DLIO joins MLPerf Storage

MLPerf Storage now uses the DLIO Benchmark to generate the I/O behavior of deep-learning training workloads. That is the whole announcement, and I am delighted by it.

What DLIO does

Deep-learning training does not stress storage in one universal way. Datasets, worker counts, epochs, checkpoints, metadata operations, and framework-specific access patterns all change the picture. Running an entire training stack to study each storage design is expensive and hard to reproduce.

DLIO makes that behavior configurable. You describe the pattern you want to exercise and it generates that pattern, repeatably, without the model. The CCGrid 2021 paper explains the design and the experiments behind it; it won the best paper award, which was a good day.

Why the MLPerf integration matters to me

A benchmark becomes real when a second community uses it to compare systems they built. MLPerf Storage has to exercise storage under defined workload configurations, and its public implementation names DLIO as the generator for those paths. Anyone can inspect the dependency, the configurations, and the rules.

Three things I take from this. DLIO left the environment that produced the paper; other people install it, configure it, and change it without our private context. Integration sharpened the interfaces, because assumptions that felt harmless in one lab became bugs across submitters’ platforms. And the benchmark now sits in a longer evidence chain: MLPerf defines the process, DLIO produces the workload, a submitter supplies the system.

Credit

DLIO started as collaborative research by Hariharan Devarajan, Huihuo Zheng, me, Xian-He Sun, and Venkatram Vishwanath. My part was shaping the benchmark and the systems argument around it. The Argonne team, contributors, and the MLCommons Storage community did the less visible work of turning a research benchmark into shared infrastructure. Being part of MLPerf does not make DLIO a standard by declaration; it means the code was useful enough for another community to build on. That is the outcome I want from research software.