The benchmark stack
-
Defines workload configurations and the process.
-
Generates the configured I/O workload.
-
The submitter’s storage runs the workload.
MLPerf Storage now uses the DLIO Benchmark to generate the I/O behavior of deep-learning training workloads. That is the whole announcement, and I am delighted by it.
What DLIO does
Deep-learning training does not stress storage in one universal way. Datasets, worker counts, epochs, checkpoints, metadata operations, and framework-specific access patterns all change the picture. Running an entire training stack to study each storage design is expensive and hard to reproduce.
DLIO makes that behavior configurable. You describe the pattern you want to exercise and it generates that pattern, repeatably, without the model. The CCGrid 2021 paper explains the design and the experiments behind it; it won the best paper award, which was a good day.
Why the MLPerf integration matters to me
A benchmark becomes real when a second community uses it to compare systems they built. MLPerf Storage has to exercise storage under defined workload configurations, and its public implementation names DLIO as the generator for those paths. Anyone can inspect the dependency, the configurations, and the rules.
Three things I take from this. DLIO left the environment that produced the paper; other people install it, configure it, and change it without its authors’ private context. Integration sharpened the interfaces, because assumptions that felt harmless in one lab became bugs across submitters’ platforms. And the benchmark now sits in a longer evidence chain: MLPerf defines the process, DLIO produces the workload, a submitter supplies the system.
Credit
DLIO is Hariharan Devarajan’s work, built with Huihuo Zheng and Venkatram Vishwanath at Argonne; I was part of the Illinois Tech team that co-advised Hari. Its growth into shared infrastructure, including its place in MLPerf Storage, is the work of Hari, the Argonne team, contributors, and the MLCommons Storage community. Being part of MLPerf does not make DLIO a standard by declaration; it means the code was useful enough for another community to build on. That is the outcome I want from research software.