akougkas.io

DLIO joins MLPerf Storage

MLPerf Storage now uses DLIO, Hariharan Devarajan's deep-learning I/O benchmark maintained at Argonne, to generate its workloads. A research tool found a second life.

Published
Updated
Reading
2 min read
Topics
systems, data, scientific computing, building

The benchmark stack

  1. MLPerf Storage

    Defines workload configurations and the process.

  2. DLIO

    Generates the configured I/O workload.

  3. System under test

    The submitter’s storage runs the workload.

MLPerf defines the process, DLIO generates the workload, and a submitter supplies the storage system under test.

MLPerf Storage now uses the DLIO Benchmark to generate the I/O behavior of deep-learning training workloads. That is the whole announcement, and I am delighted by it.

What DLIO does

Deep-learning training does not stress storage in one universal way. Datasets, worker counts, epochs, checkpoints, metadata operations, and framework-specific access patterns all change the picture. Running an entire training stack to study each storage design is expensive and hard to reproduce.

DLIO makes that behavior configurable. You describe the pattern you want to exercise and it generates that pattern, repeatably, without the model. The CCGrid 2021 paper explains the design and the experiments behind it; it won the best paper award, which was a good day.

Why the MLPerf integration matters to me

A benchmark becomes real when a second community uses it to compare systems they built. MLPerf Storage has to exercise storage under defined workload configurations, and its public implementation names DLIO as the generator for those paths. Anyone can inspect the dependency, the configurations, and the rules.

Three things I take from this. DLIO left the environment that produced the paper; other people install it, configure it, and change it without its authors’ private context. Integration sharpened the interfaces, because assumptions that felt harmless in one lab became bugs across submitters’ platforms. And the benchmark now sits in a longer evidence chain: MLPerf defines the process, DLIO produces the workload, a submitter supplies the system.

Credit

DLIO is Hariharan Devarajan’s work, built with Huihuo Zheng and Venkatram Vishwanath at Argonne; I was part of the Illinois Tech team that co-advised Hari. Its growth into shared infrastructure, including its place in MLPerf Storage, is the work of Hari, the Argonne team, contributors, and the MLCommons Storage community. Being part of MLPerf does not make DLIO a standard by declaration; it means the code was useful enough for another community to build on. That is the outcome I want from research software.

Sources, code, and records

  1. source

    MLPerf Storage repository

    The benchmark implementation that names DLIO as its workload generator.

  2. source

    DLIO Benchmark repository

    Source, documentation, releases, and contributors.

  3. source

    DLIO paper

    The CCGrid 2021 paper that defined the benchmark and won the best paper award, with the DOI and PDF.

  4. source

    MLPerf Storage at MLCommons

    The benchmark suite's page at MLCommons.

  • 2020–present
    active

    DeepIO

    Data movement and I/O analysis for scientific AI

More on DeepIO

  1. essay

    When is research software shipped?

    A working definition of "shipped" for research software, built around a named state someone else can run, evidence tied to that state, explicit limits, and a continuity signal, with Hermes and DLIO as the worked examples.

    6 min
  2. essay

    We need an Agent500

    Agent leaderboards score answers. Science needs a ranking that runs an agent through a real workflow with real tools and data and measures correctness, cost, I/O efficiency, and provenance together.

    7 min
  3. field note

    Reflecting on SC25

    What I noticed at SC25 in St. Louis, what the group brought, what a year of running the PDSW program committee taught me about how a workshop gets made, and a few notes for people attending their first SC.

    8 min