DTIO

The question

HPC simulations, Big Data frameworks, and machine-learning pipelines each speak their own I/O dialect: POSIX and HDF5 on one side, NumPy and PyTorch tensors on the other. Connecting them usually means converting a whole dataset offline before the next stage can start. The conversion is pure overhead, and it serializes a pipeline that should overlap.

DTIO asked whether data movement, translation, ordering, and dependencies could be expressed as tasks that a runtime schedules, instead of files that an application waits for.

Where it came from

DTIO is what LABIOS becomes when you take the label seriously as a unit of work. A label carries an operation and a pointer; a DataTask carries an operation, its input and output formats, ordering constraints, and dependencies, so the runtime can compose and schedule it. I proposed it to DOE ASCR with Xian-He Sun as PI and Bogdan Nicolae and Franck Cappello at Argonne as co-PIs, and it was awarded in August 2023.

Bogdan is also my host as Guest Research Faculty at Argonne, which is why DTIO was always aimed at Aurora.

The architectural bet

Treat every I/O request as a DataTask and run it through one pipeline. A shim layer intercepts POSIX, HDF5, and NumPy calls. Task composition identifies format translations, builds the dependency graph, and spots caching opportunities. Composite tasks are decomposed into atomic operations and queued. A constraint-based scheduler places them by executor load, data locality, and storage-tier latency. Executors run them on the right tier and update provenance so a failed partition can be replayed rather than recomputed.

The bet that mattered most was online translation with caching. If a consumer can read partial results while the producer is still writing, the pipeline overlaps instead of stalling.

What the paper measured

Keith Bateman led the implementation and wrote the SSDBM 2025 paper with Neeraj Rajesh, Jaime Cernuda, Luke Logan, Bogdan, Franck, Xian-He, and me. The numbers in it are the ones I am willing to repeat.

Storing DataTasks in a circular buffer so later tasks hit local memory cut IOR read time by 89.2% on average, with write overhead at worst 12%. Staging data before reads arrived improved read performance by 88.6%, and asynchronous prefetching into client buffers pushed it to 91.7%. On the PtychoNN workflow, where a producer converts HDF5 simulation data to NumPy and a consumer trains on it as PyTorch tensors, running both concurrently through DTIO improved I/O performance by 38% at 25.6 GiB and 65% at 6.4 GiB, an average of 49.6%.

Where the code went

The DTIO repository preserves the published prototype at the ssdbm25 tag. In July 2025 the team reimplemented its translation, caching, staging, aggregation, and splitting mechanisms inside CLIO Core, which is the runtime IOWarp ships. That is the ending I wanted: the paper artifact stays frozen and citable, and the mechanisms live in software with users.

The same task-and-provenance model now supports the threads that came after it at GRC: DTSchedule for compressed placement across tiers, AgentRecall for rewinding multi-agent workflows from provenance checkpoints, and Keith’s Agent Error Simulator, accepted at eScience 2026.

What I took from it

Two things. First, “task” is a better unit than “request” once the pipeline has more than one stage, because a task can carry its dependencies and its expected outputs. That is the same reason a dispatch in Clio Coder carries typed intent. Second, a research prototype should have an explicit successor before it is retired. DTIO’s mechanisms did not disappear when the award ended; they moved, and the move is on the record.