DTIO
Task-based I/O with online data transformation and provenance.
A task-based I/O runtime built with Argonne under a DOE ASCR award. Labels became composable data tasks, online format translation with caching beat offline conversion by 49.6%, and the mechanisms were reimplemented inside CLIO Core.
The question
HPC simulations, Big Data frameworks, and machine-learning pipelines each speak their own I/O dialect: POSIX and HDF5 on one side, NumPy and PyTorch tensors on the other. Connecting them usually means converting a whole dataset offline before the next stage can start. The conversion is pure overhead, and it serializes a pipeline that should overlap.
DTIO asked whether data movement, translation, ordering, and dependencies could be expressed as tasks that a runtime schedules, instead of files that an application waits for.
Where it came from
DTIO is what LABIOS becomes when you take the label seriously as a unit of work. A label carries an operation and a pointer; a DataTask carries an operation, its input and output formats, ordering constraints, and dependencies, so the runtime can compose and schedule it. I proposed it to DOE ASCR with Xian-He Sun, Bogdan Nicolae, and Franck Cappello at Argonne. The award began in August 2023.
Bogdan also hosts my guest appointment at Argonne, which is why DTIO was always aimed at Aurora.
The architectural bet
Treat every I/O request as a DataTask and run it through one pipeline. A shim layer intercepts POSIX, HDF5, and NumPy calls. Task composition identifies format translations, builds the dependency graph, and spots caching opportunities. Composite tasks are decomposed into atomic operations and queued. A constraint-based scheduler places them by executor load, data locality, and storage-tier latency. Executors run them on the right tier and update provenance so a failed partition can be replayed rather than recomputed.
The bet that mattered most was online translation with caching. If a consumer can read partial results while the producer is still writing, the pipeline overlaps instead of stalling.
What the paper measured
Keith Bateman led the implementation and wrote the SSDBM 2025 paper with Neeraj Rajesh, Jaime Cernuda, Luke Logan, Bogdan, Franck, Xian-He, and me.
Storing DataTasks in a circular buffer so later tasks hit local memory cut IOR read time by 89.2% on average, with write overhead at worst 12%. Staging data before reads arrived improved read performance by 88.6%, and asynchronous prefetching into client buffers pushed it to 91.7%. On the PtychoNN workflow, where a producer converts HDF5 simulation data to NumPy and a consumer trains on it as PyTorch tensors, running both concurrently through DTIO improved I/O performance by 38% at 25.6 GiB and by 65% at 6.4 GiB; the paper reports an average of 49.6%.
Where the code went
The DTIO repository preserves the published prototype at the ssdbm25 tag. In July 2025 the team reimplemented its translation, caching, staging, aggregation, and splitting mechanisms inside CLIO Core, which is the runtime IOWarp ships. The paper artifact remains frozen and citable while the mechanisms continue in maintained software.
The same task-and-provenance model now supports the threads that came after it at GRC: DTSchedule for compressed placement across tiers, AgentRecall for rewinding multi-agent workflows from provenance checkpoints, and Keith’s Agent Error Simulator, accepted at eScience 2026.
What I took from it
A task is a better unit than a request once a pipeline has more than one stage because it can carry dependencies, formats, placement constraints, and expected outputs together. DTIO brought that model from label-based I/O into IOWarp and CLIO, where online transformation, data placement, and provenance support HPC, analytics, and AI workflows through one context runtime.
Code, releases, papers, and records
- paper
DTIO at SSDBM 2025
The DataTask abstraction, the runtime, and the evaluation with IOR and the PtychoNN workflow.
- code
grc-iit/DTIO
The published prototype, preserved as the record of what the paper measured.
- release
ssdbm25 tag, July 2025
The code state behind the paper's numbers.
- code
iowarp/clio-core
Where the translation, caching, staging, aggregation, and splitting mechanisms were reimplemented in 2025.
- record
GRC project record
The institutional page, with the DOE award number and the full team.
Papers from this line of work
Browse these papers- 2026
Agent Error Simulator: Fault Injection for Testing Recovery in Agentic Workflows
Keith Bateman, Jaime Cernuda, Luke Logan, Bogdan Nicolae, Franck Cappello, … Anthony Kougkas. 22nd IEEE International Conference on eScience (eScience 2026)
- 2026
AgentRecall: Provenance and Rewind for Multi-Agent Workflows
Keith Bateman, Xian-He Sun, Anthony Kougkas. 13th Greater Chicago Area Systems Research Workshop (GCASR 2026)
- 2025
DTIO: Data Stack for AI-driven Workflows
Keith Alex Bateman, Neeraj Rajesh, Jaime Cernuda, Luke Logan, Bogdan Nicolae, … Anthony Kougkas. Proceedings of the 37th International Conference on Scalable Scientific Data Management
- 2025
A Unified Data Stack for AI Workflows
Keith Bateman, Neeraj Rajesh, Jaime Cernuda, Luke Logan, Bogdan Nicolae, … Anthony Kougkas. 37th International Conference on Scalable Scientific Data Management (SSDBM 2025)
- 2024
Viper: A High-Performance I/O Framework for Transparently Updating, Storing, and Transferring Deep Neural Network Models
Jie Ye, Jaime Cernuda, Neeraj Rajesh, Keith Bateman, Orcun Yildiz, … Anthony Kougkas et al. Proceedings of the 53rd International Conference on Parallel Processing
- 2022
LabStor: A modular and extensible platform for developing high-performance, customized I/O stacks in userspace
Luke Logan, Jaime Cernuda Garcia, Jay Lofstead, Xian–He Sun, Anthony Kougkas. Proceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis
- 2020
HReplica: A Dynamic Data Replication Engine with Adaptive Compression for Multi-Tiered Storage
Hariharan Devarajan, Anthony Kougkas, Xian-He Sun. Proceedings of the International Conference on Big Data
- 2020
HCL: Distributing Parallel Data Structures in Extreme Scales
Hariharan Devarajan, Anthony Kougkas, Keith Bateman, Xian-He Sun. Proceedings of the International Conference on Cluster Computing
- 2019
NIOBE: An intelligent I/O bridging engine for complex and distributed workflows
Kun Feng, Hariharan Devarajan, Anthony Kougkas, Xian-He Sun. Proceedings of the International Conference on Big Data
- 2019
An intelligent, adaptive, and flexible data compression framework
Hariharan Devarajan, Anthony Kougkas, Xian-He Sun. Proceedings of the 19th International Symposium on Cluster, Cloud and Grid Computing
- 2018
Vidya: Performing code-block I/O characterization for data access optimization
Hariharan Devarajan, Anthony Kougkas, Prajwal Challa, Xian-He Sun. Proceedings of the 25th International Conference on High Performance Computing
- 2018
IRIS: I/O Redirection via Integrated Storage
Anthony Kougkas, Hariharan Devarajan, Xian-He Sun. Proceedings of the International Conference on Supercomputing
People
- Keith Bateman First author of the SSDBM'25 paper; lead developer
- Xian-He Sun SSDBM'25 co-author
- Bogdan Nicolae Co-PI, Argonne
- Franck Cappello Co-PI, Argonne
- Neeraj Rajesh Researcher; co-author
- Jaime Cernuda Researcher; co-author
- Luke Logan Researcher; the CLIO Core port
- Rajni Pawar Researcher
Related notes
- 7 min
The label that became a patent
US Patent 11,630,834 was granted today for label-based I/O. What a label is, how the idea went from a dissertation chapter to a best-paper award to a patent, and what a patent does not mean for research software.
- 4 min
DOE funds DTIO
The Department of Energy has funded DTIO, a three-year project with Argonne to turn I/O requests into schedulable data tasks. It is what LABIOS labels become when a pipeline has more than one stage.
- 7 min
DTIO: turning labels into schedulable data tasks
A data task is a label with a schedule. What DTIO adds to label-based I/O, what the SSDBM 2025 paper measured, and what a task runtime still has to prove.
- 9 min
One destination, N tiers
Every tier supplies the application at once, and the application sees one destination. The thesis from a November 2026 talk to my group at the Gnosis Research Center, the six lines of I/O work behind it since 2016, and my bet that they are the foundation agents need for context.