DTIO: Data Stack for AI-driven Workflows
Abstract
HPC, Big Data Analytics, and Machine Learning have become increasingly intertwined as popular models such as LLMs and Diffusion Models have been driving discovery in scientific fields. However, each of these domains has its own storage infrastructure with unique I/O interfaces and storage systems, requiring feature sets that are often incompatible. Users with experience in one domain lack the expertise to change their applications to match the data stacks of the other domains, necessitating expensive conversions. There is a need for a transparent solution for the unification of disparate data stacks for the triple convergence of HPC, Big Data, and ML that can provide the required functionality while achieving higher performance. To better support converged HPC, Big Data, and ML workflows, this paper proposes DTIO, a scalable I/O runtime that unifies the disparate I/O stack for modern scientific ML workflows. DTIO utilizes a unique DataTask abstraction to express the movement of data, its ordering, and its dependencies on other data as a task. DTIO achieves a unification of scientific and ML workflows by utilizing intelligent mapping of interfaces, and automatically determines the best method to relate their unique semantics. DTIO’s online translation with DataTask caching can improve performance by 49.6% compared to offline translation methods. DTIO also offers numerous optimizations, such as asynchronous I/O and aggregation.
Finding
DTIO unifies HPC, Big Data, and ML I/O stacks through its DataTask abstraction with 9% overhead compared to offline translation, and it improves PtychoNN’s I/O performance by an average of 49.6% by avoiding producer writes.
System
- 2023–2026
DTIO
A task-based I/O runtime built with Argonne under a DOE ASCR award. Labels became composable data tasks, online format translation with caching beat offline conversion by 49.6%, and the mechanisms were reimplemented inside CLIO Core.
More papers on DTIO
Browse all papers on DTIO- 2026
Agent Error Simulator: Fault Injection for Testing Recovery in Agentic Workflows
Keith Bateman, Jaime Cernuda, Luke Logan, Bogdan Nicolae, Franck Cappello, … Anthony Kougkas. 22nd IEEE International Conference on eScience (eScience 2026)
- 2026
AgentRecall: Provenance and Rewind for Multi-Agent Workflows
Keith Bateman, Xian-He Sun, Anthony Kougkas. 13th Greater Chicago Area Systems Research Workshop (GCASR 2026)
- 2025
A Unified Data Stack for AI Workflows
Keith Bateman, Neeraj Rajesh, Jaime Cernuda, Luke Logan, Bogdan Nicolae, … Anthony Kougkas. 37th International Conference on Scalable Scientific Data Management (SSDBM 2025)
- 2024
Viper: A High-Performance I/O Framework for Transparently Updating, Storing, and Transferring Deep Neural Network Models
Jie Ye, Jaime Cernuda, Neeraj Rajesh, Keith Bateman, Orcun Yildiz, … Anthony Kougkas et al. Proceedings of the 53rd International Conference on Parallel Processing
Citation
@inproceedings{bateman2025dtio,
author = {Bateman, Keith Alex and Rajesh, Neeraj and Cernuda, Jaime and Logan, Luke and Nicolae, Bogdan and Cappello, Franck and Sun, Xian-He and Kougkas, Anthony},
booktitle = {Proceedings of the 37th International Conference on Scalable Scientific Data Management},
title = {DTIO: Data Stack for AI-driven Workflows},
year = {2025},
month = jun,
publisher = {ACM},
pages = {1--12},
doi = {10.1145/3733723.3733736},
url = {https://doi.org/10.1145/3733723.3733736},
} Related notes
- 9 min
One destination, N tiers
Every tier supplies the application at once, and the application sees one destination. The thesis from a November 2026 talk to my group at the Gnosis Research Center, the six lines of I/O work behind it since 2016, and my bet that they are the foundation agents need for context.
- 7 min
2025 reflections
A personal record of 2025: a promotion, a keynote, two dissertations, a summer of teaching, a loss, and the argument I kept making all year about what a researcher is for now.
- 8 min
From storage optimization to context orchestration
Twenty years of I/O research optimized bytes per second for programs. Agents are a different client, and each system in our lineage contributes one primitive to serving them, as argued in a seminar at Argonne.