DLIO: A data-centric benchmark for scientific deep learning applications
Abstract
Deep learning has been shown as a successful method for various tasks, and its popularity results in numerous open-source deep learning software tools. Deep learning has been applied to a broad spectrum of scientific domains such as cosmology, particle physics, computer vision, fusion, and astrophysics. Scientists have performed a great deal of work to optimize the computational performance of deep learning frameworks. However, the same cannot be said for I/O performance. As deep learning algorithms rely on big-data volume and variety to effectively train neural networks accurately, I/O is a significant bottleneck on large-scale distributed deep learning training. This study aims to provide a detailed investigation of the I/O behavior of various scientific deep learning workloads running on the Theta supercomputer at Argonne Leadership Computing Facility. In this paper, we present DLIO, a novel representative benchmark suite built based on the I/O profiling of the selected workloads. DLIO can be utilized to accurately emulate the I/O behavior of modern scientific deep learning applications. Using DLIO, application developers and system software solution architects can identify potential I/O bottlenecks in their applications and guide optimizations to boost the I/O performance leading to lower training times by up to 6.7x.
Finding
DLIO emulates the I/O behavior of scientific deep learning applications, helping developers find bottlenecks and lower training times by up to 6.7x.
System
- 2020–present
DeepIO
A line of subcontracts with Argonne, Pacific Northwest, and Lawrence Livermore that funds students working where I/O meets AI, from Viper and KV-cache scheduling for LLM inference to WisIO's I/O bottleneck analysis.
- 2018–2024
More papers on DeepIO
Browse all papers on DeepIO- 2026
Extending I/O Observability to Explainability for Targeted Optimization
Neeraj Rajesh, S. Neuwirth, Jay Lofstead, Anthony Kougkas, Xian-He Sun, Nikoli Dryden et al. International Conference for High Performance Computing, Networking, Storage, and Analysis (SC 2026)
- 2026
PKAS: Predictive KVCache-Aware Scheduling for Faster LLM and Transformer Inferences
Jie Ye, A. Maurya, K. Chitty-Venkata, Bogdan Nicolae, Anthony Kougkas, Xian-He Sun. 35th International Symposium on High-Performance Parallel and Distributed Computing (HPDC 2026)
- 2026
Pre-RoPE versus Post-RoPE: Trading Per-Step Compute for KVCache Survivability
Zia Uddin Chowdhury, Jaime Cernuda, Jie Ye, Anthony Kougkas, Xian-He Sun. 13th Greater Chicago Area Systems Research Workshop (GCASR 2026)
- 2026
PagedEviction: Structured Block-wise KV Cache Pruning for Efficient Large Language Model Inference
Krishna Teja Chitty-Venkata, Jie Ye, Siddhisanket Raskar, Xian-He Sun, Anthony Kougkas, Murali Emani et al. Findings of the Association for Computational Linguistics (EACL 2026)
Citation
@inproceedings{devarajan2021dlio,
author = {Devarajan, Hariharan and Zheng, Huihuo and Kougkas, Anthony and Sun, Xian-He and Vishwanath, Venkatram},
booktitle = {Proceedings of the 21st International Symposium on Cluster, Cloud and Internet Computing},
title = {DLIO: A data-centric benchmark for scientific deep learning applications},
year = {2021},
month = may,
publisher = {IEEE/ACM},
pages = {81--91},
doi = {10.1109/CCGrid51090.2021.00018},
url = {https://ieeexplore.ieee.org/abstract/document/9499416},
} Related notes
- 7 min
When is research software shipped?
A working definition of "shipped" for research software, built around a named state someone else can run, evidence tied to that state, explicit limits, and a continuity signal, with Hermes and DLIO as the worked examples.
- 8 min
Context is the new compute
The scarce resource for scientific AI is not FLOPS or bytes but assembled, trustworthy context, and the storage we built as tanks has to become a refinery. From the AI4Sys keynote at HPDC 2025.
- 7 min
We need an Agent500
Agent leaderboards score answers. Science needs a ranking that runs an agent through a real workflow with real tools and data and measures correctness, cost, I/O efficiency, and provenance together.