akougkas.io

DLIO: A data-centric benchmark for scientific deep learning applications

Hariharan Devarajan, Huihuo Zheng, Anthony Kougkas, Xian-He Sun, Venkatram Vishwanath

Proceedings of the 21st International Symposium on Cluster, Cloud and Internet ComputingIEEE/ACM

Award2021 IEEE/ACM CCGrid Best Paper Award

Abstract

Deep learning has been shown as a successful method for various tasks, and its popularity results in numerous open-source deep learning software tools. Deep learning has been applied to a broad spectrum of scientific domains such as cosmology, particle physics, computer vision, fusion, and astrophysics. Scientists have performed a great deal of work to optimize the computational performance of deep learning frameworks. However, the same cannot be said for I/O performance. As deep learning algorithms rely on big-data volume and variety to effectively train neural networks accurately, I/O is a significant bottleneck on large-scale distributed deep learning training. This study aims to provide a detailed investigation of the I/O behavior of various scientific deep learning workloads running on the Theta supercomputer at Argonne Leadership Computing Facility. In this paper, we present DLIO, a novel representative benchmark suite built based on the I/O profiling of the selected workloads. DLIO can be utilized to accurately emulate the I/O behavior of modern scientific deep learning applications. Using DLIO, application developers and system software solution architects can identify potential I/O bottlenecks in their applications and guide optimizations to boost the I/O performance leading to lower training times by up to 6.7x.

Finding

DLIO emulates the I/O behavior of scientific deep learning applications, helping developers find bottlenecks and lower training times by up to 6.7x.

System

  1. 2020–present
    active

    DeepIO

    A line of subcontracts with Argonne, Pacific Northwest, and Lawrence Livermore that funds students working where I/O meets AI, from Viper and KV-cache scheduling for LLM inference to WisIO's I/O bottleneck analysis.

  2. 2018–2024
    complete

    Hermes

  3. completed

    IRIS

Browse all papers on DeepIO
  1. 2026
    poster

    Extending I/O Observability to Explainability for Targeted Optimization

    Neeraj Rajesh, S. Neuwirth, Jay Lofstead, Anthony Kougkas, Xian-He Sun, Nikoli Dryden et al. International Conference for High Performance Computing, Networking, Storage, and Analysis (SC 2026)

  2. 2026
    conference

    PKAS: Predictive KVCache-Aware Scheduling for Faster LLM and Transformer Inferences

    Jie Ye, A. Maurya, K. Chitty-Venkata, Bogdan Nicolae, Anthony Kougkas, Xian-He Sun. 35th International Symposium on High-Performance Parallel and Distributed Computing (HPDC 2026)

  3. 2026
    poster

    Pre-RoPE versus Post-RoPE: Trading Per-Step Compute for KVCache Survivability

    Zia Uddin Chowdhury, Jaime Cernuda, Jie Ye, Anthony Kougkas, Xian-He Sun. 13th Greater Chicago Area Systems Research Workshop (GCASR 2026)

  4. 2026
    conference

    PagedEviction: Structured Block-wise KV Cache Pruning for Efficient Large Language Model Inference

    Krishna Teja Chitty-Venkata, Jie Ye, Siddhisanket Raskar, Xian-He Sun, Anthony Kougkas, Murali Emani et al. Findings of the Association for Computational Linguistics (EACL 2026)

Citation

@inproceedings{devarajan2021dlio,
  author = {Devarajan, Hariharan and Zheng, Huihuo and Kougkas, Anthony and Sun, Xian-He and Vishwanath, Venkatram},
  booktitle = {Proceedings of the 21st International Symposium on Cluster, Cloud and Internet Computing},
  title = {DLIO: A data-centric benchmark for scientific deep learning applications},
  year = {2021},
  month = may,
  publisher = {IEEE/ACM},
  pages = {81--91},
  doi = {10.1109/CCGrid51090.2021.00018},
  url = {https://ieeexplore.ieee.org/abstract/document/9499416},
}

Related notes

  1. essay

    When is research software shipped?

    A working definition of "shipped" for research software, built around a named state someone else can run, evidence tied to that state, explicit limits, and a continuity signal, with Hermes and DLIO as the worked examples.

    7 min
  2. essay

    Context is the new compute

    The scarce resource for scientific AI is not FLOPS or bytes but assembled, trustworthy context, and the storage we built as tanks has to become a refinery. From the AI4Sys keynote at HPDC 2025.

    8 min
  3. essay

    We need an Agent500

    Agent leaderboards score answers. Science needs a ranking that runs an agent through a real workflow with real tools and data and measures correctness, cost, I/O efficiency, and provenance together.

    7 min