akougkas.io

DeepIO

Data movement and I/O analysis for scientific AI.

A line of subcontracts with Argonne, Pacific Northwest, and Lawrence Livermore that funds students working where I/O meets AI, from Viper and KV-cache scheduling for LLM inference to WisIO's I/O bottleneck analysis.

View the code Read the papers
The CCGrid 2021 Best Paper Award, First Prize, certificate for DLIO, a data-centric benchmark for scientific deep learning applications, presented to Hariharan Devarajan, Huihuo Zheng, Anthony Kougkas, Xian-He Sun, and Venkatram Vishwanath.
The CCGrid 2021 Best Paper Award certificate for DLIO, naming Hariharan Devarajan, Huihuo Zheng, me, Xian-He Sun, and Venkatram Vishwanath.

The line

DeepIO is a series of subcontracts with three national laboratories. Argonne funded DeepMeta from 2022, then DeepIO and its 2026 renewal; Pacific Northwest funded DFlow from 2022 to 2024; Lawrence Livermore funded WisIO from 2023 and IntentIO in 2026. Each one widened the line, and together they paid for students who wanted to work where I/O meets AI.

The question they share is older than any of them. Around 2020 every AI-for-science conversation ended at the same place: the GPUs were waiting for data, and nobody could say precisely why. Training, inference, and the diagnosis of both depend on a data path that crosses storage, memory, accelerators, and files, and DeepIO studies that path as one system.

Where it started

DLIO is Hariharan Devarajan’s benchmark. He built it with Huihuo Zheng and Venkatram Vishwanath at the Argonne Leadership Computing Facility, profiling cosmology, particle-physics, and climate workloads on Theta and turning them into a configurable emulator, and I was part of the Illinois Tech team that co-advised him. The CCGrid 2021 paper won the Best Paper Award. Hari’s Stimulus followed at CCGrid 2022, connecting scientific formats to AI frameworks and speeding up ingestion by up to 5.3x on Summit.

What DLIO became afterward is Argonne’s work, and mostly Hari’s. MLPerf Storage from MLCommons runs its workloads through dlio_benchmark, and the benchmark is maintained at argonne-lcf/dlio_benchmark, with v2.0.1 as the current release.

The students’ work

TunIO, Neeraj Rajesh’s IPDPS 2024 work, searched layered I/O configurations automatically. Jie Ye moved the question to inference. Viper at ICPP 2024 schedules and transfers model updates between concurrent training and inference through memory, cutting model-update latency by roughly 9x. UnboxKV at IPDPS 2025, with Bogdan Nicolae at Argonne, characterized how KV caches behave under concurrent transformer inference, and PKAS at HPDC 2026 used predicted cache demand to schedule requests, up to 7.34x more throughput than the schedulers it was compared against.

I/O characterization

The Livermore subcontracts fund I/O characterization, with Kathryn Mohror as the partner. DFTracer, Hari’s SC 2024 tracer, captures AI workflow activity at 1 to 5% overhead. WisIO, Izzet Yildirim’s ICS 2025 system, analyzes multi-terabyte traces through time, process, and file views and explains bottlenecks rather than only counting them: up to 805x more bottleneck coverage than a single view, and up to 11x faster analysis than the tools it was compared against. WisIO v0.2.0 shipped publicly in July 2026. IntentIO, the 2026 follow-on with Nikoli Dryden, extracts I/O intent from training runs with machine learning.

Scientific AI as a data system

These threads meet at the same systems boundary. Training depends on the path from storage to accelerators. Concurrent inference depends on model movement, cache placement, and scheduling. Diagnosis depends on traces that connect application behavior to devices and files. DeepIO treats GPUs, memory, KV caches, storage, and provenance as one data path rather than a collection of isolated optimizations.

That systems view is DeepIO’s contribution to IOWarp and CLIO. Context management for scientific AI has to carry data, model state, intent, topology, and history through the same runtime decisions.

Where it stands

DeepIO is active. The 2026 Argonne renewal and the Livermore IntentIO award run this year, and Isa Muradli and Jie are studying GPU-attached storage and dynamic tensor routing for multi-GPU inference. DLIO continues at Argonne with its community. The results belong to the people whose names are on the papers.

Code, releases, papers, and records

  1. paper

    Viper at ICPP 2024

    Jie Ye's framework for moving model updates between concurrent training and inference through memory.

  2. paper

    UnboxKV at IPDPS 2025

    Characterizing KV-cache behavior under concurrent transformer inference, with Bogdan Nicolae.

  3. paper

    WisIO at ICS 2025

    Izzet Yildirim's multi-perspective I/O bottleneck analysis, built with Lawrence Livermore.

  4. paper

    DFTracer at SC 2024

    The analysis-friendly tracer for AI-driven workflows that feeds WisIO.

  5. paper

    DLIO at CCGrid 2021

    Hariharan Devarajan's data-centric benchmark for scientific deep learning I/O. Best Paper Award.

  6. code

    argonne-lcf/dlio_benchmark

    The benchmark, maintained at the Argonne Leadership Computing Facility with the community. v2.0.1 is current.

  7. code

    mlcommons/storage

    The MLPerf Storage benchmark suite. Its README documents that the simulation runs through dlio_benchmark.

  8. press

    ALCF news, July 2021

    The national laboratory announcement of DLIO winning Best Paper at CCGrid 2021.

  9. press

    HPCwire, July 2021

    Industry coverage of the ALCF and Illinois Tech DLIO benchmark and award.

  10. record

    GRC project record

    The project's page at the Gnosis Research Center.

Papers from this line of work

Browse these papers
  1. 2026
    poster

    Extending I/O Observability to Explainability for Targeted Optimization

    Neeraj Rajesh, S. Neuwirth, Jay Lofstead, Anthony Kougkas, Xian-He Sun, Nikoli Dryden et al. International Conference for High Performance Computing, Networking, Storage, and Analysis (SC 2026)

  2. 2026
    conference

    PKAS: Predictive KVCache-Aware Scheduling for Faster LLM and Transformer Inferences

    Jie Ye, A. Maurya, K. Chitty-Venkata, Bogdan Nicolae, Anthony Kougkas, Xian-He Sun. 35th International Symposium on High-Performance Parallel and Distributed Computing (HPDC 2026)

  3. 2026
    poster

    Pre-RoPE versus Post-RoPE: Trading Per-Step Compute for KVCache Survivability

    Zia Uddin Chowdhury, Jaime Cernuda, Jie Ye, Anthony Kougkas, Xian-He Sun. 13th Greater Chicago Area Systems Research Workshop (GCASR 2026)

  4. 2026
    conference

    PagedEviction: Structured Block-wise KV Cache Pruning for Efficient Large Language Model Inference

    Krishna Teja Chitty-Venkata, Jie Ye, Siddhisanket Raskar, Xian-He Sun, Anthony Kougkas, Murali Emani et al. Findings of the Association for Computational Linguistics (EACL 2026)

  5. 2025
    poster

    Insights into GPUDirect Data Transfer through NIXL Benchmarking

    Isa Muradli, Jie Ye, Luke Logan, Anthony Kougkas, Xian-He Sun. 21st IEEE International Conference on eScience (eScience ’25), September 15-18, 2025

  6. 2025
    conference

    Characterizing the Behavior and Impact of KV Caching on Transformer Inferences under Concurrency

    Jie Ye, Jaime Cernuda, Avinash Maurya, Xian-He Sun, Anthony Kougkas, Bogdan Nicolae. IPDPS’25: The 39th IEEE International Parallel and Distributed Processing Symposium

  7. 2025
    poster

    Accelerate LLM inference with Asynchronous model offload

    Jie Ye, Bogdan Nicolae, Xian-He Sun, Anthony Kougkas. 37th International Conference on Scalable Scientific Data Management (SSDBM 2025)

  8. 2024
    conference

    DFTracer: An Analysis-Friendly Data Flow Tracer for AI-Driven Workflows

    Hariharan Devarajan, Loïc Pottier, Kaushik Velusamy, Huihuo Zheng, Izzet Yildirim, … Anthony Kougkas et al. Proceedings of the International Conference for High Performance Computing, Networking, Storage, and Analysis

  9. 2024
    poster

    Uncover the Overhead and Resource Usage for Handling KV Cache Overflow in LLM Inference

    Jie Ye, Bogdan Nicolae, Anthony Kougkas, Xian-He Sun. The International Conference for High Performance Computing, Networking, Storage, and Analysis (SC’24)

  10. 2024
    conference

    DaYu: Optimizing Distributed Scientific Workflows by Decoding Dataflow Semantics and Dynamics

    Meng Tang, Jaime Cernuda, Jie Ye, Luanzheng Guo, Nathan R. Tallent, Anthony Kougkas et al. Proceedings of the IEEE International Conference on Cluster Computing

  11. 2024
    journal

    An Evaluation of DAOS for Simulation and Deep Learning HPC Workloads

    Luke Logan, Jay Lofstead, Xian-He Sun, Anthony Kougkas. ACM SIGOPS Operating Systems Review

  12. 2024
    conference

    Viper: A High-Performance I/O Framework for Transparently Updating, Storing, and Transferring Deep Neural Network Models

    Jie Ye, Jaime Cernuda, Neeraj Rajesh, Keith Bateman, Orcun Yildiz, … Anthony Kougkas et al. Proceedings of the 53rd International Conference on Parallel Processing

  13. 2024
    conference

    TunIO: An AI-powered Framework for Optimizing HPC I/O

    Neeraj Rajesh, Keith Bateman, Jean Luca Bez, Suren Byna, Anthony Kougkas, Xian-He Sun. Proceedings of the International Parallel and Distributed Processing Symposium

  14. 2023
    workshop

    An Evaluation of DAOS for Simulation and Deep Learning HPC Workloads

    Luke Logan, Jay Lofstead, Xian-He Sun, Anthony Kougkas. Proceedings of the 3rd Workshop on Challenges and Opportunities of Efficient and Performant Storage Systems

  15. 2022
    conference

    Stimulus: Accelerate Data Management for Scientific AI applications in HPC

    Hariharan Devarajan, Anthony Kougkas, Huihuo Zheng, Venkatram Vishwanath, Xian-He Sun. Proceedings of the 22nd IEEE International Symposium on Cluster, Cloud and Internet Computing

  16. 2021
    poster

    HDF5 VOL Connector to Apache Arrow

    Jie Ye, Anthony Kougkas, Xian-He Sun. The 2021 International Conference for High Performance Computing, Networking, Storage, and Analysis (SC’21), November 14–19, 2021

  17. 2021
    conference

    DLIO: A data-centric benchmark for scientific deep learning applications

    Hariharan Devarajan, Huihuo Zheng, Anthony Kougkas, Xian-He Sun, Venkatram Vishwanath. Proceedings of the 21st International Symposium on Cluster, Cloud and Internet Computing

    Award: 2021 IEEE/ACM CCGrid Best Paper Award

People

Related notes

  1. update

    DLIO joins MLPerf Storage

    MLPerf Storage now uses DLIO, Hariharan Devarajan's deep-learning I/O benchmark maintained at Argonne, to generate its workloads. A research tool found a second life.

    2 min
  2. essay

    When is research software shipped?

    A working definition of "shipped" for research software, built around a named state someone else can run, evidence tied to that state, explicit limits, and a continuity signal, with Hermes and DLIO as the worked examples.

    7 min
  3. essay

    Context is the new compute

    The scarce resource for scientific AI is not FLOPS or bytes but assembled, trustworthy context, and the storage we built as tanks has to become a refinery. From the AI4Sys keynote at HPDC 2025.

    8 min
  4. field note

    Reflecting on SC25

    What I noticed at SC25 in St. Louis, what the group brought, what a year of running the PDSW program committee taught me about how a workshop gets made, and a few notes for people attending their first SC.

    8 min
  5. essay

    We need an Agent500

    Agent leaderboards score answers. Science needs a ranking that runs an agent through a real workflow with real tools and data and measures correctness, cost, I/O efficiency, and provenance together.

    7 min

Funding

  • 2024–present

    IOWarp

    Context management infrastructure for scientific AI.

  • 2025–present

    CLIO

    A context layer for scientific AI agents and workflows.

  • 2023–2026

    DTIO

    Task-based I/O with online data transformation and provenance.