akougkas.io

DTIO

Task-based I/O with online data transformation and provenance.

A task-based I/O runtime built with Argonne under a DOE ASCR award. Labels became composable data tasks, online format translation with caching beat offline conversion by 49.6%, and the mechanisms were reimplemented inside CLIO Core.

View the code Read the papers
The DTIO pipeline as six stages drawn as a pipe. Applications issue I/O requests; interception converts them to DataTasks; task composition, decomposition and queueing, task scheduling, and task execution follow.
The six stages of DTIO. An application's POSIX, HDF5, or NumPy call is intercepted, composed into DataTasks with explicit formats and dependencies, decomposed into queues, scheduled by locality and tier latency, and executed by workers that record provenance. Open full-size figure.

The question

HPC simulations, Big Data frameworks, and machine-learning pipelines each speak their own I/O dialect: POSIX and HDF5 on one side, NumPy and PyTorch tensors on the other. Connecting them usually means converting a whole dataset offline before the next stage can start. The conversion is pure overhead, and it serializes a pipeline that should overlap.

DTIO asked whether data movement, translation, ordering, and dependencies could be expressed as tasks that a runtime schedules, instead of files that an application waits for.

Where it came from

DTIO is what LABIOS becomes when you take the label seriously as a unit of work. A label carries an operation and a pointer; a DataTask carries an operation, its input and output formats, ordering constraints, and dependencies, so the runtime can compose and schedule it. I proposed it to DOE ASCR with Xian-He Sun, Bogdan Nicolae, and Franck Cappello at Argonne. The award began in August 2023.

Bogdan also hosts my guest appointment at Argonne, which is why DTIO was always aimed at Aurora.

The architectural bet

Treat every I/O request as a DataTask and run it through one pipeline. A shim layer intercepts POSIX, HDF5, and NumPy calls. Task composition identifies format translations, builds the dependency graph, and spots caching opportunities. Composite tasks are decomposed into atomic operations and queued. A constraint-based scheduler places them by executor load, data locality, and storage-tier latency. Executors run them on the right tier and update provenance so a failed partition can be replayed rather than recomputed.

The bet that mattered most was online translation with caching. If a consumer can read partial results while the producer is still writing, the pipeline overlaps instead of stalling.

What the paper measured

Keith Bateman led the implementation and wrote the SSDBM 2025 paper with Neeraj Rajesh, Jaime Cernuda, Luke Logan, Bogdan, Franck, Xian-He, and me.

Storing DataTasks in a circular buffer so later tasks hit local memory cut IOR read time by 89.2% on average, with write overhead at worst 12%. Staging data before reads arrived improved read performance by 88.6%, and asynchronous prefetching into client buffers pushed it to 91.7%. On the PtychoNN workflow, where a producer converts HDF5 simulation data to NumPy and a consumer trains on it as PyTorch tensors, running both concurrently through DTIO improved I/O performance by 38% at 25.6 GiB and by 65% at 6.4 GiB; the paper reports an average of 49.6%.

Where the code went

The DTIO repository preserves the published prototype at the ssdbm25 tag. In July 2025 the team reimplemented its translation, caching, staging, aggregation, and splitting mechanisms inside CLIO Core, which is the runtime IOWarp ships. The paper artifact remains frozen and citable while the mechanisms continue in maintained software.

The same task-and-provenance model now supports the threads that came after it at GRC: DTSchedule for compressed placement across tiers, AgentRecall for rewinding multi-agent workflows from provenance checkpoints, and Keith’s Agent Error Simulator, accepted at eScience 2026.

What I took from it

A task is a better unit than a request once a pipeline has more than one stage because it can carry dependencies, formats, placement constraints, and expected outputs together. DTIO brought that model from label-based I/O into IOWarp and CLIO, where online transformation, data placement, and provenance support HPC, analytics, and AI workflows through one context runtime.

Code, releases, papers, and records

  1. paper

    DTIO at SSDBM 2025

    The DataTask abstraction, the runtime, and the evaluation with IOR and the PtychoNN workflow.

  2. code

    grc-iit/DTIO

    The published prototype, preserved as the record of what the paper measured.

  3. release

    ssdbm25 tag, July 2025

    The code state behind the paper's numbers.

  4. code

    iowarp/clio-core

    Where the translation, caching, staging, aggregation, and splitting mechanisms were reimplemented in 2025.

  5. record

    GRC project record

    The institutional page, with the DOE award number and the full team.

Papers from this line of work

Browse these papers
  1. 2026
    conference

    Agent Error Simulator: Fault Injection for Testing Recovery in Agentic Workflows

    Keith Bateman, Jaime Cernuda, Luke Logan, Bogdan Nicolae, Franck Cappello, … Anthony Kougkas. 22nd IEEE International Conference on eScience (eScience 2026)

  2. 2026
    poster

    AgentRecall: Provenance and Rewind for Multi-Agent Workflows

    Keith Bateman, Xian-He Sun, Anthony Kougkas. 13th Greater Chicago Area Systems Research Workshop (GCASR 2026)

  3. 2025
    conference

    DTIO: Data Stack for AI-driven Workflows

    Keith Alex Bateman, Neeraj Rajesh, Jaime Cernuda, Luke Logan, Bogdan Nicolae, … Anthony Kougkas. Proceedings of the 37th International Conference on Scalable Scientific Data Management

  4. 2025
    poster

    A Unified Data Stack for AI Workflows

    Keith Bateman, Neeraj Rajesh, Jaime Cernuda, Luke Logan, Bogdan Nicolae, … Anthony Kougkas. 37th International Conference on Scalable Scientific Data Management (SSDBM 2025)

  5. 2024
    conference

    Viper: A High-Performance I/O Framework for Transparently Updating, Storing, and Transferring Deep Neural Network Models

    Jie Ye, Jaime Cernuda, Neeraj Rajesh, Keith Bateman, Orcun Yildiz, … Anthony Kougkas et al. Proceedings of the 53rd International Conference on Parallel Processing

  6. 2022
    conference

    LabStor: A modular and extensible platform for developing high-performance, customized I/O stacks in userspace

    Luke Logan, Jaime Cernuda Garcia, Jay Lofstead, Xian–He Sun, Anthony Kougkas. Proceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis

  7. 2020
    conference

    HReplica: A Dynamic Data Replication Engine with Adaptive Compression for Multi-Tiered Storage

    Hariharan Devarajan, Anthony Kougkas, Xian-He Sun. Proceedings of the International Conference on Big Data

  8. 2020
    conference

    HCL: Distributing Parallel Data Structures in Extreme Scales

    Hariharan Devarajan, Anthony Kougkas, Keith Bateman, Xian-He Sun. Proceedings of the International Conference on Cluster Computing

  9. 2019
    conference

    NIOBE: An intelligent I/O bridging engine for complex and distributed workflows

    Kun Feng, Hariharan Devarajan, Anthony Kougkas, Xian-He Sun. Proceedings of the International Conference on Big Data

  10. 2019
    conference

    An intelligent, adaptive, and flexible data compression framework

    Hariharan Devarajan, Anthony Kougkas, Xian-He Sun. Proceedings of the 19th International Symposium on Cluster, Cloud and Grid Computing

  11. 2018
    conference

    Vidya: Performing code-block I/O characterization for data access optimization

    Hariharan Devarajan, Anthony Kougkas, Prajwal Challa, Xian-He Sun. Proceedings of the 25th International Conference on High Performance Computing

  12. 2018
    conference

    IRIS: I/O Redirection via Integrated Storage

    Anthony Kougkas, Hariharan Devarajan, Xian-He Sun. Proceedings of the International Conference on Supercomputing

People

Related notes

  1. essay

    The label that became a patent

    US Patent 11,630,834 was granted today for label-based I/O. What a label is, how the idea went from a dissertation chapter to a best-paper award to a patent, and what a patent does not mean for research software.

    7 min
  2. update

    DOE funds DTIO

    The Department of Energy has funded DTIO, a three-year project with Argonne to turn I/O requests into schedulable data tasks. It is what LABIOS labels become when a pipeline has more than one stage.

    4 min
  3. essay

    DTIO: turning labels into schedulable data tasks

    A data task is a label with a schedule. What DTIO adds to label-based I/O, what the SSDBM 2025 paper measured, and what a task runtime still has to prove.

    7 min
  4. essay

    One destination, N tiers

    Every tier supplies the application at once, and the application sees one destination. The thesis from a November 2026 talk to my group at the Gnosis Research Center, the six lines of I/O work behind it since 2016, and my bet that they are the foundation agents need for context.

    9 min

Funding

  • 2017–present

    LABIOS

    Label-based I/O for HPC and Big Data storage systems.

  • 2024–present

    IOWarp

    Context management infrastructure for scientific AI.

  • 2025–present

    CLIO

    A context layer for scientific AI agents and workflows.