# DTIO

https://akougkas.io/work/dtio/

Author: Anthony Kougkas

Updated: 2026-08-27

## The question

HPC simulations, Big Data frameworks, and machine-learning pipelines each speak their own I/O dialect: POSIX and HDF5 on one side, NumPy and PyTorch tensors on the other. Connecting them usually means converting a whole dataset offline before the next stage can start. The conversion is pure overhead, and it serializes a pipeline that should overlap.

DTIO asked whether data movement, translation, ordering, and dependencies could be expressed as tasks that a runtime schedules, instead of files that an application waits for.

## Where it came from

DTIO is what [LABIOS](https://akougkas.io/work/labios/) becomes when you take the label seriously as a unit of work. A label carries an operation and a pointer; a DataTask carries an operation, its input and output formats, ordering constraints, and dependencies, so the runtime can compose and schedule it. I proposed it to [DOE ASCR](https://www.energy.gov/science/ascr/advanced-scientific-computing-research) with [Xian-He Sun](https://www.iit.edu/directory/people/xian-he-sun), [Bogdan Nicolae](https://www.bnicolae.net/), and [Franck Cappello](https://franckcappello.github.io/) at [Argonne](https://www.anl.gov/). The award began in August 2023.

Bogdan also hosts my guest appointment at Argonne, which is why DTIO was always aimed at Aurora.

## The architectural bet

Treat every I/O request as a DataTask and run it through one pipeline. A shim layer intercepts POSIX, HDF5, and NumPy calls. Task composition identifies format translations, builds the dependency graph, and spots caching opportunities. Composite tasks are decomposed into atomic operations and queued. A constraint-based scheduler places them by executor load, data locality, and storage-tier latency. Executors run them on the right tier and update provenance so a failed partition can be replayed rather than recomputed.

The bet that mattered most was online translation with caching. If a consumer can read partial results while the producer is still writing, the pipeline overlaps instead of stalling.

## What the paper measured

[Keith Bateman](https://grc.iit.edu/center/team/keith-bateman/) led the implementation and wrote the [SSDBM 2025 paper](https://akougkas.io/publications/bateman2025dtio/) with [Neeraj Rajesh](https://grc.iit.edu/center/team/neeraj-rajesh/), [Jaime Cernuda](https://grc.iit.edu/center/team/jaime-cernuda/), [Luke Logan](https://grc.iit.edu/center/team/luke-logan/), Bogdan, Franck, Xian-He, and me.

Storing DataTasks in a circular buffer so later tasks hit local memory cut IOR read time by 89.2% on average, with write overhead at worst 12%. Staging data before reads arrived improved read performance by 88.6%, and asynchronous prefetching into client buffers pushed it to 91.7%. On the PtychoNN workflow, where a producer converts HDF5 simulation data to NumPy and a consumer trains on it as PyTorch tensors, running both concurrently through DTIO improved I/O performance by 38% at 25.6 GiB and by 65% at 6.4 GiB; the paper reports an average of 49.6%.

## Where the code went

The [DTIO repository](https://github.com/grc-iit/DTIO) preserves the published prototype at the [`ssdbm25`](https://github.com/grc-iit/dtio/releases/tag/ssdbm25) tag. In July 2025 the team reimplemented its translation, caching, staging, aggregation, and splitting mechanisms inside [CLIO Core](https://akougkas.io/work/clio/), which is the runtime [IOWarp](https://akougkas.io/work/iowarp/) ships. The paper artifact remains frozen and citable while the mechanisms continue in maintained software.

The same task-and-provenance model now supports the threads that came after it at GRC: DTSchedule for compressed placement across tiers, AgentRecall for rewinding multi-agent workflows from provenance checkpoints, and Keith’s [Agent Error Simulator](https://github.com/Keith-Bateman/agent-error-simulator), [accepted at eScience 2026](https://akougkas.io/publications/bateman2026aes/).

## What I took from it

A task is a better unit than a request once a pipeline has more than one stage because it can carry dependencies, formats, placement constraints, and expected outputs together. DTIO brought that model from label-based I/O into [IOWarp](https://akougkas.io/work/iowarp/) and [CLIO](https://akougkas.io/work/clio/), where online transformation, data placement, and provenance support HPC, analytics, and AI workflows through one context runtime.
