# One destination, N tiers

https://akougkas.io/journal/2026/one-destination-n-tiers/

Author: Anthony Kougkas

Published: 2026-11-13

On Wednesday, November 11, I gave a talk to my group at the [Gnosis Research Center](https://grc.iit.edu/) about what happens after the data leaves memory. Most of the room builds I/O systems, and many of them are building the newest pieces of the line I describe here. The talk had one thesis, and I put it on a slide by itself: every tier supplies the application at once. The application sees one destination.

[Download the slides (PDF)](https://akougkas.io/assets/pdf/kougkas-one-destination-n-tiers-2026-11.pdf)

At a glance

Anchor

GRC group meeting, November 11, 2026

Systems

[Hermes](https://akougkas.io/work/hermes/), [LABIOS](https://akougkas.io/work/labios/), [DTIO](https://akougkas.io/work/dtio/), [ChronoLog](https://akougkas.io/work/chronolog/), [IOWarp](https://akougkas.io/work/iowarp/), [CLIO](https://akougkas.io/work/clio/)

The claim

Six lines of I/O work converge on context, and agents are the client they were waiting for.

Slides

[28 pages, PDF](https://akougkas.io/assets/pdf/kougkas-one-destination-n-tiers-2026-11.pdf)

## The hierarchy nobody chooses

None of us has ever chosen which cache level a variable lives in. That invisibility is forty years of architecture work, some of it done in our own lab, and it holds because one architect designs every level of one machine.

Below DRAM there is no such invisibility. From IBM’s first disk drive in 1956 until flash reached the data center, there was disk and then tape. Then four families of hardware arrived: flash, NVMe, and burst buffers; byte-addressable persistent memory such as Optane; disaggregated memory and storage over CXL, RDMA, and NVMe over Fabrics; and storage attached directly to accelerators through HBM and GPUDirect Storage. Software support arrived one device at a time. Intel wound Optane down in 2022, and software written against Optane went with it. Software written against the hierarchy kept running.

An I/O tier is not another cache level. It is shared by many jobs and many owners, it fails on its own, it is persistent and carries durability semantics, and it answers in microseconds to seconds instead of nanoseconds. The principle is the one the lab’s memory-systems work states for the inside of the machine: match the rate of requests to the rate of supply at every layer. Same principle, different problem, different machinery.

## Where the line started

In 2015 and 2016, Cray’s DataWarp burst buffers arrived on Cori and Trinity. A new tier showed up on leadership machines and the first question was who manages it. Jobs interfered with each other on the shared tier, and [BBIO](https://akougkas.io/publications/bbio/), my eScience 2016 paper with Matthieu Dorier, Rob Latham, and Rob Ross at Argonne, coordinated burst buffers to prevent that interference and improved performance by up to 2x. [Harmonia](https://akougkas.io/publications/kougkas2018harmonia/) followed as an interference-aware scheduler for shared burst buffers, and [IRIS](https://akougkas.io/publications/kougkas2018iris/) put files and objects behind one interface, which [Kun Feng](https://grc.iit.edu/center/team/kun-feng/) carried into [NIOBE](https://akougkas.io/publications/feng2019niobe/).

Single tiers had managers: DataWarp for the burst buffer, Data Elevator between burst buffer and file system, hierarchical storage management for tape. Each of our early papers also tamed one tier or one interface. The machines already had five tiers, and nothing managed the hierarchy as one.

## Six lines

The thesis rests on six ideas. Each one has a name, a system, and people who built it.

| Line | The question | Systems | Built by |
| --- | --- | --- | --- |
| Placement | Which tier holds the data now? | Hermes, then the CLIO Context Transfer Engine | Hari Devarajan and I; runtime by Luke Logan |
| Intent | What does the application want? | LABIOS | Hari and I, with Jay Lofstead and Kathryn Mohror |
| Task | What work travels with the data? | DTIO, now in CLIO Core | Keith Bateman, with Bogdan Nicolae and Franck Cappello |
| Order | What happened first? | ChronoLog | Hari and I first; Kun Feng and Eneko Gonzalez wrote most of today’s code |
| Runtime | What executes the decision? | The Hermes runtime, LabStor, Chimaera, the CLIO Runtime | Luke Logan |
| Measurement | What is actually happening? | DLIO, DFTracer, WisIO | Hari, with Argonne and Livermore; Izzet Yildirim |

**Placement.** [Hermes](https://akougkas.io/work/hermes/), the multi-tier I/O buffering system in our [HPDC 2018 paper](https://akougkas.io/publications/kougkas2018hermes/), treats RAM, NVMe, burst buffers, and the parallel file system as one buffer. Placement is solved per request, not fixed per device: if a request fits in a layer, place it; otherwise skip to the next layer, split it, or flush and recurse on what remains. Hermes was 8x faster than no buffering and more than 2x faster than Data Elevator and DataWarp, and it needed no application changes. Once tiers exist, every classic optimization gains a new variable, which tier, and [Hariharan Devarajan](https://grc.iit.edu/center/team/hariharan-devarajan/)’s [HFetch](https://akougkas.io/publications/devarajan2020hfetch/), [HCompress](https://akougkas.io/publications/devarajan2020hcompress/), and [HReplica](https://akougkas.io/publications/devarajan2020hreplica/) answered it for prefetching, compression, and replication.

**Intent.** Every one of those algorithms inferred what the application wanted from its I/O calls. POSIX throws that away. [LABIOS](https://akougkas.io/work/labios/), our label-based I/O system ([HPDC 2019](https://akougkas.io/publications/kougkas2019labios/), with [Jay Lofstead](https://www.lofstead.org/) and [Kathryn Mohror](https://people.llnl.gov/people/mohror1)), puts it back. A label is an operation, a data pointer, and flags, and once I/O is labels, storage becomes a scheduling problem. I used to call labels an optimization. I now see them as intent preserved across a system boundary.

**Task.** [DTIO](https://akougkas.io/work/dtio/), [Keith Bateman](https://grc.iit.edu/center/team/keith-bateman/)’s task-based I/O system with [Bogdan Nicolae](https://www.bnicolae.net/) and [Franck Cappello](https://franckcappello.github.io/) at Argonne, turns the label into a DataTask that carries formats, ordering, and dependencies, so translation runs online and producer and consumer overlap. On PtychoNN it cut time by 49.6 percent against offline translation ([SSDBM 2025](https://akougkas.io/publications/bateman2025dtio/)). Its mechanisms now live in CLIO Core.

**Order.** [ChronoLog](https://akougkas.io/work/chronolog/), a distributed shared log ordered by physical time ([MSST 2020](https://akougkas.io/publications/kougkas2020chronolog/)), asks what happens if the data carries its own placement key. Physical time is the log sequence number, so a writer knows where its entry goes without asking a sequencer. The prototype sustained over a million tail operations per second. The log’s age order matches the hierarchy’s speed order, newest in RAM and oldest in the archive, and version 3.0 added SQL, publish and subscribe, and an MCP server in May 2026.

**Runtime.** Every system kept rebuilding the same runtime. [Luke Logan](https://grc.iit.edu/center/team/luke-logan/) built it once and kept rebuilding it better: the Hermes runtime, then [LabStor](https://akougkas.io/publications/logan2022labstor/), modular I/O stacks in user space, then Chimaera, where every module is a pool of tasks on shared-memory queues, and now the CLIO Runtime. His [MegaMmap](https://akougkas.io/publications/logan2024megammap/) put the same tiers behind a memory interface. Build the runtime once, and every system after it is a module.

**Measurement.** You cannot place what you cannot see. DLIO is Hari’s benchmark of how scientific deep learning does I/O, built with Huihuo Zheng and Venkat Vishwanath at Argonne. Its later development and its place as the workload engine of MLPerf Storage are Argonne’s work, mostly Hari’s, and I was part of the Illinois Tech team that co-advised him. At Livermore, Hari built DFTracer, and [Izzet Yildirim](https://grc.iit.edu/center/team/izzet-yildirim/)’s [WisIO](https://akougkas.io/publications/yildirim_wisio_2025/) reads one trace from many views to classify bottlenecks from metrics rather than fixed rules.

None of this was one person’s work, and the line passes through five dissertations. Mine, on data labels, gave Harmonia, Hermes, and LABIOS. Kun’s integrated files and objects. Hari’s made storage tier-aware and measurable. Luke’s built the runtime. [Jaime Cernuda](https://grc.iit.edu/center/team/jaime-cernuda/)’s built the data flows and pipelines that now carry CLIO Agent and Relay. Each handed its ideas to the next, and all five continue in [IOWarp](https://akougkas.io/work/iowarp/).

## The same pyramid, different hot data

In 2018 the hot data in our pyramid was a checkpoint: RAM, NVMe, burst buffer, parallel file system. In 2025 the hot data is a KV-cache, the attention state a language model keeps per token, and the pyramid is GPU HBM, host DRAM, local SSD, and networked object storage. NVIDIA Dynamo, LMCache, and Mooncake all tier KV-cache blocks down that hierarchy. Industry arrived at the same pyramid independently, and that is the best evidence I know that the problem is real. [Jie Ye](https://grc.iit.edu/center/team/jie-ye/)’s measurements of [KV-cache I/O under concurrency](https://akougkas.io/publications/ye2025unboxkv-io/) are part of how our group came to see it.

The next hot data is the context an agent needs for one decision. I made the argument that the client changed at an [Argonne seminar last year](https://akougkas.io/journal/2025/from-storage-optimization-to-context-orchestration/): a program’s unit of work is the byte and its access is periodic and knowable, while an agent’s unit of work is the decision and its requests are semantic and produced by reasoning. Context is the slice of data, metadata, provenance, system state, tool capability, and history that one decision needs. Assembling that slice on time, from every tier, is an I/O problem.

## The bet

My bet is that the line we have built since 2016 is the right foundation for agent context. Not because the systems were designed for agents, since none of them was, but because context is assembled from every tier at once and handed to a client that should see one destination. That is the thesis, with a new client.

Each line already has a home in the [CLIO](https://akougkas.io/work/clio/) stack, the context layer for scientific AI agents that I created for IOWarp. Placement lives in CLIO Core’s Context Transfer Engine, which is Hermes grown up: its policies became a chain of interposers on one task interface, cache, then compressor, then replication, then core. Intent and task live in CLIO Core. Order is ChronoLog as an agent’s history. Runtime is the CLIO Runtime. Luke leads Core engineering, and the [2.3 and 2.4 releases](https://akougkas.io/journal/2026/clio-core-2-3-and-2-4-when-a-tier-fills-or-a-node-dies/) taught it to evict from a full tier, lose no data when a tier goes offline, and run on Aurora’s Intel GPUs. Jaime leads CLIO Agent and Relay. Isa Muradli, Shazzadul Islam, Soham Sonar, and Aum Sathwara build CLIO Kit, the MCP servers for scientific formats and tools. I build [CLIO Coder](https://akougkas.io/work/clio-coder/). Two DOE Genesis Mission projects, SPOTTER-AI led by Argonne and PORTUS led by PNNL, build on CLIO, and the full stack launches at SC26 in Chicago, November 15 to 20.

The measurement line is the youngest. For agents it means characterizing agents as an I/O workload, and several students in the group are now doing exactly that. Agents read, write, and revisit context in patterns that our storage stacks were never tuned for, and nobody has measured them carefully yet. If agent context turns out to be a first-class I/O workload, every tier in the hierarchy has a new tenant to serve. The other five lines tell us where context should live and how it should move. Measurement is how we will learn whether agents need it the way I think they do.

## What the line taught me

None of these lessons is about I/O. The system existed before the project did: Hermes was a paper and a prototype before it was funded. One line of work compounds: one runtime, from Hermes to LabStor to Chimaera, now sits under everything we ship. Software makes an idea negotiate, with compilers, failures, other people’s workloads, and the cost of being understood. And the abstraction outlives the code. [Hermes’s final release](https://akougkas.io/journal/2024/hermes-is-complete/) was in 2024, and its placement model runs inside CLIO today.

None of these systems started because someone asked for them. Each started because someone in the lab saw a thing that should exist and began building it well. Pursuing excellence does not need justification.