# Hermes

https://akougkas.io/work/hermes/

Author: Anthony Kougkas

Updated: 2026-08-27

## The question

High-performance machines do not have one kind of storage. They have a hierarchy: memory, local NVMe, burst buffers, parallel file systems, and increasingly object stores. Applications knew the hierarchy existed, but they were left to manage it with brittle, machine-specific logic. Most of them simply wrote to the parallel file system and waited.

Hermes asked whether middleware could present those tiers as one coordinated buffering system while still making intelligent placement decisions on the application’s behalf.

## Where it came from

Hermes started as the center of my PhD at [Illinois Tech](https://www.iit.edu/) under [Xian-He Sun](https://www.iit.edu/directory/people/xian-he-sun). I designed the architecture, wrote the [HPDC 2018 paper](https://akougkas.io/publications/kougkas2018hermes/) with [Hariharan Devarajan](https://grc.iit.edu/center/team/hariharan-devarajan/), and then turned the paper into a proposal. In September 2018, [NSF](https://www.nsf.gov/) funded five years of it as a collaborative Frameworks award with [The HDF Group](https://www.hdfgroup.org/), with [Gerd Heber](https://www.hdfgroup.org/team/gerd-heber/) leading the HDF Group side. I led the technical work from the first day to the last release.

That order matters to me. The system existed before the money did.

## The architectural bet

The core bet was policy-driven placement. Data should move through the hierarchy according to workload behavior, device capability, capacity, and intent, not because one path was compiled into an application years earlier. The paper’s placement engine offered three policies, and `MinimizeIoTime` modeled latency, throughput, and remaining capacity to pick a tier per request.

That required more than a fast data path. Hermes needed distributed metadata, adapters that intercept POSIX, HDF5, and MPI-IO without changing the application, an RDMA-capable runtime that could coordinate placement across nodes, and deployment practices that worked outside the machine where the paper was written.

## What the team built on top of it

The follow-on papers were built inside Hermes. [HFetch](https://akougkas.io/publications/devarajan2020hfetch/) pushed data through the hierarchy from the server side using access frequency and recency. [HCompress](https://akougkas.io/publications/devarajan2020hcompress/) picked a compression method per tier, trading CPU for capacity. [HReplica](https://akougkas.io/publications/devarajan2020hreplica/) coordinated replication with placement. Multi-tier eviction decided where an evicted buffer should land as well as which one to evict. Those were Hariharan’s papers with me and Xian-He, and each of them had to fit inside the same placement model, which is what kept the architecture coherent.

The runtime that [Luke Logan](https://grc.iit.edu/center/team/luke-logan/) built to keep all of that coordinated later became Chimaera, which is now the runtime under [CLIO Core](https://akougkas.io/work/clio/). [Keith Bateman](https://grc.iit.edu/center/team/keith-bateman/), [Neeraj Rajesh](https://grc.iit.edu/center/team/neeraj-rajesh/), [Jaime Cernuda](https://grc.iit.edu/center/team/jaime-cernuda/), [Jie Ye](https://grc.iit.edu/center/team/jie-ye/), [Kun Feng](https://grc.iit.edu/center/team/kun-feng/), [Izzet Yildirim](https://grc.iit.edu/center/team/izzet-yildirim/), and [Meng Tang](https://grc.iit.edu/center/team/meng-tang/) all worked inside it at some point. For most of them it was the first large system they touched.

## What shipped

Hermes became public, versioned middleware under [The HDF Group](https://github.com/HDFGroup/hermes) organization. It was launched publicly in December 2020, acquired adapters, packaging, documentation, and tutorials at SC21, and closed with [v1.2.1](https://github.com/HDFGroup/hermes/releases/tag/v1.2.1) in August 2024. Along the way we ran it on [Argonne](https://www.anl.gov/)’s Theta and [Oak Ridge](https://www.ornl.gov/)’s Summit, and the provisional patent we filed in May 2020 became the seed of the [LABIOS](https://akougkas.io/work/labios/) patent.

That translation from architecture to maintained software is the part I return to most often. A paper can isolate a good idea. Software forces the idea to negotiate with compilers, dependencies, failures, other people’s workloads, and the cost of being understandable. Those negotiations changed the architecture for the better and exposed which abstractions were durable.

## What carried forward

The project is complete and its design is still in use. The separation between interface, runtime, and heterogeneous storage; the attention to data placement; and the demand for deployable integration all carried into [IOWarp](https://akougkas.io/work/iowarp/), where the Context Transfer Engine is Hermes’s direct descendant, and into [Coeus](https://akougkas.io/work/atlas/#project-coeus), which put computation in the same data path.

The outcome I value most is that the architecture became working material for the next system.

## What I would do differently

I would make observability first-class earlier. Placement decisions are easier to trust when a user can see why they happened. I would also design the packaging and upgrade path alongside the runtime from the beginning. Installation, diagnosis, and upgrades are part of a research system, and the work is unfinished until they exist. Those lessons now shape the broader [IOWarp](https://akougkas.io/work/iowarp/) and [CLIO](https://akougkas.io/work/clio/) program, where data placement, context movement, and the decisions behind them have to remain visible across the stack.
