akougkas.io

Hermes

Multi-tier I/O buffering for HPC storage hierarchies.

The multi-tier I/O buffering system I created during my PhD, carried through five years of NSF funding into versioned middleware released under The HDF Group, and forward into IOWarp.

View the code Read the papers
The Hermes architecture from the 2018 paper. Application nodes with RAM and NVMe on the left, burst buffer nodes in the middle, and the parallel file system at the bottom, with Hermes coordinating data movement across every tier.
The diagram from the HPDC 2018 paper. Compute-node RAM and NVMe, burst-buffer nodes, and the parallel file system, presented to the application as one buffering hierarchy. Open full-size figure.

The question

High-performance machines do not have one kind of storage. They have a hierarchy: memory, local NVMe, burst buffers, parallel file systems, and increasingly object stores. Applications knew the hierarchy existed, but they were left to manage it with brittle, machine-specific logic. Most of them simply wrote to the parallel file system and waited.

Hermes asked whether middleware could present those tiers as one coordinated buffering system while still making intelligent placement decisions on the application’s behalf.

Where it came from

Hermes started as the center of my PhD at Illinois Tech under Xian-He Sun. I designed the architecture, wrote the HPDC 2018 paper with Hariharan Devarajan, and then turned the paper into a proposal. In September 2018, NSF funded five years of it as a collaborative Frameworks award with The HDF Group, with Gerd Heber leading the HDF Group side. I led the technical work from the first day to the last release.

That order matters to me. The system existed before the money did.

The architectural bet

The core bet was policy-driven placement. Data should move through the hierarchy according to workload behavior, device capability, capacity, and intent, not because one path was compiled into an application years earlier. The paper’s placement engine offered three policies, and MinimizeIoTime modeled latency, throughput, and remaining capacity to pick a tier per request.

That required more than a fast data path. Hermes needed distributed metadata, adapters that intercept POSIX, HDF5, and MPI-IO without changing the application, an RDMA-capable runtime that could coordinate placement across nodes, and deployment practices that worked outside the machine where the paper was written.

What the team built on top of it

The follow-on papers were built inside Hermes. HFetch pushed data through the hierarchy from the server side using access frequency and recency. HCompress picked a compression method per tier, trading CPU for capacity. HReplica coordinated replication with placement. Multi-tier eviction decided where an evicted buffer should land as well as which one to evict. Those were Hariharan’s papers with me and Xian-He, and each of them had to fit inside the same placement model, which is what kept the architecture coherent.

The runtime that Luke Logan built to keep all of that coordinated later became Chimaera, which is now the runtime under CLIO Core. Keith Bateman, Neeraj Rajesh, Jaime Cernuda, Jie Ye, Kun Feng, Izzet Yildirim, and Meng Tang all worked inside it at some point. For most of them it was the first large system they touched.

What shipped

Hermes became public, versioned middleware under The HDF Group organization. It was launched publicly in December 2020, acquired adapters, packaging, documentation, and tutorials at SC21, and closed with v1.2.1 in August 2024. Along the way we ran it on Argonne’s Theta and Oak Ridge’s Summit, and the provisional patent we filed in May 2020 became the seed of the LABIOS patent.

That translation from architecture to maintained software is the part I return to most often. A paper can isolate a good idea. Software forces the idea to negotiate with compilers, dependencies, failures, other people’s workloads, and the cost of being understandable. Those negotiations changed the architecture for the better and exposed which abstractions were durable.

What carried forward

The project is complete and its design is still in use. The separation between interface, runtime, and heterogeneous storage; the attention to data placement; and the demand for deployable integration all carried into IOWarp, where the Context Transfer Engine is Hermes’s direct descendant, and into Coeus, which put computation in the same data path.

The outcome I value most is that the architecture became working material for the next system.

What I would do differently

I would make observability first-class earlier. Placement decisions are easier to trust when a user can see why they happened. I would also design the packaging and upgrade path alongside the runtime from the beginning. Installation, diagnosis, and upgrades are part of a research system, and the work is unfinished until they exist. Those lessons now shape the broader IOWarp and CLIO program, where data placement, context movement, and the decisions behind them have to remain visible across the stack.

Code, releases, papers, and records

  1. paper

    Hermes at HPDC 2018

    The architecture and its first evaluation. Kougkas, Devarajan, and Sun.

  2. code

    HDFGroup/hermes

    The deployable middleware, adapters, history, and issues, maintained under The HDF Group.

  3. release

    Hermes v1.2.1, August 2024

    The final public release in the completed project record.

  4. record

    NSF award OAC-1835764

    The public award record, 2018–2023. PI Xian-He Sun; I was the technical lead.

  5. press

    Illinois Tech news, December 2020

    The public launch of the software with The HDF Group.

  6. record

    GRC project record

    The project's page at the Gnosis Research Center.

  7. documentation

    Context Transfer Engine in CLIO Core

    The transfer engine in CLIO Core, which descends from Hermes.

  8. press

    GRC news, September 2018

    The announcement of the NSF Frameworks grant for Hermes.

Papers from this line of work

Browse these papers
  1. 2026
    poster

    Benchmarking the Performance of Semantic Search Techniques for Agent Discovery

    Rajni Pawar, Luke Logan, Xian-He Sun, Anthony Kougkas. 13th Greater Chicago Area Systems Research Workshop (GCASR 2026)

  2. 2024
    conference

    MegaMmap: Blurring the Boundary Between Memory and Storage for Data-Intensive Workloads

    Luke Logan, Anthony Kougkas, Xian-He Sun. Proceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis SC

  3. 2024
    conference

    DaYu: Optimizing Distributed Scientific Workflows by Decoding Dataflow Semantics and Dynamics

    Meng Tang, Jaime Cernuda, Jie Ye, Luanzheng Guo, Nathan R. Tallent, Anthony Kougkas et al. Proceedings of the IEEE International Conference on Cluster Computing

  4. 2024
    conference

    HStream: A hierarchical data streaming engine for high-throughput scientific applications

    Jaime Cernuda, Jie Ye, Anthony Kougkas, Xian-He Sun. Proceedings of the 53rd International Conference on Parallel Processing

  5. 2023
    conference

    Data Flow Lifecycles for Optimizing Workflow Coordination

    Hyungro Lee, Luanzheng Guo, Meng Tang, Jesun Firoz, Nathan Tallent, Anthony Kougkas et al. Proceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis

  6. 2022
    conference

    LuxIO: Intelligent Resource Provisioning and Auto-Configuration for Storage Services

    Keith Bateman, Neeraj Rajesh, Jaime Cernuda Garcia, Luke Logan, Jie Ye, … Anthony Kougkas et al. Proceedings of the 29th International Conference on High Performance Computing, Data, and Analytics

  7. 2022
    conference

    LabStor: A modular and extensible platform for developing high-performance, customized I/O stacks in userspace

    Luke Logan, Jaime Cernuda Garcia, Jay Lofstead, Xian–He Sun, Anthony Kougkas. Proceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis

  8. 2022
    conference

    Stimulus: Accelerate Data Management for Scientific AI applications in HPC

    Hariharan Devarajan, Anthony Kougkas, Huihuo Zheng, Venkatram Vishwanath, Xian-He Sun. Proceedings of the 22nd IEEE International Symposium on Cluster, Cloud and Internet Computing

  9. 2021
    poster

    Utilizing Persistent Memory in Parallel I/O Libraries

    Luke Logan, Jay Lofstead, S. Levy, P. Widener, Xian-He Sun, Anthony Kougkas. The 2021 International Conference for High Performance Computing, Networking, Storage, and Analysis (SC’21), November 14–19, 2021

  10. 2021
    workshop

    pMEMCPY: a simple, lightweight, and portable I/O library for storing data in persistent memory

    Luke Logan, Jay Lofstead, Scott Levy, Patrick Widener, Xian-He Sun, Anthony Kougkas. Proceedings of the International Conference on Cluster Computing

  11. 2021
    conference

    HFlow: A Dynamic and Elastic Multi-Layered I/O Forwarder

    Jaime Cernuda, Hariharan Devarajan, Luke Logan, Keith Bateman, Neeraj Rajesh, … Anthony Kougkas et al. Proceedings of the International Conference on Cluster Computing

  12. 2021
    conference

    Apollo: An ML-assisted real-time storage resource observer

    Neeraj Rajesh, Hariharan Devarajan, Jaime Cernuda Garcia, Keith Bateman, Luke Logan, … Anthony Kougkas et al. Proceedings of the 30th International Symposium on High-Performance Parallel and Distributed Computing

  13. 2021
    conference

    DLIO: A data-centric benchmark for scientific deep learning applications

    Hariharan Devarajan, Huihuo Zheng, Anthony Kougkas, Xian-He Sun, Venkatram Vishwanath. Proceedings of the 21st International Symposium on Cluster, Cloud and Internet Computing

    Award: 2021 IEEE/ACM CCGrid Best Paper Award
  14. 2020
    conference

    HReplica: A Dynamic Data Replication Engine with Adaptive Compression for Multi-Tiered Storage

    Hariharan Devarajan, Anthony Kougkas, Xian-He Sun. Proceedings of the International Conference on Big Data

  15. 2020
    poster

    A Dynamic Multi-Tiered Storage System for Extreme Scale Computing

    Hariharan Devarajan, Anthony Kougkas, Xian-He Sun. The International Conference for High Performance Computing, Networking, Storage and Analysis (SC’20)

  16. 2020
    poster

    Quantifying the Overheads of the Modern Linux I/O Stack

    Luke Logan, Anthony Kougkas, Xian-He Sun. The International Conference for High Performance Computing, Networking, Storage and Analysis (SC’20)

  17. 2020
    poster

    Characterizing and Approximating I/O Behavior of HDF5 Applications

    Neeraj Rajesh, Gerd Heber, Anthony Kougkas, Xian-He Sun. The International Conference for High Performance Computing, Networking, Storage and Analysis (SC’20)

  18. 2020
    conference

    HCL: Distributing Parallel Data Structures in Extreme Scales

    Hariharan Devarajan, Anthony Kougkas, Keith Bateman, Xian-He Sun. Proceedings of the International Conference on Cluster Computing

  19. 2020
    conference

    HFetch: Hierarchical Data Prefetching for Scientific Workflows in Multi-Tiered Storage Environments

    Hariharan Devarajan, Anthony Kougkas, Xian-He Sun. Proceedings of the International Parallel and Distributed Processing Symposium

  20. 2020
    conference

    HCompress: Hierarchical Data Compression for Multi-Tiered Storage Environments

    Hariharan Devarajan, Anthony Kougkas, Luke Logan, Xian-He Sun. Proceedings of the International Parallel and Distributed Processing Symposium

  21. 2020
    journal

    I/O Acceleration via Multi-Tiered Data Buffering and Prefetching

    Anthony Kougkas, Hariharan Devarajan, Xian-He Sun. Journal of Computer Science and Technology

  22. 2019
    poster

    Efficient Data Eviction across Multiple Tiers of Storage

    Jaime Cernuda, H. Trivino, Hariharan Devarajan, Anthony Kougkas, Xian-He Sun. The International Conference for High Performance Computing, Networking, Storage and Analysis (SC’19)

  23. 2019
    poster

    HFetch: Hierarchical Data Prefetching in Multi-Tiered Storage Environments

    Hariharan Devarajan, Anthony Kougkas, Xian-He Sun. The International Conference for High Performance Computing, Networking, Storage and Analysis (SC’19) Best Poster Nominee, Ph.D Forum

  24. 2019
    conference

    LABIOS: A Distributed Label-Based I/O System

    Anthony Kougkas, Hariharan Devarajan, Jay Lofstead, Xian-He Sun. Proceedings of the 28th International Symposium on High-Performance Parallel and Distributed Computing

    Award: Karsten Schwan Best Paper Award at HPDC 2019
  25. 2019
    conference

    An intelligent, adaptive, and flexible data compression framework

    Hariharan Devarajan, Anthony Kougkas, Xian-He Sun. Proceedings of the 19th International Symposium on Cluster, Cloud and Grid Computing

  26. 2018
    conference

    Vidya: Performing code-block I/O characterization for data access optimization

    Hariharan Devarajan, Anthony Kougkas, Prajwal Challa, Xian-He Sun. Proceedings of the 25th International Conference on High Performance Computing

  27. 2018
    conference

    Harmonia: An interference-aware dynamic I/O scheduler for shared non-volatile burst buffers

    Anthony Kougkas, Hariharan Devarajan, Xian-He Sun, Jay Lofstead. Proceedings of the International Conference on Cluster Computing

  28. 2018
    conference

    Hermes: a heterogeneous-aware multi-tiered distributed I/O buffering system

    Anthony Kougkas, Hariharan Devarajan, Xian-He Sun. Proceedings of the 27th International Symposium on High-Performance Parallel and Distributed Computing

  29. 2016
    conference

    Leveraging burst buffer coordination to prevent I/O interference

    Anthony Kougkas, Matthieu Dorier, Rob Latham, Rob Ross, Xian-He Sun. Proceedings of the 12th International Conference on e-Science

People

Related notes

  1. update

    CLIO Core 2.3 and 2.4: when a tier fills or a node dies

    Four CLIO Core releases between October 9 and November 2, 2026 added a device-paged GPU vector and Intel GPU support, then taught the context store to recover a full tier and keep data and directories readable when a node goes down.

    3 min
  2. essay

    When is research software shipped?

    A working definition of "shipped" for research software, built around a named state someone else can run, evidence tied to that state, explicit limits, and a continuity signal, with Hermes and DLIO as the worked examples.

    7 min
  3. update

    Hermes is complete

    Hermes v1.2.1 is the final release of a project that started as the center of my PhD and ran for five years with The HDF Group. What the multi-tier buffering bet proved, what it did not, and what moves into IOWarp.

    5 min
  4. essay

    From storage optimization to context orchestration

    Twenty years of I/O research optimized bytes per second for programs. Agents are a different client, and each system in our lineage contributes one primitive to serving them, as argued in a seminar at Argonne.

    8 min
  5. update

    Two years of IOWarp

    Two years after the NSF award, what IOWarp promised, what shipped as public releases, what changed course, and what is still a plan.

    4 min
  6. essay

    One destination, N tiers

    Every tier supplies the application at once, and the application sees one destination. The thesis from a November 2026 talk to my group at the Gnosis Research Center, the six lines of I/O work behind it since 2016, and my bet that they are the foundation agents need for context.

    9 min

Funding

  • NSF OAC-1835764, 2018–2023

    Collaborative award with The HDF Group. PI Xian-He Sun; I created and led Hermes as its architect and technical lead.

  • 2024–present

    IOWarp

    Context management infrastructure for scientific AI.

  • 2025–present

    CLIO

    A context layer for scientific AI agents and workflows.

  • 2017–present

    LABIOS

    Label-based I/O for HPC and Big Data storage systems.