Hermes
Multi-tier I/O buffering for HPC storage hierarchies.
The multi-tier I/O buffering system I created during my PhD, carried through five years of NSF funding into versioned middleware released under The HDF Group, and forward into IOWarp.
The question
High-performance machines do not have one kind of storage. They have a hierarchy: memory, local NVMe, burst buffers, parallel file systems, and increasingly object stores. Applications knew the hierarchy existed, but they were left to manage it with brittle, machine-specific logic. Most of them simply wrote to the parallel file system and waited.
Hermes asked whether middleware could present those tiers as one coordinated buffering system while still making intelligent placement decisions on the application’s behalf.
Where it came from
Hermes started as the center of my PhD at Illinois Tech under Xian-He Sun. I designed the architecture, wrote the HPDC 2018 paper with Hariharan Devarajan, and then turned the paper into a proposal. In September 2018, NSF funded five years of it as a collaborative Frameworks award with The HDF Group, with Gerd Heber leading the HDF Group side. I led the technical work from the first day to the last release.
That order matters to me. The system existed before the money did.
The architectural bet
The core bet was policy-driven placement. Data should move through the hierarchy according to workload behavior, device capability, capacity, and intent, not because one path was compiled into an application years earlier. The paper’s placement engine offered three policies, and MinimizeIoTime modeled latency, throughput, and remaining capacity to pick a tier per request.
That required more than a fast data path. Hermes needed distributed metadata, adapters that intercept POSIX, HDF5, and MPI-IO without changing the application, an RDMA-capable runtime that could coordinate placement across nodes, and deployment practices that worked outside the machine where the paper was written.
What the team built on top of it
The follow-on papers were built inside Hermes. HFetch pushed data through the hierarchy from the server side using access frequency and recency. HCompress picked a compression method per tier, trading CPU for capacity. HReplica coordinated replication with placement. Multi-tier eviction decided where an evicted buffer should land as well as which one to evict. Those were Hariharan’s papers with me and Xian-He, and each of them had to fit inside the same placement model, which is what kept the architecture coherent.
The runtime that Luke Logan built to keep all of that coordinated later became Chimaera, which is now the runtime under CLIO Core. Keith Bateman, Neeraj Rajesh, Jaime Cernuda, Jie Ye, Kun Feng, Izzet Yildirim, and Meng Tang all worked inside it at some point. For most of them it was the first large system they touched.
What shipped
Hermes became public, versioned middleware under The HDF Group organization. It was launched publicly in December 2020, acquired adapters, packaging, documentation, and tutorials at SC21, and closed with v1.2.1 in August 2024. Along the way we ran it on Argonne’s Theta and Oak Ridge’s Summit, and the provisional patent we filed in May 2020 became the seed of the LABIOS patent.
That translation from architecture to maintained software is the part I return to most often. A paper can isolate a good idea. Software forces the idea to negotiate with compilers, dependencies, failures, other people’s workloads, and the cost of being understandable. Those negotiations changed the architecture for the better and exposed which abstractions were durable.
What carried forward
The project is complete and its design is still in use. The separation between interface, runtime, and heterogeneous storage; the attention to data placement; and the demand for deployable integration all carried into IOWarp, where the Context Transfer Engine is Hermes’s direct descendant, and into Coeus, which put computation in the same data path.
The outcome I value most is that the architecture became working material for the next system.
What I would do differently
I would make observability first-class earlier. Placement decisions are easier to trust when a user can see why they happened. I would also design the packaging and upgrade path alongside the runtime from the beginning. Installation, diagnosis, and upgrades are part of a research system, and the work is unfinished until they exist. Those lessons now shape the broader IOWarp and CLIO program, where data placement, context movement, and the decisions behind them have to remain visible across the stack.
Code, releases, papers, and records
- paper
Hermes at HPDC 2018
The architecture and its first evaluation. Kougkas, Devarajan, and Sun.
- code
HDFGroup/hermes
The deployable middleware, adapters, history, and issues, maintained under The HDF Group.
- release
Hermes v1.2.1, August 2024
The final public release in the completed project record.
- record
NSF award OAC-1835764
The public award record, 2018–2023. PI Xian-He Sun; I was the technical lead.
- press
Illinois Tech news, December 2020
The public launch of the software with The HDF Group.
- record
GRC project record
The project's page at the Gnosis Research Center.
- documentation
Context Transfer Engine in CLIO Core
The transfer engine in CLIO Core, which descends from Hermes.
- press
GRC news, September 2018
The announcement of the NSF Frameworks grant for Hermes.
Papers from this line of work
Browse these papers- 2026
Benchmarking the Performance of Semantic Search Techniques for Agent Discovery
Rajni Pawar, Luke Logan, Xian-He Sun, Anthony Kougkas. 13th Greater Chicago Area Systems Research Workshop (GCASR 2026)
- 2024
MegaMmap: Blurring the Boundary Between Memory and Storage for Data-Intensive Workloads
Luke Logan, Anthony Kougkas, Xian-He Sun. Proceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis SC
- 2024
DaYu: Optimizing Distributed Scientific Workflows by Decoding Dataflow Semantics and Dynamics
Meng Tang, Jaime Cernuda, Jie Ye, Luanzheng Guo, Nathan R. Tallent, Anthony Kougkas et al. Proceedings of the IEEE International Conference on Cluster Computing
- 2024
HStream: A hierarchical data streaming engine for high-throughput scientific applications
Jaime Cernuda, Jie Ye, Anthony Kougkas, Xian-He Sun. Proceedings of the 53rd International Conference on Parallel Processing
- 2023
Data Flow Lifecycles for Optimizing Workflow Coordination
Hyungro Lee, Luanzheng Guo, Meng Tang, Jesun Firoz, Nathan Tallent, Anthony Kougkas et al. Proceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis
- 2022
LuxIO: Intelligent Resource Provisioning and Auto-Configuration for Storage Services
Keith Bateman, Neeraj Rajesh, Jaime Cernuda Garcia, Luke Logan, Jie Ye, … Anthony Kougkas et al. Proceedings of the 29th International Conference on High Performance Computing, Data, and Analytics
- 2022
LabStor: A modular and extensible platform for developing high-performance, customized I/O stacks in userspace
Luke Logan, Jaime Cernuda Garcia, Jay Lofstead, Xian–He Sun, Anthony Kougkas. Proceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis
- 2022
Stimulus: Accelerate Data Management for Scientific AI applications in HPC
Hariharan Devarajan, Anthony Kougkas, Huihuo Zheng, Venkatram Vishwanath, Xian-He Sun. Proceedings of the 22nd IEEE International Symposium on Cluster, Cloud and Internet Computing
- 2021
Utilizing Persistent Memory in Parallel I/O Libraries
Luke Logan, Jay Lofstead, S. Levy, P. Widener, Xian-He Sun, Anthony Kougkas. The 2021 International Conference for High Performance Computing, Networking, Storage, and Analysis (SC’21), November 14–19, 2021
- 2021
pMEMCPY: a simple, lightweight, and portable I/O library for storing data in persistent memory
Luke Logan, Jay Lofstead, Scott Levy, Patrick Widener, Xian-He Sun, Anthony Kougkas. Proceedings of the International Conference on Cluster Computing
- 2021
HFlow: A Dynamic and Elastic Multi-Layered I/O Forwarder
Jaime Cernuda, Hariharan Devarajan, Luke Logan, Keith Bateman, Neeraj Rajesh, … Anthony Kougkas et al. Proceedings of the International Conference on Cluster Computing
- 2021
Apollo: An ML-assisted real-time storage resource observer
Neeraj Rajesh, Hariharan Devarajan, Jaime Cernuda Garcia, Keith Bateman, Luke Logan, … Anthony Kougkas et al. Proceedings of the 30th International Symposium on High-Performance Parallel and Distributed Computing
- 2021 Award: 2021 IEEE/ACM CCGrid Best Paper Award
DLIO: A data-centric benchmark for scientific deep learning applications
Hariharan Devarajan, Huihuo Zheng, Anthony Kougkas, Xian-He Sun, Venkatram Vishwanath. Proceedings of the 21st International Symposium on Cluster, Cloud and Internet Computing
- 2020
HReplica: A Dynamic Data Replication Engine with Adaptive Compression for Multi-Tiered Storage
Hariharan Devarajan, Anthony Kougkas, Xian-He Sun. Proceedings of the International Conference on Big Data
- 2020
A Dynamic Multi-Tiered Storage System for Extreme Scale Computing
Hariharan Devarajan, Anthony Kougkas, Xian-He Sun. The International Conference for High Performance Computing, Networking, Storage and Analysis (SC’20)
- 2020
Quantifying the Overheads of the Modern Linux I/O Stack
Luke Logan, Anthony Kougkas, Xian-He Sun. The International Conference for High Performance Computing, Networking, Storage and Analysis (SC’20)
- 2020
Characterizing and Approximating I/O Behavior of HDF5 Applications
Neeraj Rajesh, Gerd Heber, Anthony Kougkas, Xian-He Sun. The International Conference for High Performance Computing, Networking, Storage and Analysis (SC’20)
- 2020
HCL: Distributing Parallel Data Structures in Extreme Scales
Hariharan Devarajan, Anthony Kougkas, Keith Bateman, Xian-He Sun. Proceedings of the International Conference on Cluster Computing
- 2020
HFetch: Hierarchical Data Prefetching for Scientific Workflows in Multi-Tiered Storage Environments
Hariharan Devarajan, Anthony Kougkas, Xian-He Sun. Proceedings of the International Parallel and Distributed Processing Symposium
- 2020
HCompress: Hierarchical Data Compression for Multi-Tiered Storage Environments
Hariharan Devarajan, Anthony Kougkas, Luke Logan, Xian-He Sun. Proceedings of the International Parallel and Distributed Processing Symposium
- 2020
I/O Acceleration via Multi-Tiered Data Buffering and Prefetching
Anthony Kougkas, Hariharan Devarajan, Xian-He Sun. Journal of Computer Science and Technology
- 2019
Efficient Data Eviction across Multiple Tiers of Storage
Jaime Cernuda, H. Trivino, Hariharan Devarajan, Anthony Kougkas, Xian-He Sun. The International Conference for High Performance Computing, Networking, Storage and Analysis (SC’19)
- 2019
HFetch: Hierarchical Data Prefetching in Multi-Tiered Storage Environments
Hariharan Devarajan, Anthony Kougkas, Xian-He Sun. The International Conference for High Performance Computing, Networking, Storage and Analysis (SC’19) Best Poster Nominee, Ph.D Forum
- 2019 Award: Karsten Schwan Best Paper Award at HPDC 2019
LABIOS: A Distributed Label-Based I/O System
Anthony Kougkas, Hariharan Devarajan, Jay Lofstead, Xian-He Sun. Proceedings of the 28th International Symposium on High-Performance Parallel and Distributed Computing
- 2019
An intelligent, adaptive, and flexible data compression framework
Hariharan Devarajan, Anthony Kougkas, Xian-He Sun. Proceedings of the 19th International Symposium on Cluster, Cloud and Grid Computing
- 2018
Vidya: Performing code-block I/O characterization for data access optimization
Hariharan Devarajan, Anthony Kougkas, Prajwal Challa, Xian-He Sun. Proceedings of the 25th International Conference on High Performance Computing
- 2018
Harmonia: An interference-aware dynamic I/O scheduler for shared non-volatile burst buffers
Anthony Kougkas, Hariharan Devarajan, Xian-He Sun, Jay Lofstead. Proceedings of the International Conference on Cluster Computing
- 2018
Hermes: a heterogeneous-aware multi-tiered distributed I/O buffering system
Anthony Kougkas, Hariharan Devarajan, Xian-He Sun. Proceedings of the 27th International Symposium on High-Performance Parallel and Distributed Computing
- 2016
Leveraging burst buffer coordination to prevent I/O interference
Anthony Kougkas, Matthieu Dorier, Rob Latham, Rob Ross, Xian-He Sun. Proceedings of the 12th International Conference on e-Science
People
- Xian-He Sun My PhD advisor; HPDC'18 co-author
- Gerd Heber Co-PI, The HDF Group
- Hariharan Devarajan HPDC'18 co-author; HFetch, HCompress, HReplica
- Luke Logan Engineer, 2019–2025; the runtime that became Chimaera
- Keith Bateman Researcher
- Neeraj Rajesh Researcher
- Jaime Cernuda Researcher
- Jie Ye Researcher
- Kun Feng Researcher
- Izzet Yildirim Researcher
- Meng Tang Researcher
Related notes
- 3 min
CLIO Core 2.3 and 2.4: when a tier fills or a node dies
Four CLIO Core releases between October 9 and November 2, 2026 added a device-paged GPU vector and Intel GPU support, then taught the context store to recover a full tier and keep data and directories readable when a node goes down.
- 7 min
When is research software shipped?
A working definition of "shipped" for research software, built around a named state someone else can run, evidence tied to that state, explicit limits, and a continuity signal, with Hermes and DLIO as the worked examples.
- 5 min
Hermes is complete
Hermes v1.2.1 is the final release of a project that started as the center of my PhD and ran for five years with The HDF Group. What the multi-tier buffering bet proved, what it did not, and what moves into IOWarp.
- 8 min
From storage optimization to context orchestration
Twenty years of I/O research optimized bytes per second for programs. Agents are a different client, and each system in our lineage contributes one primitive to serving them, as argued in a seminar at Argonne.
- 4 min
Two years of IOWarp
Two years after the NSF award, what IOWarp promised, what shipped as public releases, what changed course, and what is still a plan.
- 9 min
One destination, N tiers
Every tier supplies the application at once, and the application sees one destination. The thesis from a November 2026 talk to my group at the Gnosis Research Center, the six lines of I/O work behind it since 2016, and my bet that they are the foundation agents need for context.