akougkas.io

Hermes: a heterogeneous-aware multi-tiered distributed I/O buffering system

Anthony Kougkas, Hariharan Devarajan, Xian-He Sun

Proceedings of the 27th International Symposium on High-Performance Parallel and Distributed ComputingACM

Abstract

Modern High-Performance Computing (HPC) systems are adding extra layers to the memory and storage hierarchy named deep memory and storage hierarchy (DMSH), to increase I/O performance. New hardware technologies, such as NVMe and SSD, have been introduced in burst buffer installations to reduce the pressure for external storage and boost the burstiness of modern I/O systems. The DMSH has demonstrated its strength and potential in practice. However, each layer of DMSH is an independent heterogeneous system and data movement among more layers is significantly more complex even without considering heterogeneity. How to efficiently utilize the DMSH is a subject of research facing the HPC community. In this paper, we present the design and implementation of Hermes: a new, heterogeneous-aware, multi-tiered, dynamic, and distributed I/O buffering system. Hermes enables, manages, supervises, and, in some sense, extends I/O buffering to fully integrate into the DMSH. We introduce three novel data placement policies to efficiently utilize all layers and we present three novel techniques to perform memory, metadata, and communication management in hierarchical buffering systems. Our evaluation shows that, in addition to automatic data movement through the hierarchy, Hermes can significantly accelerate I/O and outperforms by more than 2x state-of-the-art buffering platforms.

Finding

Hermes moves data automatically through the deep memory and storage hierarchy and outperforms state-of-the-art buffering platforms by more than 2x.

The Hermes architecture from the 2018 paper. Application nodes with RAM and NVMe on the left, burst buffer nodes in the middle, and the parallel file system at the bottom, with Hermes coordinating data movement across every tier.
The diagram from the HPDC 2018 paper. Compute-node RAM and NVMe, burst-buffer nodes, and the parallel file system, presented to the application as one buffering hierarchy. Open full-size figure.

System

  1. 2018–2024
    complete

    Hermes

    The multi-tier I/O buffering system I created during my PhD, carried through five years of NSF funding into versioned middleware released under The HDF Group, and forward into IOWarp.

Browse all papers on Hermes
  1. 2026
    poster

    Benchmarking the Performance of Semantic Search Techniques for Agent Discovery

    Rajni Pawar, Luke Logan, Xian-He Sun, Anthony Kougkas. 13th Greater Chicago Area Systems Research Workshop (GCASR 2026)

  2. 2024
    conference

    MegaMmap: Blurring the Boundary Between Memory and Storage for Data-Intensive Workloads

    Luke Logan, Anthony Kougkas, Xian-He Sun. Proceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis SC

  3. 2024
    conference

    DaYu: Optimizing Distributed Scientific Workflows by Decoding Dataflow Semantics and Dynamics

    Meng Tang, Jaime Cernuda, Jie Ye, Luanzheng Guo, Nathan R. Tallent, Anthony Kougkas et al. Proceedings of the IEEE International Conference on Cluster Computing

  4. 2024
    conference

    HStream: A hierarchical data streaming engine for high-throughput scientific applications

    Jaime Cernuda, Jie Ye, Anthony Kougkas, Xian-He Sun. Proceedings of the 53rd International Conference on Parallel Processing

Citation

@inproceedings{kougkas2018hermes,
  author = {Kougkas, Anthony and Devarajan, Hariharan and Sun, Xian-He},
  booktitle = {Proceedings of the 27th International Symposium on High-Performance Parallel and Distributed Computing},
  title = {Hermes: a heterogeneous-aware multi-tiered distributed I/O buffering system},
  year = {2018},
  month = jun,
  publisher = {ACM},
  pages = {219--230},
  doi = {10.1145/3208040.3208059},
  url = {https://dl.acm.org/doi/abs/10.1145/3208040.3208059},
}

Related notes

  1. essay

    One destination, N tiers

    Every tier supplies the application at once, and the application sees one destination. The thesis from a November 2026 talk to my group at the Gnosis Research Center, the six lines of I/O work behind it since 2016, and my bet that they are the foundation agents need for context.

    9 min
  2. update

    Hermes is complete

    Hermes v1.2.1 is the final release of a project that started as the center of my PhD and ran for five years with The HDF Group. What the multi-tier buffering bet proved, what it did not, and what moves into IOWarp.

    5 min
  3. update

    CLIO Core 2.3 and 2.4: when a tier fills or a node dies

    Four CLIO Core releases between October 9 and November 2, 2026 added a device-paged GPU vector and Intel GPU support, then taught the context store to recover a full tier and keep data and directories readable when a node goes down.

    3 min