akougkas.io

HCompress: Hierarchical Data Compression for Multi-Tiered Storage Environments

Hariharan Devarajan, Anthony Kougkas, Luke Logan, Xian-He Sun

Proceedings of the International Parallel and Distributed Processing SymposiumIEEE

Abstract

Modern scientific applications read and write massive amounts of data through simulations, observations, and analysis. These applications spend the majority of their runtime in performing I/O. HPC storage solutions include fast node-local and shared storage resources to elevate applications from this bottleneck. Moreover, several middleware libraries (e.g., Hermes) are proposed to move data between these tiers transparently. Data reduction is another technique that reduces the amount of data produced and, hence, improve I/O performance. These two technologies, if used together, can benefit from each other. The effectiveness of data compression can be enhanced by selecting different compression algorithms according to the characteristics of the different tiers, and the multi-tiered hierarchy can benefit from extra capacity. In this paper, we design and implement HCompress, a hierarchical data compression library that can improve the application’s performance by harmoniously leveraging both multi-tiered storage and data compression. We have developed a novel compression selection algorithm that facilitates the optimal matching of compression libraries to the tiered storage. Our evaluation shows that HCompress can improve scientific application’s performance by 7x when compared to other state-of-the-art tiered storage solutions.

Finding

Matching compression libraries to storage tiers lets HCompress improve scientific application performance by 7x over other state-of-the-art tiered storage solutions.

System

  1. 2018–2024
    complete

    Hermes

    The multi-tier I/O buffering system I created during my PhD, carried through five years of NSF funding into versioned middleware released under The HDF Group, and forward into IOWarp.

Browse all papers on Hermes
  1. 2026
    poster

    Benchmarking the Performance of Semantic Search Techniques for Agent Discovery

    Rajni Pawar, Luke Logan, Xian-He Sun, Anthony Kougkas. 13th Greater Chicago Area Systems Research Workshop (GCASR 2026)

  2. 2024
    conference

    MegaMmap: Blurring the Boundary Between Memory and Storage for Data-Intensive Workloads

    Luke Logan, Anthony Kougkas, Xian-He Sun. Proceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis SC

  3. 2024
    conference

    DaYu: Optimizing Distributed Scientific Workflows by Decoding Dataflow Semantics and Dynamics

    Meng Tang, Jaime Cernuda, Jie Ye, Luanzheng Guo, Nathan R. Tallent, Anthony Kougkas et al. Proceedings of the IEEE International Conference on Cluster Computing

  4. 2024
    conference

    HStream: A hierarchical data streaming engine for high-throughput scientific applications

    Jaime Cernuda, Jie Ye, Anthony Kougkas, Xian-He Sun. Proceedings of the 53rd International Conference on Parallel Processing

Citation

@inproceedings{devarajan2020hcompress,
  author = {Devarajan, Hariharan and Kougkas, Anthony and Logan, Luke and Sun, Xian-He},
  booktitle = {Proceedings of the International Parallel and Distributed Processing Symposium},
  title = {{HCompress}: Hierarchical Data Compression for Multi-Tiered Storage Environments},
  year = {2020},
  month = may,
  publisher = {IEEE},
  pages = {557--566},
  doi = {10.1109/IPDPS47924.2020.00064},
  url = {https://ieeexplore.ieee.org/abstract/document/9139838},
}

Related notes

  1. essay

    One destination, N tiers

    Every tier supplies the application at once, and the application sees one destination. The thesis from a November 2026 talk to my group at the Gnosis Research Center, the six lines of I/O work behind it since 2016, and my bet that they are the foundation agents need for context.

    9 min
  2. update

    Hermes is complete

    Hermes v1.2.1 is the final release of a project that started as the center of my PhD and ran for five years with The HDF Group. What the multi-tier buffering bet proved, what it did not, and what moves into IOWarp.

    5 min
  3. update

    CLIO Core 2.3 and 2.4: when a tier fills or a node dies

    Four CLIO Core releases between October 9 and November 2, 2026 added a device-paged GPU vector and Intel GPU support, then taught the context store to recover a full tier and keep data and directories readable when a node goes down.

    3 min