HReplica: A Dynamic Data Replication Engine with Adaptive Compression for Multi-Tiered Storage
Abstract
As the diversity of big data applications increases, their requirements diverge and often conflict with one other. Managing this diversity in any supercomputer or data center is a major challenge for system designers. Data replication is a popular approach to meet several of these requirements, such as low latency, read availability, durability, etc. This approach can be enhanced using new modern heterogeneous hardware and software techniques such as data compression. However, both these enhancements work in isolation to the detriment of both. In this work, we present HReplica: a dynamic data replication engine which harmoniously leverages data compression and hierarchical storage to increase the effectiveness of data replication. We have developed a novel dynamic selection algorithm that facilitates the optimal matching of replication schemes, compression libraries, and tiered storage. Our evaluation shows that HReplica can improve scientific and cloud application performance by 5.2x when compared to other state-of-the-art replication schemes.
Finding
Jointly selecting replication schemes, compression libraries, and storage tiers lets HReplica improve application performance by 5.2x over state-of-the-art replication schemes.
System
- 2018–2024
- 2023–2026
More papers on Hermes
Browse all papers on Hermes- 2026
Benchmarking the Performance of Semantic Search Techniques for Agent Discovery
Rajni Pawar, Luke Logan, Xian-He Sun, Anthony Kougkas. 13th Greater Chicago Area Systems Research Workshop (GCASR 2026)
- 2024
MegaMmap: Blurring the Boundary Between Memory and Storage for Data-Intensive Workloads
Luke Logan, Anthony Kougkas, Xian-He Sun. Proceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis SC
- 2024
DaYu: Optimizing Distributed Scientific Workflows by Decoding Dataflow Semantics and Dynamics
Meng Tang, Jaime Cernuda, Jie Ye, Luanzheng Guo, Nathan R. Tallent, Anthony Kougkas et al. Proceedings of the IEEE International Conference on Cluster Computing
- 2024
HStream: A hierarchical data streaming engine for high-throughput scientific applications
Jaime Cernuda, Jie Ye, Anthony Kougkas, Xian-He Sun. Proceedings of the 53rd International Conference on Parallel Processing
Citation
@inproceedings{devarajan2020hreplica,
author = {Devarajan, Hariharan and Kougkas, Anthony and Sun, Xian-He},
booktitle = {Proceedings of the International Conference on Big Data},
title = {{HReplica}: A Dynamic Data Replication Engine with Adaptive Compression for Multi-Tiered Storage},
year = {2020},
month = dec,
publisher = {IEEE},
pages = {256--265},
doi = {10.1109/BigData50022.2020.9378167},
url = {https://ieeexplore.ieee.org/abstract/document/9378167},
} Related notes
- 9 min
One destination, N tiers
Every tier supplies the application at once, and the application sees one destination. The thesis from a November 2026 talk to my group at the Gnosis Research Center, the six lines of I/O work behind it since 2016, and my bet that they are the foundation agents need for context.
- 5 min
Hermes is complete
Hermes v1.2.1 is the final release of a project that started as the center of my PhD and ran for five years with The HDF Group. What the multi-tier buffering bet proved, what it did not, and what moves into IOWarp.
- 3 min
CLIO Core 2.3 and 2.4: when a tier fills or a node dies
Four CLIO Core releases between October 9 and November 2, 2026 added a device-paged GPU vector and Intel GPU support, then taught the context store to recover a full tier and keep data and directories readable when a node goes down.