akougkas.io

An intelligent, adaptive, and flexible data compression framework

Hariharan Devarajan, Anthony Kougkas, Xian-He Sun

Proceedings of the 19th International Symposium on Cluster, Cloud and Grid ComputingIEEE

Abstract

The data explosion phenomenon in modern applications causes tremendous stress on storage systems. Developers use data compression, a size-reduction technique, to address this issue. However, each compression library exhibits different strengths and weaknesses when considering the input data entry_type and format. We present Ares, an intelligent, adaptive, and flexible compression framework which can dynamically choose a compression library for a given input data based on the entry_type of the workload and provides an appropriate infrastructure to users to fine-tune the chosen library. Ares is a modular framework which unifies several compression libraries while allowing the addition of more compression libraries by the user. Ares is a unified compression engine that abstracts the complexity of using different compression libraries for each workload. Evaluation results show that under real-world applications, from both scientific and Cloud domains, Ares performed 2-6x faster than competitive solutions with a low cost of additional data analysis (i.e., overheads around 10%) and up to 10x faster against a baseline of no compression at all.

Finding

Ares chooses a compression library for each workload and ran 2-6x faster than competing solutions with overheads around 10%, and up to 10x faster than no compression.

System

  1. 2018–2024
    complete

    Hermes

  2. 2017–present
    evolving

    LABIOS

  3. 2020–present
    active

    ChronoLog

  4. 2023–2026
    complete

    DTIO

Browse all papers on Hermes
  1. 2026
    poster

    Benchmarking the Performance of Semantic Search Techniques for Agent Discovery

    Rajni Pawar, Luke Logan, Xian-He Sun, Anthony Kougkas. 13th Greater Chicago Area Systems Research Workshop (GCASR 2026)

  2. 2024
    conference

    MegaMmap: Blurring the Boundary Between Memory and Storage for Data-Intensive Workloads

    Luke Logan, Anthony Kougkas, Xian-He Sun. Proceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis SC

  3. 2024
    conference

    DaYu: Optimizing Distributed Scientific Workflows by Decoding Dataflow Semantics and Dynamics

    Meng Tang, Jaime Cernuda, Jie Ye, Luanzheng Guo, Nathan R. Tallent, Anthony Kougkas et al. Proceedings of the IEEE International Conference on Cluster Computing

  4. 2024
    conference

    HStream: A hierarchical data streaming engine for high-throughput scientific applications

    Jaime Cernuda, Jie Ye, Anthony Kougkas, Xian-He Sun. Proceedings of the 53rd International Conference on Parallel Processing

Citation

@inproceedings{devarajan2019intelligent,
  author = {Devarajan, Hariharan and Kougkas, Anthony and Sun, Xian-He},
  booktitle = {Proceedings of the 19th International Symposium on Cluster, Cloud and Grid Computing},
  title = {An intelligent, adaptive, and flexible data compression framework},
  year = {2019},
  month = may,
  publisher = {IEEE},
  pages = {82--91},
  doi = {10.1109/CCGRID.2019.00019},
  url = {https://ieeexplore.ieee.org/abstract/document/8752926},
}

Related notes

  1. essay

    One destination, N tiers

    Every tier supplies the application at once, and the application sees one destination. The thesis from a November 2026 talk to my group at the Gnosis Research Center, the six lines of I/O work behind it since 2016, and my bet that they are the foundation agents need for context.

    9 min
  2. update

    CLIO Core 2.3 and 2.4: when a tier fills or a node dies

    Four CLIO Core releases between October 9 and November 2, 2026 added a device-paged GPU vector and Intel GPU support, then taught the context store to recover a full tier and keep data and directories readable when a node goes down.

    3 min
  3. update

    Two years of IOWarp

    Two years after the NSF award, what IOWarp promised, what shipped as public releases, what changed course, and what is still a plan.

    4 min