akougkas.io

HStream: A hierarchical data streaming engine for high-throughput scientific applications

Jaime Cernuda, Jie Ye, Anthony Kougkas, Xian-He Sun

Proceedings of the 53rd International Conference on Parallel ProcessingACM

Abstract

Data streaming is gaining traction in high-performance computing (HPC) as a mechanism for continuous data transfer, but remains underutilized as a processing paradigm due to the inadequacy of existing technologies, which are primarily designed for cloud architectures and ill-equipped to tackle HPC-specific challenges. This work introduces HStream, a novel data management design for out-of-core data streaming engines. Central to the HStream design is the separation of data and computing planes at the task level. By managing them independently, issues such as memory thrashing and back-pressure, caused by the high volume, velocity, and burstiness of I/O in HPC environments, can be effectively addressed at runtime. Specifically, HStream utilizes adaptive parallelism and hierarchical memory management, enabled by this design paradigm, to alleviate memory pressure and enhance system performance. These improvements enable HStream to match the performance of state-of-the-art HPC streaming engines and achieve up to a 1.5x reduction in latency under high data loads.

Finding

HStream shows up to a 1.5x decrease in cluster-wide execution time under a multi-tenant deployment, up to a 75% reduction in latency under high data volume, and up to a 2x throughput increase over state-of-the-art HPC streaming engines as an AI inference service.

System

  1. 2024–present
    active

    IOWarp

  2. 2018–2024
    complete

    Hermes

  3. 2017–present
    evolving

    LABIOS

  4. 2020–present
    active

    ChronoLog

  5. completed

    Coeus

Browse all papers on IOWarp
  1. 2026
    conference

    Trigger, Render, Reason: Multimodal LLM Agents for Science-Aware Scientific Simulation Monitoring

    Hua Xu, Jaime Cernuda, Ana Gainaru, Scott Klasky, Anthony Kougkas, Xian-He Sun. International Conference for High Performance Computing, Networking, Storage, and Analysis (SC 2026)

  2. 2026
    workshop

    GPUH5: A GPU-first I/O Library for Scientific Data

    J. Fernandes, Luke Logan, Xian-He Sun, Anthony Kougkas. 11th International Parallel Data Systems Workshop (PDSW 2026), held in conjunction with SC26

  3. 2026
    workshop

    Trigger-Gated Multimodal LLM Agents for In-situ Scientific Simulation Monitoring

    Hua Xu, Jaime Cernuda, Anthony Kougkas, Ana Gainaru, Scott Klasky, Xian-He Sun et al. 11th International Parallel Data Systems Workshop (PDSW 2026), held in conjunction with SC26

  4. 2026
    workshop

    Type-Safe, Dynamically Defined Metadata Support for Scientific Applications

    J. Teves, C. Brinker, Anthony Kougkas, Ana Gainaru, Scott Klasky, Xian-He Sun et al. 11th International Parallel Data Systems Workshop (PDSW 2026), held in conjunction with SC26

Citation

@inproceedings{cernuda2024hstream,
  author = {Cernuda, Jaime and Ye, Jie and Kougkas, Anthony and Sun, Xian-He},
  booktitle = {Proceedings of the 53rd International Conference on Parallel Processing},
  title = {HStream: A hierarchical data streaming engine for high-throughput scientific applications},
  year = {2024},
  month = aug,
  publisher = {ACM},
  pages = {231-240},
  doi = {10.1145/3673038.3673150},
  url = {https://dl.acm.org/doi/abs/10.1145/3673038.3673150},
}

Related notes

  1. field note

    Two PhDs, one architecture

    Luke Logan and Jaime Cernuda defended in the same semester. Their dissertations are the two halves of the runtime and the agent layer we are building, and they got there by owning the problems, not by executing mine.

    4 min
  2. essay

    One destination, N tiers

    Every tier supplies the application at once, and the application sees one destination. The thesis from a November 2026 talk to my group at the Gnosis Research Center, the six lines of I/O work behind it since 2016, and my bet that they are the foundation agents need for context.

    9 min
  3. field note

    SC26 in Chicago

    What the group brings to SC26 in Chicago, the pressure from GPUs and agent fleets that runs through all of it, and an invitation to Thursday's Birds of a Feather on HDF5 in the age of AI-driven science.

    4 min