akougkas.io

Stimulus: Accelerate Data Management for Scientific AI applications in HPC

Hariharan Devarajan, Anthony Kougkas, Huihuo Zheng, Venkatram Vishwanath, Xian-He Sun

Proceedings of the 22nd IEEE International Symposium on Cluster, Cloud and Internet ComputingIEEE

Abstract

Modern scientific workflows couple simulations with AI-powered analytics by frequently exchanging data to accelerate time-to-science to reduce the complexity of the simulation planes. However, this data exchange is limited in performance and portability due to a lack of support for scientific data formats in AI frameworks. We need a cohesive mechanism to effectively integrate at scale complex scientific data formats such as HDF5, PnetCDF, ADIOS2, GNCF, and Silo into popular AI frameworks such as TensorFlow, PyTorch, and Caffe. To this end, we designed Stimulus, a data management library for ingesting scientific data effectively into the popular AI frameworks. We utilize the StimOps functions along with StimPack abstraction to enable the integration of scientific data formats with any AI framework. The evaluations show that Stimulus outperforms several large-scale applications with different use-cases such as Cosmic Tagger (consuming HDF5 dataset in PyTorch), Distributed FFN (consuming HDF5 dataset in TensorFlow), and CosmoFlow (converting HDF5 into TFRecord and then consuming that in TensorFlow) by 5.3 x, 2.9 x, and 1.9 x respectively with ideal I/O scalability up to 768 GPUs on the Summit supercomputer. Through Stimulus, we can portably extend existing popular AI frameworks to cohesively support any complex scientific data format and efficiently scale the applications on large-scale supercomputers.

Finding

Stimulus ingests scientific data formats into AI frameworks and outperforms three large-scale applications by 5.3x, 2.9x, and 1.9x, with ideal I/O scalability up to 768 GPUs on Summit.

System

  1. 2024–present
    active

    IOWarp

  2. 2018–2024
    complete

    Hermes

  3. 2020–present
    active

    DeepIO

  4. completed

    IRIS

Browse all papers on IOWarp
  1. 2026
    conference

    Trigger, Render, Reason: Multimodal LLM Agents for Science-Aware Scientific Simulation Monitoring

    Hua Xu, Jaime Cernuda, Ana Gainaru, Scott Klasky, Anthony Kougkas, Xian-He Sun. International Conference for High Performance Computing, Networking, Storage, and Analysis (SC 2026)

  2. 2026
    workshop

    GPUH5: A GPU-first I/O Library for Scientific Data

    J. Fernandes, Luke Logan, Xian-He Sun, Anthony Kougkas. 11th International Parallel Data Systems Workshop (PDSW 2026), held in conjunction with SC26

  3. 2026
    workshop

    Trigger-Gated Multimodal LLM Agents for In-situ Scientific Simulation Monitoring

    Hua Xu, Jaime Cernuda, Anthony Kougkas, Ana Gainaru, Scott Klasky, Xian-He Sun et al. 11th International Parallel Data Systems Workshop (PDSW 2026), held in conjunction with SC26

  4. 2026
    workshop

    Type-Safe, Dynamically Defined Metadata Support for Scientific Applications

    J. Teves, C. Brinker, Anthony Kougkas, Ana Gainaru, Scott Klasky, Xian-He Sun et al. 11th International Parallel Data Systems Workshop (PDSW 2026), held in conjunction with SC26

Citation

@inproceedings{devarajan2022stimulus,
  author = {Devarajan, Hariharan and Kougkas, Anthony and Zheng, Huihuo and Vishwanath, Venkatram and Sun, Xian-He},
  booktitle = {Proceedings of the 22nd IEEE International Symposium on Cluster, Cloud and Internet Computing},
  title = {Stimulus: Accelerate Data Management for Scientific AI applications in HPC},
  year = {2022},
  month = may,
  publisher = {IEEE},
  pages = {109--118},
  doi = {10.1109/CCGrid54584.2022.00020},
  url = {https://ieeexplore.ieee.org/abstract/document/9826104},
}

Related notes

  1. essay

    Context is the new compute

    The scarce resource for scientific AI is not FLOPS or bytes but assembled, trustworthy context, and the storage we built as tanks has to become a refinery. From the AI4Sys keynote at HPDC 2025.

    8 min
  2. essay

    One destination, N tiers

    Every tier supplies the application at once, and the application sees one destination. The thesis from a November 2026 talk to my group at the Gnosis Research Center, the six lines of I/O work behind it since 2016, and my bet that they are the foundation agents need for context.

    9 min
  3. field note

    SC26 in Chicago

    What the group brings to SC26 in Chicago, the pressure from GPUs and agent fleets that runs through all of it, and an invitation to Thursday's Birds of a Feather on HDF5 in the age of AI-driven science.

    4 min