akougkas.io

Rich I/O-centric Derived Quantity Operators for Advanced In Situ Processing

Hua Xu, Jaime Cernuda, Xian-He Sun, Michela Taufer, Nigel Tan, Anthony Kougkas, Ana Gainaru, Josh Teves, Kevin Assogba, Bogdan Nicolae, Line Pouchard, Curtis Brinker, M. Mustafa Rafique, Scott Klasky, Jay Lofstead

22nd IEEE International Conference on eScience (eScience 2026)IEEEPresented

Open the DOI

Abstract

Scientific simulations produce increasingly large datasets that surpass practical limits for frequent input and output operations. Although performing analysis directly as data is produced (in situ) can mitigate this problem, embedding such analysis routines into complex scientific codes is often impractical or inflexible. We address this by introducing derived quantity operators that provide a modular and pluggable framework within high-level I/O libraries, enabling sophisticated in situ analysis to be performed transparently during data output with minimal modifications to the host application. The operators bind to persistent hierarchical storage through operator-declared metadata, allowing cross-timestep and cross-run analysis to be carried out seamlessly within the I/O path. We demonstrate the capabilities of our operators by integrating correctness validation and important scientific metrics, traditionally executed after simulation runs, directly into the I/O pathway as derived quantities. Our implementation illustrates how complex, non-mathematical analyses such as runtime validation can be efficiently performed during simulation, enabling early detection of errors. Our work broadens the range of in situ analyses achievable through derived quantity operators, promoting more flexible, intelligent, and reliable scientific workflows.

Finding

Derived quantity operators bound to persistent hierarchical storage through metadata run correctness validation and scientific metrics in the I/O path across Gray-Scott, WRF, and Xcompact3D, and on three Xcompact3D benchmarks the stencil operator reduces total execution time against an in-solver ring-buffer baseline in all nine configurations while its history retrieval adds only ≈3-5% to I/O time.

System

  1. 2024–present
    active

    IOWarp

  2. completed

    Coeus

Browse all papers on IOWarp
  1. 2026
    conference

    Trigger, Render, Reason: Multimodal LLM Agents for Science-Aware Scientific Simulation Monitoring

    Hua Xu, Jaime Cernuda, Ana Gainaru, Scott Klasky, Anthony Kougkas, Xian-He Sun. International Conference for High Performance Computing, Networking, Storage, and Analysis (SC 2026)

  2. 2026
    workshop

    GPUH5: A GPU-first I/O Library for Scientific Data

    J. Fernandes, Luke Logan, Xian-He Sun, Anthony Kougkas. 11th International Parallel Data Systems Workshop (PDSW 2026), held in conjunction with SC26

  3. 2026
    workshop

    Trigger-Gated Multimodal LLM Agents for In-situ Scientific Simulation Monitoring

    Hua Xu, Jaime Cernuda, Anthony Kougkas, Ana Gainaru, Scott Klasky, Xian-He Sun et al. 11th International Parallel Data Systems Workshop (PDSW 2026), held in conjunction with SC26

  4. 2026
    workshop

    Type-Safe, Dynamically Defined Metadata Support for Scientific Applications

    J. Teves, C. Brinker, Anthony Kougkas, Ana Gainaru, Scott Klasky, Xian-He Sun et al. 11th International Parallel Data Systems Workshop (PDSW 2026), held in conjunction with SC26

Citation

@inproceedings{xu2026derivedquantity,
  author = {Xu, Hua and Cernuda, Jaime and Sun, Xian-He and Taufer, Michela and Tan, Nigel and Kougkas, Anthony and Gainaru, Ana and Teves, Josh and Assogba, Kevin and Nicolae, Bogdan and Pouchard, Line and Brinker, Curtis and Rafique, M. Mustafa and Klasky, Scott and Lofstead, Jay},
  title = {{Rich I/O-centric Derived Quantity Operators for Advanced In Situ Processing}},
  booktitle = {22nd IEEE International Conference on eScience (eScience 2026)},
  year = {2026},
  month = sep,
  doi = {10.1109/eScience70237.2026.00053},
  pages = {392--400},
  url = {https://www.escience-conference.org/2026/papers},
  publisher = {IEEE},
}

Related notes

  1. essay

    One destination, N tiers

    Every tier supplies the application at once, and the application sees one destination. The thesis from a November 2026 talk to my group at the Gnosis Research Center, the six lines of I/O work behind it since 2016, and my bet that they are the foundation agents need for context.

    9 min
  2. field note

    SC26 in Chicago

    What the group brings to SC26 in Chicago, the pressure from GPUs and agent fleets that runs through all of it, and an invitation to Thursday's Birds of a Feather on HDF5 in the age of AI-driven science.

    4 min
  3. essay

    CLIO as the entry point

    CLIO should become the single place a developer starts with IOWarp, built as an application SDK with deployment profiles rather than an umbrella package or a gateway service. This is the direction for 2027; the API and the scope are still open.

    6 min