Rich I/O-centric Derived Quantity Operators for Advanced In Situ Processing
Abstract
Scientific simulations produce increasingly large datasets that surpass practical limits for frequent input and output operations. Although performing analysis directly as data is produced (in situ) can mitigate this problem, embedding such analysis routines into complex scientific codes is often impractical or inflexible. We address this by introducing derived quantity operators that provide a modular and pluggable framework within high-level I/O libraries, enabling sophisticated in situ analysis to be performed transparently during data output with minimal modifications to the host application. The operators bind to persistent hierarchical storage through operator-declared metadata, allowing cross-timestep and cross-run analysis to be carried out seamlessly within the I/O path. We demonstrate the capabilities of our operators by integrating correctness validation and important scientific metrics, traditionally executed after simulation runs, directly into the I/O pathway as derived quantities. Our implementation illustrates how complex, non-mathematical analyses such as runtime validation can be efficiently performed during simulation, enabling early detection of errors. Our work broadens the range of in situ analyses achievable through derived quantity operators, promoting more flexible, intelligent, and reliable scientific workflows.
Finding
Derived quantity operators bound to persistent hierarchical storage through metadata run correctness validation and scientific metrics in the I/O path across Gray-Scott, WRF, and Xcompact3D, and on three Xcompact3D benchmarks the stencil operator reduces total execution time against an in-solver ring-buffer baseline in all nine configurations while its history retrieval adds only ≈3-5% to I/O time.
System
- 2024–present
More papers on IOWarp
Browse all papers on IOWarp- 2026
Trigger, Render, Reason: Multimodal LLM Agents for Science-Aware Scientific Simulation Monitoring
Hua Xu, Jaime Cernuda, Ana Gainaru, Scott Klasky, Anthony Kougkas, Xian-He Sun. International Conference for High Performance Computing, Networking, Storage, and Analysis (SC 2026)
- 2026
GPUH5: A GPU-first I/O Library for Scientific Data
J. Fernandes, Luke Logan, Xian-He Sun, Anthony Kougkas. 11th International Parallel Data Systems Workshop (PDSW 2026), held in conjunction with SC26
- 2026
Trigger-Gated Multimodal LLM Agents for In-situ Scientific Simulation Monitoring
Hua Xu, Jaime Cernuda, Anthony Kougkas, Ana Gainaru, Scott Klasky, Xian-He Sun et al. 11th International Parallel Data Systems Workshop (PDSW 2026), held in conjunction with SC26
- 2026
Type-Safe, Dynamically Defined Metadata Support for Scientific Applications
J. Teves, C. Brinker, Anthony Kougkas, Ana Gainaru, Scott Klasky, Xian-He Sun et al. 11th International Parallel Data Systems Workshop (PDSW 2026), held in conjunction with SC26
Citation
@inproceedings{xu2026derivedquantity,
author = {Xu, Hua and Cernuda, Jaime and Sun, Xian-He and Taufer, Michela and Tan, Nigel and Kougkas, Anthony and Gainaru, Ana and Teves, Josh and Assogba, Kevin and Nicolae, Bogdan and Pouchard, Line and Brinker, Curtis and Rafique, M. Mustafa and Klasky, Scott and Lofstead, Jay},
title = {{Rich I/O-centric Derived Quantity Operators for Advanced In Situ Processing}},
booktitle = {22nd IEEE International Conference on eScience (eScience 2026)},
year = {2026},
month = sep,
doi = {10.1109/eScience70237.2026.00053},
pages = {392--400},
url = {https://www.escience-conference.org/2026/papers},
publisher = {IEEE},
} Related notes
- 9 min
One destination, N tiers
Every tier supplies the application at once, and the application sees one destination. The thesis from a November 2026 talk to my group at the Gnosis Research Center, the six lines of I/O work behind it since 2016, and my bet that they are the foundation agents need for context.
- 4 min
SC26 in Chicago
What the group brings to SC26 in Chicago, the pressure from GPUs and agent fleets that runs through all of it, and an invitation to Thursday's Birds of a Feather on HDF5 in the age of AI-driven science.
- 6 min
CLIO as the entry point
CLIO should become the single place a developer starts with IOWarp, built as an application SDK with deployment profiles rather than an umbrella package or a gateway service. This is the direction for 2027; the API and the scope are still open.