To Derive or Not to Derive: I/O Libraries Take Charge of Derived Quantities Computation
Abstract
The ever-increasing volume of data produced by HPC simulations necessitates scalable methods for data exploration and knowledge extraction. Scientific data analysis often involves complex queries across distributed datasets, requiring manipulation of multiple primary variables and generating derived data that needs to be handled efficiently, creating challenges for applications that need to parse many large datasets. Relying on individual applications to handle all intermediate data generally leads to redundant computations across studies and unnecessary data transfers. In this paper, we investigate the performance of different approaches where applications define derived variables as quantities of interest (QoIs) and offload the computation and transfer of these QoIs to the I/O library. This significantly reduces redundancy and optimizes data movement across the distributed storage and processing infrastructure by allowing control over when and where derived variables are computed. We present a detailed analysis of the performance-storage trade-offs associated with different solutions and showcase results for our study on two large-scale datasets created from climate and combustion simulations.
Finding
Offloading derived variables to the I/O library exposes a performance-storage trade-off: on S3D queries over primary data, writing only statistics for derived variables performs up to 15% better in total time than the expression-only strategy in situ and 25% better in total storage than storing the derived data.
System
- 2024–present
More papers on Coeus
Browse all papers on Coeus- 2026
Trigger, Render, Reason: Multimodal LLM Agents for Science-Aware Scientific Simulation Monitoring
Hua Xu, Jaime Cernuda, Ana Gainaru, Scott Klasky, Anthony Kougkas, Xian-He Sun. International Conference for High Performance Computing, Networking, Storage, and Analysis (SC 2026)
- 2026
Trigger-Gated Multimodal LLM Agents for In-situ Scientific Simulation Monitoring
Hua Xu, Jaime Cernuda, Anthony Kougkas, Ana Gainaru, Scott Klasky, Xian-He Sun et al. 11th International Parallel Data Systems Workshop (PDSW 2026), held in conjunction with SC26
- 2026
Type-Safe, Dynamically Defined Metadata Support for Scientific Applications
J. Teves, C. Brinker, Anthony Kougkas, Ana Gainaru, Scott Klasky, Xian-He Sun et al. 11th International Parallel Data Systems Workshop (PDSW 2026), held in conjunction with SC26
- 2026
Rich I/O-centric Derived Quantity Operators for Advanced In Situ Processing
Hua Xu, Jaime Cernuda, Xian-He Sun, Michela Taufer, Nigel Tan, Anthony Kougkas et al. 22nd IEEE International Conference on eScience (eScience 2026)
Citation
@inproceedings{gainaru2024derive,
author = {Gainaru, Ana and Podhorszki, Norbert and Dulac, Liz and Gong, Qian and Klasky, Scott and Eisenhauer, Greg and Kougkas, Antonios and Sun, Xian-He and Lofstead, Jay},
booktitle = {2024 IEEE 36th International Symposium on Computer Architecture and High Performance Computing (SBAC-PAD)},
title = {To Derive or Not to Derive: I/O Libraries Take Charge of Derived Quantities Computation},
year = {2024},
month = nov,
publisher = {IEEE Press},
pages = {105-115},
doi = {10.1109/SBAC-PAD63648.2024.00030},
url = {https://ieeexplore.ieee.org/document/10763877},
}