To Derive or Not to Derive: I/O Libraries Take Charge of Derived Quantities Computation
2024 IEEE 36th International Symposium on Computer Architecture and High Performance Computing (SBAC-PAD) · IEEE Press
Abstract
Data streaming is gaining traction in high-performance computing (HPC) as a mechanism for continuous data transfer, but remains underutilized as a processing paradigm due to the inadequacy of existing technologies, which are primarily designed for cloud architectures and ill-equipped to tackle HPC-specific challenges. This work introduces a novel approach where I/O libraries take charge of computing derived quantities. By managing the computation of these quantities within the I/O stack, issues such as redundant computations and data movement can be effectively addressed at runtime. The proposed solution demonstrates significant performance improvements and reduced resource utilization in HPC environments.
Connected work
Citation
@inproceedings{gainaru2024derive,
author = {Gainaru, Ana and Podhorszki, Norbert and Dulac, Liz and Gong, Qian and Klasky, Scott and Eisenhauer, Greg and Kougkas, Antonios and Sun, Xian-He and Lofstead, Jay},
booktitle = {2024 IEEE 36th International Symposium on Computer Architecture and High Performance Computing (SBAC-PAD)},
title = {To Derive or Not to Derive: I/O Libraries Take Charge of Derived Quantities Computation},
year = {2024},
month = nov,
publisher = {IEEE Press},
volume = {},
number = {},
pages = {105-115},
keywords = {Large-scale I/O, Derived Variables, HPC Analysis, Queries for Scientific Data, Quantities of Interest},
doi = {10.1109/SBAC-PAD63648.2024.00030},
url = {https://ieeexplore.ieee.org/document/10763877},
}