Hades: A Context-Aware Active Storage Framework for Accelerating Large-Scale Data Analysis
Abstract
Modern simulation workflows generate and analyze massive amounts of data using I/O libraries like Adios2 and NetCDF. Although extensive work has optimized the I/O processes during the simulation phase, executing analytical queries—which often require iterative traversals of large files for insights—is cumbersome and usually constrained by low I/O performance. Instead of waiting for the analysis phase to process queries, quantities can be derived asynchronously during data production and cached, speeding up future queries. In this work, we introduce a context-aware I/O layer named ’Hades.’ It is designed to efficiently derive insights from selected quantities without compromising overall workflow performance. Hades actively and asynchronously computes and stores these quantities while the data is in transit. Hades leverages a hierarchical buffering system with data access-aware prefetching to ensure quick and timely access to relevant data. It offers a flexible query interface empowering users to easily define derived quantities and provide control over data placement decisions. Hades is implemented using an Adios2 plugin engine and the Hermes buffering platform, enabling transparent use by any Adios-powered application or workflow. Experimental results demonstrate performance improvements by up to 3-4x for tested real-world scientific producer-consumer workflows.
Finding
Hades computes derived quantities asynchronously while data is in transit and delivers performance improvements of up to 3-4x for real-world scientific producer-consumer workflows.
System
-
Coeus
Metadata-aware I/O adapters for in-situ scientific workflows.
More papers on Coeus
Browse all papers on Coeus- 2026
Trigger, Render, Reason: Multimodal LLM Agents for Science-Aware Scientific Simulation Monitoring
Hua Xu, Jaime Cernuda, Ana Gainaru, Scott Klasky, Anthony Kougkas, Xian-He Sun. International Conference for High Performance Computing, Networking, Storage, and Analysis (SC 2026)
- 2026
Trigger-Gated Multimodal LLM Agents for In-situ Scientific Simulation Monitoring
Hua Xu, Jaime Cernuda, Anthony Kougkas, Ana Gainaru, Scott Klasky, Xian-He Sun et al. 11th International Parallel Data Systems Workshop (PDSW 2026), held in conjunction with SC26
- 2026
Type-Safe, Dynamically Defined Metadata Support for Scientific Applications
J. Teves, C. Brinker, Anthony Kougkas, Ana Gainaru, Scott Klasky, Xian-He Sun et al. 11th International Parallel Data Systems Workshop (PDSW 2026), held in conjunction with SC26
- 2026
Rich I/O-centric Derived Quantity Operators for Advanced In Situ Processing
Hua Xu, Jaime Cernuda, Xian-He Sun, Michela Taufer, Nigel Tan, Anthony Kougkas et al. 22nd IEEE International Conference on eScience (eScience 2026)
Citation
@inproceedings{cernuda2024hades,
author = {Cernuda, Jaime and Logan, Luke and Gainaru, Ana and Klasky, Scott and Lofstead, Jay and Kougkas, Anthony and Sun, Xian-He},
booktitle = {Proceedings of the 24th International Symposium on Cluster, Cloud and Internet Computing},
title = {Hades: A Context-Aware Active Storage Framework for Accelerating Large-Scale Data Analysis},
year = {2024},
month = may,
publisher = {IEEE},
pages = {577-586},
doi = {10.1109/CCGrid59990.2024.00070},
url = {https://ieeexplore.ieee.org/document/10701392},
} Related notes
- 4 min
Two PhDs, one architecture
Luke Logan and Jaime Cernuda defended in the same semester. Their dissertations are the two halves of the runtime and the agent layer we are building, and they got there by owning the problems, not by executing mine.