Jarvis: Towards a Shared, User-Friendly, and Reproducible, I/O Infrastructure.
Abstract
Hardware is becoming increasingly heterogeneous in modern high-performance computing clusters. However, computing environments for developing tools to harness these technologies are not easily available to researchers. This work showcases the need for a new high-pace, heterogeneous I/O research cluster and presents a novel software deployment framework named Jarvis to manage its hardware diversity. Jarvis is an extensible Python framework that allows users to create Packages that deploy, manage, and monitor software, including complex applications (e.g., scientific simulations), support tools (e.g., Darshan, GDB), and storage systems (e.g., Lustre, DAOS). These packages can be combined to form complex deployment Pipelines. To ensure pipelines are portable across hardware, Jarvis defines a novel Resource Graph schema file, which is a snapshot of a cluster’s machine-specific information. This schema can be queried by Jarvis packages to deploy software across diverse hardware compositions with minimal user effort.
Finding
Jarvis uses a Resource Graph snapshot of machine-specific information to generate machine-optimized configurations for its deployment Packages and Pipelines without code modification, and it has been tested on clusters from small university systems to national laboratories.
System
- 2024–present
- 2020–present
More papers on IOWarp
Browse all papers on IOWarp- 2026
Trigger, Render, Reason: Multimodal LLM Agents for Science-Aware Scientific Simulation Monitoring
Hua Xu, Jaime Cernuda, Ana Gainaru, Scott Klasky, Anthony Kougkas, Xian-He Sun. International Conference for High Performance Computing, Networking, Storage, and Analysis (SC 2026)
- 2026
GPUH5: A GPU-first I/O Library for Scientific Data
J. Fernandes, Luke Logan, Xian-He Sun, Anthony Kougkas. 11th International Parallel Data Systems Workshop (PDSW 2026), held in conjunction with SC26
- 2026
Trigger-Gated Multimodal LLM Agents for In-situ Scientific Simulation Monitoring
Hua Xu, Jaime Cernuda, Anthony Kougkas, Ana Gainaru, Scott Klasky, Xian-He Sun et al. 11th International Parallel Data Systems Workshop (PDSW 2026), held in conjunction with SC26
- 2026
Type-Safe, Dynamically Defined Metadata Support for Scientific Applications
J. Teves, C. Brinker, Anthony Kougkas, Ana Gainaru, Scott Klasky, Xian-He Sun et al. 11th International Parallel Data Systems Workshop (PDSW 2026), held in conjunction with SC26
Citation
@inproceedings{cernuda2024jarvis,
author = {Cernuda, J. and Logan, L. and Lewis, N. and Byna, S. and Sun, X.-H. and Kougkas, A.},
title = {{Jarvis: Towards a Shared, User-Friendly, and Reproducible, I/O Infrastructure.}},
booktitle = {The International Parallel Data Systems Workshop (PDSW'24)},
year = {2024},
month = nov,
url = {http://cs.iit.edu/%7Escs/assets/files/cernuda2024jarvis.pdf},
} Related notes
- 4 min
Two PhDs, one architecture
Luke Logan and Jaime Cernuda defended in the same semester. Their dissertations are the two halves of the runtime and the agent layer we are building, and they got there by owning the problems, not by executing mine.
- 9 min
One destination, N tiers
Every tier supplies the application at once, and the application sees one destination. The thesis from a November 2026 talk to my group at the Gnosis Research Center, the six lines of I/O work behind it since 2016, and my bet that they are the foundation agents need for context.
- 4 min
SC26 in Chicago
What the group brings to SC26 in Chicago, the pressure from GPUs and agent fleets that runs through all of it, and an invitation to Thursday's Birds of a Feather on HDF5 in the age of AI-driven science.