akougkas.io

Jarvis: Towards a Shared, User-Friendly, and Reproducible, I/O Infrastructure.

Jaime Cernuda, Luke Logan, N. Lewis, Suren Byna, Xian-He Sun, Anthony Kougkas

The International Parallel Data Systems Workshop (PDSW’24)

Abstract

Hardware is becoming increasingly heterogeneous in modern high-performance computing clusters. However, computing environments for developing tools to harness these technologies are not easily available to researchers. This work showcases the need for a new high-pace, heterogeneous I/O research cluster and presents a novel software deployment framework named Jarvis to manage its hardware diversity. Jarvis is an extensible Python framework that allows users to create Packages that deploy, manage, and monitor software, including complex applications (e.g., scientific simulations), support tools (e.g., Darshan, GDB), and storage systems (e.g., Lustre, DAOS). These packages can be combined to form complex deployment Pipelines. To ensure pipelines are portable across hardware, Jarvis defines a novel Resource Graph schema file, which is a snapshot of a cluster’s machine-specific information. This schema can be queried by Jarvis packages to deploy software across diverse hardware compositions with minimal user effort.

Finding

Jarvis uses a Resource Graph snapshot of machine-specific information to generate machine-optimized configurations for its deployment Packages and Pipelines without code modification, and it has been tested on clusters from small university systems to national laboratories.

System

  1. 2024–present
    active

    IOWarp

  2. 2020–present
    active

    ChronoLog

  3. completed

    Coeus

Browse all papers on IOWarp
  1. 2026
    conference

    Trigger, Render, Reason: Multimodal LLM Agents for Science-Aware Scientific Simulation Monitoring

    Hua Xu, Jaime Cernuda, Ana Gainaru, Scott Klasky, Anthony Kougkas, Xian-He Sun. International Conference for High Performance Computing, Networking, Storage, and Analysis (SC 2026)

  2. 2026
    workshop

    GPUH5: A GPU-first I/O Library for Scientific Data

    J. Fernandes, Luke Logan, Xian-He Sun, Anthony Kougkas. 11th International Parallel Data Systems Workshop (PDSW 2026), held in conjunction with SC26

  3. 2026
    workshop

    Trigger-Gated Multimodal LLM Agents for In-situ Scientific Simulation Monitoring

    Hua Xu, Jaime Cernuda, Anthony Kougkas, Ana Gainaru, Scott Klasky, Xian-He Sun et al. 11th International Parallel Data Systems Workshop (PDSW 2026), held in conjunction with SC26

  4. 2026
    workshop

    Type-Safe, Dynamically Defined Metadata Support for Scientific Applications

    J. Teves, C. Brinker, Anthony Kougkas, Ana Gainaru, Scott Klasky, Xian-He Sun et al. 11th International Parallel Data Systems Workshop (PDSW 2026), held in conjunction with SC26

Citation

@inproceedings{cernuda2024jarvis,
  author = {Cernuda, J. and Logan, L. and Lewis, N. and Byna, S. and Sun, X.-H. and Kougkas, A.},
  title = {{Jarvis: Towards a Shared, User-Friendly, and Reproducible, I/O Infrastructure.}},
  booktitle = {The International Parallel Data Systems Workshop (PDSW'24)},
  year = {2024},
  month = nov,
  url = {http://cs.iit.edu/%7Escs/assets/files/cernuda2024jarvis.pdf},
}

Related notes

  1. field note

    Two PhDs, one architecture

    Luke Logan and Jaime Cernuda defended in the same semester. Their dissertations are the two halves of the runtime and the agent layer we are building, and they got there by owning the problems, not by executing mine.

    4 min
  2. essay

    One destination, N tiers

    Every tier supplies the application at once, and the application sees one destination. The thesis from a November 2026 talk to my group at the Gnosis Research Center, the six lines of I/O work behind it since 2016, and my bet that they are the foundation agents need for context.

    9 min
  3. field note

    SC26 in Chicago

    What the group brings to SC26 in Chicago, the pressure from GPUs and agent fleets that runs through all of it, and an invitation to Thursday's Birds of a Feather on HDF5 in the age of AI-driven science.

    4 min