akougkas.io

Viper: A High-Performance I/O Framework for Transparently Updating, Storing, and Transferring Deep Neural Network Models

Jie Ye, Jaime Cernuda, Neeraj Rajesh, Keith Bateman, Orcun Yildiz, Tom Peterka, Arnur Nigmetov, Dmitriy Morozov, Xian-He Sun, Anthony Kougkas, Bogdan Nicolae

Proceedings of the 53rd International Conference on Parallel ProcessingACM

Abstract

Scientific workflows increasingly need to train a DNN model in real-time during an experiment (e.g. using ground truth from a simulation), while using it at the same time for inferences. Instead of sharing the same model instance, the training (producer) and inference server (consumer) often use different model replicas that are kept synchronized. In addition to efficient I/O techniques to keep the model replica of the producer and consumer synchronized, there is another important trade-off: frequent model updates enhance inference quality but may slow down training; infrequent updates may lead to less precise inference results. To address these challenges, we introduce Viper: a new I/O framework designed to determine a near-optimal checkpoint schedule and accelerate the delivery of the latest model updates. Viper builds an inference performance predictor to identify the optimal checkpoint schedule to balance the trade-off between training slowdown and inference quality improvement. It also creates a memory-first model transfer engine to accelerate model delivery through direct memory-to-memory communication. Our experiments show that Viper can reduce the model update latency by ≈ 9x using the GPU-to-GPU data transfer engine and ≈ 3x using the DRAM-to-DRAM host data transfer. The checkpoint schedule obtained from Viper’s predictor also demonstrates improved cumulative inference accuracy compared to the baseline of epoch-based solutions.

Finding

Viper reduces model update latency by about 9x with GPU-to-GPU transfer and about 3x with DRAM-to-DRAM transfer, while its checkpoint schedule improves cumulative inference accuracy.

System

  1. 2020–present
    active

    DeepIO

  2. 2023–2026
    complete

    DTIO

  3. 2024–present
    active

    IOWarp

Browse all papers on DeepIO
  1. 2026
    poster

    Extending I/O Observability to Explainability for Targeted Optimization

    Neeraj Rajesh, S. Neuwirth, Jay Lofstead, Anthony Kougkas, Xian-He Sun, Nikoli Dryden et al. International Conference for High Performance Computing, Networking, Storage, and Analysis (SC 2026)

  2. 2026
    conference

    PKAS: Predictive KVCache-Aware Scheduling for Faster LLM and Transformer Inferences

    Jie Ye, A. Maurya, K. Chitty-Venkata, Bogdan Nicolae, Anthony Kougkas, Xian-He Sun. 35th International Symposium on High-Performance Parallel and Distributed Computing (HPDC 2026)

  3. 2026
    poster

    Pre-RoPE versus Post-RoPE: Trading Per-Step Compute for KVCache Survivability

    Zia Uddin Chowdhury, Jaime Cernuda, Jie Ye, Anthony Kougkas, Xian-He Sun. 13th Greater Chicago Area Systems Research Workshop (GCASR 2026)

  4. 2026
    conference

    PagedEviction: Structured Block-wise KV Cache Pruning for Efficient Large Language Model Inference

    Krishna Teja Chitty-Venkata, Jie Ye, Siddhisanket Raskar, Xian-He Sun, Anthony Kougkas, Murali Emani et al. Findings of the Association for Computational Linguistics (EACL 2026)

Citation

@inproceedings{ye2024viper,
  author = {Ye, Jie and Cernuda, Jaime and Rajesh, Neeraj and Bateman, Keith and Yildiz, Orcun and Peterka, Tom and Nigmetov, Arnur and Morozov, Dmitriy and Sun, Xian-He and Kougkas, Anthony and Nicolae, Bogdan},
  booktitle = {Proceedings of the 53rd International Conference on Parallel Processing},
  title = {Viper: A High-Performance I/O Framework for Transparently Updating, Storing, and Transferring Deep Neural Network Models},
  year = {2024},
  month = aug,
  publisher = {ACM},
  pages = {812-821},
  doi = {10.1145/3673038.3673070},
  url = {https://doi.org/10.1145/3673038.3673070},
}

Related notes

  1. essay

    When is research software shipped?

    A working definition of "shipped" for research software, built around a named state someone else can run, evidence tied to that state, explicit limits, and a continuity signal, with Hermes and DLIO as the worked examples.

    7 min
  2. essay

    We need an Agent500

    Agent leaderboards score answers. Science needs a ranking that runs an agent through a real workflow with real tools and data and measures correctness, cost, I/O efficiency, and provenance together.

    7 min
  3. field note

    Reflecting on SC25

    What I noticed at SC25 in St. Louis, what the group brought, what a year of running the PDSW program committee taught me about how a workshop gets made, and a few notes for people attending their first SC.

    8 min