Viper: A High-Performance I/O Framework for Transparently Updating, Storing, and Transferring Deep Neural Network Models
Abstract
Scientific workflows increasingly need to train a DNN model in real-time during an experiment (e.g. using ground truth from a simulation), while using it at the same time for inferences. Instead of sharing the same model instance, the training (producer) and inference server (consumer) often use different model replicas that are kept synchronized. In addition to efficient I/O techniques to keep the model replica of the producer and consumer synchronized, there is another important trade-off: frequent model updates enhance inference quality but may slow down training; infrequent updates may lead to less precise inference results. To address these challenges, we introduce Viper: a new I/O framework designed to determine a near-optimal checkpoint schedule and accelerate the delivery of the latest model updates. Viper builds an inference performance predictor to identify the optimal checkpoint schedule to balance the trade-off between training slowdown and inference quality improvement. It also creates a memory-first model transfer engine to accelerate model delivery through direct memory-to-memory communication. Our experiments show that Viper can reduce the model update latency by ≈ 9x using the GPU-to-GPU data transfer engine and ≈ 3x using the DRAM-to-DRAM host data transfer. The checkpoint schedule obtained from Viper’s predictor also demonstrates improved cumulative inference accuracy compared to the baseline of epoch-based solutions.
Finding
Viper reduces model update latency by about 9x with GPU-to-GPU transfer and about 3x with DRAM-to-DRAM transfer, while its checkpoint schedule improves cumulative inference accuracy.
System
- 2020–present
- 2023–2026
- 2024–present
More papers on DeepIO
Browse all papers on DeepIO- 2026
Extending I/O Observability to Explainability for Targeted Optimization
Neeraj Rajesh, S. Neuwirth, Jay Lofstead, Anthony Kougkas, Xian-He Sun, Nikoli Dryden et al. International Conference for High Performance Computing, Networking, Storage, and Analysis (SC 2026)
- 2026
PKAS: Predictive KVCache-Aware Scheduling for Faster LLM and Transformer Inferences
Jie Ye, A. Maurya, K. Chitty-Venkata, Bogdan Nicolae, Anthony Kougkas, Xian-He Sun. 35th International Symposium on High-Performance Parallel and Distributed Computing (HPDC 2026)
- 2026
Pre-RoPE versus Post-RoPE: Trading Per-Step Compute for KVCache Survivability
Zia Uddin Chowdhury, Jaime Cernuda, Jie Ye, Anthony Kougkas, Xian-He Sun. 13th Greater Chicago Area Systems Research Workshop (GCASR 2026)
- 2026
PagedEviction: Structured Block-wise KV Cache Pruning for Efficient Large Language Model Inference
Krishna Teja Chitty-Venkata, Jie Ye, Siddhisanket Raskar, Xian-He Sun, Anthony Kougkas, Murali Emani et al. Findings of the Association for Computational Linguistics (EACL 2026)
Citation
@inproceedings{ye2024viper,
author = {Ye, Jie and Cernuda, Jaime and Rajesh, Neeraj and Bateman, Keith and Yildiz, Orcun and Peterka, Tom and Nigmetov, Arnur and Morozov, Dmitriy and Sun, Xian-He and Kougkas, Anthony and Nicolae, Bogdan},
booktitle = {Proceedings of the 53rd International Conference on Parallel Processing},
title = {Viper: A High-Performance I/O Framework for Transparently Updating, Storing, and Transferring Deep Neural Network Models},
year = {2024},
month = aug,
publisher = {ACM},
pages = {812-821},
doi = {10.1145/3673038.3673070},
url = {https://doi.org/10.1145/3673038.3673070},
} Related notes
- 7 min
When is research software shipped?
A working definition of "shipped" for research software, built around a named state someone else can run, evidence tied to that state, explicit limits, and a continuity signal, with Hermes and DLIO as the worked examples.
- 7 min
We need an Agent500
Agent leaderboards score answers. Science needs a ranking that runs an agent through a real workflow with real tools and data and measures correctness, cost, I/O efficiency, and provenance together.
- 8 min
Reflecting on SC25
What I noticed at SC25 in St. Louis, what the group brought, what a year of running the PDSW program committee taught me about how a workshop gets made, and a few notes for people attending their first SC.