Characterizing the Behavior and Impact of KV Caching on Transformer Inferences under Concurrency
Abstract
Pre-training of LLMs and transformers is known to take weeks if not months even with powerful HPC systems. However, inferences are an equally important problem: once pre-trained, the model needs to serve a large number of inferences submitted under concurrency by multiple users. Thus, speeding up each inference request is instrumental in achieving high throughput and latency at scale. To avoid redundant recomputation in each decode iteration, a Key-Value (KV) cache is used to store previously computed keys (K) and values (V), speeding up token generation. GPU memory is primarily consumed by model weights and the remainder is used by the KV cache. Thus, the free GPU space available to the KV cache is a scarce resource that needs to be managed in an efficient way in order to minimize the overhead of redundant recomputations. There are many optimizations applied in this context: batching of inference requests to enable them to run in the same forward pass (and thus increase the parallelism and inference throughput), different KV cache eviction policies (simply drop KV entries and recompute them later vs. swap to host memory), etc. Under these circumstances, the decision of what batching strategy, what KV cache eviction policy to apply and how the KV cache impacts the inference performance is non-trivial. Unlike the case of pre-training, state-of-art studies are scarce in this context. To fill this gap, in this paper we study the impact of KV caching. Specifically, we instrument vLLM to measure and analyze fine-grain metrics (token throughput, KV cache memory access patterns, load balancing of the forward passes), during different inference stages (prefill, decode) in several scenarios that involve concurrent inference requests using several benchmarks. Based on the measurements and associated observations, we identify several opportunities to improve the design of inference frameworks.
Finding
Instrumenting vLLM under concurrent inference shows trade-offs between chunked batching and forward-pass overhead, significant KV cache idle durations, and swapping versus recomputation, which point to co-designing batching and multi-level KV cache strategies.
System
- 2020–present
DeepIO
A line of subcontracts with Argonne, Pacific Northwest, and Lawrence Livermore that funds students working where I/O meets AI, from Viper and KV-cache scheduling for LLM inference to WisIO's I/O bottleneck analysis.
More papers on DeepIO
Browse all papers on DeepIO- 2026
Extending I/O Observability to Explainability for Targeted Optimization
Neeraj Rajesh, S. Neuwirth, Jay Lofstead, Anthony Kougkas, Xian-He Sun, Nikoli Dryden et al. International Conference for High Performance Computing, Networking, Storage, and Analysis (SC 2026)
- 2026
PKAS: Predictive KVCache-Aware Scheduling for Faster LLM and Transformer Inferences
Jie Ye, A. Maurya, K. Chitty-Venkata, Bogdan Nicolae, Anthony Kougkas, Xian-He Sun. 35th International Symposium on High-Performance Parallel and Distributed Computing (HPDC 2026)
- 2026
Pre-RoPE versus Post-RoPE: Trading Per-Step Compute for KVCache Survivability
Zia Uddin Chowdhury, Jaime Cernuda, Jie Ye, Anthony Kougkas, Xian-He Sun. 13th Greater Chicago Area Systems Research Workshop (GCASR 2026)
- 2026
PagedEviction: Structured Block-wise KV Cache Pruning for Efficient Large Language Model Inference
Krishna Teja Chitty-Venkata, Jie Ye, Siddhisanket Raskar, Xian-He Sun, Anthony Kougkas, Murali Emani et al. Findings of the Association for Computational Linguistics (EACL 2026)
Citation
@inproceedings{ye2025unboxkv-io,
author = {Ye, Jie and Cernuda, Jaime and Maurya, Avinash and Sun, Xian-He and Kougkas, Anthony and Nicolae, Bogdan},
booktitle = {IPDPS'25: The 39th IEEE International Parallel and Distributed Processing Symposium},
title = {Characterizing the Behavior and Impact of KV Caching on Transformer Inferences under Concurrency},
year = {2025},
month = jun,
day = {3},
publisher = {IEEE},
pages = {1191--1202},
doi = {10.1109/IPDPS64566.2025.00108},
url = {https://doi.org/10.1109/IPDPS64566.2025.00108},
} Related notes
- 9 min
One destination, N tiers
Every tier supplies the application at once, and the application sees one destination. The thesis from a November 2026 talk to my group at the Gnosis Research Center, the six lines of I/O work behind it since 2016, and my bet that they are the foundation agents need for context.
- 8 min
High-performance agentic computing
The case for naming a discipline at the meeting point of HPC, autonomous AI, and domain science, with the multiplicative axiom behind it and my own doubt about whether it is a field or a decade-long transition.
- 7 min
2025 reflections
A personal record of 2025: a promotion, a keynote, two dissertations, a summer of teaching, a loss, and the argument I kept making all year about what a researcher is for now.