PKAS: Predictive KVCache-Aware Scheduling for Faster LLM and Transformer Inferences
Abstract
With rising popularity of LLMs, the performance, scalability, and resource-efficiency of inferences become a crucial challenge. The core part of the inference process is the KV cache, which avoids recomputing intermediate attention states, and the batching strategy that batches multiple requests per forward pass to leverage GPU parallelism. KV cache memory grows linearly with sequence length and batch sizes, easily exceeding the limited GPU memory capacity. State-of-the-art inference runtimes use continuous batching to maximize GPU utilization by interleaving the processing of new requests (i.e., prefill requests) with ongoing generation requests (i.e., decode requests). However, existing schedulers greedily admit prefill requests without considering the future KV cache memory required to successfully run the decode phases. This shortsighted approach causes frequent KV cache overflows, which in turn trigger preemption and recomputation of requests, severely degrading both throughput and latency. We propose PKAS, a Predictive KV Cache-Aware Scheduling algorithm to mitigate this inefficiency by reducing preemptions. PKAS uses a low-overhead technique to simulate future KV cache utilization and guide the admissibility for new request candidates. Combined with lightweight output-length predictions, PKAS can make better batching decisions, preventing KV cache overflows and drastically reducing preemptions. Evaluations on diverse models and workloads show that PKAS achieves up to 7.34x higher throughput and 8x lower latency compared to state-of-the-art scheduling, with the largest gains on long-context workloads where KV cache pressure is high.
Finding
By predicting future KV cache use when admitting new requests, PKAS prevents KV cache overflows and reduces preemptions, achieving up to 7.34x higher throughput and 8x lower latency than state-of-the-art scheduling, with the largest gains on long-context workloads.
System
- 2020–present
DeepIO
A line of subcontracts with Argonne, Pacific Northwest, and Lawrence Livermore that funds students working where I/O meets AI, from Viper and KV-cache scheduling for LLM inference to WisIO's I/O bottleneck analysis.
More papers on DeepIO
Browse all papers on DeepIO- 2026
Extending I/O Observability to Explainability for Targeted Optimization
Neeraj Rajesh, S. Neuwirth, Jay Lofstead, Anthony Kougkas, Xian-He Sun, Nikoli Dryden et al. International Conference for High Performance Computing, Networking, Storage, and Analysis (SC 2026)
- 2026
Pre-RoPE versus Post-RoPE: Trading Per-Step Compute for KVCache Survivability
Zia Uddin Chowdhury, Jaime Cernuda, Jie Ye, Anthony Kougkas, Xian-He Sun. 13th Greater Chicago Area Systems Research Workshop (GCASR 2026)
- 2026
PagedEviction: Structured Block-wise KV Cache Pruning for Efficient Large Language Model Inference
Krishna Teja Chitty-Venkata, Jie Ye, Siddhisanket Raskar, Xian-He Sun, Anthony Kougkas, Murali Emani et al. Findings of the Association for Computational Linguistics (EACL 2026)
- 2025
Insights into GPUDirect Data Transfer through NIXL Benchmarking
Isa Muradli, Jie Ye, Luke Logan, Anthony Kougkas, Xian-He Sun. 21st IEEE International Conference on eScience (eScience ’25), September 15-18, 2025
Citation
@inproceedings{ye2026pkas,
author = {Ye, J. and Maurya, A. and Chitty-Venkata, K. and Nicolae, B. and Kougkas, A. and Sun, X.-H.},
title = {{PKAS: Predictive KVCache-Aware Scheduling for Faster LLM and Transformer Inferences}},
booktitle = {35th International Symposium on High-Performance Parallel and Distributed Computing (HPDC 2026)},
year = {2026},
month = jul,
publisher = {ACM},
pages = {402--414},
doi = {10.1145/3806645.3807819},
url = {https://doi.org/10.1145/3806645.3807819},
} Related notes
- 7 min
When is research software shipped?
A working definition of "shipped" for research software, built around a named state someone else can run, evidence tied to that state, explicit limits, and a continuity signal, with Hermes and DLIO as the worked examples.
- 7 min
We need an Agent500
Agent leaderboards score answers. Science needs a ranking that runs an agent through a real workflow with real tools and data and measures correctness, cost, I/O efficiency, and provenance together.
- 8 min
Reflecting on SC25
What I noticed at SC25 in St. Louis, what the group brought, what a year of running the PDSW program committee taught me about how a workshop gets made, and a few notes for people attending their first SC.