akougkas.io

PKAS: Predictive KVCache-Aware Scheduling for Faster LLM and Transformer Inferences

Jie Ye, A. Maurya, K. Chitty-Venkata, Bogdan Nicolae, Anthony Kougkas, Xian-He Sun

35th International Symposium on High-Performance Parallel and Distributed Computing (HPDC 2026)ACMPublished

Abstract

With rising popularity of LLMs, the performance, scalability, and resource-efficiency of inferences become a crucial challenge. The core part of the inference process is the KV cache, which avoids recomputing intermediate attention states, and the batching strategy that batches multiple requests per forward pass to leverage GPU parallelism. KV cache memory grows linearly with sequence length and batch sizes, easily exceeding the limited GPU memory capacity. State-of-the-art inference runtimes use continuous batching to maximize GPU utilization by interleaving the processing of new requests (i.e., prefill requests) with ongoing generation requests (i.e., decode requests). However, existing schedulers greedily admit prefill requests without considering the future KV cache memory required to successfully run the decode phases. This shortsighted approach causes frequent KV cache overflows, which in turn trigger preemption and recomputation of requests, severely degrading both throughput and latency. We propose PKAS, a Predictive KV Cache-Aware Scheduling algorithm to mitigate this inefficiency by reducing preemptions. PKAS uses a low-overhead technique to simulate future KV cache utilization and guide the admissibility for new request candidates. Combined with lightweight output-length predictions, PKAS can make better batching decisions, preventing KV cache overflows and drastically reducing preemptions. Evaluations on diverse models and workloads show that PKAS achieves up to 7.34x higher throughput and 8x lower latency compared to state-of-the-art scheduling, with the largest gains on long-context workloads where KV cache pressure is high.

Finding

By predicting future KV cache use when admitting new requests, PKAS prevents KV cache overflows and reduces preemptions, achieving up to 7.34x higher throughput and 8x lower latency than state-of-the-art scheduling, with the largest gains on long-context workloads.

System

  1. 2020–present
    active

    DeepIO

    A line of subcontracts with Argonne, Pacific Northwest, and Lawrence Livermore that funds students working where I/O meets AI, from Viper and KV-cache scheduling for LLM inference to WisIO's I/O bottleneck analysis.

Browse all papers on DeepIO
  1. 2026
    poster

    Extending I/O Observability to Explainability for Targeted Optimization

    Neeraj Rajesh, S. Neuwirth, Jay Lofstead, Anthony Kougkas, Xian-He Sun, Nikoli Dryden et al. International Conference for High Performance Computing, Networking, Storage, and Analysis (SC 2026)

  2. 2026
    poster

    Pre-RoPE versus Post-RoPE: Trading Per-Step Compute for KVCache Survivability

    Zia Uddin Chowdhury, Jaime Cernuda, Jie Ye, Anthony Kougkas, Xian-He Sun. 13th Greater Chicago Area Systems Research Workshop (GCASR 2026)

  3. 2026
    conference

    PagedEviction: Structured Block-wise KV Cache Pruning for Efficient Large Language Model Inference

    Krishna Teja Chitty-Venkata, Jie Ye, Siddhisanket Raskar, Xian-He Sun, Anthony Kougkas, Murali Emani et al. Findings of the Association for Computational Linguistics (EACL 2026)

  4. 2025
    poster

    Insights into GPUDirect Data Transfer through NIXL Benchmarking

    Isa Muradli, Jie Ye, Luke Logan, Anthony Kougkas, Xian-He Sun. 21st IEEE International Conference on eScience (eScience ’25), September 15-18, 2025

Citation

@inproceedings{ye2026pkas,
  author = {Ye, J. and Maurya, A. and Chitty-Venkata, K. and Nicolae, B. and Kougkas, A. and Sun, X.-H.},
  title = {{PKAS: Predictive KVCache-Aware Scheduling for Faster LLM and Transformer Inferences}},
  booktitle = {35th International Symposium on High-Performance Parallel and Distributed Computing (HPDC 2026)},
  year = {2026},
  month = jul,
  publisher = {ACM},
  pages = {402--414},
  doi = {10.1145/3806645.3807819},
  url = {https://doi.org/10.1145/3806645.3807819},
}

Related notes

  1. essay

    When is research software shipped?

    A working definition of "shipped" for research software, built around a named state someone else can run, evidence tied to that state, explicit limits, and a continuity signal, with Hermes and DLIO as the worked examples.

    7 min
  2. essay

    We need an Agent500

    Agent leaderboards score answers. Science needs a ranking that runs an agent through a real workflow with real tools and data and measures correctness, cost, I/O efficiency, and provenance together.

    7 min
  3. field note

    Reflecting on SC25

    What I noticed at SC25 in St. Louis, what the group brought, what a year of running the PDSW program committee taught me about how a workshop gets made, and a few notes for people attending their first SC.

    8 min