PagedEviction: Structured Block-wise KV Cache Pruning for Efficient Large Language Model Inference
Abstract
KV caching significantly improves the efficiency of Large Language Model (LLM) inference by storing attention states from previously processed tokens, enabling faster generation of subsequent tokens. However, as sequence length increases, the KV cache quickly becomes a major memory bottleneck. To address this, we propose PagedEviction, a novel fine-grained, structured KV cache pruning strategy that enhances the memory efficiency of vLLM’s PagedAttention. Unlike existing approaches that rely on attention-based token importance or evict tokens across different vLLM pages, PagedEviction introduces an efficient block-wise eviction algorithm tailored for paged memory layouts. Our method integrates seamlessly with PagedAttention without requiring any modifications to its CUDA attention kernels. We evaluate PagedEviction across Llama-3.1-8B-Instruct, Llama-3.2-1B-Instruct, and Llama-3.2-3B-Instruct models on the LongBench benchmark suite, demonstrating improved memory usage with better accuracy than baselines on long context tasks.
Finding
PagedEviction stays within 0.5-1.5 ROUGE points of full-cache performance at a 1024-token budget while improving throughput by up to 3.1x over Full Cache, and across Llama 1B, 3B, and 8B models it yields 10-12% lower latency and 15-20% better accuracy than methods such as StreamingLLM.
System
- 2024–present
- 2020–present
More papers on IOWarp
Browse all papers on IOWarp- 2026
Trigger, Render, Reason: Multimodal LLM Agents for Science-Aware Scientific Simulation Monitoring
Hua Xu, Jaime Cernuda, Ana Gainaru, Scott Klasky, Anthony Kougkas, Xian-He Sun. International Conference for High Performance Computing, Networking, Storage, and Analysis (SC 2026)
- 2026
GPUH5: A GPU-first I/O Library for Scientific Data
J. Fernandes, Luke Logan, Xian-He Sun, Anthony Kougkas. 11th International Parallel Data Systems Workshop (PDSW 2026), held in conjunction with SC26
- 2026
Trigger-Gated Multimodal LLM Agents for In-situ Scientific Simulation Monitoring
Hua Xu, Jaime Cernuda, Anthony Kougkas, Ana Gainaru, Scott Klasky, Xian-He Sun et al. 11th International Parallel Data Systems Workshop (PDSW 2026), held in conjunction with SC26
- 2026
Type-Safe, Dynamically Defined Metadata Support for Scientific Applications
J. Teves, C. Brinker, Anthony Kougkas, Ana Gainaru, Scott Klasky, Xian-He Sun et al. 11th International Parallel Data Systems Workshop (PDSW 2026), held in conjunction with SC26
Citation
@inproceedings{chittyvenkata2026pagedeviction,
author = {Chitty-Venkata, Krishna Teja and Ye, Jie and Raskar, Siddhisanket and Sun, Xian-He and Kougkas, Anthony and Emani, Murali and Vishwanath, Venkatram and Nicolae, Bogdan},
title = {{PagedEviction: Structured Block-wise KV Cache Pruning for Efficient Large Language Model Inference}},
booktitle = {Findings of the Association for Computational Linguistics (EACL 2026)},
year = {2026},
month = mar,
publisher = {Association for Computational Linguistics},
pages = {3207--3218},
doi = {10.18653/v1/2026.findings-eacl.168},
url = {https://doi.org/10.18653/v1/2026.findings-eacl.168},
} Related notes
- 9 min
One destination, N tiers
Every tier supplies the application at once, and the application sees one destination. The thesis from a November 2026 talk to my group at the Gnosis Research Center, the six lines of I/O work behind it since 2016, and my bet that they are the foundation agents need for context.
- 4 min
SC26 in Chicago
What the group brings to SC26 in Chicago, the pressure from GPUs and agent fleets that runs through all of it, and an invitation to Thursday's Birds of a Feather on HDF5 in the age of AI-driven science.
- 6 min
CLIO as the entry point
CLIO should become the single place a developer starts with IOWarp, built as an application SDK with deployment profiles rather than an umbrella package or a gateway service. This is the direction for 2027; the API and the scope are still open.