akougkas.io

PagedEviction: Structured Block-wise KV Cache Pruning for Efficient Large Language Model Inference

Krishna Teja Chitty-Venkata, Jie Ye, Siddhisanket Raskar, Xian-He Sun, Anthony Kougkas, Murali Emani, Venkatram Vishwanath, Bogdan Nicolae

Findings of the Association for Computational Linguistics (EACL 2026)Association for Computational LinguisticsPublished

Abstract

KV caching significantly improves the efficiency of Large Language Model (LLM) inference by storing attention states from previously processed tokens, enabling faster generation of subsequent tokens. However, as sequence length increases, the KV cache quickly becomes a major memory bottleneck. To address this, we propose PagedEviction, a novel fine-grained, structured KV cache pruning strategy that enhances the memory efficiency of vLLM’s PagedAttention. Unlike existing approaches that rely on attention-based token importance or evict tokens across different vLLM pages, PagedEviction introduces an efficient block-wise eviction algorithm tailored for paged memory layouts. Our method integrates seamlessly with PagedAttention without requiring any modifications to its CUDA attention kernels. We evaluate PagedEviction across Llama-3.1-8B-Instruct, Llama-3.2-1B-Instruct, and Llama-3.2-3B-Instruct models on the LongBench benchmark suite, demonstrating improved memory usage with better accuracy than baselines on long context tasks.

Finding

PagedEviction stays within 0.5-1.5 ROUGE points of full-cache performance at a 1024-token budget while improving throughput by up to 3.1x over Full Cache, and across Llama 1B, 3B, and 8B models it yields 10-12% lower latency and 15-20% better accuracy than methods such as StreamingLLM.

System

  1. 2024–present
    active

    IOWarp

  2. 2020–present
    active

    DeepIO

Browse all papers on IOWarp
  1. 2026
    conference

    Trigger, Render, Reason: Multimodal LLM Agents for Science-Aware Scientific Simulation Monitoring

    Hua Xu, Jaime Cernuda, Ana Gainaru, Scott Klasky, Anthony Kougkas, Xian-He Sun. International Conference for High Performance Computing, Networking, Storage, and Analysis (SC 2026)

  2. 2026
    workshop

    GPUH5: A GPU-first I/O Library for Scientific Data

    J. Fernandes, Luke Logan, Xian-He Sun, Anthony Kougkas. 11th International Parallel Data Systems Workshop (PDSW 2026), held in conjunction with SC26

  3. 2026
    workshop

    Trigger-Gated Multimodal LLM Agents for In-situ Scientific Simulation Monitoring

    Hua Xu, Jaime Cernuda, Anthony Kougkas, Ana Gainaru, Scott Klasky, Xian-He Sun et al. 11th International Parallel Data Systems Workshop (PDSW 2026), held in conjunction with SC26

  4. 2026
    workshop

    Type-Safe, Dynamically Defined Metadata Support for Scientific Applications

    J. Teves, C. Brinker, Anthony Kougkas, Ana Gainaru, Scott Klasky, Xian-He Sun et al. 11th International Parallel Data Systems Workshop (PDSW 2026), held in conjunction with SC26

Citation

@inproceedings{chittyvenkata2026pagedeviction,
  author = {Chitty-Venkata, Krishna Teja and Ye, Jie and Raskar, Siddhisanket and Sun, Xian-He and Kougkas, Anthony and Emani, Murali and Vishwanath, Venkatram and Nicolae, Bogdan},
  title = {{PagedEviction: Structured Block-wise KV Cache Pruning for Efficient Large Language Model Inference}},
  booktitle = {Findings of the Association for Computational Linguistics (EACL 2026)},
  year = {2026},
  month = mar,
  publisher = {Association for Computational Linguistics},
  pages = {3207--3218},
  doi = {10.18653/v1/2026.findings-eacl.168},
  url = {https://doi.org/10.18653/v1/2026.findings-eacl.168},
}

Related notes

  1. essay

    One destination, N tiers

    Every tier supplies the application at once, and the application sees one destination. The thesis from a November 2026 talk to my group at the Gnosis Research Center, the six lines of I/O work behind it since 2016, and my bet that they are the foundation agents need for context.

    9 min
  2. field note

    SC26 in Chicago

    What the group brings to SC26 in Chicago, the pressure from GPUs and agent fleets that runs through all of it, and an invitation to Thursday's Birds of a Feather on HDF5 in the age of AI-driven science.

    4 min
  3. essay

    CLIO as the entry point

    CLIO should become the single place a developer starts with IOWarp, built as an application SDK with deployment profiles rather than an umbrella package or a gateway service. This is the direction for 2027; the API and the scope are still open.

    6 min