AI systems research / Five-person course collaboration / 2026
Can a cache forget less useful context
and keep the answer?
We explored semantic-aware, chunk-level KV-cache eviction for retrieval-augmented generation. The system could release meaningful cache capacity without changing answer quality at one moderate budget, but the first live result also exposed substantial latency and robustness costs.
01 / The research question
Memory is limited.
Not every context chunk is equally useful.
Long-context language-model serving stores key and value tensors for earlier tokens so the model does not recompute them on every step. That cache improves generation speed, but it also consumes enough memory to limit how many requests a system can serve.
We asked whether the system could rank retrieved context by semantic utility, evict whole low-value chunks, and preserve answer quality more coherently than dropping isolated tokens.
02 / System
Make the unit of memory
match the unit of meaning.
The system treats a retrieved passage as a chunk, computes a utility score, and evicts complete low-utility chunks from the live cache. A shared offline/online feature contract keeps training-time traces and serving-time decisions comparable.
Retrieved context
Keep meaningful passages as whole chunks rather than unrelated token fragments.
Shared features
Compute the same relevance and cache signals in offline experiments and live serving.
Utility score
Rank chunks with a parameterized combination of semantic and system features.
Chunk eviction
Release the lowest-utility cache blocks while preserving the remaining context structure.
03 / Role
My contribution
I led the offline pipeline and chunk-level utility formulation: creating trace data, defining parameterized utility signals, and keeping the shared offline/online feature contract aligned with the live vLLM path.
Offline traces
Prepared the examples and cache-level observations needed to compare eviction choices before live serving.
Utility model
Combined semantic relevance with system signals so whole chunks could be ranked under different memory budgets.
Interface alignment
Kept feature names and meanings consistent between the offline pipeline and online eviction logic.
Built with Aarul Dhawan, Yu Fu, Naman Raina, and Keshav Trikha. The numerical outcome is a team result; this page distinguishes that result from my individual contribution.
04 / Evidence
Capacity returned.
Quality held. Latency rose.
In a live warmup using 50 MuSiQue examples at 30% eviction, the system released 713 cache blocks—an estimated 654 MB—with a 0% quality delta against the unpruned answers.
The serving cost was real
The same comparison showed a 56% latency increase. At a more aggressive 40% eviction budget, answer quality degraded, so the moderate result did not extend safely to every setting.
05 / Friction
An encouraging mechanism.
Not production-ready.
Small live sample. Fifty examples are enough to expose behavior, not enough to establish a reliable quality boundary across tasks.
Latency overhead. Feature extraction and eviction logic added 56% latency in the measured warmup.
Coarse decisions. Whole chunks preserve meaning, but a mixed-quality chunk may contain both essential and disposable tokens.
Budget sensitivity. The 30% setting held quality in this sample; the 40% setting did not.
Estimated memory. The 654 MB figure is derived from released blocks rather than an independent device-memory measurement.
06 / Next action
Find the safe boundary
before optimizing beyond it.
These results come from the team’s final course report. No public report or repository is linked here while the shareable source and contribution boundaries remain under review.
Return to selected work