Beyond Perplexity: The Quality–Memory Frontier of Native Mixed-Precision KV Caching in llama.cpp

<jats:p>The key–value (KV) cache is a growing memory bottleneck for long-context autoregressive inference, but low-bit cache policies are frequently selected from perplexity alone. We implement native packed mixed-precision KV caching in llama.cpp, including Q3 formats, a four-level non-uniform Q2 value codebook, direct quantized cache writes, and mixed-format CUDA FlashAttention. A direct key-precision ablation on Qwen3-4B shows that Q3 keys increase held-out perplexity by approximately 20% and Q2 keys are catastrophic, whereas Q4 and Q8 keys have similar likelihood on the tested subsets; a 200-example GSM8K pilot nevertheless favors Q8 keys by 13 points, exposing a second likelihood–behavior mismatch. For values, the conservative Q4K-Q3V policy reduces KV storage by 13.89% and is statistically indistinguishable from Q4K-Q4V on full GSM8K, HumanEval, and MBPP. The aggressive Q4K-MixV-21 policy reduces KV storage by 20.99% and slightly lowers held-out perplexity, yet its strict GSM8K accuracy falls by 2.81 points (paired 95% CI \(\lbrack{- 5.53},{- 0.08}\rbrack\)). At 32K populated context, it saves 272 MiB of process VRAM. Occupancy-verified RULER experiments from 8K to 32K show task- and context-dependent changes rather than monotonic degradation. Offline sensitivity analysis further shows that the fixed NU-Q2 codebook is within 1.1% of the deployment-data MSE optimum and reduces reconstruction MSE by 73.7% relative to uniform Q2. These results support a two-point deployment frontier and demonstrate that perplexity-preserving KV compression is not necessarily behavior-preserving.</jats:p>