KV Cache | Learn AI

What is KV Cache?

During autoregressive generation, a transformer must compute attention over all previous tokens for every new token it generates. The KV Cache stores the Key and Value projections from previous tokens so they don't need to be recomputed. This turns generation from O(n²) to O(n) in computation per step — a massive speedup for long sequences.

Key Cache

Stores the Key projections for each token at each layer. These are used to compute attention scores between the new token and all previous tokens.

Value Cache

Stores the Value projections for each token at each layer. Once attention weights are computed, these cached values are used to produce the output.

Why It Matters

Caching avoids recomputing past keys and values during decoding. The savings grow with sequence length, while the cache itself consumes memory.

⚡

Speed

Avoids recomputing attention for all previous tokens at every step. Generation goes from quadratic to linear.

🔁

Incremental

Each new token only needs to compute its own Q, K, V and attend to the cached K, V from previous positions.

📉

Trade-off

Trades GPU memory for compute time. The cache grows linearly with sequence length and model depth.

🔬

Interactive KV Cache Explorer

Watch the cache grow token by token

Step through autoregressive generation to see how the KV cache accumulates. Compare the computation cost with and without caching — the savings become dramatic as sequences get longer.

These counters count token-wise K/V projection pairs, not all attention operations. Each new query still scores the cached keys. The sample vectors are illustrative.

Token	K	V

Without cache: 0 K/V projections

With cache: 0 K/V projections

KV memory versus context

2 × 32 × 8 × 128 × 8192 × 2 bytesContext tokens: 8192

Memory (GiB): 1.00

Worked example: 32 Layers (n_layers), 128 Head dimension, FP16.

Memory Implications

The KV cache is the primary memory bottleneck during inference. For large models with long contexts, it can consume tens of gigabytes of GPU memory.

KV Cache Memory = 2 × num_layers × seq_len × d_head × num_kv_heads × dtype_size

Standard FP16 KV tensors: 2 bytes per value. For 80 layers, 8 KV heads and head dimension 128, 8,192 tokens require 2.5 GiB per sequence; 131,072 tokens require 40 GiB. Runtime allocation and other buffers are additional.

Optimization Techniques

Multi-Query & Grouped-Query Attention (MQA/GQA)

Instead of separate K/V heads per attention head, MQA shares a single K/V head across all query heads, while GQA uses a few shared groups. This reduces KV cache size by 4-32× with minimal quality loss. Llama 3 and Mistral use GQA. See the Attention Mechanism article for more details.

Sliding Window Attention

Instead of caching all tokens, only keep the most recent W tokens in the cache. Used by Mistral models. Reduces memory from O(seq_len) to O(W), but limits the model's ability to attend to very early tokens.

Paged Attention (vLLM)

Inspired by virtual memory in operating systems. Instead of allocating contiguous memory for each sequence's KV cache, vLLM manages cache in fixed-size "pages" that can be allocated and freed dynamically. This eliminates memory fragmentation and enables efficient batching of requests with different sequence lengths.

TurboQuant, PolarQuant & QJL (Google Research, 2025)

TurboQuant studies low-bit vector compression for inference. Its quantizers and error estimators aim to preserve task quality while reducing storage. Similar benchmark results do not mean that each compressed vector can be reconstructed exactly.

PolarQuant — the core compressor

The paper uses transformations and scalar quantization to control distortion. The small worked example below uses a simpler uniform quantizer so its rounding and scale overhead remain inspectable.

QJL — the 1-bit error corrector

An unbiased estimator can be correct in expectation while individual estimates still have error. QJL is not a lossless repair code for every original vector.

Why it matters for inference

For the standard 80-layer, 8-KV-head, 128-dimension configuration, a 128K FP16 cache occupies 40 GiB per sequence. Whether compression preserves task quality depends on the quantizer, model and evaluation. Consult the paper for its measured configurations.

🔬

What rotation and quantization actually change

An orthogonal transform, a real rounding operation and a visible reconstruction error

Rotation and lossy quantization

Computed toy example, not a TurboQuant implementation. An orthogonal rotation preserves the norm. A uniform scalar quantizer then rounds values; its inverse does not recover the original exactly. Change angle and precision to see the error.

Rotation angle: 30°Bits per coordinate: 4

Worked example	Squared norm
Original `0.8200, -0.4100, 0.6700, -0.2300`	1.342300
Rotation angle `0.9151, 0.0549, 0.6952, 0.1358`	1.342300
Reconstructed `0.8230, -0.4047, 0.6727, -0.1770`	1.325084

Four FP16 values: 64 bits

Quantized storage: 48 bits (4 × 4 + 32)

Scale (FP32): 0.9151

Error (MSE): 7.121e-4

64 / 48 = 1.33×

This tiny block includes one 32-bit scale. Compression can be less than 1×. TurboQuant uses different quantizers and error-estimation techniques; its benchmark quality is not a claim of lossless reconstruction.

TurboQuant (2025)

Key Takeaways

1KV cache stores Key and Value projections from previous tokens, avoiding redundant computation during generation
2Caching reuses past K/V projections. Each new query still attends to the cached keys; per-step attention cost grows with the attended context.
3KV cache memory grows linearly with sequence length × layers × KV heads — this is the main inference memory bottleneck
4Techniques like GQA, sliding window, and paged attention address the memory cost while preserving generation speed