Bounded memory.
Long-context memory is a geometry problem.
Linear attention removed the quadratic cost of attention and introduced a quieter failure in its place: an internal memory state that grows without bound and takes recall down with it. The fix is not a cheaper update. It is controlling which directions the state is allowed to write to.
Variational Linear Attention: Stable Associative Memory for Long-Context Transformers. Vishal Pandey and Gopal Singh, Metriqual.
- Lower memory state norm
- 110×
- at T = 1,000 tokens
- Recall at 24 stored pairs
- 1.000
- DeltaNet 0.010
- Jacobian spectral norm
- 1
- exactly, at every step
- Fused kernel speedup
- 14×
- over the Python loop
Every write counts the same, so nothing survives.
Linear attention accumulates each key-value pair with equal weight regardless of what the state already holds.
Softmax attention costs quadratic time and linear memory in the sequence length, which puts sequences of a hundred thousand tokens out of reach. Linear attention removes the quadratic term by keeping a running state and adding to it, one outer product per token.
That update is unconditional. Over a sequence, the Frobenius norm of the state grows as O(T), and stored associations start interfering with each other. Retrieval decays as the context gets longer, which is precisely the regime the mechanism was built for.
DeltaNet improves on this with a learned scalar gate that decays the state. A scalar decays every direction at once, so it cannot retire only the directions that were just overwritten. Under memory load that limitation dominates.
The proposal here replaces the scalar with a matrix. The memory update becomes an online regularised least-squares problem with an adaptive penalty, maintained exactly by the Sherman-Morrison rank-1 formula. Directions that were recently written receive smaller updates, so new associations route around the old ones instead of over them.
Bounded state
The state norm is self-limiting under bounded inputs, rather than accumulating every write forever.
S_t = S_t−1 + (v_t − S_t−1 k̂_t) α̂_tᵀ
Unit Jacobian
Normalising both the key and the gating vector gives the recurrence a Jacobian with spectral norm exactly 1 at every step, so gradients neither vanish nor explode with depth.
∂S_t/∂S_t−1 = I − α̂_t k̂_tᵀ
1,630 against 15.
Both numbers are the Frobenius norm of the memory state after a thousand tokens of random input.
The mechanism behind the bound is visible in a second measurement. The penalty matrix's own norm decays from about 56 to about 10 over the same thousand steps. As it accumulates penalty mass, the residual error shrinks, because the state already fits the associations it has been asked to store. A decaying penalty norm is the model actively suppressing redundant writes.
The gate is what keeps old associations alive.
Multi-query associative recall stores n key-value pairs, then asks for all of them back in shuffled order.
| Pairs stored | VLA | DeltaNet | Linear | Softmax |
|---|---|---|---|---|
| 4 | 1.000 | 0.97 | 0.27 | 0.26 |
| 8 | 1.000 | 0.73 | — | — |
| 12 | 1.000 | 0.10 | — | — |
| 24 | 1.000 | 0.010 | 0.08 | 0.07 |
| Sequence length | VLA | DeltaNet | Linear and softmax |
|---|---|---|---|
| 64 | 1.000 | ≈0.01 | ≈0.14–0.15 |
| 128 | 1.000 | ≈0.01 | ≈0.14–0.15 |
| 256 | 1.000 | ≈0.01 | ≈0.14–0.15 |
| 512 | 1.000 | ≈0.01 | ≈0.14–0.15 |
| Variant | At 16 pairs | At 48 pairs | What it tests |
|---|---|---|---|
| VLA, full | 0.990 | 0.044 | the full model |
| Penalty matrix frozen | ≈0.091 | ≈0.043 | the adaptive update |
| No key normalisation | NaN | — | the stability proof |
| DeltaNet | 0.009 | 0.008 | scalar against matrix gate |
| Linear attention | 0.091 | 0.043 | the residual update |
Linear time, with a constant to pay first.
The update is five extra batched matrix-vector operations per token beyond DeltaNet's three.
All three linear-time mechanisms hold their O(T) slope on a log-log plot while softmax bends away from it. In Python, VLA runs about 3× slower than plain linear attention at matched length, and that overhead is an implementation cost rather than an algorithmic one: a Triton kernel that fuses every Sherman-Morrison step into a single launch recovers 14× of it at 4,096 tokens.
Even fused, it stays slower than softmax attention until roughly 43,000 tokens, where the curves cross. Short contexts should use softmax. This mechanism earns its place in the long-context regime and nowhere else, which the paper says outright.
Training setup
- Layers
- 2
- Hidden dim
- 128, 4 heads
- Per-head capacity
- 32
- Optimiser
- AdamW, 3e-4 cosine
- Batch size
- 64
- Training steps
- 3,000
- Seeds
- 42, 123, 999
- Hardware
- 1× NVIDIA T4
What has not been shown yet.
The results are clean and the tasks are synthetic. Both of those are true at once.
Every experiment is an associative recall task, copy and MQAR. They isolate memory dynamics under controlled conditions, which is exactly what makes the state-norm and capacity results legible, and they do not carry the distributional complexity of natural language. There are no perplexity numbers on WikiText-103 or The Pile, no long-document QA, no downstream fine-tuning. Showing that these advantages survive contact with real text is the next piece of work, not a claim being made here.
The capacity limit is real and is not removed by this mechanism. A per-head state holds at most as many independent associations as it has dimensions, 32 in these experiments. Past that, new associations overwrite old ones. VLA degrades more gradually than additive linear attention in the overload regime because the penalty matrix preferentially overwrites recently written directions, but 0.044 at 48 pairs is a gentler failure, not a solution. More capacity costs more memory.
The claim is about geometry, not cost.
Controlling which directions the state may update, rather than only making the update cheaper, is what decides whether a stored association survives a long sequence. That framing connects recurrent attention back to recursive least squares, which is where the stability theory comes from.