LCGA: what a negative result taught me about learned block-sparse attention

LCGA started with a simple question: can a learned router skip most of the attention blocks and still answer questions that depend on distant context?

I trained it on a frozen Qwen2.5-0.5B, on one RTX 3050 Laptop with 6 GB of VRAM. The router selects blocks at a 25% budget (64-token blocks). The base model stays frozen; only the router is trained.

The short answer: on plain language modelling, the learned router mostly learns a position prior. On retrieval-style tasks, content selection is learnable. The difference between those two regimes is the interesting result.

What I expected

Block-sparse attention usually sounds like a compute win: keep the sink and the recent window, then learn which distant blocks matter. If the router works, long context becomes cheaper without losing quality.

The first version looked great. Then I found two measurement errors:

  1. The evaluation loaded the checkpoint before patching the router modules, so it was evaluating a freshly initialised router, not the trained one.
  2. The “local” baseline was given fewer blocks than the learned router. Against a budget-matched window, the learned router was not better.

Both mistakes produced plausible numbers. Both are documented in the archive with the invalid runs kept.

The negative result

After fixing the loader, the trained router selects only a small amount of content:

context sink recent window content
1k 40.0% 58.3% 1.7%
4k 12.2% 81.8% 6.0%
8k 6.2% 86.8% 7.0%
16k 3.1% 90.2% 6.7%

About 90% of the selection is a position prior. The learned part is not zero, but it is small.

Quality against a budget-matched static window at a 25% block budget:

context dense learned static window
1k 17.323 25.166 25.051
4k 10.950 13.788 13.783
8k 10.375 12.176 12.160
16k 10.799 12.001 11.868

The learned router is within +0.005 to +0.133 PPL of the window — slightly worse, not better. The gap to dense does fall from +45% at 1k to +11% at 16k, but that is the budget helping as the window covers more of the sequence, not the router learning content.

Why the plain-LM result is not a training failure

On WikiText, the dense teacher itself puts almost no attention mass on distant blocks. If the teacher does not look far away, the student has nothing to distil. Perplexity cannot reward content selection that the teacher never uses.

This is the part I had to accept: the negative result is the correct behavior for that data and that objective, not a bug to tune away.

Where distant content actually matters

The picture changes on a needle-in-a-haystack probe. A unique four-digit code is planted at a known depth in 8k tokens, followed by: “What is the secret code? Answer:”.

At a 25% budget:

fact depth dense budget-matched window
0.1 1.00 exact 0.00
0.5 1.00 exact 0.00
0.9 (inside window) 1.00 exact 1.00

A static window can only answer when the fact happens to fall inside the window. Dense attention can answer at every depth. This is the regime where a content-selecting router can beat the static baseline, and where perplexity evaluation can never show it.

What made content selection work

The original objective — full KL over the teacher’s attention distribution — failed. At the answer rows, roughly half the dense mass sits on the sink block and a large fraction on the query block, so the loss is dominated by structure the position prior already provides.

Three changes mattered:

  • Train at the answer positions, not all query rows.
  • Remove the blocks the position prior already covers (sink, self, recent window).
  • Distil the teacher’s content argmax rather than the full distribution.

The label-free content-argmax router matched dense exact match in every out-of-window cell at a 25% budget. A shuffled-label control stayed at chance, so the effect came from the target, not the training loop.

A mixed schedule of one needle step per seven LM steps kept exact match at 1.00 in all five probe cells across three seeds (15/15), including the in-window case that a needle-only router loses. The cost was +0.60 PPL at 1k and +0.37 at 4k above the budget-matched window.

The speed result

The current sparse path gives:

  • 1.71× prefill speed at 32k
  • 1.27× graph-decode speed at 8k
  • eager decode loses

This is a compute/bandwidth result, not a KV-memory win. The sparse path does not shrink the KV cache enough to change the memory story on this card.

What still does not work

  • The 1k-trained router transfers to 8k, but not to 2k or 16k.
  • A held-out phrasing of the fact scores 0.00 exact, even when the router still selects fact-like content.
  • With multiple facts, the router selects fact-like blocks rather than the block that answers the specific question.
  • The full ablation grid is one base model (Qwen2.5-0.5B); the label-free recipe is replicated on Qwen2.5-3B, but the 1.5B host is not usable because its dense attention does not retrieve the needle at 1k.
  • The result is a research archive, not a production attention kernel.

What I would tell someone starting this

  1. Check what your evaluation actually loaded. A fresh router can look like a trained one if you patch the model after loading the checkpoint.
  2. Budget-match the baseline. A learned router with 4–16 blocks versus a “local” baseline with 2 blocks is not a comparison.
  3. Do not judge sparse attention on perplexity alone. If the task never needs distant content, the router will learn the position prior and look useless. If the task needs distant content, a static window will fail and perplexity may not show it.
  4. Publish the negative result. It saves the next person from repeating the same two mistakes.

The full archive, every raw JSON, and the invalid runs are here:

LCGA is not a claim that learned sparse attention is useless. It is a claim that, on plain language modelling at this scale, a learned router does not beat a budget-matched window — and on retrieval, the right objective makes content selection work.