GPT-6 Astra got the raw weights of Qwen3.5-0.8B and a Python kernel with no GPU. It changed 131,072 weights, 0.016% of the model, using no gradients, no optimizer and no fine-tuning data. MATH-500 went from 7.9% to 48.1%.
The thread passed 2,600 points on Hacker News. The headline leaves out that in the 7.9% baseline, 93% of the model's MATH-500 answers ran out of tokens before reaching an answer. What Astra actually did was stop a 0.8B model from looping, which is a long way from teaching it math. One HN commenter cites 77.6% on the same test for Qwen's own small reasoning model.
I'll go through what the edit changed and why the baseline is the weak spot, then show how to run the edited Qwen3.5-0.8B with transformers, vLLM, llama.cpp or Ollama so you can check it yourself.
What "weight-space programming" means
The write-up is Sparks of Weight-Space Programming by Rikiya Takehi, Jayden Teoh, Alex Zhang and Omar Khattab at MIT CSAIL, posted November 2, 2026. A general-purpose LLM reads and writes another network's weights directly, the way you'd read and patch source code.
Qwen is experiment 8 of 8. I found the earlier ones more surprising:
- Astra named the game (Breakout, Pong, Pac-Man, Freeway) each of four 17,253-weight PPO policies was trained on, from the weights alone.
- It fixed a Kinetix car-on-ramp policy that had never succeeded in 256 episodes. One edit changed 3.5% of the weights, and the car succeeded in all 1,024 evaluation episodes.
- Starting from all-zero weights, it wrote a MinAtar Breakout policy that scored 517.4. PPO scored 25.3 after 500M steps.
Only after those did they point it at a real 24-layer language model.
Qwen3.5-0.8B MATH-500, GSM8K and IFEval results
Setup: thinking on, up to 32,768 tokens, temperature 0.6, top-p 0.95, top-k 20, presence penalty 1.5. Astra could practice on 7,469 GSM8K and 7,275 MATH training problems (no test sets) and got a budget of 10, 25 or 50 forward passes. One "forward pass" means the model answers up to 512 problems or reads up to 1,024 texts.
| Model | MATH-500 | GSM8K | IFEval | MATH-500 answers hitting token limit |
|---|---|---|---|---|
| Base | 7.9% | 13.8% | 21.0% | 93.0% |
| Edited, 10 passes | 12.3% | 25.7% | 49.0% | 87.2% |
| Edited, 25 passes | 14.1% | 25.1% | 39.9% | 86.4% |
| Edited, 50 passes | 48.1% | 67.1% | 62.3% | 33.1% |
Read the last column first. With thinking on, the base model almost never finishes: 85 of its first 96 practice answers hit the 32k ceiling. Astra traced the looping to one direction written by the MLPs of layers 15 and 19 and subtracted it from 64 output columns in each layer. Once the model stops looping, it reaches the end of its reasoning and writes down answers it could already get.
The scores track the loops. At 10 and 25 passes the authors say "the loops barely went away," and MATH-500 stayed in the low teens. The whole headline comes from the 50-pass run.
Why the 7.9% baseline is the weak spot
Gemma shows the edit only fixes a bug that's already there. The team ran the same procedure on Gemma 3 270M, which hits its token limit on only 8.5% of answers. MATH-500 went 6.8% → 7.0% → 7.6% → 7.7% → 7.0% across 5 to 50 passes, which is noise. The authors' conclusion: editing can remove an obstacle, but it can't add an ability the model doesn't have.
Qwen also documents the bug itself. The Qwen3.5-0.8B model card says the 0.8B "is more prone to entering thinking loops compared to other Qwen3.5 models" and that it "operates in non-thinking mode by default." So the baseline is the model in its least reliable mode. On HN, pama wrote that "the nonthinking qwen" model scored 55.2% on MATH-500, above the edited 48.1%, and that the original Qwen reasoning model got 77.6%. They wondered whether the 7.9% was "a vllm or setup bug." Their comment says "0.6B", so it's not clear they measured this exact checkpoint. kristopolous said that turning thinking off on the base Qwen gets "52.0% on their evals." Neither is a controlled reproduction, but both point the same way. porridgeraisin added that the prompt Claude Code wrote for Astra already described the looping bug, so by that account Astra didn't discover it.
The other HN objection is that this is just a steering vector. Subtracting one direction from 64 columns in two down_proj matrices is mechanically very close to abliteration, which removes a refusal direction the same way. yurimo, who says part of their research is in weight-space learning, called it "basically a steering vector, a very established technique" and said "the blog writing and subsequent posting overstate the results." theptip said the experiments "muddy the water": the early experiments showed one-shot weight reading, while the Qwen result is "more of a classic steering vector intervention."
I agree with the critics on the headline. The Qwen result is the weakest experiment in the post, and it got the attention. The Kinetix fix and the from-scratch Breakout policy deserve it more. A frontier model reading raw weight matrices and producing a working policy with no training loop is new; turning down a loop direction in a model whose vendor already warns about loops isn't.
Credit where it's due, though: Astra found a 0.016% edit that cut truncation from 93% to 33%, and the authors published the full Astra logs. I'd call that decent interpretability work. I wouldn't call it a math result.
How to run the edited Qwen3.5-0.8B locally
The official release is RikiyaT/Qwen3.5-0.8B-weight-space-programming: Apache-2.0, safetensors, BF16/F32. Only two tensors differ from the base model, model.language_model.layers.15.mlp.down_proj.weight and layers.19.mlp.down_proj.weight. There's also a hosted demo.
The card includes an independent re-run by akhaliq with the post's vLLM settings: the edited model scored 51.0% on full MATH-500 with 31.6% of answers hitting the limit. The base model only got a 25-problem smoke test, which scored 5.0% with 95% hitting the limit. That backs the truncation story, but 25 problems is not a full baseline.
transformers
The card itself only shows vLLM. For transformers, use the multimodal class, because Qwen3.5 is image-text-to-text. The processor also needs Pillow and torchvision, or from_pretrained fails with an ImportError.
pip install -U transformers accelerate torch torchvision pillow from transformers import AutoProcessor, AutoModelForMultimodalLM repo = "RikiyaT/Qwen3.5-0.8B-weight-space-programming" processor = AutoProcessor.from_pretrained(repo) model = AutoModelForMultimodalLM.from_pretrained(repo, device_map="auto") messages = [{"role": "user", "content": [ {"type": "text", "text": "What is the sum of all positive divisors of 36?"} ]}] inputs = processor.apply_chat_template( messages, add_generation_prompt=True, tokenize=True, return_dict=True, return_tensors="pt", enable_thinking=True, ).to(model.device) out = model.generate(**inputs, max_new_tokens=32768, do_sample=True, temperature=0.6, top_p=0.95, top_k=20) print(processor.decode(out[0][inputs["input_ids"].shape[-1]:])) I ran this on transformers 5.22.1 on a 4-core CPU box with no GPU (with max_new_tokens cut to 48, because 32k tokens on CPU takes forever). It loads and starts a <think> block. Keep enable_thinking=True; the card's vLLM snippet sets it too. Without it the chat template emits an empty <think>\n\n</think> and you're testing non-thinking mode, not the setup in the table. On CPU, transformers also warns that it's falling back to slow reference kernels because causal_conv1d and flash-linear-attention aren't installed.
vLLM
transformers' generate has no presence penalty, so to match the eval, serve it with vLLM:
pip install vllm vllm serve "RikiyaT/Qwen3.5-0.8B-weight-space-programming" --dtype bfloat16 --max-model-len 34816 Then send temperature=0.6, top_p=0.95, top_k=20, presence_penalty=1.5 in your request, which matches the card. With the OpenAI client, top_k goes in extra_body.
llama.cpp and Ollama
The authors didn't publish GGUFs. A community repo, NotCherry/Qwen3.5-0.8B-weight-space-programming-GGUF, has a 542 MB Q4_K_M file and an f16 mmproj for vision.
# llama.cpp llama-cli -hf NotCherry/Qwen3.5-0.8B-weight-space-programming-GGUF:Q4_K_M \ -c 34816 --temp 0.6 --top-p 0.95 --top-k 20 --presence-penalty 1.5 # Ollama ollama run hf.co/NotCherry/Qwen3.5-0.8B-weight-space-programming-GGUF:Q4_K_M Two warnings. This is a third-party quant: the edit is a small subtraction across 131,072 weights, and the GGUF repo says it makes no evaluation or quality claims, so nothing shows the edit survives 4-bit quantization. And the gains only show up with thinking on and a long context. If your frontend caps output at 2k or 4k tokens, you're testing a different setup from the one in the table.
How to check the claim yourself
- Run the base and edited models side by side in thinking mode with the same seed, and count how many responses hit your token cap. That's the number the edit actually moves.
- Then run the base model with thinking off. If it beats both, you've reproduced pama's and kristopolous's complaint.
- If you run Qwen3.5-0.8B in production and see runaway generations, the Qwen card's advice still applies: adjust
presence_penaltybetween 0 and 2. I'd also stream output so you can kill loops early.
Is fine-tuning dead?
No. Gemma's flat line shows that removing a bug and adding a capability are separate jobs, and gradient descent is still the only tool here that does the second. Still, a model that reads down_proj matrices, finds the bad direction and patches it within 50 forward passes is a new kind of debugger, and I expect people to start using it for exactly that.
HN didn't settle the bigger question: is this the start of LLMs debugging other models' weights directly, or is it abliteration with a bigger model holding the scalpel? And does a full 500-problem run of base Qwen3.5-0.8B with thinking on land near 7.9%, or is the baseline a setup bug? If you've run it and counted truncations, post your numbers in the comments.
