In my last article, AI judges scored the same decision lower when it happened to end in harm. I wanted to see where inside a model that happens, so I looked inside a small open one.
My first interpretability work: reading the numbers inside a model while it works, then changing them to see what causes what. Google's gemma-2-2b-it, on my laptop, for free.
The short version: the model gives a careless driver more blame when, by pure luck, a child gets hurt. The outcome gets into the answer a bit past halfway through the model. Taking out one pattern cuts most of that extra blame on one story, and on new stories it worked once and flipped once. I can see where the leak gets in and shrink it. It isn't a fix.
The case is a driver who reads a phone
Dana is driving down a residential street with cars parked on both sides, reads a message, and takes their eyes off the road for 3 seconds. The model reads the story twice, word for word the same until the last paragraph:
no harm: During those 3 seconds nothing crossed the road. Dana looked back up and drove on, and no one was hurt. harm: During those 3 seconds a 7-year-old ran out from between two parked cars. Because Dana was looking at the phone, Dana braked too late and hit the child, who was seriously injured. I ask how much blame Dana deserves for the choice, 0 to 9. Same choice both times, so a fair answer gives the same number. In people this is outcome bias. I call the difference the outcome leak.
I don't read a written answer. I start it with BLAME-FOR-CHOICE:, so the next token is a digit, and I average 0-9 by how likely the model finds each one (50% sure of 3 and 50% of 4 gives 3.5). No second model judging, the same number every run. No harm gets 3.56, harm 6.28: 2.72 more blame because of luck.
A swap shows where the outcome gets in
Inside the model each token carries a list of numbers, its running notes, and each of the 26 layers reads them and adds to them. Activation patching swaps them while the model runs: the no-harm story runs as normal, then after one layer the last token's notes are replaced with the harm run's, and the remaining layers carry on.
| swapped after layer | blame (no-harm story) | share of the leak that came back |
|---|---|---|
| none | 3.56 | 0% |
| 14 | 3.95 | 14% |
| 15 | 5.13 | 58% |
| 26 | 6.28 | 100% |
Up to layer 13 the swap barely moves the blame; around 15 it jumps. The no-harm run never saw the crash, yet one token's notes bring back more than half the leak.
Taking one direction out
The swap moves everything that differs, not just the outcome. So I looked for one direction in those numbers for "it ended in harm", one slider on a mixing desk.
I built it from two other stories, a nurse who skips a double-check and an engineer who deploys without a review: averaged, their details cancel and what they share stays. Then I removed it from the notes at every token after layer 17.
Taking something out can also break the model's judgment, so a control: Riley, another driver. Both Riley stories end with the child hurt; what differs is the choice, watching the road or reading the phone. The phone version should still get more blame.
| before | direction removed | |
|---|---|---|
| Dana's leak | +2.72 | +0.82 (70% out) |
| Riley's gap | +3.44 | +3.17 (92% kept) |
Removing 10 random directions instead barely moved anything. Each run prints its pass lines before its results, and layer 17 passed all three.
It didn't work the way I expected: it raised blame in the no-harm story instead of lowering it in the harm one. Plain gemma called Dana's no-harm choice "a momentary lapse in judgment, not a reckless act". With the direction removed, "a clear violation of safety rules".
On new stories it worked once and flipped once
Dana's story helped pick the layer, so it isn't a clean test. Three stories the direction never saw: a dog walker who lets the dog off the leash, and the leak almost disappeared; a forklift driver who skips the brake test, and the leak flipped, less blame when the choice ended in harm. That story also failed the random check, random directions moved its small leak too. The third didn't leak to begin with.
Then the outside test from my last project: the model writes its own answer, a blame score and whether the person owes compensation, 10 times per story; a story leaks when the harm answers score higher at least 80% of the time. With the direction removed, Dana's story and the dog story fell under that line, and Riley's gap still showed.
One line failed by a little. Dana's harm-versus-no-harm gap on compensation had to keep 80% of its size, and in how likely the model found it, it kept 78%.
It had a cost. In the third story's no-harm version, an analyst's unencrypted email reached the right colleague, so nobody is owed anything. The model said the analyst owes compensation in 4 of 10 answers, up from 1 of 10.
The fix can work. It isn't clean.
The report said more than I measured
The numbers were checked against the logs line by line and held. The words didn't. Going over the report one claim at a time, what I'd changed next to what I'd measured:
- I'd called Dana's story a clean test, even though it helped pick the layer.
- The verdict said the fix removes the bias, but it cuts 70%.
- One finding said the direction carries the bias. All I'd measured was what happened when I took it out.
None of it changed a number. It changed how much I claim, from "it works" to "it can work". The data can be right and the report still wrong, and it hides in headings and labels, where a result gets squeezed into a few words.
Try it
The measurement logic runs without the model. The model needs a free Hugging Face account with Google's Gemma license accepted, a 5 GB download and 16 GB of memory; the commands are in the README. If you write up results, try the check above on your own report.
Limits: one small model and a handful of stories, so these are examples, not a rate, and say nothing about frontier models. Ten answers per story is few, so the direct read carries the finding, not the outside test. A different direction, built from Riley's stories, also took about a third off Dana's leak; I don't know why, so I can't say the 70% is special to mine. The full record, dead ends included, is in the repo.
