The right answer was the odd one out 67 % of the time

I build practice exams for Dutch certification exams: the VCA safety certificate, the hospitality licence (Sociale Hygiëne), the small-boat licence, marine radio and a few more. One model drafts the questions. A second model answers them blind, without seeing the drafter's notes. A third model, which neither wrote nor checked the question, verifies it independently. Only then does a question go live.

All three models were happy with the questions. Then I counted something none of them had flagged: in many questions you could find the right answer without reading the question.

The cue

Here's a made-up question in the same style:

What do you do first when you find a colleague unconscious next to an electrical cabinet?

A. Call the company's emergency response officer
B. Call the emergency number
C. Make sure the power is switched off

Two options start with "Call". One doesn't. If you've taken a lot of multiple-choice tests, your eye goes straight to C, and here it's right.

That would be fine once. It isn't fine when it's a pattern.

What I measured

I took every verified single-choice question where exactly one option starts with a different first word than the others, and counted how often that odd option was the correct answer:

Most of these exams have three options, so by chance the key would be the odd option about a third of the time. The boating exam has four options, so chance is a quarter. Every exam was above chance, and the hospitality exam was at two thirds.

A second cue was even stronger: absolute words. Where every wrong option contained a word like "only", "always", "never" or "but not", and exactly one option didn't, that option was the key in 82 % (B-VCA), 89 % (VOL-VCA), 81 % (Sociale Hygiëne), 63 % (boating) and 60 % (marine radio) of the cases. "Absolute answers are usually wrong" is one of the first tricks test-wise people learn. Our questions rewarded it.

Why it happens

My best explanation: a language model writes the correct answer carefully, as a specific action or a qualified statement. Then it writes the distractors as a set, and a set comes out parallel: same opening verb, same structure, and often an absolute word to make sure each one is clearly wrong. The key ends up as the one option that doesn't look like the others.

A model reviewing the question checks whether the key is correct and the distractors are wrong. It does that well. It doesn't ask whether the key looks different, because that isn't a question about content.

Why shuffling doesn't help

The first idea people have is to shuffle the options. That fixes a cue in the position (for example "the answer is often B"). This cue is in the wording. C is still the odd one out when it's shown as A.

What we changed

1. A mechanical check on every draft. It's a few lines of TypeScript and runs before any model looks at the question:

const firstWord = (t: string) => t.trim().split(/\s+/)[0].toLowerCase().replace(/[^\p{L}\p{N}]/gu, ""); /** True when the key is the only option with a different first word. */ function keyIsOddOneOut(options: { id: string; text: string }[], keyId: string): boolean { const key = options.find((o) => o.id === keyId)!; const others = new Set(options.filter((o) => o.id !== keyId).map((o) => firstWord(o.text))); return others.size === 1 && !others.has(firstWord(key.text)); } 

The real check has two more rules: every distractor is restricted (by an absolute word) and the key isn't, and the key is the only "yes" among "no"s (or the reverse). Each language has its own word lists. A conditional like "if" or "unless" doesn't count as a restriction.

2. A rewrite pass that only touches the options. Every live question that failed the check got new wording for its options. The correct answer, the fact behind it and the cited source stayed exactly the same. The rewrite went through the same blind check and independent verification as a new question, and the original was only retired when the replacement passed.

3. The verifying model reads for it too. The third model now treats a form cue as a reason to fail a question. On a later batch of 56 new questions, it passed 51 and failed 5, all five on form cues. Those were redrafted.

What it doesn't solve

This catches the cheapest cues, not all of them. A longer, more careful key is another classic, and so is grammar that only fits one option. The check is a floor: it makes sure that a question can't be answered from its shape alone. It doesn't make a question good.

If you generate quizzes with an LLM

• Count it. For your own bank, measure how often the key is the option with a different first word, and how often it's the only option without an absolute word. Compare with chance. It takes an afternoon.
• Write distractors like the key. Same structure, same length, same kind of qualifier. If the key is an action, the distractors are actions.
• Use absolute words in the key too, when they're true. Otherwise "never" becomes a giveaway.
• Check form separately from content. A reviewer who checks whether an answer is right won't notice that it also looks different.

The practice exams this came from are at examenzaal.nl (Dutch, with the VCA exams also in English).

I wrote this post with help from an AI assistant. The numbers come from our own question bank and code.