My first comparison of two releases showed a clean accuracy gain and two large regressions. None of the three was real. Fixing that took four bugs. Once the numbers were right, half of my wording benchmark turned out not to be held out, and a conclusion I nearly published came from comparing effects measured at different candidate-set sizes.
The setting
I maintain Ru-Jev, an open typed-decision model for Russian-language text. It takes a context and a set of candidate options and returns a probability for each option. It does not generate text. Technically, it is a pointwise cross-encoder on mmBERT-base, 307M parameters: each (context, option) pair gets its own encoder pass, and a softmax over the options gives the distribution. It runs on CPU, so it can sit inside a closed network. Weights and code are MIT.
Version 1.3 had one goal. The caller names the options, in the caller's own words, and version 1.1 had learned the exact label wordings in its training data. So 1.3 presents each label in several surface forms during training, including English intent keys transliterated into Cyrillic and a clipped form of that transliteration, and draws one form per example.
Bug one: a split that moved when I edited something else
The first table had two regressions I could not explain. Sentence acceptability fell from 0.909 to 0.647. A reasoning set fell from 0.939 to 0.596. A release that only changed label wording had no mechanism for that.
The data builder carved each task's validation slice like this:
random.shuffle(pool) n_val = max(1, int(len(pool) * val_fraction)) val, train = pool[:n_val], pool[n_val:] random.shuffle with no generator draws from the module-level stream, which every other random call in the file also consumes, including the new choice of label wording. Adding label styles to one task changed how many draws happened before the next task was carved, and every later task got a different split.
That is leakage, and it can be measured: 90.3% of 1.3's acceptability validation rows and 89.4% of its reasoning rows had been in 1.1's training data. The 0.909 was 1.1 scored on rows it had trained on. On held-out rows it scores 0.650.

The fix seeds each carve from the task name:
srng = random.Random(f"split:{task}") srng.shuffle(pool) The check is what confirms the diagnosis. I rebuilt the previous release's data from its commit and confirmed that the six tasks whose option sets never changed matched at 100%. Those tasks had no reason to move, so if they had moved, my diagnosis would have been wrong.
Bug two: matching rows by input text is not enough
With the split repaired I matched the two validation sets by input text and found nine comparable tasks. In this release a row with the same text can carry an option wording 1.1 never saw, so scoring 1.1 on it measures robustness to renaming, not accuracy. Matching on text and option wording together leaves six tasks.
A toxicity gain from 0.722 to 0.954 that I had presented as a capability gain was this bug. Version 1.1 scores 0.961 on toxicity with the labels it trained on.
Bug three: two denominators in one table
The summary row of my comparison table, 0.7948 against 0.7920 on 5,078 decisions, came from one random half of the comparable rows. The per-task table beside it counted all 10,157. A calibration advantage, 0.0172 against 0.0160, came from the same single split. Both numbers reached the published model card.
On all 10,157 rows the two releases answer exactly 8,120 correctly, disagreeing on 1,680 rows split 840 and 840. Over 25 random splits the calibration difference changes sign. Both claims were withdrawn. What survives is narrower: 1.3 takes a lower fitted temperature in all 25 splits, so its raw scores are less overconfident, but after each model gets its own temperature the residual calibration error is the same.
Bug four: an integrity check against a stale reference
To measure seed variance I retrained both recipes on a rented GPU and checked the data on the remote machine against my local copy by SHA-256, row count and per-task counts. Everything matched. The local copy predated the commit that added the transliteration styles, so the "augmented" seed runs were trained without the augmentation. Their jargon accuracy fell in the baseline's range, and I first read that as an unstable recipe.
Row counts could not catch it, because the new styles change option strings, not the number of rows. What caught it was counting distinct option strings per task: 120 in the stale copy, 198 after rebuilding. Those runs are excluded from everything.
The misleading benchmark
Once the numbers were right, I ran a wider wording test: ten label vocabularies on the same 1,000 routing questions, each scored with five candidates per question and eight of them also with all eighteen. Because the model scores every option independently, the five-candidate answer can be computed exactly from the eighteen-candidate run. I checked every probe label against all training option strings, used exact McNemar tests, a bootstrap that resamples labels rather than just questions, and a Holm correction over 23 comparisons.
Two things came out of it. First, two of my six original benchmark vocabularies, the English key and the ordinary Russian word, turned out to be training styles for both releases, and a third, the Russian phrase, was a training style for the new release only. Half of my original benchmark was not a held-out test.
Second, the effect size depended on the number of candidates, differently for different vocabularies. I had first measured jargon with five candidates and slang with eighteen, which made the slang gain look far smaller than the jargon gain. With both vocabularies in both settings:
| Vocabulary, slice | 5 options: v1.1 → v1.3 | 18 options: v1.1 → v1.3 |
|---|---|---|
| English-derived jargon, strict (no exact or prefix match to training) | 0.550 → 0.786 | 0.328 → 0.470 |
| Office slang, strict | 0.595 → 0.761 | 0.369 → 0.419 (n.s. after Holm) |
| Department names (no exact match; 2 of 18 labels share a prefix with a training option) | 0.835 → 0.869 | 0.632 → 0.733 |
| Abbreviations, clean | 0.420 → 0.360 | not scored |
| Opaque codes, strict | 0.135 → 0.168 (n.s.) | 0.018 → 0.061, both at or below chance |

On every held-out vocabulary made of words, the new release is better in both settings, though not every gain survives the Holm correction. On abbreviations and opaque codes there is no usable gain. The two releases differ in more than the augmentation (I also fixed the split and trained four epochs instead of three), so this compares two trained models rather than a clean ablation. Two replicates of the v1.1 recipe scored 0.658 and 0.622 on jargon with five candidates, against 0.633 for the release, so small differences in the table should not be read as reproducible; the jargon gains are large relative to that spread. The contrast I almost published between jargon and slang came from comparing a five-candidate effect with an eighteen-candidate one.
Accuracy and confidence also diverged. Where accuracy rose, the risk–coverage curve mostly improved, but with eighteen candidates the number of wrong answers given above 0.8 confidence went up on jargon (all labels, 79 to 109 per 1,000) and on slang, because the new release is more confident overall. On department names it went down.
A checklist
- Seed every split from the name of the thing being split, never from a stream shared with other code.
- Define comparability as everything the model consumes, then compute the overlap and print it.
- Measure leakage directly: what fraction of the new validation set appears in the old training set.
- Pick a control that must not move. A fix that does not predict what stays the same is not understood.
- Keep one denominator per table, and replace any figure read off one random split with its distribution over splits.
- Verify artefacts by the content your change is supposed to alter, not by a hash against a copy of unknown age.
- Check every probe vocabulary against the training data before calling it held out. Test wording at more than one candidate-set size, with styles that were not in your augmentation, and resample labels, not just questions.
Explore the Model
Author’s note: The name references Jev, a commercial decision model by TypeSafe AI. Ru-Jev is an independent project, not affiliated with TypeSafe AI and not trained on Jev outputs.
