TL;DR — We built ja-ocr-error-detect-v1, a detector for OCR errors in prewar Japanese printed texts, and we are not releasing it. Inside its synthetic training distribution it scored character F1 0.64. On real OCR output — 80 pages of National Diet Library modern materials, used for evaluation only — it scored F0.5 0.078 and caught about 2% of the actual errors. Three generations of fixes later it still missed the release bar we had set in advance (F0.5 ≥ 0.20 with at most 1.0 false flags per 1,000 characters), so the project is frozen. This is the full record of what we measured.
- The synthetic-to-real gap was not a gap; it was a cliff. Character F1 0.64 on synthetic data became F0.5 0.078 on real OCR text, detecting roughly 2% of real errors. The errors we had synthesized were not the errors prewar print actually produces.
- The untrained baseline beat the trained model. Before any fine-tuning, the base model's masked-language-model signal scored PR-AUC 0.137 at spotting real errors. The trained detector scored 0.067. Training on synthetic errors taught the synthetic distribution so well that it actively hurt detection of the real thing.
- The errors live in the layout, not the text. From 120 further pages we collected 2,494 real error characters and classified them. Of the extra-character errors, 74.9% sat at line starts and line ends — kanji bleeding in from adjacent lines, ruby annotations intruding into the main text. A line-at-a-time text model cannot see the cause, because the cause is the page.
The setup
The goal was a small detector for OCR errors in digitized prewar Japanese books and magazines — the kind of material where OCR struggles with old kanji forms, vertical typesetting, and ruby glosses. Detection only: flag the suspicious character spans for a human or a downstream corrector, the same point-don't-fix design as our modern-text error detector.
Training data was synthetic: errors injected into clean text. Evaluation was not: 80 pages of real OCR output from the National Diet Library's modern materials collection, held out for evaluation only and never trained on. Because a flagging tool that cries wolf is useless, we scored with F0.5, which weights precision above recall, and tracked false flags per 1,000 characters alongside it.
The release bar was fixed before the final measurements: F0.5 at least 0.20, and false flags at most 1.0 per 1,000 characters. Both conditions, not either.
What the real pages said
The first detector failed in an instructive order:
- Synthetic character F1 0.64 → real F0.5 0.078, about 2% of real errors detected
- The base model's raw MLM signal, with no training at all: PR-AUC 0.137 on real errors
- The trained detector on the same real errors: PR-AUC 0.067
Read that second pair again. We took a signal that worked modestly and trained it into one that worked half as well. The synthetic error distribution — plausible substitutions, plausible insertions, in plausible places — was learnable, and learning it crowded out the weaker, real signal.
So we stopped tuning and went to look at the errors themselves. From 120 additional pages outside the evaluation set, we harvested 2,494 real error characters and classified them by type and position. The dominant finding was positional: among extra-character errors, 74.9% occurred at the start or end of a line. These are not language errors. They are segmentation accidents — the OCR engine pulling a character from the neighboring line, or swallowing a ruby gloss into the body text. The text on the line, read alone, often looks perfectly plausible; the wrongness only exists relative to the page.
Two more generations, honestly scored
Generation 2 was rebuilt using the measured real-error patterns instead of our imagined ones. Detection rose about 2.8×. False flags rose with it: from 0.79 to 4.27 per 1,000 characters — four times over the bar. Catching more by accusing more is not progress a proofreading tool can ship.
Generation 3 attacked the cause we had measured: if errors cluster at line boundaries and arrive from adjacent lines, the model needs the adjacent lines. We added previous- and next-line context. It helped — F0.5 0.141, false flags down to 2.89 per 1,000 characters — and it was not enough. Against the bar set in advance (F0.5 ≥ 0.20, false flags ≤ 1.0), generation 3 fails on both axes.
So we stopped. The model is not released, the weights stay internal, and the project is frozen rather than quietly abandoned — this article is its record.
What didn't work / Limitations
- Text-only, line-unit detection has a ceiling, and we hit it. The largest error class is caused by page layout — line segmentation and ruby intrusion. Context lines bought a real improvement (0.078 → 0.141) and still landed far from the bar. The remaining distance is probably not in the text at all; it is in the image. A layout- or image-aware approach is a different project, and we have not tested one, so we claim nothing about whether it would clear the bar.
- The evaluation is one collection. 80 pages of National Diet Library modern materials is a real measurement, but it is one corpus, one period, one OCR pipeline. Other collections would move the numbers; nothing suggests they would move them enough to rescue this design.
- The generation-2 and generation-3 numbers are progress reports, not near-misses. 0.141 against a 0.20 bar with triple the allowed false flags is a clear miss. We are not publishing these as "almost worked."
- What we keep: the error census (2,494 real errors, classified) is the durable output. If anyone — including us, later — retries this problem, the line-boundary finding is where to start: model the page, not the line.
This is the tenth model in our small Japanese model line, and the first one we froze instead of shipping. Every other model in the line cleared a bar we set before measuring it. This one didn't, and the bar does not move after the measurement — that is the entire point of setting it first.
If you work with historical OCR: where do you put the correction effort — better OCR engines, post-correction models, or human review of flagged spans? And what error rate would a flagging tool need before you would trust it in your pipeline?
- The models that did ship, with their measured results: https://jpopendata.jp/models/
Get the next release notes and our Kaggle experiment logs by email: https://jpopendata.jp/ (newsletter signup in the footer).
