The job

Konstantin Tsiolkovsky (1857–1935) derived the rocket equation and described the multistage rocket and the orbital station decades before anyone could build them. His personal papers are held as fond 555 of the Archive of the Russian Academy of Sciences in Moscow. The archive did the essential work years ago: it scanned the fond and put the images online. What it did not provide was a catalogue you can query, full-text search or a dataset. To find anything, you click through page views one file at a time.

Over the summer I built the missing layer: a catalogue of all 2,019 files, a dating for 1,969 of them taken from the archive's own catalogue cards, a classification of every page as handwriting or typescript, and a machine transcription of every page. All of it is public: a full-fond search, the dataset with a permanent DOI (10.5281/zenodo.21705221), the code and the preprint.

One caveat belongs at the top. The machine read everything and no human has verified a single page. The corpus is a finding aid, not a scholarly edition, and every file links back to the archive's own scan, which remains the source of record.

"Digitized" is not "usable"

Three things about the archive's portal only became visible once I tried to use it as a data source.

The identifier in a page address is not the archival file number. Id 300 is file 297, because 31 files carry lettered numbers (145a, 585a and others) that take a position in the running sequence without advancing the archival count. A catalogue built on the obvious reading of the URLs would cite the wrong files.

The inventory number in the address is decorative. Two URLs that differ only in that number return the same document. The real archival address appears only inside the file paths of the scans.

Large scans, above roughly 800 KB, are served cut off at about 130 KB. A browser saves the stub silently with no error. The only way to get them whole was byte-range requests.

None of this is a complaint about the archive, which scanned and published 51,008 pages. It shows the distance between images on the internet and a collection you can ask questions of.

Why model confidence is not a metric

To measure recognition accuracy you need ground truth: the same text keyed in by a person. For an archival fond there is none. If there were, machine reading would be unnecessary.

The usual workaround is to lean on the model's own confidence. It does not work here, and I doubt it works anywhere that matters. A fluent invention satisfies a confidence score as well as a correct reading does. On century-old handwriting, fluent invention is exactly the failure you are worried about.

The answer key the typewriter left behind

Personal archives of the typewriter era often keep one text twice: the author's manuscript and a typescript copy made from it, filed together. I read both with the same pipeline and compare the two transcriptions. The text is the same and the pipeline is the same. The only variable left is how legible the page is, so the disagreement between the two readings measures the difficulty of the handwriting.

Fond 555 holds 1,759 such pairs across 224 files. At the median, the two readings agree on 37% of the words. The median longest run they share verbatim is 10 words, and in 43% of pairs no shared run is longer. That is what "machine-read early twentieth-century manuscript" honestly means.

Checking the check

A proxy metric needs its own validation, and here that was possible in a small corner of the fond: files whose text also survives in a printed edition. On those, the paired-reading estimate is unbiased to within a percentage point of the true error. It also ranks pages by difficulty the way the truth does: a rank correlation of 0.67 across all 55 available pairs, 0.92 on the 32 pairs where the printed edition is demonstrably the same text, and 0.97 on the 17 most certain.

Scored directly against published editions, the transcription reaches 98.1% character accuracy on typescript and 81.1% on handwriting, or 91.7% and 73.7% at word level. Those two figures need a precise label. They were measured on the part of the corpus read by the earlier pipeline. The rest of the fond was read later by a different model and has not been scored against editions. What was measured across both parts is the paired agreement, and it is the same for both. That supports "the newer part reads no worse than the older one". It does not support "81% across the whole fond", and I do not claim it.

This is the rule I would carry into any production system. You almost never have ground truth for everything, but you almost always have it for something. Use that small piece to calibrate a proxy, check that the proxy ranks cases the way the truth does, and only then scale it.

Three results that looked like results

Each of these first looked like a success.

Comparing the author's drafts turned out to be impossible. The fond holds Tsiolkovsky's "The Spaceship" in two variants, ideal material for studying how he revised it, and part of the reason I started. The two variants share 19% of their words. That is below the 37% at which two machine readings of one and the same page agree. Below the noise floor, authorial revision cannot be told apart from reading error. I closed that line of work and built the limit into the tool: the comparison script refuses to report differences that do not clear the measured threshold.

The page classifier passed its test and was still wrong. I separated handwriting from typescript by the variation in ink-stroke lengths along a line. On 19 hand-labelled pages it scored 19 out of 19. Later an independent signal appeared on 4,675 transcribed pages, and against it the classifier is 80% accurate, where always guessing the commoner class gives 54%. Carbon copies and faded typescript read as handwriting. A small, clean test set certifies a demo, not a system.

A model defect posed as data. On 236 pages the model looped and repeated one line of markup, 635 times at worst. Each repeat counted as an uncertainty mark and inflated the published total by 6,189. No statistical check caught it, because 236 pages out of 51,008 do not move any distribution. A person opening a file and looking at it did. The runs are now folded, each with a note on the page saying so, and the corpus carries 299,939 uncertainty marks, about six per scan.

The difference between a measurement and an impression is that a measurement can tell you that you were wrong. Publishing those corrections is what makes the surviving numbers worth trusting.

The truncation that looked like bad reading

Newer models enable reasoning by default, and it draws on the same output budget as the answer. With a 4,096-token limit, up to 3,929 tokens went to reasoning. The transcription broke off mid-page, and the stub was indistinguishable from a poor reading. With reasoning set to minimum, quality did not drop and output volume halved.

If your pipeline evaluates model outputs, a silent truncation upstream will present itself as a quality problem downstream. Check the finish reason before you check the text.

A related edge case: two pages could not be read by any model. They are a German typewritten review of a 1927 Tsiolkovsky pamphlet, and they trip the built-in filter against reproducing known printed text verbatim. They were read by a separate route and are flagged in the corpus.

Choosing the model by measurement, not by price list

I compared candidate models on the same handwritten pages, scored against the typed copy of the same text, with the existing pipeline as the baseline. On typical handwriting no model clearly outran the rest. On a 48-page sample the more expensive model was better on 37 pages and worse on 11. The edge is real but modest, because the ceiling is set by the material, not the model.

The remaining 41,212 pages were read overnight in one batch run, for about $39. The bottleneck in work like this stopped being compute some time ago. It is measurement.

What carries over to production

  • Find the duplicate. Most systems hold some outputs twice in different forms: a document and its summary, a form and its scan, a query and its report. Agreement between two independent paths is a usable proxy for error.
  • Validate the proxy on the little ground truth you have, and check that it ranks cases correctly, not only that its average is close.
  • Report the noise floor and let it constrain the product. If a difference is below it, the tool should refuse to show it.
  • Look at raw outputs. Distribution-level checks miss rare, loud failures.
  • Check finish reasons. Truncation masquerades as poor quality.
  • Benchmark on your own material against a baseline before choosing a price tier.

My previous post, The Model Was Never the Bottleneck, ended in the same place from the other side: the work that decides whether an AI system can be trusted is rarely the model itself.

Useful Resources

Author’s note: This is an independent project. It is not affiliated with or endorsed by the Archive of the Russian Academy of Sciences. The method is described in a preprint (arXiv:2608.03617); the data are released under CC0 and the code under MIT.