I trained an EfficientNet-B0 to grade diabetic retinopathy on APTOS 2019. The five-fold ensemble reached QWK 0.909 on the held-out test split. Public solutions go past 0.93, so the score itself was not the interesting part. What I wanted to know was whether the model earned it for reasons that would survive a different clinic.

The baseline that should not work

Before trusting the CNN, I trained a RandomForest on file properties only: resolution, aspect ratio, brightness, contrast and file size. It never saw a retinal pixel.

metadata only always predict 0 CNN ensemble
QWK 0.652 0.000 0.909
Accuracy 0.708 0.493 0.803

The reason is acquisition. APTOS was collected across sites with different cameras, and resolution is a camera signature. Of the 974 images at 1050x1050, 92.5% are graded No DR. Across every other resolution the share is 33.6%. Device correlates with disease prevalence, so file properties predict the label.

Within APTOS I could only narrow this down. Split by resolution, the CNN still beat the metadata model in both strata, but the 1050x1050 test stratum held four diseased eyes, and the other stratum still mixed sixteen resolutions with very different label shares. To actually remove the shortcut I needed a test set from one camera.

Writing the prediction down first

IDRiD has 455 images, all at 4288x2848, so geometry carries no label information. The prior is also inverted: every APTOS image at that resolution is diseased, while 129 of the 455 IDRiD eyes are healthy.

Before running anything I wrote down a prediction: if the model leaned on acquisition cues, it would over-call disease on IDRiD and specificity would collapse.

It did not. Same ensemble, no fine-tuning, APTOS thresholds unchanged:

APTOS test IDRiD
QWK 0.909 0.805
Referable sensitivity 0.956 0.885
Referable specificity 0.917 0.987

Specificity went up. Missed referrals (true grade 3+ called 0 or 1) were 2 of 148. That rules out exclusive reliance on the resolution proxy, though not every possible acquisition confound.

The second external set is where it broke

Messidor-2 is 1,744 gradable images from France, each graded by a panel of three retina specialists who adjudicated disagreements. APTOS labels come from single graders. I committed four predictions before downloading a single image. The first one was referable AUC of at least 0.93.

APTOS test IDRiD Messidor-2
Referable AUC 0.983 0.984 0.819
Five-way QWK 0.909 0.805 0.493

That prediction failed. At the APTOS operating point, referable sensitivity on Messidor-2 was 0.26. The failure sits in one place: 83% of the 347 Moderate eyes were graded below Moderate, against 21% on IDRiD. Grades 3 and 4 were still recognised. Moderate disease with hard exudates near the fovea scored much higher than Moderate disease without them.

My reading, made after seeing the data, was that this is about the labels rather than the images. APTOS single-grader labels disagree with themselves on 29% of duplicate pairs (43 of 148). A model trained on them learns where single graders put the Mild/Moderate boundary, and an adjudicating panel puts it lower.

Testing the labels reading

I then tested that reading on an examination-disjoint split of Messidor-2 (872 test eyes, 437 examinations). One arm was fine-tuned on the panel's labels. A control arm was fine-tuned on the same images, but with the unchanged model's own grades as labels, so it got the new camera without the new boundary.

referable AUC
Unchanged 0.8405
Fine-tuned, control labels 0.8534
Fine-tuned, adjudicated labels 0.9435

The adjudicated-minus-control difference is +0.090, with a 95% paired bootstrap interval of [+0.061, +0.120]. So most of the gain comes from the labels, not from seeing new images. Messidor-2 had already been inspected by then, so this is exploratory rather than a fresh confirmation. It also had a cost: accuracy on the APTOS test set fell from 80.6% to 64.2%. No fine-tuned model was promoted to the demo.

What I took from it

  • An internal test score tells you how well a model reproduces its training labels' source, cameras and graders included. It does not tell you which of those it learned.
  • A shortcut baseline is cheap. If a model that cannot see the content scores 0.65, the headline number needs a second look.
  • Writing predictions down before an external run made the results easier to read honestly. One held, one failed, and the failed one taught me more.

This is a research project, not a diagnostic tool. Prospective clinical use, fairness across populations and multi-seed robustness were not evaluated.

Links