We trained a small model
to route support emails.

Delivery.

Billing.

Damaged goods.

Account changes.

Forty thousand labelled emails.

Shuffle them.

Eighty percent to train,
twenty percent to test.

The test score was ninety six percent.

We celebrated.

We shipped it.

In the first month
it routed about three in four correctly.

Same model.

Same kind of emails.

The test set was not lying.

It was answering
a different question
from the one we asked.

Here is what we found
when we finally read the data.

Our biggest customers
send the same email
over and over.

Their systems use templates.

Only the order number changes.

Some threads
had been labelled twenty times,
once per reply,
each one quoting the last.

A random split
puts copies of the same email
on both sides.

So the test was full
of emails the model had already seen
in training,
with a different number in them.

It did not have to understand
damaged goods.

It only had to remember
this customer's wording.

That is leakage.

Information from the test
reached the training,
and the score measured memory.

Production has no copies.

Next month's emails
are new.

The fix is to split
the way the future arrives.

Group by customer,
or by thread,
so everything from one source
lands on one side.

In scikit-learn
that is GroupShuffleSplit,
or GroupKFold for cross validation.

Remove near duplicates
before you split,
not after.

And when the data has dates,
train on the past
and test on the most recent weeks.

Our honest score
came out at seventy nine.

Lower,
and true.

It matched production
within a point or two,
and for the first time
we knew which number to improve.

This does not only apply
to models you train.

If you test a prompt
on examples you also put in the prompt,
or tune it on the cases
you then grade it on,
the same thing happens.

Before you trust a score,
ask what the test had
that tomorrow will not.

– Serguey Asael Shinder