PowerCons is a small time series dataset: 180 training rows, 180 test rows, two classes. It is one of the 128 datasets in the UCR archive, which researchers use to compare classifiers. 67 of its 180 test rows are also in its training set, value for value. A 1-nearest-neighbour classifier gets all 67 right, because each one has its own answer sitting in the training data.
I found this while auditing the UCR and UEA archives row by row, and the audit became a preprint. This post leaves the statistics to the paper and covers the check itself. It takes about 20 lines of Python and a fraction of a second, and you can point it at your own benchmark today. I ran every code block below against the real archive files, and every output is pasted from those runs.
Why nothing warns you
A random split assumes the rows are independent. Copies break that assumption silently. Nothing throws, the shapes are right, the class counts look fine, and the score goes up. A higher score looks like a better model, so nobody goes looking for the cause.
Comparing every test row with every training row would find them, but that is n * m comparisons of long vectors. A hash set does it in one pass over each split. Give each row a fingerprint, put the training fingerprints in a set, and look up each test row.
Two passes, one lookup per test row.
Two details decide whether this is useful. Leave the label out of the hash, because a copy with a different label is still a copy, and it is one of the more interesting things you can find. And keep the row indices rather than a count, because every later step needs them.
Run it on a .ts file
UCR and UEA ship in the .ts format. audit_ts.py reads it with only the standard library. A row's identity is its text with the label cut off, so this is the whole idea:
audit_ts.py
def read_ts(path): keys, labels = [], [] in_data = False with open(path, "rb") as f: for raw in f: line = raw.strip() if not line or line.startswith(b"#"): continue if in_data: body, _, label = line.rpartition(b":") keys.append(hashlib.md5(body).digest()) labels.append(label.decode()) elif line.lower().startswith(b"@data"): in_data = True if not keys: sys.exit(f"{path}: no @data rows found") return keys, labels MD5 is fine here. Nobody is attacking this hash, and all it has to do is tell rows apart. Download the archive zip, then run it on a train and test pair:
git clone https://github.com/mahmadmanzoor/archive-audit python archive-audit/audit_ts.py PowerCons_TRAIN.ts PowerCons_TEST.ts n_train: 180 n_test: 180 duplicate_rows: 130 test_rows_with_train_twin: 67 test_twin_pct: 37.22 twin_rows_with_conflicting_label: 0 groups_with_conflicting_labels: 0 test_rows_with_train_twin is the number that matters. duplicate_rows counts across both splits: 360 rows in total, 230 distinct, so 130 are repeats. The two conflict lines count copies whose labels disagree, and here there are none.
It stays fast on bigger data. Crop has 24,000 rows, and the same command finished in 0.06 seconds and reported 2,753 of 16,800 test rows, or 16.39%. Add --json to get the row numbers.
17 of 128 UCR datasets and 3 of 30 UEA datasets leak on their default split. Eight of the UCR ones leak at least 1%.
The same check for arrays and dataframes
Your benchmark probably is not in .ts files. Once the data is numeric you can hash the values instead of the text, which also means 1.0 and 1.00 finally match.
fingerprint.py
import hashlib import numpy as np def fingerprints(X, decimals=6): """One hash per row, after rounding away float noise.""" X = np.asarray(X, dtype=np.float64) X = np.round(X, decimals) + 0.0 # + 0.0 turns -0.0 into 0.0 return [hashlib.blake2b(row.tobytes(), digest_size=16).hexdigest() for row in X] def leak_report(X_train, X_test, decimals=6): train = set(fingerprints(X_train, decimals)) test = fingerprints(X_test, decimals) twins = [i for i, h in enumerate(test) if h in train] pct = round(100 * len(twins) / len(test), 2) return {"test_rows": len(test), "twins": len(twins), "pct": pct, "indices": twins} row.tobytes() does not care about shape, so this works on a batch of images as well as on a table. If you want to follow along with the archive files, this loader turns them into arrays:
load.py
import numpy as np def load_ts(path): X, y = [], [] in_data = False for line in open(path): line = line.strip() if not line or line.startswith("#"): continue if in_data: body, label = line.rsplit(":", 1) X.append([float(v) for v in body.split(",") if v != "?"]) y.append(label) elif line.lower().startswith("@data"): in_data = True return np.array(X), np.array(y) The decimals argument needs checking, because rounding is a trade-off. Too strict and float noise hides real copies. Too loose and different rows collapse into one. So run it at several values and see if the count holds:
X_train, y_train = load_ts("PowerCons_TRAIN.ts") X_test, y_test = load_ts("PowerCons_TEST.ts") for d in (2, 3, 4, 6, 10): print(d, leak_report(X_train, X_test, decimals=d)["twins"]) noisy = X_test + 1e-9 strict = leak_report(X_train, noisy, decimals=12)["twins"] rounded = leak_report(X_train, noisy, decimals=6)["twins"] narrow = leak_report(X_train, X_test.astype(np.float32))["twins"] print("noise, decimals=12:", strict) print("noise, decimals=6: ", rounded) print("float32: ", narrow) 2 67 3 67 4 67 6 67 10 67 noise, decimals=12: 0 noise, decimals=6: 67 float32: 67 The count is flat from 2 to 10 decimals, so these are real copies and not an artefact of rounding. The noise lines show why exact bytes are not enough: add 1e-9 to every value and a strict match finds nothing at all, while decimals=6 still finds all 67. If your own count climbs as you lower decimals, you are starting to merge rows that are merely similar.
Exact matching also misses a copy that has been rescaled. If your pipeline normalises each row, hash after normalising:
def znorm(X): X = np.asarray(X, dtype=np.float64) std = X.std(axis=1, keepdims=True) std[std == 0] = 1 return (X - X.mean(axis=1, keepdims=True)) / std scaled = X_test * 2 raw = leak_report(X_train, scaled) normed = leak_report(znorm(X_train), znorm(scaled), decimals=3) print("raw: ", raw["twins"]) print("z-normalised:", normed["twins"]) raw: 0 z-normalised: 67 With a dataframe, pandas has the hash built in. Drop the label column first:
import pandas as pd from pandas.util import hash_pandas_object columns = [f"x{i}" for i in range(X_train.shape[1])] train_df = pd.DataFrame(X_train, columns=columns).assign(label=y_train) test_df = pd.DataFrame(X_test, columns=columns).assign(label=y_test) features = [c for c in train_df.columns if c != "label"] def row_hashes(df): return hash_pandas_object(df[features].round(6), index=False) is_copy = row_hashes(test_df).isin(set(row_hashes(train_df))) print(is_copy.sum(), f"{is_copy.mean():.2%}") 67 37.22% Those are the same 67 rows the array version found.
Measure what the copies do to your score
A leak rate tells you copies exist. It does not tell you what they cost. For that, split your accuracy into twin rows and the rest:
from sklearn.neighbors import KNeighborsClassifier X_train, y_train = load_ts("PowerCons_TRAIN.ts") X_test, y_test = load_ts("PowerCons_TEST.ts") report = leak_report(X_train, X_test) is_twin = np.zeros(len(X_test), dtype=bool) is_twin[report["indices"]] = True model = KNeighborsClassifier(n_neighbors=1).fit(X_train, y_train) pred = model.predict(X_test) correct = pred == y_test print(f"all rows: {correct.mean():.2%} ({len(correct)})") print(f"twin rows: {correct[is_twin].mean():.2%} ({is_twin.sum()})") print(f"the rest: {correct[~is_twin].mean():.2%} ({(~is_twin).sum()})") all rows: 97.78% (180) twin rows: 100.00% (67) the rest: 96.46% (113) This is raw values on the one default split, so treat it as a demonstration. The twin rows are a free 100%, and removing them takes the score down by over a point. Worms on the same setup goes from 45.45% to 37.31%.
For a fairer estimate the paper averages 30 resamples with and without twins:
- 1-NN on z-normalised values: PowerCons 91.67% to 86.63%, Worms 47.75% to 35.52%, Crop 71.06% to 65.47%.
- MiniRocket (2,000 kernels): PowerCons 95.52% to 92.83%, Worms 70.48% to 66.48%, Crop 69.99% to 65.44%.
So this is not only a nearest-neighbour effect. A model that trains on a row and is tested on that same row benefits whatever its design. On the published fold-0 predictions, the mean over 17 classifiers on Crop goes from 74.7% to 69.8%.
Warning: Do not try to find twins with
kneighborsanddistance == 0. On PowerCons, sklearn reported a non-zero nearest distance for 24 of the 67 rows I had confirmed identical, as high as 1.58e-06. Compare fingerprints instead.
Check both directions
Everything above asks how much of the test set is in train. InsectEPGSmallTrain shows why that is not enough:
X_a, _ = load_ts("InsectEPGSmallTrain_TRAIN.ts") X_b, _ = load_ts("InsectEPGSmallTrain_TEST.ts") in_train = leak_report(X_a, X_b)["twins"] in_test = leak_report(X_b, X_a)["twins"] print(in_train, "of", len(X_b), "test rows are in train") print(in_test, "of", len(X_a), "train rows are in test") 15 of 249 test rows are in train 15 of 17 train rows are in test Reported the usual way, this dataset leaks 6% and you move on. Flip the arguments and 15 of its 17 training rows are in the test set. Freezer's SmallTrain variant has 26 of 28, while MixedShapes' is clean. A dataset that exists to test learning from a few examples should not test on nearly all of those examples.
Same copies, very different headline.
Read the copies before you call it a bug
Not every copy is a mistake. ElectricDevices has 208 test rows with a training twin, 2.7% of its test set. But 54 of its twin groups carry conflicting labels, and 101 test rows have no training twin with their own label. Over 30 resamples, removing the twins there raises 1-NN accuracy slightly, from 69.39% to 69.65%, because twin rows only score 61.5%. So copies do not always help.
The reason shows up when you count distinct values in each twin row:
def median_distinct(X_tr, X_te): rows = X_te[leak_report(X_tr, X_te)["indices"]] counts = [len(set(row)) for row in rows] return int(np.median(counts)), rows.shape[1] X_e, _ = load_ts("ElectricDevices_TRAIN.ts") X_f, _ = load_ts("ElectricDevices_TEST.ts") print("ElectricDevices:", median_distinct(X_e, X_f)) print("PowerCons: ", median_distinct(X_train, X_test)) ElectricDevices: (2, 96) PowerCons: (77, 144) Each result is the median number of distinct values and the row length. ElectricDevices twins are mostly on/off patterns, so two different appliances can produce the same row by coincidence. PowerCons twins are far richer, which makes a chance match much less believable. A copy can also be legitimate in the sense that real deployments see it too. So I would report the score both ways, and I would not quietly delete rows.
Do not loosen the match
The tempting upgrade is near-duplicate detection, because exact matching obviously misses some copies. Here is what a strict-looking correlation threshold does on OliveOil:
X_o, _ = load_ts("OliveOil_TRAIN.ts") X_q, _ = load_ts("OliveOil_TEST.ts") best_r = (znorm(X_q) @ znorm(X_o).T / X_o.shape[1]).max(axis=1) share = (best_r >= 0.9999).mean() print("best r >= 0.9999:", f"{share:.0%}", "of test rows") print("lowest best r: ", round(float(best_r.min()), 6)) print("exact twins: ", leak_report(X_o, X_q)["twins"]) best r >= 0.9999: 100% of test rows lowest best r: 0.999963 exact twins: 0 Every test row correlates above 0.9999 with something in train, and none of them is a copy. These are smooth spectra, and smooth spectra correlate. The paper sees the same thing on Wine (92.6%) and Meat (83.3%). A near-match is fine as a lead to go and look at, but I only count exact matches as leakage, and I accept that the result is a lower bound.
Fix the split
If you control the split, make the fingerprint the group, so all copies of a row land on the same side:
from sklearn.model_selection import (GroupShuffleSplit, StratifiedGroupKFold) X = np.vstack([X_train, X_test]) y = np.concatenate([y_train, y_test]) groups = np.array(fingerprints(X)) def show(tr, te): twins = leak_report(X[tr], X[te])["twins"] print(len(tr), "train,", len(te), "test,", twins, "twins") splitter = GroupShuffleSplit(n_splits=1, test_size=0.5, random_state=0) show(*next(splitter.split(X, y, groups=groups))) folds = StratifiedGroupKFold(n_splits=2, shuffle=True, random_state=0) show(*next(folds.split(X, y, groups=groups))) 180 train, 180 test, 0 twins 181 train, 179 test, 0 twins GroupShuffleSplit does not balance classes, so use StratifiedGroupKFold when that matters. Both keep repeated rows together instead of deleting them.
