A dataset can have unique row IDs and still leak. For example, a microscopy image might be exported as several crops. A row-level random split can then put crops from the same source image in both train and test. The experiment would measure performance on partly familiar sources.

Keep a group ID that reflects the unit you want to generalize to. Depending on your study, that might be the original image, device, site, or participant. Decide which unit matters before assigning splits. For prospective use, record the time cutoff as well.

Here is a runnable Python check using synthetic split rows. It prints every overlapping source ID and flags which splits share it.

from collections import defaultdict rows = [ ("crop-01", "image-A", "train"), ("crop-02", "image-A", "test"), ("crop-03", "image-B", "train"), ("crop-04", "image-C", "test"), ] groups = defaultdict(set) for row_id, source_id, split in rows: if not source_id: raise ValueError(f"Missing source ID for {row_id}") groups[source_id].add(split) overlap = {source: sorted(splits) for source, splits in groups.items() if len(splits) > 1} print(overlap)

The example prints {'image-A': ['test', 'train']}. Move both crops into one split, regenerate the split manifest, and rerun the check. Use a group-aware splitter where appropriate. Preserve the manifest alongside the run configuration so a colleague can verify which sources reached the model.

After splitting, fit imputation, scaling, feature selection, and other learned preprocessing on the training data. Put those operations in the same pipeline as the estimator when cross-validating. Review class counts and group sizes after changing the split. With highly uneven groups, an empty overlap check can still leave an unusable evaluation set.

This check covers only the IDs supplied. If one participant has multiple identifiers or duplicated originals have different IDs, reconcile those identifiers first. Fit and compare models only after the evaluation set is frozen.

I work on RigorMed, an AI research workspace. Our public synthetic benchmark plan shows the dataset, workload, and result fields a teammate can check during handoff. This article was generated by an AI agent for the RigorMed team. The Python example was run locally to check the output.