Are twelve records twelve independent examples?
You have twelve session records from six entities, A through F. Each entity appears twice with the same signature and label. If one session enters training and the other validation, are you evaluating new entities?
Our deliberately simple predictor remembers training signatures and their classes. For an unseen signature, it predicts class 0. Its purpose is to expose overlap, not to recommend a model.
Establish an overlapping reference
Open the dataset and compare rows with matching entities. Unique row identifiers do not make those observations independent.
Use as the reference. There are six training and six validation records. Predict how many entities appear in both sets and how well a lookup can classify the validation rows. Record the run.
Change only to , keeping held-out entities D, E, F. The predictor and six-record budgets stay fixed. Predict again before running.
Six entities, two sessions each: 12 records. Each entity repeats one signature and label. Both splits contain six records. A lookup predictor memorizes training signatures; unseen signatures receive class 0. Only the split changes. This diagnoses overlap, not a realistic classifier.
0/16 runs. Each run keeps its own inputs and results. Work stays in this tab across reloads; download a report before closing it.
Write a prediction to record your reference.
Inspect the validation predictions. Explain the score change using shared signatures rather than just calling one split harder.
Compare what the two scores measure
In the reference, all six validation entities already appeared in training. Every signature is recognized, giving 100% accuracy.
With D, E, F held out, no validation signature is recognized. Their labels are 1, 0, 1, each repeated twice. The fallback predicts 0, so it gets two of six records right: about 33.33%.
The second score addresses previously unseen entities. The first score addresses repeat sessions of known entities. Which is relevant depends on deployment; neither is a precise population estimate from six entities.
Define the unit before splitting
Exact duplicate records, equal feature values, and repeat observations of one entity are different situations. Removing every repeated feature vector can discard valid measurements or conflicting labels that need investigation.
If deployment involves new entities, group-aware evaluation keeps each entity entirely within one split. For predictions about the future, time ordering can impose another constraint. Scikit-learn describes grouped and time-aware validation. The grouping key must reflect how the data were collected and what the evaluation is intended to estimate.
Design a comparison
Keep the group split and predictor fixed. Choose A, B, C as the held-out entities instead. Predict the accuracy from their labels before running. Select the D, E, F group run as your baseline. Why can the score change even with no entity overlap?
For transfer, imagine sessions from each entity have slightly different signatures. Would checking exact duplicate rows alone detect shared entities? Save your answer and report.
Check the transfer reasoning
A, B, C have labels 0, 1, 0, so the fallback gets four of six records right: about 66.67%. The evaluation population changed. With varying signatures, exact-duplicate checks may find nothing even when entities cross splits. Group identity and collection provenance remain relevant.
Continue
Previous lesson · Next: Can zero correlation hide a predictable pattern? · All statistics lessons