Which rows are allowed to fit a scaler?

Before training, you subtract a column’s mean and divide by its standard deviation. Should those statistics describe every row you have, or only the training rows?

Our fixed split has training inputs 2, 4, 6, 8 and validation inputs 10, 12. There is no target or fitted predictor: we are inspecting preprocessing itself.

Establish a training-only reference

Use Training rows only and validation shift 0. Predict the fitted mean and whether the transformed training and validation means will both be zero. Record the reference.

Change only Fit mean and scale on to All rows (leakage). Keep the split and input values fixed. Predict whether the representation of the unchanged training records will move.

Training inputs 2, 4, 6, 8; validation inputs 10, 12 plus the chosen shift. Transform both splits with the same fitted mean and standard deviation. The scaler uses variance divided by n, matching StandardScaler; no model is fitted.

Fit mean and scale onTraining rows onlyAll rows (leakage)Validation-only shift0102040

Prediction and reason

Record runReference settings

0/16 runs. Each run keeps its own inputs and results. Work stays in this tab across reloads; download a report before closing it.

Write a prediction to record your reference.

Observations: cite runs and valuesExplanation: what accounts for the change?Your next test, prediction, and a result that would challenge it

Download report

Compare the fitted statistics and transformed training values. Explain where information from the validation records entered the calculation.

Follow the information into the scaler

Training-only fitting gives mean 5 and standard deviation $\sqrt5\approx2.2361$. The transformed training mean is zero. The validation mean becomes $(11-5)/\sqrt5\approx2.6833$; held-out values need not have transformed mean zero.

Fitting on all six rows instead gives mean 7 and standard deviation $\sqrt{35/3}\approx3.4157$. Even the unchanged training value 2 gets a different representation, because validation inputs influenced the fitted transformation.

Separate fitting from transforming

The transformation is $z=(x-\mu)/\sigma$. Learn $\mu$ and $\sigma$ on the training portion, then reuse them for validation and future inputs. Here the scaler’s variance divides by $n$, matching StandardScaler’s convention. The earlier descriptive sample standard deviation divided by $n-1$; these are explicitly different conventions.

Fitting preprocessing on held-out data lets information cross the evaluation boundary. The same issue applies to learned imputation values and feature selection. In cross-validation, fit those steps separately inside each training fold; pipelines help preserve that separation. Scikit-learn documents how to prevent preprocessing leakage.

This demonstration detects leakage in the transformation. It does not claim a particular score improvement: no predictor is evaluated.

Design a leakage test

Choose a nonzero validation-only shift. First keep training-only fitting fixed and compare with its shift-0 run. Predict whether the fitted mean, fitted scale, and transformed training values will change.

Then select all-row fitting and compare with its shift-0 run. Does changing only validation data alter the training representation? Record a conclusion that distinguishes the input intervention from the fitting-policy intervention.

Check the invariance test

Training-only fitted statistics and training representations remain unchanged. Validation representations move because their inputs moved. With all-row fitting, changing validation values changes the fitted statistics and therefore the training representation. This is direct evidence that the scaler consumed information across the split.

For transfer, propose where to fit a missing-value imputer in a training/validation workflow. Download your report, then inspect the fixed split in the learning-rate experiment. Its browser model uses raw inputs; a notebook investigation could add training-fitted scaling and compare matched learning rates.

Continue

Previous lesson · Next: Can 95% accuracy mean complete failure? · All statistics lessons