When does a larger learning rate stop helping?

Read the gradient-descent article and the learning-rate experiments for the background. Here we train a line on a small synthetic dataset and isolate the effect of step size. You need basic algebra and the idea that a gradient gives a local direction of increasing loss.

Inspect the data

Open Explore dataset in a new tab. Each of the 300 records has an input $x$, a target $y$, and a fixed training or validation assignment. Sort the inputs and inspect several targets. Do similar inputs always have similar targets? Would one straight line pass through every record?

We generated inputs uniformly between −2 and 2, then used $y=2x+1+\text{noise}$, with uniform noise between −0.6 and 0.6. Seed 42 fixes both the records and their shuffled split: 240 training records and 60 validation records. Values are stored to six decimal places. No preprocessing is applied. The identifier and split are metadata, not model inputs.

Our model predicts $\hat y=wx+b$: it learns a slope $w$ and an intercept $b$. Each update averages gradients over all 240 training records. Validation records never contribute to updates. For either split, the reported loss is half its mean squared prediction error:

$$L=\frac{1}{2n}\sum_{i=1}^{n}(wx_i+b-y_i)^2.$$

Establish a reference

Keep the starting slope at $-2$, the learning rate at $0.25$, and the budget at 20 updates. The intercept always starts at zero. Predict whether training and validation loss will decrease, and whether either will reach zero. Write your reason in the bench, then select Run experiment.

Record initial and final losses and at least one intermediate slope and intercept using Inspect all calculated values for this run. This is your reference. Every subsequent run starts afresh with the same records and split; it does not continue training from the preceding run.

Change only the learning rate

Keep the starting slope and 20-update budget fixed. Predict and run learning rates $0.01$, $1$, $1.5$, and $2$, one at a time. Compare both loss curves and the parameter values against the reference. Which rates improve slowly, settle quickly, or grow unstable? Does a slope that crosses its eventual value necessarily indicate failure?

Experiment bench

Load reference settings

300 synthetic observations: y = 2x + 1 + uniform noise in [−0.6, 0.6), seed 42. Fixed split: 240 training / 60 validation. Model ŷ = wx + b; initial b = 0. Full-batch gradient descent uses training rows only. Loss = ½ mean squared error. Every run starts afresh; no preprocessing or resampling.

Load the reference, write a prediction, and run it first. For each comparison, change only the condition named in the lesson. Each run is calculated here in your browser.

Learning rate0.010.050.10.250.511.251.52Starting slope-4-2034Number of updates5102040

Prediction and reason for this run

Run experimentReference settings selected · 0/16 runs

Loading reference settings preserves your evidence. Work is saved in this tab across navigation and reloads. Download your report to keep it after closing the tab.

No run recorded yet.

Build your explanation from the evidence

Observations: which runs and values are you comparing?Interpretation: what explains the result?Limitations: what does this experiment leave unresolved?Next test: what will you change, and what result would challenge your explanation?

Download reportClear runs

Write your explanation before opening the comparison. Blue solid shows training loss; orange dashed shows validation loss. The vertical axis is $\log_{10}(1+L)$. Each run has its own scale, so compare numerical losses rather than heights across different plots.

Compare the outcomes and explain the pattern

After 20 updates, the reference has slope about $1.98264$, intercept $1.01841$, training loss $0.06156$, and validation loss $0.06321$. Rate 0.01 still has training loss about 6.436; rate 1 is already near the best training fit. Rate 1.5 oscillates while converging, whereas rate 2 grows rapidly, reaching training loss about $3.26\times10^8$ within this budget.

Both parameters use the same pre-update errors $e_i=wx_i+b-y_i$:

$$w_{\text{next}}=w-\eta\frac{1}{240}\sum_{i\in\text{train}}e_ix_i, \qquad b_{\text{next}}=b-\eta\frac{1}{240}\sum_{i\in\text{train}}e_i.$$

The best training line has slope about $1.98720$ and intercept $1.01418$. It differs slightly from the generating line because we fitted a finite noisy sample. Its training loss is about $0.06154$, not zero. A larger rate multiplies both gradient components; whether the resulting step helps depends on the inputs and the shape of the loss.

Validation loss measures prediction error on the 60 withheld records. It is not used in the gradients. Once we use those validation results to choose a rate, they are no longer an unbiased final test of our selection. This experiment has no separate test set and does not reproduce the articles’ MNIST measurements.

Design your own investigation

Choose a rate and two available starting slopes. Keep the budget identical. Predict which parts of the trajectory will change, then test your explanation against both losses and the intermediate parameters.

For a separate budget comparison, keep rate 0.05 and starting slope −2 fixed, then compare 10 with 40 updates. Explain why this answers a different question from comparing rates after exactly 20 updates.

Use Comparison baseline to select a recorded run that differs in only the condition you are testing. In the editable notebook, you can also rescale every $x$ value in both splits inside the calculation. Record the scale, keep those transformed inputs fixed across runs, and investigate whether the same learning rate remains stable.

Report and transfer

Name the runs you compared, cite both losses, and distinguish observations from your explanation. State what remains uncertain after testing only one dataset and split. Download the report to keep the input records, settings, predictions, and every calculated step.

Without running, suppose we multiply the entire loss by a positive constant $c$, and use its matching gradient. How would you adjust the learning rate to reproduce exactly the same parameter trajectory?

Check the transfer argument

The gradient is multiplied by $c$. Dividing the learning rate by $c$ leaves each parameter update unchanged. The reported loss values are still multiplied by $c$. A numerical learning rate therefore depends on the loss definition; even replacing half-MSE with MSE changes the matching rate by a factor of two.

Optional one-parameter warm-up

For one input $x=1$, target $y=2$, and a fixed zero intercept, $L(w)=\tfrac12(w-2)^2$. Starting at −2, predict one update at rates 0.25, 1, and 2 before calculating $w_{\text{next}}=w-\eta(w-2)$.

The respective results are −1, 2, and 6. If $e=w-2$, then $e_{\text{next}}=(1-\eta)e$, so a nonzero error shrinks for $0<\eta<2$. This boundary belongs to this particular one-parameter loss. It is not the boundary for our two-parameter dataset experiment.

Continue experimenting

Next: What breaks when backpropagation forgets a path? · All machine learning experiments