The effect of hyperparameters on training

The same neural network can learn quickly, make almost no progress, or become unstable depending on how we train it. What changes when we adjust the learning rate, batch size, depth, or activation function? We can investigate by changing one setting at a time and watching the training curves.

In the previous article, we built a neural network in NumPy and PyTorch. Here we use the same baseline: one hidden layer of 128 neurons, a learning rate of 0.1, and a batch size of 32.

These choices are hyperparameters: settings we choose rather than parameters the model learns. We will vary them one at a time to see how they affect training. The results provide starting points for this model and dataset; other tasks may need different settings.

Each experiment below has an interactive chart. You can click the labels to show or hide individual runs, switch between loss and accuracy views, toggle “show baseline” to include the initial random-weights measurement (epoch 0), and expand the “Computation log” section to see the raw training output. The charts show training metrics by default — our analysis is based on training values — but you can click “val” to overlay validation metrics for comparison.

We reuse the MNIST data preparation, PyTorch model, and training loop from the previous article. Each downloadable notebook includes the complete setup and helper functions used in the snippets below.

Learning rate

When we were looking at how networks learn, we saw how the learning rate controls the step size of gradient descent. Too small and training goes very slowly; too large and it overshoots the minimum, making it harder to converge toward a low loss. We demonstrated this on a 2-parameter model with an interactive widget. Now let’s see the exact same phenomenon on a real network with 100,000+ parameters.

We will compare five learning rates over ten epochs:

for lr in [0.001, 0.01, 0.1, 1.0, 10.0]:
    model = make_model()
    history = train_torch(model, X_train, y_train, epochs=10, lr=lr,
                          include_initial=True)

Let’s see how the loss evolves per epoch for each learning rate:

Turn on “show baseline” to see the measurements before training. The runs start near a loss of 2.3, with some variation from random initialization. Assigning exactly 0.1 probability to each of ten classes would give $-\log(0.1) \approx 2.3$; random weights do not guarantee an exactly uniform distribution.

At lr = 10.0, training fails in this run. Loss jumps to 12.2 in the first epoch and stays around 2.5 afterward, while training accuracy remains near 10%. These metrics show that the model is not learning useful digit distinctions; they do not by themselves tell us whether every image receives the same prediction.

Let’s toggle off lr=10.0 by clicking its label — because its spike to 12.2 compresses the y-axis and makes it hard to see the detail in the other curves. Hiding that curve makes the four remaining runs easier to compare:

• lr = 0.001 — loss reaches 0.41 after ten epochs, still above the 0.33 reached by lr=0.1 after one epoch. Learning is slow at this step size.
• lr = 0.01 — loss reaches 0.18 and is still falling. More training may help, but it need not reach the same solution as lr=0.1.
• lr = 0.1 — the lowest final training loss among these runs, about 0.03. Most of the initial improvement happens in the first two or three epochs.
• lr = 1.0 — training loss reaches about 0.15, while validation loss fluctuates around 0.20–0.24 in the final epochs. Turn on the validation curves to see that validation accuracy also declines near the end.

So far, we’ve used plain SGD and changed only the learning rate. We can also change how updates are calculated and controlled: vary the rate during training, use gradient history, adapt updates for each parameter, shrink weights, or limit large gradients. Here’s a brief overview of these techniques and how they fit together.

Learning-rate schedules. Our MNIST runs above use a fixed learning rate, but a schedule can change it at every optimizer step, even within a single epoch. For example, a one-epoch run with 10,000 updates could warm up to a peak of 0.003 over the first 100 steps (1%), then decay linearly to zero over the remaining 9,900. Warmup limits early updates; decay allows finer adjustments later. In a scheduled learning-rate sweep, the values being compared may be different peak rates.

Momentum. SGD with momentum carries information from previous gradients into each update. When gradients consistently agree, their contributions build up; when they oppose each other, they can partially cancel, reducing oscillations. Momentum works at every optimizer step and can be combined with a learning-rate schedule.

Adam. Adam tracks moving averages of both gradients and squared gradients for each parameter. The first provides momentum-like history; the second scales updates according to recent gradient magnitudes. The squared-gradient average is a second raw moment, not a variance. Adam still needs a global learning rate, which can follow the same warmup-and-decay schedule.

Weight decay. AdamW adds a separate shrinkage of selected weights toward zero, acting as regularization that can improve generalization. Its strength is controlled by a weight-decay coefficient; embeddings, normalization parameters, and biases can be excluded depending on the training setup. This shrinkage is part of each optimizer update.

Gradient clipping. Global norm clipping rescales the combined gradient vector when its length exceeds a chosen threshold, preserving its direction. For example, clipping $(3,4)$ to norm 1 gives $(0.6,0.8)$. It happens after backpropagation and before the optimizer updates its state and weights. Clipping limits gradients; weight decay shrinks weights. With momentum or Adam, clipping does not directly bound the final parameter update.

Batch size

We will compare batch sizes of 1, 32, 256, and 60,000. With 20% held out for validation, only 48,000 images are used for training, so batch_size=60000 puts those 48,000 images into a single batch.

for bs in [1, 32, 256, 60000]:
    model = make_model()
    history = train_torch(model, X_train, y_train, epochs=10, batch_size=bs,
                          include_initial=True)

Let’s see how the loss evolves per epoch for each batch size:

The middle ground — bs=32 and bs=256 — both work well, with bs=32 converging faster:

• bs = 32 — loss drops sharply from 2.4 to 0.03 by epoch 10. With 1,500 batches per epoch, the model gets frequent updates with gradients that are noisy but roughly correct.
• bs = 256 — slower but steady. The loss drops to 0.17 after 10 epochs and the curve is still going down. Each epoch has only 188 batches (vs 1,500 for bs=32), so fewer updates — but each update is based on a more reliable gradient average.

The smallest and largest batches behave differently:

• bs = 1 — unstable with lr=0.1. Each gradient comes from one image, and large updates can undo previous progress. Averaging over more images reduces this variation.
• bs = 60000 — loss falls from about 2.4 to 1.6 after ten epochs. Each epoch makes only one update, so ten epochs give ten gradient steps, compared with 15,000 at bs=32. Using the full training gradient does not compensate for so few updates at this learning rate.

Batch size 32 worked well in this experiment. Larger batches use more memory for activations and can improve hardware throughput, but a faster epoch does not necessarily mean less time to reach the same validation accuracy. Compare both training time and validation quality when changing batch size.

Tune batch size and learning rate together. In the bs=1 run, reducing lr from 0.1 to 0.001 brings loss down to 0.09. Larger batches may benefit from a larger learning rate, but the relationship is not a universal scaling rule.

Network depth and width

Our baseline network has one hidden layer of 128 neurons. What happens when we go wider, deeper, or both? Let’s find out — we’ll train five variations with everything else held constant (lr=0.1, bs=32, 10 epochs):

configs = {
    "narrow (32)":  [32],
    "baseline (128)": [128],
    "wide (512)":   [512],
    "deep (2×128)": [128, 128],
    "deep (3×128)": [128, 128, 128],
}

for name, hidden_sizes in configs.items():
    model = make_model(hidden_sizes)
    history = train_torch(model, X_train, y_train, epochs=10,
                          include_initial=True)

Let’s see how the loss evolves for each architecture:

A few things stand out:

• Width helps: “width” here means the number of neurons in a single hidden layer — going from 32 to 128 to 512 neurons steadily lowers the loss. A wider layer has more parameters to detect patterns. But 512 has 4× the parameters of 128 for only a small improvement in loss — diminishing returns.
• Depth helps too: adding a second hidden layer (2×128) reaches a lower loss than the single-layer baseline, even though the total parameter count is similar. Deeper networks can learn hierarchical features — the first layer might detect edges, the second layer might combine edges into shapes.
• A third layer did not improve this run: its final loss is close to the baseline, and its validation loss fluctuates more. These curves alone do not identify the cause or establish that gradients are vanishing.

For this MNIST task, one or two hidden layers and widths of 128–512 are useful starting points. Compare validation performance before adding capacity. Layer type also matters: convolutional layers can use the spatial structure of images, which our fully connected model ignores.

Activation functions: sigmoid vs ReLU

We will compare sigmoid and ReLU to examine the gradient behavior discussed in the theory article. Sigmoid contributes a derivative of at most 0.25 at each layer; ReLU contributes 1 for active neurons and 0 for inactive ones. These factors affect how gradients propagate through a deep network.

Choose nn.ReLU or nn.Sigmoid in our model factory. We compare networks with 1, 3, and 5 hidden layers:

for n_layers in [1, 3, 5]:
    for activation in [nn.ReLU, nn.Sigmoid]:
        model = make_model([128] * n_layers, activation)
        history = train_torch(model, X_train, y_train, epochs=10,
                              include_initial=True)

Let’s see how the loss evolves for each combination of depth and activation function:

In these runs, sigmoid becomes harder to train as depth increases:

• 1 hidden layer — both activations work well. ReLU’s loss drops to 0.03 by epoch 10, sigmoid’s to 0.15. With only one layer, the gradient passes through a single activation, so sigmoid’s gradient shrinking doesn’t matter much.
• 3 hidden layers — sigmoid starts to fall behind. ReLU’s loss reaches 0.016, sigmoid’s only gets to 0.14 — nearly 10x higher. Sigmoid is noticeably slower in the first few epochs — its loss at epoch 3 is still higher than where ReLU was after epoch 1.
• 5 hidden layers — sigmoid completely fails. The loss barely moves from its initial ~2.3 value across all 10 epochs — the model has learned essentially nothing. Meanwhile, ReLU with 5 layers reaches a loss of 0.024 — virtually identical to 1 and 3 layers.

These results are consistent with vanishing gradients. Across five sigmoid layers, the activation derivatives alone contribute a factor of at most $0.25^5 \approx 0.001$. Weight matrices also affect the full gradient, so this is not an exact estimate of the gradient reaching the first layer.

ReLU does not shrink the gradient at the activation step for active neurons, which helps explain its results here. It does not guarantee that gradients remain stable throughout the network.

ReLU is a useful starting point for this model. Leaky ReLU and ELU allow nonzero gradients for negative inputs and may help when neurons remain inactive. Other architectures use different activations, including GELU in some transformers. In a standard LSTM, sigmoid controls gates, while tanh is used for candidate values and the cell-state output transformation.

Number of epochs

In all the experiments above, we trained for 10 epochs. But how many epochs should you actually train for? Let’s find out by training our best configuration (lr=0.1, batch size 32) for 50 epochs and watching what happens:

model = make_model()
history = train_torch(model, X_train, y_train, epochs=50, include_initial=True)

Training loss falls from 0.33 in epoch 1 to 0.0015 in epoch 50. Reported training accuracy reaches 100% by epoch 29. That shows an excellent fit to the training set, but does not establish how well the model generalizes.

Validation loss improves from 0.19 at epoch 1 to 0.075 around epoch 13, then stops improving and starts slowly increasing — 0.078 at epoch 20, 0.081 at epoch 30, 0.085 at epoch 50. Meanwhile, validation accuracy plateaus around 98% from epoch 13 onward and barely moves for the remaining 37 epochs.

The rising validation loss is a sign of overfitting. Training loss continues to improve while validation accuracy stays nearly flat and validation loss worsens. The model may be becoming more confident in its remaining mistakes on unseen data.

Too few epochs can leave a model underfitted; further training can eventually lead to overfitting. The stopping point depends on the model, data, and other hyperparameters, so we monitor validation performance during training.

For early stopping, track validation loss in the training loop. Save a copy of the best weights and stop after five consecutive epochs without improvement:

from copy import deepcopy

model = make_model()
optimizer = torch.optim.SGD(model.parameters(), lr=0.1)
split = int(0.8 * len(X_train))
best_loss, stale_epochs = float("inf"), 0
best_weights = deepcopy(model.state_dict())
for epoch in range(100):
    run_epoch(model, X_train[:split], y_train[:split], optimizer=optimizer)
    val_loss, _ = run_epoch(model, X_train[split:], y_train[split:])
    if val_loss < best_loss:
        best_loss, stale_epochs = val_loss, 0
        best_weights = deepcopy(model.state_dict())
    else:
        stale_epochs += 1
        if stale_epochs >= 5:
            break
model.load_state_dict(best_weights)

The limit is 100 epochs. deepcopy(model.state_dict()) preserves the best weights independently of subsequent updates; load_state_dict restores them when training ends. The patience here is five epochs.

We’ll explore overfitting and early stopping in more depth in upcoming articles.

Automated hyperparameter search

In this article we tuned hyperparameters manually — changing one at a time and observing the effect. This builds intuition, but it doesn’t scale. When you have dozens of hyperparameters and thousands of possible combinations, you need a strategy for searching, and a tool to automate it.

Evaluating one configuration may require a complete training run; our learning-rate experiment did this five times. Search strategies differ in how they choose the next configuration and use the available evaluation budget:

• Grid search. Choose a finite set of values for each hyperparameter and evaluate every combination. Three parameters with five values each require 125 runs. This is practical for small search spaces and inexpensive evaluations. See scikit-learn’s guide and examples.
• Random search. Sample combinations from chosen ranges or distributions. When only a few parameters strongly affect the result, random search can explore more distinct values for those parameters than a grid with the same budget. See Bergstra & Bengio, 2012.
• Coordinate descent. Start with a configuration, vary one parameter at a time, and keep improvements. Repeat the process so earlier choices can be reconsidered after other parameters change. This avoids evaluating the full grid but can get stuck where no single change helps, even though changing several parameters together would. For the underlying method and its convergence conditions, see Ryan Tibshirani’s lecture slides (PDF).
• Bayesian optimization. Fit a probabilistic model to previous configurations and scores, then use it to choose the next evaluation, balancing exploration against promising results. It can reduce the number of costly evaluations, but its effectiveness depends on the search space and model. For a thorough introduction, see Roman Garnett’s freely available book, Bayesian Optimization.

Choose a search strategy based on evaluation cost, the search space, and the available budget. Random search provides a useful baseline; local or model-based search can help when evaluations are expensive. For example, the AgentDiet paper tuned four hyperparameters on 100 agent tasks by varying one parameter at a time. The authors updated two settings after the first round and found no further improvement in the second.

Weights & Biases Sweeps and Optuna can organize hyperparameter searches around a PyTorch training loop and compare validation results across configurations.

Designing how a model learns

Training a model means choosing an entire system for how it learns. Our MNIST experiments explored a few of these choices, and the optimizer overview introduced others. Larger training projects extend the same idea to data preparation, architecture, and evaluation:

Some entries are numerical hyperparameters; others are broader experimental choices. Together, they determine what a training run does and how we judge its result. The evaluation protocol should reflect the intended task and remain consistent when comparing configurations.

These choices interact. We saw that batch size 1 worked much better after reducing the learning rate. Likewise, changing depth or activation functions may call for a different initialization or optimizer configuration. A setting that works well for one setup may work poorly after another part changes.

What this means for experiments

Suppose a new architecture performs worse than our baseline. Is the architecture less suitable for the task, or have we given it settings that work well only for the baseline? Holding settings fixed tells us what happens under those settings. Comparing what each architecture can achieve requires giving both a fair opportunity to tune.

This is why experimental design matters:

• Use controlled experiments and ablations. State the hypothesis, keep a reference configuration, and change or remove the component being studied. Test combinations when the hypothesis concerns an interaction.
• Give methods comparable tuning budgets. Report search ranges, trial counts, and compute limits so readers can assess the comparison.
• Repeat runs with multiple random seeds. Initialization and data order can change the outcome. Report the spread of results alongside their average.
• Quantify uncertainty. Use statistical analysis appropriate to the experiment, and check whether an apparent improvement is large relative to run-to-run variation.
• Report compute costs. Include the search effort as well as the final model’s training cost and score.

These concerns are examined in Accounting for Variance in Machine Learning Benchmarks, which studies how data sampling, initialization, and hyperparameter choices affect comparisons. Because repeated tuning can also overfit the validation set, reserve a separate test set for the final evaluation after the choices are fixed.

You do not need to tune everything simultaneously. Start from an established baseline, keep most settings fixed, and investigate the variables relevant to your hypothesis. Revisit related settings when the evidence suggests an interaction. The goal is to establish whether your intervention caused a reliable improvement, and what it cost to obtain it.

Learning from experiments across model sizes

So far, each sweep has answered a local question: which setting works best for this model under this training budget? Answering that question gets more expensive as the model grows. Testing ten learning rates at a fixed training budget requires ten training runs, and an unsuitable rate can spend that budget on slow progress or unstable training. Repeating a broad search from scratch at every new model size can consume resources we wanted to spend on the final model.

The opportunity is to use cheaper, smaller-model experiments to narrow the expensive search. If good settings change predictably with scale, a fitted rule can suggest a promising learning-rate range before we launch the larger runs. We can spend the remaining tuning budget checking that range and testing whether the prediction transfers. The savings depend on the rule being accurate enough; an untested extrapolation can waste compute too.

This makes each small-model sweep useful beyond the model it tuned. Results from several sizes become a dataset for learning how the settings should change with scale. The next question is whether that relationship predicts useful settings at a size we held out of the fit.

Two papers illustrate how these patterns can help us plan training and choose hyperparameters.

Scaling Laws for Neural Language Models examines how language-model loss changes with model size, dataset size, and training compute. The study found smooth, predictable trends across large ranges of scale: increasing resources improved performance, but the benefit depended on whether another resource had become the bottleneck.

The practical finding is that smaller experiments can help estimate what additional training resources might buy us. This gives us evidence for deciding whether to increase model size, provide more data, or spend more compute before committing to a larger run. The paper’s estimates apply to its experimental conditions; they are a starting point for planning, rather than a universal recipe.

StepFun’s Hyperparameter Scaling Law in Large Language Model focuses on the settings used to train those models. Across its experiments, the best peak learning rate depended on both model size and training-token budget, while the best batch size depended mainly on the token budget.

At a fixed token budget, its fitted rule recommended smaller learning rates for larger models. At a fixed model size, larger token budgets called for larger peak learning rates and batch sizes. The authors tested their predictions at a held-out scale and across different model structures and data recipes.

The practical finding is that we may be able to narrow a new model’s hyperparameter search using results from earlier experiments. Instead of restarting a broad sweep at every size, we can test a smaller range around a predicted setting. That still requires checking whether the prediction transfers to our architecture, optimizer, data, and schedule.

Together, these papers motivate treating training experiments as reusable evidence: they can inform both the resources we allocate and the settings we try. We do not need their formulas to apply the habit—record the conditions and outcomes of each sweep, look for patterns across runs, and test those patterns on a setup that was not used to discover them.

Is an apparent improvement reliable?

The same care applies when comparing architectures. Suppose a baseline finishes with validation loss 2.93 and a variant without positional encoding finishes at 3.07. These are illustrative single-run outcomes: the observed gap is 0.14. We still need to learn whether that gap persists across runs and whether both configurations received suitable tuning.

Initialization, training-example order, and stochastic operations can change final loss. Repeating runs measures that variability. Our earlier notebooks fix the seed at 42; repeating them unchanged does not create independent seed trials. Make initialization and data-order seeds configurable, then record the seeds and all outcomes for each comparison.

First decide which question the experiment should answer:

Both are useful. They can produce different rankings because removing a component may change the best learning rate. Define the resource constraint too: equal steps, equal tokens, and equal training compute are different comparisons.

After tuning, freeze the configurations and evaluate fresh runs. Report the number of runs, average loss, spread, and an uncertainty interval for the difference. With deliberately matched runs, such as runs using the same training-example order, subtract the baseline’s loss from the variant’s loss for each pair, then average those differences. A positive average means the baseline had lower loss; a negative average means the variant did.

A paired analysis measures how much these differences vary across pairs. Independently collected runs need an analysis suited to independent samples. Merely giving different architectures the same seed integer does not guarantee corresponding initial weights or matched randomness. Training runs are the replicates here, not the many checkpoints along one run.

A confidence interval shows the range of mean differences compatible with the data under the chosen procedure. If it includes zero, the experiment may be too uncertain to establish a direction; this does not prove equivalence. If it excludes zero, also ask whether the effect is large enough to matter for the task and cost.

A hypothesis test can instead ask whether the results are consistent with the baseline and variant having the same average loss across repeated runs. That assumption is the null hypothesis. The p-value measures how likely a result at least as extreme as the observed one would be if that assumption and the test’s other assumptions held. A common cutoff is a p-value below 0.05, but this does not mean there is less than a 5% chance that the null hypothesis is true, nor does it tell us whether the improvement matters in practice. Choose the analysis before inspecting the results, and account for testing multiple comparisons when evaluating many variants.

Accounting for Variance in Machine Learning Benchmarks studies why initialization, data sampling, and hyperparameter selection all matter to these claims. Repeating only the winning configuration estimates variability conditional on that selection; it does not include the uncertainty of the search itself. Reserve untouched evaluation data for the final comparison, and repeat the search when the claim concerns the reliability of an entire tuning procedure.

Repeating seeds on one dataset also does not establish transfer to other datasets or distributions. The useful outcome of these experiments is a bounded claim: a scaling rule predicted good settings at held-out sizes, or an intervention improved expected loss under a stated protocol, with measured uncertainty and cost.