The effect of hyperparameters on training

In the previous article, we built a neural network in NumPy, replicated it in Keras, and reached about 97% validation accuracy on MNIST. We used one hidden layer of 128 neurons, a learning rate of 0.1, and a batch size of 32.

These choices are hyperparameters: settings we choose rather than parameters the model learns. We will vary them one at a time to see how they affect training. The results provide starting points for this model and dataset; other tasks may need different settings.

Each experiment below has an interactive chart. You can click the labels to show or hide individual runs, switch between loss and accuracy views, toggle “show baseline” to include the initial random-weights measurement (epoch 0), and expand the “Computation log” section to see the raw training output. The charts show training metrics by default — our analysis is based on training values — but you can click “val” to overlay validation metrics for comparison. Code blocks with a marimo icon (visible on hover) link to an interactive notebook where you can run the experiment yourself and modify the code.

We’ll use the same dataset and model setup from the previous article:

import keras
import numpy as np

# Training: 60000 images, Test: 10000 images
(train_images, train_labels), (test_images, test_labels) = keras.datasets.mnist.load_data()

# Flatten 28×28 → 784 and normalize to [0, 1]
X_train = train_images.reshape(-1, 784).astype("float32") / 255.0
X_test = test_images.reshape(-1, 784).astype("float32") / 255.0

y_train = train_labels
y_test = test_labels

Learning rate

When we were looking at how networks learn, we saw how the learning rate controls the step size of gradient descent. Too small and training goes very slowly; too large and it overshoots the minimum, making it harder to converge toward a low loss. We demonstrated this on a 2-parameter model with an interactive widget. Now let’s see the exact same phenomenon on a real network with 100,000+ parameters.

We will compare five learning rates over ten epochs:

for lr in [0.001, 0.01, 0.1, 1.0, 10.0]:
    model = keras.Sequential([
        keras.layers.Dense(128, activation="relu", input_shape=(784,)),
        keras.layers.Dense(10, activation="softmax"),
    ])

    model.compile(
        optimizer=keras.optimizers.SGD(learning_rate=lr),
        loss="sparse_categorical_crossentropy",
        metrics=["accuracy"],
    )

    model.fit(X_train, y_train, epochs=10, batch_size=32,
              validation_split=0.2, verbose=2)

Let’s see how the loss evolves per epoch for each learning rate:

Turn on “show baseline” to see the measurements before training. The runs start near a loss of 2.3, with some variation from random initialization. Assigning exactly 0.1 probability to each of ten classes would give $-\log(0.1) \approx 2.3$; random weights do not guarantee an exactly uniform distribution.

At lr = 10.0, training fails in this run. Loss jumps to 12.2 in the first epoch and stays around 2.5 afterward, while training accuracy remains near 10%. These metrics show that the model is not learning useful digit distinctions; they do not by themselves tell us whether every image receives the same prediction.

Let’s toggle off lr=10.0 by clicking its label — because its spike to 12.2 compresses the y-axis and makes it hard to see the detail in the other curves. Hiding that curve makes the four remaining runs easier to compare:

• lr = 0.001 — loss reaches 0.41 after ten epochs, still above the 0.33 reached by lr=0.1 after one epoch. Learning is slow at this step size.
• lr = 0.01 — loss reaches 0.18 and is still falling. More training may help, but it need not reach the same solution as lr=0.1.
• lr = 0.1 — the lowest final training loss among these runs, about 0.03. Most of the initial improvement happens in the first two or three epochs.
• lr = 1.0 — training loss reaches about 0.15, while validation loss fluctuates around 0.20–0.24 in the final epochs. Turn on the validation curves to see that validation accuracy also declines near the end.

Learning rate also depends on the optimizer. Adam uses running estimates of gradients and their squares to scale updates for each parameter, but its learning rate still needs tuning. We will examine optimizers, learning-rate schedules, and warmup in the next article.

Batch size

We will compare batch sizes of 1, 32, 256, and 60,000. With 20% held out for validation, only 48,000 images are used for training, so batch_size=60000 puts those 48,000 images into a single batch.

for bs in [1, 32, 256, 60000]:
    model = keras.Sequential([
        keras.layers.Dense(128, activation="relu", input_shape=(784,)),
        keras.layers.Dense(10, activation="softmax"),
    ])

    model.compile(
        optimizer=keras.optimizers.SGD(learning_rate=0.1),
        loss="sparse_categorical_crossentropy",
        metrics=["accuracy"],
    )

    model.fit(X_train, y_train, epochs=10, batch_size=bs,
              validation_split=0.2, verbose=2)

Let’s see how the loss evolves per epoch for each batch size:

The middle ground — bs=32 and bs=256 — both work well, with bs=32 converging faster:

• bs = 32 — loss drops sharply from 2.4 to 0.03 by epoch 10. With 1,500 batches per epoch, the model gets frequent updates with gradients that are noisy but roughly correct.
• bs = 256 — slower but steady. The loss drops to 0.17 after 10 epochs and the curve is still going down. Each epoch has only 188 batches (vs 1,500 for bs=32), so fewer updates — but each update is based on a more reliable gradient average.

The smallest and largest batches behave differently:

• bs = 1 — unstable with lr=0.1. Each gradient comes from one image, and large updates can undo previous progress. Averaging over more images reduces this variation.
• bs = 60000 — loss falls from about 2.4 to 1.6 after ten epochs. Each epoch makes only one update, so ten epochs give ten gradient steps, compared with 15,000 at bs=32. Using the full training gradient does not compensate for so few updates at this learning rate.

Batch size 32 worked well in this experiment. Larger batches use more memory for activations and can improve hardware throughput, but a faster epoch does not necessarily mean less time to reach the same validation accuracy. Compare both training time and validation quality when changing batch size.

Tune batch size and learning rate together. In the bs=1 run, reducing lr from 0.1 to 0.001 brings loss down to 0.09. Larger batches may benefit from a larger learning rate, but the relationship is not a universal scaling rule.

Network depth and width

Our baseline network has one hidden layer of 128 neurons. What happens when we go wider, deeper, or both? Let’s find out — we’ll train five variations with everything else held constant (lr=0.1, bs=32, 10 epochs):

configs = {
    "narrow (32)":  [32],
    "baseline (128)": [128],
    "wide (512)":   [512],
    "deep (2×128)": [128, 128],
    "deep (3×128)": [128, 128, 128],
}

for name, hidden_sizes in configs.items():
    model = keras.Sequential()
    model.add(keras.layers.Dense(hidden_sizes[0], activation="relu", input_shape=(784,)))
    for size in hidden_sizes[1:]:
        model.add(keras.layers.Dense(size, activation="relu"))
    model.add(keras.layers.Dense(10, activation="softmax"))

    model.compile(
        optimizer=keras.optimizers.SGD(learning_rate=0.1),
        loss="sparse_categorical_crossentropy",
        metrics=["accuracy"],
    )

    model.fit(X_train, y_train, epochs=10, batch_size=32,
              validation_split=0.2, verbose=2)

Let’s see how the loss evolves for each architecture:

A few things stand out:

• Width helps: “width” here means the number of neurons in a single hidden layer — going from 32 to 128 to 512 neurons steadily lowers the loss. A wider layer has more parameters to detect patterns. But 512 has 4× the parameters of 128 for only a small improvement in loss — diminishing returns.
• Depth helps too: adding a second hidden layer (2×128) reaches a lower loss than the single-layer baseline, even though the total parameter count is similar. Deeper networks can learn hierarchical features — the first layer might detect edges, the second layer might combine edges into shapes.
• A third layer did not improve this run: its final loss is close to the baseline, and its validation loss fluctuates more. These curves alone do not identify the cause or establish that gradients are vanishing.

For this MNIST task, one or two hidden layers and widths of 128–512 are useful starting points. Compare validation performance before adding capacity. Layer type also matters: convolutional layers can use the spatial structure of images, which our fully connected model ignores.

Activation functions: sigmoid vs ReLU

We will compare sigmoid and ReLU to examine the gradient behavior discussed in the theory article. Sigmoid contributes a derivative of at most 0.25 at each layer; ReLU contributes 1 for active neurons and 0 for inactive ones. These factors affect how gradients propagate through a deep network.

In Keras, switching activation functions is just changing a string — "relu" vs "sigmoid". We’ll train networks with 1, 3, and 5 hidden layers to see how depth interacts with the choice of activation:

for n_layers in [1, 3, 5]:
    for activation in ["relu", "sigmoid"]:
        model = keras.Sequential()
        model.add(keras.layers.Dense(128, activation=activation, input_shape=(784,)))
        for _ in range(n_layers - 1):
            model.add(keras.layers.Dense(128, activation=activation))
        model.add(keras.layers.Dense(10, activation="softmax"))

        model.compile(
            optimizer=keras.optimizers.SGD(learning_rate=0.1),
            loss="sparse_categorical_crossentropy",
            metrics=["accuracy"],
        )

        model.fit(X_train, y_train, epochs=10, batch_size=32,
                  validation_split=0.2, verbose=2)

Let’s see how the loss evolves for each combination of depth and activation function:

In these runs, sigmoid becomes harder to train as depth increases:

• 1 hidden layer — both activations work well. ReLU’s loss drops to 0.03 by epoch 10, sigmoid’s to 0.15. With only one layer, the gradient passes through a single activation, so sigmoid’s gradient shrinking doesn’t matter much.
• 3 hidden layers — sigmoid starts to fall behind. ReLU’s loss reaches 0.016, sigmoid’s only gets to 0.14 — nearly 10x higher. Sigmoid is noticeably slower in the first few epochs — its loss at epoch 3 is still higher than where ReLU was after epoch 1.
• 5 hidden layers — sigmoid completely fails. The loss barely moves from its initial ~2.3 value across all 10 epochs — the model has learned essentially nothing. Meanwhile, ReLU with 5 layers reaches a loss of 0.024 — virtually identical to 1 and 3 layers.

These results are consistent with vanishing gradients. Across five sigmoid layers, the activation derivatives alone contribute a factor of at most $0.25^5 \approx 0.001$. Weight matrices also affect the full gradient, so this is not an exact estimate of the gradient reaching the first layer.

ReLU does not shrink the gradient at the activation step for active neurons, which helps explain its results here. It does not guarantee that gradients remain stable throughout the network.

ReLU is a useful starting point for this model. Leaky ReLU and ELU allow nonzero gradients for negative inputs and may help when neurons remain inactive. Other architectures use different activations, including GELU in some transformers. In a standard LSTM, sigmoid controls gates, while tanh is used for candidate values and the cell-state output transformation.

Number of epochs

In all the experiments above, we trained for 10 epochs. But how many epochs should you actually train for? Let’s find out by training our best configuration (lr=0.1, batch size 32) for 50 epochs and watching what happens:

model = keras.Sequential([
    keras.layers.Dense(128, activation="relu", input_shape=(784,)),
    keras.layers.Dense(10, activation="softmax"),
])

model.compile(
    optimizer=keras.optimizers.SGD(learning_rate=0.1),
    loss="sparse_categorical_crossentropy",
    metrics=["accuracy"],
)

history = model.fit(X_train, y_train, epochs=50, batch_size=32,
                    validation_split=0.2, verbose=2)

Training loss falls from 0.33 in epoch 1 to 0.0015 in epoch 50. Reported training accuracy reaches 100% by epoch 29. That shows an excellent fit to the training set, but does not establish how well the model generalizes.

Validation loss improves from 0.19 at epoch 1 to 0.075 around epoch 13, then stops improving and starts slowly increasing — 0.078 at epoch 20, 0.081 at epoch 30, 0.085 at epoch 50. Meanwhile, validation accuracy plateaus around 98% from epoch 13 onward and barely moves for the remaining 37 epochs.

The rising validation loss is a sign of overfitting. Training loss continues to improve while validation accuracy stays nearly flat and validation loss worsens. The model may be becoming more confident in its remaining mistakes on unseen data.

Too few epochs can leave a model underfitted; further training can eventually lead to overfitting. The stopping point depends on the model, data, and other hyperparameters, so we monitor validation performance during training.

In general, rather than guessing the right number of epochs, use early stopping — a Keras callback that monitors validation loss and stops training automatically when it stops improving:

early_stop = keras.callbacks.EarlyStopping(
    monitor="val_loss",
    patience=5,
    restore_best_weights=True,
)

model.fit(X_train, y_train, epochs=100, batch_size=32,
          validation_split=0.2,
          callbacks=[early_stop])

Set an upper limit such as 100 epochs. With patience=5, training stops after five consecutive epochs without improvement in validation loss. restore_best_weights=True restores the weights from the best validation epoch. You still choose the monitored metric, patience, and maximum training budget.

We’ll explore overfitting and early stopping in more depth in upcoming articles.

Automated hyperparameter search

In this article we tuned hyperparameters manually — changing one at a time and observing the effect. This builds intuition, but it doesn’t scale. When you have dozens of hyperparameters and thousands of possible combinations, you need a strategy for searching, and a tool to automate it.

Search strategies differ in how they choose the next configuration and use the available evaluation budget:

• Grid search. Choose a finite set of values for each hyperparameter and evaluate every combination. Three parameters with five values each require 125 runs. This is practical for small search spaces and inexpensive evaluations.
• Random search. Sample combinations from chosen ranges or distributions. When only a few parameters strongly affect the result, random search can explore more distinct values for those parameters than a grid with the same budget. See Bergstra & Bengio, 2012.
• Coordinate descent. Start with a configuration, vary one parameter at a time, and keep improvements. Repeat the process so earlier choices can be reconsidered after other parameters change. This avoids evaluating the full grid but can get stuck where no single change helps, even though changing several parameters together would.
• Bayesian optimization. Fit a probabilistic model to previous configurations and scores, then use it to choose the next evaluation, balancing exploration against promising results. It can reduce the number of costly evaluations, but its effectiveness depends on the search space and model.

Choose a search strategy based on evaluation cost, the search space, and the available budget. Random search provides a useful baseline; local or model-based search can help when evaluations are expensive. For example, the AgentDiet paper tuned four hyperparameters on 100 agent tasks by varying one parameter at a time. The authors updated two settings after the first round and found no further improvement in the second.

Weights & Biases Sweeps and Optuna provide tools for grid, random, and model-based search, automatically tracking every run and comparing results, and Keras Tuner offers the same integrated directly into Keras. These tools help manage experiments once manual tracking becomes cumbersome.