What happens when a hidden layer has more parameters?

In what a weight matrix actually does, we followed four apples through a network with two hidden neurons. Its weights were chosen by hand so that every coordinate could be calculated exactly. A weight matrix and ReLU produced a representation that the output neuron could separate with a straight line.

That raises another question: what changes if the hidden layer has three neurons, or ten? Does the boundary in the original input space become more complicated? Does having more parameters make the network more likely to find a solution?

We will keep the same four inputs and labels, increase the hidden layer’s width, and train its parameters. The widget lets us compare the resulting decision regions, repeat training from another initialization, and inspect the weights. More neurons allow more bends, but the boundary we get also depends on training.

The same four apples

Each point represents one apple, described by a color score $x_1$ and a softness score $x_2$. Positive values mean redder or softer than a reference level; negative values mean greener or firmer. The labels are unripe (0) and ripe (1).

These measurements are invented to create XOR: each class occupies a diagonal of a square. They do not describe a real rule for apple ripeness. No single straight line can put both unripe apples on one side and both ripe apples on the other.

The widget opens with an alternative hand-chosen two-neuron solution whose hidden weights are $1$ and $-1$. The matrix article uses $1$ and $-0.8$; both choices classify the same four apples correctly. Select 3 or 10 under Hidden neurons to train a new network. Train again changes the random seed; switching widths reuses the current seed. Hand-chosen (2) restores the original solution.

{
  "W": [
    [
      1,
      -1
    ],
    [
      -1,
      1
    ]
  ],
  "b": [
    0,
    0
  ],
  "V": [
    [
      1,
      1
    ]
  ],
  "c": -0.5
}

In Top view (2D), the dots remain at their original coordinates. Teal regions predict unripe; coral regions predict ripe. White dashed segments mark score zero. A pink ring marks an apple whose prediction disagrees with its label.

In Angled view (3D), the orange outline encloses the input plane, while the blue surface shows the network’s score above or below each input position. Its intersection with score zero gives the same white boundary. The height scale adjusts to fit larger scores; the axis labels and table retain their actual values. The two horizontal axes are still color and softness, even when the hidden representation has ten coordinates.

Watch the learned representation change during training

The widget above shows the classifier’s output in the original input space. We can also watch the hidden representation itself change as training updates the parameters. We will keep two hidden neurons so that both hidden coordinates can be plotted directly.

Four examples are enough to train our small XOR network. To make the geometry easier to inspect, the widget below uses 100 fixed synthetic samples in four clusters, including the original A, B, C, and D. Inputs with matching signs have label 0; inputs with opposite signs have label 1. The clusters stay away from the axes. These are illustrative labels, not a rule for real apple ripeness.

At each training step $t$, the hidden layer computes

The original samples stay fixed. What changes is the function applied to them: training updates the weights and biases, so the same inputs acquire new hidden coordinates. Unlike an animation between the stages of a forward pass, every state in this replay comes from an actual optimizer update.

Select Play training, or drag the training-step slider to inspect a recorded update. After W + b shows the affine transformation; After ReLU includes the activation. The scales remain fixed throughout each recorded run. The plot shows the hidden layer; the output layer’s weights affect the loss and training accuracy displayed above it.

Training uses batches of 20 samples and updates all nine parameters with Adam. Different batches can pull the parameters in different directions, so the representation may fluctuate, move smoothly, or change sharply. The motion is not guaranteed to oscillate or to converge to a correct classifier. New initialization records another run on the same fixed dataset; changing the learning rate keeps the initialization seed, allowing a comparison from the same starting weights. Playback speed only changes how quickly recorded steps are shown.

What one extra neuron adds

We use one fully connected hidden layer with ReLU, followed by one output neuron. With $m$ hidden neurons, the computation is

A positive score predicts ripe; a zero or negative score predicts unripe. Each hidden neuron has two input weights and one bias. The output neuron has one weight per hidden coordinate, plus its own bias:

Adding one hidden neuron adds four parameters to this network: two weights and a bias in the hidden layer, plus a weight connecting that neuron to the output. The hidden representation gains one coordinate. The input still has two measurements, and the output still has one score.

Changing width in the widget builds a new network and trains all its parameters from scratch. It does not keep the original two neurons fixed and append extra ones. Reusing a seed makes a run reproducible for a given width; differently shaped networks still start with different parameter arrays.

Why the boundary can bend

Our original two-neuron solution computes

followed by

Its boundary is the pair of lines $x_1-x_2=\pm\tfrac12$. The middle band predicts unripe; both outer regions predict ripe. There are three regions but only two classes. A class can occupy disconnected parts of the input plane.

For a wider layer, the output score has the same general form, with more terms:

Each hidden neuron switches between zero and an affine expression at its activation line $w_{j1}x_1+w_{j2}x_2+b_j=0$. Within a region where the same neurons are active, the score is affine. Crossing an activation line can change its slope.

That is why a ReLU network’s decision boundary is made of straight pieces that can meet at bends. Additional neurons provide additional places where the slope can change. Some neurons may stay inactive, have no effect through their output weight, or contribute changes that cancel, so an extra neuron need not create a visible bend. An activation line is also not automatically a class boundary: the white segments show where the combined output score is zero.

More parameters allow a more complicated boundary; they do not require one. A wider network can even reproduce the smaller network exactly by setting the extra output weights to zero. Training chooses a particular function from the ones the architecture can represent.

Compare widths, then compare initializations

Try the following sequence:

1. Select Hand-chosen (2) and note the two parallel boundaries. This solution already classifies all four apples correctly.
2. Select 3, then 10. Both train with seed 1 after that reset. Compare how the boundaries pass between the apples, as well as the number of correct predictions.
3. Select Train again several times at one width. Watch how changing the initialization affects the learned boundary.
4. Select 2 to train a two-neuron network too. Compare its learned solution with Hand-chosen (2), which has the same architecture.

The widget reports every run, including ones that miss an apple. Representing a solution and finding it through training are different questions. Our hand-chosen construction proves that two hidden neurons are sufficient for these labels. A failed two-neuron run does not disprove that; it shows that this initialization and optimization run did not find such a solution. ReLU units can become inactive on all training examples, and optimization can stall.

Every trained run uses the same four-example batch, binary cross-entropy computed from the scores, and 6,000 Adam updates at learning rate 0.02, with no regularization. The hard classification threshold is used to report predictions, not to calculate gradients. These settings are fixed so that we can explore width and initialization. They are not a claim that this is the best training setup for every width.

The hidden input weights start from a normal distribution with standard deviation 1, hidden biases with standard deviation 0.1, and output weights with standard deviation $1/\sqrt{m}$; all have mean zero. The output bias starts at zero. Training setup and current weights shows the current parameters and loss. The displayed weights are rounded; the plotted boundaries use full precision.

Four correct predictions do not determine the whole boundary

All four apples can receive the right label under different decision regions. Training provides no labels for the spaces between those points, or for the rest of the displayed plane. A network still has to assign scores there, and different parameter choices can produce different answers.

This is a useful limit of our experiment: four out of four is training accuracy on four examples. It does not tell us which boundary would work better on new apples. A more intricate boundary is not automatically better, and its appearance alone does not establish overfitting. We would need more labeled examples, including data held out from training, to compare generalization.

The changing boundary is the original-space view of a changing representation. The input measurements do not move. Every network computes its own hidden coordinates and then uses them to calculate an output score. With three hidden neurons, the intermediate vector has three entries; with ten, it has ten. All of those computations still define a function of the original two inputs.

The same forward pass at every width

For a batch of four apples, $X$ has shape $4\times2$. The only changing dimension is $m$:

The following NumPy function evaluates a network of any of these widths, given its parameters. It computes the forward pass; the widget additionally trains those parameters.

import numpy as np

def forward(X, W, b, V, c):
    # X: one apple per row, shape (batch_size, 2).
    # W: one weight row per hidden neuron, shape (m, 2).
    # Each dot product supplies one hidden coordinate per apple.
    Z = X @ W.T + b             # (batch_size, m)
    H = np.maximum(0.0, Z)      # ReLU, coordinate by coordinate
    scores = H @ V.T + c        # V: (1, m); scores: (batch_size, 1)
    predictions = (scores > 0).astype(int)
    return H, scores, predictions

For the arithmetic and geometry behind each of these operations, return to what a weight matrix actually does. For experiments on a larger dataset, our article on network depth and width compares architectures on handwritten digits.

W = np.array([[-3.0998,  3.0998],
              [ 3.1660, -3.1659]])
b = np.array([-0.00004169, -0.00008875])

V = np.array([[4.3837, 4.2924]])
c = -5.9400