How neural networks learn: backpropagation and gradient descent

A neural network uses parameters — including weights and biases — to transform inputs into predictions. Its architecture defines the computations, and its parameter values determine how those computations respond to an input. Training adjusts those values using examples, so the network can learn tasks such as recognizing handwritten digits or translating text.

Training is the process of finding useful values for those parameters. The network makes predictions, measures their error with a loss function, and computes how the loss changes with each parameter. Backpropagation computes these derivatives, called gradients. An optimizer such as gradient descent uses them to update the parameters.

Here’s what that looks like on the simplest possible example — a model learning to fit a line. Click Step a few times and watch:

Each click runs one training iteration: make predictions, measure their error (the red dashed lines), compute gradients, and update the parameters. With the learning rate used here, the line gets closer to the data with each step.

We’ll work through each of those steps, starting with one neuron and extending the calculation to multiple layers. The widgets show how the numbers change, and the Python snippets connect those changes to code.

Start with the neuron and loss examples, then work through derivatives and the chain rule. If you already know the calculus, you can skip ahead to the chain rule across many layers.

A neural network is just parameters

We’ll use a dense feedforward network: neurons are arranged in layers, and every neuron in one layer receives the outputs of the previous layer. Each neuron combines its inputs into one output. To understand this network, we’ll start with a single neuron.

The widget below shows a neuron with 3 inputs. On the left you control the input values; on the right you set its weights, bias, and activation function. The weights and bias are learned parameters; the activation function is a choice we make when designing the network.

What you can see on the widget is a neuron multiplying each input by a corresponding weight, adding them up with a bias, and passing the result through an activation function.

Written as an equation:

Here $x_i$ is an input, $w_i$ is its weight, $b$ is the bias, and $f$ is the activation function.

Try a few configurations to build intuition:

• A weight controls how much an input matters. Set x₁ = 1.0 and the rest to 0. Now drag w₁ — the output responds directly. Set w₁ = 0 and that input is completely ignored, no matter its value.
• The sign of a weight changes its contribution. Set x₁ = 1.0, w₁ = -1.5, and everything else to 0. The weighted sum is negative, so the perceptron activation outputs 0. Now set x₁ = -1.0: the product becomes positive. A negative weight reduces the sum for a positive input and increases it for a negative input.
• The bias shifts the decision boundary. With all inputs at 0, only the bias determines the sum. With the perceptron activation, a positive bias produces 1; a negative bias means the weighted inputs must overcome it before the output becomes 1.
• The activation function shapes the output. Switch from Perceptron (hard 0/1) to Sigmoid — now the output is a smooth value between 0 and 1. Try ReLU — it passes positive values through unchanged and clips negatives to zero.

Why do activation functions matter? Without nonlinear activations, each layer computes an affine function: a linear transformation plus a bias. Composing these layers still gives one affine function, regardless of depth. Nonlinear activations let the network represent more complex relationships. For example, they can transform inputs so that classes that overlap along every straight-line boundary become separable. Chris Olah illustrates this geometry in Neural Networks, Manifolds, and Topology.

The weights and bias are the neuron’s parameters — the values it needs to learn. A neuron stores one weight per input — in our widget above, that’s a vector of 3 values. The weights determine how much each input matters, and the bias shifts the result up or down. Together, these parameters define what the neuron responds to. Different weights and biases make the same neuron detect completely different patterns in its inputs.

Since the multiply-and-sum is just a dot product, this is usually written in vector form:

Here $\mathbf{x}$ is the vector (list) of all inputs — e.g. $[x_1, x_2, x_3]$ — and $\mathbf{w}$ is the vector of all weights — e.g. $[w_1, w_2, w_3]$. The dot product $\mathbf{w} \cdot \mathbf{x}$ multiplies each pair and sums the results: $w_1 x_1 + w_2 x_2 + w_3 x_3$.

In Python, that might look like this:

import numpy as np

class Neuron:
    def __init__(self, n_inputs):
        self.w = np.random.randn(n_inputs)   # w — weight vector, e.g. [w₁, w₂, w₃]
        self.b = 0.0                         # b — bias

    def forward(self, x):                    # x — input vector, e.g. [x₁, x₂, x₃]
        z = np.dot(self.w, x) + self.b       # w · x + b — dot product + bias
        return max(0, z)                     # f(z) — activation function (ReLU)

# Create a neuron with 3 inputs and run it
neuron = Neuron(3)
output = neuron.forward(np.array([1.0, 0.5, 0.7]))

A layer is a bunch of neurons

Stack many of these neurons together and you get a layer. Put several layers together and you get a neural net. Here’s a small one — 2 inputs, two hidden layers of 3 neurons each, and 1 output. They’re called “hidden” because you only see the inputs going in and the output coming out — the layers in between are internal to the network, invisible from the outside:

Each neuron stores one weight per input and a bias, then applies an activation:

A layer typically stores all of them together in a weight matrix ($W$) — one row of weights per neuron. For the network depicted above, layer1.W is a 3×2 matrix — 3 neurons, each with 2 weights, since each neuron gets 2 inputs:

layer1.W = [[ 0.4, -0.2],    ← neuron 0: weights for x₁, x₂
            [ 0.1,  0.7],    ← neuron 1: weights for x₁, x₂
            [-0.3,  0.5]]    ← neuron 2: weights for x₁, x₂

For a single neuron, we had a dot product between one weight vector and the input: $f(\mathbf{w} \cdot \mathbf{x} + b)$. For a full layer, we stack all the weight vectors into a matrix $W$ and all the biases into a vector $\mathbf{b}$, so the same operation applies to every neuron at once:

When we compute $W\mathbf{x}$, each row of $W$ gets dot-producted with the input — that’s one neuron’s weighted sum. The matrix multiply does all of them in a single operation.

For a geometric view of this calculation, see what a weight matrix actually does. It follows four input points through a hidden layer to show how the weight matrix, bias, and nonlinear activation change their representation and make the classes separable.

In Python, that looks like this:

import numpy as np

class Layer:
    def __init__(self, n_inputs, n_neurons):
        # W is a matrix where each ROW is one neuron's weights.
        # Shape: (n_neurons, n_inputs) — so W[0] is neuron 0's weights,
        # W[1] is neuron 1's weights, etc.
        self.W = np.random.randn(n_neurons, n_inputs)
        self.b = np.zeros(n_neurons)  # b — bias vector, one per neuron

    def forward(self, x):
        # W @ x multiplies every neuron's weight row by the input,
        # computing all dot products at once
        return np.maximum(0, self.W @ x + self.b)  # ReLU activation

# Build the network from the diagram above
layer1 = Layer(2, 3)   # 2 inputs  → 3 neurons  (6 weights + 3 biases = 9)
layer2 = Layer(3, 3)   # 3 inputs  → 3 neurons  (9 weights + 3 biases = 12)
output = Layer(3, 1)   # 3 inputs  → 1 neuron   (3 weights + 1 bias   = 4)

# Forward pass — each layer's output feeds into the next
x = np.array([0.5, 0.8])
h1 = layer1.forward(x)       # input → hidden layer 1
h2 = layer2.forward(h1)      # hidden layer 1 → hidden layer 2
y  = output.forward(h2)      # hidden layer 2 → output

Matrix operations let libraries compute many neurons and many examples together, which makes them well suited to parallel execution on GPUs.

Every weight and every bias is a parameter. Layer 1 has 9, layer 2 has 12, and the output layer has 4: 25 parameters in total. Training must compute an update for each of them.

How does a network learn?

Training a neural network means adjusting every weight and bias. Weights are usually initialized randomly to break symmetry between neurons; biases can start at zero, as they do in our code. Training then updates them using gradients of the loss.

Each training iteration has four stages:

Training includes all four stages, starting with the forward pass. During inference, the network uses its current parameters to make predictions; it does not compute training gradients or update its weights.

The rest of this article focuses on how we measure error, compute gradients, and use them to update parameters.

The process has two directions — data flows forward to produce a prediction, then gradients flow backward so the optimizer can update the weights:

  Forward pass: each layer receives activations, passes output →

              activations    activations    activations
  Input ─────────▶ Layer 1 ─────────▶ Layer 2 ─────────▶ Output ──▶ Loss


  Backward pass: each layer receives gradient signal, passes it ←

              gradients      gradients      gradients
  Input ◀───────── Layer 1 ◀───────── Layer 2 ◀───────── Output ◀── ∂L
              ↓ ∂L/∂W₁           ↓ ∂L/∂W₂           ↓ ∂L/∂W₃
           (own weight       (own weight         (own weight
            gradients)        gradients)          gradients)

Notice the symmetry: in the forward pass, each layer receives activations from the previous layer and passes its output forward. In the backward pass, each layer receives a gradient signal from the next layer and passes it backward. In both directions, each layer needs an input from its neighbor to do its work.

A simple example: fitting a line

We’ll begin with one neuron, one input, one weight, and one bias. This gives us a loss with just two parameters to explore before we extend the gradient calculation to multiple layers.

With one input and an identity activation (which returns its input unchanged), the neuron computes $\hat y = wx + b$. Here $w$ and $x$ are scalars. Training this neuron to fit a line lets us study the learning algorithm without an activation derivative complicating the calculation.

Suppose someone hands you five points and says they come from a linear function $f(x) = wx + b$, and asks you to find w and b:

The data gives us inputs and outputs. We need to find the parameters connecting them: $w$, the slope of the line, and $b$, its intercept on the y-axis.

Learning a function lets us predict outputs for inputs absent from the training data. If the relationship is $f(x) = 2x + 1$, we can predict $f(1.5) = 4$ even though 1.5 is not in the table. Making accurate predictions on new inputs is called generalization. Our data follows an exact line; on real data, a close training fit alone does not establish that the model will generalize.

So let’s put those points on the graph (green dots) and try to find w and b manually by adjusting the sliders. Use the loss value to guide you — drag them and see if you can get the loss to zero. You’ll find that $w = 2$ and $b = 1$ bring the loss to zero — those are the exact parameters that generated the data, giving us $y = 2x + 1$:

The sliders report a loss: a single number summarizing prediction errors across the dataset.

As you drag the sliders, notice the red dashed lines — those are the individual errors at each data point, showing how far off the prediction is from the actual value. We compute an error for each point as $error = prediction - actual$.

Different loss functions summarize these errors in different ways:

• Mean Squared Error (MSE) — for regression (predicting numbers). Squares each error and averages them.
• Cross-Entropy — for classification (predicting categories). Measures how far predicted probabilities are from the true labels.
• Mean Absolute Error (MAE) — like MSE but uses absolute values instead of squares, less sensitive to outliers.

Since we’re fitting a line — a task known as regression (predicting a continuous number) — we’ve used Mean Squared Error (MSE): take each error, square it, then average them all. Squaring makes errors nonnegative, so positive and negative errors cannot cancel, and gives larger errors more weight:

$MSE = \frac{1}{n} \sum_{i=1}^{n} (y_{\text{pred}_i} - y_{\text{actual}_i})^2 = \frac{(y_{\text{pred}_1} - y_{\text{actual}_1})^2 + (y_{\text{pred}_2} - y_{\text{actual}_2})^2 + \cdots + (y_{\text{pred}_n} - y_{\text{actual}_n})^2}{n}$

Let’s apply this formula to our data. Say your current guess is $w = 3$, $b = 1$, so $f(x) = 3x + 1$. For each of our 5 data points, we compute the prediction, the error (how far off), and the squared error:

The MSE is 2.0. Taking its square root gives the root mean squared error (RMSE), $\sqrt{2} \approx 1.41$, in the same units as the predictions. This differs from the mean absolute error, which is $(2 + 1 + 0 + 1 + 2)/5 = 1.2$ here. At $w = 2$ and $b = 1$, all errors and both measures are zero.

We’ve managed to find the parameters manually for our simple 2-parameter function, but imagine doing this with 25 parameters, let alone millions. Later, we’ll apply everything from this article to a real task — training a network to classify handwritten digits with over 100,000 parameters. Manual tuning is impossible at that scale — no human could explore a space of millions of dimensions. We need a systematic way to look at the loss and mathematically figure out which direction to nudge each parameter to make it smaller. That’s exactly what backpropagation and gradient descent do together: backpropagation computes which way to adjust each parameter, and gradient descent takes a small step in that direction. We repeat that process to reduce the loss.

Linear least squares can also be solved directly using the normal equations or a matrix factorization. That gives a minimum of the squared-error objective; it does not guarantee a perfect fit to arbitrary data. General neural networks have no comparable direct solution, so we use iterative optimizers such as gradient descent.

The training loop, step by step

Let’s first see how this automatic algorithm works. The widget below lets you run backpropagation and gradient descent for our task step by step and watch everything happen:

• Left chart: the data points (green dots) and the model’s prediction line (blue) based on the current values of $w$ and $b$. Red dashed lines show the error at each point — the difference between the prediction and the actual value. These errors are squared and averaged to produce the MSE loss.
• Right chart: the loss after each update. At the default learning rate, it decreases toward zero. A plateau can mean convergence; rising loss can indicate that the learning rate is too large.

Click Step to run one gradient descent update, or Step x10 to run ten at once.

Keep the learning rate at its default of 0.1 and click Step repeatedly. The loss decreases as the line approaches the data. Expand Computation to inspect the predictions, errors, and gradients.

In this example, the gradients shrink as the parameters approach the minimum, so the updates lr * dw and lr * db shrink even though lr stays fixed. The parameters converge toward w = 2, b = 1. The learning rate matters: a larger value can prevent convergence, as the next widget shows.

Each click of “Step” runs one full training iteration — four steps matching the training loop we described earlier that looks like this in Python:

# 1. forward pass
y_pred = w * x + b

# 2. loss computation
error  = y_pred - y
loss   = np.mean(error ** 2)

# 3. backpropagation
dw = 2 * np.mean(error * x)
db = 2 * np.mean(error)

# 4. gradient descent
w = w - lr * dw
b = b - lr * db

Steps 1 and 2 produce predictions and compute their MSE. Step 3 computes the loss gradients. Step 4 updates each parameter by subtracting its gradient multiplied by the learning rate, lr. We will examine the learning rate first, then derive dw and db.

Choosing a learning rate

In the computation section above, you can see that gradient descent updates parameters like this:

w = w - lr * dw
b = b - lr * db

The gradient (dw, db) tells us which direction to move each parameter and by how much relative to the others. But how far should we actually step? That’s what the learning rate (lr) controls — it scales every gradient before applying it.

For a differentiable loss with a nonzero gradient, the negative gradient is a downhill direction locally. A finite step can still overshoot and increase the loss. In this particular example:

• Slow convergence (try 0.01): small updates take many steps to approach the minimum.
• Steady convergence (try 0.1): the loss decreases at each step.
• Shrinking oscillations (try 0.4): w crosses the optimum on every step, but the distance shrinks.
• Persistent oscillation (try 0.5): w alternates between -3 and 7. The bias reaches 1 immediately, but the loss stays at 50 after the first step.
• Divergence (try 1.0 or 1.5): w overshoots by an increasing amount and the loss grows.

Try it yourself — change the learning rate and click Step x10 to see the effect:

The stable learning-rate range depends on the loss. For this dataset, convergence from an arbitrary starting point requires a positive learning rate below 0.5; we will see why when we simplify the loss formula.

Larger networks require tuning the learning rate for the model, data, and optimizer. A learning rate schedule changes it over training. Adaptive optimizers such as Adam also use gradient history to scale parameter updates. These choices change the update rule while retaining the forward pass, loss computation, and backpropagation stages.

Computing backpropagation

In the computation section above, you can see that backpropagation computes the gradients like this:

dw = 2 * np.mean(error * x)
db = 2 * np.mean(error)

There’s a lot packed in these two lines. Why do we multiply error by x for dw but not for db? Where does the 2 come from? What does mean have to do with anything? Let’s unpack it step by step.

Remember, the loss is computed from the predictions, and the predictions depend on w and b. The loss is ultimately a function of the parameters — change w or b, and the loss changes. Drag w or b in the widget below and watch the loss change — the white dot moves along the curve, showing exactly where you are on the loss landscape:

Drag w: the dot moves along the left curve, while the right curve shifts vertically. Changing b does the reverse. For this centered dataset, the minimum along the w axis stays at 2 and the minimum along the b axis stays at 1. The parameters can be optimized independently here; we’ll derive the reason below.

The gradient formulas come from taking the derivative of these curves — measuring how much the loss changes when you nudge each parameter by a tiny amount. So, 2 * mean(error * x) is simply the derivative of the loss function with respect to w.

To understand how we get from the loss function to 2 * mean(error * x), we need three concepts that build on each other:

1. Derivatives — what it means to measure how a function changes
2. The chain rule — how to compute derivatives when functions are chained together
3. Partial derivatives and gradients — how to handle multiple parameters at once

By the end, we’ll trace exactly where every piece of that formula comes from. Let’s start with what a derivative actually is.

The formula we just saw — 2 * mean(error * x) — is specific to MSE loss with a linear model. Different loss functions and architectures produce different gradient formulas — but the underlying math principles are always the same. For our simple model we can derive the formula by hand; for deep networks with millions of parameters, frameworks like PyTorch and TensorFlow compute derivatives automatically using autograd (automatic differentiation).

The derivative: slope at a point

A derivative answers one question: if I nudge this input a tiny bit, how much does the output change? Think of it as the slope of a curve at a single point. If you’re standing on a hill, the derivative tells you how steep the ground is under your feet — and in which direction it goes downhill.

Take a simple function like f(x) = x². Drag the point x along the curve and watch how the slope and the derivative change:

Set x = 2 and dx = 0.5 in the widget. The yellow line ($dx$) is a nudge to the input, the green line ($df$) is how much the output changes in response. The Computation section below the chart shows how these combine into the derivative.

First, we evaluate the function at our point: $f(2) = 4$. Then we nudge the input by dx and evaluate again: $f(2.5) = 6.25$. The difference tells us how much the output changed: $df = 6.25 - 4 = 2.25$. Dividing by the nudge gives us the rate of change: $df/dx = 2.25 / 0.5 = 4.5$.

That ratio (4.5) is approximately the derivative at x = 2 — it tells you the rate: at this point, the output changes about 4x faster than the input. It’s not exactly 4 because dx = 0.5 is still a large nudge. Now let’s reduce the dx — try dragging it down to 0.1:

• $f(2) = 4$
• $f(2.1) = 4.41$
• $df = 0.41$
• $df/dx = 0.41 / 0.1 = 4.1$ — closer to 4

The derivative is the limit of this ratio as the input change shrinks toward zero:

$f(x + dx) - f(x)$ is the change in output ($df$). Divide by $dx$ to get the ratio. The $\lim_{dx \to 0}$ part just means “shrink $dx$ toward zero” — exactly what you did with the slider, watching the ratio converge to the exact value.

For $f(x) = x^2$, we can work it out:

As $dx \to 0$, that’s just $2x$. So $\frac{df}{dx} = 2x$.

Once we have the derivative formula, we can evaluate $2x$ directly instead of estimating the slope with smaller and smaller input changes.

For another visual introduction, see The Essence of Calculus by 3Blue1Brown.

The chain rule: computing derivatives of combined functions

We know how to take the derivative of a simple function like $f(x) = x^2$. But what happens when one function feeds into another? That’s called function composition — and it’s exactly what our computation does:

y_pred = w * x + b             # prediction
error  = y_pred - y            # how far off
loss   = np.mean(error ** 2)   # squared error, averaged

To derive the gradient, first consider one data point. Its squared error depends on w through three operations: prediction, subtraction of the target, and squaring. We will average over the dataset after differentiating this chain.

So we can see that computing the loss from w isn’t one function — it’s a chain of three functions, each feeding its output into the next:

Spelled out:

• $f_1(w) = w \cdot x + b$ — the model’s prediction
• $f_2(y\_pred) = y\_pred - y$ — how far off we are
• $f_3(error) = error^2$ — the squared error (what we want to minimize)

The loss is $f_3(f_2(f_1(w)))$ — three functions nested inside each other.

We can find the derivative of each individual function, but how do we combine them to get the derivative of the whole chain? The answer is the chain rule: multiply the local derivatives together.

Remember the derivative formula:

$f'(x) = \lim_{dx \to 0} \frac{f(x + dx) - f(x)}{dx}$.

The finite-change ratio approaches the derivative in the limit. We write that derivative as $df/dx$, with the output on top and the input on the bottom. For the three functions in our chain:

• $f'_1$: output is $y\_pred$, input is $w$ → $\frac{d(y\_pred)}{dw}$
• $f'_2$: output is $error$, input is $y\_pred$ → $\frac{d(\text{error})}{d(y\_pred)}$
• $f'_3$: output is $error^2$, input is $error$ → $\frac{d(\text{error}^2)}{d(\text{error})}$

Using this notation, the chain rule expands to:

Why multiplication? Because each function is nested inside the next — the output of one becomes the input of another. Think of it as a chain of nudges: if you nudge w by a tiny amount, y_pred changes by x times that nudge. Then error changes by 1 times whatever y_pred changed. For a sufficiently small change, error² changes by approximately 2·error times the change in error. Each link in the chain scales the nudge — and scaling compounds by multiplication.

Okay, so let’s compute the combined derivative of our loss chain. We showed how to find the derivative of $x^2$, which gave us $2x$. The same approach works for simpler functions: the derivative of $ax + b$ is just $a$ (a constant multiplier), and the derivative of $x - c$ is $1$ (subtraction of a constant doesn’t change the rate). This makes calculating each individual derivative straightforward:

• To compute $f'_1 = \frac{d(y\_pred)}{dw}$, we use the fact that the derivative of $ax + b$ is $a$. Since $y\_pred = w \cdot x + b$, the derivative is x.
• To compute $f'_2 = \frac{d(\text{error})}{d(y\_pred)}$, we use the fact that the derivative of $x - c$ is $1$. Since $error = y\_pred - y$, the derivative is 1.
• To compute $f'_3 = \frac{d(\text{error}^2)}{d(\text{error})}$, we use the fact that the derivative of $x^2$ is $2x$. Since the function is $error^2$, the derivative is 2 · error.

Which gives us:

That’s $2 \cdot error \cdot x$ for a single data point.

Let’s trace it with real numbers. With w = 3, b = 1, take data point x = 2, y = 5:

w = 3
  ↓  × x = ×2
y_pred = 3·2 + 1 = 7
  ↓  × 1
error = 7 - 5 = 2
  ↓  × 2·error = ×4
error² = 4

Chain rule: $f'_1 \cdot f'_2 \cdot f'_3 = 2 \cdot 1 \cdot 4 = 8$. For a small change $\Delta w$, the squared error changes by approximately $8\Delta w$. This is a local approximation: increasing w from 3 to 4 changes the squared error from 4 to 16, an increase of 12 rather than 8.

But we have 5 data points, not one. Since MSE averages the squared errors over all data points, we need to average the derivatives too. For each point we compute $2 \cdot error \cdot x$:

Average them: $\frac{8 + 2 + 0 + 2 + 8}{5} = 4$. So dw = 4 — the gradient tells us the loss increases when we increase w, so we should decrease it. (And indeed, the true value is w = 2, which is lower than our guess of 3.)

In math notation, that’s:

And in Python:

dw = 2 * np.mean(error * x)

The widget below lets you trace this chain for each data point. Click the different x= buttons to see how the local derivatives change — notice how the chain rule gives a different value for each point, because x and error are different:

For db it’s the same chain, except $f'_1$ is different: since $y\_pred = w \cdot x + b$, the derivative with respect to b is just 1 (instead of x). So:

And in our Python code this looks like this:

db = 2 * np.mean(error)

This is why the chain rule matters: any time the loss is computed through a sequence of operations (and it always is), you need it to trace back how each parameter affected the final result. For our 2-parameter model the chain has 3 steps. A deep neural network might have hundreds — one for each layer — but the principle is identical: each layer is one more function in the composition, one more local derivative to multiply.

Applying derivatives to our loss function

Now that we know how to compute individual derivatives and combine them with the chain rule, let’s apply this knowledge to our problem. The “curve” we want to minimize is our loss function — the formula we chose to measure error. We know this function exactly:

$loss(w, b) = \frac{1}{n}\sum_{i=1}^{n}(w \cdot x_i + b - y_i)^2$

loss = np.mean((w * x + b - y) ** 2)

What we don’t know is which values of w and b make it smallest. The derivative helps us find out: it tells us if I increase w by a tiny amount, does the loss go up or down? And how fast?

The data (x and y) is fixed — it’s our training data. If we also hold b constant for now (say b = 3), then the loss becomes a function of w alone, and we can plot it as a simple curve. For example, at w = 0:

y_pred = w * x + b                  # 0 * [-2,-1,0,1,2] + 3 = [3, 3, 3, 3, 3]
error  = y_pred - y                 # [3,3,3,3,3] - [-3,-1,1,3,5] = [6, 4, 2, 0, -2]
loss   = np.mean(error ** 2)        # mean([36, 16, 4, 0, 4]) = 12.0

That gives us one point on the curve: (w=0, loss=12). Do this for every w from -5 to 5 (keeping b = 3 fixed) and we get the full picture — the loss as a function of w alone:

The x-axis is w and the y-axis is the loss. The minimum occurs at w = 2, where the loss is 4, because b is still fixed at 3. Every prediction is 2 above its target. Drag the slider to inspect the predictions, errors, and squared errors that produce each point on the curve.

We can do the same for b — this time fixing w = 2 and varying b from -5 to 5:

The same parabola shape, but now the x-axis is b. The minimum is at b = 1, where the loss drops to zero. Together, w = 2 and b = 1 are the exact parameters that generated our data — y = 2x + 1.

These plots evaluate the loss at many parameter values so we can see its shape. Exploring all combinations becomes impractical as the number of parameters grows. Gradient descent instead computes derivatives at the current parameters and uses them to choose the next step.

Partial derivatives

Notice what we just did: to understand how the loss depends on w, we froze b and varied w alone. To understand how it depends on b, we froze w and varied b alone. This is exactly what a partial derivative is — the derivative of the loss with respect to one parameter, while holding all others fixed:

• ∂loss/∂w — how the loss changes when you nudge w (with b frozen)
• ∂loss/∂b — how the loss changes when you nudge b (with w frozen)

Each of the two curves above is a slice through the loss landscape along one parameter. The slope of that curve at any point is the partial derivative:

dw = 2 * np.mean(error * x)        # ∂loss/∂w — how loss changes with w
db = 2 * np.mean(error)            # ∂loss/∂b — how loss changes with b

The widget below is a combined view of the two curves we saw above, now showing the partial derivatives in action. The left chart varies w (holding b fixed) — the right chart varies b (holding w fixed). On each chart, the white dot is where you are now, the blue dashed line is the tangent (its slope is the partial derivative), and the green arrow shows which direction to move to reduce the loss.

Drag the sliders and watch what happens:

• Far from the minimum — the curve is steep, the tangent tilts sharply, and the derivative is a large number. Gradient descent takes big steps here.
• Near the minimum — the curve flattens out, the tangent is nearly horizontal, and the derivative is close to zero. Steps get tiny — the model is fine-tuning.
• At the minimum — the tangent is perfectly flat. The derivative is zero. There’s nowhere to go — you’ve arrived.

For our five data points, $y_i = 2x_i + 1$, the mean of $x$ is zero, and the mean of $x^2$ is 2. Expanding the loss gives:

The cross term disappears because the inputs sum to zero. This explains why changing w shifts the b curve vertically without moving its minimum, and vice versa. The derivatives simplify to:

After one gradient descent update, the distances from the optimum become:

Both distances shrink when $0 < lr < 0.5$. At lr = 0.5, the first multiplier is -1, so w oscillates without getting closer. At lr = 1, it is -3, so that distance triples each step.

This independence is a property of our centered linear example. With other datasets and in multilayer networks, one parameter’s gradient generally depends on the others. To take a gradient descent step, compute all gradients at the current parameters, then apply the updates.

From derivative to gradient

The gradient is simply the vector of all partial derivatives bundled together: [dw, db]. It points in the direction of steepest increase in loss. So we move in the opposite direction — that’s why the update rule subtracts: w = w - lr * dw.

With two parameters, we can plot the loss as a 3D surface: w on one axis, b on another, and loss as the height. Our loss forms a convex bowl with a unique minimum at w = 2, b = 1. Each earlier curve is a slice through this surface. Try Step (both) from different starting points; with the widget’s fixed learning rate of 0.1, the parameters approach the same minimum.

Try clicking Step (w) and Step (b) separately — you’ll see the point move along one axis at a time, creating a staircase pattern down the bowl. Then try Step (both) — this is what real gradient descent does, updating both parameters at once. You can rotate the surface by dragging to see it from different angles.

The yellow arrow shows the update direction: its components along the parameter axes are proportional to [-dw, -db], the negative gradient. Its height follows the change in loss. Step (w) and Step (b) move along one parameter axis; Step (both) combines both updates.

At the starting point w = -3, b = 3, the gradients are dw = -20 and db = 4, so the update moves farther along the w axis. The relative sizes depend on both the loss curvature and the current parameter values. At w = 2, b = 3, for example, dw = 0 and only the bias changes.

A gradient measures local sensitivity, not how much blame a parameter deserves for the error. A small gradient can occur near a minimum, but it can also occur in a flat region with high loss.

Beyond the bowl: why deep networks are non-convex

Our squared-error loss is convex because it is a sum of squares of expressions that are affine in the parameters. For this dataset, $L = 2(w-2)^2 + (b-1)^2$ has a unique minimum. Quadratics in general need not be convex: $-w^2$, for example, curves downward.

In a multilayer network, parameters from successive layers interact through products and activations, so the loss is generally non-convex. Even two scalar linear layers can show this. Let their weights be $a$ and $c$, their input be 1, and their target be 1:

Both $(a,c)=(1,1)$ and $(-1,-1)$ have zero loss. Their midpoint $(0,0)$ has loss 1. A convex function cannot have a midpoint value greater than the average of the endpoint values, so this loss is non-convex. Nonlinear activations add further structure, but they are not required for non-convexity in the parameters.

Such landscapes can contain local minima, saddle points, and flat regions. A zero gradient alone does not establish that we have found the best solution, and gradient descent does not guarantee a global minimum. The Deep Learning textbook’s optimization chapter describes these challenges in more detail.

Training aims to find parameters that make useful predictions on new data. We monitor validation performance as well as training loss. Early stopping and regularization can help limit overfitting; they do not turn a non-convex loss into a convex one.

Stochastic gradient descent

So far, we’ve been computing gradients using all of our training data at once. Every time you clicked “Step” in the widget above, dw = 2 * mean(error * x) computed 2 * error * x for each of our 5 data points individually, then averaged them into a single gradient:

dw = 2 * mean(error * x)
   = 2 * mean([-24.00, -7.00, 0.00, -3.00, -16.00])
   = -20.00

Using all 5 points gives the exact gradient of our training loss. It does not make the update immune to outliers: a large error can still dominate the MSE gradient. On a large dataset, processing every example before each update is expensive. Mini-batches let us update the parameters after processing a smaller subset.

Shuffle the training examples, then split them into mini-batches. Run the 4-stage training loop on each batch: forward pass, loss computation, backpropagation, and parameter update. Each update uses only the examples in that batch.

This is stochastic gradient descent (SGD). “Stochastic” just means random — referring to the random shuffle. The update rule is the same, averaging over the mini-batch instead of the full dataset:

Here $B$ is the current mini-batch. Its gradient is an estimate of the full-dataset gradient at the current parameters. An individual update can increase the full-dataset loss. Because parameters change between batches, one epoch of mini-batch updates is not equivalent to one full-batch update.

Once you’ve gone through every example, that’s one epoch. Shuffle again and start the next epoch. This is why you see “epoch” in training logs — each epoch means the model has seen every example in the dataset exactly once.

For our 5 data points with a batch size of 2, it looks like this:

Each batch runs the full 4-step loop (forward pass → loss → backpropagation → gradient descent), so each epoch does 3 parameter updates instead of 1. By the end of each epoch, every data point has been used exactly once — shuffling changes which examples are grouped together and reduces dependence on their original ordering.

The widget below runs both methods side by side on our 5-point dataset so you can compare them directly. Both start from the same parameters (w = -3, b = 3) and use the same learning rate. Each click of “Step” does one parameter update for each method. The blue line (full batch) uses all 5 points every step. The orange line (mini-batch) uses only batch_size points — orange circles show which ones, and the epoch bar tracks progress through the dataset.

Click “Step” a few times and watch the loss curves on the right. The blue curve (full batch) drops smoothly at the default learning rate. The orange curve (mini-batch) zigzags.

By default, fixed order is enabled so the batches are the same every run, making this zigzag pattern reproducible. Uncheck it to shuffle randomly each epoch — the orange curve will look different every time, but the overall behavior is the same.

Click through the first few steps to see why: step 1→2 the loss drops (the batch happened to give a good gradient), but step 2→3 the loss goes up — that batch pulled the parameters in a direction that helped its own points but hurt others. Then step 3→4 it drops again. This is normal: each batch only sees a slice of the data, so some steps overshoot or even go the wrong way. With the default settings in this example, the overall trend is toward lower loss despite those individual increases.

The default full-batch run approaches the solution in fewer updates, but each update processes more examples. Compare the cost for 3 parameter updates with our 5 data points:

• Full batch (3 steps): every step uses all 5 points. That’s 3 × 5 = 15 data point computations. Each point is processed 3 times.
• SGD with batch size 2 (3 steps = 1 epoch): each step uses only 2 points (or 1 for the last batch). That’s 2 + 2 + 1 = 5 data point computations. Each point is processed once.

Both methods perform 3 parameter updates, but the mini-batch run processes one third as many examples. This counts work, not progress toward a fixed loss: the two methods need not reach the same loss after those updates. On larger datasets, the useful comparison is time to reach a target validation performance, which also depends on hardware and batch size.

You can also try different batch sizes to see the difference in behavior:

• batch size = 1 — each step uses a single point, so updates can vary substantially between examples. 5 steps = 1 epoch (each point seen once). This is the original “stochastic” gradient descent.
• batch size = 2 — less noise, each epoch takes 3 steps (2 + 2 + 1 leftover). This is closer to what’s used in practice.
• batch size = 5 — that’s all our data in one batch, so it’s identical to full batch gradient descent. Both lines overlap perfectly. 1 step = 1 epoch.

Batch size trades off gradient noise, memory use, and how efficiently the hardware processes examples. Smaller batches allow more updates per pass through the data; larger batches average over more examples per update. Noise can sometimes help optimization, but it does not guarantee escape from a local minimum or convergence to a global one.

The chain rule across many layers

Remember how backpropagation uses the chain rule to multiply local derivatives together? Our example had the simplest case: a single neuron with one weight w, one bias b, and one input:

The chain from each parameter to the loss had 3 links $(f'_1 \cdot f'_2 \cdot f'_3)$ that looked like this in Python:

y_pred = w * x + b                  # f1: prediction
error  = y_pred - y                 # f2: how far off
loss   = error ** 2                 # f3: squared error

dw = x * 1 * (2 * error)           # chain rule for w: f'1 · f'2 · f'3
db = 1 * 1 * (2 * error)           # chain rule for b: same chain, different f'1

But real networks have multiple neurons per layer, each with their own weights. The gradient computation doesn’t change — we still compute a partial derivative for each individual weight using the chain rule, same as before. The difference is scale. The chain gets longer for weights that sit further from the loss — more layers to pass through, more derivatives to multiply. And it gets wider — a neuron’s output can feed into many neurons in the next layer, so the gradient must sum contributions from all of those paths.

The next diagram has two hidden layers of 3 neurons each and a scalar output. Toggle the buttons to compare a weight in the second hidden layer with one in the first:

We will use ReLU in both hidden layers and an identity activation at the output. For one training example, define the forward pass as:

Here $W_1$ has shape $3\times2$, $W_2$ has shape $3\times3$, and the output weight vector $\mathbf{u}$ has 3 entries. The vectors $\mathbf{z}$ and $\mathbf{q}$ hold weighted sums before activation; $\mathbf{h}$ and $\mathbf{g}$ hold outputs after activation. The diagram’s $w_1$ connects $x_1$ to $h_1$. Its $v_1$ connects $h_1$ to $g_1$. We will also call the weights from $h_1$ to $g_2$ and $g_3$ $v_2$ and $v_3$: these three weights form the first column of $W_2$.

Gradient for v₁ (layer 2) starts active. This weight affects the loss through $q_1$, $g_1$, and $\hat y$. Writing $error=\hat y-y$, its derivative is:

The other inputs to $g_1$ are multiplied by different weights, so they do not appear in $\partial q_1/\partial v_1$. They still affect $q_1$ and the prediction used to evaluate the derivative. Notice that $\partial L/\partial g_1 = u_1\,2\,error$: the output connection contributes its weight, even though we are differentiating a parameter in an earlier layer.

Now select Gradient for w₁ (layer 1). Changing $w_1$ changes $h_1$, which feeds all three neurons in the second hidden layer. We must add the contributions of all three paths:

Each path contributes a product of local derivatives. Where paths branch, backpropagation sums their contributions. This gives the following calculation for the two highlighted weights:

# All values come from the same forward pass; indices start at 0.
# ReLU's derivative is 1 for positive inputs, 0 for negative inputs.
# At exactly zero, we use 0 as the implementation convention.
d_output = 2 * error

# Gradients with respect to the second layer's pre-activations q.
d_q = (q > 0) * u * d_output            # shape: (3,)

# v₁ is W2[0, 0]: h₁ -> g₁.
dv1 = h[0] * d_q[0]

# Sum the three paths from h₁ through the second layer.
d_h1 = np.dot(W2[:, 0], d_q)

# w₁ is W1[0, 0]: x₁ -> h₁.
dw1 = x[0] * (z[0] > 0) * d_h1

We can compute the gradients for every weight and bias with the same operations. This code completes backpropagation for one example, then updates all parameters:

# Output layer: y_pred = u @ g + c
du = g * d_output                      # 3 weights
dc = d_output                         # scalar bias

# Second hidden layer: g = ReLU(W2 @ h + b2)
dW2 = np.outer(d_q, h)                 # shape: (3, 3)
db2 = d_q

# First hidden layer: h = ReLU(W1 @ x + b1)
d_z = (z > 0) * (W2.T @ d_q)
dW1 = np.outer(d_z, x)                 # shape: (3, 2)
db1 = d_z

# Apply updates only after every gradient has been computed.
u = u - lr * du
c = c - lr * dc
W2 = W2 - lr * dW2
b2 = b2 - lr * db2
W1 = W1 - lr * dW1
b1 = b1 - lr * db1

For a mini-batch with MSE loss, average these per-example gradients before updating. Backpropagation reuses intermediate gradients such as d_q and d_z, so it does not have to trace every path separately for every parameter.

Adding layers introduces more weight and activation factors along each path. Repeated multiplication can make the gradient much smaller or larger by the time it reaches an early layer.

The vanishing gradient problem

In a scalar chain, if each layer contributes a derivative of 0.5, passing through 100 layers multiplies the incoming gradient by $0.5^{100} \approx 7.9\times10^{-31}$. An early layer then receives very little signal for updating its parameters. This is the vanishing gradient problem. Wider networks use matrix products and sums over paths, but repeated contraction can have the same effect.

Repeated amplification can instead produce exploding gradients, leading to unstable updates.

The widget below lets you see this in action. Drag the local derivative below 1 and watch the gradient fade to nothing as it flows backwards through the layers. Then try values above 1 and watch it explode:

Sigmoid’s derivative is at most 0.25, so repeated sigmoid derivatives can shrink a gradient substantially. Weight matrices also affect its size; activation derivatives alone do not determine whether the full gradient vanishes or explodes.

Several design choices help train deeper networks:

• ReLU activation has derivative 1 for positive inputs, so that activation does not shrink the gradient on its active branch. For negative inputs its derivative is 0, which blocks the gradient through that neuron. Weight matrices can still amplify or shrink the signal.
• Residual connections add an identity path around a block of layers. For a block $h + F(h)$, the derivative includes an identity term as well as the derivative of $F$. This provides an additional route for gradients. See the ResNet paper.
• Batch normalization normalizes intermediate values using mini-batch statistics and learns a scale and offset. It can make optimization easier, but it does not force derivatives to be 1 or guarantee stable gradients. See the batch normalization paper.

Initialization, architecture, and optimizer settings all affect gradient propagation. These techniques address different parts of the problem; none guarantees that every gradient stays at a useful scale.

Ready to put all of this into practice? In the next article, we build and train a real neural network on MNIST — writing the forward pass, backpropagation, and gradient descent from scratch in NumPy, then comparing it to a Keras implementation.