The process of training a neural network involves searching for weights and biases that make its predictions more accurate. We compare those predictions with the expected answers using a loss function—a formula that turns their mismatch into a number, with a lower value indicating a better match. To choose an update, we need to know how changing each parameter would affect that loss. We could randomly nudge a parameter, calculate the loss again, and compare the results. But checking millions of parameters one at a time this way would require millions of additional evaluations.

To guide this search, we use the gradient, which collects the partial derivatives of the loss with respect to the parameters. These derivatives let us estimate how small changes in the parameters will affect the loss without recalculating it. For each parameter, the derivative describes how sensitive the loss is to that parameter while the others stay fixed.

Mathematically, this is an optimization problem, specifically minimization: we look for model parameter values that make the loss as small as possible. Although zeroth-order techniques such as MeZO can guide updates by randomly perturbing many parameters at once and evaluating the loss, the standard approach is to use derivative information to guide the search. Neural networks today are predominantly trained using gradients computed by the algorithm known as backpropagation—the approach we’ll develop here.

To understand backpropagation, we’ll build on several ideas: local derivatives, the chain rule, and computation graphs. Along the way, we’ll explore three methods for calculating derivatives: numerical differentiation, symbolic differentiation, and automatic differentiation. We’ll begin with functions small enough to differentiate by hand and compare their exact derivatives with finite-difference estimates, then move to functions with several inputs. We’ll use partial derivatives to isolate each input’s effect while holding the others fixed and collect these rates into a gradient.

Next, we’ll follow changes through connected operations, showing how the chain rule multiplies derivatives along a path and adds contributions when an input affects the output through several paths. The central idea is that knowing how to differentiate a small set of primitive operations lets us differentiate functions assembled from them. Even the derivative of a square can follow from multiplication alone. We’ll connect this idea to symbolic differentiation, then trace a forward and backward pass to see how reverse-mode autodiff reuses intermediate results to calculate gradients for many inputs—the approach used by backpropagation.

This article builds the theoretical foundations for understanding Andrej Karpathy’s Micrograd, a small automatic differentiation library with a PyTorch-like API. In his video walkthrough, he builds the library and uses it to train a neural network. The same principles underlie PyTorch’s torch.autograd.backward, which computes gradients using the chain rule.

From rate of change to derivatives

Let’s start with a simple linear relationship $z=3x$ and find its derivative by examining the slope of its graph. A derivative tells us how quickly a function’s output changes as we change its input, at a particular input value. Here, any increase in $x$ produces an increase in $z$ three times as large. Using $\Delta$ to represent a change, we can write:

$$\Delta z=3\,\Delta x.$$

When we plot $x$ horizontally and $z$ vertically, this constant rate of change gives a straight line. For a straight line, the derivative equals its slope, so we can find it by calculating the ratio of output change to input change, often called ‘rise over run’:

$$\text{slope}=\frac{\Delta z}{\Delta x}=\frac{\text{rise}}{\text{run}}$$

We can see on the graph below how the blue line $z=3x$ rises by $3$ for a run of $1$, giving a slope of $3/1=3$.

The dotted gray line $z=0.5x$ and dashed purple line $z=x$ rise by only $0.5$ and $1$ for the same run of $1$. With the same axis scales, a larger absolute slope means a steeper line. All three slopes are positive, so the lines rise as we move right; a negative slope would make a line fall.

We write this derivative as $\frac{dz}{dx}$, read as ‘the derivative of $z$ with respect to $x$’. This notation describes how quickly the output $z$ changes as the input $x$ changes. For our blue line:

$$\frac{dz}{dx}=3$$

The $d$ symbols indicate a derivative, whereas $\Delta z/\Delta x$ describes the average rate of change over a finite interval. For $z=3x$, the rate of change is constant and doesn’t depend on $x$, so the derivative is $3$ wherever we evaluate it.

We’ve used the line’s slope to find the derivative. We can also calculate it using differentiation rules. Since $z=3x$, we can replace $z$ with its expression and write:

$$\frac{dz}{dx}=\frac{d}{dx}(3x)$$

Here, the differential operator $\frac{d}{dx}$ means ‘differentiate the expression that follows with respect to $x$’. In $\frac{d}{dx}(3x)$, that expression is $3x$, treated as a function of $x$. This notation is especially useful when we treat individual terms in a larger expression as functions and differentiate them separately.

The constant-multiple rule captures the behavior we saw on the graph: multiplying an input by $3$ multiplies its changes by $3$. To differentiate $3x$, we keep the multiplier $3$ and differentiate $x$. The derivative of $x$ with respect to itself is $1$:

$$\frac{d}{dx}(3x)=3\frac{d}{dx}(x)=3\times1=3.$$

Differentiation rules give us a systematic way to calculate rates that are harder to see in more complicated expressions. We’ll introduce more of these rules when we connect several operations.

A derivative that changes with the input

Let’s now look at a function whose rate of change depends on its input. This makes the derivative’s local nature clearer: it describes how the output changes near the particular input value where we evaluate it. To demonstrate that, we’ll use $z=x^2+3$. We can also write $z(x)$ to make its dependence on $x$ explicit:

$$z(x)=x^2+3.$$

We can write this derivative as $dz/dx$, using the same notation we introduced above, called Leibniz notation. Another way to write it is $z'(x)$, read as “z prime of x,” which is called Lagrange notation, or prime notation. Both mean the derivative of $z$ with respect to $x$.

To find the derivative of $z(x)=x^2+3$, we’ll differentiate its two terms separately, then add the results. This uses three rules:

Rule	How we use it
Power rule	Differentiate $x^2$
Constant rule	Differentiate the constant term $3$
Sum rule	Add the derivatives of the two terms

Let’s first apply the power rule to $x^2$: multiply by the exponent, then reduce the exponent by one. In general,

$$\frac{d}{dx}(x^n)=nx^{n-1}$$

For $x^2$, this gives $2x$. Then we apply the constant rule to $3$. Its derivative with respect to $x$ is zero because it does not change as $x$ changes. And finally, we use the sum rule to add these derivatives. So for our function, the derivative is:

$$z'(x)=2x+0=2x$$

To see what this formula tells us, let’s calculate the function’s output and its derivative at three values of $x$:

Input $x$	Output $z(x)=x^2+3$	Derivative $z'(x)=2x$
$-5$	$(-5)^2+3=28$	$2\times(-5)=-10$
$0$	$0^2+3=3$	$2\times0=0$
$5$	$5^2+3=28$	$2\times5=10$

Unlike the constant rate of $3$ in $z=3x$, this rate depends on the current value of $x$. The function’s output is the same at $x=-5$ and $x=5$, but the derivatives have opposite signs. At $x=5$, the positive derivative means the output rises as $x$ increases slightly. At $x=-5$, the negative derivative means the output falls as $x$ increases slightly.

To reduce the function’s output, the derivative’s sign tells us which way to move from each starting point: increase $x$ at $x=-5$ and decrease $x$ at $x=5$. The green arrows below show these directions, which we can determine without trying both an increase and a decrease in $x$ and recalculating the output. Both arrows point toward $x=0$, where this function reaches its minimum and the derivative is zero. A zero derivative, however, does not always indicate a minimum.

A common way to visualize the derivative is as the slope of the tangent to the graph at a selected point. At $x=5$, the derivative is $10$, so the tangent has slope $10$. Change Parameter x below to see how the purple tangent’s slope changes along the blue curve.

Parameter x

5

Exact derivative10z′(x) = 2x

The tangent also helps us estimate how much the function’s output changes nearby. At $x=5$, where the output is $28$ and the derivative is $10$, a small change in $x$ produces a change in output approximately $10$ times as large:

$$\Delta z\approx10\,\Delta x.$$

We use $\approx$ in the expression above because we are estimating the change in output over a small step. At $x=5$, the derivative is exactly $10$, but the slope changes as we move along the curve. Setting Parameter x to $4.9$ in the widget gives a slope of $9.8$; a smaller move to $4.99$ would give $9.98$. The smaller the step, the less the slope changes, so treating it as constant at $10$ gives a closer estimate of the change in output.

Estimating the derivative with finite differences

We just used a known derivative to estimate a small change in output. We can also work in the other direction: evaluate the function at two nearby inputs and use the output change to estimate the derivative. This returns to the rise-over-run calculation we used for $z=3x$. For a curved graph, however, the slope varies, so the ratio generally gives an approximation. This approach is called finite differences, a method of numerical differentiation. It requires function evaluations, but no formula for the derivative.

The forward-difference estimate divides the output change by a small, nonzero input change $h$:

$$\frac{z(x+h)-z(x)}{h}$$

Let’s calculate the finite-difference estimate using $z(x)=x^2+3$. Starting at $x=5$, a step of $h=0.001$ takes us to $5.001$:

$$\frac{z(5.001)-z(5)}{0.001} =\frac{28.010001-28}{0.001} =10.001$$

This is the average rate of change between the two inputs. Geometrically, it is the slope of the straight line joining their points on the curve, called a secant. The derivative gives the tangent’s slope at the starting point.

In the widget below, set Parameter x to $5$, where the purple tangent has slope $10$. With Small change h at $0.001$, the orange secant has slope $10.001$. This is the finite-difference estimate. As we shrink $h$, the orange slope approaches $10$, the exact derivative.

Tangent: exact derivativeSecant: finite difference

Parameter x

2

Small change h

0.8

Exact derivative4z′(x) = 2x

Finite difference4.8(z(x + h) − z(x)) / h

We refer to this as an average because we divide the total change in output by the total change in input across the interval. The finite-difference approximation uses only the two endpoint values, so it doesn’t tell us how the function behaves between them. Its output could change more slowly in one part of the interval and more quickly in another.

Since we know the formula, we can plot the curve between the endpoints and see that distinction. The graph below uses a wider interval, from $x=1$ to $x=5$, to make the difference visible. The dashed line connecting the endpoints has the average slope; the green curve shows how the output actually changes. The shaded triangle shows the total input change $\Delta x=4$ and output change $\Delta z=24$, so the average slope is $24/4=6$.

The average slope uses only the endpoints

Whole intervalFirst halfSecond halfFunction curve

We can compare the average rate over the whole interval with the rates over its two halves. Splitting at $x=3$, where the output is $12$, gives:

Interval	Input change	Output change	Average rate
Whole interval	$1 \to 5$	$4 \to 28$	$(28-4)/(5-1)=6$
First half	$1 \to 3$	$4 \to 12$	$(12-4)/(3-1)=4$
Second half	$3 \to 5$	$12 \to 28$	$(28-12)/(5-3)=8$

The limit definition makes precise the idea of a rate of change at a single point: it is the value that the average rate of change approaches as the interval shrinks toward zero:

$$z'(x)=\lim_{h\to0}\frac{z(x+h)-z(x)}{h}$$

We let $h$ approach zero rather than substitute $h=0$, which would give $0/0$. Taking this limit lets us derive the rule $z'(x)=2x$. For our function, first simplify the ratio while keeping $h$ nonzero:

$$\begin{aligned} \frac{z(x+h)-z(x)}{h} &=\frac{(x+h)^2+3-(x^2+3)}{h}\\ &=\frac{2xh+h^2}{h}\\ &=2x+h \end{aligned}$$

This makes the distinction explicit: a finite-difference calculation at a chosen step gives $2x+h$. Taking the limit as $h$ approaches zero removes the extra $h$, giving us the derivative rule:

$$z'(x)=2x,\qquad z'(5)=10$$

This explains the rule we applied symbolically earlier. Once we have the formula $2x$, we can evaluate it directly at any input without evaluating the function at a nearby input.

Forward difference compares $x$ with $x+h$. Sampling behind $x$ gives a backward-difference estimate, while sampling equally on both sides gives a central-difference estimate. These are other types of finite-difference approximation. For the same function $z(x)=x^2+3$, we’ll now use the larger step $h=0.1$:

In exact arithmetic, as we make the step $h$ smaller, the forward, backward, and central difference estimates converge to the exact derivative—for our function at $x=5$, that value is $10$. Central difference is exact for this quadratic in exact arithmetic, but generally remains an approximation.

Software can estimate a derivative using finite differences: evaluate the function at two nearby inputs, subtract the outputs, and divide by the input difference. Choosing that difference requires care. A finite step generally introduces truncation error because we use an interval to estimate the rate at a point. Shrinking the step reduces this error, but an extremely small step creates another problem: computers store numbers with limited precision. Subtracting nearly equal function values can leave a difference dominated by rounding error, which division by a tiny $h$ amplifies.

These methods are useful for checking derivatives, but their additional function evaluations make them expensive when applied separately to millions of parameters.

Multiple inputs and partial derivatives

So far, we’ve used functions $z=3x$ and $z(x)=x^2+3$ with only one parameter $x$. Let’s now look at $z=x+y$ and $z=xy$, with two parameters $x$ and $y$ that we can vary separately. To understand each parameter’s effect on $z$, we change one while holding the other fixed. Changing both at once would mix their effects together.

Earlier, we wrote $dz/dx$ to describe how the output $z$ changes with the single input $x$. For our two-input functions, we need to specify which input we vary. To measure how $z$ changes when we vary $x$ while holding $y$ fixed, we use a partial derivative, written $\partial z/\partial x$. We read this as 'the partial derivative of $z$ with respect to $x$'. The order means the same thing as before; the symbol $\partial$ indicates that we’re varying one input while holding the others fixed.

Geometrically, the two inputs determine a point on the surface $z=f(x,y)$. A partial derivative measures the slope at that point in one input’s direction, while the other input stays fixed. We therefore need both input values to specify where we measure the slope, even though we examine changes in only one direction.

When calculating a partial derivative, we apply differentiation rules while treating the other inputs as constants.

Let’s start with addition, $z=x+y$. Holding $y$ fixed, increasing $x$ by $1$ increases $z$ by $1$, so $\partial z/\partial x=1$. Reversing the roles gives $\partial z/\partial y=1$.

For multiplication, $z=xy$, the rate depends on the input we hold fixed. The following table shows how $z$ changes as we increase $x$ while holding $y$ at the constant value $5$:

$\boldsymbol{x}$	$\boldsymbol{y}$	$\boldsymbol{z=xy}$
$3$	$5$	$15$
$4$	$5$	$20$
$5$	$5$	$25$
$6$	$5$	$30$

We can see that every time $x$ increases by $1$, $z$ increases by $5$. With $y$ held at $5$, the rate of change with respect to $x$ is therefore $5$:

$$\frac{\partial z}{\partial x}=y=5$$

As you can see, the rate of change is equal to the constant value of $y$. This uses the same constant-multiple rule we applied to $z=3x$. The multiplier in $z=3x$ was $3$. In $z=xy$, the multiplier is $y$, which we hold constant while differentiating $x$. So when we keep one variable constant, it still participates in the calculation and influences how the output responds to changes in the other variable.

Differentiating with respect to $y$ works the same way: we hold $x$ constant and apply the constant-multiple rule, giving $\partial z/\partial y=x$. The two inputs play symmetric roles, but their effects depend on the current values of $x$ and $y$. At $x=3$ and $y=5$, the output is $15$, and the partial derivatives are $5$ and $3$, respectively.

We can now summarize the local derivative rules for both operations:

Operation	$\partial z/\partial x$	$\partial z/\partial y$
$z=x+y$	$1$	$1$
$z=xy$	$y$	$x$

We can derive the formula $\partial z/\partial y=x$ without knowing the numerical value of $x$. To evaluate that derivative, however, we need its value: when $x=5$, the derivative is $5$. Holding $x$ fixed means it does not change as we vary $y$; its value still determines the rate of change.

Why does the derivative with respect to $x$ differ between these operations? In addition, $y$ shifts the output without changing how strongly $x$ affects it: increasing $x$ by $1$ increases $z$ by $1$, whatever fixed value $y$ has. In multiplication, $y$ scales that effect: increasing $x$ by $1$ changes $z$ by $y$. With $y=3$, this is exactly the relationship $z=3x$ we started with.

Holding $y$ fixed makes this scaling exact: $\Delta z=y\,\Delta x$. Holding $x$ fixed instead gives $\Delta z=x\,\Delta y$. Each operand is therefore the scaling factor for changes in the other operand. More generally, a local derivative is a scaling factor for small input changes; for a nonlinear function, that relationship is usually an approximation near the current input.

The derivative describes how an operation’s output responds to an input; it isn’t a property of $x$ alone. These derivatives are called local because they describe that relationship for a single operation. Later, we’ll connect them to follow changes through several operations.

The partial derivatives $5$ and $3$ describe sensitivity, not portions of the output $15$. For equal small changes, varying $x$ alone has $5/3$ times the effect of varying $y$ alone. To estimate the effect of changing both inputs, we need both derivatives and the proposed input changes.

To do this, we collect the partial derivatives into a vector called the gradient. We denote the gradient with the symbol $\nabla$, so for our function $z=xy$ at $x=3$ and $y=5$, we write $\nabla z=(5,3)$. This vector also has a geometric meaning in the input plane: among moves of the same small Euclidean length, it points toward the greatest first-order increase in the output. Its negative points toward steepest local decrease.

The graph below demonstrates these directions for $z=xy$. You can see how the purple arrow leads uphill from $(3,5)$, where the output is $15$, while the green arrow leads downhill. They trace surface paths as we move the inputs in the gradient direction or its opposite. The graph scales height separately; the gradient specifies a direction in the input plane.

Reset

∇z = (5, 3): local increase−∇z = (−5, −3): local decrease

Drag to rotate, or focus the graph and use the arrow keys. Height is scaled separately; the gradient describes direction in the input plane.

Once we know the partial derivatives, we can use a first-order Taylor approximation, also called a linear approximation. It allows us to estimate the combined effect of small input changes on the output: we multiply each partial derivative by the corresponding input change and add the results.

$$\Delta z\approx\frac{\partial z}{\partial x}\Delta x+\frac{\partial z}{\partial y}\Delta y$$

This is the dot product of the gradient and the input-change vector. For our function $z=xy$ at $(3,5)$, it gives:

$$\Delta z\approx5\,\Delta x+3\,\Delta y$$

For example, increasing both inputs by $0.1$ gives:

$$\Delta z\approx5(0.1)+3(0.1)=0.8$$

Increasing $x$ by $0.1$ contributes $0.5$ to the predicted change, while increasing $y$ by $0.1$ contributes $0.3$. Each contribution is a partial derivative multiplied by the corresponding input change. The derivatives describe sensitivity; the chosen changes determine how much of that sensitivity enters the prediction.

Evaluating the function at the new inputs gives an actual change of $3.1\times5.1-15=0.81$. The remaining $0.01$ is the interaction term $\Delta x\,\Delta y$, which the linear approximation leaves out. The terms $0.5$ and $0.3$ therefore describe contributions to the predicted change, not shares of the original output $15$. The same distinction applies to backpropagation: it calculates the loss’s sensitivity to each weight. Multiplying those derivatives by proposed weight changes estimates their contributions to the loss change.

Combining differentiation rules for a more complex function

In the previous section, we calculated partial derivatives for addition and multiplication. Now we’ll combine differentiation rules in an expression with several terms and powers:

$$f(x,y)=xy^2+x^3$$

We’ll compare two equivalent approaches to calculating partial derivatives at a chosen point: differentiate first and substitute the values afterward, or substitute the other input’s value first, differentiate the resulting one-input function, and then evaluate its derivative. Both approaches produce derivative formulas. Plotting the one-input functions as slices through a surface will show why each partial derivative is also an ordinary tangent slope.

Differentiate first, then evaluate

We keep $x$ and $y$ as symbols, finding a partial derivative for each input while treating the other as constant. Then we’ll evaluate both derivatives at $(x,y)=(3,2)$.

First, we differentiate with respect to $x$, holding $y$ fixed so that $y^2$ acts as a constant multiplier of the derivative of $x$. To differentiate $x^3$, we use the power rule:

$$\begin{aligned} \frac{\partial f}{\partial x} &=y^2\frac{\partial}{\partial x}(x) +\frac{\partial}{\partial x}(x^3)\\ &=y^2\cdot1+3x^2\\ &=y^2+3x^2 \end{aligned}$$

Then we differentiate with respect to $y$, reversing the roles and holding $x$ fixed. Now $x$ is a constant multiplier of $y^2$. The term $x^3$ does not depend on $y$, so differentiating it with respect to $y$ gives zero:

$$\begin{aligned} \frac{\partial f}{\partial y} &=x\frac{\partial}{\partial y}(y^2) +\frac{\partial}{\partial y}(x^3)\\ &=x\cdot2y+0\\ &=2xy \end{aligned}$$

This gives us symbolic formulas for the partial derivatives. We can evaluate them at a particular point by substituting its input values—for example, $(x,y)=(3,2)$:

Input we change	Symbolic formula	Partial derivative at $(3,2)$	Predicted output change
$x$	$\partial f/\partial x=y^2+3x^2$	$2^2+3(3^2)=31$	$\Delta f\approx31\,\Delta x$
$y$	$\partial f/\partial y=2xy$	$2(3)(2)=12$	$\Delta f\approx12\,\Delta y$

Each partial derivative tells us how quickly $f$ changes as we vary one input while holding the other fixed. As before, we collect the partial derivatives into a gradient vector. At $(3,2)$, the gradient is $\nabla f=(31,12)$. To estimate how small changes in both inputs affect the output, we multiply each change by its corresponding partial derivative and add the results. This is the dot product of the gradient and the input-change vector:

$$\Delta f\approx\nabla f(3,2)\cdot(\Delta x,\Delta y) =31\Delta x+12\Delta y$$

At $(x,y)=(3,2)$, both coordinates specify the point where we measure the slopes. Moving in the $x$-direction with $y$ fixed at $2$ gives a tangent slope of $31$; moving in the $y$-direction with $x$ fixed at $3$ gives a slope of $12$. These are two directions through the same point on the surface. Each rate is a partial derivative of the original two-input function and an ordinary derivative of its corresponding one-input slice. The gradient combines these rates into the direction of steepest local increase in the input plane.

Fix one input first, then differentiate the slice

Let’s now look at the second approach, where we fix one input first, then differentiate the slice. We substitute a value for one input in $f(x,y)=xy^2+x^3$, leaving an ordinary function of the remaining input.

Each row below starts from the original function and fixes one input:

Input held fixed	Remaining variable	Expression we will differentiate
$y=2$	$x$	$f(x,2)=x(2^2)+x^3=4x+x^3$
$x=3$	$y$	$f(3,y)=3y^2+3^3=3y^2+27$

These expressions are slices of the original function. We have only substituted input values; we have not differentiated yet.

Let’s now differentiate the slice obtained by substituting $y=2$ into $f(x,y)$. Just as before, $y^2$ acts as a constant multiplier of $x$, contributing $4$ to the derivative when $y=2$. With only $x$ remaining as a variable, we can now take the ordinary derivative with respect to $x$:

$$\frac{d}{dx}(4x+x^3)=4+3x^2 \quad\Longrightarrow\quad 4+3(3^2)=31\quad\text{at }x=3$$

Then, for the slice obtained by substituting $x=3$ into $f(x,y)$, we differentiate $3y^2+27$ with respect to $y$:

$$\frac{d}{dy}(3y^2+27)=6y \quad\Longrightarrow\quad 6(2)=12\quad\text{at }y=2$$

Both slices pass through the same selected point, where $f(3,2)=3(2^2)+3^3=39$.

The two approaches differ in the order of the steps: we can differentiate first and then substitute the fixed input’s value, or substitute that value first and then differentiate. Differentiating first gives the general formulas $y^2+3x^2$ and $2xy$. Substituting first and then differentiating gives $4+3x^2$ for the slice with $y=2$, and $6y$ for the slice with $x=3$—the same formulas we get by substituting those fixed values into the general derivatives. Evaluating the derivatives of these slices at $(3,2)$ gives the same values as our earlier calculation. Both produce:

$$\frac{\partial f}{\partial x}(3,2)=31,\qquad \frac{\partial f}{\partial y}(3,2)=12$$

We can now use this slicing technique to visualize how the function changes as we vary each input separately. At the top of the widget below, you can see the surface $f(x,y)=xy^2+x^3$: its height represents the output for each input pair. The two purple curves trace slices through the selected point—one varies $x$ while holding $y$ fixed, and the other varies $y$ while holding $x$ fixed.

Below the surface, the two plots show these slices separately: $f(x,2)=4x+x^3$ on the left and $f(3,y)=3y^2+27$ on the right at the initial point $(3,2)$. Each green tangent shows the slope at the selected point, giving the corresponding partial derivative. The orange arrow on the surface follows the gradient direction, combining both partial derivatives into a direction of steepest local increase in the input plane.

Reset

x

3

y

2

f(x, 2) = 4x + x³

∂f/∂x = 31

f(3, y) = 3y² + 27

∂f/∂y = 12

Starting from $(3,2)$, moving the $x$ slider affects the two plots differently. On the left, $x$ is the varying input, so the selected point moves along the same curve $f(x,2)=4x+x^3$ while $y=2$ stays fixed. On the right, $x$ is the fixed input that defines the slice, so changing it from $3$ to $4$ changes the formula from $f(3,y)=3y^2+27$ to $f(4,y)=4y^2+64$. In other words, moving $x$ moves the point along the left slice and changes which slice is shown on the right. Moving $y$ does the reverse.

Following derivatives through connected operations

Earlier, we used the sum rule to differentiate $xy^2+x^3$: we differentiated each term with respect to the chosen input, holding the other input fixed, and added the results. More generally, we can view each intermediate calculation as a function—for example, $g(x,y)=xy^2$ and $h(x,y)=x^3$, so our original expression is $f(x,y)=g(x,y)+h(x,y)$. Here, we combine the functions by adding their outputs. We can also multiply their outputs or use one function’s output as another’s input. Each way of combining functions has a corresponding differentiation rule.

The following table shows the rules for addition, multiplication, and composition (nesting). For simplicity, we’ll write these rules for two differentiable functions of one input, $g(x)$ and $h(x)$:

Rule	How the functions combine	Derivative with respect to $x$
Sum rule	$g(x)+h(x)$	$g'(x)+h'(x)$
Product rule	$g(x)h(x)$	$g'(x)h(x)+g(x)h'(x)$
Chain rule	$g(h(x))$	$g'(h(x))h'(x)$

These rules also cover cases where one function stays constant as $x$ changes. Its derivative is then zero: in the sum rule, its contribution disappears; in the product rule, its value remains as a multiplier. If that function is a multiplier $c$, differentiating $c\,g(x)$ gives $c\,g'(x)$. The same mechanism works with $g(x)h(x)$: if $h$ is constant with respect to $x$, then $h'(x)=0$, so $g'(x)h(x)+g(x)h'(x)$ reduces to $g'(x)h(x)$, with $h(x)$ acting as the constant multiplier. In our earlier term $xy^2$, the fixed value $y^2$ played the role of this constant multiplier when we differentiated with respect to $x$.

The chain rule is especially important when we build complex functions by connecting simpler operations, with one operation’s output becoming another’s input. Knowing each operation’s local derivative lets us describe how it responds to its immediate inputs; the chain rule connects those derivatives to determine how the final output responds to an input earlier in the calculation.

These rules offer different ways to differentiate the same expression. We can also view the product $c\,g(x)$ as a composition: first calculate $g(x)$, then multiply its output by $c$. The chain rule gives the same derivative, $c\,g'(x)$, by connecting those two operations’ local derivatives.

Let’s now look in detail at how the chain rule connects local derivatives in our function $f(x,y)=xy^2+x^3$. We’ll start by splitting the calculation into individual operations, using braces to label their results within the original expression:

$$\underbrace{ \overbrace{x\underbrace{y^2}_{u}}^{v} +\overbrace{x^3}^{w} }_{f}$$

Using these names, we can write each operation separately and show which intermediate results it uses:

$$u=y^2,\qquad v=xu,\qquad w=x^3,\qquad f=v+w$$

We can also draw the calculation as a graph, with arrows showing how inputs and intermediate results flow into subsequent operations:

For example, squaring maps $y$ to $u=y^2$, and multiplication maps $x$ and $u$ to $v=xu$. The square’s output becomes an input to multiplication. When one function’s output becomes another’s input, the functions compose—a relationship called function composition, shown as $g(h(x))$ in the table above. An operation has its own immediate inputs; connecting operations makes its result depend on earlier inputs as well.

The split follows the expression’s dependencies: $u=y^2$ must be available before $v=xu$, and both $v$ and $w$ must be available before $f=v+w$. Since $w=x^3$ does not depend on $u$ or $v$, we can calculate it before or after that branch. The split is not unique: we could express squaring and cubing as multiplications. Here, we treat them as individual operations and use the power rule for their local derivatives.

We’ll now use the chain rule to calculate derivatives through individual operations, multiplying local derivatives along each path and adding contributions from different paths. We’ll first follow one path, then examine an input that reaches the output through two paths.

The chain rule: following one path

We’ll start by differentiating with respect to $y$, holding $x$ fixed. The split shows why $y$ is a useful starting point: changing it affects $u=y^2$, then $v=xu$, while $w=x^3$ stays constant. Afterward, we’ll examine $x$, which affects both $v$ and $w$. For now, there is just one path from $y$ to the final output:

To see the difference between a local derivative and the chain rule, focus on just two operation nodes: the square $u=y^2$ and the multiplication $v=xu$, connected by $u\to v$.

Viewed on its own, the multiplication takes $x$ and $u$ as inputs. Its local partial derivatives are:

$$\frac{\partial v}{\partial x}=u, \qquad \frac{\partial v}{\partial u}=x$$

To derive each local derivative, we only need the operation’s expression and the input we differentiate with respect to. For $v=xu$, treating $x$ as constant gives $\partial v/\partial u=x$, regardless of how $u$ was calculated or where $v$ is used. We do not need the rest of the graph for this step. The square separately supplies $du/dy=2y$.

Notice how the chain rule includes the derivative of node $u=y^2$, $du/dy=2y$, separately, even though node $v=xu$ already uses $u$’s value. The local derivative $\partial v/\partial u=x$ only tells us how $v$ responds to a change in $u$. It does not include how $u$ changes with $y$. That rate is $du/dy=2y$, so we need both factors:

$$\frac{\partial v}{\partial y} =\underbrace{\frac{\partial v}{\partial u}}_{\text{how v responds to u}} \cdot\underbrace{\frac{du}{dy}}_{\text{how u responds to y}} =x\cdot2y$$

With $x$ fixed, multiplication is the outer function and squaring is the inner function. The outer derivative alone cannot tell us how the inner function changes: multiplication accepts $u$’s value without knowing whether it came from $y^2$, $2y$, or another calculation. At $y=2$, both $y^2$ and $2y$ produce $u=4$, but their derivatives are $4$ and $2$, respectively. The same value therefore does not imply the same response to a change in $y$. This distinction applies whenever we differentiate a composition.

If we substitute $u=y^2$ into $v=xu$, the complete expression $v=xy^2$ contains that inner dependency. Differentiating it directly gives $2xy$, accounting for both rates. The graph makes those two factors explicit.

The table below follows only the effect of changing $y$, with $x$ held fixed. It focuses on the two operations $u=y^2$ and $v=xu$, stopping at $v$ to calculate $\partial v/\partial y$:

Step	Symbolic form
Define the operations	$u=y^2$, then $v=xu$
Derive the local derivative formulas	$du/dy=2y$, $\partial v/\partial u=x$
Combine the formulas using the chain rule	$\partial v/\partial y=(\partial v/\partial u)(du/dy)=x\cdot2y$

The connections matter when we combine local derivatives: the path $y\to u\to v$ tells us to multiply $\partial v/\partial u$ by $du/dy$ to find $\partial v/\partial y$. The operation supplies the local rule; the graph tells us how to connect it to the other rules.

Now continue to the final addition. The original function takes $x$ and $y$, but each operation has its own inputs. For the final addition $f=v+w$, those inputs are $v$ and $w$. Its local derivative $\partial f/\partial v$ asks how the addition’s output changes when $v$ changes while $w$ stays fixed. This equals $1$, because adding a small amount to $v$ adds the same amount to $f$. The chain rule then connects these local responses back to the original input $y$.

The following table lists the local derivatives along the full path:

Operation	Local derivative
Square $y$ to obtain $u$	$du/dy=2y$
Multiply $u$ by the fixed value $x$ to obtain $v$	$\partial v/\partial u=x$
Add the fixed value $w$ to obtain $f$	$\partial f/\partial v=1$

To obtain the derivative of the final output with respect to $y$, we can now apply the chain rule, multiplying the local derivatives along $y\rightarrow u\rightarrow v\rightarrow f$:

$$\frac{\partial f}{\partial y} =\underbrace{\frac{\partial f}{\partial v}}_{f=v+w} \cdot\underbrace{\frac{\partial v}{\partial u}}_{v=xu} \cdot\underbrace{\frac{du}{dy}}_{u=y^2} =1\cdot x\cdot2y=2xy$$

Each brace names the operation whose local derivative appears above it: addition contributes $1$, multiplication by the fixed $x$ contributes $x$, and squaring contributes $2y$.

The final addition also has a local derivative $\partial f/\partial w=1$. Including both inputs to that addition gives:

$$\frac{\partial f}{\partial y} =\frac{\partial f}{\partial v}\cdot\frac{\partial v}{\partial y} +\frac{\partial f}{\partial w}\cdot\frac{\partial w}{\partial y} =1\cdot(x\cdot2y)+1\cdot0 =2xy$$

The second contribution is zero because $w=x^3$ does not change with $y$ when $x$ is held fixed: $\partial w/\partial y=0$. Although the graph has two branches into the final addition, only one carries a change from $y$ to $f$. When we differentiate with respect to $x$, both branches contribute, as we’ll see next.

This matches the formula we obtained by differentiating $xy^2+x^3$ directly. Here, $\partial f/\partial y$ equals $\partial v/\partial y$ because the addition passes on the change in $v$ while $w$ stays fixed.

The widget below renders a graph of these connected calculations. It starts with $x$ and $y$, the inputs the function depends on. The expressions inside the nodes, such as $u=y^2$ and $v=xu$, define the operations. Solid arrows point from an operation’s result to the operation that uses it. Thus $u\to v$ means that $v=xu$ uses the result of $u=y^2$, making their composition visible. Multiplication uses $u$, not its derivative $du/dy$; the derivative labels and dashed arrows to the chain-rule box show how we differentiate the calculation. Follow the orange path and multiply the local derivatives to obtain $2y\cdot x\cdot1=2xy$.

Select For Y or For X to compare which paths contribute, or For Y,X to see the two partial derivatives together. Each partial derivative holds the other input fixed. The graph starts with the derivative for $y$, holding $x$ fixed. We’ll derive the two-path result for $x$ in the next section.

For YFor XFor Y,X

Scroll the graph sideways to see the chain-rule box.

When we now take the derivative formula $2xy$ and substitute $x=3$ and $y=2$, we obtain $2(3)(2)=12$, matching the partial derivative we calculated earlier. We can evaluate this formula at any input pair without calculating the intermediate function values first.

At $x=3$ and $y=2$, a small change $h$ in $y$ produces approximately $4h$ change in $u$. Multiplication scales that change by $3$, giving approximately $12h$ change in $v$. Adding the unchanged $w$ leaves that change intact.

Multiple paths: adding derivative contributions

We’ve followed how $y$ affects the output through one path. Let’s now find the partial derivative of $f(x,y)=xy^2+x^3$ with respect to $x$, holding $y$ fixed. We use the same intermediate results:

$$u=y^2,\qquad v=xu,\qquad w=x^3,\qquad f=v+w$$

The value $u=y^2$ stays constant, but both $v=xu$ and $w=x^3$ change with $x$. This gives us two paths from $x$ to the final output:

$$x\longrightarrow v=xu\longrightarrow f=v+w \qquad\text{and}\qquad x\longrightarrow w=x^3\longrightarrow f=v+w$$

We already calculated $\partial f/\partial x=y^2+3x^2$ by differentiating $xy^2+x^3$ with respect to $x$, holding $y$ constant. The chain rule gives the same answer using the local derivatives along both paths:

Operation	Local derivative
Multiply $x$ by the fixed value $u$ to obtain $v$	$\partial v/\partial x=u=y^2$
Cube $x$ to obtain $w$	$dw/dx=3x^2$
Add $v$ and $w$ to obtain $f$, considering the effect of $v$	$\partial f/\partial v=1$
Add $v$ and $w$ to obtain $f$, considering the effect of $w$	$\partial f/\partial w=1$

Each local derivative of the addition holds its other input fixed. Both inputs change when $x$ changes, so we need both contributions.

We now apply the multivariable chain rule: multiply the local derivatives along each path, then add the two contributions to obtain the derivative of the final output with respect to $x$:

$$\frac{\partial f}{\partial x} =\frac{\partial f}{\partial v}\cdot\frac{\partial v}{\partial x} +\frac{\partial f}{\partial w}\cdot\frac{dw}{dx} =1\cdot y^2+1\cdot3x^2 =y^2+3x^2$$

The diagram below shows the two paths from $x$ to the final output: $x\to v\to f$ and $x\to w\to f$. Both track the effect of changing $x$, with $y$ held constant. The nodes $y$ and $u=y^2$ are gray because that branch stays fixed; it still supplies the value $u$ used by multiplication. The orange path contributes $1\cdot y^2$, and the purple path contributes $1\cdot3x^2$. The active operation boxes show their local derivatives. The box above multiplies the derivatives along each path, then adds the contributions to obtain $y^2+3x^2$.

The diagram starts with For X selected. Switch to For Y to compare the active paths and local derivatives while keeping the graph in place, or select For Y,X to show them together.

For YFor XFor Y,X

Scroll the graph sideways to follow the paths.

When we now take the derivative formula $y^2+3x^2$ and substitute $x=3$ and $y=2$, we obtain $2^2+3(3^2)=4+27=31$. This matches the partial derivative we calculated earlier, without calculating the intermediate function values first.

Calculating derivatives with rules: symbolic differentiation and autodiff

The examples above combine local derivative rules to differentiate a larger function. We can take that decomposition further: even squaring and cubing can be built from multiplication, so their derivatives can follow from the multiplication rule.

A square’s derivative from multiplication alone

To differentiate a square using multiplication and the chain rule, write $s=x^2$ as $s=x\cdot x$. For a multiplication $s=ab$, the local derivatives are:

$$\frac{\partial s}{\partial a}=b,\qquad \frac{\partial s}{\partial b}=a.$$

Here, both input positions refer to the same variable: $a=x$ and $b=x$. Changing $x$ affects the result through both positions, so we add their contributions:

$$\frac{ds}{dx} =\frac{\partial s}{\partial a}\cdot\frac{da}{dx} +\frac{\partial s}{\partial b}\cdot\frac{db}{dx} =x\cdot1+x\cdot1 =2x.$$

The derivative of the square follows from multiplication and the chain rule; we never needed a separate square rule. At $x=3$, the two input positions contribute $3+3=6$. Even though there is only one variable $x$, both uses of it must contribute.

The same idea works for a cube. Write $t=x\cdot x$, then $c=t\cdot x$. Combining the derivatives gives:

$$\frac{dc}{dx} =x\frac{dt}{dx}+t\frac{dx}{dx} =x(x+x)+x^2 =3x^2.$$

We can therefore differentiate positive integer powers expressed as repeated multiplications without using the power rule. Other functions require additional basic operations: for example, multiplication alone cannot express $\sin(x)$ or arbitrary fractional powers. The general principle is to combine known local derivative rules to differentiate functions built from those operations.

The same rules can produce expressions or numbers

In the square example, we kept $x$ as a symbol, constructed $x\cdot1+x\cdot1$, and simplified it to $2x$. Producing derivative expressions this way is symbolic differentiation. Software can automate that expression construction using the same local rules and dependencies.

Micrograd applies those rules to numerical values. At $x=3$, the original multiplication produces $s=9$, and its backward rule accumulates $3+3=6$ into $x$‘s gradient. This is automatic differentiation: the calculation is mathematically equivalent to evaluating $2x$ at $x=3$, without constructing that symbolic formula. Backpropagation uses reverse-mode automatic differentiation to calculate the loss gradients needed for training.

Both methods compose the same calculus rules. In this comparison, what changes is whether we construct derivative expressions or execute the derivative calculations on numbers:

Method	What it works with	What it produces
Symbolic differentiation	Symbolic expressions, including named intermediate expressions	Derivative formulas that we can evaluate at chosen inputs
Automatic differentiation	A program’s operations and their values at chosen inputs	Numerical derivatives obtained by combining local derivatives

From derivative formulas to executable rules

In the earlier one-path symbolic graph, we wrote $du/dy=2y$ beside the operation $u=y^2$. That formula also tells us how to calculate the derivative using arithmetic: take the input value and multiply it by $2$. At $y=2$, the result is $4$. This is a numerical local derivative: the local derivative evaluated at specific input values.

Autodiff does not infer primitive derivative rules from the values or discover them symbolically during execution. The programmer supplies a rule for each primitive operation, translating its known derivative formula into code. If we choose squaring as a primitive, $2y$ becomes 2 * input_value. The formula explains the rule mathematically; the code executes that same rule on numbers. At runtime, supplying the input value 2 gives 4. If we instead implement squaring as y * y, the two multiplication contributions give the same result, 2 + 2 = 4.

To differentiate combinations of those operations, we use their implemented derivative rules together with the graph and the chain rule. Each local rule is implemented once and reused whenever that operation appears:

Program operation	Known local derivative rule	Numerical calculation
u = y ** 2	$du/dy=2y$	2 * y.data
v = x * u	$\partial v/\partial x=u$, $\partial v/\partial u=x$	u.data, x.data
w = x ** 3	$dw/dx=3x^2$	3 * x.data ** 2
f = v + w	$\partial f/\partial v=1$, $\partial f/\partial w=1$	1, 1

Here, .data holds a node’s numerical value. Multiplication’s derivative with respect to $u$ is obtained by reading the other operand’s value, $x$. Applying this local rule requires only that value, regardless of how it was calculated.

An autodiff implementation also records which inputs produced each result, preserving the dependencies shown in our graph. Ordinary arithmetic calculates values; autodiff adds the dependency tracking and derivative rules. It can also follow operations executed inside loops and conditional branches, without the need to rewrite the entire calculation as a single formula.

For the graph used in our walkthrough, we choose a dedicated power primitive, keeping each square and cube as one operation. Squaring and cubing can then be implemented as y.pow(2) and x.pow(3). This is an implementation choice; repeated multiplication would also work for these powers. A pow(exponent) method applies the general power rule:

$$\frac{d}{dy}(y^n)=ny^{n-1}$$

For the positive integer exponents used here, its calculation is:

Here, this is the input node. The operation calculates its output value and records the connection to that input together with its numerical local derivative. The backward pass will use that saved derivative to propagate a gradient contribution to the input.

Calling y.pow(2) supplies the exponent $2$, so the local derivative becomes $2y^{2-1}=2y$. Calling x.pow(3) gives $3x^{3-1}=3x^2$. The exponent comes from the operation’s argument; the programmer supplies the general derivative rule. The code evaluates that rule using the supplied exponent and input value. A specialized square() method can use the same calculation with the exponent fixed at $2$.

The graph supplies the connections; the implemented rules supply the local derivative values. The chain rule tells us how to combine them.

We can now revisit the table from the symbolic example, again following only $y\to u\to v$ with $x$ held fixed to calculate $\partial v/\partial y$. Autodiff executes the same operations at chosen inputs, so it also calculates their numerical output values. It still needs local derivatives, but it evaluates the rules already implemented for each operation rather than deriving new formulas during execution:

Step	At $x=3$, $y=2$, with $x$ held fixed
Use the same operations	$u=y^2$, then $v=xu$
Calculate the operation values	$u=2^2=4$, then $v=3\cdot4=12$
Evaluate the local derivative rules	$du/dy=2y=4$, $\partial v/\partial u=x=3$
Combine the numerical derivatives using the chain rule	$\partial v/\partial y=(\partial v/\partial u)(du/dy)=3\cdot4=12$

Continuing through the final addition gives $\partial f/\partial y=1\cdot12=12$. This table isolates one input’s effect for explanation. Our symbolic calculation combined $1\cdot x\cdot2y$ into the formula $2xy$. Autodiff evaluates the individual rules at the current inputs and combines $1\cdot3\cdot4$ instead. It uses known derivative rules as numerical instructions, without constructing a symbolic derivative formula for the whole function.

The widget below uses the same graph as the one-path example. Switch between Symbolic and Autodiff to compare combining derivative expressions with combining their numerical values at $x=3$ and $y=2$.

The derivative toggle also lets you compare the one path from $y$ with the two paths from $x$ in either mode. Select For Y,X to see both formulas or both numerical derivatives together.

For YFor XFor Y,X

SymbolicAutodiff

Scroll the graph sideways to see the chain-rule box.

In symbolic mode, the solid arrows show how the functions compose: each intermediate result becomes an input to another operation. In automatic differentiation mode, the same arrows also describe the forward flow of numerical values: squaring produces $u=4$, which multiplication uses with $x=3$ to produce $v=12$. The dependencies stay the same; we now evaluate the operations at specific inputs. This comparison widget does not animate that flow. The step-by-step walkthrough below will show these forward values and the backward propagation of derivative contributions.

Calculating operation values is distinct from calculating local derivatives. Each derivative rule uses whichever values it needs: along this path, the square’s rule needs $y$, and the multiplication’s derivative with respect to $u$ needs $x$. The rows in the numerical calculation table describe different tasks, not separate passes: an implementation can evaluate local derivative rules during its forward or backward pass. In this walkthrough, we evaluate and save them during the forward pass.

The comparison widget displays the evaluated derivatives; the later walkthrough will also display the forward values. The square’s output $u=y^2$ and its local derivative $du/dy=2y$ both happen to equal $4$ at $y=2$, but they are different quantities: at $y=3$, they would be $9$ and $6$.

Evaluating the combined derivative formula gives the same result as evaluating the local derivatives first and then combining them using the chain rule. At $(3,2)$, evaluating $2xy$ gives $12$, as does multiplying $4\times3\times1$. The operations and their dependencies stay the same; what changes is whether we combine derivative expressions or numerical values.

Avoiding an expanded formula matters because repeatedly substituting intermediate expressions can make a derivative formula much larger than the original expression—a problem called expression swell. The expanded formula may contain many copies of the same subexpression, taking more space and repeating work if each copy is evaluated separately. Keeping named intermediate results lets us reuse those calculations. Symbolic differentiation can preserve and share these expressions too, so expression swell is not an unavoidable limitation of the method.

Forward and reverse modes

The widget shows which local derivatives we combine, but not the order in which we propagate them. Its dashed connections collect the factors into a product. Forward-mode and reverse-mode autodiff can use the same computation graph, and both evaluate the function from inputs to output. They differ in how derivatives propagate: forward mode carries them from inputs toward the outputs, while reverse mode carries derivative contributions from an output back toward the inputs.

Starting with one chosen input, forward mode can calculate how all the outputs respond to that input in one derivative pass. Starting with one chosen output, reverse mode can calculate how that output responds to all the inputs in one backward pass. We’ll focus on reverse mode, the approach used by backpropagation. The complete backward pass will compute both $\partial f/\partial x$ and $\partial f/\partial y$, accumulating contributions from both paths to $x$.

A forward-mode derivative pass propagates derivatives alongside the function’s values. The forward pass of our reverse-mode calculation instead evaluates the function and records what the later backward pass needs; the derivatives of the final output are accumulated during that backward pass. We’ll follow this calculation in two passes:

• The forward pass evaluates the function. In this walkthrough, each operation calculates its value and saves its input connections and numerical local derivatives.
• The backward pass starts with the derivative of the output with respect to itself, $\partial f/\partial f=1$, and uses the saved local derivatives to propagate numerical contributions backward using the chain rule. Contributions to the same input accumulate.

The table below shows what our forward pass calculates and stores at $x=3$ and $y=2$. An operation’s type determines which derivative rule runs; the stored edges pair its inputs with the evaluated local derivatives:

Calculation	Output value at $(x,y)=(3,2)$	Stored input connections and local derivatives
$u=y^2$	$4$	Input $y$, with $du/dy=4$
$v=xu$	$12$	Input $x$, with $\partial v/\partial x=4$; input $u$, with $\partial v/\partial u=3$
$w=x^3$	$27$	Input $x$, with $dw/dx=27$
$f=v+w$	$39$	Input $v$, with $\partial f/\partial v=1$; input $w$, with $\partial f/\partial w=1$

The output values and the local derivatives have different roles. For example, the multiplication uses $u=4$ to calculate its output $v=12$, and that same operand value supplies its local derivative with respect to $x$. Backward combines the stored local derivatives with accumulated sensitivities to the final output.

A model is a program, so autodiff has access to the operations that produced its output; a finite table of input–output observations alone cannot provide that information or determine exact derivatives. For a fixed training batch, the model’s prediction-and-loss calculation has many parameters and one loss value. Reverse mode reuses intermediate results to compute all the parameter derivatives in one backward pass, rather than repeating the whole calculation for each parameter.

We’ve established what the forward pass records and how the backward pass uses local derivative rules. Before tracing the full calculation, let’s look at how one multiplication uses its values during backward.

One value, two mathematical roles

In Micrograd, each Value object stores a forward value in .data and accumulates the derivative of the chosen final output in .grad. Start with two independent inputs and multiply them:

The multiplication creates a third Value connected to both inputs. After z.backward(), the objects contain:

flowchart LR x["x: Value(data=2, grad=3)"] --> z["z = x * y: Value(data=6, grad=1)"] y["y: Value(data=3, grad=2)"] --> z

Here, $z$ is the final output, so backward starts with z.grad = 1: $\partial z/\partial z=1$. The multiplication rule then gives:

Input receiving a contribution	Local scaling factor	Backward update
x	$\partial z/\partial x=y=3$	x.grad += y.data * z.grad → $3\times1=3$
y	$\partial z/\partial y=x=2$	y.grad += x.data * z.grad → $2\times1=2$

The same y.data = 3 has two mathematical roles: it is an operand used to calculate $z=6$, and it equals the scaling factor that tells us how $z$ responds to changes in $x$. The operation’s derivative rule explains why a forward value can supply a local derivative. .data continues to store the forward value throughout backward.

This is why Micrograd’s multiplication backward rule can look deceptively simple:

Three ideas meet in this line: other.data supplies a saved forward value; the multiplication rule identifies that value as the local derivative with respect to self; multiplying it by out.grad propagates a contribution using the chain rule. Micrograd evaluates that rule numerically without storing a separate symbolic derivative expression.

The same distinction applies to a neuron computing $z=wx+b$. Its local derivative with respect to the weight is $x$, so for a loss $L(z)$, the loss gradient is:

$$\frac{\partial L}{\partial w}=\frac{\partial L}{\partial z}\,x.$$

Even when we only want gradients for the weights, we still need the input values to evaluate them. This local step uses $x$ and the upstream gradient; the upstream gradient can itself depend on $w$ through the forward calculation.

There are two mathematical roles for multiplication to distinguish here: multiplication as the operation being differentiated, and multiplication of derivatives to apply the chain rule. For $z=xy$, differentiating with respect to $x$ gives $y$, and differentiating with respect to $y$ gives $x$. Obtaining either local derivative only requires reading the other operand’s value. The * in other.data * out.grad then multiplies that local derivative by the upstream gradient. This chain-rule multiplication is needed even when the forward operation is addition, whose local derivative is $1$; its backward code can omit the multiplication by $1$.

The same * syntax hides different mathematical roles

When we translate these derivative calculations into code, two mathematically distinct steps can look exactly alike. Backward code uses the same * operator both to evaluate local derivative formulas and to apply the chain rule. Both are ordinary numerical multiplication, so the code’s syntax does not distinguish their purposes. This makes compact backward expressions easy to misread.

For a square primitive $z=x^2$, both roles appear in x.grad += (2 * x.data) * z.grad. Separating that expression makes the distinction visible:

The first * evaluates the local derivative formula; the second combines that derivative with the upstream gradient. To understand a * in backward code, ask what its factors represent. In 2 * x.data, they belong to the square’s derivative formula. In local_derivative * z.grad, they are two sensitivities combined by the chain rule.

Returning to $z=xy$ at $x=2$, $y=3$, suppose we build a fresh graph, extend it with $L=z^2$, and call L.backward(). Here, $z$ is an intermediate value and $L$ names the final output whose derivatives we want. The multiplication receives z.grad = 2 * z.data = 12. It then contributes $3\times12=36$ to x.grad and $2\times12=24$ to y.grad. Its local derivatives are still $3$ and $2$; the upstream gradient now includes the effect of the later square.

The multiplication by out.grad always applies the chain rule, including when that gradient is the seed $1$. As a graph grows, the incoming gradient incorporates downstream contributions one operation at a time. A square uses the same propagation mechanism, but its local factor is 2 * input.data; using the other operand’s value directly is specific to multiplication.

Reverse-mode autodiff step by step

Let’s now follow a complete reverse-mode autodiff calculation using the same function $f(x,y)=xy^2+x^3$ at $(x,y)=(3,2)$. We’ll evaluate the four operations from the table above, then follow their derivatives backward to obtain the gradient $(31,12)$:

The forward pass gives $u=4$, $v=12$, $w=27$, and the final output $f=39$. Each result keeps its connections to the inputs that produced it, along with the information needed for its local derivative rule. The backward pass calculates how the final output responds to each node; it does not change these forward values.

The forward values are also called primals. The derivatives accumulated during the backward pass are called adjoints: each measures how the final output changes with respect to a particular input or intermediate value. In our widget, these are labeled value and grad, respectively. An adjoint is a derivative of the final output, distinct from an operation’s local derivative.

Before the backward pass, every .grad starts at zero. We seed f.grad = 1, because $\partial f/\partial f=1$.

Each scalar node’s .grad stores the derivative of the final output with respect to that node. Together, x.grad and y.grad form the gradient vector $(31,12)$. The walkthrough stores numerical local derivatives on the connections and uses them during backward: parent.grad += node.grad * local_derivative.

The earlier comparison widget followed only the path from $y$ to calculate $\partial f/\partial y$. The widget below processes the whole graph in one backward pass, calculating the derivatives with respect to both $x$ and $y$. These steps happen sequentially within the same pass. Each box shows its output value, numerical local derivatives, and accumulated gradient separately.

The inputs are fixed at $x=3$ and $y=2$, equivalent to creating x = Value(3) and y = Value(2). Select Evaluate forward to see the four operations produce the output. Then select Seed output gradient and follow the backward steps through addition, cubing, multiplication, and squaring. Each backward step highlights the operation and its inputs, then connects the symbolic chain-rule expression to its numerical substitution and the corresponding .grad update below the graph. Watch how the two contributions accumulate at $x$.

The same operations, forward and backwardf(x, y) = xy² + x³

Evaluate forwardRestart trace

Stage 0 of 6. Start with x = 3 and y = 2, then evaluate u = y², v = xu, w = x³, and f = v + w.

The following table summarizes the backward steps you can follow in the widget. The forward values stay unchanged throughout; only the accumulated gradients change.

Backward step	Gradient update
Seed	$\mathtt{f.grad}=1$
Addition	$\mathtt{v.grad}=\mathtt{f.grad}\cdot1=1$, $\mathtt{w.grad}=\mathtt{f.grad}\cdot1=1$
Cube	$\mathtt{x.grad}=\mathtt{w.grad}\cdot3x^2=1\cdot27=27$
Multiplication, contribution to $x$	Add $\mathtt{v.grad}\cdot u=1\cdot4=4$, giving $\mathtt{x.grad}=27+4=31$
Multiplication, contribution to $u$	$\mathtt{u.grad}=\mathtt{v.grad}\cdot x=1\cdot3=3$
Square	$\mathtt{y.grad}=\mathtt{u.grad}\cdot2y=3\cdot4=12$

The two paths from $x$ contribute separately, while $y$ receives its contribution through $u$ and $v$:

$$\frac{\partial f}{\partial x}=\underbrace{1\times27}_{\text{through }w}+\underbrace{1\times4}_{\text{through }v}=31, \qquad \frac{\partial f}{\partial y}=1\times3\times4=12$$

This order processes the cube branch before the multiplication branch. Other valid backward orders give the same result, provided a node receives all downstream contributions before it passes its accumulated derivative to its inputs.

The widget shows the full backward pass. Let’s now connect its multiplication $v=xu$ and square $u=y^2$ steps to the code. The multiplication’s backward step multiplies two different derivatives:

$$\underbrace{\frac{\partial f}{\partial v}}_{\text{stored in v.grad}} \times \underbrace{\frac{\partial v}{\partial x}}_{\text{u.data, the other operand's value}} =1\times4$$

The first factor tells us how the final output $f$ responds to $v$. The second tells us how $v$ responds to $x$, holding $u$ fixed. Multiplying them is the chain rule: it gives this path’s contribution to the derivative with respect to $x$.

The walkthrough saves numerical local derivatives during the forward pass. Another implementation can save the operand values and evaluate the same local rules during backward. In that form, the multiplication method __mul__(self, other) creates a result node called out, and its backward rule can be written as:

These two lines handle the two input positions of one multiplication. Every executed operation has its own backward step, so a larger computation introduces more operations, each reusing the same small rules. The incoming out.grad already accounts for how this operation’s result affects the final output. The multiplication’s backward rule only needs that accumulated derivative and the operand values.

For v = x * u, out is $v$, self is $x$, and other is $u$. Written with those names, the multiplication updates are followed by the square’s update:

The forward pass supplied $u=4$, $x=3$, and $y=2$. Multiplication uses the other operand’s value for each local derivative. The square uses $2y=4$ and multiplies it by the accumulated u.grad = 3.

Earlier, symbolic differentiation gave $\partial f/\partial x=y^2+3x^2$. The cube’s and multiplication’s x.grad += ... updates perform that addition numerically: the cube contributes $27$, and the multiplication contributes $4$, giving $31$. The += in backward code implements the addition of derivative contributions in calculus. Using = would overwrite the cube’s contribution with the multiplication’s contribution, leaving $4$ instead of $31$.

The same accumulation explains our earlier square example, s = x * x. Here, self and other reference the same Value object, so both backward lines update the same x.grad. Starting with x.grad = 0 and s.grad = 1, at $x=3$ they accumulate $3+3=6$. If we instead multiply two separately created Value(3) objects, they represent distinct input nodes, each receiving a gradient of $3$. Contributions accumulate at the node they describe; equal numerical values do not make two nodes the same input.

Code	Meaning
out.grad	Derivative of the final output with respect to this operation’s result
other.data	Other operand’s value, which equals the local derivative for multiplication
+=	Add this path’s contribution to the input’s accumulated derivative

other.data stores a value, not a derivative. The multiplication rule tells us that this value equals the local derivative we need. For addition, the local derivative is $1$, so the corresponding update is self.grad += out.grad * 1. For a power, we evaluate the power rule at the input: the cube’s local derivative is $3x^2$, and the square’s is $2y$.

In the symbolic widget, we combined the local derivative expressions along the path from $y$:

$$\frac{\partial f}{\partial y} =\frac{\partial f}{\partial v}\cdot\frac{\partial v}{\partial u}\cdot\frac{du}{dy} =1\cdot x\cdot2y =2xy$$

The code follows the same connections but combines numbers. The addition gives v.grad = 1; multiplication gives u.grad = 1 × 3 = 3; the square gives y.grad = 3 × 4 = 12. Thus u.grad already contains the first two factors of the symbolic expression. Autodiff obtains the same derivative without constructing the combined formula $2xy$.

These steps calculate derivatives at $(3,2)$; they do not update the inputs to minimize the function. The Value class automates the backward steps in a single call to f.backward().

From the calculation to an engine

A derivative measures local sensitivity. Partial derivatives isolate each input’s effect, and the gradient collects those measurements into a vector. Backpropagation calculates them by working backward through recorded operations: multiply derivatives along each path and add contributions where paths meet.

A neuron computes a weighted sum followed by an activation. The engine needs derivative rules for the primitive operations used in that calculation and in the loss, but no special derivative rule for a neuron or layer. Multiplication and addition cover the weighted sum; activations and loss functions must also be built from supported operations or supplied with their own derivative rules. The same operation rules work wherever those operations appear, so the engine can differentiate an entire network by following its computation graph.

To see these ideas implemented in Python, follow Andrej Karpathy’s Micrograd walkthrough. The local derivative rules, gradient accumulation, and backward traversal we explored here provide the foundations for understanding his implementation.

For a deeper mathematical view, explore this talk on automatic differentiation and linear maps. It connects calculus with linear algebra by showing how derivatives act as linear maps, how Jacobian matrices represent them, and how the chain rule combines them—the mathematical foundations of automatic differentiation.