What breaks when backpropagation forgets a path?

This experiment follows the shared-variable computation in From derivatives to backpropagation. You need the product rule and the idea that changing one input can affect an output through several paths.

The forward computation is $a=x+y$, followed by $z=ax$. The same $x$ appears inside the sum and as a multiplication input. We will deliberately omit a backward contribution while preserving the complete forward computation. This is a diagnostic experiment, not a replacement forward model.

Establish a reference

Use $x=2$, $y=3$, both backward paths, and difference step $h=0.001$. Predict the forward output and the total derivative with respect to $x$. Record your reasoning, then run the reference.

The bench also computes a central difference, $[z(x+h)-z(x-h)]/(2h)$, with $y$ fixed. Treat it as an independent numerical check, not an exact oracle at every step size.

Remove one backward contribution

Keep $x$, $y$, and $h$ fixed. Select Direct multiplication path only, predict the gradient and forward output, and record a run. Then select Path through the sum only and repeat the comparison.

Record both path contributions, the reported gradient, and its discrepancy from the finite-difference estimate. Does an unchanged forward result guarantee that the backward calculation is correct?

Experiment bench

Load reference settings

Forward computation: a = x + y, z = a × x. The backward-path setting changes only the reported gradient. Central differences always evaluate the complete forward function with y fixed.

Load the reference, write a prediction, and run it first. For each comparison, change only the condition named in the lesson. Each run is calculated here in your browser.

Input x-3-2-10123Fixed input y-3-2-10123Backward pathsBoth pathsDirect multiplication path onlyPath through the sum onlyDifference step h0.10.0010.0000011e-121e-16

Prediction and reason for this run

Run experimentReference settings selected · 0/16 runs

Loading reference settings preserves your evidence. Work is saved in this tab across navigation and reloads. Download your report to keep it after closing the tab.

No run recorded yet.

Build your explanation from the evidence

Observations: which runs and values are you comparing?Interpretation: what explains the result?Limitations: what does this experiment leave unresolved?Next test: what will you change, and what result would challenge your explanation?

Download reportClear runs

Explain which dependence was lost in each modified run before opening the calculation.

Compare the paths

At $(x,y)=(2,3)$, $a=5$ and $z=10$. The direct multiplication input contributes $a=5$; the route through $a$ contributes $x\times1=2$. Adding them gives 7. Keeping only one path gives 5 or 2, even though the forward output remains 10 in both cases.

Expanding $z=x(x+y)=x^2+xy$ gives $\partial z/\partial x=2x+y$. It agrees with $(x+y)+x$. A central difference is exact for this quadratic in exact arithmetic, but the computer evaluates rounded numbers.

Design a test that could miss the bug

Choose inputs for which omitting one path would produce the right numerical gradient by accident. State which contribution you expect to vanish, then test your case. Keep $h=0.001$.

Next choose a second input that exposes the same omission. Explain why a gradient check at just your first point would provide weak evidence about the implementation.

As a separate numerical experiment, restore the reference and both paths. Change only $h$ through $0.1$, $0.000001$, $10^{-12}$, and $10^{-16}$. Predict whether the smallest step must give the best check, then compare the recorded discrepancies. Distinguish a small nonzero rounding discrepancy from an estimate that collapses to zero.

For your independent comparisons, use Comparison baseline to select a recorded run that differs in only the condition you are testing.

Report and transfer

Distinguish an omitted derivative contribution from a finite-difference rounding error. Use at least two input points in your explanation and state which settings stayed fixed in each comparison.

On paper, consider $q=x\times x$. Both multiplication inputs refer to the same variable. Should a backward pass process one contribution or two? Predict the result of accidentally keeping only one, then explain how you would detect the mistake.

Check your diagnostic reasoning

At $x=0$, the contribution through the sum vanishes. At $x+y=0$, the direct contribution vanishes. An implementation that omits that contribution can pass a check at such a point and fail elsewhere. At $x=2$, adding or subtracting $10^{-16}$ can round back to the same representable input, making the difference estimate zero despite a nonzero derivative.

For $q=x\times x$, two operand edges refer to the same variable. Each contributes $x$, so the total derivative is $2x$. Deduplicating the variable must not discard one of its edge contributions.

Continue experimenting

Previous: When does a larger learning rate stop helping? · Next: What changes when we remove the hidden activation? · All machine learning experiments