What makes a function linear?
If increasing an input by 1 always increases the output by 2, have we established that the function is linear? What if its graph is a straight line but it does not pass through zero?
We will investigate using ordinary numbers first, then connect the result to a neuron’s weighted sum. You need to be comfortable substituting numbers into a formula: for example, if , then . Here the inputs can be any real numbers, including negative numbers.
Give the input the same increase
Start with . Before using the explorer, write your prediction and a reason: will the output change by the same amount when the input moves from 1 to 2 as when it moves from 2 to 3?
Record the reference at = 1. Then change only the starting point to 2. Keep the function and the input step of +1 fixed. Record both input endpoints, both output endpoints, and the output difference.
Input: 1 → 2. Input change: +1.
Output: 2 → 4. Output change: 2.
The input step is fixed at +1. Reset restores f(x) = 2x and starting x = 1. Record your settings and results in your notebook before changing a control.
Now repeat that two-row comparison for . Return the starting point to 1, select the new function, and write your prediction before moving it to 2. Finally, repeat for . Within each pair, only the starting point changes; compare different functions at matching starting points.
Use this record format in a notebook or text file:
| Function | Input interval | Output endpoints | Output difference |
|---|---|---|---|
| … | … | ||
| … | … | ||
| … | … | ||
| … | … | ||
| … | … | ||
| … | … |
Which function responds differently to the same input increase at different starting points? Which two functions have matching changes but different output values? Explain your observations before opening the calculation.
Compare the output changes
| Function | Input interval | Output endpoints | Output difference |
|---|---|---|---|
For , the same +1 input step produces a different output change at different starting points. In fact,
so that change depends on . For , the added 3 cancels when we subtract the two outputs, leaving the same change as .
Equal input increases producing different output increases can reveal nonlinearity. But observing equal increases does not establish linearity: we have tested only a few cases, and a constant offset can survive unnoticed. We need a rule about what the function preserves.
Combine before or after applying the function
A linear function preserves addition and scaling:
Both identities must hold for every allowed , , and real scale . On the left we combine the inputs first. On the right we apply the function first, then combine the outputs. A linear function gives the same answer by either route. Together, these rules also give .
Use , , and as the reference settings. Before revealing the explorer’s results, calculate both sides of each rule for . Then predict what will happen if we change only the function to , and then to .
f(0) = 0. f(x) = 2. f(y) = 4.
Addition: f(x + y) = f(3) = 6; f(x) + f(y) = 6. Matches in this case.
Scaling: f(kx) = f(2) = 4; k f(x) = 4. Matches in this case.
Reset restores f(x) = 2x, x = 1, y = 2, k = 2. A matching example is not a proof for every input. Record your settings and results in your notebook before changing a control.
Keep the three number controls fixed while switching functions. Record both sides of each rule, including the cases that fail. Explain why an output offset matters here even though it canceled in the earlier output differences.
Compare the two routes
| Function | ||||
|---|---|---|---|---|
For , applying the function separately adds the offset twice. Applying it to the combined input adds the offset only once. That is why addition fails. Squaring also fails: , whereas .
One exact failure disproves linearity. The successful row for is evidence, but the general argument uses arbitrary inputs:
These identities hold for all real inputs and scales, so this function is linear.
A straight line can be affine
For a real number as both input and output, every linear function has the form : set in the scaling rule to see that , with .
Adding an offset gives an affine function, . Its graph is still a straight line. When , however, it is not linear in the linear-algebra sense. Some school courses call any straight-line function “linear”; here we distinguish linear from affine because the addition and scaling rules matter.
Every linear function sends zero to zero: . This is a necessary condition, not a sufficient one. The function sends zero to zero too, yet our counterexample disproved its linearity.
| Form | Example | Classification |
|---|---|---|
| Linear; also a special case of affine | ||
| , with | Affine, not linear | |
| A function that cannot be written as | Nonlinear and non-affine |
Strictly, “nonlinear” means “not linear,” so it can include a nonzero affine offset. In neural-network discussions, “nonlinear activation” usually refers to a non-affine function such as ReLU. A bias alone does not provide that kind of nonlinearity.
Why positive tests can miss ReLU’s nonlinearity
ReLU is : it keeps positive inputs and replaces negative ones with zero. On positive inputs it agrees with the identity function .
In the rules explorer, select , then . Keep , , and . Record what passes. Predict whether addition will still hold if you change only to , then test it.
Restore . Write a separate prediction for changing only from 2 to , then test it. Explain which part of the definition the positive-input tests missed.
Check the negative cases
With and , addition fails:
With and , scaling fails:
ReLU does preserve scaling by nonnegative numbers. But linearity requires all real scales and all input pairs. Its two straight pieces do not make it one linear function over the whole real line.
Design a test that could change your mind
Choose a claim to investigate: “a function that sends zero to zero is linear,” or “passing addition and scaling on positive inputs establishes linearity.” Choose your own function and inputs; you can use the explorer or calculate another function on paper.
- Write the claim, your reference settings, and your prediction with a reason.
- Choose one input or scale to change. State which settings stay fixed and what observation would contradict the claim.
- Record both sides of the relevant rule before and after the change. Keep passing and failing cases.
- Explain what the evidence establishes. If everything matches, either give an algebraic argument for all inputs or propose an untested case that might expose a failure.
Hints and criteria for checking your investigation
For the first claim, find a function with , then an exact failure of addition or scaling. For the second, consider a function that behaves differently on opposite sides of zero, such as or ReLU. Choose values different from the worked example.
A counterexample must include unequal numerical results for the two sides of a required identity. Several matching examples cannot prove a statement about every real input. If your function is linear, distribute and rearrange symbols to show that both rules hold generally.
Back to a neuron
For fixed weights, a neuron’s weighted sum is linear in its input vector. Dot products preserve vector addition and scaling just as did for numbers:
Adding a nonzero bias makes affine. An identity activation leaves that calculation unchanged. A nonlinear activation such as ReLU can make the response depend on which region the input reaches. It can behave linearly within one region without being linear everywhere.
This distinction explains why a stack of affine layers still computes an affine function: expanding gives . Non-affine activations allow the network to represent functions beyond that family.
Before continuing, classify and . Both give the same output for every input. Does that make both linear? Use the rules rather than the appearance of the graph.
Check the constant functions
The zero function is linear: both sides of each rule equal zero for all inputs and scales. The function is affine and not linear: , and while .
Continue with what a weight matrix actually does to apply these distinctions to a whole layer.