What makes a function linear?

If increasing an input by 1 always increases the output by 2, have we established that the function is linear? What if its graph is a straight line but it does not pass through zero?

We will investigate using ordinary numbers first, then connect the result to a neuron’s weighted sum. You need to be comfortable substituting numbers into a formula: for example, if f(x)=2xf(x)=2x, then f(3)=6f(3)=6. Here the inputs can be any real numbers, including negative numbers.

Give the input the same increase

Start with f(x)=2xf(x)=2x. Before using the explorer, write your prediction and a reason: will the output change by the same amount when the input moves from 1 to 2 as when it moves from 2 to 3?

Record the reference at Starting x = 1. Then change only the starting point to 2. Keep the function and the input step of +1 fixed. Record both input endpoints, both output endpoints, and the output difference.

Compare equal input steps
1
f(x) = 2x: an input step of oneFrom (1, 2) to (2, 4). The output changes by 2. Plot scales stay fixed when you change the function or starting point.-4-2024-80816xf(x)
Blue: function. Orange: endpoints and the input/output changes. Axes stay fixed.

Input: 1 → 2. Input change: +1.

Output: 2 → 4. Output change: 2.

The input step is fixed at +1. Reset restores f(x) = 2x and starting x = 1. Record your settings and results in your notebook before changing a control.

Now repeat that two-row comparison for f(x) = x². Return the starting point to 1, select the new function, and write your prediction before moving it to 2. Finally, repeat for f(x) = 2x + 3. Within each pair, only the starting point changes; compare different functions at matching starting points.

Use this record format in a notebook or text file:

FunctionInput intervalOutput endpointsOutput difference
2x2x1→21\to2……
2x2x2→32\to3……
x2x^21→21\to2……
x2x^22→32\to3……
2x+32x+31→21\to2……
2x+32x+32→32\to3……

Which function responds differently to the same input increase at different starting points? Which two functions have matching changes but different output values? Explain your observations before opening the calculation.

Compare the output changes
FunctionInput intervalOutput endpointsOutput difference
2x2x1→21\to22→42\to422
2x2x2→32\to34→64\to622
x2x^21→21\to21→41\to433
x2x^22→32\to34→94\to955
2x+32x+31→21\to25→75\to722
2x+32x+32→32\to37→97\to922

For x2x^2, the same +1 input step produces a different output change at different starting points. In fact,

(x+1)2−x2=2x+1,(x+1)^2-x^2=2x+1,

so that change depends on xx. For 2x+32x+3, the added 3 cancels when we subtract the two outputs, leaving the same change as 2x2x.

Equal input increases producing different output increases can reveal nonlinearity. But observing equal increases does not establish linearity: we have tested only a few cases, and a constant offset can survive unnoticed. We need a rule about what the function preserves.

Combine before or after applying the function

A linear function preserves addition and scaling:

f(x+y)=f(x)+f(y),f(kx)=k f(x).f(x+y)=f(x)+f(y),\qquad f(kx)=k\,f(x).

Both identities must hold for every allowed xx, yy, and real scale kk. On the left we combine the inputs first. On the right we apply the function first, then combine the outputs. A linear function gives the same answer by either route. Together, these rules also give f(ax+by)=a f(x)+b f(y)f(ax+by)=a\,f(x)+b\,f(y).

Use x=1x=1, y=2y=2, and k=2k=2 as the reference settings. Before revealing the explorer’s results, calculate both sides of each rule for f(x)=2xf(x)=2x. Then predict what will happen if we change only the function to 2x+32x+3, and then to x2x^2.

Test addition and scaling
1
2
2

f(0) = 0. f(x) = 2. f(y) = 4.

Addition: f(x + y) = f(3) = 6; f(x) + f(y) = 6. Matches in this case.

Scaling: f(kx) = f(2) = 4; k f(x) = 4. Matches in this case.

Reset restores f(x) = 2x, x = 1, y = 2, k = 2. A matching example is not a proof for every input. Record your settings and results in your notebook before changing a control.

Keep the three number controls fixed while switching functions. Record both sides of each rule, including the cases that fail. Explain why an output offset matters here even though it canceled in the earlier output differences.

Compare the two routes
Functionf(1+2)f(1+2)f(1)+f(2)f(1)+f(2)f(2×1)f(2\times1)2f(1)2f(1)
2x2x66664444
2x+32x+3991212771010
x2x^299554422

For 2x+32x+3, applying the function separately adds the offset twice. Applying it to the combined input adds the offset only once. That is why addition fails. Squaring also fails: (1+2)2=9(1+2)^2=9, whereas 12+22=51^2+2^2=5.

One exact failure disproves linearity. The successful row for 2x2x is evidence, but the general argument uses arbitrary inputs:

2(x+y)=2x+2y,2(kx)=k(2x).2(x+y)=2x+2y,\qquad 2(kx)=k(2x).

These identities hold for all real inputs and scales, so this function is linear.

A straight line can be affine

For a real number as both input and output, every linear function has the form f(x)=mxf(x)=mx: set x=1x=1 in the scaling rule to see that f(k)=kf(1)f(k)=k f(1), with m=f(1)m=f(1).

Adding an offset gives an affine function, f(x)=mx+bf(x)=mx+b. Its graph is still a straight line. When b≠0b\ne0, however, it is not linear in the linear-algebra sense. Some school courses call any straight-line function “linear”; here we distinguish linear from affine because the addition and scaling rules matter.

Every linear function sends zero to zero: f(0)=f(0×x)=0f(x)=0f(0)=f(0\times x)=0f(x)=0. This is a necessary condition, not a sufficient one. The function x2x^2 sends zero to zero too, yet our counterexample disproved its linearity.

FormExampleClassification
mxmx2x2xLinear; also a special case of affine
mx+bmx+b, with b≠0b\ne02x+32x+3Affine, not linear
A function that cannot be written as mx+bmx+bx2x^2Nonlinear and non-affine

Strictly, “nonlinear” means “not linear,” so it can include a nonzero affine offset. In neural-network discussions, “nonlinear activation” usually refers to a non-affine function such as ReLU. A bias alone does not provide that kind of nonlinearity.

Why positive tests can miss ReLU’s nonlinearity

ReLU is f(x)=max⁡(0,x)f(x)=\max(0,x): it keeps positive inputs and replaces negative ones with zero. On positive inputs it agrees with the identity function f(x)=xf(x)=x.

In the rules explorer, select Reset, then f(x) = max(0, x). Keep x=1x=1, y=2y=2, and k=2k=2. Record what passes. Predict whether addition will still hold if you change only Input y to −2-2, then test it.

Restore y=2y=2. Write a separate prediction for changing only Scale k from 2 to −1-1, then test it. Explain which part of the definition the positive-input tests missed.

Check the negative cases

With x=1x=1 and y=−2y=-2, addition fails:

f(1−2)=f(−1)=0,f(1)+f(−2)=1+0=1.f(1-2)=f(-1)=0,\qquad f(1)+f(-2)=1+0=1.

With x=1x=1 and k=−1k=-1, scaling fails:

f(−1)=0,−f(1)=−1.f(-1)=0,\qquad -f(1)=-1.

ReLU does preserve scaling by nonnegative numbers. But linearity requires all real scales and all input pairs. Its two straight pieces do not make it one linear function over the whole real line.

Design a test that could change your mind

Choose a claim to investigate: “a function that sends zero to zero is linear,” or “passing addition and scaling on positive inputs establishes linearity.” Choose your own function and inputs; you can use the explorer or calculate another function on paper.

  1. Write the claim, your reference settings, and your prediction with a reason.
  2. Choose one input or scale to change. State which settings stay fixed and what observation would contradict the claim.
  3. Record both sides of the relevant rule before and after the change. Keep passing and failing cases.
  4. Explain what the evidence establishes. If everything matches, either give an algebraic argument for all inputs or propose an untested case that might expose a failure.
Hints and criteria for checking your investigation

For the first claim, find a function with f(0)=0f(0)=0, then an exact failure of addition or scaling. For the second, consider a function that behaves differently on opposite sides of zero, such as ∣x∣|x| or ReLU. Choose values different from the worked example.

A counterexample must include unequal numerical results for the two sides of a required identity. Several matching examples cannot prove a statement about every real input. If your function is linear, distribute and rearrange symbols to show that both rules hold generally.

Back to a neuron

For fixed weights, a neuron’s weighted sum c(x)=w⊤xc(x)=w^\top x is linear in its input vector. Dot products preserve vector addition and scaling just as 2x2x did for numbers:

w⊤(x+y)=w⊤x+w⊤y,w⊤(kx)=k(w⊤x).w^\top(x+y)=w^\top x+w^\top y,\qquad w^\top(kx)=k(w^\top x).

Adding a nonzero bias makes w⊤x+βw^\top x+\beta affine. An identity activation ϕ(z)=z\phi(z)=z leaves that calculation unchanged. A nonlinear activation such as ReLU can make the response depend on which region the input reaches. It can behave linearly within one region without being linear everywhere.

This distinction explains why a stack of affine layers still computes an affine function: expanding A(Bx+b)+cA(Bx+b)+c gives ABx+(Ab+c)ABx+(Ab+c). Non-affine activations allow the network to represent functions beyond that family.

Before continuing, classify f(x)=0f(x)=0 and g(x)=5g(x)=5. Both give the same output for every input. Does that make both linear? Use the rules rather than the appearance of the graph.

Check the constant functions

The zero function is linear: both sides of each rule equal zero for all inputs and scales. The function g(x)=5g(x)=5 is affine and not linear: g(0)=5g(0)=5, and g(x+y)=5g(x+y)=5 while g(x)+g(y)=10g(x)+g(y)=10.

Continue with what a weight matrix actually does to apply these distinctions to a whole layer.