Why does the angle change while the dot product stays the same?

In Feature detectors: how neurons read and write, a message becomes three word counts, ordered claim, prize, meeting. The input is x=(2,1,1)\mathbf x=(2,1,1). Two readers look at that same input:

w1=(1.5,1,0)(green, promotional),w2=(0,0,2)(purple, routine).\mathbf w_1=(1.5,1,0)\quad\text{(green, promotional)}, \qquad \mathbf w_2=(0,0,2)\quad\text{(purple, routine)}.

The puzzle appears when we increase claim from 2 to 3. The blue input arrow turns farther away from the purple reader, yet the purple dot-product score stays at 2. If the dot product measures alignment, shouldn’t it decrease?

This note investigates that specific case. You only need to know how coordinates locate an arrow and how to multiply and add its weighted inputs. We will build the geometry from there. Here, “score” means the reader’s dot product before any bias or activation, not the classifier’s final output.

First, reproduce the puzzle

Before moving a control, write a prediction and a reason: with both readers fixed, what will increasing claim do to the purple angle, purple score, and input length? Predict the green score too.

The explorer starts at the article’s reference, (2,1,1)(2,1,1). Change only Claim from 2 to 3; leave prize and meeting at 1. Compare the current row with the reference row. Record the input, its length, both reader angles, and both scores in a notebook or text file.

The plot shows the blue input and purple reader. The green reader’s measurements appear below it for comparison. Rotating the camera changes your view, not the vectors or their measurements.

The article’s input and readers
Input and purple reader in three dimensionsInput (2, 1, 1) in blue. Purple reader (0, 0, 2). Angle 65.9°.Drag to rotate, Shift-drag to pan, and scroll to zoom. Arrow keys rotate; Shift with arrows pans; plus and minus zoom; zero resets the view.claimprizemeetingθxw₂

Drag to rotate · Shift-drag to pan · Scroll to zoom

Blue: input xPurple: reader w₂Teal: projection onto w₂
Input x = (2, 1, 1) · Length 2.449

Purple w₂ = (0, 0, 2)

3D angle: 65.9° · Score: 2

w₂ᵀx = 0 × 2 + 0 × 1 + 2 × 1 = 2

Green w₁ = (1.5, 1, 0)

3D angle: 25.1° · Score: 4

w₁ᵀx = 1.5 × 2 + 1 × 1 + 0 × 1 = 4

Compare with the article reference · rounded measurements
InputLengthPurple anglePurple scoreGreen angleGreen score
Reference (2, 1, 1)2.44965.9°225.1°4
Current (2, 1, 1)2.44965.9°225.1°4

What changed? What stayed fixed? Try to explain the result in one sentence before opening the calculation.

Compare the two weighted sums

For the purple reader, only the meeting coordinate contributes:

(0,0,2)⋅(2,1,1)=0×2+0×1+2×1=2,(0,0,2)⋅(3,1,1)=0×3+0×1+2×1=2.\begin{aligned} (0,0,2)\cdot(2,1,1)&=0\times2+0\times1+2\times1=2,\\ (0,0,2)\cdot(3,1,1)&=0\times3+0\times1+2\times1=2. \end{aligned}

The extra claim is multiplied by zero. It adds nothing to this score. There is no hidden increase in another weighted contribution: the three contributions remain 0,0,20,0,2.

For the green reader, the claim weight is 1.51.5, so the same change adds 1.51.5:

(1.5,1,0)⋅(2,1,1)=1.5×2+1×1+0×1=4,(1.5,1,0)⋅(3,1,1)=1.5×3+1×1+0×1=5.5.\begin{aligned} (1.5,1,0)\cdot(2,1,1)&=1.5\times2+1\times1+0\times1=4,\\ (1.5,1,0)\cdot(3,1,1)&=1.5\times3+1\times1+0\times1=5.5. \end{aligned}

This explains the scores. To explain the changing purple angle, we need to look at the whole input arrow.

One angle between two complete arrows

The purple reader (0,0,2)(0,0,2) points straight along the positive meeting axis. Its two zero coordinates say it has no claim or prize component. They do not say that its angle to every input is zero.

An input (0,0,1)(0,0,1) also points straight along meeting: its angle to the purple reader is 0∘0^\circ. But (2,1,1)(2,1,1) points sideways as well as upward, so it has a different direction. Increasing claim moves its tip farther sideways while keeping its meeting height at 1.

The angle θ\theta is the single angle between the two complete arrows when their tails share the origin. We do not calculate three coordinate angles and add or average them. The reader’s length 2 also does not create a different direction from (0,0,1)(0,0,1); positive scaling preserves direction.

Two nonparallel arrows with a common tail lie in a plane, even when they live in 3D. We can look straight at that plane and measure an ordinary planar angle. When the arrows are parallel, their angle is already 0∘0^\circ or 180∘180^\circ and they do not determine a unique plane.

In the explorer, restore Article reference, then select Angle triangle. This looks straight at the plane containing the input and purple reader. Predict which side of the triangle will change when claim goes from 2 to 3, then try it. Record the sideways distance and meeting height, and explain the result before continuing.

Read the triangle inside the 3D picture

Write the input as x=(c,p,m)\mathbf x=(c,p,m). The two horizontal coordinates combine into one sideways distance from the meeting axis:

r=c2+p2.r=\sqrt{c^2+p^2}.

This is Pythagoras in the claim–prize plane. The triangle view’s horizontal axis is this combined distance, not the claim coordinate alone. Its vertical axis is meeting.

Apply Pythagoras again to include the height:

∥x∥=r2+m2=c2+p2+m2.\|\mathbf x\|=\sqrt{r^2+m^2}=\sqrt{c^2+p^2+m^2}.

The notation ∥x∥\|\mathbf x\| means the length of the whole input arrow. Our two triangles have:

InputSideways distance rrMeeting height mmInput length
(2,1,1)(2,1,1)5≈2.236\sqrt5\approx2.236116≈2.449\sqrt6\approx2.449
(3,1,1)(3,1,1)10≈3.162\sqrt{10}\approx3.1621111≈3.317\sqrt{11}\approx3.317

The hypotenuse gets longer and tilts farther from the vertical. The vertical side stays at 1.

How the 3D angle is calculated

For this right triangle, cosine is the adjacent side divided by the hypotenuse. Our angle is measured from the upward purple direction, so the adjacent side is the meeting height:

cos⁡θ=m∥x∥=mc2+p2+m2.\cos\theta=\frac{m}{\|\mathbf x\|} =\frac{m}{\sqrt{c^2+p^2+m^2}}.

To turn that ratio into an angle, use the inverse cosine, written arccos⁡\arccos:

θbefore=arccos⁡(16)≈65.9∘,θafter=arccos⁡(111)≈72.5∘.\begin{aligned} \theta_{\text{before}}&=\arccos\left(\frac1{\sqrt6}\right)\approx65.9^\circ,\\ \theta_{\text{after}}&=\arccos\left(\frac1{\sqrt{11}}\right)\approx72.5^\circ. \end{aligned}

Claim affects the angle because it affects the direction and length of the complete input. It appears in the denominator even though the purple reader’s claim weight is zero.

For any two nonzero real vectors, the general formula is

θ=arccos⁡(w⋅x∥w∥ ∥x∥).\theta=\arccos\left(\frac{\mathbf w\cdot\mathbf x}{\|\mathbf w\|\,\|\mathbf x\|}\right).

For our purple reader, w2⋅x=2m\mathbf w_2\cdot\mathbf x=2m and ∥w2∥=2\|\mathbf w_2\|=2. Substituting gives 2m/(2∥x∥)=m/∥x∥2m/(2\|\mathbf x\|)=m/\|\mathbf x\|, exactly the triangle’s ratio. The coordinate calculation and triangle geometry agree. The general angle formula follows from the law of cosines; see Interactive Linear Algebra, §7.1.

Switch back to 3D view and rotate the plot without changing the counts. The angle may look different on your screen because projecting 3D onto a screen can distort angles. The numerical measurement comes from the actual vectors, so it stays fixed. The flat triangle view preserves the angle and uses equal scales on its two axes.

What the purple reader actually measures

Imagine dropping a perpendicular from the blue arrow’s tip onto the meeting axis. Its landing point is (0,0,m)(0,0,m). The arrow from the origin to that point is the input’s projection onto the purple direction.

For the article’s inputs, that projection is always (0,0,1)(0,0,1). Increasing claim moves the tip sideways without moving its landing point up or down. We can split the input into two parts:

(c,p,m)⏟whole input=(0,0,m)⏟along the purple reader+(c,p,0)⏟perpendicular to it.\underbrace{(c,p,m)}_{\text{whole input}} =\underbrace{(0,0,m)}_{\text{along the purple reader}} +\underbrace{(c,p,0)}_{\text{perpendicular to it}}.

The purple reader measures the first part and gives zero response to the second. Its dot product equals the signed length along its direction, multiplied by its own length 2:

w2⋅x=2m.\mathbf w_2\cdot\mathbf x=2m.

In general, projection onto a nonzero reader is proj⁡wx=(w⋅x/∥w∥2)w\operatorname{proj}_{\mathbf w}\mathbf x=(\mathbf w\cdot\mathbf x/\|\mathbf w\|^2)\mathbf w. The remainder is perpendicular to the reader. This is orthogonal decomposition.

This gives a more direct explanation than saying two effects “cancel”: the amount of input along the purple direction has not changed. The growing sideways part changes the input’s direction but contributes nothing to this reader’s score.

Where the longer input enters the calculation

The familiar geometric dot-product formula is

w⋅x=∥w∥ ∥x∥cos⁡θ.\mathbf w\cdot\mathbf x=\|\mathbf w\|\,\|\mathbf x\|\cos\theta.

It includes both length and alignment. In our example:

Before: 2×6×16=2,After: 2×11×111=2.\begin{aligned} \text{Before: }&2\times\sqrt6\times\frac1{\sqrt6}=2,\\ \text{After: }&2\times\sqrt{11}\times\frac1{\sqrt{11}}=2. \end{aligned}

The angle increases, so its cosine decreases. But the input’s length increases by exactly the corresponding factor. Their product ∥x∥cos⁡θ\|\mathbf x\|\cos\theta is the unchanged meeting height, 1. This is a geometric identity, not an accidental numerical cancellation.

If we normalized each input to length 1, we would remove that length difference. The purple scores for the normalized inputs would then be 2/6≈0.8162/\sqrt6\approx0.816 and 2/11≈0.6032/\sqrt{11}\approx0.603. Those are scores for different, normalized inputs, not the original word counts.

When turning away really does reduce the score

“A larger angle gives a smaller dot product” holds when both vector lengths stay fixed, for angles from 0∘0^\circ to 180∘180^\circ.

The next explorer starts with the same (2,1,1)(2,1,1) and purple reader, but now the input length is fixed at 6\sqrt6. Its only input control turns the blue vector within its plane with the purple reader. The claim-to-prize ratio remains 2:1; the coordinates change together to keep the length fixed. These continuous values are geometric vectors, not literal word counts.

Before changing Angle to purple reader from about 65.9∘65.9^\circ to 75∘75^\circ, predict the meeting component and purple score. Record them along with the input length, then test. Explain how this differs from increasing claim alone before opening the result.

Keep length fixed; change the angle

Length stays √6 ≈ 2.449. Claim:prize stays 2:1 whenever the sideways component is nonzero.

The angle triangle, viewed straight onInput (2, 1, 1) in blue. Purple reader (0, 0, 2). Angle 65.9°.Horizontal distance combines claim and prize; vertical distance is meeting. Both axes use the same scale.meetingsideways distance √(claim² + prize²)θxw₂m = 1r = 2.236

A straight-on view of the input–reader plane. Both axes have the same scale.

Blue: input xPurple: reader w₂Teal: projection onto w₂
Input x = (2, 1, 1) · Length 2.449

Purple w₂ = (0, 0, 2)

3D angle: 65.9° · Score: 2

w₂ᵀx ≈ 0 × 2 + 0 × 1 + 2 × 1 ≈ 2

Compare with the article reference · rounded measurements
InputLengthPurple anglePurple score
Reference (2, 1, 1)2.44965.9°2
Current (2, 1, 1)2.44965.9°2
Compare turning at fixed length with adding claim

At 75∘75^\circ, the input’s meeting component is

m=6cos⁡75∘≈0.634,w2⋅x=2m≈1.268.m=\sqrt6\cos75^\circ\approx0.634, \qquad \mathbf w_2\cdot\mathbf x=2m\approx1.268.

The meeting height falls because the hypotenuse cannot grow. The purple score falls with it.

ExperimentWhat stays fixed?What happens as the angle increases?
Increase claim alonePrize, meeting, both readersInput length grows; purple projection and score stay fixed.
Turn at fixed lengthInput length, purple reader, plane of rotationMeeting projection shrinks; purple score decreases.

At 90∘90^\circ, the input lies entirely in the claim–prize plane: its meeting component and purple score are zero. This input is still nonzero, so its angle is well defined.

Design a case that tests your explanation

Return to the first explorer. Choose your own reference counts between 0 and 5, using a positive meeting count. Your task is to construct a second input with a different purple angle but the same purple score. You choose which coordinate to change and by how much.

Before making the change, record your reference, proposed change, prediction, and reason. Also state an observation that would contradict your explanation. Test it, record the measurements, and explain whether the result supports your claim. Keep both readers fixed.

Then design a separate comparison that changes the purple score while preserving the green score. Choose the two inputs yourself. Explain why the relevant weighted contributions do or do not change.

Hints and a way to check your investigation

With the purple reader fixed, its score is 2m2m. Preserve mm and change the sideways distance c2+p2\sqrt{c^2+p^2} to get a different angle when mm is positive. Merely swapping claim and prize preserves this distance and therefore preserves the angle too.

For the second comparison, look for a coordinate whose weight is zero in the green reader and nonzero in the purple reader. The identities s1=1.5c+ps_1=1.5c+p and s2=2ms_2=2m let you check your chosen values.

Your observations test particular cases. These formulas explain why the behavior holds for every case satisfying the same constraints.

Does this require a zero weight?

A zero weight makes changing that one coordinate invisible to the score. But a more general rule is that any change perpendicular to the reader leaves its dot product unchanged:

w⋅(x+Δx)=w⋅x+w⋅Δx.\mathbf w\cdot(\mathbf x+\Delta\mathbf x) =\mathbf w\cdot\mathbf x+\mathbf w\cdot\Delta\mathbf x.

If w⋅Δx=0\mathbf w\cdot\Delta\mathbf x=0, the last term vanishes. For the purple reader, Δx=(1,0,0)\Delta\mathbf x=(1,0,0) is such a change.

Try a new case on paper: w=(1,1,1)\mathbf w=(1,1,1), a=(1,1,1)\mathbf a=(1,1,1), and b=(2,0,1)\mathbf b=(2,0,1). None of the weights is zero. Predict whether the scores and angles match, then calculate.

Check the case with no zero weights

Both dot products are 3. The change is (1,−1,0)(1,-1,0), whose weighted contributions cancel: 1−1+0=01-1+0=0.

The first input points in exactly the reader’s direction, so its angle is 0∘0^\circ. The second has angle arccos⁡(3/15)≈39.2∘\arccos(3/\sqrt{15})\approx39.2^\circ. The angle changes while the score stays fixed, even though every weight is nonzero. Here two input coordinates changed together; it is not an example of one ignored coordinate.

Boundary cases worth distinguishing

An input (0,0,1)(0,0,1) has angle 0∘0^\circ and purple score 2. An input (1,0,0)(1,0,0) has angle 90∘90^\circ and score 0. The zero input (0,0,0)(0,0,0) also has score 0, but its angle is undefined because it has no direction. You can test all three in the first explorer.

Outside the word-count setting, negative coordinates are possible. For example, (0,0,−1)(0,0,-1) points opposite the purple reader: angle 180∘180^\circ, score −2-2. A score’s sign distinguishes projection along the reader from projection against it.

The article’s puzzle is now one specific instance of a general distinction: the angle describes the direction of the whole input; the dot product measures its signed projection along a reader, scaled by the reader’s length. A sideways change can alter the first without altering the second.

Return to the feature-detectors article, or continue with dot product and cosine similarity to explore what changes when we remove vector length from the comparison.