Why does the angle change while the dot product stays the same?
In Feature detectors: how neurons read and write, a message becomes three word counts, ordered claim, prize, meeting. The input is . Two readers look at that same input:
The puzzle appears when we increase claim from 2 to 3. The blue input arrow turns farther away from the purple reader, yet the purple dot-product score stays at 2. If the dot product measures alignment, shouldn’t it decrease?
This note investigates that specific case. You only need to know how coordinates locate an arrow and how to multiply and add its weighted inputs. We will build the geometry from there. Here, “score” means the reader’s dot product before any bias or activation, not the classifier’s final output.
First, reproduce the puzzle
Before moving a control, write a prediction and a reason: with both readers fixed, what will increasing claim do to the purple angle, purple score, and input length? Predict the green score too.
The explorer starts at the article’s reference, . Change only from 2 to 3; leave prize and meeting at 1. Compare the current row with the reference row. Record the input, its length, both reader angles, and both scores in a notebook or text file.
The plot shows the blue input and purple reader. The green reader’s measurements appear below it for comparison. Rotating the camera changes your view, not the vectors or their measurements.
Drag to rotate · Shift-drag to pan · Scroll to zoom
Purple w₂ = (0, 0, 2)
3D angle: 65.9° · Score: 2
w₂ᵀx = 0 × 2 + 0 × 1 + 2 × 1 = 2
Green w₁ = (1.5, 1, 0)
3D angle: 25.1° · Score: 4
w₁ᵀx = 1.5 × 2 + 1 × 1 + 0 × 1 = 4
| Input | Length | Purple angle | Purple score | Green angle | Green score |
|---|---|---|---|---|---|
| Reference (2, 1, 1) | 2.449 | 65.9° | 2 | 25.1° | 4 |
| Current (2, 1, 1) | 2.449 | 65.9° | 2 | 25.1° | 4 |
What changed? What stayed fixed? Try to explain the result in one sentence before opening the calculation.
Compare the two weighted sums
For the purple reader, only the meeting coordinate contributes:
The extra claim is multiplied by zero. It adds nothing to this score. There is no hidden increase in another weighted contribution: the three contributions remain .
For the green reader, the claim weight is , so the same change adds :
This explains the scores. To explain the changing purple angle, we need to look at the whole input arrow.
One angle between two complete arrows
The purple reader points straight along the positive meeting axis. Its two zero coordinates say it has no claim or prize component. They do not say that its angle to every input is zero.
An input also points straight along meeting: its angle to the purple reader is . But points sideways as well as upward, so it has a different direction. Increasing claim moves its tip farther sideways while keeping its meeting height at 1.
The angle is the single angle between the two complete arrows when their tails share the origin. We do not calculate three coordinate angles and add or average them. The reader’s length 2 also does not create a different direction from ; positive scaling preserves direction.
Two nonparallel arrows with a common tail lie in a plane, even when they live in 3D. We can look straight at that plane and measure an ordinary planar angle. When the arrows are parallel, their angle is already or and they do not determine a unique plane.
In the explorer, restore , then select . This looks straight at the plane containing the input and purple reader. Predict which side of the triangle will change when claim goes from 2 to 3, then try it. Record the sideways distance and meeting height, and explain the result before continuing.
Read the triangle inside the 3D picture
Write the input as . The two horizontal coordinates combine into one sideways distance from the meeting axis:
This is Pythagoras in the claim–prize plane. The triangle view’s horizontal axis is this combined distance, not the claim coordinate alone. Its vertical axis is meeting.
Apply Pythagoras again to include the height:
The notation means the length of the whole input arrow. Our two triangles have:
| Input | Sideways distance | Meeting height | Input length |
|---|---|---|---|
The hypotenuse gets longer and tilts farther from the vertical. The vertical side stays at 1.
How the 3D angle is calculated
For this right triangle, cosine is the adjacent side divided by the hypotenuse. Our angle is measured from the upward purple direction, so the adjacent side is the meeting height:
To turn that ratio into an angle, use the inverse cosine, written :
Claim affects the angle because it affects the direction and length of the complete input. It appears in the denominator even though the purple reader’s claim weight is zero.
For any two nonzero real vectors, the general formula is
For our purple reader, and . Substituting gives , exactly the triangle’s ratio. The coordinate calculation and triangle geometry agree. The general angle formula follows from the law of cosines; see Interactive Linear Algebra, §7.1.
Switch back to and rotate the plot without changing the counts. The angle may look different on your screen because projecting 3D onto a screen can distort angles. The numerical measurement comes from the actual vectors, so it stays fixed. The flat triangle view preserves the angle and uses equal scales on its two axes.
What the purple reader actually measures
Imagine dropping a perpendicular from the blue arrow’s tip onto the meeting axis. Its landing point is . The arrow from the origin to that point is the input’s projection onto the purple direction.
For the article’s inputs, that projection is always . Increasing claim moves the tip sideways without moving its landing point up or down. We can split the input into two parts:
The purple reader measures the first part and gives zero response to the second. Its dot product equals the signed length along its direction, multiplied by its own length 2:
In general, projection onto a nonzero reader is . The remainder is perpendicular to the reader. This is orthogonal decomposition.
This gives a more direct explanation than saying two effects “cancel”: the amount of input along the purple direction has not changed. The growing sideways part changes the input’s direction but contributes nothing to this reader’s score.
Where the longer input enters the calculation
The familiar geometric dot-product formula is
It includes both length and alignment. In our example:
The angle increases, so its cosine decreases. But the input’s length increases by exactly the corresponding factor. Their product is the unchanged meeting height, 1. This is a geometric identity, not an accidental numerical cancellation.
If we normalized each input to length 1, we would remove that length difference. The purple scores for the normalized inputs would then be and . Those are scores for different, normalized inputs, not the original word counts.
When turning away really does reduce the score
“A larger angle gives a smaller dot product” holds when both vector lengths stay fixed, for angles from to .
The next explorer starts with the same and purple reader, but now the input length is fixed at . Its only input control turns the blue vector within its plane with the purple reader. The claim-to-prize ratio remains 2:1; the coordinates change together to keep the length fixed. These continuous values are geometric vectors, not literal word counts.
Before changing from about to , predict the meeting component and purple score. Record them along with the input length, then test. Explain how this differs from increasing claim alone before opening the result.
Length stays √6 ≈ 2.449. Claim:prize stays 2:1 whenever the sideways component is nonzero.
A straight-on view of the input–reader plane. Both axes have the same scale.
Purple w₂ = (0, 0, 2)
3D angle: 65.9° · Score: 2
w₂ᵀx ≈ 0 × 2 + 0 × 1 + 2 × 1 ≈ 2
| Input | Length | Purple angle | Purple score |
|---|---|---|---|
| Reference (2, 1, 1) | 2.449 | 65.9° | 2 |
| Current (2, 1, 1) | 2.449 | 65.9° | 2 |
Compare turning at fixed length with adding claim
At , the input’s meeting component is
The meeting height falls because the hypotenuse cannot grow. The purple score falls with it.
| Experiment | What stays fixed? | What happens as the angle increases? |
|---|---|---|
| Increase claim alone | Prize, meeting, both readers | Input length grows; purple projection and score stay fixed. |
| Turn at fixed length | Input length, purple reader, plane of rotation | Meeting projection shrinks; purple score decreases. |
At , the input lies entirely in the claim–prize plane: its meeting component and purple score are zero. This input is still nonzero, so its angle is well defined.
Design a case that tests your explanation
Return to the first explorer. Choose your own reference counts between 0 and 5, using a positive meeting count. Your task is to construct a second input with a different purple angle but the same purple score. You choose which coordinate to change and by how much.
Before making the change, record your reference, proposed change, prediction, and reason. Also state an observation that would contradict your explanation. Test it, record the measurements, and explain whether the result supports your claim. Keep both readers fixed.
Then design a separate comparison that changes the purple score while preserving the green score. Choose the two inputs yourself. Explain why the relevant weighted contributions do or do not change.
Hints and a way to check your investigation
With the purple reader fixed, its score is . Preserve and change the sideways distance to get a different angle when is positive. Merely swapping claim and prize preserves this distance and therefore preserves the angle too.
For the second comparison, look for a coordinate whose weight is zero in the green reader and nonzero in the purple reader. The identities and let you check your chosen values.
Your observations test particular cases. These formulas explain why the behavior holds for every case satisfying the same constraints.
Does this require a zero weight?
A zero weight makes changing that one coordinate invisible to the score. But a more general rule is that any change perpendicular to the reader leaves its dot product unchanged:
If , the last term vanishes. For the purple reader, is such a change.
Try a new case on paper: , , and . None of the weights is zero. Predict whether the scores and angles match, then calculate.
Check the case with no zero weights
Both dot products are 3. The change is , whose weighted contributions cancel: .
The first input points in exactly the reader’s direction, so its angle is . The second has angle . The angle changes while the score stays fixed, even though every weight is nonzero. Here two input coordinates changed together; it is not an example of one ignored coordinate.
Boundary cases worth distinguishing
An input has angle and purple score 2. An input has angle and score 0. The zero input also has score 0, but its angle is undefined because it has no direction. You can test all three in the first explorer.
Outside the word-count setting, negative coordinates are possible. For example, points opposite the purple reader: angle , score . A score’s sign distinguishes projection along the reader from projection against it.
The article’s puzzle is now one specific instance of a general distinction: the angle describes the direction of the whole input; the dot product measures its signed projection along a reader, scaled by the reader’s length. A sideways change can alter the first without altering the second.
Return to the feature-detectors article, or continue with dot product and cosine similarity to explore what changes when we remove vector length from the comparison.