What is the difference between dot product and cosine similarity?
Two vectors can point in exactly the same direction and still produce very different dot products. How could we measure their directional alignment without letting their lengths affect the result?
You will use vector lengths and the coordinate dot product from the previous lesson. For two-dimensional vectors,
Stretch without turning
Start with and . The explorer reports two scores: dot product and cosine similarity. Before using the controls, write a prediction: if we double only the length of b, which score will change? Why?
Keep a record in a notebook or text file: both vectors, the angle between them, the dot product, and cosine similarity. Record the reference values first. Then change from 1 to 2, keeping a and both directions fixed. Record the results. Predict and test length 3 as a second trial against the same reference.
Directions are measured from the positive x-axis. Change a length to stretch without turning. Values are rounded to three decimals.
Dot product: 3
Cosine similarity: 0.6
Angle between vectors: 53.13°
Compare your three records. Did the direction of either arrow change? Write an explanation of the two scores before opening ours.
Compare the stretching results
| Vector b | Dot product | Cosine similarity | Angle |
|---|---|---|---|
| about | |||
| about | |||
| about |
The dot product scales with the length of b. Cosine similarity stays fixed because the directions stay fixed. Both scores describe alignment, but only the dot product also includes the vectors’ magnitudes.
On paper, the middle trial gives . Dividing by the two lengths gives .
Turn without stretching
Select . Keep a fixed and keep the length of b at 1. Predict the sign of each score if b points straight up, then straight left. Change only to 90 and then 180, recording each result against the reset reference.
Does ignoring length mean ignoring direction too? Explain your evidence before revealing the calculation.
Compare the rotation results
At direction 90°, , so the dot product is and cosine similarity is . At direction 180°, , giving and .
The control gives b’s direction from the x-axis, not the angle between the vectors. That angle is about 36.87° in the first trial and 126.87° in the second. Both scores respond to direction even though both lengths stay fixed.
What each score measures
For nonzero real vectors, the coordinate sum has this geometric interpretation:
The length product sets the scale; supplies the signed alignment factor. Dividing by both lengths removes that scale:
The dot product measures directional alignment weighted by both magnitudes. Cosine similarity measures directional alignment alone. For fixed nonzero lengths, a smaller angle between 0° and 180° gives a larger value of either score.
| Angle between vectors | Cosine similarity | Dot product |
|---|---|---|
| : same direction | ||
| : perpendicular | ||
| : opposite directions |
Cosine similarity is always between −1 and 1. The dot product has no universal bound as the lengths vary. Increasing length makes a negative dot product more negative; at a right angle, stretching preserves a dot product of zero.
For your example and ,
The angle is about 94.4°: slightly beyond perpendicular. The dot product does not mean opposite directions; cosine similarity would.
Same direction, different scores
Consider , , and . Before calculating, predict which comparison will have the largest dot product and whether cosine similarity will distinguish them.
Check the three comparisons
| Comparison | Dot product | Cosine similarity |
|---|---|---|
| a and b | ||
| a and c | ||
| b and c |
All three vectors point in the same direction. The differing length products explain the differing dot products. A cosine similarity of 1 means identical directions, not necessarily identical vectors.
Is normalization the same as making a unit vector?
Yes. Normalizing a nonzero vector to unit length means dividing every component by its Euclidean norm. It preserves direction and makes the length 1:
For , the norm is . Therefore,
Reset the explorer. Predict where the unit arrows will end and what their dot product will be, then select . The original and unit arrows use the same coordinate scale. Explain what happened to their lengths and directions.
Connect the unit arrows to cosine similarity
The unit vectors are and , with dot product . That equals the original pair’s cosine similarity. In general, dividing each vector by its own norm gives
Cosine similarity is the dot product of the two unit vectors. Normalizing only one vector generally leaves the other vector’s magnitude in the score.
Unit normalization changes length, not the number of dimensions. A vector in still has 768 components afterward. It can also lengthen a vector whose norm was less than 1.
The zero vector cannot be normalized this way: its norm is zero, so the division is undefined. Likewise, cosine similarity is undefined if either vector is zero, although their dot product is still zero. Predict the explorer’s readouts before setting either length to zero, then check.
Why this matters for semantic search
An embedding represents a query or document as a vector. Consider a toy query and documents and :
The second document receives a score ten times as large solely because of its magnitude. That does not establish ten times the relevance. Their cosine similarities are both 1, so cosine ranking ties them on directional alignment.
Cosine similarity is useful when we want magnitude differences to have no effect on ranking. Magnitude can also contain meaningful learned information, so removing it is a modeling choice, not an automatic improvement. If both queries and documents already have unit length, dot product and cosine similarity give exactly the same scores.
Connection to attention
In transformer attention, queries and keys are learned vectors whose scores determine how information is mixed. Standard scaled dot-product attention computes
Here contains queries, keys, values, and is the number of components in each query or key. Softmax turns each query’s scores into weights used to combine values. This is the formulation in Attention Is All You Need, §3.2.1.
Dividing by applies a shared scale based on dimension. It does not divide each query–key score by , so it does not turn those vectors into unit vectors. The scores still depend on both alignment and magnitude. Scaling both query and key by 2 multiplies their score by 4 while their cosine similarity stays fixed.
For the statistical reason behind that divisor, continue with variance normalization in attention, after mean and variance.
Design your own comparison
Choose two nonzero vectors in the explorer, or use coordinates on paper. Choose a positive length change for just one vector while keeping both directions and the other vector fixed. Record a reference, predict both scores after the change, test, and explain the result. What observation would contradict your claim?
Next, design a separate comparison that challenges the statement “stretching always makes the dot product larger.” You choose the starting vectors and stretch factor. Can you construct a case where the score gets more negative, or stays zero? Explain why cosine similarity does or does not change.
Check the reasoning in your investigation
For nonzero vectors and a positive scale ,
The same positive factor appears in the numerator and denominator of cosine similarity and cancels. These identities explain the general rule; your recorded examples test it in particular cases.
For a stretch factor greater than 1, an obtuse angle makes the dot product more negative; perpendicular vectors keep it at zero. A negative scale reverses direction and flips the sign of cosine similarity. A scale of zero makes cosine similarity undefined. Neither is a direction-preserving stretch.
Check on a new pair
On paper, let and . Find their dot product, cosine similarity, and unit vectors. If you normalize only v, does the new dot product equal the original cosine similarity?
Check your calculation
The lengths are 13 and 2. The dot product is , and cosine similarity is .
The unit vectors are and . Their dot product is . Normalizing only v gives , which still includes the length of u.
Continue with linear combinations and span to investigate which vectors you can build from a chosen set of directions.