Why does attention divide dot products by √d?
A dot product adds one contribution per dimension. If we add more contributions, what happens to its spread—and what should we divide by to keep that spread steady?
This is a statistics question behind attention. You will compare three score rules, then explain why dividing by is variance normalization under stated assumptions. Here is the number of coordinates in each query and key for one attention head, not the number of tokens.
Start with mean and variance and dot product versus cosine similarity if those ideas are unfamiliar.
Predict the spread of a sum
Write a dot product as a sum of coordinate products:
For our experiment, each original query and key coordinate is independently either or , with equal probability. Each product is therefore also an independent fair sign. The explorer lists every possible product pattern, rather than drawing a random sample.
At dimension 2 the patterns are , giving scores . Each pattern has probability ; repeated scores are intentional. Their mean is zero, but their variance is not:
We divide by the number of equally likely patterns because we have the complete distribution. This is not the estimate of population variance from a sample.
Set to 4, to , and to 1. Write a prediction about the mean and spread, then record the reference.
Before changing dimension to 8, predict which of the mean, variance, and standard deviation will double. Keep the sign law, score rule, and query multiplier fixed. Record this second run and compare it with the first.
Exact sign-product model: each coordinate product is independently −1 or +1 with equal probability before scaling the query. Enumerate all 2^d equally likely product patterns, without sampling. Keep the sign law fixed when changing dimension. Query and key coordinates originally have magnitude 1; multiplying only the query by c gives lengths c√d and √d. Variance uses the full distribution, with divisor 2^d.
0/16 runs. Each run keeps its own inputs and results. Work stays in this tab across reloads; download a report before closing it.
Write a prediction to record your reference.
Record the mean, variance, standard deviation, and largest absolute score. Inspect some individual patterns too. The plot groups identical scores and shows their exact probabilities; it is not an attention distribution across tokens. Explore dataset opens the reference product patterns; each run report contains its own inputs.
Explain how a mean of zero can coexist with increasing spread before opening the results.
Compare dimensions 4 and 8
| Dimension | Mean | Variance | Standard deviation | Largest absolute score |
|---|---|---|---|---|
| 4 | 0 | 4 | 2 | 4 |
| 8 | 0 | 8 | 8 |
Positive and negative sums balance at zero. Squaring their deviations prevents that cancellation when measuring spread. Doubling dimension doubles the variance; it multiplies standard deviation by , not 2. The maximum grows differently from the typical spread.
Find a divisor that holds the spread steady
Return to and select your original dimension-4 run as the . Before changing the score rule to , predict the new variance. Keep dimension and query multiplier fixed, then record the run.
Now use that scaled run as the baseline. Predict and test dimension 8 with the same scaling rule and query multiplier. Record the same four statistics. Explain what stayed constant and what still grew.
Check the scaling results
At dimension 4, division by 2 changes variance from 4 to 1 and standard deviation from 2 to 1. At dimension 8, division by also gives variance 1 and standard deviation 1.
The largest absolute scaled score is still : 2 and about 2.828 respectively. Unit variance is a statement about spread across possible scores, not a bound of 1 on every score. The distributions also need not have identical shapes.
Why the variance rule works
Let denote the probability-weighted average of a random quantity , and let . Variance is .
Multiplying every value by a constant also multiplies its mean by . Each deviation is multiplied by , so each squared deviation is multiplied by :
If has standard deviation , dividing by gives variance 1. Subtracting the mean is a separate operation: also centers the result at zero. Attention’s dimension scaling does not estimate or subtract a score mean.
To derive the dot-product variance, assume all query and key coordinates are mutually independent, have mean zero, and have variance one. For each product:
Thus . Products from different coordinates are independent, so their variances add:
Using now gives
For , the raw variance is 64 and standard deviation is 8. Divide the scores by 8, not 64: dividing by 64 would leave variance .
These equalities are exact under the assumptions. In a trained network, coordinates can be correlated and have different means and variances. The dimension factor is a useful statistical scale, not a guarantee of unit variance for every head or input. It also does not force the finite set of scores for one query to have variance 1.
Three scores, three treatments of magnitude
For nonzero vectors, .
| Operation | Formula | Effect on magnitude information |
|---|---|---|
| Dot product | Includes both lengths and alignment | |
| Cosine similarity | Removes both lengths; measures angle | |
| Scaled dot product | Preserves length dependence at fixed dimension |
For , these scores are , , and . For , they are , , and . Both pairs are perfectly aligned. Only cosine ignores the change in lengths; it is undefined if either vector is zero.
In our sign model, each original vector has length , so cosine divides by . With query multiplier 1, predict cosine’s variance at dimension 4. Compare a cosine run with a dimension-4 raw reference before opening the answer.
Check cosine’s variance in this model
Cosine’s variance is , giving here. Dividing by actual vector lengths is different from dividing by the expected standard deviation of the dot product. Outside this sign model, the length product need not equal .
Why does attention care about this spread?
For one query, softmax turns the scores for its keys into weights:
On paper or with a calculator, use raw scores and dimension . Predict whether dividing all three scores by 2 changes the highest-scoring key or how strongly it is favored. Hold the raw scores and keys fixed; change only the divisor. Calculate and record both probability vectors, then explain the difference.
Compare the attention weights
| Scores entering softmax | Key 1 | Key 2 | Key 3 |
|---|---|---|---|
| 86.7% | 11.7% | 1.6% | |
| 66.5% | 24.5% | 9.0% |
The ranking stays fixed, but the distribution becomes less concentrated. Softmax responds to score differences: adding 100 to every score changes none of the probabilities. Multiplying scores changes their gaps.
For example, becomes small near 0 or 1. This concerns the softmax derivative; the gradient of a complete training loss also depends on the rest of the computation.
Standard scaled dot-product attention uses , applied across keys for each query. The original Transformer paper motivates the divisor using the variance argument and softmax gradients. Learned vector directions and magnitudes still influence compatibility. See Vaswani et al., §3.2.1 and footnote 1.
So there are three different meanings of normalization here:
| Normalization | What it normalizes |
|---|---|
| Divide each nonzero vector by its length | Each vector has unit length; their dot product becomes cosine similarity |
| Divide a dot product by | Score variance becomes 1 under the independent, zero-mean, unit-variance assumptions |
| Softmax across keys | Exponentiated scores become nonnegative weights summing to 1 |
Cosine-based attention is also possible, often with a temperature multiplier. The variance argument explains the standard scaling choice; it does not prove that cosine-based designs cannot work.
Design a test that separates the normalizations
Choose a dimension and a query length multiplier greater than 1. Keep the key vectors and all signs fixed. Predict how stretching only the query will affect raw scores, scaled scores, and cosine scores—and their variances.
For each score rule, record a multiplier-1 reference and your chosen multiplier as a matched pair. Select the matching baseline so each comparison changes only the query multiplier. Cite individual scores as well as distribution statistics. What observation would contradict your explanation?
Check your proposed rule
For a positive query multiplier , raw and scaled dot products multiply by and their variances multiply by . Cosine stays fixed because both its numerator and its length denominator multiply by .
In this model, the three variances are , , and , respectively. A scaled-score variance different from 1 after stretching does not contradict our derivation: the query coordinates no longer have unit variance.
Transfer: challenge an assumption
On paper, replace independent products with perfectly tied ones: , where is one fair sign. Predict the variance of the sum and of the sum divided by . Can the same divisor still give unit variance?
Check the dependent case
Now , so and . Variances do not simply add when the products are correlated. Generally,
For a different transfer, keep all coordinates independent and zero-mean, but give query coordinates variance 4 and key coordinates variance 9. Each product then has variance 36. Dividing the dot product by leaves variance 36; dividing by gives variance 1.
Download your report. Include what was normalized, which assumptions your explanation used, and a case where the unit-variance conclusion fails.
Continue
Previous: Accuracy, precision, and recall · Revisit dot product and cosine similarity · All statistics notes