Why does attention divide dot products by √d?

A dot product adds one contribution per dimension. If we add more contributions, what happens to its spread—and what should we divide by to keep that spread steady?

This is a statistics question behind attention. You will compare three score rules, then explain why dividing by dk\sqrt{d_k} is variance normalization under stated assumptions. Here dkd_k is the number of coordinates in each query and key for one attention head, not the number of tokens.

Start with mean and variance and dot product versus cosine similarity if those ideas are unfamiliar.

Predict the spread of a sum

Write a dot product as a sum of coordinate products:

S=q⋅k=∑i=1dkqiki.S=q\cdot k=\sum_{i=1}^{d_k}q_i k_i.

For our experiment, each original query and key coordinate is independently either −1-1 or +1+1, with equal probability. Each product zi=qikiz_i=q_i k_i is therefore also an independent fair sign. The explorer lists every possible product pattern, rather than drawing a random sample.

At dimension 2 the patterns are (−1,−1),(−1,+1),(+1,−1),(+1,+1)(-1,-1),(-1,+1),(+1,-1),(+1,+1), giving scores −2,0,0,2-2,0,0,2. Each pattern has probability 1/41/4; repeated scores are intentional. Their mean is zero, but their variance is not:

variance=(−2−0)2+(0−0)2+(0−0)2+(2−0)24=2.\text{variance}=\frac{(-2-0)^2+(0-0)^2+(0-0)^2+(2-0)^2}{4}=2.

We divide by the number of equally likely patterns because we have the complete distribution. This is not the n−1n-1 estimate of population variance from a sample.

Set Dimension d to 4, Score rule to Dot product, and Query length multiplier to 1. Write a prediction about the mean and spread, then record the reference.

Before changing dimension to 8, predict which of the mean, variance, and standard deviation will double. Keep the sign law, score rule, and query multiplier fixed. Record this second run and compare it with the first.

Exact sign-product model: each coordinate product is independently −1 or +1 with equal probability before scaling the query. Enumerate all 2^d equally likely product patterns, without sampling. Keep the sign law fixed when changing dimension. Query and key coordinates originally have magnitude 1; multiplying only the query by c gives lengths c√d and √d. Variance uses the full distribution, with divisor 2^d.

0/16 runs. Each run keeps its own inputs and results. Work stays in this tab across reloads; download a report before closing it.

Write a prediction to record your reference.

Record the mean, variance, standard deviation, and largest absolute score. Inspect some individual patterns too. The plot groups identical scores and shows their exact probabilities; it is not an attention distribution across tokens. Explore dataset opens the reference product patterns; each run report contains its own inputs.

Explain how a mean of zero can coexist with increasing spread before opening the results.

Compare dimensions 4 and 8
DimensionMeanVarianceStandard deviationLargest absolute score
40424
8088≈2.828\sqrt8\approx2.8288

Positive and negative sums balance at zero. Squaring their deviations prevents that cancellation when measuring spread. Doubling dimension doubles the variance; it multiplies standard deviation by 2\sqrt2, not 2. The maximum grows differently from the typical spread.

Find a divisor that holds the spread steady

Return to Reference settings and select your original dimension-4 run as the Comparison baseline. Before changing the score rule to Divide by √d, predict the new variance. Keep dimension and query multiplier fixed, then record the run.

Now use that scaled run as the baseline. Predict and test dimension 8 with the same scaling rule and query multiplier. Record the same four statistics. Explain what stayed constant and what still grew.

Check the scaling results

At dimension 4, division by 2 changes variance from 4 to 1 and standard deviation from 2 to 1. At dimension 8, division by 8\sqrt8 also gives variance 1 and standard deviation 1.

The largest absolute scaled score is still d\sqrt d: 2 and about 2.828 respectively. Unit variance is a statement about spread across possible scores, not a bound of 1 on every score. The distributions also need not have identical shapes.

Why the variance rule works

Let E[X]E[X] denote the probability-weighted average of a random quantity XX, and let μ=E[X]\mu=E[X]. Variance is Var⁡(X)=E[(X−μ)2]\operatorname{Var}(X)=E[(X-\mu)^2].

Multiplying every value by a constant aa also multiplies its mean by aa. Each deviation is multiplied by aa, so each squared deviation is multiplied by a2a^2:

Var⁡(aX)=E[(aX−aμ)2]=a2Var⁡(X).\operatorname{Var}(aX) =E[(aX-a\mu)^2] =a^2\operatorname{Var}(X).

If XX has standard deviation σ>0\sigma>0, dividing by σ\sigma gives variance 1. Subtracting the mean is a separate operation: (X−μ)/σ(X-\mu)/\sigma also centers the result at zero. Attention’s dimension scaling does not estimate or subtract a score mean.

To derive the dot-product variance, assume all query and key coordinates are mutually independent, have mean zero, and have variance one. For each product:

E[qiki]=E[qi]E[ki]=0,E[(qiki)2]=E[qi2]E[ki2]=1.E[q_i k_i]=E[q_i]E[k_i]=0, \qquad E[(q_i k_i)^2]=E[q_i^2]E[k_i^2]=1.

Thus Var⁡(qiki)=1\operatorname{Var}(q_i k_i)=1. Products from different coordinates are independent, so their variances add:

Var⁡(S)=∑i=1dkVar⁡(qiki)=dk.\operatorname{Var}(S)=\sum_{i=1}^{d_k}\operatorname{Var}(q_i k_i)=d_k.

Using a=1/dka=1/\sqrt{d_k} now gives

Var⁡ ⁣(Sdk)=1dkVar⁡(S)=1.\operatorname{Var}\!\left(\frac{S}{\sqrt{d_k}}\right) =\frac{1}{d_k}\operatorname{Var}(S)=1.

For dk=64d_k=64, the raw variance is 64 and standard deviation is 8. Divide the scores by 8, not 64: dividing by 64 would leave variance 64/642=1/6464/64^2=1/64.

These equalities are exact under the assumptions. In a trained network, coordinates can be correlated and have different means and variances. The dimension factor is a useful statistical scale, not a guarantee of unit variance for every head or input. It also does not force the finite set of scores for one query to have variance 1.

Three scores, three treatments of magnitude

For nonzero vectors, q⋅k=∥q∥∥k∥cos⁡θq\cdot k=\|q\|\|k\|\cos\theta.

OperationFormulaEffect on magnitude information
Dot productq⋅kq\cdot kIncludes both lengths and alignment
Cosine similarityq⋅k∥q∥∥k∥\frac{q\cdot k}{\|q\|\|k\|}Removes both lengths; measures angle
Scaled dot productq⋅kdk\frac{q\cdot k}{\sqrt{d_k}}Preserves length dependence at fixed dimension

For q=k=(1,0)q=k=(1,0), these scores are 11, 11, and 1/21/\sqrt2. For q=k=(10,0)q=k=(10,0), they are 100100, 11, and 100/2100/\sqrt2. Both pairs are perfectly aligned. Only cosine ignores the change in lengths; it is undefined if either vector is zero.

In our sign model, each original vector has length d\sqrt d, so cosine divides by dd. With query multiplier 1, predict cosine’s variance at dimension 4. Compare a cosine run with a dimension-4 raw reference before opening the answer.

Check cosine’s variance in this model

Cosine’s variance is d/d2=1/dd/d^2=1/d, giving 1/41/4 here. Dividing by actual vector lengths is different from dividing by the expected standard deviation of the dot product. Outside this sign model, the length product need not equal dd.

Why does attention care about this spread?

For one query, softmax turns the scores for its keys into weights:

pj=esj∑ℓesℓ.p_j=\frac{e^{s_j}}{\sum_\ell e^{s_\ell}}.

On paper or with a calculator, use raw scores [4,2,0][4,2,0] and dimension dk=4d_k=4. Predict whether dividing all three scores by 2 changes the highest-scoring key or how strongly it is favored. Hold the raw scores and keys fixed; change only the divisor. Calculate and record both probability vectors, then explain the difference.

Compare the attention weights
Scores entering softmaxKey 1Key 2Key 3
[4,2,0][4,2,0]86.7%11.7%1.6%
[2,1,0][2,1,0]66.5%24.5%9.0%

The ranking stays fixed, but the distribution becomes less concentrated. Softmax responds to score differences: adding 100 to every score changes none of the probabilities. Multiplying scores changes their gaps.

For example, ∂pj/∂sj=pj(1−pj)\partial p_j/\partial s_j=p_j(1-p_j) becomes small near 0 or 1. This concerns the softmax derivative; the gradient of a complete training loss also depends on the rest of the computation.

Standard scaled dot-product attention uses softmax⁡(QK⊤/dk)\operatorname{softmax}(QK^\top/\sqrt{d_k}), applied across keys for each query. The original Transformer paper motivates the divisor using the variance argument and softmax gradients. Learned vector directions and magnitudes still influence compatibility. See Vaswani et al., §3.2.1 and footnote 1.

So there are three different meanings of normalization here:

NormalizationWhat it normalizes
Divide each nonzero vector by its lengthEach vector has unit length; their dot product becomes cosine similarity
Divide a dot product by dk\sqrt{d_k}Score variance becomes 1 under the independent, zero-mean, unit-variance assumptions
Softmax across keysExponentiated scores become nonnegative weights summing to 1

Cosine-based attention is also possible, often with a temperature multiplier. The variance argument explains the standard scaling choice; it does not prove that cosine-based designs cannot work.

Design a test that separates the normalizations

Choose a dimension and a query length multiplier greater than 1. Keep the key vectors and all signs fixed. Predict how stretching only the query will affect raw scores, scaled scores, and cosine scores—and their variances.

For each score rule, record a multiplier-1 reference and your chosen multiplier as a matched pair. Select the matching baseline so each comparison changes only the query multiplier. Cite individual scores as well as distribution statistics. What observation would contradict your explanation?

Check your proposed rule

For a positive query multiplier cc, raw and scaled dot products multiply by cc and their variances multiply by c2c^2. Cosine stays fixed because both its numerator and its length denominator multiply by cc.

In this model, the three variances are c2dc^2d, c2c^2, and 1/d1/d, respectively. A scaled-score variance different from 1 after stretching does not contradict our derivation: the query coordinates no longer have unit variance.

Transfer: challenge an assumption

On paper, replace independent products with perfectly tied ones: z1=⋯=zd=Zz_1=\cdots=z_d=Z, where ZZ is one fair sign. Predict the variance of the sum and of the sum divided by d\sqrt d. Can the same divisor still give unit variance?

Check the dependent case

Now S=dZS=dZ, so Var⁡(S)=d2\operatorname{Var}(S)=d^2 and Var⁡(S/d)=d\operatorname{Var}(S/\sqrt d)=d. Variances do not simply add when the products are correlated. Generally,

Var⁡ ⁣(∑izi)=∑iVar⁡(zi)+2∑i<jCov⁡(zi,zj).\operatorname{Var}\!\left(\sum_i z_i\right) =\sum_i\operatorname{Var}(z_i)+2\sum_{i<j}\operatorname{Cov}(z_i,z_j).

For a different transfer, keep all coordinates independent and zero-mean, but give query coordinates variance 4 and key coordinates variance 9. Each product then has variance 36. Dividing the dot product by d\sqrt d leaves variance 36; dividing by 6d6\sqrt d gives variance 1.

Download your report. Include what was normalized, which assumptions your explanation used, and a case where the unit-variance conclusion fails.

Continue

Previous: Accuracy, precision, and recall · Revisit dot product and cosine similarity · All statistics notes