In what a weight matrix does, we followed inputs through a small neural network. Each row of a matrix read a weighted combination of the inputs, and the resulting numbers became the coordinates of a new vector. Bias and ReLU changed what the next layer received.

We can use those same operations to investigate another question: how can a layer represent more features than it has neurons? Our learned bottleneck experiment investigated two features sharing one signed number. Here we’ll move into a plane, where the angles between several feature directions reveal new kinds of interference.

Imagine five feature strengths, labeled cat, dog, car, tree, and boat. We want to squeeze them into two hidden numbers and then reconstruct the five strengths. Two numbers cannot preserve every possible combination of five independently varying inputs. But suppose most examples contain just one or two active features. Could sharing the two numbers still be useful?

We’ll choose a small matrix by hand and follow its calculations. The geometry will show what causes interference, how a negative bias can remove some unwanted responses, and why larger spaces offer more room for features to coexist. The model’s architecture follows the tied-weight model studied in Toy Models of Superposition. We choose the five feature arrows and input strengths ourselves to make the arithmetic visible; these are not results from a training run.

Five inputs become two hidden numbers

Our inputs are nonnegative strengths, with zero meaning absent. The labels make the example easier to follow: these are synthetic numbers supplied directly to the network, not measurements extracted from photographs.

Put the strengths in a column vector, always in the order cat, dog, car, tree, boat. Cat alone at strength $0.6$ gives

$$x=\begin{bmatrix}0.6\\0\\0\\0\\0\end{bmatrix}.$$

The encoder is a matrix with two rows and five columns:

$$W\approx\begin{bmatrix} 1&0.309&-0.809&-0.809&0.309\\ 0&0.951&0.588&-0.588&-0.951 \end{bmatrix}, \qquad h=Wx.$$

Each row produces one hidden coordinate. For this input, $h=(0.6,0)$: two numbers now carry the cat input.

The column view tells us how that happened. Each column specifies where one input feature sends the hidden vector. Call these columns $f_{\text{cat}},f_{\text{dog}},\ldots,f_{\text{boat}}$. Then

$$h=x_{\text{cat}}f_{\text{cat}}+x_{\text{dog}}f_{\text{dog}}+ x_{\text{car}}f_{\text{car}}+x_{\text{tree}}f_{\text{tree}}+ x_{\text{boat}}f_{\text{boat}}.$$

This is the familiar rule that a matrix builds its output by adding scaled columns. Here each column has an additional interpretation: it is a feature vector, describing how an input feature contributes to the hidden representation.

We chose five unit-length arrows spaced evenly around a circle, with $72^\circ$ between neighbors. The matrix displays rounded coordinates; calculations below use the exact angles.

Dog at strength $0.6$, for example, produces approximately $(0.185,0.571)$. Both hidden neurons respond to that one feature. Cat happens to lie on an axis because we drew it there; rotating all five arrows together would preserve their lengths and pairwise angles.

Five arrows fit easily in a plane. The difficult part is reading their contributions back out.

Why perpendicular directions make readout easier

Before reading all five features, let’s isolate what an angle does in a separate two-feature example. Put the dog direction along the first coordinate and the cat direction along the second:

$$v_{\text{dog}}=(1,0),\qquad v_{\text{cat}}=(0,1).$$

Both directions have length one. Dog strength $3$ and cat strength $2$ produce

$$h=3v_{\text{dog}}+2v_{\text{cat}}=(3,2).$$

Reading with the dog direction gives $v_{\text{dog}}^T h=3$; reading with the cat direction gives $v_{\text{cat}}^T h=2$. Each reader recovers its own feature strength because the other feature contributes a dot product of zero.

Now turn only the cat direction to $(0.6,0.8)$, keeping its length and both feature strengths fixed:

$$h=3(1,0)+2(0.6,0.8)=(4.2,1.6).$$

The dog reader returns $4.2$, although dog strength is still $3$. The extra $1.2$ comes from cat: its strength $2$ multiplies the overlap $v_{\text{dog}}^T v_{\text{cat}}=0.6$.

In the widget, select Overlap (0.6, 0.8), then move Angle between features toward $90^\circ$. Watch the unwanted contribution shrink while the original strengths remain fixed.

Change the angle, keep the feature strengths fixed

Dog strength = 3 · Cat strength = 2 · Both feature directions have length 1

Angle between features

90°

Orthogonal (90°)Overlap (0.6, 0.8)

h = 3(1, 0) + 2(0, 1) = (3, 2)

Actual dog strength3

Dog dot-product readout3

Contribution from cat0

The orange projection shows how much of the cat direction the dog reader picks up. Changing the angle changes that readout, not the original feature strengths.

This is interference in a reader that uses the same direction as the feature being written. It does not mean nonorthogonal directions always lose information. In this two-dimensional example, a different decoder can recover cat strength as $h_2/0.8=2$, then dog strength as $h_1-0.6(2)=3$. The two directions are still linearly independent. Why a dot product does not automatically recover a feature develops this distinction.

Orthogonality makes these simple dot-product readers exact. It is not a requirement imposed by the training objective, and it is not available to every feature when we have more features than dimensions. Return now to our five arrows in a plane, where we must manage their overlaps.

Reading along an arrow picks up its neighbors

For the decoder, transpose the encoder matrix:

$$z=W^T h.$$

Transposing turns each feature column into a reader row. Cat’s output is $f_{\text{cat}}^T h$, dog’s is $f_{\text{dog}}^T h$, and so on. Reusing the encoder weights this way is called tying the weights. In the language of read with a key, write with a value, the same hidden-space arrow serves as a writing direction during encoding and a reading direction during decoding. It makes the example easy to inspect, but $W^T$ is not an inverse of $W$; other architectures can learn different readers and writers.

With cat alone, $h=0.6f_{\text{cat}}$, so every output has the form

$$z_j=0.6\,f_j^T f_{\text{cat}}.$$

Cat reads its own arrow and returns $0.6$. Dog’s arrow partly points in the same direction, so dog responds too. Car points mostly the other way and produces a negative response.

Output	Angle to cat	Raw output, approximately
Cat	$0^\circ$	$0.600$
Dog	$72^\circ$	$0.185$
Car	$144^\circ$	$-0.485$
Tree	$144^\circ$	$-0.485$
Boat	$72^\circ$	$0.185$

Only cat was present, yet all five readers respond. The unwanted responses are interference: one feature affects another feature’s readout.

The dot product explains the signs:

$$f_i^T f_j=\|f_i\|\,\|f_j\|\cos\theta.$$

At $90^\circ$, the cosine is zero, so neither arrow contributes to the other’s readout. Smaller angles produce positive contributions; larger angles produce negative ones. This is why perpendicular directions are useful here.

But in a two-dimensional space, at most two nonzero vectors can be mutually perpendicular. We cannot give all five features that protection. Orthogonality is a stronger condition than linear independence: cat and dog are linearly independent, yet their $72^\circ$ angle produces interference. The subspaces article explains the dimension count: we can draw arbitrarily many directions in a plane, but at most two can be linearly independent.

ReLU removes the negative outputs

Our target strengths are nonnegative. Apply ReLU to the raw outputs:

$$\hat x=\operatorname{ReLU}(W^T h), \qquad \operatorname{ReLU}(s)=\max(0,s).$$

The cat example now reconstructs approximately as

$$\hat x=(0.600,\ 0.185,\ 0,\ 0,\ 0.185).$$

Car and tree disappear from the reconstruction. Cat survives unchanged. Dog and boat remain as false positive responses.

Notice where the nonlinearity sits: ReLU acts on the five decoder outputs. The two hidden coordinates remain signed. Clipping them would change the encoding and erase some of our arrows’ contributions.

ReLU also acts on each output’s total score. It does not inspect the separate contributions and remove every negative one. When several features are active, a negative contribution can reduce a desired positive signal before ReLU sees it. We’ll encounter that limitation shortly.

A negative bias sets a threshold

Dog and boat respond weakly. Could we require a stronger score before allowing an output through?

Add a bias of $-0.25$ to each output, then apply ReLU:

$$\hat x=\operatorname{ReLU}(W^T h+b), \qquad b=(-0.25,-0.25,-0.25,-0.25,-0.25).$$

Now a raw score must exceed $0.25$ to produce a positive output.

Output	Raw score	After bias	After ReLU
Cat	$0.600$	$0.350$	$0.350$
Dog	$0.185$	$-0.065$	$0$
Car	$-0.485$	$-0.735$	$0$
Tree	$-0.485$	$-0.735$	$0$
Boat	$0.185$	$-0.065$	$0$

The threshold removes all four unwanted responses. But cat is now too small: we supplied $0.6$ and recovered $0.35$.

The bias cannot tell which part of a score we wanted. It subtracts the same amount from cat’s response as from dog’s. This is the cost of using a threshold to suppress interference.

Longer arrows compensate, with another cost

For cat alone, its own raw score is

$$f_{\text{cat}}^T(0.6f_{\text{cat}}) =0.6\|f_{\text{cat}}\|^2.$$

The length appears twice: once when the encoder writes cat into the hidden space, and again when the decoder reads it. That is why the length is squared.

To recover $0.6$ after subtracting $0.25$, we need

$$0.6\|f_{\text{cat}}\|^2-0.25=0.6, \qquad \|f_{\text{cat}}\|=\sqrt{\frac{0.85}{0.6}}\approx1.19.$$

Suppose we lengthen all five arrows to this value, preserving their directions. Each isolated feature at strength $0.6$ then gets its desired self-response back.

However, the neighboring responses grow too. Dog’s raw response to cat becomes

$$0.6\times\frac{0.85}{0.6}\times\cos72^\circ \approx0.263.$$

After the bias, about $0.013$ remains. Lengthening the arrows restores cat, but lets a little positive interference through again.

Nor does this length correct every possible cat strength. With these same weights and bias, cat’s own output is approximately $0.175$ for input $0.3$, and $1.025$ for input $0.9$. An isolated cat strength at or below $0.25/(0.85/0.6)\approx0.176$ produces no cat output at all: the threshold erases a genuine weak signal. We calibrated the compensation for one strength, $0.6$.

Training can adjust directions, lengths, and individual biases together. The useful compromise depends on which features occur, their strengths, and the cost of reconstruction errors. There is no single arrow length or threshold that solves every input distribution.

Activate two features and watch their contributions mix

Return to the original unit-length arrows and remove the bias so we can isolate the interference. Activate cat and dog at strength $0.6$ each:

$$h=0.6f_{\text{cat}}+0.6f_{\text{dog}} \approx(0.785,0.571).$$

Cat’s reader now measures

$$f_{\text{cat}}^T h =0.6+0.6\cos72^\circ \approx0.785.$$

Dog’s reader also returns about $0.785$. Both outputs are too large because each active feature adds a positive contribution to the other’s readout.

Now replace dog with car, keeping both strengths at $0.6$. Their angle is $144^\circ$, so each reads

$$0.6+0.6\cos144^\circ\approx0.115.$$

Both features remain positive after ReLU, but most of their strength has disappeared from the reconstruction. Their sum also produces a false dog response of about $0.371$: dog’s arrow lies between them. A bias of $-0.25$ would suppress the two genuine features completely while leaving that false response positive.

These comparisons explain why which features occur together matters. Nearby arrows can inflate each other’s outputs. Widely separated arrows can suppress them. A threshold that worked for one active feature need not work for a mixture.

Sparse inputs help by limiting how many contributions arrive at once. Here sparsity means few nonzero entries in the five input strengths, not necessarily zeros in the hidden coordinates. Dog alone already uses both hidden coordinates.

Rare features can also be tightly correlated: two features might appear only occasionally but always appear together. Their interference would occur whenever either was present. The learned bottleneck experiment varies frequency and co-occurrence separately to investigate that distinction.

Put all the interference in one matrix

Substituting $h=Wx$ into the decoder gives

$$\hat x=\operatorname{ReLU}(W^TWx+b).$$

The entry in row $j$, column $i$ of $W^TW$ is $f_j^T f_i$. Diagonal entries measure each feature’s response to itself. Off-diagonal entries measure how one feature affects another’s reader.

For an arbitrary output $j$, the score before ReLU is

$$z_j+b_j= \underbrace{x_j\|f_j\|^2}_{\text{own contribution}} +\underbrace{\sum_{i\ne j}x_i f_j^T f_i}_{\text{other features' contributions}} +b_j.$$

This equation collects everything we observed. Length changes the self-response. Angles and lengths determine the cross-responses. The active input strengths determine which terms enter the sum. The bias shifts the total before ReLU clips it.

If all five columns were orthonormal, $W^TW$ would be the five-dimensional identity matrix, and the unbiased decoder would reconstruct every input. But our $W$ has only two rows, so $W^TW$ has rank at most two. It cannot be that identity matrix.

Nonlinearity allows different responses in different regions of the hidden space. It does not remove the bottleneck’s ambiguity: if two allowed inputs produce the same $h$, any decoder receiving only $h$ must give them the same answer.

Higher dimensions give us more nearly perpendicular directions

In our plane, neighboring arrows are only $72^\circ$ apart. What happens if the hidden layer has 1,024 coordinates?

We can then choose 1,024 mutually orthogonal nonzero vectors. We still cannot choose more than 1,024 that are all exactly orthogonal. But approximate orthogonality is much less restrictive.

To see why, we’ll switch from our carefully spaced arrows to randomly chosen directions. This gives us a simple baseline: even without arranging the arrows to avoid each other, high-dimensional geometry makes most pairs nearly perpendicular. Learned directions need not follow this random model.

The dot product adds coordinate-wise contributions. Under this random model, positive and negative contributions tend to cancel. The high-dimensional geometry chapter of Foundations of Data Science develops the connection between random directions, concentration, and random projection.

For independently sampled, uniformly random unit vectors $u$ and $v$ in $d$ dimensions,

$$\mathbb E[u^Tv]=0, \qquad \operatorname{Std}(u^Tv)=\frac1{\sqrt d}.$$

The spread of their dot products shrinks as dimension increases. At $d=1024$, the standard deviation is $0.03125$. Since unit vectors’ dot products are their angle cosines, most pairs are nearly perpendicular. Near $90^\circ$, that spread corresponds to an angular standard deviation of approximately $1.79^\circ$.

Imagine generating 102,400 such directions: one hundred times as many directions as coordinates. A histogram of sampled pairwise angles would still concentrate around $90^\circ$. Having more vectors adds more pairs; it does not change the distribution for an independently chosen distinct pair.

This is a statement about typical pairs. With billions of possible comparisons, some pairs will overlap more strongly. A narrow histogram does not establish that every angle falls within a particular interval.

For an experiment, sampling pairs is enough to estimate that histogram. Constructing the full matrix of pairwise dot products would require over ten billion entries, about 84 GB in float64 before calculating the angles.

Optional: why a ball’s volume concentrates near its boundary

Another high-dimensional effect concerns distance from the origin. If a point is sampled uniformly from the interior of a unit ball in $d$ dimensions, the fraction of volume inside radius $r$ is $r^d$, for $0\le r\le1$. The outermost 10% of the radius therefore contains a fraction

$$1-0.9^d$$

of the total volume. At $d=100$, that is about $99.9973\%$. Change Dimensions below to compare it with lower-dimensional balls.

How much volume lies in the outer 10% of the radius?

Dimensions

100

Inner ball: radius at most 0.90.00266%

Outer shell: radius 0.9 to 199.99734%

The bar shows fractions of volume, not a cross-section of the ball. Very small inner volumes may be too narrow to see.

This statement assumes a uniform distribution inside a ball. It does not say that arbitrary model activations lie on a sphere. It also concerns radius, while our feature-interference argument concerns angles between directions. The two concentration effects are discussed in Foundations of Data Science, chapter 2.

Small overlaps can still add up

Suppose feature $j$ is absent, while $k$ other unit-length feature vectors are active with strength one. Its raw readout is

$$z_j=\sum_{i\text{ active}}f_j^T f_i.$$

Under the independent random-direction model, the terms have mean zero and variance $1/d$. Their signs fluctuate, so their typical accumulated size grows like the square root of the number of terms:

$$\operatorname{Std}(z_j)=\sqrt{\frac{k}{d}}.$$

Other active features	Raw interference standard deviation in 1,024 dimensions
$1$	$0.031$
$10$	$0.099$
$100$	$0.313$
$1{,}000$	$0.988$

These numbers describe variability across random constructions for a fixed active set. They are not reconstruction errors after thresholding, worst-case bounds, or measured results from a trained model. Different strengths change the variance to $\sum_i x_i^2/d$ under the same assumptions.

The table makes the roles of dimension and sparsity visible. More dimensions weaken typical individual overlaps. Fewer active features limit how much overlap accumulates. Bias and ReLU then act on the remaining total, trading false positives against weakened or missed true features.

Compare dimensions in the widget below. At 1,000 dimensions, one pair’s dot product has standard deviation about $0.0316$, but summing contributions from 1,000 other active features gives standard deviation $1$. Keep Hidden dimensions fixed and reduce Other active features to see why sparsity matters even when individual overlaps are small. Select 1,024 dimensions to reproduce the table above.

More dimensions, fewer active features

Independent, uniformly random unit directions · Other active features have strength 1

2D

0.7071

10D

0.3162

100D

0.1

1,000D

0.0316

10,000D

0.01

Bars use one linear scale from 0 to 1 and show 1/√d. These are theoretical standard deviations, not sampled angles or bounds on every pair.

Hidden dimensions2101001,0001,02410,000Other active features

10

One pair: 1/√d0.0316

Accumulated: √(k/d)0.1

Both readouts are standard deviations around zero. The accumulated value is for one absent feature's raw readout before bias or ReLU; it can exceed 1 because it is a sum, not a cosine.

With many inactive features, there are also many opportunities for an unwanted response. Good behavior for one reader does not guarantee good behavior for all readers at once.

Going further: what the Johnson–Lindenstrauss lemma adds

The random-pair observation has a stronger geometric relative. The Johnson–Lindenstrauss lemma says that a finite set of $m$ points can be mapped into a space with

$$d=O\!\left(\frac{\log m}{\varepsilon^2}\right)$$

dimensions while preserving all pairwise squared distances within multiplicative factors $1-\varepsilon$ and $1+\varepsilon$. The original space may have many more dimensions. A suitably scaled Gaussian random matrix supplies one construction; see these lecture notes on random projection.

To connect this to feature arrows, start with the origin and many orthogonal unit vectors. Each vector is distance one from the origin, and every pair of distinct vectors has squared distance two. Preserve those distances approximately, and the images retain approximately unit length and small pairwise dot products, because

$$u^Tv=\frac{\|u\|^2+\|v\|^2-\|u-v\|^2}{2}.$$

The logarithm means that the number of approximately separated directions can grow exponentially with dimension for a fixed nonzero tolerance, with constants and tolerance determining the rate. It does not give an exact feature capacity for a neural network.

In particular, preserving this finite collection of points does not establish recoverability of every weighted sum of them. The input combinations and the decoder still matter. Our five-arrow example already showed that individually distinguishable features can interfere when combined.

What the two neurons mean

Return to our encoder. Its first hidden coordinate is approximately

$$h_1=x_{\text{cat}}+0.309x_{\text{dog}}-0.809x_{\text{car}} -0.809x_{\text{tree}}+0.309x_{\text{boat}}.$$

A change in this neuron alone does not identify which feature changed. Several features can raise it, and several can lower it. Their effects on the second coordinate provide more context.

This is a concrete way polysemanticity can arise: one coordinate responds to several features. Our five-directions-in-two-dimensions construction also exhibits superposition, although mixed neuron responses alone do not establish that there are more features than dimensions.

We knew the five arrows because we chose them. In a trained model, finding useful feature directions is a separate problem, as is testing what those directions mean. Directions, sparse features, and how sparse autoencoders find them explores how a learned dictionary can recover ingredients from examples. The practical SAE article takes that question to activations collected from a trained network.

To see how an objective selects a representation, return to the geometry learned by the bottleneck experiment. Its trained weights show why feature co-occurrence and the decoder matter. Geometry also needs to be interpreted together with the reader: changing the geometry without changing predictions demonstrates that angles alone do not uniquely determine a model’s behavior.

Our construction makes the geometric opportunity and its cost explicit: a small activation space can hold many candidate feature directions, while the combinations we can read reliably depend on their overlaps, their activity, and the decoder.