https://chatgpt.com/c/6af0ecec-f270-83eb-8937-e96cab4a6b1b

In Feature detectors: how neurons read and write, two readers turned three word counts into promotional and routine-message responses, which a single output neuron combined into a classification score. Here we will focus on the two reader responses before that final score. For “Claim your prize. Claim it today.”, they form the pair $(4,0)$. Could these two responses share just one coordinate? We will try that next, then ask how several feature directions can share a space with fewer dimensions.

In Towards Monosemanticity, researchers decomposed activations from a layer with 512 neurons into more than 4,000 learned features. That does not mean they found 4,000 linearly independent axes inside a 512-dimensional space. They learned a larger collection of patterns and described each activation using a small selection of them.

A sparse autoencoder, or SAE, learns this kind of description. We’ll start with two features encoded in one signed number, then expand to four features in three dimensions. We’ll see how nonlinear decoding separates features, when their contributions become indistinguishable, and how shared coordinates can lead to polysemanticity. Then we’ll train a small SAE on a known dictionary so we can check what it recovers.

For the mathematical foundation, start with Feature detectors: how neurons read and write, which connects dot-product measurements to scaled vector contributions. Finding features in a tiny network then provides a learned example with brightness and contrast directions. Here we extend that picture to several features sharing a smaller set of coordinates.

Many features, a small vector

Choose a particular layer and token position in a language model. Its activation might have 128 coordinates:

$$x=(x_1,x_2,\ldots,x_{128})\in\mathbb R^{128}.$$

The coordinate count tells us the dimension of this activation space. It does not tell us how many different properties the model might represent across all the inputs it encounters.

A feature, in the picture we will explore, is a recurring property with a numerical strength. A pattern associated with French text might have a large strength on some inputs and a small or zero strength on others. Whether a particular learned pattern deserves that description has to be tested; the numbers do not arrive with labels.

For now, imagine that we know the features. Each has a fixed vector $v_i$ describing how it contributes to the activation. The activation is their weighted sum:

$$x=f_1v_1+f_2v_2+\cdots+f_nv_n.$$

The scalar $f_i$ says how strongly feature $i$ is present. The vector $v_i$ says how that strength affects the coordinates of $x$. This is an exact construction in our example. When applied to a real model, it becomes a hypothesis about its representation, usually with some reconstruction error.

Two counts appear here: the number of coordinates in $x$, and the number of possible ingredients in the sum. Understanding why they can differ is the first step toward understanding an SAE.

Two features in one number

Return to the same three-word vocabulary, in the order claim, prize, meeting. Use the promotional and routine-message readers from the previous article:

$$p=1.5x_{\mathrm{claim}}+x_{\mathrm{prize}}, \qquad r=x_{\mathrm{meeting}}.$$

Here $p$ is the promotional response and $r$ is the routine-message response. Both are nonnegative for our word-count inputs. These are two feature strengths that can be present together, not mutually exclusive class labels. We are keeping the readers fixed and changing how their responses are represented.

Message	Word counts	Responses $(p,r)$
Claim your prize. Claim it today.	$(2,1,0)$	$(4,0)$
The project meeting is tomorrow.	$(0,0,1)$	$(0,1)$
Claim your prize at the meeting.	$(1,1,1)$	$(2.5,1)$

These two responses occupy separate coordinates in the original hidden vector. Now give an encoder just one output, with weights $+1$ for $p$ and $-1$ for $r$:

$$s=E\begin{bmatrix}p\\r\end{bmatrix} =\begin{bmatrix}1&-1\end{bmatrix}\begin{bmatrix}p\\r\end{bmatrix} =p-r.$$

We call the new encoder $E$ and its scalar output $s$ to distinguish them from the previous article’s three-input weight matrix $W$ and two-coordinate activation $h$. This adds a new compression step: three word counts → two reader responses → one signed number. Our reconstruction target is the two responses, not the original word counts.

For the original promotional message, $s=4-0=4$. For the routine message, $s=0-1=-1$. If at most one response is nonzero, the sign identifies which one and the magnitude gives its strength. The two features share one coordinate but occupy opposite sides of zero.

A nonlinear decoder reads the sign

Use the transpose $E^T$ as the decoder matrix. This is called tied weights: the encoder and decoder share parameters. Here those parameters are fixed; an untied decoder would have its own weights.

Multiply the scalar by the decoder column, then apply ReLU to each output:

$$\begin{bmatrix}\hat p\\\hat r\end{bmatrix} =\operatorname{ReLU}\left(\begin{bmatrix}1\\-1\end{bmatrix}s\right) =\begin{bmatrix}\max(0,s)\\\max(0,-s)\end{bmatrix}.$$

For the promotional message, $s=4$ gives $(4,-4)$ before ReLU and $(4,0)$ afterward. For the routine message, $s=-1$ gives $(-1,1)$ before ReLU and $(0,1)$ afterward. ReLU removes the negative response in the absent feature’s coordinate; the word patterns were measured by the earlier readers.

ReLU is applied at the decoder output. The compressed scalar stays signed. Clipping $s$ itself would erase the negative routine-message code.

Without ReLU, this decoder produces a negative response for the absent feature. More generally, a bias-free linear encoder–decoder through one scalar has rank at most one, so it cannot reconstruct both independent inputs $(4,0)$ and $(0,1)$ exactly. The nonlinear decoder succeeds on the restricted set where only one response is nonzero.

Transposing the encoder does not invert it:

$$E^TE =\begin{bmatrix}1\\-1\end{bmatrix}\begin{bmatrix}1&-1\end{bmatrix} =\begin{bmatrix}1&-1\\-1&1\end{bmatrix}\ne I.$$

Before ReLU, every output lies on the line $(s,-s)$. ReLU maps its positive half onto the promotional-response axis and its negative half onto the routine-response axis.

A mixed message exposes the limitation

“Claim your prize at the meeting.” activates both readers: $p=2.5$ and $r=1$. Its compressed code is $s=1.5$, and the decoder returns $(1.5,0)$. We have lost the two original strengths.

The one-word message “Claim!” also produces $s=1.5$, because its responses are $(1.5,0)$. The compressed number cannot tell these messages’ response pairs apart.

Responses $(p,r)$	Encoded $s=p-r$	Before ReLU $(s,-s)$	Reconstruction	Exact?
Promotion only: $(4,0)$	$4$	$(4,-4)$	$(4,0)$	Yes
Routine only: $(0,1)$	$-1$	$(-1,1)$	$(0,1)$	Yes
Both: $(1,1)$	$0$	$(0,0)$	$(0,0)$	No
Neither: $(0,0)$	$0$	$(0,0)$	$(0,0)$	Yes
Promotion stronger: $(2,1)$	$1$	$(1,-1)$	$(1,0)$	No
Routine stronger: $(1,2)$	$-1$	$(-1,1)$	$(0,1)$	No
Mixed message: $(2.5,1)$	$1.5$	$(1.5,-1.5)$	$(1.5,0)$	No
“Claim!”: $(1.5,0)$	$1.5$	$(1.5,-1.5)$	$(1.5,0)$	Yes

With only one nonnegative response active, reconstruction is exact at any strength. With both active, their contributions subtract: equal strengths disappear entirely, and unequal strengths leave only their difference.

The response pairs $(1,0)$ and $(2,1)$ differ by $(1,1)$, and

$$E\begin{bmatrix}1\\1\end{bmatrix}=1-1=0.$$

That difference lies in the encoder’s null space: adding it leaves the code unchanged. No decoder receiving only $s$ can distinguish those inputs. Both features together likewise produce the same code as neither.

Select Promotion only and Routine only, then compare Both with Neither. Compare Mixed message with Same difference to see the two different response pairs that produce $s=1.5$. Switching between Linear decoder and ReLU decoder keeps the encoder fixed, isolating what the output nonlinearity does.

Two message responses, one signed number

Promotion onlyRoutine onlyBothNeitherPromotion strongerRoutine strongerMixed messageSame difference

Message: “Claim your prize. Claim it today.”

Promotional response p

4

Routine response r

0

Encode: s = 1 × 4 + (−1) × 0 = 4

Linear decoderReLU decoder

Multiply by the decoder columnEᵀs = (1 × 4, −1 × 4) = (4, -4)

Apply ReLU to each coordinate(max(0, 4), max(0, -4)) = (4, 0)

Reconstructed promotional response4max(0, s)

Reconstructed routine response0max(0, −s)

Original responses: (4, 0) · Reconstructed: (4, 0)Exact reconstruction

With only one response active, the sign identifies which one and the magnitude gives its strength. ReLU removes the negative value in the other output.

ReLU has not restored destroyed information. The signed scalar already distinguished the single-feature inputs; the nonlinear decoder reads that distinction. Once different allowed inputs produce the same scalar, the ambiguity remains.

The important condition is which feature combinations occur. Mutual exclusion makes this example exactly recoverable, but our mixed message shows that promotional and routine words need not be mutually exclusive. Sparse activity makes collisions less common when feature occurrences are independent, but two rare features can still occur together every time. Sparsity and co-occurrence are different properties of the data.

These weights were chosen by hand. In When networks learn to share a bottleneck, we train weights in synthetic examples and vary feature frequency, co-occurrence, and decoder nonlinearity. Here, we will generalize from two responses sharing a line to more features sharing a three-dimensional space.

A feature can change several coordinates

The signed example gave the two responses opposite directions on one line. To explore a larger dictionary, we will now use four abstract features, A through D, in a three-dimensional activation space. These are illustrative feature strengths, not additional word groups learned by our classifier. Suppose feature D contributes along

$$v_D=(1,1,1).$$

If its strength is $0.7$ and no other features are active, the activation is

$$x=0.7v_D=(0.7,0.7,0.7).$$

Increasing D’s strength from $0.7$ to $1$ changes the activation by

$$\Delta x=0.3v_D=(0.3,0.3,0.3).$$

All three coordinates increase together. The feature has a direction even though it has no dedicated coordinate.

Three objects have different roles here. The point $x$ represents a particular activation. The vector $v_D$ specifies the change caused by one unit of D. The coefficient $f_D$ determines how much of that vector contributes. Strictly, $v_D$ specifies both a direction and a scale; dividing it by its length preserves the direction while changing what one unit of the coefficient means.

The same geometry works in 128 dimensions. A feature vector can contain 128 nonzero entries. Changing one feature can change every coordinate of the activation.

Sparse features can produce dense activations

Give our example four possible features, A through D. Their strengths form a separate vector:

$$f=(f_A,f_B,f_C,f_D).$$

For an input with only D active, we might have

$$f=(0,0,0,1), \qquad x=(1,1,1).$$

Only one of the four feature strengths is nonzero, while all three activation coordinates are nonzero.

Sparsity describes how many feature strengths are active for an input. It does not require the resulting activation vector to contain zeros. It also does not require a feature’s direction vector to contain zeros.

Four features make the distinction visible; practical dictionaries can contain thousands or millions of candidates, with a much smaller number active on each input. Features also need not be mutually exclusive: an input can activate several at once.

For an ordinary transformer, the vector $f$ in this explanation need not exist as an explicit array stored by the model. We are proposing a way to decompose its activations. An SAE will create an explicit code that attempts to describe them in this form.

Four directions fit in three dimensions

Here is our complete dictionary:

Feature	Vector in activation space
A	$v_A=(1,0,0)$
B	$v_B=(0,1,0)$
C	$v_C=(0,0,1)$
D	$v_D=(1,1,1)$

The first three vectors are mutually orthogonal: each pair has a dot product of zero. They also have unit length. If an activation contains only contributions from A, B, and C, taking its dot product with one of these vectors recovers that feature’s strength without picking up the other two.

These vectors are also linearly independent: none can be constructed by combining the other two. They already span the entire 3D space.

D points in a fourth direction, but

$$v_D=v_A+v_B+v_C.$$

It therefore adds no independent dimension. The four vectors are linearly dependent: there is a nontrivial combination of them that equals zero. Unlike the first three directions, D is not orthogonal to any of them: $v_A^\top v_D=v_B^\top v_D=v_C^\top v_D=1$. A readout along A therefore responds to D as well. This overlap matters when we try to recover which features contributed.

A 3D space contains infinitely many possible directions, but at most three linearly independent vectors. Rotating the coordinate axes does not change that limit. Likewise, a 128-dimensional space can contain thousands of distinct feature directions while allowing at most 128 linearly independent ones.

Write the four vectors as columns of a matrix:

$$D= \begin{bmatrix} 1&0&0&1\\ 0&1&0&1\\ 0&0&1&1 \end{bmatrix}, \qquad x=Df.$$

The multiplication makes the overlap explicit:

$$x_1=f_A+f_D,\qquad x_2=f_B+f_D,\qquad x_3=f_C+f_D.$$

Each coordinate combines contributions. In particular, the vector $x=(1,1,1)$ could come from either

$$f=(0,0,0,1) \qquad\text{or}\qquad f=(1,1,1,0).$$

A fourth arrow is easy to draw. Recovering which arrows contributed is the difficult part.

Why sparsity helps—and where it stops helping

Suppose we know that each input activates at most one feature, with a nonnegative strength. In our example, a nonzero activation then lies on one of four distinct rays.

A point $(0.8,0,0)$ identifies A. A point $(0.8,0.8,0.8)$ identifies D. Their amplitudes tell us the feature strengths. Although the dictionary is linearly dependent, restricting the allowed combinations makes these cases distinguishable.

Use the controls below to explore the difference between the coefficients and the resulting point. Start with Only D, adjust Feature D, and then select A + B + C.

Four features. Three coordinates.

Only DA + B + CAll fourClear

Feature strengths f

Feature A

0.00

Feature B

0.00

Feature C

0.00

Feature D

1.00

Rotate view

Feature strengths f[0.00, 0.00, 0.00, 1.00]1 of 4 active features

Activation coordinates x[1.00, 1.00, 1.00]3 of 3 nonzero coordinates

The last switch changes three feature strengths and turns D off, but the black point returns to exactly the same location. Once both explanations are allowed, seeing the point cannot tell us which occurred.

In matrix terms,

$$D\begin{bmatrix}-1\\-1\\-1\\1\end{bmatrix}=0.$$

Adding this coefficient change leaves the activation unchanged whenever the resulting coefficients remain allowed. A decoding rule can prefer one explanation, but the activation alone cannot establish that it was the original one.

In particular, a code that favors one active entry can describe A, B, and C acting together using D alone. It has found a shorter description. That does not establish that the system producing the point had a separate D mechanism.

This is why sparsity helps without providing a universal guarantee. Recovery also depends on the dictionary geometry, the combinations that occur, and the amount of noise. Two nearly parallel directions can be hard to distinguish even when only one feature is active.

In the feature-count sense used by Toy Models of Superposition, superposition means representing more features than the activation space has dimensions, using overlapping directions. Its small networks trained on sparse synthetic features exhibit this behavior, with interference that nonlinear computation helps manage. Our hand-specified dictionary illustrates the storage and recovery problem. It does not show a task network learning to arrange the directions that way.

The number of mathematical directions is therefore not a promise of unlimited usable capacity. As more features compete for the same space, their overlap matters.

There is a separate question about why a trained network shares its available space. Do its computations share useful structure, or are unrelated computations packed together because they rarely interfere? Nick Cammarata’s Goodfire essay Neural Networks Are Already Clean focuses on the latter meaning of superposition. It challenges that account of real networks while accepting the synthetic toy results. Here we use the feature-count definition; our construction does not settle how real networks organize their computations.

Shared coordinates can produce polysemanticity

A single activation coordinate can contain contributions from several features. In the one-number example, positive $s$ indicates the promotional response and negative $s$ indicates the routine-message response when they occur separately. In the four-direction example, $x_1=f_A+f_D$, so either A or D can increase the same coordinate.

This is a way polysemanticity can arise: one neuron or coordinate carries information associated with several features. Superposition describes how features share the representation; polysemanticity describes the mixed responses observed in individual coordinates. They are related, but not interchangeable.

Mixed responses alone do not prove that there are more features than dimensions. Even an invertible rotation of two independent features into two coordinates can make both coordinates respond to both features. Nor does a neuron responding to promotional and routine messages necessarily mix unrelated concepts: it might detect a word pattern common to both. The hand-built signed code above specifies two features by construction; interpreting a trained neuron’s responses requires additional evidence.

For the larger dictionary, many feature directions can have nonzero components along the same coordinate. Examining that coordinate alone loses the context supplied by the other coordinates. This motivates searching for feature directions across the activation space and testing how downstream computations use them. An SAE will attempt to infer a sparse feature description from the full vector.

What one neuron reads from its inputs

A feature detector measures how strongly an input matches a pattern. Feature detectors: how neurons read and write introduces a simple detector as a dot product, then adds a bias and activation. Here we can ask what that detector measures when its input coordinates already mix several features.

To connect this geometry to a neuron, consider one that receives three numbers through three weighted connections. It multiplies each input $x_i$ by its weight $w_i$, adds the contributions and a bias, then applies an activation function. With ReLU, its output is

$$a=\operatorname{ReLU}(s), \qquad s=w_1x_1+w_2x_2+w_3x_3+b.$$

The inputs might be measurements supplied to the network or outputs from a previous layer. Start with a representation where each input measures a separate feature. For an illustrative image example, suppose the inputs are nonnegative strengths on a common scale:

Incoming input	Feature strength	Connection weight	Effect on the score
$x_1$	Curved-edge evidence	$w_1=2$	Increasing $x_1$ raises the score twice as much
$x_2$	Straight-edge evidence	$w_2=0$	Changing $x_2$ has no direct effect
$x_3$	Corner evidence	$w_3=-1$	Increasing $x_3$ lowers the score

With bias $b=-0.5$, this neuron computes

$$a=\operatorname{ReLU}(2x_1-x_3-0.5).$$

For $x=(0.8,0.4,0.2)$, the score is $1.6-0.2-0.5=0.9$, so the output is $0.9$. ReLU outputs zero whenever the score is zero or negative. The weights determine how the inputs contribute; the bias sets the threshold for a positive output. These illustrative weights are chosen by hand. In a trained network, learning adjusts the weights and biases.

That makes it natural to say that weights determine which features a neuron reacts to. But the interpretation depends on how the features are represented. In our dictionary example, $x_1=f_A+f_D$: one incoming number already mixes two features. A weight on $x_1$ affects the neuron’s response to both.

Substituting the feature decomposition into the neuron’s score gives

$$s=w^\top x+b =\sum_i f_i\bigl(w^\top v_i\bigr)+b.$$

An individual weight $w_j$ controls sensitivity to coordinate $x_j$. The dot product $w^\top v_i$ controls sensitivity to feature $i$‘s direction. Increasing $f_i$ by $\Delta$, with the other feature strengths fixed, changes the score by $\Delta w^\top v_i$. The activation function then determines how that score change affects the output.

For D’s direction $(1,1,1)$, the reader $w=(2,0,-1)$ has sensitivity $2+0-1=1$. It responds to D, but it also responds to A. Responding to a feature does not mean responding exclusively to it.

This is the difference between representing a feature and detecting it. The vector $v_D$ describes how D contributes to the activation; $w$ describes how a neuron reads that activation. They need not point in the same direction. A useful detector may also respond to other features, and its bias and nonlinearity determine which mixtures produce an active output.

Why a dot product does not automatically recover a feature

If D lies along $(1,1,1)$, could we recover its strength simply by projecting onto that direction?

The coefficient of the orthogonal projection onto $v_D$ is

$$\frac{v_D^\top x}{v_D^\top v_D} =\frac{x_1+x_2+x_3}{3} =f_D+\frac{f_A+f_B+f_C}{3}.$$

We get the contribution from D plus contributions from the other features. If A alone has strength one, this measurement returns $1/3$, even though D is inactive.

This is why detecting a pattern is a weaker requirement than recovering its coefficient. As more feature directions share the activation space, we must consider which other features affect each readout and how often they occur together. The SAE below learns a collection of detectors and reconstruction vectors together, under a sparsity constraint; a dot product along a known feature direction does not solve that recovery problem by itself.

This separates two uses of a vector. A writing direction specifies a change to the activation:

$$x'=x+\alpha v.$$

A reading direction supplies weights for a scalar measurement:

$$s=w^\top x.$$

After the write, that measurement changes by $\alpha w^\top v$. The dot product tells us how sensitive the reader is to the writer. It does not establish that the reader responds only to that feature.

In fact, no linear reader can be zero on all three axis vectors while responding to D. Being zero on A, B, and C forces all three reader weights to zero.

A nonlinear rule can exploit the one-active-feature assumption. For our nonnegative example, consider

$$\tilde f_D=\min(x_1,x_2,x_3), \qquad (\tilde f_A,\tilde f_B,\tilde f_C) =(x_1,x_2,x_3)-\tilde f_D(1,1,1).$$

On each of the four allowed rays, this recovers the generating coefficients exactly. But give it the activation produced by A, B, and C together, and it reports D instead. Nonlinearity lets us use a restriction on the data; it cannot resolve an exact collision between allowed inputs.

We designed that rule using the known dictionary. With model activations, both the dictionary and a useful encoding rule must be learned.

The sparse autoencoder: infer coefficients, reconstruct the vector

An autoencoder takes an input, encodes it, and tries to reconstruct it. Here the input is an activation collected from another model.

The promotional/routine-response network showed how known features can share a coordinate. An SAE starts with the mixed activation and tries to discover a sparse set of ingredients that explains it:

	Promotional/routine-response example	Sparse autoencoder for interpretability
Input	Two known feature strengths	A vector of model activations
Encoding	Two strengths → one signed scalar	Activation coordinates → a larger set of candidate feature activations
Decoding	Tied weights, then output ReLU	A weighted sum of learned decoder vectors, usually with a bias
Weights	Chosen by hand	Learned from collected activations; encoder and decoder need not be tied
Sparsity	Assumed in the input combinations	Encouraged or enforced in the learned code
What we investigate	When shared coordinates preserve enough information	Whether a sparse dictionary reconstructs activations and reveals useful features

Our small SAE maps three activation coordinates to four code entries, then back to three coordinates:

Model activationx ∈ ℝ³3 coordinates

Encoder→Estimate strengths

Sparse codez ∈ ℝ⁴Few nonzero entries

Decoder→Add feature vectors

Reconstructionx̂ ∈ ℝ³3 coordinates

A common encoder is a learned affine map followed by ReLU:

$$z=\operatorname{ReLU}(W_{\mathrm{enc}}x+b_{\mathrm{enc}}), \qquad W_{\mathrm{enc}}\in\mathbb R^{4\times3}.$$

We use $z$ for the learned code to distinguish it from the known generating coefficients $f$ in our toy data. ReLU replaces negative scores with zero. Training will encourage a small number of positive entries.

The decoder reconstructs the activation:

$$\hat x=W_{\mathrm{dec}}z+b_{\mathrm{dec}}, \qquad W_{\mathrm{dec}}\in\mathbb R^{3\times4}.$$

Each column of $W_{\mathrm{dec}}$ is a learned vector in the original 3D space. Call these columns $d_A,\ldots,d_D$:

$$\hat x=b_{\mathrm{dec}}+z_A d_A+z_B d_B+z_C d_C+z_D d_D.$$

If only two code entries are nonzero, only two dictionary vectors contribute beyond the baseline $b_{\mathrm{dec}}$. This is the same weighted-sum construction we started with, now with learned ingredients.

To inspect the decoder alone, use our four known vectors as its columns and set the bias to zero. Change the code entries below to see their contributions add up. These coefficients are supplied manually; this widget does not train an SAE or infer a code from an input.

Build a reconstruction from code entries

Only DA + B + CAll fourClear

Code entries z

Feature A

0.00

Feature B

0.00

Feature C

0.00

Feature D

1.00

Rotate view

Each code entry scales one decoder vector

z_Ad_A = 0.00 × [1.00, 0.00, 0.00][0.00, 0.00, 0.00]

z_Bd_B = 0.00 × [0.00, 1.00, 0.00][0.00, 0.00, 0.00]

z_Cd_C = 0.00 × [0.00, 0.00, 1.00][0.00, 0.00, 0.00]

z_Dd_D = 1.00 × [1.00, 1.00, 1.00][1.00, 1.00, 1.00]

Bias = [0, 0, 0]. Add these four contributions to reconstruct x̂.

Code entries z[0.00, 0.00, 0.00, 1.00]1 of 4 active code entries

Reconstructed activation x̂[1.00, 1.00, 1.00]3 of 3 nonzero coordinates

Because there are more dictionary vectors than activation dimensions, this is an overcomplete dictionary. Its columns cannot form a linearly independent basis. The four code positions provide four candidate ingredients; they do not create a fourth independent dimension in the original space.

The encoder’s rows read from the activation space; the decoder’s columns write into it. They need not be the same vectors. Nor is the encoder a general linear inverse of the decoder: recovering arbitrary four-dimensional coefficients from three numbers would be impossible.

In a typical interpretability workflow, the original model stays frozen. We collect activations on many inputs and train the SAE separately. Labels such as “French text” are not supplied by the reconstruction objective.

Sparsity in the data and sparsity in the code

In the promotional/routine-response example, sparsity was a property of the inputs: few known features were active together. In our four-feature construction, it describes how many entries of the generating vector $f$ are nonzero. The resulting activation $x$ can still be dense.

In an SAE, sparsity describes the learned code $z$. The architecture and training objective encourage or enforce few active code entries, even when every input coordinate is nonzero. ReLU alone does not guarantee a sparse code: all its inputs could be positive.

Choice	What it affects
Dataset and sampled activations	Which patterns occur and how often they appear together
Original model	How those patterns affect activation coordinates
SAE dictionary size	How many candidate vectors are available
Sparsity penalty or activity budget	How many code entries are used, or how costly their activation is
Reconstruction objective	How accurately those entries must reproduce the activation

Changing the dataset can change the dictionary an SAE learns. Changing the sparsity constraint can also change it on the same data. A sparse code is therefore a modeling choice to evaluate, not proof that we have recovered the original features. The experiment below lets us check that distinction against a known dictionary.

Learning the dictionary from examples

A common SAE training objective combines reconstruction with a sparsity penalty:

$$\mathcal L =\underbrace{\|x-\hat x\|_2^2}_{\text{reconstruction error}} +\lambda\underbrace{\|z\|_1}_{\text{sum of code magnitudes}}.$$

The first term rewards an accurate reconstruction. The second makes activation expensive, encouraging the encoder to explain each example with fewer or smaller positive coefficients. ReLU supplies exact zeros; the penalty influences which entries remain active and how large they become.

The strength $\lambda$ controls a tradeoff. A weak penalty may leave a dense code; a strong one can discard useful information. Decoder scale must also be controlled. Otherwise the model could enlarge a decoder column and shrink its corresponding coefficient, reducing the penalty without changing the reconstruction. Constraining decoder columns to unit length removes that scaling shortcut.

Another approach, Top-K, keeps only the $k$ largest encoder activations and sets the rest to zero. It can train with reconstruction loss alone because the activity budget is enforced directly. Scaling and Evaluating Sparse Autoencoders investigates this approach and evaluates feature quality beyond reconstruction error.

Let’s test the smallest version: a Top-1 SAE, which can use at most one dictionary vector per example.

For this experiment, we keep the same four directions but normalize D to $(1,1,1)/\sqrt3$. All four vectors then have length one, so a coefficient of one contributes the same vector length for every feature. This changes D’s coefficient scale, not its direction.

We generate 4,096 training activations. Each has exactly one active feature, chosen uniformly, with strength sampled uniformly between $0.5$ and $1.5$. The SAE sees only the resulting 3D points. It receives neither the dictionary nor the feature strengths used to construct them.

The encoder and decoder have no biases. The encoder applies a linear map, ReLU, and Top-1; the decoder forms the weighted sum. We train five initializations for 2,000 updates each, then evaluate on 2,048 independent examples. The decoder starts from random positive columns, and the encoder starts from their transpose; the two matrices train separately. Positive initialization is a choice suited to this positive-octant dataset.

We also train three comparisons: an unconstrained ReLU autoencoder on the same one-feature examples, and Top-1 SAEs on examples containing exactly two or four active generating features.

Relative reconstruction error divides squared reconstruction error by the error of always predicting the training-set mean. Zero is perfect reconstruction; 100% is as bad as that constant baseline.

Matched direction cosine compares the learned decoder vectors with the generating dictionary. Feature numbering is arbitrary, so we try every one-to-one matching and report the best average cosine. A score of one means all four directions align. This comparison uses the known dictionary only for evaluation.

On the one-feature data, all five Top-1 runs recover the four directions to the displayed precision and reconstruct nearly perfectly. Each input lies on one of four rays, and the architecture can select a different ray for different inputs. Four directions are useful even though the surrounding space has only three dimensions.

The unconstrained autoencoder also reconstructs nearly perfectly, but its directions align less well with the generating dictionary and it uses several code entries per example. Reconstructing the point does not establish that we recovered its original ingredients.

With two or four generating features active, the Top-1 models have a different problem: a single dictionary vector generally cannot represent their sum. The learned vectors move toward mixtures, and reconstruction error increases. These rows deliberately mismatch the activity budget to the data. They do not establish what an SAE with a larger budget could recover.

Here, recovering the features means recovering the dictionary we used to generate the data. No language model is involved, and no task network was trained to produce that dictionary. The experiment establishes what these autoencoders learn on this particular distribution. Whether a network trained for another task develops such a representation is a separate empirical question.

Experiment protocol and downloadable results

All runs use Adam with learning rate $0.01$, batches of 256 examples sampled with replacement, and decoder-column normalization after each update. The initialization seeds are 7, 11, 23, 42, and 101. Within each activity condition, all five runs share the same training and test sets. Training has a fixed duration; test results do not select the stopping point or a preferred run.

The table reports the mean and range across those five initializations. These ranges describe this experiment, rather than uncertainty about real language models. Some four-feature runs leave a code entry unused; all runs remain in the results.

The complete Python experiment includes data generation, training, dictionary matching, and checks of the article’s geometry. The results JSON contains every run’s weights, individual matched cosines, reconstruction error, coefficient error, and activity counts. The repository’s sparse-experiment directory includes the setup instructions.

For a broader set of experiments, ARENA’s Toy Models of Superposition & Sparse Autoencoders explores how sparsity and feature importance affect learned representations, then guides you through training SAEs on those toy models. It is a practical next step from our fixed dictionary to investigating representations learned by a network.

What makes a learned direction interpretable?

Our toy features are named A through D because we constructed them. We can compare a recovered vector with the exact direction that generated the data.

Real model activations provide no such answer key. A learned dictionary entry could correspond to an individual mechanism, a combination of mechanisms, or a useful statistical pattern. Reconstruction and sparsity alone do not distinguish those possibilities. The ambiguity between D and A + B + C becomes a question about computation: what produces this activation, and what reads it?

Goodfire’s essay offers a working hypothesis: many original neurons are already interpretable, and an SAE can group existing mechanisms into descriptions of particular contexts. Its comparison of neurons and dictionary entries illustrates that possibility. This motivates comparing learned entries with the original neurons and tracing their connections, alongside evaluating the dictionary’s reconstruction quality.

To attach a label to an SAE entry, researchers inspect inputs that activate it and look for a recurring pattern. An entry that activates on many Python function definitions might suggest a hypothesis about function syntax.

That hypothesis needs examples beyond the most impressive matches. Does the feature also activate on unrelated text? Does it miss many examples that should fit its label? Does the description hold across contexts? A feature can mix patterns, and several learned entries can divide what a person would regard as one concept.

Interventions ask a further question: does changing the activation along this direction affect the model’s behavior? To increase code entry $i$ by $\Delta$ while retaining the original reconstruction residual, write

$$x'=x+\Delta d_i.$$

This isolates the chosen addition from the SAE’s reconstruction error. Replacing $x$ with the whole reconstruction would also change the residual $x-\hat x$, so reconstruction alone needs its own comparison.

An observed output change is evidence about the effect of that intervention. It does not prove that the vector represents only the proposed concept, or that the same edit works in every context. Feature descriptions, reconstruction quality, and causal effects are related measurements with different jobs.

The practical payoff of an SAE is a collection of candidate patterns that we can inspect and test. A readable label is an interpretation of the evidence, rather than a label guaranteed by the loss function.

Return to transformers: projection and sparsity are different operations

There is another place where a large vector becomes a smaller one: a transformer’s query projection. For example,

$$q=W_Qx, \qquad W_Q\in\mathbb R^{128\times12288}.$$

The output has 128 coordinates because the matrix has 128 rows. Each row computes a dot product with the 12,288-dimensional input. Those rows can read overlapping combinations of coordinates.

A small version makes the calculation visible:

$$\begin{bmatrix} 1&0&1\\ 0&1&1 \end{bmatrix} \begin{bmatrix}2\\3\\4\end{bmatrix} = \begin{bmatrix}6\\7\end{bmatrix}.$$

We reduced three coordinates to two, and both output coordinates are nonzero. The operation mixes input coordinates; dimensionality reduction does not imply a sparse output.

What disappears is the ability to distinguish inputs separated by a vector in the matrix’s null space. Here,

$$\begin{bmatrix} 1&0&1\\ 0&1&1 \end{bmatrix} \begin{bmatrix}1\\1\\-1\end{bmatrix} = \begin{bmatrix}0\\0\end{bmatrix}.$$

Thus $(2,3,4)$ and $(3,4,3)$ both map to $(6,7)$. The lost distinction involves all three input coordinates. In the larger example, rank–nullity gives a null space of dimension at least $12{,}288-128=12{,}160$.

This operation has a different purpose from the SAE:

	Query projection	Sparse autoencoder
Learned for	Computing query–key attention scores	Reconstructing collected activations with a sparse code
Typical shape in this example	$12{,}288\rightarrow128$	$128\rightarrow1{,}024\rightarrow128$
Sparsity	No requirement that the query contain zeros	Code activity is penalized or constrained
Place in the workflow	Part of the original model	Usually trained separately on a frozen model

The dimensions in the last column are illustrative. An SAE could analyze queries, residual-stream activations, or another chosen location; its dictionary vectors live in whichever activation space it was trained to reconstruct.

A linear map also transforms feature directions. If we change its input by $\alpha v$, then

$$W(x+\alpha v)=Wx+\alpha Wv.$$

The feature’s input displacement becomes $Wv$ at the output, or disappears if $Wv=0$. Through a whole transformer block, nonlinear operations and input-dependent attention can make the effect depend on the starting context.

For a closer look at what a matrix reads and erases, see vector decomposition and subspaces. The house-price compression experiment explores what happens when continuously varying attributes share a smaller hidden space. Our practical SAE article takes the next step of collecting activations from a trained network.

A 128-dimensional activation can have 128 nonzero coordinates while admitting a useful explanation in terms of a few active feature directions. Those directions can come from a dictionary much larger than 128. Learning that dictionary, estimating its coefficients, and testing what the entries mean are the separate pieces that make sparse autoencoders useful.