Finding features in a tiny network: brightness, contrast, and three neurons

In the previous article on what a weight matrix actually does, we followed an input through a weight matrix and watched it become a point in hidden space. Now we’ll work backward: given that hidden space, can we find directions that represent recognizable properties of the input?

For this experiment, we’ll move from XOR classification to a linear autoencoder: an image with two pixels, a hidden layer with three neurons, and an output layer that reconstructs the two pixels. The images have two properties we can control independently: brightness and left–right contrast.

We deliberately simplify the task so that those properties have exact, checkable representations. That lets us focus on what projections, orthonormal bases, and matrix decompositions tell us about hidden space. Nonlinearity can return once we understand these tools in the simplest case.

We will train the network, freeze its weights, and investigate where those properties went. Every claim about the representation will have something we can calculate or change to check it.

Two pixels, two properties

A grayscale pixel is a number between 0 and 1. Our image is two such numbers, arranged left to right:

The average brightness is $b=(x_L+x_R)/2$. The signed contrast is $c=(x_L-x_R)/2$: positive means the left pixel is lighter; negative means the right pixel is lighter. This definition of contrast is specific to our experiment.

We sample $b$ uniformly from $[0.25,0.75]$ and $c$ independently from $[-0.25,0.25]$. These ranges keep both pixels inside $[0,1]$ without clipping, which would otherwise change the relationship between the properties and the pixels.

The training target is the image itself. We keep the property labels for analysis, but the network receives only the two pixel values.

A network with twelve weights

The encoder $W$ makes three hidden activations. The decoder $D$ turns them back into two pixels:

There are six encoder weights and six decoder weights. We start with random weights and minimize the average squared reconstruction distance:

Both layers are linear, with no biases or nonlinear activations. Their composition is another linear map. The task only requires learning the identity map on these inputs, so this is enough. The extra hidden coordinate gives us a space we can inspect in 3D; it does not add information or compress the input.

This makes a useful first experiment because we can derive exactly what the analysis should find. Later, a nonlinear activation will give us a reason to ask which conclusions survive.

The interactive below uses weights from a real NumPy training run: 2,048 images, 10,000 gradient steps, and seed 7. On 512 new images, its per-pixel reconstruction MSE is about $8.6\times10^{-31}$. That is numerical precision for this calculation, not evidence of difficult generalization: the model has learned an almost exact linear identity.

Start with Feature directions. Keep contrast fixed and move brightness. Both pixels change together, and the white point follows a straight line through hidden space. Now keep brightness fixed and move contrast: the pixels change in opposite directions, and the hidden point follows another line. Rotate the plot to see that the grid occupies a plane.

Find a direction by controlling the input

Increasing brightness by $\Delta b$ adds the same amount to both input pixels. The hidden activation changes by:

This identifies the brightness direction $v_b$. Changing contrast identifies another direction:

We could estimate these directions from pairs of activations by dividing their difference by the property change. Here we can also read them from $W$, which gives us an exact check.

For the trained encoder, rounded to four decimals:

Every coordinate responds to both properties. None of the three neurons is exclusively a brightness neuron or a contrast neuron. Increasing brightness even makes all three hidden activations more negative; a hidden coordinate need not resemble a pixel intensity.

Linearity gives us the full decomposition:

The properties are coefficients of directions in the representation. The hidden activations are the coordinates of the resulting vector along the neuron axes. Those are two different coordinate descriptions of the same point.

This is the writing operation from Feature detectors: how neurons read and write: each scalar scales a direction, and the contributions add. That article develops the underlying dot products and vector sums; here the directions come from a trained encoder, and we can test what they represent.

Three coordinates, two occupied dimensions

The ambient hidden space is $\mathbb R^3$. The two independent feature directions span a plane:

Every activation lies in this plane. The bounded collection of image activations is a patch within it; that patch is not itself a vector space, since arbitrary scaling and addition can leave it.

This is where rank becomes a question about representations. An activation matrix can have three columns, one per neuron, yet only two independent directions of variation. Two independent properties have been distributed across three coordinates. We have not packed more independent features than dimensions, so this example does not demonstrate superposition.

A projection is not automatically a property measurement

The cosine between our feature directions is about $-0.155$. They are independent, but not perpendicular.

Suppose we try to read brightness by projecting $h$ onto $v_b$ and taking the coefficient:

The extra term is a contribution from contrast. For this model, $\tilde b\approx b-0.162c$. With $b=0.5$ and $c=0.2$, it reports about $0.4676$.

Open Read brightness and contrast from h in the widget. Hold brightness fixed and vary contrast: the separate projection estimate changes even though brightness does not.

This also clarifies what nonorthogonal directions do and do not imply. Adding $\alpha v_b$ to an activation changes its brightness coefficient by $\alpha$ while leaving its contrast coefficient alone. But it can change the dot product with $v_c$. A feature coefficient and a dot product with its direction are different quantities when the basis is not orthogonal.

Gram–Schmidt gives the plane an orthonormal basis

Start by normalizing the brightness direction:

Then remove the part of the contrast direction parallel to $q_1$, and normalize what remains:

Now $q_1^\top q_2=0$ and both vectors have length 1. They span exactly the same plane as $v_b$ and $v_c$.

Select Gram–Schmidt in the widget. The data points stay where they were; only our reference arrows change. Orthogonalizing the basis does not retrain the network or change its activations.

The new coordinates are $z_1=q_1^\top h$ and $z_2=q_2^\top h$. Because $q_2$ is perpendicular to $v_b$, $z_2$ has no brightness contribution. But $z_1$ can still contain both properties. Reversing the order of Gram–Schmidt would preserve the contrast direction first and produce a different basis.

So an orthonormal basis supplies convenient coordinates. Its construction does not, by itself, attach semantic labels to those coordinates.

QR turns those coordinates back into brightness and contrast

Put the original directions into a matrix $V=[v_b\;v_c]$ and the orthonormal directions into $Q=[q_1\;q_2]$. The coefficients used by Gram–Schmidt form an upper-triangular matrix $R$:

Since $h=V[b,c]^\top$ and $Q^\top Q=I$, projecting into the orthonormal basis gives:

Read the second row first, then the first:

This is back-substitution. We recovered the property coefficients by accounting for the original directions’ overlap and lengths. The widget performs these calculations with the full-precision weights.

If an activation had a component outside the plane, this same procedure would recover the coefficients of its orthogonal projection into the plane: the least-squares solution to $Vs\approx h$. All our original image activations lie in the plane, so their representation residual is zero up to numerical error.

Fit a readout without supplying the directions

We can also ask whether brightness and contrast are linearly recoverable by fitting a separate model. Collect the frozen network’s activations into $H\in\mathbb R^{N\times3}$ and the known properties into $Y\in\mathbb R^{N\times2}$. Find:

This is a linear regression probe. It leaves the original network unchanged and learns how to read information from its activations. Alain and Bengio’s work on linear classifier probes uses the same separation between a frozen representation and an independently trained diagnostic readout, with classification targets.

H = X_train @ W.T
probe = np.linalg.lstsq(H, properties_train, rcond=None)[0]
predicted_properties = (X_test @ W.T) @ probe

$H$ has rank two, so there are multiple three-coordinate probe weights that agree on every image activation. lstsq handles that rank deficiency and returns a minimum-norm solution. A naive full-rank triangular solve on all three columns of $H$ would be inappropriate. The earlier QR calculation used the two independent columns of $V$ instead.

For a brightness readout vector $w_b$, the conditions that matter are:

$v_b$ describes how brightness is written into hidden space. $w_b$ describes how brightness is read from it. They need not point in the same direction. In our held-out examples, the fitted probe recovers both properties to numerical precision.

PCA finds the occupied plane without property labels

Center the activation matrix, then compute its singular value decomposition:

H_centered = H - H.mean(axis=0)
U, singular_values, Vt = np.linalg.svd(H_centered, full_matrices=False)
pca_axes = Vt[:2]  # Two orthonormal directions in hidden space
scores = H_centered @ pca_axes.T

The singular values are approximately $9.957$, $8.431$, and $3.4\times10^{-15}$. The last is numerical residue: there are two independent directions of variation. Their variance shares are about 58.2% and 41.8%.

Select PCA in the widget. Its arrows start at the training activation mean, because PCA analyzes centered data. They span the same plane as our feature directions and the Gram–Schmidt basis. But PCA chooses directions by variance; it does not know which property we call brightness. Its first component need not be a brightness axis.

This gives us three ways to describe one plane: controlled feature directions, an orthonormal basis built from those directions, and variance-maximizing axes found without labels. They answer different questions about the same activations.

Change the hidden activation and check the output

A successful probe establishes recoverability. We can investigate behavior more directly by changing the activation sent to the original decoder:

Since the trained network reconstructs accurately, $Dv_b\approx[1,1]^\top$. Both decoded pixels should increase by $\alpha$. Likewise, $Dv_c\approx[1,-1]^\top$, so adding $\alpha v_c$ should change contrast while preserving brightness.

Open Intervene on the hidden activation. With the initial pixels $[0.7,0.3]$, adding $0.1v_b$ produces approximately $[0.8,0.4]$. Adding $0.1v_c$ produces approximately $[0.8,0.2]$. The input remains fixed while the pink hidden point and the decoded intervention change.

The widget shows raw numerical outputs. Pixel swatches alone are clipped to $[0,1]$ when an intervention leaves the displayable range; that clipping is not part of the network.

There is one more useful comparison: subtract the orthogonal projection onto $v_b$. The result is perpendicular to $v_b$, but decoded brightness need not be zero:

This removes dependence on the original $b$, while introducing a brightness contribution determined by $c$. To set the brightness coefficient to zero, we would subtract $b v_b$ instead. Geometric projection removal and setting a property to zero are distinct operations.

Here the forward map is simple enough to prove the intervention’s effect. That is a small mechanistic explanation: a controlled input property changes a known hidden direction, and the decoder maps that direction to a predictable output change.

Reproduce the experiment

Download the complete NumPy experiment. It trains the network, implements modified Gram–Schmidt, fits the probe, computes PCA, and checks reconstruction, readouts, and interventions on held-out examples. It also checks that PCA and Gram–Schmidt produce the same plane by comparing their projection matrices.

python3 -m pip install numpy
python3 two_pixel_experiment.py
python3 two_pixel_experiment.py --seed 11

The script prints its weights and measurements. Its optional --export model.json argument writes a snapshot in the format used by the widget; the checked-in snapshot is seed 7. The browser runs the frozen encoder, decoder, and analysis calculations, rather than training during page load.

A different initialization can produce a different plane and different feature angles while reconstructing the same pixels. More generally, replacing $W$ with $AW$ and $D$ with $DA^{-1}$ for any invertible $3\times3$ matrix $A$ leaves $DW$ unchanged. Angles and neuron coordinates describe this particular learned representation; reconstruction alone does not uniquely determine them.

Make the next experiment harder in one way

We now have a baseline with known properties, an exact decomposition, several readouts, and a behavioral check. That gives us specific things to revisit:

The next step I would take is the correlated-data experiment. It keeps the same twelve-weight network and asks a harder interpretability question: when a direction appears to represent brightness, how much of that conclusion came from the way we sampled the images?