When networks learn to share a bottleneck

In Directions, sparse features, and how sparse autoencoders find them, we chose a one-number encoding by hand: $h=x_1-x_2$. Positive and negative values distinguished two nonnegative features. A ReLU decoder reconstructed either feature alone, but both together could cancel.

That construction showed what a small representation can do. It did not show that training would discover it, or when sharing the coordinate would be useful.

Here we’ll learn the encoder weights. We will vary how often each feature appears, whether the features appear together, and whether the decoder is linear or uses ReLU. The model has only two trainable numbers, making it possible to inspect every learned solution. The experiment is inspired by Toy Models of Superposition; it is a deliberately narrow study of two features in one dimension, not a reproduction of that paper’s larger experiments.

Give training a reason to preserve both features

A predictor can discard information that does not help its target. To ask whether two features survive a bottleneck, we instead train the model to reconstruct both inputs.

Let $x_1$ and $x_2$ be nonnegative feature strengths. The encoder produces a signed scalar:

The decoder reuses the same weights, a constraint called tied weights:

Both models have two trainable parameters and no biases. ReLU is applied only to the outputs; the hidden scalar can be negative. The training loss is mean squared reconstruction error across examples and both features.

With $w=(1,-1)$, the ReLU model reconstructs every input where at most one feature is active. With $w=(1,1)/\sqrt2$, both models reconstruct $(x_1,x_2)$ as two copies of $(x_1+x_2)/2$. These are different ways to use the same scalar: preserve which isolated feature occurred, or preserve the average of two co-occurring strengths.

Training is free to choose other weights. We do not initialize it at either of these solutions.

Separate sparsity from co-occurrence

Each feature is active with marginal probability $p$. When active, its strength is sampled independently and uniformly from $[0.5,1.5]$. Inactive features are zero.

We test $p=0.05$, $0.2$, and $0.5$. Smaller $p$ means more zeros in the input features. It does not describe how often the two features overlap, so we control that separately.

Write $q$ for the probability that both are active:

These probabilities preserve the same marginal activity $p$ for both features. For every value of $p$, we train three conditions:

For independent features, $q/p=p$: the probability that the second is active given that the first is active falls as $p$ falls. For always-together features, that conditional probability is $1$, even when both features are rare.

Rare does not necessarily mean isolated. This distinction lets us test whether the benefit comes from avoiding interference, rather than merely seeing many all-zero examples.

Compare the same training protocol

For each of the nine data conditions, we generate 8,192 training examples and 32,768 independent test examples. Each decoder trains from five initializations, using seeds 7, 11, 23, 42, and 101. This gives 90 trained models.

Within a condition, both decoders receive the same training data, the same minibatch sequence, and paired initial weights. Every run uses 3,000 Adam updates with batches of 256. The learning rate decays from $0.02$ toward $0.0002$ on a cosine schedule. We keep every run, including runs that do not find the opposite-sign solution.

The training MSE is divided by the training set’s mean squared input value. This rescales the objective without changing its minimizers, keeping the overall loss scale comparable when the fraction of empty inputs changes. It does not remove sampling variability or make optimization identical across conditions.

We report relative test error:

Zero means perfect reconstruction; 100% equals always predicting zero. This is an input-energy baseline, not a mean-prediction baseline. It prevents a small raw MSE on mostly empty inputs from looking impressive by itself.

The table shows the mean and range across five initializations. The inspector also separates error on single-feature examples from error on examples containing both features.

What the weights learned

Mutual exclusion makes exact reconstruction possible

At $p=0.05$, all five ReLU runs learn opposite-sign weights with magnitudes one and reconstruct the test inputs to the displayed precision. The linear runs retain roughly half the input energy as reconstruction error.

At $p=0.2$ and $p=0.5$, three of the five ReLU runs find the exact signed solution. The other two end near a solution that preserves one feature and loses the other, giving about 50% relative error. Consequently, their mean errors are about 20%, despite an exact solution being available.

The architectural possibility is unchanged across these exclusive conditions. At $p=0.5$, every example contains exactly one active feature; at $p=0.05$, most contain none. The difference in outcomes reflects this training setup, sampled data, and initialization sensitivity. Five runs are not enough to establish a general success rate or a monotonic relationship between sparsity and optimization.

Independent features reveal the interference cost

At $p=0.05$, one ReLU run finds an opposite-sign encoding and achieves 2.9% relative test error. The other four settle on same-sign mixtures, close to the linear model’s 48.3%. The five-run ReLU mean is therefore 39.2%, not 2.9%.

At $p=0.2$, again one run finds opposite signs. Its error is 14.0%, compared with approximately 40.6% for the same-sign solutions. Select Initialization seed 42 in these independent conditions to inspect that run. Its single-feature error is small; its error is concentrated on examples where both features are present.

At $p=0.5$, all five ReLU runs instead converge to same-sign mixtures, with approximately 26.9% relative error, matching the linear runs. Overlap is now common enough that encoding the two strengths as an average performs better than our fixed $h=x_1-x_2$ construction on this distribution. This observation does not prove that the learned solution is the global optimum over every possible nonlinear decoder.

The mean alone hides this behavior. Two runs can have very different internal encodings and error patterns even when they use the same architecture and data.

Always-together features favor an average

Across all three activity levels, both decoders learn weights close to $(1,1)/\sqrt2$, or its global sign reversal. Their relative errors are approximately 3.7–3.9%.

That low error does not mean both strengths have been preserved exactly. When active strengths are $1.2$ and $0.8$, the ideal equal-weight solution reconstructs $(1,1)$. It preserves the average and loses the difference.

The strengths vary independently within $[0.5,1.5]$, but both are positive and share the same presence pattern. Their average is already a good approximation. ReLU adds no benefit for these same-sign solutions because the values entering the output ReLU are nonnegative.

The inspector’s single-feature probes reveal another limitation: those inputs never occur in the always-together training or test data. A poor answer on such a probe describes behavior outside the measured distribution; it is not included in that condition’s reported test error.

Training changes the representation

The weights determine both the hidden code and how the decoder reads it. A gradient-descent update

therefore changes the representation as well as its reconstruction error. In our one-dimensional hidden space, each feature contributes a signed amount: its encoder weight sets the contribution’s direction and scale. Opposite-sign weights place the features on opposite sides of zero; same-sign weights mix them on the same side.

There is no room for two nonzero perpendicular directions in one dimension. Yet opposite signs and output ReLU can recover mutually exclusive features exactly. When features always occur together, the measured runs instead learn an average. Training is minimizing reconstruction loss on the supplied examples, not explicitly trying to make feature directions orthogonal.

In larger hidden spaces, changing the weights can also change angles between feature directions. How feature directions share a small space shows how those angles affect a dot-product reader, then explores how dimension and sparsity limit typical interference. Its arrows are chosen by hand; the experiments here show which representations this training procedure actually finds.

Check capacity separately from training success

The two-parameter model gives us useful mathematical checks.

First, $w=(1,-1)$ with output ReLU reconstructs every mutually exclusive nonnegative input exactly. This verifies representational capacity independently of whether an optimizer finds those weights.

Second, the tied linear model computes $\hat{\mathbf x}=ww^{\mathsf T}\mathbf x$. Its best nonzero rank-one reconstruction on the training set uses a unit eigenvector corresponding to the largest eigenvalue of the uncentered second-moment matrix:

We use the uncentered moment because the model has no bias or centering step. The experiment computes this linear reference directly and saves its weights and test error alongside the learned runs. It also checks that no trained linear solution beats that reference on the training objective.

Finally, the script checks its analytical tied-weight gradients against central finite differences at several test weights away from ReLU kinks. Both uses of each weight matter: changing $w_i$ affects the hidden code and its decoder output.

These checks separate errors in our implementation from limitations of the architecture and limitations of optimization.

What this establishes about superposition

The opposite-sign solution stores two synthetic features in one dimension and decodes them separately on single-feature inputs. It is a minimal example of feature sharing beyond the coordinate count. Interference appears when both features activate, and the training results show that discovering this solution is not guaranteed.

A single coordinate carrying several feature meanings is related to polysemanticity. Here we know the meanings because we constructed the two input features. A trained language model does not supply such labels, and observing a neuron respond to several kinds of input does not by itself prove superposition.

This experiment also differs from training a sparse autoencoder on a frozen model. Here we train the representation itself with a reconstruction objective, asking whether it learns to share a small bottleneck. In the SAE experiment, the activation-generating dictionary is fixed and the autoencoder tries to infer its ingredients.

Our conclusions are limited to two nonnegative features, one signed hidden scalar, tied weights, no biases, equal feature importance, and the stated distributions. More dimensions allow different geometries; biases and richer decoders change which codes can be read; different feature frequencies and importance weights change the reconstruction tradeoff. The small model makes those distinctions visible without claiming to explain every representation learned by a large network.

Reproduce and inspect the experiment

Download the complete NumPy experiment and the saved results. The JSON includes every run’s initial and final weights, test activity counts, conditional errors, diagnostic reconstructions, and the directly calculated linear reference.

From this article’s experiment/ directory:

python3 -m venv .venv
source .venv/bin/activate
python -m pip install -r requirements.txt
python experiment.py

The checked-in experiment uses NumPy 2.2.6. To run only the gradient and data-generation checks, use python experiment.py --check-only. To write the measurements elsewhere, pass --output /path/to/results.json. There is no test-based early stopping, and no initialization is removed from the results.