Suppose several components of a network contribute to the same vector. One component changes it, another adds its own contribution, and a later component uses the result. There is only one list of numbers. How can the components do different things without every change affecting every reader?

The answer depends on the directions each component can write along and the directions each reader responds to. Two components can change every coordinate of the same vector while having quite different effects on a later computation.

We will make this visible with a shared three-dimensional space, two writers, and one reader. The actors are small matrix operations with hand-chosen weights. Their simplicity lets us inspect the geometry directly before connecting it to neural networks.

The linear algebra article explains row space, column space, and null space. Feature detectors: how neurons read and write builds an individual read-and-write operation. Here we bring several actors into the same space and ask what their interactions allow.

One shared vector, several actors

Let the shared vector be $h\in\mathbb R^3$. It starts at zero. Actor A supplies a two-number result $z_A$, and actor B supplies another two-number result $z_B$. Each actor has a matrix that turns its result into a contribution to the shared space:

$$\Delta h_A=W_Az_A,\qquad \Delta h_B=W_Bz_B.$$

Both $W_A$ and $W_B$ have shape $3\times2$. We add the contributions, then let actor C read the result:

$$h=\Delta h_A+\Delta h_B, \qquad y_C=r^\top h.$$

Here $r$ is C’s reading direction. C returns one number. In the demonstration, sliders supply A’s and B’s results so we can vary their contributions independently. We are isolating how they communicate through the shared vector; we have not yet specified a task that computes those results.

There is no A field and B field inside $h$. Once added, the contributions occupy the same three coordinates. Keeping separate arrows in a diagram is our bookkeeping, not extra storage inside the vector.

Two writers with different planes

Choose three mutually perpendicular unit vectors, $u$, $v$, and $n$. Think of $u$ and $v$ as directions along a tabletop, and $n$ as its normal. These directions need not coincide with the coordinate axes.

Give A the output matrix

$$W_A=\begin{bmatrix}u&v\end{bmatrix}.$$

For a two-number result $z_A=(a_u,a_v)^\top$, A writes

$$\Delta h_A=a_u u+a_v v.$$

Whatever numbers A produces, its contribution lies in $S_A=\operatorname{span}\{u,v\}$. This plane is A’s write subspace: the column space of its output matrix.

Now give B a different matrix of exactly the same shape:

$$W_B=\begin{bmatrix}u&n\end{bmatrix}, \qquad \Delta h_B=b_u u+b_n n.$$

B writes in $S_B=\operatorname{span}\{u,n\}$. Both writers can move along $u$, but only A can supply the $v$ component, and only B can supply the $n$ component in this configuration.

The matrix shapes limit each writer to at most two independent directions. The matrix values determine which two directions. Nothing about having two inputs automatically assigns A and B different planes.

The explorer uses these exact directions:

$$u=\frac13\begin{bmatrix}1\\2\\2\end{bmatrix},\qquad v=\frac13\begin{bmatrix}2\\1\\-2\end{bmatrix},\qquad n=\frac13\begin{bmatrix}-2\\2\\-1\end{bmatrix}.$$

Each direction has nonzero entries in all three coordinates. Writing along just one of them already changes every coordinate. Yet their dot products with one another are zero. Coordinate overlap does not imply that every reader responds to every contribution.

A reader can ignore an entire writer

Let C read along $n$. Since $n^\top u=n^\top v=0$ and $n^\top n=1$,

$$y_C=n^\top h =n^\top\big((a_u+b_u)u+a_vv+b_nn\big) =b_n.$$

A can change the shared vector, including all three coordinates, without changing C’s answer. Every contribution A can make lies in the null space of C’s reading matrix $n^\top$.

This is a statement about all possible results from A, not just one contribution that happens to have no effect. C is insensitive to A’s whole write plane.

Try it before moving on:

1. Keep the reference configuration: all four writer coefficients are $1$, B’s plane angle is $90^\circ$, and C reads along $n$. Record it with Record view.
2. Write a prediction, then change only A along v from $1$ to $2$. Record that view too. Compare the coordinates of $h$ and C’s answer.
3. Explain which dot product accounts for the result. Then restore the reference and change only B along w from $1$ to $2$. At this angle, $w=n$.

The figure shows two plane patches, the individual contributions, their sum, and C’s reading direction. View rotation changes only the camera. Reset reference restores the four coefficients, plane angle, reader, and camera; it keeps your written prediction and records.

Two write planes, one shared vector, one reader

Reset reference

h starts at zero. A and B supply contributions; C reads their sum. All values below are calculated from the current controls.

View rotation

70°

Actor A · writes aᵤu + aᵥv

A along u

1

A along v

1

Actor B · writes b_uu + b_ww

B along u

1

B along w

1

B plane angle θ

90°

w = cos(θ)v + sin(θ)n

Actor C · reads rᵀh

Read directionn · perpendicular to A's planev · within A's planeu · shared by both planes

C reads from A0rᵀΔh_A

C reads from B1rᵀΔh_B

C reads the sum1rᵀh

Δh_A = (1, 1, 0)
Δh_B = (-0.333, 1.333, 0.333)
h = (0.667, 2.333, 0.333)

C's response to A's two channels: (0, 0)
C's response to B's two channels: (0, 1)

Exact directions and matrices

u = (1, 2, 2)/3; v = (2, 1, −2)/3; n = (−2, 2, −1)/3. These are mutually perpendicular unit vectors. All three use every coordinate.

W_A = [u v]; W_B = [u w]; R_C = rᵀ. Columns determine the write planes. Displayed decimals are rounded to three places.

Predict a change before moving a control

Record viewClear recordsNotes and records last until this page is reloaded.

In the first comparison, changing $a_v$ changes $h$ by $v$, but C’s answer remains $1$. In the second, changing $b_n$ changes $h$ by $n$ and raises C’s answer to $2$. The shared vector changed in both cases; the reader distinguishes them by direction.

A shared direction allows reinforcement and cancellation

The two write planes intersect along a line:

$$S_A\cap S_B=\operatorname{span}\{u\}.$$

A’s and B’s contributions along that line combine into $(a_u+b_u)u$. At the reference settings, they reinforce: the coefficient of $u$ is $2$.

Restore the reference and select $u$ using Read direction. Predict what will happen when you change only B along u from $1$ to $-1$. Record the two views.

Now the shared contribution cancels:

$$a_u u+b_u u=u-u=0.$$

C’s readout along $u$ falls from $2$ to $0$, while the $v$ and $n$ contributions remain. If C instead reads along $n$, this cancellation has no effect on its answer.

That distinction matters: cancelling a component of the shared vector and changing a particular reader’s answer are different questions. We have to specify which reader observes the change.

The total along $u$ also carries no record of its source. From that total alone, a reader cannot distinguish $a_u=1,b_u=-1$ from $a_u=0,b_u=0$. Other information could distinguish the actors, but addition has not preserved their individual coefficients here.

Interaction does not require an exact shared direction

We can rotate B’s plane around the shared line $u$. Replace its second column with

$$w_\theta=\cos(\theta)v+\sin(\theta)n, \qquad W_B(\theta)=\begin{bmatrix}u&w_\theta\end{bmatrix}.$$

At $90^\circ$, $w_\theta=n$ and we have our original configuration. At $0^\circ$, $w_\theta=v$ and B’s plane coincides with A’s. Between those endpoints, the writers share exactly the line through $u$.

Keep C reading along $n$. Its response to B is now

$$n^\top\Delta h_B=b_w\sin(\theta).$$

Restore the reference, record it, then change only B plane angle θ to $45^\circ$. B’s two coefficients stay fixed. C reads approximately $0.707$ from B, rather than $1$.

At this angle, C’s read line $\operatorname{span}\{n\}$ and B’s write plane share no nonzero vector. To produce a vector along $n$, a combination of $u$ and $w_{45^\circ}$ would have to eliminate both its $u$ and $v$ components. That forces both coefficients to zero.

Nevertheless, C responds to B because $n^\top w_{45^\circ}\ne0$. A direction can have a nonzero projection onto a reader without lying along that reader’s line.

This gives us two precise questions to ask about subspaces:

• Intersection: which exact directions can both contain?
• Sensitivity: does a direction one actor writes have a nonzero dot product with a direction the other reads?

Exact intersection explains the shared direction where A and B can cancel each other. Sensitivity explains why C can detect B even without an intersection between its read line and B’s write plane.

A matrix product describes the possible interaction

For a reader with several outputs, replace $r^\top$ with a reading matrix $R_C$. The contribution of A to its reading is

$$\Delta y_C=R_C\Delta h_A =\underbrace{R_CW_A}_{\text{response to A's output channels}}z_A.$$

The product connects two actors through the shared space. Each entry measures how one reading row responds to one writing column.

In our original configuration, C reads along $n$, so

$$R_CW_A=\begin{bmatrix}0&0\end{bmatrix}, \qquad R_CW_B=\begin{bmatrix}0&1\end{bmatrix}.$$

C ignores both of A’s channels and responds only to B’s second channel. Switching C’s reading direction to $u$ gives

$$R_CW_A=\begin{bmatrix}1&0\end{bmatrix}, \qquad R_CW_B=\begin{bmatrix}1&0\end{bmatrix}.$$

The write planes have not changed. The reader now responds to the direction both writers share.

If $R_CW_A=0$, A’s entire write subspace lies in $\ker R_C$. A cannot affect this linear reading through its direct contribution. If the product is nonzero, some possible results from A affect C. A particular result can still have zero effect, and the computations producing A’s results may restrict which possibilities actually occur.

That is why geometry establishes a possible interaction, while understanding a working network also requires examining the activations it produces.

Design an interaction yourself

Use the explorer to build a configuration where changing B’s second coefficient changes C’s answer, but changing either of A’s coefficients does not. Choose an angle other than $90^\circ$ and write down your prediction before trying it.

Record a reference and one comparison for each changed coefficient. Keep the reader, angle, and other coefficients fixed within each comparison. Explain your results using $R_CW_A$ and $R_CW_B$.

As a separate test, make the two write planes coincide by setting the angle to $0^\circ$. Can a linear reader ignore every possible contribution from A while still responding to some contribution from B?

Check the geometric reasoning

For the first task, choose C’s direction as $n$ and any angle strictly between $0^\circ$ and $90^\circ$. Then $R_CW_A=[0\;0]$ and $R_CW_B=[0\;\sin(\theta)]$. Varying B’s second coefficient changes the answer; varying either of A’s does not.

For the second task, no such linear reader exists. With identical write subspaces, a reader that annihilates every vector A can write also annihilates every vector B can write. A reader could distinguish restricted sets of actual outputs under additional assumptions, but it cannot distinguish the actors’ full capabilities in this configuration.

Where the actors appear in a neural network

A neural-network component can read from a shared representation, compute a result, and map that result back into the shared space. Its input matrix supplies the reading rows; its output matrix supplies the writing columns. These matrices can have different orientations, so what a component reads need not be what it writes.

In our example, sliders supplied the writers’ results. In a network, earlier computations supply them. The distinction between A’s possible write plane and its actual contribution remains: the matrix specifies what is reachable, while the current result selects a vector within it. Our planes were deliberately chosen; learned matrices can be aligned, distinct, redundant, or rank deficient. Matrix shape alone guarantees none of those arrangements.

Attention heads provide a larger instance of this geometry. A head’s output projection maps its result into a subspace of the residual stream, and later components read the updated stream through their own projections.

For example, take a model with a 768-dimensional stream and 12 heads, each producing a 64-dimensional result. Each head has a write map $W_O^{(r)}\in\mathbb R^{768\times64}$. Its contribution has 768 coordinates, but lies in the column space of that matrix, whose dimension is at most 64. Mapping back to the stream’s width does not make every direction reachable.

In our column-vector notation, a head writes $\Delta x_r=W_O^{(r)}z_r$, and a later linear reader sees $R\Delta x_r=RW_O^{(r)}z_r$. This is the same writer-to-reader product we inspected in three dimensions. The per-head matrices can be stored as blocks of one combined output projection; separate boxes show their mathematical contributions.

A Mathematical Framework for Transformer Circuits develops this communication view and the products connecting reading and writing matrices. Actual attention also includes token routing and nonlinear operations, and normalization can change the effective response to an update; our fixed linear readers isolate one part of that computation.

The residual-network article explains how updates accumulate and why skip connections matter for training. The attention-head article explains how a head computes its result. The question to carry into either is geometric: where can this component write, and which later readers are sensitive to those directions?