One of the most useful ways to view a vector is as a combination of component vectors. This perspective helps us understand matrix transformations. Instead of asking only what a matrix does to the whole input vector, we can ask what it does to each component. Because the transformation is linear, the transformed components add up to the same output as transforming the whole vector. Some components may strongly affect the output, some may be weakened or redirected, and some may be ignored completely.

For neural networks, this perspective is especially useful because activation vectors often contain many kinds of information at once. A particular weight matrix does not necessarily use all of that information. Understanding a transformation therefore becomes partly a question of identifying which directions in the activation space matter to the matrix and which do not.

One exact way to make this distinction is through the row space and null space of a matrix. They split the input space into directions that can influence the output and directions that the matrix completely erases. This explains how two different activation vectors can produce exactly the same result under the same weight matrix.

In the previous article, we explored how weight matrices transform vectors and why neural networks need nonlinear activations. In this article, we develop the linear algebra needed to look inside those transformations more closely.

We will first look at how vectors can be built from other vectors, why the same vector can be decomposed along different directions, and how linear combinations and dot products help us reason about a transformation. We will then study the four fundamental subspaces of a matrix—its row space, null space, column space, and left null space—and use them to understand which input components determine the output and which the matrix erases. The companion experiment puts this foundation into practice by constructing and testing a projection.

This provides the foundation for future articles, including our MNIST experiment: what a trained classifier reads. There, a hidden layer produces a vector of 128 activation values for each handwritten digit image. We identify 118 independent directions that the trained output layer ignores, remove the corresponding component of the activation vector, and preserve the classifier’s digit scores up to numerical roundoff.

Building a vector from components

A matrix defines a linear transformation from an input space to an output space. Its shape specifies the number of coordinates in those spaces; its entries determine where each input goes. In a neural network, the input space is the space in which activation vectors from the previous layer live. The output space is where the linearly transformed vectors live, before any nonlinear activation is applied.

To understand a matrix’s effect on a vector, we need to look at the vector components that it can be decomposed into. Each component is itself a vector in the input space, and the original vector is their sum. By linearity, the matrix transforms each component separately, and the resulting vectors add up to the same output as transforming the original vector directly. Some components may be stretched, shortened, or sent in a different direction, while others may map to zero and disappear entirely.

A vector has many possible decompositions, using coordinate axes or directions that cross several axes. In mechanistic interpretability and representation analysis, techniques such as PCA/SVD, linear probes, and sparse autoencoders help identify useful directions from data, revealing structure distributed across multiple entries of the activation vector.

Let’s now take a look at how to build a vector from components. Each option in the widget below builds the same blue vector $h=(5,3,2)$ using a linear combination of different component vectors: each colored arrow is a chosen vector scaled by a coefficient, and these component vectors add up to $h$. The first option uses the three coordinate basis vectors $e_1=(1,0,0)$, $e_2=(0,1,0)$, and $e_3=(0,0,1)$. Each points along its corresponding coordinate axis and is scaled by the corresponding coordinate of $h$.

Coordinate basis Other basis Two vectors

(5, 3, 2) = 5(1, 0, 0) + 3(0, 1, 0) + 2(0, 0, 1)

Scale e₁

5

Scale e₂

3

Scale e₃

2

Let’s look more closely at the widget’s Coordinate basis option to distinguish a basis vector, its coefficient, and the resulting vector component. The basis vector $e_1$ is a unit vector representing a direction: it has length one. The scalar $5$ tells us how much to scale it, and $5e_1=(5,0,0)$ is the actual vector component. The three vector components add up to our input:

$$h=5e_1+3e_2+2e_3=(5,0,0)+(0,3,0)+(0,0,2)=(5,3,2).$$

Coordinates such as $5$ are often called scalar components. In this article, vector component means the whole scaled vector, such as $(5,0,0)$.

In a neural network layer, each neuron produces one of these scalar coordinates. The widget’s coefficients $5$, $3$, and $2$ could therefore be the outputs of three neurons for a single input. Each activation scales its corresponding coordinate basis vector, giving a component vector along that axis. Together, these component vectors form the layer’s activation vector $h=(5,3,2)$, as shown below:

For example, neuron 2 outputs the scalar $3$. We represent its contribution to the layer vector as $3e_2=(0,3,0)$. Multiplying by $e_2$ and adding the component vectors is our mathematical description of assembling the coordinates; it does not require an extra layer. The same $h$ is this layer’s output and the next layer’s input.

The next layer’s weight matrix $W$ transforms these component vectors, and the results add up to a new vector: $Wh=5We_1+3We_2+2We_3$. This is the linear part of the next layer’s computation, before adding any bias and applying its activation function.

The neuron outputs provide one way to decompose the layer’s activation vector. Much of representation analysis and mechanistic interpretability seeks a more informative decomposition—directions or subspaces that help explain what the network represents and how it uses that information. Infinitely many decompositions are mathematically possible, so we need criteria and experiments to identify useful ones.

Separating direction from magnitude

Separating direction from magnitude helps us reason about subspaces and transformations. Vectors such as $(1,1,1)$ and $(2,2,2)$ span the same line despite having different lengths. Representing their direction with a unit vector lets dot products measure signed projection lengths directly, and lets us examine how a matrix transforms that direction separately from how much of it is present in the input.

To make this separation explicit, we write a vector component as $\alpha d$: $d$ is a unit vector representing the chosen direction, and $\alpha$ is a signed scalar specifying how much of it to include. In our example, $d=e_1$ and $\alpha=5$. Since length is never negative, we use the absolute value $|\alpha|$ to express the component’s length. A negative sign in $\alpha$ means the component points in the opposite direction to $d$.

How do we express a component as a scalar times a unit vector when we know only its coordinates? For any nonzero vector $v$, calculate its length $\|v\|$ and divide every coordinate by that length. This is normalization:

$$d=\frac{v}{\|v\|}, \qquad \alpha=\|v\|, \qquad v=\alpha d.$$

Choosing $d$ to point in the same direction as $v$ makes $\alpha$ positive. If we instead fix a unit direction first, a component pointing opposite to it has a negative coefficient.

We can demonstrate normalization using a vector already in the widget. One of the vectors in the Other basis set is $v=(1,1,1)$, which points across all three coordinate axes. Initially, its coefficient is $2$, so the green component is $2v=(2,2,2)$. These vectors have different lengths but the same direction.

To represent their shared direction with a unit vector, we could use L2 normalization. Normalization scales the vector to a total length of $1$ while preserving its direction. First, we calculate $v$‘s length by squaring its coordinates, adding them, and taking the square root. Then we divide every coordinate by that length. Scaling all coordinates by the same positive amount preserves the direction:

$$\|v\|=\sqrt{1^2+1^2+1^2}=\sqrt{3}, \qquad d=\frac{v}{\|v\|}=\frac{1}{\sqrt{3}}(1,1,1).$$

The resulting vector has length $\|d\|=\sqrt{\tfrac13+\tfrac13+\tfrac13}=1$. We can normalize any nonzero vector this way; the zero vector has no direction and cannot be divided by its length.

Multiplying by the original length reconstructs the basis vector: $v=\sqrt{3}\,d=(1,1,1)$. The widget’s green component can therefore be written as:

$$2v=2\sqrt{3}\,d =2\sqrt{3}\left(\frac{1}{\sqrt{3}},\frac{1}{\sqrt{3}},\frac{1}{\sqrt{3}}\right) =(2,2,2).$$

The coefficient is $2$ when we scale $v$, but $2\sqrt{3}$ when we scale the unit vector $d$. The component itself stays unchanged. Moving the first slider scales that component along the same direction; a negative value reverses it.

From component vectors to subspaces

So far, we’ve chosen coefficients to build a single vector. Keeping the vectors we’re combining fixed, we can vary those coefficients over all real numbers to build an entire set of vectors. This set is their span, and it forms a subspace.

In the widget below, the fixed vectors $(4,1,1)$ and $(1,2,1)$ span a plane. The green and pink arrows show their scaled components, and the blue vector is their sum. As you change the coefficients, the blue vector moves within the plane, showing different members of the same subspace. It adds no independent direction because it is built entirely from the two fixed vectors.

Show vector outside the planeReset

1(4, 1, 1) + 1(1, 2, 1) = (5, 3, 2)

Coefficient a

1

Coefficient b

1

Try changing the two coefficients, labeled $a$ and $b$, in the widget above. The blue vector moves within the shaded plane. Vectors outside this plane cannot be reached by combining our two fixed vectors, no matter which coefficients we choose. Select Show vector outside the plane to reveal one such example: the orange marker $(0,0,3)$.

The shaded patch shows part of the span of our two fixed vectors: the infinite plane containing all their possible linear combinations. The widget’s integer sliders show only some of these combinations; allowing $a$ and $b$ to take any real values reaches every vector in the plane. Each result has the form:

$$a(4,1,1)+b(1,2,1)=(4a+b,\;a+2b,\;a+b)$$

More generally, for fixed vectors $v_1$ and $v_2$, we write:

$$S=\operatorname{span}\{v_1,v_2\} =\{av_1+bv_2:a,b\in\mathbb R\}$$

This entire span is a subspace of $\mathbb R^3$, the surrounding three-dimensional vector space. One decomposition chooses particular coefficients and produces one vector; the subspace contains every vector our fixed spanning vectors can build. It contains the zero vector, and scaling or adding vectors within it always gives another vector in the same set. In our example, the whole plane is the subspace, and each blue vector is one member of it.

Our two independent vectors span a plane through the origin. The vector $h=(5,3,2)$ lies in that plane because we can build it with $a=b=1$. But $(0,0,3)$ is unreachable: making the first two coordinates zero forces $a=b=0$, which also makes the third zero. To reach every vector in 3D, we would need a third vector outside the plane. The earlier widget’s Coordinate basis and Other basis each provide three independent vectors, so their spans fill all of $\mathbb R^3$.

The same vector can belong to different subspaces. For example, $(1,0,0)$ can be built as $(1,1,0)+(0,-1,0)$, whose components span the $xy$-plane, or as $(1,0,1)+(0,0,-1)$, whose components span the $xz$-plane. Infinitely many planes through the origin contain this same vector.

Reset view

xy-plane · z = 0xz-plane · y = 0Shared line: the x-axis

So a vector doesn’t determine a unique larger subspace. Once we fix the spanning vectors, however, their span is uniquely determined. In the widget above, both planes contain the $x$-axis. The blue vector and its endpoint $(1,0,0)$ lie in both planes. Each shaded patch is part of an infinite plane.

Subspaces give us a way to study structure in neural network representations. A subspace can be associated with a feature or a group of related features: its spanning vectors provide directions along which the representation can vary, and changing their coefficients produces components that may encode different feature values. However, the vectors that span a subspace do not necessarily represent separate features, such as color or shape. We need experiments to find out what information, if any, they represent.

Researchers have tested such interpretations experimentally. An Othello study identified directions encoding board state and changed the model’s behavior by modifying activations along them. Another study found days and months arranged approximately in circles within two-dimensional activation subspaces; interventions changed predictions on day and month arithmetic tasks. The plane is the subspace, while positions around the circle represent feature values. These experiments provide evidence that the models use the identified representations, without implying that every point in a subspace has a meaningful interpretation.

In the examples above, we saw how two vectors span a plane and how a vector outside that plane remains unreachable, regardless of the coefficients we choose. Now, the interesting question is what determines the geometry of that span: why a plane rather than a line or all of 3D space, and what determines which plane we get? To answer that, we need to distinguish the number of coordinates from the number of independent directions.

Three coordinates do not require three independent directions

If we build a vector in $\mathbb R^3$ by adding component vectors, each component also has three coordinates. Having three coordinates tells us that the vectors live in three-dimensional space; it does not tell us the dimension of the subspace they span. The dimension of the span depends on how many of the chosen vectors are linearly independent.

Consider two independent vectors, $v_1=(1,0,0)$ and $v_2=(0,1,0)$. Their combinations have the form $(a,b,0)$. Since the third coordinate is always zero, these vectors fill the $xy$-plane. They can build $(3,5,0)$, but not $(3,5,2)$. Now replace the second vector with $(0,1,2)$:

$$a(1,0,0)+b(0,1,2)=(a,b,2b)$$

The span is still a plane, now tilted so that $z=2y$. For example, $3(1,0,0)+5(0,1,2)=(3,5,10)$ lies in it. A two-dimensional subspace in three-dimensional space does not require one coordinate to be zero. A vector can have three nonzero coordinates and still lie in this plane: the third coordinate is tied to the second, leaving only two independent choices.

Compare the flat and tilted planes below. Changing the coefficients moves the blue vectors within their planes; neither construction can reach its marked red target.

Reset

Flat plane · 2D

v₁ = (1, 0, 0)v₂ = (0, 1, 0)

3(1, 0, 0) + 5(0, 1, 0)
= (3, 5, 0)

Tilted plane · 2D

v₁ = (1, 0, 0)v₂ = (0, 1, 2)

3(1, 0, 0) + 5(0, 1, 2)
= (3, 5, 10)

a · scale v₁

3

b · scale v₂

5

Adding a third vector outside that plane, such as $(0,0,1)$, lets us build any vector in $\mathbb{R}^3$ by choosing suitable coefficients:

$$a(1,0,0)+b(0,1,2)+c(0,0,1)=(a,b,2b+c)$$

We can now build any target vector $(x,y,z)$. Our combination produces $(a,b,2b+c)$, so we choose $a=x$ and $b=y$ to match the first two coordinates, then $c=z-2y$ to match the third. For example, to build $(3,5,2)$, choose $a=3$, $b=5$, and $c=-8$. A third vector already in the plane would add no independent direction.

The next widget shows this expansion starting with the flat plane: adding $(0,0,1)$ gives the right-hand construction a third independent direction. With three freely chosen coefficients, this construction can reach any vector in $\mathbb R^3$. The initial values $a=3$, $b=5$, and $c=2$ reach the target $(3,5,2)$. Move $c$ to change its third coordinate without changing the first two.

Reset

Flat plane · 2D

v₁ = (1, 0, 0)v₂ = (0, 1, 0)

3(1, 0, 0) + 5(0, 1, 0)
= (3, 5, 0)

Full space · 3D

v₁ = (1, 0, 0)v₂ = (0, 1, 0)v₃ = (0, 0, 1)

3(1, 0, 0) + 5(0, 1, 0) + 2(0, 0, 1)
= (3, 5, 2)

a · scale v₁

3

b · scale v₂

5

c · scale v₃ (right plot)

2

From the perspective of representation analysis, a shared subspace, in this case a plane, can tell us something about the collection of representations. To connect this geometry to a neural network, consider what happens when it processes several different inputs. We can collect activation vectors from the same layer into an activation matrix, with one input per row and one neuron per column. Analyzing this matrix can reveal whether the collected activations lie in a shared lower-dimensional subspace.

Suppose three images produce the activation vectors $(1,1,0)$, $(2,0,2)$, and $(3,1,2)$. All three can be built from the same two fixed vectors, $v_1=(1,1,0)$ and $v_2=(1,-1,2)$:

$$\begin{aligned} (1,1,0)&=v_1,\\ (2,0,2)&=v_1+v_2,\\ (3,1,2)&=2v_1+v_2. \end{aligned}$$

All three activations therefore lie in the same plane, despite each having three coordinates. The spanning vectors stay fixed; only their coefficients change from image to image.

When we pass different images through the same layer, each image produces its own activation vector. To check whether these vectors share a plane, we ask whether the same two spanning vectors can build every one of them. Finding a plane that contains one image’s activation vector does not tell us whether the activation vectors from other images also lie in it.

Next, let’s see what happens when we organize our construction vectors as matrix columns.

Linear combinations: building the output from columns

Imagine two sets of possible vectors. We call them the input space and the output space. The matrix $W$ defines a mapping function between them: it takes an input vector $x$ and produces an output vector $Wx$. This mapping is a linear transformation.

If we look at how $W$ builds the output, we can see how each column contributes a component vector. Each column lies in the output space and is scaled by the corresponding coordinate of $x$—the output of one neuron in the previous layer. We saw earlier how neuron activations scale coordinate basis vectors to form a layer’s activation vector. When passed to the next layer, those same activations scale the columns of $W$, which serve as the vectors we combine to build the output. These scaled columns add up to $Wx$, making the output a linear combination of the matrix’s columns.

Writing the columns as $w_1,\ldots,w_n$ and the input coordinates as $x_1,\ldots,x_n$, we can express this construction as:

$$Wx=x_1w_1+x_2w_2+\cdots+x_nw_n.$$

In a neural network, $W$ is the layer’s weight matrix, and $x$ is the activation vector supplied by the previous layer. Each coordinate $x_i$ scales the corresponding column $w_i$, producing a component vector $x_iw_i$ of the output.

The widget below follows this construction one column at a time. Each coordinate of the input $x=(5,3,2)$ scales its corresponding column to produce a component vector. Select Next column to follow how each input coordinate scales its column. The matching arrow highlights this step, while all three contributions add up to the transformed output $Wx=(19,8,2)$.

Column 1Column 2Column 3Next column →

5 × (2, 1, 0) = (5 × 2, 5 × 1, 5 × 0) = (10, 5, 0)

These components let us track how individual input coordinates contribute to the result. To study a feature distributed across several coordinates, we may instead decompose the input along directions associated with that feature and examine how $W$ transforms those components.

Column space: which outputs can the matrix reach?

The matrix does not necessarily reach every vector in the output space; it may map all input vectors into a smaller subspace, such as a plane within three-dimensional space. Earlier, we saw how a fixed set of component vectors can build only vectors lying within their span, which forms a subspace. For a matrix, that subspace of reachable outputs is called its column space.

The name comes from the fact that the matrix’s columns are the fixed vectors spanning the subspace of possible outputs.

In the earlier widget, we built the same vector $h=(5,3,2)$ using three different sets of component vectors. Let’s now view each construction as a matrix transformation: the vectors we chose become the columns of $W$, their coefficients become the input vector $x$, and $h$ is the output $Wx$.

All three constructions produce $h$ with the chosen coefficients, but they cannot all produce the same range of other outputs. By keeping each matrix fixed and varying its input, we can see which vectors belong to its column space.

First, let’s start with the vector under the Coordinate basis option. We built $h$ as $5(1,0,0)+3(0,1,0)+2(0,0,1)$. The three coordinate vectors become the columns of $W$, and the scaling coefficients $5,3,2$ become the entries of $x$:

$$W=\begin{pmatrix}1&0&0\\0&1&0\\0&0&1\end{pmatrix}, \qquad x=\begin{pmatrix}5\\3\\2\end{pmatrix}, \qquad Wx=\begin{pmatrix}5\\3\\2\end{pmatrix}$$

Since $W$ is the identity matrix, it leaves the input unchanged: $Wx=x=(5,3,2)$.

The input vector $x$ has three coordinates, so the input space is $\mathbb R^3$. Each coordinate scales one column of $W$. These three columns span all of $\mathbb R^3$, so this matrix can reach every vector in the output space.

Let’s now look at the Other basis option. We build the same vector as $h=2(1,1,1)+1(1,-1,1)+1(2,2,-1)=(5,3,2)$. The columns are now $(1,1,1)$, $(1,-1,1)$, and $(2,2,-1)$, with input coefficients $2,1,1$:

$$W=\begin{pmatrix}1&1&2\\1&-1&2\\1&1&-1\end{pmatrix}, \qquad x=\begin{pmatrix}2\\1\\1\end{pmatrix}, \qquad Wx=\begin{pmatrix}5\\3\\2\end{pmatrix}$$

The input space and output space are both still $\mathbb R^3$. These three columns are also independent, so their linear combinations can produce every output vector. We changed the construction vectors and input coefficients, but can still reach all of $\mathbb R^3$.

Something interesting emerges when we try the Two vectors option. The same vector $h$ is now built using only two construction vectors: $h=1(4,1,1)+1(1,2,1)=(5,3,2)$. This means that our transformation matrix has only two columns. Its input contains the two coefficients $1,1$:

$$W=\begin{pmatrix}4&1\\1&2\\1&1\end{pmatrix}, \qquad x=\begin{pmatrix}1\\1\end{pmatrix}, \qquad Wx=\begin{pmatrix}5\\3\\2\end{pmatrix}$$

The input space is now $\mathbb R^2$, because we supply only two coefficients. The output space remains $\mathbb R^3$ because every output still has three coordinates: $(4a+b,\;a+2b,\;a+b)$. However, there are only two independent choices, $a$ and $b$, even though the result has three coordinates. The matrix’s shape limits the reachable subspace within the output space to at most two dimensions.

The two construction vectors we chose, and therefore the corresponding matrix columns, are independent, so they span a plane through the origin, while their values determine which plane it is. In the previous examples, the columns spanned the entire three-dimensional output space. In this case, with just two vectors, we can still reach the same vector $h=(5,3,2)$, but only because it lies in the plane they span—the matrix’s column space. Vectors outside that plane remain unreachable.

The table below compares the three constructions: which input vectors each matrix accepts and which output vectors it can produce. All three have the same three-dimensional output space, but only some of the matrices can produce every vector in that space.

Widget option	Input space: coefficients $x$	Column space: reachable outputs
Coordinate basis	$\mathbb R^3$	All of $\mathbb R^3$
Other basis	$\mathbb R^3$	All of $\mathbb R^3$
Two vectors	$\mathbb R^2$	A plane inside $\mathbb R^3$

We saw how having only two columns limits the reachable outputs to at most a plane. Interestingly, having three columns does not guarantee that a matrix can reach the entire three-dimensional output space. The columns must also be linearly independent.

In the following matrix, the first two columns are identical, so the second column adds no new direction. The columns $c_1$ and $c_3$ are independent and form a basis for the plane of reachable outputs:

$$W=\begin{pmatrix}1&1&0\\0&0&1\\1&1&1\end{pmatrix}, \qquad c_1=\begin{pmatrix}1\\0\\1\end{pmatrix},\quad c_2=\begin{pmatrix}1\\0\\1\end{pmatrix},\quad c_3=\begin{pmatrix}0\\1\\1\end{pmatrix}$$

We can see mathematically why these three columns produce outputs in only two independent directions. Let’s write the input as $h$ and the output as $z=Wh$. Since $c_2=c_1$, the first two input coordinates scale the same column vector:

$$z=Wh=h_1c_1+h_2c_2+h_3c_3 =(h_1+h_2)(1,0,1)+h_3(0,1,1)$$

If we let $s=h_1+h_2$ and $t=h_3$, every possible output has the form:

$$s(1,0,1)+t(0,1,1)=(s,t,s+t)$$

We can see that every output is determined by two values, $s=h_1+h_2$ and $t=h_3$, with the third coordinate always equal to the sum of the first two. All vectors with this relationship form a plane, which is the matrix’s column space. The dimension depends on the number of independent directions, not the number of columns.

Dot products: building the output one row at a time

In the previous section, we saw how a matrix’s columns build its output from scaled vectors. Let’s now look at the same transformation through the matrix’s rows, and examine how the input’s component vectors contribute to each output coordinate. As explained in how neurons work together to transform a vector, each row of the weight matrix takes a dot product with the whole input vector, producing one output coordinate.

The widget below follows the calculation one row at a time. Select each row to see its weights take a dot product with the whole input $h=(5,3,2)$, producing one coordinate of the output $Wh=(19,8,2)$.

Row 1Row 2Row 3Next row →

z₁ = 2 × 5 + 1 × 3 + 3 × 2 = 19

Let’s focus on the first row. Its dot product with the whole input produces the scalar $19$:

$$(2,1,3)\cdot(5,3,2)=10+3+6=19.$$

Linearity lets us express this dot product as the sum of contributions from the input’s component vectors. To see where $10$, $3$, and $6$ come from, split the input into its coordinate components:

$$h=c_1+c_2+c_3, \qquad c_1=(5,0,0),\quad c_2=(0,3,0),\quad c_3=(0,0,2).$$

For the first row, $r=(2,1,3)$, linearity gives:

$$r\cdot h=r\cdot(c_1+c_2+c_3)=r\cdot c_1+r\cdot c_2+r\cdot c_3.$$

Each component supplies one scalar contribution to this output coordinate:

$$\begin{aligned} r\cdot c_1&=(2,1,3)\cdot(5,0,0)=10,\\ r\cdot c_2&=(2,1,3)\cdot(0,3,0)=3,\\ r\cdot c_3&=(2,1,3)\cdot(0,0,2)=6. \end{aligned}$$

Adding them gives $r\cdot h=10+3+6=19$, exactly the same result as computing $(2,1,3)\cdot(5,3,2)$ directly.

Repeating this calculation for every row gives one scalar result per row. Collecting those results produces the output vector $Wh=(19,8,2)$. For each row, taking the dot product with the whole input gives the same result as adding the dot products with its components.

This works for any decomposition of the input: if $h=v_1+v_2$, then $r\cdot h=r\cdot v_1+r\cdot v_2$, even when the components point in other directions.

Geometrically, we can project each component onto the same line—the direction of the row vector $r$—and add those signed projections. Their sum equals the projection of the whole input. A component pointing partly along $r$ contributes positively, one pointing partly against it contributes negatively, and a perpendicular component contributes zero.

To see these projections clearly, the next widget uses a two-dimensional example with the unit row $r=(1,0)$ pointing along the horizontal axis. The green vector $c_1$ and pink vector $c_2$ are the two component vectors. The blue vector $h=c_1+c_2$ is the combined vector they build.

The dashed lines connect each vector’s endpoint to its projection on the horizontal axis. The matching colored arrows along that axis show the projected vectors: green for $c_1$, pink for $c_2$, and blue for their sum $h$. These projection arrows can overlap. A projection pointing right contributes positively, one pointing left contributes negatively, and a zero projection contributes nothing. Because $r$ has length one, each signed projection length equals the corresponding dot product with $r$.

Drag either component’s endpoint or change its coordinates to see both the combined vector and their projections update. Try Opposite signs, Perpendicular, and Cancelling to see negative, zero, and cancelling contributions.

Both positiveOpposite signsPerpendicularCancelling

h = (3, 2) + (1, -1) = (4, 1)

c₁ = (3, 2)

xy

c₂ = (1, -1)

xy

The widget used a unit row, so each component’s signed projection length was its contribution to the output. A weight-matrix row need not have length one: its length scales every contribution. To understand how much each component contributes, we therefore need to consider both its length and its alignment with the row.

For the unit row direction $u=r/\|r\|$, the dot product $u\cdot c$ is the signed projection length of a component $c$. Using $r$ itself multiplies that length by $\|r\|$:

$$u\cdot c=\|c\|\cos\theta, \qquad r\cdot c=\|r\|\,(u\cdot c)=\|r\|\,\|c\|\cos\theta,$$

where $\theta$ is the angle between the component and the row’s direction. For our row $r=(2,1,3)$, whose length is $\sqrt{14}$:

• $c_1=(5,0,0)$ has length $5$ and alignment $\cos\theta_1=2/\sqrt{14}$, giving a contribution of $10$.
• $c_2=(0,3,0)$ has length $3$ and alignment $\cos\theta_2=1/\sqrt{14}$, giving a contribution of $3$.
• $c_3=(0,0,2)$ has length $2$ and alignment $\cos\theta_3=3/\sqrt{14}$, giving a contribution of $6$.

From that, we can see that the third component is most closely aligned with $r$: it has the largest cosine and therefore the smallest angle to the row’s direction. But the first component contributes more because it is longer. Each input component contributes according to its length and alignment with the row’s direction, and linearity lets us add these signed contributions to obtain the row’s response to the whole input vector.

Rows and columns give two ways to calculate the same transformation:

	Columns	Rows
Where the vectors live	Output space	Input space
Role in multiplication	Scale and add to build the output	Each takes a dot product to compute one output coordinate

Row space and null space: what the matrix reads and ignores

Let’s start with two vectors $(1,0,0)$ and $(0,1,0)$. They are independent, and their linear combinations have the form

$$a(1,0,0)+b(0,1,0)=(a,b,0)$$

Their span is the $xy$-plane, with all its vectors having the form $(a,b,0)$, as shown in the widget below.

Now imagine we have a blue vector $h=(3,5,2)$ and want to see whether it lies in this subspace. One way to decompose it is into the green component $(1,1,0)$ and the pink component $(2,4,2)$.

Notice how the green component lies in the plane, while the pink component and the blue vector lie outside it. The green component lies in the plane because it can be built from the two vectors $(1,0,0)$ and $(0,1,0)$ that span it.

Now suppose we use the two fixed vectors $(1,0,0)$ and $(0,1,0)$ as the rows of a matrix:

$$W=\begin{pmatrix}1&0&0\\0&1&0\end{pmatrix}$$

In this case, each row provides a vector we can use in a linear combination. Together, these vectors span a subspace of the input space known as the row space. The plane in the widget is therefore this matrix’s row space.

Let’s now try a different decomposition of the same blue vector, using the green component $(3,5,0)$ and the pink component $(0,0,2)$. Green still lies in the row-space plane, but pink now points perpendicular to it, as shown in the widget below.

Let’s treat the blue vector $h$ as an input and examine how its components contribute to the output in each decomposition. The matrix produces the same output in both cases, but the components in each decomposition make different contributions to it.

In the first decomposition, we can calculate each component’s contribution by taking its dot product with each row of $W$. Each result gives one coordinate of that component’s contribution to the output.

	Green component $(1,1,0)$	Pink component $(2,4,2)$
Dot product with row 1	$(1,0,0)\cdot(1,1,0)=1$	$(1,0,0)\cdot(2,4,2)=2$
Dot product with row 2	$(0,1,0)\cdot(1,1,0)=1$	$(0,1,0)\cdot(2,4,2)=4$
Contribution to the output	$(1,1)$	$(2,4)$

Neither component’s third coordinate contributes to the output, because both matrix rows have a zero third coordinate. By linearity, the green and pink contributions add up to the output:

$$Wh=W(1,1,0)+W(2,4,2)=(1,1)+(2,4)=(3,5)$$

In the second decomposition, green $(3,5,0)$ contributes the entire output, while pink $(0,0,2)$ contributes nothing:

	Green component $(3,5,0)$	Pink component $(0,0,2)$
Dot product with row 1	$(1,0,0)\cdot(3,5,0)=3$	$(1,0,0)\cdot(0,0,2)=0$
Dot product with row 2	$(0,1,0)\cdot(3,5,0)=5$	$(0,1,0)\cdot(0,0,2)=0$
Contribution to the output	$(3,5)$	$(0,0)$

Just as in the first decomposition, neither component’s third coordinate contributes to the output, because both matrix rows have a zero third coordinate. By linearity, the green and pink contributions add up to the output:

$$Wh=W(3,5,0)+W(0,0,2)=(3,5)+(0,0)=(3,5)$$

Notice the difference between the two decompositions: the first pink component still contributes to the output, while the second contributes nothing. This happens because the second pink component is perpendicular to both rows, and consequently to the entire row space. We can see this in the widgets and in the dot-product calculations in the tables. In the first case, the pink component has dot products of $2$ and $4$ with the two rows, whereas in the second case, both dot products are zero. This shows that lying outside the row space does not mean a vector is completely ignored; being perpendicular to every row does.

Null space: the inputs that contribute nothing

We’ve seen that the second pink component, $(0,0,2)$, contributes nothing because it is perpendicular to every row. The same is true of every vector $(0,0,z)$ along the $z$-axis: changing its length or reversing its direction still leaves both dot products at zero.

The set of all input vectors a matrix maps to zero is called its null space. For the matrix we used in the previous sections, that set is the entire $z$-axis. The first pink component has a part in the null space; the second lies entirely in it.

The input’s null-space component is $(0,0,2)$ in both cases. The second decomposition isolates it explicitly, putting the entire row-space component in green. An input has many possible decompositions, but its split into a row-space component and a null-space component is unique.

So we can take any matrix and an input it accepts, and work out which part of that input the matrix ignores and which part affects the output. There is only one way to split the input so that one part lies entirely in the row space and the other entirely in the null space.

For our matrix and input, this is exactly the second decomposition shown in our widgets and tables:

$$(3,5,2)=\underbrace{(3,5,0)}_{\text{row-space component}}+\underbrace{(0,0,2)}_{\text{null-space component}}$$

The alternative decomposition $(1,1,0)+(2,4,2)$ builds the same input, but its second component is not entirely in the null space. The calculation is straightforward for this example; for a general matrix, methods such as SVD let us compute the split systematically.

A vector maps to zero when it is perpendicular to every row—and therefore to every vector in the row space. The null space is the perpendicular complement of the row space, which we write as:

$$\operatorname{Null}(W)=\operatorname{Row}(W)^\perp$$

The symbol $\perp$ denotes all vectors that are perpendicular to every vector in this space. Splitting an input into these two parts gives:

$$h=h_{\mathrm{row}}+h_{\mathrm{null}}, \qquad Wh=Wh_{\mathrm{row}}$$

So $Wh=Wh_{\mathrm{row}}$ shows that only the row-space component affects the output. The matrix can still change its length and direction, or map it into an output space with a different number of coordinates.

Left null space: perpendicular to every possible output

Suppose a layer’s weight matrix has a third row equal to the sum of its first two rows. This leads to a constraint on the output: the third coordinate always equals the sum of the first two. No matter what input we give it, it cannot produce $(1,2,4)$, because the first two coordinates would require the third to be $3$. The left null space helps us identify constraints like this on a layer’s output.

As we saw earlier, the column space contains all possible outputs. The left null space describes the directions perpendicular to all of them. Just as the null space is perpendicular to the row space in the input space, the left null space is perpendicular to the column space in the output space.

In our earlier column-space example, every output has the form $(s,t,s+t)$. The third coordinate always equals the sum of the first two. The vector $q=(-1,-1,1)$ captures this constraint:

$$q\cdot(s,t,s+t)=-s-t+(s+t)=0$$

It lies in the left null space because it is perpendicular to every possible output. A target such as $(0,0,1)$ breaks this constraint, so no input can make this matrix produce it. Changing the input alone cannot fix that mismatch.

This gives the left null space a practical role: it reveals constraints on what a fixed weight matrix can produce. For a linear prediction layer, it can tell us that some target outputs are unreachable even before we try to find an input that produces them. These constraints apply to the linear output $Wh$; adding a bias shifts them to $q\cdot z=q\cdot b$ for $z=Wh+b$, and a following nonlinear activation can change them further.

It’s important to stress that the left null space is not where erased inputs go; those inputs map to the origin. It describes restrictions on the output side.

Putting the four subspaces together

We can now connect the four spaces in one picture. The row space and null space split the input into perpendicular components. The matrix sends the null-space component to zero and transforms the row-space component into an output in the column space. The left null space is perpendicular to every vector in that output subspace.

The image shows how the four subspaces relate, but its flat patches are schematic: a subspace could be a line or a plane, depending on the matrix. The two widgets below make those relationships concrete. You can change the input and see how it splits into row-space and null-space components, and how each component contributes to the output.

Let’s now explore these relationships in the widget below. It uses the input vector $h=(5,3,2)$ and the matrix:

$$W=\begin{pmatrix}1&1&0\\0&0&1\\1&1&1\end{pmatrix}$$

The starting input $h=(5,3,2)$ lies outside both the row-space plane and the null-space line. It has a component in each. The three sliders control the input coordinates $h_1$, $h_2$, and $h_3$. Changing them changes the vector the matrix transforms, while the matrix itself stays fixed.

Reset

Input space · ℝ³

Row space is a plane, Null space is a line

Output space · ℝ³

Column space is a plane, Left null space is a line

Drag either plot to rotate; scroll to zoom. With a plot focused, use the arrow keys to rotate, + and − to zoom, or 0 to reset zoom.

Input h₁

5

Input h₂

3

Input h₃

2

Whole input h(5, 3, 2)→(8, 2, 10)

Row-space component hrow(4, 4, 2)→(8, 2, 10)

Null-space component hnull(1, −1, 0)→(0, 0, 0)

Here is what defines each of the four subspaces. The parameters $a$, $b$, $d$, $s$, and $t$ can be any real numbers.

The three cards show the whole input and its two components. Applying $W$ to the row-space component gives the same output as applying it to the whole input. The null-space component contributes zero, so adding or removing it leaves the output unchanged.

The image’s $h_r$ and $h_n$ correspond to the widget’s $h_{\mathrm{row}}$ and $h_{\mathrm{null}}$. Initially, the blue input splits into green and pink components:

$$\underbrace{(5,3,2)}_{h} =\underbrace{(4,4,2)}_{h_{\mathrm{row}}} +\underbrace{(1,-1,0)}_{h_{\mathrm{null}}}.$$

The arrows crossing the image represent applying $W$. In the widget, both the whole input and its green component produce $(8,2,10)$, so their output arrows overlap in the orange plane. The pink component maps to zero. The purple left-null-space line is perpendicular to the output plane; it is not where erased inputs go.

Changing the input leaves the four subspaces fixed.

Let’s now use the same input vector $h=(5,3,2)$ with this alternative matrix:

$$W=\begin{pmatrix}1&1&0\\2&2&0\\3&3&0\end{pmatrix}$$

It has the same $3\times3$ shape but rank 1: all its rows are multiples of $(1,1,0)$. Its row and column spaces are lines, and both null spaces are planes. The starting input lies outside both the row-space line and the null-space plane, with a component in each. The sliders change the input while this matrix stays fixed.

Reset

Input space · ℝ³

Row space is a line, Null space is a plane

Output space · ℝ³

Column space is a line, Left null space is a plane

Drag either plot to rotate; scroll to zoom. With a plot focused, use the arrow keys to rotate, + and − to zoom, or 0 to reset zoom.

Input h₁

5

Input h₂

3

Input h₃

2

Whole input h(5, 3, 2)→(8, 16, 24)

Row-space component hrow(4, 4, 0)→(8, 16, 24)

Null-space component hnull(1, −1, 2)→(0, 0, 0)

For this matrix, the four subspaces contain the following vectors. Again, the parameters can be any real numbers.

Here, changing $h_3$ leaves the output unchanged: the third coordinate direction lies in the null space. Only the component along the green row-space line contributes to the output; every output lies on the orange column-space line.

Matrix rank and the dimensions of the four subspaces

The last concept we need to look at is matrix rank. We’ve seen that matrices with the same shape can have different subspace dimensions. Rank tells us how many independent directions the matrix preserves and, together with its shape, determines the dimensions of all four subspaces.

For a matrix with $m$ rows and $n$ columns, the $n$ columns define an input space of $\mathbb R^n$, and the $m$ rows define an output space of $\mathbb R^m$. These are the dimensions of the full input and output spaces, determined by the matrix’s shape.

Each matrix has a property called rank $r$ that defines the shared dimension of its row and column spaces: how many independent directions the rows and columns span. These two spaces always have the same dimension, even when the full input and output spaces have different dimensions.

Equivalently, the maximum number of linearly independent rows always equals the maximum number of linearly independent columns. The rank cannot exceed either the number of rows or the number of columns, formally $r\leq\min(m,n)$.

Consider our earlier $2\times3$ matrix:

$$W=\begin{pmatrix}1&0&0\\0&1&0\end{pmatrix}$$

Its shape describes a transformation from three input coordinates to two output coordinates: $3\to2$. Its two rows are independent, so its rank is 2. The row space is the $xy$-plane inside $\mathbb R^3$, consisting of vectors $(a,b,0)$. The column space is all of $\mathbb R^2$, consisting of vectors $(a,b)$.

Hence, the row and column spaces share a dimension of two, even though the full input and output spaces have different dimensions. The row space still contains three-coordinate vectors; only two coordinates can vary independently. In the column space, those two independent coordinates fill the entire output space.

Two independent directions on each side → rank(W) = 2

Input space · ℝ³

The row space is a plane inside 3D.

Row-space dimension: 2

Null-space dimension: 3 − 2 = 1

Output space · ℝ²

The column space fills all of 2D.

Column-space dimension: 2

Left-null-space dimension: 2 − 2 = 0

The shaded planes have the same dimension but live in different ambient spaces. The labels $r_1,r_2$ denote rows in the input space $\mathbb R^3$; $c_1,c_2,c_3$ denote columns in the output space $\mathbb R^2$, with $c_3=0$ at the origin. Shading shows a portion of each infinite plane.

The table compares the full input and output spaces with the row and column spaces for this matrix:

Space	Where it lives	Its dimension
Input space	$\mathbb R^3$	3
Row space	A plane inside $\mathbb R^3$: vectors $(a,b,0)$	2
Output space	$\mathbb R^2$	2
Column space	All of $\mathbb R^2$: vectors $(a,b)$	2

This matrix keeps the first two coordinates and drops the third:

$$\underbrace{\begin{pmatrix}1&0&0\\0&1&0\end{pmatrix}}_{W} \begin{pmatrix}h_1\\h_2\\h_3\end{pmatrix} =\begin{pmatrix}h_1\\h_2\end{pmatrix}.$$

We can write this as $W(h_1,h_2,h_3)=(h_1,h_2)$, using the tuple as shorthand for a column vector. For example, both $(5,3,2)$ and $(5,3,99)$ map to $(5,3)$. The row-space component $(a,b,0)$ maps to $(a,b)$, while the null-space component $(0,0,c)$ maps to zero.

That leaves one null-space direction in the input: the third coordinate axis. On the output side, the column space already fills all of $\mathbb R^2$, so its perpendicular complement—the left null space—contains only the zero vector and has dimension zero.

The same dimension accounting works for any $m\times n$ matrix of rank $r$: the row and column spaces each have dimension $r$, the null space has dimension $n-r$, and the left null space has dimension $m-r$.

Shape sets the limit; the entries determine the rank. A different $2\times3$ matrix could have rank 1. Its full input and output spaces would still be $\mathbb R^3$ and $\mathbb R^2$, but its row and column spaces would both be lines. Its null space would then have dimension $3-1=2$, and its left null space would have dimension $2-1=1$.