A linear layer in a neural network is a matrix. A 1024 × 1024 weight matrix has just over a million parameters and looks like it should be able to do a million parameters’ worth of work. Often it can’t. The transformation it performs may live in a 50-dimensional subspace, no matter how many parameters you pour into it. The shape on disk and the expressive capacity it actually delivers are two different numbers — and the second one is called rank.

This article is about that gap. We start with what a matrix does to a point cloud in 2D, work up to the formal definitions of rank and SVD, and then look at why this one property quietly determines how big neural networks can be, how small we can compress them, and why deep transformers stay expressive at all.

The math stays light. Every formal step is preceded by a numeric or visual example, and proofs are skipped in favour of the geometric picture. By the end, the word “rank” will mean a specific picture — not a definition you memorise.

Rung 2 of the linear-algebra guided map — it assumes the dot products of cosine similarity and feeds the PCA, factorisation, and latent-bottleneck articles above it.

The hidden bottleneck

Take a 1024 × 1024 linear layer somewhere in the middle of a trained network. It has 1,048,576 parameters. Now do a singular value decomposition and look at the singular values — we’ll get to what those are shortly, but for now treat them as a sorted list of “how much work each independent direction is doing.” A common shape in practice: the first 50 or so are large, the next few hundred shrink rapidly, and the tail is essentially zero.

The layer’s 1024 → 1024 mapping, in other words, lives in a roughly 50-dimensional subspace of the output. The other ~974 dimensions are linear combinations of the first 50; they carry no new information. A 1024 × 50 matrix followed by a 50 × 1024 matrix — just over 100,000 parameters — would produce the same mapping. The original layer was paying ten times the storage and compute for the same transformation.

The number that captures this gap is the matrix’s rank (or, in this messier real-world version, its effective rank). The rest of this article is about what rank is, how to read it off a matrix, and why neural networks are full of layers whose rank is much smaller than their shape suggests.

A matrix is a transformation

Before we can talk about rank, we need a sharper picture of what a matrix actually does. Forget for a moment that matrices are grids of numbers. A matrix is a function — it takes a vector in and produces a vector out:

$$\mathbf{y} = W \mathbf{x}$$

For a 2 × 2 matrix, that function takes a 2D point and returns a 2D point. Apply it to every point on a circle and you get a new shape — usually an ellipse, but sometimes a line, and in degenerate cases a single point.

The shape of that output is what rank measures. A 2 × 2 matrix that turns a circle into a full ellipse has rank 2: both input dimensions made it through to the output. A matrix that crushes the same circle onto a line has rank 1: one dimension survived, the other was annihilated. A matrix that maps everything to the origin has rank 0.

That picture is what the rest of the article formalises. Before we get there, it’s worth seeing it move.

Demo: rank as a geometric collapse

identitystretch xrotate 30°shearrank 1 (parallel columns)rank 0 (zero matrix)

input — unit circle

output — W · (unit circle)

W = [[a, b], [c, d]]

a1.00

b0.00

c0.00

d1.00

singular values

σ₁

1.000

σ₂

1.000

rank = 2(σ > 0.0001)

Drag the sliders. Each one is one entry of a 2 × 2 matrix W. The grey circle on the left is the input — every point on the unit circle. The coloured shape on the right is what W does to it. The bars below the matrix are the two singular values of W — the lengths of the ellipse’s two axes. The rank shown beneath them is just the count of singular values that aren’t (numerically) zero.

A few things to try:

• Set W to the identity matrix. The output is the same circle. Both singular values are 1. Rank 2.
• Stretch one direction by pulling a diagonal slider. The output becomes an ellipse. Singular values become unequal. Still rank 2.
• Set the second column equal to the first. The two columns now point in the same direction — they are linearly dependent. The output ellipse collapses to a line. The smaller singular value drops to zero. Rank is now 1.
• Set every entry to zero. Output collapses to the origin. Both singular values are zero. Rank 0.

The thing the demo is teaching is that rank is not really about the numbers in the matrix; it’s about the geometric shape of the output. Rank counts how many dimensions of the input survive the trip through W. When two columns of W point in the same direction, one of them is redundant — the matrix has fewer genuinely independent directions than its shape suggests.

Linear independence and the column space

Let’s put a name to what we just saw. The columns of a matrix are vectors. For a 2 × 2 matrix

$$W = \begin{bmatrix} a & b \\ c & d \end{bmatrix}$$

the columns are $\mathbf{w}_1 = (a, c)$ and $\mathbf{w}_2 = (b, d)$. When you multiply W by an input vector $\mathbf{x} = (x_1, x_2)$, you get

$$W\mathbf{x} = x_1 \mathbf{w}_1 + x_2 \mathbf{w}_2$$

The output is a weighted combination of the columns. So the set of all possible outputs — every vector W can produce, over every possible input — is exactly the set of all weighted combinations of $\mathbf{w}_1$ and $\mathbf{w}_2$. That set is called the column space of W.

If $\mathbf{w}_1$ and $\mathbf{w}_2$ point in different directions, their combinations sweep out the entire plane — the column space is all of $\mathbb{R}^2$. If they point in the same direction (one is a scalar multiple of the other), all combinations lie on a single line — the column space is one-dimensional. If they’re both zero, the column space is just the origin.

Two vectors are linearly independent if neither is a scalar multiple of the other. More generally, a set of vectors is linearly independent if no one of them can be written as a combination of the others. The rank of a matrix is the number of linearly independent columns it has — equivalently, the dimension of its column space.

A worked example. Take

$$M = \begin{bmatrix} 1 & 2 & 3 \\ 2 & 4 & 6 \\ 3 & 6 & 9 \end{bmatrix}$$

The second column is twice the first. The third column is three times the first. Only one column is genuinely “new”; the other two carry no information that the first doesn’t already. M has rank 1 despite being a 3 × 3 matrix with 9 non-zero entries.

Rank, formally

A few facts about rank that fall out of the column-space picture, stated without proof:

• Row rank equals column rank. The number of linearly independent rows of a matrix is always equal to the number of linearly independent columns. This is not obvious from the definition; it’s a theorem. But it means “the rank” is unambiguous — you can count rows or count columns, you’ll get the same answer.
• Maximum rank is min(m, n). A matrix with shape (m, n) can have at most min(m, n) independent columns, because you only have min(m, n) directions available in the smaller of the two spaces. A 1024 × 50 matrix has rank at most 50, no matter what numbers you put in it.
• Full rank vs. rank-deficient. A matrix is full rank if its rank equals min(m, n). Otherwise it is rank-deficient. Most matrices you generate randomly are full rank. Most matrices that mean something — learned weights, real-world data — are rank-deficient in practice.

The last point is the one this article is built around. The geometry says: a layer’s effective dimensionality is whatever its rank is. And in practice, trained layers tend to have much smaller rank than their shape advertises.

Eigenvectors: directions the matrix leaves alone

A general matrix turns a vector two ways at once — it changes the vector’s length and its direction. To see this, take

$$A = \begin{bmatrix} 1 & 0 \\ 1 & -2 \end{bmatrix}$$

and feed it the vector $(1, 2)$:

$$A \begin{bmatrix} 1 \\ 2 \end{bmatrix} = \begin{bmatrix} 1\cdot 1 + 0\cdot 2 \\ 1\cdot 1 + (-2)\cdot 2 \end{bmatrix} = \begin{bmatrix} 1 \\ -3 \end{bmatrix}$$

The output $(1, -3)$ has swung to the other side of the horizontal axis and changed length — it differs from the input in both orientation and magnitude. That is the typical case: feed a square matrix almost any vector and it comes back rotated and rescaled at the same time.

But for every square matrix there are a few special directions where nothing rotational happens — the matrix only stretches or shrinks. Send a vector that lies along one of these directions through $A$ and you get back a vector pointing the same way, just longer or shorter. (It may also be flipped to point the opposite way along the same line through the origin, which still counts as the same direction.) These directions are characteristic of the matrix — which is where the name eigen comes from, German for “own” or “characteristic.” The directions themselves are the matrix’s eigenvectors, and the scalar each one is stretched by is its eigenvalue.

Formally, a non-zero vector $\mathbf{v}$ is an eigenvector of a square matrix $A$ if

$$A\mathbf{v} = \lambda \mathbf{v}$$

for some scalar $\lambda$. Multiplying by $A$ does the same thing as multiplying by the number $\lambda$ — the vector stays on its own line through the origin; only its length, and possibly its sign, changes.

The cleanest case is when those special directions line up with the coordinate axes. Take the diagonal matrix

$$A = \begin{bmatrix} 2 & 0 \\ 0 & 3 \end{bmatrix}$$

The vector $(1, 0)$ is sent to $(2, 0)$ — same direction, scaled by 2. The vector $(0, 1)$ is sent to $(0, 3)$ — same direction, scaled by 3. So $(1, 0)$ and $(0, 1)$ are eigenvectors with eigenvalues 2 and 3. Every other vector $A$ acts on is a combination of those two, and gets stretched unequally along the two axes — turning a circle into an ellipse, exactly the picture the demo earlier was showing.

Most matrices don’t line their eigenvectors up with the axes so conveniently. A neat way to see where they actually point, due to Anand Avati’s Stanford lectures, is to take every unit vector — the whole circle — and multiply each one by the matrix. The output vectors fan out into an ellipse, and the eigenvectors are the directions that didn’t swing off their own line; they fall along the ellipse’s axes.

[[1,0],[1,−2]] (non-symmetric)[[3,1],[1,2]] (symmetric)shearrotate 50° (complex)

output — each grey ray is W·(unit vector); black arrows are the eigenvectors

W = [[a, b], [c, d]]

a1.0

b0.0

c1.0

d-2.0

symmetric (b = c): no

eigenvalues: λ₁ = 1.00, λ₂ = -2.00

angle between eigenvectors: 71.6°

Two cases are worth contrasting with the buttons. The non-symmetric matrix $\left[\begin{smallmatrix} 1 & 0 \\ 1 & -2 \end{smallmatrix}\right]$ — the same one from the worked example above — has eigenvalues $1$ and $-2$ (the negative one flips its eigenvector to point the opposite way), and its two eigenvectors are not perpendicular. Now switch to the symmetric matrix $\left[\begin{smallmatrix} 3 & 1 \\ 1 & 2 \end{smallmatrix}\right]$, the kind that is mirror-equal across its main diagonal. Its eigenvectors snap to exactly $90°$ apart, lying along the major and minor axes of the ellipse. That is not a coincidence of this example: a real symmetric matrix always has real eigenvalues and orthogonal eigenvectors. (Push $b$ and $c$ apart with the sliders and watch the angle drift away from $90°$; spin up the rotation preset and the eigenvalues go complex — a pure rotation leaves no vector on its own line at all.) This orthogonality is exactly what makes the covariance matrix in PCA, and $W^\top W$ in SVD, so well-behaved — both are symmetric by construction, so their eigenvectors form a clean perpendicular frame.

That symmetry is not an accident of the example; it is built into how a covariance matrix is formed. Stack your data as an m × n matrix $X$ (m samples, n features), centre each column, and the product $X^\top X$ is symmetric by construction — its $(i, j)$ entry is the covariance between features $i$ and $j$, the same number as entry $(j, i)$. For two features — say the heights and weights of three people — it is a 2 × 2 symmetric matrix whose diagonal holds each feature’s variance and whose off-diagonal holds their covariance: large and positive when the features rise together, near zero when they don’t. Its two eigenvectors are therefore orthogonal, and they are precisely the principal components of the data — the directions of greatest variance, each tagged with how much variance (its eigenvalue) lies along it. That is the whole of PCA, and the PCA article builds it out with an interactive covariance demo.

That picture generalises. A diagonalisable n × n matrix has n linearly independent eigenvectors, and the matrix factors as

$$A = Q \Lambda Q^{-1}$$

where the columns of $Q$ are the eigenvectors and $\Lambda$ is diagonal with the eigenvalues on its diagonal. The matrix’s whole behaviour collapses to a three-step recipe: express the input in the eigenvector basis, scale each coordinate by its eigenvalue, transform back. Rank reads off $\Lambda$ directly — it is the number of non-zero eigenvalues. Zero eigenvalues are directions the matrix annihilates, the same collapse the column-space picture described.

Eigendecomposition only works for square matrices (and not all of them — some lack a full set of independent eigenvectors). Most matrices in a neural network are rectangular: a 768 × 3072 projection has no eigenvalues in the usual sense. SVD, in the next section, is the generalisation that works on any matrix. The two are tightly connected: the singular values of $W$ are the square roots of the eigenvalues of $W^\top W$, and the columns of $V$ and $U$ in the SVD are the eigenvectors of $W^\top W$ and $W W^\top$ respectively. SVD is eigendecomposition with the input promoted to a symmetric square matrix first.

The reason this is worth a section of its own is that the same handful of eigen-structures keep showing up across machine learning:

• PCA. The principal components of a dataset are the eigenvectors of its covariance matrix, sorted by eigenvalue. The largest eigenvalues point along the directions of greatest variance — exactly the few directions PCA keeps. Eigenvectors of the covariance matrix and singular vectors of the centred data matrix are the same vectors arrived at two ways.
• Spectral methods on graphs. Community detection, graph clustering, and node embeddings (Laplacian eigenmaps, normalised cuts) all hinge on the top eigenvectors of an adjacency or Laplacian matrix. The eigenvalues say how separable each cluster is; the eigenvectors say which nodes belong to it.
• Rank collapse in transformers. “Attention loses rank with depth” is, mechanically, “the dominant eigenvalue of the repeated attention operator pulls every token toward a single direction.” The residual stream and feed-forward sublayers are what stop that eigenvalue from running away.
• Optimisation diagnostics. The eigenvalues of the loss Hessian — its spectrum — describe the local curvature. The ratio of the top to the bottom eigenvalue is the condition number; second-order methods, learning-rate schedules, and sharpness-aware training all read this spectrum directly.

Once “this matrix has an interesting spectrum” becomes a thought you can have, the same lens fits PCA, spectral clustering, transformer depth dynamics, and Hessian-based optimisation. Eigenvectors are how a matrix tells you which directions in its domain actually matter to it; eigenvalues are how loudly it says so.

SVD: the rank microscope

To measure rank, we need a tool that turns a matrix into a list of “how much each independent direction is contributing.” That tool is the singular value decomposition. Any real matrix W of shape (m, n) can be written as

$$W = U \Sigma V^\top$$

where:

• $U$ is (m, m), an orthogonal matrix — its columns are perpendicular unit vectors. Orthogonal matrices are exactly the rotations and reflections of space; they don’t stretch or shrink, they just spin.
• $\Sigma$ is (m, n), diagonal with non-negative entries $\sigma_1 \ge \sigma_2 \ge \cdots \ge 0$ on the diagonal and zeros everywhere else. These are the singular values.
• $V^\top$ is (n, n), also orthogonal.

Geometrically, W is doing three things in sequence: rotate the input ($V^\top$), stretch each axis by the corresponding singular value ($\Sigma$), then rotate again ($U$). That decomposition is what the demo above is showing, even though it doesn’t draw the intermediate rotations — the final ellipse’s axis lengths are the singular values, and the axis directions are the columns of $U$.

The rank reads directly off $\Sigma$: it is the number of non-zero singular values. Zero singular values mean some directions get scaled to zero — annihilated — and the output dimensionality drops accordingly.

A worked example. The rank-1 matrix from earlier:

$$M = \begin{bmatrix} 1 & 2 & 3 \\ 2 & 4 & 6 \\ 3 & 6 & 9 \end{bmatrix} \quad\Rightarrow\quad \sigma_1 \approx 14.0, \;\; \sigma_2 = 0, \;\; \sigma_3 = 0$$

One non-zero singular value. Rank 1. The SVD agrees with the column inspection from earlier — as it must.

Effective rank — the spectrum, not the number

Pure rank — the count of non-zero singular values — is a clean definition, but in practice it’s almost useless on its own. Real matrices, especially those produced by training, almost never have exact zeros in their singular value spectrum. They have a decay: large values at the top, then a smooth fall-off, and a long tail of tiny but non-zero values. Strictly speaking, such a matrix is full rank. Functionally, it isn’t.

The number people actually care about is effective rank: how many singular values are large enough to matter. Different ways to formalise this:

• Threshold rank: count the singular values above some cutoff, e.g. $\sigma_i > 10^{-6} \cdot \sigma_1$.
• Energy rank: the smallest k such that $\sum_{i=1}^k \sigma_i^2$ captures, say, 99% of $\sum_i \sigma_i^2$. This is the “how many directions do we need to keep to preserve 99% of the matrix” question.
• Stable rank: $\|\Sigma\|_F^2 / \sigma_1^2 = (\sum_i \sigma_i^2) / \sigma_1^2$. A scalar that summarises the whole spectrum in one number.

When someone says “this 1024×1024 layer is effectively rank 50,” they are using one of these definitions. The exact choice rarely matters; what matters is the shape of the singular value plot. A trained layer’s spectrum tells you, at a glance, how much of its parameter budget is doing real work.

Neural layers under the rank lens

Here is the payoff. A linear layer y = Wx with W of shape (out, in) has out × in parameters and rank at most min(out, in). If W’s effective rank is r — and r is much smaller than min(out, in) — then the SVD tells us we can approximate W as

$$W \approx U_r \Sigma_r V_r^\top = (U_r \sqrt{\Sigma_r}) (\sqrt{\Sigma_r} V_r^\top) = B A$$

where B is (out, r) and A is (r, in). The parameter count goes from out × in to r × (out + in). For (1024, 1024) with r = 50, that’s 10× fewer parameters. The trained layer was always “really” two small matrices in a trench coat; we just hadn’t measured it.

This single observation — learned matrices have low effective rank — is the seed of three large research areas:

• LoRA and low-rank fine-tuning. Instead of updating the full weight matrix during fine-tuning, learn a low-rank update $\Delta W = B A$ with r in the single or double digits. Works because fine-tuning updates are empirically very low rank: you’re not relearning the whole transformation, you’re nudging it along a few directions.
• Pruning and low-rank compression. Truncate the SVD to the top k singular values and you’ve compressed the layer with bounded approximation error. Magnitude pruning, structured pruning, and tensor decomposition methods are all variations on this theme.
• Rank collapse in deep transformers. Stack many attention layers without residual connections and token representations converge toward a one-dimensional subspace — the network’s effective rank drops to 1 across depth, and it loses the ability to distinguish positions. Residual connections, LayerNorm, and feed-forward sublayers are partly what keep the rank healthy across depth. (See “Attention is not all you need: pure attention loses rank doubly exponentially with depth”, Dong et al. 2021.)

Each of these deserves its own article and will get one. The point of this one is the lens itself: once you start seeing every weight matrix as “shape × effective rank” rather than just “shape,” a lot of seemingly disconnected results in modern deep learning start to look like variations on the same observation.

Where to next

If this article gave you a working picture of rank, the natural follow-ups on this site are:

• LLM inference optimization techniques — quantization and low-rank decomposition as production-time exploits of the same effect.
• Inside a transformer block — where rank collapse is a live concern and where the residual / LayerNorm / FFN machinery earns its keep.
• Context assembly strategies — which sits one level up, on how to use the resulting model effectively.

The matrix is a transformation. The rank is the dimensionality of what survives. Every neural network is built out of these two ideas; once you see them, you can’t un-see them.