Principal component analysis — dimensionality reduction you can see

A lot of the data modern ML produces lives in hundreds of dimensions. Word embeddings are 300-dim; image features off a CNN backbone are 512–2048-dim; sentence embeddings are 768-dim or more; gene-expression vectors can be tens of thousands. Inspecting points in spaces that wide is impossible — we can draw two or three dimensions, not three hundred. Principal component analysis (PCA) is the standard trick for getting around this: take the d-dimensional points, find the two (or three) directions along which they vary the most, and project everything onto those directions.

The result is a 2D plot you can stare at, with the most prominent structure of the original space preserved. You throw most of the dimensions away — but the dimensions you keep are the ones that carried the most signal.

It’s worth being clear about what PCA is before we lean on it. At heart it’s a feature-engineering / dimensionality-reduction technique — a preprocessing step, not a predictive model itself. It takes many correlated features and re-expresses them as a smaller set of uncorrelated ones (the principal components), which then feed whatever comes next: a classifier, a clustering algorithm, or — when you reduce all the way to two or three dimensions — your own eyes. This article focuses on that last case, visualization, because it’s the most immediately legible; but the same machinery is doing feature extraction, noise reduction, and compute savings every time it’s used as a pre-ML step. (We come back to that framing concretely further down.)

Rung 3 of the linear-algebra guided map — it points the rank microscope of matrix rank at data instead of at a transformation.

The idea

Imagine a cloud of 300-dimensional points. Most of those 300 directions don’t carry much spread — points are tightly bunched along them, so they contribute very little to distinguishing one point from another. A few directions, on the other hand, are the ones along which the points actually stretch out. PCA finds those high-variance directions.

Formally: PCA looks for the orthogonal axes (directions) of greatest variance in the data. The first principal component is the single direction along which the points spread out the most. The second is the direction of next-greatest spread that’s perpendicular to the first. The third, perpendicular to both. And so on, up to the data’s intrinsic dimensionality.

You can compute these directions from the eigenvectors of the data’s covariance matrix (or, equivalently, via SVD on the centred data matrix). Once you have them, “projecting onto the top-2” just means taking the dot product of each data point with each of the top-2 directions — two real numbers per point, ready to plot.

What you lose: everything that lived in the discarded directions. What you keep: the dominant structure, the part that most differentiates the points from each other. For data with a few clear axes of variation, this is enormously informative. For data where variance is spread evenly across many dimensions, PCA’s 2D view will miss a lot.

Before any covariance matrices, it helps to feel what “direction of greatest variance” means. The widget below is a 2D cloud with a single candidate axis through its centre. Each point’s projection onto the axis is a hollow circle; the grey segment is the bit you’d discard. Spin the axis with the slider — or hit OPTIMIZE — and watch VAR, the variance of the projections:

Two things move together as you turn the line. When the projections spread out (high VAR), the grey residual segments get short; when they bunch up (low VAR), the residuals get long. That’s not a coincidence — by the Pythagorean theorem, total-spread = projected-variance + leftover-residual is a constant, so maximising the variance you keep is exactly the same as minimising the error you throw away. The axis that wins both is PC1. (This framing — spin the axis, read off the variance — is borrowed from Kynd’s interactive PCA explainer.)

Where the directions come from: the covariance matrix

Spinning the axis by hand finds PC1, but you don’t have to search — the principal directions fall straight out of one matrix. Take your data as an m × 2 matrix X (m people, two features — say height and weight), subtract each column’s mean so the cloud is centred on the origin, and form

This 2 × 2 is the covariance matrix. Its diagonal entries are the variance of each feature on its own; the off-diagonal is the covariance between them — how much they move together. Concretely, for three people whose heights are 5, 6, 7 feet:

• If the taller people are also heavier (weights 120, 160, 220), the off-diagonal comes out positive (≈ +100) — height and weight rise together.
• Re-shuffle so the tallest person is the lightest (weights 160, 220, 120) and the same calculation gives a negative off-diagonal (≈ −40) — the features now pull against each other.

Because XᵀX is built from the data multiplied by its own transpose, it is always square and symmetric — which, from the eigenvector picture, means its eigenvectors are guaranteed real and orthogonal. And here is the punchline that all of this was building toward:

The eigenvectors of the covariance matrix are the principal components of X. Each eigenvector points along a direction of variation in the data, and its eigenvalue is exactly the variance along that direction.

So PCA is just: centre the data, build the covariance matrix, take its eigenvectors, and sort them by eigenvalue. The widget below shows this happening. Toggle between the three datasets and watch the covariance matrix’s off-diagonal flip sign as the cloud tilts — and watch the eigenvectors (the coloured PC axes) swing to lie along the data’s spread, with PC1 always claiming the most variance:

Notice that PC1 and PC2 stay perpendicular in every dataset — that’s the symmetry of the covariance matrix doing its work. When the features are uncorrelated, the off-diagonal sits near zero and the principal components line up with the original axes; the more the cloud tilts, the larger the covariance and the more PC1 swings away from the axes to chase the spread.

There’s a second route to the very same directions. Instead of forming XᵀX and taking its eigenvectors, you can run SVD directly on the centred data matrix X = U Σ Vᵀ — the columns of V come out equal to the eigenvectors above, and the squared singular values in Σ are proportional to the eigenvalues (the variances). It’s the same line, found a different way. In practice this is the route libraries like scikit-learn actually take, because forming XᵀX squares the condition number and loses precision, whereas SVD works on X straight. It’s also the route the word2vec article takes when it compresses a word-by-word co-occurrence matrix down to dense embeddings — same “find the top directions of variation and project onto them” idea, applied to a V × V matrix instead of a 2D cloud.

A toy example: 2D → 1D

Before scaling up to 300 dimensions, it helps to see the mechanic in a setting you can actually draw. Here are ten points in 2D — five circles and five triangles — that happen to lie roughly along a 45° line:

The point of panel A is to notice that the two features x₁ and x₂ aren’t really independent. Knowing a point’s x₁ tells you a lot about its x₂, because the points stretch out along a single direction. The variance along the 45° diagonal is large; the variance perpendicular to it is small.

Panel B draws that observation. The dashed yellow line is the first principal component (PC1) — the direction along which the data spreads the most. The blue line is PC2, perpendicular to it. PC2 captures the small wobble around the diagonal.

Panel C is the projection: keep only the PC1 coordinate of each point, throw PC2 away. We’ve gone from 2D to 1D — but the circles still sit clearly to the left of the triangles. The structure that mattered for telling the two clusters apart lived entirely along PC1, so the dimension we discarded cost us almost nothing.

You can even use the projection to classify new points. Hand the model an unlabelled shape, project it onto PC1, and check whether it lands to the left or right of the boundary between the two groups — that’s enough to call it a circle or a triangle. The catch is that this only works if the new point is drawn from the same distribution as the data PCA was fit on. If the underlying geometry has shifted, PC1 is no longer the right axis to project onto, and “left of the boundary” stops meaning what it did before.

This is PCA in miniature. The same procedure scales up: in 300 dimensions, find the direction of greatest variance, then the next-greatest perpendicular to it, then the next, and keep however many you have room to plot. The 2D you keep are the 2D worth keeping.

A real example: the Iris dataset, 4D → 2D

The toy example was 2D so we could draw both the data and the projection. The point of PCA, though, is the case where you can’t draw the data. The classic example — the one that has introduced PCA to generations of statisticians — is Edgar Anderson’s iris measurements, collated by R. A. Fisher in 1936. It is 150 flowers from three species — setosa, versicolor, virginica — with four features measured on each: sepal length, sepal width, petal length, petal width. That makes a 150 × 4 data matrix. Four features means each flower is a point in 4D space, and 4D is exactly one dimension past what we can plot.

So we run the recipe from the last section, just with four features instead of two:

1. Mean-correct the 150 × 4 matrix X (centre each column).
2. Form the covariance matrix XᵀX — a 4 × 4 = (4×150)·(150×4), square and symmetric, so it has four real orthogonal eigenvectors.
3. Sort the eigenvectors by their eigenvalue and keep the top two, stacking them into a 4 × 2 matrix W_r (“W-reduced”).
4. Project: T = X · W_r, a 150 × 2 matrix. Each flower is now two numbers — its coordinates along PC1 and PC2 — and the new axes no longer mean “petal width”; each is a mix of all four original features, weighted by how much each contributes to that direction of variance.

For iris, that top-2 projection is unusually faithful: PC1 alone captures 92% of the total variance and the top two together capture 98%, so the 2D picture throws away almost nothing. Here is the result — start with just the projection, then reveal the species labels (which PCA never saw):

With no labels at all, the projection already shows one tight group sitting well apart from a larger, looser one. Colour by the true species and you see why: setosa is linearly separable along PC1, while versicolor and virginica overlap — they really are harder to tell apart from these four measurements, and PCA is being honest about that rather than inventing a separation that isn’t there.

The payoff is the third button. Because the projected points already fall into visible groups, you can hand the unlabelled 2D data to a clustering algorithm — K-means with k=3, which iterates to find three centroids and assigns each point to its nearest one — and it recovers the species roughly 89% of the time, with essentially all the errors in the versicolor/virginica overlap. PCA found the structure; K-means named it; neither was ever told what an iris is. That pairing — reduce dimensions with PCA, then cluster — is a workhorse of exploratory data analysis.

A concrete example: word2vec country/capital pairs

Word embeddings make a clean illustration. A trained word2vec model assigns every word in its vocabulary a vector in some space — typically 300 dimensions. Pairs of related words like france/paris, germany/berlin, japan/tokyo end up with consistent vector offsets: the direction france → paris is approximately the same direction as germany → berlin and japan → tokyo. That’s the famous “analogy arithmetic” — paris − france + germany ≈ berlin.

The geometry exists in 300 dimensions, but the “country → capital” offset is largely a single direction. PCA, given a handful of country/capital pairs, should find it as one of its top principal components — and the 2D projection should make the parallelism visible.

The widget below runs PCA on GloVe-300 vectors for 12 country/capital pairs, projects them to 2D, and connects each pair with a dashed line:

The dashed lines are roughly parallel. That parallelism is what makes the analogy arithmetic work: the vector you’d add to france to land at paris is approximately the same vector you’d add to germany to land at berlin, or russia to land at moscow. The “capital of” relation has a consistent direction in the original 300-dim space, and PCA preserves enough of it for the 2D projection to be visible.

A few things worth noticing about this view:

• PCA doesn’t know the labels. It wasn’t told which points are countries and which are capitals, or that they come in pairs. It found the directions of greatest variance in the raw vectors and projected onto the top two. The country/capital separation drops out as a consequence of how the points actually sit in 300-dim space — capitals genuinely sit in a different region than countries do, because they appear in different contexts in the training corpus.
• The parallelism isn’t perfect. Some lines are tighter than others. In the full 300-dim space the alignment is much cleaner; the 2D projection throws away most of the dimensions, and some of that lost variance was contributing to the “capital of” direction. So the visualization is a faithful but slightly degraded picture of the actual structure.
• Only the dominant structure survives. Other directions in the original space — verb-tense offsets, gender offsets, formal-informal offsets — exist but get squashed in this 2D view because they happen along axes that aren’t in the top 2 for this particular subset of points. If you ran PCA on a different subset of the vocabulary you’d see different structure pop out, because the top-2 directions of variance would change.

Beyond visualization: PCA as preprocessing

Everything so far has used PCA to see — squash high-dimensional data down to 2D and look at it. But the same projection is just as often a preprocessing step that feeds another model rather than a human eye. Three things make it useful there:

• Feature extraction. The principal components are new features — each a weighted mix of the originals — ordered by how much variance they carry. Keeping the top few gives a compact, decorrelated feature set that retains most of the signal.
• Noise reduction. Variance that lives in the discarded low-eigenvalue directions is often jitter or measurement noise rather than structure. Dropping those directions can act as a denoiser, smoothing out the axes along which the data barely moves.
• Compute savings. Fewer features means less to train on — faster fits, lower memory, and sometimes better generalization, since a model with fewer inputs has fewer ways to overfit.

A clean real-world example comes from anesthesiology, by way of Anil Ananthaswamy’s Why Machines Learn. Emery Brown and colleagues recorded EEG from ten patients as they were put under with propofol, turning each two-second slot into a 100-dimensional vector of power-spectral-density features. Seven subjects’ slots stack into a 37,800 × 100 matrix X (7 × 5,400 time slots). Run the recipe: mean-correct, form the 100 × 100 covariance matrix XᵀX, take its eigenvectors.

Here’s the twist that makes the example worth telling. They kept the top three eigenvectors — a 100 × 3 matrix — then threw the first one away. PC1 captures the most variance, but that dominant variation turned out to have nothing to do with consciousness; it was some other, larger effect in the EEG. Keeping it would have spent a precious axis on a nuisance factor. So Wᵣ becomes a 100 × 2 matrix — the second and third components — and projecting a subject’s 5,400 × 100 data through it (5,400 × 100 · 100 × 2) gives a 5,400 × 2 matrix: every two-second state as a single point on a plane.

Plot those points — gray circles for conscious, black triangles for unconscious — and the two states pull apart into largely separate clusters. Not perfectly: a few triangles sit among the circles and vice versa, so no straight line separates them cleanly, which means a perceptron never converges, but a naïve Bayes or k-nearest-neighbour classifier finds a usable boundary anyway. Train that classifier on the 2D points, then test on the three held-out subjects: project each of their 100-dim vectors through the same Wᵣ, ask the classifier “conscious or unconscious?”, and check against the ground truth. PCA did the heavy lifting — collapsing 100 noisy dimensions to the two that mattered — and a simple classifier finished the job.

That “drop PC1 because its variance is the wrong variance” move is the practical face of an assumption baked deep into PCA — one worth examining directly.

What PCA preserves, what it loses

PCA is a linear projection. That has two important consequences:

• It preserves linear structure — straight lines in the original space become straight lines in the projection, parallel offsets stay (roughly) parallel, and distances along the top components are preserved. That’s why “consistent vector offset” relations like the country/capital one survive.
• It can mangle non-linear structure — clusters that curve through the original space, or structure that requires non-linear combinations of dimensions to see, will look worse than they are. For those cases, non-linear methods like t-SNE or UMAP are stronger; they’re designed to preserve local neighbourhoods at the cost of distorting global geometry.

A simple way to think about the tradeoff: PCA’s 2D view answers “if I have to draw this with a single linear projection, what’s the best one?” t-SNE and UMAP answer “what 2D arrangement preserves which points are near each other?” They give different views and are useful for different questions.

There’s a deeper assumption baked into PCA that’s worth naming: it equates variance with importance. The dimensions you discard are the low-variance ones — but a direction along which the data barely moves might still be the one that actually separates the classes you care about. And the opposite happens too: a high-variance direction might just be capturing noise, scale artefacts, or some dominant nuisance factor that has nothing to do with what you’re trying to predict — exactly what happened with PC1 in the EEG example above. PCA is unsupervised — it doesn’t know your labels — so “top component” only ever means “most variance”, not “most predictive”. When that gap matters, supervised cousins like linear discriminant analysis (LDA) find the directions that separate known classes, rather than the directions that maximise overall spread.

When to reach for PCA

PCA is the right first tool when:

• You have continuous-valued, high-dimensional vectors with global structure worth seeing — embeddings, sensor readings, gene-expression profiles.
• You want a deterministic, interpretable projection — PCA’s top components are well-defined directions you can compute, store, and apply to new points (unlike t-SNE/UMAP, which fit per-dataset).
• You want to understand variance — PCA also tells you how much variance each component captures, which is genuinely informative (“the top 2 components explain 65% of total variance” is a meaningful statement).

Reach for non-linear methods when the structure you care about is local clustering rather than global axes of variation — when you’d rather distort the global layout to keep close points close than preserve a single dominant direction.

Further reading

• Principal component analysis (Wikipedia) — the formal treatment.
• Kynd’s interactive PCA explainer — the variance-maximising “spin the axis” intuition, with more interactive widgets.
• How to use t-SNE effectively (Distill) — when t-SNE plots lie, and how to read them.
• Understanding UMAP — interactive tour of UMAP’s behaviour.
• Word embeddings — word2vec and vectors that mean something — where the country/capital vectors come from.