PCA: finding the directions that matter

Real datasets are wide — dozens or hundreds of columns. But the data rarely uses all those directions; it usually spreads out along just a few. Principal Component Analysis (PCA) finds those few directions and lets you throw the rest away with almost no loss. It's the workhorse of "make this data smaller and simpler." Let's build it up.

The goal: keep the spread, drop the rest

Imagine plotting your data as a cloud of points. Some directions the cloud is long and stretched (lots of variety); other directions it's thin and flat (everything's about the same). The stretched directions carry the information; the flat ones barely matter.

PCA's plan:

1. Find the direction the data is most stretched — call it PC1 (the first principal component).
2. Find the next-most-stretched direction that's at right angles to PC1 — PC2.
3. Keep the top few; describe every point by just those coordinates. Fewer numbers, almost all the information.

"How stretched" has a precise name: variance.

Variance = how spread out

The variance of a set of numbers is the average of their squared distances from the mean:

$\mathrm{Var} = \frac{1}{n}\sum_{i=1}^{n}\left(x_i - \bar{x}\right)^2$

(Here $\bar{x}$ is the mean.) Big variance = spread out; zero variance = all the same. PCA is literally "find the direction with the biggest variance."

Projecting a point onto a direction

To measure a point's position along a direction, you project it — and projection is just a dot product. If $u$ is a unit-length direction and $x$ is a (mean-centred) point, its coordinate along $u$ is:

$t = x \cdot u$

So PCA becomes: find the unit direction $u$ that makes the variance of all the $t$'s as large as possible. That best direction is PC1.

Worked example

Take three points sitting on a diagonal line: $(1,1)$, $(2,2)$, $(3,3)$. Their mean is $(2,2)$, so the mean-centred points are $(-1,-1)$, $(0,0)$, $(1,1)$.

Project onto the diagonal direction $u = \left[\tfrac{1}{\sqrt2}, \tfrac{1}{\sqrt2}\right] \approx [0.707, 0.707]$:

(-1,-1)·u = (-1)(0.707) + (-1)(0.707) = -1.414
( 0, 0)·u = 0
( 1, 1)·u = (1)(0.707) + (1)(0.707)   = +1.414

The projections spread from −1.414 to +1.414 — big variance. Now project onto the perpendicular direction $u' = [-0.707, 0.707]$:

(-1,-1)·u' = (-1)(-0.707) + (-1)(0.707) = 0.707 - 0.707 = 0
( 0, 0)·u' = 0
( 1, 1)·u' = (1)(-0.707)  + (1)(0.707)  = 0

Every point projects to 0 — zero variance in that direction. So all the information lives along the diagonal. PCA would pick the diagonal as PC1 and let you describe each point with a single number instead of two, losing nothing. (Real data isn't perfectly on a line, so PC1 captures most — not all — of the variance, and you keep enough components to reach, say, 95%.)

Where the eigenvectors come in

Finding these best directions turns out to be an eigenvalue problem: the principal components are the eigenvectors of the data's covariance matrix, and each one's eigenvalue tells you how much variance it captures. You don't need that machinery to get the idea — but it's why, in practice, PCA is one line of code: build the covariance matrix, take its eigenvectors, keep the ones with the biggest eigenvalues.

Why this matters

• Compression & speed — 100 columns down to 10 means faster training and less storage, with little lost.
• Visualisation — squash high-dimensional data to 2-D so you can actually plot it.
• Noise removal — the tiny-variance directions are often just noise; dropping them cleans the data.

Now try it. Rotate the axis in the demo and watch the variance of the projected shadows rise and fall — it peaks exactly at PC1, the direction PCA would keep.

---

The dot product and variance here come straight from the maths foundations. Want to build PCA yourself? Try a free lesson.