SVMs and the kernel trick

For decades, before deep learning took over, the support vector machine (SVM) was the sharpest classifier around — and it's still excellent on small, clean datasets. It rests on two elegant ideas: draw the boundary with the widest possible margin, and use the kernel trick to separate data that looks hopelessly tangled. Let's unpack both.

The widest street

Suppose you have two classes of points and want a line to separate them. Usually many lines work. Which is best? An SVM picks the one with the biggest margin — the widest empty "street" between the two classes.

The line is defined by weights $w$ and bias $b$; a point $x$ is classified by which side it falls on:

$\text{class} = \text{sign}(w \cdot x + b)$

The few points sitting right on the edge of the street — the ones that "hold the boundary in place" — are the support vectors. Everything else could move around without changing the answer; only these matter. Maximising the margin makes the classifier as robust as possible: new points have the most room before they'd be misclassified.

The problem: not everything is separable by a line

Some data simply can't be split by any straight line. Classic example, in 1-D: class B sits in the middle, class A on the outside.

position:   −3   −2   −1    0    1    2    3
class:       A    A    B    B    B    A    A

No single threshold separates A from B — B is surrounded. A straight line fails.

The kernel trick: add a dimension

Watch what happens if we invent a new feature: $x^2$. Each 1-D point $x$ becomes a 2-D point $(x,\, x^2)$:

x = −3 → (−3, 9)   A
x = −2 → (−2, 4)   A
x = −1 → (−1, 1)   B
x =  0 → ( 0, 0)   B
x =  1 → ( 1, 1)   B
x =  2 → ( 2, 4)   A
x =  3 → ( 3, 9)   A

Now look at the new $x^2$ coordinate: class B has $x^2 \le 1$, class A has $x^2 \ge 4$. A simple horizontal line at $x^2 = 2$ cleanly separates them! By lifting the data into a higher dimension, a curved boundary in the original space became a straight one up top.

The catch: mapping every point into a huge (sometimes infinite) dimension would be impossibly expensive. The kernel trick is the clever escape: the SVM's maths only ever needs dot products between points. A kernel function $K(a, b)$ computes the dot product in the high-dimensional space directly from the original points — without ever building those big vectors:

$K(a, b) = \varphi(a) \cdot \varphi(b)$

where $\varphi$ is the (never-actually-computed) mapping. Popular kernels:

• Linear — just $a \cdot b$ (a plain straight boundary).
• Polynomial — separates curved boundaries like our $x^2$ example.
• RBF (Gaussian) — the workhorse; effectively an infinite-dimensional lift, able to carve very flexible boundaries.

So an SVM gets the power of a wildly high-dimensional separation for the cost of a simple function on the original points. That's the trick.

Why this matters

• On small-to-medium, clean datasets, SVMs are still competitive with — sometimes better than — neural networks, and they need far less data.
• The max-margin idea (widest safe gap) shows up across ML as a principle for robust boundaries.
• The kernel trick — "I only need dot products, so I can pretend I'm in a huge space" — is a beautiful, reusable idea that also underlies Gaussian processes and more.

Now see the margin. In the demo below, watch the SVM find the widest street between two classes and highlight the support vectors that hold it in place.

---

The dot product doing all the heavy lifting here is the same one from embeddings and attention. Want to build an SVM? Try a free lesson.