Decision trees & random forests

Some of the most powerful models on everyday (spreadsheet-style) data aren't neural networks at all — they're decision trees and the forests you build from them. The best part: a decision tree is something you already understand intuitively. It's a flowchart of yes/no questions. The clever bit is how a computer decides which questions to ask. Let's dig in.

A tree is a flowchart of questions

To decide if an email is spam, a tree might ask: "Does it contain the word 'lottery'?" → if yes, "More than 3 links?" → and so on, until it reaches a leaf that says spam or not spam. Each internal node is a yes/no question that splits the data into two groups; each leaf is a prediction.

Training a tree means answering: at each node, which question splits the data best? "Best" means the resulting groups should be as pure as possible — ideally all-spam on one side, all-not-spam on the other.

Measuring purity: Gini impurity

We need a number for "how mixed is this group?" The most common is Gini impurity. If a group has classes with proportions $p_1, p_2, \dots$, then:

$\text{Gini} = 1 - \sum_{i} p_i^2$

• A pure group (all one class): $\text{Gini} = 1 - 1^2 = 0$. Perfect.
• A 50/50 two-class group: $1 - (0.5^2 + 0.5^2) = 1 - 0.5 = 0.5$. Maximum mess.

Worked example

A node holds 8 spam and 2 not-spam (10 emails). Proportions: $p_{\text{spam}} = 0.8$, $p_{\text{not}} = 0.2$.

Gini = 1 − (0.8² + 0.2²)
     = 1 − (0.64 + 0.04)
     = 1 − 0.68
     = 0.32

Now compare a split. Suppose the question "contains 'lottery'?" sends 4 emails (all spam) one way and 6 emails (4 spam, 2 not) the other:

left  {4 spam, 0 not}: Gini = 1 − (1² + 0²)            = 0
right {4 spam, 2 not}: Gini = 1 − ((4/6)² + (2/6)²) ≈ 0.44
combined (weighted) = (4/10)×0 + (6/10)×0.44 ≈ 0.27

The split dropped impurity from 0.32 to 0.27, so it's a helpful question. The tree tries every possible question, computes this combined impurity for each, and picks the one that lowers it the most — then repeats on each child. That greedy "pick the purest split" loop is the entire training algorithm.

The catch: trees overfit

Left unchecked, a tree keeps splitting until every leaf is a single example — memorising the training data perfectly and then failing on anything new. The fixes: limit the tree's depth, require a minimum number of examples per leaf, or prune branches that don't help on held-out data.

Random forests: a crowd of trees

A single tree is twitchy — change a few data points and it can look completely different (high variance). The fix is wonderfully simple: build many trees and let them vote. That's a random forest. To make the trees usefully different from each other, each one is trained on:

• a random sample of the rows (with replacement — "bootstrapping"), and
• a random subset of the questions it's allowed to ask.

Then, to classify, every tree votes and the majority wins (for numbers, you average). Because each tree makes different mistakes, their errors tend to cancel when you average — so the forest is far steadier and more accurate than any single tree. This "combine many weak-ish models" trick is called ensembling, and it's one of the most reliable ideas in machine learning.

Why this matters

• On tabular data (the spreadsheets that run most of the real world), tree ensembles like random forests and gradient boosting (XGBoost, LightGBM) are often the best models — beating deep networks.
• They're interpretable: you can read the questions and see why a prediction was made.
• The Gini/split idea and the "average many models" idea both generalise far beyond trees.

Play with both below. First grow a tree and watch it split the data; then see a forest of trees vote — and how the crowd beats any single member.

---

Want to build a tree's split logic yourself? Try a free lesson.