Ch.39: Decision Tree Hyperparameters
Inspired by: YouTube
An unconstrained decision tree will keep splitting until every leaf is pure, or has just one row left. That gives it a well-earned reputation as a high-variance algorithm: left alone, it memorizes the training set, entropy at the root but zero at every leaf, and then fails to generalize to anything it hasn't seen. sklearn's DecisionTreeClassifier exposes a handful of hyperparameters that exist for exactly one reason: to stop the tree from growing that far.
1. Overfitting and Underfitting, the Two Failure Modes
Left unrestricted (max_depth=None, the default), a decision tree keeps splitting until every leaf is pure or has just one row left. Follow that all the way down and you eventually land on a leaf holding, say, two rows. Every decision the tree makes for any future point that lands in that leaf now rests entirely on those two rows. If they're genuine signal, fine, but if they're noise, outliers, or bad data, all the effort spent building the if-else chain above them was spent arriving at a leaf you can't actually trust. That's overfitting: the tree performs beautifully on training data (it's literally memorizing rows) but falls apart on anything new.
The geometric picture makes this concrete. A decision tree splits by drawing axis-parallel lines through the feature space, cutting it into boxes. Split enough times and you can end up with a box that's labeled, say, green, purely because a handful of green points happened to fall inside it, even though every point around that box is red. A new query point landing in that box gets confidently labeled green, when the surrounding evidence says it should almost certainly be red. This is the same failure mode nearest-neighbors has: trusting a single nearby point (or a tiny handful of them) too much.
The opposite failure, underfitting, shows up when the tree is stopped too early. With max_depth=1, a node with, say, 100 "yes" rows and 100 "no" rows gets exactly one split to work with, and every point on one side of that split is labeled by nothing more than a majority vote: whichever class has more rows on that side wins, regardless of where the point actually sits relative to the real boundary. The tree hasn't learned the data's shape at all, it's just guessed the locally dominant class, and it performs poorly on training data and test data alike.
Every hyperparameter below is a knob between these two failure modes, and tuning them is the practical answer to the question ch37 left open: when should splitting stop?
The demo dataset throughout is sklearn's make_moons, two interleaved, non-linearly-separable classes with some added noise, split 70/30 into train and test:
from sklearn.datasets import make_moons
from sklearn.model_selection import train_test_split
X, y = make_moons(n_samples=300, noise=0.25, random_state=42)
X_train, X_test, y_train, y_test = train_test_split(
X, y, test_size=0.3, random_state=42
)Every hyperparameter in the rest of this post is a live control in the explorer below, the same kind of interactive web app the video itself walks through hyperparameter by hyperparameter, running a DecisionTreeClassifier and re-fitting on each change. Push max_depth up until the boundary starts carving islands, then bring min_samples_leaf up to smooth them back out, and watch the train/test accuracy gap.
2. max_depth
The most direct lever: how many questions deep the tree is allowed to go from root to leaf. Left at its default (None), the tree grows until every leaf is pure, no matter how deep that requires.
from sklearn.tree import DecisionTreeClassifier
for depth in [1, 4, None]:
clf = DecisionTreeClassifier(max_depth=depth, random_state=42)
clf.fit(X_train, y_train)
print(depth, clf.score(X_train, y_train), clf.score(X_test, y_test))
max_depth = 1 is a single axis-aligned cut, it can't do much better than guess which half of the plane each moon mostly falls in. max_depth = None chases every noisy point, including ones that shouldn't be trusted, carving out small islands that hurt more than they help on new data. max_depth = 4 sits in between: close to the actual moon-shaped boundary, and (in this run) matched train/test accuracy rather than a gap between them. Small max_depth underfits, large or unset max_depth overfits, and the right value sits somewhere in the middle, found by comparing train and test scores rather than read off a formula. Doing that comparison by eye on a 2D toy dataset is one thing; doing it systematically on real, high-dimensional data (a proper grid search) is a separate technique, picked back up in §9.
3. criterion: Gini vs. Entropy
The split-quality measure introduced in ch38, Gini impurity by default, entropy as the alternative. Sweeping max_depth against test accuracy with each:
The two curves track each other closely, gini criterion peaked at 0.944 test accuracy (depth 6) versus entropy's peak of 0.889 (depth 5) in this run, but the shapes of both curves are nearly identical. In practice the choice rarely moves accuracy by much; gini is the default because it's cheaper to compute (no logarithm), and that's the only real reason to prefer it.
4. splitter: Best vs. Random
By default (splitter="best"), a node checks every feature and every candidate threshold on that feature, and picks whichever split gives the highest information gain. splitter="random" instead picks one random feature at each node and searches for the best threshold only on that one, trading a slightly worse-fitting tree for less overfitting, since the randomness stops the tree from over-committing to whichever feature looks marginally best on this particular training sample. Across a few random seeds, splitter="best" on this dataset lands around 0.91-0.93 test accuracy and splitter="random" around 0.89-0.93, close enough that the gap is mostly seed noise rather than a decisive win either way. That matches what the difference usually looks like in practice: it's a lighter-weight regularizer than the size-based hyperparameters below, worth trying but rarely anyone's default choice.
5. min_samples_split and min_samples_leaf
Both hyperparameters stop a split from happening once a node's row count gets too small, but they check the count on different sides of the split:
min_samples_split(default 2): a node needs at least this many rows before it's allowed to split at all.min_samples_leaf(default 1): after a split, both resulting children must end up with at least this many rows, or the split doesn't happen.
min_samples_leaf is the more common one to tune, since it directly bounds how small (and how easily swayed by one or two noisy points) any single leaf can be:
for leaf in [1, 5, 20]:
clf = DecisionTreeClassifier(min_samples_leaf=leaf, random_state=42)
clf.fit(X_train, y_train)
print(leaf, clf.score(X_train, y_train), clf.score(X_test, y_test))
min_samples_leaf = 1 lets a leaf form around a single point, the small-islands overfitting pattern again (train 1.00, test 0.92). Raising it to 5 smooths those islands away while keeping the two-moon shape largely intact (train 0.95, test 0.94). Pushed to 20 the tree can no longer represent the moon curve at all and coarsens into wide horizontal bands, underfitting both sets (train 0.83, test 0.80). Small values risk overfitting, large values risk underfitting, same trade-off as max_depth, applied at the leaf-size level instead of the depth level.
6. max_features
Restricts how many features a node is allowed to consider when searching for its best split, rather than checking all of them. Say a dataset has 100 columns and max_features is set to 50: at every single node, a random 50-column subset is drawn, nobody controls which 50, and the split search runs only over information gain on those columns. With only two features (as in the make_moons demo) this has almost no room to matter, its real effect shows up on wider datasets. The randomness it introduces makes training-data performance a little worse than an unrestricted tree would get, which sounds like a bad trade in isolation, but it's exactly the kind of hyperparameter that pays off once decision trees stop being used alone and start being combined into ensembles, a thread the closing section picks back up.
7. max_leaf_nodes
A budget on the tree's total leaf count rather than its depth or per-leaf row count, letting the tree spend that budget wherever the gain is highest instead of growing uniformly deep everywhere.
Two leaves means the tree got exactly one split to work with (train 0.82, test 0.78). Each doubling buys a noticeably better fit to the moon shapes, four leaves reaches 0.90/0.90, eight leaves reaches 0.94/0.92, until an unrestricted budget eventually reproduces the same small-islands overfitting seen with unrestricted max_depth (train 1.00, test 0.92). Same shape of trade-off as every hyperparameter so far, just controlling a different structural property of the tree.
8. min_impurity_decrease
Instead of budgeting size, this sets a minimum bar on split quality: a split only happens if it reduces weighted impurity by at least this amount. Concretely, on the root node of this dataset, the parent's Gini impurity is 0.498 (210 rows split roughly evenly between classes), and the best available split sends 84 rows to a child with Gini 0.245 and 126 rows to a child with Gini 0.328. Weight those by how many rows land in each side and subtract from the parent:
That 0.203 is the actual impurity decrease this split buys, so min_impurity_decrease values up to about 0.2 still let the root split happen; push it past that and even the best available split at the root isn't good enough, and the tree never grows past a single leaf. Sweeping the threshold confirms the pattern: 0.0 (the default) reaches its full depth of 9 and 24 leaves (train 1.00, test 0.92); 0.01 cuts it back to depth 5, 7 leaves (train 0.94, test 0.91); 0.02 allows only the root split, depth 2, 4 leaves (train 0.90, test 0.90); 0.05 stops even earlier. Set it too high and the tree stops early even where a real pattern remained to be found, underfitting; set it too low (the default) and the tree accepts splits that only marginally help, one step short of what min_samples_leaf and max_depth are already there to prevent.
9. Tuning in Practice
None of these hyperparameters has a universally correct value, they're all found the same way: fit with a candidate value, compare train accuracy against test accuracy, and adjust. A large gap (high train, low test) means overfitting, tighten a constraint. Both scores low and close together means underfitting, loosen one. The mental model that ties every hyperparameter in this post together is the same one: more freedom to split (max_depth, max_leaf_nodes unrestricted, min_samples_leaf at 1, min_impurity_decrease at 0) pushes toward overfitting, and tighter limits on any of them push back toward underfitting, until train and test accuracy land close together.
Every chart so far tuned one hyperparameter at a time, which hides how they interact. Gridding max_depth against min_samples_leaf together shows it directly:
It's worth caring about all eight of these specifically for decision trees, and not waiting to pick them up later by accident, because decision trees don't stay standalone models for long: Random Forests, bagging, and gradient boosting are all built out of many decision trees combined. Knowing what max_depth, min_samples_leaf, max_features, and the rest each do to a single tree is exactly the knowledge that carries forward once the model being tuned is an ensemble of them.
min_samples_leaf = 1 reaches the same peak as min_samples_leaf = 5, but only at exactly the right depth, push depth further and it starts overfitting again. min_samples_leaf = 5 holds that peak across a wide range of depths instead of just one, a size constraint doing part of the job a depth limit alone would otherwise have to get exactly right. In practice this is the case for tuning more than one hyperparameter together (a grid search) rather than sweeping each in isolation.