Ch.60: Gradient Boosting for Classification, the Intuition

Inspired by: YouTube

The regression posts on gradient boosting built up a specific four-step recipe: start with a simple baseline, compute a pseudo-residual, fit a tree to that residual, and update. The genuinely good news is that classification uses the exact same recipe, no new algorithm to learn. The only thing that changes is the loss function, squared error for regression, log loss for classification, and that one change ripples through every step: the baseline becomes log-odds instead of a mean, and every tree's leaf needs its own small formula to convert a residual back into something that can be added to a log-odds running total. This post works through why, step by step, on a small worked example, then shows what it looks like geometrically on a real dataset.

1. Why the Baseline Can't Just Be a Mean Anymore

In regression, stage 1 was the mean of the target column, straightforward, since salaries are numbers you can average. Classification labels are 0s and 1s, averaging them technically gives you a proportion, but that's not a quantity you can keep adding tree outputs to and expect sensible results (it would drift outside the sensible range as soon as you started adding and subtracting from it).

The fix is to work in log-odds space instead. For a small placement dataset (CGPA and IQ predicting whether a student gets placed) with 5 placed students and 3 not placed:

The initial model is:

That's the starting prediction for every row, regardless of CGPA or IQ, expressed as a log-odds value. To interpret it as something meaningful (a probability), convert with the sigmoid:

Every student starts out predicted "62% likely to be placed", which, with a 0.5 threshold, means the baseline model just predicts "placed" for everyone. Crude, but it's a sensible starting point: the training data is majority-placed (5 of 8), so guessing "placed" for everyone is the best a single constant can do, exactly analogous to how the regression baseline's mean was the best single constant for that problem.

2. The Pseudo-Residual Is Still Actual Minus Predicted

Here's the part that doesn't change: the pseudo-residual is still $y_i - \hat{y}_i$, actual minus predicted. What's different is that $\hat{y}_i$ is now a probability (0.62 above), not a log-odds value, since you can't meaningfully subtract a 0-or-1 label from a log-odds number. So residuals here are literally "actual label minus predicted probability", giving numbers between -1 and 1 rather than the arbitrary-scale residuals from the regression case.

3. Fitting a Tree to the Residual, Then the Hard Part

Same as regression: fit a shallow regression tree (a weak learner, kept small on purpose, capped at a handful of leaves) with the original features as input and the residuals as the target.

Worth pausing on that word "regression": even though this whole post is about a classification problem, the tree fit at every stage is a regression tree, not a classification tree. The reason is simple once you look at what it's actually being trained on: the target column at this step is a pseudo-residual, a continuous number like $-0.625$ or $0.375$, not a class label. A classification tree expects to predict discrete classes; a regression tree predicts numbers, which is exactly what's needed here. This holds at every stage, no matter how many trees get added, gradient boosting for classification is built entirely out of regression trees.

The tricky part is what comes next. In regression, a leaf's own averaged output was directly usable, add it straight to the running prediction. Here it isn't, because the tree was fit on residuals that are differences of probabilities, and the running total is being kept in log-odds. Mixing the two would be mixing units, so every leaf's probability-space residual has to be converted back into a log-odds contribution before it can be added to $F$.

Each leaf therefore needs its own conversion formula, one that takes that leaf's residuals (still in probability space) and outputs a single log-odds value:

Read it as: sum up the residuals that landed in this leaf (the numerator), and divide by a measure of how uncertain the previous stage's predictions were for those same rows (the denominator, biggest when PreviousProb is near 0.5, smallest when PreviousProb is confidently near 0 or 1). This formula falls out of a Newton step on the log-loss function; the full derivation is deferred to a dedicated math post, same as the regression case's mean and residual formulas were derived in a follow-up rather than the first intuition post.

For the placement dataset, tree 1 (capped at 3 leaves) splits first on CGPA (< 6.37), then, on the branch that fails that test, on IQ (< 132). Every row shares the same previous probability, the baseline's 0.625, so the residual is fixed entirely by the label: $+0.375$ if placed, $-0.625$ if not. Here's where every row lands:

Working it for Leaf B, the 2 rows whose previous probability was 0.625 and whose residuals were both $-0.625$ (both actually unplaced students the baseline was over-confident about):

Repeating this per leaf across the tree gives three leaf values: Leaf A (3 rows, all $+0.375$) comes out to $1.6$, Leaf B (above) is $-2.66$, and Leaf C (one $-0.625$ mixed with two $+0.375$s) comes out to $0.18$, each one a log-odds contribution, not a probability.

4. Combining Stages: Back and Forth Between Log-Odds and Probability

The running model's prediction for any row is $F_0$ plus whichever leaf value that row's features route it to. For a student landing in the $-2.66$ leaf:

For a student landing in the $1.6$ leaf:

Both track the training labels well (the first student's residual pushed hard toward 0, the second's toward 1), and this back-and-forth, add in log-odds space, convert to probability to compute the next stage's residuals, repeats every stage. Stage 2 fits a new tree to the new residuals ($y_i$ minus the updated probabilities), computes a new log-odds value per leaf the same way, and adds them (again in log-odds) to get $F_2$. Exactly as in the regression case, a learning rate (typically 0.1) scales each tree's contribution before it's added, so that no single stage causes an outsized jump in the running prediction.

5. A Full Prediction, End to End

For a new student with CGPA 7.2 and IQ 100, prediction means routing them through every tree and summing log-odds contributions. Say tree 1 (the $F_0$ baseline) contributes $0.51$, the first decision tree routes this student to a leaf worth $-2.66$, and the second decision tree routes them to a leaf worth $1.56$:

$0.35$ is below the 0.5 threshold, so the model predicts "not placed", the same mechanical last step as regression (sum every stage's contribution, only here a sigmoid stands between the sum and the final answer, instead of the sum being the final answer directly).

6. Seeing the Decision Boundary Sharpen, Stage by Stage

The worked example above is deliberately tiny to keep the arithmetic checkable by hand. To see the shape of what's happening, here's the same procedure (log-odds baseline, log-loss pseudo-residuals, a shallow tree per stage, the leaf log-odds-value formula from Section 3) run on a real, non-linearly-separable two-class dataset (sklearn.datasets.make_moons). The two input features ($X_1$, $X_2$) sit on the two horizontal axes, and the class itself becomes the third axis: every training point is plotted at $z=0$ (red) or $z=1$ (blue), and the model's predicted probability is drawn as a surface across the same three axes, so "the model is right" literally means "the surface passes through the points". Drag the slider below to step through the stages yourself:

Stage 0 is a single flat plane, exactly as Section 1 predicted: a constant log-odds baseline has nothing to do with any individual row's features, so its probability surface has no shape at all, it's the same height everywhere, sitting roughly halfway between the red points on the floor and the blue points near the ceiling. Each added tree bends that plane further, carving it into the blocky, stepped shape you'd expect from stacking shallow trees (their leaves are literally axis-aligned regions), and by the final stage the surface has folded itself down to hug the red points near $z=0$ and up to hug the blue points near $z=1$. This is the exact same mechanism as the toy 8-row example, applied enough times that "the surface reaches for each point's actual class" stops being an abstract description and becomes something you can drag through and watch happen.

7. The Per-Stage Loop, In One Line

Every stage after the baseline repeats the same back-and-forth between log-odds and probability. Written as a chain:

log-odds ($F$) → sigmoid → probability ($p$) → $y - p$ → pseudo-residual → fit regression tree → per-leaf log-odds value → $F \mathrel{+}= \eta \cdot \text{log-odds value}$ → back to log-odds → repeat

Two conversions happen every single stage: log-odds becomes a probability (to compute the residual), and a tree's residual-fitted output becomes a log-odds value (via the leaf formula from Section 3, to update $F$). Losing track of which quantity is in which space is the easiest way to get lost in this algorithm; this chain is the whole loop in one place.

8. What to Take Away

• Classification gradient boosting is the same four-step recipe as regression (baseline, pseudo-residual, fit a tree, update), only the loss function changes, from squared error to log loss.
• The baseline becomes log-odds ($\ln(\text{count}_1/\text{count}_0)$) instead of a mean, since labels can't be meaningfully averaged and then added to.
• The pseudo-residual stays actual-minus-predicted, just computed on probabilities rather than raw values, since log-odds and 0/1 labels can't be subtracted directly.
• Each tree's leaves need their own conversion formula ($\sum \text{residual} / \sum p(1-p)$) to turn a probability-space residual back into a log-odds-space contribution, unlike regression where a leaf's raw averaged output was already usable.
• Every stage still alternates between log-odds (for adding contributions together) and probability (for computing the next residual), converting back and forth with the sigmoid and its inverse.
• A learning rate still scales every stage's contribution, for the same overfitting-control reason as in regression.