Ch.34: Soft Margin SVM and the Hinge Loss

Inspired by: YouTube

Last post ended on a warning: hard margin SVM's constraint $y_i(w \cdot x_i + b) \geq 1$ has to hold for every point, and if the classes overlap even slightly, no $w, b$ satisfies it. There's no fallback, the optimization problem simply has no solution. Soft margin SVM is the fix, it lets points violate the constraint, at a cost the objective has to pay for.

1. Give Every Point Some Slack

For each training point, introduce a variable $\xi_i \geq 0$ (the Greek letter xi, pronounced "chi" or often just "slack $i$") and relax the hard constraint to:

$y_i(w \cdot x_i + b) \geq 1 - \xi_i$

Read $\xi_i$ as how far point $i$ is allowed to fall short of the margin requirement:

• $\xi_i = 0$: the point satisfies the original hard-margin constraint, sitting on the correct side of its supporting plane, exactly like before.
• $0 < \xi_i \leq 1$: the point is inside the margin, correctly classified but too close to the boundary, it invades the safety strip that hard margin insisted on keeping empty.
• $\xi_i > 1$: the point has crossed all the way to the wrong side of the decision boundary $w \cdot x + b = 0$, it's actually misclassified.

Every point gets its own $\xi_i$, and most of them stay at $0$, slack is only "spent" on the points that actually need it to keep the problem solvable.

2. Paying for Slack in the Objective

Slack can't be free. Set $\xi_i$ to a huge number for every point and the constraint is trivially satisfiable no matter how bad $w, b$ are, that's not a margin classifier anymore, it's nothing. The fix is to add a penalty for total slack directly into the thing being minimized:

$\min_{w, b, \xi} \ \frac{1}{2}\|w\|^2 + C\sum_{i=1}^n \xi_i \quad \text{subject to} \quad y_i(w \cdot x_i + b) \geq 1 - \xi_i, \ \ \xi_i \geq 0 \ \ \forall i$

Two competing terms now sit in one objective:

• $\frac{1}{2}\|w\|^2$ still wants a small $\|w\|$, which from ch33 means a wide margin.
• $C\sum \xi_i$ wants total slack small, meaning few violations and margins that hug the data tightly.

$C$ is the hyperparameter that decides how much the second term matters relative to the first, it's the knob that turns hard margin ($C \to \infty$, no violation tolerated) into a fully relaxed classifier ($C \to 0$, margin can grow as wide as it likes regardless of how many points end up inside it).

• Small $C$: violations are cheap, the optimizer happily lets $\xi_i$ grow to buy a wider margin. More bias, less variance, closer to underfitting if pushed too far.
• Large $C$: violations are expensive, the optimizer shrinks $\xi_i$ even if that means a narrower margin that hugs the training points closely. Less bias, more variance, closer to overfitting if pushed too far.

This is exactly the same bias-variance dial seen with logistic regression's regularization strength, just written as $C$ instead of $\lambda$, and inverted: scikit-learn's SVC(C=...) uses large $C$ for less regularization, the opposite convention from $\lambda$.

3. Turning Slack Into a Loss Function

The constraint $y_i(w \cdot x_i + b) \geq 1 - \xi_i$ combined with $\xi_i \geq 0$ pins down the smallest $\xi_i$ can possibly be for a given point: either $0$ (if the constraint is already met) or exactly $1 - y_i(w \cdot x_i + b)$ (if it isn't). That's precisely what $\max(0, \cdot)$ computes:

$\xi_i = \max\big(0, \ 1 - y_i(w \cdot x_i + b)\big)$

Substituting this back removes $\xi_i$ and the constraints entirely, collapsing soft margin SVM into a single unconstrained objective over $w, b$:

$\min_{w, b} \ \frac{1}{2}\|w\|^2 + C\sum_{i=1}^n \max\big(0, \ 1 - y_i(w \cdot x_i + b)\big)$

The term inside the sum, $\max(0, 1 - y_i(w \cdot x_i + b))$, is the hinge loss for a single point. It's zero once a point clears the margin by at least $1$, and grows linearly the moment it doesn't, which is exactly the behavior $\xi_i$ was already tracking, this is just that same quantity written as a per-point loss function instead of a constraint.

4. Hinge Loss vs. Logistic Loss

Both hinge loss and logistic regression's log loss punish a point more the further it sits on the wrong side of the boundary, and both go to zero as the point moves confidently onto the correct side. The difference is where "confident enough" kicks in:

• Log loss never actually reaches zero, it keeps decreasing (however slightly) for any finite score, always rewarding more confidence.
• Hinge loss hits exactly zero once $y_i(w \cdot x_i + b) \geq 1$ and stays there, a point that already clears the margin contributes nothing further to the objective, no matter how much more confidently correct it becomes.

That flat zero region is what makes SVM a margin classifier rather than a probability classifier: it only cares about pushing points out of the margin, not about squeezing every last bit of confidence out of points that are already safely classified. Once that's true for a point, hinge loss stops paying attention to it entirely.

5. Where This Leaves SVM

Soft margin SVM, in either form, the constrained quadratic program with slack variables or the unconstrained hinge-loss objective, is still a linear classifier: $w \cdot x + b$ is a straight line (or hyperplane) no matter how $C$ is tuned. Overlapping classes that soft margin merely tolerates are one problem; classes that no straight line could ever separate, even loosely, are a different one entirely. That's what the kernel trick exists to solve, and it's the next post.