Machine Learning Algorithms
A deep dive into how core machine learning algorithms actually work, from linear regression to ensemble methods.
Linear Regression
Post 1•Ch.1: Simple Linear Regression - The Intuition
Kicking off a new series on core ML algorithms with the first one everyone learns: linear regression, the placement dataset it's usually taught on, and what slope and intercept actually mean.
- Post 2•
Ch.2: Simple Linear Regression - The Math
Deriving the closed-form OLS solution for simple linear regression by hand, coding a linear regression class from scratch, and comparing it to sklearn on the same placement dataset.
- Post 3•
Ch.3: Regression Metrics - MAE, MSE, RMSE, R² and Adjusted R²
How to tell whether a regression line is any good: the five standard metrics, what each one measures geometrically and mathematically, why each one exists, and where each one falls short.
- Post 4•
Ch.4: Assumptions of Linear Regression
The five core assumptions of ordinary least squares linear regression, why each one matters, what happens when it is violated, and how to verify them using Python.
- Post 5•
Ch.5: Multiple Linear Regression - The Intuition
Extending simple linear regression to more than one input column: fitting a plane through 3D data instead of a line, and what the hyperplane equation looks like in n dimensions.
- Post 6•
Ch.6: Multiple Linear Regression - The Math
Deriving the closed-form normal equation for multiple linear regression in matrix form, coding a custom estimator from scratch, and validating it against sklearn on the diabetes dataset.
- Post 7•
Ch.7: Gradient Descent - Intuition, Math, and Implementation
Deriving batch gradient descent for linear regression, building a custom GDRegressor class from scratch, animating live convergence, and comparing converged parameters against OLS.
- Post 8•
Ch.8: Batch Gradient Descent for Multiple Linear Regression
Extending gradient descent from one feature to many: deriving the vectorized coefficient update rule, implementing GDRegressor from scratch, and benchmarking it against sklearn's OLS on the diabetes dataset.
- Post 9•
Ch.9: Stochastic Gradient Descent
Why batch GD breaks on large data, how SGD fixes it by updating parameters one random row at a time, implementing SGDRegressor from scratch, and understanding the noise-vs-speed tradeoff with a real convergence comparison.
- Post 10•
Ch.10: Mini-Batch Gradient Descent
The middle ground between batch GD and SGD: update on small random batches of rows instead of one row or the whole dataset, implement MBGDRegressor from scratch, and see why it converges as fast as SGD but far more smoothly.
- Post 11•
Ch.11: Polynomial Regression
Why plain linear regression fails on curved datasets, how feature transformation turns non-linear problems into linear ones with PolynomialFeatures, navigating the degree trade-off, and extending to multiple features.
- Post 12•
Ch.12: Bias-Variance Tradeoff
What bias and variance actually mean, using three models fit to the same data: one too simple, one just right, one that memorizes noise, and why the goal is always low bias and low variance together.
- Post 13•
Ch.13: Ridge Regression: Regularization and the L2 Penalty
Understanding regularization in linear models: bagging, boosting, and regularization as the three tools against overfitting, why high coefficient values signal overfitting, and how the L2 penalty term reshapes the loss function to fix it.
- Post 14•
Ch.14: Ridge Regression: The Math and the Code
Deriving the closed-form solution to Ridge Regression, first for a single feature and then in full matrix form for n dimensions, and implementing both from scratch to match Scikit-Learn's Ridge class coefficient for coefficient.
- Post 15•
Ch.15: Ridge Regression via Gradient Descent
Deriving Ridge's gradient descent update from the same matrix loss used for the closed-form solution, then implementing it from scratch to match Scikit-Learn's Ridge and SGDRegressor on the diabetes dataset.
- Post 16•
Ch.16: Five Things to Know About Ridge Regression
Five interview-style intuitions about Ridge Regression: how coefficients shrink with alpha, why larger coefficients shrink faster, the bias-variance tradeoff, how alpha reshapes the loss surface, and why it's called Ridge.
- Post 17•
Ch.17: Lasso Regression: The Intuition
Introducing Lasso (L1) Regression alongside Ridge: how the L1 penalty drives coefficients to exactly zero, performs automatic feature selection, and reshapes the loss function into a V-shaped kink that Ridge's smooth parabola never has.
- Post 18•
Ch.18: Why Lasso Zeroes Out Coefficients (And Ridge Never Does)
Deriving the single-feature closed-form solution for Lasso Regression and showing exactly why lambda subtracts from the numerator instead of adding to the denominator, the algebraic reason Lasso coefficients can hit exactly zero and Ridge coefficients never do.
- Post 19•
Ch.19: Elastic Net Regression
Elastic Net combines Ridge's L2 penalty and Lasso's L1 penalty into one loss function, controlled by two hyperparameters (or Scikit-Learn's alpha and l1_ratio), for datasets where it isn't obvious in advance whether every feature matters or only a few do.
Logistic Regression
Post 20•Ch.20: Logistic Regression: The Perceptron Trick
Building the geometric intuition behind Logistic Regression through the Perceptron Trick: representing a decision boundary as a weight vector, and deriving a single unified update rule that nudges a misclassified line toward the point it got wrong.
- Post 21•
Ch.21: Coding the Perceptron Trick (And Why It's Not Enough)
Implementing the Perceptron Trick from scratch in numpy, then comparing its decision boundary against Scikit-Learn's actual LogisticRegression to expose the real limitation: the perceptron stops the moment every point is classified correctly, while Logistic Regression keeps improving toward a better-margin line.
- Post 22•
Ch.22: Swapping the Step Function for Sigmoid
Replacing the perceptron trick's step function with sigmoid so correctly classified points keep pushing the line, closing most of the margin gap with real Logistic Regression, and setting up why one problem still remains.
- Post 23•
Ch.23: Deriving Logistic Regression's Loss Function
Using maximum likelihood to turn 'which of two lines is better' into an actual number: multiplying per-point probabilities, fixing the vanishing-product problem with logs, and arriving at binary cross-entropy.
- Post 24•
Ch.24: Logistic Regression's Gradient Descent, Derived and Coded
Rewriting log loss in matrix form, differentiating it with respect to the weight vector, and coding the resulting gradient descent update rule into a from-scratch Logistic Regression that matches scikit-learn's.
Classification Metrics
Post 25•Ch.25: Accuracy, the Confusion Matrix, and Why Accuracy Lies
Defining accuracy for classification, why there's no universal 'good enough' threshold, and why the confusion matrix (TP, FP, FN, TN) exists to fix accuracy's biggest blind spot on imbalanced datasets, illustrated with a terrorist-detection example.
- Post 26•
Ch.26: Precision, Recall, and F1 Score
Fixing accuracy's blind spot by scoring the two error types separately: precision for when false positives are the dangerous mistake, recall for when false negatives are, F1 as their harmonic mean, and how both generalize to multi-class problems via macro and weighted averages.
- Post 27•
Ch.27: The ROC Curve and AUC
Why classifiers output probabilities, not labels; how the classification threshold trades false positives against false negatives; and how the ROC curve and AUC evaluate a model across every possible threshold at once instead of just one.
Logistic Regression
Post 28•Ch.28: Softmax Regression for Multi-Class Classification
Extending Logistic Regression past two classes: the softmax function, why training one binary model per class is the intuitive but slow approach, and how a single modified loss function trains one joint multinomial model instead.
- Post 29•
Ch.29: Fitting Non-Linear Data With Polynomial Logistic Regression
Logistic Regression always draws a straight decision boundary, which fails on non-linearly separable data. Borrowing the same trick Polynomial Regression used for curves, PolynomialFeatures lets Logistic Regression fit curved boundaries too, at the cost of a new degree hyperparameter to tune.
- Post 30•
Ch.30: Logistic Regression's Hyperparameters
A practical tour of scikit-learn's LogisticRegression knobs, penalty, C, solver, max_iter, multi_class, class_weight, and how each one maps back to a concept already covered in this series.
K-Nearest Neighbors
Post 31•Ch.31: K-Nearest Neighbors, Intuition and Failure Cases
Introducing KNN: classifying a point by majority vote among its K closest training points, why it needs no real training phase, how the decision surface visualizes what a classifier actually learned, and the six situations where KNN quietly falls apart.
Support Vector Machines
Post 32•Ch.32: Support Vector Machines, the Maximum Margin Intuition
Introducing SVM: why two lines that both separate the training data perfectly aren't equally good, what 'margin' means geometrically, why the closest points (the support vectors) are the only ones that matter, and where SVM extends past a straight line.
- Post 33•
Ch.33: The Math Behind Hard Margin SVM
Turning SVM's margin intuition into an actual optimization problem: the +1/-1 supporting hyperplanes, the yi(w.xi+b) >= 1 constraint that unifies both classes into one rule, the distance-formula derivation of margin width 2/||w||, and why maximizing that margin becomes minimizing (1/2)||w||^2.
- Post 34•
Ch.34: Soft Margin SVM and the Hinge Loss
Hard margin SVM breaks the moment data isn't perfectly separable. Soft margin SVM fixes this with slack variables, one per point, that measure how badly a constraint is violated, folded into the objective through a C-weighted penalty. Rewriting that penalty as a per-point loss produces the hinge loss, SVM's answer to logistic regression's log loss.
- Post 35•
Ch.35: The Kernel Trick, Geometrically
SVM's margin-maximizing line only works if a line can separate the classes at all. The kernel trick sidesteps that limit by lifting data into a higher dimension where it becomes linearly separable, illustrated with two concrete examples: a 1D red-green-red line lifted to 2D by squaring, and 2D concentric circles lifted to 3D by an RBF-style transform.
- Post 36•
Ch.36: The Kernel Trick, In Code
sklearn's SVC turns the geometric lift from Ch.35 into one keyword argument. A linear SVM fails on concentric circles, manually engineering z = x1² + x2² fixes it, and kernel="rbf" gets the same result without ever building that feature by hand, which is the actual trick in kernel trick.
Decision Trees
Post 37•Ch.37: Decision Trees, the Core Intuition
How a decision tree is really just a nested series of if/else questions asked on the data, why it flips the usual root-at-bottom tree image upside down, what happens once features are numeric instead of categorical, and the terminology and trade-offs that come with the idea.
- Post 38•
Ch.38: Entropy, Information Gain, and Gini Impurity
The missing piece from the last post: how a decision tree actually decides which feature to split on. Deriving entropy as a measure of disorder, computing it by hand on the Play Tennis dataset, using information gain to pick the best split, meeting Gini impurity as CART's faster alternative, and extending both to numeric features via threshold search.
- Post 39•
Ch.39: Decision Tree Hyperparameters
How max_depth, min_samples_split, min_samples_leaf, criterion, splitter, max_features, max_leaf_nodes, and min_impurity_decrease each pull an unconstrained decision tree back from overfitting, with real sklearn decision boundaries showing the underfit-to-overfit trade-off for each one.