Machine Learning Algorithms
A deep dive into how core machine learning algorithms actually work, from linear regression to ensemble methods.
Linear Regression
Post 1•Ch.1: Simple Linear Regression - The Intuition
Kicking off a new series on core ML algorithms with the first one everyone learns: linear regression, the placement dataset it's usually taught on, and what slope and intercept actually mean.
- Post 2•
Ch.2: Simple Linear Regression - The Math
Deriving the closed-form OLS solution for simple linear regression by hand, coding a linear regression class from scratch, and comparing it to sklearn on the same placement dataset.
- Post 3•
Ch.3: Regression Metrics - MAE, MSE, RMSE, R² and Adjusted R²
How to tell whether a regression line is any good: the five standard metrics, what each one measures geometrically and mathematically, why each one exists, and where each one falls short.
- Post 4•
Ch.4: Assumptions of Linear Regression
The five core assumptions of ordinary least squares linear regression, why each one matters, what happens when it is violated, and how to verify them using Python.
- Post 5•
Ch.5: Multiple Linear Regression - The Intuition
Extending simple linear regression to more than one input column: fitting a plane through 3D data instead of a line, and what the hyperplane equation looks like in n dimensions.
- Post 6•
Ch.6: Multiple Linear Regression - The Math
Deriving the closed-form normal equation for multiple linear regression in matrix form, coding a custom estimator from scratch, and validating it against sklearn on the diabetes dataset.
- Post 7•
Ch.7: Gradient Descent - Intuition, Math, and Implementation
Deriving batch gradient descent for linear regression, building a custom GDRegressor class from scratch, animating live convergence, and comparing converged parameters against OLS.
- Post 8•
Ch.8: Batch Gradient Descent for Multiple Linear Regression
Extending gradient descent from one feature to many: deriving the vectorized coefficient update rule, implementing GDRegressor from scratch, and benchmarking it against sklearn's OLS on the diabetes dataset.
- Post 9•
Ch.9: Stochastic Gradient Descent
Why batch GD breaks on large data, how SGD fixes it by updating parameters one random row at a time, implementing SGDRegressor from scratch, and understanding the noise-vs-speed tradeoff with a real convergence comparison.
- Post 10•
Ch.10: Mini-Batch Gradient Descent
The middle ground between batch GD and SGD: update on small random batches of rows instead of one row or the whole dataset, implement MBGDRegressor from scratch, and see why it converges as fast as SGD but far more smoothly.
- Post 11•
Ch.11: Polynomial Regression
Why plain linear regression fails on curved datasets, how feature transformation turns non-linear problems into linear ones with PolynomialFeatures, navigating the degree trade-off, and extending to multiple features.
- Post 12•
Ch.12: Bias-Variance Tradeoff
What bias and variance actually mean, using three models fit to the same data: one too simple, one just right, one that memorizes noise, and why the goal is always low bias and low variance together.
- Post 13•
Ch.13: Ridge Regression: Regularization and the L2 Penalty
Understanding regularization in linear models: bagging, boosting, and regularization as the three tools against overfitting, why high coefficient values signal overfitting, and how the L2 penalty term reshapes the loss function to fix it.
- Post 14•
Ch.14: Ridge Regression: The Math and the Code
Deriving the closed-form solution to Ridge Regression, first for a single feature and then in full matrix form for n dimensions, and implementing both from scratch to match Scikit-Learn's Ridge class coefficient for coefficient.
- Post 15•
Ch.15: Ridge Regression via Gradient Descent
Deriving Ridge's gradient descent update from the same matrix loss used for the closed-form solution, then implementing it from scratch to match Scikit-Learn's Ridge and SGDRegressor on the diabetes dataset.
- Post 16•
Ch.16: Five Things to Know About Ridge Regression
Five interview-style intuitions about Ridge Regression: how coefficients shrink with alpha, why larger coefficients shrink faster, the bias-variance tradeoff, how alpha reshapes the loss surface, and why it's called Ridge.
- Post 17•
Ch.17: Lasso Regression: The Intuition
Introducing Lasso (L1) Regression alongside Ridge: how the L1 penalty drives coefficients to exactly zero, performs automatic feature selection, and reshapes the loss function into a V-shaped kink that Ridge's smooth parabola never has.
- Post 18•
Ch.18: Why Lasso Zeroes Out Coefficients (And Ridge Never Does)
Deriving the single-feature closed-form solution for Lasso Regression and showing exactly why lambda subtracts from the numerator instead of adding to the denominator, the algebraic reason Lasso coefficients can hit exactly zero and Ridge coefficients never do.
- Post 19•
Ch.19: Elastic Net Regression
Elastic Net combines Ridge's L2 penalty and Lasso's L1 penalty into one loss function, controlled by two hyperparameters (or Scikit-Learn's α and l1_ratio), for datasets where it isn't obvious in advance whether every feature matters or only a few do.
Logistic Regression
Post 20•Ch.20: Logistic Regression: The Perceptron Trick
Building the geometric intuition behind Logistic Regression through the Perceptron Trick: representing a decision boundary as a weight vector, and deriving a single unified update rule that nudges a misclassified line toward the point it got wrong.
- Post 21•
Ch.21: Coding the Perceptron Trick (And Why It's Not Enough)
Implementing the Perceptron Trick from scratch in numpy, then comparing its decision boundary against Scikit-Learn's actual LogisticRegression to expose the real limitation: the perceptron stops the moment every point is classified correctly, while Logistic Regression keeps improving toward a better-margin line.
- Post 22•
Ch.22: Swapping the Step Function for Sigmoid
Replacing the perceptron trick's step function with sigmoid so correctly classified points keep pushing the line, closing most of the margin gap with real Logistic Regression, and setting up why one problem still remains.
- Post 23•
Ch.23: Deriving Logistic Regression's Loss Function
Using maximum likelihood to turn 'which of two lines is better' into an actual number: multiplying per-point probabilities, fixing the vanishing-product problem with logs, and arriving at binary cross-entropy.
- Post 24•
Ch.24: Logistic Regression's Gradient Descent, Derived and Coded
Rewriting log loss in matrix form, differentiating it with respect to the weight vector, and coding the resulting gradient descent update rule into a from-scratch Logistic Regression that matches scikit-learn's.
Classification Metrics
Post 25•Ch.25: Accuracy, the Confusion Matrix, and Why Accuracy Lies
Defining accuracy for classification, why there's no universal 'good enough' threshold, and why the confusion matrix (TP, FP, FN, TN) exists to fix accuracy's biggest blind spot on imbalanced datasets, illustrated with a terrorist-detection example.
- Post 26•
Ch.26: Precision, Recall, and F1 Score
Fixing accuracy's blind spot by scoring the two error types separately: precision for when false positives are the dangerous mistake, recall for when false negatives are, F1 as their harmonic mean, and how both generalize to multi-class problems via macro and weighted averages.
- Post 27•
Ch.27: The ROC Curve and AUC
Why classifiers output probabilities, not labels; how the classification threshold trades false positives against false negatives; and how the ROC curve and AUC evaluate a model across every possible threshold at once instead of just one.
Logistic Regression
Post 28•Ch.28: Softmax Regression for Multi-Class Classification
Extending Logistic Regression past two classes: the softmax function, why training one binary model per class is the intuitive but slow approach, and how a single modified loss function trains one joint multinomial model instead.
- Post 29•
Ch.29: Fitting Non-Linear Data With Polynomial Logistic Regression
Logistic Regression always draws a straight decision boundary, which fails on non-linearly separable data. Borrowing the same trick Polynomial Regression used for curves, PolynomialFeatures lets Logistic Regression fit curved boundaries too, at the cost of a new degree hyperparameter to tune.
- Post 30•
Ch.30: Logistic Regression's Hyperparameters
A practical tour of scikit-learn's LogisticRegression knobs, penalty, C, solver, max_iter, multi_class, class_weight, and how each one maps back to a concept already covered in this series.
K-Nearest Neighbors
Post 31•Ch.31: K-Nearest Neighbors, Intuition and Failure Cases
Introducing KNN: classifying a point by majority vote among its K closest training points, why it needs no real training phase, how the decision surface visualizes what a classifier actually learned, and the six situations where KNN quietly falls apart.
Support Vector Machines
Post 32•Ch.32: Support Vector Machines, the Maximum Margin Intuition
Introducing SVM: why two lines that both separate the training data perfectly aren't equally good, what 'margin' means geometrically, why the closest points (the support vectors) are the only ones that matter, and where SVM extends past a straight line.
- Post 33•
Ch.33: The Math Behind Hard Margin SVM
Turning SVM's margin intuition into an actual optimization problem: the +1/-1 supporting hyperplanes, the yi(w.xi+b) >= 1 constraint that unifies both classes into one rule, the distance-formula derivation of margin width 2/||w||, and why maximizing that margin becomes minimizing (1/2)||w||^2.
- Post 34•
Ch.34: Soft Margin SVM and the Hinge Loss
Hard margin SVM breaks the moment data isn't perfectly separable. Soft margin SVM fixes this with slack variables, one per point, that measure how badly a constraint is violated, folded into the objective through a C-weighted penalty. Rewriting that penalty as a per-point loss produces the hinge loss, SVM's answer to logistic regression's log loss.
- Post 35•
Ch.35: The Kernel Trick, Geometrically
SVM's margin-maximizing line only works if a line can separate the classes at all. The kernel trick sidesteps that limit by lifting data into a higher dimension where it becomes linearly separable, illustrated with two concrete examples: a 1D red-green-red line lifted to 2D by squaring, and 2D concentric circles lifted to 3D by an RBF-style transform.
- Post 36•
Ch.36: The Kernel Trick, In Code
sklearn's SVC turns the geometric lift from Ch.35 into one keyword argument. A linear SVM fails on concentric circles, manually engineering z = x1² + x2² fixes it, and kernel="rbf" gets the same result without ever building that feature by hand, which is the actual trick in kernel trick.
Decision Trees
Post 37•Ch.37: Decision Trees, the Core Intuition
How a decision tree is really just a nested series of if/else questions asked on the data, why it flips the usual root-at-bottom tree image upside down, what happens once features are numeric instead of categorical, and the terminology and trade-offs that come with the idea.
- Post 38•
Ch.38: Entropy, Information Gain, and Gini Impurity
The missing piece from the last post: how a decision tree actually decides which feature to split on. Deriving entropy as a measure of disorder, computing it by hand on the Play Tennis dataset, using information gain to pick the best split, meeting Gini impurity as CART's faster alternative, and extending both to numeric features via threshold search.
- Post 39•
Ch.39: Decision Tree Hyperparameters
How max_depth, min_samples_split, min_samples_leaf, criterion, splitter, max_features, max_leaf_nodes, and min_impurity_decrease each pull an unconstrained decision tree back from overfitting, with real sklearn decision boundaries showing the underfit-to-overfit trade-off for each one.
- Post 40•
Ch.40: Decision Tree Regression
How a decision tree predicts a continuous number instead of a class: it partitions the feature space and predicts the mean of each region, chooses every split by minimising left-plus-right squared error, and produces a staircase instead of a line. Plus the hyperparameters that differ from the classifier, and feature_importances_ for feature selection.
Ensemble Learning
Post 41•Ch.41: Ensemble Learning, the Introduction
Why combining many mediocre models beats tuning one: the wisdom-of-the-crowd idea it rests on, the two ways to force base models to disagree, the four families (voting, stacking, bagging, boosting), a visual for why aggregation smooths out noise, and where ensembles fit in a real project.
- Post 42•
Ch.42: Voting Ensembles, the Intuition and the Proof
The simplest ensemble: train a few different models on the same data and take the majority vote. Why it works as a wisdom-of-the-crowd argument, the two conditions the base models must meet, and a full probability walk-through showing three independent 70%-accurate models combine into a 78.4%-accurate one, plus the mirror case where sub-coin-flip models make things worse.
- Post 43•
Ch.43: The Voting Classifier in Code
Building sklearn's VotingClassifier: hard voting (majority of predicted labels) vs soft voting (average of predicted probabilities), why soft usually edges out hard, weights as a hyperparameter with a small grid search, and using different hyperparameters of the same algorithm as an alternative source of ensemble diversity. Includes an interactive explorer with six toy datasets and five base estimators.
- Post 44•
Ch.44: The Voting Regressor in Code
Building sklearn's VotingRegressor: how it averages the predictions of different algorithms, weights as a searchable hyperparameter, and getting diversity from one algorithm at different hyperparameter settings instead of different algorithms. Includes an interactive explorer showing linear regression, SVR, and a decision tree voted together on four toy 1-D datasets.
- Post 45•
Ch.45: Bagging Ensemble, the Intuition
Why training the same algorithm on many random subsets of the data and voting the results cuts variance without touching bias: the bootstrap-then-aggregate mechanism, the row-budget argument for why the variance actually drops, an interactive explorer, and the four bagging variants (bagging, pasting, random subspaces, random patches).
- Post 46•
Ch.46: The Bagging Classifier in Code
Building sklearn's BaggingClassifier: how a fully-grown tree's overfitting gets smoothed by voting 500 of them, the four variants (bagging, pasting, random subspaces, random patches) as four settings of the same class, why bagging an SVM barely helps, the out-of-bag score as a free validation set, and tuning the whole thing with RandomizedSearchCV.
- Post 47•
Ch.47: The Bagging Regressor in Code
Building sklearn's BaggingRegressor: the same idea as the classifier but the aggregation step averages instead of votes, watching a single overfit tree smooth into a bagged curve, and benchmarking it against linear regression, KNN, and a lone decision tree on real housing data, tuned with GridSearchCV.
- Post 48•
Ch.48: Random Forest, the Name and the Intuition
Where 'random forest' gets its name, why it's bagging specialized to decision trees, and a from-scratch demo of the three sampling schemes (row, feature, combined) that make each tree in the forest different, with real trees trained and voted on real data.
- Post 49•
Ch.49: Why Random Forest Actually Works
The bias-variance argument for why random forest outperforms a single decision tree: fully-grown trees are low-bias, high-variance, and averaging many of them trained on different row subsets cancels the variance while barely touching the bias, shown on a real classification decision boundary and a real regression curve.
- Post 50•
Ch.50: Bagging vs Random Forest, the Actual Difference
The two real differences between bagging and random forest: the base-model restriction to decision trees, and the difference that actually matters, tree-level column sampling in bagging versus node-level column sampling in random forest, shown by inspecting which columns each real trained tree splits on.
- Post 51•
Ch.51: Random Forest Hyperparameters
A tour of random forest's actual tuning knobs (n_estimators, max_features, bootstrap, max_samples), what each one trades off, and real accuracy/decision-boundary curves showing why more trees plateaus and why max_samples has a sweet spot instead of a monotonic best value.
- Post 52•
Ch.52: Tuning Random Forest on a Real Dataset
Applying random forest to a heart disease classification problem end to end: out-of-the-box comparison against gradient boosting, SVC, and logistic regression, cross-validation, then GridSearchCV and RandomizedSearchCV to actually search the hyperparameter space instead of nudging one slider at a time.
- Post 53•
Ch.53: How Feature Importance Is Actually Calculated
What feature importance means, why it matters beyond feature selection (model interpretability), and the exact node-importance formula scikit-learn uses under the hood for decision trees and random forest, worked by hand on real trees and checked against feature_importances_.
- Post 54•
Ch.54: AdaBoost, Geometric Intuition
The first boosting algorithm: what a weak learner and a decision stump are, why AdaBoost uses +1/-1 labels instead of 0/1, and the stage-wise geometric picture of how up-sampled stumps combine into one strong classifier.
- Post 55•
Ch.55: AdaBoost, the α and Weight-Update Math
The actual formulas behind AdaBoost's stage-wise loop: why α equals 0.5*ln((1-error)/error), how misclassified rows get boosted and correct ones shrunk, and how cumulative-weight up-sampling builds the next stage's training set, worked through with the real source notebook's numbers.
- Post 56•
Ch.56: AdaBoost From Scratch in Code
The same three-stage AdaBoost run from the last post, this time as actual sklearn + pandas code: training decision stumps, computing error and α, updating and normalizing weights, and up-sampling into the next stage's dataset, run against the real source notebook and its decision-boundary plots, including a genuine bug in the notebook's third stage worth knowing about.
- Post 57•
Ch.57: Bagging vs Boosting, the Interview Answer
The three axes interviewers actually ask about: what kind of base model each family reaches for and why, parallel vs sequential training, and equal-vote democracy vs weighted votes, tying together everything already covered separately in the bagging and AdaBoost posts.
- Post 58•
Ch.58: Gradient Boosting, the Regression Intuition
How gradient boosting builds a regressor stage by stage: why the first model is just the mean, why every later stage is trained to predict the previous stage's leftover error instead of the target itself, and why a shared learning rate across every stage is what keeps the whole thing from memorizing the training data.
- Post 59•
Ch.59: Gradient Boosting Regression, the Actual Math
Deriving, not just asserting, why gradient boosting's first model is the mean, why the pseudo-residual works out to actual-minus-predicted, and why a fitted regression tree's own leaf values already are the formula's optimal output, all via one differentiation applied three times to squared-error loss, worked through on a real 3-row profit dataset.
- Post 60•
Ch.60: Gradient Boosting for Classification, the Intuition
The same four-step gradient boosting recipe from the regression posts, now run with log loss instead of squared error: why the starting model becomes log-odds instead of a mean, why the pseudo-residual is still actual-minus-predicted (just on probabilities), and why each leaf needs its own formula to convert a residual back into a log-odds value, worked through a small placement dataset and a real decision-boundary simulation.
- Post 61•
Ch.61: Introduction to XGBoost, Why It Took Over
XGBoost is not an algorithm, it's a library built on top of gradient boosting: what specifically made gradient boosting the right base to extend rather than AdaBoost or another algorithm, and a tour of the flexibility, speed, and performance engineering that turned it into the library every Kaggle winner reaches for.
- Post 62•
Ch.62: XGBoost for Regression, Worked by Hand
The same baseline-then-residual-trees recipe from gradient boosting, but XGBoost splits and scores its trees with a different formula: a similarity score built from squared summed residuals and a regularization term, worked through on a tiny 4-row CGPA-to-package dataset and checked against a real fitted XGBRegressor.
- Post 63•
Ch.63: XGBoost for Classification, Worked by Hand
XGBoost's classification recipe is gradient boosting's log-odds loop with XGBoost's similarity-score trees dropped in, and the only formula change from regression is that each row now contributes p(1-p) to the denominator instead of 1. Worked through by hand on a 5-row CGPA-to-placement dataset, stage by stage, and checked against a real fitted XGBClassifier.
- Post 64•
Ch.64: Deriving XGBoost's Formulas From Scratch
The similarity score and leaf output formulas used in the regression and classification walkthroughs were handed over without proof. This post derives both, plus the split gain formula, from XGBoost's actual objective function using a second-order Taylor expansion, the same route the original paper takes, verified with sympy along the way.
- Post 65•
Ch.65: Stacking and Blending, Learning How to Combine Models
Voting combines base models with a fixed rule, mean or majority vote. Stacking replaces that rule with a trained meta-model instead, which immediately creates an overfitting trap, and two different fixes for it, a single holdout split (blending) and K-fold cross-validation (stacking proper), worked through with diagrams and a real scikit-learn StackingClassifier on the heart disease dataset.
Clustering
Post 66•Ch.66: K-Means Clustering, the Geometric Intuition
K-means groups unlabeled points by repeating two steps until they stop changing anything: assign every point to its nearest centroid, then move each centroid to the mean of the points assigned to it. Worked through on a CGPA/IQ placement dataset with real generated diagrams for every round, plus the elbow method for picking K from a WCSS curve.
- Post 67•
Ch.67: K-Means in Python, Clustering Students by CGPA and IQ
Running K-means end to end with scikit-learn on a 200-student CGPA/IQ dataset: picking K from a real elbow curve built from km.inertia_, training the final model, plotting each cluster with boolean-mask indexing, and turning the clusters into an actual plan. Then the exact same code on 3-D blob data, with an interactive 3D plot, plus a feature-scaling quirk the clean-looking result hides.
- Post 68•
Ch.68: Agglomerative Hierarchical Clustering
K-means assumes clusters are roughly spherical and fails on concentric circles, crescents, elongated clusters, and noisy data. Agglomerative clustering fixes this by repeatedly merging the two closest clusters, starting from every point as its own cluster, recording the merge order as a dendrogram, and offering four different ways (single, complete, average, Ward) to define what counts as the closest pair. Worked through with real sklearn code on the classic 200-row mall customer dataset.
- Post 69•
Ch.69: DBSCAN Clustering
K-means forces every cluster to be a round blob built around a centroid, and that assumption is exactly what breaks on concentric circles, crescents, and noisy data. DBSCAN clusters by density instead: a core point, a border point, and a noise point, defined entirely by how many neighbors fall within a radius epsilon. It finds clusters of any shape, doesn't need a cluster count up front, and naturally flags outliers, at the cost of a single global density assumption and no way to place a brand-new point once it's done.
Imbalanced Data
Post 70•Ch.70: Imbalanced Data, Undersampling, Oversampling, SMOTE and Beyond
When one class vastly outnumbers another, a model quietly learns to ignore the rare class while accuracy still looks excellent. Why that happens, why it matters in nearly every real ML application, and five fixes run on the same real dataset: random undersampling, random oversampling, SMOTE, balanced random forest, and cost-sensitive learning with class weights and a custom XGBoost loss. Includes what each technique actually changes, and what it doesn't.
Hyperparameter Tuning
Post 71•Ch.71: Hyperparameter Tuning with Optuna and Bayesian Optimization
Why grid search and random search waste most of their trials, how Bayesian search uses every past trial to decide where to look next, why gradient descent can't be used to tune hyperparameters, and Optuna's full workflow on real data: studies, trials, objective functions, samplers, visualizations, and tuning several algorithms in a single search.