Ch.6: Instance-Based vs. Model-Based Learning

Inspired by: YouTube

In the previous posts, we categorized machine learning based on the amount of supervision required (Supervised vs. Unsupervised) and how the model updates in production (Batch vs. Online Learning). Today, we are looking at a third way to classify machine learning algorithms: How the model actually learns and generalizes from the data.

When human beings learn, we generally fall into one of two categories. Sometimes, we simply memorize or rote-learn the material because we just need to pass an exam. Other times, we focus on understanding the underlying concepts and principles (like deriving a physics equation).

Machine learning algorithms learn in the exact same two ways:

1. Instance-Based Learning: Memorizing the data.
2. Model-Based Learning: Understanding the underlying concepts to build a generalized rule.

The main goal of this post is not to dive deep into the complex mathematics of these algorithms, but to ensure that whenever you study a new algorithm in the future, you can instantly identify which of these two learning styles it uses.

Instance-Based Learning

Instance-Based Learning is the simplest form of machine learning. The core philosophy here is pure memorization.

The Placement Prediction Example

Imagine you have a dataset of college students with three columns:

• IQ: The student's intelligence quotient.
• CGPA: The student's academic grade.
• Placement: Whether they got a job (Yes or No).

If you plot this on a 2D graph, you might put IQ on the X-axis and CGPA on the Y-axis. You would have red dots representing students who got placed, and blue dots representing students who did not.

When you feed this training data into an Instance-Based algorithm, it does not try to find any complex mathematical patterns. It simply stores the entire dataset in its memory as-is. It essentially does nothing during the training phase.

How it Predicts

Now, a new student arrives. Their CGPA is 7.5 and their IQ is 103. You plot this new query point on the graph. How does the algorithm decide if this student will get placed?

Since the algorithm hasn't learned a "rule", it relies on a simple principle: Similarity. It calculates the physical distance between this new student and every other student in its stored memory. It then looks at the closest neighbors.

If you tell the algorithm to look at the 3 nearest neighbors, and 2 out of those 3 neighbors are red dots (placed), the algorithm will predict that this new student will also get placed.

This specific logic is the foundation of the K-Nearest Neighbors (KNN) algorithm.

The "Lazy Learner"

Because the model just sits quietly and stores data without actually doing any computational learning upfront, Instance-Based algorithms are often called "Lazy Learners". They only wake up and do the heavy lifting (calculating distances) when you explicitly ask them to make a prediction on a new point.

Model-Based Learning

Model-Based Learning takes the opposite approach. It wants to find the underlying principle in the data.

Using the exact same IQ and CGPA placement dataset, a Model-Based algorithm will look at all the red and blue dots and attempt to draw a Decision Boundary (a mathematical line or curve) that separates the placed students from the unplaced students.

This boundary is the "Model". It is represented by a mathematical equation defined by a set of parameters (like slopes, intercepts, or weights).

How it Predicts

The biggest advantage of this approach happens after the training phase is complete. Once the algorithm has successfully figured out the mathematical equation for the decision boundary, it no longer needs the training data. You can completely delete the original dataset.

When the new student (CGPA 7.5, IQ 103) arrives, the algorithm simply plugs those numbers into its mathematical equation. If the result falls on the left side of the boundary, they get placed. If it falls on the right, they don't.

Most of the popular machine learning algorithms you will encounter are Model-Based. Examples include:

• Linear Regression
• Logistic Regression
• Support Vector Machines (SVMs)
• Artificial Neural Networks

Key Differences Summary

Here is a breakdown of how the two approaches compare across critical technical dimensions:

1. Data Preparation

Both approaches require the exact same amount of initial data preprocessing. You must handle missing values, remove outliers, and encode strings into numbers regardless of the learning style.

2. Model Generation

• Model-Based: Generates an actual mathematical model defined by parameters (e.g., weights and biases in a neural network). The goal is to discover a generalized rule before seeing any new queries.
• Instance-Based: Does not generate a model or any parameters. There is no generalization beforehand. It only attempts to generalize locally when a specific query point is provided.

3. Training Phase

• Model-Based: The training phase is computationally intensive as the algorithm iteratively figures out the perfect decision boundary.
• Instance-Based: The training phase is almost instantaneous. It just stores the raw data points in memory and defers all the computational work to the prediction phase.

4. Storage Requirements

• Model-Based: Extremely storage efficient. Once trained, you discard the gigabytes of training data and only save a tiny file containing the mathematical parameters.
• Instance-Based: Highly storage intensive. Because the entire prediction logic relies on measuring distances to past examples, you must keep the entire training dataset in memory forever. If you have 1GB of training data, your model requires 1GB of storage.