Week 5: The Supervised Learning Workflow

1. Types of Learning

"Machine learning" splits into three broad problem shapes, distinguished by what feedback the model gets while learning:

• Supervised learning — every training example comes with a known correct answer (a label); the model learns to predict that label for new, unseen inputs. This week, and most of Weeks 6–20, live here.
• Unsupervised learning — there are no labels; the model finds structure in the data on its own (clusters, reduced dimensions). You'll dig into this properly in Week 7.
• Reinforcement learning — an agent takes actions in an environment and learns from a reward signal rather than a labeled example for every step. You'll meet its LLM-specific form, RLHF, in Week 20.

Within supervised learning, there are two further shapes worth naming now because you'll pick between them constantly: classification (predicting a category — spam or not spam) and regression (predicting a continuous number — a house price). The workflow in this lesson applies to both identically.

2. Train/Validation/Test Splits

Splitting data into just "train" and "test" feels sufficient, but it quietly breaks down the moment you tune anything — a hyperparameter, a feature set, a threshold — because every time you check performance on the "test" set to decide what to change next, you leak a little bit of that test set's information into your decisions. Three sets solve this:

• Training set — what the model actually learns its parameters from
• Validation set — used repeatedly, during development, to compare models and tune choices
• Test set — touched exactly once, at the very end, to report a final, honest number

from sklearn.model_selection import train_test_split

# First split off the test set -- and don't touch it again until the end
X_temp, X_test, y_temp, y_test = train_test_split(
    X, y, test_size=0.2, random_state=42, stratify=y
)

# Then split the remainder into train and validation
X_train, X_val, y_train, y_val = train_test_split(
    X_temp, y_temp, test_size=0.25, random_state=42, stratify=y_temp
)
# Overall: 60% train, 20% validation, 20% test

stratify=y ensures each split preserves the original class proportions — a 90/10 imbalanced classification problem should stay roughly 90/10 in every split, or a small validation set could end up with almost no examples of the minority class by pure chance. You'll rely on stratify again for cross-validation in Week 6.

3. The Bias–Variance Tradeoff

Every model's prediction error can be decomposed into two competing sources, plus irreducible noise:

• Bias — error from a model that's too simple to capture the real pattern (it makes systematic mistakes no matter how much data you give it)
• Variance — error from a model that's too sensitive to the specific training data it happened to see (it would give a very different answer if trained on a slightly different sample)

These trade off against each other as model complexity changes: a very simple model (like a straight line fit to curved data) has high bias and low variance — it's consistently wrong in the same way. A very complex model (like a high-degree polynomial that snakes through every training point) has low bias and high variance — it fits its training data almost perfectly but swings wildly on new data. The goal is the sweet spot between them, not the elimination of either.

4. Overfitting & Underfitting, via Learning Curves

A learning curve plots training and validation error (or accuracy) against the amount of training data used. Its shape is one of the most useful diagnostic tools in this entire course, because it turns "is my model good?" into a picture you can read at a glance.

from sklearn.model_selection import learning_curve
from sklearn.linear_model import LogisticRegression
import numpy as np

train_sizes, train_scores, val_scores = learning_curve(
    LogisticRegression(), X_train, y_train,
    train_sizes=np.linspace(0.1, 1.0, 10), cv=5,
)

train_mean = train_scores.mean(axis=1)
val_mean = val_scores.mean(axis=1)

# Underfitting signature: both curves plateau LOW and close together
# Overfitting signature: training curve stays HIGH, validation curve stays notably lower

Reading the two failure modes:

• Underfitting (high bias) — training and validation scores are both poor and close together. Fixes: use a more expressive model, add features (revisit Week 4), reduce regularization.
• Overfitting (high variance) — training score is high, but validation score is meaningfully lower and the gap doesn't close as data grows. Fixes: collect more training data, simplify the model, add regularization (you'll formalize this in Week 12's dropout and weight decay), or remove noisy features.

Notice both fixes are opposites — which is exactly why diagnosing which one you have, before reaching for a fix, is the actual skill. Trying to fix underfitting with "more regularization" makes it worse, not better.

5. Baselines & Your First Model

Before judging any model as "good," you need something trivial to compare it against — a baseline. Without one, 82% accuracy sounds impressive right up until you learn that always predicting the majority class also gets 80%.

from sklearn.dummy import DummyClassifier
from sklearn.linear_model import LogisticRegression
from sklearn.metrics import accuracy_score

# Baseline: always predict the most frequent class
baseline = DummyClassifier(strategy="most_frequent")
baseline.fit(X_train, y_train)
baseline_acc = accuracy_score(y_val, baseline.predict(X_val))

# A real first model
model = LogisticRegression(max_iter=1000)
model.fit(X_train, y_train)
model_acc = accuracy_score(y_val, model.predict(X_val))

print(f"Baseline: {baseline_acc:.3f}  |  Model: {model_acc:.3f}")

If your "real" model can't clear the baseline by a meaningful margin, that's a signal to investigate before doing anything else — check for a data problem (Week 4's leakage and cleaning), not necessarily a more complex algorithm. You'll compare against this exact baseline pattern again when choosing metrics in Week 6.

6. Hands-on Exercise

from sklearn.datasets import make_classification
from sklearn.tree import DecisionTreeClassifier
from sklearn.linear_model import LogisticRegression

X, y = make_classification(
    n_samples=300, n_features=20, n_informative=5,
    n_redundant=10, random_state=42,
)

# Candidate A: an unrestricted decision tree
model_a = DecisionTreeClassifier(random_state=42)

# Candidate B: a regularized logistic regression
model_b = LogisticRegression(max_iter=1000)

7. Knowledge Check

Four quick questions. Expand each to check your answer.