Week 6: Model Evaluation & Hyperparameter Tuning

1. Precision, Recall & the Confusion Matrix

Accuracy — the fraction of predictions that are correct — is the metric everyone reaches for first, and the one that lies most often. On a dataset where 95% of emails are not spam, a model that predicts "not spam" for everything scores 95% accuracy while catching zero spam. The confusion matrix breaks predictions into four buckets that reveal exactly this kind of failure:

from sklearn.metrics import confusion_matrix, precision_score, recall_score, f1_score

y_true = [1, 0, 1, 1, 0, 1, 0, 0, 1, 0]
y_pred = [1, 0, 0, 1, 0, 1, 1, 0, 1, 0]

tn, fp, fn, tp = confusion_matrix(y_true, y_pred).ravel()
# tp: predicted spam, actually spam       fp: predicted spam, actually NOT spam
# fn: predicted not-spam, actually spam   tn: predicted not-spam, actually not spam

precision = precision_score(y_true, y_pred)   # tp / (tp + fp) -- of predicted spam, how much really is?
recall = recall_score(y_true, y_pred)         # tp / (tp + fn) -- of all real spam, how much did we catch?
f1 = f1_score(y_true, y_pred)                 # harmonic mean of precision and recall

Precision and recall trade off against each other, and which one matters more depends entirely on the cost of each mistake: a spam filter that's too aggressive (low precision) buries real emails in the spam folder — a false positive is expensive. A cancer-screening model that's too conservative (low recall) misses real cases — a false negative is far worse there than a false alarm. F1 is a single number balancing both, useful when you don't have a strong reason to prefer one over the other.

2. ROC Curves, AUC & When Accuracy Is Misleading

Most classifiers don't just output a class — they output a probability, which you then threshold (commonly at 0.5) to get a class label. A ROC curve plots true positive rate against false positive rate across every possible threshold, showing you the full tradeoff instead of the one your default threshold happens to land on. AUC (area under that curve) condenses it into one number: 0.5 means "no better than random guessing," 1.0 means perfect separation.

from sklearn.metrics import roc_auc_score, roc_curve
from sklearn.linear_model import LogisticRegression

model = LogisticRegression(max_iter=1000).fit(X_train, y_train)
probabilities = model.predict_proba(X_val)[:, 1]   # probability of the positive class

auc = roc_auc_score(y_val, probabilities)
fpr, tpr, thresholds = roc_curve(y_val, probabilities)

AUC's key property, unlike accuracy, is that it's insensitive to class imbalance and doesn't depend on picking a threshold — which makes it a much fairer way to compare two models on the same imbalanced dataset from Section 1's spam example. It doesn't replace precision/recall, though: AUC tells you how well-separated the classes are overall, while precision/recall tell you how a specific chosen threshold actually performs in practice.

3. Regression Metrics

Regression has its own family of metrics, and — just like precision vs. recall — the choice changes what kind of error you're penalizing most.

from sklearn.metrics import mean_absolute_error, mean_squared_error, r2_score
import numpy as np

y_true = np.array([100, 150, 200, 130, 175])
y_pred = np.array([110, 140, 220, 125, 300])   # one very bad prediction (300 vs 175)

mae = mean_absolute_error(y_true, y_pred)          # average absolute error
mse = mean_squared_error(y_true, y_pred)            # penalizes large errors more (squared)
rmse = np.sqrt(mse)                                  # same units as the target, easier to interpret
r2 = r2_score(y_true, y_pred)                        # fraction of variance explained (1.0 = perfect)

MAE treats every unit of error equally. MSE/RMSE square the error first, so one very bad prediction (like the 300 vs. 175 above) drags the metric up disproportionately — appropriate when large errors are genuinely much worse than small ones (a shipping ETA off by 10 days vs. off by 1 day), and misleading if a single outlier shouldn't dominate your judgment of the model. R² gives you a scale-free sense of "how much better than just guessing the mean" your model is — 0 means no better, 1 means perfect, and it can go negative if a model is worse than that trivial mean-guessing baseline from Week 5.

4. Cross-Validation

A single train/validation split gives you one estimate of performance — but that estimate depends partly on the luck of which rows happened to land in validation. k-fold cross-validation splits the training data into k equal parts (folds), trains k times using a different fold as validation each time, and averages the results — giving a far more stable estimate, and one standard deviation you can use to judge how much that estimate might vary.

from sklearn.model_selection import cross_val_score, StratifiedKFold
from sklearn.ensemble import RandomForestClassifier

cv = StratifiedKFold(n_splits=5, shuffle=True, random_state=42)
scores = cross_val_score(
    RandomForestClassifier(random_state=42), X_train, y_train,
    cv=cv, scoring="f1",
)

print(f"F1: {scores.mean():.3f} ± {scores.std():.3f}")

StratifiedKFold is the classification default for the same reason stratify=y mattered in Week 5 — it keeps class proportions consistent across every fold. Note that the test set from Week 5 still stays completely untouched here; cross-validation only ever operates within the training set, so your final held-out test score remains a single, honest number.

5. Hyperparameter Tuning: Grid Search & Random Search

A hyperparameter is a setting you choose before training (like a tree's max depth, or a regularization strength) rather than something the model learns. Grid search exhaustively tries every combination from a specified set of values; random search samples a fixed number of random combinations — often just as good, and far cheaper when you have many hyperparameters.

from sklearn.model_selection import GridSearchCV
from sklearn.ensemble import RandomForestClassifier

param_grid = {
    "n_estimators": [100, 200, 400],
    "max_depth": [None, 5, 10, 20],
    "min_samples_leaf": [1, 2, 4],
}

search = GridSearchCV(
    RandomForestClassifier(random_state=42),
    param_grid, cv=5, scoring="f1", n_jobs=-1,
)
search.fit(X_train, y_train)   # tunes using cross-validation on the TRAINING set only

print(search.best_params_, search.best_score_)
best_model = search.best_estimator_

Notice search.fit(X_train, y_train) — never X_val or X_test. GridSearchCV runs its own internal cross-validation on the training set to compare hyperparameter combinations, which means the validation and test sets from Week 5 stay completely uninvolved in the tuning decision itself. Tuning directly against your validation or test score, rather than through cross-validation, is the hyperparameter-tuning equivalent of the scaling leak from Week 4 — it makes your final reported number optimistic in a way that won't hold up on genuinely new data.

6. Hands-on Exercise

from sklearn.datasets import make_classification

X, y = make_classification(
    n_samples=2000, n_features=15, n_informative=6,
    weights=[0.95, 0.05],   # 95% class 0 ("legitimate"), 5% class 1 ("fraud")
    random_state=42,
)

7. Knowledge Check

Four quick questions. Expand each to check your answer.