Week 8: Classical ML Mini-Project

1. Framing a Real-World Problem as an ML Task

A stakeholder asks "can we predict which customers will churn?" That sentence is not yet an ML task — it's missing a target definition, a time horizon, and a notion of what counts as a usable prediction. Framing means turning a vague goal into specifics:

• Target — exactly what column, computed exactly how? ("Churn" by day 30? By day 90? Cancelled, or just inactive?)
• Unit of prediction — one row per customer? Per customer-month? Timing matters.
• Available features at prediction time — the target-leakage question from Week 4, asked explicitly up front rather than discovered later.
• Success criterion — which Week 6 metric reflects the real cost of a wrong prediction here, and what score would actually be "good enough to act on"?

Spending twenty minutes writing these four things down before opening a notebook is the single highest-leverage step in the entire project — most failed ML projects fail here, not at the modeling step.

2. Choosing and Justifying a Model Family

With the problem framed, pick a starting model family using what you already know from Weeks 5–7, and be ready to explain the choice, not just make it:

• Tabular data, mixed types, non-linear relationships expected? Tree ensembles (Week 9) are a strong, low-effort default.
• Need an interpretable coefficient per feature for stakeholders? Logistic/linear regression, even if a tree ensemble scores slightly higher.
• High-dimensional, correlated features? Consider PCA (Week 7) before modeling, or a model robust to correlated inputs.
• No labels at all? The task may actually be unsupervised (Week 7) or need labels created first.

Always start with the baseline from Week 5 (majority class, or mean prediction) before touching a "real" model — it's the number every later choice has to beat to be worth its added complexity.

3. The Full Pipeline: EDA → Preprocessing → Modeling → Evaluation

This is every previous week, assembled into one pass. Nothing here is new — the value is in doing all of it, in order, without skipping a step because "the data looks fine."

import pandas as pd
from sklearn.model_selection import train_test_split, GridSearchCV
from sklearn.pipeline import Pipeline
from sklearn.compose import ColumnTransformer
from sklearn.metrics import classification_report

# 1. Load and look (Week 1): .head(), .info(), .describe(), a histogram or two
df = pd.read_csv("project_data.csv")

# 2. Split BEFORE anything else touches the data (Week 5 + Week 4 leakage)
X_train, X_test, y_train, y_test = train_test_split(
    df.drop(columns=["target"]), df["target"],
    test_size=0.2, stratify=df["target"], random_state=42,
)

# 3. Preprocessing pipeline (Week 4): scaling, encoding, imputation
preprocessor = ColumnTransformer([...])  # numeric_pipeline, categorical_pipeline

# 4. Model + tuning (Weeks 5, 6, 9): baseline first, then a real model, then GridSearchCV
pipeline = Pipeline([("preprocess", preprocessor), ("model", ...)])
search = GridSearchCV(pipeline, param_grid={...}, cv=5, scoring="f1")
search.fit(X_train, y_train)

# 5. Evaluate ONCE on the test set (Week 5 + Week 6)
report = classification_report(y_test, search.predict(X_test))

The order matters as much as the content: EDA before preprocessing decisions (you can't choose a scaling or encoding strategy blind), the train/test split before any fitting, and the test set touched exactly once, at the very end — every rule from Weeks 4–6 applied together rather than in isolation.

4. Communicating Results

A model's output is only useful once someone who wasn't in the room can understand what it does and doesn't do. A short written summary should cover, briefly:

• What was predicted, and from what — the framing from Section 1, in one sentence
• The metric chosen, and why — not just "F1 was 0.81," but why F1 was the right metric for this problem's cost structure
• How it compares to the baseline — a number is meaningless without this comparison
• At least one plot — a confusion matrix, ROC curve, or feature importance chart communicates more in five seconds than a paragraph of numbers
• Known limitations — what the model would likely get wrong, and any data quality caveats from your EDA

This write-up is also the fastest way to catch your own mistakes — trying to explain why a result makes sense often surfaces a leakage bug or framing error that a purely numeric evaluation would miss.

5. Common Failure Modes and How to Debug Them

A short field guide to the problems you're most likely to hit this week:

• Suspiciously perfect score — almost always leakage (Week 4). Check every feature for "would I actually have this at prediction time?"
• Model barely beats the baseline — could be genuinely hard data, could be underfitting (Week 5) — check a learning curve before concluding either way.
• Great validation score, poor test score — you likely tuned against the validation set too many times, or leaked something during preprocessing (Week 6).
• Metric looks good, but stakeholders would disagree it's "good" — revisit Section 1's framing; you may be optimizing the wrong metric for the actual cost of errors.

6. Project

7. Knowledge Check

Four quick questions. Expand each to check your answer.