Week 10: SVMs, Naive Bayes & Ensemble Methods

1. Support Vector Machines & the Kernel Trick

A support vector machine (SVM) finds the boundary between classes that maximizes the margin — the distance from the boundary to the nearest point of either class. Those nearest points are the "support vectors" that give the algorithm its name; every other point could move without changing the boundary at all.

from sklearn.svm import SVC
from sklearn.preprocessing import StandardScaler
from sklearn.pipeline import Pipeline

svm = Pipeline([
    ("scale", StandardScaler()),   # SVMs are distance-based -- scaling matters, as in Week 4/7
    ("svm", SVC(kernel="linear", C=1.0)),
])
svm.fit(X_train, y_train)

A linear boundary can't separate classes that are tangled in a circular or otherwise non-linear pattern. The kernel trick solves this without ever explicitly computing new features: it substitutes a similarity function (kernel) for the plain dot product in the SVM's math, which behaves as if the data had been projected into a much higher-dimensional space where a straight boundary does exist — without the computational cost of actually constructing that space.

# RBF (Gaussian) kernel: handles non-linear boundaries
svm_rbf = SVC(kernel="rbf", C=1.0, gamma="scale")

# C controls the margin/misclassification tradeoff (smaller C = wider margin, more tolerance for errors)
# gamma controls how far a single point's influence reaches (smaller gamma = smoother boundary)

2. Naive Bayes & Its Independence Assumption

Naive Bayes applies Week 2's Bayes' theorem directly as a classifier: for a new example, it computes P(class | features) for every possible class and predicts whichever is highest. The "naive" part is the simplifying assumption that every feature is conditionally independent given the class — clearly false in most real data (word order matters in text; features do correlate) — but the algorithm often works remarkably well anyway.

from sklearn.naive_bayes import MultinomialNB
from sklearn.feature_extraction.text import CountVectorizer

# Classic use case: text classification (spam filtering, from Week 2's example)
vectorizer = CountVectorizer()
X_counts = vectorizer.fit_transform(emails)   # word-count features

nb = MultinomialNB()
nb.fit(X_counts, labels)

Naive Bayes' real strengths are speed and data efficiency: it trains almost instantly even on very high-dimensional data (like word counts over a large vocabulary), and can produce reasonable results with far less training data than more flexible models need — exactly the profile of a strong baseline for text classification tasks, ahead of anything more expensive.

3. Bagging vs. Boosting vs. Stacking

Week 9 covered bagging (random forests) and boosting (gradient boosting) — both combine many models of the same type. Stacking takes a different approach: train several genuinely different model types, then train a final meta-model whose input is each base model's predictions, learning how to best combine them.

from sklearn.ensemble import StackingClassifier
from sklearn.svm import SVC
from sklearn.naive_bayes import GaussianNB
from sklearn.ensemble import RandomForestClassifier
from sklearn.linear_model import LogisticRegression

stack = StackingClassifier(
    estimators=[
        ("svm", SVC(probability=True, kernel="rbf")),
        ("nb", GaussianNB()),
        ("forest", RandomForestClassifier(n_estimators=200, random_state=42)),
    ],
    final_estimator=LogisticRegression(),   # the meta-model
    cv=5,   # base models are trained via cross-validation to avoid the meta-model overfitting
)
stack.fit(X_train, y_train)

The cv=5 parameter matters: if each base model's predictions on the meta-model's training data came from models that had already seen those exact rows, the meta-model would be handed suspiciously confident (leaked) inputs — the same category of problem as Week 4 and Week 6's leakage discussions, just one layer up. StackingClassifier handles this internally using cross-validation to generate honest out-of-fold predictions.

4. Voting Classifiers

A simpler alternative to stacking's learned meta-model is a voting classifier, which just combines predictions by majority vote (hard voting) or by averaging predicted probabilities (soft voting) — no meta-model to train at all.

from sklearn.ensemble import VotingClassifier

voter = VotingClassifier(
    estimators=[
        ("svm", SVC(probability=True, kernel="rbf")),
        ("nb", GaussianNB()),
        ("forest", RandomForestClassifier(n_estimators=200, random_state=42)),
    ],
    voting="soft",
)
voter.fit(X_train, y_train)

Voting tends to help most when the combined models make genuinely different kinds of mistakes — three models that all get the same examples wrong won't out-vote their shared blind spot. This is exactly why Section 3's stacking example deliberately combines an SVM, Naive Bayes and a tree ensemble rather than three variations of the same algorithm: diversity in how each model errs is the actual source of the improvement.

5. Interpretability Tradeoffs Across Algorithm Families

Every algorithm family covered so far sits at a different point on the accuracy/interpretability spectrum, and picking one is a real design decision, not just a performance contest:

• Logistic/linear regression — most interpretable; a coefficient per feature with a direct meaning.
• Decision trees — visualizable as an explicit set of rules, readable by a non-technical stakeholder.
• Naive Bayes — reasonably interpretable via each feature's contribution to the posterior, fast, and a strong baseline.
• Random forests / gradient boosting — explainable only indirectly, via feature importance and partial dependence (Week 9); usually the accuracy leader on tabular data.
• SVMs, stacked/voting ensembles — least directly interpretable; the "black box" end of the classical ML spectrum, ahead of only the deep networks starting Week 11.

In regulated or high-stakes settings (credit decisions, medical use), interpretability can outweigh a small accuracy gain — a theme that resurfaces directly when you weigh evaluation and safety tradeoffs for LLMs in Week 24.

6. Hands-on Exercise

from sklearn.datasets import make_moons

X, y = make_moons(n_samples=800, noise=0.25, random_state=42)

7. Knowledge Check

Four quick questions. Expand each to check your answer.