Week 9: Tree-Based Models & Gradient Boosting

1. Decision Trees: Splits, Impurity & Overfitting

A decision tree predicts by asking a sequence of yes/no questions about your features — "is age < 35?", then "is income > 50000?" — and arriving at a leaf that holds a prediction. It chooses each split by testing every feature and threshold, and picking whichever split most reduces impurity — how mixed the classes are within each resulting group.

from sklearn.tree import DecisionTreeClassifier, plot_tree

tree = DecisionTreeClassifier(max_depth=3, random_state=42)
tree.fit(X_train, y_train)

# plot_tree(tree, feature_names=feature_names, filled=True) -- visualize every split

Gini impurity and entropy are the two common impurity measures — both are near zero when a group is almost all one class, and highest when classes are evenly mixed. Left unrestricted, a tree keeps splitting until every leaf is perfectly pure (often a single training example per leaf) — the textbook overfitting shape from Week 5's learning curves. max_depth, min_samples_leaf and min_samples_split are the standard levers to stop it early.

2. Random Forests & Bagging

A single tree is a high-variance model — small changes in training data can produce a very different tree. Bagging (bootstrap aggregating) tackles this by training many trees on different random samples of the data (drawn with replacement) and averaging their predictions. A random forest adds one more trick: each tree also only considers a random subset of features at every split, decorrelating the trees from each other so their errors are less likely to overlap.

from sklearn.ensemble import RandomForestClassifier

forest = RandomForestClassifier(
    n_estimators=300, max_depth=None, max_features="sqrt", random_state=42,
)
forest.fit(X_train, y_train)

Averaging many high-variance, low-bias trees is a direct, practical application of the bias-variance tradeoff from Week 5: each individual tree still overfits its own bootstrap sample, but averaging cancels out much of that variance while the ensemble's bias stays low — usually a much better tradeoff than any single tree could reach on its own.

3. Gradient Boosting Fundamentals

Bagging trains trees independently and averages them. Boosting instead trains trees sequentially — each new tree is trained specifically to correct the errors (technically, the gradient of the loss) left by the ensemble so far. This typically achieves lower bias than bagging, at the cost of being more prone to overfitting if left unchecked, and being inherently sequential (harder to parallelize across trees, though modern libraries parallelize within each tree's construction).

from xgboost import XGBClassifier

boosted = XGBClassifier(
    n_estimators=300,
    max_depth=4,             # boosted trees are usually shallower than forest trees
    learning_rate=0.05,      # how much each new tree corrects the ensemble
    subsample=0.8,           # row sampling, like bagging's bootstrap
    colsample_bytree=0.8,    # feature sampling, like a random forest
    eval_metric="logloss",
)
boosted.fit(
    X_train, y_train,
    eval_set=[(X_val, y_val)],
    early_stopping_rounds=20,   # stop once validation performance stops improving
)

learning_rate and n_estimators trade off directly: a smaller learning rate needs more trees to reach the same fit, but usually generalizes better. early_stopping_rounds uses the validation set from Week 5 to stop adding trees the moment they stop helping — an automatic, data-driven way to avoid overfitting rather than guessing n_estimators up front. XGBoost and LightGBM are the two most common production-grade gradient boosting libraries, and both follow this same shape.

4. Feature Importance & Partial Dependence

Tree ensembles trade some interpretability for accuracy compared to a linear model's coefficients, but they still offer two useful explanation tools.

import pandas as pd
from sklearn.inspection import PartialDependenceDisplay

importances = pd.Series(forest.feature_importances_, index=feature_names)
importances.sort_values(ascending=False).head(10)

# Partial dependence: how predictions change as ONE feature varies,
# averaging out every other feature's effect
PartialDependenceDisplay.from_estimator(forest, X_train, features=["income"])

Feature importance ranks how much each feature contributed to reducing impurity across all trees — useful for a quick "what does the model rely on," but it says nothing about the direction of the effect, and can be misleading for highly correlated features that split the credit between them. Partial dependence fills that gap by showing, for one feature at a time, how the average prediction shifts as that feature's value changes — much closer to "what would happen if" than importance alone, though it's still a correlational tool, not a causal one (the distinction from Week 3).

5. When Tree Ensembles Beat (or Lose to) Linear Models

Tree ensembles are usually the strongest out-of-the-box choice for tabular data with non-linear relationships and feature interactions, and they need almost no preprocessing (no scaling, tolerant of outliers, handle missing values natively in some implementations). But they're not universally better:

• Very high-dimensional, sparse data (e.g. text represented as bag-of-words) — linear models often generalize better and train far faster.
• Genuinely linear relationships — a tree approximates a straight line with a staircase of splits; a linear model represents it exactly, with fewer parameters.
• Need for a simple, auditable rule — a logistic regression's coefficients are easier to explain to a regulator or stakeholder than "the average of 300 trees."
• Very small datasets — a simpler linear model's higher bias but lower variance can generalize better when there's too little data to reliably fit a complex ensemble.

The practical answer, consistent with Week 8's mini-project approach: try both, compare on your chosen metric from Week 6, and let the data — not a rule of thumb — decide.

6. Hands-on Exercise

from sklearn.datasets import make_classification

X, y = make_classification(
    n_samples=1500, n_features=25, n_informative=8,
    n_redundant=5, n_clusters_per_class=3, random_state=42,
)

7. Knowledge Check

Four quick questions. Expand each to check your answer.