Week 4: Data Wrangling & Feature Engineering

1. Feature Scaling

Many models — anything using distance or a gradient, including the neural networks you'll build starting Week 11 — are sensitive to the scale of each feature. A "years of experience" column (0–40) and a "salary" column (0–200,000) will otherwise let salary dominate purely because its numbers are bigger, not because it's more predictive.

import numpy as np
from sklearn.preprocessing import StandardScaler, MinMaxScaler

X = np.array([[5.0, 50000], [10.0, 62000], [2.0, 45000], [20.0, 90000]])

# Standardization: (x - mean) / std -- centers at 0, unit variance
standardized = StandardScaler().fit_transform(X)

# Min-max: (x - min) / (max - min) -- squashes into [0, 1]
normalized = MinMaxScaler().fit_transform(X)

Standardization is the safer default for most models — it's not distorted by a single extreme value the way min-max scaling is, since min-max anchors its entire range to whatever the min and max happen to be. Reach for min-max scaling specifically when you need values in a bounded range (like [0, 1] pixel intensities feeding a neural network).

2. Encoding Categorical Variables

Models operate on numbers, so a column like city or plan_tier needs to become numeric before it can be used — but naively converting categories to arbitrary integers invents a false ordering the model will happily "learn."

import pandas as pd

df = pd.DataFrame({
    "plan": ["free", "pro", "enterprise", "pro", "free"],
    "size": ["small", "medium", "large", "small", "large"],
})

# One-hot: a new binary column per category -- no false ordering
one_hot = pd.get_dummies(df["plan"], prefix="plan")

# Ordinal: use only when a real order exists
size_order = {"small": 0, "medium": 1, "large": 2}
df["size_ordinal"] = df["size"].map(size_order)

One-hot encoding is the right default for unordered categories (plan has no natural rank) — it creates one binary column per category so the model can't infer an ordering that isn't there. Ordinal encoding is only correct when a genuine order exists, like size here. A third option, target encoding (replacing a category with the average target value for that category), is powerful for high-cardinality columns like "zip code" but is also the single easiest way to leak information from your target — a problem you'll dig into in Section 5.

3. Handling Outliers & Skewed Distributions

Building on Week 3's descriptive statistics: an extreme outlier doesn't just distort the mean, it can dominate a model that's sensitive to scale — one $10 million transaction in a dataset of $50 purchases can single-handedly determine a linear model's coefficients.

import numpy as np
import pandas as pd

df = pd.DataFrame({"transaction": [45, 52, 48, 61, 55, 10_000_000, 50]})

# Detect outliers with the IQR method
q1, q3 = df["transaction"].quantile([0.25, 0.75])
iqr = q3 - q1
lower, upper = q1 - 1.5 * iqr, q3 + 1.5 * iqr
is_outlier = (df["transaction"] < lower) | (df["transaction"] > upper)

# A common fix for skewed positive data: log-transform
df["transaction_log"] = np.log1p(df["transaction"])

np.log1p (log of 1 + x, safely handling zero) compresses large values much more than small ones, turning a heavily right-skewed column into something closer to normal — often enough, on its own, to make a linear model behave far better without discarding a single row of real data.

4. Pipelines & ColumnTransformer

Applying scaling, encoding and imputation as separate manual steps works in a notebook, but it's fragile — it's easy to fit a scaler on the wrong data, or forget a step when you apply the same transformations to new data later. scikit-learn's Pipeline and ColumnTransformer bundle every step into one object that fits and transforms consistently.

from sklearn.pipeline import Pipeline
from sklearn.compose import ColumnTransformer
from sklearn.preprocessing import StandardScaler, OneHotEncoder
from sklearn.impute import SimpleImputer

numeric_features = ["age", "income"]
categorical_features = ["plan", "city"]

numeric_pipeline = Pipeline([
    ("impute", SimpleImputer(strategy="median")),
    ("scale", StandardScaler()),
])

categorical_pipeline = Pipeline([
    ("impute", SimpleImputer(strategy="most_frequent")),
    ("encode", OneHotEncoder(handle_unknown="ignore")),
])

preprocessor = ColumnTransformer([
    ("num", numeric_pipeline, numeric_features),
    ("cat", categorical_pipeline, categorical_features),
])

# preprocessor.fit_transform(X_train) -- fits ALL steps only on training data
# preprocessor.transform(X_test)      -- reuses those exact fitted parameters

The payoff is that fit and transform are now separate, enforced steps — which is precisely the mechanism that prevents the leakage bug in Section 5. You'll reuse this exact pipeline pattern as the first step of the Week 8 mini-project and every classical ML model after it.

5. Data Leakage

Data leakage happens when information from outside the training data — often, indirectly, information from the test set or the target itself — sneaks into training and makes a model look far better than it will ever perform in the real world. It is the single most common reason a beginner's model reports 99% accuracy and then fails completely in production.

# BUG: scaler is fit on the full dataset, BEFORE the train/test split
scaler = StandardScaler()
X_scaled = scaler.fit_transform(X)               # sees test data's mean/std!

X_train, X_test, y_train, y_test = train_test_split(X_scaled, y)
# The scaler's mean/std now leak information about the test set into training

# Split FIRST, then fit the scaler only on training data
X_train, X_test, y_train, y_test = train_test_split(X, y)

scaler = StandardScaler()
X_train_scaled = scaler.fit_transform(X_train)   # fit only on training data
X_test_scaled = scaler.transform(X_test)          # reuse those training statistics

The fix is always the same shape: split before you fit anything, then only ever call .transform() (never .fit() again) on the test set. This is exactly what the Pipeline from Section 4 enforces automatically when combined with cross-validation — another reason to prefer it over manual steps.

6. Hands-on Exercise

import pandas as pd
import numpy as np

df = pd.DataFrame({
    "age": [25, 34, np.nan, 45, 29, 52, 38],
    "income": [42000, 58000, 51000, 95000, 47000, 250000, 61000],
    "plan": ["free", "pro", "pro", "enterprise", "free", "enterprise", np.nan],
    "signup_channel": ["ad", "referral", "ad", "organic", "referral", "ad", "organic"],
    "refund_issued": [0, 0, 1, 0, 0, 1, 0],   # only ever set AFTER a churn decision
    "churned": [0, 0, 1, 0, 0, 1, 0],
})

7. Knowledge Check

Four quick questions. Expand each to check your answer.