Week 12: Training Deep Networks with PyTorch

1. Tensors, Autograd & the Training Loop

A PyTorch tensor is NumPy's ndarray with one crucial addition: if you set requires_grad=True, PyTorch records every operation performed on it, building a graph it can later walk backward through to compute gradients automatically — exactly the backpropagation you derived by hand in Week 11, generalized to networks of any size.

import torch
import torch.nn as nn

model = nn.Sequential(
    nn.Linear(20, 64), nn.ReLU(),
    nn.Linear(64, 32), nn.ReLU(),
    nn.Linear(32, 1), nn.Sigmoid(),
)

loss_fn = nn.BCELoss()
optimizer = torch.optim.Adam(model.parameters(), lr=1e-3)

for epoch in range(50):
    optimizer.zero_grad()               # clear gradients from the last step
    y_pred = model(X_train_tensor)      # forward pass
    loss = loss_fn(y_pred, y_train_tensor)
    loss.backward()                      # autograd computes every gradient
    optimizer.step()                     # apply the update (Week 11's "W -= lr * dW", generalized)

optimizer.zero_grad() exists because PyTorch accumulates gradients by default rather than overwriting them — forgetting this call is one of the most common bugs in a first PyTorch training loop, silently summing gradients across steps instead of computing each step's fresh.

2. Optimizers: SGD, Momentum & Adam

Week 11's gradient descent update — subtract the gradient, scaled by a learning rate — is plain SGD. In practice, three refinements almost always train faster and more reliably:

• SGD with momentum — accumulates a running average of past gradients, so updates build speed in a consistent direction and dampen oscillation across noisy ones.
• Adam — maintains a per-parameter adaptive learning rate based on both the gradient's recent average and its recent variance, making it far more forgiving of a poorly-chosen global learning rate.

import torch.optim as optim

sgd = optim.SGD(model.parameters(), lr=0.01, momentum=0.9)
adam = optim.Adam(model.parameters(), lr=1e-3, betas=(0.9, 0.999))

Adam is the practical default for most projects — it converges quickly with little tuning. Plain SGD with momentum still shows up in settings (some vision and large language model training runs) where its slightly different generalization behavior is worth the extra tuning effort, but as a rule: reach for Adam first, and only revisit that choice if you have a specific reason to.

3. Regularization: Dropout, Weight Decay & Early Stopping

These are Week 5's overfitting fixes, now made concrete for neural networks specifically.

model = nn.Sequential(
    nn.Linear(20, 64), nn.ReLU(), nn.Dropout(p=0.3),
    nn.Linear(64, 32), nn.ReLU(), nn.Dropout(p=0.3),
    nn.Linear(32, 1), nn.Sigmoid(),
)

# Weight decay: L2 penalty added directly by the optimizer
optimizer = torch.optim.Adam(model.parameters(), lr=1e-3, weight_decay=1e-4)

Dropout randomly zeroes out a fraction of neurons on every training step, forcing the network to not rely too heavily on any single neuron — a form of built-in ensembling, conceptually related to the bagging from Week 9. Weight decay penalizes large weight values directly in the loss, discouraging the network from fitting noise with extreme weights. Early stopping — stopping training the moment validation loss stops improving, exactly as with early_stopping_rounds in Week 9's gradient boosting — prevents a network from continuing to memorize the training set after it has already found the useful pattern.

4. Batch Normalization & Learning Rate Schedules

Batch normalization normalizes each layer's activations (subtracting the batch mean, dividing by the batch standard deviation) before the next layer sees them — the same standardization idea from Week 4, applied between layers rather than only at the input. This keeps activations in a consistent, well-behaved range throughout training, which generally allows higher learning rates and faster convergence.

model = nn.Sequential(
    nn.Linear(20, 64), nn.BatchNorm1d(64), nn.ReLU(),
    nn.Linear(64, 32), nn.BatchNorm1d(32), nn.ReLU(),
    nn.Linear(32, 1), nn.Sigmoid(),
)

optimizer = torch.optim.Adam(model.parameters(), lr=1e-3)
scheduler = torch.optim.lr_scheduler.StepLR(optimizer, step_size=10, gamma=0.5)

for epoch in range(50):
    # ... training loop ...
    scheduler.step()   # halve the learning rate every 10 epochs

A learning rate schedule lowers the learning rate over the course of training — large steps early to make fast progress, smaller steps later to settle precisely into a good minimum instead of oscillating around it. This is a direct answer to Week 11's tension between "too large" and "too small" learning rates: rather than picking one fixed value, start large and shrink it over time.

5. Reading Training/Validation Curves to Diagnose Problems

This is Week 5's learning curve, extended over training epochs rather than dataset size, and it's the single most useful debugging tool for a neural network.

• Both losses stay high and flat — underfitting; the model is too small, or the learning rate is badly chosen. Try a larger network or a different learning rate before anything else.
• Training loss keeps dropping, validation loss stops or rises — classic overfitting; apply Section 3's tools (dropout, weight decay, early stopping).
• Both losses drop together, smoothly — healthy training; keep going, or consider whether you've plateaued and could benefit from Section 4's schedule.
• Loss is NaN or explodes — almost always a learning-rate-too-high or unstabilized-activation problem; lower the learning rate and check for batch normalization.

6. Hands-on Exercise

from sklearn.datasets import make_classification
import torch

X, y = make_classification(
    n_samples=300, n_features=20, n_informative=6,
    n_redundant=8, flip_y=0.1, random_state=42,
)

7. Knowledge Check

Four quick questions. Expand each to check your answer.