Week 13: Convolutional Neural Networks

1. Convolutions, Kernels, Stride & Padding

A convolution slides a small learnable grid of weights — a kernel (e.g. 3×3) — across an image, computing a dot product (Week 2's same operation, again) between the kernel and each patch of pixels it covers. The output is a feature map that lights up wherever that particular pattern (an edge, a texture) appears in the image.

import torch.nn as nn

conv = nn.Conv2d(
    in_channels=3,     # RGB image
    out_channels=16,    # learn 16 different 3x3 kernels/filters
    kernel_size=3,
    stride=1,           # how far the kernel moves between applications
    padding=1,          # pad the image so output size matches input size
)

Two parameters control the output's size and behavior: stride — how many pixels the kernel jumps between applications (stride 2 roughly halves the output's spatial size) — and padding — adding a border of zeros around the input so the kernel can be centered on edge pixels without shrinking the output. The crucial efficiency win over a fully-connected layer: the same small kernel is reused across the entire image, so the network learns "what an edge looks like" once, rather than a separate weight for every pixel position.

2. Pooling Layers & Spatial Downsampling

Pooling reduces a feature map's spatial size by summarizing small regions — most commonly max pooling, which keeps only the largest value in each small patch.

pool = nn.MaxPool2d(kernel_size=2, stride=2)   # halves both height and width

model = nn.Sequential(
    nn.Conv2d(3, 16, 3, padding=1), nn.ReLU(), nn.MaxPool2d(2),
    nn.Conv2d(16, 32, 3, padding=1), nn.ReLU(), nn.MaxPool2d(2),
    nn.Flatten(),
    nn.Linear(32 * 8 * 8, 10),   # final classification head, e.g. 10 classes
)

Pooling does two things at once: it reduces computation for every subsequent layer (half the height and width means a quarter of the values to process), and it introduces a small amount of translation invariance — a feature shifted by a pixel or two still tends to produce the same pooled output, which is usually a desirable property for recognizing an object regardless of its exact position.

3. Classic Architectures & a Look at ResNet

Stacking convolution → activation → pooling blocks repeatedly, shrinking spatial size while increasing channel depth, is the pattern behind early CNNs like LeNet. Pushing this much deeper ran into a problem: very deep plain networks actually got worse at training, not better — gradients from Week 11's backpropagation would vanish or become unstable across dozens of stacked layers.

import torch.nn as nn

class ResidualBlock(nn.Module):
    def __init__(self, channels):
        super().__init__()
        self.conv1 = nn.Conv2d(channels, channels, 3, padding=1)
        self.conv2 = nn.Conv2d(channels, channels, 3, padding=1)
        self.relu = nn.ReLU()

    def forward(self, x):
        identity = x
        out = self.relu(self.conv1(x))
        out = self.conv2(out)
        return self.relu(out + identity)   # the "residual" (skip) connection

ResNet's key idea is the + identity skip connection: instead of forcing each block to learn a full transformation, it only has to learn a residual — the difference from simply passing the input through unchanged. This gives gradients a direct path backward through the shortcut, letting networks train successfully at far greater depth than was previously practical. You'll see this exact same residual-connection idea again in Week 16's transformer architecture — it's not CNN-specific, it's a general fix for training very deep networks.

4. Data Augmentation & Transfer Learning

Training a strong CNN from scratch typically needs far more labeled images than most projects have. Two techniques close that gap.

import torchvision.transforms as T
from torchvision.models import resnet18, ResNet18_Weights
import torch.nn as nn

# Data augmentation: synthesize variety from the same images
transform = T.Compose([
    T.RandomHorizontalFlip(),
    T.RandomRotation(10),
    T.ColorJitter(brightness=0.2, contrast=0.2),
    T.ToTensor(),
])

# Transfer learning: start from a model already trained on millions of images
model = resnet18(weights=ResNet18_Weights.DEFAULT)
for param in model.parameters():
    param.requires_grad = False           # freeze the pretrained feature extractor

model.fc = nn.Linear(model.fc.in_features, num_classes)   # replace only the final layer

Data augmentation artificially expands a small dataset by applying realistic transformations (flips, rotations, color shifts) that shouldn't change the true label — a cat flipped horizontally is still a cat. Transfer learning reuses a model already trained on a large, general dataset (like ImageNet), freezing its early layers — which have learned generic features like edges and textures useful across almost any image task — and only training a new final layer for your specific classes. This is often the single highest-leverage technique available when you have a small labeled dataset, since it borrows the equivalent of millions of training images' worth of learned structure for free.

5. Visualizing What a CNN Has Learned

CNNs are less of a black box than they first appear — you can directly inspect what a trained filter responds to by extracting a feature map at an intermediate layer.

activations = {}

def hook(module, input, output):
    activations["conv1"] = output.detach()

model.conv1.register_forward_hook(hook)
model(sample_image.unsqueeze(0))   # run one image through the network

feature_maps = activations["conv1"][0]   # shape: (out_channels, H, W)
# Plot each channel of feature_maps as a grayscale image

Early layers typically learn simple, general-purpose detectors — edges, color blobs, basic textures — while deeper layers combine those into detectors for increasingly complex, task-specific patterns (an eye, a wheel, a specific texture of fur). This progression from generic to specific is exactly why transfer learning's "freeze the early layers" strategy in Section 4 works: those early filters are useful across nearly any image task, while only the later, more specialized layers need retraining.

6. Hands-on Exercise

7. Knowledge Check

Four quick questions. Expand each to check your answer.