Week 11: Neural Network Foundations

1. Perceptrons & the Limits of Linear Models

A perceptron computes a weighted sum of its inputs, plus a bias, and passes the result through a step function — the exact dot product from Week 2's linear algebra lesson, now wearing a neural network's name.

import numpy as np

def perceptron(x, weights, bias):
    z = x @ weights + bias      # the Week 2 dot product, plus a bias term
    return 1 if z > 0 else 0

A single perceptron can only draw a straight-line (linear) decision boundary — it can learn AND and OR, but famously cannot learn XOR, because no single straight line separates XOR's true and false outputs. Stacking perceptrons into layers, with a non-linear activation function between them (Section 2), is precisely what lets a network bend that boundary into something curved enough to solve XOR — and, at much larger scale, everything from image recognition to language modeling.

2. Activation Functions

Without a non-linear activation function between layers, stacking many linear layers would still collapse into one big linear function — depth would buy you nothing. Activation functions are what let depth actually add expressive power.

import numpy as np

def sigmoid(z):
    return 1 / (1 + np.exp(-z))          # squashes to (0, 1) -- classic, but saturates for large |z|

def relu(z):
    return np.maximum(0, z)               # zero below 0, identity above -- the modern default

def softmax(z):
    shifted = z - np.max(z, axis=-1, keepdims=True)   # the stability trick from Week 2
    exp_z = np.exp(shifted)
    return exp_z / np.sum(exp_z, axis=-1, keepdims=True)

Sigmoid is intuitive (it looks like a probability) but "saturates" — for large positive or negative inputs, its output barely changes, and its gradient shrinks toward zero, slowing learning in deep networks. ReLU avoids this for positive inputs (constant gradient of 1) and is dramatically cheaper to compute, which is why it's the default choice inside hidden layers of most modern networks. Softmax, which you already built in Week 2, is reserved for a network's final layer in multi-class classification, converting raw scores into a probability distribution over classes.

3. Loss Functions

A loss function measures how wrong a prediction is — the single number the entire training process tries to minimize.

import numpy as np

def mse_loss(y_true, y_pred):
    return np.mean((y_true - y_pred) ** 2)          # regression: Week 6's MSE, revisited

def binary_cross_entropy(y_true, y_pred, eps=1e-12):
    y_pred = np.clip(y_pred, eps, 1 - eps)           # avoid log(0)
    return -np.mean(y_true * np.log(y_pred) + (1 - y_true) * np.log(1 - y_pred))

Mean squared error is standard for regression outputs. Cross-entropy is standard for classification, and it connects directly back to Week 2's probability foundations: it's exactly the negative log-likelihood of the true label under the model's predicted probability distribution — a network that assigns high probability to the correct class gets low loss; one that's confidently wrong gets heavily penalized, because -log(p) grows sharply as p approaches 0.

4. Backpropagation & the Chain Rule, by Hand

Training means adjusting every weight to reduce the loss. Backpropagation computes exactly how much each weight contributed to the final error, using the calculus chain rule to propagate the loss's gradient backward, layer by layer, from output to input.

import numpy as np

def relu_derivative(z):
    return (z > 0).astype(float)

# --- Forward pass ---
z1 = X @ W1 + b1        # hidden layer pre-activation
a1 = relu(z1)            # hidden layer activation
z2 = a1 @ W2 + b2        # output layer pre-activation
y_pred = sigmoid(z2)     # final prediction

loss = binary_cross_entropy(y_true, y_pred)

# --- Backward pass: the chain rule, one layer at a time ---
dz2 = y_pred - y_true                      # derivative of loss w.r.t. z2 (a clean result for this loss+sigmoid pairing)
dW2 = a1.T @ dz2 / len(X)
db2 = dz2.mean(axis=0)

da1 = dz2 @ W2.T                            # gradient flowing backward into the hidden layer
dz1 = da1 * relu_derivative(z1)             # chain rule through ReLU's derivative
dW1 = X.T @ dz1 / len(X)
db1 = dz1.mean(axis=0)

Read this from the bottom up and the pattern is exactly the chain rule from calculus: the gradient at each layer is the gradient from the layer after it, multiplied by that layer's own local derivative. dz2 = y_pred - y_true looks suspiciously simple — that's not a shortcut, it's what the chain rule for sigmoid-plus-cross-entropy actually simplifies to, which is exactly why that pairing is so common in practice.

5. Gradient Descent, Learning Rate & Convergence

With gradients computed, gradient descent updates every weight a small step in the direction that reduces the loss — the negative of the gradient, scaled by a learning rate.

learning_rate = 0.1

W1 -= learning_rate * dW1
b1 -= learning_rate * db1
W2 -= learning_rate * dW2
b2 -= learning_rate * db2

# Repeat forward pass -> backward pass -> update, for many epochs

The learning rate is the single most consequential hyperparameter in this whole process: too small, and training crawls, needing far more epochs to converge; too large, and updates overshoot the loss's minimum, sometimes diverging entirely (loss increasing every step instead of decreasing). Plotting the loss curve over training — a direct cousin of Week 5's learning curves — is the standard way to catch this: a healthy loss curve drops smoothly; a too-large learning rate produces a jagged or exploding one.

6. Hands-on Exercise

import numpy as np

X = np.array([[0, 0], [0, 1], [1, 0], [1, 1]])
y = np.array([[0], [1], [1], [0]])   # XOR

rng = np.random.default_rng(42)
W1 = rng.normal(scale=0.5, size=(2, 4))   # 2 inputs -> 4 hidden units
b1 = np.zeros((1, 4))
W2 = rng.normal(scale=0.5, size=(4, 1))   # 4 hidden units -> 1 output
b2 = np.zeros((1, 1))

7. Knowledge Check

Four quick questions. Expand each to check your answer.