Week 16: The Full Transformer Architecture

1. Positional Encoding

Attention, as built in Week 15, is fundamentally order-blind — Q @ K.T produces the same relevance scores no matter what order the tokens are fed in, unlike an RNN, which processes tokens strictly in sequence. Since word order clearly carries meaning ("dog bites man" vs. "man bites dog"), the model needs order injected explicitly. Positional encoding adds a vector — a function of position — directly to each token's embedding before attention ever runs.

import numpy as np

def sinusoidal_positional_encoding(seq_len, d_model):
    position = np.arange(seq_len)[:, np.newaxis]
    dim = np.arange(d_model)[np.newaxis, :]
    angle_rates = 1 / np.power(10000, (2 * (dim // 2)) / d_model)
    angles = position * angle_rates

    pe = np.zeros((seq_len, d_model))
    pe[:, 0::2] = np.sin(angles[:, 0::2])   # even dimensions: sine
    pe[:, 1::2] = np.cos(angles[:, 1::2])   # odd dimensions: cosine

    return pe

token_embeddings = X + sinusoidal_positional_encoding(seq_len, d_model)

The original transformer used this fixed sinusoidal pattern, chosen because its particular mathematical structure lets the model infer relative positions (the encoding for position 5 relative to position 3 has a consistent relationship regardless of where in the sequence those positions fall) purely through linear combinations. Many modern models instead use learned positional embeddings — a plain trainable lookup table indexed by position — letting the model discover whatever positional pattern works best for its data, at the cost of not extrapolating well beyond the lengths seen during training (directly relevant to the "context window" limits you'll examine in Week 17).

2. Residual Connections & Layer Normalization

A transformer stacks many attention and feed-forward sublayers — and just like the very deep CNNs from Week 13, training would be unstable without help. The transformer borrows exactly the same fix: a residual connection around every sublayer, adding the sublayer's input back to its output.

def layer_norm(x, eps=1e-5):
    mean = x.mean(axis=-1, keepdims=True)
    var = x.var(axis=-1, keepdims=True)
    return (x - mean) / np.sqrt(var + eps)

# The transformer block's actual pattern, per sublayer:
attn_out = multi_head_attention(X, num_heads=4, d_model=d_model)
X = layer_norm(X + attn_out)          # residual, THEN normalize

Layer normalization is Week 4 and Week 12's normalization idea applied across a single token's features (rather than across a batch, the way batch normalization does), which makes it work cleanly regardless of sequence length or batch size — an important practical difference from Week 12's batch norm, which depends on batch statistics. Together, residuals give gradients a direct path through dozens of stacked transformer blocks (exactly ResNet's argument from Week 13), while layer norm keeps activations in a stable range at every step.

3. The Feed-Forward Sublayer

Every transformer block pairs its attention sublayer with a second, simpler sublayer: a small, ordinary fully-connected network (Week 11's territory) applied independently to each token's representation.

def feed_forward(x, W1, b1, W2, b2):
    hidden = np.maximum(0, x @ W1 + b1)   # expand, typically to 4x d_model, then ReLU
    return hidden @ W2 + b2                # project back down to d_model

ff_out = feed_forward(X, W1, b1, W2, b2)
X = layer_norm(X + ff_out)                 # another residual + norm, same pattern as Section 2

Where attention lets tokens exchange information with each other, the feed-forward sublayer processes each token's representation independently, adding additional non-linear transformation capacity. A full transformer block, in order, is: attention → residual + norm → feed-forward → residual + norm — and this exact block is simply stacked N times to build the full model.

4. Encoder & Decoder Stacks

An encoder stack is a sequence of transformer blocks using plain (unmasked) self-attention — every token can attend to every other token, in both directions, which suits tasks that need to understand a complete input. A decoder stack is nearly identical, but uses the causal mask from Week 15 in its self-attention (so it can't look ahead), and — in the original encoder-decoder design — adds a cross-attention sublayer where its queries come from the decoder while keys and values come from the encoder's output.

# A decoder block, in order:
X = layer_norm(X + masked_self_attention(X))          # 1. causal self-attention
X = layer_norm(X + cross_attention(X, encoder_output))  # 2. cross-attention (encoder-decoder only)
X = layer_norm(X + feed_forward(X))                     # 3. feed-forward

5. Encoder-Only vs. Decoder-Only vs. Encoder-Decoder

These building blocks combine into three architecture families, each suited to a different kind of task:

• Encoder-only (e.g. BERT-style models) — bidirectional self-attention, good at understanding/classifying complete input text (sentiment, embeddings for search). Not naturally suited to generating new text.
• Decoder-only (e.g. GPT-style models) — causal self-attention only, generates text one token at a time, each new token attending only to what came before. This is the dominant architecture for modern LLMs, and the one you'll build in this week's exercise.
• Encoder-decoder (e.g. the original translation transformer, T5-style models) — a full encoder processes the input, a decoder generates the output while cross-attending back to it. Well-suited to tasks with a clear input→output transformation, like translation or summarization.

Decoder-only models won out as the dominant architecture for general-purpose LLMs largely because a single next-token-prediction objective (Week 17) can be applied uniformly to almost any text, and the same architecture that generates text can also be prompted to "understand" it — one architecture serving both jobs, rather than needing separate encoder and decoder specializations.

6. Hands-on Exercise

7. Knowledge Check

Four quick questions. Expand each to check your answer.