Week 14: Sequence Models — RNNs, LSTMs & an Intro to Attention

1. RNNs & the Vanishing Gradient Problem

A recurrent neural network (RNN) processes a sequence one element at a time, maintaining a hidden state that's updated at every step and carries information forward from everything seen so far.

import torch.nn as nn

rnn = nn.RNN(input_size=50, hidden_size=64, batch_first=True)

# x: (batch, sequence_length, input_size)
output, hidden = rnn(x)
# output: hidden state at every time step
# hidden: the FINAL hidden state -- a summary of the entire sequence

The same weights are reused at every time step (a form of weight sharing across time, conceptually similar to Week 13's weight sharing across space), which lets an RNN handle sequences of any length. The problem: computing gradients backward through time (backpropagation through many time steps) means repeatedly multiplying by the same weight matrix — and just like Week 11's discussion of unstable training, repeated multiplication can make gradients shrink toward zero (vanish) or grow uncontrollably (explode) over long sequences. A vanishing gradient means the network effectively can't learn dependencies between tokens that are far apart.

2. LSTMs & GRUs: Gates and Why They Help

LSTMs (Long Short-Term Memory networks) fix this by adding a separate cell state — a kind of conveyor belt that information can travel along largely unchanged — controlled by learned gates that decide what to keep, what to forget, and what to output at each step.

import torch.nn as nn

lstm = nn.LSTM(input_size=50, hidden_size=64, batch_first=True)

output, (hidden, cell) = lstm(x)
# forget gate:  decides what to discard from the cell state
# input gate:   decides what new information to add
# output gate:  decides what part of the cell state becomes the visible hidden state

Because the cell state's updates are mostly additive (rather than the repeated matrix multiplication of a plain RNN), gradients have a much more direct path backward through time, largely sidestepping the vanishing gradient problem for moderately long sequences. GRUs (Gated Recurrent Units) simplify this design into fewer gates, training faster with often comparable performance — a common practical alternative when LSTM's full complexity isn't needed.

3. Sequence-to-Sequence Modeling & the Bottleneck Problem

Sequence-to-sequence (seq2seq) models — used for tasks like translation, where an input sequence maps to an output sequence of a different length — use an encoder RNN to read the entire input and compress it into a single final hidden state, then a decoder RNN that generates the output sequence starting from that one hidden state.

# Encoder: reads the whole input sentence
_, (encoder_hidden, encoder_cell) = encoder_lstm(input_sequence)

# Decoder: must generate the ENTIRE output sequence
# starting from just this one fixed-size hidden state
decoder_output, _ = decoder_lstm(decoder_input, (encoder_hidden, encoder_cell))

Here's the problem this creates: no matter how long the input sentence is — five words or five hundred — the encoder has to squeeze everything into one fixed-size hidden vector. For long sequences, this is an information bottleneck: details from early in the input are increasingly likely to get lost by the time the encoder finishes reading a long sequence, well before the decoder ever gets a chance to use them.

4. Attention as a Fix: a First, Intuitive Look

The fix, introduced originally for exactly this seq2seq bottleneck, is attention: instead of forcing the decoder to work from one fixed summary vector, let it look back at every encoder hidden state at every decoding step, and learn to weight them by relevance for whatever it's currently generating.

# All encoder hidden states are kept, not just the final one
encoder_outputs, _ = encoder_lstm(input_sequence)   # one hidden state PER input token

# At each decoding step, compute a relevance score between the
# decoder's current state and every encoder hidden state
scores = decoder_hidden @ encoder_outputs.transpose(1, 2)   # a dot product, per Week 2

weights = softmax(scores)              # Week 2's softmax, turning scores into a distribution
context = weights @ encoder_outputs     # a weighted average -- attend more to relevant tokens

This should look familiar: a dot product to compute relevance, softmax to turn scores into weights, and a weighted sum — exactly the vectors and softmax you built in Week 2, now solving a real architectural problem. This mechanism eliminates the bottleneck entirely, since the decoder can pull directly from any part of the input, regardless of sequence length. Week 15 formalizes this exact idea — with the same query/key/value framing — into the self-attention mechanism that powers every modern transformer.

5. Where RNNs Still Make Sense vs. Where Transformers Took Over

Once attention alone (without any recurrence at all) proved sufficient — the subject of Weeks 15–16 — transformers largely replaced RNNs for language tasks, because transformers process an entire sequence in parallel (no step-by-step recurrence) and handle long-range dependencies far more directly than an RNN's sequential hidden state ever could. RNNs and LSTMs still show up in a few practical niches:

• Streaming/online settings — where you genuinely process one new token at a time as it arrives, without access to the full sequence up front
• Very resource-constrained deployments — an RNN's per-step computation can be cheaper than the attention mechanism's cost, which scales with sequence length squared (a cost you'll examine directly in Week 15)
• Some time-series forecasting tasks — where sequences are moderate-length and the inductive bias of strict left-to-right recurrence fits naturally

6. Hands-on Exercise

7. Knowledge Check

Four quick questions. Expand each to check your answer.