1. Byte-Pair Encoding & Subword Tokenization
A transformer's embedding layer needs a fixed, finite vocabulary of tokens to look up — but text is effectively infinite (any new word, typo, or made-up term is possible). Word-level tokenization (one token per whole word) breaks on anything outside a fixed dictionary; character-level tokenization avoids that problem but produces very long sequences, which is expensive given attention's quadratic cost from Week 15. Byte-Pair Encoding (BPE) is the middle ground almost every modern LLM uses: a subword vocabulary, learned from data.
# BPE, conceptually:
# 1. Start with a vocabulary of individual characters/bytes
# 2. Count every adjacent pair of tokens across the training corpus
# 3. Merge the single MOST FREQUENT pair into one new token
# 4. Repeat steps 2-3 until you reach a target vocabulary size
corpus = "low lower lowest new newer newest"
# Iteration 1: "e" + "r" is common -> merge into "er"
# Iteration 2: "low" + "er" is common -> merge into "lower"
# ... continuing until the vocabulary reaches its target size
The result is a vocabulary where common whole words ("the", "and") often become a single
token, while rarer or unfamiliar words get split into meaningful subword pieces (e.g.
"unhappiness" might become ["un", "happi", "ness"]) — and any string, even
one the tokenizer has never seen, can still be represented, by falling back to
increasingly small pieces down to individual bytes if necessary. This is precisely why an
LLM can process a made-up word or a typo without crashing: nothing is truly
"out of vocabulary."
A model that seems to "struggle counting letters in strawberry" is often a tokenization artifact, not a reasoning failure — the word might be a single token (or a couple of subword tokens) to the model, so it never actually sees individual letters the way a human reading the word does.
2. Pretraining Objectives: Causal LM vs. Masked LM
A pretraining objective is the task a model is actually trained to solve on raw, unlabeled text — no human-written labels required, which is what makes training on such enormous amounts of text possible at all.
# Causal language modeling (decoder-only, e.g. GPT-style -- Week 16's decoder-only architecture)
# Predict the NEXT token, given only everything before it
text = "The cat sat on the"
target = "mat" # trained using the causal mask from Week 15
# Masked language modeling (encoder-only, e.g. BERT-style)
# Predict RANDOMLY MASKED tokens, using context from BOTH directions
text = "The cat [MASK] on the mat"
target = "sat" # can see tokens both before AND after the mask
Causal language modeling — predict the next token, using only what came before — trains directly for the thing a decoder-only model actually does at inference time: generate text one token at a time. Masked language modeling trains a model to fill in blanks using bidirectional context, which produces excellent representations for understanding/classification tasks but doesn't naturally support open-ended generation the way causal LM does. This is exactly why Week 16's dominant decoder-only architecture pairs with causal LM: the pretraining objective and the intended use case (generation) match perfectly.
3. Building & Inspecting a Training Corpus
A pretraining corpus is typically a mixture of web text, books, code, and other sources, filtered and deduplicated at massive scale. The exact same data-quality concerns from Weeks 1–4 apply here, just at a much larger scale:
- Deduplication — repeated documents can cause the model to overweight (effectively memorize) that content, similar in spirit to a leaked feature dominating a small dataset.
- Filtering low-quality text — spam, boilerplate, and garbled text degrade what the model learns, the same "garbage in, garbage out" concern from Week 1's EDA.
- Contamination checks — making sure benchmark test sets (Week 18) haven't leaked into the training corpus, which would silently inflate reported performance — precisely Week 4's data leakage, at the scale of an entire internet-sized dataset.
You won't build a corpus at this scale yourself, but the underlying discipline — inspect your data before you trust a model trained on it — is identical to Week 1's very first lesson, just applied at a different order of magnitude.
4. Context Windows & How They Constrain a Model
The context window is the maximum number of tokens a model can attend over at once — a hard architectural limit tied directly to Week 15's positional encoding (which typically doesn't generalize cleanly beyond the lengths seen in training) and its quadratic attention cost (a longer window is dramatically more expensive to actually run).
# A context window budget must cover EVERYTHING the model sees at once:
# - System/instructions
# - Conversation history
# - Any retrieved documents (Week 22's RAG)
# - The model's own generated output so far
context_window = 128_000 # example, in tokens
system_prompt = 500
conversation_history = 20_000
retrieved_documents = 8_000
remaining_for_response = context_window - system_prompt - conversation_history - retrieved_documents
A larger context window doesn't just cost more compute — it directly determines what's even possible: a long document that doesn't fit has to be chunked (Week 22), a long conversation has to be truncated or summarized, and a large codebase can't be handed to the model in one shot. This budget arithmetic is something you'll do explicitly again when designing a RAG pipeline in Week 22 and estimating cost in Week 25.
5. Compute & Data Requirements, at a High Level
Pretraining a modern LLM from scratch requires an amount of compute (measured in floating-point operations) and data (measured in tokens) that scales with model size in a roughly predictable way — the subject of Week 18's scaling laws. For now, the practical takeaway: pretraining a frontier-scale model from scratch costs many millions of dollars in compute and requires trillions of tokens of text, which is precisely why almost nobody — including most companies — pretrains their own base model. Instead:
- Use an existing pretrained model (open-weight or via an API) as the starting point
- Adapt it to a specific task via fine-tuning (Week 19) or parameter-efficient tuning (Week 20)
- Or steer it without touching weights at all, via prompting (Week 21) or retrieval (Week 22)
Nearly every remaining week in this course is really about this same question: given a pretrained model you didn't (and won't) train from scratch, how do you get it to do what you actually need?
6. Hands-on Exercise
Train a BPE tokenizer and measure vocabulary-size tradeoffs
Use the Hugging Face tokenizers library to train a real BPE tokenizer on a custom text corpus.
from tokenizers import Tokenizer
from tokenizers.models import BPE
from tokenizers.trainers import BpeTrainer
from tokenizers.pre_tokenizers import Whitespace
tokenizer = Tokenizer(BPE(unk_token="[UNK]"))
tokenizer.pre_tokenizer = Whitespace()
trainer = BpeTrainer(vocab_size=1000, special_tokens=["[UNK]"])
# tokenizer.train(files=["your_corpus.txt"], trainer=trainer)
Requirements:
- Assemble a small text corpus (a few thousand words of any public-domain text) and train a BPE tokenizer with
vocab_size=500. - Tokenize five sentences, including at least one unusual or made-up word, and print the resulting tokens. Confirm the made-up word gets split into subword pieces rather than failing.
- Retrain with
vocab_size=2000andvocab_size=8000, and compare average tokens-per-sentence across all three vocabulary sizes. - Write one paragraph explaining the tradeoff: what improves and what gets more expensive as vocabulary size grows, in terms of sequence length, embedding table size, and how "whole-word" common tokens become.
- Identify one real word from your corpus that got split into multiple subword tokens, and explain why (hint: check how frequently that exact word appeared during training).
A larger vocabulary generally produces fewer tokens per sentence (shorter sequences, cheaper attention) but a much larger embedding table (more parameters) — there's no free lunch, just a tradeoff to reason about explicitly.
7. Knowledge Check
Four quick questions. Expand each to check your answer.
Q1
Why does BPE use subword tokens instead of whole words or individual characters?
Why does BPE use subword tokens instead of whole words or individual characters?
Whole-word tokenization can't represent words outside its fixed vocabulary. Character-level tokenization handles any text but produces much longer sequences, which is expensive given attention's quadratic cost. Subword tokenization is a middle ground: common words become single tokens for efficiency, while rare or novel words can still be represented by falling back to smaller, known pieces.
Q2
Why does causal language modeling pair naturally with a decoder-only architecture?
Why does causal language modeling pair naturally with a decoder-only architecture?
Causal LM's training task — predict the next token using only prior context — is exactly what a decoder-only model does at inference time when generating text. Training and inference use the identical causal-masked mechanism, so there's no mismatch between what the model was trained to do and what it's actually asked to do afterward.
Q3
Why is a model's context window a hard limit rather than something you can just extend at inference time?
Why is a model's context window a hard limit rather than something you can just extend at inference time?
It's tied to the positional encoding scheme the model was trained with (which often doesn't generalize cleanly to unseen lengths) and to attention's quadratic cost, which makes arbitrarily long contexts prohibitively expensive to actually compute — both are architectural and computational constraints, not a configuration setting you can freely change.
Q4
Why do most companies adapt an existing pretrained model rather than pretraining their own from scratch?
Why do most companies adapt an existing pretrained model rather than pretraining their own from scratch?
Pretraining a frontier-scale model requires trillions of tokens of data and compute costing many millions of dollars — resources far beyond what almost any individual project or company needs to invest, when fine-tuning (Week 19), parameter-efficient tuning (Week 20), or prompting (Week 21) an existing pretrained model can achieve the needed result at a small fraction of the cost.