Week 19: Fine-Tuning & Transfer Learning

1. Transfer Learning: Freezing vs. Fine-Tuning Layers

Week 13 froze a pretrained CNN's early layers (generic edge/texture detectors) and trained only a new final layer for a new set of image classes. LLM fine-tuning follows the exact same logic, but usually goes further: because language understanding is distributed across nearly every layer of a transformer (not concentrated in a few generic early layers the way low-level vision features are), full fine-tuning — updating every parameter — is far more common for LLMs than the "freeze everything but the last layer" pattern was for CNNs.

from transformers import AutoModelForCausalLM, AutoTokenizer

model = AutoModelForCausalLM.from_pretrained("some-base-model")
tokenizer = AutoTokenizer.from_pretrained("some-base-model")

# Every parameter remains trainable -- unlike Week 13's frozen CNN base
for param in model.parameters():
    param.requires_grad = True

The core idea transferred from Week 13 still holds, though: you're not starting from random weights (Week 11's initialization problem) — you're starting from weights that already encode grammar, facts, and reasoning patterns learned during pretraining, and you're nudging them toward your specific task or domain rather than teaching the model language from scratch.

2. The Full Fine-Tuning Workflow for an LLM

Mechanically, fine-tuning an LLM is the same causal language modeling objective from Week 17 — predict the next token — just continued on a new, smaller, task-specific dataset instead of a general pretraining corpus.

from torch.optim import AdamW

optimizer = AdamW(model.parameters(), lr=2e-5)   # much smaller LR than pretraining from scratch

for epoch in range(3):   # fine-tuning typically needs far fewer epochs than pretraining
    for batch in train_dataloader:
        optimizer.zero_grad()
        outputs = model(**batch, labels=batch["input_ids"])
        loss = outputs.loss              # causal LM loss, same objective as Week 17
        loss.backward()                   # Week 12's autograd, same mechanism
        optimizer.step()

Two details matter more here than in a from-scratch training run: the learning rate is typically much smaller (large updates risk destroying the pretrained knowledge you're relying on), and training runs for far fewer steps — often a single pass or a few passes over a dataset that's tiny compared to the pretraining corpus.

3. Dataset Preparation for Fine-Tuning

Fine-tuning data is usually formatted as instruction/response pairs, matching the format you actually want the model to produce at inference time.

{"instruction": "Summarize this support ticket in one sentence.", "input": "Customer reports login fails after password reset...", "output": "Customer cannot log in following a password reset."}
{"instruction": "Classify the sentiment of this review.", "input": "The product arrived broken and support was unhelpful.", "output": "Negative"}

Every data-quality lesson from Weeks 1–4 applies here at a smaller scale: check for duplicate or near-duplicate examples, verify labels are actually correct (a mislabeled fine-tuning example teaches the model the wrong behavior directly), and make sure the train/validation split (Week 5) is respected — evaluating a fine-tuned model on examples it was fine-tuned on tells you nothing about how it generalizes.

4. Catastrophic Forgetting & How to Mitigate It

Catastrophic forgetting is when fine-tuning on a narrow task degrades capabilities the model had before fine-tuning — a model fine-tuned heavily on customer support tickets might become noticeably worse at general reasoning or coding it could previously do, because its weights shifted to specialize for the new task at the expense of the old.

Standard mitigations:

• Lower learning rate — smaller updates disturb the pretrained weights less (Section 2's practical detail, directly motivated by this concern).
• Fewer epochs — stop before the model over-specializes; this is Week 5's overfitting concept, applied to "overfitting toward the new task at the expense of general ability" rather than just overfitting the new dataset's noise.
• Mix in general-purpose data — blend some broad, diverse examples into the fine-tuning set alongside the task-specific ones, so the model keeps practicing its general capabilities during fine-tuning.
• Parameter-efficient tuning — Week 20's LoRA, which updates far fewer parameters, tends to cause less forgetting simply because it changes less of the model.

5. Compute Cost of Full Fine-Tuning at Scale

Full fine-tuning updates every parameter, which means storing gradients and optimizer state (Adam, from Week 12, keeps two extra numbers per parameter) for the entire model — memory requirements that scale directly with model size, often several times the model's own size in GPU memory. For a small model this is manageable on consumer hardware; for a multi-billion-parameter model, full fine-tuning can require the same class of expensive, multi-GPU infrastructure as pretraining itself, just for far fewer steps.

This cost is precisely the motivation for Week 20's parameter-efficient methods: if updating 100% of a model's parameters requires enterprise-grade hardware, what if you could get most of the benefit by updating only 1%?

6. Hands-on Exercise

7. Knowledge Check

Four quick questions. Expand each to check your answer.