Week 20: Parameter-Efficient Tuning & RLHF/DPO

1. LoRA: Low-Rank Adapters, Explained and Implemented

LoRA (Low-Rank Adaptation) is built on an empirical observation: the change a model's weights need during fine-tuning tends to have low "intrinsic rank" — it can be well-approximated by a much smaller matrix than the full weight matrix itself. Instead of updating a weight matrix W directly, LoRA freezes W entirely and learns a small correction alongside it, expressed as the product of two much smaller matrices.

import torch
import torch.nn as nn

class LoRALinear(nn.Module):
    def __init__(self, in_features, out_features, rank=8):
        super().__init__()
        self.frozen_weight = nn.Parameter(torch.randn(out_features, in_features), requires_grad=False)

        # The ONLY trainable parameters: two small low-rank matrices
        self.A = nn.Parameter(torch.randn(rank, in_features) * 0.01)
        self.B = nn.Parameter(torch.zeros(out_features, rank))   # zero-init -> starts as a no-op

    def forward(self, x):
        base_output = x @ self.frozen_weight.T
        lora_output = x @ self.A.T @ self.B.T   # low-rank correction: (in -> rank -> out)
        return base_output + lora_output

With rank=8 on a weight matrix that might be 4096×4096 (~16.8 million parameters), LoRA's two matrices total only 8 × 4096 × 2 ≈ 65,000 trainable parameters — roughly 0.4% of the original matrix. Zero-initializing B means the LoRA correction starts at exactly zero, so training begins from the unmodified pretrained model's behavior and gradually learns a small, targeted adjustment — directly reducing Week 19's catastrophic forgetting risk, simply because far fewer weights ever change.

2. QLoRA & Quantization for Memory-Efficient Tuning

LoRA reduces trainable parameters, but the frozen base model still has to sit in memory in full precision. Quantization shrinks that memory footprint by storing the frozen weights using fewer bits per number (commonly 4-bit instead of the usual 16 or 32-bit floating point) — QLoRA combines this quantized base model with LoRA training on top.

from transformers import AutoModelForCausalLM, BitsAndBytesConfig
from peft import LoraConfig, get_peft_model

quant_config = BitsAndBytesConfig(load_in_4bit=True)   # base model stored in 4-bit
model = AutoModelForCausalLM.from_pretrained("some-base-model", quantization_config=quant_config)

lora_config = LoraConfig(r=8, lora_alpha=16, target_modules=["q_proj", "v_proj"])
model = get_peft_model(model, lora_config)   # wraps attention's Q/V projections with LoRA

Notice target_modules points specifically at the query and value projections inside attention (Week 15's Q and V) — LoRA is typically applied only to a subset of a model's weight matrices, most commonly inside attention, rather than every single matrix in the network, further shrinking the trainable footprint. The combination of 4-bit quantization plus LoRA is precisely what makes fine-tuning multi-billion-parameter models practical on a single consumer GPU, something full fine-tuning (Week 19) simply couldn't do at that scale.

3. Instruction Tuning

A raw pretrained model, trained purely on next-token prediction (Week 17), is good at continuing text plausibly — but not reliably at following instructions, because "complete this text" and "do what I asked" aren't the same skill. Instruction tuning is supervised fine-tuning (Week 19's exact workflow) specifically on a dataset of instruction/response pairs, teaching the model the specific behavior of interpreting a request and producing a helpful, direct response rather than just a plausible continuation.

This is the same mechanism as Week 19's fine-tuning — the novelty here is purely in the data: large, diverse collections of instructions across many task types, so the model generalizes "follow instructions" as a general skill rather than memorizing answers to specific questions.

4. RLHF & DPO

Instruction tuning teaches a model to follow instructions in general, but doesn't directly teach it which of several plausible responses humans actually prefer — tone, helpfulness, and safety judgments that are hard to specify as a single "correct" training example. RLHF (Reinforcement Learning from Human Feedback) addresses this in two stages: train a separate reward model to predict which of two responses a human would prefer, then use reinforcement learning (commonly PPO) to adjust the LLM to produce responses the reward model scores highly.

# Stage 1: train a reward model on human preference comparisons
# Given (prompt, response_a, response_b, human_preferred), learn to predict the preferred one

# Stage 2: use the reward model's score as a reward signal to further train the LLM
# (via PPO -- a reinforcement learning algorithm, conceptually similar to the
#  "policy gets rewarded, updates to do more of what worked" idea from Week 5's syllabus overview)

RLHF works well but is notoriously complex to implement correctly — it involves training and maintaining multiple models (policy, reward model, and often a reference model) and tuning reinforcement learning's characteristically finicky hyperparameters. DPO (Direct Preference Optimization) is a simpler alternative that skips the separate reward model and reinforcement learning loop entirely: it reformulates the same preference data into a direct supervised loss the model can be trained on with ordinary gradient descent (Week 11), reaching a similar end result with meaningfully less engineering complexity — which is why many newer open post-training recipes favor DPO over full RLHF.

5. Choosing Full Fine-Tuning vs. LoRA vs. Prompting

With three genuinely different ways to adapt a model's behavior now covered (this week plus prompting in Week 21), the practical decision comes down to what you're trying to change and what resources you have:

• Prompting (Week 21) — no training at all; fastest to iterate, no infrastructure needed, but limited by what fits in the context window (Week 17) and can be less consistent across many similar requests.
• LoRA (this week) — moderate effort, moderate hardware; ideal when you need consistent behavior change across many requests but can't afford full fine-tuning's cost or forgetting risk.
• Full fine-tuning (Week 19) — highest cost and forgetting risk, but sometimes necessary for large, fundamental behavior shifts that a low-rank correction genuinely can't capture.

A common, pragmatic sequence in practice: start with prompting, since it's nearly free to test; move to LoRA if prompting can't reliably achieve the needed consistency; reserve full fine-tuning for cases where LoRA's low-rank approximation demonstrably isn't enough.

6. Hands-on Exercise

7. Knowledge Check

Four quick questions. Expand each to check your answer.