Week 18: Scaling Laws, Model Families & Reading Model Cards

1. Scaling Laws: Parameters, Data & Compute

A scaling law is an empirical relationship: as you increase model size (parameters), training data, and compute together in the right proportions, a model's loss on held-out data decreases in a remarkably smooth, predictable curve — smooth enough that researchers can often estimate a much larger model's eventual performance before fully training it.

# Loss tends to fall off as a power law in each factor, holding others fixed:
# loss(N) ~ N ** (-alpha)     where N = number of parameters
# loss(D) ~ D ** (-beta)      where D = number of training tokens
#
# The practical finding (Chinchilla, 2022): many earlier large models were
# UNDER-trained relative to their size -- a smaller model trained on more
# tokens can outperform a larger model trained on fewer, for the same compute budget

The key practical lesson: parameters and data must scale together. A model with far more parameters than its training data can support tends to be compute-inefficient — you'd get better performance from a smaller model trained on more tokens for the same cost. This is why "bigger is always better" is an oversimplification; the actual driver is whether size and data are balanced for the compute budget spent.

2. Overview of Major Model Families & Their Design Choices

Different model families make different architectural and training tradeoffs within the same overall transformer blueprint from Week 16:

• Dense vs. mixture-of-experts (MoE) — a dense model uses every parameter for every token; an MoE model routes each token to only a subset of specialized "expert" sub-networks, letting total parameter count grow without proportionally increasing compute per token.
• Context window size — families differ substantially in how long a sequence they support, directly shaped by the positional encoding and attention-cost tradeoffs from Weeks 15–17.
• Training data composition — how much code, multilingual text, or curated instruction data went into pretraining and post-training shapes what a model is naturally good at.
• Post-training recipe — how heavily a family relies on instruction tuning, RLHF, or DPO (Week 20) shapes how well it follows instructions "out of the box" versus needing more prompt engineering (Week 21).

Rather than memorizing a snapshot of "current best models" (which will be outdated quickly), the durable skill is knowing which of these design axes to ask about when a new model is announced.

3. Open-Weight vs. Closed Models

Closed models (accessed only via an API) hide their weights and often their exact training details, but usually offer the strongest turnkey performance with no infrastructure to manage. Open-weight models publish their trained parameters for anyone to download, self-host, and fine-tune directly — enabling full control, on-premises deployment for sensitive data, and no per-token API cost, at the price of needing your own infrastructure and expertise.

# Lean toward a closed API model when:
# - You need the strongest available general performance with minimal setup
# - Your data isn't sensitive enough to require self-hosting
# - You'd rather pay per-token than manage GPU infrastructure

# Lean toward an open-weight model when:
# - You need to fine-tune the base model itself (Weeks 19-20)
# - Data residency / privacy requires the model to run in your own environment
# - You expect high, predictable volume where self-hosting is cheaper at scale

This decision resurfaces directly and concretely in Week 25's deployment discussion, where cost, latency and infrastructure tradeoffs are made precise rather than conceptual.

4. Reading a Model Card Critically

A model card is a model's accompanying documentation — training data summary, intended uses, known limitations, and evaluation results. Reading one critically means treating it the way you'd treat any claim backed by a metric, applying the same skepticism from Week 3's statistics and Week 6's evaluation lessons:

• What exactly was measured? — a benchmark score is only as meaningful as the benchmark itself represents your actual use case (Week 6's "wrong metric" problem, one level up).
• What's explicitly listed as a limitation? — a card that lists no limitations is a bigger red flag than one that lists several; every model has them.
• What's not mentioned? — training data composition, safety evaluation methodology, and known failure modes are sometimes vague or absent, which is itself informative.
• Is the comparison fair? — a model touted as "beating" another on a cherry-picked benchmark may lose on others that better reflect your task.

5. Benchmark Leaderboards & Their Limitations

Public benchmark leaderboards are useful for a rough sense of relative capability, but come with real limitations worth naming explicitly, echoing Week 17's contamination concern:

• Contamination — if a benchmark's questions leaked into a model's training data (Week 17), its score is inflated and not a fair measure of generalization.
• Benchmark-task mismatch — a model ranked highly on a general reasoning benchmark may not be the best choice for your narrow, specific task (e.g. structured data extraction).
• Overfitting to the leaderboard — teams can (consciously or not) tune specifically to popular benchmarks, a version of Week 6's "tuning against your validation set too many times" problem, but happening at an industry-wide scale.
• Point-in-time snapshots — leaderboards move fast; a ranking from even a few months ago may already be stale.

The reliable alternative, consistent with Week 6's whole philosophy: build a small, task-specific eval set of your own (formalized properly in Week 24) rather than trusting a general leaderboard to answer a specific question about your specific use case.

6. Hands-on Exercise

7. Knowledge Check

Four quick questions. Expand each to check your answer.