Week 9: Prompt Caching, Cost & Choosing a Model

1. Token Economics: What You're Actually Paying For

Every call is billed on tokens in (your prompt, system prompt, any attached documents or images) and tokens out (Claude's response) — input and output are typically priced differently, with output usually costing meaningfully more per token than input. Bigger models cost more per token than smaller ones, which is the whole reason Week 1's Opus/Sonnet/Haiku decision matters financially, not just qualitatively.

response = client.messages.create(...)
print(response.usage.input_tokens)   # what the request cost you, in tokens
print(response.usage.output_tokens)  # what the response cost you, in tokens
# Multiply each by that model's current per-token price (check the
# Anthropic pricing page — it's the one number in this lesson worth
# looking up fresh rather than memorizing, since pricing evolves).

The concept that doesn't change even as prices do: a large system prompt, a long attached document, or extensive few-shot examples (Week 2) all become part of your input tokens on every single call that includes them — repeating a 2,000- token document across 10,000 calls a day adds up fast, which is exactly the problem caching solves next.

2. Prompt Caching

If the same large block of context — a system prompt, a long document, a set of few-shot examples — gets reused across many calls, mark it as cacheable. On a cache hit, Claude skips reprocessing that content from scratch, which cuts both cost and latency for every subsequent call that reuses it.

response = client.messages.create(
    model="claude-sonnet-4-5",
    max_tokens=1024,
    system=[
        {
            "type": "text",
            "text": long_reused_instructions,  # e.g. a big set of rules/context
            "cache_control": {"type": "ephemeral"},
        }
    ],
    messages=[{"role": "user", "content": "The actual per-request question"}],
)

The cache has a limited lifetime (it expires after a period of inactivity), so caching pays off specifically for high-repeat workloads — a chatbot reusing the same long system prompt on every incoming message, or a document-analysis pipeline asking many different questions about the same attached document — not for one-off calls that never repeat that context again.

3. Choosing a Model, With Real Numbers

Week 1 framed the Opus/Sonnet/Haiku decision qualitatively — how hard is the task. Now add the cost dimension: the wrong choice compounds at scale. A task run once barely matters which model you pick; a task run 100,000 times a day makes the wrong tier expensive fast.

# A simple classification task, run at three different volumes:
#
# Run once:            tier choice barely matters — seconds and cents either way
# Run 1,000x/day:       a pricier tier is still probably fine
# Run 100,000x/day:     tier choice is now a real line item — the cheapest
#                       tier that hits your required accuracy wins decisively
#
# The task didn't change. The volume did. That's the whole decision.

The practical process: don't assume a smaller model "can't handle it" — measure. Run a real sample of your actual task through the cheaper tier first, check the accuracy against what you actually need, and only reach for a pricier tier if the cheaper one genuinely falls short on real examples, not out of caution alone.

4. The Batch API for Bulk Work

For large workloads that don't need an immediate response — classifying 50,000 support tickets overnight, generating summaries for a backlog of documents — the batch API processes requests asynchronously at a meaningful cost discount compared to the standard API, since there's no real-time latency requirement to satisfy.

# 1. Submit a large list of individual requests as one batch job
# 2. The batch processes over some time window (not instantaneous)
# 3. Poll for completion, then retrieve all results together
#
# This is exactly Week 1's "high-volume, low-complexity, Haiku-shaped"
# workload — batch is the delivery mechanism for running it cheaply at
# scale, once you don't need any single result back immediately.

The tradeoff is explicit: you give up immediacy in exchange for lower cost. That makes the batch API the wrong choice for anything user-facing in real time (a chat response, a live tool call) and the right choice for anything that's fundamentally a backend job.

5. Hands-on Exercise

6. Knowledge Check

Four quick questions. Expand each to check your answer.