Week 25: Deployment, Cost & Latency

1. Serving Options: Hosted APIs vs. Self-Hosted Inference

This is Week 18's open-vs-closed decision made operational. A hosted API (calling a provider's endpoint) means no infrastructure to manage, automatic scaling, and per-token pricing — ideal for most projects, especially at low-to-moderate volume. Self-hosting (running an open-weight model, Week 18, on your own or rented GPUs) trades that convenience for control over data residency, no per-token cost ceiling, and the ability to serve a model you fine-tuned yourself (Weeks 19–20) exactly as trained.

# Hosted API tends to win when:
# - Usage volume is low-to-moderate (per-token cost stays manageable)
# - You need the strongest available model with zero ops overhead

# Self-hosting tends to win when:
# - Volume is high and sustained (fixed GPU cost beats per-token cost at scale)
# - Data cannot leave your infrastructure
# - You're serving a custom fine-tuned/LoRA-adapted model (Weeks 19-20) directly

2. Latency, Throughput & Batching

Latency is how long a single request takes; throughput is how many requests a system can serve per unit time. For LLMs specifically, two factors dominate both: the causal generation loop (Week 17) means output tokens are produced one at a time, and attention's quadratic cost (Week 15) means longer inputs slow every step of that generation down.

# Total latency for a request roughly decomposes into:
# - Time to first token (TTFT): processing the full input context before generation starts
# - Time per output token: generation happens one token at a time, sequentially
# - Total latency ~= TTFT + (output_tokens * time_per_token)

# A long RAG context (Week 22) or a long agent history (Week 23) directly increases TTFT,
# while a longer requested response directly increases total generation time

Batching — processing multiple requests' token-generation steps together on the same GPU — is the standard way serving infrastructure improves throughput, at the cost of potentially added latency for any individual request while it waits to be batched with others. This is a direct, practical instance of a batch-size tradeoff you first saw conceptually in Week 12's training loops, now applied to inference instead of training.

3. Cost Modeling for LLM Usage at Scale

Hosted APIs typically price by token, separately for input and output — which means Week 17's context-window budgeting is directly, literally a cost calculation, not just an architectural constraint.

price_per_1k_input = 0.003    # example pricing, illustrative only
price_per_1k_output = 0.015

def estimate_cost(input_tokens, output_tokens, requests_per_day):
    daily_input_cost = (input_tokens / 1000) * price_per_1k_input * requests_per_day
    daily_output_cost = (output_tokens / 1000) * price_per_1k_output * requests_per_day
    return daily_input_cost + daily_output_cost

# A RAG pipeline (Week 22) with a large retrieved context is an INPUT-token-heavy cost;
# a long generated response is an OUTPUT-token-heavy cost -- they're priced differently

Notice output tokens are typically priced higher than input tokens (generation is more computationally expensive per token than processing existing context) — meaning "ask the model to be more concise" is a legitimate, meaningful cost lever, not just a style preference. Multiplying a per-request cost by realistic daily volume, as this snippet does, is what turns "this feature works" into "this feature is affordable" — a distinction many prototypes never actually check before shipping.

4. Caching & Prompt/Response Optimization

Two cheap wins reduce both cost and latency without touching the model at all.

import hashlib

cache = {}

def cached_generate(prompt):
    key = hashlib.sha256(prompt.encode()).hexdigest()
    if key in cache:
        return cache[key]   # skip the model call entirely for a repeated prompt
    result = call_llm(prompt)
    cache[key] = result
    return result

Response caching avoids paying for (and waiting on) an identical request twice — valuable for common queries in a support bot or repeated tool descriptions in an agent (Week 23). Many providers also support prompt caching, where a long, unchanged prefix (a system prompt, a large RAG context reused across a conversation) is cached server-side so subsequent requests reprocess only the new part of the input — directly cutting the TTFT cost from Section 2 for exactly the kind of long, mostly-static context Week 22's RAG pipelines produce.

5. Monitoring a Live LLM Application

Once deployed, an LLM application needs the same kind of operational visibility as any production system, plus a few LLM-specific signals: token usage and cost per request (to catch a cost regression early), latency percentiles (not just averages — Week 3's lesson on medians and percentiles applies directly here), and eval-suite scores (Week 24) tracked over time rather than checked only once before launch.

# Version prompts and model choices explicitly, the same way you'd version code
config = {
    "version": "v3",
    "model": "some-model-v2",
    "system_prompt": "...",
    "rollback_target": "v2",   # what to revert to if v3 regresses
}

# Track Week 24's eval suite score per version -- a prompt or model change
# that regresses the eval score is a signal to roll back, exactly like a
# failed test in ordinary software deployment

Treating prompts and model versions with the same discipline as application code — versioned, tested against a fixed eval suite before rollout, and quick to roll back — is what actually separates a prototype from a production system. It's the same engineering instinct as Week 6's "don't tune against your test set" discipline, applied to production changes instead of a training run.

6. Hands-on Exercise

7. Knowledge Check

Four quick questions. Expand each to check your answer.