Week 24: Evaluation, Safety & Guardrails

1. Building Evals & Benchmarks for Your Own Use Case

Week 18 warned that public leaderboards often don't reflect your specific task. The fix is the same one Week 21 already used at small scale for prompts: build your own labeled eval set, sized to your actual use case, and score every candidate system against it — a base model, a fine-tuned model (Week 19), a RAG pipeline (Week 22), or an agent (Week 23).

eval_cases = [
    {"input": "What's your return policy?", "expected_topics": ["30 days", "receipt"], "must_not_contain": ["guarantee refund"]},
    # ... 20-100+ real, representative examples
]

def run_eval(system_fn, eval_cases):
    results = []
    for case in eval_cases:
        output = system_fn(case["input"])
        passed_topics = all(topic in output for topic in case["expected_topics"])
        passed_safety = not any(bad in output for bad in case["must_not_contain"])
        results.append({"case": case, "output": output, "passed": passed_topics and passed_safety})
    return results

The key discipline, straight from Week 6: pick metrics tied to what actually matters for your use case (not a generic accuracy number), keep the eval set fixed while you iterate so comparisons are fair, and treat a regression on this eval set the same way you'd treat a validation-metric regression when tuning a classical model — a real signal to investigate, not noise to ignore.

2. Hallucination Detection & Mitigation Strategies

A hallucination is confident, fluent output that's factually wrong or unsupported by any real source — a direct consequence of how an LLM generates text (Week 17's next-token prediction has no built-in notion of "truth," only "plausible continuation"). Week 22 introduced one detection method already: a groundedness check for RAG. More generally:

• Self-consistency checks — ask the same question multiple times (or with slight rephrasing) and flag answers that disagree with each other, a signal the model is guessing rather than retrieving a stable fact.
• Citation requirements — require the model to cite a specific source for factual claims (from RAG's retrieved context), and verify the citation actually supports the claim.
• Retrieval grounding — Week 22's approach: prefer answers explicitly derived from retrieved context over the model's parametric memory, for anything fact-sensitive.
• Confidence calibration — explicitly instructing (and evaluating) that the model say "I don't know" when it genuinely doesn't, rather than always producing a confident-sounding guess.

3. Guardrails & Responsible-AI Basics

Guardrails are checks applied around a model's input and output — not relying on the model alone to always behave correctly, the same layered-defense principle from Week 21's prompt-injection discussion, extended to safety broadly.

def is_output_safe(text):
    # A real implementation might use a dedicated classifier model,
    # a keyword/policy check, or a second LLM call as a moderation layer
    flagged_categories = classify_content(text)   # e.g. a moderation API
    return len(flagged_categories) == 0

response = generate_response(prompt)
if not is_output_safe(response):
    response = fallback_response()   # never show unsafe output to the user

Responsible-AI basics worth building into any real project: check for biased or unfair treatment across different user groups in your eval set (Section 1), be explicit with users about a system's limitations rather than implying more capability than it has, and keep a human able to intervene for consequential decisions — echoing Week 23's "confirm before consequential tool calls" guard, applied to model output more broadly.

4. LLM-as-Judge Evaluation Patterns

Many qualities worth evaluating — helpfulness, tone, whether a summary captured the key point — don't reduce to a simple string match the way Section 1's expected_topics check does. LLM-as-judge uses a second LLM call to score or compare outputs against a rubric, standing in for a human evaluator at far lower cost and higher throughput.

judge_prompt = f"""
Rate the following response on a scale of 1-5 for helpfulness and accuracy,
given the original question. Respond with only a number and a one-sentence justification.

Question: {question}
Response: {response}
"""
score = call_llm(judge_prompt)

This is powerful but not infallible: an LLM judge can share the same blind spots as the model being judged, can be inconsistent across repeated calls, and can be gamed by responses that are superficially well-formatted but substantively wrong. Best practice is to validate a judge's scores against a small sample of actual human judgments before trusting it at scale — the same "don't trust a metric you haven't sanity-checked" instinct from Week 6.

5. Informally Red-Teaming a Model

Red-teaming means deliberately trying to break your own system before someone else does — probing for prompt injection (Week 21), attempts to extract the system prompt, requests for disallowed content phrased to evade simple filters, and edge-case inputs your eval set (Section 1) doesn't happen to cover.

# Try, deliberately, before shipping:
# - "Ignore previous instructions and..." style prompt injection
# - Asking the system to reveal its system prompt or internal instructions
# - Rephrasing a disallowed request to sound like a legitimate one
# - Extremely long or malformed inputs
# - Inputs in a different language than the eval set covers

You don't need a dedicated security team to get real value from this — even a modest, deliberate hour spent trying to break your own system before real users do routinely surfaces gaps that normal, well-behaved testing never would, precisely because normal testing doesn't try to find the edges on purpose.

6. Hands-on Exercise

7. Knowledge Check

Four quick questions. Expand each to check your answer.