Week 22: Retrieval-Augmented Generation & Embeddings

1. Embeddings & Vector Similarity

An embedding model converts a piece of text into a fixed-length vector such that semantically similar text ends up nearby in that vector space — a sentence about "returning a damaged product" lands close to "refund for a broken item," even though they share almost no exact words.

from sentence_transformers import SentenceTransformer
import numpy as np

model = SentenceTransformer("all-MiniLM-L6-v2")

query = "How do I get a refund for a broken item?"
doc = "To return a damaged product for a refund, visit the returns portal within 30 days."

query_vec = model.encode(query)
doc_vec = model.encode(doc)

def cosine_similarity(a, b):
    return np.dot(a, b) / (np.linalg.norm(a) * np.linalg.norm(b))   # exactly Week 2's formula

similarity = cosine_similarity(query_vec, doc_vec)

This is precisely Week 2's cosine similarity, applied to embeddings instead of the small hand-built word-count vectors from that lesson's exercise. RAG's entire retrieval step is built on this one idea: embed a query, embed a collection of documents, and rank documents by how close their vectors are to the query's — a real-world direct application of Week 2's dot-product-and-norm formula.

2. Chunking Strategies

A whole document is usually too long to embed as one meaningful vector (its meaning gets diluted across many topics), and too long to fit in a context window (Week 17) alongside a question and an answer. Chunking splits documents into smaller pieces before embedding and storing them.

def chunk_text(text, chunk_size=500, overlap=50):
    chunks = []
    start = 0
    while start < len(text):
        end = start + chunk_size
        chunks.append(text[start:end])
        start += chunk_size - overlap   # overlap keeps context from being cut mid-thought
    return chunks

Chunk size is a real tradeoff: too small, and a chunk loses surrounding context needed to make sense of it in isolation; too large, and its embedding blurs together multiple distinct ideas, hurting retrieval precision (Section 4). Overlap between consecutive chunks helps avoid awkwardly severing a sentence or idea exactly at a chunk boundary. More sophisticated approaches chunk along natural document structure (paragraphs, sections, headers) rather than a fixed character count — generally preferable when the document's own structure is meaningful, at the cost of more complex preprocessing.

3. Vector Databases & Approximate Nearest-Neighbor Search

Computing cosine similarity between a query and every single chunk works fine for a thousand chunks, but doesn't scale to millions — exactly the same kind of computational wall Week 15 hit with attention's quadratic cost. A vector database stores embeddings alongside an index structure built for fast approximate nearest-neighbor (ANN) search, trading a small amount of retrieval accuracy for dramatically faster lookups at scale.

import chromadb

client = chromadb.Client()
collection = client.create_collection("support_docs")

collection.add(
    embeddings=[doc_vec_1, doc_vec_2, doc_vec_3],
    documents=[chunk_1, chunk_2, chunk_3],
    ids=["doc1", "doc2", "doc3"],
)

results = collection.query(query_embeddings=[query_vec], n_results=3)   # top-3 nearest chunks

"Approximate" is the key word: rather than guaranteeing the mathematically exact nearest neighbors, ANN indexes (like HNSW, a common one under the hood) find very likely nearest neighbors far faster than an exhaustive search — a deliberate, usually worthwhile accuracy/speed tradeoff, directly reminiscent of Week 9's tradeoff between a simpler and a more accurate model when speed matters.

4. Retrieval Quality: Recall vs. Precision Tradeoffs

Retrieval has its own version of Week 6's precision/recall tension: retrieving more chunks (a higher n_results) increases the chance the truly relevant information is somewhere in there (recall), but also increases the chance of including irrelevant or distracting chunks that push up the token count (Week 17's context budget) and can confuse the model's final answer (precision).

retrieved_chunks = collection.query(query_embeddings=[query_vec], n_results=5)["documents"][0]

context = "\n\n".join(retrieved_chunks)
prompt = f"""
Answer the question using ONLY the context below. If the answer isn't in the context, say so.

Context:
{context}

Question: {query}
"""

"Answer using ONLY the context" is doing real work in that prompt — without it, the model may fall back on its pretrained knowledge (Week 17) even when the retrieved context doesn't actually contain the answer, defeating the entire purpose of grounding the response in your own data. Tuning n_results, chunk size, and this instruction together is the practical work of building a good RAG system.

5. Evaluating a RAG System's Answers for Groundedness

A RAG answer can fail in two distinct ways, and telling them apart matters: the retrieval step might fail to find the relevant chunk at all, or retrieval might succeed while the model still hallucinates — generating an answer not actually supported by the retrieved context, despite the instruction from Section 4.

# Ask a second LLM call to verify: is the answer actually supported by the retrieved context?
verification_prompt = f"""
Context: {context}
Answer: {generated_answer}

Does the answer above rely ONLY on information present in the context? Reply YES or NO, with a brief reason.
"""

This kind of check — separately verifying retrieval quality ("was the right chunk found?") and generation groundedness ("did the model actually use it correctly?") — is the foundation Week 24 builds into a proper evaluation suite, including hallucination detection as a first-class metric rather than an afterthought.

6. Hands-on Exercise

7. Knowledge Check

Four quick questions. Expand each to check your answer.