Week 2: Linear Algebra & Probability for Machine Learning

1. Vectors & Matrices

You don't need a semester of linear algebra to start — you need to recognize three objects, because they're the literal building blocks of every model from Week 11 onward.

• Scalar — a single number (e.g. a learning rate)
• Vector — an ordered list of numbers (e.g. one data point's features, or one word's embedding)
• Matrix — a 2D grid of numbers (e.g. a whole dataset, or a neural network layer's weights)

import numpy as np

v = np.array([1.5, 0.0, 3.2])           # a vector, shape (3,)
M = np.array([[1, 2, 3], [4, 5, 6]])    # a matrix, shape (2, 3) -- 2 rows, 3 columns

# The transpose flips rows and columns
M_T = M.T                                # shape (3, 2)

The transpose shows up constantly in the notation you'll read later — a weight matrix applied one way in a forward pass is frequently transposed to compute a gradient going backward, and you'll see Wᵀ throughout the transformer paper in Week 16.

2. The Dot Product & Matrix Multiplication

The one operation to internalize now is the dot product — multiply corresponding elements of two vectors, then sum the results:

features = np.array([1.5, 0.0, 3.2])   # one data point
weights  = np.array([0.4, 0.9, -0.1])  # a model's learned weights

# Dot product, by hand
manual = sum(f * w for f, w in zip(features, weights))

# Dot product, the NumPy way
dot = features @ weights          # or: np.dot(features, weights)

print(manual, dot)   # 0.28 0.28 -- same result

That single line, features @ weights, is — quite literally — what a neuron in a neural network computes before its activation function is applied. Matrix multiplication is just this same dot product done for every row/column pair at once:

A = np.array([[1, 2], [3, 4]])   # shape (2, 2)
B = np.array([[5, 6], [7, 8]])   # shape (2, 2)

C = A @ B
# C[i, j] = dot product of A's row i and B's column j
# C = [[1*5+2*7, 1*6+2*8],
#      [3*5+4*7, 3*6+4*8]]
#   = [[19, 22],
#      [43, 50]]

# The one rule that matters: inner dimensions must match.
# A (2, 3) @ B (3, 4) is valid -> result is (2, 4).
# A (2, 3) @ B (2, 4) raises a ValueError -- the 3 and 2 don't line up.

A batch of 32 data points, each with 3 features, run through a layer with 3 inputs and 4 outputs is exactly one matrix multiplication: (32, 3) @ (3, 4) → (32, 4). Every forward pass in Week 11's neural network, and every attention score in Week 15's transformer, is this same shape-matching rule applied over and over.

3. Vector Norms & Cosine Similarity

A norm measures a vector's length. The two you'll see everywhere: the L1 norm (sum of absolute values) and the L2 norm (square root of the sum of squares — ordinary Euclidean length).

v = np.array([3.0, -4.0])

l1 = np.sum(np.abs(v))            # 7.0
l2 = np.sqrt(np.sum(v ** 2))      # 5.0  -- or: np.linalg.norm(v)

Norms alone measure size, not direction. To compare how similar two vectors are — regardless of their length — you use cosine similarity: the dot product of two vectors, divided by the product of their norms. It ranges from -1 (opposite direction) to 1 (same direction), with 0 meaning perpendicular/unrelated.

def cosine_similarity(a, b):
    return np.dot(a, b) / (np.linalg.norm(a) * np.linalg.norm(b))

doc_a = np.array([1.0, 2.0, 0.0])
doc_b = np.array([2.0, 4.0, 0.0])   # same direction as doc_a, just longer
doc_c = np.array([0.0, 0.0, 5.0])   # perpendicular to doc_a

print(cosine_similarity(doc_a, doc_b))   # 1.0  -- identical direction
print(cosine_similarity(doc_a, doc_c))   # 0.0  -- unrelated

4. Probability Foundations

You'll go deeper on probability as it comes up (evaluation metrics in Week 6, sampling in Week 17), but a few ideas are worth having solid now, because they show up immediately once you start describing data or a model's confidence.

• Mean — the average value; the "center" of your data
• Standard deviation — how spread out the values typically are from that center
• Random variable — a quantity whose value comes from some uncertain process (a die roll, tomorrow's stock price, a model's next-token prediction)
• Probability distribution — an assignment of likelihood to every possible value a random variable can take, always summing (or integrating) to 1

scores = np.array([72, 85, 90, 61, 78, 95, 88])

mean = scores.mean()          # 81.28...
std = scores.std()            # 10.9... -- typical distance from the mean

normalized = (scores - mean) / std   # z-scores: how many std-devs from the mean

A small standard deviation means most values cluster tightly around the mean; a large one means they're spread wide. You'll use exactly this pair — mean and standard deviation — to normalize features before training a model in Week 5, and to interpret how "confident" a distribution over outcomes is: a tight distribution means a model is sure; a wide one means it's genuinely uncertain.

5. Conditional Probability & Bayes' Theorem

Conditional probability, written P(A | B), is "the probability of A, given that we already know B happened." Bayes' theorem is the formula for flipping a conditional probability around — going from P(evidence | hypothesis), which is usually easy to estimate, to P(hypothesis | evidence), which is usually what you actually want to know:

P(H | E) = P(E | H) * P(H) / P(E)

# H = hypothesis (e.g. "this email is spam")
# E = evidence   (e.g. "this email contains the word 'free'")

Worked example — a classic spam filter setup:

# Priors and likelihoods, estimated from a labeled dataset
p_spam = 0.30                    # P(spam)  -- 30% of all mail is spam
p_ham = 0.70                     # P(not spam)
p_free_given_spam = 0.60         # P("free" appears | spam)
p_free_given_ham = 0.05          # P("free" appears | not spam)

# P(E) -- total probability the word "free" appears at all, spam or not
p_free = (p_free_given_spam * p_spam) + (p_free_given_ham * p_ham)

# Bayes' theorem: flip P(E | H) into P(H | E)
p_spam_given_free = (p_free_given_spam * p_spam) / p_free

print(round(p_spam_given_free, 3))   # 0.837 -- 83.7% spam, given it contains "free"

Notice the result (84%) is much higher than the raw prior (30%) — seeing the word "free" is strong evidence, and Bayes' theorem is precisely the mechanism for updating a belief once new evidence arrives. This exact update rule — prior belief, new evidence, updated belief — is the conceptual backbone behind how you'll reason about model evaluation and calibration in Week 6.

6. From Raw Scores to a Probability Distribution: Softmax

A model rarely outputs a probability directly — it outputs raw, unbounded numbers called logits (one per possible outcome). Softmax is the function that turns any list of logits into a valid probability distribution: every value between 0 and 1, and the whole set summing to exactly 1.

def softmax(logits):
    shifted = logits - np.max(logits)   # subtract max first -- numerical stability
    exp_scores = np.exp(shifted)
    return exp_scores / np.sum(exp_scores)

logits = np.array([2.0, 1.0, 0.1])
probs = softmax(logits)

print(probs)          # [0.659, 0.242, 0.099]
print(probs.sum())    # 1.0 -- always, by construction

Two details worth noticing: exponentiating first means larger logits become disproportionately more likely — softmax exaggerates differences rather than preserving them linearly. And subtracting np.max(logits) before exponentiating doesn't change the final result (it cancels out in the division) but keeps the numbers from overflowing — a real numerical-stability trick you'll see in nearly every from-scratch transformer implementation in Week 16.

7. Hands-on Exercise

import numpy as np

# Vocabulary, in order: [cat, dog, python, code, bark, meow]
docs = {
    "d1": np.array([2, 0, 1, 1, 0, 1]),   # about a cat that codes in python
    "d2": np.array([0, 3, 0, 0, 2, 0]),   # about a dog that barks
    "d3": np.array([1, 1, 2, 3, 0, 0]),   # about python code, briefly mentions both animals
    "d4": np.array([0, 0, 3, 3, 0, 0]),   # purely about python code
    "d5": np.array([3, 0, 0, 0, 0, 2]),   # purely about a cat meowing
}

query = np.array([0, 0, 2, 2, 0, 0])       # a search for "python code"

8. Knowledge Check

Four quick questions. Expand each to check your answer.