Week 3: Statistics for Machine Learning

1. Descriptive Statistics

You already met mean and standard deviation in Week 2. This week starts by rounding out the toolkit for describing a dataset — because every one of these numbers can mislead you if you reach for the wrong one.

import numpy as np

salaries = np.array([42000, 45000, 47000, 44000, 46000, 250000])

mean = salaries.mean()          # 79000 -- dragged upward by one outlier
median = np.median(salaries)    # 45500 -- much more representative here
p90 = np.percentile(salaries, 90)   # 90th percentile

The mean and the median tell you different things, and they diverge exactly when it matters: with one extreme outlier (a single $250,000 salary), the mean jumps to $79,000 — a number that doesn't describe anyone in the dataset — while the median stays close to the typical value. Skewed data (like income, latency, or most real-world durations) is where this gap shows up most.

2. Probability Distributions

A distribution tells you how likely each possible value of a random variable is. You don't need to memorize formulas — you need to recognize the shape and know when each one shows up.

• Normal (Gaussian) — the classic bell curve; sums and averages of many independent effects tend toward this shape
• Binomial — the count of successes across a fixed number of yes/no trials (e.g. clicks out of 1,000 impressions)
• Poisson — the count of events in a fixed window when events happen independently at some average rate (e.g. support tickets per hour)

import numpy as np

rng = np.random.default_rng(seed=42)

# Simulate 1,000 coin flips (a Bernoulli trial) repeated 500 times
binomial_sample = rng.binomial(n=1000, p=0.5, size=500)

# Simulate ticket arrivals averaging 4 per hour, over 500 hours
poisson_sample = rng.poisson(lam=4, size=500)

print(binomial_sample.mean(), binomial_sample.std())   # ~500, ~15.8
print(poisson_sample.mean(), poisson_sample.std())     # ~4, ~2

Notice the Poisson distribution's mean and variance are both close to 4 — that's not a coincidence, it's a defining property of the distribution (variance = mean). Recognizing which distribution a real quantity follows is exactly what lets you compute a sensible confidence interval for it in Section 5.

3. Sampling & the Central Limit Theorem

You almost never have access to an entire population — you have a sample, and you use it to estimate something about the population it came from. The Central Limit Theorem (CLT) is the single result that makes this estimation trustworthy: the distribution of a sample mean, taken across many repeated samples, approaches a normal distribution as sample size grows — regardless of the shape of the original data.

import numpy as np

rng = np.random.default_rng(seed=0)

# The underlying population is heavily skewed -- NOT normal at all
population = rng.exponential(scale=2.0, size=100_000)

# Repeatedly draw samples of size 40 and record each sample's mean
sample_means = np.array([
    rng.choice(population, size=40, replace=False).mean()
    for _ in range(1000)
])

print(population.std())      # population is skewed and spread out
print(sample_means.std())    # sample means cluster much more tightly, and look normal

This is why standard error — the standard deviation of the sample mean itself, approximated as std / sqrt(n) — shrinks as you collect more data: individual observations stay just as noisy, but their average becomes an increasingly precise estimate. Every confidence interval and hypothesis test in the rest of this lesson leans on this one fact.

4. Hypothesis Testing

A hypothesis test asks a narrow, specific question: "if there were truly no difference, how surprising would this observed result be?" The null hypothesis (H₀) is "no real effect"; the p-value is the probability of seeing a result at least this extreme, assuming H₀ is true.

from scipy import stats
import numpy as np

# Conversion rate: control vs. a new button color
control = np.array([0, 1, 0, 0, 1, 0, 0, 0, 1, 0] * 50)     # 15% conversion, n=500
treatment = np.array([1, 1, 0, 1, 0, 0, 1, 0, 1, 0] * 50)   # 25% conversion, n=500

t_stat, p_value = stats.ttest_ind(control, treatment)

print(f"p-value: {p_value:.4f}")
# A common (not universal) threshold: p < 0.05 is called "statistically significant"

A small p-value means the observed difference would be unlikely under "no real effect" — evidence against the null hypothesis, not proof of the treatment's size or importance. This distinction trips people up constantly: a p-value doesn't tell you the probability the treatment is better, and a statistically significant result from a tiny effect size can still be practically meaningless with a large enough sample.

5. Confidence Intervals

A confidence interval (CI) gives a range instead of a single-point estimate, along with a stated confidence level — most commonly 95%. The correct reading is subtle: a 95% CI means that if you repeated the entire sampling process many times, about 95% of the intervals you'd construct would contain the true population value — not "there's a 95% chance the true value is in this specific interval."

import numpy as np
from scipy import stats

sample = np.array([4.2, 3.8, 5.1, 4.6, 4.9, 3.6, 4.4, 5.0, 4.1, 4.7])

mean = sample.mean()
standard_error = sample.std(ddof=1) / np.sqrt(len(sample))

ci_low, ci_high = stats.t.interval(
    confidence=0.95, df=len(sample) - 1, loc=mean, scale=standard_error
)

print(f"Mean: {mean:.2f}, 95% CI: [{ci_low:.2f}, {ci_high:.2f}]")

Notice the direct link to Section 3: the interval's width is driven by the standard error, which shrinks as sample size grows. A wide confidence interval isn't just an inconvenience — it's an honest signal that you don't yet have enough data to be precise, and reporting a single mean without it hides that uncertainty entirely.

6. Correlation vs. Causation

Two variables can move together for reasons that have nothing to do with one causing the other: a confounding variable can drive both, the relationship could run backward from what you assumed, or it could simply be coincidence. A correlation coefficient — typically Pearson's r, from -1 to 1 — measures only the strength of a linear association, never the direction of cause and effect.

import numpy as np

ice_cream_sales = np.array([10, 15, 25, 40, 55, 60])
drowning_incidents = np.array([2, 3, 5, 8, 11, 12])

correlation = np.corrcoef(ice_cream_sales, drowning_incidents)[0, 1]
print(correlation)   # close to 1.0 -- strongly correlated

# Obviously, ice cream doesn't cause drowning -- both rise with a confounder: summer heat

This isn't a pedantic distinction — it directly determines what you're allowed to conclude from a model. A feature with high correlation to your target is a candidate predictor, not automatically a lever you can pull to change the outcome; confusing the two is one of the most common ways an ML-driven business decision goes wrong.

7. Hands-on Exercise

import numpy as np

rng = np.random.default_rng(seed=7)

# 1 = converted, 0 = did not convert
control = rng.binomial(1, 0.11, size=1200)      # existing checkout page
treatment = rng.binomial(1, 0.135, size=1200)   # redesigned checkout page

8. Knowledge Check

Four quick questions. Expand each to check your answer.