Week 7: Unsupervised Learning & Dimensionality Reduction

1. k-Means Clustering & Choosing k

k-means partitions data into k groups by iterating two steps: assign every point to its nearest of k centroids, then move each centroid to the mean of the points assigned to it. Repeat until assignments stop changing. It's simple, fast, and the default first thing to try on unlabeled numeric data — but it requires you to choose k up front, which is rarely obvious.

from sklearn.cluster import KMeans
from sklearn.preprocessing import StandardScaler

# Scaling matters here -- k-means uses distance, exactly like the Week 4 discussion
X_scaled = StandardScaler().fit_transform(X)

model = KMeans(n_clusters=3, n_init=10, random_state=42)
labels = model.fit_predict(X_scaled)   # cluster id (0, 1, 2, ...) per row

Two standard tools for picking k:

from sklearn.metrics import silhouette_score

inertias = []
silhouettes = []

for k in range(2, 9):
    model = KMeans(n_clusters=k, n_init=10, random_state=42).fit(X_scaled)
    inertias.append(model.inertia_)                              # elbow method
    silhouettes.append(silhouette_score(X_scaled, model.labels_)) # silhouette score

# Elbow method: plot k vs inertia, look for where the drop-off flattens
# Silhouette score: ranges -1 to 1, higher is better-separated clusters -- just pick the max

Inertia (sum of squared distances to each point's centroid) always decreases as k grows — even up to k = n, where every point is its own cluster and inertia hits zero. The elbow method looks for the point where adding another cluster stops helping much. Silhouette score measures how much closer each point is to its own cluster than to the nearest other cluster, and doesn't have this "always improves" problem, which usually makes it the more reliable of the two.

2. Hierarchical Clustering

Rather than committing to one fixed k up front, agglomerative hierarchical clustering starts with every point as its own cluster and repeatedly merges the two closest clusters until only one remains — producing a tree (dendrogram) you can then "cut" at any height to get any number of clusters.

from scipy.cluster.hierarchy import linkage, dendrogram, fcluster

# 'ward' linkage minimizes the variance within merged clusters
Z = linkage(X_scaled, method="ward")

# Cut the dendrogram to produce exactly 3 flat clusters
labels = fcluster(Z, t=3, criterion="maxclust")

# dendrogram(Z) -- visualize the full merge tree to help decide where to cut

The dendrogram itself is often the most useful output — the height of each merge shows how dissimilar the merged clusters were, so a long vertical gap in the tree is a visual hint for a natural place to cut. This makes hierarchical clustering a good complement to k-means: use the dendrogram to sanity-check whether the k you chose in Section 1 actually corresponds to a natural break in the data.

3. Principal Component Analysis (PCA)

PCA reduces the number of features while preserving as much of the data's variance (spread, and therefore information) as possible. It works by finding new axes — principal components — that are linear combinations of your original features, ordered so the first component captures the most variance, the second captures the most remaining variance while staying perpendicular to the first, and so on.

from sklearn.decomposition import PCA

pca = PCA(n_components=2)
X_pca = pca.fit_transform(X_scaled)   # from, say, 20 features down to 2

print(pca.explained_variance_ratio_)   # e.g. [0.42, 0.19] -- 42% + 19% = 61% of variance kept

Choosing the number of components is a direct tradeoff: fewer components are easier to visualize and feed into a faster downstream model, but you lose whatever variance those discarded components carried. A common rule of thumb is to keep enough components to retain 90–95% of total variance — PCA(n_components=0.95) does this automatically, choosing however many components are needed to hit that threshold.

4. t-SNE & UMAP for Visualization

PCA is a linear projection — great for preserving global variance and for feeding into another model, but it can miss more complex, curved structure. t-SNE and UMAP are nonlinear techniques built specifically for one job: producing a 2D (or 3D) plot where points that were close together in high-dimensional space stay close together visually, even if that closeness isn't a straight-line relationship.

from sklearn.manifold import TSNE

X_tsne = TSNE(n_components=2, perplexity=30, random_state=42).fit_transform(X_scaled)

# X_tsne is now purely for plotting -- do NOT feed it into a downstream model,
# and do NOT compare distances between points as if they were meaningful

The critical caveat: t-SNE and UMAP optimize for visual cluster separation, not for preserving true distances or feeding a supervised model afterward — cluster sizes and the gaps between clusters in the plot are not reliably meaningful, only which points group together is. Use them purely to look at your data; use PCA (Section 3) when you need a projection with well-defined, reusable numeric meaning.

5. Combining PCA with a Downstream Supervised Model

PCA isn't only for visualization — reducing dozens or hundreds of correlated features down to a handful of components can speed up training, reduce overfitting risk (fewer dimensions, closer to Week 5's bias-variance sweet spot), and remove multicollinearity that destabilizes some models.

from sklearn.pipeline import Pipeline
from sklearn.decomposition import PCA
from sklearn.linear_model import LogisticRegression

pipeline = Pipeline([
    ("scale", StandardScaler()),
    ("pca", PCA(n_components=0.95)),
    ("model", LogisticRegression(max_iter=1000)),
])

# Fits scaling, PCA and the model together -- and with GridSearchCV from Week 6,
# every fold refits PCA correctly within that fold, avoiding leakage

Notice this slots directly into the Pipeline pattern from Week 4 and the GridSearchCV pattern from Week 6 — PCA becomes just another pipeline step that gets fit only on training data within each cross-validation fold, so it never sees validation or test rows during fitting.

6. Hands-on Exercise

from sklearn.datasets import make_blobs

X, _ = make_blobs(
    n_samples=500, n_features=10, centers=4,
    cluster_std=1.8, random_state=42,
)
# Pretend the true cluster labels don't exist -- you only have X

7. Knowledge Check

Four quick questions. Expand each to check your answer.