1. Leader Election, Mechanically
Many systems need exactly one node to act as "the leader" for some responsibility — Week 15's saga orchestrator, a database's primary that accepts writes, a job scheduler deciding which worker runs a task. If no node is elected leader, nothing happens; if two nodes both believe they're the leader, that's Section 3's split-brain problem. A consensus algorithm (Raft is the one most production systems use today) solves this.
All nodes start as FOLLOWERS.
Each follower has a randomized election timeout (e.g. 150-300ms).
If a follower doesn't hear from a leader before its timeout
fires, it becomes a CANDIDATE and requests votes from every
other node.
A candidate that receives votes from a MAJORITY of all nodes
(not just a majority of respondents -- an actual majority of
the whole cluster) becomes the new LEADER.
The leader sends periodic heartbeats to all followers, resetting
their election timeouts -- as long as heartbeats keep arriving,
no new election starts.
The randomized timeout is the detail that makes this actually work: if every follower used the same fixed timeout, they'd all become candidates simultaneously and split the vote repeatedly, with no majority ever forming. Randomizing means one follower's timer almost always fires first, becomes a candidate, and typically wins before others even start an election — a small, deliberate detail that resolves what would otherwise be a recurring tie.
The majority requirement is what guarantees at most one leader can be elected at a time: two different candidates cannot both receive a majority of votes from the same fixed set of nodes simultaneously — any two majorities of the same group must overlap in at least one node, and that node can only vote for one of them.
2. Distributed Locks
A distributed lock extends the familiar idea of a mutex (Week 12's LLD module) across multiple machines: ensure only one process, anywhere in the system, holds a given lock at a time — useful for "only one worker should process this job," exactly the kind of guarantee Week 15's saga steps or a scheduled cron job running on multiple redundant workers needs.
Worker A: SET lock:job-123 "worker-A" NX EX 30
-- acquires the lock, expires automatically after 30s
-- (the expiry exists so a crashed worker doesn't hold
-- the lock forever)
Worker A: ... starts processing job-123, but is unexpectedly
paused for 35 seconds (a long GC pause, a slow
disk write, a network partition)
t=30s: lock:job-123 expires automatically
Worker B: SET lock:job-123 "worker-B" NX EX 30
-- acquires the SAME lock, believing it's exclusive
Worker A: (resumes) finishes processing, unaware it lost the
lock -- NOW BOTH WORKERS THINK THEY OWN job-123
This is a genuinely dangerous, easy-to-miss bug: the lock's expiry (needed so a crashed worker doesn't hold it forever) directly conflicts with the guarantee the lock exists to provide, if a worker is merely slow rather than actually dead. The fix requires a fencing token — a monotonically increasing number issued with each lock acquisition, which the downstream resource (the database, the job queue) itself checks and rejects if a request arrives with a token older than the last one it accepted, catching exactly the scenario above even though the lock service itself was fooled.
Naming the fencing-token problem unprompted in an interview is a strong signal — it shows the difference between "I've used a distributed lock library" and "I understand why a distributed lock, by itself, doesn't guarantee mutual exclusion under real-world timing." The lock reduces the likelihood of a conflict; the fencing token at the resource itself is what actually prevents the conflict from causing damage.
3. Deep Dive: The Split-Brain Problem
Split-brain is what happens when a network partition (Week 9-10's CAP theorem, made concrete) splits a cluster into two groups, each of which believes it's the "real" cluster and elects its own leader — two leaders, both accepting writes, with no way to know about each other until the partition heals.
Before partition: [N1, N2, N3, N4, N5] -- N1 is leader
Network splits into two groups:
Group A: [N1, N2] -- 2 nodes
Group B: [N3, N4, N5] -- 3 nodes
Group A: N1 still believes it's the leader (can't reach N3-N5
to know otherwise)
Group B: has 3 of 5 nodes -- a MAJORITY -- elects a new leader,
say N3
Result: N1 and N3 BOTH believe they're the leader, simultaneously
accepting writes in their own partition -- exactly the
conflict Section 1's majority rule was supposed to prevent
The majority requirement from Section 1 is exactly what resolves this, given enough nodes: Group A (2 of 5 nodes) cannot elect a new leader, because 2 is not a majority of 5 — it can only ever have its existing leader, N1, which itself should step down once it can no longer reach a majority of the cluster to confirm its leadership (a well-implemented Raft leader does exactly this). Group B (3 of 5, a genuine majority) can safely elect N3. The system as a whole degrades to read-only or unavailable in the minority partition, rather than allowing two simultaneous writers — trading availability in the minority side for correctness, precisely the CP choice in Week 9-10's CAP framing.
This is also why coordination clusters are near-universally deployed with an odd number of nodes (3, 5, 7): an even-numbered cluster split exactly in half has no majority on either side at all, which can leave the entire system leaderless and unavailable until the partition heals — strictly worse than an odd-sized cluster, where one side always has a clear majority.
4. Coordination Services in Practice
Implementing Raft or Paxos correctly is notoriously hard to get right — subtle timing bugs are easy to introduce and hard to catch in testing. Real systems almost never implement consensus themselves; they depend on a dedicated coordination service (ZooKeeper, etcd, Consul) that has already solved it, and use it for exactly the primitives this week covered.
Leader election: "which node in my cluster is currently
the leader?" -- the coordination service
tracks this, using Section 1's algorithm
internally, so individual services don't
have to implement it themselves
Service discovery: Week 16's registry of "which instances of
Inventory Service are currently healthy"
is commonly backed by exactly this kind of
service
Distributed locks: Section 2's lock, with the fencing-token
detail usually built in as a first-class
feature, not something each team reinvents
Configuration: a single source of truth for config values
that many services need to read consistently
The pattern worth internalizing: consensus is a genuinely hard, easy-to-get-subtly- wrong problem, so the correct engineering answer is almost always "use an existing, battle-tested coordination service," not "implement Raft from the description in this lesson." Recognizing when a design needs one of these primitives — and naming the coordination service as the standard tool for it — is the actual skill an interview is testing, not the ability to reimplement consensus from scratch.
5. Hands-on Exercise
Add safe leader election to a job scheduler
A distributed job scheduler runs 5 redundant scheduler instances (for availability) but must ensure only one of them actually dispatches jobs at a time.
Requirements:
- Explain, using Section 1's mechanics, how the 5 instances agree on which one is the active leader without a human ever configuring it manually.
- Explain what happens if a network partition splits the 5 instances into a group of 2 and a group of 3, tracing through Section 3's reasoning for each group.
- Design the distributed lock the active leader uses when dispatching a specific job to a worker, including the fencing-token fix from Section 2 — describe concretely what the worker checks before actually executing the job.
- Decide whether this scheduler should implement its own leader election from scratch or use an existing coordination service (Section 4), and justify the choice.
- State why deploying exactly 5 instances (rather than 4 or 6) is a deliberate choice, not an arbitrary one.
For requirement 3: the worker itself should reject a job-execution request carrying a fencing token older than the last one it accepted for that job — that check has to live at the worker (or wherever the job's actual side effect happens), not just at the lock service, because the lock service is exactly what can be fooled by a slow-but-not-dead leader.
6. Knowledge Check
Four quick questions. Expand each to check your answer.
Q1
Why does Raft use a randomized (not fixed) election timeout for followers?
Why does Raft use a randomized (not fixed) election timeout for followers?
A fixed timeout would cause every follower to become a candidate at exactly the same moment, repeatedly splitting the vote with no majority ever forming. Randomizing the timeout means one follower's timer almost always fires first, letting it typically win the election before other followers even become candidates.
Q2
In the naive distributed lock example, why does the lock's automatic expiry create a bug rather than just a safety feature?
In the naive distributed lock example, why does the lock's automatic expiry create a bug rather than just a safety feature?
The expiry can't distinguish between "the worker crashed" (where releasing the lock is correct) and "the worker is merely slow" (a long pause, then it resumes). If a worker is only slow, the lock expiring and being granted to another worker creates a window where two workers both believe they exclusively hold it — the safety mechanism (expiry) and the guarantee (exclusivity) are in direct tension.
Q3
In a 5-node cluster split into a 2-node group and a 3-node group by a partition, why can only the 3-node group elect a leader?
In a 5-node cluster split into a 2-node group and a 3-node group by a partition, why can only the 3-node group elect a leader?
Leader election requires a majority of the entire cluster's nodes (3 of 5), not just a majority of nodes the candidate can currently reach. The 2-node group can never reach 3 votes no matter what it does, so it structurally cannot elect a new leader — only the 3-node group has enough nodes to form a genuine majority.
Q4
Why do most real systems use an existing coordination service (ZooKeeper, etcd) rather than implementing Raft or Paxos themselves?
Why do most real systems use an existing coordination service (ZooKeeper, etcd) rather than implementing Raft or Paxos themselves?
Consensus algorithms are notoriously easy to get subtly wrong, with timing bugs that are hard to catch in normal testing and only surface under specific, rare network conditions in production. Using a battle-tested, already-correct coordination service avoids reimplementing a genuinely hard, high-stakes piece of distributed systems engineering from scratch.