1. Requirements & Estimation
Scope this the way Week 1 taught: a small functional list, a demanding non-functional one, and numbers before any boxes get drawn.
Functional:
- A user can create a post (text, optionally an image)
- A user can follow other users
- A user can view a personalized feed of posts from people
they follow, ordered newest-first
Non-functional:
- Scale: 200M daily active users (DAU)
- Average user follows ~200 accounts; a small number of
"celebrity" accounts have 10M+ followers each
- Posts: 100M new posts/day
- Feed reads are extremely frequent: each DAU checks their
feed ~20 times/day on average
- Latency: feed load must feel instant, well under 200ms
- Consistency: eventual is fine -- a post appearing in a
follower's feed a few seconds late is an acceptable tradeoff
for availability and speed, unlike Week 1's URL shortener
where a broken redirect was never acceptable
Feed reads:
200,000,000 DAU * 20 checks/day = 4,000,000,000 reads/day
4,000,000,000 / 86,400 sec ≈ 46,000 reads/second average
Peak (usage concentrated in a few hours) ≈ 46,000 * 3
≈ 140,000 reads/second
Post writes:
100,000,000 posts/day / 86,400 sec ≈ 1,150 writes/second average
Fan-out writes (the number that actually shapes this design):
average follower count ~200 -> 1,150 * 200
= 230,000 feed-cache writes/second
-- and that's before accounting for the celebrity accounts,
which push this number far higher per individual post
(Section 4)
46,000 reads/second is already high, but the fan-out number is the one that should reshape the design: a single post doesn't cost one write, it costs one write per follower, and the follower-count distribution is wildly skewed — most users have a few hundred followers, a handful have tens of millions. A design that treats every post as equally cheap to fan out hasn't understood this system yet, which is exactly what Sections 2 and 4 address.
2. Fan-out on Write vs. Fan-out on Read
There are two fundamentally different ways to get a post from its author to a follower's feed, and the tradeoff between them is one of the most common deep-dive topics in a news-feed interview.
User posts --> system immediately pushes the post ID into a
precomputed feed list for EVERY follower
(stored in a cache, e.g. a Redis list per user)
Read a feed: just read that user's precomputed list -> fast,
O(1) lookup, exactly the <200ms target
Write a post: cost is proportional to follower count -- fine
for a user with 200 followers, catastrophic for
one with 10 million (Section 4)
User posts --> system just stores the post. Nothing else happens.
Read a feed: query posts from all ~200 followed accounts,
merge and sort by time on the fly -> slow, and
gets slower the more accounts a user follows
Write a post: cheap, O(1) regardless of follower count
Fan-out on write optimizes the far more frequent operation — this system does 46,000 reads/second against 1,150 writes/second, roughly a 40:1 ratio, so paying a write-time cost to make every read instant is usually the right trade. But "usually" is doing real work in that sentence: it breaks down exactly where the estimation numbers in Section 1 flagged it would, for accounts with enormous follower counts. Section 4 resolves that with a hybrid of both strategies, which is the answer real large-scale feeds actually use rather than picking one extreme.
Neither strategy is categorically better — the right answer depends on the read:write ratio and the shape of the follower-count distribution, both of which came directly out of Section 1's estimation. Presenting this as "it depends, and here's specifically what it depends on" is a stronger answer than confidently picking one and defending it as universally correct.
3. Deep Dive: The Fan-out Pipeline
Committing to fan-out on write for most accounts means designing the actual pipeline that executes it, and this is where Weeks 5, 6–7 and 8 combine into one system rather than staying separate topics.
[Author] --post--> [Post Service] --> [Post Store, sharded by author_id]
|
+--> publish fanout job --> [Message Queue]
|
[Fan-out Workers] (pool, scales
independently of Post Service)
|
for each follower:
push post_id into their
[Feed Cache] (Redis list,
sharded by user_id, capped
to most recent ~500 items)
[Reader] --load feed--> [Feed Service] --> reads own Feed Cache
entry directly -> O(1)
The queue is doing the same job it does everywhere in this course: decoupling the
author's post-creation request (which must return quickly) from the actual fan-out work
(which can take much longer for a well-followed account), and absorbing bursts — a
flood of posts during a major live event doesn't back up the Post Service, it just
grows the queue's backlog for the workers to drain. The Post Store is
sharded by author_id (Week 5) so that no single database instance owns all
100M daily posts; the Feed Cache is sharded by user_id for
the same reason on the read side, and it's capped in size (Week 7's eviction policies)
since no one needs their precomputed feed to hold every post they've ever been eligible
to see, only the most recent few hundred.
One correctness detail the pipeline has to handle: message queues typically guarantee
at-least-once delivery (Week 8), meaning a fan-out worker could process the same job
twice after a retry. Pushing a duplicate post_id into a follower's feed
cache would show the same post twice — the fix is making the fan-out write idempotent,
inserting into a set-like structure keyed by post_id (or checking for
existence before pushing) so a retried job has no visible effect the second time.
Any time a queue with at-least-once delivery appears in a design, the very next question should be "what happens if this job runs twice?" For fan-out specifically, an unprotected double-processing bug is directly user-visible — a duplicated post in someone's feed — which makes this a detail worth naming unprompted rather than waiting for the interviewer to ask.
4. Deep Dive: The Celebrity Problem & Hybrid Fan-out
Section 1 flagged the number that breaks pure fan-out on write: a celebrity account with 10 million followers posting once means 10 million feed-cache writes for that one post, on top of whatever else the fan-out workers are already processing.
Celebrity account: 10,000,000 followers
Fan-out worker pool throughput: ~50,000 writes/second (aggregate)
Time to fully fan out ONE post:
10,000,000 / 50,000 ≈ 200 seconds (over 3 minutes)
During those 3+ minutes, this single post is competing for
worker capacity against every other post in the system --
a hot key/thundering write problem, the write-side mirror of
the hot-key read problem from Week 3.
Fan-out on read has the opposite problem for this exact case: pulling from 10 million followers isn't the issue (that's a write-time problem), but a typical reader who follows one celebrity among their ~200 follows doesn't want to wait on a slow merge either. The standard resolution is a hybrid: apply fan-out on write for accounts under a follower threshold (say, 100,000), and skip fan-out entirely for accounts above it. At read time, a user's feed is assembled by combining their precomputed fan-out-on-write feed cache with a small number of direct, real-time lookups — one per celebrity account they follow — merged by timestamp before the response is returned.
User follows 200 accounts: 197 regular, 3 celebrities
Feed load:
1. Read precomputed Feed Cache (already has the 197 regular
accounts' posts merged in, via Section 3's pipeline)
2. Fetch the 3 celebrities' most recent posts directly
(celebrity accounts get their own lightweight cache of
"my last 20 posts," cheap to maintain since it's written
once per post, not fanned out)
3. Merge both lists by timestamp, return the top N
Cost at read time: 1 cache read + 3 small direct lookups --
still fast, and the celebrity's post never required 10 million
writes to become visible.
This is the same shape as Week 9's CAP framing applied to a design choice rather than a formal theorem: there's no version of this system that's cheap on both the write side and the read side for every account size simultaneously, so the hybrid design explicitly pays a slightly more expensive read (a few extra lookups) in exchange for making the write side tractable for the accounts that would otherwise break it — a small, bounded cost paid on every read, instead of an unbounded one paid occasionally on write.
The "100,000 followers" cutoff between fan-out-on-write and fan-out-on-read isn't a fixed industry number — it's a knob tuned against the actual worker throughput and read latency budget of the system it's protecting. Stating that it's a tunable threshold, and naming what you'd measure to tune it (fan-out worker saturation, p99 read latency for accounts near the boundary), reads as stronger system thinking than presenting a specific number as gospel.
5. Hands-on Exercise
Design the "just followed 500 people" backfill problem
A new user signs up and, in one session, follows 500 accounts at once (a common onboarding flow: "follow these suggested accounts"). Their feed cache from Section 3 is currently empty — pure fan-out on write only populates a feed going forward, from posts made after a follow relationship exists.
Requirements:
- Explain, in 2–3 sentences, why fan-out on write alone leaves this new user's feed empty even though they now follow 500 active accounts.
- Propose a fix using one of the two strategies from Section 2 (fan-out on write vs. fan-out on read), applied specifically to this one-time backfill case rather than as this user's permanent feed strategy.
- Decide whether this backfill should block the "follow 500 accounts" action from completing, or happen asynchronously — justify against this week's <200ms feed-load latency target.
- If asynchronous, name the component from Section 3 you'd reuse for this, and describe what the job it processes actually does differently from a normal fan-out job.
- Write one sentence on what a user sees in their feed during the window before the backfill completes, and whether that's an acceptable experience given this system's eventual-consistency stance from Section 1.
This is a one-time, bounded piece of work (500 accounts' worth of recent posts, once), not a permanent fan-out-on-read fallback for this user — think about it as a background job that pulls recent history and writes it into the same Feed Cache Section 3 already reads from, so the read path itself never needs to know the difference.
6. Knowledge Check
Four quick questions. Expand each to check your answer.
Q1
Why does fan-out on write make sense for this system's read:write ratio, and what does it cost in return?
Why does fan-out on write make sense for this system's read:write ratio, and what does it cost in return?
With roughly 40x more feed reads than posts written, paying a cost at write time to make every read an O(1) cache lookup optimizes the operation that happens far more often, keeping the dominant traffic pattern fast. The cost is that a single post's write cost is no longer constant — it scales with the author's follower count, which becomes untenable for accounts with very large followings.
Q2
Why is a message queue placed between the Post Service and the fan-out workers, rather than fanning out synchronously when a post is created?
Why is a message queue placed between the Post Service and the fan-out workers, rather than fanning out synchronously when a post is created?
Fanning out synchronously would make the author wait for every follower's cache to be updated before their post-creation request even returns, and that wait time scales with follower count — unacceptable for anyone with a large following. The queue decouples the two: the Post Service returns as soon as the post is stored and a fan-out job is enqueued, while the fan-out workers process that job independently and can scale their own throughput separately from post-creation traffic.
Q3
Why does a hybrid fan-out strategy handle the celebrity problem better than either pure fan-out on write or pure fan-out on read?
Why does a hybrid fan-out strategy handle the celebrity problem better than either pure fan-out on write or pure fan-out on read?
Pure fan-out on write makes a celebrity's single post cost millions of writes, which can take minutes to complete and starves the worker pool for everyone else; pure fan-out on read makes every feed load slow, even for users following mostly regular accounts. The hybrid keeps fan-out on write for the common case (most accounts have modest follower counts) and switches to a small, bounded number of direct read-time lookups only for the rare high-follower accounts, avoiding both failure modes.
Q4
Why must the fan-out worker's write into a follower's feed cache be idempotent?
Why must the fan-out worker's write into a follower's feed cache be idempotent?
Message queues typically guarantee at-least-once delivery, meaning the same fan-out job can be redelivered and processed twice after a retry. Without an idempotent write — checking whether the post_id is already present before inserting, or writing into a set-like structure that can't hold duplicates — a redelivered job would insert the same post into a follower's feed twice, a directly user-visible bug.