1. Object vs. Block vs. File Storage
Three different storage abstractions, each solving a different access pattern — picking the wrong one is a common design mistake worth heading off first:
- Block storage — raw, fixed-size blocks of data, addressed by block number, with no built-in concept of a "file." This is what a database's own disk volume typically sits on (Week 4-5) — the database software imposes structure on top of raw blocks.
- File storage — data organized in a hierarchical directory structure (folders and files), accessed via a filesystem API (open, read, write, close). Familiar, but the directory hierarchy becomes a scaling bottleneck at massive scale.
- Object storage — data stored as flat, independent "objects," each with a unique key and metadata, accessed over HTTP (GET/PUT by key), with no hierarchy to navigate. This is what S3, Google Cloud Storage and this week's design target actually are.
Object storage's flat structure — no real directories, just keys that happen to contain slashes for human readability — is precisely what lets it scale to trillions of objects: there's no directory-tree bottleneck to contend with, and any object can be looked up directly by its key without walking a hierarchy, one flat lookup instead of several nested ones.
2. Chunking Large Files
A 2GB video file isn't stored as one contiguous blob — it's split into fixed-size chunks (commonly 4-64MB each), and the chunks are what actually gets distributed across storage servers.
video.mp4 (2GB) --> split into 32 chunks of 64MB each
Chunk 1 --> stored on [Storage Node 14, 22, 51] (replicated,
Chunk 2 --> stored on [Storage Node 3, 40, 51] Section 4)
Chunk 3 --> stored on [Storage Node 8, 14, 33]
...
Chunk 32 --> stored on [Storage Node 9, 22, 44]
A manifest records: video.mp4 = [chunk_1_id, chunk_2_id, ...,
chunk_32_id], in order
Chunking solves several problems at once. No single storage node needs to hold an entire large file, so files can be distributed across many nodes' available capacity rather than requiring one node with 2GB free. Chunks from one file can be fetched in parallel from different nodes, speeding up downloads. And critically for a real product: uploads become resumable — if an upload fails at chunk 20 of 32, only the remaining chunks need to be sent, not the whole 2GB file again from scratch.
Smaller chunks mean more resilient resumable uploads and finer-grained parallelism, but more chunks per file means more metadata to track and more network round-trips to assemble a file. Larger chunks mean less overhead but a failed upload loses more progress and re-fetching one chunk to fix corruption moves more data. Naming this tradeoff, rather than picking a chunk size arbitrarily, is what an interview is actually listening for.
3. Separating Metadata from Data
A file's actual bytes and the information about the file — its name, owner, size, upload date, and (from Section 2) which chunks it's made of — are stored in two completely separate systems, deliberately.
[Metadata Service] -- a normal database (Week 4-5), stores:
file_id, owner_id, filename, size, created_at,
chunk_ids: [c1, c2, ..., c32]
-- small records, queried constantly ("list my files"),
benefits from indexes, joins, transactions
[Chunk Storage] -- a massive, distributed key-value store of
raw chunk bytes, keyed by chunk_id
-- huge volume, simple access pattern (get/put by key),
doesn't need any of a relational database's features
This split exists because metadata and chunk data have almost opposite access patterns and storage needs: metadata is small, relational, and frequently queried with filters ("show me files modified this week") — a great fit for a normal database. Chunk data is enormous in volume, accessed by a single key, and never needs a join or a filter — a great fit for a much simpler, purpose-built key-value store optimized purely for storing and retrieving huge binary blobs cheaply. "Upload a file" becomes: write the chunks to chunk storage, then write one metadata record pointing at them — two very different systems, each doing what it's actually good at.
4. Deep Dive: Replication vs. Erasure Coding
Disks fail. At the scale of millions of disks, failures happen constantly, not rarely — durability has to be designed for, not hoped for. Two strategies keep a chunk recoverable after a disk (or several) fail.
Chunk stored as 3 full copies, on 3 different disks/racks/
regions:
[Copy 1] [Copy 2] [Copy 3]
Storage overhead: 3x the original data size
Can survive: any 2 of the 3 copies being lost
Recovery: read one of the surviving full copies directly
Chunk split into 6 data fragments + 3 parity fragments
(computed via an algorithm like Reed-Solomon), spread across
9 disks:
[D1][D2][D3][D4][D5][D6][P1][P2][P3]
Storage overhead: only 1.5x the original data size
(9 fragments store 6 fragments' worth of data)
Can survive: any 3 of the 9 fragments being lost (the parity
fragments let the missing data be MATHEMATICALLY
RECONSTRUCTED, not just copied from elsewhere)
Recovery: read the surviving fragments, compute the missing
ones -- real CPU cost, unlike replication's simple copy
The tradeoff is explicit and worth stating in exactly these terms: replication trades storage space for simplicity (3x the storage, but recovery is just reading a spare copy); erasure coding trades computation for storage efficiency (1.5x the storage, but recovery requires a real mathematical reconstruction). At the petabyte scale object storage operates at, that difference between 3x and 1.5x storage overhead is an enormous, real cost difference — which is why large-scale object storage systems typically use erasure coding for infrequently-accessed ("cold") data, and sometimes still use replication for frequently-accessed ("hot") data where recovery speed and read simplicity matter more than storage cost.
Neither replication nor erasure coding alone protects against an entire data center failing — the copies/fragments still need to be spread across separate failure domains (different racks, different data centers, different regions, foreshadowing Week 22) so a single facility-level outage doesn't take out enough copies/fragments at once to lose the data entirely.
5. Hands-on Exercise
Design file storage for a photo backup app
A photo backup app lets users automatically upload photos from their phone. Photos average 5MB; some users have 50,000+ photos.
Requirements:
- Choose object, block or file storage (Section 1) for the actual photo bytes, and justify the choice against this app's access pattern.
- Decide whether photos this small (5MB average) should be chunked at all (Section 2) — justify your answer against the tradeoffs discussed, not just "chunking is always good."
- Design the metadata schema (Section 3) for one photo record, including at minimum the fields needed to support "show me all my photos from July, sorted newest first."
- Choose replication or erasure coding (Section 4) for this app's photo storage, and justify it against how frequently old photos are typically re-accessed versus how quickly a user expects an upload to be durably saved.
- A user's phone loses connection halfway through uploading a 5MB photo. Explain what happens next, and whether your chunking decision from requirement 2 affects the answer.
For requirement 2: chunking's benefits (parallel transfer, resumability) matter far more for large files than small ones — a 5MB photo re-upload after a dropped connection is cheap enough that treating each photo as one indivisible unit is often the simpler, equally correct choice. Chunking earns its complexity specifically at large file sizes.
6. Knowledge Check
Four quick questions. Expand each to check your answer.
Q1
Why does object storage's flat, hierarchy-free key structure scale better than file storage's nested directory structure at massive scale?
Why does object storage's flat, hierarchy-free key structure scale better than file storage's nested directory structure at massive scale?
A directory hierarchy requires walking through nested levels to resolve a path, and at massive scale that tree structure itself becomes a bottleneck. Object storage looks up any object directly by its flat key, with no tree to traverse, which is what lets it scale to trillions of objects without a hierarchy-induced bottleneck.
Q2
Why does chunking a large file make an interrupted upload resumable, rather than requiring the whole file to be re-sent?
Why does chunking a large file make an interrupted upload resumable, rather than requiring the whole file to be re-sent?
Since each chunk is uploaded and stored independently, the system can track which chunks were already successfully received before the interruption. Resuming just means sending the remaining not-yet-received chunks, rather than starting the entire large file's transfer over from byte zero.
Q3
Why are a file's metadata and its actual chunk data stored in two completely separate systems rather than one?
Why are a file's metadata and its actual chunk data stored in two completely separate systems rather than one?
Metadata is small, relational, and frequently queried with filters and joins — a good fit for a normal database. Chunk data is enormous in volume and only ever accessed by a single key, with no need for relational features — a good fit for a much simpler, purpose-built key-value store. Their access patterns and scale characteristics are different enough that one system optimized for both would do neither well.
Q4
What does erasure coding trade away to achieve lower storage overhead than 3x replication?
What does erasure coding trade away to achieve lower storage overhead than 3x replication?
Recovering lost data with erasure coding requires computing missing fragments from the surviving data and parity fragments — a real CPU cost — whereas replication's recovery is just reading an already-complete spare copy directly. Erasure coding trades computation at recovery time for a significantly smaller storage footprint (e.g. 1.5x instead of 3x).