Week 9: Async/Await Deep Dive & Streams

1. Promise Combinators

When you need to run several async operations together, which combinator you reach for changes your program's behavior in ways that matter. Promise.all waits for every promise to resolve, but rejects as soon as any one of them rejects — the results of the others are simply discarded, even if they eventually would have succeeded.

promise-all-example.ts

async function loadDashboard(userId: string) {
  // If fetchRecommendations rejects, this whole function rejects too --
  // even if fetchProfile and fetchOrders already succeeded.
  const [profile, orders, recommendations] = await Promise.all([
    fetchProfile(userId),
    fetchOrders(userId),
    fetchRecommendations(userId),
  ]);
  return { profile, orders, recommendations };
}

Promise.all is right when every result is required for the operation to make sense. When partial success is acceptable — a dashboard that can render with two out of three widgets — Promise.allSettled is the better fit: it never rejects, and instead resolves with a status for every promise, win or lose.

promise-allsettled-example.ts

async function loadDashboardResilient(userId: string) {
  const results = await Promise.allSettled([
    fetchProfile(userId),
    fetchOrders(userId),
    fetchRecommendations(userId),
  ]);

  const [profileResult, ordersResult, recsResult] = results;

  return {
    profile: profileResult.status === "fulfilled" ? profileResult.value : null,
    orders: ordersResult.status === "fulfilled" ? ordersResult.value : [],
    recommendations: recsResult.status === "fulfilled" ? recsResult.value : [],
  };
}

Promise.race settles as soon as the first promise settles — fulfilled or rejected — and ignores the rest. It's the natural tool for a timeout pattern: race the real work against a promise that rejects after a deadline.

with-timeout.ts

function withTimeout<T>(promise: Promise<T>, ms: number): Promise<T> {
  const timeout = new Promise<never>((_, reject) => {
    setTimeout(() => reject(new Error(`Timed out after ${ms}ms`)), ms);
  });
  return Promise.race([promise, timeout]);
}

// Usage: fails fast if the upstream service takes longer than 2 seconds,
// instead of letting the request hang indefinitely.
const data = await withTimeout(fetchFromSlowUpstream(), 2000);

Race doesn't cancel the loser

Promise.race settles on the first outcome, but the losing promise keeps running in the background — in the timeout example, the real request isn't actually aborted, it just stops being awaited. If you need true cancellation (to stop a real HTTP request, not just ignore its result), you need an AbortController wired into that request as well.

2. Async/Await Error-Handling Pitfalls

The most common async bug in Express code is calling an async function inside Array.prototype.forEach and expecting it to be awaited — it isn't, because forEach ignores whatever its callback returns, including a promise.

forEach-pitfall.ts

// WRONG: forEach doesn't await the async callback at all. This function
// returns before a single email has actually finished sending, and any
// rejection inside the callback becomes an unhandled promise rejection
// that Express never sees or reports.
async function notifyAllWrong(userIds: string[]) {
  userIds.forEach(async (id) => {
    await sendNotification(id);
  });
  console.log("Done!"); // logs immediately, before any notification finishes
}

// RIGHT: for...of with await runs them sequentially and is genuinely awaited.
async function notifyAllSequential(userIds: string[]) {
  for (const id of userIds) {
    await sendNotification(id);
  }
}

// RIGHT (and faster, when order doesn't matter): map to promises, then
// Promise.all them so they run concurrently but are still properly awaited.
async function notifyAllConcurrent(userIds: string[]) {
  await Promise.all(userIds.map((id) => sendNotification(id)));
}

A second common mistake is a try/catch that's scoped too broadly or too narrowly. Wrapping an entire route handler in one try/catch is fine for a single top-level error boundary, but swallowing an error inside a helper function and returning null instead of rethrowing hides exactly the information the caller needs to respond correctly.

swallowed-error.ts

// WRONG: swallows the real error, caller can't tell "not found" from
// "database connection failed" -- both look like `null`.
async function getUserSafe(id: string) {
  try {
    return await db.user.findUniqueOrThrow({ where: { id } });
  } catch {
    return null;
  }
}

// RIGHT: let unexpected errors propagate to Express's error-handling
// middleware (Week 2); only catch the specific case you can meaningfully
// handle here.
async function getUserOrNotFound(id: string) {
  const user = await db.user.findUnique({ where: { id } });
  if (!user) {
    throw new NotFoundError(`User ${id} not found`);
  }
  return user;
}

Wrap every async route handler, once

Rather than a try/catch in every single route, wrap async handlers once with a small helper that catches rejections and forwards them to next(err) — that's what routes this whole course from Week 2 onward that end in next(err) rely on to reach the centralized error middleware, instead of an unhandled rejection crashing the process.

3. Node Streams: Readable & Writable

A stream processes data in chunks as it arrives, rather than waiting for the entire payload before doing anything with it. Node models this with a small set of base classes: Readable streams produce data (a file being read, an HTTP request body arriving), Writable streams consume it (a file being written, an HTTP response being sent), and Duplex/ Transform streams do both — a TCP socket is duplex; a gzip compressor is a transform stream that reads uncompressed chunks and writes compressed ones.

stream-basics.ts

import { createReadStream } from "node:fs";

const stream = createReadStream("large-log-file.txt", { encoding: "utf-8" });

stream.on("data", (chunk) => {
  console.log(`Received ${chunk.length} characters`);
});

stream.on("end", () => {
  console.log("Finished reading the whole file, chunk by chunk");
});

stream.on("error", (err) => {
  console.error("Stream failed:", err);
});

The classic way to move data from a readable stream to a writable one is .pipe(), which also automatically manages backpressure (covered next):

pipe-example.ts

import { createReadStream, createWriteStream } from "node:fs";

const source = createReadStream("input.csv");
const destination = createWriteStream("output.csv");

source.pipe(destination);

// .pipe() does NOT forward errors between streams -- each stream can
// still emit its own "error" event independently, so both need listeners.
source.on("error", (err) => console.error("Read failed:", err));
destination.on("error", (err) => console.error("Write failed:", err));

Express request/response objects are streams

req is a readable stream (that's how express.json() reads the request body without you thinking about it) and res is a writable stream (that's what res.send() ultimately writes to). Understanding streams isn't a detour from Express — it's what's actually happening underneath every request and response you've handled since Week 2.

4. Backpressure & Streaming Large Data

Backpressure is what happens when a writable destination can't keep up with how fast a readable source is producing data. Every writable stream has an internal buffer; when you call .write() and that buffer exceeds its highWaterMark, .write() returns false as a signal to slow down. Writing more anyway just grows that buffer unboundedly — exactly the kind of memory problem streaming was supposed to avoid in the first place.

manual-backpressure.ts (illustrative -- prefer pipeline() below)

function writeAll(writable: NodeJS.WritableStream, chunks: string[]) {
  let i = 0;

  function writeNext() {
    let ok = true;
    while (i < chunks.length && ok) {
      ok = writable.write(chunks[i]);
      i++;
    }
    if (i < chunks.length) {
      // Buffer is full -- wait for "drain" before writing more.
      writable.once("drain", writeNext);
    }
  }

  writeNext();
}

You rarely have to write that loop by hand. pipeline() from node:stream/promises connects any number of streams together, forwards backpressure automatically end to end, and — critically — cleans up every stream properly if any one of them errors, which .pipe() alone does not do for you.

src/routes/downloads.ts

import { Router } from "express";
import { createReadStream, existsSync } from "node:fs";
import { pipeline } from "node:stream/promises";
import path from "node:path";

export const downloadsRouter = Router();
const UPLOADS_DIR = path.resolve("uploads");

downloadsRouter.get("/:filename", async (req, res, next) => {
  const filePath = path.join(UPLOADS_DIR, req.params.filename);

  if (!filePath.startsWith(UPLOADS_DIR) || !existsSync(filePath)) {
    res.status(404).json({ error: "File not found" });
    return;
  }

  try {
    res.setHeader("Content-Type", "application/octet-stream");
    res.setHeader("Content-Disposition", `attachment; filename="${req.params.filename}"`);

    // Streams the file straight to the response in chunks -- a multi-GB
    // file never has to be fully loaded into memory the way
    // res.send(await readFile(filePath)) would require.
    await pipeline(createReadStream(filePath), res);
  } catch (err) {
    next(err);
  }
});

Compare that to the naive alternative: res.send(await readFile(bigFile)) reads the entire file into a Buffer in memory before sending a single byte to the client. For a large file, or many concurrent downloads, that's the difference between constant memory usage and memory usage that scales with file size times concurrent requests.

Prefer pipeline() over raw .pipe() in production

.pipe() leaves you responsible for wiring up error listeners on every stream in the chain and manually destroying the others if one fails, which is easy to get wrong under real failure conditions (client disconnects mid-download, disk read error). pipeline() handles all of that for you and rejects its returned promise if anything in the chain fails, which fits naturally into the try/catch → next(err) pattern from Section 2.

5. Hands-on Exercise

Hands-on

Fix async pitfalls and add a streaming download route

Apply this week's Promise combinators, error-handling fixes, and streaming to a small script and an Express route.

Requirements:

Write a function that fetches data from 3 mock async sources with Promise.allSettled, and returns a result object that includes whichever sources succeeded plus a list of which ones failed and why.
Implement withTimeout() from Section 1 using Promise.race, and write a test (using Week 8's Vitest setup) confirming it rejects when the wrapped promise is slower than the timeout.
Take a function that uses .forEach(async ...) incorrectly, rewrite it two ways — sequential with for...of, and concurrent with Promise.all — and add a comment explaining when you'd pick each.
Add a GET /downloads/:filename Express route that streams a file from disk with pipeline() instead of buffering it fully, including a 404 for a missing file and a path-traversal guard.
Create a custom slow Writable stream (delay each _write call by 50ms) and pipe a large readable into it with pipeline(); log whenever backpressure kicks in by listening for the "drain" event on the underlying writable.

Hint

To force write() to return false and actually observe backpressure, set a small highWaterMark (e.g. { highWaterMark: 16 }) on your custom Writable and push a readable source significantly larger than that — with the default 16KB high-water mark and a small test file, you may never see the buffer fill up at all.

6. Knowledge Check

Four quick questions. Expand each to check your answer.

Why does `Promise.all` reject as soon as one promise rejects, even if the others would have succeeded — and when is `Promise.allSettled` the better choice?

Promise.all is designed for the case where every result is required — as soon as any one fails, the combined result can't be complete anyway, so it fails fast rather than waiting on operations whose results will be discarded. Promise.allSettled is the right choice whenever partial success is still useful, like a dashboard that can render two working widgets even if a third one's data source is down — it always resolves, giving you a per-promise status you can handle individually.

Why does calling an `async` function inside `Array.prototype.forEach` without awaiting it lead to silently swallowed errors?

forEach invokes its callback and completely ignores its return value, so when the callback is an async function, the promise it returns is never awaited or even referenced by forEach itself. If that promise later rejects, there's nothing attached to it to handle the rejection, which surfaces as an unhandled promise rejection rather than an error the calling code can catch — and the surrounding function has already returned as if everything succeeded.

Why is `res.send(await readFile(bigFile))` worse for a large file than piping a read stream directly into `res`?

readFile has to load the entire file into a Buffer in memory before res.send can send a single byte, so memory usage scales with file size and, worse, with the number of concurrent downloads happening at once. Streaming with pipeline(createReadStream(filePath), res) sends chunks as they're read, keeping memory usage roughly constant regardless of file size and letting the response start immediately instead of only after the whole file has been read from disk.

What does it mean when `writable.write()` returns `false`, and what should you do before calling `write()` again?

It means the writable stream's internal buffer has exceeded its highWaterMark — the destination can't drain data as fast as it's arriving, and the stream is signaling backpressure. Writing more immediately anyway just keeps growing that buffer in memory; the correct response is to pause and wait for the stream's "drain" event before writing again, which is exactly what pipeline()/.pipe() do for you automatically.