Node.js Streams Explained with Real Examples

Streams have a reputation for being the difficult corner of Node.js, and it is partly deserved: the API has accumulated three eras of design since 2011. But the underlying idea is simple. Instead of loading all of your data into memory and then acting on it, you process it in chunks as it flows past. When a client asked us to import 4 GB CSV exports into Postgres on a container with 512 MB of RAM, streams were not an optimisation — they were the only way the job could run at all.

The four kinds of stream

• Readable: a source of chunks, such as fs.createReadStream or an incoming HTTP request.
• Writable: a destination, such as fs.createWriteStream or an HTTP response.
• Duplex: both at once, like a TCP socket.
• Transform: a duplex stream that modifies data on the way through — gzip is the classic example.

Everything else in the streams world is composition of these four. An HTTP server handler in Node is literally a readable request stream and a writable response stream, which is why piping a file to a response works with no framework help at all.

Always use pipeline, never pipe

The classic pipe method has a serious flaw: it does not forward errors. If the source stream errors mid-transfer, the destination is never closed and you leak file descriptors. The pipeline function from node:stream/promises propagates errors across the whole chain and cleans up every stream on failure.

import { pipeline } from 'node:stream/promises';
import { createReadStream, createWriteStream } from 'node:fs';
import { createGzip } from 'node:zlib';

await pipeline(
  createReadStream('export.csv'),
  createGzip(),
  createWriteStream('export.csv.gz')
);

Backpressure is handled for you here as well. If the disk cannot keep up with the source, pipeline pauses the readable side until the writable side drains. This is the mechanism most hand-rolled stream code gets wrong, and it is the difference between steady memory usage and a process that balloons until the OOM killer arrives.

A real transform stream

Transform streams are where streams stop being plumbing and start being useful. Here is a simplified version of the CSV importer mentioned above: it receives lines and emits Postgres COPY-friendly rows, filtering out records we do not want.

import { Transform } from 'node:stream';

const toRow = new Transform({
  objectMode: true,
  transform(line, _enc, callback) {
    const [id, email, status] = line.split(',');
    if (status !== 'deleted') {
      this.push(id + '\t' + email.toLowerCase() + '\n');
    }
    callback();
  },
});

Note objectMode: without it, streams deal in Buffers and strings of arbitrary chunk boundaries, which is rarely what you want for record-oriented work. Pair a transform like this with a line-splitting stream and the whole import runs in constant memory regardless of file size.

One modern wrinkle worth knowing: Node also ships the WHATWG web streams API, which is what fetch responses use. The two families interoperate through Readable.toWeb and Readable.fromWeb, so you can pipe a fetch download straight into a classic Node pipeline. Newer cross-platform libraries increasingly target web streams, so expect to convert at the boundary for a while yet — happily the conversion is a single call in each direction, and backpressure survives the crossing.

When not to use streams

If your payload is a few kilobytes of JSON, streams add complexity for nothing — read it, parse it, move on. Streams earn their keep when data is large, unbounded, or arriving over time: file processing, proxying uploads straight to S3, server-sent events, log shipping. Reach for them when memory is the constraint, and prefer the promise-based pipeline API and iterating readables with for await, which make modern stream code look pleasantly ordinary.

Streams reward the afternoon it takes to learn them properly. The next time a file is too big for memory, you will have the right tool already in hand.