DevOps & CloudSoftware Engineering

Scaling Node.js: From Single Process to Distributed System

A roadmap for scaling Node.js from handling 100 to 100,000 requests per second — clustering, load balancing, caching, and horizontal scaling.

Mbeah Essilfie

May 10, 2026 at 06:04 AM

11 min read 1022 views

Scaling Node.js: From Single Process to Distributed System

The Scaling Journey

Node.js runs on a single thread by default. That single thread handles surprisingly well — often 10,000+ requests/second. But at some point, you need more.

Level 1: Optimize the Single Process

Before adding complexity, squeeze more from what you have:

// 1. Use streaming instead of buffering
app.get('/large-file', (req, res) => {
  const stream = fs.createReadStream('large-file.csv')
  stream.pipe(res) // Constant memory regardless of file size
})

// 2. Cache expensive operations
const cache = new Map<string, { data: unknown; expires: number }>()

function cached<T>(key: string, ttl: number, fn: () => Promise<T>): Promise<T> {
  const hit = cache.get(key)
  if (hit && hit.expires > Date.now()) return hit.data as Promise<T>

  const result = fn()
  result.then(data => cache.set(key, { data, expires: Date.now() + ttl }))
  return result
}

// 3. Use connection pooling
const pool = new Pool({
  max: 20,              // Match your CPU count
  idleTimeoutMillis: 30000,
  connectionTimeoutMillis: 2000,
})

Level 2: Cluster Mode

Use all CPU cores:

import cluster from 'node:cluster'
import { cpus } from 'node:os'

if (cluster.isPrimary) {
  const numWorkers = cpus().length
  console.log(`Starting ${numWorkers} workers`)

  for (let i = 0; i < numWorkers; i++) {
    cluster.fork()
  }

  cluster.on('exit', (worker) => {
    console.log(`Worker ${worker.process.pid} died, restarting...`)
    cluster.fork()
  })
} else {
  startServer()
}

Or simply use PM2:

pm2 start server.js -i max  # Uses all cores

Clustering gives you linear scaling up to your CPU core count. A 4-core machine handles roughly 4x the traffic of a single process.

Level 3: External Cache (Redis)

Move caching outside the process so all instances share it:

import Redis from 'ioredis'

const redis = new Redis(process.env.REDIS_URL)

async function getCachedPosts(page: number) {
  const key = `posts:page:${page}`
  const cached = await redis.get(key)
  if (cached) return JSON.parse(cached)

  const posts = await db.post.findMany({ skip: (page - 1) * 20, take: 20 })
  await redis.setex(key, 300, JSON.stringify(posts)) // 5 min TTL
  return posts
}

Level 4: Horizontal Scaling

Multiple servers behind a load balancer:

                    ┌─── Node Instance 1 (4 workers)
Load Balancer ─────┼─── Node Instance 2 (4 workers)
                    └─── Node Instance 3 (4 workers)

Requirements:

Stateless servers — No in-memory sessions
External session store — Redis
Shared nothing — Each instance is independent
Health checks — Load balancer removes unhealthy instances

Level 5: Read Replicas and Queue Workers

Writes ──→ Primary DB
Reads  ──→ Read Replica 1, Read Replica 2

Heavy jobs ──→ Queue (BullMQ) ──→ Worker processes

When to Scale What

Bottleneck	Solution
CPU-bound	Cluster mode → more instances
I/O-bound	Connection pooling → async optimization
Database reads	Read replicas → caching layer
Database writes	Queue + batch processing
Memory	Streaming → pagination → horizontal scale