Software Engineering

Designing Resilient Systems: Circuit Breakers and Retry Patterns

Build systems that gracefully handle failure with circuit breakers, exponential backoff, bulkheads, and timeout patterns.

Mbeah Essilfie

May 14, 2026 at 06:04 AM

9 min read 1328 views

Designing Resilient Systems: Circuit Breakers and Retry Patterns

Everything Fails

Networks are unreliable. Services go down. Databases hit connection limits. Resilient systems accept this and handle failure gracefully instead of cascading.

Retry with Exponential Backoff

async function retry<T>(
  fn: () => Promise<T>,
  options: { maxAttempts?: number; baseDelay?: number; maxDelay?: number } = {}
): Promise<T> {
  const { maxAttempts = 3, baseDelay = 1000, maxDelay = 30000 } = options

  for (let attempt = 1; attempt <= maxAttempts; attempt++) {
    try {
      return await fn()
    } catch (error) {
      if (attempt === maxAttempts) throw error

      const delay = Math.min(
        baseDelay * Math.pow(2, attempt - 1) + Math.random() * 1000,
        maxDelay,
      )
      console.warn(`Attempt ${attempt} failed, retrying in ${delay}ms...`)
      await new Promise(resolve => setTimeout(resolve, delay))
    }
  }
  throw new Error('Unreachable')
}

// Usage
const data = await retry(() => fetch('https://api.example.com/data'), {
  maxAttempts: 3,
  baseDelay: 500,
})

Always add jitter (randomness) to backoff delays. Without it, all failed clients retry at exactly the same time, causing a "thundering herd" that overwhelms the recovering service.

Circuit Breaker

Stop calling a failing service — give it time to recover:

class CircuitBreaker {
  private failures = 0
  private lastFailure = 0
  private state: 'closed' | 'open' | 'half-open' = 'closed'

  constructor(
    private threshold: number = 5,
    private resetTimeout: number = 30000,
  ) {}

  async execute<T>(fn: () => Promise<T>): Promise<T> {
    if (this.state === 'open') {
      if (Date.now() - this.lastFailure > this.resetTimeout) {
        this.state = 'half-open'
      } else {
        throw new Error('Circuit is OPEN — service unavailable')
      }
    }

    try {
      const result = await fn()
      this.onSuccess()
      return result
    } catch (error) {
      this.onFailure()
      throw error
    }
  }

  private onSuccess() {
    this.failures = 0
    this.state = 'closed'
  }

  private onFailure() {
    this.failures++
    this.lastFailure = Date.now()
    if (this.failures >= this.threshold) {
      this.state = 'open'
    }
  }
}

// Usage
const paymentBreaker = new CircuitBreaker(3, 60000)

async function processPayment(amount: number) {
  return paymentBreaker.execute(() =>
    fetch('https://payments.api/charge', {
      method: 'POST',
      body: JSON.stringify({ amount }),
    })
  )
}

Timeout Pattern

Never wait forever:

function withTimeout<T>(promise: Promise<T>, ms: number): Promise<T> {
  const timeout = new Promise<never>((_, reject) =>
    setTimeout(() => reject(new Error(`Timeout after ${ms}ms`)), ms)
  )
  return Promise.race([promise, timeout])
}

// Usage
const data = await withTimeout(fetchUserProfile(id), 5000)

Bulkhead Pattern

Isolate failures to prevent cascading:

class Bulkhead {
  private active = 0

  constructor(private maxConcurrent: number) {}

  async execute<T>(fn: () => Promise<T>): Promise<T> {
    if (this.active >= this.maxConcurrent) {
      throw new Error('Bulkhead capacity reached')
    }
    this.active++
    try {
      return await fn()
    } finally {
      this.active--
    }
  }
}

// Limit database calls to 10 concurrent
const dbBulkhead = new Bulkhead(10)

Combining Patterns

async function resilientCall<T>(fn: () => Promise<T>): Promise<T> {
  return retry(
    () => circuitBreaker.execute(
      () => withTimeout(fn(), 5000)
    ),
    { maxAttempts: 3 }
  )
}

Build for failure from day one. It's cheaper than fixing outages at 3am.