Scaling Real-Time APIs to 100k+ Concurrent Connections: WebSockets, SSE, and Redis Pub/Sub

Key Takeaway / TL;DR:

• WebSockets are the de facto choice for bidirectional, low-latency communication, but they require sticky sessions or a Redis Pub/Sub layer to scale horizontally.
• Server-Sent Events (SSE) offer a simpler, HTTP-based alternative for one-way server-to-client streaming, with automatic reconnection and better HTTP/2 multiplexing.
• Redis Pub/Sub acts as a scalable message broker to fan out events across multiple nodes, but you must handle backpressure and connection limits carefully.

The Problem & Industry Shift

Real-time features—live chat, collaborative editing, financial tickers, IoT telemetry—are no longer optional. Users expect sub-100ms updates. Traditional REST polling is wasteful and latency-bound. The industry has shifted toward persistent connections: WebSockets and SSE. However, scaling these to 100k+ concurrent connections on a single server is impossible; you need a distributed architecture.

The core challenge: a single Node.js process can handle ~10k-50k concurrent WebSocket connections (depending on memory and CPU), but beyond that, you must scale out horizontally. This introduces the sticky session problem: a client connected to server A might need to receive events triggered by another client connected to server B. Without a shared message bus, you lose events.

Architecture & Core Mechanics

A robust real-time architecture typically involves three layers:

1. Client Connection Layer: WebSocket or SSE endpoints terminated by a fleet of stateless API servers.
2. Message Broker Layer: Redis Pub/Sub (or Kafka, NATS) that relays messages between servers.
3. Persistence Layer: Optional, for message history or offline delivery.

Here's a high-level flow:

[Client A] <--WebSocket--> [Server 1] <--Redis Pub/Sub--> [Server 2] <--WebSocket--> [Client B]

When Client A sends a message, Server 1 publishes it to a Redis channel. All servers (including Server 1) subscribe to that channel and forward the message to their local clients who are interested. This decouples the connection from the message origin.

Key design decisions:

• WebSocket vs SSE: WebSockets are bidirectional and lower overhead per message (after handshake). SSE is unidirectional (server-to-client) but rides on HTTP, which simplifies firewalls and proxies. For chat or collaborative apps, WebSockets are preferred. For live feeds (stock prices, notifications), SSE is simpler and can be more efficient with HTTP/2.
• Redis Pub/Sub: It's fast (sub-millisecond) but fire-and-forget. If a subscriber is slow, messages are dropped. For critical messages, consider Redis Streams or a persistent queue.
• Connection Pooling: Redis connections are expensive. Use a single Redis client per server, not per WebSocket. Node.js Redis clients (e.g., ioredis) handle connection pooling internally.

Production Code Example

Below is a production-grade Node.js example using ws for WebSockets, ioredis for Redis Pub/Sub, and express for HTTP. It demonstrates horizontal scaling with a Redis adapter.

import express from 'express';
import http from 'http';
import { WebSocketServer, WebSocket } from 'ws';
import Redis from 'ioredis';

const app = express();
const server = http.createServer(app);

// WebSocket server with noServer to handle upgrade manually
const wss = new WebSocketServer({ noServer: true });

// Redis clients: one for publishing, one for subscribing (ioredis recommends separate connections)
const redisPublisher = new Redis({ host: 'redis-1', port: 6379, maxRetriesPerRequest: null });
const redisSubscriber = new Redis({ host: 'redis-1', port: 6379, maxRetriesPerRequest: null });

// Map to track clients per server (in-memory)
const clients = new Map<string, WebSocket>(); // clientId -> socket

// Subscribe to Redis channel for incoming messages from other servers
redisSubscriber.subscribe('chat:global');
redisSubscriber.on('message', (channel, message) => {
  if (channel === 'chat:global') {
    const { clientId, data } = JSON.parse(message);
    // Broadcast to all local clients (or target specific client)
    clients.forEach((ws, id) => {
      if (ws.readyState === WebSocket.OPEN) {
        ws.send(JSON.stringify({ from: clientId, data }));
      }
    });
  }
});

// Handle HTTP upgrade requests
server.on('upgrade', (request, socket, head) => {
  // Authenticate here (e.g., JWT in query string or cookie)
  wss.handleUpgrade(request, socket, head, (ws) => {
    wss.emit('connection', ws, request);
  });
});

wss.on('connection', (ws, request) => {
  // Extract client ID from query params (e.g., ?clientId=abc)
  const clientId = new URL(request.url, 'http://localhost').searchParams.get('clientId');
  if (!clientId) {
    ws.close(4001, 'Missing clientId');
    return;
  }

  clients.set(clientId, ws);
  console.log(`Client ${clientId} connected. Total: ${clients.size}`);

  ws.on('message', (message) => {
    // Publish to Redis so other servers can receive
    redisPublisher.publish('chat:global', JSON.stringify({ clientId, data: message.toString() }));
  });

  ws.on('close', () => {
    clients.delete(clientId);
    console.log(`Client ${clientId} disconnected. Total: ${clients.size}`);
  });

  ws.on('error', (err) => {
    console.error('WebSocket error:', err);
    ws.close(1011, 'Internal error');
  });
});

const PORT = process.env.PORT || 3000;
server.listen(PORT, () => {
  console.log(`Server listening on port ${PORT}`);
});

Critical engineering decisions:

• Separate Redis connections for pub/sub: ioredis requires a dedicated connection for subscriptions because it enters a subscriber mode. Mixing pub and sub on one connection will cause errors.
• maxRetriesPerRequest: null: Prevents the Redis client from buffering commands when the connection is lost, which could cause memory leaks. Instead, it will emit errors and you can handle reconnection.
• Graceful backoff: Implement reconnection logic for Redis and WebSocket clients. For Redis, ioredis has built-in retry strategy; configure it with exponential backoff.
• Backpressure: If a client is slow, ws.send can buffer. Use ws.bufferedAmount to monitor and drop or disconnect slow consumers.

Performance, Cost & Trade-offs

Benchmarks (approximate, from real-world deployments):

• WebSocket throughput: A single Node.js process can handle ~10k messages/sec with low latency (<5ms) on modest hardware. With Redis Pub/Sub, the bottleneck becomes Redis (can handle 100k+ ops/sec).
• Memory: Each WebSocket connection consumes ~20-50KB (including buffers). 100k connections => 2-5GB RAM per server. Plan accordingly.
• Latency: Redis Pub/Sub adds ~0.1-0.5ms overhead. End-to-end latency remains under 10ms in a well-configured cluster.

Trade-offs:

• Redis Pub/Sub vs. Redis Streams: Pub/Sub is simpler and lower latency, but messages are lost if no subscriber is present. Streams provide persistence and consumer groups but add complexity and higher latency.
• Sticky sessions vs. Redis: Sticky sessions (e.g., using a load balancer) avoid Redis but cause uneven load and failover issues. Redis decouples but adds a single point of failure (use Redis Sentinel or Cluster).
• SSE over WebSockets: SSE is easier to implement and works over HTTP/2, but it's unidirectional. For bidirectional, you need WebSockets or a combination (SSE + POST).

Security considerations:

• Authentication: Validate tokens during the WebSocket handshake. Don't rely on origin headers.
• Rate limiting: Implement per-connection message rate limits to prevent abuse.
• TLS: Use WSS (WebSocket Secure) to encrypt traffic. Terminate TLS at the load balancer to reduce CPU load on app servers.

Actionable Checklist / Summary

When building a real-time API at scale, follow these steps:

1. Choose the right protocol: WebSockets for bidirectional, SSE for one-way streaming.
2. Design for horizontal scaling: Use Redis Pub/Sub (or a similar broker) to sync across nodes.
3. Manage Redis connections: Use separate pub/sub clients, configure retry strategies with exponential backoff.
4. Handle backpressure: Monitor bufferedAmount and disconnect slow clients.
5. Implement graceful degradation: If Redis fails, fall back to local-only broadcast (with a warning).
6. Monitor metrics: Track connection count, message rates, Redis latency, and error rates.
7. Load test: Use tools like k6 or wrk to simulate 100k connections and identify bottlenecks.

References

• MDN Web Docs: WebSockets
• MDN Web Docs: Server-Sent Events
• Redis Pub/Sub Documentation
• ioredis GitHub Repository
• ws: a Node.js WebSocket library