WebSocket Infrastructure

The Problem

Applications like chat, multiplayer games, collaborative editors, and live trading platforms require low-latency bidirectional communication between client and server. HTTP request-response adds too much overhead per interaction. WebSockets upgrade an HTTP connection to a persistent, full-duplex TCP channel — but scaling millions of concurrent WebSocket connections across a fleet of servers introduces challenges around connection management, message routing, state synchronization, and fault tolerance.

Architectural Approaches

Architecture

Loading diagram...

Key Components

Data Model

Loading diagram...

Connection Lifecycle

A WebSocket connection goes through several stages:

1. HTTP Upgrade — Client sends an HTTP request with Upgrade: websocket header. Server validates auth token (from query param or cookie) and responds with 101 Switching Protocols.
2. Initialization — Server registers the connection in the registry, subscribes to relevant pub/sub channels, and sends an initial state payload (unread counts, presence data).
3. Steady State — Bidirectional message exchange. Server sends ping frames every 30s; client responds with pong. Messages are routed through the pub/sub backbone.
4. Graceful Close — Either side sends a close frame with a status code. Server unsubscribes from channels, removes the registry entry, and notifies presence subscribers.
5. Abnormal Close — Network failure or crash. Detected via ping timeout (typically 60-90s). Cleanup runs asynchronously, and the client reconnects with exponential backoff.

Reconnection: Clients should implement exponential backoff with jitter (e.g., 1s, 2s, 4s + random 0-1s). On reconnect, send the last received message ID to enable server-side replay of missed messages.

Loading diagram...

Scaling to Millions of Connections

Per-server limits: A single server with 16GB RAM can hold ~500K idle WebSocket connections (each consuming ~30KB). Active connections with message buffering need more memory. Key OS tunings:

• Set ulimit -n to 1M+ for file descriptors
• Tune net.core.somaxconn and TCP buffer sizes
• Use epoll (Linux) or kqueue (BSD/macOS) for efficient I/O multiplexing

Horizontal scaling strategy:

1. Add gateway instances behind a Layer 4 load balancer
2. Shard pub/sub channels to avoid hotspots (e.g., partition by channel ID hash)
3. Separate read and write paths — message persistence can be async
4. Use connection draining during deploys — stop accepting new connections, wait for existing ones to close or migrate

Cost optimization: WebSocket connections are long-lived and consume resources even when idle. Consider:

• Downgrading idle connections to SSE or long-polling
• Implementing connection quotas per user
• Using serverless WebSocket services (AWS API Gateway WebSocket, Cloudflare Durable Objects) for variable workloads

Message Ordering and Delivery

Within a connection: TCP guarantees in-order delivery. Messages sent on a single WebSocket connection arrive in order.

Across connections: When a user reconnects to a different gateway, message ordering depends on the pub/sub system. Solutions:

• Assign sequence numbers per channel. Clients buffer and reorder if needed.
• Use Kafka partitions keyed by channel ID for strict per-channel ordering.
• Accept eventual consistency for non-critical updates (typing indicators, presence).

Delivery guarantees:

• At-most-once: Default WebSocket behavior. Message lost if connection drops mid-delivery.
• At-least-once: Persist messages before acknowledging to sender. Replay on reconnect using last-message-ID. Clients deduplicate using message IDs.
• Exactly-once: Very expensive. Requires idempotent message processing and acknowledgment tracking. Rarely needed — at-least-once with client dedup is sufficient for most use cases.

Real-World Examples

Architectural Tradeoffs

# Learn the System Design of: WebSocket Infrastructure

You are an expert software architect and teacher. I want to understand websocket infrastructure as a practical system design capability. (Persistent bidirectional connections at scale for real-time applications).

Use the following architectural guide as a learning reference. Avoid one-size-fits-all recommendations. Keep assumptions, costs, and tradeoffs explicit.

---

import { Mermaid, ComponentList, ComponentCard, ApproachList, Approach, Example, Tradeoff } from '@/components/mdx';

## The Problem

Applications like chat, multiplayer games, collaborative editors, and live trading platforms require low-latency bidirectional communication between client and server. HTTP request-response adds too much overhead per interaction. WebSockets upgrade an HTTP connection to a persistent, full-duplex TCP channel — but scaling millions of concurrent WebSocket connections across a fleet of servers introduces challenges around connection management, message routing, state synchronization, and fault tolerance.

## Architectural Approaches

<ApproachList>
<Approach
  name="Sticky Sessions with In-Process State"
  pros={[
    'Simplest implementation — no external state store needed',
    'Low latency for messages between users on the same server',
    'Easy to reason about — each server is self-contained',
  ]}
  cons={[
    'Server failure drops all connections on that instance',
    'Cross-server messaging requires an additional routing layer',
    'Uneven load distribution as sticky sessions prevent rebalancing',
    'Horizontal scaling is limited by single-server capacity',
  ]}
>
Route each client to a specific server instance using sticky sessions (IP hash, cookie, or connection ID). Each server holds connection state in memory and routes messages locally. Simple to build but limited in scale and resilience.
</Approach>
<Approach
  name="Pub/Sub Backbone with Stateless Gateways"
  pros={[
    'Gateways are interchange...

[Full prompt copied to clipboard]