Case study boundary
Use this as a learning model.
This page explains architectural patterns and constraints inspired by ChatGPT. It should not be read as the exact private production architecture or as a one-size-fits-all recommendation.
Evidence
Public info + inference
Learn first
Concepts behind this architecture
GPU Inference
Run trained models efficiently enough to serve user requests in real time.
Backpressure
Slow producers down when consumers cannot keep up.
Embeddings
Represent text, images, users, or items as vectors that capture semantic similarity.
Vector Database
Store embeddings and retrieve nearest neighbors at scale.
Overview
ChatGPT is OpenAI’s conversational AI product, serving 200M+ weekly active users with access to GPT-4 class large language models. Its architecture addresses the unique challenges of LLM serving: extremely GPU-intensive inference, token-by-token streaming responses, multi-turn conversation context management, tool use (code execution, web browsing, image generation), and a plugin/function-calling ecosystem. The system must balance throughput (requests per second) with latency (time to first token) while managing GPU clusters costing hundreds of millions of dollars.
Scale
200M+
Weekly active users
Billions
Requests per day
1T+ (GPT-4 class)
Model parameters
10,000s of GPUs
GPU cluster size
Requirements
Functional
- Multi-turn conversational chat with context retention
- Token-by-token streaming responses
- Tool use: code interpreter, web browsing, DALL-E image generation
- File upload and document analysis
- Custom GPTs with system prompts and knowledge files
- API access for developers (Chat Completions, Assistants)
- Voice input/output mode
Non-Functional
- Low time-to-first-token (<1 second)
- High throughput across millions of concurrent conversations
- Efficient GPU utilization (>80% MFU)
- Graceful degradation under load (queuing, rate limiting)
- Content safety filtering with low false positive rate
- Global availability with regional data residency
High-Level Architecture
Key Components
API Gateway & Rate Limiter
Handles authentication (API keys, session tokens), rate limiting (per-user, per-tier), and request routing. Implements token-based rate limits — each request consumes tokens proportional to input + output length.
Model Router
Routes requests to the appropriate model cluster based on the requested model (GPT-4, GPT-4o, GPT-4o-mini), user tier (free vs Plus vs Enterprise), and current cluster load. Implements overflow routing when primary clusters are at capacity.
Conversation Manager
Manages multi-turn conversation state. Assembles the full prompt from conversation history, system prompts (for Custom GPTs), and tool results. Handles context window management — truncating or summarizing old messages when the conversation exceeds the model’s context length.
Request Scheduler
The core inference scheduler that batches incoming requests for efficient GPU utilization. Implements continuous batching — new requests join a running batch without waiting for all current requests to complete. Manages the KV cache allocation on GPU memory.
GPU Inference Cluster
Clusters of NVIDIA H100/A100 GPUs running model inference. Each model instance is distributed across multiple GPUs using tensor parallelism (within a node) and pipeline parallelism (across nodes). Uses custom inference kernels optimized with CUDA.
Tool Orchestrator
Coordinates tool calls when the model decides to use tools. Executes code interpreter sessions in sandboxed containers, fetches web pages for browsing, triggers DALL-E for image generation, and runs retrieval against uploaded documents.
Safety Classifier
Runs content moderation on both inputs and outputs. A separate lightweight model classifies content against policy categories. Operates inline with low latency to avoid delaying responses.
Data Model
LLM Inference and Continuous Batching
Serving a large language model efficiently is the central technical challenge of ChatGPT’s architecture.
Autoregressive generation: LLMs generate tokens one at a time. Each token requires a full forward pass through the model. For a 200-token response, that’s 200 sequential forward passes — the main source of latency.
KV cache: To avoid recomputing attention for all previous tokens on each step, the model caches key-value pairs from previous tokens. This KV cache grows linearly with sequence length and is the primary consumer of GPU memory.
Continuous batching: Traditional batching waits for a batch of requests to complete before starting new ones. Continuous batching (also called iteration-level batching) allows new requests to join a running batch at any iteration. When a request in the batch finishes (generates its EOS token), its GPU memory slot is immediately freed for a new request. This dramatically improves GPU utilization.
PagedAttention: The KV cache is managed using paged memory allocation (inspired by OS virtual memory). Instead of pre-allocating a contiguous block of GPU memory for each request’s maximum possible sequence length, memory is allocated in small pages as needed. This reduces memory waste by ~60%.
Tool Use and Function Calling
ChatGPT’s tool use system allows the model to interact with external services — a key architectural extension beyond pure text generation.
Function calling protocol: The model is trained to output structured JSON tool calls instead of text when it determines a tool would help. The orchestrator intercepts these, executes the tool, and feeds the result back to the model as a new message.
Code Interpreter: Runs Python code in a sandboxed Jupyter-like container. Each session gets an isolated container with a mounted filesystem for uploaded files. The container has a CPU/memory limit and a network firewall (no internet access). Execution results (text, images, files) are returned to the model.
Web Browsing: When the model issues a browse tool call, a headless browser fetches the page, extracts readable text content, and returns it to the model within the context window.
Retrieval (RAG): For Custom GPTs with knowledge files, uploaded documents are chunked, embedded, and stored in a vector database. When the user’s query is relevant, the orchestrator retrieves the top-k chunks and injects them into the model’s context.
Multi-step tool use: The model can chain multiple tool calls in a single turn — e.g., browse a URL, then write code to analyze the data, then generate a chart.
GPU Cluster and Model Parallelism
GPT-4 class models have over a trillion parameters and cannot fit on a single GPU. Serving them requires distributed inference across multiple GPUs.
Tensor parallelism: A single layer’s weight matrices are split across GPUs within a server node (typically 8 H100s). Each GPU computes a portion of the matrix multiplication, and results are all-reduced. This requires high-bandwidth interconnect (NVLink at 900 GB/s).
Pipeline parallelism: The model’s layers are split across multiple nodes. Node 1 computes layers 1-N, sends activations to Node 2 for layers N+1-2N, and so on. This allows using more GPUs but adds latency proportional to the number of pipeline stages.
Cluster management: The inference cluster runs on Kubernetes with custom GPU scheduling. Failed GPUs are automatically detected and the model shard is migrated to a healthy GPU. The cluster maintains a pool of hot spare GPUs for rapid failover.
Cost: Running inference at ChatGPT’s scale requires tens of thousands of H100 GPUs. At ~$30K per GPU, the hardware cost alone is in the hundreds of millions. Efficient utilization (measured by Model FLOPs Utilization) is critical — even a 5% improvement saves millions of dollars.
Architectural Tradeoffs
Continuous batching over static batching
Pros
- +Much higher GPU utilization
- +Lower average latency
- +Better handling of variable-length requests
Cons
- −More complex scheduler implementation
- −Memory management overhead (paging)
- −Harder to guarantee consistent per-request latency
Server-side tool execution over client-side
Pros
- +Model can autonomously chain tool calls
- +Consistent tool behavior across clients
- +Sandboxed execution for security
Cons
- −Adds latency for each tool call round-trip
- −Server cost for running sandboxed containers
- −Limited to pre-defined tool interfaces
Streaming token-by-token responses
Pros
- +Perceived latency is much lower (user reads as tokens arrive)
- +Better UX for long responses
- +Client can cancel mid-generation
Cons
- −SSE connections consume server resources longer
- −Safety filtering must work on partial outputs
- −Client rendering complexity for streaming markdown/code
Learn From ChatGPT
Copy a learning context prompt for an AI assistant. It keeps assumptions visible and asks for tradeoffs, cost drivers, failure modes, and what to measure before adding complexity.