ChatGPT System Pipeline Architecture

Streamlined real-time token processing & microservice topology

1
Phase 1 of 6 HTTP POST • JSON
Client sends prompt payload and conversation ID to start SSE response stream.

Client Apps

React / iOS / Android

SSE Pipeline Active

API Gateway

Envoy / SSL Ingress

Rate Limiter & JWT OAuth2

Message Service

Stream Orchestrator

Active SSE Streams gRPC

Moderation

Safety Guardrails

Fast-path Filter <10ms

Redis Cluster

Context Window Cache

Sliding Window N TTL 24h

LLM Gateway

Load Balancer & Queue

Token Scheduler vLLM Router

GPU Cluster Pool

NVIDIA H100 TensorRT

Token Streamer 60 tok/sec

PostgreSQL DB

Primary + Replicas

Async Write Persistent
Message Orchestrator Service
Core Stream Controller
Coordinates active Server-Sent Events (SSE) connections with clients. Fetches past N conversation turns from Redis to assemble the prompt context window before passing enriched payloads to the LLM Gateway.
Protocol: HTTP/2 SSE & gRPC
Latency Impact: < 15ms
Concurrency: 500k active streams