Design ChatGPT: A Conversational LLM Serving Platform

Asked by: OpenAI · Anthropic · xAI

"Design ChatGPT" is the single most frequently asked system design question across the AI labs right now. OpenAI, Anthropic, and xAI interviews all use some variant of it, and it is deceptively hard. It is not "design a chat app" — the chat UI is the easy 10%. The interview is really probing whether you understand how to serve a large language model as a low-latency, stateful, multi-tenant streaming service on scarce GPU hardware.

This article reconstructs a canonical version of the question and the answer a strong senior candidate would give.