Design an AI Chatbot App (RAG-Grounded Assistant)

Asked by: OpenAI

"Design an AI Chatbot App" (an OpenAI question) is the application-layer cousin of "Design ChatGPT." That distinction is the first thing to establish in the room: ChatGPT-the-platform is about the inference engine (KV cache, batching, GPU economics — see that article). An AI chatbot app — say, an assistant grounded in a company's documents — treats inference as a dependency and lives or dies on something else entirely: grounding. A chatbot that confidently makes things up is worse than no chatbot.

The crux (spend ~60% of your time here). The hard part is the retrieval-augmented generation (RAG) pipeline — ingest a knowledge base, retrieve the right context for each question, and generate a grounded, cited answer within the latency and context-window budget — plus conversation/context management and keeping the index fresh. The LLM inference and the chat UI are table-stakes here. If you spend your time on token streaming and skip retrieval quality, you've missed the question.