Debug a Mini GPT and Add a KV Cache

Asked at OpenAI · Deep Learning & Architectures · Difficulty: Hard · Last asked Jun 9, 2026

A small GPT-style Transformer is meant to generate handwritten digits. Each MNIST image is shrunk to 7x7 pixels and read row by row as a sequence of 49 tokens, one per pixel, with values 0 to 255 (vocabulary size 256). The model learns to predict the next pixel from the earlier ones, and new images are sampled one pixel at a time.

Loading the problem…