Dynamic Batching for Language Model Token Prediction

Asked at Alibaba · Coding Questions · Difficulty: Hard

Given a 'mock language model' interface: Each time you input a batch of generated token prefixes, the model returns the next token (or tokens) for each sequence. Implement a batch sampling/decoding process with the core challenge of dynamic batching: different sequences may finish at different times, leading to empty slots in the batch.

Loading the problem…