Longest-Match Tokenizer with Unknown Tokens

Asked at Anthropic · Coding Questions · Difficulty: Medium · Last asked Jun 26, 2026

Implement a tokenizer that converts a string into a list of integer ids using a fixed vocabulary. At every position it takes the longest vocabulary entry that matches, and characters that start no entry become a special unknown id.

Loading the problem…