NLP

Byte-Pair Encoding in Large Language Models: How Tokenizers Compress Text, Shape Context, and Fail

Large language models do not process strings directly. Before a single attention weight or linear projection executes, incoming text is converted into a sequence of discrete integers known as tokens. The choice and implementation of the tokenization algorithm establish the model's fundamental vocabulary, define the boundaries of its context window, dictate inference speed, and introduce unique behavioral quirks. Across modern transformer architectures, subword tokenization via Byte-Pair Encodin