Subscribe
Topic

Tokenization

How tokenizers turn text into the tokens LLMs read. Posts explain byte-pair encoding (BPE) in depth, including merge tables, byte-level fallbacks and how tokenization shapes context length, cost and failure modes.

Articles

1-3
  1. What Are Tokens? How AI Models Count Text and How Usage Is Billed

    Tokens are the chunks of text AI models read and write. How tokenizers split words, why counts differ by model and language, how images and audio count too, and how pay-per-use billing works, with sources from Google, IBM, Hugging Face and Anthropic.

    LLM
    8 min read
  2. Byte-Pair Encoding (BPE) and Modern Subword Tokenization: Mathematical Foundations, Merge Dynamics, Byte-Level Encodings, and Vocabulary Compression Mechanics

    Tokenization is the discrete boundary interface between raw textual sequences and the continuous vector representations of autoregressive large language models. Before an attention block computes a single inner product or a feed-forward network applies an activation function, an input string must be mapped into an ordered sequence of discrete integer identifiers drawn from a finite vocabulary $V$. The design and mathematical properties of this mapping govern key operational characteristics of l

    AI
    11 min read
  3. Byte-Pair Encoding in Large Language Models: How Tokenizers Compress Text, Shape Context, and Fail

    Large language models do not process strings directly. Before a single attention weight or linear projection executes, incoming text is converted into a sequence of discrete integers known as tokens. The choice and implementation of the tokenization algorithm establish the model's fundamental vocabulary, define the boundaries of its context window, dictate inference speed, and introduce unique behavioral quirks. Across modern transformer architectures, subword tokenization via Byte-Pair Encodin

    AI
    8 min read