Subscribe
Topic

Explainers

Explainers that break down AI research ideas and model techniques, with the mathematics behind them. Recent posts cover Diffusion Transformers (DiT), STaR and Quiet-STaR reasoning, Transfusion, Test-Time Training and Tree of Thoughts.

Articles

1-20
  1. What Is Weight Initialization? The Random Numbers That Decide If a Network Learns

    Every neural network starts as a pile of random numbers. Before a model has seen a single training example, its billions of weights are filled in by a random-number generator, and the choice of how that generator is set up determines whether training works at all. Getting it wrong does not produce a worse model. It produces a model that never learns. This is weight initialization, and it is one of the oldest unsolved-sounding problems in deep learning hiding inside one line of setup code. Why

    AI
    7 min read
  2. What Is NLP? The Research Field That Became Large Language Models

    Natural language processing, or NLP, is the engineering discipline of making computers work with human text and speech: translating languages, answering questions, summarizing documents, detecting sentiment, parsing grammar. For seven decades it was a field of separate, hand-built tasks. In the last five years it became something else: the research area that produced large language models, and then was largely absorbed by them. The clearest map of that arc comes from Christopher Manning of Stan

    educational
    6 min read
  3. What Is Inference? How Trained Models Turn Prompts Into Tokens

    Training ends. Inference never does. A language model spends months being trained and then, in principle, nothing more happens to its weights. Every question you ask ChatGPT, Claude or Gemini afterward triggers a different process: the model reads your prompt, runs a forward pass, and emits one token at a time. That process is called inference, and it is where almost all of the money in deployed AI now goes. Deloitte forecast that inference workloads would account for roughly two-thirds of all

    Education
    7 min read
  4. What Are Audio Tokens? How Language Models Learned to Hear and Talk

    Speech arrives as 24,000 numbers per second, far too dense for any context window. Audio tokens are the compressed integer sequences that let language models hear and talk anyway. The path from WaveNet through Whisper, SoundStream, EnCodec, VALL-E and Moshi, and what the trick costs.

    educational
    6 min read
  5. What Is Grokking? How Models Memorize First and Understand Later

    A network can fit every training example and still know nothing. Then, thousands of steps later and with no change to the setup, it clicks. What two and a half years of delayed-generalization research, from OpenAI's modular arithmetic table to mixture-of-experts pretraining, actually shows.

    educational
    7 min read
  6. What Is Pre-Training? How a Language Model Reads the Internet

    Pre-training is the first and by far the most expensive phase of building a large language model. A lab takes a neural network with random weights, feeds it hundreds of billions or trillions of tokens of raw text, and trains it on one task: predict the next token. The result is a "base model" that has absorbed the statistics of human writing at a scale no human has read. Everything a frontier model knows before anyone fine-tunes it into an assistant comes from this phase. This post is part of o

    AI
    6 min read
  7. What Is Model Collapse? How AI Models Degrade by Training on Their Own Output

    Every generative model is a copy of the data it was trained on. Model collapse is what happens when the data being copied is other models. First formally described by researchers at Oxford, the University of Toronto, and their collaborators, the effect is one of the few well-documented failure modes that becomes more probable, not less, as AI-generated text floods the web. The basic mechanism The canonical paper, published in Nature in July 2024, defines the phenomenon precisely: "indiscrimin

    AI
    6 min read
  8. What Is a Transformer? The Architecture Behind Every Modern LLM

    Every frontier language model you have used, GPT, Claude, Gemini, Llama, DeepSeek, is a transformer. The word does not describe a company, a product or a training method. It describes the circuit layout, the arrangement of matrix multiplications that turns a sequence of tokens into a prediction of the next one. Two models from different labs can share this layout while behaving completely differently, because the layout says how computation is organized, not what the model knows. The design com

    educational
    6 min read
  9. What Is RLHF? How Human Feedback Became the Standard Way to Train AI Assistants

    A pretrained large language model has one job: predict the next token. Nothing in that objective tells it to answer your question, admit uncertainty, or refuse to write malware. Model scale does not fix this. As OpenAI's InstructGPT team put it in early 2022, making language models bigger does not inherently make them better at following a user's intent, and the result can be outputs that are untruthful, toxic, or simply not helpful. The standard fix for that gap is reinforcement learning from

    AI
    5 min read
  10. How to Use Gemini in Google Slides

    How to use Gemini in Google Slides: open Ask Gemini, generate or edit slides, build a full deck, add images, and check which plans include each feature.

    Google
    10 min read