Subscribe
Explainers
6 min read

What Is a Chat Template? How Your Messages Become a Model Prompt

A model never sees your messages array. It sees one flat string of tokens built by a small Jinja program you are told never to touch. Here is what that program does and why the exact output matters.

Retro Olivetti-style brochure cover reading CHAT TEMPLATE, with navy speech bubbles riding an orange ribbon into a small machine
Retro Olivetti-style brochure cover reading CHAT TEMPLATE, with navy speech bubbles riding an orange ribbon into a small machine

A chat API takes a list of messages. Each one has a role, usually system, user or assistant, and some text. A language model takes none of that. It consumes a single flat sequence of tokens and predicts the next one. Something has to convert the structured conversation into the flat string, in exactly the form the model was trained to see. That something is the chat template, and it is the most invisible piece of infrastructure in a modern AI stack.

What a template actually is

In the Hugging Face ecosystem a chat template is a small program, written in the Jinja template language, stored inside the model's tokenizer under the attribute chat_template. The method apply_chat_template walks your list of message dictionaries and renders the whole conversation into one string. The documentation is explicit about the job: the input is "a list of dictionaries with role and content keys" and the output is "a formatted sequence" ready for the model (Hugging Face transformers docs).

The rendered string is not plain text. It is stitched together with special control tokens that live in the model's vocabulary and were introduced during training as part of the format. The exact tokens differ from family to family.

OpenAI's ChatML format, the one that started the habit, wraps every turn in two delimiters:

<|im_start|>system
You are a helpful assistant.<|im_end|>
<|im_start|>user
What is France's capital?<|im_end|>
<|im_start|>assistant

Meta's Llama 3 uses a different set. Its published prompt format documents the tokens: <|begin_of_text|> marks the start of the prompt, <|start_header_id|>{role}<|end_header_id|> encloses the role of each message, and <|eot_id|> ends a turn. The model is expected to complete the assistant message after a trailing <|start_header_id|>assistant<|end_header_id|> header, and it signals it is finished by generating <|eot_id|> (Meta Llama model cards and prompt formats). Same idea, different wire.

So a chat template is the grammar of the conversation-on-a-string: who is speaking, where each turn begins and ends, and which token tells the model its turn is over.

Speech bubbles feeding a stamping machine that outputs a tick-marked tape

Why the exact format matters

The tokens are not decoration. A chat model learns the template during post-training the same way it learns vocabulary. During instruction fine-tuning the model sees thousands of conversations rendered in one fixed format, and it learns that a role header means a speaker change, that an end-of-turn token means stop generating, and that the whole exchange follows those rules. When the inference-time rendering diverges from the training-time rendering, the model operates outside its training distribution.

Hugging Face, which maintains the template system, warns in blunt language: "Using a format different from the format a model was trained with will usually cause severe, silent performance degradation, so matching the format used during training is extremely important!" (Hugging Face blog, Introducing chat templates). Silent is the operative word. Nothing fails loudly. The model just gets worse, rambles past the end of its turn, or invents conversations with itself.

Three concrete behaviors the template controls:

  • The generation prompt. To make the model answer, the prompt must end with the opening of an assistant turn. Hugging Face exposes this as the add_generation_prompt flag on apply_chat_template, which appends the trailing assistant header so the model continues in its own voice (docs). In Llama 3 terms, that is the <|start_header_id|>assistant<|end_header_id|> at the end of the example above.
  • The stop condition. The end-of-turn token (<|im_end|>, <|eot_id|>) is what a serving stack watches to halt generation. A missing or mismatched delimiter is why a raw base model keeps "role-playing" the next user message instead of stopping.
  • Turn accounting in the tokenizer. In the ChatML era, OpenAI's own token-counting example added a fixed four tokens per message because "every message follows <im_start>{role/name}\n{content}<im_end>\n" (OpenAI community, quoting the tiktoken example). Your messages cost more than their content. That overhead is the template, billed per token.

Templates also explain the difference between a base model and its instruct variant. Meta's Llama 3 format notes that the <|end_of_text|> token "is generated only by the base models"; instruct models close turns with <|eot_id|> instead (Meta prompt format docs). The weights are the same family. The conversation contract is what was added.

A standard that standardized nothing

The closest thing the field has to a common wire format is ChatML, and it is no longer an official anything. OpenAI published a short ChatML specification as chatml.md inside its Python client library and then deleted the file when version 1.0 of the library landed in November 2023. Asked about a republication, an OpenAI staff engineer replied: "I would not expect this to be published again in the future" (OpenAI developer community; GitHub issue #506). The archived spec itself explains why the format is security-relevant. Raw-string rendering "inherently allows injections from user input containing special-token syntax, similar to SQL injections," while the structured form lets the model "tell which instructions come from the developer, the user, or its own input" (chatml.md, preserved from the openai-python repo). Delimiters are a trust boundary. If untrusted text can forge a turn boundary, it can forge a speaker.

Open source kept the habit anyway.

A segmented tape with gate markers, a scissors and a taped-in mismatched patch

Hugging Face's own guidance calls ChatML "the closest thing to a 'standard'" for chat formats, with <|im_start|> and <|im_end|> as special tokens (Hugging Face blog). Qwen and many derivative models still render conversations in ChatML. Meanwhile OpenAI's open-weights gpt-oss family moved to a different format, Harmony, which separates output into named channels such as analysis and final, again with role markup around each turn (Hugging Face blog, ChatML vs Harmony). Function calling and tool results ride on the same rails: the transformers docs describe template variants with names like tool_use, selected automatically when a caller passes a tools argument (docs).

There is no universal serialization for a conversation. There is a per-model dialect, shipped as a Jinja file, and you are expected to use the one that came with the weights.

The template in training, too

The format is not only an inference concern. Which tokens of a rendered conversation the model is graded on during supervised fine-tuning is a knob, and it has a name: prompt loss weight. A 2024 study found "a statistically-significant negative quadratic relationship" between that weight and downstream performance on short-completion data, meaning the standard practice of masking out prompt tokens is close to optimal, and it was written as "a warning to API providers" after OpenAI quietly removed the parameter from its fine-tuning API (Zhang et al., Instruction Fine-Tuning: Does Prompt Loss Matter?, arXiv:2401.13586). The boilerplate your template injects is also something the model could learn to predict, if anyone chose to train it that way.

What to do with this

If you self-host, the rule is simple: never hand-build a prompt string when the tokenizer ships a template. Call apply_chat_template, let it add the special tokens, and pass skip_special_tokens=True on decode so the delimiters never leak into your user-visible text. The docs warn that adding BOS or EOS tokens on top of a template output is "often incorrect or duplicated, hurting model performance" (Hugging Face transformers docs).

If you call an API, the template is still there. It runs on the provider's side, and it is the reason the messages array exists as an interface at all: the provider is refusing to let you, or anyone's retrieval pipeline, forge role boundaries in the raw string. The structure of your request is the format of your model, and you have simply outsourced the dialect.

The chat template is where a language model stops being a text completer and starts being a conversationalist. Someone wrote that contract in Jinja, shipped it in a tokenizer file, and told you never to touch it. Now you know what the plumbing is.

Sources

Explainers