Does Constrained Decoding Hurt LLM Accuracy? Mostly No

With structured output, you give the model a JSON schema, and the decoder blocks any token that would break it. That blocking "is often assumed to inherently degrade reasoning," in the words of a new paper on arXiv from Télécom Paris and Nokia Bell Labs. The paper tests that assumption directly and finds that, in most cases, the decoder is not the problem.

The paper, "Constrained Decoding: Harmful or Beneficial? Revisiting Its Impact on LLM Performance," is marked as an ICLR 2027 conference paper. Its main result: when the prompt and the expected output format stay the same, turning on constrained decoding matches or beats unconstrained decoding. The drop that earlier studies saw comes mostly from comparing different prompts and output formats.

The debate this paper answers

Constrained decoding (also called grammar-constrained or structured generation) works at the token level. At each step, the decoder masks every next token that the grammar does not allow, then renormalizes the probabilities of the tokens that remain. Libraries such as Outlines, XGrammar and guidance do this. For a side-by-side look at four such engines and the overhead of token masking, see our comparison of Outlines, llguidance, XGrammar and LM Format Enforcer.

In 2024, the paper "Let Me Speak Freely?" by Tam et al. reported accuracy drops when models had to reason inside a JSON schema. Other teams then proposed fixes, such as a "reason" field in the JSON, a two-step "reason freely, then format" pipeline, or a trigger token that switches from free text to constrained output. Other work disagreed with the diagnosis. The dottxt response argued that the comparisons were unfair, and "The Format Tax" by Lee et al. put the blame on the prompt.

The new paper sides with that second view and adds a formal proof and new experiments.

Three kinds of constraint

The authors split constraints into three types:

• Format constraints (FC) force the output structure, for example a JSON object with an answer field, but let the answer field hold any string.
• Validity constraints (VC) also limit what a valid answer can be, based on the task. In the experiments: a non-negative integer for GSM8K, an integer from 0 to 999 for AIME 2026, and exactly four or eight letters for the Last Letter tasks.
• Reasoning constraints (RC) force the reasoning to follow a set path. The test case was forcing the model to start with "Let's think step by step." Our chain-of-thought explainer covers how written-out reasoning steps work.

The theory part is short. If the correct answer is still allowed by the constraint, masking can only remove probability from invalid tokens, so the relative probability of the correct answer stays the same or goes up. The authors show that the gain is larger when the constraint removes more invalid tokens along the path of the correct answer. If a constraint excludes the correct answer, or excludes all good reasoning paths, it hurts. That is the case to avoid.

What the experiments show

The team tested ten open models for task solving: Qwen3 (1.7B, 4B, 8B), Qwen3.5 (2B, 4B, 9B, 27B), Llama3-8B, Gemma2-9B and SmolLM3-3B. Qwen3-32B served as the extractor in the two-step pipeline. Benchmarks were GSM8K, Last Letter Concatenation, AIME 2026 and a new LastLetter-Extended set with eight names instead of four. The authors say the last two were released or built after all the models, which reduces the risk of test leakage. They used the Litelines library, which is built on outlines-core.

Key results from the paper:

• Same prompt, same format: constraints win. Across the tables, combined format and validity constraints matched or beat unconstrained decoding. On Last Letter with the two-step pipeline, Qwen3.5-9B went from 86.8 to 89.9 accuracy, and Qwen3.5-2B went from 62.9 to 68.7.
• The large gaps come from format choice. Reasoning inside JSON still trailed free-form reasoning in several settings, as earlier work found. The authors attribute that to the prompt-level change, not to masking.
• Validity constraints give the most consistent gains. The authors report gains of up to 17.4% from validity constraints, and say they are more consistent than format constraints. Format constraints matter less when a model already follows format instructions well.
• Reasoning constraints are mixed. Forcing "Let's think step by step" generally helped on GSM8K and AIME 2026, but cut AIME 2026 accuracy by 6.6% for Qwen3.5-4B.

Why validity constraints help

The appendix shows what validity constraints fix in practice. On GSM8K, an answer of "28.00" is forced into the required form "28" and then matches. On Last Letter, a model that correctly lists the letters H, E, N, E but writes "H E N E" is pushed to "hene" by the four-letter rule. In other cases the constraint restores a letter that the model dropped from its final answer, or removes one it added, so that the answer agrees with the model's own reasoning.

The authors argue that this is more than post-processing. Spaces and extra letters can be part of tokens such as " e" or "da," so the decoder must pick different tokens that fit both the constraint and the reasoning before them.

A small extra test points at content filtering. On IMDb sentiment classification, the models had to answer without the words "Positive" or "Negative" and could only use alternatives such as "Good" or "Disappointing." Accuracy fell but stayed high. Qwen3.5-4B scored 94.6 unconstrained and 85.8 with the alternative words.

Parsing and cost

Unconstrained JSON output failed to parse much more often than the other strategies. The authors counted JSON wrapped in a Markdown code fence as a failure, which they admit may seem harsh. Constrained decoding improved format completion for every strategy, including for the larger Qwen3-32B and Qwen3.5-27B.

Cost differs by strategy. Reasoning inside JSON applies the mask for the full generation. The two-step pipeline needs a second model call and, per the paper, nearly doubles token usage. The trigger-token approach (which the authors call InWriting) added fewer than 15 tokens in their runs and constrains only the final answer.

Limits

The scope is narrow, and the authors say so. Validity constraints were tested mainly on the Last Letter tasks, plus the small IMDb test. The JSON formats were simple. Most runs used greedy decoding with thinking mode off and a 600-token reasoning cap. Only AIME 2026 used a longer 12,800-token budget. All models are open models of 32B parameters or less. The code is in the supplementary material and will be public on acceptance, according to the paper.

What it means for engineers

If structured output seems to hurt accuracy, check the prompt first. Compare constrained and unconstrained runs with the same prompt and the same expected format before you remove the schema. Then tighten the schema where the task allows it: integer ranges, enums and fixed lengths are validity constraints, and in this study they gave the most consistent gains. Be careful with constraints that force a reasoning style. And if constraint overhead matters, let the model reason in free text and constrain only the final answer, as the paper's InWriting approach does.

Sources

• Constrained Decoding: Harmful or Beneficial? (arXiv 2610.35328)
• Let Me Speak Freely? (Tam et al., EMNLP 2024 Industry Track)
• The Format Tax (Lee et al., arXiv 2604.03616)
• Say What You Mean (dottxt)
• Efficient Guided Generation for LLMs (Outlines, arXiv 2307.09702)
• XGrammar (Dong et al., arXiv 2411.15100)
• guidance on GitHub
• Litelines on GitHub