ReAct in Large Language Models: How Interleaving Reasoning and Action Traces Built the Foundation of AI Agents

Before autonomous agents could interact reliably with APIs, search engines, and bash environments, large language models (LLMs) operated in one of two disconnected paradigms: internal reasoning without external interaction, or external action generation without internal deliberation. In pure reasoning paradigms such as Chain-of-Thought (CoT) prompting, models generate intermediate natural language deduction steps using only static parametric memory, leaving them prone to factual hallucination and compounding reasoning errors. In pure action paradigms (Act-only), models emit direct API calls or environment commands without maintaining an explicit reasoning trace, making it difficult to decompose complex multi-step goals, track intermediate state, or recover from unexpected execution failures.

The Reason + Act (ReAct) paradigm, introduced by Shunyu Yao and colleagues from Princeton University and Google Research in their 2022 paper ReAct: Synergizing Reasoning and Acting in Language Models (published at ICLR 2023), solved this structural divide. By interleaving free-form natural language "thoughts" with concrete environment "actions" in an iterative loop, ReAct created a self-correcting feedback mechanism. The internal reasoning traces guide action selection and synthesize observation histories, while the external observations ground the reasoning traces in dynamic, verifiable facts.

Understanding the ReAct architecture, its mathematical formulation, its empirical failure modes, and its trade-offs is essential for analyzing modern agentic systems, tool-use protocols, and autonomous execution frameworks.

Formal Problem Formulation

In a standard interactive decision-making setup, an agent interacts with an environment $\mathcal{E}$ over discrete time steps $t = 1, \dots, T$. At each step $t$, the agent receives an observation $o_t \in \mathcal{O}$ and selects an action $a_t \in \mathcal{A}$ according to its policy $\pi(a_t \mid h_t)$, where $h_t = (c, o_1, a_1, o_2, a_2, \dots, o_{t-1}, a_{t-1}, o_t)$ denotes the interaction history and $c$ denotes the task context or initial prompt.

In traditional reinforcement learning or imitation learning setups, the action space $\mathcal{A}$ is strictly task-specific and executable by the environment (e.g., clicking a button, running a shell command, querying an API endpoint).

ReAct redefines the action space by expanding $\mathcal{A}$ into an augmented action space $\hat{\mathcal{A}} = \mathcal{A} \cup \mathcal{L}$, where $\mathcal{L}$ is the space of unconstrained natural language sequences (reasoning traces or "thoughts"):

1. Actions ($a_t \in \mathcal{A}$): Operations that interface directly with the environment $\mathcal{E}$, updating the environment state and returning a new external observation $o_{t+1} \sim \mathcal{E}(a_t)$.
2. Thoughts ($t_t \in \mathcal{L}$): Internal natural language utterances generated by the language model that do not alter the external environment state. Instead, they update the working context $h_t$, serving as explicit cognitive scaffolding for subsequent decisions.

Under this formulation, the interaction history at step $t$ becomes:

$$h_t = (c, o_1, \tau_1, o_2, \tau_2, \dots, o_{t-1}, \tau_{t-1}, o_t)$$

where each step $\tau_i$ can represent either a domain action $a_i \in \mathcal{A}$ or an internal reasoning trace $t_i \in \mathcal{L}$.

The Three Execution Paradigms

To understand why ReAct altered LLM agent design, it is necessary to compare how the three primary prompting paradigms process a multi-step objective.

1. Chain-of-Thought (Reason-Only):
   Question/Goal -> Thought 1 -> Thought 2 -> Thought 3 -> Final Answer
   (No environment interaction; completely reliant on internal parametric weights)

2. Act-Only (Action-Only):
   Question/Goal -> Action 1 -> Observation 1 -> Action 2 -> Observation 2 -> Finish
   (No internal deliberation; fails to track state, decompose goals, or synthesize data)

3. ReAct (Interleaved Thought and Action):
   Question/Goal -> Thought 1 -> Action 1 -> Observation 1 -> Thought 2 -> Action 2 -> Observation 2 -> Thought 3 -> Finish
   (Internal thoughts decompose goals; external observations ground and update thoughts)

1. Reason-Only (Chain-of-Thought)

Chain-of-Thought (Wei et al., 2022) prompts the model to generate intermediate reasoning tokens before predicting a final label or answer. While CoT unlocked substantial performance gains on arithmetic, symbolic manipulation, and commonsense reasoning tasks, it is completely ungrounded. The model cannot inspect external databases, check the current date, or verify intermediate assertions. When a multi-hop factual query requires up-to-date knowledge, an early incorrect token in the reasoning chain triggers cumulative compounding hallucinations.

2. Act-Only

Act-only systems prompt the LLM to emit API calls or environment actions sequentially without any internal natural language trace. In interactive benchmarks such as WebShop (web navigation) or ALFWorld (text-based embodied simulation), Act-only models suffer from severe working memory limitations. Because they cannot write out sub-goals or summarize dense observation histories, they frequently fall into repetitive action loops, fail to recognize when an action had no effect, and struggle to coordinate multi-variable constraints.

3. ReAct

ReAct coordinates both capabilities. The model generates a thought to decompose the problem into sub-tasks, identify information gaps, or extract relevant entities from the previous observation. It then issues an action to query the environment. When the environment returns an observation, the model generates a subsequent thought to interpret the result, assess whether the sub-goal was satisfied, and determine the next logical action.

The Functional Roles of "Thoughts" in ReAct

In the ReAct framework, natural language thoughts perform distinct cognitive functions depending on the task domain. Yao et al. categorized these roles into four main types:

1. Goal Decomposition and Sub-Goal Planning

When confronted with complex multi-hop queries (e.g., "Which magazine was started first, Arthur's Magazine or First for Women?"), the model cannot answer in a single retrieval step. A ReAct thought explicitly maps out the sequence:

• Thought 1: "I need to search Arthur's Magazine and First for Women, find their start dates, and compare them."
• Action 1: search[Arthur's Magazine]

2. Information Extraction and Observation Filtering

Raw observations from APIs or web pages often contain hundreds of irrelevant words. Thoughts act as an explicit semantic filter, extracting only the necessary variables into the model's active working context:

• Observation 1: "...Arthur's Magazine was an American literary magazine published in Philadelphia in the 19th century. First published in January 1844..."
• Thought 2: "Arthur's Magazine was started in 1844. Now I need to find the start date of First for Women."
• Action 2: search[First for Women]

3. Dynamic Plan Adjustment and Exception Handling

When an action fails (e.g., a search returns a disambiguation page or no results), an Act-only agent often repeats the identical query. ReAct enables runtime exception handling:

• Observation 2: "Could not find [First for Women]. Similar: [Woman's World], [First (magazine)]."
• Thought 3: "The search did not find First for Women directly. Let me try searching First (magazine) or check the exact title."
• Action 3: search[First (magazine)]

4. State Tracking and Final Synthesis

Once all sub-goals are resolved, a thought synthesizes the disparate observations before emitting the final answer action:

• Observation 3: "...First for Women is a woman's magazine published by Bauer Media Group in Englewood Cliffs, New Jersey. The magazine was started in 1989..."
• Thought 4: "Arthur's Magazine started in 1844. First for Women started in 1989. 1844 is earlier than 1989, so Arthur's Magazine was started first."
• Action 4: finish[Arthur's Magazine]

Empirical Benchmarks and Performance Analysis

The original ReAct paper evaluated the framework across two distinct task classes using PaLM-540B and GPT-3 (text-davinci-002):

1. Knowledge-Intensive Reasoning: Multi-hop question answering on HotpotQA and fact verification on FEVER, interacting with a Wikipedia search API (search, lookup, finish).
2. Interactive Decision Making: Embodied text tasks on ALFWorld (household object interaction) and simulated e-commerce tasks on WebShop (multi-attribute product search and purchasing).

Knowledge-Intensive Tasks: HotpotQA and FEVER

On HotpotQA (6-shot) and FEVER (3-shot), ReAct demonstrated marked advantages over Act-only and standard prompting methods, while providing distinct error trade-offs compared to CoT:

| Method | HotpotQA (Exact Match %) | FEVER (Accuracy %) | | :--- | :--- | :--- | | Standard Prompting | 28.7 | 57.1 | | Reason-Only (CoT) | 29.4 | 56.3 | | CoT with Self-Consistency (CoT-SC) | 33.4 | 60.4 | | Act-Only | 25.7 | 58.9 | | ReAct | 27.4 | 60.9 | | CoT-SC $\to$ ReAct (Fallback) | 34.2 | 64.6 | | ReAct $\to$ CoT-SC (Fallback) | 35.1 | 62.0 |

Source: Yao et al., ICLR 2023, using PaLM-540B.

On FEVER, ReAct alone (60.9%) outperformed both CoT (56.3%) and Act (58.9%). On HotpotQA, standalone ReAct achieved 27.4% Exact Match compared to CoT's 29.4%. The lower standalone EM score on HotpotQA was driven by search retrieval bottlenecks: when the Wikipedia API failed to return the target document, ReAct could not proceed.

However, combining ReAct and CoT in hybrid pipelines (ReAct $\to$ CoT-SC, where the model falls back to internal CoT reasoning if ReAct fails to retrieve within a step budget) yielded 35.1% EM on HotpotQA, establishing a substantial improvement over pure parametric reasoning.

Interactive Decision Making: ALFWorld and WebShop

On interactive environments requiring physical manipulation and web navigation, the synergy of ReAct produced its most pronounced gains:

| Method | ALFWorld Success Rate (2-shot) | WebShop Success Rate (1-shot) | WebShop Average Score | | :--- | :--- | :--- | :--- | | Imitation Learning (BUTLER / IL) | 37% (~100k samples) | 29.1% (~90k samples) | 59.9 | | Reinforcement Learning (RL) | - | 28.7% (~100k samples) | 55.7 | | Act-Only Prompting | 45% | 30.1% | 62.3 | | ReAct Prompting | 71% | 40.0% | 66.6 |

Source: Yao et al., ICLR 2023, using PaLM-540B.

On ALFWorld, ReAct achieved a 71% absolute success rate with only two in-context demonstration examples, outperforming the Act-only baseline (45%) by a relative 57.7% and beating specialized imitation learning models trained on 100,000 expert trajectories (37%).

On WebShop, one-shot ReAct achieved a 40.0% success rate, a 10% absolute gain over both Act-only and reinforcement learning baselines.

Error Analysis: Hallucination vs. Search Recovery

The pivotal finding of the ReAct paper lies in the qualitative and quantitative error breakdown between CoT and ReAct.

Error Distribution on Knowledge Verification:

Chain-of-Thought (CoT):
[ Hallucination / Fact Fabrication (56%) ] [ Reasoning Logic Errors (44%) ]

ReAct:
[ Search Failure / Incomplete Retrieval (64%) ] [ Reasoning Errors (23%) ] [ Hallucination (13%) ]

Chain-of-Thought Error Modes

In error analysis conducted on 100 random failure cases from HotpotQA:

• Hallucination accounted for 56% of CoT errors. Because CoT relies exclusively on static parametric weights, it invents plausible-sounding publication dates, names, and relationships. Once a hallucinated premise enters the sequence, subsequent reasoning steps treat it as ground truth.
• Reasoning errors accounted for 44%, where the model retrieved correct internal knowledge but failed to follow correct deductive logic.

ReAct Error Modes

In contrast, ReAct altered the failure landscape:

• Hallucination dropped to only 13% of errors. By forcing every factual claim to originate from an external Wikipedia observation, the model rarely fabricated facts.
• Search retrieval failure accounted for 64% of errors. The primary bottleneck was the rigidity of the search API (e.g., query mismatch, inability to find specific sub-sections, or running out of the maximum step budget).
• Reasoning errors dropped to 23%.

This divergence explains why hybrid methods succeed: ReAct eliminates factual fabrications by grounding the model in external search, while CoT serves as a safety net when external retrieval APIs return null or noisy documents.

Production Trade-offs and Architectural Costs

While ReAct forms the conceptual backbone of modern AI agents, deploying ReAct loops in production environments introduces several engineering trade-offs:

1. Cumulative Context Window Bloat

In a standard ReAct trajectory of $K$ steps, every step appends a thought $t_k$, an action $a_k$, and an environment observation $o_k$. At step $k$, the input prompt length is:

$$L_k = L_{\text{prompt}} + \sum_{j=1}^{k-1} (|t_j| + |a_j| + |o_j|)$$

Because LLM generation complexity scales with context length, late-stage execution steps consume significantly more prompt tokens. In production agents that run 15 to 30 steps, observation bloat can consume thousands of tokens per turn, increasing Time to First Token (TTFT) and inference cost.

2. Multi-Turn Latency Tax

A traditional one-shot LLM call requires a single forward generation pass. A $K$-step ReAct agent requires $K$ sequential LLM generations and $K-1$ round-trip environment executions (e.g., database queries, browser rendering, sandbox executions). An agent taking 8 steps with an average LLM latency of 800ms and tool latency of 400ms incurs an end-to-end user latency of nearly 10 seconds:

$$\text{Latency} = \sum_{k=1}^K \left( \text{TTFT}_k + \frac{|t_k| + |a_k|}{\text{Throughput}} + \text{ToolLatency}_k \right)$$

3. Trajectory Drift and Looping

Without explicit state compaction or loop detection, ReAct models can get trapped in repetitive action-observation cycles (e.g., repeatedly calling the same search query with minor punctuation changes). Production frameworks must implement explicit stopping criteria, maximum retry counts, and scratchpad summarization to mitigate runaway loops.

Architectural Evolution: From Text Traces to Function Calling

The original 2022 ReAct implementation relied on text parsing: the language model was prompted with text exemplars demonstrating Thought:, Action:, and Observation: delimiters, and the harness used string matching and regex parsers to intercept actions.

Over the subsequent years, the industry standardized and industrialized the ReAct loop:

1. Native Function Calling / Tool Use: Model providers (OpenAI, Anthropic, Google, Mistral) fine-tuned frontier models to output structured JSON tool calls inside specialized protocol delimiters (e.g., <tool_call>, <function>), eliminating fragile regex string parsing.
2. Hidden and Native Reasoning (Extended Thinking): Modern reasoning models (OpenAI o1/o3, Claude 3.7 Sonnet Extended Thinking, DeepSeek-R1) internalize intermediate reasoning tokens into dedicated thinking blocks before emitting tool calls, decoupling private chain-of-thought scratchpads from user-visible tool arguments.
3. Executable Code Actions (CodeAct): Modern coding agents have shifted from JSON action formats back to executable code blocks (e.g., Python scripts or bash commands). This allows the agent to execute complex loops, variable transformations, and data filtering inside an isolated execution sandbox before returning the final observation to the model context.

Despite these formatting and runtime optimizations, the underlying principle remains unchanged: reliable autonomous decision-making requires the continuous interleaving of internal reasoning state with external environment feedback.

Sources

• Yao, S., Zhao, J., Yu, D., Du, N., Shafran, I., Narasimhan, K., & Cao, Y. (2022). ReAct: Synergizing Reasoning and Acting in Language Models. International Conference on Learning Representations (ICLR 2023). arXiv:2210.03629
• Google Research. (2023). ReAct: Synergizing Reasoning and Acting in Language Models. Google Research Blog
• Wei, J., Wang, X., Schuurmans, D., Bosma, M., Xia, F., Chi, E., Le, Q. V., & Zhou, D. (2022). Chain-of-Thought Prompting Elicits Reasoning in Large Language Models. Advances in Neural Information Processing Systems (NeurIPS 2022). arXiv:2201.11903
• Wang, X., Wei, J., Schuurmans, D., Le, Q. V., Chi, E., Narang, S., Chowdhery, A., & Zhou, D. (2022). Self-Consistency Improves Chain of Thought Reasoning in Language Models. International Conference on Learning Representations (ICLR 2023). arXiv:2203.11171
• Yang, Z., Qi, P., Zhang, S., Bengio, Y., Cohen, W. W., Salakhutdinov, R., & Manning, C. D. (2018). HotpotQA: A Dataset for Diverse, Explainable Multi-hop Question Answering. EMNLP 2018. HotpotQA
• Thorne, J., Vlachos, A., Christodoulopoulos, C., & Mittal, A. (2018). FEVER: a large-scale dataset for Fact Extraction and VERification. NAACL-HLT 2018. FEVER
• Shridhar, M., Yuan, X., Côté, M. A., Bisk, Y., Trischler, A., & Hausknecht, M. (2020). ALFWorld: Aligning Text and Embodied Environments for Interactive Learning. ICLR 2021. ALFWorld
• Yao, S., Chen, H., Yang, J., & Narasimhan, K. (2022). WebShop: Towards Scalable Real-World Web Interaction with Grounded Language Agents. NeurIPS 2022. WebShop