AI Agents: Tools and Reasoning

A large language model is a function from a token sequence to a distribution over the next token. On its own it produces text and nothing else. An agent wraps that function in a loop: the model reads an observation, decides on an action, an environment executes the action, and the resulting observation feeds the next decision. The model becomes a controller rather than a generator, and the unit of study shifts from a single forward pass to a trajectory of interaction.¹

This is the formalism of a partially observed Markov decision process, but with two properties peculiar to language agents. First, the state is the text history itself: there is no hidden vector to carry forward, only the running transcript in the context window. Second, the policy is a fixed pretrained network; an agent is usually constructed by prompting and scaffolding a frozen model rather than by training one with reinforcement learning.

The loop runs until the policy emits a designated stop action (a final answer) or a budget on steps or cost is exhausted. Everything that follows is a way of filling in the three blanks: what the actions are (tools), how the policy chooses them (reasoning and search), and what the history can hold (memory and retrieval).

Tool use and function calling

The action space is made concrete by tools: named functions with typed arguments that the environment can execute. The model does not run code; it emits a structured request, an external executor runs the real function, and the return value is serialized back into the context as the next observation.

The model is taught to emit calls by training on demonstrations of the call format, so that the decision of when to call a tool and how to fill its arguments is itself learned from data rather than hand-scripted.² The control flow is a single loop: decode, dispatch, append, repeat.

Mechanically, a tool call is generated the same way as any other text. The action $a_t=(\text{name},\text{args})$ is a token string the model decodes autoregressively, so the probability of a specific call factors over its tokens,

and the choice among tools is a choice among token continuations. Two knobs govern selection. Greedy or low-temperature decoding takes the most probable call, which is the right default when the schema is unambiguous. Constrained decoding goes further: the executor supplies a grammar (the JSON schema of the registered tools) and the decoder masks any token that would violate it, so the model cannot emit an unparseable call or an unknown tool name. Grammar constraints turn a class of the failures below into impossibilities rather than errors to be caught after the fact.

The executor is the trust boundary. The model proposes; the executor validates the schema, enforces permissions, and decides what actually runs. A malformed or unauthorized call is caught here and returned as an error observation, which the model can read and correct on the next step.

Read as data flow the picture is static, but a single call is a sequence in time: the model decodes a call, control passes to the executor, the executor runs the tool, the return travels back, and only then does the model resume with the result now in its context. The horizontal axis below is time; each downward hop is a hand-off, and the model is idle (no decoding) while the tool runs.

ReAct: interleaving reasoning and action

A model that jumps straight to a tool call commits to it without deliberation. The ReAct pattern inserts a free-text thought before each action, so the trajectory alternates reasoning and acting.³ The thought is not executed; it is scratch reasoning that conditions the action the model then emits.

The thought tokens help for a mechanical reason. The action distribution is ${\pi }_{\theta }(a_t\mid h_{t-1},r_t)$; conditioning on $r_t$ lets the model spend forward-pass computation deriving and committing intermediate conclusions to the context before it must select a discrete action. Reasoning that would otherwise have to happen implicitly in a single step is made explicit and reusable across steps.

For example, take the question What is the population of the capital of the country that won the 2018 World Cup? A single forward pass would have to resolve three chained facts at once; ReAct spreads them across steps, each grounded by a tool return the model could not have known a priori.

Thought 1: I need the winner of the 2018 World Cup, then its capital, then that           city's population. Start with the winner.Action 1:  search("2018 FIFA World Cup winner")Observ. 1: France won the 2018 FIFA World Cup.Thought 2: The country is France. Its capital is Paris. I still need Paris's           population, so I will look that up rather than trust memory.Action 2:  search("population of Paris")Observ. 2: Paris has about 2.1 million residents (city proper).Thought 3: I have the chain: France -> Paris -> 2.1 million. I can answer.Action 3:  finish("About 2.1 million.")

Each observation narrows the next thought's scope, and the intermediate facts sit in the context where the final thought can read them back. Had observation 1 returned a wrong country, thought 2 would have built on the error — which is why the pattern pairs naturally with the verification and reflection mechanisms below.

ReAct also makes the agent correctable. Because reasoning is written down, a wrong observation can be diagnosed in the next thought rather than silently propagated, and the same channel carries plans, error analysis, and the decision to stop. A recoverable trajectory looks like this: the model reads an error observation, names the fault in the next thought, and reissues a corrected action.

Action 1:  calculator("17 * ")            # malformed: trailing operatorObserv. 1: ERROR: could not parse expression "17 * "Thought 2: My call was cut off. The intended computation was 17 * 23. Retry.Action 2:  calculator("17 * 23")Observ. 2: 391

The error observation is data like any other. The model that reads it can localize the mistake and repair it, provided the executor returns a message specific enough to act on rather than a bare failure code.

Planning and search over thoughts

ReAct reasons one step at a time and greedily. When a problem needs multi-step deduction, the reasoning itself becomes a search problem, and several strategies trade compute for reliability by expanding the thought process.

Chain-of-thought (CoT). Prompt the model to emit intermediate reasoning steps before the answer, so it computes $p(\text{answer},\text{reasoning}\mid x)$ and reads the answer off the end.⁴ A single sampled chain is one walk through the reasoning, and its correctness is correlated with, but not guaranteed by, the plausibility of the steps.

Self-consistency. Rather than trust one chain, sample $m$ chains independently at nonzero temperature and take a majority vote over their final answers.⁵ This marginalizes the answer over reasoning paths.

Least-to-most. Decompose the problem into an ordered list of simpler subproblems, then solve them in sequence, each conditioned on the answers to the previous ones.⁶ This makes the dependency structure of the reasoning explicit instead of leaving it to a single chain. The pattern splits into two phases: a planning pass that emits the subproblem list, and an execution pass that walks the list, feeding each solved subproblem's answer forward as context for the next.

Tree of Thoughts (ToT). Generalize the single chain to a tree: a node is a partial solution (a prefix of reasoning steps), the model proposes several next steps to branch on, and a learned or prompted value heuristic $v(\text{node})$ scores each partial solution so a search procedure (BFS or DFS) can expand the promising frontier and prune the rest.⁷

The three families increase in cost and structure. CoT is one sample; self-consistency is $m$ independent samples reduced by voting; ToT is a structured search with backtracking and a value function.

The progression trades compute for accuracy. Self-consistency helps most where a correct answer is reachable by many chains; ToT helps where the solution requires lookahead and the ability to abandon a dead end, at the price of many more model calls and a value function that must itself be reliable.

From prompted search to trained reasoning

The search strategies above are all prompting patterns wrapped around a frozen model: chain-of-thought, self-consistency, and Tree of Thoughts change what tokens the model generates, not what the weights encode. Two developments since move the reasoning inside the model, and they reframe the comparison.

Reasoning models internalize the chain. Rather than prompt for intermediate steps, a reasoning model is trained so that emitting a long internal deliberation before the answer is its default behavior, with the deliberation rewarded by outcome-based reinforcement learning on verifiable problems (math, code) where a checker can score the final answer.⁸ The effect is that self-consistency and Tree-of-Thoughts-style exploration become behaviors the model learned to run internally, spending more decoding tokens on hard problems and fewer on easy ones. Chain-of-thought stops being a prompt and becomes a trained policy.

Verification scales better than generation. Self-consistency takes a majority vote because it has no way to check which chain is right. When a verifier is available — a unit test, a proof checker, or a separately trained reward model that scores solution correctness — the agent can generate many candidates and keep the ones that pass, which is far more reliable than voting.⁹ This is the search-versus-verify asymmetry that runs through the whole subject: proposing a solution is cheap and noisy, checking one is often cheap and exact, and an agent that can verify its own steps breaks the horizon-error compounding that defeats an unchecked chain.

The reframing is that the CoT / self-consistency / ToT progression measures how much inference-time compute a prompted model spends on reasoning; reasoning models internalize the cheap strategies and let outcome rewards teach the expensive ones, and verifiers convert blind voting into filtered search. The mechanism this lesson built — a thought conditions the next action — is unchanged; what moved is whether the thoughts are prompted or trained, and whether their quality is voted on or checked.

Takeaways

• An agent is a policy ${\pi }_{\theta }:\mathcal{H}\to \Delta (\mathcal{A})$ over interaction histories: a frozen language model wrapped in a perceive–decide–act loop, where the state is the running text transcript in the context window.
• Tool use turns text generation into action: the model emits a structured call, an executor validates and runs it at the trust boundary, and the result is appended as the next observation. Constrained decoding against the tool schema turns a class of invalid calls into impossibilities.
• ReAct interleaves free-text thoughts with actions; the thought tokens let the model spend computation deliberating before committing to a discrete action, and make the trajectory diagnosable and correctable from error observations.
• Search over thoughts trades compute for reliability: chain-of-thought is one path, self-consistency marginalizes by majority vote over $m$ paths, least-to-most decomposes into ordered subproblems, and Tree of Thoughts searches a tree of partial solutions under a value heuristic with backtracking.
• Trained reasoning: reasoning models train the chain in with outcome-based RL, and verifiers (tests, checkers, reward models) turn blind voting into filtered search, breaking the horizon-error compounding an unchecked chain suffers.