--- title: "AI Agents: Tools and Reasoning" module: Large Models & Agents moduleNumber: 10 lessonNumber: 7 order: 1007 summary: > A language model that only emits text is a function from prompt to prompt; an agent closes the loop, letting that model act on an environment, read back the result, and decide again. This first part formalizes the agent as a policy over interaction histories, builds out tool calling and the executor trust boundary, the ReAct interleaving of reasoning and action (with concrete traces), and search over thoughts: chain-of-thought, self-consistency, least-to-most, and Tree of Thoughts. Memory, retrieval, reflection, and multi-agent orchestration continue in part two. topics: [Large Models & Agents] sources: - book: Chollet ref: "Ch. 11 — language models as components of larger systems" - book: Goodfellow ref: "Ch. 1 — AI systems that perceive, reason, and act" --- A [large language model](/deep-learning/large-models-and-agents/large-language-models) is a function from a token sequence to a distribution over the next token. On its own it produces text and nothing else. An **agent** wraps that function in a loop: the model reads an observation, decides on an action, an environment executes the action, and the resulting observation feeds the next decision. The model becomes a **controller** rather than a generator, and the unit of study shifts from a single forward pass to a trajectory of interaction.[^chollet-systems] > **Definition (LLM agent).** An agent is a tuple $\parens{\mathcal{O}, > \mathcal{A}, \pi, \text{env}}$ where $\mathcal{O}$ is a space of observations > (text the model reads), $\mathcal{A}$ is a space of actions the environment > exposes (tool calls, queries, or a final answer), and the policy > $$ > \pi_\theta : \mathcal{H} \to \Delta(\mathcal{A}), \qquad > \mathcal{H} = \parens{\mathcal{O} \times \mathcal{A}}^{\ast} \times \mathcal{O}, > $$ > maps an interaction history $h_t = (o_0, a_0, o_1, a_1, \dots, o_t)$ to a > distribution over the next action. The policy is realized by the language model: > the history is serialized into the context window and $a_t$ is the model's > decoded output. The environment returns $o_{t+1} = \text{env}(h_t, a_t)$. This is the formalism of a partially observed Markov decision process, but with two properties peculiar to language agents. First, the state is the _text history_ itself: there is no hidden vector to carry forward, only the running transcript in the context window. Second, the policy is a fixed pretrained network; an agent is usually constructed by _prompting and scaffolding_ a frozen model rather than by training one with reinforcement learning. $$ % caption: The agent loop. The policy reads the running history, emits an action, % the environment executes it and returns an observation appended to the history. \begin{tikzpicture}[>=stealth, font=\footnotesize, box/.style={draw, minimum width=26mm, minimum height=11mm, align=center}, env/.style={draw, minimum width=26mm, minimum height=11mm, align=center}] \definecolor{acc}{HTML}{2348F2} \definecolor{green}{HTML}{1F9D4D} \node[box, draw=acc, text=acc, thick] (pol) at (0,0) {\texttt{policy}\\\texttt{(language model)}}; \node[env, draw=green, text=green, thick] (env) at (6.4,0) {\texttt{environment}\\\texttt{(actions, world)}}; \node[box] (hist) at (3.2,-3.0) {\texttt{history} $h_t$\\\texttt{(context window)}}; % action: policy -> environment \draw[->, acc, thick] (pol.east) -- (env.west) node[midway, above, text=acc] {\texttt{action} $a_t$}; % observation: environment -> history \draw[->, green, thick] (env.south) |- (hist.east) node[pos=0.72, above, text=green] {\texttt{observation}}; % history -> policy \draw[->, black, thick] (hist.west) -| (pol.south) node[pos=0.28, below, black] {\texttt{read context}}; \node[black, anchor=south] at (3.2,1.0) {\texttt{decide, act, observe, repeat}}; \end{tikzpicture} $$ The loop runs until the policy emits a designated **stop** action (a final answer) or a budget on steps or cost is exhausted. Everything that follows is a way of filling in the three blanks: what the actions are (tools), how the policy chooses them (reasoning and search), and what the history can hold (memory and retrieval). ## Tool use and function calling The action space is made concrete by **tools**: named functions with typed arguments that the environment can execute. The model does not run code; it emits a structured request, an external **executor** runs the real function, and the return value is serialized back into the context as the next observation. > **Definition (Tool / function call).** A tool is a function > $g : \mathcal{X}_g \to \mathcal{Y}_g$ registered with the executor under a name, > with a schema describing its arguments. A function call is an action > $a_t = \parens{\text{name}, \text{args}}$ that the model emits as structured text > (typically JSON). The executor parses it, runs $y = g(\text{args})$, and appends > the serialized result $o_{t+1} = \serialize(y)$ to the history. The model is taught to emit calls by training on demonstrations of the call format, so that the decision of _when_ to call a tool and _how_ to fill its arguments is itself learned from data rather than hand-scripted.[^schick-toolformer] The control flow is a single loop: decode, dispatch, append, repeat. Mechanically, a tool call is generated the same way as any other text. The action $a_t = (\text{name}, \text{args})$ is a token string the model decodes autoregressively, so the probability of a specific call factors over its tokens, $$ \pi_\theta(a_t \mid h_{t-1}) = \prod_{i=1}^{|a_t|} p_\theta\!\parens{a_t^{(i)} \mid h_{t-1}, a_t^{(1:i-1)}}, $$ and the choice among tools is a choice among token continuations. Two knobs govern selection. Greedy or low-temperature decoding takes the most probable call, which is the right default when the schema is unambiguous. Constrained decoding goes further: the executor supplies a grammar (the JSON schema of the registered tools) and the decoder masks any token that would violate it, so the model _cannot_ emit an unparseable call or an unknown tool name. Grammar constraints turn a class of the failures below into impossibilities rather than errors to be caught after the fact. ```algorithm caption: $\textsc{ToolCallingLoop}(\text{prompt}, \text{tools})$ — run the agent until it answers $h \gets \text{prompt}$ // initialize the history repeat $a \gets \pi_\theta(h)$ // model decodes the next action if $a$ is a final answer then return $a$ $(\text{name}, \text{args}) \gets \text{parse}(a)$ if name $\notin$ tools or args invalid then $o \gets \text{error message}$ // surface the fault to the model else $o \gets \text{tools}[\text{name}](\text{args})$ // execute the tool $h \gets h \mathbin{\Vert} a \mathbin{\Vert} o$ // append call and result until step budget exhausted return best answer so far ``` The executor is the trust boundary. The model proposes; the executor validates the schema, enforces permissions, and decides what actually runs. A malformed or unauthorized call is caught here and returned as an error observation, which the model can read and correct on the next step. $$ % caption: Tool-calling data flow. The model emits a structured call; the executor % validates and dispatches it to a tool; the result is appended to the context. \begin{tikzpicture}[>=stealth, font=\footnotesize, mdl/.style={draw, draw=acc, text=acc, thick, minimum width=22mm, minimum height=10mm, align=center}, xc/.style={draw, minimum width=22mm, minimum height=10mm, align=center}, tl/.style={draw, draw=green, text=green, thick, minimum width=18mm, minimum height=7mm, align=center}] \definecolor{acc}{HTML}{2348F2} \definecolor{green}{HTML}{1F9D4D} \node[mdl] (model) at (0,0) {\texttt{model}}; \node[xc] (exec) at (4.2,0) {\texttt{executor}}; \node[tl] (t1) at (8.4,1.4) {\texttt{search}}; \node[tl] (t2) at (8.4,0.0) {\texttt{calculator}}; \node[tl] (t3) at (8.4,-1.4) {\texttt{code run}}; % call out / result back — symmetric arcs between the two faces \draw[->, acc, thick] (model.east) to[bend left=32] node[midway, above, text=acc] {\texttt{call}} (exec.west); \draw[->, green, thick] (exec.west) to[bend left=32] node[midway, below, text=green] {\texttt{result}} (model.east); % dispatch to tools \draw[->, black, thick] (exec.east) -- (t1.west); \draw[->, black, thick] (exec.east) -- (t2.west); \draw[->, black, thick] (exec.east) -- (t3.west); \node[black, anchor=south, font=\footnotesize] at (4.2,1.5) {\texttt{validate \& dispatch}}; \end{tikzpicture} $$ Read as data flow the picture is static, but a single call is a sequence in time: the model decodes a call, control passes to the executor, the executor runs the tool, the return travels back, and only then does the model resume with the result now in its context. The horizontal axis below is time; each downward hop is a hand-off, and the model is idle (no decoding) while the tool runs. $$ % caption: One tool call over time. The model decodes a call and blocks; the executor % validates and runs the tool; the return is serialized back as the next observation, % and only then does the model resume decoding. \begin{tikzpicture}[>=stealth, font=\footnotesize] \definecolor{acc}{HTML}{2348F2} \definecolor{green}{HTML}{1F9D4D} % three lifelines \node[draw, draw=acc, text=acc, thick, minimum width=20mm] (mh) at (0,0) {\texttt{model}}; \node[draw, minimum width=20mm] (eh) at (5,0) {\texttt{executor}}; \node[draw, draw=green, text=green, thick, minimum width=20mm] (th) at (10,0) {\texttt{tool}}; \draw[black, dashed] (mh.south) -- ++(0,-5.4); \draw[black, dashed] (eh.south) -- ++(0,-5.4); \draw[black, dashed] (th.south) -- ++(0,-5.4); % messages \draw[->, acc, thick] (0,-1.0) -- (5,-1.0) node[midway, above, text=acc, font=\footnotesize] {\texttt{emit call}}; \draw[->, black, thick] (5,-2.0) -- (10,-2.0) node[midway, above, font=\footnotesize] {\texttt{dispatch}}; \draw[->, green, thick] (10,-3.2) -- (5,-3.2) node[midway, above, text=green, font=\footnotesize] {\texttt{return value}}; \draw[->, green, thick] (5,-4.2) -- (0,-4.2) node[midway, above, text=green, font=\footnotesize] {\texttt{observation}}; % model idle band \draw[black, |-|] (-1.4,-1.0) -- (-1.4,-4.2) node[midway, left, text=black, font=\footnotesize, align=center] {\texttt{model}\\\texttt{idle}}; % tool active band \draw[green, line width=2pt] (10,-2.0) -- (10,-3.2); \end{tikzpicture} $$ ## ReAct: interleaving reasoning and action A model that jumps straight to a tool call commits to it without deliberation. The **ReAct** pattern inserts a free-text **thought** before each action, so the trajectory alternates reasoning and acting.[^yao-react] The thought is not executed; it is scratch reasoning that conditions the action the model then emits. > **Definition (ReAct trajectory).** A trajectory in which the policy emits, at each > step, a thought $r_t$ (free text), then an action $a_t$, and reads an observation > $o_t$, producing the alternation > $$ > r_1, a_1, o_1, \; r_2, a_2, o_2, \; \dots, \; r_T, a_T, o_T . > $$ > The thought tokens are appended to the history alongside actions and observations, > so each new action is conditioned on the full record of prior reasoning. The thought tokens help for a mechanical reason. The action distribution is $\pi_\theta(a_t \mid h_{t-1}, r_t)$; conditioning on $r_t$ lets the model spend forward-pass computation deriving and committing intermediate conclusions to the context before it must select a discrete action. Reasoning that would otherwise have to happen implicitly in a single step is made explicit and reusable across steps. $$ % caption: The ReAct loop, read left to right. Each pass reasons (Thought), acts by % calling a tool (Action), and reads the tool's return (Observation); the observation % conditions the next thought, and the loop repeats until the model emits a finish. \begin{tikzpicture}[>=stealth, font=\footnotesize, th/.style={draw, draw=acc, text=acc, thick, minimum width=22mm, minimum height=9mm, align=center}, ac/.style={draw, thick, minimum width=22mm, minimum height=9mm, align=center}, ob/.style={draw, draw=green, text=green, thick, minimum width=24mm, minimum height=9mm, align=center}, env/.style={draw, draw=black, text=black, dashed, minimum width=18mm, minimum height=7mm, align=center, font=\scriptsize}, sub/.style={font=\scriptsize, text=black}] \definecolor{acc}{HTML}{2348F2} \definecolor{green}{HTML}{1F9D4D} \node[th] (r) at (0,0) {\texttt{Thought}}; \node[ac] (a) at (3.7,0) {\texttt{Action}}; \node[ob] (o) at (7.7,0) {\texttt{Observation}}; \node[ac, draw=acc, text=acc] (f) at (11.6,0) {\texttt{finish}}; \draw[->, thick] (r) -- node[above, sub] {\texttt{reason}} (a); \draw[->, thick] (a) -- node[above, sub] {\texttt{act}} (o); \draw[->, thick] (o) -- node[above, sub] {\texttt{done}} (f); % the loop: the observation conditions the next thought \draw[->, thick, acc] (o.north) to[bend right=26] node[above, sub, text=acc] {\texttt{next thought}} (r.north); % the action's tool call: out to the tool, result returns as the observation \node[env] (t) at (5.7,-2.0) {\texttt{search tool}}; \draw[->, dashed, black] (a.south) to[bend right=18] node[midway, below, sub] {\texttt{call}} (t.west); \draw[->, dashed, black] (t.east) to[bend right=18] node[midway, below, sub] {\texttt{result}} (o.south); \end{tikzpicture} $$ For example, take the question _"What is the population of the capital of the country that won the 2018 World Cup?"_ A single forward pass would have to resolve three chained facts at once; ReAct spreads them across steps, each grounded by a tool return the model could not have known a priori. ```text Thought 1: I need the winner of the 2018 World Cup, then its capital, then that city's population. Start with the winner. Action 1: search("2018 FIFA World Cup winner") Observ. 1: France won the 2018 FIFA World Cup. Thought 2: The country is France. Its capital is Paris. I still need Paris's population, so I will look that up rather than trust memory. Action 2: search("population of Paris") Observ. 2: Paris has about 2.1 million residents (city proper). Thought 3: I have the chain: France -> Paris -> 2.1 million. I can answer. Action 3: finish("About 2.1 million.") ``` Each observation narrows the next thought's scope, and the intermediate facts sit in the context where the final thought can read them back. Had observation 1 returned a wrong country, thought 2 would have built on the error — which is why the pattern pairs naturally with the verification and reflection mechanisms below. ReAct also makes the agent _correctable_. Because reasoning is written down, a wrong observation can be diagnosed in the next thought rather than silently propagated, and the same channel carries plans, error analysis, and the decision to stop. A recoverable trajectory looks like this: the model reads an error observation, names the fault in the next thought, and reissues a corrected action. ```text Action 1: calculator("17 * ") # malformed: trailing operator Observ. 1: ERROR: could not parse expression "17 * " Thought 2: My call was cut off. The intended computation was 17 * 23. Retry. Action 2: calculator("17 * 23") Observ. 2: 391 ``` The error observation is data like any other. The model that reads it can localize the mistake and repair it, provided the executor returns a message specific enough to act on rather than a bare failure code. ## Planning and search over thoughts ReAct reasons one step at a time and greedily. When a problem needs multi-step deduction, the reasoning itself becomes a search problem, and several strategies trade compute for reliability by expanding the thought process. **Chain-of-thought (CoT).** Prompt the model to emit intermediate reasoning steps before the answer, so it computes $p(\text{answer}, \text{reasoning} \mid x)$ and reads the answer off the end.[^wei-cot] A single sampled chain is one walk through the reasoning, and its correctness is correlated with, but not guaranteed by, the plausibility of the steps. **Self-consistency.** Rather than trust one chain, sample $m$ chains independently at nonzero temperature and take a majority vote over their final answers.[^wang-sc] This marginalizes the answer over reasoning paths. > **Definition (Self-consistency decoding).** Draw $m$ chains > $\parens{r^{(j)}, y^{(j)}} \sim p_\theta(\cdot \mid x)$ and return the answer with > the most votes, > $$ > \hat{y} = \arg\max_{y} \sum_{j=1}^{m} \mathbb{1}\brackets{y^{(j)} = y}, > $$ > an estimate of $\arg\max_y \sum_{r} p_\theta(y, r \mid x)$ that marginalizes over > the latent reasoning path $r$. Correct answers are reached by many distinct chains > and accumulate votes; idiosyncratic errors scatter and do not. **Least-to-most.** Decompose the problem into an ordered list of simpler subproblems, then solve them in sequence, each conditioned on the answers to the previous ones.[^zhou-l2m] This makes the dependency structure of the reasoning explicit instead of leaving it to a single chain. The pattern splits into two phases: a _planning_ pass that emits the subproblem list, and an _execution_ pass that walks the list, feeding each solved subproblem's answer forward as context for the next. $$ % caption: Least-to-most decomposition. A planning pass turns the task into an ordered % subproblem list; execution solves them left to right, each solution conditioning the % next, until the final subproblem yields the answer. \begin{tikzpicture}[>=stealth, font=\footnotesize, task/.style={draw, draw=acc, text=acc, thick, minimum width=24mm, minimum height=10mm, align=center}, sp/.style={draw, minimum width=18mm, minimum height=9mm, align=center}, ans/.style={draw, draw=green, text=green, thick, minimum width=18mm, minimum height=9mm, align=center}] \definecolor{acc}{HTML}{2348F2} \definecolor{green}{HTML}{1F9D4D} \node[task] (task) at (0,0) {\texttt{task} $x$}; \node[sp] (s1) at (3.6,0) {\texttt{sub 1}}; \node[sp] (s2) at (6.8,0) {\texttt{sub 2}}; \node[ans] (s3) at (10.2,0) {\texttt{sub 3}}; \draw[->, acc, thick] (task) -- node[above, font=\footnotesize, text=acc] {\texttt{plan}} (s1); \draw[->, thick] (s1) -- node[above, font=\footnotesize] {\texttt{solve}} (s2); \draw[->, thick] (s2) -- node[above, font=\footnotesize] {\texttt{solve}} (s3); % answers feed forward \draw[->, green, thick] (s1.south) to[bend right=22] node[below, font=\footnotesize, text=green] {\texttt{answer 1}} (s2.south); \draw[->, green, thick] (s2.south) to[bend right=22] node[below, font=\footnotesize, text=green] {\texttt{answer 2}} (s3.south); \node[black, anchor=south, font=\footnotesize] at (5.2,1.1) {\texttt{planning}}; \node[black, anchor=south, font=\footnotesize] at (8.5,1.1) {\texttt{execution}}; \end{tikzpicture} $$ **Tree of Thoughts (ToT).** Generalize the single chain to a tree: a node is a partial solution (a prefix of reasoning steps), the model proposes several next steps to branch on, and a learned or prompted **value heuristic** $v(\text{node})$ scores each partial solution so a search procedure (BFS or DFS) can expand the promising frontier and prune the rest.[^yao-tot] $$ % caption: Tree of Thoughts. Each node is a partial solution; the model branches and % a value heuristic prunes weak nodes (red), expanding the promising frontier (blue). \begin{tikzpicture}[>=stealth, font=\footnotesize, n/.style={draw, circle, minimum size=8mm, inner sep=0pt}, good/.style={draw=acc, text=acc, thick}, bad/.style={draw=red, text=red, thick}] \definecolor{acc}{HTML}{2348F2} \definecolor{red}{HTML}{C0392B} \node[n, good] (root) at (0,0) {$s$}; \node[n, good] (a) at (-3.0,-2.0) {$a$}; \node[n, bad] (b) at (0,-2.0) {$b$}; \node[n, good] (c) at (3.0,-2.0) {$c$}; \node[n, good] (a1) at (-4.2,-4.0) {$a_1$}; \node[n, bad] (a2) at (-1.8,-4.0) {$a_2$}; \node[n, bad] (c1) at (1.8,-4.0) {$c_1$}; \node[n, good] (c2) at (4.2,-4.0) {$c_2$}; \draw[->, acc, thick] (root) -- (a); \draw[->, red, thick] (root) -- (b); \draw[->, acc, thick] (root) -- (c); \draw[->, acc, thick] (a) -- (a1); \draw[->, red, thick] (a) -- (a2); \draw[->, red, thick] (c) -- (c1); \draw[->, acc, thick] (c) -- (c2); \node[acc, anchor=west, font=\footnotesize] at (4.8,-2.0) {\texttt{kept (high value)}}; \node[red, anchor=west, font=\footnotesize] at (4.8,-3.0) {\texttt{pruned (low value)}}; \end{tikzpicture} $$ The three families increase in cost and structure. CoT is one sample; self-consistency is $m$ independent samples reduced by voting; ToT is a structured search with backtracking and a value function. | Method | What is explored | Selection rule | Backtracking | Cost (calls) | | --- | --- | --- | --- | --- | | Chain-of-thought | one reasoning path | take the final step | none | $1$ | | Self-consistency | $m$ independent paths | majority vote | none | $m$ | | Tree of Thoughts | a tree of partial solutions | value heuristic $v(\cdot)$, BFS/DFS | yes | $\gg m$ | The progression trades compute for accuracy. Self-consistency helps most where a correct answer is reachable by many chains; ToT helps where the solution requires lookahead and the ability to abandon a dead end, at the price of many more model calls and a value function that must itself be reliable. ## From prompted search to trained reasoning The search strategies above are all _prompting_ patterns wrapped around a frozen model: chain-of-thought, self-consistency, and Tree of Thoughts change what tokens the model generates, not what the weights encode. Two developments since move the reasoning inside the model, and they reframe the comparison. **Reasoning models internalize the chain.** Rather than prompt for intermediate steps, a **reasoning model** is trained so that emitting a long internal deliberation before the answer is its default behavior, with the deliberation rewarded by outcome-based reinforcement learning on verifiable problems (math, code) where a checker can score the final answer.[^deepseek-r1] The effect is that self-consistency and Tree-of-Thoughts-style exploration become behaviors the model learned to run internally, spending more decoding tokens on hard problems and fewer on easy ones. Chain-of-thought stops being a prompt and becomes a trained policy. **Verification scales better than generation.** Self-consistency takes a majority vote because it has no way to _check_ which chain is right. When a verifier is available — a unit test, a proof checker, or a separately trained reward model that scores solution correctness — the agent can generate many candidates and keep the ones that pass, which is far more reliable than voting.[^cobbe-verifiers] This is the search-versus-verify asymmetry that runs through the whole subject: proposing a solution is cheap and noisy, checking one is often cheap and exact, and an agent that can verify its own steps breaks the horizon-error compounding that defeats an unchecked chain. The reframing is that the CoT / self-consistency / ToT progression measures how much _inference-time_ compute a prompted model spends on reasoning; reasoning models internalize the cheap strategies and let outcome rewards teach the expensive ones, and verifiers convert blind voting into filtered search. The mechanism this lesson built — a thought conditions the next action — is unchanged; what moved is whether the thoughts are prompted or trained, and whether their quality is voted on or checked. ## Takeaways - An **agent** is a policy $\pi_\theta : \mathcal{H} \to \Delta(\mathcal{A})$ over interaction histories: a frozen language model wrapped in a perceive–decide–act loop, where the state is the running text transcript in the context window. - **Tool use** turns text generation into action: the model emits a structured call, an **executor** validates and runs it at the trust boundary, and the result is appended as the next observation. Constrained decoding against the tool schema turns a class of invalid calls into impossibilities. - **ReAct** interleaves free-text thoughts with actions; the thought tokens let the model spend computation deliberating before committing to a discrete action, and make the trajectory diagnosable and correctable from error observations. - **Search over thoughts** trades compute for reliability: **chain-of-thought** is one path, **self-consistency** marginalizes by majority vote over $m$ paths, **least-to-most** decomposes into ordered subproblems, and **Tree of Thoughts** searches a tree of partial solutions under a value heuristic with backtracking. - **Trained reasoning:** reasoning models train the chain in with outcome-based RL, and verifiers (tests, checkers, reward models) turn blind voting into filtered search, breaking the horizon-error compounding an unchecked chain suffers. [^chollet-systems]: **Chollet**, _Deep Learning with Python_, Ch. 11 — language models as components of larger systems: the model supplies the policy, while tools, retrieval, and control flow make it act on an environment. [^yao-react]: **Yao et al.**, _ReAct: Synergizing Reasoning and Acting in Language Models_, 2022 — interleaves chain-of-thought reasoning traces with tool actions so each action is conditioned on explicit prior reasoning. [^wei-cot]: **Wei et al.**, _Chain-of-Thought Prompting Elicits Reasoning in Large Language Models_, 2022 — prompting the model to emit intermediate steps before the answer markedly improves multi-step reasoning. [^wang-sc]: **Wang et al.**, _Self-Consistency Improves Chain of Thought Reasoning in Language Models_, 2023 — sample many reasoning paths and take a majority vote, marginalizing the answer over latent chains. [^yao-tot]: **Yao et al.**, _Tree of Thoughts: Deliberate Problem Solving with Large Language Models_, 2023 — searches a tree of partial solutions with BFS/DFS and a value heuristic that prunes weak branches. [^zhou-l2m]: **Zhou et al.**, _Least-to-Most Prompting Enables Complex Reasoning in Large Language Models_, 2023 — decompose a problem into ordered subproblems solved in sequence, each conditioned on the prior answers. [^schick-toolformer]: **Schick et al.**, _Toolformer: Language Models Can Teach Themselves to Use Tools_, 2023 — the model learns from self-supervised demonstrations when to call an API and how to fill its arguments. [^deepseek-r1]: **DeepSeek-AI**, _DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning_, 2025 — trains long chain-of-thought reasoning directly with outcome-based reinforcement learning on verifiable problems, internalizing deliberation rather than prompting for it. [^cobbe-verifiers]: **Cobbe et al.**, _Training Verifiers to Solve Math Word Problems_ (GSM8K), 2021 — a trained verifier that scores candidate solutions lets generate-and-check outperform majority voting on multi-step reasoning.