---
title: "Agent Memory, Retrieval, and Orchestration"
module: Large Models & Agents
moduleNumber: 10
lessonNumber: 8
order: 1008
summary: >
  An agent's reasoning and tool use only matter if it can remember what it learned
  and coordinate work larger than one context window. This second part builds the
  systems around the loop: short-term scratchpad versus long-term vector store,
  retrieval-augmented generation with a worked softmax over passage scores,
  reflection (Reflexion, Self-Refine), and multi-agent orchestration. It closes on
  the failure modes that bound agents — invalid tool calls, horizon-error
  compounding, context overflow, non-terminating loops — and the benchmarks that
  score the full loop.
topics: [Large Models & Agents]
sources:
  - book: Chollet
    ref: "Ch. 11 — language models as components of larger systems"
  - book: Goodfellow
    ref: "Ch. 1 — AI systems that perceive, reason, and act"
---

This builds on [AI Agents: Tools and Reasoning](/deep-learning/large-models-and-agents/ai-agents),
which formalized the agent as a policy over interaction histories and filled in two
of its three blanks: the actions (tools, run behind an executor trust boundary) and
how the policy chooses them (ReAct, and search over thoughts). The third blank is
what the history can _hold_. The context window is bounded, so a long trajectory
must offload to an external store, ground itself in a corpus too large to fit, and
sometimes split across several coordinated agents. This lesson fills that blank, then
names the failures that bound the whole design.

## Memory

The context window is the agent's working memory, and it is bounded. A trajectory
that runs for hundreds of steps, or that must recall facts from far earlier, exceeds
the window, so agents separate a small in-context **scratchpad** from a large
external store read through retrieval.

> **Definition (Short-term vs. long-term memory).** Short-term (working) memory is
> the content currently in the context window: the running history $h_t$, including
> the scratchpad of thoughts. Long-term memory is an external store
> $\mathcal{M} = \set{(k_i, v_i)}$ of key–value records outside the window, queried
> by a read operation $\operatorname{read}(\mathcal{M}, q)$ that returns the records
> most relevant to a query $q$ and splices them back into the context.

The dominant long-term store is a **vector store**: each memory $v_i$ is embedded to
a vector $k_i = \emb(v_i)$, and a read embeds the query and returns the
$k$ nearest neighbors by cosine similarity. The read is concretely
$$
\operatorname{read}(\mathcal{M}, q) = \operatorname*{top-}k_{i}\;
\frac{\langle \emb(q), k_i\rangle}{\lVert \emb(q)\rVert\,
\lVert k_i\rVert},
$$
the $k$ records whose embeddings point most nearly in the query's direction. Cosine
similarity, rather than raw inner product, normalizes away vector magnitude so that a
long memory is not favored merely for having a larger norm. The retrieved records are
then serialized and spliced into the prompt, at which point they become ordinary
context the policy reads on its next step.

Short-term memory is fast and exact but tiny; long-term memory is large and persistent
but lossy, since only what the read surfaces re-enters the context. The lossiness
forces a trade-off: raise $k$ and more of the store reaches the model, but the
retrieved text competes for the same bounded window as the working scratchpad, so the
practical setting balances recall against the token budget the reasoning itself needs.

| Property | Short-term (scratchpad) | Long-term (vector store) |
| --- | --- | --- |
| Location | inside the context window | external database |
| Capacity | bounded by window length | effectively unbounded |
| Access | direct (already in context) | retrieval by similarity |
| Persistence | lost when the window rolls over | survives across sessions |
| Cost | token budget | an embedding query per read |

## Retrieval-augmented generation

When the knowledge an agent needs lives in a corpus too large to fit any window, the
read is folded directly into generation. **Retrieval-augmented generation (RAG)**
pairs a **retriever** that selects passages with a **generator** that conditions on
them.[^lewis-rag] The retriever is typically a **dense passage retriever**: question
and passage are each embedded by an encoder, and relevance is their inner
product.[^karpukhin-dpr]

> **Definition (Dense retriever).** Encoders $E_Q, E_P$ map a query $x$ and a passage
> $z$ to vectors; the retrieval score is $s(x, z) = E_Q(x)^{T} E_P(z)$, and
> $$
> p_\eta(z \mid x) = \frac{\exp s(x, z)}{\sum_{z' \in \mathcal{Z}} \exp s(x, z')}
> $$
> is approximated by the top-$k$ passages under $s$, found by maximum-inner-product
> search over the precomputed passage index $\mathcal{Z}$.

The generator then treats the retrieved passage as a latent variable and
marginalizes the answer over the top-$k$ documents.

> **Definition (RAG marginalization).** With retrieval distribution $p_\eta(z \mid
> x)$ and generator $p_\theta(y \mid x, z)$, the answer probability marginalizes over
> retrieved passages,
> $$
> p(y \mid x) = \sum_{z \in \operatorname{top-}k(x)} p_\eta(z \mid x)\, p_\theta(y
> \mid x, z),
> $$
> the sum restricted to the top-$k$ passages so the marginal is a $k$-term mixture
> rather than a sum over the whole corpus.

For example, suppose the query embeds to
$q = E_Q(x)$ and three candidate passages score $s_1 = 8.2$, $s_2 = 7.9$, $s_3 = 4.0$
under the inner product (in practice thousands of passages are scored by
maximum-inner-product search; the top three survive). Softmax over the retained
scores gives retrieval weights
$$
w_z = \frac{e^{s_z}}{e^{8.2} + e^{7.9} + e^{4.0}}
\;\Rightarrow\; w_1 \approx 0.57,\; w_2 \approx 0.42,\; w_3 \approx 0.01 .
$$
Passage 3 is retrieved but almost ignored, because the exponential sharpens the gap
between a score of $4.0$ and the two near $8$. The generator conditions on each
passage in turn, and the answer distribution is the weighted mixture
$p(y \mid x) = \sum_z w_z\, p_\theta(y \mid x, z)$. Two design consequences follow.
Retrieval quality caps answer quality: if the correct evidence is not in the top-$k$,
the mixture cannot recover it. And the weights are soft, so a strong distractor near
the top can pull the answer even when a lower-ranked passage held the truth; in
practice, $k$, the encoder quality, and passage chunking are the settings that
matter most.

This grounds generation in retrieved evidence and updates the agent's knowledge by
editing the corpus, with no change to the model weights. It also separates two
kinds of error that a monolithic model conflates: a _retrieval_ failure (the evidence
was never surfaced) and a _generation_ failure (the evidence was present but the model
answered wrongly). Because the retrieved passages sit in the transcript, each can be
audited independently — a diagnostic advantage over asking a closed-book model why it
produced a fact.

```algorithm
caption: $\textsc{RAG}(x, \mathcal{Z}, k)$ — retrieve then generate over top-$k$ passages
$q \gets E_Q(x)$ // embed the query
$Z_k \gets$ top-$k$ passages in $\mathcal{Z}$ by $q^{T} E_P(z)$ // nearest neighbors
for each $z \in Z_k$ do
  $w_z \gets \softmax_z\!\parens{q^{T} E_P(z)}$ // retrieval weight
  $p_z \gets p_\theta(y \mid x, z)$ // condition generation on the passage
return $\sum_{z \in Z_k} w_z\, p_z$ // marginalize over retrieved passages
```

$$
% caption: The RAG pipeline. The retriever embeds the query, ranks the corpus, and
% feeds the top-$k$ passages to the generator, which marginalizes the answer over them.
\begin{tikzpicture}[>=stealth, font=\footnotesize,
  bx/.style={draw, minimum width=22mm, minimum height=10mm, align=center},
  acn/.style={draw=acc, text=acc, thick},
  grn/.style={draw=green, text=green, thick},
  doc/.style={draw, minimum width=14mm, minimum height=6mm, align=center, font=\scriptsize}]
  \definecolor{acc}{HTML}{2348F2}
  \definecolor{green}{HTML}{1F9D4D}
  \node[bx] (q) at (0,0) {\texttt{query} $x$};
  \node[bx, acn] (ret) at (3.2,0) {\texttt{retriever}};
  \node[doc] (d1) at (6.2,1.4) {\texttt{passage 1}};
  \node[doc] (d2) at (6.2,0.2) {\texttt{passage 2}};
  \node[doc] (d3) at (6.2,-1.0) {\texttt{passage k}};
  \node[bx, grn] (gen) at (9.6,0.2) {\texttt{generator}};
  \node[bx] (y) at (12.6,0.2) {\texttt{answer} $y$};
  \draw[->, thick] (q) -- (ret);
  \draw[->, acc, thick] (ret.east) -- (d1.west);
  \draw[->, acc, thick] (ret.east) -- (d2.west);
  \draw[->, acc, thick] (ret.east) -- (d3.west);
  \draw[->, green, thick] (d1.east) -- (gen.north west);
  \draw[->, green, thick] (d2.east) -- (gen.west);
  \draw[->, green, thick] (d3.east) -- (gen.south west);
  \draw[->, thick] (gen) -- (y);
  \node[black, anchor=south, font=\scriptsize] at (6.2,2.0) {top-$k$};
  \draw[->, black, thick] (q.south) .. controls (4.8,-2.2) and (7.6,-2.2) .. (gen.south);
  \node[black, anchor=north, font=\footnotesize] at (6.2,-2.1) {\texttt{query also passed to generator}};
\end{tikzpicture}
$$

## Reflection and self-improvement

An agent that can read its own transcript can also critique it. Two patterns turn a
failed or weak attempt into a better one without any weight update.

**Reflexion.** After an episode, the agent generates a verbal **self-reflection**
on what went wrong and writes it into memory, so the next attempt at the same task
is conditioned on that critique.[^shinn-reflexion] The feedback signal is natural
language, not a gradient.

> **Definition (Reflexion).** Given a trajectory $\tau$ and a (possibly sparse)
> outcome signal, a reflector model produces a textual critique
> $c = \reflect(\tau, \text{outcome})$, appended to a persistent
> memory $\mathcal{M} \gets \mathcal{M} \cup \set{c}$. The next attempt conditions on
> $\mathcal{M}$, so errors are corrected through language rather than parameter
> updates.

**Self-Refine.** Within a single task, the model drafts an answer, critiques its own
draft, and revises, iterating until the critique is satisfied or a budget is
spent.[^madaan-refine] The same network plays generator, critic, and reviser, so the
improvement loop needs no external reward model.

```algorithm
caption: $\textsc{SelfRefine}(x)$ — iterative draft, critique, revise on one task
$y \gets \pi_\theta(x)$ // initial draft
repeat
  $c \gets \pi_\theta(\text{critique} \mid x, y)$ // model critiques its own draft
  if $c$ signals "good enough" then
    return $y$
  $y \gets \pi_\theta(\text{revise} \mid x, y, c)$ // revise using the critique
until refinement budget exhausted
return $y$
```

## Multi-agent systems

A single policy can be split into several agents with distinct prompts, tools, and
roles, communicating through messages. Specialization lets each agent hold a focused
context and persona, and the composition itself becomes a design choice.

> **Definition (Multi-agent system).** A collection of policies
> $\set{\pi^{(1)}, \dots, \pi^{(n)}}$, each with its own role, tools, and context,
> that exchange messages on a shared channel. An orchestration protocol determines
> the order of turns and how each agent's output routes to the others' inputs.

Three protocols recur. In **debate**, agents argue opposing positions and a judge or
a final round resolves the answer, surfacing errors one agent alone would miss. In
**orchestrator–worker**, a planner agent decomposes a task and dispatches subtasks to
specialist workers, then composes their results. In **role-play simulation**,
persona-conditioned agents interact in an environment to produce emergent behavior,
as in generative-agent populations.[^park-agents]

$$
% caption: Orchestrator-worker. A planner decomposes the task, dispatches subtasks to
% specialist workers, and composes their returned results into a final answer.
\begin{tikzpicture}[>=stealth, font=\footnotesize,
  o/.style={draw=acc, thick, fill=acc!7, text=acc, minimum width=27mm, minimum height=13mm, align=center},
  w/.style={draw, thick, fill=black!4, minimum width=22mm, minimum height=9mm, align=center}]
  \definecolor{acc}{HTML}{2348F2}
  \node[o] (orch) at (0,0) {\texttt{orchestrator}\\\texttt{(planner)}};
  \node[w] (w1) at (5.7,2.2) {\texttt{worker: search}};
  \node[w] (w2) at (5.7,0.0) {\texttt{worker: code}};
  \node[w] (w3) at (5.7,-2.2) {\texttt{worker: write}};
  % a small accent badge in the orchestrator's corner
  \fill[acc] ([xshift=3.5pt,yshift=-3.5pt]orch.north west) rectangle ++(0.17,-0.17);
  % one clean straight two-way link per worker — subtask out, result back
  \draw[<->, acc, thick] (orch.east) -- (w1.west);
  \draw[<->, acc, thick] (orch.east) -- (w2.west);
  \draw[<->, acc, thick] (orch.east) -- (w3.west);
  \node[acc, font=\footnotesize, anchor=south] at (3.35,0.6) {\texttt{subtask} $\to$};
  \node[acc, font=\footnotesize, anchor=north] at (3.35,-0.6) {$\gets$ \texttt{result}};
  \node[black, anchor=north, font=\footnotesize] at (0,-1.6) {\texttt{composes results}};
\end{tikzpicture}
$$

## Failure modes and evaluation

Agents inherit every weakness of the underlying model and add new ones from the
loop. The dominant failures are concrete.

- **Hallucinated or invalid tool calls.** The model invents a tool that does not
  exist, calls a real one with malformed or out-of-schema arguments, or fabricates a
  plausible result instead of calling the tool at all. The executor catches schema
  violations; semantic errors (a syntactically valid but wrong call) are harder to
  detect.
- **Compounding errors over long horizons.** Each step conditions on every prior
  step, so an early mistake contaminates the rest of the trajectory. If each step is
  correct with probability $p$, a $T$-step task with no recovery succeeds with
  probability $p^{T}$, which decays fast: at $p = 0.95$ and $T = 20$, success is
  $0.95^{20} \approx 0.36$.
- **Context overflow.** A trajectory that runs long enough exceeds the window: old
  observations must be dropped, truncated, or summarized, and whatever is evicted is
  gone unless it was written to long-term memory. Overflow shows up as an agent that
  "forgets" a constraint stated hundreds of steps earlier, or that loops because the
  observation proving a subtask is done has scrolled out of view.
- **Non-terminating loops.** With no progress signal, an agent can repeat the same
  action forever — re-searching the same query, re-reading the same file — because each
  step looks locally reasonable. A step budget bounds the damage, but the cleaner fix
  is a loop detector that halts on a repeated (action, observation) pair.
- **Cost and latency.** Search over thoughts, retrieval, and multi-agent debate each
  multiply the number of model calls. A self-consistency vote over $m$ chains is
  $m\times$ the cost; a ToT search is far more, and every call adds latency.

The mitigations map onto the failures one for one, and the practical defaults follow
from the mechanisms above. Constrained decoding against the tool schema removes
unparseable and unknown-tool calls at the source. A step budget plus a repeated-state
detector bounds both loops and horizon cost. Summarizing or offloading old context to
a vector store keeps the window from overflowing. And because horizon error compounds
as $p^{T}$, the intervention that helps most is a recovery mechanism (verification,
reflection, or a retry) that lets a wrong step be caught and redone, breaking the
$p^{T}$ product.

> **Definition (Horizon-error compounding).** For a $T$-step trajectory with
> per-step success probability $p$ and no error recovery, the probability that all
> steps succeed is $p^{T}$, so the failure probability $1 - p^{T}$ grows toward $1$
> as the horizon $T$ increases. Recovery mechanisms (reflection, verification) exist
> precisely to break this product.

Evaluation uses task suites that exercise the full loop, not single-turn prompts.

| Benchmark | What it tests | Action space |
| --- | --- | --- |
| HotpotQA | multi-hop question answering | search / lookup tools |
| WebArena | task completion on realistic websites | browser actions (click, type, navigate) |
| SWE-bench | resolving real GitHub issues | edit files, run tests in a repo |
| GAIA | general assistants on multi-step real-world tasks | web, files, multimodal tools |

These benchmarks share a structure: a task with a verifiable end state, an
environment exposing real actions, and a trajectory budget. They measure exactly the
loop the agent runs, so progress on them tracks the property that matters,
end-to-end task completion rather than fluent text.[^zhou-webarena]


## What actually ships

The patterns above are research primitives; deployed systems assemble them under three constraints that explain why production agents look the way they do.

**Context engineering replaces prompt engineering.** With a bounded window and a store that competes for it, the binding decision is _what enters the context on this step_, not the wording of an instruction: which retrieved passages, how much of the scratchpad, which past-episode reflections, and in what order. Because attention degrades over very long contexts (the middle of a long prompt is attended weakly), packing the window is itself an optimization problem: put the task and the most relevant evidence where the model reads them best, and summarize or evict the rest. The RAG weights and the memory read $k$ from earlier are the two knobs that decide this trade-off directly.

**Structured protocols make tools portable.** Ad-hoc function schemas do not compose across systems, so the field converged on standard interfaces — a tool exposes a typed schema, and an agent discovers and calls it through a uniform protocol rather than bespoke glue. The payoff generalizes the executor trust boundary from part one: one validating layer mediates every tool, so permissions, rate limits, and error surfaces are enforced in one place instead of per-integration. An agent that implements a standard tool protocol can use any tool that implements it, without new code.

**Evaluation is the bottleneck.** Because a trajectory has many valid paths and a verifiable end state, the hard part of building an agent is measuring it: the benchmarks in the table above (SWE-bench, WebArena, GAIA) matter precisely because single-turn accuracy does not predict end-to-end task completion. The practical loop is to run the agent on a task suite with a programmatic checker, read the failed trajectories, and fix the scaffolding — the reflection, retrieval, and loop-detection mechanisms of this lesson — rather than the base model. The model supplies the policy; most of the reliability comes from the surrounding system.

An agent is a systems artifact as much as a model one. The language model is the policy, fixed and frozen; everything this two-part lesson added (the executor, the reasoning scaffold, the memory, the retrieval, the orchestration, the failure guards) is the engineering that turns a next-token predictor into a system that reliably completes work.

## Takeaways

- **Memory** splits into a bounded in-context **scratchpad** (short-term, fast, exact)
  and an external **vector store** read by cosine similarity (long-term, large,
  lossy); raising the read count $k$ trades recall against the token budget the
  reasoning itself needs.
- **RAG** marginalizes $p(y \mid x) = \sum_{z} p_\eta(z \mid x)\, p_\theta(y \mid x,
  z)$ over the top-$k$ passages from a dense retriever; the softmax over passage
  scores sharpens onto the top few, so retrieval quality caps answer quality and
  a strong distractor can pull the answer. It grounds generation in a corpus editable
  without retraining, and separates retrieval failures from generation failures.
- **Reflection** improves outputs through language, not gradients: **Reflexion**
  writes a verbal critique of a failed episode into memory for the next attempt;
  **Self-Refine** drafts, critiques, and revises within one task.
- **Multi-agent** systems specialize roles and compose them by debate,
  orchestrator–worker dispatch, or role-play simulation.
- The hard failures are **invalid tool calls**, **error compounding** over long
  horizons ($p^{T}$ decay: $0.95^{20} \approx 0.36$), **context overflow**, and
  **non-terminating loops**; the intervention that helps most is a recovery mechanism
  (verification, reflection, retry) that breaks the $p^T$ product. Benchmarks like
  **HotpotQA**, **WebArena**, **SWE-bench**, and **GAIA** score the full loop.
- **In production:** agents depend on **context engineering**,
  standardized **tool protocols** generalizing the executor trust boundary, and
  **evaluation** on task suites with programmatic checkers.

[^lewis-rag]: **Lewis et al.**, _Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks_, 2020 — couples a dense retriever with a generator and marginalizes the answer over the top-$k$ retrieved passages.
[^karpukhin-dpr]: **Karpukhin et al.**, _Dense Passage Retrieval for Open-Domain Question Answering_, 2020 — learns dual encoders so relevance is an inner product, retrieved by maximum-inner-product search over a passage index.
[^shinn-reflexion]: **Shinn et al.**, _Reflexion: Language Agents with Verbal Reinforcement Learning_, 2023 — the agent writes a verbal critique of a failed episode into memory and conditions the next attempt on it.
[^madaan-refine]: **Madaan et al.**, _Self-Refine: Iterative Refinement with Self-Feedback_, 2023 — one model drafts, critiques its own output, and revises in a loop with no external reward model.
[^park-agents]: **Park et al.**, _Generative Agents: Interactive Simulacra of Human Behavior_, 2023 — persona-conditioned agents with memory and reflection interact to produce emergent social behavior.
[^zhou-webarena]: **Zhou et al.**, _WebArena: A Realistic Web Environment for Building Autonomous Agents_, 2023 — benchmarks agents on multi-step tasks in realistic websites with verifiable end states.
[^chollet-systems]: **Chollet**, _Deep Learning with Python_, Ch. 11 — language models as components of larger systems: the model supplies the policy, while tools, retrieval, and control flow make it act on an environment.