Agent Memory, Retrieval, and Orchestration

This builds on AI Agents: Tools and Reasoning, which formalized the agent as a policy over interaction histories and filled in two of its three blanks: the actions (tools, run behind an executor trust boundary) and how the policy chooses them (ReAct, and search over thoughts). The third blank is what the history can hold. The context window is bounded, so a long trajectory must offload to an external store, ground itself in a corpus too large to fit, and sometimes split across several coordinated agents. This lesson fills that blank, then names the failures that bound the whole design.

Memory

The context window is the agent's working memory, and it is bounded. A trajectory that runs for hundreds of steps, or that must recall facts from far earlier, exceeds the window, so agents separate a small in-context scratchpad from a large external store read through retrieval.

The dominant long-term store is a vector store: each memory $v_i$ is embedded to a vector $k_i=\mathrm{emb}(v_i)$, and a read embeds the query and returns the $k$ nearest neighbors by cosine similarity. The read is concretely

the $k$ records whose embeddings point most nearly in the query's direction. Cosine similarity, rather than raw inner product, normalizes away vector magnitude so that a long memory is not favored merely for having a larger norm. The retrieved records are then serialized and spliced into the prompt, at which point they become ordinary context the policy reads on its next step.

Short-term memory is fast and exact but tiny; long-term memory is large and persistent but lossy, since only what the read surfaces re-enters the context. The lossiness forces a trade-off: raise $k$ and more of the store reaches the model, but the retrieved text competes for the same bounded window as the working scratchpad, so the practical setting balances recall against the token budget the reasoning itself needs.

Retrieval-augmented generation

When the knowledge an agent needs lives in a corpus too large to fit any window, the read is folded directly into generation. Retrieval-augmented generation (RAG) pairs a retriever that selects passages with a generator that conditions on them.¹ The retriever is typically a dense passage retriever: question and passage are each embedded by an encoder, and relevance is their inner product.²

The generator then treats the retrieved passage as a latent variable and marginalizes the answer over the top-$k$ documents.

For example, suppose the query embeds to $q=E_Q(x)$ and three candidate passages score $s_1=8.2$, $s_2=7.9$, $s_3=4.0$ under the inner product (in practice thousands of passages are scored by maximum-inner-product search; the top three survive). Softmax over the retained scores gives retrieval weights

Passage 3 is retrieved but almost ignored, because the exponential sharpens the gap between a score of $4.0$ and the two near $8$. The generator conditions on each passage in turn, and the answer distribution is the weighted mixture $p(y\mid x)={\sum }_z{w}_z{p}_{\theta }(y\mid x,z)$. Two design consequences follow. Retrieval quality caps answer quality: if the correct evidence is not in the top-$k$, the mixture cannot recover it. And the weights are soft, so a strong distractor near the top can pull the answer even when a lower-ranked passage held the truth; in practice, $k$, the encoder quality, and passage chunking are the settings that matter most.

This grounds generation in retrieved evidence and updates the agent's knowledge by editing the corpus, with no change to the model weights. It also separates two kinds of error that a monolithic model conflates: a retrieval failure (the evidence was never surfaced) and a generation failure (the evidence was present but the model answered wrongly). Because the retrieved passages sit in the transcript, each can be audited independently — a diagnostic advantage over asking a closed-book model why it produced a fact.

Reflection and self-improvement

An agent that can read its own transcript can also critique it. Two patterns turn a failed or weak attempt into a better one without any weight update.

Reflexion. After an episode, the agent generates a verbal self-reflection on what went wrong and writes it into memory, so the next attempt at the same task is conditioned on that critique.³ The feedback signal is natural language, not a gradient.

Self-Refine. Within a single task, the model drafts an answer, critiques its own draft, and revises, iterating until the critique is satisfied or a budget is spent.⁴ The same network plays generator, critic, and reviser, so the improvement loop needs no external reward model.

Multi-agent systems

A single policy can be split into several agents with distinct prompts, tools, and roles, communicating through messages. Specialization lets each agent hold a focused context and persona, and the composition itself becomes a design choice.

Three protocols recur. In debate, agents argue opposing positions and a judge or a final round resolves the answer, surfacing errors one agent alone would miss. In orchestrator–worker, a planner agent decomposes a task and dispatches subtasks to specialist workers, then composes their results. In role-play simulation, persona-conditioned agents interact in an environment to produce emergent behavior, as in generative-agent populations.⁵

Failure modes and evaluation

Agents inherit every weakness of the underlying model and add new ones from the loop. The dominant failures are concrete.

• Hallucinated or invalid tool calls. The model invents a tool that does not exist, calls a real one with malformed or out-of-schema arguments, or fabricates a plausible result instead of calling the tool at all. The executor catches schema violations; semantic errors (a syntactically valid but wrong call) are harder to detect.
• Compounding errors over long horizons. Each step conditions on every prior step, so an early mistake contaminates the rest of the trajectory. If each step is correct with probability $p$, a $T$-step task with no recovery succeeds with probability $p^T$, which decays fast: at $p=0.95$ and $T=20$, success is $0.95^{20}\approx 0.36$.
• Context overflow. A trajectory that runs long enough exceeds the window: old observations must be dropped, truncated, or summarized, and whatever is evicted is gone unless it was written to long-term memory. Overflow shows up as an agent that forgets a constraint stated hundreds of steps earlier, or that loops because the observation proving a subtask is done has scrolled out of view.
• Non-terminating loops. With no progress signal, an agent can repeat the same action forever — re-searching the same query, re-reading the same file — because each step looks locally reasonable. A step budget bounds the damage, but the cleaner fix is a loop detector that halts on a repeated (action, observation) pair.
• Cost and latency. Search over thoughts, retrieval, and multi-agent debate each multiply the number of model calls. A self-consistency vote over $m$ chains is $m\times$ the cost; a ToT search is far more, and every call adds latency.

The mitigations map onto the failures one for one, and the practical defaults follow from the mechanisms above. Constrained decoding against the tool schema removes unparseable and unknown-tool calls at the source. A step budget plus a repeated-state detector bounds both loops and horizon cost. Summarizing or offloading old context to a vector store keeps the window from overflowing. And because horizon error compounds as $p^T$, the intervention that helps most is a recovery mechanism (verification, reflection, or a retry) that lets a wrong step be caught and redone, breaking the $p^T$ product.

Evaluation uses task suites that exercise the full loop, not single-turn prompts.

These benchmarks share a structure: a task with a verifiable end state, an environment exposing real actions, and a trajectory budget. They measure exactly the loop the agent runs, so progress on them tracks the property that matters, end-to-end task completion rather than fluent text.⁶

What actually ships

The patterns above are research primitives; deployed systems assemble them under three constraints that explain why production agents look the way they do.

Context engineering replaces prompt engineering. With a bounded window and a store that competes for it, the binding decision is what enters the context on this step, not the wording of an instruction: which retrieved passages, how much of the scratchpad, which past-episode reflections, and in what order. Because attention degrades over very long contexts (the middle of a long prompt is attended weakly), packing the window is itself an optimization problem: put the task and the most relevant evidence where the model reads them best, and summarize or evict the rest. The RAG weights and the memory read $k$ from earlier are the two knobs that decide this trade-off directly.

Structured protocols make tools portable. Ad-hoc function schemas do not compose across systems, so the field converged on standard interfaces — a tool exposes a typed schema, and an agent discovers and calls it through a uniform protocol rather than bespoke glue. The payoff generalizes the executor trust boundary from part one: one validating layer mediates every tool, so permissions, rate limits, and error surfaces are enforced in one place instead of per-integration. An agent that implements a standard tool protocol can use any tool that implements it, without new code.

Evaluation is the bottleneck. Because a trajectory has many valid paths and a verifiable end state, the hard part of building an agent is measuring it: the benchmarks in the table above (SWE-bench, WebArena, GAIA) matter precisely because single-turn accuracy does not predict end-to-end task completion. The practical loop is to run the agent on a task suite with a programmatic checker, read the failed trajectories, and fix the scaffolding — the reflection, retrieval, and loop-detection mechanisms of this lesson — rather than the base model. The model supplies the policy; most of the reliability comes from the surrounding system.

An agent is a systems artifact as much as a model one. The language model is the policy, fixed and frozen; everything this two-part lesson added (the executor, the reasoning scaffold, the memory, the retrieval, the orchestration, the failure guards) is the engineering that turns a next-token predictor into a system that reliably completes work.

Takeaways

• Memory splits into a bounded in-context scratchpad (short-term, fast, exact) and an external vector store read by cosine similarity (long-term, large, lossy); raising the read count $k$ trades recall against the token budget the reasoning itself needs.
• RAG marginalizes $p(y\mid x)={\sum }_z{p}_{\eta }(z\mid x)p_{\theta }(y\mid x,z)$ over the top-$k$ passages from a dense retriever; the softmax over passage scores sharpens onto the top few, so retrieval quality caps answer quality and a strong distractor can pull the answer. It grounds generation in a corpus editable without retraining, and separates retrieval failures from generation failures.
• Reflection improves outputs through language, not gradients: Reflexion writes a verbal critique of a failed episode into memory for the next attempt; Self-Refine drafts, critiques, and revises within one task.
• Multi-agent systems specialize roles and compose them by debate, orchestrator–worker dispatch, or role-play simulation.
• The hard failures are invalid tool calls, error compounding over long horizons ($p^T$ decay: $0.95^{20}\approx 0.36$), context overflow, and non-terminating loops; the intervention that helps most is a recovery mechanism (verification, reflection, retry) that breaks the $p^T$ product. Benchmarks like HotpotQA, WebArena, SWE-bench, and GAIA score the full loop.
• In production: agents depend on context engineering, standardized tool protocols generalizing the executor trust boundary, and evaluation on task suites with programmatic checkers.