The Anatomy of an Agent That Actually Works

Day 2 of 7 — The Agentic Engineer’s Field Notes

Yesterday I argued that the system is the artifact, not the prompt. Today I want to open the system up and show you what’s inside.

There is a diagram I draw on whiteboards more often than any other. It has four boxes and one arrow that loops back on itself. If your team can name what goes in each box and explain why the arrow loops, you have an agent. If they cannot, you have a chatbot pretending to be one.

Here it is, in plain text:

 ┌──────────┐ │ MEMORY │ └────┬─────┘ │ ▼ ┌───────────────────────┐ │ MODEL │◄────────┐ └───────────┬───────────┘ │ │ chooses │ ▼ │ ┌─────────┐ │ │ TOOLS │ │ └────┬────┘ │ │ produces │ ▼ │ ┌─────────┐ │ │ RESULT │────────────────┘ └─────────┘ the loop

Four moving parts. One loop. That is the entire substrate of agentic computing, and almost everything interesting in this field is about getting one of these four parts right.

Let’s take them one at a time.

1. The model

The model is the brain. It is also the part you have the least control over.

You did not train it. You cannot easily fine-tune it. It will be replaced under you, sometimes silently, sometimes with weeks of notice. Its behavior on your specific task is an empirical question, not a design choice.

This sounds like bad news. It is actually the most freeing thing about agentic engineering: the model is a capability, not an implementation. You don’t optimize the model itself. You optimize how you compose around it.

The practical consequence: choose the smallest, fastest, cheapest model that gets the job done, and design the system so that you can swap models without rewriting it. The teams that bet hard on a specific model’s quirks (“Claude does this best because of X”) look brilliant for six months and stranded for the next six. The teams that treat the model as an interchangeable component compound their improvements every time the frontier moves.

A useful heuristic that has hardened into a production pattern by 2026: route by difficulty. The dominant Claude Agent SDK pattern in production today uses Opus as the orchestrator and Haiku inside sub-agents. Cognition’s most recent writeup makes the same point with cross-frontier routing — running Claude and GPT together as “a capability router rather than a difficulty escalator,” each handling subtasks where it actually excels. Use a small model for routing, classification, and summarization. Reserve the frontier model for the actual reasoning. You will pay a fraction of the cost and most of your users will not notice.

2. The tools

The tools are the hands. They are also where most agents quietly fail.

A tool is a function the model can call. The model picks the tool, picks the arguments, sees the result, and decides what to do next. From the model’s point of view, your codebase is a vocabulary — every tool is a word it can speak. If your vocabulary is bad, no amount of model intelligence will save you.

I’m going to spend an entire day on tool design later this week, so I’ll keep it short here. The key insight is one Anthropic published explicitly in their tool-design guidance for engineers: more tools don’t always lead to better outcomes. Build a few thoughtful tools targeting specific high-impact workflows, not a thin wrapper around every API endpoint you happen to have.

A human-facing API and a tool designed for an LLM are not the same thing. Hold that thought until Day 4.

3. The memory

The memory is what survives across turns.

This is more subtle than it sounds. There are at least three kinds of memory in any non-trivial agent, and confusing them is the root of an entire class of bugs.

Working memory — the conversation so far. Everything the model can see in its current context window. This is bounded, and managing it is most of the game.
Episodic memory — past conversations and runs. “Last Tuesday the user asked us to summarize a contract. Here is what we did.” Stored externally; pulled in on demand.
Semantic memory — durable facts about the world or the user. “This user prefers SQL examples in PostgreSQL syntax.” Stored externally; sometimes pinned, sometimes retrieved.

Most “memory features” you’ve seen in chatbots conflate these. They store everything in a vector database, retrieve a top-k slice, and call it memory. That works for trivial cases and breaks the moment the user’s “memory” includes contradictions, retractions, or evolving preferences. We’ll get to the fix on Day 5, including the Letta team’s hierarchical memory architecture that has become the de facto reference design.

For now, the operating principle: memory is not a bag of facts. It is a managed surface that the agent reasons over. The management — what to write, what to retrieve, what to expire — is engineering, not magic.

4. The loop

The loop is what makes it an agent and not a function call.

In its simplest form, the loop is a while:

context = system_prompt + user_request
while not done:
action = model(context)
if action == "stop":
return action.final_message
result = execute_tool(action.tool, action.args)
context = update(context, action, result)

That’s it. That is the entire control flow of an agent. Everything else is decoration.

The loop is also where almost every interesting failure mode lives:

Infinite loops — the model keeps calling the same tool because it doesn’t realize the result is the same.
Off-track loops — the model wanders, retrying, exploring, never converging on the goal.
Dead-end loops — the model gives up too early because the context contains a misleading error.
Cost blowouts — the loop runs 47 iterations on a task that should have taken three.
Context exhaustion — the loop runs long enough that the original instruction has rolled out of the model’s attention window.

Every one of these is an engineering problem, not a model problem. You add max-iteration caps. You add cycle detection. You add cost budgets (the Claude Agent SDK now ships with a max_tool_calls budget defaulting to 12, for good reason). You add periodic context summarization. You add escape hatches that hand control back to a human. The loop is the place where agentic engineering becomes most like classical systems engineering — and most teams do not give it the attention it deserves because it does not feel like AI work.

Five patterns to know by name

When Anthropic published Building Effective Agents, they did the field a quiet favor by codifying five composable patterns. They are not new individually. What’s useful is that they are now named, with shared vocabulary across teams. If you are sketching an agent on a whiteboard, you should be able to point at any region of it and say “this part is evaluator-optimizer” or “this is parallelization” — and have the room nod.

The five:

Prompt chaining — sequential decomposition. Each step’s output feeds the next. Use when the task has obvious sub-steps and you want to trade latency for accuracy. (Generate marketing copy → translate it.)
Routing — classify the input, dispatch to a specialist. Use when categories of input genuinely need different handling, or when you’re routing for cost (small model triages, big model only when the triage flags hard).
Parallelization — run multiple model calls at once, either on independent sections or as a vote on the same question. Use for speed gains or for confidence-via-redundancy.
Orchestrator-workers — a central agent breaks the problem into sub-tasks dynamically and delegates. Use when the sub-task structure depends on the input and can’t be hard-coded. (This is the territory of multi-agent debates we’ll cover on Day 7.)
Evaluator-optimizer — one model proposes; another critiques; loop until acceptable. Use when there’s a clear quality criterion and iterative refinement measurably helps. (The Anthropic engineering team has explicitly used this for translation quality.)

Most production agents are some composition of two or three of these patterns. The error most teams make is reaching for the most complicated pattern they’ve heard of, when the simpler one would have shipped in a week.

The principle Anthropic states explicitly is the one to internalize: start simple. Add complexity only when simpler solutions demonstrably underperform.

A note on frameworks

You are going to be told, repeatedly, that you need a framework. LangChain, LangGraph, AutoGen, CrewAI, the Claude Agent SDK, the OpenAI Agents SDK, Letta, and a dozen others are all competing for your import statements.

Frameworks are not free. Each one introduces its own abstractions, its own coupling, its own state machine you have to understand to debug. For simple agents, a 50-line while loop around your model SDK is shorter, clearer, and easier to debug than the equivalent in any framework. For complex agents — multi-step, branching, retry-aware, observability-rich — a good framework will save you months. The trick is knowing where the cutover is for your team.

My rule of thumb: if you can describe your agent’s control flow in one paragraph of plain English, write it in raw SDK calls. If you cannot — if it has parallel sub-tasks, conditional branches, retries with backoff, persistent state across runs — pick a framework and commit. Don’t half-adopt one. The worst codebase I’ve debugged was a “lightweight wrapper” some team had built around three different agent frameworks at once.

The other rule: your framework should not be the most interesting part of your agent. If your team meetings are mostly about which framework to use, you are not building an agent. You are building an opinion.

The simplest agent worth shipping

Here is what a minimum-viable agent looks like in production. Not in a tutorial — in production.

A model, called via an SDK, with a system prompt that defines the role and the boundaries.
A small set of tools (often three to seven) that are designed for the model, instrumented with logging, and tested for the messy cases.
A memory layer that includes at least: a scratchpad for the current task, a way to recall recent runs, and durable preferences for the user.
A loop with a hard iteration cap, a cost cap, a wallclock cap, and an escape hatch that returns control to a human or to a deterministic fallback.
Observability hooks on every model call and every tool call, so that when something goes wrong (and it will), you can replay the trajectory.
One of the five Anthropic patterns as your top-level shape — usually orchestrator-workers or evaluator-optimizer for non-trivial tasks.

That’s the whole thing. You can build the first version in a week. The next six months are about making each of the four subsystems better — and that is what this series is about.

Tomorrow: the part that’s quietly eating prompt engineering

Look at the loop diagram again. Notice the arrow back into the model. Every time it fires, the context gets bigger. The model sees more, but its attention is finite. What it pays attention to determines what it does next.

The discipline of deciding what gets into the model’s context, in what order, with what compression — that is the most important emerging skill in this entire field. It has a name now. Karpathy gave it the canonical formulation last year: “the delicate art and science of filling the context window with just the right information for the next step.” It has a four-strategy framework that’s quietly become the reference model. And the teams that have internalized it are quietly running away from everyone else.

Tomorrow we go deep on context engineering. See you then.

Day 2 of 7. Yesterday: the mental shift. Tomorrow: context engineering — and why prompt engineering is dying as a job description. Follow along.

Sources & further reading: