agent-00 · The AI agent is a computer system around an LLM processor

If you treat an AI agent as a model with a chat box, every failure starts to look like a model failure.

The model forgot. The model hallucinated. The model was not smart enough. The model needs a larger context window.

Sometimes that is true. But it is often the wrong first diagnosis.

A better frame is: an AI agent is a computer system built around an LLM processor.

The LLM is the central reasoning engine, but the real capability comes from the whole architecture around it: context, memory, retrieval, tools, drivers, feedback, permissions, tests, and the harness that coordinates the loop.

This is the architecture map I took from six recent Hung-yi Lee lectures on agents.

agent capability = model x context x tools x feedback x harness x memory

The model matters. Of course it does. So does the rest of the machine.

Why this frame helps

Computer architecture gives agent builders a cleaner debugging language.

Nobody evaluates a computer by the CPU alone. A fast processor still needs working memory, storage, buses, device drivers, I/O ports, an operating system, interrupts, permissions, diagnostics, and recovery behavior.

The same is true for agents.

A strong model inside a weak system can look confused. A smaller model inside a disciplined system can look surprisingly competent. The difference is not magic. The surrounding architecture changes what the processor can see, what it can do, how errors return, and when the system is allowed to claim success.

That is why the useful question is not only "is the model smart enough?" It is also "which layer of the system is failing?"

Architecture map

Here is the compact map:

• LLM: CPU / reasoning processor.
• Context window: active working set.
• Long-term memory: storage layer.
• Retrieval: page loader / cache fill.
• Tools: peripherals, I/O devices, and accelerators.
• Tool schemas: drivers and syscalls.
• Feedback and tests: interrupts, diagnostics, and watchdogs.
• Harness: motherboard, OS, scheduler, permission layer, and recovery system.

The analogy is not perfect. An LLM does not randomly address memory the way a CPU does. A vector database is not literally a page table. A prompt is not literally RAM.

But as a builder's map, the analogy is useful because it forces you to inspect the interfaces around the model instead of staring only at the model weights.

The LLM is the processor, not the whole computer

The LLM is the processor because it is where the central reasoning step happens.

It receives serialized input tokens. It produces output tokens. Its weights encode language ability, coding patterns, world knowledge, reasoning habits, and learned behavior from pretraining and post-training.

That is a lot. It is also not enough.

The model does not automatically know which files exist. It does not automatically remember what happened yesterday. It does not automatically know whether a shell command succeeded. It does not automatically run the verifier before saying "done." It sees what the harness loads into the prompt, and it acts through whatever tools the system exposes.

So when the agent fails, "the model is weak" is only one possible explanation. The active context may be wrong. The tool may be vague. The schema may be too loose. The feedback may be missing. The harness may have allowed the model to stop before verification.

The model is central. It is not the whole computer.

Context is the loaded working set

The first lecture is nominally about context engineering. In this frame, it is working-set management.

An LLM does not reach into all of long-term memory on demand. It sees a prompt. Whatever matters must be serialized into that prompt-visible workspace.

That makes the context window closer to the loaded working set for a process than to total system memory. It contains current instructions, recent state, retrieved facts, tool outputs, user constraints, partial plans, and whatever scratch reasoning the system chooses to preserve.

The naive agent treats this working set as an append-only transcript:

C_next = C_current + new_input + model_output

That works for short conversations. It breaks for agents.

Tool outputs get huge. File reads flood the prompt. Old mistakes stay around. Important constraints sink below the visible fold. The context window becomes a landfill instead of a working set.

The better model is:

C_next = F(C_current, new_input, model_output)

The interesting part is F. It decides what stays hot, what gets evicted, what gets summarized, what gets externalized, what gets retrieved, and what gets delegated to a sub-agent.

That is why context engineering is not prompt decoration. It is memory management for an LLM-shaped processor.

Memory and retrieval are storage plus loading policy

Prompt-visible context is not the same thing as memory.

Memory is the larger storage layer outside the prompt: files, notes, logs, embeddings, databases, traces, previous decisions, user preferences, and project conventions.

C = P + M

P is the prompt-visible working set. M is external memory.

Weak agent systems try to stuff too much of M into P. Stronger systems keep memory external, keep pointers, search deliberately, and load only the relevant pages when needed.

Retrieval is the page-loading layer. It finds the right memory chunk and brings it into the active working set. A vector index is not literally an address translation unit, but it plays a similar architectural role: it lets the system find relevant stored state without asking the model to carry everything at once.

This is also where sub-agents become interesting. They are not only parallel workers. They can be context-compression devices. The parent sends a messy branch into another process and receives a compact return: a claim, evidence, a file path, a command, a timestamp, or a reproduction handle.

Tools are peripherals, accelerators, and interfaces

Tools are not just output devices. They are the I/O system around the LLM processor.

A browser tool is a network interface plus a web client. A calculator is a math coprocessor. A shell is an execution environment. A database tool is a storage controller and query engine. Image and audio tools are media accelerators. Calendar, email, cloud, robotics, and sensor APIs are external peripherals.

The LLM can reason about an action. Tools let the system touch the world.

Tool schemas are the driver layer. They translate model intent into valid operations: argument names, types, constraints, return shapes, permissions, failure modes, and safe boundaries.

Bad drivers make the device hard to use. A vague tool schema forces the model to guess. A tool that returns a wall of unstructured text turns every observation into another context-management problem.

Agent-first tools should expose structured inputs, bounded outputs, concise errors, stable IDs, line numbers, raw-output handles, and deterministic verification commands. Human interfaces hide state behind visual affordances. Agent interfaces should make state machine-readable.

This is why the physical motherboard analogy works for the cover image. RAM, storage, ports, slots, accelerators, and diagnostic LEDs are not decoration. They are the visible parts of the agent's capability surface.

Feedback and tests are diagnostics

The self-correction lecture lands better if you treat feedback as a signal path.

An agent that only generates and reflects is like a machine with no diagnostics. It can sound thoughtful while drifting away from reality.

A useful system needs interrupts and test signals: compiler errors, failing tests, lint output, browser screenshots, user corrections, evaluator scores, reward signals, timeouts, permission denials, and watchdog checks.

The goal is not just "ask the model to check itself." The goal is to feed the processor a grounded signal from the environment.

That is why programming is such a strong agent domain. The environment can say exactly what failed. The agent can use that failure as the next input.

Tests beat vibes because tests create an interrupt the system cannot honestly ignore.

The harness is the system layer

The harness lecture is the most practical part for builders.

A small model fails a bug-fixing task because it hallucinates file content and claims success. Then the setup changes. The model gets a few boring rules: list the directory first, read the file before editing, and run the verification script before claiming the task is done.

Same model. Same task. Better harness. The model succeeds.

The model did not become smarter. The system around it became less careless.

That is why the harness is bigger than a bus. A bus moves data between components. The harness decides what can happen.

It chooses what context gets loaded, what tools exist, what permissions apply, how state is stored, how failures are surfaced, whether tests must pass, when to retry, when to stop, and how successful procedures become reusable skills.

In this analogy, the harness is motherboard plus operating system plus scheduler plus permission layer plus recovery system.

It is the layer that turns a powerful text processor into a delegatable worker.

Multi-agent systems are network topologies

"Multi-agent" is not a feature. It is a topology choice.

Chain, tree, star, mesh, debate, blackboard, manager-worker, and role-based search all behave differently. This is distributed-systems logic entering agent design.

More processors do not automatically make the computer better. More communication does not automatically improve the result. The topology changes the algorithm.

A chain is simple, but errors propagate. A mesh creates more chances for critique, but it costs more and creates more noise. A tree can branch into variations, but the direction and merge policy matter.

For builders, the useful question is not simply "one agent or many?" The useful question is: what kind of network are you building, what state crosses each boundary, and what summary is allowed to return?

Self-correction is a control loop

Self-correction is not magic introspection. It is a control-loop problem.

The system tries something, observes a signal, updates state, retries, or escalates.

That can happen at several layers. Decoding changes how tokens are selected. Workflow generates, verifies, revises, and tries again. Training can teach reasoning behavior that checks intermediate work before final output.

Self-improvement is the same idea at a larger scale. Can the system improve its own data, objectives, rewards, model, tools, memory, or harness?

We are not clearly across that river. AI can generate pseudo-labels, help write rewards, judge outputs, create tasks, train weaker models, and improve harness rules. But every step has traps: proxy reward drift, self-training plateaus, benchmark overfitting, evaluator hacking, and fake progress.

The near-term self-improvement path I trust most is harness improvement: add a verifier, tighten a tool, write a reusable skill, clean memory, or improve the workflow. That version is less dramatic than rewriting weights, but it is much easier to inspect.

A builder checklist

The practical takeaway is simple: before blaming the processor, inspect the computer.

• Was the right working set loaded?
• Was long-term memory retrieved cleanly?
• Did the tools expose explicit state?
• Did the drivers constrain valid actions?
• Did feedback arrive as a real diagnostic signal?
• Did tests or watchdogs prevent fake completion?
• Did permissions match the task?
• Did the harness know when to retry, stop, or escalate?

This is why agent engineering feels messy. There is no single knob. We are building computer systems around LLM processors, and every layer changes what the processor appears capable of.

The strongest agent builders I know do not only ask whether the model is smart enough. They ask which layer of the system is failing.

Companion deep dives

This post is the architecture map. The individual posts go deeper into the lecture topics behind each layer:

1. agent-01 · Context engineering is the agent's working memory
2. agent-02 · Multi-agent systems are topology problems
3. agent-03 · What AI agents change about research work
4. agent-04 · Harness engineering is how we actually make agents useful
5. agent-05 · Self-correction has three layers: decoding, workflow, and reasoning
6. agent-06 · Self-improving AI is a spectrum, not a switch

Sources and references

Source lectures behind this architecture map:

• AI Agent (1/3)：核心技術 Context Engineering 基本概念解說
• AI Agent (2/3): AI Agent 之間可以有什麼樣的互動
• AI Agent (3/3): AI Agent 對於工作帶來的衝擊 - 以學術研究為例
• Harness Engineering：有時候語言模型不是不夠聰明，只是沒有人類好好引導
• AI 能自我修正嗎？從 decoding、workflow 到 reasoning 的技術發展整理
• AI 要跨越盧比孔河了嗎？自我成長的 AI 離我們多遠？