agent-01 · Context engineering is the agent's working memory

The first lecture in Hung-yi Lee's AI agent series is about context engineering. That sounds like a narrow infrastructure topic. It is not. It is one of the cleanest explanations I have seen of why agents are hard to make reliable.

A language model does not wake up in the morning remembering what happened yesterday. It receives a prompt and predicts what comes next. If you want it to act like a long-running worker, you need a system around it that chooses what goes into the next prompt.

That system is the agent's working memory.

Builder takeaway: when an agent fails, inspect the context stack before blaming the model. Did it see the right prompt-visible workspace? Could it retrieve the right memory? Did tool output flood the prompt? Did compression drop the deciding detail?

The practical context stack

The useful way to read this lecture is as a taxonomy, not a single trick. Context engineering is the layer that decides what the model can see, what remains outside the prompt, what gets compressed, and what evidence survives long enough to be checked.

For agent builders, I would split the stack like this:

• Prompt-visible workspace: the small active set the model sees right now.
• External memory: files, logs, notes, vector stores, playbooks, and other state outside the prompt.
• Compression: summaries, markers, and pointers that keep long histories usable without pretending nothing was lost.
• Bounded observations: tool results shaped before they enter the model context.
• Sub-agent interfaces: messy branches that return concise conclusions plus verifiable handles.
• On-demand skills: tool and procedure descriptions loaded when relevant, not carried in every step.

That taxonomy is also a debugging checklist. If a coding agent loops, hallucinates, misses a file, or confidently says "done" without evidence, the failure may be in one of these context interfaces rather than in the base model alone.

The naive agent just appends everything

The simplest agent loop is easy to imagine. A user says something. The model replies. The tool returns an observation. The agent appends all of it to the next prompt.

In rough form:

C_next = C_current + input + output

This feels honest. Nothing gets lost. The model can see the whole history.

It also breaks quickly.

Tool outputs are long. File reads are long. Logs are long. Web pages are long. A few cycles later the prompt is full of stale details, failed attempts, giant observations, and random fragments that are only accidentally related to the current decision.

The model is not only limited by context length. It is limited by attention. Even if the context window is large, burying the one relevant fact inside pages of irrelevant text makes the job harder.

The lecture reframes the problem as a function:

C_next = F(C_current, input, output)

The core engineering problem is no longer "how do I keep all the text?" It is "what should the update function F do?"

That small shift matters.

Prompt is not context

One of the most useful distinctions in the lecture is prompt versus context.

The prompt is what the model sees right now.

Context is broader. It includes what the agent has experienced, what it can retrieve, what is stored on disk, what skills exist, what logs were written, and what memories may become relevant later.

A helpful notation is:

context = P + M

P is prompt-visible context. M is external memory.

Most bad agent designs confuse the two. They try to make P contain everything. That is how you get giant prompts full of tool manuals, previous mistakes, raw logs, and old observations.

The better design treats the prompt as a small active workspace. External memory can be much larger, but it should stay outside until the agent has a reason to read it.

This is very close to how I want coding agents to behave. Do not paste the whole repository into the model. Give it a map, search tools, file readers, and clear rules for when to inspect what.

Compression is necessary, but dangerous

Once history gets long, compression is unavoidable. You can summarize old turns, mask observations, or store long tool outputs as files and leave behind a pointer.

The lecture discusses a few patterns:

• summarize old history with another LLM
• replace a long observation with a short marker
• save raw logs to a file like log1.txt
• keep only the parts needed for the current task
• ask a sub-agent to handle a branch and return a short result

The danger is context collapse.

A summary can remove exactly the fact that later becomes necessary. The agent might have had enough information to solve the task before compression, then fail after compression because the important bit was treated as noise.

This is why summarization should not be thought of as a harmless cleanup step. It is an information bottleneck. You need to know what the task cares about.

The ACON idea mentioned in the lecture fits here: when compression causes failure, use feedback to teach the summarizer what not to drop. I like this because it treats context engineering as a learning problem, not just a prompt-formatting problem.

Observation is the real context killer

A subtle point in the lecture: the model's own reasoning and actions are not always the biggest source of context growth. Observations are.

A shell command returns 500 lines. A file read dumps a whole module. A browser scrape returns a page. A test run prints a giant stack trace. Now the model has to decide inside a swamp of text.

That suggests a different place to intervene. Instead of summarizing everything after the fact, prevent bad observations from entering the prompt in the first place.

A smarter read tool should not always mean "read the entire file." It could mean:

• read a specific line range
• search for related symbols first
• return a summary plus line references
• hide irrelevant output unless requested
• preserve raw output externally for later inspection

This is where tool design and context engineering blur together. A tool that returns better observations is also a context engineering tool.

Sub-agents are compression devices

The lecture's framing of sub-agents is useful because it cuts through some of the hype.

A sub-agent is not only a miniature coworker. It is a way to keep the main context clean.

The main agent can say: go inspect these files, compare these papers, or test this hypothesis. The sub-agent accumulates all the messy intermediate context. When it returns, the main agent receives a short answer: what was found, what failed, what matters.

That is compression with agency.

It is also risky. If the sub-agent returns a bad summary, the main agent may never see the missing evidence. So the interface matters. A good sub-agent should return not just a conclusion, but also anchors: files, line numbers, commands, timestamps, or other handles the parent can verify.

This is the same reason I do not trust "done" from an agent unless it gives me a path, diff, test result, or reproduction.

Tool descriptions should not all live in the system prompt

Another practical point: tool descriptions are themselves context.

If you expose a large tool library to a model by dumping every tool description into the prompt, the tool manual becomes a tax on every single step. It also makes tool selection harder.

The lecture mentions approaches like MCP-Zero and on-demand skill loading. The idea is straightforward: keep tool and skill descriptions searchable, then load only what is relevant.

This is basically retrieval-augmented generation, but for agent affordances rather than documents.

I think this matters a lot for long-lived personal agents. A useful personal agent may have hundreds of tiny skills: email, calendar, code review, video editing, note taking, finance, deployment, writing. It cannot keep every instruction active all the time. It needs a way to remember that a capability exists without carrying the full manual in its head.

Agentic context engineering

The final step is to let the model help manage its own context.

The lecture calls this agentic context engineering. Instead of humans hard-coding all of F, the agent can maintain a cheatsheet, update a playbook, decide what to store, or search external memory by itself.

Examples include:

• dynamic cheatsheets
• playbooks that get updated over time
• recursive language model style systems that store large context externally and keep only metadata visible

This is powerful and uncomfortable for the same reason. If the agent can edit its own working notes, it can improve. It can also write bad rules, preserve wrong lessons, or delete the one constraint that mattered.

So the system boundary matters. I would not let an agent casually rewrite its root identity or safety rules. But letting it maintain task-level notes, project conventions, and reusable tactics seems not only reasonable but necessary.

What I would actually use

If I were designing an agent after this lecture, I would turn the taxonomy into a context checklist:

1. Keep the active prompt small: make the model work in a clean workspace, not a transcript dump.
2. Store raw outputs externally: preserve logs, diffs, traces, and long reads outside the prompt.
3. Put pointers in the prompt: use file paths, line numbers, commands, timestamps, and source URLs instead of giant blobs.
4. Bound every observation: make tools return structured summaries, relevant ranges, and handles for deeper inspection.
5. Use sub-agents for messy branches: isolate exploratory context, then return only the claim, evidence, and reproduction path.
6. Require verifiable handoffs: do not accept "done" without a path, diff, test result, or other anchor.
7. Load tools and skills on demand: keep tool manuals searchable instead of making every step pay the full context tax.
8. Treat summaries as lossy: test whether compression preserves the facts future steps need.
9. Let the agent maintain a playbook: allow task-level learning while keeping root rules and safety boundaries protected.

None of this is glamorous. That is probably why it is important.

A lot of agent failures look like reasoning failures from the outside. Sometimes they are. But after this lecture, I would first ask a more basic question: did the agent have the right working memory?

Concept inventory

For reference, here are the main concepts from the lecture in one place:

• context engineering as the update function from current context, input, and output to the next context
• context window limits and why long tasks need compression
• prompt-visible context versus external memory
• P + M: prompt contents plus memory outside the prompt
• compaction, summarization, and observation masking
• memory externalization: save long outputs as files and keep only pointers active
• context collapse: losing the one detail the task later needs
• ACON-style feedback for better compression
• AgentFold and fold-style tools for turning long histories into short notes
• sub-agents as both parallel workers and context compression devices
• observation filtering before tool output enters the model context
• on-demand tool and skill loading instead of dumping every manual into the prompt
• MCP-Zero-style discovery of tools only when needed
• agentic context engineering: dynamic cheatsheets, playbooks, and model-maintained memory

Sources and references

Primary source watched for this post:

• AI Agent (1/3)：核心技術 Context Engineering 基本概念解說