agent-01 · Context engineering is the agent's working memory — Lee Hung-yi-inspired teaching style

All right, everyone, let's start class.

Today we are going to talk about context engineering. The name sounds like a narrow infrastructure topic, perhaps just a sophisticated way to say “write a better prompt.” But that intuition is too small.

In one sentence: context engineering is working-set management for an LLM-shaped processor.

Here is our roadmap. First, we let the naive append-only agent fail. Then we separate prompt, context, memory, compression, and observation. Finally, we bring in sub-agents, on-demand skills, and agent-managed context.

The important pattern is naive approach → failure → system fix. Once you see that sequence, you know which layer to debug instead of blaming the model for everything.

Style note: this is a teaching-style rewrite inspired by Lee Hung-yi. It was not written, reviewed, or endorsed by him.

The practical context stack

Before looking for tricks, let's draw the stack. If you do not know where each layer lives, every bug starts to look like “the model is not smart enough.”

A language model does not wake up remembering yesterday. It receives a prompt and predicts what comes next. If you want it to act like a long-running worker, a system around it must choose what enters the next prompt.

That system is the agent's working memory. The useful way to read the lecture is as a taxonomy, not as one special technique. Context engineering decides what the model can see, what remains outside the prompt, what gets compressed, and what evidence survives long enough to be checked.

For agent builders, I would split the stack like this:

• Prompt-visible workspace: the small active set the model sees right now.
• External memory: files, logs, notes, vector stores, playbooks, and other state outside the prompt.
• Compression: summaries, markers, and pointers that keep long histories usable without pretending nothing was lost.
• Bounded observations: tool results shaped before they enter the model context.
• Sub-agent interfaces: messy branches that return concise conclusions plus verifiable handles.
• On-demand skills: tool and procedure descriptions loaded when relevant, not carried in every step.

This taxonomy is also a debugging checklist. If a coding agent loops, hallucinates, misses a file, or confidently says “done” without evidence, the failure may be in one of these context interfaces rather than in the base model alone.

The naive agent just appends everything

What is the most naive design? Very simple: append everything. It sounds honest—nothing is hidden—but it breaks as soon as the agent works for long enough.

A user says something. The model replies. A tool returns an observation. The agent appends all of it to the next prompt.

In rough form:

C_next = C_current + input + output

This feels safe. Nothing gets lost. The model can see the whole history.

It also breaks quickly.

Tool outputs are long. File reads are long. Logs are long. Web pages are long. A few cycles later, the prompt is full of stale details, failed attempts, giant observations, and fragments that are only accidentally related to the current decision.

The model is limited not only by context length, but also by attention. Even with a large context window, burying one relevant fact inside pages of irrelevant text makes the job harder.

So the lecture replaces the append operation with a function:

C_next = F(C_current, input, output)

The core engineering problem is no longer “how do I preserve all the text?” It is “what should the update function F do?” That small shift changes the whole design.

Prompt is not context

You might ask, “Isn't the prompt just the context?” Not quite. The prompt is the small part visible now. Context is the larger system that decides what can become visible.

The prompt is what the model sees right now.

Context is broader. It includes what the agent has experienced, what it can retrieve, what is stored on disk, what skills exist, what logs were written, and what memories may become relevant later.

A helpful notation is:

context = P + M

P is prompt-visible context. M is external memory.

Weak designs confuse the two and try to make P contain everything. That gives you giant prompts full of tool manuals, previous mistakes, raw logs, and old observations.

A better design treats the prompt as a small active workspace. External memory can be much larger, but it stays outside until the agent has a reason to read it.

This is how I want coding agents to behave. Do not paste the whole repository into the model. Give it a map, search tools, file readers, and clear rules for when to inspect what.

Compression is necessary, but dangerous

Now we have a problem. Context fills up, so we must compress it. But compression is not a free lunch: it throws information away.

You can summarize old turns, mask observations, or store long tool outputs as files and leave behind a pointer.

The lecture discusses several patterns:

• summarize old history with another LLM
• replace a long observation with a short marker
• save raw logs to a file like log1.txt
• keep only the parts needed for the current task
• ask a sub-agent to handle a branch and return a short result

The danger is context collapse.

A summary can remove exactly the fact that later becomes necessary. The agent may have had enough information before compression, then fail afterward because the important bit was treated as noise.

So summarization is not a harmless cleanup step. It is an information bottleneck, and you need to know what the task cares about.

The ACON idea mentioned in the lecture fits here: when compression causes failure, use feedback to teach the summarizer what not to drop. That treats context engineering as a learning problem, not merely a prompt-formatting problem.

Observation is the real context killer

Here is a subtle point: the model's answer is often not what explodes the context. The tool observation is.

A shell command returns 500 lines. A file read dumps a whole module. A browser scrape returns a page. A test run prints a giant stack trace. Now the model must make a decision inside a swamp of text.

This suggests a better intervention point. Instead of summarizing everything afterward, prevent bad observations from entering the prompt in the first place.

A smarter read tool should not always mean “read the entire file.” It could mean:

• read a specific line range
• search for related symbols first
• return a summary plus line references
• hide irrelevant output unless requested
• preserve raw output externally for later inspection

This is where tool design and context engineering blur together. A tool that returns better observations is also a context-engineering tool.

Sub-agents are compression devices

A sub-agent is not merely another little coworker invited to a meeting. From the context perspective, it is a compressor that can do work.

The main agent can ask one to inspect files, compare papers, or test a hypothesis. The sub-agent accumulates the messy intermediate context. When it returns, the main agent receives a short answer: what was found, what failed, and what matters.

That is compression with agency.

It is also risky. If the sub-agent returns a bad summary, the main agent may never see the missing evidence. So a good sub-agent should return not just a conclusion, but anchors: files, line numbers, commands, timestamps, or other handles the parent can verify.

This is why I do not trust “done” from an agent unless it gives me a path, diff, test result, or reproduction.

Tool descriptions should not all live in the system prompt

Now consider one very practical question: should every tool manual live in the system prompt? The naive intuition says more instructions must be better. In practice, the agent pays for that manual on every step.

If you expose a large tool library by dumping every description into the prompt, the manual becomes a tax on every single step. It also makes tool selection harder.

The lecture mentions approaches such as MCP-Zero and on-demand skill loading. The idea is straightforward: keep tool and skill descriptions searchable, then load only what is relevant.

This is retrieval-augmented generation for agent affordances rather than documents.

It matters for long-lived personal agents. A useful personal agent may have hundreds of small skills—email, calendar, code review, video editing, note taking, finance, deployment, writing. It needs to remember that a capability exists without carrying the full manual in its head.

Agentic context engineering

Finally, can the agent help manage its own context? Yes—but now we must draw boundaries.

The lecture calls this agentic context engineering. Instead of humans hard-coding all of F, the agent can maintain a cheatsheet, update a playbook, decide what to store, or search external memory by itself.

Examples include:

• dynamic cheatsheets
• playbooks that get updated over time
• recursive language model style systems that store large context externally and keep only metadata visible

This is powerful and uncomfortable for the same reason. If the agent can edit its own working notes, it can improve. It can also write bad rules, preserve wrong lessons, or delete the one constraint that mattered.

I would not let an agent casually rewrite its root identity or safety rules. But letting it maintain task-level notes, project conventions, and reusable tactics seems not only reasonable but necessary.

What I would actually use

If I were designing an agent after this lecture, I would begin with a few boring rules:

1. Keep the active prompt small: make the model work in a clean workspace, not a transcript dump.
2. Store raw outputs externally: preserve logs, diffs, traces, and long reads outside the prompt.
3. Put pointers in the prompt: use file paths, line numbers, commands, timestamps, and source URLs instead of giant blobs.
4. Bound every observation: make tools return structured summaries, relevant ranges, and handles for deeper inspection.
5. Use sub-agents for messy branches: isolate exploratory context, then return only the claim, evidence, and reproduction path.
6. Require verifiable handoffs: do not accept “done” without a path, diff, test result, or other anchor.
7. Load tools and skills on demand: keep tool manuals searchable instead of making every step pay the full context tax.
8. Treat summaries as lossy: test whether compression preserves the facts future steps need.
9. Let the agent maintain a playbook: allow task-level learning while keeping root rules and safety boundaries protected.

None of this is glamorous. That is probably why it is important.

Many agent failures look like reasoning failures from the outside. Sometimes they are. But after this lesson, ask the more basic question first: did the agent have the right working memory?

Concept inventory

For reference, here are the lecture's main concepts in one place:

• context engineering as the update function from current context, input, and output to the next context
• context window limits and why long tasks need compression
• prompt-visible context versus external memory
• P + M: prompt contents plus memory outside the prompt
• compaction, summarization, and observation masking
• memory externalization: save long outputs as files and keep only pointers active
• context collapse: losing the one detail the task later needs
• ACON-style feedback for better compression
• AgentFold and fold-style tools for turning long histories into short notes
• sub-agents as both parallel workers and context compression devices
• observation filtering before tool output enters the model context
• on-demand tool and skill loading instead of dumping every manual into the prompt
• MCP-Zero-style discovery of tools only when needed
• agentic context engineering: dynamic cheatsheets, playbooks, and model-maintained memory

Sources and references

Primary source watched for this post:

• AI Agent (1/3)：核心技術 Context Engineering 基本概念解說