agent-05 · Self-correction has three layers: decoding, workflow, and reasoning — Lee Hung-yi-inspired teaching style

All right, everyone, today's question sounds easy: can AI correct itself?

The naïve experiment is to ask, “Are you sure?” The model reflects, changes a few sentences, and sounds more thoughtful. But sounding reflective is not evidence that the answer became true. It may even revise a correct answer into a wrong one.

In one sentence: self-correction needs a signal that distinguishes improvement from confident motion.

Our roadmap has three layers. First, decoding methods steer token probabilities away from an estimated error state. Second, workflow methods add checks and external feedback. Third, reasoning training tries to internalize checking before the final answer. At each layer we will ask the same question: where does the corrective signal come from?

Style note: this is a Lee Hung-yi-inspired teaching-style rewrite. It was not written, reviewed, or endorsed by Lee Hung-yi.

Builder takeaway: use tests, verifiers, retrieved evidence, contrastive baselines, or trained rewards before trusting a model's private feeling that it has fixed itself.

Layer 1: decoding

Begin at the smallest scale: one token decision. We do not retrain the model; we estimate what an error-prone distribution looks like and steer away from it.

The first family does not change the model weights. It changes how the next token is selected.

The core idea is contrastive decoding.

A normal language model produces a distribution over possible next tokens. Contrastive decoding creates another distribution that is supposed to represent a worse or more error-prone state. Then it subtracts the bad distribution from the good one.

In rough terms:

better_answer = normal_distribution - bad_distribution

The trick is how to get the bad distribution.

Early versions used a smaller model. If the big model and small model both assign high probability to a generic continuation, but the big model has more signal for the correct answer, subtracting the small model can help.

DoLa uses layers inside the same model. Earlier layers can be treated as less mature guesses; later layers are more refined. Compare them and push the final output away from the earlier, weaker distribution.

Context-aware decoding does something similar for retrieval. Run the model with the retrieved document and without the retrieved document. The no-document answer is more likely to reflect parametric prior or hallucination. Subtract it to make the model rely more on the retrieved context.

Vision and audio versions degrade the input. Blur the image, remove the image, add noise, or replace audio with silence. The degraded input reveals the model's prior. Subtract that prior from the normal answer.

This is a neat idea. It treats hallucination as a direction you can estimate and move away from.

The cost problem

The method is elegant, but there is no free lunch. Estimating a second distribution often means running extra inference, so correction itself becomes part of the budget.

Contrastive decoding often requires extra inference. You may need to run the model twice, once normally and once in the degraded condition.

That is expensive.

The lecture discusses minimum test-time intervention, or MTI, as a way to reduce the cost. Do not apply contrastive decoding on every token. Only do it when the model is uncertain, for example when entropy is high.

This makes intuitive sense. Most tokens are not life-changing. Spend extra compute only at decision points.

KV cache reuse can reduce cost further when the normal and contrastive inputs share a prefix. Again, this is the kind of engineering detail that matters in real systems. A clever method that doubles inference cost everywhere may lose to a boring baseline.

Layer 2: workflow

Now zoom out from token selection to the process around generation. A workflow can introduce new evidence after the first answer instead of asking the same model to merely feel uncertain.

The second family wraps the model in a process.

Generate an answer. Then ask the model to check it. Maybe ask it to reflect. Maybe run tests. Maybe use a checklist. Maybe search externally. Maybe send the output to a verifier.

This is the most familiar form of self-correction because it looks like how people use chat models:

Are you sure?
Check your work.
Find the bug.
Revise the answer.

The lecture is careful here. Pure self-reflection is not reliably good.

A model can start wrong and become right. It can also start right and become wrong. Asking the model to reconsider may increase willingness to edit, but that is not the same as increasing correctness.

This is why the lecture distinguishes internal self-reflection from external feedback.

Internal reflection is the model criticizing itself with no new evidence.

External feedback gives it something grounded: a compiler error, a failed test, a retrieved document, a checklist, a human comment, a simulator result.

External feedback is usually the better bet.

Reflection has a compute baseline

Suppose reflection uses three model calls. Then its opponent is not a one-call answer; the fair opponent also gets three calls. Otherwise we may be measuring extra compute and calling it self-correction.

One of the best warnings in the lecture is about majority vote.

Suppose your self-reflection workflow costs three model calls: generate, reflect, revise. A fair comparison is not one raw answer. A fair comparison may be sampling three independent answers and taking the majority or best one.

If reflection does not beat that, it may not be worth the compute.

This matters because self-correction papers can accidentally compare against weak baselines. Extra compute often helps. The question is whether the structure of reflection helps more than simply sampling more.

Verification is useful, but it is not free.

The four outcomes of revision

A revision button does not have one outcome. It can repair an error, preserve an answer, remain wrong, or damage something that was already correct. So we need to measure both correction and stability.

The lecture's breakdown of revision outcomes is simple and clarifying:

• wrong to right
• right to right
• wrong to wrong
• right to wrong

The dream is wrong to right.

The nightmare is right to wrong.

Two metrics make this concrete:

• Confidence Level: if the original answer is right, does the model keep it?
• Critic Score: if the original answer is wrong, does the model fix it?

A model that changes everything has high plasticity but low stability. A model that never changes has stability but cannot correct mistakes.

Different reflection prompts move that tradeoff. Praise can make a model hold its answer. Doubt can make it revise more. This is funny, but also important. The model's "personality" under reflection is partly an artifact of the prompt.

Layer 3: reasoning

The third layer moves the loop inside the model's generated reasoning. Instead of attaching “reflect now” after an answer, training rewards paths that check and recover before finalizing.

The third family tries to internalize correction into the model.

Instead of bolting on a workflow that says "now reflect," train the model so that checking and revising happen inside its reasoning process.

This is where RL with verifiable rewards becomes relevant. For math or code, the final answer can be checked. The model generates reasoning tokens and an answer. If the answer is right, it gets positive reward. If wrong, negative reward.

The training does not need to label every intermediate thought. It only needs the final verification signal.

A common result is emergent verification: the model starts producing reasoning where it tries a path, notices a mistake, and corrects before answering.

That looks much closer to real self-correction.

Knowing the fact is not enough

Here is an important distinction: storage is not control. A fact may exist somewhere in the weights, yet the model may fail to retrieve it at the moment when it should challenge its current path.

The lecture gives an important conceptual warning: having the correct knowledge inside the model is not the same as using it to correct yourself.

A model may know a fact in isolation, but fail to apply it when answering a different question. The knowledge exists somewhere in the weights. The self-correction behavior is about control: when to retrieve the fact, when to doubt the current path, when to revise.

This distinction matters because many people talk about reasoning as if the only issue is whether the model "knows" enough.

Sometimes the knowledge is there. The routing is wrong.

REVISE and the distribution shift problem

Error detection and error correction are separate skills, so decomposing them is sensible. But improvement moves the target: after training, tomorrow's mistakes may no longer resemble yesterday's dataset.

The lecture discusses REVISE, which separates error detection from error correction.

The model first learns to detect whether an answer needs refinement. If it does, it emits a special refine token. Then it learns to produce the corrected answer.

That decomposition makes sense. Detecting an error and fixing an error are different skills.

But there is a subtle problem. Once you fine-tune the model, the kinds of errors it makes may change. A dataset of old errors may no longer match the new model's failure distribution.

This is a common trap in self-improvement. Training on yesterday's mistakes can help, but it can also chase a moving target.

Did RL learn something new?

Now we reach the conceptual puzzle. If reinforcement learning improves an answer, did it create a new reasoning path, or simply raise the probability of a path the base model could already sample?

The lecture ends with a debate I find very important.

When RL improves reasoning, what exactly changed?

One view says the base model already had good reasoning paths, but they were low probability. RL just upweighted them. Evidence for this comes from pass@k: if you sample many times from the base model, it may eventually produce a correct path.

Another view says RL can teach genuinely new reasoning behavior, not just reweight old paths. Evidence comes from looking at the reasoning path itself, not only the final answer. If the RL model produces better chains of thought under stricter path evaluation, something deeper may have changed.

The lecture's answer is not dogmatic. Early training may mostly reweight existing paths. Later training may create new capability.

That feels right to me.

My practical takeaway

For a practical agent, climb the signal ladder from strongest and most external to weakest and most introspective. Start with a test or verifier before relying on prose that says “I checked.”

If I want an agent to correct itself, I would not start with "reflect on your answer."

I would use this ladder:

1. Can I get an external verifier?
2. Can I run a test, search, compiler, simulator, or checklist?
3. Can I compare multiple samples cheaply?
4. Can I use contrastive decoding where hallucination priors are obvious?
5. Can I train or choose a model that has stronger reasoning behavior?

Pure self-reflection would be near the bottom of my list unless the model is strong and the task has no external signal.

The main lesson

Let us put the three layers side by side. They intervene at different places, but every successful version must answer the same question: what signal tells the system that the new answer is better?

Self-correction is not one mechanism.

At the decoding layer, we steer token probabilities away from estimated error states.

At the workflow layer, we wrap generation in verification and feedback.

At the reasoning layer, we try to train models that check themselves before finalizing an answer.

The deeper lesson is that correction needs a signal. Without a signal, reflection can become style, not truth.

The best self-correcting systems will probably combine all three layers: better decoding, real external feedback, and models trained to use that feedback inside their reasoning.

Concept inventory

The lecture's main self-correction concepts:

• self-correction as error detection plus error correction without a human giving the answer
• decoding-level methods that do not update model weights
• contrastive decoding and the idea of subtracting an error-prone distribution
• alpha or omega weights controlling the strength of contrast
• smaller models as error models
• logit lens for inspecting intermediate-layer predictions
• DoLa, or Decoding by Contrasting Layers
• layer contrastive decoding in multimodal models
• instruction contrastive decoding using deliberately bad instructions
• context-aware decoding for RAG by contrasting with and without retrieved documents
• visual contrastive decoding by removing, blurring, or masking images
• audio-aware decoding by contrasting normal audio with degraded audio
• minimum test-time intervention on high-entropy tokens
• KV cache reuse to reduce contrastive decoding cost
• VISTA-style representation intervention
• attention-level contrastive methods such as ACG
• workflow-level generation, verification, reflection, and revision
• internal self-reflection versus external feedback
• RefineBench-style evaluation of multi-round refinement
• four revision outcomes: wrong to right, right to right, wrong to wrong, right to wrong
• Confidence Level and Critic Score
• the accuracy relationship ACC2 = ACC1 CL + (1 - ACC1) CS
• reflection prompt personality effects, such as praise preserving answers and doubt increasing edits
• majority vote as a compute-matched baseline
• reasoning as internalized correction
• knowing a fact versus knowing when to use it for correction
• steering vectors for correction states
• REVISE and refine tokens
• distribution shift when training on old error patterns
• RL and RLVR with verifiable rewards
• emergent verification after reasoning training
• cost of thinking and reasoning-token budgets
• reasoning as reducing sample complexity by decomposing problems
• parity or XOR as examples where multi-step reasoning compresses complexity
• pass@k and the question of whether RL reweights existing paths
• training-free sampling methods such as COTPath-style path selection or checking

Sources and references

Primary source watched for this post:

• AI 能自我修正嗎？從 decoding、workflow 到 reasoning 的技術發展整理