agent-03 · What AI agents change about research work

The third lecture in Hung-yi Lee's AI agent series is the most concrete one. It asks what agents do to work, then uses academic research as the example.

The answer is not "AI writes papers, humans go home." That is too simple. The better answer is that research gets decomposed, and different parts of the pipeline change at different speeds.

Some parts are already becoming cheap. Some remain stubbornly human.

Builder takeaway: agents do not replace research as one blob. They lower the cost of drafts, code, reviews, and analysis, which makes verification, problem selection, and feasibility judgment more important.

From tool to coworker to worker

The lecture starts with a useful progression.

At first, AI is a tool. You give one command, it does one thing.

Then AI becomes a collaborator. You discuss, iterate, and co-produce.

Now agents are starting to look like workers inside bounded tasks. Give them a goal, a repository, some tools, maybe a paper draft, and they can run for a while.

That last step changes the shape of work. It is one thing to ask a model to summarize a paper. It is another thing to ask an agent to inspect a dataset, modify code, rerun an experiment, write up the result, and check whether the output matches the original question.

The second version is not just generation. It is a workflow.

The 100x research assistant example

One example in the lecture involves Claude Code writing a paper-like analysis using an existing paper's method and new data. The result was not perfect. It made at least one data mistake. But the cost and time comparison was striking: roughly an hour and a small amount of money versus a human research assistant spending far longer.

The lesson is not that the AI result is automatically better.

The lesson is that the first draft of many research tasks is becoming much cheaper.

This shifts the economic question. Instead of comparing "AI alone" against "human alone," the relevant workflow may be:

AI generates first pass
human checks errors
AI revises
human decides if it matters

If the checking cost is low enough, this beats doing everything manually. If the checking cost is high, or if errors are subtle and dangerous, it may not.

That distinction matters. Agents do not eliminate verification. They make verification more central.

Research is not one task

The lecture breaks research into pieces, implicitly and explicitly:

• finding literature
• understanding prior work
• proposing ideas
• designing experiments
• writing code
• running models
• analyzing data
• writing the paper
• reviewing the paper
• teaching the result

AI is not equally good at all of these.

It is already useful for literature triage, code edits, drafts, and analysis. Autoresearch loops show agents changing training scripts, running experiments, looking at results, and trying again. That is real leverage.

But idea generation is harder.

The lecture discusses work where LLM-generated research ideas scored well on novelty, excitement, and overall impression, but worse on feasibility. That is exactly the failure mode I would expect. Language models are very good at combining concepts into something that sounds new. The real test is whether the idea survives contact with implementation.

A beautiful research idea that cannot be run, measured, or falsified is mostly decoration.

The feasibility bottleneck

This is probably the most important part of the lecture.

AI can generate many ideas. Some will look novel. Some will even be genuinely useful. But feasibility is where human taste still matters.

Can this experiment actually be run?

Is the dataset available?

Will the metric answer the question?

Is the expected effect large enough to detect?

Is this a real problem or just a clever recombination of terms?

These questions require domain judgment, not just fluency.

The lecture's follow-up example, where AI ideas looked worse after being implemented into short papers, is a good warning. Idea quality cannot be judged only at the pitch stage. Execution reveals whether the idea has bones.

AI reviewers are coming, but that is not the real issue

The lecture then moves to AI review. AAAI 2026 is mentioned as an example where AI reviewers and AI meta-reviewers become part of the process.

This will make people angry, but the lecture's position is more nuanced than "AI review bad."

Bad review is bad.

If a human reviewer pastes a lazy AI-generated review that misunderstands the paper, that is bad because it is useless. If an AI reviewer points out a real flaw, gives a concrete suggestion, and helps the author improve the work, that is useful.

The question is not whether the text came from AI. The question is whether the review does the job.

That said, the system needs accountability. A review process cannot become a chain of agents producing confident artifacts that nobody owns. Someone needs to decide whether the criticism is valid.

Deadline-aware review is an underrated idea

One of my favorite practical examples is the lab agent that reviews student papers with awareness of the deadline.

If the deadline is far away, it can suggest more experiments.

If the deadline is tomorrow, "run three new ablations" is not helpful. The agent should focus on clarity, framing, related work, and changes that can actually be made before submission.

This is a small example, but it captures what good agent design looks like. The agent is not just evaluating in the abstract. It is evaluating inside a human workflow.

A technically correct suggestion can be useless if it ignores time.

AI Agent for Science and the partial loop

The lecture also discusses AI Agent for Science, an experimental setup where AI is a major contributor and AI review is part of the process.

This points toward a possible research loop:

AI proposes
AI experiments
AI writes
AI reviews
AI revises

It is tempting to call this a closed loop of scientific progress.

But the details matter. Accepted work still appeared to involve more human input in the early stages: idea selection and experiment design. AI was stronger in data analysis and writing.

That division feels right. Later-stage tasks have clearer local feedback. Early-stage research taste is harder to specify.

The human job moves around

The strongest version of the lecture is not that humans become irrelevant. It is that human work moves.

If AI makes drafts, code edits, and routine analysis cheap, then human value shifts toward:

• choosing problems
• setting research taste
• designing experiments
• checking validity
• deciding what is worth publishing
• accepting responsibility for errors

This is not necessarily comforting. Some jobs are built around doing the pieces that AI is making cheap. But it is more accurate than saying "researchers are replaced."

Research was never just writing papers. At its best, research is choosing good questions and forcing reality to answer them.

What I would change in my own workflow

After this lecture, I would use agents more aggressively for first passes, but I would not trust them more.

That sounds contradictory. It is not.

I would let an agent:

• summarize a literature cluster
• draft experiment code
• create a baseline implementation
• generate tables and plots
• write a first related-work section
• review a paper for obvious gaps
• run a checklist before submission

But I would keep human control over:

• what question matters
• whether the experiment tests the claim
• whether the result is real
• whether the paper should exist

The point is to spend less human time turning the crank and more human time deciding which crank is worth turning.

The main lesson

AI agents are not a single shockwave hitting all research work equally. They are a set of tools that make some steps dramatically cheaper and leave other steps exposed as the real bottlenecks.

The cheap parts will expand. We will produce more drafts, more experiments, more reviews, more plots, more candidate ideas.

That makes taste more important, not less.

When output becomes cheap, judgment becomes the scarce resource.

Concept inventory

The main research-work concepts covered in the lecture:

• the role shift from AI as tool, to collaborator, to bounded autonomous worker
• Claude Code-style research assistance for data analysis and paper drafting
• the 100x research assistant idea: cheaper first passes plus human verification
• human verification as the bottleneck after AI generation becomes cheap
• autoresearch loops that modify training scripts, run experiments, observe results, and try again
• AI-generated research ideas with high novelty or excitement but lower feasibility
• the gap between surface novelty and executable research design
• AI reviewers and AI meta-reviewers in conference workflows
• review quality as the key issue, not whether the reviewer used AI
• deadline-aware review that changes advice based on remaining time
• AI Agent for Science-style loops with AI authorship and AI review
• the four research stages: idea, experiment design, data analysis, and paper writing
• AI being stronger in later stages and weaker in problem selection or experiment design
• the shift of human value toward taste, feasibility judgment, quality control, and responsibility

Sources and references

Primary source watched for this post:

• AI Agent (3/3): AI Agent 對於工作帶來的衝擊 - 以學術研究為例