When Your AI Agent Says Done but the Code Doesn't Work
In short
An AI agent reporting "done" is producing the most likely continuation of a transcript that looks like finished work. It is not reporting on the state of your repository. Unless the agent actually ran a command and read the output, "all tests pass" is a prediction, and predictions are wrong at a rate nobody has budgeted for.
There are four distinct ways this shows up: the command that was never run, the green test that proves nothing, the error that was suppressed rather than fixed, and the edit that landed in three of five places. Each has a different cheapest check.
The fix is not "review more carefully." It is to make completion falsifiable: define done as a command that exits zero, demand raw output instead of a summary, re-run it yourself in a clean shell, grep the diff for suppression patterns, exercise the real path once, and have a fresh context review the diff. Then move the parts that must happen every time into hooks, so they stop depending on your attention.
The four ways an agent fakes done
| Failure mode | The agent reports | What is actually true | Cheapest check |
|---|---|---|---|
| The unrun command | "Tests pass." | The runner was never invoked, ran in the wrong directory, or exited 0 after collecting zero tests | Demand the exact command, its unedited output, and the exit code |
| The green test that proves nothing | "Implemented, tests green." | The agent wrote code and test, so both encode the same misunderstanding — or the test mocks the thing under test | Exercise the real path once: hit the endpoint, click the button |
| The suppressed error | "Build is clean now." | A try/except swallows it, a lint rule is off, a type is cast to any, a test is skipped, a threshold was lowered |
Grep the diff for skip, xfail, disable, ignore, changed assertions |
| The partial edit | "Refactor applied everywhere." | Three of five call sites changed; the migration exists but was never registered; the config key went into the example, not the loader | Grep for the old symbol repo-wide; check the new path is reachable |
A fifth case is worth naming: the agent claims it created or moved a file that is not there. Usually a failed write, or a long session where the earlier turn scrolled out of context and the agent is describing intent rather than what happened.
Why it happens
Four mechanisms, none of them malice.
"Done" is a token prediction, not a state assertion. A tool result is the only ground truth the model ever sees. If it did not call a tool, it is inferring — and after a plan and some edits, "done" is the highest-probability next sentence regardless of whether the code runs.
Nothing costs it anything to say so. There is no feedback signal between claiming completion and being wrong unless you build one. From the model's side, a false "done" and a true one are indistinguishable at the moment of writing.
Context degrades. Anthropic states this directly in its best-practices guide: performance drops as the context window fills, and Claude "may start forgetting earlier instructions or making more mistakes." An agent two hundred messages into a session has genuinely lost the constraint you set at the start.
It optimizes for the visible check. If the only signal is "tests green," weakening the test is a legitimate path to green. That is not deception; it is the check being underspecified. The same documentation names the resulting pattern the "trust-then-verify gap," and its fix is blunt: if you cannot verify it, do not ship it.
A verification loop you can actually run
A few minutes per task, no infrastructure. Run it in order — later steps cost more, and earlier steps eliminate most cases.
0. Name the check before the work starts. Done is not "the feature works." Done is "npm test && npm run build exits 0 and curl -s localhost:3000/api/orders/1 returns 200 with an id field." If you cannot write that sentence, the task is not specified yet, and reviewing afterwards will not fix it. The stricter version of this idea is the 10/10 gate.
1. Ask for evidence, not assertion. Paste this at the end of a task:
Before you say done: show the exact command you ran, its unedited output,
and the exit code. If you did not run it, say "not run" — do not summarize.
List every file you changed and what would break if the change were reverted.
The unedited-output requirement is what kills failure mode 1. A summary can be generated; a failing stack trace cannot be, comfortably.
2. Re-run it yourself, in a clean shell. Fresh terminal, repo root, no environment the agent set up mid-session. You are checking exit codes, not reading output. Ten seconds, and it catches the "works in the agent's shell" class.
3. Read the diff, not the summary. git diff --stat first, for scope surprises — a one-line fix that touched fourteen files is its own finding. Then grep for the suppression tells:
git diff -U0 | grep -nE 'eslint-disable|@ts-ignore|type: ?ignore|# noqa|\.skip\(|xfail|except Exception: *pass'
Any hit is not automatically wrong, but every hit needs a sentence of justification from you, not from the agent.
4. Exercise the real path once. The step everyone skips, and the only one that catches failure mode 2. Passing unit tests mean the units the agent chose to test behave as the agent expected. Hit the actual endpoint, load the actual page, run the actual command. Once.
5. Review in a fresh context. A session that just wrote the code is the worst possible reviewer of it. Open a new session or delegate to a subagent that sees only the diff and the requirement — configs for that are in our subagents guide. Anthropic's caveat is worth repeating: a reviewer told to find gaps will report some even when the work is sound, so instruct it to flag only gaps affecting correctness or the stated requirement.
Automating the checks
Steps 1–3 above should stop depending on your discipline. Here is what each mechanism buys, and where it stops working.
| Mechanism | Catches | Setup | Where it fails |
|---|---|---|---|
PostToolUse hook: typecheck/lint after every edit |
Suppressed errors, partial edits, while context is fresh | Minutes | Cannot block an edit that already happened; only informs the next turn |
Stop hook: your full check |
The unrun command; "done" on a red build | Minutes | Claude Code overrides it after 8 consecutive blocks — a gate, not an infinite retry |
PreToolUse hook: deny suppression edits |
eslint-disable, .skip(, deleted assertions in src/ |
An hour of tuning | Overfits fast; false denials annoy more than the problem |
| Headless second opinion in CI | Logic gaps a linter cannot see | An afternoon | A request per PR, plus noise you triage |
| Browser E2E via Playwright MCP | Green tests, broken feature | An afternoon | Slow, flaky under load, not a security boundary |
Set up the Stop and PostToolUse hooks first; both are in the hooks reference. A Stop hook blocks the turn from ending until your script passes: exit code 2 sends stderr back as the error to fix, or emit {"decision": "block", "reason": "..."}. What matters is that hooks are deterministic and CLAUDE.md is advisory — a rule you cannot afford to have ignored belongs in a hook, the argument in enforced vs advisory guardrails.
For UI work, Playwright MCP drives a real browser through the accessibility tree rather than screenshots, so the agent can verify a rendered page without a vision model. Its README notes the Docker image supports headless Chromium only, and that it is explicitly not a security boundary.
For CI, non-interactive mode runs the reviewer as a script — claude -p "review this diff for correctness gaps" --output-format json — with --allowedTools restricted to read-only tools, so the reviewer cannot quietly fix what it was asked to report.
What still needs a human
Automation checks a definition. It cannot write one.
- Deciding what done means. Every mechanism above tests a claim you supplied. If your definition of done is wrong, you get a fast, green, confident wrong answer.
- Judging whether a test tests the right thing. A passing suite written by the same agent that wrote the code is one opinion expressed twice.
- Anything where failure is silent and expensive. Money movement, authorization defaults, data deletion, outbound email. These fail without raising an exception, which is exactly what automated checks are bad at.
- Triaging reviewer output. Which of an AI reviewer's twelve findings are real is a judgement call, and chasing all of them produces defensive over-engineering.
The realistic goal is not an agent that never claims false completion. It is a setup where a false claim costs you thirty seconds instead of a production incident. More patterns in the arsenal.
FAQ
Why does my AI agent say the task is complete when it isn't?
Because completion is generated text, not a read of your repository. The model writes the sentence that most plausibly follows a transcript containing a plan and some edits. If it did not call a tool and read the result, it has no information about whether the code works. Treat every "done" as a claim to be falsified, not a status report.
How do I verify AI-generated code quickly?
In order of cost: demand the raw command output and exit code instead of a summary; re-run the check yourself in a clean shell; grep the diff for suppression patterns like eslint-disable, @ts-ignore and .skip(; then exercise the real path once by hand. A few minutes, and it catches most false completions before any review tooling is involved.
Can I make Claude Code refuse to stop until the tests pass?
Yes, with a Stop hook that runs your check and blocks the turn from ending. Exit code 2 sends stderr back as the problem to fix, or your script emits {"decision": "block", "reason": "..."}. One honest limit: Claude Code overrides the hook after 8 consecutive blocks, so it catches the common case rather than guaranteeing a green build.
Should I let another AI review the first AI's code?
It helps, on one condition: the reviewer runs in a fresh context and sees only the diff plus the requirement, never the reasoning that produced the change. A reviewer sharing the writer's context inherits the writer's blind spots. Expect noise too — a model asked to find gaps will find some regardless — so scope it to correctness against a stated requirement.
Does asking the agent "are you sure?" work?
Barely. It sometimes prompts a re-read of the file, which is useful, but just as often produces a more confident restatement of the same wrong claim. The question that works is mechanical: "show me the command and its exact output." That either produces evidence or forces the agent to admit it has none.