Last updated July 27, 2026, by the Editorial Team. Originally published on Towards AI.

“My job is to write loops.”

That statement came from Boris Cherny, who leads Claude Code at Anthropic. Cherny has said he stopped prompting Claude directly and now spends his time designing the loops that prompt it for him [1]. The remark, along with several similar comments, helped trigger a wave of loop-engineering explainers this year [1][2]. After reading six of them, I built one.

More precisely, I built the two components that nearly every explainer describes but few appear to run end to end. The first was a run-until-done loop: instead of asking the model to guess again after a failure, it feeds the model its own real test failures. The second was a maker/checker setup: the model that writes the code is not allowed to be the final judge of whether the code is correct.

That distinction matters because coding-agent workflows often depend on the same separation: one system proposes a change, while another tool, test suite, or model decides whether the change is acceptable. If the feedback channel or verifier is weak, the loop can look automated without becoming meaningfully safer.

I implemented both from scratch in about 600 lines of Python, wired them to claude-opus-4-8, and evaluated them against MBPP+ [3]. MBPP+ is an EvalPlus benchmark of small Python programming tasks, which made it useful for isolating loop behavior without bringing in repository-scale complications. The total cost across every experiment discussed here was under two dollars. The second component — the one often treated as the safer half because it “actually runs tests” rather than trusting the model’s word — produced results in my measurements that those explainers had not warned about.

The short version: loop engineering’s two core pieces are simple to connect and easy to get quietly wrong. My “real feedback” loop initially looked indistinguishable from random retries until I found a bug in my own test harness. My “safe” test-running verifier had a higher false-accept rate than a checker that simply asked the model how confident it felt. Building the loop is the easy 20%.

Wiring a Loop to Nothing

Much of the writing about loop engineering stops at the wiring diagram. It lays out a trigger, a verifiable goal, tools, state, and stop rules — five boxes with arrows between them. The implied message is that once the boxes are connected, the loop works.

That is like installing a smoke detector and declaring the house safe because the detector is mounted on the ceiling and wired in, without checking whether it has a working battery.

I ran into exactly this failure with my “real feedback” loop. It was connected to the actual test output, not to a generic retry prompt. In theory, it should have clearly outperformed a loop that received only the message, “that was wrong, try again.” My first run showed otherwise.

The Grader That Grades the Grader

Before touching the loop itself, I built the component everything else depended on: a scorer that runs candidate code in an isolated subprocess with a hard timeout and grades it against hidden tests.

I did not trust the scorer until it graded itself. When given a known-good solution, it had to pass. When given a known-bad solution, it had to fail and attach the assertion error. When given an infinite loop, it had to be killed by the timeout rather than hang forever.

I then validated the entire pipeline against 75 MBPP+ reference solutions. All 75 passed. Only after that did I trust any number produced by the loop.

The Loop That Looked Correct but Was Not

The loop itself was very simple. It generated a solution, graded it, and, on failure, fed back the real stderr — not a generic “try again,” but the actual error — for as many as three attempts.

I also built a control arm because I did not want to trust a headline number without a baseline. The control used the identical loop but replaced the real error with a generic instruction: “that was wrong, write a different solution.” If real feedback did not clearly outperform that, then something in the wiring was broken.

On the first run across 35 problems, all three arms were identical. That was not evidence of a working loop. It was a red flag in the shape of one.

Instead of focusing on the top-line metric, I examined the failures and found the problem: MBPP+’s hidden-test harness was failing with a bare AssertionError. It did not include the failing input, expected value, or actual value. As a result, “real feedback” was informationally identical to “try again,” because the model received nothing it could act on.

I instrumented the harness so it reported the failing input, the expected output, and the code’s actual returned value. On the same 35 problems in a second run, real feedback recovered a problem the generic arm could not solve, at a cost of about 2,500 additional input tokens across the run.

The loop had not been broken. The signal it was connected to was empty. Only the control arm exposed that; the headline metric alone would not have done so.

The Verifier Failed in a Way the Theory Did Not Predict

A loop needs a stop rule, and “the model says it is done” is not a stop rule. I therefore built a checker that writes its own tests from the specification. It never sees the hidden tests and never sees the solution code it is judging. It then runs those tests. The policy was to accept only on a clean sweep and reject by default.

I compared that verifier with three weaker checkers on 41 candidates produced by my loop: 33 correct and 8 wrong. The measured quantity was the false-accept rate, meaning how often each checker allowed code through even though the code was actually broken.

I expected the test-running checker to win outright on false accepts. It did not. It allowed through a higher fraction of wrong code than either opinion-based checker.

The reason mattered more than the number. All eight wrong candidates came from three problems with genuinely ambiguous specifications. The checker and the fixer were the same model reading the same ambiguous sentence. The checker’s self-written tests therefore encoded the same misreading that the wrong code already contained, allowing the wrong code to pass those tests cleanly.

The opinion-based checkers “won” that false-accept comparison mostly because they were generally hesitant. That same hesitation is also why they falsely rejected four to five times more correct code.

Running tests is not a free pass against false accepts. It is a different kind of evidence: one that includes a specific input, an expected value, and an actual value instead of a general impression. That is what makes its 3% false-reject rate usable as a real gate. A checker that rejects 15% of good work can bury a workflow in retries before it prevents a bad merge.

Combining the Two Pieces

The final composition runs the feedback loop first. If that fails, it samples fresh candidates and lets the verifier — not the model’s confidence — decide what gets submitted. It also includes an explicit surrender path if nothing clears the bar.

I evaluated this setup on a held-out slice of MBPP+ that I had not used while building any of the preceding components. Grading was performed against the hidden tests, independently of whatever the verifier decided.

The loop did essentially all the work: it added 14.2 points, recovered five of six single-shot failures, and required about 10 extra API calls. The verifier stage fired exactly once, on the one problem the loop could not solve. Its first sampled candidate passed its own self-written tests and still failed the hidden tests.

That was a live false accept, matching the failure mode predicted by the earlier table. It was caught only because the runner graded submissions against a source of truth the verifier never sees. If the verifier’s own judgment had been final, that bug would have shipped.

The total cost for this stage was 45 calls, or roughly thirteen cents.

Where the Approach Breaks

This approach works when the goal is genuinely testable: a function with hidden test cases, a schema that either validates or does not, or an oracle the loop cannot talk its way around. It does not work when the specification itself is ambiguous, because a same-model checker can inherit the same misreading as the generator. In that case, the fix is not necessarily a cleverer checker. It is a clearer specification, or an independent oracle from a different model family entirely.

I also tested this only on small, standalone MBPP+ functions. The experiment did not address a large codebase with cross-file dependencies. I did not build or test worktree isolation for running multiple loops in parallel. That is a real problem, but it was not the problem this experiment measured.

The loop also stops at “verified.” It does not decide whether to auto-apply a change or escalate it to a human. That becomes a separate and harder problem as soon as the system touches anything with real write access.

Start Here

Start with the grader. Write the self-test — known-good, known-bad, and infinite-loop — before writing any loop logic. That five-minute script is the main protection against a fictional headline number.

For a codebase with even a small set of unit tests, wire the feedback loop next and build the generic-retry control arm alongside it from day one. Do not trust the improvement until it beats “try again.”

Then build a second checker and compare it with the first one. When the false-accept table disagrees with what the theory suggested — and it may — the reason will show why the loop matters more than the model underneath it.

References

[1] Rohan Mistry, “Prompt Engineering Is Dead. Loop Engineering Is Here.,” Towards AI, July 2026.

[2] Mehmet Özel, “Loop Engineering for AI Agents : Building Verifiable, Self-Correcting Coding Workflows,” Towards AI, June 2026.

[3] EvalPlus, “MBPP+ Dataset,” Hugging Face Datasets.