Does it help code quality if agents encode their intents?

The idea

A code reviewer, human or model, gets finished code and no specification. They have to work out what the code is for from the code itself, bar a PR description that is verbose enough. And code agrees with itself. The author is better placed: they have the request, in the words of whoever made it, at the moment the change is written.

So we wrote a skill for coding agents, intent, that asks for three things in a fixed order: statements of what the change must do, then a check for each statement that a machine can run, then the code. A statement looks like this:

## S2 A charge that would overdraw is refused
- kind: changes
- state: checked
- given: a balance of 10 and a charge of 11
- then: the charge is refused and the balance stays 10
- source: request: "refuse a charge that would overdraw"
- check: `python -m unittest tests.test_billing.TestCharge.test_refuses_overdraw`

The design borrows four ideas from proof assistants such as Lean, and none of the proving:

• The statement is fixed before the proof. Statements come before the code. One written from finished code is met by construction.
• A small checker decides, not the author. A script with no model in it runs every check twice: with the change, and with the change taken out. A check for new behaviour has to fail without the change, or it shows nothing about the change. A check for behaviour that must stay has to pass both times.
• Checked, refuted and open are three states. Nothing is dropped in silence.
• What was assumed is marked. Every statement names its source: the request, the existing code, or the agent's own guess.

It sounds right. We wanted to know whether it works.

Two claims in one idea

The idea makes two claims, and they are worth keeping apart:

• Discovery. Writing statements helps an agent work out what is wanted when the request is thin.
• Preservation. Once what is wanted is known, writing it down helps the agent keep to it, and helps a reviewer hold the code to it.

Our tests bear mostly on the first, for a reason that comes next.

What we tested it on

We took 54 real changes from six open-source projects (hono, httpx, cobra, urfave/cli, click and ky). Of these, 42 contain 44 bugs that the projects later fixed, each one visible in the change that introduced it. The other 12 are clean. For every bug we have the fix, so "is the bug there?" has an answer that does not depend on anyone's opinion.

Each change came with one line of request: its commit subject, such as "Add NetRCAuth() class". So the agents never had what the idea assumes an author has, the real request. Everything below is about agents that had to work out most of the intent themselves.

Read the numbers with this in mind:

• They are small. 44 bugs for the statements and the reviewers, 9 for the agents' own code, in one to four runs. Runs on the same bug are not independent of each other, so we compare bug by bug.
• The code is public. Every model may have seen these repositories and their fixes in training. That holds for both sides of every comparison.
• The harness is plain. The agents ran in a simple tool loop with the skill's text in the task, not in a product such as Claude Code. Its limits ended a quarter of DeepSeek's runs.
• Models judged. Where a judgement was needed, two models labelled apart and a third settled disagreements. The two agreed on more than nine labels in ten.
• We decided beforehand. For each test, the question, the cases and what would count as a result were written down before any model was called.

All four experiments together cost about $45 in model calls.

1. Do the statements say anything about the bugs?

We gave Claude Sonnet 5.5 the skill's text and asked it for statements, once before the code existed and once with the finished diff in hand.

Written first, the statements fix the right behaviour for the exact case that later went wrong in a third of the bugs. Written afterwards, they do about as well, with one difference: for 8 bugs they describe the buggy behaviour as what was wanted. That is the order the skill warns against, and here is the reason.

2. Does a check written from such a statement catch the bug?

For the 15 pinned-down bugs, an agent wrote a test from the statement alone, without seeing the change. We then ran it on the change as its author committed it, and again with the later fix applied.

11 of the 15 tests fail on the bug and pass on the fix. All 15 fail on the buggy change. The four that stay red after the fix do so because the statement was wrong or said too much (three), or because the code holds a second copy of the same mistake (one).

There is a cost. Of 29 checks in all, 19 were red on the change as written, and 5 of those were red because the statement was wrong. Each of the five named a source. The error was in the detail the writer added to it.

3. Does an agent write better code with the skill?

This is the question the skill was built for. We had agents write eight of these changes themselves, from the same one-line request, with the skill and without it. Hidden checks then read the cases of nine known bugs. We picked the bugs in the skill's favour: all nine are ones a skill-written statement had pinned down.

No measurable gain, on any model. Beforehand we had set the bar at five bugs better and none worse. Opus avoids all nine bugs unaided, so it had nothing to gain. Haiku and DeepSeek get about a third of the cases wrong either way, and the two-case differences between the columns are less than one run moves by.

Three things stood out.

• A stated case came out right. Where the agent's own statements covered the bug's case, the code handled it in 45 of 48 cases. That is a correlation. Without the skill, a case the agent wrote a test for came out right every time as well, 40 of 40. An agent that thinks of the case handles it, written down or not.
• The skill did not make the weaker models think of the case. They stated about three cases in five. Opus stated all nine, and without the skill it wrote a test for eight of them anyway.
• The one bug that kept coming back was never stated. A flag name repeated at two levels of a command tree was listed twice in the help, in 13 of 16 runs by the two weaker models. Not one of their intent files had a statement for it. Three mention it in a note, one in so many words: "Left as it was: not asked for."

We also learned to distrust single runs. After one run, Haiku with the skill looked worse on three bugs and better on one. After four, it was the other way round. Neither difference is more than chance.

4. Does a reviewer find more bugs with the statements?

Last, the reviewer's side. We ran Sonnet 5.5 and Haiku 5.5 as reviewers over all 54 changes, three times each, under four conditions: no statements; the statements written before the code; those written after it; and, as a ceiling, one statement per bug written by someone who knew the bug.

Bugs found in at least two of three runs, of 44:

• A statement that covers the bug works. Told the bug's case, in the neutral words of a requirement, Sonnet found 17 bugs more often than without and 2 less often. Haiku found 29 more often and none less often, and all but caught up with Sonnet. Reviewers do read statements. What this ceiling does not show is that our format adds anything over handing the reviewer the same requirement in any other way.
• Statements written before the code do not help. Sonnet found 6 bugs more often and 8 less often, Haiku 8 and 5. Neither is more than chance. Where the statements cover the bug, Haiku gains a little: 6 more often, 1 less.
• Statements written after the code did not measurably hide bugs. The eight bugs they describe as intended were found a little less often by both reviewers. Eight bugs are too few to tell.

The statements made the reviewers wrong more often

This may be the result that matters most in practice. On the 12 clean changes, reviewed three times each, the reviewers posted:

Most of the new findings cite a statement: the code "contradicts the claim that" something is kept. By a word search, that holds for 28 of Sonnet's 42 and 29 of Haiku's 36. But the claim was a model's guess from a commit subject.

Every statement carried a label saying where it came from. It did not stop the reviewers. Of the 22 wrong findings that name their statement, 16 cite one the writer had marked as its own inference and 6 one it had tied to existing code. None cites a statement quoted from the request.

A guess that is written in the form of a requirement gets enforced like one.

Two cautions on the size of this. Twelve clean changes are a small base. And in our set four changes in five have a bug, where in ordinary work most changes are clean, so this cost would weigh more in practice than it does in our totals. On the changes with bugs the findings rose as well, from 1.8 to 2.4 a change for Sonnet, and we did not judge those.

So: does it help?

Not measurably, in these tests. Requiring agents to state their intent first did not produce better code by the bar we had set, and handing a reviewer those statements did not find more bugs.

Accurate statements look valuable. Tests written from them caught the bug in 11 of 15 cases. Agents that stated the relevant case nearly always handled it. Reviewers given the relevant requirement found far more bugs. What we have not shown is that making an agent write statements causes any of this. The agents that stated the case may be the ones that would have handled it anyway, and our ceiling was written by someone who knew the bug.

What decides it is where the statement comes from. Writing intent down helps only as far as the intent is grounded in something outside the agent: the request, the existing behaviour, a person. Where it is the agent's own guess, the form lends the guess an authority it has not earned, without making it more correct. That accounts for most of what we saw:

• Before the code, a statement is a guess that pins down the right case a third of the time.
• After the code, it describes the bug as the plan.
• A wrong statement becomes a red check for the author.
• A guessed statement becomes a false finding for the reviewer, even when it is labelled a guess.
• The strongest model we tried thought of the cases without the ritual, and the weaker ones did not think of them because of it.

And the ritual is not free. The skill more than doubled what the Claude models cost, and the reviewers' wrong findings on clean code rose about three- to fourfold.

What we are publishing

We are publishing the skill as it is, under the MIT licence, with the checker and these results: rapidreview-dev/skill-intent-statements.

The part we would keep in any case is the checker. For a statement about new behaviour it runs the test with the change and with the change taken out, and the test has to tell the two apart. That is a cheap way to find tests that show nothing about the change they came with, and it needs no model.

If you try the skill on work that comes with real requests, we would like to hear what you find.

What is still open

We tested discovery, and the skill did not deliver it: a structured process did not help an agent find requirements that a one-line request left out.

We could not test preservation. That is the case the idea was made for: an author who has the real request in hand, a ticket or a conversation and not a commit subject, and an agent that writes down what is known instead of what it supposes. Our ceiling shows what a statement can do when it is right. Whether authors and their agents write that statement is open.

If we take this further, we would not make the skill more elaborate. We would look for a better way for the agent to recover what was actually asked, or to ask.