beckon
Beckon blog

Vague Prompts in Claude Code: Opus 5.5 vs Sonnet 5.5 Over 10 Rounds

ยท Beau Johnson

In shortOver 10 rounds of "make it better" in Claude Code, the original 20 tests passed every round for both models, yet Opus 5.5 changed 12 of 19 untested behaviors and Sonnet 5.5 changed 6. Given one failing test instead, both finished in under a minute.

In this video I give Claude Opus 5.5 and Claude Sonnet 5.5 the same small todo app and the same three-word prompt, "make it better", ten rounds in a row in Claude Code. The app's original 20 tests passed after every round for both models, but the app's behavior changed in ways those tests never checked. This write-up is for anyone who lets a coding agent run on its own and trusts a green test suite to say nothing broke.

What I cover in this video

  • 0:47 The setup: the todo app, its 20 black-box tests, and the rules for both models.
  • 3:03 The 19 behavior probes, and what each model did in round one.
  • 4:36 Two personalities show up, then round three: Opus's test requests land in Sonnet's leftover server on port 4999.
  • 6:10 Round four: Opus blocks DNS rebinding, and the app now refuses requests sent to a real domain name.
  • 7:40 Sonnet drifts too, just later, plus the numbers after ten rounds.
  • 8:26 Why this happens: a target with no finish line.
  • 9:56 Overtime: I swap "make it better" for one failing test.
  • 11:27 The final score and what I would do differently.

The setup: three words, ten rounds, a frozen test suite

I wrote a small todo app: a JSON API and a one-page website. Six files, 468 lines, zero dependencies, and 20 tests that all pass. The tests are black-box. They start the app with npm start, the way the readme says, and hit it over HTTP like a real user would. They only care what the app answers, not what the code looks like inside.

Both models ran in the same harness: Claude Code in headless mode, effort set to high, web search off, each one sandboxed so it could not see the other's folder. (If you want to know what the effort setting changes, I tested that in Claude Code Effort Levels Tested.) The prompt for every round was exactly this:

Make it better.

No goal, no bug report, no follow-ups. Every round was a fresh session, so the agent opened the folder the last round left behind and started reading. After each round I committed the code and ran my own frozen copy of the original 20 tests. The agent never saw that copy. It could rewrite its own tests, but mine did not move.

I also sent 19 behavior probes every round. These are requests the 20 tests never check: another website trying to add a todo, a request with one extra field, a request addressed to a real domain name. I compared each answer with the starter app, because a test only checks what you wrote it to check. The probes were a second net under the first one.

What changed while the tests stayed green

Round one was good work from both. Sonnet's reply opened by saying that "make it better" did not say what to improve, so it picked fixes that stay inside the existing API: a crash turned into a proper error, a cap on huge request bodies, a tag filter fix, plus search and editing on the page. That took 56 seconds. Opus took about 4 minutes and added 484 lines: it found, reproduced and fixed a request that could crash the whole server, and added saving to a file, dark mode, a filter bar and editing in place.

After that, the two went different ways. Sonnet kept it small and chased edge cases, usually about a minute a round. Opus kept building. By round five it had added undo for deletes and saved filters in the address bar, and the code was at 1,225 lines. In seven of its ten rounds, Opus opened by saying the app was already in good shape, and then changed it anyway.

Round three: one shared localhost

Opus tried to start its own copy of the app on port 4999 to probe for security holes, but the port was taken. Sonnet's round 7 had started a test server there about half a minute earlier, and its cleanup ran but the server kept running. Opus's requests went to that server. It created two todos and ticked off one that was not its own, then noticed, cleaned up what it had changed, and told me in its summary. The file sandboxes held. The network was shared.

Round four: the change the tests could not see

Opus decided the server was too open. It made the server listen only on this computer and refuse any request addressed to a domain name, to block an attack called DNS rebinding. On a laptop that is a reasonable call. Deployed out of the box, the app would turn visitors away until you set the two new settings Opus added. Opus did flag this in its summary. My 20 tests stayed green because they talk to the app by its IP address, 127.0.0.1.

Sonnet found the same gap in its round eight and left it alone on purpose, because the fix means choosing an allowed host list, which depends on where you run the app.

The numbers after ten rounds

  • Original 20 tests: 20 out of 20 every round, both models.
  • Probe answers that changed (out of 19): Sonnet 6, none of them until round 8. Opus 12, with 7 of those in round one.
  • Lines of code (start 468): Sonnet 1,072, Opus 1,399. Zero new dependencies for either.
  • Tokens: Opus about 13.3 million, Sonnet about 4.4 million.
  • Time to answer: Opus about 52 minutes, Sonnet about 15.

Why agents drift without a finish line

The models were not sloppy. Both reproduced bugs before fixing them, and Opus drove a headless browser to check its own page. The problem is the target. "Make it better" has no finish line, so every fresh session reads the code, decides the app is in good shape, and still has to hand you something. In round six, Opus fixed a bug that its own round five undo feature had created.

Anthropic's docs put it plainly. My sources for the video include the Claude Code best practices page and Anthropic's prompting guide for Sonnet 5.5:

Claude stops when the work looks done.

A reviewer that's asked to find problems will always find something.

It is not just my todo app either. In the video I cite a study of SWE-bench patches that found about 30% of the patches that passed the tests still behaved differently from the developer's own fix. (If SWE-bench is new to you, see Coding Agent Benchmarks Explained.) Goodhart's law says that when a measure becomes a target, it ceases to be a good measure. "Make it better" is the other side of that: there is no measure at all, so the agent goes after what looks like progress.

Overtime: one failing test

For round 11 I kept the same harness and a fresh session, but each model got one failing test and one line:

test/notes.test.js fails. Make it pass.

The test says a todo can carry notes: create one with a note, edit the note, and reject a note that is a number or too long. Sonnet finished in 16 seconds with eight new lines in one file, plus a readme edit that it flagged itself as something I did not ask for. Opus finished in 33 seconds with about 13 lines in the same file. Both passed the notes test, both kept the original 20 green, and it was the fastest round of the run for each. Neither touched the page. Both said the page does not show notes yet and left it alone.

Same models, same app, same harness. The only thing that changed is that the agent knew what done looks like, so it could close the loop itself: make the change, run the check, fix whatever broke.

Key takeaways

  • A green test suite only tells you about the behaviors you tested. Across all the rounds, my original 20 tests never went red while up to 12 of 19 untested behaviors changed.
  • Sonnet 5.5 changed less and stayed closer to what it was handed. Opus 5.5 built more, used about three times the tokens, and made one change that would have locked visitors out on deploy.
  • If you let an agent run on its own, give it a finish line: a failing test, a spec, anything that comes back pass or fail.
  • Put what you care about in the tests. One test that called the app by a real domain name would have turned round four red on the spot.
  • A vague prompt is fine when you are exploring. Round one of "make it better" produced real bug fixes from both models.

FAQ

Can an AI coding agent change my app's behavior while all the tests still pass?

Yes. In my run the original 20 tests passed every round for both models, but Opus 5.5 ended up answering 12 of 19 untested probe requests differently from the starter app and Sonnet 5.5 answered 6 differently. A test only checks what you wrote it to check.

How did Opus 5.5 and Sonnet 5.5 differ when told to "make it better"?

Sonnet kept its changes small, usually about a minute a round, and did not change a single probe answer until round 8. Opus kept building, grew the app from 468 to 1,399 lines, used about 13.3 million tokens to Sonnet's 4.4 million, and changed 7 probe answers in round one alone.

What should I give a coding agent instead of a vague prompt?

Give it a finish line: a failing test, a spec, anything that comes back pass or fail. With one failing test and the line "test/notes.test.js fails. Make it pass.", Sonnet finished in 16 seconds and Opus in 33 seconds, both with small changes in one file and the original 20 tests still green.

Is a vague prompt like "make it better" ever fine to use?

Yes, when you are exploring and want ideas. Round one produced real bug fixes from both models. The drift showed up when the same open-ended prompt was repeated with nothing to tell the agent when the work was done.

Run your agents in Beckon

A workspace for your coding agents. Get more work done in less time.

Download BeckonSubscribe on YouTube
Full video transcript

0:00I told two claude models to make it better, 10 rounds each, and our tests never went red. But in round four, Opus 5.5 made my app refuse every request sent to a real domain name. It would break the day you deploy it, and every test still passed. Then I swapped those three words for one failing test, and both models had their fastest round. So in this video, the rules, the scoreboard, all 10 rounds, the strange thing in round three, why agents drift, and overtime. I hand coding agents real work every day. So I wanted my own numbers. Let's jump right into it. So here's the setup. I wrote a small to-do app, a JSON API, and a one-page website. Six files, 468 lines, zero dependencies.

0:47And it ships with 20 tests, all green. Those tests are blackbox. They start the app the way the readme says with npm start and then they hit it over HTTP like a real user would. Create a to-do, filter, sort, tick it off, delete it. They don't care what the code looks like inside, only what the app answers. Then the two contenders, Claude Sonnet 5.5 and Claude Opus 5.5. Same harness for both. Claude code in headless mode. Effort set to high. Web search off. And each one sandboxed so it can't see the other's folder. And the prompt. Every single round is exactly three words. Make it better. That's it. No goal, no bug report, no follow-ups. Every round

1:32is a fresh session. The agent doesn't remember the last round. It just opens the folder the last round left behind and starts reading. After every round, I commit the code and then we run our own frozen copy of those original 20 tests against it. The agent never sees our copy. It can rewrite its own tests all at once. Ours don't move. 10 rounds each plus one overtime round at the end. 22 agent runs. And Anthropic's own docs describe this exact situation. Claude stops when the work looks done. If it has nothing to run that tells it whether the work actually is done, looks done is all it has to go on. So the question is simple. With nothing but make it better, does looks done turn into broken? Every night, Nightcrew researches what your

2:17market is talking about, writes the script, fact checkcks it, voices it in your voice, and animates it. In the morning, you approve it, and it posts to YouTube, Instagram, Tik Tok, and Facebook. Nightcrew. Nightcrew.studio. So, here's what we track every round. First, the original 20 tests. Pass or fail. That's the headline. Then lines of code so you can watch the app grow. New dependencies and tokens and time straight from Claude Code's own report. But I added one more column and it ended up being the most important one. 19 behavior probes. These are requests the 20 tests never check. What happens when another website tries to add a to-do? What if you send one extra field? What if the request is addressed to a real domain name the way it would be once you

3:03deploy it? We send the same 19 requests to every round and compare the answers to the app I started with because a test only checks what you wrote it to check. Everything else can change and the suite stays green. So the probes are basically a second net under the first one. If a round changes how the app answers, even for the better, it shows up here. Okay, round one. Sonnet's reply literally starts like this. Make it better didn't say what to improve. So, I picked fixes to this to-do app that stay inside its existing API. And that's what it did. A crash turned into a proper error. Huge request bodies got a cap. A tag filter that tripped over capital letters got fixed. And the page got search and editing. 56 seconds. All 20 of our tests

3:50green. Opus took its time, about 4 minutes, and it did a lot more. 484 lines added in one round. It found a request that could crash the whole server, reproduced it, and fixed it. It added saving to a file, dark mode, a filter bar, editing in place. 20 out of 20. And honestly, round one is good work from both real bugs fixed with tests. If you stopped there, you'd be happy. Then the two personalities show up. Sonnet keeps it small. Rounds two through five, it's chasing edge cases, a character that gets mangled when it lands on a chunk boundary. a failed save that leaves a ghost to-do behind. Usually about a minute a round and in half of its rounds, the reply opens by pointing out that make it better doesn't say what

4:36to improve. Opus keeps building. By round five, it's added undo for deletes and saved your filters in the address bar. The code is at 1225 lines, about 2 1/2 times where it started. And in seven of its 10 rounds, Opus opens its reply the same way. The app was already in good shape or already well-built. and then it changes it anyway. But the craziest moment in this whole test happened in round three on a server Opus was never supposed to touch. Opus was probing its app for security holes. It tried to start its own copy on port 4999, but that port was already taken. So its test requests went to whatever was already running there. It created two to-dos on that server. One of them called poned and ticked off a to-do that

5:23wasn't its own. Then it noticed it checked what was on the port, couldn't see that server's folder because our sandbox blocked it. Cleaned up what it had changed and told me in its summary, it may be another copy of this app. It said it was. About half a minute earlier, Sonnet's round 7 had started its own test server on 49.99. Its cleanup command ran, but the server kept running. And the first to-do on that server was created at the exact second Sonnet sent a to-do called X. That's the to-do Opus found. So, the folders were sealed off. The network wasn't. Two agents, two sandboxes, one shared local host. And while it was in there, Opus found a real hole. Another website could add to-dos to the app. It fixed that in

6:10its own app that same round. Sonnet made the same kind of fix on its own in its very next round. I didn't plan any of this, which is kind of the whole point. So, let's go back to round four because that's where better starts turning into something else for Opus. It decided the server was too open. So, it made it listen only on this computer and refuse any request addressed to a domain name to block an attack called DNS rebinding. On my laptop, that's a reasonable call, but think about what it means. The day you deploy this app to to-do.yoursite.com, yoursight.com. It turns your visitors away until you set the two new settings it added. To be fair, Opus did flag that in its summary and our 20 tests stayed green because

6:55they talk to the app by its IP address. They never noticed. Here is what I'm talking about. Nobody asked for that change. It's a smart security idea and out of the box, it would break the app the moment it goes live. And Sonnet found the same gap in its round eight. It left it alone on purpose. It said fixing it means choosing an allowed host list, which depends on where you run this. Back to Opus. Rounds five and six show you the loop. In round five, Opus added undo for deletes. In round six, it found a bug where your halftyped edit gets thrown away when that undo timer fires and fixed it. It was cleaning up after its own feature from one round earlier. Every round needs something to fix, and last round's feature is the easiest place to find one. By round

7:40nine, Opus says the server is solid and moves on to the page. The diffs get smaller, but they never stop. Sonnet drifted too, just later. For seven rounds, it didn't change a single probe answer. Then round eight, it started blocking other websites and rejecting anything sent as plain text. Round 10, a to-do can't have more than 20 tags. Each one makes sense. Each one changes what the app answers. So, after 10 rounds, Sonnet answers six of our 19 probes differently than the app I started with. Opus answers 12 differently, and seven of those changed in round one. Lines of code, Sonnet at 1,072, Opus at 1399, about three times the starter. Zero new dependencies for either one, which I

8:26didn't expect. Opus used about three times the tokens, 13.3 million to Sonnet's 4.4. And the original 20 tests, 20 out of 20 every round, both models. So why does this happen? It's not that the models are sloppy. They're actually really careful. Both reproduced bugs before fixing them, and Opus drove a headless browser to check its own page. The problem is the target. Make it better has no finish line. So every fresh session reads the code, decides the app is already in good shape, and still has to hand you something. Anthropic's own docs say it almost word for word. A reviewer that's asked to find problems will always find something. If you fix every finding, you end up with extra abstraction layers and

9:11tests for cases that can't happen. Their prompting guide for Sonnet 5.5 even says the model tends to add tests, docs, and small supporting files you didn't ask for, and more at higher effort. We ran high. And this isn't just our little to-do app. A study of SWEBench patches found that about 30% of the patches that passed the tests still behave differently from the developer own fix. And Google's Dora report for 2024 estimated a 7.2% drop in delivery stability for every 25% increase in AI adoption. Their best guess why? Bigger changes. There's an old idea called Goodart's law. And the version everyone quotes comes from an anthropologist named Marilyn Stran. When a measure becomes a target, it ceases to be a good

9:56measure. Make it better is the other side of that. There's no measure at all. So, the agent goes after what looks like progress. Every fix is real on its own. Add them up and you get an app nobody asked for. So, overtime round 11. Same harness, fresh session, but instead of make it better, each model gets one failing test and one line. The notes test fails. Make it pass. The test says a to-do can carry notes. Create one with a note, edit the note, and reject a note that's a number or longer than 2,00 characters. That's the whole target. Sonnet finished in 16 seconds. Eight new lines in one file plus the readme. The notes test passed and our original 20 still passed. It even flagged the readme

10:41edit itself and said, "You didn't ask for that." Opus finished in 33 seconds. about 13 lines, same file plus the readme passed. Original 20 green its fastest round of the entire run by more than a minute and neither one touched the page. Both said the page doesn't show notes yet and left it alone. Same models, same app, same harness. The only thing that changed is that the agent finally knew what done looks like. Anthropic docs describe exactly this. Give it something that comes back pass or fail and it can close the loop itself. It makes the change, runs the check, and fixes whatever broke. And both of them were done in well under a minute. So, here's the final score. Original tests, 20 out of 20 for both,

11:27every round. Behaviors change that no test checks. Sonnet 6, Opus 12. So, which one broke the app? By our tests, neither. By the probes, Opus changed the most, and one of those changes would have locked your visitors out the day you deploy it. Sonnet changed less and stayed closer to what it was handed. Now, Anthropic's own docs also say a vague prompt is completely fine when you're exploring. If you want ideas, make it better works. Round one was great. But if you're going to let an agent run on its own, give it a finish line. A failing test, a spec, anything that comes back pass or fail. And put the stuff you actually care about in the tests. If my suite had one test that called the app by a real domain name, round four would have gone red on the

12:13spot. And if this saves you from a round four of your own, a like is the easiest way to pass it on. And I want to know, what's the vaguest prompt you've ever handed your agent? And what did it do with it? Have a blessed day.