In this video I run the exact same model, Claude Sonnet 5.5, inside two different coding agents: Claude Code and OpenAI's Codex CLI. Same prompt, word for word, on four tests: a Minecraft-style game, a real client job, a 3D sailing ship and a 16-bug hunt. If you pick your coding agent by the model inside it, this one is for you, because the model was the same and the runs were not.
What I cover in this video
- 0:45 What a harness is: the system prompt, the tools, the context and the loop around the model.
- 2:18 What the leaderboards and studies say: the Artificial Analysis Coding Agent Index, Arena's HarnessTax study and the Meta-Harness paper.
- 3:49 My bench setup: one model, the same prompt, fresh folders, hidden tests and third-party judges.
- 4:34 Test one: a Minecraft-style game in Three.js.
- 5:21 Test two: a freelance client job with a requirement that changes mid-thread.
- 6:51 Test three: a wooden sailing ship in Three.js, where Codex got weird.
- 7:36 Test four: 16 planted bugs in a 42-file backend.
- 9:08 The scoreboard, three reasons the runs came out different, and my verdict.
What a harness is, and why it could matter
The model is the brain. It reads text and writes text back. It can't open a file, run your tests or see your folder. The harness is everything around it, and it comes down to four parts:
- The system prompt: the instructions the model gets before you type anything.
- The tools: read a file, edit a file, run a command.
- The context: what stays in memory, what gets summarized and what gets thrown away.
- The loop: the model asks for a tool, the harness runs it and hands back the result, over and over until the job is done.
Claude Code is Anthropic's harness. It has its own read, write and edit tools and can split work off to sub-agents. Codex is OpenAI's harness. It edits files with a patch tool and can run everything inside a sandbox. Cursor's blog adds a detail that comes back later: OpenAI's models are trained to edit code with patches, and Anthropic's models are trained to edit by replacing text. If you want more on the loop itself, I wrote up how the Claude Code agent loop works.
The public numbers disagree on how much the harness matters. On the Artificial Analysis Coding Agent Index, every row is an agent plus a model: Claude Code with Sonnet 5.5 scores 59 (tied for third) and Claude Code with Opus 5.5 scores 62 (first). Arena's HarnessTax study found that switching the harness moved success only a few percent, while cost moved up to five times. Then the Meta-Harness paper ran the same model, Opus 4.6, on Terminal-Bench 2 and got 58.0% in Claude Code, 62.9% in Terminus 2 and 81.8% in ForgeCode. Those are other people's benchmarks, so I ran my own. (For background on reading these, see coding agent benchmarks explained.)
The setup and the four tests
One model, Claude Sonnet 5.5, with the same prompt in a fresh empty folder. Effort high, web search off, no follow-ups. 45 minutes per test and 60 for the bug hunt. Codex ran Sonnet 5.5 through OpenRouter, which Codex supports as a custom provider. Each run was locked in its own sandbox, away from the hidden tests and answer keys. Scoring came from hidden tests plus judge models from labs with no stake in Anthropic or OpenAI. I wanted Cursor in this too, but our Cursor CLI wasn't signed in and I hadn't set up a key. That's on our setup, not on Cursor. And it's one run each, so treat this as a vibe check, not a final verdict.
Test one: the game
Claude Code finished in 10 minutes and 39 seconds. Codex took 13 and a half minutes. Both games are playable. Claude Code's world has flowers, tall grass and water. The bigger difference is how they got there: Claude Code made 31 tool calls (read, write, edit, test). Codex made 12 and wrote the game in big chunks through shell commands.
Test two: the client job
An accounting firm sends a messy email thread about a tax season document intake page, and halfway through one partner changes the rules. The agent has to build the app and write the reply email. Both agents passed 12 out of 12 hidden tests in about two and a half minutes, and both caught the change. Codex's reply got a 10 from both judges. Claude Code got a 10 and a 9 because it left the sign-off as a placeholder.
Test three: the ship
Claude Code finished the ship in 41 minutes. With Codex, there was nothing in its folder at 15 minutes, so our check-in stopped it and restarted it once (same rule for every model). It turned out it was building the ship in pieces in a temp folder, and it copied the finished ship in during the very last minute of the 45-minute limit. Both ships work, and they look like sister ships: three masts, full sails, a wake behind them.
Test four: the bug hunt
A backend invoicing app, 42 files and about 3,000 lines, with 16 planted bugs plus 11 fake ones that look like bugs but aren't. Out of 32 points:
- Claude Code: 13 bugs fixed, 26 out of 32, in 3.5 minutes. No regressions.
- Codex: 14 bugs fixed, 28 out of 32, in 15.5 minutes. No regressions.
The one Claude Code missed was a rounding bug (tax done with floating point math). Both missed the same two hardest bugs: a cache that goes stale in a race, and a refund that never turns payment reminders back on.
Why the same model came out different
Total time was about 58 minutes for Claude Code and 76 for Codex, and Claude Code's total cost was lower too. One caveat: Claude Code's cost is API-equivalent (what the same run would cost on the API), while Codex's is what the OpenRouter runs actually cost. I see three reasons for the gap:
- Thinking. Both were set to high effort, but each harness passes that to the model its own way. On the bug hunt, Codex let Sonnet think about 93,500 tokens and Claude Code about 12,500. That is 7.5x more thinking, 4.4x longer, and one more bug fixed. I tested effort settings on their own in this effort levels post.
- Tools. Sonnet never used Codex's patch-style edit tool, not once in four tests. It wrote every file through shell commands. In Claude Code it used Claude Code's own read, write and edit tools, and that edit tool works by replacing text. Could the patch style slow Sonnet down in Codex? Maybe. With one run each, that's a hunch, not proof.
- Missing metadata. On every run with Sonnet 5.5, Codex printed a "model metadata not found, defaulting to fallback metadata" warning, and noted that this can degrade performance.
Key takeaways
- You don't pick a model. You pick a model plus a harness.
- In my runs, switching the harness barely moved the scores. It changed the time, the cost and how the work got done, which is pretty much what Arena found too.
- With Sonnet 5.5, the harness built for it (Claude Code) was faster on three of four tests and cheaper overall. Codex won the bug hunt by two points, probably because it let the model think longer.
- On a normal client job it was basically a tie: 12 out of 12 for both in about two and a half minutes.
- Opus 5.5 in Claude Code is still my daily driver.
Full video transcript
0:00Same model, same prompt, two different coding agents. In Claude Code, Sonnet 5.5 fixed 13 hidden bugs in three and a half minutes. In OpenAI's codeex, the exact same model took over 15 minutes and fixed 14. Everybody picks a coding agent by the model inside it, but the model was the same here, and the results weren't. So, in this video, I'm running Sonnet 5.5 inside both agents on four tests. a game, a real client job, a viral 3D ship, and a 16 bug hunt. Then the scoreboard and why they came out different. I run coding agents every single day, so I had to know. Let's jump right into it. So, what is a harness exactly? The model is the brain. It
0:45reads text and writes text back. That's it. It can't open a file. It can't run your tests. It can't even see your folder. The harness is everything around the brain. And it comes down to four parts. A system prompt, which is the instructions the model gets before you type anything. The tools, things like read a file, edit a file, run a command. The context, so what stays in memory, what gets summarized, and what gets thrown away. And then the loop. The model asks for a tool. The harness runs it, hands back the result, and the model goes again over and over until the job is done. Claude code is Anthropic's harness. It has its own read, write, and edit tools, and it can split work off to sub agents. Codeex is OpenAI's harness.
1:30It edits files with a patch tool, and it can run everything inside a sandbox. And Cursor put it pretty simply on their blog. The harness and the model together decide how good the agent is. They also said something I want you to remember. OpenAI's models are trained to edit code with patches, and Anthropics models are trained to edit by replacing text. Keep that in your back pocket. It comes back. Now, I wanted cursor in this, too, but our cursor command line tool wasn't signed in, and I hadn't set up a key for it, so it couldn't run on its own. That's on our setup, not on cursor. So, this one is claude code versus codeex. Two harnesses, one brain, and four tests to see if it even matters. So, before I show you what the leaderboards say about all this, EasyFlip got 460,000 views in
2:1811 days. Nightcrew made one new short every day and posted it to four platforms. The best one hit 155,000. Want that for your business? Nightcrew.studio. So, artificial analysis has a coding agent index. And every row on it is an agent plus a model, not just a model. They say it straight up. Each row is an agent variant. Sonnet 5.5 in Claude Code scores a 59, tied for third. Opus 5.5 in Claude Code is first with a 62 and they have one pair where the same model family runs in two harnesses. Deepseek V4.1 Flash 47 in Codeex, 46 in Deepseek's own harness. Basically a tie, but it flips depending on the test. Codex wins on one, Deepseek's harness
3:04wins on the other. Then there's a study from Arena called harness tax. Seven models run Claude codeex and a third harness called PI. Switching the harness only moved success by about two to 5%. But the cost moved up to five times, and the model's own harness wasn't even the best one most of the time. A different harness won 9 out of 12 matchups. The lead author's post on that has over 360,000 views. But then there's this paper, Meta Harness. Same model, Opus 4.6, six. Same benchmark, Terminal Bench 2, 58% inside Claude Code, almost 63 in a harness called Terminus 2, and almost 82 in one called Forge Code. That's almost a 24 point swing, and nothing
3:49changed but the harness. So, which is it? Does the harness barely matter, or is it worth 24 points? Those are other people's benchmarks on tasks the models may have seen in training. So, we ran our own. So, here's how it works. One model, Claude Sonnet 5.5 with the same prompt word for word in a fresh empty folder. High effort web search off and no follow-ups. 45 minutes per test and 60 for the bug hunt. Cloud code ran on my subscription. So the cost you'll see is what the same run would cost on the API. Codeex ran the same sonnet 5.5 through open router which Codex officially supports as a custom provider. So that cost is what we actually got build and neither agent could peak. Every run was locked in its
4:34own sandbox away from the hidden tests and the answer keys. The scoring comes from hidden tests and two third party judges models from labs that have nothing to do with anthropic or open AI. And to be fair, it's one run each. So this is a vibe check, not a final verdict. Let's go. Test one. Build a Minecraft style game in 3.js. Walk around, break blocks, place blocks, a hot bar, a sky, trees, all of it. Claude Code finished in 10 minutes and 39 seconds. $137 API equivalent. Codeex took 13 1.5 minutes and 176. And here they are with the exact same inputs played into both. You can see Claude Codes world has flowers, tall grass, water, the whole thing. Codeex's works,
5:21too. blocks break, blocks place, and then it walks us right into a tree. Which one looks better to you? Now look at how they got there. Claude Code made 31 tool calls. It read, it wrote, it edited, it tested. Codeex made 12. It wrote the game in big chunks through shell commands. Same brain, totally different way of working. Test two is the one I actually care about, a real freelance job. An accounting firm sends a messy email thread about a tax season document intake page. Halfway through, one partner changes the rules. Returning clients don't have to send their photo ID or last year's return anymore. And there's one thing the partners never agree on. Married couples with two W TWS. The agent has to build the app and
6:06write the email back to the client. Then 12 hidden tests click through it, one for every requirement, including the one that changed. Claude Code 12 out of 12 in 2 and 1/2 minutes. Codeex 12 out of 12 in 2 and 1/2 minutes. Both caught the change. The reply emails, Codeex got a 10 out of 10 from both judges. Claude Code got a 10 and a nine because it left the sign off as a placeholder. Your name in brackets. Come on now. So on a normal client job, it was basically a tie. Same model, same app. Honestly, you could put these two side by side and barely tell them apart. Test three is a redo of something that blew up on X. A creator posted two 3D ships, Sonnet 5.5 in
6:51Claude Code against Gemini 4 Hydrogen in Cursor and said came out better, but that was two different models. The prompt was never shared, so we wrote our own. A wooden sailing ship in 3.js, waves, sails moving in the wind, a camera button. Cloud code took 41 minutes, 233 API equivalent, and it finished. Codeex is where it got weird. At 15 minutes, there was nothing in its folder. So, our check-in stopped it and restarted it once. Same rule for every model. And it turned out it was building the ship in pieces in a temp folder somewhere else. It copied the finished ship in during the very last minute, right as the 45minute clock ran out, about 450 on open router. Now, this is
7:36the part I didn't expect. Both ships work. Same inputs, same camera switches, and they look like sister ships. Three masts, full sails, awake behind them. His two models got different results. Our one model in two harnesses built the same kind of ship. Test four is the hard one. A real back-end code base. 42 files, about 3,000 lines, an invoicing app for small agencies. We planted 16 bugs in it, plus 11 fake ones that look like bugs but aren't. And the agents were never told how many there were. Hidden tests decide what's actually fixed. Two judges, Kimmy K 3.1 Pro and Deepseek V4 Pro, decide what's explained and take points off for false alarms. 32
8:21points max. Clog code 3 1/2 minutes. 13 bugs fixed. Nothing broken. 26 out of 32. 73 cents. Codeex 15 1.5 minutes 14 fixed nothing broken 28 out of 32 195 the one claw code missed was a rounding bug tax at 6.35% done with floating point math so the pennies drift and both of them missed the same two hardest bugs a cash that goes stale in a race and a refund that never turns the payment reminders back on both judges agreed on every score. So what's actually going on under the hood? Here's the scoreboard. First, Claude Code was faster on three of the four tests, and it finished the ship on time. Codex won the bug hunt by two points.
9:08The client job was a tie. Total time 58 minutes for Claude Code, 76 for Codeex. Total cost, 482 API equivalent against about 870 for Codeex. Reason number one, thinking. We set both to high effort, but each harness passes that to the model its own way. On the bug hunt, Codeex let Sonnet think about 93,000 tokens. Claude Code about 12 a.5 thousand. So 7 and a half times more thinking. Four times longer and one more bug found. You're paying for that thinking in time and money. Reason number two, the tools. Remember the patch thing from Cursor? Codeex's own edit tool is a patch tool and Sonnet never used it. Not once in any of the
9:54four tests. It wrote every single file through shell commands instead. In Claude Code on the game, it used Claude Code's own read, write, and edit tools. And that edit tool works by replacing text, the style cursor says anthropic trains for. Could the patch style slow sonnet down in Codeex? Maybe. With one run each, that's a hunch, not proof. And reason number three, on every single Codeex run, Codeex printed a warning. Model metadata for Sonnet 5.5 not found. Defaulting to fallback metadata. This can degrade performance. So Codeex was basically driving a car it didn't have the manual for. So let's answer the four questions. Can it make a game? Yes, in both. In our run, Claude Code was faster
10:39and to me its world looked better. Can it hunt bugs? Yes. 13 or 14 out of 16 with zero false alarms. Codex found one more. Probably because it let the model think longer. What does this mean for AI coders? You don't pick a model. You pick a model plus a harness. In our test, switching the harness barely moved the scores. It changed the time, the cost, and how the work got done, which is pretty much what Arena found, too. So, with Sonnet, in our runs, the harness built for it got there faster and cheaper. Opus 5.5 and Claude Code is still my daily driver. And how would you use it in a business? That's the client job. 12 out of 12 in 2 and 1/2 minutes. Either way, in my resale business, that's an intake page for when a shipment lands or a form my team
11:25actually uses built before my coffee gets cold. So, what are you running your model in right now? The agent its own lab built or somebody else's? And did switching just the agent ever change your results? Tell me down below. And if this made you rethink your setup, give it a like so the next person running agent sees it. Have a blessed day.