In this video I put GLM-5.3, an open weights model from Z.ai, inside Claude Code and raced it against Claude Sonnet 5.5 in the exact same harness: same prompt, same repo, same tools, max effort, zero nudges. If you run AI coding agents every day and want to know whether Claude Code is good because of Claude or because of the harness, this is the test I ran to find out.
What I cover in this video
- 0:00 An open model inside Claude Code, and the headline results
- 0:45 Harness vs model: what each one actually does
- 2:15 The setup: three settings to swap the brain, plus the rules of the race
- 3:47 The race: a CSV export ticket with 8 tests
- 5:17 The scoreboard (medians of three runs)
- 6:03 Where the open model stumbled: the game build and the bug hunt
- 8:22 The independent numbers from Artificial Analysis and Terminal-Bench
- 9:55 The verdict: what you can hand an open model today
Harness vs model
The harness is everything around the model. It's the loop that reads your files, runs your commands, edits the code and decides what to send back to the model next. Claude Code is a harness. Codex is a harness. The model is the brain inside it: it reads what the harness hands it and decides what to do next. Think of a race car. Same car, same pit crew, and we only swap the driver.
There's real data on both sides of this argument. On the official Terminal-Bench 2.0 leaderboard, the same Claude model ranged from 58% in Claude Code to 76% in a different harness. Same brain, 18 points apart. But Anthropic's own engineers also showed that just the machine setup can move a score by six points. So the harness matters, and the question is how much. If the harness is doing most of the work, an open model in the same harness should get you most of the way there, and you could keep the setup you already have and swap in a different brain.
How I put GLM-5.3 inside Claude Code
It's basically three settings:
ANTHROPIC_BASE_URL: where Claude Code sends its requests
ANTHROPIC_AUTH_TOKEN: your token
- The model name, which I set to GLM-5.3
I pointed it at OpenRouter, which has an endpoint that speaks the same language as Anthropic's. Z.ai documents this with a whole Claude Code page in their docs, and OpenRouter documents it too. One thing to pay attention to: Anthropic does not support this. Their docs say they don't support routing Claude Code to non-Claude models, and OpenRouter says it's only guaranteed with Anthropic's own provider. It works, but you're on your own.
The rules were strict: the same prompt word for word, a fresh copy of the repo for every run, max effort for both, web search off, auto approve inside its own folder, and zero nudges. One prompt, then hands off. I checked the logs, and every answer in the GLM runs came from GLM, not from Claude in the background. Two caveats: Claude Code didn't recognize GLM, so it assumed a 200,000 token context window instead of a million (neither run got close), and this was GLM via OpenRouter, not Z.ai's own servers.
The race: a CSV export ticket
The main test was a small orders API with a ticket for a CSV export. Finance wants to download their orders into a spreadsheet, and it's harder than it sounds. Commas and quotes have to be escaped correctly, amounts in cents come out as dollars with two decimals, refunds are negative, the date filter has to include the whole last day, it has to pull every order and not just the first page, and text starting with an equal sign gets neutralized for spreadsheet safety while a negative number does not. Eight tests, and before scoring I put the original test files back, so editing a test earns nothing.
GLM read the readme, the tests and every source file, wrote the CSV module, the route and the date parsing, ran the tests, poked at a few edge cases on its own, and was done in 1:33. Sonnet read everything, wrote the export, ran the tests, then kept checking: it spun up the server, hit the endpoint with real requests and looked at the raw bytes. It finished in 5:15. Both went 8 for 8, and that held across all three runs each.
The scoreboard (medians of three runs)
- Tests passed: 8/8 for both, every run
- Time: 1:33 for GLM vs 5:15 for Sonnet
- Output tokens: 17,304 vs 45,268
- Tool calls: 26 vs 32
- Errored tool calls: 0 vs 1
- Malformed tool calls: 0 across all six runs
- API-equivalent cost: GLM came in at roughly a tenth of Sonnet's
Tied on tests is not the same as tied on quality, though. Sonnet spent that extra time testing things nobody asked it to test. Sometimes that's exactly what you want, and sometimes you just want the ticket closed.
Where the open model stumbled
These next two were one run each, so treat them as a vibe check, not a verdict.
The game. Both got the same Subway Surfers prompt. GLM had a working game saved in under 9 minutes, then did something smart: it opened the game in a headless browser, took a screenshot and asked Claude Code to show it the picture. That's where the run died. GLM-5.3 is text only, and OpenRouter has no endpoint for it that takes images, so the request came back with "No endpoints found that support image input." That's the whole harness question in one moment: Claude Code assumed the model could see. Sonnet used the full 45 minutes and was still polishing at the limit. Both games play with the same inputs, and GLM's is simpler.
The bug hunt. An invoicing backend with 42 files, 16 seeded bugs and 11 decoys, and the models were never told the count. Hidden tests decided what was fixed, and two outside judges, Gemini 4 Argon and Grok 4.7 (neither from Anthropic or Z.ai), decided what was explained against a hidden answer key. GLM finished in about 8 minutes with 9 bugs fixed and explained, zero false reports and nothing broken. That's clean work, but it missed seven: the double charge race, tax rounding, email normalization, a regex that can hang the server, a double finalize race, the cache race and the refund that never turns reminders back on. Sonnet used the whole hour and fixed 15 of 16. Out of 32 points, GLM scored 18 from both judges, and Sonnet scored 30 and 27, because the judges split on three extra issues it reported.
The independent numbers
On the Artificial Analysis Intelligence Index, Sonnet 5.5 scores 54 and GLM-5.3 scores 32, both at max effort. Sonnet writes about 139 tokens a second to GLM's 76, but used about 340M tokens to run the whole index versus GLM's 140M. That matches what I saw: Sonnet writes more and checks more. On the official Terminal-Bench 4.0 leaderboard, with both run inside Claude Code at max effort, Sonnet scored 61.8% and GLM 41.8%. GLM lands around 40 in every harness it's been measured in, so Claude Code didn't give it a boost there. Z.ai's own numbers are vendor reported and never compare against Sonnet 5.5.
If you're running several agents like this side by side, that's what I built Beckon for: every coding agent on one canvas, press a key and talk to them, and every idea gets its own workspace with memory that carries over.
Key takeaways
- On a well scoped ticket with tests in a codebase it can read in one sitting, GLM-5.3 in Claude Code kept up with Sonnet 5.5: faster, fewer tokens, nothing broken and no malformed tool calls.
- On the hard bug hunt, the model made the difference. Same tools, but Sonnet knew where to look.
- Most of what GLM missed involved time: two requests landing at once, a cache filling while something clears it, a refund that should restart a reminder. Those are the bugs that cost you money.
- Swapping models takes three settings, but Anthropic doesn't support it, and a text-only model can break harness features that assume the model can see.
- How I'd split it: new exports and reports go to GLM, anything that touches payments stays on Claude, and Opus 5.5 is still my daily driver.
Full video transcript
0:00I plugged an open weights model into cloud code. And on a real feature build, it matched sonnet 5.5, eight tests for 8 in less than a third of the time. Same prompt, same repo, same tools. But one of these is not running Claude. It's GLM 5.3 from Z.AI. Three runs each and every single run eight for eight. But then we handed both of them a 16 bug codebase and that's where the gap showed up. nine bugs fixed versus 15. So, in this video, we're breaking down the harness versus the model, the race, the scoreboard, where the open model stumbled, and if you can hand it real work today. Let's jump right into it. So, what is a
0:45harness exactly? The harness is everything around the model. It's the loop that reads your files, runs your commands, edits the code, and decides what to send back to the model next. Claude code is a harness. Codeex is a harness and the model is the brain inside of it. It reads what the harness hands it and decides what to do next. So think of it like a race car. Same car, same pit crew and we only swap the driver. The driver is the brain. And builders argue about this all the time. Is Claude code good because of Claude or is Claude good because of Claude code? Here's something most people don't think about. You can actually test that. Claude code lets you point it at a different model. So, we kept the harness exactly the same and only swapped the
1:30brain. And there's real data on both sides. On the official Terminal Bench 2.0 leaderboard, the same Claude model ranged from 58% in Claude code to 76% in a different harness. Same brain, 18 points apart. But Anthropic's own engineers also showed that just the machine setup can move a score by six points. So, the harness matters. The question is how much? Because if the harness is doing most of the work, then an open model inside the same harness should get you most of the way there. And if you're running coding agents all day, that's a big deal because it means you could swap in a different brain and keep the setup you already have. So that's exactly what we tested. Still juggling cloud code, codecs, and cursor
2:15in five terminal windows. Beckin puts every coding agent on one canvas side by side. Press a key and just talk to them. Every idea gets its own workspace with memory that carries over. Becken. Hey, Becken. So, here's how you put a different brain inside Claude Code. It's basically three settings. A base URL, which is where Cloud Code sends its requests, a token, and a model name. We pointed it at Open Router, which has an endpoint that speaks the same language as anthropics, and we set the model to GLM 5.3. Z.AI actually documents this. They have a whole cloud code page in their docs. Open Router documents it too. Here's something to pay attention to. Anthropic does not support this. Their docs say they don't support routing cloud code to
3:01non-Cloud models. Open Router says it's only guaranteed with Anthropic's own provider. So, it works, but you're on your own. Now, the rules. Same prompt word for word. A fresh copy of the repo for every run. Max effort for both. Web search off. auto approve inside its own folder and zero nudges. One prompt, then hands off. And we checked the logs. Every single answer in the GLM runs came from GLM, not from Claude in the background. Two things to know. Claude Code didn't recognize GLM, so it assumed a 200,000 token context window instead of a million. Neither run got close to that. And GLM went through Open Router, so it's GLM via Open Router, not Z.AI's
3:47own servers. So, the main test, it's a small orders API, and the ticket is a CSV export. Finance wants to download their orders into a spreadsheet, and it's not as easy as it sounds. The commas and quotes have to be escaped the right way. Amounts in cents have to come out as dollars with two decimals. Refunds are negative. The date filter has to include the whole last day. It has to pull every order, not just the first page. And there's a spreadsheet safety rule where text that starts with an equal sign gets neutralized but a negative number does not. Eight tests and we didn't let them cheat. Before scoring, we put the original test files back. So editing a test gets you nothing. So let's run it. You can see
4:32GLM reads the readme, reads the tests, reads every source file, and then it just starts writing a CSV module, the route, the date parsing. It runs the tests 17 out of 17. Then it pokes at a few edge cases on its own and it's done. 1 minute 33 and Sonnet is still going. Sonnet reads everything, writes the export, runs the tests, and then it keeps checking. It spins up the server. It hits the endpoint with real requests. It looks at the raw bytes. It even tried a cleanup command that Claude code blocked for safety. 5 minutes 15. Also 8 out of eight. And we did this three times each. Every single run, both models, eight out of eight. So here's
5:17the scoreboard. And these are the medians of three runs. Tests passed. 8 for8 on both sides. Time 133 for GLM, 515 for Sonnet. That's almost three and a half times longer. Output tokens about 17,000 versus 45,000. Tool calls 26 versus 32. Errors zero versus one. And malformed tool calls, the thing everybody worries about with open models and cloud code, zero. Zero across all six runs. And the API equivalent cost of these runs, about 9 cents for GLM and about 89 cents for sonnet. So roughly a tenth. I mean, come on. On the kind of ticket most of us hand to an agent every single day, the
6:03open model tied. Here's something to pay attention to, though. Tied on tests is not the same as tied on quality. Sonnet spent that extra time testing things nobody asked it to test. Sometimes that's exactly what you want. And sometimes you just want the ticket closed. So where did the open model stumble? Two places. And like I said, the CSV test was three runs each. These next two are one run each. So it's a vibe check, not a verdict. The first one is the game. We gave both of them the same Subway Surfers prompt. GLM had a working game saved in under 9 minutes. Then it did something really smart. It opened the game in a headless browser, took a screenshot, and asked Claude Code to show it the picture. And that's where
6:49the run died. GLM 5.3 is text only. It can't look at images. And Open Router has no endpoint for it that takes images. So, the request came back with an error. No endpoints found that support image input. And that's the whole harness question in one moment. Claude can see, GLM can't. Anthropic's own engineers say every part of a harness encodes an assumption about what the model can't do on its own. This one went the other way. Claude Code assumed the model could see. The game was saved though. So, here they are. Same inputs. Sonnet used the full 45 minutes and was still polishing when time ran out. And you can see it. The shop fronts, the trees, the train. GLM's is simpler, but it plays. Which one looks better to you?
7:35The second place is the bug hunt. A 30,000line invoicing backend. 42 files, 16 real bugs hidden in it, 11 fake ones that look like bugs, and the models were not told how many. Hidden tests decide what's fixed, and two outside judges, Gemini 4 Argon and Grock 4.7, decide what was explained. Neither one is from Anthropic or Z.AI. GLM finished in about 8 minutes. Nine bugs fixed and explained. Zero false reports, zero things broken. That's clean work, honestly. But it missed seven. The double charge race, the tax rounding, the email normalization, a reax that can hang the server, a double finalize race, the cash race, and the refund that never turns reminders back on. Sonnet used the
8:22whole hour and fixed 15 of 16. It even caught the cash race, which it missed in our calibration run. The judges split on Sonnet 30 versus 27 out of 32 because of three extra issues it reported that one judge called real and the other didn't. GLM scored 18 from both. So the races and the state overtime bugs, that's where the brain shows up. The harness gave both of them the same tools. Sonnet just knew where to look on the hard ones. So does that line up with the outside numbers? Pretty much. On artificial analysis, Sonnet 5.5 scores 54 on their intelligence index and GLM 5.3 scores 32. That's at max effort for both. Sonnet writes about 139 tokens a second and GLM about 76, but Sonnet used
9:10about 340 million tokens to run their whole index and GLM about 140 million, which is exactly what we saw. Sonnet writes more and checks more, which is pretty interesting. And here's the best part for this question. On the official Terminal Bench 4.0 leaderboard, both models were run inside Claude Code at max effort. Sonnet 61.8%. GLM 41.8. And GLM lands around 40 on that test in every harness it's been measured in. That's on artificial analysis and on the leaderboard. So, Claude Code didn't give it a boost there. Same 40-ish wherever you put it. and Z.AI's own numbers, those are vendor reported and a lot of them were run inside Cloud Code, but their table never
9:55compares against Sonnet 5.5. Take that for what you will. So, the gap on the leaderboard is real. It just doesn't show up on every task. So, what can you actually hand to an open model today? I'd say this, a well scoped ticket with tests already written in a codebase it can read in one sitting. That's GLM's lane. It was faster. It used fewer tokens. It didn't break anything. And in our runs, it never sent a single malformed tool call. What I wouldn't hand it yet is the stuff that needs you to think about time. Two requests landing at once. A cash filling while something clears it. A refund that should restart a reminder. That's most of what it missed. And that's where bugs cost you money. So, here's how I'd use it in my resale business. We pull a ton
10:41of show data and sales data and we're always asking for some new export or some new report. That's a perfect GLM ticket. But anything that touches payments, I'm keeping on Claude. Opus 5.5 is still my daily driver. And to answer the question from the start, Claude Code gave both of them the exact same tools. On a clean ticket with tests, that was enough for GLM to keep up. On the hard bug hunt, the model made the difference. Sonnet just found more. So, which job are you handing to an open model first? And which one stays with Claude? Comment down below. I want to know. And if this test answered something you were wondering about, give it a like. It goes a long way. Have a blessed day.