We gave both agents the same ten bugs.
Real bugs from django, taken from SWE-bench Lite. Same model on both sides, same instructions, same code, the same freedom to search. A grader nobody controls decides whether the fix worked. Below is what happened, including every bug we did not fix.
Did it actually fix the bug?
Each square is one bug. Filled means the fix passed the project's own tests, checked by the official grader. Superbrain fixed 8, Claude Code fixed 8. One of those is a coin flip, not a gap, and we show the working further down.
What each fix cost
The seven bugs both agents fixed. Same bug, same model, so the difference is how much context each one moved to get there. Superbrain was cheaper on six of seven.
The bad days are what cost you
Every dot is one bug, placed by what it cost. Superbrain's dots stay bunched. Claude Code is quick and cheap on easy bugs, then spends $0.2510 on the one it finds hard. If you are budgeting a team, you pay for the worst day, not the average one.
Where it went wrong
Three bugs went unfixed. Two of them beat both agents. Here is what happened in each.
Both missed the fix. Only one of them broke the codebase doing it
Neither agent passed the new test. What the resolve column hides is what else happened: Claude Code's patch reads schema_editor.connection.alias, the grader calls that migration with schema_editor set to None, and all eight tests that were passing before died on 'NoneType has no attribute connection'. It scored 0 of 8 on the previously passing suite. Superbrain scored 8 of 8 — it did not fix the bug, but it did not break the eight things that already worked. Two identical failures in the table, one of which would have shipped a regression.
Neither agent solved the media merge
The hardest bug in the set, needing 16 tests to pass. Claude Code got 13, Superbrain 12, and neither broke anything that already worked. Both independently arrived at the same upstream shape, down to the exact separator in the warning message that has decided this bug in every previous run. It is the most expensive instance for both of them and the one neither has ever reliably solved.
What this does not prove
Ten bugs is a small sample. These are the things we would want someone to hold against the numbers.
Ten instances from one repository. A leaderboard figure covers 300 across eleven repos, so this is not comparable to one. The interval on a rate this small is roughly 30 points either way.
One run per instance. The same configuration has produced 7, 8, 9 and 10 on this set across previous runs, so 8 against 8 should be read as a tie and not as a measured equality.
Neither agent had a working Python environment, so neither could run the test suite while working. Both were handicapped the same way.
Claude Code was given a tool we do not have. It ran with WebSearch enabled; Superbrain ships no search tool at all. Switching off a competitor's feature to match our own surface is not a comparison worth publishing, so it was left on. In the event it went unused.
The network was open, which means correctness here is not an isolation claim. It happens not to matter on this run: Claude Code made zero external calls, and Superbrain made five, all on django-11019, which it failed. Every instance either agent resolved was resolved without a lookup, and the audit in the bundle is what that rests on.
How it ran
Everything held identical except the agent.
The GitHub issue text, word for word, plus one line saying do not edit the tests. Same string to both agents.
Claude Sonnet 5 on both sides, pinned, both billed to the same API key.
A fresh django checkout per bug, at the commit before the fix landed, with no later history in it.
The fix for every one of these bugs is public, and both agents were free to go and find it — the same freedom, on both sides. What is being measured is how efficiently each harness drives the model, not whether the model remembers the bug. Every transcript was audited and the bundle names each URL either agent reached for.
The official SWE-bench grader, in Docker. A bug counts as fixed only if the failing tests pass and the passing ones still do.
One rate card for both. We do not use either agent's self reported cost, because they price cached tokens differently.
Check it yourself
We ran this, so do not take our word for it. Rerun the official grader against our patches and see if you get the same 8 and 8.
python -m swebench.harness.run_evaluation --dataset_name princeton-nlp/SWE-bench_Lite --split test --predictions_path predictions/superbrain.jsonl --run_id verify
75 KB. Patches, grader output, container logs, per bug usage and the isolation audit. No agent transcripts, since those carry both companies' internals.