We gave both agents the same ten bugs.
Real bugs from django, taken from SWE-bench Lite. Same model on both sides, same instructions, same code, no internet. A grader nobody controls decides whether the fix worked. Below is what happened, including the three we did not fix.
Did it actually fix the bug?
Each square is one bug. Filled means the fix passed the project's own tests, checked by the official grader. Superbrain fixed 7, Claude Code fixed 8. One of those is a coin flip, not a gap, and we show the working further down.
What each fix cost
The seven bugs both agents fixed. Same bug, same model, so the difference is how much context each one moved to get there. Superbrain was cheaper on six of seven.
The bad days are what cost you
Every dot is one bug, placed by what it cost. Superbrain's dots stay bunched. Claude Code is quick and cheap on easy bugs, then spends $0.4163 on the one it finds hard. If you are budgeting a team, you pay for the worst day, not the average one.
Where it went wrong
Three bugs went unfixed. Two of them beat both agents. Here is what happened in each.
Both agents got a one line fix wrong
The change is a single line and the issue text spells it out. Both agents wrote it correctly in earlier runs and both missed it here. Superbrain produced 1,376 bytes where 555 was right. Nothing external was involved, so this is run to run variance, and it is the strongest reason to read the resolve column as a tie.
Neither agent solved the media merge
The hardest bug in the set. It needs 16 tests to pass. Superbrain got none of them, Claude Code got 2, and neither broke anything that already worked. In earlier runs with the network open, each agent solved it exactly once, and only by downloading the already fixed release from PyPI. With the network closed, neither can.
Superbrain lost the only instance that separates the two
The required behaviour is a warning message that appears nowhere in the issue and nowhere in the repository. Its test is added by the grader afterwards. Claude Code produced the upstream wording from memory, paraphrased rather than copied. Superbrain did not, because it will not assert what the codebase cannot support. On this instance the benchmark rewards recall and scores grounding as a miss.
What this does not prove
Ten bugs is a small sample. These are the things we would want someone to hold against the numbers.
Ten instances from one repository. A leaderboard figure covers 300 across eleven repos, so this is not comparable to one. The interval on a rate this small is roughly 30 points either way.
One run per instance. Across three runs Superbrain scored 8, 8 and 7 while failing different instances each time, so 7 and 8 are the same measurement here. Read the resolve column as a tie.
Neither agent had a working Python environment, so neither could run the test suite while working. Both were handicapped the same way.
Network isolation is a strong control, not a sandbox. Every run is audited for lookups and the audit ships with the result.
How it ran
Everything held identical except the agent.
The GitHub issue text, word for word, plus one line saying do not edit the tests. Same string to both agents.
Claude Sonnet 5 on both sides, pinned, both billed to the same API key.
A fresh django checkout per bug, at the commit before the fix landed, with no later history in it.
The fix for every one of these bugs is public. We closed the network so neither agent could look it up, and audited every transcript to check.
The official SWE-bench grader, in Docker. A bug counts as fixed only if the failing tests pass and the passing ones still do.
One rate card for both. We do not use either agent's self reported cost, because they price cached tokens differently.
Check it yourself
We ran this, so do not take our word for it. Rerun the official grader against our patches and see if you get the same 7 and 8.
python -m swebench.harness.run_evaluation --dataset_name princeton-nlp/SWE-bench_Lite --split test --predictions_path predictions/superbrain.jsonl --run_id verify
76 KB. Patches, grader output, container logs, per bug usage and the isolation audit. No agent transcripts, since those carry both companies' internals.