We Benchmarked Ourselves Against Four Open Source Tools and Came Last

On 42 ContextBench instances that stayed sealed until the final run, repowise get_answer found 0.876 of the files a real fix touched, against 0.610 for CodeGraph, the next best tool. Our first run on the development half scored 0.228 and came last. This post covers what was broken, what we changed, and why the sealed result is evidence of a fix and not of benchmark tuning.

We compared repowise with CodeGraph, Graphify and code-review-graph, the open source tools that build a graph or index over a repository to answer "where is this?" questions for coding agents. The test is ContextBench: a bug report from a real GitHub issue, and the set of files the actual fix changed. Grading compares returned paths with those files. No LLM judge is involved, so anyone can rerun it.

What was the retrieval benchmark protocol?

We split the 112 instances 70/42 by instance id, committed the split to a file, and sealed the 42 before building a single index. All improvement work used the 70 development instances. The sealed 42 were queried once, at the end, and whatever came back was going to be published.

We fixed the protocol first because the temptation is predictable: measure, improve against the questions you are about to report on, publish the good number. The split removes that option. It also let us commit to publishing the arc (first run, diagnosis, fix, sealed result), since a fix that did not hold on unseen instances would have been the result too.

Every arm built its own index of every instance's repository at that instance's base commit, with nothing shared or cached between tools. That took 748 index builds across the matrix.

What did the first run show?

repowise placed last on the development half. The metric is file coverage: the share of gold files that appear in what the tool returns.

Before any of this counted we checked the harness: 70 of 70 cells returned status: ok, the embedder was live on every repowise cell, and a known-perfect and a known-wrong prediction were graded to confirm the grader could tell them apart. The pooled number also hid a split by repository. On django we scored 0.312. On the 20 Go instances we scored 0.025 while every competitor cleared 0.50.

What was actually wrong?

The index and the ranking were fine. Query-time gates were returning early with an answer built from something other than the ranked candidate pool. Of the 19 instances where we surfaced a gold file, 14 had it at rank 1. A failure taxonomy over the 29 get_answer misses on the development half put 21 of them in the same bucket: an early-return gate firing and bypassing the ranking mainline. That mainline hit 33 of 37 instances that reached it.

One gate was meant to recognise questions about a data shape, using trigger words such as field, key and schema. Those words appear in most bug reports about an ORM-backed codebase, so the gate was detecting database-shaped repositories instead. It fired on 23 of 50 python instances and almost none of the Go ones, which explains most of the Go gap.

The cleanest evidence was a control. On 18 of the 19 instances where CodeGraph won and get_answer lost, our own search_codebase had already found the file, on the same index in the same run. Same bytes, same embeddings, and one query path dropped the candidate before ranking. That places the whole competitive gap at query time.

What did we change?

Two narrow changes, shipped as ordinary pull requests that every user of the tool receives. get_answer stopped discarding the ranked candidate pool the early-return gates sat on top of, and the ORM-vocabulary gate got a tighter trigger condition. There was no new ranking model and no larger context window.

search_codebase was not touched by either change and moved by exactly 0.000 on all 70 instances. That rules out index drift, embedder drift and grader drift as explanations for the get_answer jump, because an upstream change would have moved the control too.

One instance went from 1.000 to 0.000 during this run. Its gold set is a single file, so one file dropping off the tail of a served list flips the score with nothing in between. A three-rebuild stability recheck was queued, but every rebuild in that batch failed for an unrelated operational reason before running, so that instance stays in the data as measured.

How did it do on the sealed half?

Higher than on the half we developed against, at 0.876. This was the first time the 42 sealed instances were queried.

Overfitting has a signature, and it is the opposite of this. A fix shaped around the quirks of the questions it was tuned on scores worse on questions it has never seen. Ours scored better.

CodeGraph is the check on the check. Nobody tuned anything against it and nobody on its side knew our split existed. It scores 0.6093 and 0.6095 on the two halves, so they are equally hard and the gap is about the tools, not the questions.

Full sealed-half results (n=42, measured August 2026, from the published tables):

Head to head against CodeGraph per instance, get_answer recorded 19 wins, 1 loss, 22 ties, a sign test at p = 0.00004. We do not quote a pooled 112-instance figure. Averaging the halves discards the one distinction that gives the result meaning: instances the improvement work never saw.

What does this measure?

File retrieval, graded deterministically against the files a real fix touched. It shows that repowise points an agent at the right files for a bug report in Python and Go. Whether an agent then writes a better patch is a separate measurement, covered in the agent-loop section of /benchmarks. Python and Go are the languages ContextBench covers, so the claim stays within them.

Why publish the first run before the fix?

A benchmark is worth what it would survive when someone else reruns it. Showing 0.228 first, on the record and before any fix existed, is what separates a measured improvement from a number rerun until it looked right. The sealed split, the untouched control arm and the fixed-target baseline each remove one way the result could be wrong.

We wrote down how to build a result that survives a rerun in how to benchmark a codebase tool. The raw cells, including every failed one, are in the repowise-bench repository, and the /compare pages put the tools side by side.