Benchmarks

Each result names the population, competing field, control, and statistical test that produced it. Retrieval uses a sealed test split with deterministic grading. Agent-loop comparisons use byte-identical prompts and freshly built indexes on the same pinned commit.

A headline number is only useful when the denominator and the failure conditions stay attached to it.

This report keeps precision beside coverage, files served beside retrieval, and tool adoption beside token reduction. It separates one-payload compression from work saved across a complete agent session. Rows that fail their own controls are retained in the raw record and excluded from the claim.

Independent reruns explain why these controls matter. JetBrains measured 8.5% against a claimed 65% saving for Caveman, while RTK's claimed 60 to 90% became 7.6% more expensive at low reasoning effort. Greptile's 82% became 45% under Augment's rerun. Measuring one context payload is easy. Measuring the work across a complete agent loop is harder.

A call graph can fail in two directions: miss calls that are real, or invent calls that are not. Precision measures how many emitted edges are real. Recall measures how much of the real graph was found. Either number is easy to game alone, so this report keeps them paired.

Across seven compiler-graded cells, no measured tool that finds as much of the graph as Repowise gets more of it right.

The answer keys come from the Go team's own call-graph implementation and the TypeScript compiler's resolution of call sites. Repowise does not control either oracle. The field contains five tools and 37,853 oracle edges.

Compiler oracles cover Go and TypeScript. The wider audit uses the same seeded, stratified source-reading protocol on nine languages, with thirty rows per language per tool. Every graded call site and verdict is public.

Read our result in reverse too: roughly fifteen percent of the sampled Repowise edges were wrong. Four language cells separate from CodeGraph and five are ties. This is a two-tool comparison and does not transfer to the wider compiler field.

Before a tool can save an agent any work, it has to point at the right code. This section measures only that. Grading is deterministic and no LLM judge is involved anywhere in the number, which makes it the most reproducible result on the page: ContextBench ships gold file spans, and a tool either returns them or it does not.

Every figure below comes from instances this work has never seen. The 112 instances were split 70 / 42 by instance id, pinned before any of it started, and the 42 were kept sealed until the final measurement.

These are two different tools with two different profiles, and we would rather say so than average them into one claim. get_answer finds the most, from a list of about 19 files. search_codebase finds fewer but is the most efficient per file served, and if you are paying by the token that is the row to read.

We first ran this and came last, at 0.228. We published that. The cause turned out to be a bug: a query-time gate was discarding most candidates before ranking ever happened. Fixing that path is what moved the number, and it is a fix any user of the tool gets rather than a change shaped around these questions.

The check on that claim is the split, and it points the right way. Our score on the sealed half we never saw, 0.876, is higher than our score on the half we tuned against, 0.810. Overfitting makes the unseen half score worse.

And CodeGraph, which nobody tuned against either half, scores 0.6093 and 0.6095 on the two halves. Three decimal places apart. So the halves are equally hard and the gap is about the tool rather than about the questions.

We do not quote a pooled 112-instance figure, though it is easy to compute and would read 0.835. Averaging the halves loses the only number that matters, which is how the tool does on instances it has never seen.

This is the section modelled on the JetBrains reruns, and the one we would ask a skeptic to read first. Six arms: repowise, four competing tools, and a bare agent with no tools at all. Every arm got a byte-identical prompt, its full advertised tool surface, and a freshly built index on the same pinned commit. The bare-agent control was verified free of any local hooks, so it is a real control rather than a contaminated one.

Two things have to be true for a tool to be worth mounting. The agent has to actually call it, and the loop has to get leaner when it does. We ran this on two agent harnesses, because the answer turned out to depend on the harness as much as on the tools.

repowise leaves the agent with the least work to do, and gets there in the fewest steps: 3.8 tool calls against the bare agent's 7.2, and 3.0 files opened instead of 7.2. One answered question replacing roughly six greps, visible directly in the call counts rather than inferred.

Correcting for testing five tools at once, three reductions are solid and two are marginal. CodeGraph is a genuine second at -24.4%, and the honest reading is that we lead a field in which more than one tool works, not that we are the only one that does. Serena is the interesting counter-case: it writes less than the bare agent while calling tools 42% more often. Busier, not leaner.

Under Claude Code most of these tools were barely called at all. code-review-graph never once, Graphify three times in fifteen. Nothing was different about the servers, the questions or the indexes between the two runs. Claude Code loads MCP tool schemas on demand, so the agent has to go looking before it can call anything, and frequently never does.

Treat that column as unstable, including our own 15 of 15.

Rerunning the same setup on later days returned 15 of 15, then 4 of 15, then 3 of 15 for us, and 2 of 14 for CodeGraph. Whether an agent calls a codebase server at all depends more on the harness than on the server. We used Sonnet here, and Sonnet reaches for MCP tools noticeably less than Codex does under an identical setup, so harness and model cannot be separated by this design. We plan to rerun this half on Opus and will add the result when we have it.

One thing this run does not show: it is a work-saved result, not a quality result. A blind judge scored every tool in the field, ours included, a fraction below the bare agent, in a range of 0.04 to 0.25 points on a 10-point scale. None of those differences is statistically distinguishable from zero, and all of them are smaller than the 0.69 points by which this benchmark moves when we rerun it unchanged. Ours is at the low end of that band, we are watching it, and we will say so if it turns into a real effect.

Dollar cost per question is the number every tool in this category wants to quote, and it is close to meaningless as a measure of a tool. Here is the control that convinced us, run in our own harness.

A tool that did nothing came out 43% cheaper than the bare agent.

Under Claude Code, code-review-graph never called its server across all 15 questions. It is behaviourally identical to the bare agent, carrying an extra 28,118 characters of tool schema that should make it cost more. Measured on dollars, it produced a best-in-class saving.

The cause is prompt caching. Cached tokens bill at a fraction of fresh ones, so whichever arm happens to run first pays to warm the cache and every arm after it reads it cheaply. In our run the correlation between an arm's position in the cycle and its dollar cost was -0.487. That is not a property of any tool. It is a property of the schedule.

Output tokens are immune to this. They are never cached, and their correlation with run position is +0.010, which is nothing. So that is what we report. It is a smaller and less impressive number than the dollar figure would have been, and it is the one that survives. If you see a token-savings claim in this category that does not say whether it controls for cache state and arm ordering, this is the first thing to ask about.

The effect is not uniform. repowise saves more on questions that require touching more of the codebase. Splitting the question set at the median by how much work the bare agent needed, the harder half saved 34.3% against the easier half's 27.2%, more than twice as many tokens per question, and the correlation between how much work a question demands and how much we save is +0.379. The mechanism is that pre-computed structure replaces exploration, and harder questions contain more exploration to replace.

Two honest limits on that. The median split was chosen after seeing the data, so it is post-hoc and weaker evidence than the pre-registered comparisons. And every question here is answered in a single session of roughly four to seven turns: we have not measured a long multi-hour task such as designing a feature across many files, and we will not imply a number for one.

This is the measurement almost every tool in this category publishes, and we are labelling it as such. It is a real measurement. It is not section 2, and it should not be read as though it were.

Lead with the pooled figure. Pooled is sum-of-tokens over sum-of-tokens, so it weights each commit by the tokens actually at stake. A mean of per-commit ratios does not: a one-line commit where get_context returns 40 tokens contributes a huge ratio that counts equally against a commit saving a hundred thousand. That is exactly how a 35x becomes a 209x in a press release, and it is why the pooled number is the one we print first.

repowise distill compresses command output before the agent reads it: errors first, exit code preserved, every omission recoverable through an inline marker. A pytest run with 11 failures compresses 61% with all 11 failure lines kept, and a 30-commit diff compresses 86%.

Two commands in the same set saved 0%, because distill never inflates output that is already compact. Reduction is also not comprehension: the bytes removed are measured, and the evidence they were safe to remove is narrower, being preserved failure lines plus CI-asserted fixtures.

A health score is worth something only if the files it flags are the files that break. Scores are taken at a historical commit, bug fixes are counted over the following six months, and nothing after the scoring commit feeds the score. Across 21 repositories, 9 languages and 2,826 files that gives an ROC AUC of 0.74, reaching 0.90 on individual repos. It survives controlling for file size, so it is not simply flagging the big files, and it beats recent churn by +0.10 AUC.

Where CodeScene is ahead, it is a real choice rather than a weaker model. Its nominal precision lead is not statistically significant, but the behaviour behind it is worth having: it flags about 27 files where we flag 132. That is a more conservative threshold, trading recall for a short list a team will actually work through. If what you want is a handful of files to fix this quarter rather than the ranking that catches the most defects, that operating point is the better one.

Two different clocks answer two different questions. Building only the call graph, Repowise has the lowest measured memory footprint and is roughly level on speed. Building the complete index, Repowise is the slowest tool in the field because it also mines git history, generates and embeds documentation, extracts decisions, and scores code health.

So "22x slower" and "the index contains categorically more" are both true, and neither one cancels the other.

If all you want is a call graph, CodeGraph builds one in 16 seconds and you should use it. The comparison that would be dishonest is quoting the ratio without the column beside it, which is why the column is there. It is also a one-time cost: updates after the first index are incremental.