Every number we publish, with its conditions.

In July 2026, JetBrains took two popular token-saving tools and reran their headline claims on real agent work. Caveman advertised a 65% saving and measured 8.5%. RTK advertised 60 to 90% and came back 7.6% more expensiveat low reasoning effort. Greptile's 82% became 45%under Augment's rerun.

Three independent collapses is not bad luck. In this category the advertised number and the reran number are routinely different numbers, and they are different for a reason worth understanding before reading anything below.

Measuring one context load is easy. Measuring an agent loop is hard.

You can show that your representation of a file is smaller than the file. That is a real measurement and it is the one almost everybody publishes. It is also not the question anyone actually has, which is whether an agent given your tool finishes the job having done less work. Agents re-read, backtrack, re-plan and re-explore. A compression that looks like 90% on one payload routinely nets out near zero across a real session, and can go negative when the agent has to work harder to recover what you compressed away.

This page publishes both numbers and labels which is which. Section 3 is the one-payload figure. Section 2 is the agent-loop figure, and it is deliberately the smaller one.

Before a tool can save an agent any work, it has to point at the right code. This section measures only that. Grading is deterministic and no LLM judge is involved anywhere in the number, which makes it the most reproducible result on the page: ContextBench ships gold file spans, and a tool either returns them or it does not.

Every figure below comes from instances this work has never seen. The 112 instances were split 70 / 42 by instance id, pinned before any of it started, and the 42 were kept sealed until the final measurement.

These are two different tools with two different profiles, and we would rather say so than average them into one claim. get_answer finds the most, from a list of about 19 files. search_codebase finds fewer but is the most efficient per file served, and if you are paying by the token that is the row to read.

We first ran this and came last, at 0.228. We published that. The cause turned out to be a bug: a query-time gate was discarding most candidates before ranking ever happened. Fixing that path is what moved the number, and it is a fix any user of the tool gets rather than a change shaped around these questions.

The check on that claim is the split, and it points the right way. Our score on the sealed half we never saw, 0.876, is higher than our score on the half we tuned against, 0.810. Overfitting makes the unseen half score worse.

And CodeGraph, which nobody tuned against either half, scores 0.6093 and 0.6095 on the two halves. Three decimal places apart. So the halves are equally hard and the gap is about the tool rather than about the questions.

We do not quote a pooled 112-instance figure, though it is easy to compute and would read 0.835. Averaging the halves loses the only number that matters, which is how the tool does on instances it has never seen.

This is the section modelled on the JetBrains reruns, and the one we would ask a skeptic to read first. Six arms: repowise, four competing tools, and a bare agent with no tools at all. Every arm got a byte-identical prompt, its full advertised tool surface, and a freshly built index on the same pinned commit. The bare-agent control was verified free of any local hooks, so it is a real control rather than a contaminated one.

Two things have to be true for a tool to be worth mounting. The agent has to actually call it, and the loop has to get leaner when it does. We ran this on two agent harnesses, because the answer turned out to depend on the harness as much as on the tools.

repowise leaves the agent with the least work to do, and gets there in the fewest steps: 3.8 tool calls against the bare agent's 7.2, and 3.0 files opened instead of 7.2. One answered question replacing roughly six greps, visible directly in the call counts rather than inferred.

Correcting for testing five tools at once, three reductions are solid and two are marginal. CodeGraph is a genuine second at -24.4%, and the honest reading is that we lead a field in which more than one tool works, not that we are the only one that does. Serena is the interesting counter-case: it writes less than the bare agent while calling tools 42% more often. Busier, not leaner.

Under Claude Code most of these tools were barely called at all. code-review-graph never once, Graphify three times in fifteen. Nothing was different about the servers, the questions or the indexes between the two runs. Claude Code loads MCP tool schemas on demand, so the agent has to go looking before it can call anything, and frequently never does.

Treat that column as unstable, including our own 15 of 15.

Rerunning the same setup on later days returned 15 of 15, then 4 of 15, then 3 of 15 for us, and 2 of 14 for CodeGraph. Whether an agent calls a codebase server at all depends more on the harness than on the server. We used Sonnet here, and Sonnet reaches for MCP tools noticeably less than Codex does under an identical setup, so harness and model cannot be separated by this design. We plan to rerun this half on Opus and will add the result when we have it.

One thing this run does not show: it is a work-saved result, not a quality result. A blind judge scored every tool in the field, ours included, a fraction below the bare agent, in a range of 0.04 to 0.25 points on a 10-point scale. None of those differences is statistically distinguishable from zero, and all of them are smaller than the 0.69 points by which this benchmark moves when we rerun it unchanged. Ours is at the low end of that band, we are watching it, and we will say so if it turns into a real effect.

Dollar cost per question is the number every tool in this category wants to quote, and it is close to meaningless as a measure of a tool. Here is the control that convinced us, run in our own harness.

A tool that did nothing came out 43% cheaper than the bare agent.

Under Claude Code, code-review-graph never called its server across all 15 questions. It is behaviourally identical to the bare agent, carrying an extra 28,118 characters of tool schema that should make it cost more. Measured on dollars, it produced a best-in-class saving.

The cause is prompt caching. Cached tokens bill at a fraction of fresh ones, so whichever arm happens to run first pays to warm the cache and every arm after it reads it cheaply. In our run the correlation between an arm's position in the cycle and its dollar cost was -0.487. That is not a property of any tool. It is a property of the schedule.

Output tokens are immune to this. They are never cached, and their correlation with run position is +0.010, which is nothing. So that is what we report. It is a smaller and less impressive number than the dollar figure would have been, and it is the one that survives. If you see a token-savings claim in this category that does not say whether it controls for cache state and arm ordering, this is the first thing to ask about.

The effect is not uniform. repowise saves more on questions that require touching more of the codebase. Splitting the question set at the median by how much work the bare agent needed, the harder half saved 34.3%against the easier half's 27.2%, more than twice as many tokens per question, and the correlation between how much work a question demands and how much we save is +0.379. The mechanism is that pre-computed structure replaces exploration, and harder questions contain more exploration to replace.

Two honest limits on that. The median split was chosen after seeing the data, so it is post-hoc and weaker evidence than the pre-registered comparisons. And every question here is answered in a single session of roughly four to seven turns: we have not measured a long multi-hour task such as designing a feature across many files, and we will not imply a number for one.

This is the measurement almost every tool in this category publishes, and we are labelling it as such. It is a real measurement. It is not section 2, and it should not be read as though it were.

Lead with the pooled figure. Pooled is sum-of-tokens over sum-of-tokens, so it weights each commit by the tokens actually at stake. A mean of per-commit ratios does not: a one-line commit where get_context returns 40 tokens contributes a huge ratio that counts equally against a commit saving a hundred thousand. That is exactly how a 35x becomes a 209x in a press release, and it is why the pooled number is the one we print first.

repowise distill compresses command output before the agent reads it: errors first, exit code preserved, every omission recoverable through an inline marker. A pytest run with 11 failures compresses 61% with all 11 failure lines kept, and a 30-commit diff compresses 86%.

Two commands in the same set saved 0%, because distill never inflates output that is already compact. Reduction is also not comprehension: the bytes removed are measured, and the evidence they were safe to remove is narrower, being preserved failure lines plus CI-asserted fixtures.

A health score is worth something only if the files it flags are the files that break. Scores are taken at a historical commit, bug fixes are counted over the following six months, and nothing after the scoring commit feeds the score. Across 21 repositories, 9 languages and 2,826 files that gives an ROC AUC of 0.74, reaching 0.90 on individual repos. It survives controlling for file size, so it is not simply flagging the big files, and it beats recent churn by +0.10 AUC.

Where CodeScene is ahead, it is a real choice rather than a weaker model. Its nominal precision lead is not statistically significant, but the behaviour behind it is worth having: it flags about 27 files where we flag 132. That is a more conservative threshold, trading recall for a short list a team will actually work through. If what you want is a handful of files to fix this quarter rather than the ranking that catches the most defects, that operating point is the better one.

We are the slowest indexer in the field, on every repo we measured, and it is not close. The reason is not an optimisation we forgot: the tools we are measured against build a call graph, and in the same pass we also mine git history, generate and embed documentation, extract decision records and score code health.

So "22xslower" and "the index contains categorically more" are both true, and neither one cancels the other.

If all you want is a call graph, CodeGraph builds one in 16 seconds and you should use it. The comparison that would be dishonest is quoting the ratio without the column beside it, which is why the column is there. It is also a one-time cost: updates after the first index are incremental.

Beyond the ones stated in each section:

• Python and Go only. No TypeScript or JavaScript row appears anywhere on this page. That was a scope choice made for instance density in the benchmark corpus, not a statement about language support.
• Section 2 is one repository at one commit, which is in every model's training data.
• Nothing here measures a long engineering task. Every question is a single session of roughly four to seven turns.
• Two capability rows are not measured at all: documentation generation against DeepWiki, Google Code Wiki and Swimm, and PR review against CodeRabbit and Greptile. We would rather say "not measured" than let a checkmark do a number's job.

The full methodology, the pre-registration files with their commit timestamps, the arm-parity rules, the statistical tests, and the list of measurement traps that produced wrong numbers before we caught them all live in repowise-bench. That repository also holds every raw run, kept permanently, including the invalidated ones with their invalidation notes attached.