How to Benchmark a Codebase Tool So the Number Survives a Rerun

A codebase tool benchmark survives a rerun when you commit the plan before you run anything, seal part of the data, measure the whole agent loop, prove every zero, and publish every row. Those are the eight rules below, and they apply to any tool you want to measure, not only ours.

The reason to care is that headline numbers in this category keep collapsing under independent reruns. In July 2026, JetBrains reran two popular token-saving tools on real agent work with Sonnet 5. Caveman advertised 65% and JetBrains measured 8.5%. RTK advertised 60 to 90% and measured 7.6% more expensive at low reasoning effort, with no change at high effort. Greptile's advertised 82% became 45% under Augment's independent rerun.

Three collapses is a category property, not bad luck. The axis that matters is whose number survives someone else running it, and that is the standard we built our own benchmarks to.

1. Pre-register the benchmark as a commit

Write down what you are testing before the first run, as a timestamped commit. Decide which harness is primary for which claim and what result would falsify your hypothesis, and commit that before the first dollar is spent. Not in your head, and not in a planning doc you can quietly edit later.

The reason is not ceremony. A benchmark you can still redefine after seeing partial results is one you will unconsciously redefine in your own favor. Committing the plan first is what makes "we didn't tune this" a claim anyone else can check.

2. Split the data and seal half of it

Seal part of the data before you build anything, and touch it once, at publication. Our retrieval benchmark, ContextBench, was split into 70 development instances and 42 sealed instances, stratified and pinned by instance id, and committed before rung 8 built a single index. All improvement work used the development half.

The proof that the split worked is what a tool nobody tuned against does on both halves. CodeGraph scored 0.6093 on development and 0.6095 on sealed, three decimal places apart. Nobody optimized for either half, so uneven difficulty would have shown. It did not, which demonstrates the halves are equally hard rather than asserting it.

Our own result moved the direction that matters: 0.810 on development against 0.876 on sealed. Overfitting makes the unseen half score worse. Ours scored better, which is evidence against tuning precisely because it is the opposite of what overfitting predicts.

3. Measure the agent loop, not the payload

Measure whether an agent finishes the job having done less work, not whether your representation of a file is smaller. Measuring one context load is easy, and it is the flattering number almost everybody in this category publishes.

It is not the question anyone actually has. Agents re-read, backtrack, re-plan and re-explore. A compression that looks like 90% on one payload routinely nets out near zero across a real session, and can go negative when the agent works harder to recover what you compressed away. The loop is harder, slower and more expensive to measure, and it is the only version that answers the reader's question. How differently the two can read for the identical tool is in 35x or 16%? we published both, and they measure different things.

4. Prove every zero before you record it

A zero is only real once you have shown the detector works and the arm was alive. This is where the real cost of the work lives, and the specifics are what make the rule usable.

• A broken extractor. A competitor measured 0.012 on a ranking metric in our harness. Our path extractor required whitespace before a file path, and that tool emits paths as [src=path]. Its real score was 0.539, good enough for second place.
• A wrong call shape. Another competitor returned errors on 84 of 84 tool calls. Its tools carry a _tool suffix and one expected structured pattern and target parameters instead of the natural-language query string every other tool accepts. We called it correctly everywhere else and incorrectly there.
• A wrong encoding. A hook-firing counter read five real firings as zero, because the shell redirect wrote UTF-16LE and the reader expected UTF-8. The control printed "POSITIVE CONTROL FAILED" while the hook worked. We nearly shipped a regression that did not exist.

The arm that gets silently zeroed is never your own tool, because your own output format is the only one you already know cold. Every unfamiliar output shape is a place a zero can hide.

5. Give every tool its full advertised surface

An arm you handicapped is not an arm you measured. We under-allowlisted at first, and it looked like a reasonable setup. Graphify serves 10 tools and we gave it 1. Serena serves 29 and we gave it 3. code-review-graph serves 30 and we gave it 1. Our own arm ran the full surface we had pinned for ourselves.

If a tool's behavior depends on tools it never got to call, the number measures your allowlist, not the tool. The fix is mechanical: every competitor gets every tool its own documentation says it serves, and the rule applies to your own tool too, so nobody can quietly under-serve the control.

6. Isolate each arm in its own tree

Give every arm its own worktree so no arm can see another arm's index. Every tool we tested writes its index into a dotdir inside the repository it indexes. A shared checkout means each arm's build sees its predecessors' output, and the bias favors whoever runs first, which in the naive harness was us.

Clearing artifact directories between cells looks like the same fix and is not. It keeps timings clean and is fatal for any later rung that needs an earlier arm's index on disk. Isolate the trees; do not delete the indexes.

7. Publish every row and every invalidated run

Publish the whole table, including rows that read worse than the headline. Indexing time is the clearest example on our benchmarks page: repowise indexes 22x slower than CodeGraph on the same repository, and that row ships next to the retrieval results. The tools we compare against build a call graph. In the same pass repowise mines git history, generates and embeds documentation, extracts decision records and scores code health. Showing the row lets a reader weigh the extra work for themselves.

The same rule applies to runs. Invalidated runs are never deleted. They get an INVALID: banner explaining why and the numbers stay visible, because a benchmark that quietly removes its own bad runs is just harder to audit.

8. Decide in advance which failures you retry

Retry the cells that never ran, and report the cells where the agent ran out of turns. Not every failed cell means the same thing. An auth failure means the cell never ran, so it gets retried. A turn-limit exhaustion is a real agent outcome: the agent tried, ran out of turns and did not finish. That is excluded from the comparison and named as an agent failure, not a plumbing failure.

Retrying turn-limit exhaustions until they succeed quietly deletes the hardest attempts your run produced, the ones most likely to show a tool's real limits. Decide the rule before you see which cells failed, or you will retry the failures that embarrass you and accept the ones that don't.

What a rerun-proof benchmark costs

Rigor has a line item. Producing the retrieval-layer table cost 748 index builds and roughly 78 machine-hours, because every arm builds its own index of every instance's repository at that instance's own base commit, with nothing shared across instances.

Producing the agent-loop tables cost 471 agent runs. About 13 hours of machine time and roughly $44 of API spend. Around a third of those runs are proof-of-life checks, not headline numbers. That third appears in no published table, and it is most of what makes the tables trustworthy: confirming a server answers before a run counts, and confirming a control can report a tool as unused rather than passing everything by default.

A published benchmark that shows no cost like this deserves a second look, because a number that cost nothing to produce is a number nobody checked. For the pattern that first taught us this, where a dollar figure and a token figure disagreed about the identical run, see a tool that did nothing came out 43% cheaper than the bare agent.

Reproduce it

The checklist to copy: commit a pre-registration file; split and seal; run each arm in its own worktree with its full tool surface; add a proof-of-life check per arm and a control that can report a tool as unused; write the retry rule before the first run; publish every row and every INVALID: run. Full tables and raw data are on the benchmarks page, and the retrieval side is the get_answer tool described under the MCP server.