On this page
Dollar cost per question is a poor way to measure a codebase tool, because prompt caching bills by run order. In our benchmark, a tool that the agent never called came out 43% cheaper than a bare agent with no tools at all. A tool that did nothing cannot save money, so the saving belonged to the schedule.
What happened in the control
We ran six arms (five tools and a bare agent) against the same 15 questions on django/django under Claude Code. One arm, code-review-graph, never called its server on any question. n = 15 · Claude Code (claude-sonnet-5) · django/django at a single pinned commit · 2026-08. It advertises 30 tools over a graph of 40,904 nodes and 380,168 edges, and the agent never asked it anything.
A tool the agent never calls behaves exactly like the bare agent: same reasoning, same tool calls, same answer. The one difference is that its schema, about 28,118 extra characters of tool descriptions, sits in the prompt the whole time. That should make the run more expensive. Measured in dollars, it came out 43% cheaper.
Why prompt caching produces a phantom saving
Anthropic's prompt cache bills cache reads at a fraction of the base input price, so whichever arm runs first pays to warm the cache and later arms read the same prefix cheaply. Per Anthropic's prompt caching documentation (checked October 2026), a five-minute cache write costs 1.25x the base input price and a cache read costs a small fraction of it. Schema, system prompt and shared context all benefit.
So an arm's dollar figure is partly a measurement of the tool and partly a measurement of where it landed in the queue. We checked this instead of asserting it. n = 90 · Claude Code (claude-sonnet-5) · django/django at a single pinned commit · 2026-08: -0.487. A correlation that size means roughly a quarter of the variance in an arm's dollar cost is explained by nothing except when it ran.
The cache has a clock
The default cache lifetime is five minutes, so the number of arms in a rotation changes every arm's cost. How long an arm waits between two of its own cells depends on how many arms are interleaved. With six arms sharing a rotation the gap is long enough for the cache to go cold. With two arms the cache stays warm throughout.
We measured the difference on the same tool and the same questions, changing only the number of arms sharing the schedule: 39 percentage points of movement on the measured dollar delta. A vendor could report almost any dollar saving for an identical tool just by choosing how many competitors to run it alongside.
Output tokens are immune
Output tokens are never cached, so run order cannot move them. The reasoning, tool calls and final reply an agent writes are generated fresh every time. We ran the same correlation on output tokens: n = 90 · Claude Code (claude-sonnet-5) · django/django at a single pinned commit · 2026-08: +0.010, which is no relationship at all.
That is why the agent-loop benchmark reports output tokens, not dollars. On the same Claude Code run, output tokens fell 15.9% against the bare agent when the agent used the tool. It is a smaller number than the dollar figure would have been, and it is the one that survives a rerun on a different schedule. The full method is in how to benchmark a codebase tool so the number survives a rerun.
On the Codex harness, where every tool was actually called, repowise cut output tokens -31.6% against the bare agent, n = 43 · p = <0.0001 · Codex (gpt-5.6-sol) · django/django at a single pinned commit · 2026-08. For why a whole-loop measurement and a single-payload measurement answer different questions, see 35x or 16%? we published both.
What to ask of any cost claim
When you read a token or cost claim in this category, ask first whether it controls for cache state and arm order. A figure that does not say has shown you the shape of the benchmark, not the cost of the tool.
This is not specific to code-review-graph. It follows from how prompt caching bills, and it will produce the same illusion for anyone running a multi-arm comparison without controlling for it. We hit it in our own harness and fixed it by dropping the number the cache was distorting. The benchmarks page lists every tool by name with its harness and date.
FAQ
- Why did a tool that was never called come out cheaper than no tool at all?
- It ran later in the schedule than the bare agent. Prompt caching bills cache reads at a fraction of fresh input, so the arm that runs first pays to warm the cache and every later arm reads the same prefix cheaply. The saving belongs to the schedule, not the tool.
- Is this specific to code-review-graph?
- No. Any tool in any multi-arm benchmark that does not control for cache state and run order can show it. code-review-graph is the cleanest control we had, because a tool that provably did nothing still won on cost.
- How long does the prompt cache stay warm?
- The default lifetime is five minutes, with an optional one-hour setting, per Anthropic's documentation. How long an arm waits between its own cells depends on how many arms share the rotation, so the number of competitors in a benchmark changes its cost numbers.
- Why does repowise report output tokens instead of dollars?
- Output tokens are never cached, so they do not depend on run order. On the same runs, position correlated with dollar cost at -0.487 and with output tokens at +0.010.
- Does this mean dollar-cost numbers are always wrong?
- No, but they depend on the benchmark's schedule in ways most published numbers do not disclose. A figure from a single-arm run with a stable cache is more trustworthy than one from a rotating multi-arm comparison. Ask how the run was scheduled.
- Where can I see the full agent-loop numbers?
- The benchmarks page publishes the Codex and Claude Code tables, including which tools the agent called and how much the loop got leaner.
