On this page
Both numbers are correct, and they answer different questions. Loading one commit's context with repowise costs 35.6x fewer tokens than reading every changed file. Giving an agent repowise's tools cuts its output tokens by 31.6% on Codex and 15.9% on Claude Code, against a bare agent. The gap between the two is the most useful thing either one tells you.
What is the difference between a single-payload and an agent-loop token benchmark?
A single-payload benchmark compares the size of one piece of context against the thing it summarizes. An agent-loop benchmark compares the total work an agent does to finish a task with and without the tool. The first is easy to run and gives the bigger number, which is why most of this category publishes it.
Our single-payload measurement: loading one commit's context with get_context costs 393 tokens. Against 13,984 for the full contents of every changed file and 1,408 for git diff alone. We measured the 30 most recent non-merge commits of pallets/flask and counted with deterministic tiktoken, so the result does not depend on whichever tokenizer was handy.
An agent loop is messier. Agents re-read files, backtrack when a lead fails, re-plan and revisit code they already touched. The question a buyer actually has is "does an agent with this tool finish having done less work," and the agent-loop number answers it. A session is dozens of decisions about what to read next, and a smaller payload only pays off on the decisions it replaces, so the loop saving is smaller than the payload multiplier.
| Single payload | Agent loop | |
|---|---|---|
| Question answered | Is the representation small? | Does the agent do less work? |
| Our result | 35.6x fewer tokens, pooled | -31.6% (Codex), -15.9% (Claude Code) |
| Baseline | Full contents of every changed file | A bare agent with no tools |
| Effort to reproduce | Low | High |
Scroll the table sideways to see every column.
Both appear on the benchmarks page, each labeled for what it measures. The method behind that choice is in how to benchmark a codebase tool so the number survives a rerun.
Why should you pool tokens instead of averaging ratios?
Pooling means summing tokens across all commits and dividing the sums, which weights each commit by the tokens actually at stake. Averaging per-commit ratios lets a one-line commit count as much as a commit that saved a hundred thousand tokens. That is how a real 35x turns into a press-release 209x.
Pooled, meaning sum of tokens over sum of tokens. A mean of per-commit ratios would read 29.3x against naive instead of 35.6x, and that gap is exactly how a 35x becomes a 209x in a press release.
If a token-reduction claim does not say whether it is pooled or averaged per unit, ask. The two methods can differ by a wide margin on identical data, and only the pooled one tracks what you pay across a real set of commits. Against git diff alone, the pooled reduction is 3.6x.
Why does the agent-loop saving grow with question difficulty?
Pre-computed structure replaces exploration. The single-payload number measures how small that structure is, and the agent-loop number measures how much exploration it removes, so the two sit at opposite ends of one causal chain.
We split the Codex question set at the median by how much work the bare agent needed. The harder half saved 34.3%. Against 27.2% on the easier half. The median split was chosen after seeing the data, so this is post-hoc and weaker evidence than the pre-registered comparisons. Across questions, the correlation between work demanded and tokens saved is +0.379. The mechanism is that pre-computed structure replaces exploration, and harder questions contain more exploration to replace. It has now shown up on both harnesses.
Harder questions contain more exploration to replace, so they show a larger saving even though the payload is the same shape whatever the question. The split was chosen after seeing the data, so we treat it as post-hoc evidence and weight it below the pre-registered comparisons.
Which number should lead when you publish a benchmark?
Report both, name each for what it measures, and lead with the one that answers your reader's question. For most people deciding whether to adopt a tool, that is the agent-loop number, because it predicts what happens when they use it.
The single-payload number still earns its place. It is real, easy to reproduce, and a sanity check: if a representation of a commit is not meaningfully smaller than the commit, the loop number was never going to be good. It stops being useful the moment it stands in for the loop number. For a related case where the flattering number and the honest one pointed opposite ways, see a tool that did nothing came out 43% cheaper than the bare agent. To try the tools on your own repository, start with the MCP tools overview.
FAQ
- Is the 35x number fake?
- No. It is a reproducible measurement of one thing: how much smaller repowise's representation of a commit is than reading every changed file in full. It answers "is this payload small," not "does an agent do less work," which has a different and smaller answer.
- Which number should I trust when comparing tools?
- The agent-loop number, if the vendor publishes one. It is harder to produce and almost always smaller. A vendor that publishes only a single-payload multiplier has either not measured the loop or not liked the result.
- What is pooled versus mean-of-ratios, and why does it matter?
- Pooled means summing tokens over every commit and dividing the sums, so each commit counts by the tokens at stake. Mean-of-ratios averages per-commit ratios, so a tiny commit's huge percentage counts as much as a large one. On our Flask data the pooled figure is 35.6x, and the mean of ratios reads lower.
- Why does the agent-loop saving vary by question difficulty?
- Pre-computed structure replaces exploration the agent would otherwise do itself. Harder questions need more exploration, so there is more to replace, and the saving is larger on the harder half of the question set.
- Is a smaller agent-loop percentage a worse result than a bigger multiplier?
- No. Both come from the same tool and mechanism. The percentage is smaller because it accounts for everything an agent does across a session, not the size of one thing it reads, which is why it is the number to trust more.
- Where can I see the raw data behind both numbers?
- On the benchmarks page, with sample sizes, p-values, and links to the raw CSVs and run directories. Each number is shown with the conditions it was measured under.
