35x or 16%? we published both, and they measure different things

The same tool, on the same codebase, gives 35.6x fewer tokens on one measurement and a 15.9% reduction on another. Both numbers are correct. They answer different questions, and the gap between them is the most useful thing either number tells you.

Two different things called "token savings"

Measuring one context load is easy. You show that your representation of a file, or a commit, or a function, is smaller than the thing itself. That is a real measurement, and it is the one almost everybody in this category publishes, because it is the one that produces the biggest number with the least work.

Our version of that measurement: loading one commit's context with get_context costs 393 tokens. Against 13,984 for the full contents of every changed file and 1,408 for git diff alone. Measured over the 30 most recent non-merge commits of pallets/flask, counted with deterministic tiktoken so the number does not depend on which tokenizer happened to be handy that day.

Measuring an agent loop is hard. Agents re-read files, backtrack when a lead doesn't pan out, re-plan, and re-explore parts of the codebase they've already touched. The question a reader actually has is not "how small is your payload," it is "does an agent given this tool finish the job having done less work." On our own numbers, the answer is -31.6% on Codex and -15.9% on Claude Code, against a bare agent with no tools at all. Both are real reductions. Both are much smaller than 35.6x, because a session is not one payload, it is dozens of decisions about what to read next, and a smaller payload only pays off on the decisions it actually replaces.

We publish both numbers on our benchmarks page and label which is which, rather than letting the bigger one stand in for the smaller one. The methodology behind that choice, and the eight things we had to get right to make either number trustworthy, is in how to benchmark a codebase tool so the number survives a rerun.

The pooling trap

There is a second gap hiding inside the 35.6x itself, and it is worth understanding because it is exactly how a real 35x becomes a press-release 209x.

Pooled, meaning sum of tokens over sum of tokens. A mean of per-commit ratios would read 29.3x against naive rather than 35.6x, and that gap is exactly how a 35x becomes a 209x in a press release.

A one-line commit where our tool returns 40 tokens against a much larger naive read produces a huge ratio, and averaging ratios lets that one commit count exactly as much as a commit where the tool saved a hundred thousand tokens. Pooling weights by the tokens actually at stake, which is why we lead with it and why the smaller pooled number is the one that should be quoted, not the larger mean-of-ratios number that is just as easy to compute from the same data.

If you see a token-reduction number in this category with no note about whether it is pooled or averaged per unit, that is worth asking about directly. The two methods can differ by 20% or more on identical data, in either direction, and only one of them tracks what you'd actually pay across a real set of commits.

Where the agent-loop saving is largest, and why that's coherent

The two numbers are not just different measurements of the same phenomenon, they connect through a mechanism, and the connection is what makes both of them make sense together rather than contradict each other.

Splitting the 48-question Codex run at the median by how much work the bare agent needed to answer each question: the easier half saved 27.2%, and the harder half saved 34.3%, more than the easier half. Against 27.2% on the easier half. The median split was chosen after seeing the data, so this is post-hoc and weaker evidence than the pre-registered comparisons. The correlation between how much work a question demands and how much we save is +0.379. The mechanism is that pre-computed structure replaces exploration, and harder questions contain more exploration to replace. It has now shown up on both harnesses.

That is the mechanism linking the two numbers. Pre-computed structure, the thing the 35.6x measures the size of, replaces exploration, the thing the agent-loop percentage measures the reduction of. Harder questions contain more exploration to replace, so they show a larger agent-loop saving even though the payload itself is the same shape of thing regardless of question difficulty. The single-payload number and the agent-loop number are measuring two ends of the same causal chain, not two unrelated claims, which is also why it would be dishonest to quote only the end of the chain that happens to be bigger.

One honest caveat on that split: it was chosen after seeing the data, so it is post-hoc evidence, not a pre-registered result, and it should be weighted accordingly against the numbers that were pre-registered.

Publish both, label which is which

The rule we landed on, and the one we'd recommend to anyone publishing a number in this category: report both measurements when you have them, name each one for what it measures, and lead with the one that answers the question your reader actually came with. For most readers evaluating whether a tool is worth adopting, that is the agent-loop number, the smaller one, because it is the one that predicts what happens when you actually use the thing.

The single-payload number still earns its place on the page. It is real, it is easier to reproduce, and it is a useful sanity check: if your representation of a commit isn't meaningfully smaller than the commit, the agent-loop number was never going to be good either. It just isn't the number that tells you whether the tool is worth using, and treating it as if it were is the single most common way a benchmark in this category overstates itself. See a tool that did nothing came out 43% cheaper than the bare agent for a related case where the flattering number and the honest number pointed in opposite directions for a completely different reason.

FAQ

Is the 35x number fake?

No. It is a real, reproducible measurement of one thing: how much smaller our representation of a commit's context is than reading every changed file in full. It answers "is this payload small," not "does an agent do less work with this tool," which is a different question with a different, smaller answer.

Which number should I trust when comparing tools?

The agent-loop number, if the vendor publishes one. It is harder to produce and almost always smaller, which is itself a signal: a vendor who only publishes the single-payload number either hasn't measured the agent loop or didn't like what it showed.

What is pooled versus mean-of-ratios, and why does it matter?

Pooled means summing tokens across every commit and dividing sums, which weights each commit by how many tokens are actually at stake. Mean-of-ratios averages the per-commit ratios directly, which lets a tiny commit's huge percentage saving count exactly as much as a commit saving a hundred thousand tokens. The same dataset produces 35.6x pooled and 29.3x as a mean of ratios, and the gap runs to a full order of magnitude in less carefully chosen examples.

Why does the agent-loop saving vary by question difficulty?

Because the mechanism behind the saving is pre-computed structure replacing exploration the agent would otherwise do itself. Harder questions require more exploration in the first place, so there's more for the pre-computed structure to replace, and the saving is measurably larger on the harder half of a question set than the easier half.

Is a smaller agent-loop percentage a worse result than a bigger single-payload multiplier?

No, it's a more honest one. Both come from the same tool and the same underlying mechanism. The percentage is smaller because it accounts for everything an agent does across a real session rather than the size of a single thing it reads, which is exactly why it's the number to trust more.

Where can I see the raw data behind both numbers?

Both live on the benchmarks page, with sample sizes, p-values, and links to the raw CSVs and run directories for each. Neither number is asserted without the conditions it was measured under attached to it.