Case Study: We Build repowise With repowise

repowise is developed against a repowise index of its own codebase. Every pull request against the repo is scored by the same deterministic risk and health checks that ship to users, before a human reviews it. When a check is wrong, we are the ones it is wrong about.

That last part is the whole reason this post exists. "We use our own product" is a weak claim on its own, because it is unfalsifiable and every vendor says it. What follows is the specific version: what the setup actually is, what we measured and on which corpus, what our own scores look like today including the parts we are not happy with, and where the numbers stop supporting the claim.

The measurement problem with dogfooding

There is an obvious trap in measuring a tool on the codebase of the team that built it. We know where everything is. Our index is unusually well tuned because we notice the moment it degrades. Any efficiency number we produced on our own repo would be the best case by construction, and worth very little to a reader deciding whether it will work on theirs.

So the split we settled on is:

• Efficiency and quality numbers come from public repositories (Flask and scikit-learn), through a public harness, so anyone can rerun them and get their own figures.
• Our own repo is where we describe the workflow and publish our current scores, because that is a question about operating discipline, not about performance.

Nothing in the benchmark section below was measured on repowise itself. That is deliberate, and it is the main thing separating this from a testimonial.

The setup

The repowise monorepo is Python core plus TypeScript UI, roughly 1,500 source files across the architectural layers. Four things sit on top of the index in every working session.

1. A generated CLAUDE.md. repowise init writes the orientation block loaded at session start: architecture summary, key modules, entry points, hotspots with owners, and current health scores. When the index updates, the block updates. It is the same file the CLAUDE.md generator produces for any repo.
2. MCP tools over the index. Instead of walking the tree, the agent queries: get_answer for how and where and why, get_context for a triage card on a file it is about to edit, get_symbol for a verified source range, search_codebase for hybrid search, get_risk before touching a hotspot.
3. Output distillation. Noisy commands, meaning test runs, builds, git log, run through repowise distill, which puts errors first and keeps every error line while compressing the rest.
4. The PR bot and health gate. Every pull request gets a deterministic risk and health check before review.

Agent-assisted, deterministically gated

Worth stating plainly, because it is the first question an engineer reading this will have: a large share of the code in this repo is written in agent sessions. We think that is only defensible if the controls around it are stronger than the controls around hand-written code, not weaker.

The controls that matter here are the ones that do not involve a model:

• The risk and health checks on every PR are deterministic. No LLM scores them. The same input produces the same output, byte for byte, which means the gate cannot be argued with, flattered, or prompted into agreement.
• Tests, types, and lint gate merges the ordinary way. The agent workflow did not replace any of that.
• The benchmark harness is public and adversarial to us. It is the artifact that would expose the claims in this post as wrong, and it lives outside our control in the sense that anyone can run it.

The interesting property is not that an agent wrote a diff. It is that the diff had to pass a check whose scoring function we published, calibrated against a defect corpus we did not curate for the occasion. An agent that produces a plausible-looking change and a human who produces one fail that gate identically.

What we measured

The numbers below come from the public benchmark repository: harness, configs, raw results, and the report they are quoted from. Each run pairs two identical agents on identical tasks. The only variable is whether the repowise MCP tools are available.

Token efficiency: understanding a commit

Measured on recent non-merge commits of pallets/flask. The question is how many tokens each strategy needs to give a model enough context to understand a change.

That is roughly 27x fewer tokens for the same task. It is the single largest effect we have measured, and it is also the least surprising one: loading whole files to understand a small change is simply the wrong unit of retrieval.

Paired agent runs: 48 tasks on Flask, 48 on scikit-learn

SWE-QA tasks, same model, same prompts, same budget, same judge. C0 is a bare agent with Read, Grep, Glob, and Bash. C2 adds repowise MCP tools.

On Flask, 32 of 48 tasks were cheaper with repowise. On scikit-learn, 33 of 48.

Read the judge scores carefully, because they are the row that constrains the claim. Flask is a tie, 8.82 against 8.81. On scikit-learn the repowise run scored lower, 8.72 against 8.23. We are not going to describe that as noise and move on. The honest reading is that on this sample the agent reached comparable answers on Flask and slightly worse ones on scikit-learn while doing substantially less reading. The efficiency claim survives that. A quality claim would not, which is why we do not make one.

Where the cost win actually comes from

A follow-up run split the result in two, and this part matters if you care about agent economics.

On short question-and-answer tasks, the win depends on keeping the advertised tool surface small. At the time of that run the server exposed nine tools, and advertising all of them cost 4,520 schema tokens against 1,884 for a four-tool core profile, a 58% cut. With the full surface the cost saving was only 4%. With the lean surface it was 25%. Those token figures describe the surface as it was measured; the tool set has grown since, which moves the numbers but not the conclusion.

The conclusion is the useful part: on a short task, schema you advertise but never call is pure overhead, and it can eat most of the navigation saving.

On long investigation tasks the win comes from distillation instead. Routing command output through repowise distill cut cache-read tokens by 41% and cost by 26%.

What our own numbers look like

The repo carries a live health badge. Whatever it reads right now is our current average code health, served from the same index this page describes:

Code health

We are deliberately not restating that score in the text. A number typed into a blog post is correct on the day it is written and quietly wrong afterwards, which is the failure mode this whole page is arguing against. The badge is the number. Click it for the per-file breakdown.

Cross-project validation of the score itself: mean ROC AUC 0.74, 95% confidence interval 0.68 to 0.79, at identifying files that go on to receive bug fixes, reaching 0.90 on individual repos. That study covers 21 open-source repositories, 9 languages, and 2,826 files, and it predates the promotion of Scala and Ruby to the Full tier, so it spans 9 of today's 11 Full-tier languages.

Two things are worth saying about publishing the badge at all.

First, it is an average across every file, and averages on a real codebase are not flattering by nature. A repo with no 4s in it is usually a repo that has not shipped much.

Second, the score is only interesting because it is falsifiable. ROC AUC 0.74 means the score ranks a known-buggy file worse than a clean one about 74% of the time. That is meaningfully better than a coin flip and well short of an oracle, and we would rather publish the interval than a round number with no error bars on it.

What changes day to day

The benchmarks measure single tasks. On our own repo the compounding effects are the point.

• Orientation is free. A session starts already knowing which file in the analysis package has the highest churn and who owns it. Nobody re-derives the architecture each morning.
• Risk arrives before the edit, not in review. The change-risk signals flag hotspot files and likely co-changes when a change is proposed, which is when acting on that information is still cheap.
• We meet our own false positives first. The health gate runs on our PRs before it runs on yours. When a marker is noisy, it is noisy at us, on a repo we know well enough to tell the difference between a real finding and a bad one.

The honest limits

Four caveats we would want stated if we were evaluating this from the outside.

The efficiency numbers are from Flask and scikit-learn. They were chosen for reproducibility, not because they resemble your codebase. The harness is public specifically so you can generate your own figures instead of trusting ours.

Quality is at parity at best, and below parity on one of the two corpora. The claim is that the agent does the same work with far less reading. It is not that the agent becomes better at the work.

The short-task cost win depends on a lean tool profile. A fully loaded MCP surface eats most of the navigation saving on quick questions. If you enable everything and measure a small win, that is the expected result, not a defect.

The badge score and the 0.74 AUC are both real numbers with real ceilings. Neither is a claim that the codebase is clean or that the score is authoritative. They are the current state, published so the trend stays checkable.

Reproduce it

Everything above is in the benchmark repository: harness, configs, raw results, and the report the figures are quoted from. Point repowise at your own repo with repowise init, which needs no API key for the first index, or browse pre-indexed open-source repos to see the output on a codebase you already know.

FAQ

Are the 27x token numbers from the repowise codebase itself?

No, and that is deliberate. The token-efficiency and paired-agent benchmarks run on public repositories, Flask and scikit-learn, so anyone can reproduce them. The repowise codebase is where we use the tool daily. The public corpora are where we measure it.

What does "quality at parity" mean?

Both agent configurations were scored by the same judge on the same tasks. On Flask the means were 8.82 without repowise and 8.81 with it. On scikit-learn they were 8.72 and 8.23, so the repowise run scored slightly lower. repowise reduces the work an agent does to reach an answer, measured in tokens, file reads, and tool calls. It does not claim to improve the answer.

If the code is agent-assisted, why should I trust the checks?

Because the checks are deterministic and do not involve a model. The risk and health scores are computed from the same input every time and produce the same output, and the scoring function is calibrated against a public defect corpus with the cross-project validation quoted above. A change written by an agent and a change written by a person pass or fail that gate on identical terms.

Why show a live badge instead of quoting your best score?

Because a score nobody can check is not evidence of anything, and a score typed into a page goes stale the week after it is written. The badge reads from the live index, so it moves when the codebase moves, including when it moves the wrong way. The score is defect-validated at ROC AUC 0.74 with a published confidence interval, which is what makes it a measurement rather than a marketing figure.

Can I run the benchmark on my own repository?

Yes. The harness, configurations, and analysis scripts are in the public benchmark repository, and the token-efficiency script takes any git repo as input.

Does this require Claude Code?

No. The index is exposed over the Model Context Protocol, so it works with Claude Code, Cursor, Cline, Codex, and any other MCP client. One index serves every agent.