Do Code Health Flags Find Buggy Files? 199 Repos, Past Fixes Taken Out

Do code health flags find buggy files? Across the 199 public repos our current scoring engine has scored, the 20 files with the lowest health score had a bug fix in the last 180 days 61% of the time, against 17% for a file picked at random. One of the score's own inputs is past bug fixes, which partly gives the answer away. With that input taken out and the files ranked again, the flagged files still had a recent fix 51% of the time, about three times the rate for a random file.

The rest of this post covers the check, the leak, how much the result depends on the base rate, and how to read the number for your own repo. The check runs on every repo repowise indexes with enough history.

repowise's health score (1 to 10) rates how likely a file is to cause bugs and how hard it is to change, using static checks plus git history. The repo score is the average across files, weighted by lines of code. See the code health guide for how it is built.

The check

Every time we index a repo with enough history, we run the same small test:

1. Sort the files by health score, worst first.
2. Take the worst 20.
3. Count how many had at least one bug-fix commit in the last 180 days.
4. Compare that with the share of all files that had one, which is the base rate.

The ratio between the two is the lift. A lift of 1 means the score did no better than picking files at random, and a lift of 3 means the flagged files were three times as likely to have been fixed. We skip repos where the test would be noise: fewer than 25 files, or fewer than 5 files with a fix in the window.

The leak: the score sees one of its own answers

The label for "had a recent fix" comes from one of the score's own findings, called prior_defect. When a file had bug-fix commits in the last 180 days, that finding fires and takes points off the file's score. So a file that was fixed recently scores lower partly because it was fixed recently, and the check above rewards the score for seeing its own answer. prior_defect has a normal weight of 1.0, the same as most other checks, and it overlaps with the checks that look at how often a file changes.

To measure how much that matters, we downloaded every file's score and every prior_defect finding for the 199 repos from our public API, added each file's prior_defect deduction back onto its score, ranked the files again, and re-ran the check.

Taking the leak out costs about 10 percentage points of precision and lowers the median lift from 4.0 to 3.3. Most of the signal is still there, so it comes from the other checks: how complex and deeply nested the code is, how many people change the file and how often it changes. Some of those checks are themselves related to past fixes, because a file that changes a lot also tends to get fixed a lot, so the second column still isn't a fully clean test.

For day-to-day use, the last row is the useful one. Even without the leak, the least healthy fifth of a repo's files held 55% of the files that got bug fixes, so that fifth is where an afternoon of cleanup should go.

Lift depends on the base rate

Lift is a ratio, so it has a ceiling. If half the files in a repo had a fix recently, even a perfect pick of 20 can only reach a lift of 2, and if only 5% did, the ceiling is 20.

A high lift therefore partly means that the repo has few fixes and the score found them, and a low lift in a repo where every other file gets fixed doesn't say much. That's why we quote precision and the base rate together with the lift.

Where the signal is weakest

With prior_defect included, all 199 repos beat chance. Without it, 6 came in below a lift of 1, and 4 of those had zero hits. Four of the six are GitHub forks of other projects, where most of the commit history belongs to the original project. The other two are small: one has 343 files and the other 56.

The pattern we'd take from this is that the score fails where the history doesn't belong to the code. we didn't find a repo with an active, normal history where the flags clearly pointed the wrong way, but 199 repos is a limited sample.

Scope of this check

It looks backwards. We score the files as they are today and check fixes from the past 180 days. A forward test scores files at a date in the past and checks fixes after it. Our earlier study on 21 repos did that, with a time split, and got a ROC AUC of 0.74. This one trades that rigour for scale.

Bigger files get more of everything. A 2,000-line file gets more commits, more fixes and more findings than a 50-line one. The score's weights were fitted with an explicit control for size, but a look-back test like this one can't fully separate an unhealthy file from a large, busy one.

"Fix" means the commit message says so. We count a commit as a fix if its subject contains the whole word fix, bug, patch or resolve(s), or says "fixes #N" or "closes #N". Merge commits are left out, and so are commits whose subject mentions typo, bump, deps, chore, lint, format, style or docs. Because the match is on whole words, a subject like "Fixed crash" isn't counted, and projects that write vague commit messages are under-counted.

One engine version. Our scoring engine changed on 13 September 2026, and these 199 repos are the ones scored since then. Across all 899 public repos with this check, including those scored by the older engine, the flagged files had a recent fix 63% of the time against a 19% base rate, with prior_defect included. we didn't re-run the leak-free version on the older 700.

So we'd describe the result like this: with its own bug-fix input taken out, the score still points at the files where fixes happen, about three times better than chance, across 199 different codebases. That is a good sign the score measures something real, and it doesn't prove that fixing those files will prevent future bugs.

How to use this on your own code

Every repo page on repowise shows the check with prior_defect included. The overview has a "Score validation" line (for example "14/20, 5.81× baseline"), and it links to the code health page where you can see the flagged files. Paste any GitHub URL to see it for yours. If the lift is high, start your cleanup with those files. If it's near 1, the score isn't telling you much about that repo, and you should lean on other signals.

How we measured

The sample is every public repo whose latest index was scored on or after 13 September 2026 and passes the minimums above (at least 25 scored files and at least 5 with a recent fix): 199 repos, checked on 6 October 2026. "Recent fix" is a non-merge commit in the 180 days before the latest commit whose subject matches a fix pattern and no exclusion pattern. For the leak-free version, we downloaded every file's score from api.repowise.dev/snapshots/<id>/health/files and every prior_defect finding from /health/findings?biomarker=prior_defect, added each file's prior_defect deduction back (capped at 10), and ranked worst-first again. Our ranking of the original scores reproduced the stored result exactly in 198 of the 199 repos and within one file in the last one. The code for the check is open source in packages/core/src/repowise/core/analysis/health/defect_accuracy.py.