Process Metrics Beat Structural Metrics for Predicting Defects

Across the 21 repositories we studied, the files that bugs landed in were better predicted by how the file changed than by what it looked like. Co-change scatter (how many other files change whenever you touch this one) was the strongest single signal, ahead of every structural metric and ahead of file size. Change entropy and ownership risk came next. Complexity, nesting and god classes followed.

The usual instinct runs the other way. To find risky files, most tooling measures shape: cyclomatic complexity, nesting depth, function length, duplication, cohesion. That is a fair premise, since hard-to-read code feels buggier. We tested it against defect data, and shape turned out to be the second-tier signal. This post gives the finding, the method, and what to do with it. The metrics themselves are described in the complete code-health guide.

Process metrics and structural metrics, defined

Structural metrics read one snapshot of the code. They need only the file: complexity, nesting, long methods, god classes, function hotspots. Delete the .git directory and they still work. Seven of the study's 21 defect-risk markers are structural.

Process metrics (also called evolutionary or change metrics) read the repository's history. They cannot be computed from a snapshot. Nine of the 21 markers are process markers:

• Churn risk: how much a file changes.
• Co-change scatter: how widely a file's changes ripple into unrelated files (shotgun surgery).
• Hidden coupling: files that move together without importing each other.
• Change entropy: how chaotically a file's modifications are spread over time, after Ahmed Hassan's model.
• Ownership risk and developer congestion: how many contributors touch a file, and how many at once.
• Knowledge loss: code whose authors have left.
• Code age volatility: dormant code that wakes up.
• Prior defects: earlier bug-fix commits on the file.

The remaining five are test and coverage markers, a separate topic. The study used 21 markers; the shipped score has since grown to 25 defect-risk detectors, and the findings below describe the study's set.

Co-change is something you can see. The view above traces one file in the repowise repository to every file that changes in the same commits. A file with a wide fan like this is the kind the study's top marker flags.

The literature already pointed here

This result has been measured before, and the field has said the same thing for almost two decades.

• Moser, Pedrycz and Succi (ICSE 2008), A comparative analysis of the efficiency of change metrics and static code attributes for defect prediction, compared change metrics with static code attributes on Eclipse and found the change metrics substantially more predictive.
• Rahman and Devanbu (ICSE 2013), How, and Why, Process Metrics Are Better, showed across many systems that process metrics predict better and stay more stable across releases, while code metrics largely track size.
• Nagappan and Ball on relative code churn at Microsoft, and Hassan's change-entropy model, point in the same direction.

The explanation that fits all of it: static metrics are entangled with file size and say little once size is controlled. Process metrics measure something size cannot fake, which is where the team keeps struggling.

What the 21 markers show

The study fits a regularized logistic model over per-file marker hits, with file size (NLOC) included as an explicit control. Each coefficient therefore reads as defect lift beyond what size already explains. Files were scored before the bug window and checked against the bug fixes that followed (21 open-source repositories, nine languages). The full method and results are in the validation study.

The top three predictors are all process metrics: co-change scatter first, change entropy second, ownership risk third. Structural metrics appear after that, with nested complexity, complex conditionals and long methods. They carry real signal, in the second tier. The dashed line in the chart is file size itself, and co-change scatter sits above it: a marker that knows nothing about the shape of the code outpredicted the lines-of-code baseline that is notoriously hard to beat.

Two structural smells that get a lot of attention in review, duplication (dry_violation) and low cohesion, sit at or below zero on defect lift once size is controlled. They are scored under the maintainability pillar instead, where they measure real readability cost without being used as bug predictors.

Why history beats shape

Structure is the part of a file an author controls. A careful engineer keeps complexity down, splits long functions and names things well, so complexity partly measures diligence. Tidy files still contain bugs, because the bug was in the problem, not the prose.

History records what the author cannot groom away. Co-change scatter catches the file that cannot be touched without editing six others. Change entropy catches the file poked from every direction with no settled design. Ownership risk catches the file that ten people each touched once and nobody owns. These are the conditions under which people introduce bugs, and they are invisible to anything that reads only the current snapshot. A file can be beautifully factored and still sit at the unstable center of the change graph.

Measuring process metrics without leakage

Process metrics are the strongest predictors and the easiest to fake. They read a recent window of history, so measured carelessly they leak the answer. Score a file at HEAD and label it from the same recent window, and the bug-fix commits you are trying to predict are themselves churn, co-changes and entropy. The marker then detects the activity that defines the label, and the AUC looks excellent for the wrong reason.

We saw this directly. developer_congestion, which counts concurrent contributors in a recent window, looked strong under HEAD scoring. Re-run leakage-free, scoring each repository six months before its bugs landed, the signal collapsed, because the congestion was the fix commits arriving. Its weight was cut to the floor. Change entropy passed the same test as a real predictor. The only way to tell the two apart is to measure in the past, as the companion validation study describes.

Structural metrics are the floor

Structural metrics work where process metrics do not exist. A fresh checkout or shallow clone has no usable history. A pull request diff covers code that has not accumulated any. A vendored or generated file has history that is mostly noise. In all three, structure is what you have.

To measure how much structure carries alone, we scored the PROMISE/Jureczko jEdit benchmark as single-release snapshots: no git history, only structural markers active, no tuning to the dataset.

The structural-only score reached an AUC of 0.76 to 0.78 across two releases, within reach of the dataset's own cross-validated full-metric model, and it beat raw line count on the effort-aware Popt metric on both. Structure is not the strongest signal, but it is always available, deterministic and portable across languages.

The two families fail in opposite ways. Process metrics are powerful with deep, clean history and unreliable without it. Structural metrics are modest and never absent. A score built on only one of them is fragile in a predictable direction.

File size is the confound over both

File size is the strongest single predictor in defect prediction, and both families correlate with it. Every coefficient in this post is reported after controlling for size. We also weight effort-aware metrics such as Popt, which charges an inspection budget in lines, above raw AUC, which rewards flagging the big files. A claim that a metric predicts defects without a size control is, until shown otherwise, a claim that big files have more bugs.

What to do with this

• Weight evolutionary signals heavily where history supports them: co-change, change entropy and ownership.
• Measure them leakage-free or not at all. A churn or co-change number computed at HEAD against a recent label window is a confident wrong number. Ask any tool how it avoids this.
• Keep structural metrics for cold starts: fresh clones, PR diffs, generated code and new files.
• Control for size everywhere, and prefer effort-aware metrics when deciding what to review.
• Separate "looks messy" from "is risky." Duplication and low cohesion are maintainability signals. Treating them as bug predictors sends refactoring effort to tidy code while the risk sits in a clean-looking file at the center of the change graph.

Readable code is worth having on its own terms. If the question is where the bugs are, the answer is written more clearly in the history than in any single snapshot.

Try it on your repository

repowise computes both families and shows which signals fired on each file. Run repowise health in an indexed repository, or open a live example. The code health feature page covers how each marker is gated and weighted, and the benchmarks page lists the published measurements.