Incident Response: Finding the Change That Broke Production

repowise teamUpdated 8 min read
On this page

To find the change that broke production, pin the first bad deploy and the last good one, then narrow the commit range with git bisect if you can test each revision, or with a review of the files changed in that window if you cannot. Then widen the net: the offending change is often in a file that usually moves together with a file the deploy did touch. The goal in the first 30 minutes is not a perfect root cause. It is the smallest safe candidate set for a rollback decision.

Start with two anchors

Write down the first bad deploy and the last known good deploy before anything else. Those two SHAs, tags or config versions turn an incident into a bounded search.

A useful incident note is five lines:

  • service: payments-api
  • symptom: 500s on /capture
  • first bad deploy: 2026-05-20 14:12 UTC
  • last known good deploy: 2026-05-20 13:47 UTC
  • suspected area: retry middleware, HTTP client or token refresh

Also ask whether the bad state began with a deploy, a feature flag flip, a dependency bump or a config change. A flag flip leaves no commit, so check your flag and config history for the same window.

git bisect, git blame and git log -S

git bisect finds the commit that introduced a regression by binary search; git blame only reports who last edited a line. The git documentation describes bisect as a search between a known bad and a known good revision, and bisect run automates it with any command that exits 0 for good and 1 to 127 (except 125) for bad.

bash
git bisect start HEAD v1.41.0 --      # HEAD is bad, the last good tag is v1.41.0
git bisect run ./scripts/repro.sh     # exit 0 = good, 1-127 = bad, 125 = skip
git bisect reset
ApproachBest forWeak spot
git bisect runReproducible regressions with a pass/fail checkNeeds a test; slow for config issues and intermittent failures
git blameLine-level history and the right person to askPoints at the last edit, which is often a refactor, not the cause
git log -S "symbol"When a string or symbol appeared or vanishedNeeds a specific token
git diff <good>..<bad> --statA fast view of what the bad deploy changedShows what changed, not what matters
repowise get_risk and get_contextCo-change partners, owners, fix history, dependentsEvidence for where to look, not proof of cause

Scroll the table sideways to see every column.

Widen the net: what moves with the changed files

Production bugs are usually boundary bugs: a serializer changes and its parser does not, a schema moves and a migration lags. A line-level tool cannot see that. History can.

repowise reads git history during repowise init and repowise update and keeps four signals that matter in an incident, none of which needs an LLM key:

  • Co-change partners. Files that keep changing in the same commit. If the bad deploy touched file A but not its usual partner B, inspect B. See git intelligence.
  • Ownership. The primary owner and bus factor of each file, so you know who to page. The ownership layer computes it from git history.
  • Bug-fix history. How often a file was fixed in the last 180 days, with a bug_magnet flag for files that keep breaking. Churn tells you where work happens; fix history tells you where things break.
  • Dependents. Which files import the changed file, so you can see how far a bad change reaches.
Change coupling view for call_resolver.py in the repowise repository, showing the test files and modules that change in the same commits
Change coupling view for call_resolver.py in the repowise repository, showing the test files and modules that change in the same commits

The capture above traces call_resolver.py in the repowise repository: each line is a file that tends to change in the same commit, and line weight is the strength of the coupling. It is a hint about hidden relationships, not a verified dependency.

The commands

bash
repowise risk --target src/payments/retry.py   # owner, bus factor, fix history for one file
repowise context src/payments/retry.py --include ownership
repowise risk main..HEAD                       # score the bad deploy's range as one change

From an agent with the MCP server connected, each question is one call:

text
get_risk(targets=["src/payments/retry.py"])           # hotspot, fix record, owners, co-change partners
get_risk(changed_files=[...files in the bad deploy])  # PR mode: may_break, missing_cochanges, tests_to_run
get_change_risk(revspec="<good>..<bad>")              # what the range made worse, fix_history
get_why(query="retry policy for token refresh")       # decisions and git archaeology for the area

PR mode is the useful one at minute 15: missing_cochanges names files that usually change with the deploy's files but were left alone, and tests_to_run names the tests that cover the range. The MCP tools reference lists every field. repowise ships ten MCP tools.

A 30-minute workflow

Spend the first five minutes on the boundary, the next five on choosing a search mode, ten on narrowing, and ten on the decision.

Minutes 0 to 5: freeze the boundary

Record the two anchors and the exact symptom. Snapshot logs and metrics before more deploys muddy the signal. Note whether the failure is deterministic.

Minutes 5 to 10: pick the search mode

  • Reproducible with a test: git bisect run.
  • Tied to a recent deploy: review the range with git diff <good>..<bad> --stat.
  • Large service, broad suspect area: start from ownership and fix history to rank files.

Minutes 10 to 20: shrink the candidate set

  1. List the files changed in the bad deploy and drop docs-only and formatting-only changes.
  2. Run get_risk(changed_files=[...]) and read missing_cochanges and the dependents of shared helpers.
  3. Check the fix history of each remaining file. A recent run of fixes in a file raises it in your list.
  4. If one commit stands out, test it directly. If several do, bisect.

Minutes 20 to 30: roll back or patch forward

SignalRoll backPatch forward
Clear bad deploy boundary and old version still compatibleYesNo
Small surface, reproducible regressionYesLater
Cross-service falloutUsuallyRarely
Bug in a shared libraryDependsOften safer
Data migration already ranCheck firstOften

Scroll the table sideways to see every column.

Base the choice on blast radius and migration state. A rollback that meets a schema change can make the state worse.

What to capture for the postmortem

A postmortem that records which signal found the cause makes the next incident shorter. Keep one template:

  • first bad commit and last known good commit
  • changed files, primary owner, co-change partners
  • rollback decision and why
  • whether the failure was reproducible, and whether git bisect would have worked
  • which signal found it first

Because repowise stores decisions, an agent can later answer "why does this retry policy exist" with get_why instead of waiting for tribal memory.

Try it on your repo

bash
pip install repowise
repowise init
repowise risk main..HEAD

Indexing needs no API key. See the live examples, the change risk layer and the bug-fix history layer.

FAQ

How do I find who changed this code during an incident?
Start from the first bad deploy, not the editor. Use git bisect when the bug is reproducible, otherwise review the files changed in the deploy window and check each file's owner. git blame names the last editor of a line, which is not proof of cause.
Is git blame enough for incident root cause?
No. git blame reports who last touched a line, and that edit may be an old refactor while the real change was in an imported helper. Use commit-range evidence, related-file history and a pass/fail check where you can.
When should I use git bisect in production incidents?
Use it when you can test each revision quickly, ideally with a script passed to git bisect run. If the failure depends on live traffic, timing or config, combine deploy-time evidence with a review of the range.
What should I check before deciding to roll back?
Check the first bad and last good deploy, the files changed, whether shared helpers or schema code changed, and whether a migration already ran. If the old version is still compatible with the data, rollback is usually the fastest safe path.
How does repowise help find the change that broke production?
repowise adds co-change partners, owners, bug-fix history and dependents to the files in a deploy range, through repowise risk or the get_risk and get_change_risk MCP tools. It narrows where to look; it does not replace a bisect.