Building a Deterministic PR Review Bot With Zero LLM Calls

repowise teamUpdated 9 min read
On this page

A deterministic PR review bot works as a pipeline: parse the changed files, query a dependency graph and git history, score the result with fixed rules, and render one comment from templates. No model reads the diff, so the same pull request gets the same comment on every run, and the review costs nothing to run. This post shows the design we use in the repowise PR bot's Signals mode, and why a no-LLM pass is the right base layer even if you add a model later.

Part of the change risk guide.

Why build a PR review bot with no LLM calls?

Because most of what a reviewer needs and cannot see in a diff is structural, and structure can be computed. A model can read a hunk. It cannot know, from the hunk, that nine functions outside the pull request call the symbol whose signature just changed. A graph can.

Three properties follow from taking the model out of the base layer:

  • Repeatability. Run the bot twice on the same head commit and you get the same words. People start making merge decisions from a bot's comment, so a bot that changes tone between runs loses trust fast.
  • Cost. The online path is a graph query and string formatting. In repowise's own repository, measured over 30 days to 12 October 2026 across 291 pull requests, Signals mode made no model call at all. The optional AI review costs about 15 cents a pull request on average, so 100 reviewed pull requests cost about $15 of pay-as-you-go credits, with no per-seat fee.
  • Auditability. Every line of the comment traces to a fact: a call edge, a commit pair, a score. You can test the output with plain assertions.

This is a base layer, not an argument against models. Our AI review consumes the same signals and turn them into review prose. The point is that the signals do not need a model to exist. For a cost and feature comparison across the market, see best AI code review tools, and for cost per 100 pull requests, AI code review pricing, compared.

What does the pipeline look like?

Eight steps, and only the first two touch GitHub. Everything after the diff arrives is computed from an index that already exists.

  1. Receive the pull request webhook and verify its signature.
  2. Fetch the diff and the changed file list.
  3. Map each changed span to the symbols it touches.
  4. Query the dependency and call graph for callers, importers and tests.
  5. Query git history for co-change pairs, churn and ownership.
  6. Score the changed files before and after with the health model.
  7. Apply the silence rules.
  8. Render the comment, then create or edit it, and update the check run.

The cost sits in the index, not the request. The graph and git index are built once and updated incrementally on each push to the default branch, so a pull request only reparses the files it touches. Doing the whole repository per pull request is the most common way these bots end up slow.

How do you parse the diff into symbols?

With a real syntax tree, not regexes. Tree-sitter produces concrete syntax trees for many languages, which is enough to tell a function body from a class declaration and to find the symbol a changed line sits in. repowise parses 28 languages to a full AST this way.

The step that matters is changed-span mapping. If a pull request edits one function in a large file, the bot attaches its findings to that symbol, not to the whole file. That is what lets a signature change be reported at symbol level while a body-only edit stays quiet.

How do you find what a change breaks?

Store files and symbols as nodes, imports and calls as directed edges, and ask the graph. A directed graph library such as NetworkX is enough for repositories of normal size, and it is what the repowise graph builder uses. The queries a review bot needs are small:

QuestionGraph queryShows up in the comment as
Who calls the symbol whose signature changed?Incoming call edges, minus nodes inside the pull request"Called by 9 symbols outside this PR"
Which tests import what changed?Importers of the changed files that are test files"Run these test files first"
Is this a move, not a change?Same symbols, new file, edges preserved"15 symbols moved. No caller has to change."

Scroll the table sideways to see every column.

Call resolution is never perfect, so the bot reports what the graph resolved and links to a full list instead of guessing at the rest. A bot that is wrong once about a caller loses more trust than one that is occasionally incomplete. The set of callers and files a change reaches is its blast radius; the change risk entry covers how a change is scored.

How does git history add signal?

It tells you which files move together, who owns them and where bugs have landed before. None of that is in the diff. Co-change pairs are the strongest example: if two files changed together in eleven past commits and only one is in this pull request, the missing one is worth a line.

text
.../perf/dialects/python.py changed with .../perf/dialects/ts_js.py
in 11 past commits and is not in this PR

Ownership turns the same history into a reviewer suggestion: the people who own the risky files are named. Churn and bug-fix history feed the hotspot signal, which is how a bot can say a file is risky with evidence instead of a feeling. The git hotspot guide covers the underlying analysis.

How do you score health without a model?

With fixed detectors whose output is measured, normalized and combined by explicit rules. repowise's code health model runs 59 markers and keeps three signals apart instead of blending them: defect risk (23 markers), maintainability (12 markers) and performance (22 markers). A bot built on it can say which signal moved, not just that a grade changed.

The bot scores the changed files before and after the pull request and reports the movement:

text
Health of changed files: 7.1 → 7.3 (+0.1)

The scorer should be boring. If a change adds a complex method to a file that already has several, the bot says so every time, in the same words. The code health complete guide lists the markers.

What does the comment look like?

Short, ranked and specific, with the detail one click away. This is a real Signals comment on a pull request in the repowise repository: three things to check, the health movement of the changed files, and a link to the full report.

A real repowise-bot comment on a repowise-dev/repowise pull request: three things to check, health of changed files 7.1 to 7.3, and a link to the full report
A real repowise-bot comment on a repowise-dev/repowise pull request: three things to check, health of changed files 7.1 to 7.3, and a link to the full report

Every item is assembled from a template that takes facts as input. A template looks like "{file} changed with {partner} in {n} past commits and is not in this PR". That does two things. It keeps the comment terse, because reviewers ignore comments that ramble. And it makes the output testable: feed in a fixture and assert the rendered text contains the facts.

Keep the structure fixed: a risk summary, up to three pieces of evidence, one suggested action, and a link to the owner, the dependency path or the health view for anyone who wants more.

When should the bot stay silent?

Whenever it has nothing grounded to say. Silence is a feature, and it is the main reason a bot stays installed. The Signals rules speak only when something specific fires:

  • a contract changed and callers outside the pull request depend on it
  • a file that usually changes with this one is missing
  • tests that import the changed code are missing from the change
  • health of the changed files dropped

A pull request with none of these gets no comment. Improvements alone never trigger one. Docs-only and config-only changes usually fall out of this naturally, because none of the signals fire.

The same discipline covers failures. If a changed file fails to parse, the graph snapshot is missing or the diff exceeds the configured budget, the bot should post a short operational note or fail its check run with a reason. It should never fill the gap with a plausible guess.

How do you avoid spamming the pull request?

Own exactly one comment per pull request and edit it on every push. The recipe is short:

  1. Find the bot's existing comment on the pull request.
  2. If it exists, edit it in place. If not, create it.
  3. Key the stored comment ID by repository and pull request number.
  4. Treat the head SHA as the version boundary, so a replayed webhook for the same SHA is a no-op.
  5. If a later push clears every signal, delete the stale comment.

GitHub's check runs API gives structured status output a proper home next to the comment. Verify the webhook signature, deduplicate on the delivery ID, and retry with a bounded backoff. The result is that a reviewer can glance at a pull request and know the bot has spoken once.

What should you expect to ship?

Plan for permissions and trust more than for features. Teams install a bot into someone else's workflow, so the product story has to be easy to state. The one we ship is: it analyzes your code, git history and dependency structure, writes one comment, and in Signals mode sends no line of your code to a model.

  • Request the narrowest permission scope the bot needs, and say why on the install page.
  • Keep any merge gate off by default. A bot that fails the build on day one gets uninstalled on day one. repowise gates are opt-in and set per repository.
  • Publish what you measured. Our own Signals figure above comes from one repository's credit ledger over a stated 30-day window.

An open source base also helps: users can inspect the logic, self-host it and tune thresholds. repowise is AGPL-3.0 licensed, the license written for network server software (GNU AGPL text). The same signals are available outside GitHub: repowise risk scores a change from git alone, and the change risk guide shows how to gate a CI job on it.

How do you try this on your own repository?

Install the repowise GitHub App on a public repository and open a pull request. Signals mode is free there and needs no key. To see the same analysis locally before you install anything:

bash
pip install repowise
repowise init --yes
repowise risk main..HEAD

For the full set of modes and what each one costs, read the PR bot page. To see the underlying graph on real code, browse a live repository.

FAQ

What is deterministic PR review?
It is pull request feedback built from fixed rules, code structure and repository data instead of model-generated text. The same input always produces the same output, which makes the bot testable and its comments reproducible.
How is no-LLM code review different from AI code review?
A no-LLM bot computes its findings from static analysis, a dependency graph and git history. An AI review bot asks a language model to infer feedback from the diff. The two combine well: repowise's AI review starts from the same deterministic signals.
What should a GitHub bot architecture include for PR review?
At minimum: webhook intake with signature checks, a diff fetcher, a parser, a dependency graph, a scorer, a comment writer, and idempotent storage keyed by pull request and head SHA. Use the check runs API for structured status.
How do you score code health in a PR bot?
Use measurable detectors, score the changed files before and after, and report the movement and the signal that moved. repowise runs 59 markers and keeps defect risk, maintainability and performance as separate signals.
Can an open source PR bot be useful without an LLM?
Yes. Callers outside the pull request, files that usually change together, tests that import the change and health movement are all structural, and they are often more actionable than a model's prose. Add a model on top when you want findings written up as review comments.
Why use Tree-sitter and NetworkX together?
Tree-sitter gives syntax-aware parsing across languages, and NetworkX gives graph queries over what was parsed. Together they turn a repository into a map you can query per pull request in seconds.