Reviewing AI-written pull requests: where do you start?

Share on Twitter Share on LinkedIn
You will find the 🇫🇷 French version of this article here
TL;DR (AI)
  • Agents write pull requests faster than anyone can review them, so you have to pick which files to read.
  • Cribelo draws a reading map for a pull request: what is left to read once moved code is set aside, what the diff does not show, and the order to read it in.
  • There is no LLM inside. It runs on static analysis and git history, so the same diff gives the same plan on every machine, and that plan can also steer an AI reviewer.
  • I tested it against the history of 28 open-source repositories and the comments of real reviewers.
Summary generated by AI to help you skim.

Pull requests are arriving faster and getting bigger, often because an AI agent wrote them. The number of reviewers hasn’t changed, and they are getting tired.

I’m not the only one saying it:

SW Simon Willison Co-creator of Django · December 2025
… the junior engineer, empowered by some class of LLM tool, who deposits giant, untested PRs on their coworkers […] and expects the “code review” process to handle the rest. simonwillison.net ↗
RV Rémi Verschelde Godot maintainer · Bluesky, February 2026
Honestly, AI slop PRs are becoming increasingly draining and demoralizing for #Godot maintainers. bsky.app ↗

The numbers point the same way. Faros followed more than 10,000 developers. In the teams that use AI the most, merged pull requests went up 98%, their size 154%, and review time 91%.

A year ago, I wrote about a team that went from nineteen days to three to merge a pull request. The problem has changed shape since then. Review time doesn’t grow with the volume, so you scroll through the diff, spot what looks important, and read that. Reading everything is no longer an option.

That triage depends on things that have nothing to do with risk: the alphabetical order of the Files tab, the file names you recognize, the time of day. The file that got seven fix commits in three months sits somewhere in the middle, between two test files. Nothing sets it apart.

So I asked myself a simple question: if I only have time to read five files, which ones should they be?

I tried static analysis, then home-grown machine learning models, and ended up writing Cribelo, a command-line tool that answers that question. Here is the comment it posted on a pull request in AST Metrics, my own repository:

The pull request changes thirty-four files. Cribelo has me read thirteen, in the order to read them, in about forty-five minutes. For each one it separates the lines to read closely from the lines I can glance at, and every range opens the diff at the right spot.

The other twenty-one files are listed with the reason I can leave them aside. Cribelo also points out that no test runs the 25 changed lines of util.go. The diff doesn’t show that.

The reasoning is the one an experienced reviewer would follow. A central file whose complexity goes up, with few tests and a history of fixes, deserves a real read. Moved code, with no new import and no new branch, can wait.

Why an AI reviewer isn’t enough

I use AI code review tools, and they don’t solve this. They comment on everything with the same confidence, a typo as much as a regression in the aggregator. I end up with forty remarks to read on top of the pull request.

They are also slow and expensive. Reviewing a hundred-file pull request means sending those hundred files and their context to a model, on every push.

A small model like Jev, which I wrote about in September, is fast and cheap. What it lacks is the big picture: it only judges what you show it, a file or a hunk. It doesn’t know that 83 files import this one, that this function was fixed seven times this year, or that it is a copy of another function the pull request doesn’t touch.

Static analysis computes that context in a few seconds, with no GPU and no network call: the dependency graph, complexity, clones, git history. Cribelo uses it to tell you where to look, and stops there. It doesn’t comment on the code and doesn’t suggest refactorings.

Since there is no language model inside, the same diff produces the same plan on your machine, on mine, and in CI.

LLMs still have a role. An AI reviewer given six files in the right order, with the facts to check, is more likely to say something useful than one handed the whole diff. cribelo describe prints the plan as Markdown, ready to paste into a prompt.

Where the idea comes from

I have been measuring code for more than ten years, with PhpMetrics and then AST Metrics. One thing always bothered me about those metrics: you look at them after the fact, in a report, a bit like reading yesterday’s weather forecast.

Complexity, coupling, clones and fix history are fairly good at telling you where trouble will come from. But they stayed locked in a dashboard. Cribelo shows them in the pull request, right when someone is about to review it.

The tool is open source, under the MIT license. A tool that judges your code should be open to challenge: you should be able to see the weight that put a file at the top of the list and redo the math. Your code stays on your machine, with no account and no server.

How Cribelo picks the files to review

Cribellum is Latin for a small sieve. Cribelo takes the diff of your branch, parses the before and after with the AST Metrics parsers, and works in three steps. Here is the same pull request in a terminal:

Step 1: find what actually needs reading

A diff compares lines. Reformat a file and git paints it red and green from top to bottom, even though the program does exactly what it did before. Cribelo compares tokens, the words of the language as the parser splits them. When the removed and added lines hold the same tokens, whitespace aside, it is formatting, and there is nothing to read.

Moved code takes a little more work. When a function leaves one file and shows up in another, Cribelo compares the two bodies token by token, ignoring variable names, and pairs them when they are nearly identical. The method comes from RefDiff, published by Silva and Valente in 2017, which detects refactorings through token similarity without needing type information. It also recognizes a function extracted from another one, and a rename applied the same way across the whole change.

These changes can be checked at a glance, and a file that contains nothing else joins the skipped files, with its reason. Everything else changes behavior. A function that gains a branch, or was already complex, gets read closely. A function that stays simple, with no new decision, gets a glance.

Step 2: surface the internal and architectural changes a plain diff can’t show

That is the Worth knowing block. It holds no predictions, only facts you can check in a minute:

  • an import that creates a cycle;
  • an exported function whose signature changes while other files call it;
  • a changed function whose clone was left untouched;
  • a function that has already been fixed several times this year;
  • changed lines that no test runs.

Fix history comes from git log -L, which follows a function through its commits. Coverage comes from the test report, when the repository has one.

Step 3: compute the best reading order

Files that import or call each other form a group. Within a group, a definition comes before its uses and a test comes after the file it covers, because a caller is easier to understand once you have read what it calls.

That is why the plan starts with library_reach.go. The file is new, it contains a function with a complexity of 17, and 83 other files depend on it.

This order builds on the work of Baum, Schneider and Bacchelli on the order in which to review the changes of a code review. The exact rules for each step are in how it is computed.

What the measurements show

To check that Cribelo does more than look smart, I used cribelo backtest to replay the history of 28 open-source repositories across the seven languages, 2,500 merges per repository.

Out of 131,000 touched functions, the ones Cribelo marks for a glance were later repaired in 1.3% of cases, against 2.8% for the ones it marks for a close read. Glancing halves the risk. But a third of the functions repaired later were in the glanced part, so they still need a glance, not a skip.

I also compared the facts with the comments of real reviewers, on 100 pull requests from prometheus, pydantic, cargo and vite. On average, 17% of changed files get a comment. For files that carry a clone, it is four out of five. For files with a fix history, one in two, and one in three when a signature changes.

Setting aside mechanical code removes little on an ordinary pull request: less than one line in ten, even though one file in seven is left with nothing to read. It pays off on very large changes. In ruff, the pull request that splits up types/class.rs moves more than 8,000 lines without git detecting a single rename. Cribelo sets them aside.

The details of every measurement are on the how it works page.

Try it

curl -fsSL https://cribelo.github.io/install.sh | sh   # or: brew install cribelo/tap/cribelo

cribelo                      # the reading map of the current branch
cribelo --budget 30m         # what fits in thirty minutes
cribelo skip                 # the files set aside, with the reason
cribelo describe             # the map as Markdown, for the pull request description
cribelo explain path/to/file # everything Cribelo knows about one file

Linux, macOS and Windows binaries are on the releases page. Seven languages are supported for now: Go, PHP, Python, Rust, TypeScript, Java and C#. If yours is missing, you can add it: the AST Metrics contributing guide explains how to plug in a new language, and contributions are welcome.

In a pull request, twelve lines of YAML are enough. The plan arrives as a single comment, the one in the first screenshot, updated on every push, and with: { budget: 30m } caps it to the time you have.

name: cribelo
on: pull_request
permissions:
  contents: read
  pull-requests: write
jobs:
  cribelo:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
        with: { fetch-depth: 0 }
      - uses: cribelo/cribelo@v2

The Action can also fail the job when reading time goes over a threshold (fail-over) or when the pull request contains certain facts (fail-on). I would wait a few weeks before turning that on. A team happily adopts a tool that shows it where to look, much less one that blocks its merges.

To start, run cribelo on the last pull request you approved by scrolling through the diff, and look at the first file on the list. Had you opened it?

If it gets things wrong, run cribelo backtest and post your number in an issue, especially if it is a bad one. The next version will be built on those counter-examples.

An open-source project lives longer when people know about it. If Cribelo showed you a file you were about to let through, give it a star on GitHub. I’ll admit it makes my day :)

© Jean-François Lépine, 2010 - 2026 · RSS feed