What an AI Code Reviewer Actually Flags: 1,000+ Findings by Priority

I counted what an AI code reviewer has said on my own pull requests. Not what I remember, what is on the record: over a thousand inline findings from Codex, collected on 2026-10-23 through the GitHub API from my own pull requests.

The split by the priority badge the reviewer puts on each comment:

• about one in five is P1,
• about three in four is P2,
• a few percent is P3,
• none is P0.

Most of what an AI reviewer says is the middle tier. That is the first lesson: the interesting work is in deciding what P2 means for you, because that is where the volume is.

What P1 looks like

A P1 is a finding the reviewer thinks can break something real. A few titles, quoted from the comment headings:

• JSON-LD content that is not escaped, so a stray character in a title breaks structured data on the page.
• A merge step that does not require the checked commit to still be the head of the branch.

That finding states a rule my merge script now enforces. A review is a statement about one commit. If someone pushes after the review, the statement is about code that no longer exists. So the script refuses to merge unless the approval names the exact head commit.

The self-review pinned to a commit

When the external reviewer is out of quota, I run the same review myself and post it as a comment that ends with one line: self-review: <full commit sha> verdict=pass. The script reads that line. Any later push changes the head, the line no longer matches, and the merge is blocked until the review is repeated.

I have many of these on record. Almost all say pass. Two say fixes-required, which is the verdict that records a review without approving. Two out of that many looks like a rubber stamp, and I cannot rule that out from the numbers alone. What I can say is that the gate is mechanical, so it does not depend on my mood at the time.

What I did not measure

Two numbers I would like and do not have:

• How many findings were fixed before merge. I never tagged findings as accepted or dismissed.
• The false-positive rate. Findings were never labelled right or wrong, so any percentage I quoted would be invented.

I also tried bucketing the findings by topic with keywords. It was too crude to quote, so I am not quoting it.

What I would do with this

If you run an AI reviewer, count its output once. The shape of the split tells you where to spend attention: the P1 tier deserves a human every time, the P2 tier deserves a rule you write down once, and the P3 tier deserves to be ignored without guilt. Then add the one thing no reviewer gives you for free: a record of what the review was a review of.

The practical version of that, for a team, is in A Claude Code Review Workflow That Saved 30 Minutes Per Engineer a Day. The setup of the agent files that feed it is in CLAUDE.md for an iOS Team, and the hook that stops the agent merging on its own is in Claude Code hooks on a real repo: what broke.