Code Review by Model
Why review is a better fit for current models than authoring, the precision problem that determines whether it is used, and the classes of finding worth reporting.
Reviewing a change is a better-shaped task for a model than writing one. The diff is bounded, the question is specific, the output is short, and a wrong comment costs a moment of a reviewer's attention rather than a bug in production. It is also the task where precision matters more than anywhere else, because the failure mode is not a missed bug but a channel people stop reading.
The precision problem
A reviewer receiving twenty comments of which two are useful learns to skim, and once skimming starts the two useful ones are missed as well. The value of an automated reviewer is not the number of true findings but the ratio, and a system reporting fewer, higher-confidence issues is more useful than one with better recall and worse precision.
This inverts the usual instinct. It is correct to suppress a finding the system is unsure about, even a real one, because a false positive costs more than a missed issue in a channel that also has human reviewers.
What a model reviews well
Local correctness in the diff. Off-by-one errors, null handling, resource leaks, incorrect error propagation, misuse of an API in a way visible from the call. These are contained in the changed lines and their immediate context, which is where a model has full information.
Consistency with the surrounding code. Whether the change follows the conventions of the file it is in, which requires only the file and is a common review comment humans make.
Missing cases. A conditional handling three of four enum values, a new error path with no test, an added parameter not threaded through a caller.
What it reviews badly
Architectural fit, which requires knowing where the system is going, why previous decisions were made, and what is planned. None of that is in the diff.
Whether the change is the right change. A correct implementation of the wrong thing reviews cleanly.
Cross-cutting consequences, where the effect appears in a component the diff does not touch and the model did not retrieve.
Security in general. A model finds pattern-matched vulnerabilities and misses ones requiring reasoning about trust boundaries and data flow across the system, and a clean review is not evidence of absence.
When it breaks
Reviewers defer to confident comments. A fluent, specific and wrong comment gets acted on, particularly by a junior developer who assumes the tool knows something. Comments should carry their reasoning so they can be evaluated rather than obeyed.
Coverage is mistaken for assurance. A change reviewed by a model and merged without human review has been checked for the classes the model handles and not for the ones it does not. If the model's review displaces human review rather than preceding it, the net effect on quality can be negative.
Diff-only context misses the reason. Reviewing a diff without the issue it addresses or the surrounding file produces comments about code that is correct in context, which is a large share of the false positives.
The comments train the code. If developers write code to avoid triggering the reviewer, the tool has become a style enforcer with opinions nobody agreed to, and its findings stop being evidence about quality and start being evidence about compliance.
12 flashcards for this concept
Click a card to reveal the answer.