[{"data":1,"prerenderedAt":32},["ShallowReactive",2],{"blog-post-three-reviewers-18-prs":3,"blog-index":15},{"slug":4,"title":5,"description":6,"date":7,"updated":7,"tags":8,"image":12,"html":13,"reading_minutes":14},"three-reviewers-18-prs","Three AI code reviewers, same 18 PRs: two mostly found a subset of the third","On identical input, findings ranged from 40 to 2, and almost everything the smaller reviewers found, the largest found too. What a second reviewer is for.","2026-10-14",[9,10,11],"ai-review","code-review","engineering","https:\u002F\u002Fcontent.presentr.me\u002Fveridu\u002Fassets\u002Fog.66cdf9188b.png","\u003Cp>A common argument for running several AI reviewers on one pull request is coverage: each model notices different things, so together they catch more. We expected to see that when we ran three reviewers on the same 18 pull requests.\u003C\u002Fp>\n\u003Cp>We mostly did not. One reviewer raised 40 findings. The other two raised 7 and 2. Almost everything the two smaller reviewers found, the largest one found too.\u003C\u002Fp>\n\u003Cp>This post separates what we observed directly from what depends on judging whether findings were right. The main result needs no judgment about whether findings were correct, only about which findings describe the same issue.\u003C\u002Fp>\n\u003Ch2 id=\"the-setup\">The setup\u003C\u002Fh2>\n\u003Cp>We took 18 merged public pull requests from Flask, Requests, Fastify, Got, Laravel and Gin, covering Python, TypeScript and JavaScript, PHP and Go.\u003C\u002Fp>\n\u003Cp>Three reviewers ran on each one: \u003Ccode>claude-opus-5\u003C\u002Fcode>, \u003Ccode>gpt-5.5-2026-04-23\u003C\u002Fcode> and \u003Ccode>gemini-3.1-pro-preview\u003C\u002Fcode>. All three received the same prompt for each pull request, built once. Context retrieval ran once, so every reviewer saw the same surrounding code.\u003C\u002Fp>\n\u003Ch2 id=\"what-we-observed-directly\">What we observed directly\u003C\u002Fh2>\n\u003Cp>These numbers are counts. They do not depend on deciding whether any finding was correct.\u003C\u002Fp>\n\u003Ctable>\n\u003Cthead>\n\u003Ctr>\n\u003Cth>Reviewer\u003C\u002Fth>\n\u003Cth style=\"text-align:right\">Findings\u003C\u002Fth>\n\u003Cth style=\"text-align:right\">Pull requests with at least one\u003C\u002Fth>\n\u003Cth style=\"text-align:right\">Cost for all 18\u003C\u002Fth>\n\u003C\u002Ftr>\n\u003C\u002Fthead>\n\u003Ctbody>\n\u003Ctr>\n\u003Ctd>\u003Ccode>claude-opus-5\u003C\u002Fcode>\u003C\u002Ftd>\n\u003Ctd style=\"text-align:right\">40\u003C\u002Ftd>\n\u003Ctd style=\"text-align:right\">16\u003C\u002Ftd>\n\u003Ctd style=\"text-align:right\">$3.77\u003C\u002Ftd>\n\u003C\u002Ftr>\n\u003Ctr>\n\u003Ctd>\u003Ccode>gemini-3.1-pro-preview\u003C\u002Fcode>\u003C\u002Ftd>\n\u003Ctd style=\"text-align:right\">7\u003C\u002Ftd>\n\u003Ctd style=\"text-align:right\">4\u003C\u002Ftd>\n\u003Ctd style=\"text-align:right\">$1.71\u003C\u002Ftd>\n\u003C\u002Ftr>\n\u003Ctr>\n\u003Ctd>\u003Ccode>gpt-5.5-2026-04-23\u003C\u002Fcode>\u003C\u002Ftd>\n\u003Ctd style=\"text-align:right\">2\u003C\u002Ftd>\n\u003Ctd style=\"text-align:right\">2\u003C\u002Ftd>\n\u003Ctd style=\"text-align:right\">$1.72\u003C\u002Ftd>\n\u003C\u002Ftr>\n\u003C\u002Ftbody>\n\u003C\u002Ftable>\n\u003Cp>That is a twentyfold difference in volume on identical input.\u003C\u002Fp>\n\u003Cp>To measure overlap, findings that describe the same defect on the same pull request were grouped into one issue. Grouping is a judgment about whether two findings are the same, not about whether either is true. Two independent groupings came out at 40 and about 42 distinct issues. We use the first below.\u003C\u002Fp>\n\u003Cul>\n\u003Cli>8 of the 40 issues were raised by more than one reviewer.\u003C\u002Fli>\n\u003Cli>1 was raised by all three: Got's tests had started depending on live network access to \u003Ccode>example.com\u003C\u002Fcode>.\u003C\u002Fli>\n\u003Cli>\u003Cstrong>All 9 findings from the two smaller reviewers were issues the largest reviewer also raised.\u003C\u002Fstrong> Under the stricter grouping, 2 of the 9 were not, so the two smaller reviewers added at most 2 issues.\u003C\u002Fli>\n\u003Cli>32 of the 40 issues came from the largest reviewer alone.\u003C\u002Fli>\n\u003C\u002Ful>\n\u003Cp>So \"agreement is rare\" is true from one side only. From the largest reviewer's side, most of what it said went uncorroborated. From the smaller reviewers' side, nearly everything they said was corroborated.\u003C\u002Fp>\n\u003Cp>This is not the shape every comparison finds. A \u003Ca href=\"https:\u002F\u002Fdev.to\u002F_vjk\u002Fbest-ai-code-reviewer-in-2026-we-ran-4-in-parallel-for-3-weeks-146-prs-679-findings-1c0f\" rel=\"noopener\">public four-tool study\u003C\u002Fa> of 146 merged pull requests reported 93.4% of 617 locations caught by exactly one reviewer. Different tools, different code, a different result. Overlap is something to measure on your own changes, not assume.\u003C\u002Fp>\n\u003Ch2 id=\"where-they-overlapped-they-did-not-always-agree\">Where they overlapped, they did not always agree\u003C\u002Fh2>\n\u003Cp>Overlap means two reviewers looked at the same thing. It does not mean they reached the same conclusion.\u003C\u002Fp>\n\u003Cp>In one Laravel pull request, two reviewers flagged \u003Ccode>array_last()\u003C\u002Fcode>. One said it might not exist on supported PHP versions. The other said it was undefined and rated that high severity. Both claims can be checked against the repository: \u003Ccode>array_last()\u003C\u002Fcode> is native in PHP 8.5, the repository polyfills it (\u003Ccode>composer.json\u003C\u002Fcode>, line 61), and it is used throughout the codebase, including inside \u003Ccode>Arr::last()\u003C\u002Fcode>. A majority vote between those two reviewers would have sent someone to fix working code.\u003C\u002Fp>\n\u003Cp>In a Got pull request, two reviewers pointed at the same line of \u003Ccode>timed-out.ts\u003C\u002Fcode> and described opposite failures.\u003C\u002Fp>\n\u003Cp>Those disagreements were more informative than the agreements. They marked exactly the claims worth checking.\u003C\u002Fp>\n\u003Ch2 id=\"what-depends-on-adjudication\">What depends on adjudication\u003C\u002Fh2>\n\u003Cp>The rest of this section relies on judging findings, so treat it with more caution. The method and its weaknesses are below.\u003C\u002Fp>\n\u003Cp>Of the 49 findings, 45 were judged accurate, 2 plausible but unproven and 2 wrong. Accurate is not the same as worth a developer's time. Against a strict definition (a demonstrated failure under a supported contract, shown by a reproduction or a full code trace), two judges agreed on 7 issues. All 7 were raised by the largest reviewer; 2 of them were also raised by another reviewer.\u003C\u002Fp>\n\u003Cp>That suggests the same thing as the raw overlap. On these pull requests, extra reviewers did not surface important issues the largest one missed. It also says most accurate findings on mature, already-reviewed code are not worth stopping for. We do not read anything here as a ranking of the models: two of them made too few findings to rank, and finding count is not quality.\u003C\u002Fp>\n\u003Ch2 id=\"what-this-means-for-multi-reviewer-review\">What this means for multi-reviewer review\u003C\u002Fh2>\n\u003Cp>On these 18 pull requests, the second and third reviewers bought little coverage. What they did provide was a cross-check. Where they agreed, a finding had independent corroboration, which still is not proof. Where they disagreed, they pointed at the claim to verify first.\u003C\u002Fp>\n\u003Cp>That matches how Veridu uses its two reviewers. A finding can block a merge only when both report the same defect at the same place and both rate it serious enough to block. Findings from one reviewer are still shown, because in this corpus that is where most of the important issues were. Each finding is asked to state what it checked and what it concluded, so a claim like the \u003Ccode>array_last()\u003C\u002Fcode> one can be verified in seconds.\u003C\u002Fp>\n\u003Cp>Coverage of the code that matters came from somewhere else. In our \u003Ca href=\"https:\u002F\u002Fveridu.io\u002Fblog\u002Fmissed-laravel-bug\u002F\" rel=\"noopener\">first post\u003C\u002Fa>, both reviewers missed a bug that shipped in Laravel until we gave them the subclass that the change broke, from a file the pull request never touched. We did not test whether more reviewers without that context would have found it. Adding the context let both of the reviewers we had find it.\u003C\u002Fp>\n\u003Ch2 id=\"method-and-limits\">Method and limits\u003C\u002Fh2>\n\u003Cul>\n\u003Cli>\u003Cstrong>Adjudication was done by a model, not by independent humans.\u003C\u002Fstrong> Findings were judged blind, with reviewer names removed and order shuffled, against the code at each pull request's commit. The judge was one of the models under test. A different model independently re-checked the claims that could change the result and overturned one. The two materiality judges were also models; they agreed on 84% of findings (Cohen's kappa 0.61). None of this is equivalent to human validation. The overlap results above do not depend on it.\u003C\u002Fli>\n\u003Cli>\u003Cstrong>The sample is small.\u003C\u002Fstrong> 18 pull requests; two reviewers made 2 and 7 findings.\u003C\u002Fli>\n\u003Cli>\u003Cstrong>The pull requests were merged, mature open-source changes.\u003C\u002Fstrong> Code still in progress, especially private code, may produce more findings and a different overlap. That is the next thing worth measuring.\u003C\u002Fli>\n\u003Cli>\u003Cstrong>Reviewer-assigned severity skewed low:\u003C\u002Fstrong> 24 low, 22 medium, 3 high and no critical findings. That makes the corpus weak for measuring missed bugs and better for measuring noise.\u003C\u002Fli>\n\u003Cli>\u003Cstrong>Models change.\u003C\u002Fstrong> These were the versions available in September 2026.\u003C\u002Fli>\n\u003C\u002Ful>\n\u003Cp>\u003Ca href=\"https:\u002F\u002Fveridu.io\u002Fsignin\" rel=\"noopener\">Try Veridu on your own pull request\u003C\u002Fa>, or \u003Ca href=\"https:\u002F\u002Fveridu.io\u002Fdemo\" rel=\"noopener\">see a real review\u003C\u002Fa> first.\u003C\u002Fp>\n",6,{"site":16,"generated_at":17,"posts":18,"redirects":31},"veridu","2026-10-14T21:20:51.089Z",[19,22],{"slug":4,"title":5,"description":6,"date":7,"updated":7,"tags":20,"image":12,"url":21,"published_at":17},[9,10,11],"https:\u002F\u002Fveridu.io\u002Fblog\u002Fthree-reviewers-18-prs",{"slug":23,"title":24,"description":25,"date":7,"updated":7,"tags":26,"image":29,"url":30},"missed-laravel-bug","Our reviewer missed a Laravel bug. Here is what we changed.","A Laravel bug escaped diff-only review because the broken override was outside the pull request. Here is how we added bounded change-impact context.",[27,28,9],"laravel","change-impact","https:\u002F\u002Fcontent.presentr.me\u002Fveridu\u002Fassets\u002Fog.fba80c1ca1.png","https:\u002F\u002Fveridu.io\u002Fblog\u002Fmissed-laravel-bug",{},1792162752976]