We evaluated four models on the task of auto-approving our PRs to test how well they followed our approval rules.
In July of 2026, we made human code review optional at Corridor. Our internal auto-approver decides whether a code change needs to be blocked for human review or can be approved to merge into main. It reviews the change against our documented review criteria, using the surrounding code, security findings, CI results, and other code policy requirements to decide whether human review is needed.
Our auto-approver operates within our existing development workflow, alongside our Corridor Security Check, PR review bot, in-session pre-commit hooks, security merge gate, and security context guidance. PRs that the auto-approver approves must still pass the other required checks before they can merge.
While an auto-approver should be able to reason about code correctness, test coverage, and readability, the biggest risk in moving to an auto-approval process is insecure code getting shipped to main. Avoiding that is a proactive cybersecurity task: can a model keep security risk out of the codebase in the first place, by approving or blocking a PR accordingly?
We wanted an auto-approver that caught more PRs requiring human review: our historical GPT-5.5 reviewer running on Cursor, and Codex Code Review were letting through far too many PRs with security and other correctness issues we caught later in the SDLC, as displayed by the false approval rates below.
Here, a false approval means approving a PR that our reference reviewers judged to require human review.
Figure 1. False approval and false block rates for the two baselines. Rates use 161 reference-blocked and 36 reference-approved PRs, respectively.
Our historical reviewer also cost roughly $4 per review, so we wanted to see whether we could improve accuracy while bringing that cost down. So, we built an evaluation to test four cheaper replacements: GPT-5.5, GPT-5.6 Luna, GPT-5.6 Terra, and Grok-4.6 all running on the Claude Code harness.
Figure 2. GPT-5.6 Luna's false approval rate compared with the historical automated-review and Codex Code Review baselines. False approval rate uses a denominator of 161 reference-blocked PRs.
When compared with our historical baseline, we found that GPT-5.6 Luna made about 80% fewer false approvals. We also compared Luna with Codex Code Review, where Luna made 76% fewer false approvals.
Figure 3. False approval rates and estimated cost per PR for the four candidate models. False approval rate uses a denominator of 161 reference-blocked PRs.
Within the same Claude Code harness, Luna reduced false approvals from 35 to 14, a 60% reduction compared with GPT-5.5.
We also found that both GPT-5.6 models had substantially lower false approval rates at lower cost than GPT-5.5 when all four models were rerun in the evaluation harness, while Grok 4.6 approved nearly half of the PRs our reference said should have been blocked. We looked at the reviews behind those decisions to understand what each model found and whether its approval decisions followed our blocking criteria.
Methodology
Figure 4. Auto-approver evaluation pipeline. Issue coverage counts each medium- or high-severity reference issue once.
Our evaluation covered 197 historical Corridor PRs, roughly half of which had originally been auto-approved and the other half blocked for human review. We reviewed them again to label a reference set.
Two reviewer agents, Claude Opus 5 and Kimi-K3, independently reviewed each PR using the Claude Code harness. They read the description and diff alongside the surrounding codebase, including modified files, related tests, and code that could be affected indirectly. They also had access to signals available at the time of review, including Corridor's PR Reviewer security issues, CI statuses, and CODEOWNERS information.
We provided them these extra signals because weighing them is a part of the auto-approval task. Corridor's PR Reviewer may surface an issue that isn't worth blocking a merge over, and CI can be red for unrelated reasons, so the reviewer must decide which concerns justify a block.
Our reviewer prompt includes explicit guidance about which issues are blocking or non-blocking. The labelers documented these issues in their reviews and returned a final approved or blocked verdict on each PR. Our labelers agreed on this decision on 114 PRs and disagreed on 83, so a third adjudicator agent, Gemini-3.1 Pro on Claude Code, reviewed their assessments, checked the concerns against the code, and adjudicated disagreements.
Across 197 historical PR snapshots, our reference reviewers recorded 923 issues, including blocking and non-blocking observations. These findings span correctness, security, tests, and internal policy. The resulting reference set contained 161 blocked PRs and 36 approved PRs.
We evaluated all four candidates (GPT-5.5, GPT-5.6 Luna, GPT-5.6 Terra, and Grok-4.6) through Claude Code as a harness using our model adapter and compared their final decisions with the reference labels. We also used GLM-5 as a judge to identify which findings from the reference reviews appeared in each candidate's review.
Our Two Baselines
We compared our candidate's reviews with two baselines, each using the same 197-PR evaluation set.
The first was our historic automated-review baseline, which used the same approve/block verdicts recorded by GPT-5.5 running in Cursor. This produced a false approval rate of 42.9% and a false block rate of 11.1%.
Our second baseline was Codex Code Review. We recreated the historical changes as PRs in a private repository and programmatically requested hosted Codex Code Reviews by posting @codex review. We collected the completed reviews through GitHub and classified any review containing a P0 or P1 finding as a block. We chose to only block on P0/P1 findings because Codex Code Review flags only P0 and P1 issues in GitHub by default.
Figure 5. False approval and false block rates for the two baselines. Rates use 161 reference-blocked and 36 reference-approved PRs, respectively.
Results
GPT-5.6 Terra had the lowest false approval rate, with 86% fewer false approvals than Codex Code Review
A false approval is a candidate-approved PR that the reference says should have been blocked. A false block is a candidate-blocked PR that the reference says should have been approved. We measured false blocks alongside false approvals because a model that blocks more changes also creates more work for human reviewers.
The false approval rate is calculated over the 161 reference-blocked PRs, and the false block rate over the 36 reference-approved PRs.
Figure 6. Approval decisions by model. False approval rates use 161 reference-blocked PRs; false block rates use 36 reference-approved PRs.
Terra had the lowest false approval rate at 5.0%, followed by Luna at 8.7%. Both GPT-5.6 models made fewer false approvals than GPT-5.5, but more false blocks. Grok was the most permissive candidate, approving 80 (~50%) of the PRs the reference blocked and all 36 of those the reference approved.
Disputed PRs were harder for every model
We also wanted to know whether the overall rates hid differences on PRs that were harder to label consistently, so we separated the 79 PRs both labelers agreed to block from the 82 disputed PRs the adjudicator ultimately blocked.
Figure 7. False approval rates on reference-blocked PRs: 79 both labelers agreed to block and 82 blocked after adjudication.
All four candidates in the shared harness had a higher false approval rate on labeler-disputed PRs. Luna went from 1.3% on PRs both labelers agreed to block to 15.9% on disputed ones, while Grok went from 16.5% to 81.7%. Evaluating only PRs with clear reviewer agreement would have given us a much more optimistic picture of every candidate, particularly Grok.
Luna combined low cost with a low false approval rate
Figure 8. Estimated cost per PR and false approval rate.
Both GPT-5.6 models improved on GPT-5.5's false approval rate at lower cost, alleviating one of our concerns that lower cost would come with more false approvals. The remaining tradeoff was between Luna and Terra: Terra let fewer blocking PRs through, while Luna cost less and sent fewer acceptable PRs to human review.
Both GPT-5.6 models made fewer false approvals than either baseline
Figure 9. Relative changes in false approvals and false blocks compared with each baseline. Negative values indicate fewer errors; positive values indicate more.
Against our two baselines, GPT-5.6 Luna made 80% fewer false approvals than the historical baseline and 76% fewer than the Codex baseline. Luna made two more false blocks than the historical baseline and four fewer than the Codex baseline. Terra had even fewer false approvals, but matched the Codex baseline's 10 false blocks. Meanwhile, Grok had more false approvals than either baseline, although it made no false blocks.
Grok let through 24% of the high-severity issues. The other models let through almost none.
The false approval rate metric counts each incorrectly approved PR once, regardless of the severity of the issues it contains. We therefore looked at the issues inside those PRs to understand what specific issues each model would allow into the codebase.
Figure 10. Medium- and high-severity reference issues in falsely approved PRs; percentages use each category's reference count.
Figure 11. High-severity issues only, a subset of Figure 10.
Grok's false approvals contained more than a third of the medium- and high-severity reference issues, compared with less than 5% for either GPT-5.6 model. They also contained 10 high-severity issues, compared with one for Luna, none for Terra, and one for GPT-5.5. Grok's higher false approval rate extended to the most severe issues in the reference, while the other models rarely let through a high-severity issue.
Finding reference issues did not guarantee good approval decisions
We wanted to understand whether false approvals were largely due to candidate models missing key reference issues or approving PRs despite identifying them.
We first measured issue coverage across the full evaluation set: the number of medium- and high-severity reference issues matched in a model's reviews, divided by the total number of reference issues of those severities.
Figure 12. Model rankings by issue coverage across 432 medium- and high-severity reference issues and by false approval rate among the 161 reference-blocked PRs.
Luna had the highest issue coverage at 63.9%, closely followed by Terra at 61.6%. Grok covered a larger share of reference issues than GPT-5.5, but its false approval rate was much higher: 49.7% versus 21.7%. Finding more reference issues therefore did not necessarily translate into better approval decisions under our documented standards.
We then applied the same calculation to each model's false approvals: matched medium- and high-severity reference issues divided by all reference issues of those severities in its falsely approved PRs. Each model's false approvals form a different subset of PRs, so these figures describe its failures rather than provide a comparison on a common set.
Figure 13. Coverage of medium- and high-severity reference issues inside each model's false approvals. Each model's false approvals are a different subset of PRs.
Grok documented 63 of the 164 medium- and high-severity reference issues in PRs it incorrectly approved, producing a 38.4% issue coverage. On 18 of those PRs, it documented all of these issues and still approved. GPT-5.5, by comparison, documented none of them on 30 of its 35 false approvals.
These results illustrate two ways approval decisions can go wrong: relevant issues can be absent from the review, or the review can document them and still approve the PR. Grok's failures show us that finding an issue is only part of the auto-approver's task: the model also has to apply our standards for whether that issue warrants blocking the PR for human review.
Approval decisions need their own evaluation
Our reviewer prompt explicitly identifies which issues should and shouldn't block auto-approval. The models received these rules directly, although applying them to a particular PR still required judgment.
Figure 14. Examples of blocking and non-blocking criteria in the reviewer prompt. A dash means no example is listed in this summary, not that no rule applies.
We suspected that making those criteria more explicit might help Grok, so we ran another experiment where we gave Grok a standalone calibration policy with additional examples of blocking and non-blocking behavior. With that policy in place, it still approved nearly half the PRs the reference said should have been blocked.
As an aside, Grok's model card describes SecureCodeReview, an internal benchmark that evaluates fixing existing security issues while avoiding new ones. It reports the following results:
Figure 15. SecureCodeReview results reported in the Grok-4.6 model card.
GPT-5.5 and Grok scored relatively close on SecureCodeReview, so we expected their auto-approval performance would be similar. Instead, Grok falsely approved more than twice as many PRs as GPT-5.5. This gap illustrates the value of an evaluation like ours: one that tests whether a model can decide when a PR can proceed and when it needs human review under an organization's rules.
What the security issues were about
We then looked specifically at security issues to understand whether the remaining false approvals involved concrete weaknesses in the code or changes that required human review under our own security rules.
The reference contained 67 medium- and high-severity security issues, including 10 rated high severity. These covered both weaknesses in the code and changes that required human review under Corridor's policies.
For this breakdown, we grouped the reference descriptions by the rule behind the concern:
- Generic security principles: a concrete weakness remains even without an internal policy, such as an overly broad permission grant or untrusted input reaching an authorization decision.
- Corridor-specific requirements: the concern depends on an internal rule, such as requiring a named human to review a particular kind of change even when the code itself is correct.
- Borderline: the security concern is broadly applicable, but treating it as merge-blocking depends on the organization's requirements.
The table shows how many security issues each model let through by approving the PR containing them. Numbers in parentheses show how many of those issues were mentioned in the review. A mention does not mean the model agreed that the issue was blocking.
Figure 16. Medium- and high-severity security issues let through by rule type. Parentheses show how many were matched in the model's review by GLM-5.
Luna and Terra each let through one of the 49 generic security issues. Of Luna's four security issues let through, two were Corridor-specific and one was borderline, so three involved an internal requirement or a judgment about what should block a merge. Terra let through one borderline issue and no Corridor-specific issues.
Across the four models, Corridor-specific issues were let through at twice the rate of generic issues: 30.6% versus 15.3%, counting each reference issue once per model.
GPT-5.5 and Grok let through more security issues, and most were not documented in their reviews. GPT-5.5 mentioned only 1 of the 14 security issues it let through, while Grok mentioned 6 of 25. For security specifically, then, better approval decisions are only part of the problem: most of the security issues these models let through were never caught by their review in the first place.
What we found
GPT-5.6 Luna made about 80% fewer false approvals than the historical automated-review baseline, 76% fewer than Codex Code Review, and far fewer than GPT-5.5 and Grok-4.6. Its falsely approved PRs contained far fewer medium- and high-severity issues than Grok-4.6's, and it had the lowest estimated cost per PR of the four candidates.
Terra had an even lower false approval rate at 5.0%, but it cost about seven times as much as Luna and falsely blocked four more PRs.
Luna showed a substantial improvement in false approval rate over GPT-5.5 while keeping cost and unnecessary human review lower than Terra.
A few takeaways from our evaluation:
- A model that is good at detecting issues in a PR isn't necessarily good at deciding whether to approve it. Grok had higher issue coverage than GPT-5.5, but a much higher false approval rate.
- All four candidates did worse on PRs where the labelers disagreed and evaluating only the clear-cut cases would have overstated candidates' performance on our task.
- Cost didn't consistently track quality. The cheapest candidate (GPT-5.6 Luna) outperformed the most expensive one (GPT-5.5) on false approvals.
- Company-specific security rules deserve explicit testing. Across the four models, Corridor-specific issues were let through at twice the rate of generic issues in this evaluation.
Overall, the models we tested still struggled to apply our review criteria consistently: even the model with the lowest false approval rate approved 5% of the PRs our reference reviewers said needed human review, and finding the issues in a PR did not guarantee the right approval decision. That's why we use Corridor throughout the development workflow to aid in determining whether code is actually safe to ship: our PR review bot catches security issues and blocks PRs before they're merged, and in-session security scanners like our pre-commit hooks and MCP server help ensure our code is secure by default.
Future Work
We limited this first comparison to models we were considering as replacements within our existing workflow, but we plan to evaluate alternative open-weight models to test whether we can bring down the cost of auto-approval even more. Given GPT-5.6 Luna's lower estimated cost, we might be able to ensemble a few of these cheaper auto-approver models to reach lower false approval rates.
As we automate more of the software development lifecycle, we'll need to make similar decisions about the models behind other gates. We're running more evaluations on the other automated checks in our pipeline as we increase our level of software autonomy, so subscribe to our blog for more.
Appendix: Models and harness details
| Role | Model ID | Requested reasoning effort | Configured context budget | Output-token limit |
|---|---|---|---|---|
| Candidate | gpt-5.6-luna | max | 400,000 | 32,000 |
| Candidate | gpt-5.6-terra | max | 400,000 | 32,000 |
| Candidate | gpt-5.5 | high | 400,000 | 32,000 |
| Candidate | grok-4.6 | high | 200,000 (harness default) | 32,000 |
| Reference labeler | claude-opus-5 | xhigh | 200,000 | 32,000 |
| Reference labeler | kimi-k3 | max | 1,000,000 | 128,000 |
| Reference adjudicator | gemini-3.1-pro-preview | high (harness default) | 200,000 (harness default) | 32,000 |
| Finding-matching judge | glm-5 | No explicit reasoning-effort parameter sent | Provider default | 24,000 |
Review harness: @anthropic-ai/claude-code@2.1.231, pinned in both the reference-labeling and candidate-evaluation workflows.