🔍 Read the full analysis: FrontierCode: Reviewing AI Pull Requests Through A Maintainer’s Eyes on ThorstenMeyerAI.com
Get business pricing on office and shipping supplies
- Business-only prices and quantity discounts
- Tax-exempt purchasing
- Multiple users, one account, clear invoices
TL;DR
Artificial Analysis tested 23 AI models on FrontierCode, a benchmark that evaluates whether a code change meets maintainers’ review criteria, not just whether tests pass. Claude Opus 5.5 led with a 52.2% score and cleared every blocking check in 57.7% of attempts; cost per task ranged from $0.09 to $12.
Artificial Analysis tested 23 AI models on FrontierCode, a coding benchmark that grades patches against maintainer-style review criteria, and found that even the leading model cleared all blocking checks in only 57.7% of attempts. Claude Opus 5.5 ranked first with a 52.2% overall score, a result that suggests passing tests alone does not establish that an AI-generated change is ready to merge.
FrontierCode’s results, added to Artificial Analysis’s evaluations on October 12, 2026, put GPT-6.1 Sol second at 49.1% and Claude Fable 5.1 third at 49.0%. The leaderboard covers the benchmark’s 100-task Main set. The figures are benchmark scores, not the percentage of real-world pull requests that a model would get accepted.
The benchmark evaluates patches across six checks, including hidden tests, command checks, tests that must fail against the original code, adaptive tests, limits on files and diff size, and an LLM review against maintainers’ criteria. An attempt that fails any blocking check receives a score of zero, regardless of how many other checks it passes. Artificial Analysis ran models through the same open-source agent harness, Stirrup, with up to 250 turns and three attempts per task; GPT-6 Luna (medium) judged the review step.
Nine of the 100 tasks were not solved by any model on any attempt. Results also show substantial differences in cost: Artificial Analysis reported $0.09 per task for GPT-6 Luna (max), $0.81 for GPT-6.1 Sol (max), and $10.15 for Claude Opus 5.5 (max with fallback). These are benchmark task costs, not estimates of a broader deployment’s total operating expenses.
Infographic 01 · Would the maintainer merge it?
The best coding model clears the bar 58% of the time
FrontierCode asks whether an open-source maintainer would actually merge an agent’s change. A patch that fails any blocking check scores zero, even if the tests pass. 23 models, same harness, three attempts per task.
FrontierCode score, 23 models
Score in %, with the share of attempts that cleared every blocking check below it. Reasoning effort as tested.
Not on the board yet: Gemini 4 Argon, Muse Wonder, Muse Wonder Full and Mistral Large 4 Preview had no FrontierCode result as of 13 Oct 2026. Nemotron 3 Ultra has no published pass rate.
Source: Artificial Analysis, FrontierCode launch analysis (12 Oct 2026) and FrontierCode leaderboard, read 13 Oct 2026. FrontierCode 1.1 by Cognition, Main set: 100 private tasks, three attempts each, Stirrup harness, 250-turn cap.
Powered by Thorsten Meyer AI
Why Passing Tests Is Not Enough
FrontierCode tests a gap between producing working code and producing a patch maintainers can reasonably review. Checks for scope, diff size, useful tests and behavior against the original code aim to penalize patterns that can make a change difficult to verify, even when ordinary tests pass.
The results make a case for keeping human review in the merge process for systems evaluated at these levels. The leading model failed to clear every blocking check in 42.3% of attempts, based on the reported 57.7% pass rate. That rate is a finding under this benchmark’s tasks and setup; it does not measure the risk of every model-assisted change in a live software project.
Cost also matters to teams comparing models. GPT-6.1 Sol scored 3.1 percentage points below Opus at a reported per-task cost of $0.81, compared with $10.15 for Opus. That trade-off may inform evaluation choices, but benchmark scores and costs alone do not establish which model will perform best in a particular repository or workflow.
Products Worth Considering
Apple MacBook Pro 2021 14-inch with M1 Max Chip, 64 GB RAM, 2 TB SSD, Silver
Logitech MX Keys Combo for Business Keyboard and Mouse – Graphite
Dell Pro 14 Plus P1425 14-Inch Portable Monitor
As an affiliate, we earn on qualifying purchases.
How FrontierCode Grades Code Changes
Cognition developed FrontierCode with open-source maintainers; the benchmark was announced in June 2026 and version 1.1 followed in July with rules against looking up solutions online. More than 20 maintainers created tasks from 36 repositories, including Celery, Budibase, uppy and Mattermost. The source says maintainers spent more than 40 hours on each task. The full benchmark contains 150 tasks, while Artificial Analysis evaluated the 100-task Main set.
Tasks are private, according to the source material, so models cannot have encountered those exact tasks during training. FrontierCode also uses a trajectory scanner to detect attempts to look up solutions; flagged attempts score zero. The scoring system distinguishes blocking checks from quality checks: an attempt must pass every blocking check to receive a score, which can range from 70% to 100% depending on quality.
The top nine models scored between 43.2% and 52.2%. Xiaomi’s MiMo-V2.6-Pro led the reported open-weight models at 45.4%, followed by DeepSeek V4.1 Flash at 44.0% and Kimi K3 at 43.6%. The comparison in the source also shows that general intelligence rankings do not directly predict FrontierCode placement: Sonnet 5.5 ranked fifth on FrontierCode, while Grok 4.7 scored 33.8% and ranked 18th in the comparison.
“Where others grade like a CI, FrontierCode grades like a tech lead.”
— Tomer Nosrati, a Celery maintainer, in Cognition’s announcement
What These Scores Cannot Establish
The benchmark does not establish how often models’ changes would be accepted across all repositories, teams or production workflows. Its results come from a defined 100-task evaluation set, a shared agent harness, specific attempt limits and an LLM review judge. Performance in other settings may differ, and the score is not a direct estimate of the share of real pull requests that can be merged without human review.
Several models had no FrontierCode result as of October 13, including Gemini 4 Argon, Muse Wonder, Muse Wonder Full and Mistral Large 4 Preview. Their future results could change the comparative picture. The source also does not provide enough detail to determine how stable each model’s results would be across repeated benchmark runs or how the reported task costs translate to other usage patterns.
More Model Results May Change Rankings
The immediate next development to watch is whether results become available for models missing from the October 13 leaderboard, including Gemini 4 Argon and Muse Wonder Full. New entries could alter both the rankings and the cost comparisons.
For now, Artificial Analysis’s published results provide a benchmark snapshot rather than a forecast of day-to-day performance. Teams considering AI-generated pull requests would need to test models against their own codebases and review standards. The source material does not specify a date for the next FrontierCode update or identify which models will be added.
Key Questions
What is FrontierCode?
FrontierCode is a coding benchmark developed by Cognition with open-source maintainers. It evaluates whether AI-generated code changes meet tests and maintainer-oriented review criteria, including limits on scope and diff size.
Which model ranked first?
Claude Opus 5.5 (max with fallback) ranked first in Artificial Analysis’s results with a 52.2% FrontierCode score. It cleared every blocking check in 57.7% of attempts.
Does the 57.7% pass rate mean 42.3% of real pull requests fail?
No. The figure is the leading model’s pass rate on FrontierCode attempts. It does not measure outcomes for all real-world pull requests or repositories.
How much did model tasks cost in the evaluation?
Artificial Analysis reported costs from $0.09 to $12 per task across models. The source lists GPT-6 Luna (max) at $0.09 and Claude Opus 5.5 (max with fallback) at $10.15; these are evaluation costs, not necessarily costs in other workflows.
Which models were missing from the results?
As of October 13, the source reported no FrontierCode result for Gemini 4 Argon, Muse Wonder, Muse Wonder Full and Mistral Large 4 Preview. It did not give a release date for their results.
Source: ThorstenMeyerAI.com
