🔍 Read the full analysis: FrontierCode: How To Navigate Maintainer Review Of AI Contributions on ThorstenMeyerAI.com
Get business pricing on monitors, keyboards and dev gear
- Business-only prices and quantity discounts
- Tax-exempt purchasing
- Multiple users, one account, clear invoices
TL;DR
Artificial Analysis added FrontierCode to its evaluations on October 12, 2026, and tested 23 AI models on repository tasks judged by maintainer-oriented checks. Claude Opus 5.5 led with a 52.2% score, but cleared every blocking check in 57.7% of attempts; the results indicate that human review remains necessary under this benchmark.
Artificial Analysis tested 23 AI models on FrontierCode, a coding benchmark that evaluates whether a patch would meet repository maintainers’ acceptance criteria, and found that Claude Opus 5.5 scored highest at 52.2%. The leading model passed every blocking check in 57.7% of attempts, according to the results published October 12, showing a substantial gap between generating plausible code and consistently producing changes that clear review.
Artificial Analysis ran the models through the same open-source agent harness, Stirrup, with a limit of 250 turns per attempt and three attempts per task. The evaluation covers the benchmark’s 100-task Main set, drawn from a full set of 150 private tasks. Nine of those 100 tasks were not solved by any tested model on any attempt, the source report said.
GPT-6.1 Sol scored 49.1% and ranked second, followed closely by Claude Fable 5.1 at 49.0%. The next results included GPT-6 Astra at 47.0%, Claude Sonnet 5.5 at 45.8%, Xiaomi’s MiMo-V2.6-Pro at 45.4%, DeepSeek V4.1 Flash at 44.0%, Kimi K3 at 43.6% and Qwen3.8 Max at 43.2%. Artificial Analysis reported that MiMo-V2.6-Pro was the top open-weights model in this evaluation.
Scores reflect more than whether code runs. FrontierCode checks hidden tests, regressions, test quality, file scope and diff size, and includes an LLM-based code review against maintainers’ criteria. Failing any blocking check results in a zero for that attempt; passing them all produces a score between 70% and 100%, with quality checks affecting the final result. The review judge was GPT-6 Luna (medium), and attempts flagged by Cognition’s trajectory scanner for looking up solutions also score zero.
Infographic 01 · Would the maintainer merge it?
The best coding model clears the bar 58% of the time
FrontierCode asks whether an open-source maintainer would actually merge an agent’s change. A patch that fails any blocking check scores zero, even if the tests pass. 23 models, same harness, three attempts per task.
FrontierCode score, 23 models
Score in %, with the share of attempts that cleared every blocking check below it. Reasoning effort as tested.
Not on the board yet: Gemini 4 Argon, Muse Wonder, Muse Wonder Full and Mistral Large 4 Preview had no FrontierCode result as of 13 Oct 2026. Nemotron 3 Ultra has no published pass rate.
Source: Artificial Analysis, FrontierCode launch analysis (12 Oct 2026) and FrontierCode leaderboard, read 13 Oct 2026. FrontierCode 1.1 by Cognition, Main set: 100 private tasks, three attempts each, Stirrup harness, 250-turn cap.
Powered by Thorsten Meyer AI
Why Passing Review Changes the Ranking
The results matter because ordinary coding tests can show that a patch works in specified cases without showing that it is well-scoped, maintainable or suitable for a real repository. FrontierCode is designed to test those additional conditions. Its figures do not establish how every open-source team will assess AI-generated contributions, but they offer a more demanding comparison than a pass rate based only on visible tests.
The ranking also differs from general model-intelligence rankings cited in the source report. Claude Sonnet 5.5, which placed second among the compared models on the Artificial Analysis Intelligence Index, ranked fifth on FrontierCode; Grok 4.7 ranked 18th here. DeepSeek V4.1 Flash and GPT-6 Luna performed better on FrontierCode than their relative positions on that general index might suggest. General intelligence scores are not a substitute for repository-specific review performance.
Cost adds a practical constraint. Artificial Analysis listed GPT-6 Luna (max) at 41.9% for $0.09 per task, MiMo-V2.6-Pro at 45.4% for $0.18, GPT-6.1 Sol at 49.1% for $0.81, and Opus 5.5 at 52.2% for $10.15. Moving from Sol to Opus added 3.1 score points at about 12.5 times the reported cost. Those figures are benchmark estimates, not a guarantee of costs or outcomes in another team’s workflow.
As an affiliate, we earn on qualifying purchases.
How FrontierCode Tests Repository Changes
FrontierCode was developed by Cognition and open-source maintainers and announced in June 2026, according to the source report. Version 1.1 followed in July with rules against searching online for task solutions. More than 20 maintainers created tasks based on 36 established repositories, including Celery, Budibase, uppy and Mattermost; the report says maintainers spent more than 40 hours on each task.
The tasks are private to reduce the chance that models have encountered the answers during training. The benchmark’s review criteria also penalize patterns that can burden maintainers, such as excessive changes, edits outside relevant files and tests that do not establish whether the original bug or behavior has been corrected. In particular, its reverse tests check that an agent’s added tests fail against the original code. A test that passes both before and after a change would not, by itself, show that the change fixed the intended problem.
Artificial Analysis publishes results for the 100-task Main set, rather than all 150 tasks in FrontierCode. Its findings therefore describe this test set, harness, model configurations and scoring process—not every model version, repository, or human review environment.
“Where others grade like a CI, FrontierCode grades like a tech lead.”
— Tomer Nosrati, a Celery maintainer, as quoted in Cognition’s announcement
software development code review software
As an affiliate, we earn on qualifying purchases.
As an affiliate, we earn on qualifying purchases.
What the Benchmark Cannot Yet Show
The results do not show how models will perform across all software repositories or how maintainers will judge their changes outside this benchmark. The source report describes the tasks, checks and harness, but the published scores remain tied to a particular 100-task evaluation and model configurations. It is also not clear from the supplied material how results would change with different agent settings, review judges or task mixes.
Several models were missing from the reported leaderboard as of October 13, including Gemini 4 Argon, Muse Wonder, Muse Wonder Full and Mistral Large 4 Preview. Their absence means the rankings are incomplete across the models mentioned in the source. The benchmark also does not establish that a model’s score translates directly into fewer hours of human review or safely autonomous pull requests in production.
As an affiliate, we earn on qualifying purchases.
Upcoming Model Results to Watch
The next notable development will be whether results become available for the models absent from the October 13 comparison, particularly Muse Wonder Full and Gemini 4 Argon. Artificial Analysis’ current results offer a snapshot; the supplied source does not give a date for another evaluation or confirm when missing models will be tested.
For maintainers and teams considering AI coding tools, the immediate practical step is to treat these scores as comparative evidence, not a replacement for review. Future FrontierCode evaluations can clarify whether models improve on blocking checks, test quality and change scope, while real-world teams will still need to judge how those results fit their own repositories and standards.
As an affiliate, we earn on qualifying purchases.
Key Questions
What is FrontierCode?
FrontierCode is a coding benchmark developed by Cognition and open-source maintainers. It evaluates repository changes using tests, scope and quality checks, plus an LLM code review against maintainer criteria.
Which model ranked highest?
Claude Opus 5.5 (max with fallback) led the Artificial Analysis results with a 52.2% FrontierCode score. It passed every blocking check in 57.7% of attempts, according to the report.
Does the top score mean AI can submit code without human review?
No. The score measures performance on a specific benchmark, and the leading model failed to clear every blocking check in many attempts. The results do not establish that human review can be removed from real software projects.
How does FrontierCode differ from a standard coding test?
It checks more than whether a patch passes tests. The benchmark includes regression and reverse tests, file and diff limits, adaptive tests and an LLM review; failing a blocking check gives an attempt a zero.
Were all major models included?
No. The source report said Gemini 4 Argon, Muse Wonder, Muse Wonder Full and Mistral Large 4 Preview had no FrontierCode result as of October 13, 2026.
Source: ThorstenMeyerAI.com
Columbus Day / Indigenous Peoples' Day Picks
long weekend sales
As an affiliate, we earn on qualifying purchases.
