🔍 Read the full analysis: Research Novelty And AI: Can Language Models Make The Call? on ThorstenMeyerAI.com
Get business pricing on monitors, keyboards and dev gear
- Business-only prices and quantity discounts
- Tax-exempt purchasing
- Multiple users, one account, clear invoices
TL;DR
Ai2 researchers report that six language models gave unstable judgments of scientific idea novelty across different evaluation setups. In one pairwise test, small prompt wording changes shifted accuracy by more than 50 percentage points; retrieval and additional reasoning did not consistently improve results. The team released a paper, benchmark data, code and an evaluation pipeline, while noting that some benchmark labels are weak or based on reviewer signals.
Ai2 researchers report that six large language models gave inconsistent judgments of whether scientific ideas were novel, with small prompt changes shifting accuracy by more than 50 percentage points in one experiment. The team released a paper, benchmark data, code and a pipeline for further testing, raising questions about using model scores to filter research proposals, as detailed in the original analysis.
The study evaluated GPT-5.1, GPT-5.2 and GPT-5.4, along with Claude Sonnet 4.5, Claude Opus 4.5 and Claude Opus 4.6. Researchers tested both absolute novelty ratings and pairwise comparisons between ideas. They report that changing a few words in a pairwise evaluation prompt shifted accuracy by more than 50 percentage points. The supplied account does not give the underlying accuracy figures or identify the prompt variants responsible for the full change.
The team also tested whether giving models more reasoning or access to retrieved literature improved their decisions. Neither approach consistently improved judgments, according to the researchers. Two purpose-built novelty systems also performed worse than a simple prompted Claude Opus 4.6 baseline in the reported comparison. These findings describe the tested systems and setups; they do not establish that every model or novelty-evaluation method will behave the same way.
For one benchmark, the researchers used ICLR 2026 peer reviews to identify ideas with explicit reviewer comments about originality. The high-novelty set contains 154 ideas from accepted papers among the top-rated 10% in their research areas; the lower-novelty set contains 145 ideas from rejected papers among the bottom 10%. The selected reviews praised novelty in the first group and raised originality concerns in the second, with no opposing comments identified under the study’s criteria. An annotator agreed with the extracted signal direction in 47 of 50 checked cases; the authors say the other three did not change the review classifications.
Why Unstable Scores Matter for Research
Research groups may use language models to sort candidate ideas before committing time to literature searches, experiments or expert review. If an evaluation score changes substantially when a prompt is lightly edited, it may reflect the test setup as well as the idea’s originality. That makes scores harder to compare across projects, teams or studies unless the evaluation procedure is reported and tested for sensitivity.
The result also bears on efforts to use AI to support scientific discovery. Systems that generate hypotheses or synthesize papers need a credible way to assess whether a proposal adds something to existing work. Ai2’s findings do not show that all automated novelty checks fail. They do show that a strong result on one chosen prompt or benchmark is not, by itself, evidence of dependable judgment.
For readers assessing AI research tools, the practical implication is that novelty scores should not be treated as definitive verdicts without validation. The study points toward checking performance across prompts and evaluation formats, and keeping human review in the process where research decisions depend on originality.
Products Worth Considering
DJI Osmo Mobile 8 Standard Combo Phone Gimbal Stabilizer
Amazon Kindle Paperwhite Signature Edition (12th Generation, 2024), 32 GB, Metallic Black
DJI Osmo Mobile 8 3-Axis Smartphone Gimbal Stabilizer
As an affiliate, we earn on qualifying purchases.
How Ai2 Constructed Its Tests
A novelty judge is a model asked to assess whether an idea is new relative to prior work, either by rating it on its own or comparing it with another proposal. Ai2 says there is no standard evaluation recipe: researchers can choose different models, prompts, reasoning budgets, comparison formats and literature-retrieval systems. The authors argue that some judges are tested on human-written papers even when their intended use is to assess ideas generated by AI.
The study includes a second test setting pairing the 154 human-authored ideas with 154 ideas from a basic language-model generator. The researchers treated the generated proposals as less novel, but describe those labels as weak because each generated idea was not independently verified. In a separate expert check of 30 pairs—selected heavily from cases the strongest judge got wrong—the expert preferred the human-authored idea every time and judged none of the generated ideas novel. That small, selected check is not a validation of every label in the full set.
The authors caution that acceptance or rejection alone cannot establish novelty: papers may succeed or fail for reasons other than originality. Their benchmark instead uses reviewer comments and selection criteria as signals. Ai2 says it has released the paper, data, code and a pipeline intended to help others create additional evaluation sets.
““Seemingly small changes to the evaluation setup produce dramatically different judgments of the same ideas.””
— Ai2 study authors
Limits of the Novelty Labels
The benchmark’s labels are signals of relative novelty, not definitive measures. The human-idea groups rely on reviewer comments and selection criteria, while the human-versus-generated comparison assumes the generated ideas are less novel without verifying every proposal individually. The expert assessment involved only 30 pairs and was weighted toward cases where a leading judge had made an error.
The available study description does not provide full results for every model, prompt, reasoning budget or evaluation format. It also does not establish whether the same instability appears in other disciplines, with different idea generators, or when systems have broader access to research literature. The authors say some conclusions about disagreement between setups do not depend on the benchmark labels, but the supplied material does not detail how broadly that result holds.
Further Tests Across Research Fields
Ai2 has made the paper, benchmark, data, code and automated pipeline available for further evaluation. Other researchers can use those resources to test whether prompt sensitivity persists across fields, types of research ideas and judge designs, and to compare methods intended to make assessments more consistent.
The release description does not announce a follow-up study or a timetable. For now, the benchmark gives researchers a way to examine how evaluation choices affect results, while leaving open whether any combination of prompts, retrieval and model design can produce dependable novelty judgments.
Key Questions
What did the Ai2 study find?
Ai2 researchers report that six language models gave unstable novelty judgments across evaluation setups. In one pairwise test, changing a few prompt words shifted accuracy by more than 50 percentage points.
Which models were tested?
The study tested GPT-5.1, GPT-5.2, GPT-5.4, Claude Sonnet 4.5, Claude Opus 4.5 and Claude Opus 4.6, using absolute ratings and pairwise comparisons.
Did literature retrieval or extra reasoning help?
According to the researchers, adding literature retrieval or giving models more reasoning did not consistently improve their novelty judgments in the tested setups.
Are the benchmark’s novelty labels definitive?
No. The human-idea labels draw on peer-review comments and selection criteria, and the generated-idea labels were not individually verified. The authors describe some labels as weak signals rather than definitive judgments.
What has Ai2 released?
Ai2 released the study, benchmark data, code and an automated pipeline intended to support further evaluation. The source material does not specify a release date or a scheduled follow-up study.
Primary source: Hugging Face · via ThorstenMeyerAI.com
