How To Choose AI For A News Pipeline After Testing 71 Configurations

🔍 Read the full analysis: How To Choose AI For A News Pipeline After Testing 71 Configurations on ThorstenMeyerAI.com

Buying for a business?Offer from Amazon

Get business pricing on monitors, keyboards and dev gear

• Business-only prices and quantity discounts
• Tax-exempt purchasing
• Multiple users, one account, clear invoices

Create a free Business account →Free to register

As an affiliate, we earn on qualifying purchases.Hide

Thorsten Meyer says testing 71 AI configurations on news-writing tasks led him to select GPT-6 Luna for production rather than the higher-scoring GPT-6 Astra, which he judged too expensive for routine use. In a report published October 13 on ThorstenMeyerAI.com, Meyer also says two backup models were removed after generating invented quotes or failing to produce valid articles.

Meyer ran the test from September 8 to October 13, 2026, using his production prompt and two real news sources: a 6,340-character product filing and a 619-character political news item. Each source was turned into both an English and a German article. Meyer says the shorter source better reflects much of his production work.

Two AI judges, GPT-6.1 Sol and Claude Opus 5.5, scored articles on source fidelity, fabrication, epistemic clarity, news value, structure and polish; German versions were also judged for language quality. The strongest configurations were then tested on 10 additional articles, mostly based on short sources. Meyer says the results included 95% bootstrap intervals and counted an article only if it produced valid output after the pipeline’s usual repair step.

GPT-6 Astra received 4.00 out of 5 in both languages in the initial test, with a source-fidelity score of 5.00. Across the expanded run, it scored 3.93, compared with 3.67 for GPT-6 Luna. Meyer reports Astra cost about $0.12 per article and took around 50 seconds; Luna cost about $0.0012 in the test and took 17 seconds. In the past week of production, he says Luna cost about $0.0013 per article at a median of 17 seconds.

The Cost of Reliable News Drafts

The results illustrate a practical choice for publishers using AI to produce repeated, source-based drafts: the highest-scoring model may not be the most economical one to run at scale. Meyer says Astra performed better, but Luna’s lower reported price and consistent output made it the production choice. That conclusion reflects one publisher’s workflow and quality threshold, not a general ranking of models for every newsroom or assignment.

The test also points to a separate risk in automated publishing: fallback models can weaken source fidelity precisely when a primary service is unavailable. Meyer reports that the backup models generated quotations not present in the supplied material, making review of the entire failover chain relevant alongside selection of the main model. The reported scores and incidents are useful signals, but other publishers would need to test against their own sources, prompts, languages and publication standards.

How the News Pipeline Was Tested

Meyer describes the test as a production-oriented benchmark, rather than a general evaluation of intelligence. It covered API, subscription, free-tier and locally run open-weight models, with each configuration asked to turn a source into a finished article using the prompt his pipeline uses. The report argues that broad model leaderboards do not directly answer which system is suitable for a particular editorial task.

Meyer cautions that the initial comparison used only two sources and says differences below about 0.3 points should be treated as noise in that small test. He considers the additional 10-article robustness run more informative. In that run, he reports Astra’s 95% interval as 3.83–4.02 and Luna’s as 3.59–3.76. These intervals do not overlap, according to the report, though the limited test set still constrains how broadly the result can be applied.

The test also compared GPT-6 Luna across six reasoning-effort settings. Meyer says the production “none” setting scored 3.67 in English and 3.68 in German. The “max” setting scored 3.58 and 3.57, while increasing reported cost from 0.12 cents per article to 1.43 cents and time from 17 seconds to 211 seconds. He says the differences were within the test’s noise range.

“The right model is the cheapest one that clears your quality bar reliably, not the one at the top of the chart.”

— Thorsten Meyer, ThorstenMeyerAI.com

Limits of the Reported Scores

The findings come from Meyer’s own test; the source material does not describe an independent replication or a peer-reviewed evaluation. The initial test used two articles, while the follow-up added 10 more. The report does not establish how the models would perform across a wider range of topics, source formats, breaking-news conditions or newsroom editing practices.

The scoring was performed by AI judges, and the supplied material does not report human-editor scores or independent verification of the judges’ ratings. It also does not specify how often the main model fails in live use, how frequently fallbacks are activated, or whether the reported per-article costs include all pipeline expenses. The source ends with an incomplete description of the updated routing order, so the full current failover setup is not clear.

Monitoring the Updated Model Route

Meyer says GPT-6 Luna is now the production model and that the two tested fallback models were removed from the news route on October 7, 2026. The report does not identify a complete replacement failover chain or set out a date for another comparison.

The next useful evidence would be performance over a larger live-production sample, including article validity, corrections, source fidelity, cost and latency. Until those results are available, the published scores should be read as a bounded test of one pipeline, rather than a final verdict on which AI model newsrooms should use.

Key Questions

Which model scored highest in Meyer’s test?

GPT-6 Astra scored 4.00 out of 5 in both English and German in the initial test, with a source-fidelity score of 5.00. It scored 3.93 across the expanded robustness run.

Why did Meyer choose GPT-6 Luna for production?

Meyer says Luna scored 3.67 in the robustness run and cost about $0.0012 per article in testing, compared with about $0.12 for Astra. He judged Luna’s quality sufficient for his production needs at a much lower reported cost.

Did higher reasoning effort improve the articles?

Meyer reports no dependable improvement across GPT-6 Luna’s six reasoning settings. The maximum setting took about 211 seconds and cost 1.43 cents per article, compared with 17 seconds and 0.12 cents at the “none” setting.

What went wrong with the fallback models?

Meyer says GPT-4.1 nano received low source-fidelity scores and generated quotes that were not in the source. He reports that the locally run gpt-oss-120b produced no valid articles out of four attempts and also generated invented named quotations.

Does the test establish the best AI model for all newsrooms?

No. It evaluates one production prompt and workflow, two initial sources and a 10-article follow-up. Other newsrooms may get different results with different content, languages, prompts and editorial checks.

Source: ThorstenMeyerAI.com