What I Learned By Testing AI Models On My Own News Pipeline

🔍 Read the full analysis: What I Learned By Testing AI Models On My Own News Pipeline on ThorstenMeyerAI.com

Buying for a business?Offer from Amazon

Get business pricing on office and shipping supplies

• Business-only prices and quantity discounts
• Tax-exempt purchasing
• Multiple users, one account, clear invoices

Create a free Business account →Free to register

As an affiliate, we earn on qualifying purchases.Hide

Thorsten Meyer says he tested 71 AI model configurations on real English- and German-language news assignments between September 8 and October 13, 2026, to choose a model for his publishing pipeline. His results identify GPT-6 Astra as the top scorer, but he uses the much cheaper GPT-6 Luna in production and says he removed two backup models after they generated fabricated quotes or unusable output.

Meyer’s test used the production article prompt and two real news sources: a 6,340-character product filing and a 619-character political item. Each configuration produced an article in English and German. Two AI judges, GPT-6.1 Sol and Claude Opus 5.5, assessed source fidelity, fabrication, clarity about uncertainty, news value, structure and polish, with German language quality also scored. Meyer says the strongest configurations then took part in a robustness run covering 10 additional articles, mostly based on short sources.

GPT-6 Astra scored 4.00 out of 5 in both languages in the initial test and received a perfect source-fidelity score there. Across 12 articles in the robustness run, it averaged 3.93, with a reported 95% bootstrap interval of 3.83–4.02. Meyer reports a cost of about $0.12 per article and an average time of about 50 seconds. GPT-6 Luna scored 3.67 in English and 3.68 in German in the initial test; its robustness-run average was 3.67, with source fidelity of 4.71 out of 5. Meyer reports a production cost of about $0.0013 per article and a median time of 17 seconds.

Meyer says he chose Luna for production because it was consistently usable at a much lower cost, rather than paying for Astra’s higher scores. He also reports that changing Luna’s reasoning effort did not produce a reliable quality improvement: the maximum setting cost about 12 times more and took about 12 times longer than the no-effort setting, while results remained within the test’s stated noise range. The test included subscription, API, free-tier and locally run open-weight models.

Why Cost and Fallbacks Matter

The results illustrate why a general leaderboard may not answer a publisher’s operational question. A model that performs well across broad benchmarks may not be the best fit for turning source material into publishable articles. Here, Meyer’s reported difference between Astra and Luna comes with a large per-article cost gap, making the production choice dependent on both quality and volume.

The fallback findings also point to a separate risk from headline model rankings. Meyer says GPT-4.1 nano scored 1.62 in English and 1.68 in German, with a source-fidelity score of 1.00, and that judges flagged invented quotes. He reports that locally run gpt-oss-120b produced zero valid articles out of four and generated named quotes not present in the sources. If a primary model fails over to a weaker system, the resulting article may contain unsupported material or fail basic formatting checks. Meyer says both models were removed from the news route on October 7.

These are results from one publisher’s workflow, not a universal ranking of writing ability. They may be useful to other newsrooms as an example of testing models against their own sources, output requirements and cost constraints, while keeping the limits of a small evaluation in view.

How Meyer Built the Test

Meyer’s October 13 report focuses on a specific production task: writing a finished article from a supplied news source. The two test sources differed substantially in length, and he says the short political item better reflects much of his normal workload. Each model was assessed on the same prompt, with both judges shown the original source.

He says an article counted only if it produced valid output after the same repair step used in production. For the initial two-source test, Meyer treated score gaps below about 0.3 points as noise; he viewed the additional 10-article robustness run as a more dependable basis for ranking. He reports 95% bootstrap intervals for that run. The design makes the findings relevant to his own pipeline, but it does not establish how the models would perform on other prompts, subjects or editorial standards.

The report also compares models across tasks. Meyer notes that Claude Opus 5.5, which he describes as the leading model on the FrontierCode coding benchmark, scored 3.46 in English and 3.39 in German in his writing test. Claude Sonnet 5.5 scored 3.67 and 3.61, respectively, but Meyer estimates its API-equivalent cost at about $0.036 per article. His conclusion is limited to the tested workflow: coding performance and news-writing performance did not rank models in the same way.

“The right model is the cheapest one that clears your quality bar reliably, not the one at the top of the chart.”

— Thorsten Meyer, in his October 13 report

Limits of the Reported Scores

The reported findings come from one publisher’s pipeline and a limited set of test sources. The initial comparison used two sources, while the robustness run covered 10 further articles for the strongest configurations. Meyer says the additional run was more reliable, but the test does not show whether the same ordering would hold across other news topics, languages, prompts or editorial review standards.

The scoring was conducted by two AI judges rather than a panel of human editors, and the report does not establish how closely their ratings match newsroom judgments. It also does not provide results from a longer-term production comparison for every configuration. The source material describes Luna as the current production choice but ends with an incomplete description of the route’s fallback arrangement, so the full failover setup is unclear.

Production Stays on GPT-6 Luna

Meyer says GPT-6 Luna remains the production model, with reasoning effort set to “none,” and that GPT-4.1 nano and gpt-oss-120b were removed from the news route on October 7. The report does not name a replacement fallback or announce a further model test. The next evidence to watch for is whether Meyer publishes results from a broader or longer-running evaluation, and whether the current production setup changes as new models become available.

Key Questions

Which model scored highest in Meyer’s test?

GPT-6 Astra scored 4.00 out of 5 in both English and German in the initial test. It averaged 3.93 across 12 articles in the robustness run, according to Meyer.

Why did Meyer choose GPT-6 Luna for production?

Meyer says Luna produced consistently usable articles at about $0.0013 per article in production, compared with about $0.12 per article for Astra. Luna’s median production time was 17 seconds, versus about 50 seconds for Astra.

Did higher reasoning effort improve Luna’s articles?

Meyer reports no reliable improvement across Luna’s six reasoning-effort settings. The maximum setting cost and took about 12 times as much as the no-effort setting, while score differences fell within the test’s stated noise range.

Why were two fallback models removed?

Meyer says GPT-4.1 nano received low scores and judges flagged invented quotes. He reports that locally run gpt-oss-120b produced no valid articles in four attempts and generated named quotes absent from the source material. He says both were removed from the route on October 7.

Do these results show which model is best for every newsroom?

No. The results describe one publisher’s prompts, sources, languages and production checks. They do not establish that the same models would rank the same way in other newsrooms or on different assignments.

Source: ThorstenMeyerAI.com