UNESCO and UCL tested 33 LLMs on climate science
A UCL and UNESCO study tested 33 LLMs on climate science, from factual benchmarks to user pushback, as chat answers replace searchable links.

- UCL and UNESCO researchers tested 33 large language models on climate-science questions.
- Flagship models in the sample included Claude Fable 5, Claude Opus 5, Gemini 3.1 Pro Preview, Gemini 3.6 Flash and GPT-5.6 Sol.
- The study combined factual benchmarks with complex interactions, including cases where users push back on answers.
- Researchers note benchmark questions may already be in the models' training data.
- Chat answers are served ephemerally per user session, making them hard to audit unlike search results.
A study led by researchers from University College London and UNESCO put 33 large language models through a battery of climate-science questions, and the early signal is a split verdict: the models handle simple factual queries well, but their behavior gets murkier when users challenge them mid-conversation.
The study, published via UNESCO's document repository, tackles a problem that has quietly reshaped how people encounter scientific information. For years, researchers could audit climate content by examining Google's top results, social media feeds, or newspaper and TV coverage. Those channels left a trail. LLM chat does not.
Why researchers can't see what users actually see
There is no direct way for scientists to observe the climate information an individual chatbot user receives. Answers are served ephemerally, inside a private chat session, and they often differ from user to user. That makes LLM output far harder to audit than a ranked search results page, which anyone could scrape and compare.
So the UCL and UNESCO team attacked the question from multiple angles instead. They tested the models with different prompt styles and evaluated the resulting output, deliberately going beyond simple lookup questions.
The takeaway in plain words: chat interfaces have become part of the public information diet, but they resist the audit methods researchers built for search engines and social platforms.
What did the study actually test?
Two layers of testing structured the work.
- Benchmarks of simple factual questions about climate science — basic checks of whether models state established facts correctly. The researchers themselves note a wrinkle here: these benchmark questions may already sit inside the models' training data, which could inflate scores.
- More complex interactions, including the scenario researchers care most about: a user pushing back on an answer they believe — rightly or wrongly — is wrong.
That second layer matters most for search and information-quality watchers. A chatbot can ace a static quiz and still fold when a confident user disputes it, because conversational pressure can pull models toward agreement rather than accuracy.
Which models were in the sample?
The study spans 33 models, including multiple versions of the major frontier and open-weight systems. The flagship frontier models named in the report are:
- Claude Fable 5
- Claude Opus 5
- Gemini 3.1 Pro Preview
- Gemini 3.6 Flash
- GPT-5.6 Sol
The full model list appears in the report itself. The breadth of the sample is a deliberate answer to a familiar criticism of LLM research: that findings about one model get dismissed because a newer frontier model shipped after the experiment ended. By testing 33 models across families and versions, the researchers made that complaint harder to lodge.
The one-sentence version: this is a multi-model study, not a single-chatbot spot check.
Why this matters for the search world
The backdrop is a shift in where answers come from. Earlier climate-information research focused on Google's organic results — in the words of the original report's framing, back when Google was still in the business of serving links. Today, a growing share of informational queries end in an AI-generated summary or chat response rather than ten blue links, and that changes what "accuracy of public information" even means.
A search results page could be ranked, disputed, and regulated. A chat answer exists once, for one user, then disappears. Publishers, fact-checkers and platforms that built their workflows around linkable, persistent content will find that model under strain if chat answers drift on contested topics like climate.
What to watch next
The study's training-data caveat is the open flank: if benchmark questions have leaked into training sets, clean re-testing with fresh, unseen questions becomes the priority. Watch for follow-up work probing how models behave under user pushback, and whether providers disclose sycophancy-style failure rates on scientific topics — that, not the benchmark scores, will tell us how reliable chat answers really are for ordinary users.
via arstechnica.com (Source)
More from Olivia Hart
Show full bio
Staff writer covering money, funding and market shifts around Search rankings and algorithm updates, generative engine optimization at SERP Journal.
225 articles