How to evaluate an AI claim: five questions for benchmarks, studies, surveys and vendor promises

Every week brings a new AI claim. A model "beats experts" on a test. A tool "doubles productivity." A report says almost every company is "getting zero return." A vendor promises an "AI lawyer" or "expert AI-driven forecasts." Some of these claims are solid, some are half true, and some have ended in fines. From the outside they all sound equally confident.

This article gives you a short, repeatable way to sort them. It explains the four main kinds of AI claim, shows with real cases how each kind goes wrong, and ends with five questions you can ask of any claim in a few minutes. None of it needs a statistics degree. It needs the habit of asking what was measured, against what, and by whom.

Four kinds of AI claim, and why they need different checks

Most AI claims fall into one of four groups, and each has its own typical weak point.

• Benchmark claims say a model scored a certain number on a standard test, such as a set of maths problems or coding tasks. A benchmark is simply a fixed set of questions with known answers that lets people compare systems. The weak point is whether the test measures what its name suggests.
• Study claims report the result of an experiment or analysis, for example that developers finished tasks faster with AI. The weak point is the comparison: faster than what, measured how, in which people?
• Survey claims report what people or companies say about AI, for example that most firms see no return. The weak point is who was asked and how the key term was defined.
• Product claims are what vendors say their system does. The weak point is evidence: was the product ever tested against the promise?

The same sentence can mix all four. "Our AI agent outperforms human analysts" might rest on a benchmark, a small internal study, a customer survey and a marketing line. Splitting a claim into its parts is the first and most useful step.

Benchmark claims: what a score can and can't tell you

Benchmarks are the backbone of AI progress reporting, so their quality matters. The largest review to date, presented at NeurIPS 2025 (one of the field's main conferences), had 29 expert reviewers examine 445 benchmarks for large language models (LLMs, the systems behind chatbots such as ChatGPT and Claude)³. Its findings are a useful warning list.

Many benchmarks measure a fuzzy idea. Most benchmark papers (78.2%) define the thing they claim to measure, such as "reasoning" or "harmlessness." But of those definitions, 47.8% are contested, meaning experts do not agree on them³. A model that scores well on a "reasoning" test has shown it can answer that test's questions. Whether that is reasoning in the sense you care about is a separate question.

Few benchmarks check whether differences are real. Only 16.0% of the reviewed benchmarks used any statistical test when comparing models³. That matters because a test is a sample of questions. Run a slightly different sample and scores shift. An Anthropic researcher, Evan Miller, has argued that evaluation scores should be reported with their standard error, a measure of how much the score would move by chance, and that comparisons between models should report the uncertainty of the difference too⁶. A two-point lead with no error bar may be noise.

Models may have seen the test before. The review found that 38.2% of benchmarks reuse data from earlier benchmarks or human exams³, which raises the risk of contamination: test questions leaking into the enormous text collections used to train models, so the model partly remembers answers instead of working them out. One team tested this directly. They wrote 1,205 new grade-school maths problems in the style of a popular benchmark called GSM8k, without any help from AI, then compared scores. Some model families scored up to 8% lower on the fresh problems, and the models most likely to reproduce GSM8k's own questions tended to show the biggest gaps⁴. Others, including leading closed models, showed little or no drop. The lesson is not that benchmarks are worthless. It is that a score on a public, well-known test is an upper estimate until confirmed on new questions.

Leaderboards can be gamed by how models are submitted. Public leaderboards that rank models by human votes are popular because they feel closer to real use. A 2025 analysis of one of the best-known, Chatbot Arena, reported that some providers could test many private variants and publish only the best. At the extreme, the authors identified 27 private variants tested by Meta before its Llama 4 release, and estimated that Google and OpenAI had received about 19.2% and 20.4% of all the arena's data, while 83 open-weight models combined received about 29.7%⁵. The arena's operators disputed parts of the analysis. Either way, the case shows that a rank depends on the rules of the contest as well as on the model.

Study claims: the comparison is everything

A study claim is only as good as its comparison group. The strongest design is a randomized controlled trial, where chance decides who (or which task) gets the AI, so other differences even out.

One of the most instructive AI studies used exactly that design. In early 2025 the research group METR asked 16 experienced open-source developers to complete 246 real tasks from projects they knew well, with each task randomly allowed or barred from using AI tools (mostly Cursor with Claude 3.5 and 3.7 Sonnet)¹. Before starting, the developers expected AI to cut their time by 24%. Afterwards they believed it had cut their time by 20%. The measured result went the other way: with AI allowed, tasks took 19% longer¹.

Two lessons follow. First, how people feel about AI's effect is not a measurement of it. Self-reported time savings, which fill many vendor case studies and surveys, can be wrong in direction, not just in size. Second, good studies come with their own caveats, and you should read them. METR stressed that 16 developers is a small sample in a specific setting. In February 2026 it reported that a follow-up with 57 developers and more than 800 tasks gave what it called "an unreliable signal," mainly because a growing share of developers declined to take part if it meant working without AI². That refusal is itself a kind of evidence about perceived value, but it also means the people left in the study were not a fair sample. METR now labels the early-2025 result as out of date². A careful reader takes both the finding and its expiry date seriously.

A subtler problem is leakage, when information that would not be available in real use slips into a model's training or testing, making it look better than it is. Princeton researchers Sayash Kapoor and Arvind Narayanan surveyed earlier reviews and found leakage-related errors reported in 17 scientific fields, affecting 329 papers in their preprint's count⁷. In their own case study on predicting civil wars, papers claiming that complex machine-learning models beat older statistical methods failed to reproduce once the leakage was fixed: the newer models did no substantively better⁷. Leakage is hard to spot from a paper's text alone, which is why the authors propose structured "model info sheets" that force the key checks into the open.

Survey and headline claims: read the definition, not the number

In August 2025 a report from MIT's Project NANDA spread widely with a striking line: despite US$30 to 40 billion in enterprise spending on generative AI, 95% of organizations were seeing no business return¹¹. It became one of the most quoted AI statistics of the year.

The number deserves a closer look before it is repeated. According to the report as summarized in the trade press, it drew on a review of more than 300 publicly disclosed AI initiatives, 52 structured interviews and 153 survey responses from senior leaders gathered at four conferences, between January and June 2025¹¹. The "95%" refers to pilots without measurable profit-and-loss impact, not to projects that failed outright. That is a meaningful finding, but a narrower one than the headline: a small, conference-based sample, a strict definition of success, and a short time window. Efficiency gains that never show up as a separate profit line would count as "no return."

Survey claims in the other direction need the same care. When a survey says most firms "use AI," check what counted as use: one employee trying a chatbot, or a system running inside a core process? The number can only be as precise as its question.

Product claims: regulators are now checking too

Product claims are where evidence matters most for buyers, and where North American regulators have started to act.

In March 2024 the US Securities and Exchange Commission (SEC) settled its first "AI washing" cases, against two investment advisers. Delphia paid a US$225,000 penalty and Global Predictions US$175,000, US$400,000 in total, for false or misleading statements about their use of AI. Global Predictions had claimed to be the "first regulated AI financial advisor"⁸. The SEC's chair at the time put it plainly: advisers "should not mislead the public by saying they are using an AI model when they are not"⁸.

The US Federal Trade Commission (FTC) has gone after consumer claims. DoNotPay marketed itself as "the world's first robot lawyer." In January 2025 the FTC voted 5-0 to finalize an order requiring the company to pay US$193,000, notify subscribers from 2021 to 2023, and stop making unsupported claims. Its complaint said the company had not tested whether its "AI lawyer" performed at the level of a human lawyer⁹.

In Canada, the Canadian Securities Administrators published Staff Notice and Consultation 11-348 on December 5, 2024. According to legal summaries, it cautions market participants against "AI washing," warning that vague and unsubstantiated statements full of jargon should not be made, and expects AI disclosures to be fair, balanced and not misleading¹⁰.

These cases share a pattern worth remembering: the problem was not that the products used AI, but that the claims ran ahead of any test. "Has this been tested against the specific promise?" is now a legal question as well as a practical one.

The limits of any checklist

Good judgment about claims has its own limits, and it is fair to name them.

Evidence ages fast. A careful study of early-2025 tools may say little about late-2026 tools, as METR itself acknowledges². Treat any figure about AI capability as dated, and look for the date.

Better evidence is often unavailable. Many of the most useful numbers (how a model performs on your data, in your workflow) do not exist until you test them. A checklist can tell you a claim is weak. It cannot supply the missing evidence.

Scepticism can overshoot. A flawed benchmark can still point in the right direction, and a vendor with an incentive can still be right. The goal is calibration: holding each claim with the confidence its evidence supports, no more and no less.

Open questions remain. Researchers are still working out how to build benchmarks that resist contamination, how to measure open-ended work such as writing or analysis, and how to run fair productivity studies when people no longer want to work without AI²³.

What this means for you

When you meet an AI claim this week, in a pitch, a report or a headline, run it through five questions:

1. What exactly is being claimed? Split it into its parts: a test score, a study result, a survey finding or a product promise. Each needs a different check.
2. What was actually measured? Find the benchmark, task or survey question behind the claim. Ask whether it matches the thing you care about, and whether the questions could have been seen in training.
3. Compared with what? Look for a baseline: people without AI, an older system, a simple rule. Prefer measured outcomes over self-reported ones, and random assignment over self-selection.
4. How big, and how sure? Ask for the sample size and the error bar. A small lead with no uncertainty attached is not yet a lead.
5. Who says so, and has it been tested on cases like mine? Note the source's stake in the result, then run a small test on your own work, including your hardest cases, before relying on it.

If a vendor cannot answer the second and third questions, treat the claim as marketing until shown otherwise.

Sources

1. Becker, J., Rush, N., Barnes, E., & Rein, D. (2025). Measuring the impact of early-2025 AI on experienced open-source developer productivity. arXiv:2507.09089. METR. arxiv.org/abs/2507.09089 (summary: metr.org/blog/2025-07-10-early-2025-ai-experienced-os-dev-study)
2. Becker, J., Rush, N., Cunningham, T., Rein, D., & Mahamud, K. (2026, February 24). We are changing our developer productivity experiment design. METR. metr.org/blog/2026-02-24-uplift-update
3. Bean, A. M., et al. (2025). Measuring what matters: Construct validity in large language model benchmarks. NeurIPS 2025, Datasets and Benchmarks Track. arXiv:2511.04703. arxiv.org/abs/2511.04703
4. Zhang, H., Da, J., Lee, D., et al. (2024). A careful examination of large language model performance on grade school arithmetic. NeurIPS 2024, Datasets and Benchmarks Track. arXiv:2405.00332. arxiv.org/abs/2405.00332
5. Singh, S., Nan, Y., Wang, A., et al. (2025). The leaderboard illusion. arXiv:2504.20879. arxiv.org/abs/2504.20879
6. Miller, E. (2024). Adding error bars to evals: A statistical approach to language model evaluations. arXiv:2411.00640. arxiv.org/abs/2411.00640
7. Kapoor, S., & Narayanan, A. (2022). Leakage and the reproducibility crisis in ML-based science. arXiv:2207.07048 (later published in Patterns, 2023). arxiv.org/abs/2207.07048
8. US Securities and Exchange Commission. (2024, March 18). SEC charges two investment advisers with making false and misleading statements about their use of artificial intelligence (Press release 2024-36). sec.gov/newsroom/press-releases/2024-36
9. US Federal Trade Commission. (2025, February 11). FTC finalizes order with DoNotPay that prohibits deceptive "AI lawyer" claims, imposes monetary relief, and requires notice to past subscribers. ftc.gov/node/87474
10. Osler, Hoskin & Harcourt LLP. (2024, December 18). Navigating AI systems in capital markets: Recent guidance from the CSA (summary of CSA Staff Notice and Consultation 11-348, December 5, 2024). osler.com
11. Ramel, D. (2025, August 19). MIT report finds most AI business investments fail, reveals "GenAI divide" (on MIT Project NANDA, The GenAI Divide: State of AI in Business 2025, survey data). Virtualization Review. virtualizationreview.com