How to read an AI research paper: a practical method and what peer review can and can't tell you

A headline says a new AI model "beats doctors," "reasons like a mathematician," or "cuts costs by 40%." Somewhere underneath is a research paper, often a PDF on arXiv posted a few days earlier. Most people never open it. The few who do usually read the abstract, glance at a chart, and close the tab.

This article gives you a better way. You'll get a three-pass reading method used by researchers for decades, a short checklist for the specific ways AI papers mislead (usually without anyone intending to), and the current evidence on how much weight the label "peer-reviewed" can actually bear. You don't need to follow the maths to do most of this. You need to know where to look and what to ask.

Why this skill matters more every year

The volume of AI research is now far beyond what anyone can read. arXiv, the free preprint server where most AI papers first appear, received about 284,000 new submissions across all fields in 2025, up roughly 17% from about 244,000 in 2024, and set a monthly record of 32,040 in June 2026². (These are arXiv-wide totals from its public statistics. arXiv does not publish an easy-to-read computer-science share, so treat them as a measure of overall volume.) Stanford's AI Index counted over 242,000 AI publications in 2023, up from about 102,000 in 2013, and found AI had grown to 41.8% of all computer-science publications⁴.

Conferences, which are where AI research is formally judged, are straining under the same load. NeurIPS, the field's largest venue, received 21,575 submissions in 2025 and accepted 5,290, about 24.5%³. GPTZero, citing NeurIPS's own program chairs, notes submissions more than doubled from 9,467 in 2020¹¹.

Two terms are worth defining now. A preprint is a paper posted publicly before, or without, formal review. Anyone with an account can post one, and arXiv's moderators screen for basic suitability, not correctness. Peer review is the process where a venue sends a paper to independent researchers who recommend accepting or rejecting it. In AI, most important work appears as a preprint first and some never goes through review at all. That's not a scandal. It's how a fast field shares results. But it means the reader carries more of the burden of judgment.

The three-pass method

The most widely taught approach comes from S. Keshav, a computer scientist at the University of Waterloo, in a two-page note he wrote after using the method for 15 years¹. The idea is simple: don't read a paper once, top to bottom. Read it in up to three passes, each deeper than the last, and stop as soon as you know what you need.

Pass one: the bird's-eye view (5–10 minutes). Read the title, abstract and introduction carefully. Then read only the section and sub-section headings, the conclusions, and glance at the reference list¹. At the end, Keshav says you should be able to answer the "five Cs":

• Category: What kind of paper is this? A new method, a measurement study, a benchmark, a position piece?
• Context: What earlier work does it build on?
• Correctness: Do the assumptions look valid?
• Contributions: What does it claim is new?
• Clarity: Is it well written?¹

For most readers, most of the time, pass one is enough. Its purpose is to decide whether the paper deserves more of your attention.

Pass two: the content, not the details (up to an hour). Read the whole paper, skipping proofs and heavy derivations. Look hard at the figures and tables. Keshav specifically tells readers to check whether graphs have labelled axes and whether results show error bars¹. Afterward you should be able to explain the paper's main point, with its supporting evidence, to someone else.

Pass three: virtually re-implement it (one to five hours). This is for papers you need to rely on or build from. The goal is to reconstruct the work in your head and "identify and challenge every assumption in every statement"¹. Most non-specialists will rarely need this pass. If a decision with real money or risk depends on a paper, this is the point to bring in someone who can do it.

The method works because it matches effort to stakes. A product manager skimming twenty papers for a landscape review needs pass one twenty times, not pass three once.

Seven questions for AI papers specifically

Keshav's method is general. AI papers have their own recurring failure modes, documented by researchers inside the field. These seven questions target them. Most can be answered in pass two without understanding the maths.

1. What exactly was compared to what? Find the main results table. What is the new method being compared against (the "baseline")? Is the baseline a strong, recent system, or an old or weakened one? A big gain over a weak baseline tells you little.

2. Where did the improvement actually come from? In an influential critique, Zachary Lipton and Jacob Steinhardt listed four "troubling trends" in machine-learning scholarship. One was "failure to identify the sources of empirical gains": papers credit a clever new architecture when the gain really came from extra tuning⁷. Look for an ablation study, an experiment that removes one component at a time to see which part matters. No ablation means the paper's explanation of why it works is closer to a hypothesis.

3. Could the test data have leaked into training? Data leakage means information from the test set seeps into the training process, so the model is partly being tested on material it has already seen. Sayash Kapoor and Arvind Narayanan of Princeton surveyed the problem and found leakage-related errors documented in 17 scientific fields, affecting 329 papers. In one case study on predicting civil wars, every paper claiming complex machine-learning models beat simple statistical ones failed to reproduce once leakage was fixed⁸. Look for how the data was split, and whether the paper says so clearly.

A related problem for large language models is benchmark contamination: test questions appear somewhere in the enormous web-scale training data. When researchers at Scale AI wrote GSM1k, a fresh set of grade-school maths problems built to match a popular benchmark, some models showed accuracy drops of up to 8% on the new problems, a sign of partial memorization¹⁰. The same study found that many frontier models showed minimal signs of overfitting¹⁰, so contamination is a question to ask, not a verdict to assume.

4. How many runs, and how much uncertainty? AI results vary from run to run because of randomness in training. A study that won an Outstanding Paper award at NeurIPS 2021 found that much of deep reinforcement-learning research compared single average scores from only a handful of runs, and that conclusions often changed under a more careful statistical analysis⁹. Look for confidence intervals or error bars (ranges that show how much a result might move by chance). A one-point win with no error bars is not a win you can rely on.

5. Does the benchmark measure the thing in the headline? A benchmark is a standard test set used to compare systems. Scoring well on a medical-exam benchmark is not the same as treating patients. Passing coding puzzles is not the same as maintaining a large codebase. Read the benchmark's description and ask how far it is from the real-world claim.

6. Is the language doing more work than the evidence? Lipton and Steinhardt also flagged "failure to distinguish between explanation and speculation" and "misuse of language," such as words like "understands" or "reasons" used with everyday meanings the experiments don't support⁷. When you see a strong verb, find the experiment behind it.

7. Who wrote it, and what could they release? Many headline AI papers are technical reports from companies about their own models. These can be excellent and are often the only source of information. But check whether the code, data, or model weights are available so others can verify the claims, and note that the authors have a stake in the result.

What "peer-reviewed" does and doesn't tell you

It's tempting to treat acceptance at a top conference as a seal of quality. The evidence says to treat it as a weak positive signal.

Review is noisy. In 2014, NeurIPS ran an experiment: 166 submissions were independently reviewed by two separate committees. Of the papers the first committee accepted, the second accepted only about half⁵. The authors, Corinna Cortes and Neil Lawrence, caution that the true figure is uncertain (somewhere between roughly 38% and 62%), but their summary stands: review was "good for identifying poor papers, but poor for identifying good papers." Among accepted papers, reviewers' quality scores had no meaningful correlation with later citation impact⁵. This is a 2014 sample from a much smaller conference, and the field has grown more than tenfold since, so it is better read as a demonstration of noise than a current measurement.

Review is now under pressure from AI itself. For ICLR 2026, another major conference, the AI-detection company Pangram Labs analysed roughly 70,000 reviews and estimated that 21%, or 15,899, were fully AI-generated, with over half showing some AI involvement¹³. Nature reported the controversy, noting academics had flagged reviews containing hallucinated citations and vague feedback¹⁴. These figures come from a detection vendor's own model, so they are estimates with unknown error rates, not audited counts.

Errors get through. In January 2026, GPTZero, another AI-detection company, reported finding 100 fabricated ("hallucinated") citations in about 53 of the 4,841 papers accepted at NeurIPS 2025, each paper having been seen by multiple reviewers¹¹¹². A follow-up analysis put that at roughly 1% of accepted papers¹⁵. The NeurIPS board responded that incorrect references do not necessarily invalidate a paper's content¹². That's fair: a bad citation is not a bad result. But it shows that reviewers do not check everything, and readers shouldn't assume they did.

Gatekeepers are adjusting. In October 2025, arXiv's computer-science section began requiring review articles and position papers to have passed peer review before posting, citing an "unmanageable influx" and saying generative AI had added to the flood by making such papers "fast and easy to write"⁶. Original research papers are unaffected. The takeaway for readers: survey papers on arXiv CS from late 2025 onward have passed one more filter, but research preprints still have not.

The limits of reading alone

Even a careful three-pass reading has a ceiling. Kapoor and Narayanan stress that in the leakage cases they studied, none of the errors "could have been caught by reading the papers"⁸. They were found only by re-running the code. Reading tells you whether a claim is well argued. It cannot always tell you whether it is true.

That points to three habits that matter as much as reading technique:

• Wait for replication. A result reproduced by an independent group is worth far more than a single paper, however prestigious its venue.
• Look for the response. Strong claims in AI usually attract critiques within weeks, on arXiv or in public discussion. Search for the paper's title plus "replication" or "critique" before relying on it. Social media is useful here as a pointer to those responses, not as evidence itself.
• Separate the finding from the framing. The table might show a modest, real improvement on one benchmark, while the abstract, press release and headline each stretch it a little further. Anchor on the table.

There are also open questions this method doesn't settle. No one yet knows how much AI-assisted writing and reviewing is changing the average quality of published work, because detection tools themselves have uncertain accuracy. And as frontier labs publish less detail about their largest models, more of what the public knows comes from company reports that outsiders cannot fully verify.

What this means for you

You can use this the next time an AI paper crosses your desk, this week:

1. Do pass one before you share or act. Ten minutes: abstract, introduction, headings, conclusions, figures. Answer the five Cs. Most hype collapses at "Category" (it's a position paper) or "Contributions" (the claim is narrower than the headline).
2. Find the main results table and ask what it's compared to. If the baseline is old or weak, discount the gain.
3. Look for three things: an ablation, error bars, and a data split. Missing all three is a strong signal to wait for replication.
4. Check the status. Preprint, accepted paper, or company technical report? Each deserves a different level of trust, and none deserves blind trust.
5. Search for the critique. Title plus "replication" or "critique." If nothing exists yet, the claim is young.
6. Match effort to stakes. For a decision with real budget or risk, get a specialist to do pass three, or wait for independent confirmation.

Sources

1. Keshav, S. (2007). How to read a paper. ACM SIGCOMM Computer Communication Review, 37(3), 83–84. doi.org/10.1145/1273445.1273458 (full text: web.stanford.edu/class/ee384m/Handouts/HowtoReadPaper.pdf)
2. arXiv. (2026). Monthly submissions (statistics as of September 19, 2026). arxiv.org/stats/monthly_submissions
3. NeurIPS Foundation. (2025). NeurIPS 2025 fact sheet. media.neurips.cc/Conferences/NeurIPS2025/press/NeurIPS2025-Fact_Sheet.pdf
4. Stanford Institute for Human-Centered AI. (2025). Artificial Intelligence Index Report 2025, Chapter 1: Research and Development. hai.stanford.edu/assets/files/hai_ai-index-report-2025_chapter1_final.pdf
5. Cortes, C., & Lawrence, N. D. (2021). Inconsistency in conference peer review: Revisiting the 2014 NeurIPS experiment. arXiv:2109.09774. arxiv.org/abs/2109.09774
6. Boboris, K. (2025, October 31). Attention authors: Updated practice for review articles and position papers in arXiv CS category. arXiv blog. blog.arxiv.org/2025/10/31/attention-authors-updated-practice-for-review-articles-and-position-papers-in-arxiv-cs-category
7. Lipton, Z. C., & Steinhardt, J. (2019). Troubling trends in machine learning scholarship. ACM Queue, 17(1). queue.acm.org/detail.cfm?id=3328534 (preprint: arxiv.org/abs/1807.03341)
8. Kapoor, S., & Narayanan, A. (2023). Leakage and the reproducibility crisis in machine-learning-based science. Patterns, 100804. doi.org/10.1016/j.patter.2023.100804 (preprint: arxiv.org/abs/2207.07048)
9. Agarwal, R., Schwarzer, M., Castro, P. S., Courville, A., & Bellemare, M. G. (2021). Deep reinforcement learning at the edge of the statistical precipice. Advances in Neural Information Processing Systems 34 (Outstanding Paper Award). arxiv.org/abs/2108.13264
10. Zhang, H., Da, J., Lee, D., et al. (2024). A careful examination of large language model performance on grade school arithmetic. NeurIPS 2024 Datasets and Benchmarks Track. arxiv.org/abs/2405.00332
11. Shmatko, N., Adam, A., Esau, P., Cui, A., & Tian, E. (2026, January 21). GPTZero finds 100 new hallucinations in NeurIPS 2025 accepted papers. GPTZero. gptzero.me/news/neurips
12. Goldman, S. (2026, January 21). NeurIPS, one of the world's top academic AI conferences, accepted research papers with 100+ AI-hallucinated citations, new report claims. Fortune. fortune.com/2026/01/21/neurips-ai-conferences-research-papers-hallucinations
13. Emi, B. (2025, November 18). Pangram predicts 21% of ICLR reviews are AI-generated. Pangram Labs. pangram.com/blog/pangram-predicts-21-of-iclr-reviews-are-ai-generated
14. Naddaf, M. (2025). Major AI conference flooded with peer reviews written fully by AI. Nature, 648, 256–257. doi.org/10.1038/d41586-025-03506-6
15. Ansari, S. (2026). Compound deception in elite peer review: A failure mode taxonomy of 100 fabricated citations at NeurIPS 2025. arXiv:2602.05930. arxiv.org/abs/2602.05930