The AI papers that shaped today's systems: a reader's index, from backpropagation to reasoning models

Every week, someone announces that a new AI paper "changes everything." Almost none of them do. Yet a small number of papers really did change the field, and most of what you hear about AI today, from chatbots to reasoning models to protein folding, traces back to them. If you know those few papers, you can place almost any new claim on a map: is this a genuine step, or a variation on something from 2017?

This article is that map. It is not a reading list you have to work through. It is an index: for each landmark paper, a plain-language summary of what it showed, why it mattered, and its main limits. It closes with what a list like this can't tell you, and how to use it the next time a headline cites "a new study."

How this index was chosen

The field produces far more research than anyone can read. Stanford's AI Index counts AI publications in computer science rising from about 102,000 in 2013 to more than 242,000 in 2023, when AI made up 41.8% of all computer science papers, up from 21.6% a decade earlier². At that volume, "important" has to mean something narrower than "widely shared."

The papers below were selected on three tests. First, the idea changed what later researchers did, not just what they cited. Second, the result can be stated in one or two sentences a non-specialist can follow. Third, the core claim has held up as others built on it. Citation counts help but aren't decisive. In Nature's 2025 analysis of the most-cited papers of the 21st century, which took the median rank across five citation databases, the top spot went to Microsoft's 2016 "deep residual learning" paper, and several AI papers, including the transformer paper, sat in the top ten¹. Being cited a lot means a paper is useful to build on. It doesn't mean every claim in it is right.

A note on terms. A preprint is a paper posted publicly, usually on arXiv, before or without formal peer review. A benchmark is a standard test used to compare systems. A parameter is one of the adjustable numbers inside a model that training tunes. More parameters roughly means a bigger model.

Part 1: The foundations of learning, seeing and attending (1986–2017)

Backpropagation (Rumelhart, Hinton and Williams, 1986)³. Neural networks are layers of simple units connected by adjustable weights. The hard problem was how to adjust weights deep inside the network when you only see the error at the end. This short Nature paper popularized backpropagation: pass the error backwards through the layers and nudge each weight in the direction that reduces it. Nearly every model trained today still learns this way. Hinton, then at the University of Toronto, appears again in the next entry. Canadian labs figure prominently in this early history.

AlexNet (Krizhevsky, Sutskever and Hinton, 2012)⁴. A team from the University of Toronto trained a 60-million-parameter neural network on graphics chips (GPUs) and entered the ImageNet image-recognition contest. It achieved a top-5 error rate of 15.3%, against 26.2% for the second-best entry⁴. That gap persuaded much of the field that deep learning, given enough data and compute, beat hand-engineered methods. Limit: the winning figure came from combining seven networks, and image classification is a narrow task.

Residual networks, or ResNet (He et al., 2016)⁵. Very deep networks were hard to train because the learning signal faded as it passed through many layers. The Microsoft team added "shortcut" connections that let information skip layers, which made networks of up to 152 layers trainable, eight times deeper than the previous leading design⁵. An ensemble achieved 3.57% error on the ImageNet test set and won the 2015 contest⁵. Residual connections are now inside almost every large model, including language models, which helps explain why this is the most-cited paper of the century¹.

Attention (Bahdanau, Cho and Bengio, 2014)⁶. Early translation systems squeezed a whole sentence into one fixed-size summary before translating it, which broke down on long sentences. This paper, from Yoshua Bengio's group at the Université de Montréal, let the model "(soft-)search for parts of a source sentence that are relevant to predicting a target word"⁶. That mechanism, called attention, lets a model decide which earlier words matter for the word it is producing now.

The transformer (Vaswani et al., 2017)⁷. Google researchers proposed an architecture "based solely on attention mechanisms, dispensing with recurrence and convolutions entirely"⁷. Because it processes all words in parallel rather than one at a time, it trains far more efficiently on GPUs. The paper's translation model trained in 3.5 days on eight GPUs and set a new state of the art on English-to-French translation⁷. Almost every major language model since, including the GPT, Claude, Gemini and Llama families, is built on this design. Limit: the paper was about translation. Its authors did not claim, and could not have shown, what it would lead to.

Part 2: The scaling era (2018–2022)

BERT (Devlin et al., 2019)⁸. Google showed that a transformer pretrained on large amounts of unlabeled text, by learning to fill in hidden words using context on both sides, could then be adapted to many tasks with little extra training. It set new records on eleven language tasks⁸. The broader lesson was pretraining: learn general knowledge of language first, specialize later.

Scaling laws (Kaplan et al., 2020)⁹. OpenAI researchers found that a language model's error falls in a smooth, predictable way, a power law, as you increase model size, data and compute, "with some trends spanning more than seven orders of magnitude"⁹. That turned building bigger models from a gamble into an engineering plan, and it underpins the investment boom that followed.

GPT-3 (Brown et al., 2020)¹⁰. A 175-billion-parameter model, ten times larger than any previous dense language model, could perform many tasks from just a few examples in the prompt, with no further training¹⁰. This "few-shot" ability is the origin of prompting as most people now know it. Limit: the paper itself documented weaknesses, and its performance on many tasks still trailed specialized systems.

Chinchilla (Hoffmann et al., 2022)¹¹. DeepMind showed that earlier large models were undertrained for their size. For the same compute budget as its 280-billion-parameter Gopher model, a 70-billion-parameter model trained on about four times more data "uniformly and significantly" outperformed Gopher, GPT-3 and other larger models, reaching 67.5% on the MMLU knowledge benchmark¹¹. The practical point: data matters as much as size, and smaller well-trained models are cheaper to run.

Part 3: Making models useful, and making them reason (2020–2025)

InstructGPT (Ouyang et al., 2022)¹². A raw pretrained model predicts text. It doesn't reliably follow instructions. OpenAI fine-tuned GPT-3 using human ratings of its answers, a method known as reinforcement learning from human feedback (RLHF). Human evaluators preferred outputs from the 1.3-billion-parameter InstructGPT over the 175-billion-parameter GPT-3, "despite having 100x fewer parameters"¹². This is the core recipe that turned language models into assistants.

Chain-of-thought prompting (Wei et al., 2022)¹³. Google researchers showed that prompting a model with worked examples that include intermediate reasoning steps "significantly improves the ability of large language models to perform complex reasoning"¹³. With just eight such examples, a 540-billion-parameter model reached state-of-the-art accuracy on a grade-school maths benchmark¹³. Today's "thinking" models build this behaviour in through training rather than prompting.

Retrieval-augmented generation, or RAG (Lewis et al., 2020)¹⁴. Researchers at Facebook AI (now Meta) combined a language model with a searchable index of Wikipedia, so the model could look up documents before answering¹⁴. Pairing a model's built-in knowledge with retrieved sources is now the standard way businesses connect AI to their own documents.

Constitutional AI (Bai et al., 2022)¹⁵. Anthropic trained a harmless assistant "without any human labels identifying harmful outputs"¹⁵. Instead, the model critiqued and revised its own answers against a written list of principles, and then learned from AI-generated rather than human preferences. It is one of the main alternatives to relying entirely on human raters.

DeepSeek-R1 (DeepSeek-AI, Guo et al., 2025)¹⁷. The Chinese lab DeepSeek reported that reasoning ability "can be incentivized through pure reinforcement learning," without human-written examples of reasoning¹⁷. During training, one version's score on the AIME 2024 maths competition rose from 15.6% to 71.0% on a single attempt, and to 86.7% with majority voting across many attempts¹⁷. Published in Nature in September 2025, it was described in press coverage as the first major language model to go through formal peer review¹⁷¹⁸. Limit: maths and coding answers are easy to check automatically, which is what makes this kind of training work. Most real-world tasks have no such automatic checker.

Part 4: Beyond chatbots

AlphaFold (Jumper et al., 2021)¹⁶. Predicting a protein's 3D shape from its sequence of building blocks was a long-standing open problem in biology. DeepMind's system showed "accuracy competitive with experimental structures in a majority of cases" in the CASP14 assessment¹⁶. It is the strongest single example of AI producing a scientific result, not just a product.

Measuring long tasks (Kwa et al., 2025)²⁰. Benchmarks go stale quickly, so researchers at METR, a nonprofit evaluator, proposed a different yardstick: how long a task, measured in skilled-human working time, can a model complete with 50% reliability? They found that this "time horizon" had been doubling roughly every seven months since 2019, possibly faster in 2024²⁰. Limit: the tasks are mostly software tasks, and 50% reliability is far below what most businesses need.

What this index can't tell you

Citations measure usefulness, not truth. A paper becomes highly cited when others build on it¹. That is a good signal, but some influential claims are later revised. Chinchilla, for example, corrected the guidance that came out of the earlier scaling work⁹¹¹.

Few landmark AI papers went through journal review. Of the seventeen here, only three are journal articles³¹⁶¹⁷. Most appeared first, or only, as preprints or at conferences, where review is faster and lighter than at journals, and some were never formally reviewed at all. That R1's journal review was treated as news in 2025 shows how unusual it was¹⁸. The earlier article in this series, How to Read an AI Research Paper, covers how to weigh that.

The frontier has become less transparent. OpenAI's 2023 GPT-4 technical report said, citing "the competitive landscape and the safety implications," that it "contains no further details about the architecture (including model size), hardware, training compute, dataset construction, training method, or similar"¹⁹. Many of the most important systems since then are described in model cards and technical reports, not papers that others can reproduce. Meanwhile, nearly 90% of notable models in 2024 came from industry rather than academia², and in 2024 US institutions produced 40 notable models against China's 15². The papers in this list are increasingly a history of ideas, while the current state of the art lives in documents with fewer details.

Benchmark numbers are snapshots. Every score quoted above was a record when published and has since been beaten. Treat them as evidence that a method worked at the time, not as a measure of what systems can do today.

What this means for you

• Place new claims on the map. When a product or headline says "breakthrough," ask which of these ideas it builds on. Most "new" systems are a transformer, pretrained, fine-tuned with feedback and connected to retrieval. Real novelty is rarer and easier to spot once you know the baseline.
• Ask whether it was reviewed and whether it can be checked. Is it a preprint, a peer-reviewed paper, or a company blog post? Are the data and methods disclosed? You don't need to read the maths to ask those two questions.
• Look for the "limit" sentence. Every paper above has one: one narrow task, an ensemble rather than a single model, answers that are easy to check, 50% reliability. If coverage of a new study doesn't mention its limits, look them up before repeating the claim.
• If you read one paper, read the abstract and the conclusion of the transformer or InstructGPT paper. Both are short, readable and explain much of what you use every day.

Sources

1. Pearson, H., Ledford, H., Hutson, M., & Van Noorden, R. (2025, April 15). Exclusive: the most-cited papers of the twenty-first century. Nature, 640, 588–592. nature.com/articles/d41586-025-01125-9
2. Stanford Institute for Human-Centered AI. (2025). The 2025 AI Index Report, Chapter 1: Research and Development. hai.stanford.edu/assets/files/hai_ai-index-report-2025_chapter1_final.pdf
3. Rumelhart, D. E., Hinton, G. E., & Williams, R. J. (1986). Learning representations by back-propagating errors. Nature, 323, 533–536. doi.org/10.1038/323533a0
4. Krizhevsky, A., Sutskever, I., & Hinton, G. E. (2012). ImageNet classification with deep convolutional neural networks. Advances in Neural Information Processing Systems 25. papers.nips.cc/paper/2012/hash/c399862d3b9d6b76c8436e924a68c45b-Abstract.html
5. He, K., Zhang, X., Ren, S., & Sun, J. (2016). Deep residual learning for image recognition. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR). arxiv.org/abs/1512.03385
6. Bahdanau, D., Cho, K., & Bengio, Y. (2014). Neural machine translation by jointly learning to align and translate. arXiv:1409.0473 (ICLR 2015). arxiv.org/abs/1409.0473
7. Vaswani, A., et al. (2017). Attention is all you need. Advances in Neural Information Processing Systems 30. arxiv.org/abs/1706.03762
8. Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2019). BERT: Pre-training of deep bidirectional transformers for language understanding. Proceedings of NAACL-HLT 2019. arxiv.org/abs/1810.04805
9. Kaplan, J., et al. (2020). Scaling laws for neural language models. arXiv:2001.08361. arxiv.org/abs/2001.08361
10. Brown, T. B., et al. (2020). Language models are few-shot learners. Advances in Neural Information Processing Systems 33. arxiv.org/abs/2005.14165
11. Hoffmann, J., et al. (2022). Training compute-optimal large language models. Advances in Neural Information Processing Systems 35. arxiv.org/abs/2203.15556
12. Ouyang, L., et al. (2022). Training language models to follow instructions with human feedback. Advances in Neural Information Processing Systems 35. arxiv.org/abs/2203.02155
13. Wei, J., et al. (2022). Chain-of-thought prompting elicits reasoning in large language models. Advances in Neural Information Processing Systems 35. arxiv.org/abs/2201.11903
14. Lewis, P., et al. (2020). Retrieval-augmented generation for knowledge-intensive NLP tasks. Advances in Neural Information Processing Systems 33. arxiv.org/abs/2005.11401
15. Bai, Y., et al. (2022). Constitutional AI: Harmlessness from AI feedback. arXiv:2212.08073. arxiv.org/abs/2212.08073
16. Jumper, J., et al. (2021). Highly accurate protein structure prediction with AlphaFold. Nature, 596(7873), 583–589. pmc.ncbi.nlm.nih.gov/articles/PMC8371605
17. DeepSeek-AI, Guo, D., Yang, D., et al. (2025). DeepSeek-R1 incentivizes reasoning in LLMs through reinforcement learning. Nature, 645, 633–638. Preprint: arXiv:2501.12948. arxiv.org/abs/2501.12948
18. The Daily Star. (2025, September). DeepSeek R1 is now a peer-reviewed AI model. online-d11.thedailystar.net
19. OpenAI. (2023). GPT-4 technical report, Section 2. arXiv:2303.08774. arxiv.org/abs/2303.08774
20. Kwa, T., West, B., Becker, J., et al. (2025). Measuring AI ability to complete long software tasks. arXiv:2503.14499. arxiv.org/abs/2503.14499