Foundations

Machine learning basics: how computers learn from examples, and why a good test score can mislead

What "learning" means for a computer, the three main ways it happens, and the traps that make a model look better than it is.

Manelink Research15 November 2026 · 8 sources · 9 min read

In brief

Machine learning builds a program by fitting patterns in example data instead of writing rules by hand. The real test is how a model performs on data it has not seen, because a good training score proves little. The textbook overfitting rule has been revised by double descent, and simple tree-based models still beat deep networks on medium-sized tables. Leakage and shortcuts quietly inflate results, and most of the work and risk sits in the data and the system around the model.

When a bank flags a suspicious card payment, your email hides a spam message, or a streaming service suggests a show, nobody sat down and wrote the rule that made that call. A program was shown many examples and worked out the rule itself. That approach is called machine learning, and it sits underneath nearly everything currently sold as "AI," from fraud screens to chatbots.

This article explains the core idea in plain terms: what a model is, how it learns, and how we judge whether it learned something real. It then looks at what recent research says about where the classic advice still holds, where it has been revised, and the quiet failure modes that make a model look better than it is. If you buy, build or just read about AI, this is the vocabulary that makes the rest of the field easier to follow.

The core idea: learning from examples instead of writing rules

In conventional software, a person writes the rules and the computer follows them. In machine learning, a person supplies examples and a goal, and a learning method produces the rules. Michael Jordan and Tom Mitchell, two of the field's best-known researchers, put it this way in a 2015 review in Science: machine learning "addresses the question of how to build computers that improve automatically through experience"1. They also describe it as lying "at the intersection of computer science and statistics"1, which is a useful reminder that much of it is careful pattern-fitting, not magic.

The idea is old. In 1959 Arthur Samuel of IBM published a paper on teaching a computer program to play checkers through experience2. What changed since then is the amount of data and computing power available, not the basic bargain.

Three terms come up constantly:

  • Training data is the set of examples the method learns from, such as past emails each labelled "spam" or "not spam."
  • Model is the result: a mathematical function that takes a new input (a new email) and produces an output (a spam score). Its adjustable internal numbers are called parameters.
  • Training is the process of adjusting the parameters so the model's outputs match the examples as closely as possible.

Three ways machines learn

Most machine learning falls into three families, distinguished by what feedback the learner gets.

Supervised learning uses examples that come with the right answer attached. Each email is labelled, and each past loan is marked repaid or defaulted. The model learns to predict the label for new cases. This is the workhorse of business AI: fraud detection, demand forecasting, credit scoring and medical image classification are mostly supervised problems.

Unsupervised learning uses data with no labels and looks for structure on its own, for example grouping customers into segments that behave alike, or flagging transactions that look unlike anything else.

Reinforcement learning learns by trial and error from rewards. A system tries actions, gets a score, and shifts toward actions that score better. Samuel's checkers program is an early ancestor of this idea2, and it is one ingredient in how modern chatbots are tuned to follow instructions.

These families are tools, not rivals. A real product often combines them. Jordan and Mitchell's review covers all three and notes how widely they were already being applied by 20151.

The one test that matters: does it work on data it has not seen?

A model can be perfect on its training examples and useless in the real world. The extreme case is memorization: a model that simply stores every training example answers them all correctly while having learned nothing general. This failure is called overfitting. The opposite, a model too simple to capture the real pattern, is underfitting.

The standard defence is to hold some data back. Builders split their data, train on one part, and measure performance on the part the model never saw, called the test set. The test score estimates how the model will behave on new cases. Everything in the field of evaluation rests on that one separation, which is why the failures described later in this article all involve it being quietly broken.

The classic textbook picture of the trade-off is a U-shaped curve. A model that is too simple has high error because it misses the pattern. A model that is too flexible also has high error because it fits noise. The best test performance lies in a "sweet spot" in between.

The textbook curve, revised

That picture was long treated as a rule. A 2019 paper in Proceedings of the National Academy of Sciences by Mikhail Belkin, Daniel Hsu, Siyuan Ma and Soumik Mandal showed that it is incomplete3. They describe a point called the interpolation threshold, where a model is just flexible enough to fit its training data perfectly. Below it, models behave as the textbook says, following the classic U shape. Beyond it, making the model even bigger can make test error fall again, "typically going below the risk achieved at the sweet spot"3. The full pattern is called double descent.

The authors demonstrated it on the handwritten-digit dataset MNIST. In one experiment with 10,000 training examples and 10 classes, the threshold arrived at 10,000 features in the model. Larger models past that point did better, not worse3. They also noted that a dataset as large as ImageNet, with roughly a million examples and a thousand classes, "may require networks with about 10^9 parameters to achieve interpolation"3. The paper reports similar behaviour across several model families, including random forests and boosting3.

A related finding from Chiyuan Zhang and colleagues, presented at ICLR 2017, is even more striking. Standard image-recognition networks could "easily fit a random labeling of the training data," even when the images were replaced by completely unstructured random noise4. A model that can memorize noise cannot be trusted to have learned something real just because its training score is high, which is exactly why the held-out test matters. The same paper reports that this behaviour was "qualitatively unaffected by explicit regularization" (techniques designed to discourage overfitting)4.

Caveats: these are results about specific model families and datasets, mostly image tasks, and the PNAS paper shows the effect, not a recipe for how large your own model should be. The practical lesson is humbler: the old rule of thumb that a model fitting its training data perfectly must be overfit is no longer safe to apply blindly.

Newer is not always better: the case for simple models

Public attention goes to giant neural networks, but for the spreadsheet-style data that dominates business (rows of customers, transactions or sensors), the evidence is less one-sided. A 2022 NeurIPS-track study by Léo Grinsztajn, Edouard Oyallon and Gaël Varoquaux compared tree-based models (random forests and gradient-boosted trees) against several deep-learning designs on a standard set of 45 tabular datasets, using a search budget that reached 20,000 compute hours5. Their conclusion: tree-based models "remain state-of-the-art on medium-sized data (about 10K samples) even without accounting for their superior speed"5.

The authors traced this to three challenges for neural networks on tables: being robust to uninformative columns, preserving the orientation of the data, and learning irregular functions5. Limits matter here too. The datasets had a minimum of 3,000 samples, training sets were truncated to 10,000 rows for larger ones, and the paper is a preprint (arXiv, July 2022) that I could not confirm as peer-reviewed from the pages read, so it covers medium-sized tables, not every setting, and newer architectures may have narrowed the gap since5. For a business with a few thousand to a few hundred thousand rows, the practical takeaway is that a well-tuned simple model is a serious baseline, and it is also cheaper to run.

How models quietly fool us

Two failure modes deserve special attention because they produce results that look excellent.

Shortcut learning. Robert Geirhos and colleagues define shortcuts as "decision rules that perform well on standard benchmarks but fail to transfer to more challenging testing conditions, such as real-world scenarios"6. A model asked to find a disease may instead learn to spot the hospital that took the scan, if sicker patients tend to be scanned at one site. It scores well on test data from the same source, then fails elsewhere. The authors argue many seemingly separate deep-learning problems are "different symptoms of the same underlying problem: shortcut learning"6. (The abstract does not itself contain the hospital example. It is offered here as an illustration of the idea, not a finding of the paper.)

Data leakage. Leakage happens when information from the test set sneaks into training, so the test no longer measures performance on unseen data. Sayash Kapoor and Arvind Narayanan surveyed research communities that had adopted machine learning and found errors of this kind in 17 fields, collectively affecting 329 papers, and built a taxonomy of 8 types of leakage7. In a case study on civil-war prediction, they report that all papers claiming complex ML models beat simple logistic regression "fail to reproduce due to data leakage"7. Their remedy is a structured reporting form, "model info sheets," so readers can check what a model was and was not given7. The study counts papers where errors were found. It does not tell us the error rate across all machine-learning research.

The model is the small part of the system

One more reality check comes from industry. In a 2015 NeurIPS paper, D. Sculley and colleagues at Google argued that "it is common to incur massive ongoing maintenance costs in real-world ML systems"8. They list risks that traditional software lacks: tangled dependencies on data, feedback loops where a model's outputs change its future training data, undeclared consumers of its predictions, and changes in the outside world that quietly make a model stale8. This is an engineering team's analysis, not a controlled study, but it matches what many practitioners report: getting a model to work once is the easy part, and keeping it correct as the world shifts is where the cost sits.

What this means for you

  • Ask what the model learned from. A vendor claiming "AI" should be able to say what examples trained it, how recent they are, and how similar they are to your data.
  • Ask how it was tested. You want performance on data the model never saw, ideally from a different time period or site than the training data, because that is what exposes shortcuts and leakage67.
  • Ask for a simple baseline. If you have tabular data, ask whether a basic model such as a tree-based one was compared. If it performs about the same, the cheaper option is usually the safer choice5.
  • Do not trust a perfect training score. A model that fits training data flawlessly tells you little, since modern networks can fit even random noise4.
  • Budget for upkeep. Plan for monitoring and retraining. The ongoing cost is part of the price8.

Key takeaways

  • Machine learning builds a program by fitting patterns in example data instead of writing rules by hand. The three main styles are supervised, unsupervised and reinforcement learning.12
  • The only score that counts is performance on data the model has not seen. A good training score proves little because models can memorize even random noise.34
  • The classic overfitting rule has been revised (double descent), and simple tree-based models still compete with deep networks on medium-sized tabular data.35
  • Shortcut learning and data leakage can make results look far better than they are, and real systems carry ongoing maintenance costs.678

Sources

  1. Jordan, M. I., & Mitchell, T. M. (2015). Machine learning: Trends, perspectives, and prospects. Science, 349(6245), 255–260. doi.org/10.1126/science.aaa8415
  2. Samuel, A. L. (1959). Some studies in machine learning using the game of checkers. IBM Journal of Research and Development, 3(3), 210–229. doi.org/10.1147/rd.33.0210
  3. Belkin, M., Hsu, D., Ma, S., & Mandal, S. (2019). Reconciling modern machine-learning practice and the classical bias–variance trade-off. Proceedings of the National Academy of Sciences, 116(32), 15849–15854. doi.org/10.1073/pnas.1903070116
  4. Zhang, C., Bengio, S., Hardt, M., Recht, B., & Vinyals, O. (2017). Understanding deep learning requires rethinking generalization. International Conference on Learning Representations (ICLR 2017), arXiv:1611.03530. arxiv.org/abs/1611.03530
  5. Grinsztajn, L., Oyallon, E., & Varoquaux, G. (2022). Why do tree-based models still outperform deep learning on tabular data? arXiv:2207.08815. arxiv.org/abs/2207.08815
  6. Geirhos, R., Jacobsen, J.-H., Michaelis, C., Zemel, R., Brendel, W., Bethge, M., & Wichmann, F. A. (2020). Shortcut learning in deep neural networks. arXiv:2004.07780, DOI 10.1038/s42256-020-00257-z. arxiv.org/abs/2004.07780
  7. Kapoor, S., & Narayanan, A. (2023). Leakage and the reproducibility crisis in machine-learning-based science. Patterns, 4(9), 100804. doi.org/10.1016/j.patter.2023.100804, figures from the preprint, arXiv:2207.07048. arxiv.org/abs/2207.07048
  8. Sculley, D., Holt, G., Golovin, D., Davydov, E., Phillips, T., Ebner, D., Chaudhary, V., Young, M., Crespo, J.-F., & Dennison, D. (2015). Hidden technical debt in machine learning systems. Advances in Neural Information Processing Systems 28. proceedings.neurips.cc/paper/2015/hash/86df7dcfd896fcaf2674f757a2463eba-Abstract.html