Is this an AI problem? A practical test for deciding when AI fits, and when it doesn't

Someone on your team has an idea: "We should use AI for this." It might be sorting customer emails, forecasting inventory, drafting proposals or screening invoices. The idea sounds modern and the tools are easy to buy. The question that rarely gets asked first is the one that decides whether the money is well spent: is this actually a problem AI is suited to solve?

This article gives you a way to answer it. It explains what AI is good at in plain terms, shows when a simpler approach wins, and turns the best available evidence into a five-question test you can apply to any proposal, pitch or pilot. It won't tell you whether a particular product is good. It will help you avoid the most common and most expensive mistake: pointing AI at the wrong problem.

Why this question comes first

AI use is spreading quickly, but it is still a minority practice. Statistics Canada found that 19.2% of Canadian businesses used AI to produce goods or deliver services in the 12 months before its spring 2026 survey, up from 6.1% two years earlier. In the same survey, 40.0% of businesses said AI is "not relevant to the business"¹. In the United States, the Census Bureau's biweekly Business Trends and Outlook Survey put AI use at 19.8% of businesses as of May 3, 2026, with use rates of 37% among firms with 250 or more employees². (The two surveys ask different questions, so the numbers are not directly comparable.) In other words, most organizations are still deciding where AI belongs, and many have concluded that, for now, it doesn't.

That decision matters because AI projects fail often, and they tend to fail for reasons that have little to do with the technology. In a 2024 study, RAND researchers interviewed 65 experienced data scientists and engineers about why AI projects go wrong. They cite outside estimates that more than 80% of AI projects fail, about twice the rate of IT projects that don't involve AI. The first root cause on their list was that people "misunderstand" or "miscommunicate" "what problem needs to be solved using AI." Others included lacking the data to train a model, focusing "more on using the latest and greatest technology than on solving real problems," and applying AI to problems "too difficult for AI to solve"³. Two caveats. The interviews (August to December 2023) excluded projects built purely on prompting off-the-shelf language models. And the 80% figure is RAND quoting another source, not its own measurement.

Industry analysts report a similar pattern for generative AI. Gartner said in January 2026 that at least 50% of generative AI projects had been abandoned after proof of concept by the end of the previous year, citing poor data quality, inadequate risk controls, escalating costs or unclear business value. It describes this as an analysis of "hundreds" of implementations rather than a published survey⁴. In a June 2025 prediction, Gartner expected over 40% of "agentic" AI projects (systems that take multi-step actions on their own) to be canceled by the end of 2027. Its analyst noted that "many use cases positioned as agentic today don't require agentic implementations" and estimated that only about 130 of the thousands of vendors selling agentic AI offer the real thing⁵. Treat these as analyst judgments, not measured rates, but the direction is consistent: poor problem selection is a leading cause of waste.

What AI is actually good at

A useful mental model comes from three economists at the University of Toronto, Ajay Agrawal, Joshua Gans and Avi Goldfarb, who framed modern machine intelligence as a sharp drop in the cost of prediction: "Prediction is about to get way cheaper"⁶. "Prediction" here is broad. It means filling in missing information from patterns: guessing which transaction is fraudulent, which word comes next, what a blurry image shows, or what a customer probably meant. If your problem can be described as "we have to make an educated guess, many times, from messy inputs," AI may fit. If it can be described as "we have to apply a known rule correctly," it probably doesn't.

Google's People + AI Research (PAIR) team, which publishes design guidance for AI products, makes this concrete. Its guidebook lists situations where an AI approach is "probably better" than a rule-based one, including recommending different content to different users, predicting future events, understanding natural language, recognizing a whole class of things (it's "not possible to program every single face into a photo tagging app"), and detecting rare events that change over time, such as credit card fraud⁷. What these have in common is that the right answer depends on patterns too numerous or too fluid for anyone to write down as rules.

That links back to the definition in our essay What is AI?: an AI system infers how to turn inputs into outputs rather than following rules a developer wrote. If a competent person could write the complete rulebook in an afternoon, you probably don't need inference.

When simpler beats smarter

The same guidebook is equally clear about when AI is "probably not better." Its list includes situations where predictability is the most valuable part of the experience (a "Cancel" button should stay in the same place), where information is static or limited (a credit card entry form), where errors are very costly and "outweigh the benefits of a small increase in success rate," where people need to understand "precisely everything that happens in the code," where speed to market matters more than anything, and where people have said they don't want a task automated⁷. Its framing is blunt: "Often a rule or heuristic-based solution will work just as well, if not better, than an AI version," with the bonus of being "easier to build, explain, debug, and maintain."

Engineers who build large machine-learning systems say the same. The first of Google's widely used "Rules of Machine Learning," written by Martin Zinkevich, is "Don't be afraid to launch a product without machine learning." His reasoning: machine learning requires data, and "if you think that machine learning will give you a 100% boost, then a heuristic will get you 50% of the way there"⁸. (A heuristic is a simple rule of thumb, such as "rank the most recently used contacts first.") He also adds a counterweight: once a heuristic becomes complex and hard to maintain, a learned model is often the better choice. Simple rules first, AI when rules stop scaling.

The hidden cost is maintenance. A well-known 2015 Google paper on "hidden technical debt" in machine-learning systems observed that "only a small fraction of real-world ML systems is composed of the ML code," and that a mature system might be "at most 5% machine learning code and at least 95% glue code"⁹. The model is the visible tip. Data pipelines, monitoring and integration are the iceberg. Generative AI adds its own maintenance burden, because language models can produce fluent but false answers. A 2025 analysis by OpenAI researchers argues this happens partly because "training and evaluation procedures reward guessing over acknowledging uncertainty"¹⁰. Any task where an occasional confident error is unacceptable needs checking built in, and that checking has a cost.

A five-question test

Pulling this together, here is a test you can run on any proposal. It is a judgment aid distilled from the guidance above, not a validated scoring instrument.

1. Can you state the problem and the measure without mentioning AI? "Cut the time to resolve a support ticket from 40 minutes to 25" is a problem. "Deploy an AI assistant" is a solution looking for one. RAND's interviewees found that successful projects are "laser-focused on the problem to be solved, not the technology used to solve it"³.
2. Does the task require an educated guess from patterns, or the correct application of known rules? Guessing from messy inputs (language, images, behaviour, rare events) points toward AI. Applying a fixed policy points toward ordinary software⁶⁷.
3. Do you have the examples, data or context the system needs? A classic model needs historical examples. A general-purpose language model needs your documents, policies and edge cases. Missing data was among the most frequently cited causes of failure in RAND's interviews³.
4. What does a wrong answer cost, and will you catch it? If errors are cheap and visible (a draft a person edits), AI can help even when imperfect. If errors are costly or hidden (a payment sent, a diagnosis missed), you need strong review, or a different approach⁷¹⁰.
5. Does it beat the simplest alternative, on your own work? Compare against a checklist, a template, a rule or a better process, not against doing nothing⁸. The US National Institute of Standards and Technology builds this into its AI Risk Management Framework: mapping a system's context should inform "an initial go/no-go decision about whether to design, develop, or deploy an AI system," and determining whether AI is "an appropriate or necessary tool" is described as a shared responsibility¹¹.

If you can answer the first question clearly and the rest favourably, AI is worth a real pilot. If you stumble on question one, stop there.

What the evidence says about fit

The best studies show that "is this an AI problem?" depends not just on the task but on who is doing it and how.

In one of the most cited field studies, economists Erik Brynjolfsson, Danielle Li and Lindsey Raymond tracked more than 5,000 customer support agents after a generative AI assistant was introduced. Productivity, measured as issues resolved per hour, rose 14% on average, with a 34% improvement for novice and low-skilled workers but "minimal impact on experienced and highly skilled workers"¹². Customer support fits the test well: high volume, patterns learnable from past conversations, and a trained person standing between the AI's suggestion and the customer.

A contrasting result came from METR, a nonprofit that evaluates AI systems. In a randomized trial run in early 2025, 16 experienced open-source developers completed 246 real tasks on codebases they knew well. With AI tools allowed, they took 19% longer, even though they expected a 24% speedup and still believed afterward that AI had sped them up by 20%¹³. METR has since flagged that result as out of date. Its February 2026 follow-up, with 57 developers and more than 800 tasks, produced estimates whose confidence intervals span both speedups and slowdowns, and METR itself calls the new data "an unreliable signal," partly because more developers declined to take part since they didn't want to work without AI¹⁴. The lesson is not that AI helps or hurts coding. It is that experts on familiar work gain less than they think, and that your own perception is a poor measure. Test it.

Large-scale US data shows how businesses are actually applying AI. A Census Bureau working paper using survey data from late 2025 and early 2026 found that 18% of firms used AI in a business function, most in three or fewer functions, and 66% of users used it only to augment tasks rather than automate them. AI-related job cuts occurred in only 2% of firms¹⁵. That pattern fits the test: most organizations are finding narrow, assistive uses, not wholesale replacement.

Limits and open questions

This test has limits worth naming. First, the line moves. Tasks that were "too difficult for AI" in 2023 may not be in 2026, so a "no" today should be revisited, especially for language-heavy work. Second, much of the failure evidence comes from interviews, analyst estimates and surveys, not controlled studies. RAND's sample skews toward engineers, who may be more likely to blame leadership³, and Gartner does not publish its methods⁴⁵. Third, the frameworks quoted here come from organizations that build or sell AI (Google) or set standards for it (NIST). They are sensible, but they are design guidance, not experimental findings. Finally, "not an AI problem" is not the same as "no value." Sometimes the act of scoping an AI project reveals a broken process that a simpler fix can solve, which is a good outcome.

What this means for you

• Rewrite the request without the word "AI." If what's left is a clear problem with a number attached, keep going. If nothing is left, you have a technology looking for a use.
• Name the simplest alternative and price it. A template, a rule, a form or a process change. Make the AI option beat it on your data.
• Run a small test on your hardest cases. Take 20 to 50 real examples, including the awkward ones, and compare outcomes with and without AI. Measure time and errors, not impressions.
• Decide who checks the output and what happens when it's wrong before you scale, not after.
• Ask vendors two questions: what exactly does the system infer, and how does it compare with a non-AI baseline on tasks like yours?

Sources

1. Statistics Canada. (2026, June 11). Analysis on artificial intelligence use by businesses in Canada, second quarter of 2026. Analysis in Brief, 11-621-M. www150.statcan.gc.ca/n1/pub/11-621-m/11-621-m2026010-eng.htm
2. Grundy, A., Breaux, C., & Khatiwoda, D. (2026, May 26). Large firms with at least 20 employees biggest AI users. U.S. Census Bureau, America Counts. census.gov/library/stories/2026/05/ai-use-businesses.html
3. Ryseff, J., De Bruhl, B. F., & Newberry, S. J. (2024). The root causes of failure for artificial intelligence projects and how they can succeed: Avoiding the anti-patterns of AI (RR-A2680-1). RAND Corporation. rand.org/pubs/research_reports/RRA2680-1.html
4. Chandrasekaran, A. (2026, January 26). Why 50% of GenAI projects fail, and how to beat the odds. Gartner. gartner.com/en/articles/genai-project-failure
5. Gartner. (2025, June 25). Gartner predicts over 40% of agentic AI projects will be canceled by end of 2027 [Press release]. gartner.com/en/newsroom/press-releases/2025-06-25-gartner-predicts-over-40-percent-of-agentic-ai-projects-will-be-canceled-by-end-of-2027
6. Agrawal, A., Gans, J., & Goldfarb, A. (2016, November 17). The simple economics of machine intelligence. Harvard Business Review. hbr.org/2016/11/the-simple-economics-of-machine-intelligence
7. Google People + AI Research. (n.d.). User needs + defining success. In People + AI Guidebook. pair.withgoogle.com/chapter/user-needs
8. Zinkevich, M. (n.d., page updated 2025). Rules of machine learning: Best practices for ML engineering. Google for Developers. developers.google.com/machine-learning/guides/rules-of-ml
9. Sculley, D., Holt, G., Golovin, D., Davydov, E., Phillips, T., Ebner, D., Chaudhary, V., Young, M., Crespo, J.-F., & Dennison, D. (2015). Hidden technical debt in machine learning systems. Advances in Neural Information Processing Systems 28. papers.nips.cc/paper_files/paper/2015/hash/86df7dcfd896fcaf2674f757a2463eba-Abstract.html
10. Kalai, A. T., Nachum, O., Vempala, S. S., & Zhang, E. (2025). Why language models hallucinate. arXiv:2509.04664. arxiv.org/abs/2509.04664
11. National Institute of Standards and Technology. (2023). Artificial Intelligence Risk Management Framework (AI RMF 1.0) (NIST AI 100-1). doi.org/10.6028/NIST.AI.100-1
12. Brynjolfsson, E., Li, D., & Raymond, L. R. (2025). Generative AI at work. The Quarterly Journal of Economics, 140(2), 889–942. doi.org/10.1093/qje/qjae044 (figures from NBER Working Paper 31161: nber.org/papers/w31161)
13. Becker, J., Rush, N., Barnes, B., & Rein, D. (2025, July 10). Measuring the impact of early-2025 AI on experienced open-source developer productivity. METR. arXiv:2507.09089. metr.org/blog/2025-07-10-early-2025-ai-experienced-os-dev-study
14. Becker, J., Rush, N., Cunningham, T., Rein, D., & Mahamud, K. (2026, February 24). We are changing our developer productivity experiment design. METR. metr.org/blog/2026-02-24-uplift-update
15. Bonney, K., Breaux, C., Dinlersoz, E., Foster, L., Haltiwanger, J., & Pande, A. (2026, April). The microstructure of AI diffusion: Evidence from firms, business functions, and worker tasks (CES-WP-26-25). U.S. Census Bureau, Center for Economic Studies. census.gov/library/working-papers/2026/adrm/CES-WP-26-25.html