How good is AI at coding? What the benchmarks and field studies show

In 2023, the best AI model could fix fewer than 2 in 100 real software bugs on the standard test¹. By late 2026, AI companies were reporting scores above 90% on a cleaned-up version of that test⁵, and most professional developers were using AI coding tools, many of them every day¹². If you have read a headline saying "AI now writes most code" or "software engineers are next," you have seen where that leads.

The real picture is more interesting and more useful. AI has become very good at certain kinds of programming work, and the evidence on whether it makes developers faster is surprisingly mixed. This article explains what "AI coding" means, walks through the best benchmark and field evidence as of October 2026, and ends with practical questions you can use whether you write code, manage people who do, or buy software built with AI.

What "AI coding" actually means

The term covers three quite different things, and claims often blur them.

• Autocomplete. The tool suggests the next line or block as a developer types. This was the first mass-market form, popularized by GitHub Copilot.
• Chat assistance. A developer describes a problem in plain language and a model (a large language model, or LLM, the kind of system behind ChatGPT, Claude and Gemini) answers with code, explanations or fixes that the person copies, checks and adapts.
• Coding agents. An agent is an AI system that takes actions toward a goal over many steps rather than giving a single answer. A coding agent can read a whole codebase, edit several files, run the program and its tests, read the error messages and try again, often for many minutes or hours with little supervision.

Each step up hands more of the work, and more of the judgment, to the machine. Under the hood the same idea drives all three: an LLM trained to predict the next piece of text, further trained with reinforcement learning (learning by trial and reward) on problems where success can be checked automatically. Code is unusually well suited to that training, because you can run it and see whether the tests pass. That is a big part of why coding has improved faster than most other AI skills.

How we measure coding ability: the SWE-bench story

The most influential yardstick is SWE-bench, introduced by a team led by Princeton researchers and published at ICLR 2024. Instead of toy puzzles, it uses 2,294 real problems taken from GitHub issues in 12 popular Python projects. A model gets the codebase and the issue description, and it passes only if its change makes the project's own tests pass. When the benchmark launched, the best model, Claude 2, solved "a mere 1.96%" of the issues¹.

Benchmarks have flaws, and SWE-bench had some. In 2024 OpenAI had 93 professional developers review a sample of the problems. They flagged 38.3% for vague problem statements and 61.1% for tests that could reject valid solutions. The cleaned-up result, a 500-problem subset called SWE-bench Verified, became the industry's headline coding number².

That number has since run out of road. BenchLM, a tracker that compiles published results, lists top scores on SWE-bench Verified of around 95–96% as of October 17, 2026. Nearly all of those figures are reported by the model makers themselves rather than independently reproduced, and the tracker now shows the table "for reference" and declines to rank models on it, because the top results sit within a single point of each other⁵. When a test is nearly maxed out, it stops telling you much.

Two problems drove researchers to build harder tests. The first is saturation: once most of the problems are solved, the remaining gap is small and partly noise. The second is contamination: public code from open-source projects may have appeared in a model's training data, so a model can partly "remember" a fix rather than work it out. The team behind SWE-Bench Pro, released by Scale AI in September 2025, also noted that 161 of the 500 Verified problems were relatively trivial. Their benchmark uses 1,865 problems from 41 repositories, including private commercial code that models are unlikely to have seen. A typical reference solution changes about 107 lines across four files, closer to a real day's work. When published, the best models scored under 45% on the public set and under 18% on the commercial set³. On Scale's public leaderboard in October 2026, the top score is about 62%⁴.

A different lens comes from METR, a nonprofit that evaluates AI systems. Rather than counting solved problems, it measures how long a task takes a skilled human, then finds the length of task an AI agent can complete half the time. Its task suite is mostly software engineering, machine learning and cybersecurity work. Across 2019 to 2025, that "time horizon" doubled about every seven months, and its January 2026 update found the pace had quickened to roughly every three months since 2024. METR is candid that its estimates for very long tasks are uncertain, because it has measured human completion times for only 5 of its 31 tasks that take eight hours or more⁶.

Put together, the benchmark story is clear: AI went from almost useless to highly capable on well-defined, testable coding tasks in about three years, and it is steadily taking on longer and messier ones. It is not yet reliable on the hardest, least-seen professional work.

Does it make developers faster? The field evidence

Benchmarks test the model alone. What most people care about is what happens when a person and an AI tool work together on real jobs. Here the evidence is more mixed than either enthusiasts or sceptics tend to admit.

Early controlled studies showed big gains on contained tasks. In a 2023 experiment by researchers at Microsoft, GitHub and MIT, developers asked to build a simple web server in JavaScript finished 55.8% faster with GitHub Copilot than without it⁷. A larger study across three companies (Microsoft, Accenture and an anonymous Fortune 100 firm) followed 4,867 developers in their normal jobs. Those given access to Copilot completed 26.08% more tasks, measured as weekly pull requests (proposed code changes submitted for review). The standard error was large (10.3 points), so the true effect could be considerably smaller or larger. Less experienced developers adopted the tool more and appeared to gain more, though the authors describe those subgroup differences as noisy⁸. This is a working paper, not yet peer-reviewed.

Then came a surprise. In early 2025, METR ran a randomized trial with 16 experienced open-source developers working on 246 real tasks in large projects they knew well, averaging five years on each. Tasks were randomly assigned to allow or forbid AI tools. With AI, tasks took 19% longer. The developers themselves had predicted a 24% speedup, and even after the study they believed AI had made them about 20% faster⁹. The gap between how fast people felt and how fast they were is one of the most important findings in this field.

The follow-up is still unsettled. METR repeated the study in late 2025 with 57 developers and more than 800 tasks. Returning developers now showed an estimated 18% speedup, and new recruits about 4%, but in both cases the confidence intervals crossed zero, meaning a slowdown could not be ruled out. METR itself called the data "an unreliable signal." The main reason was revealing: a growing number of developers declined to take part, or held back tasks, because they did not want to work without AI. Between 30% and 50% said they were skipping some tasks for that reason. METR is redesigning the study¹⁰.

Self-reports are much rosier. A METR survey of 349 technical workers in February to April 2026 found a median self-reported 1.4x to 2x increase in the value of their work due to AI, with heavier users of agentic tools reporting more. The authors warn that their convenience sample likely suffers from "significant selection bias" and that self-reports are "not necessarily grounded in reality," pointing to their own finding that people overestimated AI's effect on their time by about 40 percentage points¹¹.

The fair summary is that AI clearly speeds up well-scoped tasks, the gains on complex work in mature codebases are real but smaller and harder to measure than many people feel, and the effect is probably growing as the tools improve.

How developers are using it, and how much they trust it

Usage is now widespread among professionals. Stack Overflow's 2026 Developer Survey, released on October 6, 2026, drew more than 30,000 responses from 169 countries. Only 17.2% of respondents said they do not use AI tools. Two-thirds (66%) use coding assistants or agents, and 26.2% use automated agent workflows. Among those using coding assistants or agents, 73% use them daily¹².

Trust, though, is conditional. In the same survey, 48% said they trust AI when they can easily check its answers, but only 16% extend that trust to most non-critical tasks, and just 6.6% are prepared to trust it with important decisions. Favourable views still outnumber unfavourable ones by almost three to one, and the most common reason given for avoiding AI was the fear of losing one's own skills (17%)¹². In short, developers treat AI like a fast, capable junior colleague whose work always gets reviewed. That is roughly what the evidence supports.

Where AI code still falls short

Security. Code that works is not the same as code that is safe. Veracode, an application-security company, has tested more than 100 AI models on coding tasks with known security pitfalls over four years. In its August 2026 report, the average security pass rate was 56%, meaning that in nearly half of cases the generated code contained a security flaw. The best model in the newest batch reached 68%, and Java was the weakest language at a 30% pass rate¹³. This is vendor research rather than peer-reviewed work, but the direction matches academic findings.

Invented dependencies. Modern software leans on packages, ready-made libraries pulled from public registries. A study analysing 576,000 code samples from 16 models found that, on average, at least 5.2% of the packages suggested by commercial models and 21.7% of those suggested by open-source models did not exist¹⁴. Attackers can register those made-up names and fill them with malicious code, an attack sometimes called "slopsquatting." A developer who installs whatever the AI suggests can be compromised without writing a single bad line.

Context and judgment. Benchmarks give the model a clear issue and a test to pass. Real work starts with a fuzzy request, unwritten team conventions and trade-offs no test captures, such as whether a change will be easy to maintain in two years. The METR results, where experts working in code they knew deeply saw little or no speedup, suggest this is exactly where today's tools add least⁹¹⁰.

Effects on careers. A study by Stanford economists using payroll data covering millions of US workers found that, by June 2026, employment of 22- to 25-year-olds in the occupations most exposed to AI stood 19% below where it would have been had it kept pace with less-exposed peers. The authors' discussion highlights software development among the exposed fields¹⁵. This shows a correlation, not proof that AI caused the gap. The wider economy and hiring cycles may also play a part. But it is a signal worth watching for anyone early in a technical career in North America.

What this means for you

If you write code: Use AI where you can check its work cheaply, such as tests, boilerplate, small well-defined changes and explaining unfamiliar code. Be more careful where you cannot, such as security-sensitive logic or decisions about how a system should be designed. Time yourself on a few real tasks with and without AI. Your gut feeling is not a reliable measure.

If you manage developers: Measure outcomes such as cycle time, defect rates and incidents, not lines of code or how many people use the tool. Keep human code review mandatory for AI-generated changes, and add automated security scanning and checks that every dependency actually exists.

If you are buying software or evaluating a vendor: Ask which benchmark their claims are based on. "90% on SWE-bench Verified" now says little. Results on harder, less-contaminated tests or on your own code say more. Ask how AI-written code is reviewed and tested before it ships.

If you are planning a career: The evidence points to AI taking over more routine coding first. Skills that remain scarce are understanding what to build, reviewing and testing code critically, and knowing how systems fail.

Sources

1. Jimenez, C. E., Yang, J., Wettig, A., Yao, S., Pei, K., Press, O., & Narasimhan, K. (2024). SWE-bench: Can language models resolve real-world GitHub issues? International Conference on Learning Representations (ICLR 2024). arxiv.org/abs/2310.06770
2. Chowdhury, N., Aung, J., Shern, C. J., Jaffe, O., Sherburn, D., Starace, G., et al. (2024, August 13, updated February 24, 2025). Introducing SWE-bench Verified. OpenAI. openai.com/index/introducing-swe-bench-verified
3. Deng, X., Da, J., et al. (2025). SWE-Bench Pro: Can AI agents solve long-horizon software engineering tasks? arXiv:2509.16941 (v2, November 14, 2025). arxiv.org/abs/2509.16941
4. Scale AI. (2026). SWE-Bench Pro (public) leaderboard (accessed October 18, 2026). labs.scale.com/leaderboard/swe_bench_pro_public
5. BenchLM. (2026). SWE-bench Verified (data verified October 17, 2026, accessed October 18, 2026). benchlm.ai/benchmarks/swe-bench-verified
6. METR. (2026, January 29). Time Horizon 1.1. metr.org/blog/2026-1-29-time-horizon-1-1
7. Peng, S., Kalliamvakou, E., Cihon, P., & Demirer, M. (2023). The impact of AI on developer productivity: Evidence from GitHub Copilot. arXiv:2302.06590. arxiv.org/abs/2302.06590
8. Cui, Z., Demirer, M., Jaffe, S., Musolff, L., Peng, S., & Salz, T. (2025). The effects of generative AI on high-skilled work: Evidence from three field experiments with software developers. SSRN working paper 4945566 (revised August 21, 2025). papers.ssrn.com/sol3/papers.cfm?abstract_id=4945566
9. Becker, J., Rush, N., Barnes, E., & Rein, D. (2025). Measuring the impact of early-2025 AI on experienced open-source developer productivity. arXiv:2507.09089. arxiv.org/abs/2507.09089
10. Becker, J., Rush, N., Cunningham, T., Rein, D., & Mahamud, K. (2026, February 24). We are changing our developer productivity experiment design. METR. metr.org/blog/2026-02-24-uplift-update
11. Becker, J. (2026, May 11). Measuring the self-reported impact of early-2026 AI on technical worker productivity. METR. metr.org/blog/2026-05-11-ai-usage-survey
12. Stack Overflow. (2026, October 6). 2026 Developer Survey: AI. survey.stackoverflow.co/2026/ai, press release: stackoverflow.co/company/press/archive/stack-overflow-annual-developer-survey-ai-trust-conditional
13. Veracode. (2026, August). 2026 GenAI Code Security Report. veracode.com/resources/analyst-reports/2026-genai-code-security-report
14. Spracklen, J., Wijewickrama, R., Sakib, A. H. M. N., Maiti, A., Viswanath, B., & Jadliwala, M. (2025). We have a package for you! A comprehensive analysis of package hallucinations by code generating LLMs. 34th USENIX Security Symposium. arXiv:2406.10279. arxiv.org/abs/2406.10279
15. Brynjolfsson, E., Chandar, B., & Chen, R. (2025, revised August 2026). Canaries in the coal mine? Six facts about the recent employment effects of artificial intelligence. Stanford Digital Economy Lab. digitaleconomy.stanford.edu/publications/canaries-in-the-coal-mine