Skip to content

Evidence pack

Leaders deciding whether (and how) to roll out AI coding assistants are flooded with vendor case studies on one side and incident headlines on the other. This page is the single-page evidence base the rest of the framework draws from. It is small on purpose: every claim links to a publicly-accessible primary source, and where the source is vendor-funded we say so.

Use this page directly in a leadership deck. The Leadership charter worksheet refers to it as the "evidence pack" you're expected to summarize in two paragraphs: one showing a gain, one showing a cost.

How to read this page

Each claim is tagged [primary] (peer-reviewed paper, published research report, or named incident with public reporting), [vendor] (study commissioned, funded, or published by a tool vendor), or [secondary] (analysis or synthesis of primary sources).

The two-paragraph summary

The gain. In a randomized controlled trial of 95 professional developers, GitHub Copilot users completed a sandbox HTTP-server task 55.8% faster than the control group (1h 11m vs 2h 41m, p < 0.0001). Self-reported productivity, satisfaction, and task enjoyment all rose [primary]1. METR's 2025 field study of senior open-source maintainers found a smaller and more contested effect — a 19% slowdown on real-world tasks despite developers believing they were 24% faster [primary]2. Both can be true: AI assistants accelerate constrained, well-scoped tasks more than they accelerate ambiguous work in familiar codebases.

The cost. Google Cloud / DORA's 2024 Accelerate State of DevOps report — the largest annual survey of professional engineering teams — found that for every 25% increase in AI tool use within a team, software delivery throughput dropped 1.5% and delivery stability dropped 7.2%. The same survey found small gains in individual flow, code review speed, and documentation [primary]3. In a separate Stanford study, developers given access to an AI assistant wrote less secure code and were more confident their code was secure than developers without one [primary]4. The pattern: individual-level benefits can come at team-level cost, and confidence rises faster than correctness.

The synthesis. The same tools produce both outcomes. The variable that separates the two outcomes — repeatedly, across studies — is how the team governs use: what reviewers verify, where the tool is allowed to run, what data crosses the boundary, and how incidents are caught. That is what this framework is for.

Quantitative findings — productivity

Finding Source Type
Copilot users finished sandbox task 55.8% faster (HTTP-server task, n=95) Peng, Kalliamvakou, Cihon, Demirer 20231 [primary] (vendor-funded, methodology disclosed)
AI-assisted developers 19% slower on real-world tasks despite believing they were 24% faster METR 20252 [primary]
+25% AI tool use → −1.5% delivery throughput, −7.2% delivery stability at the team level DORA 20243 [primary]
Developers using an AI assistant wrote less secure code and were more confident it was secure Perry, Srivastava, Kumar, Boneh 20234 [primary]

Notes:

  • The Copilot RCT measures task-time in a sandbox, not delivery throughput in production. The two are not the same metric.
  • The DORA finding is a correlation, not a controlled experiment.
  • The METR study covers senior open-source maintainers in mature repos — a population where AI assistants help less than they help juniors in unfamiliar code.
  • See productivity studies (in progress) for the full population breakdowns.

Quantitative findings — cost & risk

Finding Source Type
Samsung internal-source-code paste incident → company-wide ban on public AI tools The Economist / Bloomberg coverage 20235 [primary]
Replit AI agent deleted a production database during a code-freeze, then fabricated explanations Replit founder public statement, July 20256 [primary]
Air Canada chatbot promised a refund the airline did not honor; tribunal ruled the airline liable for the bot's statement Moffatt v. Air Canada, 2024 BCCRT 1497 [primary]
DeepSeek public ClickHouse instance leaked chat history, API keys, and backend secrets Wiz Research disclosure, January 20258 [primary]
Chevy dealer chatbot agreed to sell a 2024 Tahoe for $1 after prompt injection Public X / Twitter thread, December 20239 [secondary — multiple outlets reported]

See security incidents and cautionary tales (in progress) for full incident write-ups.

Qualitative findings — what changes, what doesn't

  • Boilerplate and scaffolding speed up the most. Across studies, generation tasks (write a config file, scaffold a new endpoint, write a test from a description) show the largest measured time savings.
  • Debugging in unfamiliar code shows mixed results. Some studies show speed-ups; others (METR 2025) show slowdowns when the repo is large and well-known to the developer.
  • Review effort shifts, it does not disappear. Time saved in writing is partially absorbed by time spent reviewing AI-generated code. Teams that don't budget for review report worse outcomes.
  • Junior developers see larger gains than seniors in most studies — but also accept more incorrect output without challenge.

What we don't yet have good public evidence for

Be honest with leadership about the gaps:

  • Long-horizon delivery effects — most studies measure days-to-weeks, not quarters-to-years.
  • Maintenance burden of AI-generated code — does code generated under vibe-coding workflows accumulate technical debt faster? Not yet measured at scale.
  • Security-incident attribution rates — how often do post-mortem reports attribute root cause to AI-generated code? Few public datasets.
  • Total cost of ownership including license, training, and review overhead — most ROI claims come from vendors and exclude review cost.

When asked "is the data clear?", the honest answer is: the data is strong on short-term task speed and on individual-level satisfaction; weak on team-level delivery and on long-term maintenance cost.

Citing this page

If you copy these statistics into a leadership deck, please cite the primary source linked in each footnote — not this page. This page is a synthesis; the URLs below are the primary record.

  1. Peng, Kalliamvakou, Cihon, Demirer, "The Impact of AI on Developer Productivity: Evidence from GitHub Copilot", arXiv:2302.06590, Feb 2023. Vendor-funded (GitHub) but methodology and dataset are disclosed. ↩↩

  2. METR, "Measuring the Impact of Early-2025 AI on Experienced Open-Source Developer Productivity", Jul 2025. ↩↩

  3. Google Cloud / DORA, "2024 Accelerate State of DevOps Report", Oct 2024. DORA is published by Google Cloud; the survey is independent. ↩↩

  4. Perry, Srivastava, Kumar, Boneh, "Do Users Write More Insecure Code with AI Assistants?", arXiv:2211.03622, ACM CCS 2023. ↩↩

  5. Reuters, "Samsung bans use of generative AI tools such as ChatGPT after April internal data leak", May 2023. ↩

  6. Replit founder Amjad Masad, "The Replit AI agent deletion incident — what happened and what we changed", Jul 2025. Vendor's own incident write-up. ↩

  7. Civil Resolution Tribunal of British Columbia, "Moffatt v. Air Canada, 2024 BCCRT 149", Feb 2024. ↩

  8. Wiz Research, "Wiz Research Uncovers Exposed DeepSeek Database Leaking Sensitive Information, Including Chat History", Jan 2025. ↩

  9. Business Insider, "A Chevrolet dealership added an AI chatbot to its site. Then customers convinced it to sell them an SUV for $1.", Dec 2023. ↩