Skip to content
AI Coding Tools: What the Evidence Says and How Mid-Market Owners and Operators Should Manage Them

AI Coding Tools: What the Evidence Says and How Mid-Market Owners and Operators Should Manage Them

2026-11-03 · By the Yellow Coop Team

The takeaway

Your developers, or your outsourced development partner, are almost certainly using AI coding tools already. The research says those tools can help, but they don’t automatically make software arrive faster or safer, and people are bad at judging their own speedup. So don’t manage AI coding by vibes. Measure delivery, keep changes small, and keep human review and testing in front of anything that ships.

What the evidence actually says

Developers feel faster than they measure

The research group METR ran a randomized trial with 16 experienced open-source developers working on 246 real tasks in projects they knew well. Each task was randomly assigned to allow or disallow AI tools. With AI allowed, tasks took 19% longer. Before starting, the developers predicted AI would cut their time by 24%; afterward, they still believed it had cut their time by 20% (METR, July 10, 2025; paper).

Newer data matters here. In February 2026, METR said those early-2025 results are out of date and that developers are likely sped up more now. Its follow-up study (57 developers, 800+ tasks) pointed toward speedups, but METR called that data only very weak evidence because many developers declined to work without AI, which skews the sample (METR, Feb. 24, 2026).

The durable lesson isn’t “AI slows developers down.” It’s that self-reported speedups are unreliable, so measure.

More code isn’t the same as better delivery

Google’s DORA research program surveys technology professionals every year. Its 2024 report estimated that a 25% increase in AI adoption was associated with a 1.5% drop in delivery throughput and a 7.2% drop in delivery stability. DORA’s hypothesis: AI makes it easy to produce more code, changes get bigger, and bigger changes are slower and more likely to break things (DORA 2024 report; Google Cloud, Oct. 22, 2024).

The 2025 report, based on nearly 5,000 respondents, found a better picture: AI adoption now had a positive relationship with throughput, but still a negative one with stability. 90% of respondents used AI at work, and 30% reported little or no trust in AI-generated code. DORA’s summary: AI “amplifies what’s already there,” so strong teams get stronger and struggling teams see their problems intensify (Google Cloud, Sept. 23, 2025).

Confidence can outrun security

Stanford researchers Neil Perry, Megha Srivastava, Deepak Kumar, and Dan Boneh ran a user study with 47 participants on five security-related programming tasks. Participants with an AI assistant wrote insecure solutions more often on four of the five tasks, and were more likely to believe their code was secure (Perry et al., ACM CCS 2023). The tools have improved since that study was first posted in November 2022, so read it as a warning about overconfidence, not a current defect rate.

Five controls owners should set

You don’t need to read code to set these. You need to ask for them.

1. Decide which tools are approved and what data they can see

Name the approved AI coding tools, the account type (business accounts, not personal ones), and whether your source code and customer data can be sent to them. Put it in your AI use policy, and make it apply to contractors too.

2. Keep changes small

DORA’s own explanation for the stability drop is batch size. Ask your team or vendor to ship in small, reviewable changes. A 3,000-line change generated in an afternoon is a review problem, not a productivity win.

3. Require human review and automated tests before anything ships

Every change gets reviewed by a person who understands it, and automated tests and security scans run before release. DORA’s 2025 guidance points to the same safety nets: strong automated testing, mature version control, and fast feedback loops (Google Cloud, Sept. 23, 2025).

4. Measure delivery, not vibes

Track a few numbers before and after rollout: how long a change takes to reach production, how often you deploy, how often a release causes a problem, and how fast you recover. These are DORA’s standard delivery measures, and they tell you whether AI is helping the business or just producing more code.

5. Put AI in your vendor conversations

If an outside firm builds your software, ask how they use AI, how they review its output, and who owns the result. Add it to the contract and the status report.

The short version

AI coding tools are worth using, but the gains show up where the basics are already solid. Approve the tools, keep changes small, review and test everything, measure delivery, and hold vendors to the same rules.

If you’re not sure whether your team or development partner is getting real value, or quietly piling up risk, Yellow Coop’s CTO work covers architecture review and outsourced partner oversight, and our CISO work includes an AI use policy that spells out which tools, which data, and who approves. Let’s talk.

Sources