Are your engineers really using AI for code, is it making delivery faster, and is quality holding up? A three-question method, with the data and queries to answer each one.
TL;DR: Three questions decide whether AI coding assistants are paying off: are we really using AI for code, is it improving delivery, and is quality holding up? Seat counts, acceptance rates and developer surveys can't answer them, and the research disagrees with developers' own perception: in METR's 2025 trial, experienced developers were 19% slower with AI while believing they were 20% faster. The reliable method is to measure depth of use (active days, share of added lines written by AI), then compare cohorts of habitual, occasional and non-users against their own pre-AI baseline on throughput, cycle time and time to first review, and finally watch quality signals such as bug-fix share, change requests and large pull requests approved without changes. Keypup's AI Usage dataset puts GitHub Copilot usage next to your pull requests, so all three answers come from one place, in a dashboard or by asking your AI assistant through the Keypup MCP Server.
Every engineering leader who rolled out an AI coding assistant eventually gets the same three questions from the CEO, the CFO or the board. Are we actually using it? Is it making us faster? And is it breaking anything?
They sound simple. They are not. The vendor console answers a different question (how many suggestions were offered), developer surveys answer a third one (how fast people feel), and the delivery dashboard rarely knows who used AI at all. The result is that most AI rollouts are judged on adoption and anecdotes, while the real effect on the software development lifecycle (SDLC) stays invisible.
This article is not about monitoring adoption: we covered that in AI Usage Monitoring: How to Measure Who Uses AI Coding Assistants. It is about reaching a verdict. We'll look at what the latest research says, why the obvious metrics mislead, and a three-question method that tells you whether AI is improving your delivery and your quality, with the exact queries to run.
AI coding assistants are now nearly universal. 90% of respondents to the 2025 DORA report use AI at work, and 84% of the 49,000 developers in Stack Overflow's 2025 survey use or plan to use AI tools. Yet the evidence on their effect points in different directions, and the people using them are not reliable judges of it.
| Source | Finding | What it means for you |
|---|---|---|
| METR randomized trial (2025) | 16 experienced open-source developers took 19% longer on 246 real issues with AI tools. They expected a 24% speed-up beforehand and still believed in a 20% speed-up afterwards. | Self-reported productivity is not evidence. Measure delivery data instead. |
| DORA report (2025) | AI adoption now has a positive relationship with delivery throughput, but still a negative relationship with delivery stability. | Speed and quality must be measured together, or speed gains hide stability losses. |
| Stack Overflow Developer Survey (2025) | 46% of developers distrust AI accuracy versus 33% who trust it; 66% cite solutions that are almost right, but not quite; 45% find debugging AI code more time-consuming. | Watch rework and review effort, not just output. |
| GitClear code quality research (2025) | Across 211 million changed lines (2020 to 2024), copy-pasted lines rose from 8.3% to 12.3% while lines associated with refactoring fell from 25% to under 10%. | AI can grow the codebase faster than it improves it. Track pull request size and rework. |
The METR authors are careful to say their study does not show that AI slows down most developers. But it shows something every leader should take seriously: how fast AI feels is a poor proxy for how fast work ships.
The DORA team summarises the whole picture in one sentence:
AI doesn't fix a team; it amplifies what's already there.
Strong teams use AI to become even better and more efficient. Struggling teams will find that AI only highlights and intensifies their existing problems.
If AI amplifies what is already there, the same tool can speed up one team and destabilise another. That is exactly why an organisation-wide average, or a vendor's global benchmark, can't tell you whether AI is working for you.
The method answers the three questions in order, because each one depends on the previous answer. You can't measure the effect of AI on delivery until you know who really uses it, and you can't call a speed gain a win until you know what it did to quality.
| Question | Misleading metric | Reliable metric | Keypup data |
|---|---|---|---|
| 1. Are we really using AI for code? | Licences assigned, acceptance rate | Active days per developer, share of added lines written by AI, usage by surface | AI Usage dataset, Commits |
| 2. Is it improving our SDLC? | Org-wide PR count, developer surveys | Throughput, cycle time and time to first review per usage cohort, against each cohort's own baseline | AI Usage dataset, Issues & Pull Requests |
| 3. Is quality holding up? | Test coverage alone, gut feeling | Bug-fix share, change requests, large PRs approved without changes, PR size | Issues & Pull Requests, Reviews |
The analyses below were produced by an AI assistant connected to the Keypup MCP Server, using GitHub Copilot usage from Keypup's AI Usage dataset and pull request data from GitHub. The figures come from a fictional sample organisation, DoeCorp, with 64 developers who adopted Copilot in spring 2026. The queries shown are the real formulas the MCP Server runs.
The first question sounds like the easiest, and it is the one most organisations get wrong. Usage is not binary. A developer who tried Copilot once in a month and a developer who uses agent mode every day both count as "active users" in a seat report, but only one of them can plausibly change your delivery numbers.
Two measures answer the question properly:
lines_added in the AI Usage dataset), compared with all lines added in pull request commits. Keypup keeps lines that were merely offered (lines_suggested_added) separate from lines that landed, so this share is not inflated by ignored suggestions.Splitting usage by surface (IDE completion, chat, agent mode, CLI, cloud agent, code review) then tells you how the code gets written, which matters for the next two questions: agent-written changes tend to be larger.
Prompt to your AI assistant, connected to the Keypup MCP Server
Are we really using GitHub Copilot? I don't want seat counts. For last month, show how many of the developers who merged code used Copilot at least once, how many used it on 8 days or more, and how many used agent mode or a cloud agent. Then show the share of lines added that were written by Copilot, month by month, since we rolled it out.
Analysis from Keypup AI
There is no universal benchmark worth copying, and acceptance rates in particular vary with how a team works. What matters is the trend in depth: the share of habitual users and the AI share of added lines should rise as enablement improves. If both stall while licences grow, the problem is enablement or fit, not the tool's potential.
This is the question behind every budget review, and the one where the METR result matters most: developers' own sense of speed is not evidence. You need delivery data, and you need to read it the right way.
Because each developer's AI usage sits in the same warehouse as their pull requests, you can split delivery metrics by usage cohort. For GitHub Copilot this works per developer: Copilot reports each user by their GitHub login, which is the same username that authors their pull requests.
Then compare each cohort with its own baseline, the same developers over the six months before the rollout. This controls for the fact that early adopters are often your strongest engineers to begin with: comparing habitual users against non-users directly would credit AI for a difference that already existed.
Four measures together show whether AI speeds up delivery or only moves the bottleneck:
Prompt to your AI assistant, connected to the Keypup MCP Server
Is AI actually making us faster? Split developers into habitual Copilot users (8 or more active days a month), occasional users (1 to 7 days) and non-users. For merged pull requests from April to September, show for each cohort the pull requests merged per developer per week, the average cycle time, the average time to first review and the average lines changed, each compared with the same developers from October to March. Add a row for the whole organisation.
Analysis from Keypup AI
This sample matches what DORA reports at industry level: AI now improves throughput, but the improvement shrinks on its way through the pipeline. Writing code is only one stage. If review, testing and release don't absorb the extra volume, the gain sits in a queue. We look at the review side in more detail in The AI Review Tax and in Measuring the Real Impact of AI Coding Assistants on Your Pull Request Cycle Time.
This is the question developers themselves raise most: two-thirds of Stack Overflow's 2025 respondents struggle with AI answers that are "almost right, but not quite," and DORA still finds a negative relationship between AI adoption and delivery stability. A throughput gain that comes with more rework is not a gain.
No single metric captures quality, so use a small set of signals that move early, again by cohort and against each cohort's baseline:
bug label, or the type of the Jira issue a pull request resolves). A rising share means more effort goes into rework.If your team tags production failures consistently, add change failure rate from Keypup's DORA metrics templates, which you can calculate from pull requests or commits using your own naming conventions.
Prompt to your AI assistant, connected to the Keypup MCP Server
Is AI hurting our code quality? For the same three Copilot cohorts and the same periods, compare the share of merged pull requests labelled bug, the share of pull requests where changes were requested, the share of pull requests over 500 lines that were approved without any change request, and the share of pull requests over 1,000 lines.
Analysis from Keypup AI
Once the three questions are answered, the summary for leadership writes itself. It should state a verdict per question, the evidence behind it, and what changes next quarter.
Prompt to your AI assistant, connected to the Keypup MCP Server
Summarize this for the leadership team: are we using AI for code, is it improving our SDLC, and is quality holding up? Give a one-word verdict per question with the supporting numbers, and recommend what we should change next quarter.
Analysis from Keypup AI
Your own verdict may be different. In some organisations the review pipeline absorbs AI-sized changes well and the answer to Question 2 is a clear yes; in others, adoption is shallow and Question 1 is the real problem. The value of the method is not the answer it gives DoeCorp. It is that every answer rests on your own delivery data.
Keypup's AI Usage dataset stores AI coding assistant activity as one row per day, broken down by developer, surface, model, client and language. GitHub Copilot is supported through the GitHub integration: enable the Copilot Metrics project in your list of GitHub projects, and Keypup imports up to one year of history on the first import. Claude Platform, OpenAI Platform and OpenAI Codex Enterprise are available in beta.
Because Copilot reports each user by their GitHub login, its usage joins directly with the pull requests those developers author. Other providers report display names or email addresses, so per-developer comparisons across providers need the identities to be matched.
The AI Usage & Monitoring dashboard template covers active developers, adoption by model and surface, suggestion acceptance, code contribution and pull request outcomes. Add delivery and quality insights next to it, or start from the DORA Metrics dashboard for throughput and change failure rate.
Connect Claude, ChatGPT, Cursor, VS Code or any MCP-compatible assistant to the Keypup MCP Server at https://hq.keypup.io/mcp, in one click with OAuth. The MCP tools are read-only: your assistant can query and analyse, never change your data. Then ask the questions above, refine them in conversation, and keep the cohort definitions consistent by writing them into your team's AI Instructions, which the MCP Server reads at the start of each conversation.
Re-run the three questions monthly with the same cohorts and baseline. The trend over a quarter tells you far more than any single month. For the broader framework, see how to measure the impact of AI on software development.
It depends on the team, and on what "faster" means. The 2025 DORA report links AI adoption with higher delivery throughput, while METR's 2025 trial found experienced developers 19% slower on their own repositories. The only reliable answer for your organisation comes from comparing usage cohorts against their own baseline on throughput, cycle time and time to first review.
Research points to risks rather than a universal answer: DORA finds AI adoption still linked to lower delivery stability, and GitClear observed more copy-pasted code and less refactoring since AI assistants spread. Measure it in your own data with bug-fix share, change requests and large pull requests approved without changes, compared by usage cohort.
Answer three questions in order: how deeply developers use AI (active days and AI share of added lines), whether delivery improves for heavy users against their own baseline, and whether quality signals hold steady. Licence cost only becomes a return when delivery improves without a rise in rework.
No. Acceptance rate only covers inline completions, not chat or agent mode, where an AI assistant can write large changes. A team can have a low acceptance rate while AI writes a large share of its code. Use it to understand how a surface is used, not to judge impact.
Expect a learning curve before usage habits change delivery numbers. Track the leading indicators (active days and AI share of added lines) from the first month, and judge delivery and quality on trends over at least a quarter, against a baseline of several months before the rollout.
Yes, but not directly. Developers who adopt AI early are often stronger engineers to begin with. Compare each cohort with its own pre-rollout baseline, then compare the size of the changes between cohorts.
Keypup's AI Usage dataset supports GitHub Copilot, with Claude Platform (including Claude Code), OpenAI Platform and OpenAI Codex Enterprise in beta, and Cursor coming next. All providers map onto the same dataset, so they can be compared side by side.
Yes. Connect your AI assistant to the Keypup MCP Server and ask the questions in plain English. The assistant reads your team's AI Instructions, turns the question into Keypup queries over your AI usage and pull request data, and explains the results.
"Are we using AI, is it making us faster, and is it breaking anything?" deserves a better answer than seat counts and developer sentiment. The research shows why: AI can speed up throughput while hurting stability, and developers can feel faster while being slower.
The three-question method gives you a verdict grounded in your own data: depth of use first, then delivery by cohort against a baseline, then quality signals that move early. With AI usage stored next to your pull requests, each answer is one query away, in a dashboard or in a conversation with your AI assistant. As AI writes a growing share of every codebase, the teams that measure its effect honestly will be the ones that get the gain without paying for it later.
Join teams already using AI to make data-driven decisions faster than ever.
Leadership asks why the roadmap is slipping. Engineering says "bugs and maintenance", but can't prove how much. Learn what engineering leaders on Reddit say about the keep-the-lights-on (KTLO) tax in large organizations, why investment allocation is so hard to measure, and how Keypup MCP uses Jira issue types, labels and linked pull requests to show where engineering capacity actually goes.
Compare the 5 best value stream analytics platforms for engineering leaders — Keypup, Pluralsight Flow, LinearB, Jellyfish and Swarmia — across DORA Metrics depth, cross-platform data unification, contextual analysis, customization, AI and natural language capabilities, and setup speed. Find out which value stream management platform turns Git, Jira and CI/CD data into prescriptive, actionable insights.
Ten-line pull requests get twenty comments. Two-thousand-line pull requests get "LGTM" in four minutes. Learn what engineering leaders on Reddit say about rubber-stamp code reviews in large organizations, why the problem stays invisible, and how Keypup MCP uses pull request size, review timestamps, approvals and comments to measure it repository by repository.