How to Measure AI Developer Productivity Without Ignoring Quality

TL;DR: To measure AI developer productivity honestly, pair throughput metrics such as PR throughput and lead time with quality metrics such as change failure rate, rework rate, escaped defects and time to restore, then compare AI-assisted work against a baseline. Self-reported speed is not enough: in the 2025 DORA research, more than 80% of respondents believed AI increased their productivity, yet AI adoption still had a negative relationship with software delivery stability.

We talk to engineering leaders every week who have rolled out AI coding assistants and now face a simple question from the CFO or the board: is it working? Most can show adoption charts. Very few can show that the team ships more working software than it did before. This guide lays out the method we recommend: a quality-adjusted view of velocity that you can set up in about 90 days with data you mostly already have.

Quick answers

How do you measure AI developer productivity?
Track team-level throughput and quality together, and split every metric by whether a change was AI-assisted. Compare against a pre-rollout baseline or a control group. A speed gain only counts if failure, rework and escaped defect rates hold steady.

Is AI suggestion acceptance rate a good productivity metric?
No. Acceptance rate tells you how often developers keep a suggestion, not whether the resulting change shipped safely or stayed shipped. Use it as an adoption signal only, never as an outcome.

How long does it take to get a reliable read?
Plan on roughly 90 days: a few weeks to instrument and baseline, then at least two full months of comparison data. Quality signals such as escaped defects lag merges by weeks, so earlier reads tend to flatter speed.

Why can't most leaders prove AI is making engineers faster?

Most leaders cannot prove it because they measure how fast developers feel, not how much working software reaches users. Perception and measured speed can point in opposite directions, and activity metrics rise even when outcomes do not.

The clearest evidence of the perception gap comes from a randomized trial by METR. In that study, 16 experienced open-source developers worked through 246 real issues, with AI tools randomly allowed or disallowed per issue. Developers expected AI to speed them up by 24%. With AI allowed, they actually took 19% longer. Even after the study, they still believed AI had sped them up by 20%. METR has since flagged these early 2025 results as out of date, and newer tools may well perform better. The lesson still holds: self-reports are a weak instrument.

Survey data shows the same pattern at scale. In the 2025 DORA research cited above, the team surveyed nearly 5,000 technology professionals and found about 90% use AI at work and more than 80% think it made them more productive. Yet the same research found that AI adoption continues to have a negative relationship with delivery stability. People are not wrong to feel faster. Typing code is faster. But a change that fails in production, gets reverted, and gets rewritten is not productivity.

Three gaps usually explain why the proof is missing:

• No baseline. Teams rolled out tools before capturing pre-rollout lead time, failure rate or rework, so there is nothing to compare against.
• No attribution. Nobody tagged which pull requests were AI-assisted, so AI and non-AI work blur together in every dashboard.
• Speed-only dashboards. Leaders report PR counts and suggestion acceptance, while quality data sits in a separate incident tool that nobody joins back to the change.

What is quality-adjusted velocity?

Quality-adjusted velocity is throughput counted only for changes that stayed shipped. It credits a team for merged, deployed work and deducts work that failed, needed rework, or leaked defects to users. It is our working definition, built on standard DORA measures, not an industry standard.

The building blocks already exist. DORA splits its five software delivery metrics into throughput metrics and instability metrics. Throughput covers change lead time, deployment frequency and failed deployment recovery time. Instability covers change fail rate and deployment rework rate, which counts unplanned deployments caused by production incidents. If you are new to these, our guide to DORA metrics walks through each one and how to collect it.

The step most teams skip is putting the two halves into one number per cohort. A simple version we use:

• Clean changes per engineer per week = merged changes per engineer per week x (1 minus change failure rate minus rework rate).
• Rework cost = failed or reworked changes x average engineer-hours to fix each one.
• Guardrails reported beside it: escaped defects per release and median time to restore service.

This is deliberately simple, and failure and rework can overlap, so treat it as a directional measure rather than an accounting identity. The point is that raw PR throughput and quality-adjusted throughput can tell very different stories, and you want both on the same slide.

Want to see quality-adjusted metrics on your own application rather than a slide? Our 12-week ContextQA pilot program ends with a before-and-after efficiency report on coverage, regression time and effort.

Which metrics should you track for AI-assisted changes?

Track four throughput metrics and four quality metrics, all at team level and all split by AI-assisted versus not. Then explicitly retire the vanity metrics that vendors and dashboards love to show.

Throughput metrics

• PR throughput: merged pull requests per engineer per week. Count merges, not opens, since AI makes opening PRs cheap.
• Change lead time: commit to production. This catches the common case where coding speeds up but review and testing queues grow.
• Review time: time from PR open to approval. AI-written diffs are often larger, and reviewers pay for that.
• Deployment frequency: a sanity check that faster merges reach users.

Quality metrics

• Change failure rate: share of deployments that need immediate intervention such as a rollback or hotfix.
• Rework rate: share of changes modified or reverted within a set window, such as 21 days, plus unplanned deployments caused by incidents.
• Escaped defects: bugs found in production per release, traced back to the change that introduced them. Our breakdown of the cost of defects shows why each escaped bug costs far more than one caught in CI.
• Time to restore: median time from incident start to recovery. If AI-assisted code is harder for the on-call engineer to understand, this number tells you.

Metrics to stop reporting as outcomes

• Lines of code: AI inflates it for free. More lines usually means more to review and maintain.
• Suggestion acceptance rate: an adoption signal. A developer can accept a suggestion and then rewrite it entirely.
• Self-reported time saved: useful for sentiment, unreliable for measurement, as the METR result shows.
• Number of PRs opened: easy to game and blind to whether anything shipped.

A useful related signal is how developers feel about the output. In the same DORA cycle, InfoQ's summary noted that 59% of respondents observed positive impacts on code quality, and the researchers found that a fail fast, fix fast approach did not offset the instability that came with AI speed. Perceived quality and measured stability are worth tracking side by side for exactly that reason.

How do you instrument AI-assisted work?

Instrumentation means tagging each change as AI-assisted or not at the source, then joining that tag to deployment, incident and test data. Without the tag, every comparison in this guide collapses into guesswork.

Here is the minimum setup we recommend:

1. Tag at the PR level. Add a required PR template checkbox or label: none, light assist, or substantially AI-generated. Some teams use a commit trailer instead. Pick one and enforce it in CI.
2. Link deploys to PRs. Your deployment tool should record which merged changes went out in each release, so a failed deploy can be traced to its changes.
3. Link incidents and bugs to changes. Require a suspected-change field on every incident and production bug ticket. This is what powers change failure rate and escaped defect attribution by cohort.
4. Track revert and follow-up edits. A short script over git history can flag files changed again within 21 days of a merge.
5. Classify test failures. Separate real regressions from flaky tests and environment noise, or AI-related quality signals get buried. Our flaky test playbook covers how to quarantine and fix noisy tests.

Also review the AI-generated changes themselves for the risks that metrics miss, such as insecure dependencies or invented APIs. Our AI-generated code checklist is a good starting point for reviewers.

How do you baseline and compare AI-assisted work?

Capture at least six to eight weeks of pre-rollout data, or keep a comparison cohort without AI tools for the same period. Then compare the change in each metric for the AI-assisted group against the change in the comparison group, which removes seasonal effects such as release freezes or holidays.

Controlled studies show why the comparison design matters. A set of three field experiments at Microsoft, Accenture and a Fortune 100 company found that developers with AI access completed 26% more tasks on average, with recent hires and junior developers gaining 27% to 39% and senior developers 8% to 13%. The researchers could not access the code produced, so they could not evaluate its quality. That is the gap a quality-adjusted measure closes inside your own organization.

Practical rules for a fair comparison:

• Compare like with like. Split by seniority and by work type, such as new features, bug fixes and migrations. The field experiments above show that gains differ sharply by experience.
• Use medians, not means. One giant migration PR can swing an average for a month.
• Wait for quality data to mature. Escaped defects and rework show up weeks after merge. Do not close a period until it is at least 30 days old.
• Keep the definitions frozen. Changing what counts as a failure mid-study makes the trend meaningless.

Worked example: a hypothetical 40-engineer team

Consider a hypothetical team of 40 engineers. These numbers are illustrative, not from a real customer. Leadership gives AI coding tools to one cohort of 20 engineers and keeps 20 as a comparison group for eight weeks, after an eight-week baseline.

During the baseline, the AI cohort merges 3.0 PRs per engineer per week, or 60 PRs a week. Change failure rate is 12% and rework rate is 9%.

After eight weeks with AI tools, the picture looks like this:

• Raw throughput: 3.9 PRs per engineer per week, or 78 a week. That is a 30% increase, and it is the number the vendor dashboard shows.
• Change failure rate: up from 12% to 17%.
• Rework rate: up from 9% to 15%.
• Comparison cohort: flat at about 3.1 PRs per engineer per week and a 12% failure rate, so the change is not seasonal.

Now apply quality-adjusted velocity. Baseline clean changes were 60 x (1 minus 0.21), or about 47 a week. After the rollout they are 78 x (1 minus 0.32), or about 53 a week. The real gain is about 12%, not 30%.

Then price the rework. If each failed or reworked change takes six engineer-hours to fix, the cohort spent about 76 hours a week on fixes before the rollout (12.6 changes) and about 150 hours after it (25 changes). That extra 74 hours is almost two engineers' worth of capacity spent cleaning up. The tool is still helping, but the bigger win is now in quality controls, not more AI seats.

This is the conversation we want leaders to have. A 12% clean gain with a clear plan to cut rework is a credible story. A 30% gain that falls apart at the next incident review is not.

What pitfalls distort AI productivity numbers?

The most common pitfalls are trusting perception, rewarding activity, and measuring individuals instead of teams. Each one makes AI look better or worse than it is, and each one is avoidable.

• Perception as evidence. Surveys are useful for adoption and sentiment. They are not a substitute for timed or system-measured outcomes.
• Goodhart effects. Once PR count becomes a target, PRs get smaller and more numerous. Report it, but never set a target on it.
• Ignoring the review bottleneck. Faster coding can move the queue to code review and QA. Watch review time and lead time together.
• Measuring too early. The first weeks after rollout mix learning curves with novelty. Exclude them or report them separately.
• Individual leaderboards. Ranking engineers by AI usage drives gaming and hides the team-level system effects you care about.
• Blaming the tool for test noise. If your suite is flaky, failure signals are unreliable for every cohort. Fix the noise first.

What should a 90-day measurement plan include?

A 90-day plan should spend the first month on instrumentation and baseline, the second on clean comparison, and the third on a decision. Here is the schedule we suggest:

1. Days 1 to 15: instrument. Add the PR tag, link deploys and incidents to changes, set the rework window, and agree on metric definitions in writing.
2. Days 16 to 30: baseline. Pull historical data where you have it. If you do not, run a short baseline with tagging switched on before expanding AI access.
3. Days 31 to 60: compare. Run the AI cohort and the comparison cohort side by side. Review the dashboard every two weeks, but do not make decisions yet.
4. Days 61 to 75: let quality mature. Keep collecting escaped defects and rework for changes merged in the comparison window.
5. Days 76 to 90: decide. Calculate quality-adjusted velocity and rework cost per cohort. Decide whether to expand, adjust guardrails, or invest in testing first.

Use this scorecard at the day 90 review:

• Green: clean changes per engineer up, change failure rate and escaped defects flat or down, time to restore flat.
• Yellow: clean changes up, but one quality metric worsened by more than a few points. Expand with added test and review guardrails.
• Red: raw throughput up but clean changes flat or down. Fix quality controls before expanding access.

If the case for expansion depends on testing investment, our guide to the ROI of test automation shows how to frame that spend alongside the productivity results.

How does ContextQA help you measure quality alongside speed?

ContextQA gives you the quality half of the equation: automated test coverage across web, mobile and API, plus diagnostics that tell you why a test failed. That makes change failure and escaped defect signals more trustworthy when you compare AI-assisted and non-AI work.

• Root cause analysis: our root cause analysis groups related failures across builds and classifies them by severity and scope, with evidence from screenshots, DOM, network and code. That helps you separate real regressions from noise before they reach your metrics.
• Self-healing tests: broken locators are repaired during the run, and every repair is logged with before-and-after context. Fewer false failures means a cleaner failure rate.
• CI/CD integration: diagnostics flow into pipelines such as Jenkins, GitHub Actions and GitLab, so test results land next to the deploy data you already collect.
• Measured pilots: the pilot program closes with a before-and-after efficiency report, which fits the baseline-and-compare method above.

We will be direct about the limit: ContextQA does not tag AI-assisted pull requests or compute DORA metrics for you. It makes the quality signals that feed those metrics more reliable, which is usually the weakest link.

Ready to put quality data next to your AI productivity numbers? Book a ContextQA demo and we will walk through your current test and failure signals.

Bottom line

AI coding tools can make teams faster, but feeling faster is not proof. Measure throughput and quality together, split by AI-assisted work, and judge the rollout on clean changes rather than raw output.

Give it 90 days, a baseline and a comparison group. If quality-adjusted velocity rises, expand with confidence. If only raw throughput rises, invest in testing and review before buying more seats.

Sources

1. Announcing the 2025 DORA Report, Google Cloud Blog, https://cloud.google.com/blog/products/ai-machine-learning/announcing-the-2025-dora-report. Cited for AI adoption (90%), perceived productivity gains (more than 80%), and the negative relationship between AI adoption and delivery stability.
2. Measuring the Impact of Early-2025 AI on Experienced Open-Source Developer Productivity, METR, https://metr.org/blog/2025-07-10-early-2025-ai-experienced-os-dev-study/. Cited for the 19% slowdown, the 24% expected speedup and the 20% believed speedup among 16 developers on 246 issues.
3. DORA's software delivery performance metrics, DORA, https://dora.dev/guides/dora-metrics/. Cited for the five DORA metrics and their split into throughput and instability.
4. DORA 2025 State of AI-assisted Software Development coverage, InfoQ, https://www.infoq.com/news/2025/09/dora-state-of-ai-in-dev-2025. Cited for 59% of respondents perceiving positive code quality impact and the fail fast, fix fast finding.
5. How generative AI affects highly skilled workers, MIT Sloan, https://mitsloan.mit.edu/ideas-made-to-matter/how-generative-ai-affects-highly-skilled-workers. Cited for the 26% increase in completed tasks, gains by seniority, and the lack of code quality measurement.