Skip to content

Measurement anti-patterns

Most AI-adoption programs are evaluated against the wrong metrics, not because nobody is paying attention, but because the wrong metrics are easier to produce, easier to explain, and produced for free by vendor dashboards. This page catalogs the traps. Each entry follows the same structure:

  • Why it's tempting. The reason it ends up on dashboards.
  • Why it misleads. The mechanism by which it produces false signal.
  • What to use instead. The metric or pair of metrics that actually answers the question the bad metric was trying to answer.

The catalog is referenced from the metrics taxonomy, the dashboards page, and the metrics baselining worksheet so the same anti-patterns can be cited consistently in training, governance, and retrospectives.

1. Suggestion acceptance rate

Why it's tempting. It comes out of the vendor dashboard for free. It is a single number. It moves predictably with rollout. It looks like adoption. It looks like productivity.

Why it misleads. A high suggestion-acceptance rate can mean two opposite things: developers are accepting good suggestions because the assistant is well-tuned to the codebase, or developers are accepting suggestions without scrutinizing them because review feels optional. The two cases produce identical acceptance numbers and opposite outcomes. Stanford's 2023 study found that developers given an AI assistant wrote less secure code and were more confident it was secure than the control group [primary]1. High acceptance rates can disguise that confidence gap.

What to use instead. Treat acceptance rate as a diagnostic paired with a quality counterweight, never as a primary metric. - Pair with regression rate on AI-touched files. - Pair with review-bounce rate on AI-touched PRs. - Pair with revert-within-14-days rate. A high acceptance rate is good news only if the counterweights are flat or improving. If the counterweights are degrading, high acceptance rate is the cause of the problem, not evidence of success.

2. Lines of code generated

Why it's tempting. Big numbers. Easy to graph. Easy to put in a press release. Easy to use in a vendor case study.

Why it misleads. Lines of code is the worst proxy in software engineering, a fact established long before AI assistants existed. More code is not better software; it is usually worse software, because the maintenance cost scales with the line count. AI assistants generate verbose code by default unless prompted not to, so a "lines generated" metric rewards verbosity. GitClear's 2023 code-churn study measured a near-doubling of code reverted within two weeks coinciding with AI-assistant adoption [primary]2; "lines generated" would not have surfaced that pattern at all.

What to use instead. - Cycle time: measures whether the work shipped, not whether the buffer filled. - Churn rate on AI-touched code: measures whether the lines that shipped are the lines that stayed. - Code-comprehension self-report. Does the team understand the code they're now responsible for? A quarterly question on the DevEx pulse.

3. Active hours / "AI usage time"

Why it's tempting. Vendor dashboards report it. It feels like "engagement." It correlates loosely with adoption.

Why it misleads. It rewards seat-time, not outcomes. An engineer who spends three hours per day in a chat panel is not necessarily producing better software than one who spends thirty minutes. Worse, "active hours" can be inflated by background sessions, re-prompting cycles, and the kind of fact-checking work that actually indicates the assistant is not helping.

What to use instead. - Cycle time: outcome metric, not activity metric. - Deep-work hours: uninterrupted blocks ≥90 minutes. This measures the opposite of "always in the chat panel," which is usually closer to what good engineering looks like. - PR throughput at constant quality: paired with change failure rate to ensure the throughput isn't coming at stability cost.

4. Percentage of PRs touching AI-generated code

Why it's tempting. Sounds like a meaningful adoption metric. Easy to instrument with a commit-message convention or PR template.

Why it misleads. Within twelve to eighteen months of broad rollout, everything is AI-touched. The metric drifts toward 100% and stops carrying any signal. By the time it stops moving, the program will have lost a leading indicator and gained nothing. Worse, this metric incentivizes tagging behavior. Engineers may mark PRs as AI-touched to demonstrate adoption, regardless of how much AI they actually used.

What to use instead. - Cycle time and quality on the whole PR stream. Once everything is AI-touched, the question is "is the stream getting faster and more stable?", which is the DORA question all along. - Comparison against the historical baseline. Keep the pre-rollout numbers visible as the dashboard's reference line so movement can be detected without needing an "AI-touched" tag at all.

5. Self-reported productivity gain

Why it's tempting. It is cheap to collect (one survey question), it tells a clear story, and it is almost always positive. Engineers genuinely feel faster with AI assistants.

Why it misleads. Self-report is the least reliable signal in the literature. METR's 2025 field study of senior open-source maintainers found that participants believed they were 24% faster with AI assistance and were measured to be 19% slower on the same tasks [primary]3. The gap between perception and measurement is wide and consistent.

Self-report has a place: it tells you about satisfaction, which is a real outcome that matters in its own right. It does not tell you about productivity, which is a different outcome.

What to use instead. - Behavior metrics first. Cycle time, throughput, change failure rate. - Self-report as the satisfaction layer, clearly labeled. - Pair them on the dashboard, never substitute one for the other. When the two diverge (satisfaction up, throughput down) the divergence is the most important finding the program will produce.

6. Vendor-supplied ROI claims

Why it's tempting. They are pre-computed. They are professionally designed. They are approximately the number leadership wants to see. They cite real-looking research.

Why it misleads. Vendor ROI calculations almost universally:

  • Exclude review cost. Time saved at write is not netted against time spent reviewing. The DORA 2024 finding suggests the net is negative at the team level [primary]4.
  • Compare against a strawman baseline, typically a "developer who must write everything from scratch with no IDE assistance," which has not been the actual baseline in any company since the early 2000s.
  • Use task-time speedups in sandbox studies as if they generalized to real-world delivery throughput. The Copilot RCT's 55.8% sandbox speedup [primary]5 is a real finding; it is also not a delivery-throughput finding.
  • Aggregate across populations that don't apply to the customer org. Junior developers in unfamiliar codebases see the largest gains; senior developers in familiar codebases often see losses.

What to use instead. - Compute ROI yourself, against your own baseline. The baselining work makes this possible. - Include all costs: license, training, review overhead, incident response, governance time. - Cite vendor numbers as [vendor] in any internal document so readers can weight them appropriately. See the evidence pack for the convention.

7. Single-team success extrapolated to the organization

Why it's tempting. The pilot succeeded. The pilot's metrics are pristine. Scaling those metrics by 50× looks like the program's expected outcome.

Why it misleads. Pilot teams are not representative. They self-select for AI enthusiasm; they tend to work on systems with better test coverage and clearer ownership; they often have a sponsor's attention that scaled rollouts cannot replicate. The Phase 1 pilot page warns explicitly against piloting only with the most enthusiastic team for this reason.

A program that extrapolates pilot results to the full organization without correction will overshoot, typically by 30–60%, and the overshoot is what produces the "the rollout disappointed" narrative two quarters later, even when the program was actually working at a realistic level.

What to use instead. - Run a control team during the pilot. See baselining: control team pattern. - Include a non-volunteer team in the pilot mix so the pilot itself is closer to representative. - Discount pilot results when projecting. A 25–40% discount on pilot productivity gains is a reasonable starting point when setting Phase 2 expectations. - Roll out in waves, per Phase 2: Scale, so the projection is calibrated against each wave's measured result, not against the pilot.

8. Adoption rate as a success metric

Why it's tempting. It is what a vendor will ask for. It is what a procurement team can defend. It is a single number that goes up.

Why it misleads. Adoption is an input, not an outcome. "60% of eligible engineers actively used the tool last week" tells you that you got value from the licenses you bought. It tells you nothing about whether the work the company ships is faster, stabler, or better. A program that rewards adoption rate will produce 100% adoption, and may produce a worse delivery system than it had before.

The Phase 2: Scale exit criteria require ≥60% active adoption, but explicitly also require the monthly KPI dashboard to be trending within ±20% of targets. Adoption is a necessary input, not the success condition.

What to use instead. - DORA + SPACE + DevEx outcomes, on the leadership dashboard. - Adoption as a denominator, not as a numerator. "Throughput per active engineer" beats "throughput" if the org is mid-rollout, but neither replaces the outcome metrics. - Charter-defined success criteria from the leadership charter. The charter should never list "high adoption" as a primary success criterion; it should list outcomes.

9. Tokenmaxxing leaderboards

Why it's tempting. When AI coding tools bill on consumption, usage data is readily available. Publishing who spends the most — or framing high consumption as evidence of "power use" — looks like a way to drive adoption and surface enthusiastic practitioners.

Why it misleads. Token consumption measures input to the model, not quality of output or impact on delivery. A developer who spends $800 re-prompting an agent to produce code that gets reverted contributes more to the leaderboard than one who spends $200 on precise, high-leverage prompts that land first time. Rewarding consumption rewards the wrong behavior. Uber's 2026 AI tooling budget was exhausted within four months before the company implemented per-tool monthly caps; usage patterns bearing no relationship to delivery outcomes drove the overshoot [secondary]6.

What to use instead. - Outcome metrics first. Cycle time, change failure rate, escape-defect rate from the metrics taxonomy. - Cost-per-outcome. Tooling spend divided by shipped features or defect escapes gives a unit-economics view that is harder to game. - Per-tool caps, not competition. A monthly per-tool spending cap contains cost without creating a race to consume. See vendor standards and enterprise adoption. - Qualitative practice review, not a consumption rank.

How to spot a vanity dashboard at a glance

A dashboard is leaning vanity when it has any of these properties:

  • Only one direction. Every metric trends up. Real systems have trade-offs; a real dashboard shows them.
  • No baseline reference. Numbers are reported without a "vs. the 90-day pre-AI baseline" comparison.
  • No cost line. No metric labeled as a cost or a counterweight. The leadership cardinal rule from dashboards is being violated.
  • Vendor logo on it. If the dashboard is rendered by the vendor's own tooling and the company has not augmented it with quality and cost metrics, it is the vendor's narrative, not the company's.
  • Acceptance rate is the headline. See anti-pattern #1: almost always a sign the rest of the metrics weren't selected with care.
  • Usage or spend leaderboard visible. A ranking of engineers by AI tool spend or token consumption is anti-pattern #9: it rewards model input, not outcome, and creates a race to consume.
  • No named owner per metric. Orphan metrics drift; orphan dashboards mislead.

If your dashboard is built on these, rebuild

A dashboard built on the metrics in this catalog is worse than no dashboard at all. It manufactures false confidence in a program whose actual performance is unknown.

Rebuild it from the metrics taxonomy up: pick from DORA, SPACE, and DevEx; pair every gain with a cost; baseline before measuring; serve audience-tiered views.

The metrics baselining worksheet is the meeting that resets the foundation. The pilot retrospective is the meeting that catches a vanity dashboard in the wild.

Cross-references

Footnotes & sources

  1. Perry, Srivastava, Kumar, Boneh, "Do Users Write More Insecure Code with AI Assistants?", arXiv:2211.03622, ACM CCS 2023. ↩

  2. GitClear, "Coding on Copilot: 2023 Data Suggests Downward Pressure on Code Quality", Jan 2024. ↩

  3. METR, "Measuring the Impact of Early-2025 AI on Experienced Open-Source Developer Productivity", Jul 2025. ↩

  4. Google Cloud / DORA, "2024 Accelerate State of DevOps Report", Oct 2024. ↩

  5. Peng, Kalliamvakou, Cihon, Demirer, "The Impact of AI on Developer Productivity: Evidence from GitHub Copilot", arXiv:2302.06590, Feb 2023. Vendor-funded; methodology disclosed. ↩

  6. Simon Willison, "Uber caps usage of AI coding tools at $1,500 per month", Jun 2026 [secondary]. ↩