Skip to content

Measurement

Most AI-adoption programs fail their measurement test before they fail anything else. The metrics that vendors push (suggestion-acceptance rate, lines of code generated, "active AI hours") almost never correlate with the outcomes leadership actually cares about: delivery throughput, delivery stability, customer-facing quality, and engineer satisfaction. Measuring the wrong thing is worse than measuring nothing, because it manufactures false confidence in the program.

Google Cloud / DORA's 2024 Accelerate State of DevOps survey, the largest annual sample of professional engineering teams, found that for every 25% increase in AI tool use, software delivery throughput dropped 1.5% and delivery stability dropped 7.2% at the team level [primary]1. That is the headline finding this section is built around. A program can show every "AI is working" indicator a vendor dashboard can produce while the underlying delivery system gets slower and less stable. The only defense is to measure what the company actually ships, not what the assistant generates.

Where this section sits

This section is the concept and method layer. The Metrics baselining worksheet is the meeting; the Pilot retrospective worksheet is the readout. Read those for cadence and facilitation; read this section for the underlying frameworks.

What's in this section

  • Metrics taxonomy DORA, SPACE, and DevEx: what each one measures, what each one misses, and how to combine them for AI-specific impact.

  • Baselining Why pre-AI numbers are non-negotiable, what 90 days of pre-rollout data should contain, and the "control team" pattern.

  • Dashboards & reporting Audience-tiered dashboards (engineer, team, leadership, board) and the rule that every leadership view must show a cost alongside a gain.

  • Anti-patterns The vanity metrics (acceptance rate, lines generated, self-reported productivity) that look like signal and aren't.

The four-step measurement workflow

Every adoption program in this framework follows the same sequence. Skipping a step is what produces the "feels faster, measures worse" gap that DORA quantified.

graph LR
    B[1. Baseline before AI] --> I[2. Instrument SPACE / DORA / DevEx];
    I --> R[3. Report monthly: gain AND cost];
    R --> V[4. Revise after the first quarter];
    V --> R;

1. Baseline before AI

Capture 90 days of pre-rollout data on the metrics you intend to track before a single seat is provisioned. Without a baseline, every later movement is attributable to AI even when it isn't, and every later disappointment is attributable to "we just need more training" even when the data says stop. See Baselining.

2. Instrument SPACE / DORA / DevEx

Pick metrics from all three frameworks rather than any one in isolation. DORA tells you about delivery; SPACE tells you about the human system around delivery; DevEx tells you about cognitive load and flow, the dimensions AI changes most directly. Size your set to the organization: small picks 3–4 metrics, mid picks 6–8, enterprise picks 10–12. See Metrics taxonomy.

3. Report monthly: gain and cost

Every leadership-facing report on AI must show at least one cost metric next to every gain metric. If the report shows only gains, it will overstate the program. If it shows only costs, the program will be cancelled before it has had a chance to mature. Honesty is a governance control. See Dashboards & reporting.

4. Revise after the first quarter

The first quarter's metrics will be wrong in at least one way: a metric that was easy to instrument turns out to be a poor proxy, a metric that's hard to instrument turns out to be the one that matters, or seasonality contaminates the baseline. Revise the metric set at the end of the first quarter, and again every quarter, using the Pilot retrospective and the framework review cadence from Phase 2: Scale.

How measurement plugs into the roadmap

Phase What measurement looks like
Phase 0: Readiness Run the baselining worksheet. Choose the metric set. Document gaps with a 90-day plan per gap.
Phase 1: Pilot Track weekly, not monthly. Survey at start and end with identical questions. Log every incident; the log is more valuable than any chart.
Phase 2: Scale Monthly KPI dashboard goes live. Audience-tiered views are built. Quarterly framework review begins.
Phase 3: Optimize Retire metrics that haven't moved leadership decisions. Add metrics that incidents revealed mattered. Re-baseline annually.

What good measurement looks like

A measurement program in good health has all of these properties:

  • Both directions on every dashboard. Throughput and change-failure rate. Cycle time and review time. Acceptance and regression rate on AI-touched files.
  • Same questions before and after. The pilot-end survey uses the identical instrument the pilot-start survey did. Otherwise the delta is unmeasurable.
  • A cost line on the leadership view. If the executive dashboard shows only positive movement, it is a marketing artifact, not a governance one.
  • An owner per metric. Every metric on every dashboard names the human responsible for its definition, source, and confidence rating. Orphan metrics drift out of meaning within two quarters.
  • A review date in the frontmatter. This page has one; every metric definition should too.

What bad measurement looks like

  • The program reports a single headline number (usually acceptance rate or "engineers using AI weekly") and nothing else.
  • No control. Everyone uses the tool from day one, so nothing is comparable.
  • Self-reported productivity is the only outcome metric. METR's 2025 field study found developers believed they were 24% faster while measuring 19% slower. Self-report is the least reliable signal in the literature [primary]2.
  • Comparison is between individuals or between teams, not between time periods of the same team. Cross-individual comparison invites performance-management misuse and breaks the dashboard for everyone.

See Anti-patterns for the full catalog.

Cross-references

Footnotes & sources

  1. Google Cloud / DORA, "2024 Accelerate State of DevOps Report", Oct 2024. DORA is published by Google Cloud; the survey instrument is independently administered. ↩

  2. METR, "Measuring the Impact of Early-2025 AI on Experienced Open-Source Developer Productivity", Jul 2025. ↩