Skip to content

Baselining

This page is the concept and method for baselining. The companion Metrics baselining worksheet is the 90-minute meeting that produces the actual snapshot. Read it for agenda, prompts, and the document template. Read this page for why the baseline matters and how to design one that won't be invalidated by the time the pilot retrospective happens.

Why baselines matter

Without a baseline, every metric movement after AI rollout is attributable to AI, even when it isn't. Without a baseline, seasonality, hiring spikes, infrastructure changes, and incidents on unrelated systems all get rolled into the AI program's narrative. Without a baseline, the loudest voice in the retrospective wins.

The DORA 2024 finding is a baseline story. The report measured a −1.5% throughput / −7.2% stability correlation with AI tool use at the team level [primary]1. None of that signal would have been visible without years of historical comparison data. Companies that started measuring at the pilot have no equivalent baseline and will simply not detect the same pattern in their own data, even if it is happening.

If you're already mid-pilot without a baseline

You can still recover by reconstructing the baseline from historical sources (git history, CI logs, incident tracker, survey archives) and explicitly tagging the resulting numbers as reconstructed. A reconstructed baseline is weaker than a pre-registered one but is still preferable to no comparison. Confidence ratings on each metric become essential.

What to baseline

The baseline must contain at least 90 days of data on every metric the pilot will be judged against. The metrics come from Metrics taxonomy: a DORA selection, a SPACE selection, a DevEx selection, and the AI-specific overlays the program will track.

For each metric capture six attributes. Without all six the metric is not yet a baseline.

  1. Definition. A single sentence anyone in the room would agree on. "Lead time for changes = median minutes from git push on main to deploy completion in production for the service in scope."
  2. Source. The system the number comes from, with a query or dashboard link if one exists.
  3. Current value. The actual number, with units.
  4. Trend. Direction and magnitude over the prior 90 days. "14 per week, flat" is meaningfully different from "14 per week, declining 5% per month."
  5. Confidence. High / Medium / Low, with a one-line reason. "Medium: service X not instrumented before March."
  6. Known caveats. Anything that complicates interpretation: a re-org during the period, a deployment-tooling migration, a notable incident, a holiday quarter.

What "90 days" actually means

A single quarter of data is the floor. It is enough to see direction and to detect order-of-magnitude shifts, but it is not enough to separate AI signal from natural variance for any metric with a long natural cadence (incident frequency, severity-1 outages, quarterly survey scores). For those metrics, anchor the baseline to a longer history (typically 12 months) even if the granular pulse is only the most recent 90 days.

The "control team" pattern

Where it is operationally feasible, hold one team out of the AI rollout for the pilot quarter to act as a control. This is the single most powerful baselining technique available, and it is almost always achievable in companies above ~50 engineers.

How a control team is constructed

  • Pick a team comparable to the pilot team on team size, system age, technology stack, and risk tier, not just one that volunteers.
  • Confirm the team's leadership accepts the role. Being the control is not punishment. Frame it explicitly as contributing to the company's evidence base.
  • Instrument the same metrics on the same cadence for the control team and the pilot team.
  • Lock the AUP for the control team for the duration of the pilot: no shadow-IT use of personal AI tools.
  • Plan for crossover. At the end of the pilot, the control team receives the tool. This both rewards them for the role and gives the program a second before/after curve to compare against.

What a control team gives you

  • A reading on how much of the change is the AI rollout vs. how much is something happening elsewhere in the company. The control team is the answer to "wasn't there a re-org that quarter?": yes, and the control team experienced it too.
  • A defense against selection bias in the pilot team. Pilot teams tend to over-recruit enthusiasts.
  • A natural comparison group for the retrospective: the same numbers, the same period, with and without the intervention.

Sizing guide: control team

  • Small (< 50): A control team is usually impractical; the company is the pilot. Compensate with a stronger historical baseline (12 months instead of 90 days where possible).
  • Mid (50–500): A single control team for the pilot quarter is achievable and high-value. Plan for crossover at pilot end.
  • Enterprise (500+): Hold out one or more comparable teams per business unit running a pilot. Phase 2's wave-based rollout makes this natural: the next wave is the control for the current wave.
  • Regulated: Control teams should be the default; the auditability gain alone justifies the operational cost.

Seasonality and confounds: why a single quarter lies

A 90-day baseline taken in the wrong quarter will be wrong by 20–40% on the noisier metrics. Common confounds:

  • End-of-quarter throughput spikes in companies with quarterly feature commitments. A pilot whose "before" period straddles a spike will look slower in the "after," even when AI helped.
  • Holiday quarters depress deploy frequency and inflate MTTR (smaller on-call rotations).
  • Hiring waves introduce a lag. Onboarding cost is real and shows up as throughput drag for ~60 days per new hire.
  • Re-orgs and team moves invalidate team-level baselines for the affected teams. Re-baseline after a re-org, or exclude affected teams from the comparison.
  • Tooling migrations (CI changes, deployment-platform changes) create a discontinuity that AI rollout will get blamed for.

How to correct for confounds

The honest correction methods, ordered from strongest to weakest:

  1. Use the control team as the natural seasonality control. If the control team's metrics moved with seasonality, subtract the movement from the pilot team's measured change.
  2. Use year-over-year comparison. Compare the pilot quarter to the same quarter the prior year, not to the immediately preceding quarter, when seasonality is suspected.
  3. Annotate the confound on the dashboard. A re-org event line on the chart so anyone reading the data can see when it happened. This is mandatory regardless of which other correction is used.
  4. Widen the baseline. When in doubt, extend the baseline window from 90 days to 180 or 365 days for the affected metric. More data dilutes a single confound.

Pre-register the correction method

Decide during the baselining workshop what correction method will be used if seasonality is detected. Pre-registration prevents the correction from being chosen post-hoc to favor a desired conclusion.

Sources of pre-existing data

Most companies have more usable historical data than they realize. The baselining workshop's primary task is often inventory, not collection.

Source What it contains Typical caveat
Git history Commit cadence, PR size, churn, file-level history. Author identity may be unreliable across re-orgs and email changes.
CI / CD logs Build duration, test pass rates, deploy frequency, deploy success rate. Often retained 90 days only; capture before any rollout.
Incident tracker Severity, MTTR, root cause, AI attribution if tagged. Pre-AI incidents won't be tagged with AI relevance; re-classify on review.
Vulnerability scanner / SCA Vulnerability count and time-to-fix trend. Tool changes invalidate cross-period comparisons; lock the tool version for the baseline.
Survey history Past eNPS, DevEx pulse, engagement results. Question wording must match the post-pilot survey exactly.
IDE telemetry (with consent) Edit time, build-error frequency, time-in-editor. Consent and privacy obligations apply; many teams should not collect this.
Code-review tool Review duration, review-comment density, approval delay. Tool migration in the period invalidates the baseline.

IDE telemetry and individual-level activity data are subject to employment-law and works-council requirements in many jurisdictions (notably Germany, France, the Netherlands, and most EU member states under GDPR plus collective-bargaining frameworks). Treat these sources as opt-in, aggregate-only, and never used for performance management. The audit-trail page covers retention and access controls. See Audit trail.

Sizing: what to baseline at what size

Sizing guide: baseline depth

  • Small (< 50): Git history + a one-question pulse survey is enough to start. Reconstruct cycle time and PR review time from git; ask "in the last 30 days, how often did you feel blocked waiting on something?" The baseline document is two pages.
  • Mid (50–500): Add CI metrics, incident tracker counts, and a 5-question DevEx pulse. The baseline document is five to ten pages, organized by metric family from the taxonomy.
  • Enterprise (500+): Instrument a formal DevEx survey on a quarterly cadence, add IDE telemetry with consent, partner with finance for cost baselines (tooling spend per engineer, cost per shipped feature). Multi-team baseline. Audit-ready.
  • Regulated: Add an Audit Liaison so the baseline document can be cited as evidence in regulatory submissions or assurance work later.

The baseline as a living document

A baseline produced once and never refreshed becomes wrong within two quarters. The baseline is version 1.0 at the end of the worksheet; the framework expects revisions.

  • Monthly: the same metric values are added to the dashboard so the baseline becomes a continuous time series, not a single snapshot.
  • Quarterly: the metric set is reviewed. Metrics that didn't move a leadership decision are candidates for retirement; new metrics revealed by incidents are candidates for addition.
  • Annually: the entire baseline is re-validated. Confounds that emerged during the year (re-orgs, tooling changes) are documented; affected metrics are re-baselined where needed.
  • After any major incident: the metric that should have caught the incident, but didn't, is added if missing, or its threshold is reviewed if it existed.

Outputs

A successful baselining produces, at minimum:

  1. Baseline snapshot document: one page per metric family, each with the six attributes above.
  2. Gap list: metrics the pilot will be judged against that cannot yet be produced, with a 90-day plan per gap.
  3. Confound register: the known seasonality, re-org, tooling, and hiring events that the metrics will need to be read against.
  4. Pre-registered correction method: what the program will do if seasonality is detected at retrospective time.
  5. Owner per metric: a named human responsible for definition, source, and confidence.
  6. Decision Record capturing the metric set the pilot will be judged against, signed by the executive sponsor.

Anti-patterns

Don't

  • Don't start measuring at the pilot. The pilot is the intervention. Measuring only the intervention is not measurement, it is observation.
  • Don't baseline only productivity metrics. Quality and security counterweights are the metrics that catch "feels-faster, measures-worse"; without them the program cannot detect the most likely failure mode.
  • Don't compare individuals. Baselines are self-comparisons across time, not league tables. Cross-individual comparison invites performance-management misuse and breaks trust in the data.
  • Don't allow "we'll figure out the baseline once we have something to show." A baseline produced after the intervention is not a baseline.
  • Don't pretend confounds don't exist. Annotate them. A re-org during the baseline period that goes undocumented becomes the loudest voice in the retrospective's interpretation.
  • Don't conflate the worksheet with the concept. Running the worksheet once does not finish baselining; refresh and revise are part of the contract.

Cross-references

Footnotes & sources

  1. Google Cloud / DORA, "2024 Accelerate State of DevOps Report", Oct 2024. ↩