Skip to content

Pilot the paved road with real builders

Run a 90-day pilot program with a small group of real builders and the real apps that reach intake, from Sandbox to Critical. Take a baseline first, fix exit and kill criteria, and name an evaluator. Judge the result on measured value against cost, support load and AI spend, not on how builders felt. Group sizes and thresholds on this page are suggestions to test, not evidence, as of October 2026.

Who this is for

IT managers and enablement leads who have the paved road (at least a thin one) and want to test it before opening it to everyone. You get the scope, a staggered training design, the measures (also as a CSV), the go/no-go rules, a scope template, a retrospective agenda and a decision record. Reading time: about 11 minutes.

What makes this a pilot and not a rollout?

A pilot can fail. If no result could produce a "no", you're running a rollout with a pilot's name on it. The pilot exists to test the assumptions this framework rests on, so each measure below is a hypothesis with a result that would prove it wrong. Government trials that published their findings show the pattern to copy: a fixed scope and period, a named evaluator, and honest limits on the data (for example, self-reported time savings that the authors flag as optimistic).1 US federal policy treats an AI pilot the same way: limited in scale and duration, tracked centrally and opt-in where possible.5 The pattern to avoid is pilot purgatory, where licences quietly become permanent because nobody defined an exit.

Treat the 90 days as a test of the program, not of the builders.

Don't: run a pilot with no owner, no metric and no end date

A pilot with no named evaluator and no exit criteria drifts into permanent unmanaged use (AP20). Write the exit and kill criteria before the first builder starts, and name the person who will call it.

Who and what is in scope?

Choose the builders on purpose, but let the apps arrive. If someone hand-picks the apps, the tier mix is whatever that person chose, and the tier hypotheses can't fail.

Decide Suggestion (a starting point, not evidence)
Builders Roughly 8 to 20 across at least three business units, including some who didn't volunteer and at least one professional developer building their own tool. Split them into two training waves (below)
Apps Three streams, none hand-picked: the apps pilot builders would build anyway, every intake from pilot units through the normal intake path, and existing apps that discovery finds in those units. Tier each with the six tier questions. If the mix misses a tier, record that; don't steer it
Critical apps Allowed through intake, tiered and counted. A Critical app doesn't go live in the pilot until its gates are met (a graduation plan, the security review and IT co-ownership). Most will wait or start graduation, and that's a finding
Platform pieces The paved road as it stands: approved tools, sign-in, private-by-default hosting, a secrets store, the registry, intake
Tools Whatever your approved list holds
People The enablement lead, two or three champions, a technical reviewer, an intake responder, a named evaluator who isn't the program's main advocate, and an executive sponsor

Don't: pilot one tool only

A pilot that tests one tool can't tell you whether the default path is attractive against the alternatives builders already use. Run the pilot on whatever your approved list holds.

Take the baseline first

Measure before anyone starts, or you'll only have opinions at day 90. Follow baselining, and capture at least:

  • how many worker-built apps you know about in the pilot units, and how many have a named owner;
  • how long an intake answer and a technical check take today, if you have either;
  • which unapproved tools builders use today (a short survey, with a promise it won't be used to punish anyone);
  • for each process a pilot app is meant to improve, one measure from the process's own records, such as cycle time, items handled per week, or errors and rework;
  • current AI spend by tool, and the hours people spend on those processes today.

How should pilot builders be trained?

Stagger the training so the pilot can tell whether training matters (A-023) and whether manager involvement matters (A-022). This is a comparison inside normal work, not a research trial.

  1. On day one, every builder gets access, the never-dos briefing and the intake form. The rules aren't staggered; only the track and the coaching are.
  2. Split builders into two waves by team, not by enthusiasm: list the teams and alternate them, or toss a coin per team. Wave 1 takes the builder track in weeks 1 and 2, before building. Wave 2 builds from day one and takes the track at about day 45.
  3. Stagger managers separately: in each wave, the managers of half the teams take the manager track at the start, and the rest at day 45.
  4. At day 45, compare the groups on the training rows below, then train everyone. Wave 2's before-and-after gives a second reading.

With 8 to 20 builders, this shows a direction, not proof. Report counts and name confounds, such as one wave getting harder apps. If 45 days without the track is unacceptable, compare against a matched team outside the pilot (the control-team pattern) and say the comparison is weaker, because that team also lacks the paved road.

Train two or three champions as technical reviewers before day one. An IT technical reviewer co-signs each champion's first three checks and re-checks about one in five after that.

What do you measure, and which assumption does each measure test?

Each row tests an assumption by its ID in the register. Write the thresholds into the scope statement before the pilot starts. They're starting points drawn from the register's falsifiers, not benchmarks. Below about 20 apps, report tier counts, not percentages: a pilot can only give early signals on A-028, A-043 and A-045.

Measure Assumption How measured Threshold Proves it wrong
Self-registration A-001 Builders register their own apps, timed; IT doesn't do it for them. Count discovered or unplanned apps registered without a chase Median 10 minutes or less; at least half registered within two weeks of the invitation Median over 10 minutes, or fewer than half register unchased
Intake completion time A-031 Median start-to-submit time; abandoned forms Median 10 minutes or less Median over 10 minutes, or many abandoned forms
Intake answer time A-030 Median working days from submit to answer, by tier 5 working days or less Median over 5 working days
Default-path use A-002 Share of new apps in pilot units on approved tools, against the baseline survey Most new apps on approved tools Heavy unmanaged use continues even with fast exception answers
Tier mix (apps not hand-picked) A-045 Tier counts from the registry for intakes and discovered apps 35% or less at Important or above Over 35% at Important or above
Critical share A-028 Count of intakes that land in Critical Under 10% 10% or more at Critical
Downgrades A-043 Downgrades against Important and Critical answers About 15% or less Over about 15%, or a downgrade whose compensating control had lapsed
Technical-check wait A-042 Median working days from request to completed check 5 working days or less Median over 5 working days
Champion check agreement A-042 IT re-checks a sample of champion technical checks Re-checks agree A re-check finds an exposure or failed data-rule test the champion passed
Trained vs not-yet-trained builders A-023 At day 45, weeks with use in tool logs, intake rework and self-check passes, wave 1 against wave 2 Wave 1 ahead on use and rework Wave 2 matches wave 1
Manager track A-022 At day 45, sustained use for teams whose managers took the track against those whose managers hadn't Trained-manager teams ahead No difference
Champions (observe only) A-024 Unapproved-tool use and registration in pilot units against non-pilot units None set: confounded, because non-pilot units lack the paved road Treat no difference as an early warning, not a result
Owner and successor A-034 Owner roles with no holder, and missing successors, at day 90 Few or none More than a small share are empty
Process outcome A-009, A-010 One before-and-after measure per app, set at intake, from the process's own records; self-report alongside, never instead Measured improvement in at least half the apps that went live Improvement in fewer than half, or self-report well above the measured figure
Support load created A-018 Questions, tickets and hours per app per month, from the office-hours log and helpdesk tags Support follows the tier's support level IT routinely supports Sandbox or Team apps, or hours per app keep rising
AI spend A-027 Spend per tool and per app from billing, against the budget and caps in AI spend Within budget; every cap breach alerted Spend varies under 25% month to month per tool without controls
Pilot cost against value A-010 People-hours of the pilot roles and builders, AI spend and platform costs, against measured value Measured value covers the next stage's projected cost, assumptions written down Measured value is below cost even on generous assumptions
Cost of the metrics A-032 Hours spent producing the monthly numbers About a day or less More than about a day of manual work

Download the measures as a CSV: pilot-measures.csv. Prompt to Prototype, Pilot the paved road with real builders, https://prompt-to-prototype.com/roadmap/pilot/, as of October 2026. Licensed CC BY-SA 4.0. Using or adapting it inside your business isn't distribution under the licence. The table on this page is the source of truth.

Self-reported time saved is a weak measure on its own. In a randomized trial, experienced developers believed AI made them faster while measured completion time went the other way.2 The UK DBT evaluation found satisfied users and time saved on some tasks, but no clear productivity gain.4 That's why the value rows use the process's own records, and why a 90-day saving that you annualize needs its assumptions written next to it.

Don't: treat satisfaction or acceptance rate as success

A pilot that ends with "builders liked it" has measured mood. Practitioners are moving away from acceptance rate as a success measure.3 Judge the pilot on the table above, plus incidents and the retrospective.

What does a go need?

Write the go rules into the scope statement before day one. A suggested go:

  • no kill criterion is open;
  • the process outcome improved, by measurement, in at least half of the apps that went live;
  • the pilot's full cost is known, and the measured value covers the next stage's projected cost;
  • support load and AI spend stayed within plan;
  • every measure is reported as met, missed or ambiguous, including the ones that missed.

A no-go is a valid result. So is a go with conditions, such as more technical reviewers before access widens.

What are the kill criteria?

Write these before the start and give one person the authority to pull the cord. Suggested stops, to adapt:

  • Any exposure of Confidential or Restricted data from a pilot app. Pause new access, run incident response, and resume only when the cause is fixed.
  • An agent or automation acting with permissions beyond what its record shows.
  • A Critical app found live without its gates.
  • Technical-check wait past ten working days for two weeks running, which means the pilot is outrunning review capacity.
  • Builders abandoning the default path in numbers, which is the answer to A-002 arriving early.

How do you support builders during the pilot?

  • Weekly office hours, run by champions in rotation, as described in enablement and training. Log every question they couldn't answer.
  • A named intake responder, so answers come within the five-day hypothesis. Count the hours as pilot cost.
  • A fortnightly sponsor readout of the measures, with the raw numbers.
  • A mid-pilot checkpoint at day 45, after the training comparison: are the measures on track, do any incidents argue for a pause, and who outside the pilot is building, under the shadow AI policy?
  • An incident and near-miss log: date, app, what happened, how it was found, impact, root cause and fix.

Worksheet: pilot scope statement template

Fill this in with the sponsor, evaluator, enablement lead and a technical reviewer in about an hour, and keep it somewhere durable. Download it as pilot-scope-template.txt. Prompt to Prototype, Pilot the paved road with real builders, https://prompt-to-prototype.com/roadmap/pilot/, as of October 2026. Licensed CC BY-SA 4.0.

# Pilot scope

Pilot ID, start date, end date (about 90 days), day-45 checkpoint date
Sponsor, evaluator, enablement lead, champions, technical reviewer, intake responder

Builders in the pilot (how they were chosen, including any who didn't volunteer)
Training waves: which teams are wave 1 and wave 2, and which managers train at the start
App streams: builders' own apps, normal intakes from pilot units, discovered apps
Critical apps: how they are tiered and counted, and the gates they must meet before going live
Out of scope: named systems, named data sets

Paved-road pieces in use: tools, sign-in, hosting, secrets, registry, intake
Baseline: where it is recorded and the date taken

Measures: each with the assumption ID, a threshold and a "proves it wrong" result
Value: the process measure for each app, and its baseline
Cost plan: people-hours by role, AI budget and caps, platform costs
Go rules, and kill criteria with who decides
Decision rights: scope changes, tool additions, pause (who and by when)
Review cadence: weekly office hours, fortnightly readout, checkpoint, retrospective

Agenda: the retrospective (days 85 to 90)

Allow two hours with the pilot builders, the sponsor, the evaluator, the enablement lead, a security reviewer and the IT or platform owner. Circulate the scope statement, the measures with baselines, the cost figures, the incident log and the training records 48 hours ahead.

Minutes Topic Lead
10 Restate the scope statement, the go rules and the kill criteria. Numbers come after this Evaluator
25 Go through each measure against its threshold: met, missed or ambiguous, and any confound (holidays, freezes, staffing changes) Evaluator
10 Cost and value: measured outcomes, people-hours, AI spend and support load against plan Sponsor
20 Walk through each incident: cause, fix, whether it would recur at larger scale Security reviewer
15 Hear from builders: what got easier, what got in the way, what they bypassed and why Champions
10 Name surprises, good and bad Everyone
20 Draft the decision: go, no-go or extend, with conditions Sponsor
10 Assign follow-ups and a date to publish Enablement lead

Keep facts and interpretations apart. "Median intake answer took three days" is a fact. "Builders trust the process" is an interpretation.

Don't: extend the pilot to avoid a no

An extension answers a new, specific question. Without one, it's avoidance. If the room is split, extend only with a named question and a date.

Record the decision

Illustrative composite

Title: Pilot outcome for the paved road · Status: proposed · Date: YYYY-MM-DD · Owner: sponsor (a role)

Context. We ran a 90-day pilot with a small group of builders in two training waves, on apps that arrived through intake and discovery, against a baseline, written go rules and kill criteria.

Decision. Go, no-go or extend. If go, name the conditions for the next stage, for example a review-capacity check before widening access (see scaling the program), and the measured value and cost the decision rests on. If no-go, record what we learned and the date we'll try again. If extend, state the one new question and the end date.

Alternatives considered. Skip the pilot and open to everyone (rejected: no evidence the assumptions hold). Pilot one tool only (rejected: can't test the default path against alternatives). Hand-pick the apps or register them for builders (rejected: the tier and registration hypotheses couldn't fail). Train everyone at the start (rejected: no comparison group).

Consequences. The enablement lead publishes what the pilot found, including the measures that missed, and updates the training and the paved road. Review the decision at the next program review.

Review by: six months from the date above.

What happens after a go, no-go or extend decision?

If the decision is go, plan the next stage in scale the program, and confirm review capacity can keep up with change volume first. Move any app that turned out to be Critical through graduation to IT. A real test of funded champions (A-024) fits the next stage: phase champion funding in unit by unit once every unit has the paved road. If the decision is no-go, don't discard the findings: log them against the assumptions, fix the weakest piece and rerun a smaller pilot.

Assumptions and evidence

  • A-001, A-002, A-009, A-010, A-018, A-022, A-023, A-027, A-030, A-031, A-032, A-034, A-042: tested in the measures table. A-022 and A-023 rest on the staggered waves and give a direction, not proof.
  • A-028, A-043, A-045: early signals only. A-024: observed, not tested.
  • Evidence for the pilot method: government trials that published findings with their limits, and a federal policy model for time-boxed, tracked pilots. Evidence on training and managers is survey and self-report, not a controlled study of non-developer builders.

Facts on this page are as of October 2026.

  1. UK Government Digital Service, "AI coding assistant trial: UK public sector findings report" (2025): bounded trial, self-reported savings flagged as optimistic, no code-quality or security metrics analyzed. https://www.gov.uk/government/publications/ai-coding-assistant-trial/ai-coding-assistant-trial-uk-public-sector-findings-report [primary], accessed 2026-10-22. ↩

  2. METR, "Measuring the impact of early-2025 AI on experienced open-source developer productivity" (2025-07-10): randomized trial of 16 developers; they expected and believed a speedup, and measured time was longer. Small sample. https://metr.org/blog/2025-07-10-early-2025-ai-experienced-os-dev-study/ [primary], accessed 2026-10-22. ↩

  3. LeadDev, "The rise and looming fall of acceptance rate" (2025). https://leaddev.com/reporting/the-rise-and-looming-fall-of-acceptance-rate [news], accessed 2026-10-22. ↩

  4. Civil Service World, "DBT trial finds Copilot saves staff time but doesn't improve productivity" (2025-09). https://www.civilserviceworld.com/news/article/dbt-trial-finds-copilot-saves-staff-time-but-doesnt-improve-productivity [news], accessed 2026-10-22. ↩

  5. US Office of Management and Budget, memorandum M-25-21 (2025-04-03), section 4(a): pilots are exempt from minimum practices only if limited in scale and duration, certified, centrally tracked and opt-in where possible. https://www.whitehouse.gov/wp-content/uploads/2025/02/M-25-21-Accelerating-Federal-Use-of-AI-through-Innovation-Governance-and-Public-Trust.pdf [primary], accessed 2026-10-22. ↩