Phase 1: Pilot (0–90 days)¶

Duration: 90 days. Goal: prove, or disprove, that AI-assisted development delivers measurable value in your context, on a bounded scope, with acceptable risk.

The test of a real pilot

A real pilot can fail. If the exit criteria cannot produce a "no" outcome, you are not running a pilot. You are running a procurement exercise with extra steps.

What Phase 1 produces¶

Output	Worksheet
Pilot scope & success criteria: one squad, one system, measurable success/failure criteria, 90-day deadline.	Pilot scope
Tool selection & onboarding: a single primary tool (or 2–3 if comparing), approved with Legal + Security.	Vendor selection
Training plan for pilot team: 4–8 hours per participant, role-differentiated.	Training needs
Weekly pulse metric dashboard: the baseline metrics from Phase 0, tracked weekly.	Metrics baselining
Documented incidents: every AI-adjacent defect, near-miss, and security finding from the pilot, with root cause.	Incident tabletop (plus running log)
Retrospective & go/no-go recommendation, signed by the pilot sponsor.	Pilot retrospective

Exit criteria¶

The pilot exits when all of these are true:

90 days have elapsed (or the pilot was stopped earlier with a documented reason).
Each pre-defined success criterion has been evaluated against evidence, not anecdote.
Every documented incident has a root cause and a mitigation for Phase 2.
The retrospective worksheet has been completed and signed.
The sponsor has made a go/no-go/extend decision, recorded in the Decision Record.

Sizing guide

Small (< 50): pilot the whole company for 30 days, not a squad for 90. Retrospective is a single meeting.
Mid (50–500): one squad of 4–8 people on one bounded system, 60–90 days.
Enterprise (500+): 2–3 parallel pilots across distinct business units (not all in the most permissive one). 90 days each.
Regulated: expect 120+ days due to additional security reviews and DPIAs. Budget for it; don't compress and ship unsafe defaults.

Pilot selection: choose with intent¶

A bad pilot choice will either:

Over-succeed (greenfield CRUD service with no regulatory constraints). Results won't generalise to your hard problems.
Over-fail (oldest, gnarliest legacy system). Results won't generalise to your normal work either.

Pick a representative workload. The Pilot scope worksheet walks through the choice. Heuristics:

Prefer	Avoid (for first pilot)
Brownfield but maintainable	Greenfield with no users
Medium-risk classification (not Low, not Critical)	Critical-tier (payments, identity)
A team that has complained about toil	A team that is against adoption on principle
A system with existing test coverage	A system with no tests
A 4–8 week feature of normal scope	A "transformation" that would take two quarters

Tool selection: one, maybe three, not seven¶

Do not let developers individually choose tools at this stage. You need concentrated signal, not scattered noise. Use the Vendor selection scorecard to pick:

One primary tool for everyone in the pilot, OR
Two or three tools in a deliberate bake-off where each pilot participant uses one.

Minimum scorecard dimensions:

Data handling. Does the vendor train on your prompts by default? Can you opt out? Where does data sit? What retention?
Enterprise controls: SSO, SCIM, audit logs, private deployment.
IP indemnification. What does the contract promise if a generated snippet infringes a license? Doe v. GitHub¹ made this table-stakes.
Security attestations: SOC 2 Type II, ISO 27001, ISO 42001.
Regional data residency, if you have EU, UK, or sectoral requirements.
Agent capabilities & guardrails. Can an agent write to prod? Can you forbid it?
Pricing & exit: per-seat cost, minimum commitment, data-export on termination.

Publicly known reference points for tool selection

Goldman Sachs piloted Copilot to ~1,000 engineers before expanding²; JPMorgan built an internal gateway ("LLM Suite") so employees didn't go to public models³; Amazon reported using Amazon Q Developer to accelerate Java 8→17 migration at scale⁴. All three resolved data-handling and IP indemnification before expanding access.

Training: lightweight, role-specific, recurring¶

The pilot team needs training before they touch the tool, not after. Four hours minimum, ideally eight. Structure:

Session	Length	Who
What the tool does, what it doesn't do, and what's public evidence on both	1 hr	Whole pilot team
The company's AUP, walked through every clause, not a link-dump	1 hr	Whole pilot team + Security
Prompting patterns (S.C.A.F.F.) with hands-on exercises	1.5 hrs	Developers
Verification patterns (V.E.R.I.F.Y., C.L.E.A.R.) with the team's real code	1.5 hrs	Developers + tech lead
Security-specific patterns (S.H.I.E.L.D.)	1 hr	Developers + AppSec
Incident reporting walkthrough: how to raise concerns	30 min	Whole pilot team

Don't outsource the AUP training

The temptation is to let the tool vendor deliver the training. The vendor will not train on your AUP. Deliver the AUP session internally; let the vendor deliver product mechanics.

Metrics: weekly, not monthly¶

During the pilot, track metrics weekly, not monthly. Ninety days is not a long time; a month-granular dashboard will only give you three data points. Weekly gives you thirteen.

Track:

Adoption signals: % of pilot team using the tool at least once per day, sessions per developer per week.
Productivity signals: cycle time, PR throughput, PR size, time-to-first-PR for new hires.
Quality signals: change-failure rate, escape defect rate, revert-within-14-days rate (GitClear's "churn" metric).
Security signals: count of AI-adjacent security findings, time-to-fix on them.
Sentiment signals: weekly 2-question micro-survey (NPS-style + open-text). Critical: ask the same questions at pilot start and end. Satisfaction often rises even when throughput falls, and that delta is the most interesting finding⁵.

See Measurement: metrics taxonomy for definitions and anti-patterns for what not to measure.

Incidents: log everything¶

Every AI-adjacent defect, near-miss, surprising output, or security finding goes in a running log. Template per entry:

RenderedMarkdown source

Incident log entry

Date: 2026-05-14
System: checkout-service
Tool: redacted for anonymity in public copy; use tool name internally
What happened: The assistant suggested using an unmaintained package (last release 2019) for JWT handling; suggestion was accepted and merged before a security reviewer flagged it.
How detected: SCA scan on next nightly pipeline.
Impact: No production exposure; caught pre-deploy.
Root cause: Prompt did not specify "actively maintained, license compatible"; reviewer did not run SCA before approving.
Mitigation applied: Added S.C.A.F.F. "Foundations" clause for dependencies; SCA promoted to pre-merge gate on pilot repo.
Status: Closed 2026-05-16.

- Date: 2026-05-14
- System: checkout-service
- Tool: <redacted for anonymity in public copy; use tool name internally>
- What happened: The assistant suggested using an unmaintained package
  (last release 2019) for JWT handling; suggestion was accepted and merged
  before a security reviewer flagged it.
- How detected: SCA scan on next nightly pipeline.
- Impact: No production exposure; caught pre-deploy.
- Root cause: Prompt did not specify "actively maintained, license compatible"; reviewer did not run SCA before approving.
- Mitigation applied: Added S.C.A.F.F. "Foundations" clause for dependencies; SCA promoted to pre-merge gate on pilot repo.
- Status: Closed 2026-05-16.

Retrospectively, this log becomes your richest artifact, richer than any metric dashboard. It is what tells a board of directors whether the program is operating safely.

The mid-pilot checkpoint (day 30–45)¶

Halfway through the pilot, run a one-hour checkpoint. Four questions:

Are we on track against the success criteria? If not, why?
Are there incidents whose root cause suggests we should pause?
Is the tool/vendor behaving as expected? Any surprises?
Is anyone outside the pilot already using the tool? (Yes. Always yes.) What does the AUP say about that, and is it being honoured?

Don't treat the checkpoint as optional. Programs that skip it almost always discover at day 90 that something was off at day 45.

The retrospective (day 85–90)¶

Use the Pilot retrospective worksheet. Produce:

A one-page summary for leadership with metrics vs. targets.
A risk register update: what did the pilot tell us about risks in Phase 2 that Phase 0 didn't anticipate?
A go / no-go / extend recommendation with rationale.
An updated AUP draft. The interim policy will need changes based on real incidents.
A training curriculum update: what did the team need that it didn't have?
A Decision Record that captures all of the above for the historical record.

Anti-patterns¶

Don't

Don't extend the pilot to avoid a "no" decision. If the data says no, say no. Extending a pilot is a rational response to new information, not to disappointing results.
Don't pilot with your most enthusiastic team only. Selection bias will produce results that don't generalise. Include one team that didn't volunteer.
Don't skip the 30-day checkpoint. It is the single highest-value hour in the whole pilot.
Don't pilot without baselines. "We feel faster" is not a measurement. See Phase 0.
Don't let the pilot's scope creep. If the pilot was "auth service for 90 days," don't let it become "auth service plus notifications plus the new admin UI." Scope creep = no comparable result.

Next¶

If the pilot said "go," move to Phase 2: Scale.
If the pilot said "no go," document it; run Phase 0 again in 6 months with updated context.
If the pilot said "extend," write down exactly what new question the extension answers. Without a new question, an extension is just avoidance.

Doe v. GitHub, OpenAI, Microsoft class action, filed Nov 2022; see Evidence: security incidents. ↩
Reuters, "Goldman Sachs rolls out generative AI tools", Jun 2024. ↩
Financial Times, "JPMorgan rolls out in-house chatbot", Jul 2024. ↩
Andy Jassy 2024 shareholder letter, aboutamazon.com, Apr 2024. ↩
Uplevel, "Can Generative AI Improve Developer Productivity?", Sep 2024. ↩