Skip to content

Incident response

Incident response here is an addendum to the company's existing IR framework, not a replacement for it. The core mechanics (detection, triage, containment, eradication, recovery, post-mortem) do not change. What changes is the actor taxonomy and the scenario library.

For the tabletop that exercises this framework, see the Incident tabletop worksheet.

What counts as an AI-adjacent incident

Any event in which an AI coding tool or agent materially contributed to a loss or near-loss of confidentiality, integrity, or availability. Including:

  • Data exposure to an AI tool (prompt leak).
  • A vulnerability introduced by accepted AI output.
  • An action taken by an autonomous AI agent that produced or threatened material impact.
  • A vendor-side incident affecting company data.
  • A case where a customer or regulator received a factually incorrect statement from an AI system.
  • An intellectual-property claim traceable to AI-generated code.

"Near-miss" incidents count. They are the most cost-effective source of learning.

Severity matrix

Severity Definition Examples
SEV-1 (Critical) Active material impact to production, customers, or regulators. Agent destructively altered prod data; customer-facing AI published misinformation; confirmed exposure of restricted data to public model.
SEV-2 (High) Confirmed breach of policy or control with plausible impact. Production credentials pasted into an AI tool; slopsquatting detected in deployed service; verbatim-match IP claim.
SEV-3 (Medium) Policy / control breach without immediate impact. Unapproved tool used on internal-class data; tool output merged without verification on a Medium-tier path.
SEV-4 (Low) Near-miss, audit finding, or procedural breach. Engineer almost pasted a key; unapproved tool requested but not used; lapsed training.

Actor taxonomy

Traditional IR taxonomies categorise incidents by actor type (internal, external, accidental, malicious). AI introduces a fifth category that must be represented explicitly:

Actor Notes
Internal accidental Human engineer error.
Internal malicious Rare; still tracked.
External attacker Via supply chain, prompt injection, model manipulation.
Vendor AI provider-side outage or exposure.
AI system An AI agent or assistant acting outside expected bounds.

The "AI system" actor is new for most organisations. It is not a blame category (the AI is not a moral agent), but it is a cause category that must appear in the taxonomy so post-mortems can address it.

Process

1. Detection

  • Engineer self-report via the AUP incident channel. One channel, one SLA, typically same business day.
  • Automated detection. SCA, SAST, egress monitoring, secret scanning, vendor status page integrations.
  • Vendor notice. Per DPA, vendor-initiated reports.
  • External notice. Journalist, security researcher, customer.

2. Triage

  • Severity call per the matrix above. Made by IR Lead (or on-call delegate). Severity may be revised up, rarely down.
  • Scope assessment: data classes involved, systems affected, users affected.
  • Actor assignment from the taxonomy.
  • Severity-driven escalation:
  • SEV-1: sponsor + CISO + Legal + Comms notified immediately.
  • SEV-2: CISO + Legal within 2 hours; sponsor within 24.
  • SEV-3: CISO next business day.
  • SEV-4: logged for trend analysis.

3. Containment

  • Revoke credentials / rotate keys if a secret is suspected exposed. Do not wait for confirmation.
  • Suspend tool access (individual, team, or tool-wide) if the tool is the contamination path.
  • Disable agent permissions if an agent is the actor; error on the side of pausing all agents until the scope is known.
  • Freeze deployments from affected paths if the incident is a supply-chain event (e.g., slopsquatting, dependency backdoor).

4. Eradication

  • Remove or remediate introduced vulnerabilities, bad dependencies, or unauthorised actions.
  • Verify via re-scan, penetration test, or data-flow review.

5. Recovery

  • Restore from backup or replay (if data was affected).
  • Re-enable tool access with conditions if tool was suspended.
  • Re-authorise agents with updated guardrails.

6. Post-mortem

  • Held within 14 days for SEV-2+.
  • Blameless format.
  • Outputs:
  • Root cause from the five-why / timeline.
  • Corrective actions with owners and deadlines.
  • Framework updates: AUP, training, controls.
  • Decision Record if material policy changed.
  • Changelog entry on the public site if public-facing framework changed.

Agent-caused incident handling

When an AI agent is the actor, additional steps apply:

  • Preserve the agent's action log: prompts, tool calls, decisions. This is the equivalent of preserving volatile memory.
  • Identify the trigger. Was the agent responding to operator instruction, self-initiated reasoning, or manipulated input (prompt injection)?
  • Scope the permission used. What was the agent authorised to do, vs. what it did? Did it exceed permissions, or were the permissions too broad?
  • Consider tool-wide implications. If one agent misbehaved, could others? Pause class-wide if uncertain.
  • Update agent permissions post-incident; do not restore autonomous write to production unless the incident's root cause is addressed.

The 2025 Replit agent incident (the agent deleted a production database during a code freeze) is the reference case for this workflow1. Its lessons: agents should not have autonomous destructive permissions; the audit log matters more than the agent's self-report; re-authorisation must follow a post-mortem, not an apology.

Vendor-side incidents

When the vendor is the actor (e.g., the DeepSeek ClickHouse exposure2), the company's response typically runs in parallel with the vendor's:

  • Assess whether company data was in the affected scope using data-flow maps and AI-tool audit logs.
  • Execute holding statements per the rehearsed Comms pattern.
  • Evaluate exit. Is this incident large enough to switch to fallback? A pre-approved exit plan (Vendor standards) makes this answerable in hours, not weeks.
  • Collect vendor RCA and add to the vendor's scorecard at next review.

Security by omission

A substantial share of AI-adjacent incidents are not caused by AI producing something wrong. They are caused by AI producing exactly what was asked for, and silently leaving out the security control the prompt didn't name. This pattern is well captured in community writing on "secure vibe coding"3: the model generates what you ask for, not what you forget to ask.

Two representative cases illustrate it:

  • A developer asks an assistant to "fetch stock prices from an API" and commits the resulting code, including the API key hardcoded as a string, to a public repository. The code worked on first test; the key was exfiltrated by a secret-scanning bot within minutes.
  • A developer asks for "a password reset function that emails a reset link." The assistant produces working code using a naïve string comparison (token == expected_token) instead of a constant-time compare (hmac.compare_digest). Legitimate users see no issue; a timing-attack path exists against anyone willing to measure response times.

Neither incident shows a bug the assistant made. Both show the assistant doing exactly what it was asked, omitting a control the prompt didn't name. Classify these as SEV-2 at minimum when they ship to production paths, and track them under the tag security-by-omission so the pattern is visible at trend time.

Operational implications:

  • Prompt templates (S.C.A.F.F.) must name the security requirements explicitly: "use constant-time comparison", "never hardcode credentials", "validate input against OWASP Top 10 cases".
  • Review (C.L.E.A.R.) must include a negative-space check: what would I expect to see for this kind of change that is not present?
  • Training must teach the pattern by example, with real stories, not abstract warnings.

Prompt injection and external manipulation

Prompt injection is a category of attack where untrusted inputs cause an AI system to deviate from its designed behavior. It is well-documented in OWASP's Top 10 for LLM Applications4. For coding agents specifically, prompt injection can appear through:

  • Repository content. A malicious comment in a file designed to influence a code-assistant agent ("ignore previous instructions and…").
  • Issue text. An attacker posts a crafted issue that the assistant reads.
  • Documentation pages. An assistant that browses vendor docs for context can be manipulated through injected content.

Treat prompt injection as a specific incident category (tagged sev-2 and external-manipulation) and log it separately so trend analysis is possible.

Communications

External communications after an AI-adjacent incident are owned by Comms (with sponsor approval for SEV-½). Principles:

  • Be accurate and non-promissory. Don't claim no data was affected until the evidence supports it.
  • Address real mitigations. "We have improved our processes" is empty. "We have moved SCA to a pre-merge gate" is real.
  • Acknowledge the AI dimension when it is material; do not attempt to obscure it. Modern disclosure expectations treat AI as a standard cause, not a novelty.
  • Separate technical incident response from sales / comms narratives. Avoid making the mitigation a product pitch for the vendor whose tool failed.
  • Breach notification per GDPR Article 33/34, HIPAA, state data breach laws, DORA for EU financial services. An AI data exposure is a data exposure.
  • Customer commitments. The Air Canada chatbot decision (Moffatt v. Air Canada, 2024 BCCRT 1495) established that a company is bound by its AI system's commitments. Customer-facing AI incidents may trigger contractual remedies, not just goodwill.
  • Litigation hold. When a claim is foreseeable, Legal initiates preservation across AI tool logs and prompts.
  • Regulator notification. Sector-specific; have the contact list pre-identified.

Metrics and reporting

  • Incident count by severity, monthly.
  • Mean time to detect (MTTD) for AI-adjacent incidents. Typically the area for biggest improvement in year 1.
  • Mean time to mitigate (MTTM).
  • Recurrence rate. The same root cause more than once indicates insufficient mitigation.
  • Training coverage vs. incident rate: lagging correlation.
  • Public-post-mortem ratio for incidents where external communication occurred.

Anti-patterns

Don't

  • Don't run the AI tabletop once and never again. 6-month cadence is the floor.
  • Don't exclude Legal and Comms. Their obligations are often the tighter timeline.
  • Don't let "the AI did it" become the root cause. AI systems don't have agency in the blame sense, but they are a cause category, and the right root cause is the policy / control that permitted the agent to act that way.
  • Don't treat vendor-side incidents as "their problem." Your data, your customers, your obligations.
  • Don't forget near-misses. The near-miss log is the richest prevention data you will have.

External references

  • NIST SP 800-61r3, Computer Security Incident Handling Guide6.
  • OWASP Top 10 for LLM Applications4.
  • MITRE ATLAS, adversarial-ML taxonomy7.
  • CISA guidance on AI incident reporting8 (evolving; check for updates).

Next

  1. Replit AI agent deletion incident, Jul 2025. See Evidence: security incidents. ↩

  2. Wiz Research, "Wiz Research Uncovers Exposed DeepSeek Database", Jan 2025. ↩

  3. See, for example, the publicly discussed Secure Vibe Coding Guide pattern and related community writing on the "security by omission" phenomenon in AI-assisted code generation. ↩

  4. OWASP, Top 10 for LLM Applications. ↩↩

  5. Moffatt v. Air Canada, 2024 BCCRT 149, decision online. ↩

  6. NIST, SP 800-61 Revision 3, 2024. ↩

  7. MITRE, ATLAS Adversarial Threat Landscape for AI Systems. ↩

  8. US CISA, AI Roadmap. ↩