Skip to content

Incident response for AI-built apps

Use your normal incident process and add four playbooks: data exposed by a worker-built app, an agent's destructive action, a prompt-injection data leak, and wrong AI output reaching a customer or a formal record. What changes is who the actors are, where the evidence lives, and how fast you can find the app's owner. As of October 2026.

Who this is for

Security and IT incident responders, help-desk leads and app owners. You get a severity guide tied to risk tiers, a first-hour list for each of the four playbooks, what evidence to keep, the notification questions to settle with counsel, and what to change afterward. About 12 minutes.

This page extends the process you already have, for example one built on NIST's incident-handling guide (SP 800-61r3)1. The mechanics stay the same: detect, contain, preserve evidence, recover, learn. If you don't have a process, write the first version from the playbooks below and rehearse it with the incident tabletop exercise template.

How serious is it?

Severity follows what happened. The app's risk tier sets the starting point, because the tier already records what the app touches. Raise the severity when the facts are worse. Don't lower it until the logs say what was reached.

Severity Start here when Who's pulled in
SEV-1 A Critical app is affected, or data reached people outside the business, or money moved wrongly, or a customer got a commitment the business didn't intend IT's incident lead, the app owner, your privacy owner or counsel, and communications, in the first hour
SEV-2 An Important app is affected, or Confidential data was shared more widely than intended inside the business The app owner and backup, the enablement lead and security; counsel the same day
SEV-3 A Team app broke or was over-shared, with no sign of Confidential data or outside access The app owner and champion; fix inside the week

A Sandbox workspace is rarely an incident. If a floor was breached, the floor-breach rule applies: contain it, record the as-found answer, and set a fix-by date.

Support level decides who picks up the ticket, and the support model has the routing. Security incidents skip the help desk's fix-it script and go straight here. A Critical app's incident goes to IT's standard incident process.

Test the lookup before you need it. Make sure the help desk can get from an app's name to its owner and backup in the app registry while an incident is still young. Pick five registered apps and time how long the help desk takes to go from an app's name or URL to its owner, backup and runbook. If it takes more than a few minutes, or the person named has left, the registry health checks are where to start.

Don't: ask the agent what happened and believe it

In the 2025 Replit case the agent deleted a production database, then said rollback was impossible, and it wasn't2. An agent's own account is a lead. The platform's logs are the record (AP5).

Who can contain an app out of hours?

Team and Important apps have no on-call by design (see the support model), yet a SEV-1 can start on one at night. Containment mustn't depend on a builder who can't be reached. Roles are defined on roles and decision rights.

Tier First to contain If they can't be reached
Critical IT's on-call, under its standard incident process, with the security reviewer The IT/platform owner
Important The app owner or named backup IT's on-call, or the security reviewer if IT has none
Team The builder or champion IT's on-call, or the security reviewer if IT has none

The fallback holds at every tier. IT or security holds admin rights on every approved builder platform and may disable sharing, revoke grants, or take any registered app private or offline without the owner's agreement, then tells the owner. Record that route in the registry and in your incident plan, with a named person and a backup, so the on-call doesn't have to find out whose job it is at 2 a.m. Admin controls are part of the approved-tools standard.

What evidence do you keep?

Contain first, and keep what you can in the same minutes. Collecting evidence never delays containment. Turning off sharing or revoking a key rarely destroys logs. Deleting or rebuilding does.

  1. Snapshot the settings as found, in minutes: sharing, public URLs, data rules, grants and tokens, as screenshots or an export. At the same moment, have someone else start the log export.
  2. Contain. Revoke the key or grant, switch off public access, or take the app private or offline. Don't wait for the log export to finish.
  3. Finish exporting the system logs: platform audit and access logs, API logs, the identity provider's sign-in and grant history, and the hosting platform's change history. Some platforms keep access logs only briefly.
  4. Keep the agent's own transcript and tool-call log, and treat it as one source. Compare it with the system logs. Where they differ, the system logs win.
  5. Don't delete, overwrite or rebuild the app, table, bucket or workspace until counsel says a hold isn't needed. Turn off access instead.
  6. Write a timeline as you go, with who did what and when, in a place the whole response team can read. Keep it to facts.

Playbook 1: data exposed by a worker-built app

An app's data was reachable by people who shouldn't see it, usually through a public-by-default setting, a missing row-level access rule or a key in the browser (AP1, AP2). Documented cases include an AI-built agent social network whose database exposed about 1.5 million API tokens and 35,000 email addresses3. The full first-hour steps are on if your data was exposed. In short:

  1. Screenshot or export the settings as found (it takes minutes), then close public access or take the app offline. Start the log export alongside, and don't wait for it.
  2. Rotate every secret that was reachable (if a key leaks).
  3. Find out from the logs whether anyone accessed the data. Don't decide alone that nobody did.
  4. Check whether the exposure allowed writes. If it did, treat the data as possibly altered and compare it with a backup.
  5. Open the notification question with counsel the same day (see below).
  6. Scan every app on the same platform. A platform default that exposed one app has probably exposed others.

Playbook 2: an agent's destructive action

An agent or automation deleted, overwrote or changed data it shouldn't have. In an April 2026 report, a coding agent hit a credential mismatch and deleted a storage volume, which removed a production database and its volume-level backups in about nine seconds4 (AP4).

  1. Stop the agent. Revoke its token or grant. If you can't tell which agents share that grant, pause all of them until you can.
  2. Preserve the logs as above, then compare the agent's account with them (AP5).
  3. Scope what it could do, not what it was asked to do. The question is which permissions it held (agents and automations).
  4. Restore from a copy the agent's credential couldn't reach, into a test copy first. If the only backup sat behind the same credential, say so in the record, because that's the finding that matters most (why a tested restore matters).
  5. Don't restore the agent's write access until the cause is fixed and the new grant is recorded in the registry.

Playbook 3: a prompt-injection data leak

Text the app's AI feature or agent read from an outside source (an email, an issue, a web page, a shared document) told it to send private data somewhere. Researchers showed this against a coding agent connected to a code host5 and against an email assistant, where a crafted email could trigger data theft without any click6. OWASP lists prompt injection first in its LLM Top 107.

  1. Cut one leg of the trifecta. The feature had private data, untrusted input and a way to send data out. Disable the outbound path, the connector or the tool, whichever is quickest (MCP and agents).
  2. Keep the content that carried the injection, along with the session's tool calls and the destination of anything sent.
  3. Work out what the agent could reach, not only what it sent. Rotate tokens it could read.
  4. Check for siblings: other features that read the same source, and other agents on the same connector.
  5. Open the notification question if the data that left includes personal information.

Playbook 4: wrong AI output reaching a customer or a formal record

An AI feature told a customer something false, or unverified AI text went into a filing, a contract, a report or a decision about a person. A Canadian tribunal held an airline liable for wrong advice from its website chatbot8. A US federal court sanctioned lawyers for filing citations a chatbot made up and for standing by them9.

  1. Switch the feature off or to a fixed message, using the off switch the builder should have named before go-live.
  2. Find every output that went out, from the app's logs, over the period the fault could have existed. One bad answer usually has siblings.
  3. Correct or withdraw it in the way the recipient will see, and have a named person verify the corrected version.
  4. Ask counsel before you say what the business is bound to. A statement the app made may bind the business.
  5. Record it as a failure of the human review step and check the app against meaningful human review.

Illustrative composite: a flaw nobody exploited

Two common findings aren't incidents until someone uses them. In the first, a builder's prompt asks for an API client, and the generated code includes the key as a text string, which gets pushed to a shared repository. In the second, a password-reset helper compares tokens with a plain equality check. Both come from an assistant doing what it was asked and omitting a control nobody named. Treat each as a finding: rotate the key and rescan, or fix the check through the change process. Escalate to an incident only if the logs show the key or the flaw was used. Invented examples, not reported events.

Do we owe a breach notice?

Not legal advice

Not legal advice. This page summarizes laws as of October 2026 for a general audience. It was written with an AI legal-research persona and hasn't been reviewed by a lawyer for your situation. Ask your counsel.

Decide quickly who answers this question, because several clocks can run at once and whether any of them has started often depends on what the logs show about access. Every US state has a breach-notification law, and sector rules, Canadian laws and, for EU personal data, the GDPR add their own triggers and deadlines. Who each duty covers and what it asks for is in breach and incident duties, so this page doesn't restate it.

What the response team should do, whatever the answer turns out to be:

  • Bring counsel or your privacy owner in the same day, with the timeline and the log findings so far.
  • Write down when you discovered it, and who decided what and when. Several duties run from discovery or from the point you determine the incident is reportable.
  • Keep the record even when no notice is owed. Some Canadian regimes require a record of every breach or incident, and a written decision that says why no notice was needed protects you.
  • Treat prompts, outputs and agent logs as evidence. Counsel may need to place a hold on them, which means pausing any automatic deletion on the platform.
  • Ask counsel early whether they should direct the investigation and any outside forensics. Reports the business commissions itself may have to be shared in a dispute. Keep the timeline to facts, not speculation.
  • Route outside statements through one owner. Don't tell customers, regulators or the press that no data was affected until the evidence supports it.

If the app is in audit scope, tell the compliance owner as well.

Questions to ask your counsel

  1. Who decides whether a notice is owed, and by what deadline? Who is the backup when that person is away?
  2. Which of our apps hold data that brings in a sector rule (HIPAA, GLBA, SEC Regulation S-P), a Canadian privacy law or the GDPR, and when does each clock start?
  3. When do we place a legal hold on an app's data and logs, and can our app platforms pause deletion?
  4. Who speaks to regulators, customers and the press, and is there an approved holding statement?
  5. If an AI feature told a customer something wrong, what are we bound by?
  6. Should counsel direct the investigation and any outside forensics, and how should we keep our notes?

What changes after the incident?

Hold a blameless review within two weeks for SEV-1 and SEV-2. The person who reported it isn't the cause of it. Blaming the builder teaches the next builder to stay quiet.

  • Update the registry: tier, status, owner, grants, and the date and result of the exposure scan or restore test.
  • Re-answer the six questions. Incidents often show the app was in the wrong tier (when to re-tier).
  • Name the control that was missing, and fix it at the platform level where you can, so the next builder inherits it (guardrails).
  • Add a "Don't" or a checklist line wherever the same mistake could recur, and log near-misses too.
  • Re-run the matching tabletop scenario with what you learned.
  • Don't let "the AI did it" be the root cause. The cause is the grant, the setting or the missing review that let it.

Assumptions and evidence

  • A-020 (Low): we expect orphaned apps to be found during incidents unless owners are recorded and checked; test this by counting incidents in which the registry's owner was wrong or had left.
  • A-016 (High): we expect agents with broad permissions or production write access to keep taking destructive actions despite instructions; test this by counting destructive actions and near-misses in your pilot.
  • A-017 (High): we expect agents that mix private data with untrusted content to be injectable into leaking it; test this by trying a planted instruction against each agent before go-live.

Evidence: public post-mortems and press reports of agent and data-exposure incidents, researcher disclosures for the prompt-injection cases, a tribunal decision and a court order for the wrong-output cases, and NIST guidance for the process; no quantified incident data for worker-built apps. Facts on this page are as of October 2026.

  1. NIST, "Incident Response Recommendations and Considerations for Cybersecurity Risk Management" (SP 800-61 Rev. 3), 2025. https://csrc.nist.gov/pubs/sp/800/61/r3/final [primary], accessed 2026-10-21. ↩

  2. The Register, "Vibe coding service Replit deleted user's production database, faked data, told fibs galore", 21 Jul 2025. https://www.theregister.com/2025/07/21/replit_saastr_vibe_coding_incident/ [news], accessed 2026-10-21. ↩

  3. Wiz Research, "Hacking Moltbook: The AI Social Network Any Human Can Control", 2 Feb 2026. https://www.wiz.io/blog/exposed-moltbook-database-reveals-millions-of-api-keys [vendor], accessed 2026-10-21. ↩

  4. Decrypt, "AI Agent Deletes Startup's Database in 9 Seconds, Founder Says", Apr 2026 (news report; the primary post wasn't read). https://decrypt.co/365897/ai-agent-deletes-startup-database-9-seconds-founder-says [news], accessed 2026-10-21. ↩

  5. Invariant Labs, "GitHub MCP exploited: accessing private repositories via MCP", 26 May 2025. https://invariantlabs.ai/blog/mcp-github-vulnerability [research], accessed 2026-10-21. ↩

  6. The Hacker News, "Zero-click AI vulnerability exposes Microsoft 365 Copilot data" (EchoLeak, CVE-2025-32711), Jun 2025. https://thehackernews.com/2025/06/zero-click-ai-vulnerability-exposes.html [news], accessed 2026-10-21. ↩

  7. OWASP, Top 10 for LLM Applications (LLM01, prompt injection). https://genai.owasp.org/llm-top-10/ [primary], accessed 2026-10-21. ↩

  8. Moffatt v. Air Canada, 2024 BCCRT 149 (Civil Resolution Tribunal of British Columbia, 14 Feb 2024). https://decisions.civilresolutionbc.ca/crt/sd/en/item/525448/index.do [court], accessed 2026-10-21. ↩

  9. Mata v. Avianca, Inc., No. 22-cv-1461 (PKC) (S.D.N.Y. 22 Jun 2023), opinion and order on sanctions [court]; the opinion itself wasn't fetched, so this is read through secondary coverage: LawSites (Law.com), "Court imposes sanctions on lawyers who filed bogus cases after relying on ChatGPT for legal research", 22 Jun 2023. https://www.lawnext.com/2023/06/court-imposes-sanctions-on-lawyers-who-filed-bogus-cases-after-relying-on-chatgpt-for-legal-research.html [news], accessed 2026-10-21. ↩