Skip to content

Secure AI features in business apps

An app that calls an AI model at runtime carries risks that an app merely built with AI doesn't. Anything the model reads can steer it (prompt injection: text in a document, email or web page that carries instructions the model then follows), its answers can be wrong and still sound right, and your data flows to the model provider. Design it so the model can't take consequential actions on its own, limit what it can read and send, disclose AI use where the law requires it, and gate launch on evals. As of October 2026, the main risk list for this is the OWASP Top 10 for LLM Applications, 2025 edition.1

Who this is for

Builders adding an AI feature, technical reviewers, and security leads. You'll get the threat model, the controls for each risk, and the legal points to raise. About 12 minutes.

Is this an AI feature, or an app built with AI?

These are two different things with different risks:

AI-assisted building
You use an AI tool to write or change the app. Once it's built, the app runs ordinary code. The risks are in the code and configuration, and the security checklist covers them.
AI feature
The app itself sends data to a model while it runs: a summary, a classification, a drafted reply, a chat answer, a search over your documents, an agent step. The risks on this page apply on top of the checklist.

Built-in SaaS AI features count. An AI formula in a spreadsheet or an AI step in a workflow tool is an AI feature when it processes the app's data at runtime. AI used only to write the app isn't. A built-in feature passes the approval gate when it's already on the approved-tools list for the data class involved; if it isn't listed for that class, it's blocked like any other unapproved service.

We expect most worker-built apps to contain no AI feature at all (A-007, a hypothesis). Record in the app registry which ones do, so the count is measured rather than guessed.

How do AI features change an app's risk tier?

Every worker-built app sits in one of four risk tiers (Sandbox, Team, Important or Critical), set by the highest answer to six questions about its data, users, internet reach, dependence, decisions about people and actions, and two floors apply at every tier. AI features add three modifiers (full wording):

Modifier When it applies What it does
M1. AI feature The app calls an AI model at runtime, including built-in SaaS AI features The tier is unchanged. Approval gate (every tier): if the app sends any data to a model or other service that isn't approved for that data class, it's blocked until that service is approved. In Sandbox, the approval gate and Floor 1 are the only M1 requirements. From Team up, add the AI-feature controls: an eval set with a pass threshold, an AI-feature review, the provider's data terms, and a disclosure check
M2. Customer-facing AI Model output reaches a customer, applicant or member of the public without a person approving each output Critical. A draft that a person reads and sends is not customer-facing AI
M3. Autonomous agent or automation It runs on a schedule or trigger, or chains tool calls, under a person's identity or a service identity The tier is unchanged, but Q1–Q6 are answered for every granted permission. Its identity, OAuth grants and tokens are recorded in the registry under the app owner's role, and they're revoked at retirement or when the owner changes

The trifecta rule applies at every tier, Sandbox included. No agent or AI feature may combine private data, untrusted content and a way to send data out (the "lethal trifecta").3 Remove one leg by design before it runs; don't rely on the model to refuse. The canonical explanation, the real-world cases and how to remove each leg are on the lethal trifecta (AP6).

What can go wrong? The OWASP LLM Top 10 risks for business apps

Use these seven OWASP 2025 entries as the review list for any AI feature.1 Of the other three, LLM08 (vector and embedding weaknesses) applies if the feature searches your documents, including files uploaded to a builder's knowledge base: make sure retrieval honours each user's access (see LLM02). LLM03 (supply chain) and LLM04 (data and model poisoning) matter mainly if you train or fine-tune models.

Risk (OWASP 2025) What it looks like in a worker-built app Controls
LLM01 Prompt injection Text in a document, email, web page or form field tells the model to ignore its instructions, reveal data or call a tool. "Indirect" injection arrives in content the app reads, not from the user2 Treat everything the model reads as untrusted. Give the model no permission the content's author shouldn't have. Keep tools read-only, or have a person approve each action. Test it with injection cases in your eval set
LLM02 Sensitive information disclosure The model repeats data from its context to someone who shouldn't see it, such as one customer's record shown to another Retrieve only what this user may see, enforced by the app's access rules, not by the prompt. Send the minimum data per call
LLM05 Improper output handling Model output is run as code, put into a database query, shown as HTML, or used as a link Treat output as untrusted input: check it against the expected format, escape it, never execute it. See input handling
LLM06 Excessive agency The feature can send, update, delete or pay without a person approving Scope tools to the task. A person approves each consequential action. Answer Q6 for every permission the model can reach
LLM07 System prompt leakage Users coax the model into printing its instructions, including any secrets or internal rules in them Put no secrets, keys or access rules in prompts. Assume the prompt is public
LLM09 Misinformation Fluent, confident, wrong answers: an invented policy, a wrong total, a fake citation Ground answers in your own sources and link them. Evals with expected answers and must-refuse cases. Show the source to the reader. Keep a human fallback
LLM10 Unbounded consumption A loop, a huge upload or abuse runs up the bill or exhausts quotas Per-app spend caps and alerts, rate limits, input size limits. See AI spend

Don't: rely on the system prompt to keep secrets or enforce access (LLM07)

Instructions such as "never reveal the discount codes" or "only answer about the user's own account" are requests to the model, not controls, and prompt injection gets around them.2 Hold secrets server-side (secrets), and enforce access in code before data reaches the model.

Why model-side defenses aren't enough. Vendors train models to resist injection, and that helps, but OWASP notes it isn't clear injection can be fully prevented.2 Design as if the model will sometimes follow the attacker, and rely on controls that don't depend on its judgment: permissions, approvals, data minimization and output checks.

Where does your data go?

Every call sends data to the model provider, and sometimes to its sub-processors. Before the first call, the M1 approval gate asks: is this service approved for this data class? The answer comes from the approved-tools list and its contract terms: training on your data, retention and staff access, sub-processors and data location, breach notice, audit rights and deletion. Then minimize. Send only the fields the task needs, strip identifiers where the task works without them, and never put data into a feature that isn't approved for its class (the four data classes).

Depends on your circumstances (C8, C9)

If you pay per token, or several AI services are already in use, a model-access layer (an AI gateway) can route every call through one place for logging, spend caps, data-loss checks and switching models. As of October 2026 we found no independent data on how widely businesses use gateways. The cheaper alternative is the provider's own organization controls: per-project keys, spend limits and usage exports. Start there, and add a gateway when you have several providers or need central data-loss checks.

What should a person check, and what's the fallback?

A person approves each output that commits the business, and the app still works, or fails safely, when the model is wrong or unavailable.

  1. Keep a person between the model and the consequence. Drafts, not sends. Suggestions, not updates. If a person approves each output before it reaches a customer, M2 doesn't apply.
  2. Make the review real. Show the reviewer the source the answer came from. A reviewer who can't check an answer is rubber-stamping it.
  3. Build the fallback. An "AI unavailable" path, a manual process the team still knows, and a switch the app owner can use to turn the feature off.
  4. Pin the model version, and plan for its retirement. Providers retire versions on published schedules, and the notice varies by provider and model type, so plan for the shortest (notice periods) and don't run preview models in production. Built-in SaaS AI features often can't be pinned at all (what to do then). Re-run your evals before any model change (the upgrade procedure).
  5. Log enough to investigate. For each call, keep a reference to the input, the prompt version, the model version, the output and what the person did with it. Logs inherit the data class of what was sent; set their retention and access to match. If the feature helps decide something about a person, store the record and the reasons at decision time. Re-running the model later won't reproduce the answer, and a model's explanation of its own output isn't its real reasons.

Don't: call a feature \"human-reviewed\" when nobody approves each output

Checking a sample, or reviewing complaints afterward, isn't approving each output. If output can reach a customer before a person approves it, the feature is customer-facing AI (M2), which is Critical. Label it honestly and tier it that way.

Don't cut the human fallback before you have quality evidence (AP21, on monitoring an AI feature).

What goes wrong with customer-facing bots?

Three public cases show the pattern: a bot that speaks for the business, with no grounding, no evals and no person checking.

  • A tribunal held a company liable for its chatbot (February 2024). In Moffatt v. Air Canada, the airline's website chatbot gave wrong advice about bereavement fares. The British Columbia Civil Resolution Tribunal rejected the argument that the chatbot was responsible for its own statements, and held the airline liable for the information on its site.4 It's a small-claims decision on negligent misrepresentation from one Canadian tribunal, so it isn't binding outside that tribunal, but its reasoning is widely cited.
  • A city's business chatbot gave unlawful advice (March 2024). New York City's MyCity bot told business owners that they could take a cut of workers' tips and that landlords needn't accept housing vouchers, both contrary to local law.5 The city moved to shut it down in January 2026.6
  • A dealer's chatbot was manipulated (December 2023). A chatbot on a car dealer's website agreed in chat to a $1 sale after manipulation; no sale resulted.7

Don't: launch a customer-facing bot without evals and a kill switch (AP14)

These bots were fluent and wrong. Before any output reaches a customer, run an eval set with the questions you'd least like it to get wrong and with injection attempts. Keep it to answers grounded in your own approved sources, hand off to a person when it's unsure, and give the app owner a switch to turn it off. Customer-facing AI without approval of each output is Critical. See AP14.

What must we disclose, and are we bound by what the AI says?

Not legal advice

Not legal advice. This page summarizes laws as of October 2026 for a general audience. It was written with an AI legal-research persona and hasn't been reviewed by a lawyer for your situation. Ask your counsel.

This section names the duties to raise. The law itself is on US and Canadian laws and the EU AI Act. Before you build, check the stop-and-call-counsel list, the site's single list of facts that stop a build until counsel has looked.

  • What the AI tells customers. In Moffatt, a tribunal held a business to its chatbot's statements.4 It isn't binding outside that tribunal, but the safe course is the same everywhere: treat bot answers as the business's own words, ground them in approved content, and log them.
  • What you claim about the AI. The FTC Act prohibits deceptive claims, and the FTC has acted against unsupported claims about AI, for example a final order in August 2025 requiring an AI-detection vendor to back its accuracy claims with evidence. The FTC's AI enforcement posture changed in 2025: in December 2025 it set aside one order from its 2024 AI sweep, but deception cases continue.8 A claim about accuracy, "human-level" quality or automation needs evidence. Your eval results are that evidence.
  • Telling people they're dealing with AI. As of October 2026, California (bots used online to sell, with intent to mislead about their artificial identity), Maine (any chatbot in trade that could make a reasonable consumer think it's a person) and Utah (when a consumer asks) require it for bots that deal with consumers, and from 1 January 2027 Colorado's HB 26-1263 adds age estimation, protections for minors and a self-harm protocol for publicly accessible chatbots. Who each covers, and the citations, are on chatbot disclosure laws. Where the EU AI Act reaches the feature, Article 50(1) has required telling people, staff included, that they're interacting with AI since 2 August 2026 (Article 50). A plain "you're chatting with an AI assistant" line is cheap. The M1 disclosure check applies from Team up; at Important, the review gate adds an AI-disclosure check when outside people use the feature or receive what it drafts.
  • Decisions about people. If the feature scores, ranks, flags, summarizes or recommends anything about a person's job, pay, credit, housing or similar, Q5 applies even if the model is "only advising". Apps that make decisions about people covers the extra tests and records.

Questions to ask your counsel

  1. Must we tell users they're dealing with AI, and how, in each place our users are? Does any public chatbot we run fall under Colorado's chatbot law, and which of its duties apply?
  2. Are we bound by what our AI feature tells a customer, and does any disclaimer change that?
  3. Which of our claims about AI features need substantiation, and is our eval evidence enough?
  4. Do our AI providers' terms on training, retention and staff access fit the data classes we plan to send?
  5. If a feature's output is used in the EU, are we a provider under the EU AI Act?

Assumptions and evidence

  • A-007 (Low): Hypothesis. We expect most worker-built apps to contain no AI feature at runtime; test this by counting M1 apps in the app registry after the pilot.
  • A-017 (High): We expect AI features and agents that combine private data with untrusted content to be open to prompt injection; test this by running injection cases against each such feature before launch.
  • A-037 (Low): Hypothesis. We expect a small eval set, including refusals and injection attempts, to catch most regressions; test this by tracking production incidents in AI features whose evals passed.
  • Evidence: a community risk list (OWASP), a tribunal decision, public incident reports and news coverage, provider documentation, and the statutes summarized on the linked legal pages. No controlled study measures how well these controls work in worker-built apps.
  • Facts on this page are as of October 2026.

  1. OWASP GenAI Security Project, Top 10 for LLM Applications 2025, https://genai.owasp.org/llm-top-10/ [primary], accessed 2026-10-21. ↩↩

  2. OWASP GenAI Security Project, LLM01:2025 Prompt Injection, https://genai.owasp.org/llmrisk/llm01-prompt-injection/ [primary], accessed 2026-10-21. ↩↩↩

  3. Simon Willison, The lethal trifecta for AI agents, 16 Jun 2025, https://simonwillison.net/2025/Jun/16/the-lethal-trifecta/ [primary], accessed 2026-10-21. ↩

  4. Civil Resolution Tribunal of British Columbia, Moffatt v. Air Canada, 2024 BCCRT 149, 14 Feb 2024, https://decisions.civilresolutionbc.ca/crt/sd/en/item/525448/index.do; also on CanLII, https://www.canlii.org/en/bc/bccrt/doc/2024/2024bccrt149/2024bccrt149.html [court; both pages refused our tools on 2026-10-21, so the holding is stated from law-firm and news coverage], accessed 2026-10-21. ↩↩

  5. The Markup, NYC's AI chatbot tells businesses to break the law, 29 Mar 2024, https://themarkup.org/news/2024/03/29/nycs-ai-chatbot-tells-businesses-to-break-the-law [news], accessed 2026-10-21. ↩

  6. The Markup, on the plan to shut down the MyCity chatbot, 30 Jan 2026, https://themarkup.org/artificial-intelligence/2026/01/30/mamdani-to-kill-the-nyc-ai-chatbot-we-caught-telling-businesses-to-break-the-law [news], accessed 2026-10-21. ↩

  7. The Decoder, People buy brand-new Chevrolets for $1 from a ChatGPT chatbot, 19 Dec 2023, https://the-decoder.com/people-buy-brand-new-chevrolets-for-1-from-a-chatgpt-chatbot/; AI Incident Database, report 6682, https://incidentdatabase.ai/reports/6682 [news; secondary sources only, no court or regulator record], accessed 2026-10-21. ↩

  8. 15 U.S.C. §45, https://uscode.house.gov/view.xhtml?req=granuleid:USC-prelim-title15-section45&num=0&edition=prelim [statute]; FTC, Operation AI Comply, 25 Sep 2024, https://www.ftc.gov/news-events/news/press-releases/2024/09/ftc-announces-crackdown-deceptive-ai-claims-schemes; FTC, final order against Workado, LLC, 28 Aug 2025, https://www.ftc.gov/news-events/news/press-releases/2025/08/ftc-approves-final-order-against-workado-llc-which-misrepresented-accuracy-its-artificial; FTC, Rytr final order reopened and set aside, 22 Dec 2025, https://www.ftc.gov/news-events/news/press-releases/2025/12/ftc-reopens-sets-aside-rytr-final-order-response-trump-administrations-ai-action-plan [regulator], accessed 2026-10-21. ↩