Skip to content

Principles for safe worker-built apps

This framework rests on eight principles: enable workers to build instead of banning it; classify each app by what it does, not how it was built; make the safe path the fast path; give every shared app an owner who isn't only the builder; treat verification as a property of the system, not a personal virtue; give agents and tools the least privilege, enforced by the platform; measure outcomes, not feelings; and plan an app's end at its start. Each principle is backed by the evidence linked below, and each rules out a documented failure.

Who this is for

Leaders and practitioners deciding whether to trust this framework, and anyone citing it. You'll get the reasoning behind the rules on every other page. Reading time: about 7 minutes.

If two recommendations conflict, the principles decide; if a principle proves wrong, the pages built on it change.

1. Should a company ban AI app building? Enable, don't ban

Why. Bans hide AI use; they don't stop it. In a 2024 survey of privacy and security professionals, 27% said their organization had banned generative AI at least temporarily, and 48% of the professionals surveyed admitted entering non-public company information into such tools.1 By 2026, vendor surveys found between about half and two-thirds of office workers using AI tools they believed weren't allowed.2 A well-known 2023 ban was later followed by a broad rollout of approved tools (all staff in Korea and one global division), according to the vendor's announcement.3

This principle is a policy stance: don't prohibit AI-assisted building (for the term most people search for, see what vibe coding means for a business). Principle 3 is about how the approved route is designed.

What it rules out. AP12: the ban-and-hope AI policy (failure patterns).

Where it's applied. Should staff build apps with AI?, the acceptable use policy and the unsanctioned AI use policy.

Don't: start with a ban

A ban without an approved alternative (AP12) pushes building into personal accounts and unapproved tools, out of sight.1 Offer an approved tool and a private sandbox first, then restrict what's genuinely too risky.

2. Classify by what the app does, not how it was built

Why. The harm an app can do comes from its data, its users, its reach and its actions, not from whether AI wrote it. In North America, the legal duties mostly come from effects too, such as handling personal data or deciding things about people, rather than from AI-assisted building as such. California's automated-decisionmaking rules are one example: they key on using technology to make significant decisions.4 The survey is on US and Canadian laws; EU rules, such as the AI Act's AI-literacy duty for deployers, reach businesses with an EU establishment or whose AI output is used there. The UK National Cyber Security Centre's June 2026 guidance on AI-assisted building also scales scrutiny with the stakes, not with the tool.5

What it rules out. Treating every AI-built app as high risk, which drives builders around the process, and treating AI-built apps as safe because a platform generated them (AP3: "the vendor secured it").

Where it's applied. Every worker-built app sits in one of four risk tiers (Sandbox, Team, Important or Critical), set by the highest answer to six questions about its data, users, internet reach, dependence, decisions about people and actions, and two floors apply at every tier. See the six tier questions.

3. Make the safe path the fast path

Why. People route around friction. Vendor platform guidance starts from a low-power default (private workspaces, sharing limits, new connectors off) with a quick route to more,6 and platform research finds that adoption comes from pull, not mandates.7 If the approved route is slower than an unapproved tool, builders take the unapproved tool.

What it rules out. Approval queues measured in weeks, and paved roads nobody chooses.

Where it's applied. The paved road, intake with its five-working-day target (a target we'll measure, A-030), and the builder handbook.

4. Every shared app has an owner who isn't only the builder

Why. Worker-built apps outlive their builders' attention and employment. OWASP's citizen development risk list names abandoned apps that stay live, unpatched and connected as a distinct risk.8 Research on internal developer platforms found that "clear ownership" was the outcome teams most wanted, though that research comes from a vendor.9

What it rules out. AP19: orphaned apps.

Where it's applied. App ownership, when the builder leaves and the app registry.

5. Verification is a system property, not a personal virtue

Why. People don't reliably catch AI mistakes by reading. In a randomized study, participants using an AI assistant wrote less secure code and were more confident it was secure.10 Across model generations, about 40–45% of generated code samples on security-relevant tasks contained a flaw, and newer or larger models didn't fix it (vendor benchmark).11 So checks must be built into the process: tests that must pass, automated scans, and review depth that rises with the tier.

What it rules out. AP10: fluent-but-wrong code accepted as reviewed, and AP5: believing the agent's own report of what it did.

Where it's applied. Verify before release, test your app and the security checklist.

Don't: rely on the builder's confidence

"I checked it" isn't evidence. AP10 happens precisely when the person is confident.10 Require test cases that pass on each release and a recorded self-check from Team up, then a business review and technical check at Important, and independent tests at Critical.

6. Least privilege for agents and tools, enforced by the platform

Why. Many documented agent incidents come from standing permissions: an agent holding credentials it should never have had, then deleting data despite instructions.12 Others come from prompt injection, where text the agent reads tells it to send private data out.14 The MCP specification says hosts must get explicit user consent before invoking a tool and should treat tool descriptions as untrusted, and it admits the protocol itself can't enforce this.13 So enforcement has to come from the platform: scoped credentials and approval gates, not policy text.

What it rules out. AP4: an agent with standing production access; AP6: the lethal-trifecta agent; AP8: permission-bypass as default.

Where it's applied. The five never-dos, coding agents, MCP and agents and guardrails.

7. Measure outcomes, not feelings

Why. Perceived speed is a poor guide to measured speed. In a 2025 randomized trial, experienced developers using AI took 19% longer on their tasks while believing they were about 20% faster.15 Satisfaction and self-reported time saved can't justify a program on their own.

What it rules out. AP22: trusting self-reported productivity.

Where it's applied. Measurement, baselining and citizen developer ROI.

Don't: report time saved as the result

Self-reported time saved is AP22. Measure a baseline before the app, then the process outcome after it, alongside a stability measure such as errors or rework.15

8. Plan the end at the start

Why. The UK government's AI Playbook asks for the full lifecycle to be managed, including closing a system down securely at the end of its useful life.17 Many AI initiatives do stop: in a 2025 survey, 42% of companies said they had abandoned most of their AI initiatives, up from 17% a year earlier.16 So many apps will be stopped, and each needs a clean way to stop. An app that was never designed to retire leaves behind live data, open URLs and forgotten grants.

What it rules out. AP20: pilot purgatory, with no owner or value measure; and AP19 again.

Where it's applied. Kill criteria at intake, the retire-by date in the app registry, and retire an app.

What this framework isn't

  • It isn't a certification. Following it doesn't make you conform to any law or standard. Where it lines up with frameworks such as the NIST AI RMF or ISO/IEC 42001, we say "aligned with" and show the mapping. Those frameworks structure governance; they aren't checklists you can pass.
  • It isn't size-tiered. It gives one recommendation for a typical business and names the specific circumstances that change it, such as handling regulated data or already running a platform team, never a headcount.
  • It isn't vendor-specific. It describes capabilities ("a secrets manager", "an AI app builder") and names products only as examples, with alternatives.

Assumptions and evidence

  • A-004 (Medium): we expect a ban without an approved alternative to hide AI use, not stop it. Test this by comparing discovered unapproved use before and after your approved tool launches.
  • A-018 (Medium): we expect four named tiers set by the highest answer to six questions to be enough to route support, review and graduation. Test this in the pilot: it's falsified if many apps don't fit a tier, tier disputes exceed a few per month, more than about 10% of registered apps land in Critical, or more than about 15% of Important and Critical answers are downgraded.
  • A-030 (Low): we expect intake answers to be given within five working days. Test this by tracking the median intake answer time in the pilot.
  • A-002 (Low): we expect builders to use a low-power default path if exceptions are answered within days. Test this by tracking exception wait times against unapproved tool discovery.
  • A-034 (Low): we expect ownership recorded as a role with a named successor to survive people leaving. Test this by counting owner roles with no current holder each quarter.
  • A-021 (Medium): we expect builders to review AI output reliably only when the process makes review visible. Test this by sampling recorded self-checks against defects found later.
  • A-016 (High): we expect agents with broad standing permissions to sometimes take destructive actions despite instructions. Test this by reviewing agent grants and near-misses in your pilot.
  • A-009 (Medium): we expect self-reported time saved to overstate measured gains. Test this by comparing a measured baseline with builders' estimates.
  • A-011 (Medium): we expect a large share of AI initiatives to stop before sustained use. Test this by tracking how many approved pilots reach production use.

Evidence: a mix of randomized studies, government guidance and policy, standards bodies, and vendor surveys and benchmarks (labelled); there's no controlled study of a worker-built app program as a whole. Facts on this page are as of October 2026.

  1. Cisco, 2024 Data Privacy Benchmark Study release, Jan 2024 [vendor], https://newsroom.cisco.com/c/r/newsroom/en/us/a/y2024/m01/organizations-ban-use-of-generative-ai-over-data-privacy-security-cisco-study.html, accessed 2026-10-21. An older baseline, still the clearest measure of bans versus behavior. ↩↩

  2. PagerDuty, survey release on unauthorized AI tool use, 11 Jun 2026 [vendor], https://s206.q4cdn.com/635206389/files/doc_news/PagerDuty-Report-Finds-Two-Thirds-66-of-Office-Professionals-Have-Used-Unauthorized-AI-Tools-at-Work-2026.pdf, accessed 2026-10-21; and The Register on an Okta survey, 27 May 2026 [news], https://www.theregister.com/ai-ml/2026/05/27/bosses-blinded-by-confidence-about-shadow-ai-use-by-workers/5247275, accessed 2026-10-21. ↩

  3. Bloomberg, "Samsung bans ChatGPT and other generative AI use by staff after leak", 2 May 2023 [news], https://www.bloomberg.com/news/articles/2023-05-02/samsung-bans-chatgpt-and-other-generative-ai-use-by-staff-after-leak; and OpenAI, Samsung Electronics deployment announcement, Jun 2026 [vendor], https://openai.com/index/samsung-electronics-chatgpt-codex-deployment, accessed 2026-10-21. ↩

  4. California Privacy Protection Agency, approved regulations on automated decisionmaking technology and risk assessments, effective 1 Jan 2026 [regulator], https://cppa.ca.gov/regulations/pdf/ccpa_updates_cyber_risk_admt_appr_text.pdf, accessed 2026-10-21. The rules key on the use of technology to make significant decisions, not on how software was built. ↩

  5. UK National Cyber Security Centre, "The vibe coding spectrum: an approach to AI-assisted software development", 18 Jun 2026 [primary], https://www.ncsc.gov.uk/blogs/the-vibe-coding-spectrum-approach-to-ai-assisted-software-development, accessed 2026-10-21. ↩

  6. Microsoft, Power Platform adoption guidance on securing the default environment, Aug 2026 [vendor], https://learn.microsoft.com/en-us/power-platform/guidance/adoption/secure-default-environment, accessed 2026-10-21. Cited for the pattern, not the product. ↩

  7. CNCF TAG App Delivery, "Platform engineering maturity model", Oct 2023 [research], https://tag-app-delivery.cncf.io/whitepapers/platform-eng-maturity-model/, accessed 2026-10-21. Older, but the finding that adoption follows value is a durable platform principle. ↩

  8. OWASP Citizen Development Top 10, "CD-SEC-09: Asset Management Failures" [primary], https://owasp.github.io/www-project-citizen-development-top10-security-risks/content/2022/en/CD-SEC-09-Asset-Management-Failures, accessed 2026-10-21. ↩

  9. CNCF, Backstage insights summary (research originated by Spotify), 2023 [vendor], https://contribute.cncf.io/techdocs/analyses/2023/backstage/insights-summary, accessed 2026-10-21. ↩

  10. Perry, Srivastava, Kumar and Boneh, "Do Users Write More Insecure Code with AI Assistants?", ACM CCS 2023 [research], https://arxiv.org/abs/2211.03622, accessed 2026-10-21. A 2022-era model; the overconfidence finding is a human-factors result that still holds. ↩↩

  11. Veracode, "GenAI Code Security Report", Jul 2025 [vendor], https://www.veracode.com/resources/analyst-reports/2025-genai-code-security-report/, accessed 2026-10-21; and Pearce et al., "Asleep at the Keyboard?", IEEE S&P 2022 [research], https://arxiv.org/abs/2108.09293, accessed 2026-10-21. ↩

  12. The Register, report on a coding agent deleting a production database (Replit and SaaStr), 21 Jul 2025 [news], https://www.theregister.com/2025/07/21/replit_saastr_vibe_coding_incident/, accessed 2026-10-21. ↩

  13. Model Context Protocol, specification version 2026-07-28 [primary], https://modelcontextprotocol.io/specification/2026-07-28, accessed 2026-10-21. ↩

  14. Simon Willison, "The lethal trifecta for AI agents", 16 Jun 2025 [primary], https://simonwillison.net/2025/Jun/16/the-lethal-trifecta/, accessed 2026-10-21. It collects documented prompt-injection data-exfiltration cases. ↩

  15. METR, "Measuring the impact of early-2025 AI on experienced open-source developer productivity", Jul 2025 [research], https://metr.org/blog/2025-07-10-early-2025-ai-experienced-os-dev-study/, accessed 2026-10-21. Sixteen developers on their own repositories; METR's Feb 2026 follow-up suggests speedups have grown but couldn't measure them reliably, https://metr.org/blog/2026-02-24-uplift-update/. ↩↩

  16. CIO Dive, report on S&P Global Market Intelligence survey data on abandoned AI projects, Mar 2025 [news], https://www.ciodive.com/news/AI-project-fail-data-SPGlobal/742590/, accessed 2026-10-21. ↩

  17. UK Government, "Artificial Intelligence Playbook for the UK Government", 10 Feb 2025 [primary], https://www.gov.uk/government/publications/ai-playbook-for-the-uk-government/artificial-intelligence-playbook-for-the-uk-government-html, accessed 2026-10-21. ↩