Skip to content

Data handling

Most publicly-reported AI incidents in software engineering are, at their root, data-handling incidents. Samsung banned ChatGPT after engineers pasted source code into the public model1. The DeepSeek ClickHouse exposure was a vendor-side data-handling failure that customers had to respond to on their own time2. Chat histories being used for training have surfaced in lawsuits and consumer complaints repeatedly.

Data-handling rules exist so these failures are foreseeable instead of surprising.

The data-class matrix

Class Examples Allowed in approved enterprise tool? Allowed in public / consumer tool?
Public Published marketing, published documentation Yes Yes
Internal Non-sensitive internal code, non-PII internal docs Yes No
Confidential Crown-jewel source code, pricing models, strategy docs Yes (enterprise tenant only) Never
Restricted Customer PII, regulated data (PHI, PCI, classified), authentication material (secrets, tokens, keys) Conditional (see below) Never

The Restricted class

Restricted data is the default-deny class. It flows to AI tools only under explicit addenda:

  • Customer PII: no AI tool unless an enterprise tenant with BAA (for PHI), SCCs (for EU cross-border), or equivalent is in place and the tool documents no training on tenant data.
  • Secrets, tokens, keys: never, regardless of tool. If they need to be tested, use synthetic equivalents.
  • Regulated categories, per sector rules:
  • PHI (US health): HIPAA BAA required; many vendors do not support this for AI coding assistants at Restricted volumes.
  • PCI cardholder data: typically prohibited from AI tool exposure entirely; use synthetic test data.
  • Classified / CJIS / export-controlled data: prohibited except in specifically-accredited environments (e.g., GovCloud equivalents).

Prompts as records

Prompts are company records. They may be:

  • Attributable to an engineer (user metadata).
  • Traceable to a PR or work item.
  • Retained beyond the engineer's memory.

Treat prompts the same way you treat other engineering artifacts for the purposes of retention, audit, and discovery.

Purpose Retention default Owner
Troubleshooting / incident reproduction 30–90 days DevEx
Audit evidence Per regulatory retention (typically 1–7 years) Compliance / CoE
Training-data feedback Off by default; explicit opt-in per DR CoE

Prompt hygiene: what engineers must actually do

Hygiene is the behavioral layer of data handling. It lives in training materials and the AUP.

  • Never paste secrets or keys, even to "ask about format".
  • Prefer references over copies. "Describe the behavior of the function in auth/session.ts at line 142" beats pasting the function.
  • Redact before asking. Replace customer names, IDs, and tokens with placeholders.
  • When in doubt, don't. If uncertainty exists, stop and ask the AppSec or Privacy contact.
  • Report leaks immediately. Blame-free; reporting is required, not optional. See Incident response.

Cross-border and residency

AI services commonly process data in the vendor's chosen regions. For GDPR-regulated data, cross-border transfer requires a valid transfer mechanism:

  • Adequacy decision (e.g., EU–UK) where applicable.
  • Standard Contractual Clauses (2021 revision) or UK IDTA.
  • Binding Corporate Rules for intra-group transfers.
  • Derogations (consent-based, contract-necessity) are rarely appropriate for systematic engineering use.

For highly-regulated sectors, stricter rules apply (e.g., French health-data hosting requires HDS-certified providers; US federal use requires FedRAMP; DORA applies in EU financial services).

Document the data-flow map

For every approved tool, produce a data-flow map:

  • Where do prompts and completions travel (engineer → tool region → sub-processor regions)?
  • Which data classes may traverse each hop?
  • What legal mechanism covers each cross-border hop?
  • Who is the internal owner?

The map is reviewed annually (more often in sector regulation).

Retention

Align retention with classification and regulatory context:

Retention purpose Typical window Notes
Incident response 90 days Shorter if the tool's retention is shorter
Engineering records 1–2 years With the PR lineage
Regulatory (general) 3–7 years Per sector: PCI (3 years post-relationship), HIPAA (6 years), GDPR (varies by lawful basis)
Litigation hold Until released Separate workflow; Legal-initiated

Deletion and right to erasure

Where individuals can request deletion (GDPR Article 17, CCPA deletion, HIPAA rights):

  • The AI tool's retention and the company's internal retention must both honour the request.
  • Training-data pipelines that absorbed prompts before Phase 0 are a common blind spot. Document whether any such data needs to be unwound, and how.
  • Derivative data (e.g., fine-tuned models) may hold personal data in non-retrievable form; consult legal and document the position.

Anti-patterns

Don't

  • Don't treat "the code is ours, not personal data" as a license to paste anything. Secrets and keys are not personal data but are equally damaging to leak.
  • Don't rely on vendor "we don't train on your data" without contractual enforcement. The default in the MSA must match.
  • Don't ignore sub-processors. The tool vendor you signed may route to sub-processors whose terms you never reviewed.
  • Don't keep prompts forever "just in case". Retention has a cost (storage, legal exposure, privacy risk). Set it deliberately.

Worked examples

Example 1: An engineer asks about a customer payment failure

Prompt (bad): "Why did customer John Smith (user id 48293, card ending 4921) fail validation on 2026-03-14 at 14:02?"

Prompt (good): "Given our validator returns error code VALIDATOR_PATTERN_X for a transaction, walk me through the likely causes."

The first leaks PII and a partial PAN. The second achieves the same debugging goal without any restricted data.

Example 2: Reviewing a change in a restricted service

A critical-tier service is classified such that source code must not leave the enterprise tenant. The engineer's approved enterprise tool is permitted. The engineer's personal Claude or ChatGPT account is not, regardless of how convenient that might be.

Example 3: Cross-border contractor

A contractor in a region without adequacy uses an approved tool configured for EU-only processing. The contractor's prompts should be routed consistent with the data class; if the tool's regional controls are insufficient, the contractor is not approved for restricted data categories in this tool.

External references

  • EDPB guidance on cross-border transfers3.
  • UK ICO AI and data protection guidance4.
  • NIST AI RMF GenAI Profile, Govern 3 and Map 3 functions5.

Next

  1. Bloomberg, "Samsung Bans Staff's AI Use After Spotting ChatGPT Data Leak", May 2023. ↩

  2. Wiz Research, "Wiz Research Uncovers Exposed DeepSeek Database", Jan 2025. ↩

  3. EDPB, Guidelines on International Transfers. ↩

  4. UK ICO, AI and data protection. ↩

  5. NIST, NIST AI 600-1 GenAI Profile, Jul 2024. ↩