Where Personal Data Actually Leaks in LLM Apps (GDPR & HIPAA)

Where Personal Data Actually Leaks in LLM Apps (GDPR & HIPAA)

Most teams building AI features have the basics: a privacy policy, encrypted databases, access controls. Then they add an LLM call — and quietly create a new path for personal data to leave their control, often without anyone mapping it.

If you handle personal data under GDPR, or protected health information under HIPAA, an LLM workflow is a new data flow that your existing compliance posture may not cover. Here's where the leaks actually happen, and what the regulations require you to do about each.

The Core Shift: Data Leaves Your Perimeter

Traditional data protection assumes your data lives in systems you control. You encrypt it, you limit access, you log who touches it.

An LLM call breaks that assumption. Every time a workflow sends text to a third-party model, a copy of whatever is in that text leaves your perimeter and enters the provider's. Under GDPR, you've just transferred personal data to a processor. Under HIPAA, if that text contains PHI, you've just disclosed it to a third party. Both regimes have specific requirements the moment that happens — and most of the compliance gaps we see are teams that never registered the transfer as a transfer.

Leak Point 1: The Prompt Payload

The most direct leak is the obvious one: personal data in the prompt itself.

A support-ticket summarizer sends the full ticket — including the customer's name, email, and whatever they typed — to the model. A resume screener sends the candidate's resume. A medical-notes assistant sends the clinical note. In each case, personal or health data is in the payload.

What the regulations require:

• GDPR: A lawful basis for the processing, a data processing agreement with the provider, and — if the provider is outside your region — a valid transfer mechanism. Data minimization also applies: are you sending fields the model doesn't need?
• HIPAA: A Business Associate Agreement (BAA) with the provider before any PHI is sent. Many model providers offer a BAA only on specific plans or endpoints — using the wrong one is a violation regardless of how the data is handled after.

The question: For each workflow, what personal data is in the payload, and do we have the agreement that makes sending it lawful?

Leak Point 2: Retrieved Context (RAG)

RAG systems retrieve documents and inject them into the prompt as context. If those documents contain personal data, every query potentially sends personal data to the model — even when the user's query itself is innocuous.

This leak is easy to miss because the data isn't in the user's input; it's pulled in automatically from your document store. A team that carefully scrubbed the user-facing input can still be shipping personal data through the retrieval layer.

The question: Does our retrieval corpus contain personal data, and does it end up in prompts?

Leak Point 3: Logs and Traces

Good observability means logging the rendered prompt and the model's output. But if the prompt contains personal data, your logs now contain personal data — often in systems with looser access controls and longer retention than your primary database.

This is a genuine tension: you need to log prompts to debug and monitor AI workflows, but those logs are a new copy of personal data. Under GDPR, logged personal data is subject to the same retention limits and deletion rights as any other copy. A data subject's deletion request has to reach your prompt logs too.

The question: Do our AI logs contain personal data, who can access them, how long are they kept, and can we delete a specific person's data from them?

Leak Point 4: Training and Retention by the Provider

Even with a data processing agreement, the provider's default data-use policy matters. Some providers retain API inputs for a period; some use them to improve models unless you opt out or use a specific enterprise tier.

If personal data is retained or used for training without the right agreement and configuration, that's a compliance gap you own — not the provider's. The provider gave you the settings; using the wrong ones is on you.

The question: For each provider we use, what's the retention and training policy on the specific plan and endpoint we're calling, and have we configured it correctly?

Leak Point 5: The Model's Output

Sometimes the leak is in what comes back. A model with access to one user's data in context can, through error or manipulation, surface it to another user. Or a model trained or fine-tuned on personal data can regurgitate it.

This connects to the "sensitive information disclosure" risk in the OWASP LLM Top 10 — and it's the leak customers worry about most: "will your AI show my data to someone else?"

The question: Can data belonging to one user ever surface in output shown to another?

What "Compliant" Actually Requires

Putting it together, a defensible position on personal data in AI workflows means being able to show, for every workflow:

1. What personal data it touches — in the prompt, in retrieved context, and in logs.

2. Which providers it goes to, and that you have the right agreement (DPA under GDPR, BAA under HIPAA) with each.

3. That the provider's retention and training settings are configured correctly for the plan you're on.

4. That deletion rights reach every copy — including prompt logs.

5. That output can't disclose one person's data to another.

The reason this is hard isn't the rules — it's that you can't answer any of it without first knowing every workflow that touches personal data. That inventory is the foundation. Once you have it, each question becomes a checklist item instead of an investigation.

Scan your codebase to inventory every AI workflow and see which ones touch data that carries GDPR or HIPAA obligations. Or read more about compliance readiness.