- Personal data flows through an AI system along four paths: into prompts, into retrieval corpora, into logs and traces, and out to a model provider. Each hop is a place PII can leak or leave your boundary, so treat privacy as a core design risk rather than a compliance afterthought.
- Control the provider boundary with no-train and data-processing terms you verify rather than assume, and self-host the most sensitive workloads; minimize and redact PII before it reaches a model or a log; scope retrieval and cap retention so records do not reach the wrong user or linger.
- The US backdrop leads with state privacy laws such as CCPA and CPRA and their peers, plus sector rules like HIPAA and GLBA; obligations shift, so build privacy controls into the platform and keep an explicit mapping from control to rule.
Privacy is a first-class AI risk
Most AI privacy incidents are not exotic. They are ordinary personal data arriving somewhere nobody designed it to go: a support transcript pasted into a prompt, a Social Security number captured verbatim in a trace, a customer record retrieved for the wrong account. AI systems are unusually good at absorbing this material because their entire value proposition is ingesting unstructured text, and unstructured text is where personal data hides best.
The structural problem is that an AI request is rarely a single hop. A prompt travels to a retrieval layer, gets enriched with documents, is packaged for a model provider, and is recorded in a trace on the way back. Each of those hops is a boundary, and each boundary is a place where personal data can be logged in the clear, retained past its usefulness, or shipped to a third party under terms nobody read closely. The model itself is often the least of the exposure; the plumbing around it carries more.
The reason to treat this as a first-class risk, rather than a checkbox handled late, is that the exposure compounds with the architecture. Retrieval widens the blast radius of a bad access-control decision. Tracing, which you rightly want for debugging, becomes a durable copy of every prompt. Agentic tool calling adds hops you did not have last year. Privacy that is bolted on after an incident tends to be a redaction filter taped to one endpoint while the other three keep leaking.
Where PII flows
You cannot protect what you have not traced. Before choosing any control, map the actual paths personal data takes through your system, because the exposure points are the hops between components, not the components themselves. In a typical retrieval-augmented deployment there are four such paths, and they tend to be owned by different teams, which is precisely why they leak.
Personal data enters at the prompt, whether typed by a user, pulled from an upstream form, or injected by an application. It enters the retrieval corpus when you index documents that contain personal records. It accumulates in logs and traces as you capture inputs and outputs for observability. And it leaves your boundary entirely on the provider call, where prompt plus retrieved context crosses to a third party. The diagram below marks each exposure point.
USER / UPSTREAM APP
| (1) prompt: PII typed or injected
v
+--------------+ (2) retrieval: records indexed
| ORCHESTRATOR |<----------- VECTOR STORE / CORPUS
+--------------+
| \
| \____ (3) trace/log: prompts + context
| stored for observability
v
MODEL PROVIDER (4) boundary crossing: leaves control
Two observations follow from the map. First, the paths are not independent: the same personal record can land in a prompt, get embedded into the index, and be written to a trace within a single request, so a naive fix at one point leaves the record exposed at three others. Second, the boundary crossing at point four is qualitatively different from the rest, because after it the data is governed by someone else's terms and retention. The next section takes that boundary on directly; the ones after it handle intake, tracing, and retrieval.
The provider boundary
The moment a prompt leaves your infrastructure for a model provider, the data is subject to that provider's handling, retention, and access practices rather than yours. For a public API this is the sharpest boundary in the whole system: you are trusting a contract and a configuration to govern personal data you no longer physically control. That trust is reasonable when it is verified and reckless when it is assumed.
Three controls, in increasing order of assurance, govern this boundary. A no-train agreement commits the provider not to train on your inputs and outputs, which addresses the fear that your customers' data becomes part of a future model. A data-processing agreement and the surrounding enterprise terms govern retention windows, sub-processors, access, and deletion, and they are what a regulator or auditor will ask to see. For the most sensitive workloads, self-hosting an open-weight model, or using a dedicated single-tenant deployment, keeps the data inside a boundary you operate end to end. Each step trades some capability or cost for control, and the right choice is per workload rather than global.
The recurring failure is not choosing the wrong control; it is assuming a control that was never actually in place. Consumer chat tiers and default API settings do not carry the same terms as negotiated enterprise agreements, and the difference is not visible in the code. Verify the terms that apply to the specific endpoint, tier, and region you call, and re-verify when the contract or the product changes.
Detection and redaction
The cheapest personal data to protect is the data that never arrives. The governing principle is data minimization: do not send a model, or write to a log, any personal data the task does not require. Everything else in this section is a technique for enforcing that principle when the raw material unavoidably contains PII.
Detection comes first, and it is imperfect. Pattern-based detectors catch structured identifiers such as card numbers, emails, and Social Security numbers well; free-text names, addresses, and health details need statistical or model-based recognizers, which trade recall against precision. Detect too little and PII slips through; detect too aggressively and you strip meaning the task needed. Run detection before the provider call and before the trace write, since those are the two hops where data most often escapes.
What you do after detection depends on whether the downstream task needs the real value, a stable stand-in, or nothing at all. The table sketches the common techniques and their trade-offs.
| Technique | What it does | Trade-off |
|---|---|---|
| Redaction / masking | Removes or blanks the value before it reaches the model or log | Simplest and safest; destroys any downstream use of the value |
| Pseudonymization | Replaces the value with a consistent token; a separate mapping can reverse it | Preserves references and joins; the mapping itself becomes sensitive |
| De-identification | Strips or generalizes identifiers so records no longer point to a person | Enables analytics; re-identification risk rises as fields combine |
| Data minimization | Never collects or forwards the field in the first place | Strongest protection; requires task-level discipline up front |
These are layers, not alternatives. A mature pipeline minimizes at intake, pseudonymizes the identifiers a task genuinely needs to carry, redacts the rest before the provider call, and de-identifies anything that flows into analytics or evaluation sets. Treat detection quality as a metric you monitor over time rather than a filter you install once, because the distribution of personal data in your prompts drifts as usage grows.
Retrieval, memory and retention
Retrieval turns a privacy question into an access-control question. When a model answers from a corpus of documents, the retrieval layer is effectively deciding which records a given user is allowed to see, and if it is not scoped to that user's entitlements, it will happily surface a sensitive record to the wrong person. On a shared vector store the classic failure is one tenant's query returning another tenant's documents because retrieval was never filtered by tenant at all.
The fix is to make entitlement a property of retrieval rather than an afterthought applied to the answer. Scope every query with per-tenant namespaces, metadata filters, or separate indexes, and enforce the user's document-level permissions at retrieval time, not by hoping the model declines to repeat what it was handed. This is the same discipline covered in Identity and Multi-Tenancy, and privacy is one of the strongest reasons to get it right: a retrieval boundary that leaks is a privacy breach with a person's record on the other side of it.
Memory and observability create the second, slower exposure. Traces and logs capture prompts and context precisely because you want them for debugging, and in doing so they become a durable copy of every piece of personal data that passed through, often in a store with looser access controls than the source system. Conversation memory persists PII across sessions. Both quietly extend the retention of personal data far beyond the request that produced it.
- Scope retrieval to the requesting user or tenant, and prove the boundary with tests rather than assuming it holds.
- Redact before the trace write, so observability data does not become your largest unmanaged PII store; see Tracing & Observability.
- Cap retention with explicit lifetimes and deletion for logs, traces, and memory, matched to what the task and the applicable rules actually require.
- Gate access to trace and memory stores as tightly as the source data, since a copy of the record is still the record.
The US regulatory backdrop
The obligations that shape these controls are, for a US-based enterprise, driven first by state privacy laws and sector rules rather than any single federal statute. As of mid-2026 there is no omnibus federal privacy law, and the operative landscape is a set of state comprehensive privacy acts layered over long-standing sector regulation. The specific duties below are illustrative and shift; treat them as a prompt to verify current obligations for your jurisdictions and data types, not as legal advice.
State comprehensive laws lead. California's CCPA, as amended by CPRA, gives residents rights over their personal information, including access, deletion, and control over certain uses, and a growing roster of peer states has enacted comparable statutes with their own definitions and thresholds. For an AI system the practical consequence is that personal data in prompts, indexes, logs, and provider calls remains subject to those access and deletion rights, which is difficult to honor if you cannot say where a person's data currently sits.
Sector rules bind on top of the state layer and often carry sharper penalties. HIPAA governs protected health information and reaches any AI workflow touching clinical or health-plan data, including the terms under which a model provider would act as a business associate. GLBA governs nonpublic personal information at financial institutions. Where your system operates in these sectors, the sector rule tends to set the tighter constraint and should drive the design baseline.
The architect view
Privacy in AI systems is not a model property; it is a property of the boundaries between components, and it holds only when each hop is designed to protect the personal data crossing it. The through-line of this article is that the exposure is distributed, so the controls must be too. A redaction filter on one endpoint while retrieval, tracing, and the provider call go ungoverned is the shape of most incidents, not the shape of a defense.
Four moves cover the ground. Minimize and redact so the least personal data possible reaches any model or log. Control the provider boundary with terms you verify rather than assume, and self-host the workloads too sensitive to send. Scope retrieval to the requesting user and cap retention on logs, traces, and memory so records neither reach the wrong person nor linger past their purpose. Map each control to the state and sector obligations it satisfies, and re-verify as those obligations move.
The organizing decision is where these controls live. Built into the platform (at the model gateway, the retrieval layer, and the tracing pipeline) they apply uniformly and by default, and every new application inherits them for free. Bolted on per team after an incident, they drift, diverge, and miss the hop that leaks next. Make privacy a platform default, prove the boundaries with tests, and let each new AI use case start already inside the protections rather than negotiating its way back to them. You can pressure-test these controls interactively in the Governance Sandbox.