Meeting HIPAA and GDPR compliance requirements while running AI inference requires control over every prompt, document, embedding, and log. Both frameworks also place restrictions over who can read that data and how long you can store it.
Self-hosting open-source models on infrastructure you control is the most direct way to manage such requirements, but it’s not the only option.
Agents make the whole situation even more complicated. One user query can turn into a dozen model calls, and every call represents another point where sensitive data can end up.
TL;DR Neither HIPAA nor GDPR requires self-hosting. OpenAI and Anthropic both offer HIPAA-related contracts for eligible features. Self-hosting may be a better option when feature scope, data retention, data location, version pinning, or volume make a hosted contract impractical.
Where Does Sensitive Data Leak in an AI Agent?
Let’s see where an AI agent can expose sensitive data. Is the prompt the only thing that needs protection? Spoiler: its just the tip of the iceberg.
| Location | What it holds | Why it matters |
|---|---|---|
| Prompts and chat history | User input, often containing health and other personal details | Obvious. |
| Retrieved context (Retrieval-Augmented Generation, or RAG) | Chunks of records pulled from your index | Often more sensitive than the prompt itself. |
| Embeddings and vector databases | Numeric versions of your documents | Vec2Text demonstrated exact recovery of 92% of 32-token inputs, including names from clinical notes. |
| Queues and payload stores | Requests waiting for a GPU | Data at rest is usually not considered. |
| Logs and traces | Full request bodies when debug logging is on | Ends up in a third-party log tool. |
| Agent tools and MCP (Model Context Protocol) servers | Whatever the agent sends to them | Each external tool is a new data processor. |
| Fine-tuned weights and adapters | Patterns from your training data | Models can repeat parts of what they were trained on. |
HIPAA vs. GDPR for Inference Layers
Both laws require contracts, appropriate safeguarding, and minimal data. However, they phrase it differently.
| Topic | HIPAA (US) | GDPR (EU) |
|---|---|---|
| Data covered | Protected Health Information (PHI) held by covered entities and their vendors | Any personal data; health data is a "special category" under Article 9 |
| Vendor contract | Business Associate Agreement (BAA), per 45 CFR 164.502(e) and 164.504(e) | Data Processing Agreement (DPA), per Article 28 |
| Security rules | Technical safeguards in 164.312: access control, audit controls, integrity, authentication, transmission security | "Appropriate" measures under Article 32 |
| Risk assessment | Security risk analysis | Data Protection Impact Assessment (DPIA) under Article 35 |
| Where data can go | No federal location rule | Transfers outside the EU need a legal basis (Chapter V) |
| End of contract | Vendor must return or destroy PHI | Storage limitation and right to erasure |
Do You Have to Self-Host to be Compliant?
No, you don’t need to rely on self-hosting. There are several hosted options that offer protections for sensitive data, but they also come with their own set of limitations.
| Hosted option | Contract | The catch | Best for |
|---|---|---|---|
| BAA through a Business Associate and Healthcare Addendum | Only BAA-eligible endpoints; Zero Data Retention (ZDR) needs separate approval. | Teams that need OpenAI's frontier models. | |
| BAA for a HIPAA-ready API organization | Batch, Files, Code Execution, and Web Fetch not covered; covered models require 30-day retention and aren’t available with ZDR. | Teams that need Claude for reasoning-heavy steps. | |
| Cloud model platforms (for example AWS) | Covered under the cloud provider's BAA | Coverage depends on each service being on the provider's eligible list. | Teams already under that cloud's BAA. |
When Does a Hosted API Still Make Sense?
- Your volume is low or uneven. Superlinked's self-hosting analysis recommends the API when your monthly inference bill is still under about $2K to $3K.
- You truly need a frontier model.
When Does Self-Hosting Make Sense?
- You want to keep your vendor list short. Every additional third party that has access to PHI adds a contract, a potential point of exposure, and another set of terms that can change. If you already have a BAA with your cloud provider, self-hosting in that account adds no new vendor.
- Your agent needs features outside the BAA scope (batch jobs, file uploads, code execution).
- Your policy says "no retention," and the covered option keeps data for 30 days.
- EU data must stay in the EU. There are also state laws, like Texas SB 1188, that limit where records can live.
- Volume is steady and high.
Which Open-Source Inference Servers Can You Self-Host?
The short, skimmable list:
| Tool | Best for | Limitations |
|---|---|---|
| High-throughput LLM serving with broad model support | One model per server; you build routing and auth around it | |
| Multi-turn chat, structured output, RAG with shared prefixes | Smaller ecosystem than vLLM | |
| Local development and single-user setups | Not designed for multi-tenant production | |
| TEI (Text Embeddings Inference) | Embeddings and rerankers | One model per server; no generation |
| TGI (Text Generation Inference) | Existing deployments only | Put in maintenance mode by Hugging Face |
| SIE (Superlinked Inference Engine) | Many small models (embed, rerank, extract, generate) behind one API | You still run Kubernetes; not built for one huge LLM across many GPUs |
For more depth, see Best Inference Servers for Running Open-Source Models in Production and Top Tools for Serving Open-Source Models on Kubernetes.
Your Self-Hosting Compliance Checklist
Contracts and paperwork
- Sign a BAA (HIPAA) or DPA (GDPR) with your cloud provider, and use only its eligible services.
- Complete a security risk analysis and, for GDPR, a DPIA.
Network and access
- Keep inference endpoints on a private network with no public address.
- Turn on authentication. Some servers accept anonymous requests by default (SIE documents how to enable token auth in its SECURITY.md).
- Require MFA for anyone who can reach the cluster.
Logs and telemetry
- Log metadata (model, latency, request ID), not request bodies.
- Keep an audit trail of who called which model and when.
- Review or turn off any usage telemetry.
Data and models
- Redact personal data before it reaches an LLM, a log, or an outside tool.
- Set retention and deletion rules for vectors, caches, and fine-tuning data.
- Pin model versions and keep evaluation results for each one.
- List every tool and MCP server your agent can call.
How Can You Skip Most of This Setup Work?
You may have read the checklist and thought, “Well, that’s a lot!” If so, I’ve got good news: that's where SIE jumps in. (Full disclosure: we’re behind SIE. 👋)
SIE is an Apache 2.0 inference server that runs encoders, rerankers, extractors, and LLMs from one cluster in your own cloud:
- No outbound internet: The air-gapped deployment guide mirrors images and weights to private storage, so workers never call Hugging Face.
- Practical example: The private compliance RAG example runs a fine-tuned encoder and reranker on one GPU.
If you want help with the security review itself, Native Inference puts Superlinked engineers behind an SIE cluster in your account. Superlinked has no access to the data unless you grant it.
⚠️ Good to know: Superlinked doesn’t currently sign BAAs. For HIPAA workloads, the recommended path is running SIE inside your own infrastructure, with any vendor support kept away from PHI.

