Wichtigste Erkenntnisse
- HIPAA (healthcare): Patient data cannot touch cloud APIs. Local LLM on isolated network, encrypted storage, access logs.
- PCI-DSS (payment cards): Payment card data cannot be processed by LLMs at all. Use for analytics only, never full PAN (card numbers).
- Legal (attorney-client privilege): Privileged documents cannot leave attorney's control. Air-gapped machine, no network, hardcopy output only.
- Setup: vLLM on isolated Linux server, encrypted filesystem (LUKS), audit logging (ELK stack), no internet.
- Cost: $3,000–5,000 hardware + $2,000/year managed security updates. Free (open source) software.
- vs Cloud APIs: Cloud = $0 upfront, $50K+/year breach liability. Local = $5K upfront, $0 breach liability.
- Compliance verification: Penetration test, SOC 2 audit, third-party validation before production.
- Data retention: Local LLM logs must be purged per HIPAA (6-year retention) / GDPR (right to deletion).
Why Local LLMs for Compliance
Cloud APIs (ChatGPT, Claude, Gemini) cannot be used with regulated data:
- Data transmission to cloud = breach of confidentiality (HIPAA, legal privilege).
- No "private mode" option. Data trains models eventually.
- Vendor lock-in: if vendor breaches or shuts down, you lose data + compliance.
Local LLM guarantees:
- Zero data egress (air-gapped = no network).
- Audit trail (every access logged, cryptographically signed).
- Control (you own the data, not the vendor).
- Cost predictability (no per-token charges after initial setup).
HIPAA-Compliant Setup (Healthcare)
PHI (Protected Health Information) cannot be processed by untrusted systems.
- 1Isolate the server: Dedicated Linux machine, no internet, encrypted USB for data transfer in/out.
- 2Encrypt storage: LUKS full-disk encryption, passphrase-protected.
- 3vLLM on private network: No external connections. Access via VPN (staff only) or secure terminal.
- 4Audit logging: Every LLM query logged with: timestamp, user ID, document hash (not plaintext), response length.
- 5Access control: Role-based (doctor vs. admin vs. researcher). MFA for login.
- 6Retention policy: Delete inference logs after 6 years (HIPAA requirement). Automated deletion scripts.
- 7Business Associate Agreement (BAA): Model provider must sign BAA (open-source models: Llama, Mistral all covered).
- 8Annual penetration test: Third-party security audit to verify no data leaks.
PCI-DSS Compliant Setup (Finance)
Payment card data (PAN = Primary Account Number) cannot be processed by LLMs in plaintext.
- 1Never input card numbers into LLM. Not allowed. Use tokenized representations (e.g., last-4-digits only).
- 2Encrypt at rest & in transit: AES-256 encrypted files, TLS 1.3 for network.
- 3Network segmentation: LLM server on isolated VLAN, no access to internet.
- 4Hardware security module (HSM): Store encryption keys in tamper-evident device (Thales, Yubico).
- 5Logging & monitoring: Real-time alerts on file access, login attempts, data exfiltration.
- 6Quarterly compliance scan: Automated PCI-DSS scanning (Qualys, Rapid7).
- 7Vendor certification: Model providers (Ollama, vLLM) must document PCI-DSS alignment (open-source: pre-screened).
Air-Gapped Deployment
Most secure option: machine has zero network connection.
- 1Physical isolation: Server in locked room, no Ethernet cable, WiFi disabled in BIOS.
- 2Model loading: Pre-download models on connected machine, transfer via encrypted USB.
- 3Data transfer in: Users transfer documents via encrypted USB (GPG-encrypted files).
- 4Inference: Run LLM locally, output saved to USB.
- 5Data transfer out: Encrypted USB returned, decrypted on separate machine.
- 6Trade-off: Latency (manual USB sneakernet) vs. absolute security (zero network risk).
- 7Use case: Legal discovery, healthcare image analysis, financial model training (batch processing OK).
Audit Logging & Data Governance
What to log: Every LLM query (timestamp, user, prompt hash, response length), file access, login/logout.
Where to store: Encrypted syslog server, separate from application server.
Tamper-evidence: Cryptographic signatures on logs (no deletion without breaking chain of trust).
Tools: ELK Stack (Elasticsearch/Logstash/Kibana) for aggregation; Splunk for enterprise.
Retention: HIPAA = 6 years, GDPR = right to deletion (need process to anonymize), PCI-DSS = 1 year.
Compliance verification: Monthly log review, quarterly data lineage audit, annual third-party assessment.
Common Compliance Failures
- Using cloud ChatGPT with healthcare data. Immediate HIPAA violation, $10K–$50K penalties.
- Air-gapped server with unlocked door. Physical security = zero if anyone can walk in.
- Logs stored on same server as data. Breach of logs = breach of audit trail. Separate systems required.
FAQ
Can I use cloud LLMs with compliance data if I hash PII?
No. Hashed data is still regulated. Cloud = violation. Use local LLM only.
Do I need a BAA with Llama or Mistral models?
Technically no (open-source, no vendor). But document your compliance internally. BAA template available online.
Is air-gapped overkill for HIPAA?
Not overkill. If data is highly sensitive (genetics, psychiatric), air-gapped is best practice.
How do I handle employee termination securely?
Disable VPN access immediately. Audit all queries by that user in past 6 months. Verify no data exported.
Can I use local LLMs for legal discovery?
Yes. Air-gapped + attorney supervision + attorney-client privilege maintained (no third-party access).
What if there's a breach of the local server?
Encrypted-at-rest = limited damage. Audit logs reveal what was accessed. Notify affected parties within 30 days (HIPAA/GDPR).
Sources
- HIPAA Privacy Rule: 45 CFR 164 (US Department of Health & Human Services)
- PCI Data Security Standard v4.0 (PCI Security Standards Council)
- GDPR Article 32: Security of Processing (EU regulation)