Medical AI Compliance

HIPAA's Privacy Rule protects all "Individually Identifiable Health Information" (PHI) — including names, dates, Medical Record Numbers (MRNs), diagnoses, and more. Its Security Rule mandates administrative, physical, and technical safeguards for any system that stores or transmits PHI. For AI deployments, two provisions are critical: the Minimum Necessary Standard (only access the data the task strictly requires) and the Business Associate Agreement (BAA) requirement (any third-party system handling PHI must sign one).

• ▸PHI transmitted to a public LLM: The agent sends the patient's full record — name, DOB, diagnosis history, medications — to a cloud-hosted LLM with no BAA. This is a reportable data breach, even if the response is never stored.
• ▸Over-fetching violates Minimum Necessary: To find the current medication list, the agent reads the entire 10-year patient record. Regulators consider this a violation even if the extra data is never used in the response.
• ▸No audit trail: If a breach occurs, HIPAA requires you to prove exactly what data was accessed, by whom, and when. A vanilla LLM integration provides none of this.

HITECH dramatically strengthened HIPAA enforcement by extending liability directly to Business Associates and by mandating breach notification. If a breach exposes more than 500 records, you must notify affected patients, the Secretary of HHS, and in many cases the media — within 60 days of discovery. HITECH also introduced tiered financial penalties, with wilful neglect carrying fines up to $1.9 million per category per year. For AI systems, the critical implication is that a compromised or manipulated agent is a reportable breach event, not just a technical failure.

• ▸Prompt injection as a breach vector: An LLM that can read patient data and make outbound calls is a prime target. A successful injection that exfiltrates even a small number of records triggers HITECH's mandatory notification regime.
• ▸Unrestricted data scope: If an agent has broad read access "for convenience," a single successful attack becomes a large-scale breach. An agent that can query any table is an agent that can expose any table.
• ▸No detection capability: Without monitoring on tool calls and output payloads, you may not know the breach occurred until you receive an external complaint — starting the 60-day clock retroactively.

Under GDPR, health data is classified as "Special Category Data" — the most sensitive tier. Processing it requires either explicit patient consent or strict necessity under a recognised legal basis. Article 22 adds another dimension for AI specifically: individuals have the right not to be subject to solely automated decisions that produce significant effects on them. Organisations must also honour the Right to Erasure — if a patient requests deletion of their data, it must be fully removed.

• ▸Fine-tuning on patient data: Training or fine-tuning a model on patient records — even internally — requires explicit consent. Without it, this is an unlawful processing activity regardless of clinical benefit.
• ▸Right to Erasure becomes unenforceable: If patient data has been used to update model weights, "forgetting" the patient becomes technically intractable. You cannot delete a person from a neural network's parameters.
• ▸Sole automated decision-making: An agent that autonomously selects a treatment pathway without a human in the loop likely triggers Article 22 rights, requiring the patient to be able to request human review.

The FDA classifies software as a Medical Device (SaMD) if it is intended to diagnose, treat, mitigate, or prevent disease — or to inform clinical management in a way that could affect patient outcomes. This covers a wider range of AI tools than most developers expect. Once classified, the software must meet Design Controls (21 CFR 820.30), including documented requirements, risk analysis, and software validation. The FDA's AI/ML-based SaMD guidance also requires a Predetermined Change Control Plan — a roadmap for how the system will be updated without losing regulatory clearance.

• ▸Non-determinism: Ask the same dosage question twice and get different answers. The FDA requires validated, reproducible outputs — a property that standard LLM generation fundamentally lacks.
• ▸No traceable logic: "The neural network said so" is not a valid Design Control justification. Regulators need to see requirement → implementation → test evidence, not a probability distribution over tokens.
• ▸Model updates break clearance: Updating (or your LLM provider silently updating) the underlying model constitutes a device change, potentially invalidating your 510(k) clearance if not pre-approved.

The EU MDR imposes rigorous Clinical Evaluation requirements — you must demonstrate clinical evidence that your device is safe and effective throughout its entire lifecycle, not just at launch. Post-Market Surveillance (PMS) is mandatory and ongoing: you must continuously collect and analyse real-world performance data. For AI-based SaMD, this includes monitoring for model drift — the gradual degradation of accuracy as the real-world distribution of patients or clinical language shifts away from validation conditions.

• ▸Model drift goes undetected: Without continuous monitoring, performance degradation is invisible until a clinical incident occurs. At that point, the lack of ongoing PMS data is itself a compliance failure.
• ▸No mechanism for PSURs: The MDR requires documented Periodic Safety Update Reports. Without structured logging of agent behaviour, there is nothing to put in them.
• ▸Provider-side model updates: If you rely on a third-party LLM, silent model updates can alter your device's behaviour without triggering your change control process — a direct MDR violation.

IEC 62304 defines the software development lifecycle for medical devices. It assigns three safety classes — A (no injury possible), B (non-serious injury), and C (death or serious injury possible) — each with increasing rigour. For Class C, you must demonstrate complete traceability from every safety requirement, through the implementation, to a corresponding test case. Every safety-critical function must be traceable to a specific, testable line of code.

• ▸Untraceable logic: You cannot map a safety requirement to a specific neural network weight. There is no line of code an auditor can point to that implements "check for allergies" — the behaviour emerges from billions of parameters, non-deterministically.
• ▸No enforceable guarantees: Even if the agent behaves correctly in testing, nothing prevents it from skipping the allergy check when given an unusual prompt in production. The behaviour is probabilistic, not contractual.
• ▸Test evidence gap: Class C requires systematic test evidence. Testing a probabilistic model against a fixed requirement is fundamentally different from testing a deterministic function, and the evidence is weaker by design.

The 21st Century Cures Act mandates health data interoperability through FHIR (Fast Healthcare Interoperability Resources) standards and explicitly prohibits "information blocking" — any practice that interferes with the access, exchange, or use of electronic health information. For AI deployments, systems must not trap data in proprietary formats, fail to surface available information, or obstruct access to records — even unintentionally.

• ▸Context window silently drops data: A naive agent loads all available records into its context window. If combined records exceed the window, earlier data is silently dropped — the agent unknowingly blocks information without any system-level constraint.
• ▸Proprietary reformatting: An agent that reformats FHIR resources into its own internal representation may lose structured field metadata, breaking downstream interoperability and potentially constituting information blocking.
• ▸Failure to surface all sources: If the agent only queries one EHR when multiple are available, it effectively blocks access to the others — a compliance violation even if no malicious intent exists.

Section 1557 prohibits discrimination on the basis of race, colour, national origin, sex, age, or disability in health programmes receiving federal funding. Updated HHS guidance explicitly applies this to clinical decision-support tools and AI systems used in patient care. Any algorithm that produces systematically different outcomes for protected groups — even without discriminatory intent — creates civil rights liability. Providers are responsible for the discriminatory effects of algorithms they deploy, including commercial tools.

• ▸Latent bias in training data: LLMs trained on historical healthcare data can absorb and replicate historical biases. Insurance type, postcode, or language are proxies for protected characteristics, and an agent using these signals — even implicitly — creates liability.
• ▸Unexplainable prioritisation: If a regulator asks "why was my patient's appointment scheduled later?", the answer "the AI decided" is not legally defensible under Section 1557.
• ▸No audit mechanism: Without a record of the exact logic used to rank patients, demonstrating that discrimination did not occur is impossible.

21 CFR Part 11 establishes that electronic records and signatures in FDA-regulated industries (pharma, biotech, medical devices) must be trustworthy, reliable, and legally equivalent to paper records and handwritten signatures. It requires strict audit trails showing who created, modified, or approved records and when. Any computer system used to create or maintain these records must be validated, access-controlled, and equipped with an audit trail that cannot be altered or disabled.

• ▸No human attribution: Standard AI agents produce outputs attributed to "the system," not a specific employee. If an agent reviews and effectively endorses a batch record, tracing that decision to a regulated, accountable human is essential under Part 11.
• ▸Mutable or absent audit trails: If the agent's reasoning process, the data it reviewed, and the actions it took are not captured in an immutable, timestamped log, the electronic record is non-compliant regardless of the outcome.
• ▸Unvalidated systems: Part 11 requires computer system validation. Deploying an LLM without a structured validation protocol and documented evidence is a direct compliance gap in a regulated environment.

ICH E6 (R3) is the international standard for Good Clinical Practice in clinical trials. Its core principles are participant protection, data credibility, and trial record integrity. The R3 revision explicitly addresses computerised systems and data integrity — requiring that any system used to generate, process, or store trial data meets the ALCOA+ framework: Attributable, Legible, Contemporaneous, Original, and Accurate, with audit trails for all data changes.

• ▸Hallucinated corrections compromise trial integrity: If the agent invents a "corrected" value rather than flagging the anomaly for human review, it introduces fabricated data into a regulated trial dataset. This can invalidate the entire trial.
• ▸No ALCOA+ compliance: The "A" in ALCOA stands for Attributable — every data point must be traceable to the person who created or changed it. An AI correction with no human attribution fails this requirement.
• ▸Unvalidated computerised system: GCP requires that any computerised system used in a trial is validated to demonstrate it performs as intended. An unvalidated LLM integration is a finding in any regulatory inspection.