I nearly signed off on an "auto‑close" feature for an AI that triaged evidence requests. It would have saved the team hours a week. We didn't enable it. To be fair, automation is seductive when your CAPA queue looks like a climbing route on the Wetterhorn, but granting an AI the power to act on regulated records is the moment you voluntarily take on provider obligations under the EU AI Act's high‑risk rules (Title III). In practice this means a much heavier conformity and post‑market burden — and the auditors will want to see the human thread.
My rule is simple and enforceable: AI proposes, a human approves and signs. Below is why that matters, how we implemented it, and the checklist I hand to engineers and vendors when "AI assistance" starts looking like autonomy.
Why "acting" matters under the EU AI Act
The Act's high‑risk classification hinges on how a system is used and the impact of its outputs. If an AI makes or executes decisions that materially affect safety, regulatory records, or legal obligations, you shift from using a decision‑support tool to placing an automated decision‑maker into scope. Consequences include:
- conformity assessment requirements,
- additional technical documentation and lifecycle evidence,
- stricter post‑market monitoring and reporting obligations,
- potentially higher liability exposure.
Granted, a lot depends on specifics and national practice will vary, but in my experience with notified bodies and auditors, the safe path is to avoid ambiguity. If an AI can autonomously change or close a regulated record, expect a long conversation about whether the system is now a "high‑risk" product component.
The principle: AI proposes, a human approves and signs
This principle is deliberately binary and audit‑friendly.
- "Proposes" = the AI generates suggestions, drafts, prioritisation scores, or recommended dispositions.
- "Approves and signs" = a named, trained person reviews the suggestion, makes any edits, and applies their electronic signature or recorded approval before the system executes the change.
Why this is clean:
- It preserves human accountability and a clear audit trail.
- It keeps the system demonstrably in the "decision support" bucket rather than the "automated decision" bucket.
- It fits naturally into existing QMS controls: traceability, change control, CAPA ownership, and reviewability.
How we implemented it (practical steps)
When we rebuilt the workflow for AI‑assisted evidence triage and automated CAPA drafting, we followed a short checklist that I now insist every project meets before any "auto" label is used.
Technical controls
- The AI can write or tag, but cannot change the record status field. Status transitions require an authenticated user action.
- Mandatory "review comments" field — user must add a rationale before approval. The field is time‑stamped and non‑editable afterwards.
- Immutable audit logs for the AI's input, the AI's output (exact text), and the human reviewer’s decision (approve/reject/edit).
- Distinct UI affordances that separate "suggested" content from "authorised" content — no opt‑out default toggles that let suggestions auto‑execute.
Procedural controls
- SOP for AI‑assisted work covering: roles, who may approve, required evidence for approvals, and escalation paths.
- Training records for approvers logged in the QMS.
- Change control record for any configuration changes to the AI model or prompt logic.
- Periodic review (PMCF/PMPF style) of AI outputs for drift and bias — this is treated like supplier oversight.
Integration with QMS
- Suggestions link back to the originating data and risk assessment (traceability).
- When the AI drafts a CAPA, it creates a task assigned to a named CAPA owner; the owner reviews, edits, and then raises the CAPA formally with their signature.
- Use connected workflow so the AI’s output becomes an attachable artefact in the relevant Technical File or device record, not a live change.
Edge cases and emergency handling
We had one scenario where a time‑critical field safety correction needed an immediate admin change. Our workaround: allow a narrowly‑scoped emergency action by a senior authorised person, followed by mandatory after‑the‑fact review and explanation logged within 24 hours. The key is to document the exception and ensure it remains an exception.
Automated monitoring (alerts) is fine; automated execution is not. If an AI flags a critical issue, it should only create an alert/task — not close the ticket.
What auditors actually want to see
- Clear SOPs showing the human approval requirement.
- Audit logs proving the AI only proposed and could not change status.
- Evidence that approvers are trained and accountable.
- Change records whenever the AI or its prompts are modified.
To be fair, some vendors will argue their model is explainable or that their confidence thresholds make auto‑actions safe. That may be true in narrow cases, but "explainable" does not equal "compliant" and the burden of proof sits with you.
Final practical checklist (quick)
- Configure AI as read/write‑for‑suggestions only, locked on status fields.
- Require human approval with electronic signature before execution.
- Log AI inputs, outputs, and the human decision in an immutable audit trail.
- Link suggestions to traceable risk assessments and CAPA records.
- Treat model/ prompt changes as controlled changes under your QMS.
- Define emergency SOPs with mandatory after‑action review.
I still like hiking automation trails — they save time and reduce errors. But when it comes to regulated records, think of the summit: you want to be sure someone human planted the flag. "AI proposes, a human approves and signs" is the simplest, most audit‑resilient rule I've found.
How have you drawn the line between AI assistance and AI action in your organisation — and what did your auditors ask to prove it?
