Answers · By Hugh Donatello, Atlas Bio · Updated 2026-10-29

AI Agent Errors in GxP and R&D: Deviations and CAPA

How should a biotech handle an AI agent error: as a deviation, an incident or a research error? It depends on where the output went. An error that stays in discovery work with no GxP or submission use is a research error, handled by the team's own rules. Once the output feeds a GxP record, process or study, use your existing deviation, incident and CAPA procedures. Contain first: stop or roll back the agent. Then find the root cause, which may lie in data, prompt, model update, tool permission, human review or intended use. Link each CAPA to change control and record each step.

Status of the guidance this page relies on

No regulator text cited here mentions AI agents, prompts or kill switches by name, and none sets a separate deviation or CAPA process for AI. This page applies the existing deviation, incident and CAPA requirements to AI agents. Where it does that, it says it is our reading. Status as checked on 21 October 2026:

Binding US requirements: the drug CGMP regulations, including 21 CFR 211.22, 211.100 and 211.192, and, for non-clinical laboratory studies, the GLP regulations in 21 CFR Part 58. EU GMP: EudraLex Volume 4, Part I, Chapter 1 (Pharmaceutical Quality System, in operation since 31 January 2013) and Annex 11 (Computerised Systems, January 2011 revision, in operation since 30 June 2011) are current. The European Commission consulted from 7 July to 7 October 2025 on a revised Annex 11 and a new Annex 22 on artificial intelligence; both remain drafts and may change before adoption. ICH Q10 (Pharmaceutical Quality System) is published by FDA as final guidance (April 2009) and is listed in EudraLex Volume 4, Part III. ICH E6(R3) Good Clinical Practice reached ICH Step 4 in January 2025 and FDA issued it as final guidance in September 2025. FDA's January 2025 guidance on AI to support regulatory decision-making for drugs and biologics is still a draft, not for implementation. EMA's reflection paper on AI in the medicinal product lifecycle (September 2024) is a reflection paper, not binding guidance. EMA's guideline on computerised systems and electronic data in clinical trials (EMA/INS/GCP/112288/2023) is a final guideline.

This page is general information, not legal or regulatory advice. Decisions on GxP scope, deviation classification, root cause, CAPA and any report to a regulator belong to your quality and regulatory functions.

Deviation, incident or research error: where the output went decides

The same mistake by an agent can be handled in three different ways depending on where its output ended up. The question to ask first is not 'what did the AI do wrong?' but 'which record, process, study or decision did the output reach, and what did the agent do?'

If the work is GMP, the existing rules apply as written. 21 CFR 211.100(b) says production and process control procedures 'shall be followed' and that 'any deviation from the written procedures shall be recorded and justified.' 21 CFR 211.192 says any unexplained discrepancy 'shall be thoroughly investigated', that the investigation 'shall extend to other batches of the same drug product and other drug products that may have been associated with the specific failure or discrepancy', and that 'a written record of the investigation shall be made and shall include the conclusions and followup.' Under 21 CFR 211.22(a), the quality control unit has the authority to review production records to assure that no errors have occurred or, if they have, that they have been fully investigated. EU GMP Chapter 1, paragraph 1.8(vii), asks that any significant deviations be fully recorded and investigated to find the root cause, with appropriate corrective and preventive action. Annex 11 section 13 adds a computerised-system rule: 'All incidents, not only system failures and data errors, should be reported and assessed.' The draft revised Annex 11, section 3.1(i), would make it explicit that all deviations during validation or operation of computerised systems are recorded, and significant ones investigated for root cause and impact on product quality, patient safety or data integrity.

In non-clinical laboratory studies, 21 CFR 58.33(c) makes the study director responsible for assuring that 'unforeseen circumstances that may affect the quality and integrity of the nonclinical laboratory study are noted when they occur, and corrective action is taken and documented.' Under 58.81(a), 'all deviations in a study from standard operating procedures shall be authorized by the study director and shall be documented in the raw data', and under 58.185(a)(9) the final report must include 'a description of all circumstances that may have affected the quality or integrity of the data.'

In clinical trials, ICH E6(R3) section 3.12.2 says that where noncompliance with the protocol, SOPs, GCP or applicable regulations significantly affects, or has the potential to significantly affect, participants' rights, safety or well-being or the reliability of trial results, the sponsor should perform a root cause analysis, implement appropriate corrective and preventive actions and confirm their adequacy unless otherwise justified. Where the sponsor identifies issues likely to significantly impact the rights, safety or well-being of trial participants or the reliability of trial results (serious noncompliance), the sponsor should notify the regulatory authority and/or IRB/IEC in accordance with applicable regulatory requirements. EMA's guideline on computerised systems in clinical trials notes that the sponsor has a legal responsibility under Regulation (EU) No 536/2014 to report serious breaches, including important data and security breaches, to authorities within seven days, and asks that agreements with service providers specify which information should be escalated immediately. In our reading, an agent error is a GCP matter in this sense when it is, or causes, noncompliance of that kind.

Discovery work outside GxP is different. EMA's reflection paper on AI says AI in drug discovery 'can be low regulatory impact if non-optimal performance only affects the developer', but that if results contribute to the body of evidence presented for regulatory review, the principles for non-clinical development should be followed. FDA's draft AI guidance states that it does not address AI used in drug discovery, or for operational efficiencies that do not affect patient safety, drug quality or the reliability of results from a non-clinical or clinical study. In our reading, an agent error in discovery is a research error: log it under the team's own rules and fix it. It becomes a GxP matter as soon as you find the output reached a GxP record, a study or a regulatory submission, and that check should be part of every research-error review.

The categories in the table are not exclusive. One event can be both a computerised-system incident and a GMP deviation, handled under both procedures.

Where the output went, or what the agent didIn our reading, how it is usually handled (illustrative)Primary text it rests on
Discovery work only; not used in any GxP record, study or submissionResearch error, logged and fixed under the team's own rules; check whether the output was reused downstreamEMA AI reflection paper section 2.3.1; FDA draft AI guidance scope
Caught and rejected by the reviewer the procedure requires, before it reached a GxP recordUsually not a deviation if the procedure foresees rejections: the control worked. Still record the rejection with its reason, and assess it as Annex 11 section 13 asks for all incidents; where the agent's output is itself GMP data, the rejection is part of the record. Trend rejections as a sign of drift, and escalate to a deviation if the error is one validation should have ruled out or rejections pass a set limit. Your quality function decides where the line sitsAnnex 11 section 13; Annex 11 section 11 (periodic evaluation includes, where appropriate, incidents and problems); draft Annex 22 sections 10.3 and 10.5 (borrowed principles)
Entered a GMP record, or the procedure was not followed (for example, the required human review was skipped)Deviation: record, justify and investigate under the deviation procedure, with QA assessing impact21 CFR 211.100(b), 211.192, 211.22(a); EU GMP Chapter 1, 1.8(vii); Annex 11 section 13
Agent acted outside its permissions, or wrote to or changed a record it should not reachIncident under Annex 11 section 13, likely critical; contain at once, and run the security incident procedure as well if access controls failedAnnex 11 sections 12 and 13; in clinical trials, EMA computerised systems guideline A4.11 (security incidents)
Unforeseen circumstance or SOP deviation in a GLP studyNoted when it occurs, with corrective action taken and documented, under the study director; SOP deviations authorised by the study director and documented in the raw data; described in the final report if it may have affected data quality or integrity21 CFR 58.33(c), 58.81(a), 58.185(a)(9)
Clinical trial, where the error is or causes noncompliance that significantly affects, or could significantly affect, participants' rights, safety or well-being or the reliability of resultsRoot cause analysis and CAPA by the sponsor, with their adequacy confirmed; notify the regulatory authority and/or IRB/IEC if the noncompliance is serious; in the EU, assess whether it is a serious breach to report within seven daysICH E6(R3) 3.12.2; EMA clinical-trials computerised systems guideline (service-provider escalation; A4.11 for security incidents); Regulation (EU) No 536/2014

Containment: stop the agent, roll back, protect the records

Containment comes before root cause. No regulator text cited here requires a 'kill switch'. In our reading, two texts point the same way. EMA's reflection paper, which is not binding, says that for all models, especially those with no human in the loop, a system risk management plan should define likely failure modes and mitigation, 'such as how to trigger a suspension/decommission of the model and how to suspend or decommission it.' Annex 11 section 16 asks that, for computerised systems supporting critical processes, provisions be made to keep those processes going if the system breaks down, 'e.g. a manual or alternative system', and that these arrangements be 'adequately documented and tested.' That covers the fallback, not the act of stopping the system.

In our reading, that means four things should be written down and tried before an agent goes live. Who can stop it, and how: one named role, one documented action, such as revoking the agent's credentials or disabling its tool access, that works without the vendor's help. What the process falls back to while the agent is stopped, usually the manual method it replaced. What the last known good configuration is: the exact model version, prompts, tools, permissions and retrieval sources that passed validation. And how to go back to it.

Simply stopping the agent is containment and can be handled under the incident procedure. Rolling back is different: in our reading it is a change. Annex 11 section 10 says changes to a computerised system, 'including system configurations', should only be made 'in a controlled manner in accordance with a defined procedure.' If your change procedure has an emergency route, name rollback to the last validated configuration as an allowed use of it: authorised at the time by the designated quality role, with documentation, impact assessment and QA approval completed promptly afterwards. Confirm that the earlier configuration is still valid in the current environment, since connectors, indexes or other dependencies may have changed since it was validated. A hosted model version can also be retired by its provider, so rolling back to it may not be possible; plan the fallback for that case.

Stopping the agent does not undo what it already wrote. Identify every output produced under the faulty configuration and quarantine or review it. In our reading, this mirrors the logic of 21 CFR 211.192, which requires an investigation to extend to other batches that may be associated with the failure; where the outputs touched batch records and a discrepancy is under investigation, that requirement applies directly. If any affected output reached a batch record or release decision, place the affected batches on hold and have QA assess whether distributed product is affected.

Root-cause categories for AI agent errors

EU GMP Chapter 1, paragraph 1.4(xiv), asks for 'an appropriate level of root cause analysis' during investigation of deviations and other problems, which can be determined using quality risk management. Where the true root cause cannot be found, consideration should be given to identifying the most likely causes and addressing those. It adds a warning that fits AI well: where human error is suspected, this should be justified, 'having taken care to ensure that process, procedural or system-based errors or problems have not been overlooked'. ICH Q10 section 3.2.2 asks for 'a structured approach to the investigation process' with the objective of determining the root cause, with effort and formality matched to the level of risk.

In our reading, 'the AI hallucinated' is not a root cause, any more than 'operator error' is. It describes the symptom. The table below groups the causes we would look for first. It is our illustrative starting point, not regulator text and not a GAMP classification; most incidents will have more than one cause, and the question is which control should have stopped the error and did not.

CategoryWhat to look for (illustrative)Where the CAPA usually lands
Data and retrievalThe agent used a superseded, wrong or uncontrolled document; an input outside the range it was validated for; an index that still returned an old versionDocument control and index checks; input monitoring; the defined intended use
Prompt and configurationA prompt or template edit, a changed setting, or an instruction that did not cover the caseChange control for the configuration; added test cases; prompt review
Model updateA new model version, chosen by you or pushed by the provider, that behaves differently on the same inputSupplier agreement and notice terms; version pinning; scheduled reruns of a fixed test set
Tool permissionThe agent could call a tool, reach a system or write a record beyond what its intended use neededAccess review and least-privilege permissions; tests that prohibited actions are refused
Human reviewThe required check was skipped, rushed or not able to catch this kind of errorReview procedure and training; monitoring the reviewer's performance like any manual process
Intended useThe agent was used for a task it was never validated forInventory and intended-use records; user training; access limited to validated uses

Linking CAPA to change control

For an AI agent, almost every corrective action is a change: a new prompt, a pinned model version, a narrower permission, an added test case or a revised review step. ICH Q10 section 3.2.3 says that innovation, continual improvement, the outputs of process performance and product quality monitoring 'and CAPA drive change', and that proposed changes should be evaluated using quality risk management. Annex 11 section 10 brings configuration changes within change control. The draft Annex 22 covers only static machine-learning models with deterministic output in critical GMP uses. It states that generative AI and large language models should not be used in critical GMP applications, and that in non-critical GMP uses qualified, trained personnel should always be responsible for ensuring their outputs are suitable for the intended use. It is a consultation draft and may change. Its section 10.1 still gives a principle worth borrowing: any change 'should be documented and evaluated to determine if the model needs to be retested', and any decision not to retest 'should be fully justified.'

In our reading, that gives a simple rule: each CAPA action on an agent should open, or point to, a change record, and the change record should decide the retest. Our separate answer on change control for AI models and prompts sets out when a change needs revalidation.

Then check that the fix worked. EU GMP Chapter 1, paragraph 1.4(xiv), says the effectiveness of CAPAs 'should be monitored and assessed', and the draft revised Annex 11 section 3.1(i) says the effectiveness of CAPA 'should be verified.' For an agent, in our reading, that usually means rerunning the case that failed and the related test cases after the change, then watching the same monitoring signal for a set period. Annex 11 section 11 lists deviation records, incidents and problems among the things periodic evaluation should cover where appropriate, so a pattern of small agent errors should come up at periodic review even if no single one was critical.

Two separate questions decide whether a regulator needs to hear about it. First, the incident itself: if affected product may have been distributed, or a trial's participants or data were affected, your deviation, quality-defect, safety-reporting or serious-breach procedures decide whether a report is due, often against short deadlines. Change control does not decide that. Second, the fix: change control evaluates whether the change itself needs to be reported. FDA's draft AI guidance, using manufacturing as its example, says changes to an AI model should be evaluated by the manufacturer's change management system within its pharmaceutical quality system, and that a change affecting model performance should be reported to the Agency in accordance with regulatory requirements. Whether any specific incident or change needs a report, variation or supplement is for your regulatory affairs function.

What to record

21 CFR 211.192 says 'a written record of the investigation shall be made and shall include the conclusions and followup.' Annex 11 section 9 asks that the reason be documented for any change or deletion of GMP-relevant data. ICH Q10 asks that the effort, formality and documentation of an investigation match the level of risk. In our reading, an AI agent incident record should let someone who was not there reconstruct what happened. As a starting list:

Identification: the deviation or incident reference and the date it was opened; which agent, and the exact configuration in use at the time, including the model and version, prompt version, tools and permissions, and retrieval sources. Detection: when and how the error was found, and by whom. The event: the inputs, the output, the record or decision it reached, and who reviewed it. Evidence: logs, inputs, outputs and a snapshot of the configuration, kept for the records retention period. Containment and immediate correction: what was stopped or rolled back, when, by whom, what fallback was used, and any data corrections, made under good documentation practice with the original kept. Impact: every other output produced under the same configuration, and the records, batches, studies or participants they may affect, with the quality unit's product impact and batch disposition decision where product is involved. Classification: research error, deviation or incident, the procedure used, the risk rating with its justification, and the outcome of the assessment of whether any report to a regulator is due. Root cause: the categories considered, the evidence, and the most likely cause where the true one cannot be found. Conclusions: what the investigation found, recorded separately from the root cause. CAPA: each action, its owner and due date, and the change record it opened. Timelines: justification for any extension. Effectiveness: the retest and monitoring results. Closure: approval by the quality unit, with the date.

Much of this comes from the agent's own audit trail, if it captures who did what, when and with which configuration. Our separate answer on audit trails for AI agents covers what that trail should hold. The roles behind each step, including who can stop an agent and who decides GxP impact and CAPA, are in our answer on who owns AI agent governance in a biotech.

Incident handling is easier to plan once you know which AI agents and tools already touch regulated work, and what they can reach. Atlas Bio's Secure AI for Life Sciences starts with an AI risk assessment of current AI use and data exposure. Status, October 2026. In use in Atlas Bio's own agent platform today: a separate verification step that checks agent work, and a hold that stops a task until a person decides. Built and tested on synthetic data, not yet used with a client: the AI risk assessment tooling (AI discovery, data-flow and vendor-terms mapping, and a NIST AI RMF gap report) and a record of what data an AI used, what it produced and who reviewed it. In development: a customer-controlled AI environment in your own Azure tenant, an AI gateway, agent permissions, evidence linkage and notifications. Each engagement states which controls are built. Decisions on GxP scope, deviation classification, root cause, CAPA and regulatory reporting stay with your quality and regulatory functions.

Is an AI agent error automatically a GMP deviation?

No. It depends on where the output went. If it stayed in discovery work with no GxP or submission use, it is a research error. If a required procedure such as human review was not followed, 21 CFR 211.100(b) requires the deviation to be recorded and justified; under 211.192, an unexplained discrepancy must be thoroughly investigated; and EU GMP Chapter 1 asks that significant deviations be recorded and investigated. In our reading, a faulty output that entered a GMP record should go through your deviation procedure, with QA deciding its classification. An output rejected by the reviewer the procedure requires is usually the control working as designed, if the procedure foresees rejections; in our reading, record it, assess it under the incident procedure and trend rejections. Your quality function decides where the line sits.

Do regulators require a kill switch for AI agents?

No regulator text cited here requires one by that name. In our reading, two texts point the same way. EMA's AI reflection paper, which is not binding, says a system risk management plan should define how to trigger a suspension or decommissioning of a model and how to carry it out. Annex 11 section 16 asks for documented, tested fallback arrangements, such as a manual alternative, to keep critical processes going if a system breaks down.

Is rolling back a model or prompt version a change that needs change control?

In our reading, yes. Annex 11 section 10 covers changes 'including system configurations', and a prompt or model version is part of an agent's configuration. If your change procedure has an emergency route, name rollback to the last validated configuration as an allowed use of it: authorised at the time by the designated quality role, with documentation, impact assessment and QA approval completed promptly afterwards. Check that the earlier configuration is still valid in the current environment, and that a hosted model version has not been retired by its provider. Simply stopping the agent is containment, handled under the incident procedure.

Can we record the root cause as 'the AI hallucinated'?

In our reading, no. That is a symptom. EU GMP Chapter 1, paragraph 1.4(xiv), warns against settling on human error without checking for process, procedural or system causes, and the same logic applies to blaming the model. Look at data and retrieval, prompts and configuration, model updates, tool permissions, human review and intended use.

How does CAPA connect to change control for an AI agent?

Almost every corrective action on an agent is a change, such as a new prompt, a pinned version or a narrower permission. ICH Q10 says CAPA drives change and that changes are evaluated with quality risk management. In our reading, each CAPA should open a change record that decides the retest, and CAPA effectiveness should be monitored and assessed, as EU GMP Chapter 1 asks; the draft revised Annex 11 would say 'verified'.

What about AI agent errors in a clinical trial?

Where an agent error is, or causes, noncompliance that significantly affects, or could significantly affect, participants' rights, safety or well-being or the reliability of trial results, ICH E6(R3) section 3.12.2 asks the sponsor to perform a root cause analysis, implement corrective and preventive actions and confirm their adequacy, and, where the noncompliance is serious, notify the regulatory authority and/or IRB/IEC under applicable requirements. EMA's computerised systems guideline notes the sponsor's legal duty under Regulation (EU) No 536/2014 to report serious breaches within seven days, and asks that agreements with service providers specify what is escalated immediately.

  1. eCFR: 21 CFR 211.22 - Responsibilities of quality control unit
  2. eCFR: 21 CFR 211.100 - Written procedures; deviations
  3. eCFR: 21 CFR 211.192 - Production record review
  4. eCFR: 21 CFR 58.33 - Study director
  5. eCFR: 21 CFR 58.81 - Standard operating procedures
  6. eCFR: 21 CFR 58.185 - Reporting of nonclinical laboratory study results
  7. European Commission: EudraLex Volume 4, Part I, Chapter 1 - Pharmaceutical Quality System (in operation since 31 January 2013)
  8. European Commission: EudraLex Volume 4, Annex 11 - Computerised Systems (January 2011 revision, in operation since 30 June 2011)
  9. European Commission: EudraLex Volume 4 - Good Manufacturing Practice guidelines (index)
  10. European Commission: Stakeholders' consultation on EudraLex Volume 4 - Chapter 4, Annex 11 and new Annex 22 (closed; 7 July to 7 October 2025)
  11. European Commission: Revised Annex 11 Computerised Systems - consultation draft (PDF)
  12. European Commission: Annex 22 Artificial Intelligence - consultation draft (PDF)
  13. FDA guidance: Q10 Pharmaceutical Quality System (final, April 2009)
  14. ICH: Q10 Pharmaceutical Quality System guideline (PDF)
  15. FDA guidance: E6(R3) Good Clinical Practice (final, September 2025)
  16. ICH: E6(R3) Guideline for Good Clinical Practice, Step 4 final guideline, January 2025 (PDF)
  17. European Medicines Agency: Guideline on computerised systems and electronic data in clinical trials (EMA/INS/GCP/112288/2023)
  18. European Medicines Agency: Reflection paper on the use of artificial intelligence in the medicinal product lifecycle (EMA/CHMP/CVMP/83833/2023, September 2024)
  19. FDA draft guidance: Considerations for the Use of Artificial Intelligence to Support Regulatory Decision-Making for Drug and Biological Products (January 2025; not for implementation)
  20. FDA draft guidance on AI to support regulatory decision-making, full text (PDF)

Talk to Atlas Bio.

Sponsors, researchers and clinicians: tell us the decision you are trying to make and we will show what the platform can and cannot answer.

Contact Us →