Answers · By Hugh Donatello, Atlas Bio · Updated 2026-10-29

GxP AI Agent Validation: 6 Risk-Based Steps

How do you validate an AI agent used in a GxP process? Validate the agent for one defined intended use, not as a general tool. Write down the task, inputs, outputs and records it touches. Assess what happens if it fails and scale validation to that risk, as EU GMP Annex 11 and FDA's drug data integrity guidance expect. Set acceptance criteria before testing, allowing for outputs that vary. Name where a trained reviewer checks the output. Keep the agent under change control, so a model, prompt or tool change gets a documented retest decision.

Download: Validation plan outline for an AI agent in a GxP process (PDF)
A fill-in outline, also available as an editable Word file. Free to copy and use inside your organization. General guidance, not regulatory advice.
Editable Word version (.docx)

Status of the guidance this page relies on

None of the regulator documents cited on this page sets out how to validate an AI agent built on a large language model. The closest, the European Commission's draft Annex 22, excludes generative AI and large language models from its scope. What the cited documents give you is general computerised-system guidance, AI-specific drafts and industry good practice. Their status matters, so it is stated up front, as checked on 19 October 2026.

EU GMP Annex 11 (Computerised Systems, in operation since 30 June 2011) is current EU GMP guidance. FDA's Data Integrity and Compliance With Drug CGMP questions and answers (December 2018) is final guidance for drug CGMP. FDA's Computer Software Assurance for Production and Quality Management System Software (CSA), issued on 3 February 2026 by CDRH and CBER in consultation with CDER, is final, non-binding guidance; its stated scope is medical device production and quality system software, so for drug and biologic CGMP it is context, not a requirement. The European Commission consulted from 7 July to 7 October 2025 on a revised Annex 11 and a new Annex 22 on artificial intelligence; both are drafts, and the consultation page does not show an adopted version. FDA's January 2025 guidance on AI to support regulatory decision-making for drugs and biologics is a draft, not for implementation. The ISPE GAMP Guide: Artificial Intelligence (July 2025) is industry good practice, not regulation.

Two further documents set direction without validation detail: EMA's reflection paper on AI in the medicinal product lifecycle (current version published 30 September 2024), and the Guiding Principles of Good AI Practice in Drug Development from FDA's CDER and CBER with EMA (January 2026), which lists principles such as a risk-based approach, a clear context of use and life cycle management for industry to consider. Neither is a validation procedure. This page is not legal or regulatory advice; validation decisions belong to your quality function.

Step 1: Write the intended use statement

Everything else depends on this. FDA's 2018 data integrity guidance says a CGMP workflow is an intended use of a computer system to be checked through validation, that the extent of validation should be commensurate with the risk posed by the automated system, and that if you validate the computer system but not its intended use, you cannot know if your workflow runs correctly. Its example is a manufacturing execution system: qualifying the platform does not show that a given master production and control record it generates contains the correct calculations. For an agent, the platform is the model and the agent software; the intended use is the specific task you give it.

The CSA guidance makes a similar point at a finer level. It recommends examining the intended uses of individual features, functions and operations, because one piece of software can carry several uses with different risks, and documenting whether each is used as part of production or the quality system. An agent that drafts a deviation summary, looks up a specification and updates a tracking field has three operations, and they need not carry the same risk.

A useful intended use statement for an agent names the process step and the records it touches, the inputs it may use, the actions it can take and the tools it can call, the outputs it produces, who receives them, and what it must not do. The draft Annex 22 asks for this level of detail for the static, deterministic models it covers, and it excludes large language models, so here its principles are borrowed rather than applied: a detailed description of the intended use based on in-depth knowledge of the process, a characterisation of the input data including common and rare variations, identified limitations and erroneous or biased inputs, and approval by a process subject matter expert before acceptance testing starts. FDA's draft AI guidance uses a related term, context of use, meaning the specific role and scope of the AI model used to address a question of interest; that framework is aimed at regulatory decision-making for drugs and biologics, and it is a draft.

Step 2: Assess the risk of each operation

Annex 11 section 1 says decisions on the extent of validation and data integrity controls should be based on a justified and documented risk assessment, taking into account patient safety, data integrity and product quality. Its principle adds that where a computerised system replaces a manual operation, there should be no resultant decrease in product quality, process control or quality assurance, and no increase in the overall risk of the process.

CSA offers a practical test, written for devices. It treats a software feature, function or operation as high process risk when its failure to perform as intended may result in a quality problem that foreseeably compromises safety, which CSA defines as a medical device risk. Its examples of generally high-risk functions include those that measure, inspect, analyse or determine the acceptability of product or process with limited or no additional human awareness or review. Functions that collect and record data for monitoring and review without a direct impact on production or process performance, route CAPA records, manage data or automate an existing calculation are generally not high process risk. CSA asks manufacturers to consider failures that are reasonably foreseeable, not only likely. For drug and biologic CGMP, rate each operation against patient safety, product quality and data integrity as Annex 11 section 1 asks: a failure that corrupts a required record matters even where no safety consequence is foreseeable.

In our reading, the reasonably foreseeable failures for an agent include output that is plausible but wrong, an answer drawn from the wrong document, an action taken on the wrong record, and instructions hidden in input content that change what the agent does. Rate each operation against them. Where a trained reviewer checks the output before it can affect product or a required record, the risk usually falls; where the agent acts with no human review, it usually rises. The table is illustrative, not a CSA classification.

Agent operation (illustrative)Our illustrative reading (not a CSA classification)Why
Summarising batch documentation for a reviewer who reads the sourceLower, if the review is realA trained reviewer checks the output before any decision; in CSA's restocking example, a qualified person's check before use is why the manufacturer rated the feature intermediate, not high
Drafting a deviation or CAPA record that a person edits and approvesLower in most cases, but it touches data integrityBy analogy with CSA's CAPA routing example; the approved record is the person's, yet the draft still shapes a GMP record
Entering or calculating values in a required record without reviewHigher under drug CGMPThe agent creates or changes data a predicate rule requires, with no human check; this is a data integrity risk even where CSA's device-safety test alone might not rate it high
Deciding whether product or a test result is acceptable, with limited or no human reviewGenerally high in CSA's termsCSA names determining acceptability with limited or no additional human awareness or review as generally high process risk

Where the draft Annex 22 draws the line

The ratings above use CSA's logic only. The draft Annex 22 states that generative AI and large language models should not be used in critical GMP applications, meaning those with direct impact on patient safety, product quality or data integrity, and that in non-critical applications personnel with adequate qualification and training should always be responsible for ensuring the outputs are suitable for the intended use, a human-in-the-loop. In our reading, rows three and four fall within the draft's definition of critical use, and rows one and two may too, depending on whether the review makes the impact indirect; an operation CSA would not rate high may still touch data integrity. If the agent is built on a large language model and the use is critical, the draft's position is not to use it there, not to validate it more heavily. The draft is not adopted, and whether a given use is critical is a decision for your quality and regulatory functions.

Step 3: Choose risk-based testing, CSA style

CSA describes computer software assurance as a risk-based approach for establishing and maintaining confidence that software is fit for its intended use, with the burden of validation no more than necessary to address the risk. It names two families of testing. Scripted testing records test cases with expected results and can be robust or limited. Unscripted testing includes scenario testing, error guessing and exploratory testing; for exploratory testing CSA still expects high-level objectives with pass or fail criteria for each one. CSA says higher-risk functions may call for scripted testing or a hybrid of scripted and unscripted testing, and lower-risk functions may use unscripted methods, while noting that the categories are not exclusive. Annex 11 section 4.7 asks for evidence of appropriate test methods and test scenarios, with particular attention to parameter limits, data limits and error handling.

In our reading, agents suit this mix. Scripted cases check the known paths: the right inputs are retrieved, the right fields are drafted, prohibited actions are refused. Error guessing and exploratory testing look for the failures scripts miss: ambiguous instructions, missing documents, conflicting sources, unusual records, instructions injected in input content, and what the agent does when it cannot complete the task.

CSA also lets you take credit for controls that already exist, such as supplier evaluation, downstream verification, process monitoring and the system's own logs. For the record, it recommends capturing the intended use, the result of the risk analysis, a description of the testing, issues found and their resolution, a conclusion on acceptability for the intended use, who tested and when, and review and approval where appropriate. It favours digital records such as system logs and audit trails over screenshots.

Step 4: Set acceptance criteria for outputs that vary

This is where agents differ most from conventional software. The draft Annex 22 covers only static models with deterministic output, meaning identical inputs give identical outputs. It states that it does not apply to generative AI and large language models, and that its principles may be considered where applicable in non-critical uses. Neither Annex 11 nor the draft Annex 22 sets acceptance criteria for a large language model agent's output; what follows borrows principles, it does not apply a requirement.

The principles that transfer from the draft are these. Define test metrics that measure performance for the intended use, and set acceptance criteria for them, with a process subject matter expert responsible for defining them and approval before acceptance testing begins. Criteria may differ by subgroup of inputs. Acceptance criteria should be at least as high as the performance of the process the model replaces, which means knowing how well that process performs today; Annex 11's own principle of no decrease in quality points the same way. Test data should represent the full range of intended inputs, be large enough for the metrics to have adequate statistical confidence, be labelled through a process that ensures a very high degree of correctness, and be kept independent of anything used to build or tune the model.

Because the same input can produce different outputs, a single passing run proves little. One practical approach is to run each test case several times, judge each output against written criteria, and set the acceptance criterion on the results across runs and across the test set, with a separate, stricter criterion for the errors that matter most. Fix the model version, configuration and instructions for the test, record them, and treat any change to them as a change to the system. This is our reading of how the principles above can be applied; it is not language from FDA, the Commission or ISPE.

Step 5: Design the human review points

Review points belong in the intended use statement, not added afterwards. For the models it covers, the draft Annex 22 says that where a model gives an input to a decision made by a human operator and the effort to test the model has been diminished, the description of intended use should include the operator's responsibility, the operator's training and consistent performance should be monitored like any other manual process, and records should be kept, which, depending on the criticality of the process and how much the model was tested, may mean consistent review of every output according to a procedure. These clauses are written for static, deterministic models; here they are borrowed as principles.

A review point only counts if the reviewer can do it. In our reading, that means showing the reviewer the agent's sources and what it changed, giving them time and authority to reject, and training them on what the agent cannot check. Signing remains a person's act; see our answer on whether an AI agent can sign a GxP record. Whether the records involved fall under 21 CFR Part 11 is covered in our answer on Part 11 and AI tools.

Step 6: Define revalidation triggers and keep the agent in a validated state

Annex 11 section 10 says changes to a computerised system, including system configurations, should only be made in a controlled manner in accordance with a defined procedure, and section 11 asks for periodic evaluation to confirm that systems remain in a valid state. The draft revised Annex 11 adds that periodic reviews should verify whether the system remains fit for intended use and in a validated state, or whether re-validation, complete or in parts, is required. For the models it covers, the draft Annex 22 says any change to the model, the system or the process should be documented and evaluated to decide whether the model needs retesting, with any decision not to retest fully justified; it also asks for configuration control, monitoring of performance against the defined metrics, and monitoring of whether inputs stay within the intended sample space.

The trigger list that follows is our practice, not regulator text. Write the triggers into the validation plan in advance. Typical ones are a new model or model version, including one a supplier pushes; a change to the system prompt or instructions; a new or changed tool, permission or connected system; a change to the documents or data it retrieves from; a change to the process or records it supports; drift in input types or a fall in monitored performance; and a deviation or incident involving the agent, which Annex 11 section 13 expects to be reported and assessed. For each, the plan should say who assesses the change, what retest applies and how the decision is recorded. Annex 11 sections 3 and 4.5 expect suppliers to be assessed, and section 3.1 expects formal agreements with clear responsibilities, so the agreement should say how you will learn about model changes.

Where ISPE's GAMP material fits

CSA itself points to GAMP 5 (Second Edition) as an industry resource on software testing methods and validation approaches. ISPE's GAMP Guide: Artificial Intelligence, published in July 2025, applies the GAMP risk-based approach to AI-enabled systems; its public table of contents includes context of use and scope definition, an initial risk assessment, testing of AI-enabled computerized systems, system monitoring, operational change and configuration management, and periodic review. The guide is paid, and its validation framework is set out in the guide itself, not in ISPE's public material. Our separate answer summarises what ISPE has published about it.

The free validation plan outline

The free outline turns these six steps into a fill-in validation plan for one agent and one intended use: system description, intended use statement, risk assessment by operation, test strategy, acceptance criteria, human review points, traceability, change and revalidation triggers, periodic review and the validation conclusion. Each section names the source it draws on and whether that source is final, current, a draft or industry good practice. It is general guidance, not regulatory advice; using it does not make a system compliant, and it must be adapted to your own procedures and approved by your quality function.

Validation starts with knowing where AI already touches regulated work. Atlas Bio's Secure AI for Life Sciences starts with an AI risk assessment of current AI use and data exposure. Status, October 2026. In use in Atlas Bio's own agent platform today: a separate verification step that checks agent work, and a hold that stops a task until a person decides. Built and tested on synthetic data, not yet used with a client: the AI risk assessment tooling (AI discovery, data-flow and vendor-terms mapping, and a NIST AI RMF gap report) and a record of what data an AI used, what it produced and who reviewed it. In development: a customer-controlled AI environment in your own Azure tenant, an AI gateway, agent permissions, evidence linkage and notifications. Each engagement states which controls are built. Decisions on GxP scope, validation and compliance stay with your quality function.

Do I have to validate an AI agent the same way as any other GxP system?

The same principles apply: validation for the intended use, scaled to a documented risk assessment, under change control and periodic review. FDA's 2018 data integrity guidance and EU GMP Annex 11 do not exempt AI. What differs is how you set acceptance criteria for outputs that vary and how often model or prompt changes force a retest decision.

Does FDA's CSA guidance apply to drug manufacturers?

Its stated scope is medical device production and quality system software, so drug and biologic manufacturers can use its risk-based approach as context; their requirements come from drug CGMP. Our answer on Part 11 and AI tools covers how CSA relates to drug records.

Can an LLM-based agent be used in critical GMP applications in the EU?

The draft Annex 22 says generative AI and large language models should not be used in critical GMP applications, and that in non-critical uses qualified, trained people should be responsible for checking outputs. It is a draft, not adopted guidance; it shows the position the Commission put out for consultation in 2025.

Is a change to the agent's prompt a change that needs revalidation?

Treat it as a configuration change. Annex 11 section 10 says changes, including system configurations, should only be made in a controlled manner under a defined procedure. As a borrowed principle, the draft Annex 22 asks that any change be evaluated for retesting, with a justified decision if no retest is done. Whether a full, partial or no retest is needed depends on the risk assessment.

Who should approve the acceptance criteria?

For the models it covers, which exclude large language models, the draft Annex 22 says a process subject matter expert should be responsible for defining acceptance criteria, documented and approved before acceptance testing starts; that is a sound principle to borrow. Your quality function approves the validation plan and the conclusion that the agent is fit for its intended use.

  1. European Commission: EudraLex Volume 4, Annex 11 - Computerised Systems (in operation since 30 June 2011)
  2. FDA guidance: Data Integrity and Compliance With Drug CGMP - Questions and Answers (final, December 2018), full text (PDF)
  3. FDA guidance: Computer Software Assurance for Production and Quality Management System Software (final, issued 3 February 2026)
  4. FDA: Computer Software Assurance guidance, full text (PDF)
  5. European Commission: Stakeholders' consultation on EudraLex Volume 4 - Chapter 4, Annex 11 and new Annex 22 (consultation closed 7 October 2025)
  6. European Commission: Annex 22 Artificial Intelligence - consultation draft (PDF)
  7. European Commission: Revised Annex 11 Computerised Systems - consultation draft (PDF)
  8. FDA draft guidance: Considerations for the Use of Artificial Intelligence To Support Regulatory Decision-Making for Drug and Biological Products (January 2025, not for implementation)
  9. FDA (CDER, CBER) with EMA: Guiding Principles of Good AI Practice in Drug Development (January 2026)
  10. EMA: Reflection paper on the use of AI in the medicinal product lifecycle (current version published 30 September 2024)
  11. ISPE: GAMP Guide: Artificial Intelligence (product page, published July 2025)
  12. ISPE: GAMP Guide: Artificial Intelligence, public table of contents (PDF)
  13. ISPE: GAMP 5 - A Risk-Based Approach to Compliant GxP Computerized Systems (Second Edition)

Talk to Atlas Bio.

Sponsors, researchers and clinicians: tell us the decision you are trying to make and we will show what the platform can and cannot answer.

Contact Us →