Answers · By Hugh Donatello, Atlas Bio · Updated 2026-10-29

What Is AI Agent Governance in Pharma R&D and GxP?

What is AI agent governance in a life-science company? AI agent governance is the operating model that decides which agents exist, who owns each one, what it may touch and do, which actions need a person's approval, how its work is logged, tested and changed, and what happens when it fails. In a life-science company it connects those decisions to GxP and regulatory expectations, using frameworks such as the NIST AI Risk Management Framework's GOVERN, MAP, MEASURE and MANAGE functions.

What does GxP AI governance for AI agents require?

No regulation is written specifically for AI agents. In GxP work, governance means showing each agent is fit for its intended use under the applicable predicate rules and, where they cover the records involved, Part 11 and EU GMP Annex 11, commonly through a risk-based approach such as GAMP 5; inventorying agents and permissions; controlling changes; keeping audit trails; and sign-off by an accountable person before agent output is approved or relied on in GxP work.

Governance is an operating model, not a control list

Security controls stop an agent from doing harm. Governance decides, before that, whether the agent should exist, what job it has, who answers for it and how that answer is kept current. It covers decisions that no firewall makes: whether an agent may draft a deviation report, who reviews its output, when a model upgrade needs re-testing, and who can switch the agent off.

NIST AI 100-1, the AI Risk Management Framework (AI RMF 1.0, January 2023), organises this work into four functions: GOVERN, MAP, MEASURE and MANAGE. GOVERN applies across the others; it is where policies, accountability and culture sit. NIST AI 600-1 (July 2024) is a companion profile for generative AI that adds suggested actions for risks such as confabulation and information security.

ISO/IEC 42001:2023 takes a management-system approach. ISO describes it as specifying requirements for establishing, implementing, maintaining and continually improving an AI management system within an organization. One approach is to use the NIST functions to decide what to do and a management system to make sure it keeps being done.

This page is not legal or regulatory advice.

The eight parts of an agent governance programme

Each part below is anchored to the closest NIST AI RMF subcategory, some more loosely than others, so the programme can be explained to an auditor in a shared vocabulary rather than an internal one.

Governance elementWhat it answersNIST AI RMF anchor
InventoryWhich agents, models and tools are in use, and whereGOVERN 1.6: mechanisms to inventory AI systems
OwnershipWhich named person is accountable for each agentGOVERN 2.1: roles and responsibilities documented
Context of useWhat the agent is for, and the laws and expectations that applyMAP 1.1: intended purposes and context documented
PermissionsWhich data, systems and tools the agent may useMEASURE 2.7: security and resilience evaluated
Human approvalWhich actions need a person before they happenMAP 3.5: processes for human oversight defined
Logging and audit trailWhat the agent did, on what, under whose authorityGOVERN 4.3: practices for testing, incident identification and information sharing
Evaluation and testingWhether the agent does its job well enough for its useMEASURE 2.9: model explained, validated and documented
Incidents and shutdownHow failures are found, contained and learned fromMANAGE 2.4 (disengage or deactivate) and MANAGE 4.3 (incidents tracked and documented)

Inventory, ownership and context of use

Governance starts with knowing what exists. GOVERN 1.6 calls for mechanisms to inventory AI systems, and NIST AI 600-1 suggests enumerating generative AI systems for that inventory (GV-1.6-001) and defining any exemptions for generative AI embedded in application software (GV-1.6-002). For agents, the inventory should record the model and version, the tools the agent can call, the data it reads and writes, and the workflow it serves.

Every agent needs a named owner. GOVERN 2.1 expects roles and responsibilities to be documented and clear across the organization. The owner approves the agent's scope, signs off changes and is the person called when it misbehaves.

Context of use is where life-science governance differs most from general IT. MAP 1.1 asks that intended purposes and context-specific laws, norms and expectations be understood and documented. FDA's January 2025 draft guidance on AI to support regulatory decision-making for drugs and biologics describes a risk-based credibility assessment framework for an AI model in a particular context of use. EMA's reflection paper on AI in the medicinal product lifecycle, adopted in final form by CHMP on 9 September 2024, says it is the responsibility of the sponsor, applicant, holder or manufacturer to ensure algorithms, models, datasets and pipelines are fit for purpose and in line with GxP standards. The FDA document is a draft and not for implementation; the EMA paper is a reflection paper, not binding law.

Permissions and human approval

OWASP's Top 10 for LLM Applications 2025 names Excessive Agency as LLM06 and says its root cause is typically excessive functionality, excessive permissions or excessive autonomy. Its mitigations read like a governance checklist: minimise extensions and their functionality, avoid open-ended extensions such as running shell commands, minimise extension permissions, execute actions in the user's context, require a human to approve high-impact actions, and enforce authorization in downstream systems rather than relying on the model to decide.

Governance turns those into decisions recorded per agent: what it may read, what it may change, and which actions are consequential enough to need a person. MAP 3.5 asks that processes for human oversight be defined, assessed and documented in line with the organization's policies. In GxP work, signing or approving a record is always one of those actions; an agent can prepare and route, but an accountable person signs.

Logging, testing and change control

An agent you cannot reconstruct is an agent you cannot defend. Where agents create or change electronic records under FDA requirements, 21 CFR 11.10(e) already expects secure, computer-generated, time-stamped audit trails. Logging should capture the agent's identity, inputs, model version, tool calls, outputs and who accepted them. OWASP lists logging and monitoring of extension activity as a way to limit damage, while noting it does not prevent Excessive Agency on its own.

Testing should be tied to the context of use rather than to a generic benchmark. MEASURE 2.9 asks that the model be explained, validated and documented and its output interpreted in context. NIST AI 600-1 suggests keeping a document retention policy for the history of test, evaluation, validation and verification (GV-1.5-003).

Change control matters because agents change underneath you: a new model version, a new tool, a revised prompt. EMA's reflection paper says that for high patient risk and high regulatory impact use cases, all non-trivial changes in the software and hardware stack supporting a model, including key dependencies, require a re-evaluation of system performance. EU GMP Annex 11 section 10 says changes to computerised systems, including system configurations, should only be made in a controlled manner in accordance with a defined procedure.

How this fits GAMP 5 and FDA CSA

Where an agent works inside a GxP process, the agent, its model and the tools it calls become part of a computerised system that has to be shown fit for its intended use. A widely referenced industry guide for that work is ISPE's GAMP 5 Second Edition, A Risk-Based Approach to Compliant GxP Computerized Systems (July 2022). It asks knowledgeable, experienced subject-matter experts to apply critical thinking when choosing the approach, and its expanded appendices cover artificial intelligence and machine learning.

ISPE's GAMP Guide: Artificial Intelligence (July 2025) is designed to be used in parallel with GAMP 5. ISPE describes it as covering the development and use of AI-enabled computerised systems in GxP areas across their life cycle, including the responsibilities of internal and external suppliers, with risk-based effort as a stated aim.

FDA's Computer Software Assurance for Production and Quality Management System Software is final guidance (final September 2025, updated February 2026), issued by CDRH and CBER. It describes a risk-based approach to establishing confidence in software used in medical device production or the quality management system under 21 CFR Part 820. FDA guidance is not binding, and this one is scoped to device production and quality systems, not drug CGMP, so treat it as context for risk-based assurance in drug or biologic work.

In governance terms these documents connect to the elements above. The context of use and the risk an agent's work poses to patients, product quality and data integrity decide how much testing, documentation and human review it needs. The inventory and ownership records are where its validation documents start, and change control decides when a new model, prompt or tool needs re-testing. GAMP guides are industry guidance, not regulation; the regulatory requirements remain the predicate rules and 21 CFR Part 11 in the US and, for GMP computerised systems in the EU, the GMP rules including Annex 11.

Incident response and switching an agent off

MANAGE 2.4 asks that mechanisms be in place, with responsibilities assigned, to supersede, disengage or deactivate AI systems whose performance or outcomes are inconsistent with intended use. NIST AI 600-1 adds procedures for escalating generative AI incidents to the organization's risk management authority when criteria for deactivation are met (MG-2.4-002), and after-action reviews of incident response (GV-1.5-002). MANAGE 4.3 asks that incidents and errors be communicated to relevant parties and that tracking, response and recovery be followed and documented.

In practice that means each agent has a defined way to stop it, a person authorised to use it, criteria for when to use it, and a record of what happened that is preserved before anyone decides what it means. Third-party models and tools belong in the same process: GOVERN 6.1 asks for policies that address AI risks from third-party entities.

Agent governance works best when it is built into the environment the agents run in. Atlas Bio's Secure AI for Life Sciences starts with an AI risk assessment of current AI use and data exposure. Status, October 2026. In use in Atlas Bio's own agent platform today: a separate verification step that checks agent work, and a hold that stops a task until a person decides. Built and tested on synthetic data, not yet used with a client: the AI risk assessment tooling (AI discovery, data-flow and vendor-terms mapping, and a NIST AI RMF gap report) and a record of what data an AI used, what it produced and who reviewed it. In development: a customer-controlled AI environment in your own Azure tenant, an AI gateway, agent permissions, evidence linkage and notifications. Each engagement states which controls are built. Decisions on GxP scope and validation stay with your quality function.

What does GxP AI governance for AI agents require?

No regulation is written specifically for AI agents. In GxP work, governance means showing each agent is fit for its intended use under the applicable predicate rules and, where they cover the records involved, Part 11 and EU GMP Annex 11, commonly through a risk-based approach such as GAMP 5; inventorying agents and permissions; controlling changes; keeping audit trails; and sign-off by an accountable person before agent output is approved or relied on in GxP work.

How is agent governance different from AI security?

Security controls what an agent can do technically. Governance decides whether it should exist, who owns it, what it is for, which actions need a person, and how it is tested, changed and retired. Each needs the other.

Is NIST AI RMF mandatory for life-science companies?

No. NIST describes the AI RMF as intended for voluntary use. It is useful because it gives auditors, partners and staff a shared vocabulary; GxP obligations such as 21 CFR Part 11 and EU GMP Annex 11 apply separately where they cover the records involved.

Does GAMP 5 apply to AI agents?

GAMP 5 is industry guidance, not law. Where an agent is part of a computerised system used in GxP work, GAMP 5 Second Edition is commonly referenced, together with ISPE's GAMP Guide: Artificial Intelligence, when deciding how to show the system is fit for its intended use. Agents used only in non-GxP research are generally outside its scope.

Do we need ISO/IEC 42001 certification?

Not to govern agents well. ISO/IEC 42001 sets requirements for an AI management system; some organisations pursue certification so partners can rely on it, others use it as a reference model. The decision is a business one.

Who should own an AI agent?

A named person in the business function the agent serves, with authority to approve its scope and changes and to switch it off. IT or security can operate the controls, but accountability should sit with the process owner.

Where should a small biotech start?

With an inventory of every agent and AI tool in use, a named owner for each, and a written list of actions that always need a person. Those three steps make later testing, logging and change control possible.

  1. NIST AI 100-1, Artificial Intelligence Risk Management Framework (AI RMF 1.0)
  2. NIST AI 600-1, AI RMF: Generative Artificial Intelligence Profile
  3. OWASP GenAI Security Project: LLM06:2025 Excessive Agency
  4. ISO/IEC 42001:2023 - Information technology - Artificial intelligence - Management system
  5. FDA draft guidance: Considerations for the Use of Artificial Intelligence to Support Regulatory Decision-Making for Drug and Biological Products
  6. EMA: Reflection paper on the use of artificial intelligence in the medicinal product lifecycle
  7. eCFR: 21 CFR Part 11 - Electronic Records; Electronic Signatures
  8. European Commission: EudraLex Volume 4, Annex 11 - Computerised Systems
  9. ISPE GAMP 5: A Risk-Based Approach to Compliant GxP Computerized Systems (Second Edition, July 2022)
  10. ISPE GAMP Guide: Artificial Intelligence (July 2025)
  11. FDA guidance: Computer Software Assurance for Production and Quality Management System Software (final September 2025, updated February 2026)

Talk to Atlas Bio.

Sponsors, researchers and clinicians: tell us the decision you are trying to make and we will show what the platform can and cannot answer.

Contact Us →