Answers · By Hugh Donatello, Atlas Bio · Updated 2026-10-29

Prompt Injection Risk for AI Research Agents

What is prompt injection, and why does it matter for research AI agents? Prompt injection is input that alters what a language model does in ways its operator did not intend. Indirect prompt injection hides those instructions in content the model retrieves - a paper, a supplementary file, a web page. Research agents read exactly that kind of content and can act through tools, so an injected instruction can lead to data leaving the organisation or records being changed. OWASP says it is unclear whether fool-proof prevention exists, so design to limit the damage.

What prompt injection is

OWASP's Top 10 for LLM Applications 2025 lists prompt injection first, as LLM01:2025. It defines a prompt injection vulnerability as occurring "when user prompts alter the LLM's behavior or output in unintended ways", and notes that the input does not need to be visible or readable to a person, only parsed by the model.

OWASP distinguishes two types. Direct prompt injection comes from the person typing into the system, deliberately or by accident. Indirect prompt injection arrives through external sources such as websites or files: content that, when the model interprets it, changes the model's behaviour. NIST's Generative AI Profile (NIST AI 600-1) describes indirect attacks as adversaries remotely exploiting LLM-integrated applications "by injecting prompts into data likely to be retrieved".

NIST AI 100-2 E2025, Adversarial Machine Learning: A Taxonomy and Terminology of Attacks and Mitigations, explains why this works: generative models combine the data and instruction channels, so whoever controls a resource the system reads can influence what it does. NIST classes indirect prompt injection (NISTAML.015) separately from direct prompt injection (NISTAML.018) and notes that indirect attacks are mounted by a third party, often harming the model's primary user.

Why research agents are especially exposed

A research agent's job is to read material the organisation did not write. Literature PDFs, preprints, supplementary data files, vendor documents, regulatory web pages and partner reports all pass through it, and any of them could carry text addressed to the model rather than the reader. OWASP notes that multimodal systems add risk because instructions can be hidden in images that accompany benign text - relevant to agents that read figures.

The second factor is agency. NIST AI 100-2 observes that because agents act through tools, prompt injection can let an attacker hijack an agent to execute code or exfiltrate data from the environment it runs in, and that security research on agents is still in its early stages. An agent that only summarises can be misled; an agent that can also query internal systems and send messages can be turned against its own organisation.

In a research setting the consequences map to familiar concerns: unpublished results or assay data sent outside the boundary, a literature summary quietly skewed toward a conclusion, a draft record altered before review, or an agent's tools invoked for something its owner never intended.

Where injected instructions can enter a research workflow

The useful question is not whether a source is trustworthy but whether the agent treats it as data or as instructions. The table lists common entry points and what an injected instruction could try to do.

Entry pointHow instructions can be carriedWhat an injection could attempt
Literature PDFs and preprintsText hidden in the document, invisible to a reader but parsed by the modelSkew a summary or evidence assessment; trigger a tool call
Supplementary files and datasetsInstructions embedded in spreadsheets, annotations or metadataAlter extracted values; push the agent to fetch or send something
Web pages and search resultsHidden or off-screen text on a page the agent retrievesExfiltrate conversation or context data through a link or request
Vendor and partner documentsInstructions in a proposal, report or data packageRedirect the agent's task or request internal information
Internal knowledge bases used for retrievalA modified document returned for a routine queryOWASP's scenario: misleading results generated from tampered content

An illustrative scenario

Picture an agent that screens new papers on a target and drafts an evidence summary. One PDF it retrieves contains a line of text a reader would not see, telling the model to include the team's internal assay results in its next search query. If the agent holds a general web-fetch tool and read access to the internal results, the request leaves the building inside a URL. Nothing about the paper looks wrong to the scientist who later reads the summary.

This is an illustration, not a reported incident, but it follows the pattern NIST AI 100-2 describes: injection attacks that force a model to exfiltrate data by querying an attacker-controlled URL. Now change the design. The agent reads the literature folder but has no access to internal results, its only outbound tool is a fixed search interface that cannot carry arbitrary text, and anything sent elsewhere needs approval. The same hidden line now has nothing to work with.

Mitigations that limit the damage

OWASP is explicit that, given how generative models work, "it is unclear if there are fool-proof methods of prevention for prompt injection." NIST AI 100-2 says current mitigations do not offer full protection against all attacker techniques, and suggests designing systems on the assumption that prompt injection is possible whenever a model is exposed to untrusted input. The goal is therefore containment, layered as follows.

Why least privilege does more than filtering

Filters and detection try to recognise malicious text, and attackers adapt. OWASP's example scenarios include instructions written in multiple languages or encoded, for example in Base64, to evade filters, and malicious prompts split across parts of a single uploaded document so that no one fragment looks harmful. Least privilege works differently: it does not need to recognise the attack, because it limits what any successful attack can do.

That is why prompt injection and OWASP's LLM06 Excessive Agency are best handled together. An agent with read-only access to one project and no send rights can still be misled, but it cannot exfiltrate a dataset or rewrite a record. NIST AI 100-2 points the same way, describing designs that use multiple models with different permissions or that allow models to reach untrusted data only through well-defined interfaces.

Independent checking helps with the integrity side. A second process that re-verifies citations against the actual sources and recomputes stated numbers can catch a summary distorted by injected text, though it is a check on outputs, not a guarantee that an injection did not occur.

Research agents read untrusted content by design, so the damage an injected instruction can do has to be limited. Atlas Bio's Secure AI for Life Sciences starts with an AI risk assessment of current AI use and data exposure. Status, October 2026. In use in Atlas Bio's own agent platform today: a separate verification step that checks agent work, and a hold that stops a task until a person decides. Built and tested on synthetic data, not yet used with a client: the AI risk assessment tooling (AI discovery, data-flow and vendor-terms mapping, and a NIST AI RMF gap report) and a record of what data an AI used, what it produced and who reviewed it. In development: a customer-controlled AI environment in your own Azure tenant, an AI gateway, agent permissions, evidence linkage and notifications. Each engagement states which controls are built.

Is prompt injection the same as jailbreaking?

They are related. OWASP describes jailbreaking as a form of prompt injection in which the attacker causes the model to disregard its safety protocols entirely. Prompt injection is the broader category of inputs that alter model behaviour in unintended ways.

Does retrieval-augmented generation or fine-tuning prevent prompt injection?

No. OWASP notes that research shows RAG and fine-tuning do not fully mitigate prompt injection. Under OWASP's definition of indirect prompt injection, any retrieved document is a possible carrier of injected instructions, so adding retrieval adds places an injection can come from.

Can we stop prompt injection with a better system prompt?

A system prompt that constrains the model's role helps, and it is OWASP's first mitigation, but it is not sufficient on its own. Enforce limits in the systems the agent calls, not only in its instructions.

Should research agents be allowed to read external content at all?

Usually yes - reading literature is the point. The safer design is to let them read widely while restricting what they can do: narrow tools, no default send rights and human approval before anything leaves the boundary.

How would we know if an agent had been injected?

Often only through its actions or outputs, which is why logging matters. Record the content the agent read, its tool calls and anything it sent, and review unexpected outbound requests or unusual tool use.

  1. OWASP GenAI Security Project: LLM01:2025 Prompt Injection
  2. NIST AI 100-2 E2025, Adversarial Machine Learning: A Taxonomy and Terminology of Attacks and Mitigations
  3. NIST AI 100-2 E2025 (PDF)
  4. NIST AI 600-1: Artificial Intelligence Risk Management Framework: Generative Artificial Intelligence Profile
  5. OWASP GenAI Security Project: LLM06:2025 Excessive Agency

Talk to Atlas Bio.

Sponsors, researchers and clinicians: tell us the decision you are trying to make and we will show what the platform can and cannot answer.

Contact Us →