Prompt Injection Detection in Production LLM Agent Regression Tests

Prompt injection in an agentic pipeline is not a variant of the chatbot problem, it is a different problem with a different failure surface. Every point where an agent retrieves or processes outside content, a tool call, a document fetch, a memory read, becomes a place where an attacker can plant instructions the model later treats as trustworthy. This piece lays out why launch-time testing cannot catch that, and what a regression suite built from production traces needs to look like to actually catch it.

Prompt injection defense in agentic pipelines versus chatbots

A chatbot has one real channel for attack: the text a user types into the box. Direct injection, where a user tries to override the system prompt, is a known quantity by now, and pattern classifiers catch a reasonable share of it. Agents don't have one channel. They have as many channels as they have tools, retrieval sources, and memory stores, and the harder problem lives in indirect injection, where the malicious instruction never comes from the user. It arrives inside a web page the agent fetches, a document a retrieval step pulls in, or a response from a third-party API the agent calls on the user's behalf, and the agent reads that content as part of its own trusted context.

OWASP's guidance on LLM01:2025 states that the stochastic nature of language models means no single technique can guarantee complete mitigation. That's an architectural instruction: defense has to happen in layers, because no one layer holds on its own. For an agent, those layers have to cover at least four distinct entry points, and each one behaves differently. Tool descriptions get loaded into context at the start of a session and can carry instructions the model never questions. Tool output comes back mid-run and gets treated as trusted information, even though it is untrusted input. Retrieval results from a RAG pipeline bring in whatever text sits in the index, regardless of who wrote it. Persistent memory stores carry content from one session into the next, so a payload you plant today can activate a week later. A defense tuned against one of these paths says nothing about the other three.

Khodayari et al. (arXiv:2604.27202, April 2026) give this a scale that's hard to dismiss. Put the two findings together: a defense validated on a short, clean benchmark tells you almost nothing about how an agent behaves against the kind of content it will actually retrieve in production. Production behavior is the only ground truth that counts.

How prompt injection failure looks inside an agent trajectory

When an indirect injection succeeds, it leaves a recognizable shape in the agent's trajectory. The early steps look exactly like a normal, legitimate run. Then an external observation comes back, a tool response, a fetched document, a retrieved chunk, and the injected instruction rides in inside it. From that point forward, the steps the agent takes start serving whatever the injected text told it to do. That shift, benign prefix, poisoned observation, compromised suffix, is the pattern a regression test needs to catch. You can't just scan a log somewhere for text that looks malicious. The thing to detect is the behavioral change that text produces across the whole run.

What makes this dangerous for engineering teams specifically is that the pattern doesn't stay fixed once you've tested for it. Three kinds of harness change can reopen a vulnerability that was closed at launch. Prompt changes are the most common: an edit made for an unrelated reason, tightening a response format or adding a new instruction, can quietly weaken the constraint that used to stop a particular injection from taking hold. Retrieved content drift is the third: the corpus an agent queries is not static, and a document indexed into a RAG store after launch can carry a payload that simply didn't exist when the agent was first tested.

A further complication is non-locality. If you poison an observation at step N, you don't always see a failure at step N. It can take three or five more steps before the compromised behavior appears, and by the time it does, the trace near the failure carries almost no diagnostic signal pointing back to where the problem started. An agent that was safe at launch can become exploitable later for reasons that have nothing to do with the model changing. The prompt changed, or the tool changed, or the documents it retrieves changed. That's the reason a one-time launch check is structurally incapable of catching injection drift: the vulnerability moves with the engineering, not with the model weights.

Synthetic attack benchmarks as an insufficient regression line

Synthetic injection benchmarks have a real use: they tell you whether an agent resisted a known set of attack patterns at a single point in time, under conditions a researcher designed. What they cannot tell you is whether that resistance holds once the harness around the agent keeps changing, because a synthetic benchmark has no connection to the actual prompts, tools, and retrieved content the agent runs against in production.

LongPIBench makes the gap concrete. If a defense looks solid on short-context benchmarks, it can still break down under the long-context conditions real RAG pipelines and document-processing workloads create, so the benchmark score and the production outcome end up measuring two different things. The web-scale study from Khodayari et al. adds a second angle on the same problem: injection content in the wild is highly structured, with 54 recurring prompt templates accounting for the overwhelming majority of confirmed cases. Those templates reflect what actual site owners and actual adversaries do, which is not the same distribution a research team curates when building an adversarial test set.

A benchmark can report a clean pass even if tool-output injection was never in scope, memory poisoning was never represented, and the specific tool schema the agent actually uses in production was never tested. The deeper mismatch is that synthetic benchmarks test a model's resistance to attack content in isolation, while a real injection in production exploits the interaction between that content and the specific harness configuration it meets, the prompt template in use, the tool set available, the retrieval corpus indexed at that moment. Every time one of those three things changes, you get a configuration that no benchmark run has ever tested. Continuous testing against the live configuration is the only way to keep the attack surface mapped as it actually is.

What production traces capture that synthetic tests miss

A production trace records the real interaction between the agent's harness and real external content, so you get the injection surface and the agent's behavioral response in the same artifact. That combination is what a synthetic test can only approximate. A trace useful for injection regression needs to capture the full tool-call sequence, the arguments passed into each call, the content each tool returned, the documents a retrieval step surfaced, the model's intermediate reasoning, and the final action taken, all stitched together into one record that can be replayed later exactly as it happened.

That structure makes two things possible that no synthetic test can offer. The first is locating the actual injection point, the specific step where compromised content entered the trajectory, instead of guessing at it backward from a bad final outcome. The second is replaying the exact configuration active at the moment of the run, not a generalized stand-in for it, which matters given how much a prompt edit or a tool schema change can shift behavior.

Production traces also show what real adversaries are actually trying to do against a specific agent; researchers can only anticipate what they might try. The web-scale study identified six distinct objective categories behind real injection content: System Disruption or Degradation, Reputation Manipulation, Data Exfiltration, Data Protection, AI-Bot Identification, and Generic Content Override. A regression suite needs to reflect the mix its own agent actually faces, and only production traffic shows what that mix is.

Detection accuracy depends on reading the whole trajectory rather than scanning individual steps for suspicious keywords. Research on trajectory classification backs this up directly: methods that read the full tool-call sequence substantially outperform methods that check steps one at a time, and a surface-level keyword baseline catches only 11.1% of partial hijacks in the AgentDrift dataset. The gap is widest exactly where non-locality makes detection hardest, in partial hijacks and delayed executions where the injection point and the eventual bad behavior sit several steps apart. Traces also capture the cases where nothing went wrong, where the agent encountered injection content and handled it correctly. Those negative examples matter as much as the failures, because they keep a detection system calibrated and keep engineers from getting flooded with alerts on completely normal behavior.

Building an injection-specific regression suite from production traces

A working regression suite needs three pieces: a process for seeding it with candidate cases from live traffic, a labeling protocol that confirms what each case actually shows, and a replay harness that runs the current agent configuration against the confirmed cases.

Seeding starts with the highest-value material: traces where the agent took an action inconsistent with what the user asked for, especially right after a tool call or a retrieval step. These confirmed-compromise traces anchor the suite. Near-miss traces matter almost as much, cases where the agent hesitated, backtracked, or took an unusual number of steps right around an external content ingestion point. The web-scale study's finding that roughly 70% of real injection content sits in non-rendered HTML matters directly here: an agent parsing structured web content will run into payloads that leave no trace in the rendered output an engineer glances at during review. The trace is the only place that content is visible.

Labeling needs to work at the level of individual steps, not just a single verdict for the whole run. Each step should get classified as benign, injection point, hijacked, or failed injection, because a trajectory-level label of "compromised" on its own gives no basis for fixing anything. Confirmed-safe traces, runs where the agent met injection content and didn't deviate, become the negative examples that keep the suite calibrated against false alarms.

The replay harness itself needs to store, for each case, the exact system prompt version active at the time of the run, the tool schemas active at that time, the external content that triggered the case (or a faithful reproduction of its structure), and the expected trajectory and final action. The suite should be versioned alongside the harness itself, so that a prompt change, a tool schema update, or a retrieval index update each triggers a replay of the case subset tied to that change, not only the cases someone happened to tag by hand. Evaluation should split along two methods: deterministic checks for clearly-defined failure modes, did the agent call a tool it shouldn't have, did its output match a disallowed pattern, and LLM-as-judge evaluation for the subtler behavioral deviations where a strict match fails to capture what actually went wrong. Coverage reporting has to stay honest about what the suite actually tests: which injection categories, direct, tool-output indirect, RAG-indirect, memory poisoning, have real regression cases, and which don't. If a suite has zero memory-poisoning cases and passes cleanly, that says nothing about resistance to memory poisoning. It says only that the question was never asked.

Triggering regression runs as the harness evolves

A regression suite that only runs at release time is a snapshot, not a regression line. The value comes from the triggers that fire it continuously as the harness changes, and most of those triggers map onto infrastructure teams already have: CI/CD pipelines, schema registries, retrieval index versioning.

If you change a prompt template, run the full suite, because prompt edits are the single most common way a previously-neutralized injection pattern becomes exploitable again. If an MCP server gets updated, a new server connects, or an existing one's tool descriptions change, treat it exactly like a tool schema change, because MCP tool descriptions enter the model's context as trusted material the moment discovery happens. A model version change should also trigger the full suite, because a new model version can change instruction-following behavior in ways its own aggregate benchmark scores won't predict.

A separate trigger matters for a different reason: the external content environment shifts on its own, without any engineering change. None of that appears in a change log, so a scheduled run, a full replay of the suite against current production on a weekly cadence, catches drift that the world introduced on its own. Without that scheduled check, the suite only catches the regressions engineers themselves created. New cases should keep entering the suite from live traffic as they're confirmed, so the case distribution tracks the actual threat landscape the agent faces rather than freezing at whatever it looked like on the day the suite was built.

Attributing injection failures to the harness layer that caused them

Catching a regression is only half the job. If it traces to retrieval, the fix belongs in the pipeline that indexes and filters content before it ever reaches the model's context.

Treating every injection failure as one generic category, "the agent got hijacked," erases the information that would actually prevent it from happening again. The specific layer that failed, prompt, tool schema, or retrieval, determines where the engineering effort goes next, and a regression suite built from real production traces makes that attribution possible.