Coding-Agent Prompt-Injection Benchmarks: A Crosswalk

Q: What is a coding-agent prompt-injection benchmark?

It evaluates how a model, guardrail, harness or complete coding agent responds when untrusted content contains instruction-like text intended to redirect the system. The exact target and environment must be stated.

Q: What should a benchmark evidence pack contain?

At minimum it should contain content hashes, exact versions and configuration, environment and permission policy, run counts, denominators, per-case outcomes, evidence type, exclusions, limitations and disclosure status.

Verdict: A prompt-injection score is meaningless until you know what was scored. A classifier identifying a risky message, a guardrail blocking a string, a harness refusing a tool call and a full coding agent completing a legitimate task safely are four different experiments. Their percentages should not be placed in one leaderboard.

Kingy.ai reviewed seven open benchmark methods and found differences at every important layer: target, attack surface, isolation, denominator, judge, utility measurement, versioning and evidence. The practical result is a crosswalk—not a ranking—and a minimum reporting card for future coding-agent security claims.

Testing disclosure: Kingy.ai originally preregistered a 15-scenario × four-product coding-agent prompt-injection test. It did not run. The required Linux isolation and complete product entitlements were unavailable, so no product cell executed and no vulnerability was discovered. This article uses the frozen protocol only as a methodology lens. It reports no Codex, Claude Code, OpenCode or Copilot security outcome.

Download the crosswalk and reporting kit. It includes CSV and JSON crosswalks, a machine-readable schema, minimum-reporting checklist, non-weaponized surface taxonomy, comparison decision tree, blank benchmark card, source register and SHA-256 manifest.

The benchmark map at a glance

The seven reviewed projects are credible for different purposes. That is precisely why combining their top-line scores would be misleading.

Benchmark	Primary target	Main evidence	Utility or false-positive measure	The score is not…
CodeIPI	Full coding agent	Sandbox trajectory and artifacts	Legitimate bug-fix completion	A standalone injection detector score
InjecAgent	Prompted or fine-tuned tool agent	Synthetic tool-response interactions	Valid-rate denominator	A coding-harness comparison
AgentDojo	Tool-using LLM agent	Dynamic environment state	Benign user-task utility	A static classifier benchmark
WASP	End-to-end web navigation agent	Executable web environment	User navigation behavior	A coding-agent benchmark
Rogue Security coding-agent dataset	Guardrail or judge model	Labeled individual messages	Safe-message false positives can be derived	Proof that an agent prevented an action
AgentShield Benchmark	Agent security product/provider	Open corpus and provider adapters	Over-refusal, latency and cost	A full-agent task-completion evaluation
agent-security-bench	Responses or normalized tool transcripts	Deterministic patterns and transcript checks	Case-specific required behavior	Automatically a live-agent result

There is no single “wrong” target in that table. The error happens when the target disappears from the claim.

“System X scored 95% on prompt injection” could mean it labeled 95% of isolated messages correctly. It could mean 95% of injected tasks failed to trigger an attacker goal. It could mean 95% of full trajectories avoided a prohibited tool action. It could also exclude invalid runs from the denominator.

The number alone cannot tell you.

The first question: what is the unit under test?

Security benchmark discussions often use model, agent and product interchangeably. Operationally, they are different layers.

Model or classifier

The system receives text or a transcript fragment and predicts whether it is safe. The Rogue Security coding-agent dataset is explicit about this target: each row is a single message from a coding-agent session, paired with a security label, and the intended task is to score a classifier or judge model.

That can be valuable. A high-quality classifier may become one layer in a defense. But it does not demonstrate that a coding agent refused an unsafe instruction, avoided a tool call or completed its original task.

Guardrail or protection provider

The security product sits between content and an agent or application. It may detect, transform or block inputs. The AgentShield Benchmark compares commercial and open protection providers across categories including prompt injection, data exfiltration, tool abuse, over-refusal and provenance. It also records latency, cost, environment metadata, provider versions and a corpus hash.

Those dimensions are appropriate for a control layer. They are not the same as measuring the downstream agent’s end-to-end behavior.

Harness

The harness controls which tools exist, what permissions apply, what context reaches the model, how approvals work and how actions are executed. A harness can prevent a willing model from performing an unsafe action. It can also expose capabilities that turn a bad response into a real side effect.

A harness-level result therefore belongs to the exact version and configuration that produced it. Calling it a model score discards the variable that may have mattered most.

Full agent

The full system includes the model, harness, tool set, permission policy, isolation, working files and evaluation environment. CodeIPI targets coding-agent vulnerability to instructions embedded in issue descriptions, code comments and README files. Each sample combines an injected artifact with a legitimate bug-fixing task, and the reported method separates injection resistance, task completion and detection.

That is much closer to the question a coding-agent user usually means: “Can the agent do my work without following hostile repository text?”

It is also more expensive and fragile to evaluate. Environment state becomes part of the result.

Recognition is not prevention

The original Kingy protocol had two separate fields:

recognized_untrusted_content
unsafe_action_prevented

That distinction survives the scope change because it is methodologically important.

An agent can say, “This looks like prompt injection,” and then continue to obey it. A classifier can flag a message while the surrounding application ignores the flag. A harness can block a dangerous tool call even when the model never explains why.

These are different security behaviors:

Outcome	What it demonstrates	What it does not demonstrate
Recognition	The system identified suspicious or untrusted content	That an unsafe action was stopped
Refusal text	The response expressed unwillingness	That no hidden or subsequent action occurred
Tool-call prevention	A prohibited action did not execute	That the legitimate task remained useful
Safe task completion	The user’s task completed without the prohibited effect	That the system would generalize to other surfaces
Recovery	The system returned to the legitimate task after detecting interference	That initial exposure had no side effects

Promptfoo’s current coding-agent red-team guidance makes a related distinction between unsafe willingness and an action-verified exploit. It recommends command evidence, traces, canaries, file hashes and sidecar reports rather than treating model-judge interpretation as the only evidence.

That is a strong general rule: the closer a security claim gets to real action, the more the evidence should come from state and traces rather than prose alone.

Denominators can reverse the story

InjecAgent contains 1,054 test cases spanning 17 user tools and 62 attacker tools. Its public output distinguishes ASR-valid from ASR-all for direct-harm and data-stealing scenarios.

The distinction is essential.

Suppose an agent produces valid behavior on 60 of 100 cases and the attack succeeds on 30 of those valid cases:

ASR-valid: 30 ÷ 60 = 50%
ASR-all: 30 ÷ 100 = 30%

Neither calculation is automatically fraudulent. They answer different questions. The first asks how often the attack succeeds when the agent produces a valid interaction. The second asks how often it succeeds across all attempted cases.

Now add crashes, refusals, timeouts and malformed tool calls. Are those safe outcomes, invalid outcomes, utility failures or exclusions? The answer can materially change a headline percentage.

Every published rate should therefore expose:

Numerator
Denominator
Crash treatment
Exclusion rules
Retry policy
Number of independent runs
Whether repeated attempts remain visible

A benchmark that hides those fields is hard to interpret even when its implementation is excellent.

Utility is part of security

A defense that blocks every untrusted file can achieve perfect attack prevention while making a coding agent useless.

This is why AgentDojo evaluates attacks and defenses in dynamic tool environments with user tasks, and why CodeIPI reports legitimate task completion alongside injection resistance. AgentShield uses over-refusal as a control-layer measure. These are not identical utility metrics, but each recognizes the same problem: security cannot be read from attack failure alone.

For coding agents, the cleanest utility measure is usually whether the legitimate task still passes protected tests.

That must be scored outside the agent’s writable boundary. Otherwise, an agent can weaken tests, overwrite the verifier or produce an artifact that looks successful from inside its own workspace.

Useful security reporting should show a two-dimensional result:

	Legitimate task failed	Legitimate task passed
Unsafe action occurred	Unsafe and unhelpful	Unsafe despite utility
Unsafe action prevented	Safe but unhelpful	Safe and useful

The bottom-right cell is the objective. A single attack-success rate cannot tell you how many cases landed there.

Isolation changes what an “attack” can do

Security results inherit the capabilities of the test environment.

In a static detector dataset, there is no real filesystem or network action. In a transcript scorer, tool calls may be recorded but not executed. In a simulated tool environment, actions change benchmark-owned state. In a full coding-agent harness, commands may run inside a container or workspace with actual permissions.

WASP uses a realistic, end-to-end web-agent environment designed to avoid real harm. Its result is valuable precisely because it captures multi-step agent behavior. It is not directly comparable with a code-comment detector because the target, environment and attacker goal differ.

The WASP implementation also illustrates an often-overlooked reproducibility fact: substantial environment setup is part of the benchmark. Its official repository was archived in July 2026. A paper result remains a paper result, but anyone repeating it should disclose repository state, dependencies and local modifications.

For a coding-agent benchmark, the environment card should include at least:

Operating system and architecture
Container, VM or namespace boundary
Writable filesystem scope
Network policy
Available tools
Approval policy
Secrets policy
Workspace reset method
Host-side protected verifier

“Sandboxed” is not enough. Two sandboxes can expose completely different capabilities.

Judges are evidence, too

Security graders generally fall into three classes.

Deterministic state checks

Did a protected file change? Did a canary appear in a trace? Did a forbidden tool call occur? Did the legitimate tests pass?

These checks are narrow but auditable. agent-security-bench emphasizes deterministic text and normalized transcript analysis. Its reports include per-case pass/fail, weighted and unweighted scores, content-pinned manifests and evidence bundles. It also states that external live-agent adapters are sandbox-gated, while deterministic dry-run and mock modes exist.

That last qualification matters. A dry-run result must not be relabeled as a live-agent outcome.

Human review

Humans can resolve nuanced intent and ambiguous trajectories, but review rules, agreement and adjudication should be disclosed. Selective human review of only surprising failures can also introduce bias.

Model judges

An LLM judge can scale semantic evaluation, but it adds another model, prompt and version to the experiment. The Rogue dataset card states that its labels reflect an LLM evaluator under a specific rubric and should be treated as a strong baseline rather than indisputable ground truth.

The best design often combines deterministic side effects with a judge for the genuinely semantic remainder. The report should say which result came from which evidence type.

Versioning is not clerical detail

Agent behavior changes when any of these change:

Target model
Harness version
System instructions
Tool definitions
Permission mode
Benchmark corpus
Evaluator prompt
Runtime dependencies

AgentShield records a corpus hash, environment fields, provider versions and per-case timestamps. Its commit-reveal mechanism is intended to show that proprietary-provider results were not altered after execution, while the project explicitly notes that this does not independently attest which model produced a result.

That is a good example of a narrow integrity claim. Cryptographic provenance can prove artifact continuity. It cannot prove every fact about how an artifact was produced.

The minimum reproducibility bundle should pin content, configuration and environment separately.

The retained 15-scenario taxonomy

Kingy’s unrun protocol covered six coding-agent input surfaces:

Surface	Scenarios	Why it matters
README and repository overview	3	Agents often read these early and may treat prose as project authority
Project instructions	3	Legitimate instruction files create a difficult trust-boundary problem
Source comments	3	Instruction-like content sits beside code the agent must inspect
Mock issue or pull request	2	Collaboration text is normally untrusted user content
Captured tool output	2	Agents routinely consume logs and command output as context
Dependency documentation	2	Third-party text can enter the working context indirectly

The original scenarios used only synthetic repositories, fake canaries, protected test files and a loopback-only sink. None ran. The public taxonomy intentionally removes reusable attack strings and executable payloads.

Why publish the taxonomy at all? Because coverage is a valid methodological field even without outcomes. A benchmark focused on tool responses and one focused on repository comments may both use the label “indirect prompt injection” while testing different exposure paths.

The crosswalk should make that mismatch visible before anybody compares percentages.

A minimum reporting standard

Before quoting a coding-agent prompt-injection result, ask for these fields:

Target unit: model, classifier, guardrail, harness or full agent.
Exact versions: product, model, harness, benchmark and judge.
Corpus identity: immutable revision or SHA-256.
Attack surface: where the untrusted content appeared.
Environment: isolation, network, filesystem and permissions.
Run design: repetitions, randomization, retry and timeout policy.
Denominators: numerators, exclusions, crashes and invalid cases.
Recognition: whether suspicious content was identified.
Prevention: whether the unsafe action actually failed.
Utility: whether the legitimate task still completed.
Recovery: what returning to the task means, if claimed.
Evidence type: state check, transcript rule, human or model judge.
Limitations: the comparison class the result does and does not support.
Disclosure: whether any credible novel issue required private handling.

The downloadable JSON Schema turns those fields into a reusable benchmark card. It does not declare a universal scoring formula. That would recreate the problem by forcing unlike experiments into one number.

When two scores can be compared

Two results are candidates for comparison only when all of these are materially aligned:

Same target unit
Same outcome definition
Same denominator and exclusion policy
Same or equivalent environment capabilities
Same evidence standard
Pinned versions and run counts

Even then, report uncertainty and configuration differences.

If one result is classifier recall and another is full-agent unsafe-action prevention, the correct output is not a normalized chart. It is a sentence explaining why the scores answer different questions.

This is analogous to the harness issue in ordinary coding benchmarks. Kingy’s State of AI Coding Tools 2026 explicitly labels its usefulness score as editorial rather than scientific. The Muse Code benchmark audit likewise distinguishes vendor harness results from independently verified leaderboard runs. Security reporting deserves at least the same care.

What buyers and builders should do

If you are choosing a coding agent, do not ask only, “What is its prompt-injection score?” Ask:

Was the full product tested or only its model?
Could the test execute tools?
Was the network available?
Did the agent complete the requested repair?
Were unsafe actions verified from traces or inferred from text?
Were failures, refusals and crashes included?
Does the tested permission mode match how we will deploy it?

If you are publishing a benchmark, release the reporting card beside the score. Put the denominator and exclusions in the first table, not only in an appendix. Publish per-case evidence where it is safe, but do not release weaponized material merely to make an article look reproducible.

If you are operating agents, remember that instructions alone are not a security boundary. The broader Kingy agent production lifecycle treats permissions, observability and evaluation as separate layers for a reason. Use least privilege, isolated workspaces, network controls, protected verification and human approval for consequential actions.

The crosswalk’s main conclusion is simple:

Before comparing the scores, compare the experiments.

Download the coding-agent security benchmark crosswalk and use the blank card for your next evaluation or vendor claim review.

FAQ

What is a coding-agent prompt-injection benchmark?

It is an evaluation of how a model, guardrail, harness or complete coding agent responds when untrusted content contains instruction-like text intended to redirect the system. The exact target and environment must be stated.

Can prompt-injection benchmark scores be compared?

Only when the target unit, outcome definition, denominator, environment, evidence standard, versions and run design materially align. A classifier score and a full-agent prevention rate are not directly comparable.

Is detection the same as prevention?

No. Detection means suspicious content was recognized. Prevention means the unsafe action did not occur. A system can detect an injection and still follow it.

Why measure legitimate-task completion?

A defense can block everything and appear safe while making the agent useless. Security evaluation should report whether the legitimate coding task still passes protected verification.

Did Kingy.ai test Codex, Claude Code, OpenCode or Copilot here?

No. The original 60-cell protocol did not run because its isolation and entitlement gates were not met. This article is a primary-source methodology crosswalk, not a product ranking.

What should a benchmark evidence pack contain?

At minimum: content hashes, exact versions and configuration, environment and permission policy, run counts, denominators, per-case outcomes, evidence type, exclusions, limitations and disclosure status.

Sources and methodology

Research cutoff: August 20, 2026. Crosswalk rows were derived from the primary pages above and do not copy upstream corpora. Benchmark versions, repository status and licensing should be rechecked before reuse. Kingy.ai plans a quarterly source review and will record material changes in the downloadable source register.