Benchmarks
Measured on public data. Every figure read from a committed benchcard.
A guardrail is only as good as the number behind it — so we grade ours against public, labeled, third-party datasets, never hand-authored samples. Every figure on this page is rendered from a benchcard the harness wrote at run time (dataset id + row count + date; the upstream revision where the card records one), through a generated index that CI checks against the cards. Where we can't yet grade a pillar on public data, we say so instead of showing a number.
Honest protocol — train/test dedup by normalized text, threshold picked on a held-out val fold, scored once on each held-out public test split. The xTRam1 and deepset test splits are same-distribution held-out splits: their train splits (8,236 + 546 rows) are in the student's 9,492-row training pool. Sources it never trained on are scored separately (out-of-sample, below).
And the one we lose
AgentDojo agent-trajectory replay — 43.1% replay-ASR against a recorded undefended floor of 47.7% from the same bytes, with the cause named and the card's own caveats printed. Read it.
And what the market publishes
7 of 11 rival figures on record here cannot be re-run by anyone outside the publisher. The register.
F1 98.0% on a same-distribution held-out split.
on 2,060 xTRam1-test rows — a same-distribution held-out split: xTRam1-train (8,236 rows) and deepset-train (546 rows) are in the student's 9,492-row training pool
First-party logistic student — word + character n-gram features over engineered injection cues. ~570 KB minified (~840 KB on disk) JSON, no GPU. Ships on install as the default injection detector at threshold 0.46.
Latency (detector-slm-vs-student card, logistic student): p50 0.091 ms / p99 0.601 ms on typical prompts (median 92 chars); p50 0.614 ms / p99 2.354 ms on long prompts (median 1,958 chars). Measured on Intel(R) Xeon(R) Processor @ 2.10GHz (4 cores), as recorded by the card.
| tier | xTRam1-test F1 (n=2,060, same-distribution held-out) | deepset-test F1 (n=116, same-distribution held-out) |
|---|---|---|
| regex heuristic (fail-soft floor) | 14.0% P 100 / R 7.5 TP 49 · FP 0 · TN 1,410 · FN 601 | 0% P 0 / R 0 TP 0 · FP 0 · TN 56 · FN 60 |
| ★ logistic student (shipped default) | 98.0% P 98.4 / R 97.5 TP 634 · FP 10 · TN 1,400 · FN 16 | 65.2% P 100 / R 48.3 TP 29 · FP 0 · TN 56 · FN 31 |
Shipped-default row: the studentCommitted row in deepset-injection-fullstack.test.json was scored at threshold 0.5 (the bench's fixed cut for the committed model), while the shipped artifact uses 0.46. The model file's embedded meta reports a different held-out figure measured at 0.46; pending a re-run, the card is the cited source.
Out-of-sample (sources the student never trained on)
| set | shipped student F1 (detector-slm-vs-student) |
|---|---|
| Tensor Trust hijack/extraction + Alpaca (n=2,100) | 84.3% P 93.9 / R 76.5 TP 459 · FP 30 · TN 1,470 · FN 141 |
| in-the-wild jailbreaks + Alpaca (n=1,940) Caveat: jailbreak positives (verazuo/jailbreak_llms 2023-12-25) ARE de-duplicated against the student's training pool: 160 overlapping positives were dropped before scoring (lowercase + collapsed whitespace (jailbreak-bench normKey)), so this figure is genuinely out of sample. The earlier, higher number on this row was measured without that step. | 88.2% P 92.5 / R 84.3 TP 371 · FP 30 · TN 1,470 · FN 69 |
30 false positives on 1,500 benign Alpaca prompts in each set (specificity 98.0%).
deepset-test is small (116) and partly non-English (German jailbreaks, roleplay paraphrases). The student is precision-strong there (100%) but recall-bound (48.3%) — the remaining misses are the hardest multilingual/novel attacks, which the optional operator DeBERTa upgrade tier targets. The 98.0% headline is the large held-out split of a dataset whose train split the student learned from; the out-of-sample rows below are the harder test.
- xTRam1/safe-guard-prompt-injectionApache-2.0— train + held-out test — 2,060 test rows scored; 8,236 train rows in the pool
- deepset/prompt-injectionsApache-2.0— train + held-out test — 116 test rows (hard, partly non-English); 546 train rows in the pool
- Lakera/gandalf_ignore_instructionsMIT— train only — 777 real-world positives
- HumanCompatibleAI/tensor-trust-dataGitHub, commit-pinned URL— positives — hijacking + extraction robustness sets
- verazuo/jailbreak_llmsGitHub, commit-pinned URL— positives — jailbreak prompts, 2023-12-25 snapshot (de-duplicated vs training pool)
- tatsu-lab/stanford_alpacaGitHub, commit-pinned URL— clean benign — 1,500 instructions
92.8% F1 on real in-the-wild jailbreaks.
88.9% coverage of real DAN-style jailbreaks · 99.2% specificity on xTRam1 benign · 98.0% on out-of-training Alpaca benign
The SAME shipped injection student, scored with no retraining on a DISTINCT, harder attack family: real in-the-wild jailbreaks (DAN roleplay, persona-hijacks) scraped from Discord/Reddit. Rows overlapping the training pool are dropped first: 378 of 800 fetched jailbreaks (47%) overlapped it and were dropped, leaving 422.
| tier | jailbreak coverage (recall) (n=422) |
|---|---|
| regex heuristic (fail-soft floor) | 5.9% |
| ★ logistic student (shipped default) | 88.9% |
The dataset's own “regular” split is NOT clean-benign — a large fraction is unlabeled jailbreaks (“As DAN…”, “developer mode on”), and our detector flags 415/800 of it (the card records only that it tripped, not how many are true jailbreaks). So we measure specificity on genuinely-clean benign sets instead — and report the finding rather than bury it. Combined F1 92.8% (P 97.2 / R 88.9).
- TrustAIRLab/in-the-wild-jailbreak-promptspublic (CCS'24) · rev a10aab8 recorded (not pinned)— 422 confirmed jailbreaks kept after dropping 378 train-pool overlaps — Shen et al. “Do Anything Now”
100% precision — hits land on real labeled PII.
structured recall 72.8% · regex hits scored against 6,324 labeled spans
scanPii — regex + Shannon-entropy recognizer for STRUCTURED PII (email · phone · SSN · credit-card · IPv4). Zero-I/O, first-party. The structured floor; unstructured PII (names, orgs, locations) is the NER tier's job (ner-pii.ts, Presidio-class).
Precision is 100% as recorded (the card rounds to one decimal): regex hits land on truly-labeled spans. Credit-card recall is low because ai4privacy's card numbers are synthetic (invalid IINs, fail Luhn), which no real card detector flags. Overall recall across ALL 6,324 spans is 9.3% by design: the regex floor targets the 599 structured spans; names/orgs/locations are the operator NER tier.
- ai4privacy/pii-masking-200kpublic · rev 7c22e68 recorded (not pinned)— span-labeled test — 2,000 English rows / 6,324 spans
100% specificity — it doesn't cry wolf on your codebase.
0 false alarms across 2,100 real OSS files
scanSecrets — provider token patterns (OpenAI · Anthropic · AWS · GitHub · Stripe · Slack · JWT) plus a Shannon-entropy gate on generic high-entropy tokens. Zero-I/O, first-party.
Specificity on REAL open-source code a team would paste through the gateway: it fired on 0 of 2,100 files. Reported as a flag rate, not a claimed FP rate — a public file could carry a genuine committed secret. Recall is a separate axis, defined by the covered provider formats.
- google/code_x_glue_ct_code_to_textpublic (CodeXGLUE) · rev 2678080 recorded (not pinned)— negative corpus — 2,100 real GitHub functions, 6 languages
recall@10 42.8% on LoCoMo — the set Mem0 and Zep publish on.
gold-evidence retrieval on LoCoMo · ~602 turns/question · CPU, no LLM
The shipped hybrid retriever (multiSignalSearch): semantic trigram vectors + BM25 keyword + entity overlap, RRF-fused — the zero-dependency CPU recall path. Each dialog turn is ingested as a memory; we rank against the question and check whether LoCoMo's annotated evidence turns land in the top-k.
| k | recall@k | hit@k |
|---|---|---|
| 1 | 13.6% | 15.1% |
| 5 | 33.7% | 36.9% |
| ★ 10 | 42.8% | 46.9% |
| 20 | 50.8% | 56.0% |
This is a RETRIEVAL number (does the evidence turn surface?), a different axis from the end-to-end QA accuracy Mem0 and Zep headline — that needs an LLM to answer and a judge to grade. A measured surprise: swapping in a learned sentence-embedder (all-MiniLM-L6-v2 (ONNX, operator tier)) does NOT lift raw-turn retrieval — dense gets 42.6% recall@10 vs this 42.8% — because LoCoMo's fine-grained factual questions favour BM25/entity matching on rare tokens (names, dates) over dense similarity on short, context-dependent turns. The expected lift is LLM fact-extraction (turning turns into atomic memories first), the same LLM-budget axis as the end-to-end score. So 42.8% is an honest ceiling for extraction-free retrieval with these embedders.
End-to-end QA: a partial full-context baseline run exists (1 of 10 conversations, 152 QA, self-judged qwen2.5:7b-instruct; it scores the full-context baseline, not the memory stack); not published as a score.
- snap-research/locomopublic (ACL 2024) · GitHub, main branch (no revision recorded)— 1,977 QA with gold evidence over 10 long multi-session conversations
98.8% in-scope recall on real reverse-shell payloads.
0 false alarms on 2,100 real OSS files · real reverse-shell payloads
scanCode — a CodeShield-class scanner (à la Meta CodeShield in LlamaFirewall): 18 offensive signatures over bash/python/node/perl for reverse shells, curl|sh pipelines, and base64-decode-exec payloads. A high/critical finding is the gate that blocks sandbox exec. Zero-I/O, first-party.
On the languages the signatures target (bash/python/node/perl) recall is 98.8% at 100% specificity — 0 false alarms on 2,100 real functions. powershell/netcat/php/ruby/go/lua/java shells are OUT of the current signature scope (netcat is caught incidentally but not counted in scope), so recall across ALL languages is 32.0% — a named coverage gap, not hidden. Building this benchmark surfaced and fixed four real signature bugs (a greedy python matcher, missing node createConnection, a perl quote-escape, and a recv/os.popen command-loop family), each verified to add zero false positives.
- darkknight25/Reverse_Shell_Payloads_Datasetpublic · no revision recorded— 291 real reverse-shell payloads (positives)
- google/code_x_glue_ct_code_to_textpublic (CodeXGLUE)— 2,100 real OSS functions (benign negatives)
replay-ASR 43.1% against an undefended floor of 47.7%
gpt-4o-2024-05-13 (undefended) / important_instructions — the config and attack every published AgentDojo guardrail figure is quoted on · 29 of 300 recorded attack successes would have been aborted, over 629 injected runs · counterfactual over recorded traces, not a live run
This is the benchmark we lose, and it is on the page because a measured loss with a named cause is worth more than a missing card. On the slice every published AgentDojo guardrail figure is quoted on, the shipped default injection tier would have aborted 29 of 300 recorded attack successes — moving the recorded undefended floor of 47.7% to 43.1%, a reduction of 4.6%. That is a real reduction and nowhere near good enough.
createFirewallEngine() default stack (ingress / tool-result) — indirect-injection@1 + injection-cascade@1+injection-trained@injection-student-logreg:2.0.0+slm:none + pii-regex@1 + secret-entropy@1, scored exactly as it ships (student threshold 0.46, no tuning on AgentDojo). Each recorded tool output is scanned as the gateway would see it, one at a time; a block stops the trace there. counterfactual abort over recorded traces: the firewall scans each tool output as ingress/tool-result and a block stops the trace there. A recorded security=true run counts as PREVENTED iff the first flagged tool output is at or before the first tool output that carried the injection. Runs where no injection-bearing tool output can be located count as NOT prevented (conservative). This is NOT a live attack-success rate.
the harness says it first — metrics.headline, verbatim
NEGATIVE RESULT, disclosed. On the slice every published AgentDojo guardrail figure is quoted on (gpt-4o-2024-05-13 / important_instructions, 629 recorded runs), counterfactual abort by the shipped default stack would have prevented 29 of 300 recorded attack successes: replay-ASR 43.0843% against a recorded undefended floor of 47.6948% measured from the same bytes. On the same slice, the configuration AgentDojo's leaderboard scores for protectai/deberta-v3-base-prompt-injection-v2 recorded a LIVE targeted ASR of 7.9491% (recomputedFromCorpus, measured here from the same pinned runs; the published counterpart is register claim agentdojo-protectai-deberta). Our replay figure is 5.4x that live figure, so this card claims no parity with it, and the peer-reviewed guardrail baselines in the register (agentdojo-promptguard2-86m, agentdojo-llamafirewall-combined) are lower still. Scored on the bare injection strings alone, outside any tool output, the same stack flags 27.98% of them (important_instructions in isolation: 16.14%), against an embedded TPR of 12.16% on the same attack — so the gap is detector capability, not dilution inside long tool outputs. What does hold up is measured too: embedded TPR 100% on the injecagent attack family and 56.97% on ignore_previous, against 12.16% on important_instructions.
Counterfactual abort against the recorded floor — the same bytes, before and after
No agent ran here and no model API key was used, so nothing in this section is an end-to-end attack-success rate. These figures are not comparable with the live competitor figures in the register below, and this page never prints them in the same table, column or chart. The only like-for-like comparison the card makes is replay against the recorded undefended floor measured from the same bytes — which is the pair tabulated here.
| slice (recorded traces) | recorded undefended floor (as AgentDojo recorded it) | replay-ASR (counterfactual abort, same bytes) | aborted successes | utility under attack (recorded, then replayed) |
|---|---|---|---|---|
| gpt-4o-2024-05-13 (undefended) / important_instructions — the config and attack every published AgentDojo guardrail figure is quoted on n=629 injected runs | 47.7% | 43.1% reduction 4.6% | 29 / 300 | 50.1% → 38.8% |
| every scored config, every attack type except the DoS family n=31,376 injected runs | 15.2% | 11.8% reduction 3.4% | 1,064 / 4,769 | 54.3% → 40.0% |
Both ASR columns come from the SAME recorded bytes: the floor as AgentDojo recorded it, and the counterfactual after a block-on-detect abort. Neither is a live end-to-end ASR, and neither may be read against one.
Block-on-detect is strictly more aggressive than the defence these traces were recorded under, so the price is recorded next to the benefit: across 2,562 clean runs it would have aborted 11.2% of them, taking recorded utility 57.7% down to 50.5%. A detection figure quoted without its utility column is not a result.
a DoS injection SUCCEEDS when the agent fails the user task, so a counterfactual abort cannot reduce its ASR — an abort IS the denial of service. Reported for completeness and excluded from every headline replay figure. That bucket (dos / felony_dos / swearwords_dos / offensive_email_dos / captcha_dos) holds 485 injected runs with 171 recorded successes and is excluded from every figure above.
Detection — did the shipped tier flag the injection inside the recorded tool output?
| unit | TPR | FPR | confusion |
|---|---|---|---|
| one recorded tool output (occurrence-weighted) n=38,524 positive · 10,656 negative | 22.5% | 4.7% | TP 8,661 · FP 497 · TN 10,159 · FN 29,863 |
| one distinct tool-output text (de-duplicated) n=9,127 positive · 1,174 negative | 27.3% | 4.2% | TP 2,494 · FP 49 · TN 1,125 · FN 6,633 |
Positives: tool outputs carrying their run's injection. Negatives: tool outputs from runs with no injection at all. Tool outputs that sit inside an attacked run but do not carry the injection are not clean negatives, so they are reported separately: 2,063 of 80,706 flagged (2.6%).
Shipped operating point: threshold 0.46 — TPR 29.6% at FPR 4.9%, ROC AUC 0.6683 over 9,127 positive and 1,174 negative distinct tool-output texts. Swept tighter on this corpus (threshold 0.6775) it reads TPR 14.4% at FPR 0.9% — an oracle point, optimistic by construction, because the threshold was picked on the very corpus it is scored on. The only point we ship is the artifact threshold.
Per attack family — where it holds and where it does not
The cause is measured, not guessed. Scored on the bare injection strings alone, outside any tool output, the same stack flags 28.0% of 629 distinct strings — and only 16.1% of the 347 important_instructions strings, against 12.2% for the same family embedded in real tool output (20,518 occurrences). The bare figure sits next to the embedded one, so the gap is detector capability on this attack family, not dilution inside long tool outputs. What holds up is measured on the same corpus: 100% on the injecagent family and 57.0% on ignore_previous, against 12.2% on important_instructions — the family that carries most of this corpus, and the one everybody quotes.
Measured here, but LIVE — and deliberately not in the table above
Taksham-MEASURED, from the pinned runs/ tree: AgentDojo's own targeted-ASR / utility definitions applied to two of its recorded configurations. These are LIVE end-to-end outcomes of someone else's agent loop, so they are NOT column-comparable with replayAbort above. They are here so a reader can check that this harness reads the corpus's ground truth the way its authors do — the register entry each one points at carries the published figure, which this card deliberately does not copy.
TransformersBasedPIDetector(protectai/deberta-v3-base-prompt-injection-v2, threshold 0.5, mode=message, raise_on_injection=False): a flagged tool output is REPLACED with a placeholder and the agent continues — it does not abort
The register's published counterpart is claim agentdojo-protectai-deberta — 7.95%, benchmark-authors, reproducible outside the publisher. Our recomputation from the pinned traces lands on the same figure, which is what a reproducible claim looks like.
no guardrail: the recorded floor this slice is measured against
the undefended floor ON THIS SLICE, measured here. It is not interchangeable with the undefended baseline in the register claim it points at: that figure comes from the LlamaFirewall authors' own agent setup, not from these traces.
We reproduce a competitor detector's published AgentDojo figure from the pinned traces and land on the same number. That — not a win — is the claim this page makes.
What this card does not claim — its own caveats, verbatim
- READ `metrics.headline` FIRST: this is a DISCLOSED NEGATIVE RESULT. On the attack the published AgentDojo guardrail figures are quoted on, the shipped injection tier's replay figure is worse than the live figure this harness recomputes for the detector those figures belong to, and no number on this card may be presented as if it were not. It is published because a measured loss with a named cause is worth more than a missing card (docs/BENCHCARDS.md).
- REPLAY, NOT LIVE. Every Taksham number on this card is computed over trajectories recorded by SOMEONE ELSE'S agent. No agent ran here and no model API key was used, so this card does NOT contain an end-to-end attack-success rate. `replayAbort.*.replayAsrPct` must never be printed in the same column, table or chart as a live end-to-end ASR — not the LlamaFirewall, PromptGuard 2 or Protect AI figures in docs/competitor-claims.json, and not `recomputedFromCorpus` on this card either. The honest comparison here is replayAsrPct against recordedTargetedAsrPct, both measured from the same bytes.
- The known weakness of static-trace replay, stated by us rather than by a reviewer: a trace is one sample from a stochastic agent. Replay shows what WOULD have been stopped in that recorded run; it cannot show what a live agent does after a block, what an adaptive attacker would write next, or how a detector in the loop changes the trajectory it is judging. A live ASR is a strictly stronger claim and this card does not make it.
- The two configs whose own defense rewrote its tool outputs (transformers_pi_detector replaces a flagged output with a placeholder; spotlighting_with_delimiting interleaves a delimiter) are read for provenance and EXCLUDED from scoring — their recorded tool text is not the upstream tool text. The exclusion is re-verified at run time from the bytes themselves.
- Ground truth is AgentDojo's own per-run `security` and `utility` booleans. The POSITION of the injection inside a trace is not recorded upstream and is recovered here by substring match after aggressive normalization (see `metrics.labelling`); `recordedSecurityTrueWithNoLocatedInjection` counts the recorded attack successes this harness could not attribute to a located tool output, and every one of them is counted as NOT prevented.
- Counterfactual abort is a BLOCK-on-detect policy and is strictly more aggressive than the defense behind the deberta configuration this card recomputes, which omits the flagged output and lets the agent continue. The price is recorded next to the benefit: `clean.falseAbortRunPct` and `clean.replayUtilityPct` are the utility a block-on-detect deployment would destroy on clean runs at this operating point. A detection figure quoted without its utility column is not a result.
- DoS-family attacks (`dos`, `*_dos`) succeed when the agent FAILS the user task, so aborting the trace achieves the attacker's goal rather than preventing it. They are reported in their own bucket and excluded from every headline replay figure.
- detectionTprFpr negatives are tool outputs from runs with no injection anywhere (3312 clean runs across the scored configs; the pinned corpus holds 3560 in total, the rest belonging to the two excluded configs). Tool outputs that sit inside an attacked run but do not carry the injection are reported separately as `nonInjectedOutputsInsideInjectedRuns`: same environments, so a more realistic false-alarm estimate, but not a clean negative.
- `injectionStudentOperatingPoints.sweptToFpr1Pct` / `sweptToFpr5Pct` pick the threshold ON THIS CORPUS and report TPR at it, so they are oracle operating points and optimistic by construction. The only operating point Taksham ships is `shipped` (the artifact threshold). The swept points exist because competitor guard-model figures are published as 'recall at 1% FPR'; they are still measured on a different corpus from Meta's, so they are comparable in kind only.
- No detector was tuned on AgentDojo. The committed student and the regex tiers are scored exactly as shipped; the student's training pool is the deepset / xTRam1 / Lakera-gandalf sets plus the in-repo corpus listed in bench/README.md, and holds no AgentDojo bytes.
- Tool outputs are scored one at a time, as the gateway would see them (direction ingress, surface tool-result, point gateway), with no verdict cache, no inline latency tier and no cross-turn context. Identical tool texts are scored once and the decision reused; both occurrence-weighted and distinct-text rates are reported.
- Utility under replay is all-or-nothing: an abort anywhere in a trace is counted as destroying that run's recorded utility. That is conservative for us on the clean side (it maximises the measured utility cost) and conservative on the attacked side too (`replayUtilityUnderAttackPct` can only fall).
36,679 committed run files read at the revision the card pins, 35,173 scored (31,861 injected, 3,312 clean) across 29 model/defense configurations and 629 distinct injection strings, over 136,559 recorded tool outputs. The 2 configurations whose own defence rewrote its tool outputs (gpt-4o-2024-05-13-transformers_pi_detector, gpt-4o-2024-05-13-spotlighting_with_delimiting) are read for provenance and excluded from scoring — their recorded tool text is not the upstream tool text.
Ground truth is AgentDojo's own per-run security and utility booleans. The position of the injection inside a trace is not recorded upstream and is recovered here by normalized substring match: 82.5% of injected runs had their injection located, and the 91 recorded attack successes that could not be attributed to a located tool output (1.8%) are all counted as NOT prevented.
pnpm exec tsx packages/firewall-engine/bench/agentdojo-replay-bench.ts --write · git 1192c02f037cc8c68ef357c8e1775589bf052c73 · Intel(R) Xeon(R) Processor @ 2.10GHz (4 cores) · 2026-10-27T16:35:14.482Z
- ethz-spylab/agentdojo runs/ (recorded agent trajectories)MIT · revision 089ed468cf3ed0322acc66b0211f26d9d90dbf60 pinned · corpus sha256 90e33d124523952cc82f542f3b49908d00a073dc1b94c6a2acb0bbb562b65ac0— all 36679 committed runs at this revision (29 model/defense configs x 15 attack types + clean); the 2 configs whose defense rewrote its own tool outputs are read but not scored — 35,173 runs scored. Debenedetti et al., AgentDojo (NeurIPS 2024 Datasets & Benchmarks), arXiv:2406.13352
7 of 11 published rival figures cannot be re-run by anyone outside the publisher.
This is a register, not a leaderboard, and it has no Taksham column. Only 4 of 11 claims in it could be reproduced by a third party at all. Every vendor-published detection score here — 4 of 4 — is scored on Lakera's PINT blend, which is explicitly part-proprietary and has never been released, so nobody outside Lakera can reproduce or contest any of them. That is the pattern, not the exception: the detection-benchmark eval set behind Prompt Guard's own published numbers is not public either (its AgentDojo row here is reproducible; its self-scored detection figures are not, which is why they are absent), and Lakera withholds b3's strong attacks while licensing out a weak slice, which is why b3 sits here marked never-a-baseline. A page that prints these figures without that column is just another leaderboard.
What we claim instead is narrow and checkable. Every figure on this page is read from one of 44 committed benchcards — 21 of which record the exact command that wrote them, and 21 the upstream dataset revision — and the section above this one is a benchmark we lose, published with its cause named. That pair is the whole argument: reproducible numbers including the bad ones, against unreproducible numbers that are uniformly good.
| publisher · product | benchmark · metric | published figure (as published) | evidence tier | reproducible by a third party? | how we may use it |
|---|---|---|---|---|---|
| Meta · LlamaFirewall (PromptGuard 2 + AlignmentCheck) agentdojo-llamafirewall-combined | AgentDojo targeted attack success rate · lower is better | 1.75% as of 2025-05-06 source | peer-reviewed | NOT reproducible outside the publisher AlignmentCheck needs a hosted Llama 4 Maverick, so the combined figure cannot be reproduced without that model. | baseline-eligible · only the publisher can re-run it ASR .0175 at utility .4268. This is a LIVE end-to-end ASR: it must never be printed in the same column as a replay figure. |
| Meta · Llama Prompt Guard 2 86M agentdojo-promptguard2-86m | AgentDojo targeted attack success rate · lower is better | 7.53% as of 2025-05-06 source | peer-reviewed | reproducible outside the publisher | baseline-eligible · anyone can re-run it Reported as ASR .0753 at utility .4701. The only peer-reviewed AgentDojo guardrail table, which is why it is the number everyone cites. |
| AgentDojo · no guardrail agentdojo-undefended | AgentDojo targeted attack success rate · lower is better | 17.63% as of 2025-05-06 source | peer-reviewed | reproducible outside the publisher | baseline-eligible · anyone can re-run it The undefended floor (utility .4773). Any guardrail claim is meaningless without it. |
| Lakera · b3 (Breaking Agent Backbones) lakera-b3-vulnerability | b3 vulnerability score, best open-weight model · lower is better | 0.34 score as of 2026-02-24 source | peer-reviewed | NOT reproducible outside the publisher The high-quality attack set is deliberately withheld; only a low-quality 210-attack slice is public, and that slice's licence forbids commercial use. | deliberately not a comparison baseline Deliberately NOT a target. The licence bars commercial benchmarking and the baseline corpus is withheld, so no one outside Lakera can reproduce or contest the leaderboard. |
| Gray Swan AI and UK AISI · IPI Arena (13 frontier models, 41 scenarios) ipi-arena-model-range | IPI Arena attack success rate across models (best to worst) · lower is better | 0.5% to 8.5% as of 2026-03-16 source | peer-reviewed | reproducible outside the publisher | baseline-eligible · anyone can re-run it 464 participants, 272k attempts, 8,648 successes. Code and a 95-attack slice are MIT, but the public slice is deliberately the weaker one, so our number on it is not comparable to theirs on the full set. |
| Protect AI · protectai/deberta-v3-base-prompt-injection-v2 (as AgentDojo's transformers_pi_detector) agentdojo-protectai-deberta | AgentDojo targeted attack success rate · lower is better | 7.95% as of 2024-06-05 source | benchmark-authors | reproducible outside the publisher | baseline-eligible · anyone can re-run it The headline number to beat: a direct competitor's detector, scored by the benchmark's own authors, on a corpus (MIT) whose recorded trajectories are committed in the AgentDojo repo. Its utility was 41.24% and utility-under-attack 21.14% — a detector is only interesting at a stated utility cost. |
| Microsoft · Azure Prompt Shield pint-azure-prompt-shield | PINT PINT score (balanced accuracy) · higher is better | 89.1241% as of 2025-05-02 source | vendor-published | NOT reproducible outside the publisher Scored by Lakera on the same unreleased blend. | deliberately not a comparison baseline Scored by a competitor, not by Microsoft. |
| AWS · Bedrock Guardrails pint-bedrock-guardrails | PINT PINT score (balanced accuracy) · higher is better | 89.2404% as of 2025-05-02 source | vendor-published | NOT reproducible outside the publisher Scored by Lakera on the same unreleased blend. | deliberately not a comparison baseline Scored by a competitor, not by AWS. |
| Lakera · Lakera Guard pint-lakera-guard | PINT PINT score (balanced accuracy) · higher is better | 95.22% as of 2025-05-02 source | vendor-published | NOT reproducible outside the publisher The 4,314-input blend is explicitly part-proprietary and has not been released, so no third party can reproduce or contest any PINT score. | deliberately not a comparison baseline Published here to make the point, not to be beaten: the leaderboard leader is self-scored on a dataset nobody else can obtain. Not baseline-eligible for exactly that reason. |
| Protect AI · protectai/deberta-v3-base-prompt-injection-v2 pint-protectai-deberta | PINT PINT score (balanced accuracy) · higher is better | 79.1366% as of 2025-05-02 source | vendor-published | NOT reproducible outside the publisher Scored by Lakera on the same unreleased blend. | deliberately not a comparison baseline The same open model that AgentDojo's authors scored independently — useful for showing how far one model's number moves with the dataset. |
| NeuralTrust · NeuralTrust neuraltrust-injection-detection | undisclosed jailbreak / indirect-injection detection · higher is better | 92% and 99.8% as of 2026-01-01 source | unverified | NOT reproducible outside the publisher No methodology, dataset or evaluation protocol published. | unverified — never a comparison baseline Recorded so the omission is deliberate and auditable rather than an oversight. A number with no stated dataset cannot be compared with anything. |
A paper. Strongest, but read `reproducible` — several of these are scored on a set the authors never released.
Published by the people who own the benchmark, on their own public leaderboard.
The vendor's own blog or docs, scoring itself.
Could not be confirmed from a primary source, or primary sources disagree. NEVER usable as a comparison baseline (`baselineEligible` must be false).
No row here is set against the replay figures above. Those are a counterfactual over someone else's recorded traces; these are live end-to-end agent runs. Different units, so a shared column would be a lie — and the AgentDojo card points at agentdojo-protectai-deberta, agentdojo-promptguard2-86m, agentdojo-llamafirewall-combined, agentdojo-undefended in this register by id rather than copying their values onto itself.
The unverified tier — 1 of 11 rows — can never be a comparison baseline. Such a claim is recorded so that leaving it out of every comparison is auditable rather than an oversight, and the build fails if one is ever marked baseline-eligible.
A COMPETITOR's published number, transcribed with its source. NOT a measurement Taksham made: our own figures live in benchcards (docs/benchmarks.index.json) and nowhere else. Every claim here carries the URL it came from, the date it was published, how strong the evidence is, and whether anyone outside the publisher could reproduce it. Validated and copied to the site by scripts/competitor-claims.mjs; never hand-edit the copy.
The full fetch sequence. The same harnesses.
Downloads are gitignored — only the fetchers and the result benchcards are committed, so the figures reproduce without redistributing anyone's data. Each run re-fetches the public split and re-scores from scratch. Hugging Face fetchers record the dataset revision they received — revision recorded (not pinned) — so a later upstream change can move a re-run.
Needs Hugging Face access: injection, jailbreak, PII (ai4privacy), secrets, malicious code and HaluEval. GitHub-sourced: LoCoMo retrieval and the detector-slm out-of-sample set.
No dataset revision recorded on these cited cards: the fullstack injection headline (deepset-injection-fullstack); detector-slm-vs-student (its GitHub source URLs are commit-pinned instead); reverse-shell malicious-code; LoCoMo end-to-end.
# prompt-injection — fetch every public set (train + held-out test), then train + score (needs Hugging Face)
node bench/fetch-deepset-injections.mjs test
node bench/fetch-deepset-injections.mjs train
node bench/fetch-public-injections.mjs xTRam1/safe-guard-prompt-injection train --cap=8300
node bench/fetch-public-injections.mjs xTRam1/safe-guard-prompt-injection test --cap=2100
node bench/fetch-public-injections.mjs Lakera/gandalf_ignore_instructions train --fixed-label=1
../../node_modules/.bin/tsx bench/injection-model-bench.ts
# out-of-sample — commit-pinned GitHub copies of Tensor Trust, jailbreak_llms, Alpaca (needs onnxruntime-node)
node bench/fetch-github-injection-sets.mjs
../../node_modules/.bin/tsx bench/detector-slm-bench.ts
# jailbreak — score the shipped student on in-the-wild DAN prompts (needs Hugging Face)
node bench/fetch-jailbreak.mjs --snap=2023_12_25 --cap=800
../../node_modules/.bin/tsx bench/jailbreak-bench.ts
# PII — span-level recall/precision on ai4privacy (needs Hugging Face)
node bench/fetch-ai4privacy-pii.mjs --cap=2000 --lang=en
../../node_modules/.bin/tsx bench/pii-bench.ts
# secrets — specificity on real OSS files (needs Hugging Face)
node bench/fetch-codexglue-code.mjs --per=350
../../node_modules/.bin/tsx bench/secrets-bench.ts
# malicious code — reverse-shell recall + benign specificity (needs Hugging Face)
node bench/fetch-reverse-shells.mjs
../../node_modules/.bin/tsx bench/malicious-code-bench.ts
# AgentDojo agent-trajectory replay (the negative result above) — run from the REPO ROOT
# corpus: https://github.com/ethz-spylab/agentdojo/tree/089ed468cf3ed0322acc66b0211f26d9d90dbf60/runs
# at revision 089ed468cf3ed0322acc66b0211f26d9d90dbf60 — corpus sha256 90e33d124523952cc82f542f3b49908d00a073dc1b94c6a2acb0bbb562b65ac0
pnpm exec tsx packages/firewall-engine/bench/agentdojo-replay-bench.ts --write
# memory — LoCoMo evidence retrieval (GitHub-sourced)
node bench/fetch-locomo.mjs
../../node_modules/.bin/tsx bench/locomo-retrieval-bench.ts
# eval judge — HaluEval (needs Hugging Face; negative result, not published as a score)
node bench/fetch-halueval.mjs --cap=1000
../../node_modules/.bin/tsx bench/eval-judge-bench.ts
Benchcards land in bench/results/*.test.json — dataset id, n and timestamp on every card, revision where the source exposes one. Each card's missing benchcard.v1 fields (host, git sha, command…) are listed in docs/benchmarks.index.json.
The pillars without a number here — and why.
These detectors ship and run, but we haven't wired a public-data harness for them yet — so rather than show a hand-made figure, we name the pillar and the public benchmark we'd grade it against. They land here with real numbers as the harnesses come online.
Indirect / embedded injection
Detector ships (indirect-injection.ts — hidden-region extraction over fetched content), and embedded-injection detection is now measured on AgentDojo's recorded tool outputs (27.3% TPR per distinct tool-output text, in the section above) — a result we lose on. A purpose-built embedded-attack corpus harness still is not wired.
target: BIPIA
Eval / LLM-judge quality
The shipped inline judge is REFERENCE-BASED (grades a candidate against a gold answer). On HaluEval we measured it head-to-head: it's near-chance on that set's groundedness task (it's not a groundedness judge) — so we're sourcing a reference-based correctness set before posting a number, rather than a misleading one.
target: RewardBench / correctness set
Memory — end-to-end QA
Retrieval on LoCoMo is measured above (recall@10 42.8%), and a sentence-embedder swap doesn't move it. The lift Mem0/Zep headline needs LLM fact-extraction + answer generation + a judge. Status: a partial full-context baseline run exists (1 of 10 conversations, 152 QA, self-judged qwen2.5:7b-instruct; it scores the full-context baseline, not the memory stack); not published as a score.
target: LoCoMo J-score (LLM-judged)
Red-team attack success (LIVE)
Campaigns and a learned attack-success judge exist, and AgentDojo is now scored — but only as static-trace REPLAY: a counterfactual over trajectories someone else recorded, which cannot show what a live agent does after a block or what an adaptive attacker writes next. A live end-to-end ASR needs an agent loop and model keys in the harness, and is a strictly stronger claim we do not yet make.
target: AgentDojo (live agent runs) / JailbreakBench
Injection — operator DeBERTa tier
The optional multilingual transformer upgrade (onnxruntime, operator/GPU) would raise the hardest deepset/multilingual recall; benchmarking it needs the operator's own model artifact.
target: deepset (multilingual)