How good is the detector, really?
We publish our numbers instead of asking you to trust a marketing claim. Below is the deterministic detector scored against a labeled dataset of 96 tool definitions (63 tool-poisoning attacks, 33 legitimate tools). The dataset lives in the repo, so anyone can rerun it and get these exact numbers.
100%
Overt attacks caught
0%
False positives
100%
Precision
90.5%
All attacks caught (deterministic)
63
Attack samples
33
Benign samples
The honest part: the deterministic layer catches every overt attack with zero false positives, but only 90.5% of attacks overall, because 6 attacks in the set are subtle(no keyword tells). Those are what the live product's AI-judge layer is for: it catches or escalates most of them for a human, though no detector catches everything and we don't claim otherwise. We show the deterministic number because it is the one you can reproduce without an API key.
Detection by attack type
admin0/1 flagged
analytics0/1 flagged
annotation_spoof3/3 flagged
auth0/1 flagged
calendar0/2 flagged
canary3/3 flagged
chat_template2/2 flagged
covert_channel2/2 flagged
crm0/2 flagged
cross_tool1/1 flagged
db0/2 flagged
delegation_abuse2/2 flagged
devops0/1 flagged
docs0/1 flagged
email0/3 flagged
encoded_payload3/3 flagged
exfiltration6/6 flagged
files0/5 flagged
full_schema_injection2/2 flagged
hidden_instruction7/7 flagged
impersonation2/2 flagged
memory_persistence2/2 flagged
network0/3 flagged
notes0/1 flagged
oauth_confused_deputy2/2 flagged
over_broad_scope4/4 flagged
payments0/2 flagged
prompt_extraction2/2 flagged
readonly0/1 flagged
scope_mismatch2/2 flagged
search0/2 flagged
secret_harvest4/4 flagged
sensitive_file2/2 flagged
settings0/1 flagged
ssrf_metadata2/2 flagged
storage0/1 flagged
subtle0/6 flagged
terminal_escape2/2 flagged
translate0/1 flagged
unicode_smuggling2/2 flagged
utility0/2 flagged
Methodology
- Dataset: 63 attacks modeled on documented incidents (Postmark MCP exfiltration, GitHub MCP cross-repo hijack) and the OWASP Agentic Top 10, plus 33 realistic legitimate tools including edge cases that mention files, email, and data.
- Flagged = risk score ≥ 30 (not cleared as safe). Precision = attacks correctly flagged / all tools flagged.
- Scored with the deterministic detector only (no model, no network), so results are exactly reproducible. The live API additionally runs the AI-judge and provenance layers.
- Reproduce the scores: GET /api/trust/benchmark.
- Get the full labeled dataset and run it against your own detector: GET /api/trust/benchmark/dataset. Every case, its label, and the scoring rule are published, so our accuracy claim is yours to check.
Generated 7/12/2026, 7:38:18 AM · deterministic detector mcp-tool@1