SECURITY_REVIEW intent
Two security reviews. One is quietly wrong.
Each case has a sound review and one that is plausible but wrong in a single decisive fact: a fake advisory id, a wrong version range, a false alarm. Pick the review you would trust, then see the flaw, how shortcut judges did, and what our evaluator chose.
real cases from the public dataset. Every advisory fact is checked against OSV.
Case 1 of 1
Tap the review you trust, or press A or B.
- Flaw type
- The flaw
- Longer-answer judge
- Keyword judge
- Our evaluator
- Never trained on this case.
- Source
- Source advisory
Why judges get fooled
accuracy on held-out pairsThe orange tick marks a coin flip. Ranges cover two training runs. These validation pairs come from the same author as the training data, so they overstate real accuracy; an independent test set decides registration.
What a paid answer is worth
3 rounds, 3,680 paid calls, testnetWe bought the same 46 security questions from every reachable Telegraph miner on three days, under a rule fixed before the first call, and reconciled the payments with the chain or the node's ledger (one $0.01 charge unmatched).
95% CI 75 to 89%83%
95% CI 57 to 75%67%
95% CI 15 to 30%21%
- Rank vs accuracy
- Kendall's τ +0.02, −0.18, then +0.29 in round 4, when a rebuilt miner reached rank 1 (17 of 36 right; it gives no verdict on clean packages).
- Verified Ask
- Our free check of each paid answer against OSV labelled none of 142 wrong answers as verified, and all 220 verified answers were correct.
- Cost per correct
- $0.014 to $0.017 from the best miner; $0.06 to $0.10 across all miners.
Accuracy is pooled over 108 answerable cases (36 per round). Testnet miners, one payer wallet, 46 cases per round, so read overlapping ranges as ties. Full method and data: the pooled report.
How it works
- [1] Sourcereal advisories from OSV and GitHub
- [2] Pairidentical answers, one decisive fact changed
- [3] Verifyevery fact and label checked; shortcut judges near 50%
- [4] Reviewa person approves every pair
- [5] Poollive in the Telegraph evaluation pool