AdversarialEval ↗
Evaluation data live on Telegraph testnet /
SECURITY_REVIEW intent

Two security reviews. One is quietly wrong.

Each case has a sound review and one that is plausible but wrong in a single decisive fact: a fake advisory id, a wrong version range, a false alarm. Pick the review you would trust, then see the flaw, how shortcut judges did, and what our evaluator chose.

real cases from the public dataset. Every advisory fact is checked against OSV.

Case 1 of 1

You 0/0 Longer-answer judge 0 Keyword judge 0
[ Request ]

    

Tap the review you trust, or press A or B.

Why judges get fooled

accuracy on held-out pairs

The orange tick marks a coin flip. Ranges cover two training runs. These validation pairs come from the same author as the training data, so they overstate real accuracy; an independent test set decides registration.

What a paid answer is worth

3 rounds, 3,680 paid calls, testnet

We bought the same 46 security questions from every reachable Telegraph miner on three days, under a rule fixed before the first call, and reconciled the payments with the chain or the node's ledger (one $0.01 charge unmatched).

Rank vs accuracy
Kendall's τ +0.02, −0.18, then +0.29 in round 4, when a rebuilt miner reached rank 1 (17 of 36 right; it gives no verdict on clean packages).
Verified Ask
Our free check of each paid answer against OSV labelled none of 142 wrong answers as verified, and all 220 verified answers were correct.
Cost per correct
$0.014 to $0.017 from the best miner; $0.06 to $0.10 across all miners.

Accuracy is pooled over 108 answerable cases (36 per round). Testnet miners, one payer wallet, 46 cases per round, so read overlapping ranges as ties. Full method and data: the pooled report.

How it works

  1. [1] Sourcereal advisories from OSV and GitHub
  2. [2] Pairidentical answers, one decisive fact changed
  3. [3] Verifyevery fact and label checked; shortcut judges near 50%
  4. [4] Reviewa person approves every pair
  5. [5] Poollive in the Telegraph evaluation pool

Ten ways a review can be wrong