How we grade reasoning
Final answers are cheap. ReasonEval scores the whole trace — the plan, the steps, the self-corrections, and the confidence behind every claim.
The evaluation pipeline
Curate & decontaminate
Problems are sourced from competitions, formal corpora and novel authoring, then n-gram and embedding-scanned against public training data. Anything contaminated is dropped or rewritten.
Elicit traces
Models solve each problem with visible reasoning at a fixed budget. Each problem ships with adversarial variants — reordered premises, changed constants, added distractors.
Grade every step
A dual-judge ensemble plus formal verifiers (CAS, test runners, proof checkers) scores each step on the rubric. Judge disagreements go to arbitration; agreement is 93.4%.
Aggregate & calibrate
Suite scores combine with rubric weights, bootstrap confidence intervals, and an ELO system for head-to-head trace comparisons. Calibration errors feed the trust penalty.
What goes into the R-score
A single composite per suite, then across suites — always with confidence intervals, never without.
| Dimension | Weight | What it measures |
|---|---|---|
| Correctness | 40% | Final answers validated by verifiable ground truth (CAS, tests, proof checkers) |
| Process quality | 25% | Per-step validity, plan coherence, efficient use of intermediate results, recovery from errors |
| Calibration | 20% | Stated confidence vs. empirical accuracy (ECE), honest "I don't know" behavior on unknowable items |
| Robustness | 15% | Score stability across adversarial perturbations of the same problem; memorization detectors |
Judges, verifiers, arbitration
Dual-judge ensemble
Two independent judge models grade each trace against the rubric. When they disagree beyond tolerance, a third arbiter judge resolves; residual disagreements are flagged in the public data.
Formal verifiers
Wherever ground truth is machine-checkable, we don't ask judges at all: computer-algebra systems, sandboxed test runners and proof checkers decide. Judges grade only the parts that cannot be formalized — the process.
Publication checks
Every leaderboard entry ships with: per-problem results, judge transcripts, variant deltas and the exact prompt templates. If we can't reproduce it, it doesn't ship.
Run it on your model
The harness is open source and works with any OpenAI-compatible endpoint. Full traces, per-step grades and calibration curves land in a single HTML report.
View on GitHub$ export OPENAI_BASE_URL="https://api.example.com/v1" $ reasoneval run --model "my-model" \ --suite math,logic --variants "adversarial" --out report/ ▸ 3,050 problems · 4 variants · 12,200 traces ▸ R-score 81.3 (±1.1) · calibration ECE 0.071 ✓ report/index.html · report/traces.jsonl