Guides · 50 benchmarks

Benchmark guides

Deep dives on the AI evaluations that actually move procurement decisions. Each guide covers methodology, scoring, common gaming patterns, and how to ship a 50-page-attested score nobody can challenge.

164 Python programming problems, pass@1 via deterministic test execution.

EvalPlus hardening — ~80× more test cases than HumanEval.

974 entry-level Python problems with auto-graded tests.

EvalPlus hardening over the sanitized MBPP subset.

1140 realistic Python tasks invoking complex libraries (pandas, matplotlib, sklearn).

Contest-style coding problems with monthly time windows — contamination-resistant.

SWE-bench Verified

500 real GitHub issues, human-vetted, Docker test harness.

300-issue lightweight subset of SWE-bench.

57 subjects, 4-choice MCQ. The classic knowledge benchmark.

MMLU's hardened successor — 10 options, harder reasoning.

PhD-level physics, chemistry, biology. Google-proof.

23 challenging reasoning tasks pulled from BIG-Bench.

Grade-school science MCQ, harder subset.

Commonsense sentence-completion MCQ.

Commonsense coreference, binary choice (Winograd-style).

Physical commonsense reasoning, binary choice.

Multistep soft reasoning: murder mysteries, team allocation.

Human-exam questions: SAT, GRE, Chinese civil service.

8.5k grade-school arithmetic word problems.

Competition math problems, LaTeX boxed answers.

American Invitational Math Exam. Integer answers 0-999.

Multilingual GSM8K across 11 languages.

Theorem-grounded STEM problems requiring numeric/expression answers.

agent framework

Multi-turn tool-use trajectories in retail and airline environments.

agent framework

General AI assistants — 466 real-world tasks, exact-match scoring.

agent framework

Realistic web navigation tasks in hosted shopping / CMS / GitLab environments.

Monthly-refreshed SWE-bench variant, contamination-resistant.

817 tricky questions, single-correct multiple choice.

OpenAI's factuality benchmark. 4,326 short-answer questions, GPT-4o judged.

5-way MCQ over ConceptNet-derived commonsense.

NVIDIA's 13-task long-context suite at configurable window sizes.

NIAH · Needle in a Haystack

Retrieve a seeded fact from 4k → 128k token contexts. Deterministic generator.

Long-conversation memory Q&A. GPT-4o canonical judge.

11.5k multimodal questions across 30 disciplines.

Math reasoning with visual context (1k testmini).

SWE-bench Multimodal

JavaScript issues with visual context. Docker harness required.

Creative Writing

Monthly-refreshed benchmark across math, coding, reasoning, data analysis, language.

Scalable Agentic Bench

USMLE-style medical licensing exam questions.

Humanity Last Exam

Chollet's abstraction and reasoning corpus. Grid-puzzle transformations.