How do you test whether an AI agent can really do a job?

You give it real jobs that people have already done, hide the answer key, and check the result with tests. One example is SWE-bench, which has 2,294 tasks taken from software issues that were actually filed, and actually fixed, in 12 Python repositories. The agent is shown the code as it stood before the fix, plus the text of the issue, and has to write a patch. It never sees the tests that will judge the result.

When it was introduced in October 2023, researchers at Princeton University and the University of Chicago found that the strongest model they tried, Claude 2, fixed only 1.96% of the tasks. By February 2026 the leading published result on a human-checked subset called SWE-bench Verified was 80.9%.

Then the benchmark itself was questioned. OpenAI said it would stop quoting the Verified figure: the models being scored had read the repositories the tasks come from, and every frontier model OpenAI tried in a sample could reproduce wording from the problem statements or from the developers' own fix. A separate study, by Reem Aleithan and colleagues, of the system then top of the leaderboard in 2024 found that in 32.67% of its supposedly solved tasks the fix was written out in the issue or its comments, and another 31.08% passed only because the tests were too weak. Take both out and its score dropped from 12.47% to 3.97%.

Wikipedia's article on language model benchmarks calls this Goodhart's law: if models are designed or selected to score highly on a benchmark, it may cease to be a good indicator of model quality. These figures are as of early 2026.