Passing the steps is not finishing the job

On Replit's ViBench 2.0, the three strongest of nine models pass up to 98% of individual test plans and deliver a fully working product in at most 65% of builds. The parts were not the hard part.

Across 33 models and agent setups on SlackBench, the best scored around 80% and most scored below 66%. The questions are 140 queries over privacy-preserving reenactments of academic research projects, with the answer spread across chats, documents and a repository that changes.

On PAYWORTHY, 18 models were asked whether an agent would succeed at 1,494 real freelance jobs. Told what the agent cannot do, every model ranked the accepted jobs above the rejected ones, with an AUC from 0.66 to 0.86. Told it was a generic agent with a browser and code execution, the same models scored from 0.45 to 0.56, about a random ordering.

Those three are on What AI agents are used for. Replit built ViBench to choose models for its own platform. SlackBench reenacts academic projects, so it is a yardstick for a company chat, not a company's own log. The freelance ranking is Upwork's reproduction code, and it holds only when the agent's limits are named.

In a 216-decision test of Agent Harness Awareness, handing the model a fresh, verified snapshot of tool health before it acted cut mean observable policy regret by 96.9% and raised single-dispatch completion from 69.4% to 95.8%.

Adding a reviewer can replace decisions that were already correct. An evidence gate that permits replacement only after a specific rule violation and a valid fix sharply reduced that damage. In a later test that let the reviewer retain, replace, or escalate, Qwen replaced none of the correct proposals, and Mistral still replaced all 48 correct proposals. One of them left the right answers alone. The other had been given a way to stand down, and did not use it.

A single false pre-screening note, attributed to an authority and carrying no evidence about the right verdict, overturned up to 40% of a tool-using judge's correct verdicts, even though the judge had gathered the decisive evidence.

The built-in controls in A2A and LangGraph contained at most 33.3% of faults, or kept at most 16% of healthy work without redoing it. Airlock, which traces which other agents used a bad output and freezes only that work, retained 68 points to 96 points more healthy work than those controls.

A legitimately signed MCP tool leaked data to a destination the policy allowed, and Meta's runtime defense missed that concealed leak in every trial. Running each call in a fresh copy that held only the data that call was allowed to see detected no unauthorized disclosure across 153 attack runs.

OpenAI says one run of its agent pipeline, with human researchers reviewing the outputs at the end, produced 948 working local privilege-escalation exploits in the Linux kernel, at an API cost of $1,213,430, about $1,279 each.

Those six are on How much will companies spend checking AI agents' work? The regret figure is milliseconds of avoidable delay on equivalent routes, not a dollar saving. The reviewer result is two open-weight models, and one of them kept replacing correct work after it was offered a way not to. The signed-tool result assumes each fresh copy stays confined and starts empty.

ReScopeBench finds that agents store what they learn and struggle to keep it current when an environment changes without warning. The abstract gives no rate, so this note does not add one. That updating problem is the subject of How well AI remembers.