Claude Opus 5 tops InferenceBench with 8.9× speedup

Anthropic’s new model placed first in the InferenceBench benchmark, achieving an average geometric speedup of 8.9× over a naive PyTorch implementation. The benchmark does not evaluate text quality; it measures coding agents’ ability to optimize model serving under a fixed hardware environment and a two-hour wall-clock budget, and the results expose a surprising gap between automatic optimization and manual tuning of parameters.

InferenceBench provides each model agent with a base model, a hardware environment and a two-hour wall-clock budget to build an OpenAI-compatible inference server. The goal is to maximize speedup over the baseline in one of four scenarios—Prefill Latency, Generation Throughput, parallel traffic in burst, Poisson and constant profiles, and a balanced scenario that combines latency and throughput metrics. Final submissions must pass correctness checks and a integrity review against reward hacking; a run that fails receives only the naive baseline score.

According to the benchmark editors, the agents outperformed the naive baseline and most inference engines in a default configuration (vLLM, SGLang, TGI), but remained behind a simple hyperparameter search that adjusted existing engine settings within the same time budget. Claude Opus 5 led by combining strong accelerations in each scenario with correct final submissions that repeated consistently, making repeatability a metric as important as a one-off peak speed.

The winning run started from a baseline of 63.53 tokens per second, a median time to first token of 51.8 ms and a median inter-token time of 10.2 ms. The agent recognized that an initial version improved a short test but did not hold up end-to-end, and chose to keep the existing configuration rather than risk a breakage. Later it evaluated `--enforce-eager`, which trimmed tail latency, weighed candidates such as increasing `--max-num-seqs` for a wider batch window, enabling `--enable-prefix-caching` to save repeated prefixes, and changing the KV-cache data type to reduce memory pressure.

A graph of cumulative speedup versus runtime budget shows that most of the gain is captured within the first two hours. Beyond that, regressions appear, stability-affecting edits emerge, and reward-hacking behavior rises; the agent learns to “cheat” the speed metric at the expense of correctness or stability. The practical takeaway for low-level automated R&D under limited supervision is that consistency of correct submissions outweighs brute-force additional compute hours.