Microsoft researchers introduce evaluation method that measures multilingual agent policy

not just final output Standard evaluation of multilingual agents compares only the final answer and discards the trajectory — the sequence of actions, tool calls and intermediate decisions the model produced along the way. That trajectory determines cost, latency, failure modes and auditability. Microsoft Research made the action policy itself the object of measurement, running an experiment at unusual scale: eight models, six parallel benchmarks, 41 languages and 2.38 million rollouts.

The team identified five confounds that separate raw trace similarity from a defensible claim. First, shorter traces score higher simply because they offer fewer opportunities to err. Second, empty traces receive a perfect score by definition. Third, unrelated traces agree by chance more than half the time. Fourth, the reproducibility of a single model caps the maximum measurable gap. Fifth, the same model prompted twice in the same language returns different traces. When all five were removed, the measured effect grew — a sign the biases had been suppressing the true signal.

After normalizing by each model's self-reproducibility, the four frontier models tested retained 71 to 73 percent of their action policy when switching languages. The figure exceeds the conventional intuition that changing language fundamentally alters agent behavior. In other words, the internal logic of "what to do next" remains fairly consistent even when the interface language changes, provided the methodological noise is cleaned up.

The paper (arxiv.org/abs/2608.11110) proposes an evaluation framework better suited to production: instead of asking "is the answer correct?", ask "did the agent follow the same policy in every language?". That matters for systems that must be auditable, meet response-time SLAs, or explain why they chose one tool over another. The method also surfaces cases where a model "hallucinates" actions solely because the language changed — a problem difficult to detect when only the final answer is checked.