IBM researchers report that agents which appear accurate on average can still be unreliable across repeated runs. On the AppWorld benchmark, a ReAct agent using GPT-4.1 achieved a Mean@5 of 77.4% but a Pass^5 of only 53.0%, producing a 24.4-point consistency gap: many tasks succeed sometimes and fail other times with no change to the task.
Measuring the problem
The team distinguishes three metrics. Mean@k is the common average pass rate across k runs. Pass^k is the fraction of tasks the agent solves on all k independent runs. Pass@k asks whether at least one of k attempts succeeds. Pass^k ≤ Mean@k ≤ Pass@k always, and the researchers call Mean@k − Pass^k the consistency gap.
In their setup the ReAct agent ran at temperature 0.0, so ordinary sampling variance was not the source; instead, the authors attribute flip-prone behavior to “flat” token distributions at particular decision points, where near-tied options can reorder under small perturbations.
Diagnosis and targeted fixes
To locate unstable steps, the researchers built the Consistency Analyzer as part of ALTK-Evolve. The analyzer replays a single recorded trajectory and, for each decision step, issues one additional model call that requests k completions (k=5 by default) against the recorded context. This black-box resampling produces a consistency score per decision step without requiring ground truth, environment replay, or logits access.
Flagged steps are converted into targeted “consistency guidelines” in ALTK-Evolve format and injected at inference time. Examples include preferring line-anchored regex matches for counting checkbox markers and verifying multiple search results before proceeding—patterns that recur across tasks rather than patching a single trace.
On AppWorld test_normal (168 tasks) with GPT-4.1, generating guidelines from one baseline trajectory per task and testing on five fresh runs raised aggregate Pass^5 from 53.0% to 69.0% while Mean@5 moved from 77.4% to 81.0%, reducing the consistency gap from 24.4pp to 12.0pp. Medium and Hard tiers showed the largest relative gains (Medium +22.9pp, Hard +14.3pp; Easy +12.2pp). A weaker model, gpt-oss-120b, saw same-task Pass^5 rise from 10.1% to 16.1% (+6.0pp), with similar-task generalization gains that in one case exceeded the same-task improvement.
The authors recommend reporting Pass^k alongside Mean@k and note that the Consistency Analyzer requires only one extra LLM call per decision step, making it usable on production traces that cannot be replayed end-to-end. The ALTK-Evolve open-source repository and a technical report on arXiv contain the full methodology and implementation details.
Original source: Hugging Face Blog