OpenAI published an analysis showing that two changes to the evaluation harness substantially altered performance on the ARC-AGI-3 benchmark. The company said that enabling retained reasoning and compaction—features of its Responses API—raised GPT-5.6 Sol’s score on the ARC-AGI-3 public set from 13.3% with the official harness to 38.3% with the model-friendly harness, while reducing output tokens by roughly six times.
What the harness did
ARC-AGI-3 is a set of 2D puzzle games designed to test how agents infer game rules and learn across episodes. OpenAI reported that the benchmark uses a deliberately generic harness that discards private reasoning messages after each action and applies a rolling truncation policy to older messages once the conversation grows large.
According to OpenAI, the combination of discarded private reasoning and rolling truncation prevented models from carrying forward their internal thoughts and, in some cases, from seeing earlier actions. The company noted earlier observations that GPT-5.6 Sol initially scored as low as 7.8% on ARC-AGI-3 and that GPT-5.5 scored 0.4% in early runs.
Effects of retained reasoning and compaction
OpenAI said that when the harness was refactored to use the Responses API, passing the previous response ID automatically preserved private reasoning across turns. Retained reasoning reduced the model’s per-action deliberation and helped it build coherent strategies. Replacing rolling truncation with compaction preserved older knowledge while keeping context manageable; the official harness had been discarding messages once the history exceeded 175,000 characters.
Measured on Relative Human Action Efficiency (RHAE), OpenAI reported a roughly threefold score increase for GPT-5.6 Sol and an approximately sixfold reduction in output tokens on the public set. OpenAI also estimated the average human tester scored 48% on the same metric, based on official gameplay logs.
OpenAI recommended that developers who seek similar performance use the Responses API, retain reasoning, and enable compaction, and suggested that model comparisons rely on evals that mirror these settings. The company pointed readers to public ARC-AGI-3 games for direct testing.
Original source: OpenAI News