Four levels of difficulty
StateBench defines four increasing levels of state-tracking complexity to mirror different kinds of variable assignment. The report labels them r-trivial (immutable assignment), aperiodic (reassignment by value), soluble (self-referential updates like x = x + 5) and general finite (assignment by reference, including swaps and the S₅-style permutation problems).
The authors connect these levels to Krohn–Rhodes theory, arguing that each class corresponds to problems built from different atomic primitives. To compare difficulty fairly, the benchmark converts a single sequence of assignments into four formats that preserve assignment locations and values while varying the update type.
Findings from evaluations
Pretrained open models were tested first. The report says two 7-billion-parameter models—Olmo 3 (a transformer) and Olmo Hybrid (which adds Gated DeltaNet layers)—and smaller models (275M and 2.7B parameters) performed well on r-trivial tasks, degraded on aperiodic tasks as inputs lengthened, and remained at chance on general finite problems. Format also mattered: the 275M hybrid and Gated DeltaNet models scored near ceiling on RULER-style r-trivial problems but near chance on the same problems framed as spreadsheet edits.
Four frontier models—Gemini 3.8 Flash, Claude Opus 5.5, and GPT-6 Sol and Luna—handled r-trivial and aperiodic problems but, according to the report, accuracy fell sharply on soluble and general finite tasks, dropping to 13% or lower on the longest general finite sequences tested.
Direct training and open questions
To separate training from architectural limits, researchers trained six 275M models from scratch (two transformers, two hybrids, and two Gated DeltaNet variants) on sequences up to 256 assignments and evaluated them on sequences up to 64,000 assignments. The report states that Gated DeltaNet and hybrid models generalized almost perfectly on r-trivial, aperiodic, and soluble problems out to the longest lengths tested, while transformers degraded on longer sequences and failed to learn general finite problems. A Gated DeltaNet variant with a more flexible memory update generalized farthest—near perfect to roughly ten times its training length.
The report notes unresolved issues: the exact placement of some Gated DeltaNet variants in the hierarchy, effects of numerical precision and attention, and the current framework’s fixed-number-of-variables assumption. The researchers released StateBench and associated datasets for further study.
Original source: Hugging Face Blog