Ai2's StateBench: Frontier Models Fall to 13% or Lower When Long Documents Keep Changing

Long-context tests usually ask a model to find something: a fact buried in a long document. Real work often asks for more. A model reading a codebase has to follow variables that are reassigned, and an agent working through a long task has to remember what changed three steps ago. StateBench measures that second skill directly.

The four levels

What Ai2 found

• Frontier models: Gemini 3.8 Flash, Claude Opus 5.5, GPT-6 Sol and GPT-6 Luna did well on r-trivial and aperiodic problems. On soluble and general finite problems, accuracy fell sharply as inputs grew, to 13% or lower on the longest general finite problems. Ai2 doesn't give a per-model breakdown in its post.
• Open pretrained models: Olmo 3 (a transformer) and Olmo Hybrid (with Gated DeltaNet layers), both 7B, and smaller 275M and 2.7B models, all trained on the same data with contexts up to 64,000 tokens. Almost all did well on r-trivial, lost accuracy on aperiodic as inputs grew, and stayed at chance on general finite.
• Format matters: small hybrid and Gated DeltaNet models were near-perfect on retrieval-style r-trivial problems but near chance on the same problems written as spreadsheet edits, which is why StateBench renders each problem in several formats.
• Trained directly on the task: 275M-parameter models trained on up to 256 assignments were tested on up to 64,000. Gated DeltaNet and hybrids generalised almost perfectly on the first three levels. Transformers degraded on longer inputs and never learned general finite. The most flexible Gated DeltaNet variant stayed near perfect on the hardest level to about 10 times its training length.

Limits Ai2 points out

• StateBench uses a fixed number of variables. In real conversations, new people and objects keep appearing, and the benchmark doesn't cover that yet.
• Where one Gated DeltaNet variant sits in the hierarchy is still an open question in Ai2's theory.
• The frontier results come from asking models to read a sequence of assignments and output every final value, not from a real coding or agent task.

What this means if you use AI on long documents or agents

• Retrieval scores don't tell you whether a model can follow changes. A model that aces a needle-in-a-haystack test can still lose track of a value updated many times.
• For long agent runs, keep state outside the model: write the current values to a file or database the agent re-reads, instead of trusting it to remember a long chain of edits.
• Watch the architecture. Ai2's results support hybrid models with recurrent layers, such as its own Olmo Hybrid, for tasks that depend on following long chains of changes.