The complexity cliff: what Apple's puzzle study measured

Reasoning models think harder as problems get harder. Then, at some point, they quietly stop.

In June 2025 a group of Apple researchers published "The Illusion of Thinking", one of the most argued-over AI papers of the year. Rather than run standard benchmarks, they used controllable puzzles: Tower of Hanoi, River Crossing and similar. The advantage is that you can turn a single difficulty dial while the logic stays identical, and you can read the model's reasoning trace at every level. They tested frontier reasoning models, including OpenAI's o-series, Claude 3.7 Sonnet Thinking and DeepSeek R1.

Three regimes emerged, comparing reasoning models against standard models at equal compute. At low complexity the standard models win, because the extra thinking adds cost and sometimes talks the model out of a correct answer. At medium complexity the reasoning models win, and the step-by-step traces genuinely help. This is the zone the demos show you. At high complexity both collapse: beyond a threshold accuracy falls to zero, and both kinds of model go over the same cliff.

The strangest detail is what happened to effort. As the puzzles got harder the models spent more thinking tokens, as you would expect. Then, past a threshold, they spent fewer, despite having budget left. Approaching the cliff, they reduced effort. Even when handed the exact solution algorithm, they failed at the same point. Whatever the traces are, they behave differently from a person executing a procedure.

Then came the rebuttal. Within weeks, a comment paper by Alex Lawsen (arXiv:2506.09250) argued that much of the collapse was an artefact of the test. Large Tower of Hanoi solutions exceeded output token limits, and some River Crossing puzzles were mathematically unsolvable, yet the models were scored as failures on them. When models were asked to output a compact solution program instead of every individual move, they handled instances previously marked as complete failures.

So part of the collapse was test design. Part survived scrutiny: the decline in effort near the threshold, and the failure to follow an explicit algorithm, remain live findings. Both sides of the debate are worth holding.

The practical point is that every task you delegate to a reasoning model sits somewhere on that complexity dial, and the model gives no warning as it approaches its own limit. The output looks confident right up to the edge. The lever that helps is decomposition: splitting one high-complexity task into several medium-complexity ones moves the work into the zone where these models are strong.

Two rules follow for delegation. Decompose before you delegate, breaking large problems into steps a model can verify one at a time, which moves the work down the complexity dial and out of the collapse zone. And test at your real complexity, because vendor demos live in the middle zone: before you trust a model with a workflow, probe it with your hardest real cases and find the limit yourself.

Capability is real, and so is the cliff. Reasoning models beat standard ones in a middle zone of difficulty and fail past a threshold without warning. The debate is about where that threshold sits and how much of it was the test. Know which zone your work is in, because the model will not tell you.