Why Frontier AI Stumbles on Children's Puzzles

Imagine a machine that has memorized a significant portion of the internet, can write sophisticated software, and casually passes graduate-level exams. Now imagine that same machine completely losing its computational mind over a simple game of Tower of Hanoi.

Games and puzzles have been the ultimate testing grounds for artificial intelligence since 1959, when IBM computer scientist Arthur Samuel popularized the term "machine learning" with a checker-playing algorithm. Fast forward to today, and AI's puzzle-solving skills are advancing at a breakneck pace. A study from Columbia University showed that in late 2024, top models could only solve 18% of the New York Times' Connections puzzles. By early 2025, they were acing them almost every time.

But if you look closely at the puzzles AI still fails to solve, a fascinating picture of machine cognition emerges. It turns out that AI doesn't "think" like we do—and its massive computational power comes with bizarre blind spots.

Take spatial reasoning, for example. Humans possess an intuitive ability to perform "mental rotation"—looking at a 2D image of a 3D object and imagining what it looks like from another angle. Despite recent leaps in multimodal capabilities, large language models (LLMs) fail abysmally at this. They lack the intrinsic understanding of physical space that a human architect or mechanic uses effortlessly.

Even more revealing is how an AI's vast memory can actually become a liability. Frontier models are trained on monstrous volumes of text, meaning they have likely "seen" almost every classic logic puzzle ever published. In a 2024 study, researchers from Google and the University of Illinois Urbana-Champaign tested models on the classic "Knights and Knaves" logic riddles, but introduced subtle variations. Instead of reasoning through the new rules, the models often whizzed past the changes and simply regurgitated the standard answer they had memorized during training. When faced with slight twists, human adaptability wins out over machine recall.

Scale and complexity also expose AI's limits. Research from Apple demonstrated that while LLMs can successfully solve simple versions of the Tower of Hanoi or river-crossing puzzles, their logic shatters as soon as the number of disks or people reaches six. They can handle the pattern, but they cannot sustain the underlying reasoning as the variables increase.

Interestingly, humans have our own cognitive foibles. We frequently fall for intuitive math traps that AI, with its deliberative processing, easily avoids. But in visual reasoning benchmarks like ARC-AGI, humans rely on simple, elegant visual concepts, while AI models attempt to brute-force solutions using byzantine, non-generalizable rules.

These puzzles serve as a humbling reminder: AI may be incredibly powerful, but it is not a human brain. It simulates reasoning through vast pattern recognition. So, the next time you feel intimidated by AI's capabilities, try giving it a spatial brain-teaser. You might just outsmart the machine.

Key Points

• Despite rapid improvements, AI models still struggle significantly with 3D spatial reasoning and mental rotation.
• AI's vast training data can cause it to fail on modified logic puzzles, as it tends to recite memorized answers rather than adapt to new rules.
• AI reasoning breaks down quickly when the complexity of sequential logic puzzles, like the Tower of Hanoi, increases beyond a few steps.
• Puzzles highlight the fundamental difference between human intuitive reasoning and AI's reliance on pattern matching.

Why It Matters

Seeing where AI fails at simple logic tasks helps demystify the technology, revealing that it relies on pattern matching rather than possessing true, human-like comprehension.

Sources:

• AI models flub these intelligence tests. Can you fare any better? — MIT Technology Review - AI