Blog
Lost in Translation: What LLMs Actually Understand About Mathematics
The 'LLMs hallucinate math' narrative is a coarse oversimplification. Recent research reveals that models encode deep mathematical structure internally — the real bottleneck is the lossy translation from latent space to human-readable output.
Ask anyone in tech whether LLMs can do math, and you’ll get a confident answer: they can’t. They hallucinate. They confabulate. They generate plausible-sounding nonsense and dress it up in notation.
I was skeptical of this narrative from the start. People have an inherent mistrust of what they don’t understand, and they tend to treat AI as a black box — probing it like an animal behavioral study and drawing sweeping conclusions from surface-level observations. Yes, next-token prediction doesn’t sound like how mathematicians learn. Mathematicians don’t study plausibility — they solve problems, construct proofs, and build rigorous arguments from first principles.
But the reasoning capabilities of these models always made me feel like “hallucination” was a coarse, overused word. And a recent paper finally gave that intuition a precise, empirical foundation.
The Collatz Paper: When Failure Modes Tell the Real Story
In 2025, Francois Charton and Ashvni Narayanan published Transformers Know More Than They Can Tell — a study that trained transformers on long steps of the Collatz sequence and then carefully analyzed how and why they failed.
For those unfamiliar: the Collatz conjecture is one of mathematics’ simplest-to-state, hardest-to-prove problems. Take any positive integer. If it’s even, divide by two. If it’s odd, multiply by three and add one. Repeat. The conjecture says you always eventually reach 1. The sequences themselves can be extremely long and structurally complex.
What Charton and Narayanan found was striking. The models didn’t fail randomly. They learned genuine deep mathematical structure — binary representations, residue classes, loop patterns. In over 90% of failures, the model performed the correct arithmetic but misjudged the control structure — getting the loop length wrong, not the math itself.
This is the finding that stood out to me most: the errors are highly structured. The model isn’t guessing or confabulating. It has internalized the mathematical operations. The difficulty is in learning the meta-reasoning — the control flow of the computation, not the computation itself.
That’s a fundamentally different diagnosis than “LLMs hallucinate math.”
The Latent Space Knows More Than the Output Shows
Here’s an analogy that captures what’s going on. Imagine a brilliant mathematician who is fluent in advanced algebra, topology, and number theory — but can only communicate through a game of telephone. The knowledge is real. The understanding is genuine. But every time the mathematician tries to express a result, it passes through a lossy channel that garbles the precision.
That’s the relationship between an LLM’s latent representations and its natural language output. The Collatz paper demonstrated that transformers internalize deep mathematical properties — binary representations, residue classes, structural patterns — that don’t fully survive the decoding phase into human-readable text. The math is in there. Something gets lost in translation.
Natural language, it turns out, is a remarkably poor format for conveying precise mathematical reasoning. It’s ambiguous, it’s sequential, and it forces the model to serialize complex structural relationships into a flat stream of tokens. A model that has internalized the modular arithmetic of Collatz sequences has to compress that understanding into English sentences — and compression means loss.
Representation Shapes Understanding
The Collatz paper revealed another result that deserves more attention: the encoding base dramatically affects model accuracy. The same model, the same task, the same architecture — but accuracy swung from 99.7% in base 32 down to 25% in base 3.
| Encoding Base | Model Accuracy |
|---|---|
| Base 32 | 99.7% |
| Base 10 | ~85% |
| Base 3 | 25% |
That’s not a marginal effect. The way you represent mathematical objects to an AI fundamentally shapes what it can learn and express. This echoes one of the key insights from DeepMind’s AlphaGeometry system, which achieved breakthroughs in geometry reasoning by encoding geometric objects into a symbolic language specifically designed for the task. Representation isn’t a detail — it’s the whole game.
This has direct implications for how we design AI systems for mathematical reasoning. If we’re feeding models mathematical problems in natural language and then judging their mathematical understanding by their natural language output, we’re measuring the channel, not the knowledge.
Reframing Hallucination
The word “hallucination” implies the model is making things up — confabulating, generating fiction with confident delivery. But the Collatz results suggest something qualitatively different. The model has real mathematical knowledge that gets degraded through the output channel. That’s not hallucination. It’s mistranslation.
This distinction matters beyond semantics. If you believe LLMs hallucinate math, you conclude they need fundamentally different architectures for mathematical reasoning. You look for external scaffolding — calculators, code execution, retrieval systems. Those aren’t bad ideas, but they treat the model as mathematically empty and try to fill it from the outside.
If you believe LLMs mistranslate math, you draw a different conclusion: the knowledge is already there, and the challenge is building a better output channel — one that preserves the precision of what the model has actually learned. That’s a different engineering problem with a different set of solutions.
For founders and investors evaluating AI capabilities, this reframing is worth sitting with. The common conclusion — that LLMs are inherently unsuited for rigorous reasoning — may be premature. The bottleneck might not be as far from being solved as it appears on the surface. But solving it requires looking within the models, not just bolting on external tools.
The Open Question: Mathematical Smell
Terence Tao — perhaps the most respected living mathematician — has made an observation that ties this together. He’s noted that AIs have developed what you might call “smell” for games like Go and Chess: an intuitive sense of whether a position is good or bad, developed through deep training. But they haven’t yet developed that same intuition for mathematical proofs.
“Mathematical smell”: This is a thing. It’s not clear how to get the AIs to duplicate that.
— Terence Tao
Tao’s observation is precise. But the Collatz paper suggests that the gap between where models are and where they need to be might be smaller than it looks from the outside. If transformers are already internalizing deep structural properties of mathematical sequences — just failing to express them — then the intuition layer may be forming. The question is whether we can build an interface that lets it out.
What would that interface look like? Not natural language — we’ve established that it’s lossy. Something structured. Something unambiguous. Something where every step must be mechanically verified.
That interface already exists. It’s called formal verification. And it changes everything.
This is the first in a series of three posts exploring the intersection of AI and mathematical reasoning. Next: Beyond Next-Token Prediction: The Case for Immutable Verification — on how formal verification acts as the lossless channel that unlocks what LLMs already know.
References
- Charton, F. & Narayanan, A. (2025). Transformers Know More Than They Can Tell: Learning the Collatz Sequence.
- Lewkowycz, A. et al. (2022). Solving Quantitative Reasoning Problems with Language Models (Minerva). Google Research.
- Trinh, T. H. et al. (2024). Solving Olympiad Geometry without Human Demonstrations (AlphaGeometry). DeepMind / Nature.
- Tao, T. Various public remarks on formalization, proof assistants, and “mathematical smell.”