The Tokenizer Trap in Claude Sonnet 5

Imagine renting a car where the daily rate stays exactly the same, but the rental company secretly shortens the definition of a "mile" by 30%. Your road trip just got significantly more expensive. This is essentially what is happening in the rapidly evolving economy of generative AI.

Anthropic recently launched Claude Sonnet 5, a model that promises the robust performance of their premium Opus 4.8 tier but at a fraction of the cost. On paper, the specs are staggering: a sprawling 1-million-token context window, the ability to generate up to 128,000 tokens in a single output, and an "adaptive thinking" mode that is turned on by default. Interestingly, Anthropic noted that they deliberately throttled the model's cyber capabilities compared to an unreleased "Mythos 5" model, a safeguard that allowed them to deploy it without facing pushback from the US government.

The real headline for developers, however, was the pricing. The base cost remains identical to its predecessor, Sonnet 4.6, at $3 per million input tokens, complete with an introductory discount running through August.

But a closer look at the developer documentation reveals a significant catch. Sonnet 5 introduces a new tokenizer—the underlying dictionary mechanism that chops human language into the data fragments (tokens) that AI actually processes and bills for. For the exact same text, this new tokenizer produces roughly 30% more tokens than the previous version.

Independent tests highlight how drastically this affects real-world budgets. When running the Universal Declaration of Human Rights through the new system, the English text produced 42% more tokens than before. Spanish text jumped by 33%, and a standard Python script saw a 28% increase. Essentially, while the price-per-token hasn't changed, users are forced to buy far more tokens to process the exact same prompts.

Fascinatingly, this "shrinkflation" doesn't impact all languages equally. The same tests showed that Simplified Mandarin remained almost entirely unaffected, with a negligible 1% increase in token count.

The update brings a few other quirks as well. Anthropic has removed manual developer controls for creativity, such as temperature settings, handing full control over to the model's adaptive systems. And despite its advanced reasoning, it still struggles with absurd visual edge cases—when shown an image of a pelican riding a bicycle, Sonnet 5 confidently identified it as a goose.

As AI models grow more sophisticated, so does the complexity of understanding what we are actually paying for. The sticker price of an AI model is no longer the whole story; the real cost is increasingly hidden in how the machine counts our words.

Key Points

• Claude Sonnet 5 offers Opus 4.8-level performance with a massive 1-million-token context window.
• Anthropic limited the model's cyber capabilities to avoid US government deployment blocks.
• While the official price per token remains unchanged, a new tokenizer inflates token counts by roughly 30%.
• English and Python inputs cost significantly more to process, while Simplified Chinese costs remain virtually flat.

Why It Matters

As AI companies shift how their models process language, users must look beyond the official pricing page to understand the true operational costs of generative AI.

Sources:

• What's new in Claude Sonnet 5 — Simon Willison's Weblog