Tokenization explained: why AI sees chunks, not letters
aieducation
A closer look at tokens, the building blocks behind how language models read text, and why they cause counting and spelling errors.
Tokenization is the first step in how a language model reads text, and it explains many odd behaviors, from miscounted letters to inconsistent word counts.
What a token is
A token is a chunk of text: a whole word, part of a word, a number fragment or punctuation. The model never receives raw letters. It receives a sequence of token IDs.
Why "strawberry" is hard
The word may be split into pieces like "straw" and "berry." The letters inside each piece are sealed off from the model, so counting the Rs becomes a guess from learned patterns instead of a direct count.
How numbers get split
Long numbers are often broken into several tokens, sometimes in uneven ways. A figure like 347.60 may be read as separate fragments. This makes digit-level arithmetic harder, because place value is not always visible to the model.
What this means in practice
- Word counts drift: a request for 150 words may return 118.
- Letter tasks fail: spelling, anagrams and character counts are unreliable.
- Formatting-sensitive math slips: decimals, commas and currency symbols change how numbers are split.
How to work around it
- Use code for counting and exact arithmetic.
- Have the system verify output before it is shown.
- Keep humans in the loop for high-stakes figures.
Tokens also affect cost and memory limits. See context windows and model choice and how AI thinks.
Understanding tokens helps set realistic expectations: models are strong with language, and they need support for exact counting.