What Are Tokens In AI?

A museum team wants to compare how much text 2 language models can accept. They count the words in a visitor’s comment and assume the same count applies to both models. But the model processes a tokenizer’s sequence. What would the team have to count, and what configuration must

travel with that number? A tokenizer converts an input into units and identifiers under its vocabulary and algorithm. A unit may be a whole word, a fragment, punctuation, a whitespace pattern, a byte-related piece, or a special symbol. The exact split is part of the configuration.

To make that concrete, we will use 2 invented classroom tokenizers whose rules are supplied, rather than report a measurement from a production model. Give both classroom tokenizers the comment unbelievable, followed by an exclamation mark. Tokenizer A has the complete word in its vocabulary and separates the punctuation.

It emits 2 units: unbelievable and the exclamation mark. Tokenizer B represents the word with 3 supplied fragments, un, believ, and able, then emits the exclamation mark. It emits 4 units. Rejoining the pieces recovers the

same displayed comment, while the model sequence has a different length. The fragments need not be dictionary words or a linguist’s preferred division. The vocabulary and splitting procedure determine them. Character or byte-related fallback can help a tokenizer represent material that lacks a convenient whole-word entry.

Repeated patterns and vocabulary design can make some text compact and other text longer. Code, names, spelling, scripts and formatting all matter. A fixed number of English words cannot establish a universal token count. There is a historical reason to look below whole words.

In their August 2016 paper, Sennrich, Haddow and Birch used subword units to address rare and unseen words in neural machine translation. Their method represented words through smaller reusable sequences, including a segmentation method adapted from byte pair encoding. They evaluated English-to-German and English-to-Russian translation in the reported setting.

The case explains the representation problem; it supplies no measured token count for today’s museum inputs. Each emitted unit selects an identifier in its tokenizer’s vocabulary. In our classroom B vocabulary, suppose un has ID 12, believ has ID 903, and able has ID 35.

Those numbers are lookup identities. 903 does not mean that fragment has more meaning than 35, nor does subtraction between those IDs measure semantic similarity. The next lesson will show the learned vector selected by an ID. A chat input also has structure. A user message and an assistant

message are different parts of a conversation. Special units or formatting can mark their boundaries and roles, and indicate where generation should begin. In a simplified classroom format, adding 1 user marker and 1 end marker adds 2 units to our comment sequence. A’s 2 content units become 4

formatted input units; B’s 4 become 6. Those counts follow the supplied teaching rules, with no claim that a particular service uses these markers or counts. A real implementation example appears in Hugging Face’s Transformers version 4.57.1 chat-template documentation. It contrasts Mistral 7B Instruct and Zephyr,

which were fine-tuned from the same base family but use different message-control formats. The example makes a practical point: keep the tokenizer and expected chat template with the chosen model. Copying the visible words into another model’s control format changes the input the model was trained to interpret.

The documentation version describes that example, rather than every current service’s protocol. This is why a useful token measurement names the tokenizer, its version or revision, the exact input and the formatting procedure. Count the final representation under the relevant convention. A visible document may contain headings,

whitespace, tables and message wrappers that a word count omits. If an application reserves room for an answer, record that separate allowance too. Published limits and billing conventions need the exact service’s definition. File bytes answer another question. A compact PDF can contain

many pages of text, or scanned page images with no extracted text. Extraction or a vision-processing path may be needed before the application obtains the representation used by its model. Image and audio inputs can have different encodings and accounting. A small file on disk

therefore supplies no direct token-budget guarantee. We will examine context capacity and temporary memory in more detail later. Here is a checkpoint. Under our supplied classroom rules, tokenizer B emits un, believ, able and the exclamation mark, then 1 user marker and 1 end marker are added.

How many units are in that formatted input? Does the largest token ID identify the most important fragment? Pause and decide. There are 6 units: 4 content units and 2 markers. The identifier values name entries;

their numerical order does not measure importance. This answer depends on the classroom rules we supplied. A real comparison would measure the chosen tokenizer and final formatting, then keep their identities in the experiment record. For a separate transfer, a team translates a notice and

gives both versions the same character limit. They claim that context use and cost are now equal. What would you measure before accepting that claim, and what other performance question remains? Pause and make a brief plan. Measure each exact input using the tokenizer and formatting matched to its model.

Record the service’s input and output accounting, rather than infer tokens from characters. Then evaluate whether the model handles the language and task reliably. Equal token counts would still leave that capability question open; unequal token counts explain why the same character limit may consume different budgets.

The museum can now replace its word-count shortcut with a reproducible token measurement. Keep the input, tokenizer, vocabulary revision and chat format together. The sequence’s IDs still do not tell us how the model represents meaning. Next, we will follow an ID into its learned vector and see how

surrounding context changes a hidden representation.