Part 5
Turning text into numbers
The model has never seen a letter in its life. Before anything else happens, your text is chopped into tokens and each one is swapped for a number.
A token is usually a common word, a fragment of a longer word, or a punctuation mark. Frequent words get a token to themselves; rarer ones are assembled from pieces. WillMe knows exactly 32,768 of them, and everything you type has to be expressed using that fixed set.
This is why a model can seem oddly precise about common phrases and strangely clumsy with unusual names: the common phrase is one or two tokens it has seen constantly, while the unusual name might be five fragments it has barely encountered in that order.
Try it
This runs WillMe GPT 3.2's real tokenizer on whatever you type.
The amber tokens hold part of a character. Emoji and rarer symbols are not in the vocabulary, so they are stored as raw bytes and split across several tokens — none of which is a readable character by itself.
What the model actually receives
Why the odd splits?
You will notice tokens that start mid-word, and that a leading space belongs to the token after it rather than the one before. That is deliberate: it lets the model tell “model” at the start of a sentence apart from “ model” in the middle of one, without needing two separate entries for every word.
The vocabulary was built by starting from raw bytes and repeatedly merging the most common neighbouring pair. Common English and Swedish words survive as single tokens because they came up constantly; anything unusual falls back to smaller pieces, and in the worst case to individual bytes. Nothing is unrepresentable — emoji, Swedish characters and typos all encode fine, just at the cost of more tokens.
Byte-level BPE and the control tokens
The tokenizer is a 32,768-entry byte-level BPE with Unicode NFC normalisation, ByteLevel pre-tokenisation and decoding, and an initial byte alphabet — so every possible UTF-8 string has an exact representation.
Eight IDs are reserved as fixed control tokens, and they are how a conversation gets encoded as a single flat sequence:
| ID | Token | Purpose |
|---|---|---|
| 0 | <unk> | Unknown-token fallback |
| 1 | <bos> | Beginning of sequence |
| 2 | <eos> | End of a document |
| 3 | <pad> | Batch padding |
| 4 | <|system|> | System-message marker |
| 5 | <|user|> | User-message marker |
| 6 | <|assistant|> | Assistant-message marker |
| 7 | <|end|> | End of an assistant reply |
A chat turn is serialised literally as
<|user|>…<|assistant|>…<|end|>. To ask for
a new reply the server appends \n<|assistant|> and lets
the model continue until it emits <|end|>. Those
markers are rejected inside message text, so no one can forge a role
boundary by typing one.