Part 6

Predicting the next token

Every reply is built one token at a time. Here is what happens between your words going in and a single word coming out.

From numbers to meaning

A token ID on its own means nothing — token 2065 is no more “language” than the number 2065 is. So the first thing the model does is look each ID up in a large table and replace it with a list of 768 numbers.

That list is the token's embedding, and it is where meaning starts to live. Tokens used in similar ways drift towards similar lists during training, so cat and dog end up closer together than cat and bureaucracy. Nobody wrote those relationships down; they are a side effect of predicting text well.

What one token becomes

32,768 scores, every time

After the layers have done their work, the model produces one score for every single token it knows — all 32,768 of them. Not a word, not a sentence: a score for every possibility, saying how well each one would continue the text so far.

Those scores get squashed into percentages that add up to 100%. Usually a handful of candidates hold nearly all of it and the rest are somewhere near zero.

Temperature

The same scores, made flatter or sharper before a word is picked.

“I made coffee this morning and completely ___”

Temperature 0.8

Illustrative numbers run through the real softmax formula, so the way the bars respond is genuine even though the starting scores are invented.

Picking a word

With percentages in hand there are two ways to choose. Always take the highest, and the model is completely predictable — the same prompt gives the same answer forever. Or roll a weighted die, and it varies between attempts.

WillMe takes the highest by default. That is why asking the same thing twice gives you the same reply. It is worth being clear about what that does and does not buy you: reproducible is not the same as correct. A confidently wrong answer will be confidently wrong every time.

The chosen token is added to the text and the whole process runs again for the next one, until the model produces its “I am finished” token or hits the length limit.

Logits, softmax and the sampling settings

The final hidden state is multiplied by the transpose of the embedding matrix — the same parameters used on the way in — giving logits of shape [batch, position, 32768]. Reusing the embedding as the output head is called weight tying and saves 25,165,824 parameters, a quarter of the whole model.

Scores are converted to FP32, a repetition penalty of 1.10 is applied to every token already present, and then either argmax is taken (the default) or, when sampling is explicitly requested, the scores are divided by the temperature and filtered:

temperature	0.8
top_k	50
top_p	0.95

Softmax itself is p_i = exp(z_i / T) / Σ exp(z_j / T). As T approaches zero this becomes argmax; as it grows, the distribution flattens towards uniform. Generation stops at <|end|> or <eos>, or when prompt plus reply reaches 2,048 tokens.