Part 10
How the model
is trained
Nobody programmed WillMe to answer questions. It was shown an enormous amount of text and corrected, two billion times, until it got good at guessing.
Guess, check, adjust
Training is one loop repeated until the numbers stop improving. Show the model some real text, hide what comes next, let it guess, then compare its guess with the truth. Every one of its 100 million parameters gets nudged a little in the direction that would have made the right answer more likely.
Any single nudge is far too small to notice. Sixty-one thousand rounds of them, over two billion tokens, is what turns random numbers into something that writes sentences.
The elegant part is that no one has to label anything. The text is the answer key: for every position, the correct next token is simply the one that was actually there.
Three stages, one continuous run
The context window is widened as training goes on — short passages first, because they are cheaper, then longer ones.
Sequence length goes 512 → 1,024 → 2,048 tokens across the three stages.
From text-continuer to assistant
After all that, the model is good at continuing text and useless as an assistant. Ask it a question and it might carry on with three more questions, because that is what a page of questions usually looks like.
So there is a second, much smaller phase. It is shown example conversations — a user message, then a good assistant reply — and this time it is only corrected on the assistant's words. It is not being taught new facts so much as a format: when you see this marker, produce a reply, then stop.
A third phase, smaller still, tunes the model towards sounding like William. That is what the comparison study exists to measure.
The three phases, to scale
Pretraining
Two billion tokens. Learns language, facts, and how sentences work.
Conversation tuning
Learns the shape of a reply, and when to stop talking.
Personality tuning
Nudges the style towards William's.
Why the last step is risky
Personality tuning updates every parameter in the model, not a separate “style” component bolted on the side. There is no such component. The same numbers that hold the model's grip on Swedish grammar and its scraps of factual knowledge are the ones being adjusted to make it sound more like William.
So the risk is real: teaching it a voice can quietly cost it accuracy or coherence. That is why the earlier, more general checkpoint is kept as a fallback, and why the preference study asks people to judge whether replies are appropriate as well as William-like.
The exact training recipe
Pretraining. One continuous AdamW run over three stages:
| Stage | Sequence | Updates | Tokens |
|---|---|---|---|
| Warm foundation | 512 | 6,104 | 200,015,872 |
| Main training | 1,024 | 45,776 | 1,499,987,968 |
| Context extension | 2,048 | 9,155 | 299,991,040 |
| Total | 61,035 | 1,999,994,880 |
Global batch 32,768 tokens; AdamW with betas (0.9, 0.95),
epsilon 1e-8, weight decay 0.1, gradient
clipping 1.0, 610 warmup steps, cosine decay from a peak
learning rate of 4e-4 down to 10% of it.
The loss.
L = CrossEntropy(logits[:, 0:T-1], labels[:, 1:T]) —
every position predicts the next one. Labels of -100 are
ignored, which is how SFT masks everything that is not an assistant
token. Cross-entropy runs in FP32 even though the forward pass is under
BF16 autocast.
Personality SFT. Full-parameter fine-tuning from the
preserved remediation checkpoint — no LoRA, no adapters, no frozen
layers, no runtime personality module. Micro batch 1, gradient
accumulation 16, peak learning rate 1e-6 (four hundred times
smaller than pretraining), weight decay 0.01.
Activation checkpointing is on per block during training: instead of keeping every intermediate value, each block is recomputed in the backward pass. Cheaper in memory, more expensive in compute, and inactive during generation.