Part 4
What makes it a
language model
A network that fits a line does not write anything. Four ideas turn one into a model that does: a self-supervised objective, a context window, a two-stage training diet, and a sampler.
Nobody labelled the training data Self-supervised: the label comes from the data itself. Every position in every sentence is a free training example, because the next word is right there.
The clever part of a language model is not the architecture. It is that the training data labels itself. Take any sentence, hide everything after position t, and ask the model what comes next — the answer is sitting in the text. One 2,048-token document is not one training example, it is 2,047 of them, all scored at once.
That is why these models can be trained on raw text at all. WillMe GPT 3.3's 1,999,994,880 training tokens needed nobody to annotate them.
Drag through the sentence: at each position the model sees only the left side and is scored on the very next token. Real mechanism, one short sentence.
The context window is the whole memory There is no other memory. Close the tab and nothing about your conversation survives; go past the window and the beginning falls out of it.
A model sees a fixed number of tokens at once, and that window is the entirety of what it knows about your conversation. WillMe GPT 3.3's is 2,048 tokens, roughly 1,500 English words. Everything — your question, the chat history, the reply being written — shares that budget, and the reply grows into it token by token.
The bar is to scale. Slide the conversation length and watch the room for a reply disappear, then watch the oldest turns fall out entirely.
Two stages, two different jobs
Training happens twice, and the two runs are not variations of each other. In pre-training the model reads a very large pile of general text to learn how language behaves at all. In fine-tuning it reads a much smaller, carefully chosen set to learn how one particular person or assistant answers.
The scale gap is the thing to notice. For WillMe GPT 3.3 it is two billion tokens of general text against 2,600 conversations of William.
Both stages drawn on the same time axis, with WillMe GPT 3.3's real figures. Pre-training is the long haul; fine-tuning is the short one that decides who the model sounds like.
The three pre-training stages, in order
WillMe GPT 3.3's pre-training run is split by sequence length, so the model learns short-range structure cheaply before paying for long contexts:
| Stage | Tokens | Sequence length |
|---|---|---|
| Warm-up | 200M | 512 |
| Main training | 1.5B | 1,024 |
| Longer context | 300M | 2,048 |
1,999,994,880 tokens in total, across 61,035 optimiser updates. Fine-tuning then runs at a learning rate of 1e-6 — four hundred times smaller than pre-training's peak of 4e-4 — because the goal is to adjust a working model, not rebuild it.
Sampling: the same model, different temperaments The model always produces the same probabilities for a given input. Sampling is what you do with them — and it is where nearly all of a chatbot's personality settings live.
A model does not output a word. It outputs a probability for every word it knows, and something else has to choose. Always taking the most likely token gives repetitive, flat text. Sampling in proportion to the probabilities gives variety, and occasionally nonsense. Temperature, top-k and top-p are the dials between those two failure modes.
Invented starting scores run through the real softmax, top-k and top-p rules, so how the bars respond to each dial is genuine even though the numbers going in are made up.
What WillMe GPT 3.3 actually uses at inference
Greedy decoding by default — plain argmax over the scores, no
randomness at all. When sampling is explicitly requested it uses temperature 0.8,
top_k 50 and top_p 0.95. A repetition penalty of 1.10 is applied to every token
already present before any of that, and the scores are converted to FP32 first.
Generation stops at <|end|> or <eos>, or when
prompt plus reply reaches 2,048 tokens.
Size is not the only axis
Parameter counts get the headlines, but a model's behaviour comes from three numbers pulling against each other: how many parameters it has, how many tokens it was trained on, and how much of the conversation it can see. WillMe's own lineup makes the trade-offs visible, because GPT 3.3 is smaller than its predecessor on purpose.
| Model | Parameters | Context | Architecture |
|---|---|---|---|
| GRU 1 | 15M | 256 | GRU |
| GRU 2 | ~40M | 512 | GRU |
| GPT 3.1 | 163M | 2,048 | RWKV-like |
| GPT 3.2 | 163M | 2,048 | RWKV-like |
| GPT 3.3 | 100.7M | 2,048 | Transformer |
Every figure here is the same one the compare page reports. GPT 3.3 trades 62M parameters and 9B training tokens for a cleaner dataset and a different architecture.
Why smaller can be better
A model's quality depends at least as much on what it read as on how big it is. GPT 3.2 trained on 11B tokens; GPT 3.3 trains on a revised and cleaned 2B. The fine-tuning set went the same way — source messages cleaned and reduced by 84%, leaving 2,600 conversations where 3.1 and 3.2 used sets of 53,000.
Whether that trade pays off is an open question and the reason GPT 3.3 exists. The compare page will show the answer once there are ratings to show.
Next: the machinery
Those are the ideas. The next three parts open the box: how text becomes numbers, how one next-token guess is actually produced, and what attention does inside a transformer block.