Learn
How does it
actually work?
Eleven parts, starting from a single neuron and ending inside our own WillMe GPT 3.3. Every diagram is interactive and most of the numbers are computed in your browser as you read. No maths needed — but the real figures are here if you want them.
The whole idea, in one sentence
A language model reads the text so far and guesses what comes next. That is the entire trick. Everything else — the size, the training, the architecture — exists to make that one guess better.
It does not look anything up. There is no database behind it, no search engine, no encyclopaedia it consults. When WillMe answers a question, it is producing the words that its training makes most likely to follow your question. Sometimes that is exactly right. Sometimes it is confidently wrong, which is a habit worth understanding rather than being surprised by.
One guess at a time
Press the button to watch a sentence get built the way a model builds it.
Press start…
Illustrative probabilities, not live model output — the shape of the process is real, the exact numbers are made up for readability.
The model we will use
WillMe GPT 3.3 is small as language models go. That is what makes it a good teaching example: everything about it is a number you can actually hold in your head, and its mistakes are easy to see.
100.7M
parameters
12
layers
32,768
known tokens
2,048
token memory
For comparison, the models behind well-known commercial assistants are roughly a thousand times larger. WillMe is a model you could read the complete specification of in an afternoon — which is exactly what the next five pages are for.
The exact specification
WillMe GPT 3.3 is a decoder-only, autoregressive Transformer with exactly 100,682,496 trainable parameters.
| Vocabulary | 32,768 |
|---|---|
| Transformer blocks | 12 |
Hidden width (d_model) | 768 |
| Feed-forward width | 2,048 |
| Query heads | 12 |
| Key/value heads | 4 |
| Head dimension | 64 |
| Maximum context | 2,048 |
Attention is grouped-query and causal, positions come from rotary
embeddings (RoPE, theta 10,000), normalisation is pre-norm RMSNorm with
epsilon 1e-5, and the feed-forward network is SwiGLU with a
SiLU gate. There are no bias terms and no dropout, and the input
embedding is reused as the output head.