Part 8 · Intermediate
Inside WillMe
GPT 3.3
The whole model, part by part, with every one of its 100,682,496 parameters accounted for. This part assumes the previous ones: tokens, next-token prediction, attention.
The specification, on one line
WillMe GPT 3.3 is a decoder-only autoregressive transformer: 12 blocks, a width of 768, grouped-query attention with 12 query heads and 4 key/value heads, SwiGLU feed-forward layers with a hidden size of 2,048, RMSNorm, rotary position embeddings, a 32,768-token vocabulary and a 2,048-token context window.
Each of those choices costs parameters, and the budget below is the whole model. Tap a slice to see what it buys.
Every number here is exact and adds up: the model is the embedding table plus twelve identical blocks plus a final norm. The output head is free, because it reuses the embedding.
The arithmetic, term by term
| Part | Working | Parameters |
|---|---|---|
| Token embedding | 32,768 × 768 | 25,165,824 |
| Attention, per block | 589,824 + 196,608 + 196,608 + 589,824 | 1,572,864 |
| SwiGLU, per block | 3 × 1,572,864 | 4,718,592 |
| Two RMSNorms, per block | 2 × 768 | 1,536 |
| One block | 1,572,864 + 4,718,592 + 1,536 | 6,292,992 |
| Twelve blocks | 12 × 6,292,992 | 75,515,904 |
| Output head | tied to the embedding | 0 |
25,165,824 + 75,515,904 = 100,681,728, and the remaining 768 are the final RMSNorm's per-channel scales, which brings it to exactly 100,682,496. RoPE tables, causal masks and KV caches hold no trainable parameters at all.
One block, five stages The residual stream is the running total every block adds into. Nothing overwrites it — each block contributes a correction, which is why deep stacks train at all.
All twelve blocks are identical in shape. A vector of 768 numbers per token comes in, and the block adds two contributions to it: one from attention, which mixes information between positions, and one from the feed-forward network, which works on each position alone. Step through it below.
The shapes are real. Note that the residual stream never changes width: 768 in, 768 out, every block, all twelve times.
Grouped-query attention, and why the cache is the point
Queries are projected to 12 heads of 64 dimensions, but keys and values only to 4 of them — each key/value head is shared by three query heads. That keeps 12 distinct query patterns while cutting the K and V projections to a third (196,608 parameters each, against 589,824 for Q and O).
The saving that matters is not the parameters, it is the cache. While generating, the keys and values for every earlier token have to be kept in memory. With 4 KV heads instead of 12 that cache is a third of the size, which is what makes a 2,048-token context affordable on the hardware this project serves from.
From Discord to a checkpoint
The architecture is only half of it. The other half is the pipeline that turns years of group chat into something a model can train on — and for GPT 3.3 that pipeline throws away far more than it keeps.
Each stage is a real step in this repository. Select one to see what it does and what comes out the other side.
What happens when you send a message
Generation is not one pass through the model. It is one pass per token, each one reading everything written so far, and the reply grows a token at a time until a stop token appears or the window fills.
Reply so far
—
The path is real; the words are a short illustration. Watch the cache grow — that is the cost that makes long conversations slower than short ones.
Why the second token is faster than the first
The first forward pass has to compute keys and values for every token in your prompt. Every pass after that only computes them for the single new token and reads the rest out of the KV cache. That is why a long prompt costs a noticeable pause up front and then streams smoothly, and why the cache size — the thing grouped-query attention shrinks — sets the practical limit on context length.
Honest limitations of this specific model
GPT 3.3 is small. A hundred million parameters is roughly a thousandth of what the well-known commercial assistants carry, and it was trained on two billion tokens where they use trillions. It will invent facts, lose the thread of an argument, and fail at arithmetic. It was also fine-tuned on 2,600 conversations from one Swedish teenager and his friends, so its idea of a normal reply is calibrated to that and nothing else.
None of that is a bug to be fixed. It is the shape of the experiment: how much of one person's voice can a small, cleanly trained model actually carry? Part 11 is about the limits in more detail.