Part 7

Attention and the
transformer block

To guess the next word well, a model has to work out which earlier words matter. That mechanism is called attention, and it is most of what a transformer is.

Which words matter?

Take “William made coffee because he was tired”. To handle “he” sensibly, something has to connect it back to “William”. Attention is how: at every position, the model looks back over everything before it and decides how much weight to give each earlier token.

Click a word below to see which earlier words it would lean on.

Illustrative weights chosen to show the mechanism. A real head's pattern is rarely this tidy, and each of the 12 heads attends to something different.

Only backwards

Notice that every word can only look left. This is the causal part, and it is enforced, not learned: while predicting position 5 the model is blocked from seeing positions 6 onwards.

Without that rule, training would be pointless — predicting the next word is trivial if you are allowed to peek at it. The same restriction is why the model cannot revise: once a token is out, it is part of the context, and the model carries on from there rather than going back to reconsider.

One of the twelve layers

input RMSNorm steady scale Attention look back 12 heads + the original is kept and added back SwiGLU think about what it found + the output of this block becomes the input of the next, twelve times over

Look, then think

Each of the twelve layers does the same two things. First attention: gather information from earlier words. Then a small feed-forward network: do something with what was gathered. Both are wrapped in a habit worth noticing — the input is kept and the result is added to it, rather than replacing it.

That addition is what lets twelve layers stack without the signal dissolving. Each layer nudges the representation rather than overwriting it, so early layers can settle basic questions like which word this is, and later ones can work on what the sentence is doing.

Grouped-query attention, RoPE and SwiGLU

Each block is pre-norm with two residual sublayers: Y = X + Attention(RMSNorm(X)) then X' = Y + SwiGLU(RMSNorm(Y)). No dropout, no biases anywhere.

Grouped-query attention. Queries are projected to 12 heads of 64 dimensions, but keys and values only to 4 — each KV head is shared by three query heads. That keeps 12 distinct query patterns while cutting the KV projection and, more importantly, the generation cache to a third of what 12 KV heads would need.

Q projection768 × 768589,824
K projection768 × 256196,608
V projection768 × 256196,608
Output projection768 × 768589,824

RoPE. There is no learned position vector added to the residual stream. Instead, query and key pairs are rotated by an angle proportional to their position, using inv_freq_i = 1 / 10000^(2i/64). Because a dot product between two rotated vectors depends on the difference of their angles, attention sees relative distance for free.

SwiGLU. The feed-forward network expands 768 to 2,048 twice over, uses one copy as a gate — Z = SiLU(G) ⊙ U — and projects back down. The gate lets the network decide which expanded features reach the residual stream. Three matrices of 1,572,864 parameters each, 4,718,592 in total, which makes the MLP three times the size of the attention sublayer.

Softmax attention itself is softmax((Q Kᵀ)/√64 + causal_mask) V, with the mask set so position t cannot see anything after it.

Where the 100,682,496 parameters live
Embedding, shared with the output head25,165,824
Attention, per block1,572,864
SwiGLU, per block4,718,592
Two RMSNorm vectors, per block1,536
One complete block6,292,992
Twelve blocks75,515,904
Final RMSNorm768
Total100,682,496

RoPE tables, causal masks and KV caches hold no trainable parameters — they are computed at runtime.