Part 9
Transformers
vs RWKV
Two answers to one question: how should a model remember what it already read? WillMe has shipped both, which makes this the rare comparison we can settle with our own models rather than someone else's benchmark.
The same problem, two shapes
Every model in this project has to answer the same question at every position: given everything so far, what comes next? The difference is in how much of “everything so far” it can reach.
A transformer looks at all of it, every time. Each new token compares itself against every earlier token and decides what to pay attention to. A recurrent model — RWKV, or a GRU before it — never looks back. It carries a fixed-size summary forward and updates it as it goes, so everything it knows about the past has to fit inside that summary.
Press play to run the same eight-token sentence through both. The transformer's connections grow with every step; the recurrent state never gets bigger.
The cost of looking at everything Quadratic means doubling the length quadruples the work. It is fine at 2,048 tokens and ruinous at 200,000, which is why long-context models need a different trick.
Attention's strength is also its bill. To let every token see every earlier one, a transformer computes a score for each pair — so the work grows with the square of the sequence length. A recurrent model does the same fixed amount of work per token no matter how long the text is, and its memory for the past never grows at all.
Move the slider and watch the two curves separate. Both are plotted from their real complexity, scaled so 2,048 tokens — WillMe's context window — reads as 1×.
Relative cost, both normalised to 1× at 2,048 tokens. The shapes are the real complexity classes — n² against n — not measurements of our hardware.
Where the n² actually comes from
Attention computes softmax((Q Kᵀ)/√64 + causal_mask) V. The
Q Kᵀ term is a T × T matrix — one score for
every pair of positions — so both the compute and the attention memory scale
with T². At 2,048 tokens that is 4.2 million pairs per head, per
block, and WillMe GPT 3.3 has 12 heads across 12 blocks.
RWKV replaces that matrix with a running state updated by a learned per-channel decay.
Each new token costs the same as the last, and nothing of size T²
is ever built. The catch is what that buys you, which is the next section.
What the fixed state costs you
A recurrent model's memory is a bottleneck in the literal sense: everything from the past has to squeeze through a fixed number of channels. If a detail from forty tokens ago was not judged worth keeping when it went past, it is gone — and the model cannot go back and look.
A transformer keeps every token available and decides what matters at the moment it needs it, which is exactly the case where recurrence struggles: a question whose answer depends on something specific and unremarkable said much earlier.
Illustrative, but the failure mode is real and it is the one GPT 3.3 was built to test. Slide the distance and watch what each architecture can still reach.
Side by side
| Transformer | RWKV | |
|---|---|---|
| Training cost | Grows with T² | Grows with T |
| Training speed | All positions in parallel | Parallel form exists, but it is harder to get right |
| Generation memory | KV cache grows with every token | One fixed-size state, forever |
| Reaching back | Exact, any position in the window | Whatever survived in the state |
| Context limit | Set by compute and cache | No hard limit, but detail fades |
| In WillMe | GPT 3.3 | GPT 3.1, GPT 3.2 |
Nothing here says one architecture is better. They are better at different things, and which matters depends on what you are building.
Where the GRUs fit
WillMe GRU 1 and GRU 2 are recurrent too, and much older in spirit. A GRU carries a hidden state through two learned gates — one deciding what to forget, one deciding what to let in — with a tanh squashing the update. RWKV is a descendant of that idea, rebuilt so the whole sequence can be trained in parallel rather than strictly one step at a time.
The three-way progression is visible in the context windows alone: 256 tokens for GRU 1, 512 for GRU 2, 2,048 for everything in Gen 3.
Reading the two blocks next to each other
The RWKV block from v3_model.py, and the GRU that serves WillMe GRU 2:
# RWKV: a learned per-channel decay on a running state self.time_decay = nn.Parameter(torch.zeros(c)) self.time_first = nn.Parameter(torch.zeros(c)) self.key = nn.Linear(c, c, bias=False) self.value = nn.Linear(c, c, bias=False) self.receptance = nn.Linear(c, c, bias=False) # GRU: the same idea, with the gates supplied by PyTorch self.gru = nn.GRU(hidden_size, hidden_size, batch_first=True)
Both keep one vector and update it per token. The difference is that RWKV's decay is learned per channel and its update is arranged so training can be parallelised over the sequence — a GRU's cannot, which is a large part of why it does not scale.
So which won?
We do not know yet, and that is the honest answer. GPT 3.3 exists precisely because GPT 3.1 and 3.2 underperformed where attention would have been expected to help, so it swaps the architecture while shrinking the model from 163M parameters to 100.7M and the pre-training set from 11B tokens to a cleaned 2B.
Two variables moved at once, which makes it an imperfect experiment — if 3.3 comes out ahead, some of the credit belongs to the cleaner data rather than to attention. The ratings on the compare page will be the first real evidence either way, and there are none for GPT 3.3 yet because it is still in training.
What comes after is already sketched: if the transformer validates, Gen 4 becomes a full transformer generation, and Gen 5 is planned to explore a hybrid with linear-attention-style mixing — which is to say, both of the ideas on this page at once.