Part 2

How a neural
network learns

Before a language model, a neuron. This part builds one from nothing, stacks a few into a network, and then lets you watch the network teach itself — with every number on this page computed in your browser as you move the sliders.

A neuron is a weighted sum

Strip away the name and a neuron does something a spreadsheet could do. It takes some numbers in, multiplies each by a weight, adds them up, adds one more number called the bias, and passes the result through a simple function. That is the whole unit. A large model is a hundred million of them, and nothing more exotic than that.

The weight decides how much an input matters, and its sign decides which way. The bias shifts the whole thing up or down — it is how a neuron says “unless I hear otherwise, lean this way”. Drag the two sliders below and watch what each one does to the line.

One neuron, two knobs
Weight (w) 1.00
Bias (b) 0.00
output = ReLU(w × input + b)
input 1.0 → 1.00
input 2.0 → 2.00
input out

The line is drawn from the numbers on the left, recomputed on every drag. Weight tilts it; bias slides it. That is the entire behaviour of a single unit.

Why the squashing function matters Without a non-linear step between layers, stacking layers is pointless: two matrix multiplications in a row are just one matrix multiplication.

That last step — the function the sum passes through — is called the activation. It looks like a detail and it is the reason deep networks work at all. Without it, a hundred stacked layers collapse into the arithmetic of a single one, and the whole network can only ever draw a straight line.

Three activations

Each curve is plotted from its real formula. ReLU is what WillMe's feed-forward layers are built on; the gate inside SwiGLU is a smoothed version of it.

Neurons in layers

One neuron draws one line. Put several side by side and you get a layer; stack layers and each one works on what the last one produced. Early layers pick up crude patterns, later layers combine those into something more useful. Nobody decides what each neuron specialises in — that falls out of training.

Press the button below to push a value through a small network. Every edge you see is one weight, and every glow is one weighted sum being computed.

A forward pass
Illustrative
input hidden output
Idle

Three inputs, a hidden layer of five, two outputs. Twenty-five weights in total. WillMe GPT 3.3 has 100,682,496 of them and works exactly this way.

Learning is rolling downhill The loss is one number saying how wrong the model was. Training is the search for weights that make it small.

A fresh network is random, so its first guesses are nonsense. Training needs two things: a way to score how wrong it was — the loss — and a way to work out which direction each weight should move to make that score smaller. That direction is the gradient, and following it downhill is gradient descent.

The step size is called the learning rate, and it is the setting people get wrong most often. Too small and training crawls. Too large and the steps overshoot the bottom and climb the far wall instead. Try it: the ball below really does follow the slope of the curve.

Gradient descent, one step at a time
weight value
Learning rate 0.30

The curve is a real function and each step moves by learning rate times the slope at that point. Push the rate past about 0.9 and watch it bounce out of the valley entirely.

The three lines of maths behind that ball

The curve is L(w) = 0.35 w² + 0.6 sin(1.6 w) + 1.2, chosen because it has a shallow local dip on the left and a deeper minimum near the middle — a miniature version of the landscape a real loss surface has in a hundred million dimensions.

Its slope is the derivative L′(w) = 0.7 w + 0.96 cos(1.6 w), and one training step is w ← w − η L′(w) where η is the learning rate. That single line is what runs 61,035 times when WillMe GPT 3.3 is trained — over a hundred million weights at once instead of one, with the gradient supplied by backpropagation rather than by hand.

A network that actually trains

Everything so far in one place. Below are some scattered points and a straight line the model has to fit. The line starts random and knows nothing. Each step measures the error, computes the gradient for both the slope and the intercept, and nudges them — the same loop, in miniature, that runs on the real thing.

Fitting a line from scratch

Real gradient descent on real (randomly generated) points: mean squared error, gradients computed exactly, no scripted outcome. Reset and the points change.

What is different in a real model

Three things, and none of them change the idea. The loss is cross-entropy over 32,768 possible next tokens rather than squared error on one number. The gradients come from backpropagation, which applies the chain rule backwards through every layer instead of being written out by hand. And the optimiser is AdamW, which keeps a running memory of recent gradients so each weight gets its own effective step size.

The loop is still: guess, measure how wrong, nudge every weight a little in the direction that would have been less wrong. Repeat 61,035 times.

So where does language come in?

Nowhere yet — and that is the point. Nothing on this page knows what a word is. A language model is this machinery pointed at one specific job: given the text so far, score every token that could come next. The next part shows what that looks like in code, and the part after it turns the machinery into something that reads.