Neural networks

A neuron is six lines of arithmetic, a layer is a matrix multiply, and training is a loop of four boxes. This level builds each one with numbers small enough to check by hand, then verifies the hand-written gradients against a numerical check.

Picture learning to throw darts blindfolded, with a friend calling out "a bit high, a bit left". You throw, hear how far off you were, and adjust your arm a little. After enough throws your arm "knows" where the bullseye is. A neural network learns the same way. Its "arm" is millions of numbers called weights, and the friend is a formula (the loss) that says how far off each guess was and which way to adjust.

Two words carry this whole lesson. The derivative (or slope) of the loss with respect to a weight answers "if I nudge this weight up a tiny bit, does the loss go up or down, and how fast?" The gradient is simply the list of those slopes, one per weight. Stepping against the gradient is walking downhill. (New to the notation? primer.notation builds every symbol used here from zero.)

Worked example with a single knob: a model with one weight w predicts y = w·x and should learn y = 3x from the example x = 1, y = 3. The loss is (w·1 − 3)². Its slope with respect to w is 2(w − 3). Start at w = 0 and step against the slope with step size 0.25:

Each step halves the distance to 3. That's all training is.

flowchart LR
  D[Batch of examples] --> F[Forward pass<br/>make predictions]
  F --> L[Loss<br/>how wrong were we?]
  L --> B[Backpropagation<br/>gradient per weight]
  B --> O[Optimizer step<br/>adjust weights]
  O -->|next batch| D

The update rule, for every weight at once, with learning rate η:

w, eta, target = 0.0, 0.25, 3.0
for step in range(3):
    # ∂L/∂w for the loss (w - 3)²
    slope = 2 * (w - target)
    # w ← w - η ∂L/∂w
    w = w - eta * slope
    print(w)  # → 1.5 2.25 2.625

learn_one_weight runs the table above; train runs the loop for a real network.

Why it matters this loop is identical for a spam filter and for a frontier language model. Only the size of the network, the data, and the loss change. When a training run misbehaves, the fault is in one of these four boxes.

Think of a judge at a cooking contest. They score taste, presentation and originality, but care about taste most, so they weight it more. They add a baseline ("everyone starts at half a point"), then apply a rule to turn the total into a verdict ("negative totals count as zero"). That is a neuron: weights are how much each input matters, the bias is the baseline, and the activation function is the rule.

Worked example: inputs (2, 3), weights (0.5, 1.0), bias 0.5. Weighted sum: 2 × 0.5 + 3 × 1.0 + 0.5 = 4.5. The ReLU rule keeps positive values, so the output is 4.5.

flowchart LR
  X1((x1 = 2)) -- "× w1 = 0.5" --> S["Sum + bias<br/>1.0 + 3.0 + 0.5 = 4.5"]
  X2((x2 = 3)) -- "× w2 = 1.0" --> S
  B((bias 0.5)) --> S
  S --> A["Activation φ<br/>ReLU(4.5) = 4.5"]
  A --> Y((y = 4.5))

x = [2, 3]
w = [0.5, 1.0]
b = 0.5
# ReLU: keep positives, zero the rest
def phi(z): return max(0.0, z)
# w · x = Σ_i w_i x_i
sum(w_i * x_i for w_i, x_i in zip(w, x))  # → 4.0
# y = φ(w · x + b)
phi(sum(w_i * x_i for w_i, x_i in zip(w, x)) + b)  # → 4.5

Why it matters every dense layer in every model is this, including the feed-forward half of each transformer block, where most of a language model's parameters live. MLP.forward is two of these layers in six lines.

Picture an assembly line that produced a faulty product. The inspector at the end measures how bad it is and passes the complaint back. Each station works out how much of the fault was its own doing, based on what it received and what it did to it, and passes the rest further back. Every station ends up knowing exactly how to adjust its own machine. Backpropagation is that blame-passing, done with derivatives.

Worked example, continuing the neuron: the target was 5, so the loss is (5 − 4.5)² = 0.25.

One step with learning rate 0.01 moves weights to (0.52, 1.03) and bias to 0.51. The new output is 4.64 and the loss drops from 0.25 to 0.1296. The weight whose input was bigger (3) received the bigger share of blame and moved more.

flowchart RL
  L["Loss (5 − y)² = 0.25"] -- "dL/dy = −1" --> Y[y = ReLU z]
  Y -- "dy/dz = 1" --> Z[z = w·x + b]
  Z -- "× x1 = 2 → dL/dw1 = −2" --> W1[w1]
  Z -- "× x2 = 3 → dL/dw2 = −3" --> W2[w2]
  Z -- "× 1 → dL/db = −1" --> Bb[b]

This is the chain rule: when a change passes through several steps, the overall rate of change is the product of each step's rate of change. If turning a knob moves a gear 3× as fast, and that gear moves a needle 2× as fast, the knob moves the needle 6× as fast.

Symbols

In words: "the blame on a weight is how much the loss cares about the output, times how much the output cares about the sum, times how much the sum cares about that weight."

With the numbers: for : ; for : .

In Python:

x, t, z = [2, 3], 5, 4.5
# ReLU(4.5)
y = max(0.0, z)
# ∂L/∂y
dL_dy = 2 * (y - t)
# φ'(z): ReLU's slope
dy_dz = 1 if z > 0 else 0
# × ∂z/∂w_i = x_i, for w_1 and w_2
[dL_dy * dy_dz * x_i for x_i in x]  # → [-2.0, -3.0]
# one step, learning rate 0.01
w = [0.5 - 0.01 * -2.0, 1.0 - 0.01 * -3.0]
b = 0.5 - 0.01 * -1.0
y_new = w[0] * x[0] + w[1] * x[1] + b
# the new output and the smaller loss
round(y_new, 2), round((t - y_new) ** 2, 4)  # → (4.64, 0.1296)

one_neuron_worked_example computes every row of the table.

Why it matters PyTorch's loss.backward() does exactly this, for billions of weights, by recording the forward computation and replaying it backwards. Knowing it is just the chain rule is what lets you reason about vanishing gradients, memory use, and why some architectures train and others don't.

Stack three sheets of tinted glass and you get one darker sheet of glass: no arrangement of flat panes can bend light around a corner. Linear layers are flat panes. However many you stack, the result is one linear layer, which can only separate classes with a straight line. The activation function is the bend.

Worked example in one dimension: a layer that multiplies by 3, followed by a layer that multiplies by 2, is the same as one layer that multiplies by 6. With a ReLU between them, inputs −1 and 1 give 0 and 6. No single multiplication does that.

flowchart LR
  subgraph NoAct["Without activation"]
    a1[X] --> a2[·W1] --> a3[·W2] --> a4[·W3] --> a5["= X·(W1W2W3)<br/>one matrix"]
  end
  subgraph WithAct["With activation"]
    b1[X] --> b2[·W1] --> b3[φ] --> b4[·W2] --> b5[φ] --> b6[·W3] --> b7[curved<br/>decision boundary]
  end

W_1, W_2 = 3, 2
# ReLU
def phi(z): return max(0, z)
# (X W_1) W_2 ...
[(x * W_1) * W_2 for x in [-1, 1]]  # → [-6, 6]
# ... equals X (W_1 W_2): one layer of 6
[x * (W_1 * W_2) for x in [-1, 1]]  # → [-6, 6]
# φ(X W_1) W_2: no single W' gives 0 and 6
[phi(x * W_1) * W_2 for x in [-1, 1]]  # → [0, 6]

In code: make_moons builds the two interleaving half-moons (and make_xor the four-point XOR puzzle); train_logistic_regression fits the straight-line baseline in the left panel.

Why it matters "depth" is only worth anything because of the activations between layers. linear_stack_collapses shows five stacked linear layers matching their single-matrix product to 1e-16.

Think of different kinds of switch. ReLU is a one-way valve: water flows freely forward, not at all backward. Sigmoid is a dimmer that squeezes any input into 0–1 but barely moves at the extremes. Tanh is the same dimmer centred on zero. GELU is a valve that leaks a little when nearly closed.

Worked example, values you can check on a calculator (the ′ mark, as in sigmoid′, means "the slope of"):

Reading the formulas: e is Euler's number, about 2.718, and e^z means "2.718 raised to the power z", which is always positive and grows fast. Φ(z) is the fraction of a bell curve (the standard normal distribution) that lies below z: Φ(0) = 0.5, Φ(1) = 0.841. Softmax turns a list of scores into shares that are all positive and add up to 1: raise e to each score, then divide each by the total (Σ means "add them all up"). See primer.notation for each symbol, and primer.ml.attention for softmax worked through by hand.

In code: relu, sigmoid, tanh and gelu compute each rule, and relu_grad, sigmoid_grad, tanh_grad and gelu_grad compute its slope; gelu_tanh is the cheaper approximation GPT-2 uses, and softmax subtracts the largest score first so nothing overflows.

Why it matters activation choice decides whether gradients survive a deep stack. Sigmoid everywhere is why deep networks were hard to train before 2010; ReLU and residual connections are why they aren't now (see primer.ml.deep_nets).

Now the assembly line has two stations. The blame from the end first reaches the output weights, then travels back through the hidden layer to the first weights. On the way it passes through the hidden activation, which can shrink it, just as a station that barely changed the product can take only a little of the blame.

Worked example, one input, one hidden unit, one output: x = 1, w1 = 0.5, tanh, w2 = 2, sigmoid, target 1.

flowchart LR
  subgraph Forward
    X[X] --> Z1["z1 = X·W1 + b1"] --> H["h = tanh z1"] --> Z2["z2 = h·W2 + b2"] --> P["ŷ = σ z2"] --> L["L = BCE ŷ, y"]
  end
  L -. "ŷ − y" .-> dZ2[dL/dz2]
  dZ2 -. "hᵀ · dz2" .-> dW2[dL/dW2]
  dZ2 -. "dz2 · W2ᵀ" .-> dH[dL/dh]
  dH -. "⊙ 1 − h²" .-> dZ1[dL/dz1]
  dZ1 -. "Xᵀ · dz1" .-> dW1[dL/dW1]
  H -. cached .-> dW2
  H -. cached .-> dZ1
  X -. cached .-> dW1

For a batch, with binary cross-entropy (the loss for yes/no predictions, ; see primer.ml.losses), which fuses with sigmoid into the simple error ŷ − y:

import math
X, W_1, W_2, y = 1, 0.5, 2, 1
z_1 = X * W_1
h = math.tanh(z_1)
z_2 = h * W_2
# σ(z_2)
y_hat = 1 / (1 + math.exp(-z_2))
# ŷ - y
dL_dz2 = y_hat - y
# hᵀ ∂L/∂z_2
dL_dW2 = h * dL_dz2
# ∂L/∂z_2 W_2ᵀ
dL_dh = dL_dz2 * W_2
# ⊙ (1 - h²), tanh's slope
dL_dz1 = dL_dh * (1 - h ** 2)
# Xᵀ ∂L/∂z_1
dL_dW1 = X * dL_dz1
[round(v, 4) for v in (dL_dz2, dL_dW2, dL_dh, dL_dz1, dL_dW1)]  # → [-0.2841, -0.1313, -0.5682, -0.4469, -0.4469]

The hand-written gradients are checked two ways: a numerical gradient check (nudge each weight by ±ε and measure the loss change; see gradient_check) and, in the tests, PyTorch autograd.

In code: tiny_two_layer_example computes every row of the worked table; MLP holds both layers' weights and biases, MLP.loss scores a batch with binary_cross_entropy, and numerical_gradient is the slow nudge-every-weight answer that gradient_check compares against.

Why it matters the factor (1 − h²) is where gradients shrink. Chain fifty of them and the first layers hear almost nothing: the vanishing gradient problem. The cached activations are why training a model needs several times the memory of serving it.

Imagine studying a deck of 400 flashcards. You work through a pile of 32, then pause to update your notes; that pause is a step. The pile is a batch. Going through the whole deck once is an epoch; then you shuffle and start again.

Worked example: 400 examples in batches of 32 is 12 full batches plus one of 16, so ⌈400/32⌉ = 13 steps per epoch. Two epochs: 26 steps.

flowchart LR
  E[Epoch: one pass over all 400 examples] --> SH[Shuffle]
  SH --> B1[Batch 1<br/>32 examples] --> S1[Step 1<br/>update weights]
  S1 --> B2[Batch 2] --> S2[Step 2]
  S2 --> D[...] --> B13[Batch 13<br/>last 16 examples] --> S13[Step 13]
  S13 -->|next epoch| E

import math
N, B, epochs = 400, 32, 2
N / B  # → 12.5
# ⌈N / B⌉: the last, smaller batch still counts
math.ceil(N / B)  # → 13
epochs * math.ceil(N / B)  # → 26

In code: train reshuffles the data every epoch, takes one plain gradient step per batch and records the loss after each epoch; MLP.accuracy reports the fraction of examples classified correctly.

Why it matters larger batches give smoother gradient estimates and keep GPUs busy, but need more memory. Pretraining a language model runs roughly one epoch over trillions of tokens (it rarely sees the same text twice); fine-tuning runs several epochs over a small dataset, which is why fine-tunes can overfit.

Answer each one out loud or on paper before you open it. If you can explain it, you know it.