Before transformers, two designs dominated deep learning, and both still matter. Convolutional networks (CNNs) see images by sliding small pattern detectors across them. Recurrent networks (RNNs) read sequences one item at a time, carrying a running memory. Each builds in an assumption about its data (patterns in images are local; sequences unfold in order), and each has a limit that the transformer removed. Knowing both stories explains why modern models look the way they do.

Part A: convolutional networks

Everyday picture You are in a dark room with a large photograph and a small flashlight. You're looking for one thing, say a place where dark turns to bright from left to right. You sweep the flashlight across the photo, one step at a time, and at every spot you jot down a score: high if the lit patch matches what you're looking for, near zero if it doesn't. When you finish, your notes form a new, smaller picture: a map of where the pattern appears. That map is called a feature map, and the pattern you were looking for, written as a small grid of numbers, is the filter (or kernel).

Tiny worked example A 5×5 image: two dark columns (0) then three bright columns (1), so there is a vertical edge between columns 2 and 3. The filter is a 3×3 "vertical edge" detector: −1 on the left column, 0 in the middle, +1 on the right. It rewards "bright on the right, dark on the left".

image X                 filter K
0 0 1 1 1               -1 0 1
0 0 1 1 1               -1 0 1
0 0 1 1 1               -1 0 1
0 0 1 1 1
0 0 1 1 1

Put the filter on the top-left 3×3 patch of the image. Multiply each image number by the filter number on top of it, and add all nine products:

patch      × filter    = products
0 0 1      -1 0 1        0 0 1
0 0 1      -1 0 1        0 0 1      sum = 1 + 1 + 1 = 3
0 0 1      -1 0 1        0 0 1

So the top-left cell of the feature map is 3. Slide one step right: the patch is 0 1 1 in every row; products 0 0 1 per row; sum again 3 (the edge is still under the flashlight). One more step: the patch is 1 1 1, products −1 0 1, sum 0 (flat bright area, no edge). Doing the same for every row gives the full 3×3 feature map:

3 3 0
3 3 0
3 3 0

The high numbers sit exactly where the edge is. Run a horizontal edge filter over the same image and every cell is 0: this image has no top-to-bottom change. Each filter answers one question.

flowchart LR
  W[Take the next k×k patch<br/>of the image] --> M[Multiply each pixel by the<br/>filter weight on top of it]
  M --> S[Add all the products<br/>into one number]
  S --> C[Write it into the<br/>feature map]
  C --> N{More positions?}
  N -->|slide by the stride| W
  N -->|no| F[Feature map complete]

# the 5×5 image: one channel, so Σ_c has one term
X = [[0, 0, 1, 1, 1] for _ in range(5)]
# the vertical-edge filter
K = [[-1, 0, 1] for _ in range(3)]
n, k, s, p = 5, 3, 1, 0
# ⌊(n + 2p - k) / s⌋ + 1
out = (n + 2 * p - k) // s + 1
out  # → 3
# Σ_u Σ_v K[u, v] X[i s + u, j s + v]
Y = [[sum(K[u][v] * X[i * s + u][j * s + v]
          for u in range(k) for v in range(k))
      for j in range(out)]
     for i in range(out)]
Y  # → [[3, 3, 0], [3, 3, 0], [3, 3, 0]]
# 224 pixels, 7×7 filter, stride 2, padding 3
(224 + 2 * 3 - 7) // 2 + 1  # → 112

In code: conv2d is the loop in the diagram, one multiply-and-add per position, and conv_output_size is the output-size formula.

Everyday picture A manager asks each of four teams for one number: the strongest signal anyone on the team saw. The report is four times shorter and still says where something important happened.

Tiny worked example 2×2 max pooling on a 4×4 map keeps the largest value in each quarter:

1 3 | 0 0
2 4 | 0 1        ->   4 1
----+----             1 6
0 0 | 5 2
1 0 | 1 6

In code: max_pool2d keeps the largest value in each non-overlapping block.

Everyday picture A rubber stamp: you carve the pattern once and use it everywhere on the page. A CNN uses the same filter at every position, so a cat detector works in any corner of the photo and costs the same number of weights however large the photo is.

Tiny worked example 64 filters of size 3×3 over a colour image need 3·3·3·64 + 64 = 1,792 parameters, for any image size. A fully connected layer mapping a 224×224×3 image to an output of the same size as those 64 feature maps would need 150,528 × 3,211,264 ≈ 483 billion weights.

As layers stack, each neuron sees more of the original image: its receptive field. One 3×3 layer sees 3×3 pixels; two see 5×5; three see 7×7. Pooling between layers speeds this up: conv 3×3, pool 2×2, conv 3×3 already sees 8×8. That's why two stacked 3×3 filters (18 weights, and two nonlinearities) replaced single 5×5 filters (25 weights) in VGG.

# (k_ℓ, s_ℓ): conv 3, pool 2 with stride 2, conv 3
layers = [(3, 1), (2, 2), (3, 1)]
# r_0 = j_0 = 1
r, j = 1, 1
for k_l, s_l in layers:
    # r_ℓ = r_{ℓ-1} + (k_ℓ - 1) j_{ℓ-1}
    r = r + (k_l - 1) * j
    # j_ℓ = j_{ℓ-1} s_ℓ
    j = j * s_l
    print(r, j)  # → 3 1 4 2 8 2

In code: conv_params and dense_params count the two layers' parameters, and receptive_field applies the recurrence to a stack of (kernel, stride) layers.

Everyday picture Reading starts with strokes, then letters, then words, then sentences. A CNN's first layer learns strokes (edges and colour blobs), the next combines them into textures and corners, deeper layers into parts (eyes, wheels), and the last into whole objects.

flowchart LR
  I[Image pixels] --> C1[Conv layer<br/>edges]
  C1 --> P1[Pool]
  P1 --> C2[Conv layer<br/>textures, parts]
  C2 --> P2[Pool]
  P2 --> C3[Conv layer<br/>whole objects]
  C3 --> FC[Dense layer]
  FC --> O[Prediction<br/>cat 94%]

In code: the figure runs each first-layer filter over the L with conv2d, and the corner detector multiplies two of those feature maps entry by entry.

• AlexNet (2012) won the ImageNet competition by a wide margin by training a deep CNN on GPUs, which started the deep-learning boom.
• VGG (2014) showed that deep stacks of small 3×3 filters work well.
• ResNet (2015) added residual (skip) connections, letting gradients bypass layers, and made networks of 100+ layers trainable. The same idea sits inside every transformer (see primer.ml.deep_nets).
• Vision Transformers (2020) cut an image into patches and treat each patch as a token, then apply ordinary attention.

Everyday picture for patches. Cut a photo into a grid of jigsaw pieces, lay them out in a row, and read them like the words of a sentence.

Tiny worked example A 224×224 colour image cut into 16×16 patches gives (224/16)² = 196 patches, each flattened to 16·16·3 = 768 numbers: 196 tokens of 768 values.

flowchart LR
  IMG[224×224×3 image] --> CUT[Cut into 16×16 patches<br/>14 × 14 = 196 pieces]
  CUT --> FLAT[Flatten each patch<br/>768 numbers]
  FLAT --> PROJ[Linear projection<br/>to model width]
  PROJ --> POS[Add position<br/>embeddings]
  POS --> TF[Transformer blocks<br/>attention across patches]

In code: patchify does the cut-and-flatten step, turning an image into one row of numbers per patch.

Why it matters in practice. CNNs remain efficient and strong for small vision tasks, on-device models and limited data, because their built-in assumptions (locality, weight sharing) mean they learn from less. At large scale, Vision Transformers dominate.

Part B: recurrent networks

Everyday picture You read a book one word at a time, and you are allowed to keep only a single page of notes. After every word you rewrite the page: blend what the page said with the new word. At the end, the page is all you have. That page is the hidden state.

Tiny worked example The sentence "not very good", with each word turned into one number: not = −1, very = 0.5, good = 1. The summary is a single number h, starting at 0. The rule: new h = tanh(0.5 × old h + 1 × word). (tanh squashes any number into the range −1 to 1: tanh(0) = 0, tanh(1) = 0.76, tanh(−1) = −0.76.)

The final summary is strongly positive: the "not" at the start has been almost washed out, because the page was rewritten twice since. This is the central weakness of RNNs, in three lines of arithmetic.

flowchart LR
  H0[h0 = 0] --> C1[RNN cell]
  X1[x1: not] --> C1
  C1 --> H1[h1 = −0.762]
  H1 --> C2[RNN cell<br/>same weights]
  X2[x2: very] --> C2
  C2 --> H2[h2 = 0.119]
  H2 --> C3[RNN cell<br/>same weights]
  X3[x3: good] --> C3
  C3 --> H3[h3 = 0.785<br/>final summary]

import math
W_h, W_x, b = 0.5, 1, 0
# not, very, good
x = [-1, 0.5, 1]
# h_0: a blank page
h = 0
for x_t in x:
    # h_t = tanh(W_h h_{t-1} + W_x x_t + b)
    h = math.tanh(W_h * h + W_x * x_t + b)
    print(round(h, 3))  # → -0.762 0.119 0.785

In code: RNNCell holds W_h, W_x and b; RNNCell.step is the formula once, RNNCell.run applies it along a sequence, and RNNCell.scalar builds the one-number cell of the worked example.

Everyday picture A photocopy of a photocopy of a photocopy. Each copy loses a little; after twenty copies the original is unreadable. Training an RNN sends a correction signal backwards through every step, and each step multiplies it by a factor. Factors below 1 fade the signal to nothing (vanishing gradient); factors above 1 blow it up (exploding gradient).

Tiny worked example With a recurrent weight of 0.5 and zero inputs, each step back multiplies the signal by exactly 0.5. Ten steps back: 0.5¹⁰ = 0.00098, about a thousandth. With a weight of 1.5 instead: 1.5¹⁰ = 57.7, and training diverges.

W_h = 0.5
# zero inputs keep every h_t at 0
h = [0.0] * 10
influence = 1
for h_t in h:
    # Π over t of tanh's slope times W_h
    influence *= (1 - h_t ** 2) * W_h
round(influence, 5)  # → 0.00098
# the same product with a weight of 1.5
round(1.5 ** 10, 1)  # → 57.7

In code: RNNCell.influence_of_start multiplies out the product above for one sequence, and gradient_through_time measures both lines of the figure.

Everyday picture Instead of rewriting the whole page after every word, keep a notebook (the cell state) and three tools, each controlled by a dial from 0 to 1 that the network sets for itself at every step:

• the eraser (forget gate f): how much of each line to keep;
• the pen (input gate i): how much of the new note to write in;
• the highlighter (output gate o): how much of the notebook to show the outside world right now.

Because the notebook is edited rather than rewritten, information can pass through many steps untouched: eraser off, pen off, and the line survives.

Tiny worked example One-line notebook holding 0.8.

flowchart LR
  CP[notebook c_prev] --> FX((× f<br/>eraser))
  FX --> ADD((+))
  G[new note g] --> IX((× i<br/>pen))
  IX --> ADD
  ADD --> C[notebook c]
  C --> T[tanh]
  T --> OX((× o<br/>highlighter))
  OX --> H[shown h]
  XH[input x and previous h] -.-> FX & IX & OX & G

import math
def sigma(z):
    # σ: any score becomes a dial between 0 and 1
    return 1 / (1 + math.exp(-z))
[round(sigma(z), 3) for z in (-4, 0, 4)]  # → [0.018, 0.5, 0.982]
def lstm_step(f, i, g, o, c_prev=0.8):
    # c_t = f ⊙ c_{t-1} + i ⊙ g
    c = f * c_prev + i * g
    # h_t = o ⊙ tanh(c_t)
    h = o * math.tanh(c)
    return c, h
for f, i, g, o in [(1, 0, 0, 1), (0, 0, 0, 1), (0, 1, 0.5, 1), (1, 0, 0, 0)]:
    # the four rows of the table, dials pinned by hand
    c, h = lstm_step(f, i, g, o)
    print(c, round(h, 3))  # → 0.8 0.664 0.0 0.0 0.5 0.462 0.8 0.0

GRUs simplify this to two dials and no separate notebook: an update gate z chooses between keeping the old state (z = 1) and taking a new candidate (z = 0), and a reset gate decides how much old state feeds that candidate: h = (1 − z) ⊙ n + z ⊙ h_prev, where n is the candidate. Similar performance, fewer parameters.

In code: LSTMCell holds the four gates' stacked weights; LSTMCell.step computes the dials and updates the notebook, LSTMCell.run carries it along a sequence, and LSTMCell.fixed_gates pins the dials to replay the table above; GRUCell.step is the two-dial version.

Everyday picture An RNN is a line of people passing a note: the last person hears about the first only through everyone in between, and nobody can start until the person before them finishes. A transformer is a meeting where everyone can speak to everyone directly, all at once.

Tiny worked example 1,000 tokens. An RNN needs 1,000 steps one after another, and information from token 1 reaches token 1,000 through 999 hand-offs. A transformer layer processes all 1,000 in 1 parallel step, and any token reaches any other in 1 hop of attention.

flowchart LR
  subgraph RNN["RNN: one step at a time"]
    r1[The] --> r2[cat] --> r3[sat] --> r4[down]
  end
  subgraph TF["Transformer: all at once"]
    t1[The] & t2[cat] & t3[sat] & t4[down] --> A[Attention<br/>every pair compared]
  end

In code: sequential_steps and path_length return the two counts in the worked example for an RNN or a transformer.

State-space models such as Mamba revisit recurrence with a design that trains in parallel and runs in time linear in sequence length. They carry a compressed state like an RNN but avoid its training bottleneck, and some hybrid models mix them with attention.

Answer each one out loud or on paper before you open it. If you can explain it, you know it.