The big picture

This lesson wires the real pieces from the other lessons into one working pipeline: the tokenizer from primer.ml.tokenization, the transformer from primer.ml.transformer, and a sampling loop. Keep this one picture in your head; every other lesson zooms into one box of it.

flowchart LR
  A[Prompt text] --> B[Tokenizer<br/>text to IDs]
  B --> C[Embedding lookup<br/>IDs to vectors]
  C --> D[Add position info]
  D --> E[Transformer blocks<br/>repeated N times]
  E --> F[Output layer<br/>score per vocab token]
  F --> G[Softmax + sampling]
  G --> H[Next token]
  H -->|append and repeat| E

Everyday picture A coat check. You hand over a coat (a piece of text) and get back a numbered ticket (a token id). The model only ever handles the tickets.

Tiny worked example With this module's toy tokenizer, "Reset your password" becomes 5 tickets: Re set y our password → [346, 377, 309, 328, 291]. The common word " password" got a single ticket; the rarer pieces got several.

flowchart LR
  T["'Reset your password'"] --> TK["tokenizer<br/>(learned kit of 400 pieces)"]
  TK --> I["[346, 377, 309, 328, 291]"]

The code tok.encode(prompt); how the kit is learned is the whole of primer.ml.tokenization.

In code: build_pipeline trains the toy tokenizer (primer.ml.tokenization.ByteBPE) and builds an untrained primer.ml.transformer.TinyGPT sized to its vocabulary.

Why it matters Prompt length, price and context limits are all counted in these tickets.

Everyday picture A dictionary where ticket number 291 opens to page 291, and each page holds a list of numbers describing that token. A second dictionary, indexed by seat number, describes where the token sits.

Tiny worked example The table has 400 rows (one per token) of 32 numbers. Id 291 fetches row 291. The 5 ids fetch 5 rows: a 5 × 32 grid. Row i of the position table (i = 0 to 4) is added to row i of that grid.

flowchart LR
  I["ids (5)"] --> E["token table<br/>400 × 32"]
  E --> X["5 × 32: what each token is"]
  P["positions 0..4"] --> PT["position table<br/>64 × 32"]
  PT --> Y["5 × 32: where each token is"]
  X --> ADD(("+"))
  Y --> ADD
  ADD --> OUT["5 × 32 input to the blocks"]

The math and the code

# row 291 of the token table (3 numbers, not 32)
E_291 = [0.2, -0.1, 0.5]
# row 4 of the position table
P_4 = [0.1, 0.3, -0.2]
# E_(t_i) + P_i, number by number
x_4 = [round(e + p, 2) for e, p in zip(E_291, P_4)]
x_4  # → [0.3, 0.2, 0.3]

In code: trace does both lookups and the add, and keeps every stage's result so you can inspect the grid before and after positions are mixed in.

Why it matters This is the only place a token's identity enters the model; every later step works on these vectors.

Everyday picture The team from primer.ml.transformer: each round is a meeting where every token listens to the others (attention), then desk work where each token thinks alone (feed-forward). This toy runs 2 rounds; large models run dozens.

Tiny worked example The 5 × 32 grid goes into block 1 and comes out 5 × 32; the same through block 2. By the end, the vector at the last position ("password") has absorbed information from "Re", "set", "y" and "our".

flowchart LR
  X["5 × 32"] --> B1["block 1<br/>meeting + desk work"] --> B2["block 2"] --> LN["final norm"] --> H["5 × 32<br/>context-aware vectors"]

The math and the code

import statistics
x = [1.0, 2.0, 3.0]
# a stand-in edit
def block_1(v): return [v_i + d for v_i, d in zip(v, [0, 1, 0])]
def block_2(v): return [v_i + d for v_i, d in zip(v, [0, 0, 2])]
def LN(v):
    mu, sigma = statistics.fmean(v), statistics.pstdev(v)
    # centre, then rescale
    return [round((v_i - mu) / sigma, 2) for v_i in v]
h = x
# Block_1 first, then Block_2 ... up to Block_N
for block in [block_1, block_2]:
    h = block(h)
h  # → [1.0, 3.0, 5.0]
LN(h)  # → [-1.22, 0.0, 1.22]

Why it matters This is where nearly all the compute and all the "understanding" happen.

Everyday picture A scoreboard with one line for every token in the vocabulary. Softmax turns the scores into shares of a pie. Temperature sets how adventurous the pick is: low temperature almost always takes the biggest slice; high temperature gives the small slices a real chance.

Tiny worked example Three candidate tokens scored 2.0, 1.0 and 0.5.

flowchart LR
  H["last row of h<br/>32 numbers"] --> S["× token tableᵀ<br/>400 scores"]
  S --> T["÷ temperature"]
  T --> SM["softmax<br/>400 probabilities"]
  SM --> PICK["draw one token"]

The math and the code Softmax raises e (≈ 2.718) to the power of each score, so bigger scores get disproportionately bigger shares, then divides by the total so the shares add up to 1:

import math
z = [2.0, 1.0, 0.5]
T = 0.5
# e^(z_i / T) for each candidate
exps = [math.exp(z_i / T) for z_i in z]
[round(e, 2) for e in exps]  # → [54.6, 7.39, 2.72]
# Σ_j e^(z_j / T)
total = sum(exps)
round(total, 1)  # → 64.7
# p_i: each share of the total
[round(e / total, 3) for e in exps]  # → [0.844, 0.114, 0.042]

Why it matters Use low temperature for extraction and tool calls, where you want the most likely answer, and higher temperature for creative writing. Temperature 0 reduces randomness but doesn't guarantee identical outputs on real serving hardware.

Everyday picture Writing a sentence one word at a time, and re-reading everything you've written before choosing each new word.

Tiny worked example A 2-token prompt, 4 new tokens. Step 1 reads 2 tokens, step 2 reads 3, step 3 reads 4, step 4 reads 5: 14 token positions to produce 4 tokens. With a 20-token prompt and 200 new tokens, the naive loop reads 23,900 positions; a KV cache reads 219.

flowchart LR
  S["ids so far"] --> M["full forward pass<br/>over ALL ids"]
  M --> P["probabilities for the next id"]
  P --> D["draw one id"]
  D --> A["append it"]
  A -->|"not done"| S
  A -->|"stop token or length limit"| OUT["decode ids to text"]

The math Total positions processed without a cache:

p, n = 2, 4
# Σ over t = 0 .. n-1 of (p + t): 2 + 3 + 4 + 5
sum(p + t for t in range(n))  # → 14
# the closed form gives the same count
n * p + n * (n - 1) // 2  # → 14

In code: Generation holds the generated ids, the decoded text and the positions count; naive_vs_cached_work counts the positions processed with and without a KV cache for the figure.

Why it matters Output length drives latency and cost; this is why every serving system caches keys and values.

Everyday picture A guessing game with instant feedback. Cover the next word, guess it, uncover it, and note how surprised you were. Training nudges every weight to make the surprise smaller next time.

Tiny worked example If the model gives the right next token probability 0.5, the loss is −ln 0.5 = 0.693. Our untrained model averages 5.97 on a real sentence, close to ln 400 = 5.99, the score of a blind uniform guess.

flowchart LR
  T["training text"] --> F["forward pass<br/>(this whole lesson)"]
  F --> P["probabilities at every position"]
  P --> L["loss: −ln p(actual next token)<br/>averaged over positions"]
  L --> B["backpropagation<br/>gradient for every weight"]
  B --> U["optimizer nudges weights"]
  U -->|next batch| F

The math and the code A logarithm answers "to what power must I raise e to get this number?" For probabilities between 0 and 1 it is negative, so we flip the sign; ln 1 = 0 (no surprise) and ln of a tiny number is very negative (huge surprise).

import math
# one guess that gave the true token p = 0.5
round(-math.log(0.5), 3)  # → 0.693
n = 12
# a uniform guess gives every true token 1/400
p = [1 / 400] * (n - 1)
# -(1/(n-1)) Σ ln p
L = -sum(math.log(p_i) for p_i in p) / (n - 1)
round(L, 2)  # → 5.99

Why it matters Pretraining is exactly this, over trillions of tokens: predicting the next token well forces the model to absorb grammar, facts and reasoning patterns. Random weights produce gibberish, and training is what turns the same machinery into a useful model.

Answer each one out loud or on paper before you open it. If you can explain it, you know it.