Math notation, from zero

Machine learning papers look impenetrable mostly because of notation: Greek letters, big sigmas, little superscript Ts. Almost every symbol is shorthand for a short loop you could write in a few lines of Python. This lesson writes each one out as that loop, so that when a formula appears elsewhere in this primer you can read it aloud.

Every function below is deliberately written the slow, obvious way, with plain Python loops, so the code is the definition. The tests check each one against NumPy, which does the same thing fast.

Everyday picture A vector is a list of numbers, like a shopping receipt: (apples 3, bread 1, milk 2). A matrix is a table of numbers, like a spreadsheet with rows and columns.

Tiny example is a vector with 3 entries. Its entries are written with a small subscript: , , . A matrix with 2 rows and 3 columns has shape , and means "row 2, column 3".

In practice, a word's embedding is a vector, a batch of embeddings is a matrix (one row per word), and a model's weights are mostly matrices.

Everyday picture Totting up a receipt.

x = [1, 2, 3, 4]
total = 0
# Σ: visit each x_i, from i = 1 to n ...
for x_i in x:
    # ... and add it to a running total
    total += x_i
total  # → 10
# Python's built-in sum is the same loop
sum(x)  # → 10

Π (capital pi) is the same idea with multiplication: . It shows up whenever independent chances combine. For example, ten steps that each succeed 95% of the time all succeed with probability . This single fact explains why long chains of AI agent steps fail so often (primer.agents.planning).

In Python:

x = [1, 2, 3, 4]
product = 1
# Π: the same loop, multiplying instead of adding
for x_i in x:
    product *= x_i
product  # → 24
# ten steps that each succeed 95% of the time
round(0.95 ** 10, 2)  # → 0.6

In code: product is Π written the same way: a loop with a running product that starts at 1.

Everyday picture A recipe needs 2 eggs, 3 cups of flour and 1 cup of sugar, and eggs cost $1, flour $0.50 and sugar $2 per unit. The total cost is 2·1 + 3·0.5 + 1·2 = $5.50: multiply matching items, then add. That is a dot product.

a = [1, 2]
b = [3, 0.5]
# Σ over i of a_i times b_i
sum(a_i * b_i for a_i, b_i in zip(a, b))  # → 4.0

Geometrically, the dot product is large when two arrows point the same way, zero when they are at right angles, and negative when they point apart. That is why it is the standard similarity score for attention and for embeddings.

flowchart LR
  A["a = (1, 2)"] --> M1["1 × 3 = 3"]
  B["b = (3, 0.5)"] --> M1
  A --> M2["2 × 0.5 = 1"]
  B --> M2
  M1 --> S["3 + 1 = 4"]
  M2 --> S

In code: dot pairs up matching entries, multiplies them and adds the products with summation, refusing lists of different lengths.

Everyday picture Walk 3 blocks east and 4 blocks north. As the crow flies, you are 5 blocks from where you started (Pythagoras).

import math
def length(x):
    # √ of Σ x_i²
    return math.sqrt(sum(x_i ** 2 for x_i in x))
length([3, 4])  # → 5.0
length([1, 2, 2])  # → 3.0

Dividing a vector by its length gives a unit vector of length 1 that points the same way. Embedding systems do this constantly, because then the dot product measures only direction (see primer.ml.embeddings.similarity).

In code: norm is the square root of a vector's dot product with itself.

Transpose, written ("A transpose"), flips a table on its diagonal so rows become columns: a matrix becomes .

Matrix multiply, written with nothing in between, is a whole grid of dot products. The cell in row , column of the answer is row of dotted with column of .

A = [[1, 2], [3, 4]]
B = [[5, 6], [7, 8]]
# A has m columns, B has m rows: they must match
m = len(B)
# (AB)_ij = Σ_k A_ik B_kj
AB = [[sum(A[i][k] * B[k][j] for k in range(m))
       for j in range(len(B[0]))]
      for i in range(len(A))]
AB  # → [[19, 22], [43, 50]]
# the transpose: rows become columns
[list(column) for column in zip(*A)]  # → [[1, 3], [2, 4]]

flowchart LR
  R["row 1 of A<br/>(1, 2)"] --> D["dot product<br/>1·5 + 2·7 = 19"]
  C["column 1 of B<br/>(5, 7)"] --> D
  D --> O["answer, row 1, column 1 = 19"]

In code: transpose turns columns into rows; matmul transposes B once, then fills each cell with one dot of a row of A and a column of B.

Everyday picture Money in an account that compounds continuously at 100% a year grows by a factor of e ≈ 2.718 in one year. is that growth run for years. The natural logarithm (often just in ML papers) answers the reverse question: how many years of that growth turn 1 into ?

Two facts carry most of machine learning:

1. is always positive and grows fast, which is why softmax uses it.
2. . Logs turn multiplying into adding. The probability of a whole sentence is a product of thousands of small numbers, which would underflow to 0 on a computer. Its log is a sum of manageable negative numbers. This is why models work with "log-probabilities", and why the standard training loss is (see primer.ml.losses).

softmax turns a list of scores into shares that are positive and sum to

1. It is covered step by step, with its symbols decoded, in primer.ml.attention:

import math
z = [2.0, 1.0, 0.5]
# e^(z_i) for each score
exps = [math.exp(z_i) for z_i in z]
# Σ_j e^(z_j)
total = sum(exps)
# each share of the total
[round(e / total, 2) for e in exps]  # → [0.63, 0.23, 0.14]
# argmax: the position of the largest
max(range(len(z)), key=lambda i: z[i])  # → 0

argmax is simpler: it answers "which position holds the largest value?", not "what is the largest value?". argmax(0.1, 7.0, 3.0) = position 2 (index 1 in Python). "Greedy decoding" in a language model is picking the argmax token every step.

In code: softmax subtracts the largest score before exponentiating, so the exponentials cannot overflow; argmax walks the list and remembers the position of the biggest value.

Everyday picture Two classes both average 70% on a test. In one, everyone scored 68 to 72; in the other, scores ran from 30 to 100. The mean is the same; the spread is not.

import math
x = [2, 4, 4, 4, 5, 5, 7, 9]
n = len(x)
# μ = (1/n) Σ x_i
mu = sum(x) / n
mu  # → 5.0
# the squared distances from μ
[(x_i - mu) ** 2 for x_i in x]  # → [9.0, 1.0, 1.0, 1.0, 0.0, 0.0, 4.0, 16.0]
# σ² = their average
variance = sum((x_i - mu) ** 2 for x_i in x) / n
# σ is its square root
variance, math.sqrt(variance)  # → (4.0, 2.0)

Spread matters constantly in neural networks. If numbers flowing through a network spread out layer after layer, training blows up; normalization layers and careful initialization exist to hold σ near 1 (see primer.ml.deep_nets and the √d_k in primer.ml.attention).

In code: mean, variance and std are the three formulas above, each a short loop built on summation.

Everyday picture You're on a hillside in thick fog and want to reach the valley. You can't see it, but you can feel the slope under your feet. Take a small step in the steepest downhill direction, feel again, and repeat. That is how every neural network is trained (gradient descent).

The derivative of a function at a point is its slope there: how much the output changes per tiny nudge of the input.

def f(x):
    return x ** 2
h = 0.00001
# rise over run across a tiny step
round((f(3 + h) - f(3 - h)) / (2 * h), 6)  # → 6.0

With many inputs, nudge each one separately. The slope in each direction is a partial derivative, written (the curly ∂ just means "only this input moves, the others stay fixed"). Collect them into a vector and you have the gradient, written ("nabla f" or "grad f"). It points in the steepest uphill direction, so training steps the opposite way:

# (x, y)
theta = [1.0, 2.0]
# ∇L for L = x² + y²: each slope is 2 times the value
gradient = [2 * theta[0], 2 * theta[1]]
gradient  # → [2.0, 4.0]
eta = 0.1
# θ_new = θ − η ∇L
[t - eta * g for t, g in zip(theta, gradient)]  # → [0.8, 1.6]

The chain rule says how slopes combine when functions are chained: if , then . Multiply the slopes along the chain. Backpropagation is the chain rule applied backwards through every layer of a network, reusing work as it goes (see primer.ml.neural_net).

In code: derivative measures a slope by nudging the input a hair up and a hair down; gradient does that for each input in turn and collects the slopes into one list.

, "order n squared", describes how cost grows as the input grows, ignoring constant factors. If doubles, an cost doubles and an cost quadruples. Attention is in sequence length, which is why long context is expensive (primer.ml.attention).

Answer each one out loud or on paper before you open it. If you can explain it, you know it.