Looking inside the model

Everyday picture A car makes a strange noise. You can take it for a test drive and note when the noise happens: that is testing the car's behaviour. Or you can open the hood, put a stethoscope on the engine, and swap parts until the noise stops: that is looking at the mechanism. Test drives tell you what the car does. Only the open hood tells you why.

Every other lesson in this primer treats a trained model as something to build, train or call. Evaluations (primer.agents.evals) are test drives: they measure what the model says. Interpretability opens the hood. It asks what the model's billions of internal numbers represent, and which of them cause the answer. Three reasons to care:

• Debugging. When a model gets something wrong, you want to know which step failed, the same way you read a stack trace rather than just the error message.
• Trust. A model can be right for the wrong reason. An image classifier that "detects" wolves by looking for snow in the background scores well until it meets a wolf on grass.
• Safety. The output is only part of what a model computes. Some questions (is it relying on a stereotype? does it know more than it says?) can only be answered by reading the computation itself.

A tiny worked example. This lesson asks four questions of one kind of sentence, "the Eiffel Tower is in …" → "Paris", each with its own tool, on toy models small enough to check by hand:

flowchart LR
  M["A trained model<br/>(frozen)"] --> A["Hidden activations<br/>at every layer and word"]
  A --> P["Probe:<br/>is property X in here?"]
  A --> L["Logit lens:<br/>what would it predict now?"]
  A --> AP["Activation patching:<br/>does this activation cause the answer?"]
  A --> S["Sparse autoencoder:<br/>which features make up this state?"]
  P & L --> R["Reading<br/>(correlation)"]
  AP --> C["Intervening<br/>(causation)"]
  S --> U["Units of analysis<br/>(what to read and intervene on)"]

Everyday picture A band with two instruments plays through two speakers. The sound engineer pans the guitar mostly to the right speaker and the piano mostly to the left, but each speaker plays a mix of both. If you unplug one speaker you don't lose "the guitar"; you lose part of everything. The instruments are not the speakers. Each instrument is a setting across all speakers, a direction, and the sound in the room is the sum of every instrument times how loudly it plays.

A model's hidden state works the same way. The speakers are neurons: the individual numbers in a layer's output vector. The instruments are features: the things the model has learned to track, such as "this review is positive" or "this verb is in the past tense". Each feature is stored as a direction in the space of neurons, and the hidden state is the sum of every active feature's direction, scaled by how strongly it is present.

A tiny worked example. Take a hidden layer with two neurons and two features whose directions are "positive" = (0.6, 0.8) and "past tense" = (0.8, −0.6). A sentence that is quite positive (0.9) and a little past-tense (0.4) has the hidden state

0.9 · (0.6, 0.8) + 0.4 · (0.8, −0.6) = (0.54 + 0.32, 0.72 − 0.24) = (0.86, 0.48).

Neuron 1 reads 0.86 and neuron 2 reads 0.48. Neither number is "how positive" or "how past-tense": each neuron is a blend of both features. To get the features back, take the dot product (multiply matching entries and add; see primer.notation) of the hidden state with each direction:

• positive: 0.6 · 0.86 + 0.8 · 0.48 = 0.516 + 0.384 = 0.9
• past tense: 0.8 · 0.86 − 0.6 · 0.48 = 0.688 − 0.288 = 0.4

positive = [0.6, 0.8]
past = [0.8, -0.6]
# h = Σ f_i d_i
h = [0.9 * p + 0.4 * q for p, q in zip(positive, past)]
[round(x, 2) for x in h]  # → [0.86, 0.48]
# f̂_i = d_i · h
round(sum(d * x for d, x in zip(positive, h)), 2)  # → 0.9
round(sum(d * x for d, x in zip(past, h)), 2)  # → 0.4
# the directions are perpendicular: their dot product is 0
round(sum(p * q for p, q in zip(positive, past)), 2)  # → 0.0

That features are directions, and that most of what a model tracks can be read with a dot product, is called the linear representation hypothesis. It is a hypothesis, not a law, but it holds often enough to power everything below. You have met it before: word-vector arithmetic such as king − man + woman ≈ queen (primer.ml.embeddings.word2vec) works because "royalty" and "gender" are directions.

In code: compose_features builds a hidden state from feature amounts and read_features reads them back with dot products.

Everyday picture A doctor can't ask your liver how it is doing, but a blood test can measure a marker that tells them. A probe is a blood test for a hidden state: a tiny classifier, trained by you, that looks at a layer's activations and answers one yes/no question, such as "is this review positive?". The model itself is frozen; only the probe learns.

A tiny worked example. A probe for "positive" on a two-neuron layer has weights w = (1, 0.5) and bias b = 0. On the hidden state h = (2, −1):

1. Score it: 1 · 2 + 0.5 · (−1) + 0 = 1.5.
2. Squash the score into a probability with the sigmoid σ(z) = 1 / (1 + e^−z), which maps any number into the range 0 to 1: σ(1.5) = 1 / (1 + 0.223) = 0.818.

The probe is 82% sure this hidden state belongs to a positive review. On h = (−1, 1) the score is −1 + 0.5 = −0.5 and σ(−0.5) = 0.378: probably negative.

import math
w = [1.0, 0.5]
b = 0.0
h = [2.0, -1.0]
# w · h + b
z = sum(wi * hi for wi, hi in zip(w, h)) + b
z  # → 1.5
# σ(z)
round(1 / (1 + math.exp(-z)), 3)  # → 0.818
# a second hidden state, (−1, 1), reads as probably negative
round(1 / (1 + math.exp(-(-1 * 1.0 + 1 * 0.5))), 3)  # → 0.378

This is exactly logistic regression (primer.ml.neural_net), trained the usual way: show it examples whose answer you know, measure its binary cross-entropy (primer.ml.losses), and nudge w and b downhill. The gradient of that loss with respect to the score is simply (p − y), the gap between the probe's probability and the true 0/1 label, which makes each update one line of code.

flowchart LR
  T["Text with a known label<br/>(positive or negative)"] --> M["Frozen model<br/>(no weights change)"]
  M --> H["Hidden state h<br/>at the layer under study"]
  H --> P["Probe: σ(w · h + b)"]
  P --> Y["Probability the label is 'positive'"]
  Y --> G["Compare with the true label<br/>(cross-entropy)"]
  G -. "gradient updates w and b only" .-> P

On our toy model. PlantedModel stores two known features in a 32-neuron hidden layer: sentiment and tense, each as ±1 along its own random direction, plus random noise of size 0.4 on every neuron. No single neuron holds either feature. A probe trained on just 100 examples reads sentiment correctly on 99% of 1,000 examples it never saw.

Why keep probes simple, and check them against a control. A probe can succeed for the wrong reason. Give a probe random labels, a control task with nothing real to find, and it still fits 73% of its 100 training examples, because 33 adjustable numbers can memorize a lot of noise. On unseen examples it scores 50%, a coin flip. Two habits follow: always score a probe on examples it never saw, and compare it with a control task, so that "the probe works" means "the hidden state encodes this" rather than "the probe is clever". This is also why probes are kept linear: a powerful probe (a deep network) could compute the property from raw ingredients by itself, and then its success would tell you about the probe, not the model.

Decodable is not the same as used

Everyday picture A library holds a book nobody ever borrows. Finding it on the shelf proves the library has it, not that it shaped anything any reader did.

A tiny worked example. Our toy model's output reads sentiment and, by construction, gives tense a weight of exactly zero. A probe still reads tense at 98% on unseen examples. Now intervene: take 200 examples, flip only one feature's label, keep everything else fixed, and measure how much the model's output moves.

A probe finding information does not mean the model uses it. To find out what the model uses you have to change something and watch the output, which is what the rest of this lesson does.

In code: train_probe fits a Probe by gradient descent on frozen hidden states from PlantedModel; probe_report trains the three probes, and flip_effect performs the intervention.

Everyday picture A writer keeps every draft of an article. Draft 1 says the landmark is "in a European capital", draft 2 says "probably Paris", the final says "Paris". Reading the drafts shows when the writer made up their mind. The logit lens reads a language model's drafts.

It works because of how a transformer is built (primer.ml.transformer). Each word has a running vector, the residual stream. Every layer reads it and adds a correction to it, rather than replacing it. At the very end, the output layer (the unembedding) turns the final vector into one score per word in the vocabulary. Because every layer writes into the same stream, you can apply that final step early, after any layer, and ask "what would the model predict if it stopped here?"

A tiny worked example. A vocabulary of three words, Paris, London and Rome, a residual stream of two numbers, and an unembedding that scores Paris = first number, London = second number, Rome = minus the first:

The first draft is a vague "some capital" that happens to lean London; layer 1 tips it to Paris; layer 2 commits.

import math
# rows: Paris, London, Rome
W_U = [[1, 0], [0, 1], [-1, 0]]
h = [0.2, 0.3]
writes = [[0.6, 0.0], [1.2, -0.3]]
# a residual layer adds its write to the stream
for w in writes:
    h = [a + b for a, b in zip(h, w)]
[round(x, 2) for x in h]  # → [2.0, 0.0]
# W_U h: one score per word
scores = [sum(u * x for u, x in zip(row, h)) for row in W_U]
[round(s, 2) for s in scores]  # → [2.0, 0.0, -2.0]
# softmax
exps = [math.exp(s) for s in scores]
[round(e / sum(exps), 2) for e in exps]  # → [0.87, 0.12, 0.02]

flowchart LR
  E["Embedding"] --> H0(("h₀")) --> L1["Layer 1<br/>adds its write"] --> H1(("h₁")) --> L2["Layer 2<br/>adds its write"] --> H2(("h₂")) --> OUT["Final norm + W_U<br/>(the real output)"]
  H0 -.-> LENS0["lens: norm + W_U<br/>London 0.40"]
  H1 -.-> LENS1["lens: norm + W_U<br/>Paris 0.55"]
  H2 -.-> LENS2["lens: norm + W_U<br/>Paris 0.87"]

On a model where we know the answer. LandmarkModel is a two-layer, one-head transformer built by hand to complete "Eiffel is in" with "Paris". Layer 1 is an MLP that looks up a fact at every word (landmark in, city out); layer 2 is an attention head that lets the last word, "in", find the landmark and copy its city.

flowchart LR
  T["Eiffel · is · in"] --> EMB["Embed each word"]
  EMB --> MLP["Layer 1: MLP at every word<br/>Eiffel → writes 'Paris'"]
  MLP --> ATT["Layer 2: attention head<br/>'in' looks for the landmark,<br/>copies its city"]
  ATT --> U["Output at 'in':<br/>Paris 3, Rome 0, London 0"]

The logit lens was first described for GPT-2 in 2020, where middle layers already "guess" the next word surprisingly well. In some models the early layers read as nonsense through the final output layer, because they do not yet speak its "language"; the tuned lens fixes this by training a small translator for each layer before applying the output layer.

In code: logit_lens applies an output layer to any residual states, worked_lens runs the table above, residual_stream and tinygpt_logit_lens do the same for primer.ml.transformer.TinyGPT, and lens_map builds the grid for LandmarkModel.

Everyday picture Two cars of the same model sit side by side: one starts, one doesn't. You move parts from the good car into the bad one, one part at a time, and try the ignition after each swap. The part whose swap makes the bad car start is the part that mattered. Activation patching (also called causal tracing) does this with a model's internal activations.

A tiny worked example. Run the landmark model twice:

• the clean prompt "Eiffel is in": scores Paris 3, Rome 0, so the logit difference Paris − Rome is +3;
• the corrupted prompt "Colosseum is in": Paris 0, Rome 3, so the difference is −3.

Now run the corrupted prompt again, but at one chosen place overwrite the activation with the one from the clean run, and see how much of the gap between −3 and +3 comes back:

• patch the MLP output at the landmark's position: the difference jumps back to +3, so 100% of the answer is restored;
• patch the state at "is": it stays at −3, 0% restored ("is" is the same word in both prompts, so its state carries nothing about the landmark).

clean, corrupt = 3.0, -3.0
# restored = (LD_patched − LD_corrupt) / (LD_clean − LD_corrupt)
def restored(patched):
    return (patched - corrupt) / (clean - corrupt)
# the landmark's MLP output, patched in
restored(3.0)  # → 1.0
# the state at "is", patched in
restored(-3.0)  # → 0.0
# a patch that only reaches a tie
restored(0.0)  # → 0.5

Why a difference of two scores, rather than the probability of Paris? Because the corrupted prompt differs from the clean one in exactly one fact, the difference measures exactly that fact, and it moves smoothly, where a probability can saturate near 0 or 1 and hide a change.

flowchart TB
  subgraph C["1. Clean run: 'Eiffel is in'"]
    c1["save every activation"] --> c2["Paris − Rome = +3"]
  end
  subgraph K["2. Corrupted run: 'Colosseum is in'"]
    k1["Paris − Rome = −3"]
  end
  subgraph P["3. Patched run: corrupted prompt,<br/>ONE activation taken from the clean run"]
    p1["Paris − Rome = ?"]
  end
  c1 -- "copy one activation" --> P
  P --> R["fraction restored =<br/>(? − (−3)) / (3 − (−3))"]
  K --> R
  C --> R

Compare with the lens grid above. After layer 2, the lens reads "Paris" at the Eiffel position with probability 0.995, yet patching that state restores 0.00: after the last layer nothing reads that position any more. The information is there, and it no longer matters. Readable is not the same as used, again.

Why it matters in practice. This is how researchers located where GPT-style models store facts: causal tracing on prompts like ours found factual recall concentrated in middle-layer MLPs at the subject's last token, and then used that to edit single facts. The same method, one site at a time, has traced whole circuits, such as the one GPT-2 small uses to fill in "When Mary and John went to the store, John gave a drink to" → "Mary".

In code: LandmarkModel is the hand-built model and LandmarkModel.run accepts patches; fraction_restored is the formula and patching_map patches every layer and position in turn.

Everyday picture Back to the band, but now five instruments play through two speakers. The engineer pans each instrument to its own position around the room: hard left, front right, back left, and so on. When one instrument plays alone, you can tell which one by where the sound comes from. When two play at once, the positions blur together and you might mistake the pair for a third instrument. The trick works because in this band, most of the time, only one instrument is playing.

Models face the same squeeze. There are far more concepts in the world than neurons in a layer, but in any one sentence almost all of them are absent: features are sparse. So models store more features than they have dimensions, as directions that are nearly perpendicular rather than exactly. That is superposition.

A tiny worked example. Put five features in two neurons, spread 72° apart like the points of a pentagon. Feature 's direction is (cos 72k°, sin 72k°): feature 0 is (1, 0), feature 1 is (0.309, 0.951), and so on. Switch on feature 0 alone, at strength 1. The hidden state is (1, 0). Reading every feature back with a dot product gives

Five directions can't all be perpendicular in two dimensions, so reading feature 0 leaks 0.309 into each neighbour: interference. The fix is a small negative bias and a ReLU (which turns negatives into zero): the leak of 0.309 falls below the bias of 0.31 and is filtered to 0, and the real feature survives at 0.69. The cost comes when two neighbours are on at once: each then reads 1 + 0.309 − 0.31 = 0.999 instead of 0.69, too high. Sparsity is a bet that such collisions are rare.

import math
# feature k's direction: (cos 72k°, sin 72k°)
W = [[math.cos(math.radians(72 * k)), math.sin(math.radians(72 * k))] for k in range(5)]
def read_back(x, bias, relu=True):
    # h = W x: two numbers
    h = [sum(W[k][n] * x[k] for k in range(5)) for n in range(2)]
    # Wᵀ h + b: one number per feature
    z = [W[k][0] * h[0] + W[k][1] * h[1] + bias for k in range(5)]
    # ReLU keeps positives and turns negatives into 0
    return [round(max(0.0, v) if relu else v, 3) for v in z]
# the leak: feature 0 alone, no bias, no ReLU
read_back([1, 0, 0, 0, 0], bias=0, relu=False)  # → [1.0, 0.309, -0.809, -0.809, 0.309]
# the bias and ReLU filter it
read_back([1, 0, 0, 0, 0], bias=-0.31)  # → [0.69, 0.0, 0.0, 0.0, 0.0]
# two neighbours on: each reads too high
read_back([1, 1, 0, 0, 0], bias=-0.31)  # → [0.999, 0.999, 0.0, 0.0, 0.0]

flowchart LR
  X["5 features<br/>x₀ … x₄<br/>(mostly zero)"] --> W["squeeze: W<br/>(2 × 5)"]
  W --> H["hidden state<br/>2 neurons"]
  H --> WT["read back: Wᵀ<br/>(5 × 2)"]
  WT --> B["+ bias b<br/>(negative)"]
  B --> R["ReLU"]
  R --> XH["5 features<br/>read back"]

Training it. Let the model learn and itself, by gradient descent on the reconstruction error, with features that matter less and less (feature is weighted ). Compare two worlds:

• Dense: every feature is on in every example. The model keeps the two most important features, perpendicular, at length 1.00, and gives the other three length 0: it simply drops them.
• Sparse: each feature is on only 5% of the time. The model keeps all five, 72° apart, at length about 1.1, and learns a negative bias of about −0.23 to filter the leaks.

That last point is why looking at single neurons so often fails. A neuron that responds to several unrelated features is polysemantic. Vision researchers found neurons that respond to cat faces and to the fronts of cars; language models are full of neurons like that. Superposition explains why: when features outnumber neurons, the features cannot line up with the neurons, so every neuron is a mixture.

In code: pentagon builds the five directions, superposition_readout is the formula, train_superposition learns W and b with gradients from superposition_loss_and_grads (checked against finite_difference_gradient), and features_a_neuron_responds_to lists a neuron's features.

Everyday picture A sound engineer receives the two-speaker recording of the five-instrument band, with no notes on who played when. She knows one thing about this band: usually only one instrument plays at a time. Many different scores could produce the same sound, but she writes down the one that uses the fewest instruments. That preference is what lets her recover the real parts rather than some arbitrary mixture.

A sparse autoencoder (SAE) does this for a model's hidden states. It learns a dictionary: many more directions than the layer has neurons (it is overcomplete), trained so that every hidden state can be rebuilt from just a few of them. Each learned direction is a candidate feature, and each one's strength on an input is its latent activation.

A tiny worked example. The hidden state is h = (0.8, 0.6) and the dictionary has three directions, (1, 0), (0, 1) and (0.8, 0.6). Two codes rebuild h perfectly:

Rebuilding alone can't choose between them. The penalty on the total strength, the L1 penalty (primer.ml.regularization), prefers the code that uses one direction. One side effect: the best strength for direction 3 is not quite 1. With strength , the loss is , which is lowest at (loss 0.0975): the penalty always pulls strengths a little toward zero, known as shrinkage.

dictionary = [[1, 0], [0, 1], [0.8, 0.6]]
h = [0.8, 0.6]
lam = 0.1
def loss(code):
    # ĥ = Σ f_i · direction_i
    rebuilt = [sum(f * d[n] for f, d in zip(code, dictionary)) for n in range(2)]
    # ‖h − ĥ‖² + λ Σ |f_i|
    return round(sum((a - b) ** 2 for a, b in zip(h, rebuilt)) + lam * sum(abs(f) for f in code), 4)
loss([0.8, 0.6, 0])  # → 0.14
loss([0, 0, 1.0])  # → 0.1
# shrinkage: a slightly weaker code is cheaper still
loss([0, 0, 0.95])  # → 0.0975

flowchart LR
  H["hidden state h<br/>(d numbers)"] --> ENC["encoder<br/>ReLU(W_e(h − b_d) + b_e)"]
  ENC --> F["code f<br/>(m ≫ d numbers,<br/>almost all exactly 0)"]
  F --> DEC["decoder<br/>W_d f + b_d"]
  DEC --> HH["rebuilt ĥ"]
  HH --> LOSS["loss = ‖h − ĥ‖² + λ Σ|f|"]
  H --> LOSS
  F --> LOSS

On the superposition toy. Take 20,000 hidden states from the pentagon model (each feature on 5% of the time), train an SAE with 5 latents and λ = 0.3, and compare the learned decoder columns with the five planted directions. Every planted feature is matched by a latent with cosine similarity above 0.99 (1 would be the same direction), and an example with a feature on lights up about 1.07 latents on average. The price is shrinkage: 11% of the variance goes unexplained. With a tiny penalty, λ = 0.01, the SAE rebuilds the data perfectly (0.0% unexplained), yet its worst match to a real feature is only 0.80: any two directions can rebuild a two-dimensional space, so without sparsity pressure nothing forces the dictionary to find the model's actual features.

Why it matters in practice. This is the tool that turned interpretability from "a few hand-picked neurons" into something that scales. Anthropic's Towards Monosemanticity (2023) trained SAEs on a small transformer and found thousands of features that each fire for one recognizable thing. Scaling Monosemanticity (2024) did the same inside a production model, Claude 3 Sonnet, and found features for concepts such as the Golden Gate Bridge. Turning that one feature up made the model bring up the bridge in almost every answer, which is the causal check that the feature means what it seems to mean.

In code: sae_objective computes the loss for one proposed code, SparseAutoencoder holds the encoder and decoder with its hand-derived SparseAutoencoder.loss_and_grads, train_sae fits it, and match_features compares learned directions with planted ones.

Everyday picture A brain scan shows which regions light up while a person reads, but not the sentence they are thinking. Every tool in this lesson is like that: a real instrument that measures something true and partial.

A tiny worked example. Each of our own toys already showed a limit:

flowchart LR
  A["Correlation<br/>probes, the logit lens:<br/>'this information is here'"] --> B["Intervention<br/>patching, steering:<br/>'changing this changes the output'"]
  B --> C["Mechanism<br/>a circuit explained end to end:<br/>'this is how the output is computed'"]

The main open problems, in plain words:

• Choices shape answers. Patching results depend on how the corrupted prompt is built (swap one word? add noise?). SAE features depend on the dictionary's size and λ: a bigger dictionary can split one feature into several finer ones.
• Redundancy hides importance. Patching one site at a time can miss parts that back each other up. Models have been observed to partly repair themselves when one component is knocked out, so "patching this restores 0%" does not always mean "this plays no role".
• Coverage. The unexplained part of an SAE's rebuild is not noise to ignore: it is model behaviour no feature describes yet.
• Scale and labour. A circuit that explains one behaviour of a small model can take researchers weeks. No one has a complete account of how a frontier model produces any long answer.
• Labels are human guesses. Naming a feature "Golden Gate Bridge" summarizes the inputs that make it fire; checking that the name is right needs interventions like the one above.

Why it matters in practice. Treat these tools as evidence, stacked next to behavioural evaluations (primer.agents.evals), never as a certificate. A probe or a lens is a cheap first look; a patching or steering experiment is the minimum for a causal claim; and a clean result on a toy, like every result in this lesson, is where understanding starts, not where it ends.

Answer each one out loud or on paper before you open it. If you can explain it, you know it.