Fine-tuning in practice

Picture a skilled cook who joins your restaurant. They already know how to cook (that is the pretrained model). You want them to cook your menu, your way. You can hand them a note with every order (a prompt), give them the recipe binder to look things up in (retrieval, RAG), or send them on a course about your kitchen (fine-tuning). The course is the only option that changes the cook, and it comes with the risks every teacher knows: the lessons can be badly written, the cook can cram the practice exam instead of learning, and a month of drilling one cuisine can make them rusty at everything else.

Everything below is one of those risks, measured on a model small enough to train in a fraction of a second.

Everyday picture Giving the cook a note with every order costs a little each time. The course costs a lot once, and it has to be repeated whenever you hire a new cook (switch to a newer base model). The course pays off only when the notes would otherwise be long, sent millions of times, or simply not enough to make the cook consistent.

Tiny worked example A support bot sends a 3,000-token prompt full of instructions and examples on every request. A fine-tuned model has learned that behaviour and needs only 300 tokens of prompt. Take these illustrative prices (real ones vary by provider and change often): $2 per million input tokens for the general model, $4 per million for the tuned one, because hosting a custom model usually costs more per token.

Each request saves $0.0048. Writing and checking 1,000 training examples plus the training run costs, say, $600 once. After 600 / 0.0048 = 125,000 requests the course has paid for itself: 25 days at 5,000 requests a day.

flowchart TD
  E[Build the eval set first] --> P[Best prompt you can write]
  P --> M1{Good enough<br/>on the eval?}
  M1 -->|yes| SHIP[Ship the prompt]
  M1 -->|"no: missing facts"| R[Add retrieval, RAG]
  R --> M2{Good enough<br/>on the eval?}
  M2 -->|yes| SHIP
  M2 -->|"no: wrong behaviour, format or style"| FT[Fine-tune, often with LoRA]
  FT --> M3{Beats the prompt<br/>on the SAME eval?}
  M3 -->|yes, and pays off| SHIPFT[Ship the fine-tune]
  M3 -->|no| P

# c = t·p / 1,000,000 for each option
c_prompt = 3000 * 2.0 / 1_000_000
c_tuned = 300 * 4.0 / 1_000_000
c_prompt, c_tuned  # → (0.006, 0.0012)
C_once = 600
# N* = C_once / (c_prompt − c_tuned)
N_star = C_once / (c_prompt - c_tuned)
round(N_star)  # → 125000
# days to break even at 5,000 requests a day
round(N_star / 5000, 1)  # → 25.0

In code: per_request_cost prices one request and break_even_requests returns , or infinity when the tuned model saves nothing per request.

Why it matters in practice. Three costs hide outside this formula. Every new base model means repeating the fine-tune, so recurs. Facts learned by fine-tuning go stale and are hard to update, which is why knowledge belongs in retrieval (primer.agents.rag). And the engineering time to build an eval is spent whichever way you go, which is why it comes first.

Everyday picture Before the course, someone writes the course book. Every recipe must be written in the same layout, no recipe may appear five times, some recipes are locked in a drawer for the final exam before the cook ever sees the book, and the recipes must actually be right. Most fine-tuning failures are course-book failures.

flowchart LR
  RAW[Raw examples<br/>logs, experts, drafts] --> FMT[Format as chat<br/>messages]
  FMT --> VAL[Validate<br/>roles, empty turns]
  VAL --> DD[Deduplicate<br/>near-copies]
  DD --> SPLIT[Set aside the<br/>held-out set, frozen]
  SPLIT --> LEAK[Drop training examples<br/>that copy held-out ones]
  LEAK --> AUDIT[Audit a sample<br/>of the labels]
  AUDIT --> TRAIN[Training set]
  SPLIT --> EVAL[Held-out set]

2a. Formatting chat examples

Everyday picture A play script: every line starts with who speaks it. The model learns the play by reading scripts, and it is graded only on the lines of the character it will play, the assistant.

Tiny worked example One example as a list of role-tagged messages:

Rendered into text, it becomes <|system|>Be brief.<|end|><|user|>Capital of France?<|end|><|assistant|>Paris.<|end|>, and only the last segment counts towards the loss.

flowchart LR
  S["system: Be brief."] --> U["user: Capital of France?"] --> A["assistant: Paris."]
  S -.->|"context only, no loss"| L[Loss]
  U -.->|"context only, no loss"| L
  A ==>|"every token scored"| L

The rules that matter. Use the exact chat template the base model was trained with; the role markers above are illustrative, and each model family has its own. Train with the same system prompt you will deploy with. Reject examples whose last turn is not the assistant's (nothing to learn), whose turns are empty, or whose roles are unknown.

In code: chat_example builds the message list, render_chat flattens it into (segment, trained?) pairs, and validate_chat lists every problem with an example.

Why it matters in practice. A template mismatch is silent: training succeeds, the loss falls, and the deployed model sees markers it never learned, so it behaves like the base model or worse.

2b. Deduplication

Everyday picture A flashcard deck with the same card in it five times. You study that card five times as often, and you start answering every question with it.

Tiny worked example Compare questions by their word pairs (two neighbouring words; a window of k words is called a shingle), after lowercasing and dropping punctuation:

The first two share 5 of the 6 distinct pairs between them: a near-duplicate. The first and third share 2 of 8: different questions.

def pairs(text):
    words = text.lower().replace("?", "").replace(",", "").split()
    return {(a, b) for a, b in zip(words, words[1:])}
A = pairs("How do I reset my password?")
B = pairs("how do I reset my password, please")
C = pairs("How do I change my email?")
# |A ∩ B| and |A ∪ B|
len(A & B), len(A | B)  # → (5, 6)
# J(A, B)
round(len(A & B) / len(A | B), 2)  # → 0.83
# J(A, C)
len(A & C) / len(A | C)  # → 0.25

flowchart LR
  T[Next example] --> CMP{Jaccard with any<br/>kept example ≥ 0.7?}
  CMP -->|yes| DROP[Drop it:<br/>a near-copy]
  CMP -->|no| KEEP[Keep it]
  KEEP --> T
  DROP --> T

In code: normalize lowercases and strips punctuation, shingles collects the word pairs, jaccard scores two texts and deduplicate returns the indices worth keeping. Comparing every pair is fine for thousands of examples; at web scale, MinHash estimates the same Jaccard without comparing every pair.

Why it matters in practice. Duplicates overweight a few examples, so the model parrots them. Worse, a near-copy of an eval question in the training set lets the model recite the answer, and the eval score becomes a memory test. Lee et al. found training sets with thousands of near-duplicates, and removing them made models memorise less.

2c. The held-out set comes first

Everyday picture A good teacher writes the final exam before teaching the course and locks it in a drawer. If the exam were written afterwards, it would drift towards what the class happened to practise.

Tiny worked example Fifty support questions are deduplicated, shuffled with a fixed seed, and 10 (20%) go in the drawer. Only then are the other 40 used, and any of the 40 that nearly copies one of the 10 is removed. Every later choice (prompt wording, learning rate, which checkpoint to ship) is scored on those 10. But how much can 10, or even 100, questions tell you?

import math
a, n, z = 0.8, 100, 1.96
# the standard error sqrt(a(1 − a)/n)
round(math.sqrt(a * (1 - a) / n), 4)  # → 0.04
# the 95% margin
round(z * math.sqrt(a * (1 - a) / n), 4)  # → 0.0784
# four times as many examples halves it
round(z * math.sqrt(a * (1 - a) / 400), 4)  # → 0.0392

flowchart LR
  ALL[All examples,<br/>deduplicated] --> SH[Shuffle with<br/>a fixed seed]
  SH --> EV[20% held out<br/>frozen, never trained on]
  SH --> TR[80% training]
  EV --> CHK{Training example<br/>near-copies one?}
  TR --> CHK
  CHK -->|yes| X[Remove from training]
  CHK -->|no| OK[Keep for training]

In code: split_before_training deduplicates, shuffles with a seed, freezes the held-out set and removes leaks with remove_leaks; margin_of_error gives the ± for any accuracy and size.

Why it matters in practice. Every time you look at held-out scores and change something, the held-out set leaks a little into your decisions. A held-out set built first and used sparingly is the only honest measure of whether the fine-tune helped. See primer.ml.regularization for train, validation and test splits, and primer.agents.evals for building evals.

2d. Label quality

Everyday picture An answer key with typos. A student who gets every question right is marked wrong wherever the key is wrong, and a student who makes the same mistake as the key is marked right.

Tiny worked example A model is truly right 90% of the time, and 10% of the reference labels are wrong. It scores a point when it is right on a right label (0.9 × 0.9 = 0.81), or when it is wrong on a wrong label and the two mistakes cancel (0.1 × 0.1 = 0.01). The eval reports 82%, not 90%.

a, eps = 0.9, 0.1
# right on a right label, plus wrong on a wrong label
round(a * (1 - eps) + (1 - a) * eps, 2)  # → 0.82
# even a perfect model scores only 1 − ε
round(1.0 * (1 - eps), 2)  # → 0.9
# two labelers, five items: how often do they agree?
labeler_1 = ["yes", "no", "yes", "yes", "no"]
labeler_2 = ["yes", "no", "no", "yes", "no"]
sum(p == q for p, q in zip(labeler_1, labeler_2)) / len(labeler_1)  # → 0.8

flowchart TD
  X[One eval item] --> R{Model right?}
  R -->|"yes, 0.9"| LR{Label right?}
  R -->|"no, 0.1"| LW{Label right?}
  LR -->|"yes, 0.9"| P1["scored right: 0.81"]
  LR -->|"no, 0.1"| P2["scored wrong: 0.09"]
  LW -->|"yes, 0.9"| P3["scored wrong: 0.09"]
  LW -->|"no, 0.1"| P4["scored right: 0.01"]

How to check label quality. Have two people label the same sample independently and measure how often they agree. Above, they agree on 4 of 5 items, 80%: one item in five is ambiguous or mislabelled, and your eval can't resolve differences smaller than that noise. Read every disagreement; they are usually unclear instructions, not careless labelers.

In code: measured_accuracy applies the formula and label_agreement compares two labelers.

Why it matters in practice. Bad training labels teach the model the mistakes (section 4 shows a small model memorising three of them). Bad eval labels hide real improvements. Zhou et al. (LIMA) fine-tuned a large model on just 1,000 carefully chosen examples and got a strong assistant: quality of examples beats quantity.

2e. How many examples?

Everyday picture Teaching a house style is not teaching a language. A new cook learns how you plate a dish from a few dozen good examples; they don't need ten thousand.

Tiny worked example Fine-tuning teaches a behaviour the base model can almost do already, so the useful sizes are small: a few dozen examples to show a format, hundreds for a consistent style or a narrow task, thousands for a harder specialised skill. Section 4 fine-tunes on 16 examples and shows the other side: with so few, the model soon memorises them, noise included.

flowchart LR
  S[Start with 50 to 100<br/>high-quality examples] --> T[Fine-tune]
  T --> E[Score on the<br/>held-out set]
  E --> Q{Gained more than<br/>the margin of error?}
  Q -->|yes| D[Double the data] --> T
  Q -->|no| STOP[Stop adding data:<br/>fix quality or approach]

Why it matters in practice. Labelled data is the expensive part of fine-tuning. Doubling until the gains flatten spends it where it helps, and the held-out set tells you when to stop.

Everyday picture Someone who learned to drive in Britain, keeping left, moves to the United States and practises keeping right every day for a month. The new habit wins, and on a trip home they drift to the wrong side of the road. Nobody told them to forget the old rule; the new practice simply rewrote the reflex both rules use. Neural networks do this to an extreme: train on a new task alone and an old one can vanish. This is catastrophic forgetting.

The model we'll fine-tune. To watch it happen, we need a model small enough to train instantly. Each example is four numbers, and the answer is yes or no:

A and B read the same two inputs and apply opposite rules in different regions, so one model can learn both, but only by paying attention to the region. C reads inputs that A and B never touch.

flowchart LR
  X["4 inputs<br/>x1 x2 x3 x4"] --> H["16 hidden units<br/>tanh"]
  H --> O["1 output<br/>probability of yes"]
  TH["θ: all 97 weights<br/>in one vector"] -.-> H
  TH -.-> O

The base model is this network trained on the general task (99% accuracy on held-out examples). Every fine-tune below starts from it and runs plain gradient descent on 200 examples.

In code: TinyNet holds θ and computes predictions, loss and TinyNet.gradient; make_task draws examples for each of the TASKS; base_model pretrains the base and fine_tune trains a copy, leaving the starting model untouched.

3a. The general skill fades while you teach a new one

Everyday picture A month of drilling one cuisine makes the cook a little rusty at everything else, and a second month of drilling it makes them rustier still, even though they had mastered the cuisine in the first week.

Tiny worked example Fine-tune the base on task A (learning rate 0.5) and check both skills on held-out examples:

Task A is learned in 5 steps. The next 295 steps teach nothing new about A but keep eroding the general skill, from 0.825 to 0.655. The last column explains why: the weights keep travelling away from the base.

import math
theta_base = [1.0, 0.0, 2.0]
theta_t = [1.5, -1.0, 2.0]
# each weight's change
[t - b for t, b in zip(theta_t, theta_base)]  # → [0.5, -1.0, 0.0]
# ‖θ_t − θ_base‖
round(math.sqrt(sum((t - b) ** 2 for t, b in zip(theta_t, theta_base))), 3)  # → 1.118

flowchart LR
  DA[Task A examples] -->|gradient| W["Shared weights θ"]
  W --> SA[Answers on task A]
  W --> SG[Answers on the<br/>general skill]
  SG -.->|"no examples,<br/>no gradient"| W

Mitigations that shorten the trip. Three knobs limit the distance, and the toy measures two of them:

• Fewer steps. Stop once the held-out score on the new task stops improving. Here, stopping at step 10 keeps the general skill at 0.79 instead of 0.655, with A at 0.995.
• A lower learning rate. Smaller steps travel less far for the same result: 0.79 instead of 0.655 after 300 steps.
• A smaller update. LoRA (primer.ml.training_stages, section 4) freezes the base weights and allows only a low-rank change. Biderman et al. measured this on real language models and summed it up in their title: LoRA learns less and forgets less.

In code: general_skill_run fine-tunes on A and records, after every step, accuracy on A, the general skill and the distance from the base.

Why it matters in practice. A fine-tuned assistant that has become worse at everything outside its narrow task is the most common fine-tuning disappointment. Always score the general skills you care about alongside the new task, and prefer the earliest checkpoint that has learned the task.

3b. A new task that contradicts an old one

Everyday picture Back to the driver. Practising "keep right" does not merely add a skill; it pushes directly against "keep left", because both use the same reflex.

Tiny worked example Take the model fine-tuned on A (98% on A), then fine-tune it on B alone for 300 steps:

After B, the model does not merely forget A; it answers A's questions backwards, because it learned "yes when x₂ < 0" everywhere. After C, A is barely touched: C's inputs never flowed through the weights A relies on most.

acc_before, acc_after_B, acc_after_C = 0.98, 0.025, 0.97
# F after the contradicting task B
round(acc_before - acc_after_B, 3)  # → 0.955
# F after task C, on separate inputs
round(acc_before - acc_after_C, 3)  # → 0.01

flowchart LR
  BASE[Base] -->|fine-tune on A| MA["Model knows A<br/>A: 0.98"]
  MA -->|fine-tune on B only| MB["Model knows B<br/>A: 0.025, B: 1.00"]
  MA -->|fine-tune on C only| MC["Model knows A and C<br/>A: 0.97, C: 0.98"]

Does a lower learning rate help here? Only in the sense of slowing the slide. At learning rate 0.02, or stopping after 10 steps, B reaches 0.975 and A still drops to 0.355.

In code: sequential_run starts from the A fine-tune, trains on a new task (optionally with replay), and records both accuracies at every step; forgetting computes F.

Why it matters in practice. Real fine-tunes contradict the base model more often than you'd think: "always answer in JSON" contradicts "chat naturally"; "be terse" contradicts "explain in detail". Expect the old behaviour to vanish wherever your data overrides it, and test for it.

3c. Replay: keep practising the old skill

Everyday picture A pianist learning a new piece plays one old piece at the start of every practice session. It costs a few minutes and keeps the old repertoire alive.

Tiny worked example Add just 10 of task A's 200 training examples to B's 200: under 5% of the mix. The result: A 0.945, B 0.99 (up from A 0.025 without replay). Why can so few examples do so much? Look at the loss. Suppose that early in training the model already scores B well (loss 0.05 per example) but has started forgetting A (loss 3.0 on each replayed example):

n_new, L_new = 200, 0.05
n_old, L_old = 10, 3.0
# the replayed share of the data
round(n_old / (n_new + n_old), 3)  # → 0.048
# each group's contribution to the average loss
round(n_new * L_new / 210, 3), round(n_old * L_old / 210, 3)  # → (0.048, 0.143)
# L_mix
round((n_new * L_new + n_old * L_old) / (n_new + n_old), 2)  # → 0.19

flowchart LR
  NB["New task B<br/>200 examples"] --> MIX[Shuffle together<br/>210 examples]
  OA["Old task A<br/>10 kept examples"] --> MIX
  MIX --> FT[Fine-tune]
  FT --> BOTH["Knows B: 0.99<br/>and A: 0.945"]

In code: replay_mix appends the old examples, mixed_loss is the formula, and sequential_run with n_replay=10 runs the experiment.

Why it matters in practice. When fine-tuning a language model, mix some general instruction-following data into your task data, so the model keeps being a good assistant while it learns your task. When you can't replay (the old data is gone or private), methods such as elastic weight consolidation (Kirkpatrick et al.) instead penalise changes to the weights the old task relied on most.

Everyday picture A student with only 16 flashcards, three of which have the wrong answer on the back. For a while, studying teaches the pattern. Keep drilling and the student memorises every card word for word, including the three wrong answers, and gets worse on new questions. primer.ml.regularization builds this idea from scratch; here it is in a fine-tune.

Tiny worked example Fine-tune the base on only 16 examples of task A, 3 of them deliberately mislabelled, for 1,500 epochs. (With full-batch training, one step is one pass over the data, one epoch.)

At the best epoch, training loss is higher than validation loss: the model is refusing to fit the three wrong labels, which is exactly right. By the end, training loss is nearly zero, so the wrong labels have been memorised, and validation loss has quadrupled.

L_train = {"best": 0.373, "end": 0.005}
L_val = {"best": 0.319, "end": 1.277}
# g = L_val − L_train at each point
{t: round(L_val[t] - L_train[t], 3) for t in L_val}  # → {'best': -0.054, 'end': 1.272}

flowchart LR
  EP[Train one epoch] --> SV[Save a checkpoint]
  SV --> SC[Score it on the<br/>held-out set]
  SC --> Q{Best so far?}
  Q -->|yes| MARK[Mark it best] --> EP
  Q -->|"no, patience used up"| SHIP[Ship the best checkpoint,<br/>not the last]
  Q -->|"no, patience left"| EP

In code: overfitting_run fine-tunes on the 16 examples and records both losses every epoch; primer.ml.regularization.early_stopping replays the validation curve and returns the best epoch and the stopping epoch.

Why it matters in practice. Fine-tuning datasets are small compared to pretraining, and big models memorise quickly, so fine-tunes typically run for only a few epochs. Save checkpoints, score each on the held-out set, and ship the best one.

5a. Weight averaging and task arithmetic

Everyday picture Two editors each take a copy of the same draft and make tracked changes: one fixes the grammar, the other tightens the argument. You can apply both sets of changes to the original. If instead you "average" the two edited copies, each change is applied at half strength: half the grammar fixed, half the argument tightened.

Tiny worked example A three-weight base (1, 0, 2). One fine-tune moves it to (1.5, 0, 2); another to (1, −1, 2).

The change a fine-tune made, θ_ft − θ_base, is its task vector. Adding task vectors to the base is task arithmetic.

theta_base = [1.0, 0.0, 2.0]
theta_A = [1.5, 0.0, 2.0]
theta_C = [1.0, -1.0, 2.0]
# τ_t = θ_t − θ_base
tau_A = [a - b for a, b in zip(theta_A, theta_base)]
tau_C = [c - b for c, b in zip(theta_C, theta_base)]
tau_A, tau_C  # → ([0.5, 0.0, 0.0], [0.0, -1.0, 0.0])
# θ_merged at λ = 1
[b + 1.0 * (a + c) for b, a, c in zip(theta_base, tau_A, tau_C)]  # → [1.5, -1.0, 2.0]
# λ = 1/2 ...
[b + 0.5 * (a + c) for b, a, c in zip(theta_base, tau_A, tau_C)]  # → [1.25, -0.5, 2.0]
# ... is the plain average of the two fine-tunes
[(a + c) / 2 for a, c in zip(theta_A, theta_C)]  # → [1.25, -0.5, 2.0]

flowchart LR
  B[Base θ] -->|fine-tune on A| FA[θ_A]
  B -->|fine-tune on C| FC[θ_C]
  FA --> TA["τ_A = θ_A − θ_base"]
  FC --> TC["τ_C = θ_C − θ_base"]
  TA --> SUM["λ × (τ_A + τ_C)"]
  TC --> SUM
  B --> ADD((+))
  SUM --> ADD
  ADD --> M[Merged model:<br/>no extra training]

In code: task_vector subtracts the base, merge adds scaled task vectors back, and merge_run merges two fine-tunes at several λ and scores the result.

Why it matters in practice. Merging combines skills trained separately, by different teams or on data that can't be pooled, without any further training and at no extra inference cost. Averaging several fine-tunes of the same task ("model soups", Wortsman et al.) often beats the best single one. Task vectors can also be subtracted: Ilharco et al. negated a task vector learned from toxic text to make a model less toxic.

5b. Interference: when task vectors collide

Everyday picture Two editors rewrote the same sentence in opposite directions, one making it warmer and one making it colder. Applying both sets of tracked changes gives a sentence neither intended.

Tiny worked example Our three-weight changes again, plus a new one: τ_A = (0.5, 0, 0), τ_C = (0, −1, 0) and τ_B = (−0.4, 0, 0.3). τ_A and τ_C touch different weights: no conflict. τ_B pulls the first weight the other way from τ_A. The cosine measures how aligned two changes are:

import math
tau_A = [0.5, 0.0, 0.0]
tau_B = [-0.4, 0.0, 0.3]
tau_C = [0.0, -1.0, 0.0]
def cos(u, v):
    dot = sum(a * b for a, b in zip(u, v))
    return dot / (math.sqrt(sum(a * a for a in u)) * math.sqrt(sum(b * b for b in v)))
# opposed changes
round(cos(tau_A, tau_B), 2)  # → -0.8
# changes to different weights
cos(tau_A, tau_C)  # → 0.0

In the toy, the fine-tunes on A and on B (which reverses A's rule) have task vectors with cosine −0.25, against −0.04 for A and C. Each fine-tune learned its rule everywhere, not just in its own region, so the two vectors rewrite the same weights in opposite directions.

flowchart TD
  TV[Task vectors from<br/>the same base] --> COS{Cosine between them}
  COS -->|"near 0: separate weights"| ADD[Add them:<br/>task arithmetic]
  COS -->|"clearly negative: conflict"| FIX{Can you retrain?}
  FIX -->|yes| JOINT[Train one model on<br/>both datasets, or replay]
  FIX -->|no| TIES["Resolve conflicts:<br/>trim small changes,<br/>agree on a sign per weight"]
  ADD --> EV[Score the merge on<br/>every task's held-out set]
  TIES --> EV
  JOINT --> EV

In code: cosine measures the angle between two task vectors, and merge_run reports it alongside the merged accuracies.

Why it matters in practice. Merges are free to try, which makes them tempting to trust. A merged model can quietly lose a skill both parents had, so it is evaluated like any new model. The cosine tells you in advance which merges to be nervous about.

Answer each one out loud or on paper before you open it. If you can explain it, you know it.