advanced35 minby DevFox

Fine-Tuning, Distillation, and When Not To

A decision framework: what fine-tuning fixes, what it never fixes, the data and evaluation it demands, and the maintenance cost that arrives the day the base model is deprecated.

The quarterly plan says: fine-tune a small model on two years of support transcripts, cut cost and latency, and make it sound like us. Six weeks later you have a model that does sound like you, a labelling contract nobody wants to renew, and not one number that says whether it beats the prompt you were running in March.

That project is not rare and it is not stupid. It is what happens when a real technique is applied to a problem it does not address. By the end of this lesson you will know precisely what training on your data moves and what it leaves untouched, the problems it solves better than anything else and the ones it cannot touch at all, what the dataset and the evaluation cost you after the training run, how adapters change the serving story, and a checklist with one gate you are allowed to fail — because failing it early is the cheapest outcome available here.

What training on your data actually moves

Fine-tuning is continued gradient descent on a released checkpoint, using your own input/output pairs, with the loss computed over the target tokens. Every step nudges the trainable weights in whatever direction makes your target text more likely. That is the entire mechanism, and everything else in this lesson follows from it.

What survives thousands of such steps is whatever was consistent across them. If every example closes with the same three fields in the same order, that regularity is reinforced from every direction at once and comes out reliably. If one example happens to mention that your Frankfurt region uses a fourteen-day window, that fact appears once, against a tide of unrelated gradients, and lands as a faint pull on the output distribution rather than a record you can read back.

Think of an accent against a phone number. Live somewhere for a year and the accent arrives without effort, assembled from thousands of overlapping exposures none of which you could point to. Nobody acquires a phone number that way — you write it down, and you look it up. Weights are the accent. Context is the notebook. The beginner course put fine-tuning on a decision map next to prompting, retrieval and tools; this is the reason it sits where it does.

A fact pressed into weights lacks three things you rely on everywhere else in your system. It has no address, so you cannot read it, cite it, or show a user where the answer came from. It has no date, so it cannot go stale in a way anything detects. And it has no delete, so correcting it means assembling new data and training again.

There is a fourth property that is worse, because it looks like success. A tuned model interpolates. Show it fourteen days for Frankfurt and thirty for the United States, then ask about Austria, and it will produce a confident number in exactly your house format. Training on documents that contain facts teaches the model to generate text of that shape, and shape is most of what makes an invented fact convincing. A tuned model hallucinates in your voice, which makes its errors harder to spot, not easier.

So the rule that organises the whole subject: facts belong in context, form belongs in weights.

The problems it genuinely solves

Three of them are ordinary supervised learning wearing new clothes. The fourth, distillation, is different enough to get its own section.

A consistent output format or voice. Prompt instructions get you the right shape most of the time; the tail is where it costs you. Every rule you add competes for attention with every other rule, and a forty-rule style guide in the system prompt is both paid for on every call and least reliable exactly where the cases are unusual. A few thousand examples encode the same guide in zero prompt tokens and no attention budget. The signal that you are in this case is specific: your reviewers cannot write the rule, but they identify a violation instantly and they agree with each other when they do.

A domain vocabulary. Internal DSLs, clinical shorthand, citation conventions, a coding scheme your industry uses and the public internet barely does. The model's priors here are thin, so it produces output that has the look of the domain without being valid in it. A glossary in the prompt supplies definitions; it does not supply a distribution. Training does, and the common forms start coming out right by default. Note the limit carefully: this teaches the model the shape of the vocabulary, not which code is correct for a given case. That still needs the manual in context or a lookup tool.

A narrow classification done cheaper and faster. Fixed label set, thousands of labelled examples, no explanation required in the output. A small tuned model routinely matches a large general one on that single boundary, because narrowness is precisely what the tune trades for. You give up everything else the small model might have done, and on one task you did not want it to do anything else. The maintenance term is the label set: change the taxonomy and you retrain.

Distillation: buying latency with a teacher

Distillation is fine-tuning where the labels come from a model instead of a person. You run a stronger model — the teacher — over your inputs, keep its outputs, and train a smaller student on those pairs. Where you have access to the teacher's output distribution you can train the student to match the full distribution, which transfers more signal per example. Through a hosted API you usually have only the sampled text, which is response-level distillation and needs more examples for the same result.

What this buys is unit cost and latency at volume, which is sometimes the difference between a feature shipping and not. What it does not buy is general capability. The student receives a compressed copy of the teacher's behaviour on the distribution you sampled from. Step off that distribution and it is a small model again, with no warning at the boundary.

Two disciplines separate a distillation that works from one that quietly poisons itself. The first is where the inputs come from: sample real traffic. If you generate the inputs with a model too, you have distilled the teacher's beliefs about your users rather than your users. The second is what you keep.

Bad — trains the student on everything the teacher produced, including the answers that were wrong.

Good — keeps only the teacher outputs that pass the check the product already applies.

A teacher that is right 94% of the time hands you a training set in which six percent of the examples teach the student to be wrong in exactly the teacher's way — and the student's version of that mistake is in its weights, where a prompt change cannot reach it.

Measure the student against the teacher on the same frozen set, broken out by segment. The average matching while the hardest ten percent of inputs collapses is the ordinary shape of a distillation result, and an aggregate score is designed not to show it to you.

The problems it never solves

Facts that change
A tuned model gives you no mechanism for revocation. When a customer exercises a deletion right over data that went into your training set, there is no row to delete.
The honest answer is a retrain, and that is much better discovered while writing the data-processing agreement than while answering the request.
A task you have not specified
Labels are a specification written by example. If two competent labellers disagree on a fifth of your cases, training averages the disagreement into a model that satisfies neither.
Measure agreement on a hundred examples before you buy ten thousand. Low agreement is a reason to write the rule you have been avoiding, not to hire better labellers.
Quality a prompt or retrieval fix would solve
The common one, and it survives because fine-tuning is the only option on the list that feels like real engineering.
The specific wrong turn is tuning before you have an evaluation set — so you cannot say what was wrong, and will not be able to say whether anything improved.

Three problems that survive a successful training run entirely intact

Bad — spot-checks the tuned model against examples it was trained on, so there is no before number and no honest after number.

Good — scores the base model on a frozen held-out set first, so the tune has a number to beat.

The first version cannot distinguish an improvement from a regression from nothing at all, and it took six weeks to not tell you. The second gives you a comparison on day one — and that frozen set scores the cheaper options too, which is how the project often ends there instead. Whether a two-point gap like that one is real at all is a question about sample size and variance, and the harness lesson at the start of this course is where it is settled.

The costs that arrive later

The training run is the cheap part. Three commitments outlive it.

The dataset is a maintained asset, not an artefact. It needs provenance for every example, a train/dev/test split you hold to so that evaluation stays honest, and an owner. It also goes wrong silently: when your product changes, the examples describing the old behaviour do not raise an error, they just teach the model to keep doing last quarter's job. Every dataset is a liability with the same shape as the codebase it describes.

The evaluation doubles. Before tuning you evaluated one thing: does the feature work. After tuning you evaluate two, and the second is regression on everything the base model could do before you touched it. Narrow training erodes unrelated capability — the effect is called catastrophic forgetting, and in practice it shows up as a model that has become excellent at your classification and noticeably worse at refusing, at following an unrelated output schema, at handling a long input, or at the languages you did not train in.

Bad — gates the release on the task the model was tuned for.

Good — same gate, plus the behaviours tuning is known to erode, each held against the base model's own number.

The first gate ships a model that triages beautifully and has forgotten how to decline an out-of-policy request, and you find that out from a user rather than from CI.

The maintenance liability lands the day the base is deprecated. Your tuned model is a fork of a snapshot. Weights do not port: a new base version, even in the same family, means collecting the data again, training again, and running both evaluation suites again — and none of your previous tuning work transfers. Meanwhile the untuned frontier keeps moving, so the question to ask every couple of quarters is whether your tuned small model still beats the current base model with a good prompt. Often enough it does not, and the sunk cost is the only argument left for keeping it. Managing that transition deliberately is the subject of Drift, Regression, and Model Upgrades later in this course.

Adapters, full tuning, and serving

The two ways to run a tune differ in what they produce, and the artefact decides your serving story.

Full fine-tuning updates every weight. Training needs the parameters, the gradients and the optimizer state resident at once, which is several times the model's size in memory, and it produces a complete new checkpoint of the same size as the base. Serving it means dedicated capacity for a whole distinct model. Two variants are two deployments.

Adapter tuning freezes the base and trains a small number of new parameters injected alongside it. The dominant form is low-rank: for a frozen weight matrix of shape d by k, you learn two small matrices — d by r and r by k, with the rank r small — and the layer computes the frozen product plus the low-rank one. Trainable parameters drop by orders of magnitude and the artefact is megabytes rather than gigabytes. Rank is the capacity knob: too low and the adapter cannot represent the change you want, too high and it overfits a small dataset while giving back the size advantage.

That size difference is not a storage convenience, it is a serving architecture. Because the base is untouched, one loaded base model can host many adapters, selected per request and batched together. A per-customer or per-task variant becomes feasible on shared capacity, and rollback becomes pointing at a different adapter identifier rather than a redeploy. You can also merge an adapter into the base weights, which removes the small per-layer overhead of computing it separately — at the cost of producing a full-size checkpoint again and giving up swapping. Serve unmerged while you are still iterating; merge when a variant is stable and latency-critical.

Choose adapters by default. Style, format, domain vocabulary and classification all sit comfortably inside a low-rank update. Reach for full tuning when the behaviour change is large, when you have a great deal of data, or when you must own and host the weights outright for reasons that are not about quality.

The checklist, and the gate you may fail

Work down this list in order. The first gate is different from the others: failing it does not mean pick another approach, it means stop and go build something you need regardless.

Gate 0 — is there a frozen evaluation set with a baseline number on it? If not, stop here. Build it. Every question below is unanswerable without one, and the set is worth having whether or not you ever tune. Most projects that die at this gate should have.

1. Has the ladder been climbed and measured? The current prompt, a genuinely better prompt, few-shot with real failing cases, retrieval for the missing facts, a stronger model. Each scored on the frozen set. If you cannot name the score for each rung, you do not yet know what fine-tuning would be fixing.

2. Is the failure in form or in fact? Wrong voice, wrong structure, wrong judgement call made consistently — that is form, and a candidate. Wrong information — that is supply, and no amount of training fixes it.

3. Can you produce the data, and keep producing it? Enough labelled examples, with agreement between labellers high enough that the task is real, provenance recorded, and a named owner for the quarter the product changes underneath it.

4. Does the serving shape work? Adapter or full, merged or swapped, and what the rollback is when the tuned variant misbehaves in production.

5. Who retrains, and what triggers it? Name the person and the signal — a base deprecation notice, a taxonomy change, a scheduled comparison against the current untuned base.

Reach the bottom with answers and fine-tuning is the right call. Start with an adapter, ship it behind the same flag you would use for any other model change, and keep the frozen set as the thing that decides.

A cheat sheet to keep

Where to go next

You can now tell, in one conversation, whether a fine-tune is the answer to the problem in front of you — and more often, name the cheaper thing that is. The costs that decide it are not the training run; they are the dataset you now maintain, the second evaluation suite you now own, and the deprecation notice that eventually arrives for the base you forked.

Next comes Routing and Model Portfolios, which answers the question this lesson deliberately left open: once you do have a tuned small model, what decides whether a given request goes to it or to the general one, how a cascade falls back when the small model is out of its depth, and how to measure whether the split saved anything real once you count the requests that had to be done twice.

The thing to do today takes an afternoon. Take the most recent complaint that the model is not good enough, and instead of fixing it, write the fifty-example frozen set that would settle it. Score your current prompt against it. Then score a better prompt, and few-shot with three of the failing cases. Most teams who do this never reach the tuning step, and the ones who do arrive with a number that makes the case for them.