Few-Shot Examples That Teach
When showing beats telling, how to choose examples that generalise instead of ones the model copies verbatim, and how a badly-chosen example set narrows the model instead of guiding it.
When showing beats telling, how to choose examples that generalise instead of ones the model copies verbatim, and how a badly-chosen example set narrows the model instead of guiding it.
Your changelog prompt is three paragraphs long by now. You have told the model to write for customers, keep each entry to one line, use plain language, avoid internal jargon and never mention ticket numbers. It still hands you "This commit refactors the discount validation logic in the cart module." You add a sixth sentence. It gets slightly worse.
Some rules are far quicker to demonstrate than to describe. Few-shot prompting — putting a handful of worked input-and-output pairs into the prompt, before the real input — is how you demonstrate. By the end of this lesson you will know when an example beats another sentence of instruction, how to choose a set that generalises instead of one the model traces over, how a narrow set quietly shrinks what the model will do for you, and why every example is a bill that arrives on every call.
There are two kinds of rule in a prompt, and only one of them can be written as a sentence.
One sentence covers it.
Reply in English. Never include a customer's email address. Stay under two hundred words.
These stay as instructions, because a model treats an instruction as a constraint.
Every sentence makes it fuzzier.
"Sound like our release notes." "The right level of detail." "Not too formal."
You know exactly what you mean and recognise it in a second when you see it — which is precisely the property an example has and a description does not.
Think about teaching someone to make coffee the way you make it. You can write out a recipe — grind size, water temperature, thirty seconds of bloom — or you can make two cups in front of them and let them taste. The recipe is precise about whatever was easy to measure and silent about the rest. The demonstration carries the parts you never thought to name.
That is the whole idea. A prompt with no examples is zero-shot: you describe the job and hope the description was complete. A prompt with a few worked pairs in it is few-shot. The heuristic is blunt and it works: when you find yourself writing a fourth sentence about how the answer should look, stop writing and show one instead.
Here is the same changelog job with the instructions cut down to two sentences and three worked pairs added underneath.
Turn each commit message into one line for our public
release notes. Write for a customer.
Commit: fix(cart): guard against null discount code
Line: Fixed a crash when applying an empty discount code.
Commit: feat(export): add CSV export to the orders table
Line: You can now export your orders as a CSV file.
Commit: fix(auth): correct expiry check on reset links
Line: Password reset links now expire when they should.
Commit: feat(billing): let admins pause a subscription
Line:The model completes the pattern:
Line: Admins can now pause a subscription instead of
cancelling it.Read that against what you actually wrote. You never said to
drop the feat(billing): prefix, or to describe repairs in the
past tense and new abilities in the present, or to keep it to
one sentence ending in a full stop, or to leave out file and
module names. Rules you did not have to be able to state,
carried by three pairs.
Examples teach by resemblance, and resemblance does not distinguish between the properties you meant and the ones your examples happened to share.
Bad — two real examples that are both, by accident, bug fixes.
Commit: fix(cart): guard against null discount code
Line: Fixed a crash when applying an empty discount code.
Commit: fix(auth): correct expiry check on reset links
Line: Password reset links now expire when they should.
Commit: feat(billing): let admins pause a subscription
Line: Fixed subscription pausing for admins.Good — the same two slots, spanning the kinds of change you actually ship.
Commit: fix(cart): guard against null discount code
Line: Fixed a crash when applying an empty discount code.
Commit: refactor(api)!: rename order_ref to order_reference
Line: The API field order_ref is now order_reference. Update
any integration that reads it.
Commit: feat(billing): let admins pause a subscription
Line: Admins can now pause a subscription instead of
cancelling it.The changelog is the one page customers read to find out what is new. The first set turns every entry into a repair, so the feature you spent a month building is announced as something that used to be broken — and nothing errors, so nobody notices until a customer asks when it is coming.
The instinct is to pick representative examples — ordinary, middle-of-the-road inputs showing the job at its most typical. That instinct picks exactly the inputs the model already handles without help.
Better: your examples are not a summary of your inputs, they are a map of the edges — the boundary of what you will actually receive. Write down how your real inputs vary. Commit messages vary in length, in kind (repair, new ability, removal, breaking change, internal housekeeping), and in whether they belong in a customer-facing changelog at all. Then fill four roles rather than picking three favourites.
A typical example anchors the format: one plain input, handled the plain way, teaching labels, length and tense.
An edge example sits at an extreme — the commit with a five-line body where one line matters to a customer, or the two-word commit with almost nothing in it. It stops the model assuming every input looks like the last one it saw.
A hard example is one where a reasonable person would hesitate. Is a breaking rename good news or a warning? The example is where you settle it, and it is the highest-value slot in the set: precisely where the model would otherwise guess, and it will guess the same way every time without ever telling you it guessed.
A refusal example is the input that should not get the normal treatment at all. That one earns its own section.
Look at what a set of confident answers teaches on top of
format: that every input gets an answer. Every input in, a
polished line out. Hand it
chore(deps): bump the test runner to 9.2 and it will write
you a polished line about that too, because it has never seen
this prompt do anything else.
You can add "output SKIP for internal changes" as an instruction, and you should. But one sentence competes with every demonstration you gave, and the demonstrations are the thing whose shape the model can see. Show the refusal once and it becomes an available move:
Commit: chore(deps): bump the test runner to 9.2
Line: SKIPGive the refusal a concrete, recognisable form — one word, in the same slot as a normal answer, so the code around your prompt can test for it with a single comparison. "I don't think this belongs in the changelog, but let me know if you'd like a line anyway" is a refusal you have to parse, and parsing polite prose is how a well-behaved model ends up posting an apology to your release notes.
Everything inside the examples is signal, including the parts you thought were packaging. Labels, order, capitalisation, quotes, the blank line between pairs — nothing tells the model which of those you meant and which you were careless about.
Bad — three pairs, three different shapes.
Commit: fix(cart): guard against null discount code
Line: Fixed a crash when applying an empty discount code.
Message: feat(export): add CSV export to the orders table
Changelog entry: "You can now export your orders as a CSV file."
fix(auth): correct expiry check on reset links
-> Password reset links now expire when they should.Good — one shape, repeated without variation.
Commit: fix(cart): guard against null discount code
Line: Fixed a crash when applying an empty discount code.
Commit: feat(export): add CSV export to the orders table
Line: You can now export your orders as a CSV file.With the first set, some answers come back wrapped in quotes,
some carry a Changelog entry: prefix, and some arrive bare.
Anything downstream that reads that text — a script pasting
entries into a release draft — breaks on roughly one in twenty.
That is the worst rate there is: too rare to show up while you
are testing, common enough to be wrong in public every release.
Order matters more than people expect, too. The example closest to the real input tends to weigh heaviest, so put your most representative one last. When you are unsure about an example, move it and see whether the output moves with it.
Here is the part that catches people out. Examples do not only add; they subtract.
Before you add any, the model's range on this task is enormous. Every example pulls that range towards itself — that is the entire mechanism, and it is what you wanted. But nothing separates the narrowing you intended from the narrowing that came along with it.
Say all four of your examples happen to open with "You can now" or "Fixed". Feed the prompt a removal and you get "You can now no longer export in the old format." No competent writer produces that sentence, and the model was entirely capable of "The old export format has been removed" — you talked it out of it. The same effect crushes a two-sentence change into one because every example was one sentence, and forces an unfamiliar input into the mould of the nearest example instead of handling it on its own terms.
The dangerous word is silently. Nothing errors, and your spot checks look excellent, because you spot check the kind of input you had in mind when you wrote the examples. The damage lives in the inputs you did not think of, which is where damage always lives.
Run it with every example deleted.
Same prompt, same inputs, no examples. If the no-example version handles a case better, your examples caused that loss.
The five that make you wince.
The real inputs that least resemble any example you wrote — not the ones you are proud of.
An instruction is written once and read on every request. So is an example — and examples are usually the largest thing in the prompt by a wide margin.
Four pairs of a sentence or two each is a solid paragraph riding along on every call your feature makes, every hour, forever. You are charged for what you send, so that paragraph has a running cost, and it is read before the first word comes back, so it is part of your latency too. At a thousand calls a day none of that is worth a meeting. At a hundred thousand it is a line item.
So examples are a trade, not a free win, and each has to earn its slot. Add them one at a time and stop when the next one changes nothing you can see. Then run the subtractive test: delete each in turn, re-run your awkward inputs, and keep only the ones whose removal makes something worse. Sets that survive that usually come out at two to four, not ten.
If you are at fifteen examples and still patching failures, few-shot has stopped being the right tool — the task is probably two tasks wanting two calls, or a constraint you keep demonstrating that should have been stated outright.
Example or instruction?
format, tone, level of detail -> example # hard to state
fourth clarifying sentence -> example # showing is faster
a taste call, not a hard limit -> example # show the call
must never be broken -> instruct # examples only nudge
one unambiguous sentence -> instruct # cheaper than a pair
a constraint, not a style -> instruct # "reply in English"
Fill four roles, not four favourites
typical a plain, middle-of-road input # anchors the format
edge longest / shortest / weirdest # stops literal copying
hard genuinely ambiguous input # highest value slot
refusal input that gets no normal answer # permission to skip
Keep the set honest
identical labels and order every time # drift becomes output
vary what you do not want copied # kind, length, opening
real inputs from logs, names removed # invented ones cluster
most representative example last # nearest the real input
refusals in a fixed, testable form # SKIP, not an apology
Before you ship
run it with every example deleted # the zero-shot control
run the five inputs least like your set # finds the narrowing
delete each example in turn # keep what earns its slot
count what they add to every call # paid on all of themYou can now tell a rule worth stating from one worth showing, fill the four slots instead of picking three comfortable inputs, keep one format throughout, give the model a way to decline, and check what your examples cost in both quality and money.
The natural next step is why models hallucinate. You have given the model permission to skip a commit; the neighbouring question is what happens when it has no such permission and no idea of the answer — why confident invention is the default rather than a malfunction, and which popular fixes genuinely reduce it. Examples shape how an answer looks. That lesson is about whether the answer is true.
Tonight, run the subtractive test on a prompt you already use. Delete one example, run your ten most awkward inputs, and see whether anything gets worse. Then delete every example and run the same ten again. One of those runs will teach you something about your own prompt — usually that an example you were proud of was doing nothing, or that one was quietly holding the model back.