Giving the Model Your Data
The decision map every AI feature runs into: put it in the prompt, retrieve it at query time, give the model a tool to fetch it, or train it in. What each option costs and which problems it actually solves.
The decision map every AI feature runs into: put it in the prompt, retrieve it at query time, give the model a tool to fetch it, or train it in. What each option costs and which problems it actually solves.
Your support assistant has been live for a week, and it is answering refund questions warmly, fluently, and with a thirty-day return window. Your return window is fourteen days. Nobody typed thirty anywhere. The model needed a number, and thirty is the number the rest of the internet uses.
Nothing is broken. The model has never seen your policy, and no amount of rewording will make it see one. By the end of this lesson you will know the four ways to actually hand a model your data, what each costs in freshness, size, speed, money and effort, and a question order that picks one in about a minute. You will also know the expensive wrong turn — reaching for training when the real problem was that nobody supplied the facts.
A model is trained once, on an enormous pile of text, and then its weights — the numbers that make it behave the way it does — are frozen and shipped. Everything it appears to know is a compressed impression of that pile, which was public text gathered up to some cutoff date.
Your handbook is not in that pile. Neither is your ticket history, your price list, the row describing the person currently typing, or an order placed nine minutes ago. Some of that is private; the rest did not exist yet.
Think of a brilliant new hire on their first morning. They have read every book ever published and can talk about anything — and nobody has given them a login. They cannot tell you your refund window, not because they are slow, but because nobody told them. This is a supply problem, not an intelligence problem, and a smarter model does not fix it.
Worse, the gap does not feel like a gap from the inside. You already know why a model produces a confident sentence rather than admitting it has nothing. Missing facts do not throw an error; they come back as fluent wrong answers.
There is exactly one channel into the model: the request you send. Each call starts from nothing — there is no drawer where last week's documents accumulate. So "giving the model your data" can only mean one of four things. Put the data in this request. Go and find the part worth putting in this request. Let the model ask for it mid-request. Or change the model itself.
Start with the boring one, because it is right more often than people expect. If you have a document and the answer is in it, paste the document into the request above the question.
# ask_model() is whichever provider SDK you are using.
def answer(question: str) -> str:
# Read per request, so an edit to the file is live at once
# without a deploy.
policy = read_text("refund-policy.md")
return ask_model(
f"Refund policy:\n{policy}\n\n"
f"Answer using only the policy above.\n\n"
f"Question: {question}"
)That is the whole technique, and it has real virtues. The data is as fresh as the file, there is nothing to build or keep in sync, and when an answer is wrong you can read the exact text the model was given.
It has one hard limit and one soft one. The hard limit is the context window — the most text one request can carry. Past it, something gets dropped. The soft limit arrives earlier: you pay for every token you send on every single call, and a fat prompt is a slow prompt.
Bad — sends all 180 pages of the handbook on every question, including "how do I reset my password".
HANDBOOK = read_text("handbook.md") # about 180 pages
def answer(question: str) -> str:
return ask_model(
f"Company handbook:\n{HANDBOOK}\n\n"
f"Question: {question}"
)Good — sends the one section the question is already known to be about.
SECTIONS = load_sections("handbook.md") # topic -> section text
def answer(question: str, topic: str) -> str:
return ask_model(
f"Handbook section '{topic}':\n{SECTIONS[topic]}\n\n"
f"Question: {question}"
)The first version pays for 180 pages on every question anyone asks, gets slower with every page added to the handbook, and hits the context limit one day without warning — while the paragraph that mattered sat buried in a haystack you built.
Notice what made the good version possible: the caller knew the topic, because the user picked a category or clicked a help page. When you do not know which part is relevant, you have to go and find it — which is the next option.
One handbook you can slice by hand. Forty thousand support tickets, six years of engineering docs, and every contract your company has signed — you cannot paste those, and you cannot pick the right three paragraphs by hand for each question.
So you put a search step in front of the model. The question arrives, you search your own content for the passages most likely to contain the answer, you paste those few passages into the prompt, and you ask. That is retrieval, and the pattern of retrieving-then-generating is what people mean by RAG.
def answer(question: str) -> str:
passages = search_docs(question, limit=5)
context = "\n\n---\n\n".join(passages)
return ask_model(
f"Excerpts from our documentation:\n{context}\n\n"
f"Answer only from the excerpts. If they do not say,\n"
f"reply: not covered in our documentation.\n\n"
f"Question: {question}"
)Read that carefully and you will see it is option one with a
lookup bolted to the front. The model's side of the deal is
unchanged: text arrives in the request, the model uses it.
Everything interesting has moved into search_docs.
What retrieval costs is a second system. There is an index to build and keep current, a search step added to every request's latency, and a failure mode that catches everyone once: retrieval failure looks exactly like model failure. If the search returns the wrong five paragraphs, the model answers wrongly, confidently, in beautiful prose, and your logs show a bad answer rather than a bad search.
Making that search genuinely good — how to split documents up, how to match on meaning as well as exact words, how to reorder results — is a substantial subject, taught properly in Retrieval That Actually Retrieves in the intermediate course. For now, keep the shape in mind: search first, paste the winners, ask.
Retrieval assumes the answer sits in a passage somewhere. Plenty of data is not like that.
Where is order 88213 right now? How many are in stock? How much credit does the person typing have left? None of that lives in a document. It is a query against a live system, the answer changes hour to hour, and you cannot pre-load it because when you build the prompt you do not yet know which order the user is about to mention.
So you invert the arrangement. Instead of deciding what to include, you describe a capability — "you can look up an order by its number" — and let the model ask for it in the middle of answering. It replies with a request to use that tool, your code runs the actual lookup, you hand the result back, and the model continues with a real answer in front of it.
user : where is order 88213?
model : call lookup_order(order_id="88213")
your code: runs the query, returns
status="in transit", eta="2 Oct"
model : "It left the warehouse and is due 2 October."The model never touches your database; it asks, and your function decides whether to comply. That boundary is the whole reason this is safe enough to do.
The costs are specific. Every tool call adds a full extra round trip, so answers get slower in steps rather than smoothly. The model decides whether to call, so it sometimes does not when it should, or calls with an argument it invented. And a tool runs code and touches real data, so it deserves the same narrow scope you would give any other public entry point. How tool calls are declared, invoked and recovered from is a subject of its own in the intermediate course.
The fourth option is to change the model. Fine-tuning takes a released model and continues training it on thousands of your own examples, producing a variant whose weights have shifted toward your material.
This is the option people reach for first, because it is the one that sounds like teaching. For the problem in this lesson it is usually the wrong one, and the most expensive mistake available.
Here is the distinction that matters. Fine-tuning reliably changes behaviour: tone, format, house style, how the model draws a fuzzy classification line, a shape you cannot describe in a paragraph but can show two thousand times. It changes facts poorly. A fact pressed into weights has no source you can cite, no date, and no off switch. You cannot edit it. You can only gather new data and train again.
Bad — bakes the refund policy into a model, where it cannot be read, cited or corrected.
rows = [
{"question": q, "answer": a}
for q, a in read_faq_pairs("refund-policy.md")
]
write_jsonl("refunds.jsonl", rows)
start_fine_tune("refunds.jsonl") # hours, then evaluateGood — supplies the policy at question time, from the file that is the policy.
def answer(question: str) -> str:
policy = read_text("refund-policy.md")
return ask_model(
f"Refund policy (authoritative):\n{policy}\n\n"
f"Answer only from the policy above. If it does not\n"
f"say, reply: not covered by the policy.\n\n"
f"Question: {question}"
)The day legal changes the window from thirty days to fourteen, the second version is correct on the very next request. The first stays confidently wrong until someone notices, and fixing it means re-collecting the data and retraining — and no answer it gave came with a source anyone could have checked.
Fine-tuning is a real tool with real uses, and it has a whole lesson to itself, Fine-Tuning, Distillation, and When Not To, in the advanced course. The rule to carry out of this one: facts go in the request, not into the weights.
Five things vary across the four options, and every real argument about them is an argument about one of the five.
Free in time, paid for every call.
As current as whatever you pasted. Limited to what fits in the context window. An afternoon to build, and a recurring bill that scales with how much you send.
Scales to any corpus, adds a search.
Current with its source, small per call, and a system to build and look after — days, not hours.
Current to the second, costs a round trip.
Sometimes several round trips. The only option that reaches live state, and the slowest per answer.
Frozen the day you trained it.
Drifts further from the truth every week. Weeks of effort, needs a real evaluation to tell whether it helped, and needs redoing when the base model is retired.
Cheap per call, because the prompt stays short. That is the one column it wins.
Ask these in order and stop at the first yes.
Does it already answer this reliably?
Then supply nothing. Padding the prompt with a document it did not need makes the answer worse, not better.
Is the data small, and do you know which part?
Put it in the prompt. Do not build a retrieval system to search four files.
Is it large, so the hard part is finding the piece?
That is retrieval.
Does it depend on live state, or a named record?
That is a tool — the model asks for what it needs, mid-answer.
Is it wrong in form rather than in fact?
Right information, wrong voice, wrong structure, wrong judgement call made a thousand times a day.
Only then is fine-tuning the conversation — and it is a conversation about behaviour, never about facts.
One more thing: the four options are not multiple choice. Most real features use two or three at once — durable rules in the system prompt, retrieval for the documentation, a tool for the one live lookup. Ask these questions per piece of data, not per feature.
fresh? size latency cost/call effort
in the prompt yes small none high hours
retrieval yes any + search medium days
tool call live any + a trip medium days
fine-tuning no n/a none low weeks
Ask in order, stop at the first yes:
model already knows it? -> supply nothing
small, and you know which part? -> put it in the prompt
large, and you must find it? -> retrieval
live, or names a record? -> tool call
wrong form, not wrong facts? -> fine-tuning
Rules worth keeping:
facts go in the request, not into the weights
retrieval failure looks exactly like model failure
log what you sent before you blame what came back
every token in the prompt is paid for on every call
most features use two of these, not oneYou can now look at any AI feature and say where its facts come from — and, more usefully, notice when the answer is "nowhere, and we were hoping." That one question catches a large share of the AI features that disappoint.
Next comes Structured Output Instead of Prose, the other half of the round trip. This lesson got your data into the model in a form it can use; that one gets the answer out in a form your code can use — because a paragraph you have to parse with regular expressions is a bug waiting to happen, and asking for a schema turns a model call into an ordinary function you can validate and test.
The thing to try today takes ten minutes and no code. Pick an AI feature — yours, or one you use — and list every fact it needs to answer a real question well. Beside each, write which of the four options supplies it. The interesting rows are the ones where you have to write "nothing": those are the answers it is currently inventing, and you now know which option to reach for.