What to Never Hand a Model
Secrets, personal data, and irreversible actions: the categories that need a boundary rather than a careful prompt, plus what retention and training policies mean for data you send to a provider.
Secrets, personal data, and irreversible actions: the categories that need a boundary rather than a careful prompt, plus what retention and training policies mean for data you send to a provider.
It is ten past four and the nightly payment job has failed again. You copy the stack trace out of the log viewer, drop it into a chat window, and ask what is wrong. Twenty seconds later you have the answer, and it is a good one.
What you also pasted, without reading it, was the database connection string on line three — password included — and the two customer rows the job choked on, names and addresses and all. Nothing bad happens today, or this week, which is exactly why the habit sticks. By the end of this lesson you will have three rules you can apply on Monday, and the questions to ask a provider before their tool touches anything that matters.
Everything you have learned about prompts so far is a form of persuasion. You describe the job, you show examples, you ask for a particular shape of answer, and most of the time you get it. That is a good deal for almost everything — and a bad deal for a few things, where "most of the time" is the wrong guarantee because the failure cannot be undone.
So this lesson is about boundaries rather than prompts.
Persuasion. Works most of the time.
You describe the job, show examples, ask for a shape of answer, and usually get it.
Right for anything where the worst case is a slightly worse reply.
Enforced by something that is not the model.
Your code, the credentials you handed it, a person clicking a button. The model cannot argue with one, because it is not addressed to the model.
Right when the failure cannot be undone — "a customer's address is now in another company's log file".
Two questions drive every judgement in this lesson:
A secret is a value that is only useful because nobody else has it: an API key, a database password, a session token, the long random string in a webhook URL. Its job is to prove that whoever holds it is you. Every extra copy weakens it, and a copy in a chat log is a copy in someone else's system.
Secrets rarely travel alone, which is why this goes wrong for
careful people. Nobody sits there deciding to send a password;
they paste a config file, a docker-compose block, or an error
message that helpfully printed the full URL it tried, and the
secret rides along in line three.
The rule is easier than it sounds: the model needs the shape of your config, never the values.
Bad — puts a live key in the request body, where it lands in the provider's storage and your own chat history.
prompt = """
Here is our config. Write a curl command that calls the
billing API and retries on a 429.
BILLING_API_KEY=sk_live_x8Qv2mBnLp4TfWr9
DATABASE_URL=postgres://app:hunter2@db.internal:5432/orders
"""Good — sends the names and leaves the values where they started.
prompt = """
Here is our config, values removed. Write a curl command
that calls the billing API and retries on a 429.
BILLING_API_KEY=<read from environment>
DATABASE_URL=postgres://<user>:<pass>@<host>:5432/orders
"""The curl command you get back is identical either way. The model was never using the key — only the name of it. The first version bought you nothing and spent a live credential on it.
The same rule applies to code that builds prompts. A function that reads a config file and drops the whole thing into a prompt string sends secrets on every call, and nobody reads line three then either.
Personal data is anything that identifies a person, or can be joined with something else to identify one. The obvious members are names, emails, phone numbers, addresses and ID numbers. The forgotten ones are quieter: an order history, an IP address, a photo, any free-text box where somebody typed more about themselves than the form asked for.
Here is how these leaks begin. On Monday you paste one support ticket to see whether the model can summarise it. It can, and it is good. On Thursday you paste twenty because you are behind. The following sprint someone writes the loop — the manual version was clearly working — and four thousand customer records go to a third party every night.
Nobody decided that. The only decision anyone made was on Monday, about one record, and it looked entirely reasonable. The habit scales; the judgement that approved it does not scale with it.
There is a second reason to stop, unrelated to risk appetite: the data is not yours. A customer gave it to your company to get their order fixed, not to become training material or a line in a vendor's log.
So: redact before you send. Redaction here means replacing
real values with stable stand-ins — CUSTOMER_1, EMAIL_1 —
keeping the mapping on your own machine, and swapping them back
into the answer if you need them there.
Bad — sends the whole ticket, because the summariser "only needs the gist".
ticket = load_ticket(ticket_id)
summary = model.complete(
f"Summarise this support ticket in one sentence:\n"
f"{ticket.body}"
)Good — swaps identifiers for placeholders first and restores them locally.
ticket = load_ticket(ticket_id)
clean_body, mapping = redact_identifiers(ticket.body)
summary = model.complete(
f"Summarise this support ticket in one sentence:\n"
f"{clean_body}"
)
summary = restore_identifiers(summary, mapping)One ticket a day is a judgement call somebody can defend. The same code inside a loop is a bulk data transfer nobody reviewed — and the redacted version is the one that stays defensible when the loop arrives.
Redaction also comes with a free sanity check. If replacing
"Maria Alvarez, 4 Kingsway, Leeds" with CUSTOMER_1 changes the
answer, the model was leaning on something it had no business
leaning on, and you have found a bug rather than lost a feature.
Sort things not by how sensitive they feel, but by what recovery looks like the morning after.
Another person's medical detail, a government ID number, a signed agreement that is not public yet.
There is no reissue path, because you cannot rotate a person.
An internal roadmap, an unannounced launch date, the reasoning behind a pricing change. Recovery is a conversation with unhappy people, and it works.
Keys, passwords, tokens. Recovery is a command and an afternoon of deploys — unpleasant, finite, and yours to fix.
Which gives you a check for the two seconds before your thumb hits paste. Ask what you would do if this exact text turned up on a public page in a year. If the answer is a command you could run, proceed with care. If it is a phone call to somebody else, you are at a one-way door — redact or summarise until you are not.
Everything so far has been about data leaving. The mirror-image risk is actions leaving, and it arrives the moment you give a model tools — functions it is allowed to call, like looking up an order, searching your docs, or sending an email.
Run the same reversibility question over the verbs. Reading, searching and looking things up change nothing in the world, so they are safe to automate. Writing something a person reviews — a draft, a proposed change, a staged deployment — is safe too, because the mistake sits still and waits for you.
Then there is the other group: send, delete, pay, publish, deploy. These have no undo, and most land in front of another human being. A draft in the wrong thread is fixed before lunch. A sent message is an incident in somebody else's inbox.
The rule: put a human in front of irreversible actions, and put them there in code.
Bad — the model's decision is the last one before the mail leaves.
def send_email(to: str, subject: str, body: str) -> str:
mail.send(to=to, subject=subject, body=body)
return "sent"
tools = [send_email]Good — the model can only prepare; a person releases it.
def draft_email(to: str, subject: str, body: str) -> str:
draft = mail.create_draft(to=to, subject=subject,
body=body)
notify_reviewer(draft.id)
return f"draft {draft.id} is waiting for approval"
tools = [draft_email]On every ordinary day these two behave identically. They differ on the day the model misreads which thread it is in, or the day the message it is reading contains text aimed at it rather than at you — the trust-boundary problem the intermediate course covers — and on that day the first version has already sent the mail.
The same instinct applies to the credentials behind each tool. Give a tool the smallest one that does its job: a read-only database user for a lookup, an account that sees one mailbox rather than the whole domain, a card with a low limit. A tool that physically cannot delete needs no wording to stop it deleting.
Two words decide what happens to your data after the reply comes back, and both live in the provider's policy rather than in your prompt.
Retention is how long a copy of your request and its response stays on their systems. It is rarely zero, usually for defensible reasons — debugging, abuse monitoring, a legal obligation. Windows are commonly days or weeks, differ by product, and change.
Training is whether what you send is used as material to improve future models. One company often answers this differently across its own products: free consumer chat tools commonly default to using your conversations, paid business and developer tiers commonly default to not using them. That is a product decision, not a rule of the industry — check it per product, and again after a plan change.
Neither is fixable from inside the prompt. Writing "do not store this, do not train on this" at the top of your message is an instruction to the model, and the model is not the party doing the storing. Your text is the data the policy governs; it cannot also be the policy.
Which makes this a procurement question — settled in writing, with the company, before the tool is approved, rather than re-decided per message. It needs whoever handles contracts where you work, and the questions are short:
That last one matters more than it looks. A toggle on a settings page is real, and it is also something any account admin can flip and any product update can reset. A signed agreement is what "approved for customer data" ought to mean.
Then there is the copy nobody remembers: yours. Your application logs the prompt for debugging, your error tracker captures it in the request body, your analytics tool records someone typing it. That copy is the one you control, which makes it the one you can actually fix. Log the request id, the token counts and the latency — and if you must log text, log the redacted version.
THE TWO QUESTIONS
Can I take this back? no -> boundary, not a better prompt
Who can see it now? provider + your logs + screenshots
NEVER IN A PROMPT
API keys, passwords keep in env, send the name only
Tokens, private keys same rule; rotate if one ever leaked
Raw personal data redact first, map locally
Someone else's records you cannot rotate a person
Unreleased or legal text recovery is a phone call, not a command
REDACT BEFORE SENDING
Replace names, emails stable tokens: CUSTOMER_1, EMAIL_1
Keep the mapping local restore only on your own screen
Sanity check answer changes? it was using the name
ACTIONS: HUMAN IN FRONT
Read, search, look up safe to automate
Draft, propose, stage safe, a person releases it
Send, delete, pay, publish approval in code, never in the prompt
Tool credentials smallest one that does the job
ASK THE PROVIDER, NOT THE MODEL
Trained on by default? differs per product, re-check on change
Retention window? get a shorter one in writing if needed
Processing region? decides what you are allowed to send
Sub-processors? who else receives it
Deletion path? how it works, and how long it takes
Written agreement? a contract, not a settings toggle
YOUR OWN COPIES COUNT
Log ids, counts, latency not the prompt text
Must log text? log the redacted versionThree rules carry almost all of this. Keep secrets in the environment and send the shape of your config, never the values. Redact personal data before it leaves your machine, in the code path rather than in your intentions. Put a person in front of anything that sends, deletes, pays or publishes, in a step the model cannot reach.
Next comes Measuring Instead of Vibing, which answers the question this lesson set aside: not "is this safe to send" but "is any of it actually working". Rules keep you out of trouble and say nothing about whether your feature is good. That needs a fixed set of real inputs, an expected outcome for each, and a number you can compare between two versions of a prompt.
Before you move on, do one small thing. Open the last handful of prompts you sent this week and read them line by line, looking for a key, an email address, a name, a postcode. Then do the same to one log line your own application writes. Most people find something on the first attempt — and the finding is not the point. The point is that you now look before you paste, which is the only moment when looking still helps.