The Shape of an AI Feature
Assembling everything so far into one small end-to-end feature — input handling, prompt, schema, validation, fallback and evaluation — and seeing which parts are AI and which are ordinary software.
Assembling everything so far into one small end-to-end feature — input handling, prompt, schema, validation, fallback and evaluation — and seeing which parts are AI and which are ordinary software.
Two hundred support emails land overnight. Every morning someone spends the first hour reading them and deciding, for each one, which of five buckets it belongs in, how urgent it is, and whose queue it goes to. It is not hard work. It is just work, and it happens again tomorrow.
You now know eleven separate things about language models. This lesson spends all of them at once, on that inbox. You will build one small feature end to end — contract, input guarding, instructions, schema, the call, validation, fallback, logging, score — and then count the lines. That count is the point: the model call is nine lines in the middle of roughly a hundred and fifty, and the rest is ordinary software you already write.
The instinct is to open a chat window and start tuning wording. Resist it. Start with the feature contract: what goes in, what comes out, and what comes out when things go wrong. Everything after it — schema, validation, eval — is derived from it.
from dataclasses import dataclass
CATEGORIES = ("billing", "bug", "access", "feature", "other")
URGENCIES = ("low", "normal", "high")
@dataclass
class Triage:
category: str # one of CATEGORIES
urgency: str # one of URGENCIES
owner: str # a name from our on-call roster
summary: str # one line, 120 characters or fewer
@dataclass
class Unclear:
reason: str # what was missing from the emailTwo return types, not one. Unclear is not an error — it is an
outcome you designed on purpose, because an email that says
"it's broken, please fix" does not contain enough to route it. A
feature that must always answer will always answer, and some of
those answers will be confident nonsense.
Nothing above mentions a model, and that ordering is what keeps the model in its place: it is hired to fill in a structure you defined, not to decide what the structure is.
Before a character leaves your server the input gets three jobs done to it: reject what cannot be processed, cut what is too large, strip what must never leave the building.
Size, because a support thread is rarely one message. A customer forwards forty replies and you are billed for every token in all of them — tokens being the chunks of text a model measures and prices in — while the signal sits in the first paragraph.
Stripping matters more. Support inboxes are where people paste card numbers, passwords and one-time codes without being asked. Sending those to a vendor should be a deliberate decision, which is the whole subject of the lesson on what to never hand a model.
Bad — sends whatever arrived, at whatever length, containing whatever the customer pasted.
def triage_email(subject, body, sender_domain):
return ask_model(subject, body, sender_domain)Good — bounds and cleans the input before anything is sent anywhere.
def triage_email(subject, body, sender_domain):
body = guard_input(body)
if body is None:
return Unclear(reason="empty or unusable message")
return ask_model(subject, body, sender_domain)
def guard_input(body):
body = redact_secrets(body.strip()) # cards, keys, codes
if not body:
return None
return clip_to_token_budget(body, limit=1500)The bad version bills you for a forty-message thread to classify one sentence, and on the day a customer pastes a card number into a complaint you have handed it to a vendor and written it into your own logs. Neither is recoverable by improving the prompt.
Your instructions split by how often they change. The durable half — the job, the categories, the urgency rule, the permission to give up — is identical for all two hundred emails and belongs in the system instructions, the layer above every request. The per-request half is only this email.
SYSTEM_INSTRUCTIONS = """
You triage inbound support email for a payments company.
Choose exactly one category:
billing - invoices, charges, refunds, plan changes
bug - something worked before and does not now
access - cannot log in, locked out, lost 2FA
feature - a request for something that does not exist
other - a real message that fits none of the above
Urgency is high only when the customer is blocked from
paying us or using the product right now. Annoyed is not
high. Long is not high.
If the message does not let you choose a category with
confidence, answer with category "unclear" and say what
is missing. That is a correct answer, not a failure.
"""
def build_prompt(subject, body, sender_domain):
return (
f"Sender domain: {sender_domain}\n"
f"Subject: {subject}\n"
f"Body:\n{body}\n"
)That split is the four-part prompt from earlier in the course, rearranged by lifetime rather than topic: instruction and examples in the durable layer, context in the per-request one, and the output contract moved out of prose entirely into a schema — the next section.
The roster gets one sentence. Three names and the categories they take is nine words that change twice a year, so it lives in the durable layer: no retrieval, no tool, no lookup at request time. The decision map for getting your data to a model has a cheapest option, and this is it.
Parsing prose is how you end up with a regex that breaks the day the model writes "Category: Billing" instead of "category: billing". Ask for a schema instead — a machine-readable description of the object you want back — and the answer arrives as a dictionary.
TRIAGE_SCHEMA = {
"type": "object",
"properties": {
"category": {"enum": [*CATEGORIES, "unclear"]},
"urgency": {"enum": [*URGENCIES]},
"summary": {"type": "string", "maxLength": 120},
"reason": {"type": "string"},
},
"required": ["category"],
}Two details are doing real work. Every fixed field is an enum
rather than a string, so "Billing" and "billing issue" are not
new values you have to normalise later. And "unclear" belongs
to the category enum — the escape hatch is a legal answer inside
the shape, not a special case arriving some other way.
Then show it, because a rule stated in a paragraph competes with everything else in that paragraph. Two examples are enough, and one of them has to be an unclear:
EXAMPLES = """
Subject: charged twice for March
Body: I see two identical charges on the 3rd.
-> category=billing urgency=normal
summary=Duplicate March charge, wants one refunded.
Subject: it's broken
Body: please fix asap
-> category=unclear
reason=No product, symptom or timeframe given.
"""The second one is the important one. Without it the model has been told the escape hatch exists but has never seen anyone use it, and models weight what you show far more heavily than what you say.
Here is the whole of the model contact surface for this feature.
def ask_model(subject, body, sender_domain):
response = model.generate(
system=SYSTEM_INSTRUCTIONS + EXAMPLES,
user=build_prompt(subject, body, sender_domain),
schema=TRIAGE_SCHEMA,
temperature=0,
max_output_tokens=200,
)
return response.parsedtemperature=0 because this is a classification, not a piece of
writing. Temperature trades variety against consistency, and a
triage that puts the same email in two different queues on two
mornings is a bug you will spend a week failing to reproduce.
Turn it as low as the provider allows — and remember low is not
off. The token cap is the same instinct: you asked for four
short fields, so if the model starts writing an essay you would
rather pay for two hundred tokens than four thousand.
Now keep the first lesson of this course in mind, because it explains the next section. The model is predicting plausible next tokens. It will produce something that looks exactly like a valid triage whether or not it had any idea what the email was about. Well-formed is free. True is not.
The schema constrains the shape and says nothing about whether the content is right — and schema enforcement itself varies by provider and by mode, so write your validation as though the schema were only a strong suggestion.
Bad — takes the response at its word and puts it straight into a human's work queue.
result = ask_model(subject, body, sender_domain)
queue.assign(result["owner"], result)Good — checks every claim against facts the application owns.
result = ask_model(subject, body, sender_domain)
triage = validate(result)
if triage is None:
return Unclear(reason="response failed validation")
queue.assign(triage.owner, triage)The bad version puts a fabricated detail in front of a support agent with the authority of a system. The first anyone hears of it is a customer being told about a refund that was never issued.
The validator itself is unremarkable, and that is the point:
OWNER_FOR = {
"billing": "priya",
"bug": "marcus",
"access": "marcus",
"feature": "dani",
"other": "dani",
}
def validate(result):
category = result.get("category")
if category not in CATEGORIES:
return None
summary = (result.get("summary") or "").strip()
if not summary or len(summary) > 120:
return None
urgency = result.get("urgency")
if urgency not in URGENCIES:
urgency = "normal" # a default, not a guess
# The roster is ours, so we look the owner up instead
# of accepting whatever name we were handed.
return Triage(category, urgency,
OWNER_FOR[category], summary)Notice the owner is never asked for. It is a pure function of
the category, and a model that has never seen your staff list
will still return a confident, plausible, fictional name — the
invention the lesson on hallucination describes. The fix is not
asking more firmly; it is refusing to treat the field as
authoritative. summary is free text and cannot be defended
this way, which is why it is the one field a human reads and
nothing routes on.
Now assemble it. Every path through the feature ends in one of three places, and two of them are the same place.
def triage_email(subject, body, sender_domain):
body = guard_input(body)
if body is None:
return Unclear(reason="empty or unusable message")
try:
result = ask_model(subject, body, sender_domain)
except ModelUnavailable:
return Unclear(reason="triage service unavailable")
if result.get("category") == "unclear":
return Unclear(reason=result.get("reason", ""))
triage = validate(result)
if triage is None:
return Unclear(reason="response failed validation")
return triageFour different failures collapse into one outcome the rest of your system already handles.
The input was unusable
Empty, or nothing survived the guard. Never reaches the model at all.
The model declined
It answered unclear, which you made a legal value precisely
so it could.
The model was unreachable
A timeout, a rate limit, an outage. Ordinary software failure, handled the ordinary way.
The answer failed validation
Well-formed and wrong: a category outside your enum, an owner who is not on the roster.
Next Tuesday someone shows you a billing email routed to
feature and asks what happened. You need to reconstruct that
call, so log the identifiers rather than the content.
log.info(
"triage.decided",
email_id=email_id,
prompt_version="triage-v3",
model=model.name,
outcome=outcome, # triage | unclear | invalid
category=category,
input_tokens=usage.input_tokens,
output_tokens=usage.output_tokens,
latency_ms=elapsed_ms,
)prompt_version is the field everyone leaves out and everyone
regrets. When accuracy drops on a Thursday the question is
always "what changed", and the prompt changes more often than
the code around it. Bump that string every time you edit the
instructions, and a bad week becomes a two-minute query.
Note what is absent: the email body. You cleaned it on the way in for a reason, and copying it into a logging system undoes that work in a different building. Log the id and look it up in the support tool, where it already lives.
The previous lesson had you build an evaluation set — forty real emails with an agreed correct category. This is what it was for. Run the whole function against it, not the prompt: a validator that rejects good answers costs you exactly as much as a prompt that produces bad ones, and only an end-to-end run sees both.
def score(cases):
correct = unclear = wrong = 0
for case in cases:
out = triage_email(case.subject, case.body,
case.sender_domain)
if isinstance(out, Unclear):
unclear += 1
elif out.category == case.expected_category:
correct += 1
else:
wrong += 1
return correct, unclear, wrong40 cases: 31 correct, 6 unclear, 3 wrong
prompt triage-v3 accuracy 0.775 unclear 0.150Two numbers, deliberately. Wrong is expensive — it sends a locked-out customer to the feature-request pile. Unclear is cheap — ten seconds of someone's reading. A prompt change that turns two wrongs into two unclears has lowered your accuracy and improved your feature, and trying it by hand a few times would never have told you that.
Now count the file: roughly fifteen lines of contract, twelve of input guarding, thirty of instructions and examples, fifteen of schema, twenty-two of validation, sixteen of orchestration, twelve of logging, fifteen of scoring — and nine of model call. Eight of those nine groups are types, guards, lookups, error handling, logging and a loop, the same work as any other feature you have written.
That is the shape. A language model is a component with unusual properties: it is non-deterministic, priced by the token, and occasionally wrong with complete confidence. AI engineering is almost entirely the ordinary code you put around it to make those three properties survivable.
1 contract two return types: a result and an unclear
2 guard input reject empty, clip to a token budget,
redact what must not leave the building
3 system layer categories, urgency rule, roster, and
explicit permission to answer "unclear"
4 request layer only this email: domain, subject, body
5 schema enum every fixed field, cap free text,
make "unclear" a legal value inside it
6 examples two, and one of them is an unclear
7 the call temperature as low as it goes, output
tokens capped, nine lines total
8 validate enums re-checked, owner looked up from
your roster, length enforced, else None
9 fall back declined, unreachable and invalid all
end in the human queue
10 log id, prompt version, model, outcome,
token counts, latency - never the body
11 score run the whole function over the eval set
report correct / unclear / wrongYou can now build a small AI feature that behaves: it refuses when it should, fails into the old process, and has a number attached to it. Plenty of shipped features do not clear that bar.
It is the bar for one email at a time. The next course, Building Reliable AI Systems, takes this feature into production and fixes what it only sketched: treating context as a budget rather than a bucket, giving the model tools it can call instead of stuffing everything into the prompt, retrieval done properly with real chunking and reranking, agent loops that know how to stop, defence against prompt injection when the text you are triaging was written by a stranger, and observability for a system that never does the same thing twice. Every one is something this function would need before it ran unsupervised.
Before you move on, do one experiment. Add a sixth category to
CATEGORIES and run score three times: with only the schema
changed, then after describing the category in the system
instructions, then after adding an example of it. Watch which of
the three moves the number. That answer is worth more than any
advice about prompt wording you will read this year.