intermediate7 h 30 min15 lessons

Building Reliable AI Systems

Turning a prompt that works in a notebook into a feature that holds up in production. Context engineering, tools, skills, retrieval, agent loops, evaluation, injection defence, observability and cost — the practices that make AI systems reliable, whichever provider you use.

by DevFox

Start course

What you will cover

  1. Designing the Context WindowContext is a budget, not a bucket. What to include, in what order, and why adding more relevant material can make answers worse rather than better.30 min
  2. Tools the Model Can CallHow tool calling actually works — schema, invocation, result, continuation — and what to do about the model calling the wrong tool, the right tool with wrong arguments, or no tool at all.30 min
  3. Designing Tools a Model Can Use WellA tool description is a prompt. Naming, granularity, argument shape, error messages that teach the model to recover, and why fewer sharper tools beat a large flexible one.30 min
  4. Skills and Reusable CapabilityPackaging instructions, examples and assets into named units the model loads only when relevant. Progressive disclosure, when a skill beats a bigger system prompt, and how to keep a library of them from rotting.30 min
  5. Retrieval That Actually RetrievesWhy naive vector search disappoints: chunking that destroys meaning, embeddings that miss exact terms, and no reranking. Hybrid retrieval, chunk design, and measuring recall before you blame the model.30 min
  6. Grounding and CitationMaking an answer traceable to its source, forcing a refusal when the sources do not support one, and checking citations mechanically instead of trusting that they exist.30 min
  7. Structured Output You Can TrustSchema-constrained generation, validating at the boundary, repair loops that converge instead of looping forever, and handling partial objects while a response is still streaming.30 min
  8. The Agent LoopThink, act, observe, repeat — and the four termination conditions that stop it running away. Step budgets, progress detection, and why an agent that cannot stop is worse than one that cannot start.30 min
  9. Memory and State Across TurnsWhat to carry between turns and what to drop: rolling summaries, durable facts, retrieval over past conversation, and the failure where a stale memory outranks the truth in front of it.30 min
  10. Failure Modes, Retries, and FallbacksTimeouts, rate limits, content filters, truncated output and provider outages — which are worth retrying, which need a different model, and which must surface to the user immediately.30 min
  11. Prompt Injection and the Trust BoundaryEvery document, web page and tool result is untrusted input that reaches the same context as your instructions. Where the boundary goes, least privilege for tools, and confirming side effects with a human.30 min
  12. Evaluating an AI FeatureGolden sets, rubrics, and regression suites that run in CI. Choosing metrics that move when quality moves, and keeping an evaluation honest as the feature it measures changes.30 min
  13. Observability for Non-Deterministic SystemsTracing a request through prompts, tools and retries; logging enough to reproduce a bad answer without logging things you should not keep; and watching cost and latency as first-class signals.30 min
  14. Cost and Latency EngineeringPrompt caching, model routing, batching, streaming for perceived speed, and trimming context that earns nothing. Where the money actually goes, measured rather than assumed.30 min
  15. Shipping an AI Feature SafelyRolling out something you cannot fully predict: flags, canaries, a kill switch, a feedback path from users back into your evaluation set, and what to do the first time it embarrasses you.30 min