advanced8 h 10 min14 lessons

Production AI Engineering

Operating AI systems at scale and under adversarial conditions. Evaluation harnesses and their statistics, judges and their biases, multi-agent orchestration, MCP, the threat model, caching, routing, model upgrades, incident response, and the unit economics that decide whether any of it survives.

by DevFox

Start course

What you will cover

  1. An Eval Harness You Can TrustBuilding an evaluation you would bet a release on: sample size, variance across runs, statistical significance, and why a two-point score improvement is usually noise.35 min
  2. LLM-as-Judge and Its Failure ModesPosition bias, self-preference, verbosity bias and miscalibration. How to build a judge that correlates with human labels, and how to know when it has stopped.35 min
  3. Context Engineering at ScaleLong contexts degrade in ways short ones do not. Compaction, hierarchical summarisation, isolating work in sub-agents with their own context, and measuring attention loss instead of assuming it away.35 min
  4. Multi-Agent OrchestrationWhen splitting work across agents genuinely helps and when it is expensive theatre. Fan-out and verify, adversarial review, coordination failure, and the token cost nobody budgets for.35 min
  5. Model Context Protocol and Tool EcosystemsAn open standard for connecting models to tools and data: servers, clients, transports, and what changes about your threat model once a third party can define the tools your agent sees.35 min
  6. The AI Threat ModelIndirect injection chains, exfiltration through tool arguments, the confused deputy, and a supply chain that now includes prompts and skills. Defence in depth for systems that will be attacked through their inputs.35 min
  7. Guardrails and Policy EnforcementInput and output filters, allowlists, and classifiers — where each belongs in the stack, what they genuinely stop, and why a guardrail that runs on the same model it guards is not a control.35 min
  8. Caching Across a Model StackPrompt caching and its layout rules, semantic caching and its correctness risk, and invalidation in a system where the same input may legitimately produce a different answer tomorrow.35 min
  9. Fine-Tuning, Distillation, and When Not ToA decision framework: what fine-tuning fixes, what it never fixes, the data and evaluation it demands, and the maintenance cost that arrives the day the base model is deprecated.35 min
  10. Routing and Model PortfoliosCascades, confidence signals and cheap-first strategies; abstracting over providers without abstracting away the differences that matter; and measuring whether routing actually saved anything.35 min
  11. Drift, Regression, and Model UpgradesWhy a better model breaks your prompts, how to pin and how to move, shadow traffic, and an upgrade playbook that turns a scary migration into a measured one.35 min
  12. Incidents in Non-Deterministic SystemsResponding when there is no stack trace: reproducing from traces, replaying a request, rolling back a prompt, and writing a postmortem whose root cause is a probability rather than a line of code.35 min
  13. Human-in-the-Loop DesignApproval gates, confidence thresholds and escalation paths that fit real work. Designing against reviewer fatigue, where rubber-stamping makes the human a liability rather than a control.35 min
  14. The Economics of an AI ProductUnit economics per request, margin as a design constraint, abuse and capacity planning, and how cost pressure quietly reshapes architecture whether or not you plan for it.35 min