Production architecture — Where AI engineering becomes engineering

The gap between a working prototype and a system real users depend on is not model quality. It is latency, failure isolation, cost, caching and observability — ordinary distributed-systems work, applied to a component that is slow, expensive and occasionally wrong.

Track 5 of the AI engineering roadmap by K Laxman — about 6 hours across 2 stages.

A production case study worth reading twice

You can reason about an AI system the way you'd reason about any distributed system.

streaming & stateful inference · full-duplex voice · asynchronous delegation · WebRTC · latency budgets · session affinity · prompt caching · failure isolation · observability

In short:

  1. Separate the fast path from the slow path. The user-facing loop must answer within a latency budget; long work (search, tools, other agents) runs asynchronously and reports back.
  2. Stream everything. Start speaking or rendering at the first token. Time-to-first-token matters more to how fast it feels than total generation time.
  3. State has an address. Stateful sessions (voice, WebRTC) must reach the same server or share state. Session affinity and prompt caching keep them fast and cheap.

Build this — Instrument something you already built. Take the Track 3 agent. Add a latency budget per turn, a cost counter, provider failover, and a trace you can actually read when it misbehaves at 2am.

You're done when: You can draw your system's fast path and slow path as separate lines, and say what happens when the slow one stalls.

Common traps:

Check yourself:

  1. Your voice agent must look up an order (3 seconds). What does the user hear meanwhile? Answer: An immediate acknowledgement from the fast path, while the lookup runs asynchronously and its result is spoken when it arrives.
  2. Why does prompt caching matter for a long session? Answer: The long, unchanging prefix (system prompt, tools, early history) is reused instead of reprocessed, which cuts latency and input cost on every turn.

Put to work in: Jarvis — per-turn provider failover and streaming · Dialogflow CX voicebot — production conversational system

What happens when the provider is down

You can name every way your system fails and say what it does instead of failing.

timeouts · provider failover · graceful degradation · rate limits and backoff · approval gates · cost ceilings

In short:

  1. Every call needs a timeout. A provider that hangs is worse than one that errors. Set timeouts per call and treat a timeout as a failure to route around.
  2. Fail over, then degrade. Retry with backoff on rate limits, switch to a second provider on outages, and when everything fails, give a reduced answer rather than an error page.
  3. Put ceilings in code. Cost per request, calls per user and actions that need approval are enforced by the system, not requested in the prompt.

There is no resource here on purpose — I have not read one that beats the lesson. This stage is learned the day a provider deprecates a model without telling you and every call in production starts returning 404. Write down, for your own system, what happens on: a timeout, a 429, a malformed reply, a tool that hangs, and a model id that no longer exists. If any answer is 'the request just fails', that is your next afternoon's work.

Build this — Break it on purpose. Point your primary provider at a dead endpoint and run your happy path. Then rate-limit it. Then make a tool hang for sixty seconds. Fix what you find, in that order — the fixes are a timeout, a backoff, and a fallback, and they are perhaps eighty lines between them.

You're done when: You have run your own system with its primary provider unplugged, and a user still got an answer.

Common traps:

Check yourself:

  1. Your primary LLM provider returns 503 for ten minutes. What happens to a user request? Answer: It times out quickly, fails over to a second provider (tested with the same evals), and if that fails, returns a clear degraded response — not a spinner.
  2. Why add jitter to retry backoff? Answer: So thousands of clients don't retry at the same instant and knock the recovering service over again.

Put to work in: Jarvis — per-turn provider failover and streaming · OpsPilot — agents behind an approval gate

Explore more

GitHub · LinkedIn · Email