System design & interview — A separate rail — not about building, about being hired

This track is honestly part of the path I walked, but it is a different kind of thing to the six above: it is about communicating the work rather than doing it. Pick it up at the end, or in parallel once you have something built to talk about. Nothing here helps you if the previous tracks are empty.

Track 6 of the AI engineering roadmap by K Laxman — about 8 hours across 2 stages.

System design fundamentals, then real problems

You can hold a 45-minute design conversation without hand-waving the storage layer.

storage · scalability · networking & latency · fault tolerance · CAP · databases · Redis · Kafka · load balancers · CDNs

In short:

  1. Requirements before boxes. Start with users, scale, latency targets and what must never be lost. The numbers decide the design; the diagram comes after.
  2. Every choice is a trade-off. Consistency vs availability, cache freshness vs speed, queue durability vs latency. Name what each choice costs, not just what it gives.
  3. An LLM is a slow, costly, unreliable dependency. Design around it like any other: timeouts, caching, fallbacks, rate limits and a clear contract for its inputs and outputs.

You're done when: You can design a system that includes an LLM without the LLM becoming a magic box in the middle of the diagram.

Common traps:

Check yourself:

  1. Where does the LLM sit in your design, and what happens when it's slow? Answer: Behind a service with a timeout, a cache for repeated requests, a fallback provider and a degraded response — ideally off the synchronous path where possible.
  2. What does CAP make you choose, in practice? Answer: During a network partition, whether to reject requests (consistency) or serve possibly stale data (availability).

Talking about AI systems specifically

You can answer 'how would you build this with an LLM?' like an engineer, not an enthusiast.

cost and latency as design constraints · eval strategy as an answer · failure modes · when NOT to use a model

In short:

  1. Lead with the eval. "How would you know it works?" is the question behind every AI design question. Answer it with a dataset, a metric and a threshold.
  2. Put numbers on cost and latency. Tokens per request × requests per day × price, and p95 latency per step. Being specific is what separates experience from reading.
  3. Know when not to use a model. A rule, a regex or a classic classifier is often cheaper, faster and more reliable. Say what the model adds over that baseline.

There is no course for this one. The material is everything above: say the eval strategy out loud, name the failure modes, put a number on the latency, and know when the correct answer is a regex and a lookup table. The strongest signal you can give in an AI interview is knowing where the model does not belong.

You're done when: Asked to add AI to a product, you can describe a version that does not use a model — and say what the model buys over it.

Common traps:

Check yourself:

  1. "Add AI to our support inbox." What's the version without a model? Answer: Keyword routing rules and canned replies for the top intents. The model earns its place by handling the long tail and drafting replies — measured against that baseline.
  2. Estimate the daily cost of 100k requests at 2k input and 300 output tokens. Answer: 200M input and 30M output tokens a day, multiplied by the model's per-million prices. Say the formula out loud, then the number.

Explore more

GitHub · LinkedIn · Email