AI Engineer, interview-ready — Build what AI engineering interviews ask about, then learn to explain it
The route I would take if the goal were an AI engineering role rather than a side project. It covers the systems these interviews keep returning to — retrieval, agents, evaluation and running things in production — and finishes on the system-design rail, because a build you cannot explain on a whiteboard does not help you in the room.
Who this is for: You want to be ready to interview for an AI engineering role.
6 tracks · 52 hours · one of 4 paths in the learning roadmap by K Laxman. An optional 4-hour prelude, Just enough to not be bluffing, covers the underlying ideas first.
Track 1 — LLM fundamentals
Understand the thing everything else sits on (~6 hours)
If you only ever do one item on this page, do the first one. Almost every confused conversation I have about AI systems traces back to someone never having watched this.
The whole pipeline, end to end
You can trace a token from raw internet text all the way to a chat reply, naming every stage.
pretraining · tokenisation · transformers · inference · SFT · RLHF · hallucination · tool use · reasoning models
In short:
- Pretraining makes a document completer. Trained to predict the next token over a huge slice of the internet, the base model continues text. It does not follow instructions yet.
- Post-training makes an assistant. Supervised fine-tuning on example conversations, then reinforcement learning from human (or AI) feedback, turns the completer into something that answers.
- Hallucination is the default. The model always produces a fluent continuation. Unless training taught it to recognise what it doesn't know, a confident guess is the most likely text.
- Deep Dive into LLMs like ChatGPT — Andrej Karpathy (video, free, ~3.5h, rated 10/10). Do this first. Nothing else on the internet takes you from raw pretraining data to RLHF and reasoning models in one honest, unhurried pass — including why hallucination happens rather than just that it does.
- Attention Is All You Need — Vaswani et al., 2017 (paper, free, ~1.5h, rated 9/10). Read it AFTER the video, not before. It is eight pages and it will feel obvious, which is exactly the point at which you have understood it.
Build this — Re-implement decoding. Take any provider that returns logprobs and implement greedy, temperature and top-p sampling yourself. Write a README explaining, in your own words, when you'd use each. It is thirty lines of code and it makes §0.5 permanent.
You're done when: You can explain, without notes, why a model hallucinates confidently rather than saying 'I don't know'.
Common traps:
- Treating the model as a database. It stores statistical patterns, not records — look facts up and put them in the context.
- Assuming a reasoning model is always the better choice. It is slower and costlier; many tasks don't need it.
Check yourself:
- Why does a model hallucinate confidently instead of saying "I don't know"? Answer: It was trained to produce likely text, and in its training data questions are usually followed by confident answers. Saying "I don't know" has to be specifically trained in, and it only works for things the model can tell it doesn't know.
- What is the difference between a base model and an assistant model? Answer: The base model continues documents. The assistant is the same network after SFT and RLHF, trained to respond to instructions in a conversation format.
Put to work in: Jarvis — six interchangeable providers behind one loop · CAMSCode — enterprise AI coding assistant
Getting output your code can actually parse
You can make a model return JSON your program can trust, and you know what to do the one call in a thousand where it doesn't.
JSON schema · constrained decoding · refusals · repair passes · streaming partials
In short:
- Ask for a schema, not "JSON please". Provider structured-output modes constrain decoding to your JSON schema, so the reply is valid JSON of the right shape — not just valid-looking text.
- Valid is not correct. The schema guarantees shape. It says nothing about whether the values are true. Validate the business rules in your own code.
- Plan for the one in a thousand. Refusals, truncation at the token limit and schema features the provider doesn't support all still happen. Parse, and on failure repair once or fail cleanly.
- Structured Outputs — OpenAI (docs, free, ~0.5h, rated 8/10). Short, and the only one of these guides that is honest about the failure cases — refusals, and what a schema cannot express. Read it even if you use a different provider; the shape of the problem is the same everywhere.
Build this — A parse-or-repair loop. Ask for a schema, parse the reply, and on a parse failure send the error text back for one repair attempt before giving up. Log how often the repair fires. That number is the difference between a demo and something you can put in front of a customer.
You're done when: You can say what your system does when the model returns prose instead of JSON — and it isn't 'crash'.
Common traps:
- Parsing with a regex over free text. Use the provider's structured mode, and a real JSON parser.
- Setting max tokens too low for the output. A truncated object is invalid JSON, every time.
Check yourself:
- The model returns prose instead of JSON. What does your system do? Answer: Catches the parse error, sends the error text back for one repair attempt, and if that also fails, returns a clear error or fallback — logged and counted.
- The JSON parses and matches the schema. What can still be wrong? Answer: The values. A date can be in the wrong year, a total can fail to add up. Schema validation checks shape; your own checks cover meaning.
Put to work in: CAMSCode — enterprise AI coding assistant · Intelligent document processing
Track 2 — Retrieval-Augmented Generation
The part most people do badly (~10 hours)
Everyone can build a RAG demo in an afternoon. Almost nobody can tell you why theirs returns the wrong chunk. The difference between those two people is this track — and the eval set at the end of it.
RAG properly — all fifteen techniques
You can name, and choose between, the retrieval strategies. Not just 'embed, store, search'.
indexing · retrieval · generation · multi-query · RAG-Fusion · query decomposition · step-back prompting · HyDE · routing · query construction · multi-representation indexing · RAPTOR · ColBERT · CRAG · adaptive RAG
In short:
- Three stages, three failure points. Indexing (split and embed), retrieval (find the right chunks) and generation (answer from them). A wrong answer came from exactly one of them.
- Most techniques fix the query or the index. Multi-query, HyDE, step-back and decomposition rewrite the question; multi-representation indexing and RAPTOR change what gets stored. Each fixes a specific miss.
- Corrective and adaptive RAG add judgement. Grade the retrieved chunks before answering, and re-retrieve or fall back when they are weak, instead of generating from whatever came back.
- Learn RAG From Scratch — LangChain / freeCodeCamp (video, free, ~2.2h, rated 10/10). The reason this beats every 'build a RAG bot in 10 minutes' video is the list of concepts on the left. It goes well past embed-and-retrieve into routing, decomposition, HyDE, RAPTOR and CRAG — the techniques you reach for when the naive version is returning garbage.
- langchain-ai/rag-from-scratch — LangChain (repo, free, ~4h, rated 9/10). Run every notebook. Watching RAG is not learning RAG — the chunking decisions only become real when your own document set makes them go wrong.
Build this — RAG over your own notes — plus a 30-question eval set. The bot is the easy half. The assignment is the eval set: thirty real questions with known answers, and a score you can quote. Without it you have a demo, and you will not be able to tell whether tomorrow's change helped.
You're done when: Given a wrong answer, you can separate retrieval failure from generation failure in under a minute.
Common traps:
- Stacking techniques before measuring. Every one adds latency and cost; add one only when a measured failure calls for it.
- Blaming the model when the right chunk was never retrieved. Look at what was retrieved first.
Check yourself:
- An answer is wrong. What is the one-minute test that tells you where? Answer: Look at the retrieved chunks. If the answer isn't in them, it's a retrieval failure. If it is and the model still got it wrong, it's a generation failure.
- When does HyDE help? Answer: When questions are short and phrased very differently from the documents. Embedding a hypothetical answer lands closer to real answers than the bare question does.
Put to work in: CAMS Lens — SEBI-grounded compliance intelligence · Agentic Research & Decision Intelligence Assistant
Chunking — the decision that quietly decides everything
You can explain why your chunk size is what it is, instead of because a tutorial said 512.
fixed vs semantic splitting · overlap · chunk size vs recall · metadata on the chunk · contextual retrieval
Interactive explainer on the live page: context budget — You can explain why your chunk size is what it is, instead of because a tutorial said 512.
In short:
- The chunk is the unit of retrieval. You can only retrieve what you stored. If the answer is split across two chunks, or buried in a huge one, retrieval can't deliver it cleanly.
- Size is a trade-off. Small chunks match precisely but lose context; large chunks keep context but dilute the embedding. Overlap reduces boundary losses at the cost of duplication.
- Give each chunk its context. Prepending the document title, section heading or a short generated summary (contextual retrieval) makes a chunk findable on its own.
- Introducing Contextual Retrieval — Anthropic (article, free, ~0.75h, rated 9/10). The single change with the best effort-to-payoff ratio in RAG: prepend a sentence of document context to each chunk before embedding it. Measured, with the failure rate reported rather than a vibe. Read this the day your naive retrieval starts missing.
- Chunking Strategies for LLM Applications — Pinecone (article, free, ~0.5h, rated 7/10). Vendor content, and worth it anyway — it is the clearest side-by-side of the splitting strategies. Take the taxonomy, ignore the product.
Build this — Same questions, four chunkings. Take the eval set from the stage above and run it against four indexes: 256 fixed, 512 fixed, 512 with overlap, and semantic. One table, four scores. You will never again accept a chunk size on faith, and you now own a number you can quote in an interview.
You're done when: You can name the chunk size you use, the one you tried before it, and the score difference between them.
Common traps:
- Choosing a chunk size once and never testing it. It is a parameter to measure on your own eval set.
- Splitting through tables and code blocks. Split on the document's own structure — headings, paragraphs, rows — before falling back to character counts.
Check yourself:
- Your chunks say "the policy above applies" and retrieval returns them useless. What is the fix? Answer: Add context to each chunk — document and section titles, or a generated one-line summary — so it stands on its own when retrieved.
- How do you choose between 300-token and 1,000-token chunks? Answer: Index both and run the same questions against each. Compare recall@k and answer quality; pick with numbers, not intuition.
Put to work in: CAMS Lens — SEBI-grounded compliance intelligence
When retrieval is the bug
You can tell a retrieval problem from a generation problem before you touch the prompt.
hybrid search · reranking · recall@k · query rewriting · the missing-document case
In short:
- Measure recall@k first. For a set of questions with known source documents, how often is the right one in the top k results? Until you know that, prompt tweaks are guesses.
- Hybrid search covers both kinds of miss. Keyword search (BM25) catches exact terms and identifiers; vectors catch paraphrase. Combining them fixes misses either one makes alone.
- Rerank a wide net. Retrieve 50 cheaply, then let a cross-encoder reranker pick the best 5. The reranker reads query and chunk together, which is far more precise.
- Rerank — overview — Cohere (docs, free, ~0.5h, rated 8/10). Retrieve twenty, rerank, keep four. It is one extra call and it is the cheapest quality jump available to you — usually larger than swapping the generation model, which is where people reach first.
Build this — Measure recall@k before you tune anything. For each question in your eval set, record whether the answer-bearing chunk appeared in the top k at all. If it didn't, no prompt on earth was going to save that answer. Split your failures into 'never retrieved' and 'retrieved and ignored' — they have completely different fixes.
You're done when: You have a recall@k number for your own corpus, and you fixed retrieval before you touched the prompt.
Common traps:
- Tuning the prompt to fix answers whose source was never retrieved.
- Not handling the missing-document case. When nothing relevant exists, the system should say so, not answer from the nearest irrelevant chunk.
Check yourself:
- Users search for error code "E4031" and get nothing useful. Why, and what fixes it? Answer: Embeddings don't represent exact identifiers well. Add keyword search (hybrid) so exact matches get found.
- Recall@5 is 60% but recall@50 is 95%. What do you add? Answer: A reranker. The right documents are being found; they just aren't ranked high enough. Rerank the top 50 down to 5.
Put to work in: CAMS Lens — SEBI-grounded compliance intelligence · Agentic Research & Decision Intelligence Assistant
Track 3 — Agents & tools
A model, in a loop, with tools. That's it. (~17 hours)
This is the track I care most about, because it is the one where the industry's marketing is furthest from the engineering. An agent is not a mysterious autonomous intelligence. It is a while loop. Write one by hand and the mystique never comes back — and then build the harness around it, which is where the actual work has moved.
The agent loop, by hand
You can write an agent without a framework and explain every line of it.
environment · tool schemas · the while loop · context accumulation · approval gates
Interactive explainer on the live page: agent loop — You can write an agent without a framework and explain every line of it.
In short:
- An agent is a while loop. Call the model; if it asks for a tool, run it, append the result to the conversation, and call again. Stop when it answers without a tool call.
- The context is the agent's memory. Every tool result goes back into the context. The model knows only what the loop has put there so far.
- You own the stopping conditions. A step limit, a cost ceiling and approval gates for risky actions belong in your code, not in the prompt.
- How We Build Effective Agents — Barry Zhang · Anthropic (video, free, ~0.5h, rated 10/10). Kills the mystique in twenty minutes. An agent is a model acting on an environment through tools, in a loop — environment, tools, system prompt, while True. Everything else in the ecosystem is a wrapper around those four lines.
- Building effective agents — Erik Schluntz & Barry Zhang · Anthropic (article, free, ~0.5h, rated 10/10). The written companion to the talk, and the most-cited page in agent engineering for a reason. It draws the line between workflows (you write the path) and agents (the model picks it), and its core advice — start with plain API calls, add complexity only when it measurably helps — is the whole philosophy of this track.
- Tips for building AI agents — Barry Zhang, Erik Schluntz & Alex Albert · Anthropic (video, free, ~0.3h, rated 8/10). Eighteen minutes of the same people arguing about where agents are over-hyped and where they genuinely earn their cost. Worth it for the workflow-vs-agent judgement calls, which no article captures as well as hearing them disagree.
- Building agents with the Claude Agent SDK — Thariq Shihipar · Anthropic (article, free, ~0.3h, rated 8/10). Names the loop you are about to write: gather context, take action, verify the work, repeat. Read it for the verify step — rules-based checks, visual checks, a model as judge — because that is the part hand-written loops forget.
Build this — Write the loop. No framework.. A while loop, three tools, JSON-schema arguments, one approval gate before anything destructive. This is the keystone exercise of the whole page — do not substitute a framework tutorial for it.
You're done when: You can draw the loop on a whiteboard and point at exactly where a tool result re-enters the context.
Common traps:
- No iteration cap. A confused agent will call the same tool forever and bill you for each turn.
- Reaching for an agent when the steps are known in advance. A fixed workflow is cheaper, faster and easier to test.
Check yourself:
- Where exactly does a tool result re-enter the loop? Answer: It is appended to the message history as a tool-result message, and the whole history is sent on the next model call.
- When should you NOT build an agent? Answer: When you can write the steps down in advance. Use a fixed pipeline, and keep agents for tasks where the path genuinely depends on what is found along the way.
Put to work in: Jarvis — that loop, in production, running live in your browser · OpsPilot — agentic request triage & remediation
Workflows and graph loops — the five patterns
You can pick the simplest pattern that solves a task, and draw any agent system as a graph of steps with a loop in it.
prompt chaining · routing · parallelization · orchestrator-workers · evaluator-optimizer · state, nodes and edges · conditional edges and cycles · checkpointing
In short:
- Five patterns cover most systems. Prompt chaining (fixed steps), routing (classify, then specialise), parallelization (split or vote), orchestrator-workers (a model plans the subtasks) and evaluator-optimizer (generate, critique, revise).
- A graph is the loop, drawn. Nodes are steps, edges are control flow, and state is what travels between them. An agent is a graph with a conditional edge pointing back to an earlier node.
- Every cycle needs an exit. A loop edge must have a condition that ends it — a passing check, a round limit, a budget. Without one, the graph is an infinite loop with a bill attached.
- Agent patterns — runnable notebooks — Anthropic · claude-cookbooks (repo, free, ~2h, rated 9/10). The five patterns from 'Building effective agents' as short notebooks: basic workflows (chaining, routing, parallelization), orchestrator-workers and evaluator-optimizer, plus multi-agent latency experiments. Plain Python and the API — run them before you ever see a graph library.
- Workflows and agents — LangChain · LangGraph docs (docs, free, ~0.75h, rated 8/10). The same patterns again, written as graphs: nodes for steps, edges for control flow, and a conditional edge that points backwards — which is all a 'graph loop' is. Read it after the notebooks and the mapping from loop to graph becomes obvious.
Build this — One task, two patterns, a number for each. Take a writing or extraction task. Solve it once as a prompt chain and once as an evaluator-optimizer loop (generate → critique → revise, with a round limit). Run both on the same 20 inputs and record quality, latency and cost. Then draw the second one as a graph: which edge is the loop, and what stops it?
You're done when: Given a new task, you can say which pattern fits, why the simpler ones don't, and where the loop's exit condition lives.
Common traps:
- Using an autonomous agent where a fixed chain would do. Start with the simplest pattern and move up only when a measured failure demands it.
- An evaluator-optimizer loop with a vague critique prompt. If the evaluator can't say what to fix, revision just rewrites the same mistakes.
Check yourself:
- Customer emails go to billing, technical or sales, each with its own prompt. Which pattern is that? Answer: Routing — classify the input, then send it to a specialised prompt or model.
- What turns a workflow graph into an agent? Answer: A cycle whose next step the model decides: a conditional edge back to the model node, taken until the model chooses to stop (or a limit stops it).
Put to work in: Research Intelligence Agent — orchestration in practice · OpsPilot — routing and remediation workflows
Tools and MCP — connecting agents to real systems
You can design a tool an LLM actually uses correctly, and expose it over MCP.
tool design · argument validation · MCP architecture · tool servers · when a tool is too broad
In short:
- The tool description is a prompt. The model chooses tools and fills arguments from the name, description and schema alone. Write them for a new colleague with no other context.
- Narrow tools, typed arguments. One clear job per tool, enums instead of free strings where possible, and validation on every call — the model's arguments are untrusted input.
- MCP standardises the plumbing. An MCP server exposes tools and resources once; any MCP-capable client (desktop apps, IDEs, your own agent) can then use them without custom glue.
- Building Agents with Model Context Protocol — Mahesh Murag · Anthropic (video, free, ~1.75h, rated 9/10). The full workshop: what MCP is, building with it, MCP with agents, the architecture, and where it is going. Watch it once you have hand-written a tool call, so you can see what the protocol is standardising.
- Model Context Protocol — getting started — MCP (docs, free, ~0.5h, rated 8/10). The spec's own introduction, and the page to have open while you write the server rather than while you watch the workshop. Skim the concepts, then go straight to the transport section — that is the part everyone gets wrong first.
- Tool use — overview — Anthropic (docs, free, ~0.75h, rated 9/10). The best writing anywhere on how to describe a tool so the model picks it correctly. The description field is prompt engineering, and this is the page that finally made me treat it that way.
- Writing effective tools for agents — with agents — Anthropic (article, free, ~0.4h, rated 9/10). The best single guide to tool design. Fewer, more purposeful tools; namespaced names; human-readable results instead of UUIDs; pagination and truncation by default — and, the real lesson, evaluate tools like any other component and let the model help rewrite their descriptions.
- Code execution with MCP — Anthropic (article, free, ~0.3h, rated 8/10). What to do when you have hundreds of tools: let the agent discover them as files and call them from code, filtering data before it reaches the context. Their example goes from roughly 150,000 tokens of tool definitions to 2,000.
- Don't Build Agents, Build Skills Instead — Barry Zhang & Mahesh Murag · Anthropic (video, free, ~0.3h, rated 8/10). Sixteen minutes on Agent Skills: folders of instructions and scripts an agent loads only when a task needs them. The idea worth keeping even if you never use Claude is progressive disclosure — context loaded in layers, not all up front.
Build this — Wrap your Track 2 RAG bot as an MCP server. One tool, a real schema, connected to a client you didn't write. The moment someone else's agent can call your thing, tool design stops being theoretical.
You're done when: You can say why a tool named `do_stuff(input: string)` will be called wrongly, and rewrite it so it won't be.
Common traps:
- Giving an agent a tool that can do anything, like a raw SQL or shell executor, without a permission boundary.
- Returning huge, unfiltered payloads from tools. The result lands in the context window; return what the model needs.
Check yourself:
- Why will `do_stuff(input: string)` get called wrongly? Answer: The name and schema tell the model nothing about when to call it or what to pass. Split it into specific tools with typed, described parameters.
- What does MCP give you that a function call in your own code doesn't? Answer: Reuse: the same server works across every MCP client, and tools can be added or updated without changing the agent's code.
Put to work in: Intelligent Document Processing — multi-agent system
Harness engineering — the part that isn't the model
You can say what a harness is, name its five pieces, and point at the one your own agent is missing.
guides vs sensors · context assembly · tool orchestration · verification loops · cost envelopes · behavioural evals
Interactive explainer on the live page: harness — You can say what a harness is, name its five pieces, and point at the one your own agent is missing.
In short:
- The harness is everything but the model. Context assembly, tool orchestration, verification, limits and logging. It decides what the model sees and what happens to what it produces.
- Guides steer, sensors check. Guides (instructions, examples, conventions) shape behaviour before the action; sensors (tests, linters, validators) check the result after it.
- Close the loop with verification. Let the agent run the check itself — the test suite, the type checker — and feed failures back. That catches errors no prompt prevents.
The shift this track has been building toward. Prompt engineering was how you got one good answer out of a 4k context window; it stopped scaling the moment a task ran longer than a few hundred lines. What replaced it is not a better prompt — it is everything around the model: what you put in the context, which tools it can reach, what checks its work, what stops it spending your money, and what you can read afterwards when it went wrong. Swapping models moves output quality by ten or fifteen per cent. Changing the harness decides whether the system works at all.
- Effective harnesses for long-running agents — Justin Young · Anthropic (article, free, ~0.2h, rated 10/10). Short and concrete: how an agent keeps making progress across many context windows. An initializer session sets up the repo and a feature list marked pass/fail; every later session reads a progress file and git history, does one feature, tests it end to end, commits. Copy this structure.
- Harness design for long-running application development — Prithvi Rajasekaran · Anthropic (article, free, ~0.4h, rated 9/10). The follow-up: planner, generator and evaluator agents, with the evaluator testing the running app in a browser and the two negotiating what 'done' means before work starts. Read to the end — they removed scaffolding once a newer model no longer needed it, which is the lesson.
- Effective context engineering for AI agents — Anthropic (article, free, ~0.4h, rated 9/10). Context assembly is the first part of any harness, and this is the reference for it: compaction near the limit, structured notes outside the window, sub-agents that return short summaries, and loading data just in time by path or query instead of up front.
- Scaling Managed Agents: decoupling the brain from the hands — Anthropic (article, free, ~0.3h, rated 8/10). How Anthropic structures a hosted agent service: a stateless harness (the brain), replaceable sandboxes and tools (the hands), and a durable session log outside the context window. Useful architecture vocabulary once your harness has to survive restarts.
- Prompting for Agents — Hannah Moran & Jeremy Hadfield · Anthropic (video, free, ~0.5h, rated 8/10). Half an hour on writing the guides part of a harness: how to prompt a model that runs for many turns with tools, where short-prompt habits stop working, and how to read transcripts to find out why an agent went wrong.
- Claude Agent SDK — full workshop — Thariq Shihipar · Anthropic (video, free, ~1.9h, rated 8/10). A two-hour build-along using the harness behind Claude Code as a library: tools, subagents, permissions and verification wired up live. Do it after writing your own harness, so you recognise every piece rather than copying it.
- What is harness engineering? — harness-engineering.ai (article, free, ~0.4h, rated 9/10). The cleanest definition going: everything between user input and agent output except the model itself, broken into five parts — context engineering, tool orchestration, verification loops, cost envelopes, observability. Read this one first and the other four stop sounding like different topics.
- Harness engineering: leveraging Codex in an agent-first world — OpenAI (article, free, ~0.5h, rated 9/10). Five months, a production beta, about a million lines, and no human typing any of them — every PR opened and reviewed by Codex agents. Read it for the admission at the end rather than the number: with the writing automated, their hard problems became designing environments, feedback loops and control systems. That is the job description this stage is about.
- Harness engineering for coding agent users — Martin Fowler / Thoughtworks (article, free, ~0.75h, rated 10/10). The one that gives you the vocabulary to design with: guides (architecture docs, convention files, bootstrap scripts) steer before generation, sensors (linters, type checks, structural tests, review agents) catch after it, and each is either computational — deterministic, milliseconds — or inferential, an LLM judging. It also says the quiet part: a good harness is not trying to remove you, it is trying to spend your attention where it actually matters.
- The anatomy of harness engineering — evaluate, iterate and guard AI coding agents — Google Developers Blog (article, free, ~0.5h, rated 8/10). How you keep a harness honest as models move under you: behavioural evals that test discrete observable actions rather than end-to-end outcomes, strict assertions where the task is simple and an LLM judge where it isn't, run in batches so you watch the trend instead of panicking at one bad run. Integration tests for your agent.
- Learn Harness Engineering — Walking Labs (course, free, ~6h, rated 8/10). Free and open source: fourteen lectures from agent failure modes through to state management and control systems, projects to build against, copy-ready AGENTS.md and feature-list templates, and teardowns of real harnesses including Claude Code and Codex. Use it as the reference you come back to, not a sitting.
- Beyond the prompt — why harness engineering is the lever — PALO IT (article, free, ~0.3h, rated 7/10). The shortest version of the argument, plus an order to build in: conventions file, then deterministic hooks for the rules that must not be reasoned about, then subagents as independent verifiers, then state on disk, and only then an automated retry loop. Its warning is the one to keep — a loop without sensors produces hallucinated completion at high speed.
Build this — Put a harness around the loop you already wrote. Take your Track 3 agent and add the five pieces one at a time, measuring after each: a conventions file it always reads, a deterministic check it cannot talk its way past, one verification gate that re-runs its work before the result counts, a per-task token ceiling that kills the run, and a trace you can read afterwards. Keep the model fixed the whole time — the point is to see how far the harness alone moves it.
You're done when: You can point at one failure your harness now catches that the model used to get away with — and say which of the five parts caught it.
Common traps:
- Answering every failure with a longer system prompt. If a check can catch it, add a sensor instead.
- No cost envelope. Long-running agents need a budget per task, enforced in code.
- Letting a long-running agent decide it's finished. Keep an external feature list marked pass/fail by tests, and a progress file each new session reads first.
Check yourself:
- Your coding agent keeps breaking imports. Guide or sensor? Answer: Both, but the sensor matters more: run the build or type check after each edit and feed errors back. A prompt rule alone will be ignored eventually.
- Name the five parts of the harness in the explainer above. Answer: Context (what the model sees), tools (what it can do), verify (checks on what it produced), budget (steps and cost it may spend), and trace (a record of every run).
Put to work in: Jarvis — tool allow-lists, call budgets and a visible trace
Multi-agent systems — when one context window isn't enough
You can tell when splitting work across agents pays for itself, and build an orchestrator whose sub-agents don't trip over each other.
orchestrator and sub-agents · parallel context windows · token cost vs quality · task locking · shared state via git or files · outcome-based evaluation
In short:
- Sub-agents buy context, not intelligence. Each sub-agent explores in its own clean context window and returns a short summary. The win is breadth and parallelism on tasks too big for one window.
- It costs a multiple of the tokens. Anthropic reports multi-agent research using roughly 15× the tokens of a chat. It pays off only when the task is valuable and genuinely splits into independent parts.
- Coordination is the hard part. Agents need clear task boundaries, a way to claim work (a lock file, a queue) and shared state they can merge (git, files). Most failures are two agents doing the same thing.
- How we built our multi-agent research system — Anthropic (article, free, ~0.4h, rated 10/10). The production write-up behind Claude's Research feature. A lead agent spawns parallel sub-agents, each with its own context window. The numbers are the point: roughly 15× the tokens of a chat, a large gain on broad research tasks, and token usage explaining most of the difference in quality. Read the prompting and evaluation lessons twice.
- Building a C compiler with a team of parallel Claudes — Nicholas Carlini · Anthropic (article, free, ~0.4h, rated 9/10). Sixteen agents, one repository, about 2,000 sessions: a Rust C compiler that builds the Linux kernel. The coordination is refreshingly plain — lock files to claim tasks, git to merge — and the real lesson is that high-quality tests are what let autonomous agents make progress at all.
Build this — An orchestrator with three researchers. Build a lead agent that splits a research question into three sub-questions, runs three sub-agents in parallel with separate contexts, and merges their cited summaries. Compare against a single agent on 10 questions: quality (judged against your own rubric), tokens and wall-clock time. Keep the multi-agent version only if the numbers justify the cost.
You're done when: You can quote what your multi-agent version costs in tokens compared with one agent, and what it buys.
Common traps:
- Splitting tightly coupled work, like one feature across many files, between agents that can't see each other's changes.
- Evaluating whether each agent followed the expected steps. Judge the final outcome — agents take different valid paths.
Check yourself:
- When is a single agent the better choice? Answer: When the subtasks depend on each other, share most of their context, or the task fits in one window. Parallel agents add cost and coordination for no gain there.
- Two parallel agents keep fixing the same bug. What do you add? Answer: A way to claim tasks — a lock file or queue entry per task — checked before starting work, plus merging shared state often.
Put to work in: Research Intelligence Agent — agentic research & decision support
Frameworks — deliberately last
You choose a framework because it removes work you understand, not because you're lost.
LangGraph and friends · what they abstract · what they hide
In short:
- A framework hides the loop. LangGraph and similar libraries give you state, retries, persistence and graphs of steps. They are the same loop you wrote, with decisions made for you.
- Adopt for features, not for learning. Durable state, human-in-the-loop pauses and tracing are real reasons. "I don't know how agents work" is not — that is what the previous stages fixed.
- Know the escape hatch. Before committing, find out how to see the exact prompt sent and how to override one step. Debugging something you can't see is the real cost.
Framework material comes last on purpose. Read one only after you have written the loop, the patterns and a harness yourself. You will find you already know what every abstraction is for — and you will notice the ones that are solving a problem you don't have.
- Introduction to LangGraph — LangChain Academy (course, free, ~6h, rated 8/10). If you choose a graph framework, this is the free, structured way in: state, nodes, edges and cycles, then memory, human-in-the-loop interrupts, checkpointing and deployment. The parts worth paying attention to are persistence and interrupts — those are the features that are genuinely tedious to build yourself.
You're done when: You can name one thing your chosen framework does that you would rather do yourself, and say why.
Common traps:
- Adopting a framework and never looking at the raw prompts it sends.
- Choosing by GitHub stars. Choose by the two or three features you actually need.
Check yourself:
- Name one good reason to adopt a framework. Answer: Durable, resumable state for long workflows, or built-in human-approval pauses — features that are genuinely tedious to build and test yourself.
- What should you check in your framework before production? Answer: That you can see and log the exact prompt and tool calls for any run, and override any single step without forking the library.
Track 4 — Evals, tracing & observability
The stage everyone skips (~5 hours)
This is the one that separates a demo from a product. As a system gets complicated, evals and tracing stop being nice-to-have and become the only way you can change anything without breaking it. If you cannot state your system's accuracy as a number with a dataset behind it, you do not have a system — you have a vibe.
Dataset → evaluator → task → interpretation
You can build an evaluation harness before you build the feature.
datasets · evaluators · LLM-as-judge and its failure modes · tracing · regression detection
In short:
- An eval is a dataset plus a grader. Inputs with expected outputs or criteria, run through your system, scored the same way every time. Without one, every change is a guess.
- Start small and real. Thirty real questions with checked answers beat a thousand synthetic ones. Add every production failure to the set as you find it.
- Trace everything. Record each run's prompts, retrieved chunks, tool calls and outputs. When a score drops, the trace shows which step changed.
- Why Evals Matter — LangSmith Evaluations, Part 1 — LangChain (video, free, ~0.4h, rated 10/10). Four pieces — dataset, evaluator, task, interpreting the result — and once you have that frame, every other eval tool is just an implementation of it. Part 1 is mandatory; carry on through the series as far as your patience holds.
- Demystifying evals for AI agents — Anthropic (article, free, ~0.3h, rated 9/10). The vocabulary for evaluating agents rather than single answers: tasks and trials, code-based, model-based and human graders, transcripts versus final outcomes, and the difference between pass@k (succeeds at least once) and pass^k (succeeds every time) — the one production cares about.
Build this — Put a number on the RAG bot. Add the eval set and tracing to what you built in Track 2. Report the score in the README. Then change the chunk size and report it again — that delta is the whole discipline.
You're done when: You can say what your system's accuracy is, with a number and a dataset behind it.
Common traps:
- Judging a change by trying five prompts by hand. You will remember the wins and forget the regressions.
- Only measuring the final answer in a multi-step system. Score the retrieval and tool steps too, or you can't tell where it broke.
Check yourself:
- "The new prompt feels better." What do you ask for? Answer: The score on the eval set before and after, on the same data, and the examples that got worse.
- Where should new eval examples come from once you're live? Answer: Production failures — user reports, low ratings and traced errors — added to the dataset so they can't regress silently.
Put to work in: CAMS Lens — grounded answers that have to be right
LLM-as-judge — and where it stops being trustworthy
You can grade an open-ended answer automatically, and you know which of those grades to disbelieve.
pairwise vs pointwise · position bias · agreement with humans · error analysis · when to just read the outputs
In short:
- A judge is a model you must evaluate. Label a sample yourself and measure how often the judge agrees. Until you have that number, the judge's scores are unverified.
- Pairwise is steadier than pointwise. "Which of these two is better?" is more reliable than "rate this 1–10". Swap the order and check again to cancel position bias.
- Binary criteria beat vague scales. "Does the answer cite a source from the context? yes/no" is far easier to judge consistently than "rate helpfulness".
- Creating a LLM-as-a-Judge That Drives Business Results — Hamel Husain (article, free, ~1h, rated 10/10). The correction to everything else written on this subject. The judge is not the work — aligning the judge with a human who actually looked at the data is the work, and most teams skip straight past it and then trust a number that means nothing.
- Evaluation concepts — LangSmith (docs, free, ~0.5h, rated 8/10). The reference to keep open while you wire it up — the vocabulary the video introduces, written down where you can search it.
Build this — Grade your judge. Hand-label thirty of your own outputs. Have the judge label the same thirty. Report the agreement rate. Below about 80% the judge is telling you a story, and you have just learned that for the price of an afternoon.
You're done when: You can quote the agreement rate between your judge and your own labels — and you know which failure category it gets wrong.
Common traps:
- Judges favour longer answers, the first option shown and outputs from their own model family. Control for each.
- Using a judge for facts it can't check. Give it the reference answer or the source context.
Check yourself:
- How do you know your judge is trustworthy? Answer: Hand-label 50–100 examples and measure agreement with the judge. Then read the disagreements to see which failure type it misses.
- What is position bias, and how do you control it? Answer: The judge prefers whichever answer it sees first (or second). Run each pair in both orders and only count consistent verdicts.
Put to work in: CAMS Lens — grounded answers that have to be right
Track 5 — Production architecture
Where AI engineering becomes engineering (~6 hours)
The gap between a working prototype and a system real users depend on is not model quality. It is latency, failure isolation, cost, caching and observability — ordinary distributed-systems work, applied to a component that is slow, expensive and occasionally wrong.
A production case study worth reading twice
You can reason about an AI system the way you'd reason about any distributed system.
streaming & stateful inference · full-duplex voice · asynchronous delegation · WebRTC · latency budgets · session affinity · prompt caching · failure isolation · observability
In short:
- Separate the fast path from the slow path. The user-facing loop must answer within a latency budget; long work (search, tools, other agents) runs asynchronously and reports back.
- Stream everything. Start speaking or rendering at the first token. Time-to-first-token matters more to how fast it feels than total generation time.
- State has an address. Stateful sessions (voice, WebRTC) must reach the same server or share state. Session affinity and prompt caching keep them fast and cheap.
- How we built a realtime system for responsive voice AI in six months — OpenAI (article, free, ~1h, rated 10/10). The single best architecture read on this page. The decision that matters: they isolated the real-time media path from slower tool and reasoning work, so a slow tool can never stall the audio stream. That one idea — keep the fast path free of the slow path — is what turns a mid-level interview answer into a senior one.
Build this — Instrument something you already built. Take the Track 3 agent. Add a latency budget per turn, a cost counter, provider failover, and a trace you can actually read when it misbehaves at 2am.
You're done when: You can draw your system's fast path and slow path as separate lines, and say what happens when the slow one stalls.
Common traps:
- Putting a slow tool call on the critical path of a real-time reply.
- Measuring average latency. Users feel the p95 — track it.
Check yourself:
- Your voice agent must look up an order (3 seconds). What does the user hear meanwhile? Answer: An immediate acknowledgement from the fast path, while the lookup runs asynchronously and its result is spoken when it arrives.
- Why does prompt caching matter for a long session? Answer: The long, unchanging prefix (system prompt, tools, early history) is reused instead of reprocessed, which cuts latency and input cost on every turn.
Put to work in: Jarvis — per-turn provider failover and streaming · Dialogflow CX voicebot — production conversational system
What happens when the provider is down
You can name every way your system fails and say what it does instead of failing.
timeouts · provider failover · graceful degradation · rate limits and backoff · approval gates · cost ceilings
In short:
- Every call needs a timeout. A provider that hangs is worse than one that errors. Set timeouts per call and treat a timeout as a failure to route around.
- Fail over, then degrade. Retry with backoff on rate limits, switch to a second provider on outages, and when everything fails, give a reduced answer rather than an error page.
- Put ceilings in code. Cost per request, calls per user and actions that need approval are enforced by the system, not requested in the prompt.
There is no resource here on purpose — I have not read one that beats the lesson. This stage is learned the day a provider deprecates a model without telling you and every call in production starts returning 404. Write down, for your own system, what happens on: a timeout, a 429, a malformed reply, a tool that hangs, and a model id that no longer exists. If any answer is 'the request just fails', that is your next afternoon's work.
Build this — Break it on purpose. Point your primary provider at a dead endpoint and run your happy path. Then rate-limit it. Then make a tool hang for sixty seconds. Fix what you find, in that order — the fixes are a timeout, a backoff, and a fallback, and they are perhaps eighty lines between them.
You're done when: You have run your own system with its primary provider unplugged, and a user still got an answer.
Common traps:
- Retrying instantly in a tight loop on a 429 — it makes the rate limit worse. Use exponential backoff with jitter.
- Failing over to a model with a different prompt format or tool schema, untested. Run the fallback path in your evals too.
Check yourself:
- Your primary LLM provider returns 503 for ten minutes. What happens to a user request? Answer: It times out quickly, fails over to a second provider (tested with the same evals), and if that fails, returns a clear degraded response — not a spinner.
- Why add jitter to retry backoff? Answer: So thousands of clients don't retry at the same instant and knock the recovering service over again.
Put to work in: Jarvis — per-turn provider failover and streaming · OpsPilot — agents behind an approval gate
Track 6 — System design & interview
A separate rail — not about building, about being hired (~8 hours)
This track is honestly part of the path I walked, but it is a different kind of thing to the six above: it is about communicating the work rather than doing it. Pick it up at the end, or in parallel once you have something built to talk about. Nothing here helps you if the previous tracks are empty.
System design fundamentals, then real problems
You can hold a 45-minute design conversation without hand-waving the storage layer.
storage · scalability · networking & latency · fault tolerance · CAP · databases · Redis · Kafka · load balancers · CDNs
In short:
- Requirements before boxes. Start with users, scale, latency targets and what must never be lost. The numbers decide the design; the diagram comes after.
- Every choice is a trade-off. Consistency vs availability, cache freshness vs speed, queue durability vs latency. Name what each choice costs, not just what it gives.
- An LLM is a slow, costly, unreliable dependency. Design around it like any other: timeouts, caching, fallbacks, rate limits and a clear contract for its inputs and outputs.
- How to Prepare for System Design Interviews — Hello Interview · with a Meta staff engineer (video, free, ~1h, rated 9/10). Fundamentals first — storage, scale, networking, latency, fault tolerance, CAP, Redis, Kafka, load balancers, CDNs — and then actual design problems. The right order, which most system-design content gets backwards.
- Hello Interview — Hello Interview (channel, free, ~5h, rated 9/10). Their system design and behavioural material, plus real interview walkthroughs. Watching someone fail a question well is worth more than reading ten cheat sheets.
- IGotAnOffer — Engineering & AI-engineering coaching — IGotAnOffer (service, paid). Listed for completeness — paid coaching and mock interviews, including an AI-engineering track. I have not used it myself, so treat this as a pointer rather than a recommendation; their free written material stands on its own.
You're done when: You can design a system that includes an LLM without the LLM becoming a magic box in the middle of the diagram.
Common traps:
- Drawing Kafka, Redis and a load balancer before asking about scale.
- Putting "LLM" in the middle of the diagram with no latency, cost or failure behaviour attached.
Check yourself:
- Where does the LLM sit in your design, and what happens when it's slow? Answer: Behind a service with a timeout, a cache for repeated requests, a fallback provider and a degraded response — ideally off the synchronous path where possible.
- What does CAP make you choose, in practice? Answer: During a network partition, whether to reject requests (consistency) or serve possibly stale data (availability).
Talking about AI systems specifically
You can answer 'how would you build this with an LLM?' like an engineer, not an enthusiast.
cost and latency as design constraints · eval strategy as an answer · failure modes · when NOT to use a model
In short:
- Lead with the eval. "How would you know it works?" is the question behind every AI design question. Answer it with a dataset, a metric and a threshold.
- Put numbers on cost and latency. Tokens per request × requests per day × price, and p95 latency per step. Being specific is what separates experience from reading.
- Know when not to use a model. A rule, a regex or a classic classifier is often cheaper, faster and more reliable. Say what the model adds over that baseline.
There is no course for this one. The material is everything above: say the eval strategy out loud, name the failure modes, put a number on the latency, and know when the correct answer is a regex and a lookup table. The strongest signal you can give in an AI interview is knowing where the model does not belong.
You're done when: Asked to add AI to a product, you can describe a version that does not use a model — and say what the model buys over it.
Common traps:
- Describing the happy path only. Interviewers want the failure modes.
- Naming frameworks instead of explaining the mechanism underneath.
Check yourself:
- "Add AI to our support inbox." What's the version without a model? Answer: Keyword routing rules and canned replies for the top intents. The model earns its place by handling the long tail and drafting replies — measured against that baseline.
- Estimate the daily cost of 100k requests at 2k input and 300 output tokens. Answer: 200M input and 30M output tokens a day, multiplied by the model's per-million prices. Say the formula out loud, then the number.
Explore more
- Home — overview, skills and a built-in AI assistant
- Experience — roles at CGI, Think360 AI - CAMS, CAMS Mutual Funds, RAVI - IIT Delhi
- Projects — GenAI, LLM, RAG and full-stack builds
- Education — IIT Delhi, M.Tech & B.Tech Computer Science
- Learn — a working AI engineer's roadmap: LLMs, RAG, agents, evals, production
- GitHub Activity — open-source contributions
- Contact / Hire me