How to learn AI, ML and Agentic AI engineering — the path I actually walked

A learning path written by K Laxman, a Senior AI Engineer who ships production Generative AI systems — not a generic link list. It starts from "you can already code" and goes straight at large language models, retrieval and agents. Every stage ends in something you build, and links to the production system it was used to build.

The order is the argument: LLM fundamentals → RAG → Agent loops → Tools / MCP → Harness engineering → Evals + tracing → Production architecture → System design.

27 stages across 8 tracks, about 66 hours of study plus the builds. 45 of 46 resources are free.

Choose a path

AI — Understand the model, then ground it in your own data (21h). You want to build products on top of LLMs.
- LLM fundamentals (~6h): Understand the thing everything else sits on
- Retrieval-Augmented Generation (~10h): The part most people do badly
- Evals, tracing & observability (~5h): The stage everyone skips
ML — The classical craft — rows, columns, and not fooling yourself (15h). Your data lives in a database, not in documents.
- Classic ML (~10h): Still what wins on the data most companies actually have
- Evals, tracing & observability (~5h): The stage everyone skips
Agentic AI Engineering — A model, in a loop, with tools — and the harness that makes it reliable (34h). You want the thing to do the work, not just answer.
- LLM fundamentals (~6h): Understand the thing everything else sits on
- Agents & tools (~17h): A model, in a loop, with tools. That's it.
- Evals, tracing & observability (~5h): The stage everyone skips
- Production architecture (~6h): Where AI engineering becomes engineering

Projects to build — Generative AI and agentic engineering

Token and cost lens (Foundation, ~4h). A small CLI or web page that takes a prompt and shows its tokens, its cost across three providers, and how much of a 128k window it uses — then the same for a Hindi or Telugu version of the text. Done when: You can predict a monthly bill from traffic numbers within 20%.
Structured extraction service (Foundation, ~8h). An API that turns invoices, resumes or receipts into validated JSON: schema-constrained output, a parse-or-repair retry, business-rule checks, and a 30-document labelled test set. Done when: Field-level accuracy on your test set, and a logged repair rate.
RAG over your own documents, with evals (Core, ~16h). A question-answering bot over a real corpus (your notes, a product's docs). Hybrid search, a reranker, cited answers, a refusal when nothing relevant exists — and a recall@k and answer-quality report. Done when: Recall@5 and answer accuracy before and after each change, in the README.
An agent loop with no framework (Core, ~10h). A while loop, three real tools (search, a file read, a calculator or database query), typed arguments, a step limit, a cost ceiling, an approval gate before any write, and a trace of every step. Done when: Task success rate on 20 tasks, with average steps and cost per task.
Workflow patterns, then the same thing as a graph (Core, ~10h). Implement routing and evaluator-optimizer in plain code. Then rebuild the evaluator-optimizer as a LangGraph graph with a conditional edge and checkpointing, and compare the two honestly. Done when: Quality, latency and cost for each version on the same 20 inputs.
An MCP server for a system you use (Core, ~8h). Expose something real — your RAG bot, a database, a calendar, a GitHub repo — as an MCP server with a few well-named, well-described tools. Use it from an MCP client, then improve the tool descriptions using failed transcripts. Done when: Correct-tool-choice rate before and after rewriting the descriptions.
A long-running coding agent harness (Advanced, ~20h). Wrap an agent so it can build a small app across many sessions: an initializer that writes a feature list marked pass/fail, a progress file each session reads first, one feature per session, tests or a browser check before every commit. Done when: Features passing after N sessions, and how often a session broke a passing feature.
A multi-agent research assistant (Advanced, ~16h). A lead agent that plans, three parallel sub-agents that search and summarise with citations, and a final synthesis. Compare it with a single agent on the same questions. Done when: Quality per question (judged against your rubric) and tokens per answer, single vs multi.
Evals and tracing in CI (Advanced, ~10h). Take any project above and add a regression suite that runs on every change: code-based graders first, an LLM judge you have checked against your own labels, traces stored per run, and a build that fails when the score drops. Done when: A pull request that was blocked by your eval, and the example that caught it.
A real-time voice agent that survives outages (Capstone, ~24h). Streaming speech in and out, a fast path that answers within a latency budget, slow tools run asynchronously, and failover to a second LLM and TTS provider when the first one times out. Done when: p95 time-to-first-audio, and a recorded session with the primary provider switched off.
Ship one to real users (Capstone, ~30h). Pick the project above you are proudest of and put it in front of people: authentication, rate limits, a cost ceiling, monitoring, a fallback, and a short write-up of the architecture and its numbers. Done when: Real users, a dashboard, and a one-page write-up you could walk an interviewer through.

AI — Understand the model, then ground it in your own data

The most common job going, and the one with the widest gap between a demo and something you would put in front of a customer. Understand what the model actually is, then learn to feed it your data without it inventing things.

Track 1 — LLM fundamentals

Understand the thing everything else sits on (~6 hours)

If you only ever do one item on this page, do the first one. Almost every confused conversation I have about AI systems traces back to someone never having watched this.

The whole pipeline, end to end

You can trace a token from raw internet text all the way to a chat reply, naming every stage.

pretraining · tokenisation · transformers · inference · SFT · RLHF · hallucination · tool use · reasoning models

In short:

Pretraining makes a document completer. Trained to predict the next token over a huge slice of the internet, the base model continues text. It does not follow instructions yet.
Post-training makes an assistant. Supervised fine-tuning on example conversations, then reinforcement learning from human (or AI) feedback, turns the completer into something that answers.
Hallucination is the default. The model always produces a fluent continuation. Unless training taught it to recognise what it doesn't know, a confident guess is the most likely text.

Deep Dive into LLMs like ChatGPT — Andrej Karpathy (video, free, ~3.5h, rated 10/10). Do this first. Nothing else on the internet takes you from raw pretraining data to RLHF and reasoning models in one honest, unhurried pass — including why hallucination happens rather than just that it does.
Attention Is All You Need — Vaswani et al., 2017 (paper, free, ~1.5h, rated 9/10). Read it AFTER the video, not before. It is eight pages and it will feel obvious, which is exactly the point at which you have understood it.

Build this — Re-implement decoding. Take any provider that returns logprobs and implement greedy, temperature and top-p sampling yourself. Write a README explaining, in your own words, when you'd use each. It is thirty lines of code and it makes §0.5 permanent.

You're done when: You can explain, without notes, why a model hallucinates confidently rather than saying 'I don't know'.

Common traps:

Treating the model as a database. It stores statistical patterns, not records — look facts up and put them in the context.
Assuming a reasoning model is always the better choice. It is slower and costlier; many tasks don't need it.

Check yourself:

Why does a model hallucinate confidently instead of saying "I don't know"? Answer: It was trained to produce likely text, and in its training data questions are usually followed by confident answers. Saying "I don't know" has to be specifically trained in, and it only works for things the model can tell it doesn't know.
What is the difference between a base model and an assistant model? Answer: The base model continues documents. The assistant is the same network after SFT and RLHF, trained to respond to instructions in a conversation format.

Put to work in: Jarvis — six interchangeable providers behind one loop · CAMSCode — enterprise AI coding assistant

Getting output your code can actually parse

You can make a model return JSON your program can trust, and you know what to do the one call in a thousand where it doesn't.

JSON schema · constrained decoding · refusals · repair passes · streaming partials

In short:

Ask for a schema, not "JSON please". Provider structured-output modes constrain decoding to your JSON schema, so the reply is valid JSON of the right shape — not just valid-looking text.
Valid is not correct. The schema guarantees shape. It says nothing about whether the values are true. Validate the business rules in your own code.
Plan for the one in a thousand. Refusals, truncation at the token limit and schema features the provider doesn't support all still happen. Parse, and on failure repair once or fail cleanly.

Structured Outputs — OpenAI (docs, free, ~0.5h, rated 8/10). Short, and the only one of these guides that is honest about the failure cases — refusals, and what a schema cannot express. Read it even if you use a different provider; the shape of the problem is the same everywhere.

Build this — A parse-or-repair loop. Ask for a schema, parse the reply, and on a parse failure send the error text back for one repair attempt before giving up. Log how often the repair fires. That number is the difference between a demo and something you can put in front of a customer.

You're done when: You can say what your system does when the model returns prose instead of JSON — and it isn't 'crash'.

Common traps:

Parsing with a regex over free text. Use the provider's structured mode, and a real JSON parser.
Setting max tokens too low for the output. A truncated object is invalid JSON, every time.

Check yourself:

The model returns prose instead of JSON. What does your system do? Answer: Catches the parse error, sends the error text back for one repair attempt, and if that also fails, returns a clear error or fallback — logged and counted.
The JSON parses and matches the schema. What can still be wrong? Answer: The values. A date can be in the wrong year, a total can fail to add up. Schema validation checks shape; your own checks cover meaning.

Put to work in: CAMSCode — enterprise AI coding assistant · Intelligent document processing

Track 2 — Retrieval-Augmented Generation

The part most people do badly (~10 hours)

Everyone can build a RAG demo in an afternoon. Almost nobody can tell you why theirs returns the wrong chunk. The difference between those two people is this track — and the eval set at the end of it.

RAG properly — all fifteen techniques

You can name, and choose between, the retrieval strategies. Not just 'embed, store, search'.

indexing · retrieval · generation · multi-query · RAG-Fusion · query decomposition · step-back prompting · HyDE · routing · query construction · multi-representation indexing · RAPTOR · ColBERT · CRAG · adaptive RAG

In short:

Three stages, three failure points. Indexing (split and embed), retrieval (find the right chunks) and generation (answer from them). A wrong answer came from exactly one of them.
Most techniques fix the query or the index. Multi-query, HyDE, step-back and decomposition rewrite the question; multi-representation indexing and RAPTOR change what gets stored. Each fixes a specific miss.
Corrective and adaptive RAG add judgement. Grade the retrieved chunks before answering, and re-retrieve or fall back when they are weak, instead of generating from whatever came back.

Learn RAG From Scratch — LangChain / freeCodeCamp (video, free, ~2.2h, rated 10/10). The reason this beats every 'build a RAG bot in 10 minutes' video is the list of concepts on the left. It goes well past embed-and-retrieve into routing, decomposition, HyDE, RAPTOR and CRAG — the techniques you reach for when the naive version is returning garbage.
langchain-ai/rag-from-scratch — LangChain (repo, free, ~4h, rated 9/10). Run every notebook. Watching RAG is not learning RAG — the chunking decisions only become real when your own document set makes them go wrong.

Build this — RAG over your own notes — plus a 30-question eval set. The bot is the easy half. The assignment is the eval set: thirty real questions with known answers, and a score you can quote. Without it you have a demo, and you will not be able to tell whether tomorrow's change helped.

You're done when: Given a wrong answer, you can separate retrieval failure from generation failure in under a minute.

Common traps:

Stacking techniques before measuring. Every one adds latency and cost; add one only when a measured failure calls for it.
Blaming the model when the right chunk was never retrieved. Look at what was retrieved first.

Check yourself:

An answer is wrong. What is the one-minute test that tells you where? Answer: Look at the retrieved chunks. If the answer isn't in them, it's a retrieval failure. If it is and the model still got it wrong, it's a generation failure.
When does HyDE help? Answer: When questions are short and phrased very differently from the documents. Embedding a hypothetical answer lands closer to real answers than the bare question does.

Put to work in: CAMS Lens — SEBI-grounded compliance intelligence · Agentic Research & Decision Intelligence Assistant

Chunking — the decision that quietly decides everything

You can explain why your chunk size is what it is, instead of because a tutorial said 512.

fixed vs semantic splitting · overlap · chunk size vs recall · metadata on the chunk · contextual retrieval

Interactive explainer on the live page: context budget — You can explain why your chunk size is what it is, instead of because a tutorial said 512.

In short:

The chunk is the unit of retrieval. You can only retrieve what you stored. If the answer is split across two chunks, or buried in a huge one, retrieval can't deliver it cleanly.
Size is a trade-off. Small chunks match precisely but lose context; large chunks keep context but dilute the embedding. Overlap reduces boundary losses at the cost of duplication.
Give each chunk its context. Prepending the document title, section heading or a short generated summary (contextual retrieval) makes a chunk findable on its own.

Introducing Contextual Retrieval — Anthropic (article, free, ~0.75h, rated 9/10). The single change with the best effort-to-payoff ratio in RAG: prepend a sentence of document context to each chunk before embedding it. Measured, with the failure rate reported rather than a vibe. Read this the day your naive retrieval starts missing.
Chunking Strategies for LLM Applications — Pinecone (article, free, ~0.5h, rated 7/10). Vendor content, and worth it anyway — it is the clearest side-by-side of the splitting strategies. Take the taxonomy, ignore the product.

Build this — Same questions, four chunkings. Take the eval set from the stage above and run it against four indexes: 256 fixed, 512 fixed, 512 with overlap, and semantic. One table, four scores. You will never again accept a chunk size on faith, and you now own a number you can quote in an interview.

You're done when: You can name the chunk size you use, the one you tried before it, and the score difference between them.

Common traps:

Choosing a chunk size once and never testing it. It is a parameter to measure on your own eval set.
Splitting through tables and code blocks. Split on the document's own structure — headings, paragraphs, rows — before falling back to character counts.

Check yourself:

Your chunks say "the policy above applies" and retrieval returns them useless. What is the fix? Answer: Add context to each chunk — document and section titles, or a generated one-line summary — so it stands on its own when retrieved.
How do you choose between 300-token and 1,000-token chunks? Answer: Index both and run the same questions against each. Compare recall@k and answer quality; pick with numbers, not intuition.

Put to work in: CAMS Lens — SEBI-grounded compliance intelligence

When retrieval is the bug

You can tell a retrieval problem from a generation problem before you touch the prompt.

hybrid search · reranking · recall@k · query rewriting · the missing-document case

In short:

Measure recall@k first. For a set of questions with known source documents, how often is the right one in the top k results? Until you know that, prompt tweaks are guesses.
Hybrid search covers both kinds of miss. Keyword search (BM25) catches exact terms and identifiers; vectors catch paraphrase. Combining them fixes misses either one makes alone.
Rerank a wide net. Retrieve 50 cheaply, then let a cross-encoder reranker pick the best 5. The reranker reads query and chunk together, which is far more precise.

Rerank — overview — Cohere (docs, free, ~0.5h, rated 8/10). Retrieve twenty, rerank, keep four. It is one extra call and it is the cheapest quality jump available to you — usually larger than swapping the generation model, which is where people reach first.

Build this — Measure recall@k before you tune anything. For each question in your eval set, record whether the answer-bearing chunk appeared in the top k at all. If it didn't, no prompt on earth was going to save that answer. Split your failures into 'never retrieved' and 'retrieved and ignored' — they have completely different fixes.

You're done when: You have a recall@k number for your own corpus, and you fixed retrieval before you touched the prompt.

Common traps:

Tuning the prompt to fix answers whose source was never retrieved.
Not handling the missing-document case. When nothing relevant exists, the system should say so, not answer from the nearest irrelevant chunk.

Check yourself:

Users search for error code "E4031" and get nothing useful. Why, and what fixes it? Answer: Embeddings don't represent exact identifiers well. Add keyword search (hybrid) so exact matches get found.
Recall@5 is 60% but recall@50 is 95%. What do you add? Answer: A reranker. The right documents are being found; they just aren't ranked high enough. Rerank the top 50 down to 5.

Put to work in: CAMS Lens — SEBI-grounded compliance intelligence · Agentic Research & Decision Intelligence Assistant

Track 4 — Evals, tracing & observability

The stage everyone skips (~5 hours)

This is the one that separates a demo from a product. As a system gets complicated, evals and tracing stop being nice-to-have and become the only way you can change anything without breaking it. If you cannot state your system's accuracy as a number with a dataset behind it, you do not have a system — you have a vibe.

Dataset → evaluator → task → interpretation

You can build an evaluation harness before you build the feature.

datasets · evaluators · LLM-as-judge and its failure modes · tracing · regression detection

In short:

An eval is a dataset plus a grader. Inputs with expected outputs or criteria, run through your system, scored the same way every time. Without one, every change is a guess.
Start small and real. Thirty real questions with checked answers beat a thousand synthetic ones. Add every production failure to the set as you find it.
Trace everything. Record each run's prompts, retrieved chunks, tool calls and outputs. When a score drops, the trace shows which step changed.

Why Evals Matter — LangSmith Evaluations, Part 1 — LangChain (video, free, ~0.4h, rated 10/10). Four pieces — dataset, evaluator, task, interpreting the result — and once you have that frame, every other eval tool is just an implementation of it. Part 1 is mandatory; carry on through the series as far as your patience holds.
Demystifying evals for AI agents — Anthropic (article, free, ~0.3h, rated 9/10). The vocabulary for evaluating agents rather than single answers: tasks and trials, code-based, model-based and human graders, transcripts versus final outcomes, and the difference between pass@k (succeeds at least once) and pass^k (succeeds every time) — the one production cares about.

Build this — Put a number on the RAG bot. Add the eval set and tracing to what you built in Track 2. Report the score in the README. Then change the chunk size and report it again — that delta is the whole discipline.

You're done when: You can say what your system's accuracy is, with a number and a dataset behind it.

Common traps:

Judging a change by trying five prompts by hand. You will remember the wins and forget the regressions.
Only measuring the final answer in a multi-step system. Score the retrieval and tool steps too, or you can't tell where it broke.

Check yourself:

"The new prompt feels better." What do you ask for? Answer: The score on the eval set before and after, on the same data, and the examples that got worse.
Where should new eval examples come from once you're live? Answer: Production failures — user reports, low ratings and traced errors — added to the dataset so they can't regress silently.

Put to work in: CAMS Lens — grounded answers that have to be right

LLM-as-judge — and where it stops being trustworthy

You can grade an open-ended answer automatically, and you know which of those grades to disbelieve.

pairwise vs pointwise · position bias · agreement with humans · error analysis · when to just read the outputs

In short:

A judge is a model you must evaluate. Label a sample yourself and measure how often the judge agrees. Until you have that number, the judge's scores are unverified.
Pairwise is steadier than pointwise. "Which of these two is better?" is more reliable than "rate this 1–10". Swap the order and check again to cancel position bias.
Binary criteria beat vague scales. "Does the answer cite a source from the context? yes/no" is far easier to judge consistently than "rate helpfulness".

Creating a LLM-as-a-Judge That Drives Business Results — Hamel Husain (article, free, ~1h, rated 10/10). The correction to everything else written on this subject. The judge is not the work — aligning the judge with a human who actually looked at the data is the work, and most teams skip straight past it and then trust a number that means nothing.
Evaluation concepts — LangSmith (docs, free, ~0.5h, rated 8/10). The reference to keep open while you wire it up — the vocabulary the video introduces, written down where you can search it.

Build this — Grade your judge. Hand-label thirty of your own outputs. Have the judge label the same thirty. Report the agreement rate. Below about 80% the judge is telling you a story, and you have just learned that for the price of an afternoon.

You're done when: You can quote the agreement rate between your judge and your own labels — and you know which failure category it gets wrong.

Common traps:

Judges favour longer answers, the first option shown and outputs from their own model family. Control for each.
Using a judge for facts it can't check. Give it the reference answer or the source context.

Check yourself:

How do you know your judge is trustworthy? Answer: Hand-label 50–100 examples and measure agreement with the judge. Then read the disagreements to see which failure type it misses.
What is position bias, and how do you control it? Answer: The judge prefers whichever answer it sees first (or second). Run each pair in both orders and only count consistent verdicts.

Put to work in: CAMS Lens — grounded answers that have to be right

ML — The classical craft — rows, columns, and not fooling yourself

Quieter than the AI headlines and still what most companies actually need. A table, a column you want to predict, and the discipline to know whether it worked. Boosted trees have not stopped winning just because language models arrived.

Track 4 — Classic ML

Still what wins on the data most companies actually have (~10 hours)

The half of the job the AI headlines skipped, and the half that quietly pays for most ML salaries: a table of rows, a column you want to predict, and the discipline not to fool yourself about how well it worked. Boosted trees did not stop winning on tabular data just because language models arrived — if your data lives in a database rather than in documents, this is the work.

Supervised learning, end to end

You can go from a CSV to a model with an honest score, and say what the score means.

train/validation/test · baselines · choosing the metric · cross-validation · over- and underfitting

In short:

Three splits, three jobs. Train to fit, validation to choose settings, test to report — once, at the end. Reuse the test set for decisions and it stops being a test.
Beat a baseline first. Predict the majority class, or last week's value. If your model can't clearly beat that, it isn't learning anything useful.
The metric is a product decision. Accuracy, precision, recall and cost-weighted error reward different mistakes. Pick the one that matches what a wrong answer costs.

scikit-learn User Guide — scikit-learn (docs, free, ~6h, rated 9/10). Not a tutorial — a reference written by people who had to make these algorithms actually work. Read the section for whatever you are using, every time, until you stop needing to.
StatQuest — Josh Starmer (channel, free, ~4h, rated 9/10). The clearest explanations of the classical algorithms anywhere, and short enough to watch one the night before you need it. Reach for it whenever a term in the docs above doesn't land.

Build this — A model with a baseline you have to beat. Pick any tabular dataset. Write down the dumbest possible predictor first — the mean, the majority class — and score it. Then beat it, and report both numbers. If you cannot beat the dumb one, that is the finding.

You're done when: You can say why a 99%-accurate model can be worthless, without looking it up.

Common traps:

Reporting accuracy on imbalanced data. 99% accuracy on a 1%-positive problem can mean the model never finds a single positive.
Tuning against the test set until the number looks good.

Check yourself:

Why can a 99%-accurate fraud model be worthless? Answer: If 1% of transactions are fraud, predicting "not fraud" every time scores 99%. Look at recall and precision on the fraud class.
What is the validation set for, if you have a test set? Answer: Making choices — hyperparameters, features, which model. The test set is kept untouched so the final number is honest.

Put to work in: Driver Drowsiness Detection

Leakage, imbalance, and the ways you fool yourself

You can spot the mistakes that make a model look brilliant offline and useless in production.

data leakage · time-based splits · class imbalance · distribution shift · error analysis

In short:

Leakage is information from the future. A feature that wouldn't exist at prediction time (an outcome field, a post-event timestamp) makes offline scores look great and production scores collapse.
Split on time when time matters. If the real system predicts next month from this month, validate the same way. Random splits let the model peek ahead.
Read the errors. Error analysis — looking at 50 wrong predictions by hand — finds leakage, label noise and shifts that no aggregate metric will show.

There is no single video for this one, and that is telling: it is the part of ML that gets learned by being burned. The two habits that matter — split on time whenever the real system will predict the future, and look at fifty actual errors before you touch the model — will save you more than any algorithm choice.

Build this — Break your own model on purpose. Take the model from the previous stage and introduce a leaked feature deliberately. Watch the score jump. That jump is exactly what a subtle leak looks like, and now you know its shape.

You're done when: Given a suspiciously good result, leakage is the first thing you check — and you know where to look.

Common traps:

Scaling or imputing on the full dataset before splitting. The statistics leak test information into training.
The same customer, patient or device appearing in both train and test sets.

Check yourself:

Your churn model scores 0.99 AUC. What do you check first? Answer: Leakage. Look for features that are only known after the customer churned — cancellation date, final bill, support tickets about closing.
Why is a random split wrong for forecasting next month's demand? Answer: Training rows from after the test rows let the model use the future. Train on the past and test on a later period.

Trees, boosting — and when to reach for deep learning

You choose a model class on the shape of the data, not on what is fashionable.

decision trees · random forests · gradient boosting · when deep learning wins · when it does not

In short:

Tables → gradient boosting. On structured, tabular data, gradient-boosted trees (XGBoost, LightGBM, CatBoost) are usually the strongest and cheapest choice.
Images, audio, text → deep learning. Neural networks win where the raw input is unstructured and the features must be learned rather than engineered.
Pretrained beats from-scratch. With little data, fine-tuning a pretrained model — or using an LLM or embedding model directly — beats training a network from zero.

Practical Deep Learning for Coders — fast.ai · Jeremy Howard (course, free, ~12h, rated 9/10). Top-down: you train a working model in lesson one and only then peel back the layers. If you want the neural-network half of ML properly — beyond the Track 0 pictures — this is still the best on-ramp there is. Take it as far as you need and no further.

Build this — Same dataset, three model classes. Boosted trees, a linear model, and a small neural net on identical splits. Report all three, plus training time. The answer on tabular data will usually surprise anyone who only reads AI news.

You're done when: Asked whether to use deep learning, your first question is about the data, not the model.

Common traps:

Reaching for a neural network on a 5,000-row spreadsheet.
Comparing model classes on different preprocessing or splits, then crediting the model for the difference.

Check yourself:

Tabular data, 50k rows, 30 columns. What do you try first? Answer: A gradient-boosted tree model, against a simple baseline. Deep learning rarely wins here, and costs far more to tune.
Someone asks "should we use deep learning?" What is your first question? Answer: What is the data — its type and how much of it is labelled. That decides the model class far more than the model does.

Put to work in: Table Recognition System

Track 4 — Evals, tracing & observability

The stage everyone skips (~5 hours)

Dataset → evaluator → task → interpretation

You can build an evaluation harness before you build the feature.

datasets · evaluators · LLM-as-judge and its failure modes · tracing · regression detection

In short:

An eval is a dataset plus a grader. Inputs with expected outputs or criteria, run through your system, scored the same way every time. Without one, every change is a guess.
Start small and real. Thirty real questions with checked answers beat a thousand synthetic ones. Add every production failure to the set as you find it.
Trace everything. Record each run's prompts, retrieved chunks, tool calls and outputs. When a score drops, the trace shows which step changed.

Why Evals Matter — LangSmith Evaluations, Part 1 — LangChain (video, free, ~0.4h, rated 10/10). Four pieces — dataset, evaluator, task, interpreting the result — and once you have that frame, every other eval tool is just an implementation of it. Part 1 is mandatory; carry on through the series as far as your patience holds.
Demystifying evals for AI agents — Anthropic (article, free, ~0.3h, rated 9/10). The vocabulary for evaluating agents rather than single answers: tasks and trials, code-based, model-based and human graders, transcripts versus final outcomes, and the difference between pass@k (succeeds at least once) and pass^k (succeeds every time) — the one production cares about.

You're done when: You can say what your system's accuracy is, with a number and a dataset behind it.

Common traps:

Judging a change by trying five prompts by hand. You will remember the wins and forget the regressions.
Only measuring the final answer in a multi-step system. Score the retrieval and tool steps too, or you can't tell where it broke.

Check yourself:

"The new prompt feels better." What do you ask for? Answer: The score on the eval set before and after, on the same data, and the examples that got worse.
Where should new eval examples come from once you're live? Answer: Production failures — user reports, low ratings and traced errors — added to the dataset so they can't regress silently.

Put to work in: CAMS Lens — grounded answers that have to be right

LLM-as-judge — and where it stops being trustworthy

You can grade an open-ended answer automatically, and you know which of those grades to disbelieve.

pairwise vs pointwise · position bias · agreement with humans · error analysis · when to just read the outputs

In short:

A judge is a model you must evaluate. Label a sample yourself and measure how often the judge agrees. Until you have that number, the judge's scores are unverified.
Pairwise is steadier than pointwise. "Which of these two is better?" is more reliable than "rate this 1–10". Swap the order and check again to cancel position bias.
Binary criteria beat vague scales. "Does the answer cite a source from the context? yes/no" is far easier to judge consistently than "rate helpfulness".

Creating a LLM-as-a-Judge That Drives Business Results — Hamel Husain (article, free, ~1h, rated 10/10). The correction to everything else written on this subject. The judge is not the work — aligning the judge with a human who actually looked at the data is the work, and most teams skip straight past it and then trust a number that means nothing.
Evaluation concepts — LangSmith (docs, free, ~0.5h, rated 8/10). The reference to keep open while you wire it up — the vocabulary the video introduces, written down where you can search it.

You're done when: You can quote the agreement rate between your judge and your own labels — and you know which failure category it gets wrong.

Common traps:

Judges favour longer answers, the first option shown and outputs from their own model family. Control for each.
Using a judge for facts it can't check. Give it the reference answer or the source context.

Check yourself:

How do you know your judge is trustworthy? Answer: Hand-label 50–100 examples and measure agreement with the judge. Then read the disagreements to see which failure type it misses.
What is position bias, and how do you control it? Answer: The judge prefers whichever answer it sees first (or second). Run each pair in both orders and only count consistent verdicts.

Put to work in: CAMS Lens — grounded answers that have to be right

Agentic AI Engineering — A model, in a loop, with tools — and the harness that makes it reliable

Where I spend my own time, and where the marketing is furthest from the engineering. It is a while loop with an approval gate — the hard parts are the tools, the evaluation and the failure isolation, not the magic. The job title for that work is harness engineering, and it is the shift: the model is bought in, the harness is what you build.

Track 1 — LLM fundamentals

Understand the thing everything else sits on (~6 hours)

If you only ever do one item on this page, do the first one. Almost every confused conversation I have about AI systems traces back to someone never having watched this.

The whole pipeline, end to end

You can trace a token from raw internet text all the way to a chat reply, naming every stage.

pretraining · tokenisation · transformers · inference · SFT · RLHF · hallucination · tool use · reasoning models

In short:

Pretraining makes a document completer. Trained to predict the next token over a huge slice of the internet, the base model continues text. It does not follow instructions yet.
Post-training makes an assistant. Supervised fine-tuning on example conversations, then reinforcement learning from human (or AI) feedback, turns the completer into something that answers.
Hallucination is the default. The model always produces a fluent continuation. Unless training taught it to recognise what it doesn't know, a confident guess is the most likely text.

Deep Dive into LLMs like ChatGPT — Andrej Karpathy (video, free, ~3.5h, rated 10/10). Do this first. Nothing else on the internet takes you from raw pretraining data to RLHF and reasoning models in one honest, unhurried pass — including why hallucination happens rather than just that it does.
Attention Is All You Need — Vaswani et al., 2017 (paper, free, ~1.5h, rated 9/10). Read it AFTER the video, not before. It is eight pages and it will feel obvious, which is exactly the point at which you have understood it.

You're done when: You can explain, without notes, why a model hallucinates confidently rather than saying 'I don't know'.

Common traps:

Treating the model as a database. It stores statistical patterns, not records — look facts up and put them in the context.
Assuming a reasoning model is always the better choice. It is slower and costlier; many tasks don't need it.

Check yourself:

Why does a model hallucinate confidently instead of saying "I don't know"? Answer: It was trained to produce likely text, and in its training data questions are usually followed by confident answers. Saying "I don't know" has to be specifically trained in, and it only works for things the model can tell it doesn't know.
What is the difference between a base model and an assistant model? Answer: The base model continues documents. The assistant is the same network after SFT and RLHF, trained to respond to instructions in a conversation format.

Put to work in: Jarvis — six interchangeable providers behind one loop · CAMSCode — enterprise AI coding assistant

Getting output your code can actually parse

You can make a model return JSON your program can trust, and you know what to do the one call in a thousand where it doesn't.

JSON schema · constrained decoding · refusals · repair passes · streaming partials

In short:

Ask for a schema, not "JSON please". Provider structured-output modes constrain decoding to your JSON schema, so the reply is valid JSON of the right shape — not just valid-looking text.
Valid is not correct. The schema guarantees shape. It says nothing about whether the values are true. Validate the business rules in your own code.
Plan for the one in a thousand. Refusals, truncation at the token limit and schema features the provider doesn't support all still happen. Parse, and on failure repair once or fail cleanly.

Structured Outputs — OpenAI (docs, free, ~0.5h, rated 8/10). Short, and the only one of these guides that is honest about the failure cases — refusals, and what a schema cannot express. Read it even if you use a different provider; the shape of the problem is the same everywhere.

You're done when: You can say what your system does when the model returns prose instead of JSON — and it isn't 'crash'.

Common traps:

Parsing with a regex over free text. Use the provider's structured mode, and a real JSON parser.
Setting max tokens too low for the output. A truncated object is invalid JSON, every time.

Check yourself:

The model returns prose instead of JSON. What does your system do? Answer: Catches the parse error, sends the error text back for one repair attempt, and if that also fails, returns a clear error or fallback — logged and counted.
The JSON parses and matches the schema. What can still be wrong? Answer: The values. A date can be in the wrong year, a total can fail to add up. Schema validation checks shape; your own checks cover meaning.

Put to work in: CAMSCode — enterprise AI coding assistant · Intelligent document processing

Track 3 — Agents & tools

A model, in a loop, with tools. That's it. (~17 hours)

This is the track I care most about, because it is the one where the industry's marketing is furthest from the engineering. An agent is not a mysterious autonomous intelligence. It is a while loop. Write one by hand and the mystique never comes back — and then build the harness around it, which is where the actual work has moved.

The agent loop, by hand

You can write an agent without a framework and explain every line of it.

environment · tool schemas · the while loop · context accumulation · approval gates

Interactive explainer on the live page: agent loop — You can write an agent without a framework and explain every line of it.

In short:

An agent is a while loop. Call the model; if it asks for a tool, run it, append the result to the conversation, and call again. Stop when it answers without a tool call.
The context is the agent's memory. Every tool result goes back into the context. The model knows only what the loop has put there so far.
You own the stopping conditions. A step limit, a cost ceiling and approval gates for risky actions belong in your code, not in the prompt.

How We Build Effective Agents — Barry Zhang · Anthropic (video, free, ~0.5h, rated 10/10). Kills the mystique in twenty minutes. An agent is a model acting on an environment through tools, in a loop — environment, tools, system prompt, while True. Everything else in the ecosystem is a wrapper around those four lines.
Building effective agents — Erik Schluntz & Barry Zhang · Anthropic (article, free, ~0.5h, rated 10/10). The written companion to the talk, and the most-cited page in agent engineering for a reason. It draws the line between workflows (you write the path) and agents (the model picks it), and its core advice — start with plain API calls, add complexity only when it measurably helps — is the whole philosophy of this track.
Tips for building AI agents — Barry Zhang, Erik Schluntz & Alex Albert · Anthropic (video, free, ~0.3h, rated 8/10). Eighteen minutes of the same people arguing about where agents are over-hyped and where they genuinely earn their cost. Worth it for the workflow-vs-agent judgement calls, which no article captures as well as hearing them disagree.
Building agents with the Claude Agent SDK — Thariq Shihipar · Anthropic (article, free, ~0.3h, rated 8/10). Names the loop you are about to write: gather context, take action, verify the work, repeat. Read it for the verify step — rules-based checks, visual checks, a model as judge — because that is the part hand-written loops forget.

Build this — Write the loop. No framework.. A while loop, three tools, JSON-schema arguments, one approval gate before anything destructive. This is the keystone exercise of the whole page — do not substitute a framework tutorial for it.

You're done when: You can draw the loop on a whiteboard and point at exactly where a tool result re-enters the context.

Common traps:

No iteration cap. A confused agent will call the same tool forever and bill you for each turn.
Reaching for an agent when the steps are known in advance. A fixed workflow is cheaper, faster and easier to test.

Check yourself:

Where exactly does a tool result re-enter the loop? Answer: It is appended to the message history as a tool-result message, and the whole history is sent on the next model call.
When should you NOT build an agent? Answer: When you can write the steps down in advance. Use a fixed pipeline, and keep agents for tasks where the path genuinely depends on what is found along the way.

Put to work in: Jarvis — that loop, in production, running live in your browser · OpsPilot — agentic request triage & remediation

Workflows and graph loops — the five patterns

You can pick the simplest pattern that solves a task, and draw any agent system as a graph of steps with a loop in it.

prompt chaining · routing · parallelization · orchestrator-workers · evaluator-optimizer · state, nodes and edges · conditional edges and cycles · checkpointing

In short:

Five patterns cover most systems. Prompt chaining (fixed steps), routing (classify, then specialise), parallelization (split or vote), orchestrator-workers (a model plans the subtasks) and evaluator-optimizer (generate, critique, revise).
A graph is the loop, drawn. Nodes are steps, edges are control flow, and state is what travels between them. An agent is a graph with a conditional edge pointing back to an earlier node.
Every cycle needs an exit. A loop edge must have a condition that ends it — a passing check, a round limit, a budget. Without one, the graph is an infinite loop with a bill attached.

Agent patterns — runnable notebooks — Anthropic · claude-cookbooks (repo, free, ~2h, rated 9/10). The five patterns from 'Building effective agents' as short notebooks: basic workflows (chaining, routing, parallelization), orchestrator-workers and evaluator-optimizer, plus multi-agent latency experiments. Plain Python and the API — run them before you ever see a graph library.
Workflows and agents — LangChain · LangGraph docs (docs, free, ~0.75h, rated 8/10). The same patterns again, written as graphs: nodes for steps, edges for control flow, and a conditional edge that points backwards — which is all a 'graph loop' is. Read it after the notebooks and the mapping from loop to graph becomes obvious.

Build this — One task, two patterns, a number for each. Take a writing or extraction task. Solve it once as a prompt chain and once as an evaluator-optimizer loop (generate → critique → revise, with a round limit). Run both on the same 20 inputs and record quality, latency and cost. Then draw the second one as a graph: which edge is the loop, and what stops it?

You're done when: Given a new task, you can say which pattern fits, why the simpler ones don't, and where the loop's exit condition lives.

Common traps:

Using an autonomous agent where a fixed chain would do. Start with the simplest pattern and move up only when a measured failure demands it.
An evaluator-optimizer loop with a vague critique prompt. If the evaluator can't say what to fix, revision just rewrites the same mistakes.

Check yourself:

Customer emails go to billing, technical or sales, each with its own prompt. Which pattern is that? Answer: Routing — classify the input, then send it to a specialised prompt or model.
What turns a workflow graph into an agent? Answer: A cycle whose next step the model decides: a conditional edge back to the model node, taken until the model chooses to stop (or a limit stops it).

Put to work in: Research Intelligence Agent — orchestration in practice · OpsPilot — routing and remediation workflows

Tools and MCP — connecting agents to real systems

You can design a tool an LLM actually uses correctly, and expose it over MCP.

tool design · argument validation · MCP architecture · tool servers · when a tool is too broad

In short:

The tool description is a prompt. The model chooses tools and fills arguments from the name, description and schema alone. Write them for a new colleague with no other context.
Narrow tools, typed arguments. One clear job per tool, enums instead of free strings where possible, and validation on every call — the model's arguments are untrusted input.
MCP standardises the plumbing. An MCP server exposes tools and resources once; any MCP-capable client (desktop apps, IDEs, your own agent) can then use them without custom glue.

Building Agents with Model Context Protocol — Mahesh Murag · Anthropic (video, free, ~1.75h, rated 9/10). The full workshop: what MCP is, building with it, MCP with agents, the architecture, and where it is going. Watch it once you have hand-written a tool call, so you can see what the protocol is standardising.
Model Context Protocol — getting started — MCP (docs, free, ~0.5h, rated 8/10). The spec's own introduction, and the page to have open while you write the server rather than while you watch the workshop. Skim the concepts, then go straight to the transport section — that is the part everyone gets wrong first.
Tool use — overview — Anthropic (docs, free, ~0.75h, rated 9/10). The best writing anywhere on how to describe a tool so the model picks it correctly. The description field is prompt engineering, and this is the page that finally made me treat it that way.
Writing effective tools for agents — with agents — Anthropic (article, free, ~0.4h, rated 9/10). The best single guide to tool design. Fewer, more purposeful tools; namespaced names; human-readable results instead of UUIDs; pagination and truncation by default — and, the real lesson, evaluate tools like any other component and let the model help rewrite their descriptions.
Code execution with MCP — Anthropic (article, free, ~0.3h, rated 8/10). What to do when you have hundreds of tools: let the agent discover them as files and call them from code, filtering data before it reaches the context. Their example goes from roughly 150,000 tokens of tool definitions to 2,000.
Don't Build Agents, Build Skills Instead — Barry Zhang & Mahesh Murag · Anthropic (video, free, ~0.3h, rated 8/10). Sixteen minutes on Agent Skills: folders of instructions and scripts an agent loads only when a task needs them. The idea worth keeping even if you never use Claude is progressive disclosure — context loaded in layers, not all up front.

Build this — Wrap your Track 2 RAG bot as an MCP server. One tool, a real schema, connected to a client you didn't write. The moment someone else's agent can call your thing, tool design stops being theoretical.

You're done when: You can say why a tool named `do_stuff(input: string)` will be called wrongly, and rewrite it so it won't be.

Common traps:

Giving an agent a tool that can do anything, like a raw SQL or shell executor, without a permission boundary.
Returning huge, unfiltered payloads from tools. The result lands in the context window; return what the model needs.

Check yourself:

Why will `do_stuff(input: string)` get called wrongly? Answer: The name and schema tell the model nothing about when to call it or what to pass. Split it into specific tools with typed, described parameters.
What does MCP give you that a function call in your own code doesn't? Answer: Reuse: the same server works across every MCP client, and tools can be added or updated without changing the agent's code.

Put to work in: Intelligent Document Processing — multi-agent system

Harness engineering — the part that isn't the model

You can say what a harness is, name its five pieces, and point at the one your own agent is missing.

guides vs sensors · context assembly · tool orchestration · verification loops · cost envelopes · behavioural evals

Interactive explainer on the live page: harness — You can say what a harness is, name its five pieces, and point at the one your own agent is missing.

In short:

The harness is everything but the model. Context assembly, tool orchestration, verification, limits and logging. It decides what the model sees and what happens to what it produces.
Guides steer, sensors check. Guides (instructions, examples, conventions) shape behaviour before the action; sensors (tests, linters, validators) check the result after it.
Close the loop with verification. Let the agent run the check itself — the test suite, the type checker — and feed failures back. That catches errors no prompt prevents.

The shift this track has been building toward. Prompt engineering was how you got one good answer out of a 4k context window; it stopped scaling the moment a task ran longer than a few hundred lines. What replaced it is not a better prompt — it is everything around the model: what you put in the context, which tools it can reach, what checks its work, what stops it spending your money, and what you can read afterwards when it went wrong. Swapping models moves output quality by ten or fifteen per cent. Changing the harness decides whether the system works at all.

Effective harnesses for long-running agents — Justin Young · Anthropic (article, free, ~0.2h, rated 10/10). Short and concrete: how an agent keeps making progress across many context windows. An initializer session sets up the repo and a feature list marked pass/fail; every later session reads a progress file and git history, does one feature, tests it end to end, commits. Copy this structure.
Harness design for long-running application development — Prithvi Rajasekaran · Anthropic (article, free, ~0.4h, rated 9/10). The follow-up: planner, generator and evaluator agents, with the evaluator testing the running app in a browser and the two negotiating what 'done' means before work starts. Read to the end — they removed scaffolding once a newer model no longer needed it, which is the lesson.
Effective context engineering for AI agents — Anthropic (article, free, ~0.4h, rated 9/10). Context assembly is the first part of any harness, and this is the reference for it: compaction near the limit, structured notes outside the window, sub-agents that return short summaries, and loading data just in time by path or query instead of up front.
Scaling Managed Agents: decoupling the brain from the hands — Anthropic (article, free, ~0.3h, rated 8/10). How Anthropic structures a hosted agent service: a stateless harness (the brain), replaceable sandboxes and tools (the hands), and a durable session log outside the context window. Useful architecture vocabulary once your harness has to survive restarts.
Prompting for Agents — Hannah Moran & Jeremy Hadfield · Anthropic (video, free, ~0.5h, rated 8/10). Half an hour on writing the guides part of a harness: how to prompt a model that runs for many turns with tools, where short-prompt habits stop working, and how to read transcripts to find out why an agent went wrong.
Claude Agent SDK — full workshop — Thariq Shihipar · Anthropic (video, free, ~1.9h, rated 8/10). A two-hour build-along using the harness behind Claude Code as a library: tools, subagents, permissions and verification wired up live. Do it after writing your own harness, so you recognise every piece rather than copying it.
What is harness engineering? — harness-engineering.ai (article, free, ~0.4h, rated 9/10). The cleanest definition going: everything between user input and agent output except the model itself, broken into five parts — context engineering, tool orchestration, verification loops, cost envelopes, observability. Read this one first and the other four stop sounding like different topics.
Harness engineering: leveraging Codex in an agent-first world — OpenAI (article, free, ~0.5h, rated 9/10). Five months, a production beta, about a million lines, and no human typing any of them — every PR opened and reviewed by Codex agents. Read it for the admission at the end rather than the number: with the writing automated, their hard problems became designing environments, feedback loops and control systems. That is the job description this stage is about.
Harness engineering for coding agent users — Martin Fowler / Thoughtworks (article, free, ~0.75h, rated 10/10). The one that gives you the vocabulary to design with: guides (architecture docs, convention files, bootstrap scripts) steer before generation, sensors (linters, type checks, structural tests, review agents) catch after it, and each is either computational — deterministic, milliseconds — or inferential, an LLM judging. It also says the quiet part: a good harness is not trying to remove you, it is trying to spend your attention where it actually matters.
The anatomy of harness engineering — evaluate, iterate and guard AI coding agents — Google Developers Blog (article, free, ~0.5h, rated 8/10). How you keep a harness honest as models move under you: behavioural evals that test discrete observable actions rather than end-to-end outcomes, strict assertions where the task is simple and an LLM judge where it isn't, run in batches so you watch the trend instead of panicking at one bad run. Integration tests for your agent.
Learn Harness Engineering — Walking Labs (course, free, ~6h, rated 8/10). Free and open source: fourteen lectures from agent failure modes through to state management and control systems, projects to build against, copy-ready AGENTS.md and feature-list templates, and teardowns of real harnesses including Claude Code and Codex. Use it as the reference you come back to, not a sitting.
Beyond the prompt — why harness engineering is the lever — PALO IT (article, free, ~0.3h, rated 7/10). The shortest version of the argument, plus an order to build in: conventions file, then deterministic hooks for the rules that must not be reasoned about, then subagents as independent verifiers, then state on disk, and only then an automated retry loop. Its warning is the one to keep — a loop without sensors produces hallucinated completion at high speed.

Build this — Put a harness around the loop you already wrote. Take your Track 3 agent and add the five pieces one at a time, measuring after each: a conventions file it always reads, a deterministic check it cannot talk its way past, one verification gate that re-runs its work before the result counts, a per-task token ceiling that kills the run, and a trace you can read afterwards. Keep the model fixed the whole time — the point is to see how far the harness alone moves it.

You're done when: You can point at one failure your harness now catches that the model used to get away with — and say which of the five parts caught it.

Common traps:

Answering every failure with a longer system prompt. If a check can catch it, add a sensor instead.
No cost envelope. Long-running agents need a budget per task, enforced in code.
Letting a long-running agent decide it's finished. Keep an external feature list marked pass/fail by tests, and a progress file each new session reads first.

Check yourself:

Your coding agent keeps breaking imports. Guide or sensor? Answer: Both, but the sensor matters more: run the build or type check after each edit and feed errors back. A prompt rule alone will be ignored eventually.
Name the five parts of the harness in the explainer above. Answer: Context (what the model sees), tools (what it can do), verify (checks on what it produced), budget (steps and cost it may spend), and trace (a record of every run).

Put to work in: Jarvis — tool allow-lists, call budgets and a visible trace

Multi-agent systems — when one context window isn't enough

You can tell when splitting work across agents pays for itself, and build an orchestrator whose sub-agents don't trip over each other.

orchestrator and sub-agents · parallel context windows · token cost vs quality · task locking · shared state via git or files · outcome-based evaluation

In short:

Sub-agents buy context, not intelligence. Each sub-agent explores in its own clean context window and returns a short summary. The win is breadth and parallelism on tasks too big for one window.
It costs a multiple of the tokens. Anthropic reports multi-agent research using roughly 15× the tokens of a chat. It pays off only when the task is valuable and genuinely splits into independent parts.
Coordination is the hard part. Agents need clear task boundaries, a way to claim work (a lock file, a queue) and shared state they can merge (git, files). Most failures are two agents doing the same thing.

How we built our multi-agent research system — Anthropic (article, free, ~0.4h, rated 10/10). The production write-up behind Claude's Research feature. A lead agent spawns parallel sub-agents, each with its own context window. The numbers are the point: roughly 15× the tokens of a chat, a large gain on broad research tasks, and token usage explaining most of the difference in quality. Read the prompting and evaluation lessons twice.
Building a C compiler with a team of parallel Claudes — Nicholas Carlini · Anthropic (article, free, ~0.4h, rated 9/10). Sixteen agents, one repository, about 2,000 sessions: a Rust C compiler that builds the Linux kernel. The coordination is refreshingly plain — lock files to claim tasks, git to merge — and the real lesson is that high-quality tests are what let autonomous agents make progress at all.

Build this — An orchestrator with three researchers. Build a lead agent that splits a research question into three sub-questions, runs three sub-agents in parallel with separate contexts, and merges their cited summaries. Compare against a single agent on 10 questions: quality (judged against your own rubric), tokens and wall-clock time. Keep the multi-agent version only if the numbers justify the cost.

You're done when: You can quote what your multi-agent version costs in tokens compared with one agent, and what it buys.

Common traps:

Splitting tightly coupled work, like one feature across many files, between agents that can't see each other's changes.
Evaluating whether each agent followed the expected steps. Judge the final outcome — agents take different valid paths.

Check yourself:

When is a single agent the better choice? Answer: When the subtasks depend on each other, share most of their context, or the task fits in one window. Parallel agents add cost and coordination for no gain there.
Two parallel agents keep fixing the same bug. What do you add? Answer: A way to claim tasks — a lock file or queue entry per task — checked before starting work, plus merging shared state often.

Put to work in: Research Intelligence Agent — agentic research & decision support

Frameworks — deliberately last

You choose a framework because it removes work you understand, not because you're lost.

LangGraph and friends · what they abstract · what they hide

In short:

A framework hides the loop. LangGraph and similar libraries give you state, retries, persistence and graphs of steps. They are the same loop you wrote, with decisions made for you.
Adopt for features, not for learning. Durable state, human-in-the-loop pauses and tracing are real reasons. "I don't know how agents work" is not — that is what the previous stages fixed.
Know the escape hatch. Before committing, find out how to see the exact prompt sent and how to override one step. Debugging something you can't see is the real cost.

Framework material comes last on purpose. Read one only after you have written the loop, the patterns and a harness yourself. You will find you already know what every abstraction is for — and you will notice the ones that are solving a problem you don't have.

Introduction to LangGraph — LangChain Academy (course, free, ~6h, rated 8/10). If you choose a graph framework, this is the free, structured way in: state, nodes, edges and cycles, then memory, human-in-the-loop interrupts, checkpointing and deployment. The parts worth paying attention to are persistence and interrupts — those are the features that are genuinely tedious to build yourself.

You're done when: You can name one thing your chosen framework does that you would rather do yourself, and say why.

Common traps:

Adopting a framework and never looking at the raw prompts it sends.
Choosing by GitHub stars. Choose by the two or three features you actually need.

Check yourself:

Name one good reason to adopt a framework. Answer: Durable, resumable state for long workflows, or built-in human-approval pauses — features that are genuinely tedious to build and test yourself.
What should you check in your framework before production? Answer: That you can see and log the exact prompt and tool calls for any run, and override any single step without forking the library.

Track 4 — Evals, tracing & observability

The stage everyone skips (~5 hours)

Dataset → evaluator → task → interpretation

You can build an evaluation harness before you build the feature.

datasets · evaluators · LLM-as-judge and its failure modes · tracing · regression detection

In short:

An eval is a dataset plus a grader. Inputs with expected outputs or criteria, run through your system, scored the same way every time. Without one, every change is a guess.
Start small and real. Thirty real questions with checked answers beat a thousand synthetic ones. Add every production failure to the set as you find it.
Trace everything. Record each run's prompts, retrieved chunks, tool calls and outputs. When a score drops, the trace shows which step changed.

Why Evals Matter — LangSmith Evaluations, Part 1 — LangChain (video, free, ~0.4h, rated 10/10). Four pieces — dataset, evaluator, task, interpreting the result — and once you have that frame, every other eval tool is just an implementation of it. Part 1 is mandatory; carry on through the series as far as your patience holds.
Demystifying evals for AI agents — Anthropic (article, free, ~0.3h, rated 9/10). The vocabulary for evaluating agents rather than single answers: tasks and trials, code-based, model-based and human graders, transcripts versus final outcomes, and the difference between pass@k (succeeds at least once) and pass^k (succeeds every time) — the one production cares about.

You're done when: You can say what your system's accuracy is, with a number and a dataset behind it.

Common traps:

Judging a change by trying five prompts by hand. You will remember the wins and forget the regressions.
Only measuring the final answer in a multi-step system. Score the retrieval and tool steps too, or you can't tell where it broke.

Check yourself:

"The new prompt feels better." What do you ask for? Answer: The score on the eval set before and after, on the same data, and the examples that got worse.
Where should new eval examples come from once you're live? Answer: Production failures — user reports, low ratings and traced errors — added to the dataset so they can't regress silently.

Put to work in: CAMS Lens — grounded answers that have to be right

LLM-as-judge — and where it stops being trustworthy

You can grade an open-ended answer automatically, and you know which of those grades to disbelieve.

pairwise vs pointwise · position bias · agreement with humans · error analysis · when to just read the outputs

In short:

A judge is a model you must evaluate. Label a sample yourself and measure how often the judge agrees. Until you have that number, the judge's scores are unverified.
Pairwise is steadier than pointwise. "Which of these two is better?" is more reliable than "rate this 1–10". Swap the order and check again to cancel position bias.
Binary criteria beat vague scales. "Does the answer cite a source from the context? yes/no" is far easier to judge consistently than "rate helpfulness".

Creating a LLM-as-a-Judge That Drives Business Results — Hamel Husain (article, free, ~1h, rated 10/10). The correction to everything else written on this subject. The judge is not the work — aligning the judge with a human who actually looked at the data is the work, and most teams skip straight past it and then trust a number that means nothing.
Evaluation concepts — LangSmith (docs, free, ~0.5h, rated 8/10). The reference to keep open while you wire it up — the vocabulary the video introduces, written down where you can search it.

You're done when: You can quote the agreement rate between your judge and your own labels — and you know which failure category it gets wrong.

Common traps:

Judges favour longer answers, the first option shown and outputs from their own model family. Control for each.
Using a judge for facts it can't check. Give it the reference answer or the source context.

Check yourself:

How do you know your judge is trustworthy? Answer: Hand-label 50–100 examples and measure agreement with the judge. Then read the disagreements to see which failure type it misses.
What is position bias, and how do you control it? Answer: The judge prefers whichever answer it sees first (or second). Run each pair in both orders and only count consistent verdicts.

Put to work in: CAMS Lens — grounded answers that have to be right

Track 5 — Production architecture

Where AI engineering becomes engineering (~6 hours)

The gap between a working prototype and a system real users depend on is not model quality. It is latency, failure isolation, cost, caching and observability — ordinary distributed-systems work, applied to a component that is slow, expensive and occasionally wrong.

A production case study worth reading twice

You can reason about an AI system the way you'd reason about any distributed system.

streaming & stateful inference · full-duplex voice · asynchronous delegation · WebRTC · latency budgets · session affinity · prompt caching · failure isolation · observability

In short:

Separate the fast path from the slow path. The user-facing loop must answer within a latency budget; long work (search, tools, other agents) runs asynchronously and reports back.
Stream everything. Start speaking or rendering at the first token. Time-to-first-token matters more to how fast it feels than total generation time.
State has an address. Stateful sessions (voice, WebRTC) must reach the same server or share state. Session affinity and prompt caching keep them fast and cheap.

How we built a realtime system for responsive voice AI in six months — OpenAI (article, free, ~1h, rated 10/10). The single best architecture read on this page. The decision that matters: they isolated the real-time media path from slower tool and reasoning work, so a slow tool can never stall the audio stream. That one idea — keep the fast path free of the slow path — is what turns a mid-level interview answer into a senior one.

Build this — Instrument something you already built. Take the Track 3 agent. Add a latency budget per turn, a cost counter, provider failover, and a trace you can actually read when it misbehaves at 2am.

You're done when: You can draw your system's fast path and slow path as separate lines, and say what happens when the slow one stalls.

Common traps:

Putting a slow tool call on the critical path of a real-time reply.
Measuring average latency. Users feel the p95 — track it.

Check yourself:

Your voice agent must look up an order (3 seconds). What does the user hear meanwhile? Answer: An immediate acknowledgement from the fast path, while the lookup runs asynchronously and its result is spoken when it arrives.
Why does prompt caching matter for a long session? Answer: The long, unchanging prefix (system prompt, tools, early history) is reused instead of reprocessed, which cuts latency and input cost on every turn.

Put to work in: Jarvis — per-turn provider failover and streaming · Dialogflow CX voicebot — production conversational system

What happens when the provider is down

You can name every way your system fails and say what it does instead of failing.

timeouts · provider failover · graceful degradation · rate limits and backoff · approval gates · cost ceilings

In short:

Every call needs a timeout. A provider that hangs is worse than one that errors. Set timeouts per call and treat a timeout as a failure to route around.
Fail over, then degrade. Retry with backoff on rate limits, switch to a second provider on outages, and when everything fails, give a reduced answer rather than an error page.
Put ceilings in code. Cost per request, calls per user and actions that need approval are enforced by the system, not requested in the prompt.

There is no resource here on purpose — I have not read one that beats the lesson. This stage is learned the day a provider deprecates a model without telling you and every call in production starts returning 404. Write down, for your own system, what happens on: a timeout, a 429, a malformed reply, a tool that hangs, and a model id that no longer exists. If any answer is 'the request just fails', that is your next afternoon's work.

Build this — Break it on purpose. Point your primary provider at a dead endpoint and run your happy path. Then rate-limit it. Then make a tool hang for sixty seconds. Fix what you find, in that order — the fixes are a timeout, a backoff, and a fallback, and they are perhaps eighty lines between them.

You're done when: You have run your own system with its primary provider unplugged, and a user still got an answer.

Common traps:

Retrying instantly in a tight loop on a 429 — it makes the rate limit worse. Use exponential backoff with jitter.
Failing over to a model with a different prompt format or tool schema, untested. Run the fallback path in your evals too.

Check yourself:

Your primary LLM provider returns 503 for ten minutes. What happens to a user request? Answer: It times out quickly, fails over to a second provider (tested with the same evals), and if that fails, returns a clear degraded response — not a spinner.
Why add jitter to retry backoff? Answer: So thousands of clients don't retry at the same instant and knock the recovering service over again.

Put to work in: Jarvis — per-turn provider failover and streaming · OpsPilot — agents behind an approval gate

Explore more

Home — overview, skills and a built-in AI assistant
Experience — roles at Think360 AI (CAMS), CAMS Mutual Funds and IIT Delhi
Projects — GenAI, LLM, RAG and full-stack builds
Education — IIT Delhi, M.Tech & B.Tech Computer Science
Learn — a working AI engineer's roadmap: LLMs, RAG, agents, evals, production
GitHub Activity — open-source contributions
Contact / Hire me

GitHub · LinkedIn · Email