Evals, tracing & observability — The stage everyone skips
This is the one that separates a demo from a product. As a system gets complicated, evals and tracing stop being nice-to-have and become the only way you can change anything without breaking it. If you cannot state your system's accuracy as a number with a dataset behind it, you do not have a system — you have a vibe.
Track 4 of the AI engineering roadmap by K Laxman — about 5 hours across 1 stage.
Dataset → evaluator → task → interpretation
You can build an evaluation harness before you build the feature.
datasets · evaluators · LLM-as-judge and its failure modes · tracing · regression detection
- Why Evals Matter — LangSmith Evaluations, Part 1 — LangChain (video, free, ~0.4h, rated 10/10). Four pieces — dataset, evaluator, task, interpreting the result — and once you have that frame, every other eval tool is just an implementation of it. Part 1 is mandatory; carry on through the series as far as your patience holds.
Build this — Put a number on the RAG bot. Add the eval set and tracing to what you built in Track 2. Report the score in the README. Then change the chunk size and report it again — that delta is the whole discipline.
You're done when: You can say what your system's accuracy is, with a number and a dataset behind it.
Put to work in: CAMS Lens — grounded answers that have to be right
Explore more
- Home — overview, skills and a built-in AI assistant
- Experience — roles at Think360 AI (CAMS), CAMS Mutual Funds and IIT Delhi
- Projects — GenAI, LLM, RAG and full-stack builds
- Education — IIT Delhi, M.Tech & B.Tech Computer Science
- Learn — a working AI engineer's roadmap: LLMs, RAG, agents, evals, production
- GitHub Activity — open-source contributions
- Contact / Hire me