AI — Understand the model, then ground it in your own data

The most common job going, and the one with the widest gap between a demo and something you would put in front of a customer. Understand what the model actually is, then learn to feed it your data without it inventing things.

Who this is for: You want to build products on top of LLMs.

3 tracks · 21 hours · one of three paths in the learning roadmap by K Laxman. An optional 4-hour prelude, Just enough to not be bluffing, covers the underlying ideas first.

Track 1 — LLM fundamentals

Understand the thing everything else sits on (~6 hours)

If you only ever do one item on this page, do the first one. Almost every confused conversation I have about AI systems traces back to someone never having watched this.

The whole pipeline, end to end

You can trace a token from raw internet text all the way to a chat reply, naming every stage.

pretraining · tokenisation · transformers · inference · SFT · RLHF · hallucination · tool use · reasoning models

In short:

  1. Pretraining makes a document completer. Trained to predict the next token over a huge slice of the internet, the base model continues text. It does not follow instructions yet.
  2. Post-training makes an assistant. Supervised fine-tuning on example conversations, then reinforcement learning from human (or AI) feedback, turns the completer into something that answers.
  3. Hallucination is the default. The model always produces a fluent continuation. Unless training taught it to recognise what it doesn't know, a confident guess is the most likely text.

Build this — Re-implement decoding. Take any provider that returns logprobs and implement greedy, temperature and top-p sampling yourself. Write a README explaining, in your own words, when you'd use each. It is thirty lines of code and it makes §0.5 permanent.

You're done when: You can explain, without notes, why a model hallucinates confidently rather than saying 'I don't know'.

Common traps:

Check yourself:

  1. Why does a model hallucinate confidently instead of saying "I don't know"? Answer: It was trained to produce likely text, and in its training data questions are usually followed by confident answers. Saying "I don't know" has to be specifically trained in, and it only works for things the model can tell it doesn't know.
  2. What is the difference between a base model and an assistant model? Answer: The base model continues documents. The assistant is the same network after SFT and RLHF, trained to respond to instructions in a conversation format.

Put to work in: Jarvis — six interchangeable providers behind one loop · CAMSCode — enterprise AI coding assistant

Getting output your code can actually parse

You can make a model return JSON your program can trust, and you know what to do the one call in a thousand where it doesn't.

JSON schema · constrained decoding · refusals · repair passes · streaming partials

In short:

  1. Ask for a schema, not "JSON please". Provider structured-output modes constrain decoding to your JSON schema, so the reply is valid JSON of the right shape — not just valid-looking text.
  2. Valid is not correct. The schema guarantees shape. It says nothing about whether the values are true. Validate the business rules in your own code.
  3. Plan for the one in a thousand. Refusals, truncation at the token limit and schema features the provider doesn't support all still happen. Parse, and on failure repair once or fail cleanly.

Build this — A parse-or-repair loop. Ask for a schema, parse the reply, and on a parse failure send the error text back for one repair attempt before giving up. Log how often the repair fires. That number is the difference between a demo and something you can put in front of a customer.

You're done when: You can say what your system does when the model returns prose instead of JSON — and it isn't 'crash'.

Common traps:

Check yourself:

  1. The model returns prose instead of JSON. What does your system do? Answer: Catches the parse error, sends the error text back for one repair attempt, and if that also fails, returns a clear error or fallback — logged and counted.
  2. The JSON parses and matches the schema. What can still be wrong? Answer: The values. A date can be in the wrong year, a total can fail to add up. Schema validation checks shape; your own checks cover meaning.

Put to work in: CAMSCode — enterprise AI coding assistant · Intelligent document processing

Track 2 — Retrieval-Augmented Generation

The part most people do badly (~10 hours)

Everyone can build a RAG demo in an afternoon. Almost nobody can tell you why theirs returns the wrong chunk. The difference between those two people is this track — and the eval set at the end of it.

RAG properly — all fifteen techniques

You can name, and choose between, the retrieval strategies. Not just 'embed, store, search'.

indexing · retrieval · generation · multi-query · RAG-Fusion · query decomposition · step-back prompting · HyDE · routing · query construction · multi-representation indexing · RAPTOR · ColBERT · CRAG · adaptive RAG

In short:

  1. Three stages, three failure points. Indexing (split and embed), retrieval (find the right chunks) and generation (answer from them). A wrong answer came from exactly one of them.
  2. Most techniques fix the query or the index. Multi-query, HyDE, step-back and decomposition rewrite the question; multi-representation indexing and RAPTOR change what gets stored. Each fixes a specific miss.
  3. Corrective and adaptive RAG add judgement. Grade the retrieved chunks before answering, and re-retrieve or fall back when they are weak, instead of generating from whatever came back.

Build this — RAG over your own notes — plus a 30-question eval set. The bot is the easy half. The assignment is the eval set: thirty real questions with known answers, and a score you can quote. Without it you have a demo, and you will not be able to tell whether tomorrow's change helped.

You're done when: Given a wrong answer, you can separate retrieval failure from generation failure in under a minute.

Common traps:

Check yourself:

  1. An answer is wrong. What is the one-minute test that tells you where? Answer: Look at the retrieved chunks. If the answer isn't in them, it's a retrieval failure. If it is and the model still got it wrong, it's a generation failure.
  2. When does HyDE help? Answer: When questions are short and phrased very differently from the documents. Embedding a hypothetical answer lands closer to real answers than the bare question does.

Put to work in: CAMS Lens — SEBI-grounded compliance intelligence · Agentic Research & Decision Intelligence Assistant

Chunking — the decision that quietly decides everything

You can explain why your chunk size is what it is, instead of because a tutorial said 512.

fixed vs semantic splitting · overlap · chunk size vs recall · metadata on the chunk · contextual retrieval

Interactive explainer on the live page: context budget — You can explain why your chunk size is what it is, instead of because a tutorial said 512.

In short:

  1. The chunk is the unit of retrieval. You can only retrieve what you stored. If the answer is split across two chunks, or buried in a huge one, retrieval can't deliver it cleanly.
  2. Size is a trade-off. Small chunks match precisely but lose context; large chunks keep context but dilute the embedding. Overlap reduces boundary losses at the cost of duplication.
  3. Give each chunk its context. Prepending the document title, section heading or a short generated summary (contextual retrieval) makes a chunk findable on its own.

Build this — Same questions, four chunkings. Take the eval set from the stage above and run it against four indexes: 256 fixed, 512 fixed, 512 with overlap, and semantic. One table, four scores. You will never again accept a chunk size on faith, and you now own a number you can quote in an interview.

You're done when: You can name the chunk size you use, the one you tried before it, and the score difference between them.

Common traps:

Check yourself:

  1. Your chunks say "the policy above applies" and retrieval returns them useless. What is the fix? Answer: Add context to each chunk — document and section titles, or a generated one-line summary — so it stands on its own when retrieved.
  2. How do you choose between 300-token and 1,000-token chunks? Answer: Index both and run the same questions against each. Compare recall@k and answer quality; pick with numbers, not intuition.

Put to work in: CAMS Lens — SEBI-grounded compliance intelligence

When retrieval is the bug

You can tell a retrieval problem from a generation problem before you touch the prompt.

hybrid search · reranking · recall@k · query rewriting · the missing-document case

In short:

  1. Measure recall@k first. For a set of questions with known source documents, how often is the right one in the top k results? Until you know that, prompt tweaks are guesses.
  2. Hybrid search covers both kinds of miss. Keyword search (BM25) catches exact terms and identifiers; vectors catch paraphrase. Combining them fixes misses either one makes alone.
  3. Rerank a wide net. Retrieve 50 cheaply, then let a cross-encoder reranker pick the best 5. The reranker reads query and chunk together, which is far more precise.

Build this — Measure recall@k before you tune anything. For each question in your eval set, record whether the answer-bearing chunk appeared in the top k at all. If it didn't, no prompt on earth was going to save that answer. Split your failures into 'never retrieved' and 'retrieved and ignored' — they have completely different fixes.

You're done when: You have a recall@k number for your own corpus, and you fixed retrieval before you touched the prompt.

Common traps:

Check yourself:

  1. Users search for error code "E4031" and get nothing useful. Why, and what fixes it? Answer: Embeddings don't represent exact identifiers well. Add keyword search (hybrid) so exact matches get found.
  2. Recall@5 is 60% but recall@50 is 95%. What do you add? Answer: A reranker. The right documents are being found; they just aren't ranked high enough. Rerank the top 50 down to 5.

Put to work in: CAMS Lens — SEBI-grounded compliance intelligence · Agentic Research & Decision Intelligence Assistant

Track 4 — Evals, tracing & observability

The stage everyone skips (~5 hours)

This is the one that separates a demo from a product. As a system gets complicated, evals and tracing stop being nice-to-have and become the only way you can change anything without breaking it. If you cannot state your system's accuracy as a number with a dataset behind it, you do not have a system — you have a vibe.

Dataset → evaluator → task → interpretation

You can build an evaluation harness before you build the feature.

datasets · evaluators · LLM-as-judge and its failure modes · tracing · regression detection

In short:

  1. An eval is a dataset plus a grader. Inputs with expected outputs or criteria, run through your system, scored the same way every time. Without one, every change is a guess.
  2. Start small and real. Thirty real questions with checked answers beat a thousand synthetic ones. Add every production failure to the set as you find it.
  3. Trace everything. Record each run's prompts, retrieved chunks, tool calls and outputs. When a score drops, the trace shows which step changed.

Build this — Put a number on the RAG bot. Add the eval set and tracing to what you built in Track 2. Report the score in the README. Then change the chunk size and report it again — that delta is the whole discipline.

You're done when: You can say what your system's accuracy is, with a number and a dataset behind it.

Common traps:

Check yourself:

  1. "The new prompt feels better." What do you ask for? Answer: The score on the eval set before and after, on the same data, and the examples that got worse.
  2. Where should new eval examples come from once you're live? Answer: Production failures — user reports, low ratings and traced errors — added to the dataset so they can't regress silently.

Put to work in: CAMS Lens — grounded answers that have to be right

LLM-as-judge — and where it stops being trustworthy

You can grade an open-ended answer automatically, and you know which of those grades to disbelieve.

pairwise vs pointwise · position bias · agreement with humans · error analysis · when to just read the outputs

In short:

  1. A judge is a model you must evaluate. Label a sample yourself and measure how often the judge agrees. Until you have that number, the judge's scores are unverified.
  2. Pairwise is steadier than pointwise. "Which of these two is better?" is more reliable than "rate this 1–10". Swap the order and check again to cancel position bias.
  3. Binary criteria beat vague scales. "Does the answer cite a source from the context? yes/no" is far easier to judge consistently than "rate helpfulness".

Build this — Grade your judge. Hand-label thirty of your own outputs. Have the judge label the same thirty. Report the agreement rate. Below about 80% the judge is telling you a story, and you have just learned that for the price of an afternoon.

You're done when: You can quote the agreement rate between your judge and your own labels — and you know which failure category it gets wrong.

Common traps:

Check yourself:

  1. How do you know your judge is trustworthy? Answer: Hand-label 50–100 examples and measure agreement with the judge. Then read the disagreements to see which failure type it misses.
  2. What is position bias, and how do you control it? Answer: The judge prefers whichever answer it sees first (or second). Run each pair in both orders and only count consistent verdicts.

Put to work in: CAMS Lens — grounded answers that have to be right

Explore more

GitHub · LinkedIn · Email