LLM fundamentals — Understand the thing everything else sits on

If you only ever do one item on this page, do the first one. Almost every confused conversation I have about AI systems traces back to someone never having watched this.

Track 1 of the AI engineering roadmap by K Laxman — about 6 hours across 2 stages.

The whole pipeline, end to end

You can trace a token from raw internet text all the way to a chat reply, naming every stage.

pretraining · tokenisation · transformers · inference · SFT · RLHF · hallucination · tool use · reasoning models

In short:

  1. Pretraining makes a document completer. Trained to predict the next token over a huge slice of the internet, the base model continues text. It does not follow instructions yet.
  2. Post-training makes an assistant. Supervised fine-tuning on example conversations, then reinforcement learning from human (or AI) feedback, turns the completer into something that answers.
  3. Hallucination is the default. The model always produces a fluent continuation. Unless training taught it to recognise what it doesn't know, a confident guess is the most likely text.

Build this — Re-implement decoding. Take any provider that returns logprobs and implement greedy, temperature and top-p sampling yourself. Write a README explaining, in your own words, when you'd use each. It is thirty lines of code and it makes §0.5 permanent.

You're done when: You can explain, without notes, why a model hallucinates confidently rather than saying 'I don't know'.

Common traps:

Check yourself:

  1. Why does a model hallucinate confidently instead of saying "I don't know"? Answer: It was trained to produce likely text, and in its training data questions are usually followed by confident answers. Saying "I don't know" has to be specifically trained in, and it only works for things the model can tell it doesn't know.
  2. What is the difference between a base model and an assistant model? Answer: The base model continues documents. The assistant is the same network after SFT and RLHF, trained to respond to instructions in a conversation format.

Put to work in: Jarvis — six interchangeable providers behind one loop · CAMSCode — enterprise AI coding assistant

Getting output your code can actually parse

You can make a model return JSON your program can trust, and you know what to do the one call in a thousand where it doesn't.

JSON schema · constrained decoding · refusals · repair passes · streaming partials

In short:

  1. Ask for a schema, not "JSON please". Provider structured-output modes constrain decoding to your JSON schema, so the reply is valid JSON of the right shape — not just valid-looking text.
  2. Valid is not correct. The schema guarantees shape. It says nothing about whether the values are true. Validate the business rules in your own code.
  3. Plan for the one in a thousand. Refusals, truncation at the token limit and schema features the provider doesn't support all still happen. Parse, and on failure repair once or fail cleanly.

Build this — A parse-or-repair loop. Ask for a schema, parse the reply, and on a parse failure send the error text back for one repair attempt before giving up. Log how often the repair fires. That number is the difference between a demo and something you can put in front of a customer.

You're done when: You can say what your system does when the model returns prose instead of JSON — and it isn't 'crash'.

Common traps:

Check yourself:

  1. The model returns prose instead of JSON. What does your system do? Answer: Catches the parse error, sends the error text back for one repair attempt, and if that also fails, returns a clear error or fallback — logged and counted.
  2. The JSON parses and matches the schema. What can still be wrong? Answer: The values. A date can be in the wrong year, a total can fail to add up. Schema validation checks shape; your own checks cover meaning.

Put to work in: CAMSCode — enterprise AI coding assistant · Intelligent document processing

Explore more

GitHub · LinkedIn · Email