Agentic AI Engineering — A model, in a loop, with tools — and the harness that makes it reliable

Where I spend my own time, and where the marketing is furthest from the engineering. It is a while loop with an approval gate — the hard parts are the tools, the evaluation and the failure isolation, not the magic. The job title for that work is harness engineering, and it is the shift: the model is bought in, the harness is what you build.

Who this is for: You want the thing to do the work, not just answer.

4 tracks · 34 hours · one of three paths in the learning roadmap by K Laxman. An optional 4-hour prelude, Just enough to not be bluffing, covers the underlying ideas first.

Track 1 — LLM fundamentals

Understand the thing everything else sits on (~6 hours)

If you only ever do one item on this page, do the first one. Almost every confused conversation I have about AI systems traces back to someone never having watched this.

The whole pipeline, end to end

You can trace a token from raw internet text all the way to a chat reply, naming every stage.

pretraining · tokenisation · transformers · inference · SFT · RLHF · hallucination · tool use · reasoning models

In short:

  1. Pretraining makes a document completer. Trained to predict the next token over a huge slice of the internet, the base model continues text. It does not follow instructions yet.
  2. Post-training makes an assistant. Supervised fine-tuning on example conversations, then reinforcement learning from human (or AI) feedback, turns the completer into something that answers.
  3. Hallucination is the default. The model always produces a fluent continuation. Unless training taught it to recognise what it doesn't know, a confident guess is the most likely text.

Build this — Re-implement decoding. Take any provider that returns logprobs and implement greedy, temperature and top-p sampling yourself. Write a README explaining, in your own words, when you'd use each. It is thirty lines of code and it makes §0.5 permanent.

You're done when: You can explain, without notes, why a model hallucinates confidently rather than saying 'I don't know'.

Common traps:

Check yourself:

  1. Why does a model hallucinate confidently instead of saying "I don't know"? Answer: It was trained to produce likely text, and in its training data questions are usually followed by confident answers. Saying "I don't know" has to be specifically trained in, and it only works for things the model can tell it doesn't know.
  2. What is the difference between a base model and an assistant model? Answer: The base model continues documents. The assistant is the same network after SFT and RLHF, trained to respond to instructions in a conversation format.

Put to work in: Jarvis — six interchangeable providers behind one loop · CAMSCode — enterprise AI coding assistant

Getting output your code can actually parse

You can make a model return JSON your program can trust, and you know what to do the one call in a thousand where it doesn't.

JSON schema · constrained decoding · refusals · repair passes · streaming partials

In short:

  1. Ask for a schema, not "JSON please". Provider structured-output modes constrain decoding to your JSON schema, so the reply is valid JSON of the right shape — not just valid-looking text.
  2. Valid is not correct. The schema guarantees shape. It says nothing about whether the values are true. Validate the business rules in your own code.
  3. Plan for the one in a thousand. Refusals, truncation at the token limit and schema features the provider doesn't support all still happen. Parse, and on failure repair once or fail cleanly.

Build this — A parse-or-repair loop. Ask for a schema, parse the reply, and on a parse failure send the error text back for one repair attempt before giving up. Log how often the repair fires. That number is the difference between a demo and something you can put in front of a customer.

You're done when: You can say what your system does when the model returns prose instead of JSON — and it isn't 'crash'.

Common traps:

Check yourself:

  1. The model returns prose instead of JSON. What does your system do? Answer: Catches the parse error, sends the error text back for one repair attempt, and if that also fails, returns a clear error or fallback — logged and counted.
  2. The JSON parses and matches the schema. What can still be wrong? Answer: The values. A date can be in the wrong year, a total can fail to add up. Schema validation checks shape; your own checks cover meaning.

Put to work in: CAMSCode — enterprise AI coding assistant · Intelligent document processing

Track 3 — Agents & tools

A model, in a loop, with tools. That's it. (~17 hours)

This is the track I care most about, because it is the one where the industry's marketing is furthest from the engineering. An agent is not a mysterious autonomous intelligence. It is a while loop. Write one by hand and the mystique never comes back — and then build the harness around it, which is where the actual work has moved.

The agent loop, by hand

You can write an agent without a framework and explain every line of it.

environment · tool schemas · the while loop · context accumulation · approval gates

Interactive explainer on the live page: agent loop — You can write an agent without a framework and explain every line of it.

In short:

  1. An agent is a while loop. Call the model; if it asks for a tool, run it, append the result to the conversation, and call again. Stop when it answers without a tool call.
  2. The context is the agent's memory. Every tool result goes back into the context. The model knows only what the loop has put there so far.
  3. You own the stopping conditions. A step limit, a cost ceiling and approval gates for risky actions belong in your code, not in the prompt.

Build this — Write the loop. No framework.. A while loop, three tools, JSON-schema arguments, one approval gate before anything destructive. This is the keystone exercise of the whole page — do not substitute a framework tutorial for it.

You're done when: You can draw the loop on a whiteboard and point at exactly where a tool result re-enters the context.

Common traps:

Check yourself:

  1. Where exactly does a tool result re-enter the loop? Answer: It is appended to the message history as a tool-result message, and the whole history is sent on the next model call.
  2. When should you NOT build an agent? Answer: When you can write the steps down in advance. Use a fixed pipeline, and keep agents for tasks where the path genuinely depends on what is found along the way.

Put to work in: Jarvis — that loop, in production, running live in your browser · OpsPilot — agentic request triage & remediation

Workflows and graph loops — the five patterns

You can pick the simplest pattern that solves a task, and draw any agent system as a graph of steps with a loop in it.

prompt chaining · routing · parallelization · orchestrator-workers · evaluator-optimizer · state, nodes and edges · conditional edges and cycles · checkpointing

In short:

  1. Five patterns cover most systems. Prompt chaining (fixed steps), routing (classify, then specialise), parallelization (split or vote), orchestrator-workers (a model plans the subtasks) and evaluator-optimizer (generate, critique, revise).
  2. A graph is the loop, drawn. Nodes are steps, edges are control flow, and state is what travels between them. An agent is a graph with a conditional edge pointing back to an earlier node.
  3. Every cycle needs an exit. A loop edge must have a condition that ends it — a passing check, a round limit, a budget. Without one, the graph is an infinite loop with a bill attached.

Build this — One task, two patterns, a number for each. Take a writing or extraction task. Solve it once as a prompt chain and once as an evaluator-optimizer loop (generate → critique → revise, with a round limit). Run both on the same 20 inputs and record quality, latency and cost. Then draw the second one as a graph: which edge is the loop, and what stops it?

You're done when: Given a new task, you can say which pattern fits, why the simpler ones don't, and where the loop's exit condition lives.

Common traps:

Check yourself:

  1. Customer emails go to billing, technical or sales, each with its own prompt. Which pattern is that? Answer: Routing — classify the input, then send it to a specialised prompt or model.
  2. What turns a workflow graph into an agent? Answer: A cycle whose next step the model decides: a conditional edge back to the model node, taken until the model chooses to stop (or a limit stops it).

Put to work in: Research Intelligence Agent — orchestration in practice · OpsPilot — routing and remediation workflows

Tools and MCP — connecting agents to real systems

You can design a tool an LLM actually uses correctly, and expose it over MCP.

tool design · argument validation · MCP architecture · tool servers · when a tool is too broad

In short:

  1. The tool description is a prompt. The model chooses tools and fills arguments from the name, description and schema alone. Write them for a new colleague with no other context.
  2. Narrow tools, typed arguments. One clear job per tool, enums instead of free strings where possible, and validation on every call — the model's arguments are untrusted input.
  3. MCP standardises the plumbing. An MCP server exposes tools and resources once; any MCP-capable client (desktop apps, IDEs, your own agent) can then use them without custom glue.

Build this — Wrap your Track 2 RAG bot as an MCP server. One tool, a real schema, connected to a client you didn't write. The moment someone else's agent can call your thing, tool design stops being theoretical.

You're done when: You can say why a tool named `do_stuff(input: string)` will be called wrongly, and rewrite it so it won't be.

Common traps:

Check yourself:

  1. Why will `do_stuff(input: string)` get called wrongly? Answer: The name and schema tell the model nothing about when to call it or what to pass. Split it into specific tools with typed, described parameters.
  2. What does MCP give you that a function call in your own code doesn't? Answer: Reuse: the same server works across every MCP client, and tools can be added or updated without changing the agent's code.

Put to work in: Intelligent Document Processing — multi-agent system

Harness engineering — the part that isn't the model

You can say what a harness is, name its five pieces, and point at the one your own agent is missing.

guides vs sensors · context assembly · tool orchestration · verification loops · cost envelopes · behavioural evals

Interactive explainer on the live page: harness — You can say what a harness is, name its five pieces, and point at the one your own agent is missing.

In short:

  1. The harness is everything but the model. Context assembly, tool orchestration, verification, limits and logging. It decides what the model sees and what happens to what it produces.
  2. Guides steer, sensors check. Guides (instructions, examples, conventions) shape behaviour before the action; sensors (tests, linters, validators) check the result after it.
  3. Close the loop with verification. Let the agent run the check itself — the test suite, the type checker — and feed failures back. That catches errors no prompt prevents.

The shift this track has been building toward. Prompt engineering was how you got one good answer out of a 4k context window; it stopped scaling the moment a task ran longer than a few hundred lines. What replaced it is not a better prompt — it is everything around the model: what you put in the context, which tools it can reach, what checks its work, what stops it spending your money, and what you can read afterwards when it went wrong. Swapping models moves output quality by ten or fifteen per cent. Changing the harness decides whether the system works at all.

Build this — Put a harness around the loop you already wrote. Take your Track 3 agent and add the five pieces one at a time, measuring after each: a conventions file it always reads, a deterministic check it cannot talk its way past, one verification gate that re-runs its work before the result counts, a per-task token ceiling that kills the run, and a trace you can read afterwards. Keep the model fixed the whole time — the point is to see how far the harness alone moves it.

You're done when: You can point at one failure your harness now catches that the model used to get away with — and say which of the five parts caught it.

Common traps:

Check yourself:

  1. Your coding agent keeps breaking imports. Guide or sensor? Answer: Both, but the sensor matters more: run the build or type check after each edit and feed errors back. A prompt rule alone will be ignored eventually.
  2. Name the five parts of the harness in the explainer above. Answer: Context (what the model sees), tools (what it can do), verify (checks on what it produced), budget (steps and cost it may spend), and trace (a record of every run).

Put to work in: Jarvis — tool allow-lists, call budgets and a visible trace

Multi-agent systems — when one context window isn't enough

You can tell when splitting work across agents pays for itself, and build an orchestrator whose sub-agents don't trip over each other.

orchestrator and sub-agents · parallel context windows · token cost vs quality · task locking · shared state via git or files · outcome-based evaluation

In short:

  1. Sub-agents buy context, not intelligence. Each sub-agent explores in its own clean context window and returns a short summary. The win is breadth and parallelism on tasks too big for one window.
  2. It costs a multiple of the tokens. Anthropic reports multi-agent research using roughly 15× the tokens of a chat. It pays off only when the task is valuable and genuinely splits into independent parts.
  3. Coordination is the hard part. Agents need clear task boundaries, a way to claim work (a lock file, a queue) and shared state they can merge (git, files). Most failures are two agents doing the same thing.

Build this — An orchestrator with three researchers. Build a lead agent that splits a research question into three sub-questions, runs three sub-agents in parallel with separate contexts, and merges their cited summaries. Compare against a single agent on 10 questions: quality (judged against your own rubric), tokens and wall-clock time. Keep the multi-agent version only if the numbers justify the cost.

You're done when: You can quote what your multi-agent version costs in tokens compared with one agent, and what it buys.

Common traps:

Check yourself:

  1. When is a single agent the better choice? Answer: When the subtasks depend on each other, share most of their context, or the task fits in one window. Parallel agents add cost and coordination for no gain there.
  2. Two parallel agents keep fixing the same bug. What do you add? Answer: A way to claim tasks — a lock file or queue entry per task — checked before starting work, plus merging shared state often.

Put to work in: Research Intelligence Agent — agentic research & decision support

Frameworks — deliberately last

You choose a framework because it removes work you understand, not because you're lost.

LangGraph and friends · what they abstract · what they hide

In short:

  1. A framework hides the loop. LangGraph and similar libraries give you state, retries, persistence and graphs of steps. They are the same loop you wrote, with decisions made for you.
  2. Adopt for features, not for learning. Durable state, human-in-the-loop pauses and tracing are real reasons. "I don't know how agents work" is not — that is what the previous stages fixed.
  3. Know the escape hatch. Before committing, find out how to see the exact prompt sent and how to override one step. Debugging something you can't see is the real cost.

Framework material comes last on purpose. Read one only after you have written the loop, the patterns and a harness yourself. You will find you already know what every abstraction is for — and you will notice the ones that are solving a problem you don't have.

You're done when: You can name one thing your chosen framework does that you would rather do yourself, and say why.

Common traps:

Check yourself:

  1. Name one good reason to adopt a framework. Answer: Durable, resumable state for long workflows, or built-in human-approval pauses — features that are genuinely tedious to build and test yourself.
  2. What should you check in your framework before production? Answer: That you can see and log the exact prompt and tool calls for any run, and override any single step without forking the library.

Track 4 — Evals, tracing & observability

The stage everyone skips (~5 hours)

This is the one that separates a demo from a product. As a system gets complicated, evals and tracing stop being nice-to-have and become the only way you can change anything without breaking it. If you cannot state your system's accuracy as a number with a dataset behind it, you do not have a system — you have a vibe.

Dataset → evaluator → task → interpretation

You can build an evaluation harness before you build the feature.

datasets · evaluators · LLM-as-judge and its failure modes · tracing · regression detection

In short:

  1. An eval is a dataset plus a grader. Inputs with expected outputs or criteria, run through your system, scored the same way every time. Without one, every change is a guess.
  2. Start small and real. Thirty real questions with checked answers beat a thousand synthetic ones. Add every production failure to the set as you find it.
  3. Trace everything. Record each run's prompts, retrieved chunks, tool calls and outputs. When a score drops, the trace shows which step changed.

Build this — Put a number on the RAG bot. Add the eval set and tracing to what you built in Track 2. Report the score in the README. Then change the chunk size and report it again — that delta is the whole discipline.

You're done when: You can say what your system's accuracy is, with a number and a dataset behind it.

Common traps:

Check yourself:

  1. "The new prompt feels better." What do you ask for? Answer: The score on the eval set before and after, on the same data, and the examples that got worse.
  2. Where should new eval examples come from once you're live? Answer: Production failures — user reports, low ratings and traced errors — added to the dataset so they can't regress silently.

Put to work in: CAMS Lens — grounded answers that have to be right

LLM-as-judge — and where it stops being trustworthy

You can grade an open-ended answer automatically, and you know which of those grades to disbelieve.

pairwise vs pointwise · position bias · agreement with humans · error analysis · when to just read the outputs

In short:

  1. A judge is a model you must evaluate. Label a sample yourself and measure how often the judge agrees. Until you have that number, the judge's scores are unverified.
  2. Pairwise is steadier than pointwise. "Which of these two is better?" is more reliable than "rate this 1–10". Swap the order and check again to cancel position bias.
  3. Binary criteria beat vague scales. "Does the answer cite a source from the context? yes/no" is far easier to judge consistently than "rate helpfulness".

Build this — Grade your judge. Hand-label thirty of your own outputs. Have the judge label the same thirty. Report the agreement rate. Below about 80% the judge is telling you a story, and you have just learned that for the price of an afternoon.

You're done when: You can quote the agreement rate between your judge and your own labels — and you know which failure category it gets wrong.

Common traps:

Check yourself:

  1. How do you know your judge is trustworthy? Answer: Hand-label 50–100 examples and measure agreement with the judge. Then read the disagreements to see which failure type it misses.
  2. What is position bias, and how do you control it? Answer: The judge prefers whichever answer it sees first (or second). Run each pair in both orders and only count consistent verdicts.

Put to work in: CAMS Lens — grounded answers that have to be right

Track 5 — Production architecture

Where AI engineering becomes engineering (~6 hours)

The gap between a working prototype and a system real users depend on is not model quality. It is latency, failure isolation, cost, caching and observability — ordinary distributed-systems work, applied to a component that is slow, expensive and occasionally wrong.

A production case study worth reading twice

You can reason about an AI system the way you'd reason about any distributed system.

streaming & stateful inference · full-duplex voice · asynchronous delegation · WebRTC · latency budgets · session affinity · prompt caching · failure isolation · observability

In short:

  1. Separate the fast path from the slow path. The user-facing loop must answer within a latency budget; long work (search, tools, other agents) runs asynchronously and reports back.
  2. Stream everything. Start speaking or rendering at the first token. Time-to-first-token matters more to how fast it feels than total generation time.
  3. State has an address. Stateful sessions (voice, WebRTC) must reach the same server or share state. Session affinity and prompt caching keep them fast and cheap.

Build this — Instrument something you already built. Take the Track 3 agent. Add a latency budget per turn, a cost counter, provider failover, and a trace you can actually read when it misbehaves at 2am.

You're done when: You can draw your system's fast path and slow path as separate lines, and say what happens when the slow one stalls.

Common traps:

Check yourself:

  1. Your voice agent must look up an order (3 seconds). What does the user hear meanwhile? Answer: An immediate acknowledgement from the fast path, while the lookup runs asynchronously and its result is spoken when it arrives.
  2. Why does prompt caching matter for a long session? Answer: The long, unchanging prefix (system prompt, tools, early history) is reused instead of reprocessed, which cuts latency and input cost on every turn.

Put to work in: Jarvis — per-turn provider failover and streaming · Dialogflow CX voicebot — production conversational system

What happens when the provider is down

You can name every way your system fails and say what it does instead of failing.

timeouts · provider failover · graceful degradation · rate limits and backoff · approval gates · cost ceilings

In short:

  1. Every call needs a timeout. A provider that hangs is worse than one that errors. Set timeouts per call and treat a timeout as a failure to route around.
  2. Fail over, then degrade. Retry with backoff on rate limits, switch to a second provider on outages, and when everything fails, give a reduced answer rather than an error page.
  3. Put ceilings in code. Cost per request, calls per user and actions that need approval are enforced by the system, not requested in the prompt.

There is no resource here on purpose — I have not read one that beats the lesson. This stage is learned the day a provider deprecates a model without telling you and every call in production starts returning 404. Write down, for your own system, what happens on: a timeout, a 429, a malformed reply, a tool that hangs, and a model id that no longer exists. If any answer is 'the request just fails', that is your next afternoon's work.

Build this — Break it on purpose. Point your primary provider at a dead endpoint and run your happy path. Then rate-limit it. Then make a tool hang for sixty seconds. Fix what you find, in that order — the fixes are a timeout, a backoff, and a fallback, and they are perhaps eighty lines between them.

You're done when: You have run your own system with its primary provider unplugged, and a user still got an answer.

Common traps:

Check yourself:

  1. Your primary LLM provider returns 503 for ten minutes. What happens to a user request? Answer: It times out quickly, fails over to a second provider (tested with the same evals), and if that fails, returns a clear degraded response — not a spinner.
  2. Why add jitter to retry backoff? Answer: So thousands of clients don't retry at the same instant and knock the recovering service over again.

Put to work in: Jarvis — per-turn provider failover and streaming · OpsPilot — agents behind an approval gate

Explore more

GitHub · LinkedIn · Email