ML — The classical craft — rows, columns, and not fooling yourself

Quieter than the AI headlines and still what most companies actually need. A table, a column you want to predict, and the discipline to know whether it worked. Boosted trees have not stopped winning just because language models arrived.

Who this is for: Your data lives in a database, not in documents.

2 tracks · 15 hours · one of three paths in the learning roadmap by K Laxman. An optional 4-hour prelude, Just enough to not be bluffing, covers the underlying ideas first.

Track 4 — Classic ML

Still what wins on the data most companies actually have (~10 hours)

The half of the job the AI headlines skipped, and the half that quietly pays for most ML salaries: a table of rows, a column you want to predict, and the discipline not to fool yourself about how well it worked. Boosted trees did not stop winning on tabular data just because language models arrived — if your data lives in a database rather than in documents, this is the work.

Supervised learning, end to end

You can go from a CSV to a model with an honest score, and say what the score means.

train/validation/test · baselines · choosing the metric · cross-validation · over- and underfitting

In short:

  1. Three splits, three jobs. Train to fit, validation to choose settings, test to report — once, at the end. Reuse the test set for decisions and it stops being a test.
  2. Beat a baseline first. Predict the majority class, or last week's value. If your model can't clearly beat that, it isn't learning anything useful.
  3. The metric is a product decision. Accuracy, precision, recall and cost-weighted error reward different mistakes. Pick the one that matches what a wrong answer costs.

Build this — A model with a baseline you have to beat. Pick any tabular dataset. Write down the dumbest possible predictor first — the mean, the majority class — and score it. Then beat it, and report both numbers. If you cannot beat the dumb one, that is the finding.

You're done when: You can say why a 99%-accurate model can be worthless, without looking it up.

Common traps:

Check yourself:

  1. Why can a 99%-accurate fraud model be worthless? Answer: If 1% of transactions are fraud, predicting "not fraud" every time scores 99%. Look at recall and precision on the fraud class.
  2. What is the validation set for, if you have a test set? Answer: Making choices — hyperparameters, features, which model. The test set is kept untouched so the final number is honest.

Put to work in: Driver Drowsiness Detection

Leakage, imbalance, and the ways you fool yourself

You can spot the mistakes that make a model look brilliant offline and useless in production.

data leakage · time-based splits · class imbalance · distribution shift · error analysis

In short:

  1. Leakage is information from the future. A feature that wouldn't exist at prediction time (an outcome field, a post-event timestamp) makes offline scores look great and production scores collapse.
  2. Split on time when time matters. If the real system predicts next month from this month, validate the same way. Random splits let the model peek ahead.
  3. Read the errors. Error analysis — looking at 50 wrong predictions by hand — finds leakage, label noise and shifts that no aggregate metric will show.

There is no single video for this one, and that is telling: it is the part of ML that gets learned by being burned. The two habits that matter — split on time whenever the real system will predict the future, and look at fifty actual errors before you touch the model — will save you more than any algorithm choice.

Build this — Break your own model on purpose. Take the model from the previous stage and introduce a leaked feature deliberately. Watch the score jump. That jump is exactly what a subtle leak looks like, and now you know its shape.

You're done when: Given a suspiciously good result, leakage is the first thing you check — and you know where to look.

Common traps:

Check yourself:

  1. Your churn model scores 0.99 AUC. What do you check first? Answer: Leakage. Look for features that are only known after the customer churned — cancellation date, final bill, support tickets about closing.
  2. Why is a random split wrong for forecasting next month's demand? Answer: Training rows from after the test rows let the model use the future. Train on the past and test on a later period.

Trees, boosting — and when to reach for deep learning

You choose a model class on the shape of the data, not on what is fashionable.

decision trees · random forests · gradient boosting · when deep learning wins · when it does not

In short:

  1. Tables → gradient boosting. On structured, tabular data, gradient-boosted trees (XGBoost, LightGBM, CatBoost) are usually the strongest and cheapest choice.
  2. Images, audio, text → deep learning. Neural networks win where the raw input is unstructured and the features must be learned rather than engineered.
  3. Pretrained beats from-scratch. With little data, fine-tuning a pretrained model — or using an LLM or embedding model directly — beats training a network from zero.

Build this — Same dataset, three model classes. Boosted trees, a linear model, and a small neural net on identical splits. Report all three, plus training time. The answer on tabular data will usually surprise anyone who only reads AI news.

You're done when: Asked whether to use deep learning, your first question is about the data, not the model.

Common traps:

Check yourself:

  1. Tabular data, 50k rows, 30 columns. What do you try first? Answer: A gradient-boosted tree model, against a simple baseline. Deep learning rarely wins here, and costs far more to tune.
  2. Someone asks "should we use deep learning?" What is your first question? Answer: What is the data — its type and how much of it is labelled. That decides the model class far more than the model does.

Put to work in: Table Recognition System

Track 4 — Evals, tracing & observability

The stage everyone skips (~5 hours)

This is the one that separates a demo from a product. As a system gets complicated, evals and tracing stop being nice-to-have and become the only way you can change anything without breaking it. If you cannot state your system's accuracy as a number with a dataset behind it, you do not have a system — you have a vibe.

Dataset → evaluator → task → interpretation

You can build an evaluation harness before you build the feature.

datasets · evaluators · LLM-as-judge and its failure modes · tracing · regression detection

In short:

  1. An eval is a dataset plus a grader. Inputs with expected outputs or criteria, run through your system, scored the same way every time. Without one, every change is a guess.
  2. Start small and real. Thirty real questions with checked answers beat a thousand synthetic ones. Add every production failure to the set as you find it.
  3. Trace everything. Record each run's prompts, retrieved chunks, tool calls and outputs. When a score drops, the trace shows which step changed.

Build this — Put a number on the RAG bot. Add the eval set and tracing to what you built in Track 2. Report the score in the README. Then change the chunk size and report it again — that delta is the whole discipline.

You're done when: You can say what your system's accuracy is, with a number and a dataset behind it.

Common traps:

Check yourself:

  1. "The new prompt feels better." What do you ask for? Answer: The score on the eval set before and after, on the same data, and the examples that got worse.
  2. Where should new eval examples come from once you're live? Answer: Production failures — user reports, low ratings and traced errors — added to the dataset so they can't regress silently.

Put to work in: CAMS Lens — grounded answers that have to be right

LLM-as-judge — and where it stops being trustworthy

You can grade an open-ended answer automatically, and you know which of those grades to disbelieve.

pairwise vs pointwise · position bias · agreement with humans · error analysis · when to just read the outputs

In short:

  1. A judge is a model you must evaluate. Label a sample yourself and measure how often the judge agrees. Until you have that number, the judge's scores are unverified.
  2. Pairwise is steadier than pointwise. "Which of these two is better?" is more reliable than "rate this 1–10". Swap the order and check again to cancel position bias.
  3. Binary criteria beat vague scales. "Does the answer cite a source from the context? yes/no" is far easier to judge consistently than "rate helpfulness".

Build this — Grade your judge. Hand-label thirty of your own outputs. Have the judge label the same thirty. Report the agreement rate. Below about 80% the judge is telling you a story, and you have just learned that for the price of an afternoon.

You're done when: You can quote the agreement rate between your judge and your own labels — and you know which failure category it gets wrong.

Common traps:

Check yourself:

  1. How do you know your judge is trustworthy? Answer: Hand-label 50–100 examples and measure agreement with the judge. Then read the disagreements to see which failure type it misses.
  2. What is position bias, and how do you control it? Answer: The judge prefers whichever answer it sees first (or second). Run each pair in both orders and only count consistent verdicts.

Put to work in: CAMS Lens — grounded answers that have to be right

Explore more

GitHub · LinkedIn · Email