Machine Learning
Old-school software is a stack of hand-written if/else rules. Machine learning flips that: you show the computer lots of examples and it discovers the pattern by itself. This is the Data Workshop: a bench where you feed in examples and watch a model take shape.
Plain words first. No math, no jargon. Just examples, and a machine that notices patterns.
The big idea
Suppose you want to tell cats from dogs in photos. With rules you'd try to write them yourself: "if pointy ears and whiskers then cat...", and you'd fail fast, because there are endless exceptions. With machine learning you instead show the computer thousands of labelled photos and let it figure out the pattern.
A rules program is a fixed recipe. A learner is a chef who tastes many dishes and learns what "good" means, then can judge a brand-new dish it has never seen.
Rule of thumb: if you can write the exact rule, do. If the pattern is fuzzy and you have lots of examples, learn it.
Read each scenario and decide: would you solve it with hand-written rules, or by learning from examples? Click your answer for instant feedback.
-
The raw material
An example is one thing you know about (one house, one email). Its features are the measurable clues (size, bedrooms, location). The label is the answer you want to predict (the price). A pile of examples, each with features and a label, is a dataset.
Good features carry signal about the label. Useless features are noise; they only distract the model.
| size (m²) | beds | shoe size of owner | price (k) |
|---|---|---|---|
| 60 | 2 | 42 | 180 |
| 95 | 3 | 38 | 270 |
| 120 | 4 | 44 | 340 |
Two columns are features that predict price; one is pure noise. Can you spot the useless one? Play the game below.
You're predicting house price. Toggle the features you think actually predict the price. Useful ones boost your signal; noise drags it down. Aim for the strongest predictive set.
Three flavours of learning
Supervised: you have labelled examples (questions and answers). The model learns to predict the answer.
Unsupervised: no labels, just data. The model finds structure on its own, like grouping similar things together.
Reinforcement: an agent acts in a world and learns from rewards and penalties, trying to score as much as possible over time.
Sort each real scenario into the right family. Instant feedback explains the why.
-
We collect 10,000 emails, each marked spam or not spam (labels). Features might be: count of the word "free", number of links, ALL-CAPS ratio. The model reads all 10,000 and learns weights for each feature. On a brand-new email it adds up the evidence and outputs a probability of spam.
This is supervised classification, exactly what we build hands-on in the next level.
How learning actually happens
A model doesn't get it right in one shot. It runs a little loop: take some data, let the model make a prediction, compare that to the right answer to get an error, then update the model a tiny bit to shrink that error. Round and round, the error gets smaller and the model gets smarter.
That's the whole secret of "learning from examples": there's no magic, just thousands of tiny corrections.
Watch a packet of evidence travel the loop: data → model → prediction → error → update, then back to the start. Each lap, the error bar on the right shrinks a little: that's the model improving. Press pause if you want to study a single stage.
Now we actually fit models: a line for numbers, a boundary for classes.
Two jobs
Regression: the answer is a quantity on a scale, like a price, a temperature, a height. "How much?"
Classification: the answer is one of a few buckets: spam/not-spam, cat/dog, yes/no. "Which one?"
The model is the straight line \(\hat y = wx + b\). Drag the slope and intercept sliders (or drag the line's endpoints on the chart) to push the line through the cloud of points and minimise the error. Lower MSE is better. Then hit Auto-fit to watch gradient descent solve it.
Two classes of points sit on the plane. Move and rotate a straight dividing line so that one class lands on each side. Your score is the share of points classified correctly. Can you beat 95%?
In words: the line score \(z = w^\top x + b\) is a raw number from \(-\infty\) to \(+\infty\). The sigmoid \(\sigma\) squashes it into a probability between 0 and 1. We then call it class 1 when that probability passes a threshold (0.5 by default). This is logistic regression, the exact rule the "Auto-fit (logistic)" button above is solving for.
Don't fool yourself
If you let a model memorise the answers, it'll ace the questions it studied and flop on new ones. So we hide a slice of data (the test set), train on the rest, and judge honestly on the hidden slice. That measures generalization.
Toggle "memorise hard" and watch train accuracy stay high while test accuracy collapses: the classic overfit gap.
How "Auto-fit" really works
Every choice of slope \(w\) and intercept \(b\) gives some error. Plot that error for all choices and you get a bowl-shaped loss surface. Training just means standing somewhere on that bowl and repeatedly stepping downhill (in the direction the slope drops fastest) until you reach the bottom: the best line.
The step size is the learning rate. Too small and you crawl; too big and you bounce around or even fly out of the bowl. Watch both failure modes below.
Each step nudges the weights opposite the gradient (the uphill direction): \(\;w \leftarrow w - \eta\,\dfrac{\partial L}{\partial w}\). Big gradient → big step; near the bottom the gradient flattens and the steps shrink automatically.
The heat-map is the error for every \((w,b)\): dark valley = low error, bright rim = high. Click anywhere to drop the ball there, then watch it roll downhill in real time. Change the learning rate and re-run to see it crawl, glide, or overshoot.
We learn \(\hat y = wx + b\) where \(x\) is the bill and \(y\) the tip. After training we get \(w=0.18,\; b=0.5\). For a \(\$50\) bill: \(\hat y = 0.18(50) + 0.5 = \$9.5\). For a \(\$20\) bill: \(\hat y = 0.18(20) + 0.5 = \$4.1\). The slope \(w\) is "tip per dollar", the intercept \(b\) is a base tip.
Training is just finding the \(w\) and \(b\) that make these predictions as close to real tips as possible, which is exactly the MSE you minimised in the game.
Loss functions, the bias-variance trade-off, regularization, and metrics, typeset properly.
How we measure "wrong"
Training is minimising a loss. Regression usually uses Mean Squared Error; classification uses cross-entropy.
In words: for each example, measure how far the prediction missed, square it (so misses count and signs can't cancel), then average over all examples. This is Mean Squared Error (MSE).
In words: when the true label is 1 it punishes a low predicted probability; when the true label is 0 it punishes a high one. Confident-and-wrong is penalised harshly. This is binary cross-entropy (log loss).
Left: MSE grows like a bowl as a single prediction drifts from its true value. Right: cross-entropy shoots toward infinity as a confident probability heads the wrong way.
The central trade-off
A model too simple underfits (high bias: misses the real pattern). A model too flexible overfits (high variance: memorises noise). The art is the middle.
Slide the polynomial degree from 1 (a straight line) to 12 (a wild squiggle). Watch the fit on the data, plus train vs validation error. Pick the degree that generalizes best (lowest validation error) and lock it in to score.
In words: add a penalty for big weights to the loss. The model now has to justify every large weight with a real drop in error, so it stays smoother and overfits less. This is L2 (ridge) regularization.
A degree-9 curve is fitted to noisy data. Raise \(\lambda\) and watch the wild wiggles relax into a calm, general curve.
Judging a classifier
In words: precision asks "of the things I flagged positive, how many were right?"; recall asks "of all the real positives, how many did I catch?"; F1 is their harmonic mean, high only when both are high.
Each dot is an example with a predicted score; reds are truly positive, greys truly negative. Slide the decision threshold: everything to the right is called "positive". Watch the confusion matrix and the metrics update live.
Every threshold at once
A single threshold gives one precision/recall pair. But which threshold? The ROC curve sweeps the threshold from high to low and plots the true-positive rate against the false-positive rate at each step. The area under that curve (AUC) is one tidy number: the chance the model ranks a random positive above a random negative. 0.5 is a coin flip; 1.0 is perfect.
In words: the true-positive rate is the share of real positives we catch; the false-positive rate is the share of real negatives we wrongly flag. As we lower the threshold both rise; the ROC curve traces that trade-off, and AUC is the total area beneath it.
Left: the same scored examples, with a threshold sliding from right (catch nothing) to left (catch everything). Right: each threshold plots one point of the ROC curve, and the area underneath fills in as the curve is drawn. Drag the threshold yourself, or press Sweep to animate the whole curve. A harder dataset (more overlap) pulls the curve toward the diagonal and drops AUC.
A picture of the trade-off
Bias is being consistently off-target (your shots cluster, but in the wrong place). Variance is being scattered (your shots spray everywhere, even if they average out near the centre). The four dartboards below animate every combination, and the goal is the bottom-left: low bias and low variance.
Each board is a different model. Watch the darts (repeated trainings on fresh data) land: tight-and-centred is ideal; tight-but-off is high bias; spread-but-centred is high variance; spread-and-off is the worst of both.
In words: the dartboards above are this formula made visible. Expected squared error splits into exactly three pieces: how far off-centre the model is on average (bias), how much it jitters from one training set to the next (variance), and irreducible noise in the data you can never fit away. Lowering one usually raises another; that is the trade-off.
Diagnose with data size
Plot training and validation error as the training set grows. Two telltale shapes appear. High bias: both curves flatten at a high error and hug each other, so more data won't help, you need a richer model. High variance: a big gap between low training error and high validation error, and more data will help close it.
Drag the model from high bias (too simple) to high variance (too flexible) and watch the two error curves redraw as the dataset grows from a handful of points to many. Read the gap: a stubborn gap means more data helps; converged-but-high means you've hit the model's ceiling.
A test flags 100 emails as spam. 80 really are spam (\(TP=80\)); 20 were innocent (\(FP=20\)). It also missed 40 real spams (\(FN=40\)).
Precision \(=\dfrac{80}{80+20}=0.80\). Recall \(=\dfrac{80}{80+40}=0.667\). \(F_1=\dfrac{2(0.8)(0.667)}{0.8+0.667}\approx 0.727\).
It's precise (few false alarms) but misses a third of real spam (lower recall). Lowering the threshold would catch more spam at the cost of more false alarms.
Clustering, neighbours, trees, scaling, cross-validation, real code, and a final challenge.
Unsupervised · clustering
You only have points, no answers. k-means drops \(k\) centre points (centroids), then repeats two steps: assign each point to its nearest centroid, and move each centroid to the average of its points. Repeat until nothing moves.
We score a clustering by inertia: the total squared distance from every point to its centroid. Lower inertia = tighter, better-placed clusters.
Click on the canvas to drop \(k\) centroids where you think the cluster centres are. Then press Run iteration to do one assign-and-update step, or Auto-run to converge. Try to reach the lowest inertia, and beat your best.
Supervised · neighbours
To classify a new point, k-nearest-neighbours looks at the \(k\) closest known points and takes a majority vote. No training at all, just remember the data and measure distances at prediction time.
The big hollow dot is a query with unknown class. Drag it around. The \(k\) nearest known points are circled and they vote on its colour. Change \(k\) and watch the prediction flip near the boundary.
Supervised · the widest street
Many lines can separate two classes, but which is best? An SVM picks the one that leaves the widest empty street between the classes. The points that touch the edges of that street are the support vectors; they alone define the boundary. A fatter margin generalises better to new points.
For a line \(w^\top x + b = 0\), the margin width is \(\dfrac{2}{\lVert w\rVert}\). SVM maximises that while keeping every point on its correct side: a "maximum-margin" classifier.
In words: real data rarely splits cleanly, so the soft-margin SVM lets points violate the street for a price. The first term widens the margin (small \(\lVert w\rVert\)); the hinge loss term charges for every point inside the margin or on the wrong side. The knob \(C\) sets the exchange rate between the two.
Two classes sit on the plane. Rotate and shift a separating line (or drag on the chart) so it splits them and leaves the widest possible gap. The shaded band is your margin; the circled points are the nearest on each side (the support vectors). Score the widest correct margin, then hit Optimal to see the SVM's answer.
Unsupervised · dimensionality reduction
Sometimes two features are really telling you one thing. Principal component analysis finds the single direction along which the data varies most (the principal axis) and projects every point onto it. You trade a dimension for simplicity while keeping the maximum possible variance (information).
Project onto a direction \(u\); the captured variance is \(u^\top \Sigma\, u\). PCA picks the \(u\) that maximises it: the top eigenvector of the covariance \(\Sigma\). Spin the line below and watch the captured fraction peak exactly on that axis.
The cloud is a stretched 2D blob. Rotate the projection line; each point drops a foot onto it (its 1D coordinate). The readout shows what fraction of the total spread that single line captures. Hit Snap to PC1 to jump to the optimal direction PCA would choose.
Scree chart (right): PCA's two axes ranked by the spread they hold, and PC1 always captures the most. The marker shows what fraction of PC1's variance your current line is reaching.
Wisdom of the crowd
One shallow tree is jagged and unreliable. But train many of them, each on a slightly different random sample of the data, and let them vote: the wobbles cancel out and a smooth, stable boundary emerges. That's bagging (a random forest is exactly this). The crowd beats any single member.
Averaging independent, noisy guesses keeps the signal and cancels the noise; it cuts variance without adding much bias. More voters → a smoother, steadier decision boundary.
Each faint line is one weak classifier trained on a random resample of the data. Add more voters and watch their jagged individual guesses average into the bold, smooth ensemble boundary. The background tint shows how confidently the crowd votes each region.
Expert · how a tree carves space
A decision tree classifies by asking yes/no questions like "is \(x > 0.4\)?" Each question is an axis-aligned cut that splits a region in two. Deeper trees keep cutting until each box is (nearly) one colour. The depth slider animates the cuts appearing, and shows how a deep tree can carve tiny boxes around single points (overfitting).
Two coloured classes sit on the plane. Raise the depth and the tree recursively cuts the space into axis-aligned regions, each shaded by its majority class. Watch purity climb toward 100%, and notice the boxes getting suspiciously small at high depth.
A tree asks yes/no questions, splitting the data until each leaf is pure. Deeper trees fit more, and overfit more. Drag the depth.
If one feature spans 0 to 1000 and another 0 to 1, distance-based models obsess over the big one. Standardize: \(x' = \dfrac{x-\mu}{\sigma}\). Toggle scaling.
Instead of one test set, split data into \(k\) folds. Train on \(k-1\), test on the held-out fold, rotate, and average. A more honest score.
from sklearn.linear_model import LinearRegression, LogisticRegression
from sklearn.model_selection import train_test_split
# split: keep 20% hidden to test generalization
X_train, X_test, y_train, y_test = train_test_split(
X, y, test_size=0.2, random_state=0)
# regression: predict a number (y_hat = w·x + b)
reg = LinearRegression().fit(X_train, y_train)
print(reg.score(X_test, y_test)) # R^2 on unseen data
# classification: predict a class via the sigmoid
clf = LogisticRegression().fit(X_train, y_train)
print(clf.predict_proba(X_test)[:5]) # probabilities, not just labelsimport torch, torch.nn as nn
model = nn.Linear(n_features, 1) # y_hat = w·x + b
opt = torch.optim.SGD(model.parameters(), lr=0.05)
loss_fn = nn.MSELoss() # regression loss
for epoch in range(200):
opt.zero_grad() # clear old gradients
y_hat = model(X_train) # forward pass
loss = loss_fn(y_hat, y_train) # how wrong?
loss.backward() # gradients of loss w.r.t w,b
opt.step() # w <- w - lr * grad (gradient descent)
# for classification: swap to nn.BCEWithLogitsLoss + a sigmoid outputtrain_test_split - hides part of the data so the score reflects unseen examples, not memorisation.
LinearRegression / nn.Linear - both fit \(\hat y = wx + b\); sklearn solves it directly, PyTorch via gradient descent.
LogisticRegression - squashes the line through a sigmoid into a probability, then thresholds for a class.
the training loop - forward, measure loss, backpropagate gradients, take a descent step, repeat.
Six points on a line: \(\{1,\,2,\,3,\,9,\,10,\,11\}\), and \(k=2\). Start the centroids badly at \(c_1 = 2\) and \(c_2 = 4\).
Assign: each point joins its nearest centroid. Points \(1,2,3\) are closer to \(c_1=2\); points \(9,10,11\) are closer to \(c_2=4\) (e.g. \(|9-4|=5 < |9-2|=7\)). So \(A_1=\{1,2,3\}\), \(A_2=\{9,10,11\}\).
Move: each centroid jumps to the mean of its members: \(c_1 = \tfrac{1+2+3}{3} = 2\), \(c_2 = \tfrac{9+10+11}{3} = 10\).
Check: re-assigning with \(c_1=2,\,c_2=10\) changes nothing, so the algorithm has converged in one update. Final inertia \(= (1{-}2)^2+(2{-}2)^2+(3{-}2)^2 + (9{-}10)^2+(10{-}10)^2+(11{-}10)^2 = 4\).
k-means only ever lowers inertia, so it always converges, but to a local minimum that depends on the start. Bad seeds give bad clusters; that is why practice uses k-means++ seeding (spread the initial centroids apart) and several restarts, keeping the lowest-inertia run.
Frontier · research-grade
The levels above are the practitioner's toolkit. This tier is the why: when does fitting the data generalize, what error is reducible, how kernels buy nonlinearity for free, how boosting turns weak rules strong, and why k-means always converges. Each topic is a guided lesson with step-through proofs, a worked example, a visualization, and citations.