How a network
learns to be less wrong

What a neural network actually is

A neural network is a function. You hand it numbers, it hands numbers back. What makes it interesting is that the function contains thousands or billions of adjustable dials — called parameters — and there is a mechanical procedure for turning those dials so the outputs get closer to the answers you wanted.

That is the entire idea. Everything else in this course is detail about the shape of the function (modules 2–4), how you measure "closer" (module 5), and how you turn the dials (modules 6–7).

The one-sentence version

Traditional software is written rule by rule: a human decides that if the email contains "wire transfer urgently" it goes to spam. A neural network is not written that way. You supply examples — thousands of emails already labelled spam or not — and the training procedure discovers the weighting of features that separates them. You specify the architecture and the objective; the data specifies the rules.

Why "neural"

The vocabulary is borrowed from biology and the borrowing is loose. A biological neuron fires a spike when incoming signals cross a threshold; an artificial neuron computes a weighted sum and passes it through a smooth function. The resemblance is a historical metaphor, not a claim about the brain. It is more useful to think of a network as a very large, very flexible curve-fitter than as a digital mind.

The neuron: weights, bias, activation

One artificial neuron does three things in order. It weights each incoming number, adds a bias, and passes the total through an activation function. That is the whole unit, and a network is nothing but many copies of it wired together.

• Inputs (x) — the numbers arriving, either raw features or activations from the previous layer.
• Weights (w) — one per input, learned. A weight is importance: large positive means "this input pushes my output up", negative means it pushes down, near zero means "ignore it". Weights are where the knowledge lives.
• Bias (b) — a learned offset added regardless of input. It shifts the threshold at which the neuron becomes active, so a neuron isn't forced to output zero when its inputs are zero.
• Activation g(z) — a fixed, non-linear function applied to the sum.

Why the activation function is non-negotiable

Without it, every neuron is a linear function, and stacking linear functions produces… another linear function. A hundred layers of pure multiplication collapse algebraically into a single equivalent layer, capable only of drawing straight lines. Non-linearity is what buys you depth. It is the ingredient that lets layers compose into curves, corners and concepts.

Layers, depth, width and parameters

Neurons are organised into layers, and layers into three roles:

• Input layer — not really neurons, just the entry point. Its size is fixed by your data: 784 for a 28×28 greyscale image, one per column for a spreadsheet.
• Hidden layers — everything in between, where representation happens. Their number and size are yours to choose.
• Output layer — sized by the task: one neuron for a price or a yes/no, ten for ten digit classes.

In a fully connected (dense) network, every neuron in a layer receives every output from the layer before. Depth is how many layers you stack; width is how many neurons sit in each.

What depth actually buys

Each layer transforms the representation handed to it. In a vision network, early layers respond to edges and gradients, middle layers to textures and parts, later layers to whole objects. Nobody assigns those jobs — the hierarchy emerges because a deep network is rewarded for reusing simple features to build complicated ones. Depth is compositional efficiency: a deep narrow network often expresses with hundreds of parameters what a shallow wide one needs millions to match.

Counting parameters

Between two adjacent layers, every connection is one weight, and every neuron in the receiving layer adds one bias:

Parameter count is the honest measure of a model's size. It drives memory, training time, and how much data you need before the model starts memorising instead of generalising.

Forward propagation: making a prediction

Forward propagation is the act of pushing one example through the network from input to output. Layer by layer: multiply by the weights, add the biases, apply the activation, hand the result on. In production this is the whole job — a trained model doing inference is only ever running a forward pass.

Written for a whole layer at once, it is a single matrix multiplication, which is exactly why GPUs matter:

Every value produced along the way is an activation. Hold on to them — backpropagation in module 6 needs them.

Loss: putting a number on wrong

The network has made a prediction. Now you need a single number saying how bad it was, because "improve" is meaningless until "worse" is measurable. That number comes from a loss function (the error on one example) averaged into a cost function (the error across a batch or the whole dataset).

The loss function is where you encode what you actually care about. Change it and you change what the network optimises for — it is the most consequential design choice after the data itself.

The loss landscape

Now hold the training data fixed and imagine sweeping every parameter across every possible value, recording the cost each time. The result is a surface in a space with one dimension per parameter — millions of dimensions, but the intuition survives in three. Hills are bad parameter settings, valleys are good ones. Training is the search for a low valley, and the tool for that search is slope.

Backpropagation and gradient descent

Two ideas often mashed into one word. Keep them apart:

• Backpropagation answers which direction. It computes the gradient — how much the loss would change if each individual parameter changed a little.
• Gradient descent answers what to do about it. It takes that gradient and steps every parameter a small distance in the downhill direction.

Gradient descent in one line

The derivative says which way is uphill; the minus sign turns you around; the learning rate decides how far you commit. That is the entire update rule, applied to every weight and bias in the network, over and over.

How backpropagation gets the gradient

A network is a chain of nested functions, and calculus already has a rule for differentiating those: the chain rule. Backpropagation applies it in reverse, from the loss backwards to the input, reusing each layer's result as it goes.

1. Run a forward pass and keep every activation.
2. At the output, compute how much the loss changes with the prediction.
3. Push that sensitivity back through the output layer's activation and weights to get the gradient for those weights — and the error attributable to the layer below.
4. Repeat layer by layer to the input. Each layer's blame is computed from the blame of the layer above it.
5. Update every parameter using the rule above.

The efficiency is the point. Estimating each gradient by nudging parameters one at a time would take one forward pass per parameter — billions of passes. Backpropagation gets all of them in a single backward sweep, which is the algorithmic breakthrough that made deep learning feasible at all.

Local minima, saddles, and why it works anyway

That curve has two valleys, and descent lands in whichever one it started nearest. Real loss landscapes have vastly more, which sounds fatal and mostly isn't: in very high dimensions, points where every direction curves upward are rare, most flat spots are saddles that momentum escapes, and the many decent minima tend to perform about as well as each other. You are not looking for the global minimum. You are looking for a low valley that generalises.

Optimisers: gradient descent with better habits

• Stochastic gradient descent (SGD) — update on small random batches instead of the whole dataset. Noisier steps, far faster, and the noise itself helps escape shallow traps.
• Momentum — accumulate a running velocity so the path keeps rolling through flat regions and dampens side-to-side oscillation.
• Adam — adapts a separate effective step size per parameter using running estimates of gradient mean and variance. The pragmatic default for most work.

Training in practice

The mechanics are settled. What remains is the craft: the choices that decide whether a technically correct network is actually useful.

Split the data before you touch it

• Training set (~70%) — the examples the model learns from.
• Validation set (~15%) — held out to tune architecture, learning rate and stopping point. The model never trains on it, but you make decisions from it, so it slowly leaks into your choices.
• Test set (~15%) — touched once, at the end. It is your only honest estimate of real-world performance.

Also scale your inputs. Features on wildly different ranges — age 0–100 next to income 0–500,000 — distort the loss landscape into a steep ravine that gradient descent struggles to navigate. Standardising or normalising is a one-line fix that routinely rescues a "broken" model.

Epochs, batches and steps

Overfitting and underfitting

The core tension of all machine learning. Underfitting means the model is too simple to capture the real pattern — it is wrong on training and test data alike. Overfitting means it has enough capacity to memorise the training examples including their noise, so training loss keeps dropping while validation loss turns and climbs. That divergence is the signal to watch.

Regularisation: the tools that fight overfitting

• More and better data — always the strongest remedy. Data augmentation (crops, flips, noise, paraphrases) manufactures variety for free.
• Early stopping — halt when validation loss stops improving, and keep the best checkpoint. Cheap and effective.
• Dropout — randomly switch off a fraction of neurons each training step so no unit can rely on a specific partner. Disabled at inference.
• Weight decay (L2) — add a penalty on large weights, preferring smoother functions over spiky memorisation.
• Batch / layer normalisation — stabilise the distribution of activations between layers; speeds training and adds mild regularisation.
• Reduce capacity — the underrated option. A smaller model that generalises beats a bigger one that memorises.

Beyond the basic net: CNNs, RNNs, transformers

Everything so far describes a multilayer perceptron — fully connected layers, no assumptions about the data. It works, but it treats an image as an unordered list of pixels and a sentence as an unordered bag of words. The famous architectures are all variations that build a structural assumption into the wiring itself.

Convolutional networks, briefly

Instead of a weight per pixel, a CNN learns a small filter — say 3×3 — and slides it across the whole image. Two consequences follow. The parameter count collapses, because one filter is reused everywhere rather than relearned per position. And the feature becomes translation-invariant: an edge detector trained on the top-left corner works in the bottom-right too. Stack these layers and you get the edges → textures → parts → objects hierarchy from module 3.

Transformers and attention

A transformer processes a whole sequence at once and uses attention to let every element look at every other and weigh how relevant each one is to it. In "the trophy didn't fit in the case because it was too big", attention is the mechanism that lets "it" bind to "trophy". Because there is no left-to-right recurrence, the computation parallelises across a sequence — which is precisely what made training on internet-scale text practical.

Best practices and common pitfalls

Practices worth making habits

• Establish a dumb baseline first — a linear model, or predicting the average. If the network can't beat it, the problem is your data or framing, not your architecture.
• Split the data before any exploration, and keep the test set sealed until the end.
• Scale and clean inputs before blaming the model. Most "the network won't learn" tickets are data problems.
• Start with the smallest architecture that could work, and grow only in response to underfitting.
• Tune the learning rate first, on a log scale (0.1, 0.01, 0.001). It outranks every other hyperparameter.
• Deliberately overfit a handful of examples as a smoke test. If the model can't memorise ten rows, the pipeline is broken.
• Plot training and validation loss every run. The shape of those two curves diagnoses nearly everything.
• Set random seeds and log hyperparameters, so a good result can be reproduced tomorrow.
• Evaluate with metrics that match the stakes — accuracy is meaningless on a 99:1 class split, where precision, recall and a confusion matrix are not.

Pitfalls that catch capable people

• Data leakage — scaling or imputing across the full dataset before splitting, or including a feature that encodes the answer. Produces beautiful validation scores and a model that fails in production.
• Judging on the test set repeatedly — every peek turns it into a second validation set and inflates your estimate.
• Chasing architecture before fixing data — more labels and cleaner labels beat a fancier model almost every time.
• Unbalanced classes ignored — 99% accuracy on a 99% negative dataset is a model that learned to say "no".
• Learning rate left at default — too high and loss oscillates or becomes NaN; too low and it crawls, or stalls in the first flat spot it meets.
• Training loss watched alone — it only ever goes down. Validation loss is the one that tells the truth.
• Assuming yesterday's distribution — the world drifts. Monitor live performance and plan to retrain.
• Treating the output as an explanation — a confidence score is not a reason. Where decisions affect people, budget for interpretability and human review.

Test yourself — 10 questions

Pick one answer per question, then score. Explanations appear with your result.