Neural networks.

What a neural network is

A neural network is a function that learns a mapping from inputs to outputs by looking at examples, instead of following rules a person wrote by hand. You feed it many pairs of input and correct answer, and it adjusts a large set of internal numbers until its output fits the examples. Those numbers are the weights, and adjusting them is what "learning" means here. A neural network is not a separate thing from machine learning: it is one family of machine-learning model.

The building block is the artificial neuron. A neuron takes several numbers in, multiplies each by its own weight, adds them up together with one extra number called the bias, and then passes the result through a non-linear activation function. In short: output = f(w₁x₁ + w₂x₂ + … + b). On its own, a single neuron is just a simple classifier; the power comes from connecting many of them together in layers.

The idea is old. In 1943 Warren McCulloch and Walter Pitts described a mathematical model of a neuron as a simple threshold logic unit. In 1958 Frank Rosenblatt published the perceptron, a neuron that weighs its inputs and outputs one or zero depending on whether the weighted sum crosses a threshold. The perceptron could learn to separate data with a straight line, and it was built as real hardware, the Mark I Perceptron, first shown in public in 1960.

"Brain-inspired", honestly. Neural networks were loosely inspired by biological neurons, and that is where the name and the word "neuron" come from. But a modern neural network is not a model of the brain. Real neurons are far more complex than a weighted sum and a threshold. Treat the brain analogy as history and a naming convention, not as a claim that these systems think the way we do.

Where it fits: artificial intelligence contains machine learning, machine learning contains neural networks, and deep learning is simply neural networks with many layers. This course is about the ordinary feedforward network that everything else is built on.

How a network is built

A network is organised into layers. The input layer is just the data: a vector of numbers, such as the pixel values of an image. In between come one or more hidden layers of neurons, and at the end an output layer that produces the prediction. In a plain feedforward network each neuron in a layer is connected to every neuron in the next, and information flows one way, from input to output.

Each connection has a weight, and each neuron has a bias. Together, the weights and biases are the parameters that learning adjusts; everything a network "knows" is stored in them. Computing the output is called the forward pass: each layer takes the numbers from the layer before, works out the weighted sums and biases, applies its activation function, and passes the result on.

The activation function is applied to each neuron's weighted sum. Three are worth knowing:

• Sigmoid squashes any number into the range zero to one. It was common historically, but it saturates: for large positive or negative inputs its output barely changes, which makes learning stall. It is rarely used in hidden layers today.
• Tanh squashes into minus one to one. It is centred on zero, which helps training, but it still saturates.
• ReLU, which outputs the input if it is positive and zero otherwise, is the usual default. It is cheap to compute and does not saturate for positive values, so networks train faster. Its one weakness is that a neuron can get stuck always outputting zero.

Why have an activation function at all? Because without a non-linear one, the whole network collapses. Stacking linear steps on top of each other just gives another linear step, so a deep network with no non-linearity could only ever represent a straight-line relationship, however many layers it had. The non-linear activation is exactly what lets a network represent complicated, curved patterns.

How much can they represent? The universal approximation theorem (Cybenko in 1989, generalised by Hornik in 1991) says that a network with even a single hidden layer can approximate any continuous function to any accuracy you like. Two honest cautions: it only says such a network exists, not how large it must be, and it says nothing about whether training will actually find it. That gap is why we still rely on depth and on good training methods.

How a network learns

Learning means finding weights that make the network's output match the correct answers. To do that we need a way to measure how wrong it is. That is the loss (or cost) function: a single number that is large when the predictions are far from the right answers and small when they are close. Mean squared error is a common choice for predicting a number; cross-entropy is common for choosing a category. Learning is the search for weights that make the loss as small as possible.

The method is gradient descent. Imagine the loss as a landscape of hills and valleys, with the weights as the position. The gradient points in the direction the loss increases fastest, so we take a small step in the opposite direction to go downhill. The size of that step is the learning rate. We repeat: measure the loss, step downhill, measure again. In practice the gradient is usually computed not over the whole dataset at once (batch), nor from a single example (stochastic), but over a small mini-batch of examples at a time, which is faster and more stable.

The remaining question is how to work out the gradient for every weight in a large network efficiently. That is backpropagation. It is an application of the chain rule from calculus, run from the output back towards the input. Each operation in the network can work out its own local gradient; backpropagation combines these along each path, multiplying gradients that follow one after another and adding where a value feeds into several places. The result is the contribution of every single weight to the overall error.

One distinction is worth getting right, because it is often blurred. Backpropagation computes the gradients; gradient descent uses them to update the weights. They are two different steps. Backpropagation is the engine that works out the derivatives; it is not itself the learning rule. A full training step is a forward pass to make a prediction and measure the loss, then a backward pass to compute the gradients, and finally a nudge to every weight in the downhill direction.

Training it without fooling yourself

A network that does well on the data it was trained on has not necessarily learned anything useful. To find out, split the data into three parts: a training set to fit the weights, a validation set to check progress and tune choices while you work, and a test set kept aside for a single, honest estimate at the very end. The golden rule is never to tune anything on the test set; the moment you do, it stops being an honest measure.

Two words describe the passage through the data. An epoch is one full pass over the whole training set. An iteration (or step) is one weight update on one mini-batch. If you have a thousand examples and a batch size of a hundred, one epoch is about ten iterations, and training usually runs for many epochs.

The learning rate is the single most important setting to get right. Too high, and training overshoots: the loss swings about or blows up. Too low, and training crawls or stalls. It is often reduced gradually as training goes on.

The central danger is overfitting: the network learns the training data so closely, including its noise and accidents, that it fails on anything new. You can see it happening when the training loss keeps falling but the validation loss starts to rise. The opposite problem, underfitting, is a model too simple or trained too little, which does poorly everywhere.

Regularisation is the name for techniques that curb overfitting:

• Weight decay (also called L2) discourages large weights, keeping the fit smoother.
• Dropout randomly switches off some neurons on each training step, so the network cannot lean too hard on any one of them.
• Early stopping halts training when the validation loss stops improving.
• More data, or sensibly augmented data, is the most reliable cure of all.

A useful rule of thumb from practice: it is usually better to use a larger network and control overfitting with regularisation than to use a network too small to learn the task in the first place.

What neural networks are good and bad at

The networks described so far are feedforward networks: information flows one way, from input through the hidden layers to the output, with no loops. This is the baseline that everything else builds on, and it is where their strengths and weaknesses are clearest.

What they are genuinely good at. Neural networks shine when the pattern is real but too complex or messy to write down as rules, and when there is plenty of data to learn from. They are strongest on perception tasks, reading images, sound and text, where hand-written rules have always struggled. A large part of their value is that they learn their own features: instead of a person deciding in advance which measurements matter, the network works out useful internal representations from the raw data.

What they are bad at, honestly. They are hungry, needing large amounts of labelled data and real computing power to train. They are opaque: it is hard to say exactly why a network made a particular prediction, which is a serious problem anywhere a decision has to be explained or defended. They can be brittle, failing in surprising ways on inputs unlike their training data, and they are only as fair as their data, so they can absorb and repeat the biases in it. And, as the universal approximation theorem reminded us, there is no guarantee training finds the best possible solution.

Where it goes next. "Deep" learning simply means stacking many hidden layers, so the network builds features up in stages, from simple to complex, such as edges, then shapes, then whole objects in an image. Two specialised deep designs are worth naming. Convolutional neural networks (CNNs) reuse small filters that slide across the input to pick up local patterns, and they dominate image and vision tasks; the design was shown to work on real documents by LeCun and colleagues in 1998. Recurrent neural networks (RNNs) carry a memory of what came before, which lets them process sequences such as text or time series. Both start from exactly the neuron, the layers, the activations and the backpropagation you have met here.

Your certificate of completion

Enter your name as you want it to appear, then save the certificate as a PDF. In the print window, choose Save as PDF. A certificate is issued once per completion of the course.

Saolix does not record who takes this course, so it cannot verify these certificates. The certificate confirms completion of a free self-paced course and is not an accredited qualification.