Deep learning.

What makes learning deep

"Deep" has a plain, technical meaning. A deep neural network is simply a network with more than one hidden layer between the input and the output. The word refers to how many layers are stacked, not to how clever the system is or how much data it has. Deep learning is the part of machine learning that uses these many-layered networks.

The reason depth matters is representation learning. Each layer transforms the output of the layer before it into a slightly more abstract description. In a network that reads images, the earliest layers tend to respond to edges and colours, middle layers to textures and small shapes, and later layers to object parts and whole objects. The network builds understanding up in stages, from simple to complex, instead of being told what to look for.

This is the clearest difference from classic machine learning. In classic ML a person first designs the features by hand, deciding which measurements of the raw data matter, and then fits a model such as a decision tree or a linear classifier on top of them. Deep learning learns the features itself, as part of training, directly from raw or lightly prepared data. Removing the hand-crafted step is much of why it works so well on messy inputs like images, audio and text.

Why does depth help? Stacking layers lets a network re-use and combine the patterns it has already found, so a complicated function can be represented more compactly, with fewer units, than a shallow network would need for the same thing.

Three honest cautions keep this in perspective. First, depth is about efficiency in practice, not raw capability: as the neural networks course noted, even a single wide hidden layer can in principle approximate any continuous function. Second, deeper networks are harder to train (the next lesson explains why). Third, more layers is not automatically better; past a point you get diminishing returns, overfitting or training failures unless you also add the right techniques.

Why deep learning took off

The ideas behind deep learning are decades old, but it only became dramatically useful in the 2010s. Three things had to arrive together.

• Large labelled datasets. Networks with millions of parameters need many examples to learn from without simply memorising. ImageNet, released in 2009, provided millions of labelled images across a thousand categories and became the proving ground for the field.
• Computing power, especially GPUs. Graphics processors turned out to be very good at the dense arithmetic that training a network needs, making it practical to train large models in a reasonable time.
• Better algorithms. A handful of techniques made deep networks trainable: the ReLU activation (which trains faster than the old saturating functions), better ways to initialise the weights, dropout (randomly switching off units during training to curb overfitting), and batch normalisation (which steadies and speeds up training).

The turning point was 2012. A deep convolutional network called AlexNet, trained on GPUs, won the ImageNet competition with a top-five error of about 15.3%, against roughly 26.2% for the next-best, non-deep entry. Nearly halving the gap in one step convinced the field that deep learning was not a curiosity, and the modern era began.

Depth also creates its own problem, and it is worth understanding. Training works by sending an error signal backwards through the layers, and that signal is a long product of terms, one per layer. With many layers and the older saturating activations, the product tends to shrink towards zero, so the early layers barely learn; this is the vanishing gradient problem. Sometimes it does the opposite and blows up (the exploding gradient). This is precisely why deep networks were historically so hard to train.

The fixes are the same ones that powered the takeoff, plus a few more: ReLU activations that do not saturate for positive values, careful initialisation, batch normalisation, clipping gradients when they explode, and, in deeper modern networks, skip connections that let the signal bypass layers. For sequences, special gated units do the job, as the fourth lesson describes.

Convolutional networks for images

A convolutional neural network (CNN) is the design that made deep learning work so well on images. It rests on three ideas, each matched to the way images are actually structured.

• Convolution. A small grid of weights called a filter or kernel, often just three by three, slides across the image. At each position it multiplies and sums the pixels under it, producing a number. Doing this everywhere gives a feature map: a picture of where that filter's pattern, such as a particular edge, appears. A layer has many filters, each tuned to a different pattern.
• Local receptive fields. Each neuron looks only at a small local patch of the layer before it, not the whole image. This fits the fact that the meaningful structure in an image, an edge or a corner, is local.
• Weight sharing. The same filter is used at every position in the image. This is the key economy: instead of learning separate weights for every pixel, the network learns one filter and reuses it everywhere. It cuts the number of parameters enormously and means a pattern is recognised wherever it appears, not only where it was first seen.

Between convolutional layers, pooling (for example taking the maximum value in each small neighbourhood) shrinks the feature maps. This reduces the amount of computation and makes the network a little less sensitive to exactly where a feature sits.

Stacking convolution and pooling layers produces the simple-to-complex hierarchy from the first lesson, which is why CNNs suit images, and other grid-like data such as audio spectrograms, so well. The canonical early example is LeNet-5, used in 1998 to read handwritten digits. One honest note for today: CNNs are a foundational vision architecture but no longer the only one, since transformer-based models (the next lesson) now also do well on images.

From RNNs to transformers

Images are grids, but much data is a sequence: the words in a sentence, a run of transactions, a signal over time. The first deep design for sequences was the recurrent neural network (RNN). An RNN reads a sequence one step at a time and carries a hidden state forward from step to step, a kind of running memory of what it has seen.

Plain RNNs have a serious weakness. Training them means sending the error signal back through every step, and the vanishing gradient problem from the second lesson is especially severe over long sequences. In practice this means an RNN struggles to connect something early in a sequence to something much later: it forgets. These are called long-range dependencies, and they are exactly what plain RNNs fail at.

The fix was the LSTM (Long Short-Term Memory), introduced in 1997. It adds a protected cell state and a set of gates that decide what to keep, what to update and what to discard, so information and the training signal can survive over long spans. A later, simpler variant with fewer gates is the GRU. For roughly a decade these gated networks were the standard tools for language and other sequences.

Then in 2017 came the transformer, from a paper whose title, "Attention Is All You Need," makes the point: it set recurrence aside entirely and relied on self-attention. Self-attention lets every position in a sequence look directly at every other position, so a distant word is one step away rather than many, and, unlike an RNN, the whole sequence can be processed in parallel, which makes training at huge scale possible. Transformers largely replaced RNNs for language and are the architecture underneath today's large language models, the subject of the next course. A caution worth keeping: CNNs, RNNs and transformers are different designs for different kinds of data; a transformer is not a kind of RNN, it is what replaced it for many tasks.

Using deep learning responsibly

Deep learning is powerful, but it is not the right tool for every job, and it comes with real costs and risks that anyone using it should weigh honestly.

• It is data-hungry. Deep networks usually need large labelled datasets to generalise well; with little data they overfit, and simpler methods often do better. One important mitigation is transfer learning: start from a model already trained on a large dataset, then fine-tune it on your smaller problem, which greatly reduces how much labelled data you need.
• It is compute-hungry. Training large models costs significant time, hardware and energy, which is a genuine constraint on where and how they can be used.
• It is opaque. A deep model is largely a black box: it is hard to explain exactly why it made a particular decision. That matters a great deal in regulated or high-stakes settings where a decision must be justified.
• It can be biased. A model learns the patterns in its training data, including the biases in that data. A skewed dataset produces a skewed model, so the data and the results need checking across different groups.
• It can be brittle. Small, carefully chosen changes to an input, sometimes imperceptible to a person, can make a confident model give the wrong answer. These adversarial examples are a reminder that high accuracy is not the same as robustness.

So when is plain machine learning the better choice? When the dataset is small; when the data is tabular or structured, where methods like gradient-boosted trees often match or beat deep networks; when you need the model to be interpretable; and when you are working within a tight compute or latency budget. The honest headline to carry away is that deep learning does not always beat classic machine learning — the right choice depends on the data, the constraints and what you need to be able to explain.

Your certificate of completion

Enter your name as you want it to appear, then save the certificate as a PDF. In the print window, choose Save as PDF. A certificate is issued once per completion of the course.

Saolix does not record who takes this course, so it cannot verify these certificates. The certificate confirms completion of a free self-paced course and is not an accredited qualification.