Training models in practice.

Data and features

A model is only ever as good as the data it learns from. Garbage in, garbage out: a model learns the patterns in its training data, and also its errors and biases. Before any clever modelling, the first job is to get the data right.

The foundation is the three-way split. You divide your data into a training set (used to fit the model), a validation set (used to tune choices and settings as you work), and a test set (used once, at the very end, for an honest estimate of how the model will do on new data). There is no single correct ratio; larger datasets can keep a smaller slice aside.

The test set must be kept sacred. Every time you look at the test score and then change the model, a little knowledge of the test set leaks into your choices, and the score quietly becomes too optimistic. You touch it once, at the end, and never tune against it.

The subtler trap is data leakage: using information when building the model that would not be available at prediction time, which makes results look great in testing and then collapse in the real world. The classic silent cause is preparing the data before splitting it, for example working out the average to fill in missing values, or the scaling numbers, from the whole dataset instead of from the training part only. The discipline is simple: split first, then learn any such transformation from the training data alone and apply it to the rest.

With that in place, feature engineering is turning raw data into inputs that expose useful structure. Scaling or normalising features onto comparable ranges matters because features on wildly different scales can distort many methods. Missing values are either dropped or filled in (imputed), and categorical values are turned into numbers the model can use. All of these are learned from the training set only, to avoid leakage.

A last warning about data: class imbalance. When one outcome is rare, such as fraud among ordinary payments, a lazy model can be right almost always by predicting the common case, while being useless at the thing you care about. That is exactly why the third lesson looks past plain accuracy.

Fitting without overfitting

Training a model means adjusting its parameters to make its predictions fit the training data. How wrong it is on that data is measured by a loss function (for example squared error when predicting a number, or cross-entropy when choosing a category). Gradient descent repeatedly nudges the parameters in the direction that reduces the loss most, a little at a time.

A few practical controls shape this. The learning rate is the step size: too large and training overshoots and will not settle; too small and it crawls. An epoch is one full pass over the training data, made up of many smaller batches, each used for one update. Too few epochs and the model has not learned enough; too many and it starts to memorise.

This points to the central tension of the whole course: underfitting versus overfitting.

• Underfitting is a model too simple to capture the real pattern. It has high bias and does poorly on both the training data and new data.
• Overfitting is a model that fits the training data too closely, including its noise. It has high variance: low error on training data but high error on new data, a large gap between the two.

This is the bias–variance trade-off. Simpler models tend to have more bias and less variance; more complex models the reverse. The aim is not zero training error, which usually means overfitting, but the sweet spot in between where the model generalises best.

Regularisation is the set of tools that curb overfitting:

• L2 (weight decay) discourages large weights, giving a smoother model; L1 can push some weights to exactly zero, effectively dropping features.
• Dropout randomly switches off units during training so a network cannot over-rely on any one path.
• Early stopping halts training once the validation error stops improving, even if training error keeps falling.
• More data, or sensibly augmented data, is the most reliable cure of all.

Evaluating honestly

Once a model is trained, the hard question is whether it is actually any good, and the honest answer is rarely "look at the accuracy". Accuracy, the fraction of predictions that are correct, is badly misleading when the classes are imbalanced: if ninety-nine in a hundred cases are negative, a model that always says "negative" scores ninety-nine per cent while catching none of the cases that matter.

The starting point for classification is the confusion matrix, which counts four outcomes: true positives and true negatives (correct), false positives (false alarms) and false negatives (misses). Every useful metric is built from these.

• Precision = true positives ÷ all predicted positives. Of the things the model flagged, how many were real. It matters when a false positive is costly, such as blocking a genuine customer.
• Recall = true positives ÷ all actual positives. Of the things that were real, how many the model caught. It matters when a false negative is costly, such as missing a fraud or a disease.
• Moving the decision threshold trades one against the other. F1 combines them into one number: the harmonic mean, 2 × precision × recall ÷ (precision + recall), which stays low unless both are high. It is a harmonic, not a plain, average.

Two more tools round this out. The ROC curve plots the true-positive rate against the false-positive rate across all thresholds, and the AUC (area under it) is the chance the model ranks a random positive above a random negative: 0.5 is no better than guessing, 1.0 is perfect. For predicting numbers, use MAE (average absolute error), RMSE (which punishes big errors more), or R² (the share of variation explained, which can even go negative for a model worse than guessing the average).

Finally, one split can mislead by luck. k-fold cross-validation rotates the validation fold through the data and averages the results, which is both steadier and more data-efficient, especially on small datasets. The test set is still kept aside for the final word.

Tuning and choosing a model

It helps to be precise about two words that are often confused. Parameters are the numbers the model learns from the data during training, such as the weights of a network or the coefficients of a regression. Hyperparameters are the settings you choose before and around training to control how it learns, such as the learning rate, the number of layers, the depth of a tree, or the strength of regularisation. Gradient descent never learns the learning rate; you set it.

Choosing good hyperparameters is a search. Grid search tries every combination in a list you specify: thorough, but the cost explodes as you add options. Random search samples combinations at random and often finds a good one faster, because usually only a few hyperparameters really matter. Whichever you use, it is scored on the validation data (or by cross-validation), never on the test set.

That rule bears repeating, because breaking it is the most common way people fool themselves. If you pick your settings by how they do on the test set, the test set has quietly become part of training, and its score no longer tells you anything honest about new data. Validation and cross-validation are for tuning; the test set is for one final, untouched read.

Good practice also means comparing fairly. Start with a baseline: something trivial like always predicting the most common class or the average value, or a simple model. A complex model is only worth its cost if it beats that baseline. When comparing candidates, evaluate them on the same data and the same metric chosen for the problem, report the spread across folds rather than a single lucky number, and weigh practical things too, such as speed, cost and how easy the model is to explain.

Putting a model into production

Training a good model is only part of the job. In a real system the model itself is a small piece of a much larger whole: the data collection, the feature pipelines, the checks, the configuration, the serving and the monitoring around it are far bigger, and they carry ongoing maintenance costs and their own risks. A model that scored well in a notebook is not yet a working system.

Serving the model means packaging it and making its predictions available, whether in large batches or one at a time through an interface. A common failure here is training–serving skew: if the live data is prepared even slightly differently from the training data, the model sees inputs it was never trained on and quietly degrades. The serving pipeline must reproduce the training pipeline.

The most important thing to accept is that a model decays. The world does not hold still, so performance drifts over time in two ways. Data drift is when the inputs change: the kinds of transactions, customers or images shift. Concept drift is when the relationship itself changes: what counted as fraud last year is not quite what counts this year. Because of drift, a deployed model must be monitored and retrained on fresh data, on a schedule or when its quality is seen to slip.

Two habits make all of this trustworthy. Reproducibility: version not only the code but the data, the model and the settings, so any result can be reproduced and any regression traced. And documentation: a short record, often called a model card, that states what the model is for, how it was evaluated and across which groups, and where it should not be used. None of this needs personal data; it is about being able to explain and stand behind what the model does.

Your certificate of completion

Enter your name as you want it to appear, then save the certificate as a PDF. In the print window, choose Save as PDF. A certificate is issued once per completion of the course.

Saolix does not record who takes this course, so it cannot verify these certificates. The certificate confirms completion of a free self-paced course and is not an accredited qualification.