Overfitting means your model is so closely aligned to training data sets that it does not know how to respond to new situations. This chapter discusses:

• Why overfitting is difficult to avoid
• How to correct overfitting
• How to prevent overfitting

Why overfitting is difficult to avoid

One of the reasons overfitting is difficult to avoid is that it is often the result of insufficient training data. The person responsible for the model may not be the same person responsible for gathering the data, who may not even realize how much data is needed to supply training and testing phases.

An overfit model returns very few errors, which makes it look attractive at first glance. Unfortunately, there are too many parameters in the model in relation to the underlying system. The training algorithm tunes these parameters to minimize the loss function, but this results in the model being overfit to the training data, rather than the desired behavior of generalizing the underlying trends. When large amounts of new data are introduced to the network, the algorithm cannot cope, and large errors begin to surface.

Ideally, your machine learning model should be as simple as possible and accurate enough to produce meaningful results. The more complex your model, the more prone it will be to overfitting.

How do you avoid overfitting?

The best way to avoid overfitting is by making sure you are using enough training data. Conventional wisdom says you need a minimum of 10,000 data points to test and train a model, but that number depends greatly on the type of task you’re performing (naive Bayes and k-nearest neighbor both require far more sample points). That data also needs to accurately reflect the complexity and diversity of the data the model will be expected to work with.