When using neural networks for predictive analytics, overfit can occur. This is when your model predicts too many features or parameters of the data set you are trying to predict with accuracy.
In other words, your model becomes very complex and does not generalize well. You may have noticed this when experimenting with different architectures or structures of the network before!
The solution? Use regularization. Introduce constraints into the learning process that help prevent it from fitting only the patterns of the training dataset.
Regularization can be done through parameter limits, input constraint, or cost function limitations. These all work towards preventing overfitting by limiting how much the algorithm is able to optimize out.
This article will go more in depth about some specific types of regularization and why and/or examples of them being used in practice.
Avoid overfitting by being consistent
When developing your model, make sure you are not using the same training data set to test it! This is called over-fitting the model onto the dataset.
When developing deep learning models, overfitting can occur when there is too much emphasis put into fitting every detail of the model onto the train set.
This happens because most AI algorithms rely heavily on mathematics that require long calculations or large amounts of computer time to run through.
By having the algorithm “think” about all these different equations at once, it may start to create its own internal equation or pattern from those patterns seen before.
Once it has done this, it will try hard to match those shapes and formulas with no gaps, which sometimes cannot be matched with new data that was not part of the original train set!
By trying so hard to fit everything, the model ends up matching the parts very well but disregarding what comes after it, leaving you with a bad prediction.
Select your dataset
A common beginner mistake is choosing an overly complex or difficult dataset for training. This can result in overfit, which is when the model becomes too dependent on the structure of the data it was trained with.
A similar situation occurs when the models are very general, without specializations into specific categories or patterns. These types of networks do not improve as you add more examples in the test set that they were not trained on!
By using simple datasets, you will find that most pre-trained state-of-the-art networks work quite well. Some of the best performing networks use only toy images (for example, pictures of house objects) and natural language processing (NLP) corpora, such as the IMDB movie review corpus.
Using simpler datasets helps prevent overfitted networks because there are not many opportunities for them to get “stuck” on complicated structures used for feature extraction during training. You also want to make sure your chosen dataset has enough instances so that the network does not suffer due to lack of statistical power.
Ensure your dataset is representative
A common cause of overfit in deep learning are datasets that contain too many features or instances of examples in the training set. This can be due to there being too much data, or individual features containing excessive amounts of information.
When you train a neural network on such a large amount of data, it learns all sorts of patterns instead of focusing only on predictive relationships between inputs and outputs.
Because of this, when you test the model by using an unseen sample that is similar to some pattern in the training set, the algorithm predicts that it will also produce a similar output because it learned that pattern during training.
This effect becomes more significant as the number of layers and neurons in the networks increase, longer chains of logic are possible, and thus the models become more powerful.
However, since these long chains of predictions rely so heavily on the input data, if the input data is not sufficiently representative then the model may fail to make any meaningful predictions even for samples that look very similarly to ones in the training set.
Use a good validation set
A common beginner mistake is using a large test dataset as your validation set. Using a large test set can lead to overfit!
When you use a larger training dataset, it becomes harder for the algorithm to make predictions outside of the training data. This happens because the model learns more complex features of the training sets than necessary.
These elaborate features are then applied to new examples that were not included in the training process. Because these models have high accuracy on the training datasets, they become even more accurate when tested on new datasets!
This effect is called over-training or overfitting. Overfitted networks do not generalize well — they perform very well on the training datasets, but fail on new samples.
Using a smaller validation set will prevent this from happening by teaching the algorithm how to differentiate between the training sets and the testing sets.
Validation sets should be separate and distinct from the test sets so that the learning does not influence which samples it predicts correctly.
Use a good test set
A lot of people make the mistake of using their validation or testing sets as a way to determine how well their model is fitting the data. This can easily lead to over-optimistic results because your model may be learning bad generalization patterns.
If you use the same training, validation, and test sets throughout all of your experiments, this will not happen. You should always have one separate set that you use for evaluating how well your models are performing without any additional information.
This way, you can rely on the accuracy of your tests instead of the accuracy of your fit. Obviously, if your model does not perform well on its own, then it is likely overfit the dataset it was trained on!
Using different partitions of the same dataset for evaluation is the best approach to take when experimenting with neural networks.
Use a regularization technique
When using neural networks for predictive modeling, over-training is a very common issue! This occurs when the network gets good accuracy by memorizing patterns of the training data instead of learning how to apply the model to new situations.
When you are tuning your hyperparameters (the variables that determine the performance of the model), it is easy to test different values and get better results with each try. Unfortunately, these improved predictions sometimes remain limited to the train set only!
This phenomenon is called overfit. By limiting the number of parameters in the model, regularization can prevent this from happening. A popular way to do this is via an L2 penalty term in the loss function.
Regularized regression uses such a penalty to ensure all coefficients of the regressor are within a certain range. For instance, if there was no coefficient outside the [-1, 1] interval, then the resulting regressor would be trivial — always predicting 0 or -1 depending on whether the input value is positive or negative!
In practice, most models have many more free parameters than needed for solving the given problem. Adding the regularization term helps enforce constraints on those extra parameters so they become meaningful.
Use a variety of models
When it comes down to it, overfiters don’t necessarily do anything wrong! They may be trying too hard to fit every little detail of the dataset, which is great if there are no discrepancies or errors in the data.
If you want to test whether or not your model is overfitted, then simply apply it to new datasets! If the accuracy stays consistent across many different situations, then your model isn’t overly specific to the training set.
This is totally fine though, as most deep learning methods require lots of generalization power, so having a few models that are slightly off is okay. It just means that some features of the image aren’t important to predicting the outcome, which makes sense since they’re not present!
Using several different architectures and regularizers can help prevent overfitting as well. There’s really nothing special about using LSTMs over other types of NNs, for example, but maybe try adding dropout at a certain layer or changing how much momentum you have at each epoch.
Try different model architectures
A common way to prevent overfit is by using more complex models- these are referred to as “bigger” or “higher dimensional” neural networks. More powerful models can fit the data better, which means they will produce thinner test sets with higher accuracy.
The problem is that big models usually take longer to train, use more computing power, and require larger amounts of training data! This makes it harder to achieve convergence, an important metric for evaluating how well your network is fitting the data.
By incorporating additional layers into our net, we get thicker (more neurons) inner layers that perform feature extraction and deeper nets that go even further than two layers.
These extra features help us capture higher level concepts that may not be picked up easily by just adding another layer onto our current architecture. For instance, if we trained a classifier to determine whether a picture contained a car, a cat would probably have some sort of head shape, whereas a dog might have ears or fur. So instead of having one neuron dedicated to cars and cats, we now have a few neurons that recognize other shapes and characteristics.
