Saving a neural network model is not as simple as just copying the architecture and weights of the net you want to save! There are several reasons for this.
Firstly, most people use deep learning networks with one loss function at a time. This means that there is only one metric being used to determine if the model is performing well.
However, some applications require multiple models to be trained simultaneously or sequentially. In these cases, it can become very difficult to choose which parts of the network should be saved because they all work equally well!
Secondly, even when using a single model at a time, different users may have their own ideas about how to improve the performance of the network. These individuals could modify part of the overall structure, or add new layers or features to help achieve better results.
Thirdly, depending on what kind of data set the net was trained on, it is possible to get poor test accuracy by simply changing the way the training sets were organized! For example, if the train and testing datasets have similar distributions then removing samples from one side will lower the generalization ability of the classifier.
All three of these things make it hard to keep the original net intact while also improving upon it. That is why almost every state-of-the-art algorithm has its own software package or toolset that allows you to easily perform transfer learning, create new architectures, and optimize individual parameters.
Make a backup of your model
It is very important to make sure you have a complete copy of your model before deleting it! If you don’t, then you will not be able to use that deleted version again.
The most common way to do this is by using one of the many online storage services or creating your own folder structure in the cloud.
Some people also save their models as JSON files which are just plain text. This can easily be done in any popular editor like Microsoft Word, Google Docs, or Notepad++.
By doing this, you keep control over what information the model contains because you choose where to store the file.
Store the model on a hard drive or external storage drive
It makes no sense to lose all of your saved models! If you work in an industry that requires you to have lots of different tools for your job, then it is totally normal to keep every one of these settings offline and sync them only when you are back at home.
Likewise, if you are a content creator who needs access to tons of pre-made designs and textures, you can save yourself some space by storing your models somewhere else than in the default location.
By using separate locations for each setting, you will give yourself more flexibility in how you organize things and what types of media you want to store.
This article will help you learn how to save your deep learning models effectively.
Encrypt your model
While it is not advised to share the weights of your neural network, encrypting them is! This can be done using either one of the two most common types of encryption for deep learning models-TensorFlow’s BIP32 (Barth et al.) or PyTorch’s AES (Application Specific Expression) modes.
The first way uses off-the-shelf cryptographic software such as PrivatePy or Pyro. These tools allow you to create an encrypted file that contains the trained weight values of your NN. When you are ready to deploy your model, you decrypt this file and use these saved weights directly in your code!
This method also makes it very difficult to access the weights due to the security measures put into place by the crypto library. Only someone with special equipment and knowledge of the algorithm could possibly do so. Therefore, even if someone manages to grab all of your training data, settings, and files, they would still have no clue what your net was originally configured to look like!
The second option is slightly more involved but has some additional benefits over the first. With AES mode, instead of just having the key to unlock the model, you get an expression for the key as well! This allows you to tweak certain parts of the model without needing the original password.
For example, say you wanted to test whether your model would work under different conditions.
Compress your model
A common way to save computer memory is to compress the model you have already trained. This can be done at either the layer level or the whole network level!
Layer-level compression comes in two main types, shrinking or pruning. Shrinking means reducing the size of an individual layer by using less powerful activation functions or thinner kernels. Pruning deletes parts of the layers that are no longer needed.
Network-level compressing uses different architectures of the same model, which have become very popular in recent years. These include things like inception modules or Xception networks, which are almost exclusively used for image classification tasks.
By replacing larger fully connected layers with smaller ones, you can get similar results while consuming less RAM.
Use a cloud backup service
If you are running into issues saving your model, make sure you have adequate backups of it! There are many free and paid services that allow you to back up all of your files online.
Cloud storage is becoming the standard for people to create and access their documents, photos, and other things. It is easy to use and can be accessed from anywhere!
Most good cloud backup sites offer very fast upload speeds and community support, making it more trustworthy. Some even offer you rewards points or monthly credits depending on how much data you store with them!
Google Cloud Storage is one such site that we recommend. Not only does it have incredible speed but also special pricing structures for professional users. You get 2GB per month for free which will enough space for almost anything!
To save your current neural network, open Google Cloud Storage and then select “Create new folder”. Type in a descriptive name like ‘SaveModel-DateX’ where DateX is the date and time of this writing (August 31, 2017 1:35 AM). Now, click “create.”
Next, go back to your computer and navigate to the newly created folder using Google Drive or another compatible app. Download the latest version of PyTorch by navigating to its website and downloading it directly there.
Once downloaded, extract the.py file so that you have the source code.
Update your model with the most recent changes
A very common cause of poor performance is outdated models!
Deep learning algorithms are extremely powerful, but also complex. Because of this, engineers have designed internal compartments within neural networks that need to be updated periodically.
These internal components are referred to as “layers” or sometimes even just “weights” because they influence the network by scaling the overall strength and size of certain equations.
When these layers become too old, they may not perform correctly due to rounding errors or numerical instability.
It is important to update your model when there has been an significant change to prevent inefficient training and loss of accuracy.
Surprise surprise, saving and loading a trained machine learning model is pretty easy! Let us dive in and learn how.
Test your model thoroughly
A very important part of any learning process is testing out what you have learned! This means not only using your model, but also practicing it on new data or settings that are different from those for which it was trained.
If you learn how to bake cookies, then trying to use your newly-acquired baking skills to make an oatmeal raisin cookie will probably fail spectacularly! By this we mean there’s just too much variability in the ingredients and processes used to achieve successful baked goods for you to be able to take into account all possible situations when deploying your deep learning model.
So rather than wasting your time by attempting to apply your knowledge directly to scenarios that may not work at all, try instead applying domain specific knowledge — in other words, knowing something about medicine so you can check if your model works for diagnosing diseases! Or perhaps you know some basic physics, so you could test whether your model would work as well predicting weather patterns.
The more diverse your source of information, the better! We recommend starting with things that seem related to the problem domain of your DL model and working from there.
Domain adaptation is another term sometimes referred to as “transfer learning” or “fine tuning”. It comes down to solving the same problem in a slightly altered way. For example, say you built a machine vision classifier to identify cars.
Back up your model again
It’s important to remember that even after you have saved all of your weights, there is still one more step before you can use your network for predictions. You will need to actually create an instance of the network or else you won’t be able to train it properly.
This is because the deep learning algorithm works in tandem with another set of codes called “back propagation.” When you are training your neural net, back propagation must occur so that the neurons learn how to connect with each other and perform functions.
If you don’t have this code already then you will need to either download it or find someone who does as you will not be able to continue training! Luckily, most people online these days leave their back propagation algorithms open source which makes things easy enough for anyone.
