This is one of the greatest challenges posed. There really is no determinant in what makes the current network sufficient or similar enough to the one you are trying to apply. In fact, the algorithm can perform even worse if it fails to find this connection. The model may even do worse than if it was never even trained in the first place.


Transfer Learning: What is similar "enough"?

- Need very large dataset
- Data must be general and not domain-specific
- Only works if initial and current problem are similar enough


University of Michigan - Ann Arbor

The longest part of the development of a deep learning model is that of the training stage. However, if the problem that is being tackled has already been studied and worked on by someone else and the model has been released to the public including the trained weights then a technique to substantially increase the training process is to utilize transfer learning.

Transfer Learning in Deep Learning

https://towardsdatascience.com/transfer-learning-using-the-fastai-library-d686b238213e

Transfer Learning Reference

Disadvantages of Transfer Learning

According to DeepMind CEO Demis Hassabis, transfer learning is also one of the most promising techniques that could lead to artificial general intelligence (AGI) someday.

ideas about transfer learning from the specialist

Transfer from Task A to Task B

- Tasks A and B have the same input x
- You have a lot more data for Task A than Task B
- Low-level features from A could be helpful for learning B

When Transfer Learning in Deep Learning Makes Sense

- Developing your own model
- Using a pre-trained model

Two Approaches to Transfer Learning in Deep Learning

An example of this concept exists in image classification. If someone wants to classify kinds of clouds it can be very difficult to compile enough data to train a model from scratch. However, models such as ImageNet have already been trained on classifying a lot more and different kinds of images so under the assumption that the earlier layers in the model hold some fundamental information about understanding images then transfer learning can be used. Then, we need to replace the output layer of the pre-trained model with a new output layer(s) that match the number of classes of the new output variable and then train the model. 

Transfer Learning in Deep Learning Example

Transfer learning has several benefits, but the main advantages are  saving training time, better performance of neural networks (in most cases), and not needing a lot of data. 

Usually, a lot of data is needed to train a neural network from scratch but access to that data isn't always available — this is where transfer learning comes in handy. With transfer learning a solid machine learning model can be built with comparatively little training data because the model is already pre-trained. This is especially valuable in natural language processing because mostly expert knowledge is required to create large labeled datasets. Additionally, training time is reduced because it can sometimes take days or even weeks to train a deep neural network from scratch on a complex task.

Learn Before

Related

Learn After