The only times multi-task learning hurts performance compared to training separate neural networks is if the neural network isn't big enough. But if a big enough neural network can be trained, then multi-task learning certainly should not or should very rarely hurt performance. It might even help performance compared to if you were training neural networks to do these different tasks in isolation.

When Multi-Task Learning Doesn't Make Sense

An Overview of Multi-Task Learning in Deep Neural Networks

1) Hard Parameter Sharing

2) Soft Parameter Sharing

MTL Methods for Deep Learning

Instead of one output, we use avector of binary outputs in the final layer. For any datapoint, any number of these outputs can be 0 or 1. This is in contrast to the Softmax activation function where only one label could be true.

Because we have a series of (probably independent) outputs, the loss function is Sigmoid, similar to Logistic regression, but instead of one output variable, y insidcates a vector of binary outputs.

Implementing Multi-task Learning in Deep Learning

- **Training on a set of tasks that could benefit from having shared low-level features:** For example, recognizing traffic lights, cars, and pedestrians should have similar features that could help recognize stop signs, because these are all features of roads. This would save time and result in better performance.
- **The amount of data you have for each task is similar:** However, this rule may not be always valid. For example, we had 4 tasks in the self-driving example and each of these tasks has 100 examples. Now because the learning is simultaneous, each task does not need to solely depend on its own examples and can use some of the 300 other examples from the other tasks. When the number of tasks, is increased to, say 100, each task has potentially 9900 examples, 100 for each of the 99 other tasks to draw from.
- **Can train a big enough neural network to do well on all the tasks:** The alternative would be to train a separate neural network for each task. So rather than training one neural network for detecting all the pedestrians, cars, stop signs, and traffic lights, you could have trained a neural network for each of the tasks. But if some of the earlier features in neural network can be shared between these different types of objects, then training one neural network to do four things results in better performance than training four neural networks to do the four tasks separately.

When Multi-task Learning in Deep Learning Makes Sense

A self-driving car would need to detect multiple things simultaneously, such as pedestrians, stop signs, other cars, and traffic lights. So in an input image, we would have four labels to check for each of the four objects. This gives us a y(i) vector of dimension 4 x 1, and consequently our training data labels would be of dimension 4 x m, where m is the number of y(i) vectors taken. We can now use a neural network to predict values of y by inputting an image and getting an output four dimensional matrix. This method also works if a few of the labels are missing in some of the input images.

Example of a Self-Driving Car for Multi-task Learning in Deep Learning

In multi-task learning, the goal is to try to have one neural network classify/predict multiple outputs at the same time, with each of the tasks helping in the execution of the other tasks, i.e., learning from multiple tasks. This contrasts with transfer learning where the tasks are done sequentially, with the previous task helping in the execution of the next.

University of Michigan - Ann Arbor

Initial weights are applied to all the neurons.
It is necessary to set initial weights for the first forward pass. Two basic options are to set weights to zero or to randomize them. However, this can result in a vanishing or exploding gradient, which will make it difficult to train the model. To mitigate this problem, you can use a heuristic (a formula tied to the number of neuron layers) to determine the weights. A common heuristic used for the Tanh activation is called Xavier initialization.

On top of that if the architecture being used has already been trained by someone else (such as ImageNet) the node values can be initialized to those values. This is known as transfer learning and it can be very useful to substantially increase training speed and model accuracy.

Deep Learning Weight Initialization

https://www.coursera.org/learn/machine-learning-projects?specialization=deep-learning

Structuring Machine Learning Projects

For a single neuron example: you want to train a deep network without the weights exploding or vanishing.

Example of Weight Initialization 

In a neural network with many time steps or layers, a gradient at the early layer is the product of all the terms from the later layers, which leads to an inherently unstable situation. Especially when the value of gradient has become so small, it no longer updates properly or is vanished eventually. Exploding gradient can be considered as the opposite of vanishing process. The updated weights using gradient descent become so large that they cause the whole network to become unstable, which leads to numerical overflow. 

Vanishing/exploding gradient

Symmetry breaking refers to a requirement of initializing machine learning models like neural networks.

When a neural network model has weights, all initialized to the same value, it can be difficult or impossible for the weights to differ as the model is trained. This is known as the “symmetry” problem.

Initializing the model to small random values breaks the symmetry and allows different weights to learn independently of each other.

Symmetry Breaking in Deep Learning


To prevent the gradients of the network’s activations from vanishing or exploding, we will stick to the following rules of thumb:

- The mean of the activations should be zero.
- The variance of the activations should stay the same across every layer.

Under these two assumptions, the backpropagated gradient signal should not be multiplied by values too small or too large in any layer.

Ensuring zero-mean and maintaining the value of the variance of the input of every layer guarantees no exploding/vanishing signal

How to Initialization Weights to Prevent Vanishing/Exploding Gradients

The longest part of the development of a deep learning model is that of the training stage. However, if the problem that is being tackled has already been studied and worked on by someone else and the model has been released to the public including the trained weights then a technique to substantially increase the training process is to utilize transfer learning.

Transfer Learning in Deep Learning

Multi-task Learning in Deep Learning

When analyzing the scale distribution of an output $$o_i$$ for a fully connected layer without nonlinearities, the output is computed as $$o_i = \sum_{j=1}^{n_	extrm{in}} w_{ij} x_j$$. Assuming the inputs $$x_j$$ and weights $$w_{ij}$$ are drawn independently with a mean of $$0$$ and variances of $$\gamma^2$$ and $$\sigma^2$$ respectively, the expected value $$E[o_i]$$ is $$0$$. We can compute the variance as $$	extrm{Var}[o_i] = E[o_i^2] - (E[o_i])^2 = \sum_{j=1}^{n_	extrm{in}} E[w^2_{ij} x^2_j] - 0 = \sum_{j=1}^{n_	extrm{in}} E[w^2_{ij}] E[x^2_j] = n_	extrm{in} \sigma^2 \gamma^2$$. Note that the distribution does not have to be Gaussian, but the mean and variance must exist. To keep this variance fixed during forward propagation and prevent it from changing across layers, the initialization must satisfy the condition $$n_	extrm{in} \sigma^2 = 1$$.

Variance of Layer Output in Forward Propagation

When building a neural network, if a specific parameter initialization method is not explicitly defined by the user, the deep learning framework will apply a default random initialization method. This default approach, such as drawing weight values from a standard normal distribution, is often sufficient and works well in practice for models with moderate problem sizes.

Default Random Initialization

Xavier initialization, named after its creators Glorot and Bengio (2010), is a standard technique designed to mitigate vanishing and exploding gradients by carefully setting the initial weights of a neural network layer. To balance the variance during both forward and backward propagation, it typically samples weights from a Gaussian distribution with a mean of $$0$$ and a variance of $$\sigma^2 = \frac{2}{n_	extrm{in} + n_	extrm{out}}$$, where $$n_	extrm{in}$$ and $$n_	extrm{out}$$ represent the number of inputs and outputs of the layer respectively. While the underlying assumption of linear activations is often violated in practice, this initialization method has proven highly effective.

Learn Before

Related

Learn After