MTL effectively increases the sample size that we are using for training our model. Learning just task A bears the risk of overfitting to task A, while learning A and B jointly enables the model to obtain a better representation F through averaging the noise patterns.

Implicit data augmentation

MTL can help the model focus its attention on those features that actually matter as other tasks will provide additional evidence for the relevance or irrelevance of those features.

Attention Focusing

Some features G are easy to learn for some task B, while being difficult to learn for another task A. Through MTL, we can allow the model to eavesdrop, i.e. learn G through task B.

Eavesdropping

MTL biases the model to prefer representations that other tasks also prefer. This will also help the model to generalize to new tasks in the future as a hypothesis space that performs well for a sufficiently large number of training tasks will also perform well for learning novel tasks as long as they are from the same environment

Representation Bias

MTL acts as a regularizer by introducing an inductive bias. As such, it reduces the risk of overfitting and improves its ability to fit random noise.

MTL as a Regularizer

For all examples, we will assume that we have two related tasks A and B, which rely on a common hidden layer representation F.

Hamilton College

In Machine Learning (ML), we typically care about optimizing for a particular metric. While we can generally achieve acceptable performance this way, by only focusing on a single task, we ignore information from training signals of related tasks that might help us do even better on the metric we care about. By contrast, MTL improves generalization by leveraging the domain-specific information contained in the training signals of related tasks. 

Multi-task Learning

Ruder, S. (n.d.). An Overview of Multi-Task Learning in Deep Neural Networks *. Retrieved May 22, 2022, from https://arxiv-download.xixiaoyao.cn/pdf/1706.05098.pdf

‌

Multi-Task Learning in Deep Neural Networks

1. Biological perspective: we can see multi-task learning as being inspired by human learning. For learning new tasks, we often apply the knowledge we have acquired by learning related tasks. 


2. Pedagogical perspective: we often learn tasks that provide us with the necessary skills to master more complex techniques first.


3. Machine learning perspective: We can view multi-task learning as a form of inductive transfer. Inductive transfer can help improve a model by introducing an inductive bias, which causes a model to prefer some hypotheses over others.


Motivations of MTL

In the context of Deep Learning, multi-task learning is typically done with either hard or soft parameter sharing of hidden layers.


Two MTL Methods for Deep Learning


Reasons Why MTL Works

There are two main ideas that have been pervasive throughout the history of multi-task learning: enforcing sparsity across tasks through norm regularization; and modeling the relationships between tasks.


MTL in Non-neural Models

While many recent Deep Learning approaches have more or less used multi-task learning as part of their model , they all employed either hard or soft parameter sharing, and sometimes both.


Learn Before

Related

Learn After