MTL effectively increases the sample size that we are using for training our model. Learning just task A bears the risk of overfitting to task A, while learning A and B jointly enables the model to obtain a better representation F through averaging the noise patterns.

Hamilton College

For all examples, we will assume that we have two related tasks A and B, which rely on a common hidden layer representation F.

Reasons Why MTL Works

Ruder, S. (n.d.). An Overview of Multi-Task Learning in Deep Neural Networks *. Retrieved May 22, 2022, from https://arxiv-download.xixiaoyao.cn/pdf/1706.05098.pdf

‌

Multi-Task Learning in Deep Neural Networks

Implicit data augmentation

MTL can help the model focus its attention on those features that actually matter as other tasks will provide additional evidence for the relevance or irrelevance of those features.

Attention Focusing

Some features G are easy to learn for some task B, while being difficult to learn for another task A. Through MTL, we can allow the model to eavesdrop, i.e. learn G through task B.

Eavesdropping

MTL biases the model to prefer representations that other tasks also prefer. This will also help the model to generalize to new tasks in the future as a hypothesis space that performs well for a sufficiently large number of training tasks will also perform well for learning novel tasks as long as they are from the same environment

Representation Bias

MTL acts as a regularizer by introducing an inductive bias. As such, it reduces the risk of overfitting and improves its ability to fit random noise.

Learn Before

Related