In MTL for computer vision, approaches often share the convolutional layers, while learning task-specific fully-connected layers. In addition to the structure of shared and task-specific layers, they place matrix priors on the fully connected layers, which allow the model to learn the relationship between tasks. 


Deep Relationship Networks

Fully-adaptive feature sharing is a bottom-up approach. It starts with a thin network and dynamically widens it greedily during training using a criterion that promotes grouping of similar tasks. However, the greedy method might not be able to discover a model that is globally optimal, while assigning each branch to exactly one task does not allow the model to learn more complex interactions between tasks.

Fully-Adaptive Feature Sharing

Cross-stitch networks start out with two separate model architectures just as in soft parameter sharing. They then use what they refer to as cross-stitch units to allow the model to determine in what way the task-specific networks leverage the knowledge of the other task by learning a linear combination of the output of the previous layers. Their architecture can be seen in the figure below, in which they only place cross-stitch units after pooling and fully-connected layers.


Cross-stitch Networks


A pre-defined hierarchical architecture consisting of several NLP tasks, which can be seen in the figure below, as a joint model for multi-task learning.


A Joint Many-Task Model

It's a model that generalizes Deep-Learning-based MTL approaches such as hard parameter sharing and cross-stitch networks, block-sparse regularization approaches, as well as recent NLP approaches that create a task hierarchy. The model allows to learn what layers and subspaces should be shared, as well as at what layers the network has learned the best representations of the input sequences.


Sluice Networks

Instead of learning the structure of sharing, this orthogonal approach considers the uncertainty of each task. They then adjust each task’s relative weight in the cost function by deriving a multi-task loss function based on maximizing the Gaussian likelihood with task-dependent uncertainty.


Weighting Losses with Uncertainty

While many recent Deep Learning approaches have more or less used multi-task learning as part of their model , they all employed either hard or soft parameter sharing, and sometimes both.


Hamilton College

In Machine Learning (ML), we typically care about optimizing for a particular metric. While we can generally achieve acceptable performance this way, by only focusing on a single task, we ignore information from training signals of related tasks that might help us do even better on the metric we care about. By contrast, MTL improves generalization by leveraging the domain-specific information contained in the training signals of related tasks. 

Multi-task Learning

Ruder, S. (n.d.). An Overview of Multi-Task Learning in Deep Neural Networks *. Retrieved May 22, 2022, from https://arxiv-download.xixiaoyao.cn/pdf/1706.05098.pdf

‌

Multi-Task Learning in Deep Neural Networks

1. Biological perspective: we can see multi-task learning as being inspired by human learning. For learning new tasks, we often apply the knowledge we have acquired by learning related tasks. 


2. Pedagogical perspective: we often learn tasks that provide us with the necessary skills to master more complex techniques first.


3. Machine learning perspective: We can view multi-task learning as a form of inductive transfer. Inductive transfer can help improve a model by introducing an inductive bias, which causes a model to prefer some hypotheses over others.


Motivations of MTL

In the context of Deep Learning, multi-task learning is typically done with either hard or soft parameter sharing of hidden layers.


Two MTL Methods for Deep Learning


For all examples, we will assume that we have two related tasks A and B, which rely on a common hidden layer representation F.

Reasons Why MTL Works

There are two main ideas that have been pervasive throughout the history of multi-task learning: enforcing sparsity across tasks through norm regularization; and modeling the relationships between tasks.


Learn Before

Related

Learn After