Rather than sparsity, we would thus like to leverage prior knowledge indicating that some tasks are related while others are not. In this scenario, a constraint that enforces a clustering of tasks might be more appropriate. To learn task relationships, we can impose a clustering constraint by penalizing both the norms of our task column vectors a·,1, . . . , a·,t as well as their variance with the following constraint:

Hamilton College

There are two main ideas that have been pervasive throughout the history of multi-task learning: enforcing sparsity across tasks through norm regularization; and modeling the relationships between tasks.


MTL in Non-neural Models

Ruder, S. (n.d.). An Overview of Multi-Task Learning in Deep Neural Networks *. Retrieved May 22, 2022, from https://arxiv-download.xixiaoyao.cn/pdf/1706.05098.pdf

‌

Multi-Task Learning in Deep Neural Networks

While in the single-task setting, the l1 norm is computed based on the parameter vector at of the respective task t, whereas for MTL we compute it over our task parameter matrix A.  Depending on what constraint we would like to place on each row, we can use a different lq. In general, we refer to these mixed-norm constraints as l1/lq norms. They are also known as block-sparse regularization, as they lead to entire rows of A being set to 0. 

Learn Before

Related