- Locality
- Prior from lower modules
- multi-task adapters
- attention with only prior -> attention distribution that is independent of pair-wise interaction between inputs. In other words, their models exploit only a prior attention distribution

Carleton College

$Prior$ is an attention distribution that comes from a different source than previous from inputs (e.g., softmax(QK$^T$)in vanilla Transformer). Attention with prior is the fusion of two attention distributions, which can be done by computing a weighted sum of the scores corresponding to the prior and generated attention before applying softmax

Attention with Prior

Lin, Tianyang & Wang, Yuxin & Liu, Xiangyang & Qiu, Xipeng. (2021). A Survey of Transformers. 

Learn Before

Related