These LMs are typically trained in two phases: a “pre-training” phase, in which unlabeled data from one or multiple languages may be used and a “fine-tuning” phase, in which task-specific labeled data is used to build a system capable of solving the task.

University of Michigan - Ann Arbor

language technologies are not yet capable of processing C-S as seamlessly as monolingual data. 

We identify three main limitations of the current state of computational processing of C-S: data, evaluation and user-facing applications.

Computational Approaches to C-S

 Since C-S languages tend to be low resourced, building Deep Learningbased models is challenging due to the lack of large C-S datasets.

Challenge of the use of deep Neural Networks

 multilingual BERT have shown promise in enabling the coverage of lowresource languages without any labeled data by using the zero-shot framework

Massive multilingual Language Models (LMs)

Mechanism of Language Models

multilingual BERT outperforms older task-specific models on C-S tasks, however, the performance on C-S is much worse than the performance on the same tasks in a monolingual setting.

they are either not exposed to C-S data at all during training, or they miss out on several language pairs

Learn Before

Related