Multilingual language models (LMs) are typically trained in two phases: a “pre-training” phase, in which unlabeled data from one or multiple languages may be used, and a “fine-tuning” phase, in which task-specific labeled data is used to build a system capable of solving a target task.

Two-Phase Training of Multilingual Language Models

Massive multilingual language models, such as multilingual BERT, have shown promise in enabling the coverage of low-resource languages without any labeled data, by using a zero-shot framework.

University of Michigan - Ann Arbor

Google

Language technologies cannot yet process code-switching (C-S)—alternating between two or more languages within a single conversation—as seamlessly as monolingual data. The current state of computational C-S processing faces three main limitations: data (scarce C-S datasets), evaluation (models such as multilingual BERT underperform on C-S relative to monolingual settings), and user-facing applications (few end-to-end tools serve multilingual communities).

Computational Approaches to Code-Switching (C-S)

While multilingual BERT outperforms older task-specific models on C-S tasks, its performance on C-S data is significantly worse than on the same tasks in a monolingual setting. This inefficiency occurs because these models are either not exposed to C-S data at all during training, or they miss out on several language pairs.

Inefficiency of Multilingual BERT on C-S Tasks

Since C-S languages tend to be low-resourced, building deep learning-based models is challenging due to the lack of large C-S datasets.

Challenge of Deep Learning for C-S Languages

Massive Multilingual Language Models (LMs)

Limitation:  the lack of end-to-end userfacing applications that interact directly with users in multilingual communities.

1) Although computational implementations of grammatical models of C-S exist (Bhat et al., 2016), they do not necessarily generate natural C-S utterances that a bilingual speaker would produce

2) Most crucially, current computational approaches to C-S language technologies do not usually take into account the linguistic and social factors that influence why and when speakers/users choose to code-switch.

Learn Before

Related

Learn After