Google

Learning multilingual text representations shared across languages plays an important role in many cross-lingual NLP tasks.
Examples:

- Multilingual BERT3 (mBERT): It is pre-trained by MLM with the shared vocabulary and weights on Wikipedia text from the top 104 languages. Each training sample is a monolingual document, and there are no cross-lingual objectives specifically designed nor any cross-lingual data. Even so, mBERT performs cross-lingual generalization surprisingly well.

- Cross-Lingual Language Model (XLM): XLM improves mBERT by incorporating a cross-lingual task, translation language modeling (TLM), which performs MLM on a concatenation of parallel bilingual sentence pairs.

Although multilingual PTMs perform well on many languages, recent work showed that PTMs trained on a single language significantly outperform the multilingual results. For Chinese, which does not have explicit word boundaries, modeling larger granularity and multigranularity word representations have shown great success.
Some monolingual PTMs have been released for different languages, such as CamemBERT and FlauBERT for French, Fin-BERT for Finnish, BERTje and RobBERT for
Dutch, AraBERT for Arabic language.

Multilingual and Language-Specific PTMs

Cross-lingual learning is a technique where a model is trained for a specific task in one language and then utilized to perform the same task in a different language. This approach leverages knowledge gained from a data-rich language to apply to languages with fewer resources, using multi-lingual pre-trained models as a foundation.

Cross-Lingual Learning

A significant improvement for multilingual pre-trained models, such as mBERT, involves incorporating bilingual data into the pre-training process. In contrast to training on separate monolingual corpora, this approach explicitly models the relationships between tokens from two different languages. This method equips the model with inherent cross-lingual transfer abilities, making it more readily adaptable to new languages.

Bilingual Pre-training for Multilingual Models

A key advantage of multi-lingual pre-trained models is their natural capacity to effectively process and handle texts that feature code-switching.

Benefit of Multilingual Pre-trained Models: Handling Code-Switching

In multilingual pre-trained models, tokens from different languages are not explicitly identified by their source language. Instead, they are all treated as entries within a single, unified vocabulary. This approach effectively creates a composite 'language' that includes the vocabularies of all processed languages, allowing the model to handle multilingual text seamlessly.

Shared Vocabulary in Multilingual Models

The effectiveness of a multilingual pre-trained model, assuming a fixed architecture, is determined by several key configuration choices. These include the size of the shared vocabulary, the proportion of training data allocated to each language, and the overall size of the model itself.

Factors Influencing Multilingual Pre-training

A company is developing a sentiment analysis tool. Their primary market is in France, for which they have a massive, high-quality dataset. They also need to provide functional support for Spanish and German, but have very limited data for these languages. The highest priority is achieving state-of-the-art performance for the French market, while still being able to handle the other languages. Given these requirements, which strategy for choosing a foundational model is most appropriate?

Based on the principles of pre-training language models, analyze why Model B is the expected top performer in this scenario. What is the core trade-off between the multilingual and language-specific approaches that this case illustrates?

Learn Before

Related