Multi-lingual BERT (mBERT)
Multi-lingual BERT (mBERT) is a version of BERT trained on text from 104 different languages. Its main distinction from monolingual BERT is the use of a significantly larger vocabulary to accommodate tokens from this diverse set of languages. This design allows mBERT to map representations from different languages into a common vector space, which enables the model to share and transfer knowledge across languages.
0
1
Tags
Ch.1 Pre-training - Foundations of Large Language Models
Foundations of Large Language Models
Foundations of Large Language Models Course
Computing Sciences
Related
Multi-lingual BERT (mBERT)
Multilingual and Language-Specific PTMs
Language Model Development Strategy
A startup with limited computational resources is developing a feature to classify customer support tickets across 20 different languages. Several of these are low-resource languages with small datasets. Considering the trade-offs between performance, cost, and data availability, which strategy for building the underlying language model is most advisable?
A team is developing a natural language processing system for a global audience. They are considering two different strategies for handling multiple languages. Match each strategy with its most significant trade-off.
Learn After
A key design choice for a multilingual language model is to train it on a combined text corpus from many languages using a single, shared vocabulary. What is the most significant functional consequence of this design?
Cross-Lingual System Development Strategy
Architectural Trade-offs in Multilingual Models