we increase the depth and width of standard Transformer architectures.

Two direction:

The first direction focuses on fitting a larger model on single device by reducing the memory required by activations and optimizer states during the training process. The second direction focuses on efficient training of even larger models through model parallelism e.g. splitting a model across multiple devices.

Dense Scaling

several strategies to increase the capacity of a sequence-to-sequence Transformer model in the context of multilingual machine translation.

Training Large Dense Models

a layer whose parameters are split by language or language group based on similarity in vocabulary.

Each translation direction only accesses a subset of these parameters, allowing the model capacity to scale without significantly affecting the training and inference time.

Scaling Model Capacity with Language-Specific Parameters

Goal: 

is to build a single model capable of translating 9, 900 language directions covering 100 languages. 

Challenge: 

This creates several challenges for models with insufficient capacity to capture that many languages and scripts adequatel

University of Michigan - Ann Arbor

1) Language selection \n 
2) Evaluation
3) Covering the Language Matrix by Mining

How to Build a Many-to-Many Parallel Dataset for 100 Languages

The first step of establishing a Many-to-Many dataset is to select 100 languages for which there already exist high-quality, annotated datasets that can be used for model evaluation.

                 
       Therefore, We cover a diversity of scripts and resource levels (as shown in Table 1) to have high coverage of languages worldwide. (widely spoken language)

Language selection

Supervised translation systems rely on large quantities of parallel sentences

Hence, we use mining technique

Covering the Language Matrix

Components for Scaling Multilingual Translation Models

It ends with a human evaluation study to understand the quality of our model translations.

We focus on 20 different directions, none of them involving English. We include languages commonly spoken in the same region, such as Japanese-Chinese, Hindi-Tamil, and RussianUkrainian

Human Evaluation on model translation process

We use publicly available evaluation datasets to evaluate the performance of all of our models. 

To cover our set of 100 languages and 2200 direction

Learn Before

Related

Learn After