Able to scale Transformers and directly translate between 100 languages without pivoting through English at a performance that is competitive with bilingual models on many competitive benchmarks, including WMT

University of Michigan - Ann Arbor

Background: The quantity of data in a Many-to-Many dataset increases quadratically with the number of languages, making neural networks with standard capacity underfit rapidly.
What they did outstandingly: "we leverage progress in scaling (Kaplan et al., 2020; Arora et al., 2018) to train models that are over 50 times larger than current bilingual models with model parallelism"

Learn Before

Related