Background: The quantity of data in a Many-to-Many dataset increases quadratically with the number of languages, making neural networks with standard capacity underfit rapidly.
What they did outstandingly: "we leverage progress in scaling (Kaplan et al., 2020; Arora et al., 2018) to train models that are over 50 times larger than current bilingual models with model parallelism"

Size of models

1) Language selection \n 
2) Evaluation
3) Covering the Language Matrix by Mining

How to Build a Many-to-Many Parallel Dataset for 100 Languages

1) Mechanism
2) Data collection
3) Methods

University of Michigan - Ann Arbor

Natural language documents are read and studied by computers so they can answer the questions posed.

Machine Reading and Comprehension

Hermann, K. M., Kocisky, T., Grefenstette, E., Espeholt, L., Kay, W., Suleyman, M., & Blunsom, P. (2015). Teaching machines to read and comprehend. Advances in neural information processing systems, 28, 1693-1701.
https://arxiv.org/abs/1506.03340v3

Teaching Machines to Read and Comprehend

- The researchers want to establish the difficulty of their machine reading task by applying a wide range of models to it
- The researchers want to compare the performance of parse-based methods to neural models
- Within the group of neural models examined, the researchers want to determine what each component contributes to the end performance; that is, we want to analyze the extent to which an LSTM can solve this task, and to what extent various attention mechanisms impact performance.

Teaching Machines to Read and Comprehend Purpose

The reading comprehension task is approached by creating two machine reading corpora of online newspaper articles and summaries from two news providers. Since the summary bullet points are abstractive, a corpus of document-query-answer triples is constructed in which the bullet points are replaced one at a time with a placeholder. In this approach, the researchers are seeking the estimation of conditional probability $p(a|c,q)$.

Supervised Training Data

- The researchers predict the neural models should in principle be well suited for this task
- The researchers predict the simple recurrent models such as the LSTM probably have insufficient expressive power for solving tasks that require complex inference
- The researchers predict the attention-based models would therefore outperform the pure LSTM-based approaches

Teaching Machines to Read and Comprehend Hypothesis

Cloze form queries
ex1) The hi-tech bra that helps you beat breast X
ex2) Could Saccharin help beat X ?
ex3) Can fish oils help fight prostate X ?

An n-gram language model trained on the Daily Mail would easily correctly predict that (X= cancer), regardless of the context document, since it is a very frequently cured entity in the Dataset(Daily Mail corpus).


Entity Replacement and Permutation

Better datasets and tasks to evaluate the capabilities of machine reading models are needed. 
Proceed by describing a number of baselines, benchmarks and new models to evaluate against this paradigm.

For the reading comprehension task, the paper used 2 simple base line models, 2 symbolic matching models, and 4 RNN models.

Learn Before

Related

Learn After