All entities are replaced by entity markers and using a coreference.
The anonymised entity markers are constantly permuted during training and testing.

Original version and Anonymised Version

Cloze form queries
ex1) The hi-tech bra that helps you beat breast X
ex2) Could Saccharin help beat X ?
ex3) Can fish oils help fight prostate X ?

An n-gram language model trained on the Daily Mail would easily correctly predict that (X= cancer), regardless of the context document, since it is a very frequently cured entity in the Dataset(Daily Mail corpus).


State University of New York at Stony Brook

Natural language documents are read and studied by computers so they can answer the questions posed.

Machine Reading and Comprehension

Hermann, K. M., Kocisky, T., Grefenstette, E., Espeholt, L., Kay, W., Suleyman, M., & Blunsom, P. (2015). Teaching machines to read and comprehend. Advances in neural information processing systems, 28, 1693-1701.
https://arxiv.org/abs/1506.03340v3

Teaching Machines to Read and Comprehend

- The researchers want to establish the difficulty of their machine reading task by applying a wide range of models to it
- The researchers want to compare the performance of parse-based methods to neural models
- Within the group of neural models examined, the researchers want to determine what each component contributes to the end performance; that is, we want to analyze the extent to which an LSTM can solve this task, and to what extent various attention mechanisms impact performance.

Teaching Machines to Read and Comprehend Purpose

The reading comprehension task is approached by creating two machine reading corpora of online newspaper articles and summaries from two news providers. Since the summary bullet points are abstractive, a corpus of document-query-answer triples is constructed in which the bullet points are replaced one at a time with a placeholder. In this approach, the researchers are seeking the estimation of conditional probability $p(a|c,q)$.

Supervised Training Data

- The researchers predict the neural models should in principle be well suited for this task
- The researchers predict the simple recurrent models such as the LSTM probably have insufficient expressive power for solving tasks that require complex inference
- The researchers predict the attention-based models would therefore outperform the pure LSTM-based approaches

Teaching Machines to Read and Comprehend Hypothesis

Entity Replacement and Permutation

Better datasets and tasks to evaluate the capabilities of machine reading models are needed. 
Proceed by describing a number of baselines, benchmarks and new models to evaluate against this paradigm.

For the reading comprehension task, the paper used 2 simple base line models, 2 symbolic matching models, and 4 RNN models.

Models

1) Mechanism
2) Data collection
3) Methods

Learn Before

Related

Learn After