All entities are replaced by entity markers and using a coreference.
The anonymised entity markers are constantly permuted during training and testing.

State University of New York at Stony Brook

Cloze form queries
ex1) The hi-tech bra that helps you beat breast X
ex2) Could Saccharin help beat X ?
ex3) Can fish oils help fight prostate X ?

An n-gram language model trained on the Daily Mail would easily correctly predict that (X= cancer), regardless of the context document, since it is a very frequently cured entity in the Dataset(Daily Mail corpus).


Entity Replacement and Permutation

Hermann, K. M., Kocisky, T., Grefenstette, E., Espeholt, L., Kay, W., Suleyman, M., & Blunsom, P. (2015). Teaching machines to read and comprehend. Advances in neural information processing systems, 28, 1693-1701.
https://arxiv.org/abs/1506.03340v3

Learn Before