A dataset focuses on contextual word similarity, which gives judgments on whether a certain word has same meaning in two given sentential contexts, which is a binary classification task in nature.

University of Michigan - Ann Arbor

It is important to compare the vector model results with other models, as well as practicing intrinsic evaluations, which is to test its performance compared to gold-standard datasets, by computing the correlation between the word similarity scores of the model, and those scores from human judgements.

Evaluating Vector Models

An on-going but a helpful book resource about NLP
https://web.stanford.edu/~jurafsky/slp3/

Speech and Language Processing (3rd ed. draft) 

A dataset that contains 353 noun pairs, with their similarity score with scale 0-10. The average score is 5.77 in the dataset.

WordSim-353 dataset

A dataset that comprises 666 Noun-Noun pairs, 222 Verb-Verb pairs and 111 Adjective-Adjective pairs. It explicitly focuses on quantifying similarity rather than relatedness, compared to WordSim-353.


SimLex-999 dataset

A dataset of 80 questions, each question consists of 4 word choices, and 1 target word, e.g. Levied is closest in meaning to: imposed, believed, requested, correlated.

TOEFL dataset

A dataset focuses on contextual word similarity, which gives human judgments on pairs of words in their sentential context. For each word in a given pair, its context and part-of-speech tag are given; The context is the word sequences around the target word.

Learn Before

Related