Word analogy tasks demonstrate that pretrained word vectors successfully capture meaningful semantic information from corpora. For example, evaluating a "male-female" analogy by computing "man":"woman"::"son":? correctly yields the word "daughter". Furthermore, word embeddings capture geographic semantics; completing a "capital-country" analogy like "beijing":"china"::"tokyo":? accurately predicts the word "japan".

Claude

Word vectors can be applied to word analogy tasks. A word analogy completion task is defined by the relationship $$a:b::c:d$$, meaning "$$a$$ is to $$b$$ as $$c$$ is to $$d$$". Given the first three words $$a$$, $$b$$, and $$c$$, the objective is to find the missing word $$d$$. If we denote the numerical representation of a word $$w$$ as $$\textrm{vec}(w)$$, we can complete the analogy by evaluating the vector arithmetic $$\textrm{vec}(c) + \textrm{vec}(b) - \textrm{vec}(a)$$ and finding the word in the vocabulary whose vector is most similar to this result.

Word Analogy Task

Dive into Deep Learning

The word analogy completion task can be implemented programmatically by defining a function, such as `get_analogy`, that takes the first three text tokens ($$a$$, $$b$$, and $$c$$) along with a pretrained embedding object. It extracts the numerical vectors for the three words, evaluates the expression $$\textrm{vec}(c) + \textrm{vec}(b) - \textrm{vec}(a)$$, and uses a nearest neighbors function (`knn`) to find the most similar vector in the embedding matrix, returning its corresponding word.

```python
def get_analogy(token_a, token_b, token_c, embed):
    vecs = embed[[token_a, token_b, token_c]]
    x = vecs[1] - vecs[0] + vecs[2]
    topk, cos = knn(embed.idx_to_vec, x, 1)
    return embed.idx_to_token[int(topk[0])]  # Remove unknown words
```

get_analogy Function Implementation

Examples of Semantic Information in Pretrained Word Vectors

Beyond semantic relationships, pretrained word vectors also encode syntactic information about a language. Word analogy tasks can verify this syntax retention. For instance, testing an "adjective-superlative adjective" analogy such as "bad":"worst"::"big":? correctly produces the word "biggest". Similarly, word embeddings capture the notion of verb tenses, as demonstrated by the "present tense-past tense" analogy where "do":"did"::"go":? correctly yields the word "went".

Learn Before

Related