- Fasttext addresses the issues associated with word2vec .
- Word2vec is not good on words not seen during the training phase.
- Word2vec also fails when the words are too sparse.
- Fasttext uses subword model to address the problems associated with word2vec.

Fasttext

- Short form for Global Vectors.
- Captures global corpus statistics.
- Based on ratios of probabilities from the word-word co-occurrence matrix, also incorporates linear structure properties from word2vec.

GloVe

Other static embeddings available are
- fasttext
- GloVe

San Jose State University

Word2vec (also called Skip-gram) algorithm is the method to obtain short dense vectors of words. 

This method is self-supervision: training the model by just running the text without explicit labels for data.

Word2vec applies binary logistic regression classifier.


Word2vec

An on-going but a helpful book resource about NLP
https://web.stanford.edu/~jurafsky/slp3/

Speech and Language Processing (3rd ed. draft) 

The classifier for word2vec is a binary logistic regression, which applies sigmoid function.
given a tuple of words$$(w,c)$$, where $$w $$is the target word, $$c $$is one of the context words:
The possibility that $$c$$ is a context word: 
$$
P\left ( +|w,c \right ) = \sigma(c \cdot w)=\frac{1}{1+exp(- \, c  \cdot w)}
$$
The possibility that $$c$$ is not a context word:
$$
P\left ( -|w,c \right ) = 1-P\left ( +|w,c \right )=\sigma(- \,\ c\cdot w)= \frac{1}{1+exp(c \cdot w)}
$$
But there are several context words:
$$
P\left ( +|w,c_{1:L} \right ) = \prod^L_{i=1}\sigma(c_i \cdot w)=\prod^L_{i=1}\frac{1}{1+exp(- \, c_i  \cdot w)}
$$

Classifier for word2vec

- A corpus of text is taken as input for learning the skip-gram embeddings by the learning algorithm.
- A vocabulary of size N is also chosen.
- For each word skip-gram algorithm learns two embeddings target embedding and context embedding.

Learn Before

Related

Learn After