- The figure below shows an example of training data, apricot being the target word and 'tablespoon ,of ,jam and a' being the context word.
- Context window is $ L = \pm 2 $.              

Skip-Gram Training data

From the given set of positive and negative training examples and initial weight for embeddings the goal of the learning algorithm is to learn to maximize the similarity of target word and context word from the positive examples and to minimize the similarity of target word and context word from the negative examples.

How to learn vectors

- A corpus of text is taken as input for learning the skip-gram embeddings by the learning algorithm.
- A vocabulary of size N is also chosen.
- For each word skip-gram algorithm learns two embeddings target embedding and context embedding.

San Jose State University

Word2vec (also called Skip-gram) algorithm is the method to obtain short dense vectors of words. 

This method is self-supervision: training the model by just running the text without explicit labels for data.

Word2vec applies binary logistic regression classifier.


Word2vec

An on-going but a helpful book resource about NLP
https://web.stanford.edu/~jurafsky/slp3/

Speech and Language Processing (3rd ed. draft) 

The classifier for word2vec is a binary logistic regression, which applies sigmoid function.
given a tuple of words$$(w,c)$$, where $$w $$is the target word, $$c $$is one of the context words:
The possibility that $$c$$ is a context word: 
$$
P\left ( +|w,c \right ) = \sigma(c \cdot w)=\frac{1}{1+exp(- \, c  \cdot w)}
$$
The possibility that $$c$$ is not a context word:
$$
P\left ( -|w,c \right ) = 1-P\left ( +|w,c \right )=\sigma(- \,\ c\cdot w)= \frac{1}{1+exp(c \cdot w)}
$$
But there are several context words:
$$
P\left ( +|w,c_{1:L} \right ) = \prod^L_{i=1}\sigma(c_i \cdot w)=\prod^L_{i=1}\frac{1}{1+exp(- \, c_i  \cdot w)}
$$

Classifier for word2vec

Learning skip-gram embeddings


Other static embeddings available are
- fasttext
- GloVe

Learn Before

Related

Learn After