The classifier for word2vec is a binary logistic regression, which applies sigmoid function.
given a tuple of words$$(w,c)$$, where $$w $$is the target word, $$c $$is one of the context words:
The possibility that $$c$$ is a context word: 
$$
P\left ( +|w,c \right ) = \sigma(c \cdot w)=\frac{1}{1+exp(- \, c  \cdot w)}
$$
The possibility that $$c$$ is not a context word:
$$
P\left ( -|w,c \right ) = 1-P\left ( +|w,c \right )=\sigma(- \,\ c\cdot w)= \frac{1}{1+exp(c \cdot w)}
$$
But there are several context words:
$$
P\left ( +|w,c_{1:L} \right ) = \prod^L_{i=1}\sigma(c_i \cdot w)=\prod^L_{i=1}\frac{1}{1+exp(- \, c_i  \cdot w)}
$$

Classifier for word2vec

- A corpus of text is taken as input for learning the skip-gram embeddings by the learning algorithm.
- A vocabulary of size N is also chosen.
- For each word skip-gram algorithm learns two embeddings target embedding and context embedding.

Learning skip-gram embeddings


Other static embeddings available are
- fasttext
- GloVe

 Other kinds of static embeddings

The skip-gram model is one of the two primary architectures contained within the word2vec tool. It operates on the core assumption that a specific word can be utilized to generate its surrounding context words within a text sequence. By relying on conditional probabilities to predict these context words from a central word in an unlabeled text corpus, it functions as a self-supervised model to generate semantically meaningful, fixed-length word representations.

skip-gram

The continuous bag of words (CBOW) is one of the two core models that make up the word2vec tool. It operates on the foundational assumption that a center word is generated based on its surrounding context words. Functioning as a self-supervised model, it learns semantically meaningful word representations by utilizing conditional probabilities to predict a target word from its surrounding context in unlabeled corpora.

Continuous Bag of Words (CBOW)

The Penn Tree Bank (PTB) is a widely used corpus in natural language processing, sampled from Wall Street Journal articles. It is typically divided into training, validation, and test sets. When formatting the dataset for word embedding models, each line represents a sentence with words separated by spaces, allowing individual words to be extracted and processed as discrete tokens. Notably, the original dataset explicitly contains `<unk>` tokens to represent rare or unknown words.

Penn Tree Bank (PTB) Dataset

A fundamental preprocessing step in natural language processing involves constructing a vocabulary from a given text corpus. This procedure identifies the unique tokens present in the dataset and establishes a mapping for them. To limit the vocabulary size and manage data sparsity, words that occur less frequently than a specified minimum threshold are excluded from the primary vocabulary and are instead mapped to a common placeholder.

Building a Vocabulary

A context-independent representation of a token $$x$$ is formally defined as a function $$f(x)$$ that takes only $$x$$ as its input, completely ignoring the surrounding text. Consequently, models such as word2vec and GloVe assign a single, fixed pretrained vector to a word regardless of the context in which it appears. This approach has notable limitations when dealing with polysemy and complex semantics; for example, the word "crane" has entirely different meanings in the phrases "a crane is flying" and "a crane driver came", yet a context-independent model assigns the exact same mathematical representation to both instances.

Context-Independent Representation

The word2vec tool maps each word to a fixed-length vector to effectively express similarity and analogy relationships among different words. It comprises two distinct models: the skip-gram model and the continuous bag of words (CBOW) model. To learn semantically meaningful representations, word2vec relies on conditional probabilities, specifically predicting words using their surrounding context in a corpus. Because this supervision is extracted directly from the unlabeled data, word2vec acts as a self-supervised model.

Claude

University of Colorado at Boulder

Vectors semantics is the standard way to represent word meaning in NLP. The idea of vector semantics is to represent a word as a point in a multidimensional semantic space that is derived from the distributions of the term is sometimes more strictly applied only to dense vectors like word2vec, rather than sparse tf-idf or PPMI vectors. The word 'embedding' derives from its mathematical sense as a mapping form one space or structure to another, although the meaning has shifted.

Vector Semantics and Embeddings

An on-going but a helpful book resource about NLP
https://web.stanford.edu/~jurafsky/slp3/

Speech and Language Processing (3rd ed. draft) 

Dive into Deep Learning

The tf-idf algorithm is a method for weighting co-occurrence matrices in information retrieval, usually used when the dimensions are documents. The algorithm is the product of two terms:
- Term frequency is the frequency of the word $t$ in the document $d$, commonly denoted in log weighting as $tf_{t, d} = log_{10}(count(t, d) + 1)$.
-  Inverse document frequency is also usually denoted in a log function as $idf_t = log_{10}\left(\frac{N}{df_t}\right)$, where document frequency $df_t$ of a term $t$ is the number of documents in which it occurs and $N$ is the total number of documents in the collection.
The tf-idf weighted value $w_{t, d}$ for word $t$ in document $d$ is thus defined as 
$$w_{t, d} = tf_{t, d} \times idf_t$$

TF-IDF Algorithm

Raw frequency is not the best measure of association between words, as it is very skewed and not very discriminative. 
- We want to learn words that occur nearby frequently, such as "digital" and "information", as being more important than words that only appear once or twice.
- On the other hand, we also want to indicate words that are too frequent, such as "the" and "good", as unimportant. 

Common solutions are the tf-idf algorithm and the PPMI algorithm.

Problem with Raw Frequency

The amount of meaning difference between two words correspond to the amount of difference in their environments e.g. words which are synonyms (like oculist and eye-doctor) tended to occur in the same environment

Distributional Hypothesis

Pointwise Mutual Information (PMI) is a measure of how often two events occur, compared with what we would expect if they were independent.

An alternative weighting function to tf-idf, positive pointwise mutual information is used for term-term-matrices, when the vector dimensions correspond to words rather than documents.

$(PMI(w,c) =  \log _{2} (P(w,c)/P(w)P(c))$

$(PPMI(w,c) =  \max(log _{2} (P(w,c)/P(w)P(c)),0)$



Pointwise Mutual Information

The construction of efficient models of word meaning, in all vector sizes, are used to understand, apply, and improve current models by begin able to visualize acquired data better. To accomplish that; there are three different methods : GloVe (Global Vectors for Word Representation), Hierarchical Clustering , Downward Projection.

Visualizing Embeddings 

It is important to compare the vector model results with other models, as well as practicing intrinsic evaluations, which is to test its performance compared to gold-standard datasets, by computing the correlation between the word similarity scores of the model, and those scores from human judgements.

Evaluating Vector Models

Properties of embeddings refers: to window discrepancies, historical impact, and optimization of structure with word relation, depending on data being portrayed.

Learn Before

Related

Learn After