The Positive Pointwise Mutual Information (PPMI) Algorithm is an algorithm for weighting term-term matrices.

Positive Pointwise Mutual Information (PPMI)

Pointwise Mutual Information (PMI) is a measure of how often two events occur compared to what would be expected if they were independent. In natural language processing, it is used as an alternative weighting function to tf-idf for term-term matrices, particularly when the vector dimensions correspond to words rather than documents.

The Pointwise Mutual Information between a word $$w$$ and a context word $$c$$ is defined as:
$$PMI(w,c) = \log_{2} \frac{P(w,c)}{P(w)P(c)}$$

State University of New York at Stony Brook

University of Michigan - Ann Arbor

Google

Vectors semantics is the standard way to represent word meaning in NLP. The idea of vector semantics is to represent a word as a point in a multidimensional semantic space that is derived from the distributions of the term is sometimes more strictly applied only to dense vectors like word2vec, rather than sparse tf-idf or PPMI vectors. The word 'embedding' derives from its mathematical sense as a mapping form one space or structure to another, although the meaning has shifted.

Vector Semantics and Embeddings

An on-going but a helpful book resource about NLP
https://web.stanford.edu/~jurafsky/slp3/

Speech and Language Processing (3rd ed. draft) 

Raw frequency is not the best measure of association between words, as it is very skewed and not very discriminative. 
- We want to learn words that occur nearby frequently, such as "digital" and "information", as being more important than words that only appear once or twice.
- On the other hand, we also want to indicate words that are too frequent, such as "the" and "good", as unimportant. 

Common solutions are the tf-idf algorithm and the PPMI algorithm.

Problem with Raw Frequency

The amount of meaning difference between two words correspond to the amount of difference in their environments e.g. words which are synonyms (like oculist and eye-doctor) tended to occur in the same environment

Distributional Hypothesis

The construction of efficient models of word meaning, in all vector sizes, are used to understand, apply, and improve current models by begin able to visualize acquired data better. To accomplish that; there are three different methods : GloVe (Global Vectors for Word Representation), Hierarchical Clustering , Downward Projection.

Visualizing Embeddings 

It is important to compare the vector model results with other models, as well as practicing intrinsic evaluations, which is to test its performance compared to gold-standard datasets, by computing the correlation between the word similarity scores of the model, and those scores from human judgements.

Evaluating Vector Models

Properties of embeddings refers: to window discrepancies, historical impact, and optimization of structure with word relation, depending on data being portrayed.

Semantic Properties

The word2vec tool maps each word to a fixed-length vector to effectively express similarity and analogy relationships among different words. It comprises two distinct models: the skip-gram model and the continuous bag of words (CBOW) model. To learn semantically meaningful representations, word2vec relies on conditional probabilities, specifically predicting words using their surrounding context in a corpus. Because this supervision is extracted directly from the unlabeled data, word2vec acts as a self-supervised model.

word2vec

The tf-idf algorithm is a method for weighting co-occurrence matrices in information retrieval, usually used when the dimensions are documents. The algorithm is the product of two terms: - Term frequency is the frequency of the word $$t$$ in the document $$d$$, commonly denoted in log weighting as $$tf_{t, d} = log_{10}(count(t, d) + 1)$$. - Inverse document frequency is also usually denoted in a log function as $$idf_t = log_{10}\left(\frac{N}{df_t}\right)$$, where document frequency $$df_t$$ of a term $$t$$ is the number of documents in which it occurs and $$N$$ is the total number of documents in the collection. The tf-idf weighted value $$w_{t, d}$$ for word $$t$$ in document $$d$$ is thus defined as $$w_{t, d} = tf_{t, d} \times idf_t$$

Learn Before

Related

Learn After