Raw frequency is not the best measure of association between words, as it is very skewed and not very discriminative. 
- We want to learn words that occur nearby frequently, such as "digital" and "information", as being more important than words that only appear once or twice.
- On the other hand, we also want to indicate words that are too frequent, such as "the" and "good", as unimportant. 

Common solutions are the tf-idf algorithm and the PPMI algorithm.

University of Michigan - Ann Arbor

Vectors semantics is the standard way to represent word meaning in NLP. The idea of vector semantics is to represent a word as a point in a multidimensional semantic space that is derived from the distributions of the term is sometimes more strictly applied only to dense vectors like word2vec, rather than sparse tf-idf or PPMI vectors. The word 'embedding' derives from its mathematical sense as a mapping form one space or structure to another, although the meaning has shifted.

Vector Semantics and Embeddings

An on-going but a helpful book resource about NLP
https://web.stanford.edu/~jurafsky/slp3/

Speech and Language Processing (3rd ed. draft) 

The tf-idf algorithm is a method for weighting co-occurrence matrices in information retrieval, usually used when the dimensions are documents. The algorithm is the product of two terms:
- Term frequency is the frequency of the word $t$ in the document $d$, commonly denoted in log weighting as $tf_{t, d} = log_{10}(count(t, d) + 1)$.
-  Inverse document frequency is also usually denoted in a log function as $idf_t = log_{10}\left(\frac{N}{df_t}\right)$, where document frequency $df_t$ of a term $t$ is the number of documents in which it occurs and $N$ is the total number of documents in the collection.
The tf-idf weighted value $w_{t, d}$ for word $t$ in document $d$ is thus defined as 
$$w_{t, d} = tf_{t, d} \times idf_t$$

TF-IDF Algorithm

Problem with Raw Frequency

The amount of meaning difference between two words correspond to the amount of difference in their environments e.g. words which are synonyms (like oculist and eye-doctor) tended to occur in the same environment

Distributional Hypothesis

Pointwise Mutual Information (PMI) is a measure of how often two events occur, compared with what we would expect if they were independent.

An alternative weighting function to tf-idf, positive pointwise mutual information is used for term-term-matrices, when the vector dimensions correspond to words rather than documents.

$(PMI(w,c) =  \log _{2} (P(w,c)/P(w)P(c))$

$(PPMI(w,c) =  \max(log _{2} (P(w,c)/P(w)P(c)),0)$



Pointwise Mutual Information

Word2vec (also called Skip-gram) algorithm is the method to obtain short dense vectors of words. 

This method is self-supervision: training the model by just running the text without explicit labels for data.

Word2vec applies binary logistic regression classifier.


Word2vec

The construction of efficient models of word meaning, in all vector sizes, are used to understand, apply, and improve current models by begin able to visualize acquired data better. To accomplish that; there are three different methods : GloVe (Global Vectors for Word Representation), Hierarchical Clustering , Downward Projection.

Visualizing Embeddings 

It is important to compare the vector model results with other models, as well as practicing intrinsic evaluations, which is to test its performance compared to gold-standard datasets, by computing the correlation between the word similarity scores of the model, and those scores from human judgements.

Evaluating Vector Models

Properties of embeddings refers: to window discrepancies, historical impact, and optimization of structure with word relation, depending on data being portrayed.

Learn Before

Related