Depth Size is important for both sparse and dense vectors. When vectors are computed with short context windows we collect a syntagmatic association (neighboring words) resulting in relational association, and when vectors are computed from long context windows we collect a Paradigmatic association (similar neighbors) resulting in topic word association.


Association depth

By using respective analogy solutions, a relational property association can be correlated for words. However, this property alone will not suffice on the growing, non-general, complexity of human cognitive processes.

Association by analogy.

By utilizing word2vectors you can correspond words to their associative relations based on their historical usage and correlational significance.

Historical Association

Semantic drift involves an erroneous pattern leading to the introduction of the erroneous tuples and the patterns based on these tuples leaning to the creation of more erroneous patterns in the database. 

Semantic Drift

Properties of embeddings refers: to window discrepancies, historical impact, and optimization of structure with word relation, depending on data being portrayed.

University of Texas at San Antonio

Vectors semantics is the standard way to represent word meaning in NLP. The idea of vector semantics is to represent a word as a point in a multidimensional semantic space that is derived from the distributions of the term is sometimes more strictly applied only to dense vectors like word2vec, rather than sparse tf-idf or PPMI vectors. The word 'embedding' derives from its mathematical sense as a mapping form one space or structure to another, although the meaning has shifted.

Vector Semantics and Embeddings

An on-going but a helpful book resource about NLP
https://web.stanford.edu/~jurafsky/slp3/

Speech and Language Processing (3rd ed. draft) 

The tf-idf algorithm is a method for weighting co-occurrence matrices in information retrieval, usually used when the dimensions are documents. The algorithm is the product of two terms:
- Term frequency is the frequency of the word $t$ in the document $d$, commonly denoted in log weighting as $tf_{t, d} = log_{10}(count(t, d) + 1)$.
-  Inverse document frequency is also usually denoted in a log function as $idf_t = log_{10}\left(\frac{N}{df_t}\right)$, where document frequency $df_t$ of a term $t$ is the number of documents in which it occurs and $N$ is the total number of documents in the collection.
The tf-idf weighted value $w_{t, d}$ for word $t$ in document $d$ is thus defined as 
$$w_{t, d} = tf_{t, d} \times idf_t$$

TF-IDF Algorithm

Raw frequency is not the best measure of association between words, as it is very skewed and not very discriminative. 
- We want to learn words that occur nearby frequently, such as "digital" and "information", as being more important than words that only appear once or twice.
- On the other hand, we also want to indicate words that are too frequent, such as "the" and "good", as unimportant. 

Common solutions are the tf-idf algorithm and the PPMI algorithm.

Problem with Raw Frequency

The amount of meaning difference between two words correspond to the amount of difference in their environments e.g. words which are synonyms (like oculist and eye-doctor) tended to occur in the same environment

Distributional Hypothesis

Pointwise Mutual Information (PMI) is a measure of how often two events occur, compared with what we would expect if they were independent.

An alternative weighting function to tf-idf, positive pointwise mutual information is used for term-term-matrices, when the vector dimensions correspond to words rather than documents.

$(PMI(w,c) =  \log _{2} (P(w,c)/P(w)P(c))$

$(PPMI(w,c) =  \max(log _{2} (P(w,c)/P(w)P(c)),0)$



Pointwise Mutual Information

Word2vec (also called Skip-gram) algorithm is the method to obtain short dense vectors of words. 

This method is self-supervision: training the model by just running the text without explicit labels for data.

Word2vec applies binary logistic regression classifier.


Word2vec

The construction of efficient models of word meaning, in all vector sizes, are used to understand, apply, and improve current models by begin able to visualize acquired data better. To accomplish that; there are three different methods : GloVe (Global Vectors for Word Representation), Hierarchical Clustering , Downward Projection.

Visualizing Embeddings 

It is important to compare the vector model results with other models, as well as practicing intrinsic evaluations, which is to test its performance compared to gold-standard datasets, by computing the correlation between the word similarity scores of the model, and those scores from human judgements.

Learn Before

Related

Learn After