Because for regular maximum likelihood estimation, it's too cunsuming to calculate the normalization factor. But without normalization factor, we can't calculate the conditional probability used in maximum likelihood estimation.

Why we need NCE?

The key idea of noise constractive estimation is to learn the feature of data distribution by learning from a dichotomy problem of selecting the data points from the original dataset from a mixed dataset made up of noises and the original datasets.

University of Michigan - Ann Arbor

1. LSA based method.
This kind of method is based on the intuition of SVD similar way. They try to do SVD on term-document matrices so that get the representation vectors.
2. local context based method.
This kind of method try to use the information gotten from the local context.
e.g.Word2Vec,Glove(Glove is actually a combination of those two methods.)

Algorithms used in word embedding

Multiple approaches have been used to deal with the cost of high-dimensional output layers over large vocabulary sets, including:
- Use of a Short Vocabulary List
- Hierarchical Softmax
- Importance Sampling
- Ranking Loss
- Noise-Contrastive Estimation

Solutions to the High Computational Cost of High-Dimensional Outputs

Goodfellow, I., Bengio, Y., & Courville, A. (2016). $\mathit{Deep \ Learning.}$ MIT Press. Retrieved from [www.deeplearningbook.org](https://www.deeplearningbook.org) 

Deep Learning

Word2vec is a technique for natural language processing. The word2vec algorithm uses a neural network model to learn word associations from a large corpus of text. Once trained, such a model can detect synonymous words or suggest additional words for a partial sentence.

Word2Vec

GloVe is an unsupervised learning algorithm for obtaining vector representations for words. Training is performed on aggregated global word-word co-occurrence statistics from a corpus, and the resulting representations showcase interesting linear substructures of the word vector space.

Glove

Elmo is a well-known word embedding model as well. It uses CNN to transfer words to raw word vectors. Then it uses bi-lstm and calculate the weighted average vectors of the raw word vectors and the two hidden layers to get the final word vectors. It's proposed as a pre-trained language model mainly to solve polysemy problem.

Elmo

https://en.wikipedia.org/wiki/Latent_semantic_analysis

Latent Semantic Analysis(LSA)-Reference

LSA is a well-known and widely used language model in natural language processing area.
LSA assumes that words that are close in meaning will occur in similar pieces of text (the distributional hypothesis). A matrix containing word counts per document (rows represent unique words and columns represent each document) is constructed from a large piece of text and a mathematical technique called singular value decomposition (SVD) is used to reduce the number of rows while preserving the similarity structure among columns. Documents are then compared by taking the cosine of the angle between the two vectors (or the dot product between the normalizations of the two vectors) formed by any two columns. Values close to 1 represent very similar documents while values close to 0 represent very dissimilar documents.
It's the basic method of Glove and many other modern methods of word embedding.

Latent Semantic Analysis

Noise Contractive Estimation

Ranking loss focuses more on relative distance instead of measuring the direct difference between the output with the expectation. 

Ranking Loss

Ensemble learning is a simple and effective technique for building a strong predictive model by combining the outputs of multiple, often weaker, individual models. This method aggregates the predictions from these constituent models to achieve a more robust and superior performance than any single model could attain on its own.

Ensemble Learning

BERT (Bidirectional Encoder Representations from Transformers) stands out as one of the most popular and extensively used pre-trained sequence encoding models in the field of Natural Language Processing. As a foundational model, it is trained using a self-supervised approach that combines two tasks: masked language modeling (MLM) and next sentence prediction (NSP). In MLM, the model predicts randomly masked words from their context, enabling it to learn deep bidirectional language representations. This dual-task training makes BERT a versatile foundation model adaptable to a wide array of NLP applications.

BERT (Bidirectional Encoder Representations from Transformers)

- Decomposing probabilities hierarchically, building nested categories of words in a tree-like structure over large vocabulary sets $\mathbb{V}$
- Instead of needing the amount of computations to be proportional $|\mathbb{V}|$, ( and to the number of hidden units $n_h$)
- In a balanced tree, the tree has a depth of $O(log|\mathbb{V}|)$
- The probability of choosing a word is equal to the product of the probabilities of choosing the branch leading to that word at every node on a path from the root of the tree to the leaf containing the word
- To predict the conditional probabilities that are needed for each node, often a logistic regression model is used, providing the same context $C$ as input. Since the correct output is encoded in the training set, you can use supervised learning to train the models, typically done using a cross-entropy loss corresponding to maximizing the log-likelihood of the correct sequence of decisions.
- Because the output log-likelihood can be efficiently computed, its gradients may also be efficiently computed (both with respect to the outer parameters as well as the hidden layer activations)
- It is often possible to optimize the tree structure to minimize the expected number of computations, however, it is usually not practical to do so, since the computation of outer probabilities is only one part of the total computation in the model.
- Instead of optimizing a tree with branching factor of 2, it is simpler to define a tree with a depth of 2 and a branching factor of $\sqrt{|\mathbb{V}|}$, i.e. defining a set of mutually exclusive word classes, capturing most of the benefit of the hierarchical strategy
- Question remains of how to best define the word classes, or the hierarchy in general. One could use discrete optimization to approximately optimize the partition of words
- Computing the probability of all $|\mathbb{V}|$ words will remain expensive, and the tree structure does not provide an exact solution to selecting the most likely word in a given context
- Produces computational benefits at both training time and test time, but tends to give worse results than sampling-based methods, potentially due to a poor choice of word classes.

Learn Before

Related

Learn After