Learn Before
Optimization Techniques in Word2Vec
Training a standard Word2Vec model on a large vocabulary is computationally expensive because the naive approach requires calculating and normalizing probabilities across all words in the output layer for every training sample. To significantly reduce this computational cost, two primary optimization techniques are utilized:
-
Hierarchical Softmax: Replaces the flat output layer with a Huffman tree-based model, reducing the time complexity of the output probability calculation and weight updates. It evaluates a path of binary decisions instead of calculating probabilities for every word.
-
Negative Sampling: Instead of updating all output vectors, this approach updates only the target word and a small, randomly selected sample of incorrect, or "negative", words. The probability of selecting a word as a negative sample is positively correlated with its frequency in the corpus, which also helps mitigate the influence of overly common words (e.g., "the").
0
2
Contributors are:
Who are from:
Tags
Data Science