The fastText model introduces a subword embedding approach to address the limitations of traditional word representations. Building upon the skip-gram architecture found in word2vec, fastText calculates the continuous vector for a center word by summing the individual vectors of its constituent subwords.

Claude

San Jose State University

Other static embeddings available are
- fasttext
- GloVe

 Other kinds of static embeddings

The skip-gram model is one of the two primary architectures contained within the word2vec tool. It operates on the core assumption that a specific word can be utilized to generate its surrounding context words within a text sequence. By relying on conditional probabilities to predict these context words from a central word in an unlabeled text corpus, it functions as a self-supervised model to generate semantically meaningful, fixed-length word representations.

skip-gram

Subword embedding is a text representation technique that breaks words down into smaller functional units. This approach can enhance the quality of vector representations, particularly for rare words and out-of-dictionary words that were not explicitly encountered during a model's initial training phase.

Subword Embedding

An on-going but a helpful book resource about NLP
https://web.stanford.edu/~jurafsky/slp3/

Speech and Language Processing (3rd ed. draft) 

Dive into Deep Learning

GloVe, short for Global Vectors, is an unsupervised learning model used to obtain dense vector representations for words. It captures global corpus statistics by modeling the ratios of probabilities from a global word-word co-occurrence matrix. By doing so, it incorporates both the advantages of global matrix factorization and the local context window methods (like word2vec) to effectively represent the linear substructures of the word vector space.

GloVe

fastText

An embedding layer operates by mapping a token's integer index $$i$$ directly to the $$i^\textrm{th}$$ row of its learnable weight matrix. The shape of this weight matrix is defined by the dictionary size as the number of rows and the vector dimension as the number of columns. When an embedding layer processes a minibatch of token indices, it retrieves the corresponding vector for each index, effectively appending the vector dimension to the input's shape. For instance, an input of shape $$(2, 3)$$ mapped to a vector dimension of $$4$$ will result in an output tensor of shape $$(2, 3, 4)$$.

Embedding Layer Mapping Token Indices to Vectors

In the forward propagation of the skip-gram model, the input consists of center word indices of shape $$(\text{batch size}, 1)$$ and concatenated context and noise word indices of shape $$(\text{batch size}, \text{max\_len})$$. These two sets of indices are first transformed into dense vectors via an embedding layer. Following this transformation, a batch matrix multiplication is performed between the embedded center words and the embedded context and noise words. This operation returns an output of shape $$(\text{batch size}, 1, \text{max\_len})$$, where each individual element represents the dot product between a center word vector and a context or noise word vector.

Learn Before

Related