1- Learn word embeddings from large text corpus.

2- Transfer embedding to new task with smaller training set.

3- Optional: Continue to fine tune the word embedding with new data.

Transfer Learning and Word Embedding

When there are many data in the learning step, and less data in the new task, such as:

1- Name entity recognition

2- Text summarization



But not:
Language Modeling, Machine Translation,...


Word embedding Applications

1. LSA based method.
This kind of method is based on the intuition of SVD similar way. They try to do SVD on term-document matrices so that get the representation vectors.
2. local context based method.
This kind of method try to use the information gotten from the local context.
e.g.Word2Vec,Glove(Glove is actually a combination of those two methods.)

Algorithms used in word embedding

When learning word embeddings, we create an artificial task of estimating 
P(target/context). It is okay if we do poorly on this artificial prediction task; the more important by-product of this task is that we learn a useful set of word embeddings. 

 word embeddings

In a simple word,  word embeddings are numerical representations of texts.
In detail, when computer treat differents words, they will turn them into some atomic symbols. e.g. cat might be a, dogs might be b. But how can computer find the relationship between them? That's what word embeddings are. They willl build some relationships between words.

Word Embeddings

t-Distributed Stochastic Neighbor Embedding (t-SNE) is a non-linear dimensionality reduction technique used to visualize high-dimensional data. In the context of word embeddings, it maps high-dimensional feature vectors into a 2D or 3D space, ensuring that semantically similar words are positioned in close proximity for visualization.

t-SNE Algorithm

Instead of one-hot encoding vector, we use featured representation to capture relationship between words.

In simple words, what are extracted features for each word? For example, in the following matrix "Man" and "Woman" have high score for "Gender" feature and low score for "Food" feature. As a result, the computer can capture their relation.



University of Michigan - Ann Arbor

Each word is represented as a vector, with value 1 for the word and 0 for all other words in the dictionary (similar to one-hot encoding)

Representing  Words

https://www.coursera.org/learn/nlp-sequence-models

Learn Before

Related

Learn After