You have trained word embeddings using a text dataset of $$m_1$$ words. You are considering using these word embeddings for a language task, for which you have a separate labeled dataset of $$m_2$$ words. Keeping in mind that using word embeddings is a form of transfer learning, under which of these circumstance would you expect the word embeddings to be helpful?

University of Michigan - Ann Arbor

Sequence models are statistical supervised learning models that predict/classify based on sequence data.

Similar to other supervised statistical models, sequence models are in two types:
- Sequence prediction
- Sequence classification

Sequence Models

https://machinelearningmastery.com/sequence-prediction/

Predictions with Sequences

Sequence prediction is used to predict the next value of a given input sequence. One example is product recommendations, which predicts a customer's next purchase given a sequence of their past purchases. Another example is stock market prediction.

Sequence Prediction Models

Sequence classification predicts a class label for a given input sequence. Sentiment analysis is one example, which takes a sequence of text and predicts the label for the sentiment (positive or negative). 

Sequence Classification Models

Recurrent neural networks, or RNNs, are a family of neural networks for processing sequential data. A recurrent neural network is specialized for processing a sequence of inputs $x^{(1)}, . . . , x^{(τ)}$ and each time add additional layers of comprehension on top of the previous inputs. 


Recurrent Neural Network (RNN)

Sequence Model Question #1

In the word2vec algorithm, you estimate $$P\left(t|c\right)$$, where t is the target word and c is a context word. How are t and c chosen from the training set?

Sequence Model Question #2

Suppose you learn a word embedding for a vocabulary of 10000 words. Then the embedding vectors should be 10000 dimensional, so as to capture the full range of variation and meaning in those words.

Sequence Moel Question #4

In beam search, if you increase the beam width B, which of the following would you expect to be true? Check all that apply.

Sequence Model Question #3

Tokenization is the process of converting a sequence of text into smaller units, known as tokens. It is a foundational step in Natural Language Processing, and there are numerous different methods and strategies for how a text can be tokenized.

Tokenization

In sequence modeling architectures, standard notation is used to represent the inputs and outputs. A source sequence is typically denoted as $$x_1...x_m$$, where $$m$$ represents the length of the source sequence. Its corresponding sequence of embedding vectors is denoted as $$\mathbf{e}_{1}^{x}...\mathbf{e}_{m}^{x}$$. Similarly, the target sequence that the model aims to generate is denoted as $$y_1...y_n$$, with length $$n$$, and its corresponding embedding sequence is represented as $$\mathbf{e}_{1}^{y}...\mathbf{e}_{n}^{y}$$.

Learn Before

Related