https://www.jmlr.org/papers/volume3/bengio03a/bengio03a.pdf

A Neural Probabilistic Language Model Paper

Man is to Computer Programmer as Woman is to Homemaker? Debiasing Word Embeddings Paper

In addition to their ability to learn word meaning from text, embeddings, alas, also reproduce the implicit biases and stereotypes that were latent in the text. As the prior section just showed, embeddings can roughly model relational similarity: ‘queen’ as the closest word to ‘king’ -‘man’ + ‘woman’ implies the analogy man:woman::king:queen.

Bias in Word Embeddings

In a simple word,  word embeddings are numerical representations of texts.
In detail, when computer treat differents words, they will turn them into some atomic symbols. e.g. cat might be a, dogs might be b. But how can computer find the relationship between them? That's what word embeddings are. They willl build some relationships between words.

University of Michigan - Ann Arbor

An LSTM cell allows the network to pick up such pertinent information and save it, injecting it back into the model when necessary.

While a basic RNN has only a simple activation function, an LSTM cell has four. Three sigmoid activation functions output numbers between 0 and 1, and lead to a pointwise multiplication gate. This gate determines whether information should enter the cell or not—0 means no information enters, and 1 means all the information enters. These three cells are used to save pertinent information for later stages of the learning process. 

In the training process, the network learns what are the optimal values for the gates, or how much of the information should be retained to help the network make the most accurate prediction.

LSTM Cell

Instead of one-hot encoding vector, we use featured representation to capture relationship between words.

In simple words, what are extracted features for each word? For example, in the following matrix "Man" and "Woman" have high score for "Gender" feature and low score for "Food" feature. As a result, the computer can capture their relation.



Featured representation: Word Embedding

Medium

The forget gate is
$$\Gamma_f=\sigma(W_f[a^{<t-1>}, x^{<t>}]+b_f)$$
, where $\sigma$ denotes the sigmoid activation function, $W_f$ is weight, $b_f$ is a bias term, $a$ denotes the hidden state, $t$ is the t-th time/neuron, and $[a^{<t-1>}, x^{<t>}]$ means $a^{<t-1>}$ and $x^{<t>}$ are concatenated together.
Then compute the update gate in two steps. First, the update gate is
$$\Gamma_u=\sigma(W_u[a^{<t-1>}, x^{<t>}]+b_u)$$
Second, the intermediate cell state candidate is
$$\tilde c^{<t>}=tanh(W_c[a^{<t-1>}, x^{<t>}]+b_c)$$
, where $tanh$ denotes the tanh activation function.
Using the results from formulas above, we can calculate the current cell state,
$$c^{<t>}=\Gamma_u*\tilde c^{<t>}+\Gamma_f*c^{<t-1>}$$
At last, the third gate, output gate is
$$\Gamma_o=\sigma(W_o[a^{<t-1>}, x^{<t>}]+b_o)$$
, and using the output gate and the current cell state, we can compute the current hidden state
$$a^{<t>}=\Gamma_o*tanh(c^{<t>})$$

Math behind LSTMs

Word Embeddings

https://www.bioinf.jku.at/publications/older/2604.pdf

Long Short-Term Memory (LSTM) Paper

Gates are made use of by LSTM units to control the flow of information into and out of the units that comprise the network layers.
- The Forget Gate deletes information from the context that is no longer needed
- The Add Gate selects the information to add to the current context
- The Output Gate decides what information is required for the current hidden state

Gates in LSTMs

Given the appropriate weights for the various gates, an LSTM cell accepts three inputs at each time step: the memory cell internal state (context vector) and the hidden state from the previous time step, along with the current input vector. The cell then produces updated memory cell internal state and hidden state vectors as output. Crucially, only the hidden state is passed into the output layer or forwarded to subsequent layers in a stacked RNN, while the memory cell internal state remains entirely internal to the LSTM unit and is never exposed outside it.

Inputs and Outputs of LSTM Cells

The neural units used in LSTMs are obviously much more complex than those used in basic feedforward networks. Fortunately, this complexity is encapsulated within the basic processing units, allowing us to maintain modularity and easily experiment with different architectures. The figure below demonstrates this by illustrating the inputs and outputs associated with each kind of unit.

The increased complexity of the LSTM units is encapsulated within the unit itself. The only additional external complexity for the LSTM over the basic recurrent unit is the presence of the additional context vector as an input and output.

The Encapsulated Complexity of LSTMs

When implementing recurrent architectures like Long Short-Term Memory (LSTM) networks in JAX, using a standard Python for-loop to iterate over sequence time steps causes extremely long Just-In-Time (JIT) compilation times on the first run. To bypass this performance bottleneck, JAX provides the jax.lax.scan utility transformation. It accepts an initial carry state and an input array, efficiently scanning across the leading axis of the inputs to update the state and yield stacked outputs.

JAX lax.scan in RNN Iteration

When implementing a Long Short-Term Memory (LSTM) network, processing a new sequence without a prior state requires the explicit initialization of two distinct state variables. Both the hidden state ($$H$$) and the memory cell internal state ($$C$$) must be initialized, typically as zero tensors with a shape dictated by the batch size and the number of hidden units. This dual-state initialization provides a neutral starting point for the gating mechanisms and the input node before the first sequence token is processed.

LSTM Initial States Initialization

t-Distributed Stochastic Neighbor Embedding (t-SNE) is a non-linear technique for dimensionality reduction that is particularly well suited for the visualization of high-dimensional datasets.

Converting featured vector algorithm from the word embedding featured vector gives us a 2D visualization showing related words close to each other.

T-SNE algorithm

1- Learn word embeddings from large text corpus.

2- Transfer embedding to new task with smaller training set.

3- Optional: Continue to fine tune the word embedding with new data.

Transfer Learning and Word Embedding

When there are many data in the learning step, and less data in the new task, such as:

1- Name entity recognition

2- Text summarization



But not:
Language Modeling, Machine Translation,...


Word embedding Applications

1. LSA based method.
This kind of method is based on the intuition of SVD similar way. They try to do SVD on term-document matrices so that get the representation vectors.
2. local context based method.
This kind of method try to use the information gotten from the local context.
e.g.Word2Vec,Glove(Glove is actually a combination of those two methods.)

Algorithms used in word embedding

When learning word embeddings, we create an artificial task of estimating 
P(target/context). It is okay if we do poorly on this artificial prediction task; the more important by-product of this task is that we learn a useful set of word embeddings. 

Learn Before

Related

Learn After