The forget gate is
$$\Gamma_f=\sigma(W_f[a^{<t-1>}, x^{<t>}]+b_f)$$
, where $\sigma$ denotes the sigmoid activation function, $W_f$ is weight, $b_f$ is a bias term, $a$ denotes the hidden state, $t$ is the t-th time/neuron, and $[a^{<t-1>}, x^{<t>}]$ means $a^{<t-1>}$ and $x^{<t>}$ are concatenated together.
Then compute the update gate in two steps. First, the update gate is
$$\Gamma_u=\sigma(W_u[a^{<t-1>}, x^{<t>}]+b_u)$$
Second, the intermediate cell state candidate is
$$\tilde c^{<t>}=tanh(W_c[a^{<t-1>}, x^{<t>}]+b_c)$$
, where $tanh$ denotes the tanh activation function.
Using the results from formulas above, we can calculate the current cell state,
$$c^{<t>}=\Gamma_u*\tilde c^{<t>}+\Gamma_f*c^{<t-1>}$$
At last, the third gate, output gate is
$$\Gamma_o=\sigma(W_o[a^{<t-1>}, x^{<t>}]+b_o)$$
, and using the output gate and the current cell state, we can compute the current hidden state
$$a^{<t>}=\Gamma_o*tanh(c^{<t>})$$

Math behind LSTMs

In a simple word,  word embeddings are numerical representations of texts.
In detail, when computer treat differents words, they will turn them into some atomic symbols. e.g. cat might be a, dogs might be b. But how can computer find the relationship between them? That's what word embeddings are. They willl build some relationships between words.

Word Embeddings

https://www.bioinf.jku.at/publications/older/2604.pdf

Long Short-Term Memory (LSTM) Paper

Gates are made use of by LSTM units to control the flow of information into and out of the units that comprise the network layers.
- The Forget Gate deletes information from the context that is no longer needed
- The Add Gate selects the information to add to the current context
- The Output Gate decides what information is required for the current hidden state

Gates in LSTMs

Given the appropriate weights for the various gates, an LSTM accepts as input the context layer, and hidden layer from the previous time step, along with the current input vector.

The cell then generates updated context and hidden vectors as output. The hidden layer can be used as input to subsequent layers in a stacked RNN, or to generate an output for the final layer of a network.

Inputs and Outputs of LSTM Cells

The neural units used in LSTMs are obviously much more complex than those used in basic feedforward networks. Fortunately, this complexity is encapsulated within the basic processing units, allowing us to maintain modularity and easily experiment with different architectures. The figure below demonstrates this by illustrating the inputs and outputs associated with each kind of unit.

The increased complexity of the LSTM units is encapsulated within the unit itself. The only additional external complexity for the LSTM over the basic recurrent unit is the presence of the additional context vector as an input and output.

The Encapsulated Complexity of LSTMs

An LSTM cell allows the network to pick up such pertinent information and save it, injecting it back into the model when necessary.

While a basic RNN has only a simple activation function, an LSTM cell has four. Three sigmoid activation functions output numbers between 0 and 1, and lead to a pointwise multiplication gate. This gate determines whether information should enter the cell or not—0 means no information enters, and 1 means all the information enters. These three cells are used to save pertinent information for later stages of the learning process. 

In the training process, the network learns what are the optimal values for the gates, or how much of the information should be retained to help the network make the most accurate prediction.

University of Michigan - Ann Arbor

The long short-term memory (LSTM) network is an extension to RNNs, in addressing problems such as the vanishing gradient problem and the inability of RNNs to carry forward critical information.

LSTMs divide the context management problem into two sub-problems: 
- Removing information no longer needed from the context
- Adding information likely to be needed for later decision making

Long Short-Term Memory (LSTM) Network

A helpful website that introduces neural networks:
https://missinglink.ai/guides/neural-network-concepts/

Neural Network Reference

LSTM Cell

LSTMs can be applied to a variety of deep learning tasks that mostly include prediction based on previous information. Three noteworthy examples include:
 - Text prediction:
The long-term memory capabilities of LSTM means it excels at predicting text sequences. In order to predict the next word in a sentence, the network has to retain all the words that preceded it. One of the most common applications of text prediction is in chatbots used by eCommerce sites.
 - Stock prediction:
Simple Machine Learning (SML) models are able to predict stock values and prices based on inputs such as the opening value and the volume of the stock. While these values do take part in stock prediction, they lack a key component. To properly predict a stock value with high accuracy, the model needs to take into account one of the biggest factors—the trend of the stock. To do so, the model needs to identify the trend based on the values recorded over the preceding days—a task suited to an LSTM network.
- Music composition: LSTM can be applied considering that music is built using long sequences of notes, much like text uses long sequences of words.

Applictaions of Long Short-Term Memory Networks (LSTMs)

In the late 1990s, German researchers Hochreiter and Schmidhuber proposed the concept of Long Short-Term Memory (LSTM), which helps Recurrent Neural Networks (RNNs) retain information over extended sequences, rather than just between consecutive time steps. For example, a character’s name introduced at the beginning of a paragraph may be necessary to understand the context at the end. An LSTM-based RNN shares the same high-level architecture as a basic RNN (whether simple, bidirectional, or deep), but replaces standard activation functions with specialized LSTM cells.

Learn Before

Related

Learn After