A course in the "Deep Learning Specialization"

University of Michigan - Ann Arbor

The forget gate is
$$\Gamma_f=\sigma(W_f[a^{<t-1>}, x^{<t>}]+b_f)$$
, where $\sigma$ denotes the sigmoid activation function, $W_f$ is weight, $b_f$ is a bias term, $a$ denotes the hidden state, $t$ is the t-th time/neuron, and $[a^{<t-1>}, x^{<t>}]$ means $a^{<t-1>}$ and $x^{<t>}$ are concatenated together.
Then compute the update gate in two steps. First, the update gate is
$$\Gamma_u=\sigma(W_u[a^{<t-1>}, x^{<t>}]+b_u)$$
Second, the intermediate cell state candidate is
$$\tilde c^{<t>}=tanh(W_c[a^{<t-1>}, x^{<t>}]+b_c)$$
, where $tanh$ denotes the tanh activation function.
Using the results from formulas above, we can calculate the current cell state,
$$c^{<t>}=\Gamma_u*\tilde c^{<t>}+\Gamma_f*c^{<t-1>}$$
At last, the third gate, output gate is
$$\Gamma_o=\sigma(W_o[a^{<t-1>}, x^{<t>}]+b_o)$$
, and using the output gate and the current cell state, we can compute the current hidden state
$$a^{<t>}=\Gamma_o*tanh(c^{<t>})$$

Math behind LSTMs

A blog explaining LSTM in detail.
https://colah.github.io/posts/2015-08-Understanding-LSTMs/

Learn Before

Related