A course in the "Deep Learning Specialization"

University of Michigan - Ann Arbor

The forget gate is $$\Gamma_f=\sigma(W_f[a^{<t-1>}, x^{<t>}]+b_f)$$, where $$\sigma$$ denotes the sigmoid activation function, $$W_f$$ is the weight matrix, $$b_f$$ is a bias term, $$a^{<t-1>}$$ denotes the hidden state from the previous time step, and $$x^{<t>}$$ is the input at the $$t$$-th time step. The notation $$[a^{<t-1>}, x^{<t>}]$$ means that $$a^{<t-1>}$$ and $$x^{<t>}$$ are concatenated. Then, compute the update gate in two steps. First, the update gate is $$\Gamma_u=\sigma(W_u[a^{<t-1>}, x^{<t>}]+b_u)$$. Second, the intermediate cell state candidate is $$\tilde{c}^{<t>}=\tanh(W_c[a^{<t-1>}, x^{<t>}]+b_c)$$, where $$\tanh$$ denotes the hyperbolic tangent activation function. Using the results from the formulas above, we can calculate the current cell state as $$c^{<t>}=\Gamma_u*\tilde{c}^{<t>}+\Gamma_f*c^{<t-1>}$$. Finally, the third gate, the output gate, is $$\Gamma_o=\sigma(W_o[a^{<t-1>}, x^{<t>}]+b_o)$$, and using the output gate and the current cell state, we can compute the current hidden state as $$a^{<t>}=\Gamma_o*\tanh(c^{<t>})$$.

LSTM Forward Pass Equations

A blog explaining LSTM in detail.
https://colah.github.io/posts/2015-08-Understanding-LSTMs/

Learn Before

Related

Related