The reset/relevance gate is
$$\Gamma_r=\sigma(W_r[h^{<t-1>}, x^{<t>}]+b_r)$$
, where $\sigma$ denotes the sigmoid activation function, $W$ is parameter matrix, $b$ is a bias term, $t$ is the t-th time/neuron, and $[h^{<t-1>}, x^{<t>}]$ means $h^{<t-1>}$ and $x^{<t>}$ are concatenated together.
The update gate is 
$$\Gamma_u=\sigma(W_u[h^{<t-1>}, x^{<t>}]+b_u)$$
Then the intermediate hidden state candidate is
$$h^{'<t>}=tanh(W_h[\Gamma_r*h^{<t-1>}, x^{<t>}]+b_h)$$
Then the current hidden state is 
$$h^{<t>}=(1-\Gamma_u)*h^{<t-1>}+\Gamma_u*h^{'<t>}$$
, and the current output is
$$\hat y^{<t>}=g(W_y*h^{<t>}+b_y)$$
, where $g$ is an activation function.

University of Michigan - Ann Arbor

Gated Recurrent Units (GRUs) resolve the vanishing gradients problem in simple RNNs and also ease the burden of introducing a considerable number of additional parameters in the LSTMs, by dispensing with the use of a separate context vector, and by reducing the number of gates to 2 — a reset/relevance gate and an update gate.
The purpose of the reset gate is to decide which aspects of the previous hidden state are relevant to the current context and what can be ignored. It computes an intermediate representation for the new hidden state at the current time.
The purpose of the update gate is to determine which aspects of this new state will be used directly in the new hidden state and which aspects of the previous state need to be preserved for future use.

Gated recurrent unit (GRU)

An on-going but a helpful book resource about NLP
https://web.stanford.edu/~jurafsky/slp3/

Learn Before

Related