It is not that different from a usual decoder. The only difference is that we store all the hidden layer outputs as we go over time to pass to the decoder

University of California, Berkeley

Attention is one of the most important innovations in deep learning for the last few years. The papers that introduced this mechanism also can consider the example of machine translation. Let’s quickly review the encoder-decoder architecture. We have an encoder and decoder parts. Encoder part runs an RNN through the input, and returns the final one context vector which we lately use during the decoding phase where we feed it to another RNN as initial hidden inputs. One big problem with that is when sentences get long the performance drops considerably, even though LSTM are supposed to keep the long term information. To fight with long sentences, researchers came up with the technique called attention. 
The attention mechanism tries to mimic our thinking because we are first focusing on different elements in the sentence or in the image before describing what is in there. In this case instead of only one vector benign passed to the decoder, we pass all the hidden layer vectors from each time stamp.


Attention Motivation

One of the main papers that introduces the attention mechanism 

Bahdanau, D., Cho, K., & Bengio, Y. (2014). Neural machine translation by jointly learning to align and translate. arXiv preprint arXiv:1409.0473.

Neural Machine Translation by Jointly Learning to Align and Translate

Also another main papers on attention mechanism
Luong, M. T., Pham, H., & Manning, C. D. (2015). Effective approaches to attention-based neural machine translation. arXiv preprint arXiv:1508.04025.

Effective Approaches to Attention-based Neural Machine Translation

Attention Encoder

The most upgrades are here. The RNN cell is left untouched but we use the encoder hidden states to generate words based on decoder hidden states. So the word we generate is a function of all encoder hidden states and current decoder state. But for inputs of different length, we are going to have different amounts of encoder hidden units. How do we fix the input dimension for that output function. One way is to add weight to each encoder hidden state based on its connection to the current decoder state and add them up. So for each hidden state we will define a score function:

Score( encoder hidden_state, decoder_hidden_state)

Encoder States: $h_{1} , h_{2}, h_{3}$  with  Decoder current state: $h'_{t}$

Scores: $ score(h_{1}, h'_{t}), score(h_{2}, h'_{t}), score(h_{3}, h'_{t}) $

Softmax: $s_{1} , s_{2}, s_{3}$ 

Context vector $ c_{t} = s_{1}  * h_{1} + s_{2} * h_{2} + s_{3} * h_{3} $

Over here you can see how the scoring function will work for each of the decoder states. As you can see here we score each encoder state and then apply a softmax function on it in order to normalize the scores and then we add them up. Then this context vector will be used further along with the decoder state to determine current word 


Learn Before

Related