The concept of using multiple recurrent neural networks (RNNs) with relationships between their hidden units can be expanded to input models of any dimensionality. While standard bidirectional RNNs use two networks for one-dimensional sequences (one for the past and one for the future), this approach can be extended to $$n$$ dimensions. For example, in two-dimensional image recognition, four recurrent neural networks can be used: two for the vertical axis and two for the horizontal axis.

University of Connecticut

Google

A bidirectional RNN assumes that the correct output not only depends on the previous inputs in the time series but also on future inputs. For example, in translation models, it is often necessary to use a word from the end of the source sentence in order to predict a word early in the target sentence. 

To make this possible, two RNNs are stacked on top of each other, one going from beginning to end and the other from end to beginning, and the output is computed based on hidden states of both networks.

Bidirectional RNNs

Goodfellow, I., Bengio, Y., & Courville, A. (2016). $\mathit{Deep \ Learning.}$ MIT Press. Retrieved from [www.deeplearningbook.org](https://www.deeplearningbook.org) 

Deep Learning

A deep RNN uses multiple neuron layers per time step. This is, in essence, a three-dimensional neural network which progresses from left to right, from right to left, and from bottom to top. It doesn't matter if the model performs forward propagation from left to right or from right to left, but it needs to propagate information through time first and then through layers. This architecture provides higher learning capacity, but also requires more training data.  

Deep (Bidirectional) RNNs

An ordinary RNN is useful because the model takes in information about your output at a previous time step to predict your current and future outputs. Similarly, a Bidirectional RNN depends upon both outputs at previous and FUTURE outputs. This is useful in applications where your predictions in your model not only depend on the past, but the future. This would include things like handwriting recognition, speech recognition, and bioinformatics. 

Motivation for Bidirectional RNNs

Bidirectional Recurrent Neural Networks (RNNs) are primarily advantageous for sequence encoding and for estimating observations when a complete bidirectional context is available. By utilizing information from both the past and the future at each time step, these networks effectively capture the full semantic context of a sequence before making downstream predictions.

Bidirectional RNN Sequence Encoding

Training bidirectional Recurrent Neural Networks (RNNs) is a computationally expensive process. This high cost is primarily due to the very long gradient chains that form during backpropagation, as the network must propagate errors backwards through time across both the forward and reverse sequences simultaneously.

Learn Before

Related