Training bidirectional Recurrent Neural Networks (RNNs) is a computationally expensive process. This high cost is primarily due to the very long gradient chains that form during backpropagation, as the network must propagate errors backwards through time across both the forward and reverse sequences simultaneously.

Claude

A bidirectional RNN assumes that the correct output not only depends on the previous inputs in the time series but also on future inputs. For example, in translation models, it is often necessary to use a word from the end of the source sentence in order to predict a word early in the target sentence. 

To make this possible, two RNNs are stacked on top of each other, one going from beginning to end and the other from end to beginning, and the output is computed based on hidden states of both networks.

Bidirectional RNNs

Dive into Deep Learning

A deep RNN uses multiple neuron layers per time step. This is, in essence, a three-dimensional neural network which progresses from left to right, from right to left, and from bottom to top. It doesn't matter if the model performs forward propagation from left to right or from right to left, but it needs to propagate information through time first and then through layers. This architecture provides higher learning capacity, but also requires more training data.  

Deep (Bidirectional) RNNs

An ordinary RNN is useful because the model takes in information about your output at a previous time step to predict your current and future outputs. Similarly, a Bidirectional RNN depends upon both outputs at previous and FUTURE outputs. This is useful in applications where your predictions in your model not only depend on the past, but the future. This would include things like handwriting recognition, speech recognition, and bioinformatics. 

Motivation for Bidirectional RNNs

The concept of using more than one RNN that have some type of relationship between the hidden units is expandable to any number of dimensions of inputs you may have. For example, in our 1 dimensional example of "Bidirectional RNNs", the proposal was to use two networks with hidden units feeding into each other (one net for the past and one for the future).

This concept can be used with any $n$ dimensional input model. In image recognition, four recursive neural networks can be used, two for the vertical axis and two for the horizontal axis.

Implementation of Bidirectional RNNs

Bidirectional Recurrent Neural Networks (RNNs) are primarily advantageous for sequence encoding and for estimating observations when a complete bidirectional context is available. By utilizing information from both the past and the future at each time step, these networks effectively capture the full semantic context of a sequence before making downstream predictions.

Learn Before

Related