A deep RNN uses multiple neuron layers per time step. This is, in essence, a three-dimensional neural network which progresses from left to right, from right to left, and from bottom to top. It doesn't matter if the model performs forward propagation from left to right or from right to left, but it needs to propagate information through time first and then through layers. This architecture provides higher learning capacity, but also requires more training data.