For each neuron in the sequence, the output $$\hat y^{<t>}$$ and the hidden state $$h^{<t>}$$ at time step $$t$$ can be calculated as follows. $$h^{<t>}=g_1(Wh^{<t-1>} \oplus Ux^{<t>}+b_h)$$ $$\hat y^{<t>}=g_2(Vh^{<t>}+b_y)$$ , where $$g_1, g_2$$ are activation functions, $$U, V, W$$ are trainable parameter matrices, $$b_h$$ and $$b_y$$ are bias terms, and $$\oplus$$ denotes element-wise addition. When the network is trained, it not only assigns weights $$U$$ to each neuron’s inputs, but also discovers the weight parameters $$W$$ of the hidden state function. These parameters define how much of the information from the previous steps should be carried forward to each subsequent step.

Simple RNN Forward Pass Equations

An unrolled Recurrent Neural Network (RNN) can be visualized as a sequence of identical network copies, with one copy for each input in the sequence (e.g., $$X_0, X_1, \dots, X_t$$).

At each time step $$t$$, the network processes the current input and generates two key components:
- **An output ($$\hat{y}^{(t)}$$):** The model's prediction at the current step.
- **A hidden state ($$h^{(t)}$$):** The network's short-term memory vector. It is computed using an activation function that takes both the previous hidden state ($$h^{(t-1)}$$) and the current input ($$X_t$$) as inputs, allowing the network to carry over information from previous steps.

Google

University of Michigan - Ann Arbor

The simplest form of RNN (“vanilla” RNN) is similar to a regular neural network, only it contains a loop that allows the model to carry forward results from previous neuron layers.

RNN Basic Structure

A helpful website that introduces neural networks:
https://missinglink.ai/guides/neural-network-concepts/

Learn Before

Related

Learn After