The formula for the Nesterov algorithm is:

$step_{t} = \beta step_{t-1} + \alpha \nabla J(W^{t-1} - \beta step_{t-1})$

$W^{t} = W^{t-1} - step_{t}$

Where:

• $\nabla J(W^{t-1} - \beta step_{t-1})$ is the gradient calculated for the new point where $J$ is the cost function.
• $step_{t}$ is the step at time stamp $t$.
• $W^{t}$ are the parameters for the layer at the time stamp $t$.
• $\alpha$ is the learning rate.
• $\beta$ is another hyperparameter (mostly people use 0.9).

Also the same process is done for bias parameters.