In model-based reinforcement learning, the model may be known or learned. In the latter case, we run a base policy, like a random or any educated policy, and observe the trajectory.

1. run base policy $\pi_0(s_t, a_t)$ to collect $D = \{ (s,a,s')_i \}$
2. learn dynamics model $f(s,a)$ to minimize $\sum_i ||f(s_i,a_i) - s_i' ||^2$
3. backpropagate through $f(s,a)$ into the policy to optimize $\pi_{\theta} (s_t,a_t)$
4. run $\pi_{\theta} (s_t,a_t)$ add the resulting data $\{ (x,u,x')_j \}$ to $D$
5. repeat from step 2

In step 2 above, we use supervised learning to train a model to minimize the least square error from the sampled trajectory

In step 3, we can use the model to predict the next state given an action, then we use the policy to decide the next action, and use the state and action to computer the cost. Finally, we backpropagate the cost to train the policy.

We continue sample and fit the model as we move along the path.