The combination of on-policy and off-policy strategies is adjusted by adjusting the weights in their loss function. According to their experiments, they claim several points. First, the usage of the off-policy value target provides a clear benefit. Second, in an environment with intermediate rewards, the off-policy value target might not provide any new information. But they are leading the model to find and quickly overfit to a
sub-optimal strategy. And as related weights decay, they will find optimal strategy finally and lead to a faster convergence.

University of Michigan - Ann Arbor

Experiments on three different conditions with different policies are proposed by this paper. In environments with sparse rewards, runs that use the off-policy value target with decay are able to have faster initial convergence and achieve higher rewards than MuZero. An environment with intermediate rewards, these runs were able to provide a faster initial convergence than MuZero, but stagnated to lower values. The combined algorithm converges faster than MuZero and no longer needs to assume that the environment is reversible.

Experimental Evaluation of combining on and off-policy steategies

Borges, Alexandre and Arlindo L. Oliveira. “Combining Off and On-Policy Training in Model-Based Reinforcement Learning.” ArXiv abs/2102.12194 (2021): n. pag.
https://arxiv.org/pdf/2102.12194.pdf

Combining Off and On-Policy Training in Model-Based
Reinforcement Learning

Across all environments, provides faster initial convergence speeds. However, if we compare M0FFV runs with M0GB runs, the paper finds that using solely this target is not enough, and needs to be combined with the on-policy value target.

Off-Policy Value Target $\gamma$

Only provided benefits in Cartpole. Runs had fast initial convergence, but quickly stagnated to values lower than MuZero, and thus it seemingly does not improve training and impairs convergence. More dependent on the number of simulations than the value target, however, based on h on the M0ALL training curve, the target seems to be useful at the beginning.

Learn Before

Related