It's a normal bug when we are feeding a RL algorithm. It indicates that forgetting to set penalty term for some wrong behavior or the penalty terms may be with too small absolute values.

University of Michigan - Ann Arbor

It's very important to know some common cases of abnormal behaviors due to improper reward setting. There are three types of them: rackless behavior, timid behavior and greedy behavior.  Reackless behavior will cause the agent ignoring some given serious side affect and leading to meaningless results especially in multi-task RL conditions. Timid Behavior will cause the agent stagnant. And greedy behavior will cause serious meaningless iterations and make the agent ignoring long-term reward. 

Abnormal Behavior Types Due to Improper Reward Setting

It's due to too many penalty term with high absolute value compared to the principal rewards.

Common Reason of Timid Behavior

Common Reason of Rackless Behavior

It always happens in RL with sparse reward conditions. It will be helpful to consider changing some tiny rewards to punishment penalty to force the agent to pursue the main reward.

Learn Before

Related