It's due to too many penalty term with high absolute value compared to the principal rewards.

Common Reason of Timid Behavior

It always happens in RL with sparse reward conditions. It will be helpful to consider changing some tiny rewards to punishment penalty to force the agent to pursue the main reward.

Common Reason of Greedy Behavior

In reinforcement learning (RL), reckless behavior typically occurs because penalty terms for undesirable actions are either omitted from the reward function or assigned absolute values that are too small to deter the agent.

Common Reason of Reckless Behavior

Reckless behavior in reinforcement learning (RL) is an abnormal agent behavior where the agent ignores critical negative side effects or constraints in order to maximize its reward. This behavior often leads to meaningless or unsafe outcomes, particularly in multi-task RL environments.

Reckless Behavior in Reinforcement Learning

Timid behavior in reinforcement learning is an abnormal pattern in which an agent becomes stagnant because penalties are too numerous or too large relative to the principal rewards.

Timid Behavior in Reinforcement Learning

Misconfigured reward functions can produce three abnormal behavior patterns in reinforcement learning: reckless, timid, and greedy behavior. A reckless agent ignores important negative side effects or constraints while maximizing reward. A timid agent becomes stagnant when penalties are too numerous or too large relative to the principal rewards. A greedy agent engages in unproductive iterations and neglects long-term reward, particularly under sparse reward conditions.

University of Michigan - Ann Arbor

Google

Rewards focus on the immediate context while value functions can focus on the long term. For instance, an action can have a low immediate reward while in long term can have a high value.

Reward vs. Value Function

1. Define dynamic potentials from reward function
2. Inverse Reinforcement Learning.
3. Reward Shaping via Meta-Learning.

Reward Construction Direction without a Prior Estimate

Reward shaping is a technique used to address the challenge of sparse rewards by providing more frequent, intermediate feedback to an agent. As proposed by Andrew Ng, it involves augmenting the original reward function with a potential-based function that depends only on the state. This addition guides the agent's learning without changing the optimal policy, helping to solve problems like meaningless iteration that can arise from delayed rewards.

Reward Shaping as a Solution for Sparse Rewards

In reinforcement learning, there is a mathematical proof that for every specific task, an optimal reward function exists that achieves the fastest convergence. However, identifying such a function is extremely difficult. The challenge of finding this ideal function is known as the Optimal Reward Problem (ORP).

Learn Before

Related

Learn After