Value-Based Reward Shaping Formula

A reinforcement learning engineer wants to add an extra reward signal, denoted as a function f, to an agent's learning process to encourage more efficient exploration. They have access to a function Φ(s) which provides a numerical estimate of a state's value, and a discount factor γ. To guarantee that this additional reward signal does not alter the agent's optimal long-term behavior, which of the following structures must the function f have for a transition from state s_t to s_{t+1}?

Analyzing a Flawed Reward Shaping Implementation

Validating a Potential-Based Shaping Function