Proximal Policy Optimization (PPO)

In a reinforcement learning context, a policy is updated by maximizing an objective function. Consider an objective function that incorporates two distinct mechanisms to control the size of policy updates relative to a reference policy:

1. A 'clipping' mechanism that puts a hard limit on the probability ratio between the new and reference policies, effectively creating a boundary beyond which the objective does not increase for a given sample.
2. A 'penalty' term that is subtracted from the obj

Diagnosing Training Instability in Reinforcement Learning

Complementary Roles of Policy Update Constraints

Composite Objective for PPO-Clip