In value-based methods, we calculate the Q values for all possible actions in the action space, but in real world , the number of actions can be a lot or can’t be imaginable. For example, the Robot walking problem has continuous and high dimensional action space. 

In policy based methods, we directly learn our policy function π without worrying about a value function, which means we can choose actions without calculate the Q(S,A) values.

University of Michigan - Ann Arbor

Policy-based methods learn the policy function directly, without calculating a value function for each action. These methods use a learning signal derived from sampling
instantiations of policy parameters and the set of policies is developed
towards policies that achieve better returns.

An example of a policy-based algorithm is Policy Gradient Methods.

Policy-Based Methods for Deep Reinforcement Learning

The value-based class of algorithms aims to build a value function, which subsequently lets us define a policy. Most frequently used algorithms including Q-learning and deep Q-learning.

Value-based Methods for Deep Reinforcement Learning

A helpful website that explains how policy-based deep reinforcement learning works:
https://medium.com/deep-math-machine-learning-ai/ch-13-deep-reinforcement-learning-deep-q-learning-and-policy-gradients-towards-agi-a2a0b611617e

Reference for Policy-Based Method in Deep Reinforcement Learning

In policy gradients , we directly learn the policy function π which gives the probability distribution of actions.
$\pi(s,a;\theta)\rightarrow [0,1]$ Probability of taking action a given state s with parameters $\theta$.

Neural networks can be used to find the policy function.  The network takes state as an input and produces the probability distribution of actions.

Process in simple terms:
 - Takes in a state and gets the probability of each action
 - Selects the most probable action and observe next state & reward
 - Repeats until the end of the game and evaluates the total rewards
 - Updates the parameters in the network, based on the rewards, using backpropagation

In this way, the network allows the agent to play freely, but with every successive game, it provides better probabilities for actions that will lead the agent to a positive result.

Policy Gradient Methods for Deep Reinforcement Learning

Value-based Methods vs. Policy-based Methods

One value function we frequently used in value-based methods is the action-value function Q(s,a), which represents the total value of taking action a in state s. It is the sum of the future rewards r, adjusted by a discount factor gamma.
$$Q^{*}(s, a)=\max _{\pi} \mathbb{E}\left[r_{t}+\gamma r_{t+1}+\gamma^{2} r_{t+2}+\ldots \mid s_{t}=s, a_{t}=a, \pi\right]$$

The basic steps of deep Q-learning algorithms:
1. Train convolutional neural network to extract the essential features that can help the agent make the decision.
2. Calculate the Q-Value of each possible action.
3. Perform back-propagation to find the most accurate Q-Values.

Deep Q-learning

Valued-based methods aim to find value functions. The advantage of learning the value function is that we can now select actions without a model of the Markov Decision Process.
E.g., in Q learning, the optimal policy is given by
$$
\pi^{\star}(s)=\underset{a \in \mathcal{A}}{\operatorname{argmax}} \hat{Q}^{\star}(s, a)
$$

Learn Before

Related