In value-based methods, we calculate the Q values for all possible actions in the action space, but in real world , the number of actions can be a lot or can’t be imaginable. For example, the Robot walking problem has continuous and high dimensional action space. 

In policy based methods, we directly learn our policy function π without worrying about a value function, which means we can choose actions without calculate the Q(S,A) values.

Value-based Methods vs. Policy-based Methods

One value function we frequently used in value-based methods is the action-value function Q(s,a), which represents the total value of taking action a in state s. It is the sum of the future rewards r, adjusted by a discount factor gamma.
$$Q^{*}(s, a)=\max _{\pi} \mathbb{E}\left[r_{t}+\gamma r_{t+1}+\gamma^{2} r_{t+2}+\ldots \mid s_{t}=s, a_{t}=a, \pi\right]$$

The basic steps of deep Q-learning algorithms:
1. Train convolutional neural network to extract the essential features that can help the agent make the decision.
2. Calculate the Q-Value of each possible action.
3. Perform back-propagation to find the most accurate Q-Values.

Deep Q-learning

Valued-based methods aim to find value functions. The advantage of learning the value function is that we can now select actions without a model of the Markov Decision Process.
E.g., in Q learning, the optimal policy is given by
$$
\pi^{\star}(s)=\underset{a \in \mathcal{A}}{\operatorname{argmax}} \hat{Q}^{\star}(s, a)
$$

Benefits of Valued-based Methods

The value-based class of algorithms aims to build a value function, which subsequently lets us define a policy. Most frequently used algorithms including Q-learning and deep Q-learning.

University of Michigan - Ann Arbor

There are different types of deep reinforcement learning methods.

If we use either value functions or policies to act on the environment, the method is called Model-free reinforcement learning:
   * Value-based methods
   * Policy-based methods

If we make use of models of the environment, it is referred to as Model-based reinforcement learning.

Deep Reinforcement Learning Methods

https://missinglink.ai/guides/neural-network-concepts/complete-guide-deep-reinforcement-learning-concepts-process-real-world-applications/

Complete Guide to Deep Reinforcement Learning: Concepts, Process, and Real World Applications

Policy-based methods learn the policy function directly, without calculating a value function for each action. These methods use a learning signal derived from sampling
instantiations of policy parameters and the set of policies is developed
towards policies that achieve better returns.

An example of a policy-based algorithm is Policy Gradient Methods.

Policy-Based Methods for Deep Reinforcement Learning

Value-based Methods for Deep Reinforcement Learning

When a model of the environment ( (the estimated transition function and
the estimated reward function) is available, the model can then act as a proxy for the actual environment. 
 - In many games, the rule of the game is the model.
 - In other cases, it can be the law of Physics. Sometimes, we know how to model it and build simulators for it.

We can define this model with rules or equations. Or, we can use the Gaussian Process, Gaussian Mixture Model (GMM) or deep networks. To fit these models, we run a controller to collect sample trajectories and train the models with supervised learning.




Model-Based Methods for Deep Reinforcement Learning

The respective strengths of the model-free versus model-based approaches depend on different factors.

 - First, the best suited approach depends on whether the agent has access to a model of the environment. If that’s not the case, the learned model usually has some inaccuracies that should be taken into account.
 - Second, a model-based approach requires working in conjunction with a planning algorithm (or controller), which is often computationally demanding. The time constraints for computing the policy π(s) need to be taken into account.
 - Third, for some tasks, the structure of the policy (or value function) is the easiest one to learn, but for other tasks, the model of the environment may be learned more efficiently due to the particular structure of the task. Thus, which one performs better depends on the structure of the model, policy, and value function

Learn Before

Related

Learn After