 $Loss = targetQ - currentQ = R + \gamma Q'(s',a') - Q(s,a)$

University of Michigan - Ann Arbor

DQN is a method using neural network to generate Q-value and select actions instead of forms in original Q-learning method.

Deep Q Networks (DQN)

Playing Atari with Deep Reinforcement Learning : https://www.cs.toronto.edu/~vmnih/docs/dqn.pdf
Human-level control through deep reinforcement learning : https://storage.googleapis.com/deepmind-media/dqn/DQNNaturePaper.pdf

DQN paper

A "Double DQN" system is an implementation of both the Double Q Learning Algorithm and a DQN.

In order to fully implement a Double DQN, two DQN networks would be trained in parallel, similarly to how two tabular datasets are trained in traditional Double Q Learning.

The paper proposing the idea of a Double DQN used stored experience replay values instead to simulate the existence of a secondary network.

As these values were already being used to train the traditional DQN network, the performance benefits are of zero cost in comparison.

Double DQN

Loss function of DQN

Experience Replay is a replay memory technique used in reinforcement learning where we store the agent’s experiences at each time-step, pooled over many episodes into a replay memory. We then usually sample the memory randomly for a mini-batch of experience, and use this to learn off-policy, as with Deep Q-Networks. 

Experience Replay

When updating the network for calculating the Q value, the network of selecting the best actions is not updated.

Learn Before

Related