The loss function of a Deep Q-Network (DQN) is calculated using the difference between the target Q-value and the current Q-value: $$Loss = targetQ - currentQ = R + \gamma Q'(s',a') - Q(s,a)$$.

University of Michigan - Ann Arbor

Google

DQN is a method using neural network to generate Q-value and select actions instead of forms in original Q-learning method.

Deep Q Networks (DQN)

Playing Atari with Deep Reinforcement Learning : https://www.cs.toronto.edu/~vmnih/docs/dqn.pdf
Human-level control through deep reinforcement learning : https://storage.googleapis.com/deepmind-media/dqn/DQNNaturePaper.pdf

DQN paper

A "Double DQN" system is an implementation of both the Double Q Learning Algorithm and a DQN.

In order to fully implement a Double DQN, two DQN networks would be trained in parallel, similarly to how two tabular datasets are trained in traditional Double Q Learning.

The paper proposing the idea of a Double DQN used stored experience replay values instead to simulate the existence of a secondary network.

As these values were already being used to train the traditional DQN network, the performance benefits are of zero cost in comparison.

Double DQN

Experience Replay is a replay memory technique used in reinforcement learning where we store the agent’s experiences at each time-step, pooled over many episodes into a replay memory. We then usually sample the memory randomly for a mini-batch of experience, and use this to learn off-policy, as with Deep Q-Networks. 

Experience Replay

When updating the network for calculating the Q value, the network of selecting the best actions is not updated.

Learn Before

Related