Fixed Q Target is to protect targetQ changing because of the update of the neural network. TargetQ changing according to the update of the neural network could make the training process unstable.

Why we need fixed Q-Target?

When updating the network for calculating the Q value, the network of selecting the best actions is not updated.

University of Michigan - Ann Arbor

DQN is a method using neural network to generate Q-value and select actions instead of forms in original Q-learning method.

Deep Q Networks (DQN)

Playing Atari with Deep Reinforcement Learning : https://www.cs.toronto.edu/~vmnih/docs/dqn.pdf
Human-level control through deep reinforcement learning : https://storage.googleapis.com/deepmind-media/dqn/DQNNaturePaper.pdf

DQN paper

A "Double DQN" system is an implementation of both the Double Q Learning Algorithm and a DQN.

In order to fully implement a Double DQN, two DQN networks would be trained in parallel, similarly to how two tabular datasets are trained in traditional Double Q Learning.

The paper proposing the idea of a Double DQN used stored experience replay values instead to simulate the existence of a secondary network.

As these values were already being used to train the traditional DQN network, the performance benefits are of zero cost in comparison.

Double DQN

 $Loss = targetQ - currentQ = R + \gamma Q'(s',a') - Q(s,a)$

Loss function of DQN

Experience Replay is a replay memory technique used in reinforcement learning where we store the agent’s experiences at each time-step, pooled over many episodes into a replay memory. We then usually sample the memory randomly for a mini-batch of experience, and use this to learn off-policy, as with Deep Q-Networks. 

Learn Before

Related

Learn After