A "Double DQN" system is an implementation of both the Double Q Learning Algorithm and a DQN.

In order to fully implement a Double DQN, two DQN networks would be trained in parallel, similarly to how two tabular datasets are trained in traditional Double Q Learning.

The paper proposing the idea of a Double DQN used stored experience replay values instead to simulate the existence of a secondary network.

As these values were already being used to train the traditional DQN network, the performance benefits are of zero cost in comparison.

Double DQN

 $Loss = targetQ - currentQ = R + \gamma Q'(s',a') - Q(s,a)$

Loss function of DQN

Playing Atari with Deep Reinforcement Learning : https://www.cs.toronto.edu/~vmnih/docs/dqn.pdf
Human-level control through deep reinforcement learning : https://storage.googleapis.com/deepmind-media/dqn/DQNNaturePaper.pdf

DQN paper

Experience Replay is a replay memory technique used in reinforcement learning where we store the agent’s experiences at each time-step, pooled over many episodes into a replay memory. We then usually sample the memory randomly for a mini-batch of experience, and use this to learn off-policy, as with Deep Q-Networks. 

Experience Replay

When updating the network for calculating the Q value, the network of selecting the best actions is not updated.

Fixed Q Target

DQN is a method using neural network to generate Q-value and select actions instead of forms in original Q-learning method.

University of Michigan - Ann Arbor

University of Toledo

One of the most popular reinforcement learning algorithms, known to sometimes learn unrealistically high action values since it includes a maximization step that tends to prefer overestimated values.

Q-learning

van Hasselt, H., Guez, A., & Silver, D. (2016). Deep Reinforcement Learning with Double Q-Learning. Proceedings of the AAAI Conference on Artificial Intelligence, 30(1). Retrieved from https://paperswithcode.com/paper/deep-reinforcement-learning-with-double-q

Deep Reinforcement Learning with Double Q-learning

Previously attributed to insufficiently flexible function approximation and noise. However, overestimations can occur regardless of the source of approximation error. Since imprecise value estimates are often used for learning, overestimations may be much more common.

The Problem With Q-learning

Deep Q Networks (DQN)

Double Q learning introduces double estimator to deal with the over-estimation problem. For update of Q values, both estimators use the value of the other estimator. For selecting options, it's still a greedy algorithm based on the average value of two estimators.

Learn Before

Related

Learn After