A "Double DQN" system is an implementation of both the Double Q Learning Algorithm and a DQN.

In order to fully implement a Double DQN, two DQN networks would be trained in parallel, similarly to how two tabular datasets are trained in traditional Double Q Learning.

The paper proposing the idea of a Double DQN used stored experience replay values instead to simulate the existence of a secondary network.

As these values were already being used to train the traditional DQN network, the performance benefits are of zero cost in comparison.

Double DQN

Double Q learning introduces double estimator to deal with the over-estimation problem. For update of Q values, both estimators use the value of the other estimator. For selecting options, it's still a greedy algorithm based on the average value of two estimators.

University of Michigan - Ann Arbor

University of Toledo

One of the most popular reinforcement learning algorithms, known to sometimes learn unrealistically high action values since it includes a maximization step that tends to prefer overestimated values.

Q-learning

van Hasselt, H., Guez, A., & Silver, D. (2016). Deep Reinforcement Learning with Double Q-Learning. Proceedings of the AAAI Conference on Artificial Intelligence, 30(1). Retrieved from https://paperswithcode.com/paper/deep-reinforcement-learning-with-double-q

Deep Reinforcement Learning with Double Q-learning

Previously attributed to insufficiently flexible function approximation and noise. However, overestimations can occur regardless of the source of approximation error. Since imprecise value estimates are often used for learning, overestimations may be much more common.

The Problem With Q-learning

DQN is a method using neural network to generate Q-value and select actions instead of forms in original Q-learning method.

Learn Before

Related

Learn After