Neural network needs a lot of data for small steps updating. Using one step data can't achieve satisfying performance and is a waste of data. Also, reinforcement learning works on sequential condition. Training neural network using one-step data is not beneficial for neural network's generalization.

Why do we need experience replay?

Experience Replay is a replay memory technique used in reinforcement learning where we store the agent’s experiences at each time-step, pooled over many episodes into a replay memory. We then usually sample the memory randomly for a mini-batch of experience, and use this to learn off-policy, as with Deep Q-Networks. 

University of Michigan - Ann Arbor

DQN is a method using neural network to generate Q-value and select actions instead of forms in original Q-learning method.

Deep Q Networks (DQN)

Playing Atari with Deep Reinforcement Learning : https://www.cs.toronto.edu/~vmnih/docs/dqn.pdf
Human-level control through deep reinforcement learning : https://storage.googleapis.com/deepmind-media/dqn/DQNNaturePaper.pdf

DQN paper

A "Double DQN" system is an implementation of both the Double Q Learning Algorithm and a DQN.

In order to fully implement a Double DQN, two DQN networks would be trained in parallel, similarly to how two tabular datasets are trained in traditional Double Q Learning.

The paper proposing the idea of a Double DQN used stored experience replay values instead to simulate the existence of a secondary network.

As these values were already being used to train the traditional DQN network, the performance benefits are of zero cost in comparison.

Double DQN

 $Loss = targetQ - currentQ = R + \gamma Q'(s',a') - Q(s,a)$

Loss function of DQN

Experience Replay

When updating the network for calculating the Q value, the network of selecting the best actions is not updated.

Learn Before

Related

Learn After