Another problem of DDDQN may be its experience replay. In fact it can lead to more non-stationarity since experiences related to the other agents' behaviour collected in the past and stored in the buffers might be replayed to the network when they are outdated [1]}. We can have an idea of this effect looking at Figure REF ; the two different plots show respectively the same simulation performed using the two DDDQN and A2C. The former, that uses \(\epsilon \) -greedy as exploration strategy makes all the agents explore at the same time, leading to a drop of the total community reward due to competition when the policy starts to be more deterministic. The total reward will be recovered only when a new equilibrium will be reestablished among the agents. This reason is also probably responsible of the further instability, that is not noticeable in the A2C simulation plot, in which the exploration is connected to the entropy level of the policy, allowing the agents to explore not necessary all at the same time with the same rate. In addition, the positive correlation between number of agents and specialization is confirmed and clearer in the experiments performed with A2C.
