Other approaches such as EnsembleDQN [1]} and MaxminDQN [2]} maintain ensembles of Q-functions to estimate an unbiased Q-function. EnsembleDQN estimates the state-action values by adding the current observed reward and the maximal state-action value from the average of Q-functions from the ensemble. MaxminDQN creates a proxy Q-function by selecting the minimum Q-value for each action from all the Q-functions and using the maximal state-action value from the proxy Q-function to estimate an unbiased Q-function. Both EnsembleDQN and MaxminDQN have been shown to perform better than Double DQN. The primary insight of this paper is that the performance of ensemble based methods is contingent on maintaining sufficient dissimilarity in the representation space between the Q-functions in the ensembles. If the Q-functions in the ensembles converge to a common representation (and indeed, in this paper we will show that this is the case in many scenarios), the performance of these approaches fall backs to the performance of DQN.
