Addressing overestimation bias is a long standing research topic not only in reinforcement learning but other fields of science such as economics and statistics. It is commonly known as max-operator bias in statistics [1]} and as the winner's curse in economics [2]}, [3]}. As we already discussed above, the max operator in the Q-learning models naturally lead to overestimation bias. To address this, [4]} proposed Double Q-learning, subsequently adapted to a neural network based function approximators as Double DQN [5]}. Alternatively, [6]}, [7]} proposed weighted estimators of Double Q-learning and  [8]} introduced a bias correction term. Other approaches to address the overestimation are based on averaging and ensembling. Techniques include averaging Q-values from previous \(N\)  versions of the Q-network [9]}, taking linear combinations of \(\min \)  and \(\max \)  over the pool of Q-values [10]}, or using a random mixture from the pool [11]}.
