A possible solution is to introduce underestimation bias in the estimation of the Q-value. Double Q-learning [1]} maintains two independent state-action value estimators (Q-functions). The state-action value of estimator one is calculated by adding observed reward and maximal state-action value from the other estimator. Double DQN [2]} applied this idea using neural networks, and was shown to provide better performance than DQN. More recent actor-critic type deep RL algorithms such as TD3 [3]} and SAC [4]} also use two Q function estimators (in combination with other techniques).
