Hazan et al. [1]} have proposed an alternative approach that maximizes the entropy of the state distribution directly, which allows an agent to cover state space completely. Proposed maximum entropy policy optimization has strong theoretical guarantees. Still, it makes some assumptions about the existence of efficient planning oracle, which makes this algorithm hardly applicable for most modern reinforcement learning tasks.
