In on-policy AC, the agent is restricted to use the current policy iterate to collect samples, which may not be exploratory. Moreover, on-policy sampling might be of high risk (e.g. self driving cars [1]}), high cost (e.g. robotics [2]}, [3]}), or might be unethical (e.g. in clinical trials [4]}, [5]}, [6]}). Off-policy AC, on the other hand, is more practical than on-policy sampling [3]}. Specifically, off-policy sampling enables the agent to learn using the historical data, hence decouples the sampling process and the learning process. This allows the agent to learn in an off-line manner, and makes RL applicable in high-stake problems mentioned earlier. In addition, it is empirically observed that by using a suitable behavior policy, one can rectify the exploration issue in on-policy AC. As a result, off-policy learning successfully solved many practical problems in different areas, such as board game [8]}, city navigation [9]}, education [10]}, and healthcare [11]}.
