Mean-Semivariance Policy Optimization via Risk-Averse Reinforcement Learning
Keeping risk under control is often more crucial than maximizing expected
reward in real-world decision-making situations, such as finance, robotics,
autonomous driving, etc. The most natural choice of risk measures is variance,
while it penalizes the upside volatility as much as the downside part. Instead,
the (downside) semivariance, which captures negative deviation of a random
variable under its mean, is more suitable for risk-averse proposes. This paper
aims at optimizing the mean-semivariance (MSV) criterion in reinforcement
learning w.r.t. steady rewards. Since semivariance is time-inconsistent and
does not satisfy the standard Bellman equation, the traditional dynamic
programming methods are inapplicable to MSV problems directly. To tackle this
challenge, we resort to the Perturbation Analysis (PA) theory and establish the
performance difference formula for MSV. We reveal that the MSV problem can be
solved by iteratively solving a sequence of RL problems with a policy-dependent
reward function. Further, we propose two on-policy algorithms based on the
policy gradient theory and the trust region method. Finally, we conduct diverse
experiments from simple bandit problems to continuous control tasks in MuJoCo,
which demonstrate the effectiveness of our proposed methods.