In reinforcement learning, the standard criterion to evaluate policies in a state is the expectation of (discounted) sum of rewards. However, this criterion may not always be suitable, we consider an alternative criterion based on the notion of quantiles. In the case of episodic reinforcement learning problems, we propose an algorithm based on stochastic approximation with two timescales. We evaluate our proposition on a simple model of the TV show, Who wants to be a millionaire.
Recent work in Markov decision process and reinforcement learning considered conditional Value-at-risk (CVaR), a criterion related to quantile, as a risk measure. @cite_7 proved the existence of deterministic wealth-Markovian policies optimal with respect to CVaR. @cite_4 proposed gradient-based algorithms for CVaR optimization. In contrast, @cite_18 used CVaR in constraints instead of as objective function.
@cite_18 @cite_4 @cite_7
2128347943 2949963197 2949207039 2619529385
We propose a new constrained Markov decision process framework with risk-type constraints. The risk metric we use is Conditional Value-at-Risk (CVaR), which is gaining popularity in finance. It is a conditional expectation but the conditioning is defined in terms of the level of the tail probability. We propose an iterative offline algorithm to find the risk-contrained optimal control policy. A stochastic approximation-inspired ‘learning’ variant is also sketched. Conditional Value at Risk (CVaR) is a prominent risk measure that is being used extensively in various domains. We develop a new formula for the gradient of the CVaR in the form of a conditional expectation. Based on this formula, we propose a novel sampling-based estimator for the CVaR gradient, in the spirit of the likelihood-ratio method. We analyze the bias of the estimator, and prove the convergence of a corresponding stochastic gradient descent algorithm to a local CVaR optimum. Our method allows to consider CVaR optimization in new domains. As an example, we consider a reinforcement learning application, and learn a risk-sensitive controller for the game of Tetris. In many sequential decision-making problems we may want to manage risk by minimizing some measure of variability in costs in addition to minimizing a standard criterion. Conditional value-at-risk (CVaR) is a relatively new risk measure that addresses some of the shortcomings of the well-known variance-related risk measures, and because of its computational efficiencies has gained popularity in finance and operations research. In this paper, we consider the mean-CVaR optimization problem in MDPs. We first derive a formula for computing the gradient of this risk-sensitive objective function. We then devise policy gradient and actor-critic algorithms that each uses a specific method to estimate this gradient and updates the policy parameters in the descent direction. We establish the convergence of our algorithms to locally risk-sensitive optimal policies. Finally, we demonstrate the usefulness of our algorithms in an optimal stopping problem. Abstract This paper proposes a stochastic and dynamic mixed-integer linear program (SD-MILP) for optimal coordinated bidding of a risk-averse profit-maximizing hydropower producer. The day-ahead, intra-day, and real-time markets are considered. To model and predict day-ahead, intra-day, and real-time prices, the Holt–Winter (HW) and the Generalized Autoregressive Conditional Heteroscedastic (GARCH) predictors are combined using a proposed Markov switch. The discrete behavior of intra-day and real-time prices is modeled as different Markov states. The proposed Markov-based HW–GARCH model with a standard scenario generation-reduction technique is used to capture the uncertainty in day-ahead, intra-day, and real-time prices. The time-dependent conditional value at risk (T-CVaR) is proposed to model the risk of trading in different considered markets. The convex combination of the expected profit and T-CVaR is used as the objective of SD-MILP. The Markov-based HW–GARCH is modeled in Matlab and the SD-MILP is coded in GAMS. The Markov-based HW–GARCH predictor and the SD-MILP are used to develop the bidding curve of a three-reservoir hydropower producer using the electricity prices from the Nordic power market. To further examine the developed models a seven-reservoir hydropower producer is also studied. For these two cases, the coordinated bidding curves are derived and discussed.