Policy evaluation is a key process in reinforcement learning. It assesses a given policy using estimation of the corresponding value function. When using a parameterized function to approximate the value, it is common to optimize the set of parameters by minimizing the sum of squared Bellman Temporal Differences errors. However, this approach ignores certain distributional properties of both the errors and value parameters. Taking these distributions into account in the optimization process can provide useful information on the amount of confidence in value estimation. In this work we propose to optimize the value by minimizing a regularized objective function which forms a trust region over its parameters. We present a novel optimization method, the Kalman Optimization for Value Approximation (KOVA), based on the Extended Kalman Filter. KOVA minimizes the regularized objective function by adopting a Bayesian perspective over both the value parameters and noisy observed returns. This distributional property provides information on parameter uncertainty in addition to value estimates. We provide theoretical results of our approach and analyze the performance of our proposed optimizer on domains with large state and action spaces.
Distributional perspective on values and observations: Distributional RL @cite_6 treats the full (general) distribution of total return, and considers VF parameters as deterministic. In our work we assume Gaussian distribution over the total return and in addition Gaussian distribution over the VF parameters.
@cite_6
2132849848 2102288976 2144100131 2021472931
We present a novel Bayesian approach to the problem of value function estimation in continuous state spaces. We define a probabilistic generative model for the value function by imposing a Gaussian prior over value functions and assuming a Gaussian noise model. Due to the Gaussian nature of the random processes involved, the posterior distribution of the value function is also Gaussian and is therefore described entirely by its mean and covariance. We derive exact expressions for the posterior process moments, and utilizing an efficient sequential sparsification method, we describe an on-line algorithm for learning them. We demonstrate the operation of the algorithm on a 2-dimensional continuous spatial navigation domain. Standard Reinforcement Learning (RL) aims to optimize decision-making rules in terms of the expected return. However, especially for risk-management purposes, other criteria such as the expected shortfall are sometimes preferred. Here, we describe a method of approximating the distribution of returns, which allows us to derive various kinds of information about the returns. We first show that the Bellman equation, which is a recursive formula for the expected return, can be extended to the cumulative return distribution. Then we derive a nonparametric return distribution estimator with particle smoothing based on this extended Bellman equation. A key aspect of the proposed algorithm is to represent the recursion relation in the extended Bellman equation by a simple replacement procedure of particles associated with a state by using those of the successor state. We show that our algorithm leads to a risk-sensitive RL paradigm. The usefulness of the proposed approach is demonstrated through numerical experiments. This paper characterizes the eigenvalue distributions of full-rank Hermitian matrices generated from a set of independent (non)zero-mean proper complex Gaussian random vectors with a scaled-identity covariance matrix. More specifically, the joint and marginal cumulative distribution function (CDF) of any subset of unordered eigenvalues of the so-called complex (non)central Wishart matrices, as well as new simple and tractable expressions for their joint probability density function (PDF), are derived in terms of a finite sum of determinants. As corollaries to these new results, explicit expressions for the statistics of the smallest and largest eigenvalues, of (non)central Wishart matrices, can be easily obtained. Moreover, capitalizing on the foregoing distributions, it becomes possible to evaluate exactly the mean, variance, and other higher order statistics such as the skewness and kurtosis of the random channel capacity, in the case of uncorrelated multiple-input multiple-output (MIMO) Ricean and Rayleigh fading channels. Doing so bridges the gap between Telatar's initial approach for evaluating the average MIMO channel capacity (Telatar, 1999), and the subsequently widely adopted moment generating function (MGF) approach, thereby setting the basis for a PDF-based framework for characterizing the capacity statistics of MIMO Ricean and Rayleigh fading channels. Classical formulations of the portfolio optimization problem, such as mean-variance or Value-at-Risk (VaR) approaches, can result in a portfolio extremely sensitive to errors in the data, such as mean and covariance matrix of the returns. In this paper we propose a way to alleviate this problem in a tractable manner. We assume that the distribution of returns is partially known, in the sense that onlybounds on the mean and covariance matrix are available. We define the worst-case Value-at-Risk as the largest VaR attainable, given the partial information on the returns' distribution. We consider the problem of computing and optimizing the worst-case VaR, and we show that these problems can be cast as semidefinite programs. We extend our approach to various other partial information on the distribution, including uncertainty in factor models, support constraints, and relative entropy information.