Policy evaluation is a key process in reinforcement learning. It assesses a given policy using estimation of the corresponding value function. When using a parameterized function to approximate the value, it is common to optimize the set of parameters by minimizing the sum of squared Bellman Temporal Differences errors. However, this approach ignores certain distributional properties of both the errors and value parameters. Taking these distributions into account in the optimization process can provide useful information on the amount of confidence in value estimation. In this work we propose to optimize the value by minimizing a regularized objective function which forms a trust region over its parameters. We present a novel optimization method, the Kalman Optimization for Value Approximation (KOVA), based on the Extended Kalman Filter. KOVA minimizes the regularized objective function by adopting a Bayesian perspective over both the value parameters and noisy observed returns. This distributional property provides information on parameter uncertainty in addition to value estimates. We provide theoretical results of our approach and analyze the performance of our proposed optimizer on domains with large state and action spaces.
Kalman filters: Outside of the RL framework, the use of Kalman filter as an optimization method is discussed in @cite_11 @cite_1 @cite_30 . solve the dynamics of each parameter with Kalman filtering. use Kalman filter for normalizing batches. In our work we use Kalman filtering for VF optimization in the context of RL. EKF is connected with the incremental Gauss-Newton method @cite_23 , and with the on-line natural gradient @cite_3 . These methods require inversing the @math -dimensional Fisher information matrix (for @math -dimensional parameter), thus require high computational resources. Our method avoids this inversion in the update step which is more computationally efficient.
@cite_30 @cite_1 @cite_3 @cite_23 @cite_11
1749494163 2079511563 2160337655 2008336375
This paper points out the flaws in using the extended Kalman filter (EKE) and introduces an improvement, the unscented Kalman filter (UKF), proposed by Julier and Uhlman (1997). A central and vital operation performed in the Kalman filter is the propagation of a Gaussian random variable (GRV) through the system dynamics. In the EKF the state distribution is approximated by a GRV, which is then propagated analytically through the first-order linearization of the nonlinear system. This can introduce large errors in the true posterior mean and covariance of the transformed GRV, which may lead to sub-optimal performance and sometimes divergence of the filter. The UKF addresses this problem by using a deterministic sampling approach. The state distribution is again approximated by a GRV, but is now represented using a minimal set of carefully chosen sample points. These sample points completely capture the true mean and covariance of the GRV, and when propagated through the true nonlinear system, captures the posterior mean and covariance accurately to the 3rd order (Taylor series expansion) for any nonlinearity. The EKF in contrast, only achieves first-order accuracy. Remarkably, the computational complexity of the UKF is the same order as that of the EKF. Julier and Uhlman demonstrated the substantial performance gains of the UKF in the context of state-estimation for nonlinear control. Machine learning problems were not considered. We extend the use of the UKF to a broader class of nonlinear estimation problems, including nonlinear system identification, training of neural networks, and dual estimation problems. In this paper, the algorithms are further developed and illustrated with a number of additional examples. Extended Kalman filter (EKF) has been a popular choice to solve simultaneous localization and mapping (SLAM) problems for mobile robots or vehicles. However, the performance of the EKF depends on the correct a priori knowledge of process and sensor measurement noise covariance matrices (Q and R, respectively). Imprecise knowledge of these statistics can cause significant degradation in performance. The present paper proposes the development of a new neurofuzzy based adaptive Kalman filtering algorithm for simultaneous localization and mapping of mobile robots or vehicles, which attempts to estimate the elements of the R matrix of the EKF algorithm, at each sampling instant when a ldquomeasurement updaterdquo step is carried out. The neuro-fuzzy based supervision for the EKF algorithm is carried out with the aim of reducing the mismatch between the theoretical and the actual covariance of the innovation sequences. The free parameters of the neuro-fuzzy system are learned offline, by employing particle swarm optimization in the training phase, which configures the training problem as a high-dimensional stochastic optimization problem. By employing a mobile robot to localize and simultaneously acquire the map of the environment, under several benchmark environment situations with varying landmarks and under several conditions of wrong knowledge of sensor statistics, the performance of the proposed scheme has been evaluated. It has been successfully demonstrated that in each case, the neuro-fuzzy assistance is able to improve highly unpredictable, degrading performance of the EKF and can provide robust and accurate solutions. Increasingly, for many application areas, it is becoming important to include elements of nonlinearity and non-Gaussianity in order to model accurately the underlying dynamics of a physical system. Moreover, it is typically crucial to process data on-line as it arrives, both from the point of view of storage costs as well as for rapid adaptation to changing signal characteristics. In this paper, we review both optimal and suboptimal Bayesian algorithms for nonlinear non-Gaussian tracking problems, with a focus on particle filters. Particle filters are sequential Monte Carlo methods based on point mass (or "particle") representations of probability densities, which can be applied to any state-space model and which generalize the traditional Kalman filtering methods. Several variants of the particle filter such as SIR, ASIR, and RPF are introduced within a generic framework of the sequential importance sampling (SIS) algorithm. These are discussed and compared with the standard EKF through an illustrative example. In this paper, the application of the fuzzy interacting multiple model unscented Kalman filter (FUZZY-IMMUKF) approach to integrated navigation processing for the maneuvering vehicle is presented. The unscented Kalman filter (UKF) employs a set of sigma points through deterministic sampling, such that a linearization process is not necessary, and therefore the errors caused by linearization as in the traditional extended Kalman filter (EKF) can be avoided. The nonlinear filters naturally suffer, to some extent, the same problem as the EKF for which the uncertainty of the process noise and measurement noise will degrade the performance. As a structural adaptation (model switching) mechanism, the interacting multiple model (IMM), which describes a set of switching models, can be utilized for determining the adequate value of process noise covariance. The fuzzy logic adaptive system (FLAS) is employed to determine the lower and upper bounds of the system noise through the fuzzy inference system (FIS). The resulting sensor fusion strategy can efficiently deal with the nonlinear problem for the vehicle navigation. The proposed FUZZY-IMMUKF algorithm shows remarkable improvement in the navigation estimation accuracy as compared to the relatively conventional approaches such as the UKF and IMMUKF.