The temporal-difference methods TD( @math ) and Sarsa( @math ) form a core part of modern reinforcement learning. Their appeal comes from their good performance, low computational cost, and their simple interpretation, given by their forward view. Recently, new versions of these methods were introduced, called true online TD( @math ) and true online Sarsa( @math ), respectively (van Seijen & Sutton, 2014). These new versions maintain an exact equivalence with the forward view at all times, whereas the traditional versions only approximate it for small step-sizes. We hypothesize that these true online methods not only have better theoretical properties, but also dominate the regular methods empirically. In this article, we put this hypothesis to the test by performing an extensive empirical comparison. Specifically, we compare the performance of true online TD( @math ) Sarsa( @math ) with regular TD( @math ) Sarsa( @math ) on random MRPs, a real-world myoelectric prosthetic arm, and a domain from the Arcade Learning Environment. We use linear function approximation with tabular, binary, and non-binary features. Our results suggest that the true online methods indeed dominate the regular methods. Across all domains representations the learning speed of the true online methods are often better, but never worse than that of the regular methods. An additional advantage is that no choice between traces has to be made for the true online methods. Besides the empirical results, we provide an in-depth analysis of the theory behind true online temporal-difference learning. In addition, we show that new true online temporal-difference methods can be derived by making changes to the online forward view and then rewriting the update equations.
Furthermore, @cite_0 derived backward views for a series of increasingly complex forward views. The derivation of the true online TD( @math ) equations in Appendix is similar to those derivations.
@cite_0
2200297111
We provide sharp worst-case evaluation complexity bounds for nonconvex minimization problems with general inexpensive constraints, i.e. problems where the cost of evaluating enforcing of the (possibly nonconvex or even disconnected) constraints, if any, is negligible compared to that of evaluating the objective function. These bounds unify, extend or improve all known upper and lower complexity bounds for unconstrained and convexly-constrained problems. It is shown that, given an accuracy level @math , a degree of highest available Lipschitz continuous derivatives @math and a desired optimality order @math between one and @math , a conceptual regularization algorithm requires no more than @math evaluations of the objective function and its derivatives to compute a suitably approximate @math -th order minimizer. With an appropriate choice of the regularization, a similar result also holds if the @math -th derivative is merely H "older rather than Lipschitz continuous. We provide an example that shows that the above complexity bound is sharp for unconstrained and a wide class of constrained problems, we also give reasons for the optimality of regularization methods from a worst-case complexity point of view, within a large class of algorithms that use the same derivative information. Let us consider the differential equation @math with an @math from @math to @math and suppose that there exists a transformation @math from @math to @math ( @math ) such that @math obeys a differential equation @math with some function @math ; then the first equation is said to be lumpable to the second by @math . Here mainly the case is investigated when the original differential equation has been induced by a complex chemical reaction. We provided a series of necessary and sufficient conditions for the existence of such functions @math and @math ; some of them are formulated in terms of @math and @math only. Beyond these conditions our main concern here is how lumping changes those properties of the solutions which are either interesting from the point of view of the qualitative theory of differential equations or from the point ... We extend the De Giorgi--Nash--Moser theory to a class of kinetic Fokker-Planck equations and deduce new results on the Landau-Coulomb equation. More precisely, we first study the H o lder regularity and establish a Harnack inequality for solutions to a general linear equation of Fokker-Planck type whose coefficients are merely measurable and essentially bounded, i.e. assuming no regularity on the coefficients in order to later derive results for non-linear problems. This general equation has the formal structure of the hypoelliptic equations "of type II" , sometimes also called ultraparabolic equations of Kolmogorov type, but with rough coefficients: it combines a first-order skew-symmetric operator with a second-order elliptic operator involving derivatives along only part of the coordinates and with rough coefficients. These general results are then applied to the non-negative essentially bounded weak solutions of the Landau equation with inverse-power law @math @math [--d, 1] whose mass, energy and entropy density are bounded and mass is bounded away from 0, and we deduce the H o lder regularity of these solutions. We investigate the asymptotic number of elements of size @math in a particular class of closed lambda-terms (so-called @math -terms) which are related to axiom systems of combinatory logic. By deriving a differential equation for the generating function of the counting sequence we obtain a recurrence relation which can be solved asymptotically. We derive differential equations for the generating functions of the counting sequences of other more general classes of terms as well: the class of @math -terms and that of closed lambda-terms. Using elementary arguments we obtain upper and lower estimates for the number of closed lambda-terms of size @math . Moreover, a recurrence relation is derived which allows an efficient computation of the counting sequence. @math -terms are discussed briefly. Data is increasingly being bought and sold online, and Web-based marketplace services have emerged to facilitate these activities. However, current mechanisms for pricing data are very simple: buyers can choose only from a set of explicit views, each with a specific price. In this paper, we propose a framework for pricing data on the Internet that, given the price of a few views, allows the price of any query to be derived automatically. We call this capability "query-based pricing." We first identify two important properties that the pricing function must satisfy, called arbitrage-free and discount-free. Then, we prove that there exists a unique function that satisfies these properties and extends the seller's explicit prices to all queries. When both the views and the query are Unions of Conjunctive Queries, the complexity of computing the price is high. To ensure tractability, we restrict the explicit prices to be defined only on selection views (which is the common practice today). We give an algorithm with polynomial time data complexity for computing the price of any chain query by reducing the problem to network flow. Furthermore, we completely characterize the class of Conjunctive Queries without self-joins that have PTIME data complexity (this class is slightly larger than chain queries), and prove that pricing all other queries is NP-complete, thus establishing a dichotomy on the complexity of the pricing problem when all views are selection queries. We introduce a number of new results in the context of multi-view geometry from general algebraic curves. We start with the recovery of camera geometry from matching curves. We first show how one can compute, without any knowledge on the camera, the homography induced by a single planar curve. Then we continue with the derivation of the extended Kruppa's equations which are responsible for describing the epipolar constraint of two projections of a general algebraic curve. As part of the derivation of those constraints we address the issue of dimension analysis and as a result establish the minimal number of algebraic curves required for a solution of the epipolar geometry as a function of their degree and genus. We then establish new results on the reconstruction of general algebraic curves from multiple views. We address three different representations of curves: (i) the regular point representation in which we show that the reconstruction from two views of a curve of degree d admits two solutions, one of degree d and the other of degree d(d − 1). Moreover using this representation, we address the problem of homography recovery for planar curves, (ii) dual space representation (tangents) for which we derive a lower bound for the number of views necessary for reconstruction as a function of the curve degree and genus, and (iii) a new representation (to computer vision) based on the set of lines meeting the curve which does not require any curve fitting in image space, for which we also derive lower bounds for the number of views necessary for reconstruction as a function of curve degree alone. We propose a trust region algorithm for solving nonconvex smooth optimization problems. For any @math ∈¯?(0,?), the algorithm requires at most @math O(∈-3 2) iterations, function evaluations, and derivative evaluations to drive the norm of the gradient of the objective function below any @math ∈?(0,∈¯]. This improves upon the @math O(∈-2) bound known to hold for some other trust region algorithms and matches the @math O(∈-3 2) bound for the recently proposed Adaptive Regularisation framework using Cubics, also known as the arc algorithm. Our algorithm, entitled trace, follows a trust region framework, but employs modified step acceptance criteria and a novel trust region update mechanism that allow the algorithm to achieve such a worst-case global complexity bound. Importantly, we prove that our algorithm also attains global and fast local convergence guarantees under similar assumptions as for other trust region algorithms. We also prove a worst-case upper bound on the number of iterations, function evaluations, and derivative evaluations that the algorithm requires to obtain an approximate second-order stationary point. The word2vec model and application by have attracted a great amount of attention in recent two years. The vector representations of words learned by word2vec models have been shown to carry semantic meanings and are useful in various NLP tasks. As an increasing number of researchers would like to experiment with word2vec or similar techniques, I notice that there lacks a material that comprehensively explains the parameter learning process of word embedding models in details, thus preventing researchers that are non-experts in neural networks from understanding the working mechanism of such models. This note provides detailed derivations and explanations of the parameter update equations of the word2vec models, including the original continuous bag-of-word (CBOW) and skip-gram (SG) models, as well as advanced optimization techniques, including hierarchical softmax and negative sampling. Intuitive interpretations of the gradient equations are also provided alongside mathematical derivations. In the appendix, a review on the basics of neuron networks and backpropagation is provided. I also created an interactive demo, wevi, to facilitate the intuitive understanding of the model. We propose RUDDER, a novel reinforcement learning approach for delayed rewards in finite Markov decision processes (MDPs). RUDDER is based on two main ideas: (i) a backward view approach and (ii) the concept of return-equivalent MDPs. Forward view approaches, like deep Q-networks (DQNs) or Monte Carlo tree search (MCTS), have to average over a large number of probabilistic future state-action paths that increases exponentially with the delay of the reward. Backward view approaches, in contrast, identify actions and states that cause a delayed reward by analyzing already chosen paths. RUDDER's backward view transforms tasks of estimating future returns into regression tasks at which deep learning excels. RUDDER decomposes the return into new, non-delayed rewards by redistributing the original reward across the episode, thereby creating a new MDP that is return-equivalent to the original MDP. "Return-equivalent MDPs" is a new concept ensuring that both MDPs have the same optimal policies. If the return decomposition is optimal, then the new MDP will be stripped of any delayed rewards. In this case, action-value estimates are unbiased and the future expected return is always zero. On several artificial tasks with delayed rewards RUDDER significantly outperforms Monte Carlo, MCTS, temporal difference, TD( ), and reward shaping approaches. RUDDER is even exponentially faster than the last three. RUDDER on top of a Proximal Policy Optimization (PPO) baseline improves the scores on Atari games and excels for delayed rewards. For long delayed rewards as in Bowling, Frostbite, PrivateEye and Venture RUDDER yields exceptional results. TD( ) is a popular family of algorithms for approximate policy evaluation in large MDPs. TD( ) works by incrementally updating the value function after each observed transition. It has two major drawbacks: it makes ine cient use of data, and it requires the user to manually tune a stepsize schedule for good performance. For the case of linear value function approximations and = 0, the Least-Squares TD (LSTD) algorithm of Bradtke and Barto (Bradtke and Barto, 1996) eliminates all stepsize parameters and improves data e ciency. This paper extends Bradtke and Barto's work in three signi cant ways. First, it presents a simpler derivation of the LSTD algorithm. Second, it generalizes from = 0 to arbitrary values of ; at the extreme of = 1, the resulting algorithm is shown to be a practical formulation of supervised linear regression. Third, it presents a novel, intuitive interpretation of LSTD as a model-based reinforcement learning technique. This book consists of two independent works: Part I is 'Solutions of the Einstein Vacuum Equations', by Lydia Bieri. Part II is 'Solutions of the Einstein-Maxwell Equations', by Nina Zipser. A famous result of Christodoulou and Klainerman is the global nonlinear stability of Minkowski spacetime. In this book, Bieri and Zipser provide two extensions to this result. In the first part, Bieri solves the Cauchy problem for the Einstein vacuum equations with more general, asymptotically flat initial data, and describes precisely the asymptotic behavior. In particular, she assumes less decay in the power of @math and one less derivative than in the Christodoulou-Klainerman result. She proves that in this case, too, the initial data, being globally close to the trivial data, yields a solution which is a complete spacetime, tending to the Minkowski spacetime at infinity along any geodesic. In contrast to the original situation, certain estimates in this proof are borderline in view of decay, indicating that the conditions in the main theorem on the decay at infinity on the initial data are sharp. In the second part, Zipser proves the existence of smooth, global solutions to the Einstein-Maxwell equations. A nontrivial solution of these equations is a curved spacetime with an electromagnetic field. To prove the existence of solutions to the Einstein-Maxwell equations, Zipser follows the argument and methodology introduced by Christodoulou and Klainerman. To generalize the original results, she needs to contend with the additional curvature terms that arise due to the presence of the electromagnetic field @math ; in her case the Ricci curvature of the spacetime is not identically zero but rather represented by a quadratic in the components of @math . In particular the Ricci curvature is a constant multiple of the stress-energy tensor for @math . Furthermore, the traceless part of the Riemann curvature tensor no longer satisfies the homogeneous Bianchi equations but rather inhomogeneous equations including components of the spacetime Ricci curvature. Therefore, the second part of this book focuses primarily on the derivation of estimates for the new terms that arise due to the presence of the electromagnetic field. This paper contains the second part of a two-part series on the stability and instability of extreme Reissner–Nordstrom spacetimes for linear scalar perturbations. We continue our study of solutions to the linear wave equation ( g =0 ) on a suitable globally hyperbolic subset of such a spacetime, arising from regular initial data prescribed on a Cauchy hypersurface Σ0 crossing the future event horizon ( H ^ + ). We here obtain definitive energy and pointwise decay, non-decay and blow-up results. Our estimates hold up to and including the horizon ( H ^ + ). A hierarchy of conservations laws on degenerate horizons is also derived. We study some Riemannian metrics on the space of regular smooth curves in the plane, viewed as the orbit space of maps from the circle to the plane modulo the group of diffeomorphisms of the circle, acting as reparameterizations. In particular we investigate the L^2 inner product with respect to 1 plus curvature squared times arclength as the measure along a curve, applied to normal vector field to the curve. The curvature squared term acts as a sort of geometric Tikhonov regularization because, without it, the geodesic distance between any 2 distinct curves is 0, while in our case the distance is always positive. We give some lower bounds for the distance function, derive the geodesic equation and the sectional curvature, solve the geodesic equation with simple endpoints numerically, and pose some open questions. The space has an interesting split personality: among large smooth curves, all its sectional curvatures are positive or 0, while for curves with high curvature or perturbations of high frequency, the curvatures are negative. Two infinite series expressions are derived for the TWDP fading PDF and CDF. The derived series are shown to rapidly converge over the range of practical TWDP fading parameters. An efficient low-complexity algorithm is also proposed for evaluation of these series. Infinite series solution are derived for the TWDP moments, and the integral of the TWDP fading PDF against the complementary error function for performance analysis purposes. The BER of a BPSK system operating in TWDP fading is evaluated as an application of the results. We describe an expansion of the solution of the wave equation on the De Sitter–Schwarzschild metric in terms of resonances. The principal term in the expansion is due to a resonance at 0. The error term decays polynomially if we permit a logarithmic derivative loss in the angular directions and exponentially if we permit an ( ) derivative loss in the angular directions. Accurately estimating the cardinality of aggregate views is crucial for logical and physical design of data warehouses. While the warehouse is under development and data are not available yet, the approaches based on accessing data cannot be adopted. This paper proposes an approach to estimate the cardinality of views based on a-priori information derived from the application domain. We face the problem by first computing satisfactory bounds for the cardinality, then by capitalizing on these bounds to determine a good probabilistic estimate for it. Bounds are determined by using, besides the functional dependencies expressed by the multidimensional scheme, additional domain-derived information in the form of cardinality constraints which may bound either the cardinality of a given view or the ratio between the cardinalities of two given views. In particular, we propose a bounding strategy which achieves an effective trade-off between the tightness of the bounds produced and the computational complexity. Abstract This study presents an analytical approach for obtaining Green's function of the two-dimensional radiative transfer equation to the boundary-value problem of a layered medium. A conventional Fourier transform and a modified Fourier series which is defined in a rotated reference frame are applied to derive an analytical solution of the radiance in the transformed space. The Monte Carlo method was used for a successful validation of the derived solutions. Sutton, Szepesvari and Maei (2009) recently introduced the first temporal-difference learning algorithm compatible with both linear function approximation and off-policy training, and whose complexity scales only linearly in the size of the function approximator. Although their gradient temporal difference (GTD) algorithm converges reliably, it can be very slow compared to conventional linear TD (on on-policy problems where TD is convergent), calling into question its practical utility. In this paper we introduce two new related algorithms with better convergence rates. The first algorithm, GTD2, is derived and proved convergent just as GTD was, but uses a different objective function and converges significantly faster (but still not as fast as conventional TD). The second new algorithm, linear TD with gradient correction, or TDC, uses the same update rule as conventional TD except for an additional term which is initially zero. In our experiments on small test problems and in a Computer Go application with a million features, the learning rate of this algorithm was comparable to that of conventional TD. This algorithm appears to extend linear TD to off-policy learning with no penalty in performance while only doubling computational requirements. This paper examine the Euler-Lagrange equations for the solution of the large deformation diffeomorphic metric mapping problem studied in (1998) and Trouve (1995) in which two images I 0, I 1 are given and connected via the diffeomorphic change of coordinates I 0???1=I 1 where ?=?1 is the end point at t= 1 of curve ? t , t?[0, 1] satisfying .? t =v t (? t ), t? [0,1] with ?0=id. The variational problem takes the form @math where ?v t? V is an appropriate Sobolev norm on the velocity field v t(·), and the second term enforces matching of the images with ?·?L 2 representing the squared-error norm. In this paper we derive the Euler-Lagrange equations characterizing the minimizing vector fields v t, t?[0, 1] assuming sufficient smoothness of the norm to guarantee existence of solutions in the space of diffeomorphisms. We describe the implementation of the Euler equations using semi-lagrangian method of computing particle flows and show the solutions for various examples. As well, we compute the metric distance on several anatomical configurations as measured by ?0 1?v t? V dt on the geodesic shortest paths. The aim of computational molecular design is the identification of promising hypothetical molecules with a predefined set of desired properties. We address the issue of accelerating the material discovery with state-of-the-art machine learning techniques. The method involves two different types of prediction; the forward and backward predictions. The objective of the forward prediction is to create a set of machine learning models on various properties of a given molecule. Inverting the trained forward models through Bayes’ law, we derive a posterior distribution for the backward prediction, which is conditioned by a desired property requirement. Exploring high-probability regions of the posterior with a sequential Monte Carlo technique, molecules that exhibit the desired properties can computationally be created. One major difficulty in the computational creation of molecules is the exclusion of the occurrence of chemically unfavorable structures. To circumvent this issue, we derive a chemical language model that acquires commonly occurring patterns of chemical fragments through natural language processing of ASCII strings of existing compounds, which follow the SMILES chemical language notation. In the backward prediction, the trained language model is used to refine chemical strings such that the properties of the resulting structures fall within the desired property region while chemically unfavorable structures are successfully removed. The present method is demonstrated through the design of small organic molecules with the property requirements on HOMO-LUMO gap and internal energy. The R package iqspr is available at the CRAN repository.