Fluid approximations have seen great success in approximating the macro-scale behaviour of Markov systems with a large number of discrete states. However, these methods rely on the continuous-time Markov chain (CTMC) having a particular population structure which suggests a natural continuous state-space endowed with a dynamics for the approximating process. We construct here a general method based on spectral analysis of the transition matrix of the CTMC, without the need for a population structure. Specifically, we use the popular manifold learning method of diffusion maps to analyse the transition matrix as the operator of a hidden continuous process. An embedding of states in a continuous space is recovered, and the space is endowed with a drift vector field inferred via Gaussian process regression. In this manner, we construct an ODE whose solution approximates the evolution of the CTMC mean, mapped onto the continuous space (known as the fluid limit).
There are many ways to satisfy the above criteria, but a common one (used in pCTMCs) is hydrodynamic scaling'', where the increments of the @math -state Markov process mapped to the Euclidean space are @math and the jump rate is @math . The criteria above are derived formally in @cite_17 @cite_18 and ensure that:
@cite_18 @cite_17
2046434141 143740007 2911793117 2002565409
Consider a graph, G, for which the vertices can have two modes, 0 or 1. Suppose that a particle moves around on G according to a discrete time Markov chain with the following rules. With (strictly positive) probabilities pm, pc and pr it moves to a randomly chosen neighbour, changes the mode of the vertex it is at or just stands still, respectively. We call such a random process a (pm, pc, pr)-lamplighter process on G. Assume that the process starts with the particle in a fixed position and with all vertices having mode 0. The convergence rate to stationarity in terms of the total variation norm is studied for the special cases with G = KN, the complete graph with N vertices, and when G = mod N. In the former case we prove that as N --> [infinity], ((2pc + pm) 4pcpm)N log N is a threshold for the convergence rate. In the latter case we show that the convergence rate is asymptotically determined by the cover time CN in that the total variation norm after aN2 steps is given by P(CN > aN2). The limit of this probability can in turn be calculated by considering a Brownian motion with two absorbing barriers. In particular, this means that there is no threshold for this case. From a theoretical perspective, scale invariance, or simply scaling, can fruitfully be modeled with classes of multifractal stochastic processes, designed from positive multiplicative martingales (or cascades). From a practical perspective, scaling in real-world data is often analyzed by means of multiresolution quantities. The present contribution focuses on three different types of such multiresolution quantities, namely increment, wavelet and Leader coefficients, as well as on a specific multifractal processes, referred to as Infinitely Divisible Motions and fractional Brownian motion in multifractal time. It aims at studying, both analytically and by numerical simulations, the impact of varying the number of vanishing moments of the mother wavelet and the order of the increments on the decay rate of the (higher order) covariance functions of the (q-th power of the absolute values of these) multiresolution coefficients. The key result obtained here consist of the fact that, though it fastens the decay of the covariance functions, as is the case for fractional Brownian motions, increasing the number of vanishing moments of the mother wavelet or the order of the increments does not induce any faster decay for the (higher order) covariance functions Consider a Markov decision process (MDP) that admits a set of state-action features, which can linearly express the process's probabilistic transition model. We propose a parametric Q-learning algorithm that finds an approximate-optimal policy using a sample size proportional to the feature dimension @math and invariant with respect to the size of the state space. To further improve its sample efficiency, we exploit the monotonicity property and intrinsic noise structure of the Bellman operator, provided the existence of anchor state-actions that imply implicit non-negativity in the feature space. We augment the algorithm using techniques of variance reduction, monotonicity preservation, and confidence bounds. It is proved to find a policy which is @math -optimal from any initial state with high probability using @math sample transitions for arbitrarily large-scale MDP with a discount factor @math . A matching information-theoretical lower bound is proved, confirming the sample optimality of the proposed method with respect to all parameters (up to polylog factors). We study a zero range process on scale-free networks in order to investigate how network structure influences particle dynamics. The zero range process is defined with the particle jumping rate function @math . We show analytically that a complete condensation occurs when @math where @math is the degree distribution exponent of the underlying networks. In the complete condensation, those nodes whose degree is higher than a threshold are occupied by macroscopic numbers of particles, while the other nodes are occupied by negligible numbers of particles. We also show numerically that the relaxation time follows a power-law scaling @math with the network size @math and a dynamic exponent @math in the condensed phase.