A dynamical model consists of a continuous self-map @math of a compact state space @math and a continuous observation function @math . This paper considers the fitting of a parametrized family of dynamical models to an observed real-valued stochastic process using empirical risk minimization. The limiting behavior of the minimum risk parameters is studied in a general setting. We establish a general convergence theorem for minimum risk estimators and ergodic observations. We then study conditions under which empirical risk minimization can effectively separate the signal from the noise in an additive observational noise model. The key, necessary condition in the latter results is that the family of dynamical models has limited complexity, which is quantified through a notion of entropy for families of infinite sequences. Close connections between entropy and limiting average mean widths for stationary processes are established.
We study the density estimation problem with observations generated by certain dynamical systems that admit a unique underlying invariant Lebesgue density. Observations drawn from dynamical systems are not independent and moreover, usual mixing concepts may not be appropriate for measuring the dependence among these observations. By employing the @math -mixing concept to measure the dependence, we conduct statistical analysis on the consistency and convergence of the kernel density estimator. Our main results are as follows: First, we show that with properly chosen bandwidth, the kernel density estimator is universally consistent under @math -norm; Second, we establish convergence rates for the estimator with respect to several classes of dynamical systems under @math -norm. In the analysis, the density function @math is only assumed to be H " o lder continuous which is a weak assumption in the literature of nonparametric density estimation and also more realistic in the dynamical system context. Last but not least, we prove that the same convergence rates of the estimator under @math -norm and @math -norm can be achieved when the density function is H " o lder continuous, compactly supported and bounded. The bandwidth selection problem of the kernel density estimator for dynamical system is also discussed in our study via numerical simulations. This paper is concerned with the problem of recovering a finite, deterministic time series from observations that are corrupted by additive, independent noise. A distinctive feature of this problem is that the available data exhibit long-range dependence and, as a consequence, existing statistical theory and methods are not readily applicable. This paper gives an analysis of the denoising problem that extends recent work of Lalley, but begins from first principles. Both positive and negative results are established. The positive results show that denoising is possible under somewhat restrictive conditions on the additive noise. The negative results show that, under more general conditions on the noise, no procedure can recover the underlying deterministic series. The paper uses empirical process techniques to study the asymptotics of the least-squares estimator (LSE) for the fitting of a nonlinear regression function. By combining and extending ideas of Wu and Van de Geer, it establishes new consistency and central limit theorems that hold under only second moment assumptions on the errors. An application to a delicate example of Wu's illustrates the use of the new theorems, leading to a normal approximation to the LSE with unusual logarithmic rescalings. The problem of extracting a signal x n from a noise-corrupted time series y n = x n + e n is considered. The signal x n is assumed to be generated by a discrete-time, deterministic, chaotic dynamical system F, in particular, x n = F n (x 0 ), where the initial point x 0 is assumed to lie in a compact hyperbolic F-invariant set. It is shown that (1) if the noise sequence e n is Gaussian then it is impossible to consistently recover the signal x n , but (2) if the noise sequence consists of i.i.d. random vectors uniformly bounded by a constant δ > 0, then it is possible to recover the signal x n provided δ < 5Δ, where Δ is a separation threshold for F. A filtering algorithm for the latter situation is presented.  We consider the asymptotic consistency of maximum likelihood parameter estimation for dynamical systems observed with noise. Under suitable conditions on the dynamical systems and the observations, we show that maximum likelihood parameter estimation is consistent. Our proof involves ideas from both information theory and dynamical systems. Furthermore, we show how some well-studied properties of dynamical systems imply the general statistical properties related to maximum likelihood estimation. Finally, we exhibit classical families of dynamical systems for which maximum likelihood estimation is consistent. Examples include shifts of finite type with Gibbs measures and Axiom A attractors with SRB measures. Suppose we are given a time series or a signal @math for @math . We consider the problem of predicting the signal in the interval @math based on a knowledge of its history and nothing more. We ask the following question: what is the largest value of @math for which a prediction can be made? We show that the answer to this question is contained in a fundamental result of information theory due to Wyner, Ziv, Ornstein, and Weiss. In particular, for the class of chaotic signals, the upper bound is @math in the limit @math , with @math being entropy in a sense that is explained in the text. If @math is small for @math , where @math is of the order of a characteristic time scale, the pattern of events leading up to @math is similar to the pattern of events leading up to @math . It is reasonable to expect @math to be a good predictor of @math . All existing methods for prediction use this idea...  We consider the problem of forecasting the next (observable) state of an unknown ergodic dynamical system from a noisy observation of the present state. Our main result shows, for example, that support vector machines (SVMs) using Gaussian RBF kernels can learn the best forecaster from a sequence of noisy observations if (a) the unknown observational noise process is bounded and has a summable α-mixing rate and (b) the unknown ergodic dynamical system is defined by a Lipschitz continuous function on some compact subset of R d and has a summable decay of correlations for Lipschitz continuous functions. In order to prove this result we first establish a general consistency result for SVMs and all stochastic processes that satisfy a mixing notion that is substantially weaker than α-mixing. We establish a Bernstein-type inequality for a class of stochastic processes that include the classical geometrically φ-mixing processes, Rio’s generalization of these processes, as well as many time-discrete dynamical systems. Modulo a logarithmic factor and some constants, our Bernsteintype inequality coincides with the classical Bernstein inequality for i.i.d. data. We further use this new Bernstein-type inequality to derive an oracle inequality for generic regularized empirical risk minimization algorithms and data generated by such processes. Applying this oracle inequality to support vector machines using the Gaussian kernels for both least squares and quantile regression, it turns out that the resulting learning rates match, up to some arbitrarily small extra term in the exponent, the optimal rates for i.i.d. processes.