We propose to tackle a time series regression problem by computing temporal evolution of a probability density function to provide a probabilistic forecast. A Recurrent Neural Network (RNN) based model is employed to learn a nonlinear operator for temporal evolution of a probability density function. We use a softmax layer for a numerical discretization of a smooth probability density functions, which transforms a function approximation problem to a classification task. Explicit and implicit regularization strategies are introduced to impose a smoothness condition on the estimated probability distribution. A Monte Carlo procedure to compute the temporal evolution of the distribution for a multiple-step forecast is presented. The evaluation of the proposed algorithm on three synthetic and two real data sets shows advantage over the compared baselines. Application of the deep learning for manufacturing processes has attracted a great attention as one of the core technologies in Industry 4.0 BID15 . In many manufacturing processes, e.g. blast furnace, smelter, and milling, the complexity of the overall system makes it almost impossible or impractical to develop a simulation model from the first principles. Hence, system identification from sensor observations has been a long-standing research topic BID24 . Still, when the observation is noisy and there is no prior knowledge on the underlying dynamics, there is only a very limited number of methods for the reconstruction of nonlinear dynamics.In this work, we consider the following class of problems, where the system is driven by a complex underlying dynamical system, e.g., ∂y ∂t = F(y(t), y(t − τ ), u(t)).Here , y(t) is a continuous process, F is a nonlinear operator, τ is a delay-time parameter, and u(t) is an exogenous forcing, such as control parameters. At time step t, we then observe a noisy measurement of y(t) which can be defined by the following noise model DISPLAYFORM0 where ν t is a multiplicative and t is an additive noise process. In FORMULA0 and FORMULA1 , we place no assumption on function F, do not assume any distributional properties of noises ν t and t , but assume the knowledge of the control parameters u(t).Since the noise components, ν t and t , are stochastic processes, the observationŷ t is a random variable. In this work, we are interested in computing temporal evolution of the probability density function (PDF) ofŷ, given the observations up to time step t, i.e., p(ŷ t+n | Y 0:t , U 0:t+n−1 ) for n ≥ 1, where Y 0:t = (ŷ 0 , · · · ,ŷ t ) is a trajectory of the past observations and U 0:t+n−1 = (u 0 , · · · , u t+n−1 ) consists of the history of the known control actions, U 0:t−1 , and a future control scenario, U t:t+n−1 . We show , in Section 3, a class of problems, where simple regression problem of forecasting the value ofŷ t+n is not sufficient or not possible, e.g., chaotic systems. Note that the computation of time evolution of a PDF has been a long-standing topic in statistical physics. For a simple Markov process, there are well-established theories based on the Fokker-Planck equation. However, it is very difficult to extend those theories to a more general problem, such as delay-time dynamical systems, or apply it to complex nonlinear systems.Modeling of the system (1) has been extensively studied in the past, in particular, under the linearity assumptions on F and certain noise models, e.g., Gaussian t and ν t = 1 in (2). The approaches based on auto-regressive processes BID18 and Kalman filter BID9 are good examples. Although these methods do estimate the predictive probability distribution and enable the computation of the forecast uncertainty, the assumptions on the noise and linearity in many cases make it challenging to model real nonlinear dynamical systems.Recently, a nonlinear state-space model based on the Gaussian process, called the Gaussian Process State Space Model (GPSSM), has been extended for the identification of nonlinear system BID5 BID4 . GPSSM is capable of representing a nonlinear system and is particularly advantageous when the size of the data set is relatively small that it is difficult to train a deep learning model. However, the joint Gaussian assumption of GPSSM may restrict the representation capability for a complex non-Gaussian noise.A recent success of deep learning created a flurry of new approaches for time series modeling and prediction. The ability of deep neural networks, such as RNN, to learn complex nonlinear spatiotemporal relationships in the data enabled these methods to outperform the classical time series approaches. For example, in the recent works of BID20 BID11 ; BID3 , the authors proposed different variants of the RNN-based algorithms to perform time series predictions and showed their advantage over the traditional methods. Although encouraging , these approaches lack the ability to estimate the probability distribution of the predictions since RNN is a deterministic model and unable to fully capture the stochastic nature of the data.To enable RNN to model the stochastic properties of the data, BID2 augmented RNN with a latent random variable included in the hidden state and proposed to estimate the resulting model using variational inference. In a similar vein, the works of BID0 ; BID14 extend the traditional Kalman filter to handle nonlinear dynamics when the inference becomes intractable. Their approach is based on formulating the variational lower bound and optimizing it under the assumption of Gaussian posterior.Another recent line of works enabled stochasticity in the RNN-based models by drawing a connection between Bayesian variation inference and a dropout technique. In particular, BID6 showed that the model parameter uncertainty (which then leads to uncertainty in model predictions), that traditionally was estimated using variational inference, can be approximated using a dropout method (a random removal of some connections in the network structure). The prediction uncertainty is then estimated by evaluating the model outputs at different realizations of the dropout weights. Following the ideas of BID6 , BID27 proposed additional ways (besides modeling the parameter uncertainty) to quantify the forecast uncertainty in RNN, which included the model mis-specification error and the inherent noise of the data.
Proposed RNN-based algorithm to estimate predictive distribution in one- and multi-step forecasts in time series prediction problems
{"src": "scitldr-AIC_train", "paper_id": "BkDB51WR-"}