We propose Significance-Offset Convolutional Neural Network, a deep convolutional network architecture for regression of multivariate asynchronous time series.   The model is inspired by standard autoregressive (AR) models and gating mechanisms used in recurrent neural networks.   It involves an AR-like weighting system, where the final predictor is obtained as a weighted sum of adjusted regressors, while the weights are data-dependent functions learnt through a convolutional network. The architecture was designed for applications on asynchronous time series and is evaluated on such datasets: a hedge fund proprietary dataset of over 2 million quotes for a credit derivative index, an artificially generated noisy autoregressive  series  and  household  electricity  consumption  dataset.    The  pro-posed architecture achieves promising results as compared to convolutional and recurrent neural networks. The code for the numerical experiments and the architecture implementation will be shared online to make the research reproducible. Time series forecasting is focused on modeling the predictors of future values of time series given their past. As in many cases the relationship between past and future observations is not deterministic, this amounts to expressing the conditional probability distribution as a function of the past observations: p(X t+d |X t , X t−1 , . . .) = f (X t , X t−1 , . . .).This forecasting problem has been approached almost independently by econometrics and machine learning communities.In this paper we examine the capabilities of convolutional neural networks (CNNs), BID25 in modeling the conditional mean of the distribution of future observations; in other words, the problem of autoregression. We focus on time series with multivariate and noisy signal. In particular , we work with financial data which has received limited public attention from the deep learning community and for which nonparametric methods are not commonly applied. Financial time series are particularly challenging to predict due to their low signal-to-noise ratio (cf. applications of Random Matrix Theory in econophysics BID24 BID3 ) and heavy-tailed distributions BID8 . Moreover, the predictability of financial market returns remains an open problem and is discussed in many publications (cf. efficient market hypothesis of BID11 ).A common situation with financial data is that the same signal (e.g. value of an asset) is observed from different sources (e.g. financial news, analysts, portfolio managers in hedge funds, marketmakers in investment banks) in asynchronous moments of time. Each of these sources may have a different bias and noise with respect to the original signal that needs to be recovered (cf. time series in FIG0 ). Moreover, these sources are usually strongly correlated and lead-lag relationships are possible (e.g. a market-maker with more clients can update its view more frequently and precisely than one with fewer clients). Therefore, the significance of each of the available past observations might be dependent on some other factors that can change in time. Hence, the traditional econometric models such as AR, VAR, VARMA (Hamilton, 1994) might not be sufficient. Yet their relatively good performance motivates coupling such linear models with deep neural networks that are capable of learning highly nonlinear relationships. Quotes from four different market participants (sources) for the same CDS 1 throughout one day. Each trader displays from time to time the prices for which he offers to buy (bid) and sell (ask) the underlying CDS. The filled area marks the difference between the best sell and buy offers (spread) at each time.For these reasons, we propose SignificanceOffset Convolutional Neural Network, a Convolutional Network extension of standard autoregressive models BID34 BID35 equipped with a nonlinear weighting mechanism and provide empirical evidence on its competitiveness with standard multilayer CNN and recurrent Long-Short Term Memory network BID18 . The mechanism is inspired by the gating systems that proved successful in recurrent neural networks BID18 BID6 and highway networks BID37 .2 RELATED WORK 2.1 TIME SERIES FORECASTING Literature in time series forecasting is rich and has a long history in the field of econometrics which makes extensive use of linear stochastic models such as AR, ARIMA and GARCH processes to mention a few. Unlike in machine learning, research in econometrics is more focused on explaining variables rather than improving out-of-sample prediction power. In practice, one can notice that these models 'over-fit ' on financial time series: their parameters are unstable and out-of-sample performance is poor.Reading through recent proceedings of the main machine learning venues (e.g. ICML, NIPS, AIS-TATS, UAI), one can notice that time series are often forecast using Gaussian processes BID31 BID38 BID19 , especially when time series are irregularly sampled BID9 BID26 . Though still largely independent, researchers have started to "bring together the machine learning and econometrics communities" by building on top of their respective fundamental models yielding to, for example, the Gaussian Copula Process Volatility model BID42 . Our paper is in line with this emerging trend by coupling AR models and neural networks.Over the past 5 years, deep neural networks have surpassed results from most of the existing literature in many fields BID33 : computer vision BID23 , audio signal processing and speech recognition BID32 , natural language processing (NLP) BID1 BID7 BID14 BID21 . Although sequence modeling in NLP, i.e. prediction of the next character or word, is related to our forecasting problem (1), the nature of the sequences is too dissimilar to allow using the same cost functions and architectures. Same applies to the adversarial training proposed by BID28 for video frame prediciton, as such approach favors most plausible scenarios rather than outputs close to all possible outputs, while the latter is usually required in financial time series due to stochasticity of the considered processes.Literature on deep learning for time series forecasting is still scarce (cf. BID12 for a recent review). Literature on deep learning for financial time series forecasting is even scarcer though interest in using neural networks for financial predictions is not new BID30 BID29 . More recent papers include BID36 that used 4-layer perceptrons in modeling price change distributions in Limit Order Books, and BID2 who applied more recent WaveNet architecture of van den BID39 to several short univariate and bivariate time-series (including financial ones). Despite claim of applying deep learning, BID17 use autoencoders with a single hidden layer to compress multivariate financial data. Besides these and claims of secretive hedge funds (it can be marketing surfing on the deep learning hype), no promising results or innovative architectures were publicly published so far, to the best of our knowledge. In this paper, we investigate the gold standard architectures' (simple Convolutional Neural Network (CNN), Residual Network, multi-layer LSTM) capabilities on AR-like artificial asynchronous and noisy time series, and on real financial data from the credit default swap market where some inefficiencies may exist, i.e. time series may not be totally random. In this article, we proposed a weighting mechanism that, coupled with convolutional networks, forms a new neural network architecture for time series prediction. The proposed architecture is designed for regression tasks on asynchronous signals in the presence of high amount of noise. This approach has proved to be successful in forecasting financial and artificially generated asynchronous time series outperforming popular convolutional and recurrent networks.The proposed model can be further extended by adding intermediate weighting layers of the same type in the network structure. Another possible generalization that requires further empirical studies can be obtained by leaving the assumption of independent offset values for each past observation, i.e. considering not only 1x1 convolutional kernels in the offset sub-network.Finally, we aim at testing the performance of the proposed architecture on other real-life datasets with relevant characteristics. We observe that there exists a strong need for common 'econometric' datasets benchmark and, more generally, for time series (stochastic processes) regression.APPENDIX A NONLINEARITY IN THE ASYNCHRONOUSLY SAMPLED AUTOREGRESSIVE TIME SERIES Lemma 1. Let X(t) be an AR(2) time series given by DISPLAYFORM0 where (ε(t)) t=1,2,... are i.i.d. error terms. Then DISPLAYFORM1 for any t > k ≥ 2, where a k , b k are rational functions of a and b.Proof. The proof follows a simple induction. It is sufficient to show that DISPLAYFORM2 where DISPLAYFORM3 and E k (t) is a linear combination of {ε(t − i), i = 0, 1, . . . , k − 2}. Basis of the induction is trivially satisfied via 15. In the induction step, we assume that 17 holds for k. For t > k + 1 we have DISPLAYFORM4 . Multiplying sides of this equation by b and adding av k X(t − 1) we obtain DISPLAYFORM5 Since aX(t − 1) + bX(t − 2) = X(t) − ε(t) we get DISPLAYFORM6 As DISPLAYFORM7 is a linear combination of {ε(t − i), i = 0, 1, . . . , k − 1}, the above equation proves 17 for k = k + 1.
Convolutional architecture for learning data-dependent weights for autoregressive forecasting of time series.
{"src": "scitldr-AIC_train", "paper_id": "rJaE2alRW"}