in this paper , we study an analog of the periodogram for extremal events . in classical time series analysis ,the periodogram is a method of moments estimator of the spectral density of a second order stationary time series ; see , for example , the standard monographs brillinger  , brockwell and davis , grenander and rosenblatt , hannan , priestley .the notions of spectral density and periodogram are the respective frequency domain analogs of the autocorrelation function and the sample autocorrelation function in the time domain . in the context of extremal events , these notions are not meaningful since second order characteristics are not suited for describing the occurrence of rare events .  however , davis and mikosch introduced a time domain analog of the autocorrelation function , the _ extremogram _ for rare events .for an -valued strictly stationary time series and a borel set bounded away from zero , _ the extremogram at lag _ is given as the limit this definition requires that the support of ( here and in what follows , denotes a generic element of any stationary sequence ) is unbounded and , more importantly , that the limit on the right - hand side exists . in general, these limits do not exist .a sufficient condition for their existence is _ regular variation _ of all pairs or , more generally , _regular variation of the finite - dimensional distributions _ of the process . a precise definition of regular variation will be given in section [ subsecregvar ] .since is assumed to be bounded away from zero , the probabilities converge to zero as .then the following calculation is straightforward for  : ^ 2}{p(x^{-1 } x\in a)(1- p(x^{-1 } x\in a ) ) } \\ & = & \lim_{{x\to\infty } } p\bigl(x^{-1 } x_h\in a \mid x^{-1 } x_0\in a\bigr)=\rho_{a}(h).\end{aligned}\ ] ] for fixed , constitutes a strictly stationary sequence .the limit sequence inherits the property of correlation function from .therefore , in an asymptotic sense , one can use the notions of classical time series analysis ( such as the autocorrelation function ) for the sequences of indicator functions .of course , there are several crucial differences to classical time series analysis .  *the notion of autocorrelation function is only defined in an asymptotic sense . * the strictly stationary sequence of indicator functions depends on the threshold , that is, we are dealing with an array of strictly stationary processes .* by definition , the values can not be negative .davis and mikosch introduced the extremogram and calculated the extremogram for various standard regularly varying time series models such as the garch model , stochastic volatility and linear processes with regularly varying noise , and infinite variance stable processes ; see also section [ subsecexam ] .they studied the basic asymptotic properties of the extremogram ( consistency , asymptotic normality ) and also introduced a frequency domain analog of the correlation function  given as the fourier series .\ ] ] a natural estimator of is found by replacing the correlations by sample analogs .the convergence in the mean square sense of such an analog of the classical periodogram estimator towards the spectral density at a fixed frequency was shown in .however , the periodogram of used in had to be truncated to achieve consistency ; the truncation level depended on some mixing rate which is unknown for real - life data . in this paper, we overcome this inconvenience .in addition , we study the periodogram ordinates of the indicator functions at finitely many frequencies .we show that the limiting vector of the periodogram ordinates at distinct fixed or fourier frequencies converges in distribution to a vector of independent exponential random variables .this property parallels the asymptotic theory for the periodogram of a second order stationary sequence ; see , for example , brockwell and davis , chapter 10 .  in classical time series analysis ,the asymptotic independence of the periodogram at distinct frequenc ies is the theoretical basis for consistent estimation of the spectral density via weighted averages or kernel based methods .we show that weighted average estimators of the periodogram evaluated at fourier frequencies in the neighborhood of a fixed non - zero frequency are consistent estimators of the limiting spectral density .the paper is organized as follows . in section [ secprelim ], we introduce basic notions and conditions used throughout this paper . in section [ subsecregvar ] , we define regular variation of a strictly stationary sequence . in section [ subsecmix ] ,we consider those mixing conditions which are relevant for the results of this paper .the periodogram of extreme events is introduced in section [ subsecper ] . in section [ subsecexam ], we discuss some regularly varying strictly stationary sequences . among themare linear , stochastic volatility and max - moving average processes with regularly varying noise .we give expressions for the extremogram and , if possible , for the corresponding spectral density . in section [ secprop ] ,we give the main results of this paper .we start in section [ subsecperiod ] by showing that the periodogram ordinates of extreme events are asymptotically uncorrelated at distinct fixed or fourier frequencies in the interval .next , in section [ subsecclt ] we show that the periodogram ordinates at distinct fixed or fourier frequencies converge to independent exponential random variables .this property is exploited in section [ subsecsmooth ] to show that weighted averages of periodogram ordinates evaluated at fourier frequencies in a small neighborhood of a fixed frequency yield consistent estimates of the underlying spectral density at the given frequency . in section [ sec6 ] , we give a short discussion of work related to the extremogram or the spectral analysis of sequences of indicator functions .the proofs depend on various calculations involving formulas for sums of trigonometric functions .some of these formulas and related calculations are given in the .it was mentioned in section [ secintrod ] that one needs conditions to ensure that the limits in ( [ eq1 ] ) exist . a sufficient condition for this to hold is _ regular variation _ of the strictly stationary sequence .regular variation is a convenient tool for modeling multivariate heavy - tail phenomena and serial extremal dependence in a time series ; see resnick s monographs , resnick  , basrak and segers , davis and hsing , embrechts _ et al . _ , jakubowski , bartkiewicz _ et al . _ , and the references therein .regular variation is particularly useful for modeling extremes in financial time series ; see basrak _ , mikosch and stric  , davis and mikosch ; cf ._ and the references therein .see also the examples in section [ subsecexam ] .  a random vector with values in for some is _ regularly varying _ if there exists a non - null radon measure on the borel -field of , where , such that here denotes _ vague convergence _ on the borel -field of ; for definitions see kallenberg  , resnick . in this context , bounded sets are those which are bounded away from zero and the radon measure charges finite mass to these sets .then , necessarily , there exists an such that , , for all in the borel -field of .we refer to _ regular variation of with limiting measure and index ._ a multivariate -distributed random vector is regularly varying and the index is the degree of freedom .other well known multivariate regularly varying distributions are the multivariate - and frchet distributions ; see resnick , chapter 5 , in particular section 5.4.2 .we will often use an equivalent sequential version of ( [ eq2 ] ) : there exists such that as and a possible choice of is given by the -quantile of .now , a strictly stationary -dimensional sequence is _ regularly varying _ if the lagged vectors , , are regularly varying with index .of course , the limiting non - null radon measures in ( [ eq2 ] ) now depend on the lag and the normalization in ( [ eq3 ] ) would also change with . in the context of this paperit is convenient to choose the normalizations of the rare event probabilities independently of . in particular, we will use the following relations for , where satisfies , as .these relations are equivalent to the definitions ( [ eq2 ] ) and ( [ eq3 ] ) of regular variation of .now we are in the position to verify that the limits in ( [ eq1 ] ) exist for any borel set bounded away from zero .write and .these sets are bounded away from zero in .if these sets are continuity sets with respect to we obtain from the sequential definition of regular variation of for ,    the results in davis and mikosch were proved under the following mixing / dependence condition on the sequence .the sequence is strongly mixing with rate function .there exist and such that and and and for all ,   condition ( [ eq33 ] ) is similar in spirit to condition ( 2.8 ) used in davis and hsing for establishing convergence of a sequence of point processes to a limiting cluster point process .it is much weaker than the anti - clustering condition of leadbetter which is well known in the extreme value literature ; see leadbetter _ or embrechts _ et al .since we choose such that as , ( [ eq33 ] ) is equivalent to in addition , we also need the following technical condition , using the same notation as in ( m ) .  the sequences , ,  ] , =qx^{-\alpha} ] which in general does not vanish .this measure is in agreement with the asymptotic tail independence conditions of ledford and tawn  .  the mentioned literature focuses on stochastic volatility processes with i.i.d .regularly varying noise with index and stochastic volatility satisfying the moment condition for some .mikosch and rezapur consider regularly varying stochastic volatility processes with index when the sequence is regularly varying with index , for some and they give examples with and for bounded away from zero .the aforementioned comments about mixing also apply in this setting .consider the linear process where is an i.i.d .one - dimensional regularly varying sequence with index and tail balance condition ( [ eq6 ] ) .we choose the coefficients from the arma equation , , where for integers , and the coefficients are chosen such that and have no common zeros and for .it is well known that is regularly varying with index ; see , for example , appendix a3.3 in embrechts _ or mikosch and samorodnitsky .the proofs in the latter references use the fact that , , is regularly varying as a simple consequence of the fact that linear combinations of i.i.d .regularly varying random variables are regularly varying ; see feller , page 278 ; cf .lemma 1.3.1 in .moreover , then it follows from lemma 3.6 in jessen and mikosch that is regularly varying .the vector is also regularly varying with index .this fact follows from an application of a multivariate version of breiman s lemma ( see basrak _et al . _ ) or the fact that linear operations preserve regular variation ; see lemma 4.6 in . since ( [ eq9 ] ) holds a straightforward multivariate extension of lemma 3.6 in yields that is regularly varying for every .  the same arguments leading to the asymptotic tail behavior of ( see , e.g. , appendix a3.3 in embrechts _ et al . _ , mikosch and samorodnitsky ) yield for , } { \sum_{i=0}^{\infty }  [ p ( \psi_i^{+})^\alpha+ q ( \psi_i^{-})^\alpha ] } , \qquad h\ge1.\ ] ] this formula was given in for symmetric when .doukhan , theorem 6 on page 99 , shows that is -mixing , hence strongly mixing , with geometric rate if has a positive lebesgue density in some neighborhood of the expected value of ( provided it exists ) and pham and tran proved the same statement under the condition that has a lebesgue density and a finite moment for some .hence ( [ eqll ] ) , ( m1 ) and ( m2 ) are satisfied under these conditions ; see remarks [ remrm1 ] and [ remrm2 ] .next , we verify condition ( [ eq33 ] ) .we observe that it trivially holds for an -dependent sequence for any integer hence , it is satisfied for any moving average of order , in particular for the truncated sequence .for ease of presentation , we assume .since and are independent we have recall that there exist such that for sufficiently large ; see brockwell and davis , chapter 3 .we have for a positive constant , for every , ( here and in what follows , denotes any constant whose value is not of interest . ) for sufficiently large , we have in view of the uniform convergence theorem for regularly varying functions ( see bingham _ , section 1.2 ) , and the right - hand side converges to zero as .thus we proved that ( m ) , ( m1 ) and ( m2 ) hold for arma processes if the noise has some lebesgue density .if relation ( [ eq10 ] ) bears some similarity with the autocorrelation function of given by . replacing in ( [ eq5 ] ) by , one obtains the well - known spectral density of a causal arma process ( up to a constant multiple ) : ,  ] the same remark applies to the quantities evaluated at fourier frequencies .we start by calculating the asymptotic covariances .any of the covariances can be written in the form \\ & = & \frac{2m_n}{n } \biggl(\sum_{1\le t = s\le n}+ \sum _ { 1\le s\neq t\le n } \biggr ) \bigl(p_{|s - t|}-p_0 ^ 2 \bigr ) f_1 ( \lambda s ) f_2( \omega t ) \\ & = & j_1+j_2 , \end{aligned}\ ] ] where and are cosine or sine functions and we estimate separately for each possible combination of sine and cosine functions .we start with and .then , if are fourier frequencies , so are and therefore =0.\end{aligned}\ ] ] if are fixed frequencies , we conclude from ( [ eqb ] ) that the sum on the right - hand side is bounded .hence , .  for get .\end{aligned}\ ] ] if are fourier frequencies , so are and then the right - hand side vanishes unless . however , if the second sum vanishes and the first sum is bounded. therefore , .if are fixed it follows from ( [ eqa ] ) that the sum on the right - hand side is bounded and therefore .for we have .\end{aligned}\ ] ] the same arguments as above show that both for fourier and fixed frequencies .next , we consider .we start with .if is a fourier frequency , we have .hence , by ( [ eqe ] ) , \\ & = & - \frac { 2m_n}{n } \sum_{h=1}^{n-1 } \bigl(p_{h}-p_0 ^ 2\bigr)\sin({\lambda}h).\end{aligned}\ ] ] by definition of strong mixing , .then , by condition ( m ) , the same argument applies for a fixed frequency since the expressions in ( [ eqe ] ) are bounded for every and .if are fixed frequencies , we conclude from ( [ eqf])([eqh ] ) and condition ( m ) that there exist constants such that now we consider the case of two distinct fourier frequencies .we start with and .if and are bounded away from zero , we can use the argument for general distinct frequencies .now assume that say . since are fourier frequencies a glance at ( [ eqf])([eqh ] )shows that one has to find suitable bounds for if taylor expansions for the nominator and the denominator show that the right - hand side is bounded by . if bound the nominator by 1 and taylor expand the denominator to conclude that the right - hand side is bounded by for some constant as well .then , by ( [ eqf ] ) , for fixed , .\ ] ] the right - hand side vanishes by virtue of condition ( m ) , first letting and then . the case of small , say , can be treated analogously .the remaining cases and can be treated in the same way by exploiting ( [ eqg ] ) and ( [ eqh ] ) .now we turn to the asymptotic variances .we restrict ourselves to for fixed ; the variance of and the case of fourier frequencies can be treated analogously .write  we have .\end{aligned}\ ] ] for any frequency bounded away from zero and , the relation holds .moreover , .}\ ] ] therefore , since and is continuous , we have the statement of the theorem then follows if we can show that .we observe that in view of condition ( [ eqweight ] ) it suffices to show that and  j_1\ne j_2 ] may simplify : if or , =p_h ] ; if , =p_{t_1-t_2,h - t_1+t_2} ] .if we take into account these different cases , we obtain applying ( [ eqhelp1 ] ) , the mixing condition ( m2 ) and lemma [ lemaa ] imply that as regards , apply ( [ eqhelp2 ] ) and split the -index set into and .then ( m2 ) and lemma [ lemaa ] imply that now consider . abusing notation ,we will write instead of .introduce the index sets now introduce the mixing coefficients and use lemma [ lemaa ] : \\ & & { } + c \biggl [ \frac{m_nr_n}{n } m_n \sum _ { h_1=r_n+1}^{n-1}\xi_{h_1}+ \frac{r_n}{n}(m_np_0)^2 \bigl(\sin(\lambda_{j}/2)\bigr)^{-2 } \biggr ] \\ & & { } + c \biggl [ m_n^2 \sum_{h_1=r_n+1}^{n-1 } \xi_{h_1}+(m_np_0)m_n\sum _ { h_2=r_n+1}^{n-1}\xi_{h_2}+\frac{1}{m_n}(m_np_0)^3 \bigl(\sin(\lambda _ { j}/2)\bigr)^{-2 } \biggr].\end{aligned}\ ] ] the right - hand side vanishes as by virtue of ( m2 ) .the same idea of proof applies to the relation .thus , we showed that .we split the summation over disjoint index sets , depending on the ordering of : , , , , , , , and .consider the index sets ( we recycle the notation here ) we write for short .then \\ & = & \sum_{i=1}^6 j_{2i}.\end{aligned}\ ] ] the terms , , involve probabilities of the form .these terms can be treated in the same way as and shown to be negligible .we omit details .the remaining s contain probabilities of the form .we illustrate how one can deal with these pieces .we start with \\ & \le & c \frac{m_n r_n^3}{n}+ c \frac{m_n^2 r_n^2}{n } \sum _ { h_2=r_n+1}^\infty\xi_{h_2}+c \frac{r_n^2}{n } ( m_n p_0)^2 \bigl(\sin ( \lambda_j/2)\bigr)^{-2}.\end{aligned}\ ] ] in the last step we used lemma [ lemaa ] .the right - hand side in the latter relation converges to zero in view of the assumptions on and ( m2 ) .the remaining expressions which contain probabilities over index sets such that or can be shown to be negligible by using similar arguments .we omit details .those sums which contain probabilities over index sets such that are most difficult to deal with .the corresponding bounds follow from the next lemma .let  ] .they were particularly interested in the case of asymptotic independence when and ^ 2\to1 ] as alternative measure of serial extremal dependence in the case when the extremogram vanishes . considered lag - dependent tail dependence coefficients under regular variation conditions on the process .these coefficients can be interpreted as special extremograms .hill showed a pre - asymptotic functional central limit theorem for the sample extremogram of univariate time series over classes of upper quadrants .his mixing and domain of maximum domain of attraction are not directly comparable with strong mixing and regular variation od stationary sequence s but the results are similar in spirit to theorem 3.2 in davis and mikosch , where multivariate time series can be treated but uniform convergence over certain classes of sets was not considered .recently , various articles on the spectral analysis of indicator functions and their covariances based on a strictly stationary time series were written ; see , for example , dette _ et al ._ and the references therein , hagemann , lee and subba rao .the results are similar to those of classical time series analysis .the aforementioned papers do not deal with the spectral analysis of serial extremal dependence .in particular , they do not involve sequences of indicator functions of the form for sets bounded away from zero .therefore , these papers do not need additional conditions such as regular variation of which are typical for extreme value theory and they do not require to consider the normalization of the periodogram but use the classical constants .the present paper focuses on the basic properties of the extremal periodogram .these properties parallel the results of classical time series analysis , but the proofs are different because of the triangular array nature of the stochastic processes . in particular , the calculation of sufficiently high moments necessary to prove central limit theorems becomes rather technical .the central limit theorem for the smoothed periodogram is still an open question .the ( smoothed ) periodogram as such contains information about the length of random cycles between extremal events .but it also opens the door to the methods and procedures of classical time series analysis , including the rich theory related to the integrated periodogram with applications to parameter estimation ( e.g. , whittle estimation ) , goodness - of - fit tests , change point analysis , detection of long - range dependence effects and other problems .the solution to these problems is again rather technical and will be treated in future work .equations ( [ eqa ] ) and ( [ eqb ] ) are given in gradshteyn and ryzhik , 1.341 on page 29 ; ( [ eqc ] ) and ( [ eqd ] ) are 1.352 on page 31 ; and ( [ eqe1 ] ) and ( [ eqe2 ] ) are listed as 1.353 on page 31 . for any and ,the following identities hold using these formulas , direct calculation yields for any frequency , \nonumber \\[-10pt ] \\[-10pt ] \nonumber & & \quad= \sum_{s=1}^{n - h } \sin(2\lambda s+{\lambda}h ) = \frac{\sin({\lambda}n)\sin({\lambda}(n - h+1))}{\sin{\lambda}}-\sin({\lambda}h).\vspace*{-1pt}\end{aligned}\ ] ] for distinct frequencies we then obtain \nonumber \\[-2pt ] & & \quad= 0.5 \sum_{s=1}^{n - h } \bigl[\sin\bigl ( ( { \lambda}+{\omega})s + { \omega}h\bigr)-\sin\bigl(({\lambda}-{\omega})s -{\omega}h\bigr)\bigr ] \nonumber \\[-2pt ] & & \qquad{}+0.5 \sum_{s=1}^{n - h } \bigl[\sin\bigl ( ( { \lambda}+{\omega})s + { \lambda}h\bigr)-\sin\bigl(({\lambda}-{\omega})s + { \lambda}h\bigr)\bigr ] \nonumber \\[-2pt ] & & \quad= -\sin({\omega}h ) \nonumber \\[-9pt ] \\[-9pt ] \nonumber & & \qquad{}+0.5\frac{\sin((n - h+1)({\lambda}+{\omega})/2)}{\sin(({\lambda}+{\omega})/2)}\\[-2pt ] & & \qquad\phantom{+}{}\times \bigl[\sin\bigl({\omega}h + ( n - h ) ( { \lambda}+{\omega})/2\bigr)+\sin\bigl ( { \lambda}h + ( n - h ) ( { \lambda}+{\omega})/2\bigr ) \bigr ] \nonumber \\[-2pt ] & & \qquad { } -0.5 \frac{\sin((n - h+1)({\lambda}-{\omega})/2)}{\sin(({\lambda}-{\omega})/2)}\nonumber\\[-2pt ] & & \qquad\phantom{-}{}\times \bigl[\sin \bigl(-{\omega}h+ ( n - h ) ( { \lambda}-{\omega})/2\bigr)+\sin \bigl({\lambda}h + (n - h ) ( { \lambda}-{\omega})/2\bigr ) \bigr ] , \nonumber\end{aligned}\ ] ] \nonumber \\[-2pt ] & & \quad=  0.5\sum_{s=1}^{n - h } \bigl [ \cos\bigl ( ( \lambda+\omega ) s + \omega h\bigr ) + \cos\bigl((\lambda-\omega)s - \omega h\bigr ) + \cos\bigl((\lambda+\omega)s + \lambda h\bigr)\nonumber \\[-2pt ] & & \hspace*{51pt}{}+\cos\bigl((\lambda-\omega)s + \lambda h\bigr ) \bigr ] \nonumber \\[-2pt ] & & \quad=  - \cos(\omega h ) -\cos(\lambda h ) \nonumber \\[-9pt ] \\[-9pt ] \nonumber & & \qquad{}+0.5 \frac{\sin((n - h+1)(\lambda+\omega)/2)}{\sin((\lambda+ \omega)/2 ) } \\[-2pt ] & & \qquad\phantom{+}{}\times\bigl [ \cos\bigl(\omega h + ( n - h ) ( \lambda+\omega)/2 \bigr ) + \cos \bigl(\lambda h + ( n - h ) ( \lambda+\omega)/2\bigr ) \bigr ] \nonumber \\[-2pt ] & & \qquad { } + 0.5 \frac{\sin((n - h+1)(\lambda-\omega)/2)}{\sin((\lambda- \omega)/2 ) } \nonumber\\[-2pt ] & & \qquad\phantom{+}{}\times\bigl [ \cos\bigl(-\omega h + ( n - h ) ( \lambda- \omega)/2\bigr ) + \cos \bigl(\lambda h + ( n - h ) ( \lambda-\omega)/2\bigr ) \bigr ] , \nonumber\end{aligned}\ ] ] \nonumber \\ & & \quad =  0.5\sum_{s=1}^{n - h } \bigl [ \cos \bigl((\lambda+\omega ) s + \omega h\bigr ) -\cos\bigl((\lambda-\omega)s - \omega h \bigr ) + \cos\bigl((\lambda+\omega)s + \lambda h\bigr)\nonumber \\ & & \hspace*{51pt}{}-\cos\bigl((\lambda-\omega)s + \lambda h\bigr ) \bigr ] \nonumber \\ & & \quad=  0.5 \frac{\sin((n - h+1)(\lambda-\omega)/2)}{\sin((\lambda- \omega)/2 ) } \\ & & \qquad{}\times\bigl [ \cos\bigl(-\omega h + ( n - h ) ( \lambda-\omega)/2\bigr ) + \cos \bigl(\lambda h + ( n - h ) ( \lambda-\omega)/2\bigr ) \bigr ] \nonumber \\ & & \qquad { } -0.5 \frac{\sin((n - h+1)(\lambda+\omega)/2)}{\sin((\lambda+ \omega)/2)}\nonumber\\ & & \qquad\phantom{-}{}\times \bigl [ \cos\bigl(\omega h + ( n - h ) ( \lambda+\omega)/2 \bigr ) + \cos \bigl(\lambda h + ( n - h ) ( \lambda+\omega)/2\bigr ) \bigr ] .\nonumber\end{aligned}\ ] ] next , assume the conditions of theorem [ thmsmoth ] . then a direct application of ( [ eqa])([eqd ] ) yields for the following relations : & & \quad=  n \biggl ( \frac{\sin(x+(n-1))\lambda/2 \sin(n\lambda/2 ) } { \sin(\lambda/2)}\\[-2pt ] & & \hspace*{11pt}\qquad { } -\frac{\sin(x+r_n \lambda/2 ) \sin((r_n + 1)\lambda/2 ) } { \sin(\lambda/2 ) } \biggr ) \\[-2pt ] & & \qquad{}+ \sin(x ) \biggl ( \frac{n\sin((2n-1)\lambda/2)}{2\sin(\lambda /2 ) } -\frac{(r_n+1)\sin((2r_n-1)\lambda/2)}{2\sin(\lambda/2)}\\[-2pt ] & & \hspace*{43pt}\qquad { } - \frac{\cos((r_n+1)\lambda)-\cos(n\lambda)}{4(\sin(\lambda /2))^2 } \biggr ) \\[-2pt ] & & \qquad { } + \cos(x ) \biggl ( \frac{n\cos((2n-1)\lambda/2)}{2\sin(\lambda /2 ) } -\frac{(r_n+1)\cos((2r_n-1)\lambda/2)}{2\sin(\lambda/2)}\\[-2pt ] & & \hspace*{43pt}\qquad { } - \frac{\sin(n \lambda)-\sin((r_n+1)\lambda)}{4(\sin(\lambda /2))^2 } \biggr)\end{aligned}\ ] ]   [ lemaa ] under the assumptions of theorem [ thmsmoth ] the following relations hold uniformly for , as , this section , we calculate the spectral density for an process and the set .the process is given as the stationary causal solution to the difference equation where and . from brockwell and davis , ( 2.3.3 ) , we obtain the coefficients of the linear process representation of ( cf .( [ eq8 ] ) ) : we assume that is an i.i.d . regularly varying sequence with index .+ _ the case , , . _ a direct application of ( [ eq10 ] ) yields that  and write ( see appendix [ secappendix ] )  then where the corresponding spectral density is given by _ the case , , ._ in view of ( [ eq10 ] ) , we have _ the case , , . _ if for integer the summand in ( [ eq10 ] ) vanishes for .thus , } .\ ] ] for , } { p + \sum_{i=1}^{\infty }  [ p |\psi_{2i-1}|^\alpha+ q |\psi_{2i}|^\alpha ] } .\ ] ] define .then , where the corresponding spectral density is \\ & & { } + 2c^{(6)}_{\alpha } ( \phi , \theta ) l^{(2 ) } ( \infty , 0 , \alpha , 2\lambda).\end{aligned}\ ] ] _ the case , , . _ if for integer the summand in ( [ eq10 ] ) vanishes for .thus , for , } { \sum_{i=0}^{\infty }  [ p|\psi _ { 2i}|^\alpha+q|\psi_{2i+1}|^\alpha ] } .\ ] ] define .then where would like to thank the reviewers of our paper for careful reading and comments , in particular for pointing out several useful references .thomas mikosch s research is partly supported by the danish research council ( fnu ) grants 09 - 072331 `` point process modelling and statistical inference '' and 10 - 084172 `` heavy tail phenomena : modeling and estimation '' .the research of yuwei zhao is supported by the danish research council grant 10 - 084172 .

Abstract: the _ extremogram _ is an asymptotic correlogram for extreme events constructed from a regularly varying stationary sequence . in this paper , we define a frequency domain analog of the correlogram : a periodogram generated from a suitable sequence of indicator functions of rare events . we derive basic properties of the periodogram such as the asymptotic independence at the fourier frequencies and use this property to show that weighted versions of the periodogram are consistent estimators of a spectral density derived from the extremogram .


it is well known that due to their accuracy and efficiency , spectral methods have great advantages over local methods in solving applicable scientific and engineering problems . given a hyperbolic system of nonlinear conservation laws with initial condition its solution may not exist in the classical sense because of possible discontinuities . the direct use of spectral methods to this problem will encounter the gibbs oscillations , which lead to blow - up in the time integration .therefore , it has been of great interest in making spectral methods applicable to the hyperbolic conservation law systems in the past two decades .the expectation , as point out by gottlieb , is that the use of spectral methods will enable us not merely to capture the shock , but also to capture the delicate features and structures of the flow .obviously , spectral approaches will be extremely valuable to a class of aerodynamical problems that involve the interaction of both turbulence and shock .such an interaction is some of the most challenging problems in computation fluid dynamics .the main objective of previous studies is to recover smooth solutions from those that are contaminated by gibbs oscillations and , meanwhile , to improve the rate of convergence . for shock - capturing ,many up - to - date local methods , such as weighted essentially non - oscillatory ( weno ) scheme , central schemes , arbitrary - order non - oscillatory advection scheme , gas kinetic , and image processing based schemes , perform well .the success of these local shock - capturing schemes lies in their appropriate amount of intrinsic numerical dissipation , which is introduced either by explicit artificial viscosity terms , or by upwinding , or by appropriate local average ( non - oscillatory central schemes ) , or by relaxation .the characteristic decomposition based on roe s mean matrix can also be considered as a local averaging on the jacobian matrix . in his recent work, leveque outlined the relation between some approximate riemann solvers and relaxation schemes .the relative amount of numerical dissipations may explain why the local characteristic decomposition is not necessary in low - order methods while it seems to be indispensable in high - order methods .however , when these local methods are used in the cases where flow structures with fine details are needed to be resolved together with shocks , numerical dissipations are usually found to be so large that they smear the fine details .the spectral methods , on the contrary , contribute very little numerical dissipation and dispersion in principle when used to approximate spatial derivatives .nevertheless , when they are applied to the approximation of spatial derivatives on a domain containing discontinuities , the gibbs oscillations again have to be suppressed by appropriate means , for which there are two general routines : ( 1 ) explicit artificial viscosity , e.g. spectral viscosity ( sv ) method proposed by tadmor and ( 2 ) filtering , which is the central issue of the present study .  at present, most filters are constructed in the spectral domain and they can be called spectral filters ._ suggested a list of typical spectral filers , i.e. lanzos filter , raised cosine filter , sharpened raised cosine filer and exponential cutoff filter .more sophisticate and effective filters in spectral domain are vandeven s -th order filter , cai _ et al ._ s sawtooth function and gottlieb and tadmor s regularized dirichlet function .  apart from methods that are implemented in the spectral domain ,filters in the physical domain are also developed . to design an appropriate filter in the physical domainis generally more difficult than in the fourier domain .a straightforward procedure is to make use of numerical dissipation partially contained in _ high - order _ shock - capturing schemes , such as the eno scheme , where , actually , numerical dissipations were introduced both in the fourier domain ( via an exponential filter ) and in the physical domain ( via eno polynomial interpolation ) .such a strategy was generalized by yee _ to the so - called characteristic filter method ( with finite difference or finite volume ) which has been successfully applied to shock - capturing and turbulence simulations .another procedure was initiated by gottlieb by using the gegenbauer polynomial to resolve the oscillatory partial fourier summation .some promising numerical results from the filter approach can be found in refs . .nevertheless , some prominent issues may hinder the success of filtered spectral methods for practical numerical computations involving shocks .in our view , there are two very important issues that are vita to the success of a filtered spectral scheme .the first issue is how to select optimal filters for shock - capturing .the issue is very complex and can not have a unique answer at this point because there are so many properties to mind , such as flatness , ripple , filter length , effective frequency range and length of transition band , to name only a few . loosely speaking ,it is desirable to have filters that are free of dispersion errors , flat while having very small transition band , short in length while having high resolution .moreover , what adds to the complexity is that solutions to different hyperbolic conservation law systems may have different fourier spectral distributions .consequently , one faces the difficulty that an optimal filter for a problem may not even work for another one .therefore , it is desirable to have a filter which accounts for this change by an adjustable parameter .the second issue also concerns the change of solution , but is due to the time evolution for a given hyperbolic conservation law system .as the fourier spectral distribution of a given problem changes with time , the question is how to implement the filter ?it will be too dissipative for many systems if a filter is applied at each time step of the integration .therefore , an adaptive implementation , which is controlled by a sensor during the time integration , is appropriate .given the complexity , it is unlikely that there will be an ideal solution to these issues in the immediate future .obviously , these challenging and interesting problems call for the further study of spectral filter approaches .the objective of this work is to construct , analyze and implement a class of high - order lowpass filters which can be used either in the fourier domain or in the physical domain .these filters are constructed within the framework of the discrete singular convolution ( dsc ) algorithm and they are used to suppress gibbs oscillations arising in the fourier pseudospectral method ( fpm ) .such lowpass filters have a wide effective wavenumber range that makes it possible to capture the fine flow structures , which is a desirable objective of spectral methods for hyperbolic conservation law systems .it is this effective wavenumber range that controls the resolution of the overall method . in our design, this effective range can be varied according to the resolution requirement by a parameter .different dsc kernels have different magnitude responses and adjustability in the fourier domain , which in turn influence the accuracy and resolution .the dsc filters used in this work were originally designed in a conjugate filter oscillation reduction ( cfor ) scheme . by `conjugate filters ' it means that the effective wavenumber range of the lowpass filter is largely overlapped with that of the highpass filter which is used for the approximation of spatial derivatives .the resulting fourier domain algorithm is essentially a windowed fourier pseudospectral method for shock - capturing .the rest of this paper is organized as follows . in section ii , a brief introduction will be devoted to the construction of the dsc lowpass filters followed by a discussion of two types of implementations of these filters to spectral methods .extensive numerical experiments are carried out in section iii , including burgers equation , shock - tube problems , shock - entropy wave interaction , shock - vortex interaction . the shock - capturing ability and the high resolution feature of the present methodare addressed . concluding remarks end this paper .in the context of distribution theory , a singular convolution is defined by where is a singular kernel and is an element of the space of test functions .interesting singular kernels include that of hilbert type , abel type and delta type . the former two play important roles in the theory of analytical functions , processing of analytical signals , theory of linear responses and radon transform . since delta type kernels are the key element in the theory of approximation and the numerical solution of differential equations , we focus on singular kernels of the delta type where superscript denotes the - order ` derivative ' of the delta distribution , with respect to , which should be understood as generalized derivatives of distributions . when , the kernel is important for the interpolation of functions . in this work ,only the case of will be involved .one has to find appropriate approximations to the above singular kernel , which can not be directly realized in computers .to this end , we consider a sequence of approximations where is a parameter which characterizes the approximation with the being a generalized limit . among various candidates of approximation kernels , regularized shannon kernel ( rsk ) is frequently used due to its close link to the shannon sampling theorem . in this formula , is the grid spacing and determines the width of the gaussian envelop . for a given , the limit of reproduces the delta kernel ( distribution ) . with the rsk, a function can be approximated by a discrete convolution where are a set of discrete grid points which are centered at , and is the computational bandwidth , or effective kernel support , which is usually smaller than the computational bandwidth of the spectral method - the entire domain span . equation ( [ eq : dsc ] ) is often referred as a dsc and the rsk is used as the prototype of a dsc lowpass filter .the feasibility of the dsc - rsk as lowpass filters in numerical computation is based on an important observation : by adjusting the ratio ( ) of the regularizer , one can control the effective wavenumber region and thus , control the resolution of the lowpass filter .we plot the frequency response profile of the dsc - rsk lowpass filter in fig .[ dsckernel ] .a distinct property of the rsk lowpass filter is that there is no ripples in its passband ; thus the application of this filter will not impart the low frequency .in contrast , some lowpass filters constructed with certain polynomials may have ripples in their passband , which would rule out their application to numerical integrations , especially for long time evolution , where the solution may be distorted after repeated filtering .the maximum effective wavenumber ( frequency ) range of the dsc - rsk is determined by the ratio , which in turn is determined by computational bandwidth .the philosophy is simple since with the increase of , the gaussian window should also be enlarged to fully utilize the stencil .the optimal values at different can be estimated and they characterize the effective wavenumber range . the larger the optimal is , the wider the effective wavenumber range will be .when value is chosen larger than the optimal value for a given , the accuracy of the resulting filter will be degraded by oscillations .hence we do not recommend the use of beyond the optimal value unless in some circumstances where the filer is utilized to stabilize a long - time integration .we denote the proposed dsc - rsk lowpass filter in the physical domain  ] and the flow field is initialized with where and are the amplitude and the wave number of the entropy wave before the shock .this problem is significant due to its relevance to the interaction of shock - turbulence .spectral methods , by nature of their high accuracy and negligible dispersive and dissipative errors , are ideally suited to the numerical study of turbulence and have already been widely applied . herewe would like to argue the feasibility of the present spectral method to the shock - turbulence interaction .in our test , we vary the wavenumber of the pre - shock wave while keeping its amplitude unchanged ( ) . as the wavenumber increases , the problem becomes more and more challenging because the amplified high - frequency entropy waves after the shock are always mixed with spurious oscillations .as the wavenumber increases , it is very difficult to distinguish the high - frequency waves from the spurious oscillations . a low - order scheme may dramatically damp the transmitted high frequency waves .even some popular high - order schemes also encounter the difficulty in preserving the amplitudes of the entropy waves due to their excessive dissipation with a given mesh size .therefore , a successful shock - capturing method should be able to eliminate gibbs oscillation while capturing the shock and preserving the high - frequency entropy waves .  in our test, we let the shock move from to . for the purpose of comparison with the previous results ,we only give the results at the interval  ] and the flow field is initialized with where =0.2 and =5 .[ shu](a ) and fig .[ shu](b ) show the solution of the density with 128 and 256 cells , respectively .the ` exact ' solution here is the solution computed by the third - order eno with 1200 cells . as evident in the solution ,the complicated flow field behind the shock is well resolved and the shock remains sharp .however , small oscillations near the shock still exist due to the fact that we do not attempt to locate this shock intentionally .if we predict the position of the shock and apply a local filter near the shock at the final time step , it is believed that more satisfactory results can be obtained .   in two dimensions ,the euler equation for gas dynamics in a vector notation takes the conservation form with  where is the 2d fluid velocity and is used in all computations .  _ _ as the first example , we use our spectral scheme to study the interaction between a normal shock and a weak entropy wave which makes an angle against the -axis . if , we have essentially the 1d problem ( see example 7 ) .the initial conditions are defined as follows : the right state of the shock is given as =(1 , 0 , 0 , 1 ) with a given shock mach number .we add a small entropy wave to the flow on the right of the shock which is equivalent to changing only the density of the flow on the right of the shock . in our problemwe change the density in the right state of the shock by multiplying  with and being the amplitude and wavenumber , respectively . in order to carry out a long time integration and to enforce periodic boundary condition in -direction, the computational domain is taken to be \times [ 0,\f{2\pi}{\kappa \sin \theta}] ] , and comparing them with the amplitude predicted by the linear analysis , which is 0.08744786 .we use 513 grid points in the -direction and 32 grid points in the -direction in our test . in order to avoid peak / valley losses of the entropy waves due to insufficient grid points near these positions , we interpolate our final results to a denser grid when plotting them . fig .[ enshock ] displays the amplitudes of the amplified entropy waves .obviously , the compressed entropy waves are well resolved in the post - shock regime even though small oscillations remain near the location of the shock .it is believed that such a trivial flaw is acceptable .this example has further demonstrated the capability of our spectral scheme for capturing small scale waves in the presence of a shock .  __ this model problem describes the interaction between a stationary shock and a vortex .it has many potential applications and has been treated by many researchers using various techniques .early numerical studies in this area using spectral methods focused primarily on shock - fitting techniques . however , for the cases in which a vortex causes a shock to deform at the point of bifurcation , it is generally difficult for a shock - fitting method to apply. this problem was successfully simulated in refs . by using the chebyshev spectral method . in this paper, we also adopt this model to access the capability of the proposed spectral shock - capturing method .this problem is defined as follows .the initialization is imposed on the domain \times [ 0 , 1] ] with the center of the vortex being initially located at , the geometrical center of the computational domain .since there is no presence of shock in this problem , spectral method on its own can already provide excellent results if the time integration period is small enough .as such , the lowpass filter is not needed in an initial time period . however , as time progresses , errors would accumulate rapidly and the computation could become unstable if the lowpass filter is not activated to efficiently control the dramatically nonlinear growth of the errors . in our numerical experiment which is not shown here ,the computation blows up at around =13 if no filter is used .two experiments are conducted in this study .one is to examine the accuracy of our spectral code and the other is to investigate the stability of our scheme for a long time integration . for the first experiment , we compute the solution of at the time =2 using three sets of meshes ( ) . to make the spatial error dominant, we optimize the cfl number to be 0.01 .  two error measures , and , have been used . to be consistent with the literature , two errors used in this problemare defined as where is the numerical result and the exact solution .note that they are not the standard definitions .the errors for the density with respect to the exact solution are listed in table i in which we also list the results of some other schemes for a comparison . as the table highlights , the accuracy of the fourier spectral method is comparative to that of the dsc method , while both fourier spectral and dsc methods are much more accurate than the fourth - order accurate , conservative central scheme ( c4 ) .our next numerical experiment concerns the performance of our spectral code for a long time integration . here ,we compute the solution of at the time =100 , 200 , 400 , 600 , 800 and 1000 , with the grid .note that we choose the gradient of the solution in this case .we list the and errors for the density in table ii .it can be seen that our spectral code is highly accurate even after such a long time evolution .the results indicate that our spectral scheme together with the dsc lowpass filter is reliable for the numerical approximation of conservation laws ._ _ finally , we consider the problem of a supersonic flow past a cylinder .this example is used to mainly examine the ability of our dsc filter spectral scheme in handling a non - rectangular domain .the physical domain is on the plane , and the computational domain is chosen to be [ 0 , 1 ] [ 0 , 1 ] on the plane .the mapping between the computational domain and the physical domain is where we take , and .the flow field is initialized with a mach 3 shock moving from the inlet , i.e. , , toward the cylinder .the state in front of the shock is given as .we impose a reflective boundary condition on the surface of the cylinder , i.e. , , and the outflow boundary condition at and . in our simulation , a uniform mesh of in the computation domain[ cylin](a ) depicts a diagram of the mesh ( drawing every other grid line ) in the physical domain . like the case on a rectangular domain , we double the computational domain in both and directions when we approximate the first order derivative of the flux using the fourier spectral method .the numerical result given in terms of pressure contours ( obtained at =4.5 ) is shown in fig .[ cylin](b ) . in summary, our spectral scheme can be easily applied to the nonrectangular domain , as long as the domain can be smoothly transformed to the rectangular one .in this work , we introduce the use of discrete singular convolution ( dsc ) filters in the fourier pseudospectral method ( fpm ) for the simulation of hyperbolic conservation law systems .the fast fourier transform ( fft ) is used as the basic scheme , while dsc low - pass filters are used to suppress unphysical oscillations .the dsc filters are implemented in either the fourier domain or the physical domain , depending on the physical problem under consideration .the fourier domain implementation is easy and cost efficient , whereas the physical domain implementation allows more flexibility to handle some special boundary conditions . a sensor technique developed in the conjugate filter oscillation reduction ( cfor ) scheme adopted in the present work to appropriately activate the dsc filter .the fourth - order runge - kutta scheme is utilized for the time integration .extensive numerical experiments are considered to validate the proposed approach and to demonstrate its usefulness .excellent numerical results are obtained for all problems examined .the dsc filters turn out to be very effective in removing the gibbs oscillations produced by the spectral approximation while maintaining a high accuracy .moreover , it is also highlighted that these filters can dramatically improve the stability of the spectral method for the long time integration .the main features of proposed method are the follows : + ( 1 ) the basic scheme , the fpm is of spectral accuracy , which is important for problems that require not only capturing shock waves , but also resolving fine flow structures .+ ( 2 ) the use of fft endows the method high efficience , which is desirable for large scale problems in science and engineering .+ ( 3 ) numerical dissipation of the dsc filters is adjustable for a given problem , which endows the proposed method the ability to handle problems that are sensitive to numerical dissipation .the interaction of turbulence and shock is arguably a problem of this kind .+ ( 4 ) the fourier domain algorithm can be regarded as a windowed fourier pseudospectral method for shock - capturing .the alternative use of the dsc filter in the physical domain makes the proposed method feasible to problems requires special boundary conditions , e.g. , reflective boundary condition .+ ( 5 ) the proposed scheme by - passes finding the characteristics which is required in many other shock - capturing schemes .+ ( 6 ) the use of the method in higher dimensions is straitforward .it is note that the state of art shock - capturing schemes , such as weno and central schemes , do not depend on any adjustable parameter . to make the proposed scheme robust and user friendly, future work will focus on developing sophisticated sensors so that the filter parameter will be chosen automatically for each given problem .we believe that this can be accomplished by analyzing the fourier power spectrum distribution of the approximate solution at selected time steps .s. abarbanel , d. gottlieb and e. tadmor , spectral methods for discontinuous problems , in _ numerical methods for fluid dynamics _ , ( k. morton and m. baines eds . ) , ( oxford university press , new york , 129 - 153 1986 ) .d. gottlieb , m. y. hussaini and s. a. orszag , introduction : theory and applications of spectral methods , in _ spectral methods for partial differential equations _ , r. g. voigt , d. gottlieb and m. y. hussaini eds . , ( siam , philadelphia , 1984 ) .  d. gottlieb and e. tadmor , recovering pointwise values of discontinuous data within spectral accuracy , in _ progress and supercomputing in computational fluid dynamics _ , earll m. murman and saul s. abarbanel eds . , ( birkh user , 357 - 375 1985 ) .t. grahs and t. sonar , entropy - controlled artificial anisotropic diffusion for the numerical solution of conservation laws based on algorithms from image processing _j. vis . commun .image r. _ , * 13 * , 176 - 194 ( 2002 ) .e. f. toro , r. c. millington and v. a. titarev , ader : arbitrary - order non - oscillatory advection scheme , _ proc .8th international conference on non - linear hyperbolic problems _ , ( magdeburg , germany , march , 2000 ) .g. w. wei , y. b. zhao and y. xiang , discrete singular convolution and its application to the analysis of plates with internal supports .i. theory and algorithm , _ int .j. numer . meth ._ , * 55 * , 913 - 946 ( 2002 ) .s. y. yang , y. c. zhou , and g. w. wei , comparison of the discrete singular convolution algorithm and the fourier pseudospectral method for solving partial differential equations , _ comput ._ , * 143 * , 113 - 135 ( 2002 ).             .advective 2d isentropic vortex . and errors for the density at =2 .the cfl number is 0.01 for all schemes .c4 denotes the fourth - order accurate , conservative centered scheme . [ cols="^,^,^,^,^",options="header " , ]

Abstract: a class of high - order lowpass filters , the discrete singular convolution ( dsc ) filters , is utilized to facilitate the fourier pseudospectral method for the solution of hyperbolic conservation law systems . the dsc filters are implemented directly in the fourier domain ( i.e. , windowed fourier pseudospectral method ) , while a physical domain algorithm is also given to enable the treatment of some special boundary conditions . by adjusting the effective wavenumber region of the dsc filter , the gibbs oscillations can be removed effectively while the high resolution feature of the spectral method can be retained . the utility and effectiveness of the present approach is validated by extensive numerical experiments .   u   * key words * : hyperbolic conservation laws ; fourier pseudospectral method ; discrete singular convolution filters ; gibbs oscillations


regularization through noise has regained focus recently in the training of supervised neural networks . randomly dropping out units while performing backpropagation has been shown to consistently improve the performance of large neural networks .stochastic pooling , where a set of input units are gated based off their activations , has also been shown to improve performance in convolutional nets over noiseless max and average pooling .the role of input noise in training unsupervised networks has also been extensively explored in recent years . injecting noise into the input layer of autoencoders has been shown to yield useful representations in these denoising autoencoders .  motivated by the success of noise injection at the input layer in autoencoders , and at the hidden layers in supervised learning settings, we systematically explore the role of noise injection at all layers in unsupervised feature learning models .as we shall see , this provides a unified framework for unsupervised learning based on the principle that hidden representations should be robust to noise .this yields an extension of prior methods for regularizing autoencoders that we call the noisy autoencoder ( nae ) . for certain types of naes , we are able to marginalize out the noise , and derive a set of penalties that relate noise injection with contractive autoencoders , sparse autoencoders , dropout , and ica .experiments on mnist and cifar-10 validate the effectiveness of noisy autoencoders at learning useful features for classification .building upon the recent success of dropout , we also experiment with further supervised fine - tuning of naes in which the noise is also injected at the supervised stage .we show that training noisy autoencoders with dropout noise , and supervised fine - tuning with dropout noise , allows us to waste less capacity in larger autoencoder networks .we also show that purely supervised training with additive gaussian noise beats dropout on mnist .we compare the effect of these different types of noise , and argue that these results point to an interaction between noise in unsupervised and supervised learning that may be more complex .these results suggest that we may be able to optimize stacked learning strategies by introducing different types of noise for the unsupervised pretraining relative to the supervised fine - tuning .an autoencoder is a type of one layer neural network that is trained to reconstruct its inputs . in the complete case ,this can trivially be accomplished using an identity transformation . however , if the network is constrained in some manner , then the autoencoder tends to learn a more interesting representation of the input data that can be useful in other tasks such as object recognition .the autoencoder consists of an encoder that maps inputs to a hidden representation : , and a decoder that maps the hidden representation back to the inputs : .the composition of the encoder and decoder yield the reconstruction function : .the typical training criterion for autoencoders is minimizing the reconstruction error , with respect to some loss , typically either squared error or the binary cross - entropy .denoising autoencoders ( daes ) are an extension of autoencoders trained to reconstruct a clean version of an input from its corrupted version .the denoising task requires the network to learn representations that are able to remove the corrupted noise .prior work has shown that daes can be stacked to learn more complex representations that are useful for a variety of tasks .the objective for the dae can be written as } ] .when using the nae to extract features or perform denoising on testing data we can compute the expectation of the noisy hidden activation or reconstruction by sampling from the nae . however , this can be prohibitively costly on large datasets . instead , we can approximate the expectation by scaling each of the corrupted variables by their expectation as in dropout . in practicewe only use noise where the corruption of the input variable does not alter the mean so that no scaling is needed .the test - time hidden representation and reconstruction are then computed in the same way as the vanilla autoencoder .     due to the parameterization of noise in the noisy autoencoder, there are many possible choices of noise , that will yield equivalent effective noise on the reconstruction .in particular , we can always rewrite a nae so that the only source of noise is on the hidden activations . to analyze the effect of introducing noise before the encoder nonlinearity, we perform a first - order taylor expansion of the encoder function : . thus , for small noise , adding noise to the linear encoder part is equivalent to having hidden noise : .if the input noise is gaussian with some covariance , then the equivalent hidden unit input noise is gaussian with covariance . if the singular values of are sufficiently small , then we can use the above result to get the effective hidden noise : .if the input noise is dropout ( multiplicative bernoulli ) , then we can use the result from to approximate as a gaussian with some mean and diagonal covariance .these simple relationships allow us to identify the covariance structure of noise at the hidden unit activations that corresponds to input and hidden unit input noise .   to understand the impact that these additional sources of noise have on autoencoder training we analyze a simplified nae .we assume that the decoder is linear ( ) , the loss function used is squared error , and that all the corruptions are applied independently to each neuron .this allows us to exactly marginalize out the noise on the hidden unit activations using a result from : } = \| x-r(x)\|^2 + { \mathrm{tr}(ww^t{\mathrm{var}(\tilde{h}|h)})}\ ] ] we can then apply the approximation results from above to yield cost functions corresponding to marginalizing out each different type of noise independently : \\ { { c_\mathcal{h}}}(x ) & = \textstyle\sum_{i=1}^d { \mathrm{var}({\tilde{h}}_i|h ) } \|w_i\|^2 \\ { { c_\mathcal{z}}}(x ) & \approx  \textstyle\sum_{i=1}^d { \mathrm{var}({\tilde{z}}_i|z ) } ( f'(w_i^tx ) \|w_i\|)^2\\ { { c_\mathcal{i}}}(x ) & \approx \textstyle\|ww^t{\mathrm{diag}(f'(wx))}{\mathrm{var}({\tilde{x}}|x)}\|_f^2\end{aligned}\ ] ] where is the number of hidden units , and is the column of .note that the penalty is exact , while the penalties for noise at the input and hidden unit inputs will only be accurate when the variance of their noises are small .these penalties allow us to relate injection of noise in a nae with regularization penalties from other regularized autoencoders .the noisy autoencoder and associated marginalized noise penalties provide a framework for comparing many types of regularized autoencoders , and help to explain the utility of injecting noise into more general neural networks .the marginalized input noise penalty from the noisy autoencoder provides an intuition in the success of * denoising autoencoders*. if we use tied weights , and additive gaussian corrupting noise with variance then the penalty becomes : this penalty encourages the hidden units to learn orthogonal representations of the input , and provides a contractive - like penalty on individual filters .a similar type of penalty is found when learning overcomplete representations with reconstruction ica , where they use to encourage a diverse set of filters .when stacking denoising autoencoders , we end up with a similar structure to a noisy autoencoder . in the case of building a two layer network , the standard practice is to first train a denoising autoencoder on the input , then compute the first layer representation by encoding the clean inputs .training a denoising autoencoder on the second layer will corrupt the first layer representation , but will not impact the prior encoding model learned for the first layer .in contrast , training a nae with noise on the hidden unit activations allows for the first layer representation to learn to be robust to noise in its hidden representation . *contractive autoencoders * aim to learn a representation that is insensitive to small changes in the input space .they penalize the frobenius norm of the jacobian of the encoder function , .if we inject additive white gaussian noise at the hidden inputs , then we recover the same penalty .alternatively , we can inject additive noise with covariance at the hidden unit activations and recover the exact penalty .this result has been previously reported in , and motivated the contractive penalty .  *sparse autoencoders * force all hidden units to have similar mean activations. we can not directly relate this penalty to a form of noise , but we can recover a penalty that encourages sparsity on hidden unit activations . if we inject additive gaussian noise on the activations of the hidden units with variance equal to the uncorrupted hidden unit activation then the marginalized noise penalty becomes : . if activations are non - negative then this penalty will force many of the hidden unit activations to zero .we note that experimental results from neuroscience have shown that cortical neurons exhibit poisson - like noise statistics with a fano factor of 1 .dropout is a simple regularizer used in training neural networks that applies multiplicative bernoulli noise to all units in a neural network .this noise has been shown to effectively regularize deep neural networks , and has been linked to preventing co - adaptation of neurons and model averaging .the primary motivation for naes was the success of dropout in improving generalization performance .we can analyze the effect of dropout noise in naes by computing the corresponding marginalized noise penalty : thus dropout in a nae shrinks the average size of the projective fields ( ) of the hidden units .shrinking the size of the projective field helps to reduce the sensitivity of the reconstruction to dropping out hidden units .recent work in neuroscience has shown that single - layer models trained with input and output noise and optimized to maximize mutual information yield receptive fields resembling those found in biological systems .similar to our work , these models show the importance of input noise and hidden activation noise on learning representations .  in *semantic hashing * , gaussian noise is injected at an intermediate layer to learn binary representations .these representations allow for fast matching in lower - dimensional space using binary codes . to the best of our knowledgethe advantage of adding noise in terms of accuracy of the fully trained autoencoder is not discussed .our theoretical analysis of noisy autoencoders shows that naes can implement a variety of regularized autoencoder penalties .here we evaluate the effectiveness of noisy autoencoders at learning representations through a variety of experiments on natural images , mnist , and cifar-10 .all experiments used stochastic gradient descent with momentum to train models .we found that momentum was critical in training both autoencoders and supervised deep networks . learning rates , batch size , and additional hyperparameters were selected through a combination of manual and automatic grid searches on validation sets .we consider autoencoders with a sigmoidal encoder , linear decoder , and squared error loss .we experimented with isotropic gaussian input noise with fixed variance , isotropic gaussian hidden unit input noise with fixed variance , and hidden unit activation noise that was either dropout ( with inclusion probability ) , additive zero mean gaussian with variance or multiplicative gaussian with mean 1 and variance .unless otherwise specified , we fix the input noise to be , and the number of hidden units to be 1000 .all experiments were run in python using the pylearn2 framework on a single intel xeon machine with an nvidia gtx 660 gpu .   in our first experiment, we evaluated the effect of dropout noise on the generalization performance of a noisy autoencoder .we trained two naes on 12x12 patches drawn from the van hateren natural image dataset .the first nae had noise on the input but not hidden activations ( simply a dae ) , while the second additionally had dropout noise with on the hidden activations .we evaluated denoising performance on an independent set of image patches with noise variance equal to the corrupting input noise of the nae , and computed the average reconstruction error over 1000 noisy inputs .the nae with and without dropout had average reconstruction errors of 2.5 , and 3.2 respectively .thus naes are able to improve denoising performance over typical daes .to better understand the impact of dropout on nae features , we trained a set of models with varying levels of input noise and dropout noise on hidden activations .we used smaller networks for this experiment , with 250 hidden units and training on only the first 10,000 digits of mnist .  in figure[ fig : feats ] ( left ) , we show the effect of input noise and hidden unit activation noise on the learned features . with no input or hidden activation noise , the noisy autoencoder reduces to a vanilla autoencoder and tends to learn features which do not capture interesting structure in the data .as we increase the input noise , we learn features that capture increasingly larger local structure in the digits , as found in daes . increasing hidden activation noise leads to more global features that tend to resemble large strokes or segments of digits .if we increase _ both _ input noise and activation noise , we find features which resemble larger sections of digits .we also evaluated the classification error for these different models by using them to initialize a multilayer perceptron with a softmax classifier on top of the learned hidden representation .importantly , we find the best performing model for classification contains both input noise and hidden activation noise ( figure [ fig : feats ] , right ) ..comparison of classification performance for autoencoder models trained on mnist with 2000 hidden units .columns indicate different pretraining methods , and rows indicate different supervised training methods [ cols=">,^,^,^,^,^,^,^,^,^,^,^",options="header " , ]   to better evaluate the impact of hidden unit input and activation noise on nae classification performance , we trained larger models with 2000 hidden units and fixed the gaussian input noise variance at .we considered gaussian noise at the hidden unit inputs , and both dropout and gaussian noise at the hidden unit activations .these models were used as initialization for a mlp that was trained with standard backpropagation . the level of hidden noise on 10000 heldout examples from the training set was used to optimize the noise level .we also trained standard autoencoders , daes , and caes of the same architecture .we found that naes with a dropout of achieved the lowest test error of 138 .given the pretrained nae , we can also perform noisy backpropagation where we continue injecting noise into the model while training the classifier. this noise can be the same as when we perform pretraining , but it can also be an entirely different type of noise .we consider performing noisy backpropagation with , and hidden noise that is either dropout with , or gaussian additive noise with .we find that the nae dropout model tuned with dropout noise achieves a test error of 99 . to the best of our knowledge ,this is the lowest classification error achieved on mnist with a single layer model .this model is able to waste less capacity in a large network , but requires regularization while performing supervised fine - tuning to prevent overfitting.  supervised training with gaussian hidden noise also improves performance relative to standard backpropagation .however , gaussian hidden noise does not help very much in the unsupervised features as it simply does weight decay .   to validate the utility of naes in other models, we also analyze the cifar-10 dataset .we train a nae with , dropout hidden activation noise with , and 800 units on 6x6 patches extracted from the cifar-10 dataset . using these features ,we extract a high level representation of full size images using the pipeline from .training an svm classifier on the resulting representation yielded an accuracy of 74.5% .this accuracy is slightly lower than that reported for the higher - order cae , but better than all other regularized autoencoder representations . training a dae results in a lower accuracy of 73.6% , indicating that adding dropout noise while learning features is helpful .we have shown that different types of noise can be used to regularize hidden representations and improve classification performance on mnist and cifar-10 .furthermore , we found that both dropout and additive gaussian noise on hidden activations while fine - tuning can improve classification error .here we experiment with a deep mnist model from .this model consists of two hidden layers of 1200 rectified linear units , and was trained with dropout on the inputs and hidden activations .it was formerly the state - of - the - art result for single models not incorporating prior domain knowledge or pretraining , but has recently been surpassed by maxout networks .both these networks utilize dropout while training , and the same scaling we perform at test time . instead of training with dropout , we use other types of noise on the input and hidden unit activations .we experimented with additive gaussian noise with a fixed variance , and additive gaussian noise whose variance was proportional to the mean activation ( we call this poisson noise as a shorthand ) .we optimized these networks using sgd with momentum using the same parameters as in .as in previous experiments , we fixed , and selected through cross validation for additive gaussian noise .we found that our best gaussian model had 85 errors on the test set and the best poisson model has 92 errors , beating both dropout and maxout networks .  to better understand why these other types of noise beats dropout and the noiseless version , we visualized the hidden representation for both networks ( figure [ fig : layer2 ] ) .we find that gaussian noise leads to less noisy first layer filters , and tends to group together more similar second layer features . to further understand the influence of noise on the network, we analyzed the activations across the 3 different types of noise , and the noiseless network .we found that all types of noise increase both the lifetime and population sparsity of neurons in both layers of the network .gaussian noise yielded the sparsest second layer representations , while dropout noise yieldest the sparsest first layer representations both in lifetime and population sparseness ( figure [ fig : sparsity ] , rows 1 and 2 ) .we also found that noise decorrelated activations relative to the noiseless network ( figure [ fig : sparsity ] , row 3 ) .this decorrelation with noise also flattened the spectrum of the activations , where the cumulative variance explained grew slower for the noisy networks ( figure [ fig : sparsity ] , row 4 ) .these results show that noise acts to sparsity and decorrelate representations , and spread information more evenly across the network .in summary , by proposing a unifying principle for auto - encoders , namely robustness of auto - encoders to injections of noise at all levels of internal representation , we have generated a framework for viewing a wide variety of previous training algorithms through a unified lens .different choices of where to inject noise , and what type of noise , lead to different algorithms .this allows us to generate many new training algorithms by designing noise , not only in nae s but also in networks engaged in direct supervised learning .  using these techniqueswe can achieve very good performance on benchmark tasks .we are able to obtain the best performing pretrained one layer model on mnist , as well as the best performing deep model that does not incorporate prior knowledge or pretraining .however , we stress that we have perhaps not even begun to explore the full power of naes , as we have not systematically explored the huge design space for what type of noise to inject , and where and when to inject it. a great deal of research lies ahead in understanding this design space and generating the algorithms and theories for automatically making data dependent design choices in this space .furthermore , both the space of noise that is useful for supervised training must also be explored , as well as how noise in naes and noise while supervised fine - tuning interact to optimize performance .  finally , intuitively , why might noise injection in the internal representations of a deep network be a good idea ?we hypothesize that because each training data point has an inherent noise ball around its hidden representation , classification will not be possible unless data points of different classes lie outside each other s noise balls .this introduces an effective repulsion between data points of different classes in the space of hidden representations . on the other hand, the invariance of the classifier required to properly categorize different inputs of the same class introduces an inherent compressive force in the projective map from the input level to the hidden level .thus noise on internal representations leads to a spreading of representations of different classes , and a contraction within classes that may be beneficial for generalization .

Abstract: autoencoders have emerged as a useful framework for unsupervised learning of internal representations , and a wide variety of apparently conceptually disparate regularization techniques have been proposed to generate useful features . here we extend existing denoising autoencoders to additionally inject noise before the nonlinearity , and at the hidden unit activations . we show that a wide variety of previous methods , including denoising , contractive , and sparse autoencoders , as well as dropout can be interpreted using this framework . this noise injection framework reaps practical benefits by providing a unified strategy to develop new internal representations by designing the nature of the injected noise . we show that noisy autoencoders outperform denoising autoencoders at the very task of denoising , and are competitive with other single - layer techniques on mnist , and cifar-10 . we also show that types of noise other than dropout improve performance in a deep network through sparsifying , decorrelating , and spreading information across representations .