testing for , and quantifying scaling is central to the application of statistical theories to ` real - world ' extended systems . a broad range of theoretical frameworks such as turbulence , critical phenomena and complex systems frame their predictions in terms of the statistical properties of ( arbitrarily large ) ensembles as a function of scale. under the assumption of ergodicity the statistical scaling property of an extended systemis captured to some extent by a reduced ( embedded ) set of observations or measurements ; so that a 1-d cut through a dimensional system will be sufficient to indicate whether scaling is present , and in a quantitative way can usefully restrict the scaling exponents of the system as a whole .this approach is pragmatic in physical systems it is generally not practicable to capture and analyze the full spatiotemporal information of all points in the system on all scales .a key observable is then the quantitative scaling properties of such a one dimensional sample or time series .an example of this is the use of the taylor hypothesis in turbulence , where the time series at a single point is used as a proxy for the statistical properties of the flow .time series are also often parsed into sub - intervals to isolate processes of interest , or to remove features which might contaminate the calculation of the quantity of interest .examples of this in the study of solar wind turbulence are the separating of fast and slow wind , and open / closed field line regions ; isolating or removing signals of interplanetary shocks , magnetosheath crossings , and coronal mass ejection remnants ; or where the interval is restricted by a secular change in parameters as the spacecraft changes location .examples in the study of the earth s geomagnetic field include isolating ` quasi - stationary ' and ` quiet ' intervals in magnetic field data ; and the effects of finite sample size in the power spectral exponent estimates in the ionosphere by ground - based measurements .` locally stationary processes ' are also discussed in speech signal analysis and physiological non - stationary signals ; and of course statistical forecasting , whether in the context of seasonal weather or the financial markets , is based on time series histories which rely on the stationarity assumption . in all of these cases , it is intuitively apparent that smaller data intervals will result in poorer statistics , which will be manifest in the variance of the observed values of the exponents .the observation of a ( non - secular ) variation in the scaling exponents therefore has two interpretations ; either it is due to intrinsic fluctuations as a result of the finite interval , or it is a consequence of non - time stationarity of the time series i.e. different scaling behaviour due to different physical phenomena .if the properties of the underlying process are not known _ a priori _ we need a method to distinguish these two interpretations in a quantitative manner ; or at best to put a degree of confidence that it is due to one and not the other .the most commonly used tool to both establish and quantify scaling in a time series is to test for scaling of the power spectral density , and obtain the exponent . in a physical system , such scaling can only be observed over a finite range , limited by the interval ( in time ) of observations over which the system is measured . from large - sample theory ( asymptotic limit of )the variance in the power spectral exponent computed using least squares regression from an interval of samples is known for finite variance processes to vary as as .one method to obtain more complete information about the scaling properties of a stochastic process is captured by how the statistical properties of the increments vary with the differencing scale ; the differencing being a particular type of coarse - graining operation which has been chosen due to the easy analogy with random walks , return probabilities etc .however , there exist other coarse - graining operations which although more involved , possess additional highly desirable properties when studying scaling .in particular , wavelets which ( with some wavelet functions ) when combined with their detrending capabilities have been shown to be a natural and computationally efficient way of studying scale - by - scale statistical behaviour . in this paperwe will discuss the behaviour of the scaling properties of the second order moment .we may anticipate that the statistical properties of this scaling exponent follow that of ; indeed there exist many results for a range of different estimators of the  that directly show the asymptotic behaviour discussed above . in practice ,the convergence to this behaviour will depend on the details of the process and the estimator and , as we shall show in this paper , is often slow .an essential tool in the analysis of ` real world ' time series in the context of scaling is then a prescription for the variance in the scaling exponents of as a function of the number of observations in the chosen interval . in this paperwe make some first steps in this direction by obtaining empirical estimates from the study of a variety of stochastic processes that have been used as models for physical systems .we focus on finite size realizations of self - affine cases with gaussian distributed increments in the form of a standard brownian motion and fractional brownian motion ( fbm ) ; and those with heavy tails , namely -stable lvy motion and linear fractional stable motion ( lfsm ) . a representative case for multifractal scalingis provided by the -model , often used to characterize observations of turbulence .the fundamental property of ergodicity in systems that exhibit scaling implies time stationarity . in its strong sense time stationarity implies that the probability density function ( pdf ) of does not change with time ; this is known as strict stationarity .pragmatically , weak stationarity , that is time independence of the variance or second order moment is usually adopted the latter convenience is usually assumed due to the special place that the central limit theorem and the gaussian distribution hold in statistics . in this paperwe are concerned specifically with the behaviour of scaling exponents which are characterized through the statistical properties of the increments , rather than the time series ; hence we will use as our test time series examples that have stationarity in , and not in .we will focus on the statistics of the scaling exponent of the second order moment of the increments , as this also captures the power spectral exponent , and for self - affine finite - variance processes the hurst exponent ( see next section and also for the infinite variance case ) .we will study these processes for a range of values of that are feasable for realisable physical systems ; and find that in particularly for the heavy - tailed processes , the variance in the exponent with shows a significant departure from the asymptotic behaviour .nevertheless , for these heavy - tailed processes , we find empirical evidence of an intermediate range of scaling with .we will estimate the time series interval required to capture the scaling exponent to reasonable precision ; this places a lower limit on the sample size . a related study to thiswas conducted to investigate and compare the effects of finite sample size on different statistical estimators for the hurst exponent for a gaussian white noise process .stationarity also implies a particular pdf of the values of the exponent obtained from many , length , realisations of a given process .this is known asymptotically for for the processes based on gaussian increments ( generalizable to finite variance processes ) and is also known in this asymptotic sense for infinite variance processes ; both processes approaching a gaussian distribution for the scaling exponents as  ( using least squares and maximum likelihood estimation schemes ) . for the intermediate stage of finite find intermediate distributions for the exponents ; resembling both the asymptotic gaussian forms and , for heavy - tailed data , gumbel max - stable ( extreme value type i ) distributions .comparing these results with that found for real - world time series may provide an additional test for stationarity in the increments . in this spiritwe finally illustrate these ideas with some examples of real - world time series in the form of _ in - situ _ observations of magnetic field and velocity in the turbulent solar wind using data from spacecraft at 1au in the ecliptic ; and comment on the statistical properties of their scaling exponents in light of the representative synthetic toy models .we will focus attention on the scaling exponent of the second order moment of the increments also known as the second order structure function : where we have assumed that the increment process is at least second order stationary i.e. ( weak - stationarity ) .in particular , this implies that the power spectral density of a discrete time random walk of _ i.i.d . _ stationary increments with finite variance , scales as  where the scaling exponent is related to the power spectral exponent of by . for self - affine process with hurst exponent pdf of the increments obeys the scaling relation ( for the case of -stable processes with finite see , and the discussion to follow ) where the pdf at any scale can be collapsed onto a unique scaling function .the scaling relation ( [ eq:1 ] ) implies that the scaling of the structure functions to all orders  is given by where ; and thus we have that . our results concerning the statistical behaviour of with will thus also apply to the power spectral exponent for all the models concerned and the hurst exponent for the self - affine models ; both are commonly used to characterize statistical scaling .our remarks can also be generalized to the scaling exponents of structure functions of higher - order positive moments .these are relevant for multifractal processes where the are a nonlinear function of and so or are not sufficient to determine the complete statistical scaling of the .our study consists of partitioning a given time series into equal intervals of sample size denoted by where .each of these intervals are then differenced on scale to produce a time series of the increments of the process .we will look for scaling of the second order moment ( structure function ) with such that .again , the index indicates the interval over which the exponents are calculated and tracks any ( real or statistical ) time variation in the value of . in an infinitely large interval , , the limits of the integral ; here however each interval of the time series will impose different finite extremal values . for the heavy - tailed processes in particular ,the statistics of the can be anticipated to have a significant effect on the statistics of the ; this has been discussed for the case of - stable lvy processes in .these lvy processes , possess heavy tails in the pdfs of their increments , with tails that fall as power - laws .the -stable lvy processes have divergent moments for and above ; for a finite sized sample the moments exist but can be dominated by the behaviour of rare outlying points in the tails which introduce a pathological bias when estimating scaling exponents from the moments ( for a wider discussion see ) .we circumvent these difficulties , at least for self - affine time series , by restricting our analysis to the scaling of the second order moment , and by using the iterative conditioning technique .this simple and robust technique for exponent estimation removes a small percentage of the extreme data values which are poorly sampled statistically . in some pathological cases such as the -stable lvy distributionsthese rare extreme points are of the order of and sometimes larger than the whole sum . because they are so large they tend to dominate the statistics and thus the scaling of the higher order moments .this can be clearly seen if we look at the discrete definition of the moments of order  the reasoning and full illustration of this iterative conditioning method to heavy - tailed non - gaussian distributions is discussed in .although not discussed in this paper , the iterative conditioning technique is also an unbiased robust technique for distinguishing self - affine ( monofractal ) from multifractal behaviour .we will focus here on parameter stationarity as opposed to trend stationarity .the former refers to the change in the intrinsic dynamics of the process of interest as characterized by its quantitative statistical properties ( the behaviour of the moments ) ; as opposed to the latter which is simply an additive trend to the signal .in particular we will focus on the stationarity of the scaling of the moments as captured by the exponents .if secular trends are present in the time series then the time series of increments will be approximately trend - free provided our differencing scale is sufficiently small .a secular trend can also be removed by detrending or by the method of studying the scaling of moments of wavelet coefficients where an appropriate wavelet is chosen with a large number of zero - moments .the more complex case of mixed dynamics i.e. two or more intrinsically different processes represented in different sections of a time series will not be considered here .we will consider synthetically generated signals that are both stationary and nonstationary with respect to their increments .the signals with stationary independent increments will consist of a standard brownian motion and standard symmetric -stable lvy motion for four values of the exponent  ; the latter being highly non - gaussian and heavy - tailed with very large excursions in their time series . to survey a broad range of such processes we have also included non - markovian versions of the above processes .these include a long - memory fractional brownian motion ( fbm ) , and a long - memory persistent and anti - persistent linear fractional stable motion ( lfsm ) see for more details on these processes and in particular for the algorithm and matlab code for the lfsm .we also investigate a multifractal time series generated from a discrete multiplicative cascade process in the form of the -model .the -model is used as a model for intermittent turbulence .the intermittency of the -model time series leads to non - time stationary finite moments ; however the set of scaling exponents are stationary .  the nonstationary time series we will consider are a standard brownian motion with linearly varying standard deviation of its increments with time , and cyclically varying standard deviation ( cyclically stationary ) .all of the above synthetic time series were generated in matlab with appropriate random seeding and sample sizes of .lastly , we will consider three real - world time series which have been found to exhibit scaling .these consist of two time series of 100 second resolution magnetic field and speed from the nasa wind spacecraft at 1au in the solar minimum year 1996 ; and a 64 second resolution one year long time series of the magnetic field energy density from the nasa ace spacecraft in the solar maximum year 2000 .all of these time series consist of data samples and can be downloaded from cda web http://cdaweb.gsfc.nasa.gov/.we study the variation of the scaling exponent of the second order moment with sample size .the process by which the exponent is estimated for contiguous intervals of points of a time series is illustrated in figure [ fig:1 ] for the -model .we begin with the time series in figure [ fig:1](a ) which we parse into intervals . for each of these intervalswe obtain an estimate of as the gradient of a linear least squares regression to a log - log plot of the second - order moment versus the scale or differencing parameter .this method of obtaining the scaling exponents is also known as the structure function technique and is closely related to variance plot , correlogram and log - periodogram techniques  in the latter reference it is identical to the variogram technique .we focus on this particular method to estimate as it provides a point of contact with asymptotic estimates of the variance of the power spectral exponent which are based on linear regression over a finite range power law power spectrum . in both cases ,the variance in the estimated exponent will depend upon the details of the linear regression .for the second order moment these details include the range of values of over which is a power law ; the number of different for which we calculate and use in the linear regression ; and the uncertainty of each value . in all cases considered herewe optimize these details to minimize the linear regression error but importantly use the same algorithm for all of the sample time series that we discuss .the linear fit is obtained by linear least - squares regression which also provides an estimate of the error .we augment this estimate of the error by varying the start and end points of the regression by a few points and obtaining the difference in the exponents .the linear regression was done over values of the scale parameter , where was increased geometrically as , where and was chosen to be .the fit was done over this reduced set of measurements at values of so that a fair comparison can be made with the real - world data ( to be discussed later ) where only a limited power - law range is seen .( color online ) a. ) time series of length for the -model ( ) .b. ) variation of the second order moment of the increments for time - scale of the above time series where the original time series has been partitioned into and equal sized intervals .c. ) variation of conditioned with time for the -model with the same segmentation as in b. ) also shown are the mean values of the exponents for different partitioning corresponding to different sample sizes ; d. ) same as ( c. ) but with errors explicitly shown . ]  due to its highly intermittent nature the -model is not time stationary in its finite moments and this can be seen in figure [ fig:1](b ) where we plot consecutive values of the second order moment obtained for each of the intervals of points , shown for and two values of . for the -model timeseries shown here , the second order moment follows the local amplitude of fluctuations in the time series itself ; comparing the ratio of the amplitude of these fluctuations to the signal amplitude is one of the classical ` first base ' techniques for establishing whether the signal is stationary ( in the weak sense) . as one would expect from ( [ eq:5pt5 ] ) , this variation of the second - order moment with the amplitude of the time series is emphasized as we decrease as any estimates of the statistics from smaller sample size will naturally mimic the more local features of the time series .this behaviour is more pronounced in very intermittent signals i.e. those with heavy - tailed fluctuation pdfs .we also plot in figures [ fig:1](c ) and ( d ) the corresponding estimates of for each interval .these two panels show the same data , that is , the estimates of plotted without ( c ) and with ( d ) error bars obtained from the linear regression and the error augmentation outlined above .as intuitively expected , if we decrease the sample size over which the are computed , the scatter increases .however unlike the moments , there is no clear trend with the amplitude of the signal , indicating stationarity of the scaling exponent .this latter phenomenon will also be encountered in the non - stationary brownian time series we will study .the estimates of can be seen to vary by up to a factor of two for for this realization of the -model .this underlies the difficulty of obtaining physically meaningful estimates of scaling exponents for physically realizable .we can see that the error bars approximately capture the fluctuations in the estimates of for the case of the -model .as we wish to include strongly non - gaussian processes in our study , we will henceforth present numerical estimates of the variance of obtained directly from computing many values of rather than the linear regression error .( color online ) a. ) the variance of conditioned with sample size for all the synthetic finite variance processes studied shown on a log - log plot .b. ) same as in a. ) for all the synthetic infinite variance processes studied .the diagonal dashed line on both these plots indicates a negative slope of unit gradient so that comparison with theoretically expected asymptotic behaviour can be made .the vertical black dashed lines indicate the areas outside of which errors begin to dominate due to i. ) ( bottom vertical line ) lack of values of to make a decent estimate of ; and ii . )( top vertical line ) failure of the iterative conditioning technique to obtain unbiased estimates of . ]the essential point is that quite significant variation in the scaling exponents can arise in time stationary , but intermittent , time series ; even when these are estimated over intervals of data that might intuitively be considered to be of adequate length . in order to distinguish variation in the scaling exponents that is statistical ( finite effect ) as shown above , from that whichreflects intrinsic non - time stationarity , some estimate of the dependence of the variance of is needed ; this will also point to an estimate of the minimum number of observations needed to obtain a ` reasonably accurate ' estimate of .we will now explore the variance of as a function of .  in the limit , , when estimated via a log - periodogram varies as \sim 1/n ] for our moment scaling estimation also .however , we do not know the rate of convergence with to this limiting behaviour and can also anticipate that this will depend upon whether the pdf of the increments is heavy - tailed , and whether or not the increments are dependent both of which introduce further difficulties in obtaining an unbiased estimator .in figure [ fig:2 ] we plot the variance of against the sample size on log - log axes , for a range of that are feasable in realistic realizations of physical systems .figure [ fig:2](a ) shows the behaviour of a subset of our synthetic time series that are intrinsically finite variance processes ; figure [ fig:2](b ) shows all the synthetic time series from infinite variance processes that we consider . plotting these on log - log axes reveals a characteristic power law trend for all the processes : = c n^{-\gamma}\ .\label{eq:8}\ ] ] we see that indeed , for the intrinsically finite variance processes . more pragmatically , we can use this plot to make an estimate of the minimum sample size needed in order to estimate such that the error introduced from the small sample size is , say , .we propose a simple criterion }}{\zeta(2)\vert_{l=1 } } \lesssim 0.05 \ , \label{eq:6}\ ] ] where is the value of estimated for the entire time series ( assuming that the scaling is stationary ) .this leads to \lesssim \left ( 0.05\zeta(2)\vert_{l=1}\right)^{2}\ , \label{eq:7}\ ] ] where the value of is extrapolated from the plot of  ] , even from a single realisation . in this contextwe should mention the use of bootstrap re - sampling methods for providing distributions , confidence intervals and statistical significance for parameter estimates in situations when one is limited by a single realisation .although the convergence and consistent properties of such techniques in the case of heavy - tailed distributions and especially infinite - variance processes are still unclear we envisage the use of such methods in future research .finally , one could in principle increase the available number of values of by overlapping intervals of size within a given single realisation .we have , however , found that this introduces a significant systematic bias in the computed values of  ] values for the examples of real - world data sets discussed earlier in the introduction .the plot detailing this study is shown in figure [ fig:4 ] .for comparison we have also included on this plot the variation of  ] . for the ace data we can see a clear systematic departure from the synthetic models. we will discuss this latter data set in the next section below .the problem with a single length realisation is that we can not calculate the errors on $ ] as done in the previous section ; and thus have no way of gauging how close these graphs are to the expected asymptotic behaviour predicted by large - sample theory .however , one can still estimate an error for measurements of obtained from a finite data size , in the same way as was done in equations ( [ eq:6])-([eq:9 ] ) .for example , in the case of the ace data this would indicate that a sample size would introduce an error of in the estimated values of using the iteratively conditioned moment scaling technique .( color online ) plot of the variance of conditioned with sample size for all the real - world data sets studied shown on a log - log plot .the dashed line on both of these plots indicates a negative slope of unit gradient .also included for comparison are the archetypal synthetic data sets for the finite variance and infinite variance processes . ]we plot in figure [ fig:5 ] the pdfs for three of the representative models we have studied along with the pdfs for one of the real - world data sets . for each of these time series , pdfshave been constructed for two different sample sizes .we see that apart from the -stable case , these pdfs are well described by a gaussian distribution , as can be seen by the maximum likelihood fits .the -stable lvy case is shown in figures [ fig:5]b i. ) and b ii . ) to be well described by a gumbel max - stable extreme value distribution . to see why nearly all of our pdfs corresponding to finite variance processes are close to gaussian we appeal to large sample - theory .    to facilitate understanding we employ more heuristic arguments at the expense of mathematical rigour .interested readers can find more on the mathematical details and proofs in which deal with spectral parameter estimates of strong long - range dependent gaussian stationary time series ; for a non - stationary generalization of these ; for a pseudo - variogram estimator ( similar to the method in this paper ) to long - range dependent gaussian processes with stationary increments ; and the more recent and extensive paper by stoev , pipiras and taqqu which extends the proofs and arguments of to infinite variance processes in the form of -stable and linear fractional stable processes .this latter reference will be our main source and point of contact for what follows .a survey of many of these papers and parameter estimation techniques can be found in .as mentioned above , we have chosen stoev _ as a point of contact from amongst the extensive literature concerning asymptotic large sample behaviour of parameter estimates .this is primarily because this work has dealt with infinite variance processes of the type discussed here ; also our moment scaling technique is a particular form of one of the main estimators used in ( see also ) .we have also used stoev s matlab algorithm for generating the lfsm realisations used in this paper .similar to our study stoev _ et ._ use a least squares regression on the moments which they refer to as the ` power ' estimator .however , instead of taking moments of the increments as we do , they take the moments of coefficients for a finite impulse response transformation ( firt ) of the discrete time - series , which is characterised by a discrete filter of members .our increments are one of the simplest forms of these firt coefficients if we take the filter to be comprised of a set of members .however , any extra benefits of having more than one zero - moment ( moments which are equal to zero ) will be lost due to this simplicity .this also applies to the wavelet coefficients used in the study of stoev _ et ._ where our increments result from taking the mother wavelet to be the superposition of two delta functions ( one positive , one negative ) separated by a scale  also known in the literature as the ` poor man s wavelet ' .also , an important fact to note when comparing the methods of stoev _ et ._ to ours is that we differ with the ` power ' method of these authors by using the iterative conditioning technique which by censoring and excluding the poorly sampled large extreme events , correct for the bias which is pathological in the case of heavy - tailed distributions .recall that the second order moment is scaling as and we will be estimating from the gradient of a log - log plot for the discrete data the gradient can be estimated via least squares linear regression , and the problem can be set out as where is the vector of the observations ( or dependent variables ) ; contains the vector of the scales ( or independent variables ) ; is the vector of the parameters needed to be estimated ; and is the vector of estimation errors between the sample measurements and those of the true expected values ( denoted by ) for which the scaling relation in ( [ eq:10 ] ) actually holds .the solution to equation ( [ eq:11 ] ) is then given by the well known ordinary linear least squares estimator to the parameters as where superscript represents a matrix transpose .( [ eq:15pt5 ] ) is simply a linear combination of the dependent variable i.e. a sum , so that for one can write the ordinary least squares estimate as where the are all the elements of the appropriate vector from .we will return to this form of the ordinary least squares solution shortly .  adapted to the notation used in this paper , theorem 3.1 in states that if is the firt coefficient estimator ( using least squares regression ) for the scaling exponent and the true expected value ; then as  where is a normal distribution with mean and variance . strictly speaking this theoremrequires that the firt coefficients obey the following inequality between the number of zero - moments of the firt filter , the self - similarity parameter and the tail ( stability ) index  which in the case of the ordinary brownian motion and the -stable lvy processes , where , generalises to the moment scaling scheme based on the raw increments has only one zero moment , hence ; and as a result , except for the -model for which and are unknown ( or not applicable ) , the above inequality does not hold for any of the synthetic models . however as mentioned in for gaussian processes , where in equation ( [ eq:17 ] ) ,the case is sufficient for the above theorem to hold as long as . for our fbm model and we find that the results of the theorem ( [ eq:16 ] ) still hold .thus we can conjecture that the criterion in ( [ eq:17 ] ) and ( [ eq:18 ] ) can be relaxed a little so that the inequality becomes an approximate inequality .also , stoev _ et ._ show via simulations that the estimators continue to work well even when the criterion in ( [ eq:17 ] ) and ( [ eq:18 ] ) are not satisfied .the essential reason why this criteria was initially introduced , was so that the estimator could distinguish between actual long - memory effects and trends .we now consider the exception that we have found to this behaviour that of the infinite variance processes at finite .we would expect that in the limit the results of the above theorem will also hold for the infinite variance processes .however , we believe that in this case the convergence will be slow and will depend upon the number of scales that were used to conduct our linear least squares regression.  we will now go beyond the above asymptotic arguments to discuss the non - gaussian intermediate finite behaviour that we see here in figure [ fig:5 ] ( b - i ) and ( b - ii ) .recall the expression for given in equation ( [ eq:5pt5 ] ) ( for ) .we will discuss in detail here the case where the sum in ( [ eq:5pt5 ] ) consists of _ i.i.d ._ random variables ; this is the case for some of our synthetic time series these arguments can be developed for other cases .the pdf of this sum will by the central limit theorem tend to a gaussian and for finite will probably take the form of a pearson type variable with degrees of freedom ( see for more details ) .however , for an infinite variance process , the sum in ( [ eq:5pt5 ] ) will be dominated by the largest extreme events , which in some cases can be of the order of the rest of the sum . this will still be the case even when we have excluded some of these extreme events due to the iterative conditioning scheme .thus the sum will be distributed as the extreme values of an -stable lvy distribution which is given by a max - stable frecht distribution ( see ) .note that this will be the case for any , even in the asymptotic large case . without too much detailthe form of the pdf of will be of the type where any scale parameters have been absorbed into the .one can then convert this frecht pdf to a pdf corresponding to the dependent variable in the least squares scheme in ( [ eq:15pt75 ] ) , which under a simple conservation of probability will be given by which is in the form of a gumbel extreme value distribution ; this is another max - stable distribution .the gumbel max - stable pdf has a long slow exponentially decreasing right tail ; this will imply that a sum of random variables such as ( [ eq:15pt75 ] ) , each distributed with this pdf , will eventually tend to a gaussian distributed random variable , but slowly due to the heavy right tail .this then captures our result in figures [ fig:5 ] ( b - i ) and ( b - ii ) , and may also explain why we do not obtain the behaviour in the plots of vs. for the -stable lvy cases .  as discussed above , for the case of the finite variance synthetic time series the be well described by a gaussian or ( more realistically for finite ) a pearson pdf . in the same wayas was done with the infinite variance processes above , it can be shown that the pdf of can be written as a gumbel min - stable pdf .gumbel min - stable pdf s have long slow exponentially decreasing left tails , which in our case will be limited by the fact that .the right tail of gumbel min - stable pdfs is more compact with a rapidly decaying exponential tail . due to the more compact nature of the pdf, a sum of variables will tend to a gaussian under the central limit theorem much faster than the infinite variance processes above , hence explaining why the pdfs for the finite variance processes are well described by a gaussian .finally , there is the open question of the behaviour of the real - world data .the ace data pdfs of show that they can be well described by a gaussian ; however the scaling of with using our estimation shows a significant deviation from the behaviour implied by ( [ eq:16 ] ). this will be the topic of future work .we have investigated finite - sample size ( ) effects on quantifying the statistical scaling properties of time series .we focus on the scaling exponent of the variance or second moment which for a wide class of processes also gives the spectral exponent of the ( power law ) power spectrum .if too small a sample size is used then these fluctuations effectively introduce a pseudo - nonstationarity in the estimates for the scaling exponents . to achieve an error in the exponent of less than , we find that the number of data points needed varies significantly with the details of the underlying process and is in the range of for the synthetic models used in this paper .the variance in the exponent when computed from an interval of samples is known to vary as for ; however , the convergence to this behaviour will also depend on the details of the process and more importantly on the parameter estimating technique used .in particular we have shown that heavy tailed processes can be far from this limiting behaviour for observationally realisable .we have also considered the case where the scaling exponents are time independent , but where there is a secular time dependence in other parameters such as the standard deviation . for the case of a brownian motion ,the estimate of the scaling exponent is not affected by this time dependence .it may thus be too premature to reject a time series for scaling analysis simply because of the non - stationarity of certain parameters i.e. a running mean or standard deviation .this also highlights the need to distinguish time variation in the moments from that in scaling exponents that are derived from them .we have focussed here on the moment order scaling technique to calculate the scaling exponents in order to highlight the issue of apparent non - time stationarity .although there exists extensive statistics literature on the asymptotic limit of various estimation techniques , further work is needed to investigate how these details pass over to the more pressing and pragmatic need for their implications on quantifying scaling in finite and realisable -sized samples .the authors would like to thank b. hnat , g. rowlands , f. m. poli , t. dudok de wit and v. keinhorst for helpful discussions and suggestions .we acknowledge the financial support of the uk stfc and epsrc ; and r. p. lepping and k. oglivie for ace and wind data .

Abstract: the accurate estimation of scaling exponents is central in the observational study of scale - invariant phenomena . natural systems unavoidably provide observations over restricted intervals ; consequently a stationary stochastic process ( time series ) can yield anomalous time variation in the scaling exponents , suggestive of non - stationarity . the variance in the estimates of scaling exponents computed from an interval of observations is known for finite variance processes to vary as as for certain statistical estimators ; however , the convergence to this behaviour will depend on the details of the process , and may be slow . we study the variation in the scaling of second order moments of the time series increments with , for a variety of synthetic and ` real world ' time series ; and find that in particular for heavy tailed processes , for realizable , one is far from this limiting behaviour . we propose a semi - empirical estimate for the minimum needed to make a meaningful estimate of the scaling exponents for model stochastic processes and compare these with some ` real world ' time series .


systems engineering ensures that the total is more than the sum of its parts . systemsengineering is essential for the successful design of polarimeters where apparently unrelated effects such as instrumental polarization due to oblique reflections in the telescope and detector non - linearity couple to generate substantial systematic errors ( keller 1996 ) .when designing a polarimeter , it is therefore crucial to optimize the entire system and not just the individual parts as has often been done in the past .while systems engineering is common in standard optics , it has been largely absent in polarimetry because errors in polarization can not easily be expressed as a scalar and optical elements such as polarizers and retarders have a major influence on the polarization . in the followingwe will present initial thoughts on establishing systems engineering approaches for polarimetry that will enable better polarimeter designs with predictable performance .systems engineering for polarimetry models and helps us understand the performance of polarimeter designs .it must address at least the following issues :  * a definition of the polarimeter performance to quantitatively compare different polarimeter designs with ideal components ; * a polarization error budget that can predict the performance based on known error statistics of real components ; * methods to maximize the performance of a polarimeter design .while the first issue has been addressed in the literature , error budgets , a crucial systems engineering tool , have been lacking . in the following sections we present some initial ideas on systems engineering approaches for polarimeter designs .errors in polarization measurements fall into two classes : statistical ( random ) errors and systematic ( instrumental ) errors .systems engineering must provide approaches to quantify and balance these two , very different , error sources .   the influence of statistical errors on the final polarization measurement can be calculated using error propagation .these calculations become relatively simple if the following assumptions hold :  * there is a linear relation between the stokes parameters of the incoming light and the signals that are measured ; * the noise does not depend on the position of a measurement in a sequence ; * the noise is independent of the signal , which is the case if * * the noise is dominated by signal - independent detector noise ( e.g. read - out noise ) or * * the light that is analyzed is only slightly polarized ; * the noise statistic has a gaussian distribution .  for the large number of photons , which are needed for accurate polarimetry , a gaussian distribution is a good approximation to the poisson distribution of photon noise .if the degree of polarization of the incoming beam is small , the measurements will all have very similar light levels , thereby justifying the assumption of independence of noise and signal .the intensities measured by the detector can be combined into a signal vector , which is related to the incoming stokes vector , by the signal matrix ( sometimes also called the synthesis matrix , e.g. tyo 2002 ) ,  is a 4 by matrix , where is the number of intensity measurements that contribute to the polarization measurement .for example , for most systems that use liquid crystals , while for a rotating retarder . is a function of the free parameters of the polarimeter design .since the polarimeter optics can be described by mueller matrices , each row of corresponds to the first row of the mueller matrix describing the particular intensity measurement as a function of the incoming stokes vector .  to determine the stokes vector from the measurements , needs to be inverted . with the standard deviations of the stokes parameters , , are given by where is the standard deviation of the intensity in measurement .in many cases , the latter does not depend on the measurement number and one can rewrite the equation as these errors can then subsequently be propagated into errors in the degree of circular , linear polarization and angle of linear polarization or polarization ellipse parameters , and then into , e.g. , solar magnetic field parameters .the last step is important because the relations between stokes parameters and magnetic field parameters or other physical quantities of interest are not linear , and one should optimize a polarimeter design according to what one wants to measure .   because of the large photon flux from the sun , solar polarimetersare largely limited by systematic ( instrumental ) errors rather than by statistical errors . a list and discussion of instrumental error sourcescan be found in keller ( 2002 ) .our goal here is not to understand the individual error sources but to discuss ways to add their influence such that we can predict the performance of a polarimeter under non - ideal conditions .if one is able to translate the science requirements into requirements for the measurement of the stokes parameters , then the concept of a _ polarimetric efficiency _ is useful .any definition of the polarimetric efficiency must have the following properties :  * comparable between different polarimeter designs and measurement approaches ; * larger values should correspond to better designs ; * independent of the intensity throughput ; * consist of 4 quantities ( `` stokes efficiency '' ) ; * the theoretical maximum efficiency shall be 1 .  the polarimetric efficiency with which the component of the stokes vector is measured was defined by del toro iniesta and collados ( 2000 ) as note that this definition is independent of the number of measurements that contribute to the measurement and fulfills all the requirements listed above if .since the polarimetric efficiency is independent of the intensity throughput , it is important to not directly compare polarimetric efficiencies of different designs but to also properly take into account the throughput of the corresponding design .for instance , a polarizing beam - splitter makes use of all photons , while a regular linear polarizer will not transmit more than 50% of all photons .   when optimizing the design of a polarimeter , it is convenient to define a scalar function of the free design parameters such that the maximum or minimum of this _ merit function _ corresponds to the optimum polarimeter design .the polarimetric efficiency provides a well - defined _ merit function _ for optimizing a polarimeter design , in particular when one demands a fixed ratio ( often unity ) between the efficiencies for the different polarized stokes parameters .various merit functions can be found in the literature , but they are all closely related to the polarimetric efficiency .  for certain cases of merit functions , one can derive equations for the properties of the optimum polarimeter ( e.g. del toro iniesta and collados , 2000 ; tyo 2002 ) . in practice , one may often use numerical optimization schemes , but the analytic approach does reveal some of the basic properties of an _ optimum _ polarimeter design and allows us to derive the _maximum performance_. the _ maximum performance _ is the best performance of all polarimeter designs , while the _ optimum performance _ is the best performance that can be achieved with a given polarimeter design . for a polarimeter with maximum performance , and .we extend eq .[ equ : keller_signalmatrix ] into the basic measurement equation by explicitly showing the dependence on the free design parameters and the addition of random , zero - mean noise .a minimum of measurements is required to determine all four stokes parameters .  to determine an estimate of the stokes vector from the measurements , needs to be inverted .if ( sometimes called the analysis matrix or demodulation matrix ) is the inverse of , we obtain our goal is to choose and such as to minimize the difference between and given the standard deviations of the measurement errors .we tackle this problem in three steps : 1 ) derive an equation for the optimum for a given ; 2 ) derive an optimum for the optimum ; and 3 ) choose optimum and to obtain a polarimeter with maximum performance .   for a given , is not necessarily unique .it is obvious that apart from fulfilling , should minimize , given the standard deviations .this can be written as a minimization problem , the solution of which is given by the generalized inverse ( e.g.albert 1972 ) among all possible that fulfill , the generalized inverse minimizes the sum of squares of its rows .therefore , the generalized inverse is the optimum synthesis matrix , given a signal matrix . since is symmetric and positive definite , it can be inverted with standard matrix inversion algorithms .  realizing that the sum of squares of the rows of are the diagonal elements of , the optimum polarimetric efficiencies for a given signal matrix are derived ( after some linear algebra )as    we now derive the properties of the signal matrix that provides maximum performance , i.e. minimizes the sum of squares of the rows of under the condition that . using eq .[ equ : keller_optpoleff ] , del toro iniesta and collados ( 2000 ) showed that the squares of the maximum possible polarimetric efficiencies are given by hence the maximum possible efficiency is given by the sum of squares of the elements of the _ columns _ of the signal matrix , normalized with the number of measurements .  for to hold ( remember that we have only considered the diagonal elements of this relation so far ), we must also require that where is equal to 0 unless where it takes on the value 1 .inserting into the above equation and multiplying both sides of the equation with the denominator of the left side yields for , the equation is obviously correct .for , the right side is zero and we conclude that has to be diagonal for a polarimeter to achieve the maximum performance .a polarimeter that achieves its maximum possible polarimetric efficiency therefore has a signal matrix such that    we now go beyond the work of del toro iniesta and collados ( 2000 ) to provide more insight into the properties of the signal matrix for a polarimeter with maximum efficiency .since each row of the signal matrix is the first row of a mueller matrix , it obeys the inequality where the equal sign applies if there are no depolarizing elements between the source and the detector .this is a direct consequence of the properties of a mueller matrix .the optimum polarimeter has an intensity efficiency of .the other three efficiencies have to obey .this implies that the optimum efficiency for measuring polarized stokes components with equal efficiencies is given by where is the number of polarized stokes parameters that are measured .if all stokes parameters are measured , the maximum efficiency is , if only two are measured , the maximum efficiency is , and for a single polarized stokes parameter , the maximum efficiency is obviously 1 .  in a polarimeter with maximum efficiency , the elements of the first column of the signal matrix are equal to 1 . since the columns of the signal matrix need to be orthogonal , and the scalar product of the first column with any of the other columns corresponds to the sum over all elements of the other product , we conclude that   for a polarimeter reaching maximum efficiency , there will be no depolarizing elements , and we have since the rows of correspond to the first rows of mueller matrices .we can thus consider each row of the signal matrix as a point on the poincar sphere . in the following ,we will show that by maximizing the average distance squared between these points ( on the poincar sphere ) , we obtain a signal matrix that provides maximum polarimetric efficiency .  to maximize the separation between points on the poincar sphere , we maximize the functions where are lagrange multipliers .these equations are valid independent of the number of polarized components that are considered since the elements are zero for polarized components of the stokes vector that we do not want to measure .  by setting the derivatives of this function with respect to to zero ,we obtain by squaring both sides of this equation , summing over , and remembering that , we obtain since the right side is independent of , we conclude that all lagrange multipliers take on the same value , i.e.  . using this relation in eq .[ equ : optcolumnsum ] and summing over all , we obtain because is not a valid lagrange multiplier , the only way to fulfill this equation is the requirement that the sums of columns of for vanish , i.e. in other words , the points have to be distributed on the poincar sphere in such a way that their center of gravity is always at the origin of the sphere .this is the same requirement as for a polarimeter with maximum efficiency .we conclude with calculating the distance between points on the poincar sphere .the average distance squared between one point and all the other points is given by which reduces to we finally obtain for we obtain , which corresponds to opposite sides of the poincar sphere .for we obtain , which corresponds to the side length of a triangle whose plane includes the origin of the poincar sphere .for we obtain , which corresponds to the side length of a tetrahedron inside the poincar sphere .note , that only corresponds to the distance between points if the distance between _ all _ points is the same , which is the case for these examples .it should not be surprising that this average distance is independent of the number of polarized stokes components that are considered , since at no point in our derivation did we make any assumptions about the number of components that we want to measure .  unfortunately , maximizing the average square distance of points is a necessary but not a sufficient criteria for generating a signal matrix corresponding to a polarimeter with maximum performance .the following signal matrix also has an average distance of , but is obviously not diagonal .once a polarization analysis system has been designed , we need to determine the optimum way to calibrate it , i.e. to experimentally determine the signal matrix .as it turns out , much of the previously derived results can also be applied to find the optimum calibration approach , i.e. based on the measured signal matrix , we need to determine the optimum matrix . to determine all 16 elements of , we need to make ( at least ) 16 measurements of the signal , which corresponds to ( at least ) 4 different input stokes vectors with ( hopefully ) known properties .  based on the approach by azzam et al .( 1988 ) , we group the 4 calibration input stokes vectors and the corresponding signal vectors into 4 by matrices .we can then write with the signal matrix is given by with .we need to choose such as to minimize the error in , given errors in the measurements .this is the same problem that we faced when optimizing the polarimetric efficiency , and we can indeed apply the same reasoning .the calibration input stokes vectors for maximum calibration accuracy should therefore obey the same relations as the rows of the signal matrix of a polarimeter with maximum performance .  for a polarimeter that only measures one polarized component ( e.g. a circular polarimeter ), we would use the two corresponding orthogonal polarization states . for a system that measures two polarized components ( e.g. a linear polarimeter ) , we would use three calibration stokes vectors whose points are the corners of an equilateral triangle . for the linear polarimeter , we could use a rotating linear polarizer positioned apart . for a vector - polarimeter, we would choose the corners of a tetrahedron as suggested by azzam et al .( 1988 ) .  in practice, however , one has to take many more measurements so that the properties of the non - ideal polarization calibration optics can also be determined .error budgets are a classical tool in systems engineering to derive requirements for the individual parts of a system such that the system as a whole meets the requirements while minimizing the total complexity and/or cost .a typical example is the wavefront aberration in an optical system that contains many optical components .an academic example is shown in tab .[ tab : keller_errorbudgetoptical ] .lllrrr level 1 item&level 2 item&level 3 item & level 1 & level 2 & level 3 + atmosphere & & & 0.50 & & + telescope & & & 0.25 & + & primary mirror & & & 0.17 & + & & mirror polishing & & & 0.10 + & & mirror support & & & 0.10 + & & thermal distortion & & & 0.10 + & secondary mirror & & & 0.17 & + & & mirror polishing & & & 0.10 + & & mirror support & & & 0.10 + & & thermal distortion & & & 0.10 + instrument & & & 0.25 & & + total & & & 0.61 & & +  an error budget can have several levels since parts can again be looked at as a combinations of smaller parts . since the overall error is estimated from the combination of many sources , mistakes in the estimates of errors of individual componentstend to average out .furthermore , only the overall error has a requirement attached to it , and the individual errors of each level of the error budget can be allocated in different ways .this error allocation is an iterative process where one tends to minimize the complexity and cost of the system while focusing on the main contributors to the system error , which are quickly identifiable in the error budget .  for an error budget to make sense, one needs to figure out how to add errors . in the case of aberrations of an optical system ,the wavefront errors of individual elements are not correlated , and one can simply assume that the final error is given by the square root of the sum of errors squared of the individual elements , the so - called _ root sum of squares _ ( rss ) .for a polarimeter , the issue is much more complicated since 1 ) polarization is a vector quantity and not a scalar ; 2 ) retarders and polarizers affect the polarization in a major way and not in a minor way as required for rss to make sense ; and 3 ) errors of very different nature are combined non - linearly , such as the instrumental polarization of the optical system that feeds the polarimeter and the non - linearity of the light detector system ( keller 1996 ) .another difficulty comes from the fact that a polarimeter is , in most cases , calibrated experimentally .this means that some ( but not all ) errors can be drastically reduced , and one should only include the error that remain after the calibration .this is somewhat analogous to wavefront aberration error budgets for optics that include active and/or adaptive optical elements .lll level 1 item&level 2 item&level 3 item + source variation & & + atmosphere & & + telescope & & + polarimeter & & + & polarizer & + & detector system & + & & undetected bias variation + & & nonlinearity + calibration & & + & polarizer & + & retarder & + & & positioning repeatability + & & temperature change of retarder + data reduction & & +  because of all these difficulties , _ polarization _ error budgets have historically not been used in the design of polarimeters .first attempts have been described by boger et al .( 2003 ) , but this does not go further than a list of potential errors .table [ tab : keller_errorbudgetpolarized ] shows several potential error sources in polarimeters , separated into three levels .if an optical element in a polarimeter can be described by a mueller matrix , then any small error associated with this element alone can be approximated by a linear change in the mueller matrix elements .for example , a mueller matrix describes an optical element with two parameters and , e.g. a retarder with retardation and fast axis orientation . a taylor approximation for the mueller matrix with scalar errors and in the respective scalar parameters and be written as for some parameters of some elements it can be that the first - order taylor expansion is insufficient and that the second order has to be considered . for simplicity , we will assume in the following that the first - order expansion is adequate . instead of working with specific errors and , we need to look at eq .[ equ : keller_taylormatrix ] in a statistical sense such that and correspond to characteristic values of the magnitude of the expected error , e.g. the standard deviation for a zero - mean gaussian distribution or the extreme values of a uniform distribution .the matrices can then be interpreted as normalized standard deviations of the elements of the mueller matrix .  for a uniform distribution of errors in over can calculate the variance of the mueller matrix with respect to errors in as where the does not indicate a matrix multiplication but the square of the individual matrix elements .the normalized standard deviation then becomes hence , the mueller matrix with errors can be written as   as an example , we show a linear retarder with fast axis angle and retardance .the corresponding mueller matrix is for a uniform error distribution in retardance we obtain the normalized standard deviation matrix with respect to as    in the best case scenario , the statistical distribution for the random errors is known from a large number of components with identical requirements .however , this is often not the case and one has to make assumptions . while a gaussian distribution might be a natural choice for certain alignment errors , it is often too optimistic an assumption , in particular for manufacturing errors .when components are manufactured , the manufacturer often stops the processing once the component meets the specifications .it is therefore likely that the component will have properties close to the maximum errors , which is a very different distribution from a normal distribution where the parameter is most likely to be at the specified value .a uniform distribution of the errors is therefore often a better choice .stenflo ( 1994 ) showed that weakly polarizing mueller matrices can be written as where is the unity matrix and and that the product of such matrices can be approximated by their sum .therefore , rss can be applied to weakly polarizing and retarding elements .this result can be generalized to strongly polarizing and retarding elements written as and again .however , is not the unity matrix , and all matrix elements can be of order unity .the product of such matrices can then be approximated according to the resulting product becomes the sum of the product of error - free matrices and all the mueller matrix errors transformed using the ideal mueller matrices of elements before and after the current element . because the transformed errors are additive , we can rss these transformed mueller matrix errors and use a classic error budget approach to estimate the contribution of individual errors to the overall system .to make these tools useful , systematic errors that can not be expressed in terms of mueller matrices must be included , and calibration and data reduction errors must be added . a library of normalized standard deviation matrices for common polarimetric components and their respective parameters must be collected .the polarimetry error budget described here neglects the coupled effect of simultaneous errors in all design parameters .a monte carlo simulation avoids this drawback by considering simultaneous errors .however , such end - to - end simulations require substantial efforts and do not provide a direct insight into the main error contributors and how their effects can be balanced .albert , a. , 1972 , regression and the moore - penrose pseudoinverse ( new york : academic press ) azzam , r. m. a. , elminyawi , i. m. & el - saba , a. m. , 1988 , j. opt .am . a , 5 , 681 boger , j. k. , stokes , s. d. , bowers , d. l. , ratliff , b. m. & fetrow , m. p. , 2003, spie 5158 , 113 keller c. u. , 1996 , 164 , 243 keller c. u. , 2002 , in astrophysical spectropolarimetry , ed . j. trujillo - bueno , f. moreno - insertis & f. sanchez ( cambridge : cambridge university press ) , 303 stenflo , j. o. 1994 , solar magnetic fields ( dordrecht : kluwer ) del toro iniesta , j. c. & collados , m. 2000 , , 39 , 1637 tyo , j. s. , 2002 , , 41 , 619

Abstract: ground - based solar polarimetry has made great progress over the last decade . nevertheless , polarimetry is still an afterthought in most telescope and instrument designs , and most polarimeters are designed based on experience and rules of thumb rather than using more formal systems engineering approaches as is common in standard optical design efforts . here we present the first steps in creating a set of systems engineering approaches to the design of polarimeters that makes sure that the final telescope - instrument - polarimeter system is more than the sum of its parts .


many systems of current interest to the scientific community can usefully be represented as networks  .examples include the internet  and the world - wide web  , social networks  , citation networks  , food webs  , and biochemical networks  .each of these networks consists of a set of nodes or vertices representing , for instance , computers or routers on the internet or people in a social network , connected together by links or edges , representing data connections between computers , friendships between people , and so forth .one network feature that has been emphasized in recent work is community structure , the gathering of vertices into groups such that there is a higher density of edges within groups than between them  .the problem of detecting such communities within networks has been well studied .early approaches such as the kernighan lin algorithm  , spectral partitioning  , or hierarchical clustering  work well for specific types of problems ( particularly graph bisection or problems with well defined vertex similarity measures ) , but perform poorly in more general cases  .  to combat this problem a number of new algorithmshave been proposed in recent years .girvan and newman  proposed a divisive algorithm that uses edge betweenness as a metric to identify the boundaries of communities . this algorithm has been applied successfully to a variety of networks , including networks of email messages , human and animal social networks , networks of collaborations between scientists and musicians , metabolic networks and gene networks  .however , as noted in  , the algorithm makes heavy demands on computational resources , running in time on an arbitrary network with edges and vertices , or time on a sparse graph ( one in which , which covers most real - world networks of interest ) .this restricts the algorithm s use to networks of at most a few thousand vertices with current hardware .more recently a number of faster algorithms have been proposed  . in  ,one of us proposed an algorithm based on the greedy optimization of the quantity known as modularity  .this method appears to work well both in contrived test cases and in real - world situations , and is substantially faster than the algorithm of girvan and newman .a naive implementation runs in time , or on a sparse graph .here we propose a new algorithm that performs the same greedy optimization as the algorithm of  and therefore gives identical results for the communities found .however , by exploiting some shortcuts in the optimization problem and using more sophisticated data structures , it runs far more quickly , in time where is the depth of the `` dendrogram '' describing the network s community structure .many real - world networks are sparse , so that ; and moreover , for networks that have a hierarchical structure with communities at many scales , . for such networksour algorithm has essentially linear running time , .this is not merely a technical advance but has substantial practical implications , bringing within reach the analysis of extremely large networks .networks of ten million vertices or more should be possible in reasonable run times .as an example , we give results from the application of the algorithm to a recommender network of books from the online bookseller amazon.com , which has more than vertices and two million edges .modularity  is a property of a network and a specific proposed division of that network into communities .it measures when the division is a good one , in the sense that there are many edges within communities and only a few between them .let be an element of the adjacency matrix of the network thus : and suppose the vertices are divided into communities such that vertex  belongs to community  .then the fraction of edges that fall within communities , i.e. , that connect vertices that both lie in the same community , is where the -function is 1 if and 0 otherwise , and is the number of edges in the graph .this quantity will be large for good divisions of the network , in the sense of having many within - community edges , but it is not , on its own , a good measure of community structure since it takes its largest value of 1 in the trivial case where all vertices belong to a single community .however , if we subtract from it the expected value of the same quantity in the case of a randomized network , we do get a useful measure .the degree  of a vertex  is defined to be the number of edges incident upon it : the probability of an edge existing between vertices and if connections are made at random but respecting vertex degrees is .we define the modularity  to be    \delta(c_v , c_w ) .\label{modularity}\ ] ] if the fraction of within - community edges is no different from what we would expect for the randomized network , then this quantity will be zero .nonzero values represent deviations from randomness , and in practice it is found that a value above about 0.3 is a good indicator of significant community structure in a network .if high values of the modularity correspond to good divisions of a network into communities , then one should be able to find such good divisions by searching through the possible candidates for ones with high modularity . while finding the global maximum modularity over all possible divisions seems hard in general , reasonably good solutions can be found with approximate optimization techniques .the algorithm proposed in  uses a greedy optimization in which , starting with each vertex being the sole member of a community of one , we repeatedly join together the two communities whose amalgamation produces the largest increase in  . for a network of vertices ,after such joins we are left with a single community and the algorithm stops .the entire process can be represented as a tree whose leaves are the vertices of the original network and whose internal nodes correspond to the joins .this dendrogram represents a hierarchical decomposition of the network into communities at all levels .the most straightforward implementation of this idea ( and the only one considered in  ) involves storing the adjacency matrix of the graph as an array of integers and repeatedly merging pairs of rows and columns as the corresponding communities are merged . for the case of the sparse graphs that are of primary interest in the field, however , this approach wastes a good deal of time and memory space on the storage and merging of matrix elements with value 0 , which is the vast majority of the adjacency matrix .the algorithm proposed in this paper achieves speed ( and memory efficiency ) by eliminating these needless operations .  to simplify the description of our algorithmlet us define the following two quantities : which is the fraction of edges that join vertices in community  to vertices in community  , and which is the fraction of ends of edges that are attached to vertices in community  . then , writing , we have , from eq .( [ modularity ] )     \sum_i\delta(c_v , i)\delta(c_w , i)\nonumber\\  & = & \sum_i \biggl [ { 1\over2 m } \sum_{vw } a_{vw } \,\delta(c_v , i)\delta(c_w , i )     \nonumber\\  & & \qquad { } - { 1\over2m}\sum_v k_v \,\delta(c_v , i )       { 1\over2m}\sum_w k_w\delta(c_w , i ) \biggr]\nonumber\\  & = & \sum_i ( e_{ii } - a_i^2).\end{aligned}\ ] ] the operation of the algorithm involves finding the changes in that would result from the amalgamation of each pair of communities , choosing the largest of them , and performing the corresponding amalgamation .one way to envisage ( and implement ) this process is to think of network as a multigraph , in which a whole community is represented by a vertex , bundles of edges connect one vertex to another , and edges internal to communities are represented by self - edges .the adjacency matrix of this multigraph has elements , and the joining of two communities and corresponds to replacing the and rows and columns by their sum . in the algorithm of  this operation is done explicitly on the entire matrix , but if the adjacency matrix is sparse ( which we expect in the early stages of the process ) the operation can be carried out more efficiently using data structures for sparse matrices .unfortunately , calculating and finding the pair with the largest then becomes time - consuming .  in our new algorithm , rather than maintaining the adjacency matrix and calculating  , we instead maintain and update a matrix of value of .since joining two communities with no edge between them can never produce an increase in  , we need only store for those pairs that are joined by one or more edges . since this matrix has the same support as the adjacency matrix , it will be similarly sparse , so we can again represent it with efficient data structures .in addition , we make use of an efficient data structure to keep track of the largest .these improvements result in a considerable saving of both memory and time .  in total, we maintain three data structures :  1 .  a sparse matrix containing for each pair of communities with at least one edge between them .we store each row of the matrix both as a balanced binary tree ( so that elements can be found or inserted in time ) and as a max - heap ( so that the largest element can be found in constant time ) .2 .  a max - heap containing the largest element of each row of the matrix along with the labels of the corresponding pair of communities .3 .  an ordinary vector array with elements  .  as described above we start off with each vertex being the sole member of a community of one , in which case if and are connected and zero otherwise , and .thus we initially set and for each  .( this assumes the graph is unweighted ; weighted graphs are a simple generalization  . )our algorithm can now be defined as follows .  1 .calculate the initial values of and according to ( [ eq : qinit ] ) and ( [ eq : ainit ] ) , and populate the max - heap with the largest element of each row of the matrix .2 .  select the largest from , join the corresponding communities , update the matrix , the heap and ( as described below ) and increment by .repeat step 2 until only one community remains .our data structures allow us to carry out the updates in step 2 quickly .first , note that we need only adjust a few of the elements of .if we join communities and  , labeling the combined community  , say , we need only update the row and column , and remove the row and column altogether .the update rules are as follows .  if community is connected to both and , then if is connected to but not to  , then if is connected to but not to , then   note that these equations imply that has a single peak over the course of the algorithm , since after the largest becomes negative all the can only decrease .  to analyze how long the algorithm takes using our data structures , let us denote the degrees of and in the reduced graph i.e ., the numbers of neighboring communities as and respectively .the first operation in a step of the algorithm is to update the row .to implement eq .( [ eq : both ] ) , we insert the elements of the row into the row , summing them wherever an element exists in both columns .since we store the rows as balanced binary trees , each of these insertions takes time .we then update the other elements of the row , of which there are at most , according to eqs .( [ eq : justi ] ) and ( [ eq : justj ] ) . in the row, we update a single element , taking time , and there are at most values of for which we have to do this .all of this thus takes time .we also have to update the max - heaps for each row and the overall max - heap . reforming the max - heap corresponding to the rowcan be done in time  .updating the max - heap for the row by inserting , raising , or lowering takes time . since we have changed the maximum element on at most rows , we need to do at most updates of , each of which takes time , for a total of .  finally ,the update ( and ) is trivial and can be done in constant time .since each join takes time , the total running time is at most times the sum over all nodes of the dendrogram of the degrees of the corresponding communities .let us make the worst - case assumption that the degree of a community is the sum of the degrees of all the vertices in the original network comprising it . in that case, each vertex of the original network contributes its degree to all of the communities it is a part of , along the path in the dendrogram from it to the root .if the dendrogram has depth  , there are at most nodes in this path , and since the total degree of all the vertices is  , we have a running time of as stated .we note that , if the dendrogram is unbalanced , some time savings can be gained by inserting the sparser row into the less sparse one .in addition , we have found that in practical situations it is usually unnecessary to maintain the separate max - heaps for each row .these heaps are used to find the largest element in a row quickly , but their maintenance takes a moderate amount of effort and this effort is wasted if the largest element in a row does not change when two rows are amalgamated , which turns out often to be the case .thus we find that the following simpler implementation works quite well in realistic situations : if the largest element of the row was or and is now reduced by eq .( [ eq : justi ] ) or ( [ eq : justj ] ) , we simply scan the row to find the new largest element .although the worst - case running time of this approach has an additional factor of  , the average - case running time is often better than that of the more sophisticated algorithm .it should be noted that the hierarchies generated by these two versions of our algorithm will differ slightly as a result of the differences in how ties are broken for the maximum element in a row .however , we find that in practice these differences do not cause significant deviations in the modularity , the community size distribution , or the composition of the largest communities .over the course of the algorithm ( the axis shows the number of joins ) . its maximum value is , where the partition consists of communities . ]the output of the algorithm described above is precisely the same as that of the slower hierarchical algorithm of  .the much improved speed of our algorithm however makes possible studies of very large networks for which previous methods were too slow to produce useful results . herewe give one example , the analysis of a co - purchasing or `` recommender '' network from the online vendor amazon.com .amazon sells a variety of products , particularly books and music , and as part of their web sales operation they list for each item a the ten other items most frequently purchased by buyers of a. this information can be represented as a directed network in which vertices represent items and there is a edge from item a to another item b if b was frequently purchased by buyers of a. in our study we have ignored the directed nature of the network ( as is common in community structure calculations ) , assuming any link between two items , regardless of direction , to be an indication of their similarity .the network we study consists of items listed on the amazon web site in august 2003 .we concentrate on the largest component of the network , which has items and edges .the dendrogram for this calculation is of course too big to draw , but fig .[ fig : q ] illustrates the modularity over the course of the algorithm as vertices are joined into larger and larger groups .the maximum value is , which is high as calculations of this type go  and indicates strong community structure in the network .the maximum occurs when there are communities with a mean size of items each .[ fig : visual ] gives a visualization of the community structure , including the major communities , smaller `` satellite '' communities connected to them , and `` bridge '' communities that connect two major communities with each other .  [ cols="^,>,<",options="header " , ]   looking at the largest communities in the network , we find that they tend to consist of items ( books , music ) in similar genres or on similar topics . in table [ table : labels ] , we give informal descriptions of the ten largest communities , which account for about 87% of the entire network .the remainder is generally divided into small , densely connected communities that represent highly specific co - purchasing habits , e.g. , major works of science fiction ( items ) , music by john cougar mellencamp ( items ) , and books about ( mostly female ) spies in the american civil war ( items ) .it is worth noting that because few real - world networks have community metadata associated with them to which we may compare the inferred communities , this type of manual check of the veracity and coherence of the algorithm s output is often necessary .  , which corresponds to an exponent of for the raw probability distribution . ]one interesting property recently noted in some networks  is that when partitioned at the point of maximum modularity , the distribution of community sizes  appears to have a power - law form for some constant  , at least over some significant range .the amazon co - purchasing network also seems to exhibit this property , as we show in fig .[ fig : distribution ] , with an exponent .it is unclear why such a distribution should arise , but we speculate that it could be a result either of the sociology of the network ( a power - law distribution in the number of people interested in various topics ) or of the dynamics of the community structure algorithm .we propose this as a direction for further research .we have described a new algorithm for inferring community structure from network topology which works by greedily optimizing the modularity .our algorithm runs in time for a network with vertices and edges where is the depth of the dendrogram . for networks that are hierarchical , in the sense that there are communities at many scales and the dendrogram is roughly balanced , we have . if the network is also sparse , , then the running time is essentially linear , is considerably faster than most previous general algorithms , and allows us to extend community structure analysis to networks that had been considered too large to be tractable .we have demonstrated our algorithm with an application to a large network of co - purchasing data from the online retailer amazon.com .our algorithm discovers clear communities within this network that correspond to specific topics or genres of books or music , indicating that the co - purchasing tendencies of amazon customers are strongly correlated with subject matter. our algorithm should allow researchers to analyze even larger networks with millions of vertices and tens of millions of edges using current computing resources , and we look forward to seeing such applications .the authors are grateful to amazon.com and eric promislow for providing the purchasing network data .this work was funded in part by the national science foundation under grant phy-0200909 ( ac , cm ) and by a grant from the james s. mcdonell foundation ( mejn ) .j. m. kleinberg , s. r. kumar , p. raghavan , s. rajagopalan , and a. tomkins , the web as a graph : measurements , models and methods . in _ proceedings of the international conference on combinatorics and computing _ , number 1627 in lecture notes in computer science , pp .118 , springer , berlin ( 1999 ) .community structure is sometimes referred to as `` clustering '' in sociology or computer science , but this term is commonly used to mean something else in the physics literature  , so to prevent confusion we avoid it here .we note also that the problem of finding communities in a network is somewhat ill - posed , since we havent defined precisely what a community is .a number of definitions have been proposed  , but none is standard .p. holme and m. huss , discovery and analysis of biochemical subnetwork hierarchies . in r. gauges , u. kummer , j. pahle , and u. rost ( eds . ) , _ proceedings of the 3rd workshop on computation of biochemical pathways and genetic networks _ , pp .39 , logos , berlin ( 2003 ) .j. r. tyler , d. m. wilkinson , and b. a. huberman , email as spectroscopy : automated discovery of community structure within organizations . in m.huysman , e. wenger , and v. wulf ( eds . ) , _ proceedings of the first international conference on communities and technologies _ ,kluwer , dordrecht ( 2003 ) .

Abstract: the discovery and analysis of community structure in networks is a topic of considerable recent interest within the physics community , but most methods proposed so far are unsuitable for very large networks because of their computational cost . here we present a hierarchical agglomeration algorithm for detecting community structure which is faster than many competing algorithms : its running time on a network with vertices and edges is where is the depth of the dendrogram describing the community structure . many real - world networks are sparse and hierarchical , with and , in which case our algorithm runs in essentially linear time , . as an example of the application of this algorithm we use it to analyze a network of items for sale on the web - site of a large online retailer , items in the network being linked if they are frequently purchased by the same buyer . the network has more than vertices and 2 million edges . we show that our algorithm can extract meaningful communities from this network , revealing large - scale patterns present in the purchasing habits of customers .