the nature and distribution of dark matter is one of the great mysteries of modern cosmology .although most of the matter in the universe may be dark , we can only directly observe baryons . by measuring statistical properties of visible universe ( galaxies and clusters )we can not infer statistical properties of the underlying dark matter without further assumptions .typically one has to assume that light in some way traces mass , but this has often been questioned and models have been proposed where galaxy formation is a non - local ( heyl et al .1995 ) or stochastic process ( dekel & lahav 1997 ) . until recently the only two direct tracers of matter distribution on large scales with existing data were velocity flows and cosmic microwave background ( cmb ) anisotropies .the former , while promising , still suffers from a small number of galaxies with accurate distances and a number of inherent biases present in any of the analysis methods ( ) . in recent yearscmb has emerged as the most promising way to measure the dark matter distribution , with the potential of measuring several cosmological parameters with a few percent accuracy over the next decade ( ) .the main advantage of cmb over other methods is that it traces the universe in linear regime , which makes the interpretation of the data relatively straightforward .another advantage is that it is relatively free of systematic effects and foreground emission seems to be subdominant over a large range of angular scales .however , cmb by itself can not determine all of the cosmological information ( ) .in particular , it does not directly probe the matter power spectrum , which although related to the cmb power spectrum , is sensitive to somewhat different physical processes and can provide valuable information by itself .for example , massive neutrinos have only a minor effect on the cmb spectrum and it will be rather difficult to distinguish between different values of neutrino mass from the cmb data alone .their effect on the matter power spectrum is significantly more important and by measuring it one could determine the value of the neutrino mass or set an upper limit .for this reason it is important to investigate other direct tracers of matter , which may allow one a more direct determination of matter power spectrum .it has long been recognized that gravitational lensing offers the possibility of tracing the dark matter distribution directly . as light propagates through the universe it is deflected by the inhomogeneities along the line of sight andthis causes a distortion in the images of background galaxies .the distortions are small and for each individual galaxy one can not separate them from an intrinsic ellipticity of the galaxy . only by averaging over several galaxies and assuming that the ellipticities of galaxies are uncorrelatedcan one detect this so called weak lensing effect .the effect has already been detected in clusters , where the higher projected mass density and typical radial pattern simplify the detection ( see for a review ) .the challenge for the future is to detect the weak lensing away from clusters in the field , where the distortions are significantly smaller , but the rewards potentially higher ( for some preliminary results see ) .although the typical distortion in the field is smaller the survey area can be larger ( specially with the advent of composite ccd cameras ) , which allows for a possibility of a statistical detection of structures on large scales through the two - point correlations , power spectrum or correlation function . often this will be the most useful information anyways , since on large scales , where the matter distribution is likely to be described as a gaussian random field , power spectrum contains all the information needed for a statistical description of matter distribution .power spectrum is the starting point for the extraction of various cosmological parameters .cosmological model predictions for two - point statistics , both correlation functions in real space and power spectrum in fourier space , have been investigated by a number of groups .the first calculations were those by and based on the work by .recently they have been extended to a general robertson - walker universe by , kaiser ( 1997 ) , stebbins ( 1996 ) , seljak ( 1996 ) and jain & seljak ( 1997 ) .the latter have also extended the calculations to a nonlinear regime using mapping of the power spectrum .most of these works have been theoretical and have only partially addressed the issue of extraction of the signal from the data .kaiser ( 1997 ) suggests a simple power spectrum estimator and shows that it gives an unbiased estimate of a convolved power spectrum .he also gives an estimate of the errors under the assumption that the data have gaussian distribution .another way to obtain such a quantity is to transform the data to fourier space first and then form a scalar quantity , as suggested by kaiser ( 1992 , 1997 ) .this method uses all the available information , but as we will show in this paper does not optimally weight the data which leads to a loss of information . propose a new statistic called mass - aperture ( previously introduced by for estimating the mass of clusters ) as a tool for estimating the power spectrum and higher order statistics .the statistic radially averages the tangential component of the shear .it was originally developed for clusters where for circular symmetry the shear pattern is indeed tangential . in the fieldthis is no longer the case and the motivation for using this statistic becomes less obvious .one reason for introducing this statistic is to obtain a localized scalar quantity from which third and higher moments can be extracted .both of these papers do not address the question of how efficiently the methods use the data and how much information is lost in the process of reducing the data to this form .we will show in this paper that there are methods which can formally be shown to be optimal for the power spectrum reconstruction .this is particularly important on large scales , where sampling variance ( finite number of modes of a given wavelength in the survey box ) is the dominant source of error on the power spectrum .another topic that was not addressed in the context of weak lensing so far is the determination of 3-d power spectrum from 2-d data sets .one method proposed by and applied to apm galaxy survey is to compute 2-d power spectrum and invert it to obtain 3-d power spectrum .the inversion procedure is nonlinear and requires an iterative scheme .the main problem with this analysis is the treatment of errors . on large scalesthe 2-d estimators have a large sampling variance and unless this is properly taken into account they contribute too much weight to the 3-d estimators and the inversion is unstable .we will show in this paper that there is a well defined answer how to perform this procedure that formally minimizes the error and provides a correct error estimate for the 3-d power spectrum . with weak lensing dataone can extract not only the power spectrum , but also its time evolution , if one uses photometric information to assign a distance to the background galaxies and we address how to extract this information as well .power spectrum is not the only information of interest when analyzing such data sets . on somewhat smaller scaleshigher order moments develop , which add additional information on the statistical properties of the process and can independently constrain some of the cosmological parameters . for example , ratio of the third moment to the square of second moment ( i.e. skewness ) of convergence depends in the second order perturbation theory only on the matter density and the shape of the power spectrum , but not on its amplitude .this may allow one to measure directly the cosmic density ( bernardeau et al .1997 , ) . because the measured shear is a tensorial quantity it has a vanishing skewness and can not be used for this test . to use as a statistic on which to apply this test , but as mentioned above this statistic does not make an optimal use of the data .we will show here that one can define a scalar quantity where all the information from the data is used and provided all the statistical properties are properly included this does not lead to a loss in information .one would also like to map large scale structures such as filaments , sheets and superclusters , which are of interest by themselves , both for cosmographical purposes and for learning about the cosmological model from their properties .again the question arises what is the optimal way to do this .we will show that for large scale structure this question has a well - defined answer , which moreover has a solution closely related to the optimal power spectrum reconstruction , so that both can be extracted within the same formalism .  in this paperwe develop the so - called minimum variance linear methods for projected mass reconstruction , 2-d and 3-d power spectrum estimation from the weak lensing data .we apply the methods to simulated data sets which demonstrate that the estimators are unbiased and reach the theoretical minimum variance limit .as such the estimators are superior to the estimators mentioned above and should be used at least on large scales , where the nearly gaussian distribution guarantees their optimal performance .although the methods developed here work best for reconstruction of lss , they can be applied to the cluster reconstruction as well , where they still minimize the variance in the class of linear reconstruction methods . in this casethey are no longer optimal for every application and instead can be viewed only as one of many possible linear filtering methods .other linear or nonlinear methods may be better depending on the application one has in mind .let us assume the observations consist of galaxies at angular positions with measured ratios of short to long axis and position angle of major axis .we can introduce ellipticity from which we can form a two - component entity of observable quantities .we arrange all the data into a component vector . because of gravitational lensing the `` true '' surface brightness ( i.e. the surface brightness one would see in the absence of any lensing )is mapped into the observed one , , where is the angular deflection of a photon caused by intervening mass distribution .this induces ellipticity correlations between the galaxies , information on which is contained in the symmetric deformation tensor , here is the gravitational potential and is the radial comoving distance with being the horizon distance . the radial distribution of background galaxies is described with the normalized distribution .the comoving angular distance introduced above can be expressed in terms of as curvature can be expressed as , where is the hubble constant and , the vacuum and matter densities , respectively .deformation tensor can be decomposed into its trace component convergence and two traceless components of the shear , , and .the observable ellipticities are related to these via the expression ( ; we are assuming that there are no critical lines where and between which this relation changes , see section 4 ) and so only a combination of and can be determined .however , since the distortions induced by lss are small , i.e. , one can in the first approximation ignore in the above expression and then shear becomes directly expressed in terms of the observed ellipticity .nonlinear corrections are addressed in more detail in section 4 .although there are 3 components of the deformation tensor they are in fact not independent , because they can all be expressed as a derivative of a single deflection potential ( equation [ shear ] ) .the quantity that we are mostly interested in is the convergence or surface mass density , which is related to the projected mass distribution over the window given in equation ( [ shear ] ) .we would like to reconstruct it and compute its power spectrum from the observables .the relation between the shear and convergence is most easily expressed in fourier space ( throughout the paper we will assume that the scales are sufficiently small for curvature of the sky to be unimportant and so fourier analysis to be adequate ; see for a generalization of this to all - sky measurements ) , so we first decompose shear and convergence into a fourier series , where is the power per mode and the sum is taken in the limit where the spacing between the modes .the two shear components can be similarly expanded and the relation between them and convergence in fourier space is ( kaiser 1992 ) where is the direction angle of mode .the data can be modelled in the form where is the underlying field in fourier space , is the noise vector , is a response matrix of the form for the two components of at position and is the number of modes we use in the model . instead of working with the complex response matrix and the complex field ( which satisfies the condition ) we can also work with real valued response matrix by replacing with and and replacing the complex with two real components .underlying field always has an infinite number of coefficients , but for computational reasons only a finite number of these can be included .the required number depends on the size of the survey and signal to noise ratio .the spacing between the modes is determined by the size of the survey , , since one can not resolve the modes more finely spaced than this .one way to see this is to embed the survey area into a box of size and impose periodic boundary conditions on the modes to make them orthogonal .these modes are spaced by and form a complete and orthonormal system in the box , so that any distribution can be expanded into this series .such an expansion will impose periodic boundary conditions on the data . to avoid spurios effects from periodic boundary conditionit is necessary to have the box somewhat larger than the survey and typically 20% zero padding on each side works well ( see section 4 ) .even better is to expand in a box of twice the linear size , which will completely eliminate the boundary condition problems but with an increase in the number of modes .the upper cutoff in the expansion is determined by the signal to noise ratio : because of the noise the data do not contain information about the structure on very small scales and do not need to be modelled .one can always include more modes than necessary or embed the survey into a larger box , so in this sense their actual number is not very important , except for computational reasons .in addition to the contribution from lss each measurement also has a noise contribution , which is a combination of intrinsic ellipticity of the galaxy ( dominant for bright and/or large sources ) and the measurement error ( significant for faint and/or small sources ) .we will assume that the noise covariance matrix is diagonal and given by , where is the transpose ( and conjugate for complex fields ) of the vector .measurement error can differ from one galaxy to another and intrinsic rms ellipticities can vary with galaxy types and/or redshift , so in principle the variances for different galaxies do not have to be equal and one can include this information if necessary . for a given galaxythe errors on , are assumed equal and uncorrelated .the fluctuations in the underlying field are parametrized with a diagonal covariance matrix , , where is the convergence power spectrum at , amplitude of mode .it can be expressed as an integral over the matter density power spectrum ( kaiser 1992 , ) \chi , \label{ps}\ ] ] where is the density power spectrum at conformal time and the expansion factor , related to the radial distance via the relation ^{1/2} ] , where is half the distance to the sources , from where the dominant contribution to the weak lensing signal is coming ( ) .this form minimizes the degeneracy between the amplitude of the power spectrum and the growth factor .we can ask how to extract from the data . expanding the likelihood function around the maximumgives the estimator of equations ( [ fisher])-([wffll ] ) , replacing with , the derivative of the correlation function with respect to , , where is the correlation function computed by using $ ] instead of .for a narrow dynamic range in distance and scale the growth factor will be degenerate with the shape of the power spectrum , which will be reflected by the covariance terms and one has to invert the matrix to obtain an estimate of the error on the growth factor that does not depend on the power spectrum .in this section we apply the formalism presented in previous section to estimate the power spectrum from simulated weak lensing data . to generate a simulated section of a fieldwe first make a random realization of the convergence in fourier space and compute the two shear components based on equation ( [ shearkappa ] ) .we then fourier transform the shear to real space and select a smaller area as our observed field ( we pay special attention not to select the area to be an integer fraction of the total simulated area to avoid any effects from periodic boundary conditions ) .we randomly generate the positions of galaxies with a specified surface density and randomly generate noise for each component of the shear from a gaussian distribution with a specified rms error .we add this value to the value of the shear at the galaxy position and use this map as our simulated data .we use and as our fiducial cosmological model and place all the galaxies at .these numbers should be typical for a several hour exposure at a 4 m class telescope with a limiting magnitude around .we then compute the power spectrum estimator using the expressions in previous section using the expressions in fourier space , because they require less computational time than the corresponding expressions in real space .figure [ fig4_1]a ( left ) shows the power spectrum measured on a 1 square degree field , using galaxies with rms ellipticity error of .the estimators are given as points with attached error bars computed from the fisher matrix , connected with a solid line .we compared the errors computed analytically with the errors obtained from monte carlo realizations and found excellent agreement .the errors on small scales are dominated by noise , while on large scales the dominant contribution comes from sampling ( cosmic ) variance .these errors are based on gaussian approximation , which underestimates them in the nonlinear regime . to estimate the scale where this may be important we plot on the figure both the input nonlinear power spectrum ( thick solid line ) and the linear power spectrum ( thin solid line ) .we see that on the largest scales the two agree and the sampling variance is correctly estimated , while on smaller scales the nonlinear power spectrum significantly deviates from the linear one and the sampling variance would be underestimated. however , where this becomes important noise will often be the dominant source of error .noise can be well approximated with the gaussian probability distribution because of the large number of galaxies , even if the intrinsic distribution of ellipticities is not gaussian .for the example we have chosen uniform dense sampling of galaxies prevents any significant amount of aliasing of small scale power and we find almost identical results ignoring aliasing terms in equations ( [ wffll ] ) .this is no longer the case when the sampling is sparse , as shown below .we can also test whether the signal is gravitational by rotating the galaxies by .the dashed curve in figure [ fig4_1 ] is the result of power spectrum estimation on the data rotated by . for perfect data this rotation would eliminate the gravity mode and only excite the vorticity mode ( ) .since there were no vorticity modes put in the simulation ( and indeed almost none are expected to be present in the real data ) one can hope to detect any systematic problems through this test . the dashed curve in figure [ fig4_1 ] is the result of power spectrum estimation on this rotated data .the resulting power spectrum is significantly lower than the one using original data , which confirms the usefulness of this test .note however that the power spectrum is not consistent with zero and there is an excess of power present in this mode .the reason is aliasing of power from the gravity mode to the vorticity mode .if this aliasing is not included in the analysis ( as it was not in this example ) it will artificially create power in this mode .we tried to analyze the data without putting the power into the gravity mode and found the recovered power spectrum consistent with zero .if however there is true power present in the data it will act as an additional noise for the vorticity mode and has to be taken into account .this example shows that one has to be careful to use the rotation test as a way to monitor the contamination of the data by some source that adds equal power to the two modes .one has to account for the aliasing first by computing the power spectrum of the gravity mode and use it to compute the total noise contribution in analogy with the bias term in equation ( [ wffll ] ) . \label{vorb}\ ] ] here is the correlation matrix ( equation [ corr ] ) and the response matrix for vorticity modes , which can be obtained from by replacing with .similarly , fisher matrix also has to be modified to account for this additional noise term .  on large scalesthe errors are completely dominated by sampling variance and as suggested by kaiser ( 1997 ) sparse sampling would be very useful to reduce the errors . as an example we computed the power spectrum by randomly placing 100 observations over an area of 100 square degrees ( figure [ fig4_1]b ) . in this exampleeach observation consists of a single exposure detecting 10000 galaxies , which can be grouped together into a single measurement with rms noise .an example would be a number of short ( a few minute ) exposures on a field , with a sampling factor of 4 , or a longer exposure with more galaxies per exposure and correspondingly lower sampling rate , both of which would only require a couple of nights with the composite ccd cameras .this choice of sampling is not necessary the optimal one for measuring the power spectrum , which in detail depends on the properties of noise , signal and aliasing power ( ) .probably even sparser sampling would give an even higher reward .even so it is clear that this strategy offers a significant promise to measure the power spectrum on very large scales where the potential rewards are largest .when the data are sparsely sampled it is important to properly remove the small scale aliasing : the power spectrum estimator without the subtraction of the aliasing term would be strongly biased towards the large values and since this term is only known to the extent that it has been measured well on small scales it can easily dominate the errors attached to the estimator .similarly , the fisher matrix had to be inverted to compare the estimators to the input ( rather than convolved ) power spectrum , otherwise the estimators would have been biased ( this effect can be minimized by estimating not the power spectrum directly , but only its departure from some fiducial power spectrum , as discussed in section 2.4 ) .this is again a consequence of strong mixing between the modes in different power spectrum bins .note that for such sparse sampling rotation test gives a lot of power in the vorticity mode , specially on small scales .for even sparser sampling the vorticity mode easily exceeds the input power spectrum because of the small scale aliasing .this has to be taken into account for this test to be a useful diagnostic of any nongravitational contribution .one can do this by using equation ( [ vorb ] ) when estimating the noise .[ fig4_1 ]  we also tried the 3-d power spectrum estimation using the approach outlined in section 2.3 .the results for the 1 square degree filled and 100 square degree sparse survey corresponding to examples in figure [ fig4_1 ] are shown in figure [ fig4_2 ] .we show the results for the convolved power spectrum and individual estimates are strongly correlated .this , logarithmic binning and the use of logarithmic scale make the results visually more impressive than in figure [ fig4_1]b , although the information content is the same .nevertheless , the results are quite impressive and an even larger ( and sparser ) survey would give an even bigger reward of determining the turnover in the power spectrum quite precisely .we verified the analytic error estimates from the fisher matrix by comparing them to monte carlo estimates and they agree quite well . on small scales where the nonlinear effects become important the sampling erroris underestimated because we used the gaussian approximation and one has to correct for that if necessary .in this section we apply the formalism developed in section 2 to reconstruct the projected dark matter distribution from weak lensing observations . we start with the reconstruction of large scale structure , where the techniques developed here work best .we then apply the same methods to the cluster reconstruction , pointing out its advantages and disadvantages and comparing them to other reconstructions . for this casewe also generalize them to include nonlinear effects and discuss how to treat the redshift distribution of the sources .we will assume throughout this section that the power spectrum of the data is known in advance , using the techniques developed in section 3 .the power spectrum is only known where the signal exceeds the noise , beyond that one has to assume its form by parametrizing it with a power law .we will use in the applications the actual power spectrum obtained from the simulated data directly .this of course can not be obtained with the real data , but it turns out that in practice the reconstruction is not very sensitive to this , because the spectral index of the data power spectrum differs significantly from the noise spectral index 0 and so the only important parameter is the scale where the signal power spectrum exceeds the noise . because we are expanding the data with modes that have periodic boundary conditions in a box it is better to use a somewhat larger box than the size of the observed field and zero pad the rest . for the examples here we used 10% padding on each side and in most cases this eliminated the spurious effects associated with periodic boundary conditions .if this is not sufficient one can always increase the size of zero padded area .we begin with the reconstruction of large scale structure . for this purposewe generated a projected density map from an n - body simulation .the simulated data in figure [ fig3_4 ] were obtained by randomly placing galaxies on a 1 square degree field .upper left panel of figure [ fig3_4 ] shows the input data , which have a lot of small scale structure . smoothing the input data at the scale where signal power spectrum equals the noise power spectrum produces upper right panel of figure [ fig3_4 ] .wf reconstruction is given in lower left panel of figure [ fig3_4 ] .we see that wf strikes the balance between the resolution and signal to noise ratio .the reconstruction is heavily smoothed because of the low signal to noise on small scales . on the other hand ,most of the reconstructed structures are real ( compare upper right with lower left panel ) , because wf preferentially keeps the modes that are above the noise .wf is thus particularly useful for identifying the large scale structures , such as filaments or superclusters in the data .rotation of the galaxies by eliminates most of the signal and the reconstructed field is now significantly lower than the typical structures in the reconstruction , as given in lower right panel of figure [ fig3_4 ] .this does not however completely eliminate the signal , because as discussed in section 3 aliasing of power from the gravity mode into the vorticity mode makes the latter nonvanishing even in the limit of small noise .the effect is rather small because of the large number of background galaxies , but would be more important if sampling was sparse .[ fig3_4 ]   the methods we developed in this paper are optimal for gaussian random fields , but they can also be applied to nongaussian situations , such as cluster reconstruction from weak lensing . in this casewf by definition still minimizes the variance in the class of linear estimators as shown in section 2.1 and so remains a useful reconstruction technique . to test itwe apply it to reconstruct a massive cluster obtained from an n - body simulation ( figure [ fig3_1]a ) .we have rescaled the projected mass density of the cluster in units of critical density , so that it is nearly critical .the cluster core is very concentrated and shows double nucleus structure .the comoving size of the box is 3.2 and we will take the area to be 10 across with 5000 galaxies in it .in this subsection we want to compare the reconstruction properties of various filtering methods and we will ignore the nonlinear corrections . those will be addressed in the next subsection .  there exist a number of cluster reconstruction techniques in the literature .squires & kaiser ( 1996 ) present a comprehensive review of most of these and conclude that their so - called maximum probability model is the best in the sense that it has the least amount of long - wavelength noise in the noise power spectrum .this is not surprising , since their model is identical to wf ( in fourier space ) , except that they advocate the prior power spectrum to be white noise with adjustable amplitude .the latter was taken to be larger than the noise , so that it does not suppress the modes on any scale ( the only reason add the theoretical prior is to regularize the inversion of matrix ) .because for long wavelength modes the actual signal is typically larger than the noise ( as shown in figure [ fig3_3 ] for the cluster used here ) the results will be similar to wf and both methods will reconstruct long wavelength modes without any suppression .however , on scales where the power spectrum drops below the noise the two methods differ : white noise prior does not filter the modes and is reconstructing mainly noise , while wf suppresses the modes , returning zero in the regions of low signal to noise .this is what one expects from a minimum variance method , since zero deviates from the true field less than the random noise does when the power spectrum of the latter exceeds the former .this is shown in figure [ fig3_3 ] for the simulated cluster . for short wavelengths white noise prioris no longer optimal in the sense of minimizing the variance and so in this sense wf is a minimum variance reconstruction among linear estimators even for a cluster .but as we show next this may not be the most desirable feature in the reconstruction and other methods ( such as the one investigated by seitz and schneider 1996 ) may result in a better overall reconstruction .  because the power spectrum has less power on small scales than the white noise one expects wf to result in a smoother reconstruction than the one assuming white noise as a theoretical prior .wf reconstruction of the cluster is shown in upper right panel of figure [ fig3_1 ] and indeed is rather smooth compared to the white noise prior wf reconstruction .note that the uniform weighting with variable theoretical variance ( equation [ wf ] ) shown in the middle left panel of figure [ fig3_1 ] gives almost identical results , but is much easier to compute .wf reconstruction shows a clear signature of the cluster above the background or above the rotation reconstruction in the middle right panel of figure [ fig3_1 ] , which does not show any evidence of a cluster .the noise is mostly eliminated from the map and most of the large structures one sees are real .these properties make wf to be a useful method even for cluster reconstruction .however , there is a high price to pay for this . by suppressing the small scale modes andso the noise across the field wf also suppresses the central peak of the cluster .some amount of smoothing is of course inevitable if the data do not have sufficient resolution , but wf tends to heavily smooth the data by suppressing all the small scale modes , regardless of their position .the nongaussian nature of the cluster center may give sufficient signal to noise to reconstruct it , but because all of the power on small scales is concentrated in this small region it may not give statistically significant excess above the noise in the power spectrum analysis .this in fact becomes exacerbated as we increase the area of the reconstruction , because by attempting to reconstruct the whole area ( with no power on small scales over most of the region ) wf will further suppress the central part of the cluster .for some applications one may be interested more in the central part of the cluster and willing to accept higher levels of noise at the outskirts of the cluster in which case wf will not be ideal method .[ fig3_3 ]  for cluster reconstruction it is therefore worth extending the family of linear estimators to make the prior power spectrum a free parameter .one example in this class is white noise prior advocated by squires & kaiser ( 1996 ) , which can reconstruct the structure on small scales .it should be stressed that with white noise prior one still has to impose a high frequency cutoff , otherwise the noise on small scales completely dominates the reconstruction . the way squires & kaiser ( 1996 ) impose this is by restricting themselves to a small number of modes in the expansion ,although the actual number was left unspecified .this method therefore amounts to a simple low - pass filtering , where the filter is a step function , as opposed to the wf , where the filter is more gradual and depends on the actual signal to noise ratio in the power spectrum of the data . if we extend the filtering scale beyond the scale where signal and noise power spectra are equal we obtain the reconstruction shown in lower right panel of figure [ fig3_1 ] for one particular choice of cutoff in .most of the reconstructed structure on small scale is noise , which clearly does not minimize the variance as defined in equation ( [ res ] ) and so is not optimal in this sense . on the other hand ,the central peak of the cluster is now significantly higher and its detection is more significant than in the case of wf . in this case reconstruction with smaller filtering scalemay be more acceptable , despite being significantly noisier .nonlinear methods such as maximum entropy method ( ) are likely to do even better in such applications . note that the spike at the corner of the white noise prior reconstruction is an artifact of the periodic boundary conditions and would be eliminated if one used 20% zero padding on each side of the box .this example shows that one has to be careful about the size of the zero padding area , which will be somewhat dependent on the type of the filter one is using .[ fig3_1 ]  wf reconstruction is also not unique and changes with the size of the observed area .if we reduce the size then the cluster becomes more important and the contribution from nongaussian modes to the power spectrum can dominate over the noise to smaller scales than in previous example , resulting in keeping more small scale modes in the reconstruction .an example of this is shown in figure [ fig3_2 ] , where we increased the physical size of the cluster by a factor of two , keeping the other parameters unchanged .this enhanced the signal power spectrum and added the small scale structure to the reconstruction .most of this structure outside the cluster is noise , but now this noise does not dominate the power spectrum and the minimum variance estimator is allowed to keep this structure and reconstruct the true structure in the center as well .it is clear from this discussion that wf is no longer optimal for every application and can only be treated as one example in a wider class of linear filters . for a conservative reconstruction of the whole measured areawf will in general outperform other methods .other low - pass filters may however be more appropriate for more specific applications .we may conclude that using the measured power spectrum from the data in wf is a useful starting procedure , which should be supplemented by other linear filters by varying the prior power spectrum .nonlinear methods , such as maximum entropy method , may provide an even better reconstruction of strongly nongaussian and point - like structures .[ fig3_2 ]   so far we ignored the term in relating observed ellipticity to underlying shear field by assuming that is small .when the clusters are nearly critical , such as in the core of the cluster we used above , the corrections become important and have to be included .because the final result of the reconstruction is convergence one can include the correction using an iterative scheme : one first reconstructs using equation ( [ wf ] ) ignoring the correction and in the next step one uses the value of reconstructed to obtain the components of the shear from the observables  . \label{ge}\ ] ] these corrections tend to suppress the reconstructed convergence , because the shear is reduced for the same measured ellipticity .a similar iterative scheme has been proposed by .assuming that is not close to unity then the error on the shear will continue to be dominated by the error on ellipticity and one can repeat the analysis as before with new values of the shear .otherwise one can improve this by adding additional term to the error matrix that corresponds to the error contribution from the reconstructed convergence , which is given by ( equation [ var ] ) [ \hat{\kappa}(\bi{\theta})-\kappa(\bi{\theta})]^{\dag}\rangle = \bi{r}_{\kappa}\bi{d}^{-1}\bi{r}_{\kappa}\ ] ] and properly add the two error contributions . if the cluster contains a critical line where then inside outer line ( but outside inner critical line ) one should replace with ( ) and one way to handle this case is to identify the critical lines at the previous iteration , replace with where necessary and repeat the iteration until the convergence .we implicitly assumed that the zero point of convergence is known , which is something that can not be reconstructed from the ellipticity measurements alone . when reconstructing lss this will not be a major problem , since the mean convergence fluctuates by a few percent at most ( the power from the modes larger than the size of the field is small ) .one can thus impose the mean convergence to vanish across the survey by setting the unreconstructed mode to zero . for cluster reconstructionit is more appropriate to set the zero point so that the convergence at the outskirts of the cluster is zero and this is somewhat more problematic when the cluster extends to the edge of the field . for the example we used in this paperthis is not so much of a problem , as one can separate the cluster from the surrounding area where the mean convergence should be small .one can in fact completely avoid this problem by using a local relation between derivatives of shear and convergence ( ) , although it does not appear straight - forward to implement this in the methods developed here .in section 3 we discussed how to handle the redshift distribution of the sources for the power spectrum estimation .one approach was to divide the galaxies into bins localized in distance and perform the analysis on each of these . the same approach should also be used for lss reconstruction , because the deflections come from a broad region along the line of sight and can only be determined in a statistical sense . in the case of cluster reconstructionthe situation differs from lss because the deflector is in a single lens plane and the distance to it is assumed to be known . instead of using convergence , which depends on the lens and source positions , it is better to model directly the surface density , which is a physical quantity independent of the redshift distribution .it is related to the convergence via the relation where and are the radial comoving distances to the source and lens , respectively . when the sources are all at the same distance one can work with and then obtain using the above relation .when the sources are not at the same distance then changes with for the same surface density .the observable can be expressed directly in terms of in analogy to equations ( [ e ] ) , ( [ ge ] ) as because the galaxy distance is not known is a stochastic variable and can be treated as the estimate plus additional noise term . assuming each galaxy has a known probability distribution of distances ( in the absence of any distance information this is just the overall galaxy distribution ) , we can define the mean critical density and the dispersion around it for each galaxy we use the mean critical density in equation ( [ ecrit ] ) as the estimate and the dispersion as an additional source of error in the observable . this additional noise term will be negligible for clusters at low ( where is rather insensitive to the redshift distribution of sources ) .even for high clusters this error is typically subdominant compared to the error on the ellipticity if the convergence is small .the only case when this error becomes important is for clusters that are close to critical , where the uncertainty on in the term of equation ( [ ecrit ] ) is amplified and can dominate over the ellipticity error , in which case we can add this error to the overall error budget of that galaxy .an alternative treatment of this problem has been discussed by for the case when the overall probability distribution of galaxy redshifts is given .the nice feature of our approach is that the mean distance and its error can be attached to each galaxy individually , which can take advantage of the multicolor photometric information . by assigning to each galaxy its photometric redshift and the corresponding error estimate we can compute for each galaxy separately its critical density and the dispersion around it .another example when this is important is when the measurement errors increase for fainter and presumably more distant sources . in this casethe more nearby galaxies carry more weight than expected from their number density and one can correct for this by assigning to each galaxy a mean distance according to its observed flux .in this paper we address the question of optimal power spectrum estimation and projected mass density reconstruction from weak lensing data .we show that for gaussian random processes both have a well defined answer , which are in fact related and can be obtained from the same calculation . for the power spectrumthe final answer is given by the unbiased estimator and its covariance matrix ( which also acts as a window function ) .the estimator is equivalent to the maximum likelihood method , but is significantly faster to compute .the covariance matrix of estimators includes the contributions from noise , sampling variance and aliasing .it is possible to invert from the 2-d power spectrum or to compute directly from the data the 3-d power spectrum , which will also be unbiased and minimum variance . finally , if we have distance information for the galaxies based on their photometric properties we can solve for both the power spectrum and its time dependence .the method can be used in the nonlinear regime , where it remains unbiased , but one has to compute the covariance matrix of the estimators , which now depend also on the reduced four point correlator .this can be either estimated from the data themselves , obtained from n - body simulations or computed using perturbation theory or hierarchical scaling relations .it is important to realize that even though the estimators were derived from the maximum likelihood method , the final result is always a weighted quadratic average of the data , where the weighting is the inverse of covariance matrix .this is the most expensive part of the calculation and can be replaced by some simpler weighting if necessary , such as uniform or inverse noise weighting . even in this casethe method will remain unbiased and will in fact remain close to optimal on small scales . similarly, if one wants to parametrize the power spectrum with some other parameters ( such as its amplitude and slope ) one can derive unbiased quadratic estimators specifically tailored for those parameters instead of going through the power spectrum .this is however not necessary , since in the process of reducing the data to the power spectrum no information has been lost if minimum variance estimator has been used .another topic of interest that we did not discuss in detail here is the question of optimal measurement of higher moments .one attempt how to measure these has been presented by schneider et al . (1997 ) using mass aperture statistic .as mentioned in the introduction this statistic uses the tangential component of the shear to obtain a quantity that is a scalar ( as opposed to shear which is a tensor ) , which is needed to define a nonvanishing third moment in real space .alternative way to obtain a scalar quantity is to reconstruct the convergence in fourier space as done in this paper . within the context of the methods presented here the general strategy for determining the higher momentsis in fact similar to the one leading to the minimum variance power spectrum methods , at least in the limit where the higher moments are small .one first multiplies the measurements with the inverse of correlation matrix .this is just a consequence of inverse variance weighting : if a given measurement has a large measurement error or it is strongly correlated with neighbouring points then it has to be downweighted , because it does not add new information to the data . on small scalesthis weighting reduces to the simple uniform weighting as discussed in section 2 .one then projects the data to fourier space to make a scalar quantity .because we reconstruct the fourier modes first one can attempt to measure higher order moments directly in fourier space ( e.g. bispectrum for the case of three - point statistic ) and this has some nice advantages ( e.g. ) .one can for example average over all triple products with , keeping the shape of the triangle the same and then vary this as a function of scale and angle between the wavevectors . as for the case of the power spectrum one has to compute the window for this statistic to make the estimator unbiased and to place errors on the estimates .alternatively , one can transform back to real space by applying operator to obtain a real space quantity that is a scalar , which can then be used to measure its skewness as a function of smoothing radius ( filtering scale ) . because all the operations are linear on the dataone can obtain the covariance matrix of the estimates and compute the window function needed to make the estimator unbiased and provide error estimate .  for projected mass density reconstruction the wf method investigated here works best on large scales , where the distribution is close to gaussian .in that case wf preferentially keeps the structure that is above the noise and filters out scales that are noise dominated . to apply wf one has to compute the power spectrum on the data first using the methods discussed above .when the structures become nonlinear wf remains a useful reconstruction technique and still minimizes the variance in the class of linear estimators , but tends to oversmooth the data in very dense regions such as clusters . in that caseit should be supplemented with linear filtering with a shorter smoothing scale or with one of the nonlinear methods , depending on the particular application one has in mind .this will be particularly important if one is searching for clusters in a random survey , where their positions are not given in advance and one wants to maximize the signal to noise for detecting such structures .baugh , c. m. , & efstathiou , g. 1994 , , 267 , 323 bernardeau , f. , van waerbeke , l. , & mellier , y. 1997 , a&a , 322 , 1 blandford , r. d. , saust , a. b. , brainerd , t. g. , & villumsen , j. v. 1991 , , 251 , 600 bond , j. r. , knox , l. , & jaffe , a. h. 1997 , astro - ph/9708203 bond , j. r. , efstathiou , g. , & tegmark , m. 1997 , astro - ph/9702100 bunn , e. f. , hoffman , y. , & silk , j. 1996 , , 464 , 1 connolly , a. j. , scabai , i. , szalay , a. s. , koo , d. c. , kron , r. c. , & munn , j. a. , , 110 , 2655 dekel , a. , & lahav , o. 1997 , in preparation fort , b. , mellier , y. 1994 , a&a review , r , 239 , 292 gazta~ naga , e. , & baugh , c. m. ( 1997 ) , in press , astro - ph/9704246 g ' orski , k. m. 1994 , , 430 , l85 gunn , j. e. 1967 , , 147 , 61 jungman , g. , kamionkowski , m. , kosowsky , a. & spergel , d. n. 1996 , phys .d , 54 , 1332 hamilton , a. j. s. , kumar , p. , lu , e. , & matthews , a. 1991 , , 374 , l1 hamilton , a. j. s. 1997 , , 289 , 285 heyl , j. s. , cole , s. , frenk , c. s. , & navarro , j. f. 1995 , , 274 , 755 .jain , b. , & seljak , u. 1997 , 484 , 560 kaiser , n. 1992 , , 388 , 272 kaiser , n. 1995 , , 439 , l1 kaiser , n. 1997 , , in press , astro - ph/9610120 kaiser , n. , squires , g. , fahlman , g. & woods , d. 1994 , in : clusters of galaxies , eds .f. durret , a. mazure & tran thanh van , editiones frontieres kendall , m. g. , & stuart , a. 1969 , ` the advanced theory of statistics ' , volume ii ( griffin , london ) kochanek , c. s. 1990 , , 247 , 135 miralda - escude , j. 1991 , , 380 , 1 narayan , r. , & nityananda , r. 1986 , , 24 , 127 oh , s. p. , spergel , d. n. , & hinshaw , g. 1997 , in preparation rybicki , g. , & press , w. h. 1992 , , 398 , 169 scoccimarro , r. , colombi , s. , fry , j. n. , frieman , j. a. , hivon , e. , & melott , a. 1997 , astro - ph/9704075 schneider , p , van waerbeke , jain , b. , & kruse , g. 1997 , astro - ph/9708143 seitz , c. , & schneider , p. 1995, a&a , 297 , 287 seitz , s. , & schneider , p. 1996 , a&a , 305 , 383 seitz , c. , & schneider , p. 1996 , a&a , 318 , 687 seljak , u. 1996 , , 463 , 1 seljak , u. 1997 , submitted to apj , astro - ph/9710269 squires , g. , & kaiser , n. 1996 , , 473 , 65 strauss , m. a. , willick , j. a. 1995 , physics reports , 261 , 271 stebbins .a. 1996 , preprint astro - ph/9609149 tegmark , m. 1997 , , 55 , 5895 tegmark , m. , taylor , a. , & heavens , a. 1997 , , 480 , 22 villumsen , j. v. 1996 , , 281 , 369 zaldarriaga , m. , spergel d. & seljak , u. 1997 , , 488 , 1 zaroubi , s. , hoffman , y. , fisher , k. , b. , & lahav , o. 1995 , , 449 , 446

Abstract: large - scale structure distorts the images of background galaxies , which allows one to measure directly the projected distribution of dark matter in the universe and determine its power spectrum . here we address the question of how to extract this information from the observations . we derive minimum variance estimators for projected density reconstruction and its power spectrum and apply them to simulated data sets , showing that they give a good agreement with the theoretical minimum variance expectations . the same estimator can also be applied to the cluster reconstruction , where it remains a useful reconstruction technique , although it is no longer optimal for every application . the method can be generalized to include nonlinear cluster reconstruction and photometric information on redshifts of background galaxies in the analysis . we also address the question of how to obtain directly the 3-d power spectrum from the weak lensing data . we derive a minimum variance quadratic estimator , which maximizes the likelihood function for the 3-d power spectrum and can be computed either from the measurements directly or from the 2-d power spectrum . the estimator correctly propagates the errors and provides a full correlation matrix of the estimates . it can be generalized to the case where redshift distribution depends on the galaxy photometric properties , which allows one to measure both the 3-d power spectrum and its time evolution .   # 1


many tools of nonlinear time series analysis are based on the theory of ( deterministic ) dynamical systems  , i.e. , the time evolution of the system under study is considered in some phase space spanned by the relevant dynamical variables . among others ,the recurrence of previous states in phase space  is a particular fundamental property of dynamical systems with a finite phase space volume ( e.g. , attractors of a dissipative system , hamiltonian systems with a bound phase space , or even stationary stochastic systems in finite time ) .the concept of recurrence implies that the dynamics of a system returns to an arbitrarily small neighborhood of any of its previously assumed states within a finite ( but possibly large ) amount of time . for deterministic - chaotic systems ,this is guaranteed by the invariance of the set which forms the support of the attractor  .recently , complex network representations have been proposed to characterize statistical properties of the underlying system associated with its geometry in phase space  . for this purpose , a proper transformation from the set of state vectors in phase space to a complex network representation is required . in this work ,we focus on the recurrence network ( rn ) approach , the vertices of the network are given by the individual state vectors sampled from a given trajectory , whereas network connectivity is established according to their mutual closeness in phase space ( i.e. , whether or not their mutual distance is smaller than a pre - defined threshold ) .mathematically , given two state vectors and ( where and denote time indices associated with two possibly different points and in time ) , the adjacency matrix of the rn is defined as where is the heaviside `` function '' , is the prescribed maximum distance , a norm in phase space ( e.g. , euclidean , manhattan , or maximum norm ) , and is the kronecker delta .rn analysis originates from the recurrence plot concept  and its basic assumption is , as the term indicates , the unambiguous presence of recurrence behavior .stationarity is a condition required by most tools of both linear and nonlinear time series analysis  , including the rn approach .a signal is ( strongly ) stationary if all joint probabilities of finding the system at some time in one state and at some later time in another state are independent of time within the observation period .the minimal requirement for most approaches is weak stationarity , that is , mean and variance of the underlying process are constant and the auto - covariances depend only on the time lag .  in turn , many real - world processes are non - stationary .for instance , climate or hydrological data often show seasonal variations .economic and financial time series typically exhibit ( irregular ) cycles of all orders of magnitude .non - stationary behaviors can be expressed in terms of trends , cycles , random walks , or combinations of the latter three .often , long - range dependence and self - similarity are involved .one classical example of a class of such non - stationary processes is fractional brownian motion ( fbm ) , which has long - range temporal correlations as its defining property  .specifically , for an fbm process , the variance scales as ( i.e. , non - stationarity in variance ) .the long range of the process is characterized by the hurst exponent when positively correlated ( persistence ) for , while suppressed ( anti - persistence ) for . corresponds to the classical brownian motion .non - stationarity provides a great challenge to both linear and nonlinear time series analysis , including complex network approaches to analyze time series data .there are some methods that are specifically tailored to cope with non - stationarity . among others ,detrended fluctuation analysis ( dfa )  and related techniques have been widely used for estimating the hurst exponent from non - stationary model data as well as real - world applications from various fields ( e.g. , ) .in contrast , regarding the rn approach , non - stationarity due to time - dependent system parameters can cause a systematic loss of recurrences .anyway , rns have been recently proposed to characterize fbm  .notably , the results of the latter study have been obtained only numerically and not explained theoretically so far . however , as we will demonstrate in the course of this work , they have rather limited physical interpretation .more generally , we will discuss how spurious results and pitfalls of rn analysis may be produced when this method is inappropriately applied to study fbm or other non - stationary stochastic processes , and that the results of are mainly of such spurious nature .this paper is organized as follows : in section [ sec : fbm ] , we discuss the construction of rns from non - stationary fbm data .we demonstrate that it is not possible to define generally applicable embedding parameters as required for a systematic investigation of the potential effect of on the rn properties .specifically , we provide numerical evidence that the latter properties ( for given embedding parameters ) depend explicitly on the system size , which generally does not apply to stationary systems provided that the sample size is large enough and sampling artifacts as well as transient behavior are avoided .subsequently , in section [ sec : fgn ] , we turn to the rn properties of the closely related fractional gaussian noise ( fgn ) , the incremental process associated with fbm , which is stationary . for the latter , the dependence of the network characteristics on is in contrast to fbm well - behaved. however , the considered embedding dimension still plays an important role when characterizing the rn structures .all results are summarized and further discussed in section [ sec : conclusions ] .the application of rns to the analysis of nonlinear time series implicitly assumes the validity of two fundamental assumptions : ( i ) the intrinsic model parameters and statistical characteristics of the system remain constant over time and ( ii ) the system under study is sufficiently sampled ( i.e. , time resolution and time series length are sufficient to approximate the system ) .the first assumption is equivalent to the condition of stationarity , while the second one mainly requires a proper coverage of phase space by a suitably embedded time series .both requirements are consequences of the fact that we approach the system s dynamics by a single finite time series , which is common to time series analysis problems .note that there have been attempts to characterize non - stationary systems by means of rn analysis using sliding windows approaches , which have provided interesting results regarding the presence of bifurcation or other qualitative changes in the dynamical regime  .however , these considerations have been related to systems with supposed time - varying parameters rather than non - stationary stochastic systems where the parameters are constant .therefore , this approach might not be helpful in the present context dealing with non - stationary variance .in the following , we will focus on two important algorithmic parameters of the rn approach , embedding dimension and delay .the impact of other parameters such as recurrence threshold , sampling rate , or even the selection of variables in multi - dimensional systems has been extensively discussed elsewhere  for deterministic systems , but not yet for stochastic ones . for the sake of brevity , we present only a brief corresponding discussion here .specifically , since we consider discrete - time univariate stochastic processes , only is relevant , but can be treated mostly alongside the theoretical considerations presented in  .  by means of conceptual considerations as well as numerical experiments , in the remainder of this section, we will address the following three questions : ( i ) can we use embedding techniques for fbm ( or , more generally , non - stationary stochastic processes ) ?( ii ) what are the intrinsic limitations of this approach ? ( iii ) which implications do these limitations have for rn analysis ?   as the most prominent subject of recent studies involving rn analysis  , chaotic attractors exhibit some complex geometric structure in their respective phase space , motivating the term `` strange attractors '' .typically , this structure is associated with self - similar ( fractal ) characteristics .( notably , there are examples for strange non - chaotic attractors as well  . )strange attractors emerge in deterministic dynamical systems , and the resulting asymptotic set of state vectors approached by the system forms some finite object in phase space .the dynamical properties of the system and the geometric characteristics of the attractor are commonly closely interrelated  .taking this idea further , it is a natural approach to describe dynamical systems of whatever kind by a geometric object in some appropriately defined phase space .this is the basis of rn analysis , which takes the existence of such a phase space ( at least in an abstract sense ) as a fundamental requirement .given this fact , rn analysis may be applied if the available data series provides enough information to describe ( or approximate ) the geometric structure of the sampled trajectory sufficiently .specifically , transient dynamics has to be excluded , data length and sampling frequency need to be appropriate , and the data object in phase space needs to be dynamically invariant or at least bound in phase space with stationary properties .  given a scalar time series ( ) , in order to apply rn analysis we first have to convert the data into state vectors in some appropriately reconstructed phase space . a common method from dynamical systems theory to define such a phase space is time - delay embedding  .in fact , the concept of a phase space representation rather than a `` simple '' time or frequency domain approach is the hallmark of many methods of nonlinear time series analysis , requiring embedding as the first step . here , we define to obtain an -dimensional time - delay embedding of with embedding delay for obtaining state vectors in phase space  . it has been proven that for deterministic dynamical systems , the thus reconstructed phase space is topologically equivalent to the original space if , where is the fractal dimension of the support of the invariant measure generated by the dynamics in the true ( but often at most partially observed ) state space .note that can be much smaller than the dimension of the underlying original ( physical ) phase space spanned by all relevant system variables .  from a practical perspective , when analyzing a scalar time series of whatever origin , neither embedding dimension nor delay known a priori .the false nearest - neighbors ( fnn ) method  was introduced to derive a reasonable guess of how to choose based on studying whether or not proximity relations between state vectors are lost when the embedding dimension is successively increased .if a reasonable embedding dimension is found , all dynamically relevant coordinates of the system are appropriately represented , so that all proximity relationships are correct and not due to lower - dimensional projection effects . in a similar spirit, the first root of the auto - correlation function ( acf ) of a time series often yields a good estimate for .a more refined method is to use time - delayed mutual information  .  while the aforementioned approaches to determining and commonly work well for data from deterministic dynamical systems , applying them to fbm leads to severe conceptual problems :  on the one hand, we note that the concepts of a fractal dimension has two aspects when being applied to a stochastic process instead of a deterministic dynamical system . from the phase space perspective, the fractal dimension is commonly defined as some scaling property described by a parameter that converges to a fixed value as and is sufficiently high .this fact is used , for example , in the famous grassberger - procaccia algorithm for estimating the correlation dimension of chaotic attractors  . however , according to the latter viewpoint , stochastic behavior is characterized by an absence of such convergence , formally leading to .finite estimates of are spurious due to the finite amount of data used .the latter result is reasonable since an infinite amount of data ( i.e. , the innovations at each time step ) are necessary to fully describe the evolution of a stochastic process .  as an alternative perspective , the fractal dimension of a stochastic process is often defined via the fractal dimension of its graph . for a one - dimensional process, this graph is represented in the -plane , and its dimension is hence bound from above by .specifically , for fbm with a hurst exponent , it has been shown that , taking the different scaling behavior in association with the process self - similarity into account .however , the latter aspect is clearly distinct from the notion of fractal dimensions used in the phase space context .thus , from a conceptual perspective , the embedding dimension should be chosen infinitely large . in turn, finite will necessarily cause spurious results since the full complexity of the system s ( discrete ) trajectory is not captured .  on the other hand ,the embedding delay is not considered in the mathematical embedding theorems for deterministic dynamical systems .embeddings with the same embedding dimension but different are topologically equivalent in the mathematical sense  , but in reality a good choice of facilitates further analysis . if is small compared to the relevant internal time - scales of the system , successive elements of the delay vectors are strongly correlated .this leads to the practical requirement that the embedding delay should cover a much longer time interval than the largest characteristic time - scale that is relevant for the dynamics of the system .however , in fbm arbitrarily long time - scales are relevant due to the self - similar nature of the process .this makes finding a feasible value of a challenging ( and , regarding formal optimality criteria , even theoretically impossible ) task .  in summary, we emphasize that in the case of non - stationary fbm , the fundamental concepts of phase space reconstruction and low - dimensional dynamics do not apply ( not even approximately ) anymore .therefore , any attempt to applying rn analysis to fbm directly necessarily yields results that hold only for the particular embedding parameters chosen and the specific length of the given time series  .we will demonstrate some numerical results illustrating these points in more detail in the following .estimating the acf of a stationary time series at lag is straightforward as long as is small compared to the total length of the time series , .for stationary stochastic processes , the functional form and rate of decay of the acf depends on the specific properties of the process .specifically , for a stationary long - range correlated process , the acf decays like a power - law with the characteristic exponent being directly related with  .   and ( b ) the corresponding acfs .( c ) average acfs taken over 200 independent realizations of fbm with the same hurst exponent .different line colors correspond to different values of from to in steps of ( from bottom to top at small ) . in all cases ,the time series length has been set to .( d ) as ( c ) for .[ fig : fbm_acf ] ]  in contrast to this , for the non - stationary fbm sample estimates of the acf decay extremely slowly beyond the `` normal '' behavior of stationary long - range dependent processes , which can be seen clearly in fig .[ fig : fbm_acf ] ( in fact , the concept of acf is not appropriate for describing the serial dependence structure of non - stationary processes ) . specifically , we show three example trajectories of fbms with , and their corresponding nave acf estimates . due to the stochastic nature of the process , the de - correlation time ( which can be expressed as or , i.e. , the time lags after which the estimated acf has decayed to or , respectively )depends on the specific realization of the process ( fig .[ fig : fbm_acf]b ) .even more , the corresponding ensemble spread does not exclusively originate from the finite sample size , but is enhanced by the inherent non - stationarity of fbm .  taking an ensemble average over a variety of independent realizations , we numerically observe that the location of the first root of the estimated acf hardly depends at all on the hurst exponent , which is shown in fig .[ fig : fbm_acf]c. however , as expected from theoretical study of fbm , it appears to systematically increase as the length of time series is increased to ( fig .[ fig : fbm_acf]d , note the different scales in figs .[ fig : fbm_acf]c and d ) . more specifically ,if we extend the length of the realization by a factor of 4 , the first root of the acf estimate also shifts to a four times larger lag .irrespective of the sample size , the spectrum of the fbm process has a significant amount of energy in frequencies that are not much larger than ( i.e. , in the low - frequency part ) .this explains why the first root of the acf estimate appears at larger time lags as is increased .consequently , the de - correlation time increases for longer time series . from the viewpoint of time - delay embedding (given it is performed disregarding the conceptual concerns detailed above ) , this hampers the proper choice of the embedding delay . in turn, the increasing persistence yields an increase in and as well , as can be seen from the mutual offset of the different lines in fig .[ fig : fbm_acf]c , d .   in a two - dimensional reconstructed phase space with embedding delays ( a ) and ( b ) ( ) .[ fig : fbm_phasespace ] ]  to further illustrate the practical consequences of the observed behavior of the sample acf when using embedding techniques , fig .[ fig : fbm_phasespace ] displays the same realization of a fbm embedded in a two - dimensional space with different embedding delays .notably , the two embedding components are highly correlated for small but less correlated for larger , leading to an entirely different geometric shape of the data object in the reconstructed phase space .the same behavior will be necessarily observed in higher embedding dimensions . as a consequence , a `` practical '' choice of the embedding delay forfbm should be independent of , but depend on .the numerical results presented above suggest for and for , possibly generalizing to .this is a rather large value , clearly far larger than those used by liu _ et al ._  ( for ) .the determination of a reasonable embedding dimension is often achieved by the fnn method  .the criterion for the embedding dimension being high enough is that the fraction of false nearest - neighbors is zero or at least sufficiently small .figure [ fig : fgn_fnn]a displays our corresponding numerical results for fbm for three different lengths , which consistently suggest .   and different time series lengths ( black : ,blue : , and red : ) .[ fig : fgn_fnn ] ]   an appropriate choice of the recurrence threshold has attracted great interest in the literature on rns  .the most wide - spread procedure is fixing the resulting recurrence rate ( i.e. , the fraction of recurrences ) and adjusting accordingly . as a rule - of - thumb, is often taken between about 0.01 and 0.05 for typical rn sizes of a few thousand vertices  , presenting a trade - off between the necessity of avoiding a largely disconnected network ( too small ) and the interest in the geometric fine structure of the system in its phase space , which is hidden when considering too large spatial domains .the latter requirement has been more precisely formulated in , emphasizing on the empirically expected relationship for the rn s average path length , , which has been numerically confirmed .recently , suggested using the percolation threshold of the random geometric graph constructed from the given distribution of observed state vectors in phase space as a suitable lower bound to .as shown by , the scaling of the rn s average path length breaks down if falls below the limit for which the rn decomposed into disjoint components , which is a necessary consequence of the fact that the averaging involved in the calculation of is commonly considered only over pairs of vertices that are mutually reachable  . however , when disregarding shortest path - based rn characteristics , there is no reason why one should restrict oneself to connected - networks , since other graph properties are hardly affected by the presence of more than one component . in particular , requesting the existence of a single component can lead to rather large due to the presence of outliers in the data  , especially in case of stochastic processes .  in this spirit , we recommend fixing at some reasonable value instead of tuning according to the percolation threshold .notably , in this case results obtained for different data sets still correspond to different when they originate from independent realizations of stochastic processes .however , the problem of the dependence of some network measures on the number of edges in the rn is relieved in this case .note that for fbm , due to the non - stationarity in variance the spread of state vectors in any reconstructed phase space necessarily grows with the sample size .based on our discussion presented in the previous section , we conclude that the results recently presented in  hold only for the particular choices of the algorithmic parameters ( for instance , length of time series , embeddings etc ) , showing limited physical interpretations . moreover , using non - stationary time series data necessarily produces unreliable and spurious results .one solution to the problem could be transforming the process in a way so that it becomes stationary . in recent applications to non - stationary real - world time series  , the authors have removed non - stationarities in the mean by removing averages taken within sliding windows from the data . in the particular case of fbm , where non - stationary affects the variance , the underlying stochastic process can be transformed into a stationary one by a first - order difference filter , i.e. , by considering its increments .the transformed series is commonly referred to as fractional gaussian noise ( fgn ) in analogy with the classical brownian motion arising from an aggregation of gaussian innovations .notably , fgn retains the long - range correlations and gaussian probability density function ( pdf ) from the underlying fbm process . for illustration purposes , three independent realizations of fgn with the same characteristic hurst parameter shown in fig .[ fig : fgn_acf]a .visual inspection clearly suggests the absence of non - stationarity in both mean and variance .because of its stationarity , for fgn the estimated acf shows a much faster decay and less ensemble spread than for fbm ( fig .[ fig : fgn_acf]b ) . therefore , disregarding the conceptual limitations of this approach when considering stochastic processes , embedding parameters can be chosen more properly for fgn than for fbm . concerning embedding delay , one easily sees that is a natural choice for according to the classical acf criterion , since the corresponding process is anti - persistent .specifically , in this case the acf drops to negative value at lag one ( as shown in fig .[ fig : fgn_acf]c ) , i.e. , subsequent values are negatively correlated the defining property of anti - persistence . in contrast , for we use the de - correlation time as an estimator for embedding delay , which increases with rising as one would expect since larger indicates a longer temporal range of correlations .   for fgn obtained by differencing the previous fbms .in ( b ) , the additional green line represents the averaged acf over 200 independent realizations .[ fig : fgn_acf ] ]  as before , the embedding dimension is chosen via the fnn method . in fig .[ fig : fgn_fnn]b , we show the fraction of false nearest - neighbors as is varied . unlike for fbm , our results suggest that the optimal value rises with an increasing length of the time series . in general , considerably higher values of are suggested than for fbm , which matches the theoretical expectations more closely . however , due to the finite sample size , we still find a vanishing fnn rate at a finite embedding dimension , which is probably related to a lack of proper neighbors when high dimensions are considered .given a proper representation of the considered system by its phase space reconstruction , the rn properties can be computed analytically from estimates of the underlying -dimensional state density  . in this spirit , an appropriate representation requires that the sample size is sufficient to cover all relevant parts of phase space , and that the sampling interval is reasonably chosen ( i.e. , to avoid sampling times co - prime with natural frequencies of continuous - time systems ) . for fbm, the latter condition can not be fulfilled due to the non - stationarity of the process , whereas it is technically met for fgn processes .  making use of the analytical results of , we expect that the degree distribution of the obtained rns should be the same for any stationary process with gaussian pdf given the same embedding dimension .specifically , this distribution has a complex shape that is independent of ( note that we may fix the mean degree by selecting a given ) .in fact , this invariance is a direct consequence of the fact that the geometry of the data in phase space is not affected by when considering sufficiently de - correlated components , a requirement that has not been met by in their recent investigation of fbm as discussed above .we emphasize again that the above considerations require a stationary gaussian process and an embedding for which all components are as close as possible to being linearly independent .otherwise , dependences between the components of the embedding vector lead to a deformation of the data distribution in phase space and , hence , possibly different geometric properties such as a too small effective dimension ( i.e. , smaller than ) .   in , we have recently demonstrated that the rn characteristics transitivity and global clustering coefficient provide relevant information for characterizing the geometry of the resulted rns , which has been numerically supported for various deterministic - chaotic systems .however , given the theory presented in  , the corresponding considerations can be extended to any kind of process or , more generally , any kind of random geometric graph  with a given state density . here, we exemplify these considerations for the case of fgn and examine how the transitivity properties of rns arising from such stationary long - range correlated stochastic processes depend on the characteristic hurst exponent as well as the underlying algorithmic parameters .   and ( b ) global clustering coefficient for fgn on the hurst exponent for different embedding dimensions ( : , : , : , : ) , taken over 200 independent realizations and using a rn edge density of . the embedding delay has been kept at the same value for all realizations with the same according to the de - correlation time . in all cases , . [fig : fgn_trans_h ] ]   and global clustering coefficient on the embedding dimension for fgn with ( averages over 200 realizations ) for two different values of ( ( a ) : , ( b ) : ) .the dashed line corresponds to the expected analytical values for -dimensional gaussian processes . in all cases , .[ fig : fgn_trans_m ] ]  for , fig .[ fig : fgn_trans_h ] shows that for a given embedding dimension , both transitivity and global clustering coefficient do not depend on .following our above considerations , this is expected since the -dimensional gaussian pdf of the process does not depend on , and the components are sufficiently de - correlated so that any marked geometric deformation of the embedded data is avoided .hence , we construct rns from the same pdf in all cases . some minor deviation from the constant valuescan be observed at close to 1 , i.e. , close to the non - stationary limit case represented by -noise , which might be due to numerical effects since the corresponding processes are harder to simulate than such with moderate .  for ,the behavior changes markedly : both and rise with decreasing hurst exponent .the reason for this behavior is that is the recommended , but still not `` optimal '' embedding delay for anti - persistent processes .specifically , the closer approaches 0 , the stronger is the anti - correlation at lag one .this means that with the same embedding delay , the smaller the stronger are the mutual correlations between the different components of the embedding vector . as a consequence, the state vectors do not form a homogeneous -dimensional gaussian pdf with independent components in the reconstructed phase space , but are stretched and squeezed along certain directions , so that the resulting geometric structure appears significantly lower - dimensional than .  given that ( ) is related to a geometric notion of the global ( average local ) dimension of the data , a reduced dimensionality of the data object results in a positive bias of both properties , which is exactly what we observe here ( fig .[ fig : fgn_trans_h ] ) .following the latter considerations , it is also easy to explain why both and systematically decrease with increasing embedding dimension ( figs .[ fig : fgn_trans_h ] , [ fig : fgn_trans_m ] ) .specifically , for a random geometric graph in dimensions ( computed with the maximum norm as also used in this work ) , one can show analytically that  ( similar considerations apply to  ) . for a fixed sample size , however , this theoretical expectation is only met at low embedding dimensions , whereas we find a systematic upward bias of both and as increases ( fig .[ fig : fgn_trans_m ] ) .we explain the latter observation by the finite sample size together with the problem that proximity relationships become more ambiguous in higher dimensions when fixing a certain value of .therefore , it can be expected that the bias should be systematically reduced when using larger sample sizes together with smaller edge densities ( for the latter effect , cf .[ fig : fgn_trans_m]a , b ) .  it would be straightforward to extend this kind of analysis to other network measures , since the available analytical description of rns allows for their calculation as well  .we leave a corresponding discussion as a subject of future work .by a critical reassessment of previous work  , we have identified several sources of errors when applying recurrence network analysis ( or , in a similar way , other concepts based on recurrences in phase space ) to long - range correlated stochastic processes . in summary ,the main conclusions of this analysis are as follows :  a.  rn analysis is based on phase space concepts originated in the theory of deterministic dynamical systems. therefore , its potential application to stochastic processes requires special care .b.  the rn theory  holds only for stationary processes .a direct application of rn analysis to typical non - stationary processes ( in particular fbm ) therefore has to fail , since the pdf of the process in the considered phase space changes with time . without correcting for non - stationarity by a proper transformation of the series , the obtained results are commonly spurious .c.  a major problem associated with non - stationary processes is that embedding can not be properly defined . in particular , the necessary selection of an embedding delay is ambiguous since auto - correlation function and related measures of serial dependences are not well - defined anymore . d.  for stationary stochastic processes , an embedding delay can be formally estimated from the data. however , the problem of selecting an embedding dimension remains , since stochastic processes are ( in the viewpoint of dynamical systems theory ) infinite - dimensional .hence , any low - dimensional embedding of a stochastic process necessarily loses relevant information , which is a major cause of spurious results .  despite the aforementioned conceptual problems and pitfalls resulting thereof, rn can still be used for obtaining interesting information on stationary stochastic processes .drawing upon the interpretation of rns as random geometric graphs  in some reconstructed phase space , the network properties could in principle be computed solely from the multi - dimensional pdf of the embedded process .deviations from the expectations are related to statistical dependencies between the different embedding components as well as finite - sample and finite - scale effects .the latter are also relevant for deterministic - chaotic processes , where in turn the underlying pdf can often not be calculated or at least estimated with high accuracy . in this spirit , deriving information based on stochastic processes can indeed help by providing benchmarks for studies of deterministic dynamics.  in general , applying rn analysis to scalar measurements requires an appropriate choice of embedding parameters .we do not claim that all choices made in this work have been based on fully objective quantitative criteria .the concepts like de - correlation time and false nearest - neighbors applied in this work rather present heuristics capturing only some aspects relevant for obtaining a proper phase space reconstruction . in this spirit , the results reported in  are conceptually interesting but practically difficult to interpret . for systematic applications ,the choice of embedding parameters depends on the particular process under consideration and should involve careful statistical evaluation beyond visual inspection .  finally , we emphasize that for non - stationary systems , embedding parameters can not be properly defined in general , so that any rn analysis ( as well as other time series analysis techniques ) necessarily yields systematic errors .this particularly applies to fbm and related processes arising from an integration of stationary processes ( e.g. , fractional lvy motion , ( f)arima models , etc . ) . in such cases ,a proper transformation is required to remove the particular type of non - stationarity from the data .this can be achieved by additive detrending , phase adjustment ( de - seasonalization ) , difference filtering ( incrementation ) or other techniques , with the one mentioned last being the proper tool for the particular case of fbm transforming the original process into stationary fgn . applying rn analysis to the latter indeed provides meaningful results .it should be noted that this observation is consistent with some wide - spread conceptual ideas beyond successful methodological alternatives for non - stationary time series analysis such as dfa  , which commonly make use of detrending and/or time series differentiation / aggregation .a more systematic exploration of corresponding approaches in combination with recurrence - based techniques is general , and rn analysis in particular , could be an interesting subject of future work .yz acknowledges financial support by the nnsf of china ( grant nos .11305062 , 11135001 ) , the specialized research fund ( srf ) for the doctoral program ( 20130076120003 ) , the srf for rocs , sem , the open project program of state key laboratory of theoretical physics , institute of theoretical physics , chinese academy of sciences , china ( no.y4kf151cj1 ) , and the german academic exchange service ( daad ) .rvd has been funded by the federal ministry for education and research ( bmbf ) via the young investigator s group cosy - cc ( project no . 01ln1306a ) .

Abstract: long - range correlated processes are ubiquitous , ranging from climate variables to financial time series . one paradigmatic example for such processes is fractional brownian motion ( fbm ) . in this work , we highlight the potentials and conceptual as well as practical limitations when applying the recently proposed recurrence network ( rn ) approach to fbm and related stochastic processes . in particular , we demonstrate that the results of a previous application of rn analysis to fbm ( liu _ et al . , _ phys . rev . e * 89 * , 032814 ( 2014 ) ) are mainly due to an inappropriate treatment disregarding the intrinsic non - stationarity of such processes . complementarily , we analyze some rn properties of the closely related stationary fractional gaussian noise ( fgn ) processes and find that the resulting network properties are well - defined and behave as one would expect from basic conceptual considerations . our results demonstrate that rn analysis can indeed provide meaningful results for stationary stochastic processes , given a proper selection of its intrinsic methodological parameters , whereas it is prone to fail to uniquely retrieve rn properties for non - stationary stochastic processes like fbm .