the ability to produce large , unexpected changes represents one of the most remarkable dynamical features of complex systems .examples include crashes in financial markets , jams in traffic and data networks , and the collapse of the immune system in humans .the large changes arising in real - world systems might be exogenous and hence triggered by some event in the environment which is external to the system - however the most intriguing case concerns the _ endogenous _ large changes which seem to arise ` out of nowhere ' and can not be blamed on any particular external cause .the bursting of the dot - com bubble around april 2000 represents a recent finance - related example of an endogenous large change .the practical motivation for studying large changes or so - called ` extreme events ' is obvious : they are rare , but the damage they cause can be catastrophic .present practice in assessing the risk of such events is purely statistical .this has two shortcomings .first , these events are rare and hence reliable statistics are difficult to determine .second , large events such as crashes arise from subtle temporal correlations in the time - series and hence can last over many time - steps . such changes are therefore not easily captured by examining probability distribution functions involving a fixed number of time - steps  .the academic motivation is equally strong , since large changes provide a visible demonstration of the system s internal dynamics and correlations .reference  quotes bacon from novum organum : `` whoever knows the ways of nature will more easily notice her deviations ; and , on the other hand , whoever knows her deviations will more accurately describe her ways '' . in short , the largest changes will tend to ` scrape the barrel ' in some way : hence they may provide new insight into the structure of the barrel ( i.e. the system ) and contents ( i.e. the constituent parts and their interactions ) .  in this paper , we examine large changes using a simple , yet highly non - trivial , model for a complex system comprising a population of objects or ` agents ' repeatedly competing for a limited global resource . in particular , we study the many possible paths the multi - agent system may take in the vicinity of a possible large change .this ` many - worlds ' view allows us not only to predict the start and end of a large change in this system , but also to investigate how such a large change might be avoided .we find that the system can be ` immunized ' : by provoking a small response today , the future robustness of the system against large changes can be boosted .agent - based models are becoming a standard tool for investigating complex system behavior across many disciplines .typically each agent has access to a limited set of recent global outcomes of the system ; she then combines this information with her limited strategy set , chosen randomly at the start of the game ( i.e. quenched disorder ) in order to decide an action at the given timestep .these decisions by the agents feed back to generate the system s output .the minority game ( mg ) introduced by challet and zhang offers possibly the simplest paradigm for such a complex , adaptive system and has therefore been the subject of much research activity .most studies of the mg have focused on a calculation of both time and configuration ( i.e. quenched disorder ) averaged quantities - in particular the ` volatility ' , where is the standard deviation of the number of agents taking a given decision .our previous work has shown that the variation of this averaged as a function of memory size can be quantitatively explained in terms of ` crowd - anticrowd ' collective behavior . in common with other mg theories ,our crowd - anticrowd theory implicitly assumes both time - averaging and configuration - averaging .the approach of studying realization - averaged quantities is standard in physics .it yields meaningful results for a wide range of physical systems because of the natural self - averaging which occurs in many - particle systems over the time - scales and length - scales of a typical experiment .however it is arguably of limited use in the study of real - world complex systems .this is because the observed evolution of a complex system only provides a _ single _ realization of the many possible paths that the dynamics can take . a financial market , for example, evolves with the knowledge that a crash happened at a specific moment in the past : there is a definite pre- and post - crash dynamics which will tend to condition the possible future paths .for example , the future behavior of the financial markets is dictated in part by the spectre of the recent dot - com boom and bust . in the context of human evolution ,social revolutions dominate the subsequent dynamics within a society .consequently one would gain little insight into the present state of the world by including all possible scenarios from the past , such as ones where the dinosaurs did not become extinct or ones where us did not gain independence . with this in mind , we have recently introduced two alternative theoretical approaches to analyse the mg , both of which focus on the dynamics for a _ single _ realization of the quenched disorder . the first approach views the mg as a stochastically perturbed deterministic system .averaging over the sources of stochasticity , in particular the coin - tosses used to break ties in strategy scores , yields a set of deterministic mapping equations . hence , in this system any particular realization of the quenched disorder will give rise to a single path for the future dynamics . the second approach takes a complementary , stochastic viewpoint based on a markov chain analysis .it is this latter theoretical approach which we will build upon in the present paper , focusing on the effect that the stochasticity can have in creating alternative future paths in the system s evolution .a further distinguishing feature of the present work is therefore its ` many worlds ' view of the future evolution of a multi - agent system .the underlying motivation for such an approach is our desire to understand the degree to which information about an upcoming large change is already ` encoded ' in the system , and hence the degree to which the large change itself might be avoidable .the basic mg comprises an odd number of agents ( e.g. traders or drivers ) choosing repeatedly between option 0 ( e.g. sell or choose route 0 ) and option 1 ( e.g. buy or choose route 1 ) .the winners are those in the minority group ; e.g. sellers win if there is an excess of buyers , drivers choosing route 0 encounter less traffic if most other drivers choose route 1 .the outcome at each timestep represents the winning decision , 0 or 1 .a common bit - string of the most recent outcomes is made available to the agents at each timestep .the agents randomly pick strategies at the beginning of the game , with repetitions allowed .this initial strategy allocation represents a quenched disorder .each strategy is a bit - string of length which predicts the next outcome for each of the possible histories . after each turn, the agent assigns one ( virtual ) point to each strategy which would have predicted the correct outcome . at each turn of the game, the agent uses the most successful strategy , i.e. the one with the most virtual points , among her strategies .any ties between equally successful strategies are resolved by the toss of a coin .our grand canonical minority game ( gcmg ) was first presented in ref .the main generalizations from the basic mg are as follows : ( i ) an agent only participates if she possesses a strategy that has a score equal to , or greater than , a predefined threshold , and ( ii ) the strategy performance is only recorded over the previous timesteps .the gcmg hence incorporates the following real - world effects : ( i ) agents ( e.g. traders ) only participate if they have a certain level of confidence in the strategies that they possess , and ( ii ) the agents will tend to forget ( or consider irrelevant ) results that lie too far back in history .these seemingly trivial modifications of the basic mg yield a dynamical system with surprisingly rich dynamics .these dynamics depend strongly on whether the participants are fed real ( as opposed to random ) history strings , and on the nature of the quenched disorder corresponding to the initial conditions .we focus on the low regime because of the richer dynamics , however our formalism can be applied for all .the number of agents holding a particular combination of strategies can be written as a ( terms ) dimensional tensor , where is the total number of available strategies . for strategies per agent ,this is simply a matrix where the entry represents the number of agents who picked strategy and then .the strategy labels are given by the decimal representation of the strategy plus unity , for example the strategy 0101 for has strategy label 5 + 1=6 .this quenched disorder is fixed at the beginning of the game and is written using the full strategy space .the value of describes the prediction of strategy given the history , where is the decimal number corresponding to the -bit binary history string .hence denotes a prediction of option ` 0 ' while denotes a prediction of option ` 1 ' .  as discussed in sec .i , we are interested in the detailed dynamics of the game for a given realization of the initial quenched disorder .hence we imagine that a particular has already been chosen .since the game involves a coin - toss to break ties in strategy scores , this stochasticity also means that different runs for a given will also differ - we return to this point below . the number of traders making decision ( the ` instantaneous crowd ' ) minus the number of traders making decision ( the ` instantaneous anticrowd ' ) defines the net ` attendance '  ] is made up of two groups of traders at any one timestep :  \(a )  ] traders who act in an ` undecided ' way , i.e. they require the toss of a coin to decide which option to choose .this is because they have two ( or more ) tied , highest - scoring strategies _ and _ these give different responses to the history at that turn of the game .hence the attendance at timestep is given by =a_{d}[t]+a_{u}[t]\ \ \ .\ ] ] the state of the game at the beginning of timestep depends on the strategy score vector  and history .henceforth we will drop the variable from the notation , but note that it remains an implicit variable through the time - dependence of and .we also focus on strategies per agent , although the formalism can be generalized in a straightforward way . at timestep , is given exactly by )\mathrm{h}[s_{r}-r]\underline{\underline{\psi } } % _ { r , r^{^{\prime } } } \ ] ] where the symmetrized matrix with representing the quenched disorder .the element gives the number of agents picking strategy and then .the function  ] ' is standard notation to denote a binomial distribution in which there are trials with probability of obtaining a given outcome .having obtained  , it is now possible to calculate the probability that the next winning decision is a 0 , given knowledge of  and .this probability is given as follows : )\ ] ] we note that eqs .( 1)-(5 ) represent a generalization to the gcmg , of the equations presented in ref . to describe the basic mg within the markov chain formalism .the dynamics of the gcmg are rich , and exhibit large changes .this paper gives explicit results for one particular ( randomly - chosen ) crash .however we stress that this crash is typical , and that similar conclusions will hold for other such typical crashes .the same analysis can be applied to other realizations without loss of generality .emphasis is placed on looking at the many paths along which the system can evolve , starting well before the large change itself .the quenched disorder is defined by .the exact numerical form of is too cumbersome to show but can be obtained from the authors .the initial conditions are defined by the score vector  and the corresponding history state . for the purpose of calculating the values of and  in subsequent timesteps ,we also define the last  winning outcomes of the gcmg .the specific initial conditions employed were taken from a typical run of the game , once initial transients had disappeared .equation ( 5 ) states the probability that the next winning outcome is a 0 , given knowledge of  and .it is possible to go further than this , by stating the mean attendance given that the attendance is positive ( i.e. winning decision of 0 ) and/or the mean attendance given that the attendance is negative ( i.e. winning decision of 1 ) .note that given astate of  and , may be greater than 0 but less than 1 , and as such both winning decisions are possible for this state .if however has a value of 0 or 1 , then there is only one winning decision via which the complex system may evolve at that time - step .the system is therefore deterministic at this time - step .there are several scenarios for  and  that can be examined .if and  then the price change is given by the value of , hence only one winning outcome is possible . if and , then the winning outcome has equal probability of being 0 or 1 . in this casethe recorded attendance is zero : however the two possible winning outcomes are noted for the purpose of calculating the outcomes for subsequent timesteps .if and , then the attendance can take possible values , each with their associated probabilities calculated using simple binomial theory in a similar way to eq .the magnitude of the attendance at has no significance for the future states of the system at times .only information concerning the sign of the attendance is carried forward , via the winning outcome .it is therefore sufficient to record the attendance for that timestep as being the average possible value , i.e. if and .showing all possible price changes will not provide us with any extra useful information concerning the system - instead , showing the average value will enable us to present a less cluttered image of the many worlds ( see later ) . if and , then both winning outcomes 0 and 1 are possible . for ease of calculation we shall approximate the binomial distribution in using a normal distribution . hence we can extract two possible values for the attendance : one corresponding to a winning outcome of 0 , and the other a winning outcome of 1 . to calculate these attendances ,one should split the normal distribution into two parts with the dividing line at the point of cut - off in terms of determining the winning outcome .if the winning outcome is to be a 0 , then the corresponding attendance contributed by becomes the first moment of the distribution above the cut - off .figure 1 shows an example of this scenario , and indicates the two corresponding attendances that are to be calculated .we define a ` path ' as the trajectory of the complex system during one possible run of the game .the game is set up with agents , each with two strategies scored over the last timesteps .the agents have identical confidence levels given by .figure 2a shows all 949 different paths in terms of the cumulative attendance ( e.g. price in a financial market ) . that this complex system can take over 56 timesteps , starting from the given initial conditions .figure 2b indicates the probabilities of occurrence of the paths in one run of the game , as a function of the paths end points .figure 2c shows a measure of the spread in price - increment between all the different paths as a function of time .this measure of the spread in price - increment is defined as follows : p(x_{i})\ ] ] where is the price change between neighboring timesteps for a price path denoted by .the probability that path , given the initial conditions , is given by .we note that for the basic mg ,  would take a value of if the attendances were random . in the present gcmg case ,the random limit would be larger due to the fluctuation in the total number of agents playing at a given time - step .it becomes very difficult to resolve the different paths in fig .2a after timestep 40 , even though it is only the average , historically - distinct paths which have been shown .the thick line shows the weighted average of _ all _ these paths .[ equation ( 5 ) allows us to calculate the probability of each path being the actual path taken in a single run of the game ] .it is easy to identify by eye that a large change ( i.e. so - called drawdown or crash ) occurs between time - steps 33 and 50 .figure 2c shows that  takes a very small value at the start of the crash : this signifies that the price - changes are similar at this point in time , in terms of the direction and size of movement , for the large majority of separate paths .similarly at the end of the crash the quantity takes a very large value , signifying a point in time where there is great disparity in the sign and size of price - changes between separate paths .we note that  is zero for the first 12 steps , since there is only one path possible up until timestep 13 .figure 2c demonstrates that it is possible to identify the start of a crash ahead of time , by the collective movements of the ` parallel worlds ' . in the caseshown , the start of the crash is indicated by the collective crashes of the large majority of the many parallel worlds .furthermore , it is possible in this system to identify the point in time when the system is released from the crash by observing when there is significant disparity between the parallel - world path movements . starting from a time 33 steps prior to the start of the crash, it can be seen that a crash is highly likely .  despite being highly likely ,such a crash is not however inevitable . in order to gain insight into how the crash may be managed , reduced or even avoided, we now turn to investigate the differences between the paths which perform better / worse than the many - world average .we will take ` better than ' as being a path that has an end value that is numerically greater than that of the mean at timestep 56 ( c.f .2a ) - similarly for ` worse than ' .hence we shall split up the paths into these two corresponding groups : better - than - the - mean shall be labelled set a , and worse - than - the - mean shall be labelled set b. although the end of the crash is signified to be at timestep 51 by the observable , for the purpose of analytics the reference point from which to define paths belonging to set a ( i.e. better than average path ) and paths belonging to set b ( i.e. worse than average path ) is taken to be at timestep 56 .paths ending significantly far from the mean path will not be affected by a slight change in the reference point - only those paths within the vicinity will be affected . due to the large value that takes at the end of the crash , the trajectories during this period vary considerably with respect to each other , and those paths withinthe vicinity of the mean path may switch from being ` better than ' to ` worse than ' from one timestep to the next .for this reason , the reference point for the end of the crash is taken to be some convenient time after the timestep represented by a high value , such that the system is in a relatively settled state .figure 3a shows the probability tree of a path belonging to set a , for the first 33 timesteps .each branching in this figure corresponds to a branching in fig .measuring the cumulative sign of attendance ( i.e. price in a financial market ) at timestep 33 , one gets three values for the particular initial conditions : , and .it is interesting to segregate those paths that have a cumulative movement of , , and , and plot the equivalent graphs to those shown in fig .3a together with their weighted means .figure 3b shows those paths with a net movement of .figure 3c shows those paths with a net movement of .figure 3d shows those with a net movement of .the mean probability of ending up as a member of set a is _ greater _ for those paths with a net movement of as compared to those with .similarly , the mean probability of ending up as a member of set a is _ greater _ for those paths with a net movement of as compared to those with .  we next compare weighted histograms of the cumulative _ sign _ of attendance during a period prior to the crash , and a period after the start of the crash . this will reinforce our findings concerning the differences between the paths in set a and set b. to do this , we set the period prior to the crash as timesteps 11 to 35 , and the period during the crash as timesteps 36 to 56 .figure 4a shows the histogram for set a prior to the crash .figure 4b shows the histogram for set b prior to the crash .it can be seen that the paths in set b generally have _ greater _ cumulative signs of attendance than the paths in set a. figure 4c shows the histogram for set a during the crash , while fig .4d shows the histogram for set b also during the crash .now the paths in set b generally have _ smaller _ cumulative signs of attendance than the paths in set a.  we now examine the occurrence of ( m+1)-bit words contained in the sequence of winning outcomes during the period from timestep 11 to 35 , for the paths in set a and set b. [ studying the occurrence of ( m+1)-bit words is equivalent to studying the occurrence of a 0 or a 1 following an m - bit history ] .figure 5 shows the frequency of occurrence of an ( m+1)-bit word for set a , divided by that for set b. this indicates how much more likely a particular ( m+1)-bit word is in set a as compared to set b , prior to the crash .hence it is just over twice as likely that 000 will be followed by a 0 in a path belonging to set a , as compared to a path belonging to set b. the period preceding the crash can therefore be seen as one of ` imbalance ' or ` tension ' , which is subsequently released by the crash .figure 6 shows an de - bruijn graph .the edges represent transitions between history states .transitions having peak values above in fig .5 , are shown with darker arrows .transitions that are particularly weak are represented by dashed arrows .the paths which are members of set a , i.e. those which perform _ better _ than the mean , tend to have more _ down _ movements in their history prior to the crash .by contrast , those paths which are members of set b , i.e. those which perform _ worse _ than the mean , tend to have more _ up _ movements in their history prior to the crash .if one therefore wanted to avoid the crash in a run of the game , then it would be wise to force the system to undergo _ down _ movements prior to the crash , so as to relax the build up of any unwanted bias in the strategy scores .this would then keep the system to the left - hand side of the de - bruijn graph where the arrows are darker , as seen in figure 6 .a controlling force or ` complex systems manager ' could intervene at moments where there is little cost in order to manipulate a winning outcome , i.e. at time - steps where it may require as little as one agent to change the path of the system . in this way the controlling force could avoid the crash with minimal effort or investment .this result that a large change can be prevented by creating an earlier , smaller movement _ in the same direction _ , reminds us of the idea of ` immunization ' .injecting a small amount of a virus provokes a minor response which ` primes ' the system s defences against a much larger , and potentially fatal , response later on . in the same way ,a small crash induced today , can be used to prevent a much larger crash from arising later on .it has recently been shown that the large endogenous changes which arise in the gcmg complex system , exhibit a degree of predictability .information signaling an increased threat of a large change is ` encoded ' in the system s microstructure well in advance .the present results for the gcmg are therefore consistent with these earlier findings . in real - world complex systems, such encoding would clearly be far more complicated to unravel .however there is the hope that _ if _ the system is viewed with the right ` spectacles ' , such signals might be uncovered .we refer the reader to the large body of work of sornette et al .( see ref . and references therein ) for the fascinating proposal of a log - periodic signature prior to large changes in a range of physical and natural systems , including financial markets .finally , we would like to draw attention to the nature of the large change studied in this paper .this event is described by a convergence of the many ` world lines ' which project into the immediate future .this convergence arises in spite of the stochastic fluctuations which occur along each ` world line ' .it is this effect that gives the system its predictability , and enables us to identify the birth of the crash .we note that if an observer was limited to knowledge of just a single ` world line ' , then little could be said about the crash s appearance until it became completely obvious .irrespective of the fact that a crash arose , there is arguably a more fundamental ` event ' which occurred in the system s dynamics .this ` event ' is the seemingly spontaneous convergence of the diversity of world - lines into the same direction .this convergence underlies the robustness of the resulting large change and hence the inevitability of the crash - in short the crash happened because of ` fate ' as opposed to chance .this leads us to speculate that an ` event ' in which a wide range of world - lines converge , may be fairly universal in complex systems , and nature in general . in short, the reality that is observed in the physical world might correspond to robust ` events ' involving convergence of world - lines , and as such is relatively unaffected by external noise / fluctuations .we have investigated large changes in the grand canonical minority game ( gcmg ) .the gcmg provides a simple , yet highly non - trivial , model for a complex system comprising a population of objects or ` agents ' repeatedly competing for a limited global resource .we have focused on the many possible paths the multi - agent system may take in the vicinity of a possible large change .this ` many - worlds ' view allowed us not only to predict the start and end of a large change , but also to investigate how such a large change might be avoided .we uncover the remarkable result that the complex system can be ` immunized ' in order to protect it against large changes .in particular we find that by provoking a small response today in the system , we can protect it against much larger responses in the future .d. challet and y.c .zhang , physica a * 246 * , 407 ( 1997 ) ; _ ibid . _ * 256 * , 514 ( 1998 ) ; _ ibid . _ * 269 * , 30 ( 1999 ) ; b.w .arthur , am .papers and proc * 84 * , 406 , ( 1994 ) ; d. challet and m. marsili , phys .e * 60 * , r6271 ( 1999 ) ; d. challet , m. marsili , and r. zecchina , phys .lett . * 84 * , 1824 ( 2000 ) ; m. marsili , d. challet and r. zecchina cond - mat/9908480 ; m marsili and d. challet cond - mat/0102257 ; r. savit , r. manuca and r. riolo , phys . rev .lett . * 82 * , 2203 ( 1999 ) ; r. dhulst and g.j .rodgers , physica a * 270 * , 514 ( 1999 ) ; a. cavagna , j.p .garrahan , i. giardina and d. sherrington , phys .83 * , 4429 ( 1999 ) ; j.a.f .heimel and a.c.c .coolen , cond - mat/0012045 .m. hart , p. jefferies , n.f .johnson and p.m. hui , cond - mat/0003486 ; phys .e * 63 * , 017102 ( 2001 ) ; physica a * 298 * , 537 ( 2001 ) ; eur . j. phys .b 2001 * 20 * , 547 - 550 ; p. jefferies , m. hart , n.f . johnson and p.m. hui , j. phys .a : math . gen . * 33 * l409 ( 2000 ) .johnson , m.l .hart , p.m. hui and d. zheng , theoretical and applied finance .* 3 * , 3 , ( 2000 ) .this work on the gcmg was first presented at the apfa1 conference in dublin ( 1999 ) and at the first workshop of applications of statistical physics to financial analysis in palermo , 1998 . for a complimentary discussion of grand - canonical gamessee , f. slanina and y.c .zhang , physica a * 272 * , 257 ( 1999 ) .    fig .demonstration of the two possible values for the attendance given that and .note that the area under the distribution between and , is equal to the area between  and the top edge of the distribution .similarly the area between and , is equal to the area between  and the bottom edge of the distribution .              fig .6 . an de - bruijn graph .the edges show transitions between history states . those edges with peak values above 1.15 in fig . 5 , have darker arrows .the transitions that are particularly weak are represented by dashed arrows .

Abstract: complex systems can exhibit unexpected large changes , e.g. a crash in a financial market . we examine the large endogenous changes arising within a non - trivial generalization of the minority game : the grand canonical minority game ( gcmg ) . using a markov chain description , we study the many possible paths the system may take . this ` many - worlds ' view not only allows us to predict the start and end of a crash in this system , but also to investigate how such a crash may be avoided . we find that the system can be ` immunized ' against large changes : by inducing small changes today , much larger changes in the future can be prevented .   : 01.75.+m ; 02.50.le ; 05.40.+j ; 87.23.ge   : complex adaptive systems ; econophysics ; agent - based models   corresponding author . tel + 44 - 1865 - 272356 . fax + 44 - 1865 - 272400 .   e - mail address : michael.hart.ox.ac.uk ( michael l. hart )


the idea of using cluster surveys as probes of dark energy has generated considerable interest of late . as the largest virialized objects in the universe , it is thought that clusters may be less affected by unknown or poorly understood astrophysical mechanisms and it is hoped that it will be reasonably straightforward to compare observed catalogs with theoretical estimates .while many recent works have investigated the power of upcoming surveys and the challenges due to poorly understood selection effects , there has been relatively less attention in the literature to methods of analysis .it has been generally assumed that the problem can be separated into analyzing the differential number counts as a function of redshift and investigating the correlations ( either angular or three - dimensional ) in the cluster catalog .studies of galaxy surveys have traditionally used either the power spectrum or point correlation functions to construct likelihood estimates , and most studies of the galaxy cluster abundance have assumed poisson statistics .however , it has been recently emphasized that poisson statistics for galaxy cluster surveys may not be a particularly good assumption .the importance of sample variance considerations is two - fold .it broadens the probability distribution of the counts in cells , weakening constraints on the true number density , and the amount of this broadening is in principle directly calculable from theory , adding new information which in principle can be used to tighten constraints . as will be shown below, the effect of sample variance on cluster counts is not negligible and treating the counts as a poisson process is incorrect .  in this work, we will go back to the work of in galaxy clustering and update it to the context of galaxy cluster studies .in particular , we will derive the correct likelihood function to use for binned estimates of the cluster number counts and we write down an explicit expression for the likelihood of observing a particular galaxy cluster catalog as a function of theoretical models for the correlation structure of the density field .we will present some very preliminary tests using the catalogs from the hubble volume simulation , demonstrating the regimes where these methods are robust and where more work clearly needs to be done .we adopt the formalism of dodelson , hui , & jaffe ( 1997 ; hereafter dhj97 ) , which in turn closely follows the development of white ( 1979 ; hereafter w79 ) .all that follows below is clearly outlined in w79 and the interested reader is referred to that paper for a full exposition .the fundamental assumption is that the probability of observing an object at a particular location ( in the absence of any other knowledge ) is simply proportional to the number density of objects : where is the number density , is the survey weight at that point ( the probability of detecting an object if it was at that point ) , and is an infinitesimal volume around the position .  by similar reasoning , the probability of observing two objects at two points in space and can be expressed as dv_1  dv_2 \end{aligned}\ ] ] where is the two - point correlation function , indicating the excess probability ( over random ) of detecting two objects at the given positions .this reasoning can be extended to three objects : dv_1  dv_2 dv_3 \nonumber\end{aligned}\ ] ] with the three - point correlation function , and the sum refers to summing over all unique .obviously this can continue to even longer lists of objects .a given catalog will contain objects , and the probability of detecting those objects at those positions can be constructed from the hierarchy of correlation functions , but a very useful component of the catalog is that _ there are no objects detected at other points . _a quantity of interest is therefore the probability of detecting objects at observed and no other objects at the other positions .a recipe for incorporating this extra information was described in w79 . instead of using the functions that were used above, one can simply replace each occurrence of with , where this latter quantity is simply   it is nice to have an exact likelihood formulation for a given catalog , but a hierarchy of infinite sums can be a bit unwieldy .dhj97 noted that for galaxy surveys there can easily be collections on the order of terms , even in the limit of gaussianity and weak correlations .it will be shown below that galaxy cluster surveys the number of terms is quite manageable , at least in the limit of the cluster distribution being gaussian and for a reasonably sparse survey .cluster surveys can effectively be considered as perturbatively different from a poisson distribution , allowing the probability distributions to be calculated quite efficiently .  in the gaussian limit ,it is useful to consider a few : where is the survey volume and the have all been set to unity for simplicity .note that this is not a requirement and is a straightforward way to include survey non - uniformity .this allows an expression to be written down for observing a collection of points as a function of a theory which predicts the number density and two - point correlation function .to first order in the two - point function ( dhj97 ) , \nonumber\end{aligned}\ ] ]  in the limit of sample variance being negligible ( i.e. , in the limit where approaches ) , this approaches the likelihood one would expect for the number counts ( the first term and the first term in brackets approaches the poisson expression ) combined independently with the correlation function .the second term in brackets closely resembles the likelihood one would obtain for the correlation function assuming that the probability of observing a pair at a given pair of locations is a poisson process with mean given by . when sample variance is not negligible, however , the probability will not be cleanly separable .furthermore , the same formalism was used by w79 to write down a compact expression for the counts - in - cells probability : = { ( -nv)^n \over n ! } { d^n \over dn^n } exp[w_0(v ) ] \label{eqn : p_phi}\end{aligned}\ ] ] where indicates observing objects ( and only ) in a volume .  again , assuming weak correlations and working in the gaussian limit , the counts in cells probability can be derived .some details are derived in the appendix , but the result is straightforward .the probability of observing objects in a volume ( to first order in the clustering ) is = { x^n e^{-x } \over n!}\bigl[1- n^2 v^2 { \bar{w } \over 2}\bigl (  { n - n^2 \over x^2 } + 2{n \over x } -1 \bigr)\bigr ] \label{eqn : countsprob}\ ] ] where is again the observed number , , and is the mean correlation function over the survey geometry . in calculations in this paperwe use the second order expression to generate the figures , but a careful analysis of terms indicates that if the second order term is important then there is a good chance that all higher order terms must be included .  where might this expression be expected to break down ?in w79 it is clear that starts to become a challenging series when becomes of order 0.1 or higher .we should therefore expect that this likelihood calculation will be most effective in that case and that it is suspect outside that regime .the breakdown appears to arise because the assumption of gaussianity must always break down : the density field can not be negative . with sparse samplingthe probability of encountering a negative density is negligible and the assumption of gaussianity is sufficient .however , at higher number densities the intrinsic non - gaussianity becomes important .notice that this approach is straightforward to extend to objects that are tagged with an observable such as flux or richness .going back to eqn [ eqn : catprob ] , it should be sufficient to replace the initial number density with a sum over the individual number densities in each observable bin ( e.g. , theoretical number density as a function of optical richness or sz flux ) , and then keep in mind that each will depend on the mass of the object or pairs of object under consideration .the efficacy of this technique is currently a topic of investigation .we use the cluster catalogs derived from the hubble volume simulations to test the range of validity of the likelihood expression derived for the counts in cells .a straightforward test is to use the cluster catalogs , where evolution effects can be neglected .we further specialize to the simulation , where the transfer function is easy to reproduce .the simulation parameters are and and a box size of .the volume is divided into subregions of cubic geometry and the above expressions are tested for a variety of cell sizes and mass thresholds . the number of clusters within each cell is then used to construct the probability distribution of the counts in cells .mass thresholds were used to construct counts in cells for all objects above a threshold mass , with the mass defined as the number of particles ( each of mass ) found by the friends of friends algorithm with a linking parameter of 0.2 .the mean correlation function of the matter distribution was derived as where is the window function for the cells ( not related to the above ! ) . for cubic cells ,the window function is simply a product of functions .the mean correlation function of the galaxy clusters needs to be corrected for the bias function .this is non - trivial ; several fitting functions exist in the literature but typical errors appear to be on the order of 10% when applied in a regime where they have not been calibrated .we use the bias relation of with the mean bias for the cluster sample above a mass threshold derived using a mass function weighted bias factor .the mass function of was used , which has been calibrated using the simulation used here . in principle , given a mass correlation function one can estimate the bias as a function of an observable cluster property and use this to estimate the cluster mass as a function of cluster properties . however , current uncertainty in the theoretical understanding of bias as a function of mass in arbitrary cosmologies complicates any such undertaking .an example is shown in figure 1 . taking a relatively high threshold of and a cell size of 154 ,the counts in cells distribution can be synthesized .we average over ten random cell offsets to smooth the observed distribution and compare it to the exact calculation of eqn [ eqn : countsprob ] . in the limit of massive ( and therefore rare and sparse ) clusters the likelihood for the counts in cells is quite accurate .the assumption of a gaussian random field will break down when is of order ( w79 ) . to investigate the nature of the breakdown , we show in figure 2 the first breakdown for cells of size 100 mpc and 133 mpc . at higher masses ( i.e. , lower number densities )the theoretical result matches the simulations quite well .it is clear that there is noticeable deviation starting around .the mass thresholds were selected to straddle the transition . at yet lower masses , the theoretical result breaks down fairly comprehensively , developing bimodality .this is an indication that the theory is simply not well defined in this regime .it is impossible to set up a distribution of positive mass concentrations with vanishing correlation functions above the two - point function .as the mass threshold is lowered , clusters become more numerous . as the cell size decreases , the mean number per cell will decrease but the mean correlation function over the cell volume will increase . for reasonable power spectra it turns out that the number per cell is slightly more important in this case for the purposes of calculating , making it slightly advantageous to move to smaller cell sizes .  to investigate the regime where this likelihood approach can be trusted , a grid of masses and cell sizes was investigated .the quantity was calculated and the lines in figure 3 indicate the locus of pairs of and cell size that lead to . for a model with likelihood approximation breaks down below , with the breaking down around the same mass .however , this is a very steep function of the amplitude of the power spectrum .for example , using drops the threshold of applicability to below .this is particularly relevant because this corresponds to the amplitude of fluctuations at if .any survey for galaxy clusters at and targeting masses above a few is squarely is the regime where this likelihood calculation can be used .the preliminary ( and very highly idealized ) tests outlined above demonstrate that many cluster surveys can be considered to be in the regime where catalog likelihood methods will be readily applicable .tests on counts in cells at indicate that the non - gaussianity of the cluster distribution becomes important around masses of several times .applications of the cluster catalog likelihood ( eqn [ eqn : catprob ] ) will likely have similar problems at lower masses and may also have difficulties at small separations .this is a topic of ongoing investigation .the tests performed in this work used a simulation snapshot , rather than a lightcone selected volume .this was done to cleanly separate the possible complications and there is no conceptual problem with generalizing this to include redshift evolution .in fact , to generalize this one simply goes back to eqn [ eqn : countsprob ] and allows to be functions of and cosmological parameters , including the evolution of the relevant window functions .non - cubic cell volumes that are functions of cosmological parameters and redshift will certainly complicate the analysis , but all the results presented here should be directly applicable .there are several directions for more work in this area .tests on existing cluster catalogs will yield insight into exactly how hard it is to implement these ideas in practice .one of the lessons learned is that this method works best on sparse catalogs ( i.e. , massive clusters ) , so in existing data it would be most easily tested on the most massive x - ray cluster surveys .it is unfortunate that the method appears to break down in a regime where there are high quality local cluster surveys .  on the theoretical side ,an obvious next step is to apply these methods using clusters that have been additionally tagged with an observable such as x - ray or sz flux or optical richness . within this formalismsuch a modification is entirely straightforward , as all that is required are number densities and correlation strengths .incorporating scatter in something like a mass - observable relation would be in principle straightforward , as would allowing for non - trivial ( non power - law ) evolution in such relations .however , uncertainty in the bias relations as a function of cosmology and mass must be improved to realize the full potential of these methods .given that the breakdown in this formalism appears to coincide with the onset of non - linearity , it may be possible to extend these results to lower masses with some modelling of the effects of weak non - linearity .the excellent match between theory and simulations at higher masses demonstrates that non - linearity and non - gaussianity are not important in the high - mass regime .in this paper we outlined a likelihood approach to galaxy cluster surveys that is uniquely applicable to the case of massive galaxy clusters .essentially , for small enough sample and/or weak enough clustering strengths one can treat cluster catalogs as being expected to be drawn from a distribution that can be thought of as a poisson distribution with some perturbations . by virtue of being only perturbatively different from a poisson process , cluster catalogs lend themselves well to efficient likelihood estimation techniques that naturally include both shot noise and sample variance .this does not require an arbitrary conceptual breaking of the problem into `` counts '' and `` clustering '' but instead simply addresses the question of how likely it is to obtain a set of points given an assumed underlying correlation structure .  for galaxies ,the number of terms involved in the estimate is prohibitive , and it is also important to understand the higher order correlations .that does not appear to be the case for massive galaxy clusters .the gaussian weak correlation limit does a very good job of matching the counts in cells distribution for massive clusters in the hubble volume simulation . as the mass threshold is lower , and the number density therefore increases ,some disagreement arises , likely due to the number density becoming high enough that the distribution becomes sensitive to non - gaussianity .it should be emphasized that it is clearly not an advantage to have fewer clusters .one could always thin the number density of a sample until the catalog is in a regime where a catalog likelihood can be calculated .this would obviously be throwing away a huge amount of information and it seems unlikely that the gain in simplicity would be worth the loss of information .these techniques are straightforward to generalize to explicitly include inhomogeneous survey selection and differential number densities and clustering strengths as a function of galaxy cluster properties .  with galaxy cluster surveys now being used for cosmological parameter estimation, it is essential to include the effects of sample variance to obtain accurate parameter estimates , so obtaining accurate probability distributions for counts in cells should be a high priority .the techniques here are a step in that direction , showing that such accurate distributions are easily calculated for massive galaxy clusters .  as this article neared completion , an article appeared by hu & cohn discussing very similar ideas from a slightly different perspective .the results here largely agree with theirs in the aspects where they overlap .the calculation of the likelihood for counts in cells can be derived from equation [ eqn : p_phi ] with straightforward algebra and accounting in the limit of weak clustering .some of the details are spelled out below .define and using , the first step is to expand the exponential term as a taylor series . for convenience , define and the counts in cells probability distribution can be written   expanding the part inside the sum with the binomial expansion gives =  { ( -x)^n \over n ! } { d^n \over dx^n}\left (  \sum_{i=0}^{\infty }  \sum_{k=0}^{i }  { 1 \over k ! ( i - k ) ! } ( -x)^{i+k } \alpha ^k \right)\ ] ]  the effect of the derivatives is that all terms in the sums with are set to zero and all other terms are multiplied by : =  { ( -x)^n \over n ! } \left ( \sum_{i=0}^{\infty } \sum_{k = max(0,n - i)}^{i }  ( -1)^n { ( i+k ) ! \over k ! ( i - k ) ! ( i+k - n ) ! } \alpha ^k ( -x)^{i+k - n }  \right)\ ] ]  up to this point , the only assumption implicit in this analysis is that the correlation functions beyond the two - point function are negligible . to see the structure in the probability distribution, we now assume that is small and investigate the relevant terms in the above equation . in particular , for small the only important terms will be terms with small . this immediately identifies the important terms as being the ones with close to .to first order in , one can move to one lower value of : and to second order in we can continue down : with subsequent orders obtained similarly .these sums are straightforward to implement using software such as maple but can become tedious . in all that was implemented in this work the serieswas truncated at second order .the sums have an interesting property that each term seems to be of order , since is of the same order as . at each levelthe terms are actually of lower order in due to cancellations in the sums .for example , for the expression in brackets in equation [ eqn : secondorder ] reduces to , making this term roughly .nonetheless , it is clear that simply having small is not sufficient .for this formalism to be useful it is required that be small , at least , and it is preferable that be small to be sure that all higher order terms are negligible .it is apparent from the expression for that bad things are happening as is becoming large , since it is becoming dubious that the void probability function is bounded to be less than one ( i.e. , must be negative ) .this is likely a breakdown of the physical picture of poisson sampling of a gaussian random field , rather than a breakdown of any approximations .it is impossible to poisson sample with a negative mean , which a gaussian random field allows .the requirement that the density is non - negative necessitates some amount of non - gaussianity . with enough samples ,this non - gaussianity must manifest itself in the observed distribution .

Abstract: galaxy cluster surveys offer great promise for measuring cosmological parameters , but survey analysis methods have not been widely studied . using methods developed decades ago for galaxy clustering studies , it is shown that nearly exact likelihood functions can be written down for galaxy cluster surveys . the sparse sampling of the density field by galaxy clusters allows simplifications that are not possible for galaxy surveys . an application to counts in cells is explicitly tested using cluster catalogs from numerical simulations and it is found that the calculated probability distributions are very accurate at masses above several times at and lower masses at higher redshift .


in this paper we provide a new approach , along with extensions , to results in two important papers closely tied to the statistical analysis of fmri ( functional magnetic resonance imaging ) brain data .the first , by david siegmund and the late keith worsley  , treated the problem of testing for a spatial signal with unknown location and scale perturbed by a gaussian random field on . while their motivation came from fmri , their results are also relevant to areas such as astronomy and genetics , or indeed to many applications in which one searches a large space of noisy data for a relatively small number of signals which manifest as `` bumps '' in the random noise .  in a second paper , together with shafie and sigal , they treated an extended version of the same basic problem , allowing the asymmetric bumps to also have a direction associated to them , in addition to scale changes that were direction dependent .this added an additional `` rotation parameter '' to the model and complicated it to the extent that they were unable to treat anything beyond the two - dimensional ( ) case .thus , for example , while their results could be used to examine cortical surface images , they could not be applied to three - dimensional fmri analysis .  for etymological reasons , the first of these caseswas called the _ scale space _ case and the second the _ rotation space _ case . in both cases , the mathematics of these papers concentrated on approximating the exceedence probabilities for large , a smooth , gaussian , real - valued , random field , and some parameter space . for reasons that will become clearer soon , in the -dimensional scale space case ,the effective dimension of was , while in the rotation space case it jumped to ( or 5 , for the only case they could handle , when ) .the aims of the current paper are twofold .first , we shall generalize the results of the rotation space to any dimension , but by a very different method .rather than taking a `` from first principles '' approach , as was the case in  , we shall compute everything from a basic result known as the _ gaussian kinematic formula _ [ gkf , cf . ( [ pmainequation ] ) below ] , using techniques of riemannian geometry .in fact , we shall derive the rotation space results from a more general setting , which is of interest in itself .this is quite different from the analysis in , in which calculations were so involved that they could only be handled by symbolic calculus using computers and , then , as already mentioned , only in two dimensions . according to the `` no free lunch '' principle , there is a price to pay here , in that the specific approximations of  have a slightly higher level of precision .second , we shall recover the results of the scale space case , again using the gkf .we shall do this in two ways .first , we shall obtain it as an immediate corollary to the `` more general setting '' mentioned above .this is quick , but once again does not give the best results .however , we shall also derive the `` best '' results from a more detailed calculation , something which seems too difficult to do for rotation space fields .  at first thismight seem as if we are attempting to kill the proverbial ant with a sledge hammer , but it turns out that there are at least three advantages to this approach .first of all , the ant in question is a rather large one , and any way of dealing with it requires considerable effort .the basic calculations in both and here take about 10 pages , and the essential difference is whether one prefers to work with riemannian topology or integral geometry , hard calculus and occasionally symbolic ( machine ) calculus. this may be more a question of mathematical taste than anything else .the second advantage of using the gkf as the basic starting point , which is significant and no longer merely an issue of taste , is that while the results of  and apply , as stated , only to gaussian fields , the gkf allows immediate extension to a much wider class of examples .section [ gkfsubsec ] explains this .finally , both of these cases are instructive in terms of how to use the gkf in more general settings , and our hope is that presenting them in detail will provide examples which will make the gkf easier to use in the future for other , perhaps quite different , examples .we shall say no more in the way of motivation for this paper .discussions and examples can be found in both  and as well as the papers cited there , and the many more recent papers citing  and  . in particular , it is explained there why ( [ exceedence1equn ] ) is an important quantity in statistical hypothesis testing . rather , we shall , in the following section , immediately turn to formal definitions of scale and rotation random fields , a description of the gkf and its relation to  and  , and a statement of the main results of the paper in section [ lkcrotatesubsec ] .fuller results for scale space fields , which go considerably beyond those of section [ lkcrotatesubsec ] , are given at the end of the paper , in section [ scalesec ] .understanding or using the results of the paper does not require a deeper knowledge of differential geometry beyond that which we describe along the way .this is not true of the proofs , and in section [ beforesection ] we point to what else is needed .the remainder of the paper is taken up with proofs .the common element behind all the random fields in this paper is gaussian white noise .the simplest way to define gaussian white noise on is as a random field indexed by the borel sets , finitely additive , taking independent values on disjoint sets , and such that , for all with finite lebesgue measure , .it is standard fare to define integrals , for _ deterministic _ , which are mean zero , gaussian random variable with covariances    the gaussian scale space random field is obtained by smoothing white noise with a spatial filter over a range of filter widths or scales .formally , for a `` filter '' it is defined as typically ranges over a nice subset of , and we take  ] , where is the volume of the unit ball in .the , known as the gaussian minkowski functionals of , are determined via the tube expansion where and dist is the usual euclidean distance . in the case it typically involves no more than multivariate calculus to compute the , and many examples are given in  and  .we shall give one example in a moment .the most important elements of ( [ pmainequation ] ) are the , known as lipschitz killing curvatures ( lkcs ) .the first of these is rather simple , as it is always the euler characteristic , that is , for all nice .thus , ( [ pmainequation ] ) , with , provides us with an expression for , which is what is needed to approximate exceedance probabilities , as explained in the following section .  for a simple but extremely important example , suppose that and is the identity .then it is easy to compute the to see that , with , and satisfying the conditions above , ( [ pmainequation ] ) becomes where is an excursion set , the , , are hermite polynomials , and ^{-1}\int_x^\infty\varphi(u ) \,du ] where the sum is taken over all distinct choices of subscripts .note that it will always be true that is the volume of , while is half the surface measure of .furthermore , .  inwhat follows , however , we shall need to work in a non - euclidean setting , as the lipschitz killing curvatures in the gkf , and thus in ( [ eecgaussianequn ] ) , are computed relative to a riemannian metric induced by a gaussian field .this metric on is defined by for any and for , the tangent space to at .this can also be written in terms of the covariance function as   both of the gaussian fields of interest to us the scale and rotation space fields which we defined on compact sets , can easily be extended , to and , respectively , and so ( [ metric ] ) defines metrics on these spaces . in all that follows , unless stated otherwise , these metrics , or their restrictions , are what we shall work with , and it will be in relation to them that hausdorff measures and lipschitz killing curvatures are defined . when we consider a submanifold of either of these spaces the notation , will be used .when the standard euclidean metric is used these will be replaced by and .a riemannian metric defines a volume form , a riemannian curvature and a second fundamental form on ( and , indeed , on each the ) .then is its riemannian volume and so is an integral of the volume form . is closely related to the ( riemannian ) area of , the ( )-dimensional boundary of .the remaining lipschitz killing curvatures involve integrals of both curvatures and second fundamental forms , and are more complicated .rather than give the general form ( for which you can see chapter 10 of ) , we shall give specific definitions for each of the cases we treat , in the following sections .now , however , we shall give ( relatively ) simple expressions for the cases of interest to us .these are the main results of this paper .we shall present the results of this section in stages . in the first we shall represent the lipschitz killing curvatures as an integral for a situation more general than that needed to handle rotation space fields .this is theorem [ decompositiontheorem ] , and it shows how to write these integrals as products of lebesgue measures of subsets of and integrals over submanifolds of . in theorem [ rotatetheorem ] we shall compute these integrals for certain situations , which will lead to corollaries [ rotatecorollary ] and [ corscale ] .these give relatively simple , quite explicit , expressions for the highest - order lipschitz killing curvatures for rotation and scale space fields , respectively .more detailed information for scale space fields is given in section [ scalesec ] .we say that a manifold is orthogonally invariant if for every , the set of orthogonal matrices . for such of dimension , we define where , is the haar probability measure on , and for a matrix , denotes the upper left submatrix of size .the measure is the haar measure induced by the restriction of the metric induced by the field to a fiber , which , as we shall see , is independent of the choice of . with these definitions, we can now state the following :  [ decompositiontheorem ] let be the restriction of the rotation space random field defined in section [ rotationspacesubsec ] , to the product of an -dimensional regular stratified manifold and a -dimensional regular stratified manifold . furthermore , suppose that each stratum , , is orthogonally invariant and that the regularity conditions of section [ regularitysubsec ] vis - a - vis all hold .then , ,\end{aligned}\ ] ] where is given by ( [ ecscaledequn ] ) or ( [ eqconstants ] ) .it is not hard to see that is a symmetric function of the eigenvalues of .furthermore , as we shall see , as each stratum of is orthogonally invariant , its hausdorff measure is in fact invariant under , for any orthogonal .hence , the are expressible as integrals over certain functions of the eigenvalues of .in particular , for rotation space , with search region , we arrive at the following explicit expressions .[ rotatetheorem ] with the notation defined above , under the conditions of theorem [ decompositiontheorem ] , and for a spherically symmetric kernel , \,d \lambda,\end{aligned}\ ] ] where and are given in ( [ eqdefck ] ) and ( [ eqdefdk ] ) .theorem [ decompositiontheorem ] implies the following corollary , with all s as above .[ rotatecorollary ] under the conditions of theorem [ rotatetheorem ] and the regularity conditions of section [ regularitysubsec ] , the top two lipschitz killing curvatures for the rotation random field on are   to obtain a parallel result for scale space fields , we use the fact that the scale space random field on  ] are and where is as in ( [ eqdefctildek ] ) .  in section [ scalesec ]we analyze scale space fields independently of the rotation space case for a manifold with a boundary and without assuming spherical symmetry .the formulae for all lipschitz killing curvatures are given via ( [ ecsumoflsequn ] ) , the terms of which are all computed in section [ scalelkcssubsubsec ] .   while the formulae of theorem [ rotatetheorem ] may look rather forbidding , one should note two facts . the first is that , for dimension , they are the first of their kind . the second is that although for dimension they are implicit in , one computation relied on computer algebra and gave a large , unstructured formula , while the second , based on tube formulae , does not precisely give the lkcs .( this is not unreasonable , since that computation was directed explicitly at approximating exceedance probabilities , and not computing lkcs . ) to recoup the results of  , we treat , as an example , the case ( so that ) . in that case , it is possible to carry out the integrations needed to compute and .a page or so of calculus shows that , for the gaussian kernel , ,\ ] ] where is the simple euclidean area of and .  the lkc is a little harder to calculate , since while the first term inside the brackets is again amenable to simple calculus , the second term leads us to special functions .however , a simple answer is attainable , and where is the length of , and , with the elliptic integral , we have .note also that , as stated , theorem [ rotatetheorem ] is weaker than the result for the scale case in section [ scalesec ] in that it gives only the highest two lkcs , rather than all of them .however , while more would be desirable , it is the higher ones that are most important for the euler characteristic heuristic discussed in the following section .we shall explain there why this is the case .as intimated in the , signal plus noise problems often boil down to being able to compute the exceedance probabilities ( [ exceedence1equn ] ) . for smooth gaussian processes, there is typically no way to directly compute these probabilities , and they must be approximated , using approximations that are accurate for the tail 5% or so of the distribution .there are a number of ways to carry out such approximations , and two of these are at the center of the papers  and  , discussed in the .one relies on _ tube formulae _ and moves the problem from one about gaussian processes on euclidean sets to one involving computation of volumes of tubes around embeddings of in high - dimensional spheres .the other is the euler characteristic heuristic which provides a generally easier route , along with the advantage that it also provides information on sample paths at all levels .the ech states that for smooth random fields  , on nice , locally convex , compact sets ( which will be the case in our scenarios if is locally convex , since and are convex ) and large ,   in the case of smooth , gaussian , , with zero mean and constant unit variance ( but with no stationarity assumptions ) the approximation in ( [ approxequn ] ) can be quantified ( cf . ) , and it is known that where we write for the difference of the two terms in ( [ approxequn ] ) , and is an -dependent constant , that is , in many cases , computable .the importance of ( [ validitymainequation ] ) is that it shows that the two terms in ( [ approxequn ] ) , both of which are known from the general theory of gaussian processes to be of order , differ by a term of exponentially smaller order , viz . of order fact is what justifies the use of the ech for gaussian fields ( in which case it is no longer heuristic ) and in large part motivates the computations of this paper , and also explains the claim of the previous section , that the leading lipschitz killing curvatures are the important ones in applying the ech .as we wrote earlier , it should have been possible to read the paper up to this point without a knowledge of differential geometry. this will not be true for the proofs .the derivations of the leading lipschitz killing curvatures for the rotation space case , which is what we have concentrated on so far , and all lkcs for the scale space case , which will be treated in section [ scalesec ] , begin in the same fashion .each starts by taking the parameter space , writing it as a stratified manifold , and carefully identifying the strata of ( [ ppartitionequation ] ) .the next step will be to take the generic formulae for lkcs , given in ( necessary , but sometimes painful ) detail in  , and rewrite them for the current setting , that is , for the specific stratification , and for the induced riemannian metric ( [ metric ] ) .we shall not explain in detail how this is done other than to say , at each stage , which generic formula we are using .then , the riemannian metric induced by the field is computed .  at this pointthe proofs diverge . for scale spacewe compute all lkcs , and so geometric objects such as curvature and second fundamental forms arise . for rotation space , with only the two leading lkcs , we can avoid them .the problem lies not in the the higher dimension of rotation space , but in its more complicated riemannian metric .nevertheless , in section [ secscale ] we shall see that once the geometry of rotation space is understood , the two leading lkcs of scale space are effortlessly computed .  throughout, we have tried to carry out computations as carefully as possible , so that not only can the interested reader follow the calculations , but also can see how they need to be adjusted for other cases .we shall break the proof of theorem [ decompositiontheorem ] into stages , the first of which relates to the crucial orthogonality properties mentioned in the .our aim in this subsection is to establish the orthogonality of the two subspaces under the inner product defined by the riemannian metric induced by the process .another factor that comes into play is a structured dependence of the restrictions of the riemannian metric on to fibers of the form .this will turn out to be the key that allows us to carry out further detailed computations .  to establish this independence , let be a point in and let be two derivations in , respectively . using the connection to the covariance functiongiven in ( [ eqriemannianmetric ] ) , and writing and to denote the elementwise derivatives of the identity functions on and , respectively , we have & = & 2{\vert}s{\vert}^{{1}/{4 } } \bigl(y { \vert}s{\vert}^{{1}/{4 } } \bigr ) \int\bigl((xt)'su\bigr ) \dot{k}\bigl ( u'su\bigr ) k\bigl(u'su\bigr ) \,du \\[-2pt ] & & { } + 2{\vert}s{\vert}^{{1}/{2 } } \int \bigl((xt)'su \bigr ) \bigl(u'(ys)u\bigr ) \bigl[\dot{k}\bigl ( u'su \bigr)\bigr]^2 \,du\\[-2pt ] & = & 0,\end{aligned}\ ] ] where the last equality follows since both integrands are odd functions .this shows that the subspaces and are orthogonal at each point .we now compute the restriction of the riemannian metric to a fiber , again based on ( [ eqriemannianmetric ] ) .writing , again , for the spatial derivative of with respect to the elements of , the gram matrix of the riemannian metric on , relative to the standard euclidean basis , is & & \qquad = { \vert}s{\vert}^{{1}/{2}}\int_{\mathbb{r}^{n } } \nabla_{t}k \bigl ( ( t - u ) 's ( t - u ) \bigr ) \bigl ( \nabla_{t}k \bigl ( ( t - u ) 's ( t - u ) \bigr ) \bigr)'\,du \\[-2pt ] & & \qquad = 4{\vert}s{\vert}^{{1}/{2}}\int_{\mathbb{r}^{n}}\dot { k } \bigl ( w'sw \bigr ) sw \bigl(\dot{k } \bigl ( w'sw \bigr ) sw \bigr)'\,dw \\[-2pt ] & & \qquad = 4{\vert}s{\vert}^{{1}/{2}}s \biggl[\int_{\mathbb { r}^{n } } \bigl(\dot{k } \bigl ( w'sw \bigr ) \bigr)^{2}ww'\,dw \biggr]s \\[-2pt ] & & \qquad = 4s^{{1}/{2 } } \biggl[\int _ { \mathbb{r}^{n } } \bigl(\dot { k } \bigl ( v'v \bigr ) \bigr)^{2}vv'\,dv \biggr]s^{{1}/{2 } } \\[-2pt ] & & \qquad = \gamma s.\end{aligned}\ ] ]   relying on ( [ eqriemannianmetric ] ) to express the riemannian metric and using a simple change of variables , it can easily be seen that the restriction of to is independent of .denote this metric on by . using the identity , valid on , we have & & \qquad = \tfrac{1}{4}{\vert}s{\vert}^{{1}/{4}}s^{-1}k \bigl ( ( t - u ) 's ( t - u ) \bigr ) \\[-2pt ] & & \qquad\quad { } + { \vert}s{\vert}^{{1}/{4}}\dot { k } \bigl ( ( t - u ) 's ( t - u ) \bigr ) ( t - u ) ( t - u ) ' .\end{aligned}\ ] ] therefore , for tangent vectors , & & \hspace*{15pt } { } \times\bigl\langle\nabla_s \bigl({\vert}s{\vert}^{{1}/{4}}k \bigl ( ( t - u ) 's ( t - u ) \bigr ) \bigr),b \bigr\rangle \,du \\[-2pt ] & = & { \vert}s{\vert}^{{1}/{2}}\int_{\mathbb{r}^{n } } \biggl [ \frac { 1}{4}k \bigl ( u'su \bigr ) \bigl\langle s^{-1},a \bigr\rangle+\dot { k } \bigl ( u'su \bigr ) \bigl \langle uu',a \bigr\rangle \biggr ] \\[-2pt ] & & \hspace*{43.4pt } { } \times\biggl[\frac{1}{4}k \bigl ( u'su \bigr ) \bigl\langle s^{-1},b \bigr\rangle+\dot{k } \bigl ( u'su \bigr ) \bigl \langle uu',b \bigr\rangle \biggr]\,du.\end{aligned}\ ] ] after some slightly tedious calculus , and based on the fact that [ see ( [ eqkappaidentity ] ) ] , all of the terms above can be expressed in terms of moments of the kernel and its derivative as follows :   when is the gaussian kernel , and therefore , simple calculations show that , leaving only the term above .let denote the riemannian measures on induced by and let denote the hausdorff measure , that is , riemannian measure under the standard euclidean metric on .a standard calculation gives where is defined to be the determinant of for some matrix , the rows of which form an orthonormal basis of ( under the euclidean inner product ) .the determinant , of course , is independent of the choice of basis .  working in a chart and relying on the orthogonality established above ,it is easy to see that & = & \int _ { \partial_{k}b_k}\int_{\partial_{l}a_l}d\mathcal { h}_{l}^{a_l \times\{s\ } } ( t ) \,d\mathcal{h}_{k}^{b_k } ( s ) \\[-2pt ] & = & \gamma^{l/2 } \int_{\partial_l a_l}\int _ { \partial_k b_k}{\vert}s{\vert}_{t_{t}\partial_la_l}^{{1}/{2}}\,d\mathcal { h}_{k}^{b_k } ( s ) \,d\mathcal{h}_{l}^{e } ( t ) . \nonumber\end{aligned}\ ] ] the equality above relies on the expression for the radon nikodym derivative above and a fubini theorem .we recall now our assumption that is orthogonally invariant , that is , for any .thus , the mapping is a diffeomorphism from to itself .moreover , for the covariance function of the rotation field , we have therefore , is also a riemannian isometry .thus , the measure , induced by the riemannian metric , is also invariant under .this implies that we can drop the dependence of the integral in ( [ eq41 ] ) on the tangent space .that is , we can replace it , for all , with a fixed -dimensional subspace of our choice or a random one .therefore , \\[-8pt ] & = & \gamma^{l/2 } { { \cal h}}_l(\partial_la_l ) f_{k , l}(\partial_kb_k ) .\nonumber \ ] ]  to complete the proof of the theorem , one must repeat essentially the same calculation for the codimension part of .in this section we perform the computations necessary to evaluate the expression in corollary [ corscale ] .these calculations are much simpler than those for rotation space .the set can be stratified as clearly , each stratum is orthogonally invariant .consequently , theorem [ decompositiontheorem ] implies we need only compute the following three terms , each of which is straightforward : where is the appropriate density with respect to the lebesgue measure .above , we expressed the necessary integrals in terms of the parametrization of by the curve .hence , therefore , start the proof of theorem [ rotatetheorem ] by describing a stratification of , which is one of the regularity conditions required for applying the gkf .in fact , we shall actually only use the strata of codimension 0 and 1 .our first step is to express the manifold as a whitney stratified manifold .note that a stratification for yields a stratification for as the product stratification , given that a whitney stratification of is assumed given .thus , our first goal is to stratify for arbitrary .semialgebraic orbits of the lie group action , , constitute a whitney stratification of ( cf . , page 21 ) .since are obtained by applying affine transformations to the strata of ( [ eqnonegdef - stratification ] ) , they form whitney stratifications of   the final step in stratifying is to verify that these stratifications intersect transversely . for this , it suffices to show that for any such that , a matrix and nonzero matrix , there exist smooth paths , with such that for any ,   based on the fact that the group action above preserves rank , the problem of finding the paths is equivalent to defining them by and finding appropriate invertible matrices .  even under the restriction a solution exists . substitution in ( [ eq3 ] ) gives which is equivalent to   for small , define to be the unique positive definite square root of . substituting in ( [ eq4 ] ) and ( [ eq5 ] ) defines , for small , the required paths , , and proves transversality .the transversality of the intersections , together with the stratifications in ( [ eq9 ] ) , yield the whitney stratification where .that is , we have found that the partition of according to the geometrical multiplicity of and is a whitney stratification .note for later use that the submanifold of of matrices of rank , and thus and , are of codimension .since , for with , and intersect tranversely , the codimension of is /2 ] , for some .we start by stratifying the parameter space into manifolds of common dimension . in particular , we write the -dimensional parameter space  ] ) , along with ( [ ecxinuzeroequn ] ) to the above , gives and we have the last of the three cases we were seeking .with the relevant connections determined , we can now turn to computing second fundamental forms and curvature matrices along the three stratifications of . , _ the interior _ : since the normal space in the interior is empty , there are no normals , no second fundamental form and no curvature matrix here .  \2 . : _ the side _ : a convenient choice for an orthonormal ( in the metric ) basis for the tangent space at any point on the side is given by=-1 =0 with outward unit normal vector .the scalar second fundamental forms of interest are therefore all of which can be computed directly using weingarten s equation and ( [ ecconnection1equn])([eqmixedconnection ] ) .we summarize them in a curvature matrix , writing where is the euclidean curvature matrix of in the basis .recall for later use that , from section 7.2 of  , in the current scenario , the trace in ( [ eqcurvaturemeasuresgaussiangeneral ] ) can be replaced by where is our usual sum of determinants of principal minors . , _ the bottom and the top _ : starting with the bottom , the outward unit normal vector is , and a convenient orthonormal basis is given by .the curvature matrix is therefore with entries by weingarten s equation and ( [ ecconnection0equn ] ) .thus , for the bottom , the outward curvature matrix is , while along the top the same arguments give it as . : _ the edges and _ : for the edges , we need to consider the scalar second fundamental form itself and not just the curvature matrix . asabove , an orthonormal basis for the tangent space is , but now an orthonormal basis for the normal space is .  applying the weingarten equation , we need to compute for arbitrary . applying ( [ ecconnection1equn ] )gives that this is , where , with a minor abuse of notation , we now use to denote the euclidean curvature matrix of at .we now have all the pieces we need to compute the lipschitz killing curvatures , and could actually proceed to the final computation . however , in justifying the formula ( [ eqcurvaturemeasuresgaussiangeneral ] ) , we used the fact that has constant negative curvature .now we shall take a moment to prove this .let } k=0 k=2n+1 k=2n$ is even.}\nonumber\end{aligned}\ ] ]  \1 . , _ the interior _ : since the second fundamental form is zero in , the only nonzero lipschitz killing curvature occurs when in ( [ eqcurvaturemeasuresgaussiangeneral ] ) . in this case & = & \kappa^{1/2 } \gamma^{n/2 } \int_{s_1}^{s_2 } \int_{{t}^{\circ } } e^{ns } \,dt \,ds \\[-2pt ] & = & \kappa^{1/2 } \gamma^{n/2 } \frac{e^{ns_2 } - e^{ns_1}}{n } { { \cal l}}_n^e\bigl(t ; { t}^{\circ}\bigr ) , \nonumber\end{aligned}\ ] ] where is computed in the standard euclidean sense . : _ the side _ : for this case , all the lipschitz killing curvatures need to be computed , and we replace the trace in ( [ eqcurvaturemeasuresgaussiangeneral ] ) by the determinants of the curvature matrix as in ( [ ectrace - detequn ] ). then & & \qquad = ( 2\pi)^{-(n - j)/2 } \ee \bigl\{z_2^{n - j } \indic_{\{z_2 > 0\ } } \bigr\ } \nonumber\\ & & \qquad\quad { } \times\int_{(s_1,s_2 ) \times\partial t } { \operatorname{detr}}_{n - j } \pmatrix { e^{-s } c_t & 0 \cr 0 & 0 } \,d{{\cal h}}_n(s , t ) \nonumber\\[-9pt]\\[-9pt ] & & \qquad = ( 2\pi)^{-(n - j)/2 } \kappa^{1/2}\ee \bigl \{z_2^{n - j}\indic_{\{z_2 > 0\ } } \bigr\ } \nonumber\\[-2pt ] & & \qquad\quad { } \times\int_{s_1}^{s_2 } \int _ { \partial t } \bigl(\gamma^{-1/2}e^{-s } \bigr)^{n - j } \bigl(\gamma^{1/2}e^s \bigr)^{n-1 } { \operatorname{detr}}_{n - j}(c_t ) \,dt\,ds \nonumber\\[-2pt ] & & \qquad = \kappa^{1/2}\gamma^{(j-1)/2 } \frac { e^{(j-1)s_2}-e^{(j-1)s_1}}{j-1 } { { \cal l}}^e_{j-1}(t ; \partial t)\nonumber\end{aligned}\ ] ] in a parallel notation to ( [ eclkckappa1equn ] ) . , _ the bottom and the top _ : beginning with the bottom , , for we have & & \qquad = ( 2\pi)^{-(n - j)/2 } \ee \bigl\{z_1^{n - j } \indic_{\{z_1 > 0\ } } \bigr\ } \nonumber\\[-9pt]\\[-9pt ] & & \qquad\quad { } \times\int_{\{s_1\ } \times { t}^{\circ } } { \operatorname{detr}}_{n - j } \bigl ( \kappa^{1/2}i_{n \times n } \bigr ) \,d{{\cal h}}_{n}(t ) \nonumber \\[-2pt ] & & \qquad = \bigl(2\pi\kappa^{-1}\bigr)^{-(n - j)/2 } \mu_{n - j } \pmatrix{n \cr j } \gamma^{n/2 } e^{ns_1 } { { \cal l}}_n^e\bigl(t ; { t}^{\circ}\bigr).\nonumber\end{aligned}\ ] ] a similar result holds for the top , , namely , \\[-9pt ] & & \qquad = ( -1)^{n - j}\bigl(2\pi\kappa^{-1 } \bigr)^{-(n - j)/2 } \mu_{n - j } \pmatrix{n \cr j } \gamma^{n/2 } e^{ns_2}{{\cal l}}_n^e\bigl(t ; { t}^{\circ } \bigr).\nonumber\end{aligned}\ ] ]  \4 . : _ the edges and _ : we start with the top edge , , & & \qquad = ( 2\pi)^{-(n-1-j)/2 } \\[-2pt ] & & \qquad\quad{}\times\int_{\{s_2\ } \times\partial t } \ee \bigl\ { { \operatorname{detr}}_{n-1-j}\bigl ( z_1 \kappa^{-1/2 } i - z_2 \gamma^{-1/2}e^{-s } c_t \bigr ) \\[-2pt ] & & \hspace*{161pt}\qquad\quad { } \times\indic_{\{z_1 > 0\ } } \indic_{\{z_2 > 0\ } } \bigr\ } \,d { { \cal h}}_{n-1}(s , t).\end{aligned}\ ] ] use now the easily checked expansion that , for , to expand the term in the expectation above and see that & & \qquad = ( 2\pi)^{-(n-1-j)/2 } \nonumber\\[-2pt ] & & \qquad\quad{}\times\sum_{m=0}^{n-1-j } \kappa^{-(n-1-j - m)/2 } \pmatrix{n-1-m \cr j } \nonumber\\[-2pt ] & & \hspace*{40pt}\qquad\quad { } \times\ee \bigl\{z_1^{n-1-j - m } \indic_{\{z_1 > 0\ } } \bigr\ } \bigl(\gamma^{1/2 } e^{s_2 } \bigr)^{n-1 } \ee \bigl\{z_2^{m } \indic_{\{z_2 > 0 \ } } \bigr\ } \nonumber\\[-2pt ] & & \hspace*{40pt}\qquad\quad { } \times\int_{\partial t } { \operatorname{detr}}_{m}\bigl ( \gamma^{-1/2}e^{-s_2}c_t\bigr ) \,d { { \cal h}}_{n-1}(t ) \nonumber\\[-9pt]\\[-9pt ] & & \qquad = \bigl(2\pi\kappa^{-1}\bigr)^{-(n-1-j)/2}\nonumber\\[-2pt ] & & \qquad\quad{}\times \sum _ { m=0}^{n-1-j } \bigl(\gamma^{1/2}e^{s_2 } \bigr)^{n-1-m}\pmatrix{n-1-m \cr j } \kappa^{m/2 } \nonumber\\[-2pt ] & & \hspace*{40pt}\qquad\quad { } \times\mu_{n-1-j - m } \ee\bigl\{z_2^{m }\indic_{\ { z_2 > 0 \}}\bigr\ } \int_{\partial t } { \operatorname{detr}}_{m}(c_t ) \,d{{\cal h}}_{n-1}(t ) \nonumber\\[-2pt ] & & \qquad = \kappa^{(n-1-j)/2 } \bigl(\gamma^{1/2}e^{s_2 } \bigr)^{n-1 } \nonumber\sum_{m=0}^{n-1-j } \pmatrix{n-1-m \cr j } \bigl(\gamma^{-1/2}e^{-s_2 } \kappa^{1/2}\bigr)^m\\[-2pt ] & & \hspace*{139pt}\qquad\quad { } \times \mu_{n-1-j - m } { { \cal l}}^e_{n-1-m}(t ; \partial t).\nonumber\end{aligned}\ ] ]  a similar argument also works for the bottom edge , the only change being that becomes , giving the final form & & \qquad = \kappa^{(n-1-j)/2 } \bigl(\gamma^{1/2}e^{s_1 } \bigr)^{n-1 } \nonumber\\[-9pt]\\[-9pt ] & & \qquad\quad{}\times\sum_{m=0}^{n-1-j } \pmatrix{n-1-m \cr j } \bigl(\gamma^{-1/2}e^{-s_1 } \kappa^{1/2}\bigr)^m \nonumber\\[-2pt ] & & \hspace*{41.5pt}\qquad\quad { } \times(-1)^{n-1-j - m } \mu_{n-1-j - m } { { \cal l}}^e_{n-1-m}(t ; \partial t ) .\nonumber\end{aligned}\ ] ]

Abstract: we provide a new approach , along with extensions , to results in two important papers of worsley , siegmund and coworkers closely tied to the statistical analysis of fmri ( functional magnetic resonance imaging ) brain data . these papers studied approximations for the exceedence probabilities of scale and rotation space random fields , the latter playing an important role in the statistical analysis of fmri data . the techniques used there came either from the euler characteristic heuristic or via tube formulae , and to a large extent were carefully attuned to the specific examples of the paper .   this paper treats the same problem , but via calculations based on the so - called gaussian kinematic formula . this allows for extensions of the worsley siegmund results to a wide class of non - gaussian cases . in addition , it allows one to obtain results for rotation space random fields in any dimension via reasonably straightforward riemannian geometric calculations . previously only the two - dimensional case could be covered , and then only via computer algebra .  by adopting this more structured approach to this particular problem , a solution path for other , related problems becomes clearer .   ,   +