while gravity is the dominant force in star formation , radiative processes are crucial as well .most basically , radiation removes energy , allowing a collapsing cloud to maintain a nearly constant , low temperature as its density and gravitational binding energy rise by many orders of magnitude .radiative cooling is what makes star formation a dynamic rather than a quasi - static process , a point understood quite early ( see the review by * ? ? ?at densities below h atoms , cooling occurs primarily via collisionally - excited atomic or molecular line emission , while at higher densities dust and gas become thermally well - coupled via collisions , and thermal radiation by dust grains is the dominant cooling process ( e.g. * ? ? ?* ) . at yet higher densitiesthe coupled dust - gas fluid becomes optically thick to its own cooling radiation , causing the temperature to rise in a quasi - adiabatic fashion .  while one - dimensional simulations of this collapse can include quite sophisticated treatments of radiative transfer between gas parcels ( e.g. * ? ? ?* ) , the computational difficulty of handling rhd in three dimensions led to a long period in which most 3d simulations of star formation simply dispensed with radiation entirely , choosing instead to represent the cooling processes either by a cooling function that removes energy at a rate dependent only on local gas properties , or by an even simpler equation of state that prescribes the gas temperature as a function of density .however , this approach has a major flaw : it does not allow energy exchange between parcels of gas , or between stars and diffuse gas . the last few years have seen a number of studies pointing out that this energy exchange is crucial to the dynamics of star formation , and that it can be dealt with only by rhd simulations .the most important radiative transfer processes can be broken into three rough categories : thermal feedback , in which collapsing gas and stars heat the gas and thereby change its pressure ; force feedback , in which radiation exerts forces on the gas that alter its motion ; and chemical feedback , in which radiation changes the chemical state of the gas ( e.g. by ionizing it ) , and this chemical change somehow affects the dynamics .each of these processes has been the object of intense study in the last few years .  with regard to thermal feedback , pointed out that how gas fragments , and thus the stellar mass distribution that results from fragmentation , is extremely sensitive to how the temperature varies with density . and showed that one consequence of this sensitivity is that radiation produced by the first few stars to form in a given region can heat the gas around them , reducing the ability of that gas to fragment , favoring monolithic collapse to massive stars and suppressing formation of low mass objects .subsequent numerical simulations using a variety of methods have confirmed this effect .force feedback becomes important in the context of massive stars , which produce such high luminosities that the radiative force they exert on dusty interstellar gas can exceed their gravitational force .a number of authors have argued based on one - dimensional models that this process sets an upper limit on stellar masses .more recent work in two ( ; also see the contribution by kuiper et al . in these proceedings ) and three dimensions instead suggests that force feedback does not limit stellar masses .deciding the question requires radiation - hydrodynamic simulations .the most important type of chemical feedback is photoionization , which raises the gas from k to k. this causes a number of important dynamic effects , including ejecting mass from star - forming clouds , altering the magnetic field structure , and driving turbulent motions .radiation also drives other chemical processes , but these are usually not important for dynamics , and thus may be handled by post - processing simulations . in this reviewi limit myself to radiative effects that are dynamically important and must therefore be simulated in tandem with the hydrodynamic evolution of the system .the equations of radiation hydrodynamics ( rhd ) in conservation form read  where , , , and are the gas density , velocity , specific energy , and pressure .the source term on the right hand side represents the rate at which radiation transfers momentum and energy to the gas . for simplicityi have omitted terms describing gravity and magnetic fields , which would appear as additional sources on the right hand side .the rate at which radiation transfers energy and momentum to the gas is described by the radiation four - force vector , \left ( \begin{array}{c } 1 \\ { \mathbf{n}}\end{array } \right)\ ] ] where is a unit vector , is the radiation intensity at frequency in direction , and and are the extinction coefficient and emissivity of the gas as a function of frequency and direction .finally , the intensity is governed by the transfer equation , for simplicity i have omitted terms describing scattering , which is generally not important for the radiation - hydrodynamics of star formation .the equations of rhd are , like the ordinary equations of fluid dynamics , characterized by dimensionless numbers .the two most important ones for radiation - hydrodynmics are , where is the characteristic value of , and , where is the size - scale of the system and is the photon mean free path .the first ratio characterizes how relativistic the system is . in writing equation ( [ eq : conslaws ] ) in non - relativistic form, we have already assumed that .the second , , characterizes how optically thick it is .systems with are described as being in the streaming limit , and are characterized by weak matter - radiation interaction .those with are in the diffusion limit and have strong - matter radiation coupling .both limits appears in star formation problems .the full system of equations ( [ eq : conslaws ] ) and ( [ eq : transfer ] ) is seven - dimensional , with quantities varying in time , space ( 3 dimensions ) , direction ( 2 dimensions ) , and frequency .unfortunately , this makes full numerical solution prohibitively expensive even on modern supercomputers .we are therefore reduced to approximations .there are two broad classes of approximation in common use in star formation simulations : moment methods and characteristic methods .a third category , monte carlo methods , has been used extensively in post - processing radiative transfer calculations , but has not been used extensively in radiation - hydrodynamic simulations .i will not discuss it in detail .   in the discussion that followsi introduce the most common methods used in amr rhd methods , and in table [ tab : codes ] i summarize which method(s ) are used in each code .the codes i include in the table are as follows : * orion * is the oldest and probably best - tested amr rhd code , but is not publicly available , and currently lacks a magnetohydrodynamic ( mhd ) capability .* ramses * is an amr mhd code that was recently upgraded with a radiation solver , although the latter is not ( as of this writing ) publicly available . *flash * is an open source amr mhd code with extensive chemistry capabilities and several add - on modules that handle radiation in different ways .* enzo * is an open source amr hd code ( an mhd version exists but is not public ) with cosmology capabilities and two different radiation methods . *pluto * is an open source amr mhd code with a newly - developed radiation solver .there are also a number of fixed , nested grid codes in use , but i will not mention these by name .lccclcc + & & + & * resolution * & * frame * & & * code * & * resolution * & * ray scheme * + orion & mg , 2 t & mixed & & orion & 1f & healpix + ramses & 2 t & comoving & & flash & mg & hybrid characteristics + enzo & 2 t & comoving & & enzo & 1f & healpix + pluto & 1 t & comoving & & pluto & mg & sphere +    the basic idea of a moment method is to take moments of the transfer equation , in exact analogy to the chapman - enskog procedure used to derive the equations of fluid dynamics from the kinetic theory of gases . to take the zeroth momentwe integrate the transfer equation over all directions ; to obtain the first moment we multiply both sides by the unit vector and then integrate ; for the second moment we multiply by the rank two tensor and integrate , and so forth . as in the analogous fluid case, the procedure yields an exact solution if carried out to infinitely many orders , but one instead makes an approximation by truncating the procedure after finitely many orders . usually finitely many " here means one or two , and the first two moments of equation ( [ eq : transfer ] ) are where are the first three moments of the radiation intensity : the radiation energy density , radiation flux , and radiation pressure tensor . in these equations the direction - dependence in the transfer equation is removed and the dimensionality of the problem is lowered by two , but at the price of introducing the radiation pressure tensor , for which we do not have an equation ( since we have not expanded the transfer equation to the next order ) and must instead make an approximation .  because moment methods necessarily involve smoothing the angular dependence of the radiation field , they are best suited to representing diffuse , smooth radiation fields .this makes them ideal for handling thermal and force feedback , which tend to be dominated by diffuse infrared light re - radiated by dust grains .they are less suited for chemical feedback , which is usually dominated by direct , highly beamed radiation from a handful of stellar sources .a further advantage of moment methods is that their computational cost is independent of the number of sources , and usually scales only as or , where is the number of cells .they also parallelize fairly easily .moment methods can be classified based on several design choices that are made in their construction , all of which involve some tradeoff between computational expense and accuracy .see table [ tab : codes ] for a list of the design choices made in each of the popular codes ._ closure order ._ the first choice is whether to retain both of the first two moment equations , or to retain only the zeroth moment and adopt a closure approximation for the radiation flux ; all the moment methods currently used in practical star formation simulations make the latter choice , using a closure known as flux - limited diffusion ( fld ) . in a medium where the photon mean free path is small compared to the size of the system , eddington showed that the radiation flux in the fluid rest frame approaches where is the rosseland mean opacity , is the planck function .this is known as the diffusion approximation .while it is quite good in optically thick media , it breaks down in optically thin regions because as the signal speed approaches infinity rather than properly limiting to .the fld approximation is an attempt to fix this problem by instead setting the radiation flux to where is a function that has the property that in optically thick regions and in optically thin regions , so that , its correct limiting value .many choices for the function are possible ; the most common one in astrophysical applications is the limiter , , where .however , all of these limiters are of unknown accuracy in the intermediate optical depth regime , and all fld methods suffer from the problem that they discard information about the directionality and momentum content of the radiation field .higher order moment methods exist and can solve some of these problems ( e.g. variable tensor eddington factor methods , , and the m1 closure method , ) , but none have yet proven cheap and robust enough for use in practical amr star formation simulations ._ frequency resolution . _the second choice in setting up an rhd moment method is the level of resolution in frequency .most accurate is the multigroup ( mg ) method , in which one discretizes equation ( [ eq : momenteqn ] ) in frequency , solving one version of the equation for each frequency bin . all the bins are coupled via the matter temperature , which affects the extinction and emissivity and thus the radiation four - force vector ( equation [ eq : forcevec ] ) .next most accurate is the 2 t method , in which one takes the radiation spectrum to be a planck function characterized by a single temperature ( thus removing a dimension from the problem ) , but allows to be different than the gas temperature .simplest of all is the 1 t method , in which one assumes , allowing a further simplification of the equations .as one might expect , this choice involves a tradeoff between accuracy and cost ; the 1 t method is cheapest but badly misestimates both the radiation spectrum and in the streaming regime . in comparison 2 t methods still produce incorrect radiation spectra in the streaming regime , but make much smaller errors in the matter temperature .they are intermediate in cost .mg provides a good representation of both the spectrum and the matter temperature but is most expensive ._ choice of frame ._ the third design choice is whether to formulate the equations in the comoving frame or using mixed frames .extinction and emissivity are simple , isotropic functions in the frame comoving with the fluid , but in other frames they contain complex directional dependence as a result of relativistic boosting .one might think that this effect is small in non - relativistic flows , but ignoring it is equivalent to neglecting the work done by radiation on the gas , which is obviously unacceptable if radiation forces are non - negligible .to avoid complex velocity dependence in and , it is highly desirable to write them in the comoving frame .the simplest choice after that is to write the radiation quantities ( , ) in the comoving frame as well ( e.g. see ) , but this carries a significant cost .since the comoving frame is non - inertial in any system with non - constant fluid velocity , the comoving radiation energy is not a conserved quantity , and thus comoving - frame codes can not be exactly conservative . moreover , every time the resolution changes and the velocity field is refined , the reference frame change as well , introducing errors in conservation that are likely to accumulate with increasing refinement .the alternative is a mixed - frame formulation in which one writes the radiation quantities in the inertial lab frame .deriving this formulation is tricky , since one must carefully account for the lorentz transformations between frames , but the resulting equations are explicitly conservative , and can be discretized in a manner that conserves energy to machine precision .   in characteristic methodsone solves the transfer equation ( [ eq : transfer ] ) directly , but only along selected rays .given this solution , one can compute the radiation four - force vector ( equation [ eq : forcevec ] ) directly .this provides much greater accuracy in computing the radiation intensity along those rays , but at the cost of neglecting all other rays .for this reason it is best used for chemical feedback , which tends to be dominated by the rays coming from a small number of sources .it is not well - suited to handling diffuse radiation fields , and simulations based on this technique are generally no better than non - radiative simulations when it comes to handling effects like fragmentation being altered by a diffuse ir radiation field .the cost of these methods scales as the number of cells times the number of sources , but with a coefficient that is generally smaller than for moment methods .thus these methods are cheaper than moment methods when the number of sources and rays is small , but become impractically expensive if the number of sources is even a small fraction of the number of cells . as with moment methods , there are several design choices to be made in setting up a characteristic method ._ frequency resolution ._ as in the moment method case , one can choose either to retain the frequency - dependence in the transfer equation by using multiple frequency groups ( the mg method ) , or one can integrate over frequency . in this caseone does not generally assign a temperature to the radiation ; instead , since for chemical feedback one cares about photons only above a certain energy , the usual approximation is to adopt an average photon energy .this is the single - frequency ( 1f ) approximation ._ ray - drawing scheme ._ there are also a number of schemes for drawing rays from the point source(s ) .the simplest is the spherical ray technique : one uses spherical coordinates , requires that the source be at the origin , and simply draws rays aligned with the computational grid .this parallelizes almost perfectly and is very simple to code , but is obviously unsuited to any problem with multiple stellar sources or where the locations at which stars form is not known _ a priori_. the other two schemes in wide use are healpix - based adaptive trees and the hybrid characteristics , both of which operate on top of an underlying cartesian grid .the healpix scheme is based on the hierarchical equal area isolatitude pixelization of the sphere , which divides the sphere into a equal area pixels that can be subdivided indefinitely , allowing the angular resolution to adapt to match that of the underlying grid .the hybrid characteristics scheme works by combining a method of short characteristics within individual amr grids with a method of long characteristics between grids .neither suffer from the limitations of the spherical ray method .hybrid methods attempt to combine the best features of characteristic and moment methods .characteristics work well for the highly beamed radiation coming directly from a star or some other point source .however , once this radiation is absorbed , following its re - emission with characteristics becomes unreasonably expensive , and a moment method is a far better choice .the underlying idea of a hybrid method , first used in multidimensional simulations by and recently implemented in the pluto code , is to use characteristics for the first absorption " , then switch to a moment method .one does this by performing a characteristic trace from the star or stars to determine the rate of energy and momentum input into each cell by direct radiation .one adds this as a source term on the right hand side of the moment equation ( [ eq : momenteqn ] ) , then solves as one would in a pure moment method .the computationally - cheap characteristic step can use mg , while the more expensive moment solve uses the 1 t or 2 t approximation . since the first absorption is generally the part of the problem where the frequency dependence is most important , this method achieves some of the accuracy of a fully frequency - dependent calculation at significantly lower cost .i close this review with a brief discussion of possible future directions for amr rhd in the star formation context .one such likely direction comes from combining realistic treatments of thermal , force , and chemical feedback into a single simulation , rather than treating only one or two of them at a time as in current simulations .since the first two effects are most easily handled by moment methods and the third by characteristic methods , hybrid techniques are the natural solution . to be competitive in handling thermal and force feedback , however, these methods will have to be linked to more advanced moment methods than has been attempted before .nonetheless , a hybrid method combining frequency - dependent characteristic tracing with a 2 t , mixed frame fld method is a straightforward extension of current techniques , and seems a logical place to start .further in the future , second order moment methods are likely to become important .developing them to the point where they can run reliably in a parallel environment on adaptive grids will be a major algorithmic undertaking , one that is likely to require the assistance of computer scientists and applied mathematicians in addition to astronomers .if successful , these methods will be able to handle both beamed and diffuse radiation fields within a single framework , and will remove many of the limitations of the fld approximation .monte carlo methods are another candidate to replace the current dominance of fld and characteristic methods .they are very computationally costly , since many photons are required to suppress poisson noise , but they have the advantage of near perfect parallelization . since the future of supercomputing seems to be heading toward the development of ever - larger numbers of processors , each of which is no faster than the processors of the previous generation , this may prove to be a decisive advantage .  finally , a caveat : for all the limitations of our current rhd methods , it is not entirely obvious that our numerical techniques are the limiting factor in the accuracy of our simulations .the main agent for coupling radiation and gas at the high densities where stars form is dust .our knowledge of dust grain properties , such as sublimation temperatures and grain size distributions in dense environments , is quite limited , and appears likely to advance less quickly than either computer power or algorithmic technique. it may be the case in the future that our knowledge of dust becomes the limiting factor in the accuracy of rhd simulations ., m. l. , bryan , g. l. , harkness , r. , bordner , j. , reynolds , d. , oshea , b. , & wagner , r. 2008 , in petascale computing : algorithms and applications , ed .d. a. bader , computational science series ( chapman & hall / crc ) , 83101 , arxiv:0705.1556           i thank richard klein , chris mckee , and loius howell for helpful discussions .i acknowledge support from : an alfred p. sloan fellowship ; nasa through atfp grant nnx09ak31 g and through the spitzer theoretical research program , through a contract issued by the jpl ; and the nsf through grant ast-0807739 .

Abstract: i provide a pedagogic review of adaptive mesh refinement ( amr ) radiation hydrodynamics ( rhd ) methods and codes used in simulations of star formation , at a level suitable for researchers who are not computational experts . i begin with a brief overview of the types of rhd processes that are most important to star formation , and then i formally introduce the equations of rhd and the approximations one uses to render them computationally tractable . i discuss strategies for solving these approximate equations on adaptive grids , with particular emphasis on identifying the main advantages and disadvantages of various approximations and numerical approaches . finally , i conclude by discussing areas ripe for improvement .


the community of program optimisation and analysis , code performance evaluation , parallelisation and optimising compilation has published since many decades hundreds of research and engineering articles in major conferences and journals .these articles study efficient algorithms , strategies and techniques to accelerate programs execution times , or optimise other performance metrics ( mips , code size , energy / power , mflops , etc . ) .the efficiency of a code optimisation technique is generally published according to two principles , non necessarily disjoint .the first principle is to provide a mathematical proof given a theoretical model that the published research result is correct or / and efficient : this is the hard part of research in computer science , since if the model is too simple , it would not represent real world , and if the model is too close to real world , mathematics become too complex to digest .a second principle is to propose and implement a code optimisation technique and to practice it on a set of chosen benchmarks in order to evaluate its efficiency .this article concerns this last point : how can we convince the community by rigorous statistics that the experimental study publishes correct and fair results ?hard natural sciences such as physics , chemistry and biology impose strict experimental methodologies and rigorous statistical measures in order to guarantee the reproducibility of the results .the reproducibility of the experimental results in our community is , namely , our dark point .given a research article , it is in practice impossible or too difficult to reproduce the published performance .if our results are not reproducible , we can not say that we are doing science !some aspects make a research article non - reproducible :  * non using precise scientific languages such as mathematics . ideally , mathematics must always be preferred to describe ideas , if possible , with an accessible difficulty . *non available software , non released software , non communicated precise data . *not providing formal algorithms or protocols make impossible to reproduce exactly the ideas . for instance, the authors in spent large efforts to re - implement some branch predictor algorithms based on the published research articles , but they fail to reproduce the initial results of the authors . simply because the initial articles describing the branch predictors are not formal , so they can be interpreted differently .* hide many experimental details . as demonstrated by ,bringing small modification on the execution environment brings contradictory experimental results .for instance , just changing the size of the linux shell variables or the order of linking an application alter the conclusions .as pointed by the authors in , a lot of published articles in major conferences hide these details , meaning that their experimental results are meaningless .* usage of deprecated machines , deprecated os , exotic environment , etc .if we take a research article published five years after the experiences for instance , there is a high chance that the workstations that served the experiences have already died or already changed their behaviour ( usury of hardware , software patches , etc . ) .  with the huge amount of published articles in the code optimisation community , with the impressive published speedups ,an external reviewer of our community has the right to ask the following naive question : _ if we combine all the published speedups ( accelerations ) on the well known public benchmarks since four decades , why do nt we observe execution times approaching to zero ? _ this question is justified , and brings a reforming malaise to us .now , we are asked to be clear about our statistics , some initiatives start to collect published performance data in order to compare them .the malaise raised by the above question is not a suspicion of a general _ cheating _ in research .we believe that our community is honest in publishing data , but the published observed speedups are sometimes _ rare _ events far from what we could observe if we redo the experiences multiple times .even if we take an ideal situation where we use exactly the original experimental machines and software , it is too difficult to reproduce exactly the same performance numbers again and again , experience after experience .usually , published speedups are computed with bias describing pretty _ rare _ events . frankly ,if a computer scientist succeeds in reproducing the performance numbers of his colleagues ( with a reasonable error ratio ) , it would be equivalent to what rigorous probabilists and statisticians call a _surprise_.   what makes a binary program execution time to vary , even if we use the same data input , the same binary , the same execution environment ?  * background tasks , concurrent jobs , os process scheduling ; * interrupts ; * input / output ; * starting loader address ; * branch predictor initial state ; * cache effects ; * non deterministic dynamic instruction scheduler ; * temperature of the room ( dynamic voltage / frequency scaling service )  one of the reasons of the non - reproducibility of the results is the variation of execution times of the same program given the same input and the same experimental environment . with the massive introduction of multicore architectures, we believe that the variations of executions times will become exacerbated because of the complex dynamic features influencing the execution : threads scheduling policy , synchronisation barriers , resource sharing between threads , hardware mechanisms for speculative execution , etc . consequently, if you execute a program ( with a fixed input and environment ) times , it is possible to obtain distinct execution times .the mistake here is to assume that these variations are minor , and are stable in general .the variation of execution times is something that we observe everyday , we can not neglect it .an usual error in the community is to replace all the execution times by one value , such that the minimum , the mean or the maximum .doing that would produce _ sexier _ speedups to publish , but does not reflect the reality with fair numbers .considering the minimum value of the observed execution times is unfair because :  * nothing guarantees that this minimum execution time is an ideal execution of the program . *nothing guarantees that this minimum execution time is a consequence of the optimisation technique under study .maybe this minimum execution time is an accident , or a consequence of dynamic voltage scaling , or anything else . *if this minimal execution time is a rare event , all your statistics describe rare speedups .so , they become non - reproducible easily.   we base our reasoning here on common well known results in statistics , especially on some results explained in the book of raj jain .we propose a first step towards a rigorous statistical methodology to evaluate program optimisation techniques .this article recalls some common mistakes in performance evaluation , explains which statistics should be used in a particular situation , and provide practical examples .furthermore , we show how to use the free software called r to compute these statistics .our article is organised to help computer scientists ( and of course phd students ) willing to make correct and rigorous statistical study of their code optimisation method .the question is how to convince real experts by statistics , provided a confidence level 0\% , 100\% [ ] be the confidence interval computed by the student test .if then we can say that would be faster in average than in of the future executions ( considering the same data input and experimental environment ) .an example is illustrated later .the speedup factor for the program can be defined as the fraction between the sample mean times , as follows : . here, is the usual notation of the sample arithmetic mean : , .the problem with this definition of speedup is that it is sensitive to _ outliers_. indeed , the distributions of the values of and may be biased .  for the above reasons , we prefer using the median as suggested by instead of the sample mean of the execution times .consequently , the speedup becomes    remember that this speedup has no sense if the student test fails to determine if 0 is outside the confidence interval . for the remaining part of the article , we note by and the observed median of the execution times of the program and resp .let be a initial program with its `` representative '' data input .we are willing to statistically demonstrate with a confidence level that an optimisation technique transforms it into and produces benefit in terms of execution speed . for doing this ,i should execute and at least 30 times . for the sake of the example ,i consider here only 5 executions for and .using the software r , i introduce the values of execution times ( in seconds ) of and as two vectors and resp .  ....> library(stats ) > t1<- c(2.799 , 2.046 , 1.259 , 1.877 , 2.244 ) > t2 < - c(1.046 , 0.259 , 0.877 , 1.244 , 1.799 ) ....  we must not hurry to conclude and to publish the following result : the resulted speedup for this program is equal to .publishing such performance gain ( acceleration of factor equal to 4.86 ) is a statistical mistake .since we have only 5 observations instead of 30 , we should check the normality of the values of and resp . using the test of shapiro - wilk .  ....shapiro.test(t1 )   shapiro - wilk normality test data :  t1  w = 0.9862 , p - value = 0.9647 ....  the test of shapiro - wilk on the data computes here a value . in order to say that the test succeeds with confidence level , the value must be greater or equal to the value of the shapiro - wilk table ( this table can be found on internet for instance ) .i use here a confidence level .the shapiro - wilk table for ( number of values ) and indicates the value of 0.986 .consequently , the normality test succeeds for .idem for .> shapiro.test(t2 )   shapiro - wilk normality test data :  t2  w = 0.9862 , p - value = 0.9647 ....  since , the values of follows a normal distribution with a confidence level of .it is important to notice here that if the normality test fails for a program ( or ) , we must run it at least 30 times .i can now continue with the student test to check if is faster than with a very high confidence level .  ....> t.test(t1,t2 , alternative="greater " , conf.level=0.99 ) welch two sample t - test ... 99 percent confidence interval : -0.02574667     inf  ... ....  the obtained confidence interval for the difference between the mean execution times is  ] , it does not include 0 .consequently , we can assert with 95% confidence level that we obtained a speedup .in other words , the risk ( of error ) of not obtaining an acceleration for the future executions is equal to 5% .the obtained speedup is .if the confidence level used for the student test is too low , it is not impossible that we reach a situation where the student test detects a speedup while the computed speedup is .the following example shows that low confidence levels may bring incoherent speedup measure .let take the same previous example with and .we apply a student test with a confidence level equal to 1% to ensure that is slower than . in the previous example, we showed the contrary with a confidence level equal to 95% .  ....> t.test(t2 , t1 , alternative="greater " , conf.level=0.01 ) ... 1 percent confidence interval :  0.02574667     inf  ... ....  as you can see , the test of student succeeds , so we have 1% of chance that is slower than .the computed speedup ( either by considering the sample mean of the median ) is as follows :  ....  > mean(t2)/mean(t1 ) [ 1 ] 0.5110024 > median(t2)/median(t1 ) [ 1 ] 0.5112414 ....  as you can see , the speedup here is below 1 . is this a contradictionno of course , remember that the confidence level of this speedup is only 1% .this section explained how to check with a confidence level that a code optimisation technique produces a faster transformed program ( for a fixed data input and experimental environment ) .we also provided a formula for quantifying the speedup .the following section explains how to compute an overall average of speedups of a set of benchmarks .when we implement a code optimisation technique , we are generally asked to test it on a set of benchmarks , not on a unique one .let be the number of considered benchmarks .ideally , the code optimisation technique should produce speedups on the programs ( at least no slowdown ) with the same confidence level .unfortunately , this situation is rare nowadays .usually , only a fraction of programs among would benefit from an acceleration .let be the obtained speedup for the program .while this is not correct in statistics , some reviewers ask an average speedup of all the benchmarks . in statistics , we can not provide a fair average because the programs are different , and their weights are different too .so , asking for an overall speedup for a set of benchmarks will highly bring unfair value . neither an arithmetic mean , nor a geometric or harmonic mean can be used to synthesise in a unique speedup of the whole set of benchmarks .the arithmetic mean does not distinguish between short and long programs : for instance , having a speedup of 105% on a program which lasts 3 days must not have the same impact as a speedup of 300% obtained on a program which lasts 3 seconds . in the former ,we save 5% of 3 days ( = 216 minutes ) , while in the latter we save 200% of 3 seconds ( = 2 seconds ) .if we use the arithmetic mean , we would obtain an overall speedup equal to ( 105 + 300)/2=202% , this does not reflect the reality with a fair number .the geometric mean can not be applied here because we are not faced to a succession of accelerations on the same program , but to accelerations to distinct programs .the harmonic mean in our case is not meaningful too because the quantity represents also a sort of speedup , so we can provide the same criticism as the arithmetic mean .  in order to compute an overall _ performance gain factor _( not an overall speedup ) that represents the weights of the different programs , we can use the following method .the confidence level of this performance gain factor is equal to the minimal value of confidence levels used in the student tests to validate individual speedups .first , an interesting question is to decide if we should neglect the programs where no speedup has been validated by the student test .that is , the performance gain factor is computed for a subset of programs , not on all the benchmarks .we believe we neglect the programs that fail in the student test if we study afterwards ( in the next section ) the confidence interval of the proportion : studying this proportion helps us to decide if the reported overall gain is meaningful . if we decide to include all the programs for computing the overall performance gain factor , this is also fair , but the reported gain may be negative since it includes the slowdowns .second , we associate a weight to each program . the general characteristics of a weight function is .if not , we should normalise the weights so that they sum to 1 .the weight of each benchmark can be chosen by the community , by the benchmark organisation , by the user , or we can simply decide to associate the same weight to all benchmarks . also , it is legitimate to choose the weight as the fraction between the observed execution time and the sum of all observed execution times : .here we choose to put , ie , the median of all the observed execution times of the program .someone would argue that this would give more weight on long running time programs : the answer is yes , because what we want to optimise at the end is the absolute execution time , not the relative one .third , transforming a program into allows to reduce the execution time by .this absolute gain should not be considered as it is , but should be multiplied by the weight of the program as follows : .fourth and last , the overall performance gain factor is defined as the fraction between weighted gains and the sum of weighted initial execution times : . by simplification ,we obtain : by definition , the overall gain , since the execution times of the optimised programs are hopefully non zero values ( ) .let a program p1 that initially lasts 3 seconds .assume we succeed to accelerate it with a factor of 300% with a confidence level .thus , its new median execution time becomes 1 second .let p2 be a program that initially lasts 1 hour and has been accelerated with a factor of 105% with a confidence level .thus , its new median execution time becomes 3428 seconds .the arithmetic mean of these two speedups is 202.5% , the geometric mean is 177.48% and the harmonic mean is 155.56% .none of these means is suggested for publications as explained before .the weights of the programs p1 and p2 are resp . and .the obtained weighted gain for each program is : and .the overall performance gain factor is then and the confidence level is equal to . if we consider that the weights are unit , , then the overall performance gain factor is then and the confidence level is still equal to . as can be remarked , there is not a direct comparison between the overall gain and the individual speedups .the following section gives a method to evaluate the quality of a code optimisation method .precisely , we want to evaluate the chance that a code optimisation technique produces a speedup on a program that does not belong to the initial set of experimented benchmarks .computing the overall performance gain for a sample of programs does not allow to estimate the quality nor the efficiency of the code optimisation technique .in fact , within the programs , only a fraction of benchmarks have got a speedup , and programs got a slowdown . if we take this sample of program as a basis , we can measure the chance of getting the fraction of accelerated programs as .the higher is this proportion , better would be the quality of the code optimisation .in fact , we want to estimate if the code optimisation technique is beneficial for a large fraction of programs .the proportion has been observed on a sample of programs .the confidence interval for this proportion ( with a confidence level ) is given by the equation , where .in other words , the confidence interval of the proportion is equal to  ] . or , if we do not want to reduce the confidence level , we need to do more experiences on more benchmarks .the next formula gives the minimal number of benchmarks requested if we want to estimate the confidence interval with a precision equal to with a confidence level :   in the previous example , we have got an initial proportion equal to .if i want to estimate the confidence interval with a precision equal to 5% with a confidence level of 95% , i put and i read in the quartiles tables .the minimal number of benchmarks to observe is then equal to : .we need to experiment 378 benchmarks in order to assert that we have 95% of chances that the proportions of accelerated programs are in the interval .the discussion that we can have here is on the quality or on the representativeness of the sample of benchmarks .this is outside the scope of the paper ! until now, we do not know what does a set of representative programs means .program performance evaluation and their optimisation techniques suffer from the disparity of the published results .it is of course very difficult to reproduce exactly the experimental environment since we do not always know all the details or factors influencing it .this article treats a part of the problem by recalling some principles in statistics allowing to consider the variance of program execution times .the variance of program execution times is not a chaotic phenomena to neglect or to smooth ; we should keep it under control and incorporate it inside the statistics we publish .this would allows us to assert with a certain confidence level that the results we publish are reproducible under similar experimental environment .  using simulators instead of real executions provide reproducible results ,since simulators are deterministic : usually , simulating a program multiple times should always produce the same performance numbers .this article assumes that the observations have been done on the physical machine not by simulation .if the physical machine does not exist , the observations based on simulation can not be studied exactly with the methods described in this article .the study should more be concentrated on the statistical quality of the simulator . as far as we know, it does not exist yet a simulator that has been rigorously validated by statistics as described in .usual error ratios reported by simulators are not sufficient alone to judge about their quality .this article does not treat performance evaluation with multiple data inputs of a program .in fact , the speedups defined in this article are computed for a unique set of data input . experimenting multiple sets of datainput to measure a speedup is let for a future work .we conclude with a short discussion about the confidence level we should use in this sort of statistical study .indeed , there is not a unique answer to this crucial question . in each context of codeoptimisation we may be asked to be more or less confident in our statistics . in the case of hard real time applications ,the confidence level should be high enough ( more than 95% for instance ) , requiring more experiments and benchmarks . in the case of soft real time applications ( multimedia , mobile phone , gps , etc . ), the confidence level can be more than 80% . in the case of desktop applications, the confidence level should not be necessarily high . in any case, the used confidence level for statistics must be declared for publication .  9  raj jain . _the art of computer systems performance analysis : techniques for experimental design , measurement , simulation , and modelling_. john wiley and sons , inc ., new york , 1991 .pierre - andr cornillon , arnaud guyader , franois husson , nicolas jgou , julie josse , maella kloareg .ric matzner - lober , laurent rouvire ._ statistiques avec r_. presses universitaires de rennes , socit franaise de statistique , 2008 .  .r foundation for statistical computing , vienna , austria , 2008 .3 - 900051 - 07 - 0 .grigori fursin and olivier temam ._ collective optimization_. the 4th international conference on high performance and embedded architectures and compilers ( hipeac ) .a. c. davison and d. v. hinkley _ bootstrap methods and their application _ cambridge university press .1997  todd mytkowicz et amer diwan et peter f. sweeney et mathias hauswirth _ producing wrong data without doing anything obviously wrong ! _ to appear in asplos 2009 .daniel gracia prez and gilles mouchard and olivier temam ._ microlib : a case for the quantitative comparison of micro - architecture mechanisms . _micro 2004 : 43 - 54we would like to thank sebastien briais from the university of versailles saint - quentin en yvelines for his helpful remarks to improve this document .

Abstract: the community of program optimisation and analysis , code performance evaluation , parallelisation and optimising compilation has published since many decades hundreds of research and engineering articles in major conferences and journals . these articles study efficient algorithms , strategies and techniques to accelerate programs execution times , or optimise other performance metrics ( mips , code size , energy / power , mflops , etc . ) . many speedups are published , but nobody is able to reproduce them exactly . the non - reproducibility of our research results is a dark point of the art , and we can not be qualified as _ computer scientists _ if we do not provide rigorous experimental methodology .   this article provides a first effort towards a correct statistical protocol for analysing and measuring speedups . as we will see , some common mistakes are done by the community inside published articles , explaining part of the non - reproducibility of the results . our current article is not sufficient by its own to deliver a complete experimental methodology , further efforts must be done by the community to decide about a common protocol for our future experiences . anyway , our community should take care about the aspect of reproducibility of the results in the future .  [ [ keywords ] ] keywords : + + + + + + + + +  program optimisation , statistical performance evaluation


the immense popularity of generalized estimating equations ( gee ; * ? ? ? * ) for longitudinal data analysis owes much to the fact that it can account for within - cluster correlations without requiring correct specification of the working variance - covariance structure .it achieves this via the much - celebrated sandwich estimator of variance , which is consistent for the true variance matrix of the estimated parameters even when the assumed variance - correlation structure is incorrectly specified .the application of gees with a sandwich estimator of variance to general regression problems with vector - valued responses is alluded to in ( * ? ? ?* chapter 6 ) , and is closely related to the mean - covariance estimating equations method of .this note aims to make the connection between gees and vector regression more explicit , as well as to highlight the key differences between some existing and related methods .vector regression shares many features with longitudinal data both exhibit within - vector correlations and independence across vectors .however , vector regression is distinct from the longitudinal setup in two important ways :  1 .each vector component may be measurements on different types of variables .in general , we may specify marginal generalized linear models for each component of a response vector from subject via for some set of mean functions and some set of variance functions .here , , where is the number of components in each vector response , and the parameters and are the regression coefficients and dispersion parameter , respectively , for each component .the are the corresponding vectors of covariates for each component , which we can collate into a covariate matrix .the within - vector correlation between two components and , may depend on a vector of correlation parameters as well as on the marginal means and .  in a balanced longitudinal framework ,each component is a repeated measure of the same variable , and the mean and variance models ( [ eq : mean ] ) and ( [ eq : var ] ) typically have the same form for each component .that is , moreover , the correlations are typically functions of the correlation parameters only .it is possible for some components to share mean parameters .sometimes this may be due to symmetry arguments , exchangeability , or other special structures in the data .this can be thought of as a hybrid between the general vector regression and longitudinal cases .it is possible for some components to be missing for some observations see section [ se : missing ] .as is typical in regression problems , the primary interest is in estimating and carrying out joint inferences on the regression parameters .the variance correlation structure itself is of secondary importance , coming into consideration when evaluating the sampling variability of parameter estimates .a key feature of the approach examined in this note is that the variance and correlation structures need not be correctly - specified . as in the longitudinal setting of ,we find in section [ se : estimation ] that the sandwich estimator of variance is consistent even if the working variances and correlations are misspecified .of course , the closer the working variance - correlation structure is to the truth , the more efficient and accurate the gee framework tends to be for finite sample sizes , as demonstrated in the simulation studies in section [ se : sim ] .there are two main existing frameworks for vector regression .the method described in ( * ? ? ? * chapter 6 ) uses copula functions to combine marginal distributions with association structures to construct full probability models for the data .the gaussian copula is the most popular choice , although other copula functions exist and can also be used .a main drawback of copula - based methods , as with other parametric models , is that the marginal distributions , association structures and copula function are all implicitly assumed to be correctly - specified .an alternative to full probability models is an estimating equations approach .the recent book from , which accompanies the state - of - the - art r package ` vgam ` , covers marginal models and estimating equations for vector glms that are similar in aims to the current proposal .however , the function ` vglm ` for fitting vector generalized linear models currently can not handle vectors of mixed type , such as continuous - binary pairs . for vectors with components of the same type , the approach examined here can be implemented as a simple plug - in " to ` vglm ` , offering an adjustment to model - based standard errors in cases where the working model is incorrectly specified .this typically involves just a few lines of code ; an example is given in appendix [ ap : plugin ] .  for vector responses of mixed type , an alternative estimating equations approach is the mean - covariance estimating equations ( mcee ) framework of which is motivated from a working quadratic exponential family for the responses .more precisely , the response vectors are assumed to have joint densities of the form where , are functions that characterises the shape " of the joint densities , and and are the canonical parameters in the quadratic exponential family which in turn define the marginal means , , and the variance and covariances , . a full parametric model specification would involve specifying the functions , but the mcee can be formulated without reference to any specific probability distribution , because the score equations for the mean , variance and covariance parameters can be shown to have the same functional form regardless of the underlying quadratic exponential family ( see appendix 2 of * ? ? ?the subsequent estimating equations are given by equation 6 in , from which it can be seen that in addition to specifying the marginal means ( [ eq : mean ] ) , marginal variances ( [ eq : var ] ) and within - vector correlations ( [ eq : corr ] ) , there are additional requirements of a working model for , the variance of the empirical covariance vectors , where for , and a working model for , the covariance between the response vectors and ( see equation 7 of * ? ? ?apart from some special cases , it is generally difficult to specify appropriate models for these additional higher - order terms .  the mcee approach is useful when there is direct interest in the variance - covariance structure . if the interest lies primarily in the regression parameters , then the introduction of a second - order model via the quadratic exponential family ( [ eq : quadexp ] ) adds an extra level of model complexity that is not required .moreover , the quadratic exponential family model is typically inappropriate if some of the components are non - gaussian .for example , if one of the components is binary , then the mean completely determines the variance and the corresponding quadratic terms need to be eliminated from the quadratic exponential family ( [ eq : quadexp ] ) and subsequent estimating equations in order to avoid degeneracy ( see section 4 of * ? ? ? * ) .if one of the components is a gamma - type random variable , say , then it is perhaps more sensible to consider a second - order exponential family defined in terms of the sufficient statistics rather than .the latter is appropriate for approximately gaussian responses .in contrast , the gee approach we examine here is simpler to specify , and can be formulated without reference to any underlying joint distribution for the data one only needs to specify a set of marginal mean models ( [ eq : mean ] ) , a set of working variance functions ( [ eq : var ] ) , and a set of working within - vector correlation functions ( [ eq : corr ] ) . of course ,context - specific parametric models , such as the hierarchical continuous - binary model of , the joint continuous - discrete models of ( * ? ? ?* chapter 24 ) or the latent variable models of , can also be used to motivate the working variances ( [ eq : var ] ) and correlations ( [ eq : corr ] ) , but the gee approach can be formulated without actually imposing any of these underlying joint distributions .the sandwich estimator of variance can then be applied for asymptotically correct inferences on the mean model ( [ eq : mean ] ) even under misspecification of the working variance and/or correlation model .we highlight the differences between the copula , gee and mcee approaches using the following two examples .the first is a dataset with vector responses of mixed type , and the second is a dataset with bivariate responses on a finite lattice .it is difficult to specify joint distributions for the data in either scenario , and it is in such cases where a gee approach proves invaluable .  in the following ,we drop the first subscript in our notation and simply write for the component of a generic response vector , and for the corresponding row of a generic design matrix , for clarity of exposition .the full dataset can then be considered as independent copies of the generic predictor - response pair . describes a dataset from containing 981 cases of burn injuries , with two response variables of mixed type .these are severity of burns as measured by and disposition of death as measured by for death and 0 for survival .it is of interest to see whether severity of burns and probability of death is related to age .burn severity is continuous and disposition of death is binary , so a default set of marginal mean functions is with a corresponding set of marginal variance functions , here , the dispersion parameters are taken to be and , respectively .  for a copula - based approach , the marginal mean and variance functions are first formalized into marginal distributions ,say , then , a joint distribution for the response pair can be constructed using a gaussian copula , for example , via here , is the cumulative distribution function ( cdf ) of a , is the cdf of a bernoulli , is cdf of a standard normal , and is the cdf of a bivariate normal with mean 0 , unit variances and correlation , which describes the level of association between burn severity and incidence of death .other copula functions can also be used .the model parameters are then estimated via maximum likelihood , but the procedure is typically computationally demanding as second order derivatives of the log - likelihood are not readily available ( see * ? ? ?* section 6.4 ) .alternative computational methods include the maximization - by - parts approach from and the gauss - newton type approach of .the r package ` gcmr ` of implements gaussian copula regression models via importance sampling , but currently can not handle data of mixed type .in contrast , a gee approach simply requires specifying a working correlation model for the data .this can be motivated from a working joint distribution , such as that induced by a copula approach , or specified without reference to any joint distribution .for example , a simple model here would be that severity of burns has an arbitrary constant correlation with death status within each individual , a more flexible model is to allow the correlation to change with the marginal means .as we show in section [ se : estimation ] , any reasonable working variance and correlation model can be used , with model misspecifications being adjusted for using a sandwich estimator of variance .  for a mcee model , in addition to a working correlation model , one needs to specify working models for and , where .note that has been removed from the empirical covariances as the mean of a binary response determines its variance . offer three ways of specifying these additional terms :  1 .independence working matrices , with and 2 .gaussian working matrices , with and 3 .gaussian matrices with common third and fourth - order correlations , with and where and are additional parameters to be estimated . prescribe a method - of - moments approach for estimating these parameters that are -consistent .  as mentioned in the introduction ,the mcee approach is useful when the variance - covariance structure is directly of interest .if the interest lies primarily in the regression parameters , then the introduction of these additional terms adds an extra level of model complexity over the gee approach that is not required . describes a dataset consisting of pairs of itching scores from left and right eyes of 41 subjects in a sorbinil retinopathy trial .the scores range from 0 ( no itch at all ) to 4 ( severe incapacitating itch ) in increments of 0.5 .the active treatment was sorbinil , which could be applied to both eyes , just the left eye , just the right eye , or to neither eye .the data are displayed in table [ tab:1 ] of appendix [ se : appa ] .ccccccccccc + sorbinil & sorbinil & & sorbinil & placebo & & placebo & sorbinil & & placebo & placebo + left & right & & left & right & & left & right & & left & right + 2.0 & 2.0 & & 1.0 & 1.5 & & 2.5 & 2.0 & & 3.0 & 3.0 + 1.0 & 1.0 & & 2.0 & 2.5 & & 2.5 & 2.5 & & 2.0 & 3.0 + 0.5 & 2.0 & & 3.0 & 1.0 & & 3.0 & 3.0 & & 2.5 & 2.5 + 2.5 & 1.0 & & 2.0 & 3.0 & & 2.5 & 2.0 & & 1.0 & 3.0 + 3.0 & 2.5 & & 3.0 & 2.5 & & 1.0 & 0.5 & & 2.0 & 2.5 + 2.0 & 2.5 & & 2.0 & 3.0 & & 2.0 & 0.0 & & 2.0 & 1.0 + & & & 3.0 & 3.0 & & 3.0 & 2.5 & & 2.0 & 2.0 + & & & 0.5 & 1.5 & & 3.0 & 1.0 & & & + & & & 3.0 & 3.0 & & 2.0 & 1.5 & & & + & & & 3.0 & 3.0 & & 0.5 & 0.0 & & & + & & & 3.0 & 3.0 & & 2.5 & 1.5 & & & + & & & 1.0 & 2.0 & & 2.0 & 2.0 & & & + & & & 1.0 & 2.0 & & 2.5 & 2.5 & & & + & & & 1.5 & 2.5 & & 2.5 & 2.5 & & & +  [ tab:1 ]  both and are scores between 0 and 4 inclusive , so the mean functions should also be constrained in this range for any value of the covariate(s ) .moreover , it is sensible for the variance functions to satisfy as for either .in particular , constant variance is not appropriate here .one way to take these constraints into consideration is to transform the itchiness scores into the unit interval $ ] via and , and then treat the transformed responses as pseudo - proportions ( c.f .example 9.2.4 in * ? ? ?* ) using a pair of logistic mean functions , }{1+\exp\left[\beta_{l0 } + \beta_{l1 } i(\mbox{sorbinil}_l = 1 ) \right ] } \ , \\ e(\tilde y_r|\mbox { treatment } ) & = & \mu_r = \frac{\exp \left[\beta_{r0 } + \beta_{r1 } i(\mbox{sorbinil}_r = 1)\right]}{1+\exp \left[\beta_{r0 } + \beta_{r1 } i(\mbox{sorbinil}_r = 1)\right ] } \ , \end{aligned}\ ] ] where is an indicator for sorbinil treatment in the left eye , and similarly for .a corresponding pair of working variance functions is here , we have assumed equal dispersion in the two eyes , so that the dispersion parameters are .  for a copula - based approach ,the marginal mean models and variance functions need to be formalized into marginal distributions .however , it is not easy to specify distributions on the finite set of values , or on the original scale , that satisfy the marginal mean and variance constraints .thus , a semiparametric estimating equations approach proves to be invaluable here , as one only needs to specify a working correlation model without necessarily invoking marginal or joint distributions for the data .  for a gee approach ,a simple working model for the within - subject correlation between the two eyes is or perhaps a more flexible model that allows the correlation to change with the marginal means , such as that induced by the odds ratio model of .this is considered in more detail in section [ se : sim2 ] .again , any reasonable working model can be used here , with potential misspecifications being adjusted for using a sandwich estimator of variance .  to formulate a mcee model , in addition to a working correlation model one needs to specify working models for and , where .note that and have been removed from the empirical covariances because the variances of proportions are determined by their means .the following are three specifications described in :  1 .independence , with and .2 .  independence with structural nonzeros , with 3 .gaussian scores , with and .the primary interest here is to estimate and make inferences on the treatment effect of sorbinil on reducing eye irritation .however , we may also be interested in testing for _ symmetry _ in the two eyes , and whether treatment applied to one eye _interferes _ with itchiness levels of the other eye .the gee approach enables us to make valid inferences on the regression parameters without having to specify correct marginal variances , within - subject correlation , and working models for higher - order moments .in addition to being simpler to formulate , the gee approach is also straightforward to implement .section [ se : estimation ] outlines the estimation and inference procedure in the gee vector regression framework .given a set of variance functions ( [ eq : var ] ) and correlation functions ( [ eq : corr ] ) , a working variance - covariance matrix for each observation can be constructed via where is a diagonal matrix of the variances of each component of , and is the correlation matrix induced by ( [ eq : corr ] ) .note that , unlike the longitudinal case , dispersion parameters are generally included in the working covariance matrices .this is because each component may have different dispersion parameters which can not be factorised out .if a working independence correlation structure is used , however , then the dispersion parameters drop out , as in the longitudinal setting .an estimator of can be defined as the solution to the set of generalized estimating equations , where are the residuals and is a matrix of derivatives .we iteratively solve for the regression coefficients and the correlation and dispersion parameters by alternating the following two steps .given current estimates , and , an update for can be obtained via fisher scoring ( c.f . equation 8 in * ? ? ?* ) conversely , given an estimate of , consistent estimates of the dispersion and correlation parameters can be obtained via method - of - moment approaches analogous to those from section 4 of .for example , in the general case where each component has its own dispersion parameter , we can estimate via for an unstructured working correlation model , can be estimated via note that other popular correlation models from the longitudinal setting , such as exchangeable , -dependence , auto - regressive and band diagonal structures , are generally not applicable to vector regression because the indices of the vector components are exchangeable labels .thus , the working independence and unstructured correlation models are perhaps the only two generally applicable working correlation structures for vector regression . other dependence models , such as the odds ratio model for bivariate binary regression of , will be specific to the problem at hand .  the following results are vector regression analogues of the results in , concerning the asymptotic normality of the gee estimator and consistency of the sandwich estimator for the asymptotic variance .[ result:1 ] under conditions of theorem 2 from , we have where   [ result:2 ] can be consistently estimated by where , , and are the plug - in estimators .  in result[ result:2 ] , the term on the outside is the unadjusted , or model - based , estimator of variance .it is consistent for the asymptotic variance only if the working variance covariance model is correct .in contrast , the sandwich estimator of variance is consistent regardless of the true variance covariance structure . underlying this ,of course , is the assumption that the mean model ( [ eq : mean ] ) is correct .results [ result:1 ] and [ result:2 ] can then be combined to carry out joint inferences on regression coefficients , as prescribed in result [ result:3 ] below .[ result:3 ] under the null hypothesis , where is a given matrix of full rank and a given vector , the quadratic form has an approximate distribution for large , where is the total number of regression parameters .to assess the finite - sample accuracy of our asymptotic results , particularly under model misspecification , we carry out two sets of simulations . the first is based on the continuous - binary burn injury dataset from example [ se : ex1 ] , and focuses on the accuracy of the sandwich estimator and its inverse for estimating the variance and precision matrices , respectively , of the gee estimator .the second is based on the bivariate sorbinil eye trial dataset from example [ se : ex2 ] , and focuses on comparing the gee method with the copula approach under different working correlation models . in both settings , the type i errors for testing model parameters using the -test of result [ result:3 ] , with and without sandwich adjustments , are compared to their nominal levels .we simulate from the hierarchical continuous - binary model of to emulate the burn injury data from example [ se : ex1 ] . more precisely , to generate each dataset we first sample or ages from the 981 original observations , then , conditional on the ages , we generate burn severity and incidence of death via the hierarchical model , by generating conditional on , we induce within - vector correlation for each subject , with the parameter controlling the strength of this correlation . here , we set .the parameter values are taken from the fitted model to the original dataset , so as to emulate the original values .we repeat this procedure to generate 1,000 datasets .  for each synthetic dataset, we fit the gee model from example [ se : ex1 ] with the working independence assumption , computing the gee estimate of along with its estimated variance matrix using the sandwich estimator from result [ result:2 ] .note that the marginal variance function for is misspecified , along with the working independence assumption .it is in such cases where a robust adjustment to the variances might be invaluable .the code for computing the sandwich adjustment is particularly simple and provided in appendix [ ap : b1 ] . in the special case of working independence ,the gee model coincides with an independent copula model with gaussian and binary marginal distributions .  to assess the accuracy of the sandwich variance estimator prescribed in result [ result:2 ] , we compute the matrix 1-norm for each simulation , where is the empirical true " variance - covariance matrix across our simulations .we also examine the estimation accuracy for the precision matrix , , which is arguably more pertinent for inferences than the variance . for each simulation , we compute the analogous matrix norm .the results are summarised in rows 1112 of table [ tab:2 ] .we see that the sandwich adjustment offers much improved performance in estimating both the variance and precision matrices , with the improvement becoming more pronounced when the sample size increases .in contrast , the unadjusted precision estimator actually diverges with increasing sample size , which leads to biased inferences on the regression parameters .indeed , the type 1 errors ( rows 1315 in table [ tab:2 ] ) for jointly testing the true parameter values and at nominal 1% , 5% and 10% levels are seen to be severely biased when the unadjusted variances are used . on the other hand , inferences usingthe adjusted variances have type i error rates that are much closer to their nominal levels .these tests were carried out using the -test from result [ result:3 ] , calibrated against the -distribution with 4 and degrees of freedoms .table [ tab:2 ] also displays the accuracy of the adjusted and unadjusted variance estimators at a component - wise level .more precisely , the adjusted and unadjusted means and standard deviations of the estimated standard deviations and correlations of are given in rows 110 .we see that the adjusted estimator leads to more precise estimation of each standard deviation and correlation term .  .burn injury simulations ( i ) component - wise means ( and standard deviations ) of estimated asymptotic standard deviations and correlations , ( ii ) mean accuracy of estimated variance and precision matrices , and ( iii ) type i errors for jointly testing all model parameters at nominal 1% , 5% and 10% levels . simulations each with sample sizes and .[ cols= " < , > , > , > , > , > , > , > , > " , ]   [ table:5 ]   the gee framework for vector regression is readily adaptable to handle cases where some components are missing completely at random for some observations .suppose , for example , that component is missing from observation . then , the contribution of the observation to the estimating equation can be modified to , where and denote the matrices and , respectively , with row and column removed , and is the vector with element removed .   for responses that are missing at random, developed a class of inverse - probability weighted gee models for longitudinal data .this was extended by to handle nonignorable missingness .adapting these methods to vector regression settings is a topic for future research .a reviewer pointed out that there exists a multitude of joint models for repeated measures and longitudinal data ( e.g. , * ? ? ? * ; * ? ? ?* chapter 24 ) , some of which may be adaptable for vector regression problems . however , the general applicability of such methods to vector regression settings is not immediate .for example , the model from ( * ? ? ?* chapter 24.2.1 ) dichotomizes one component of a bivariate normal vector to generate a probit normal joint model for binary - continuous response pairs , but it is not clear how this dichotomization can be generalized to a discretization for handling count continuous responses . moreover , even if an analogous discretization can be formulated , the resulting mean model will generally not be of an easily - interpretable form , such as a log - linear model .in contrast , model specification in the gee vector regression approach is done simply via the first two moments , so that count components can be handled by specifying , say , a log - linear mean model in ( [ eq : mean ] ) and a linear variance function in ( [ eq : var ] ) ; this is no harder to specify than for continuous or binary components .a more generic joint modelling approach is the generalized linear mixed model ( glmm ) formulation ( e.g. , * ? ? ?* chapter 24.2.3 ) which induces within - vector correlations via subject - specific random effects .while mixed models for response vectors of mixed type are conceptually no different to mixed models for repeated measures of the same type , on a practical level it is well - known that for non - gaussian responses misspecification of the random effects distribution can lead to severely biased estimation and inferences on model parameters ( e.g. , * ? ? ?moreover , mixed models generally do not preserve marginal mean structures when non - identity links are used .for example , including random effects in a binary logistic model typically leads to a marginal mean model that is no longer logistic .in contrast , gee vector regression explicitly models the marginal means via ( [ eq : mean ] ) , so that model specification and parameter interpretation are both simple and invariant to the underlying dependence structure . formulated a class of joint models for a sequence of longitudinal measures and an associated sequence of event times .this class covers the models of and as special cases .the association between the two sequences of observations is characterized through a latent bivariate gaussian process indexed by time .if there is no longitudinal aspect , this approach reduces to a glmm with normal random intercepts , and so the drawbacks of mixed models also hold here .additionally , inferences on model parameters assume that the underlying latent gaussian process is correctly specified , and ( currently ) no adjustments are available in the case of model misspecification . however , one very attractive feature of the approach is that it can handle within - subject dependence both between the bivariate vector components and across time .the extension of the gee vector regression approach to allow for longitudinal dependence will be an important and invaluable development .the idea of extending gees to general vector regression problems has been suggested in the literature , but generally in an ad hoc manner .this paper formalises the connection .the main attraction of the gee approach is its simplicity in model specification and the guarantee of asymptotically correct inferences without requiring correct models for the variances , correlations , or underlying joint distributions for the data .this is particularly useful when a full distributional model for the response vectors is hard to specify , such as in examples [ se : ex1 ] and [ se : ex2 ] .the guarantee of asymptotically valid inferences relies on the sandwich estimator of variance ( result [ result:2 ] ) . although our simulations demonstrate that gee vector regression can be more robust to model misspecification than copula - based methods for moderate sample sizes , it is well - known that the sandwich estimator can be severely biased in small - sample settings .small - sample modifications of the sandwich estimator have recently been proposed for longitudinal settings ( e.g. , * ? ? ?* ; * ? ? ?extending these modifications to the general vector regression case is a direction for future research .r software for implementing general gee vector regression models is currently in development .one arm of the development is to implement plug - ins " to the many multivariate families available in ` vglm ` from the ` vgam ` package . this will be useful for vector responses of the same type .a second arm will implement gee models for vectors of mixed type , with scope for handling a wide range of working correlation structures . although a general software package is not yet available , gee vector regression models are straightforward to implement on a case - by - case basis , as demonstrated via the simple r routines for implementing examples [ se : ex1 ] and [ se : ex2 ] provided in the appendix .  99  dunson , d. b. ( 2000 ) ` bayesian latent variable models for clustered mixed outcomes ' , _ journal of the royal statistical society : series b _ , * 62 * , 355 - 366.  fan , j. and gijbels , i. ( 1996 ) _ local polynomial modelling and its applications_. london : chapman and hall .fay , m. p. , and graubard , b. i. ( 2001 ) , ` small - sample adjustments for wald - type tests using sandwich estimators ' , _ biometrics _ , * 57 * , 11981206 .fitzmaurice , g. m. and laird , n. m. ( 1995 ) ` regression models for a bivariate discrete and continuous outcome with clustering ' , _ journal of the american statistical association _ , * 90 * , 845852 .henderson , r. , diggle , p. and dobson , a. ( 2000 ) ` joint modelling of longitudinal measurements and event time data ' , _ biostatistics _ , * 1 * , 465480 .liang , k - l . and zeger , s. ( 1986)`longitudinal data analysis using generalized linear models ' , _ biometrika _ , * 73 * , 1322 .litire , s. , alonso , a. , and molenberghs , g. ( 2008 ) ` the impact of a misspecified random - effects distribution on the estimation and the performance of inferential procedures in generalized linear mixed models ' , _ stat med _ , * 27 * , 31253144 .mancl , l. a. , and derouen , t. a. ( 2001 ) , ` a covariance estimator for gee with improved small - sample properties ' , _ biometrics _ , * 57 * , 126134 .masorotto , g. and varian , c. ( 2015 ) _ gcmr , _ r - package .mccullagh , p. and nelder , j. a. ( 1989 ) _ generalized linear models_. 2nd edition .london : chapman and hall .molenberghs , g. and verbeke , g. ( 2005 ) _models for discrete longitudinal data_. new york : springer series in statistics .prentice , r. l. and zhao , l. p. ( 1991 ) ` estimating equations for parameters in means and covariances of multivariate discrete and continuous responses ' , _ biometrics _ , * 47 * , 825839 .robins j.m . , rotnitzky a. and zhao l.p .( 1995 ) ` analysis of semiparametric regression models for repeated outcomes in the presence of missing data ' , _ journal of the american statistical association _ , * 90 * , 106121 .rosner , b. , glynn , r. j. , and lee , m - l .t. ( 2006 ) ` extension of the rank sum test for clustered data : two - group comparisons with group membership defined at the subunit level ' , _ biometrics _ , * 62 * , 12511259 .rotnitzky a. , robins j.m . , and scharfstein d.o .( 1998 ) ` semiparametric regression for repeated outcomes with nonignorable nonresponse ' , _ journal of the american statistical association _ , * 93 * , 13211339 .ruppert , d. ( 2005 ) comment on maximization by parts in likelihood inference" , _ journal of the american statistical association _ , * 100 * , 11611163 .song , p. x - k .( 2007 ) _ correlated data analysis : modeling , analytics and applications_. springer series in statistics .song , p. x - k . , fan , y. , and kalbfleisch , j. d. ( 2005 ) ` maximization by parts in likelihood inference ' , _ journal of the american statistical association _ , * 100 * , 11451167 .tsiatis , a. a. , degruttola , v. , and wulfsohn , m. s. ( 1995 ) ` modeling the relationship of survival to longitudinal data measured with error .applications to survival and cd4 counts in patients with aids ' , _ journal of the american statistical association _ , * 90 * , 2737 .wulfsohn , m. s. , and tsiatis , a. a. ( 1997 ) ` a joint model for survival and longitudinal data measured with error ' , _ biometrics _ , * 53 * , 330339 .yee , t. w. ( 2015 ) _ vector generalized linear and additive models_. springer series in statistics .yee , t. w. ( 2016 ) _vgam_. r - package .zeileis , a ( 2006 ) ` object - oriented computation of sandwich estimators ' , _ journal of statistical software _ ,* 16 * , 116 .zhao , l. p. and prentice , r. l. ( 1990 ) ` correlated binary regression using a quadratic exponential model ' , _ biometrika _ , * 77 * , 642648 .table 1 goes here .# compute sandwich estimator of variance meat = 0 for ( i in 1:n ) {   di = bdiag(model.matrix(fit1)[i,],(mu2[i]-mu2[i]^2)*model.matrix(fit2)[i , ] )   wi.inv = diag(c(1/w1[i ] , 1/w2[i ] ) )   si = rbind(y1[i ] - mu1[i ] , y2[i]-mu2[i ] )   meat = meat + di%*%wi.inv%*%si%*%t(si)%*%wi.inv%*%t(di ) } sandwich.vcov = naive.cov%*%meat%*%naive.cov ....      # fitted marginal probabilities mu.c = apply(fitted[,c(5,6,7,8 ) ] , 1 , sum ) mu.b = apply(fitted[,c(3,4,7,8 ) ] , 1 , sum ) mu.k = apply(fitted[,c(2,4,6,8 ) ] , 1 , sum ) mu.cb = apply(fitted[,c(7,8 ) ] , 1 , sum ) mu.ck = apply(fitted[,c(6,8 ) ] , 1 , sum ) mu.bk = apply(fitted[,c(4,8 ) ] , 1 , sum )   # compute sandwich estimator of variance   meat = 0 ; bread = 0 for ( i in 1:392 ) {  wi = rbind(c(var.cc[i ] , cov.cb[i ] , cov.ck[i ] ) ,        c(cov.cb[i ] , var.bb[i ] , cov.bk[i ] ) ,        c(cov.ck[i ] , cov.bk[i ] , var.kk[i ] ) )  di = bdiag(mu.c[i]*(1-mu.c[i])*cbind(1 , altitude[i ] ) ,        mu.b[i]*(1-mu.b[i])*cbind(1 , altitude[i ] ) ,        mu.k[i]*(1-mu.k[i])*cbind(1 , altitude[i ] ) )  si = rbind(cyadea[i ] - mu.c[i ] , beitaw[i ] - mu.b[i ] , kniexc[i ] - mu.k[i ] )  meat = meat + t(di)%*%solve(wi)%*%si%*%t(si)%*%solve(wi)%*%di  bread = bread + t(di)%*%solve(wi)%*%di } sandwich.vcov = solve(bread)%*%(meat)%*%solve(bread ) ....

Abstract: generalized estimating equations ( gee ; * ? ? ? * ) for regression problems with vector - valued responses are examined . when the response vectors are of mixed type ( e.g. continuous binary response pairs ) , the gee approach is a semiparametric alternative to full - likelihood copula methods , and is closely related to the mean - covariance estimation equations approach of . when the response vectors are of the same type ( e.g. measurements on left and right eyes ) , the gee approach can be viewed as a  plug - in " to existing methods , such as the ` vglm ` function from the state - of - the - art ` vgam ` r package of . in either scenario , the gee approach offers asymptotically correct inferences on model parameters regardless of whether the working variance - covariance model is correctly or incorrectly specified . the finite - sample performance of the method is assessed using simulation studies based on a burn injury dataset and a sorbinil eye trial dataset . the method is applied to data analysis examples using the same two datasets , as well as on a trivariate binary dataset on three plant species in the hunua ranges of auckland .