the characteristic approach has been used successfully to carry out numerical simulations of space - times with and without sources  .a significant computational effort is nevertheless necessary to extend its range of applicability to the simulation of astrophysically relevant sources of gravitational radiation , such as the black hole - neutron star binary problem , where the approach can be most useful .as work on the characteristic formulation to date illustrates  , it is clear that most three - dimensional characteristic simulations , even vacuum simulations  are resolution - limited .this is particularly true of three - dimensional simulations of systems containing compact matter sources  , even when the matter source is an extended one . in general, all these simulations have been limited in resolution primarily because of the time required to integrate the equations numerically .for instance , at the finest resolution simulation considered in  , tracking a neutron star in ( close ) orbit around a black hole requires approximately 1.5 months even on one of the fastest processors currently available .this is so even though the grid in question ( angular points , points radially ) is fairly moderate by today s standards . to a lesser extent ,characteristic simulations are also limited because of memory requirements , although the characteristic scheme is particularly economic in this regard . even though it is feasible to equip a single - processor workstation with the 1.4 gbytes of memory required by that moderate grid size , the time required for the numerical solution on even the fastest processor would make such serial simulations highly impractical .most of the past code development in the characteristic approach  has been geared towards vector or single processor machines .the computational platforms available today require instead a parallel programming approach in order to perform large resolution simulations in a reasonable time , thus a parallel version of the characteristic code is clearly needed . in the present workwe show how , with a well thought out yet modest programming effort , it is not only possible to produce an efficient , highly scalable parallel implementation of characteristic codes , but to do so in such a way that it becomes straightforward to extend our parallel implementation to new physical models .we aim for our numerical implementation to be particularly useful for long time simulations of sources of astrophysical interest , which are very demanding in terms of the number of grid points on which to advance the solution , and thus on the number of floating point operations required . with that end in mind ,our numerical code must scale well in platforms with a large number of processors .we show here that our implementation meets this goal , making it a potentially valuable tool when applied to some of the most interesting astrophysical applications of numerical relativity , such as the study of black hole neutron star binary systems , in the close orbit regime up to the tidal disruption of the companion star .the first astrophysical application we have in mind is a characteristic simulation of boson stars in orbit about a black hole .for that purpose we calibrate the current code for a massless scalar field minimally coupled with gravitation .this let us , beyond numerical tests , compare and calibrate our code with linear versions of it , radial codes , and analytic ( perturbative ) results reported in the literature about quasi - normal modes .the extension to a black hole boson star system is straightforward but not trivial , deserving a detailed study of its own and enormous computational resources .we want to stress that the production of gravitational waves by the scattering of scalar waves has many mathematical features in common with the production of gravitational waves by the motion of fluid bodies .the article is organized as follows : in sec .[ sec : standard ] we review briefly the standard numerical implementation of the _ eth _ approach  , and discuss some of its drawbacks when applied to high resolution simulations in the characteristic approach to numerical relativity . in sec .[ sec : gnomic ] we present an implementation of the and operators which combines their standard description in terms of stereographic coordinates with their numerical representation on non conformal grid coordinates on the sphere .as this approach differs significantly from our previous work , we provide further motivation for this departure . sec .[ sec : numerics ] provides a detailed description of the numerical implementation of the approach outlined in sec .[ sec : gnomic ] . in sec .[ sec : ekg3d ] we illustrate how the parallel , scalable characteristic code framework ( leo ) that we have developed , can be used to implement the model problem of a scalar field minimally coupled to gravity in three dimensions .[ sec : addnumer ] expands on additional numerical considerations specific to the hypersurface and evolution equations , and to the boundary conditions , and presents convergence tests of our numerical implementation . in sec .[ sec : results ] , the viability of the approach is demonstrated by clearly resolving several problems which could not be tackled previously .we close in sec .[ sec : conclusions ] with concluding remarks and an outline of future work .the characteristic approach to numerical relativity is based on null coordinates , with the retarded time , a luminosity distance and coordinates on the sphere  . in its 3-dimensional implementation , the angular coordinates chosen are stereographic coordinates on the sphere , which is covered with two stereographic coordinate patches , as first presented in ref .we summarize here the salient aspects of the approach to provide the motivation for ( and highlight the differences with ) , the implementation described in this article . the standard _ eth _approach  is a straightforward numerical implementation in stereographic coordinates of the , operators introduced by newman and penrose  .two stereographic patches are used to cover the unit sphere , with coordinates in the north patch and in the south patch , respectively , where are standard angular coordinates .  in terms of the dyad , where , vectors on the sphere are represented by a spin - weight 1 field ( or alternatively , by a spin - weight -1 field ) .this treatment generalizes to tensors on the sphere , which are represented in terms of spin - weighted functions obtained by contracting them with the dyad and its complex conjugate , i.e. the spin of the resulting scalar function is .angular derivatives of tensor fields are represented by the action of the spin - raising and lowering operators and .for example , the angular derivatives of a vector field , where are the derivatives compatible with the flat sphere metric in the coordinates , are represented by the spin-2 field and spin-0 field given by the and operators acting on a spin - weight function are equivalently defined by    [ eq : eth ]  where , in terms of the ( real ) coordinates , , , .functions on the sphere with spin - weight transform between patches according to   we implement this numerically by laying down a two - dimensional grid on each patch , with coordinates , , such that , , , and with the grid point indices in the range .this grid covers the coordinate range .ghost zones are used on each side of the grid for the discretization of angular derivatives by centered , second - order - accurate finite difference stencils .function values at these ghost zones are obtained by interpolation from the function values on the opposite patch . in order to compute second angular derivatives to second order accuracy, the interpolations must be evaluated to fourth - order accuracy , which can be readily attained using a sixteen - point stencil in two dimensions  , provided the grid covers no less than the range indicated above . for some applications , we find it necessary to extend the grid to a finite overlap , i.e. , with .the set of ghost zones required for the north patch , maps onto the south patch ( and vice - versa ) as per the transformation , into a cloverleaf shape .regardless of the number of grid points ( or of ghost zones ) , there is a finite overlap between the north and south patches . while the points in the overlap area of each grid are redundant , we carry them all because we find it more efficient to work with rectangular grids .a potential problem of doing so is the development of two different numerical solutions in the overlap area of each patch , only loosely coupled at the stereographic patch edge .we now discuss briefly some of the drawbacks of the standard `` eth '' approach when applied to high resolution simulations in the characteristic approach to numerical relativity , and the motivation for the changes that we propose in the next section .the first objection that we encounter is in the process of parallelizing our characteristic codes .because of the radial march implicit in the radial integration of the hypersurface equations , the natural way to parallelize a characteristic simulation is to distribute the angular grid among processors , which would integrate the equations along a `` pencil '' of null rays . in the computational `` eth '' approachthis means assigning the computation of the solution over a subset of each stereographic patch to a given processor . a similar arrangement , in the context of axisymmetric simulations ,was explored earlier by bishop et . .thus , given processors , we can simply partition the stereographic grid on each patch , assigning equal square subgrids of extent on each direction to each processor .load - balancing ( the requirement that all processors in a parallel computation do approximately the same amount of work ) would in principle be achieved , so long as we restricted ourselves to explicit methods , thus guaranteeing that the time spent per subgrid remains constant .the communication pattern imposed by the two stereographic patches implementation of the `` eth '' approach  does present an obstacle to effective scaling .the mapping of ghost zones at the edge of the grid to grid points in the opposing patch is not restricted to nearest neighbors . to provide values for these ghost zonesrequires data from a set of grid points whose values are scattered among processors in an irregular pattern ( in the sense that , depending on its location on the angular grid , the ghost zones may be obtained from one , two , or more processors ) .this procedure is not only cumbersome to program , but intrinsically inefficient .if the data required for these ghost zones could be obtained just from grid points at the edge of an adjoining grid , the procedure would simplify considerably . the time spent in communicationwould be substantially reduced and remain constant over the set of processors , with a significant impact on the scalability and overall efficiency of a code .unfortunately such an arrangement is not possible with a stereographic grid .in addition , as we point out at the end of sec .[ sec : standard ] , because of the fixed overlap between patches , a significant portion of the grid is wasted .while this might be acceptable in small scale simulations  , it needs to be addressed in the context of large scale computations as in that case it translates into a serious waste of computational resources .  yet another problem that arises from the angular grid layout is that of highly non - uniform angular resolution , as a direct consequence of using a stereographic grid . considering the expression for the area element in stereographic coordinates , it can be seen that there is a marked disparity between the resolution at grid points at various places on the sphere .considerable better resolution is attained near the equator than at the poles , with the largest disparity between a grid zone at the pole ( ) and a point at a corner of the grid ( ) where the respective area elements have a ratio of 9:1 .for some situations , such as the case of a matter source in equatorial orbit around a black hole , this feature works to our advantage .conversely , for a matter source in a polar orbit , the matter source would be resolved three times as poorly when it lies along the axis ( ) compared to the resolution obtained when it crosses the equator , which in a second - order accurate code translates into a nine - fold increase in the intrinsic error in the numerical solution .one possible correction for this effect would be to maintain the standard stereographic grid as the basic computational grid , but introduce a physical grid related to the computational grid by a fish - eye stretch in the angular coordinates , and allow the refined portion of the grid to follow the compact object .a related issue is that of proper resolution of angular features , which is critical for characteristic simulations of compact objects in orbit around a black hole . inspherical coordinates , angular resolution decreases with distance to the center , as pointed out in  .there is then a limit to the distance at which we can initialize a characteristic simulation , a limit which is not dictated by physical considerations , such as the need to avoid the formation of caustics , which would lead to a break down of the coordinate system .we are also limited by the number of points which can in practice be devoted to resolve a companion object .a proposed extension to the characteristic approach  would include an adaptive mesh refinement ( amr ) strategy , which indirectly would address both grid resolution issues mentioned .while this extension may prove necessary in simulations of the last stages of capture or disruption of a companion star by a black hole , the necessary technology has not yet been developed in the context of the characteristic approach ; to our knowledge , applications of amr in the characteristic framework have been made only in simplified one- dimensional models  .the intermediate stages of the black hole neutron star problem , where the companion remains approximately contained in a finite region could , at least in principle , be equally well resolved with fixed mesh refinement , an approach successfully used , for example , in  . the introduction of amr carries with it a whole new set of issues , not the least of which is the problem of load - balancing in a massively parallel computer .current characteristic codes present serious obstacles to the implementation of an amr strategy by the nature of the angular grid alone . to be effective, an amr implementation would have to be capable of dealing with refined meshes in multiple coordinate patches .to our knowledge , few existing implementations have this capability  , and none have been applied in numerical relativity . in the present work we present a parallel implementation on a single distributed grid , and we defer the discussion of possible techniques for fixed and adaptive mesh refinement for future work .a key consideration for the present work is that different numerical representations of the approach can be developed by laying down different types of grids on the sphere .we consider here an alternative to the standard `` eth '' approach , based on the `` cubed sphere '' or `` gnomic '' covering of the sphere introduced by ronchi _et al . _ , based on earlier work of sadourny  .this approach is now in common use in global weather simulations , such as in the mit general circulation model ( mitgcm ) , for example  .it has also been applied in astrophysical simulations  , in the study of wave propagation methods on the sphere  , and more recently , in the evolution of scalar fields in a fixed background  . while preparing this manuscript , we learned  of work being carried out by bishop _et al . _ , on a similar grid arrangement , based on work by thornburg  .    in the ` cubed sphere ' method , a covering of the unit sphere with six non - overlapping patches results from projecting the sphere from its center onto the six faces of a circumscribing cube ,whose edges have length two .for example , for points on the sphere with cartesian coordinates and angular coordinate ( where ) , we project the point by tracing a line from the center of the sphere though the point to the face of the circumscribing cube , determining a point with cartesian coordinates , we then label the point on the sphere according to the cartesian coordinates of its projection on the plane .  in order to obtain a covering of the sphere with nearly uniform area , we label the points on the sphere by angular coordinates , where , , introducing an equally spaced grid in the angular coordinates , i.e. , , .similar projections from the center of the sphere to the other faces of the cube provide a covering of the sphere with six patches .  for a finite - difference code , this arrangement is ideal , as the grid on each patch is equally spaced in the angular coordinates , and the same angular coordinate is used on any two patches in the direction perpendicular to a boundary .thus any additional layers of ghost zones in the adjacent grid will fall on coordinate lines parallel to the boundary . evaluating function values at those ghost zonesrequires only one dimensional interpolation along coordinate lines parallel to the boundary .for example , an -th order centered stencil requires additional layers of ghost cells to be supplied along the edge of each spherical cap .the order of the interpolations used to supply these points can be selected so as to preserve the accuracy and maintain the desired dissipation properties of the numerical scheme . in the standard configuration  ,the spherical caps share common points along the edges of the grid , where two grids abut , and at the corners of each patch , where three grids meet .these common points must have a unique value on each of the grids that share them .an approach advocated in the literature  is to replace the function values at these shared points by some form of weighted average .here we dispense with this procedure , by carefully selecting the range of the angular coordinates in each patch , in a way that precludes the existence of points common to two ( or three ) cubed sphere patches .an advantage of a _ gnomic _ decomposition of the sphere , crucial to an efficient parallel implementation of the numerical approach , is that the patches can be laid out so that they are non - overlapping .in contrast , in a stereographic covering , there is a finite overlap , which does not shrink in size as we increase the grid resolution .although it is possible to construct a six patch stereographic covering of the sphere , in which the amount of overlap is reduced with respect to the two patch covering , the area of the overlap zone remains constant , regardless of grid size .non - conformal projections are based on non - orthogonal coordinates , thus there is no symmetry in some operators such as the laplacian on the sphere .it is still straightforward to couple a gnomic grid layout with the existing approach , i.e. while continuing to express the and operators of ref . on stereographic coordinates . in the following wedetail how this is implemented by expressing the angular ( stereographic ) derivatives in terms of angular derivatives in gnomic coordinates .a gnomic covering of the sphere is given by six coordinate patches    where , are the cartesian coordinates of the points , are coordinates on the sphere in the range , and .( numerical grid points can be set equally spaced in the coordinates , related to by , with ) .the coordinate lines at ( ) are great circles on the sphere which pass through the points where the cartesian axis intersect the sphere .for instance , the lines at are great circles spun around the axis , while those of are great circles rotated around the axis , with the respective rotation angles . similarly , on each patch, we define stereographic coordinates , where the angular coordinates on each patch are related to cartesian coordinates by    with .the gnomic coordinates and stereographic coordinates on each patch are related by or , in more compact form , in terms of a complex gnomic coordinate ,    we can express the angular derivatives in stereographic coordinates , , that enter into the and operators in terms of angular derivatives through the relations and with the jacobian we cast derivatives with respect to stereographic in terms of derivatives with respect to gnomic coordinates , which are in turn related to derivatives with respect to gnomic coordinates , by the spin - raising and lowering operators and  acting on a spin function , eq .( [ eq : eth ] ) , can be written as where the values of for each grid point are computed from the respective values as per eq .( [ eq : xi_zeta ] ) .  to complete the prescription of the operator on the cubed spherewe must specify the transformation rule for spin - weighted functions on the sphere .the stereographic coordinates on the various patches transform according to    [ eq : transf ]  where eqs .( [ eq : transfeq ] ) relate neighboring equatorial patches , eqs .( [ eq : transfneq ] ) and ( [ eq : transfseq ] ) supply the coordinate transformations between equatorial patches and the north and south patch , respectively .( [ eq : transfop ] ) , which relate diametrically opposed patches on the sphere , are not strictly necessary for a numerical implementation and can be deduced from ( [ eq : transfneq ] ) and ( [ eq : transfseq ] ) .our convention for the gnomic parameterization follows ref . , and it has the advantage that the orientation of the axes on each patch is chosen so as to reduce the amount of book keeping needed to transfer information between patches . from eq .( [ eq : transf ] ) , adjacent patches with coordinates and are related by , where and , while opposite patches are related by , with , in particular , we recover the coordinates transformation between `` north '' ( ) and `` south '' ( ) patches , , as in ref .  .given the dyad , , and expressing in two coordinate systems and , it can be seen that where and where the substitution is understood in the right hand side of ( [ eq : wzzp ] ) . from thiswe deduce the transformation rule for spin- functions on the sphere , given here only for the case of adjacent patches for the case of functions with spin - weight zero the transformations reduce to .we give here a summary of the numerical techniques used so far in the leo framework .it is worth noting that the framework is easily extensible , and thus we are not restricted , for instance , to the particular choice of radial grid made here , nor to the choice of radial or time integration schemes used in the present work . in the subsection on finite - difference operators on the sphere , for instance, we describe higher order extensions that we have elected not to use in the example application considered here , as they are inconsistent with the radial and time integration schemes , which we have taken unchanged from  .following  , we take the computational radial grid to be equally - spaced in the compactified coordinate , restricted to the range , i.e. , , , with .we express radial derivatives in terms of the compactified grid , via the relation , e.g.    we construct an equally - spaced grid on the gnomic coordinates , with , , and , . the useful part , exclusive of ghost zones , for each of the coordinates ranges from to . with this arrangement ,the points with or are excluded , and thus we avoid storing double values for the points at the edges of each patch , and triple values for the points on the corners where three patches meet . adding ghost zones on each side of the gridallows us to evaluate derivatives to order with centered stencils of the form the coefficients for the derivatives , up to -th order , are given in table [ tab : derivatives ] ..coefficients for centered angular derivatives .[ cols="<,^,^,^,^",options="header " , ]   we can verify that the coefficients of table [ tab : derivatives ] for each of the derivatives are correct by noting that the numerical error of derivatives of order is within the level of round - off when applied to a polynomial test function of order or lower .we have also verified the proper convergence rate of the numerical and operators when applied to spin - weighted spherical harmonics  .[ fig : eth - conv ] shows the proper convergence rates of the operators constructed from derivatives of second , fourth , sixth , and eighth order when applied to the spin-2 spherical harmonic on grid sizes ranging from to .operator , built upon angular derivatives of order 2 , 4 , 6 , and 8 ( indicated in the graph as circles , squares , diamonds and triangles , respectively ) , acting on , and with grid sizes ranging from to .,width=312,height=312 ]   since the ghost zones required to evaluate derivatives fall on coordinate lines parallel to the boundary , we can obtain function values at these ghost zones with one dimensional interpolations .we use standard lagrangian interpolation formulae to -th order accuracy , adapted to equally spaced grids , i.e. . fig .[ fig : interp ] shows the calibration of the interpolation routines with a test function consisting of a polynomial of order , i.e. for , where the coefficients are chosen randomly , subject to the condition .the interpolants display convergence to the correct order ( 3 , 5 , 7 , and 9-th order , respectively ) , for grid sizes in the range .order ( circles ) , order ( squares ) , order ( diamonds ) , and order ( triangles ) interpolators . for the highest order interpolatorused ( -th order ) , the error goes down to double - precision round - off level ( ) when more than angular points per patch are used.,width=312,height=312 ]  for this range of values ( ) , there are from 32 to 1024 points in the great circles determined by the intersection of the sphere with any of the cartesian coordinate planes .as expected , for smooth data such as our test function , for sufficiently large grid sizes , the error goes down to round - off level when using the higher - order schemes .this saturation effect is already visible in the plot for the -th order interpolator when angular grid sizes reach approximately . as indicated in sec .[ sec : gnomic ] , we have chosen the range of the spherical coordinates so that there are no overlapping points at the edge of each patch .this avoids the awkward procedure of averaging values from different patches to obtain a single - valued function throughout the computational grid .  the number of ghost zones , the order of the finite difference approximations and the order of interpolation , while related , are not directly tied to each other .one requirement is that we must have enough ghost zones ( ) to compute the finite - difference approximation to the desired order , ; and since in general we want to use centered differences , the relation must hold .if we wish to maintain the symmetry of the interpolation stencils , must also hold .for the cases we have considered , we find that our algorithms are stable if , with the inequality required only in the case of , the lowest order of finite - differences that we considered .we are otherwise free to vary the number of ghost zones as dictated by efficiency considerations .integrals over the sphere and volume integrals arise naturally , in particular when computing norms of various quantities .we evaluate integrals on the sphere to second order accuracy by evaluating the area element in gnomic coordinates , evaluating the function value on grid cell centers , , and summing over grid cells , where stands for the grid spacing on both coordinates ( , which we have taken to be the same .note that since the spherical patches do not overlap , the integral over the sphere is just the sum of the integrals over the individual patches .[ fig : integral - conv ] shows the converge of the integral of the area element itself to the correct answer of for grid sizes in the range of to .   to .the markers indicate the error of the area element at the corresponding resolution , the line is the least - squares fit , yielding a convergence rate of .,width=312,height=312 ]  the measured convergence rate is , in full agreement with the expected result .volume integrals are computed similarly to integrals over the sphere , but in this case evaluating the spherical contributions mid - point in between radial points , i.e. we replace the flat volume element , , with the volume element corresponding to a bondi metric , , when appropriate . to speed up the evaluation of integrals , we pre - compute the area element on the sphere , eq .( [ eq : area ] ) .we make use of spin - weighted spherical harmonics throughout this paper , following the convention of  . in order to estimate the error introduced when we perform a spin - weighted spherical harmonic decomposition , we look at how well the orthonormality condition is preserved ( at the numerical level ) for spherical harmonics with spin - weight , 1 and 2 , for a range of values of and and angular grid sizes .as expected , the numerical value of the integral converges to the analytic result to second order on the grid spacing , since we have chosen to use a second - order integration algorithm .[ fig : f_l=6 ] illustrates one instance , where we have taken , , with , and varied the angular grid size from to ., for the case , , on grid sizes ranging from to ,width=312,height=312 ]  we can also place an estimate on the accuracy of the projection of a spin weight function , = \int_{s } f \ { } _ { s } \bar{y}_{lm } d\omega , \ ] ] based on the magnitude of the off - diagonal values in ( [ ylm_ortho ] ) for a given grid size .when projecting the test functions into the spherical harmonics for , , at the analytic level we would expect to obtain zero for all coefficients , except for which would be identically one .we find that grid sizes of and larger are sufficient to keep the error in the coefficients to within one part in , which again is consistent with our integration scheme being second - order in the angular discretization .[ fig : proj_error ] shows the error in the coefficients computed for , , on a grid with points .we have omitted from the graph those coefficients for which the error is already at the level of round - off .when the function being projected is the spherical harmonic , on a grid of points .coefficients whose error is at round off level are not shown.,width=312,height=312 ]  the preceding description of the numerical implementation is complete but for one key aspect , namely our parallelization strategy . in our framework , the six cubed - sphere grid patches are decomposed into _ computational _ sub - patches , each with the same number of points on the angular directions , for efficiency reasons .these sub - patches are distributed among processors , and the ghost - zone values required for the computation of angular derivatives are communicated by the use of message - passing calls  .the radial direction is not distributed , as the characteristic algorithm requires a radial march for the integration of the hypersurface equations as well as the evolution equations  .the computational complexity of a parallel implementation via message passing lies in that , knowing the location of its assigned grid sub - patch on the global grid , each processor must determine which processors are its nearest neighbors , i.e. to which processes it must supply ghost - zone information ( and also receive that information from ) . due to the relative orientation of the cubed - sphere patches , we need to know whether the order in which the ghost zones are traversed must be reversed for sub - patches on the edge of a cubed - sphere patch .since the sub - patch to processor mapping remains constant during a simulation , the relevant information needs to be computed only once , and at any rate , it incurs no measurable overhead in the computation involved in a simulation .an efficient and scaling implementation of the message passing itself requires only a small subset of the full mpi functionality : a few calls to set up the appropriate groups of processors ; sends , receives and waits ( for ghost zone communication ) ; some additional reduction operations ( to accumulate integrated values ) , and some broadcasts ( to propagate parameters ) .exclusive of file access operations , only 14 mpi functions in all are invoked .having established that all the key computational aspects of the framework are in place , and have been correctly implemented , we proceed next to consider specific applications of the framework to systems of physical interest .we use the numerical formalism developed in the preceding sections to solve numerically a model problem consisting of a self - gravitating massless scalar field in three dimensions .our starting points are ref . for a description of the vacuum problem , and ref . for the coupling of the scalar field to the gravitational metric fields .we use coordinates based upon a family of outgoing null hypersurfaces , and we let label these hypersurfaces , ( ) label the null rays and be a surface area coordinate . in the resulting coordinates ,the metric takes the bondi sachs form ^{2}- 2e^{2\beta } du dr \nonumber \\ & -&2r^{2}h_{ab}u^{b}dudx^{a}+r^{2}h_{ab}dx^{a}dx^{b},\end{aligned}\ ] ] where is related to the more usual bondi sachs variable by , and where and , with a unit sphere metric , given in terms of a complex dyad satisfying , , , with and .we also use the intermediate variable .we represent tensors on the sphere by spin - wighted variables .the conformal metric , is represented by the complex function , and by the real function , where .the metric functions are similarly encoded in the complex function .thus , it is necessary to introduce the intermediate spin - weighted variable , as well as the ( complex differential ) operators and ( see for full details ) .treating the einstein - klein - gordon model problem consistently within the leo framework requires some modifications to  , specifically to the wave equation for the scalar field ( ) which is given by eqs .( 21)-(27 ) of  .we substitute all second - order angular derivatives of the metric fields in terms of and operators acting on the additional fields , and introduced in ref . , whenever possible .a consistent treatment is obtained by introducing the additional variable where , so that the scalar field equation is also in first - order differential form in the angular variables , on a par with the approach of  for the metric equations . the bondi sachs hierarchy of hypersurface equations ,  \nonumber \\   & + &  \frac{r^2}{2k^2 }      \left [ \bar \nu \left(j_{,r } - j^2 \bar j_{,r } \right )         + \eth j \left(\bar{j}_{,r } - \bar{j}^2 j_{,r } \right )      \right ] \nonumber \\   & &  +  2 r^2 b_{,r } -4r b          + 16\pi r\phi _ { , r}\psi\ , ,    \label{eq : q } \\r^2 u_{,r }  & = & e^{2\beta } \left ( k q - j \bar q \right ) ,    \label{eq : u } \\    ( r^2 \tilde{w})_{,r }  & = & \re \bigg\ { e^{2\beta }   \left ( \frac{\cal r}{2 } -k \left ( \bar\eth b + b \bar b\right )   + \bar j \left ( \eth b + b^2 \right ) \right . \nonumber \\ & + & \left .( \nu - k ) \bar b \right )   -  1 + 2\,r \bar \eth u + \frac{r^2}{2 } \bar\eth u_{,r } \nonumber \\ & &  -  e^{-2 \beta } \frac{r^4}{4 }  \bar u_{,r } \left ( k u_{,r } + j \bar u_{,r } \right ) \bigg\ } \nonumber \\  & - & 2\pi \frac{e^{2\beta } } { r^2 } \left [     2k\bar{\psi } \psi -j \bar{\psi}^{2 }     -\bar{j } \psi^{2 }              \right]\ , ,  \label{eq : w } \\   \psi_{,r } & = & \eth\chi_{,r}\ , ,   \label{eq : psi}\end{aligned}\ ] ] now includes an additional consistency condition , eq .( [ eq : psi ] ) , and the equations for , and are modified to include the source terms as shown above .the evolution equation for the metric field is given by with the quantities , and as in eqs .( 24)(26 ) of  .the scalar field evolution equation follows from eq .( 21 ) of  , the source term is      \nonumber \\  & &  - \frac{1}{r}(u \bar{\psi } + \bar{u } \psi )  - \frac{r}{2 } \phi_{,r } ( \bar{\eth } u + \eth\bar{u } )     \nonumber \\  & &  - [ u ( \bar{\psi}_{,r}-\bar\psi ) + \bar{u } ( \psi_{,r}-\psi ) ] .\label{eq : nphi}\end{aligned}\ ] ] following  , we have used the shorthand , and eliminated the radial derivatives and using eq .( [ eq : u ] ) ,   the data required on the initial null cone are the evolution variables and .given boundary values at a fixed value of , the remaining variables ( , , , , , and ) can be determined on the initial null cone by explicit integration of the hypersurface equations ( see for details ) .the evolution equations ( [ eq : j ] ) and ( [ eq : phi ] ) can then be used to find and on the next null cone , and the process repeated to determine the spacetime to the future of the initial slice .the above system of equations describes a self gravitating scalar field . in the limit of small amplitudes , ,the scalar field can be treated as a perturbation propagating on a fixed background .this considerably simpler model is contained in the fully nonlinear case , and is implemented in our code by integrating only eqs .( [ eq : psi ] ) and ( [ eq : phi ] ) . for a scharwzschild background , the metric fields , , , , and are zero , and . the source term in eq .( [ eq : nphi ] ) reduces to , and we are left with the system for the simulations we discuss in the present work , we will be interested in solutions of the scalar field on a fixed background with definite angular dependence , as discussed in the next sub - section .   the linear equation for the scalar field on a fixed background , eq .( [ eq : l_phi ] ) is separable , i.e. its solutions can be written in the form with the coordinates in the sphere , and where each of the satisfies the one - dimensional wave equation in the plane ) where we have used the property  .( [ eq : sph ] ) is the usual equation governing the scalar perturbations of a schwarzschild black hole  , written here in characteristic coordinates .it can be put in a more familiar form by writing it in the coordinates , with , and where is the usual `` tortoise '' coordinate , . where and the potential is given by \chi , \label{eq : potential}\ ] ] and we have denoted it by to avoid confusion with bondi s which we use throughout this paper .( [ eq : qnm ] ) has been studied extensively  , its most salient feature being the existence of quasi - normal modes , whose frequencies have been tabulated , see for example  . note that the right - hand side of eq .( [ eq : sph ] ) is the correct form of the potential in coordinates .it differs by a factor of from the potential as given in eq .( [ eq : qnm ] ) , see  , because that factor is precisely the jacobian of the coordinate transformation , .  in the remainder of the present work we will use both the quasi - normal mode equation , eq .( [ eq : sph ] ) , and the linear system , eq .( [ eq : l_phi ] ) , as tests of the validity of our numerical implementation .we do this in an incremental fashion , solving eq .( [ eq : sph ] ) for fixed values of , and comparing the effectiveness of the numerical integration scheme and of our boundary conditions in reproducing the quasi - normal modes . to this end, we implement a purely radial code for eq .( [ eq : sph ] ) that employs the same numerical integration scheme that is used in the `` linear '' code ( which solves eq .( [ eq : sph ] ) and eq .( [ eq : l_phi ] ) ) , and in the full nonlinear code .using this radial code , we can isolate the effects arising from the inner - boundary treatment at by implementing eq .( [ eq : sph ] ) as indicated , in outgoing null coordinates , using both a non - compactified coordinate , with a simple extrapolative boundary condition at the outer boundary , and the compactified coordinate , where the outer boundary lies at future null infinity .the use of a non compactified coordinate allows us to isolate any effects that may arise due to the non - uniform coordinate velocity introduced by the compactified coordinate .conversely , simulations using the compactified coordinate avoid the effects of placing the outer boundary at a finite distance .we also implement the equivalent of eq .( [ eq : sph ] ) in _ ingoing _ null coordinates , , with , namely {,r } =  \nonumber \\ \left[\frac{2m}{r^3}+ \frac{\ell(\ell+1)}{r^2 } \right ] \chi .\label{eq : sph_v}\end{aligned}\ ] ] in ingoing null coordinates , the slices at penetrate the event horizon , effectively providing for an excision scheme , where evolution can be stopped at a finite number of points inside the boundary , because the behavior of the field inside the horizon does not affect the solution outside .evolutions in ingoing coordinates are carried out on a non - compactified radial grid , for which boundary data are required at a fixed value of .because of the presence of this outer boundary , simulations in ingoing coordinates can only be run for a limited time , typically , before outer boundary effects influence the signal extracted .a similar effect is seen when using outgoing , non compactified null coordinates .when using compactified coordinates , no such effects are seen , as expected .a detailed comparison between ingoing and outgoing versions of characteristic systems of equations , in compactified as well as non compactified coordinates , along with their relative advantages and disadvantages for specific applications , is worthwhile but lies outside the scope of the present work and will be reported elsewhere .we will refer only briefly to these issues in the remainder of this work .   as a useful physical indicator we calculate the balance of the scalar field energy contained between the inner boundary and null infinity .the expressions we give here are valid in the linear case , where the background metric is that of schwarzschild . for a more general approach to this issue , the linkage integrals have to be calculated , specifically the asymptotic killing vector field must be parallely propagated from null infinity  .restricted to the background case then , given a killing vector field of the metric , , we can define the conserved quantity in particular , selecting the time like killing vector , and for a surface of constant , is the energy contained on the surface , where is the volume element of the surface at constant . for a sphere at constant , represents the energy flux across the surface , with the solid angle element .the relevant components of the stress energy tensor for a massless scalar field are \nonumber \\        & - & \frac{1}{2 } e^{-2\beta } \phi_{,r } \left (           \bar u \eth\phi + u \bar\eth\phi                         \right)\ , ,  \\  t^{r}{}_{u } & = &  e^{-2\beta } \phi_{,u } \nonumber \\       & \times&\left [           \phi_{,u } -\frac{v}{r } \phi_{,r }          + \frac{1}{2 } \left ( \bar u \eth\phi + u \bar\eth\phi \right )                \right ] \ .\end{aligned}\ ] ] in the case of a linear scalar perturbation on a schwarzschild background , the energy content of a hypersurface at constant is given by d\omega \ .\label{eq : econs}\ ] ] the power radiated at time across a surface of constant , such as the inner boundary , which in our simulations we place close enough to the schwarzschild black hole , is ^2d\omega \ .\ ] ] for the flux across the inner boundary , the integral as well as the spatial and time derivatives are to be taken as evaluated at .analogously , the power radiated at time at null infinity is the limiting form ( as ) of the above expression , i.e. where we have used the behavior of the scalar field near null infinity to simplify the expression . with these definitions , the following global energy conservation law holds du ' \equiv const .\label{eq : gecons}\ ] ]  even though the expressions given above hold _ only _ in the limit in which is a killing vector of the metric , we expect them to hold in an approximate sense for our nonlinear evolutions , so we use them as a criterion for code testing .as stated previously , we use the radial code to calibrate the fully nonlinear , three - dimensional leo code in the linear regime .when computing the energy in the radial code , we make use of the property ( see ) .since the data we pose are pure spherical harmonics , the integral in the right hand side is proportional to the norm .( [ eq : angint ] ) allows us then to properly account for the contribution of the angular derivatives of the field to the energy ( [ eq : energy ] ) when using only the radial code .the integration of the hypersurface equations does not present any inherent difficulty as they are discretized at mid - point between grid points as per  .an important issue which arises because of the parallel implementation of our algorithm is that after each step in the radial march , that is , after each hypersurface equation has been advanced radially one grid point , we must synchronize the variable which has just been integrated . by thiswe mean that we communicate the ghost zone values to the processors carrying out the integration in neighboring patches . since communication is an expensive operation even on the most tightly coupled parallel computers , we take the approach of explicitly synchronizing a variable only if an ( or ) operator will be applied to the variable in question .an alternative approach would be to incorporate the synchronization into the ( and ) operators .the first approach requires more book - keeping on our part , whereas the second is more straightforward .because of the number of ( or ) operations that appear in the full nonlinear equations , however , the performance difference between these two approaches is significant .for this reason we take the first approach , reducing to the minimum possible the amount of communications , with a substantial increase in performance .the evolution equation ( [ eq : j ] ) for is treated as reported in , except that the first two radial points are subject to the boundary condition explained below .the evolution equation for the scalar field is recast in terms of the two dimensional wave operator ,  \label{eq : swe}\ ] ] where and eq .( [ eq : phi ] ) reduces then to where since all two dimensional wave operators are conformally flat , with conformal weight , we can apply to ( [ phi ] ) a flat space identity relating the values of at the four corners , , and of a null parallelogram , with sides formed by incoming and outgoing radial characteristics . in terms of , this relation leads to an integral form of the evolution equation for the scalar field the corners of the null parallelogram can not be chosen to lie exactly on radial grid points , thus the values of at the vertices of the parallelogram are approximated to second order accuracy by linear interpolations between nearest neighbor grid points on the same outgoing characteristic . approximating the integrand by its value at the center of the parallelogram ( evaluated using average values between the points and ) , we have then the evolution algorithm for the metric function follows the procedure outlined in  . as with the hypersurface equations , we synchronize the fields and , i.e. we communicate the ghost zone information from each patch to their neighbors , immediately after advancing radially these two fields with their respective evolution equations .   for the ingoing formulation ,we set the field values , and we march inwards until a few points beyond the black hole horizon ( ) . since the past light cones tilts outwards once inside the horizon , the values computed just inside the horizon can never affect those points of the grid that lie outside .this scheme provides then an extremely simple and effective form of excision , as discussed in  in the context of characteristic evolution , and in  in the context of 3 + 1 simulations in the bondi - sachs gauge .  for the outgoing formulation on a non compactified radial grid , we use simple extrapolative boundary conditions at the outermost point , i.e.the field at the last point is set equal to the value of at the point immediately before .this approximation is justified for sufficiently large as the field behaves , to leading order , as .our treatment of the inner boundary is motivated by physical considerations that arise naturally in the study of quasi - normal modes .it can be seen from eq .( [ eq : potential ] ) that when the potential goes to zero , as it does in the limits and , the solutions to eq .( [ eq : qnm ] ) are traveling waves , and . in the linear approximation then , it is consistent to apply an _ open boundary _ condition to the the scalar field based on the assumption that , at the inner boundary , the field behaves as a left - travelling wave , .it follows also that the same condition must be applied to the spin - weight 2 metric field . in the linear approximation , eq .( [ eq : j ] ) reduces to eq .( [ eq : qnm ] ) , with the potential corresponding to that of a spin weight field , see  .this open boundary condition is equivalent to stating that the fields and propagate towards the horizon along the incoming characteristics of the two - dimensional wave operator , eq .( [ eq : swe ] ) . in practice, we implement this condition for the first two points of the radial grid , and use the evolution equations for and elsewhere .  in the non - linear case ,the horizon can no longer be assumed to be static , rather it is dynamically distorted and grows as the scalar field accretes into the black hole .our boundary condition is applied always to the same set of points , which are subsequently enveloped by the growing horizon , thus any inaccuracy we might have introduced in those first two points can not have any effect on the exterior spacetime .our approach suggests the following iterative method to treat the inner boundary , in a manner which is consistent with the _ open boundary _ condition : ( 1 ) as a first approximation , solve the homogeneous equation ( [ eq : qnm ] ) for the first two radial points , i.e. assume the evolved fields propagate along incoming characteristics up to the retarded time , and ( 2 ) with the values predicted for the fields at time , correct the right - hand side of the full evolution equations .the simulations for the tests reported in the remainder of this section are conducted in compactified outgoing ( retarded ) null coordinates . to verify that the numerical algorithm is globally second - order convergent, we compute the norm of the relative residuals for three grid sizes , e.g. ^ 2 dx\,d\omega , \nonumber \\ { \mathcal q}_{mf } & = & \int [ \chi_m  - \chi_f]^2 dx\,d\omega , \label{eq : qcmf}\end{aligned}\ ] ] where the , and subscripts denote the field as computed on coarse , medium and fine grids , respectively . the field is evolved from an initial retarded time and the integrals ( [ eq : qcmf ] ) are calculated at the same final retarded time , using the same set of spatial grid points , obtained by appropriately subsampling from the fine and medium grids to the coarse grid . herewe take the angular ( and radial ) grids to be in a proportion of .grids in these ratios have a common set of points that align directly , and thus do not require interpolating cell values from the finest to the coarser grids . in this case , given the values and , it can be shown that the order of convergence of the algorithm can be read by solving for the following equation for this test we evolve the initial data with , whose the radial profile is characterized by , , and whose angular dependence is given by and , from up to .we perform three simulations , on the angular grid sizes and the corresponding radial grid sizes , for which we take time - steps , respectively . from ( [eq : order ] ) we find that the measured order of convergence is , in excellent agreement with the expected second - order convergence .it should be noted that this procedure tests the cauchy convergence of the code , providing a basic check of the consistency of the discretization . for low amplitudes ( in the perturbative regime ) , and for a given value of , the scalar field profiles computed with the fully three - dimensional code match , to within second order , the profiles obtained with a purely radial code which solves eq .( [ eq : sph ] ) , as expected .we want to stress that the boundary conditions , the initial data and the marching algorithm for the scalar field used in this numerical test are all the same as those which we have used to calibrate the radial code , the solutions of which we use here in place of an analytic solution .in fact , the convergence rate for the radial code is exactly for the radial grid sizes of , measured at with its respective subsampling , as per eqs.([eq : qcmf])-([eq : order ] ) .  for sufficiently low values of , the angular grid sizes used in the convergence test are adequate . for a given angular grid size , it is also possible to reduce the angular error by increasing the order of the angular derivatives , for example , to fourth order or higher .the increased computational expense is offset by the increased accuracy obtained ; in a parallel application there is also the potential for additional overhead because more ghost cells must be communicated . in practicewe observe that , for the smallest angular grid size considered ( ) , changing the discretization of the angular derivatives from second to fourth - order increases the execution time by about 20% .  in the work reported here , since the radial and time integrationare carried out with a scheme that is second - order convergent , we have opted to use second order accurate angular derivatives , as with this choice the nonlinear code exhibits second - order accurate cauchy convergence .  for the initial data considered here , the radial resolution must be at least to guarantee second - order convergence , as the radial features are the dominant source of numerical error .we note that if we repeat the test above using the same angular grid sizes , , but using instead radial grids with fewer points , i.e. , the measured convergence rate is lower , namely .  for the numerical simulations we present in the remainder of this article we have chosen grid sizes such that the numerical algorithm is always in the second - order convergence regime . for more details on the convergence properties of the radial evolution algorithm ,see  .   at as a function of bondi time , showing the quasi - normal mode regime oscillations for , .the solid line is the output from leo for and , when the initial data and boundary conditions are given as for the convergence test , the dashed line is the quasi - normal mode extracted from the data.,width=312,height=312 ]at as a function of bondi time . parameters and conditions are the same of fig . [fig : qnm_ell_1 ] .the solid line is the output from leo , dashed)line is the quasi - normal mode extracted the data.,width=312,height=312 ]   at as a function of bondi time , showing the quasi - normal mode regime oscillations for , .the solid line is the output from leo for and , when the initial data and boundary conditions are given as for the convergence test , except that ; the dashed line is the quasi - normal mode extracted from the data.,width=312,height=312 ]   at as a function of bondi time .parameters and conditions are the same of fig .[ fig : qnm_ell_2 ] .the solid line is the output from leo , the dashed line is the quasi - normal mode extracted from the data.,width=312,height=312 ]  the simulations reported in this and subsequent sections are all carried out in compactified , outgoing ( retarded ) null coordinates . because these coordinates allow us to read off scalar radiation patterns at null infinity .( the treatment of the inner boundary is as described in sec .[ subsec : bc ] . )the quality of the waveforms extracted depends in part on the location of the inner boundary and other factors .we describe here the method used and analyze the sources of error . for the simulations in this section we use a grid with sizes , ; the initial data corresponds to eq .( [ initial_data ] ) , with , , , and .  to extract the quasi - normal modes we have used the free software package harminv  , which employs a low - storage filter diagonalization method ( fdm ) for finding the quasi - normal modes in a given frequency interval .this software package is based on the fdm algorithm described in .the advantage of using harminv is that fdm methods provide better accuracy than what can be obtained with a fast fourier transform ( fft )  , and are more robust than least - squares fit  .we find it surprising that this approach , to our knowledge , has not been used in the context of reading quasi - normal modes in gravitational simulations .  in performing a fit with harminv to the scalar field waveforms, we find sometimes necessary to factor out , at least approximately , the exponential decay of the signal .this happens when the magnitude of the imaginary part of the frequency ( the decay rate ) is comparable to the real ( oscillatory ) part , where the fdm method fails to find a fitting frequency . in those cases, we pre - multiply the signal by an exponentially increasing function , perform the fit with harminv , and adjust the frequency obtained accordingly .when an analytic value for the frequency is available , we take its imaginary part as the value for . in general ,when the imaginary part of the frequency is not known , it suffices to use a rough estimate of the decay rate , which can obtained graphically .we also need to decide what range of values of to use to extract this information .we do this by plotting the signal and noting when the waveform is clearly periodic with an exponentially decaying envelope . for example , in fig .[ fig : qnm_ell_1_log ] , one can clearly see that this regime starts at about .we take the end of the fitting interval when the signal no longer appears to be a damped sinusoidal . for initial data of the form ( [ initial_data ] ) , with , we use harminv to extract the frequency , using as the fitting interval  ] , with the comparison between the computed signal and  shown in figs .[ fig : qnm_ell_2]-[fig : qnm_ell_2_log ] .we have observed that the relative percent error for the decay rate is larger for because it depends strongly on the value selected for the location of the boundary , .numerical experiments with the radial code confirm this and suggest that , by carefully tuning the location of the inner boundary , better accuracy can be achieved for any one value of .we have done this only partially in computing the frequency for the case .we want to emphasize that the dependence of the frequency on the boundary is not a numerical artifact of the code , but a consequence of the choice of outgoing null coordinates .this is confirmed by numerical experiments with the radial code in ingoing coordinates , in which case we find that the frequency can be read off with an error of less than for the same initial data and grid sizes .( solid line ) ; ( dotted line ) ; ( long dashed line ) .this calculation was done using the same grid parameters as for fig .[ fig : qnm_ell_2 ] except for . for each specific ( line type ; color )the descending curve corresponds to energy given by eq .( [ eq : econs ] ) .the ascending curve corresponds to the algebraic sum of and .thus , in accordance with eq .( [ eq : gecons ] ) , the horizontal curve represents the global conservation of energy.,width=312,height=312 ]   with respect to as a function of bondi time for ( circles ) , ( squares ) , and ( triangles ) .the graph shows that energy is conserved to within less than of the energy content of the initial surface.,width=312,height=312 ]   as a function of bondi time for : ( circles ) , ( squares ) , ( diamonds ) , ( triangles ) , and ( stars ) .this calculation was done using the grid parameters and .the initial data and boundary conditions are the same as in the convergence test.,width=312,height=312 ]   as a function of bondi time for : ( circles ) , ( squares ) , ( diamonds ) , ( triangles ) , and ( stars ) .this calculation was done using the same conditions of fig .[ fig : energy_iii].,width=312,height=312 ]  , as a function of bondi time for : ( circles ) , ( squares ) , ( diamonds ) , ( triangles ) , and ( stars ) . both curves for and saturate eventually without crossing for .this calculation was done using the same conditions of fig .[ fig : energy_iii ] . , width=312,height=312 ]  for initial data of the form ( [ initial_data ] ) with , fig .[ fig : energy ] shows that energy is conserved in the linear regime .it is immediately clear from the graph that the energy contained on the initial slice is larger the larger the value of . in all cases energyis clearly conserved , however , we have seen also that if the resolution is not sufficient for a given , this fact shows up clearly in the graph of energy conservation .thus , we can use energy conservation , as well as the results from running the same initial data on the radial code , to debug and calibrate the nonlinear code , as well as to estimate the evolution time needed and its computational requirements . from fig .[ fig : energy ] alone the reader might be left to guess as to the extent of the deviation of the total energy from a straight line , since that deviation is clearly so small that it does not show up in the plot for any of three simulations reported in fig .[ fig : energy ] . fig .[ fig : energy_percentage ] shows the variation in the energy balance , defined as the percentage variation in relative to the initial value , , i.e. it can be seen from fig .[ fig : energy_percentage ] that the relative change stays below during the simulation .we will revisit energy conservation in the context of large resolution simulations in sec .[ subsec : largeres ] .  fig .[ fig : energy_iii ] shows the energy content as a function of bondi time for a sequence of simulations with initial data ( [ initial_data ] ) with varying values of . for lower values of  ,the energy content decays slowly at first , then drops rather sharply , and afterwards it decays again slowly , at a much lower rate . for higher values of , , the energy decays approximately monotonically from the beginning of the simulation .we also observe , see fig . [fig : energy_out ] , that in general , increasing corresponds to an increase of the energy radiated at .the oscillations observed in the profiles are higher the higher the value of , as would be expected . the most interesting observation in the analysis of energy balance arises from fig .[ fig : energy_in ] , and is the following : for values of from 0 to 2 , the total energy flux towards the black hole ( as measured by as ) increases with the value of ; however , for values of , the total flux of energy towards the black hole diminishes with increasing values of .we have confirmed that this is the case with the radial code , so this is not a non - linear effect .it is also clear that the sudden change of energy for is due to the energy carried away by the scalar field as it falls into the black hole . at about , the radiation into the black hole saturates , and for higher values of , i.e. for , the centrifugal potential barrier prevents much of the field from falling into the black holethus , for the same amplitude , configurations with higher angular momentum ( larger values ) carry more energy , most of which will be radiated away and less of which will fall into the black hole , so in that sense these configurations are proportionally more efficient at carrying energy out to .    at for ( top ) and ( bottom ) .the parameters of the initial data are , , , , .the grid size is , .,title="fig:",width=312 ]  at for ( top ) and ( bottom ) .the parameters of the initial data are , , , , .the grid size is , .,title="fig:",width=312 ]   at for ( top ) and ( bottom ) .the parameters of the initial data and grid size are the same as in fig.[fig : phi].,title="fig:",width=312 ]  at for ( top ) and ( bottom ) .the parameters of the initial data and grid size are the same as in fig.[fig : phi].,title="fig:",width=312 ]   at for ( top ) and ( bottom ) .the parameters of the initial data and grid size are the same as in fig.[fig : phi].,title="fig:",width=312 ]  at for ( top ) and ( bottom ) .the parameters of the initial data and grid size are the same as in fig.[fig : phi].,title="fig:",width=312 ]   at for ( top ) and ( bottom ) .the parameters of the initial data and grid size are the same as in fig.[fig : phi].,title="fig:",width=312 ]  at for ( top ) and ( bottom ) .the parameters of the initial data and grid size are the same as in fig.[fig : phi].,title="fig:",width=312 ]   at for ( top ) and ( bottom ) .the parameters of the initial data and grid size are the same as in fig.[fig : phi].,title="fig:",width=312 ]  at for ( top ) and ( bottom ) .the parameters of the initial data and grid size are the same as in fig.[fig : phi].,title="fig:",width=312 ]  [fig : w ] .the solid line corresponds to the energy content at successive times , the dashed line to the sum of the energy radiated through the inner ( ) and outer ( ) boundaries , and the dot - dashed line to the sum , , respectively .the insert graph shows the percentage variation in , relative to its final value at .,width=312 ]  in order to get a first glimpse of the type of simulations that our framework enables us to perform , and to perform a final calibration check of the nonlinear code , we select initial data given by eq .( [ initial_data ] ) , with , , , , , and evolve this configuration until .this simulation is performed in compactified outgoing coordinates , with the treatment of the inner boundary as described in sec .[ subsec : bc ] .the plots shown are of quantities computed at null infinity , .the angular grid has size , that is , there are 372 points on a great circle on the sphere , while the radial grid has points .this simulation requires 27 hours on 54 processors , for a total of 1458 processor hours , or the equivalent of two months of a single - processor run .it is not by far the largest simulation we could run with our framework : we have performed scaling studies that indicate the code scales linearly well into the 4000 + processor range , but it suffices as a demonstration of the resolution that can be achieved and the typical turn around times .we assign no particular significance to the initial data selected , other than the fact that its angular complexity provides an excellent test of the code . on any large simulation ,data analysis and visualization is always a challenge . in leo, visualization is performed by having each processor write its own data set at run time , the individual data files are then post - processed , and graphs of the desired quantities generated with paraview  .paraview allows us to easily generate graphs of slices at constant coordinate lines and volumetric renderings of various fields .of particular interest to us is the behavior of the various metric quantities at null infinity .[ fig : phi]-[fig : w ] display the code variables , , , and as functions on the sphere at null infinity , at and at . in each case , the graphs show the corresponding field on the six cubed sphere caps ( the gap between the caps is the actual size of the spacing between grid cells ) .the north pole is at the top of the figure , rotated 45 degrees towards the observer .for those fields that are complex ( and which have spin different from zero ) , i.e. , , we display for ease of visualization the combinations and , which are real and have spin zero .clearly visible in fig .[ fig : phi ] is the azimuthal dependence of the field , marked by the presence of six maxima and minima .it is also apparent that oscillates in time , as the maxima and minima alternate between the top and bottom figures .the angular dependence is preserved by the evolution , as expected , as the only change between the two figures is in the overall amplitude ( by a factor of in between the two times shown ) .the graphs of , fig .[ fig : j ] , are clearly different in their angular dependence , showing the presence of various harmonics at earlier time .this can be understood since in our initial data , thus develops from , i.e. , and it is not until later times that a definite profile for has formed .the graph of , fig .[ fig : beta ] , shows precisely the angular dependence resulting from the contribution to the source term in eq .( [ eq : beta ] ) , and remains constant throughout the simulation , up to an overall amplitude .the graphs of and , figs .[ fig : u]-[fig : w ] , show higher order angular dependence arising from the angular derivatives of , which in turn are essentially driven by the source term in the eq .( [ eq : q ] ) , i.e. by .  fig .[ fig : energy_iv ] shows again that energy is conserved during the entire simulation .the variation in the energy balance is well below , thus is indistinguishable from a straight line at the resolution of the graph , as noted in sec .[ subsec : energycons ] . tomore fully appreciate to what extent energy is conserved , the graph insert in fig .[ fig : energy_iv ] shows the percentage variation of , normalized to its value at . from the graph insertwe see that the total energy varies by at most during the simulation . as we stated earlier ,energy conservation is a requisite for accuracy in the waveforms ; note that energy is conserved during this simulation to within tighter limits than in the simulations of sec .[ subsec : qnm ] , which is due primarily to the increased angular resolution . for this simulation we took in order to accurately resolve the higher order harmonic angular dependence , where in sec .[ subsec : qnm ] we set ; this amounts to ( approximately ) an 8-fold increase in resolution , and for the same radial resolution , a 64-fold increase on the computing resources required.  we would be remiss if we did not discuss at least briefly the performance characteristics of our code .as part of our calibration and testing , we have performed detailed profiling studies , which we will not go into detail here . suffice to say that in its current configuration , the code performs at approximately 20% of peak on the cray xt3 .its weak scaling is linear ( that is , its performance solving progressively larger configurations , while keeping the load per processor constant , scales linearly with the number of cores ) , while running on up to 4056 cpu cores on the cray xt3 at psc .we have presented a new computational framework ( leo ) , which we can use to perform large scale , high resolution calculations in the context of the characteristic approach in numerical relativity .this highly parallel and easily extensible implementation has been used to solve the model problem of a massless scalar field minimally coupled to gravity ( the three dimensional einstein klein gordon problem ) .we have shown that the nonlinear code thus implemented is globally second - order convergent , and how accurately we can follow quasi - normal mode ringing .we have studied the balance of energy for a number of initial data sets with different angular structure . aside from the interesting result of energy flow saturation through the schwarzschild horizon ,the leo framework offers a good prospect to study new configurations beyond the linear regime and the grid sizes used in this work .future directions we are currently exploring include the application of the leo framework to a consistent , quasilinear , fully first order formalism derived from  , and the extension of the model problem considered here to massive scalar fields .the later case is particularly important because it will allow us to simulate a boson star orbiting a black hole .we will compare the performance of the characteristic framework in ingoing versus outgoing null coordinates in the extraction of quasi - normal modes , and in the study of nonlinear effects in the neighborhood of the central black hole .the underlying framework can be applied equally well to formulations of the einstein equations in spherical coordinates , in particular to a generalization to three dimensions of the bondi sachs gauge of  , and finally to matched and characteristic evolutions  .we have not addressed in the present work some outstanding problems with the calculation of the news  , some of which arise from second angular derivatives of the metric fields at entering in the computation of the news , a feature which can lead to substantial propagation of errors .we have observed during our simulations that the metric fields computed at are smooth , as evident in figs .[ fig : phi]-[fig : w ] ( and so are those fields which represent their angular derivatives , although these are not shown here ) .it should be noted that , in our formulation  , the first angular derivatives of some fields have been promoted to auxiliary variables , for which a hypersurface equation is integrated radially .( we do this for those fields whose second angular derivatives enter in the computation of the news ) . in practice, this means that only first order angular derivatives of any of the fields we evolve need to be computed to calculate the news .it is possible that the cubed - sphere approach leads to substantial improvements in the computed waveforms , and this issue remains to be addressed .although of potential importance for the accuracy of gravitational waveforms , such a study lies outside the scope of the present work .  r.g . wishes to thank raghurama reddy for many enlightening discussions on the parallelization of the characteristic code , and acknowledges the hospitality of the universidad de los andes , mrida , venezuela , and the kavli institute for theoretical physics at santa barbara , where portions of this project were carried out .thanks the pittsburgh supercomputing center for hospitality .this work was supported in part by the national science foundation under grants no .phy0135390 to carnegie mellon university , and no .phy0244752 and phy-0555218 to duquesne university ; by fonacit under grants s198003270 and f2002000426 , and by cdcht ula under grant c12670405a .code development and simulations were carried out on the cray xt3 at the pittsburgh supercomputing center , under grants phy060004p and phy070022n .additional computer time was provided by the centro nacional de clculo cientfico , universidad de los andes ( cecalcula ) .

Abstract: we present a new computational framework ( leo ) , that enables us to carry out the very first large - scale , high - resolution computations in the context of the characteristic approach in numerical relativity . at the analytic level , our approach is based on a new implementation of the `` eth '' formalism , using a non - standard representation of the spin - raising and lowering angular operators in terms of non - conformal coordinates on the sphere ; we couple this formalism to a partially first - order reduction ( in the angular variables ) of the einstein equations . the numerical implementation of our approach supplies the basic building blocks for a highly parallel , easily extensible numerical code . we demonstrate the adaptability and excellent scaling of our numerical code by solving , within our numerical framework , for a scalar field minimally coupled to gravity ( the einstein - klein - gordon problem ) in the 3-dimensions . the nonlinear code is globally second - order convergent , and has been extensively tested using as reference a calibrated code with the same boundary initial data and radial marching algorithm . in this context , we show how accurately we can follow quasi - normal mode ringing . in the linear regime , we show energy conservation for a number of initial data sets with varying angular structure . a striking result that arises in this context is the saturation of the flow of energy through the schwarzschild radius . as a final calibration check we perform a large simulation with resolution never achieved before .