meta - analysis refers to methods focused on contrasting and combining results from different studies , in the hope of identifying patterns among study results , sources of disagreement among those results , or other interesting relationships that may come to light in the context of multiple studies . in its simplest form , this is normally done by identification of a common measure of effect size , of which a weighted average might be the output of a meta - analysis . the weighting might be related to sample sizes within the individual studies [ 2 , 3 ] . more generally there are other differences between the studies that need to be allowed for , but the general aim of a meta - analysis is to more powerfully estimate the true effect size as opposed to a less precise effect size derived in a single study under a given single set of assumptions and conditions . for reviews on meta - analysis models , these include odds ratios and relative risks ; standardized mean difference , cohen 's d , hedges ' g , and glass 's  ; correlation coefficient and relative metrics ; sensitivity and specificity from diagnostic accuracy studies ; and p - values . for more comprehensive reviews publication bias is a threat to any research that attempts to use the published literature , and its potential presence is perhaps the greatest threat to the validity of a meta - analysis . publication bias exists because research with statistically significant results is more likely to be submitted and published than work with null or nonsignificant results . this issue was memorably termed as the file - drawer problem by rosenthal ; nonsignificant results are stored in file drawers without ever being published . in addition to publication bias , other related types of bias exist including pipeline bias , subjective reporting bias , duplicate reporting bias , or language bias ( see [ 1215 ] for definitions and examples ) . the implication of these various types of bias is that combining only the identified published studies uncritically may lead to an incorrect , usually over optimistic , conclusion [ 10 , 16 ] . the ability to detect publication bias in a given field is a key strength of meta - analysis because identification of publication bias will challenge the validity of common views in that area and will spur further investigations . there are two types of statistical procedures for dealing with publication bias in meta - analysis : methods for identifying the existence of publication bias and methods for assessing the impact of publications bias . the first includes the funnel plot ( and other visualisation methods such as the normal quantile plot ) and regression / correlation - based tests , while the second includes the fail - safe ( also called file - drawer ) number , the trim and fill method , and selection model approaches [ 10 , 14 , 18 ] . this assumes that the results from smaller studies will be more widely spread around the mean effect because of larger random error . the next most frequent method used to assess publication bias is rosenthal 's fail - safe number . two recent reviews examining the assessment of publication bias in psychology and ecology reported that funnel plots were the most frequently used ( 24% and 40% resp . ) , followed by rosenthal 's fail - safe number ( 22% and 30% , resp . ) . assessing publication bias can be performed by trying to estimate the number of unpublished studies in the given area a meta - analysis is studying . the fail - safe number represents the number of studies required to refute significant meta - analytic means . although apparently intuitive , it is in reality difficult to interpret not only because the number of data points ( i.e. , sample size ) for each of k studies is not defined , but also because no benchmarks regarding the fail - safe number exist , unlike cohen 's benchmarks for effect size statistics . however , these versions have been heavily criticised , mainly because such numbers are often misused and misinterpreted . the main reason for the criticism is that , depending on which method is used to estimate the fail - safe n , the number of studies can greatly vary . although rosenthal 's fail - safe number of publication bias was proposed as early as 1979 and is frequently cited in the literature   ( over 2000 citations ) , little attention has been given to the statistical properties of this estimator . this is the aim of the present paper , which is discussed in further detail in section 3 . his concern was that some statistically nonsignificant studies may be missing from an analysis ( i.e. , placed in a file drawer ) and that these studies , if included , would nullify the observed effect . by nullify , he meant to reduce the effect to a level not statistically significantly different from zero . rosenthal suggested that rather than speculate on whether the file drawer problem existed , the actual number of studies that would be required to nullify the effect could be calculated . this method calculates the significance of multiple studies by calculating the significance of the mean of the standard normal deviates of each study . rosenthal 's method calculates the number of additional studies nr , with the mean null result necessary to reduce the combined significance to a desired  level ( usually 0.05 ) . the necessary prerequisite is that each study examines a directional null hypothesis such that the effect sizes i from each study are examined under i  0 or ( i  0 ) . the test statistic for this is 
 ( 2)zs=i=1kzik , 
 with zi = i / si , where si are the standard errors of i . under the null hypothesis so we get that the number of additional studies nr , with mean null result necessary to reduce the combined significance to a desired  level ( usually 0.05 [ 7 , 11 ] ) , is found after solving 
 ( 3)z=i=1kzinr+k . so , nr is calculated as 
 ( 4)nr=i=1kzi2z2k , 
 where k is the number of studies and z is the one - tailed z score associated with the desired  level of significance . rosenthal further suggested that if nr > 5k + 10 , the likelihood of publication bias would be minimal . cooper   and cooper and rosenthal   called this number the fail - safe sample size or fail - safe number . if this number is large , one might be more confident that the effect , although possibly inflated by the exclusion of some studies , is , nevertheless , not zero . first , it assumes that the association in the hidden studies is zero , rather than considering the possibility that some of the studies could have an effect in the reverse direction or an effect that is small but not zero . therefore , the number of studies required to nullify the effect may be different than the fail - safe number , either larger or smaller . second , this approach focuses on statistical significance rather than practical or substantive significance ( effect sizes ) . that is , it may allow one to assert that the mean correlation is not zero , but it does not provide an estimate of what the correlation might be ( how it has changed in size ) after the missing studies are included [ 23 , 3234 ] . however , for many fields it remains the gold standard to assess publication bias , since its presentation is conceptually simple and eloquent . iyengar and greenhouse   proposed an alternative formula for rosenthal 's fail - safe number , in which the sum of the unpublished studies ' standard variates is not zero . in this case the number of unpublished studies n is approached through the following equation : 
 ( 5)z=i=1kzi+nmn+k , 
 where m( ) = (z)/(z ) ( this results immediately from the definition of truncated normal distribution ) and  is the desired level of significance . this is justified by the author that the unpublished studies follow a truncated normal distribution with x  z.  ( ) and ( ) denote the cumulative distribution function ( cdf ) and probability distribution function ( pdf ) , respectively , of a standard normal distribution . there are certain other fail - safe numbers which have been described , but their explanation goes beyond the scope of the present article . duval and tweedie [ 36 , 37 ] present three different estimators for the number of missing studies and the method to calculate this has been named trim and fill method . orwin 's   approach is very similar to rosenthal 's   approach without considering the normal variates but taking cohen 's d   to compute a fail - safe number . rosenberg 's fail - safe number is very similar to rosenthal 's and orwin 's fail - safe number . its difference is that it takes into account the meta - analytic estimate under investigation by incorporating individual weights per study . gleser and olkin   proposed a model under which the number of unpublished studies in a field where a meta - analysis is undertaken could be estimated . the maximum likelihood estimator of their fail - safe number only needs the number of studies and the maximum p value of the studies . finally , the eberly and casella fail - safe number assumes a bayesian methodology which aims to estimate the number of unpublished studies in a certain field where a meta - analysis is undertaken . the aim of the present paper is to study the statistical properties of rosenthal 's   fail - safe number . in the next section we introduce the statistical theory for computing confidence intervals for rosenthal 's   fail - safe number . we initially compute the probability distribution function of n^r , which gives formulas for variance and expectation ; next , we suggest distributional assumptions for the standard normal variates used in rosenthal 's fail - safe number and finally suggest confidence intervals . the estimator n^r of unpublished studies is approached through rosenthal 's formula : 
 ( 6)n^r=i=1kzi2z2k . let zi , i = 1,2 ,  , i ,  , k , be i.i.d . random variables with e[zi ] =  and var[zi ] = . we discern two cases:(a)k  is fixed or(b)k is random , assuming additionally that k ~ pois( ) . this is reasonable since the number of studies included in a meta - analysis is like observing counts . other distributions might be assumed , such as the gamma distribution , but this would require more information or assumptions to compute the parameters of the distribution.in both cases , estimators of  ,  , and  can be calculated without distributional assumptions for the zi with the method of moments or with distributional assumptions regarding the zi . this is reasonable since the number of studies included in a meta - analysis is like observing counts . other distributions might be assumed , such as the gamma distribution , but this would require more information or assumptions to compute the parameters of the distribution . z2 ,  , zi ,  , zk in the formula of the estimator n^r  ( 6 ) are i.i.d . distributed with e[zi ] =  and var[zi ] = . let s = i=1zi and according to the lindeberg - lvy central limit theorem , we have 
 ( 7)kskdn0,2sdnk,k2 . z2 ,  , zi ,  , zk in the formula of the estimator n^r  ( 6 ) are i.i.d . distributed with e[zi ] =  and var[zi ] = . let s = i=1zi and according to the lindeberg - lvy central limit theorem , we have 
 ( 7)kskdn0,2sdnk,k2 . the pdf of rosenthal 's n^r can be retrieved from a truncated version of ( 8) . from ( 3 ) , we get that 
 ( 9)s = zn^r+k . we advocate that rosenthal 's equations ( 3 ) and ( 9 ) implicitly impose two conditions which must be taken into account when we seek to estimate the distribution of nr : 
 ( 10)s0,(11)n^r0 . 
 expression ( 10 ) is justified by the fact that the right hand side of  ( 9 ) is positive , so then s  0 . expression ( 11 ) is justified by the fact that nr expresses the number of studies , so it must be at least 0 . hence , expressions ( 10 ) and ( 11 ) are satisfied when s is a truncated normal random variable.the truncated normal distribution is a probability distribution related to the normal distribution . given a normally distributed random variable x with mean t and variance t , a  b  . then x conditional on a < x < b has a truncated normal distribution with pdf : fx(x ) = ( 1/)((x  t)/t)/(((b  t)/t ) ((a    t)/t ) ) , for a  x  b and fx(x ) = 0 otherwise .let it be s , such that szk . so the pdf of s then becomes 
 ( 12)fss=12k2expsk22k2,hhhhhhhhhhhhhhhihhhhhszk , 
 where =(k-z)/. 
  the pdf of rosenthal 's n^r can be retrieved from a truncated version of ( 8) . from ( 3 ) , we get that 
 ( 9)s = zn^r+k . we advocate that rosenthal 's equations ( 3 ) and ( 9 ) implicitly impose two conditions which must be taken into account when we seek to estimate the distribution of nr : 
 ( 10)s0,(11)n^r0 . expression ( 10 ) is justified by the fact that the right hand side of  ( 9 ) is positive , so then s  0 . expression ( 11 ) is justified by the fact that nr expresses the number of studies , so it must be at least 0 . hence , expressions ( 10 ) and ( 11 ) are satisfied when s is a truncated normal random variable . given a normally distributed random variable x with mean t and variance t , let it be that x  ( a , b ) ,   a  b  . then x conditional on a < x < b has a truncated normal distribution with pdf : fx(x ) = ( 1/)((x  t)/t)/(((b  t)/t ) ((a    t)/t ) ) , for a  x  b and fx(x ) = 0 otherwise . so the pdf of s then becomes 
 ( 12)fss=12k2expsk22k2,hhhhhhhhhhhhhhhihhhhhszk , 
 where =(k-z)/. then , we have 
 ( 13)fn^rnr = fssdsdnr9,12fn^rnr = z22k2nr+k  expznr+kk22k2 ,  nr0 . 
 the characteristic function is 
 ( 14)n^rt = eexpitnr=1+/1  zexpk22it/(z22k2it)kitz22k2it1/2 , 
 where i=-1 , 1=2kkit/(z2 - 2k2it ) , 1 = z/(z  2kit ) . from ( 14 ) we get 
 ( 15)en^r = k22+k2z2k+ , 
 where =(/)(k(k+z)/z2 ) . also , 
 ( 16)varn^r=2k222k2+2z4+ , 
 where 
 ( 17)=k235k+za2z4  hhhhhhh+k3/22k+za2z4 . proofs for expressions ( 14 ) , ( 15 ) , and ( 16 ) are given in the appendix . also we get 
 ( 19)en^r = k22+k2z2k,(20)varn^r=2k222k2+2z4.(2)a limiting element of this computation is that n^r takes discrete values because it describes number of studies , but it has been described by a continuous distribution . for a significantly large k a limiting element of this computation is that n^r takes discrete values because it describes number of studies , but it has been described by a continuous distribution . so taking into account the result from the distribution of n^r for a fixed k we get that the joint distribution of k and n^r is 
 ( 21)fn^r , nnr , k = fn^rnr    k = kpk = kfn^r , nnr , k = z22k2nr+k  expznr+kk22k2kk!,hhhhhhhhhhhhhhhnr0 ,  k=0,1,2,  . ( a)when k is fixed , expressions ( 19 ) and ( 20 ) denote the expectation and variance , respectively , for n^r . this is derived from the pdf of n^r ; an additional proof without reference to the pdf is given in the appendix.(b)when k is random with k ~ pois( ) , the expectation and variance of n^r are 
 ( 22)en^r=22+2+2z2,varn^r=43 + 62+4  + 43 + 162 + 622  + 22 + 34  z41222+2+2z2+.proofs are given in the appendix . when k is fixed , expressions ( 19 ) and ( 20 ) denote the expectation and variance , respectively , for n^r . this is derived from the pdf of n^r ; an additional proof without reference to the pdf is given in the appendix . when k is random with k ~ pois( ) , the expectation and variance of n^r are 
 ( 22)en^r=22+2+2z2,varn^r=43 + 62+4  + 43 + 162 + 622  + 22 + 34  z41222+2+2z2+. having now computed a formula for the variance which is necessary for a confidence interval , we need to estimate  ,  , and . in both cases , estimators of  ,  , and  can be calculated without distributional assumptions for the zi with the method of moments or with distributional assumptions regarding the zi . if we suppose that the zi follow a distribution , we would replace the values of  and  with their distributional values . the zi follow a standard normal distribution ; that is , zi ~ n(0,1 ) . although the origin of the zi is from the standard normal distribution , the studies in a meta - analysis are a selected sample of published studies . for this reason , the next distribution is suggested as better .  
 half normal distribution . here we propose that the zi follow a half normal distribution hn(0,1 ) , which is a special case of folded normal distribution . before we explain the rational of this distribution , a definition of this type of distribution is provided . a half normal distribution is also a special case of a truncated normal distribution . given a normally distributed random variable x with mean f and variance f , the random variable y = |x| has a folded normal distribution [ 43 , 45 , 46 ] . given a normally distributed random variable x with mean f and variance f , the random variable y = |x| has a folded normal distribution [ 43 , 45 , 46 ] . the folded normal distribution has the following properties:(a)probability density function ( pdf ) : 
 ( 26)fyy=1f2expyf22f2  + 1f2expyf22f2,hhhhhhllhhhhhhhhhhhfor  y0,(b)(27)ey=f2expf22f2+f12ff , vary  = f2+f2  f2expf22f2+f12ff2 . the folded normal distribution has the following properties:(a)probability density function ( pdf ) : 
 ( 26)fyy=1f2expyf22f2  + 1f2expyf22f2,hhhhhhllhhhhhhhhhhhfor  y0,(b)(27)ey=f2expf22f2+f12ff , vary  = f2+f2  f2expf22f2+f12ff2 . probability density function ( pdf ) : 
 ( 26)fyy=1f2expyf22f2  + 1f2expyf22f2,hhhhhhllhhhhhhhhhhhfor  y0 , ( 27)ey=f2expf22f2+f12ff , vary  = f2+f2  f2expf22f2+f12ff2 this distribution is identical to the truncated normal distribution , with left truncation point 0 and no right truncation point . for this distribution we have the following.(a)fyy=2/fexp  ( -y2/2f2 ) , for y  0.(b)ey=f2/ , var[y ] = f(1  2/ ) . 
  when f = 0 , the distribution of y is a half normal distribution . this distribution is identical to the truncated normal distribution , with left truncation point 0 and no right truncation point . for this distribution we have the following.(a)fyy=2/fexp  ( -y2/2f2 ) , for y  0.(b)ey=f2/ , var[y ] = f(1  2/ ) . the zi in rosenthal 's estimator nr are derived from a half normal distribution , based on a normal distribution n(0,1 ) . the zi in rosenthal 's estimator nr are derived from a half normal distribution , based on a normal distribution n(0,1 ) . when a researcher begins to perform a meta - analysis , the sample of studies is drawn from those studies that are already published . so his sample is most likely biased by some sort of selection bias , produced via a specific selection process . thus , although when we study rosenthal 's nr assuming that all zi are drawn from the normal distribution , they are in essence drawn from a truncated normal distribution . this has been commented on by iyengar and greenhouse   and schonemann and scargle . we would like to advocate that the half normal distribution , based on a normal distribution n(0,1 ) , is the one best representing the zi rosenthal uses to compute his fail - safe nr . the reasons for this are as follows.(1)firstly , assuming that all zi are of the same sign does not impede the significance of the results from each study . z1/2 occurs.(2)however , when a researcher begins to perform a meta - analysis of studies , many times zi can be either positive or negative . although this is true , when the researcher is interested in doing a meta - analysis , usually the zi that have been published are indicative of a significant effect of the same direction ( thus zi have the same sign ) or are at least indicative of such an association without being statistically significant , thus producing zi of the same sign but not producing significance ( e.g. , the confidence interval of the effect might include the null value).(3)there will definitely be studies that produce a totally opposite effect , thus producing an effect of opposite direction , but these will definitely be a minority of the studies . also there is the case that these other signed zi are not significant.hence , in this case 
 ( 28)^=k ,  =2 ,  2=12. firstly , assuming that all zi are of the same sign does not impede the significance of the results from each study . that is , the test is significant when either zi > z/2 or zi < z1/2 occurs . however , when a researcher begins to perform a meta - analysis of studies , many times zi can be either positive or negative . although this is true , when the researcher is interested in doing a meta - analysis , usually the zi that have been published are indicative of a significant effect of the same direction ( thus zi have the same sign ) or are at least indicative of such an association without being statistically significant , thus producing zi of the same sign but not producing significance ( e.g. , the confidence interval of the effect might include the null value ) . there will definitely be studies that produce a totally opposite effect , thus producing an effect of opposite direction , but these will definitely be a minority of the studies . also there is the case that these other signed zi are not significant . we propose that the zi follow a skew normal distribution ; that is , zi ~ sn( ,  ,  ) . the skew normal distribution is a continuous probability distribution that generalises the normal distribution to allow for nonzero skewness . a random variable x follows a univariate skew normal distribution with location parameter   r , scale parameter   r , and skewness parameter   r , if it has the density 
 ( 29)fxx=2xx  xr . note that if  = 0 , the density of x reduces to the n( ,  ) . the skew normal distribution is a continuous probability distribution that generalises the normal distribution to allow for nonzero skewness . a random variable x follows a univariate skew normal distribution with location parameter   r , scale parameter   r , and skewness parameter   r , if it has the density 
 ( 29)fxx=2xx  xr . note that if  = 0 , the density of x reduces to the n( ,  ) . remark 1 . the expectation and variance of x are 
 ( 30)ex=+2 ,  where  =1+2,varx=2122. 
  the expectation and variance of x are 
 ( 30)ex=+2 ,  where  =1+2,varx=2122. 
 remark 2 . the methods of moments estimators for  ,  , and  are [ 50 , 51 ] 
 ( 31)~=m1a1m3b11/3 ,  ~2=m2a12m3b12/3,~=a12+m2b1m32/31/2 , 
 where a1=2/ , b1 = ( 4/  1)a1 , m1 = ni=1xi , m2 = ni=1(xi  m1 ) , and m3 = ni=1(xi  m1 ) . the sign of  ~ is taken to be the sign of m3 . the methods of moments estimators for  ,  , and  are [ 50 , 51 ] 
 ( 31)~=m1a1m3b11/3 ,  ~2=m2a12m3b12/3,~=a12+m2b1m32/31/2 , 
 where a1=2/ , b1 = ( 4/  1)a1 , m1 = ni=1xi , m2 = ni=1(xi  m1 ) , and m3 = ni=1(xi  m1 ) . the sign of  ~ is taken to be the sign of m3 . the skew normal distribution allows for a dynamic way to fit the available z - scores . the fact that there is ambiguity on the derivation of the standard deviates from each study from a normal or a truncated normal distribution creates the possibility that the distribution could be a skew normal , with the skewness being attributed that we are including only the published z - scores in the estimation of rosenthal 's   estimator . hence , in this case and taking the method of moments estimators of  ,  , and  , we get 
 ( 32)^=n ,  ^=~+~~2 ,  ^2=~212~2 , 
 where a1=2/ , b1 = ( 4/  1)a1 , m1 = ni=1zi , m2 = ni=1(zi  m1 ) , and m3 = ni=1(zi  m1 ) . in the previous section , we compute asymptotic ( 1  /2)% confidence intervals for nr as 
 ( 33)n^rlow , n^rup  = n^rz1/2var^n^r  , n^r+z1/2var^n^r , 
 where z1/2 is the ( 1  /2)th quantile of the standard normal distribution . the variance of n^r for a given set of values zi depends firstly on whether the number of studies k is fixed or random and secondly on whether the estimators of  ,  , and  are derived from the method of moments or from the distributional assumptions . bootstrap is a well - known resampling methodology for obtaining nonparametric confidence intervals of a parameter [ 52 , 53 ] . in most statistical problems one needs an estimator of a parameter of interest as well as some assessment of its variability . in many such problems the estimators are complicated functionals of the empirical distribution function and it is difficult to derive trustworthy analytical variance estimates for them . the primary objective of this technique is to estimate the sampling distribution of a statistic . essentially , bootstrap is a method that mimics the process of sampling from a population , like one does in monte carlo simulations , but instead drawing samples from the observed sampling data . this process is explained properly by efron and tibshirani   and davison and hinkley , who also noted that bootstrap confidence intervals are approximate , yet better than the standard ones . nevertheless , they do not try to replace the theoretical ones and bootstrap is not a substitute for precise parametric results , but rather a way to reasonably proceed when such results are unavailable . nonparametric resampling makes no assumptions concerning the distribution of , or model for , the data . our data is assumed to be a vector zobs of k independent observations , and we are interested in a confidence interval for ^zobs . the general algorithm for a nonparametric bootstrap is as follows.(1)sample k observations randomly with replacement from zobs to obtain a bootstrap data set , denoted by z.(2)calculate the bootstrap version of the statistic of interest ^=^z.(3)repeat steps ( 1 ) and ( 2 ) several times , say b , to obtain an estimate of the bootstrap distribution.in our case(1)compute a random sample from the initial sample of zi , size k,(2)compute nr from this sample,(3)repeat these processes b times.then the bootstrap estimator of nr is 
 ( 34)nr_bootstrap=nrb . sample k observations randomly with replacement from zobs to obtain a bootstrap data set , denoted by z. calculate the bootstrap version of the statistic of interest ^=^z. repeat steps ( 1 ) and ( 2 ) several times , say b , to obtain an estimate of the bootstrap distribution . compute a random sample from the initial sample of zi , size k , compute nr from this sample , repeat these processes b times . in the next section , the method for simulations is as follows.(1)initially we draw random numbers from the following distributions:(a)standard normal distribution,(b)half normal distribution ( 0,1),(c)skew normal distribution with negative skewness sn( = 0.5 ,  = 0 ,  = 1),(d)skew normal distribution with positive skewness sn( = 0.5 ,  = 0 ,  = 1).(2)the numbers drawn from each distribution represent the number of studies in a meta - analysis and we have chosen k = 5 , 15 , 30 , and 50 . when k is assumed to be random , then the parameter  is equal to the values chosen for the simulation , that is , 5 , 15 , 30 , and 50 , respectively.(3)we compute the normal approximation confidence interval with the formulas described in section 3 and the bootstrap confidence interval . we also discern whether the number of studies is fixed or random . for the computation of the bootstrap confidence interval , we also study the performance of the different distributional estimators in cases where the distributional assumption is not met , thus comparing each of the six confidence interval estimators under all four distributions.(4)we compute the coverage probability comparing with the true value of rosenthal 's fail - safe number . when the number of studies is fixed the true value of rosenthal 's number is 
 ( 35)en^r = k22+k2z2k . 
 when the number of studies is random [ from a poisson (  ) distribution ] the true value of rosenthal 's number is 
 ( 36)en^r=22+2+2z2. 
 we execute the above procedure 10,000 times each time . initially we draw random numbers from the following distributions : ( a)standard normal distribution,(b)half normal distribution ( 0,1),(c)skew normal distribution with negative skewness sn( = 0.5 ,  = 0 ,  = 1),(d)skew normal distribution with positive skewness sn( = 0.5 ,  = 0 ,  = 1 ) . standard normal distribution , half normal distribution ( 0,1 ) , skew normal distribution with negative skewness sn( = 0.5 ,  = 0 ,  = 1 ) , skew normal distribution with positive skewness sn( = 0.5 ,  = 0 ,  = 1 ) . the numbers drawn from each distribution represent the number of studies in a meta - analysis and we have chosen k = 5 , 15 , 30 , and 50 . when k is assumed to be random , then the parameter  is equal to the values chosen for the simulation , that is , 5 , 15 , 30 , and 50 , respectively . we compute the normal approximation confidence interval with the formulas described in section 3 and the bootstrap confidence interval . we also discern whether the number of studies is fixed or random . for the computation of the bootstrap confidence interval , we also study the performance of the different distributional estimators in cases where the distributional assumption is not met , thus comparing each of the six confidence interval estimators under all four distributions . we compute the coverage probability comparing with the true value of rosenthal 's fail - safe number . when the number of studies is fixed the true value of rosenthal 's number is 
 ( 35)en^r = k22+k2z2k . 
 when the number of studies is random [ from a poisson (  ) distribution ] the true value of rosenthal 's number is 
 ( 36)en^r=22+2+2z2. 
 we execute the above procedure 10,000 times each time . all simulations were performed in r and the code is shown in the supplementary materials ( see supplementary materials available online at http://dx.doi.org/10.1155/2014/825383 ) . we observe from table 2 and figure 1 that the bootstrap confidence intervals perform the poorest both when the number of studies is considered fixed or random . the only case in which they perform acceptably is when the distribution is half normal and the number of studies is fixed . the moment estimators of variance perform either poorly or too efficiently in all cases , with coverages being under 90% or near 100% . the most acceptable confidence intervals for rosenthal 's estimator appear to be in the distribution - based method and to be much better for a fixed number of studies than for random number of studies . we also observe that , for the distribution - based confidence intervals in the fixed category , the half normal distribution hn(0,1 ) produces coverages which are all 95% . when the distributional assumption is not met the coverage is poor except for the cases of the positive and negative skewness skew normal distributions which perform similarly , possibly due to symmetry . in the next sections , we give certain examples and we present the lower limits of confidence intervals for testing whether nr > 5k + 10 , according to the suggested rule of thumb by . we choose only the variance from a fixed number of studies when the zi are drawn from a half normal distribution hn(0,1 ) . in this section , we present two examples of meta - analyses from the literature . the first study is a meta - analysis of the effect of probiotics for preventing antibiotic - associated diarrhoea and included 63 studies . the second meta - analysis comes from the psychological literature and is a meta - analysis examining reward , cooperation , and punishment , including analysis of 148 effect sizes . for each meta - analysis , we computed rosenthal 's fail - safe number and the respective confidence interval with the methods described above ( table 3 ) . we observe that both fail - safe numbers exceed rosenthal 's rule of thumb , but some lower confidence intervals , especially in the first example , go as low as 369 , which only slightly surpasses the rule of thumb ( 563 + 10 = 325 in this case ) . hence the confidence interval , especially the lower confidence interval value , is important to establish whether the fail - safe number surpasses the rule of thumb . in the next section , we present a table with values according to which future researchers can get advice on whether their value truly supersedes the rule of thumb . we wish to answer the question whether nr > 5k + 10 for a given level of significance and the estimator n^r , which is the rule of thumb suggested by rosenthal . we formulate a hypothesis test according to which 
 ( 37)h0:nr5k+10h1:nr>5k+10 . 
 an asymptotic test statistic for this so we reject the null hypothesis if  ( n^r-5k-10)/varn^r > zn^r > zvarn^r+5k+10 . in table 4 we give the limits of nr above which we are 95% confident that nr > 5k + 10 . for example if a researcher performs a meta - analysis of 25 studies , the rule of thumb suggests that over 5  25 + 10 = 135 studies there is no publication bias . the present approach and the values of table 4 suggest that we are 95% confident for this when nr exceeds 209 studies . so this approach allows for inferences about rosenthal 's n^r and is also slightly more conservative especially when rosenthal 's fail - safe number is characterised from overestimating the number of published studies . the purpose of the present paper was to assess the efficacy of confidence intervals for rosenthal 's fail - safe number . we initially defined publication bias and described an overview of the available literature on fail - safe calculations in meta - analysis . although rosenthal 's estimator is highly used by researchers , its properties and usefulness have been questioned [ 48 , 58 ] . the original contributions of the present paper are its theoretical and empirical results . first , we developed statistical theory allowing us to produce confidence intervals for rosenthal 's fail - safe number . this was produced by discerning whether the number of studies analysed in a meta - analysis is fixed or random . each case produces different variance estimators . for a given number of studies and a given distribution , we provided five variance estimators : moment- and distribution - based estimators based on whether the number of studies is fixed or random and on bootstrap confidence intervals secondly , we examined four distributions by which we can simulate and test our hypotheses of variance , namely , standard normal distribution , half normal distribution , a positive skew normal distribution , and a negative skew normal distribution . these four distributions were chosen as closest to the nature of the zis . the half normal distribution variance estimator appears to present the best coverage for the confidence intervals . hence , this might support the hypothesis that zis are derived from a half normal distribution . the limitations of the study initially stem from the flaws associated with rosenthal 's estimator . this usually means that the number of negative studies needed to disprove the result is highly overestimated . we could try more values for the skew normal distribution , for which we tried only two values in present paper . the implications of this research for applied researchers in psychology , medicine , and social sciences , which are the fields that predominantly use rosenthal 's fail - safe number , are immediate . table 4 provides an accessible reference for researchers to consult and apply this more conservative rule for rosenthal 's number . secondly , the formulas for the variance estimator are all available to researchers , so they can compute normal approximation confidence intervals on their own . the future step that needs to be attempted is to develop an r - package program or a stata program to execute this quickly and efficiently and make it available to the public domain . the present study is the first in the literature to study the statistical properties of rosenthal 's fail - safe number . statistical theory and simulations were presented and tables for applied researchers were also provided . despite the limitations of rosenthal 's fail - safe number , it can be a trustworthy way to assess publication bias , especially under the more conservative nature of the present paper .