But the problem here is not publication bias per se . Any experimental activity ( including but not limited to publication ) that involves preferentially following up on one 's most statistically promising initial results presents the same challenge . For example , a standard study design in genetics is to scan the genome , one position at a time , for statistical evidence of a genetic variant with an effect on some phenotype in one or more families , in order to find the best - supported genomic position ; and then to follow up with additional data , e.g. , using a new set of families , in order to corroborate the result at that position . This procedure seems scientifically unassailable , but any attempt at evidence amalgamation across the two stages of the study violates the same statistical assumption as does publication bias . When following up on our most promising findings , we can expect the p value to regress to the mean regardless of whether the evidence is going up or down , by virtue of having selected a location for follow - up on the basis of a notably small p value .