Mutational pattern of a sample from a critical branching population
We study a universal object for the genealogy of a sample in populations with mutations: the critical birth-death process with Poissonian mutations, conditioned on its population size at a fixed time horizon. We show how this process arises as the law of the genealogy of a sample in a large class of nearly critical branching populations with rare mutations at birth, namely populations converging, in a large population asymptotic, towards the continuum random tree. We extend this model to populations with random foundation times, with (potentially improper) prior distributions $$g_i:x\mapsto x^{-i}$$gi:x↦x-i, $$i\in \mathbb Z_+$$i∈Z+, including the so-called uniform ($$i=0$$i=0) and log-uniform ($$i=1$$i=1) priors. We first investigate the mutational patterns arising from these models, by studying the site frequency spectrum of a sample with fixed size, i.e. the number of mutations carried by k individuals in the sample. Explicit formulae for the expected frequency spectrum of a sample are provided, in the cases of a fixed foundation time, and of a uniform and log-uniform prior on the foundation time. Second, we establish the convergence in distribution, for large sample sizes, of the (suitably renormalized) tree spanned by the sample with prior $$g_i$$gi on the time of origin. We finally prove that the limiting genealogies with different priors can all be embedded in the same realization of a given Poisson point measure.