Without knowing the experimental population in advance (i.e. we did not know which of the users we invited would actually participate in the experiment), we were unable to use more sophisticated experimental designs such as completely-randomized or block-randomized assignment. This is a common challenge when running randomized experiments online [1]}.
