We also note that the pseudo-code contains an unconventional computation of mean pseudo-statistics, τ B , for shiftto-zero. Rather than speculate over whether these issues with the original paper were simply presentational glitches or the actual basis of the experiments reported on in the paper, we present a normalized version of the two-sided bootstrap algorithm in Figure 1 , and report on the results of our own experiments in Section 4. We compare this method with approximate randomization and also paired bootstrap resampling<cite> (Koehn, 2004)</cite> , which is widely used in MT evaluation.