A bootstrap pseudo-sample consists of the translations by the two systems (X b , Y b ) of a bootstrapped test set<cite> (Koehn, 2004)</cite> , constructed by sampling with replacement from the original test set translations.