\begin{table}
\begin{tabular}{l|l} \hline \hline \multicolumn{1}{c}{**Transformation**} & \multicolumn{1}{c}{**Training example**} \\ \hline None & Cruise was born on July 3, 1962, in Syracuse, New York, to Mary \\  & Lee Pfeiffer. \\ \hline Word reversal & Preiffer Lee Mary to, York New, Syracuse in, 1962, 3 July on born \\  & was Cruise \\ \hline Entity-preserving reversal & Mary Lee Pfeiffer to, Syracuse, New York in, 1962, 3 July on born \\  & was Cruise \\ \hline Random segment reversal & [REV] York, to Mary Lee Pfeiffer. [REV] in Syracuse, New [REV] \\  & on July 3, 1962, [REV] born [REV] Cruise was \\ \hline \hline \end{tabular}
\end{table}
Table 2: **Reversal transformations: examples of different reversal types on a given string. In practice, training examples can be much longer (e.g., entire documents during pre-training). The language model is still trained left-to-right on such transformations, and in the word reversal case is essentially predicting the sentence backwards (right-to-left) starting from the last word. The entities which have their word ordering preserved are highlighted by underlines. In random segment reversal, the segments are separated by “[REV]”. _Reverse training_ on both the standard (”None” transformation) and reversed examples, hence doubling the amount of training tokens.