During the pre-training stage, [1]} train two separate models—forward and backward—on the 1-billion word corpus [2]}. The pre-training task is a standard character level language modelling task that predicts the next character given a sequence of characters. Taking the backward model illustrated in Figure REF  as an example, the output state of the 12-nd character is taken as input to a classifier to predict the next character, whose ground truth in this example is the character `n'. Once the pre-training finishes, the pre-trained models are frozen and used as part of the mapping function for downstream supervised task (Figure REF ).
<FIGURE>