Following the original BERT setup <cite>(Devlin et al., 2018)</cite> , we format the inputs as "[CLS] TRANS/BERT  108M  12  768  768  12  6.0X  Base  TRANS-BLSTM-SMALL  152M  12  768  768  12  3.3X  TRANS-BLSTM  237M  12  768  768  12  2.5X  Large TRANS/BERT  334M  24  1024  1024  16  2.8X  TRANS-BLSTM-SMALL  487M  24  1024  1024  16  1.4X  TRANS-BLSTM  789M  24  1024  1024  16  1   Table 1 : Parameter size and training speed for TRANS/BERT, TRANS-BLSTM-SMALL, and TRANS-BLSTM on base and large settings respectively.