Since BERT-Adam <cite>[2]</cite> has excellent performance, in our experiments, we use it as an optimizer with β 1 = 0.9, β 2 = 0.999,L 2 -weight decay of 0.01.We apply a dropout trick on all layers and set the dropout probability as 0.1.