For each task, we fine-tune on the BERT-Base Muultilingual Cased model relesed by the authors [1]} https://github.com/google-research/bert/blob/master/multilingual.md. The model was pre-trained on Wikipedia of 104 languages (including Arabic) with 12 layer, 768 hidden units each, 12 attention heads, and has 110M parameters in entire model. The vocabulary of the model is 119,547 shared WordPices. We fine-tune the model with maximum sequence length of 50 tokens and a batch size of 32. We set the learning rate to \(2e-5\)  and train for 15 epochs. We use the same network architecture and parameters across the 3 tasks. As Table REF  shows, comparing with GRU, BERT is 3.16% better for age, 4.85% better for dialect, and 2.45% higher for gender.
