We study the BERT language representation model and the sequence generation model with BERT encoder for multi-label text classification task. We experiment with both models and explore their special qualities for this setting. We also introduce and examine experimentally a mixed model, which is an ensemble of multi-label BERT and sequence generating BERT models. Our experiments demonstrated that BERT-based models and the mixed model, in particular, outperform current baselines in several metrics achieving state-of-the-art results on three well-studied multi-label classification datasets with English texts and two private Yandex Taxi datasets with Russian texts. Multi-label text classification (MLTC) is an important natural language processing task with many applications, such as document categorization, automatic text annotation, protein function prediction (Wehrmann et al., 2018) , intent detection in dialogue systems, and tickets tagging in client support systems (Molino et al., 2018) . In this task, text samples are assigned to multiple labels from a finite label set. In recent years, it became clear that deep learning approaches can go a long way toward solving text classification tasks. However, most of the widely used approaches in MLTC tend to neglect correlation between labels. One of the promising yet fairly less studied methods to tackle this problem is using sequence-to-sequence modeling. In this approach, a model treats an input text as a sequence of tokens and predict labels in a sequential way taking into account previously predicted labels. Nam et al. (2017) used Seq2Seq architecture with GRU encoder and attention-based GRU decoder, achieving an improvement over a standard GRU model on several datasets and metrics. Yang et al. (2018b) continued this idea by introducing Sequence Generation Model (SGM) consisting of BiLSTM-based encoder and LSTM decoder coupled with additive attention mechanism . In this paper, we argue that the encoder part of SGM can be successfully replaced with a heavy language representation model such as BERT (Devlin et al., 2018) . We propose Sequence Generating BERT model (BERT+SGM) and a mixed model which is an ensemble of vanilla BERT and BERT+SGM models. We show that BERT+SGM model achieves decent results after less than a half of an epoch of training, while the standard BERT model needs to be trained for 5-6 epochs just to achieve the same accuracy and several dozens epochs more to converge. On public datasets, we obtain 0.4%, 0.8%, and 1.6% average improvement in miF 1 , maF 1 , and accuracy respectively in comparison with BERT. On datasets with hierarchically structured classes, we achieve 2.8% and 1.5% average improvement in maF 1 and accuracy. Our main contributions are as follows: 1. We present the results of BERT as an encoder in the sequence-to-sequence framework for MLTC datasets with and without a given hierarchical tree structure over classes. 2. We introduce and examine experimentally a novel mixed model for MLTC. 3. We fine-tune the vanilla BERT model to perform multi-label text classification. To the best of our knowledge, this is the first work to experiment with BERT and explore its particular properties for the multi-label setting and hierarchical text classification. 4. We demonstrate state-of-the-art results on three well-studied MLTC datasets with English texts and two private Yandex Taxi datasets with Russian texts. We present the results of the suggested models and baselines on the five considered datasets in Table  2 . First, we can see that both BERT and BERT+SGM show favorable results on multi-label classification datasets mostly outperforming other baselines by a significant margin. On RCV1-v2 dataset, it is clear that the BERT-based models perform the best in micro-F 1 metrics. The methods dealing with the class structure (tree hierarchy in HMCN and HiLAP, label frequency in BERT+SGM) also have the highest macro-F 1 score. In some cases, BERT performs better than the sequence-to-sequence version, which is especially evident on the Reuters-21578 dataset. Since BERT+SGM has more learnable parameters, a possible reason might be a fewer number of samples provided on the dataset. However, sometimes BERT+SGM might be a more preferable option: on RCV1-v2 dataset the macro-F 1 metrics of BERT + SGM is much larger while other metrics are still comparable with the BERT's results. Also, for both Yandex Taxi datasets on the Russian language, we can see that the hamming accuracy and the set accuracy of the BERT+SGM model is higher compared to other models. On Y.Taxi Riders there is also an improvement in terms of macro-F 1 metrics. In most cases, better performance can be achieved after mixing BERT and BERT+SGM. On public datasets, we see 0.4%, 0.8%, and 1.6% average improvement in miF 1 , maF 1 , and accuracy respectively in comparison with BERT. On datasets with tree hierarchy over classes, we observe 2.8% and 1.5% average improvement in maF 1 and accuracy. Metrics of interest for the mixed model depending on α on RCV1-v2 validation set are shown in Figure 4 . Visualization of feature importance for BERT and sequence generating BERT models is provided in Appendix A. In our experiments, we also found that BERT for multi-label text classification tasks takes far more epochs to converge compared to 3-4 epochs needed for multi-class datasets (Devlin et al., 2018) . For AAPD, we performed 20 epochs of training; for RCV1-v2 and Reuters-21578 -around 30 epochs; for Russian datasets -45-50 epochs. BERT + SGM achieves decent accuracy much faster than multi-label BERT and converges after 8-12 epochs. The behavior of performance of both models on the validation set of Reuters-21578 during the training process is shown in Figure 3 . Another finding of our experiments is that the beam size in the inference stage does not appear to influence much on the performance. We obtained optimal results with the beam size in the range from 5 to 9. However, a greedy approach with the beam size 1 still gives similar results with less than 1.5% difference in the metrics. A possible explanation for this might be that, while in neural machine translation (NMT) the word ordering in the output sequence matters a lot and there might be confusing options, label set generation task is much simpler and we do not have any problems with ordering. Also, due to a quite limited 'vocabulary' size |L|, we may not have as many options here to perform a beam search as in NMT or another natural sequence generation task. In this research work, we examine BERT and sequence generating BERT on the multi-label setting. We experiment with both models and explore their particular properties for this task. We also introduce and examine experimentally a mixed model which is an ensemble of vanilla BERT and sequence-to-sequence BERT models. Our experimental studies showed that BERT-based models and the mixed model, in particular, outperform current baselines by several metrics achieving state-of-the-art results on three well-studied multi-label classification datasets with English texts and two private Yandex Taxi datasets with Russian texts. We established that multi-label BERT typically needs several dozens of epochs to converge, unlike to BERT+SGM model which demonstrates decent results just after a few hundreds of iterations (less than a half of an epoch).
On using BERT as an encoder for sequential prediction of labels in multi-label text classification task
{"src": "scitldr-AIC_train", "paper_id": "BJeHFlBYvB"}