CU-UD: text-mining drug and chemical-protein interactions with ensembles of BERT-based models

Identifying the relations between chemicals and proteins is an important text mining task. BioCreative VII track 1 DrugProt task aims to promote the development and evaluation of systems that can automatically detect relations between chemical compounds/drugs and genes/proteins in PubMed abstracts. In this paper, we describe our submission, which is an ensemble system, including multiple BERT-based language models. We combine the outputs of individual models using majority voting and multilayer perceptron. Our system obtained 0.7708 in precision and 0.7770 in recall, for an F1 score of 0.7739, demonstrating the effectiveness of using ensembles of BERT-based language models for automatically detecting relations between chemicals and proteins. Our code is available at https://github.com/bionlplab/drugprot_bcvii.


I. INTRODUCTION
Relation extraction is an important task in the biomedical domain that aims to identify the semantic relationship between the biomedical entities mentioned in the text. Information about these interactions can be helpful for other biomedical research such as drug discovery, database curation and question answering systems. In recent years, there have been lots of efforts to automatically extract the relations from the biomedical articles (e.g., (1,2)). BioCreative events have focused on different relation extraction tasks over the years. One of the tasks in BioCreative VI was ChemProt (3), which aims to promote the development and evaluation of systems that can automatically detect relations between chemical compounds and genes/proteins in PubMed abstracts. This task has been continued as a DrugProt task in BioCreative VII (4). In this paper, we discuss our participation in this task, describing ensemble approaches and incorporating multiple BERT-based language representation models.

A. Dataset
Organizers of the DrugProt track created a manually annotated corpus of 5,000 abstracts from PubMed (4). The dataset was split into a training set (3,500 abstracts), a development set (750), and a test set (750) ( Table I). Following the settings of previous BioCreative tracks, the test set consists of a large collection of records containing a subset of a total of 750 Gold Standard records that was used for evaluation purposes (3). * co-corresponding

B. Preprocessing
Since all relations in the DrugProt corpus are enclosed in one sentence, our system considers interactions only occur within a sentence. We split the raw text into individual sentences by Stanza sentence splitter (5). These sentences are passed onto the BERT framework which tokenizes them using Wordpiece tokenizer (6).
Given a sentence with multiple entity mentions marked, we treated the relation extraction problem as a multi-class classification problem. Curators of the DrugProt corpus annotated thirteen key relations of biomedical importance. A sentence instance with a marked protein and a chemical mention is considered one of these thirteen DrugProt relation classes if there is a relationship between a protein and chemical occurrences. Otherwise, this instance is negative. There may be multiple interacting entities and non-interacting entities marked in the same sentence. We consider only a single chemical-protein pair in that sentence per sentence instance. Therefore, we formulate the task as predicting one of the fourteen relation types for a sentence instance.
We consider two entity tagging schemes (Table II). In the first scheme, we anonymized target named entities in a sentence using pre-defined tags to prevent overfitting to entities In the second scheme, we directly inserted the start and end entity markers into the original sentence. For example, we used the format <PROTEIN-B> human type 12 RDH <PROTEIN-E> to enclose the protein mention "human type 12 RDH" in the sentence.

C. Model Development
We hypothesize that an ensemble system that combines results of different models could lead to better predictive performance than using a single model (1). The superior performance of ensemble models is achieved when there is enough diversity among the individual models (8). In this study, we propose two different ensemble approaches which incorporate multiple BERT-based language representation models. Here, we chose BERT as the backbone because recent studies demonstrate it achieved state-of-the-art performance on a number of natural language understanding tasks, including relation extraction task (7,9,10).
We explore various architectures and pre-trained models to achieve the goal of diversifying BERT-based models. We will discuss these techniques and the ensemble algorithms in the following subsections. Due to the space limitations, we explain only the most relevant parts of the BERT model to the DrugProt task in the following. We refer the reader to the original paper for the details.

1) Single models:
We built three different architectures. The first one uses the last layer of the [CLS] token for the relation extraction task because it aggregates the information of the whole sentence. Fig. 1a demonstrates the fine-tuning process of BERT on the relation extraction task. A sentence instance is tokenized and embedded as in the original BERT paper. Inspired by the work of Su et al. (11), we also explore two other methods of summarizing the information in the last layer: Long Short Term Memory (LSTM) on the last layer (Fig. 1b) and attention mechanism on all units of the last layer (Fig. 1c).
To obtain diversity among the models used in our ensembles, we chose three BERT-based models in our approach, BioBERT (12), PubMedBERT (13), and BioM-ELECTRA  (14,15), since they differ in some pre-training aspects as well as masking, vocabulary, and the learning objectives techniques. The pre-training of BioBERT used PubMed and PMC after the initial pre-training of BERT, whereas PubMedBERT pretraining using PubMed alone started from scratch. These two models are BERT-Base with 12 bidirectional self-attention heads. To study the effectiveness of large models for this task, we also used BioM-ELECTRAL. BioM-ELECTRAL is pretrained on PubMed abstracts with no previous pre-training and is based on the ELECTRA architecture (14).
2) Class-weighted loss function: Since we have created a negative instance for each non-interacted chemical and protein in the same sentence, we have substantially more negative examples in the training dataset. To mitigate this unbalance,  we assigned class weights inversely proportional to their respective frequencies.
3) Majority voting: Majority voting is the simplest yet effective weighting method for output fusion. We select the relation type that gets the most votes out of five individual models. If the models cannot agree on the majority vote, we assign this instance as negative.

4) Meta-learning method:
We also employ another ensemble method using multilayer perceptron (MLP) as a metalearner to learn from individual classifiers. Specifically, we use MLP to combine the outputs of each individual model in an ensemble (Fig. 2). After each individual model is fine-tuned, we extract the last output layer of the [CLS] token from each model. We then use the development set that we reserve for the ensemble learning to obtain the activation vector for the [CLS] token. Afterward, we concatenate them into one vector and pass it to another MLP. We insert a softmax layer on top of the MLP to obtain final predictions.
Individual model predictions were combined using a stacking approach involving a MLP with one hidden layer having 512 units as described in Section II-C4. Run 3 involves the same Group B and uses both stacking and the majority  While participant teams were provided with the training data set and the development data set, we combined them together to develop the models. For Runs 1 and 2, we first used 70% of the total data for training, 20% for development (tuning the hyper-parameters for the individual models) and 10% to train the ensemble models. For the final models, we trained the individual models on the original training data and ensemble models on the development data with the chosen hyper-parameters.
For Run 3, we used the same approach of 70-20-10 split, but repeated this process five times (Fig 3). Thus, we obtained a group of five models corresponding to each 70-20-10 partition. Next, we trained a separate MLP for each group of models using the 10% as described above. As a result, we have five predictions for each instance corresponding to each MLP. Finally, we combined them using majority voting to obtain the final predictions.
For Run 4, we fine-tuned five BioM-ELECTRA L models using 80% of total data in each fold and then combined their predictions using majority voting.
Run 5 involved only one model, and hence 100% of the data was used for its fine-tuning.
Our models are implemented in Tensorflow 1.14 and all the experiments are carried out on NVIDIA 2080Ti GPUs. We employ AdamW optimizer with an initial learning rate of 2e-5. The maximum epoch is set as 10. Table III and IV shows the results we obtained for 5 Runs on the development set in iteration 1 (Fig 3) and the official test set, respectively. Runs 3 and 4 have the best F1 performance, which is substantially better than the average F1 across participant teams. Interestingly, F1 score of the single large model is almost the same as those of the ensembles of  Table V shows the results of individual relation types using our best system (Run 3). Our system obtained the best performance of 0.8902 in F1 for the ANTAGONIST relation, and 0.8570 for the INHIBITOR relation. As shown in Table I, INHIBITOR is the most frequent relation in the dataset. Thus its classification is probably well trained. On the other hand, ANTAGONIST is one of the least frequent relations. Further error analysis is needed to understand why our model can effectively extract it with such few cases. Compared to results of individual relation types by other Runs, we observed that Runs 3 and 4 have a uniform increase in F1 performance for almost all individual relation types. On the other hand, Run 1 has the best F1 performance on the AGONIST type with a 2.5% increase over Run 3.

IV. CONCLUSION
In this manuscript, we describe our submission in the BioCreative VII DrugProt task. The results demonstrate that our ensemble system can effectively detect the chemicalprotein relations from biomedical literature.