Models. For text span-pair relation extraction, we use the pre-trained BERTBASE modelhttps://github.com/huggingface/pytorch-pretrained-BERT [1]} as our backbone model to address the low-resource issue of our RE dataset generated from the limited CTA data.
On this basis, we implement model variants of injecting context information awareness (Sec. REF ) to utilize the full information in our dataset, which includes: hidden states Masking (MaskAVG and MaskMAX), Context position as Attention (C. Attn.) and Context postion as input Embedding (C. Emb.). For hidden states Masking, the different subscriptions represent different hidden state pooling methods (avg pooling and max pooling) For the two models using context position, we empirically use \(E=30\)  as the embedding size and truncate the context position sequence (Sec. REF ) by \(\pm 10\) . In addition, we experiment on the hierarchical BiLSTM model [2]} and Piecewise Convolution Neural Network [3]} as the non-pretrained baseline models in comparison. Results are aggregated from 5 runs with different initialization seeds for all experiments.
<TABLE>