As described in Section , in addition to the BERT model, we constructed a LSTM model as one of our baselines, considering its ability to account for long-distance relations between words. In this model, firstly, an embedding layer initialized with fastText’s Common Crawl 300-dimensional embeddings [1]}. We experimented with GloVe 300-dimensional embeddings [2]} and a concatenation of GloVe and fastText in our initial experiments and selected fastText because it outperformed others. This layer is followed by two bi-directional LSTM layers with a dense layer on top. The final prediction is obtained via a softmax layer. For our training parameters, we used a maximum sequence length of 128, batch size of 32, a learning rate of \(1e^{-3}\)  with Adam optimizer and epochs of 20 with early stopping patience of 5. The obtained results are summarized in Table REF . We observed that performance of this system is significantly lower than BERT for all scenarios we tested. However, conclusions regarding the transferability of PDTB-3 and CTB Bal onto CNC (based on F1 and MCC scores) holds.
