Previous studies [1]}, [2]} have shown that applying intermediate auxiliary task training to an encoder such as BERT can improve performance on a target task. We designed an intermediate task for BERT using CausalBank \(\mathcal {B}\) , employing margin loss [3]}, [4]} in the objective function: \(L(\Theta ) = \sum _{(c,e)\in \mathcal {B}}(\max (0, m - f(c,e) + f(c^{\prime },e^{\prime }))) +\frac{\lambda }{2} ||\Theta ||^2\) ,
where \(f(c,e)\)  is the score of true CE pair given by BERT model, \(f(c^{\prime },e^{\prime })\)  is the score of corrupted CE pair by replacing \(c\)  or \(e\)  with randomly sampled negative cause \(c^{\prime }\)  or effect \(e^{\prime }\)  from other examples in \(\mathcal {B}\) . \(m > 0\)  is the margin loss function parameter, which is set to 0.3. \(\Theta \)  is the set of BERT model parameters. \(\lambda \)  is the parameter for L2 regularization, which is set to 0.00001.
