Leaf Nodes (F1, AllCorrect):
Does the predicted tree use the correct leaf sentences? We compute an F1 score by comparing leaf sentences \(S_{pred}\)  to \(S_{gold}\) . The “AllCorrect” score is 1 if the entire set of nodes is identified correctly (F1=1.0), 0 otherwise.
Steps (F1, AllCorrect):
Does the tree have the correct entailment step structure? Here we compute F1 and “AllCorrect” scores
in a way similar to leaf nodes by comparing
the set of entailment steps in the aligned proof \(T_{pred}^{^{\prime }}\)  with \(T_{gold}\) .
Intermediates (BLEURT, AllCorrect):
Are the synthesized intermediate nodes \(Int_{pred}^{^{\prime }}\)  AllCorrect.
For this metric, we use the state-of-the-art BLEURT metric for evaluating generations, using the off-the-shelf BLEURT-Large-512 model.We fine-tuned and evaluated the BLEURT metric further using 300 hand-labeled examples, however it performs similarly to the vanilla model. [1]}. Our analysis based on 300 hand-labelled examples from the development set suggests
that it correlates well with human ratings (\(correlation = 0.67\) , \(sensitivity=0.88\) , \(specificity=0.80\) ). Further, we define generation correctness as 1 if an aligned pair of \(int_{pred}^{^{\prime }}\) , \(int_{gold}\)  gives \(BLEURT > 0.28\) ,The BLEURT threshold was picked using a subset of 300 manually labeled pairs. When
we test this threshold
on the rest of the labeled pairs we get a high (89%) F1 score, indicating the threshold is reasonable. 0 otherwise.
Note that a correct intermediate doesn't necessarily imply a correct entailment, as the ancestor nodes may be wrong.
Finally, “Intermediates AllCorrect” per question is 1 if all the generated intermediates in the tree are correct, 0 otherwise.
Overall Proof (AllCorrect):
We define the overall “AllCorrect” score for a generated proof as
1 only if all of the leaves, steps, and intermediates are all correct, i.e., the tree
completely matches \(T_{gold}\) . Otherwise it scores 0. This is a strict metric: any error in
the generated tree will result in a score of 0.

