
Incorrect leaves results in invalid/missing entailment steps (56%):
For example, for the question “Why do mosquitoes instinctively move towards carbon dioxide...? A: It helps mosquitoes find food”, the predicted tree misses using the critical input fact that “mosquitoes eat animal blood”, hence cannot infer “animals are a source of food for mosquitoes”, hence cannot infer the importance of moving towards carbon dioxide.

Correct leaves, but invalid steps (18%): For example,
for a question asking “Can a person see someone in a dark room? A: No”, the
model selects the correct leaf sentences but stitches them together in the
wrong order. As a result, an invalid intermediate conclusion is generated.
Specifically, it incorrectly tries to draw an entailment from “a person is in a dark room” and “a person is looking into the dark room”, producing “the person outside can see the person in the dark room”, an invalid step and one that directly contradicts the target QA pair.

Correct steps, but incorrect intermediate conclusions (2%):
For example, for a question with H:“compression waves cause objects to move in the same direction of the wave”,
the model gets the proof structure right in terms of placement of relevant leaves. But instead of concluding a gold intermediate conclusion “\(int1_{gold}\) : longitudinal waves are also called compression waves” it prematurely predicts the final conclusion H for the intermediate \(int1_{pred}\)  (and then re-predicts it in the final entailment step).

Disconnected trees (4%):
We found 2 examples where the generated entailment tree had intermediate conclusions that were not used later towards proving the hypothesis.
E.g. for question with H: “the plants in the gardens will receive the most sunlight in summer to grow during the day”, the model accurately compose leaf facts to reach the intermediate conclusion “the plants in the gardens will receive sunlight to grow” but the predicted proof does not then use it in later steps. One way to avoid this would be to apply structural constraints on the output, enforcing a (single) tree structure
[1]}.

Imperfect evaluation (24%): For these cases we found that the generated entailment tree was correct but our metrics could not detect the equivalence in the predicted vs gold tree. Failure cases included 1) alternate valid entailment trees missing in our gold annotations 2) the generated proof is finer-grained or coarse-grained with respect to the gold proof. An improved evaluation metric would help alleviate these scoring discrepancies.

