To identify the pool of relevant facts to show the annotator for each question, we could simply perform information retrieval over the corpus,
using the QA pair as the query. However, for this dataset, we are able to do better than this: First, we train
two “relevant sentence” classifiers (using BERT [1]} and RoBERTa [2]} respectively) using additional WorldTree annotations.WorldTree includes annotations about which WorldTree table rows are relevant to which questions, i.e., which rows
are supporting evidence (“rationales”) for which question. Although these rationales do not identify all relevant sentences,
they can be used as distant supervision (along with random negative facts drawn from the corpus) to train a “relevant sentence” classifier.
Then, for each question, both models exhaustively score each fact in the corpus, and the top 20 facts from each are retrieved, reranked using Tensorflow-Ranking-BERT [3]}, and presented as a ranked list to the entailment tree annotator based on their final scores.
