We present FAKTA which is a unified framework that integrates various components of a fact checking process: document retrieval from media sources with various types of reliability, stance detection of documents with respect to given claims, evidence extraction, and linguistic analysis. FAKTA predicts the factuality of given claims and provides evidence at the document and sentence level to explain its predictions
These steps are typically handled in isolation. For example, previous works @cite_2 @cite_19 proposed algorithms to predict factuality of claims by mainly focusing on only input claims (i.e., step () and their metadata information (e.g., the speaker of the claim). In addition, recent works on the Fact Extraction and VERification (FEVER) @cite_13 has focused on a specific domain (e.g., Wikipedia).
@cite_19 @cite_13 @cite_2
2909693640 2963961878 2963416784
This work was supported by the National Science Foundation Science and Technology Center Award CCF-123121. In this paper we introduce a new publicly available dataset for verification against textual sources, FEVER: Fact Extraction and VERification. It consists of 185,445 claims generated by altering sentences extracted from Wikipedia and subsequently verified without knowledge of the sentence they were derived from. The claims are classified as SUPPORTED, REFUTED or NOTENOUGHINFO by annotators achieving 0.6841 in Fleiss κ. For the first two classes, the annotators also recorded the sentence(s) forming the necessary evidence for their judgment. To characterize the challenge of the dataset presented, we develop a pipeline approach and compare it to suitably designed oracles. The best accuracy we achieve on labeling a claim accompanied by the correct evidence is 31.87 , while if we ignore the evidence we achieve 50.91 . Thus we believe that FEVER is a challenging testbed that will help stimulate progress on claim verification against textual sources.