@inproceedings{
ashok2023mathbbscimathbbfix,
title={\${\textbackslash}mathbb\{S\}\$ci\${\textbackslash}mathbb\{F\}\$ix: Outperforming {GPT}3 on Scientific Factual Error Correction},
author={Dhananjay Ashok and Atharva Kulkarni and Hai Pham and Barnabas Poczos},
booktitle={NeurIPS 2023 Workshop on Synthetic Data Generation with Generative AI},
year={2023},
url={https://openreview.net/forum?id=yJ1dQFVAgU}
}
Due to the prohibitively high cost of creating error correction datasets, most Factual Claim Correction methods rely on a powerful verification model to guide the correction process. This leads to a significant drop in performance in domains like Scientific Claim Correction, where good verification models do not always exist. In this work we introduce SciFix, a claim correction system that does not require a verifier but is able to outperform existing methods by a considerable margin — achieving correction accuracy of 84% on the SciFact dataset, 77% on SciFact-Open and 72% on the CovidFact dataset, compared to next best accuracies of 7%, 5% and 15% on the same datasets respectively. Our method leverages the power of prompting with LLMs during training to create a richly annotated dataset that can be used for fully supervised training and regularization. We additionally use a claim-aware decoding procedure to improve the quality of corrected claims. Our method outperforms the very LLM that was used to generate the annotated dataset — with FewShot Prompting on GPT3.5 achieving 58%, 61% and 64% on the respective datasets, a consistently lower correction accuracy, despite using nearly 800 times as many parameters as our model.
𝕊ci𝔽ix: Outperforming GPT3 on Scientific Factual Error Correction
Dhananjay Ashok Atharva Kulkarni Hai Pham Barnabas Poczos