Citation count of a paper is a commonly used proxy for evaluating the significance of a paper in the scientific community. Yet citation measures are widely criticized for failing to accurately reflect the true impact of a paper.
Thus, we proposeCausalCite, a new way to measure the significance of a paper by assessing the causal impact of the paper on its follow-up papers.CausalCiteis based on a novel causal inference method,TextMatch, which adapts the traditional matching framework to high-dimensional text embeddings.TextMatchencodes each paper using text embeddings from large language models (LLMs) , extracts similar samples by cosine similarity, and synthesizes a counterfactual sample as the weighted average of similar papers according to their similarity values.
We demonstrate the effectiveness ofCausalCiteon various criteria, such as high correlation with paper impact as reported by scientific experts on a previous dataset of 1K papers, (test-of-time)  awards for past papers, and its stability across various subfields of AI. We also provide a set of findings that can serve as suggested ways for future researchers to use our metric for a better understanding of the quality of a paper.111Our code is available athttps://github.com/causalNLP/causal-cite.