Users have tremendous potential to aid in the construction and maintenance of knowledges bases (KBs) through the contribution of feedback that identifies incorrect and missing entity attributes and relations. However, as new data is added to the KB, the KB entities, which are constructed by running entity resolution (ER), can change, rendering the intended targets of user feedback unknown–a problem we term identity uncertainty. In this work, we present a framework for integrating user feedback into KBs in the presence of identity uncertainty. Our approach is based on having user feedback participate alongside mentions in ER. We propose a specific representation of user feedback as feedback mentions and introduce a new online algorithm for integrating these mentions into an existing KB. In experiments, we demonstrate that our proposed approach outperforms the baselines in 70% of experimental conditions. Structured knowledge bases (KBs) of entities and relations are often incomplete and noisy, whether constructed by hand or automatically. For example, it has been reported that 71% of people in Freebase are missing a place of birth attribute and 75% have no known nationality BID5 . Similarly, while YAGO2 is estimated to be about 95% accurate on facts extracted from Wikipedia, this translates to roughly 5.7 million incorrect facts involving 2.6 million entities 1 BID11 . The vast research in cleaning and correction of databases is further evidence of the permeation of errors throughout KB construction in multiple domains BID5 ,b, Wang et al., 2015 .As the primary consumers of KBs, human users have significant potential to aid in KB construction and maintenance. From a user's standpoint, a KB contains a set of entities, each entity possessing attributes and optionally participating in relationships with other entities. Thus , KB errors manifest as spurious and missing attributes and relationships. However , the data that gives rise to a KB is a collection of raw evidence, which can be understood as mentions that require clustering by entity resolution (ER) into a set of inferred entities. The attributes and relations of the inferred KB entities with which the user interacts are drawn from this underlying clustering of the mentions. Therefore, the t = 0 t = 1 t = 2 c a u s e s s p l i t Tables 1a and 1b contain the results of the expertise and title experiments, respectively. Each table reports the paired t-statistic between each baseline method and our proposed Example feedback is shown for a concise target. The packaging and payload for authorship FM would contain the two titles mentioned with positive and negative weights respectively. approach (FM), under detailed and concise feedback generation schemes, with respect to the number of pieces of feedback required to discover the ground-truth partition of the mentions. Each row represents a canopy in which the experiment is performed, and each column corresponds to a baseline method and feedback generation setting. Each cell contains the difference between the mean number of rounds required by the FM approach and a baseline approach to discover the ground-truth partition (higher is better). Positive numbers are bolded; asterisks (*) indicate statistical significance (p < 0.05) and two asterisks (**) indicate statistical significance (p < 0.01). Rows are omitted if the initial partition of the mentions, constructed by the OG algorithm and subject to no user feedback, is correct.The paired-t statistics compare our proposed feedback representation (FM) to the three baseline feedback representations. We find that FM outperforms pack in both the detailed and concise settings of Experiment I on all but two of the canopies. In 7 out of 14 canopies, the results are statistically significant. These results underscore the importance of using only certain attributes (stored in the packaging) during the initial nearest neighbor search. We hypothesize that storing shared attributes in the payload is especially important because otherwise they can interfere with initial routing. When feedback is made with respect to attributes that are not shared, as in Experiment II, separating packaging and payload is less important. This is evidenced by the pack approach slightly outperforming FMs in the detailed setting, but never significantly. FMs generally outperform pack in the concise setting. We hypothesize that this is a result of better initial placement of the feedback in the tree by the OG algorithm.In comparing, FM and assign we find that our proposed approach typically performs better in Experiment II while the baseline performs better in Experiment I. We note that the feedback in Experiment I is more ambiguous than Experiment II (because expertise is a shared attribute). We hypothesize that assign's better performance in Experiment I is due to the baseline's approach of deleting feedback to mitigate errors caused by identity uncertainty with respect to user feedback. We note that this agrees with the observation Table 1 : Paired-t statistic. Each cell represents that difference in mean number of feedback-rounds required to discover the ground-truth entities over 25 runs between a baseline, denoted by the column heading, and our proposed approach (FM). Positive numbers indicate that FM requires fewer rounds of feedback than its competitor (larger numbers are better). Two asterisks (**) indicates that the statistic is significant at a 0.01 significance level; one asterisk indicates statistical significance at the 0.05 level. The mcguire j canopy is excluded from Tables 1a and 1b and the robinson h canopy is excluded from Table 1b since in these canopies, either: 0 or 1 edits are required to discover the ground-truth entities across baselines.that FM generally outperforms assign-m in both experiments, in that assign-m is similar to the assign strategy but never deletes feedback. This work presents a framework for reasoning about user feedback under identity uncertainty during KB construction. We advocate representing user feedback as feedback mentions that participate in ER alongside standard mentions. Our feedback mentions are endowed with a packaging-used to identify similar mentions during ER-and a payload-that is used to add missing attributes to inferred entities, correct mistakes and influence future ER decision. We give a hierarchical model of inferred entities and present the OG algorithm for performing online ER amongst standard and feedback mentions. In experiments, we show that our approach often outperforms baseline approaches in terms of efficiency with respect to recovering the ground-truth partition in ER. Our work is a foundational step in addressing a significant and under-explored problem in automatic KB construction whose solution could improve the accuracy and efficacy of integrating expressive user feedback with KB content.
This paper develops a framework for integrating user feedback under identity uncertainty in knowledge bases. 
{"src": "scitldr-AIC_train", "paper_id": "SygLHbcapm"}