A Survey on Truth Discovery <s> General Principle of Truth Discovery <s> Worker reliability is a longstanding issue in crowdsourcing, and the automatic discovery of high quality workers is an important practical problem. Most previous work on this problem mainly focuses on estimating the quality of each individual worker jointly with the true answer of each task. However, in practice, for some tasks, worker quality could be associated with some explicit characteristics of the worker, such as education level, major and age. So the following question arises: how do we automatically discover related worker attributes for a given task, and further utilize the findings to improve data quality? In this paper, we propose a general crowd targeting framework that can automatically discover, for a given task, if any group of workers based on their attributes have higher quality on average; and target such groups, if they exist, for future work on the same task. Our crowd targeting framework is complementary to traditional worker quality estimation approaches. Furthermore, an advantage of our framework is that it is more budget efficient because we are able to target potentially good workers before they actually do the task. Experiments on real datasets show that the accuracy of final prediction can be improved significantly for the same budget (or even less budget in some cases). Our framework can be applied to many real word tasks and can be easily integrated in current crowdsourcing platforms. <s> BIB001 </s> A Survey on Truth Discovery <s> General Principle of Truth Discovery <s> Information Extraction using multiple information sources and systems is beneficial due to multisource/system consolidation and challenging due to the resulting inconsistency and redundancy. We integrate IE and truth-finding research and present a novel unsupervised multi-dimensional truth finding framework which incorporates signals from multiple sources, multiple systems and multiple pieces of evidence by knowledge graph construction through multi-layer deep linguistic analysis. Experiments on the case study of Slot Filling Validation demonstrate that our approach can find truths accurately (9.4% higher F-score than supervised methods) and efficiently (finding 90% truths with only one half the cost of a baseline without credibility estimation). <s> BIB002
In this section, we discuss the general principle adopted by truth discovery approaches, and then describe three popular ways to model it in practice. After comparing these three ways, a general truth discovery procedure is given. As mentioned in Section 1, the most important feature of truth discovery is its ability to estimate source reliabilities. To identify the trustworthy information (truths), weighted aggregation of the multi-source data is performed based on the estimated source reliabilities. As both source reliabilities and truths are unknown, the general principle of truth discovery works as follows: If a source provides trustworthy information frequently, it will be assigned a high reliability; meanwhile, if one piece of information is supported by sources with high reliabilities, it will have big chance to be selected as truth. To better illustrate this principle, consider the example in Table 1 . There are three sources providing the birthplace information of six politicians (objects), and the goal is to infer the true birthplace for each politician based on the conflicting multi-source information. The last two rows in Table 1 show the identified birthplaces by majority voting and truth discovery respectively. By comparing the results given by majority voting and truth discovery in Table 1 , we can see that truth discovery methods outperform majority voting in the following cases: (1) If a tie case is observed in the information provided by sources for an object, majority voting can only randomly select one value to break the tie because each candidate value receives equal votes. However, truth discovery methods are able to distinguish sources by estimating their reliability degrees. Therefore, they can easily break the tie and output the value obtained by weighted aggregation. In this running example, Mahatma Gandhi represents a tie case. (2) More importantly, truth discovery approaches are able to output a minority value as the aggregated result. Let's take a close look at Barack Obama in Table 1 . The final aggregated result given by truth discovery is "Hawaii", which is a minority value provided by only Source 2 among these three sources. If most sources are unreliable and they provide the same incorrect information (consider the "Mount Everest" example in Section 1), majority voting has no chance to select the correct value as it is claimed by minority. In contrast, truth discovery can distinguish reliable and unreliable sources by inferring their reliability degrees, so it is able to derive the correct information by conducting weighted aggregation. As a result, truth discovery approaches are labeled as the methods that can discover "the wisdom of minority" BIB001 BIB002 . Next, we discuss three popular ways to incorporate this general principle in truth discovery methods.