A Survey on Truth Discovery <s> Enriched meaning of source reliability <s> ABSTRACT This paper addresses the challenge of truth discovery from noisy social sensing data. The work is motivated by the emergence of social sensing as a data collection paradigm of growing interest, where humans perform sensory data collection tasks. A challenge in social sensing applications lies in the noisy nature of data. Unlike the case with well-calibrated and well-tested infrastructure sensors, humans are less reliable, and the likelihood that participants' measurements are correct is often unknown a priori. Given a set of human participants of unknown reliability together with their sensory measurements, this paper poses the question of whether one can use this information alone to determine, in an analytically founded manner, the probability that a given measurement is true. The paper focuses on binary measurements. While some previous work approached the answer in a heuristic manner, we offer the first optimal solution to the above truth discovery problem. Optimality, in the sense of maximum likelihood estimation, is attained by solving an expectation maximization problem that returns the best guess regarding the correctness of each measurement. The approach is shown to outperform the state of the art fact-finding heuristics, as well as simple baselines such as majority voting. <s> BIB001 </s> A Survey on Truth Discovery <s> Enriched meaning of source reliability <s> In practical data integration systems, it is common for the data sources being integrated to provide conflicting information about the same entity. Consequently, a major challenge for data integration is to derive the most complete and accurate integrated records from diverse and sometimes conflicting sources. We term this challenge the truth finding problem. We observe that some sources are generally more reliable than others, and therefore a good model of source quality is the key to solving the truth finding problem. In this work, we propose a probabilistic graphical model that can automatically infer true records and source quality without any supervision. In contrast to previous methods, our principled approach leverages a generative process of two types of errors (false positive and false negative) by modeling two different aspects of source quality. In so doing, ours is also the first approach designed to merge multi-valued attribute types. Our method is scalable, due to an efficient sampling-based inference algorithm that needs very few iterations in practice and enjoys linear time complexity, with an even faster incremental variant. Experiments on two real world datasets show that our new method outperforms existing state-of-the-art approaches to the truth finding problem. <s> BIB002 </s> A Survey on Truth Discovery <s> Enriched meaning of source reliability <s> A frequent problem when dealing with data gathered from multiple sources on the web (ranging from booksellers to Wikipedia pages to stock analyst predictions) is that these sources disagree, and we must decide which of their (often mutually exclusive) claims we should accept. Current state-of-the-art information credibility algorithms known as "fact-finders" are transitive voting systems with rules specifying how votes iteratively flow from sources to claims and then back to sources. While this is quite tractable and often effective, fact-finders also suffer from substantial limitations; in particular, a lack of transparency obfuscates their credibility decisions and makes them difficult to adapt and analyze: knowing the mechanics of how votes are calculated does not readily tell us what those votes mean, and finding, for example, that a source has a score of 6 is not informative. We introduce a new approach to information credibility, Latent Credibility Analysis (LCA), constructing strongly principled, probabilistic models where the truth of each claim is a latent variable and the credibility of a source is captured by a set of model parameters. This gives LCA models clear semantics and modularity that make extending them to capture additional observed and latent credibility factors straightforward. Experiments over four real-world datasets demonstrate that LCA models can outperform the best fact-finders in both unsupervised and semi-supervised settings. <s> BIB003 </s> A Survey on Truth Discovery <s> Enriched meaning of source reliability <s> In many real world applications, the same item may be described by multiple sources. As a consequence, conflicts among these sources are inevitable, which leads to an important task: how to identify which piece of information is trustworthy, i.e., the truth discovery task. Intuitively, if the piece of information is from a reliable source, then it is more trustworthy, and the source that provides trustworthy information is more reliable. Based on this principle, truth discovery approaches have been proposed to infer source reliability degrees and the most trustworthy information (i.e., the truth) simultaneously. However, existing approaches overlook the ubiquitous long-tail phenomenon in the tasks, i.e., most sources only provide a few claims and only a few sources make plenty of claims, which causes the source reliability estimation for small sources to be unreasonable. To tackle this challenge, we propose a confidence-aware truth discovery (CATD) method to automatically detect truths from conflicting data with long-tail phenomenon. The proposed method not only estimates source reliability, but also considers the confidence interval of the estimation, so that it can effectively reflect real source reliability for sources with various levels of participation. Experiments on four real world tasks as well as simulated multi-source long-tail datasets demonstrate that the proposed method outperforms existing state-of-the-art truth discovery approaches by successful discounting the effect of small sources. <s> BIB004 </s> A Survey on Truth Discovery <s> Enriched meaning of source reliability <s> Data integration is a challenging task due to the large numbers of autonomous data sources. This necessitates the development of techniques to reason about the benefits and costs of acquiring and integrating data. Recently the problem of source selection (i.e., identifying the subset of sources that maximizes the profit from integration) was introduced as a preprocessing step before the actual integration. The problem was studied for static sources and used the accuracy of data fusion to quantify the integration profit. In this paper, we study the problem of source selection considering dynamic data sources whose content changes over time. We define a set of time-dependent metrics, including coverage, freshness and accuracy, to characterize the quality of integrated data. We show how statistical models for the evolution of sources can be used to estimate these metrics. While source selection is NP-complete, we show that for a large class of practical cases, near-optimal solutions can be found, propose an algorithmic framework with theoretical guarantees for our problem and show its effectiveness with an extensive experimental evaluation on both real-world and synthetic data. <s> BIB005 </s> A Survey on Truth Discovery <s> Enriched meaning of source reliability <s> When integrating information from multiple sources, it is common to encounter conflicting answers to the same question. Truth discovery is to infer the most accurate and complete integrated answers from conflicting sources. In some cases, there exist questions for which the true answers are excluded from the candidate answers provided by all sources. Without any prior knowledge, these questions, named no-truth questions, are difficult to be distinguished from the questions that have true answers, named has-truth questions. In particular, these no-truth questions degrade the precision of the answer integration system. We address such a challenge by introducing source quality, which is made up of three fine-grained measures: silent rate, false spoken rate and true spoken rate. By incorporating these three measures, we propose a probabilistic graphical model, which simultaneously infers truth as well as source quality without any a priori training involving ground truth answers. Moreover, since inferring this graphical model requires parameter tuning of the prior of truth, we propose an initialization scheme based upon a quantity named truth existence score, which synthesizes two indicators, namely, participation rate and consistency rate. Compared with existing methods, our method can effectively filter out no-truth questions, which results in more accurate source quality estimation. Consequently, our method provides more accurate and complete answers to both has-truth and no-truth questions. Experiments on three real-world datasets illustrate the notable advantage of our method over existing state-of-the-art truth discovery methods. <s> BIB006
In most truth discovery work, source reliability is a parameter that is positively correlated with the probability of a source asserting truths. However in work BIB004  BIB003 BIB005 BIB001 BIB002 BIB006 , the meaning of this parameter is further enriched to fit more complex application scenarios. In many applications of crowd/social sensing, people want to infer an observation is true or false, for example, whether or not a specific gas station runs out of gas. Thus in BIB001 , the authors model the observations from participants as binary variables, and capture the source (participant) reliability by estimating both true positive rate and false positive rate. In LTM BIB002 , the authors assume that the true information about an object can contains more than one value, for example, the authors of a book can be more than one. To estimate the quality of a source, it is insufficient to only calculate the probability that this source's provided information is accurate (precision). Furthermore, the source reliability should also capture the probability that this source fails to provide truths (recall). Thus, in the proposed method LTM, both precision and recall are adopted to estimate the source reliability. On the other hand, it is possible that there is no truth for some objects, e.g., the death date of someone who is still alive. Under this scenario, the authors in BIB006 propose several new metrics, such as silent rate, false spoken rate and true spoken rate, to describe the source quality. When some sources only provide one or two pieces of information, truth discovery methods may not be able to accurately estimate reliability degrees for these sources. However, such sources are commonly observed and they may contribute to the main part of the collected data. To better deal with such "small" sources, in BIB004 , the authors propose a confidence-aware truth discovery approach that outputs the confidence interval of the source reliability estimation. In  , the authors assume that the truths are subjective rather than objective. That is, the identified truths are dependent on the end-users: For different users, the truths can be different. To fit this scenario, instead of representing the reliability of a source as a single scalar, they represent it as three interrelated, but separate values: truthfulness, completeness, and bias. Specifically, truthfulness reflects the probability of a source asserting the truth, completeness reflects the proportion of objects which are covered by a source, and bias is the extent to which a source tends to support a favored position. These three metrics are calculated according to the importance of claims with respect to a specific user, so that they can better reflect users' prior knowledge and preferences. In BIB005 , the authors study the problem of selecting a subset of sources in dynamic scenarios for truth discovery. To better capture the time-dependent source quality, they define a set of metrics, including accuracy (truthfulness in  ), coverage (completeness in  ), and freshness (the frequency of updates provided by a source). In the probabilistic model of BIB003 , the source reliability has different semantic meanings in different settings: It can indicate the probability of a source asserting truth, or the probability of a source both knowing and telling the truth, or even the probability of a source intending to tell the truth. The model selection depends on the specific application scenarios.