A Survey on Truth Discovery <s> RELATED AREAS <s> This paper examines eight published reviews each reporting results from several related trials. Each review pools the results from the relevant trials in order to evaluate the efficacy of a certain treatment for a specified medical condition. These reviews lack consistent assessment of homogeneity of treatment effect before pooling. We discuss a random effects approach to combining evidence from a series of experiments comparing two treatments. This approach incorporates the heterogeneity of effects in the analysis of the overall treatment efficacy. The model can be extended to include relevant covariates which would reduce the heterogeneity and allow for more specific therapeutic recommendations. We suggest a simple noniterative procedure for characterizing the distribution of treatment effects in a series of studies. <s> BIB001 </s> A Survey on Truth Discovery <s> RELATED AREAS <s> We consider the problem of using a large unlabeled sample to boost performance of a learning algorit,hrn when only a small set of labeled examples is available. In particular, we consider a problem setting motivated by the task of learning to classify web pages, in which the description of each example can be partitioned into two distinct views. For example, the description of a web page can be partitioned into the words occurring on that page, and the words occurring in hyperlinks t,hat point to that page. We assume that either view of the example would be sufficient for learning if we had enough labeled data, but our goal is to use both views together to allow inexpensive unlabeled data to augment, a much smaller set of labeled examples. Specifically, the presence of two distinct views of each example suggests strategies in which two learning algorithms are trained separately on each view, and then each algorithm’s predictions on new unlabeled examples are used to enlarge the training set of the other. Our goal in this paper is to provide a PAC-style analysis for this setting, and, more broadly, a PAC-style framework for the general problem of learning from both labeled and unlabeled data. We also provide empirical results on real web-page data indicating that this use of unlabeled examples can lead to significant improvement of hypotheses in practice. *This research was supported in part by the DARPA HPKB program under contract F30602-97-1-0215 and by NSF National Young investigator grant CCR-9357793. Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. TO copy otherwise, to republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. COLT 98 Madison WI USA Copyright ACM 1998 l-58113-057--0/98/ 7...%5.00 92 Tom Mitchell School of Computer Science Carnegie Mellon University Pittsburgh, PA 15213-3891 mitchell+@cs.cmu.edu <s> BIB002 </s> A Survey on Truth Discovery <s> RELATED AREAS <s> We consider the problem of combining ranking results from various sources. In the context of the Web, the main applications include building meta-search engines, combining ranking functions, selecting documents based on multiple criteria, and improving search precision through word associations. We develop a set of techniques for the rank aggregation problem and compare their performance to that of well-known methods. A primary goal of our work is to design rank aggregation techniques that can e ectively combat \spam," a serious problem in Web searches. Experiments show that our methods are simple, e cient, and e ective. <s> BIB003 </s> A Survey on Truth Discovery <s> RELATED AREAS <s> Multisensor fusion and integration is a rapidly evolving research area and requires interdisciplinary knowledge in control theory, signal processing, artificial intelligence, probability and statistics, etc. The advantages gained through the use of redundant, complementary, or more timely information in a system can provide more reliable and accurate information. This paper provides an overview of current sensor technologies and describes the paradigm of multisensor fusion and integration as well as fusion techniques at different fusion levels. Applications of multisensor fusion in robotics, biomedical system, equipment monitoring, remote sensing, and transportation system are also discussed. Finally, future research directions of multisensor fusion technology including microsensors, smart sensors, and adaptive fusion techniques are presented. <s> BIB004 </s> A Survey on Truth Discovery <s> RELATED AREAS <s> We consider clustering problems in which the available attributes can be split into two independent subsets, such that either subset suffices for learning. Example applications of this multi-view setting include clustering of Web pages which have an intrinsic view (the pages themselves) and an extrinsic view (e.g., anchor texts of inbound hyperlinks); multi-view learning has so far been studied in the context of classification. We develop and study partitioning and agglomerative, hierarchical multi-view clustering algorithms for text data. We find empirically that the multi-view versions of k-means and EM greatly improve on their single-view counterparts. By contrast, we obtain negative results for agglomerative hierarchical multi-view clustering. Our analysis explains this surprising phenomenon. <s> BIB005 </s> A Survey on Truth Discovery <s> RELATED AREAS <s> Recommender systems have proven to be an important response to the information overload problem, by providing users with more proactive and personalized information services. And collaborative filtering techniques have proven to be an vital component of many such recommender systems as they facilitate the generation of high-quality recom-mendations by leveraging the preferences of communities of similar users. In this paper we suggest that the traditional emphasis on user similarity may be overstated. We argue that additional factors have an important role to play in guiding recommendation. Specifically we propose that the trustworthiness of users must be an important consideration. We present two computational models of trust and show how they can be readily incorporated into standard collaborative filtering frameworks in a variety of ways. We also show how these trust models can lead to improved predictive accuracy during recommendation. <s> BIB006 </s> A Survey on Truth Discovery <s> RELATED AREAS <s> This textbook provides a comprehensive introduction to the theories and techniques of multi-sensor data fusion. It is aimed at advanced undergraduate and first-year graduate students in electrical engineering and computer science, as well as researchers and professional engineers. The book is intended to be self-contained. No previous knowledge of multi-sensor data fusion is assumed, although some familiarity with the basic tools of linear algebra, calculus and simple probability theory is recommended. Although conceptually simple, the study of multi-sensor data fusion presents challenges that are unique within the education of the electrical engineer or computer scientist. To become competent in the field the student must become familiar with tools taken from a wide range of diverse subjects including: neural networks, signal processing, statistical estimation, tracking algorithms, computer vision and control theory. All too often the student views multi-sensor data fusion as a miscellaneous assortment of different processes which bear no relationship to each other. In this book the processes are described using a common statistical framework. As a consequence, the underlying pattern of relationships that exists between the different methodologies is made evident. <s> BIB007 </s> A Survey on Truth Discovery <s> RELATED AREAS <s> This paper analyzes the trustor and trustee factors that lead to inter-personal trust using a well studied Trust Antecedent framework in management science \cite{mayer}. To apply these factors to trust ranking problem in online rating systems, we derive features that correspond to each factor and develop different trust ranking models. The advantage of this approach is that features relevant to trust can be systematically derived so as to achieve good prediction accuracy. Through a series of experiments on real data from Epinions, we show that even a simple model using the derived features yields good accuracy and outperforms MoleTrust, a trust propagation based model. SVM classifiers using these features also show improvements. <s> BIB008 </s> A Survey on Truth Discovery <s> RELATED AREAS <s> This article provides an overview of rank aggregation methods and algorithms, with an emphasis on modern biological applications. Rank aggregation methods have traditionally been used extensively in marketing and advertisement research, and in applied psychology in general. In recent years, rank aggregation methods have emerged as an important tool for combining information from different Internet search engines or from different omics-scale biological studies. We discuss three classes of methods, namely distributional based, heuristic, and stochastic search. The original Thurstone's scaling and its extensions represent the first class of methods that are most appropriate for aggregating many short ranked lists. Aggregating results from consumer rankings of products falls into this category of problems. Its application to biological problems is also being explored. On the other hand, heuristic algorithms and stochastic search methods are applicable to the situation of aggregating a small number of long lists, the so-called ‘high-level’ meta-analysis scenario. Combining results from different search engines/criteria and a number of omics-scale biological applications fall into this category. Heuristic algorithms are deterministic in nature, ranging from simple arithmetic averages of ranks to Markov chains and stationary distributions. Stochastic search algorithms, on the other hand, aim at maximizing a particular criterion such as that following the Kemeny guideline. Several examples will be provided to illustrate, compare, and contrast the methods and algorithms. The examples range from simple and contrive to representing realistic scenarios. In particular, an application to aggregating results from gene expression microarray studies is provided to demonstrate applications of the methods to modern biological problems. Copyright © 2010 John Wiley & Sons, Inc. ::: ::: ::: ::: For further resources related to this article, please visit the WIREs website. <s> BIB009 </s> A Survey on Truth Discovery <s> RELATED AREAS <s> Ensemble methods have been called the most influential development in Data Mining and Machine Learning in the past decade. They combine multiple models into one usually more accurate than the best of its components. Ensembles can provide a critical boost to industrial challenges -- from investment timing to drug discovery, and fraud detection to recommendation systems -- where predictive accuracy is more vital than model interpretability. Ensembles are useful with all modeling algorithms, but this book focuses on decision trees to explain them most clearly. After describing trees and their strengths and weaknesses, the authors provide an overview of regularization -- today understood to be a key reason for the superior performance of modern ensembling algorithms. The book continues with a clear description of two recent developments: Importance Sampling (IS) and Rule Ensembles (RE). IS reveals classic ensemble methods -- bagging, random forests, and boosting -- to be special cases of a single algorithm, thereby showing how to improve their accuracy and speed. REs are linear rule models derived from decision tree ensembles. They are the most interpretable version of ensembles, which is essential to applications such as credit scoring and fault diagnosis. Lastly, the authors explain the paradox of how ensembles achieve greater accuracy on new data despite their (apparently much greater) complexity.This book is aimed at novice and advanced analytic researchers and practitioners -- especially in Engineering, Statistics, and Computer Science. Those with little exposure to ensembles will learn why and how to employ this breakthrough method, and advanced practitioners will gain insight into building even more powerful models. Throughout, snippets of code in R are provided to illustrate the algorithms described and to encourage the reader to try the techniques. (edited by author) <s> BIB010 </s> A Survey on Truth Discovery <s> RELATED AREAS <s> A Cyber-Physical System (CPS) is an integration of sensor networks with informational devices. CPS can be used for many promising applications, such as traffic observation, battlefield surveillance, and sensor-network-based monitoring. One key issue in CPS research is trustworthiness analysis of sensor data. Due to technology limitations and environmental influences, the sensor data collected by CPS are inherently noisy and may trigger many false alarms. It is highly desirable to sift meaningful information from a large volume of noisy data. In this study, we propose a method called Tru-Alarm, which increases the capability of a CPS to recognize trustworthy alarms. Tru-Alarm estimates the locations of objects causing alarms, constructs an object-alarm graph and carries out trustworthiness inference based on the graph links. The study also reveals that the alarm trustworthiness and sensor reliability could be mutually enhanced. The property is used to help prune the large search space of object-alarm graph, filter out the alarms generated by unreliable sensors and improve the algorithm@?s efficiency. Extensive experiments are conducted on both real and synthetic datasets, and the results show that Tru-Alarm filters out noise and false information efficiently and effectively, while ensuring that no meaningful alarms are missed. <s> BIB011 </s> A Survey on Truth Discovery <s> RELATED AREAS <s> The rapid development of social media exacerbates the information overload and credibility problems. Trust, providing information about with whom we can trust to share information and from whom we can accept information, plays an important role in helping users collect relevant and reliable information in social media. Trust has become a research topic of increasing importance and of practical significance. In this tutorial, we illustrate properties and representation models of trust, elucidate trust measurements with representative algorithms, and demonstrate real-world applications where trust is explicitly used. As a new dimension of the trust study, we discuss the concept of distrust and its roles in trust measurements and applications. <s> BIB012
From a broader view, there are some research topics in the area of aggregation and fusion that are relevant to truth discovery. These research topics include multi-view learning/clustering, rank aggregation, sensor data fusion, meta analysis, and ensemble learning. However, the problem settings of these research topics are different from that of truth discovery. Specifically, in multi-view learning/clustering BIB005 BIB002  , the input is data of various feature sets (views) and the task is to conduct classification or clustering on multiple views jointly. Rank aggregation BIB003 BIB009 has a different input/output space with truth discovery as it focuses on the aggregation of ranking functions. In sensor data fusion BIB004 BIB007 , the source reliability is not considered and usually all the sources are treated indistinguishably. In meta analysis BIB001  , different lab studies are combined via weighted aggregation, but the source weight is derived mostly based on the size of the sample used in each lab study. In ensemble learning BIB010  , the weights of base models are inferred through supervised training whereas truth discovery is unsupervised. Compared with these research topics, truth discovery methods aggregate data collected from various sources on the same set of features on the same set of objects, and the goal of truth discovery is to resolve the conflicts among multiple sources by automatically estimating source reliability based on multi-source data only. Another relevant research area is information trustworthiness analysis. First, in some trustworthiness scoring systems, various methods have been developed for evaluating the reputation or trustworthiness of users, accounts or nodes in the context of social media platforms BIB008 BIB012 , or sensor networks  BIB011 . However, these systems focus on the computation of trustworthiness or reliability degrees by considering the content and/or links among sources. Second, in trust-enhanced recommendation systems BIB006  , the recommendation for one user on one item can be computed by a weighted combination of ratings from other users on this item, where weights incorporate trust between users. In such systems, trust and recommendation are derived separately from different types of data -Trust is derived based on users' provided trust values about others or the recommendation history information, while recommendation is conducted based on current user-item ratings and the derived trust. Compared with these studies, truth discovery tightly combines the process of source reliability estimation and truth computation to discover both of them from multisource data, and source reliability is usually defined as the probability of a source giving true claims.