Energy big data: A survey <s> 1) QUALITY CONTROL <s> We describe a probabilistic approach for supervised learning when we have multiple experts/annotators providing (possibly noisy) labels but no absolute gold standard. The proposed algorithm evaluates the different experts and also gives an estimate of the actual hidden labels. Experimental results indicate that the proposed method is superior to the commonly used majority voting baseline. <s> BIB001 </s> Energy big data: A survey <s> 1) QUALITY CONTROL <s> We consider a crowdsourcing database system that may cleanse, populate, or filter its data by using human workers. Just like a conventional DB system, such a crowdsourcing DB system requires data manipulation functions such as select, aggregate, maximum, average, and so on, except that now it must rely on human operators (that for example compare two objects) with very different latency, cost and accuracy characteristics. In this paper, we focus on one such function, maximum, that finds the highest ranked object or tuple in a set. In particularm we study two problems: given a set of votes (pairwise comparisons among objects), how do we select the maximum? And how do we improve our estimate by requesting additional votes? We show that in a crowdsourcing DB system, the optimal solution to both problems is NP-Hard. We then provide heuristic functions to select the maximum given evidence, and to select additional votes. We experimentally evaluate our functions to highlight their strengths and weaknesses. <s> BIB002 </s> Energy big data: A survey <s> 1) QUALITY CONTROL <s> It is universal to see people obtain knowledge on micro-blog services by asking others decision making questions. In this paper, we study the Jury Selection Problem(JSP) by utilizing crowdsourcing for decision making tasks on micro-blog services. Specifically, the problem is to enroll a subset of crowd under a limited budget, whose aggregated wisdom via Majority Voting scheme has the lowest probability of drawing a wrong answer(Jury Error Rate-JER). ::: ::: Due to various individual error-rates of the crowd, the calculation of JER is non-trivial. Firstly, we explicitly state that JER is the probability when the number of wrong jurors is larger than half of the size of a jury. To avoid the exponentially increasing calculation of JER, we propose two efficient algorithms and an effective bounding technique. Furthermore, we study the Jury Selection Problem on two crowdsourcing models, one is for altruistic users(AltrM) and the other is for incentive-requiring users(PayM) who require extra payment when enrolled into a task. For the AltrM model, we prove the monotonicity of JER on individual error rate and propose an efficient exact algorithm for JSP. For the PayM model, we prove the NP-hardness of JSP on PayM and propose an efficient greedy-based heuristic algorithm. Finally, we conduct a series of experiments to investigate the traits of JSP, and validate the efficiency and effectiveness of our proposed algorithms on both synthetic and real micro-blog data. <s> BIB003 </s> Energy big data: A survey <s> 1) QUALITY CONTROL <s> Worker quality control is a crucial aspect of crowdsourcing systems; typically occupying a large fraction of the time and money invested on crowdsourcing. In this work, we devise techniques to generate confidence intervals for worker error rate estimates, thereby enabling a better evaluation of worker quality. We show that our techniques generate correct confidence intervals on a range of real-world datasets, and demonstrate wide applicability by using them to evict poorly performing workers, and provide confidence intervals on the accuracy of the answers. <s> BIB004
The first step of the quality control is to characterize workers' quality, including modeling workers and defining parameters. The current researches have proposed several methods to model a worker's quality. We summarize some as follows. • Worker Probability BIB002 - BIB003 In worker probability, each worker's quality is taken as a single parameter p ∈ [0, 1], which indicates the accuracy rate of the worker's answer: p = Prob(the worker s answer = the task s true answer). For example, if a worker's answer has a probability of 80 percent to be correct, the workers quality would be characterized as p = 0.8. Since the accuracy parameter p is defined in the range of [0, 1], some studies extend it with two approaches: i) Extending to a wider range of (−∞, +∞), which enables a higher probability to the correct answer  . ii) Extending another parameter as a confidence interval to describe how confident the calculated p is. For example, if the confidence interval is ranged among [0.3, 0.8], it may narrow to [0.55, 0.6] after a worker answering more tasks BIB004 . • Confusion Matrix BIB001 Confusion matrix is often used to represent a worker's ability to process single-choice tasks with n potential answers. A confusion matrix is a n × n matrix: where the i-th (i ∈ [1, n]) row represents the distribution of the worker's answers for the i-th task's answer and the j-th (j ∈ [1, n]) column represents the correct answer probability of the j-th worker's answer. Therefore,