Energy big data: A survey <s> the worker s answer is j | the task s true answer is i). <s> We describe a probabilistic approach for supervised learning when we have multiple experts/annotators providing (possibly noisy) labels but no absolute gold standard. The proposed algorithm evaluates the different experts and also gives an estimate of the actual hidden labels. Experimental results indicate that the proposed method is superior to the commonly used majority voting baseline. <s> BIB001 </s> Energy big data: A survey <s> the worker s answer is j | the task s true answer is i). <s> Some complex problems, such as image tagging and natural language processing, are very challenging for computers, where even state-of-the-art technology is yet able to provide satisfactory accuracy. Therefore, rather than relying solely on developing new and better algorithms to handle such tasks, we look to the crowdsourcing solution -- employing human participation -- to make good the shortfall in current technology. Crowdsourcing is a good supplement to many computer tasks. A complex job may be divided into computer-oriented tasks and human-oriented tasks, which are then assigned to machines and humans respectively. ::: ::: To leverage the power of crowdsourcing, we design and implement a Crowdsourcing Data Analytics System, CDAS. CDAS is a framework designed to support the deployment of various crowdsourcing applications. The core part of CDAS is a quality-sensitive answering model, which guides the crowdsourcing engine to process and monitor the human tasks. In this paper, we introduce the principles of our quality-sensitive model. To satisfy user required accuracy, the model guides the crowdsourcing query engine for the design and processing of the corresponding crowdsourcing jobs. It provides an estimated accuracy for each generated result based on the human workers' historical performances. When verifying the quality of the result, the model employs an online strategy to reduce waiting time. To show the effectiveness of the model, we implement and deploy two analytics jobs on CDAS, a twitter sentiment analytics job and an image tagging job. We use real Twitter and Flickr data as our queries respectively. We compare our approaches with state-of-the-art classification and image annotation techniques. The results show that the human-assisted methods can indeed achieve a much higher accuracy. By embedding the quality-sensitive model into crowdsourcing query engine, we effectively reduce the processing cost while maintaining the required query answer quality. <s> BIB002
Some techniques are studied in order to compute the parameters in worker models as above, and we just introduce two typical examples: • Qualification Test BIB001 The qualification test sets a series of tasks with known true answers. Only when the workers pass the qualification test and achieve good marks can they answer the real answer tasks. Referred to the worker's test answers, the worker's quality can be deduced. For instance, if a worker is modeled by Worker Probability and he answers correctly 8 out of 10 test tasks, his quality is p = 0.8. • Test-Injected Method BIB002 The gold-injected method integrates test tasks and real tasks. For instance, if 10 percent of tasks are set as test tasks, each time when a worker is assigned with 10 tasks, he will take 1 test task and 9 real tasks. Based on the accuracy of worker's answers on test tasks, his quality can be calculated. However, this method is different from the qualification test because worker doesn't know that some tasks are the tests. When the workers are in the low quality, worker elimination is essential to improve the whole quality. There are many approaches to detect low-quality workers. A simple one is still using qualification test or test-injected method. It uses test tasks to compute the worker's quality and workers with low quality (e.g. p ≤ 0.5) are not allowed to process the real tasks. Therefore, the overall quality can keep ranging in a high level.