With the rise of linked data and knowledge graphs, the need becomes compelling to find suitable solutions to increase the coverage and correctness of datasets, to add missing knowledge and to identify and remove errors. Several approaches – mostly relying on machine learning and NLP techniques – have been proposed to address this refinement goal; they usually need a partial gold standard, i.e. some “ground truth” to train automatic models. Gold standards are manually constructed, either by involving domain experts or by adopting crowdsourcing and human computation solutions.
Machine learning methods, however, need a partial to train automated models; those training sets are usually created manually by experts: while this usually leads to higher quality trained models, it is also an expensive process, so those ground truth'' datasets are usually small. Involving humans at scale in an effective way is, on the other hand, the goal of crowdsourcing @cite_7 and human computation @cite_2 . Indeed, microtask workers have been employed as a mean to perform manual quality assessment of linked data @cite_27 @cite_9 .
@cite_9 @cite_27 @cite_7 @cite_2
187967972 31535402 2170989440
Many aspects of Linked Data management - including exposing legacy data and applications to semantic formats, designing vocabularies to describe RDF data, identifying links between entities, query processing, and data curation - are necessarily tackled through the combination of human effort with algorithmic techniques. In the literature on traditional data management the theoretical and technical groundwork to realize and manage such combinations is being established. In this paper we build upon and extend these ideas to propose a framework by which human and computational intelligence can co-exist by augmenting existing Linked Data and Linked Service technology with crowdsourcing functionality. Starting from a motivational scenario we introduce a set of generic tasks which may feasibly be approached using crowdsourcing platforms such as Amazon's Mechanical Turk, explain how these tasks can be decomposed and translated into MTurk projects, and roadmap the extensions to SPARQL, D2RQ R2R and Linked Data browsing that are required to achieve this vision. In this paper we look into the use of crowdsourcing as a means to handle Linked Data quality problems that are challenging to be solved automatically. We analyzed the most common errors encountered in Linked Data sources and classified them according to the extent to which they are likely to be amenable to a specific form of crowdsourcing. Based on this analysis, we implemented a quality assessment methodology for Linked Data that leverages the wisdom of the crowds in different ways: (i) a contest targeting an expert crowd of researchers and Linked Data enthusiasts; complemented by (ii) paid microtasks published on Amazon Mechanical Turk.We empirically evaluated how this methodology could efficiently spot quality issues in DBpedia. We also investigated how the contributions of the two types of crowds could be optimally integrated into Linked Data curation processes. The results show that the two styles of crowdsourcing are complementary and that crowdsourcing-enabled quality assessment is a promising and affordable way to enhance the quality of Linked Data. The rapid growth of human computation within research and industry has produced many novel ideas aimed at organizing web users to do great things. However, the growth is not adequately supported by a framework with which to understand each new system in the context of the old. We classify human computation systems to help identify parallels between different systems and reveal "holes" in the existing work as opportunities for new research. Since human computation is often confused with "crowdsourcing" and other terms, we explore the position of human computation with respect to these related topics.