In mobile crowdsourcing (MCS), mobile users accomplish outsourced human intelligence tasks. MCS requires an appropriate task assignment strategy, since different workers may have different performance in terms of acceptance rate and quality. Task assignment is challenging, since a worker’s performance 1) may fluctuate, depending on both the worker’s current personal context and the task context and 2) is not known a priori, but has to be learned over time. Moreover, learning context-specific worker performance requires access to context information, which may not be available at a central entity due to communication overhead or privacy concerns. In addition, evaluating worker performance might require costly quality assessments. In this paper, we propose a context-aware hierarchical online learning algorithm addressing the problem of performance maximization in MCS. In our algorithm, a local controller (LC) in the mobile device of a worker regularly observes the worker’s context, her his decisions to accept or decline tasks and the quality in completing tasks. Based on these observations, the LC regularly estimates the worker’s context-specific performance. The mobile crowdsourcing platform (MCSP) then selects workers based on performance estimates received from the LCs. This hierarchical approach enables the LCs to learn context-specific worker performance and it enables the MCSP to select suitable workers. In addition, our algorithm preserves worker context locally, and it keeps the number of required quality assessments low. We prove that our algorithm converges to the optimal task assignment strategy. Moreover, the algorithm outperforms simpler task assignment strategies in experiments based on synthetic and real data.
Research has put some effort in theoretically defining and classifying CS systems, such as web-based @cite_50 , mobile @cite_21 and spatial @cite_43 CS. AhujaSchaar2016 LiuLiu2017 SlivkinsVaughan2013 Below, we give an overview on related work on task assignment in general, mobile and spatial CS systems, as relevant for our scenario. Note that strategic behavior of workers and task owners in CS systems, e.g., concerning pricing and effort spent in task completion @cite_36 is out of the scope of this paper. Also note that we assume that it is possible to assess the quality of a completed task. A different line of work on CS deals with quality estimation in case of missing ground truth, recently also using online learning @cite_22 .
@cite_22 @cite_36 @cite_21 @cite_43 @cite_50
2605194399 2522887313 1973875719 2467923433 2020740057
We consider a crowd-sourcing problem where in the process of labeling massive data sets, multiple labelers with unknown annotation quality must be selected to perform the labeling task for each incoming data sample or task, with the results aggregated using for example simple or weighted majority voting rule. In this paper, we approach this labeler selection problem in an online learning framework, whereby the quality of the labeling outcome by a specific set of labelers is estimated so that the learning algorithm over time learns to use the most effective combinations of labelers. This type of online learning in some sense falls under the family of multi-armed bandit (MAB) problems, but with a distinct feature not commonly seen: since the data is unlabeled to begin with and the labelers’ quality is unknown, their labeling outcome (or reward in the MAB context) cannot be readily verified; it can only be estimated against the crowd and be known probabilistically. We design an efficient online algorithm LS_OL using a simple majority voting rule that can differentiate high and low quality labelers over time, and is shown to have a regret (with respect to always using the optimal set of labelers) of @math uniformly in time under mild assumptions on the collective quality of the crowd, thus regret free in the average sense. We discuss further performance improvement by using a more sophisticated majority voting rule, and show how to detect and filter out “bad” (dishonest, malicious or very incompetent) labelers to further enhance the quality of crowd-sourcing. Extension to the case when a labeler’s quality is task-type dependent is also discussed using techniques from the literature on continuous arms. We establish a lower bound on the order of @math , where @math is an arbitrary function such that @math . We further provide a matching upper bound through a minor modification of the algorithm we proposed and studied earlier on. We present numerical results using both simulation and set of images labeled by amazon mechanic turks. In many two-sided markets, each side has incomplete information about the other but has an opportunity to learn (some) relevant information before final matches are made. For instance, clients seeking workers to perform tasks often conduct interviews that require the workers to perform some tasks and thereby provide information to both sides. The performance of a worker in such an interview assessment - and hence the information revealed - depends both on the inherent characteristics of the worker and the task and also on the actions taken by the worker (e.g. the effort expended); thus there is both adverse selection (on both sides) and moral hazard (on one side). When interactions are ongoing, incentives for workers to expend effort in the current assessment can be provided by the payment rule used and also by the matching rule that assesses and determines the tasks to which the worker is assigned in the future; thus workers have career concerns. We derive mechanisms - payment, assessment and matching rules - that lead to final matchings that are stable in the long run and achieve close to the optimal performance (profit or social welfare maximizing) in equilibrium (unique) thus mitigating both adverse selection and moral hazard (in many settings). With the proliferation of increasingly powerful mobile devices, mobile users can collaboratively form a mobile cloud to provide pervasive services, such as data collecting, processing, and computing. With this mobile cloud, mobile crowdsourcing, as an emerging service paradigm, can enable mobile users to take over the outsourced tasks. By leveraging the sensing capabilities of mobile devices and integrating humanintelligence and machine-computation, mobile crowdsourcing has the potential to revolutionize the approach of data collecting and processing. In this article we investigate the mobile crowdsourcing architecture and applications, then discuss some research challenges and countermeasures for developing mobile crowdsourcing. Some research orientations are finally envisioned for further studies. Crowdsourcing relies on the contributions of a large number of workers to accomplish spatial tasks, and it has drawn more attention in recent years. Many crowdsourcing tasks are completed online due to its convenience and efficiency. However, sometimes this traditional method may not work due to special requirements involving actual physical locations. Thus, a new paradigm of data collection, called spatial crowdsourcing, has emerged in the past few years. Spatial crowdsourcing consists of location-specific tasks that require people to physically be at specific locations to complete them. In this article we discuss unique challenges of spatial crowdsourcing, provide a comprehensive view of this new paradigm by introducing the taxonomy, and give future directions.