This paper describes the COCO-Text dataset. In recent years large-scale datasets like SUN and Imagenet drove the advancement of scene understanding and object recognition. The goal of COCO-Text is to advance state-of-the-art in text detection and recognition in natural images. The dataset is based on the MS COCO dataset, which contains images of complex everyday scenes. The images were not collected with text in mind and thus contain a broad variety of text instances. To reflect the diversity of text in natural scenes, we annotate text with (a) location in terms of a bounding box, (b) fine-grained classification into machine printed text and handwritten text, (c) classification into legible and illegible text, (d) script of the text and (e) transcriptions of legible text. The dataset contains over 173k text annotations in over 63k images. We provide a statistical analysis of the accuracy of our annotations. In addition, we present an analysis of three leading state-of-the-art photo Optical Character Recognition (OCR) approaches on our dataset. While scene text detection and recognition enjoys strong advances in recent years, we identify significant shortcomings motivating future work.
Another related stream of research focuses on repeated labeling in the face of noisy labels as well as combining human workers with machine classifiers. In early work, @cite_20 looked into improving label quality by taking worker accuracy into account. Further, Wilber at al. @cite_17 investigate the use of grid questions, where workers select answers from a grid of images to take advantage of the parallelism of human perception. We also use similar grid interfaces, but our approach differs in that we do not require a specific number of responses, because we perform binary classification whereas they do relative comparisons. Closer to our work, @cite_14 propose a framework that combines object detectors with human annotators to annotate a dataset. While we also combine results from object detectors and human annotators, our work differs in that we do not have access to the detectors during the annotation process, but only initial detections as input. In this area, the work closest to ours is the approach by @cite_9 that proposes strategies to optimize the task allocation to human workers with constrained budgets. We adapt their approach in that we increase annotation redundancy only the supposedly most difficult annotations.
@cite_9 @cite_14 @cite_20 @cite_17
2273307612 1908985308 2051105555 1878518397
When crowdsourcing systems are used in combination with machine inference systems in the real world, they benefit the most when the machine system is deeply integrated with the crowd workers. However, if researchers wish to integrate the crowd with "off-the-shelf" machine classifiers, this deep integration is not always possible. This work explores two strategies to increase accuracy and decrease cost under this setting. First, we show that reordering tasks presented to the human can create a significant accuracy improvement. Further, we show that greedily choosing parameters to maximize machine accuracy is sub-optimal, and joint optimization of the combined system improves performance. The long-standing goal of localizing every object in an image remains elusive. Manually annotating objects is quite expensive despite crowd engineering innovations. Current state-of-the-art automatic object detectors can accurately detect at most a few objects per image. This paper brings together the latest advancements in object detection and in crowd engineering into a principled framework for accurately and efficiently localizing objects in images. The input to the system is an image to annotate and a set of annotation constraints: desired precision, utility and or human cost of the labeling. The output is a set of object annotations, informed by human feedback and computer vision. Our model seamlessly integrates multiple computer vision models with multiple sources of human input in a Markov Decision Process. We empirically validate the effectiveness of our human-in-the-loop labeling approach on the ILSVRC2014 object detection dataset. This paper presents and tests a formal mathematical model for the analysis of informant responses to systematic interview questions. We assume a situation in which the ethnographer does not know how much each informant knows about the cultural domain under consideration nor the answers to the questions. The model simultaneously provides an estimate of the cultural competence or knowledge of each informant and an estimate of the correct answer to each question asked of the informant. The model currently handles true-false, multiple-choice, andfill-in-the-blank type question formats. In familiar cultural domains the model produces good results from as few as four informants. The paper includes a table showing the number of informants needed to provide stated levels of confidence given the mean level of knowledge among the informants. Implications are discussed. Similarity comparisons of the form "Is object a more similar to b than to c?" are useful for computer vision and machine learning applications. Unfortunately, an embedding of @math points is specified by @math triplets, making collecting every triplet an expensive task. In noticing this difficulty, other researchers have investigated more intelligent triplet sampling techniques, but they do not study their effectiveness or their potential drawbacks. Although it is important to reduce the number of collected triplets, it is also important to understand how best to display a triplet collection task to a user. In this work we explore an alternative display for collecting triplets and analyze the monetary cost and speed of the display. We propose best practices for creating cost effective human intelligence tasks for collecting triplets. We show that rather than changing the sampling algorithm, simple changes to the crowdsourcing UI can lead to much higher quality embeddings. We also provide a dataset as well as the labels collected from crowd workers.