The number of redundant answers to collect from the crowd is predicted to efficiently capture the diversity of all answers from all visual questions.
ABSTRACTIVE
FLOAT SELECTED: Fig. 6: We propose a novel application of predicting the number of redundant answers to collect from the crowd per visual question to efficiently capture the diversity of all answers for all visual questions. (a) For a batch of visual questions, our system first produces a relative ordering using the predicted confidence in whether a crowd would agree on an answer (upper half). Then, the system allocates a minimum number of annotations to all visual questions (bottom, left half) and then the extra available human budget to visual questions most confidently predicted to lead to crowd disagreement (bottom, right half). (b) For 121,512 visual questions, we show results for our system, a related VQA algorithm, and today’s status quo of random predictions. Boundary conditions are one answer (leftmost) and five answers (rightmost) for all visual questions. Our approach typically accelerates the capture of answer diversity by over 20% from today’s Status Quo selection; e.g., 21% for 70% of the answer diversity and 23% for 86% of the answer diversity. This translates to saving over 19 40-hour work weeks and $1800, assuming 30 seconds and $0.02 per answer.
1608.08188
Introduction </s> What Ġwould Ġbe Ġpossible Ġif Ġa Ġperson Ġhad Ġan Ġor acle Ġthat Ġcould Ġimmediately Ġprovide Ġthe Ġanswer Ġto Ġany Ġquestion Ġabout Ġthe Ġvisual Ġworld ? ĠSight - im pa ired Ġusers Ġcould Ġquickly Ġand Ġreliably Ġfigure Ġout Ġthe Ġdenomination Ġof Ġtheir Ġcurrency Ġand Ġso Ġwhether Ġthey Ġspent Ġthe Ġappropriate Ġamount Ġfor Ġa Ġproduct ĠB IB REF 0 Ġ. ĠH ikers Ġcould Ġimmediately Ġlearn Ġabout Ġtheir Ġbug Ġbites Ġand Ġwhether Ġto Ġseek Ġout Ġemergency Ġmedical Ġcare . ĠPil ots Ġcould Ġlearn Ġhow Ġmany Ġbirds Ġare Ġin Ġtheir Ġpath Ġto Ġdecide Ġwhether Ġto Ġchange Ġcourse Ġand Ġso Ġavoid Ġcostly , Ġlife - threatening Ġcollisions . ĠThese Ġexamples Ġillustrate Ġseveral Ġof Ġthe Ġinterests Ġfrom Ġa Ġvisual Ġquestion Ġanswering Ġ( V Q A ) Ġsystem , Ġincluding Ġtackling Ġproblems Ġthat Ġinvolve Ġclassification , Ġdetection , Ġand Ġcounting . ĠMore Ġgenerally , Ġthe Ġgoal Ġfor ĠV Q A Ġis Ġto Ġhave Ġa Ġsingle Ġsystem Ġthat Ġcan Ġaccurately Ġanswer Ġany Ġnatural Ġlanguage Ġquestion Ġabout Ġan Ġimage Ġor Ġvideo ĠB IB REF 1 Ġ, ĠB IB REF 2 Ġ, ĠB IB REF 3 Ġ. </s> Ent angled Ġin Ġthe Ġdream Ġof Ġa ĠV Q A Ġsystem Ġis Ġan Ġunavoidable Ġissue Ġthat , Ġwhen Ġasking Ġmultiple Ġpeople Ġa Ġvisual Ġquestion , Ġsometimes Ġthey Ġall Ġagree Ġon Ġa Ġsingle Ġanswer Ġwhile Ġother Ġtimes Ġthey Ġoffer Ġdifferent Ġanswers Ġ( Figure ĠFIG REF 1 Ġ). ĠIn Ġfact , Ġas Ġwe Ġshow Ġin Ġthe Ġpaper , Ġthese Ġtwo Ġoutcomes Ġarise Ġin Ġapproximately Ġequal Ġproportions Ġin Ġtoday 's Ġlargest Ġpublicly - shared ĠV Q A Ġbenchmark Ġthat Ġcontains Ġover Ġ450 , 000 Ġvisual Ġquestions . ĠFigure ĠFIG REF 1 Ġillustrates Ġthat Ġhuman Ġdisagreements Ġarise Ġfor Ġa Ġvariety Ġof Ġreasons Ġincluding Ġdifferent Ġdescriptions Ġof Ġthe Ġsame Ġconcept Ġ( e . g ., ĠâĢ ľ min or " Ġand ĠâĢ ľ under age "), Ġdifferent Ġconcepts Ġ( e . g ., ĠâĢ ľ ghost " Ġand ĠâĢ ľ phot oshop "), Ġand Ġirrelevant Ġresponses Ġ( e . g ., ĠâĢ ľ no "). </s> Our Ġgoal Ġis Ġto Ġaccount Ġfor Ġwhether Ġdifferent Ġpeople Ġwould Ġagree Ġon Ġa Ġsingle Ġanswer Ġto Ġa Ġvisual Ġquestion Ġto Ġimprove Ġupon Ġtoday 's ĠV Q A Ġsystems . ĠWe Ġpropose Ġmultiple Ġprediction Ġsystems Ġto Ġautomatically Ġdecide Ġwhether Ġa Ġvisual Ġquestion Ġwill Ġlead Ġto Ġhuman Ġagreement Ġand Ġdemonstrate Ġthe Ġvalue Ġof Ġthese Ġpredictions Ġfor Ġa Ġnew Ġtask Ġof Ġcapturing Ġthe Ġdiversity Ġof Ġall Ġplausible Ġanswers Ġwith Ġless Ġhuman Ġeffort . </s> Our Ġwork Ġis Ġpartially Ġinspired Ġby Ġthe Ġgoal Ġto Ġimprove Ġhow Ġto Ġemploy Ġcrowds Ġas Ġthe Ġcomputing Ġpower Ġat Ġrun - time . ĠTowards Ġsatisfying Ġexisting Ġusers , Ġgaining Ġnew Ġusers , Ġand Ġsupporting Ġa Ġwide Ġrange Ġof Ġapplications , Ġa Ġcrowd - powered ĠV Q A Ġsystem Ġshould Ġbe Ġlow Ġcost , Ġhave Ġfast Ġresponse Ġtimes , Ġand Ġyield Ġhigh Ġquality Ġanswers . ĠToday 's Ġstatus Ġquo Ġis Ġto Ġassume Ġa Ġfixed Ġnumber Ġof Ġhuman Ġresponses Ġper Ġvisual Ġquestion Ġand Ġso Ġa Ġfixed Ġcost , Ġdelay , Ġand Ġpotential Ġdiversity Ġof Ġanswers Ġfor Ġevery Ġvisual Ġquestion ĠB IB REF 2 Ġ, ĠB IB REF 0 Ġ, ĠB IB REF 4 Ġ. ĠWe Ġinstead Ġpropose Ġto Ġdynamically Ġsolicit Ġthe Ġnumber Ġof Ġhuman Ġresponses Ġbased Ġon Ġeach Ġvisual Ġquestion . ĠIn Ġparticular , Ġwe Ġaim Ġto Ġacc rue Ġadditional Ġcosts Ġand Ġdelays Ġfrom Ġcollecting Ġextra Ġanswers Ġonly Ġwhen Ġextra Ġresponses Ġare Ġneeded Ġto Ġdiscover Ġall Ġplausible Ġanswers . ĠWe Ġshow Ġin Ġour Ġexperiments Ġthat Ġour Ġsystem Ġsaves Ġ19 Ġ40 - hour Ġwork Ġweeks Ġand Ġ$ 1800 Ġto Ġanswer Ġ121 , 512 Ġvisual Ġquestions , Ġcompared Ġto Ġtoday 's Ġstatus Ġquo Ġapproach ĠB IB REF 0 Ġ. </s> Our Ġwork Ġis Ġalso Ġinspired Ġby Ġthe Ġgoal Ġto Ġimprove Ġhow Ġto Ġemploy Ġcrowds Ġto Ġproduce Ġthe Ġinformation Ġneeded Ġto Ġtrain Ġand Ġevaluate Ġautomated Ġmethods . ĠSpecifically , Ġresearchers Ġin Ġfields Ġas Ġdiverse Ġas Ġcomputer Ġvision ĠB IB REF 2 Ġ, Ġcomputational Ġlingu istics ĠB IB REF 1 Ġ, Ġand Ġmachine Ġlearning ĠB IB REF 3 Ġrely Ġon Ġlarge Ġdatasets Ġto Ġimprove Ġtheir ĠV Q A Ġalgorithms . ĠThese Ġdatasets Ġinclude Ġvisual Ġquestions Ġand Ġhuman - supp lied Ġanswers . ĠSuch Ġdata Ġis Ġcritical Ġfor Ġteaching Ġmachine Ġlearning Ġalgorithms Ġhow Ġto Ġanswer Ġquestions Ġby Ġexample . ĠSuch Ġdata Ġis Ġalso Ġcritical Ġfor Ġevaluating Ġhow Ġwell ĠV Q A Ġalgorithms Ġperform . ĠIn Ġgeneral , ĠâĢ ľ big ger " Ġdata Ġis Ġbetter . ĠCurrent Ġmethods Ġto Ġcreate Ġthese Ġdatasets Ġassume Ġa Ġfixed Ġnumber Ġof Ġhuman Ġanswers Ġper Ġvisual Ġquestion ĠB IB REF 2 Ġ, ĠB IB REF 4 Ġ, Ġthereby Ġeither Ġcompromising Ġon Ġquality Ġby Ġnot Ġcollecting Ġall Ġplausible Ġanswers Ġor Ġcost Ġby Ġcollecting Ġadditional Ġanswers Ġwhen Ġthey Ġare Ġredundant . ĠWe Ġoffer Ġan Ġeconomical Ġway Ġto Ġspend Ġa Ġhuman Ġbudget Ġto Ġcollect Ġanswers Ġfrom Ġcrowd Ġworkers . ĠIn Ġparticular , Ġwe Ġaim Ġto Ġactively Ġallocate Ġadditional Ġanswers Ġonly Ġto Ġvisual Ġquestions Ġlikely Ġto Ġhave Ġmultiple Ġanswers . </s> The Ġkey Ġcontributions Ġof Ġour Ġwork Ġare Ġas Ġfollows : </s> Paper ĠOverview </s> The Ġremainder Ġof Ġthe Ġpaper Ġis Ġorganized Ġinto Ġfour Ġsections . ĠWe Ġfirst Ġdescribe Ġa Ġstudy Ġwhere Ġwe Ġinvestigate : Ġ1 ) ĠHow Ġmuch Ġanswer Ġdiversity Ġarises Ġfor Ġvisual Ġquestions ? Ġand Ġ2 ) ĠWhy Ġdo Ġpeople Ġdisagree Ġ( Section ĠSEC REF 4 Ġ) ? ĠNext , Ġwe Ġexplore Ġthe Ġfollowing Ġtwo Ġquestions : Ġ1 ) ĠGiven Ġa Ġnovel Ġvisual Ġquestion , Ġcan Ġa Ġmachine Ġcorrectly Ġpredict Ġwhether Ġmultiple Ġindependent Ġmembers Ġof Ġa Ġcrowd Ġwould Ġsupply Ġthe Ġsame Ġanswer ? Ġand Ġ2 ) ĠIf Ġso , Ġwhat Ġinsights Ġdoes Ġour Ġmachine - learn ed Ġsystem Ġreveal Ġregarding Ġwhat Ġhumans Ġare Ġmost Ġlikely Ġto Ġagree Ġabout Ġ( Section ĠSEC REF 5 Ġ) ? ĠIn Ġthe Ġfollowing Ġsection , Ġwe Ġpropose Ġa Ġnovel Ġresource Ġallocation Ġsystem Ġfor Ġefficiently Ġcapturing Ġthe Ġdiversity Ġof Ġall Ġanswers Ġfor Ġa Ġset Ġof Ġvisual Ġquestions Ġ( Section ĠSEC REF 6 Ġ). ĠFinally , Ġwe Ġend Ġwith Ġconcluding Ġremarks Ġ( Section ĠSEC REF 7 Ġ). </s> V Q A Ġ- ĠAnalysis Ġof ĠAnswer Ġ( Dis ) Ag reements </s> Our Ġfirst Ġaim Ġis Ġto Ġanswer Ġthe Ġfollowing Ġquestions : Ġ1 ) ĠHow Ġmuch Ġanswer Ġdiversity Ġarises Ġfor Ġvisual Ġquestions ? Ġand Ġ2 ) ĠWhy Ġdo Ġpeople Ġdisagree ? </s> Visual ĠQuestion Ġ- ĠPredict ing ĠIf Ġa ĠCrowd ĠWill Ġ( Dis ) Ag ree Ġon Ġan ĠAnswer </s> We Ġnow Ġexplore Ġthe Ġfollowing Ġtwo Ġquestions : Ġ1 ) ĠGiven Ġa Ġnovel Ġvisual Ġquestion , Ġcan Ġa Ġmachine Ġcorrectly Ġpredict Ġwhether Ġmultiple Ġindependent Ġmembers Ġof Ġa Ġcrowd Ġwould Ġsupply Ġthe Ġsame Ġanswer ? Ġand Ġ2 ) ĠIf Ġso , Ġwhat Ġinsights Ġdoes Ġour Ġmachine - learn ed Ġsystem Ġreveal Ġregarding Ġwhat Ġhumans Ġare Ġmost Ġlikely Ġto Ġagree Ġabout ? </s> Pred iction ĠSystems </s> We Ġpose Ġthe Ġprediction Ġtask Ġas Ġa Ġbinary Ġclassification Ġproblem . ĠSpecifically , Ġgiven Ġan Ġimage Ġand Ġassociated Ġquestion , Ġa Ġsystem Ġoutputs Ġa Ġbinary Ġlabel Ġindicating Ġwhether Ġa Ġcrowd Ġwill Ġagree Ġon Ġthe Ġsame Ġanswer . ĠOur Ġgoal Ġis Ġto Ġdesign Ġa Ġsystem Ġthat Ġcan Ġdetect Ġwhich Ġvisual Ġquestions Ġto Ġassign Ġa Ġdisagreement Ġlabel , Ġregardless Ġof Ġthe Ġdisagreement Ġcause Ġ( e . g ., Ġsubject ivity , Ġambiguity , Ġdifficulty ). ĠWe Ġimplement Ġboth Ġrandom Ġforest Ġand Ġdeep Ġlearning Ġclass ifiers . </s> A Ġvisual Ġquestion Ġis Ġassigned Ġeither Ġan Ġanswer Ġagreement Ġor Ġdisagreement Ġlabel . ĠTo Ġassign Ġlabels , Ġwe Ġemploy Ġ10 Ġcrowds ourced Ġanswers Ġfor Ġeach Ġvisual Ġquestion . ĠA Ġvisual Ġquestion Ġis Ġassigned Ġan Ġanswer Ġagreement Ġlabel Ġwhen Ġthere Ġis Ġan Ġexact Ġstring Ġmatch Ġfor Ġ9 Ġof Ġthe Ġ10 Ġcrowds ourced Ġanswers Ġ( after Ġanswer Ġpre - pre processing , Ġas Ġdiscussed Ġin Ġthe Ġprevious Ġsection ) Ġand Ġan Ġanswer Ġdisagreement Ġlabel Ġotherwise . ĠOur Ġrationale Ġis Ġto Ġpermit Ġthe Ġpossibility Ġof Ġup Ġto Ġone ĠâĢ ľ care less / sp am " Ġanswer Ġper Ġvisual Ġquestion . ĠThe Ġoutcome Ġof Ġour Ġlabeling Ġscheme Ġis Ġthat Ġa Ġdisagreement Ġlabel Ġis Ġag nostic Ġto Ġthe Ġspecific Ġcause Ġof Ġdisagreement Ġand Ġrather Ġrepresents Ġthe Ġmany Ġcauses Ġ( described Ġabove ). </s> For Ġour Ġfirst Ġsystem , Ġwe Ġuse Ġdomain Ġknowledge Ġto Ġguide Ġthe Ġlearning Ġprocess . ĠWe Ġcompile Ġa Ġset Ġof Ġfeatures Ġthat Ġwe Ġhypothes ize Ġinform Ġwhether Ġa Ġcrowd Ġwill Ġarrive Ġat Ġan Ġundis puted , Ġsingle Ġanswer . ĠThen Ġwe Ġapply Ġa Ġmachine Ġlearning Ġtool Ġto Ġreveal Ġthe Ġsignificance Ġof Ġeach Ġfeature . ĠWe Ġpropose Ġfeatures Ġbased Ġon Ġthe Ġobservation Ġthat Ġanswer Ġagreement Ġoften Ġarises Ġwhen Ġ1 ) Ġa Ġlay Ġperson 's Ġattention Ġcan Ġbe Ġeasily Ġconcentrated Ġto Ġa Ġsingle , Ġundis puted Ġregion Ġin Ġan Ġimage Ġand Ġ2 ) Ġa Ġlay Ġperson Ġwould Ġfind Ġthe Ġrequested Ġtask Ġeasy Ġto Ġaddress . </s> We Ġemploy Ġfive Ġimage - based Ġfeatures Ġcoming Ġfrom Ġthe Ġsalient Ġobject Ġsub it izing ĠB IB REF 22 Ġ( S OS ) Ġmethod , Ġwhich Ġproduces Ġfive Ġprobabilities Ġthat Ġindicate Ġwhether Ġan Ġimage Ġcontains Ġ0 , Ġ1 , Ġ2 , Ġ3 , Ġor Ġ4 + Ġsalient Ġobjects . ĠInt uitive ly , Ġthe Ġnumber Ġof Ġsalient Ġobjects Ġshows Ġhow Ġmany Ġregions Ġin Ġan Ġimage Ġare Ġcompeting Ġfor Ġan Ġobserver 's Ġattention , Ġand Ġso Ġmay Ġcorrelate Ġwith Ġthe Ġease Ġin Ġidentifying Ġa Ġregion Ġof Ġinterest . ĠMoreover , Ġwe Ġhypothes ize Ġthis Ġfeature Ġwill Ġcapture Ġour Ġobservation Ġfrom Ġthe Ġprevious Ġstudy Ġthat Ġcounting Ġproblems Ġtypically Ġleads Ġto Ġdisagreement Ġfor Ġimages Ġshowing Ġmany Ġobjects , Ġand Ġagreement Ġotherwise . </s> We Ġemploy Ġa Ġ2 , 492 - dimensional Ġfeature Ġvector Ġto Ġrepresent Ġthe Ġquestion - based Ġfeatures . ĠOne Ġfeature Ġis Ġthe Ġnumber Ġof Ġwords Ġin Ġthe Ġquestion . ĠInt uitive ly , Ġa Ġlonger Ġquestion Ġoffers Ġmore Ġinformation Ġand Ġwe Ġhypothes ize Ġadditional Ġinformation Ġmakes Ġa Ġquestion Ġmore Ġprecise . ĠThe Ġremaining Ġfeatures Ġcome Ġfrom Ġtwo Ġone - hot Ġvectors Ġdescribing Ġeach Ġof Ġthe Ġfirst Ġtwo Ġwords Ġin Ġthe Ġquestion . ĠEach Ġone - hot Ġvector Ġis Ġcreated Ġusing Ġthe Ġlearned Ġvoc ab ular ies Ġthat Ġdefine Ġall Ġpossible Ġwords Ġat Ġthe Ġfirst Ġand Ġsecond Ġword Ġlocation Ġof Ġa Ġquestion Ġrespectively Ġ( using Ġtraining Ġdata , Ġas Ġdescribed Ġin Ġthe Ġnext Ġsection ). ĠInt uitive ly , Ġearly Ġwords Ġin Ġa Ġquestion Ġinform Ġthe Ġtype Ġof Ġanswers Ġthat Ġmight Ġbe Ġpossible Ġand , Ġin Ġturn , Ġpossible Ġreasons / frequency Ġfor Ġanswer Ġdisagreement . ĠFor Ġexample , Ġwe Ġexpect ĠâĢ ľ why Ġis " Ġto Ġregularly Ġelicit Ġmany Ġopinions Ġand Ġso Ġdisagreement . ĠThis Ġintuition Ġabout Ġthe Ġbeginning Ġwords Ġof Ġa Ġquestion Ġis Ġalso Ġsupported Ġby Ġour Ġanalysis Ġin Ġthe Ġprevious Ġsection Ġwhich Ġshows Ġthat Ġdifferent Ġanswer Ġtypes Ġyield Ġdifferent Ġbiases Ġof Ġelic iting Ġanswer Ġagreement Ġversus Ġdisagreement . </s> We Ġleverage Ġa Ġrandom Ġforest Ġclassification Ġmodel ĠB IB REF 23 Ġto Ġpredict Ġan Ġanswer Ġ( dis ) ag reement Ġlabel Ġfor Ġa Ġgiven Ġvisual Ġquestion . ĠThis Ġmodel Ġconsists Ġof Ġan Ġensemble Ġof Ġdecision Ġtree Ġclass ifiers . ĠWe Ġtrain Ġthe Ġsystem Ġto Ġlearn Ġthe Ġunique Ġweighted Ġcombinations Ġof Ġthe Ġaforementioned Ġ2 , 497 Ġfeatures Ġthat Ġeach Ġdecision Ġtree Ġapplies Ġto Ġmake Ġa Ġprediction . ĠAt Ġtest Ġtime , Ġgiven Ġa Ġnovel Ġvisual Ġquestion , Ġthe Ġtrained Ġsystem Ġconverts Ġa Ġ2 , 497 Ġfeature Ġdescriptor Ġof Ġthe Ġvisual Ġquestion Ġinto Ġa Ġfinal Ġprediction Ġthat Ġreflects Ġthe Ġmajority Ġvote Ġprediction Ġfrom Ġthe Ġensemble Ġof Ġdecision Ġtrees . ĠThe Ġsystem Ġreturns Ġthe Ġfinal Ġprediction Ġalong Ġwith Ġa Ġprobability Ġindicating Ġthe Ġsystem 's Ġconfidence Ġin Ġthat Ġprediction . ĠWe Ġemploy Ġthe ĠMat lab Ġimplementation Ġof Ġrandom Ġforests , Ġusing Ġ25 Ġtrees Ġand Ġthe Ġdefault Ġparameters . </s> We Ġnext Ġadapt Ġa ĠV Q A Ġdeep Ġlearning Ġarchitecture ĠB IB REF 24 Ġto Ġlearn Ġthe Ġpredictive Ġcombination Ġof Ġvisual Ġand Ġtextual Ġfeatures . ĠThe Ġquestion Ġis Ġencoded Ġwith Ġa Ġ1024 - dimensional ĠL ST M Ġmodel Ġthat Ġtakes Ġin Ġa Ġone - hot Ġdescriptor Ġof Ġeach Ġword Ġin Ġthe Ġquestion . ĠThe Ġimage Ġis Ġdescribed Ġwith Ġthe Ġ4096 - dimensional Ġoutput Ġfrom Ġthe Ġlast Ġfully Ġconnected Ġlayer Ġof Ġthe ĠConv olution al ĠNeural ĠNetwork Ġ( CNN ), ĠV GG 16 ĠB IB REF 25 Ġ. ĠThe Ġsystem Ġperforms Ġan Ġelement - wise Ġmultiplication Ġof Ġthe Ġimage Ġand Ġquestion Ġfeatures , Ġafter Ġlin early Ġtransforming Ġthe Ġimage Ġdescriptor Ġto Ġ1024 Ġdimensions . ĠThe Ġfinal Ġlayer Ġof Ġthe Ġarchitecture Ġis Ġa Ġsoft max Ġlayer . </s> We Ġtrain Ġthe Ġsystem Ġto Ġpredict Ġ( dis ) ag reement Ġlabels Ġwith Ġtraining Ġexamples , Ġwhere Ġeach Ġexample Ġincludes Ġan Ġimage Ġand Ġquestion . ĠAt Ġtest Ġtime , Ġgiven Ġa Ġnovel Ġvisual Ġquestion , Ġthe Ġsystem Ġoutputs Ġan Ġun normal ized Ġlog Ġprobability Ġindicating Ġits Ġbelief Ġin Ġboth Ġthe Ġagreement Ġand Ġdisagreement Ġlabel . ĠFor Ġour Ġsystem 's Ġprediction , Ġwe Ġconvert Ġthe Ġbelief Ġin Ġthe Ġdisagreement Ġlabel Ġinto Ġa Ġnormalized Ġprobability . ĠConsequently , Ġpredicted Ġvalues Ġrange Ġfrom Ġ0 Ġto Ġ1 Ġwith Ġlower Ġvalues Ġreflecting Ġgreater Ġlikelihood Ġfor Ġcrowd Ġagreement . </s> Analysis Ġof ĠPrediction ĠSystem </s> We Ġnow Ġdescribe Ġour Ġstudies Ġto Ġassess Ġthe Ġpredictive Ġpower Ġof Ġour Ġclassification Ġsystems Ġto Ġdecide Ġwhether Ġvisual Ġquestions Ġwill Ġlead Ġto Ġanswer Ġ( dis ) ag reement . </s> We Ġcapitalize Ġon Ġtoday 's Ġlargest Ġvisual Ġquestion Ġanswering Ġdataset ĠB IB REF 2 Ġto Ġevaluate Ġour Ġprediction Ġsystem , Ġwhich Ġincludes Ġ369 , 86 1 Ġvisual Ġquestions Ġabout Ġreal Ġimages . ĠOf Ġthese , Ġ248 , 349 Ġvisual Ġquestions Ġ( i . e ., ĠTraining Ġquestions Ġ2015 Ġv 1 . 0 ) Ġare Ġkept Ġfor Ġfor Ġtraining Ġand Ġthe Ġremaining Ġ121 , 512 Ġvisual Ġquestions Ġ( i . e ., ĠVal idation Ġquestions Ġ2015 Ġv 1 . 0 ) Ġare Ġemployed Ġfor Ġtesting Ġour Ġclassification Ġsystem . ĠThis Ġseparation Ġof Ġtraining Ġand Ġtesting Ġsamples Ġenables Ġus Ġto Ġestimate Ġhow Ġwell Ġa Ġclass ifier Ġwill Ġgeneral ize Ġwhen Ġapplied Ġto Ġan Ġunseen , Ġindependent Ġset Ġof Ġvisual Ġquestions . </s> To Ġour Ġknowledge , Ġno Ġprior Ġwork Ġhas Ġdirectly Ġaddressed Ġpredicting Ġanswer Ġ( dis ) ag reement Ġfor Ġvisual Ġquestions . ĠTherefore , Ġwe Ġemploy Ġas Ġa Ġbaseline Ġa Ġrelated ĠV Q A Ġalgorithm ĠB IB REF 24 Ġ, ĠB IB REF 2 Ġwhich Ġproduces Ġfor Ġa Ġgiven Ġvisual Ġquestion Ġan Ġanswer Ġwith Ġa Ġconfidence Ġscore . ĠThis Ġsystem Ġparallels Ġthe Ġdeep Ġlearning Ġarchitecture Ġwe Ġadapt . ĠHowever , Ġit Ġpredicts Ġthe Ġsystem 's Ġuncertainty Ġin Ġits Ġown Ġanswer , Ġwhereas Ġwe Ġare Ġinterested Ġin Ġthe Ġhumans ' Ġcollective Ġdisagreement Ġon Ġthe Ġanswer . ĠStill , Ġit Ġis Ġa Ġuseful Ġbaseline Ġto Ġsee Ġif Ġan Ġexisting Ġalgorithm Ġcould Ġserve Ġour Ġpurpose . </s> We Ġevaluate Ġthe Ġpredictive Ġpower Ġof Ġthe Ġclassification Ġsystems Ġbased Ġon Ġeach Ġclass ifier 's Ġpredictions Ġfor Ġthe Ġ121 , 512 Ġvisual Ġquestions Ġin Ġthe Ġtest Ġdataset . ĠWe Ġfirst Ġshow Ġperformance Ġof Ġthe Ġbaseline Ġand Ġour Ġtwo Ġprediction Ġsystems Ġusing Ġprecision - rec all Ġcurves . ĠThe Ġgoals Ġare Ġto Ġachieve Ġa Ġhigh Ġprecision , Ġto Ġminimize Ġwasting Ġcrowd Ġeffort Ġwhen Ġtheir Ġefforts Ġwill Ġbe Ġredundant , Ġand Ġa Ġhigh Ġrecall , Ġto Ġavoid Ġmissing Ġout Ġon Ġcollecting Ġthe Ġdiversity Ġof Ġaccepted Ġanswers Ġfrom Ġa Ġcrowd . ĠWe Ġalso Ġreport Ġthe Ġaverage Ġprecision Ġ( AP ), Ġwhich Ġindicates Ġthe Ġarea Ġunder Ġa Ġprecision - rec all Ġcurve . ĠAP Ġvalues Ġrange Ġfrom Ġ0 Ġto Ġ1 Ġwith Ġbetter - performing Ġprediction Ġsystems Ġhaving Ġlarger Ġvalues . </s> Figure ĠFIG REF 8 Ġa Ġshows Ġprecision - rec all Ġcurves Ġfor Ġall Ġprediction Ġsystems . ĠBoth Ġour Ġproposed Ġclassification Ġsystems Ġoutper form Ġthe ĠV Q A ĠAl gorithm ĠB IB REF 2 Ġbaseline ; Ġe . g ., ĠO urs Ġ- ĠRF Ġyields Ġa Ġ12 Ġpercentage Ġpoint Ġimprovement Ġwith Ġrespect Ġto ĠAP . ĠThis Ġis Ġinteresting Ġbecause Ġit Ġshows Ġthere Ġis Ġvalue Ġin Ġlearning Ġthe Ġdisagreement Ġtask Ġspecifically , Ġrather Ġthan Ġemploying Ġan Ġalgorithm 's Ġconfidence Ġin Ġits Ġanswers . ĠMore Ġgenerally , Ġour Ġresults Ġdemonstrate Ġit Ġis Ġpossible Ġto Ġpredict Ġwhether Ġa Ġcrowd Ġwill Ġagree Ġon Ġa Ġsingle Ġanswer Ġfrom Ġa Ġgiven Ġimage Ġand Ġassociated Ġquestion . ĠDespite Ġthe Ġsignificant Ġvariety Ġof Ġquestions Ġand Ġimage Ġcontent , Ġand Ġdespite Ġthe Ġvariety Ġof Ġreasons Ġfor Ġwhich Ġthe Ġcrowd Ġcan Ġdisagree , Ġour Ġlearned Ġmodel Ġis Ġable Ġto Ġproduce Ġquite Ġaccurate Ġresults . </s> We Ġobserve Ġour ĠRandom ĠForest Ġclass ifier Ġoutper forms Ġour Ġdeep Ġlearning Ġclass ifier ; Ġe . g ., ĠO urs : ĠRF Ġyields Ġa Ġthree Ġpercentage Ġpoint Ġimprovement Ġwith Ġrespect Ġto ĠAP Ġwhile Ġconsistently Ġyielding Ġimproved Ġprecision - rec all Ġvalues Ġover ĠO urs : ĠL ST M - CNN Ġ( Figure ĠFIG REF 8 Ġa ). ĠIn Ġgeneral , Ġdeep Ġlearning Ġsystems Ġhold Ġpromise Ġto Ġreplace Ġhand crafted Ġfeatures Ġto Ġpick Ġout Ġthe Ġdiscrim in ative Ġfeatures . ĠOur Ġbas elines Ġhighlight Ġa Ġpossible Ġvalue Ġin Ġdeveloping Ġa Ġdifferent Ġdeep Ġlearning Ġarchitecture Ġfor Ġthe Ġproblem Ġof Ġlearning Ġanswer Ġdisagreement Ġthan Ġapplied Ġfor Ġpredicting Ġanswers Ġto Ġvisual Ġquestions . </s> We Ġshow Ġexamples Ġof Ġprediction Ġresults Ġwhere Ġour Ġtop - performing ĠRF Ġclass ifier Ġmakes Ġits Ġmost Ġconfident Ġpredictions Ġ( Figure ĠFIG REF 8 Ġb ). ĠIn Ġthese Ġexamples , Ġthe Ġpredictor Ġexpects Ġhuman Ġagreement Ġfor ĠâĢ ľ what Ġroom ... Ġ ?" Ġvisual Ġquestions Ġand Ġdisagreement Ġfor ĠâĢ ľ why ... Ġ ?" Ġvisual Ġquestions . ĠThese Ġexamples Ġhighlight Ġthat Ġthe Ġclass ifier Ġmay Ġhave Ġa Ġstrong Ġlanguage Ġprior Ġtowards Ġmaking Ġpredictions , Ġas Ġwe Ġwill Ġdiscuss Ġin Ġthe Ġnext Ġsection . </s> We Ġnow Ġexplore Ġwhat Ġmakes Ġa Ġvisual Ġquestion Ġlead Ġto Ġcrowd Ġanswer Ġagreement Ġversus Ġdisagreement . ĠWe Ġexamine Ġthe Ġinfluence Ġof Ġwhether Ġvisual Ġquestions Ġlead Ġto Ġthe Ġthree Ġtypes Ġof Ġanswers Ġ( âĢ ľ yes / no ", ĠâĢ ľ number ", ĠâĢ ľ other ") Ġfor Ġboth Ġour Ġrandom Ġforest Ġ( RF ) Ġand Ġdeep Ġlearning Ġ( DL ) Ġclassification Ġsystems . ĠWe Ġenrich Ġour Ġanalysis Ġby Ġexamining Ġthe Ġpredictive Ġperformance Ġof Ġboth Ġclass ifiers Ġwhen Ġthey Ġare Ġtrained Ġand Ġtested Ġexclusively Ġwith Ġimage Ġand Ġquestion Ġfeatures Ġrespectively . ĠFigure ĠFIG REF 9 Ġshows Ġprecision - rec all Ġcurves Ġfor Ġboth Ġclassification Ġsystems Ġwith Ġquestion Ġfeatures Ġalone Ġ( Q ), Ġimage Ġfeatures Ġalone Ġ( I ), Ġand Ġboth Ġquestion Ġand Ġimage Ġfeatures Ġtogether Ġ( Q + I ). </s> When Ġcomparing ĠAP Ġscores Ġ( Figure ĠFIG REF 9 Ġ), Ġwe Ġobserve Ġour ĠQ + I Ġpredict ors Ġyield Ġthe Ġgreatest Ġpredictive Ġperformance Ġfor Ġvisual Ġquestions Ġthat Ġlead Ġto ĠâĢ ľ other " Ġanswers , Ġfollowed Ġby ĠâĢ ľ number " Ġanswers , Ġand Ġfinally ĠâĢ ľ yes / no " Ġanswers . ĠOne Ġpossible Ġreason Ġfor Ġthis Ġfinding Ġis Ġthat Ġthe Ġquestion Ġwording Ġstrongly Ġdrives Ġwhether Ġa Ġcrowd Ġwill Ġdisagree Ġfor ĠâĢ ľ other " Ġvisual Ġquestions , Ġwhereas Ġsome Ġnotion Ġof Ġcommon Ġsense Ġmay Ġbe Ġrequired Ġto Ġlearn Ġwhether Ġa Ġcrowd Ġwill Ġagree Ġfor ĠâĢ ľ yes / no " Ġvisual Ġquestions Ġ( e . g ., ĠFigure ĠFIG REF 7 Ġa Ġvs ĠFigure ĠFIG REF 7 Ġg ). </s> We Ġobserve Ġthat Ġquestion - based Ġfeatures Ġyield Ġgreater Ġpredictive Ġperformance Ġthan Ġimage - based Ġfeatures Ġfor Ġall Ġvisual Ġquestions , Ġwhen Ġcomparing ĠAP Ġscores Ġfor ĠQ Ġand ĠI Ġclassification Ġresults Ġ( Figure ĠFIG REF 9 Ġ). ĠIn Ġfact , Ġimage Ġfeatures Ġcontribute Ġto Ġperformance Ġimprovements Ġonly Ġfor Ġour Ġrandom Ġforest Ġclass ifier Ġfor Ġvisual Ġquestions Ġthat Ġlead Ġto ĠâĢ ľ number " Ġanswers , Ġas Ġillustrated Ġby Ġcomparing ĠAP Ġscores Ġfor ĠOur ĠRF : ĠQ + I Ġand ĠOur ĠRF : ĠQ Ġ( Figure ĠFIG REF 9 Ġb ). ĠOur Ġoverall Ġfinding Ġthat Ġmost Ġof Ġthe Ġpredictive Ġpower Ġstems Ġfrom Ġlanguage - based Ġfeatures Ġparallels Ġfeature Ġanalysis Ġfindings Ġin Ġthe Ġautomated ĠV Q A Ġliterature ĠB IB REF 2 Ġ, ĠB IB REF 21 Ġ. ĠThis Ġdoes Ġnot Ġmean , Ġhowever , Ġthat Ġthe Ġimage Ġcontent Ġis Ġnot Ġpredictive . ĠFurther Ġwork Ġimproving Ġvisual Ġcontent Ġcues Ġfor ĠV Q A Ġagreement Ġis Ġwarranted . </s> Our Ġfindings Ġsuggest Ġthat Ġour ĠRandom ĠForest Ġclass ifier 's Ġoverall Ġadvantage Ġover Ġour Ġdeep Ġlearning Ġsystem Ġarises Ġbecause Ġof Ġcounting Ġquestions , Ġas Ġindicated Ġby Ġhigher ĠAP Ġscores Ġ( Figure ĠFIG REF 9 Ġ). ĠFor Ġexample , Ġthe Ġadvantage Ġof Ġthe Ġinitial Ġhigher Ġprecision Ġ( Figure ĠFIG REF 8 Ġa ; ĠO urs : ĠRF Ġvs ĠO urs : ĠDL ) Ġis Ġalso Ġobserved Ġfor Ġcounting Ġquestions Ġ( Figure ĠFIG REF 9 Ġb ; ĠO urs : ĠRF Ġ- ĠQ + I Ġvs ĠO urs : ĠDL Ġ- ĠQ + I ). ĠWe Ġhypothes ize Ġthis Ġadvantage Ġarises Ġdue Ġto Ġthe Ġstrength Ġof Ġthe ĠRandom ĠForest Ġclass ifier Ġin Ġpairing Ġthe Ġquestion Ġprior Ġ( âĢ ľ How Ġmany ?" ) Ġwith Ġthe Ġimage - based ĠSOS Ġfeatures Ġthat Ġindicates Ġthe Ġnumber Ġof Ġobjects Ġin Ġan Ġimage . ĠSpecifically , Ġwe Ġexpect ĠâĢ ľ how Ġmany " Ġto Ġlead Ġto Ġagreement Ġonly Ġfor Ġsmall Ġcounting Ġproblems . </s> Capt uring ĠAnswer ĠDiversity Ġwith ĠLess ĠEff ort </s> We Ġnext Ġpresent Ġa Ġnovel Ġresource Ġallocation Ġsystem Ġfor Ġefficiently Ġcapturing Ġthe Ġdiversity Ġof Ġtrue Ġanswers Ġfor Ġa Ġbatch Ġof Ġvisual Ġquestions . ĠToday 's Ġstatus Ġquo Ġis Ġto Ġeither Ġuniformly Ġcollect ĠIN LINE FORM 0 Ġanswers Ġfor Ġevery Ġvisual Ġquestion ĠB IB REF 2 Ġor Ġcollect Ġmultiple Ġanswers Ġwhere Ġthe Ġnumber Ġis Ġdetermined Ġby Ġexternal Ġcrowds ourcing Ġconditions ĠB IB REF 0 Ġ. ĠOur Ġsystem Ġinstead Ġspends Ġa Ġhuman Ġbudget Ġby Ġpredicting Ġthe Ġnumber Ġof Ġanswers Ġto Ġcollect Ġfor Ġeach Ġvisual Ġquestion Ġbased Ġon Ġwhether Ġmultiple Ġhuman Ġanswers Ġare Ġpredicted Ġto Ġbe Ġredundant . </s> Answer ĠCollection ĠSystem </s> Supp ose Ġwe Ġhave Ġa Ġbudget ĠIN LINE FORM 0 Ġwhich Ġwe Ġcan Ġallocate Ġto Ġcollect Ġextra Ġanswers Ġfor Ġa Ġsubset Ġof Ġvisual Ġquestions . ĠOur Ġsystem Ġautomatically Ġdecides Ġto Ġwhich Ġvisual Ġquestions Ġto Ġallocate Ġthe ĠâĢ ľ extra " Ġanswers Ġin Ġorder Ġto Ġmaximize Ġcaptured Ġanswer Ġdiversity Ġfor Ġall Ġvisual Ġquestions . </s> The Ġaim Ġof Ġour Ġsystem Ġis Ġto Ġacc rue Ġadditional Ġcosts Ġand Ġdelays Ġfrom Ġcollecting Ġextra Ġanswers Ġonly Ġwhen Ġextra Ġresponses Ġwill Ġprovide Ġmore Ġinformation . ĠTowards Ġthis Ġaim , Ġour Ġsystem Ġinvolves Ġthree Ġsteps Ġto Ġcollect Ġanswers Ġfor Ġall ĠIN LINE FORM 0 Ġvisual Ġquestions Ġ( Figure ĠFIG REF 11 Ġa ). ĠFirst , Ġthe Ġsystem Ġapplies Ġour Ġtop - performing Ġrandom Ġforest Ġclass ifier Ġto Ġevery Ġvisual Ġquestion Ġin Ġthe Ġbatch . ĠThen , Ġthe Ġsystem Ġranks Ġthe ĠIN LINE FORM 1 Ġvisual Ġquestions Ġbased Ġon Ġpredicted Ġscores Ġfrom Ġthe Ġclass ifier , Ġfrom Ġvisual Ġquestions Ġmost Ġconfidently Ġpredicted Ġto Ġlead Ġto Ġanswer ĠâĢ ľ ag reement " Ġfrom Ġa Ġcrowd Ġto Ġthose Ġmost Ġconfidently Ġpredicted Ġto Ġlead Ġto Ġanswer ĠâĢ ľ dis ag reement " Ġfrom Ġa Ġcrowd . ĠFinally , Ġthe Ġsystem Ġsolic its Ġmore Ġ( ĠIN LINE FORM 2 Ġ) Ġhuman Ġanswers Ġfor Ġthe ĠIN LINE FORM 3 Ġvisual Ġquestions Ġpredicted Ġto Ġreflect Ġthe Ġgreatest Ġlikelihood Ġfor Ġcrowd Ġdisagreement Ġand Ġfewer Ġ( ĠIN LINE FORM 4 Ġ) Ġhuman Ġanswers Ġfor Ġthe Ġremaining Ġvisual Ġquestions . ĠMore Ġdetails Ġbelow . </s> Analysis Ġof ĠAnswer ĠCollection ĠSystem </s> We Ġnow Ġdescribe Ġour Ġstudies Ġto Ġassess Ġthe Ġbenefit Ġof Ġour Ġallocation Ġsystem Ġto Ġreduce Ġhuman Ġeffort Ġto Ġcapture Ġthe Ġdiversity Ġof Ġall Ġanswers Ġto Ġvisual Ġquestions . </s> We Ġevaluate Ġthe Ġimpact Ġof Ġactively Ġall ocating Ġextra Ġhuman Ġeffort Ġto Ġanswer Ġvisual Ġquestions Ġas Ġa Ġfunction Ġof Ġthe Ġavailable Ġbudget Ġof Ġhuman Ġeffort . ĠSpecifically , Ġfor Ġa Ġrange Ġof Ġbudget Ġlevels , Ġwe Ġcompute Ġthe Ġtotal Ġmeasured Ġanswer Ġdiversity Ġ( as Ġdefined Ġbelow ) Ġresulting Ġfor Ġthe Ġbatch Ġof Ġvisual Ġquestions . ĠThe Ġgoal Ġis Ġto Ġcapture Ġa Ġlarge Ġamount Ġof Ġanswer Ġdiversity Ġwith Ġlittle Ġhuman Ġeffort . </s> We Ġconduct Ġour Ġstudies Ġon Ġthe Ġ121 , 512 Ġtest Ġvisual Ġquestions Ġabout Ġreal Ġimages Ġ( i . e ., ĠVal idation Ġquestions Ġ2015 Ġv 1 . 0 ). ĠFor Ġeach Ġvisual Ġquestion , Ġwe Ġestablish Ġthe Ġset Ġof Ġtrue Ġanswers Ġas Ġall Ġunique Ġanswers Ġwhich Ġare Ġobserved Ġat Ġleast Ġtwice Ġin Ġthe Ġ10 Ġcrowds ourced Ġanswers Ġper Ġvisual Ġquestion . ĠWe Ġrequire Ġagreement Ġby Ġtwo Ġworkers Ġto Ġavoid Ġthe Ġpossibility Ġthat ĠâĢ ľ care less / sp am " Ġanswers Ġare Ġtreated Ġas Ġground Ġtruth . </s> We Ġcollect Ġeither Ġthe Ġminimum Ġof ĠIN LINE FORM 0 Ġanswer Ġper Ġvisual Ġquestion Ġor Ġthe Ġmaximum Ġof ĠIN LINE FORM 1 Ġanswers Ġper Ġvisual Ġquestion . ĠOur Ġnumber Ġof Ġanswers Ġroughly Ġalign s Ġwith Ġexisting Ġcrowd - powered ĠV Q A Ġsystems , Ġfor Ġexample Ġwith ĠViz W iz , ĠâĢ ľ On Ġaverage , Ġparticipants Ġreceived Ġ3 . 3 Ġ( SD = 1 . 8 ) Ġanswers Ġfor Ġeach Ġquestion " ĠB IB REF 0 Ġ. ĠOur Ġmaximum Ġnumber Ġof Ġanswers Ġalso Ġsupports Ġthe Ġpossibility Ġof Ġcapturing Ġthe Ġmaximum Ġof Ġthree Ġunique , Ġvalid Ġanswers Ġtypically Ġobserved Ġin Ġpractice Ġ( rec all Ġstudy Ġabove ). ĠWhile Ġmore Ġelaborate Ġschemes Ġfor Ġdistributing Ġresponses Ġmay Ġbe Ġpossible , Ġwe Ġwill Ġshow Ġthis Ġapproach Ġalready Ġproves Ġquite Ġeffective Ġin Ġour Ġexperiments . ĠWe Ġsimulate Ġanswer Ġcollection Ġby Ġrandomly Ġselecting Ġanswers Ġfrom Ġthe Ġ10 Ġcrowd Ġanswers Ġper Ġvisual Ġquestion . </s> We Ġcompare Ġour Ġapproach Ġto Ġthe Ġfollowing Ġbas elines : </s> As Ġin Ġthe Ġprevious Ġsection , Ġwe Ġleverage Ġthe Ġoutput Ġconfidence Ġscore Ġfrom Ġthe Ġpublicly - shared Ġmodel ĠB IB REF 24 Ġlearned Ġfrom Ġa ĠL ST M - CNN Ġdeep Ġlearning Ġarchitecture Ġto Ġrank Ġthe Ġorder Ġof Ġpriority Ġfor Ġvisual Ġquestions Ġto Ġreceive Ġredundancy . </s> The Ġsystem Ġrandomly Ġprior itiz es Ġwhich Ġimages Ġreceive Ġredundancy . ĠThis Ġpredictor Ġillustrates Ġthe Ġbest Ġa Ġuser Ġcan Ġachieve Ġtoday Ġwith Ġcrowd - powered Ġsystems ĠB IB REF 0 Ġ, ĠB IB REF 5 Ġor Ġwith Ġcurrent Ġdataset Ġcollection Ġmethods ĠB IB REF 2 Ġ, ĠB IB REF 4 Ġ. </s> We Ġquantify Ġthe Ġtotal Ġdiversity Ġof Ġanswers Ġcaptured Ġby Ġa Ġresource Ġallocation Ġsystem Ġfor Ġa Ġbatch Ġof Ġvisual Ġquestions ĠIN LINE FORM 0 Ġas Ġfollows : ĠDIS PLAY FORM 0 </s> where ĠIN LINE FORM 0 Ġrepresents Ġthe Ġset Ġof Ġall Ġtrue Ġanswers Ġfor Ġthe ĠIN LINE FORM 1 Ġ- th Ġvisual Ġquestion , ĠIN LINE FORM 2 Ġrepresents Ġthe Ġset Ġof Ġunique Ġanswers Ġcaptured Ġin Ġthe ĠIN LINE FORM 3 Ġanswers Ġcollected Ġfor Ġthe ĠIN LINE FORM 4 Ġ- th Ġvisual Ġquestion , Ġand ĠIN LINE FORM 5 Ġrepresents Ġthe Ġset Ġof Ġunique Ġanswers Ġcaptured Ġin Ġthe ĠIN LINE FORM 6 Ġanswers Ġcollected Ġfor Ġthe ĠIN LINE FORM 7 Ġ- th Ġvisual Ġquestion . ĠGiven Ġno Ġextra Ġhuman Ġbudget , Ġtotal Ġdiversity Ġcomes Ġfrom Ġthe Ġsecond Ġterm Ġwhich Ġindicates Ġthe Ġdiversity Ġcaptured Ġwhen Ġonly ĠIN LINE FORM 8 Ġanswers Ġare Ġcollected Ġfor Ġevery Ġvisual Ġquestion . ĠGiven Ġa Ġmaximum Ġavailable Ġextra Ġhuman Ġbudget Ġ( ĠIN LINE FORM 9 Ġ), Ġtotal Ġdiversity Ġcomes Ġfrom Ġthe Ġfirst Ġterm Ġwhich Ġindicates Ġthe Ġdiversity Ġcaptured Ġwhen ĠIN LINE FORM 10 Ġanswers Ġare Ġcollected Ġfor Ġevery Ġvisual Ġquestion . ĠGiven Ġa Ġpartial Ġextra Ġhuman Ġbudget Ġ( ĠIN LINE FORM 11 Ġ), Ġthe Ġaim Ġis Ġto Ġhave Ġperfect Ġpredictions Ġsuch Ġthat Ġthe Ġminimum Ġnumber Ġof Ġanswers Ġ( ĠIN LINE FORM 12 Ġ) Ġare Ġallocated Ġonly Ġfor Ġvisual Ġquestions Ġwith Ġone Ġtrue Ġanswer Ġso Ġthat Ġall Ġdiverse Ġanswers Ġare Ġsafely Ġcaptured . </s> We Ġmeasure Ġdiversity Ġper Ġvisual Ġquestion Ġas Ġthe Ġnumber Ġof Ġall Ġtrue Ġanswers Ġcollected Ġper Ġvisual Ġquestion Ġ( ĠIN LINE FORM 0 Ġ). ĠL arger Ġvalues Ġreflect Ġgreater Ġcaptured Ġdiversity . ĠThe Ġmotivation Ġfor Ġthis Ġmeasure Ġis Ġto Ġonly Ġgive Ġtotal Ġcredit Ġto Ġvisual Ġquestions Ġwhen Ġall Ġvalid , Ġunique Ġhuman Ġanswers Ġare Ġcollected . </s> Our Ġsystem Ġconsistently Ġoffers Ġsignificant Ġgains Ġover Ġtoday 's Ġstatus Ġquo Ġapproach Ġ( Figure ĠFIG REF 11 Ġb ). ĠFor Ġexample , Ġour Ġsystem Ġacceler ates Ġthe Ġcollection Ġof Ġ70 % Ġof Ġthe Ġdiversity Ġby Ġ21 % Ġover Ġthe ĠStatus ĠQu o Ġbaseline . ĠIn Ġaddition , Ġour Ġsystem Ġacceler ates Ġthe Ġcollection Ġof Ġthe Ġ82 % Ġof Ġdiversity Ġone Ġwould Ġobserve Ġwith ĠViz W iz Ġby Ġ23 % Ġ( i . e ., Ġaverage Ġof Ġ3 . 3 Ġanswers Ġper Ġvisual Ġquestion ). ĠIn Ġabsolute Ġterms , Ġthis Ġmeans Ġeliminating Ġthe Ġcollection Ġof Ġ92 , 180 Ġanswers Ġwith Ġno Ġloss Ġto Ġcaptured Ġanswer Ġdiversity . ĠThis Ġtranslates Ġto Ġeliminating Ġ19 Ġ40 - hour Ġwork Ġweeks Ġand Ġsaving Ġover Ġ$ 1800 , Ġassuming Ġworkers Ġare Ġpaid Ġ$ 0 . 02 Ġper Ġanswer Ġand Ġtake Ġ30 Ġseconds Ġto Ġanswer Ġa Ġvisual Ġquestion . ĠOur Ġapproach Ġfills Ġan Ġimportant Ġgap Ġin Ġthe Ġcrowds ourcing Ġanswer Ġcollection Ġliterature Ġfor Ġtargeting Ġthe Ġallocation Ġof Ġextra Ġanswers Ġonly Ġto Ġvisual Ġquestions Ġwhere Ġa Ġdiversity Ġof Ġanswers Ġis Ġexpected . </s> Figure ĠFIG REF 11 Ġb Ġalso Ġillustrates Ġthe Ġadvantage Ġof Ġour Ġsystem Ġover Ġa Ġrelated ĠV Q A Ġalgorithm ĠB IB REF 2 Ġfor Ġour Ġnovel Ġapplication Ġof Ġcost - sensitive Ġanswer Ġcollection Ġfrom Ġa Ġcrowd . ĠAs Ġobserved , Ġrelying Ġon Ġan Ġalgorithm 's Ġconfidence Ġin Ġits Ġanswer Ġoffers Ġa Ġvaluable Ġindicator Ġover Ġtoday 's Ġstatus Ġquo Ġof Ġpassively Ġbudget ing . ĠWhile Ġwe Ġacknowledge Ġthis Ġmethod Ġis Ġnot Ġintended Ġfor Ġour Ġtask Ġspecifically , Ġit Ġstill Ġserves Ġas Ġan Ġimportant Ġbaseline Ġ( as Ġdiscussed Ġabove ). ĠWe Ġattribute Ġthe Ġfurther Ġperformance Ġgains Ġof Ġour Ġprediction Ġsystem Ġto Ġit Ġdirectly Ġpredicting Ġwhether Ġhumans Ġwill Ġdisagree Ġrather Ġthan Ġpredicting Ġa Ġproperty Ġof Ġa Ġspecific Ġalgorithm Ġ( e . g ., Ġconfidence Ġof Ġthe ĠAnt ol Ġet Ġal . Ġalgorithm Ġin Ġits Ġanswer Ġprediction ). </s> Con clusions </s> We Ġproposed Ġa Ġnew Ġproblem Ġof Ġpredicting Ġwhether Ġdifferent Ġpeople Ġwould Ġanswer Ġwith Ġthe Ġsame Ġresponse Ġto Ġthe Ġsame Ġvisual Ġquestion . ĠTowards Ġmotivating Ġthe Ġpractical Ġimplications Ġfor Ġthis Ġproblem , Ġwe Ġanalyzed Ġnearly Ġhalf Ġa Ġmillion Ġvisual Ġquestions Ġand Ġdemonstrated Ġthere Ġis Ġnearly Ġa Ġ50 / 50 Ġsplit Ġbetween Ġvisual Ġquestions Ġthat Ġlead Ġto Ġanswer Ġagreement Ġversus Ġdisagreement . ĠWe Ġobserved Ġthat Ġcrowd Ġdisagreement Ġarose Ġfor Ġvarious Ġtypes Ġof Ġanswers Ġ( yes / no , Ġcounting , Ġother ) Ġfor Ġmany Ġdifferent Ġreasons . ĠWe Ġnext Ġproposed Ġa Ġsystem Ġthat Ġautomatically Ġpredicts Ġwhether Ġa Ġvisual Ġquestion Ġwill Ġlead Ġto Ġa Ġsingle Ġversus Ġmultiple Ġanswers Ġfrom Ġa Ġcrowd . ĠOur Ġmethod Ġoutper forms Ġa Ġstrong Ġexisting ĠV Q A Ġsystem Ġlimited Ġto Ġestimating Ġsystem Ġuncertainty Ġrather Ġthan Ġcrowd Ġdisagreement . ĠFinally , Ġwe Ġdemonstrated Ġhow Ġto Ġemploy Ġthe Ġprediction Ġsystem Ġto Ġaccelerate Ġthe Ġcollection Ġof Ġdiverse Ġanswers Ġfrom Ġa Ġcrowd Ġby Ġtypically Ġat Ġleast Ġ20 % Ġover Ġtoday 's Ġstatus Ġquo Ġof Ġfixed Ġredundancy Ġallocation . </s> Acknowled gments </s> The Ġauthors Ġgrate fully Ġacknowledge Ġfunding Ġfrom Ġthe ĠOffice Ġof ĠNaval ĠResearch Ġ( ON R ĠY IP ĠN 0001 4 - 12 - 1 - 0 754 ) Ġand ĠNational ĠScience ĠFoundation Ġ( I IS - 10 65 390 ). ĠWe Ġthank ĠD ines h ĠJay ar aman , ĠYu - Ch uan ĠSu , ĠSu y og ĠJ ain , Ġand ĠChao - Y eh ĠChen Ġfor Ġtheir Ġassistance Ġwith Ġexperiments .
Introduction What would be possible if a person had an oracle that could immediately provide the answer to any question about the visual world? Sight-impaired users could quickly and reliably figure out the denomination of their currency and so whether they spent the appropriate amount for a product BIBREF0 . Hikers could immediately learn about their bug bites and whether to seek out emergency medical care. Pilots could learn how many birds are in their path to decide whether to change course and so avoid costly, life-threatening collisions. These examples illustrate several of the interests from a visual question answering (VQA) system, including tackling problems that involve classification, detection, and counting. More generally, the goal for VQA is to have a single system that can accurately answer any natural language question about an image or video BIBREF1 , BIBREF2 , BIBREF3 . Entangled in the dream of a VQA system is an unavoidable issue that, when asking multiple people a visual question, sometimes they all agree on a single answer while other times they offer different answers (Figure FIGREF1 ). In fact, as we show in the paper, these two outcomes arise in approximately equal proportions in today's largest publicly-shared VQA benchmark that contains over 450,000 visual questions. Figure FIGREF1 illustrates that human disagreements arise for a variety of reasons including different descriptions of the same concept (e.g., “minor" and “underage"), different concepts (e.g., “ghost" and “photoshop"), and irrelevant responses (e.g., “no"). Our goal is to account for whether different people would agree on a single answer to a visual question to improve upon today's VQA systems. We propose multiple prediction systems to automatically decide whether a visual question will lead to human agreement and demonstrate the value of these predictions for a new task of capturing the diversity of all plausible answers with less human effort. Our work is partially inspired by the goal to improve how to employ crowds as the computing power at run-time. Towards satisfying existing users, gaining new users, and supporting a wide range of applications, a crowd-powered VQA system should be low cost, have fast response times, and yield high quality answers. Today's status quo is to assume a fixed number of human responses per visual question and so a fixed cost, delay, and potential diversity of answers for every visual question BIBREF2 , BIBREF0 , BIBREF4 . We instead propose to dynamically solicit the number of human responses based on each visual question. In particular, we aim to accrue additional costs and delays from collecting extra answers only when extra responses are needed to discover all plausible answers. We show in our experiments that our system saves 19 40-hour work weeks and $1800 to answer 121,512 visual questions, compared to today's status quo approach BIBREF0 . Our work is also inspired by the goal to improve how to employ crowds to produce the information needed to train and evaluate automated methods. Specifically, researchers in fields as diverse as computer vision BIBREF2 , computational linguistics BIBREF1 , and machine learning BIBREF3 rely on large datasets to improve their VQA algorithms. These datasets include visual questions and human-supplied answers. Such data is critical for teaching machine learning algorithms how to answer questions by example. Such data is also critical for evaluating how well VQA algorithms perform. In general, “bigger" data is better. Current methods to create these datasets assume a fixed number of human answers per visual question BIBREF2 , BIBREF4 , thereby either compromising on quality by not collecting all plausible answers or cost by collecting additional answers when they are redundant. We offer an economical way to spend a human budget to collect answers from crowd workers. In particular, we aim to actively allocate additional answers only to visual questions likely to have multiple answers. The key contributions of our work are as follows: Paper Overview The remainder of the paper is organized into four sections. We first describe a study where we investigate: 1) How much answer diversity arises for visual questions? and 2) Why do people disagree (Section SECREF4 )? Next, we explore the following two questions: 1) Given a novel visual question, can a machine correctly predict whether multiple independent members of a crowd would supply the same answer? and 2) If so, what insights does our machine-learned system reveal regarding what humans are most likely to agree about (Section SECREF5 )? In the following section, we propose a novel resource allocation system for efficiently capturing the diversity of all answers for a set of visual questions (Section SECREF6 ). Finally, we end with concluding remarks (Section SECREF7 ). VQA - Analysis of Answer (Dis)Agreements Our first aim is to answer the following questions: 1) How much answer diversity arises for visual questions? and 2) Why do people disagree? Visual Question - Predicting If a Crowd Will (Dis)Agree on an Answer We now explore the following two questions: 1) Given a novel visual question, can a machine correctly predict whether multiple independent members of a crowd would supply the same answer? and 2) If so, what insights does our machine-learned system reveal regarding what humans are most likely to agree about? Prediction Systems We pose the prediction task as a binary classification problem. Specifically, given an image and associated question, a system outputs a binary label indicating whether a crowd will agree on the same answer. Our goal is to design a system that can detect which visual questions to assign a disagreement label, regardless of the disagreement cause (e.g., subjectivity, ambiguity, difficulty). We implement both random forest and deep learning classifiers. A visual question is assigned either an answer agreement or disagreement label. To assign labels, we employ 10 crowdsourced answers for each visual question. A visual question is assigned an answer agreement label when there is an exact string match for 9 of the 10 crowdsourced answers (after answer pre-preprocessing, as discussed in the previous section) and an answer disagreement label otherwise. Our rationale is to permit the possibility of up to one “careless/spam" answer per visual question. The outcome of our labeling scheme is that a disagreement label is agnostic to the specific cause of disagreement and rather represents the many causes (described above). For our first system, we use domain knowledge to guide the learning process. We compile a set of features that we hypothesize inform whether a crowd will arrive at an undisputed, single answer. Then we apply a machine learning tool to reveal the significance of each feature. We propose features based on the observation that answer agreement often arises when 1) a lay person's attention can be easily concentrated to a single, undisputed region in an image and 2) a lay person would find the requested task easy to address. We employ five image-based features coming from the salient object subitizing BIBREF22 (SOS) method, which produces five probabilities that indicate whether an image contains 0, 1, 2, 3, or 4+ salient objects. Intuitively, the number of salient objects shows how many regions in an image are competing for an observer's attention, and so may correlate with the ease in identifying a region of interest. Moreover, we hypothesize this feature will capture our observation from the previous study that counting problems typically leads to disagreement for images showing many objects, and agreement otherwise. We employ a 2,492-dimensional feature vector to represent the question-based features. One feature is the number of words in the question. Intuitively, a longer question offers more information and we hypothesize additional information makes a question more precise. The remaining features come from two one-hot vectors describing each of the first two words in the question. Each one-hot vector is created using the learned vocabularies that define all possible words at the first and second word location of a question respectively (using training data, as described in the next section). Intuitively, early words in a question inform the type of answers that might be possible and, in turn, possible reasons/frequency for answer disagreement. For example, we expect “why is" to regularly elicit many opinions and so disagreement. This intuition about the beginning words of a question is also supported by our analysis in the previous section which shows that different answer types yield different biases of eliciting answer agreement versus disagreement. We leverage a random forest classification model BIBREF23 to predict an answer (dis)agreement label for a given visual question. This model consists of an ensemble of decision tree classifiers. We train the system to learn the unique weighted combinations of the aforementioned 2,497 features that each decision tree applies to make a prediction. At test time, given a novel visual question, the trained system converts a 2,497 feature descriptor of the visual question into a final prediction that reflects the majority vote prediction from the ensemble of decision trees. The system returns the final prediction along with a probability indicating the system's confidence in that prediction. We employ the Matlab implementation of random forests, using 25 trees and the default parameters. We next adapt a VQA deep learning architecture BIBREF24 to learn the predictive combination of visual and textual features. The question is encoded with a 1024-dimensional LSTM model that takes in a one-hot descriptor of each word in the question. The image is described with the 4096-dimensional output from the last fully connected layer of the Convolutional Neural Network (CNN), VGG16 BIBREF25 . The system performs an element-wise multiplication of the image and question features, after linearly transforming the image descriptor to 1024 dimensions. The final layer of the architecture is a softmax layer. We train the system to predict (dis)agreement labels with training examples, where each example includes an image and question. At test time, given a novel visual question, the system outputs an unnormalized log probability indicating its belief in both the agreement and disagreement label. For our system's prediction, we convert the belief in the disagreement label into a normalized probability. Consequently, predicted values range from 0 to 1 with lower values reflecting greater likelihood for crowd agreement. Analysis of Prediction System We now describe our studies to assess the predictive power of our classification systems to decide whether visual questions will lead to answer (dis)agreement. We capitalize on today's largest visual question answering dataset BIBREF2 to evaluate our prediction system, which includes 369,861 visual questions about real images. Of these, 248,349 visual questions (i.e., Training questions 2015 v1.0) are kept for for training and the remaining 121,512 visual questions (i.e., Validation questions 2015 v1.0) are employed for testing our classification system. This separation of training and testing samples enables us to estimate how well a classifier will generalize when applied to an unseen, independent set of visual questions. To our knowledge, no prior work has directly addressed predicting answer (dis)agreement for visual questions. Therefore, we employ as a baseline a related VQA algorithm BIBREF24 , BIBREF2 which produces for a given visual question an answer with a confidence score. This system parallels the deep learning architecture we adapt. However, it predicts the system's uncertainty in its own answer, whereas we are interested in the humans' collective disagreement on the answer. Still, it is a useful baseline to see if an existing algorithm could serve our purpose. We evaluate the predictive power of the classification systems based on each classifier's predictions for the 121,512 visual questions in the test dataset. We first show performance of the baseline and our two prediction systems using precision-recall curves. The goals are to achieve a high precision, to minimize wasting crowd effort when their efforts will be redundant, and a high recall, to avoid missing out on collecting the diversity of accepted answers from a crowd. We also report the average precision (AP), which indicates the area under a precision-recall curve. AP values range from 0 to 1 with better-performing prediction systems having larger values. Figure FIGREF8 a shows precision-recall curves for all prediction systems. Both our proposed classification systems outperform the VQA Algorithm BIBREF2 baseline; e.g., Ours - RF yields a 12 percentage point improvement with respect to AP. This is interesting because it shows there is value in learning the disagreement task specifically, rather than employing an algorithm's confidence in its answers. More generally, our results demonstrate it is possible to predict whether a crowd will agree on a single answer from a given image and associated question. Despite the significant variety of questions and image content, and despite the variety of reasons for which the crowd can disagree, our learned model is able to produce quite accurate results. We observe our Random Forest classifier outperforms our deep learning classifier; e.g., Ours: RF yields a three percentage point improvement with respect to AP while consistently yielding improved precision-recall values over Ours: LSTM-CNN (Figure FIGREF8 a). In general, deep learning systems hold promise to replace handcrafted features to pick out the discriminative features. Our baselines highlight a possible value in developing a different deep learning architecture for the problem of learning answer disagreement than applied for predicting answers to visual questions. We show examples of prediction results where our top-performing RF classifier makes its most confident predictions (Figure FIGREF8 b). In these examples, the predictor expects human agreement for “what room... ?" visual questions and disagreement for “why... ?" visual questions. These examples highlight that the classifier may have a strong language prior towards making predictions, as we will discuss in the next section. We now explore what makes a visual question lead to crowd answer agreement versus disagreement. We examine the influence of whether visual questions lead to the three types of answers (“yes/no", “number", “other") for both our random forest (RF) and deep learning (DL) classification systems. We enrich our analysis by examining the predictive performance of both classifiers when they are trained and tested exclusively with image and question features respectively. Figure FIGREF9 shows precision-recall curves for both classification systems with question features alone (Q), image features alone (I), and both question and image features together (Q+I). When comparing AP scores (Figure FIGREF9 ), we observe our Q+I predictors yield the greatest predictive performance for visual questions that lead to “other" answers, followed by “number" answers, and finally “yes/no" answers. One possible reason for this finding is that the question wording strongly drives whether a crowd will disagree for “other" visual questions, whereas some notion of common sense may be required to learn whether a crowd will agree for “yes/no" visual questions (e.g., Figure FIGREF7 a vs Figure FIGREF7 g). We observe that question-based features yield greater predictive performance than image-based features for all visual questions, when comparing AP scores for Q and I classification results (Figure FIGREF9 ). In fact, image features contribute to performance improvements only for our random forest classifier for visual questions that lead to “number" answers, as illustrated by comparing AP scores for Our RF: Q+I and Our RF: Q (Figure FIGREF9 b). Our overall finding that most of the predictive power stems from language-based features parallels feature analysis findings in the automated VQA literature BIBREF2 , BIBREF21 . This does not mean, however, that the image content is not predictive. Further work improving visual content cues for VQA agreement is warranted. Our findings suggest that our Random Forest classifier's overall advantage over our deep learning system arises because of counting questions, as indicated by higher AP scores (Figure FIGREF9 ). For example, the advantage of the initial higher precision (Figure FIGREF8 a; Ours: RF vs Ours: DL) is also observed for counting questions (Figure FIGREF9 b; Ours: RF - Q+I vs Ours: DL - Q+I). We hypothesize this advantage arises due to the strength of the Random Forest classifier in pairing the question prior (“How many?") with the image-based SOS features that indicates the number of objects in an image. Specifically, we expect “how many" to lead to agreement only for small counting problems. Capturing Answer Diversity with Less Effort We next present a novel resource allocation system for efficiently capturing the diversity of true answers for a batch of visual questions. Today's status quo is to either uniformly collect INLINEFORM0 answers for every visual question BIBREF2 or collect multiple answers where the number is determined by external crowdsourcing conditions BIBREF0 . Our system instead spends a human budget by predicting the number of answers to collect for each visual question based on whether multiple human answers are predicted to be redundant. Answer Collection System Suppose we have a budget INLINEFORM0 which we can allocate to collect extra answers for a subset of visual questions. Our system automatically decides to which visual questions to allocate the “extra" answers in order to maximize captured answer diversity for all visual questions. The aim of our system is to accrue additional costs and delays from collecting extra answers only when extra responses will provide more information. Towards this aim, our system involves three steps to collect answers for all INLINEFORM0 visual questions (Figure FIGREF11 a). First, the system applies our top-performing random forest classifier to every visual question in the batch. Then, the system ranks the INLINEFORM1 visual questions based on predicted scores from the classifier, from visual questions most confidently predicted to lead to answer “agreement" from a crowd to those most confidently predicted to lead to answer “disagreement" from a crowd. Finally, the system solicits more ( INLINEFORM2 ) human answers for the INLINEFORM3 visual questions predicted to reflect the greatest likelihood for crowd disagreement and fewer ( INLINEFORM4 ) human answers for the remaining visual questions. More details below. Analysis of Answer Collection System We now describe our studies to assess the benefit of our allocation system to reduce human effort to capture the diversity of all answers to visual questions. We evaluate the impact of actively allocating extra human effort to answer visual questions as a function of the available budget of human effort. Specifically, for a range of budget levels, we compute the total measured answer diversity (as defined below) resulting for the batch of visual questions. The goal is to capture a large amount of answer diversity with little human effort. We conduct our studies on the 121,512 test visual questions about real images (i.e., Validation questions 2015 v1.0). For each visual question, we establish the set of true answers as all unique answers which are observed at least twice in the 10 crowdsourced answers per visual question. We require agreement by two workers to avoid the possibility that “careless/spam" answers are treated as ground truth. We collect either the minimum of INLINEFORM0 answer per visual question or the maximum of INLINEFORM1 answers per visual question. Our number of answers roughly aligns with existing crowd-powered VQA systems, for example with VizWiz, “On average, participants received 3.3 (SD=1.8) answers for each question" BIBREF0 . Our maximum number of answers also supports the possibility of capturing the maximum of three unique, valid answers typically observed in practice (recall study above). While more elaborate schemes for distributing responses may be possible, we will show this approach already proves quite effective in our experiments. We simulate answer collection by randomly selecting answers from the 10 crowd answers per visual question. We compare our approach to the following baselines: As in the previous section, we leverage the output confidence score from the publicly-shared model BIBREF24 learned from a LSTM-CNN deep learning architecture to rank the order of priority for visual questions to receive redundancy. The system randomly prioritizes which images receive redundancy. This predictor illustrates the best a user can achieve today with crowd-powered systems BIBREF0 , BIBREF5 or with current dataset collection methods BIBREF2 , BIBREF4 . We quantify the total diversity of answers captured by a resource allocation system for a batch of visual questions INLINEFORM0 as follows: DISPLAYFORM0 where INLINEFORM0 represents the set of all true answers for the INLINEFORM1 -th visual question, INLINEFORM2 represents the set of unique answers captured in the INLINEFORM3 answers collected for the INLINEFORM4 -th visual question, and INLINEFORM5 represents the set of unique answers captured in the INLINEFORM6 answers collected for the INLINEFORM7 -th visual question. Given no extra human budget, total diversity comes from the second term which indicates the diversity captured when only INLINEFORM8 answers are collected for every visual question. Given a maximum available extra human budget ( INLINEFORM9 ), total diversity comes from the first term which indicates the diversity captured when INLINEFORM10 answers are collected for every visual question. Given a partial extra human budget ( INLINEFORM11 ), the aim is to have perfect predictions such that the minimum number of answers ( INLINEFORM12 ) are allocated only for visual questions with one true answer so that all diverse answers are safely captured. We measure diversity per visual question as the number of all true answers collected per visual question ( INLINEFORM0 ). Larger values reflect greater captured diversity. The motivation for this measure is to only give total credit to visual questions when all valid, unique human answers are collected. Our system consistently offers significant gains over today's status quo approach (Figure FIGREF11 b). For example, our system accelerates the collection of 70% of the diversity by 21% over the Status Quo baseline. In addition, our system accelerates the collection of the 82% of diversity one would observe with VizWiz by 23% (i.e., average of 3.3 answers per visual question). In absolute terms, this means eliminating the collection of 92,180 answers with no loss to captured answer diversity. This translates to eliminating 19 40-hour work weeks and saving over $1800, assuming workers are paid $0.02 per answer and take 30 seconds to answer a visual question. Our approach fills an important gap in the crowdsourcing answer collection literature for targeting the allocation of extra answers only to visual questions where a diversity of answers is expected. Figure FIGREF11 b also illustrates the advantage of our system over a related VQA algorithm BIBREF2 for our novel application of cost-sensitive answer collection from a crowd. As observed, relying on an algorithm's confidence in its answer offers a valuable indicator over today's status quo of passively budgeting. While we acknowledge this method is not intended for our task specifically, it still serves as an important baseline (as discussed above). We attribute the further performance gains of our prediction system to it directly predicting whether humans will disagree rather than predicting a property of a specific algorithm (e.g., confidence of the Antol et al. algorithm in its answer prediction). Conclusions We proposed a new problem of predicting whether different people would answer with the same response to the same visual question. Towards motivating the practical implications for this problem, we analyzed nearly half a million visual questions and demonstrated there is nearly a 50/50 split between visual questions that lead to answer agreement versus disagreement. We observed that crowd disagreement arose for various types of answers (yes/no, counting, other) for many different reasons. We next proposed a system that automatically predicts whether a visual question will lead to a single versus multiple answers from a crowd. Our method outperforms a strong existing VQA system limited to estimating system uncertainty rather than crowd disagreement. Finally, we demonstrated how to employ the prediction system to accelerate the collection of diverse answers from a crowd by typically at least 20% over today's status quo of fixed redundancy allocation. Acknowledgments The authors gratefully acknowledge funding from the Office of Naval Research (ONR YIP N00014-12-1-0754) and National Science Foundation (IIS-1065390). We thank Dinesh Jayaraman, Yu-Chuan Su, Suyog Jain, and Chao-Yeh Chen for their assistance with experiments.
How is the data used for training annotated?
4a4ce942a7a6efd1fa1d6c91dedf7a89af64b729
How Ġis Ġthe Ġdata Ġused Ġfor Ġtraining Ġannot ated ?
The Ġnumber Ġof Ġredundant Ġanswers Ġto Ġcollect Ġfrom Ġthe Ġcrowd Ġis Ġpredicted Ġto Ġefficiently Ġcapture Ġthe Ġdiversity Ġof Ġall Ġanswers Ġfrom Ġall Ġvisual Ġquestions .
<s> How Ġis Ġthe Ġdata Ġused Ġfor Ġtraining Ġannot ated ? </s> Introduction </s> What Ġwould Ġbe Ġpossible Ġif Ġa Ġperson Ġhad Ġan Ġor acle Ġthat Ġcould Ġimmediately Ġprovide Ġthe Ġanswer Ġto Ġany Ġquestion Ġabout Ġthe Ġvisual Ġworld ? ĠSight - im pa ired Ġusers Ġcould Ġquickly Ġand Ġreliably Ġfigure Ġout Ġthe Ġdenomination Ġof Ġtheir Ġcurrency Ġand Ġso Ġwhether Ġthey Ġspent Ġthe Ġappropriate Ġamount Ġfor Ġa Ġproduct ĠB IB REF 0 Ġ. ĠH ikers Ġcould Ġimmediately Ġlearn Ġabout Ġtheir Ġbug Ġbites Ġand Ġwhether Ġto Ġseek Ġout Ġemergency Ġmedical Ġcare . ĠPil ots Ġcould Ġlearn Ġhow Ġmany Ġbirds Ġare Ġin Ġtheir Ġpath Ġto Ġdecide Ġwhether Ġto Ġchange Ġcourse Ġand Ġso Ġavoid Ġcostly , Ġlife - threatening Ġcollisions . ĠThese Ġexamples Ġillustrate Ġseveral Ġof Ġthe Ġinterests Ġfrom Ġa Ġvisual Ġquestion Ġanswering Ġ( V Q A ) Ġsystem , Ġincluding Ġtackling Ġproblems Ġthat Ġinvolve Ġclassification , Ġdetection , Ġand Ġcounting . ĠMore Ġgenerally , Ġthe Ġgoal Ġfor ĠV Q A Ġis Ġto Ġhave Ġa Ġsingle Ġsystem Ġthat Ġcan Ġaccurately Ġanswer Ġany Ġnatural Ġlanguage Ġquestion Ġabout Ġan Ġimage Ġor Ġvideo ĠB IB REF 1 Ġ, ĠB IB REF 2 Ġ, ĠB IB REF 3 Ġ. </s> Ent angled Ġin Ġthe Ġdream Ġof Ġa ĠV Q A Ġsystem Ġis Ġan Ġunavoidable Ġissue Ġthat , Ġwhen Ġasking Ġmultiple Ġpeople Ġa Ġvisual Ġquestion , Ġsometimes Ġthey Ġall Ġagree Ġon Ġa Ġsingle Ġanswer Ġwhile Ġother Ġtimes Ġthey Ġoffer Ġdifferent Ġanswers Ġ( Figure ĠFIG REF 1 Ġ). ĠIn Ġfact , Ġas Ġwe Ġshow Ġin Ġthe Ġpaper , Ġthese Ġtwo Ġoutcomes Ġarise Ġin Ġapproximately Ġequal Ġproportions Ġin Ġtoday 's Ġlargest Ġpublicly - shared ĠV Q A Ġbenchmark Ġthat Ġcontains Ġover Ġ450 , 000 Ġvisual Ġquestions . ĠFigure ĠFIG REF 1 Ġillustrates Ġthat Ġhuman Ġdisagreements Ġarise Ġfor Ġa Ġvariety Ġof Ġreasons Ġincluding Ġdifferent Ġdescriptions Ġof Ġthe Ġsame Ġconcept Ġ( e . g ., ĠâĢ ľ min or " Ġand ĠâĢ ľ under age "), Ġdifferent Ġconcepts Ġ( e . g ., ĠâĢ ľ ghost " Ġand ĠâĢ ľ phot oshop "), Ġand Ġirrelevant Ġresponses Ġ( e . g ., ĠâĢ ľ no "). </s> Our Ġgoal Ġis Ġto Ġaccount Ġfor Ġwhether Ġdifferent Ġpeople Ġwould Ġagree Ġon Ġa Ġsingle Ġanswer Ġto Ġa Ġvisual Ġquestion Ġto Ġimprove Ġupon Ġtoday 's ĠV Q A Ġsystems . ĠWe Ġpropose Ġmultiple Ġprediction Ġsystems Ġto Ġautomatically Ġdecide Ġwhether Ġa Ġvisual Ġquestion Ġwill Ġlead Ġto Ġhuman Ġagreement Ġand Ġdemonstrate Ġthe Ġvalue Ġof Ġthese Ġpredictions Ġfor Ġa Ġnew Ġtask Ġof Ġcapturing Ġthe Ġdiversity Ġof Ġall Ġplausible Ġanswers Ġwith Ġless Ġhuman Ġeffort . </s> Our Ġwork Ġis Ġpartially Ġinspired Ġby Ġthe Ġgoal Ġto Ġimprove Ġhow Ġto Ġemploy Ġcrowds Ġas Ġthe Ġcomputing Ġpower Ġat Ġrun - time . ĠTowards Ġsatisfying Ġexisting Ġusers , Ġgaining Ġnew Ġusers , Ġand Ġsupporting Ġa Ġwide Ġrange Ġof Ġapplications , Ġa Ġcrowd - powered ĠV Q A Ġsystem Ġshould Ġbe Ġlow Ġcost , Ġhave Ġfast Ġresponse Ġtimes , Ġand Ġyield Ġhigh Ġquality Ġanswers . ĠToday 's Ġstatus Ġquo Ġis Ġto Ġassume Ġa Ġfixed Ġnumber Ġof Ġhuman Ġresponses Ġper Ġvisual Ġquestion Ġand Ġso Ġa Ġfixed Ġcost , Ġdelay , Ġand Ġpotential Ġdiversity Ġof Ġanswers Ġfor Ġevery Ġvisual Ġquestion ĠB IB REF 2 Ġ, ĠB IB REF 0 Ġ, ĠB IB REF 4 Ġ. ĠWe Ġinstead Ġpropose Ġto Ġdynamically Ġsolicit Ġthe Ġnumber Ġof Ġhuman Ġresponses Ġbased Ġon Ġeach Ġvisual Ġquestion . ĠIn Ġparticular , Ġwe Ġaim Ġto Ġacc rue Ġadditional Ġcosts Ġand Ġdelays Ġfrom Ġcollecting Ġextra Ġanswers Ġonly Ġwhen Ġextra Ġresponses Ġare Ġneeded Ġto Ġdiscover Ġall Ġplausible Ġanswers . ĠWe Ġshow Ġin Ġour Ġexperiments Ġthat Ġour Ġsystem Ġsaves Ġ19 Ġ40 - hour Ġwork Ġweeks Ġand Ġ$ 1800 Ġto Ġanswer Ġ121 , 512 Ġvisual Ġquestions , Ġcompared Ġto Ġtoday 's Ġstatus Ġquo Ġapproach ĠB IB REF 0 Ġ. </s> Our Ġwork Ġis Ġalso Ġinspired Ġby Ġthe Ġgoal Ġto Ġimprove Ġhow Ġto Ġemploy Ġcrowds Ġto Ġproduce Ġthe Ġinformation Ġneeded Ġto Ġtrain Ġand Ġevaluate Ġautomated Ġmethods . ĠSpecifically , Ġresearchers Ġin Ġfields Ġas Ġdiverse Ġas Ġcomputer Ġvision ĠB IB REF 2 Ġ, Ġcomputational Ġlingu istics ĠB IB REF 1 Ġ, Ġand Ġmachine Ġlearning ĠB IB REF 3 Ġrely Ġon Ġlarge Ġdatasets Ġto Ġimprove Ġtheir ĠV Q A Ġalgorithms . ĠThese Ġdatasets Ġinclude Ġvisual Ġquestions Ġand Ġhuman - supp lied Ġanswers . ĠSuch Ġdata Ġis Ġcritical Ġfor Ġteaching Ġmachine Ġlearning Ġalgorithms Ġhow Ġto Ġanswer Ġquestions Ġby Ġexample . ĠSuch Ġdata Ġis Ġalso Ġcritical Ġfor Ġevaluating Ġhow Ġwell ĠV Q A Ġalgorithms Ġperform . ĠIn Ġgeneral , ĠâĢ ľ big ger " Ġdata Ġis Ġbetter . ĠCurrent Ġmethods Ġto Ġcreate Ġthese Ġdatasets Ġassume Ġa Ġfixed Ġnumber Ġof Ġhuman Ġanswers Ġper Ġvisual Ġquestion ĠB IB REF 2 Ġ, ĠB IB REF 4 Ġ, Ġthereby Ġeither Ġcompromising Ġon Ġquality Ġby Ġnot Ġcollecting Ġall Ġplausible Ġanswers Ġor Ġcost Ġby Ġcollecting Ġadditional Ġanswers Ġwhen Ġthey Ġare Ġredundant . ĠWe Ġoffer Ġan Ġeconomical Ġway Ġto Ġspend Ġa Ġhuman Ġbudget Ġto Ġcollect Ġanswers Ġfrom Ġcrowd Ġworkers . ĠIn Ġparticular , Ġwe Ġaim Ġto Ġactively Ġallocate Ġadditional Ġanswers Ġonly Ġto Ġvisual Ġquestions Ġlikely Ġto Ġhave Ġmultiple Ġanswers . </s> The Ġkey Ġcontributions Ġof Ġour Ġwork Ġare Ġas Ġfollows : </s> Paper ĠOverview </s> The Ġremainder Ġof Ġthe Ġpaper Ġis Ġorganized Ġinto Ġfour Ġsections . ĠWe Ġfirst Ġdescribe Ġa Ġstudy Ġwhere Ġwe Ġinvestigate : Ġ1 ) ĠHow Ġmuch Ġanswer Ġdiversity Ġarises Ġfor Ġvisual Ġquestions ? Ġand Ġ2 ) ĠWhy Ġdo Ġpeople Ġdisagree Ġ( Section ĠSEC REF 4 Ġ) ? ĠNext , Ġwe Ġexplore Ġthe Ġfollowing Ġtwo Ġquestions : Ġ1 ) ĠGiven Ġa Ġnovel Ġvisual Ġquestion , Ġcan Ġa Ġmachine Ġcorrectly Ġpredict Ġwhether Ġmultiple Ġindependent Ġmembers Ġof Ġa Ġcrowd Ġwould Ġsupply Ġthe Ġsame Ġanswer ? Ġand Ġ2 ) ĠIf Ġso , Ġwhat Ġinsights Ġdoes Ġour Ġmachine - learn ed Ġsystem Ġreveal Ġregarding Ġwhat Ġhumans Ġare Ġmost Ġlikely Ġto Ġagree Ġabout Ġ( Section ĠSEC REF 5 Ġ) ? ĠIn Ġthe Ġfollowing Ġsection , Ġwe Ġpropose Ġa Ġnovel Ġresource Ġallocation Ġsystem Ġfor Ġefficiently Ġcapturing Ġthe Ġdiversity Ġof Ġall Ġanswers Ġfor Ġa Ġset Ġof Ġvisual Ġquestions Ġ( Section ĠSEC REF 6 Ġ). ĠFinally , Ġwe Ġend Ġwith Ġconcluding Ġremarks Ġ( Section ĠSEC REF 7 Ġ). </s> V Q A Ġ- ĠAnalysis Ġof ĠAnswer Ġ( Dis ) Ag reements </s> Our Ġfirst Ġaim Ġis Ġto Ġanswer Ġthe Ġfollowing Ġquestions : Ġ1 ) ĠHow Ġmuch Ġanswer Ġdiversity Ġarises Ġfor Ġvisual Ġquestions ? Ġand Ġ2 ) ĠWhy Ġdo Ġpeople Ġdisagree ? </s> Visual ĠQuestion Ġ- ĠPredict ing ĠIf Ġa ĠCrowd ĠWill Ġ( Dis ) Ag ree Ġon Ġan ĠAnswer </s> We Ġnow Ġexplore Ġthe Ġfollowing Ġtwo Ġquestions : Ġ1 ) ĠGiven Ġa Ġnovel Ġvisual Ġquestion , Ġcan Ġa Ġmachine Ġcorrectly Ġpredict Ġwhether Ġmultiple Ġindependent Ġmembers Ġof Ġa Ġcrowd Ġwould Ġsupply Ġthe Ġsame Ġanswer ? Ġand Ġ2 ) ĠIf Ġso , Ġwhat Ġinsights Ġdoes Ġour Ġmachine - learn ed Ġsystem Ġreveal Ġregarding Ġwhat Ġhumans Ġare Ġmost Ġlikely Ġto Ġagree Ġabout ? </s> Pred iction ĠSystems </s> We Ġpose Ġthe Ġprediction Ġtask Ġas Ġa Ġbinary Ġclassification Ġproblem . ĠSpecifically , Ġgiven Ġan Ġimage Ġand Ġassociated Ġquestion , Ġa Ġsystem Ġoutputs Ġa Ġbinary Ġlabel Ġindicating Ġwhether Ġa Ġcrowd Ġwill Ġagree Ġon Ġthe Ġsame Ġanswer . ĠOur Ġgoal Ġis Ġto Ġdesign Ġa Ġsystem Ġthat Ġcan Ġdetect Ġwhich Ġvisual Ġquestions Ġto Ġassign Ġa Ġdisagreement Ġlabel , Ġregardless Ġof Ġthe Ġdisagreement Ġcause Ġ( e . g ., Ġsubject ivity , Ġambiguity , Ġdifficulty ). ĠWe Ġimplement Ġboth Ġrandom Ġforest Ġand Ġdeep Ġlearning Ġclass ifiers . </s> A Ġvisual Ġquestion Ġis Ġassigned Ġeither Ġan Ġanswer Ġagreement Ġor Ġdisagreement Ġlabel . ĠTo Ġassign Ġlabels , Ġwe Ġemploy Ġ10 Ġcrowds ourced Ġanswers Ġfor Ġeach Ġvisual Ġquestion . ĠA Ġvisual Ġquestion Ġis Ġassigned Ġan Ġanswer Ġagreement Ġlabel Ġwhen Ġthere Ġis Ġan Ġexact Ġstring Ġmatch Ġfor Ġ9 Ġof Ġthe Ġ10 Ġcrowds ourced Ġanswers Ġ( after Ġanswer Ġpre - pre processing , Ġas Ġdiscussed Ġin Ġthe Ġprevious Ġsection ) Ġand Ġan Ġanswer Ġdisagreement Ġlabel Ġotherwise . ĠOur Ġrationale Ġis Ġto Ġpermit Ġthe Ġpossibility Ġof Ġup Ġto Ġone ĠâĢ ľ care less / sp am " Ġanswer Ġper Ġvisual Ġquestion . ĠThe Ġoutcome Ġof Ġour Ġlabeling Ġscheme Ġis Ġthat Ġa Ġdisagreement Ġlabel Ġis Ġag nostic Ġto Ġthe Ġspecific Ġcause Ġof Ġdisagreement Ġand Ġrather Ġrepresents Ġthe Ġmany Ġcauses Ġ( described Ġabove ). </s> For Ġour Ġfirst Ġsystem , Ġwe Ġuse Ġdomain Ġknowledge Ġto Ġguide Ġthe Ġlearning Ġprocess . ĠWe Ġcompile Ġa Ġset Ġof Ġfeatures Ġthat Ġwe Ġhypothes ize Ġinform Ġwhether Ġa Ġcrowd Ġwill Ġarrive Ġat Ġan Ġundis puted , Ġsingle Ġanswer . ĠThen Ġwe Ġapply Ġa Ġmachine Ġlearning Ġtool Ġto Ġreveal Ġthe Ġsignificance Ġof Ġeach Ġfeature . ĠWe Ġpropose Ġfeatures Ġbased Ġon Ġthe Ġobservation Ġthat Ġanswer Ġagreement Ġoften Ġarises Ġwhen Ġ1 ) Ġa Ġlay Ġperson 's Ġattention Ġcan Ġbe Ġeasily Ġconcentrated Ġto Ġa Ġsingle , Ġundis puted Ġregion Ġin Ġan Ġimage Ġand Ġ2 ) Ġa Ġlay Ġperson Ġwould Ġfind Ġthe Ġrequested Ġtask Ġeasy Ġto Ġaddress . </s> We Ġemploy Ġfive Ġimage - based Ġfeatures Ġcoming Ġfrom Ġthe Ġsalient Ġobject Ġsub it izing ĠB IB REF 22 Ġ( S OS ) Ġmethod , Ġwhich Ġproduces Ġfive Ġprobabilities Ġthat Ġindicate Ġwhether Ġan Ġimage Ġcontains Ġ0 , Ġ1 , Ġ2 , Ġ3 , Ġor Ġ4 + Ġsalient Ġobjects . ĠInt uitive ly , Ġthe Ġnumber Ġof Ġsalient Ġobjects Ġshows Ġhow Ġmany Ġregions Ġin Ġan Ġimage Ġare Ġcompeting Ġfor Ġan Ġobserver 's Ġattention , Ġand Ġso Ġmay Ġcorrelate Ġwith Ġthe Ġease Ġin Ġidentifying Ġa Ġregion Ġof Ġinterest . ĠMoreover , Ġwe Ġhypothes ize Ġthis Ġfeature Ġwill Ġcapture Ġour Ġobservation Ġfrom Ġthe Ġprevious Ġstudy Ġthat Ġcounting Ġproblems Ġtypically Ġleads Ġto Ġdisagreement Ġfor Ġimages Ġshowing Ġmany Ġobjects , Ġand Ġagreement Ġotherwise . </s> We Ġemploy Ġa Ġ2 , 492 - dimensional Ġfeature Ġvector Ġto Ġrepresent Ġthe Ġquestion - based Ġfeatures . ĠOne Ġfeature Ġis Ġthe Ġnumber Ġof Ġwords Ġin Ġthe Ġquestion . ĠInt uitive ly , Ġa Ġlonger Ġquestion Ġoffers Ġmore Ġinformation Ġand Ġwe Ġhypothes ize Ġadditional Ġinformation Ġmakes Ġa Ġquestion Ġmore Ġprecise . ĠThe Ġremaining Ġfeatures Ġcome Ġfrom Ġtwo Ġone - hot Ġvectors Ġdescribing Ġeach Ġof Ġthe Ġfirst Ġtwo Ġwords Ġin Ġthe Ġquestion . ĠEach Ġone - hot Ġvector Ġis Ġcreated Ġusing Ġthe Ġlearned Ġvoc ab ular ies Ġthat Ġdefine Ġall Ġpossible Ġwords Ġat Ġthe Ġfirst Ġand Ġsecond Ġword Ġlocation Ġof Ġa Ġquestion Ġrespectively Ġ( using Ġtraining Ġdata , Ġas Ġdescribed Ġin Ġthe Ġnext Ġsection ). ĠInt uitive ly , Ġearly Ġwords Ġin Ġa Ġquestion Ġinform Ġthe Ġtype Ġof Ġanswers Ġthat Ġmight Ġbe Ġpossible Ġand , Ġin Ġturn , Ġpossible Ġreasons / frequency Ġfor Ġanswer Ġdisagreement . ĠFor Ġexample , Ġwe Ġexpect ĠâĢ ľ why Ġis " Ġto Ġregularly Ġelicit Ġmany Ġopinions Ġand Ġso Ġdisagreement . ĠThis Ġintuition Ġabout Ġthe Ġbeginning Ġwords Ġof Ġa Ġquestion Ġis Ġalso Ġsupported Ġby Ġour Ġanalysis Ġin Ġthe Ġprevious Ġsection Ġwhich Ġshows Ġthat Ġdifferent Ġanswer Ġtypes Ġyield Ġdifferent Ġbiases Ġof Ġelic iting Ġanswer Ġagreement Ġversus Ġdisagreement . </s> We Ġleverage Ġa Ġrandom Ġforest Ġclassification Ġmodel ĠB IB REF 23 Ġto Ġpredict Ġan Ġanswer Ġ( dis ) ag reement Ġlabel Ġfor Ġa Ġgiven Ġvisual Ġquestion . ĠThis Ġmodel Ġconsists Ġof Ġan Ġensemble Ġof Ġdecision Ġtree Ġclass ifiers . ĠWe Ġtrain Ġthe Ġsystem Ġto Ġlearn Ġthe Ġunique Ġweighted Ġcombinations Ġof Ġthe Ġaforementioned Ġ2 , 497 Ġfeatures Ġthat Ġeach Ġdecision Ġtree Ġapplies Ġto Ġmake Ġa Ġprediction . ĠAt Ġtest Ġtime , Ġgiven Ġa Ġnovel Ġvisual Ġquestion , Ġthe Ġtrained Ġsystem Ġconverts Ġa Ġ2 , 497 Ġfeature Ġdescriptor Ġof Ġthe Ġvisual Ġquestion Ġinto Ġa Ġfinal Ġprediction Ġthat Ġreflects Ġthe Ġmajority Ġvote Ġprediction Ġfrom Ġthe Ġensemble Ġof Ġdecision Ġtrees . ĠThe Ġsystem Ġreturns Ġthe Ġfinal Ġprediction Ġalong Ġwith Ġa Ġprobability Ġindicating Ġthe Ġsystem 's Ġconfidence Ġin Ġthat Ġprediction . ĠWe Ġemploy Ġthe ĠMat lab Ġimplementation Ġof Ġrandom Ġforests , Ġusing Ġ25 Ġtrees Ġand Ġthe Ġdefault Ġparameters . </s> We Ġnext Ġadapt Ġa ĠV Q A Ġdeep Ġlearning Ġarchitecture ĠB IB REF 24 Ġto Ġlearn Ġthe Ġpredictive Ġcombination Ġof Ġvisual Ġand Ġtextual Ġfeatures . ĠThe Ġquestion Ġis Ġencoded Ġwith Ġa Ġ1024 - dimensional ĠL ST M Ġmodel Ġthat Ġtakes Ġin Ġa Ġone - hot Ġdescriptor Ġof Ġeach Ġword Ġin Ġthe Ġquestion . ĠThe Ġimage Ġis Ġdescribed Ġwith Ġthe Ġ4096 - dimensional Ġoutput Ġfrom Ġthe Ġlast Ġfully Ġconnected Ġlayer Ġof Ġthe ĠConv olution al ĠNeural ĠNetwork Ġ( CNN ), ĠV GG 16 ĠB IB REF 25 Ġ. ĠThe Ġsystem Ġperforms Ġan Ġelement - wise Ġmultiplication Ġof Ġthe Ġimage Ġand Ġquestion Ġfeatures , Ġafter Ġlin early Ġtransforming Ġthe Ġimage Ġdescriptor Ġto Ġ1024 Ġdimensions . ĠThe Ġfinal Ġlayer Ġof Ġthe Ġarchitecture Ġis Ġa Ġsoft max Ġlayer . </s> We Ġtrain Ġthe Ġsystem Ġto Ġpredict Ġ( dis ) ag reement Ġlabels Ġwith Ġtraining Ġexamples , Ġwhere Ġeach Ġexample Ġincludes Ġan Ġimage Ġand Ġquestion . ĠAt Ġtest Ġtime , Ġgiven Ġa Ġnovel Ġvisual Ġquestion , Ġthe Ġsystem Ġoutputs Ġan Ġun normal ized Ġlog Ġprobability Ġindicating Ġits Ġbelief Ġin Ġboth Ġthe Ġagreement Ġand Ġdisagreement Ġlabel . ĠFor Ġour Ġsystem 's Ġprediction , Ġwe Ġconvert Ġthe Ġbelief Ġin Ġthe Ġdisagreement Ġlabel Ġinto Ġa Ġnormalized Ġprobability . ĠConsequently , Ġpredicted Ġvalues Ġrange Ġfrom Ġ0 Ġto Ġ1 Ġwith Ġlower Ġvalues Ġreflecting Ġgreater Ġlikelihood Ġfor Ġcrowd Ġagreement . </s> Analysis Ġof ĠPrediction ĠSystem </s> We Ġnow Ġdescribe Ġour Ġstudies Ġto Ġassess Ġthe Ġpredictive Ġpower Ġof Ġour Ġclassification Ġsystems Ġto Ġdecide Ġwhether Ġvisual Ġquestions Ġwill Ġlead Ġto Ġanswer Ġ( dis ) ag reement . </s> We Ġcapitalize Ġon Ġtoday 's Ġlargest Ġvisual Ġquestion Ġanswering Ġdataset ĠB IB REF 2 Ġto Ġevaluate Ġour Ġprediction Ġsystem , Ġwhich Ġincludes Ġ369 , 86 1 Ġvisual Ġquestions Ġabout Ġreal Ġimages . ĠOf Ġthese , Ġ248 , 349 Ġvisual Ġquestions Ġ( i . e ., ĠTraining Ġquestions Ġ2015 Ġv 1 . 0 ) Ġare Ġkept Ġfor Ġfor Ġtraining Ġand Ġthe Ġremaining Ġ121 , 512 Ġvisual Ġquestions Ġ( i . e ., ĠVal idation Ġquestions Ġ2015 Ġv 1 . 0 ) Ġare Ġemployed Ġfor Ġtesting Ġour Ġclassification Ġsystem . ĠThis Ġseparation Ġof Ġtraining Ġand Ġtesting Ġsamples Ġenables Ġus Ġto Ġestimate Ġhow Ġwell Ġa Ġclass ifier Ġwill Ġgeneral ize Ġwhen Ġapplied Ġto Ġan Ġunseen , Ġindependent Ġset Ġof Ġvisual Ġquestions . </s> To Ġour Ġknowledge , Ġno Ġprior Ġwork Ġhas Ġdirectly Ġaddressed Ġpredicting Ġanswer Ġ( dis ) ag reement Ġfor Ġvisual Ġquestions . ĠTherefore , Ġwe Ġemploy Ġas Ġa Ġbaseline Ġa Ġrelated ĠV Q A Ġalgorithm ĠB IB REF 24 Ġ, ĠB IB REF 2 Ġwhich Ġproduces Ġfor Ġa Ġgiven Ġvisual Ġquestion Ġan Ġanswer Ġwith Ġa Ġconfidence Ġscore . ĠThis Ġsystem Ġparallels Ġthe Ġdeep Ġlearning Ġarchitecture Ġwe Ġadapt . ĠHowever , Ġit Ġpredicts Ġthe Ġsystem 's Ġuncertainty Ġin Ġits Ġown Ġanswer , Ġwhereas Ġwe Ġare Ġinterested Ġin Ġthe Ġhumans ' Ġcollective Ġdisagreement Ġon Ġthe Ġanswer . ĠStill , Ġit Ġis Ġa Ġuseful Ġbaseline Ġto Ġsee Ġif Ġan Ġexisting Ġalgorithm Ġcould Ġserve Ġour Ġpurpose . </s> We Ġevaluate Ġthe Ġpredictive Ġpower Ġof Ġthe Ġclassification Ġsystems Ġbased Ġon Ġeach Ġclass ifier 's Ġpredictions Ġfor Ġthe Ġ121 , 512 Ġvisual Ġquestions Ġin Ġthe Ġtest Ġdataset . ĠWe Ġfirst Ġshow Ġperformance Ġof Ġthe Ġbaseline Ġand Ġour Ġtwo Ġprediction Ġsystems Ġusing Ġprecision - rec all Ġcurves . ĠThe Ġgoals Ġare Ġto Ġachieve Ġa Ġhigh Ġprecision , Ġto Ġminimize Ġwasting Ġcrowd Ġeffort Ġwhen Ġtheir Ġefforts Ġwill Ġbe Ġredundant , Ġand Ġa Ġhigh Ġrecall , Ġto Ġavoid Ġmissing Ġout Ġon Ġcollecting Ġthe Ġdiversity Ġof Ġaccepted Ġanswers Ġfrom Ġa Ġcrowd . ĠWe Ġalso Ġreport Ġthe Ġaverage Ġprecision Ġ( AP ), Ġwhich Ġindicates Ġthe Ġarea Ġunder Ġa Ġprecision - rec all Ġcurve . ĠAP Ġvalues Ġrange Ġfrom Ġ0 Ġto Ġ1 Ġwith Ġbetter - performing Ġprediction Ġsystems Ġhaving Ġlarger Ġvalues . </s> Figure ĠFIG REF 8 Ġa Ġshows Ġprecision - rec all Ġcurves Ġfor Ġall Ġprediction Ġsystems . ĠBoth Ġour Ġproposed Ġclassification Ġsystems Ġoutper form Ġthe ĠV Q A ĠAl gorithm ĠB IB REF 2 Ġbaseline ; Ġe . g ., ĠO urs Ġ- ĠRF Ġyields Ġa Ġ12 Ġpercentage Ġpoint Ġimprovement Ġwith Ġrespect Ġto ĠAP . ĠThis Ġis Ġinteresting Ġbecause Ġit Ġshows Ġthere Ġis Ġvalue Ġin Ġlearning Ġthe Ġdisagreement Ġtask Ġspecifically , Ġrather Ġthan Ġemploying Ġan Ġalgorithm 's Ġconfidence Ġin Ġits Ġanswers . ĠMore Ġgenerally , Ġour Ġresults Ġdemonstrate Ġit Ġis Ġpossible Ġto Ġpredict Ġwhether Ġa Ġcrowd Ġwill Ġagree Ġon Ġa Ġsingle Ġanswer Ġfrom Ġa Ġgiven Ġimage Ġand Ġassociated Ġquestion . ĠDespite Ġthe Ġsignificant Ġvariety Ġof Ġquestions Ġand Ġimage Ġcontent , Ġand Ġdespite Ġthe Ġvariety Ġof Ġreasons Ġfor Ġwhich Ġthe Ġcrowd Ġcan Ġdisagree , Ġour Ġlearned Ġmodel Ġis Ġable Ġto Ġproduce Ġquite Ġaccurate Ġresults . </s> We Ġobserve Ġour ĠRandom ĠForest Ġclass ifier Ġoutper forms Ġour Ġdeep Ġlearning Ġclass ifier ; Ġe . g ., ĠO urs : ĠRF Ġyields Ġa Ġthree Ġpercentage Ġpoint Ġimprovement Ġwith Ġrespect Ġto ĠAP Ġwhile Ġconsistently Ġyielding Ġimproved Ġprecision - rec all Ġvalues Ġover ĠO urs : ĠL ST M - CNN Ġ( Figure ĠFIG REF 8 Ġa ). ĠIn Ġgeneral , Ġdeep Ġlearning Ġsystems Ġhold Ġpromise Ġto Ġreplace Ġhand crafted Ġfeatures Ġto Ġpick Ġout Ġthe Ġdiscrim in ative Ġfeatures . ĠOur Ġbas elines Ġhighlight Ġa Ġpossible Ġvalue Ġin Ġdeveloping Ġa Ġdifferent Ġdeep Ġlearning Ġarchitecture Ġfor Ġthe Ġproblem Ġof Ġlearning Ġanswer Ġdisagreement Ġthan Ġapplied Ġfor Ġpredicting Ġanswers Ġto Ġvisual Ġquestions . </s> We Ġshow Ġexamples Ġof Ġprediction Ġresults Ġwhere Ġour Ġtop - performing ĠRF Ġclass ifier Ġmakes Ġits Ġmost Ġconfident Ġpredictions Ġ( Figure ĠFIG REF 8 Ġb ). ĠIn Ġthese Ġexamples , Ġthe Ġpredictor Ġexpects Ġhuman Ġagreement Ġfor ĠâĢ ľ what Ġroom ... Ġ ?" Ġvisual Ġquestions Ġand Ġdisagreement Ġfor ĠâĢ ľ why ... Ġ ?" Ġvisual Ġquestions . ĠThese Ġexamples Ġhighlight Ġthat Ġthe Ġclass ifier Ġmay Ġhave Ġa Ġstrong Ġlanguage Ġprior Ġtowards Ġmaking Ġpredictions , Ġas Ġwe Ġwill Ġdiscuss Ġin Ġthe Ġnext Ġsection . </s> We Ġnow Ġexplore Ġwhat Ġmakes Ġa Ġvisual Ġquestion Ġlead Ġto Ġcrowd Ġanswer Ġagreement Ġversus Ġdisagreement . ĠWe Ġexamine Ġthe Ġinfluence Ġof Ġwhether Ġvisual Ġquestions Ġlead Ġto Ġthe Ġthree Ġtypes Ġof Ġanswers Ġ( âĢ ľ yes / no ", ĠâĢ ľ number ", ĠâĢ ľ other ") Ġfor Ġboth Ġour Ġrandom Ġforest Ġ( RF ) Ġand Ġdeep Ġlearning Ġ( DL ) Ġclassification Ġsystems . ĠWe Ġenrich Ġour Ġanalysis Ġby Ġexamining Ġthe Ġpredictive Ġperformance Ġof Ġboth Ġclass ifiers Ġwhen Ġthey Ġare Ġtrained Ġand Ġtested Ġexclusively Ġwith Ġimage Ġand Ġquestion Ġfeatures Ġrespectively . ĠFigure ĠFIG REF 9 Ġshows Ġprecision - rec all Ġcurves Ġfor Ġboth Ġclassification Ġsystems Ġwith Ġquestion Ġfeatures Ġalone Ġ( Q ), Ġimage Ġfeatures Ġalone Ġ( I ), Ġand Ġboth Ġquestion Ġand Ġimage Ġfeatures Ġtogether Ġ( Q + I ). </s> When Ġcomparing ĠAP Ġscores Ġ( Figure ĠFIG REF 9 Ġ), Ġwe Ġobserve Ġour ĠQ + I Ġpredict ors Ġyield Ġthe Ġgreatest Ġpredictive Ġperformance Ġfor Ġvisual Ġquestions Ġthat Ġlead Ġto ĠâĢ ľ other " Ġanswers , Ġfollowed Ġby ĠâĢ ľ number " Ġanswers , Ġand Ġfinally ĠâĢ ľ yes / no " Ġanswers . ĠOne Ġpossible Ġreason Ġfor Ġthis Ġfinding Ġis Ġthat Ġthe Ġquestion Ġwording Ġstrongly Ġdrives Ġwhether Ġa Ġcrowd Ġwill Ġdisagree Ġfor ĠâĢ ľ other " Ġvisual Ġquestions , Ġwhereas Ġsome Ġnotion Ġof Ġcommon Ġsense Ġmay Ġbe Ġrequired Ġto Ġlearn Ġwhether Ġa Ġcrowd Ġwill Ġagree Ġfor ĠâĢ ľ yes / no " Ġvisual Ġquestions Ġ( e . g ., ĠFigure ĠFIG REF 7 Ġa Ġvs ĠFigure ĠFIG REF 7 Ġg ). </s> We Ġobserve Ġthat Ġquestion - based Ġfeatures Ġyield Ġgreater Ġpredictive Ġperformance Ġthan Ġimage - based Ġfeatures Ġfor Ġall Ġvisual Ġquestions , Ġwhen Ġcomparing ĠAP Ġscores Ġfor ĠQ Ġand ĠI Ġclassification Ġresults Ġ( Figure ĠFIG REF 9 Ġ). ĠIn Ġfact , Ġimage Ġfeatures Ġcontribute Ġto Ġperformance Ġimprovements Ġonly Ġfor Ġour Ġrandom Ġforest Ġclass ifier Ġfor Ġvisual Ġquestions Ġthat Ġlead Ġto ĠâĢ ľ number " Ġanswers , Ġas Ġillustrated Ġby Ġcomparing ĠAP Ġscores Ġfor ĠOur ĠRF : ĠQ + I Ġand ĠOur ĠRF : ĠQ Ġ( Figure ĠFIG REF 9 Ġb ). ĠOur Ġoverall Ġfinding Ġthat Ġmost Ġof Ġthe Ġpredictive Ġpower Ġstems Ġfrom Ġlanguage - based Ġfeatures Ġparallels Ġfeature Ġanalysis Ġfindings Ġin Ġthe Ġautomated ĠV Q A Ġliterature ĠB IB REF 2 Ġ, ĠB IB REF 21 Ġ. ĠThis Ġdoes Ġnot Ġmean , Ġhowever , Ġthat Ġthe Ġimage Ġcontent Ġis Ġnot Ġpredictive . ĠFurther Ġwork Ġimproving Ġvisual Ġcontent Ġcues Ġfor ĠV Q A Ġagreement Ġis Ġwarranted . </s> Our Ġfindings Ġsuggest Ġthat Ġour ĠRandom ĠForest Ġclass ifier 's Ġoverall Ġadvantage Ġover Ġour Ġdeep Ġlearning Ġsystem Ġarises Ġbecause Ġof Ġcounting Ġquestions , Ġas Ġindicated Ġby Ġhigher ĠAP Ġscores Ġ( Figure ĠFIG REF 9 Ġ). ĠFor Ġexample , Ġthe Ġadvantage Ġof Ġthe Ġinitial Ġhigher Ġprecision Ġ( Figure ĠFIG REF 8 Ġa ; ĠO urs : ĠRF Ġvs ĠO urs : ĠDL ) Ġis Ġalso Ġobserved Ġfor Ġcounting Ġquestions Ġ( Figure ĠFIG REF 9 Ġb ; ĠO urs : ĠRF Ġ- ĠQ + I Ġvs ĠO urs : ĠDL Ġ- ĠQ + I ). ĠWe Ġhypothes ize Ġthis Ġadvantage Ġarises Ġdue Ġto Ġthe Ġstrength Ġof Ġthe ĠRandom ĠForest Ġclass ifier Ġin Ġpairing Ġthe Ġquestion Ġprior Ġ( âĢ ľ How Ġmany ?" ) Ġwith Ġthe Ġimage - based ĠSOS Ġfeatures Ġthat Ġindicates Ġthe Ġnumber Ġof Ġobjects Ġin Ġan Ġimage . ĠSpecifically , Ġwe Ġexpect ĠâĢ ľ how Ġmany " Ġto Ġlead Ġto Ġagreement Ġonly Ġfor Ġsmall Ġcounting Ġproblems . </s> Capt uring ĠAnswer ĠDiversity Ġwith ĠLess ĠEff ort </s> We Ġnext Ġpresent Ġa Ġnovel Ġresource Ġallocation Ġsystem Ġfor Ġefficiently Ġcapturing Ġthe Ġdiversity Ġof Ġtrue Ġanswers Ġfor Ġa Ġbatch Ġof Ġvisual Ġquestions . ĠToday 's Ġstatus Ġquo Ġis Ġto Ġeither Ġuniformly Ġcollect ĠIN LINE FORM 0 Ġanswers Ġfor Ġevery Ġvisual Ġquestion ĠB IB REF 2 Ġor Ġcollect Ġmultiple Ġanswers Ġwhere Ġthe Ġnumber Ġis Ġdetermined Ġby Ġexternal Ġcrowds ourcing Ġconditions ĠB IB REF 0 Ġ. ĠOur Ġsystem Ġinstead Ġspends Ġa Ġhuman Ġbudget Ġby Ġpredicting Ġthe Ġnumber Ġof Ġanswers Ġto Ġcollect Ġfor Ġeach Ġvisual Ġquestion Ġbased Ġon Ġwhether Ġmultiple Ġhuman Ġanswers Ġare Ġpredicted Ġto Ġbe Ġredundant . </s> Answer ĠCollection ĠSystem </s> Supp ose Ġwe Ġhave Ġa Ġbudget ĠIN LINE FORM 0 Ġwhich Ġwe Ġcan Ġallocate Ġto Ġcollect Ġextra Ġanswers Ġfor Ġa Ġsubset Ġof Ġvisual Ġquestions . ĠOur Ġsystem Ġautomatically Ġdecides Ġto Ġwhich Ġvisual Ġquestions Ġto Ġallocate Ġthe ĠâĢ ľ extra " Ġanswers Ġin Ġorder Ġto Ġmaximize Ġcaptured Ġanswer Ġdiversity Ġfor Ġall Ġvisual Ġquestions . </s> The Ġaim Ġof Ġour Ġsystem Ġis Ġto Ġacc rue Ġadditional Ġcosts Ġand Ġdelays Ġfrom Ġcollecting Ġextra Ġanswers Ġonly Ġwhen Ġextra Ġresponses Ġwill Ġprovide Ġmore Ġinformation . ĠTowards Ġthis Ġaim , Ġour Ġsystem Ġinvolves Ġthree Ġsteps Ġto Ġcollect Ġanswers Ġfor Ġall ĠIN LINE FORM 0 Ġvisual Ġquestions Ġ( Figure ĠFIG REF 11 Ġa ). ĠFirst , Ġthe Ġsystem Ġapplies Ġour Ġtop - performing Ġrandom Ġforest Ġclass ifier Ġto Ġevery Ġvisual Ġquestion Ġin Ġthe Ġbatch . ĠThen , Ġthe Ġsystem Ġranks Ġthe ĠIN LINE FORM 1 Ġvisual Ġquestions Ġbased Ġon Ġpredicted Ġscores Ġfrom Ġthe Ġclass ifier , Ġfrom Ġvisual Ġquestions Ġmost Ġconfidently Ġpredicted Ġto Ġlead Ġto Ġanswer ĠâĢ ľ ag reement " Ġfrom Ġa Ġcrowd Ġto Ġthose Ġmost Ġconfidently Ġpredicted Ġto Ġlead Ġto Ġanswer ĠâĢ ľ dis ag reement " Ġfrom Ġa Ġcrowd . ĠFinally , Ġthe Ġsystem Ġsolic its Ġmore Ġ( ĠIN LINE FORM 2 Ġ) Ġhuman Ġanswers Ġfor Ġthe ĠIN LINE FORM 3 Ġvisual Ġquestions Ġpredicted Ġto Ġreflect Ġthe Ġgreatest Ġlikelihood Ġfor Ġcrowd Ġdisagreement Ġand Ġfewer Ġ( ĠIN LINE FORM 4 Ġ) Ġhuman Ġanswers Ġfor Ġthe Ġremaining Ġvisual Ġquestions . ĠMore Ġdetails Ġbelow . </s> Analysis Ġof ĠAnswer ĠCollection ĠSystem </s> We Ġnow Ġdescribe Ġour Ġstudies Ġto Ġassess Ġthe Ġbenefit Ġof Ġour Ġallocation Ġsystem Ġto Ġreduce Ġhuman Ġeffort Ġto Ġcapture Ġthe Ġdiversity Ġof Ġall Ġanswers Ġto Ġvisual Ġquestions . </s> We Ġevaluate Ġthe Ġimpact Ġof Ġactively Ġall ocating Ġextra Ġhuman Ġeffort Ġto Ġanswer Ġvisual Ġquestions Ġas Ġa Ġfunction Ġof Ġthe Ġavailable Ġbudget Ġof Ġhuman Ġeffort . ĠSpecifically , Ġfor Ġa Ġrange Ġof Ġbudget Ġlevels , Ġwe Ġcompute Ġthe Ġtotal Ġmeasured Ġanswer Ġdiversity Ġ( as Ġdefined Ġbelow ) Ġresulting Ġfor Ġthe Ġbatch Ġof Ġvisual Ġquestions . ĠThe Ġgoal Ġis Ġto Ġcapture Ġa Ġlarge Ġamount Ġof Ġanswer Ġdiversity Ġwith Ġlittle Ġhuman Ġeffort . </s> We Ġconduct Ġour Ġstudies Ġon Ġthe Ġ121 , 512 Ġtest Ġvisual Ġquestions Ġabout Ġreal Ġimages Ġ( i . e ., ĠVal idation Ġquestions Ġ2015 Ġv 1 . 0 ). ĠFor Ġeach Ġvisual Ġquestion , Ġwe Ġestablish Ġthe Ġset Ġof Ġtrue Ġanswers Ġas Ġall Ġunique Ġanswers Ġwhich Ġare Ġobserved Ġat Ġleast Ġtwice Ġin Ġthe Ġ10 Ġcrowds ourced Ġanswers Ġper Ġvisual Ġquestion . ĠWe Ġrequire Ġagreement Ġby Ġtwo Ġworkers Ġto Ġavoid Ġthe Ġpossibility Ġthat ĠâĢ ľ care less / sp am " Ġanswers Ġare Ġtreated Ġas Ġground Ġtruth . </s> We Ġcollect Ġeither Ġthe Ġminimum Ġof ĠIN LINE FORM 0 Ġanswer Ġper Ġvisual Ġquestion Ġor Ġthe Ġmaximum Ġof ĠIN LINE FORM 1 Ġanswers Ġper Ġvisual Ġquestion . ĠOur Ġnumber Ġof Ġanswers Ġroughly Ġalign s Ġwith Ġexisting Ġcrowd - powered ĠV Q A Ġsystems , Ġfor Ġexample Ġwith ĠViz W iz , ĠâĢ ľ On Ġaverage , Ġparticipants Ġreceived Ġ3 . 3 Ġ( SD = 1 . 8 ) Ġanswers Ġfor Ġeach Ġquestion " ĠB IB REF 0 Ġ. ĠOur Ġmaximum Ġnumber Ġof Ġanswers Ġalso Ġsupports Ġthe Ġpossibility Ġof Ġcapturing Ġthe Ġmaximum Ġof Ġthree Ġunique , Ġvalid Ġanswers Ġtypically Ġobserved Ġin Ġpractice Ġ( rec all Ġstudy Ġabove ). ĠWhile Ġmore Ġelaborate Ġschemes Ġfor Ġdistributing Ġresponses Ġmay Ġbe Ġpossible , Ġwe Ġwill Ġshow Ġthis Ġapproach Ġalready Ġproves Ġquite Ġeffective Ġin Ġour Ġexperiments . ĠWe Ġsimulate Ġanswer Ġcollection Ġby Ġrandomly Ġselecting Ġanswers Ġfrom Ġthe Ġ10 Ġcrowd Ġanswers Ġper Ġvisual Ġquestion . </s> We Ġcompare Ġour Ġapproach Ġto Ġthe Ġfollowing Ġbas elines : </s> As Ġin Ġthe Ġprevious Ġsection , Ġwe Ġleverage Ġthe Ġoutput Ġconfidence Ġscore Ġfrom Ġthe Ġpublicly - shared Ġmodel ĠB IB REF 24 Ġlearned Ġfrom Ġa ĠL ST M - CNN Ġdeep Ġlearning Ġarchitecture Ġto Ġrank Ġthe Ġorder Ġof Ġpriority Ġfor Ġvisual Ġquestions Ġto Ġreceive Ġredundancy . </s> The Ġsystem Ġrandomly Ġprior itiz es Ġwhich Ġimages Ġreceive Ġredundancy . ĠThis Ġpredictor Ġillustrates Ġthe Ġbest Ġa Ġuser Ġcan Ġachieve Ġtoday Ġwith Ġcrowd - powered Ġsystems ĠB IB REF 0 Ġ, ĠB IB REF 5 Ġor Ġwith Ġcurrent Ġdataset Ġcollection Ġmethods ĠB IB REF 2 Ġ, ĠB IB REF 4 Ġ. </s> We Ġquantify Ġthe Ġtotal Ġdiversity Ġof Ġanswers Ġcaptured Ġby Ġa Ġresource Ġallocation Ġsystem Ġfor Ġa Ġbatch Ġof Ġvisual Ġquestions ĠIN LINE FORM 0 Ġas Ġfollows : ĠDIS PLAY FORM 0 </s> where ĠIN LINE FORM 0 Ġrepresents Ġthe Ġset Ġof Ġall Ġtrue Ġanswers Ġfor Ġthe ĠIN LINE FORM 1 Ġ- th Ġvisual Ġquestion , ĠIN LINE FORM 2 Ġrepresents Ġthe Ġset Ġof Ġunique Ġanswers Ġcaptured Ġin Ġthe ĠIN LINE FORM 3 Ġanswers Ġcollected Ġfor Ġthe ĠIN LINE FORM 4 Ġ- th Ġvisual Ġquestion , Ġand ĠIN LINE FORM 5 Ġrepresents Ġthe Ġset Ġof Ġunique Ġanswers Ġcaptured Ġin Ġthe ĠIN LINE FORM 6 Ġanswers Ġcollected Ġfor Ġthe ĠIN LINE FORM 7 Ġ- th Ġvisual Ġquestion . ĠGiven Ġno Ġextra Ġhuman Ġbudget , Ġtotal Ġdiversity Ġcomes Ġfrom Ġthe Ġsecond Ġterm Ġwhich Ġindicates Ġthe Ġdiversity Ġcaptured Ġwhen Ġonly ĠIN LINE FORM 8 Ġanswers Ġare Ġcollected Ġfor Ġevery Ġvisual Ġquestion . ĠGiven Ġa Ġmaximum Ġavailable Ġextra Ġhuman Ġbudget Ġ( ĠIN LINE FORM 9 Ġ), Ġtotal Ġdiversity Ġcomes Ġfrom Ġthe Ġfirst Ġterm Ġwhich Ġindicates Ġthe Ġdiversity Ġcaptured Ġwhen ĠIN LINE FORM 10 Ġanswers Ġare Ġcollected Ġfor Ġevery Ġvisual Ġquestion . ĠGiven Ġa Ġpartial Ġextra Ġhuman Ġbudget Ġ( ĠIN LINE FORM 11 Ġ), Ġthe Ġaim Ġis Ġto Ġhave Ġperfect Ġpredictions Ġsuch Ġthat Ġthe Ġminimum Ġnumber Ġof Ġanswers Ġ( ĠIN LINE FORM 12 Ġ) Ġare Ġallocated Ġonly Ġfor Ġvisual Ġquestions Ġwith Ġone Ġtrue Ġanswer Ġso Ġthat Ġall Ġdiverse Ġanswers Ġare Ġsafely Ġcaptured . </s> We Ġmeasure Ġdiversity Ġper Ġvisual Ġquestion Ġas Ġthe Ġnumber Ġof Ġall Ġtrue Ġanswers Ġcollected Ġper Ġvisual Ġquestion Ġ( ĠIN LINE FORM 0 Ġ). ĠL arger Ġvalues Ġreflect Ġgreater Ġcaptured Ġdiversity . ĠThe Ġmotivation Ġfor Ġthis Ġmeasure Ġis Ġto Ġonly Ġgive Ġtotal Ġcredit Ġto Ġvisual Ġquestions Ġwhen Ġall Ġvalid , Ġunique Ġhuman Ġanswers Ġare Ġcollected . </s> Our Ġsystem Ġconsistently Ġoffers Ġsignificant Ġgains Ġover Ġtoday 's Ġstatus Ġquo Ġapproach Ġ( Figure ĠFIG REF 11 Ġb ). ĠFor Ġexample , Ġour Ġsystem Ġacceler ates Ġthe Ġcollection Ġof Ġ70 % Ġof Ġthe Ġdiversity Ġby Ġ21 % Ġover Ġthe ĠStatus ĠQu o Ġbaseline . ĠIn Ġaddition , Ġour Ġsystem Ġacceler ates Ġthe Ġcollection Ġof Ġthe Ġ82 % Ġof Ġdiversity Ġone Ġwould Ġobserve Ġwith ĠViz W iz Ġby Ġ23 % Ġ( i . e ., Ġaverage Ġof Ġ3 . 3 Ġanswers Ġper Ġvisual Ġquestion ). ĠIn Ġabsolute Ġterms , Ġthis Ġmeans Ġeliminating Ġthe Ġcollection Ġof Ġ92 , 180 Ġanswers Ġwith Ġno Ġloss Ġto Ġcaptured Ġanswer Ġdiversity . ĠThis Ġtranslates Ġto Ġeliminating Ġ19 Ġ40 - hour Ġwork Ġweeks Ġand Ġsaving Ġover Ġ$ 1800 , Ġassuming Ġworkers Ġare Ġpaid Ġ$ 0 . 02 Ġper Ġanswer Ġand Ġtake Ġ30 Ġseconds Ġto Ġanswer Ġa Ġvisual Ġquestion . ĠOur Ġapproach Ġfills Ġan Ġimportant Ġgap Ġin Ġthe Ġcrowds ourcing Ġanswer Ġcollection Ġliterature Ġfor Ġtargeting Ġthe Ġallocation Ġof Ġextra Ġanswers Ġonly Ġto Ġvisual Ġquestions Ġwhere Ġa Ġdiversity Ġof Ġanswers Ġis Ġexpected . </s> Figure ĠFIG REF 11 Ġb Ġalso Ġillustrates Ġthe Ġadvantage Ġof Ġour Ġsystem Ġover Ġa Ġrelated ĠV Q A Ġalgorithm ĠB IB REF 2 Ġfor Ġour Ġnovel Ġapplication Ġof Ġcost - sensitive Ġanswer Ġcollection Ġfrom Ġa Ġcrowd . ĠAs Ġobserved , Ġrelying Ġon Ġan Ġalgorithm 's Ġconfidence Ġin Ġits Ġanswer Ġoffers Ġa Ġvaluable Ġindicator Ġover Ġtoday 's Ġstatus Ġquo Ġof Ġpassively Ġbudget ing . ĠWhile Ġwe Ġacknowledge Ġthis Ġmethod Ġis Ġnot Ġintended Ġfor Ġour Ġtask Ġspecifically , Ġit Ġstill Ġserves Ġas Ġan Ġimportant Ġbaseline Ġ( as Ġdiscussed Ġabove ). ĠWe Ġattribute Ġthe Ġfurther Ġperformance Ġgains Ġof Ġour Ġprediction Ġsystem Ġto Ġit Ġdirectly Ġpredicting Ġwhether Ġhumans Ġwill Ġdisagree Ġrather Ġthan Ġpredicting Ġa Ġproperty Ġof Ġa Ġspecific Ġalgorithm Ġ( e . g ., Ġconfidence Ġof Ġthe ĠAnt ol Ġet Ġal . Ġalgorithm Ġin Ġits Ġanswer Ġprediction ). </s> Con clusions </s> We Ġproposed Ġa Ġnew Ġproblem Ġof Ġpredicting Ġwhether Ġdifferent Ġpeople Ġwould Ġanswer Ġwith Ġthe Ġsame Ġresponse Ġto Ġthe Ġsame Ġvisual Ġquestion . ĠTowards Ġmotivating Ġthe Ġpractical Ġimplications Ġfor Ġthis Ġproblem , Ġwe Ġanalyzed Ġnearly Ġhalf Ġa Ġmillion Ġvisual Ġquestions Ġand Ġdemonstrated Ġthere Ġis Ġnearly Ġa Ġ50 / 50 Ġsplit Ġbetween Ġvisual Ġquestions Ġthat Ġlead Ġto Ġanswer Ġagreement Ġversus Ġdisagreement . ĠWe Ġobserved Ġthat Ġcrowd Ġdisagreement Ġarose Ġfor Ġvarious Ġtypes Ġof Ġanswers Ġ( yes / no , Ġcounting , Ġother ) Ġfor Ġmany Ġdifferent Ġreasons . ĠWe Ġnext Ġproposed Ġa Ġsystem Ġthat Ġautomatically Ġpredicts Ġwhether Ġa Ġvisual Ġquestion Ġwill Ġlead Ġto Ġa Ġsingle Ġversus Ġmultiple Ġanswers Ġfrom Ġa Ġcrowd . ĠOur Ġmethod Ġoutper forms Ġa Ġstrong Ġexisting ĠV Q A Ġsystem Ġlimited Ġto Ġestimating Ġsystem Ġuncertainty Ġrather Ġthan Ġcrowd Ġdisagreement . ĠFinally , Ġwe Ġdemonstrated Ġhow Ġto Ġemploy Ġthe Ġprediction Ġsystem Ġto Ġaccelerate Ġthe Ġcollection Ġof Ġdiverse Ġanswers Ġfrom Ġa Ġcrowd Ġby Ġtypically Ġat Ġleast Ġ20 % Ġover Ġtoday 's Ġstatus Ġquo Ġof Ġfixed Ġredundancy Ġallocation . </s> Acknowled gments </s> The Ġauthors Ġgrate fully Ġacknowledge Ġfunding Ġfrom Ġthe ĠOffice Ġof ĠNaval ĠResearch Ġ( ON R ĠY IP ĠN 0001 4 - 12 - 1 - 0 754 ) Ġand ĠNational ĠScience ĠFoundation Ġ( I IS - 10 65 390 ). ĠWe Ġthank ĠD ines h ĠJay ar aman , ĠYu - Ch uan ĠSu , ĠSu y og ĠJ ain , Ġand ĠChao - Y eh ĠChen Ġfor Ġtheir Ġassistance Ġwith Ġexperiments .