No
BOOLEAN
Prior work has aimed to make privacy policies easier to understand. Prescriptive approaches towards communicating privacy information BIBREF21, BIBREF22, BIBREF23 have not been widely adopted by industry. Recently, there have been significant research effort devoted to understanding privacy policies by leveraging NLP techniques BIBREF24, BIBREF25, BIBREF26, BIBREF27, BIBREF28, especially by identifying specific data practices within a privacy policy. We adopt a personalized approach to understanding privacy policies, that allows users to query a document and selectively explore content salient to them. Most similar is the PolisisQA corpus BIBREF29, which examines questions users ask corporations on Twitter. Our approach differs in several ways: 1) The PrivacyQA dataset is larger, containing 10x as many questions and answers. 2) Answers are formulated by domain experts with legal training. 3) PrivacyQA includes diverse question types, including unanswerable and subjective questions.
1911.00841
Introduction </s> Privacy Ġpolicies Ġare Ġthe Ġdocuments Ġwhich Ġdisclose Ġthe Ġways Ġin Ġwhich Ġa Ġcompany Ġgathers , Ġuses , Ġshares Ġand Ġmanages Ġa Ġuser 's Ġdata . ĠAs Ġlegal Ġdocuments , Ġthey Ġfunction Ġusing Ġthe Ġprinciple Ġof Ġnotice Ġand Ġchoice ĠB IB REF 0 , Ġwhere Ġcompanies Ġpost Ġtheir Ġpolicies , Ġand Ġtheoretically , Ġusers Ġread Ġthe Ġpolicies Ġand Ġdecide Ġto Ġuse Ġa Ġcompany 's Ġproducts Ġor Ġservices Ġonly Ġif Ġthey Ġfind Ġthe Ġconditions Ġoutlined Ġin Ġits Ġprivacy Ġpolicy Ġacceptable . ĠMany Ġlegal Ġjurisdictions Ġaround Ġthe Ġworld Ġaccept Ġthis Ġframework , Ġincluding Ġthe ĠUnited ĠStates Ġand Ġthe ĠEuropean ĠUnion ĠB IB REF 1 , ĠB IB REF 2 . ĠHowever , Ġthe Ġlegitimacy Ġof Ġthis Ġframework Ġdepends Ġupon Ġusers Ġactually Ġreading Ġand Ġunderstanding Ġprivacy Ġpolicies Ġto Ġdetermine Ġwhether Ġcompany Ġpractices Ġare Ġacceptable Ġto Ġthem ĠB IB REF 3 . ĠIn Ġpractice Ġthis Ġis Ġseldom Ġthe Ġcase ĠB IB REF 4 , ĠB IB REF 5 , ĠB IB REF 6 , ĠB IB REF 7 , ĠB IB REF 8 , ĠB IB REF 9 , ĠB IB REF 10 . ĠThis Ġis Ġfurther Ġcomplicated Ġby Ġthe Ġhighly Ġindividual Ġand Ġnuanced Ġcompromises Ġthat Ġusers Ġare Ġwilling Ġto Ġmake Ġwith Ġtheir Ġdata ĠB IB REF 11 , Ġdiscouraging Ġa Ġ` one - size - fits - all ' Ġapproach Ġto Ġnotice Ġof Ġdata Ġpractices Ġin Ġprivacy Ġdocuments . </s> With Ġdevices Ġconstantly Ġmonitoring Ġour Ġenvironment , Ġincluding Ġour Ġpersonal Ġspace Ġand Ġour Ġbodies , Ġlack Ġof Ġawareness Ġof Ġhow Ġour Ġdata Ġis Ġbeing Ġused Ġeasily Ġleads Ġto Ġproblematic Ġsituations Ġwhere Ġusers Ġare Ġoutraged Ġby Ġinformation Ġmisuse , Ġbut Ġcompanies Ġinsist Ġthat Ġusers Ġhave Ġcons ented . ĠThe Ġdiscovery Ġof Ġincreasingly Ġegregious Ġuses Ġof Ġdata Ġby Ġcompanies , Ġsuch Ġas Ġthe Ġscandals Ġinvolving ĠFacebook Ġand ĠCambridge ĠAnaly t ica ĠB IB REF 12 , Ġhave Ġfurther Ġbrought Ġpublic Ġattention Ġto Ġthe Ġprivacy Ġconcerns Ġof Ġthe Ġinternet Ġand Ġubiquitous Ġcomputing . ĠThis Ġmakes Ġprivacy Ġa Ġwell - mot ivated Ġapplication Ġdomain Ġfor ĠN LP Ġresearchers , Ġwhere Ġadvances Ġin Ġenabling Ġusers Ġto Ġquickly Ġidentify Ġthe Ġprivacy Ġissues Ġmost Ġsalient Ġto Ġthem Ġcan Ġpotentially Ġhave Ġlarge Ġreal - world Ġimpact . </s> [ 1 ] https :// play . google . com / store / apps / details ? id = com . g oto keep . keep . int l Ġ[ 2 ] https :// play . google . com / store / apps / details ? id = com . v iber . vo ip Ġ[ 3 ] A Ġquestion Ġmight Ġnot Ġhave Ġany Ġsupporting Ġevidence Ġfor Ġan Ġanswer Ġwithin Ġthe Ġprivacy Ġpolicy . </s> Mot ivated Ġby Ġthis Ġneed , Ġwe Ġcontribute ĠPrivacy Q A , Ġa Ġcorpus Ġconsisting Ġof Ġ17 50 Ġquestions Ġabout Ġthe Ġcontents Ġof Ġprivacy Ġpolicies , Ġpaired Ġwith Ġover Ġ35 00 Ġexpert Ġannotations . ĠThe Ġgoal Ġof Ġthis Ġeffort Ġis Ġto Ġkick start Ġthe Ġdevelopment Ġof Ġquestion - ans w ering Ġmethods Ġfor Ġthis Ġdomain , Ġto Ġaddress Ġthe Ġ( un real istic ) Ġexpectation Ġthat Ġa Ġlarge Ġpopulation Ġshould Ġbe Ġreading Ġmany Ġpolicies Ġper Ġday . ĠIn Ġdoing Ġso , Ġwe Ġidentify Ġseveral Ġunder stud ied Ġchallenges Ġto Ġour Ġability Ġto Ġanswer Ġthese Ġquestions , Ġwith Ġbroad Ġimplications Ġfor Ġsystems Ġseeking Ġto Ġserve Ġusers ' Ġinformation - seeking Ġintent . ĠBy Ġreleasing Ġthis Ġresource , Ġwe Ġhope Ġto Ġprovide Ġan Ġimpetus Ġto Ġdevelop Ġsystems Ġcapable Ġof Ġlanguage Ġunderstanding Ġin Ġthis Ġincreasingly Ġimportant Ġdomain . </s> Related ĠWork </s> Prior Ġwork Ġhas Ġaimed Ġto Ġmake Ġprivacy Ġpolicies Ġeasier Ġto Ġunderstand . ĠPres cript ive Ġapproaches Ġtowards Ġcommunicating Ġprivacy Ġinformation ĠB IB REF 21 , ĠB IB REF 22 , ĠB IB REF 23 Ġhave Ġnot Ġbeen Ġwidely Ġadopted Ġby Ġindustry . ĠRecently , Ġthere Ġhave Ġbeen Ġsignificant Ġresearch Ġeffort Ġdevoted Ġto Ġunderstanding Ġprivacy Ġpolicies Ġby Ġleveraging ĠN LP Ġtechniques ĠB IB REF 24 , ĠB IB REF 25 , ĠB IB REF 26 , ĠB IB REF 27 , ĠB IB REF 28 , Ġespecially Ġby Ġidentifying Ġspecific Ġdata Ġpractices Ġwithin Ġa Ġprivacy Ġpolicy . ĠWe Ġadopt Ġa Ġpersonalized Ġapproach Ġto Ġunderstanding Ġprivacy Ġpolicies , Ġthat Ġallows Ġusers Ġto Ġquery Ġa Ġdocument Ġand Ġselectively Ġexplore Ġcontent Ġsalient Ġto Ġthem . ĠMost Ġsimilar Ġis Ġthe ĠPol is is Q A Ġcorpus ĠB IB REF 29 , Ġwhich Ġexamines Ġquestions Ġusers Ġask Ġcorporations Ġon ĠTwitter . ĠOur Ġapproach Ġdiffers Ġin Ġseveral Ġways : Ġ1 ) ĠThe ĠPrivacy Q A Ġdataset Ġis Ġlarger , Ġcontaining Ġ10 x Ġas Ġmany Ġquestions Ġand Ġanswers . Ġ2 ) ĠAnswers Ġare Ġformulated Ġby Ġdomain Ġexperts Ġwith Ġlegal Ġtraining . Ġ3 ) ĠPrivacy Q A Ġincludes Ġdiverse Ġquestion Ġtypes , Ġincluding Ġun answer able Ġand Ġsubjective Ġquestions . </s> Our Ġwork Ġis Ġalso Ġrelated Ġto Ġreading Ġcomprehension Ġin Ġthe Ġopen Ġdomain , Ġwhich Ġis Ġfrequently Ġbased Ġupon ĠWikipedia Ġpassages ĠB IB REF 16 , ĠB IB REF 17 , ĠB IB REF 15 , ĠB IB REF 30 Ġand Ġnews Ġarticles ĠB IB REF 20 , ĠB IB REF 31 , ĠB IB REF 32 . ĠTable . TA B REF 4 Ġpresents Ġthe Ġdesirable Ġattributes Ġour Ġdataset Ġshares Ġwith Ġpast Ġapproaches . ĠThis Ġwork Ġis Ġalso Ġtied Ġinto Ġresearch Ġin Ġapplying ĠN LP Ġapproaches Ġto Ġlegal Ġdocuments ĠB IB REF 33 , ĠB IB REF 34 , ĠB IB REF 35 , ĠB IB REF 36 , ĠB IB REF 37 , ĠB IB REF 38 , ĠB IB REF 39 . ĠWhile Ġprivacy Ġpolicies Ġhave Ġlegal Ġimplications , Ġtheir Ġintended Ġaudience Ġconsists Ġof Ġthe Ġgeneral Ġpublic Ġrather Ġthan Ġindividuals Ġwith Ġlegal Ġexpertise . ĠThis Ġarrangement Ġis Ġproblematic Ġbecause Ġthe Ġentities Ġthat Ġwrite Ġprivacy Ġpolicies Ġoften Ġhave Ġdifferent Ġgoals Ġthan Ġthe Ġaudience . Ġf eng 2015 app lying , Ġtan - E t Al : 2016 : P 16 - 1 Ġexamine Ġquestion Ġanswering Ġin Ġthe Ġinsurance Ġdomain , Ġanother Ġspecialized Ġdomain Ġsimilar Ġto Ġprivacy , Ġwhere Ġthe Ġintended Ġaudience Ġis Ġthe Ġgeneral Ġpublic . </s> Data ĠCollection </s> We Ġdescribe Ġthe Ġdata Ġcollection Ġmethodology Ġused Ġto Ġconstruct ĠPrivacy Q A . ĠWith Ġthe Ġgoal Ġof Ġachieving Ġbroad Ġcoverage Ġacross Ġapplication Ġtypes , Ġwe Ġcollect Ġprivacy Ġpolicies Ġfrom Ġ35 Ġmobile Ġapplications Ġrepresenting Ġa Ġnumber Ġof Ġdifferent Ġcategories Ġin Ġthe ĠGoogle ĠPlay ĠStore . ĠOne Ġof Ġour Ġgoals Ġis Ġto Ġinclude Ġboth Ġpolicies Ġfrom Ġwell - known Ġapplications , Ġwhich Ġare Ġlikely Ġto Ġhave Ġcarefully - con structed Ġprivacy Ġpolicies , Ġand Ġlesser - known Ġapplications Ġwith Ġsmaller Ġinstall Ġbases , Ġwhose Ġpolicies Ġmight Ġbe Ġconsiderably Ġless Ġsophisticated . ĠThus , Ġsetting Ġ5 Ġmillion Ġinstalls Ġas Ġa Ġthreshold , Ġwe Ġensure Ġeach Ġcategory Ġincludes Ġapplications Ġwith Ġinstalls Ġon Ġboth Ġsides Ġof Ġthis Ġthreshold . ĠAll Ġpolicies Ġincluded Ġin Ġthe Ġcorpus Ġare Ġin ĠEnglish , Ġand Ġwere Ġcollected Ġbefore ĠApril Ġ1 , Ġ2018 , Ġpred ating Ġmany Ġcompanies ' ĠGD PR - focused ĠB IB REF 41 Ġupdates . ĠWe Ġleave Ġit Ġto Ġfuture Ġstudies ĠB IB REF 42 Ġto Ġlook Ġat Ġthe Ġimpact Ġof Ġthe ĠGD PR Ġ( e . g ., Ġto Ġwhat Ġextent ĠGD PR Ġrequirements Ġcontribute Ġto Ġmaking Ġit Ġpossible Ġto Ġprovide Ġusers Ġwith Ġmore Ġinformative Ġanswers , Ġand Ġto Ġwhat Ġextent Ġtheir Ġdisclosures Ġcontinue Ġto Ġomit Ġissues Ġthat Ġmatter Ġto Ġusers ). </s> Data ĠCollection Ġ: :: ĠCrowd s ourced ĠQuestion ĠE lic itation </s> The Ġintended Ġaudience Ġfor Ġprivacy Ġpolicies Ġconsists Ġof Ġthe Ġgeneral Ġpublic . ĠThis Ġinforms Ġthe Ġdecision Ġto Ġelicit Ġquestions Ġfrom Ġcrowd workers Ġon Ġthe Ġcontents Ġof Ġprivacy Ġpolicies . ĠWe Ġchoose Ġnot Ġto Ġshow Ġthe Ġcontents Ġof Ġprivacy Ġpolicies Ġto Ġcrowd workers , Ġa Ġprocedure Ġmotivated Ġby Ġa Ġdesire Ġto Ġavoid Ġinadvert ent Ġbiases ĠB IB REF 43 , ĠB IB REF 44 , ĠB IB REF 45 , ĠB IB REF 46 , ĠB IB REF 47 , Ġand Ġencourage Ġcrowd workers Ġto Ġask Ġa Ġvariety Ġof Ġquestions Ġbeyond Ġonly Ġasking Ġquestions Ġbased Ġon Ġpractices Ġdescribed Ġin Ġthe Ġdocument . </s> Instead , Ġcrowd workers Ġare Ġpresented Ġwith Ġpublic Ġinformation Ġabout Ġa Ġmobile Ġapplication Ġavailable Ġon Ġthe ĠGoogle ĠPlay ĠStore Ġincluding Ġits Ġname , Ġdescription Ġand Ġnavig able Ġscreenshots . ĠFigure ĠFIG REF 9 Ġshows Ġan Ġexample Ġof Ġour Ġuser Ġinterface . ĠCrowd workers Ġare Ġasked Ġto Ġimagine Ġthey Ġhave Ġaccess Ġto Ġa Ġtrusted Ġthird - party Ġprivacy Ġassistant , Ġto Ġwhom Ġthey Ġcan Ġask Ġany Ġprivacy Ġquestion Ġabout Ġa Ġgiven Ġmobile Ġapplication . ĠWe Ġuse Ġthe ĠAmazon ĠMechanical ĠTurk Ġplatform Ġand Ġrecruit Ġcrowd workers Ġwho Ġhave Ġbeen Ġconferred ĠâĢ ľ master âĢ Ŀ Ġstatus Ġand Ġare Ġlocated Ġwithin Ġthe ĠUnited ĠStates Ġof ĠAmerica . ĠTurk ers Ġare Ġasked Ġto Ġprovide Ġfive Ġquestions Ġper Ġmobile Ġapplication , Ġand Ġare Ġpaid Ġ$ 2 Ġper Ġassignment , Ġtaking Ġ~ eight Ġminutes Ġto Ġcomplete Ġthe Ġtask . </s> Data ĠCollection Ġ: :: ĠAnswer ĠSelection </s> To Ġidentify Ġlegally Ġsound Ġanswers , Ġwe Ġrecruit Ġseven Ġexperts Ġwith Ġlegal Ġtraining Ġto Ġconstruct Ġanswers Ġto ĠTur ker Ġquestions . ĠExperts Ġidentify Ġrelevant Ġevidence Ġwithin Ġthe Ġprivacy Ġpolicy , Ġas Ġwell Ġas Ġprovide Ġmeta - ann otation Ġon Ġthe Ġquestion 's Ġrelevance , Ġsubject ivity , ĠO PP - 115 Ġcategory ĠB IB REF 49 , Ġand Ġhow Ġlikely Ġany Ġprivacy Ġpolicy Ġis Ġto Ġcontain Ġthe Ġanswer Ġto Ġthe Ġquestion Ġasked . </s> Data ĠCollection Ġ: :: ĠAnalysis </s> Table . TA B REF 17 Ġpresents Ġaggregate Ġstatistics Ġof Ġthe ĠPrivacy Q A Ġdataset . Ġ17 50 Ġquestions Ġare Ġposed Ġto Ġour Ġimaginary Ġprivacy Ġassistant Ġover Ġ35 Ġmobile Ġapplications Ġand Ġtheir Ġassociated Ġprivacy Ġdocuments . ĠAs Ġan Ġinitial Ġstep , Ġwe Ġformulate Ġthe Ġproblem Ġof Ġanswering Ġuser Ġquestions Ġas Ġan Ġextract ive Ġsentence Ġselection Ġtask , Ġignoring Ġfor Ġnow Ġbackground Ġknowledge , Ġstatistical Ġdata Ġand Ġlegal Ġexpertise Ġthat Ġcould Ġotherwise Ġbe Ġbrought Ġto Ġbear . ĠThe Ġdataset Ġis Ġpartition ed Ġinto Ġa Ġtraining Ġset Ġfeaturing Ġ27 Ġmobile Ġapplications Ġand Ġ13 50 Ġquestions , Ġand Ġa Ġtest Ġset Ġconsisting Ġof Ġ400 Ġquestions Ġover Ġ8 Ġpolicy Ġdocuments . ĠThis Ġensures Ġthat Ġdocuments Ġin Ġtraining Ġand Ġtest Ġsplits Ġare Ġmutually Ġexclusive . ĠEvery Ġquestion Ġis Ġanswered Ġby Ġat Ġleast Ġone Ġexpert . ĠIn Ġaddition , Ġin Ġorder Ġto Ġestimate Ġannotation Ġreliability Ġand Ġprovide Ġfor Ġbetter Ġevaluation , Ġevery Ġquestion Ġin Ġthe Ġtest Ġset Ġis Ġanswered Ġby Ġat Ġleast Ġtwo Ġadditional Ġexperts . </s> Table ĠT AB REF 14 Ġdescribes Ġthe Ġdistribution Ġover Ġfirst Ġwords Ġof Ġquestions Ġposed Ġby Ġcrowd workers . ĠWe Ġalso Ġobserve Ġlow Ġredundancy Ġin Ġthe Ġquestions Ġposed Ġby Ġcrowd workers Ġover Ġeach Ġpolicy , Ġwith Ġeach Ġpolicy Ġreceiving Ġ~ 49 . 94 Ġunique Ġquestions Ġdespite Ġcrowd workers Ġindependently Ġposing Ġquestions . ĠQuestions Ġare Ġon Ġaverage Ġ8 . 4 Ġwords Ġlong . ĠAs Ġdeclining Ġto Ġanswer Ġa Ġquestion Ġcan Ġbe Ġa Ġlegally Ġsound Ġresponse Ġbut Ġis Ġseldom Ġpractically Ġuseful , Ġanswers Ġto Ġquestions Ġwhere Ġa Ġminority Ġof Ġexperts Ġabst ain Ġto Ġanswer Ġare Ġfiltered Ġfrom Ġthe Ġdataset . ĠPrivacy Ġpolicies Ġare Ġ~ 3000 Ġwords Ġlong Ġon Ġaverage . ĠThe Ġanswers Ġto Ġthe Ġquestion Ġasked Ġby Ġthe Ġusers Ġtypically Ġhave Ġ~ 100 Ġwords Ġof Ġevidence Ġin Ġthe Ġprivacy Ġpolicy Ġdocument . </s> Data ĠCollection Ġ: :: ĠAnalysis Ġ: :: ĠCategories Ġof ĠQuestions </s> Questions Ġare Ġorganized Ġunder Ġnine Ġcategories Ġfrom Ġthe ĠO PP - 115 ĠCorpus Ġannotation Ġscheme ĠB IB REF 49 : </s> First ĠParty ĠCollection / Use : ĠWhat , Ġwhy Ġand Ġhow Ġinformation Ġis Ġcollected Ġby Ġthe Ġservice Ġprovider </s> Third ĠParty ĠSharing / Collection : ĠWhat , Ġwhy Ġand Ġhow Ġinformation Ġshared Ġwith Ġor Ġcollected Ġby Ġthird Ġparties </s> Data ĠSecurity : ĠProtection Ġmeasures Ġfor Ġuser Ġinformation </s> Data ĠRet ention : ĠHow Ġlong Ġuser Ġinformation Ġwill Ġbe Ġstored </s> User ĠChoice / Control : ĠControl Ġoptions Ġavailable Ġto Ġusers </s> User ĠAccess , ĠEdit Ġand ĠDe let ion : ĠIf / how Ġusers Ġcan Ġaccess , Ġedit Ġor Ġdelete Ġinformation </s> Policy ĠChange : ĠIn forming Ġusers Ġif Ġpolicy Ġinformation Ġhas Ġbeen Ġchanged </s> International Ġand ĠSpecific ĠAud iences : ĠPractices Ġpertaining Ġto Ġa Ġspecific Ġgroup Ġof Ġusers </s> Other : ĠGeneral Ġtext , Ġcontact Ġinformation Ġor Ġpractices Ġnot Ġcovered Ġby Ġother Ġcategories . </s> For Ġeach Ġquestion , Ġdomain Ġexperts Ġindicate Ġone Ġor Ġmore Ġrelevant ĠO PP - 115 Ġcategories . ĠWe Ġmark Ġa Ġcategory Ġas Ġrelevant Ġto Ġa Ġquestion Ġif Ġit Ġis Ġidentified Ġas Ġsuch Ġby Ġat Ġleast Ġtwo Ġannot ators . ĠIf Ġno Ġsuch Ġcategory Ġexists , Ġthe Ġcategory Ġis Ġmarked Ġas Ġ` Other ' Ġif Ġat le ast Ġone Ġannot ator Ġhas Ġidentified Ġthe Ġ` Other ' Ġcategory Ġto Ġbe Ġrelevant . ĠIf Ġneither Ġof Ġthese Ġconditions Ġis Ġsatisfied , Ġwe Ġlabel Ġthe Ġquestion Ġas Ġhaving Ġno Ġagreement . ĠThe Ġdistribution Ġof Ġquestions Ġin Ġthe Ġcorpus Ġacross ĠO PP - 115 Ġcategories Ġis Ġas Ġshown Ġin ĠTable . TA B REF 16 . ĠFirst Ġparty Ġand Ġthird Ġparty Ġrelated Ġquestions Ġare Ġthe Ġlargest Ġcategories , Ġforming Ġnearly Ġ66 . 4 % Ġof Ġall Ġquestions Ġasked Ġto Ġthe Ġprivacy Ġassistant . </s> Data ĠCollection Ġ: :: ĠAnalysis Ġ: :: ĠAnswer ĠVal idation </s> When Ġdo Ġexperts Ġdisagree ? ĠWe Ġwould Ġlike Ġto Ġanalyze Ġthe Ġreasons Ġfor Ġpotential Ġdisagreement Ġon Ġthe Ġannotation Ġtask , Ġto Ġensure Ġdisagreements Ġarise Ġdue Ġto Ġvalid Ġdifferences Ġin Ġopinion Ġrather Ġthan Ġlack Ġof Ġadequate Ġspecification Ġin Ġannotation Ġguidelines . ĠIt Ġis Ġimportant Ġto Ġnote Ġthat Ġthe Ġannot ators Ġare Ġexperts Ġrather Ġthan Ġcrowd workers . ĠAccordingly , Ġtheir Ġjud gements Ġcan Ġbe Ġconsidered Ġvalid , Ġlegally - informed Ġopinions Ġeven Ġwhen Ġtheir Ġperspectives Ġdiffer . ĠFor Ġthe Ġsake Ġof Ġthis Ġquestion Ġwe Ġrandomly Ġsample Ġ100 Ġinstances Ġin Ġthe Ġtest Ġdata Ġand Ġanalyze Ġthem Ġfor Ġlikely Ġreasons Ġfor Ġdisagreements . ĠWe Ġconsider Ġa Ġdisagreement Ġto Ġhave Ġoccurred Ġwhen Ġmore Ġthan Ġone Ġexpert Ġdoes Ġnot Ġagree Ġwith Ġthe Ġmajority Ġconsensus . ĠBy Ġdisagreement Ġwe Ġmean Ġthere Ġis Ġno Ġoverlap Ġbetween Ġthe Ġtext Ġidentified Ġas Ġrelevant Ġby Ġone Ġexpert Ġand Ġanother . </s> We Ġfind Ġthat Ġthe Ġannot ators Ġagree Ġon Ġthe Ġanswer Ġfor Ġ74 % Ġof Ġthe Ġquestions , Ġeven Ġif Ġthe Ġsupporting Ġevidence Ġthey Ġidentify Ġis Ġnot Ġidentical Ġi . e Ġfull Ġoverlap . ĠThey Ġdisagree Ġon Ġthe Ġremaining Ġ26 %. ĠSources Ġof Ġapparent Ġdisagreement Ġcorrespond Ġto Ġsituations Ġwhen Ġdifferent Ġexperts : Ġhave Ġdiffering Ġinterpretations Ġof Ġquestion Ġintent Ġ( 11 %) Ġ( for Ġexample , Ġwhen Ġa Ġuser Ġasks Ġ' who Ġcan Ġcontact Ġme Ġthrough Ġthe Ġapp ', Ġthe Ġquestions Ġadmits Ġmultiple Ġinterpretations , Ġincluding Ġseeking Ġinformation Ġabout Ġthe Ġfeatures Ġof Ġthe Ġapp , Ġasking Ġabout Ġfirst Ġparty Ġcollection / use Ġof Ġdata Ġor Ġasking Ġabout Ġthird Ġparty Ġcollection / use Ġof Ġdata ), Ġidentify Ġdifferent Ġsources Ġof Ġevidence Ġfor Ġquestions Ġthat Ġask Ġif Ġa Ġpractice Ġis Ġperformed Ġor Ġnot Ġ( 4 %), Ġhave Ġdiffering Ġinterpretations Ġof Ġpolicy Ġcontent Ġ( 3 %), Ġidentify Ġa Ġpartial Ġanswer Ġto Ġa Ġquestion Ġin Ġthe Ġprivacy Ġpolicy Ġ( 2 %) Ġ( for Ġexample , Ġwhen Ġthe Ġuser Ġasks Ġ` who Ġis Ġallowed Ġto Ġuse Ġthe Ġapp ' Ġa Ġmajority Ġof Ġour Ġannot ators Ġdecline Ġto Ġanswer , Ġbut Ġthe Ġremaining Ġannot ators Ġhighlight Ġpartial Ġevidence Ġin Ġthe Ġprivacy Ġpolicy Ġwhich Ġstates Ġthat Ġchildren Ġunder Ġthe Ġage Ġof Ġ13 Ġare Ġnot Ġallowed Ġto Ġuse Ġthe Ġapp ), Ġand Ġother Ġlegitimate Ġsources Ġof Ġdisagreement Ġ( 6 %) Ġwhich Ġinclude Ġpersonal Ġsubjective Ġviews Ġof Ġthe Ġannot ators Ġ( for Ġexample , Ġwhen Ġthe Ġuser Ġasks Ġ` is Ġmy ĠDNA Ġinformation Ġused Ġin Ġany Ġway Ġother Ġthan Ġwhat Ġis Ġspecified ', Ġsome Ġexperts Ġconsider Ġthe Ġboiler plate Ġtext Ġof Ġthe Ġprivacy Ġpolicy Ġwhich Ġstates Ġthat Ġit Ġab ides Ġto Ġpractices Ġdescribed Ġin Ġthe Ġpolicy Ġdocument Ġas Ġsufficient Ġevidence Ġto Ġanswer Ġthis Ġquestion , Ġwhereas Ġothers Ġdo Ġnot ). </s> Exper imental ĠSetup </s> We Ġevaluate Ġthe Ġability Ġof Ġmachine Ġlearning Ġmethods Ġto Ġidentify Ġrelevant Ġevidence Ġfor Ġquestions Ġin Ġthe Ġprivacy Ġdomain . ĠWe Ġestablish Ġbas elines Ġfor Ġthe Ġsubt ask Ġof Ġdeciding Ġon Ġthe Ġanswer ability Ġ( Â§ SEC REF 33 ) Ġof Ġa Ġquestion , Ġas Ġwell Ġas Ġthe Ġoverall Ġtask Ġof Ġidentifying Ġevidence Ġfor Ġquestions Ġfrom Ġpolicies Ġ( Â§ SEC REF 37 ). ĠWe Ġdescribe Ġaspects Ġof Ġthe Ġquestion Ġthat Ġcan Ġrender Ġit Ġun answer able Ġwithin Ġthe Ġprivacy Ġdomain Ġ( Â§ SEC REF 41 ). </s> Exper imental ĠSetup Ġ: :: ĠAnswer ability ĠIdentification ĠBas elines </s> We Ġdefine Ġanswer ability Ġidentification Ġas Ġa Ġbinary Ġclassification Ġtask , Ġevaluating Ġmodel Ġability Ġto Ġpredict Ġif Ġa Ġquestion Ġcan Ġbe Ġanswered , Ġgiven Ġa Ġquestion Ġin Ġisolation . ĠThis Ġcan Ġserve Ġas Ġa Ġprior Ġfor Ġdownstream Ġquestion - ans w ering . ĠWe Ġdescribe Ġthree Ġbas elines Ġon Ġthe Ġanswer ability Ġtask , Ġand Ġfind Ġthey Ġconsiderably Ġimprove Ġperformance Ġover Ġa Ġmajority - class Ġbaseline . </s> S VM : ĠWe Ġdefine Ġ3 Ġsets Ġof Ġfeatures Ġto Ġcharacterize Ġeach Ġquestion . ĠThe Ġfirst Ġis Ġa Ġsimple Ġbag - of - words Ġset Ġof Ġfeatures Ġover Ġthe Ġquestion Ġ( S VM - B OW ), Ġthe Ġsecond Ġis Ġbag - of - words Ġfeatures Ġof Ġthe Ġquestion Ġas Ġwell Ġas Ġlength Ġof Ġthe Ġquestion Ġin Ġwords Ġ( S VM - B OW Ġ+ ĠL EN ), Ġand Ġlast ly Ġwe Ġextract Ġbag - of - words Ġfeatures , Ġlength Ġof Ġthe Ġquestion Ġin Ġwords Ġas Ġwell Ġas Ġpart - of - speech Ġtags Ġfor Ġthe Ġquestion Ġ( S VM - B OW Ġ+ ĠL EN Ġ+ ĠPOS ). ĠThis Ġresults Ġin Ġvectors Ġof Ġ200 , Ġ201 Ġand Ġ228 Ġdimensions Ġrespectively , Ġwhich Ġare Ġprovided Ġto Ġan ĠS VM Ġwith Ġa Ġlinear Ġkernel . </s> CNN : ĠWe Ġutilize Ġa ĠCNN Ġneural Ġenc oder Ġfor Ġanswer ability Ġprediction . ĠWe Ġuse ĠGl o Ve Ġword Ġembed d ings ĠB IB REF 50 , Ġand Ġa Ġfilter Ġsize Ġof Ġ5 Ġwith Ġ64 Ġfilters Ġto Ġencode Ġquestions . </s> BER T : ĠB ERT ĠB IB REF 51 Ġis Ġa Ġbid irection al Ġtransformer - based Ġlanguage - model ĠB IB REF 52 . ĠWe Ġfine - t une ĠB ERT - base Ġon Ġour Ġbinary Ġanswer ability Ġidentification Ġtask Ġwith Ġa Ġlearning Ġrate Ġof Ġ2 e - 5 Ġfor Ġ3 Ġepoch s , Ġwith Ġa Ġmaximum Ġsequence Ġlength Ġof Ġ128 . </s> Exper imental ĠSetup Ġ: :: ĠPrivacy ĠQuestion ĠAn sw ering </s> Our Ġgoal Ġis Ġto Ġidentify Ġevidence Ġwithin Ġa Ġprivacy Ġpolicy Ġfor Ġquestions Ġasked Ġby Ġa Ġuser . ĠThis Ġis Ġframed Ġas Ġan Ġanswer Ġsentence Ġselection Ġtask , Ġwhere Ġmodels Ġidentify Ġa Ġset Ġof Ġevidence Ġsentences Ġfrom Ġall Ġcandidate Ġsentences Ġin Ġeach Ġpolicy . </s> Exper imental ĠSetup Ġ: :: ĠPrivacy ĠQuestion ĠAn sw ering Ġ: :: ĠEvaluation ĠMet ric </s> Our Ġevaluation Ġmetric Ġfor Ġanswer - sent ence Ġselection Ġis Ġsentence - level ĠF 1 , Ġimplemented Ġsimilar Ġto ĠB IB REF 30 , ĠB IB REF 16 . ĠPrecision Ġand Ġrecall Ġare Ġimplemented Ġby Ġmeasuring Ġthe Ġoverlap Ġbetween Ġpredicted Ġsentences Ġand Ġsets Ġof Ġgold - reference Ġsentences . ĠWe Ġreport Ġthe Ġaverage Ġof Ġthe Ġmaximum ĠF 1 Ġfrom Ġeach Ġn $ -$ 1 Ġsubset , Ġin Ġrelation Ġto Ġthe Ġheld out Ġreference . </s> Exper imental ĠSetup Ġ: :: ĠPrivacy ĠQuestion ĠAn sw ering Ġ: :: ĠBas elines </s> We Ġdescribe Ġbas elines Ġon Ġthis Ġtask , Ġincluding Ġa Ġhuman Ġperformance Ġbaseline . </s> No - Answer ĠBas eline Ġ( NA ) Ġ: ĠMost Ġof Ġthe Ġquestions Ġwe Ġreceive Ġare Ġdifficult Ġto Ġanswer Ġin Ġa Ġlegally - sound Ġway Ġon Ġthe Ġbasis Ġof Ġinformation Ġpresent Ġin Ġthe Ġprivacy Ġpolicy . ĠWe Ġestablish Ġa Ġsimple Ġbaseline Ġto Ġquantify Ġthe Ġeffect Ġof Ġidentifying Ġevery Ġquestion Ġas Ġun answer able . </s> Word ĠCount ĠBas eline Ġ: ĠTo Ġquantify Ġthe Ġeffect Ġof Ġusing Ġsimple Ġlex ical Ġmatching Ġto Ġanswer Ġthe Ġquestions , Ġwe Ġretrieve Ġthe Ġtop Ġcandidate Ġpolicy Ġsentences Ġfor Ġeach Ġquestion Ġusing Ġa Ġword Ġcount Ġbaseline ĠB IB REF 53 , Ġwhich Ġcounts Ġthe Ġnumber Ġof Ġquestion Ġwords Ġthat Ġalso Ġappear Ġin Ġa Ġsentence . ĠWe Ġinclude Ġthe Ġtop Ġ2 , Ġ3 Ġand Ġ5 Ġcandidates Ġas Ġbas elines . </s> BER T : ĠWe Ġimplement Ġtwo ĠB ERT - based Ġbas elines ĠB IB REF 51 Ġfor Ġevidence Ġidentification . ĠFirst , Ġwe Ġtrain ĠB ERT Ġon Ġeach Ġquery - policy Ġsentence Ġpair Ġas Ġa Ġbinary Ġclassification Ġtask Ġto Ġidentify Ġif Ġthe Ġsentence Ġis Ġevidence Ġfor Ġthe Ġquestion Ġor Ġnot Ġ( B ert ). ĠWe Ġalso Ġexperiment Ġwith Ġa Ġtwo - stage Ġclass ifier , Ġwhere Ġwe Ġseparately Ġtrain Ġthe Ġmodel Ġon Ġquestions Ġonly Ġto Ġpredict Ġanswer ability . ĠAt Ġinference Ġtime , Ġif Ġthe Ġanswer able Ġclass ifier Ġpredicts Ġthe Ġquestion Ġis Ġanswer able , Ġthe Ġevidence Ġidentification Ġclass ifier Ġproduces Ġa Ġset Ġof Ġcandidate Ġsentences Ġ( B ert Ġ+ ĠUn answer able ). </s> Human ĠPerformance : ĠWe Ġpick Ġeach Ġreference Ġanswer Ġprovided Ġby Ġan Ġannot ator , Ġand Ġcompute Ġthe ĠF 1 Ġwith Ġrespect Ġto Ġthe Ġremaining Ġreferences , Ġas Ġdescribed Ġin Ġsection Ġ4 . 2 . 1 . ĠEach Ġreference Ġanswer Ġis Ġtreated Ġas Ġthe Ġprediction , Ġand Ġthe Ġremaining Ġn - 1 Ġanswers Ġare Ġtreated Ġas Ġthe Ġgold Ġreference . ĠThe Ġaverage Ġof Ġthe Ġmaximum ĠF 1 Ġacross Ġall Ġreference Ġanswers Ġis Ġcomputed Ġas Ġthe Ġhuman Ġbaseline . </s> Results Ġand ĠDiscussion </s> The Ġresults Ġof Ġthe Ġanswer ability Ġbas elines Ġare Ġpresented Ġin ĠTable ĠT AB REF 31 , Ġand Ġon Ġanswer Ġsentence Ġselection Ġin ĠTable ĠT AB REF 32 . ĠWe Ġobserve Ġthat Ġb ert Ġexhibits Ġthe Ġbest Ġperformance Ġon Ġa Ġbinary Ġanswer ability Ġidentification Ġtask . ĠHowever , Ġmost Ġbas elines Ġconsiderably Ġexceed Ġthe Ġperformance Ġof Ġa Ġmajority - class Ġbaseline . ĠThis Ġsuggests Ġconsiderable Ġinformation Ġin Ġthe Ġquestion , Ġindicating Ġit 's Ġpossible Ġanswer ability Ġwithin Ġthis Ġdomain . </s> Table . TA B REF 32 Ġdescribes Ġthe Ġperformance Ġof Ġour Ġbas elines Ġon Ġthe Ġanswer Ġsentence Ġselection Ġtask . ĠThe ĠNo - answer Ġ( NA ) Ġbaseline Ġperforms Ġat Ġ28 ĠF 1 , Ġproviding Ġa Ġlower Ġbound Ġon Ġperformance Ġat Ġthis Ġtask . ĠWe Ġobserve Ġthat Ġour Ġbest - performing Ġbaseline , ĠBert Ġ+ ĠUn answer able Ġachieves Ġan ĠF 1 Ġof Ġ39 . 8 . ĠThis Ġsuggest Ġthat Ġb ert Ġis Ġcapable Ġof Ġmaking Ġsome Ġprogress Ġtowards Ġanswering Ġquestions Ġin Ġthis Ġdifficult Ġdomain , Ġwhile Ġstill Ġleaving Ġconsiderable Ġhead room Ġfor Ġimprovement Ġto Ġreach Ġhuman Ġperformance . ĠBert Ġ+ ĠUn answer able Ġperformance Ġsuggests Ġthat Ġincorporating Ġinformation Ġabout Ġanswer ability Ġcan Ġhelp Ġin Ġthis Ġdifficult Ġdomain . ĠWe Ġexamine Ġthis Ġchallenging Ġphenomena Ġof Ġun answer ability Ġfurther Ġin ĠSection Ġ. </s> Results Ġand ĠDiscussion Ġ: :: ĠError ĠAnalysis </s> Dis ag reements Ġare Ġanalyzed Ġbased Ġon Ġthe ĠO PP - 115 Ġcategories Ġof Ġeach Ġquestion Ġ( Table . TA B REF 34 ). ĠWe Ġcompare Ġour Ġbest Ġperforming ĠB ERT Ġvariant Ġagainst Ġthe ĠNA Ġmodel Ġand Ġhuman Ġperformance . ĠWe Ġobserve Ġsignificant Ġroom Ġfor Ġimprovement Ġacross Ġall Ġcategories Ġof Ġquestions Ġbut Ġespecially Ġfor Ġfirst Ġparty , Ġthird Ġparty Ġand Ġdata Ġretention Ġcategories . </s> We Ġanalyze Ġthe Ġperformance Ġof Ġour Ġstrongest ĠB ERT Ġvariant , Ġto Ġidentify Ġclasses Ġof Ġerrors Ġand Ġdirections Ġfor Ġfuture Ġimprovement Ġ( Table . 8 ). ĠWe Ġobserve Ġthat Ġa Ġmajority Ġof Ġanswer ability Ġmistakes Ġmade Ġby Ġthe ĠB ERT Ġmodel Ġare Ġquestions Ġwhich Ġare Ġin Ġfact Ġanswer able , Ġbut Ġare Ġidentified Ġas Ġun answer able Ġby ĠB ERT . ĠWe Ġobserve Ġthat ĠB ERT Ġmakes Ġ124 Ġsuch Ġmistakes Ġon Ġthe Ġtest Ġset . ĠWe Ġcollect Ġexpert Ġjudgments Ġon Ġrelevance , Ġsubject ivity Ġ, Ġsilence Ġand Ġinformation Ġabout Ġhow Ġlikely Ġthe Ġquestion Ġis Ġto Ġbe Ġanswered Ġfrom Ġthe Ġprivacy Ġpolicy Ġfrom Ġour Ġexperts . ĠWe Ġfind Ġthat Ġmost Ġof Ġthese Ġmistakes Ġare Ġrelevant Ġquestions . ĠHowever Ġmany Ġof Ġthem Ġwere Ġidentified Ġas Ġsubjective Ġby Ġthe Ġannot ators , Ġand Ġat Ġleast Ġone Ġannot ator Ġmarked Ġ19 Ġof Ġthese Ġquestions Ġas Ġhaving Ġno Ġanswer Ġwithin Ġthe Ġprivacy Ġpolicy . ĠHowever , Ġonly Ġ6 Ġof Ġthese Ġquestions Ġwere Ġunexpected Ġor Ġdo Ġnot Ġusually Ġhave Ġan Ġanswer Ġin Ġprivacy Ġpolicies . ĠThese Ġfindings Ġsuggest Ġthat Ġa Ġmore Ġnuanced Ġunderstanding Ġof Ġanswer ability Ġmight Ġhelp Ġimprove Ġmodel Ġperformance Ġin Ġhis Ġchallenging Ġdomain . </s> Results Ġand ĠDiscussion Ġ: :: ĠWhat Ġmakes ĠQuestions ĠUn answer able ? </s> We Ġfurther Ġask Ġlegal Ġexperts Ġto Ġidentify Ġpotential Ġcauses Ġof Ġun answer ability Ġof Ġquestions . ĠThis Ġanalysis Ġhas Ġconsiderable Ġimplications . ĠWhile Ġpast Ġwork ĠB IB REF 17 Ġhas Ġtreated Ġun answer able Ġquestions Ġas Ġhom ogeneous , Ġa Ġquestion Ġanswering Ġsystem Ġmight Ġwish Ġto Ġhave Ġdifferent Ġtreatments Ġfor Ġdifferent Ġcategories Ġof Ġ` un answer able ' Ġquestions . ĠThe Ġfollowing Ġfactors Ġwere Ġidentified Ġto Ġplay Ġa Ġrole Ġin Ġun answer ability : </s> In com pre hens ibility : ĠIf Ġa Ġquestion Ġis Ġincomprehensible Ġto Ġthe Ġextent Ġthat Ġits Ġmeaning Ġis Ġnot Ġintellig ible . </s> Re lev ance : ĠIs Ġthis Ġquestion Ġin Ġthe Ġscope Ġof Ġwhat Ġcould Ġbe Ġanswered Ġby Ġreading Ġthe Ġprivacy Ġpolicy . </s> Ill - formed ness : ĠIs Ġthis Ġquestion Ġambiguous Ġor Ġvague . ĠAn Ġambiguous Ġstatement Ġwill Ġtypically Ġcontain Ġexpressions Ġthat Ġcan Ġrefer Ġto Ġmultiple Ġpotential Ġexplanations , Ġwhereas Ġa Ġvague Ġstatement Ġcarries Ġa Ġconcept Ġwith Ġan Ġunclear Ġor Ġsoft Ġdefinition . </s> Sil ence : ĠOther Ġpolicies Ġanswer Ġthis Ġtype Ġof Ġquestion Ġbut Ġthis Ġone Ġdoes Ġnot . </s> A typ ical ity : ĠThe Ġquestion Ġis Ġof Ġa Ġnature Ġsuch Ġthat Ġit Ġis Ġunlikely Ġfor Ġany Ġpolicy Ġpolicy Ġto Ġhave Ġan Ġanswer Ġto Ġthe Ġquestion . </s> Our Ġexperts Ġattempt Ġto Ġidentify Ġthe Ġdifferent Ġ` un answer able ' Ġfactors Ġfor Ġall Ġ5 73 Ġsuch Ġquestions Ġin Ġthe Ġcorpus . Ġ4 . 18 % Ġof Ġthe Ġquestions Ġwere Ġidentified Ġas Ġbeing Ġincomprehensible Ġ( for Ġexample , Ġ` any Ġdifficulties Ġto Ġoccupy Ġthe Ġprivacy Ġassistant '). ĠAmong st Ġthe Ġcomprehend able Ġquestions , Ġ50 % Ġwere Ġidentified Ġas Ġlikely Ġto Ġhave Ġan Ġanswer Ġwithin Ġthe Ġprivacy Ġpolicy , Ġ33 . 1 % Ġwere Ġidentified Ġas Ġbeing Ġprivacy - related Ġquestions Ġbut Ġnot Ġwithin Ġthe Ġscope Ġof Ġa Ġprivacy Ġpolicy Ġ( e . g ., Ġ' has ĠV iber Ġhad Ġany Ġprivacy Ġbreaches Ġin Ġthe Ġpast ?' ) Ġand Ġ16 . 9 % Ġof Ġquestions Ġwere Ġidentified Ġas Ġcompletely Ġout - of - scope Ġ( e . g ., Ġ` ' will Ġthe Ġapp Ġconsume Ġmuch Ġspace ?' ). ĠIn Ġthe Ġquestions Ġidentified Ġas Ġrelevant , Ġ32 % Ġwere Ġill - formed Ġquestions Ġthat Ġwere Ġphr ased Ġby Ġthe Ġuser Ġin Ġa Ġmanner Ġconsidered Ġvague Ġor Ġambiguous . ĠOf Ġthe Ġquestions Ġthat Ġwere Ġboth Ġrelevant Ġas Ġwell Ġas Ġ` well - formed ', Ġ95 . 7 % Ġof Ġthe Ġquestions Ġwere Ġnot Ġanswered Ġby Ġthe Ġpolicy Ġin Ġquestion Ġbut Ġit Ġwas Ġreasonable Ġto Ġexpect Ġthat Ġa Ġprivacy Ġpolicy Ġwould Ġcontain Ġan Ġanswer . ĠThe Ġremaining Ġ4 . 3 % Ġwere Ġdescribed Ġas Ġreasonable Ġquestions , Ġbut Ġof Ġa Ġnature Ġgenerally Ġnot Ġdiscussed Ġin Ġprivacy Ġpolicies . ĠThis Ġsuggests Ġthat Ġthe Ġanswer ability Ġof Ġquestions Ġover Ġprivacy Ġpolicies Ġis Ġa Ġcomplex Ġissue , Ġand Ġfuture Ġsystems Ġshould Ġconsider Ġeach Ġof Ġthese Ġfactors Ġwhen Ġserving Ġuser 's Ġinformation Ġseeking Ġintent . </s> We Ġexamine Ġa Ġlarge - scale Ġdataset Ġof ĠâĢ ľ natural âĢ Ŀ Ġun answer able Ġquestions ĠB IB REF 54 Ġbased Ġon Ġreal Ġuser Ġsearch Ġengine Ġqueries Ġto Ġidentify Ġif Ġsimilar Ġun answer ability Ġfactors Ġexist . ĠIt Ġis Ġimportant Ġto Ġnote Ġthat Ġthese Ġquestions Ġhave Ġpreviously Ġbeen Ġfiltered , Ġaccording Ġto Ġa Ġcriteria Ġfor Ġbad Ġquestions Ġdefined Ġas ĠâĢ ľ ( quest ions Ġthat Ġare ) Ġambiguous , Ġincomprehensible , Ġdependent Ġon Ġclear Ġfalse Ġpresupp os itions , Ġopinion - seeking , Ġor Ġnot Ġclearly Ġa Ġrequest Ġfor Ġfactual Ġinformation . âĢ Ŀ ĠAn not ators Ġmade Ġthe Ġdecision Ġbased Ġon Ġthe Ġcontent Ġof Ġthe Ġquestion Ġwithout Ġviewing Ġthe Ġequivalent ĠWikipedia Ġpage . ĠWe Ġrandomly Ġsample Ġ100 Ġquestions Ġfrom Ġthe Ġdevelopment Ġset Ġwhich Ġwere Ġidentified Ġas Ġun answer able , Ġand Ġfind Ġthat Ġ20 % Ġof Ġthe Ġquestions Ġare Ġnot Ġquestions Ġ( e . g ., ĠâĢ ľ all ĠI Ġwant Ġfor Ġchrist mas Ġis Ġyou Ġmar iah Ġcare y Ġtour âĢ Ŀ ). Ġ12 % Ġof Ġquestions Ġare Ġunlikely Ġto Ġever Ġcontain Ġan Ġanswer Ġon ĠWikipedia , Ġcorresponding Ġclosely Ġto Ġour Ġat yp ical ity Ġcategory . Ġ3 % Ġof Ġquestions Ġare Ġunlikely Ġto Ġhave Ġan Ġanswer Ġanywhere Ġ( e . g ., Ġ` what Ġguides ĠSanta Ġhome Ġafter Ġhe Ġhas Ġdelivered Ġpresents ?' ). Ġ7 % Ġof Ġquestions Ġare Ġincomplete Ġor Ġopen - ended Ġ( e . g ., Ġ` the Ġsouth Ġwest Ġwind Ġblows Ġacross Ġnig eria Ġbetween '). Ġ3 % Ġof Ġquestions Ġhave Ġan Ġunres ol vable Ġcore ference Ġ( e . g ., Ġ` how Ġdo Ġi Ġget Ġto ĠWarsaw ĠMissouri Ġfrom Ġhere '). Ġ4 % Ġof Ġquestions Ġare Ġvague , Ġand Ġa Ġfurther Ġ7 % Ġhave Ġunknown Ġsources Ġof Ġerror . Ġ2 % Ġstill Ġcontain Ġfalse Ġpresupp os itions Ġ( e . g ., Ġ` what Ġis Ġthe Ġonly Ġfruit Ġthat Ġdoes Ġnot Ġhave Ġseeds ?' ) Ġand Ġthe Ġremaining Ġ42 % Ġdo Ġnot Ġhave Ġan Ġanswer Ġwithin Ġthe Ġdocument . ĠThis Ġreinforces Ġour Ġbelief Ġthat Ġthough Ġthey Ġhave Ġbeen Ġunder stud ied Ġin Ġpast Ġwork , Ġany Ġquestion Ġanswering Ġsystem Ġinteracting Ġwith Ġreal Ġusers Ġshould Ġexpect Ġto Ġreceive Ġsuch Ġun anticipated Ġand Ġun answer able Ġquestions . </s> Conclusion </s> We Ġpresent ĠPrivacy Q A , Ġthe Ġfirst Ġsignificant Ġcorpus Ġof Ġprivacy Ġpolicy Ġquestions Ġand Ġmore Ġthan Ġ35 00 Ġexpert Ġannotations Ġof Ġrelevant Ġanswers . ĠThe Ġgoal Ġof Ġthis Ġwork Ġis Ġto Ġpromote Ġquestion - ans w ering Ġresearch Ġin Ġthe Ġspecialized Ġprivacy Ġdomain , Ġwhere Ġit Ġcan Ġhave Ġlarge Ġreal - world Ġimpact . ĠStrong Ġneural Ġbas elines Ġon ĠPrivacy Q A Ġachieve Ġa Ġperformance Ġof Ġonly Ġ39 . 8 ĠF 1 Ġon Ġthis Ġcorpus , Ġindicating Ġconsiderable Ġroom Ġfor Ġfuture Ġresearch . ĠFurther , Ġwe Ġshed Ġlight Ġon Ġseveral Ġimportant Ġconsiderations Ġthat Ġaffect Ġthe Ġanswer ability Ġof Ġquestions . ĠWe Ġhope Ġthis Ġcontribution Ġleads Ġto Ġmult idis ciplinary Ġefforts Ġto Ġprecisely Ġunderstand Ġuser Ġintent Ġand Ġreconcile Ġit Ġwith Ġinformation Ġin Ġpolicy Ġdocuments , Ġfrom Ġboth Ġthe Ġprivacy Ġand ĠN LP Ġcommunities . </s> Acknowled gements </s> This Ġresearch Ġwas Ġsupported Ġin Ġpart Ġby Ġgrants Ġfrom Ġthe ĠNational ĠScience ĠFoundation ĠSecure Ġand ĠTrust worthy ĠComputing Ġprogram Ġ( C NS - 13 30 596 , ĠCNS - 13 30 214 , ĠCNS - 15 - 139 57 , ĠCNS - 18 013 16 , ĠCNS - 19 14 486 , ĠCNS - 19 14 444 ) Ġand Ġa ĠDAR PA ĠBrand e is Ġgrant Ġon ĠPersonal ized ĠPrivacy ĠAssist ants Ġ( FA 87 50 - 15 - 2 - 0 277 ). ĠThe ĠUS ĠGovernment Ġis Ġauthorized Ġto Ġreproduce Ġand Ġdistribute Ġre prints Ġfor ĠGovernment al Ġpurposes Ġnot Ġwith standing Ġany Ġcopyright Ġnotation . ĠThe Ġviews Ġand Ġconclusions Ġcontained Ġherein Ġare Ġthose Ġof Ġthe Ġauthors Ġand Ġshould Ġnot Ġbe Ġinterpreted Ġas Ġnecessarily Ġrepresenting Ġthe Ġofficial Ġpolicies Ġor Ġendorsements , Ġeither Ġexpressed Ġor Ġimplied , Ġof Ġthe ĠNS F , ĠDAR PA , Ġor Ġthe ĠUS ĠGovernment . ĠThe Ġauthors Ġwould Ġlike Ġto Ġextend Ġtheir Ġgratitude Ġto ĠElias ĠWright , ĠGian ĠMas ci oli , ĠKi ara ĠPill ay , ĠHarrison ĠKay , ĠEl iel ĠTal o , ĠAlexander ĠF age lla Ġand ĠN . ĠCameron ĠRussell Ġfor Ġproviding Ġtheir Ġvaluable Ġexpertise Ġand Ġinsight Ġto Ġthis Ġeffort . ĠThe Ġauthors Ġare Ġalso Ġgrateful Ġto ĠEdu ard ĠH ovy , ĠLor rie ĠCran or , ĠFlor ian ĠScha ub , ĠJoel ĠRe iden berg , ĠAd ity a ĠPot uk uchi Ġand ĠIgor ĠSh aly min ov Ġfor Ġhelpful Ġdiscussions Ġrelated Ġto Ġthis Ġwork , Ġand Ġto Ġthe Ġthree Ġanonymous Ġreviewers Ġof Ġthis Ġdraft Ġfor Ġtheir Ġconstructive Ġfeedback . ĠFinally , Ġthe Ġauthors Ġwould Ġlike Ġto Ġthank Ġall Ġcrowd workers Ġwho Ġcons ented Ġto Ġparticipate Ġin Ġthis Ġstudy .
Introduction Privacy policies are the documents which disclose the ways in which a company gathers, uses, shares and manages a user's data. As legal documents, they function using the principle of notice and choice BIBREF0, where companies post their policies, and theoretically, users read the policies and decide to use a company's products or services only if they find the conditions outlined in its privacy policy acceptable. Many legal jurisdictions around the world accept this framework, including the United States and the European Union BIBREF1, BIBREF2. However, the legitimacy of this framework depends upon users actually reading and understanding privacy policies to determine whether company practices are acceptable to them BIBREF3. In practice this is seldom the case BIBREF4, BIBREF5, BIBREF6, BIBREF7, BIBREF8, BIBREF9, BIBREF10. This is further complicated by the highly individual and nuanced compromises that users are willing to make with their data BIBREF11, discouraging a `one-size-fits-all' approach to notice of data practices in privacy documents. With devices constantly monitoring our environment, including our personal space and our bodies, lack of awareness of how our data is being used easily leads to problematic situations where users are outraged by information misuse, but companies insist that users have consented. The discovery of increasingly egregious uses of data by companies, such as the scandals involving Facebook and Cambridge Analytica BIBREF12, have further brought public attention to the privacy concerns of the internet and ubiquitous computing. This makes privacy a well-motivated application domain for NLP researchers, where advances in enabling users to quickly identify the privacy issues most salient to them can potentially have large real-world impact. [1]https://play.google.com/store/apps/details?id=com.gotokeep.keep.intl [2]https://play.google.com/store/apps/details?id=com.viber.voip [3]A question might not have any supporting evidence for an answer within the privacy policy. Motivated by this need, we contribute PrivacyQA, a corpus consisting of 1750 questions about the contents of privacy policies, paired with over 3500 expert annotations. The goal of this effort is to kickstart the development of question-answering methods for this domain, to address the (unrealistic) expectation that a large population should be reading many policies per day. In doing so, we identify several understudied challenges to our ability to answer these questions, with broad implications for systems seeking to serve users' information-seeking intent. By releasing this resource, we hope to provide an impetus to develop systems capable of language understanding in this increasingly important domain. Related Work Prior work has aimed to make privacy policies easier to understand. Prescriptive approaches towards communicating privacy information BIBREF21, BIBREF22, BIBREF23 have not been widely adopted by industry. Recently, there have been significant research effort devoted to understanding privacy policies by leveraging NLP techniques BIBREF24, BIBREF25, BIBREF26, BIBREF27, BIBREF28, especially by identifying specific data practices within a privacy policy. We adopt a personalized approach to understanding privacy policies, that allows users to query a document and selectively explore content salient to them. Most similar is the PolisisQA corpus BIBREF29, which examines questions users ask corporations on Twitter. Our approach differs in several ways: 1) The PrivacyQA dataset is larger, containing 10x as many questions and answers. 2) Answers are formulated by domain experts with legal training. 3) PrivacyQA includes diverse question types, including unanswerable and subjective questions. Our work is also related to reading comprehension in the open domain, which is frequently based upon Wikipedia passages BIBREF16, BIBREF17, BIBREF15, BIBREF30 and news articles BIBREF20, BIBREF31, BIBREF32. Table.TABREF4 presents the desirable attributes our dataset shares with past approaches. This work is also tied into research in applying NLP approaches to legal documents BIBREF33, BIBREF34, BIBREF35, BIBREF36, BIBREF37, BIBREF38, BIBREF39. While privacy policies have legal implications, their intended audience consists of the general public rather than individuals with legal expertise. This arrangement is problematic because the entities that write privacy policies often have different goals than the audience. feng2015applying, tan-EtAl:2016:P16-1 examine question answering in the insurance domain, another specialized domain similar to privacy, where the intended audience is the general public. Data Collection We describe the data collection methodology used to construct PrivacyQA. With the goal of achieving broad coverage across application types, we collect privacy policies from 35 mobile applications representing a number of different categories in the Google Play Store. One of our goals is to include both policies from well-known applications, which are likely to have carefully-constructed privacy policies, and lesser-known applications with smaller install bases, whose policies might be considerably less sophisticated. Thus, setting 5 million installs as a threshold, we ensure each category includes applications with installs on both sides of this threshold. All policies included in the corpus are in English, and were collected before April 1, 2018, predating many companies' GDPR-focused BIBREF41 updates. We leave it to future studies BIBREF42 to look at the impact of the GDPR (e.g., to what extent GDPR requirements contribute to making it possible to provide users with more informative answers, and to what extent their disclosures continue to omit issues that matter to users). Data Collection ::: Crowdsourced Question Elicitation The intended audience for privacy policies consists of the general public. This informs the decision to elicit questions from crowdworkers on the contents of privacy policies. We choose not to show the contents of privacy policies to crowdworkers, a procedure motivated by a desire to avoid inadvertent biases BIBREF43, BIBREF44, BIBREF45, BIBREF46, BIBREF47, and encourage crowdworkers to ask a variety of questions beyond only asking questions based on practices described in the document. Instead, crowdworkers are presented with public information about a mobile application available on the Google Play Store including its name, description and navigable screenshots. Figure FIGREF9 shows an example of our user interface. Crowdworkers are asked to imagine they have access to a trusted third-party privacy assistant, to whom they can ask any privacy question about a given mobile application. We use the Amazon Mechanical Turk platform and recruit crowdworkers who have been conferred “master” status and are located within the United States of America. Turkers are asked to provide five questions per mobile application, and are paid $2 per assignment, taking ~eight minutes to complete the task. Data Collection ::: Answer Selection To identify legally sound answers, we recruit seven experts with legal training to construct answers to Turker questions. Experts identify relevant evidence within the privacy policy, as well as provide meta-annotation on the question's relevance, subjectivity, OPP-115 category BIBREF49, and how likely any privacy policy is to contain the answer to the question asked. Data Collection ::: Analysis Table.TABREF17 presents aggregate statistics of the PrivacyQA dataset. 1750 questions are posed to our imaginary privacy assistant over 35 mobile applications and their associated privacy documents. As an initial step, we formulate the problem of answering user questions as an extractive sentence selection task, ignoring for now background knowledge, statistical data and legal expertise that could otherwise be brought to bear. The dataset is partitioned into a training set featuring 27 mobile applications and 1350 questions, and a test set consisting of 400 questions over 8 policy documents. This ensures that documents in training and test splits are mutually exclusive. Every question is answered by at least one expert. In addition, in order to estimate annotation reliability and provide for better evaluation, every question in the test set is answered by at least two additional experts. Table TABREF14 describes the distribution over first words of questions posed by crowdworkers. We also observe low redundancy in the questions posed by crowdworkers over each policy, with each policy receiving ~49.94 unique questions despite crowdworkers independently posing questions. Questions are on average 8.4 words long. As declining to answer a question can be a legally sound response but is seldom practically useful, answers to questions where a minority of experts abstain to answer are filtered from the dataset. Privacy policies are ~3000 words long on average. The answers to the question asked by the users typically have ~100 words of evidence in the privacy policy document. Data Collection ::: Analysis ::: Categories of Questions Questions are organized under nine categories from the OPP-115 Corpus annotation scheme BIBREF49: First Party Collection/Use: What, why and how information is collected by the service provider Third Party Sharing/Collection: What, why and how information shared with or collected by third parties Data Security: Protection measures for user information Data Retention: How long user information will be stored User Choice/Control: Control options available to users User Access, Edit and Deletion: If/how users can access, edit or delete information Policy Change: Informing users if policy information has been changed International and Specific Audiences: Practices pertaining to a specific group of users Other: General text, contact information or practices not covered by other categories. For each question, domain experts indicate one or more relevant OPP-115 categories. We mark a category as relevant to a question if it is identified as such by at least two annotators. If no such category exists, the category is marked as `Other' if atleast one annotator has identified the `Other' category to be relevant. If neither of these conditions is satisfied, we label the question as having no agreement. The distribution of questions in the corpus across OPP-115 categories is as shown in Table.TABREF16. First party and third party related questions are the largest categories, forming nearly 66.4% of all questions asked to the privacy assistant. Data Collection ::: Analysis ::: Answer Validation When do experts disagree? We would like to analyze the reasons for potential disagreement on the annotation task, to ensure disagreements arise due to valid differences in opinion rather than lack of adequate specification in annotation guidelines. It is important to note that the annotators are experts rather than crowdworkers. Accordingly, their judgements can be considered valid, legally-informed opinions even when their perspectives differ. For the sake of this question we randomly sample 100 instances in the test data and analyze them for likely reasons for disagreements. We consider a disagreement to have occurred when more than one expert does not agree with the majority consensus. By disagreement we mean there is no overlap between the text identified as relevant by one expert and another. We find that the annotators agree on the answer for 74% of the questions, even if the supporting evidence they identify is not identical i.e full overlap. They disagree on the remaining 26%. Sources of apparent disagreement correspond to situations when different experts: have differing interpretations of question intent (11%) (for example, when a user asks 'who can contact me through the app', the questions admits multiple interpretations, including seeking information about the features of the app, asking about first party collection/use of data or asking about third party collection/use of data), identify different sources of evidence for questions that ask if a practice is performed or not (4%), have differing interpretations of policy content (3%), identify a partial answer to a question in the privacy policy (2%) (for example, when the user asks `who is allowed to use the app' a majority of our annotators decline to answer, but the remaining annotators highlight partial evidence in the privacy policy which states that children under the age of 13 are not allowed to use the app), and other legitimate sources of disagreement (6%) which include personal subjective views of the annotators (for example, when the user asks `is my DNA information used in any way other than what is specified', some experts consider the boilerplate text of the privacy policy which states that it abides to practices described in the policy document as sufficient evidence to answer this question, whereas others do not). Experimental Setup We evaluate the ability of machine learning methods to identify relevant evidence for questions in the privacy domain. We establish baselines for the subtask of deciding on the answerability (§SECREF33) of a question, as well as the overall task of identifying evidence for questions from policies (§SECREF37). We describe aspects of the question that can render it unanswerable within the privacy domain (§SECREF41). Experimental Setup ::: Answerability Identification Baselines We define answerability identification as a binary classification task, evaluating model ability to predict if a question can be answered, given a question in isolation. This can serve as a prior for downstream question-answering. We describe three baselines on the answerability task, and find they considerably improve performance over a majority-class baseline. SVM: We define 3 sets of features to characterize each question. The first is a simple bag-of-words set of features over the question (SVM-BOW), the second is bag-of-words features of the question as well as length of the question in words (SVM-BOW + LEN), and lastly we extract bag-of-words features, length of the question in words as well as part-of-speech tags for the question (SVM-BOW + LEN + POS). This results in vectors of 200, 201 and 228 dimensions respectively, which are provided to an SVM with a linear kernel. CNN: We utilize a CNN neural encoder for answerability prediction. We use GloVe word embeddings BIBREF50, and a filter size of 5 with 64 filters to encode questions. BERT: BERT BIBREF51 is a bidirectional transformer-based language-model BIBREF52. We fine-tune BERT-base on our binary answerability identification task with a learning rate of 2e-5 for 3 epochs, with a maximum sequence length of 128. Experimental Setup ::: Privacy Question Answering Our goal is to identify evidence within a privacy policy for questions asked by a user. This is framed as an answer sentence selection task, where models identify a set of evidence sentences from all candidate sentences in each policy. Experimental Setup ::: Privacy Question Answering ::: Evaluation Metric Our evaluation metric for answer-sentence selection is sentence-level F1, implemented similar to BIBREF30, BIBREF16. Precision and recall are implemented by measuring the overlap between predicted sentences and sets of gold-reference sentences. We report the average of the maximum F1 from each n$-$1 subset, in relation to the heldout reference. Experimental Setup ::: Privacy Question Answering ::: Baselines We describe baselines on this task, including a human performance baseline. No-Answer Baseline (NA) : Most of the questions we receive are difficult to answer in a legally-sound way on the basis of information present in the privacy policy. We establish a simple baseline to quantify the effect of identifying every question as unanswerable. Word Count Baseline : To quantify the effect of using simple lexical matching to answer the questions, we retrieve the top candidate policy sentences for each question using a word count baseline BIBREF53, which counts the number of question words that also appear in a sentence. We include the top 2, 3 and 5 candidates as baselines. BERT: We implement two BERT-based baselines BIBREF51 for evidence identification. First, we train BERT on each query-policy sentence pair as a binary classification task to identify if the sentence is evidence for the question or not (Bert). We also experiment with a two-stage classifier, where we separately train the model on questions only to predict answerability. At inference time, if the answerable classifier predicts the question is answerable, the evidence identification classifier produces a set of candidate sentences (Bert + Unanswerable). Human Performance: We pick each reference answer provided by an annotator, and compute the F1 with respect to the remaining references, as described in section 4.2.1. Each reference answer is treated as the prediction, and the remaining n-1 answers are treated as the gold reference. The average of the maximum F1 across all reference answers is computed as the human baseline. Results and Discussion The results of the answerability baselines are presented in Table TABREF31, and on answer sentence selection in Table TABREF32. We observe that bert exhibits the best performance on a binary answerability identification task. However, most baselines considerably exceed the performance of a majority-class baseline. This suggests considerable information in the question, indicating it's possible answerability within this domain. Table.TABREF32 describes the performance of our baselines on the answer sentence selection task. The No-answer (NA) baseline performs at 28 F1, providing a lower bound on performance at this task. We observe that our best-performing baseline, Bert + Unanswerable achieves an F1 of 39.8. This suggest that bert is capable of making some progress towards answering questions in this difficult domain, while still leaving considerable headroom for improvement to reach human performance. Bert + Unanswerable performance suggests that incorporating information about answerability can help in this difficult domain. We examine this challenging phenomena of unanswerability further in Section . Results and Discussion ::: Error Analysis Disagreements are analyzed based on the OPP-115 categories of each question (Table.TABREF34). We compare our best performing BERT variant against the NA model and human performance. We observe significant room for improvement across all categories of questions but especially for first party, third party and data retention categories. We analyze the performance of our strongest BERT variant, to identify classes of errors and directions for future improvement (Table.8). We observe that a majority of answerability mistakes made by the BERT model are questions which are in fact answerable, but are identified as unanswerable by BERT. We observe that BERT makes 124 such mistakes on the test set. We collect expert judgments on relevance, subjectivity , silence and information about how likely the question is to be answered from the privacy policy from our experts. We find that most of these mistakes are relevant questions. However many of them were identified as subjective by the annotators, and at least one annotator marked 19 of these questions as having no answer within the privacy policy. However, only 6 of these questions were unexpected or do not usually have an answer in privacy policies. These findings suggest that a more nuanced understanding of answerability might help improve model performance in his challenging domain. Results and Discussion ::: What makes Questions Unanswerable? We further ask legal experts to identify potential causes of unanswerability of questions. This analysis has considerable implications. While past work BIBREF17 has treated unanswerable questions as homogeneous, a question answering system might wish to have different treatments for different categories of `unanswerable' questions. The following factors were identified to play a role in unanswerability: Incomprehensibility: If a question is incomprehensible to the extent that its meaning is not intelligible. Relevance: Is this question in the scope of what could be answered by reading the privacy policy. Ill-formedness: Is this question ambiguous or vague. An ambiguous statement will typically contain expressions that can refer to multiple potential explanations, whereas a vague statement carries a concept with an unclear or soft definition. Silence: Other policies answer this type of question but this one does not. Atypicality: The question is of a nature such that it is unlikely for any policy policy to have an answer to the question. Our experts attempt to identify the different `unanswerable' factors for all 573 such questions in the corpus. 4.18% of the questions were identified as being incomprehensible (for example, `any difficulties to occupy the privacy assistant'). Amongst the comprehendable questions, 50% were identified as likely to have an answer within the privacy policy, 33.1% were identified as being privacy-related questions but not within the scope of a privacy policy (e.g., 'has Viber had any privacy breaches in the past?') and 16.9% of questions were identified as completely out-of-scope (e.g., `'will the app consume much space?'). In the questions identified as relevant, 32% were ill-formed questions that were phrased by the user in a manner considered vague or ambiguous. Of the questions that were both relevant as well as `well-formed', 95.7% of the questions were not answered by the policy in question but it was reasonable to expect that a privacy policy would contain an answer. The remaining 4.3% were described as reasonable questions, but of a nature generally not discussed in privacy policies. This suggests that the answerability of questions over privacy policies is a complex issue, and future systems should consider each of these factors when serving user's information seeking intent. We examine a large-scale dataset of “natural” unanswerable questions BIBREF54 based on real user search engine queries to identify if similar unanswerability factors exist. It is important to note that these questions have previously been filtered, according to a criteria for bad questions defined as “(questions that are) ambiguous, incomprehensible, dependent on clear false presuppositions, opinion-seeking, or not clearly a request for factual information.” Annotators made the decision based on the content of the question without viewing the equivalent Wikipedia page. We randomly sample 100 questions from the development set which were identified as unanswerable, and find that 20% of the questions are not questions (e.g., “all I want for christmas is you mariah carey tour”). 12% of questions are unlikely to ever contain an answer on Wikipedia, corresponding closely to our atypicality category. 3% of questions are unlikely to have an answer anywhere (e.g., `what guides Santa home after he has delivered presents?'). 7% of questions are incomplete or open-ended (e.g., `the south west wind blows across nigeria between'). 3% of questions have an unresolvable coreference (e.g., `how do i get to Warsaw Missouri from here'). 4% of questions are vague, and a further 7% have unknown sources of error. 2% still contain false presuppositions (e.g., `what is the only fruit that does not have seeds?') and the remaining 42% do not have an answer within the document. This reinforces our belief that though they have been understudied in past work, any question answering system interacting with real users should expect to receive such unanticipated and unanswerable questions. Conclusion We present PrivacyQA, the first significant corpus of privacy policy questions and more than 3500 expert annotations of relevant answers. The goal of this work is to promote question-answering research in the specialized privacy domain, where it can have large real-world impact. Strong neural baselines on PrivacyQA achieve a performance of only 39.8 F1 on this corpus, indicating considerable room for future research. Further, we shed light on several important considerations that affect the answerability of questions. We hope this contribution leads to multidisciplinary efforts to precisely understand user intent and reconcile it with information in policy documents, from both the privacy and NLP communities. Acknowledgements This research was supported in part by grants from the National Science Foundation Secure and Trustworthy Computing program (CNS-1330596, CNS-1330214, CNS-15-13957, CNS-1801316, CNS-1914486, CNS-1914444) and a DARPA Brandeis grant on Personalized Privacy Assistants (FA8750-15-2-0277). The US Government is authorized to reproduce and distribute reprints for Governmental purposes not withstanding any copyright notation. The views and conclusions contained herein are those of the authors and should not be interpreted as necessarily representing the official policies or endorsements, either expressed or implied, of the NSF, DARPA, or the US Government. The authors would like to extend their gratitude to Elias Wright, Gian Mascioli, Kiara Pillay, Harrison Kay, Eliel Talo, Alexander Fagella and N. Cameron Russell for providing their valuable expertise and insight to this effort. The authors are also grateful to Eduard Hovy, Lorrie Cranor, Florian Schaub, Joel Reidenberg, Aditya Potukuchi and Igor Shalyminov for helpful discussions related to this work, and to the three anonymous reviewers of this draft for their constructive feedback. Finally, the authors would like to thank all crowdworkers who consented to participate in this study.
Are the experts comparable to real-world users?
3c3807f226ba72fc41f59f0338f12a49a0c35605
Are Ġthe Ġexperts Ġcomparable Ġto Ġreal - world Ġusers ?
No
<s> Are Ġthe Ġexperts Ġcomparable Ġto Ġreal - world Ġusers ? </s> Introduction </s> Privacy Ġpolicies Ġare Ġthe Ġdocuments Ġwhich Ġdisclose Ġthe Ġways Ġin Ġwhich Ġa Ġcompany Ġgathers , Ġuses , Ġshares Ġand Ġmanages Ġa Ġuser 's Ġdata . ĠAs Ġlegal Ġdocuments , Ġthey Ġfunction Ġusing Ġthe Ġprinciple Ġof Ġnotice Ġand Ġchoice ĠB IB REF 0 , Ġwhere Ġcompanies Ġpost Ġtheir Ġpolicies , Ġand Ġtheoretically , Ġusers Ġread Ġthe Ġpolicies Ġand Ġdecide Ġto Ġuse Ġa Ġcompany 's Ġproducts Ġor Ġservices Ġonly Ġif Ġthey Ġfind Ġthe Ġconditions Ġoutlined Ġin Ġits Ġprivacy Ġpolicy Ġacceptable . ĠMany Ġlegal Ġjurisdictions Ġaround Ġthe Ġworld Ġaccept Ġthis Ġframework , Ġincluding Ġthe ĠUnited ĠStates Ġand Ġthe ĠEuropean ĠUnion ĠB IB REF 1 , ĠB IB REF 2 . ĠHowever , Ġthe Ġlegitimacy Ġof Ġthis Ġframework Ġdepends Ġupon Ġusers Ġactually Ġreading Ġand Ġunderstanding Ġprivacy Ġpolicies Ġto Ġdetermine Ġwhether Ġcompany Ġpractices Ġare Ġacceptable Ġto Ġthem ĠB IB REF 3 . ĠIn Ġpractice Ġthis Ġis Ġseldom Ġthe Ġcase ĠB IB REF 4 , ĠB IB REF 5 , ĠB IB REF 6 , ĠB IB REF 7 , ĠB IB REF 8 , ĠB IB REF 9 , ĠB IB REF 10 . ĠThis Ġis Ġfurther Ġcomplicated Ġby Ġthe Ġhighly Ġindividual Ġand Ġnuanced Ġcompromises Ġthat Ġusers Ġare Ġwilling Ġto Ġmake Ġwith Ġtheir Ġdata ĠB IB REF 11 , Ġdiscouraging Ġa Ġ` one - size - fits - all ' Ġapproach Ġto Ġnotice Ġof Ġdata Ġpractices Ġin Ġprivacy Ġdocuments . </s> With Ġdevices Ġconstantly Ġmonitoring Ġour Ġenvironment , Ġincluding Ġour Ġpersonal Ġspace Ġand Ġour Ġbodies , Ġlack Ġof Ġawareness Ġof Ġhow Ġour Ġdata Ġis Ġbeing Ġused Ġeasily Ġleads Ġto Ġproblematic Ġsituations Ġwhere Ġusers Ġare Ġoutraged Ġby Ġinformation Ġmisuse , Ġbut Ġcompanies Ġinsist Ġthat Ġusers Ġhave Ġcons ented . ĠThe Ġdiscovery Ġof Ġincreasingly Ġegregious Ġuses Ġof Ġdata Ġby Ġcompanies , Ġsuch Ġas Ġthe Ġscandals Ġinvolving ĠFacebook Ġand ĠCambridge ĠAnaly t ica ĠB IB REF 12 , Ġhave Ġfurther Ġbrought Ġpublic Ġattention Ġto Ġthe Ġprivacy Ġconcerns Ġof Ġthe Ġinternet Ġand Ġubiquitous Ġcomputing . ĠThis Ġmakes Ġprivacy Ġa Ġwell - mot ivated Ġapplication Ġdomain Ġfor ĠN LP Ġresearchers , Ġwhere Ġadvances Ġin Ġenabling Ġusers Ġto Ġquickly Ġidentify Ġthe Ġprivacy Ġissues Ġmost Ġsalient Ġto Ġthem Ġcan Ġpotentially Ġhave Ġlarge Ġreal - world Ġimpact . </s> [ 1 ] https :// play . google . com / store / apps / details ? id = com . g oto keep . keep . int l Ġ[ 2 ] https :// play . google . com / store / apps / details ? id = com . v iber . vo ip Ġ[ 3 ] A Ġquestion Ġmight Ġnot Ġhave Ġany Ġsupporting Ġevidence Ġfor Ġan Ġanswer Ġwithin Ġthe Ġprivacy Ġpolicy . </s> Mot ivated Ġby Ġthis Ġneed , Ġwe Ġcontribute ĠPrivacy Q A , Ġa Ġcorpus Ġconsisting Ġof Ġ17 50 Ġquestions Ġabout Ġthe Ġcontents Ġof Ġprivacy Ġpolicies , Ġpaired Ġwith Ġover Ġ35 00 Ġexpert Ġannotations . ĠThe Ġgoal Ġof Ġthis Ġeffort Ġis Ġto Ġkick start Ġthe Ġdevelopment Ġof Ġquestion - ans w ering Ġmethods Ġfor Ġthis Ġdomain , Ġto Ġaddress Ġthe Ġ( un real istic ) Ġexpectation Ġthat Ġa Ġlarge Ġpopulation Ġshould Ġbe Ġreading Ġmany Ġpolicies Ġper Ġday . ĠIn Ġdoing Ġso , Ġwe Ġidentify Ġseveral Ġunder stud ied Ġchallenges Ġto Ġour Ġability Ġto Ġanswer Ġthese Ġquestions , Ġwith Ġbroad Ġimplications Ġfor Ġsystems Ġseeking Ġto Ġserve Ġusers ' Ġinformation - seeking Ġintent . ĠBy Ġreleasing Ġthis Ġresource , Ġwe Ġhope Ġto Ġprovide Ġan Ġimpetus Ġto Ġdevelop Ġsystems Ġcapable Ġof Ġlanguage Ġunderstanding Ġin Ġthis Ġincreasingly Ġimportant Ġdomain . </s> Related ĠWork </s> Prior Ġwork Ġhas Ġaimed Ġto Ġmake Ġprivacy Ġpolicies Ġeasier Ġto Ġunderstand . ĠPres cript ive Ġapproaches Ġtowards Ġcommunicating Ġprivacy Ġinformation ĠB IB REF 21 , ĠB IB REF 22 , ĠB IB REF 23 Ġhave Ġnot Ġbeen Ġwidely Ġadopted Ġby Ġindustry . ĠRecently , Ġthere Ġhave Ġbeen Ġsignificant Ġresearch Ġeffort Ġdevoted Ġto Ġunderstanding Ġprivacy Ġpolicies Ġby Ġleveraging ĠN LP Ġtechniques ĠB IB REF 24 , ĠB IB REF 25 , ĠB IB REF 26 , ĠB IB REF 27 , ĠB IB REF 28 , Ġespecially Ġby Ġidentifying Ġspecific Ġdata Ġpractices Ġwithin Ġa Ġprivacy Ġpolicy . ĠWe Ġadopt Ġa Ġpersonalized Ġapproach Ġto Ġunderstanding Ġprivacy Ġpolicies , Ġthat Ġallows Ġusers Ġto Ġquery Ġa Ġdocument Ġand Ġselectively Ġexplore Ġcontent Ġsalient Ġto Ġthem . ĠMost Ġsimilar Ġis Ġthe ĠPol is is Q A Ġcorpus ĠB IB REF 29 , Ġwhich Ġexamines Ġquestions Ġusers Ġask Ġcorporations Ġon ĠTwitter . ĠOur Ġapproach Ġdiffers Ġin Ġseveral Ġways : Ġ1 ) ĠThe ĠPrivacy Q A Ġdataset Ġis Ġlarger , Ġcontaining Ġ10 x Ġas Ġmany Ġquestions Ġand Ġanswers . Ġ2 ) ĠAnswers Ġare Ġformulated Ġby Ġdomain Ġexperts Ġwith Ġlegal Ġtraining . Ġ3 ) ĠPrivacy Q A Ġincludes Ġdiverse Ġquestion Ġtypes , Ġincluding Ġun answer able Ġand Ġsubjective Ġquestions . </s> Our Ġwork Ġis Ġalso Ġrelated Ġto Ġreading Ġcomprehension Ġin Ġthe Ġopen Ġdomain , Ġwhich Ġis Ġfrequently Ġbased Ġupon ĠWikipedia Ġpassages ĠB IB REF 16 , ĠB IB REF 17 , ĠB IB REF 15 , ĠB IB REF 30 Ġand Ġnews Ġarticles ĠB IB REF 20 , ĠB IB REF 31 , ĠB IB REF 32 . ĠTable . TA B REF 4 Ġpresents Ġthe Ġdesirable Ġattributes Ġour Ġdataset Ġshares Ġwith Ġpast Ġapproaches . ĠThis Ġwork Ġis Ġalso Ġtied Ġinto Ġresearch Ġin Ġapplying ĠN LP Ġapproaches Ġto Ġlegal Ġdocuments ĠB IB REF 33 , ĠB IB REF 34 , ĠB IB REF 35 , ĠB IB REF 36 , ĠB IB REF 37 , ĠB IB REF 38 , ĠB IB REF 39 . ĠWhile Ġprivacy Ġpolicies Ġhave Ġlegal Ġimplications , Ġtheir Ġintended Ġaudience Ġconsists Ġof Ġthe Ġgeneral Ġpublic Ġrather Ġthan Ġindividuals Ġwith Ġlegal Ġexpertise . ĠThis Ġarrangement Ġis Ġproblematic Ġbecause Ġthe Ġentities Ġthat Ġwrite Ġprivacy Ġpolicies Ġoften Ġhave Ġdifferent Ġgoals Ġthan Ġthe Ġaudience . Ġf eng 2015 app lying , Ġtan - E t Al : 2016 : P 16 - 1 Ġexamine Ġquestion Ġanswering Ġin Ġthe Ġinsurance Ġdomain , Ġanother Ġspecialized Ġdomain Ġsimilar Ġto Ġprivacy , Ġwhere Ġthe Ġintended Ġaudience Ġis Ġthe Ġgeneral Ġpublic . </s> Data ĠCollection </s> We Ġdescribe Ġthe Ġdata Ġcollection Ġmethodology Ġused Ġto Ġconstruct ĠPrivacy Q A . ĠWith Ġthe Ġgoal Ġof Ġachieving Ġbroad Ġcoverage Ġacross Ġapplication Ġtypes , Ġwe Ġcollect Ġprivacy Ġpolicies Ġfrom Ġ35 Ġmobile Ġapplications Ġrepresenting Ġa Ġnumber Ġof Ġdifferent Ġcategories Ġin Ġthe ĠGoogle ĠPlay ĠStore . ĠOne Ġof Ġour Ġgoals Ġis Ġto Ġinclude Ġboth Ġpolicies Ġfrom Ġwell - known Ġapplications , Ġwhich Ġare Ġlikely Ġto Ġhave Ġcarefully - con structed Ġprivacy Ġpolicies , Ġand Ġlesser - known Ġapplications Ġwith Ġsmaller Ġinstall Ġbases , Ġwhose Ġpolicies Ġmight Ġbe Ġconsiderably Ġless Ġsophisticated . ĠThus , Ġsetting Ġ5 Ġmillion Ġinstalls Ġas Ġa Ġthreshold , Ġwe Ġensure Ġeach Ġcategory Ġincludes Ġapplications Ġwith Ġinstalls Ġon Ġboth Ġsides Ġof Ġthis Ġthreshold . ĠAll Ġpolicies Ġincluded Ġin Ġthe Ġcorpus Ġare Ġin ĠEnglish , Ġand Ġwere Ġcollected Ġbefore ĠApril Ġ1 , Ġ2018 , Ġpred ating Ġmany Ġcompanies ' ĠGD PR - focused ĠB IB REF 41 Ġupdates . ĠWe Ġleave Ġit Ġto Ġfuture Ġstudies ĠB IB REF 42 Ġto Ġlook Ġat Ġthe Ġimpact Ġof Ġthe ĠGD PR Ġ( e . g ., Ġto Ġwhat Ġextent ĠGD PR Ġrequirements Ġcontribute Ġto Ġmaking Ġit Ġpossible Ġto Ġprovide Ġusers Ġwith Ġmore Ġinformative Ġanswers , Ġand Ġto Ġwhat Ġextent Ġtheir Ġdisclosures Ġcontinue Ġto Ġomit Ġissues Ġthat Ġmatter Ġto Ġusers ). </s> Data ĠCollection Ġ: :: ĠCrowd s ourced ĠQuestion ĠE lic itation </s> The Ġintended Ġaudience Ġfor Ġprivacy Ġpolicies Ġconsists Ġof Ġthe Ġgeneral Ġpublic . ĠThis Ġinforms Ġthe Ġdecision Ġto Ġelicit Ġquestions Ġfrom Ġcrowd workers Ġon Ġthe Ġcontents Ġof Ġprivacy Ġpolicies . ĠWe Ġchoose Ġnot Ġto Ġshow Ġthe Ġcontents Ġof Ġprivacy Ġpolicies Ġto Ġcrowd workers , Ġa Ġprocedure Ġmotivated Ġby Ġa Ġdesire Ġto Ġavoid Ġinadvert ent Ġbiases ĠB IB REF 43 , ĠB IB REF 44 , ĠB IB REF 45 , ĠB IB REF 46 , ĠB IB REF 47 , Ġand Ġencourage Ġcrowd workers Ġto Ġask Ġa Ġvariety Ġof Ġquestions Ġbeyond Ġonly Ġasking Ġquestions Ġbased Ġon Ġpractices Ġdescribed Ġin Ġthe Ġdocument . </s> Instead , Ġcrowd workers Ġare Ġpresented Ġwith Ġpublic Ġinformation Ġabout Ġa Ġmobile Ġapplication Ġavailable Ġon Ġthe ĠGoogle ĠPlay ĠStore Ġincluding Ġits Ġname , Ġdescription Ġand Ġnavig able Ġscreenshots . ĠFigure ĠFIG REF 9 Ġshows Ġan Ġexample Ġof Ġour Ġuser Ġinterface . ĠCrowd workers Ġare Ġasked Ġto Ġimagine Ġthey Ġhave Ġaccess Ġto Ġa Ġtrusted Ġthird - party Ġprivacy Ġassistant , Ġto Ġwhom Ġthey Ġcan Ġask Ġany Ġprivacy Ġquestion Ġabout Ġa Ġgiven Ġmobile Ġapplication . ĠWe Ġuse Ġthe ĠAmazon ĠMechanical ĠTurk Ġplatform Ġand Ġrecruit Ġcrowd workers Ġwho Ġhave Ġbeen Ġconferred ĠâĢ ľ master âĢ Ŀ Ġstatus Ġand Ġare Ġlocated Ġwithin Ġthe ĠUnited ĠStates Ġof ĠAmerica . ĠTurk ers Ġare Ġasked Ġto Ġprovide Ġfive Ġquestions Ġper Ġmobile Ġapplication , Ġand Ġare Ġpaid Ġ$ 2 Ġper Ġassignment , Ġtaking Ġ~ eight Ġminutes Ġto Ġcomplete Ġthe Ġtask . </s> Data ĠCollection Ġ: :: ĠAnswer ĠSelection </s> To Ġidentify Ġlegally Ġsound Ġanswers , Ġwe Ġrecruit Ġseven Ġexperts Ġwith Ġlegal Ġtraining Ġto Ġconstruct Ġanswers Ġto ĠTur ker Ġquestions . ĠExperts Ġidentify Ġrelevant Ġevidence Ġwithin Ġthe Ġprivacy Ġpolicy , Ġas Ġwell Ġas Ġprovide Ġmeta - ann otation Ġon Ġthe Ġquestion 's Ġrelevance , Ġsubject ivity , ĠO PP - 115 Ġcategory ĠB IB REF 49 , Ġand Ġhow Ġlikely Ġany Ġprivacy Ġpolicy Ġis Ġto Ġcontain Ġthe Ġanswer Ġto Ġthe Ġquestion Ġasked . </s> Data ĠCollection Ġ: :: ĠAnalysis </s> Table . TA B REF 17 Ġpresents Ġaggregate Ġstatistics Ġof Ġthe ĠPrivacy Q A Ġdataset . Ġ17 50 Ġquestions Ġare Ġposed Ġto Ġour Ġimaginary Ġprivacy Ġassistant Ġover Ġ35 Ġmobile Ġapplications Ġand Ġtheir Ġassociated Ġprivacy Ġdocuments . ĠAs Ġan Ġinitial Ġstep , Ġwe Ġformulate Ġthe Ġproblem Ġof Ġanswering Ġuser Ġquestions Ġas Ġan Ġextract ive Ġsentence Ġselection Ġtask , Ġignoring Ġfor Ġnow Ġbackground Ġknowledge , Ġstatistical Ġdata Ġand Ġlegal Ġexpertise Ġthat Ġcould Ġotherwise Ġbe Ġbrought Ġto Ġbear . ĠThe Ġdataset Ġis Ġpartition ed Ġinto Ġa Ġtraining Ġset Ġfeaturing Ġ27 Ġmobile Ġapplications Ġand Ġ13 50 Ġquestions , Ġand Ġa Ġtest Ġset Ġconsisting Ġof Ġ400 Ġquestions Ġover Ġ8 Ġpolicy Ġdocuments . ĠThis Ġensures Ġthat Ġdocuments Ġin Ġtraining Ġand Ġtest Ġsplits Ġare Ġmutually Ġexclusive . ĠEvery Ġquestion Ġis Ġanswered Ġby Ġat Ġleast Ġone Ġexpert . ĠIn Ġaddition , Ġin Ġorder Ġto Ġestimate Ġannotation Ġreliability Ġand Ġprovide Ġfor Ġbetter Ġevaluation , Ġevery Ġquestion Ġin Ġthe Ġtest Ġset Ġis Ġanswered Ġby Ġat Ġleast Ġtwo Ġadditional Ġexperts . </s> Table ĠT AB REF 14 Ġdescribes Ġthe Ġdistribution Ġover Ġfirst Ġwords Ġof Ġquestions Ġposed Ġby Ġcrowd workers . ĠWe Ġalso Ġobserve Ġlow Ġredundancy Ġin Ġthe Ġquestions Ġposed Ġby Ġcrowd workers Ġover Ġeach Ġpolicy , Ġwith Ġeach Ġpolicy Ġreceiving Ġ~ 49 . 94 Ġunique Ġquestions Ġdespite Ġcrowd workers Ġindependently Ġposing Ġquestions . ĠQuestions Ġare Ġon Ġaverage Ġ8 . 4 Ġwords Ġlong . ĠAs Ġdeclining Ġto Ġanswer Ġa Ġquestion Ġcan Ġbe Ġa Ġlegally Ġsound Ġresponse Ġbut Ġis Ġseldom Ġpractically Ġuseful , Ġanswers Ġto Ġquestions Ġwhere Ġa Ġminority Ġof Ġexperts Ġabst ain Ġto Ġanswer Ġare Ġfiltered Ġfrom Ġthe Ġdataset . ĠPrivacy Ġpolicies Ġare Ġ~ 3000 Ġwords Ġlong Ġon Ġaverage . ĠThe Ġanswers Ġto Ġthe Ġquestion Ġasked Ġby Ġthe Ġusers Ġtypically Ġhave Ġ~ 100 Ġwords Ġof Ġevidence Ġin Ġthe Ġprivacy Ġpolicy Ġdocument . </s> Data ĠCollection Ġ: :: ĠAnalysis Ġ: :: ĠCategories Ġof ĠQuestions </s> Questions Ġare Ġorganized Ġunder Ġnine Ġcategories Ġfrom Ġthe ĠO PP - 115 ĠCorpus Ġannotation Ġscheme ĠB IB REF 49 : </s> First ĠParty ĠCollection / Use : ĠWhat , Ġwhy Ġand Ġhow Ġinformation Ġis Ġcollected Ġby Ġthe Ġservice Ġprovider </s> Third ĠParty ĠSharing / Collection : ĠWhat , Ġwhy Ġand Ġhow Ġinformation Ġshared Ġwith Ġor Ġcollected Ġby Ġthird Ġparties </s> Data ĠSecurity : ĠProtection Ġmeasures Ġfor Ġuser Ġinformation </s> Data ĠRet ention : ĠHow Ġlong Ġuser Ġinformation Ġwill Ġbe Ġstored </s> User ĠChoice / Control : ĠControl Ġoptions Ġavailable Ġto Ġusers </s> User ĠAccess , ĠEdit Ġand ĠDe let ion : ĠIf / how Ġusers Ġcan Ġaccess , Ġedit Ġor Ġdelete Ġinformation </s> Policy ĠChange : ĠIn forming Ġusers Ġif Ġpolicy Ġinformation Ġhas Ġbeen Ġchanged </s> International Ġand ĠSpecific ĠAud iences : ĠPractices Ġpertaining Ġto Ġa Ġspecific Ġgroup Ġof Ġusers </s> Other : ĠGeneral Ġtext , Ġcontact Ġinformation Ġor Ġpractices Ġnot Ġcovered Ġby Ġother Ġcategories . </s> For Ġeach Ġquestion , Ġdomain Ġexperts Ġindicate Ġone Ġor Ġmore Ġrelevant ĠO PP - 115 Ġcategories . ĠWe Ġmark Ġa Ġcategory Ġas Ġrelevant Ġto Ġa Ġquestion Ġif Ġit Ġis Ġidentified Ġas Ġsuch Ġby Ġat Ġleast Ġtwo Ġannot ators . ĠIf Ġno Ġsuch Ġcategory Ġexists , Ġthe Ġcategory Ġis Ġmarked Ġas Ġ` Other ' Ġif Ġat le ast Ġone Ġannot ator Ġhas Ġidentified Ġthe Ġ` Other ' Ġcategory Ġto Ġbe Ġrelevant . ĠIf Ġneither Ġof Ġthese Ġconditions Ġis Ġsatisfied , Ġwe Ġlabel Ġthe Ġquestion Ġas Ġhaving Ġno Ġagreement . ĠThe Ġdistribution Ġof Ġquestions Ġin Ġthe Ġcorpus Ġacross ĠO PP - 115 Ġcategories Ġis Ġas Ġshown Ġin ĠTable . TA B REF 16 . ĠFirst Ġparty Ġand Ġthird Ġparty Ġrelated Ġquestions Ġare Ġthe Ġlargest Ġcategories , Ġforming Ġnearly Ġ66 . 4 % Ġof Ġall Ġquestions Ġasked Ġto Ġthe Ġprivacy Ġassistant . </s> Data ĠCollection Ġ: :: ĠAnalysis Ġ: :: ĠAnswer ĠVal idation </s> When Ġdo Ġexperts Ġdisagree ? ĠWe Ġwould Ġlike Ġto Ġanalyze Ġthe Ġreasons Ġfor Ġpotential Ġdisagreement Ġon Ġthe Ġannotation Ġtask , Ġto Ġensure Ġdisagreements Ġarise Ġdue Ġto Ġvalid Ġdifferences Ġin Ġopinion Ġrather Ġthan Ġlack Ġof Ġadequate Ġspecification Ġin Ġannotation Ġguidelines . ĠIt Ġis Ġimportant Ġto Ġnote Ġthat Ġthe Ġannot ators Ġare Ġexperts Ġrather Ġthan Ġcrowd workers . ĠAccordingly , Ġtheir Ġjud gements Ġcan Ġbe Ġconsidered Ġvalid , Ġlegally - informed Ġopinions Ġeven Ġwhen Ġtheir Ġperspectives Ġdiffer . ĠFor Ġthe Ġsake Ġof Ġthis Ġquestion Ġwe Ġrandomly Ġsample Ġ100 Ġinstances Ġin Ġthe Ġtest Ġdata Ġand Ġanalyze Ġthem Ġfor Ġlikely Ġreasons Ġfor Ġdisagreements . ĠWe Ġconsider Ġa Ġdisagreement Ġto Ġhave Ġoccurred Ġwhen Ġmore Ġthan Ġone Ġexpert Ġdoes Ġnot Ġagree Ġwith Ġthe Ġmajority Ġconsensus . ĠBy Ġdisagreement Ġwe Ġmean Ġthere Ġis Ġno Ġoverlap Ġbetween Ġthe Ġtext Ġidentified Ġas Ġrelevant Ġby Ġone Ġexpert Ġand Ġanother . </s> We Ġfind Ġthat Ġthe Ġannot ators Ġagree Ġon Ġthe Ġanswer Ġfor Ġ74 % Ġof Ġthe Ġquestions , Ġeven Ġif Ġthe Ġsupporting Ġevidence Ġthey Ġidentify Ġis Ġnot Ġidentical Ġi . e Ġfull Ġoverlap . ĠThey Ġdisagree Ġon Ġthe Ġremaining Ġ26 %. ĠSources Ġof Ġapparent Ġdisagreement Ġcorrespond Ġto Ġsituations Ġwhen Ġdifferent Ġexperts : Ġhave Ġdiffering Ġinterpretations Ġof Ġquestion Ġintent Ġ( 11 %) Ġ( for Ġexample , Ġwhen Ġa Ġuser Ġasks Ġ' who Ġcan Ġcontact Ġme Ġthrough Ġthe Ġapp ', Ġthe Ġquestions Ġadmits Ġmultiple Ġinterpretations , Ġincluding Ġseeking Ġinformation Ġabout Ġthe Ġfeatures Ġof Ġthe Ġapp , Ġasking Ġabout Ġfirst Ġparty Ġcollection / use Ġof Ġdata Ġor Ġasking Ġabout Ġthird Ġparty Ġcollection / use Ġof Ġdata ), Ġidentify Ġdifferent Ġsources Ġof Ġevidence Ġfor Ġquestions Ġthat Ġask Ġif Ġa Ġpractice Ġis Ġperformed Ġor Ġnot Ġ( 4 %), Ġhave Ġdiffering Ġinterpretations Ġof Ġpolicy Ġcontent Ġ( 3 %), Ġidentify Ġa Ġpartial Ġanswer Ġto Ġa Ġquestion Ġin Ġthe Ġprivacy Ġpolicy Ġ( 2 %) Ġ( for Ġexample , Ġwhen Ġthe Ġuser Ġasks Ġ` who Ġis Ġallowed Ġto Ġuse Ġthe Ġapp ' Ġa Ġmajority Ġof Ġour Ġannot ators Ġdecline Ġto Ġanswer , Ġbut Ġthe Ġremaining Ġannot ators Ġhighlight Ġpartial Ġevidence Ġin Ġthe Ġprivacy Ġpolicy Ġwhich Ġstates Ġthat Ġchildren Ġunder Ġthe Ġage Ġof Ġ13 Ġare Ġnot Ġallowed Ġto Ġuse Ġthe Ġapp ), Ġand Ġother Ġlegitimate Ġsources Ġof Ġdisagreement Ġ( 6 %) Ġwhich Ġinclude Ġpersonal Ġsubjective Ġviews Ġof Ġthe Ġannot ators Ġ( for Ġexample , Ġwhen Ġthe Ġuser Ġasks Ġ` is Ġmy ĠDNA Ġinformation Ġused Ġin Ġany Ġway Ġother Ġthan Ġwhat Ġis Ġspecified ', Ġsome Ġexperts Ġconsider Ġthe Ġboiler plate Ġtext Ġof Ġthe Ġprivacy Ġpolicy Ġwhich Ġstates Ġthat Ġit Ġab ides Ġto Ġpractices Ġdescribed Ġin Ġthe Ġpolicy Ġdocument Ġas Ġsufficient Ġevidence Ġto Ġanswer Ġthis Ġquestion , Ġwhereas Ġothers Ġdo Ġnot ). </s> Exper imental ĠSetup </s> We Ġevaluate Ġthe Ġability Ġof Ġmachine Ġlearning Ġmethods Ġto Ġidentify Ġrelevant Ġevidence Ġfor Ġquestions Ġin Ġthe Ġprivacy Ġdomain . ĠWe Ġestablish Ġbas elines Ġfor Ġthe Ġsubt ask Ġof Ġdeciding Ġon Ġthe Ġanswer ability Ġ( Â§ SEC REF 33 ) Ġof Ġa Ġquestion , Ġas Ġwell Ġas Ġthe Ġoverall Ġtask Ġof Ġidentifying Ġevidence Ġfor Ġquestions Ġfrom Ġpolicies Ġ( Â§ SEC REF 37 ). ĠWe Ġdescribe Ġaspects Ġof Ġthe Ġquestion Ġthat Ġcan Ġrender Ġit Ġun answer able Ġwithin Ġthe Ġprivacy Ġdomain Ġ( Â§ SEC REF 41 ). </s> Exper imental ĠSetup Ġ: :: ĠAnswer ability ĠIdentification ĠBas elines </s> We Ġdefine Ġanswer ability Ġidentification Ġas Ġa Ġbinary Ġclassification Ġtask , Ġevaluating Ġmodel Ġability Ġto Ġpredict Ġif Ġa Ġquestion Ġcan Ġbe Ġanswered , Ġgiven Ġa Ġquestion Ġin Ġisolation . ĠThis Ġcan Ġserve Ġas Ġa Ġprior Ġfor Ġdownstream Ġquestion - ans w ering . ĠWe Ġdescribe Ġthree Ġbas elines Ġon Ġthe Ġanswer ability Ġtask , Ġand Ġfind Ġthey Ġconsiderably Ġimprove Ġperformance Ġover Ġa Ġmajority - class Ġbaseline . </s> S VM : ĠWe Ġdefine Ġ3 Ġsets Ġof Ġfeatures Ġto Ġcharacterize Ġeach Ġquestion . ĠThe Ġfirst Ġis Ġa Ġsimple Ġbag - of - words Ġset Ġof Ġfeatures Ġover Ġthe Ġquestion Ġ( S VM - B OW ), Ġthe Ġsecond Ġis Ġbag - of - words Ġfeatures Ġof Ġthe Ġquestion Ġas Ġwell Ġas Ġlength Ġof Ġthe Ġquestion Ġin Ġwords Ġ( S VM - B OW Ġ+ ĠL EN ), Ġand Ġlast ly Ġwe Ġextract Ġbag - of - words Ġfeatures , Ġlength Ġof Ġthe Ġquestion Ġin Ġwords Ġas Ġwell Ġas Ġpart - of - speech Ġtags Ġfor Ġthe Ġquestion Ġ( S VM - B OW Ġ+ ĠL EN Ġ+ ĠPOS ). ĠThis Ġresults Ġin Ġvectors Ġof Ġ200 , Ġ201 Ġand Ġ228 Ġdimensions Ġrespectively , Ġwhich Ġare Ġprovided Ġto Ġan ĠS VM Ġwith Ġa Ġlinear Ġkernel . </s> CNN : ĠWe Ġutilize Ġa ĠCNN Ġneural Ġenc oder Ġfor Ġanswer ability Ġprediction . ĠWe Ġuse ĠGl o Ve Ġword Ġembed d ings ĠB IB REF 50 , Ġand Ġa Ġfilter Ġsize Ġof Ġ5 Ġwith Ġ64 Ġfilters Ġto Ġencode Ġquestions . </s> BER T : ĠB ERT ĠB IB REF 51 Ġis Ġa Ġbid irection al Ġtransformer - based Ġlanguage - model ĠB IB REF 52 . ĠWe Ġfine - t une ĠB ERT - base Ġon Ġour Ġbinary Ġanswer ability Ġidentification Ġtask Ġwith Ġa Ġlearning Ġrate Ġof Ġ2 e - 5 Ġfor Ġ3 Ġepoch s , Ġwith Ġa Ġmaximum Ġsequence Ġlength Ġof Ġ128 . </s> Exper imental ĠSetup Ġ: :: ĠPrivacy ĠQuestion ĠAn sw ering </s> Our Ġgoal Ġis Ġto Ġidentify Ġevidence Ġwithin Ġa Ġprivacy Ġpolicy Ġfor Ġquestions Ġasked Ġby Ġa Ġuser . ĠThis Ġis Ġframed Ġas Ġan Ġanswer Ġsentence Ġselection Ġtask , Ġwhere Ġmodels Ġidentify Ġa Ġset Ġof Ġevidence Ġsentences Ġfrom Ġall Ġcandidate Ġsentences Ġin Ġeach Ġpolicy . </s> Exper imental ĠSetup Ġ: :: ĠPrivacy ĠQuestion ĠAn sw ering Ġ: :: ĠEvaluation ĠMet ric </s> Our Ġevaluation Ġmetric Ġfor Ġanswer - sent ence Ġselection Ġis Ġsentence - level ĠF 1 , Ġimplemented Ġsimilar Ġto ĠB IB REF 30 , ĠB IB REF 16 . ĠPrecision Ġand Ġrecall Ġare Ġimplemented Ġby Ġmeasuring Ġthe Ġoverlap Ġbetween Ġpredicted Ġsentences Ġand Ġsets Ġof Ġgold - reference Ġsentences . ĠWe Ġreport Ġthe Ġaverage Ġof Ġthe Ġmaximum ĠF 1 Ġfrom Ġeach Ġn $ -$ 1 Ġsubset , Ġin Ġrelation Ġto Ġthe Ġheld out Ġreference . </s> Exper imental ĠSetup Ġ: :: ĠPrivacy ĠQuestion ĠAn sw ering Ġ: :: ĠBas elines </s> We Ġdescribe Ġbas elines Ġon Ġthis Ġtask , Ġincluding Ġa Ġhuman Ġperformance Ġbaseline . </s> No - Answer ĠBas eline Ġ( NA ) Ġ: ĠMost Ġof Ġthe Ġquestions Ġwe Ġreceive Ġare Ġdifficult Ġto Ġanswer Ġin Ġa Ġlegally - sound Ġway Ġon Ġthe Ġbasis Ġof Ġinformation Ġpresent Ġin Ġthe Ġprivacy Ġpolicy . ĠWe Ġestablish Ġa Ġsimple Ġbaseline Ġto Ġquantify Ġthe Ġeffect Ġof Ġidentifying Ġevery Ġquestion Ġas Ġun answer able . </s> Word ĠCount ĠBas eline Ġ: ĠTo Ġquantify Ġthe Ġeffect Ġof Ġusing Ġsimple Ġlex ical Ġmatching Ġto Ġanswer Ġthe Ġquestions , Ġwe Ġretrieve Ġthe Ġtop Ġcandidate Ġpolicy Ġsentences Ġfor Ġeach Ġquestion Ġusing Ġa Ġword Ġcount Ġbaseline ĠB IB REF 53 , Ġwhich Ġcounts Ġthe Ġnumber Ġof Ġquestion Ġwords Ġthat Ġalso Ġappear Ġin Ġa Ġsentence . ĠWe Ġinclude Ġthe Ġtop Ġ2 , Ġ3 Ġand Ġ5 Ġcandidates Ġas Ġbas elines . </s> BER T : ĠWe Ġimplement Ġtwo ĠB ERT - based Ġbas elines ĠB IB REF 51 Ġfor Ġevidence Ġidentification . ĠFirst , Ġwe Ġtrain ĠB ERT Ġon Ġeach Ġquery - policy Ġsentence Ġpair Ġas Ġa Ġbinary Ġclassification Ġtask Ġto Ġidentify Ġif Ġthe Ġsentence Ġis Ġevidence Ġfor Ġthe Ġquestion Ġor Ġnot Ġ( B ert ). ĠWe Ġalso Ġexperiment Ġwith Ġa Ġtwo - stage Ġclass ifier , Ġwhere Ġwe Ġseparately Ġtrain Ġthe Ġmodel Ġon Ġquestions Ġonly Ġto Ġpredict Ġanswer ability . ĠAt Ġinference Ġtime , Ġif Ġthe Ġanswer able Ġclass ifier Ġpredicts Ġthe Ġquestion Ġis Ġanswer able , Ġthe Ġevidence Ġidentification Ġclass ifier Ġproduces Ġa Ġset Ġof Ġcandidate Ġsentences Ġ( B ert Ġ+ ĠUn answer able ). </s> Human ĠPerformance : ĠWe Ġpick Ġeach Ġreference Ġanswer Ġprovided Ġby Ġan Ġannot ator , Ġand Ġcompute Ġthe ĠF 1 Ġwith Ġrespect Ġto Ġthe Ġremaining Ġreferences , Ġas Ġdescribed Ġin Ġsection Ġ4 . 2 . 1 . ĠEach Ġreference Ġanswer Ġis Ġtreated Ġas Ġthe Ġprediction , Ġand Ġthe Ġremaining Ġn - 1 Ġanswers Ġare Ġtreated Ġas Ġthe Ġgold Ġreference . ĠThe Ġaverage Ġof Ġthe Ġmaximum ĠF 1 Ġacross Ġall Ġreference Ġanswers Ġis Ġcomputed Ġas Ġthe Ġhuman Ġbaseline . </s> Results Ġand ĠDiscussion </s> The Ġresults Ġof Ġthe Ġanswer ability Ġbas elines Ġare Ġpresented Ġin ĠTable ĠT AB REF 31 , Ġand Ġon Ġanswer Ġsentence Ġselection Ġin ĠTable ĠT AB REF 32 . ĠWe Ġobserve Ġthat Ġb ert Ġexhibits Ġthe Ġbest Ġperformance Ġon Ġa Ġbinary Ġanswer ability Ġidentification Ġtask . ĠHowever , Ġmost Ġbas elines Ġconsiderably Ġexceed Ġthe Ġperformance Ġof Ġa Ġmajority - class Ġbaseline . ĠThis Ġsuggests Ġconsiderable Ġinformation Ġin Ġthe Ġquestion , Ġindicating Ġit 's Ġpossible Ġanswer ability Ġwithin Ġthis Ġdomain . </s> Table . TA B REF 32 Ġdescribes Ġthe Ġperformance Ġof Ġour Ġbas elines Ġon Ġthe Ġanswer Ġsentence Ġselection Ġtask . ĠThe ĠNo - answer Ġ( NA ) Ġbaseline Ġperforms Ġat Ġ28 ĠF 1 , Ġproviding Ġa Ġlower Ġbound Ġon Ġperformance Ġat Ġthis Ġtask . ĠWe Ġobserve Ġthat Ġour Ġbest - performing Ġbaseline , ĠBert Ġ+ ĠUn answer able Ġachieves Ġan ĠF 1 Ġof Ġ39 . 8 . ĠThis Ġsuggest Ġthat Ġb ert Ġis Ġcapable Ġof Ġmaking Ġsome Ġprogress Ġtowards Ġanswering Ġquestions Ġin Ġthis Ġdifficult Ġdomain , Ġwhile Ġstill Ġleaving Ġconsiderable Ġhead room Ġfor Ġimprovement Ġto Ġreach Ġhuman Ġperformance . ĠBert Ġ+ ĠUn answer able Ġperformance Ġsuggests Ġthat Ġincorporating Ġinformation Ġabout Ġanswer ability Ġcan Ġhelp Ġin Ġthis Ġdifficult Ġdomain . ĠWe Ġexamine Ġthis Ġchallenging Ġphenomena Ġof Ġun answer ability Ġfurther Ġin ĠSection Ġ. </s> Results Ġand ĠDiscussion Ġ: :: ĠError ĠAnalysis </s> Dis ag reements Ġare Ġanalyzed Ġbased Ġon Ġthe ĠO PP - 115 Ġcategories Ġof Ġeach Ġquestion Ġ( Table . TA B REF 34 ). ĠWe Ġcompare Ġour Ġbest Ġperforming ĠB ERT Ġvariant Ġagainst Ġthe ĠNA Ġmodel Ġand Ġhuman Ġperformance . ĠWe Ġobserve Ġsignificant Ġroom Ġfor Ġimprovement Ġacross Ġall Ġcategories Ġof Ġquestions Ġbut Ġespecially Ġfor Ġfirst Ġparty , Ġthird Ġparty Ġand Ġdata Ġretention Ġcategories . </s> We Ġanalyze Ġthe Ġperformance Ġof Ġour Ġstrongest ĠB ERT Ġvariant , Ġto Ġidentify Ġclasses Ġof Ġerrors Ġand Ġdirections Ġfor Ġfuture Ġimprovement Ġ( Table . 8 ). ĠWe Ġobserve Ġthat Ġa Ġmajority Ġof Ġanswer ability Ġmistakes Ġmade Ġby Ġthe ĠB ERT Ġmodel Ġare Ġquestions Ġwhich Ġare Ġin Ġfact Ġanswer able , Ġbut Ġare Ġidentified Ġas Ġun answer able Ġby ĠB ERT . ĠWe Ġobserve Ġthat ĠB ERT Ġmakes Ġ124 Ġsuch Ġmistakes Ġon Ġthe Ġtest Ġset . ĠWe Ġcollect Ġexpert Ġjudgments Ġon Ġrelevance , Ġsubject ivity Ġ, Ġsilence Ġand Ġinformation Ġabout Ġhow Ġlikely Ġthe Ġquestion Ġis Ġto Ġbe Ġanswered Ġfrom Ġthe Ġprivacy Ġpolicy Ġfrom Ġour Ġexperts . ĠWe Ġfind Ġthat Ġmost Ġof Ġthese Ġmistakes Ġare Ġrelevant Ġquestions . ĠHowever Ġmany Ġof Ġthem Ġwere Ġidentified Ġas Ġsubjective Ġby Ġthe Ġannot ators , Ġand Ġat Ġleast Ġone Ġannot ator Ġmarked Ġ19 Ġof Ġthese Ġquestions Ġas Ġhaving Ġno Ġanswer Ġwithin Ġthe Ġprivacy Ġpolicy . ĠHowever , Ġonly Ġ6 Ġof Ġthese Ġquestions Ġwere Ġunexpected Ġor Ġdo Ġnot Ġusually Ġhave Ġan Ġanswer Ġin Ġprivacy Ġpolicies . ĠThese Ġfindings Ġsuggest Ġthat Ġa Ġmore Ġnuanced Ġunderstanding Ġof Ġanswer ability Ġmight Ġhelp Ġimprove Ġmodel Ġperformance Ġin Ġhis Ġchallenging Ġdomain . </s> Results Ġand ĠDiscussion Ġ: :: ĠWhat Ġmakes ĠQuestions ĠUn answer able ? </s> We Ġfurther Ġask Ġlegal Ġexperts Ġto Ġidentify Ġpotential Ġcauses Ġof Ġun answer ability Ġof Ġquestions . ĠThis Ġanalysis Ġhas Ġconsiderable Ġimplications . ĠWhile Ġpast Ġwork ĠB IB REF 17 Ġhas Ġtreated Ġun answer able Ġquestions Ġas Ġhom ogeneous , Ġa Ġquestion Ġanswering Ġsystem Ġmight Ġwish Ġto Ġhave Ġdifferent Ġtreatments Ġfor Ġdifferent Ġcategories Ġof Ġ` un answer able ' Ġquestions . ĠThe Ġfollowing Ġfactors Ġwere Ġidentified Ġto Ġplay Ġa Ġrole Ġin Ġun answer ability : </s> In com pre hens ibility : ĠIf Ġa Ġquestion Ġis Ġincomprehensible Ġto Ġthe Ġextent Ġthat Ġits Ġmeaning Ġis Ġnot Ġintellig ible . </s> Re lev ance : ĠIs Ġthis Ġquestion Ġin Ġthe Ġscope Ġof Ġwhat Ġcould Ġbe Ġanswered Ġby Ġreading Ġthe Ġprivacy Ġpolicy . </s> Ill - formed ness : ĠIs Ġthis Ġquestion Ġambiguous Ġor Ġvague . ĠAn Ġambiguous Ġstatement Ġwill Ġtypically Ġcontain Ġexpressions Ġthat Ġcan Ġrefer Ġto Ġmultiple Ġpotential Ġexplanations , Ġwhereas Ġa Ġvague Ġstatement Ġcarries Ġa Ġconcept Ġwith Ġan Ġunclear Ġor Ġsoft Ġdefinition . </s> Sil ence : ĠOther Ġpolicies Ġanswer Ġthis Ġtype Ġof Ġquestion Ġbut Ġthis Ġone Ġdoes Ġnot . </s> A typ ical ity : ĠThe Ġquestion Ġis Ġof Ġa Ġnature Ġsuch Ġthat Ġit Ġis Ġunlikely Ġfor Ġany Ġpolicy Ġpolicy Ġto Ġhave Ġan Ġanswer Ġto Ġthe Ġquestion . </s> Our Ġexperts Ġattempt Ġto Ġidentify Ġthe Ġdifferent Ġ` un answer able ' Ġfactors Ġfor Ġall Ġ5 73 Ġsuch Ġquestions Ġin Ġthe Ġcorpus . Ġ4 . 18 % Ġof Ġthe Ġquestions Ġwere Ġidentified Ġas Ġbeing Ġincomprehensible Ġ( for Ġexample , Ġ` any Ġdifficulties Ġto Ġoccupy Ġthe Ġprivacy Ġassistant '). ĠAmong st Ġthe Ġcomprehend able Ġquestions , Ġ50 % Ġwere Ġidentified Ġas Ġlikely Ġto Ġhave Ġan Ġanswer Ġwithin Ġthe Ġprivacy Ġpolicy , Ġ33 . 1 % Ġwere Ġidentified Ġas Ġbeing Ġprivacy - related Ġquestions Ġbut Ġnot Ġwithin Ġthe Ġscope Ġof Ġa Ġprivacy Ġpolicy Ġ( e . g ., Ġ' has ĠV iber Ġhad Ġany Ġprivacy Ġbreaches Ġin Ġthe Ġpast ?' ) Ġand Ġ16 . 9 % Ġof Ġquestions Ġwere Ġidentified Ġas Ġcompletely Ġout - of - scope Ġ( e . g ., Ġ` ' will Ġthe Ġapp Ġconsume Ġmuch Ġspace ?' ). ĠIn Ġthe Ġquestions Ġidentified Ġas Ġrelevant , Ġ32 % Ġwere Ġill - formed Ġquestions Ġthat Ġwere Ġphr ased Ġby Ġthe Ġuser Ġin Ġa Ġmanner Ġconsidered Ġvague Ġor Ġambiguous . ĠOf Ġthe Ġquestions Ġthat Ġwere Ġboth Ġrelevant Ġas Ġwell Ġas Ġ` well - formed ', Ġ95 . 7 % Ġof Ġthe Ġquestions Ġwere Ġnot Ġanswered Ġby Ġthe Ġpolicy Ġin Ġquestion Ġbut Ġit Ġwas Ġreasonable Ġto Ġexpect Ġthat Ġa Ġprivacy Ġpolicy Ġwould Ġcontain Ġan Ġanswer . ĠThe Ġremaining Ġ4 . 3 % Ġwere Ġdescribed Ġas Ġreasonable Ġquestions , Ġbut Ġof Ġa Ġnature Ġgenerally Ġnot Ġdiscussed Ġin Ġprivacy Ġpolicies . ĠThis Ġsuggests Ġthat Ġthe Ġanswer ability Ġof Ġquestions Ġover Ġprivacy Ġpolicies Ġis Ġa Ġcomplex Ġissue , Ġand Ġfuture Ġsystems Ġshould Ġconsider Ġeach Ġof Ġthese Ġfactors Ġwhen Ġserving Ġuser 's Ġinformation Ġseeking Ġintent . </s> We Ġexamine Ġa Ġlarge - scale Ġdataset Ġof ĠâĢ ľ natural âĢ Ŀ Ġun answer able Ġquestions ĠB IB REF 54 Ġbased Ġon Ġreal Ġuser Ġsearch Ġengine Ġqueries Ġto Ġidentify Ġif Ġsimilar Ġun answer ability Ġfactors Ġexist . ĠIt Ġis Ġimportant Ġto Ġnote Ġthat Ġthese Ġquestions Ġhave Ġpreviously Ġbeen Ġfiltered , Ġaccording Ġto Ġa Ġcriteria Ġfor Ġbad Ġquestions Ġdefined Ġas ĠâĢ ľ ( quest ions Ġthat Ġare ) Ġambiguous , Ġincomprehensible , Ġdependent Ġon Ġclear Ġfalse Ġpresupp os itions , Ġopinion - seeking , Ġor Ġnot Ġclearly Ġa Ġrequest Ġfor Ġfactual Ġinformation . âĢ Ŀ ĠAn not ators Ġmade Ġthe Ġdecision Ġbased Ġon Ġthe Ġcontent Ġof Ġthe Ġquestion Ġwithout Ġviewing Ġthe Ġequivalent ĠWikipedia Ġpage . ĠWe Ġrandomly Ġsample Ġ100 Ġquestions Ġfrom Ġthe Ġdevelopment Ġset Ġwhich Ġwere Ġidentified Ġas Ġun answer able , Ġand Ġfind Ġthat Ġ20 % Ġof Ġthe Ġquestions Ġare Ġnot Ġquestions Ġ( e . g ., ĠâĢ ľ all ĠI Ġwant Ġfor Ġchrist mas Ġis Ġyou Ġmar iah Ġcare y Ġtour âĢ Ŀ ). Ġ12 % Ġof Ġquestions Ġare Ġunlikely Ġto Ġever Ġcontain Ġan Ġanswer Ġon ĠWikipedia , Ġcorresponding Ġclosely Ġto Ġour Ġat yp ical ity Ġcategory . Ġ3 % Ġof Ġquestions Ġare Ġunlikely Ġto Ġhave Ġan Ġanswer Ġanywhere Ġ( e . g ., Ġ` what Ġguides ĠSanta Ġhome Ġafter Ġhe Ġhas Ġdelivered Ġpresents ?' ). Ġ7 % Ġof Ġquestions Ġare Ġincomplete Ġor Ġopen - ended Ġ( e . g ., Ġ` the Ġsouth Ġwest Ġwind Ġblows Ġacross Ġnig eria Ġbetween '). Ġ3 % Ġof Ġquestions Ġhave Ġan Ġunres ol vable Ġcore ference Ġ( e . g ., Ġ` how Ġdo Ġi Ġget Ġto ĠWarsaw ĠMissouri Ġfrom Ġhere '). Ġ4 % Ġof Ġquestions Ġare Ġvague , Ġand Ġa Ġfurther Ġ7 % Ġhave Ġunknown Ġsources Ġof Ġerror . Ġ2 % Ġstill Ġcontain Ġfalse Ġpresupp os itions Ġ( e . g ., Ġ` what Ġis Ġthe Ġonly Ġfruit Ġthat Ġdoes Ġnot Ġhave Ġseeds ?' ) Ġand Ġthe Ġremaining Ġ42 % Ġdo Ġnot Ġhave Ġan Ġanswer Ġwithin Ġthe Ġdocument . ĠThis Ġreinforces Ġour Ġbelief Ġthat Ġthough Ġthey Ġhave Ġbeen Ġunder stud ied Ġin Ġpast Ġwork , Ġany Ġquestion Ġanswering Ġsystem Ġinteracting Ġwith Ġreal Ġusers Ġshould Ġexpect Ġto Ġreceive Ġsuch Ġun anticipated Ġand Ġun answer able Ġquestions . </s> Conclusion </s> We Ġpresent ĠPrivacy Q A , Ġthe Ġfirst Ġsignificant Ġcorpus Ġof Ġprivacy Ġpolicy Ġquestions Ġand Ġmore Ġthan Ġ35 00 Ġexpert Ġannotations Ġof Ġrelevant Ġanswers . ĠThe Ġgoal Ġof Ġthis Ġwork Ġis Ġto Ġpromote Ġquestion - ans w ering Ġresearch Ġin Ġthe Ġspecialized Ġprivacy Ġdomain , Ġwhere Ġit Ġcan Ġhave Ġlarge Ġreal - world Ġimpact . ĠStrong Ġneural Ġbas elines Ġon ĠPrivacy Q A Ġachieve Ġa Ġperformance Ġof Ġonly Ġ39 . 8 ĠF 1 Ġon Ġthis Ġcorpus , Ġindicating Ġconsiderable Ġroom Ġfor Ġfuture Ġresearch . ĠFurther , Ġwe Ġshed Ġlight Ġon Ġseveral Ġimportant Ġconsiderations Ġthat Ġaffect Ġthe Ġanswer ability Ġof Ġquestions . ĠWe Ġhope Ġthis Ġcontribution Ġleads Ġto Ġmult idis ciplinary Ġefforts Ġto Ġprecisely Ġunderstand Ġuser Ġintent Ġand Ġreconcile Ġit Ġwith Ġinformation Ġin Ġpolicy Ġdocuments , Ġfrom Ġboth Ġthe Ġprivacy Ġand ĠN LP Ġcommunities . </s> Acknowled gements </s> This Ġresearch Ġwas Ġsupported Ġin Ġpart Ġby Ġgrants Ġfrom Ġthe ĠNational ĠScience ĠFoundation ĠSecure Ġand ĠTrust worthy ĠComputing Ġprogram Ġ( C NS - 13 30 596 , ĠCNS - 13 30 214 , ĠCNS - 15 - 139 57 , ĠCNS - 18 013 16 , ĠCNS - 19 14 486 , ĠCNS - 19 14 444 ) Ġand Ġa ĠDAR PA ĠBrand e is Ġgrant Ġon ĠPersonal ized ĠPrivacy ĠAssist ants Ġ( FA 87 50 - 15 - 2 - 0 277 ). ĠThe ĠUS ĠGovernment Ġis Ġauthorized Ġto Ġreproduce Ġand Ġdistribute Ġre prints Ġfor ĠGovernment al Ġpurposes Ġnot Ġwith standing Ġany Ġcopyright Ġnotation . ĠThe Ġviews Ġand Ġconclusions Ġcontained Ġherein Ġare Ġthose Ġof Ġthe Ġauthors Ġand Ġshould Ġnot Ġbe Ġinterpreted Ġas Ġnecessarily Ġrepresenting Ġthe Ġofficial Ġpolicies Ġor Ġendorsements , Ġeither Ġexpressed Ġor Ġimplied , Ġof Ġthe ĠNS F , ĠDAR PA , Ġor Ġthe ĠUS ĠGovernment . ĠThe Ġauthors Ġwould Ġlike Ġto Ġextend Ġtheir Ġgratitude Ġto ĠElias ĠWright , ĠGian ĠMas ci oli , ĠKi ara ĠPill ay , ĠHarrison ĠKay , ĠEl iel ĠTal o , ĠAlexander ĠF age lla Ġand ĠN . ĠCameron ĠRussell Ġfor Ġproviding Ġtheir Ġvaluable Ġexpertise Ġand Ġinsight Ġto Ġthis Ġeffort . ĠThe Ġauthors Ġare Ġalso Ġgrateful Ġto ĠEdu ard ĠH ovy , ĠLor rie ĠCran or , ĠFlor ian ĠScha ub , ĠJoel ĠRe iden berg , ĠAd ity a ĠPot uk uchi Ġand ĠIgor ĠSh aly min ov Ġfor Ġhelpful Ġdiscussions Ġrelated Ġto Ġthis Ġwork , Ġand Ġto Ġthe Ġthree Ġanonymous Ġreviewers Ġof Ġthis Ġdraft Ġfor Ġtheir Ġconstructive Ġfeedback . ĠFinally , Ġthe Ġauthors Ġwould Ġlike Ġto Ġthank Ġall Ġcrowd workers Ġwho Ġcons ented Ġto Ġparticipate Ġin Ġthis Ġstudy .