Unanswerable
NONE
1907.01468
Introduction </s> In ĠJune Ġ2015 , Ġthe Ġoperators Ġof Ġthe Ġonline Ġdiscussion Ġsite ĠReddit Ġbanned Ġseveral Ġcommunities Ġunder Ġnew Ġanti - har assment Ġrules . ĠB IB REF 0 Ġused Ġthis Ġopportunity Ġto Ġcombine Ġrich Ġonline Ġdata Ġwith Ġcomputational Ġmethods Ġto Ġstudy Ġa Ġcurrent Ġquestion : ĠDoes Ġeliminating Ġthese ĠâĢ ľ echo Ġchambers âĢ Ŀ Ġdiminish Ġthe Ġamount Ġof Ġhate Ġspeech Ġoverall ? ĠExc iting Ġopportunities Ġlike Ġthese , Ġat Ġthe Ġintersection Ġof ĠâĢ ľ th ick âĢ Ŀ Ġcultural Ġand Ġsocietal Ġquestions Ġon Ġthe Ġone Ġhand , Ġand Ġthe Ġcomputational Ġanalysis Ġof Ġrich Ġtextual Ġdata Ġon Ġlarger - than - human Ġscales Ġon Ġthe Ġother , Ġare Ġbecoming Ġincreasingly Ġcommon . </s> Indeed , Ġcomputational Ġanalysis Ġis Ġopening Ġnew Ġpossibilities Ġfor Ġexploring Ġchallenging Ġquestions Ġat Ġthe Ġheart Ġof Ġsome Ġof Ġthe Ġmost Ġpressing Ġcontemporary Ġcultural Ġand Ġsocial Ġissues . ĠWhile Ġa Ġhuman Ġreader Ġis Ġbetter Ġequipped Ġto Ġmake Ġlogical Ġinf erences , Ġresolve Ġambig u ities , Ġand Ġapply Ġcultural Ġknowledge Ġthan Ġa Ġcomputer , Ġhuman Ġtime Ġand Ġattention Ġare Ġlimited . ĠMoreover , Ġmany Ġpatterns Ġare Ġnot Ġobvious Ġin Ġany Ġspecific Ġcontext , Ġbut Ġonly Ġstand Ġout Ġin Ġthe Ġaggregate . ĠFor Ġexample , Ġin Ġa Ġlandmark Ġstudy , ĠB IB REF 1 Ġanalyzed Ġthe Ġauthors hip Ġof ĠThe ĠFederal ist ĠPapers Ġusing Ġa Ġstatistical Ġtext Ġanalysis Ġby Ġfocusing Ġon Ġstyle , Ġbased Ġon Ġthe Ġdistribution Ġof Ġfunction Ġwords , Ġrather Ġthan Ġcontent . ĠAs Ġanother Ġexample , ĠB IB REF 2 Ġstudied Ġwhat Ġdefines ĠEnglish Ġha iku Ġand Ġshowed Ġhow Ġcomputational Ġanalysis Ġand Ġclose Ġreading Ġcan Ġcomplement Ġeach Ġother . ĠComput ational Ġapproaches Ġare Ġvaluable Ġprecisely Ġbecause Ġthey Ġhelp Ġus Ġidentify Ġpatterns Ġthat Ġwould Ġnot Ġotherwise Ġbe Ġdiscern ible . </s> Yet Ġthese Ġapproaches Ġare Ġnot Ġa Ġpan acea . ĠExam ining Ġthick Ġsocial Ġand Ġcultural Ġquestions Ġusing Ġcomputational Ġtext Ġanalysis Ġcarries Ġsignificant Ġchallenges . ĠFor Ġone , Ġtexts Ġare Ġculturally Ġand Ġsocially Ġsituated . ĠThey Ġreflect Ġthe Ġideas , Ġvalues Ġand Ġbeliefs Ġof Ġboth Ġtheir Ġauthors Ġand Ġtheir Ġtarget Ġaudiences , Ġand Ġsuch Ġsubt let ies Ġof Ġmeaning Ġand Ġinterpretation Ġare Ġdifficult Ġto Ġincorporate Ġin Ġcomputational Ġapproaches . ĠFor Ġanother , Ġmany Ġof Ġthe Ġsocial Ġand Ġcultural Ġconcepts Ġwe Ġseek Ġto Ġexamine Ġare Ġhighly Ġcontested ĠâĢĶ Ġhate Ġspeech Ġis Ġjust Ġone Ġsuch Ġexample . ĠCho ices Ġregarding Ġhow Ġto Ġoperational ize Ġand Ġanalyze Ġthese Ġconcepts Ġcan Ġraise Ġserious Ġconcerns Ġabout Ġconceptual Ġvalidity Ġand Ġmay Ġlead Ġto Ġshallow Ġor Ġobvious Ġconclusions , Ġrather Ġthan Ġfindings Ġthat Ġreflect Ġthe Ġdepth Ġof Ġthe Ġquestions Ġwe Ġseek Ġto Ġaddress . </s> These Ġare Ġjust Ġa Ġsmall Ġsample Ġof Ġthe Ġmany Ġopportunities Ġand Ġchallenges Ġfaced Ġin Ġcomputational Ġanalyses Ġof Ġtextual Ġdata . ĠNew Ġpossibilities Ġand Ġfrustrating Ġobstacles Ġemerge Ġat Ġevery Ġstage Ġof Ġresearch , Ġfrom Ġidentification Ġof Ġthe Ġresearch Ġquestion Ġto Ġinterpretation Ġof Ġthe Ġresults . ĠIn Ġthis Ġarticle , Ġwe Ġtake Ġthe Ġreader Ġthrough Ġa Ġtypical Ġresearch Ġprocess Ġthat Ġinvolves Ġmeasuring Ġsocial Ġor Ġcultural Ġconcepts Ġusing Ġcomputational Ġmethods , Ġdiscussing Ġboth Ġthe Ġopportunities Ġand Ġcomplications Ġthat Ġoften Ġarise . ĠIn Ġthe ĠReddit Ġcase , Ġfor Ġexample , Ġhate Ġspeech Ġis Ġmeasured , Ġhowever Ġimperfect ly , Ġby Ġthe Ġpresence Ġof Ġparticular Ġwords Ġsemi - aut om atically Ġextracted Ġfrom Ġa Ġmachine Ġlearning Ġalgorithm . ĠOper ational izations Ġare Ġnever Ġperfect Ġtranslations , Ġand Ġare Ġoften Ġrefined Ġover Ġthe Ġcourse Ġof Ġan Ġinvestigation , Ġbut Ġthey Ġare Ġcrucial . </s> We Ġbegin Ġour Ġexploration Ġwith Ġthe Ġidentification Ġof Ġresearch Ġquestions , Ġproceed Ġthrough Ġdata Ġselection , Ġconceptual ization , Ġand Ġoperational ization , Ġand Ġend Ġwith Ġanalysis Ġand Ġthe Ġinterpretation Ġof Ġresults . ĠThe Ġresearch Ġprocess Ġsounds Ġmore Ġor Ġless Ġlinear Ġthis Ġway , Ġbut Ġeach Ġof Ġthese Ġphases Ġoverl aps , Ġand Ġin Ġsome Ġinstances Ġturns Ġback Ġupon Ġitself . ĠThe Ġanalysis Ġphase , Ġfor Ġexample , Ġoften Ġfeeds Ġback Ġinto Ġthe Ġoriginal Ġresearch Ġquestions , Ġwhich Ġmay Ġcontinue Ġto Ġevolve Ġfor Ġmuch Ġof Ġthe Ġproject . ĠAt Ġeach Ġstage , Ġour Ġdiscussion Ġis Ġcritically Ġinformed Ġby Ġinsights Ġfrom Ġthe Ġhumanities Ġand Ġsocial Ġsciences , Ġfields Ġthat Ġhave Ġfocused Ġon , Ġand Ġworked Ġto Ġtackle , Ġthe Ġchallenges Ġof Ġtextual Ġanalysis âĢĶ albeit Ġat Ġsmaller Ġscales âĢĶ since Ġtheir Ġinception . </s> In Ġdescribing Ġour Ġexperiences Ġwith Ġcomputational Ġtext Ġanalysis , Ġwe Ġhope Ġto Ġachieve Ġthree Ġprimary Ġgoals . ĠFirst , Ġwe Ġaim Ġto Ġshed Ġlight Ġon Ġthorn y Ġissues Ġnot Ġalways Ġat Ġthe Ġforefront Ġof Ġdiscussions Ġabout Ġcomputational Ġtext Ġanalysis Ġmethods . ĠSecond , Ġwe Ġhope Ġto Ġprovide Ġa Ġset Ġof Ġbest Ġpractices Ġfor Ġworking Ġwith Ġthick Ġsocial Ġand Ġcultural Ġconcepts . ĠOur Ġguidance Ġis Ġbased Ġon Ġour Ġown Ġexperiences Ġand Ġis Ġtherefore Ġinherently Ġimperfect . ĠStill , Ġgiven Ġour Ġdiversity Ġof Ġdisciplinary Ġbackgrounds Ġand Ġresearch Ġpractices , Ġwe Ġhope Ġto Ġcapture Ġa Ġrange Ġof Ġideas Ġand Ġidentify Ġcommon alities Ġthat Ġwill Ġresonate Ġfor Ġmany . ĠAnd Ġthis Ġleads Ġto Ġour Ġfinal Ġgoal : Ġto Ġhelp Ġpromote Ġinter disciplinary Ġcollaborations . ĠInter disciplinary Ġinsights Ġand Ġpartnerships Ġare Ġessential Ġfor Ġrealizing Ġthe Ġfull Ġpotential Ġof Ġany Ġcomputational Ġtext Ġanalysis Ġthat Ġinvolves Ġsocial Ġand Ġcultural Ġconcepts , Ġand Ġthe Ġmore Ġwe Ġare Ġable Ġto Ġbridge Ġthese Ġdivides , Ġthe Ġmore Ġfruitful Ġwe Ġbelieve Ġour Ġwork Ġwill Ġbe . </s> Research Ġquestions </s> We Ġtypically Ġstart Ġby Ġidentifying Ġthe Ġquestions Ġwe Ġwish Ġto Ġexplore . ĠCan Ġtext Ġanalysis Ġprovide Ġa Ġnew Ġperspective Ġon Ġa ĠâĢ ľ big Ġquestion âĢ Ŀ Ġthat Ġhas Ġbeen Ġattracting Ġinterest Ġfor Ġyears ? ĠOr Ġcan Ġwe Ġraise Ġnew Ġquestions Ġthat Ġhave Ġonly Ġrecently Ġemerged , Ġfor Ġexample Ġabout Ġsocial Ġmedia ? ĠFor Ġsocial Ġscientists Ġworking Ġin Ġcomputational Ġanalysis , Ġthe Ġquestions Ġare Ġoften Ġgrounded Ġin Ġtheory , Ġasking : ĠHow Ġcan Ġwe Ġexplain Ġwhat Ġwe Ġobserve ? ĠThese Ġquestions Ġare Ġalso Ġinfluenced Ġby Ġthe Ġavailability Ġand Ġaccessibility Ġof Ġdata Ġsources . ĠFor Ġexample , Ġthe Ġchoice Ġto Ġwork Ġwith Ġdata Ġfrom Ġa Ġparticular Ġsocial Ġmedia Ġplatform Ġmay Ġbe Ġpartly Ġdetermined Ġby Ġthe Ġfact Ġthat Ġit Ġis Ġfreely Ġavailable , Ġand Ġthis Ġwill Ġin Ġturn Ġshape Ġthe Ġkinds Ġof Ġquestions Ġthat Ġcan Ġbe Ġasked . ĠA Ġkey Ġoutput Ġof Ġthis Ġphase Ġare Ġthe Ġconcepts Ġto Ġmeasure , Ġfor Ġexample : Ġinfluence ; Ġcopying Ġand Ġreproduction ; Ġthe Ġcreation Ġof Ġpatterns Ġof Ġlanguage Ġuse ; Ġhate Ġspeech . ĠComput ational Ġanalysis Ġof Ġtext Ġmotivated Ġby Ġthese Ġquestions Ġis Ġinsight Ġdriven : Ġwe Ġaim Ġto Ġdescribe Ġa Ġphenomenon Ġor Ġexplain Ġhow Ġit Ġcame Ġabout . ĠFor Ġexample , Ġwhat Ġcan Ġwe Ġlearn Ġabout Ġhow Ġand Ġwhy Ġhate Ġspeech Ġis Ġused Ġor Ġhow Ġthis Ġchanges Ġover Ġtime ? ĠIs Ġhate Ġspeech Ġone Ġthing , Ġor Ġdoes Ġit Ġcomprise Ġmultiple Ġforms Ġof Ġexpression ? ĠIs Ġthere Ġa Ġclear Ġboundary Ġbetween Ġhate Ġspeech Ġand Ġother Ġtypes Ġof Ġspeech , Ġand Ġwhat Ġfeatures Ġmake Ġit Ġmore Ġor Ġless Ġambiguous ? ĠIn Ġthese Ġcases , Ġit Ġis Ġcritical Ġto Ġcommunicate Ġhigh - level Ġpatterns Ġin Ġterms Ġthat Ġare Ġrecognizable . </s> This Ġcontrasts Ġwith Ġmuch Ġof Ġthe Ġwork Ġin Ġcomputational Ġtext Ġanalysis , Ġwhich Ġtends Ġto Ġfocus Ġon Ġautom ating Ġtasks Ġthat Ġhumans Ġperform Ġinefficient ly . ĠThese Ġtasks Ġrange Ġfrom Ġcore Ġlingu istically Ġmotivated Ġtasks Ġthat Ġconstitute Ġthe Ġbackbone Ġof Ġnatural Ġlanguage Ġprocessing , Ġsuch Ġas Ġpart - of - speech Ġtagging Ġand Ġparsing , Ġto Ġfiltering Ġspam Ġand Ġdetecting Ġsentiment . ĠMany Ġtasks Ġare Ġmotivated Ġby Ġapplications , Ġfor Ġexample Ġto Ġautomatically Ġblock Ġonline Ġtrolls . ĠSuccess , Ġthen , Ġis Ġoften Ġmeasured Ġby Ġperformance , Ġand Ġcommunicating Ġwhy Ġa Ġcertain Ġprediction Ġwas Ġmade âĢĶ for Ġexample , Ġwhy Ġa Ġdocument Ġwas Ġlabeled Ġas Ġpositive Ġsentiment , Ġor Ġwhy Ġa Ġword Ġwas Ġclassified Ġas Ġa Ġnoun âĢĶ is Ġless Ġimportant Ġthan Ġthe Ġaccuracy Ġof Ġthe Ġprediction Ġitself . ĠThe Ġapproaches Ġwe Ġuse Ġand Ġwhat Ġwe Ġmean Ġby Ġ` success ' Ġare Ġthus Ġguided Ġby Ġour Ġresearch Ġquestions . </s> Domain Ġexperts Ġand Ġfellow Ġresearchers Ġcan Ġprovide Ġfeedback Ġon Ġquestions Ġand Ġhelp Ġwith Ġdynamically Ġrev ising Ġthem . ĠFor Ġexample , Ġthey Ġmay Ġsay ĠâĢ ľ we Ġalready Ġthink Ġwe Ġknow Ġthat âĢ Ŀ , ĠâĢ ľ that 's Ġtoo ĠnaÃ¯ve âĢ Ŀ , ĠâĢ ľ that Ġdoesn 't Ġreflect Ġsocial Ġreality âĢ Ŀ Ġ( negative ); ĠâĢ ľ two Ġmajor Ġcamps Ġin Ġthe Ġfield Ġwould Ġgive Ġdifferent Ġanswers Ġto Ġthat Ġquestion âĢ Ŀ Ġ( neutral ); ĠâĢ ľ we Ġtried Ġto Ġlook Ġat Ġthat Ġback Ġin Ġthe Ġ1960 s , Ġbut Ġwe Ġdidn 't Ġhave Ġthe Ġtechnology âĢ Ŀ Ġ( positive ); Ġand ĠâĢ ľ that Ġsounds Ġlike Ġsomething Ġthat Ġpeople Ġwho Ġmade Ġthat Ġarchive Ġwould Ġlove âĢ Ŀ , ĠâĢ ľ that 's Ġa Ġreally Ġfundamental Ġquestion âĢ Ŀ Ġ( very Ġpositive ). </s> Sometimes Ġwe Ġalso Ġhope Ġto Ġconnect Ġto Ġmultiple Ġdisciplines . ĠFor Ġexample , Ġwhile Ġfocusing Ġon Ġthe Ġhuman istic Ġconcerns Ġof Ġan Ġarchive , Ġwe Ġcould Ġalso Ġask Ġsocial Ġquestions Ġsuch Ġas ĠâĢ ľ is Ġthis Ġarchive Ġmore Ġabout Ġcollaborative Ġprocesses , Ġculture - building Ġor Ġnorm Ġcreation ? âĢ Ŀ Ġor ĠâĢ ľ how Ġwell Ġdoes Ġthis Ġarchive Ġreflect Ġthe Ġsociety Ġin Ġwhich Ġit Ġis Ġembedded ?" ĠB IB REF 3 Ġused Ġquantitative Ġmethods Ġto Ġtell Ġa Ġstory Ġabout ĠDarwin 's Ġintellectual Ġdevelopment âĢĶ an Ġessential Ġbi ographical Ġquestion Ġfor Ġa Ġkey Ġfigure Ġin Ġthe Ġhistory Ġof Ġscience . ĠAt Ġthe Ġsame Ġtime , Ġtheir Ġmethods Ġconnected ĠDarwin 's Ġdevelopment Ġto Ġthe Ġchanging Ġlandscape Ġof ĠVictorian Ġscientific Ġculture , Ġallowing Ġthem Ġto Ġcontrast ĠDarwin 's ĠâĢ ľ for aging âĢ Ŀ Ġin Ġthe Ġscientific Ġliterature Ġof Ġhis Ġtime Ġto Ġthe Ġways Ġin Ġwhich Ġthat Ġliterature Ġwas Ġitself Ġproduced . ĠFinally , Ġtheir Ġmethods Ġprovided Ġa Ġcase Ġstudy , Ġand Ġvalidation Ġof Ġtechnical Ġapproaches , Ġfor Ġcognitive Ġscientists Ġwho Ġare Ġinterested Ġin Ġhow Ġpeople Ġexplore Ġand Ġexploit Ġsources Ġof Ġknowledge . </s> Questions Ġabout Ġpotential ĠâĢ ľ du al Ġuse âĢ Ŀ Ġmay Ġalso Ġarise . ĠReturning Ġto Ġour Ġintroductory Ġexample , ĠB IB REF 0 Ġstarted Ġwith Ġa Ġde cept ively Ġsimple Ġquestion : Ġif Ġan Ġinternet Ġplatform Ġeliminates Ġforums Ġfor Ġhate Ġspeech , Ġdoes Ġthis Ġimpact Ġhate Ġspeech Ġin Ġother Ġforums ? ĠThe Ġresearch Ġwas Ġmotivated Ġby Ġthe Ġbelief Ġthat Ġa Ġrising Ġtide Ġof Ġonline Ġhate Ġspeech Ġwas Ġ( and Ġis ) Ġmaking Ġthe Ġinternet Ġincreasingly Ġunf riend ly Ġfor Ġdis em powered Ġgroups , Ġincluding Ġminorities , Ġwomen , Ġand ĠLB GT Q Ġindividuals . ĠYet Ġthe Ġpossibility Ġof Ġdual Ġuse Ġtroubled Ġthe Ġresearchers Ġfrom Ġthe Ġonset . ĠCould Ġthe Ġmethodology Ġbe Ġadopted Ġto Ġtarget Ġthe Ġspeech Ġof Ġgroups Ġlike ĠBlack ĠLives ĠMatter ? ĠCould Ġit Ġbe Ġadopted Ġby Ġrepressive Ġgovernments Ġto Ġminimize Ġonline Ġdissent ? ĠWhile Ġthese Ġconcerns Ġremained , Ġthey Ġconcluded Ġthat Ġhypothetical Ġdual Ġuse Ġscenarios Ġdid Ġnot Ġoutweigh Ġthe Ġtangible Ġcontribution Ġthis Ġresearch Ġcould Ġoffer Ġtowards Ġmaking Ġthe Ġonline Ġenvironment Ġmore Ġequal Ġand Ġjust . </s> Data </s> The Ġnext Ġstep Ġinvolves Ġdeciding Ġon Ġthe Ġdata Ġsources , Ġcollecting Ġand Ġcompiling Ġthe Ġdataset , Ġand Ġinspecting Ġits Ġmetadata . </s> Data Ġacquisition </s> Many Ġscholars Ġin Ġthe Ġhumanities Ġand Ġthe Ġsocial Ġsciences Ġwork Ġwith Ġsources Ġthat Ġare Ġnot Ġavailable Ġin Ġdigital Ġform , Ġand Ġindeed Ġmay Ġnever Ġbe Ġdigit ized . ĠOthers Ġwork Ġwith Ġboth Ġanalogue Ġand Ġdigit ized Ġmaterials , Ġand Ġthe Ġincreasing Ġdigit ization Ġof Ġarchives Ġhas Ġopened Ġopportunities Ġto Ġstudy Ġthese Ġarchives Ġin Ġnew Ġways . ĠWe Ġcan Ġgo Ġto Ġthe Ġcanonical Ġarchive Ġor Ġopen Ġup Ġsomething Ġthat Ġnobody Ġhas Ġstudied Ġbefore . ĠFor Ġexample , Ġwe Ġmight Ġfocus Ġon Ġmajor Ġhistorical Ġmoments Ġ( French ĠRevolution , Ġpost - Mil ose vic ĠSerbia ) Ġor Ġcritical Ġepoch s Ġ( Britain Ġentering Ġthe ĠVictorian Ġera , Ġthe Ġtransition Ġfrom ĠLatin Ġto Ġproto - Rom ance ). ĠOr , Ġwe Ġcould Ġlook Ġfor Ġrecords Ġof Ġhow Ġpeople Ġconducted Ġscience , Ġwrote Ġand Ġconsumed Ġliterature , Ġand Ġworked Ġout Ġtheir Ġphilosophies . </s> A Ġgrowing Ġnumber Ġof Ġresearchers Ġwork Ġwith Ġborn - digital Ġsources Ġor Ġdata . ĠBorn - digital Ġdata , Ġe . g ., Ġfrom Ġsocial Ġmedia , Ġgenerally Ġdo Ġnot Ġinvolve Ġdirect Ġelic itation Ġfrom Ġparticipants Ġand Ġtherefore Ġenable Ġunob tr usive Ġmeasurements ĠB IB REF 5 Ġ, ĠB IB REF 6 Ġ. ĠIn Ġcontrast , Ġmethods Ġlike Ġsurveys Ġsometimes Ġelicit Ġaltered Ġresponses Ġfrom Ġparticipants , Ġwho Ġmight Ġadapt Ġtheir Ġresponses Ġto Ġwhat Ġthey Ġthink Ġis Ġexpected . ĠMoreover , Ġborn - digital Ġdata Ġis Ġoften Ġmassive , Ġenabling Ġlarge - scale Ġstudies Ġof Ġlanguage Ġand Ġbehavior Ġin Ġa Ġvariety Ġof Ġsocial Ġcontexts . </s> Still , Ġmany Ġscholars Ġin Ġthe Ġsocial Ġsciences Ġand Ġhumanities Ġwork Ġwith Ġmultiple Ġdata Ġsources . ĠThe Ġvariety Ġof Ġsources Ġtypically Ġused Ġmeans Ġthat Ġmore Ġthan Ġone Ġdata Ġcollection Ġmethod Ġis Ġoften Ġrequired . ĠFor Ġexample , Ġa Ġproject Ġexamining Ġcoverage Ġof Ġa ĠUK ĠGeneral ĠElection , Ġcould Ġdraw Ġdata Ġfrom Ġtraditional Ġmedia , Ġweb Ġarchives , ĠTwitter Ġand ĠFacebook , Ġcampaign Ġmanifest os , Ġetc . Ġand Ġmight Ġcombine Ġtextual Ġanalysis Ġof Ġthese Ġmaterials Ġwith Ġsurveys , Ġlaboratory Ġexperiments , Ġor Ġfield Ġobservations Ġoffline . ĠIn Ġcontrast , Ġmany Ġcomputational Ġstudies Ġbased Ġon Ġborn - digital Ġdata Ġhave Ġfocused Ġon Ġone Ġspecific Ġsource , Ġsuch Ġas ĠTwitter . </s> The Ġuse Ġof Ġborn - digital Ġdata Ġraises Ġethical Ġconcerns . ĠAlthough Ġearly Ġstudies Ġoften Ġtreated Ġprivacy Ġas Ġa Ġbinary Ġconstruct , Ġmany Ġnow Ġacknowledge Ġits Ġcomplexity ĠB IB REF 7 Ġ. ĠConvers ations Ġon Ġprivate Ġmatters Ġcan Ġbe Ġposted Ġonline , Ġvisible Ġfor Ġall , Ġbut Ġsocial Ġnorms Ġregarding Ġwhat Ġshould Ġbe Ġconsidered Ġpublic Ġinformation Ġmay Ġdiffer Ġfrom Ġthe Ġdata 's Ġexplicit Ġvisibility Ġsettings . ĠOften Ġno Ġinformed Ġconsent Ġhas Ġbeen Ġobtained , Ġraising Ġconcerns Ġand Ġchallenges Ġregarding Ġpublishing Ġcontent Ġand Ġpotentially Ġharmful Ġsecondary Ġuses ĠB IB REF 8 Ġ, ĠB IB REF 4 Ġ. </s> Recently , Ġconcerns Ġabout Ġpotential Ġharms Ġstemming Ġfrom Ġsecondary Ġuses Ġhave Ġled Ġa Ġnumber Ġof Ġdigital Ġservice Ġproviders Ġto Ġrestrict Ġaccess Ġto Ġborn - digital Ġdata . ĠFacebook Ġand ĠTwitter , Ġfor Ġexample , Ġhave Ġreduced Ġor Ġeliminated Ġpublic Ġaccess Ġto Ġtheir Ġapplication Ġprogramming Ġinterfaces Ġ( AP Is ) Ġand Ġexpressed Ġhesitation Ġabout Ġallowing Ġacademic Ġresearchers Ġto Ġuse Ġdata Ġfrom Ġtheir Ġplatforms Ġto Ġexamine Ġcertain Ġsensitive Ġor Ġcontroversial Ġtopics . ĠDespite Ġthe Ġseeming Ġabundance Ġof Ġborn - digital Ġdata , Ġwe Ġtherefore Ġcannot Ġtake Ġits Ġavailability Ġfor Ġgranted . </s> Working Ġwith Ġdata Ġthat Ġsomeone Ġelse Ġhas Ġacquired Ġpresents Ġadditional Ġproblems Ġrelated Ġto Ġproven ance Ġand Ġcontextual isation . ĠIt Ġmay Ġnot Ġalways Ġbe Ġpossible Ġto Ġdetermine Ġthe Ġcriteria Ġapplied Ġduring Ġthe Ġcreation Ġprocess . ĠFor Ġexample , Ġwhy Ġwere Ġcertain Ġnewspapers Ġdigit ized Ġbut Ġnot Ġothers , Ġand Ġwhat Ġdoes Ġthis Ġsay Ġabout Ġthe Ġcollection ? ĠSimilar Ġquestions Ġarise Ġwith Ġthe Ġuse Ġof Ġborn - digital Ġdata . ĠFor Ġinstance , Ġwhen Ġusing Ġthe ĠInternet ĠArchive âĢ Ļ s ĠWay back ĠMachine Ġto Ġgather Ġdata Ġfrom Ġarchived Ġweb Ġpages , Ġwe Ġneed Ġto Ġconsider Ġwhat Ġpages Ġwere Ġcaptured , Ġwhich Ġare Ġlikely Ġmissing , Ġand Ġwhy . </s> We Ġmust Ġoften Ġrep ur pose Ġborn - digital Ġdata Ġ( e . g ., ĠTwitter Ġwas Ġnot Ġdesigned Ġto Ġmeasure Ġpublic Ġopinion ), Ġbut Ġdata Ġbiases Ġmay Ġlead Ġto Ġspurious Ġresults Ġand Ġlimit Ġjustification Ġfor Ġgeneral ization . ĠIn Ġparticular , Ġdata Ġcollected Ġvia Ġblack Ġbox ĠAPIs Ġdesigned Ġfor Ġcommercial , Ġnot Ġresearch , Ġpurposes Ġare Ġlikely Ġto Ġintroduce Ġbiases Ġinto Ġthe Ġinf erences Ġwe Ġdraw , Ġand Ġthe Ġclosed Ġnature Ġof Ġthese ĠAPIs Ġmeans Ġwe Ġrarely Ġknow Ġwhat Ġbiases Ġare Ġintroduced , Ġlet Ġalone Ġhow Ġseverely Ġthey Ġmight Ġimpact Ġour Ġresearch ĠB IB REF 10 Ġ. ĠThese , Ġhowever , Ġare Ġnot Ġnew Ġproblems . ĠHist orians , Ġfor Ġexample , Ġhave Ġalways Ġunderstood Ġthat Ġtheir Ġsources Ġwere Ġproduced Ġwithin Ġparticular Ġcontexts Ġand Ġfor Ġparticular Ġpurposes , Ġwhich Ġare Ġnot Ġalways Ġapparent Ġto Ġus . </s> Non - represent ative Ġdata Ġcan Ġstill Ġbe Ġuseful Ġfor Ġmaking Ġcomparisons Ġwithin Ġa Ġsample . ĠIn Ġthe Ġintroductory Ġexample Ġon Ġhate Ġspeech ĠB IB REF 0 Ġ, Ġthe ĠReddit Ġforums Ġdo Ġnot Ġpresent Ġa Ġcomprehensive Ġor Ġbalanced Ġpicture Ġof Ġhate Ġspeech : Ġthe Ġwriting Ġis Ġalmost Ġexclusively Ġin ĠEnglish , Ġthe Ġtargets Ġof Ġhate Ġspeech Ġare Ġmainly Ġrestricted Ġ( e . g ., Ġto Ġblack Ġpeople , Ġor Ġwomen ), Ġand Ġthe Ġpopulation Ġof Ġwriters Ġis Ġshaped Ġby ĠReddit 's Ġdemographics , Ġwhich Ġskew Ġtowards Ġyoung Ġwhite Ġmen . ĠThese Ġbiases Ġlimit Ġthe Ġgeneral iz ability Ġof Ġthe Ġfindings , Ġwhich Ġcannot Ġbe Ġextrap olated Ġto Ġother Ġlanguages , Ġother Ġtypes Ġof Ġhate Ġspeech , Ġand Ġother Ġdemographic Ġgroups . ĠHowever , Ġbecause Ġthe Ġfindings Ġare Ġbased Ġon Ġmeasurements Ġon Ġthe Ġsame Ġsort Ġof Ġhate Ġspeech Ġand Ġthe Ġsame Ġpopulation Ġof Ġwriters , Ġas Ġlong Ġas Ġthe Ġcollected Ġdata Ġare Ġrepresentative Ġof Ġthis Ġspecific Ġpopulation , Ġthese Ġbiases Ġdo Ġnot Ġpose Ġan Ġint ract able Ġvalidity Ġproblem Ġif Ġclaims Ġare Ġproperly Ġrestricted . </s> The Ġsize Ġof Ġmany Ġnewly Ġavailable Ġdatasets Ġis Ġone Ġof Ġtheir Ġmost Ġappealing Ġcharacteristics . ĠBig ger Ġdatasets Ġoften Ġmake Ġstatistics Ġmore Ġrobust . ĠThe Ġsize Ġneeded Ġfor Ġa Ġcomputational Ġtext Ġanalysis Ġdepends Ġon Ġthe Ġresearch Ġgoal : ĠWhen Ġit Ġinvolves Ġstudying Ġrare Ġevents , Ġbigger Ġdatasets Ġare Ġneeded . ĠHowever , Ġlarger Ġis Ġnot Ġalways Ġbetter . ĠSome Ġvery Ġlarge Ġarchives Ġare ĠâĢ ľ secret ly âĢ Ŀ Ġcollections Ġof Ġmultiple Ġand Ġdistinct Ġprocesses Ġthat Ġno Ġin - field Ġscholar Ġwould Ġconsider Ġrelated . ĠFor Ġexample , ĠGoogle ĠBooks Ġis Ġfrequently Ġused Ġto Ġstudy Ġcultural Ġpatterns , Ġbut Ġthe Ġover - represent ation Ġof Ġscientific Ġarticles Ġin ĠGoogle Ġbooks Ġcan Ġbe Ġproblematic ĠB IB REF 11 Ġ. ĠEven Ġvery Ġlarge Ġborn - digital Ġdatasets Ġusually Ġcover Ġlimited Ġtimes p ans Ġcompared Ġto , Ġe . g ., Ġthe ĠGutenberg Ġarchive Ġof ĠBritish Ġnovels . </s> This Ġstage Ġof Ġthe Ġresearch Ġalso Ġraises Ġimportant Ġquestions Ġabout Ġfairness . ĠAre Ġmarginalized Ġgroups , Ġfor Ġexample , Ġrepresented Ġin Ġthe Ġtweets Ġwe Ġhave Ġcollected ? ĠIf Ġnot , Ġwhat Ġtypes Ġof Ġbiases Ġmight Ġresult Ġfrom Ġanalyses Ġrelying Ġon Ġthose Ġtweets ? </s> Local Ġexperts Ġand ĠâĢ ľ in form ants âĢ Ŀ Ġcan Ġhelp Ġnavigate Ġthe Ġdata . ĠThey Ġcan Ġhelp Ġunderstand Ġthe Ġrole Ġan Ġarchive Ġplays Ġin Ġthe Ġtime Ġand Ġplace . ĠThey Ġmight Ġtell Ġus : ĠIs Ġthis Ġthe Ġcentral Ġarchive , Ġor Ġa Ġperipheral Ġone ? ĠWhat Ġmakes Ġit Ġunusual ? ĠOr Ġthey Ġmight Ġtell Ġus Ġhow Ġcertain Ġunder represented Ġcommunities Ġuse Ġa Ġsocial Ġmedia Ġplatform Ġand Ġadvise Ġus Ġon Ġstrategies Ġfor Ġensuring Ġour Ġdata Ġcollection Ġincludes Ġtheir Ġperspectives . </s> However , Ġwhen Ġit Ġis Ġpractically Ġinf eas ible Ġto Ġnavigate Ġthe Ġdata Ġin Ġthis Ġway âĢĶ for Ġinstance , Ġwhen Ġwe Ġcannot Ġdetermine Ġwhat Ġis Ġmissing Ġfrom ĠTwitter 's ĠStreaming ĠAPI Ġor Ġwhat Ġweb pages Ġare Ġleft Ġout Ġof Ġthe ĠInternet ĠArchive âĢĶ we Ġshould Ġbe Ġopen Ġabout Ġthe Ġlimitations Ġof Ġour Ġanalyses , Ġacknowledging Ġthe Ġflaws Ġin Ġour Ġdata Ġand Ġdrawing Ġcautious Ġand Ġreasonable Ġconclusions Ġfrom Ġthem . ĠIn Ġall Ġcases , Ġwe Ġshould Ġreport Ġthe Ġchoices Ġwe Ġhave Ġmade Ġwhen Ġcreating Ġor Ġre - using Ġany Ġdataset . </s> Comp iling Ġdata </s> After Ġidentifying Ġthe Ġdata Ġsource ( s ), Ġthe Ġnext Ġstep Ġis Ġcompiling Ġthe Ġdata . ĠThis Ġstep Ġis Ġfundamental : Ġif Ġthe Ġsources Ġcannot Ġsupport Ġa Ġconvincing Ġresult , Ġno Ġresult Ġwill Ġbe Ġconvincing . ĠIn Ġmany Ġcases , Ġthis Ġinvolves Ġdefining Ġa ĠâĢ ľ core " Ġset Ġof Ġdocuments Ġand Ġa ĠâĢ ľ com par ison " Ġset . ĠWe Ġoften Ġhave Ġa Ġspecific Ġset Ġof Ġdocuments Ġin Ġmind : Ġan Ġauthor 's Ġwork , Ġa Ġparticular Ġjournal , Ġa Ġtime Ġperiod . ĠBut Ġif Ġwe Ġwant Ġto Ġsay Ġthat Ġthis ĠâĢ ľ core " Ġset Ġhas Ġsome Ġdistinctive Ġproperty , Ġwe Ġneed Ġa ĠâĢ ľ com par ison " Ġset . ĠExp anding Ġthe Ġcollection Ġbeyond Ġthe Ġdocuments Ġthat Ġwe Ġwould Ġimmediately Ġthink Ġof Ġhas Ġthe Ġbeneficial Ġeffect Ġof Ġincreasing Ġour Ġsample Ġsize . ĠHaving Ġmore Ġsources Ġincreases Ġthe Ġchance Ġthat Ġwe Ġwill Ġnotice Ġsomething Ġconsistent Ġacross Ġmany Ġindividually Ġvarying Ġcontexts . </s> Comp aring Ġsets Ġof Ġdocuments Ġcan Ġsometimes Ġsupport Ġcausal Ġinference , Ġpresented Ġas Ġa Ġcontrast Ġbetween Ġa Ġtreatment Ġgroup Ġand Ġa Ġcontrol . ĠIn ĠB IB REF 0 Ġ, Ġthe Ġtreatment Ġconsisted Ġof Ġthe Ġtext Ġwritten Ġin Ġthe Ġtwo Ġforums Ġthat Ġwere Ġeventually Ġclosed Ġby ĠReddit . ĠHowever , Ġidentifying Ġa Ġcontrol Ġgroup Ġrequired Ġa Ġconsiderable Ġamount Ġof Ġtime Ġand Ġeffort . ĠReddit Ġis Ġa Ġdiverse Ġplatform , Ġwith Ġa Ġwide Ġvariety Ġof Ġinteract ional Ġand Ġlinguistic Ġstyles ; Ġit Ġwould Ġbe Ġpointless Ġto Ġcompare Ġhate Ġspeech Ġforums Ġagainst Ġforums Ġdedicated Ġto , Ġsay , Ġpictures Ġof Ġwrecked Ġbicycles . ĠChand rase k har an Ġet Ġal . Ġused Ġa Ġmatching Ġdesign , Ġpop ulating Ġthe Ġcontrol Ġgroup Ġwith Ġforums Ġthat Ġwere Ġas Ġsimilar Ġas Ġpossible Ġto Ġthe Ġtreatment Ġgroup , Ġbut Ġwere Ġnot Ġbanned Ġfrom ĠReddit . ĠThe Ġgoal Ġis Ġto Ġestimate Ġthe Ġcounter fact ual Ġscenario : Ġin Ġthis Ġcase , Ġwhat Ġwould Ġhave Ġhappened Ġhad Ġthe Ġsite Ġnot Ġtaken Ġaction Ġagainst Ġthese Ġspecific Ġforums ? ĠAn Ġideal Ġcontrol Ġwould Ġmake Ġit Ġpossible Ġto Ġdistinguish Ġthe Ġeffect Ġof Ġthe Ġtreatment ĠâĢĶ Ġclosing Ġthe Ġforums ĠâĢĶ Ġfrom Ġother Ġidiosyncr atic Ġproperties Ġof Ġtexts Ġthat Ġwere Ġtreated . </s> We Ġalso Ġlook Ġfor Ġcategories Ġof Ġdocuments Ġthat Ġmight Ġnot Ġbe Ġuseful . ĠWe Ġmight Ġremove Ġdocuments Ġthat Ġare Ġmeta - disc ourse , Ġlike Ġintrodu ctions Ġand Ġnotes , Ġor Ġdocuments Ġthat Ġare Ġin Ġa Ġlanguage Ġthat Ġis Ġnot Ġthe Ġprimary Ġlanguage Ġof Ġthe Ġcollection , Ġor Ġdupl icates Ġwhen Ġwe Ġare Ġworking Ġwith Ġarchived Ġweb Ġpages . ĠHowever , Ġwe Ġneed Ġto Ġcarefully Ġconsider Ġthe Ġpotential Ġconsequences Ġof Ġinformation Ġwe Ġremove . ĠDoes Ġits Ġremoval Ġalter Ġthe Ġdata , Ġor Ġthe Ġinterpretation Ġof Ġthe Ġdata , Ġwe Ġare Ġanalyzing ? ĠAre Ġwe Ġlosing Ġanything Ġthat Ġmight Ġbe Ġvaluable Ġat Ġa Ġlater Ġstage ? </s> Lab els Ġand Ġmetadata </s> Sometimes Ġall Ġwe Ġhave Ġis Ġdocuments , Ġbut Ġoften Ġwe Ġwant Ġto Ġlook Ġat Ġdocuments Ġin Ġthe Ġcontext Ġof Ġsome Ġadditional Ġinformation , Ġor Ġmetadata . ĠThis Ġadditional Ġinformation Ġcould Ġtell Ġus Ġabout Ġthe Ġcreation Ġof Ġdocuments Ġ( date , Ġauthor , Ġforum ), Ġor Ġabout Ġthe Ġreception Ġof Ġdocuments Ġ( fl agged Ġas Ġhate Ġspeech , Ġhelpful Ġreview ). ĠInformation Ġabout Ġtext Ġsegments Ġcan Ġbe Ġextremely Ġvaluable , Ġbut Ġit Ġis Ġalso Ġprone Ġto Ġerrors , Ġinconsistencies , Ġbias , Ġand Ġmissing Ġinformation . ĠExam ining Ġmetadata Ġis Ġa Ġgood Ġway Ġto Ġcheck Ġa Ġcollection 's Ġbalance Ġand Ġrepresent at iveness . ĠAre Ġsources Ġdisproportionately Ġof Ġone Ġform ? ĠIs Ġthe Ġcollection Ġmissing Ġa Ġspecific Ġtime Ġwindow ? ĠThis Ġtype Ġof Ġcur ation Ġcan Ġbe Ġextremely Ġtime Ġconsuming Ġas Ġit Ġmay Ġrequire Ġexpert Ġlabeling , Ġbut Ġit Ġoften Ġleads Ġto Ġthe Ġmost Ġcompelling Ġresults . ĠSometimes Ġmetadata Ġare Ġalso Ġused Ġas Ġtarget Ġlabels Ġto Ġdevelop Ġmachine Ġlearning Ġmodels . ĠBut Ġusing Ġthem Ġas Ġa ĠâĢ ľ ground Ġtruth âĢ Ŀ Ġrequires Ġcaution . ĠLab els Ġsometimes Ġmean Ġsomething Ġdifferent Ġthan Ġwe Ġexpect . ĠFor Ġexample , Ġa Ġdown Ġvote Ġfor Ġa Ġsocial Ġmedia Ġpost Ġcould Ġindicate Ġthat Ġthe Ġcontent Ġis Ġoffensive , Ġor Ġthat Ġthe Ġvoter Ġsimply Ġdisagreed Ġwith Ġthe Ġexpressed Ġview . </s> Con cept ual ization </s> A Ġcore Ġstep Ġin Ġmany Ġanalyses Ġis Ġtranslating Ġsocial Ġand Ġcultural Ġconcepts Ġ( such Ġas Ġhate Ġspeech , Ġrumor , Ġor Ġconversion ) Ġinto Ġmeasurable Ġquantities . ĠBefore Ġwe Ġcan Ġdevelop Ġmeasurements Ġfor Ġthese Ġconcepts Ġ( the Ġoperational ization Ġstep , Ġor Ġthe ĠâĢ ľ im plementation âĢ Ŀ Ġstep Ġas Ġden oted Ġby ĠB IB REF 12 Ġ), Ġwe Ġneed Ġto Ġdefine Ġthem . ĠIn Ġthe Ġconceptual ization Ġphase Ġwe Ġoften Ġstart Ġwith Ġquestions Ġsuch Ġas : Ġwho Ġare Ġthe Ġdomain Ġexperts , Ġand Ġhow Ġhave Ġthey Ġapproached Ġthe Ġtopic ? ĠWe Ġare Ġlooking Ġfor Ġa Ġdefinition Ġof Ġthe Ġconcept Ġthat Ġis Ġflexible Ġenough Ġto Ġapply Ġon Ġour Ġdataset , Ġyet Ġformal Ġenough Ġfor Ġcomputational Ġresearch . ĠFor Ġexample , Ġour Ġintroductory Ġstudy Ġon Ġhate Ġspeech ĠB IB REF 0 Ġused Ġa Ġstatement Ġon Ġhate Ġspeech Ġproduced Ġby Ġthe ĠEuropean ĠUnion ĠCourt Ġof ĠHuman ĠRights . ĠThe Ġgoal Ġwas Ġnot Ġto Ġimplement Ġthis Ġdefinition Ġdirectly Ġin Ġsoftware Ġbut Ġto Ġuse Ġit Ġas Ġa Ġreference Ġpoint Ġto Ġanchor Ġsubsequent Ġanalyses . </s> If Ġwe Ġwant Ġto Ġmove Ġbeyond Ġthe Ġuse Ġof Ġad Ġhoc Ġdefinitions , Ġit Ġcan Ġbe Ġuseful Ġto Ġdistinguish Ġbetween Ġwhat Ġpolitical Ġscientists ĠAd cock Ġand ĠColl ier Ġcall Ġthe ĠâĢ ľ background Ġconcept âĢ Ŀ Ġand Ġthe ĠâĢ ľ system at ized Ġconcept âĢ Ŀ ĠB IB REF 13 Ġ. ĠThe Ġbackground Ġconcept Ġcomprises Ġthe Ġfull Ġand Ġdiverse Ġset Ġof Ġmeanings Ġthat Ġmight Ġbe Ġassociated Ġwith Ġa Ġparticular Ġterm . ĠThis Ġinvolves Ġdel ving Ġinto Ġtheoretical , Ġconceptual , Ġand Ġempirical Ġstudies Ġto Ġassess Ġhow Ġa Ġconcept Ġhas Ġbeen Ġdefined Ġby Ġother Ġscholars Ġand , Ġmost Ġimportantly , Ġto Ġdetermine Ġwhich Ġdefinition Ġis Ġmost Ġappropriate Ġfor Ġthe Ġparticular Ġresearch Ġquestion Ġand Ġthe Ġtheoretical Ġframework Ġin Ġwhich Ġit Ġis Ġsituated . ĠThat Ġdefinition , Ġin Ġturn , Ġrepresents Ġthe Ġsystem at ized Ġconcept : Ġthe Ġformulation Ġthat Ġis Ġadopted Ġfor Ġthe Ġstudy . </s> It Ġis Ġimportant Ġto Ġconsider Ġthat Ġfor Ġsocial Ġand Ġcultural Ġconcepts Ġthere Ġis Ġno Ġabsolute Ġground Ġtruth . ĠThere Ġare Ġoften Ġmultiple Ġvalid Ġdefinitions Ġfor Ġa Ġconcept Ġ( the ĠâĢ ľ background âĢ Ŀ Ġconcept Ġin Ġthe Ġterms Ġof ĠAd cock Ġand ĠColl ier ), Ġand Ġdefinitions Ġmight Ġbe Ġcontested Ġover Ġtime . ĠThis Ġmay Ġbe Ġuncomfortable Ġfor Ġcomputer Ġscientists , Ġwhose Ġprimary Ġmeasure Ġof Ġsuccess Ġis Ġoften Ġbased Ġon Ġcomparing Ġa Ġmodel 's Ġoutput Ġagainst ĠâĢ ľ ground Ġtruth âĢ Ŀ Ġor Ġa ĠâĢ ľ gold Ġstandard âĢ Ŀ , Ġe . g ., Ġby Ġcomparing Ġa Ġsentiment Ġclass ifier 's Ġoutput Ġagainst Ġmanual Ġannotations . ĠHowever , Ġthe Ġnotion Ġof Ġground Ġtruth Ġis Ġuncommon Ġin Ġthe Ġhumanities Ġand Ġthe Ġsocial Ġsciences Ġand Ġit Ġis Ġoften Ġtaken Ġtoo Ġfar Ġin Ġmachine Ġlearning . ĠB IB REF 14 Ġnotes Ġthat Ġin Ġliterary Ġcriticism Ġand Ġthe Ġdigital Ġhumanities Ġmore Ġbroadly ĠâĢ ľ interpret ation , Ġambiguity , Ġand Ġargument ation Ġare Ġprized Ġfar Ġabove Ġground Ġtruth Ġand Ġdefinitive Ġconclusions ". ĠB IB REF 15 Ġdraw Ġattention Ġto Ġthe Ġdifferent Ġattitudes Ġof Ġliterary Ġscholars Ġand Ġcomputational Ġlingu ists Ġtowards Ġambiguity , Ġstating Ġthat ĠâĢ ľ In ĠComput ational ĠL ingu istics Ġ[ .. ] Ġambiguity Ġis Ġalmost Ġuniformly Ġtreated Ġas Ġa Ġproblem Ġto Ġbe Ġsolved ; Ġthe Ġfocus Ġis Ġon Ġdis amb ig uation , Ġwith Ġthe Ġassumption Ġthat Ġone Ġtrue , Ġcorrect Ġinterpretation Ġexists ." ĠThe Ġlatter Ġis Ġprobably Ġtrue Ġfor Ġtasks Ġsuch Ġas Ġspam Ġfiltering , Ġbut Ġin Ġthe Ġsocial Ġsciences Ġand Ġthe Ġhumanities Ġmany Ġrelevant Ġconcepts Ġare Ġfundamentally Ġunob serv able , Ġsuch Ġas Ġlatent Ġtraits Ġof Ġpolitical Ġactors ĠB IB REF 16 Ġor Ġcultural Ġfit Ġin Ġorganizations ĠB IB REF 17 Ġ, Ġleading Ġto Ġvalidation Ġchallenges . ĠMoreover , Ġwhen Ġthe Ġground Ġtruth Ġcomes Ġfrom Ġpeople , Ġit Ġmay Ġbe Ġinfluenced Ġby Ġideological Ġpri ors , Ġprim ing , Ġsimple Ġdifferences Ġof Ġopinion Ġor Ġperspective , Ġand Ġmany Ġother Ġfactors ĠB IB REF 18 Ġ. ĠWe Ġreturn Ġto Ġthis Ġissue Ġin Ġour Ġdiscussions Ġon Ġvalidation Ġand Ġanalysis . </s> Oper ational ization </s> In Ġthis Ġphase Ġwe Ġdevelop Ġmeasures Ġ( or , ĠâĢ ľ oper ational izations âĢ Ŀ , Ġor ĠâĢ ľ ind icators âĢ Ŀ ) Ġfor Ġthe Ġconcepts Ġof Ġinterest , Ġa Ġprocess Ġcalled ĠâĢ ľ oper ational ization âĢ Ŀ . ĠRegardless Ġof Ġwhether Ġwe Ġare Ġworking Ġwith Ġcomputers , Ġthe Ġoutput Ġproduced Ġcoincides Ġwith ĠAd cock Ġand ĠColl ier 's ĠâĢ ľ sc ores âĢ Ŀ âĢĶ the Ġconcrete Ġtranslation Ġand Ġoutput Ġof Ġthe Ġsystem at ized Ġconcept Ġinto Ġnumbers Ġor Ġlabels ĠB IB REF 13 Ġ. ĠCho ices Ġmade Ġduring Ġthis Ġphase Ġare Ġalways Ġtied Ġto Ġthe Ġquestion ĠâĢ ľ Are Ġwe Ġmeasuring Ġwhat Ġwe Ġintend Ġto Ġmeasure ? âĢ Ŀ ĠDoes Ġour Ġoperational ization Ġmatch Ġour Ġconceptual Ġdefinition ? ĠTo Ġensure Ġvalidity Ġwe Ġmust Ġrecognize Ġgaps Ġbetween Ġwhat Ġis Ġimportant Ġand Ġwhat Ġis Ġeasy Ġto Ġmeasure . ĠWe Ġfirst Ġdiscuss Ġmodeling Ġconsiderations . ĠNext , Ġwe Ġdescribe Ġseveral Ġfrequently Ġused Ġcomputational Ġapproaches Ġand Ġtheir Ġlimitations Ġand Ġstrengths . </s> Mod eling Ġconsiderations </s> The Ġvariables Ġ( both Ġpredict ors Ġand Ġoutcomes ) Ġare Ġrarely Ġsimply Ġbinary Ġor Ġcateg orical . ĠFor Ġexample , Ġa Ġstudy Ġon Ġlanguage Ġuse Ġand Ġage Ġcould Ġfocus Ġon Ġchronological Ġage Ġ( instead Ġof , Ġe . g ., Ġsocial Ġage ĠB IB REF 19 Ġ). ĠHowever , Ġeven Ġthen , Ġage Ġcan Ġbe Ġmodeled Ġin Ġdifferent Ġways . ĠDisc ret ization Ġcan Ġmake Ġthe Ġmodeling Ġeasier Ġand Ġvarious ĠN LP Ġstudies Ġhave Ġmodeled Ġage Ġas Ġa Ġcateg orical Ġvariable ĠB IB REF 20 Ġ. ĠBut Ġany Ġdisc ret ization Ġraises Ġquestions : ĠHow Ġmany Ġcategories ? ĠWhere Ġto Ġplace Ġthe Ġboundaries ? ĠFine Ġdistinctions Ġmight Ġnot Ġalways Ġbe Ġmeaningful Ġfor Ġthe Ġanalysis Ġwe Ġare Ġinterested Ġin , Ġbut Ġcategories Ġthat Ġare Ġtoo Ġbroad Ġcan Ġthreaten Ġvalidity . ĠOther Ġinteresting Ġvariables Ġinclude Ġtime , Ġspace , Ġand Ġeven Ġthe Ġsocial Ġnetwork Ġposition Ġof Ġthe Ġauthor . ĠIt Ġis Ġoften Ġpreferable Ġto Ġkeep Ġthe Ġvariable Ġin Ġits Ġmost Ġprecise Ġform . ĠFor Ġexample , ĠB IB REF 21 Ġperform Ġexploration Ġin Ġthe Ġcontext Ġof Ġhypothesis Ġtesting Ġby Ġusing Ġlatitude Ġand Ġlong itude Ġcoordinates ĠâĢĶ Ġthe Ġoriginal Ġmetadata Ġattached Ġto Ġge ot agged Ġsocial Ġmedia Ġsuch Ġas Ġtweets ĠâĢĶ Ġrather Ġthan Ġaggreg ating Ġinto Ġadministrative Ġunits Ġsuch Ġas Ġcounties Ġor Ġcities . ĠThis Ġis Ġnecessary Ġwhen Ġsuch Ġadministrative Ġunits Ġare Ġunlikely Ġto Ġbe Ġrelated Ġto Ġthe Ġtarget Ġconcept , Ġas Ġis Ġthe Ġcase Ġin Ġtheir Ġanalysis Ġof Ġdialect Ġdifferences . ĠF ocusing Ġon Ġprecise Ġgeographical Ġcoordinates Ġalso Ġmakes Ġit Ġpossible Ġto Ġrecognize Ġfine - gr ained Ġeffects , Ġsuch Ġas Ġlanguage Ġvariation Ġacross Ġthe Ġgeography Ġof Ġa Ġcity . </s> Using Ġa Ġparticular Ġclassification Ġscheme Ġmeans Ġdeciding Ġwhich Ġvariations Ġare Ġvisible , Ġand Ġwhich Ġones Ġare Ġhidden ĠB IB REF 22 Ġ. ĠWe Ġare Ġlooking Ġfor Ġa Ġcategor ization Ġscheme Ġfor Ġwhich Ġit Ġis Ġfeasible Ġto Ġcollect Ġa Ġlarge Ġenough Ġlabeled Ġdocument Ġcollection Ġ( e . g ., Ġto Ġtrain Ġsupervised Ġmodels ), Ġbut Ġwhich Ġis Ġalso Ġfine - gr ained Ġenough Ġfor Ġour Ġpurposes . ĠClassification Ġschemes Ġrarely Ġexhibit Ġthe Ġideal Ġproperties , Ġi . e ., Ġthat Ġthey Ġare Ġconsistent , Ġtheir Ġcategories Ġare Ġmutually Ġexclusive , Ġand Ġthat Ġthe Ġsystem Ġis Ġcomplete ĠB IB REF 22 Ġ. ĠBorder line Ġcases Ġare Ġchallenging , Ġespecially Ġwith Ġsocial Ġand Ġcultural Ġconcepts , Ġwhere Ġthe Ġboundaries Ġare Ġoften Ġnot Ġclear - cut . ĠThe Ġchoice Ġof Ġscheme Ġcan Ġalso Ġhave Ġethical Ġimplications ĠB IB REF 22 Ġ. ĠFor Ġexample , Ġgender Ġis Ġusually Ġrepresented Ġas Ġa Ġbinary Ġvariable Ġin ĠN LP Ġand Ġcomputational Ġmodels Ġtend Ġto Ġlearn Ġgender - st ere otyp ical Ġpatterns . ĠThe Ġoperational ization Ġof Ġgender Ġin ĠN LP Ġhas Ġbeen Ġchallenged Ġonly Ġrecently ĠB IB REF 23 Ġ, ĠB IB REF 24 Ġ, ĠB IB REF 25 Ġ. </s> Super vised Ġand Ġun super vised Ġlearning Ġare Ġthe Ġmost Ġcommon Ġapproaches Ġto Ġlearning Ġfrom Ġdata . ĠWith Ġsupervised Ġlearning , Ġa Ġmodel Ġlearns Ġfrom Ġlabeled Ġdata Ġ( e . g ., Ġsocial Ġmedia Ġmessages Ġlabeled Ġby Ġsentiment ) Ġto Ġinfer Ġ( or Ġpredict ) Ġthese Ġlabels Ġfrom Ġunl abel ed Ġtexts . ĠIn Ġcontrast , Ġun super vised Ġlearning Ġuses Ġunl abel ed Ġdata . ĠSuper vised Ġapproaches Ġare Ġespecially Ġsuitable Ġwhen Ġwe Ġhave Ġa Ġclear Ġdefinition Ġof Ġthe Ġconcept Ġof Ġinterest Ġand Ġwhen Ġlabels Ġare Ġavailable Ġ( either Ġannot ated Ġor Ġnative Ġto Ġthe Ġdata ). ĠUn super vised Ġapproaches , Ġsuch Ġas Ġtopic Ġmodels , Ġare Ġespecially Ġuseful Ġfor Ġexploration . ĠIn Ġthis Ġsetting , Ġconceptual ization Ġand Ġoperational ization Ġmay Ġoccur Ġsimultaneously , Ġwith Ġtheory Ġemerging Ġfrom Ġthe Ġdata ĠB IB REF 26 Ġ. ĠUn super vised Ġapproaches Ġare Ġalso Ġused Ġwhen Ġthere Ġis Ġa Ġclear Ġway Ġof Ġmeasuring Ġa Ġconcept , Ġoften Ġbased Ġon Ġstrong Ġassumptions . ĠFor Ġexample , ĠB IB REF 3 Ġmeasure ĠâĢ ľ sur prise âĢ Ŀ Ġin Ġan Ġanalysis Ġof ĠDarwin 's Ġreading Ġdecisions Ġbased Ġon Ġthe Ġdivergence Ġbetween Ġtwo Ġprobability Ġdistributions . </s> From Ġan Ġanalysis Ġperspective , Ġthe Ġunit Ġof Ġtext Ġthat Ġwe Ġare Ġlabeling Ġ( or Ġannot ating , Ġor Ġcoding ), Ġeither Ġautomatic Ġor Ġmanual , Ġcan Ġsometimes Ġbe Ġdifferent Ġthan Ġone 's Ġfinal Ġunit Ġof Ġanalysis . ĠFor Ġexample , Ġif Ġin Ġa Ġstudy Ġon Ġmedia Ġframes Ġin Ġnews Ġstories , Ġthe Ġtheoretical Ġframework Ġand Ġresearch Ġquestion Ġpoint Ġtoward Ġframes Ġat Ġthe Ġstory Ġlevel Ġ( e . g ., Ġwhat Ġis Ġthe Ġoverall Ġcausal Ġanalysis Ġof Ġthe Ġnews Ġarticle ?), Ġthe Ġstory Ġmust Ġbe Ġthe Ġunit Ġof Ġanalysis . ĠYet Ġit Ġis Ġoften Ġdifficult Ġto Ġvalid ly Ġand Ġreliably Ġcode Ġa Ġsingle Ġframe Ġat Ġthe Ġstory Ġlevel . ĠMultiple Ġperspectives Ġare Ġlikely Ġto Ġsit Ġside - by - side Ġin Ġa Ġstory . ĠThus , Ġan Ġarticle Ġon Ġincome Ġinequality Ġmight Ġpoint Ġto Ġmultiple Ġcauses , Ġsuch Ġas Ġglobalization , Ġeducation , Ġand Ġtax Ġpolicies . ĠC oding Ġat Ġthe Ġsentence Ġlevel Ġwould Ġdetect Ġeach Ġof Ġthese Ġcausal Ġexplanations Ġindividually , Ġbut Ġthis Ġinformation Ġwould Ġneed Ġto Ġbe Ġsomehow Ġaggreg ated Ġto Ġdetermine Ġthe Ġoverall Ġstory - level Ġframe . ĠSometimes Ġscholars Ġsolve Ġthis Ġproblem Ġby Ġonly Ġexamining Ġheadlines Ġand Ġlead Ġparagraphs , Ġarguing Ġthat Ġbased Ġon Ġjournalistic Ġconvention , Ġthe Ġmost Ġimportant Ġinformation Ġcan Ġbe Ġfound Ġat Ġthe Ġbeginning Ġof Ġa Ġstory . ĠHowever , Ġthis Ġleads Ġto Ġa Ġreturn Ġto Ġa Ġshorter , Ġless Ġnuanced Ġanalysis . </s> From Ġa Ġcomputational Ġperspective , Ġthe Ġunit Ġof Ġtext Ġcan Ġalso Ġmake Ġa Ġhuge Ġdifference , Ġespecially Ġwhen Ġwe Ġare Ġusing Ġbag - of - words Ġmodels , Ġwhere Ġword Ġorder Ġwithin Ġa Ġunit Ġdoes Ġnot Ġmatter . ĠSmall Ġsegments , Ġlike Ġtweets , Ġsometimes Ġdo Ġnot Ġhave Ġenough Ġinformation Ġto Ġmake Ġtheir Ġsemantic Ġcontext Ġclear . ĠIn Ġcontrast , Ġlarger Ġsegments , Ġlike Ġnovels , Ġhave Ġtoo Ġmuch Ġvariation , Ġmaking Ġit Ġdifficult Ġto Ġtrain Ġfocused Ġmodels . ĠFinding Ġa Ġgood Ġsegment ation Ġsometimes Ġmeans Ġcombining Ġshort Ġdocuments Ġand Ġsub d ivid ing Ġlong Ġdocuments . ĠThe Ġword ĠâĢ ľ document " Ġcan Ġtherefore Ġbe Ġmisleading . ĠBut Ġit Ġis Ġso Ġingrained Ġin Ġthe Ġcommon ĠN LP Ġlex icon Ġthat Ġwe Ġuse Ġit Ġanyway Ġin Ġthis Ġarticle . </s> For Ġinsight - driven Ġtext Ġanalysis , Ġit Ġis Ġoften Ġcritical Ġthat Ġhigh - level Ġpatterns Ġcan Ġbe Ġcommunicated . ĠFurthermore , Ġinterpret able Ġmodels Ġmake Ġit Ġeasier Ġto Ġfind Ġspurious Ġfeatures , Ġto Ġdo Ġerror Ġanalysis , Ġand Ġto Ġsupport Ġinterpretation Ġof Ġresults . ĠSome Ġapproaches Ġare Ġeffective Ġfor Ġprediction , Ġbut Ġharder Ġto Ġinterpret . ĠThe Ġvalue Ġwe Ġplace Ġon Ġinterpret ability Ġcan Ġtherefore Ġinfluence Ġthe Ġapproach Ġwe Ġchoose . ĠThere Ġis Ġan Ġincreasing Ġinterest Ġin Ġdeveloping Ġinterpret able Ġor Ġtransparent Ġmodels Ġin Ġthe ĠN LP Ġand Ġmachine Ġlearning Ġcommunities . </s> An notation </s> Many Ġstudies Ġinvolve Ġhuman Ġcod ers . ĠSometimes Ġthe Ġgoal Ġis Ġto Ġfully Ġcode Ġthe Ġdata , Ġbut Ġin Ġa Ġcomputational Ġanalysis Ġwe Ġoften Ġuse Ġthe Ġlabels Ġ( or Ġannotations ) Ġto Ġtrain Ġmachine Ġlearning Ġmodels Ġto Ġautomatically Ġrecognize Ġthem , Ġand Ġto Ġidentify Ġlanguage Ġpatterns Ġthat Ġare Ġassociated Ġwith Ġthese Ġlabels . ĠFor Ġexample , Ġfor Ġa Ġproject Ġanalyzing Ġrumors Ġonline ĠB IB REF 27 Ġ, Ġconversation Ġthreads Ġwere Ġannot ated Ġalong Ġdifferent Ġdimensions , Ġincluding Ġrumor Ġversus Ġnon - rum or Ġand Ġstance Ġtowards Ġa Ġrumor . </s> The Ġcollection Ġof Ġannotation Ġchoices Ġmake Ġup Ġan Ġannotation Ġscheme Ġ( or ĠâĢ ľ code book âĢ Ŀ ). ĠEx isting Ġschemes Ġand Ġannotations Ġcan Ġbe Ġuseful Ġas Ġstarting Ġpoints . ĠUsually Ġsettling Ġon Ġan Ġannotation Ġscheme Ġrequires Ġseveral Ġiterations , Ġin Ġwhich Ġthe Ġguidelines Ġare Ġupdated Ġand Ġannotation Ġexamples Ġare Ġadded . ĠFor Ġexample , Ġa Ġpolitical Ġscientist Ġcould Ġuse Ġa Ġmixed Ġdeduct ive - ind uctive Ġstrategy Ġfor Ġdeveloping Ġa Ġcode book . ĠShe Ġstarts Ġby Ġlaying Ġout Ġa Ġset Ġof Ġtheory - driven Ġdeduct ive Ġcoding Ġrules , Ġwhich Ġmeans Ġthat Ġthe Ġbroad Ġprinciples Ġof Ġthe Ġcoding Ġrules Ġare Ġlaid Ġout Ġwithout Ġexamining Ġexamples Ġfirst . ĠThese Ġare Ġthen Ġtested Ġ( and Ġpossibly Ġadjusted ) Ġbased Ġon Ġa Ġsample Ġof Ġthe Ġdata . ĠIn Ġline Ġwith ĠAd cock Ġand ĠColl ier 's Ġnotion Ġof ĠâĢ ľ content Ġvalidity âĢ Ŀ ĠB IB REF 13 Ġ, Ġthe Ġgoal Ġis Ġto Ġassess Ġwhether Ġthe Ġcode book Ġadequately Ġcaptures Ġthe Ġsystem at ized Ġconcept . ĠBy Ġlooking Ġat Ġthe Ġdata Ġthemselves , Ġshe Ġgains Ġa Ġbetter Ġsense Ġof Ġwhether Ġsome Ġthings Ġhave Ġbeen Ġleft Ġout Ġof Ġthe Ġcoding Ġrules Ġand Ġwhether Ġanything Ġis Ġsuperflu ous , Ġmisleading , Ġor Ġconfusing . ĠAdjust ments Ġare Ġmade Ġand Ġthe Ġprocess Ġis Ġrepeated , Ġoften Ġwith Ġanother Ġresearcher Ġinvolved . </s> The Ġfinal Ġannotations Ġcan Ġbe Ġcollected Ġusing Ġa Ġcrowds ourcing Ġplatform , Ġa Ġsmaller Ġnumber Ġof Ġhighly - trained Ġannot ators , Ġor Ġa Ġgroup Ġof Ġexperts . ĠWhich Ġtype Ġof Ġannot ator Ġto Ġuse Ġshould Ġbe Ġinformed Ġby Ġthe Ġcomplexity Ġand Ġspecificity Ġof Ġthe Ġconcept . ĠFor Ġmore Ġcomplex Ġconcepts , Ġhighly - trained Ġor Ġexpert Ġannot ators Ġtend Ġto Ġproduce Ġmore Ġreliable Ġresults . ĠHowever , Ġcomplex Ġconcepts Ġcan Ġsometimes Ġbe Ġbroken Ġdown Ġinto Ġmicro - t asks Ġthat Ġcan Ġbe Ġperformed Ġindependently Ġin Ġparallel Ġby Ġcrowds ourced Ġannot ators . ĠConcepts Ġfrom Ġhighly Ġspecialized Ġdomains Ġmay Ġrequire Ġexpert Ġannot ators . ĠIn Ġall Ġcases , Ġhowever , Ġsome Ġtraining Ġwill Ġbe Ġrequired , Ġand Ġthe Ġtraining Ġphase Ġshould Ġinvolve Ġcontinual Ġchecks Ġof Ġinter - annot ator Ġagreement Ġ( i . e . Ġinter c oder Ġreliability ) Ġor Ġchecks Ġagainst Ġa Ġgold Ġstandard Ġ( e . g . Ġquizz es Ġin Ġcrowds ourcing Ġplatforms ). </s> We Ġalso Ġneed Ġto Ġdecide Ġhow Ġinter - annot ator Ġagreement Ġwill Ġbe Ġmeasured Ġand Ġwhat Ġan Ġacceptable Ġlevel Ġof Ġagreement Ġwould Ġbe . ĠK ripp endor ff 's Ġalpha Ġis Ġfrequently Ġused Ġin Ġthe Ġsocial Ġsciences , Ġbut Ġthe Ġright Ġmeasure Ġdepends Ġon Ġthe Ġtype Ġof Ġdata Ġand Ġtask . ĠFor Ġmanual Ġcoding , Ġwe Ġcan Ġcontinually Ġcheck Ġinter - annot ator Ġagreement Ġand Ġbegin Ġintroducing Ġchecks Ġof Ġintra - annot ator Ġagreement , Ġtoo . ĠFor Ġmost Ġcommunication Ġscholars Ġusing Ġonly Ġmanual Ġcontent Ġanalysis , Ġan Ġacceptable Ġrate Ġof Ġagreement Ġis Ġachieved Ġwhen ĠK ripp endor f 's Ġalpha Ġreaches Ġ0 . 80 Ġor Ġabove . ĠWhen Ġhuman - coded Ġdata Ġare Ġused Ġto Ġvalidate Ġmachine Ġlearning Ġalgorithms , Ġthe Ġreliability Ġof Ġthe Ġhuman - coded Ġdata Ġis Ġeven Ġmore Ġimportant . ĠDis ag reement Ġbetween Ġannot ators Ġcan Ġsignal Ġweaknesses Ġof Ġthe Ġannotation Ġscheme , Ġor Ġhighlight Ġthe Ġinherent Ġambiguity Ġin Ġwhat Ġwe Ġare Ġtrying Ġto Ġmeasure . ĠDis ag reement Ġitself Ġcan Ġbe Ġmeaningful Ġand Ġcan Ġbe Ġintegrated Ġin Ġsubsequent Ġanalyses ĠB IB REF 28 Ġ, ĠB IB REF 29 Ġ. </s> Data Ġpre - processing </s> Prep aring Ġthe Ġdata Ġcan Ġbe Ġa Ġcomplex Ġand Ġtime - consuming Ġprocess , Ġoften Ġinvolving Ġworking Ġwith Ġpartially Ġor Ġwholly Ġun struct ured Ġdata . ĠThe Ġpre - processing Ġsteps Ġhave Ġa Ġbig Ġimpact Ġon Ġthe Ġoperational izations , Ġsubsequent Ġanalyses Ġand Ġreprodu c ibility Ġefforts ĠB IB REF 30 Ġ, Ġand Ġthey Ġare Ġusually Ġtightly Ġlinked Ġto Ġwhat Ġwe Ġintend Ġto Ġmeasure . ĠUnfortunately , Ġthese Ġsteps Ġtend Ġto Ġbe Ġunder reported , Ġbut Ġdocumenting Ġthe Ġpre - processing Ġchoices Ġmade Ġis Ġessential Ġand Ġis Ġanalogous Ġto Ġrecording Ġthe Ġdecisions Ġtaken Ġduring Ġthe Ġproduction Ġof Ġa Ġscholarly Ġedition Ġor Ġprotocols Ġin Ġbiomedical Ġresearch . ĠData Ġmay Ġalso Ġvary Ġenormously Ġin Ġquality , Ġdepending Ġon Ġhow Ġit Ġhas Ġbeen Ġgenerated . ĠMany Ġhistorians , Ġfor Ġexample , Ġwork Ġwith Ġtext Ġproduced Ġfrom Ġan Ġanalogue Ġoriginal Ġusing ĠOptical ĠCharacter ĠRecogn ition Ġ( OC R ). ĠOften , Ġthere Ġwill Ġbe Ġlimited Ġinformation Ġavailable Ġregarding Ġthe Ġaccuracy Ġof Ġthe ĠO CR , Ġand Ġthe Ġdegree Ġof Ġaccuracy Ġmay Ġeven Ġvary Ġwithin Ġa Ġsingle Ġcorpus Ġ( e . g . Ġwhere Ġdigit ized Ġtext Ġhas Ġbeen Ġproduced Ġover Ġa Ġperiod Ġof Ġyears , Ġand Ġthe Ġsoftware Ġhas Ġgradually Ġimproved ). ĠThe Ġfirst Ġstep , Ġthen , Ġis Ġto Ġtry Ġto Ġcorrect Ġfor Ġcommon ĠO CR Ġerrors . ĠThese Ġwill Ġvary Ġdepending Ġon Ġthe Ġtype Ġof Ġtext , Ġthe Ġdate Ġat Ġwhich Ġthe Ġ` original ' Ġwas Ġproduced , Ġand Ġthe Ġnature Ġof Ġthe Ġfont Ġand Ġtypes etting . </s> One Ġstep Ġthat Ġalmost Ġeveryone Ġtakes Ġis Ġto Ġtoken ize Ġthe Ġoriginal Ġcharacter Ġsequence Ġinto Ġthe Ġwords Ġand Ġword - like Ġunits . ĠToken ization Ġis Ġa Ġmore Ġsubtle Ġand Ġmore Ġpowerful Ġprocess Ġthan Ġpeople Ġexpect . ĠIt Ġis Ġoften Ġdone Ġusing Ġregular Ġexpressions Ġor Ġscripts Ġthat Ġhave Ġbeen Ġcirculating Ġwithin Ġthe ĠN LP Ġcommunity . ĠToken ization Ġhe ur istics , Ġhowever , Ġcan Ġbe Ġbadly Ġconfused Ġby Ġemot icons , Ġcreative Ġorth ography Ġ( e . g ., ĠU $ A , Ġsh ! t ), Ġand Ġmissing Ġwhites pace . ĠMulti - word Ġterms Ġare Ġalso Ġchallenging . ĠTreat ing Ġthem Ġas Ġa Ġsingle Ġunit Ġcan Ġdramatically Ġalter Ġthe Ġpatterns Ġin Ġtext . ĠMany Ġwords Ġthat Ġare Ġindividually Ġambiguous Ġhave Ġclear , Ġunmist akable Ġmeanings Ġas Ġterms , Ġlike ĠâĢ ľ black Ġhole " Ġor ĠâĢ ľ European ĠUnion ". ĠHowever , Ġdeciding Ġwhat Ġconstitutes Ġa Ġmulti - word Ġterm Ġis Ġa Ġdifficult Ġproblem . ĠIn Ġwriting Ġsystems Ġlike ĠChinese , Ġtoken ization Ġis Ġa Ġresearch Ġproblem Ġin Ġits Ġown Ġright . </s> Beyond Ġtoken ization , Ġcommon Ġsteps Ġinclude Ġlower c asing , Ġremoving Ġpunct uation , Ġstemming Ġ( rem oving Ġsuffix es ), Ġle mm at ization Ġ( con verting Ġinf lections Ġto Ġa Ġbase Ġle mma ), Ġand Ġnormal ization , Ġwhich Ġhas Ġnever Ġbeen Ġclearly Ġdefined , Ġbut Ġoften Ġincludes Ġgrouping Ġabbrevi ations Ġlike ĠâĢ ľ U . S . A ." Ġand ĠâĢ ľ USA ", Ġord inals Ġlike ĠâĢ ľ 1 st " Ġand ĠâĢ ľ first ", Ġand Ġvariant Ġspell ings Ġlike ĠâĢ ľ n oooo oo ". ĠThe Ġmain Ġgoal Ġof Ġthese Ġsteps Ġis Ġto Ġimprove Ġthe Ġratio Ġof Ġtokens Ġ( individual Ġoccurrences ) Ġto Ġtypes Ġ( the Ġdistinct Ġthings Ġin Ġa Ġcorpus ). ĠEach Ġstep Ġrequires Ġmaking Ġadditional Ġassumptions Ġabout Ġwhich Ġdistinctions Ġare Ġrelevant : Ġis ĠâĢ ľ apple âĢ Ŀ Ġdifferent Ġfrom ĠâĢ ľ Apple âĢ Ŀ ? ĠIs ĠâĢ ľ bur nt âĢ Ŀ Ġdifferent Ġfrom ĠâĢ ľ burn ed âĢ Ŀ ? ĠIs ĠâĢ ľ cool " Ġdifferent Ġfrom ĠâĢ ľ co oo ool "? ĠSometimes Ġthese Ġsteps Ġcan Ġactively Ġhide Ġuseful Ġpatterns , Ġlike Ġsocial Ġmeaning ĠB IB REF 32 Ġ. ĠSome Ġof Ġus Ġtherefore Ġtry Ġdo Ġas Ġlittle Ġmodification Ġas Ġpossible . </s> From Ġa Ġmult ilingual Ġperspective , ĠEnglish Ġand ĠChinese Ġhave Ġan Ġunusually Ġsimple Ġinf lection al Ġsystem , Ġand Ġso Ġit Ġis Ġstatistically Ġreasonable Ġto Ġtreat Ġeach Ġinf lection Ġas Ġa Ġunique Ġword Ġtype . ĠRomance Ġlanguages Ġhave Ġconsiderably Ġmore Ġinf lections Ġthan ĠEnglish ; Ġmany Ġindigenous ĠNorth ĠAmerican Ġlanguages Ġhave Ġstill Ġmore . ĠFor Ġthese Ġlanguages , Ġunseen Ġdata Ġis Ġfar Ġmore Ġlikely Ġto Ġinclude Ġpreviously - un seen Ġinf lections , Ġand Ġtherefore , Ġdealing Ġwith Ġinf lections Ġis Ġmore Ġimportant . ĠOn Ġthe Ġother Ġhand , Ġthe Ġresources Ġfor Ġhandling Ġinf lections Ġvary Ġgreatly Ġby Ġlanguage , Ġwith ĠEuropean Ġlanguages Ġdominating Ġthe Ġattention Ġof Ġthe Ġcomputational Ġlingu istics Ġcommunity Ġthus Ġfar . </s> We Ġsometimes Ġalso Ġremove Ġwords Ġthat Ġare Ġnot Ġrelevant Ġto Ġour Ġgoals , Ġfor Ġexample Ġby Ġcalculating Ġvocabulary Ġfrequencies . ĠWe Ġconstruct Ġa ĠâĢ ľ stop list âĢ Ŀ Ġof Ġwords Ġthat Ġwe Ġare Ġnot Ġinterested Ġin . ĠIf Ġwe Ġare Ġlooking Ġfor Ġsemantic Ġthemes Ġwe Ġmight Ġremove Ġfunction Ġwords Ġlike Ġdetermin ers Ġand Ġpre pos itions . ĠIf Ġwe Ġare Ġlooking Ġfor Ġauthor - specific Ġstyles , Ġwe Ġmight Ġremove Ġall Ġwords Ġexcept Ġfunction Ġwords . ĠSome Ġwords Ġare Ġgenerally Ġmeaningful Ġbut Ġtoo Ġfrequent Ġto Ġbe Ġuseful Ġwithin Ġa Ġspecific Ġcollection . ĠWe Ġsometimes Ġalso Ġremove Ġvery Ġinf requent Ġwords . ĠTheir Ġoccurrences Ġare Ġtoo Ġlow Ġfor Ġrobust Ġpatterns Ġand Ġremoving Ġthem Ġhelps Ġreducing Ġthe Ġvocabulary Ġsize . </s> The Ġchoice Ġof Ġprocessing Ġsteps Ġcan Ġbe Ġguided Ġby Ġtheory Ġor Ġknowledge Ġabout Ġthe Ġdomain Ġas Ġwell Ġas Ġexperimental Ġinvestigation . ĠWhen Ġwe Ġhave Ġlabels , Ġpredictive Ġaccuracy Ġof Ġa Ġmodel Ġis Ġa Ġway Ġto Ġassess Ġthe Ġeffect Ġof Ġthe Ġprocessing Ġsteps . ĠIn Ġun super vised Ġsettings , Ġit Ġis Ġmore Ġchallenging Ġto Ġunderstand Ġthe Ġeffects Ġof Ġdifferent Ġsteps . ĠInf erences Ġdrawn Ġfrom Ġun super vised Ġsettings Ġcan Ġbe Ġsensitive Ġto Ġpre - processing Ġchoices ĠB IB REF 33 Ġ. ĠSt em ming Ġhas Ġbeen Ġfound Ġto Ġprovide Ġlittle Ġmeasurable Ġbenefits Ġfor Ġtopic Ġmodeling Ġand Ġcan Ġsometimes Ġeven Ġbe Ġharmful ĠB IB REF 34 Ġ. ĠAll Ġin Ġall , Ġthis Ġagain Ġhighlights Ġthe Ġneed Ġto Ġdocument Ġthese Ġsteps . </s> Finally , Ġwe Ġcan Ġalso Ġmark Ġup Ġthe Ġdata , Ġe . g ., Ġby Ġidentifying Ġentities Ġ( people , Ġplaces , Ġorganizations , Ġetc .) Ġor Ġparts Ġof Ġspeech . ĠAlthough Ġmany ĠN LP Ġtools Ġare Ġavailable Ġfor Ġsuch Ġtasks , Ġthey Ġare Ġoften Ġchallenged Ġby Ġlinguistic Ġvariation , Ġsuch Ġas Ġorth ographic Ġvariation Ġin Ġhistorical Ġtexts ĠB IB REF 35 Ġand Ġsocial Ġmedia ĠB IB REF 32 Ġ. ĠMoreover , Ġthe Ġperformance Ġof ĠN LP Ġtools Ġoften Ġdrops Ġwhen Ġapplying Ġthem Ġoutside Ġthe Ġtraining Ġdomain , Ġsuch Ġas Ġapplying Ġtools Ġdeveloped Ġon Ġnews wire Ġtexts Ġto Ġtexts Ġwritten Ġby Ġyounger Ġauthors ĠB IB REF 36 Ġ. ĠProblems Ġ( e . g ., Ġdis amb ig uation Ġin Ġnamed Ġentity Ġrecognition ) Ġare Ġsometimes Ġresolved Ġusing Ġconsiderable Ġmanual Ġintervention . ĠThis Ġcombination Ġof Ġthe Ġautomated Ġand Ġthe Ġmanual , Ġhowever , Ġbecomes Ġmore Ġdifficult Ġas Ġthe Ġscale Ġof Ġthe Ġdata Ġincreases , Ġand Ġthe Ġ` certain ty ' Ġbrought Ġby Ġthe Ġlatter Ġmay Ġhave Ġto Ġbe Ġabandoned . </s> D ictionary - based Ġapproaches </s> D iction aries Ġare Ġfrequently Ġused Ġto Ġcode Ġtexts Ġin Ġcontent Ġanalyses ĠB IB REF 37 Ġ. ĠD iction aries Ġconsist Ġof Ġone Ġor Ġmore Ġcategories Ġ( i . e . Ġword Ġlists ). ĠSometimes Ġthe Ġoutput Ġis Ġsimply Ġthe Ġnumber Ġof Ġcategory Ġoccurrences Ġ( e . g ., Ġpositive Ġsentiment ), Ġthus Ġweight ing Ġwords Ġwithin Ġa Ġcategory Ġequally . ĠIn Ġsome Ġother Ġcases , Ġwords Ġare Ġassigned Ġcontinuous Ġscores . ĠThe Ġhigh Ġtransparency Ġof Ġdiction aries Ġmakes Ġthem Ġsometimes Ġmore Ġsuitable Ġthan Ġsupervised Ġmachine Ġlearning Ġmodels . ĠHowever , Ġdiction aries Ġshould Ġonly Ġbe Ġused Ġif Ġthe Ġscores Ġassigned Ġto Ġwords Ġmatch Ġhow Ġthe Ġwords Ġare Ġused Ġin Ġthe Ġdata Ġ( see ĠB IB REF 38 Ġfor Ġa Ġdetailed Ġdiscussion Ġon Ġlimitations ). ĠThere Ġare Ġmany Ġoff - the - she lf Ġdiction aries Ġavailable Ġ( e . g ., ĠLI WC ĠB IB REF 39 Ġ). ĠThese Ġare Ġoften Ġwell - valid ated , Ġbut Ġapplying Ġthem Ġon Ġa Ġnew Ġdomain Ġmay Ġnot Ġbe Ġappropriate Ġwithout Ġadditional Ġvalidation . ĠCorpus - Ġor Ġdomain - specific Ġdiction aries Ġcan Ġovercome Ġlimitations Ġof Ġgeneral - purpose Ġdiction aries . </s> The Ġdiction aries Ġare Ġoften Ġmanually Ġcompiled , Ġbut Ġincreasingly Ġthey Ġare Ġconstructed Ġsemi - aut om atically Ġ( e . g ., ĠB IB REF 40 Ġ). ĠWhen Ġwe Ġsemi - aut om atically Ġcreate Ġa Ġword Ġlist , Ġwe Ġuse Ġautomation Ġto Ġidentify Ġan Ġinitial Ġword Ġlist , Ġand Ġhuman Ġinsight Ġto Ġfilter Ġit . ĠBy Ġautomatically Ġgenerating Ġthe Ġinitial Ġwords Ġlists , Ġwords Ġcan Ġbe Ġidentified Ġthat Ġhuman Ġannot ators Ġmight Ġhave Ġdifficulty Ġintu iting . ĠBy Ġmanually Ġfiltering Ġthe Ġlists , Ġwe Ġuse Ġour Ġtheoretical Ġunderstanding Ġof Ġthe Ġtarget Ġconcept Ġto Ġremove Ġspurious Ġfeatures . </s> In Ġthe Ġintroduction Ġstudy , ĠS AGE ĠB IB REF 41 Ġwas Ġused Ġto Ġobtain Ġa Ġlist Ġof Ġwords Ġthat Ġdistinguished Ġthe Ġtext Ġin Ġthe Ġtreatment Ġgroup Ġ( sub reddits Ġthat Ġwere Ġclosed Ġby ĠReddit ) Ġfrom Ġtext Ġin Ġthe Ġcontrol Ġgroup Ġ( similar Ġsubreddits Ġthat Ġwere Ġnot Ġclosed ). ĠThe Ġresearchers Ġthen Ġreturned Ġto Ġthe Ġhate Ġspeech Ġdefinition Ġprovided Ġby Ġthe ĠEuropean ĠCourt Ġof ĠHuman ĠRights , Ġand Ġmanually Ġfiltered Ġthe Ġtop ĠS AGE Ġwords Ġbased Ġon Ġthis Ġdefinition . ĠNot Ġall Ġidentified Ġwords Ġfitted Ġthe Ġdefinition . ĠThe Ġothers Ġincluded : Ġthe Ġnames Ġof Ġthe Ġsubreddits Ġthemselves , Ġnames Ġof Ġrelated Ġsubreddits , Ġcommunity - specific Ġjargon Ġthat Ġwas Ġnot Ġdirectly Ġrelated Ġto Ġhate Ġspeech , Ġand Ġterms Ġsuch Ġas ĠIQ Ġand Ġwelfare , Ġwhich Ġwere Ġfrequently Ġused Ġin Ġdisc ourses Ġof Ġhate Ġspeech , Ġbut Ġhad Ġsignificant Ġother Ġuses . ĠThe Ġword Ġlists Ġprovided Ġthe Ġmeasurement Ġinstrument Ġfor Ġtheir Ġmain Ġresult , Ġwhich Ġis Ġthat Ġthe Ġuse Ġof Ġhate Ġspeech Ġthroughout ĠReddit Ġdeclined Ġafter Ġthe Ġtwo Ġtreatment Ġsubreddits Ġwere Ġclosed . </s> Super vised Ġmodels </s> Super vised Ġlearning Ġis Ġfrequently Ġused Ġto Ġscale Ġup Ġanalyses . ĠFor Ġexample , ĠB IB REF 42 Ġwanted Ġto Ġanalyze Ġthe Ġmotivations Ġof ĠM ovember Ġcampaign Ġparticipants . ĠBy Ġdeveloping Ġa Ġclass ifier Ġbased Ġon Ġa Ġsmall Ġset Ġof Ġannotations , Ġthey Ġwere Ġable Ġto Ġexpand Ġthe Ġanalysis Ġto Ġover Ġ90 k Ġparticipants . </s> The Ġchoice Ġof Ġsupervised Ġlearning Ġmodel Ġis Ġoften Ġguided Ġby Ġthe Ġtask Ġdefinition Ġand Ġthe Ġlabel Ġtypes . ĠFor Ġexample , Ġto Ġidentify Ġstance Ġtowards Ġrumors Ġbased Ġon Ġsequential Ġannotations , Ġan Ġalgorithm Ġfor Ġlearning Ġfrom Ġsequential ĠB IB REF 43 Ġor Ġtime Ġseries Ġdata ĠB IB REF 44 Ġcould Ġbe Ġused . ĠThe Ġfeatures Ġ( sometimes Ġcalled Ġvariables Ġor Ġpredict ors ) Ġare Ġused Ġby Ġthe Ġmodel Ġto Ġmake Ġthe Ġpredictions . ĠThey Ġmay Ġvary Ġfrom Ġcontent - based Ġfeatures Ġsuch Ġas Ġsingle Ġwords , Ġsequences Ġof Ġwords , Ġor Ġinformation Ġabout Ġtheir Ġsynt actic Ġstructure , Ġto Ġmeta - information Ġsuch Ġas Ġuser Ġor Ġnetwork Ġinformation . ĠDec iding Ġon Ġthe Ġfeatures Ġrequires Ġexperimentation Ġand Ġexpert Ġinsight Ġand Ġis Ġoften Ġcalled Ġfeature Ġengineering . ĠFor Ġinsight - driven Ġanalysis , Ġwe Ġare Ġoften Ġinterested Ġin Ġwhy Ġa Ġprediction Ġhas Ġbeen Ġmade Ġand Ġfeatures Ġthat Ġcan Ġbe Ġinterpreted Ġby Ġhumans Ġmay Ġbe Ġpreferred . ĠRecent Ġneural Ġnetwork Ġapproaches Ġoften Ġuse Ġsimple Ġfeatures Ġas Ġinput Ġ( such Ġas Ġword Ġembed d ings Ġor Ġcharacter Ġsequences ), Ġwhich Ġrequires Ġless Ġfeature Ġengineering Ġbut Ġmake Ġinterpretation Ġmore Ġdifficult . </s> Super vised Ġmodels Ġare Ġpowerful , Ġbut Ġthey Ġcan Ġlatch Ġon Ġto Ġspurious Ġfeatures Ġof Ġthe Ġdataset . ĠThis Ġis Ġparticularly Ġtrue Ġfor Ġdatasets Ġthat Ġare Ġnot Ġwell - balanced , Ġand Ġfor Ġannotations Ġthat Ġare Ġnoisy . ĠIn Ġour Ġintroductory Ġexample Ġon Ġhate Ġspeech Ġin ĠReddit ĠB IB REF 0 Ġ, Ġthe Ġannotations Ġare Ġautomatically Ġderived Ġfrom Ġthe Ġforum Ġin Ġwhich Ġeach Ġpost Ġappears , Ġand Ġindeed , Ġmany Ġof Ġthe Ġposts Ġin Ġthe Ġforums Ġ( sub reddits ) Ġthat Ġwere Ġbanned Ġby ĠReddit Ġwould Ġbe Ġperceived Ġby Ġmany Ġas Ġhate Ġspeech . ĠBut Ġeven Ġin Ġbanned Ġsubreddits , Ġnot Ġall Ġof Ġthe Ġcontent Ġis Ġhate Ġspeech Ġ( e . g ., Ġsome Ġof Ġthe Ġtop Ġfeatures Ġwere Ġself - ref erential Ġlike Ġthe Ġname Ġof Ġthe Ġsubreddit ) Ġbut Ġa Ġclass ifier Ġwould Ġlearn Ġa Ġhigh Ġweight Ġfor Ġthese Ġfeatures . </s> Even Ġwhen Ġexpert Ġannotations Ġare Ġavailable Ġon Ġthe Ġlevel Ġof Ġindividual Ġposts , Ġspurious Ġfeatures Ġmay Ġremain . ĠB IB REF 45 Ġproduced Ġexpert Ġannotations Ġof Ġhate Ġspeech Ġon ĠTwitter . ĠThey Ġfound Ġthat Ġone Ġof Ġthe Ġstrongest Ġfeatures Ġfor Ġsexism Ġis Ġthe Ġname Ġof Ġan ĠAustralian ĠTV Ġshow , Ġbecause Ġpeople Ġlike Ġto Ġpost Ġsexist Ġcomments Ġabout Ġthe Ġcontestants . ĠIf Ġwe Ġare Ġtrying Ġto Ġmake Ġclaims Ġabout Ġwhat Ġinhibits Ġor Ġencourages Ġhate Ġspeech , Ġwe Ġwould Ġnot Ġwant Ġthose Ġclaims Ġto Ġbe Ġtied Ġto Ġthe ĠTV Ġshow 's Ġpopularity . ĠSuch Ġproblems Ġare Ġinevitable Ġwhen Ġdatasets Ġare Ġnot Ġwell - balanced Ġover Ġtime , Ġacross Ġgenres , Ġtopics , Ġetc . ĠEspecially Ġwith Ġsocial Ġmedia Ġdata , Ġwe Ġlack Ġa Ġclear Ġand Ġobjective Ġdefinition Ġof Ġ` balance ' Ġat Ġthis Ġtime . </s> The Ġrisk Ġof Ġsupervised Ġmodels Ġl atching Ġon Ġto Ġspurious Ġfeatures Ġreinforces Ġthe Ġneed Ġfor Ġinterpret ability . ĠAlthough Ġthe Ġdevelopment Ġof Ġsupervised Ġmodels Ġis Ġusually Ġperformance Ġdriven , Ġplacing Ġmore Ġemphasis Ġon Ġinterpret ability Ġcould Ġincrease Ġthe Ġadoption Ġof Ġthese Ġmodels Ġin Ġinsight - driven Ġanalyses . ĠOne Ġway Ġwould Ġbe Ġto Ġonly Ġuse Ġmodels Ġthat Ġare Ġalready Ġsomewhat Ġinterpret able , Ġfor Ġexample Ġmodels Ġthat Ġuse Ġa Ġsmall Ġnumber Ġof Ġhuman - interpret able Ġfeatures . ĠRather Ġthan Ġimposing Ġsuch Ġrestrictions , Ġthere Ġis Ġalso Ġwork Ġon Ġgenerating Ġpost - h oc Ġexplanations Ġfor Ġindividual Ġpredictions Ġ( e . g ., ĠB IB REF 46 Ġ), Ġeven Ġwhen Ġthe Ġunderlying Ġmodel Ġitself Ġis Ġvery Ġcomplex . </s> Topic Ġmodeling </s> Topic Ġmodels Ġ( e . g ., ĠL DA ĠB IB REF 47 Ġ) Ġare Ġusually Ġun super vised Ġand Ġtherefore Ġless Ġbiased Ġtowards Ġhuman - defined Ġcategories . ĠThey Ġare Ġespecially Ġsuited Ġfor Ġinsight - driven Ġanalysis , Ġbecause Ġthey Ġare Ġconstrained Ġin Ġways Ġthat Ġmake Ġtheir Ġoutput Ġinterpret able . ĠAlthough Ġthere Ġis Ġno Ġguarantee Ġthat Ġa ĠâĢ ľ topic âĢ Ŀ Ġwill Ġcorrespond Ġto Ġa Ġrecognizable Ġtheme Ġor Ġevent Ġor Ġdiscourse , Ġthey Ġoften Ġdo Ġso Ġin Ġways Ġthat Ġother Ġmethods Ġdo Ġnot . ĠTheir Ġeasy Ġapplic ability Ġwithout Ġsupervision Ġand Ġready Ġinterpret ability Ġmake Ġtopic Ġmodels Ġgood Ġfor Ġexploration . ĠTopic Ġmodels Ġare Ġless Ġsuccessful Ġfor Ġmany Ġperformance - driven Ġapplications . ĠRaw Ġword Ġfeatures Ġare Ġalmost Ġalways Ġbetter Ġthan Ġtopics Ġfor Ġsearch Ġand Ġdocument Ġclassification . ĠL ST Ms Ġand Ġother Ġneural Ġnetwork Ġmodels Ġare Ġbetter Ġas Ġlanguage Ġmodels . ĠContinuous Ġword Ġembed d ings Ġhave Ġmore Ġexpressive Ġpower Ġto Ġrepresent Ġfine - gr ained Ġsemantic Ġsimilarities Ġbetween Ġwords . </s> A Ġtopic Ġmodel Ġprovides Ġa Ġdifferent Ġperspective Ġon Ġa Ġcollection . ĠIt Ġcreates Ġa Ġset Ġof Ġprobability Ġdistributions Ġover Ġthe Ġvocabulary Ġof Ġthe Ġcollection , Ġwhich , Ġwhen Ġcombined Ġtogether Ġin Ġdifferent Ġproportions , Ġbest Ġmatch Ġthe Ġcontent Ġof Ġthe Ġcollection . ĠWe Ġcan Ġsort Ġthe Ġwords Ġin Ġeach Ġof Ġthese Ġdistributions Ġin Ġdescending Ġorder Ġby Ġprobability , Ġtake Ġsome Ġarbitrary Ġnumber Ġof Ġmost - pro bable Ġwords , Ġand Ġget Ġa Ġsense Ġof Ġwhat Ġ( if Ġanything ) Ġthe Ġtopic Ġis ĠâĢ ľ about âĢ Ŀ . ĠEach Ġof Ġthe Ġtext Ġsegments Ġalso Ġhas Ġits Ġown Ġdistribution Ġover Ġthe Ġtopics , Ġand Ġwe Ġcan Ġsort Ġthese Ġsegments Ġby Ġtheir Ġprobability Ġwithin Ġa Ġgiven Ġtopic Ġto Ġget Ġa Ġsense Ġof Ġhow Ġthat Ġtopic Ġis Ġused . </s> One Ġof Ġthe Ġmost Ġcommon Ġquestions Ġabout Ġtopic Ġmodels Ġis Ġhow Ġmany Ġtopics Ġto Ġuse , Ġusually Ġwith Ġthe Ġimplicit Ġassumption Ġthat Ġthere Ġis Ġa ĠâĢ ľ right âĢ Ŀ Ġnumber Ġthat Ġis Ġinherent Ġin Ġthe Ġcollection . ĠWe Ġprefer Ġto Ġthink Ġof Ġthis Ġparameter Ġas Ġmore Ġlike Ġthe Ġscale Ġof Ġa Ġmap Ġor Ġthe Ġmagnification Ġof Ġa Ġmicroscope . ĠThe ĠâĢ ľ right âĢ Ŀ Ġnumber Ġis Ġdetermined Ġby Ġthe Ġneeds Ġof Ġthe Ġuser , Ġnot Ġby Ġthe Ġcollection . ĠIf Ġthe Ġanalyst Ġis Ġlooking Ġfor Ġa Ġbroad Ġoverview , Ġa Ġrelatively Ġsmall Ġnumber Ġof Ġtopics Ġmay Ġbe Ġbest . ĠIf Ġthe Ġanalyst Ġis Ġlooking Ġfor Ġfine - gr ained Ġphenomena , Ġa Ġlarger Ġnumber Ġis Ġbetter . </s> After Ġfitting Ġthe Ġmodel , Ġit Ġmay Ġbe Ġnecessary Ġto Ġcircle Ġback Ġto Ġan Ġearlier Ġphase . ĠTopic Ġmodels Ġfind Ġconsistent Ġpatterns . ĠWhen Ġauthors Ġrepeatedly Ġuse Ġa Ġparticular Ġtheme Ġor Ġdiscourse , Ġthat Ġrepetition Ġcreates Ġa Ġconsistent Ġpattern . ĠBut Ġother Ġfactors Ġcan Ġalso Ġcreate Ġsimilar Ġpatterns , Ġwhich Ġlook Ġas Ġgood Ġto Ġthe Ġalgorithm . ĠWe Ġmight Ġnotice Ġa Ġtopic Ġthat Ġhas Ġhighest Ġprobability Ġon ĠFrench Ġstop words , Ġindicating Ġthat Ġwe Ġneed Ġto Ġdo Ġa Ġbetter Ġjob Ġof Ġfiltering Ġby Ġlanguage . ĠWe Ġmight Ġnotice Ġa Ġtopic Ġof Ġword Ġfragments , Ġsuch Ġas ĠâĢ ľ ing âĢ Ŀ , ĠâĢ ľ tion âĢ Ŀ , ĠâĢ ľ inter âĢ Ŀ , Ġindicating Ġthat Ġwe Ġare Ġnot Ġhandling Ġend - of - line Ġhyp hen ation Ġcorrectly . ĠWe Ġmay Ġneed Ġto Ġadd Ġto Ġour Ġstop list Ġor Ġchange Ġhow Ġwe Ġcur ate Ġmulti - word Ġterms . </s> Val idation </s> The Ġoutput Ġof Ġour Ġmeasurement Ġprocedures Ġ( in Ġthe Ġsocial Ġsciences Ġoften Ġcalled Ġthe ĠâĢ ľ sc ores âĢ Ŀ ) Ġmust Ġnow Ġbe Ġassessed Ġin Ġterms Ġof Ġtheir Ġreliability Ġand Ġvalidity Ġwith Ġregard Ġto Ġthe Ġ( system ized ) Ġconcept . ĠRel iability Ġaims Ġto Ġcapture Ġrepeat ability , Ġi . e . Ġthe Ġextent Ġto Ġwhich Ġa Ġgiven Ġtool Ġprovides Ġconsistent Ġresults . </s> Val idity Ġassess es Ġthe Ġextent Ġto Ġwhich Ġa Ġgiven Ġmeasurement Ġtool Ġmeasures Ġwhat Ġit Ġis Ġsupposed Ġto Ġmeasure . ĠIn ĠN LP Ġand Ġmachine Ġlearning , Ġmost Ġmodels Ġare Ġprimarily Ġevaluated Ġby Ġcomparing Ġthe Ġmachine - generated Ġlabels Ġagainst Ġan Ġannot ated Ġsample . ĠThis Ġapproach Ġpres umes Ġthat Ġthe Ġhuman Ġoutput Ġis Ġthe ĠâĢ ľ gold Ġstandard " Ġagainst Ġwhich Ġperformance Ġshould Ġbe Ġtested . ĠIn Ġcontrast , Ġwhen Ġthe Ġreliability Ġis Ġmeasured Ġbased Ġon Ġthe Ġoutput Ġof Ġdifferent Ġannot ators , Ġno Ġc oder Ġis Ġtaken Ġas Ġthe Ġstandard Ġand Ġthe Ġlikelihood Ġof Ġcod ers Ġreaching Ġagreement Ġby Ġchance Ġ( rather Ġthan Ġbecause Ġthey Ġare ĠâĢ ľ correct ") Ġis Ġfact ored Ġinto Ġthe Ġresulting Ġstatistic . ĠComp aring Ġagainst Ġa ĠâĢ ľ gold Ġstandard âĢ Ŀ Ġsuggests Ġthat Ġthe Ġthreshold Ġfor Ġhuman Ġinter - Ġand Ġintra - c oder Ġreliability Ġshould Ġbe Ġparticularly Ġhigh . </s> Acc uracy , Ġas Ġwell Ġas Ġother Ġmeasures Ġsuch Ġas Ġprecision , Ġrecall Ġand ĠF - score , Ġare Ġsometimes Ġpresented Ġas Ġa Ġmeasure Ġof Ġvalidity , Ġbut Ġif Ġwe Ġdo Ġnot Ġhave Ġa Ġgenuinely Ġobjective Ġdetermination Ġof Ġwhat Ġsomething Ġis Ġsupposed Ġmeasure âĢĶ as Ġis Ġoften Ġthe Ġcase Ġin Ġtext Ġanalysis âĢĶ then Ġaccuracy Ġis Ġperhaps Ġa Ġbetter Ġindication Ġof Ġreliability Ġthan Ġof Ġvalidity . ĠIn Ġthat Ġcase , Ġvalidity Ġneeds Ġto Ġbe Ġassessed Ġbased Ġon Ġother Ġtechniques Ġlike Ġthose Ġwe Ġdiscuss Ġlater Ġin Ġthis Ġsection . ĠIt Ġis Ġalso Ġworth Ġasking Ġwhat Ġlevel Ġof Ġaccuracy Ġis Ġsufficient Ġfor Ġour Ġanalysis Ġand Ġto Ġwhat Ġextent Ġthere Ġmay Ġbe Ġan Ġupper Ġbound , Ġespecially Ġwhen Ġthe Ġlabels Ġare Ġnative Ġto Ġthe Ġdata Ġor Ġwhen Ġthe Ġnotion Ġof Ġa ĠâĢ ľ gold Ġstandard âĢ Ŀ Ġis Ġnot Ġappropriate . </s> For Ġsome Ġin Ġthe Ġhumanities , Ġvalidation Ġtakes Ġthe Ġform Ġof Ġclose Ġreading , Ġnot Ġdesigned Ġto Ġconfirm Ġwhether Ġthe Ġmodel Ġoutput Ġis Ġcorrect , Ġbut Ġto Ġpresent Ġwhat ĠB IB REF 48 Ġrefers Ġto Ġas Ġa Ġform Ġof ĠâĢ ľ f urther Ġdiscovery Ġin Ġtwo Ġdirections âĢ Ŀ . ĠModel Ġoutputs Ġtell Ġus Ġsomething Ġabout Ġthe Ġtexts , Ġwhile Ġa Ġclose Ġreading Ġof Ġthe Ġtexts Ġalongside Ġthose Ġoutputs Ġtells Ġus Ġsomething Ġabout Ġthe Ġmodels Ġthat Ġcan Ġbe Ġused Ġfor Ġmore Ġeffective Ġmodel Ġbuilding . ĠApp lying Ġthis Ġcircular , Ġiter ative Ġprocess Ġto Ġ450 Ġ18 th - century Ġnovels Ġwritten Ġin Ġthree Ġlanguages , ĠPiper Ġwas Ġable Ġto Ġuncover Ġa Ġnew Ġform Ġof ĠâĢ ľ con vers ional Ġnovel âĢ Ŀ Ġthat Ġwas Ġnot Ġpreviously Ġcaptured Ġin ĠâĢ ľ liter ary Ġhistory 's Ġreceived Ġcritical Ġcategories âĢ Ŀ ĠB IB REF 48 Ġ. </s> Along Ġsimilar Ġlines , Ġwe Ġcan Ġsubject Ġboth Ġthe Ġmachine - generated Ġoutput Ġand Ġthe Ġhuman Ġannotations Ġto Ġanother Ġround Ġof Ġcontent Ġvalidation . ĠThat Ġis , Ġtake Ġa Ġstrat ified Ġrandom Ġsample , Ġselecting Ġobservations Ġfrom Ġthe Ġfull Ġrange Ġof Ġscores , Ġand Ġask : ĠDo Ġthese Ġmake Ġsense Ġin Ġlight Ġof Ġthe Ġsystem at ized Ġconcept ? ĠIf Ġnot , Ġwhat Ġseems Ġto Ġbe Ġmissing ? ĠOr Ġis Ġsomething Ġextr aneous Ġbeing Ġcaptured ? ĠThis Ġis Ġprimarily Ġa Ġqualitative Ġprocess Ġthat Ġrequires Ġreturning Ġto Ġtheory Ġand Ġinterrog ating Ġthe Ġsystem at ized Ġconcept , Ġindicators , Ġand Ġscores Ġtogether . ĠThis Ġtype Ġof Ġvalidation Ġis Ġrarely Ġdone Ġin ĠN LP , Ġbut Ġit Ġis Ġespecially Ġimportant Ġwhen Ġit Ġis Ġdifficult Ġto Ġassess Ġwhat Ġdrives Ġa Ġgiven Ġmachine Ġlearning Ġmodel . ĠIf Ġthere Ġis Ġa Ġmismatch Ġbetween Ġthe Ġscores Ġand Ġsystem at ized Ġconcept Ġat Ġthis Ġstage , Ġthe Ġcode book Ġmay Ġneed Ġto Ġbe Ġadjusted , Ġhuman Ġcod ers Ġret rained , Ġmore Ġtraining Ġdata Ġprepared , Ġalgorithms Ġadjusted , Ġor Ġin Ġsome Ġinstances , Ġeven Ġa Ġnew Ġanalytical Ġmethod Ġadopted . </s> Other Ġtypes Ġof Ġvalidation Ġare Ġalso Ġpossible , Ġsuch Ġas Ġcomparing Ġwith Ġother Ġapproaches Ġthat Ġaim Ġto Ġcapture Ġthe Ġsame Ġconcept , Ġor Ġcomparing Ġthe Ġoutput Ġwith Ġexternal Ġmeasures Ġ( e . g ., Ġpublic Ġopinion Ġpolls , Ġthe Ġoccurrence Ġof Ġfuture Ġevents ). ĠWe Ġcan Ġalso Ġgo Ġbeyond Ġonly Ġevaluating Ġthe Ġlabels Ġ( or Ġpoint Ġestimates ). ĠB IB REF 16 Ġused Ġhuman Ġjudgments Ġto Ġnot Ġonly Ġassess Ġthe Ġpositional Ġestimates Ġfrom Ġa Ġscaling Ġmethod Ġof Ġlatent Ġpolitical Ġtraits Ġbut Ġalso Ġto Ġassess Ġuncertainty Ġintervals . ĠUsing Ġdifferent Ġtypes Ġof Ġvalidation Ġcan Ġincrease Ġour Ġconfidence Ġin Ġthe Ġapproach , Ġespecially Ġwhen Ġthere Ġis Ġno Ġclear Ġnotion Ġof Ġground Ġtruth . </s> Besides Ġfocusing Ġon Ġrather Ġabstract Ġevaluation Ġmeasures , Ġwe Ġcould Ġalso Ġassess Ġthe Ġmodels Ġin Ġtask - based Ġsettings Ġusing Ġhuman Ġexperts . ĠFurthermore , Ġfor Ġinsight - driven Ġanalyses , Ġit Ġcan Ġbe Ġmore Ġuseful Ġto Ġfocus Ġon Ġimproving Ġexplanatory Ġpower Ġthan Ġmaking Ġsmall Ġimprovements Ġin Ġpredictive Ġperformance . </s> Analysis </s> In Ġthis Ġphase , Ġwe Ġuse Ġour Ġmodels Ġto Ġexplore Ġor Ġanswer Ġour Ġresearch Ġquestions . ĠFor Ġexample , Ġgiven Ġa Ġtopic Ġmodel Ġwe Ġcan Ġlook Ġat Ġthe Ġconnection Ġbetween Ġtopics Ġand Ġmetadata Ġelements . ĠTags Ġsuch Ġas ĠâĢ ľ hate Ġspeech " Ġor Ġmetadata Ġinformation Ġimply Ġa Ġcertain Ġway Ġof Ġorganizing Ġthe Ġcollection . ĠComput ational Ġmodels Ġprovide Ġanother Ġorganization , Ġwhich Ġmay Ġdiffer Ġin Ġways Ġthat Ġprovide Ġmore Ġinsight Ġinto Ġhow Ġthese Ġcategories Ġmanifest Ġthemselves , Ġor Ġfail Ġto Ġdo Ġso . </s> Moreover , Ġwhen Ġusing Ġa Ġsupervised Ġapproach , Ġthe ĠâĢ ľ errors âĢ Ŀ , Ġi . e . Ġdisagreement Ġbetween Ġthe Ġsystem Ġoutput Ġand Ġhuman - provided Ġlabels , Ġcan Ġpoint Ġtowards Ġinteresting Ġcases Ġfor Ġcloser Ġanalysis Ġand Ġhelp Ġus Ġreflect Ġon Ġour Ġconceptual izations . ĠIn Ġthe Ġwords Ġof ĠB IB REF 2 Ġ, Ġthey Ġcan Ġbe ĠâĢ ľ opp ortun ities Ġfor Ġinterpretation âĢ Ŀ . ĠOther Ġtypes Ġof ĠâĢ ľ fail ures âĢ Ŀ Ġcan Ġbe Ġinsightful Ġas Ġwell . ĠSometimes Ġthere Ġis Ġa ĠâĢ ľ dog Ġthat Ġdidn 't Ġbark âĢ Ŀ ĠB IB REF 49 ĠâĢĵ i . e ., Ġsomething Ġthat Ġeveryone Ġthinks Ġwe Ġshould Ġhave Ġfound , Ġbut Ġwe Ġdid Ġnot . ĠOr , Ġsometimes Ġthe Ġfailures Ġare Ġtelling Ġus Ġabout Ġthe Ġexistence Ġof Ġsomething Ġin Ġthe Ġdata Ġthat Ġnobody Ġnoticed , Ġor Ġthought Ġimportant , Ġuntil Ġthen Ġ( e . g ., Ġthe Ġlarge Ġnumber Ġof Ġtravel Ġjournals Ġin ĠDarwin 's Ġreading Ġlists ). </s> Com put ational Ġtext Ġanalysis Ġis Ġnot Ġa Ġreplacement Ġfor Ġbut Ġrather Ġan Ġaddition Ġto Ġthe Ġapproaches Ġone Ġcan Ġtake Ġto Ġanalyze Ġsocial Ġand Ġcultural Ġphenomena Ġusing Ġtextual Ġdata . ĠBy Ġmoving Ġback Ġand Ġforth Ġbetween Ġlarge - scale Ġcomputational Ġanalyses Ġand Ġsmall - scale Ġqualitative Ġanalyses , Ġwe Ġcan Ġcombine Ġtheir Ġstrengths Ġso Ġthat Ġwe Ġcan Ġidentify Ġlarge - scale Ġand Ġlong - term Ġtrends , Ġbut Ġalso Ġtell Ġindividual Ġstories . ĠFor Ġexample , Ġthe ĠReddit Ġstudy Ġon Ġhate Ġspeech ĠB IB REF 0 Ġraised Ġvarious Ġfollow - up Ġquestions : ĠCan Ġwe Ġdistinguish Ġhate Ġspeech Ġfrom Ġpeople Ġtalking Ġabout Ġhate Ġspeech ? ĠDid Ġpeople Ġfind Ġnew Ġways Ġto Ġexpress Ġhate Ġspeech ? ĠIf Ġso , Ġdid Ġthe Ġtotal Ġamount Ġof Ġonline Ġhate Ġspeech Ġdecrease Ġafter Ġall ? ĠAs Ġpossible Ġnext Ġsteps , Ġa Ġqualitative Ġdiscourse Ġanalyst Ġmight Ġexamine Ġa Ġsmaller Ġcorpus Ġto Ġinvestigate Ġwhether Ġcommenters Ġwere Ġindeed Ġexpressing Ġhate Ġspeech Ġin Ġnew Ġways ; Ġa Ġspecialist Ġin Ġinterview Ġmethod ologies Ġmight Ġreach Ġout Ġto Ġcommenters Ġto Ġbetter Ġunderstand Ġthe Ġrole Ġof Ġonline Ġhate Ġspeech Ġin Ġtheir Ġlives . ĠComput ational Ġtext Ġanalysis Ġrepresents Ġa Ġstep Ġtowards Ġbetter Ġunderstanding Ġsocial Ġand Ġcultural Ġphenomena , Ġand Ġit Ġis Ġin Ġmany Ġcases Ġbetter Ġsuited Ġtowards Ġopening Ġquestions Ġrather Ġthan Ġclosing Ġthem . </s> Conclusion </s> In sight - driven Ġcomputational Ġanalysis Ġof Ġtext Ġis Ġbecoming Ġincreasingly Ġcommon . ĠIt Ġnot Ġonly Ġhelps Ġus Ġsee Ġmore Ġbroadly , Ġit Ġhelps Ġus Ġsee Ġsubtle Ġpatterns Ġmore Ġclearly Ġand Ġallows Ġus Ġto Ġexplore Ġradical Ġnew Ġquestions Ġabout Ġculture Ġand Ġsociety . ĠIn Ġthis Ġarticle Ġwe Ġhave Ġconsolidated Ġour Ġexperiences , Ġas Ġscholars Ġfrom Ġvery Ġdifferent Ġdisciplines , Ġin Ġanalyzing Ġtext Ġas Ġsocial Ġand Ġcultural Ġdata Ġand Ġdescribed Ġhow Ġthe Ġresearch Ġprocess Ġoften Ġunfolds . ĠEach Ġof Ġthe Ġsteps Ġin Ġthe Ġprocess Ġis Ġtime - consuming Ġand Ġlabor - intensive . ĠEach Ġpresents Ġchallenges . ĠAnd Ġespecially Ġwhen Ġworking Ġacross Ġdisciplines , Ġthe Ġresearch Ġoften Ġinvolves Ġa Ġfair Ġamount Ġof Ġdiscussion âĢĶ even Ġnegotiation âĢĶ about Ġwhat Ġmeans Ġof Ġoperational ization Ġand Ġapproaches Ġto Ġanalysis Ġare Ġappropriate Ġand Ġfeasible . ĠAnd Ġyet , Ġwith Ġa Ġbit Ġof Ġpersever ance Ġand Ġmutual Ġunderstanding , Ġconcept ually Ġsound Ġand Ġmeaningful Ġwork Ġresults Ġso Ġthat Ġwe Ġcan Ġtruly Ġmake Ġuse Ġof Ġthe Ġexciting Ġopportunities Ġrich Ġtextual Ġdata Ġoffers . </s> Acknowled gements </s> This Ġwork Ġwas Ġsupported Ġby ĠThe ĠAlan ĠTuring ĠInstitute Ġunder Ġthe ĠEPS RC Ġgrant ĠEP / N 51 01 29 / 1 . ĠDong ĠNguyen Ġis Ġsupported Ġwith Ġan ĠAlan ĠTuring ĠInstitute ĠFellowship Ġ( T U / A / 00000 6 ). ĠMaria ĠLi ak ata Ġis Ġa ĠTuring Ġfellow Ġat Ġ40 %. ĠWe Ġwould Ġalso Ġlike Ġto Ġthank Ġthe Ġparticipants Ġof Ġthe ĠâĢ ľ B rid ging Ġdisciplines Ġin Ġanalys ing Ġtext Ġas Ġsocial Ġand Ġcultural Ġdata âĢ Ŀ Ġworkshop Ġheld Ġat Ġthe ĠTuring ĠInstitute Ġ( 2017 ) Ġfor Ġinsightful Ġdiscussions . ĠThe Ġworkshop Ġwas Ġfunded Ġby Ġa ĠTuring ĠInstitute Ġseed Ġfunding Ġaward Ġto ĠNguyen Ġand ĠLi ak ata .
Introduction In June 2015, the operators of the online discussion site Reddit banned several communities under new anti-harassment rules. BIBREF0 used this opportunity to combine rich online data with computational methods to study a current question: Does eliminating these “echo chambers” diminish the amount of hate speech overall? Exciting opportunities like these, at the intersection of “thick” cultural and societal questions on the one hand, and the computational analysis of rich textual data on larger-than-human scales on the other, are becoming increasingly common. Indeed, computational analysis is opening new possibilities for exploring challenging questions at the heart of some of the most pressing contemporary cultural and social issues. While a human reader is better equipped to make logical inferences, resolve ambiguities, and apply cultural knowledge than a computer, human time and attention are limited. Moreover, many patterns are not obvious in any specific context, but only stand out in the aggregate. For example, in a landmark study, BIBREF1 analyzed the authorship of The Federalist Papers using a statistical text analysis by focusing on style, based on the distribution of function words, rather than content. As another example, BIBREF2 studied what defines English haiku and showed how computational analysis and close reading can complement each other. Computational approaches are valuable precisely because they help us identify patterns that would not otherwise be discernible. Yet these approaches are not a panacea. Examining thick social and cultural questions using computational text analysis carries significant challenges. For one, texts are culturally and socially situated. They reflect the ideas, values and beliefs of both their authors and their target audiences, and such subtleties of meaning and interpretation are difficult to incorporate in computational approaches. For another, many of the social and cultural concepts we seek to examine are highly contested — hate speech is just one such example. Choices regarding how to operationalize and analyze these concepts can raise serious concerns about conceptual validity and may lead to shallow or obvious conclusions, rather than findings that reflect the depth of the questions we seek to address. These are just a small sample of the many opportunities and challenges faced in computational analyses of textual data. New possibilities and frustrating obstacles emerge at every stage of research, from identification of the research question to interpretation of the results. In this article, we take the reader through a typical research process that involves measuring social or cultural concepts using computational methods, discussing both the opportunities and complications that often arise. In the Reddit case, for example, hate speech is measured, however imperfectly, by the presence of particular words semi-automatically extracted from a machine learning algorithm. Operationalizations are never perfect translations, and are often refined over the course of an investigation, but they are crucial. We begin our exploration with the identification of research questions, proceed through data selection, conceptualization, and operationalization, and end with analysis and the interpretation of results. The research process sounds more or less linear this way, but each of these phases overlaps, and in some instances turns back upon itself. The analysis phase, for example, often feeds back into the original research questions, which may continue to evolve for much of the project. At each stage, our discussion is critically informed by insights from the humanities and social sciences, fields that have focused on, and worked to tackle, the challenges of textual analysis—albeit at smaller scales—since their inception. In describing our experiences with computational text analysis, we hope to achieve three primary goals. First, we aim to shed light on thorny issues not always at the forefront of discussions about computational text analysis methods. Second, we hope to provide a set of best practices for working with thick social and cultural concepts. Our guidance is based on our own experiences and is therefore inherently imperfect. Still, given our diversity of disciplinary backgrounds and research practices, we hope to capture a range of ideas and identify commonalities that will resonate for many. And this leads to our final goal: to help promote interdisciplinary collaborations. Interdisciplinary insights and partnerships are essential for realizing the full potential of any computational text analysis that involves social and cultural concepts, and the more we are able to bridge these divides, the more fruitful we believe our work will be. Research questions We typically start by identifying the questions we wish to explore. Can text analysis provide a new perspective on a “big question” that has been attracting interest for years? Or can we raise new questions that have only recently emerged, for example about social media? For social scientists working in computational analysis, the questions are often grounded in theory, asking: How can we explain what we observe? These questions are also influenced by the availability and accessibility of data sources. For example, the choice to work with data from a particular social media platform may be partly determined by the fact that it is freely available, and this will in turn shape the kinds of questions that can be asked. A key output of this phase are the concepts to measure, for example: influence; copying and reproduction; the creation of patterns of language use; hate speech. Computational analysis of text motivated by these questions is insight driven: we aim to describe a phenomenon or explain how it came about. For example, what can we learn about how and why hate speech is used or how this changes over time? Is hate speech one thing, or does it comprise multiple forms of expression? Is there a clear boundary between hate speech and other types of speech, and what features make it more or less ambiguous? In these cases, it is critical to communicate high-level patterns in terms that are recognizable. This contrasts with much of the work in computational text analysis, which tends to focus on automating tasks that humans perform inefficiently. These tasks range from core linguistically motivated tasks that constitute the backbone of natural language processing, such as part-of-speech tagging and parsing, to filtering spam and detecting sentiment. Many tasks are motivated by applications, for example to automatically block online trolls. Success, then, is often measured by performance, and communicating why a certain prediction was made—for example, why a document was labeled as positive sentiment, or why a word was classified as a noun—is less important than the accuracy of the prediction itself. The approaches we use and what we mean by `success' are thus guided by our research questions. Domain experts and fellow researchers can provide feedback on questions and help with dynamically revising them. For example, they may say “we already think we know that”, “that's too naïve”, “that doesn't reflect social reality” (negative); “two major camps in the field would give different answers to that question” (neutral); “we tried to look at that back in the 1960s, but we didn't have the technology” (positive); and “that sounds like something that people who made that archive would love”, “that's a really fundamental question” (very positive). Sometimes we also hope to connect to multiple disciplines. For example, while focusing on the humanistic concerns of an archive, we could also ask social questions such as “is this archive more about collaborative processes, culture-building or norm creation?” or “how well does this archive reflect the society in which it is embedded?" BIBREF3 used quantitative methods to tell a story about Darwin's intellectual development—an essential biographical question for a key figure in the history of science. At the same time, their methods connected Darwin's development to the changing landscape of Victorian scientific culture, allowing them to contrast Darwin's “foraging” in the scientific literature of his time to the ways in which that literature was itself produced. Finally, their methods provided a case study, and validation of technical approaches, for cognitive scientists who are interested in how people explore and exploit sources of knowledge. Questions about potential “dual use” may also arise. Returning to our introductory example, BIBREF0 started with a deceptively simple question: if an internet platform eliminates forums for hate speech, does this impact hate speech in other forums? The research was motivated by the belief that a rising tide of online hate speech was (and is) making the internet increasingly unfriendly for disempowered groups, including minorities, women, and LBGTQ individuals. Yet the possibility of dual use troubled the researchers from the onset. Could the methodology be adopted to target the speech of groups like Black Lives Matter? Could it be adopted by repressive governments to minimize online dissent? While these concerns remained, they concluded that hypothetical dual use scenarios did not outweigh the tangible contribution this research could offer towards making the online environment more equal and just. Data The next step involves deciding on the data sources, collecting and compiling the dataset, and inspecting its metadata. Data acquisition Many scholars in the humanities and the social sciences work with sources that are not available in digital form, and indeed may never be digitized. Others work with both analogue and digitized materials, and the increasing digitization of archives has opened opportunities to study these archives in new ways. We can go to the canonical archive or open up something that nobody has studied before. For example, we might focus on major historical moments (French Revolution, post-Milosevic Serbia) or critical epochs (Britain entering the Victorian era, the transition from Latin to proto-Romance). Or, we could look for records of how people conducted science, wrote and consumed literature, and worked out their philosophies. A growing number of researchers work with born-digital sources or data. Born-digital data, e.g., from social media, generally do not involve direct elicitation from participants and therefore enable unobtrusive measurements BIBREF5 , BIBREF6 . In contrast, methods like surveys sometimes elicit altered responses from participants, who might adapt their responses to what they think is expected. Moreover, born-digital data is often massive, enabling large-scale studies of language and behavior in a variety of social contexts. Still, many scholars in the social sciences and humanities work with multiple data sources. The variety of sources typically used means that more than one data collection method is often required. For example, a project examining coverage of a UK General Election, could draw data from traditional media, web archives, Twitter and Facebook, campaign manifestos, etc. and might combine textual analysis of these materials with surveys, laboratory experiments, or field observations offline. In contrast, many computational studies based on born-digital data have focused on one specific source, such as Twitter. The use of born-digital data raises ethical concerns. Although early studies often treated privacy as a binary construct, many now acknowledge its complexity BIBREF7 . Conversations on private matters can be posted online, visible for all, but social norms regarding what should be considered public information may differ from the data's explicit visibility settings. Often no informed consent has been obtained, raising concerns and challenges regarding publishing content and potentially harmful secondary uses BIBREF8 , BIBREF4 . Recently, concerns about potential harms stemming from secondary uses have led a number of digital service providers to restrict access to born-digital data. Facebook and Twitter, for example, have reduced or eliminated public access to their application programming interfaces (APIs) and expressed hesitation about allowing academic researchers to use data from their platforms to examine certain sensitive or controversial topics. Despite the seeming abundance of born-digital data, we therefore cannot take its availability for granted. Working with data that someone else has acquired presents additional problems related to provenance and contextualisation. It may not always be possible to determine the criteria applied during the creation process. For example, why were certain newspapers digitized but not others, and what does this say about the collection? Similar questions arise with the use of born-digital data. For instance, when using the Internet Archive’s Wayback Machine to gather data from archived web pages, we need to consider what pages were captured, which are likely missing, and why. We must often repurpose born-digital data (e.g., Twitter was not designed to measure public opinion), but data biases may lead to spurious results and limit justification for generalization. In particular, data collected via black box APIs designed for commercial, not research, purposes are likely to introduce biases into the inferences we draw, and the closed nature of these APIs means we rarely know what biases are introduced, let alone how severely they might impact our research BIBREF10 . These, however, are not new problems. Historians, for example, have always understood that their sources were produced within particular contexts and for particular purposes, which are not always apparent to us. Non-representative data can still be useful for making comparisons within a sample. In the introductory example on hate speech BIBREF0 , the Reddit forums do not present a comprehensive or balanced picture of hate speech: the writing is almost exclusively in English, the targets of hate speech are mainly restricted (e.g., to black people, or women), and the population of writers is shaped by Reddit's demographics, which skew towards young white men. These biases limit the generalizability of the findings, which cannot be extrapolated to other languages, other types of hate speech, and other demographic groups. However, because the findings are based on measurements on the same sort of hate speech and the same population of writers, as long as the collected data are representative of this specific population, these biases do not pose an intractable validity problem if claims are properly restricted. The size of many newly available datasets is one of their most appealing characteristics. Bigger datasets often make statistics more robust. The size needed for a computational text analysis depends on the research goal: When it involves studying rare events, bigger datasets are needed. However, larger is not always better. Some very large archives are “secretly” collections of multiple and distinct processes that no in-field scholar would consider related. For example, Google Books is frequently used to study cultural patterns, but the over-representation of scientific articles in Google books can be problematic BIBREF11 . Even very large born-digital datasets usually cover limited timespans compared to, e.g., the Gutenberg archive of British novels. This stage of the research also raises important questions about fairness. Are marginalized groups, for example, represented in the tweets we have collected? If not, what types of biases might result from analyses relying on those tweets? Local experts and “informants” can help navigate the data. They can help understand the role an archive plays in the time and place. They might tell us: Is this the central archive, or a peripheral one? What makes it unusual? Or they might tell us how certain underrepresented communities use a social media platform and advise us on strategies for ensuring our data collection includes their perspectives. However, when it is practically infeasible to navigate the data in this way—for instance, when we cannot determine what is missing from Twitter's Streaming API or what webpages are left out of the Internet Archive—we should be open about the limitations of our analyses, acknowledging the flaws in our data and drawing cautious and reasonable conclusions from them. In all cases, we should report the choices we have made when creating or re-using any dataset. Compiling data After identifying the data source(s), the next step is compiling the data. This step is fundamental: if the sources cannot support a convincing result, no result will be convincing. In many cases, this involves defining a “core" set of documents and a “comparison" set. We often have a specific set of documents in mind: an author's work, a particular journal, a time period. But if we want to say that this “core" set has some distinctive property, we need a “comparison" set. Expanding the collection beyond the documents that we would immediately think of has the beneficial effect of increasing our sample size. Having more sources increases the chance that we will notice something consistent across many individually varying contexts. Comparing sets of documents can sometimes support causal inference, presented as a contrast between a treatment group and a control. In BIBREF0 , the treatment consisted of the text written in the two forums that were eventually closed by Reddit. However, identifying a control group required a considerable amount of time and effort. Reddit is a diverse platform, with a wide variety of interactional and linguistic styles; it would be pointless to compare hate speech forums against forums dedicated to, say, pictures of wrecked bicycles. Chandrasekharan et al. used a matching design, populating the control group with forums that were as similar as possible to the treatment group, but were not banned from Reddit. The goal is to estimate the counterfactual scenario: in this case, what would have happened had the site not taken action against these specific forums? An ideal control would make it possible to distinguish the effect of the treatment — closing the forums — from other idiosyncratic properties of texts that were treated. We also look for categories of documents that might not be useful. We might remove documents that are meta-discourse, like introductions and notes, or documents that are in a language that is not the primary language of the collection, or duplicates when we are working with archived web pages. However, we need to carefully consider the potential consequences of information we remove. Does its removal alter the data, or the interpretation of the data, we are analyzing? Are we losing anything that might be valuable at a later stage? Labels and metadata Sometimes all we have is documents, but often we want to look at documents in the context of some additional information, or metadata. This additional information could tell us about the creation of documents (date, author, forum), or about the reception of documents (flagged as hate speech, helpful review). Information about text segments can be extremely valuable, but it is also prone to errors, inconsistencies, bias, and missing information. Examining metadata is a good way to check a collection's balance and representativeness. Are sources disproportionately of one form? Is the collection missing a specific time window? This type of curation can be extremely time consuming as it may require expert labeling, but it often leads to the most compelling results. Sometimes metadata are also used as target labels to develop machine learning models. But using them as a “ground truth” requires caution. Labels sometimes mean something different than we expect. For example, a down vote for a social media post could indicate that the content is offensive, or that the voter simply disagreed with the expressed view. Conceptualization A core step in many analyses is translating social and cultural concepts (such as hate speech, rumor, or conversion) into measurable quantities. Before we can develop measurements for these concepts (the operationalization step, or the “implementation” step as denoted by BIBREF12 ), we need to define them. In the conceptualization phase we often start with questions such as: who are the domain experts, and how have they approached the topic? We are looking for a definition of the concept that is flexible enough to apply on our dataset, yet formal enough for computational research. For example, our introductory study on hate speech BIBREF0 used a statement on hate speech produced by the European Union Court of Human Rights. The goal was not to implement this definition directly in software but to use it as a reference point to anchor subsequent analyses. If we want to move beyond the use of ad hoc definitions, it can be useful to distinguish between what political scientists Adcock and Collier call the “background concept” and the “systematized concept” BIBREF13 . The background concept comprises the full and diverse set of meanings that might be associated with a particular term. This involves delving into theoretical, conceptual, and empirical studies to assess how a concept has been defined by other scholars and, most importantly, to determine which definition is most appropriate for the particular research question and the theoretical framework in which it is situated. That definition, in turn, represents the systematized concept: the formulation that is adopted for the study. It is important to consider that for social and cultural concepts there is no absolute ground truth. There are often multiple valid definitions for a concept (the “background” concept in the terms of Adcock and Collier), and definitions might be contested over time. This may be uncomfortable for computer scientists, whose primary measure of success is often based on comparing a model's output against “ground truth” or a “gold standard”, e.g., by comparing a sentiment classifier's output against manual annotations. However, the notion of ground truth is uncommon in the humanities and the social sciences and it is often taken too far in machine learning. BIBREF14 notes that in literary criticism and the digital humanities more broadly “interpretation, ambiguity, and argumentation are prized far above ground truth and definitive conclusions". BIBREF15 draw attention to the different attitudes of literary scholars and computational linguists towards ambiguity, stating that “In Computational Linguistics [..] ambiguity is almost uniformly treated as a problem to be solved; the focus is on disambiguation, with the assumption that one true, correct interpretation exists." The latter is probably true for tasks such as spam filtering, but in the social sciences and the humanities many relevant concepts are fundamentally unobservable, such as latent traits of political actors BIBREF16 or cultural fit in organizations BIBREF17 , leading to validation challenges. Moreover, when the ground truth comes from people, it may be influenced by ideological priors, priming, simple differences of opinion or perspective, and many other factors BIBREF18 . We return to this issue in our discussions on validation and analysis. Operationalization In this phase we develop measures (or, “operationalizations”, or “indicators”) for the concepts of interest, a process called “operationalization”. Regardless of whether we are working with computers, the output produced coincides with Adcock and Collier's “scores”—the concrete translation and output of the systematized concept into numbers or labels BIBREF13 . Choices made during this phase are always tied to the question “Are we measuring what we intend to measure?” Does our operationalization match our conceptual definition? To ensure validity we must recognize gaps between what is important and what is easy to measure. We first discuss modeling considerations. Next, we describe several frequently used computational approaches and their limitations and strengths. Modeling considerations The variables (both predictors and outcomes) are rarely simply binary or categorical. For example, a study on language use and age could focus on chronological age (instead of, e.g., social age BIBREF19 ). However, even then, age can be modeled in different ways. Discretization can make the modeling easier and various NLP studies have modeled age as a categorical variable BIBREF20 . But any discretization raises questions: How many categories? Where to place the boundaries? Fine distinctions might not always be meaningful for the analysis we are interested in, but categories that are too broad can threaten validity. Other interesting variables include time, space, and even the social network position of the author. It is often preferable to keep the variable in its most precise form. For example, BIBREF21 perform exploration in the context of hypothesis testing by using latitude and longitude coordinates — the original metadata attached to geotagged social media such as tweets — rather than aggregating into administrative units such as counties or cities. This is necessary when such administrative units are unlikely to be related to the target concept, as is the case in their analysis of dialect differences. Focusing on precise geographical coordinates also makes it possible to recognize fine-grained effects, such as language variation across the geography of a city. Using a particular classification scheme means deciding which variations are visible, and which ones are hidden BIBREF22 . We are looking for a categorization scheme for which it is feasible to collect a large enough labeled document collection (e.g., to train supervised models), but which is also fine-grained enough for our purposes. Classification schemes rarely exhibit the ideal properties, i.e., that they are consistent, their categories are mutually exclusive, and that the system is complete BIBREF22 . Borderline cases are challenging, especially with social and cultural concepts, where the boundaries are often not clear-cut. The choice of scheme can also have ethical implications BIBREF22 . For example, gender is usually represented as a binary variable in NLP and computational models tend to learn gender-stereotypical patterns. The operationalization of gender in NLP has been challenged only recently BIBREF23 , BIBREF24 , BIBREF25 . Supervised and unsupervised learning are the most common approaches to learning from data. With supervised learning, a model learns from labeled data (e.g., social media messages labeled by sentiment) to infer (or predict) these labels from unlabeled texts. In contrast, unsupervised learning uses unlabeled data. Supervised approaches are especially suitable when we have a clear definition of the concept of interest and when labels are available (either annotated or native to the data). Unsupervised approaches, such as topic models, are especially useful for exploration. In this setting, conceptualization and operationalization may occur simultaneously, with theory emerging from the data BIBREF26 . Unsupervised approaches are also used when there is a clear way of measuring a concept, often based on strong assumptions. For example, BIBREF3 measure “surprise” in an analysis of Darwin's reading decisions based on the divergence between two probability distributions. From an analysis perspective, the unit of text that we are labeling (or annotating, or coding), either automatic or manual, can sometimes be different than one's final unit of analysis. For example, if in a study on media frames in news stories, the theoretical framework and research question point toward frames at the story level (e.g., what is the overall causal analysis of the news article?), the story must be the unit of analysis. Yet it is often difficult to validly and reliably code a single frame at the story level. Multiple perspectives are likely to sit side-by-side in a story. Thus, an article on income inequality might point to multiple causes, such as globalization, education, and tax policies. Coding at the sentence level would detect each of these causal explanations individually, but this information would need to be somehow aggregated to determine the overall story-level frame. Sometimes scholars solve this problem by only examining headlines and lead paragraphs, arguing that based on journalistic convention, the most important information can be found at the beginning of a story. However, this leads to a return to a shorter, less nuanced analysis. From a computational perspective, the unit of text can also make a huge difference, especially when we are using bag-of-words models, where word order within a unit does not matter. Small segments, like tweets, sometimes do not have enough information to make their semantic context clear. In contrast, larger segments, like novels, have too much variation, making it difficult to train focused models. Finding a good segmentation sometimes means combining short documents and subdividing long documents. The word “document" can therefore be misleading. But it is so ingrained in the common NLP lexicon that we use it anyway in this article. For insight-driven text analysis, it is often critical that high-level patterns can be communicated. Furthermore, interpretable models make it easier to find spurious features, to do error analysis, and to support interpretation of results. Some approaches are effective for prediction, but harder to interpret. The value we place on interpretability can therefore influence the approach we choose. There is an increasing interest in developing interpretable or transparent models in the NLP and machine learning communities. Annotation Many studies involve human coders. Sometimes the goal is to fully code the data, but in a computational analysis we often use the labels (or annotations) to train machine learning models to automatically recognize them, and to identify language patterns that are associated with these labels. For example, for a project analyzing rumors online BIBREF27 , conversation threads were annotated along different dimensions, including rumor versus non-rumor and stance towards a rumor. The collection of annotation choices make up an annotation scheme (or “codebook”). Existing schemes and annotations can be useful as starting points. Usually settling on an annotation scheme requires several iterations, in which the guidelines are updated and annotation examples are added. For example, a political scientist could use a mixed deductive-inductive strategy for developing a codebook. She starts by laying out a set of theory-driven deductive coding rules, which means that the broad principles of the coding rules are laid out without examining examples first. These are then tested (and possibly adjusted) based on a sample of the data. In line with Adcock and Collier's notion of “content validity” BIBREF13 , the goal is to assess whether the codebook adequately captures the systematized concept. By looking at the data themselves, she gains a better sense of whether some things have been left out of the coding rules and whether anything is superfluous, misleading, or confusing. Adjustments are made and the process is repeated, often with another researcher involved. The final annotations can be collected using a crowdsourcing platform, a smaller number of highly-trained annotators, or a group of experts. Which type of annotator to use should be informed by the complexity and specificity of the concept. For more complex concepts, highly-trained or expert annotators tend to produce more reliable results. However, complex concepts can sometimes be broken down into micro-tasks that can be performed independently in parallel by crowdsourced annotators. Concepts from highly specialized domains may require expert annotators. In all cases, however, some training will be required, and the training phase should involve continual checks of inter-annotator agreement (i.e. intercoder reliability) or checks against a gold standard (e.g. quizzes in crowdsourcing platforms). We also need to decide how inter-annotator agreement will be measured and what an acceptable level of agreement would be. Krippendorff's alpha is frequently used in the social sciences, but the right measure depends on the type of data and task. For manual coding, we can continually check inter-annotator agreement and begin introducing checks of intra-annotator agreement, too. For most communication scholars using only manual content analysis, an acceptable rate of agreement is achieved when Krippendorf's alpha reaches 0.80 or above. When human-coded data are used to validate machine learning algorithms, the reliability of the human-coded data is even more important. Disagreement between annotators can signal weaknesses of the annotation scheme, or highlight the inherent ambiguity in what we are trying to measure. Disagreement itself can be meaningful and can be integrated in subsequent analyses BIBREF28 , BIBREF29 . Data pre-processing Preparing the data can be a complex and time-consuming process, often involving working with partially or wholly unstructured data. The pre-processing steps have a big impact on the operationalizations, subsequent analyses and reproducibility efforts BIBREF30 , and they are usually tightly linked to what we intend to measure. Unfortunately, these steps tend to be underreported, but documenting the pre-processing choices made is essential and is analogous to recording the decisions taken during the production of a scholarly edition or protocols in biomedical research. Data may also vary enormously in quality, depending on how it has been generated. Many historians, for example, work with text produced from an analogue original using Optical Character Recognition (OCR). Often, there will be limited information available regarding the accuracy of the OCR, and the degree of accuracy may even vary within a single corpus (e.g. where digitized text has been produced over a period of years, and the software has gradually improved). The first step, then, is to try to correct for common OCR errors. These will vary depending on the type of text, the date at which the `original' was produced, and the nature of the font and typesetting. One step that almost everyone takes is to tokenize the original character sequence into the words and word-like units. Tokenization is a more subtle and more powerful process than people expect. It is often done using regular expressions or scripts that have been circulating within the NLP community. Tokenization heuristics, however, can be badly confused by emoticons, creative orthography (e.g., U$A, sh!t), and missing whitespace. Multi-word terms are also challenging. Treating them as a single unit can dramatically alter the patterns in text. Many words that are individually ambiguous have clear, unmistakable meanings as terms, like “black hole" or “European Union". However, deciding what constitutes a multi-word term is a difficult problem. In writing systems like Chinese, tokenization is a research problem in its own right. Beyond tokenization, common steps include lowercasing, removing punctuation, stemming (removing suffixes), lemmatization (converting inflections to a base lemma), and normalization, which has never been clearly defined, but often includes grouping abbreviations like “U.S.A." and “USA", ordinals like “1st" and “first", and variant spellings like “noooooo". The main goal of these steps is to improve the ratio of tokens (individual occurrences) to types (the distinct things in a corpus). Each step requires making additional assumptions about which distinctions are relevant: is “apple” different from “Apple”? Is “burnt” different from “burned”? Is “cool" different from “coooool"? Sometimes these steps can actively hide useful patterns, like social meaning BIBREF32 . Some of us therefore try do as little modification as possible. From a multilingual perspective, English and Chinese have an unusually simple inflectional system, and so it is statistically reasonable to treat each inflection as a unique word type. Romance languages have considerably more inflections than English; many indigenous North American languages have still more. For these languages, unseen data is far more likely to include previously-unseen inflections, and therefore, dealing with inflections is more important. On the other hand, the resources for handling inflections vary greatly by language, with European languages dominating the attention of the computational linguistics community thus far. We sometimes also remove words that are not relevant to our goals, for example by calculating vocabulary frequencies. We construct a “stoplist” of words that we are not interested in. If we are looking for semantic themes we might remove function words like determiners and prepositions. If we are looking for author-specific styles, we might remove all words except function words. Some words are generally meaningful but too frequent to be useful within a specific collection. We sometimes also remove very infrequent words. Their occurrences are too low for robust patterns and removing them helps reducing the vocabulary size. The choice of processing steps can be guided by theory or knowledge about the domain as well as experimental investigation. When we have labels, predictive accuracy of a model is a way to assess the effect of the processing steps. In unsupervised settings, it is more challenging to understand the effects of different steps. Inferences drawn from unsupervised settings can be sensitive to pre-processing choices BIBREF33 . Stemming has been found to provide little measurable benefits for topic modeling and can sometimes even be harmful BIBREF34 . All in all, this again highlights the need to document these steps. Finally, we can also mark up the data, e.g., by identifying entities (people, places, organizations, etc.) or parts of speech. Although many NLP tools are available for such tasks, they are often challenged by linguistic variation, such as orthographic variation in historical texts BIBREF35 and social media BIBREF32 . Moreover, the performance of NLP tools often drops when applying them outside the training domain, such as applying tools developed on newswire texts to texts written by younger authors BIBREF36 . Problems (e.g., disambiguation in named entity recognition) are sometimes resolved using considerable manual intervention. This combination of the automated and the manual, however, becomes more difficult as the scale of the data increases, and the `certainty' brought by the latter may have to be abandoned. Dictionary-based approaches Dictionaries are frequently used to code texts in content analyses BIBREF37 . Dictionaries consist of one or more categories (i.e. word lists). Sometimes the output is simply the number of category occurrences (e.g., positive sentiment), thus weighting words within a category equally. In some other cases, words are assigned continuous scores. The high transparency of dictionaries makes them sometimes more suitable than supervised machine learning models. However, dictionaries should only be used if the scores assigned to words match how the words are used in the data (see BIBREF38 for a detailed discussion on limitations). There are many off-the-shelf dictionaries available (e.g., LIWC BIBREF39 ). These are often well-validated, but applying them on a new domain may not be appropriate without additional validation. Corpus- or domain-specific dictionaries can overcome limitations of general-purpose dictionaries. The dictionaries are often manually compiled, but increasingly they are constructed semi-automatically (e.g., BIBREF40 ). When we semi-automatically create a word list, we use automation to identify an initial word list, and human insight to filter it. By automatically generating the initial words lists, words can be identified that human annotators might have difficulty intuiting. By manually filtering the lists, we use our theoretical understanding of the target concept to remove spurious features. In the introduction study, SAGE BIBREF41 was used to obtain a list of words that distinguished the text in the treatment group (subreddits that were closed by Reddit) from text in the control group (similar subreddits that were not closed). The researchers then returned to the hate speech definition provided by the European Court of Human Rights, and manually filtered the top SAGE words based on this definition. Not all identified words fitted the definition. The others included: the names of the subreddits themselves, names of related subreddits, community-specific jargon that was not directly related to hate speech, and terms such as IQ and welfare, which were frequently used in discourses of hate speech, but had significant other uses. The word lists provided the measurement instrument for their main result, which is that the use of hate speech throughout Reddit declined after the two treatment subreddits were closed. Supervised models Supervised learning is frequently used to scale up analyses. For example, BIBREF42 wanted to analyze the motivations of Movember campaign participants. By developing a classifier based on a small set of annotations, they were able to expand the analysis to over 90k participants. The choice of supervised learning model is often guided by the task definition and the label types. For example, to identify stance towards rumors based on sequential annotations, an algorithm for learning from sequential BIBREF43 or time series data BIBREF44 could be used. The features (sometimes called variables or predictors) are used by the model to make the predictions. They may vary from content-based features such as single words, sequences of words, or information about their syntactic structure, to meta-information such as user or network information. Deciding on the features requires experimentation and expert insight and is often called feature engineering. For insight-driven analysis, we are often interested in why a prediction has been made and features that can be interpreted by humans may be preferred. Recent neural network approaches often use simple features as input (such as word embeddings or character sequences), which requires less feature engineering but make interpretation more difficult. Supervised models are powerful, but they can latch on to spurious features of the dataset. This is particularly true for datasets that are not well-balanced, and for annotations that are noisy. In our introductory example on hate speech in Reddit BIBREF0 , the annotations are automatically derived from the forum in which each post appears, and indeed, many of the posts in the forums (subreddits) that were banned by Reddit would be perceived by many as hate speech. But even in banned subreddits, not all of the content is hate speech (e.g., some of the top features were self-referential like the name of the subreddit) but a classifier would learn a high weight for these features. Even when expert annotations are available on the level of individual posts, spurious features may remain. BIBREF45 produced expert annotations of hate speech on Twitter. They found that one of the strongest features for sexism is the name of an Australian TV show, because people like to post sexist comments about the contestants. If we are trying to make claims about what inhibits or encourages hate speech, we would not want those claims to be tied to the TV show's popularity. Such problems are inevitable when datasets are not well-balanced over time, across genres, topics, etc. Especially with social media data, we lack a clear and objective definition of `balance' at this time. The risk of supervised models latching on to spurious features reinforces the need for interpretability. Although the development of supervised models is usually performance driven, placing more emphasis on interpretability could increase the adoption of these models in insight-driven analyses. One way would be to only use models that are already somewhat interpretable, for example models that use a small number of human-interpretable features. Rather than imposing such restrictions, there is also work on generating post-hoc explanations for individual predictions (e.g., BIBREF46 ), even when the underlying model itself is very complex. Topic modeling Topic models (e.g., LDA BIBREF47 ) are usually unsupervised and therefore less biased towards human-defined categories. They are especially suited for insight-driven analysis, because they are constrained in ways that make their output interpretable. Although there is no guarantee that a “topic” will correspond to a recognizable theme or event or discourse, they often do so in ways that other methods do not. Their easy applicability without supervision and ready interpretability make topic models good for exploration. Topic models are less successful for many performance-driven applications. Raw word features are almost always better than topics for search and document classification. LSTMs and other neural network models are better as language models. Continuous word embeddings have more expressive power to represent fine-grained semantic similarities between words. A topic model provides a different perspective on a collection. It creates a set of probability distributions over the vocabulary of the collection, which, when combined together in different proportions, best match the content of the collection. We can sort the words in each of these distributions in descending order by probability, take some arbitrary number of most-probable words, and get a sense of what (if anything) the topic is “about”. Each of the text segments also has its own distribution over the topics, and we can sort these segments by their probability within a given topic to get a sense of how that topic is used. One of the most common questions about topic models is how many topics to use, usually with the implicit assumption that there is a “right” number that is inherent in the collection. We prefer to think of this parameter as more like the scale of a map or the magnification of a microscope. The “right” number is determined by the needs of the user, not by the collection. If the analyst is looking for a broad overview, a relatively small number of topics may be best. If the analyst is looking for fine-grained phenomena, a larger number is better. After fitting the model, it may be necessary to circle back to an earlier phase. Topic models find consistent patterns. When authors repeatedly use a particular theme or discourse, that repetition creates a consistent pattern. But other factors can also create similar patterns, which look as good to the algorithm. We might notice a topic that has highest probability on French stopwords, indicating that we need to do a better job of filtering by language. We might notice a topic of word fragments, such as “ing”, “tion”, “inter”, indicating that we are not handling end-of-line hyphenation correctly. We may need to add to our stoplist or change how we curate multi-word terms. Validation The output of our measurement procedures (in the social sciences often called the “scores”) must now be assessed in terms of their reliability and validity with regard to the (systemized) concept. Reliability aims to capture repeatability, i.e. the extent to which a given tool provides consistent results. Validity assesses the extent to which a given measurement tool measures what it is supposed to measure. In NLP and machine learning, most models are primarily evaluated by comparing the machine-generated labels against an annotated sample. This approach presumes that the human output is the “gold standard" against which performance should be tested. In contrast, when the reliability is measured based on the output of different annotators, no coder is taken as the standard and the likelihood of coders reaching agreement by chance (rather than because they are “correct") is factored into the resulting statistic. Comparing against a “gold standard” suggests that the threshold for human inter- and intra-coder reliability should be particularly high. Accuracy, as well as other measures such as precision, recall and F-score, are sometimes presented as a measure of validity, but if we do not have a genuinely objective determination of what something is supposed measure—as is often the case in text analysis—then accuracy is perhaps a better indication of reliability than of validity. In that case, validity needs to be assessed based on other techniques like those we discuss later in this section. It is also worth asking what level of accuracy is sufficient for our analysis and to what extent there may be an upper bound, especially when the labels are native to the data or when the notion of a “gold standard” is not appropriate. For some in the humanities, validation takes the form of close reading, not designed to confirm whether the model output is correct, but to present what BIBREF48 refers to as a form of “further discovery in two directions”. Model outputs tell us something about the texts, while a close reading of the texts alongside those outputs tells us something about the models that can be used for more effective model building. Applying this circular, iterative process to 450 18th-century novels written in three languages, Piper was able to uncover a new form of “conversional novel” that was not previously captured in “literary history's received critical categories” BIBREF48 . Along similar lines, we can subject both the machine-generated output and the human annotations to another round of content validation. That is, take a stratified random sample, selecting observations from the full range of scores, and ask: Do these make sense in light of the systematized concept? If not, what seems to be missing? Or is something extraneous being captured? This is primarily a qualitative process that requires returning to theory and interrogating the systematized concept, indicators, and scores together. This type of validation is rarely done in NLP, but it is especially important when it is difficult to assess what drives a given machine learning model. If there is a mismatch between the scores and systematized concept at this stage, the codebook may need to be adjusted, human coders retrained, more training data prepared, algorithms adjusted, or in some instances, even a new analytical method adopted. Other types of validation are also possible, such as comparing with other approaches that aim to capture the same concept, or comparing the output with external measures (e.g., public opinion polls, the occurrence of future events). We can also go beyond only evaluating the labels (or point estimates). BIBREF16 used human judgments to not only assess the positional estimates from a scaling method of latent political traits but also to assess uncertainty intervals. Using different types of validation can increase our confidence in the approach, especially when there is no clear notion of ground truth. Besides focusing on rather abstract evaluation measures, we could also assess the models in task-based settings using human experts. Furthermore, for insight-driven analyses, it can be more useful to focus on improving explanatory power than making small improvements in predictive performance. Analysis In this phase, we use our models to explore or answer our research questions. For example, given a topic model we can look at the connection between topics and metadata elements. Tags such as “hate speech" or metadata information imply a certain way of organizing the collection. Computational models provide another organization, which may differ in ways that provide more insight into how these categories manifest themselves, or fail to do so. Moreover, when using a supervised approach, the “errors”, i.e. disagreement between the system output and human-provided labels, can point towards interesting cases for closer analysis and help us reflect on our conceptualizations. In the words of BIBREF2 , they can be “opportunities for interpretation”. Other types of “failures” can be insightful as well. Sometimes there is a “dog that didn't bark” BIBREF49 –i.e., something that everyone thinks we should have found, but we did not. Or, sometimes the failures are telling us about the existence of something in the data that nobody noticed, or thought important, until then (e.g., the large number of travel journals in Darwin's reading lists). Computational text analysis is not a replacement for but rather an addition to the approaches one can take to analyze social and cultural phenomena using textual data. By moving back and forth between large-scale computational analyses and small-scale qualitative analyses, we can combine their strengths so that we can identify large-scale and long-term trends, but also tell individual stories. For example, the Reddit study on hate speech BIBREF0 raised various follow-up questions: Can we distinguish hate speech from people talking about hate speech? Did people find new ways to express hate speech? If so, did the total amount of online hate speech decrease after all? As possible next steps, a qualitative discourse analyst might examine a smaller corpus to investigate whether commenters were indeed expressing hate speech in new ways; a specialist in interview methodologies might reach out to commenters to better understand the role of online hate speech in their lives. Computational text analysis represents a step towards better understanding social and cultural phenomena, and it is in many cases better suited towards opening questions rather than closing them. Conclusion Insight-driven computational analysis of text is becoming increasingly common. It not only helps us see more broadly, it helps us see subtle patterns more clearly and allows us to explore radical new questions about culture and society. In this article we have consolidated our experiences, as scholars from very different disciplines, in analyzing text as social and cultural data and described how the research process often unfolds. Each of the steps in the process is time-consuming and labor-intensive. Each presents challenges. And especially when working across disciplines, the research often involves a fair amount of discussion—even negotiation—about what means of operationalization and approaches to analysis are appropriate and feasible. And yet, with a bit of perseverance and mutual understanding, conceptually sound and meaningful work results so that we can truly make use of the exciting opportunities rich textual data offers. Acknowledgements This work was supported by The Alan Turing Institute under the EPSRC grant EP/N510129/1. Dong Nguyen is supported with an Alan Turing Institute Fellowship (TU/A/000006). Maria Liakata is a Turing fellow at 40%. We would also like to thank the participants of the “Bridging disciplines in analysing text as social and cultural data” workshop held at the Turing Institute (2017) for insightful discussions. The workshop was funded by a Turing Institute seed funding award to Nguyen and Liakata.
What dataset do they use for analysis?
f903396d943541a8cc65edefb04ca37814ed30dd
What Ġdataset Ġdo Ġthey Ġuse Ġfor Ġanalysis ?
Un answer able
<s> What Ġdataset Ġdo Ġthey Ġuse Ġfor Ġanalysis ? </s> Introduction </s> In ĠJune Ġ2015 , Ġthe Ġoperators Ġof Ġthe Ġonline Ġdiscussion Ġsite ĠReddit Ġbanned Ġseveral Ġcommunities Ġunder Ġnew Ġanti - har assment Ġrules . ĠB IB REF 0 Ġused Ġthis Ġopportunity Ġto Ġcombine Ġrich Ġonline Ġdata Ġwith Ġcomputational Ġmethods Ġto Ġstudy Ġa Ġcurrent Ġquestion : ĠDoes Ġeliminating Ġthese ĠâĢ ľ echo Ġchambers âĢ Ŀ Ġdiminish Ġthe Ġamount Ġof Ġhate Ġspeech Ġoverall ? ĠExc iting Ġopportunities Ġlike Ġthese , Ġat Ġthe Ġintersection Ġof ĠâĢ ľ th ick âĢ Ŀ Ġcultural Ġand Ġsocietal Ġquestions Ġon Ġthe Ġone Ġhand , Ġand Ġthe Ġcomputational Ġanalysis Ġof Ġrich Ġtextual Ġdata Ġon Ġlarger - than - human Ġscales Ġon Ġthe Ġother , Ġare Ġbecoming Ġincreasingly Ġcommon . </s> Indeed , Ġcomputational Ġanalysis Ġis Ġopening Ġnew Ġpossibilities Ġfor Ġexploring Ġchallenging Ġquestions Ġat Ġthe Ġheart Ġof Ġsome Ġof Ġthe Ġmost Ġpressing Ġcontemporary Ġcultural Ġand Ġsocial Ġissues . ĠWhile Ġa Ġhuman Ġreader Ġis Ġbetter Ġequipped Ġto Ġmake Ġlogical Ġinf erences , Ġresolve Ġambig u ities , Ġand Ġapply Ġcultural Ġknowledge Ġthan Ġa Ġcomputer , Ġhuman Ġtime Ġand Ġattention Ġare Ġlimited . ĠMoreover , Ġmany Ġpatterns Ġare Ġnot Ġobvious Ġin Ġany Ġspecific Ġcontext , Ġbut Ġonly Ġstand Ġout Ġin Ġthe Ġaggregate . ĠFor Ġexample , Ġin Ġa Ġlandmark Ġstudy , ĠB IB REF 1 Ġanalyzed Ġthe Ġauthors hip Ġof ĠThe ĠFederal ist ĠPapers Ġusing Ġa Ġstatistical Ġtext Ġanalysis Ġby Ġfocusing Ġon Ġstyle , Ġbased Ġon Ġthe Ġdistribution Ġof Ġfunction Ġwords , Ġrather Ġthan Ġcontent . ĠAs Ġanother Ġexample , ĠB IB REF 2 Ġstudied Ġwhat Ġdefines ĠEnglish Ġha iku Ġand Ġshowed Ġhow Ġcomputational Ġanalysis Ġand Ġclose Ġreading Ġcan Ġcomplement Ġeach Ġother . ĠComput ational Ġapproaches Ġare Ġvaluable Ġprecisely Ġbecause Ġthey Ġhelp Ġus Ġidentify Ġpatterns Ġthat Ġwould Ġnot Ġotherwise Ġbe Ġdiscern ible . </s> Yet Ġthese Ġapproaches Ġare Ġnot Ġa Ġpan acea . ĠExam ining Ġthick Ġsocial Ġand Ġcultural Ġquestions Ġusing Ġcomputational Ġtext Ġanalysis Ġcarries Ġsignificant Ġchallenges . ĠFor Ġone , Ġtexts Ġare Ġculturally Ġand Ġsocially Ġsituated . ĠThey Ġreflect Ġthe Ġideas , Ġvalues Ġand Ġbeliefs Ġof Ġboth Ġtheir Ġauthors Ġand Ġtheir Ġtarget Ġaudiences , Ġand Ġsuch Ġsubt let ies Ġof Ġmeaning Ġand Ġinterpretation Ġare Ġdifficult Ġto Ġincorporate Ġin Ġcomputational Ġapproaches . ĠFor Ġanother , Ġmany Ġof Ġthe Ġsocial Ġand Ġcultural Ġconcepts Ġwe Ġseek Ġto Ġexamine Ġare Ġhighly Ġcontested ĠâĢĶ Ġhate Ġspeech Ġis Ġjust Ġone Ġsuch Ġexample . ĠCho ices Ġregarding Ġhow Ġto Ġoperational ize Ġand Ġanalyze Ġthese Ġconcepts Ġcan Ġraise Ġserious Ġconcerns Ġabout Ġconceptual Ġvalidity Ġand Ġmay Ġlead Ġto Ġshallow Ġor Ġobvious Ġconclusions , Ġrather Ġthan Ġfindings Ġthat Ġreflect Ġthe Ġdepth Ġof Ġthe Ġquestions Ġwe Ġseek Ġto Ġaddress . </s> These Ġare Ġjust Ġa Ġsmall Ġsample Ġof Ġthe Ġmany Ġopportunities Ġand Ġchallenges Ġfaced Ġin Ġcomputational Ġanalyses Ġof Ġtextual Ġdata . ĠNew Ġpossibilities Ġand Ġfrustrating Ġobstacles Ġemerge Ġat Ġevery Ġstage Ġof Ġresearch , Ġfrom Ġidentification Ġof Ġthe Ġresearch Ġquestion Ġto Ġinterpretation Ġof Ġthe Ġresults . ĠIn Ġthis Ġarticle , Ġwe Ġtake Ġthe Ġreader Ġthrough Ġa Ġtypical Ġresearch Ġprocess Ġthat Ġinvolves Ġmeasuring Ġsocial Ġor Ġcultural Ġconcepts Ġusing Ġcomputational Ġmethods , Ġdiscussing Ġboth Ġthe Ġopportunities Ġand Ġcomplications Ġthat Ġoften Ġarise . ĠIn Ġthe ĠReddit Ġcase , Ġfor Ġexample , Ġhate Ġspeech Ġis Ġmeasured , Ġhowever Ġimperfect ly , Ġby Ġthe Ġpresence Ġof Ġparticular Ġwords Ġsemi - aut om atically Ġextracted Ġfrom Ġa Ġmachine Ġlearning Ġalgorithm . ĠOper ational izations Ġare Ġnever Ġperfect Ġtranslations , Ġand Ġare Ġoften Ġrefined Ġover Ġthe Ġcourse Ġof Ġan Ġinvestigation , Ġbut Ġthey Ġare Ġcrucial . </s> We Ġbegin Ġour Ġexploration Ġwith Ġthe Ġidentification Ġof Ġresearch Ġquestions , Ġproceed Ġthrough Ġdata Ġselection , Ġconceptual ization , Ġand Ġoperational ization , Ġand Ġend Ġwith Ġanalysis Ġand Ġthe Ġinterpretation Ġof Ġresults . ĠThe Ġresearch Ġprocess Ġsounds Ġmore Ġor Ġless Ġlinear Ġthis Ġway , Ġbut Ġeach Ġof Ġthese Ġphases Ġoverl aps , Ġand Ġin Ġsome Ġinstances Ġturns Ġback Ġupon Ġitself . ĠThe Ġanalysis Ġphase , Ġfor Ġexample , Ġoften Ġfeeds Ġback Ġinto Ġthe Ġoriginal Ġresearch Ġquestions , Ġwhich Ġmay Ġcontinue Ġto Ġevolve Ġfor Ġmuch Ġof Ġthe Ġproject . ĠAt Ġeach Ġstage , Ġour Ġdiscussion Ġis Ġcritically Ġinformed Ġby Ġinsights Ġfrom Ġthe Ġhumanities Ġand Ġsocial Ġsciences , Ġfields Ġthat Ġhave Ġfocused Ġon , Ġand Ġworked Ġto Ġtackle , Ġthe Ġchallenges Ġof Ġtextual Ġanalysis âĢĶ albeit Ġat Ġsmaller Ġscales âĢĶ since Ġtheir Ġinception . </s> In Ġdescribing Ġour Ġexperiences Ġwith Ġcomputational Ġtext Ġanalysis , Ġwe Ġhope Ġto Ġachieve Ġthree Ġprimary Ġgoals . ĠFirst , Ġwe Ġaim Ġto Ġshed Ġlight Ġon Ġthorn y Ġissues Ġnot Ġalways Ġat Ġthe Ġforefront Ġof Ġdiscussions Ġabout Ġcomputational Ġtext Ġanalysis Ġmethods . ĠSecond , Ġwe Ġhope Ġto Ġprovide Ġa Ġset Ġof Ġbest Ġpractices Ġfor Ġworking Ġwith Ġthick Ġsocial Ġand Ġcultural Ġconcepts . ĠOur Ġguidance Ġis Ġbased Ġon Ġour Ġown Ġexperiences Ġand Ġis Ġtherefore Ġinherently Ġimperfect . ĠStill , Ġgiven Ġour Ġdiversity Ġof Ġdisciplinary Ġbackgrounds Ġand Ġresearch Ġpractices , Ġwe Ġhope Ġto Ġcapture Ġa Ġrange Ġof Ġideas Ġand Ġidentify Ġcommon alities Ġthat Ġwill Ġresonate Ġfor Ġmany . ĠAnd Ġthis Ġleads Ġto Ġour Ġfinal Ġgoal : Ġto Ġhelp Ġpromote Ġinter disciplinary Ġcollaborations . ĠInter disciplinary Ġinsights Ġand Ġpartnerships Ġare Ġessential Ġfor Ġrealizing Ġthe Ġfull Ġpotential Ġof Ġany Ġcomputational Ġtext Ġanalysis Ġthat Ġinvolves Ġsocial Ġand Ġcultural Ġconcepts , Ġand Ġthe Ġmore Ġwe Ġare Ġable Ġto Ġbridge Ġthese Ġdivides , Ġthe Ġmore Ġfruitful Ġwe Ġbelieve Ġour Ġwork Ġwill Ġbe . </s> Research Ġquestions </s> We Ġtypically Ġstart Ġby Ġidentifying Ġthe Ġquestions Ġwe Ġwish Ġto Ġexplore . ĠCan Ġtext Ġanalysis Ġprovide Ġa Ġnew Ġperspective Ġon Ġa ĠâĢ ľ big Ġquestion âĢ Ŀ Ġthat Ġhas Ġbeen Ġattracting Ġinterest Ġfor Ġyears ? ĠOr Ġcan Ġwe Ġraise Ġnew Ġquestions Ġthat Ġhave Ġonly Ġrecently Ġemerged , Ġfor Ġexample Ġabout Ġsocial Ġmedia ? ĠFor Ġsocial Ġscientists Ġworking Ġin Ġcomputational Ġanalysis , Ġthe Ġquestions Ġare Ġoften Ġgrounded Ġin Ġtheory , Ġasking : ĠHow Ġcan Ġwe Ġexplain Ġwhat Ġwe Ġobserve ? ĠThese Ġquestions Ġare Ġalso Ġinfluenced Ġby Ġthe Ġavailability Ġand Ġaccessibility Ġof Ġdata Ġsources . ĠFor Ġexample , Ġthe Ġchoice Ġto Ġwork Ġwith Ġdata Ġfrom Ġa Ġparticular Ġsocial Ġmedia Ġplatform Ġmay Ġbe Ġpartly Ġdetermined Ġby Ġthe Ġfact Ġthat Ġit Ġis Ġfreely Ġavailable , Ġand Ġthis Ġwill Ġin Ġturn Ġshape Ġthe Ġkinds Ġof Ġquestions Ġthat Ġcan Ġbe Ġasked . ĠA Ġkey Ġoutput Ġof Ġthis Ġphase Ġare Ġthe Ġconcepts Ġto Ġmeasure , Ġfor Ġexample : Ġinfluence ; Ġcopying Ġand Ġreproduction ; Ġthe Ġcreation Ġof Ġpatterns Ġof Ġlanguage Ġuse ; Ġhate Ġspeech . ĠComput ational Ġanalysis Ġof Ġtext Ġmotivated Ġby Ġthese Ġquestions Ġis Ġinsight Ġdriven : Ġwe Ġaim Ġto Ġdescribe Ġa Ġphenomenon Ġor Ġexplain Ġhow Ġit Ġcame Ġabout . ĠFor Ġexample , Ġwhat Ġcan Ġwe Ġlearn Ġabout Ġhow Ġand Ġwhy Ġhate Ġspeech Ġis Ġused Ġor Ġhow Ġthis Ġchanges Ġover Ġtime ? ĠIs Ġhate Ġspeech Ġone Ġthing , Ġor Ġdoes Ġit Ġcomprise Ġmultiple Ġforms Ġof Ġexpression ? ĠIs Ġthere Ġa Ġclear Ġboundary Ġbetween Ġhate Ġspeech Ġand Ġother Ġtypes Ġof Ġspeech , Ġand Ġwhat Ġfeatures Ġmake Ġit Ġmore Ġor Ġless Ġambiguous ? ĠIn Ġthese Ġcases , Ġit Ġis Ġcritical Ġto Ġcommunicate Ġhigh - level Ġpatterns Ġin Ġterms Ġthat Ġare Ġrecognizable . </s> This Ġcontrasts Ġwith Ġmuch Ġof Ġthe Ġwork Ġin Ġcomputational Ġtext Ġanalysis , Ġwhich Ġtends Ġto Ġfocus Ġon Ġautom ating Ġtasks Ġthat Ġhumans Ġperform Ġinefficient ly . ĠThese Ġtasks Ġrange Ġfrom Ġcore Ġlingu istically Ġmotivated Ġtasks Ġthat Ġconstitute Ġthe Ġbackbone Ġof Ġnatural Ġlanguage Ġprocessing , Ġsuch Ġas Ġpart - of - speech Ġtagging Ġand Ġparsing , Ġto Ġfiltering Ġspam Ġand Ġdetecting Ġsentiment . ĠMany Ġtasks Ġare Ġmotivated Ġby Ġapplications , Ġfor Ġexample Ġto Ġautomatically Ġblock Ġonline Ġtrolls . ĠSuccess , Ġthen , Ġis Ġoften Ġmeasured Ġby Ġperformance , Ġand Ġcommunicating Ġwhy Ġa Ġcertain Ġprediction Ġwas Ġmade âĢĶ for Ġexample , Ġwhy Ġa Ġdocument Ġwas Ġlabeled Ġas Ġpositive Ġsentiment , Ġor Ġwhy Ġa Ġword Ġwas Ġclassified Ġas Ġa Ġnoun âĢĶ is Ġless Ġimportant Ġthan Ġthe Ġaccuracy Ġof Ġthe Ġprediction Ġitself . ĠThe Ġapproaches Ġwe Ġuse Ġand Ġwhat Ġwe Ġmean Ġby Ġ` success ' Ġare Ġthus Ġguided Ġby Ġour Ġresearch Ġquestions . </s> Domain Ġexperts Ġand Ġfellow Ġresearchers Ġcan Ġprovide Ġfeedback Ġon Ġquestions Ġand Ġhelp Ġwith Ġdynamically Ġrev ising Ġthem . ĠFor Ġexample , Ġthey Ġmay Ġsay ĠâĢ ľ we Ġalready Ġthink Ġwe Ġknow Ġthat âĢ Ŀ , ĠâĢ ľ that 's Ġtoo ĠnaÃ¯ve âĢ Ŀ , ĠâĢ ľ that Ġdoesn 't Ġreflect Ġsocial Ġreality âĢ Ŀ Ġ( negative ); ĠâĢ ľ two Ġmajor Ġcamps Ġin Ġthe Ġfield Ġwould Ġgive Ġdifferent Ġanswers Ġto Ġthat Ġquestion âĢ Ŀ Ġ( neutral ); ĠâĢ ľ we Ġtried Ġto Ġlook Ġat Ġthat Ġback Ġin Ġthe Ġ1960 s , Ġbut Ġwe Ġdidn 't Ġhave Ġthe Ġtechnology âĢ Ŀ Ġ( positive ); Ġand ĠâĢ ľ that Ġsounds Ġlike Ġsomething Ġthat Ġpeople Ġwho Ġmade Ġthat Ġarchive Ġwould Ġlove âĢ Ŀ , ĠâĢ ľ that 's Ġa Ġreally Ġfundamental Ġquestion âĢ Ŀ Ġ( very Ġpositive ). </s> Sometimes Ġwe Ġalso Ġhope Ġto Ġconnect Ġto Ġmultiple Ġdisciplines . ĠFor Ġexample , Ġwhile Ġfocusing Ġon Ġthe Ġhuman istic Ġconcerns Ġof Ġan Ġarchive , Ġwe Ġcould Ġalso Ġask Ġsocial Ġquestions Ġsuch Ġas ĠâĢ ľ is Ġthis Ġarchive Ġmore Ġabout Ġcollaborative Ġprocesses , Ġculture - building Ġor Ġnorm Ġcreation ? âĢ Ŀ Ġor ĠâĢ ľ how Ġwell Ġdoes Ġthis Ġarchive Ġreflect Ġthe Ġsociety Ġin Ġwhich Ġit Ġis Ġembedded ?" ĠB IB REF 3 Ġused Ġquantitative Ġmethods Ġto Ġtell Ġa Ġstory Ġabout ĠDarwin 's Ġintellectual Ġdevelopment âĢĶ an Ġessential Ġbi ographical Ġquestion Ġfor Ġa Ġkey Ġfigure Ġin Ġthe Ġhistory Ġof Ġscience . ĠAt Ġthe Ġsame Ġtime , Ġtheir Ġmethods Ġconnected ĠDarwin 's Ġdevelopment Ġto Ġthe Ġchanging Ġlandscape Ġof ĠVictorian Ġscientific Ġculture , Ġallowing Ġthem Ġto Ġcontrast ĠDarwin 's ĠâĢ ľ for aging âĢ Ŀ Ġin Ġthe Ġscientific Ġliterature Ġof Ġhis Ġtime Ġto Ġthe Ġways Ġin Ġwhich Ġthat Ġliterature Ġwas Ġitself Ġproduced . ĠFinally , Ġtheir Ġmethods Ġprovided Ġa Ġcase Ġstudy , Ġand Ġvalidation Ġof Ġtechnical Ġapproaches , Ġfor Ġcognitive Ġscientists Ġwho Ġare Ġinterested Ġin Ġhow Ġpeople Ġexplore Ġand Ġexploit Ġsources Ġof Ġknowledge . </s> Questions Ġabout Ġpotential ĠâĢ ľ du al Ġuse âĢ Ŀ Ġmay Ġalso Ġarise . ĠReturning Ġto Ġour Ġintroductory Ġexample , ĠB IB REF 0 Ġstarted Ġwith Ġa Ġde cept ively Ġsimple Ġquestion : Ġif Ġan Ġinternet Ġplatform Ġeliminates Ġforums Ġfor Ġhate Ġspeech , Ġdoes Ġthis Ġimpact Ġhate Ġspeech Ġin Ġother Ġforums ? ĠThe Ġresearch Ġwas Ġmotivated Ġby Ġthe Ġbelief Ġthat Ġa Ġrising Ġtide Ġof Ġonline Ġhate Ġspeech Ġwas Ġ( and Ġis ) Ġmaking Ġthe Ġinternet Ġincreasingly Ġunf riend ly Ġfor Ġdis em powered Ġgroups , Ġincluding Ġminorities , Ġwomen , Ġand ĠLB GT Q Ġindividuals . ĠYet Ġthe Ġpossibility Ġof Ġdual Ġuse Ġtroubled Ġthe Ġresearchers Ġfrom Ġthe Ġonset . ĠCould Ġthe Ġmethodology Ġbe Ġadopted Ġto Ġtarget Ġthe Ġspeech Ġof Ġgroups Ġlike ĠBlack ĠLives ĠMatter ? ĠCould Ġit Ġbe Ġadopted Ġby Ġrepressive Ġgovernments Ġto Ġminimize Ġonline Ġdissent ? ĠWhile Ġthese Ġconcerns Ġremained , Ġthey Ġconcluded Ġthat Ġhypothetical Ġdual Ġuse Ġscenarios Ġdid Ġnot Ġoutweigh Ġthe Ġtangible Ġcontribution Ġthis Ġresearch Ġcould Ġoffer Ġtowards Ġmaking Ġthe Ġonline Ġenvironment Ġmore Ġequal Ġand Ġjust . </s> Data </s> The Ġnext Ġstep Ġinvolves Ġdeciding Ġon Ġthe Ġdata Ġsources , Ġcollecting Ġand Ġcompiling Ġthe Ġdataset , Ġand Ġinspecting Ġits Ġmetadata . </s> Data Ġacquisition </s> Many Ġscholars Ġin Ġthe Ġhumanities Ġand Ġthe Ġsocial Ġsciences Ġwork Ġwith Ġsources Ġthat Ġare Ġnot Ġavailable Ġin Ġdigital Ġform , Ġand Ġindeed Ġmay Ġnever Ġbe Ġdigit ized . ĠOthers Ġwork Ġwith Ġboth Ġanalogue Ġand Ġdigit ized Ġmaterials , Ġand Ġthe Ġincreasing Ġdigit ization Ġof Ġarchives Ġhas Ġopened Ġopportunities Ġto Ġstudy Ġthese Ġarchives Ġin Ġnew Ġways . ĠWe Ġcan Ġgo Ġto Ġthe Ġcanonical Ġarchive Ġor Ġopen Ġup Ġsomething Ġthat Ġnobody Ġhas Ġstudied Ġbefore . ĠFor Ġexample , Ġwe Ġmight Ġfocus Ġon Ġmajor Ġhistorical Ġmoments Ġ( French ĠRevolution , Ġpost - Mil ose vic ĠSerbia ) Ġor Ġcritical Ġepoch s Ġ( Britain Ġentering Ġthe ĠVictorian Ġera , Ġthe Ġtransition Ġfrom ĠLatin Ġto Ġproto - Rom ance ). ĠOr , Ġwe Ġcould Ġlook Ġfor Ġrecords Ġof Ġhow Ġpeople Ġconducted Ġscience , Ġwrote Ġand Ġconsumed Ġliterature , Ġand Ġworked Ġout Ġtheir Ġphilosophies . </s> A Ġgrowing Ġnumber Ġof Ġresearchers Ġwork Ġwith Ġborn - digital Ġsources Ġor Ġdata . ĠBorn - digital Ġdata , Ġe . g ., Ġfrom Ġsocial Ġmedia , Ġgenerally Ġdo Ġnot Ġinvolve Ġdirect Ġelic itation Ġfrom Ġparticipants Ġand Ġtherefore Ġenable Ġunob tr usive Ġmeasurements ĠB IB REF 5 Ġ, ĠB IB REF 6 Ġ. ĠIn Ġcontrast , Ġmethods Ġlike Ġsurveys Ġsometimes Ġelicit Ġaltered Ġresponses Ġfrom Ġparticipants , Ġwho Ġmight Ġadapt Ġtheir Ġresponses Ġto Ġwhat Ġthey Ġthink Ġis Ġexpected . ĠMoreover , Ġborn - digital Ġdata Ġis Ġoften Ġmassive , Ġenabling Ġlarge - scale Ġstudies Ġof Ġlanguage Ġand Ġbehavior Ġin Ġa Ġvariety Ġof Ġsocial Ġcontexts . </s> Still , Ġmany Ġscholars Ġin Ġthe Ġsocial Ġsciences Ġand Ġhumanities Ġwork Ġwith Ġmultiple Ġdata Ġsources . ĠThe Ġvariety Ġof Ġsources Ġtypically Ġused Ġmeans Ġthat Ġmore Ġthan Ġone Ġdata Ġcollection Ġmethod Ġis Ġoften Ġrequired . ĠFor Ġexample , Ġa Ġproject Ġexamining Ġcoverage Ġof Ġa ĠUK ĠGeneral ĠElection , Ġcould Ġdraw Ġdata Ġfrom Ġtraditional Ġmedia , Ġweb Ġarchives , ĠTwitter Ġand ĠFacebook , Ġcampaign Ġmanifest os , Ġetc . Ġand Ġmight Ġcombine Ġtextual Ġanalysis Ġof Ġthese Ġmaterials Ġwith Ġsurveys , Ġlaboratory Ġexperiments , Ġor Ġfield Ġobservations Ġoffline . ĠIn Ġcontrast , Ġmany Ġcomputational Ġstudies Ġbased Ġon Ġborn - digital Ġdata Ġhave Ġfocused Ġon Ġone Ġspecific Ġsource , Ġsuch Ġas ĠTwitter . </s> The Ġuse Ġof Ġborn - digital Ġdata Ġraises Ġethical Ġconcerns . ĠAlthough Ġearly Ġstudies Ġoften Ġtreated Ġprivacy Ġas Ġa Ġbinary Ġconstruct , Ġmany Ġnow Ġacknowledge Ġits Ġcomplexity ĠB IB REF 7 Ġ. ĠConvers ations Ġon Ġprivate Ġmatters Ġcan Ġbe Ġposted Ġonline , Ġvisible Ġfor Ġall , Ġbut Ġsocial Ġnorms Ġregarding Ġwhat Ġshould Ġbe Ġconsidered Ġpublic Ġinformation Ġmay Ġdiffer Ġfrom Ġthe Ġdata 's Ġexplicit Ġvisibility Ġsettings . ĠOften Ġno Ġinformed Ġconsent Ġhas Ġbeen Ġobtained , Ġraising Ġconcerns Ġand Ġchallenges Ġregarding Ġpublishing Ġcontent Ġand Ġpotentially Ġharmful Ġsecondary Ġuses ĠB IB REF 8 Ġ, ĠB IB REF 4 Ġ. </s> Recently , Ġconcerns Ġabout Ġpotential Ġharms Ġstemming Ġfrom Ġsecondary Ġuses Ġhave Ġled Ġa Ġnumber Ġof Ġdigital Ġservice Ġproviders Ġto Ġrestrict Ġaccess Ġto Ġborn - digital Ġdata . ĠFacebook Ġand ĠTwitter , Ġfor Ġexample , Ġhave Ġreduced Ġor Ġeliminated Ġpublic Ġaccess Ġto Ġtheir Ġapplication Ġprogramming Ġinterfaces Ġ( AP Is ) Ġand Ġexpressed Ġhesitation Ġabout Ġallowing Ġacademic Ġresearchers Ġto Ġuse Ġdata Ġfrom Ġtheir Ġplatforms Ġto Ġexamine Ġcertain Ġsensitive Ġor Ġcontroversial Ġtopics . ĠDespite Ġthe Ġseeming Ġabundance Ġof Ġborn - digital Ġdata , Ġwe Ġtherefore Ġcannot Ġtake Ġits Ġavailability Ġfor Ġgranted . </s> Working Ġwith Ġdata Ġthat Ġsomeone Ġelse Ġhas Ġacquired Ġpresents Ġadditional Ġproblems Ġrelated Ġto Ġproven ance Ġand Ġcontextual isation . ĠIt Ġmay Ġnot Ġalways Ġbe Ġpossible Ġto Ġdetermine Ġthe Ġcriteria Ġapplied Ġduring Ġthe Ġcreation Ġprocess . ĠFor Ġexample , Ġwhy Ġwere Ġcertain Ġnewspapers Ġdigit ized Ġbut Ġnot Ġothers , Ġand Ġwhat Ġdoes Ġthis Ġsay Ġabout Ġthe Ġcollection ? ĠSimilar Ġquestions Ġarise Ġwith Ġthe Ġuse Ġof Ġborn - digital Ġdata . ĠFor Ġinstance , Ġwhen Ġusing Ġthe ĠInternet ĠArchive âĢ Ļ s ĠWay back ĠMachine Ġto Ġgather Ġdata Ġfrom Ġarchived Ġweb Ġpages , Ġwe Ġneed Ġto Ġconsider Ġwhat Ġpages Ġwere Ġcaptured , Ġwhich Ġare Ġlikely Ġmissing , Ġand Ġwhy . </s> We Ġmust Ġoften Ġrep ur pose Ġborn - digital Ġdata Ġ( e . g ., ĠTwitter Ġwas Ġnot Ġdesigned Ġto Ġmeasure Ġpublic Ġopinion ), Ġbut Ġdata Ġbiases Ġmay Ġlead Ġto Ġspurious Ġresults Ġand Ġlimit Ġjustification Ġfor Ġgeneral ization . ĠIn Ġparticular , Ġdata Ġcollected Ġvia Ġblack Ġbox ĠAPIs Ġdesigned Ġfor Ġcommercial , Ġnot Ġresearch , Ġpurposes Ġare Ġlikely Ġto Ġintroduce Ġbiases Ġinto Ġthe Ġinf erences Ġwe Ġdraw , Ġand Ġthe Ġclosed Ġnature Ġof Ġthese ĠAPIs Ġmeans Ġwe Ġrarely Ġknow Ġwhat Ġbiases Ġare Ġintroduced , Ġlet Ġalone Ġhow Ġseverely Ġthey Ġmight Ġimpact Ġour Ġresearch ĠB IB REF 10 Ġ. ĠThese , Ġhowever , Ġare Ġnot Ġnew Ġproblems . ĠHist orians , Ġfor Ġexample , Ġhave Ġalways Ġunderstood Ġthat Ġtheir Ġsources Ġwere Ġproduced Ġwithin Ġparticular Ġcontexts Ġand Ġfor Ġparticular Ġpurposes , Ġwhich Ġare Ġnot Ġalways Ġapparent Ġto Ġus . </s> Non - represent ative Ġdata Ġcan Ġstill Ġbe Ġuseful Ġfor Ġmaking Ġcomparisons Ġwithin Ġa Ġsample . ĠIn Ġthe Ġintroductory Ġexample Ġon Ġhate Ġspeech ĠB IB REF 0 Ġ, Ġthe ĠReddit Ġforums Ġdo Ġnot Ġpresent Ġa Ġcomprehensive Ġor Ġbalanced Ġpicture Ġof Ġhate Ġspeech : Ġthe Ġwriting Ġis Ġalmost Ġexclusively Ġin ĠEnglish , Ġthe Ġtargets Ġof Ġhate Ġspeech Ġare Ġmainly Ġrestricted Ġ( e . g ., Ġto Ġblack Ġpeople , Ġor Ġwomen ), Ġand Ġthe Ġpopulation Ġof Ġwriters Ġis Ġshaped Ġby ĠReddit 's Ġdemographics , Ġwhich Ġskew Ġtowards Ġyoung Ġwhite Ġmen . ĠThese Ġbiases Ġlimit Ġthe Ġgeneral iz ability Ġof Ġthe Ġfindings , Ġwhich Ġcannot Ġbe Ġextrap olated Ġto Ġother Ġlanguages , Ġother Ġtypes Ġof Ġhate Ġspeech , Ġand Ġother Ġdemographic Ġgroups . ĠHowever , Ġbecause Ġthe Ġfindings Ġare Ġbased Ġon Ġmeasurements Ġon Ġthe Ġsame Ġsort Ġof Ġhate Ġspeech Ġand Ġthe Ġsame Ġpopulation Ġof Ġwriters , Ġas Ġlong Ġas Ġthe Ġcollected Ġdata Ġare Ġrepresentative Ġof Ġthis Ġspecific Ġpopulation , Ġthese Ġbiases Ġdo Ġnot Ġpose Ġan Ġint ract able Ġvalidity Ġproblem Ġif Ġclaims Ġare Ġproperly Ġrestricted . </s> The Ġsize Ġof Ġmany Ġnewly Ġavailable Ġdatasets Ġis Ġone Ġof Ġtheir Ġmost Ġappealing Ġcharacteristics . ĠBig ger Ġdatasets Ġoften Ġmake Ġstatistics Ġmore Ġrobust . ĠThe Ġsize Ġneeded Ġfor Ġa Ġcomputational Ġtext Ġanalysis Ġdepends Ġon Ġthe Ġresearch Ġgoal : ĠWhen Ġit Ġinvolves Ġstudying Ġrare Ġevents , Ġbigger Ġdatasets Ġare Ġneeded . ĠHowever , Ġlarger Ġis Ġnot Ġalways Ġbetter . ĠSome Ġvery Ġlarge Ġarchives Ġare ĠâĢ ľ secret ly âĢ Ŀ Ġcollections Ġof Ġmultiple Ġand Ġdistinct Ġprocesses Ġthat Ġno Ġin - field Ġscholar Ġwould Ġconsider Ġrelated . ĠFor Ġexample , ĠGoogle ĠBooks Ġis Ġfrequently Ġused Ġto Ġstudy Ġcultural Ġpatterns , Ġbut Ġthe Ġover - represent ation Ġof Ġscientific Ġarticles Ġin ĠGoogle Ġbooks Ġcan Ġbe Ġproblematic ĠB IB REF 11 Ġ. ĠEven Ġvery Ġlarge Ġborn - digital Ġdatasets Ġusually Ġcover Ġlimited Ġtimes p ans Ġcompared Ġto , Ġe . g ., Ġthe ĠGutenberg Ġarchive Ġof ĠBritish Ġnovels . </s> This Ġstage Ġof Ġthe Ġresearch Ġalso Ġraises Ġimportant Ġquestions Ġabout Ġfairness . ĠAre Ġmarginalized Ġgroups , Ġfor Ġexample , Ġrepresented Ġin Ġthe Ġtweets Ġwe Ġhave Ġcollected ? ĠIf Ġnot , Ġwhat Ġtypes Ġof Ġbiases Ġmight Ġresult Ġfrom Ġanalyses Ġrelying Ġon Ġthose Ġtweets ? </s> Local Ġexperts Ġand ĠâĢ ľ in form ants âĢ Ŀ Ġcan Ġhelp Ġnavigate Ġthe Ġdata . ĠThey Ġcan Ġhelp Ġunderstand Ġthe Ġrole Ġan Ġarchive Ġplays Ġin Ġthe Ġtime Ġand Ġplace . ĠThey Ġmight Ġtell Ġus : ĠIs Ġthis Ġthe Ġcentral Ġarchive , Ġor Ġa Ġperipheral Ġone ? ĠWhat Ġmakes Ġit Ġunusual ? ĠOr Ġthey Ġmight Ġtell Ġus Ġhow Ġcertain Ġunder represented Ġcommunities Ġuse Ġa Ġsocial Ġmedia Ġplatform Ġand Ġadvise Ġus Ġon Ġstrategies Ġfor Ġensuring Ġour Ġdata Ġcollection Ġincludes Ġtheir Ġperspectives . </s> However , Ġwhen Ġit Ġis Ġpractically Ġinf eas ible Ġto Ġnavigate Ġthe Ġdata Ġin Ġthis Ġway âĢĶ for Ġinstance , Ġwhen Ġwe Ġcannot Ġdetermine Ġwhat Ġis Ġmissing Ġfrom ĠTwitter 's ĠStreaming ĠAPI Ġor Ġwhat Ġweb pages Ġare Ġleft Ġout Ġof Ġthe ĠInternet ĠArchive âĢĶ we Ġshould Ġbe Ġopen Ġabout Ġthe Ġlimitations Ġof Ġour Ġanalyses , Ġacknowledging Ġthe Ġflaws Ġin Ġour Ġdata Ġand Ġdrawing Ġcautious Ġand Ġreasonable Ġconclusions Ġfrom Ġthem . ĠIn Ġall Ġcases , Ġwe Ġshould Ġreport Ġthe Ġchoices Ġwe Ġhave Ġmade Ġwhen Ġcreating Ġor Ġre - using Ġany Ġdataset . </s> Comp iling Ġdata </s> After Ġidentifying Ġthe Ġdata Ġsource ( s ), Ġthe Ġnext Ġstep Ġis Ġcompiling Ġthe Ġdata . ĠThis Ġstep Ġis Ġfundamental : Ġif Ġthe Ġsources Ġcannot Ġsupport Ġa Ġconvincing Ġresult , Ġno Ġresult Ġwill Ġbe Ġconvincing . ĠIn Ġmany Ġcases , Ġthis Ġinvolves Ġdefining Ġa ĠâĢ ľ core " Ġset Ġof Ġdocuments Ġand Ġa ĠâĢ ľ com par ison " Ġset . ĠWe Ġoften Ġhave Ġa Ġspecific Ġset Ġof Ġdocuments Ġin Ġmind : Ġan Ġauthor 's Ġwork , Ġa Ġparticular Ġjournal , Ġa Ġtime Ġperiod . ĠBut Ġif Ġwe Ġwant Ġto Ġsay Ġthat Ġthis ĠâĢ ľ core " Ġset Ġhas Ġsome Ġdistinctive Ġproperty , Ġwe Ġneed Ġa ĠâĢ ľ com par ison " Ġset . ĠExp anding Ġthe Ġcollection Ġbeyond Ġthe Ġdocuments Ġthat Ġwe Ġwould Ġimmediately Ġthink Ġof Ġhas Ġthe Ġbeneficial Ġeffect Ġof Ġincreasing Ġour Ġsample Ġsize . ĠHaving Ġmore Ġsources Ġincreases Ġthe Ġchance Ġthat Ġwe Ġwill Ġnotice Ġsomething Ġconsistent Ġacross Ġmany Ġindividually Ġvarying Ġcontexts . </s> Comp aring Ġsets Ġof Ġdocuments Ġcan Ġsometimes Ġsupport Ġcausal Ġinference , Ġpresented Ġas Ġa Ġcontrast Ġbetween Ġa Ġtreatment Ġgroup Ġand Ġa Ġcontrol . ĠIn ĠB IB REF 0 Ġ, Ġthe Ġtreatment Ġconsisted Ġof Ġthe Ġtext Ġwritten Ġin Ġthe Ġtwo Ġforums Ġthat Ġwere Ġeventually Ġclosed Ġby ĠReddit . ĠHowever , Ġidentifying Ġa Ġcontrol Ġgroup Ġrequired Ġa Ġconsiderable Ġamount Ġof Ġtime Ġand Ġeffort . ĠReddit Ġis Ġa Ġdiverse Ġplatform , Ġwith Ġa Ġwide Ġvariety Ġof Ġinteract ional Ġand Ġlinguistic Ġstyles ; Ġit Ġwould Ġbe Ġpointless Ġto Ġcompare Ġhate Ġspeech Ġforums Ġagainst Ġforums Ġdedicated Ġto , Ġsay , Ġpictures Ġof Ġwrecked Ġbicycles . ĠChand rase k har an Ġet Ġal . Ġused Ġa Ġmatching Ġdesign , Ġpop ulating Ġthe Ġcontrol Ġgroup Ġwith Ġforums Ġthat Ġwere Ġas Ġsimilar Ġas Ġpossible Ġto Ġthe Ġtreatment Ġgroup , Ġbut Ġwere Ġnot Ġbanned Ġfrom ĠReddit . ĠThe Ġgoal Ġis Ġto Ġestimate Ġthe Ġcounter fact ual Ġscenario : Ġin Ġthis Ġcase , Ġwhat Ġwould Ġhave Ġhappened Ġhad Ġthe Ġsite Ġnot Ġtaken Ġaction Ġagainst Ġthese Ġspecific Ġforums ? ĠAn Ġideal Ġcontrol Ġwould Ġmake Ġit Ġpossible Ġto Ġdistinguish Ġthe Ġeffect Ġof Ġthe Ġtreatment ĠâĢĶ Ġclosing Ġthe Ġforums ĠâĢĶ Ġfrom Ġother Ġidiosyncr atic Ġproperties Ġof Ġtexts Ġthat Ġwere Ġtreated . </s> We Ġalso Ġlook Ġfor Ġcategories Ġof Ġdocuments Ġthat Ġmight Ġnot Ġbe Ġuseful . ĠWe Ġmight Ġremove Ġdocuments Ġthat Ġare Ġmeta - disc ourse , Ġlike Ġintrodu ctions Ġand Ġnotes , Ġor Ġdocuments Ġthat Ġare Ġin Ġa Ġlanguage Ġthat Ġis Ġnot Ġthe Ġprimary Ġlanguage Ġof Ġthe Ġcollection , Ġor Ġdupl icates Ġwhen Ġwe Ġare Ġworking Ġwith Ġarchived Ġweb Ġpages . ĠHowever , Ġwe Ġneed Ġto Ġcarefully Ġconsider Ġthe Ġpotential Ġconsequences Ġof Ġinformation Ġwe Ġremove . ĠDoes Ġits Ġremoval Ġalter Ġthe Ġdata , Ġor Ġthe Ġinterpretation Ġof Ġthe Ġdata , Ġwe Ġare Ġanalyzing ? ĠAre Ġwe Ġlosing Ġanything Ġthat Ġmight Ġbe Ġvaluable Ġat Ġa Ġlater Ġstage ? </s> Lab els Ġand Ġmetadata </s> Sometimes Ġall Ġwe Ġhave Ġis Ġdocuments , Ġbut Ġoften Ġwe Ġwant Ġto Ġlook Ġat Ġdocuments Ġin Ġthe Ġcontext Ġof Ġsome Ġadditional Ġinformation , Ġor Ġmetadata . ĠThis Ġadditional Ġinformation Ġcould Ġtell Ġus Ġabout Ġthe Ġcreation Ġof Ġdocuments Ġ( date , Ġauthor , Ġforum ), Ġor Ġabout Ġthe Ġreception Ġof Ġdocuments Ġ( fl agged Ġas Ġhate Ġspeech , Ġhelpful Ġreview ). ĠInformation Ġabout Ġtext Ġsegments Ġcan Ġbe Ġextremely Ġvaluable , Ġbut Ġit Ġis Ġalso Ġprone Ġto Ġerrors , Ġinconsistencies , Ġbias , Ġand Ġmissing Ġinformation . ĠExam ining Ġmetadata Ġis Ġa Ġgood Ġway Ġto Ġcheck Ġa Ġcollection 's Ġbalance Ġand Ġrepresent at iveness . ĠAre Ġsources Ġdisproportionately Ġof Ġone Ġform ? ĠIs Ġthe Ġcollection Ġmissing Ġa Ġspecific Ġtime Ġwindow ? ĠThis Ġtype Ġof Ġcur ation Ġcan Ġbe Ġextremely Ġtime Ġconsuming Ġas Ġit Ġmay Ġrequire Ġexpert Ġlabeling , Ġbut Ġit Ġoften Ġleads Ġto Ġthe Ġmost Ġcompelling Ġresults . ĠSometimes Ġmetadata Ġare Ġalso Ġused Ġas Ġtarget Ġlabels Ġto Ġdevelop Ġmachine Ġlearning Ġmodels . ĠBut Ġusing Ġthem Ġas Ġa ĠâĢ ľ ground Ġtruth âĢ Ŀ Ġrequires Ġcaution . ĠLab els Ġsometimes Ġmean Ġsomething Ġdifferent Ġthan Ġwe Ġexpect . ĠFor Ġexample , Ġa Ġdown Ġvote Ġfor Ġa Ġsocial Ġmedia Ġpost Ġcould Ġindicate Ġthat Ġthe Ġcontent Ġis Ġoffensive , Ġor Ġthat Ġthe Ġvoter Ġsimply Ġdisagreed Ġwith Ġthe Ġexpressed Ġview . </s> Con cept ual ization </s> A Ġcore Ġstep Ġin Ġmany Ġanalyses Ġis Ġtranslating Ġsocial Ġand Ġcultural Ġconcepts Ġ( such Ġas Ġhate Ġspeech , Ġrumor , Ġor Ġconversion ) Ġinto Ġmeasurable Ġquantities . ĠBefore Ġwe Ġcan Ġdevelop Ġmeasurements Ġfor Ġthese Ġconcepts Ġ( the Ġoperational ization Ġstep , Ġor Ġthe ĠâĢ ľ im plementation âĢ Ŀ Ġstep Ġas Ġden oted Ġby ĠB IB REF 12 Ġ), Ġwe Ġneed Ġto Ġdefine Ġthem . ĠIn Ġthe Ġconceptual ization Ġphase Ġwe Ġoften Ġstart Ġwith Ġquestions Ġsuch Ġas : Ġwho Ġare Ġthe Ġdomain Ġexperts , Ġand Ġhow Ġhave Ġthey Ġapproached Ġthe Ġtopic ? ĠWe Ġare Ġlooking Ġfor Ġa Ġdefinition Ġof Ġthe Ġconcept Ġthat Ġis Ġflexible Ġenough Ġto Ġapply Ġon Ġour Ġdataset , Ġyet Ġformal Ġenough Ġfor Ġcomputational Ġresearch . ĠFor Ġexample , Ġour Ġintroductory Ġstudy Ġon Ġhate Ġspeech ĠB IB REF 0 Ġused Ġa Ġstatement Ġon Ġhate Ġspeech Ġproduced Ġby Ġthe ĠEuropean ĠUnion ĠCourt Ġof ĠHuman ĠRights . ĠThe Ġgoal Ġwas Ġnot Ġto Ġimplement Ġthis Ġdefinition Ġdirectly Ġin Ġsoftware Ġbut Ġto Ġuse Ġit Ġas Ġa Ġreference Ġpoint Ġto Ġanchor Ġsubsequent Ġanalyses . </s> If Ġwe Ġwant Ġto Ġmove Ġbeyond Ġthe Ġuse Ġof Ġad Ġhoc Ġdefinitions , Ġit Ġcan Ġbe Ġuseful Ġto Ġdistinguish Ġbetween Ġwhat Ġpolitical Ġscientists ĠAd cock Ġand ĠColl ier Ġcall Ġthe ĠâĢ ľ background Ġconcept âĢ Ŀ Ġand Ġthe ĠâĢ ľ system at ized Ġconcept âĢ Ŀ ĠB IB REF 13 Ġ. ĠThe Ġbackground Ġconcept Ġcomprises Ġthe Ġfull Ġand Ġdiverse Ġset Ġof Ġmeanings Ġthat Ġmight Ġbe Ġassociated Ġwith Ġa Ġparticular Ġterm . ĠThis Ġinvolves Ġdel ving Ġinto Ġtheoretical , Ġconceptual , Ġand Ġempirical Ġstudies Ġto Ġassess Ġhow Ġa Ġconcept Ġhas Ġbeen Ġdefined Ġby Ġother Ġscholars Ġand , Ġmost Ġimportantly , Ġto Ġdetermine Ġwhich Ġdefinition Ġis Ġmost Ġappropriate Ġfor Ġthe Ġparticular Ġresearch Ġquestion Ġand Ġthe Ġtheoretical Ġframework Ġin Ġwhich Ġit Ġis Ġsituated . ĠThat Ġdefinition , Ġin Ġturn , Ġrepresents Ġthe Ġsystem at ized Ġconcept : Ġthe Ġformulation Ġthat Ġis Ġadopted Ġfor Ġthe Ġstudy . </s> It Ġis Ġimportant Ġto Ġconsider Ġthat Ġfor Ġsocial Ġand Ġcultural Ġconcepts Ġthere Ġis Ġno Ġabsolute Ġground Ġtruth . ĠThere Ġare Ġoften Ġmultiple Ġvalid Ġdefinitions Ġfor Ġa Ġconcept Ġ( the ĠâĢ ľ background âĢ Ŀ Ġconcept Ġin Ġthe Ġterms Ġof ĠAd cock Ġand ĠColl ier ), Ġand Ġdefinitions Ġmight Ġbe Ġcontested Ġover Ġtime . ĠThis Ġmay Ġbe Ġuncomfortable Ġfor Ġcomputer Ġscientists , Ġwhose Ġprimary Ġmeasure Ġof Ġsuccess Ġis Ġoften Ġbased Ġon Ġcomparing Ġa Ġmodel 's Ġoutput Ġagainst ĠâĢ ľ ground Ġtruth âĢ Ŀ Ġor Ġa ĠâĢ ľ gold Ġstandard âĢ Ŀ , Ġe . g ., Ġby Ġcomparing Ġa Ġsentiment Ġclass ifier 's Ġoutput Ġagainst Ġmanual Ġannotations . ĠHowever , Ġthe Ġnotion Ġof Ġground Ġtruth Ġis Ġuncommon Ġin Ġthe Ġhumanities Ġand Ġthe Ġsocial Ġsciences Ġand Ġit Ġis Ġoften Ġtaken Ġtoo Ġfar Ġin Ġmachine Ġlearning . ĠB IB REF 14 Ġnotes Ġthat Ġin Ġliterary Ġcriticism Ġand Ġthe Ġdigital Ġhumanities Ġmore Ġbroadly ĠâĢ ľ interpret ation , Ġambiguity , Ġand Ġargument ation Ġare Ġprized Ġfar Ġabove Ġground Ġtruth Ġand Ġdefinitive Ġconclusions ". ĠB IB REF 15 Ġdraw Ġattention Ġto Ġthe Ġdifferent Ġattitudes Ġof Ġliterary Ġscholars Ġand Ġcomputational Ġlingu ists Ġtowards Ġambiguity , Ġstating Ġthat ĠâĢ ľ In ĠComput ational ĠL ingu istics Ġ[ .. ] Ġambiguity Ġis Ġalmost Ġuniformly Ġtreated Ġas Ġa Ġproblem Ġto Ġbe Ġsolved ; Ġthe Ġfocus Ġis Ġon Ġdis amb ig uation , Ġwith Ġthe Ġassumption Ġthat Ġone Ġtrue , Ġcorrect Ġinterpretation Ġexists ." ĠThe Ġlatter Ġis Ġprobably Ġtrue Ġfor Ġtasks Ġsuch Ġas Ġspam Ġfiltering , Ġbut Ġin Ġthe Ġsocial Ġsciences Ġand Ġthe Ġhumanities Ġmany Ġrelevant Ġconcepts Ġare Ġfundamentally Ġunob serv able , Ġsuch Ġas Ġlatent Ġtraits Ġof Ġpolitical Ġactors ĠB IB REF 16 Ġor Ġcultural Ġfit Ġin Ġorganizations ĠB IB REF 17 Ġ, Ġleading Ġto Ġvalidation Ġchallenges . ĠMoreover , Ġwhen Ġthe Ġground Ġtruth Ġcomes Ġfrom Ġpeople , Ġit Ġmay Ġbe Ġinfluenced Ġby Ġideological Ġpri ors , Ġprim ing , Ġsimple Ġdifferences Ġof Ġopinion Ġor Ġperspective , Ġand Ġmany Ġother Ġfactors ĠB IB REF 18 Ġ. ĠWe Ġreturn Ġto Ġthis Ġissue Ġin Ġour Ġdiscussions Ġon Ġvalidation Ġand Ġanalysis . </s> Oper ational ization </s> In Ġthis Ġphase Ġwe Ġdevelop Ġmeasures Ġ( or , ĠâĢ ľ oper ational izations âĢ Ŀ , Ġor ĠâĢ ľ ind icators âĢ Ŀ ) Ġfor Ġthe Ġconcepts Ġof Ġinterest , Ġa Ġprocess Ġcalled ĠâĢ ľ oper ational ization âĢ Ŀ . ĠRegardless Ġof Ġwhether Ġwe Ġare Ġworking Ġwith Ġcomputers , Ġthe Ġoutput Ġproduced Ġcoincides Ġwith ĠAd cock Ġand ĠColl ier 's ĠâĢ ľ sc ores âĢ Ŀ âĢĶ the Ġconcrete Ġtranslation Ġand Ġoutput Ġof Ġthe Ġsystem at ized Ġconcept Ġinto Ġnumbers Ġor Ġlabels ĠB IB REF 13 Ġ. ĠCho ices Ġmade Ġduring Ġthis Ġphase Ġare Ġalways Ġtied Ġto Ġthe Ġquestion ĠâĢ ľ Are Ġwe Ġmeasuring Ġwhat Ġwe Ġintend Ġto Ġmeasure ? âĢ Ŀ ĠDoes Ġour Ġoperational ization Ġmatch Ġour Ġconceptual Ġdefinition ? ĠTo Ġensure Ġvalidity Ġwe Ġmust Ġrecognize Ġgaps Ġbetween Ġwhat Ġis Ġimportant Ġand Ġwhat Ġis Ġeasy Ġto Ġmeasure . ĠWe Ġfirst Ġdiscuss Ġmodeling Ġconsiderations . ĠNext , Ġwe Ġdescribe Ġseveral Ġfrequently Ġused Ġcomputational Ġapproaches Ġand Ġtheir Ġlimitations Ġand Ġstrengths . </s> Mod eling Ġconsiderations </s> The Ġvariables Ġ( both Ġpredict ors Ġand Ġoutcomes ) Ġare Ġrarely Ġsimply Ġbinary Ġor Ġcateg orical . ĠFor Ġexample , Ġa Ġstudy Ġon Ġlanguage Ġuse Ġand Ġage Ġcould Ġfocus Ġon Ġchronological Ġage Ġ( instead Ġof , Ġe . g ., Ġsocial Ġage ĠB IB REF 19 Ġ). ĠHowever , Ġeven Ġthen , Ġage Ġcan Ġbe Ġmodeled Ġin Ġdifferent Ġways . ĠDisc ret ization Ġcan Ġmake Ġthe Ġmodeling Ġeasier Ġand Ġvarious ĠN LP Ġstudies Ġhave Ġmodeled Ġage Ġas Ġa Ġcateg orical Ġvariable ĠB IB REF 20 Ġ. ĠBut Ġany Ġdisc ret ization Ġraises Ġquestions : ĠHow Ġmany Ġcategories ? ĠWhere Ġto Ġplace Ġthe Ġboundaries ? ĠFine Ġdistinctions Ġmight Ġnot Ġalways Ġbe Ġmeaningful Ġfor Ġthe Ġanalysis Ġwe Ġare Ġinterested Ġin , Ġbut Ġcategories Ġthat Ġare Ġtoo Ġbroad Ġcan Ġthreaten Ġvalidity . ĠOther Ġinteresting Ġvariables Ġinclude Ġtime , Ġspace , Ġand Ġeven Ġthe Ġsocial Ġnetwork Ġposition Ġof Ġthe Ġauthor . ĠIt Ġis Ġoften Ġpreferable Ġto Ġkeep Ġthe Ġvariable Ġin Ġits Ġmost Ġprecise Ġform . ĠFor Ġexample , ĠB IB REF 21 Ġperform Ġexploration Ġin Ġthe Ġcontext Ġof Ġhypothesis Ġtesting Ġby Ġusing Ġlatitude Ġand Ġlong itude Ġcoordinates ĠâĢĶ Ġthe Ġoriginal Ġmetadata Ġattached Ġto Ġge ot agged Ġsocial Ġmedia Ġsuch Ġas Ġtweets ĠâĢĶ Ġrather Ġthan Ġaggreg ating Ġinto Ġadministrative Ġunits Ġsuch Ġas Ġcounties Ġor Ġcities . ĠThis Ġis Ġnecessary Ġwhen Ġsuch Ġadministrative Ġunits Ġare Ġunlikely Ġto Ġbe Ġrelated Ġto Ġthe Ġtarget Ġconcept , Ġas Ġis Ġthe Ġcase Ġin Ġtheir Ġanalysis Ġof Ġdialect Ġdifferences . ĠF ocusing Ġon Ġprecise Ġgeographical Ġcoordinates Ġalso Ġmakes Ġit Ġpossible Ġto Ġrecognize Ġfine - gr ained Ġeffects , Ġsuch Ġas Ġlanguage Ġvariation Ġacross Ġthe Ġgeography Ġof Ġa Ġcity . </s> Using Ġa Ġparticular Ġclassification Ġscheme Ġmeans Ġdeciding Ġwhich Ġvariations Ġare Ġvisible , Ġand Ġwhich Ġones Ġare Ġhidden ĠB IB REF 22 Ġ. ĠWe Ġare Ġlooking Ġfor Ġa Ġcategor ization Ġscheme Ġfor Ġwhich Ġit Ġis Ġfeasible Ġto Ġcollect Ġa Ġlarge Ġenough Ġlabeled Ġdocument Ġcollection Ġ( e . g ., Ġto Ġtrain Ġsupervised Ġmodels ), Ġbut Ġwhich Ġis Ġalso Ġfine - gr ained Ġenough Ġfor Ġour Ġpurposes . ĠClassification Ġschemes Ġrarely Ġexhibit Ġthe Ġideal Ġproperties , Ġi . e ., Ġthat Ġthey Ġare Ġconsistent , Ġtheir Ġcategories Ġare Ġmutually Ġexclusive , Ġand Ġthat Ġthe Ġsystem Ġis Ġcomplete ĠB IB REF 22 Ġ. ĠBorder line Ġcases Ġare Ġchallenging , Ġespecially Ġwith Ġsocial Ġand Ġcultural Ġconcepts , Ġwhere Ġthe Ġboundaries Ġare Ġoften Ġnot Ġclear - cut . ĠThe Ġchoice Ġof Ġscheme Ġcan Ġalso Ġhave Ġethical Ġimplications ĠB IB REF 22 Ġ. ĠFor Ġexample , Ġgender Ġis Ġusually Ġrepresented Ġas Ġa Ġbinary Ġvariable Ġin ĠN LP Ġand Ġcomputational Ġmodels Ġtend Ġto Ġlearn Ġgender - st ere otyp ical Ġpatterns . ĠThe Ġoperational ization Ġof Ġgender Ġin ĠN LP Ġhas Ġbeen Ġchallenged Ġonly Ġrecently ĠB IB REF 23 Ġ, ĠB IB REF 24 Ġ, ĠB IB REF 25 Ġ. </s> Super vised Ġand Ġun super vised Ġlearning Ġare Ġthe Ġmost Ġcommon Ġapproaches Ġto Ġlearning Ġfrom Ġdata . ĠWith Ġsupervised Ġlearning , Ġa Ġmodel Ġlearns Ġfrom Ġlabeled Ġdata Ġ( e . g ., Ġsocial Ġmedia Ġmessages Ġlabeled Ġby Ġsentiment ) Ġto Ġinfer Ġ( or Ġpredict ) Ġthese Ġlabels Ġfrom Ġunl abel ed Ġtexts . ĠIn Ġcontrast , Ġun super vised Ġlearning Ġuses Ġunl abel ed Ġdata . ĠSuper vised Ġapproaches Ġare Ġespecially Ġsuitable Ġwhen Ġwe Ġhave Ġa Ġclear Ġdefinition Ġof Ġthe Ġconcept Ġof Ġinterest Ġand Ġwhen Ġlabels Ġare Ġavailable Ġ( either Ġannot ated Ġor Ġnative Ġto Ġthe Ġdata ). ĠUn super vised Ġapproaches , Ġsuch Ġas Ġtopic Ġmodels , Ġare Ġespecially Ġuseful Ġfor Ġexploration . ĠIn Ġthis Ġsetting , Ġconceptual ization Ġand Ġoperational ization Ġmay Ġoccur Ġsimultaneously , Ġwith Ġtheory Ġemerging Ġfrom Ġthe Ġdata ĠB IB REF 26 Ġ. ĠUn super vised Ġapproaches Ġare Ġalso Ġused Ġwhen Ġthere Ġis Ġa Ġclear Ġway Ġof Ġmeasuring Ġa Ġconcept , Ġoften Ġbased Ġon Ġstrong Ġassumptions . ĠFor Ġexample , ĠB IB REF 3 Ġmeasure ĠâĢ ľ sur prise âĢ Ŀ Ġin Ġan Ġanalysis Ġof ĠDarwin 's Ġreading Ġdecisions Ġbased Ġon Ġthe Ġdivergence Ġbetween Ġtwo Ġprobability Ġdistributions . </s> From Ġan Ġanalysis Ġperspective , Ġthe Ġunit Ġof Ġtext Ġthat Ġwe Ġare Ġlabeling Ġ( or Ġannot ating , Ġor Ġcoding ), Ġeither Ġautomatic Ġor Ġmanual , Ġcan Ġsometimes Ġbe Ġdifferent Ġthan Ġone 's Ġfinal Ġunit Ġof Ġanalysis . ĠFor Ġexample , Ġif Ġin Ġa Ġstudy Ġon Ġmedia Ġframes Ġin Ġnews Ġstories , Ġthe Ġtheoretical Ġframework Ġand Ġresearch Ġquestion Ġpoint Ġtoward Ġframes Ġat Ġthe Ġstory Ġlevel Ġ( e . g ., Ġwhat Ġis Ġthe Ġoverall Ġcausal Ġanalysis Ġof Ġthe Ġnews Ġarticle ?), Ġthe Ġstory Ġmust Ġbe Ġthe Ġunit Ġof Ġanalysis . ĠYet Ġit Ġis Ġoften Ġdifficult Ġto Ġvalid ly Ġand Ġreliably Ġcode Ġa Ġsingle Ġframe Ġat Ġthe Ġstory Ġlevel . ĠMultiple Ġperspectives Ġare Ġlikely Ġto Ġsit Ġside - by - side Ġin Ġa Ġstory . ĠThus , Ġan Ġarticle Ġon Ġincome Ġinequality Ġmight Ġpoint Ġto Ġmultiple Ġcauses , Ġsuch Ġas Ġglobalization , Ġeducation , Ġand Ġtax Ġpolicies . ĠC oding Ġat Ġthe Ġsentence Ġlevel Ġwould Ġdetect Ġeach Ġof Ġthese Ġcausal Ġexplanations Ġindividually , Ġbut Ġthis Ġinformation Ġwould Ġneed Ġto Ġbe Ġsomehow Ġaggreg ated Ġto Ġdetermine Ġthe Ġoverall Ġstory - level Ġframe . ĠSometimes Ġscholars Ġsolve Ġthis Ġproblem Ġby Ġonly Ġexamining Ġheadlines Ġand Ġlead Ġparagraphs , Ġarguing Ġthat Ġbased Ġon Ġjournalistic Ġconvention , Ġthe Ġmost Ġimportant Ġinformation Ġcan Ġbe Ġfound Ġat Ġthe Ġbeginning Ġof Ġa Ġstory . ĠHowever , Ġthis Ġleads Ġto Ġa Ġreturn Ġto Ġa Ġshorter , Ġless Ġnuanced Ġanalysis . </s> From Ġa Ġcomputational Ġperspective , Ġthe Ġunit Ġof Ġtext Ġcan Ġalso Ġmake Ġa Ġhuge Ġdifference , Ġespecially Ġwhen Ġwe Ġare Ġusing Ġbag - of - words Ġmodels , Ġwhere Ġword Ġorder Ġwithin Ġa Ġunit Ġdoes Ġnot Ġmatter . ĠSmall Ġsegments , Ġlike Ġtweets , Ġsometimes Ġdo Ġnot Ġhave Ġenough Ġinformation Ġto Ġmake Ġtheir Ġsemantic Ġcontext Ġclear . ĠIn Ġcontrast , Ġlarger Ġsegments , Ġlike Ġnovels , Ġhave Ġtoo Ġmuch Ġvariation , Ġmaking Ġit Ġdifficult Ġto Ġtrain Ġfocused Ġmodels . ĠFinding Ġa Ġgood Ġsegment ation Ġsometimes Ġmeans Ġcombining Ġshort Ġdocuments Ġand Ġsub d ivid ing Ġlong Ġdocuments . ĠThe Ġword ĠâĢ ľ document " Ġcan Ġtherefore Ġbe Ġmisleading . ĠBut Ġit Ġis Ġso Ġingrained Ġin Ġthe Ġcommon ĠN LP Ġlex icon Ġthat Ġwe Ġuse Ġit Ġanyway Ġin Ġthis Ġarticle . </s> For Ġinsight - driven Ġtext Ġanalysis , Ġit Ġis Ġoften Ġcritical Ġthat Ġhigh - level Ġpatterns Ġcan Ġbe Ġcommunicated . ĠFurthermore , Ġinterpret able Ġmodels Ġmake Ġit Ġeasier Ġto Ġfind Ġspurious Ġfeatures , Ġto Ġdo Ġerror Ġanalysis , Ġand Ġto Ġsupport Ġinterpretation Ġof Ġresults . ĠSome Ġapproaches Ġare Ġeffective Ġfor Ġprediction , Ġbut Ġharder Ġto Ġinterpret . ĠThe Ġvalue Ġwe Ġplace Ġon Ġinterpret ability Ġcan Ġtherefore Ġinfluence Ġthe Ġapproach Ġwe Ġchoose . ĠThere Ġis Ġan Ġincreasing Ġinterest Ġin Ġdeveloping Ġinterpret able Ġor Ġtransparent Ġmodels Ġin Ġthe ĠN LP Ġand Ġmachine Ġlearning Ġcommunities . </s> An notation </s> Many Ġstudies Ġinvolve Ġhuman Ġcod ers . ĠSometimes Ġthe Ġgoal Ġis Ġto Ġfully Ġcode Ġthe Ġdata , Ġbut Ġin Ġa Ġcomputational Ġanalysis Ġwe Ġoften Ġuse Ġthe Ġlabels Ġ( or Ġannotations ) Ġto Ġtrain Ġmachine Ġlearning Ġmodels Ġto Ġautomatically Ġrecognize Ġthem , Ġand Ġto Ġidentify Ġlanguage Ġpatterns Ġthat Ġare Ġassociated Ġwith Ġthese Ġlabels . ĠFor Ġexample , Ġfor Ġa Ġproject Ġanalyzing Ġrumors Ġonline ĠB IB REF 27 Ġ, Ġconversation Ġthreads Ġwere Ġannot ated Ġalong Ġdifferent Ġdimensions , Ġincluding Ġrumor Ġversus Ġnon - rum or Ġand Ġstance Ġtowards Ġa Ġrumor . </s> The Ġcollection Ġof Ġannotation Ġchoices Ġmake Ġup Ġan Ġannotation Ġscheme Ġ( or ĠâĢ ľ code book âĢ Ŀ ). ĠEx isting Ġschemes Ġand Ġannotations Ġcan Ġbe Ġuseful Ġas Ġstarting Ġpoints . ĠUsually Ġsettling Ġon Ġan Ġannotation Ġscheme Ġrequires Ġseveral Ġiterations , Ġin Ġwhich Ġthe Ġguidelines Ġare Ġupdated Ġand Ġannotation Ġexamples Ġare Ġadded . ĠFor Ġexample , Ġa Ġpolitical Ġscientist Ġcould Ġuse Ġa Ġmixed Ġdeduct ive - ind uctive Ġstrategy Ġfor Ġdeveloping Ġa Ġcode book . ĠShe Ġstarts Ġby Ġlaying Ġout Ġa Ġset Ġof Ġtheory - driven Ġdeduct ive Ġcoding Ġrules , Ġwhich Ġmeans Ġthat Ġthe Ġbroad Ġprinciples Ġof Ġthe Ġcoding Ġrules Ġare Ġlaid Ġout Ġwithout Ġexamining Ġexamples Ġfirst . ĠThese Ġare Ġthen Ġtested Ġ( and Ġpossibly Ġadjusted ) Ġbased Ġon Ġa Ġsample Ġof Ġthe Ġdata . ĠIn Ġline Ġwith ĠAd cock Ġand ĠColl ier 's Ġnotion Ġof ĠâĢ ľ content Ġvalidity âĢ Ŀ ĠB IB REF 13 Ġ, Ġthe Ġgoal Ġis Ġto Ġassess Ġwhether Ġthe Ġcode book Ġadequately Ġcaptures Ġthe Ġsystem at ized Ġconcept . ĠBy Ġlooking Ġat Ġthe Ġdata Ġthemselves , Ġshe Ġgains Ġa Ġbetter Ġsense Ġof Ġwhether Ġsome Ġthings Ġhave Ġbeen Ġleft Ġout Ġof Ġthe Ġcoding Ġrules Ġand Ġwhether Ġanything Ġis Ġsuperflu ous , Ġmisleading , Ġor Ġconfusing . ĠAdjust ments Ġare Ġmade Ġand Ġthe Ġprocess Ġis Ġrepeated , Ġoften Ġwith Ġanother Ġresearcher Ġinvolved . </s> The Ġfinal Ġannotations Ġcan Ġbe Ġcollected Ġusing Ġa Ġcrowds ourcing Ġplatform , Ġa Ġsmaller Ġnumber Ġof Ġhighly - trained Ġannot ators , Ġor Ġa Ġgroup Ġof Ġexperts . ĠWhich Ġtype Ġof Ġannot ator Ġto Ġuse Ġshould Ġbe Ġinformed Ġby Ġthe Ġcomplexity Ġand Ġspecificity Ġof Ġthe Ġconcept . ĠFor Ġmore Ġcomplex Ġconcepts , Ġhighly - trained Ġor Ġexpert Ġannot ators Ġtend Ġto Ġproduce Ġmore Ġreliable Ġresults . ĠHowever , Ġcomplex Ġconcepts Ġcan Ġsometimes Ġbe Ġbroken Ġdown Ġinto Ġmicro - t asks Ġthat Ġcan Ġbe Ġperformed Ġindependently Ġin Ġparallel Ġby Ġcrowds ourced Ġannot ators . ĠConcepts Ġfrom Ġhighly Ġspecialized Ġdomains Ġmay Ġrequire Ġexpert Ġannot ators . ĠIn Ġall Ġcases , Ġhowever , Ġsome Ġtraining Ġwill Ġbe Ġrequired , Ġand Ġthe Ġtraining Ġphase Ġshould Ġinvolve Ġcontinual Ġchecks Ġof Ġinter - annot ator Ġagreement Ġ( i . e . Ġinter c oder Ġreliability ) Ġor Ġchecks Ġagainst Ġa Ġgold Ġstandard Ġ( e . g . Ġquizz es Ġin Ġcrowds ourcing Ġplatforms ). </s> We Ġalso Ġneed Ġto Ġdecide Ġhow Ġinter - annot ator Ġagreement Ġwill Ġbe Ġmeasured Ġand Ġwhat Ġan Ġacceptable Ġlevel Ġof Ġagreement Ġwould Ġbe . ĠK ripp endor ff 's Ġalpha Ġis Ġfrequently Ġused Ġin Ġthe Ġsocial Ġsciences , Ġbut Ġthe Ġright Ġmeasure Ġdepends Ġon Ġthe Ġtype Ġof Ġdata Ġand Ġtask . ĠFor Ġmanual Ġcoding , Ġwe Ġcan Ġcontinually Ġcheck Ġinter - annot ator Ġagreement Ġand Ġbegin Ġintroducing Ġchecks Ġof Ġintra - annot ator Ġagreement , Ġtoo . ĠFor Ġmost Ġcommunication Ġscholars Ġusing Ġonly Ġmanual Ġcontent Ġanalysis , Ġan Ġacceptable Ġrate Ġof Ġagreement Ġis Ġachieved Ġwhen ĠK ripp endor f 's Ġalpha Ġreaches Ġ0 . 80 Ġor Ġabove . ĠWhen Ġhuman - coded Ġdata Ġare Ġused Ġto Ġvalidate Ġmachine Ġlearning Ġalgorithms , Ġthe Ġreliability Ġof Ġthe Ġhuman - coded Ġdata Ġis Ġeven Ġmore Ġimportant . ĠDis ag reement Ġbetween Ġannot ators Ġcan Ġsignal Ġweaknesses Ġof Ġthe Ġannotation Ġscheme , Ġor Ġhighlight Ġthe Ġinherent Ġambiguity Ġin Ġwhat Ġwe Ġare Ġtrying Ġto Ġmeasure . ĠDis ag reement Ġitself Ġcan Ġbe Ġmeaningful Ġand Ġcan Ġbe Ġintegrated Ġin Ġsubsequent Ġanalyses ĠB IB REF 28 Ġ, ĠB IB REF 29 Ġ. </s> Data Ġpre - processing </s> Prep aring Ġthe Ġdata Ġcan Ġbe Ġa Ġcomplex Ġand Ġtime - consuming Ġprocess , Ġoften Ġinvolving Ġworking Ġwith Ġpartially Ġor Ġwholly Ġun struct ured Ġdata . ĠThe Ġpre - processing Ġsteps Ġhave Ġa Ġbig Ġimpact Ġon Ġthe Ġoperational izations , Ġsubsequent Ġanalyses Ġand Ġreprodu c ibility Ġefforts ĠB IB REF 30 Ġ, Ġand Ġthey Ġare Ġusually Ġtightly Ġlinked Ġto Ġwhat Ġwe Ġintend Ġto Ġmeasure . ĠUnfortunately , Ġthese Ġsteps Ġtend Ġto Ġbe Ġunder reported , Ġbut Ġdocumenting Ġthe Ġpre - processing Ġchoices Ġmade Ġis Ġessential Ġand Ġis Ġanalogous Ġto Ġrecording Ġthe Ġdecisions Ġtaken Ġduring Ġthe Ġproduction Ġof Ġa Ġscholarly Ġedition Ġor Ġprotocols Ġin Ġbiomedical Ġresearch . ĠData Ġmay Ġalso Ġvary Ġenormously Ġin Ġquality , Ġdepending Ġon Ġhow Ġit Ġhas Ġbeen Ġgenerated . ĠMany Ġhistorians , Ġfor Ġexample , Ġwork Ġwith Ġtext Ġproduced Ġfrom Ġan Ġanalogue Ġoriginal Ġusing ĠOptical ĠCharacter ĠRecogn ition Ġ( OC R ). ĠOften , Ġthere Ġwill Ġbe Ġlimited Ġinformation Ġavailable Ġregarding Ġthe Ġaccuracy Ġof Ġthe ĠO CR , Ġand Ġthe Ġdegree Ġof Ġaccuracy Ġmay Ġeven Ġvary Ġwithin Ġa Ġsingle Ġcorpus Ġ( e . g . Ġwhere Ġdigit ized Ġtext Ġhas Ġbeen Ġproduced Ġover Ġa Ġperiod Ġof Ġyears , Ġand Ġthe Ġsoftware Ġhas Ġgradually Ġimproved ). ĠThe Ġfirst Ġstep , Ġthen , Ġis Ġto Ġtry Ġto Ġcorrect Ġfor Ġcommon ĠO CR Ġerrors . ĠThese Ġwill Ġvary Ġdepending Ġon Ġthe Ġtype Ġof Ġtext , Ġthe Ġdate Ġat Ġwhich Ġthe Ġ` original ' Ġwas Ġproduced , Ġand Ġthe Ġnature Ġof Ġthe Ġfont Ġand Ġtypes etting . </s> One Ġstep Ġthat Ġalmost Ġeveryone Ġtakes Ġis Ġto Ġtoken ize Ġthe Ġoriginal Ġcharacter Ġsequence Ġinto Ġthe Ġwords Ġand Ġword - like Ġunits . ĠToken ization Ġis Ġa Ġmore Ġsubtle Ġand Ġmore Ġpowerful Ġprocess Ġthan Ġpeople Ġexpect . ĠIt Ġis Ġoften Ġdone Ġusing Ġregular Ġexpressions Ġor Ġscripts Ġthat Ġhave Ġbeen Ġcirculating Ġwithin Ġthe ĠN LP Ġcommunity . ĠToken ization Ġhe ur istics , Ġhowever , Ġcan Ġbe Ġbadly Ġconfused Ġby Ġemot icons , Ġcreative Ġorth ography Ġ( e . g ., ĠU $ A , Ġsh ! t ), Ġand Ġmissing Ġwhites pace . ĠMulti - word Ġterms Ġare Ġalso Ġchallenging . ĠTreat ing Ġthem Ġas Ġa Ġsingle Ġunit Ġcan Ġdramatically Ġalter Ġthe Ġpatterns Ġin Ġtext . ĠMany Ġwords Ġthat Ġare Ġindividually Ġambiguous Ġhave Ġclear , Ġunmist akable Ġmeanings Ġas Ġterms , Ġlike ĠâĢ ľ black Ġhole " Ġor ĠâĢ ľ European ĠUnion ". ĠHowever , Ġdeciding Ġwhat Ġconstitutes Ġa Ġmulti - word Ġterm Ġis Ġa Ġdifficult Ġproblem . ĠIn Ġwriting Ġsystems Ġlike ĠChinese , Ġtoken ization Ġis Ġa Ġresearch Ġproblem Ġin Ġits Ġown Ġright . </s> Beyond Ġtoken ization , Ġcommon Ġsteps Ġinclude Ġlower c asing , Ġremoving Ġpunct uation , Ġstemming Ġ( rem oving Ġsuffix es ), Ġle mm at ization Ġ( con verting Ġinf lections Ġto Ġa Ġbase Ġle mma ), Ġand Ġnormal ization , Ġwhich Ġhas Ġnever Ġbeen Ġclearly Ġdefined , Ġbut Ġoften Ġincludes Ġgrouping Ġabbrevi ations Ġlike ĠâĢ ľ U . S . A ." Ġand ĠâĢ ľ USA ", Ġord inals Ġlike ĠâĢ ľ 1 st " Ġand ĠâĢ ľ first ", Ġand Ġvariant Ġspell ings Ġlike ĠâĢ ľ n oooo oo ". ĠThe Ġmain Ġgoal Ġof Ġthese Ġsteps Ġis Ġto Ġimprove Ġthe Ġratio Ġof Ġtokens Ġ( individual Ġoccurrences ) Ġto Ġtypes Ġ( the Ġdistinct Ġthings Ġin Ġa Ġcorpus ). ĠEach Ġstep Ġrequires Ġmaking Ġadditional Ġassumptions Ġabout Ġwhich Ġdistinctions Ġare Ġrelevant : Ġis ĠâĢ ľ apple âĢ Ŀ Ġdifferent Ġfrom ĠâĢ ľ Apple âĢ Ŀ ? ĠIs ĠâĢ ľ bur nt âĢ Ŀ Ġdifferent Ġfrom ĠâĢ ľ burn ed âĢ Ŀ ? ĠIs ĠâĢ ľ cool " Ġdifferent Ġfrom ĠâĢ ľ co oo ool "? ĠSometimes Ġthese Ġsteps Ġcan Ġactively Ġhide Ġuseful Ġpatterns , Ġlike Ġsocial Ġmeaning ĠB IB REF 32 Ġ. ĠSome Ġof Ġus Ġtherefore Ġtry Ġdo Ġas Ġlittle Ġmodification Ġas Ġpossible . </s> From Ġa Ġmult ilingual Ġperspective , ĠEnglish Ġand ĠChinese Ġhave Ġan Ġunusually Ġsimple Ġinf lection al Ġsystem , Ġand Ġso Ġit Ġis Ġstatistically Ġreasonable Ġto Ġtreat Ġeach Ġinf lection Ġas Ġa Ġunique Ġword Ġtype . ĠRomance Ġlanguages Ġhave Ġconsiderably Ġmore Ġinf lections Ġthan ĠEnglish ; Ġmany Ġindigenous ĠNorth ĠAmerican Ġlanguages Ġhave Ġstill Ġmore . ĠFor Ġthese Ġlanguages , Ġunseen Ġdata Ġis Ġfar Ġmore Ġlikely Ġto Ġinclude Ġpreviously - un seen Ġinf lections , Ġand Ġtherefore , Ġdealing Ġwith Ġinf lections Ġis Ġmore Ġimportant . ĠOn Ġthe Ġother Ġhand , Ġthe Ġresources Ġfor Ġhandling Ġinf lections Ġvary Ġgreatly Ġby Ġlanguage , Ġwith ĠEuropean Ġlanguages Ġdominating Ġthe Ġattention Ġof Ġthe Ġcomputational Ġlingu istics Ġcommunity Ġthus Ġfar . </s> We Ġsometimes Ġalso Ġremove Ġwords Ġthat Ġare Ġnot Ġrelevant Ġto Ġour Ġgoals , Ġfor Ġexample Ġby Ġcalculating Ġvocabulary Ġfrequencies . ĠWe Ġconstruct Ġa ĠâĢ ľ stop list âĢ Ŀ Ġof Ġwords Ġthat Ġwe Ġare Ġnot Ġinterested Ġin . ĠIf Ġwe Ġare Ġlooking Ġfor Ġsemantic Ġthemes Ġwe Ġmight Ġremove Ġfunction Ġwords Ġlike Ġdetermin ers Ġand Ġpre pos itions . ĠIf Ġwe Ġare Ġlooking Ġfor Ġauthor - specific Ġstyles , Ġwe Ġmight Ġremove Ġall Ġwords Ġexcept Ġfunction Ġwords . ĠSome Ġwords Ġare Ġgenerally Ġmeaningful Ġbut Ġtoo Ġfrequent Ġto Ġbe Ġuseful Ġwithin Ġa Ġspecific Ġcollection . ĠWe Ġsometimes Ġalso Ġremove Ġvery Ġinf requent Ġwords . ĠTheir Ġoccurrences Ġare Ġtoo Ġlow Ġfor Ġrobust Ġpatterns Ġand Ġremoving Ġthem Ġhelps Ġreducing Ġthe Ġvocabulary Ġsize . </s> The Ġchoice Ġof Ġprocessing Ġsteps Ġcan Ġbe Ġguided Ġby Ġtheory Ġor Ġknowledge Ġabout Ġthe Ġdomain Ġas Ġwell Ġas Ġexperimental Ġinvestigation . ĠWhen Ġwe Ġhave Ġlabels , Ġpredictive Ġaccuracy Ġof Ġa Ġmodel Ġis Ġa Ġway Ġto Ġassess Ġthe Ġeffect Ġof Ġthe Ġprocessing Ġsteps . ĠIn Ġun super vised Ġsettings , Ġit Ġis Ġmore Ġchallenging Ġto Ġunderstand Ġthe Ġeffects Ġof Ġdifferent Ġsteps . ĠInf erences Ġdrawn Ġfrom Ġun super vised Ġsettings Ġcan Ġbe Ġsensitive Ġto Ġpre - processing Ġchoices ĠB IB REF 33 Ġ. ĠSt em ming Ġhas Ġbeen Ġfound Ġto Ġprovide Ġlittle Ġmeasurable Ġbenefits Ġfor Ġtopic Ġmodeling Ġand Ġcan Ġsometimes Ġeven Ġbe Ġharmful ĠB IB REF 34 Ġ. ĠAll Ġin Ġall , Ġthis Ġagain Ġhighlights Ġthe Ġneed Ġto Ġdocument Ġthese Ġsteps . </s> Finally , Ġwe Ġcan Ġalso Ġmark Ġup Ġthe Ġdata , Ġe . g ., Ġby Ġidentifying Ġentities Ġ( people , Ġplaces , Ġorganizations , Ġetc .) Ġor Ġparts Ġof Ġspeech . ĠAlthough Ġmany ĠN LP Ġtools Ġare Ġavailable Ġfor Ġsuch Ġtasks , Ġthey Ġare Ġoften Ġchallenged Ġby Ġlinguistic Ġvariation , Ġsuch Ġas Ġorth ographic Ġvariation Ġin Ġhistorical Ġtexts ĠB IB REF 35 Ġand Ġsocial Ġmedia ĠB IB REF 32 Ġ. ĠMoreover , Ġthe Ġperformance Ġof ĠN LP Ġtools Ġoften Ġdrops Ġwhen Ġapplying Ġthem Ġoutside Ġthe Ġtraining Ġdomain , Ġsuch Ġas Ġapplying Ġtools Ġdeveloped Ġon Ġnews wire Ġtexts Ġto Ġtexts Ġwritten Ġby Ġyounger Ġauthors ĠB IB REF 36 Ġ. ĠProblems Ġ( e . g ., Ġdis amb ig uation Ġin Ġnamed Ġentity Ġrecognition ) Ġare Ġsometimes Ġresolved Ġusing Ġconsiderable Ġmanual Ġintervention . ĠThis Ġcombination Ġof Ġthe Ġautomated Ġand Ġthe Ġmanual , Ġhowever , Ġbecomes Ġmore Ġdifficult Ġas Ġthe Ġscale Ġof Ġthe Ġdata Ġincreases , Ġand Ġthe Ġ` certain ty ' Ġbrought Ġby Ġthe Ġlatter Ġmay Ġhave Ġto Ġbe Ġabandoned . </s> D ictionary - based Ġapproaches </s> D iction aries Ġare Ġfrequently Ġused Ġto Ġcode Ġtexts Ġin Ġcontent Ġanalyses ĠB IB REF 37 Ġ. ĠD iction aries Ġconsist Ġof Ġone Ġor Ġmore Ġcategories Ġ( i . e . Ġword Ġlists ). ĠSometimes Ġthe Ġoutput Ġis Ġsimply Ġthe Ġnumber Ġof Ġcategory Ġoccurrences Ġ( e . g ., Ġpositive Ġsentiment ), Ġthus Ġweight ing Ġwords Ġwithin Ġa Ġcategory Ġequally . ĠIn Ġsome Ġother Ġcases , Ġwords Ġare Ġassigned Ġcontinuous Ġscores . ĠThe Ġhigh Ġtransparency Ġof Ġdiction aries Ġmakes Ġthem Ġsometimes Ġmore Ġsuitable Ġthan Ġsupervised Ġmachine Ġlearning Ġmodels . ĠHowever , Ġdiction aries Ġshould Ġonly Ġbe Ġused Ġif Ġthe Ġscores Ġassigned Ġto Ġwords Ġmatch Ġhow Ġthe Ġwords Ġare Ġused Ġin Ġthe Ġdata Ġ( see ĠB IB REF 38 Ġfor Ġa Ġdetailed Ġdiscussion Ġon Ġlimitations ). ĠThere Ġare Ġmany Ġoff - the - she lf Ġdiction aries Ġavailable Ġ( e . g ., ĠLI WC ĠB IB REF 39 Ġ). ĠThese Ġare Ġoften Ġwell - valid ated , Ġbut Ġapplying Ġthem Ġon Ġa Ġnew Ġdomain Ġmay Ġnot Ġbe Ġappropriate Ġwithout Ġadditional Ġvalidation . ĠCorpus - Ġor Ġdomain - specific Ġdiction aries Ġcan Ġovercome Ġlimitations Ġof Ġgeneral - purpose Ġdiction aries . </s> The Ġdiction aries Ġare Ġoften Ġmanually Ġcompiled , Ġbut Ġincreasingly Ġthey Ġare Ġconstructed Ġsemi - aut om atically Ġ( e . g ., ĠB IB REF 40 Ġ). ĠWhen Ġwe Ġsemi - aut om atically Ġcreate Ġa Ġword Ġlist , Ġwe Ġuse Ġautomation Ġto Ġidentify Ġan Ġinitial Ġword Ġlist , Ġand Ġhuman Ġinsight Ġto Ġfilter Ġit . ĠBy Ġautomatically Ġgenerating Ġthe Ġinitial Ġwords Ġlists , Ġwords Ġcan Ġbe Ġidentified Ġthat Ġhuman Ġannot ators Ġmight Ġhave Ġdifficulty Ġintu iting . ĠBy Ġmanually Ġfiltering Ġthe Ġlists , Ġwe Ġuse Ġour Ġtheoretical Ġunderstanding Ġof Ġthe Ġtarget Ġconcept Ġto Ġremove Ġspurious Ġfeatures . </s> In Ġthe Ġintroduction Ġstudy , ĠS AGE ĠB IB REF 41 Ġwas Ġused Ġto Ġobtain Ġa Ġlist Ġof Ġwords Ġthat Ġdistinguished Ġthe Ġtext Ġin Ġthe Ġtreatment Ġgroup Ġ( sub reddits Ġthat Ġwere Ġclosed Ġby ĠReddit ) Ġfrom Ġtext Ġin Ġthe Ġcontrol Ġgroup Ġ( similar Ġsubreddits Ġthat Ġwere Ġnot Ġclosed ). ĠThe Ġresearchers Ġthen Ġreturned Ġto Ġthe Ġhate Ġspeech Ġdefinition Ġprovided Ġby Ġthe ĠEuropean ĠCourt Ġof ĠHuman ĠRights , Ġand Ġmanually Ġfiltered Ġthe Ġtop ĠS AGE Ġwords Ġbased Ġon Ġthis Ġdefinition . ĠNot Ġall Ġidentified Ġwords Ġfitted Ġthe Ġdefinition . ĠThe Ġothers Ġincluded : Ġthe Ġnames Ġof Ġthe Ġsubreddits Ġthemselves , Ġnames Ġof Ġrelated Ġsubreddits , Ġcommunity - specific Ġjargon Ġthat Ġwas Ġnot Ġdirectly Ġrelated Ġto Ġhate Ġspeech , Ġand Ġterms Ġsuch Ġas ĠIQ Ġand Ġwelfare , Ġwhich Ġwere Ġfrequently Ġused Ġin Ġdisc ourses Ġof Ġhate Ġspeech , Ġbut Ġhad Ġsignificant Ġother Ġuses . ĠThe Ġword Ġlists Ġprovided Ġthe Ġmeasurement Ġinstrument Ġfor Ġtheir Ġmain Ġresult , Ġwhich Ġis Ġthat Ġthe Ġuse Ġof Ġhate Ġspeech Ġthroughout ĠReddit Ġdeclined Ġafter Ġthe Ġtwo Ġtreatment Ġsubreddits Ġwere Ġclosed . </s> Super vised Ġmodels </s> Super vised Ġlearning Ġis Ġfrequently Ġused Ġto Ġscale Ġup Ġanalyses . ĠFor Ġexample , ĠB IB REF 42 Ġwanted Ġto Ġanalyze Ġthe Ġmotivations Ġof ĠM ovember Ġcampaign Ġparticipants . ĠBy Ġdeveloping Ġa Ġclass ifier Ġbased Ġon Ġa Ġsmall Ġset Ġof Ġannotations , Ġthey Ġwere Ġable Ġto Ġexpand Ġthe Ġanalysis Ġto Ġover Ġ90 k Ġparticipants . </s> The Ġchoice Ġof Ġsupervised Ġlearning Ġmodel Ġis Ġoften Ġguided Ġby Ġthe Ġtask Ġdefinition Ġand Ġthe Ġlabel Ġtypes . ĠFor Ġexample , Ġto Ġidentify Ġstance Ġtowards Ġrumors Ġbased Ġon Ġsequential Ġannotations , Ġan Ġalgorithm Ġfor Ġlearning Ġfrom Ġsequential ĠB IB REF 43 Ġor Ġtime Ġseries Ġdata ĠB IB REF 44 Ġcould Ġbe Ġused . ĠThe Ġfeatures Ġ( sometimes Ġcalled Ġvariables Ġor Ġpredict ors ) Ġare Ġused Ġby Ġthe Ġmodel Ġto Ġmake Ġthe Ġpredictions . ĠThey Ġmay Ġvary Ġfrom Ġcontent - based Ġfeatures Ġsuch Ġas Ġsingle Ġwords , Ġsequences Ġof Ġwords , Ġor Ġinformation Ġabout Ġtheir Ġsynt actic Ġstructure , Ġto Ġmeta - information Ġsuch Ġas Ġuser Ġor Ġnetwork Ġinformation . ĠDec iding Ġon Ġthe Ġfeatures Ġrequires Ġexperimentation Ġand Ġexpert Ġinsight Ġand Ġis Ġoften Ġcalled Ġfeature Ġengineering . ĠFor Ġinsight - driven Ġanalysis , Ġwe Ġare Ġoften Ġinterested Ġin Ġwhy Ġa Ġprediction Ġhas Ġbeen Ġmade Ġand Ġfeatures Ġthat Ġcan Ġbe Ġinterpreted Ġby Ġhumans Ġmay Ġbe Ġpreferred . ĠRecent Ġneural Ġnetwork Ġapproaches Ġoften Ġuse Ġsimple Ġfeatures Ġas Ġinput Ġ( such Ġas Ġword Ġembed d ings Ġor Ġcharacter Ġsequences ), Ġwhich Ġrequires Ġless Ġfeature Ġengineering Ġbut Ġmake Ġinterpretation Ġmore Ġdifficult . </s> Super vised Ġmodels Ġare Ġpowerful , Ġbut Ġthey Ġcan Ġlatch Ġon Ġto Ġspurious Ġfeatures Ġof Ġthe Ġdataset . ĠThis Ġis Ġparticularly Ġtrue Ġfor Ġdatasets Ġthat Ġare Ġnot Ġwell - balanced , Ġand Ġfor Ġannotations Ġthat Ġare Ġnoisy . ĠIn Ġour Ġintroductory Ġexample Ġon Ġhate Ġspeech Ġin ĠReddit ĠB IB REF 0 Ġ, Ġthe Ġannotations Ġare Ġautomatically Ġderived Ġfrom Ġthe Ġforum Ġin Ġwhich Ġeach Ġpost Ġappears , Ġand Ġindeed , Ġmany Ġof Ġthe Ġposts Ġin Ġthe Ġforums Ġ( sub reddits ) Ġthat Ġwere Ġbanned Ġby ĠReddit Ġwould Ġbe Ġperceived Ġby Ġmany Ġas Ġhate Ġspeech . ĠBut Ġeven Ġin Ġbanned Ġsubreddits , Ġnot Ġall Ġof Ġthe Ġcontent Ġis Ġhate Ġspeech Ġ( e . g ., Ġsome Ġof Ġthe Ġtop Ġfeatures Ġwere Ġself - ref erential Ġlike Ġthe Ġname Ġof Ġthe Ġsubreddit ) Ġbut Ġa Ġclass ifier Ġwould Ġlearn Ġa Ġhigh Ġweight Ġfor Ġthese Ġfeatures . </s> Even Ġwhen Ġexpert Ġannotations Ġare Ġavailable Ġon Ġthe Ġlevel Ġof Ġindividual Ġposts , Ġspurious Ġfeatures Ġmay Ġremain . ĠB IB REF 45 Ġproduced Ġexpert Ġannotations Ġof Ġhate Ġspeech Ġon ĠTwitter . ĠThey Ġfound Ġthat Ġone Ġof Ġthe Ġstrongest Ġfeatures Ġfor Ġsexism Ġis Ġthe Ġname Ġof Ġan ĠAustralian ĠTV Ġshow , Ġbecause Ġpeople Ġlike Ġto Ġpost Ġsexist Ġcomments Ġabout Ġthe Ġcontestants . ĠIf Ġwe Ġare Ġtrying Ġto Ġmake Ġclaims Ġabout Ġwhat Ġinhibits Ġor Ġencourages Ġhate Ġspeech , Ġwe Ġwould Ġnot Ġwant Ġthose Ġclaims Ġto Ġbe Ġtied Ġto Ġthe ĠTV Ġshow 's Ġpopularity . ĠSuch Ġproblems Ġare Ġinevitable Ġwhen Ġdatasets Ġare Ġnot Ġwell - balanced Ġover Ġtime , Ġacross Ġgenres , Ġtopics , Ġetc . ĠEspecially Ġwith Ġsocial Ġmedia Ġdata , Ġwe Ġlack Ġa Ġclear Ġand Ġobjective Ġdefinition Ġof Ġ` balance ' Ġat Ġthis Ġtime . </s> The Ġrisk Ġof Ġsupervised Ġmodels Ġl atching Ġon Ġto Ġspurious Ġfeatures Ġreinforces Ġthe Ġneed Ġfor Ġinterpret ability . ĠAlthough Ġthe Ġdevelopment Ġof Ġsupervised Ġmodels Ġis Ġusually Ġperformance Ġdriven , Ġplacing Ġmore Ġemphasis Ġon Ġinterpret ability Ġcould Ġincrease Ġthe Ġadoption Ġof Ġthese Ġmodels Ġin Ġinsight - driven Ġanalyses . ĠOne Ġway Ġwould Ġbe Ġto Ġonly Ġuse Ġmodels Ġthat Ġare Ġalready Ġsomewhat Ġinterpret able , Ġfor Ġexample Ġmodels Ġthat Ġuse Ġa Ġsmall Ġnumber Ġof Ġhuman - interpret able Ġfeatures . ĠRather Ġthan Ġimposing Ġsuch Ġrestrictions , Ġthere Ġis Ġalso Ġwork Ġon Ġgenerating Ġpost - h oc Ġexplanations Ġfor Ġindividual Ġpredictions Ġ( e . g ., ĠB IB REF 46 Ġ), Ġeven Ġwhen Ġthe Ġunderlying Ġmodel Ġitself Ġis Ġvery Ġcomplex . </s> Topic Ġmodeling </s> Topic Ġmodels Ġ( e . g ., ĠL DA ĠB IB REF 47 Ġ) Ġare Ġusually Ġun super vised Ġand Ġtherefore Ġless Ġbiased Ġtowards Ġhuman - defined Ġcategories . ĠThey Ġare Ġespecially Ġsuited Ġfor Ġinsight - driven Ġanalysis , Ġbecause Ġthey Ġare Ġconstrained Ġin Ġways Ġthat Ġmake Ġtheir Ġoutput Ġinterpret able . ĠAlthough Ġthere Ġis Ġno Ġguarantee Ġthat Ġa ĠâĢ ľ topic âĢ Ŀ Ġwill Ġcorrespond Ġto Ġa Ġrecognizable Ġtheme Ġor Ġevent Ġor Ġdiscourse , Ġthey Ġoften Ġdo Ġso Ġin Ġways Ġthat Ġother Ġmethods Ġdo Ġnot . ĠTheir Ġeasy Ġapplic ability Ġwithout Ġsupervision Ġand Ġready Ġinterpret ability Ġmake Ġtopic Ġmodels Ġgood Ġfor Ġexploration . ĠTopic Ġmodels Ġare Ġless Ġsuccessful Ġfor Ġmany Ġperformance - driven Ġapplications . ĠRaw Ġword Ġfeatures Ġare Ġalmost Ġalways Ġbetter Ġthan Ġtopics Ġfor Ġsearch Ġand Ġdocument Ġclassification . ĠL ST Ms Ġand Ġother Ġneural Ġnetwork Ġmodels Ġare Ġbetter Ġas Ġlanguage Ġmodels . ĠContinuous Ġword Ġembed d ings Ġhave Ġmore Ġexpressive Ġpower Ġto Ġrepresent Ġfine - gr ained Ġsemantic Ġsimilarities Ġbetween Ġwords . </s> A Ġtopic Ġmodel Ġprovides Ġa Ġdifferent Ġperspective Ġon Ġa Ġcollection . ĠIt Ġcreates Ġa Ġset Ġof Ġprobability Ġdistributions Ġover Ġthe Ġvocabulary Ġof Ġthe Ġcollection , Ġwhich , Ġwhen Ġcombined Ġtogether Ġin Ġdifferent Ġproportions , Ġbest Ġmatch Ġthe Ġcontent Ġof Ġthe Ġcollection . ĠWe Ġcan Ġsort Ġthe Ġwords Ġin Ġeach Ġof Ġthese Ġdistributions Ġin Ġdescending Ġorder Ġby Ġprobability , Ġtake Ġsome Ġarbitrary Ġnumber Ġof Ġmost - pro bable Ġwords , Ġand Ġget Ġa Ġsense Ġof Ġwhat Ġ( if Ġanything ) Ġthe Ġtopic Ġis ĠâĢ ľ about âĢ Ŀ . ĠEach Ġof Ġthe Ġtext Ġsegments Ġalso Ġhas Ġits Ġown Ġdistribution Ġover Ġthe Ġtopics , Ġand Ġwe Ġcan Ġsort Ġthese Ġsegments Ġby Ġtheir Ġprobability Ġwithin Ġa Ġgiven Ġtopic Ġto Ġget Ġa Ġsense Ġof Ġhow Ġthat Ġtopic Ġis Ġused . </s> One Ġof Ġthe Ġmost Ġcommon Ġquestions Ġabout Ġtopic Ġmodels Ġis Ġhow Ġmany Ġtopics Ġto Ġuse , Ġusually Ġwith Ġthe Ġimplicit Ġassumption Ġthat Ġthere Ġis Ġa ĠâĢ ľ right âĢ Ŀ Ġnumber Ġthat Ġis Ġinherent Ġin Ġthe Ġcollection . ĠWe Ġprefer Ġto Ġthink Ġof Ġthis Ġparameter Ġas Ġmore Ġlike Ġthe Ġscale Ġof Ġa Ġmap Ġor Ġthe Ġmagnification Ġof Ġa Ġmicroscope . ĠThe ĠâĢ ľ right âĢ Ŀ Ġnumber Ġis Ġdetermined Ġby Ġthe Ġneeds Ġof Ġthe Ġuser , Ġnot Ġby Ġthe Ġcollection . ĠIf Ġthe Ġanalyst Ġis Ġlooking Ġfor Ġa Ġbroad Ġoverview , Ġa Ġrelatively Ġsmall Ġnumber Ġof Ġtopics Ġmay Ġbe Ġbest . ĠIf Ġthe Ġanalyst Ġis Ġlooking Ġfor Ġfine - gr ained Ġphenomena , Ġa Ġlarger Ġnumber Ġis Ġbetter . </s> After Ġfitting Ġthe Ġmodel , Ġit Ġmay Ġbe Ġnecessary Ġto Ġcircle Ġback Ġto Ġan Ġearlier Ġphase . ĠTopic Ġmodels Ġfind Ġconsistent Ġpatterns . ĠWhen Ġauthors Ġrepeatedly Ġuse Ġa Ġparticular Ġtheme Ġor Ġdiscourse , Ġthat Ġrepetition Ġcreates Ġa Ġconsistent Ġpattern . ĠBut Ġother Ġfactors Ġcan Ġalso Ġcreate Ġsimilar Ġpatterns , Ġwhich Ġlook Ġas Ġgood Ġto Ġthe Ġalgorithm . ĠWe Ġmight Ġnotice Ġa Ġtopic Ġthat Ġhas Ġhighest Ġprobability Ġon ĠFrench Ġstop words , Ġindicating Ġthat Ġwe Ġneed Ġto Ġdo Ġa Ġbetter Ġjob Ġof Ġfiltering Ġby Ġlanguage . ĠWe Ġmight Ġnotice Ġa Ġtopic Ġof Ġword Ġfragments , Ġsuch Ġas ĠâĢ ľ ing âĢ Ŀ , ĠâĢ ľ tion âĢ Ŀ , ĠâĢ ľ inter âĢ Ŀ , Ġindicating Ġthat Ġwe Ġare Ġnot Ġhandling Ġend - of - line Ġhyp hen ation Ġcorrectly . ĠWe Ġmay Ġneed Ġto Ġadd Ġto Ġour Ġstop list Ġor Ġchange Ġhow Ġwe Ġcur ate Ġmulti - word Ġterms . </s> Val idation </s> The Ġoutput Ġof Ġour Ġmeasurement Ġprocedures Ġ( in Ġthe Ġsocial Ġsciences Ġoften Ġcalled Ġthe ĠâĢ ľ sc ores âĢ Ŀ ) Ġmust Ġnow Ġbe Ġassessed Ġin Ġterms Ġof Ġtheir Ġreliability Ġand Ġvalidity Ġwith Ġregard Ġto Ġthe Ġ( system ized ) Ġconcept . ĠRel iability Ġaims Ġto Ġcapture Ġrepeat ability , Ġi . e . Ġthe Ġextent Ġto Ġwhich Ġa Ġgiven Ġtool Ġprovides Ġconsistent Ġresults . </s> Val idity Ġassess es Ġthe Ġextent Ġto Ġwhich Ġa Ġgiven Ġmeasurement Ġtool Ġmeasures Ġwhat Ġit Ġis Ġsupposed Ġto Ġmeasure . ĠIn ĠN LP Ġand Ġmachine Ġlearning , Ġmost Ġmodels Ġare Ġprimarily Ġevaluated Ġby Ġcomparing Ġthe Ġmachine - generated Ġlabels Ġagainst Ġan Ġannot ated Ġsample . ĠThis Ġapproach Ġpres umes Ġthat Ġthe Ġhuman Ġoutput Ġis Ġthe ĠâĢ ľ gold Ġstandard " Ġagainst Ġwhich Ġperformance Ġshould Ġbe Ġtested . ĠIn Ġcontrast , Ġwhen Ġthe Ġreliability Ġis Ġmeasured Ġbased Ġon Ġthe Ġoutput Ġof Ġdifferent Ġannot ators , Ġno Ġc oder Ġis Ġtaken Ġas Ġthe Ġstandard Ġand Ġthe Ġlikelihood Ġof Ġcod ers Ġreaching Ġagreement Ġby Ġchance Ġ( rather Ġthan Ġbecause Ġthey Ġare ĠâĢ ľ correct ") Ġis Ġfact ored Ġinto Ġthe Ġresulting Ġstatistic . ĠComp aring Ġagainst Ġa ĠâĢ ľ gold Ġstandard âĢ Ŀ Ġsuggests Ġthat Ġthe Ġthreshold Ġfor Ġhuman Ġinter - Ġand Ġintra - c oder Ġreliability Ġshould Ġbe Ġparticularly Ġhigh . </s> Acc uracy , Ġas Ġwell Ġas Ġother Ġmeasures Ġsuch Ġas Ġprecision , Ġrecall Ġand ĠF - score , Ġare Ġsometimes Ġpresented Ġas Ġa Ġmeasure Ġof Ġvalidity , Ġbut Ġif Ġwe Ġdo Ġnot Ġhave Ġa Ġgenuinely Ġobjective Ġdetermination Ġof Ġwhat Ġsomething Ġis Ġsupposed Ġmeasure âĢĶ as Ġis Ġoften Ġthe Ġcase Ġin Ġtext Ġanalysis âĢĶ then Ġaccuracy Ġis Ġperhaps Ġa Ġbetter Ġindication Ġof Ġreliability Ġthan Ġof Ġvalidity . ĠIn Ġthat Ġcase , Ġvalidity Ġneeds Ġto Ġbe Ġassessed Ġbased Ġon Ġother Ġtechniques Ġlike Ġthose Ġwe Ġdiscuss Ġlater Ġin Ġthis Ġsection . ĠIt Ġis Ġalso Ġworth Ġasking Ġwhat Ġlevel Ġof Ġaccuracy Ġis Ġsufficient Ġfor Ġour Ġanalysis Ġand Ġto Ġwhat Ġextent Ġthere Ġmay Ġbe Ġan Ġupper Ġbound , Ġespecially Ġwhen Ġthe Ġlabels Ġare Ġnative Ġto Ġthe Ġdata Ġor Ġwhen Ġthe Ġnotion Ġof Ġa ĠâĢ ľ gold Ġstandard âĢ Ŀ Ġis Ġnot Ġappropriate . </s> For Ġsome Ġin Ġthe Ġhumanities , Ġvalidation Ġtakes Ġthe Ġform Ġof Ġclose Ġreading , Ġnot Ġdesigned Ġto Ġconfirm Ġwhether Ġthe Ġmodel Ġoutput Ġis Ġcorrect , Ġbut Ġto Ġpresent Ġwhat ĠB IB REF 48 Ġrefers Ġto Ġas Ġa Ġform Ġof ĠâĢ ľ f urther Ġdiscovery Ġin Ġtwo Ġdirections âĢ Ŀ . ĠModel Ġoutputs Ġtell Ġus Ġsomething Ġabout Ġthe Ġtexts , Ġwhile Ġa Ġclose Ġreading Ġof Ġthe Ġtexts Ġalongside Ġthose Ġoutputs Ġtells Ġus Ġsomething Ġabout Ġthe Ġmodels Ġthat Ġcan Ġbe Ġused Ġfor Ġmore Ġeffective Ġmodel Ġbuilding . ĠApp lying Ġthis Ġcircular , Ġiter ative Ġprocess Ġto Ġ450 Ġ18 th - century Ġnovels Ġwritten Ġin Ġthree Ġlanguages , ĠPiper Ġwas Ġable Ġto Ġuncover Ġa Ġnew Ġform Ġof ĠâĢ ľ con vers ional Ġnovel âĢ Ŀ Ġthat Ġwas Ġnot Ġpreviously Ġcaptured Ġin ĠâĢ ľ liter ary Ġhistory 's Ġreceived Ġcritical Ġcategories âĢ Ŀ ĠB IB REF 48 Ġ. </s> Along Ġsimilar Ġlines , Ġwe Ġcan Ġsubject Ġboth Ġthe Ġmachine - generated Ġoutput Ġand Ġthe Ġhuman Ġannotations Ġto Ġanother Ġround Ġof Ġcontent Ġvalidation . ĠThat Ġis , Ġtake Ġa Ġstrat ified Ġrandom Ġsample , Ġselecting Ġobservations Ġfrom Ġthe Ġfull Ġrange Ġof Ġscores , Ġand Ġask : ĠDo Ġthese Ġmake Ġsense Ġin Ġlight Ġof Ġthe Ġsystem at ized Ġconcept ? ĠIf Ġnot , Ġwhat Ġseems Ġto Ġbe Ġmissing ? ĠOr Ġis Ġsomething Ġextr aneous Ġbeing Ġcaptured ? ĠThis Ġis Ġprimarily Ġa Ġqualitative Ġprocess Ġthat Ġrequires Ġreturning Ġto Ġtheory Ġand Ġinterrog ating Ġthe Ġsystem at ized Ġconcept , Ġindicators , Ġand Ġscores Ġtogether . ĠThis Ġtype Ġof Ġvalidation Ġis Ġrarely Ġdone Ġin ĠN LP , Ġbut Ġit Ġis Ġespecially Ġimportant Ġwhen Ġit Ġis Ġdifficult Ġto Ġassess Ġwhat Ġdrives Ġa Ġgiven Ġmachine Ġlearning Ġmodel . ĠIf Ġthere Ġis Ġa Ġmismatch Ġbetween Ġthe Ġscores Ġand Ġsystem at ized Ġconcept Ġat Ġthis Ġstage , Ġthe Ġcode book Ġmay Ġneed Ġto Ġbe Ġadjusted , Ġhuman Ġcod ers Ġret rained , Ġmore Ġtraining Ġdata Ġprepared , Ġalgorithms Ġadjusted , Ġor Ġin Ġsome Ġinstances , Ġeven Ġa Ġnew Ġanalytical Ġmethod Ġadopted . </s> Other Ġtypes Ġof Ġvalidation Ġare Ġalso Ġpossible , Ġsuch Ġas Ġcomparing Ġwith Ġother Ġapproaches Ġthat Ġaim Ġto Ġcapture Ġthe Ġsame Ġconcept , Ġor Ġcomparing Ġthe Ġoutput Ġwith Ġexternal Ġmeasures Ġ( e . g ., Ġpublic Ġopinion Ġpolls , Ġthe Ġoccurrence Ġof Ġfuture Ġevents ). ĠWe Ġcan Ġalso Ġgo Ġbeyond Ġonly Ġevaluating Ġthe Ġlabels Ġ( or Ġpoint Ġestimates ). ĠB IB REF 16 Ġused Ġhuman Ġjudgments Ġto Ġnot Ġonly Ġassess Ġthe Ġpositional Ġestimates Ġfrom Ġa Ġscaling Ġmethod Ġof Ġlatent Ġpolitical Ġtraits Ġbut Ġalso Ġto Ġassess Ġuncertainty Ġintervals . ĠUsing Ġdifferent Ġtypes Ġof Ġvalidation Ġcan Ġincrease Ġour Ġconfidence Ġin Ġthe Ġapproach , Ġespecially Ġwhen Ġthere Ġis Ġno Ġclear Ġnotion Ġof Ġground Ġtruth . </s> Besides Ġfocusing Ġon Ġrather Ġabstract Ġevaluation Ġmeasures , Ġwe Ġcould Ġalso Ġassess Ġthe Ġmodels Ġin Ġtask - based Ġsettings Ġusing Ġhuman Ġexperts . ĠFurthermore , Ġfor Ġinsight - driven Ġanalyses , Ġit Ġcan Ġbe Ġmore Ġuseful Ġto Ġfocus Ġon Ġimproving Ġexplanatory Ġpower Ġthan Ġmaking Ġsmall Ġimprovements Ġin Ġpredictive Ġperformance . </s> Analysis </s> In Ġthis Ġphase , Ġwe Ġuse Ġour Ġmodels Ġto Ġexplore Ġor Ġanswer Ġour Ġresearch Ġquestions . ĠFor Ġexample , Ġgiven Ġa Ġtopic Ġmodel Ġwe Ġcan Ġlook Ġat Ġthe Ġconnection Ġbetween Ġtopics Ġand Ġmetadata Ġelements . ĠTags Ġsuch Ġas ĠâĢ ľ hate Ġspeech " Ġor Ġmetadata Ġinformation Ġimply Ġa Ġcertain Ġway Ġof Ġorganizing Ġthe Ġcollection . ĠComput ational Ġmodels Ġprovide Ġanother Ġorganization , Ġwhich Ġmay Ġdiffer Ġin Ġways Ġthat Ġprovide Ġmore Ġinsight Ġinto Ġhow Ġthese Ġcategories Ġmanifest Ġthemselves , Ġor Ġfail Ġto Ġdo Ġso . </s> Moreover , Ġwhen Ġusing Ġa Ġsupervised Ġapproach , Ġthe ĠâĢ ľ errors âĢ Ŀ , Ġi . e . Ġdisagreement Ġbetween Ġthe Ġsystem Ġoutput Ġand Ġhuman - provided Ġlabels , Ġcan Ġpoint Ġtowards Ġinteresting Ġcases Ġfor Ġcloser Ġanalysis Ġand Ġhelp Ġus Ġreflect Ġon Ġour Ġconceptual izations . ĠIn Ġthe Ġwords Ġof ĠB IB REF 2 Ġ, Ġthey Ġcan Ġbe ĠâĢ ľ opp ortun ities Ġfor Ġinterpretation âĢ Ŀ . ĠOther Ġtypes Ġof ĠâĢ ľ fail ures âĢ Ŀ Ġcan Ġbe Ġinsightful Ġas Ġwell . ĠSometimes Ġthere Ġis Ġa ĠâĢ ľ dog Ġthat Ġdidn 't Ġbark âĢ Ŀ ĠB IB REF 49 ĠâĢĵ i . e ., Ġsomething Ġthat Ġeveryone Ġthinks Ġwe Ġshould Ġhave Ġfound , Ġbut Ġwe Ġdid Ġnot . ĠOr , Ġsometimes Ġthe Ġfailures Ġare Ġtelling Ġus Ġabout Ġthe Ġexistence Ġof Ġsomething Ġin Ġthe Ġdata Ġthat Ġnobody Ġnoticed , Ġor Ġthought Ġimportant , Ġuntil Ġthen Ġ( e . g ., Ġthe Ġlarge Ġnumber Ġof Ġtravel Ġjournals Ġin ĠDarwin 's Ġreading Ġlists ). </s> Com put ational Ġtext Ġanalysis Ġis Ġnot Ġa Ġreplacement Ġfor Ġbut Ġrather Ġan Ġaddition Ġto Ġthe Ġapproaches Ġone Ġcan Ġtake Ġto Ġanalyze Ġsocial Ġand Ġcultural Ġphenomena Ġusing Ġtextual Ġdata . ĠBy Ġmoving Ġback Ġand Ġforth Ġbetween Ġlarge - scale Ġcomputational Ġanalyses Ġand Ġsmall - scale Ġqualitative Ġanalyses , Ġwe Ġcan Ġcombine Ġtheir Ġstrengths Ġso Ġthat Ġwe Ġcan Ġidentify Ġlarge - scale Ġand Ġlong - term Ġtrends , Ġbut Ġalso Ġtell Ġindividual Ġstories . ĠFor Ġexample , Ġthe ĠReddit Ġstudy Ġon Ġhate Ġspeech ĠB IB REF 0 Ġraised Ġvarious Ġfollow - up Ġquestions : ĠCan Ġwe Ġdistinguish Ġhate Ġspeech Ġfrom Ġpeople Ġtalking Ġabout Ġhate Ġspeech ? ĠDid Ġpeople Ġfind Ġnew Ġways Ġto Ġexpress Ġhate Ġspeech ? ĠIf Ġso , Ġdid Ġthe Ġtotal Ġamount Ġof Ġonline Ġhate Ġspeech Ġdecrease Ġafter Ġall ? ĠAs Ġpossible Ġnext Ġsteps , Ġa Ġqualitative Ġdiscourse Ġanalyst Ġmight Ġexamine Ġa Ġsmaller Ġcorpus Ġto Ġinvestigate Ġwhether Ġcommenters Ġwere Ġindeed Ġexpressing Ġhate Ġspeech Ġin Ġnew Ġways ; Ġa Ġspecialist Ġin Ġinterview Ġmethod ologies Ġmight Ġreach Ġout Ġto Ġcommenters Ġto Ġbetter Ġunderstand Ġthe Ġrole Ġof Ġonline Ġhate Ġspeech Ġin Ġtheir Ġlives . ĠComput ational Ġtext Ġanalysis Ġrepresents Ġa Ġstep Ġtowards Ġbetter Ġunderstanding Ġsocial Ġand Ġcultural Ġphenomena , Ġand Ġit Ġis Ġin Ġmany Ġcases Ġbetter Ġsuited Ġtowards Ġopening Ġquestions Ġrather Ġthan Ġclosing Ġthem . </s> Conclusion </s> In sight - driven Ġcomputational Ġanalysis Ġof Ġtext Ġis Ġbecoming Ġincreasingly Ġcommon . ĠIt Ġnot Ġonly Ġhelps Ġus Ġsee Ġmore Ġbroadly , Ġit Ġhelps Ġus Ġsee Ġsubtle Ġpatterns Ġmore Ġclearly Ġand Ġallows Ġus Ġto Ġexplore Ġradical Ġnew Ġquestions Ġabout Ġculture Ġand Ġsociety . ĠIn Ġthis Ġarticle Ġwe Ġhave Ġconsolidated Ġour Ġexperiences , Ġas Ġscholars Ġfrom Ġvery Ġdifferent Ġdisciplines , Ġin Ġanalyzing Ġtext Ġas Ġsocial Ġand Ġcultural Ġdata Ġand Ġdescribed Ġhow Ġthe Ġresearch Ġprocess Ġoften Ġunfolds . ĠEach Ġof Ġthe Ġsteps Ġin Ġthe Ġprocess Ġis Ġtime - consuming Ġand Ġlabor - intensive . ĠEach Ġpresents Ġchallenges . ĠAnd Ġespecially Ġwhen Ġworking Ġacross Ġdisciplines , Ġthe Ġresearch Ġoften Ġinvolves Ġa Ġfair Ġamount Ġof Ġdiscussion âĢĶ even Ġnegotiation âĢĶ about Ġwhat Ġmeans Ġof Ġoperational ization Ġand Ġapproaches Ġto Ġanalysis Ġare Ġappropriate Ġand Ġfeasible . ĠAnd Ġyet , Ġwith Ġa Ġbit Ġof Ġpersever ance Ġand Ġmutual Ġunderstanding , Ġconcept ually Ġsound Ġand Ġmeaningful Ġwork Ġresults Ġso Ġthat Ġwe Ġcan Ġtruly Ġmake Ġuse Ġof Ġthe Ġexciting Ġopportunities Ġrich Ġtextual Ġdata Ġoffers . </s> Acknowled gements </s> This Ġwork Ġwas Ġsupported Ġby ĠThe ĠAlan ĠTuring ĠInstitute Ġunder Ġthe ĠEPS RC Ġgrant ĠEP / N 51 01 29 / 1 . ĠDong ĠNguyen Ġis Ġsupported Ġwith Ġan ĠAlan ĠTuring ĠInstitute ĠFellowship Ġ( T U / A / 00000 6 ). ĠMaria ĠLi ak ata Ġis Ġa ĠTuring Ġfellow Ġat Ġ40 %. ĠWe Ġwould Ġalso Ġlike Ġto Ġthank Ġthe Ġparticipants Ġof Ġthe ĠâĢ ľ B rid ging Ġdisciplines Ġin Ġanalys ing Ġtext Ġas Ġsocial Ġand Ġcultural Ġdata âĢ Ŀ Ġworkshop Ġheld Ġat Ġthe ĠTuring ĠInstitute Ġ( 2017 ) Ġfor Ġinsightful Ġdiscussions . ĠThe Ġworkshop Ġwas Ġfunded Ġby Ġa ĠTuring ĠInstitute Ġseed Ġfunding Ġaward Ġto ĠNguyen Ġand ĠLi ak ata .