politics, business, science, and AskReddit, and 1000 additional posts from the Reddit frontpage. 
EXTRACTIVE
training data has posts from politics, business, science and other popular topics; the trained model is applied to millions of unannotated posts on all of Reddit
ABSTRACTIVE
Data collection. Subreddits are sub-communities on Reddit oriented around specific interests or topics, such as technology or politics. Sampling from Reddit as a whole would bias the model towards the most commonly discussed content. But by sampling posts from individual subreddits, we can control the kinds of posts we use to train our model. To collect a diverse training dataset, we have randomly sampled 1000 posts each from the subreddits politics, business, science, and AskReddit, and 1000 additional posts from the Reddit frontpage. All posts in our sample appeared between January 2007 and March 2015, and to control for length effects, contain between 300 and 400 characters. This results in a total training dataset of 5000 posts.
Data collection. Subreddits are sub-communities on Reddit oriented around specific interests or topics, such as technology or politics. Sampling from Reddit as a whole would bias the model towards the most commonly discussed content. But by sampling posts from individual subreddits, we can control the kinds of posts we use to train our model. To collect a diverse training dataset, we have randomly sampled 1000 posts each from the subreddits politics, business, science, and AskReddit, and 1000 additional posts from the Reddit frontpage. All posts in our sample appeared between January 2007 and March 2015, and to control for length effects, contain between 300 and 400 characters. This results in a total training dataset of 5000 posts. We now apply our dogmatism classifier to a larger dataset of posts, examining how dogmatic language shapes the Reddit community. Concretely, we apply the BOW+LING model trained on the full Reddit dataset to millions of new unannotated posts, labeling these posts with a probability of dogmatism according to the classifier (0=non-dogmatic, 1=dogmatic). We then use these dogmatism annotations to address four research questions.
1609.00425
Introduction </s> âĢ ľ I 'm Ġsupposed Ġto Ġtrust Ġthe Ġopinion Ġof Ġa ĠMS Ġminion ? ĠThe Ġpeople Ġthat Ġproduced ĠWindows ĠME , ĠVista Ġand Ġ8 ? ĠThey Ġdon 't Ġeven Ġunderstand Ġpeople , Ġyet Ġthey Ġthink Ġthey Ġcan Ġpredict Ġthe Ġbehavior Ġof Ġnew , Ġself - gu iding ĠAI ? âĢ Ŀ ĠâĢĵ an onymous </s> âĢ ľ I Ġthink Ġan ĠAI Ġwould Ġmake Ġit Ġeasier Ġfor ĠPatients Ġto Ġconf ide Ġtheir Ġinformation Ġbecause Ġby Ġnature , Ġa Ġrobot Ġcannot Ġjudge Ġthem . ĠWin - win ? Ġ: D âĢ Ŀ ' ĠâĢĵ an onymous </s> Dog mat ism Ġdescribes Ġthe Ġtendency Ġto Ġlay Ġdown Ġopinions Ġas Ġinc ont ro vert ibly Ġtrue , Ġwithout Ġrespect Ġfor Ġconflicting Ġevidence Ġor Ġthe Ġopinions Ġof Ġothers ĠB IB REF 0 Ġ. ĠWhich Ġuser Ġis Ġmore Ġdog matic Ġin Ġthe Ġexamples Ġabove ? ĠThis Ġquestion Ġis Ġsimple Ġfor Ġhumans . ĠPh r ases Ġlike ĠâĢ ľ they Ġthink âĢ Ŀ Ġand ĠâĢ ľ they Ġdon 't Ġeven Ġunderstand , âĢ Ŀ Ġsuggest Ġan Ġint ract ability Ġof Ġopinion , Ġwhile ĠâĢ ľ I Ġthink âĢ Ŀ Ġand ĠâĢ ľ win - win ? âĢ Ŀ Ġsuggest Ġthe Ġopposite . ĠCan Ġwe Ġtrain Ġcomputers Ġto Ġdraw Ġsimilar Ġdistinctions ? ĠWork Ġin Ġpsychology Ġhas Ġcalled Ġout Ġmany Ġaspects Ġof Ġdog mat ism Ġthat Ġcan Ġbe Ġmodeled Ġcomput ationally Ġvia Ġnatural Ġlanguage , Ġsuch Ġas Ġover - confidence Ġand Ġstrong Ġemotions ĠB IB REF 1 Ġ. </s> We Ġpresent Ġa Ġstatistical Ġmodel Ġof Ġdog mat ism Ġthat Ġaddresses Ġtwo Ġcomplementary Ġgoals . ĠFirst , Ġwe Ġvalidate Ġpsychological Ġtheories Ġby Ġexamining Ġthe Ġpredictive Ġpower Ġof Ġfeature Ġsets Ġthat Ġguide Ġthe Ġmodel 's Ġpredictions . ĠFor Ġexample , Ġdo Ġlinguistic Ġsignals Ġof Ġcertainty Ġhelp Ġto Ġpredict Ġa Ġpost Ġis Ġdog matic , Ġas Ġtheory Ġwould Ġsuggest ? ĠSecond , Ġwe Ġapply Ġour Ġmodel Ġto Ġanswer Ġfour Ġquestions : </s> R 1 : ĠWhat Ġkinds Ġof Ġtopics Ġ( e . g ., Ġguns , ĠLGBT ) Ġattract Ġthe Ġhighest Ġlevels Ġof Ġdog mat ism ? </s> R 2 : ĠHow Ġdo Ġdog matic Ġbeliefs Ġcluster ? </s> R 3 : ĠHow Ġdoes Ġdog mat ism Ġinfluence Ġa Ġconversation Ġon Ġsocial Ġmedia ? ĠR 4 : ĠHow Ġdo Ġother Ġuser Ġbehaviors Ġ( e . g ., Ġfrequency Ġand Ġbreadth Ġof Ġposts ) Ġrelate Ġto Ġdog mat ism ? </s> We Ġtrain Ġa Ġpredictive Ġmodel Ġto Ġclassify Ġdog matic Ġposts Ġfrom ĠReddit , Ġone Ġof Ġthe Ġmost Ġpopular Ġdiscussion Ġcommunities Ġon Ġthe Ġweb . ĠPosts Ġon ĠReddit Ġcapture Ġdiscussion Ġand Ġdebate Ġacross Ġa Ġdiverse Ġset Ġof Ġdomains Ġand Ġtopics ĠâĢĵ Ġusers Ġtalk Ġabout Ġeverything Ġfrom Ġclimate Ġchange Ġand Ġabortion , Ġto Ġworld Ġnews Ġand Ġrelationship Ġadvice , Ġto Ġthe Ġfuture Ġof Ġartificial Ġintelligence . ĠAs Ġa Ġprerequisite Ġto Ġtraining Ġour Ġmodel , Ġwe Ġhave Ġcreated Ġa Ġcorpus Ġof Ġ5 , 000 ĠReddit Ġposts Ġannot ated Ġwith Ġlevels Ġof Ġdog mat ism , Ġwhich Ġwe Ġare Ġreleasing Ġto Ġshare Ġwith Ġother Ġresearchers . </s> Using Ġthe Ġmodel , Ġwe Ġoperational ize Ġkey Ġdomain - independent Ġaspects Ġof Ġpsychological Ġtheories Ġof Ġdog mat ism Ġdrawn Ġfrom Ġthe Ġliterature . ĠWe Ġfind Ġthese Ġfeatures Ġhave Ġpredictive Ġpower Ġthat Ġlargely Ġsupports Ġthe Ġunderlying Ġtheory . ĠFor Ġexample , Ġposts Ġthat Ġuse Ġless Ġconfident Ġlanguage Ġtend Ġto Ġbe Ġless Ġdog matic . ĠWe Ġalso Ġdiscover Ġevidence Ġfor Ġnew Ġattributes Ġof Ġdog mat ism . ĠFor Ġexample , Ġdog matic Ġposts Ġtend Ġnot Ġto Ġverbal ize Ġcognition , Ġthrough Ġterms Ġsuch Ġas ĠâĢ ľ I Ġthink , âĢ Ŀ ĠâĢ ľ possibly , âĢ Ŀ Ġor ĠâĢ ľ might Ġbe . âĢ Ŀ </s> Our Ġmodel Ġis Ġtrained Ġon Ġonly Ġ5 , 000 Ġannot ated Ġposts , Ġbut Ġonce Ġtrained , Ġwe Ġuse Ġit Ġto Ġanalyze Ġmillions Ġof Ġother ĠReddit Ġposts Ġto Ġanswer Ġour Ġresearch Ġquestions . ĠWe Ġfind Ġa Ġdiverse Ġset Ġof Ġtopics Ġare Ġcolored Ġby Ġdog matic Ġlanguage Ġ( e . g ., Ġpeople Ġare Ġdog matic Ġabout Ġreligion , Ġbut Ġalso Ġabout ĠLGBT Ġissues ). ĠFurther , Ġwe Ġfind Ġsome Ġevidence Ġfor Ġdog mat ism Ġas Ġa Ġdeeper Ġpersonality Ġtrait ĠâĢĵ Ġpeople Ġwho Ġare Ġstrongly Ġdog matic Ġabout Ġone Ġtopic Ġare Ġmore Ġlikely Ġto Ġexpress Ġdog matic Ġviews Ġabout Ġothers Ġas Ġwell . ĠFinally , Ġin Ġconversation , Ġwe Ġdiscover Ġthat Ġone Ġuser 's Ġdog mat ism Ġtends Ġto Ġbring Ġout Ġdog mat ism Ġin Ġtheir Ġconvers ational Ġpartner , Ġforming Ġa Ġvicious Ġcycle . </s> Dog mat ism Ġdata </s> Posts Ġon ĠReddit Ġcapture Ġdebate Ġand Ġdiscussion Ġacross Ġa Ġdiverse Ġset Ġof Ġtopics , Ġmaking Ġthem Ġa Ġnatural Ġstarting Ġpoint Ġfor Ġunt angling Ġdomain - independent Ġlinguistic Ġfeatures Ġof Ġdog mat ism . </s> Data Ġcollection . ĠSub reddits Ġare Ġsub - commun ities Ġon ĠReddit Ġoriented Ġaround Ġspecific Ġinterests Ġor Ġtopics , Ġsuch Ġas Ġtechnology Ġor Ġpolitics . ĠSam pling Ġfrom ĠReddit Ġas Ġa Ġwhole Ġwould Ġbias Ġthe Ġmodel Ġtowards Ġthe Ġmost Ġcommonly Ġdiscussed Ġcontent . ĠBut Ġby Ġsampling Ġposts Ġfrom Ġindividual Ġsubreddits , Ġwe Ġcan Ġcontrol Ġthe Ġkinds Ġof Ġposts Ġwe Ġuse Ġto Ġtrain Ġour Ġmodel . ĠTo Ġcollect Ġa Ġdiverse Ġtraining Ġdataset , Ġwe Ġhave Ġrandomly Ġsampled Ġ1000 Ġposts Ġeach Ġfrom Ġthe Ġsubreddits Ġpolitics , Ġbusiness , Ġscience , Ġand ĠAsk Reddit , Ġand Ġ1000 Ġadditional Ġposts Ġfrom Ġthe ĠReddit Ġfront page . ĠAll Ġposts Ġin Ġour Ġsample Ġappeared Ġbetween ĠJanuary Ġ2007 Ġand ĠMarch Ġ2015 , Ġand Ġto Ġcontrol Ġfor Ġlength Ġeffects , Ġcontain Ġbetween Ġ300 Ġand Ġ400 Ġcharacters . ĠThis Ġresults Ġin Ġa Ġtotal Ġtraining Ġdataset Ġof Ġ5000 Ġposts . </s> Dog mat ism Ġannotations . ĠBuilding Ġa Ġuseful Ġcomputational Ġmodel Ġrequires Ġlabeled Ġtraining Ġdata . ĠWe Ġlabeled Ġthe ĠReddit Ġdataset Ġusing Ġcrowd workers Ġon ĠAmazon ĠMechanical ĠTurk Ġ( AM T ), Ġcreating Ġthe Ġfirst Ġpublic Ġcorpus Ġannot ated Ġwith Ġlevels Ġof Ġdog mat ism . ĠWe Ġasked Ġcrowd workers Ġto Ġrate Ġlevels Ġof Ġdog mat ism Ġon Ġa Ġ5 - point ĠL ik ert Ġscale , Ġas Ġsupported Ġby Ġsimilar Ġannotation Ġtasks Ġin Ġprior Ġwork ĠB IB REF 2 Ġ. ĠCon crete ly , Ġwe Ġgave Ġcrowd workers Ġthe Ġfollowing Ġtask : </s> Given Ġa Ġcomment , Ġimagine Ġyou Ġhold Ġa Ġwell - informed , Ġdifferent Ġopinion Ġfrom Ġthe Ġcommenter Ġin Ġquestion . ĠWe 'd Ġlike Ġyou Ġto Ġtell Ġus Ġhow Ġlikely Ġthat Ġcommenter Ġwould Ġbe Ġto Ġengage Ġyou Ġin Ġa Ġconstructive Ġconversation Ġabout Ġyour Ġdisagreement , Ġwhere Ġyou Ġeach Ġare Ġable Ġto Ġexplore Ġthe Ġother 's Ġbeliefs . ĠThe Ġoptions Ġare : </s> ( 5 ): ĠIt 's Ġunlikely Ġyou 'll Ġbe Ġable Ġto Ġengage Ġin Ġany Ġsubstantive Ġconversation . ĠWhen Ġyou Ġrespectfully Ġexpress Ġyour Ġdisagreement , Ġthey Ġare Ġlikely Ġto Ġignore Ġyou Ġor Ġinsult Ġyou Ġor Ġotherwise Ġlower Ġthe Ġlevel Ġof Ġdiscourse . </s> ( 4 ): ĠThey Ġare Ġdeeply Ġrooted Ġin Ġtheir Ġopinion , Ġbut Ġyou Ġare Ġable Ġto Ġexchange Ġyour Ġviews Ġwithout Ġthe Ġconversation Ġdegener ating Ġtoo Ġmuch . </s> ( 3 ): ĠIt 's Ġnot Ġlikely Ġyou 'll Ġbe Ġable Ġto Ġchange Ġtheir Ġmind , Ġbut Ġyou 're Ġeasily Ġable Ġto Ġtalk Ġand Ġunderstand Ġeach Ġother 's Ġpoint Ġof Ġview . </s> ( 2 ): ĠThey Ġmay Ġhave Ġa Ġclear Ġopinion Ġabout Ġthe Ġsubject , Ġbut Ġwould Ġlikely Ġbe Ġopen Ġto Ġdiscussing Ġalternative Ġviewpoints . </s> ( 1 ): ĠThey Ġare Ġnot Ġset Ġin Ġtheir Ġopinion , Ġand Ġit 's Ġpossible Ġyou Ġmight Ġchange Ġtheir Ġmind . ĠIf Ġthe Ġcomment Ġdoes Ġnot Ġconvey Ġan Ġopinion Ġof Ġany Ġkind , Ġyou Ġmay Ġalso Ġselect Ġthis Ġoption . </s> To Ġensure Ġquality Ġwork , Ġwe Ġrestricted Ġthe Ġtask Ġto ĠMasters Ġworkers Ġand Ġprovided Ġexamples Ġcorresponding Ġto Ġeach Ġpoint Ġon Ġthe Ġscale . ĠIncluding Ġexamples Ġin Ġa Ġtask Ġhas Ġbeen Ġshown Ġto Ġsignificantly Ġincrease Ġthe Ġagreement Ġand Ġquality Ġof Ġcrowd work ĠB IB REF 3 Ġ. ĠFor Ġinstance , Ġhere Ġis Ġan Ġexample Ġof Ġa Ġhighly Ġdog matic Ġ( 5 ) Ġcomment : </s> I Ġwon 't Ġbe Ġhappy Ġuntil ĠI Ġsee Ġthe Ġexecutive Ġsuite Ġof ĠB of A , ĠWells , Ġand Ġall Ġthe Ġothers , Ġfrog - mar ched Ġinto Ġwaiting Ġsquad Ġcars . ĠIt 's ĠAL READ Y ĠBE EN ĠEST AB L ISH ED Ġthat ... </s> And Ġa Ġminim ally Ġdog matic Ġ( 1 ) Ġcomment : </s> I Ġagree . ĠI Ġwould Ġlike Ġto Ġcompile Ġa Ġplaylist Ġfor Ġus Ġtrance Ġyog i 's , Ġeven Ġif Ġyou Ġjust Ġwould Ġlike Ġto Ġexperiment Ġwith Ġit . ĠIs Ġthere Ġany Ġpreference Ġon Ġwhich Ġplatform Ġto Ġuse ? </s> Each Ġcomment Ġhas Ġbeen Ġannot ated Ġby Ġthree Ġindependent Ġworkers Ġon ĠAM T , Ġwhich Ġis Ġenough Ġto Ġproduce Ġreliable Ġresults Ġin Ġmost Ġlabeling Ġtasks ĠB IB REF 4 Ġ. ĠTo Ġcompute Ġan Ġaggregate Ġmeasure Ġof Ġdog mat ism Ġfor Ġeach Ġcomment , Ġwe Ġsummed Ġthe Ġscores Ġof Ġall Ġthree Ġworkers . ĠWe Ġshow Ġthe Ġresulting Ġdistribution Ġof Ġannotations Ġin ĠFigure Ġ1 Ġ. </s> Inter - annot ator Ġagreement . ĠTo Ġevaluate Ġthe Ġreliability Ġof Ġannotations Ġwe Ġcompute ĠK ripp endor ff 's Ġ$\ alpha Ġ$ Ġ, Ġa Ġmeasure Ġof Ġagreement Ġdesigned Ġfor Ġvariable Ġlevels Ġof Ġmeasurement Ġsuch Ġas Ġa ĠL ik ert Ġscale ĠB IB REF 5 Ġ. ĠAn Ġ$\ alpha Ġ$ Ġof Ġ0 Ġindicates Ġagreement Ġindistinguishable Ġfrom Ġchance , Ġwhile Ġan Ġ$\ alpha Ġ$ Ġof Ġ1 Ġindicates Ġperfect Ġagreement . ĠAcross Ġall Ġannotations Ġwe Ġfind Ġ$\ alpha Ġ= 0 . 44 $ Ġ. ĠWhile Ġworkers Ġagree Ġmuch Ġmore Ġthan Ġchance , Ġclearly Ġdog mat ism Ġis Ġalso Ġsubjective . ĠIn Ġfact , Ġwhen Ġwe Ġexamine Ġonly Ġthe Ġmiddle Ġtwo Ġquart iles Ġof Ġthe Ġdog mat ism Ġannotations , Ġwe Ġfind Ġagreement Ġis Ġno Ġbetter Ġthan Ġchance . ĠAlternatively , Ġwhen Ġwe Ġmeasure Ġagreement Ġonly Ġamong Ġthe Ġtop Ġand Ġbottom Ġquart iles Ġof Ġannotations , Ġwe Ġfind Ġagreement Ġof Ġ$\ alpha Ġ= 0 . 69 $ Ġ. ĠThis Ġsuggests Ġcomments Ġwith Ġscores Ġthat Ġare Ġonly Ġslightly Ġdog matic Ġare Ġunreliable Ġand Ġoften Ġsubject Ġto Ġhuman Ġdisagreement . ĠFor Ġthis Ġreason , Ġwe Ġuse Ġonly Ġthe Ġtop Ġand Ġbottom Ġquart iles Ġof Ġcomments Ġwhen Ġtraining Ġour Ġmodel . </s> App ro aches Ġto ĠIdent ifying ĠDog mat ism </s> We Ġnow Ġconsider Ġstrategies Ġfor Ġidentifying Ġdog mat ism Ġbased Ġon Ġprior Ġwork Ġin Ġpsychology . ĠWe Ġstart Ġwith Ġthe ĠL ingu istic ĠInquiry Ġand ĠWord ĠCount Ġ( LI WC ), Ġa Ġlex icon Ġpopular Ġin Ġthe Ġsocial Ġsciences ĠB IB REF 6 Ġ. ĠLI WC Ġprovides Ġhuman Ġvalidated Ġlists Ġof Ġwords Ġthat Ġcorrespond Ġto Ġhigh - level Ġpsychological Ġcategories Ġsuch Ġas Ġcertainty Ġor Ġperception . ĠIn Ġother Ġstudies , ĠLI WC Ġhas Ġuncovered Ġlinguistic Ġsignals Ġrelating Ġto Ġpolit eness ĠB IB REF 2 Ġ, Ġdeception ĠB IB REF 7 Ġ, Ġor Ġauthority Ġin Ġtexts ĠB IB REF 8 Ġ. ĠHere , Ġwe Ġexamine Ġhow Ġdog mat ism Ġrelates Ġto Ġ17 Ġof ĠLI WC 's Ġcategories Ġ( Table Ġ1 Ġ). </s> To Ġcompute Ġthe Ġrelationships Ġbetween ĠLI WC Ġcategories Ġand Ġdog mat ism , Ġwe Ġfirst Ġcount Ġthe Ġrelevant Ġcategory Ġterms Ġthat Ġappear Ġin Ġeach Ġannot ated ĠReddit Ġcomment , Ġnormalized Ġby Ġits Ġword Ġcount . ĠWe Ġthen Ġcalculate Ġodds Ġratios Ġon Ġthe Ġaggregate Ġcounts Ġof Ġeach ĠLI WC Ġcategory Ġover Ġthe Ġtop Ġand Ġbottom Ġquart iles Ġof Ġdog matic Ġcomments . ĠAs Ġwe Ġhave Ġdiscussed , Ġusing Ġthe Ġtop Ġand Ġbottom Ġquart iles Ġof Ġcomments Ġprovides Ġa Ġmore Ġreliable Ġsignal Ġof Ġdog mat ism . ĠWe Ġcheck Ġfor Ġsignificant Ġdifferences Ġin Ġcategories Ġbetween Ġdog matic Ġand Ġnon - dog matic Ġcomments Ġusing Ġthe ĠMann - Whit ney ĠU Ġtest Ġand Ġapply ĠHolmes Ġmethod Ġfor Ġcorrection . ĠAll Ġodds Ġwe Ġreport Ġin Ġthis Ġsection Ġare Ġsignificant Ġafter Ġcorrection . </s> Dog matic Ġstatements Ġtend Ġto Ġexpress Ġa Ġhigh Ġdegree Ġof Ġcertainty ĠB IB REF 1 Ġ. ĠHere Ġwe Ġconsider ĠLI WC Ġcategories Ġthat Ġexpress Ġcertainty Ġboth Ġpositively Ġ( certain ty ) Ġand Ġnegatively Ġ( t ent at iveness ). ĠFor Ġexample , Ġthe Ġword ĠâĢ ľ always âĢ Ŀ Ġis Ġcertain , Ġwhile ĠâĢ ľ possibly âĢ Ŀ Ġis Ġtentative . ĠCon forming Ġto Ġexisting Ġtheory , Ġcertainty Ġis Ġmore Ġassociated Ġwith Ġdog matic Ġcomments Ġ( 1 . 52 Ġodds ), Ġwhile Ġtent at iveness Ġis Ġmore Ġassociated Ġwith Ġthe Ġabsence Ġof Ġdog mat ism Ġ( 0 . 88 Ġodds ). </s> Ter ms Ġused Ġto Ġverbal ize Ġcognition Ġcan Ġact Ġas Ġa Ġhedge Ġthat Ġoften Ġcharacter izes Ġnon - dog matic Ġlanguage . ĠLI WC 's Ġinsight Ġcategory Ġcaptures Ġthis Ġeffect Ġthrough Ġwords Ġsuch Ġas ĠâĢ ľ think , âĢ Ŀ ĠâĢ ľ know , âĢ Ŀ Ġor ĠâĢ ľ bel ieve . âĢ Ŀ ĠThese Ġwords Ġadd Ġnuance Ġto Ġa Ġstatement ĠB IB REF 9 Ġ, Ġsignaling Ġit Ġis Ġthe Ġproduct Ġof Ġsomeone 's Ġmind Ġ( âĢ ľ I Ġthink Ġyou Ġshould Ġgive Ġthis Ġpaper Ġa Ġgood Ġreview âĢ Ŀ ) Ġand Ġnot Ġmeant Ġto Ġbe Ġinterpreted Ġas Ġan Ġobjective Ġtruth . ĠAlong Ġthese Ġlines , Ġwe Ġfind Ġthe Ġuse Ġof Ġterms Ġin Ġthe Ġinsight Ġcategory Ġis Ġassociated Ġwith Ġnon - dog matic Ġcomments Ġ( 0 . 83 Ġodds ). </s> S ens ory Ġlanguage , Ġwith Ġits Ġfocus Ġon Ġdescription Ġand Ġdetail , Ġoften Ġsignals Ġa Ġlack Ġof Ġany Ġkind Ġof Ġopinion , Ġdog matic Ġor Ġotherwise . ĠLI WC 's Ġperception Ġcategory Ġcaptures Ġthis Ġidea Ġthrough Ġwords Ġassociated Ġwith Ġhearing , Ġfeeling , Ġor Ġseeing . ĠFor Ġexample , Ġthese Ġwords Ġmight Ġoccur Ġwhen Ġrecount ing Ġa Ġpersonal Ġexperience Ġ( âĢ ľ I Ġsaw Ġhis Ġincoming Ġfist âĢ Ŀ ), Ġwhich Ġeven Ġif Ġemotionally Ġcharged Ġor Ġnegative , Ġis Ġless Ġlikely Ġto Ġbe Ġdog matic . ĠWe Ġfind Ġperception Ġis Ġassociated Ġwith Ġnon - dog matic Ġcomments Ġat Ġ0 . 77 Ġodds . </s> Draw ing Ġcomparisons Ġor Ġqualifying Ġsomething Ġas Ġrelative Ġto Ġsomething Ġelse Ġconve ys Ġa Ġnuance Ġthat Ġis Ġabsent Ġfrom Ġtraditionally Ġdog matic Ġlanguage . ĠThe ĠLI WC Ġcategories Ġcomparison Ġand Ġrelativity Ġcapture Ġthese Ġeffects Ġthrough Ġcomparison Ġwords Ġsuch Ġas ĠâĢ ľ than âĢ Ŀ Ġor ĠâĢ ľ as âĢ Ŀ Ġand Ġqualifying Ġwords Ġsuch Ġas ĠâĢ ľ during âĢ Ŀ Ġor ĠâĢ ľ when . âĢ Ŀ ĠFor Ġexample , Ġthe Ġstatement ĠâĢ ľ I Ġhate Ġpoliticians âĢ Ŀ Ġis Ġmore Ġdog matic Ġthan ĠâĢ ľ I Ġhate Ġpoliticians Ġwhen Ġthey Ġcan 't Ġget Ġanything Ġdone .' ĠRel ativity Ġis Ġassociated Ġwith Ġnon - dog matic Ġcomments Ġat Ġ0 . 80 Ġodds , Ġbut Ġcomparison Ġdoes Ġnot Ġreach Ġsignificance . </s> P ron oun s Ġcan Ġbe Ġsurprisingly Ġrevealing Ġindicators Ġof Ġlanguage : Ġfor Ġexample , Ġsignaling Ġone 's Ġgender Ġor Ġhierarchical Ġstatus Ġin Ġa Ġconversation ĠB IB REF 10 Ġ. ĠWe Ġfind Ġfirst Ġperson Ġsingular Ġpronouns Ġare Ġa Ġuseful Ġnegative Ġsignal Ġfor Ġdog mat ism Ġ( 0 . 46 Ġodds ), Ġwhile Ġsecond Ġperson Ġsingular Ġpronouns Ġ( 2 . 18 Ġodds ) Ġand Ġthird Ġperson Ġplural Ġ( 1 . 63 Ġodds ) Ġare Ġa Ġuseful Ġpositive Ġsignal . ĠLooking Ġacross Ġthe Ġcorpus , Ġwe Ġsee ĠI Ġoften Ġused Ġwith Ġa Ġhedge Ġ( âĢ ľ I Ġthink âĢ Ŀ Ġor ĠâĢ ľ I Ġknow âĢ Ŀ ), Ġwhile Ġyou Ġand Ġthey Ġtend Ġto Ġcharacterize Ġthe Ġbeliefs Ġof Ġothers , Ġoften Ġin Ġa Ġstrongly Ġopinion ated Ġway Ġ( âĢ ľ you Ġare Ġa Ġmor on âĢ Ŀ Ġor ĠâĢ ľ they Ġare Ġkeeping Ġus Ġdown âĢ Ŀ ). ĠOther Ġpronoun Ġtypes Ġdo Ġnot Ġshow Ġsignificant Ġrelationships . </s> Like Ġpronouns , Ġverb Ġtense Ġcan Ġreveal Ġsubtle Ġsignals Ġin Ġlanguage Ġuse , Ġsuch Ġas Ġthe Ġtendency Ġof Ġmedical Ġin pat ients Ġto Ġfocus Ġon Ġthe Ġpast ĠB IB REF 11 Ġ. ĠOn Ġsocial Ġmedia , Ġcomments Ġwritten Ġin Ġthe Ġpresent Ġtense Ġare Ġmore Ġlikely Ġto Ġbe Ġoriented Ġtowards Ġa Ġuser 's Ġcurrent Ġinteraction Ġ( âĢ ľ this Ġis Ġall Ġso Ġstupid âĢ Ŀ ), Ġcreating Ġopportunities Ġto Ġsignal Ġdog mat ism . ĠAlternatively , Ġcomments Ġin Ġthe Ġpast Ġtense Ġare Ġmore Ġlikely Ġto Ġrefer Ġto Ġoutside Ġexperiences Ġ( âĢ ľ it Ġwas Ġan Ġawful Ġparty âĢ Ŀ ), Ġspeaking Ġless Ġto Ġa Ġuser 's Ġstance Ġtowards Ġan Ġongoing Ġdiscussion . ĠWe Ġfind Ġpresent Ġtense Ġis Ġa Ġpositive Ġsignal Ġfor Ġdog mat ism Ġ( 1 . 11 Ġodds ) Ġand Ġpast Ġtense Ġis Ġa Ġnegative Ġsignal Ġ( 0 . 69 Ġodds ). </s> Dog matic Ġlanguage Ġcan Ġbe Ġeither Ġpositively Ġor Ġnegatively Ġcharged Ġin Ġsentiment : Ġfor Ġexample , Ġconsider Ġthe Ġpositive Ġstatement ĠâĢ ľ Trump Ġis Ġthe ĠS AV IOR Ġof Ġthis Ġcountry !!! âĢ Ŀ Ġor Ġthe Ġnegative Ġstatement ĠâĢ ľ Are Ġyou ĠREALLY Ġthat Ġstupid ?? ĠEducation Ġis Ġthe Ġonly Ġway Ġout Ġof Ġthis Ġhorrible Ġmess . ĠIt 's Ġhard Ġto Ġimagine Ġhow Ġanyone Ġcould Ġbe Ġso Ġdel uded . âĢ Ŀ ĠIn Ġdiverse Ġcommunities , Ġwhere Ġpeople Ġhold Ġmany Ġdifferent Ġkinds Ġof Ġopinions , Ġdog matic Ġopinions Ġwill Ġoften Ġtend Ġto Ġcome Ġinto Ġconflict Ġwith Ġone Ġanother ĠB IB REF 12 Ġ, Ġproducing Ġa Ġgreater Ġlikelihood Ġof Ġnegative Ġsentiment . ĠPerhaps Ġfor Ġthis Ġreason , Ġnegative Ġemotion Ġ( 2 . 09 Ġodds ) Ġand Ġswearing Ġ( 3 . 80 Ġodds ) Ġare Ġuseful Ġpositive Ġsignals Ġof Ġdog mat ism , Ġwhile Ġpositive Ġemotion Ġshows Ġno Ġsignificant Ġrelationship . </s> Finally , Ġwe Ġfind Ġthat Ġinterrog ative Ġlanguage Ġ( 1 . 12 Ġodds ) Ġand Ġneg ation Ġ( 1 . 35 Ġodds ) Ġare Ġtwo Ġadditional Ġpositive Ġsignals Ġof Ġdog mat ism . ĠWhile Ġinterrog ative Ġwords Ġlike ĠâĢ ľ how âĢ Ŀ Ġor ĠâĢ ľ what âĢ Ŀ Ġhave Ġmany Ġbenign Ġuses , Ġthey Ġdisproportionately Ġappear Ġin Ġour Ġdata Ġin Ġthe Ġform Ġof Ġrhetorical Ġor Ġemotionally Ġcharged Ġquestions , Ġsuch Ġas ĠâĢ ľ how Ġcan Ġanyone Ġbe Ġthat Ġdumb ? âĢ Ŀ </s> Many Ġof Ġthese Ġlinguistic Ġsignals Ġare Ġcorrelated Ġwith Ġeach Ġother , Ġsuggesting Ġthat Ġdog mat ism Ġis Ġthe Ġcumulative Ġeffect Ġof Ġmany Ġcomponent Ġrelationships . ĠFor Ġexample , Ġconsider Ġthe Ġrelatively Ġnon - dog matic Ġstatement : ĠâĢ ľ I Ġthink Ġthe Ġreviewers Ġare Ġwrong Ġin Ġthis Ġinstance . âĢ Ŀ ĠRem oving Ġsignals Ġof Ġinsight , Ġwe Ġhave : ĠâĢ ľ the Ġreviewers Ġare Ġwrong Ġin Ġthis Ġinstance , âĢ Ŀ Ġwhich Ġis Ġslightly Ġmore Ġdog matic . ĠThen Ġremoving Ġrelativity , Ġwe Ġhave : ĠâĢ ľ the Ġreviewers Ġare Ġwrong . âĢ Ŀ ĠAnd Ġfinally , Ġadding Ġcertainty , Ġwe Ġhave Ġa Ġdog matic Ġstatement : ĠâĢ ľ the Ġreviewers Ġare Ġalways Ġwrong . âĢ Ŀ </s> P redict ing Ġdog mat ism </s> We Ġnow Ġshow Ġhow Ġwe Ġcan Ġuse Ġthe Ġlinguistic Ġfeature Ġsets Ġwe Ġhave Ġdescribed Ġto Ġbuild Ġa Ġclass ifier Ġthat Ġpredicts Ġdog mat ism Ġin Ġcomments . ĠA Ġpredictive Ġmodel Ġfurther Ġvalid ates Ġour Ġfeature Ġsets , Ġand Ġalso Ġallows Ġus Ġto Ġanalyze Ġdog mat ism Ġin Ġmillions Ġof Ġother ĠReddit Ġcomments Ġin Ġa Ġscalable Ġway , Ġwith Ġmultiple Ġuses Ġin Ġongoing , Ġdownstream Ġanalyses . </s> Pred iction Ġtask . ĠOur Ġgoal Ġis Ġ( 1 ) Ġto Ġunderstand Ġhow Ġwell Ġwe Ġcan Ġuse Ġthe Ġstrategies Ġin ĠSection Ġ3 Ġto Ġpredict Ġdog mat ism , Ġand Ġ( 2 ) Ġto Ġtest Ġthe Ġdomain - independence Ġof Ġthese Ġstrategies . ĠFirst , Ġwe Ġtest Ġthe Ġperformance Ġof Ġour Ġmodel Ġunder Ġcross - valid ation Ġwithin Ġthe ĠReddit Ġcomment Ġdataset . ĠWe Ġthen Ġevaluate Ġthe ĠReddit - based Ġmodel Ġon Ġa Ġheld Ġout Ġcorpus Ġof ĠNew ĠYork ĠTimes Ġcomments Ġannot ated Ġusing Ġthe Ġtechnique Ġin ĠSection Ġ2 . ĠWe Ġdid Ġnot Ġrefer Ġto Ġthis Ġsecond Ġdataset Ġduring Ġfeature Ġconstruction . </s> For Ġclassification , Ġwe Ġconsider Ġtwo Ġclasses Ġof Ġcomments : Ġdog matic Ġand Ġnon - dog matic . ĠAs Ġin Ġthe Ġprior Ġanalysis , Ġwe Ġdraw Ġthese Ġcomments Ġfrom Ġthe Ġtop Ġand Ġbottom Ġquart iles Ġof Ġthe Ġdog mat ism Ġdistribution . ĠThis Ġmeans Ġthe Ġclasses Ġare Ġbalanced , Ġwith Ġ2 , 500 Ġtotal Ġcomments Ġin Ġthe ĠReddit Ġtraining Ġdata Ġand Ġ500 Ġtotal Ġcomments Ġin Ġthe ĠNew ĠYork ĠTimes Ġtesting Ġdata . </s> We Ġcompare Ġthe Ġpredictions Ġof Ġlog istic Ġregression Ġmodels Ġbased Ġon Ġun ig ram Ġbag - of - words Ġfeatures Ġ( B OW ), Ġsentiment Ġsignals Ġ( S ENT ), Ġthe Ġlinguistic Ġfeatures Ġfrom Ġour Ġearlier Ġanalyses Ġ( L ING ), Ġand Ġcombinations Ġof Ġthese Ġfeatures . ĠB OW Ġand ĠS ENT Ġprovide Ġbas elines Ġfor Ġthe Ġtask . ĠWe Ġcompute ĠB OW Ġfeatures Ġusing Ġterm Ġfrequency - in verse Ġdocument Ġfrequency Ġ( TF - ID F ) Ġand Ġcategory - based Ġfeatures Ġby Ġnormal izing Ġcounts Ġfor Ġeach Ġcategory Ġby Ġthe Ġnumber Ġof Ġwords Ġin Ġeach Ġdocument . ĠThe ĠB OW Ġclass ifiers Ġare Ġtrained Ġwith Ġregular ization Ġ( L 2 Ġpenalties Ġof Ġ1 . 5 ). </s> Class ification Ġresults . ĠWe Ġpresent Ġclassification Ġaccuracy Ġin ĠTable Ġ2 Ġ. ĠB OW Ġshows Ġan ĠA UC Ġof Ġ0 . 8 53 Ġwithin ĠReddit Ġand Ġ0 . 776 Ġon Ġthe Ġheld Ġout ĠNew ĠYork ĠTimes Ġcomments . ĠThe Ġlinguistic Ġfeatures Ġboost Ġclassification Ġresults Ġwithin ĠReddit Ġ( 0 . 88 1 ) Ġand Ġon Ġthe Ġheld Ġout ĠNew ĠYork ĠTimes Ġcomments Ġ( 0 . 79 1 ). ĠWhile Ġlinguistic Ġsignals Ġby Ġthemselves Ġprovide Ġstrong Ġpredictive Ġpower Ġ( 0 . 801 ĠA UC Ġwithin Ġdomain ), Ġsentiment Ġsignals Ġare Ġmuch Ġless Ġpredictive . </s> These Ġresults Ġsuggest Ġthat Ġlinguistic Ġfeatures Ġinspired Ġby Ġprior Ġefforts Ġin Ġpsychology Ġare Ġuseful Ġfor Ġpredicting Ġdog mat ism Ġin Ġpractice Ġand Ġgeneral ize Ġacross Ġnew Ġdomains . </s> Dog mat ism Ġin Ġthe ĠReddit ĠCommunity Ġ </s> We Ġnow Ġapply Ġour Ġdog mat ism Ġclass ifier Ġto Ġa Ġlarger Ġdataset Ġof Ġposts , Ġexamining Ġhow Ġdog matic Ġlanguage Ġshapes Ġthe ĠReddit Ġcommunity . ĠCon crete ly , Ġwe Ġapply Ġthe ĠB OW + L ING Ġmodel Ġtrained Ġon Ġthe Ġfull ĠReddit Ġdataset Ġto Ġmillions Ġof Ġnew Ġun annot ated Ġposts , Ġlabeling Ġthese Ġposts Ġwith Ġa Ġprobability Ġof Ġdog mat ism Ġaccording Ġto Ġthe Ġclass ifier Ġ( 0 = non - dog matic , Ġ1 = dog matic ). ĠWe Ġthen Ġuse Ġthese Ġdog mat ism Ġannotations Ġto Ġaddress Ġfour Ġresearch Ġquestions . </s> What Ġsubreddits Ġhave Ġthe Ġhighest Ġand Ġlowest Ġlevels Ġof Ġdog mat ism ? Ġ( R 1 ) </s> A Ġnatural Ġstarting Ġpoint Ġfor Ġanalyzing Ġdog mat ism Ġon ĠReddit Ġis Ġto Ġexamine Ġhow Ġit Ġcharacter izes Ġthe Ġsite 's Ġsub - commun ities . ĠFor Ġexample , Ġwe Ġmight Ġexpect Ġto Ġsee Ġthat Ġsubreddits Ġoriented Ġaround Ġtopics Ġsuch Ġas Ġabortion Ġor Ġclimate Ġchange Ġare Ġmore Ġdog matic , Ġand Ġsubreddits Ġabout Ġcooking Ġare Ġless Ġso . </s> To Ġanswer Ġthis Ġquestion , Ġwe Ġrandomly Ġsample Ġ1 . 6 Ġmillion Ġposts Ġfrom Ġthe Ġentire ĠReddit Ġcommunity Ġbetween Ġ2007 Ġand Ġ2015 . ĠWe Ġthen Ġannot ate Ġeach Ġof Ġthese Ġposts Ġwith Ġdog mat ism Ġusing Ġour Ġclass ifier , Ġand Ġcompute Ġthe Ġaverage Ġdog mat ism Ġlevel Ġfor Ġeach Ġsubreddit Ġin Ġthe Ġsample Ġwith Ġat Ġleast Ġ100 Ġposts . </s> We Ġpresent Ġthe Ġresults Ġof Ġthis Ġanalysis Ġin ĠTable Ġ3 Ġ. ĠThe Ġsubreddits Ġwith Ġthe Ġhighest Ġlevels Ġof Ġdog mat ism Ġtend Ġto Ġbe Ġoriented Ġaround Ġpolitics Ġand Ġreligion Ġ( Deb ate A Christian Ġor Ġu k politics ), Ġwhile Ġthose Ġwith Ġthe Ġlowest Ġlevels Ġtend Ġto Ġfocus Ġon Ġhobbies Ġ( phot ography Ġor Ġhomebrew ing ). ĠThe Ġsubreddit Ġwith Ġthe Ġhighest Ġaverage Ġdog mat ism Ġlevel , Ġcr ing ep ics , Ġis Ġa Ġplace Ġto Ġmake Ġfun Ġof Ġsocially Ġawkward Ġmessages , Ġoften Ġfrom Ġwould - be Ġromantic Ġpartners . ĠDog mat ism Ġhere Ġtends Ġto Ġtake Ġthe Ġform Ġof ĠâĢ ľ how Ġcould Ġsomeone Ġbe Ġthat Ġstupid âĢ Ŀ Ġand Ġis Ġdirected Ġat Ġthe Ġsubject Ġof Ġthe Ġpost , Ġas Ġopposed Ġto Ġother Ġmembers Ġof Ġthe Ġcommunity . </s> Similarly , ĠSub reddit D rama Ġis Ġa Ġcommunity Ġwhere Ġpeople Ġcome Ġto Ġtalk Ġabout Ġfights Ġon Ġthe Ġinternet Ġor Ġsocial Ġmedia . ĠThese Ġfights Ġare Ġoften Ġthen Ġextended Ġin Ġdiscussion , Ġfor Ġexample : ĠâĢ ľ If Ġthe Ġbest Ġyou Ġcan Ġcome Ġup Ġwith Ġis Ġthat Ġsomething Ġyou Ġdid Ġwas Ġlegal , Ġit 's Ġprobably Ġtime Ġto Ġown Ġup Ġto Ġbeing Ġan Ġass . âĢ Ŀ ĠThe Ġpresence Ġof Ġthis Ġsubreddit Ġin Ġour Ġanalysis Ġprovides Ġa Ġfurther Ġsanity Ġcheck Ġthat Ġour Ġmodel Ġis Ġcapturing Ġa Ġrobust Ġsignal Ġof Ġdog mat ism . </s> How Ġdo Ġdog matic Ġbeliefs Ġcluster ? Ġ( R 2 ) </s> Dog mat ism Ġis Ġwidely Ġconsidered Ġto Ġbe Ġa Ġdomain - specific Ġattitude Ġ( for Ġexample , Ġoriented Ġtowards Ġreligion Ġor Ġpolitics ) Ġas Ġopposed Ġto Ġa Ġdeeper Ġpersonality Ġtrait ĠB IB REF 1 Ġ. ĠHere Ġwe Ġuse ĠReddit Ġas Ġa Ġlens Ġto Ġexamine Ġthis Ġidea Ġmore Ġclosely . ĠAre Ġusers Ġwho Ġare Ġdog matic Ġabout Ġone Ġtopic Ġlikely Ġto Ġbe Ġdog matic Ġabout Ġothers ? ĠDo Ġclusters Ġof Ġdog mat ism Ġexist Ġaround Ġparticular Ġtopics ? ĠTo Ġfind Ġout , Ġwe Ġexamine Ġthe Ġrelationships Ġbetween Ġsubreddits Ġover Ġwhich Ġindividual Ġusers Ġare Ġdog matic . ĠFor Ġexample , Ġif Ġmany Ġusers Ġoften Ġpost Ġdog matic Ġcomments Ġon Ġboth Ġthe Ġpolitics Ġand ĠChristianity Ġsubreddits , Ġbut Ġless Ġoften Ġon Ġworld news , Ġthat Ġwould Ġsuggest Ġpolitics Ġand ĠChristianity Ġare Ġlinked Ġper Ġa Ġboost Ġin Ġlikelihood Ġof Ġindividuals Ġbeing Ġdog matic Ġin Ġboth . </s> We Ġsample Ġ1000 ĠReddit Ġusers Ġwho Ġposted Ġat Ġleast Ġonce Ġa Ġyear Ġbetween Ġ2007 Ġand Ġ2015 Ġto Ġconstruct Ġa Ġcorpus Ġof Ġ10 Ġmillion Ġposts Ġthat Ġconstitute Ġtheir Ġentire Ġpost Ġhistory . ĠWe Ġthen Ġannot ate Ġthese Ġposts Ġusing Ġthe Ġclass ifier Ġand Ġcompute Ġthe Ġaverage Ġdog mat ism Ġscore Ġper Ġsubreddit Ġper Ġuser . ĠFor Ġexample , Ġone Ġuser Ġmight Ġhave Ġan Ġaverage Ġdog mat ism Ġlevel Ġof Ġ0 . 55 Ġfor Ġthe Ġpolitics Ġsubreddit Ġand Ġ0 . 45 Ġfor Ġthe Ġeconomics Ġsubreddit . ĠMost Ġusers Ġdo Ġnot Ġpost Ġin Ġall Ġsubreddits , Ġso Ġwe Ġtrack Ġonly Ġsubreddits Ġfor Ġwhich Ġa Ġuser Ġhad Ġposted Ġat Ġleast Ġ10 Ġtimes . ĠAny Ġsubreddits Ġwith Ġan Ġaverage Ġdog mat ism Ġscore Ġhigher Ġthan Ġ0 . 50 Ġwe Ġconsider Ġto Ġbe Ġa Ġuser 's Ġdog matic Ġsubreddits . ĠWe Ġthen Ġcount Ġall Ġpairs Ġof Ġthese Ġdog matic Ġsubreddits . ĠFor Ġexample , Ġ45 Ġusers Ġhave Ġpolitics Ġand Ġtechnology Ġamong Ġtheir Ġdog matic Ġsubreddits , Ġso Ġwe Ġconsider Ġpolitics Ġand Ġtechnology Ġas Ġlinked Ġ45 Ġtimes . ĠWe Ġcompute Ġthe Ġmutual Ġinformation ĠB IB REF 13 Ġbetween Ġthese Ġlinks , Ġwhich Ġgives Ġus Ġa Ġmeasure Ġof Ġthe Ġsubreddits Ġthat Ġare Ġmost Ġrelated Ġthrough Ġdog mat ism . </s> We Ġpresent Ġthe Ġresults Ġof Ġthis Ġanalysis Ġin ĠTable Ġ4 Ġ, Ġchoosing Ġclusters Ġthat Ġrepresent Ġa Ġdiverse Ġset Ġof Ġtopics . ĠFor Ġexample , ĠLibertarian ism Ġis Ġlinked Ġthrough Ġdog mat ism Ġto Ġother Ġpolitical Ġcommunities Ġlike ĠAnarch o _ Capital ism , Ġr on p aul , Ġor Ġu k politics , Ġas Ġwell Ġas Ġother Ġtopical Ġsubreddits Ġlike Ġguns Ġor Ġeconomy . ĠSimilarly , Ġpeople Ġwho Ġare Ġdog matic Ġin Ġthe Ġbusiness Ġsubreddit Ġalso Ġtend Ġto Ġbe Ġdog matic Ġin Ġsubreddits Ġfor ĠBitcoin , Ġsocialism , Ġand Ġtechnology . ĠNotably , Ġwhen Ġwe Ġapply Ġthe Ġsame Ġmutual Ġinformation Ġanalysis Ġto Ġlinks Ġdefined Ġby Ġsubreddits Ġposted Ġin Ġby Ġthe Ġsame Ġuser , Ġwe Ġsee Ġdramatically Ġdifferent Ġresults . ĠFor Ġexample , Ġthe Ġsubreddits Ġmost Ġlinked Ġto Ġscience Ġthrough Ġuser Ġposts Ġare ĠU pl ifting News , Ġphot oshop b att les , Ġand Ġfirst world an arch ist , Ġand Ġmillionaire makers . </s> Finally , Ġwe Ġsee Ġless Ġobvious Ġconnections Ġbetween Ġsubreddits Ġthat Ġsuggest Ġsome Ġpeople Ġmay Ġbe Ġdog matic Ġby Ġnature . ĠFor Ġexample , Ġamong Ġthe Ġusers Ġwho Ġare Ġdog matic Ġon Ġpolitics , Ġthey Ġare Ġalso Ġdisproportionately Ġdog matic Ġon Ġunrelated Ġsubreddits Ġsuch Ġas Ġscience Ġ( Ġ$ p < 0 . 001 $ Ġ), Ġtechnology Ġ( Ġ$ p < 0 . 001 $ Ġ), ĠI Am A Ġ( Ġ$ p < 0 . 001 $ Ġ), Ġand ĠAsk Reddit Ġ( Ġ$ p < 0 . 05 $ Ġ), Ġwith Ġp - values Ġcomputed Ġunder Ġa Ġbin omial Ġtest . </s> What Ġuser Ġbehaviors Ġare Ġpredictive Ġof Ġdog mat ism ? Ġ( R 3 ) </s> We Ġhave Ġshown Ġdog mat ism Ġis Ġcaptured Ġby Ġmany Ġlinguistic Ġfeatures , Ġbut Ġcan Ġwe Ġdiscover Ġother Ġhigh - level Ġuser Ġbehaviors Ġthat Ġare Ġsimilarly Ġpredictive ? </s> To Ġfind Ġout , Ġwe Ġcompute Ġmetrics Ġof Ġuser Ġbehavior Ġusing Ġthe Ġdata Ġsample Ġof Ġ1000 Ġusers Ġand Ġ10 Ġmillion Ġposts Ġdescribed Ġin ĠSection Ġ5 . 2 . ĠSpecifically , Ġwe Ġcalculate Ġ( 1 ) Ġactivity : Ġa Ġuser 's Ġtotal Ġnumber Ġof Ġposts , Ġ( 2 ) Ġbreadth : Ġthe Ġnumber Ġof Ġsubreddits Ġa Ġuser Ġhas Ġposted Ġin , Ġ( 3 ) Ġfocus : Ġthe Ġproportion Ġof Ġa Ġuser 's Ġposts Ġthat Ġappear Ġin Ġthe Ġsubreddit Ġwhere Ġthey Ġare Ġmost Ġactive , Ġand Ġ( 4 ) Ġengagement : Ġthe Ġaverage Ġnumber Ġof Ġposts Ġa Ġuser Ġcontributes Ġto Ġeach Ġdiscussion Ġthey Ġengage Ġin . ĠWe Ġthen Ġfit Ġthese Ġbehavioral Ġfeatures Ġto Ġa Ġlinear Ġregression Ġmodel Ġwhere Ġwe Ġpredict Ġeach Ġuser 's Ġaverage Ġdog mat ism Ġlevel . ĠPositive Ġcoefficients Ġin Ġthis Ġmodel Ġare Ġpositively Ġpredictive Ġof Ġdog mat ism , Ġwhile Ġnegative Ġcoefficients Ġare Ġnegatively Ġpredictive . ĠWe Ġfind Ġthis Ġmodel Ġis Ġsignificantly Ġpredic itive Ġof Ġdog mat ism Ġ( Ġ$ R ^ 2 = 0 . 1 $ Ġ, Ġ$ p < 0 . 001 $ Ġ), Ġwith Ġall Ġfeatures Ġreaching Ġstatistical Ġsignificance Ġ( Ġ$ p < 0 . 001 $ Ġ). ĠActivity Ġand Ġfocus Ġare Ġpositively Ġassociated Ġwith Ġdog mat ism , Ġwhile Ġbreadth Ġand Ġengagement Ġare Ġnegatively Ġassociated Ġ( Table Ġ5 Ġ). ĠTogether , Ġthese Ġresults Ġsuggest Ġdog matic Ġusers Ġtend Ġto Ġpost Ġfrequently Ġand Ġin Ġspecific Ġcommunities , Ġbut Ġare Ġnot Ġas Ġinclined Ġto Ġcontinue Ġto Ġengage Ġwith Ġa Ġdiscussion , Ġonce Ġit Ġhas Ġbegun . </s> How Ġdoes Ġdog mat ism Ġimpact Ġa Ġconversation ? Ġ( R 4 ) </s> How Ġdoes Ġinteracting Ġwith Ġa Ġdog matic Ġcomment Ġimpact Ġa Ġconversation ? ĠAre Ġusers Ġable Ġto Ġshrug Ġit Ġoff ? ĠOr Ġdo Ġotherwise Ġnon - dog matic Ġusers Ġbecome Ġmore Ġdog matic Ġthemselves ? </s> To Ġanswer Ġthis Ġquestion , Ġwe Ġsample Ġ600 , 000 Ġconversations Ġtri ples Ġfrom ĠReddit . ĠThese Ġconversations Ġconsist Ġof Ġtwo Ġpeople Ġ( A Ġand ĠB ) Ġtalking , Ġwith Ġthe Ġstructure : ĠA 1 Ġ$\ right arrow Ġ$ ĠB Ġ$\ right arrow Ġ$ ĠA 2 . ĠThis Ġallows Ġus Ġto Ġmeasure Ġthe Ġimpact Ġof ĠB 's Ġdog mat ism Ġon ĠA 's Ġresponse , Ġwhile Ġalso Ġcontrolling Ġfor Ġthe Ġdog mat ism Ġlevel Ġinitially Ġset Ġby ĠA . ĠCon crete ly , Ġwe Ġmodel Ġthe Ġimpact Ġof Ġdog mat ism Ġon Ġthese Ġconversations Ġthrough Ġa Ġlinear Ġregression . ĠThis Ġmodel Ġtakes Ġtwo Ġfeatures , Ġthe Ġdog mat ism Ġlevels Ġof ĠA 1 Ġand ĠB , Ġand Ġpredicts Ġthe Ġdog mat ism Ġresponse Ġof ĠA 2 . ĠIf ĠB 's Ġdog mat ism Ġhas Ġno Ġeffect Ġon ĠA 's Ġresponse , Ġthe Ġcoefficient Ġthat Ġcorresponds Ġto ĠB Ġwill Ġnot Ġbe Ġsignificant Ġin Ġthe Ġmodel . ĠAlternatively , Ġif ĠB 's Ġdog mat ism Ġdoes Ġhave Ġsome Ġeffect , Ġit Ġwill Ġbe Ġcaptured Ġby Ġthe Ġmodel 's Ġcoefficient . </s> We Ġfind Ġthe Ġcoefficient Ġof Ġthe ĠB Ġfeature Ġin Ġthe Ġmodel Ġis Ġpositively Ġassociated Ġwith Ġdog mat ism Ġ( Ġ$ p < 0 . 001 $ Ġ). ĠIn Ġother Ġwords , Ġengagement Ġwith Ġa Ġdog matic Ġcomment Ġtends Ġto Ġmake Ġa Ġuser Ġmore Ġdog matic Ġthemselves . ĠThis Ġeffect Ġholds Ġwhen Ġwe Ġrun Ġthe Ġsame Ġmodel Ġon Ġdata Ġsubs ets Ġconsisting Ġonly Ġof Ġdog matic Ġor Ġnon - dog matic Ġusers , Ġand Ġalso Ġwhen Ġwe Ġconservative ly Ġremove Ġall Ġwords Ġused Ġby ĠB Ġfrom ĠA 's Ġresponse Ġ( i . e ., Ġcontrolling Ġfor Ġquoting Ġeffects ). </s> Related ĠWork </s> In Ġcontrast Ġto Ġthe Ġcomputational Ġmodels Ġwe Ġhave Ġpresented , Ġdog mat ism Ġis Ġusually Ġmeasured Ġin Ġpsychology Ġthrough Ġsurvey Ġscales , Ġin Ġwhich Ġstudy Ġparticipants Ġanswer Ġquestions Ġdesigned Ġto Ġreveal Ġunderlying Ġpersonality Ġattributes ĠB IB REF 1 Ġ. ĠOver Ġtime , Ġthese Ġsurveys Ġhave Ġbeen Ġupdated ĠB IB REF 14 Ġand Ġimproved Ġto Ġmeet Ġstandards Ġof Ġpsych ometric Ġvalidity ĠB IB REF 15 Ġ. </s> These Ġsurveys Ġare Ġoften Ġused Ġto Ġstudy Ġthe Ġrelationship Ġbetween Ġdog mat ism Ġand Ġother Ġpsychological Ġphenomena . ĠFor Ġexample , Ġdog matic Ġpeople Ġtend Ġto Ġshow Ġan Ġincreased Ġtendency Ġfor Ġconfrontation ĠB IB REF 16 Ġor Ġmoral Ġconviction Ġand Ġrelig iosity ĠB IB REF 17 Ġ, Ġand Ġless Ġlikelihood Ġof Ġcognitive Ġflexibility ĠB IB REF 18 Ġ, Ġeven Ġamong Ġstereotyp ically Ġnon - dog matic Ġgroups Ġlike Ġatheists ĠB IB REF 19 Ġ. ĠFrom Ġa Ġbehavioral Ġstandpoint , Ġdog matic Ġpeople Ġsolve Ġproblems Ġdifferently , Ġspending Ġless Ġtime Ġframing Ġa Ġproblem Ġand Ġexpressing Ġmore Ġcertainty Ġin Ġtheir Ġsolution ĠB IB REF 20 Ġ. ĠHere Ġwe Ġsimilarly Ġexamine Ġhow Ġuser Ġbehaviors Ġon ĠReddit Ġrelate Ġto Ġa Ġlanguage Ġmodel Ġof Ġdog mat ism . </s> Er tel Ġsought Ġto Ġcapture Ġdog mat ism Ġlingu istically , Ġthough Ġa Ġsmall Ġlex icon Ġof Ġwords Ġthat Ġcorrespond Ġwith Ġhigh - level Ġconcepts Ġlike Ġcertainty Ġand Ġcompromise Ġd ota . ĠMcK enny Ġthen Ġused Ġthis Ġdictionary Ġto Ġrelate Ġdog mat ism Ġto Ġargument Ġquality Ġin Ġstudent Ġessays Ġdog mat ism - ess ays . ĠOur Ġwork Ġexpands Ġon Ġthis Ġapproach , Ġapplying Ġsupervised Ġmodels Ġbased Ġon Ġa Ġbroader Ġset Ġof Ġlinguistic Ġcategories Ġto Ġidentify Ġdog mat ism Ġin Ġtext . </s> Other Ġresearchers Ġhave Ġstudied Ġtopics Ġsimilar Ġto Ġdog mat ism , Ġsuch Ġas Ġsignals Ġof Ġcognitive Ġstyle Ġin Ġright - wing Ġpolitical Ġthought ĠB IB REF 21 Ġ, Ġthe Ġlanguage Ġused Ġby Ġtrolls Ġon Ġsocial Ġmedia ĠB IB REF 22 Ġ, Ġor Ġwhat Ġmakes Ġfor Ġimpartial Ġlanguage Ġon Ġtwitter ĠB IB REF 23 Ġ. ĠA Ġsimilar Ġflavor Ġof Ġwork Ġhas Ġexamined Ġlinguistic Ġmodels Ġthat Ġcapture Ġpolit eness ĠB IB REF 2 Ġ, Ġdeception ĠB IB REF 24 Ġ, Ġand Ġauthority ĠB IB REF 8 Ġ. ĠWe Ġtook Ġinspiration Ġfrom Ġthese Ġmodels Ġwhen Ġconstructing Ġthe Ġfeature Ġsets Ġin Ġour Ġwork . </s> Finally , Ġwhile Ġwe Ġexamine Ġwhat Ġmakes Ġan Ġopinion Ġdog matic , Ġother Ġwork Ġhas Ġpushed Ġfurther Ġinto Ġthe Ġstructure Ġof Ġarguments , Ġfor Ġexample Ġclass ifying Ġtheir Ġjust ifications ĠB IB REF 25 Ġ, Ġor Ġwhat Ġmakes Ġan Ġargument Ġlikely Ġto Ġwin ĠB IB REF 26 Ġ. ĠOur Ġmodel Ġmay Ġallow Ġfuture Ġresearchers Ġto Ġprobe Ġthese Ġquestions Ġmore Ġdeeply . </s> Conclusion </s> We Ġhave Ġconstructed Ġthe Ġfirst Ġcorpus Ġof Ġsocial Ġmedia Ġposts Ġannot ated Ġwith Ġdog mat ism Ġscores , Ġallowing Ġus Ġto Ġexplore Ġlinguistic Ġfeatures Ġof Ġdog mat ism Ġand Ġbuild Ġa Ġpredictive Ġmodel Ġthat Ġanaly zes Ġnew Ġcontent . ĠWe Ġapply Ġthis Ġmodel Ġto ĠReddit , Ġwhere Ġwe Ġdiscover Ġbehavioral Ġpredict ors Ġof Ġdog mat ism Ġand Ġtopical Ġpatterns Ġin Ġthe Ġcomments Ġof Ġdog matic Ġusers . </s> Could Ġwe Ġuse Ġthis Ġcomputational Ġmodel Ġto Ġhelp Ġusers Ġshed Ġtheir Ġdog matic Ġbeliefs ? ĠLooking Ġforward , Ġour Ġwork Ġmakes Ġpossible Ġnew Ġavenues Ġfor Ġencouraging Ġpro - social Ġbehavior Ġin Ġonline Ġcommunities .
Introduction “I'm supposed to trust the opinion of a MS minion? The people that produced Windows ME, Vista and 8? They don't even understand people, yet they think they can predict the behavior of new, self-guiding AI?” –anonymous “I think an AI would make it easier for Patients to confide their information because by nature, a robot cannot judge them. Win-win? :D”' –anonymous Dogmatism describes the tendency to lay down opinions as incontrovertibly true, without respect for conflicting evidence or the opinions of others BIBREF0 . Which user is more dogmatic in the examples above? This question is simple for humans. Phrases like “they think” and “they don't even understand,” suggest an intractability of opinion, while “I think” and “win-win?” suggest the opposite. Can we train computers to draw similar distinctions? Work in psychology has called out many aspects of dogmatism that can be modeled computationally via natural language, such as over-confidence and strong emotions BIBREF1 . We present a statistical model of dogmatism that addresses two complementary goals. First, we validate psychological theories by examining the predictive power of feature sets that guide the model's predictions. For example, do linguistic signals of certainty help to predict a post is dogmatic, as theory would suggest? Second, we apply our model to answer four questions: R1: What kinds of topics (e.g., guns, LGBT) attract the highest levels of dogmatism? R2: How do dogmatic beliefs cluster? R3: How does dogmatism influence a conversation on social media? R4: How do other user behaviors (e.g., frequency and breadth of posts) relate to dogmatism? We train a predictive model to classify dogmatic posts from Reddit, one of the most popular discussion communities on the web. Posts on Reddit capture discussion and debate across a diverse set of domains and topics – users talk about everything from climate change and abortion, to world news and relationship advice, to the future of artificial intelligence. As a prerequisite to training our model, we have created a corpus of 5,000 Reddit posts annotated with levels of dogmatism, which we are releasing to share with other researchers. Using the model, we operationalize key domain-independent aspects of psychological theories of dogmatism drawn from the literature. We find these features have predictive power that largely supports the underlying theory. For example, posts that use less confident language tend to be less dogmatic. We also discover evidence for new attributes of dogmatism. For example, dogmatic posts tend not to verbalize cognition, through terms such as “I think,” “possibly,” or “might be.” Our model is trained on only 5,000 annotated posts, but once trained, we use it to analyze millions of other Reddit posts to answer our research questions. We find a diverse set of topics are colored by dogmatic language (e.g., people are dogmatic about religion, but also about LGBT issues). Further, we find some evidence for dogmatism as a deeper personality trait – people who are strongly dogmatic about one topic are more likely to express dogmatic views about others as well. Finally, in conversation, we discover that one user's dogmatism tends to bring out dogmatism in their conversational partner, forming a vicious cycle. Dogmatism data Posts on Reddit capture debate and discussion across a diverse set of topics, making them a natural starting point for untangling domain-independent linguistic features of dogmatism. Data collection. Subreddits are sub-communities on Reddit oriented around specific interests or topics, such as technology or politics. Sampling from Reddit as a whole would bias the model towards the most commonly discussed content. But by sampling posts from individual subreddits, we can control the kinds of posts we use to train our model. To collect a diverse training dataset, we have randomly sampled 1000 posts each from the subreddits politics, business, science, and AskReddit, and 1000 additional posts from the Reddit frontpage. All posts in our sample appeared between January 2007 and March 2015, and to control for length effects, contain between 300 and 400 characters. This results in a total training dataset of 5000 posts. Dogmatism annotations. Building a useful computational model requires labeled training data. We labeled the Reddit dataset using crowdworkers on Amazon Mechanical Turk (AMT), creating the first public corpus annotated with levels of dogmatism. We asked crowdworkers to rate levels of dogmatism on a 5-point Likert scale, as supported by similar annotation tasks in prior work BIBREF2 . Concretely, we gave crowdworkers the following task: Given a comment, imagine you hold a well-informed, different opinion from the commenter in question. We'd like you to tell us how likely that commenter would be to engage you in a constructive conversation about your disagreement, where you each are able to explore the other's beliefs. The options are: (5): It's unlikely you'll be able to engage in any substantive conversation. When you respectfully express your disagreement, they are likely to ignore you or insult you or otherwise lower the level of discourse. (4): They are deeply rooted in their opinion, but you are able to exchange your views without the conversation degenerating too much. (3): It's not likely you'll be able to change their mind, but you're easily able to talk and understand each other's point of view. (2): They may have a clear opinion about the subject, but would likely be open to discussing alternative viewpoints. (1): They are not set in their opinion, and it's possible you might change their mind. If the comment does not convey an opinion of any kind, you may also select this option. To ensure quality work, we restricted the task to Masters workers and provided examples corresponding to each point on the scale. Including examples in a task has been shown to significantly increase the agreement and quality of crowdwork BIBREF3 . For instance, here is an example of a highly dogmatic (5) comment: I won't be happy until I see the executive suite of BofA, Wells, and all the others, frog-marched into waiting squad cars. It's ALREADY BEEN ESTABLISHED that... And a minimally dogmatic (1) comment: I agree. I would like to compile a playlist for us trance yogi's, even if you just would like to experiment with it. Is there any preference on which platform to use? Each comment has been annotated by three independent workers on AMT, which is enough to produce reliable results in most labeling tasks BIBREF4 . To compute an aggregate measure of dogmatism for each comment, we summed the scores of all three workers. We show the resulting distribution of annotations in Figure 1 . Inter-annotator agreement. To evaluate the reliability of annotations we compute Krippendorff's $\alpha $ , a measure of agreement designed for variable levels of measurement such as a Likert scale BIBREF5 . An $\alpha $ of 0 indicates agreement indistinguishable from chance, while an $\alpha $ of 1 indicates perfect agreement. Across all annotations we find $\alpha =0.44$ . While workers agree much more than chance, clearly dogmatism is also subjective. In fact, when we examine only the middle two quartiles of the dogmatism annotations, we find agreement is no better than chance. Alternatively, when we measure agreement only among the top and bottom quartiles of annotations, we find agreement of $\alpha =0.69$ . This suggests comments with scores that are only slightly dogmatic are unreliable and often subject to human disagreement. For this reason, we use only the top and bottom quartiles of comments when training our model. Approaches to Identifying Dogmatism We now consider strategies for identifying dogmatism based on prior work in psychology. We start with the Linguistic Inquiry and Word Count (LIWC), a lexicon popular in the social sciences BIBREF6 . LIWC provides human validated lists of words that correspond to high-level psychological categories such as certainty or perception. In other studies, LIWC has uncovered linguistic signals relating to politeness BIBREF2 , deception BIBREF7 , or authority in texts BIBREF8 . Here, we examine how dogmatism relates to 17 of LIWC's categories (Table 1 ). To compute the relationships between LIWC categories and dogmatism, we first count the relevant category terms that appear in each annotated Reddit comment, normalized by its word count. We then calculate odds ratios on the aggregate counts of each LIWC category over the top and bottom quartiles of dogmatic comments. As we have discussed, using the top and bottom quartiles of comments provides a more reliable signal of dogmatism. We check for significant differences in categories between dogmatic and non-dogmatic comments using the Mann-Whitney U test and apply Holmes method for correction. All odds we report in this section are significant after correction. Dogmatic statements tend to express a high degree of certainty BIBREF1 . Here we consider LIWC categories that express certainty both positively (certainty) and negatively (tentativeness). For example, the word “always” is certain, while “possibly” is tentative. Conforming to existing theory, certainty is more associated with dogmatic comments (1.52 odds), while tentativeness is more associated with the absence of dogmatism (0.88 odds). Terms used to verbalize cognition can act as a hedge that often characterizes non-dogmatic language. LIWC's insight category captures this effect through words such as “think,” “know,” or “believe.” These words add nuance to a statement BIBREF9 , signaling it is the product of someone's mind (“I think you should give this paper a good review”) and not meant to be interpreted as an objective truth. Along these lines, we find the use of terms in the insight category is associated with non-dogmatic comments (0.83 odds). Sensory language, with its focus on description and detail, often signals a lack of any kind of opinion, dogmatic or otherwise. LIWC's perception category captures this idea through words associated with hearing, feeling, or seeing. For example, these words might occur when recounting a personal experience (“I saw his incoming fist”), which even if emotionally charged or negative, is less likely to be dogmatic. We find perception is associated with non-dogmatic comments at 0.77 odds. Drawing comparisons or qualifying something as relative to something else conveys a nuance that is absent from traditionally dogmatic language. The LIWC categories comparison and relativity capture these effects through comparison words such as “than” or “as” and qualifying words such as “during” or “when.” For example, the statement “I hate politicians” is more dogmatic than “I hate politicians when they can't get anything done.' Relativity is associated with non-dogmatic comments at 0.80 odds, but comparison does not reach significance. Pronouns can be surprisingly revealing indicators of language: for example, signaling one's gender or hierarchical status in a conversation BIBREF10 . We find first person singular pronouns are a useful negative signal for dogmatism (0.46 odds), while second person singular pronouns (2.18 odds) and third person plural (1.63 odds) are a useful positive signal. Looking across the corpus, we see I often used with a hedge (“I think” or “I know”), while you and they tend to characterize the beliefs of others, often in a strongly opinionated way (“you are a moron” or “they are keeping us down”). Other pronoun types do not show significant relationships. Like pronouns, verb tense can reveal subtle signals in language use, such as the tendency of medical inpatients to focus on the past BIBREF11 . On social media, comments written in the present tense are more likely to be oriented towards a user's current interaction (“this is all so stupid”), creating opportunities to signal dogmatism. Alternatively, comments in the past tense are more likely to refer to outside experiences (“it was an awful party”), speaking less to a user's stance towards an ongoing discussion. We find present tense is a positive signal for dogmatism (1.11 odds) and past tense is a negative signal (0.69 odds). Dogmatic language can be either positively or negatively charged in sentiment: for example, consider the positive statement “Trump is the SAVIOR of this country!!!” or the negative statement “Are you REALLY that stupid?? Education is the only way out of this horrible mess. It's hard to imagine how anyone could be so deluded.” In diverse communities, where people hold many different kinds of opinions, dogmatic opinions will often tend to come into conflict with one another BIBREF12 , producing a greater likelihood of negative sentiment. Perhaps for this reason, negative emotion (2.09 odds) and swearing (3.80 odds) are useful positive signals of dogmatism, while positive emotion shows no significant relationship. Finally, we find that interrogative language (1.12 odds) and negation (1.35 odds) are two additional positive signals of dogmatism. While interrogative words like “how” or “what” have many benign uses, they disproportionately appear in our data in the form of rhetorical or emotionally charged questions, such as “how can anyone be that dumb?” Many of these linguistic signals are correlated with each other, suggesting that dogmatism is the cumulative effect of many component relationships. For example, consider the relatively non-dogmatic statement: “I think the reviewers are wrong in this instance.” Removing signals of insight, we have: “the reviewers are wrong in this instance,” which is slightly more dogmatic. Then removing relativity, we have: “the reviewers are wrong.” And finally, adding certainty, we have a dogmatic statement: “the reviewers are always wrong.” Predicting dogmatism We now show how we can use the linguistic feature sets we have described to build a classifier that predicts dogmatism in comments. A predictive model further validates our feature sets, and also allows us to analyze dogmatism in millions of other Reddit comments in a scalable way, with multiple uses in ongoing, downstream analyses. Prediction task. Our goal is (1) to understand how well we can use the strategies in Section 3 to predict dogmatism, and (2) to test the domain-independence of these strategies. First, we test the performance of our model under cross-validation within the Reddit comment dataset. We then evaluate the Reddit-based model on a held out corpus of New York Times comments annotated using the technique in Section 2. We did not refer to this second dataset during feature construction. For classification, we consider two classes of comments: dogmatic and non-dogmatic. As in the prior analysis, we draw these comments from the top and bottom quartiles of the dogmatism distribution. This means the classes are balanced, with 2,500 total comments in the Reddit training data and 500 total comments in the New York Times testing data. We compare the predictions of logistic regression models based on unigram bag-of-words features (BOW), sentiment signals (SENT), the linguistic features from our earlier analyses (LING), and combinations of these features. BOW and SENT provide baselines for the task. We compute BOW features using term frequency-inverse document frequency (TF-IDF) and category-based features by normalizing counts for each category by the number of words in each document. The BOW classifiers are trained with regularization (L2 penalties of 1.5). Classification results. We present classification accuracy in Table 2 . BOW shows an AUC of 0.853 within Reddit and 0.776 on the held out New York Times comments. The linguistic features boost classification results within Reddit (0.881) and on the held out New York Times comments (0.791). While linguistic signals by themselves provide strong predictive power (0.801 AUC within domain), sentiment signals are much less predictive. These results suggest that linguistic features inspired by prior efforts in psychology are useful for predicting dogmatism in practice and generalize across new domains. Dogmatism in the Reddit Community  We now apply our dogmatism classifier to a larger dataset of posts, examining how dogmatic language shapes the Reddit community. Concretely, we apply the BOW+LING model trained on the full Reddit dataset to millions of new unannotated posts, labeling these posts with a probability of dogmatism according to the classifier (0=non-dogmatic, 1=dogmatic). We then use these dogmatism annotations to address four research questions. What subreddits have the highest and lowest levels of dogmatism? (R1) A natural starting point for analyzing dogmatism on Reddit is to examine how it characterizes the site's sub-communities. For example, we might expect to see that subreddits oriented around topics such as abortion or climate change are more dogmatic, and subreddits about cooking are less so. To answer this question, we randomly sample 1.6 million posts from the entire Reddit community between 2007 and 2015. We then annotate each of these posts with dogmatism using our classifier, and compute the average dogmatism level for each subreddit in the sample with at least 100 posts. We present the results of this analysis in Table 3 . The subreddits with the highest levels of dogmatism tend to be oriented around politics and religion (DebateAChristian or ukpolitics), while those with the lowest levels tend to focus on hobbies (photography or homebrewing). The subreddit with the highest average dogmatism level, cringepics, is a place to make fun of socially awkward messages, often from would-be romantic partners. Dogmatism here tends to take the form of “how could someone be that stupid” and is directed at the subject of the post, as opposed to other members of the community. Similarly, SubredditDrama is a community where people come to talk about fights on the internet or social media. These fights are often then extended in discussion, for example: “If the best you can come up with is that something you did was legal, it's probably time to own up to being an ass.” The presence of this subreddit in our analysis provides a further sanity check that our model is capturing a robust signal of dogmatism. How do dogmatic beliefs cluster? (R2) Dogmatism is widely considered to be a domain-specific attitude (for example, oriented towards religion or politics) as opposed to a deeper personality trait BIBREF1 . Here we use Reddit as a lens to examine this idea more closely. Are users who are dogmatic about one topic likely to be dogmatic about others? Do clusters of dogmatism exist around particular topics? To find out, we examine the relationships between subreddits over which individual users are dogmatic. For example, if many users often post dogmatic comments on both the politics and Christianity subreddits, but less often on worldnews, that would suggest politics and Christianity are linked per a boost in likelihood of individuals being dogmatic in both. We sample 1000 Reddit users who posted at least once a year between 2007 and 2015 to construct a corpus of 10 million posts that constitute their entire post history. We then annotate these posts using the classifier and compute the average dogmatism score per subreddit per user. For example, one user might have an average dogmatism level of 0.55 for the politics subreddit and 0.45 for the economics subreddit. Most users do not post in all subreddits, so we track only subreddits for which a user had posted at least 10 times. Any subreddits with an average dogmatism score higher than 0.50 we consider to be a user's dogmatic subreddits. We then count all pairs of these dogmatic subreddits. For example, 45 users have politics and technology among their dogmatic subreddits, so we consider politics and technology as linked 45 times. We compute the mutual information BIBREF13 between these links, which gives us a measure of the subreddits that are most related through dogmatism. We present the results of this analysis in Table 4 , choosing clusters that represent a diverse set of topics. For example, Libertarianism is linked through dogmatism to other political communities like Anarcho_Capitalism, ronpaul, or ukpolitics, as well as other topical subreddits like guns or economy. Similarly, people who are dogmatic in the business subreddit also tend to be dogmatic in subreddits for Bitcoin, socialism, and technology. Notably, when we apply the same mutual information analysis to links defined by subreddits posted in by the same user, we see dramatically different results. For example, the subreddits most linked to science through user posts are UpliftingNews, photoshopbattles, and firstworldanarchist, and millionairemakers. Finally, we see less obvious connections between subreddits that suggest some people may be dogmatic by nature. For example, among the users who are dogmatic on politics, they are also disproportionately dogmatic on unrelated subreddits such as science ( $p<0.001$ ), technology ( $p<0.001$ ), IAmA ( $p<0.001$ ), and AskReddit ( $p<0.05$ ), with p-values computed under a binomial test. What user behaviors are predictive of dogmatism? (R3) We have shown dogmatism is captured by many linguistic features, but can we discover other high-level user behaviors that are similarly predictive? To find out, we compute metrics of user behavior using the data sample of 1000 users and 10 million posts described in Section 5.2. Specifically, we calculate (1) activity: a user's total number of posts, (2) breadth: the number of subreddits a user has posted in, (3) focus: the proportion of a user's posts that appear in the subreddit where they are most active, and (4) engagement: the average number of posts a user contributes to each discussion they engage in. We then fit these behavioral features to a linear regression model where we predict each user's average dogmatism level. Positive coefficients in this model are positively predictive of dogmatism, while negative coefficients are negatively predictive. We find this model is significantly predicitive of dogmatism ( $R^2=0.1$ , $p<0.001$ ), with all features reaching statistical significance ( $p<0.001$ ). Activity and focus are positively associated with dogmatism, while breadth and engagement are negatively associated (Table 5 ). Together, these results suggest dogmatic users tend to post frequently and in specific communities, but are not as inclined to continue to engage with a discussion, once it has begun. How does dogmatism impact a conversation? (R4) How does interacting with a dogmatic comment impact a conversation? Are users able to shrug it off? Or do otherwise non-dogmatic users become more dogmatic themselves? To answer this question, we sample 600,000 conversations triples from Reddit. These conversations consist of two people (A and B) talking, with the structure: A1 $\rightarrow $ B $\rightarrow $ A2. This allows us to measure the impact of B's dogmatism on A's response, while also controlling for the dogmatism level initially set by A. Concretely, we model the impact of dogmatism on these conversations through a linear regression. This model takes two features, the dogmatism levels of A1 and B, and predicts the dogmatism response of A2. If B's dogmatism has no effect on A's response, the coefficient that corresponds to B will not be significant in the model. Alternatively, if B's dogmatism does have some effect, it will be captured by the model's coefficient. We find the coefficient of the B feature in the model is positively associated with dogmatism ( $p<0.001$ ). In other words, engagement with a dogmatic comment tends to make a user more dogmatic themselves. This effect holds when we run the same model on data subsets consisting only of dogmatic or non-dogmatic users, and also when we conservatively remove all words used by B from A's response (i.e., controlling for quoting effects). Related Work In contrast to the computational models we have presented, dogmatism is usually measured in psychology through survey scales, in which study participants answer questions designed to reveal underlying personality attributes BIBREF1 . Over time, these surveys have been updated BIBREF14 and improved to meet standards of psychometric validity BIBREF15 . These surveys are often used to study the relationship between dogmatism and other psychological phenomena. For example, dogmatic people tend to show an increased tendency for confrontation BIBREF16 or moral conviction and religiosity BIBREF17 , and less likelihood of cognitive flexibility BIBREF18 , even among stereotypically non-dogmatic groups like atheists BIBREF19 . From a behavioral standpoint, dogmatic people solve problems differently, spending less time framing a problem and expressing more certainty in their solution BIBREF20 . Here we similarly examine how user behaviors on Reddit relate to a language model of dogmatism. Ertel sought to capture dogmatism linguistically, though a small lexicon of words that correspond with high-level concepts like certainty and compromise dota. McKenny then used this dictionary to relate dogmatism to argument quality in student essays dogmatism-essays. Our work expands on this approach, applying supervised models based on a broader set of linguistic categories to identify dogmatism in text. Other researchers have studied topics similar to dogmatism, such as signals of cognitive style in right-wing political thought BIBREF21 , the language used by trolls on social media BIBREF22 , or what makes for impartial language on twitter BIBREF23 . A similar flavor of work has examined linguistic models that capture politeness BIBREF2 , deception BIBREF24 , and authority BIBREF8 . We took inspiration from these models when constructing the feature sets in our work. Finally, while we examine what makes an opinion dogmatic, other work has pushed further into the structure of arguments, for example classifying their justifications BIBREF25 , or what makes an argument likely to win BIBREF26 . Our model may allow future researchers to probe these questions more deeply. Conclusion We have constructed the first corpus of social media posts annotated with dogmatism scores, allowing us to explore linguistic features of dogmatism and build a predictive model that analyzes new content. We apply this model to Reddit, where we discover behavioral predictors of dogmatism and topical patterns in the comments of dogmatic users. Could we use this computational model to help users shed their dogmatic beliefs? Looking forward, our work makes possible new avenues for encouraging pro-social behavior in online communities.
what are the topics pulled from Reddit?
b6ae8e10c6a0d34c834f18f66ab730b670fb528c
what Ġare Ġthe Ġtopics Ġpulled Ġfrom ĠReddit ?
politics , Ġbusiness , Ġscience , Ġand ĠAsk Reddit , Ġand Ġ1000 Ġadditional Ġposts Ġfrom Ġthe ĠReddit Ġfront page . Ġ
<s> what Ġare Ġthe Ġtopics Ġpulled Ġfrom ĠReddit ? </s> Introduction </s> âĢ ľ I 'm Ġsupposed Ġto Ġtrust Ġthe Ġopinion Ġof Ġa ĠMS Ġminion ? ĠThe Ġpeople Ġthat Ġproduced ĠWindows ĠME , ĠVista Ġand Ġ8 ? ĠThey Ġdon 't Ġeven Ġunderstand Ġpeople , Ġyet Ġthey Ġthink Ġthey Ġcan Ġpredict Ġthe Ġbehavior Ġof Ġnew , Ġself - gu iding ĠAI ? âĢ Ŀ ĠâĢĵ an onymous </s> âĢ ľ I Ġthink Ġan ĠAI Ġwould Ġmake Ġit Ġeasier Ġfor ĠPatients Ġto Ġconf ide Ġtheir Ġinformation Ġbecause Ġby Ġnature , Ġa Ġrobot Ġcannot Ġjudge Ġthem . ĠWin - win ? Ġ: D âĢ Ŀ ' ĠâĢĵ an onymous </s> Dog mat ism Ġdescribes Ġthe Ġtendency Ġto Ġlay Ġdown Ġopinions Ġas Ġinc ont ro vert ibly Ġtrue , Ġwithout Ġrespect Ġfor Ġconflicting Ġevidence Ġor Ġthe Ġopinions Ġof Ġothers ĠB IB REF 0 Ġ. ĠWhich Ġuser Ġis Ġmore Ġdog matic Ġin Ġthe Ġexamples Ġabove ? ĠThis Ġquestion Ġis Ġsimple Ġfor Ġhumans . ĠPh r ases Ġlike ĠâĢ ľ they Ġthink âĢ Ŀ Ġand ĠâĢ ľ they Ġdon 't Ġeven Ġunderstand , âĢ Ŀ Ġsuggest Ġan Ġint ract ability Ġof Ġopinion , Ġwhile ĠâĢ ľ I Ġthink âĢ Ŀ Ġand ĠâĢ ľ win - win ? âĢ Ŀ Ġsuggest Ġthe Ġopposite . ĠCan Ġwe Ġtrain Ġcomputers Ġto Ġdraw Ġsimilar Ġdistinctions ? ĠWork Ġin Ġpsychology Ġhas Ġcalled Ġout Ġmany Ġaspects Ġof Ġdog mat ism Ġthat Ġcan Ġbe Ġmodeled Ġcomput ationally Ġvia Ġnatural Ġlanguage , Ġsuch Ġas Ġover - confidence Ġand Ġstrong Ġemotions ĠB IB REF 1 Ġ. </s> We Ġpresent Ġa Ġstatistical Ġmodel Ġof Ġdog mat ism Ġthat Ġaddresses Ġtwo Ġcomplementary Ġgoals . ĠFirst , Ġwe Ġvalidate Ġpsychological Ġtheories Ġby Ġexamining Ġthe Ġpredictive Ġpower Ġof Ġfeature Ġsets Ġthat Ġguide Ġthe Ġmodel 's Ġpredictions . ĠFor Ġexample , Ġdo Ġlinguistic Ġsignals Ġof Ġcertainty Ġhelp Ġto Ġpredict Ġa Ġpost Ġis Ġdog matic , Ġas Ġtheory Ġwould Ġsuggest ? ĠSecond , Ġwe Ġapply Ġour Ġmodel Ġto Ġanswer Ġfour Ġquestions : </s> R 1 : ĠWhat Ġkinds Ġof Ġtopics Ġ( e . g ., Ġguns , ĠLGBT ) Ġattract Ġthe Ġhighest Ġlevels Ġof Ġdog mat ism ? </s> R 2 : ĠHow Ġdo Ġdog matic Ġbeliefs Ġcluster ? </s> R 3 : ĠHow Ġdoes Ġdog mat ism Ġinfluence Ġa Ġconversation Ġon Ġsocial Ġmedia ? ĠR 4 : ĠHow Ġdo Ġother Ġuser Ġbehaviors Ġ( e . g ., Ġfrequency Ġand Ġbreadth Ġof Ġposts ) Ġrelate Ġto Ġdog mat ism ? </s> We Ġtrain Ġa Ġpredictive Ġmodel Ġto Ġclassify Ġdog matic Ġposts Ġfrom ĠReddit , Ġone Ġof Ġthe Ġmost Ġpopular Ġdiscussion Ġcommunities Ġon Ġthe Ġweb . ĠPosts Ġon ĠReddit Ġcapture Ġdiscussion Ġand Ġdebate Ġacross Ġa Ġdiverse Ġset Ġof Ġdomains Ġand Ġtopics ĠâĢĵ Ġusers Ġtalk Ġabout Ġeverything Ġfrom Ġclimate Ġchange Ġand Ġabortion , Ġto Ġworld Ġnews Ġand Ġrelationship Ġadvice , Ġto Ġthe Ġfuture Ġof Ġartificial Ġintelligence . ĠAs Ġa Ġprerequisite Ġto Ġtraining Ġour Ġmodel , Ġwe Ġhave Ġcreated Ġa Ġcorpus Ġof Ġ5 , 000 ĠReddit Ġposts Ġannot ated Ġwith Ġlevels Ġof Ġdog mat ism , Ġwhich Ġwe Ġare Ġreleasing Ġto Ġshare Ġwith Ġother Ġresearchers . </s> Using Ġthe Ġmodel , Ġwe Ġoperational ize Ġkey Ġdomain - independent Ġaspects Ġof Ġpsychological Ġtheories Ġof Ġdog mat ism Ġdrawn Ġfrom Ġthe Ġliterature . ĠWe Ġfind Ġthese Ġfeatures Ġhave Ġpredictive Ġpower Ġthat Ġlargely Ġsupports Ġthe Ġunderlying Ġtheory . ĠFor Ġexample , Ġposts Ġthat Ġuse Ġless Ġconfident Ġlanguage Ġtend Ġto Ġbe Ġless Ġdog matic . ĠWe Ġalso Ġdiscover Ġevidence Ġfor Ġnew Ġattributes Ġof Ġdog mat ism . ĠFor Ġexample , Ġdog matic Ġposts Ġtend Ġnot Ġto Ġverbal ize Ġcognition , Ġthrough Ġterms Ġsuch Ġas ĠâĢ ľ I Ġthink , âĢ Ŀ ĠâĢ ľ possibly , âĢ Ŀ Ġor ĠâĢ ľ might Ġbe . âĢ Ŀ </s> Our Ġmodel Ġis Ġtrained Ġon Ġonly Ġ5 , 000 Ġannot ated Ġposts , Ġbut Ġonce Ġtrained , Ġwe Ġuse Ġit Ġto Ġanalyze Ġmillions Ġof Ġother ĠReddit Ġposts Ġto Ġanswer Ġour Ġresearch Ġquestions . ĠWe Ġfind Ġa Ġdiverse Ġset Ġof Ġtopics Ġare Ġcolored Ġby Ġdog matic Ġlanguage Ġ( e . g ., Ġpeople Ġare Ġdog matic Ġabout Ġreligion , Ġbut Ġalso Ġabout ĠLGBT Ġissues ). ĠFurther , Ġwe Ġfind Ġsome Ġevidence Ġfor Ġdog mat ism Ġas Ġa Ġdeeper Ġpersonality Ġtrait ĠâĢĵ Ġpeople Ġwho Ġare Ġstrongly Ġdog matic Ġabout Ġone Ġtopic Ġare Ġmore Ġlikely Ġto Ġexpress Ġdog matic Ġviews Ġabout Ġothers Ġas Ġwell . ĠFinally , Ġin Ġconversation , Ġwe Ġdiscover Ġthat Ġone Ġuser 's Ġdog mat ism Ġtends Ġto Ġbring Ġout Ġdog mat ism Ġin Ġtheir Ġconvers ational Ġpartner , Ġforming Ġa Ġvicious Ġcycle . </s> Dog mat ism Ġdata </s> Posts Ġon ĠReddit Ġcapture Ġdebate Ġand Ġdiscussion Ġacross Ġa Ġdiverse Ġset Ġof Ġtopics , Ġmaking Ġthem Ġa Ġnatural Ġstarting Ġpoint Ġfor Ġunt angling Ġdomain - independent Ġlinguistic Ġfeatures Ġof Ġdog mat ism . </s> Data Ġcollection . ĠSub reddits Ġare Ġsub - commun ities Ġon ĠReddit Ġoriented Ġaround Ġspecific Ġinterests Ġor Ġtopics , Ġsuch Ġas Ġtechnology Ġor Ġpolitics . ĠSam pling Ġfrom ĠReddit Ġas Ġa Ġwhole Ġwould Ġbias Ġthe Ġmodel Ġtowards Ġthe Ġmost Ġcommonly Ġdiscussed Ġcontent . ĠBut Ġby Ġsampling Ġposts Ġfrom Ġindividual Ġsubreddits , Ġwe Ġcan Ġcontrol Ġthe Ġkinds Ġof Ġposts Ġwe Ġuse Ġto Ġtrain Ġour Ġmodel . ĠTo Ġcollect Ġa Ġdiverse Ġtraining Ġdataset , Ġwe Ġhave Ġrandomly Ġsampled Ġ1000 Ġposts Ġeach Ġfrom Ġthe Ġsubreddits Ġpolitics , Ġbusiness , Ġscience , Ġand ĠAsk Reddit , Ġand Ġ1000 Ġadditional Ġposts Ġfrom Ġthe ĠReddit Ġfront page . ĠAll Ġposts Ġin Ġour Ġsample Ġappeared Ġbetween ĠJanuary Ġ2007 Ġand ĠMarch Ġ2015 , Ġand Ġto Ġcontrol Ġfor Ġlength Ġeffects , Ġcontain Ġbetween Ġ300 Ġand Ġ400 Ġcharacters . ĠThis Ġresults Ġin Ġa Ġtotal Ġtraining Ġdataset Ġof Ġ5000 Ġposts . </s> Dog mat ism Ġannotations . ĠBuilding Ġa Ġuseful Ġcomputational Ġmodel Ġrequires Ġlabeled Ġtraining Ġdata . ĠWe Ġlabeled Ġthe ĠReddit Ġdataset Ġusing Ġcrowd workers Ġon ĠAmazon ĠMechanical ĠTurk Ġ( AM T ), Ġcreating Ġthe Ġfirst Ġpublic Ġcorpus Ġannot ated Ġwith Ġlevels Ġof Ġdog mat ism . ĠWe Ġasked Ġcrowd workers Ġto Ġrate Ġlevels Ġof Ġdog mat ism Ġon Ġa Ġ5 - point ĠL ik ert Ġscale , Ġas Ġsupported Ġby Ġsimilar Ġannotation Ġtasks Ġin Ġprior Ġwork ĠB IB REF 2 Ġ. ĠCon crete ly , Ġwe Ġgave Ġcrowd workers Ġthe Ġfollowing Ġtask : </s> Given Ġa Ġcomment , Ġimagine Ġyou Ġhold Ġa Ġwell - informed , Ġdifferent Ġopinion Ġfrom Ġthe Ġcommenter Ġin Ġquestion . ĠWe 'd Ġlike Ġyou Ġto Ġtell Ġus Ġhow Ġlikely Ġthat Ġcommenter Ġwould Ġbe Ġto Ġengage Ġyou Ġin Ġa Ġconstructive Ġconversation Ġabout Ġyour Ġdisagreement , Ġwhere Ġyou Ġeach Ġare Ġable Ġto Ġexplore Ġthe Ġother 's Ġbeliefs . ĠThe Ġoptions Ġare : </s> ( 5 ): ĠIt 's Ġunlikely Ġyou 'll Ġbe Ġable Ġto Ġengage Ġin Ġany Ġsubstantive Ġconversation . ĠWhen Ġyou Ġrespectfully Ġexpress Ġyour Ġdisagreement , Ġthey Ġare Ġlikely Ġto Ġignore Ġyou Ġor Ġinsult Ġyou Ġor Ġotherwise Ġlower Ġthe Ġlevel Ġof Ġdiscourse . </s> ( 4 ): ĠThey Ġare Ġdeeply Ġrooted Ġin Ġtheir Ġopinion , Ġbut Ġyou Ġare Ġable Ġto Ġexchange Ġyour Ġviews Ġwithout Ġthe Ġconversation Ġdegener ating Ġtoo Ġmuch . </s> ( 3 ): ĠIt 's Ġnot Ġlikely Ġyou 'll Ġbe Ġable Ġto Ġchange Ġtheir Ġmind , Ġbut Ġyou 're Ġeasily Ġable Ġto Ġtalk Ġand Ġunderstand Ġeach Ġother 's Ġpoint Ġof Ġview . </s> ( 2 ): ĠThey Ġmay Ġhave Ġa Ġclear Ġopinion Ġabout Ġthe Ġsubject , Ġbut Ġwould Ġlikely Ġbe Ġopen Ġto Ġdiscussing Ġalternative Ġviewpoints . </s> ( 1 ): ĠThey Ġare Ġnot Ġset Ġin Ġtheir Ġopinion , Ġand Ġit 's Ġpossible Ġyou Ġmight Ġchange Ġtheir Ġmind . ĠIf Ġthe Ġcomment Ġdoes Ġnot Ġconvey Ġan Ġopinion Ġof Ġany Ġkind , Ġyou Ġmay Ġalso Ġselect Ġthis Ġoption . </s> To Ġensure Ġquality Ġwork , Ġwe Ġrestricted Ġthe Ġtask Ġto ĠMasters Ġworkers Ġand Ġprovided Ġexamples Ġcorresponding Ġto Ġeach Ġpoint Ġon Ġthe Ġscale . ĠIncluding Ġexamples Ġin Ġa Ġtask Ġhas Ġbeen Ġshown Ġto Ġsignificantly Ġincrease Ġthe Ġagreement Ġand Ġquality Ġof Ġcrowd work ĠB IB REF 3 Ġ. ĠFor Ġinstance , Ġhere Ġis Ġan Ġexample Ġof Ġa Ġhighly Ġdog matic Ġ( 5 ) Ġcomment : </s> I Ġwon 't Ġbe Ġhappy Ġuntil ĠI Ġsee Ġthe Ġexecutive Ġsuite Ġof ĠB of A , ĠWells , Ġand Ġall Ġthe Ġothers , Ġfrog - mar ched Ġinto Ġwaiting Ġsquad Ġcars . ĠIt 's ĠAL READ Y ĠBE EN ĠEST AB L ISH ED Ġthat ... </s> And Ġa Ġminim ally Ġdog matic Ġ( 1 ) Ġcomment : </s> I Ġagree . ĠI Ġwould Ġlike Ġto Ġcompile Ġa Ġplaylist Ġfor Ġus Ġtrance Ġyog i 's , Ġeven Ġif Ġyou Ġjust Ġwould Ġlike Ġto Ġexperiment Ġwith Ġit . ĠIs Ġthere Ġany Ġpreference Ġon Ġwhich Ġplatform Ġto Ġuse ? </s> Each Ġcomment Ġhas Ġbeen Ġannot ated Ġby Ġthree Ġindependent Ġworkers Ġon ĠAM T , Ġwhich Ġis Ġenough Ġto Ġproduce Ġreliable Ġresults Ġin Ġmost Ġlabeling Ġtasks ĠB IB REF 4 Ġ. ĠTo Ġcompute Ġan Ġaggregate Ġmeasure Ġof Ġdog mat ism Ġfor Ġeach Ġcomment , Ġwe Ġsummed Ġthe Ġscores Ġof Ġall Ġthree Ġworkers . ĠWe Ġshow Ġthe Ġresulting Ġdistribution Ġof Ġannotations Ġin ĠFigure Ġ1 Ġ. </s> Inter - annot ator Ġagreement . ĠTo Ġevaluate Ġthe Ġreliability Ġof Ġannotations Ġwe Ġcompute ĠK ripp endor ff 's Ġ$\ alpha Ġ$ Ġ, Ġa Ġmeasure Ġof Ġagreement Ġdesigned Ġfor Ġvariable Ġlevels Ġof Ġmeasurement Ġsuch Ġas Ġa ĠL ik ert Ġscale ĠB IB REF 5 Ġ. ĠAn Ġ$\ alpha Ġ$ Ġof Ġ0 Ġindicates Ġagreement Ġindistinguishable Ġfrom Ġchance , Ġwhile Ġan Ġ$\ alpha Ġ$ Ġof Ġ1 Ġindicates Ġperfect Ġagreement . ĠAcross Ġall Ġannotations Ġwe Ġfind Ġ$\ alpha Ġ= 0 . 44 $ Ġ. ĠWhile Ġworkers Ġagree Ġmuch Ġmore Ġthan Ġchance , Ġclearly Ġdog mat ism Ġis Ġalso Ġsubjective . ĠIn Ġfact , Ġwhen Ġwe Ġexamine Ġonly Ġthe Ġmiddle Ġtwo Ġquart iles Ġof Ġthe Ġdog mat ism Ġannotations , Ġwe Ġfind Ġagreement Ġis Ġno Ġbetter Ġthan Ġchance . ĠAlternatively , Ġwhen Ġwe Ġmeasure Ġagreement Ġonly Ġamong Ġthe Ġtop Ġand Ġbottom Ġquart iles Ġof Ġannotations , Ġwe Ġfind Ġagreement Ġof Ġ$\ alpha Ġ= 0 . 69 $ Ġ. ĠThis Ġsuggests Ġcomments Ġwith Ġscores Ġthat Ġare Ġonly Ġslightly Ġdog matic Ġare Ġunreliable Ġand Ġoften Ġsubject Ġto Ġhuman Ġdisagreement . ĠFor Ġthis Ġreason , Ġwe Ġuse Ġonly Ġthe Ġtop Ġand Ġbottom Ġquart iles Ġof Ġcomments Ġwhen Ġtraining Ġour Ġmodel . </s> App ro aches Ġto ĠIdent ifying ĠDog mat ism </s> We Ġnow Ġconsider Ġstrategies Ġfor Ġidentifying Ġdog mat ism Ġbased Ġon Ġprior Ġwork Ġin Ġpsychology . ĠWe Ġstart Ġwith Ġthe ĠL ingu istic ĠInquiry Ġand ĠWord ĠCount Ġ( LI WC ), Ġa Ġlex icon Ġpopular Ġin Ġthe Ġsocial Ġsciences ĠB IB REF 6 Ġ. ĠLI WC Ġprovides Ġhuman Ġvalidated Ġlists Ġof Ġwords Ġthat Ġcorrespond Ġto Ġhigh - level Ġpsychological Ġcategories Ġsuch Ġas Ġcertainty Ġor Ġperception . ĠIn Ġother Ġstudies , ĠLI WC Ġhas Ġuncovered Ġlinguistic Ġsignals Ġrelating Ġto Ġpolit eness ĠB IB REF 2 Ġ, Ġdeception ĠB IB REF 7 Ġ, Ġor Ġauthority Ġin Ġtexts ĠB IB REF 8 Ġ. ĠHere , Ġwe Ġexamine Ġhow Ġdog mat ism Ġrelates Ġto Ġ17 Ġof ĠLI WC 's Ġcategories Ġ( Table Ġ1 Ġ). </s> To Ġcompute Ġthe Ġrelationships Ġbetween ĠLI WC Ġcategories Ġand Ġdog mat ism , Ġwe Ġfirst Ġcount Ġthe Ġrelevant Ġcategory Ġterms Ġthat Ġappear Ġin Ġeach Ġannot ated ĠReddit Ġcomment , Ġnormalized Ġby Ġits Ġword Ġcount . ĠWe Ġthen Ġcalculate Ġodds Ġratios Ġon Ġthe Ġaggregate Ġcounts Ġof Ġeach ĠLI WC Ġcategory Ġover Ġthe Ġtop Ġand Ġbottom Ġquart iles Ġof Ġdog matic Ġcomments . ĠAs Ġwe Ġhave Ġdiscussed , Ġusing Ġthe Ġtop Ġand Ġbottom Ġquart iles Ġof Ġcomments Ġprovides Ġa Ġmore Ġreliable Ġsignal Ġof Ġdog mat ism . ĠWe Ġcheck Ġfor Ġsignificant Ġdifferences Ġin Ġcategories Ġbetween Ġdog matic Ġand Ġnon - dog matic Ġcomments Ġusing Ġthe ĠMann - Whit ney ĠU Ġtest Ġand Ġapply ĠHolmes Ġmethod Ġfor Ġcorrection . ĠAll Ġodds Ġwe Ġreport Ġin Ġthis Ġsection Ġare Ġsignificant Ġafter Ġcorrection . </s> Dog matic Ġstatements Ġtend Ġto Ġexpress Ġa Ġhigh Ġdegree Ġof Ġcertainty ĠB IB REF 1 Ġ. ĠHere Ġwe Ġconsider ĠLI WC Ġcategories Ġthat Ġexpress Ġcertainty Ġboth Ġpositively Ġ( certain ty ) Ġand Ġnegatively Ġ( t ent at iveness ). ĠFor Ġexample , Ġthe Ġword ĠâĢ ľ always âĢ Ŀ Ġis Ġcertain , Ġwhile ĠâĢ ľ possibly âĢ Ŀ Ġis Ġtentative . ĠCon forming Ġto Ġexisting Ġtheory , Ġcertainty Ġis Ġmore Ġassociated Ġwith Ġdog matic Ġcomments Ġ( 1 . 52 Ġodds ), Ġwhile Ġtent at iveness Ġis Ġmore Ġassociated Ġwith Ġthe Ġabsence Ġof Ġdog mat ism Ġ( 0 . 88 Ġodds ). </s> Ter ms Ġused Ġto Ġverbal ize Ġcognition Ġcan Ġact Ġas Ġa Ġhedge Ġthat Ġoften Ġcharacter izes Ġnon - dog matic Ġlanguage . ĠLI WC 's Ġinsight Ġcategory Ġcaptures Ġthis Ġeffect Ġthrough Ġwords Ġsuch Ġas ĠâĢ ľ think , âĢ Ŀ ĠâĢ ľ know , âĢ Ŀ Ġor ĠâĢ ľ bel ieve . âĢ Ŀ ĠThese Ġwords Ġadd Ġnuance Ġto Ġa Ġstatement ĠB IB REF 9 Ġ, Ġsignaling Ġit Ġis Ġthe Ġproduct Ġof Ġsomeone 's Ġmind Ġ( âĢ ľ I Ġthink Ġyou Ġshould Ġgive Ġthis Ġpaper Ġa Ġgood Ġreview âĢ Ŀ ) Ġand Ġnot Ġmeant Ġto Ġbe Ġinterpreted Ġas Ġan Ġobjective Ġtruth . ĠAlong Ġthese Ġlines , Ġwe Ġfind Ġthe Ġuse Ġof Ġterms Ġin Ġthe Ġinsight Ġcategory Ġis Ġassociated Ġwith Ġnon - dog matic Ġcomments Ġ( 0 . 83 Ġodds ). </s> S ens ory Ġlanguage , Ġwith Ġits Ġfocus Ġon Ġdescription Ġand Ġdetail , Ġoften Ġsignals Ġa Ġlack Ġof Ġany Ġkind Ġof Ġopinion , Ġdog matic Ġor Ġotherwise . ĠLI WC 's Ġperception Ġcategory Ġcaptures Ġthis Ġidea Ġthrough Ġwords Ġassociated Ġwith Ġhearing , Ġfeeling , Ġor Ġseeing . ĠFor Ġexample , Ġthese Ġwords Ġmight Ġoccur Ġwhen Ġrecount ing Ġa Ġpersonal Ġexperience Ġ( âĢ ľ I Ġsaw Ġhis Ġincoming Ġfist âĢ Ŀ ), Ġwhich Ġeven Ġif Ġemotionally Ġcharged Ġor Ġnegative , Ġis Ġless Ġlikely Ġto Ġbe Ġdog matic . ĠWe Ġfind Ġperception Ġis Ġassociated Ġwith Ġnon - dog matic Ġcomments Ġat Ġ0 . 77 Ġodds . </s> Draw ing Ġcomparisons Ġor Ġqualifying Ġsomething Ġas Ġrelative Ġto Ġsomething Ġelse Ġconve ys Ġa Ġnuance Ġthat Ġis Ġabsent Ġfrom Ġtraditionally Ġdog matic Ġlanguage . ĠThe ĠLI WC Ġcategories Ġcomparison Ġand Ġrelativity Ġcapture Ġthese Ġeffects Ġthrough Ġcomparison Ġwords Ġsuch Ġas ĠâĢ ľ than âĢ Ŀ Ġor ĠâĢ ľ as âĢ Ŀ Ġand Ġqualifying Ġwords Ġsuch Ġas ĠâĢ ľ during âĢ Ŀ Ġor ĠâĢ ľ when . âĢ Ŀ ĠFor Ġexample , Ġthe Ġstatement ĠâĢ ľ I Ġhate Ġpoliticians âĢ Ŀ Ġis Ġmore Ġdog matic Ġthan ĠâĢ ľ I Ġhate Ġpoliticians Ġwhen Ġthey Ġcan 't Ġget Ġanything Ġdone .' ĠRel ativity Ġis Ġassociated Ġwith Ġnon - dog matic Ġcomments Ġat Ġ0 . 80 Ġodds , Ġbut Ġcomparison Ġdoes Ġnot Ġreach Ġsignificance . </s> P ron oun s Ġcan Ġbe Ġsurprisingly Ġrevealing Ġindicators Ġof Ġlanguage : Ġfor Ġexample , Ġsignaling Ġone 's Ġgender Ġor Ġhierarchical Ġstatus Ġin Ġa Ġconversation ĠB IB REF 10 Ġ. ĠWe Ġfind Ġfirst Ġperson Ġsingular Ġpronouns Ġare Ġa Ġuseful Ġnegative Ġsignal Ġfor Ġdog mat ism Ġ( 0 . 46 Ġodds ), Ġwhile Ġsecond Ġperson Ġsingular Ġpronouns Ġ( 2 . 18 Ġodds ) Ġand Ġthird Ġperson Ġplural Ġ( 1 . 63 Ġodds ) Ġare Ġa Ġuseful Ġpositive Ġsignal . ĠLooking Ġacross Ġthe Ġcorpus , Ġwe Ġsee ĠI Ġoften Ġused Ġwith Ġa Ġhedge Ġ( âĢ ľ I Ġthink âĢ Ŀ Ġor ĠâĢ ľ I Ġknow âĢ Ŀ ), Ġwhile Ġyou Ġand Ġthey Ġtend Ġto Ġcharacterize Ġthe Ġbeliefs Ġof Ġothers , Ġoften Ġin Ġa Ġstrongly Ġopinion ated Ġway Ġ( âĢ ľ you Ġare Ġa Ġmor on âĢ Ŀ Ġor ĠâĢ ľ they Ġare Ġkeeping Ġus Ġdown âĢ Ŀ ). ĠOther Ġpronoun Ġtypes Ġdo Ġnot Ġshow Ġsignificant Ġrelationships . </s> Like Ġpronouns , Ġverb Ġtense Ġcan Ġreveal Ġsubtle Ġsignals Ġin Ġlanguage Ġuse , Ġsuch Ġas Ġthe Ġtendency Ġof Ġmedical Ġin pat ients Ġto Ġfocus Ġon Ġthe Ġpast ĠB IB REF 11 Ġ. ĠOn Ġsocial Ġmedia , Ġcomments Ġwritten Ġin Ġthe Ġpresent Ġtense Ġare Ġmore Ġlikely Ġto Ġbe Ġoriented Ġtowards Ġa Ġuser 's Ġcurrent Ġinteraction Ġ( âĢ ľ this Ġis Ġall Ġso Ġstupid âĢ Ŀ ), Ġcreating Ġopportunities Ġto Ġsignal Ġdog mat ism . ĠAlternatively , Ġcomments Ġin Ġthe Ġpast Ġtense Ġare Ġmore Ġlikely Ġto Ġrefer Ġto Ġoutside Ġexperiences Ġ( âĢ ľ it Ġwas Ġan Ġawful Ġparty âĢ Ŀ ), Ġspeaking Ġless Ġto Ġa Ġuser 's Ġstance Ġtowards Ġan Ġongoing Ġdiscussion . ĠWe Ġfind Ġpresent Ġtense Ġis Ġa Ġpositive Ġsignal Ġfor Ġdog mat ism Ġ( 1 . 11 Ġodds ) Ġand Ġpast Ġtense Ġis Ġa Ġnegative Ġsignal Ġ( 0 . 69 Ġodds ). </s> Dog matic Ġlanguage Ġcan Ġbe Ġeither Ġpositively Ġor Ġnegatively Ġcharged Ġin Ġsentiment : Ġfor Ġexample , Ġconsider Ġthe Ġpositive Ġstatement ĠâĢ ľ Trump Ġis Ġthe ĠS AV IOR Ġof Ġthis Ġcountry !!! âĢ Ŀ Ġor Ġthe Ġnegative Ġstatement ĠâĢ ľ Are Ġyou ĠREALLY Ġthat Ġstupid ?? ĠEducation Ġis Ġthe Ġonly Ġway Ġout Ġof Ġthis Ġhorrible Ġmess . ĠIt 's Ġhard Ġto Ġimagine Ġhow Ġanyone Ġcould Ġbe Ġso Ġdel uded . âĢ Ŀ ĠIn Ġdiverse Ġcommunities , Ġwhere Ġpeople Ġhold Ġmany Ġdifferent Ġkinds Ġof Ġopinions , Ġdog matic Ġopinions Ġwill Ġoften Ġtend Ġto Ġcome Ġinto Ġconflict Ġwith Ġone Ġanother ĠB IB REF 12 Ġ, Ġproducing Ġa Ġgreater Ġlikelihood Ġof Ġnegative Ġsentiment . ĠPerhaps Ġfor Ġthis Ġreason , Ġnegative Ġemotion Ġ( 2 . 09 Ġodds ) Ġand Ġswearing Ġ( 3 . 80 Ġodds ) Ġare Ġuseful Ġpositive Ġsignals Ġof Ġdog mat ism , Ġwhile Ġpositive Ġemotion Ġshows Ġno Ġsignificant Ġrelationship . </s> Finally , Ġwe Ġfind Ġthat Ġinterrog ative Ġlanguage Ġ( 1 . 12 Ġodds ) Ġand Ġneg ation Ġ( 1 . 35 Ġodds ) Ġare Ġtwo Ġadditional Ġpositive Ġsignals Ġof Ġdog mat ism . ĠWhile Ġinterrog ative Ġwords Ġlike ĠâĢ ľ how âĢ Ŀ Ġor ĠâĢ ľ what âĢ Ŀ Ġhave Ġmany Ġbenign Ġuses , Ġthey Ġdisproportionately Ġappear Ġin Ġour Ġdata Ġin Ġthe Ġform Ġof Ġrhetorical Ġor Ġemotionally Ġcharged Ġquestions , Ġsuch Ġas ĠâĢ ľ how Ġcan Ġanyone Ġbe Ġthat Ġdumb ? âĢ Ŀ </s> Many Ġof Ġthese Ġlinguistic Ġsignals Ġare Ġcorrelated Ġwith Ġeach Ġother , Ġsuggesting Ġthat Ġdog mat ism Ġis Ġthe Ġcumulative Ġeffect Ġof Ġmany Ġcomponent Ġrelationships . ĠFor Ġexample , Ġconsider Ġthe Ġrelatively Ġnon - dog matic Ġstatement : ĠâĢ ľ I Ġthink Ġthe Ġreviewers Ġare Ġwrong Ġin Ġthis Ġinstance . âĢ Ŀ ĠRem oving Ġsignals Ġof Ġinsight , Ġwe Ġhave : ĠâĢ ľ the Ġreviewers Ġare Ġwrong Ġin Ġthis Ġinstance , âĢ Ŀ Ġwhich Ġis Ġslightly Ġmore Ġdog matic . ĠThen Ġremoving Ġrelativity , Ġwe Ġhave : ĠâĢ ľ the Ġreviewers Ġare Ġwrong . âĢ Ŀ ĠAnd Ġfinally , Ġadding Ġcertainty , Ġwe Ġhave Ġa Ġdog matic Ġstatement : ĠâĢ ľ the Ġreviewers Ġare Ġalways Ġwrong . âĢ Ŀ </s> P redict ing Ġdog mat ism </s> We Ġnow Ġshow Ġhow Ġwe Ġcan Ġuse Ġthe Ġlinguistic Ġfeature Ġsets Ġwe Ġhave Ġdescribed Ġto Ġbuild Ġa Ġclass ifier Ġthat Ġpredicts Ġdog mat ism Ġin Ġcomments . ĠA Ġpredictive Ġmodel Ġfurther Ġvalid ates Ġour Ġfeature Ġsets , Ġand Ġalso Ġallows Ġus Ġto Ġanalyze Ġdog mat ism Ġin Ġmillions Ġof Ġother ĠReddit Ġcomments Ġin Ġa Ġscalable Ġway , Ġwith Ġmultiple Ġuses Ġin Ġongoing , Ġdownstream Ġanalyses . </s> Pred iction Ġtask . ĠOur Ġgoal Ġis Ġ( 1 ) Ġto Ġunderstand Ġhow Ġwell Ġwe Ġcan Ġuse Ġthe Ġstrategies Ġin ĠSection Ġ3 Ġto Ġpredict Ġdog mat ism , Ġand Ġ( 2 ) Ġto Ġtest Ġthe Ġdomain - independence Ġof Ġthese Ġstrategies . ĠFirst , Ġwe Ġtest Ġthe Ġperformance Ġof Ġour Ġmodel Ġunder Ġcross - valid ation Ġwithin Ġthe ĠReddit Ġcomment Ġdataset . ĠWe Ġthen Ġevaluate Ġthe ĠReddit - based Ġmodel Ġon Ġa Ġheld Ġout Ġcorpus Ġof ĠNew ĠYork ĠTimes Ġcomments Ġannot ated Ġusing Ġthe Ġtechnique Ġin ĠSection Ġ2 . ĠWe Ġdid Ġnot Ġrefer Ġto Ġthis Ġsecond Ġdataset Ġduring Ġfeature Ġconstruction . </s> For Ġclassification , Ġwe Ġconsider Ġtwo Ġclasses Ġof Ġcomments : Ġdog matic Ġand Ġnon - dog matic . ĠAs Ġin Ġthe Ġprior Ġanalysis , Ġwe Ġdraw Ġthese Ġcomments Ġfrom Ġthe Ġtop Ġand Ġbottom Ġquart iles Ġof Ġthe Ġdog mat ism Ġdistribution . ĠThis Ġmeans Ġthe Ġclasses Ġare Ġbalanced , Ġwith Ġ2 , 500 Ġtotal Ġcomments Ġin Ġthe ĠReddit Ġtraining Ġdata Ġand Ġ500 Ġtotal Ġcomments Ġin Ġthe ĠNew ĠYork ĠTimes Ġtesting Ġdata . </s> We Ġcompare Ġthe Ġpredictions Ġof Ġlog istic Ġregression Ġmodels Ġbased Ġon Ġun ig ram Ġbag - of - words Ġfeatures Ġ( B OW ), Ġsentiment Ġsignals Ġ( S ENT ), Ġthe Ġlinguistic Ġfeatures Ġfrom Ġour Ġearlier Ġanalyses Ġ( L ING ), Ġand Ġcombinations Ġof Ġthese Ġfeatures . ĠB OW Ġand ĠS ENT Ġprovide Ġbas elines Ġfor Ġthe Ġtask . ĠWe Ġcompute ĠB OW Ġfeatures Ġusing Ġterm Ġfrequency - in verse Ġdocument Ġfrequency Ġ( TF - ID F ) Ġand Ġcategory - based Ġfeatures Ġby Ġnormal izing Ġcounts Ġfor Ġeach Ġcategory Ġby Ġthe Ġnumber Ġof Ġwords Ġin Ġeach Ġdocument . ĠThe ĠB OW Ġclass ifiers Ġare Ġtrained Ġwith Ġregular ization Ġ( L 2 Ġpenalties Ġof Ġ1 . 5 ). </s> Class ification Ġresults . ĠWe Ġpresent Ġclassification Ġaccuracy Ġin ĠTable Ġ2 Ġ. ĠB OW Ġshows Ġan ĠA UC Ġof Ġ0 . 8 53 Ġwithin ĠReddit Ġand Ġ0 . 776 Ġon Ġthe Ġheld Ġout ĠNew ĠYork ĠTimes Ġcomments . ĠThe Ġlinguistic Ġfeatures Ġboost Ġclassification Ġresults Ġwithin ĠReddit Ġ( 0 . 88 1 ) Ġand Ġon Ġthe Ġheld Ġout ĠNew ĠYork ĠTimes Ġcomments Ġ( 0 . 79 1 ). ĠWhile Ġlinguistic Ġsignals Ġby Ġthemselves Ġprovide Ġstrong Ġpredictive Ġpower Ġ( 0 . 801 ĠA UC Ġwithin Ġdomain ), Ġsentiment Ġsignals Ġare Ġmuch Ġless Ġpredictive . </s> These Ġresults Ġsuggest Ġthat Ġlinguistic Ġfeatures Ġinspired Ġby Ġprior Ġefforts Ġin Ġpsychology Ġare Ġuseful Ġfor Ġpredicting Ġdog mat ism Ġin Ġpractice Ġand Ġgeneral ize Ġacross Ġnew Ġdomains . </s> Dog mat ism Ġin Ġthe ĠReddit ĠCommunity Ġ </s> We Ġnow Ġapply Ġour Ġdog mat ism Ġclass ifier Ġto Ġa Ġlarger Ġdataset Ġof Ġposts , Ġexamining Ġhow Ġdog matic Ġlanguage Ġshapes Ġthe ĠReddit Ġcommunity . ĠCon crete ly , Ġwe Ġapply Ġthe ĠB OW + L ING Ġmodel Ġtrained Ġon Ġthe Ġfull ĠReddit Ġdataset Ġto Ġmillions Ġof Ġnew Ġun annot ated Ġposts , Ġlabeling Ġthese Ġposts Ġwith Ġa Ġprobability Ġof Ġdog mat ism Ġaccording Ġto Ġthe Ġclass ifier Ġ( 0 = non - dog matic , Ġ1 = dog matic ). ĠWe Ġthen Ġuse Ġthese Ġdog mat ism Ġannotations Ġto Ġaddress Ġfour Ġresearch Ġquestions . </s> What Ġsubreddits Ġhave Ġthe Ġhighest Ġand Ġlowest Ġlevels Ġof Ġdog mat ism ? Ġ( R 1 ) </s> A Ġnatural Ġstarting Ġpoint Ġfor Ġanalyzing Ġdog mat ism Ġon ĠReddit Ġis Ġto Ġexamine Ġhow Ġit Ġcharacter izes Ġthe Ġsite 's Ġsub - commun ities . ĠFor Ġexample , Ġwe Ġmight Ġexpect Ġto Ġsee Ġthat Ġsubreddits Ġoriented Ġaround Ġtopics Ġsuch Ġas Ġabortion Ġor Ġclimate Ġchange Ġare Ġmore Ġdog matic , Ġand Ġsubreddits Ġabout Ġcooking Ġare Ġless Ġso . </s> To Ġanswer Ġthis Ġquestion , Ġwe Ġrandomly Ġsample Ġ1 . 6 Ġmillion Ġposts Ġfrom Ġthe Ġentire ĠReddit Ġcommunity Ġbetween Ġ2007 Ġand Ġ2015 . ĠWe Ġthen Ġannot ate Ġeach Ġof Ġthese Ġposts Ġwith Ġdog mat ism Ġusing Ġour Ġclass ifier , Ġand Ġcompute Ġthe Ġaverage Ġdog mat ism Ġlevel Ġfor Ġeach Ġsubreddit Ġin Ġthe Ġsample Ġwith Ġat Ġleast Ġ100 Ġposts . </s> We Ġpresent Ġthe Ġresults Ġof Ġthis Ġanalysis Ġin ĠTable Ġ3 Ġ. ĠThe Ġsubreddits Ġwith Ġthe Ġhighest Ġlevels Ġof Ġdog mat ism Ġtend Ġto Ġbe Ġoriented Ġaround Ġpolitics Ġand Ġreligion Ġ( Deb ate A Christian Ġor Ġu k politics ), Ġwhile Ġthose Ġwith Ġthe Ġlowest Ġlevels Ġtend Ġto Ġfocus Ġon Ġhobbies Ġ( phot ography Ġor Ġhomebrew ing ). ĠThe Ġsubreddit Ġwith Ġthe Ġhighest Ġaverage Ġdog mat ism Ġlevel , Ġcr ing ep ics , Ġis Ġa Ġplace Ġto Ġmake Ġfun Ġof Ġsocially Ġawkward Ġmessages , Ġoften Ġfrom Ġwould - be Ġromantic Ġpartners . ĠDog mat ism Ġhere Ġtends Ġto Ġtake Ġthe Ġform Ġof ĠâĢ ľ how Ġcould Ġsomeone Ġbe Ġthat Ġstupid âĢ Ŀ Ġand Ġis Ġdirected Ġat Ġthe Ġsubject Ġof Ġthe Ġpost , Ġas Ġopposed Ġto Ġother Ġmembers Ġof Ġthe Ġcommunity . </s> Similarly , ĠSub reddit D rama Ġis Ġa Ġcommunity Ġwhere Ġpeople Ġcome Ġto Ġtalk Ġabout Ġfights Ġon Ġthe Ġinternet Ġor Ġsocial Ġmedia . ĠThese Ġfights Ġare Ġoften Ġthen Ġextended Ġin Ġdiscussion , Ġfor Ġexample : ĠâĢ ľ If Ġthe Ġbest Ġyou Ġcan Ġcome Ġup Ġwith Ġis Ġthat Ġsomething Ġyou Ġdid Ġwas Ġlegal , Ġit 's Ġprobably Ġtime Ġto Ġown Ġup Ġto Ġbeing Ġan Ġass . âĢ Ŀ ĠThe Ġpresence Ġof Ġthis Ġsubreddit Ġin Ġour Ġanalysis Ġprovides Ġa Ġfurther Ġsanity Ġcheck Ġthat Ġour Ġmodel Ġis Ġcapturing Ġa Ġrobust Ġsignal Ġof Ġdog mat ism . </s> How Ġdo Ġdog matic Ġbeliefs Ġcluster ? Ġ( R 2 ) </s> Dog mat ism Ġis Ġwidely Ġconsidered Ġto Ġbe Ġa Ġdomain - specific Ġattitude Ġ( for Ġexample , Ġoriented Ġtowards Ġreligion Ġor Ġpolitics ) Ġas Ġopposed Ġto Ġa Ġdeeper Ġpersonality Ġtrait ĠB IB REF 1 Ġ. ĠHere Ġwe Ġuse ĠReddit Ġas Ġa Ġlens Ġto Ġexamine Ġthis Ġidea Ġmore Ġclosely . ĠAre Ġusers Ġwho Ġare Ġdog matic Ġabout Ġone Ġtopic Ġlikely Ġto Ġbe Ġdog matic Ġabout Ġothers ? ĠDo Ġclusters Ġof Ġdog mat ism Ġexist Ġaround Ġparticular Ġtopics ? ĠTo Ġfind Ġout , Ġwe Ġexamine Ġthe Ġrelationships Ġbetween Ġsubreddits Ġover Ġwhich Ġindividual Ġusers Ġare Ġdog matic . ĠFor Ġexample , Ġif Ġmany Ġusers Ġoften Ġpost Ġdog matic Ġcomments Ġon Ġboth Ġthe Ġpolitics Ġand ĠChristianity Ġsubreddits , Ġbut Ġless Ġoften Ġon Ġworld news , Ġthat Ġwould Ġsuggest Ġpolitics Ġand ĠChristianity Ġare Ġlinked Ġper Ġa Ġboost Ġin Ġlikelihood Ġof Ġindividuals Ġbeing Ġdog matic Ġin Ġboth . </s> We Ġsample Ġ1000 ĠReddit Ġusers Ġwho Ġposted Ġat Ġleast Ġonce Ġa Ġyear Ġbetween Ġ2007 Ġand Ġ2015 Ġto Ġconstruct Ġa Ġcorpus Ġof Ġ10 Ġmillion Ġposts Ġthat Ġconstitute Ġtheir Ġentire Ġpost Ġhistory . ĠWe Ġthen Ġannot ate Ġthese Ġposts Ġusing Ġthe Ġclass ifier Ġand Ġcompute Ġthe Ġaverage Ġdog mat ism Ġscore Ġper Ġsubreddit Ġper Ġuser . ĠFor Ġexample , Ġone Ġuser Ġmight Ġhave Ġan Ġaverage Ġdog mat ism Ġlevel Ġof Ġ0 . 55 Ġfor Ġthe Ġpolitics Ġsubreddit Ġand Ġ0 . 45 Ġfor Ġthe Ġeconomics Ġsubreddit . ĠMost Ġusers Ġdo Ġnot Ġpost Ġin Ġall Ġsubreddits , Ġso Ġwe Ġtrack Ġonly Ġsubreddits Ġfor Ġwhich Ġa Ġuser Ġhad Ġposted Ġat Ġleast Ġ10 Ġtimes . ĠAny Ġsubreddits Ġwith Ġan Ġaverage Ġdog mat ism Ġscore Ġhigher Ġthan Ġ0 . 50 Ġwe Ġconsider Ġto Ġbe Ġa Ġuser 's Ġdog matic Ġsubreddits . ĠWe Ġthen Ġcount Ġall Ġpairs Ġof Ġthese Ġdog matic Ġsubreddits . ĠFor Ġexample , Ġ45 Ġusers Ġhave Ġpolitics Ġand Ġtechnology Ġamong Ġtheir Ġdog matic Ġsubreddits , Ġso Ġwe Ġconsider Ġpolitics Ġand Ġtechnology Ġas Ġlinked Ġ45 Ġtimes . ĠWe Ġcompute Ġthe Ġmutual Ġinformation ĠB IB REF 13 Ġbetween Ġthese Ġlinks , Ġwhich Ġgives Ġus Ġa Ġmeasure Ġof Ġthe Ġsubreddits Ġthat Ġare Ġmost Ġrelated Ġthrough Ġdog mat ism . </s> We Ġpresent Ġthe Ġresults Ġof Ġthis Ġanalysis Ġin ĠTable Ġ4 Ġ, Ġchoosing Ġclusters Ġthat Ġrepresent Ġa Ġdiverse Ġset Ġof Ġtopics . ĠFor Ġexample , ĠLibertarian ism Ġis Ġlinked Ġthrough Ġdog mat ism Ġto Ġother Ġpolitical Ġcommunities Ġlike ĠAnarch o _ Capital ism , Ġr on p aul , Ġor Ġu k politics , Ġas Ġwell Ġas Ġother Ġtopical Ġsubreddits Ġlike Ġguns Ġor Ġeconomy . ĠSimilarly , Ġpeople Ġwho Ġare Ġdog matic Ġin Ġthe Ġbusiness Ġsubreddit Ġalso Ġtend Ġto Ġbe Ġdog matic Ġin Ġsubreddits Ġfor ĠBitcoin , Ġsocialism , Ġand Ġtechnology . ĠNotably , Ġwhen Ġwe Ġapply Ġthe Ġsame Ġmutual Ġinformation Ġanalysis Ġto Ġlinks Ġdefined Ġby Ġsubreddits Ġposted Ġin Ġby Ġthe Ġsame Ġuser , Ġwe Ġsee Ġdramatically Ġdifferent Ġresults . ĠFor Ġexample , Ġthe Ġsubreddits Ġmost Ġlinked Ġto Ġscience Ġthrough Ġuser Ġposts Ġare ĠU pl ifting News , Ġphot oshop b att les , Ġand Ġfirst world an arch ist , Ġand Ġmillionaire makers . </s> Finally , Ġwe Ġsee Ġless Ġobvious Ġconnections Ġbetween Ġsubreddits Ġthat Ġsuggest Ġsome Ġpeople Ġmay Ġbe Ġdog matic Ġby Ġnature . ĠFor Ġexample , Ġamong Ġthe Ġusers Ġwho Ġare Ġdog matic Ġon Ġpolitics , Ġthey Ġare Ġalso Ġdisproportionately Ġdog matic Ġon Ġunrelated Ġsubreddits Ġsuch Ġas Ġscience Ġ( Ġ$ p < 0 . 001 $ Ġ), Ġtechnology Ġ( Ġ$ p < 0 . 001 $ Ġ), ĠI Am A Ġ( Ġ$ p < 0 . 001 $ Ġ), Ġand ĠAsk Reddit Ġ( Ġ$ p < 0 . 05 $ Ġ), Ġwith Ġp - values Ġcomputed Ġunder Ġa Ġbin omial Ġtest . </s> What Ġuser Ġbehaviors Ġare Ġpredictive Ġof Ġdog mat ism ? Ġ( R 3 ) </s> We Ġhave Ġshown Ġdog mat ism Ġis Ġcaptured Ġby Ġmany Ġlinguistic Ġfeatures , Ġbut Ġcan Ġwe Ġdiscover Ġother Ġhigh - level Ġuser Ġbehaviors Ġthat Ġare Ġsimilarly Ġpredictive ? </s> To Ġfind Ġout , Ġwe Ġcompute Ġmetrics Ġof Ġuser Ġbehavior Ġusing Ġthe Ġdata Ġsample Ġof Ġ1000 Ġusers Ġand Ġ10 Ġmillion Ġposts Ġdescribed Ġin ĠSection Ġ5 . 2 . ĠSpecifically , Ġwe Ġcalculate Ġ( 1 ) Ġactivity : Ġa Ġuser 's Ġtotal Ġnumber Ġof Ġposts , Ġ( 2 ) Ġbreadth : Ġthe Ġnumber Ġof Ġsubreddits Ġa Ġuser Ġhas Ġposted Ġin , Ġ( 3 ) Ġfocus : Ġthe Ġproportion Ġof Ġa Ġuser 's Ġposts Ġthat Ġappear Ġin Ġthe Ġsubreddit Ġwhere Ġthey Ġare Ġmost Ġactive , Ġand Ġ( 4 ) Ġengagement : Ġthe Ġaverage Ġnumber Ġof Ġposts Ġa Ġuser Ġcontributes Ġto Ġeach Ġdiscussion Ġthey Ġengage Ġin . ĠWe Ġthen Ġfit Ġthese Ġbehavioral Ġfeatures Ġto Ġa Ġlinear Ġregression Ġmodel Ġwhere Ġwe Ġpredict Ġeach Ġuser 's Ġaverage Ġdog mat ism Ġlevel . ĠPositive Ġcoefficients Ġin Ġthis Ġmodel Ġare Ġpositively Ġpredictive Ġof Ġdog mat ism , Ġwhile Ġnegative Ġcoefficients Ġare Ġnegatively Ġpredictive . ĠWe Ġfind Ġthis Ġmodel Ġis Ġsignificantly Ġpredic itive Ġof Ġdog mat ism Ġ( Ġ$ R ^ 2 = 0 . 1 $ Ġ, Ġ$ p < 0 . 001 $ Ġ), Ġwith Ġall Ġfeatures Ġreaching Ġstatistical Ġsignificance Ġ( Ġ$ p < 0 . 001 $ Ġ). ĠActivity Ġand Ġfocus Ġare Ġpositively Ġassociated Ġwith Ġdog mat ism , Ġwhile Ġbreadth Ġand Ġengagement Ġare Ġnegatively Ġassociated Ġ( Table Ġ5 Ġ). ĠTogether , Ġthese Ġresults Ġsuggest Ġdog matic Ġusers Ġtend Ġto Ġpost Ġfrequently Ġand Ġin Ġspecific Ġcommunities , Ġbut Ġare Ġnot Ġas Ġinclined Ġto Ġcontinue Ġto Ġengage Ġwith Ġa Ġdiscussion , Ġonce Ġit Ġhas Ġbegun . </s> How Ġdoes Ġdog mat ism Ġimpact Ġa Ġconversation ? Ġ( R 4 ) </s> How Ġdoes Ġinteracting Ġwith Ġa Ġdog matic Ġcomment Ġimpact Ġa Ġconversation ? ĠAre Ġusers Ġable Ġto Ġshrug Ġit Ġoff ? ĠOr Ġdo Ġotherwise Ġnon - dog matic Ġusers Ġbecome Ġmore Ġdog matic Ġthemselves ? </s> To Ġanswer Ġthis Ġquestion , Ġwe Ġsample Ġ600 , 000 Ġconversations Ġtri ples Ġfrom ĠReddit . ĠThese Ġconversations Ġconsist Ġof Ġtwo Ġpeople Ġ( A Ġand ĠB ) Ġtalking , Ġwith Ġthe Ġstructure : ĠA 1 Ġ$\ right arrow Ġ$ ĠB Ġ$\ right arrow Ġ$ ĠA 2 . ĠThis Ġallows Ġus Ġto Ġmeasure Ġthe Ġimpact Ġof ĠB 's Ġdog mat ism Ġon ĠA 's Ġresponse , Ġwhile Ġalso Ġcontrolling Ġfor Ġthe Ġdog mat ism Ġlevel Ġinitially Ġset Ġby ĠA . ĠCon crete ly , Ġwe Ġmodel Ġthe Ġimpact Ġof Ġdog mat ism Ġon Ġthese Ġconversations Ġthrough Ġa Ġlinear Ġregression . ĠThis Ġmodel Ġtakes Ġtwo Ġfeatures , Ġthe Ġdog mat ism Ġlevels Ġof ĠA 1 Ġand ĠB , Ġand Ġpredicts Ġthe Ġdog mat ism Ġresponse Ġof ĠA 2 . ĠIf ĠB 's Ġdog mat ism Ġhas Ġno Ġeffect Ġon ĠA 's Ġresponse , Ġthe Ġcoefficient Ġthat Ġcorresponds Ġto ĠB Ġwill Ġnot Ġbe Ġsignificant Ġin Ġthe Ġmodel . ĠAlternatively , Ġif ĠB 's Ġdog mat ism Ġdoes Ġhave Ġsome Ġeffect , Ġit Ġwill Ġbe Ġcaptured Ġby Ġthe Ġmodel 's Ġcoefficient . </s> We Ġfind Ġthe Ġcoefficient Ġof Ġthe ĠB Ġfeature Ġin Ġthe Ġmodel Ġis Ġpositively Ġassociated Ġwith Ġdog mat ism Ġ( Ġ$ p < 0 . 001 $ Ġ). ĠIn Ġother Ġwords , Ġengagement Ġwith Ġa Ġdog matic Ġcomment Ġtends Ġto Ġmake Ġa Ġuser Ġmore Ġdog matic Ġthemselves . ĠThis Ġeffect Ġholds Ġwhen Ġwe Ġrun Ġthe Ġsame Ġmodel Ġon Ġdata Ġsubs ets Ġconsisting Ġonly Ġof Ġdog matic Ġor Ġnon - dog matic Ġusers , Ġand Ġalso Ġwhen Ġwe Ġconservative ly Ġremove Ġall Ġwords Ġused Ġby ĠB Ġfrom ĠA 's Ġresponse Ġ( i . e ., Ġcontrolling Ġfor Ġquoting Ġeffects ). </s> Related ĠWork </s> In Ġcontrast Ġto Ġthe Ġcomputational Ġmodels Ġwe Ġhave Ġpresented , Ġdog mat ism Ġis Ġusually Ġmeasured Ġin Ġpsychology Ġthrough Ġsurvey Ġscales , Ġin Ġwhich Ġstudy Ġparticipants Ġanswer Ġquestions Ġdesigned Ġto Ġreveal Ġunderlying Ġpersonality Ġattributes ĠB IB REF 1 Ġ. ĠOver Ġtime , Ġthese Ġsurveys Ġhave Ġbeen Ġupdated ĠB IB REF 14 Ġand Ġimproved Ġto Ġmeet Ġstandards Ġof Ġpsych ometric Ġvalidity ĠB IB REF 15 Ġ. </s> These Ġsurveys Ġare Ġoften Ġused Ġto Ġstudy Ġthe Ġrelationship Ġbetween Ġdog mat ism Ġand Ġother Ġpsychological Ġphenomena . ĠFor Ġexample , Ġdog matic Ġpeople Ġtend Ġto Ġshow Ġan Ġincreased Ġtendency Ġfor Ġconfrontation ĠB IB REF 16 Ġor Ġmoral Ġconviction Ġand Ġrelig iosity ĠB IB REF 17 Ġ, Ġand Ġless Ġlikelihood Ġof Ġcognitive Ġflexibility ĠB IB REF 18 Ġ, Ġeven Ġamong Ġstereotyp ically Ġnon - dog matic Ġgroups Ġlike Ġatheists ĠB IB REF 19 Ġ. ĠFrom Ġa Ġbehavioral Ġstandpoint , Ġdog matic Ġpeople Ġsolve Ġproblems Ġdifferently , Ġspending Ġless Ġtime Ġframing Ġa Ġproblem Ġand Ġexpressing Ġmore Ġcertainty Ġin Ġtheir Ġsolution ĠB IB REF 20 Ġ. ĠHere Ġwe Ġsimilarly Ġexamine Ġhow Ġuser Ġbehaviors Ġon ĠReddit Ġrelate Ġto Ġa Ġlanguage Ġmodel Ġof Ġdog mat ism . </s> Er tel Ġsought Ġto Ġcapture Ġdog mat ism Ġlingu istically , Ġthough Ġa Ġsmall Ġlex icon Ġof Ġwords Ġthat Ġcorrespond Ġwith Ġhigh - level Ġconcepts Ġlike Ġcertainty Ġand Ġcompromise Ġd ota . ĠMcK enny Ġthen Ġused Ġthis Ġdictionary Ġto Ġrelate Ġdog mat ism Ġto Ġargument Ġquality Ġin Ġstudent Ġessays Ġdog mat ism - ess ays . ĠOur Ġwork Ġexpands Ġon Ġthis Ġapproach , Ġapplying Ġsupervised Ġmodels Ġbased Ġon Ġa Ġbroader Ġset Ġof Ġlinguistic Ġcategories Ġto Ġidentify Ġdog mat ism Ġin Ġtext . </s> Other Ġresearchers Ġhave Ġstudied Ġtopics Ġsimilar Ġto Ġdog mat ism , Ġsuch Ġas Ġsignals Ġof Ġcognitive Ġstyle Ġin Ġright - wing Ġpolitical Ġthought ĠB IB REF 21 Ġ, Ġthe Ġlanguage Ġused Ġby Ġtrolls Ġon Ġsocial Ġmedia ĠB IB REF 22 Ġ, Ġor Ġwhat Ġmakes Ġfor Ġimpartial Ġlanguage Ġon Ġtwitter ĠB IB REF 23 Ġ. ĠA Ġsimilar Ġflavor Ġof Ġwork Ġhas Ġexamined Ġlinguistic Ġmodels Ġthat Ġcapture Ġpolit eness ĠB IB REF 2 Ġ, Ġdeception ĠB IB REF 24 Ġ, Ġand Ġauthority ĠB IB REF 8 Ġ. ĠWe Ġtook Ġinspiration Ġfrom Ġthese Ġmodels Ġwhen Ġconstructing Ġthe Ġfeature Ġsets Ġin Ġour Ġwork . </s> Finally , Ġwhile Ġwe Ġexamine Ġwhat Ġmakes Ġan Ġopinion Ġdog matic , Ġother Ġwork Ġhas Ġpushed Ġfurther Ġinto Ġthe Ġstructure Ġof Ġarguments , Ġfor Ġexample Ġclass ifying Ġtheir Ġjust ifications ĠB IB REF 25 Ġ, Ġor Ġwhat Ġmakes Ġan Ġargument Ġlikely Ġto Ġwin ĠB IB REF 26 Ġ. ĠOur Ġmodel Ġmay Ġallow Ġfuture Ġresearchers Ġto Ġprobe Ġthese Ġquestions Ġmore Ġdeeply . </s> Conclusion </s> We Ġhave Ġconstructed Ġthe Ġfirst Ġcorpus Ġof Ġsocial Ġmedia Ġposts Ġannot ated Ġwith Ġdog mat ism Ġscores , Ġallowing Ġus Ġto Ġexplore Ġlinguistic Ġfeatures Ġof Ġdog mat ism Ġand Ġbuild Ġa Ġpredictive Ġmodel Ġthat Ġanaly zes Ġnew Ġcontent . ĠWe Ġapply Ġthis Ġmodel Ġto ĠReddit , Ġwhere Ġwe Ġdiscover Ġbehavioral Ġpredict ors Ġof Ġdog mat ism Ġand Ġtopical Ġpatterns Ġin Ġthe Ġcomments Ġof Ġdog matic Ġusers . </s> Could Ġwe Ġuse Ġthis Ġcomputational Ġmodel Ġto Ġhelp Ġusers Ġshed Ġtheir Ġdog matic Ġbeliefs ? ĠLooking Ġforward , Ġour Ġwork Ġmakes Ġpossible Ġnew Ġavenues Ġfor Ġencouraging Ġpro - social Ġbehavior Ġin Ġonline Ġcommunities .