BIBREF12 , BIBREF13
EXTRACTIVE
an existing, annotated Twitter dataset that was constructed based on a hierarchical model of depression-related symptoms BIBREF12 , BIBREF13
EXTRACTIVE
Specifically, we conducted a feature ablation study to assess the informativeness of each feature group and a feature elimination study to determine the optimal feature sets for classifying Twitter tweets. We leveraged an existing, annotated Twitter dataset that was constructed based on a hierarchical model of depression-related symptoms BIBREF12 , BIBREF13 . The dataset contains 9,473 annotations for 9,300 tweets. Each tweet is annotated as no evidence of depression (e.g., “Citizens fear an economic depression") or evidence of depression (e.g., “depressed over disappointment"). If a tweet is annotated evidence of depression, then it is further annotated with one or more depressive symptoms, for example, depressed mood (e.g., “feeling down in the dumps"), disturbed sleep (e.g., “another restless night"), or fatigue or loss of energy (e.g., “the fatigue is unbearable") BIBREF10 . For each class, every annotation (9,473 tweets) is binarized as the positive class e.g., depressed mood=1 or negative class e.g., not depressed mood=0.
Specifically, we conducted a feature ablation study to assess the informativeness of each feature group and a feature elimination study to determine the optimal feature sets for classifying Twitter tweets. We leveraged an existing, annotated Twitter dataset that was constructed based on a hierarchical model of depression-related symptoms BIBREF12 , BIBREF13 . The dataset contains 9,473 annotations for 9,300 tweets. Each tweet is annotated as no evidence of depression (e.g., “Citizens fear an economic depression") or evidence of depression (e.g., “depressed over disappointment"). If a tweet is annotated evidence of depression, then it is further annotated with one or more depressive symptoms, for example, depressed mood (e.g., “feeling down in the dumps"), disturbed sleep (e.g., “another restless night"), or fatigue or loss of energy (e.g., “the fatigue is unbearable") BIBREF10 . For each class, every annotation (9,473 tweets) is binarized as the positive class e.g., depressed mood=1 or negative class e.g., not depressed mood=0.
1701.08229
Introduction </s> In Ġrecent Ġyears , Ġthere Ġhas Ġbeen Ġa Ġmovement Ġto Ġleverage Ġsocial Ġmedial Ġdata Ġto Ġdetect , Ġestimate , Ġand Ġtrack Ġthe Ġchange Ġin Ġprevalence Ġof Ġdisease . ĠFor Ġexample , Ġeating Ġdisorders Ġin ĠSpanish Ġlanguage ĠTwitter Ġtweets ĠB IB REF 0 Ġand Ġinfluenza Ġsurveillance ĠB IB REF 1 Ġ. ĠMore Ġrecently , Ġsocial Ġmedia Ġhas Ġbeen Ġlever aged Ġto Ġmonitor Ġsocial Ġrisks Ġsuch Ġas Ġprescription Ġdrug Ġand Ġsmoking Ġbehaviors ĠB IB REF 2 Ġ, ĠB IB REF 3 Ġ, ĠB IB REF 4 Ġas Ġwell Ġas Ġa Ġvariety Ġof Ġmental Ġhealth Ġdisorders Ġincluding Ġsuicidal Ġide ation ĠB IB REF 5 Ġ, Ġattention Ġdeficient Ġhyper activity Ġdisorder ĠB IB REF 6 Ġand Ġmajor Ġdepressive Ġdisorder ĠB IB REF 7 Ġ. ĠIn Ġthe Ġcase Ġof Ġmajor Ġdepressive Ġdisorder , Ġrecent Ġefforts Ġrange Ġfrom Ġcharacter izing Ġlinguistic Ġphenomena Ġassociated Ġwith Ġdepression ĠB IB REF 8 Ġand Ġits Ġsub types Ġe . g ., Ġpost part um Ġdepression ĠB IB REF 5 Ġ, Ġto Ġidentifying Ġspecific Ġdepressive Ġsymptoms ĠB IB REF 9 Ġ, ĠB IB REF 10 Ġe . g ., Ġdepressed Ġmood . ĠHowever , Ġmore Ġresearch Ġis Ġneeded Ġto Ġbetter Ġunderstand Ġthe Ġpredictive Ġpower Ġof Ġsupervised Ġmachine Ġlearning Ġclass ifiers Ġand Ġthe Ġinfluence Ġof Ġfeature Ġgroups Ġand Ġfeature Ġsets Ġfor Ġefficiently Ġclass ifying Ġdepression - related Ġtweets Ġto Ġsupport Ġmental Ġhealth Ġmonitoring Ġat Ġthe Ġpopulation - level ĠB IB REF 11 Ġ. </s> This Ġpaper Ġbuilds Ġupon Ġrelated Ġworks Ġtoward Ġclass ifying ĠTwitter Ġtweets Ġrepresenting Ġsymptoms Ġof Ġmajor Ġdepressive Ġdisorder Ġby Ġassessing Ġthe Ġcontribution Ġof Ġlex ical Ġfeatures Ġ( e . g ., Ġun ig rams ) Ġand Ġemotion Ġ( e . g ., Ġstrongly Ġnegative ) Ġto Ġclassification Ġperformance , Ġand Ġby Ġapplying Ġmethods Ġto Ġeliminate Ġlow - value Ġfeatures . </s> METHOD S </s> Specifically , Ġwe Ġconducted Ġa Ġfeature Ġab lation Ġstudy Ġto Ġassess Ġthe Ġinform at iveness Ġof Ġeach Ġfeature Ġgroup Ġand Ġa Ġfeature Ġelimination Ġstudy Ġto Ġdetermine Ġthe Ġoptimal Ġfeature Ġsets Ġfor Ġclass ifying ĠTwitter Ġtweets . ĠWe Ġlever aged Ġan Ġexisting , Ġannot ated ĠTwitter Ġdataset Ġthat Ġwas Ġconstructed Ġbased Ġon Ġa Ġhierarchical Ġmodel Ġof Ġdepression - related Ġsymptoms ĠB IB REF 12 Ġ, ĠB IB REF 13 Ġ. ĠThe Ġdataset Ġcontains Ġ9 , 473 Ġannotations Ġfor Ġ9 , 300 Ġtweets . ĠEach Ġtweet Ġis Ġannot ated Ġas Ġno Ġevidence Ġof Ġdepression Ġ( e . g ., ĠâĢ ľ C itizens Ġfear Ġan Ġeconomic Ġdepression ") Ġor Ġevidence Ġof Ġdepression Ġ( e . g ., ĠâĢ ľ dep ressed Ġover Ġdisappointment "). ĠIf Ġa Ġtweet Ġis Ġannot ated Ġevidence Ġof Ġdepression , Ġthen Ġit Ġis Ġfurther Ġannot ated Ġwith Ġone Ġor Ġmore Ġdepressive Ġsymptoms , Ġfor Ġexample , Ġdepressed Ġmood Ġ( e . g ., ĠâĢ ľ fe eling Ġdown Ġin Ġthe Ġdumps "), Ġdisturbed Ġsleep Ġ( e . g ., ĠâĢ ľ another Ġrestless Ġnight "), Ġor Ġfatigue Ġor Ġloss Ġof Ġenergy Ġ( e . g ., ĠâĢ ľ the Ġfatigue Ġis Ġunbearable ") ĠB IB REF 10 Ġ. ĠFor Ġeach Ġclass , Ġevery Ġannotation Ġ( 9 , 473 Ġtweets ) Ġis Ġbin ar ized Ġas Ġthe Ġpositive Ġclass Ġe . g ., Ġdepressed Ġmood = 1 Ġor Ġnegative Ġclass Ġe . g ., Ġnot Ġdepressed Ġmood = 0 . </s> Features </s> Furthermore , Ġthis Ġdataset Ġwas Ġencoded Ġwith Ġ7 Ġfeature Ġgroups Ġwith Ġassociated Ġfeature Ġvalues Ġbin ar ized Ġ( i . e ., Ġpresent = 1 Ġor Ġabsent = 0 ) Ġto Ġrepresent Ġpotentially Ġinformative Ġfeatures Ġfor Ġclass ifying Ġdepression - related Ġclasses . ĠWe Ġdescribe Ġthe Ġfeature Ġgroups Ġby Ġtype , Ġsub type , Ġand Ġprovide Ġone Ġor Ġmore Ġexamples Ġof Ġwords Ġrepresenting Ġthe Ġfeature Ġsub type Ġfrom Ġa Ġtweet : </s> lex ical Ġfeatures , Ġun ig rams , Ġe . g ., ĠâĢ ľ dep ressed âĢ Ŀ ; </s> sy nt actic Ġfeatures , Ġparts Ġof Ġspeech , Ġe . g ., ĠâĢ ľ c ried âĢ Ŀ Ġencoded Ġas ĠV Ġfor Ġverb ; </s> em otion Ġfeatures , Ġemot icons , Ġe . g ., Ġ:( Ġencoded Ġas ĠS AD ; </s> dem ographic Ġfeatures , Ġage Ġand Ġgender Ġe . g ., ĠâĢ ľ this Ġsemester âĢ Ŀ Ġencoded Ġas Ġan Ġindicator Ġof Ġ19 - 22 Ġyears Ġof Ġage Ġand ĠâĢ ľ my Ġgirlfriend âĢ Ŀ Ġencoded Ġas Ġan Ġindicator Ġof Ġmale Ġgender , Ġrespectively ; </s> sent iment Ġfeatures , Ġpol arity Ġand Ġsubject ivity Ġterms Ġwith Ġstrengths , Ġe . g ., ĠâĢ ľ ter rible âĢ Ŀ Ġencoded Ġas Ġstrongly Ġnegative Ġand Ġstrongly Ġsubjective ; </s> person ality Ġtraits , Ġneurot icism Ġe . g ., ĠâĢ ľ p iss ed Ġoff âĢ Ŀ Ġimplies Ġneurot icism ; </s> LI WC ĠFeatures , Ġindicators Ġof Ġan Ġindividual 's Ġthoughts , Ġfeelings , Ġpersonality , Ġand Ġmotivations , Ġe . g ., ĠâĢ ľ fe eling âĢ Ŀ Ġsuggestions Ġperception , Ġfeeling , Ġinsight , Ġand Ġcognitive Ġmechanisms Ġexperienced Ġby Ġthe ĠTwitter Ġuser . </s> A Ġmore Ġdetailed Ġdescription Ġof Ġlever aged Ġfeatures Ġand Ġtheir Ġvalues , Ġincluding ĠLI WC Ġcategories , Ġcan Ġbe Ġfound Ġin ĠB IB REF 10 Ġ. </s> Based Ġon Ġour Ġprior Ġinitial Ġexperiments Ġusing Ġthese Ġfeature Ġgroups ĠB IB REF 10 Ġ, Ġwe Ġlearned Ġthat Ġsupport Ġvector Ġmachines Ġperform Ġwith Ġthe Ġhighest ĠF 1 - score Ġcompared Ġto Ġother Ġsupervised Ġapproaches . ĠFor Ġthis Ġstudy , Ġwe Ġaim Ġto Ġbuild Ġupon Ġthis Ġwork Ġby Ġconducting Ġtwo Ġexperiments : Ġ1 ) Ġto Ġassess Ġthe Ġcontribution Ġof Ġeach Ġfeature Ġgroup Ġand Ġ2 ) Ġto Ġdetermine Ġthe Ġoptimal Ġpercentile Ġof Ġtop Ġranked Ġfeatures Ġfor Ġclass ifying ĠTwitter Ġtweets Ġin Ġthe Ġdepression Ġschema Ġhierarchy . </s> Feature ĠCont ribution </s> Feature Ġab lation Ġstudies Ġare Ġconducted Ġto Ġassess Ġthe Ġinform at iveness Ġof Ġa Ġfeature Ġgroup Ġby Ġquant ifying Ġthe Ġchange Ġin Ġpredictive Ġpower Ġwhen Ġcomparing Ġthe Ġperformance Ġof Ġa Ġclass ifier Ġtrained Ġwith Ġthe Ġall Ġfeature Ġgroups Ġversus Ġthe Ġperformance Ġwithout Ġa Ġparticular Ġfeature Ġgroup . ĠWe Ġconducted Ġa Ġfeature Ġab lation Ġstudy Ġby Ġholding Ġout Ġ( s ans ) Ġeach Ġfeature Ġgroup Ġand Ġtraining Ġand Ġtesting Ġthe Ġsupport Ġvector Ġmodel Ġusing Ġa Ġlinear Ġkernel Ġand Ġ5 - fold , Ġstrat ified Ġcross - valid ation . ĠWe Ġreport Ġthe Ġaverage ĠF 1 - score Ġfrom Ġour Ġbaseline Ġapproach Ġ( all Ġfeature Ġgroups ) Ġand Ġreport Ġthe Ġpoint Ġdifference Ġ(+ Ġor Ġ- ) Ġin ĠF 1 - score Ġperformance Ġobserved Ġby Ġabl ating Ġeach Ġfeature Ġset . </s> By Ġabl ating Ġeach Ġfeature Ġgroup Ġfrom Ġthe Ġfull Ġdataset , Ġwe Ġobserved Ġthe Ġfollowing Ġcount Ġof Ġfeatures Ġ- Ġsans Ġlex ical : Ġ185 , Ġsans Ġsynt actic : Ġ16 , 9 35 , Ġsans Ġemotion : Ġ16 , 954 , Ġsans Ġdemographics : Ġ16 , 9 46 , Ġsans Ġsentiment : Ġ16 , 950 , Ġsans Ġpersonality : Ġ16 , 9 46 , Ġand Ġsans ĠLI WC : Ġ16 , 8 32 . ĠIn ĠFigure Ġ1 , Ġcompared Ġto Ġthe Ġbaseline Ġperformance , Ġsignificant Ġdrops Ġin ĠF 1 - sc ores Ġresulted Ġfrom Ġsans Ġlex ical Ġfor Ġdepressed Ġmood Ġ(- 35 Ġpoints ), Ġdisturbed Ġsleep Ġ(- 43 Ġpoints ), Ġand Ġdepressive Ġsymptoms Ġ(- 45 Ġpoints ). ĠLess Ġextensive Ġdrops Ġalso Ġoccurred Ġfor Ġevidence Ġof Ġdepression Ġ(- 14 Ġpoints ) Ġand Ġfatigue Ġor Ġloss Ġof Ġenergy Ġ(- 3 Ġpoints ). ĠIn Ġcontrast , Ġa Ġ3 Ġpoint Ġgain Ġin ĠF 1 - score Ġwas Ġobserved Ġfor Ġno Ġevidence Ġof Ġdepression . ĠWe Ġalso Ġobserved Ġnotable Ġdrops Ġin ĠF 1 - sc ores Ġfor Ġdisturbed Ġsleep Ġby Ġabl ating Ġdemographics Ġ(- 7 Ġpoints ), Ġemotion Ġ(- 5 Ġpoints ), Ġand Ġsentiment Ġ(- 5 Ġpoints ) Ġfeatures . ĠThese ĠF 1 - score Ġdrops Ġwere Ġaccompanied Ġby Ġdrops Ġin Ġboth Ġrecall Ġand Ġprecision . ĠWe Ġfound Ġequal Ġor Ġhigher ĠF 1 - sc ores Ġby Ġremoving Ġnon - lex ical Ġfeature Ġgroups Ġfor Ġno Ġevidence Ġof Ġdepression Ġ( 0 - 1 Ġpoints ), Ġevidence Ġof Ġdepression Ġ( 0 - 1 Ġpoints ), Ġand Ġdepressive Ġsymptoms Ġ( 2 Ġpoints ). </s> Un surprisingly , Ġlex ical Ġfeatures Ġ( un ig rams ) Ġwere Ġthe Ġlargest Ġcontributor Ġto Ġfeature Ġcounts Ġin Ġthe Ġdataset . ĠWe Ġobserved Ġthat Ġlex ical Ġfeatures Ġare Ġalso Ġcritical Ġfor Ġidentifying Ġdepressive Ġsymptoms , Ġspecifically Ġfor Ġdepressed Ġmood Ġand Ġfor Ġdisturbed Ġsleep . ĠFor Ġthe Ġclasses Ġhigher Ġin Ġthe Ġhierarchy Ġ- Ġno Ġevidence Ġof Ġdepression , Ġevidence Ġof Ġdepression , Ġand Ġdepressive Ġsymptoms Ġ- Ġthe Ġclass ifier Ġproduced Ġconsistent ĠF 1 - sc ores , Ġeven Ġslightly Ġabove Ġthe Ġbaseline Ġfor Ġdepressive Ġsymptoms Ġand Ġminor Ġfluctuations Ġof Ġchange Ġin Ġrecall Ġand Ġprecision Ġwhen Ġremoving Ġother Ġfeature Ġgroups Ġsuggesting Ġthat Ġthe Ġcontribution Ġof Ġnon - lex ical Ġfeatures Ġto Ġclassification Ġperformance Ġwas Ġlimited . ĠHowever , Ġnotable Ġchanges Ġin ĠF 1 - score Ġwere Ġobserved Ġfor Ġthe Ġclasses Ġlower Ġin Ġthe Ġhierarchy Ġincluding Ġdisturbed Ġsleep Ġand Ġfatigue Ġor Ġloss Ġof Ġenergy . ĠFor Ġinstance , Ġchanges Ġin ĠF 1 - sc ores Ġdriven Ġby Ġboth Ġrecall Ġand Ġprecision Ġwere Ġobserved Ġfor Ġdisturbed Ġsleep Ġby Ġabl ating Ġdemographics , Ġemotion , Ġand Ġsentiment Ġfeatures , Ġsuggesting Ġthat Ġage Ġor Ġgender Ġ( âĢ ľ mid - se mes ter Ġexams Ġhave Ġme Ġrestless âĢ Ŀ ), Ġpol arity Ġand Ġsubjective Ġterms Ġ( âĢ ľ l ack Ġof Ġsleep Ġis Ġkilling Ġme âĢ Ŀ ), Ġand Ġemot icons Ġ( âĢ ľ wide Ġawake Ġ:( âĢ Ŀ ) Ġcould Ġbe Ġimportant Ġfor Ġboth Ġidentifying Ġand Ġcorrectly Ġclass ifying Ġa Ġsubset Ġof Ġthese Ġtweets . </s> Feature ĠElim ination </s> Feature Ġelimination Ġstrategies Ġare Ġoften Ġtaken Ġ1 ) Ġto Ġremove Ġirrelevant Ġor Ġnoisy Ġfeatures , Ġ2 ) Ġto Ġimprove Ġclass ifier Ġperformance , Ġand Ġ3 ) Ġto Ġreduce Ġtraining Ġand Ġrun Ġtimes . ĠWe Ġconducted Ġan Ġexperiment Ġto Ġdetermine Ġwhether Ġwe Ġcould Ġmaintain Ġor Ġimprove Ġclass ifier Ġperformances Ġby Ġapplying Ġthe Ġfollowing Ġthree - tie red Ġfeature Ġelimination Ġapproach : </s> Red uction ĠWe Ġreduced Ġthe Ġdataset Ġencoded Ġfor Ġeach Ġclass Ġby Ġeliminating Ġfeatures Ġthat Ġoccur Ġless Ġthan Ġtwice Ġin Ġthe Ġfull Ġdataset . </s> Se lection ĠWe Ġiter atively Ġapplied ĠChi - Square Ġfeature Ġselection Ġon Ġthe Ġreduced Ġdataset , Ġselecting Ġthe Ġtop Ġpercentile Ġof Ġhighest Ġranked Ġfeatures Ġin Ġincrements Ġof Ġ5 Ġpercent Ġto Ġtrain Ġand Ġtest Ġthe Ġsupport Ġvector Ġmodel Ġusing Ġa Ġlinear Ġkernel Ġand Ġ5 - fold , Ġstrat ified Ġcross - valid ation . </s> Rank ĠWe Ġcum ul atively Ġplotted Ġthe Ġaverage ĠF 1 - score Ġperformances Ġof Ġeach Ġincrement ally Ġadded Ġpercentile Ġof Ġtop Ġranked Ġfeatures . ĠWe Ġreport Ġthe Ġpercentile Ġand Ġcount Ġof Ġfeatures Ġresulting Ġin Ġthe Ġfirst Ġoccurrence Ġof Ġthe Ġhighest Ġaverage ĠF 1 - score Ġfor Ġeach Ġclass . </s> All Ġexperiments Ġwere Ġprogrammed Ġusing Ġsc ik it - learn Ġ0 . 18 . </s> The Ġinitial Ġmat rices Ġof Ġalmost Ġ17 , 000 Ġfeatures Ġwere Ġreduced Ġby Ġeliminating Ġfeatures Ġthat Ġonly Ġoccurred Ġonce Ġin Ġthe Ġfull Ġdataset , Ġresulting Ġin Ġ5 , 76 1 Ġfeatures . ĠWe Ġapplied ĠChi - Square Ġfeature Ġselection Ġand Ġplotted Ġthe Ġtop - ranked Ġsubset Ġof Ġfeatures Ġfor Ġeach Ġpercentile Ġ( at Ġ5 Ġpercent Ġintervals Ġcum ul atively Ġadded ) Ġand Ġevaluated Ġtheir Ġpredictive Ġcontribution Ġusing Ġthe Ġsupport Ġvector Ġmachine Ġwith Ġlinear Ġkernel Ġand Ġstrat ified , Ġ5 - fold Ġcross Ġvalidation . </s> In ĠFigure Ġ2 , Ġwe Ġobserved Ġoptimal ĠF 1 - score Ġperformance Ġusing Ġthe Ġfollowing Ġtop Ġfeature Ġcounts : Ġno Ġevidence Ġof Ġdepression : ĠF 1 : Ġ87 Ġ( 15 th Ġpercentile , Ġ8 64 Ġfeatures ), Ġevidence Ġof Ġdepression : ĠF 1 : Ġ59 Ġ( 30 th Ġpercentile , Ġ1 , 728 Ġfeatures ), Ġdepressive Ġsymptoms : ĠF 1 : Ġ55 Ġ( 15 th Ġpercentile , Ġ8 64 Ġfeatures ), Ġdepressed Ġmood : ĠF 1 : Ġ39 Ġ( 55 th Ġpercentile , Ġ3 , 168 Ġfeatures ), Ġdisturbed Ġsleep : ĠF 1 : Ġ46 Ġ( 10 th Ġpercentile , Ġ5 76 Ġfeatures ), Ġand Ġfatigue Ġor Ġloss Ġof Ġenergy : ĠF 1 : Ġ72 Ġ( 5 th Ġpercentile , Ġ288 Ġfeatures ) Ġ( Figure Ġ1 ). ĠWe Ġnote ĠF 1 - score Ġimprovements Ġfor Ġdepressed Ġmood Ġfrom ĠF 1 : Ġ13 Ġat Ġthe Ġ1 st Ġpercentile Ġto ĠF 1 : Ġ33 Ġat Ġthe Ġ20 th Ġpercentile . </s> We Ġobserved Ġpeak ĠF 1 - score Ġperformances Ġat Ġlow Ġpercent iles Ġfor Ġfatigue Ġor Ġloss Ġof Ġenergy Ġ( 5 th Ġpercentile ), Ġdisturbed Ġsleep Ġ( 10 th Ġpercentile ) Ġas Ġwell Ġas Ġdepressive Ġsymptoms Ġand Ġno Ġevidence Ġof Ġdepression Ġ( both Ġ15 th Ġpercentile ) Ġsuggesting Ġfewer Ġfeatures Ġare Ġneeded Ġto Ġreach Ġoptimal Ġperformance . ĠIn Ġcontrast , Ġpeak ĠF 1 - score Ġperformances Ġoccurred Ġat Ġmoderate Ġpercent iles Ġfor Ġevidence Ġof Ġdepression Ġ( 30 th Ġpercentile ) Ġand Ġdepressed Ġmood Ġ( 55 th Ġpercentile ) Ġsuggesting Ġthat Ġmore Ġfeatures Ġare Ġneeded Ġto Ġreach Ġoptimal Ġperformance . ĠHowever , Ġone Ġnotable Ġdifference Ġbetween Ġthese Ġtwo Ġclasses Ġis Ġthe Ġdramatic ĠF 1 - score Ġimprovements Ġfor Ġdepressed Ġmood Ġi . e ., Ġ20 Ġpoint Ġincrease Ġfrom Ġthe Ġ1 st Ġpercentile Ġto Ġthe Ġ20 th Ġpercentile Ġcompared Ġto Ġthe Ġmore Ġgradual ĠF 1 - score Ġimprovements Ġfor Ġevidence Ġof Ġdepression Ġi . e ., Ġ11 Ġpoint Ġincrease Ġfrom Ġthe Ġ1 st Ġpercentile Ġto Ġthe Ġ20 th Ġpercentile . ĠThis Ġfinding Ġsuggests Ġthat Ġfor Ġidentifying Ġdepressed Ġmood Ġa Ġvariety Ġof Ġfeatures Ġare Ġneeded Ġbefore Ġincremental Ġgains Ġare Ġobserved . </s> RESULTS </s> From Ġour Ġannot ated Ġdataset Ġof ĠTwitter Ġtweets Ġ( n = 9 , 300 Ġtweets ), Ġwe Ġconducted Ġtwo Ġfeature Ġstudies Ġto Ġbetter Ġunderstand Ġthe Ġpredictive Ġpower Ġof Ġseveral Ġfeature Ġgroups Ġfor Ġclass ifying Ġwhether Ġor Ġnot Ġa Ġtweet Ġcontains Ġno Ġevidence Ġof Ġdepression Ġ( n = 6 , 8 29 Ġtweets ) Ġor Ġevidence Ġof Ġdepression Ġ( n = 2 , 644 Ġtweets ). ĠIf Ġthere Ġwas Ġevidence Ġof Ġdepression , Ġwe Ġdetermined Ġwhether Ġthe Ġtweet Ġcontained Ġone Ġor Ġmore Ġdepressive Ġsymptoms Ġ( n = 1 , 656 Ġtweets ) Ġand Ġfurther Ġclassified Ġthe Ġsymptom Ġsub type Ġof Ġdepressed Ġmood Ġ( n = 1 , 010 Ġtweets ), Ġdisturbed Ġsleep Ġ( n = 98 Ġtweets ), Ġor Ġfatigue Ġor Ġloss Ġof Ġenergy Ġ( n = 427 Ġtweets ) Ġusing Ġsupport Ġvector Ġmachines . ĠFrom Ġour Ġprior Ġwork ĠB IB REF 10 Ġand Ġin ĠFigure Ġ1 , Ġwe Ġreport Ġthe Ġperformance Ġfor Ġprediction Ġmodels Ġbuilt Ġby Ġtraining Ġa Ġsupport Ġvector Ġmachine Ġusing Ġ5 - fold , Ġstrat ified Ġcross - valid ation Ġwith Ġall Ġfeature Ġgroups Ġas Ġa Ġbaseline Ġfor Ġeach Ġclass . ĠWe Ġobserved Ġhigh Ġperformance Ġfor Ġno Ġevidence Ġof Ġdepression Ġand Ġfatigue Ġor Ġloss Ġof Ġenergy Ġand Ġmoderate Ġperformance Ġfor Ġall Ġremaining Ġclasses . </s> Discussion </s> We Ġconducted Ġtwo Ġfeature Ġstudy Ġexperiments : Ġ1 ) Ġa Ġfeature Ġab lation Ġstudy Ġto Ġassess Ġthe Ġcontribution Ġof Ġfeature Ġgroups Ġand Ġ2 ) Ġa Ġfeature Ġelimination Ġstudy Ġto Ġdetermine Ġthe Ġoptimal Ġpercentile Ġof Ġtop Ġranked Ġfeatures Ġfor Ġclass ifying ĠTwitter Ġtweets Ġin Ġthe Ġdepression Ġschema Ġhierarchy . </s> Future ĠWork </s> Our Ġnext Ġstep Ġis Ġto Ġaddress Ġthe Ġclassification Ġof Ġrare r Ġdepressive Ġsymptoms Ġsuggestive Ġof Ġmajor Ġdepressive Ġdisorder Ġfrom Ġour Ġdataset Ġand Ġhierarchy Ġincluding Ġinappropriate Ġguilt , Ġdifficulty Ġconcentrating , Ġpsych om otor Ġagitation Ġor Ġretard ation , Ġweight Ġloss Ġor Ġgain , Ġand Ġan hed onia ĠB IB REF 15 Ġ, ĠB IB REF 16 Ġ. ĠWe Ġare Ġdeveloping Ġa Ġpopulation - level Ġmonitoring Ġframework Ġdesigned Ġto Ġestimate Ġthe Ġprevalence Ġof Ġdepression Ġ( and Ġdepression - related Ġsymptoms Ġand Ġpsycho - social Ġstress ors ) Ġover Ġmillions Ġof ĠUnited ĠStates - ge oc oded Ġtweets . ĠIdent ifying Ġthe Ġmost Ġdiscriminating Ġfeature Ġsets Ġand Ġnatural Ġlanguage Ġprocessing Ġclass ifiers Ġfor Ġeach Ġdepression Ġsymptom Ġis Ġvital Ġfor Ġthis Ġgoal . </s> Con clusions </s> In Ġsummary , Ġwe Ġconducted Ġtwo Ġfeature Ġstudy Ġexperiments Ġto Ġassess Ġthe Ġcontribution Ġof Ġfeature Ġgroups Ġand Ġto Ġdetermine Ġthe Ġoptimal Ġpercentile Ġof Ġtop Ġranked Ġfeatures Ġfor Ġclass ifying ĠTwitter Ġtweets Ġin Ġthe Ġdepression Ġschema Ġhierarchy . ĠFrom Ġthese Ġexperiments , Ġwe Ġconclude Ġthat Ġsimple Ġlex ical Ġfeatures Ġand Ġreduced Ġfeature Ġsets Ġcan Ġproduce Ġcomparable Ġresults Ġto Ġthe Ġmuch Ġlarger Ġfeature Ġdataset . </s> Acknowled gments </s> Research Ġreported Ġin Ġthis Ġpublication Ġwas Ġsupported Ġby Ġthe ĠNational ĠLibrary Ġof ĠMedicine Ġof Ġthe Ġ[ United ĠStates ] ĠNational ĠInstitutes Ġof ĠHealth Ġunder Ġaward Ġnumbers ĠK 99 LM 01 13 93 Ġand ĠR 00 LM 01 13 93 . ĠThis Ġstudy Ġwas Ġgranted Ġan Ġexemption Ġfrom Ġreview Ġby Ġthe ĠUniversity Ġof ĠUtah ĠInst itutional ĠReview ĠBoard Ġ( IR B Ġ000 76 188 ). ĠNote Ġthat Ġin Ġorder Ġto Ġprotect Ġtwe eter Ġanonymity , Ġwe Ġhave Ġnot Ġreproduced Ġtweets Ġver bat im . ĠExample Ġtweets Ġshown Ġwere Ġgenerated Ġby Ġthe Ġresearchers Ġas Ġexempl ars Ġonly . ĠFinally , Ġwe Ġwould Ġlike Ġto Ġthank Ġthe Ġanonymous Ġreviewers Ġof Ġthis Ġpaper Ġfor Ġtheir Ġvaluable Ġcomments .
Introduction In recent years, there has been a movement to leverage social medial data to detect, estimate, and track the change in prevalence of disease. For example, eating disorders in Spanish language Twitter tweets BIBREF0 and influenza surveillance BIBREF1 . More recently, social media has been leveraged to monitor social risks such as prescription drug and smoking behaviors BIBREF2 , BIBREF3 , BIBREF4 as well as a variety of mental health disorders including suicidal ideation BIBREF5 , attention deficient hyperactivity disorder BIBREF6 and major depressive disorder BIBREF7 . In the case of major depressive disorder, recent efforts range from characterizing linguistic phenomena associated with depression BIBREF8 and its subtypes e.g., postpartum depression BIBREF5 , to identifying specific depressive symptoms BIBREF9 , BIBREF10 e.g., depressed mood. However, more research is needed to better understand the predictive power of supervised machine learning classifiers and the influence of feature groups and feature sets for efficiently classifying depression-related tweets to support mental health monitoring at the population-level BIBREF11 . This paper builds upon related works toward classifying Twitter tweets representing symptoms of major depressive disorder by assessing the contribution of lexical features (e.g., unigrams) and emotion (e.g., strongly negative) to classification performance, and by applying methods to eliminate low-value features. METHODS Specifically, we conducted a feature ablation study to assess the informativeness of each feature group and a feature elimination study to determine the optimal feature sets for classifying Twitter tweets. We leveraged an existing, annotated Twitter dataset that was constructed based on a hierarchical model of depression-related symptoms BIBREF12 , BIBREF13 . The dataset contains 9,473 annotations for 9,300 tweets. Each tweet is annotated as no evidence of depression (e.g., “Citizens fear an economic depression") or evidence of depression (e.g., “depressed over disappointment"). If a tweet is annotated evidence of depression, then it is further annotated with one or more depressive symptoms, for example, depressed mood (e.g., “feeling down in the dumps"), disturbed sleep (e.g., “another restless night"), or fatigue or loss of energy (e.g., “the fatigue is unbearable") BIBREF10 . For each class, every annotation (9,473 tweets) is binarized as the positive class e.g., depressed mood=1 or negative class e.g., not depressed mood=0. Features Furthermore, this dataset was encoded with 7 feature groups with associated feature values binarized (i.e., present=1 or absent=0) to represent potentially informative features for classifying depression-related classes. We describe the feature groups by type, subtype, and provide one or more examples of words representing the feature subtype from a tweet: lexical features, unigrams, e.g., “depressed”; syntactic features, parts of speech, e.g., “cried” encoded as V for verb; emotion features, emoticons, e.g., :( encoded as SAD; demographic features, age and gender e.g., “this semester” encoded as an indicator of 19-22 years of age and “my girlfriend” encoded as an indicator of male gender, respectively; sentiment features, polarity and subjectivity terms with strengths, e.g., “terrible” encoded as strongly negative and strongly subjective; personality traits, neuroticism e.g., “pissed off” implies neuroticism; LIWC Features, indicators of an individual's thoughts, feelings, personality, and motivations, e.g., “feeling” suggestions perception, feeling, insight, and cognitive mechanisms experienced by the Twitter user. A more detailed description of leveraged features and their values, including LIWC categories, can be found in BIBREF10 . Based on our prior initial experiments using these feature groups BIBREF10 , we learned that support vector machines perform with the highest F1-score compared to other supervised approaches. For this study, we aim to build upon this work by conducting two experiments: 1) to assess the contribution of each feature group and 2) to determine the optimal percentile of top ranked features for classifying Twitter tweets in the depression schema hierarchy. Feature Contribution Feature ablation studies are conducted to assess the informativeness of a feature group by quantifying the change in predictive power when comparing the performance of a classifier trained with the all feature groups versus the performance without a particular feature group. We conducted a feature ablation study by holding out (sans) each feature group and training and testing the support vector model using a linear kernel and 5-fold, stratified cross-validation. We report the average F1-score from our baseline approach (all feature groups) and report the point difference (+ or -) in F1-score performance observed by ablating each feature set. By ablating each feature group from the full dataset, we observed the following count of features - sans lexical: 185, sans syntactic: 16,935, sans emotion: 16,954, sans demographics: 16,946, sans sentiment: 16,950, sans personality: 16,946, and sans LIWC: 16,832. In Figure 1, compared to the baseline performance, significant drops in F1-scores resulted from sans lexical for depressed mood (-35 points), disturbed sleep (-43 points), and depressive symptoms (-45 points). Less extensive drops also occurred for evidence of depression (-14 points) and fatigue or loss of energy (-3 points). In contrast, a 3 point gain in F1-score was observed for no evidence of depression. We also observed notable drops in F1-scores for disturbed sleep by ablating demographics (-7 points), emotion (-5 points), and sentiment (-5 points) features. These F1-score drops were accompanied by drops in both recall and precision. We found equal or higher F1-scores by removing non-lexical feature groups for no evidence of depression (0-1 points), evidence of depression (0-1 points), and depressive symptoms (2 points). Unsurprisingly, lexical features (unigrams) were the largest contributor to feature counts in the dataset. We observed that lexical features are also critical for identifying depressive symptoms, specifically for depressed mood and for disturbed sleep. For the classes higher in the hierarchy - no evidence of depression, evidence of depression, and depressive symptoms - the classifier produced consistent F1-scores, even slightly above the baseline for depressive symptoms and minor fluctuations of change in recall and precision when removing other feature groups suggesting that the contribution of non-lexical features to classification performance was limited. However, notable changes in F1-score were observed for the classes lower in the hierarchy including disturbed sleep and fatigue or loss of energy. For instance, changes in F1-scores driven by both recall and precision were observed for disturbed sleep by ablating demographics, emotion, and sentiment features, suggesting that age or gender (“mid-semester exams have me restless”), polarity and subjective terms (“lack of sleep is killing me”), and emoticons (“wide awake :(”) could be important for both identifying and correctly classifying a subset of these tweets. Feature Elimination Feature elimination strategies are often taken 1) to remove irrelevant or noisy features, 2) to improve classifier performance, and 3) to reduce training and run times. We conducted an experiment to determine whether we could maintain or improve classifier performances by applying the following three-tiered feature elimination approach: Reduction We reduced the dataset encoded for each class by eliminating features that occur less than twice in the full dataset. Selection We iteratively applied Chi-Square feature selection on the reduced dataset, selecting the top percentile of highest ranked features in increments of 5 percent to train and test the support vector model using a linear kernel and 5-fold, stratified cross-validation. Rank We cumulatively plotted the average F1-score performances of each incrementally added percentile of top ranked features. We report the percentile and count of features resulting in the first occurrence of the highest average F1-score for each class. All experiments were programmed using scikit-learn 0.18. The initial matrices of almost 17,000 features were reduced by eliminating features that only occurred once in the full dataset, resulting in 5,761 features. We applied Chi-Square feature selection and plotted the top-ranked subset of features for each percentile (at 5 percent intervals cumulatively added) and evaluated their predictive contribution using the support vector machine with linear kernel and stratified, 5-fold cross validation. In Figure 2, we observed optimal F1-score performance using the following top feature counts: no evidence of depression: F1: 87 (15th percentile, 864 features), evidence of depression: F1: 59 (30th percentile, 1,728 features), depressive symptoms: F1: 55 (15th percentile, 864 features), depressed mood: F1: 39 (55th percentile, 3,168 features), disturbed sleep: F1: 46 (10th percentile, 576 features), and fatigue or loss of energy: F1: 72 (5th percentile, 288 features) (Figure 1). We note F1-score improvements for depressed mood from F1: 13 at the 1st percentile to F1: 33 at the 20th percentile. We observed peak F1-score performances at low percentiles for fatigue or loss of energy (5th percentile), disturbed sleep (10th percentile) as well as depressive symptoms and no evidence of depression (both 15th percentile) suggesting fewer features are needed to reach optimal performance. In contrast, peak F1-score performances occurred at moderate percentiles for evidence of depression (30th percentile) and depressed mood (55th percentile) suggesting that more features are needed to reach optimal performance. However, one notable difference between these two classes is the dramatic F1-score improvements for depressed mood i.e., 20 point increase from the 1st percentile to the 20th percentile compared to the more gradual F1-score improvements for evidence of depression i.e., 11 point increase from the 1st percentile to the 20th percentile. This finding suggests that for identifying depressed mood a variety of features are needed before incremental gains are observed. RESULTS From our annotated dataset of Twitter tweets (n=9,300 tweets), we conducted two feature studies to better understand the predictive power of several feature groups for classifying whether or not a tweet contains no evidence of depression (n=6,829 tweets) or evidence of depression (n=2,644 tweets). If there was evidence of depression, we determined whether the tweet contained one or more depressive symptoms (n=1,656 tweets) and further classified the symptom subtype of depressed mood (n=1,010 tweets), disturbed sleep (n=98 tweets), or fatigue or loss of energy (n=427 tweets) using support vector machines. From our prior work BIBREF10 and in Figure 1, we report the performance for prediction models built by training a support vector machine using 5-fold, stratified cross-validation with all feature groups as a baseline for each class. We observed high performance for no evidence of depression and fatigue or loss of energy and moderate performance for all remaining classes. Discussion We conducted two feature study experiments: 1) a feature ablation study to assess the contribution of feature groups and 2) a feature elimination study to determine the optimal percentile of top ranked features for classifying Twitter tweets in the depression schema hierarchy. Future Work Our next step is to address the classification of rarer depressive symptoms suggestive of major depressive disorder from our dataset and hierarchy including inappropriate guilt, difficulty concentrating, psychomotor agitation or retardation, weight loss or gain, and anhedonia BIBREF15 , BIBREF16 . We are developing a population-level monitoring framework designed to estimate the prevalence of depression (and depression-related symptoms and psycho-social stressors) over millions of United States-geocoded tweets. Identifying the most discriminating feature sets and natural language processing classifiers for each depression symptom is vital for this goal. Conclusions In summary, we conducted two feature study experiments to assess the contribution of feature groups and to determine the optimal percentile of top ranked features for classifying Twitter tweets in the depression schema hierarchy. From these experiments, we conclude that simple lexical features and reduced feature sets can produce comparable results to the much larger feature dataset. Acknowledgments Research reported in this publication was supported by the National Library of Medicine of the [United States] National Institutes of Health under award numbers K99LM011393 and R00LM011393. This study was granted an exemption from review by the University of Utah Institutional Review Board (IRB 00076188). Note that in order to protect tweeter anonymity, we have not reproduced tweets verbatim. Example tweets shown were generated by the researchers as exemplars only. Finally, we would like to thank the anonymous reviewers of this paper for their valuable comments.
What dataset is used for this study?
5a81732d52f64e81f1f83e8fd3514251227efbc7
What Ġdataset Ġis Ġused Ġfor Ġthis Ġstudy ?
BI B REF 12 Ġ, ĠB IB REF 13
<s> What Ġdataset Ġis Ġused Ġfor Ġthis Ġstudy ? </s> Introduction </s> In Ġrecent Ġyears , Ġthere Ġhas Ġbeen Ġa Ġmovement Ġto Ġleverage Ġsocial Ġmedial Ġdata Ġto Ġdetect , Ġestimate , Ġand Ġtrack Ġthe Ġchange Ġin Ġprevalence Ġof Ġdisease . ĠFor Ġexample , Ġeating Ġdisorders Ġin ĠSpanish Ġlanguage ĠTwitter Ġtweets ĠB IB REF 0 Ġand Ġinfluenza Ġsurveillance ĠB IB REF 1 Ġ. ĠMore Ġrecently , Ġsocial Ġmedia Ġhas Ġbeen Ġlever aged Ġto Ġmonitor Ġsocial Ġrisks Ġsuch Ġas Ġprescription Ġdrug Ġand Ġsmoking Ġbehaviors ĠB IB REF 2 Ġ, ĠB IB REF 3 Ġ, ĠB IB REF 4 Ġas Ġwell Ġas Ġa Ġvariety Ġof Ġmental Ġhealth Ġdisorders Ġincluding Ġsuicidal Ġide ation ĠB IB REF 5 Ġ, Ġattention Ġdeficient Ġhyper activity Ġdisorder ĠB IB REF 6 Ġand Ġmajor Ġdepressive Ġdisorder ĠB IB REF 7 Ġ. ĠIn Ġthe Ġcase Ġof Ġmajor Ġdepressive Ġdisorder , Ġrecent Ġefforts Ġrange Ġfrom Ġcharacter izing Ġlinguistic Ġphenomena Ġassociated Ġwith Ġdepression ĠB IB REF 8 Ġand Ġits Ġsub types Ġe . g ., Ġpost part um Ġdepression ĠB IB REF 5 Ġ, Ġto Ġidentifying Ġspecific Ġdepressive Ġsymptoms ĠB IB REF 9 Ġ, ĠB IB REF 10 Ġe . g ., Ġdepressed Ġmood . ĠHowever , Ġmore Ġresearch Ġis Ġneeded Ġto Ġbetter Ġunderstand Ġthe Ġpredictive Ġpower Ġof Ġsupervised Ġmachine Ġlearning Ġclass ifiers Ġand Ġthe Ġinfluence Ġof Ġfeature Ġgroups Ġand Ġfeature Ġsets Ġfor Ġefficiently Ġclass ifying Ġdepression - related Ġtweets Ġto Ġsupport Ġmental Ġhealth Ġmonitoring Ġat Ġthe Ġpopulation - level ĠB IB REF 11 Ġ. </s> This Ġpaper Ġbuilds Ġupon Ġrelated Ġworks Ġtoward Ġclass ifying ĠTwitter Ġtweets Ġrepresenting Ġsymptoms Ġof Ġmajor Ġdepressive Ġdisorder Ġby Ġassessing Ġthe Ġcontribution Ġof Ġlex ical Ġfeatures Ġ( e . g ., Ġun ig rams ) Ġand Ġemotion Ġ( e . g ., Ġstrongly Ġnegative ) Ġto Ġclassification Ġperformance , Ġand Ġby Ġapplying Ġmethods Ġto Ġeliminate Ġlow - value Ġfeatures . </s> METHOD S </s> Specifically , Ġwe Ġconducted Ġa Ġfeature Ġab lation Ġstudy Ġto Ġassess Ġthe Ġinform at iveness Ġof Ġeach Ġfeature Ġgroup Ġand Ġa Ġfeature Ġelimination Ġstudy Ġto Ġdetermine Ġthe Ġoptimal Ġfeature Ġsets Ġfor Ġclass ifying ĠTwitter Ġtweets . ĠWe Ġlever aged Ġan Ġexisting , Ġannot ated ĠTwitter Ġdataset Ġthat Ġwas Ġconstructed Ġbased Ġon Ġa Ġhierarchical Ġmodel Ġof Ġdepression - related Ġsymptoms ĠB IB REF 12 Ġ, ĠB IB REF 13 Ġ. ĠThe Ġdataset Ġcontains Ġ9 , 473 Ġannotations Ġfor Ġ9 , 300 Ġtweets . ĠEach Ġtweet Ġis Ġannot ated Ġas Ġno Ġevidence Ġof Ġdepression Ġ( e . g ., ĠâĢ ľ C itizens Ġfear Ġan Ġeconomic Ġdepression ") Ġor Ġevidence Ġof Ġdepression Ġ( e . g ., ĠâĢ ľ dep ressed Ġover Ġdisappointment "). ĠIf Ġa Ġtweet Ġis Ġannot ated Ġevidence Ġof Ġdepression , Ġthen Ġit Ġis Ġfurther Ġannot ated Ġwith Ġone Ġor Ġmore Ġdepressive Ġsymptoms , Ġfor Ġexample , Ġdepressed Ġmood Ġ( e . g ., ĠâĢ ľ fe eling Ġdown Ġin Ġthe Ġdumps "), Ġdisturbed Ġsleep Ġ( e . g ., ĠâĢ ľ another Ġrestless Ġnight "), Ġor Ġfatigue Ġor Ġloss Ġof Ġenergy Ġ( e . g ., ĠâĢ ľ the Ġfatigue Ġis Ġunbearable ") ĠB IB REF 10 Ġ. ĠFor Ġeach Ġclass , Ġevery Ġannotation Ġ( 9 , 473 Ġtweets ) Ġis Ġbin ar ized Ġas Ġthe Ġpositive Ġclass Ġe . g ., Ġdepressed Ġmood = 1 Ġor Ġnegative Ġclass Ġe . g ., Ġnot Ġdepressed Ġmood = 0 . </s> Features </s> Furthermore , Ġthis Ġdataset Ġwas Ġencoded Ġwith Ġ7 Ġfeature Ġgroups Ġwith Ġassociated Ġfeature Ġvalues Ġbin ar ized Ġ( i . e ., Ġpresent = 1 Ġor Ġabsent = 0 ) Ġto Ġrepresent Ġpotentially Ġinformative Ġfeatures Ġfor Ġclass ifying Ġdepression - related Ġclasses . ĠWe Ġdescribe Ġthe Ġfeature Ġgroups Ġby Ġtype , Ġsub type , Ġand Ġprovide Ġone Ġor Ġmore Ġexamples Ġof Ġwords Ġrepresenting Ġthe Ġfeature Ġsub type Ġfrom Ġa Ġtweet : </s> lex ical Ġfeatures , Ġun ig rams , Ġe . g ., ĠâĢ ľ dep ressed âĢ Ŀ ; </s> sy nt actic Ġfeatures , Ġparts Ġof Ġspeech , Ġe . g ., ĠâĢ ľ c ried âĢ Ŀ Ġencoded Ġas ĠV Ġfor Ġverb ; </s> em otion Ġfeatures , Ġemot icons , Ġe . g ., Ġ:( Ġencoded Ġas ĠS AD ; </s> dem ographic Ġfeatures , Ġage Ġand Ġgender Ġe . g ., ĠâĢ ľ this Ġsemester âĢ Ŀ Ġencoded Ġas Ġan Ġindicator Ġof Ġ19 - 22 Ġyears Ġof Ġage Ġand ĠâĢ ľ my Ġgirlfriend âĢ Ŀ Ġencoded Ġas Ġan Ġindicator Ġof Ġmale Ġgender , Ġrespectively ; </s> sent iment Ġfeatures , Ġpol arity Ġand Ġsubject ivity Ġterms Ġwith Ġstrengths , Ġe . g ., ĠâĢ ľ ter rible âĢ Ŀ Ġencoded Ġas Ġstrongly Ġnegative Ġand Ġstrongly Ġsubjective ; </s> person ality Ġtraits , Ġneurot icism Ġe . g ., ĠâĢ ľ p iss ed Ġoff âĢ Ŀ Ġimplies Ġneurot icism ; </s> LI WC ĠFeatures , Ġindicators Ġof Ġan Ġindividual 's Ġthoughts , Ġfeelings , Ġpersonality , Ġand Ġmotivations , Ġe . g ., ĠâĢ ľ fe eling âĢ Ŀ Ġsuggestions Ġperception , Ġfeeling , Ġinsight , Ġand Ġcognitive Ġmechanisms Ġexperienced Ġby Ġthe ĠTwitter Ġuser . </s> A Ġmore Ġdetailed Ġdescription Ġof Ġlever aged Ġfeatures Ġand Ġtheir Ġvalues , Ġincluding ĠLI WC Ġcategories , Ġcan Ġbe Ġfound Ġin ĠB IB REF 10 Ġ. </s> Based Ġon Ġour Ġprior Ġinitial Ġexperiments Ġusing Ġthese Ġfeature Ġgroups ĠB IB REF 10 Ġ, Ġwe Ġlearned Ġthat Ġsupport Ġvector Ġmachines Ġperform Ġwith Ġthe Ġhighest ĠF 1 - score Ġcompared Ġto Ġother Ġsupervised Ġapproaches . ĠFor Ġthis Ġstudy , Ġwe Ġaim Ġto Ġbuild Ġupon Ġthis Ġwork Ġby Ġconducting Ġtwo Ġexperiments : Ġ1 ) Ġto Ġassess Ġthe Ġcontribution Ġof Ġeach Ġfeature Ġgroup Ġand Ġ2 ) Ġto Ġdetermine Ġthe Ġoptimal Ġpercentile Ġof Ġtop Ġranked Ġfeatures Ġfor Ġclass ifying ĠTwitter Ġtweets Ġin Ġthe Ġdepression Ġschema Ġhierarchy . </s> Feature ĠCont ribution </s> Feature Ġab lation Ġstudies Ġare Ġconducted Ġto Ġassess Ġthe Ġinform at iveness Ġof Ġa Ġfeature Ġgroup Ġby Ġquant ifying Ġthe Ġchange Ġin Ġpredictive Ġpower Ġwhen Ġcomparing Ġthe Ġperformance Ġof Ġa Ġclass ifier Ġtrained Ġwith Ġthe Ġall Ġfeature Ġgroups Ġversus Ġthe Ġperformance Ġwithout Ġa Ġparticular Ġfeature Ġgroup . ĠWe Ġconducted Ġa Ġfeature Ġab lation Ġstudy Ġby Ġholding Ġout Ġ( s ans ) Ġeach Ġfeature Ġgroup Ġand Ġtraining Ġand Ġtesting Ġthe Ġsupport Ġvector Ġmodel Ġusing Ġa Ġlinear Ġkernel Ġand Ġ5 - fold , Ġstrat ified Ġcross - valid ation . ĠWe Ġreport Ġthe Ġaverage ĠF 1 - score Ġfrom Ġour Ġbaseline Ġapproach Ġ( all Ġfeature Ġgroups ) Ġand Ġreport Ġthe Ġpoint Ġdifference Ġ(+ Ġor Ġ- ) Ġin ĠF 1 - score Ġperformance Ġobserved Ġby Ġabl ating Ġeach Ġfeature Ġset . </s> By Ġabl ating Ġeach Ġfeature Ġgroup Ġfrom Ġthe Ġfull Ġdataset , Ġwe Ġobserved Ġthe Ġfollowing Ġcount Ġof Ġfeatures Ġ- Ġsans Ġlex ical : Ġ185 , Ġsans Ġsynt actic : Ġ16 , 9 35 , Ġsans Ġemotion : Ġ16 , 954 , Ġsans Ġdemographics : Ġ16 , 9 46 , Ġsans Ġsentiment : Ġ16 , 950 , Ġsans Ġpersonality : Ġ16 , 9 46 , Ġand Ġsans ĠLI WC : Ġ16 , 8 32 . ĠIn ĠFigure Ġ1 , Ġcompared Ġto Ġthe Ġbaseline Ġperformance , Ġsignificant Ġdrops Ġin ĠF 1 - sc ores Ġresulted Ġfrom Ġsans Ġlex ical Ġfor Ġdepressed Ġmood Ġ(- 35 Ġpoints ), Ġdisturbed Ġsleep Ġ(- 43 Ġpoints ), Ġand Ġdepressive Ġsymptoms Ġ(- 45 Ġpoints ). ĠLess Ġextensive Ġdrops Ġalso Ġoccurred Ġfor Ġevidence Ġof Ġdepression Ġ(- 14 Ġpoints ) Ġand Ġfatigue Ġor Ġloss Ġof Ġenergy Ġ(- 3 Ġpoints ). ĠIn Ġcontrast , Ġa Ġ3 Ġpoint Ġgain Ġin ĠF 1 - score Ġwas Ġobserved Ġfor Ġno Ġevidence Ġof Ġdepression . ĠWe Ġalso Ġobserved Ġnotable Ġdrops Ġin ĠF 1 - sc ores Ġfor Ġdisturbed Ġsleep Ġby Ġabl ating Ġdemographics Ġ(- 7 Ġpoints ), Ġemotion Ġ(- 5 Ġpoints ), Ġand Ġsentiment Ġ(- 5 Ġpoints ) Ġfeatures . ĠThese ĠF 1 - score Ġdrops Ġwere Ġaccompanied Ġby Ġdrops Ġin Ġboth Ġrecall Ġand Ġprecision . ĠWe Ġfound Ġequal Ġor Ġhigher ĠF 1 - sc ores Ġby Ġremoving Ġnon - lex ical Ġfeature Ġgroups Ġfor Ġno Ġevidence Ġof Ġdepression Ġ( 0 - 1 Ġpoints ), Ġevidence Ġof Ġdepression Ġ( 0 - 1 Ġpoints ), Ġand Ġdepressive Ġsymptoms Ġ( 2 Ġpoints ). </s> Un surprisingly , Ġlex ical Ġfeatures Ġ( un ig rams ) Ġwere Ġthe Ġlargest Ġcontributor Ġto Ġfeature Ġcounts Ġin Ġthe Ġdataset . ĠWe Ġobserved Ġthat Ġlex ical Ġfeatures Ġare Ġalso Ġcritical Ġfor Ġidentifying Ġdepressive Ġsymptoms , Ġspecifically Ġfor Ġdepressed Ġmood Ġand Ġfor Ġdisturbed Ġsleep . ĠFor Ġthe Ġclasses Ġhigher Ġin Ġthe Ġhierarchy Ġ- Ġno Ġevidence Ġof Ġdepression , Ġevidence Ġof Ġdepression , Ġand Ġdepressive Ġsymptoms Ġ- Ġthe Ġclass ifier Ġproduced Ġconsistent ĠF 1 - sc ores , Ġeven Ġslightly Ġabove Ġthe Ġbaseline Ġfor Ġdepressive Ġsymptoms Ġand Ġminor Ġfluctuations Ġof Ġchange Ġin Ġrecall Ġand Ġprecision Ġwhen Ġremoving Ġother Ġfeature Ġgroups Ġsuggesting Ġthat Ġthe Ġcontribution Ġof Ġnon - lex ical Ġfeatures Ġto Ġclassification Ġperformance Ġwas Ġlimited . ĠHowever , Ġnotable Ġchanges Ġin ĠF 1 - score Ġwere Ġobserved Ġfor Ġthe Ġclasses Ġlower Ġin Ġthe Ġhierarchy Ġincluding Ġdisturbed Ġsleep Ġand Ġfatigue Ġor Ġloss Ġof Ġenergy . ĠFor Ġinstance , Ġchanges Ġin ĠF 1 - sc ores Ġdriven Ġby Ġboth Ġrecall Ġand Ġprecision Ġwere Ġobserved Ġfor Ġdisturbed Ġsleep Ġby Ġabl ating Ġdemographics , Ġemotion , Ġand Ġsentiment Ġfeatures , Ġsuggesting Ġthat Ġage Ġor Ġgender Ġ( âĢ ľ mid - se mes ter Ġexams Ġhave Ġme Ġrestless âĢ Ŀ ), Ġpol arity Ġand Ġsubjective Ġterms Ġ( âĢ ľ l ack Ġof Ġsleep Ġis Ġkilling Ġme âĢ Ŀ ), Ġand Ġemot icons Ġ( âĢ ľ wide Ġawake Ġ:( âĢ Ŀ ) Ġcould Ġbe Ġimportant Ġfor Ġboth Ġidentifying Ġand Ġcorrectly Ġclass ifying Ġa Ġsubset Ġof Ġthese Ġtweets . </s> Feature ĠElim ination </s> Feature Ġelimination Ġstrategies Ġare Ġoften Ġtaken Ġ1 ) Ġto Ġremove Ġirrelevant Ġor Ġnoisy Ġfeatures , Ġ2 ) Ġto Ġimprove Ġclass ifier Ġperformance , Ġand Ġ3 ) Ġto Ġreduce Ġtraining Ġand Ġrun Ġtimes . ĠWe Ġconducted Ġan Ġexperiment Ġto Ġdetermine Ġwhether Ġwe Ġcould Ġmaintain Ġor Ġimprove Ġclass ifier Ġperformances Ġby Ġapplying Ġthe Ġfollowing Ġthree - tie red Ġfeature Ġelimination Ġapproach : </s> Red uction ĠWe Ġreduced Ġthe Ġdataset Ġencoded Ġfor Ġeach Ġclass Ġby Ġeliminating Ġfeatures Ġthat Ġoccur Ġless Ġthan Ġtwice Ġin Ġthe Ġfull Ġdataset . </s> Se lection ĠWe Ġiter atively Ġapplied ĠChi - Square Ġfeature Ġselection Ġon Ġthe Ġreduced Ġdataset , Ġselecting Ġthe Ġtop Ġpercentile Ġof Ġhighest Ġranked Ġfeatures Ġin Ġincrements Ġof Ġ5 Ġpercent Ġto Ġtrain Ġand Ġtest Ġthe Ġsupport Ġvector Ġmodel Ġusing Ġa Ġlinear Ġkernel Ġand Ġ5 - fold , Ġstrat ified Ġcross - valid ation . </s> Rank ĠWe Ġcum ul atively Ġplotted Ġthe Ġaverage ĠF 1 - score Ġperformances Ġof Ġeach Ġincrement ally Ġadded Ġpercentile Ġof Ġtop Ġranked Ġfeatures . ĠWe Ġreport Ġthe Ġpercentile Ġand Ġcount Ġof Ġfeatures Ġresulting Ġin Ġthe Ġfirst Ġoccurrence Ġof Ġthe Ġhighest Ġaverage ĠF 1 - score Ġfor Ġeach Ġclass . </s> All Ġexperiments Ġwere Ġprogrammed Ġusing Ġsc ik it - learn Ġ0 . 18 . </s> The Ġinitial Ġmat rices Ġof Ġalmost Ġ17 , 000 Ġfeatures Ġwere Ġreduced Ġby Ġeliminating Ġfeatures Ġthat Ġonly Ġoccurred Ġonce Ġin Ġthe Ġfull Ġdataset , Ġresulting Ġin Ġ5 , 76 1 Ġfeatures . ĠWe Ġapplied ĠChi - Square Ġfeature Ġselection Ġand Ġplotted Ġthe Ġtop - ranked Ġsubset Ġof Ġfeatures Ġfor Ġeach Ġpercentile Ġ( at Ġ5 Ġpercent Ġintervals Ġcum ul atively Ġadded ) Ġand Ġevaluated Ġtheir Ġpredictive Ġcontribution Ġusing Ġthe Ġsupport Ġvector Ġmachine Ġwith Ġlinear Ġkernel Ġand Ġstrat ified , Ġ5 - fold Ġcross Ġvalidation . </s> In ĠFigure Ġ2 , Ġwe Ġobserved Ġoptimal ĠF 1 - score Ġperformance Ġusing Ġthe Ġfollowing Ġtop Ġfeature Ġcounts : Ġno Ġevidence Ġof Ġdepression : ĠF 1 : Ġ87 Ġ( 15 th Ġpercentile , Ġ8 64 Ġfeatures ), Ġevidence Ġof Ġdepression : ĠF 1 : Ġ59 Ġ( 30 th Ġpercentile , Ġ1 , 728 Ġfeatures ), Ġdepressive Ġsymptoms : ĠF 1 : Ġ55 Ġ( 15 th Ġpercentile , Ġ8 64 Ġfeatures ), Ġdepressed Ġmood : ĠF 1 : Ġ39 Ġ( 55 th Ġpercentile , Ġ3 , 168 Ġfeatures ), Ġdisturbed Ġsleep : ĠF 1 : Ġ46 Ġ( 10 th Ġpercentile , Ġ5 76 Ġfeatures ), Ġand Ġfatigue Ġor Ġloss Ġof Ġenergy : ĠF 1 : Ġ72 Ġ( 5 th Ġpercentile , Ġ288 Ġfeatures ) Ġ( Figure Ġ1 ). ĠWe Ġnote ĠF 1 - score Ġimprovements Ġfor Ġdepressed Ġmood Ġfrom ĠF 1 : Ġ13 Ġat Ġthe Ġ1 st Ġpercentile Ġto ĠF 1 : Ġ33 Ġat Ġthe Ġ20 th Ġpercentile . </s> We Ġobserved Ġpeak ĠF 1 - score Ġperformances Ġat Ġlow Ġpercent iles Ġfor Ġfatigue Ġor Ġloss Ġof Ġenergy Ġ( 5 th Ġpercentile ), Ġdisturbed Ġsleep Ġ( 10 th Ġpercentile ) Ġas Ġwell Ġas Ġdepressive Ġsymptoms Ġand Ġno Ġevidence Ġof Ġdepression Ġ( both Ġ15 th Ġpercentile ) Ġsuggesting Ġfewer Ġfeatures Ġare Ġneeded Ġto Ġreach Ġoptimal Ġperformance . ĠIn Ġcontrast , Ġpeak ĠF 1 - score Ġperformances Ġoccurred Ġat Ġmoderate Ġpercent iles Ġfor Ġevidence Ġof Ġdepression Ġ( 30 th Ġpercentile ) Ġand Ġdepressed Ġmood Ġ( 55 th Ġpercentile ) Ġsuggesting Ġthat Ġmore Ġfeatures Ġare Ġneeded Ġto Ġreach Ġoptimal Ġperformance . ĠHowever , Ġone Ġnotable Ġdifference Ġbetween Ġthese Ġtwo Ġclasses Ġis Ġthe Ġdramatic ĠF 1 - score Ġimprovements Ġfor Ġdepressed Ġmood Ġi . e ., Ġ20 Ġpoint Ġincrease Ġfrom Ġthe Ġ1 st Ġpercentile Ġto Ġthe Ġ20 th Ġpercentile Ġcompared Ġto Ġthe Ġmore Ġgradual ĠF 1 - score Ġimprovements Ġfor Ġevidence Ġof Ġdepression Ġi . e ., Ġ11 Ġpoint Ġincrease Ġfrom Ġthe Ġ1 st Ġpercentile Ġto Ġthe Ġ20 th Ġpercentile . ĠThis Ġfinding Ġsuggests Ġthat Ġfor Ġidentifying Ġdepressed Ġmood Ġa Ġvariety Ġof Ġfeatures Ġare Ġneeded Ġbefore Ġincremental Ġgains Ġare Ġobserved . </s> RESULTS </s> From Ġour Ġannot ated Ġdataset Ġof ĠTwitter Ġtweets Ġ( n = 9 , 300 Ġtweets ), Ġwe Ġconducted Ġtwo Ġfeature Ġstudies Ġto Ġbetter Ġunderstand Ġthe Ġpredictive Ġpower Ġof Ġseveral Ġfeature Ġgroups Ġfor Ġclass ifying Ġwhether Ġor Ġnot Ġa Ġtweet Ġcontains Ġno Ġevidence Ġof Ġdepression Ġ( n = 6 , 8 29 Ġtweets ) Ġor Ġevidence Ġof Ġdepression Ġ( n = 2 , 644 Ġtweets ). ĠIf Ġthere Ġwas Ġevidence Ġof Ġdepression , Ġwe Ġdetermined Ġwhether Ġthe Ġtweet Ġcontained Ġone Ġor Ġmore Ġdepressive Ġsymptoms Ġ( n = 1 , 656 Ġtweets ) Ġand Ġfurther Ġclassified Ġthe Ġsymptom Ġsub type Ġof Ġdepressed Ġmood Ġ( n = 1 , 010 Ġtweets ), Ġdisturbed Ġsleep Ġ( n = 98 Ġtweets ), Ġor Ġfatigue Ġor Ġloss Ġof Ġenergy Ġ( n = 427 Ġtweets ) Ġusing Ġsupport Ġvector Ġmachines . ĠFrom Ġour Ġprior Ġwork ĠB IB REF 10 Ġand Ġin ĠFigure Ġ1 , Ġwe Ġreport Ġthe Ġperformance Ġfor Ġprediction Ġmodels Ġbuilt Ġby Ġtraining Ġa Ġsupport Ġvector Ġmachine Ġusing Ġ5 - fold , Ġstrat ified Ġcross - valid ation Ġwith Ġall Ġfeature Ġgroups Ġas Ġa Ġbaseline Ġfor Ġeach Ġclass . ĠWe Ġobserved Ġhigh Ġperformance Ġfor Ġno Ġevidence Ġof Ġdepression Ġand Ġfatigue Ġor Ġloss Ġof Ġenergy Ġand Ġmoderate Ġperformance Ġfor Ġall Ġremaining Ġclasses . </s> Discussion </s> We Ġconducted Ġtwo Ġfeature Ġstudy Ġexperiments : Ġ1 ) Ġa Ġfeature Ġab lation Ġstudy Ġto Ġassess Ġthe Ġcontribution Ġof Ġfeature Ġgroups Ġand Ġ2 ) Ġa Ġfeature Ġelimination Ġstudy Ġto Ġdetermine Ġthe Ġoptimal Ġpercentile Ġof Ġtop Ġranked Ġfeatures Ġfor Ġclass ifying ĠTwitter Ġtweets Ġin Ġthe Ġdepression Ġschema Ġhierarchy . </s> Future ĠWork </s> Our Ġnext Ġstep Ġis Ġto Ġaddress Ġthe Ġclassification Ġof Ġrare r Ġdepressive Ġsymptoms Ġsuggestive Ġof Ġmajor Ġdepressive Ġdisorder Ġfrom Ġour Ġdataset Ġand Ġhierarchy Ġincluding Ġinappropriate Ġguilt , Ġdifficulty Ġconcentrating , Ġpsych om otor Ġagitation Ġor Ġretard ation , Ġweight Ġloss Ġor Ġgain , Ġand Ġan hed onia ĠB IB REF 15 Ġ, ĠB IB REF 16 Ġ. ĠWe Ġare Ġdeveloping Ġa Ġpopulation - level Ġmonitoring Ġframework Ġdesigned Ġto Ġestimate Ġthe Ġprevalence Ġof Ġdepression Ġ( and Ġdepression - related Ġsymptoms Ġand Ġpsycho - social Ġstress ors ) Ġover Ġmillions Ġof ĠUnited ĠStates - ge oc oded Ġtweets . ĠIdent ifying Ġthe Ġmost Ġdiscriminating Ġfeature Ġsets Ġand Ġnatural Ġlanguage Ġprocessing Ġclass ifiers Ġfor Ġeach Ġdepression Ġsymptom Ġis Ġvital Ġfor Ġthis Ġgoal . </s> Con clusions </s> In Ġsummary , Ġwe Ġconducted Ġtwo Ġfeature Ġstudy Ġexperiments Ġto Ġassess Ġthe Ġcontribution Ġof Ġfeature Ġgroups Ġand Ġto Ġdetermine Ġthe Ġoptimal Ġpercentile Ġof Ġtop Ġranked Ġfeatures Ġfor Ġclass ifying ĠTwitter Ġtweets Ġin Ġthe Ġdepression Ġschema Ġhierarchy . ĠFrom Ġthese Ġexperiments , Ġwe Ġconclude Ġthat Ġsimple Ġlex ical Ġfeatures Ġand Ġreduced Ġfeature Ġsets Ġcan Ġproduce Ġcomparable Ġresults Ġto Ġthe Ġmuch Ġlarger Ġfeature Ġdataset . </s> Acknowled gments </s> Research Ġreported Ġin Ġthis Ġpublication Ġwas Ġsupported Ġby Ġthe ĠNational ĠLibrary Ġof ĠMedicine Ġof Ġthe Ġ[ United ĠStates ] ĠNational ĠInstitutes Ġof ĠHealth Ġunder Ġaward Ġnumbers ĠK 99 LM 01 13 93 Ġand ĠR 00 LM 01 13 93 . ĠThis Ġstudy Ġwas Ġgranted Ġan Ġexemption Ġfrom Ġreview Ġby Ġthe ĠUniversity Ġof ĠUtah ĠInst itutional ĠReview ĠBoard Ġ( IR B Ġ000 76 188 ). ĠNote Ġthat Ġin Ġorder Ġto Ġprotect Ġtwe eter Ġanonymity , Ġwe Ġhave Ġnot Ġreproduced Ġtweets Ġver bat im . ĠExample Ġtweets Ġshown Ġwere Ġgenerated Ġby Ġthe Ġresearchers Ġas Ġexempl ars Ġonly . ĠFinally , Ġwe Ġwould Ġlike Ġto Ġthank Ġthe Ġanonymous Ġreviewers Ġof Ġthis Ġpaper Ġfor Ġtheir Ġvaluable Ġcomments .