Reuters-8 dataset without stop words
EXTRACTIVE
The Reuters-8 dataset (with stop words removed)
ABSTRACTIVE
In this section we describe the experiments performed to demonstrate the validity of our proposed method and its extension. We used the Reuters-8 dataset without stop words from BIBREF27 aiming at single-label classification, which is a preprocessed format of the Reuters-21578. Words in the texts were considered as they appeared, without performing stemming or typo correction. This database has eight different classes with the number of samples varying from 51 to over 3000 documents, as can be seen in Table 1 .
In this section we describe the experiments performed to demonstrate the validity of our proposed method and its extension. We used the Reuters-8 dataset without stop words from BIBREF27 aiming at single-label classification, which is a preprocessed format of the Reuters-21578. Words in the texts were considered as they appeared, without performing stemming or typo correction. This database has eight different classes with the number of samples varying from 51 to over 3000 documents, as can be seen in Table 1 .
1806.03125
Introduction </s> Text Ġclassification Ġhas Ġbecome Ġan Ġindispensable Ġtask Ġdue Ġto Ġthe Ġrapid Ġgrowth Ġin Ġthe Ġnumber Ġof Ġtexts Ġin Ġdigital Ġform Ġavailable Ġonline . ĠIt Ġaims Ġto Ġclassify Ġdifferent Ġtexts , Ġalso Ġcalled Ġdocuments , Ġinto Ġa Ġfixed Ġnumber Ġof Ġpred efined Ġcategories , Ġhelping Ġto Ġorganize Ġdata , Ġand Ġmaking Ġeasier Ġfor Ġusers Ġto Ġfind Ġthe Ġdesired Ġinformation . ĠOver Ġthe Ġpast Ġthree Ġdecades , Ġmany Ġmethods Ġbased Ġon Ġmachine Ġlearning Ġand Ġstatistical Ġmodels Ġhave Ġbeen Ġapplied Ġto Ġperform Ġthis Ġtask , Ġsuch Ġas Ġlatent Ġsemantic Ġanalysis Ġ( L SA ), Ġsupport Ġvector Ġmachines Ġ( S VM ), Ġand Ġmult in omial Ġnaive ĠBay es Ġ( M NB ). </s> The Ġfirst Ġstep Ġin Ġutilizing Ġsuch Ġmethods Ġto Ġcategor ize Ġtextual Ġdata Ġis Ġto Ġconvert Ġthe Ġtexts Ġinto Ġa Ġvector Ġrepresentation . ĠOne Ġof Ġthe Ġmost Ġpopular Ġtext Ġrepresentation Ġmodels Ġis Ġthe Ġbag - of - words Ġmodel ĠB IB REF 0 Ġ, Ġwhich Ġrepresents Ġeach Ġdocument Ġin Ġa Ġcollection Ġas Ġa Ġvector Ġin Ġa Ġvector Ġspace . ĠEach Ġdimension Ġof Ġthe Ġvectors Ġrepresents Ġa Ġterm Ġ( e . g ., Ġa Ġword , Ġa Ġsequence Ġof Ġwords ), Ġand Ġits Ġvalue Ġenc odes Ġa Ġweight , Ġwhich Ġcan Ġbe Ġhow Ġmany Ġtimes Ġthe Ġterm Ġoccurs Ġin Ġthe Ġdocument . </s> Despite Ġshowing Ġpositive Ġresults Ġin Ġtasks Ġsuch Ġas Ġlanguage Ġmodeling Ġand Ġclassification ĠB IB REF 1 Ġ, ĠB IB REF 2 Ġ, ĠB IB REF 3 Ġ, Ġthe ĠB OW Ġrepresentation Ġhas Ġlimitations : Ġfirst , Ġfeature Ġvectors Ġare Ġcommonly Ġvery Ġhigh - dimensional , Ġresulting Ġin Ġsparse Ġdocument Ġrepresentations , Ġwhich Ġare Ġhard Ġto Ġmodel Ġdue Ġto Ġspace Ġand Ġtime Ġcomplexity . ĠSecond , ĠB OW Ġdoes Ġnot Ġconsider Ġthe Ġproximity Ġof Ġwords Ġand Ġtheir Ġposition Ġin Ġthe Ġtext Ġand Ġconsequently Ġcannot Ġencode Ġthe Ġwords Ġsemantic Ġmeanings . </s> To Ġsolve Ġthese Ġproblems , Ġneural Ġnetworks Ġhave Ġbeen Ġemployed Ġto Ġlearn Ġvector Ġrepresentations Ġof Ġwords ĠB IB REF 4 Ġ, ĠB IB REF 5 Ġ, ĠB IB REF 6 Ġ, ĠB IB REF 7 Ġ. ĠIn Ġparticular , Ġthe Ġword 2 vec Ġrepresentation ĠB IB REF 8 Ġhas Ġgained Ġattention . ĠGiven Ġa Ġtraining Ġcorpus , Ġword 2 vec Ġcan Ġgenerate Ġa Ġvector Ġfor Ġeach Ġword Ġin Ġthe Ġcorpus Ġthat Ġenc odes Ġits Ġsemantic Ġinformation . ĠThese Ġword Ġvectors Ġare Ġdistributed Ġin Ġsuch Ġa Ġway Ġthat Ġwords Ġfrom Ġsimilar Ġcontexts Ġare Ġrepresented Ġby Ġword Ġvectors Ġwith Ġhigh Ġcorrelation , Ġwhile Ġwords Ġfrom Ġdifferent Ġcontexts Ġare Ġrepresented Ġby Ġword Ġvectors Ġwith Ġlow Ġcorrelation . </s> One Ġcrucial Ġaspect Ġof Ġthe Ġword 2 vec Ġrepresentation Ġis Ġthat Ġarithmetic Ġand Ġdistance Ġcalculation Ġbetween Ġtwo Ġword Ġvectors Ġcan Ġbe Ġperformed , Ġgiving Ġinformation Ġabout Ġtheir Ġsemantic Ġrelationship . ĠHowever , Ġrather Ġthan Ġlooking Ġat Ġpairs Ġof Ġword Ġvectors , Ġwe Ġare Ġinterested Ġin Ġstudying Ġthe Ġrelationship Ġbetween Ġsets Ġof Ġvectors Ġas Ġa Ġwhole Ġand , Ġtherefore , Ġit Ġis Ġdesirable Ġto Ġhave Ġa Ġtext Ġrepresentation Ġbased Ġon Ġa Ġset Ġof Ġthese Ġword Ġvectors . </s> To Ġtackle Ġthis Ġproblem , Ġwe Ġintroduce Ġthe Ġnovel Ġconcept Ġof Ġword Ġsub space . ĠIt Ġis Ġmat hematically Ġdefined Ġas Ġa Ġlow Ġdimensional Ġlinear Ġsub space Ġin Ġa Ġword Ġvector Ġspace Ġwith Ġhigh Ġdimension ality . ĠGiven Ġthat Ġwords Ġfrom Ġtexts Ġof Ġthe Ġsame Ġclass Ġbelong Ġto Ġthe Ġsame Ġcontext , Ġit Ġis Ġpossible Ġto Ġmodel Ġword Ġvectors Ġof Ġeach Ġclass Ġas Ġword Ġsub sp aces Ġand Ġefficiently Ġcompare Ġthem Ġin Ġterms Ġof Ġsimilarity Ġby Ġusing Ġcanonical Ġangles Ġbetween Ġthe Ġword Ġsub sp aces . ĠThrough Ġthis Ġrepresentation , Ġmost Ġof Ġthe Ġvariability Ġof Ġthe Ġclass Ġis Ġretained . ĠConsequently , Ġa Ġword Ġsub space Ġcan Ġeffectively Ġand Ġcompact ly Ġrepresent Ġthe Ġcontext Ġof Ġthe Ġcorresponding Ġtext . ĠWe Ġachieve Ġthis Ġframework Ġthrough Ġthe Ġmutual Ġsub space Ġmethod Ġ( MS M ) ĠB IB REF 9 Ġ. </s> The Ġword Ġsub space Ġof Ġeach Ġtext Ġclass Ġis Ġmodeled Ġby Ġapplying ĠPC A Ġwithout Ġdata Ġcent ering Ġto Ġthe Ġset Ġof Ġword Ġvectors Ġof Ġthe Ġclass . ĠWhen Ġmodeling Ġthe Ġword Ġsub sp aces , Ġwe Ġassume Ġonly Ġone Ġoccurrence Ġof Ġeach Ġword Ġinside Ġthe Ġclass . </s> However , Ġas Ġseen Ġin Ġthe ĠB OW Ġapproach , Ġthe Ġfrequency Ġof Ġwords Ġinside Ġa Ġtext Ġis Ġan Ġinformative Ġfeature Ġthat Ġshould Ġbe Ġconsidered . ĠIn Ġorder Ġto Ġintroduce Ġthis Ġfeature Ġin Ġthe Ġword Ġsub space Ġmodeling Ġand Ġenhance Ġits Ġperformance , Ġwe Ġfurther Ġextend Ġthe Ġconcept Ġof Ġword Ġsub space Ġto Ġthe Ġterm - frequency Ġ( TF ) Ġweighted Ġword Ġsub space . </s> In Ġthis Ġextension , Ġwe Ġconsider Ġa Ġset Ġof Ġweights , Ġwhich Ġenc odes Ġthe Ġwords Ġfrequencies , Ġwhen Ġperforming Ġthe ĠPC A . ĠText Ġclassification Ġwith ĠTF Ġweighted Ġword Ġsub space Ġcan Ġalso Ġbe Ġperformed Ġunder Ġthe Ġframework Ġof ĠMSM . ĠWe Ġshow Ġthe Ġvalidity Ġof Ġour Ġmodeling Ġthrough Ġexperiments Ġon Ġthe ĠReuters Ġdatabase , Ġan Ġestablished Ġdatabase Ġfor Ġnatural Ġlanguage Ġprocessing Ġtasks . ĠWe Ġdemonstrate Ġthe Ġeffectiveness Ġof Ġthe Ġword Ġsub space Ġformulation Ġand Ġits Ġextension , Ġcomparing Ġour Ġmethods ' Ġperformance Ġto Ġvarious Ġstate - of - art Ġmethods . </s> The Ġmain Ġcontributions Ġof Ġour Ġwork Ġare : </s> The Ġremainder Ġof Ġthis Ġpaper Ġis Ġorganized Ġas Ġfollows . ĠIn ĠSection Ġ" Related ĠWork " Ġ, Ġwe Ġdescribe Ġthe Ġmain Ġworks Ġrelated Ġto Ġtext Ġclassification . ĠIn ĠSection Ġ" Word Ġsub space " Ġ, Ġwe Ġpresent Ġthe Ġformulation Ġof Ġour Ġproposed Ġword Ġsub space . ĠIn ĠSection Ġ" Con ventional Ġtext Ġclassification Ġmethods " Ġ, Ġwe Ġexplain Ġhow Ġtext Ġclassification Ġwith Ġword Ġsub sp aces Ġis Ġperformed Ġunder Ġthe ĠMSM Ġframework . ĠThen , Ġwe Ġpresent Ġthe ĠTF Ġweighted Ġword Ġsub space Ġextension Ġin ĠSection Ġ" TF Ġweighted Ġword Ġsub space " Ġ. ĠEvaluation Ġexperiments Ġand Ġtheir Ġresults Ġare Ġdescribed Ġin ĠSection Ġ" Exper imental ĠEvaluation " Ġ. ĠFurther Ġdiscussion Ġis Ġthen Ġpresented Ġin ĠSection Ġ" Discussion " Ġ, Ġand Ġour Ġconclusions Ġare Ġdescribed Ġin ĠSection Ġ" Con clusions Ġand ĠFuture ĠWork " Ġ. </s> Related ĠWork </s> In Ġthis Ġsection , Ġwe Ġoutline Ġrelevant Ġwork Ġtowards Ġtext Ġclassification . ĠWe Ġstart Ġby Ġdescribing Ġhow Ġtext Ġdata Ġis Ġconvention ally Ġrepresented Ġusing Ġthe Ġbag - of - words Ġmodel Ġand Ġthen Ġfollow Ġto Ġdescribe Ġthe Ġconventional Ġmethods Ġutilized Ġin Ġtext Ġclassification . </s> Text ĠRepresent ation Ġwith Ġbag - of - words </s> The Ġbag - of - words Ġrepresentation Ġcomes Ġfrom Ġthe Ġhypothesis Ġthat Ġfrequencies Ġof Ġwords Ġin Ġa Ġdocument Ġcan Ġindicate Ġthe Ġrelevance Ġof Ġthe Ġdocument Ġto Ġa Ġquery ĠB IB REF 0 Ġ, Ġthat Ġis , Ġif Ġdocuments Ġand Ġa Ġquery Ġhave Ġsimilar Ġfrequencies Ġfor Ġthe Ġsame Ġwords , Ġthey Ġmight Ġhave Ġa Ġsimilar Ġmeaning . ĠThis Ġrepresentation Ġis Ġbased Ġon Ġthe Ġvector Ġspace Ġmodel Ġ( V SM ), Ġthat Ġwas Ġdeveloped Ġfor Ġthe ĠSM ART Ġinformation Ġretrieval Ġsystem ĠB IB REF 10 Ġ. ĠIn Ġthe ĠV SM , Ġthe Ġmain Ġidea Ġis Ġthat Ġdocuments Ġin Ġa Ġcollection Ġcan Ġbe Ġrepresented Ġas Ġa Ġvector Ġin Ġa Ġvector Ġspace , Ġwhere Ġvectors Ġclose Ġto Ġeach Ġother Ġrepresent Ġsem antically Ġsimilar Ġdocuments . </s> More Ġformally , Ġa Ġdocument Ġ$ d $ Ġcan Ġbe Ġrepresented Ġby Ġa Ġvector Ġin Ġ$\ math bb Ġ{ R } ^{ n } $ Ġ, Ġwhere Ġeach Ġdimension Ġrepresents Ġa Ġdifferent Ġterm . ĠA Ġterm Ġcan Ġbe Ġa Ġsingle Ġword , Ġconstit uting Ġthe Ġconventional Ġbag - of - words , Ġor Ġcombinations Ġof Ġ$ N $ Ġwords , Ġconstit uting Ġthe Ġbag - of - N - gram s . ĠIf Ġa Ġterm Ġoccurs Ġin Ġthe Ġdocument , Ġits Ġposition Ġin Ġthe Ġvector Ġwill Ġhave Ġa Ġnon - zero Ġvalue , Ġalso Ġknown Ġas Ġterm Ġweight . ĠTwo Ġdocuments Ġin Ġthe ĠV SM Ġcan Ġbe Ġcompared Ġto Ġeach Ġother Ġby Ġtaking Ġthe Ġcos ine Ġdistance Ġbetween Ġthem ĠB IB REF 1 Ġ. </s> There Ġare Ġseveral Ġways Ġto Ġcompute Ġthe Ġterm Ġweights . ĠAmong Ġthem , Ġwe Ġcan Ġhighlight Ġsome : ĠBinary Ġweights , Ġterm - frequency Ġ( TF ) Ġweights , Ġand Ġterm - frequency Ġinverse Ġdocument - frequency Ġ( TF - ID F ) Ġweights . </s> Consider Ġa Ġcorpus Ġwith Ġdocuments Ġ$ D Ġ= Ġ\ l brace Ġd _ i \ r brace Ġ_ { i = 1 } ^{ | D | } $ Ġand Ġa Ġvocabulary Ġwith Ġall Ġterms Ġin Ġthe Ġcorpus Ġ$ V Ġ= Ġ\ l brace Ġw _ i \ r brace Ġ_ { i = 1 } ^{ | V | } $ Ġ. ĠThe Ġterm Ġweights Ġcan Ġbe Ġdefined Ġas : </s> B inary Ġweight : ĠIf Ġa Ġterm Ġoccurs Ġin Ġthe Ġdocument , Ġits Ġweight Ġis Ġ1 . ĠOtherwise , Ġit Ġis Ġzero . </s> Term - frequency Ġweight Ġ( TF ): ĠThe Ġweight Ġof Ġa Ġterm Ġ$ w $ Ġis Ġdefined Ġby Ġthe Ġnumber Ġof Ġtimes Ġit Ġoccurs Ġin Ġthe Ġdocument Ġ$ d $ Ġ. </s> $$ TF ( w , d ) Ġ= Ġn _ d ^ w $$ Ġ Ġ Ġ( E q . Ġ8 ) </s> In verse Ġdocument - frequency : ĠThe Ġweight Ġof Ġa Ġterm Ġ$ w $ Ġ, Ġgiven Ġthe Ġcorpus Ġ$ D $ Ġ, Ġis Ġdefined Ġas Ġthe Ġtotal Ġnumber Ġof Ġdocuments Ġ$ | D | $ Ġdivided Ġby Ġthe Ġnumber Ġof Ġdocuments Ġthat Ġhave Ġthe Ġterm Ġ$ w $ Ġ, Ġ$ | D ^ w | $ Ġ. </s> $$ ID F ( w Ġ| ĠD ) Ġ= Ġ\ frac { | D | }{ | D ^ w | } $$ Ġ Ġ Ġ( E q . Ġ10 ) </s> Term - frequency Ġinverse Ġdocument - frequency Ġ( TF - ID F ): ĠThe Ġweight Ġof Ġa Ġterm Ġ$ w $ Ġis Ġdefined Ġby Ġthe Ġmultiplication Ġof Ġits Ġterm - frequency Ġand Ġits Ġinverse Ġdocument - frequency . ĠWhen Ġconsidering Ġonly Ġthe ĠTF Ġweights , Ġall Ġterms Ġhave Ġthe Ġsame Ġimportance Ġamong Ġthe Ġcorpus . ĠBy Ġusing Ġthe ĠIDF Ġweight , Ġwords Ġthat Ġare Ġmore Ġcommon Ġacross Ġall Ġdocuments Ġin Ġ$ D $ Ġreceive Ġa Ġsmaller Ġweight , Ġgiving Ġmore Ġimportance Ġto Ġrare Ġterms Ġin Ġthe Ġcorpus . </s> $$ TF ID F ( w , d Ġ| ĠD )= TF Ġ\ times ĠIDF $$ Ġ Ġ Ġ( E q . Ġ12 ) </s> In Ġvery Ġlarge Ġcorpus , Ġit Ġis Ġcommon Ġto Ġconsider Ġthe Ġlog ar ith m Ġof Ġthe ĠIDF Ġin Ġorder Ġto Ġdamp en Ġits Ġeffect . </s> $$ TF ID F ( w , d Ġ| ĠD )= TF Ġ\ times Ġlog _{ 10 } ( ID F ) $$ Ġ Ġ Ġ( E q . Ġ13 ) </s> Con ventional Ġtext Ġclassification Ġmethods </s> Multi - vari ate ĠBern ou lli Ġ( M VB ) Ġand Ġmult in omial Ġnaive ĠBay es Ġ( M NB ) Ġare Ġtwo Ġgener ative Ġmodels Ġbased Ġon Ġthe Ġnaive ĠBay es Ġassumption . ĠIn Ġother Ġwords , Ġthey Ġassume Ġthat Ġall Ġattributes Ġ( e . g ., Ġthe Ġfrequency Ġof Ġeach Ġword , Ġthe Ġpresence Ġor Ġabsence Ġof Ġa Ġword ) Ġof Ġeach Ġtext Ġare Ġindependent Ġof Ġeach Ġother Ġgiven Ġthe Ġcontext Ġof Ġthe Ġclass ĠB IB REF 11 Ġ. </s> In Ġthe ĠMV B Ġmodel , Ġa Ġdocument Ġis Ġrepresented Ġby Ġa Ġvector Ġgenerated Ġby Ġa Ġbag - of - words Ġwith Ġbinary Ġweights . ĠIn Ġthis Ġcase , Ġa Ġdocument Ġcan Ġbe Ġconsidered Ġan Ġevent , Ġand Ġthe Ġpresence Ġor Ġthe Ġabsence Ġof Ġthe Ġwords Ġto Ġbe Ġthe Ġattributes Ġof Ġthe Ġevent . ĠOn Ġthe Ġother Ġhand , Ġthe ĠMN B Ġmodel Ġrepresents Ġeach Ġdocument Ġas Ġa Ġvector Ġgenerated Ġby Ġa Ġbag - of - words Ġwith ĠTF Ġweights . ĠHere , Ġthe Ġindividual Ġword Ġoccurrences Ġare Ġconsidered Ġas Ġevents Ġand Ġthe Ġdocument Ġis Ġa Ġcollection Ġof Ġword Ġevents . </s> Both Ġthese Ġmodels Ġuse Ġthe ĠBay es Ġrule Ġto Ġclassify Ġa Ġdocument . ĠConsider Ġthat Ġeach Ġdocument Ġshould Ġbe Ġclassified Ġinto Ġone Ġof Ġthe Ġclasses Ġin Ġ$ C = \ l brace Ġc _ j \ r brace Ġ_ { j = 1 } ^{ | C | } $ Ġ. ĠThe Ġprobability Ġof Ġeach Ġclass Ġgiven Ġthe Ġdocument Ġis Ġdefined Ġas : </s> $$ P ( c _ j | d _ i ) Ġ= Ġ\ frac { P ( d _ i | c _ j ) P ( c _ j ) }{ P ( d _ i ) }. $$ Ġ Ġ Ġ( E q . Ġ16 ) </s> The Ġprior Ġ$ P ( d _ i ) $ Ġis Ġthe Ġsame Ġfor Ġall Ġclasses , Ġso Ġto Ġdetermine Ġthe Ġclass Ġto Ġwhich Ġ$ d _ i $ Ġbelongs Ġto , Ġthe Ġfollowing Ġequation Ġcan Ġbe Ġused : </s> $$ pred iction ( d _ i ) Ġ= Ġarg max _{ c _ j } P ( d _ i | c _ j ) P ( c _ j ) $$ Ġ Ġ Ġ( E q . Ġ17 ) </s> The Ġprior Ġ$ P ( c _ j ) $ Ġcan Ġbe Ġobtained Ġby Ġthe Ġfollowing Ġequation : </s> $$ P ( c _ j ) Ġ= Ġ\ frac { 1 + | D _ j | }{ | C | + | D | }, $$ Ġ Ġ Ġ( E q . Ġ18 ) </s> where Ġ$ | D _ j | $ Ġis Ġthe Ġnumber Ġof Ġdocuments Ġin Ġclass Ġ$ c _ j $ Ġ. </s> As Ġfor Ġthe Ġposterior Ġ$ P ( d _ i | c _ j ) $ Ġ, Ġdifferent Ġcalculations Ġare Ġperformed Ġfor Ġeach Ġmodel . ĠFor ĠMV B , Ġit Ġis Ġdefined Ġas : </s> $$ P ( d _ i | c _ j ) Ġ= Ġ\ pro d Ġ_ { k = 1 } ^{ | V | } P ( w _ k | c _ j ) ^{ t _ i ^ k } ( 1 - P ( w _ k | c _ j )) ^{ 1 - t _ i ^ k }, $$ Ġ Ġ Ġ( E q . Ġ19 ) </s> where Ġ$ w _ k $ Ġis Ġthe Ġk - th Ġword Ġin Ġthe Ġvocabulary Ġ$ V $ Ġ, Ġand Ġ$ t _ i ^ k $ Ġis Ġthe Ġvalue Ġ( 0 Ġor Ġ1 ) Ġof Ġthe Ġk - th Ġelement Ġof Ġthe Ġvector Ġof Ġdocument Ġ$ d _ i $ Ġ. </s> For Ġthe ĠMN B , Ġit Ġis Ġdefined Ġas : </s> $$ P ( d _ i | c _ j ) Ġ= ĠP ( | d _ i | )| d _ i | ! \ pro d Ġ_ { k = 1 } ^{ | V | }\ frac { P ( w _ k | c _ j ) ^{ n _ i ^ k }} { n _ i ^ k ! }, $$ Ġ Ġ Ġ( E q . Ġ20 ) </s> where Ġ$ | d _ i | $ Ġis Ġthe Ġnumber Ġof Ġwords Ġin Ġdocument Ġ$ d _ i $ Ġand Ġ$ n _ i ^ k $ Ġis Ġthe Ġk - th Ġelement Ġof Ġthe Ġvector Ġof Ġdocument Ġ$ d _ i $ Ġand Ġit Ġrepresents Ġhow Ġmany Ġtimes Ġword Ġ$ w _ k $ Ġoccurs Ġin Ġ$ d _ i $ Ġ. </s> Finally , Ġthe Ġposterior Ġ$ P ( w _ k | c _ j ) $ Ġcan Ġbe Ġobtained Ġby Ġthe Ġfollowing Ġequation : </s> $$ P ( w _ k | c _ j ) Ġ= Ġ\ frac { 1 + | D _ j ^ k | }{ | C | + | D | }, $$ Ġ Ġ Ġ( E q . Ġ21 ) </s> where Ġ$ | D _ j ^ k | $ Ġis Ġthe Ġnumber Ġof Ġdocuments Ġin Ġclass Ġ$ c _ j $ Ġthat Ġcontain Ġthe Ġword Ġ$ w _ k $ Ġ. </s> In Ġgeneral , ĠMV B Ġtends Ġto Ġperform Ġbetter Ġthan ĠMN B Ġat Ġsmall Ġvocabulary Ġsizes Ġwhereas ĠMN B Ġis Ġmore Ġefficient Ġon Ġlarge Ġvoc ab ular ies . </s> Despite Ġbeing Ġrobust Ġtools Ġfor Ġtext Ġclassification , Ġboth Ġthese Ġmodels Ġdepend Ġdirectly Ġon Ġthe Ġbag - of - words Ġfeatures Ġand Ġdo Ġnot Ġnaturally Ġwork Ġwith Ġrepresentations Ġsuch Ġas Ġword 2 vec . </s> Lat ent Ġsemantic Ġanalysis Ġ( L SA ), Ġor Ġlatent Ġsemantic Ġindex ing Ġ( LS I ), Ġwas Ġproposed Ġin ĠB IB REF 12 Ġ, Ġand Ġit Ġextends Ġthe Ġvector Ġspace Ġmodel Ġby Ġusing Ġsingular Ġvalue Ġdecom position Ġ( S VD ) Ġto Ġfind Ġa Ġset Ġof Ġunderlying Ġlatent Ġvariables Ġwhich Ġspans Ġthe Ġmeaning Ġof Ġtexts . </s> It Ġis Ġbuilt Ġfrom Ġa Ġterm - document Ġmatrix , Ġin Ġwhich Ġeach Ġrow Ġrepresents Ġa Ġterm , Ġand Ġeach Ġcolumn Ġrepresents Ġa Ġdocument . ĠThis Ġmatrix Ġcan Ġbe Ġbuilt Ġby Ġconc aten ating Ġthe Ġvectors Ġof Ġall Ġdocuments Ġin Ġa Ġcorpus , Ġobtained Ġusing Ġthe Ġbag - of - words Ġmodel , Ġthat Ġis , Ġ$ Ġ{ X } Ġ= Ġ[ Ġ{ v } _ 1 , Ġ{ v } _ 2 , Ġ... , Ġ{ v } _{ | D | } ] $ Ġ, Ġwhere Ġ${ v } _ i $ Ġis Ġthe Ġvector Ġrepresentation Ġobtained Ġusing Ġthe Ġbag - of - words Ġmodel . </s> In Ġthis Ġmethod , Ġthe Ġterm - document Ġmatrix Ġis Ġdecomp osed Ġusing Ġthe Ġsingular Ġvalue Ġdecom position , </s> $$ { X } Ġ= Ġ{ U \ S igma ĠV } ^ \ top Ġ, $$ Ġ Ġ Ġ( E q . Ġ23 ) </s> where Ġ$ U $ Ġand Ġ$ V $ Ġare Ġorth og onal Ġmat rices Ġand Ġcorrespond Ġto Ġthe Ġleft Ġsingular Ġvectors Ġand Ġright Ġsingular Ġvectors Ġof Ġ$ X $ Ġ, Ġrespectively . Ġ$\ S igma Ġ$ Ġis Ġa Ġdiagonal Ġmatrix , Ġand Ġit Ġcontains Ġthe Ġsquare Ġroots Ġof Ġthe Ġe igen values Ġof Ġ$ X ^ TX $ Ġand Ġ$ XX ^ T $ Ġ. ĠL SA Ġfinds Ġa Ġlow - rank Ġapproximation Ġof Ġ$ X $ Ġby Ġselecting Ġonly Ġthe Ġ$ k $ Ġlargest Ġsingular Ġvalues Ġand Ġits Ġrespective Ġsingular Ġvectors , </s> $$ { X } _ k Ġ= Ġ{ U } _ k {\ S igma Ġ} _ k Ġ{ V } _ k ^ {\ top Ġ} . $$ Ġ Ġ Ġ( E q . Ġ24 ) </s> To Ġcompare Ġtwo Ġdocuments , Ġwe Ġproject Ġboth Ġof Ġthem Ġinto Ġthis Ġlower Ġdimension Ġspace Ġand Ġcalculate Ġthe Ġcos ine Ġdistance Ġbetween Ġthem . ĠThe Ġprojection Ġ${ \ hat { d }} $ Ġof Ġdocument Ġ${ d } $ Ġis Ġobtained Ġby Ġthe Ġfollowing Ġequation : </s> $$ {\ hat { d }} Ġ= Ġ{\ S igma Ġ} _ k ^{ - 1 } Ġ{ U } _ k ^ \ top Ġ{ d }. $$ Ġ Ġ Ġ( E q . Ġ25 ) </s> Despite Ġits Ġextensive Ġapplication Ġon Ġtext Ġclassification ĠB IB REF 13 Ġ, ĠB IB REF 14 Ġ, ĠB IB REF 15 Ġ, Ġthis Ġmethod Ġwas Ġinitially Ġproposed Ġfor Ġdocument Ġindex ing Ġand , Ġtherefore , Ġdoes Ġnot Ġencode Ġany Ġclass Ġinformation Ġwhen Ġmodeling Ġthe Ġlow - rank Ġapproximation . ĠTo Ġperform Ġclassification , Ġ1 - ne arest Ġneighbor Ġis Ġusually Ġperformed , Ġplacing Ġa Ġquery Ġdocument Ġinto Ġthe Ġclass Ġof Ġthe Ġnearest Ġtraining Ġdocument . </s> The Ġsupport Ġvector Ġmachine Ġ( S VM ) Ġwas Ġfirst Ġpresented Ġin ĠB IB REF 16 Ġand Ġperforms Ġthe Ġseparation Ġbetween Ġsamples Ġof Ġtwo Ġdifferent Ġclasses Ġby Ġprojecting Ġthem Ġonto Ġa Ġhigher Ġdimension ality Ġspace . ĠIt Ġwas Ġfirst Ġapplied Ġin Ġtext Ġclassification Ġby ĠB IB REF 17 Ġand Ġhave Ġsince Ġbeen Ġsuccessfully Ġapplied Ġin Ġmany Ġtasks Ġrelated Ġto Ġnatural Ġlanguage Ġprocessing ĠB IB REF 18 Ġ, ĠB IB REF 19 Ġ. </s> Consider Ġa Ġtraining Ġdata Ġset Ġ$ D $ Ġ, Ġwith Ġ$ n $ Ġsamples </s> $$ D Ġ= Ġ\ l brace Ġ({ x } _ i , c _ i )| { x } _ i \ in Ġ\ math bb Ġ{ R } ^ p , Ġc _ i Ġ\ in Ġ\ l brace Ġ- 1 , 1 \ r brace Ġ Ġ\ r brace Ġ_ { i = 1 } ^{ n }, $$ Ġ Ġ Ġ( E q . Ġ27 ) </s> where Ġ$ c _ i $ Ġrepresents Ġthe Ġclass Ġto Ġwhich Ġ${ x } _ i $ Ġbelongs Ġto . ĠEach Ġ${ x } _ i $ Ġis Ġa Ġ$ p $ Ġ- dimensional Ġvector . ĠThe Ġgoal Ġis Ġto Ġfind Ġthe Ġhyper plane Ġthat Ġdivides Ġthe Ġpoints Ġfrom Ġ$ c _ i Ġ= Ġ1 $ Ġfrom Ġthe Ġpoints Ġfrom Ġ$ c _ i Ġ= Ġ- 1 $ Ġ. ĠThis Ġhyper plane Ġcan Ġbe Ġwritten Ġas Ġa Ġset Ġof Ġpoints Ġ$ x $ Ġsatisfying : </s> $$ { w } Ġ\ cd ot Ġ{ x } Ġ- Ġb Ġ= Ġ0 , $$ Ġ Ġ Ġ( E q . Ġ28 ) </s> where Ġ$\ cd ot Ġ$ Ġdenotes Ġthe Ġdot Ġproduct . ĠThe Ġvector Ġ${ w } $ Ġis Ġperpendicular Ġto Ġthe Ġhyper plane . ĠThe Ġparameter Ġ$\ frac { b }{ \ Vert Ġ{ w }\ Vert Ġ} $ Ġdetermines Ġthe Ġoffset Ġof Ġthe Ġhyper plane Ġfrom Ġthe Ġorigin Ġalong Ġthe Ġnormal Ġvector Ġ${ w } $ Ġ. </s> We Ġwish Ġto Ġchoose Ġ${ w } $ Ġand Ġ$ b $ Ġ, Ġso Ġthey Ġmaximize Ġthe Ġdistance Ġbetween Ġthe Ġparallel Ġhyper planes Ġthat Ġare Ġas Ġfar Ġapart Ġas Ġpossible , Ġwhile Ġstill Ġseparating Ġthe Ġdata . </s> If Ġthe Ġtraining Ġdata Ġis Ġlin early Ġsepar able , Ġwe Ġcan Ġselect Ġtwo Ġhyper planes Ġin Ġa Ġway Ġthat Ġthere Ġare Ġno Ġpoints Ġbetween Ġthem Ġand Ġthen Ġtry Ġto Ġmaximize Ġthe Ġdistance . ĠIn Ġother Ġwords , Ġminimize Ġ$\ Vert Ġ{ w }\ Vert Ġ$ Ġsubject Ġto Ġ$ c _ i ({ w }\ cd ot Ġ{ x } _ u - b ) Ġ\ ge Ġ1 , Ġi = \ l brace Ġ1 , 2 ,... , n \ r brace Ġ$ Ġ. ĠIf Ġthe Ġtraining Ġdata Ġis Ġnot Ġlin early Ġsepar able , Ġthe Ġkernel Ġtrick Ġcan Ġbe Ġapplied , Ġwhere Ġevery Ġdot Ġproduct Ġis Ġreplaced Ġby Ġa Ġnon - linear Ġkernel Ġfunction . </s> Word Ġsub space </s> All Ġmethods Ġmentioned Ġabove Ġutilize Ġthe ĠB OW Ġfeatures Ġto Ġrepresent Ġa Ġdocument . ĠAlthough Ġthis Ġrepresentation Ġis Ġsimple Ġand Ġpowerful , Ġits Ġmain Ġproblem Ġlies Ġon Ġdisreg arding Ġthe Ġword Ġsemantics Ġwithin Ġa Ġdocument , Ġwhere Ġthe Ġcontext Ġand Ġmeaning Ġcould Ġoffer Ġmany Ġbenefits Ġto Ġthe Ġmodel Ġsuch Ġas Ġidentification Ġof Ġsyn onyms . </s> In Ġour Ġformulation , Ġwords Ġare Ġrepresented Ġas Ġvectors Ġin Ġa Ġreal - valued Ġfeature Ġvector Ġspace Ġ$\ math bb Ġ{ R } ^{ p } $ Ġ, Ġby Ġusing Ġword 2 vec ĠB IB REF 8 Ġ. ĠThrough Ġthis Ġrepresentation , Ġit Ġis Ġpossible Ġto Ġcalculate Ġthe Ġdistance Ġbetween Ġtwo Ġwords , Ġwhere Ġwords Ġfrom Ġsimilar Ġcontexts Ġare Ġrepresented Ġby Ġvectors Ġclose Ġto Ġeach Ġother , Ġwhile Ġwords Ġfrom Ġdifferent Ġcontexts Ġare Ġrepresented Ġas Ġfar Ġapart Ġvectors . ĠAlso , Ġthis Ġrepresentation Ġbrings Ġthe Ġnew Ġconcept Ġof Ġarithmetic Ġoperations Ġbetween Ġwords , Ġwhere Ġoperations Ġsuch Ġas Ġaddition Ġand Ġsubt raction Ġcarry Ġmeaning Ġ( eg ., ĠâĢ ľ king âĢ Ŀ - âĢ ľ man âĢ Ŀ + âĢ ľ woman âĢ Ŀ = âĢ ľ que en âĢ Ŀ ) ĠB IB REF 20 Ġ. </s> Consider Ġa Ġset Ġof Ġdocuments Ġwhich Ġbelong Ġto Ġthe Ġsame Ġcontext Ġ$ D _ c Ġ= Ġ\ l brace Ġd _ i \ r brace Ġ_ { i = 1 } ^{ | D _ c | } $ Ġ. ĠEach Ġdocument Ġ$ d _ i $ Ġis Ġrepresented Ġby Ġa Ġset Ġof Ġ$ N _ i $ Ġwords , Ġ$ d _ i Ġ= Ġ\ l brace Ġw _ k \ r brace Ġ_ { k = 1 } ^{ N _ i } $ Ġ. ĠBy Ġconsidering Ġthat Ġall Ġwords Ġfrom Ġdocuments Ġof Ġthe Ġsame Ġcontext Ġbelong Ġto Ġthe Ġsame Ġdistribution , Ġa Ġset Ġof Ġwords Ġ$ W _ c Ġ= Ġ\ l brace Ġw _ k \ r brace Ġ_ { k = 1 } ^{ N _ c } $ Ġwith Ġthe Ġwords Ġin Ġthe Ġcontext Ġ$ c $ Ġis Ġobtained . </s> We Ġthen Ġtranslate Ġthese Ġwords Ġinto Ġword Ġvectors Ġusing Ġword 2 vec , Ġresulting Ġin Ġa Ġset Ġof Ġword Ġvectors Ġ$ X _ c Ġ= Ġ\ l brace Ġ{ x } ^ k _ c \ r brace Ġ_ { k = 1 } ^{ N _ c } Ġ\ in Ġ\ math bb Ġ{ R } ^ p $ Ġ. ĠThis Ġset Ġof Ġword Ġvectors Ġis Ġmodeled Ġinto Ġa Ġword Ġsub space , Ġwhich Ġis Ġa Ġcompact , Ġscalable Ġand Ġmeaningful Ġrepresentation Ġof Ġthe Ġwhole Ġset . ĠSuch Ġa Ġword Ġsub space Ġis Ġgenerated Ġby Ġapplying ĠPC A Ġto Ġthe Ġset Ġof Ġword Ġvectors . </s> First , Ġwe Ġcompute Ġan Ġaut oc or relation Ġmatrix , Ġ${ R } _ c $ Ġ: </s> $$ { R } _ c Ġ= Ġ\ frac { 1 }{ N _ c }\ sum Ġ_ { i = 1 } ^{ N _ c }{ x } ^{ i } _ c { x } _ c ^{ i ^ {\ top Ġ} }. $$ Ġ Ġ Ġ( E q . Ġ29 ) </s> The Ġorth on ormal Ġbasis Ġvectors Ġof Ġ$ m _ c $ Ġ- dimensional Ġsub space Ġ${ Y } _ c $ Ġare Ġobtained Ġas Ġthe Ġe igen ve ctors Ġwith Ġthe Ġ$ m _ c $ Ġlargest Ġe igen values Ġof Ġthe Ġmatrix Ġ${ R } _ c $ Ġ. ĠWe Ġrepresent Ġa Ġsub space Ġ${ Y } _ c $ Ġby Ġthe Ġmatrix Ġ${ Y } _ c Ġ\ in Ġ\ math bb Ġ{ R } ^{ p Ġ\ times Ġm _ c } $ Ġ, Ġwhich Ġhas Ġthe Ġcorresponding Ġorth on ormal Ġbasis Ġvectors Ġas Ġits Ġcolumn Ġvectors . </s> Text Ġclassification Ġbased Ġon Ġword Ġsub space </s> We Ġformulate Ġour Ġproblem Ġas Ġa Ġsingle Ġlabel Ġclassification Ġproblem . ĠGiven Ġa Ġset Ġof Ġtraining Ġdocuments , Ġwhich Ġwe Ġwill Ġrefer Ġas Ġcorpus , Ġ$ D Ġ= Ġ\ l brace Ġd _ i \ r brace Ġ_ { i = 1 } ^{ | D | } $ Ġ, Ġwith Ġknown Ġclasses Ġ$ C Ġ= Ġ\ l brace Ġc _ j \ r brace Ġ_ { j = 1 } ^{ | C | } $ Ġ, Ġwe Ġwish Ġto Ġclassify Ġa Ġquery Ġdocument Ġ$ d _ q $ Ġinto Ġone Ġof Ġthe Ġclasses Ġin Ġ$ C $ Ġ. </s> Text Ġclassification Ġbased Ġon Ġword Ġsub space Ġcan Ġbe Ġperformed Ġunder Ġthe Ġframework Ġof Ġmutual Ġsub space Ġmethod Ġ( MS M ). ĠThis Ġtask Ġinvolves Ġtwo Ġdifferent Ġstages : ĠA Ġlearning Ġstage , Ġwhere Ġthe Ġword Ġsub space Ġfor Ġeach Ġclass Ġis Ġmodeled , Ġand Ġa Ġclassification Ġstage , Ġwhere Ġthe Ġword Ġsub space Ġfor Ġa Ġquery Ġis Ġmodeled Ġand Ġcompared Ġto Ġthe Ġword Ġsub sp aces Ġof Ġthe Ġclasses . </s> In Ġthe Ġlearning Ġstage , Ġit Ġis Ġassumed Ġthat Ġall Ġdocuments Ġof Ġthe Ġsame Ġclass Ġbelong Ġto Ġthe Ġsame Ġcontext , Ġresulting Ġin Ġa Ġset Ġof Ġwords Ġ$ W _ c Ġ= Ġ\ l brace Ġw _ c ^ k \ r brace Ġ_ { k = 1 } ^{ N _ c } $ Ġ. ĠThis Ġset Ġassumes Ġthat Ġeach Ġword Ġappears Ġonly Ġonce Ġin Ġeach Ġclass . ĠEach Ġset Ġ$\ l brace ĠW _ c \ r brace Ġ_ { c = 1 } ^{ | C | } $ Ġis Ġthen Ġmodeled Ġinto Ġa Ġword Ġsub space Ġ${ Y } _ c $ Ġ, Ġas Ġexplained Ġin ĠSection Ġ" Word Ġsub space " Ġ. ĠAs Ġthe Ġnumber Ġof Ġwords Ġin Ġeach Ġclass Ġmay Ġvary Ġlargely , Ġthe Ġdimension Ġ$ m _ c $ Ġof Ġeach Ġclass Ġword Ġsub space Ġis Ġnot Ġset Ġto Ġthe Ġsame Ġvalue . </s> In Ġthe Ġclassification Ġstage , Ġfor Ġa Ġquery Ġdocument Ġ$ d _ q $ Ġ, Ġit Ġis Ġalso Ġassumed Ġthat Ġeach Ġword Ġoccurs Ġonly Ġonce , Ġgenerating Ġa Ġsub space Ġ${ Y } _ q $ Ġ. </s> To Ġmeasure Ġthe Ġsimilarity Ġbetween Ġa Ġclass Ġword Ġsub space Ġ${ Y } _ c $ Ġand Ġa Ġquery Ġword Ġsub space Ġ${ Y } _ q $ Ġ, Ġthe Ġcanonical Ġangles Ġbetween Ġthe Ġtwo Ġword Ġsub sp aces Ġare Ġused ĠB IB REF 21 Ġ. ĠThere Ġare Ġseveral Ġmethods Ġfor Ġcalculating Ġcanonical Ġangles ĠB IB REF 22 Ġ, ĠB IB REF 23 Ġ, Ġand ĠB IB REF 24 Ġ, Ġamong Ġwhich Ġthe Ġsimplest Ġand Ġmost Ġpractical Ġis Ġthe Ġsingular Ġvalue Ġdecom position Ġ( S VD ). ĠConsider , Ġfor Ġexample , Ġtwo Ġsub sp aces , Ġone Ġfrom Ġthe Ġtraining Ġdata Ġand Ġanother Ġfrom Ġthe Ġquery , Ġrepresented Ġas Ġmat rices Ġof Ġbases , Ġ${ Y } _{ c } Ġ= Ġ[ {\ Ph i Ġ} _{ 1 } Ġ\ ld ots Ġ{\ Ph i Ġ} _{ m _ c } ] Ġ\ in Ġ\ math bb Ġ{ R } ^{ p Ġ\ times Ġm _ c } $ Ġand Ġ${ Y } _{ q } Ġ= Ġ[ {\ Ps i Ġ} _{ 1 } Ġ\ ld ots Ġ{\ Ps i Ġ} _{ m _ q } ] Ġ\ in Ġ\ math bb Ġ{ R } ^{ p Ġ\ times Ġm _ q } $ Ġ, Ġwhere Ġ${ \ Ph i Ġ} _{ i } $ Ġare Ġthe Ġbases Ġfor Ġ${ Y } _ c $ Ġand Ġ${ \ Ps i Ġ} _{ i } $ Ġare Ġthe Ġbases Ġfor Ġ${ Y } _ q $ Ġ. ĠLet Ġthe ĠS VD Ġof Ġ${ Y } _ c ^ {\ top Ġ} { Y } _ q Ġ\ in Ġ\ math bb Ġ{ R } ^{ m _ c Ġ\ times Ġm _ q } $ Ġbe Ġ${ Y } _ c ^ {\ top Ġ} { Y } _ q Ġ= Ġ{ U Ġ\ S igma ĠV } ^ {\ top Ġ} $ Ġ, Ġwhere Ġ${ Y } _ q $ 0 Ġ, Ġ${ Y } _ q $ 1 Ġrepresents Ġthe Ġset Ġof Ġsingular Ġvalues . ĠThe Ġcanonical Ġangles Ġ${ Y } _ q $ 2 Ġcan Ġbe Ġobtained Ġas Ġ${ Y } _ q $ 3 Ġ${ Y } _ q $ 4 Ġ. ĠThe Ġsimilarity Ġbetween Ġthe Ġtwo Ġsub sp aces Ġis Ġmeasured Ġby Ġ${ Y } _ q $ 5 Ġangles Ġas Ġfollows : </s> $$ S _{ ({ Y } _ c , { Y } _ q )} [ t ] Ġ= Ġ\ frac { 1 }{ t }\ sum Ġ_ { i Ġ= Ġ1 } ^{ t } Ġ\ cos Ġ^ { 2 } Ġ\ the ta Ġ_ { i }, \ ; Ġ1 Ġ\ le Ġt Ġ\ le Ġm _ q , Ġ\ ; Ġm _ q Ġ\ le Ġm _ c . $$ Ġ Ġ Ġ( E q . Ġ30 ) </s> Fig . Ġ1 Ġshows Ġthe Ġmodeling Ġand Ġcomparison Ġof Ġsets Ġof Ġwords Ġby ĠMSM . ĠThis Ġmethod Ġcan Ġcompare Ġsets Ġof Ġdifferent Ġsizes , Ġand Ġnaturally Ġenc odes Ġproximity Ġbetween Ġsets Ġwith Ġrelated Ġwords . </s> Finally , Ġthe Ġclass Ġwith Ġthe Ġhighest Ġsimilarity Ġwith Ġ$ d _ q $ Ġis Ġassigned Ġas Ġthe Ġclass Ġof Ġ$ d _ q $ Ġ: </s> $$ pred iction ( d _ q ) Ġ= Ġarg max _ c ( S _{ ({ Y } _ c , { Y } _ q )} ). $$ Ġ Ġ Ġ( E q . Ġ32 ) </s> TF Ġweighted Ġword Ġsub space </s> The Ġword Ġsub space Ġformulation Ġpresented Ġin ĠSection Ġ" Word Ġsub space " Ġis Ġa Ġpractical Ġand Ġcompact Ġway Ġto Ġrepresent Ġsets Ġof Ġword Ġvectors , Ġretaining Ġmost Ġof Ġthe Ġvariability Ġof Ġfeatures . ĠHowever , Ġas Ġseen Ġin Ġthe ĠB OW Ġfeatures , Ġthe Ġfrequency Ġof Ġwords Ġis Ġrelevant Ġinformation Ġthat Ġcan Ġimprove Ġthe Ġcharacterization Ġof Ġa Ġtext . ĠTo Ġincorporate Ġthis Ġinformation Ġinto Ġthe Ġword Ġsub space Ġmodeling , Ġwe Ġpropose Ġan Ġextension Ġof Ġthe Ġword Ġsub space , Ġcalled Ġthe Ġterm - frequency Ġ( TF ) Ġweighted Ġword Ġsub space . </s> Like Ġthe Ġword Ġsub space , Ġthe ĠTF Ġweighted Ġword Ġsub space Ġis Ġmat hematically Ġdefined Ġas Ġa Ġlow - dimensional Ġlinear Ġsub space Ġin Ġa Ġword Ġvector Ġspace Ġwith Ġhigh Ġdimension ality . ĠHowever , Ġa Ġweighted Ġversion Ġof Ġthe ĠPC A ĠB IB REF 25 Ġ, ĠB IB REF 26 Ġis Ġutilized Ġto Ġincorporate Ġthe Ġinformation Ġgiven Ġby Ġthe Ġfrequencies Ġof Ġwords Ġ( term - f requ encies ). ĠThis ĠTF Ġweighted Ġword Ġsub space Ġis Ġequivalent Ġto Ġthe Ġword Ġsub space Ġif Ġwe Ġconsider Ġall Ġoccurrences Ġof Ġthe Ġwords . </s> Consider Ġthe Ġset Ġof Ġword Ġvectors Ġ$\ l brace Ġ{ x } _ c ^ k \ r brace Ġ_ { k = 1 } ^{ N _ c } Ġ\ in Ġ\ math bb Ġ{ R } ^{ p } $ Ġ, Ġwhich Ġrepresents Ġeach Ġword Ġin Ġthe Ġcontext Ġ$ c $ Ġ, Ġand Ġthe Ġset Ġof Ġweights Ġ$\ l brace Ġ\ ome ga Ġ_ i \ r brace Ġ_ { i = 1 } ^{ N _ c } $ Ġ, Ġwhich Ġrepresent Ġthe Ġfrequencies Ġof Ġthe Ġwords Ġin Ġthe Ġcontext Ġ$ c $ Ġ. </s> We Ġincorporate Ġthese Ġfrequencies Ġinto Ġthe Ġsub space Ġcalculation Ġby Ġweight ing Ġthe Ġdata Ġmatrix Ġ${ X } $ Ġas Ġfollows : </s> $$ {\ wid et ilde { X }} ={ X }{ \ O mega Ġ} ^{ 1 / 2 }, $$ Ġ Ġ Ġ( E q . Ġ33 ) </s> where Ġ${ X } Ġ\ in Ġ\ math bb Ġ{ R } ^{ p Ġ\ times ĠN _ c } $ Ġis Ġa Ġmatrix Ġcontaining Ġthe Ġword Ġvectors Ġ$\ l brace Ġ{ x } _ c ^ k \ r brace Ġ_ { k = 1 } ^{ N _ c } $ Ġand Ġ${ \ O mega Ġ} $ Ġis Ġa Ġdiagonal Ġmatrix Ġcontaining Ġthe Ġweights Ġ$\ l brace Ġ\ ome ga Ġ_ i \ r brace Ġ_ { i = 1 } ^{ N _ c } $ Ġ. </s> We Ġthen Ġperform ĠPC A Ġby Ġsolving Ġthe ĠS VD Ġof Ġthe Ġmatrix Ġ${ \ wid et ilde { X }} $ Ġ: </s> $$ {\ wid et ilde { X }} ={ AM B } ^ {\ top Ġ}, $$ Ġ Ġ Ġ( E q . Ġ34 ) </s> where Ġthe Ġcolumns Ġof Ġthe Ġorth og onal Ġmat rices Ġ${ A } $ Ġand Ġ${ B } $ Ġare , Ġrespectively , Ġthe Ġleft - sing ular Ġvectors Ġand Ġright - sing ular Ġvectors Ġof Ġthe Ġmatrix Ġ${ \ wid et ilde { X }} $ Ġ, Ġand Ġthe Ġdiagonal Ġmatrix Ġ${ M } $ Ġcontains Ġsingular Ġvalues Ġof Ġ${ \ wid et ilde { X }} $ Ġ. </s> Finally , Ġthe Ġorth on ormal Ġbasis Ġvectors Ġof Ġthe Ġ$ m _ c $ Ġ- dimensional ĠTF Ġweighted Ġsub space Ġ${ W } $ Ġare Ġthe Ġcolumn Ġvectors Ġin Ġ${ A } $ Ġcorresponding Ġto Ġthe Ġ$ m _ c $ Ġlargest Ġsingular Ġvalues Ġin Ġ${ M } $ Ġ. </s> Text Ġclassification Ġwith ĠTF Ġweighted Ġword Ġsub space Ġcan Ġalso Ġbe Ġperformed Ġunder Ġthe Ġframework Ġof ĠMSM . ĠIn Ġthis Ġpaper , Ġwe Ġwill Ġrefer Ġto ĠMSM Ġwith ĠTF Ġweighted Ġword Ġsub space Ġas ĠTF - MS M . </s> Exper imental ĠEvaluation </s> In Ġthis Ġsection Ġwe Ġdescribe Ġthe Ġexperiments Ġperformed Ġto Ġdemonstrate Ġthe Ġvalidity Ġof Ġour Ġproposed Ġmethod Ġand Ġits Ġextension . ĠWe Ġused Ġthe ĠReuters - 8 Ġdataset Ġwithout Ġstop Ġwords Ġfrom ĠB IB REF 27 Ġaiming Ġat Ġsingle - label Ġclassification , Ġwhich Ġis Ġa Ġpre process ed Ġformat Ġof Ġthe ĠReuters - 215 78 . ĠWords Ġin Ġthe Ġtexts Ġwere Ġconsidered Ġas Ġthey Ġappeared , Ġwithout Ġperforming Ġstemming Ġor Ġtypo Ġcorrection . ĠThis Ġdatabase Ġhas Ġeight Ġdifferent Ġclasses Ġwith Ġthe Ġnumber Ġof Ġsamples Ġvarying Ġfrom Ġ51 Ġto Ġover Ġ3000 Ġdocuments , Ġas Ġcan Ġbe Ġseen Ġin ĠTable Ġ1 Ġ. </s> To Ġobtain Ġthe Ġvector Ġrepresentation Ġof Ġwords , Ġwe Ġused Ġa Ġfreely Ġavailable Ġword 2 vec Ġmodel , Ġtrained Ġby ĠB IB REF 8 Ġ, Ġon Ġapproximately Ġ100 Ġbillion Ġwords , Ġwhich Ġenc odes Ġthe Ġvector Ġrepresentation Ġin Ġ$\ math bb Ġ{ R } ^{ 300 } $ Ġof Ġover Ġ3 Ġmillion Ġwords Ġfrom Ġseveral Ġdifferent Ġlanguages . ĠSince Ġwe Ġdecided Ġto Ġfocus Ġon ĠEnglish Ġwords Ġonly , Ġwe Ġfiltered Ġthese Ġvectors Ġto Ġabout Ġ800 Ġthousand Ġwords , Ġexcluding Ġall Ġwords Ġwith Ġnon - roman Ġcharacters . </s> To Ġshow Ġthe Ġvalidity Ġof Ġour Ġword Ġsub space Ġrepresentation Ġfor Ġtext Ġclassification Ġand Ġthe Ġproposed Ġextension , Ġwe Ġdivided Ġour Ġexperiment Ġsection Ġinto Ġtwo Ġparts : ĠThe Ġfirst Ġone Ġaims Ġto Ġverify Ġif Ġsets Ġof Ġword Ġvectors Ġare Ġsuitable Ġfor Ġsub space Ġrepresentation , Ġand Ġthe Ġsecond Ġone Ġputs Ġour Ġmethods Ġin Ġpractice Ġin Ġa Ġtext Ġclassification Ġtest , Ġcomparing Ġour Ġresults Ġwith Ġthe Ġconventional Ġmethods Ġdescribed Ġin ĠSection Ġ" Related ĠWork " Ġ. </s> E val uation Ġof Ġthe Ġword Ġsub space Ġrepresentation </s> In Ġthis Ġexperiment , Ġwe Ġmodeled Ġthe Ġword Ġvectors Ġfrom Ġeach Ġclass Ġin Ġthe ĠReuters - 8 Ġdatabase Ġinto Ġa Ġword Ġsub space . ĠThe Ġprimary Ġgoal Ġis Ġto Ġvisualize Ġhow Ġmuch Ġof Ġthe Ġtext Ġdata Ġcan Ġbe Ġrepresented Ġby Ġa Ġlower Ġdimensional Ġsub space . </s> Sub space Ġrepresentations Ġare Ġvery Ġefficient Ġin Ġcompact ly Ġrepresent Ġdata Ġthat Ġis Ġclose Ġto Ġa Ġnormal Ġdistribution . ĠThis Ġcharacteristic Ġis Ġdue Ġto Ġthe Ġapplication Ġof Ġthe ĠPC A , Ġthat Ġis Ġoptimal Ġto Ġfind Ġthe Ġdirection Ġwith Ġthe Ġhighest Ġvariation Ġwithin Ġthe Ġdata . </s> In ĠPC A , Ġthe Ġprincipal Ġcomponents Ġgive Ġthe Ġdirections Ġof Ġmaximum Ġvariance , Ġwhile Ġtheir Ġcorresponding Ġe igen values Ġgive Ġthe Ġvariance Ġof Ġthe Ġdata Ġin Ġeach Ġof Ġthem . ĠTherefore , Ġby Ġobserving Ġthe Ġdistribution Ġof Ġthe Ġe igen values Ġcomputed Ġwhen Ġperforming ĠPC A Ġin Ġthe Ġmodeling Ġof Ġthe Ġsub sp aces , Ġwe Ġcan Ġsuggest Ġif Ġthe Ġdata Ġis Ġsuitable Ġor Ġnot Ġfor Ġsub space Ġrepresentation . </s> For Ġeach Ġclass , Ġwe Ġnormalized Ġthe Ġe igen values Ġby Ġthe Ġlargest Ġone Ġof Ġthe Ġclass . ĠFig . Ġ2 Ġshows Ġthe Ġmean Ġof Ġthe Ġe igen values Ġand Ġthe Ġstandard Ġdeviation Ġamong Ġclasses . ĠIt Ġis Ġpossible Ġto Ġsee Ġthat Ġthe Ġfirst Ġlargest Ġe igen values Ġretain Ġlarger Ġvariance Ġthan Ġthe Ġsmallest Ġones . ĠIn Ġfact , Ġlooking Ġat Ġthe Ġfirst Ġ150 Ġlargest Ġe igen values , Ġwe Ġcan Ġsee Ġthat Ġthey Ġretain , Ġon Ġaverage , Ġ86 . 37 % Ġof Ġthe Ġdata Ġvariance . ĠAlso , Ġby Ġobserving Ġthe Ġstandard Ġdeviation , Ġwe Ġcan Ġunderstand Ġthat Ġthe Ġe igen values Ġdistribution Ġamong Ġclasses Ġfollows Ġthe Ġsame Ġpattern , Ġthat Ġis , Ġmost Ġof Ġthe Ġvariance Ġis Ġin Ġthe Ġfirst Ġdimensions . ĠThis Ġplot Ġindicates Ġthat Ġtext Ġdata Ġrepresented Ġby Ġvectors Ġgenerated Ġwith Ġword 2 vec Ġis Ġsuitable Ġfor Ġsub space Ġrepresentation . </s> Text Ġclassification Ġexperiment </s> In Ġthis Ġexperiment , Ġwe Ġperformed Ġtext Ġclassification Ġamong Ġthe Ġclasses Ġin Ġthe ĠReuters - 8 Ġdatabase . ĠWe Ġcompared Ġthe Ġclassification Ġusing Ġthe Ġword Ġsub space , Ġand Ġits Ġweighted Ġextension , Ġbased Ġon ĠMSM Ġ( to Ġwhich Ġwe Ġwill Ġrefer Ġas ĠMSM Ġand ĠTF - MS M , Ġrespectively ) Ġwith Ġthe Ġbas elines Ġpresented Ġin ĠSection Ġ" Related ĠWork " Ġ: ĠMV B , ĠMN B , ĠL SA , Ġand ĠS VM . ĠSince Ġnone Ġof Ġthe Ġbaseline Ġmethods Ġwork Ġwith Ġvector Ġset Ġclassification , Ġwe Ġalso Ġcompared Ġto Ġa Ġsimple Ġbaseline Ġfor Ġcomparing Ġsets Ġof Ġvectors , Ġdefined Ġas Ġthe Ġaverage Ġof Ġsimilarities Ġbetween Ġall Ġvector Ġpair Ġcombinations Ġof Ġtwo Ġgiven Ġsets . ĠFor Ġtwo Ġmat rices Ġ${ A } $ Ġand Ġ${ B } $ Ġ, Ġcontaining Ġthe Ġsets Ġof Ġvectors Ġ$\ l brace Ġ Ġ{ x } ^{ i } _ a Ġ\ r brace Ġ_ { i Ġ= Ġ1 } ^{ N _ A } $ Ġand Ġ$\ l brace Ġ Ġ{ x } ^{ i } _ b Ġ\ r brace Ġ_ { i Ġ= Ġ1 } ^{ N _ B } $ Ġ, Ġrespectively , Ġwhere Ġ$ N _ A $ Ġand Ġ$ N _ B $ Ġare Ġthe Ġnumber Ġof Ġmain Ġwords Ġin Ġeach Ġset , Ġthe Ġsimilarity Ġis Ġdefined Ġas : </s> $$ Sim _{ ( A , B )} Ġ= Ġ\ frac { 1 }{ N _ A ĠN _ B }\ sum Ġ_ { i } ^{ N _ A }\ sum Ġ_ { j } ^{ N _ B }{ { x } _ a ^ i } ^ {\ top Ġ} { x } _ b ^ j . $$ Ġ Ġ Ġ( E q . Ġ41 ) </s> We Ġrefer Ġto Ġthis Ġbaseline Ġas Ġsimilarity Ġaverage Ġ( SA ). ĠFor Ġthis Ġmethod , Ġwe Ġonly Ġconsidered Ġone Ġoccurrence Ġof Ġeach Ġword Ġin Ġeach Ġset . </s> Different Ġfeatures Ġwere Ġused , Ġdepending Ġon Ġthe Ġmethod . ĠClassification Ġwith ĠSA , ĠMSM , Ġand ĠTF - MS M Ġwas Ġperformed Ġusing Ġword 2 vec Ġfeatures , Ġto Ġwhich Ġwe Ġrefer Ġas Ġw 2 v . ĠFor ĠMV B , Ġdue Ġto Ġits Ġnature , Ġonly Ġbag - of - words Ġfeatures Ġwith Ġbinary Ġweights Ġwere Ġused Ġ( bin B OW ). ĠFor Ġthe Ġsame Ġreason , Ġwe Ġonly Ġused Ġbag - of - words Ġfeatures Ġwith Ġterm - frequency Ġweights Ġ( tf B OW ) Ġwith ĠMN B . ĠClassification Ġwith ĠL SA Ġand ĠS VM Ġis Ġusually Ġperformed Ġusing Ġbag - of - words Ġfeatures Ġand , Ġtherefore , Ġwe Ġtested Ġwith Ġbin B OW , Ġtf B OW , Ġand Ġwith Ġthe Ġterm - frequency Ġinverse Ġdocument - frequency Ġweight , Ġtf id f B OW . ĠWe Ġalso Ġtested Ġthem Ġusing Ġword 2 vec Ġvectors . ĠIn Ġthis Ġcase , Ġwe Ġconsidered Ġeach Ġword Ġvector Ġfrom Ġall Ġdocuments Ġin Ġeach Ġclass Ġto Ġbe Ġa Ġsingle Ġsample . </s> To Ġdetermine Ġthe Ġdimensions Ġof Ġthe Ġclass Ġsub sp aces Ġand Ġquery Ġsub space Ġof ĠMSM Ġand ĠTF - MS M , Ġand Ġthe Ġdimension Ġof Ġthe Ġapproximation Ġperformed Ġby ĠL SA , Ġwe Ġperformed Ġa Ġ10 - fold Ġcross Ġvalidation , Ġwherein Ġeach Ġfold , Ġthe Ġdata Ġwere Ġrandomly Ġdivided Ġinto Ġtrain Ġ( 60 %), Ġvalidation Ġ( 20 %) Ġand Ġtest Ġset Ġ( 20 %). </s> The Ġresults Ġcan Ġbe Ġseen Ġin ĠTable Ġ2 Ġ. ĠThe Ġsimplest Ġbaseline , ĠSA Ġwith Ġw 2 v , Ġachieved Ġan Ġaccuracy Ġrate Ġof Ġ78 . 73 %. ĠThis Ġresult Ġis Ġimportant Ġbecause Ġit Ġshows Ġthe Ġvalidity Ġof Ġthe Ġword 2 vec Ġrepresentation , Ġperforming Ġbetter Ġthan Ġmore Ġelaborate Ġmethods Ġbased Ġon ĠB OW , Ġsuch Ġas ĠMV B Ġwith Ġbin B OW . </s> L SA Ġwith ĠB OW Ġfeatures Ġwas Ġalmost Ġ10 % Ġmore Ġaccurate Ġthan ĠSA , Ġwhere Ġthe Ġbest Ġresults Ġwith Ġbinary Ġweights Ġwere Ġachieved Ġwith Ġan Ġapproximation Ġwith Ġ130 Ġdimensions , Ġwith ĠTF Ġweights Ġwere Ġachieved Ġwith Ġ50 Ġdimensions , Ġand Ġwith ĠTF - ID F Ġweights Ġwere Ġachieved Ġwith Ġ30 Ġdimensions . ĠS VM Ġwith ĠB OW Ġfeatures Ġwas Ġabout Ġ3 % Ġmore Ġaccurate Ġthan ĠL SA , Ġwith Ġbinary Ġweights Ġleading Ġto Ġa Ġhigher Ġaccuracy Ġrate . </s> It Ġis Ġinteresting Ġto Ġnote Ġthat Ġdespite Ġthe Ġreasonably Ġhigh Ġaccuracy Ġrates Ġachieved Ġusing ĠL SA Ġand ĠS VM Ġwith ĠB OW Ġfeatures , Ġthey Ġpoorly Ġperformed Ġwhen Ġusing Ġw 2 v Ġfeatures . </s> Among Ġthe Ġbas elines , Ġthe Ġbest Ġmethod Ġwas ĠMN B Ġwith Ġtf B OW Ġfeatures , Ġwith Ġan Ġaccuracy Ġof Ġ91 . 47 %, Ġbeing Ġthe Ġonly Ġconventional Ġmethod Ġto Ġoutper form ĠMSM . ĠMSM Ġwith Ġw 2 v Ġhad Ġan Ġaccuracy Ġrate Ġof Ġ90 . 62 %, Ġwith Ġthe Ġbest Ġresults Ġachieved Ġwith Ġword Ġsub space Ġdimensions Ġfor Ġthe Ġtraining Ġclasses Ġranging Ġfrom Ġ150 Ġto Ġ181 , Ġand Ġfor Ġthe Ġquery Ġranging Ġfrom Ġ3 Ġto Ġ217 . ĠInc orpor ating Ġthe Ġfrequency Ġinformation Ġin Ġthe Ġsub space Ġmodeling Ġresulted Ġin Ġhigher Ġaccuracy , Ġwith ĠTF - MS M Ġachieving Ġ92 . 01 %, Ġwith Ġdimensions Ġof Ġword Ġsub sp aces Ġfor Ġtraining Ġclasses Ġranging Ġfrom Ġ150 Ġto Ġ172 , Ġand Ġfor Ġthe Ġquery , Ġranging Ġfrom Ġ2 Ġto Ġ109 . ĠTo Ġconfirm Ġthat ĠTF - MS M Ġis Ġsignificantly Ġmore Ġaccurate Ġthan ĠMN B , Ġwe Ġperformed Ġa Ġt - test Ġto Ġcompare Ġtheir Ġresults . ĠIt Ġresulted Ġin Ġa Ġp - value Ġof Ġ0 . 031 , Ġwhich Ġshows Ġthat Ġat Ġa Ġ95 % Ġsignificance Ġlevel , ĠTF - MS M Ġhas Ġproduced Ġbetter Ġresults . </s> Discussion </s> Given Ġthe Ġobservation Ġof Ġthe Ġe igen values Ġdistribution Ġof Ġword Ġvectors , Ġwe Ġcould Ġsee Ġthat Ġword Ġvectors Ġthat Ġbelong Ġto Ġthe Ġsame Ġcontext , Ġi . e ., Ġsame Ġclass , Ġare Ġsuitable Ġfor Ġsub space Ġrepresentation . ĠOur Ġanalysis Ġshowed Ġthat Ġhalf Ġof Ġthe Ġword Ġvector Ġspace Ġdimensions Ġsuffice Ġto Ġrepresent Ġmost Ġof Ġthe Ġvariability Ġof Ġthe Ġdata Ġin Ġeach Ġclass Ġof Ġthe ĠReuters - 8 Ġdatabase . </s> The Ġresults Ġfrom Ġthe Ġtext Ġclassification Ġexperiment Ġshowed Ġthat Ġsub space - based Ġmethods Ġperformed Ġbetter Ġthan Ġthe Ġtext Ġclassification Ġmethods Ġdiscussed Ġin Ġthis Ġwork . ĠUltimately , Ġour Ġproposed ĠTF Ġweighted Ġword Ġsub space Ġwith ĠMSM Ġsurpassed Ġall Ġthe Ġother Ġmethods . Ġword 2 vec Ġfeatures Ġare Ġreliable Ġtools Ġto Ġrepresent Ġthe Ġsemantic Ġmeaning Ġof Ġthe Ġwords Ġand Ġwhen Ġtreated Ġas Ġsets Ġof Ġword Ġvectors , Ġthey Ġare Ġcapable Ġof Ġrepresenting Ġthe Ġcontent Ġof Ġtexts . ĠHowever , Ġdespite Ġthe Ġfact Ġthat Ġword Ġvectors Ġcan Ġbe Ġtreated Ġseparately , Ġconventional Ġmethods Ġsuch Ġas ĠS VM Ġand ĠL SA Ġmay Ġnot Ġbe Ġsuitable Ġfor Ġtext Ġclassification Ġusing Ġword Ġvectors . </s> Among Ġthe Ġconventional Ġmethods , ĠL SA Ġand ĠS VM Ġachieved Ġabout Ġ86 % Ġand Ġ89 %, Ġrespectively , Ġwhen Ġusing Ġbag - of - words Ġfeatures . ĠInterestingly , Ġboth Ġmethods Ġhad Ġbetter Ġperformance Ġwhen Ġusing Ġbinary Ġweights . ĠFor ĠL SA , Ġwe Ġcan Ġsee Ġthat Ġdespite Ġthe Ġslight Ġdifferences Ġin Ġthe Ġperformance , Ġtf id f B OW Ġrequired Ġapprox im ations Ġwith Ġsmaller Ġdimensions . ĠS VM Ġhad Ġthe Ġlowest Ġaccuracy Ġrate Ġwhen Ġusing Ġthe Ġtf id f B OW Ġfeatures . ĠOne Ġpossible Ġexplanation Ġfor Ġthis Ġis Ġthat ĠTF - ID F Ġweights Ġare Ġuseful Ġwhen Ġrare Ġwords Ġand Ġvery Ġfrequent Ġwords Ġexist Ġin Ġthe Ġcorpus , Ġgiving Ġhigher Ġweights Ġfor Ġrare Ġwords Ġand Ġlower Ġweights Ġfor Ġcommon Ġwords . ĠSince Ġwe Ġremoved Ġthe Ġstop Ġwords , Ġthe Ġmost Ġfrequent Ġwords Ġamong Ġthe Ġtraining Ġdocuments Ġwere Ġnot Ġconsidered Ġand , Ġtherefore , Ġusing ĠTF - ID F Ġweights Ġdid Ġnot Ġimprove Ġthe Ġresults . </s> Only ĠMN B Ġwith Ġtf B OW Ġperformed Ġbetter Ġthan ĠMSM . ĠThis Ġresult Ġmay Ġbe Ġbecause Ġtf B OW Ġfeatures Ġencode Ġthe Ġword Ġfrequencies , Ġwhile ĠMSM Ġonly Ġconsiders Ġa Ġsingle Ġoccurrence Ġof Ġwords . ĠWhen Ġincorporating Ġthe Ġword Ġfrequencies Ġwith Ġour ĠTF Ġweighted Ġword Ġsub space , Ġwe Ġachieved Ġa Ġhigher Ġaccuracy Ġof Ġ92 . 01 %, Ġperforming Ġbetter Ġthan ĠMN B Ġat Ġa Ġsignificance Ġlevel Ġof Ġ95 %. </s> Con clusions Ġand ĠFuture ĠWork </s> In Ġthis Ġpaper , Ġwe Ġproposed Ġa Ġnew Ġmethod Ġfor Ġtext Ġclassification , Ġbased Ġon Ġthe Ġnovel Ġconcept Ġof Ġword Ġsub space Ġunder Ġthe ĠMSM Ġframework . ĠWe Ġalso Ġproposed Ġthe Ġterm - frequency Ġweighted Ġword Ġsub space Ġwhich Ġcan Ġincorporate Ġthe Ġfrequency Ġof Ġwords Ġdirectly Ġin Ġthe Ġmodeling Ġof Ġthe Ġsub space Ġby Ġusing Ġa Ġweighted Ġversion Ġof ĠPC A . </s> Most Ġof Ġthe Ġconventional Ġtext Ġclassification Ġmethods Ġare Ġbased Ġon Ġthe Ġbag - of - words Ġfeatures , Ġwhich Ġare Ġvery Ġsimple Ġto Ġcompute Ġand Ġhad Ġbeen Ġproved Ġto Ġproduce Ġpositive Ġresults . ĠHowever , Ġbag - of - words Ġare Ġcommonly Ġhigh Ġdimensional Ġmodels , Ġwith Ġa Ġsparse Ġrepresentation , Ġwhich Ġis Ġcomput ationally Ġheavy Ġto Ġmodel . ĠAlso , Ġbag - of - words Ġfail Ġto Ġconvey Ġthe Ġsemantic Ġmeaning Ġof Ġwords Ġinside Ġa Ġtext . ĠDue Ġto Ġthese Ġproblems , Ġneural Ġnetworks Ġstarted Ġto Ġbe Ġapplied Ġto Ġgenerate Ġa Ġvector Ġrepresentation Ġof Ġwords . ĠDespite Ġthe Ġfact Ġthat Ġthese Ġrepresentations Ġcan Ġencode Ġthe Ġsemantic Ġmeaning Ġof Ġwords , Ġconventional Ġmethods Ġdo Ġnot Ġwork Ġwell Ġwhen Ġconsidering Ġword Ġvectors Ġseparately . </s> In Ġour Ġwork , Ġwe Ġfocused Ġon Ġthe Ġword 2 vec Ġrepresentation , Ġwhich Ġcan Ġembed Ġthe Ġsemantic Ġstructure Ġof Ġwords , Ġrendering Ġvector Ġangles Ġas Ġa Ġuseful Ġmetric Ġto Ġshow Ġmeaningful Ġsimilarities Ġbetween Ġwords . ĠOur Ġexperiments Ġshowed Ġthat Ġour Ġword Ġsub space Ġmodeling Ġalong Ġwith Ġthe ĠMSM Ġoutper forms Ġmost Ġof Ġthe Ġconventional Ġmethods . ĠUltimately , Ġour ĠTF Ġweighted Ġsub space Ġformulation Ġresulted Ġin Ġsignificantly Ġhigher Ġaccuracy Ġwhen Ġcompared Ġto Ġall Ġconventional Ġtext Ġclassification Ġmethods Ġdiscussed Ġin Ġthis Ġwork . ĠIt Ġis Ġimportant Ġto Ġnote Ġthat Ġour Ġmethod Ġdoes Ġnot Ġconsider Ġthe Ġorder Ġof Ġthe Ġwords Ġin Ġa Ġtext , Ġresulting Ġin Ġa Ġloss Ġof Ġcontext Ġinformation . ĠAs Ġa Ġfuture Ġwork , Ġwe Ġwish Ġto Ġextend Ġour Ġword Ġsub space Ġconcept Ġfurther Ġin Ġmainly Ġtwo Ġdirections . ĠFirst , Ġwe Ġseek Ġto Ġencode Ġword Ġorder , Ġwhich Ġmay Ġenrich Ġthe Ġrepresentation Ġof Ġcontext Ġinformation . ĠSecond , Ġwe Ġwish Ġto Ġmodel Ġdynamic Ġcontext Ġchange , Ġenabling Ġanalysis Ġof Ġlarge Ġdocuments , Ġby Ġhaving Ġa Ġlong - short Ġmemory Ġto Ġinterpret Ġinformation Ġusing Ġcues Ġfrom Ġdifferent Ġparts Ġof Ġa Ġtext . </s> Acknowled gment </s> This Ġwork Ġis Ġsupported Ġby ĠJS PS ĠK AK EN HI ĠGrant ĠNumber ĠJP 16 H 028 42 Ġand Ġthe ĠJapanese ĠMinistry Ġof ĠEducation , ĠCulture , ĠSports , ĠScience , Ġand ĠTechnology Ġ( M EXT ) Ġscholarship .
Introduction Text classification has become an indispensable task due to the rapid growth in the number of texts in digital form available online. It aims to classify different texts, also called documents, into a fixed number of predefined categories, helping to organize data, and making easier for users to find the desired information. Over the past three decades, many methods based on machine learning and statistical models have been applied to perform this task, such as latent semantic analysis (LSA), support vector machines (SVM), and multinomial naive Bayes (MNB). The first step in utilizing such methods to categorize textual data is to convert the texts into a vector representation. One of the most popular text representation models is the bag-of-words model BIBREF0 , which represents each document in a collection as a vector in a vector space. Each dimension of the vectors represents a term (e.g., a word, a sequence of words), and its value encodes a weight, which can be how many times the term occurs in the document. Despite showing positive results in tasks such as language modeling and classification BIBREF1 , BIBREF2 , BIBREF3 , the BOW representation has limitations: first, feature vectors are commonly very high-dimensional, resulting in sparse document representations, which are hard to model due to space and time complexity. Second, BOW does not consider the proximity of words and their position in the text and consequently cannot encode the words semantic meanings. To solve these problems, neural networks have been employed to learn vector representations of words BIBREF4 , BIBREF5 , BIBREF6 , BIBREF7 . In particular, the word2vec representation BIBREF8 has gained attention. Given a training corpus, word2vec can generate a vector for each word in the corpus that encodes its semantic information. These word vectors are distributed in such a way that words from similar contexts are represented by word vectors with high correlation, while words from different contexts are represented by word vectors with low correlation. One crucial aspect of the word2vec representation is that arithmetic and distance calculation between two word vectors can be performed, giving information about their semantic relationship. However, rather than looking at pairs of word vectors, we are interested in studying the relationship between sets of vectors as a whole and, therefore, it is desirable to have a text representation based on a set of these word vectors. To tackle this problem, we introduce the novel concept of word subspace. It is mathematically defined as a low dimensional linear subspace in a word vector space with high dimensionality. Given that words from texts of the same class belong to the same context, it is possible to model word vectors of each class as word subspaces and efficiently compare them in terms of similarity by using canonical angles between the word subspaces. Through this representation, most of the variability of the class is retained. Consequently, a word subspace can effectively and compactly represent the context of the corresponding text. We achieve this framework through the mutual subspace method (MSM) BIBREF9 . The word subspace of each text class is modeled by applying PCA without data centering to the set of word vectors of the class. When modeling the word subspaces, we assume only one occurrence of each word inside the class. However, as seen in the BOW approach, the frequency of words inside a text is an informative feature that should be considered. In order to introduce this feature in the word subspace modeling and enhance its performance, we further extend the concept of word subspace to the term-frequency (TF) weighted word subspace. In this extension, we consider a set of weights, which encodes the words frequencies, when performing the PCA. Text classification with TF weighted word subspace can also be performed under the framework of MSM. We show the validity of our modeling through experiments on the Reuters database, an established database for natural language processing tasks. We demonstrate the effectiveness of the word subspace formulation and its extension, comparing our methods' performance to various state-of-art methods. The main contributions of our work are: The remainder of this paper is organized as follows. In Section "Related Work" , we describe the main works related to text classification. In Section "Word subspace" , we present the formulation of our proposed word subspace. In Section "Conventional text classification methods" , we explain how text classification with word subspaces is performed under the MSM framework. Then, we present the TF weighted word subspace extension in Section "TF weighted word subspace" . Evaluation experiments and their results are described in Section "Experimental Evaluation" . Further discussion is then presented in Section "Discussion" , and our conclusions are described in Section "Conclusions and Future Work" . Related Work In this section, we outline relevant work towards text classification. We start by describing how text data is conventionally represented using the bag-of-words model and then follow to describe the conventional methods utilized in text classification. Text Representation with bag-of-words The bag-of-words representation comes from the hypothesis that frequencies of words in a document can indicate the relevance of the document to a query BIBREF0 , that is, if documents and a query have similar frequencies for the same words, they might have a similar meaning. This representation is based on the vector space model (VSM), that was developed for the SMART information retrieval system BIBREF10 . In the VSM, the main idea is that documents in a collection can be represented as a vector in a vector space, where vectors close to each other represent semantically similar documents. More formally, a document $d$ can be represented by a vector in $\mathbb {R}^{n}$ , where each dimension represents a different term. A term can be a single word, constituting the conventional bag-of-words, or combinations of $N$ words, constituting the bag-of-N-grams. If a term occurs in the document, its position in the vector will have a non-zero value, also known as term weight. Two documents in the VSM can be compared to each other by taking the cosine distance between them BIBREF1 . There are several ways to compute the term weights. Among them, we can highlight some: Binary weights, term-frequency (TF) weights, and term-frequency inverse document-frequency (TF-IDF) weights. Consider a corpus with documents $D = \lbrace d_i\rbrace _{i=1}^{|D|}$ and a vocabulary with all terms in the corpus $V = \lbrace w_i\rbrace _{i=1}^{|V|}$ . The term weights can be defined as: Binary weight: If a term occurs in the document, its weight is 1. Otherwise, it is zero. Term-frequency weight (TF): The weight of a term $w$ is defined by the number of times it occurs in the document $d$ . $$TF(w,d) = n_d^w$$   (Eq. 8) Inverse document-frequency: The weight of a term $w$ , given the corpus $D$ , is defined as the total number of documents $|D|$ divided by the number of documents that have the term $w$ , $|D^w|$ . $$IDF(w | D) = \frac{|D|}{|D^w|}$$   (Eq. 10) Term-frequency inverse document-frequency (TF-IDF): The weight of a term $w$ is defined by the multiplication of its term-frequency and its inverse document-frequency. When considering only the TF weights, all terms have the same importance among the corpus. By using the IDF weight, words that are more common across all documents in $D$ receive a smaller weight, giving more importance to rare terms in the corpus. $$TFIDF(w,d | D)=TF \times IDF$$   (Eq. 12) In very large corpus, it is common to consider the logarithm of the IDF in order to dampen its effect. $$TFIDF(w,d | D)=TF \times log_{10}(IDF)$$   (Eq. 13) Conventional text classification methods Multi-variate Bernoulli (MVB) and multinomial naive Bayes (MNB) are two generative models based on the naive Bayes assumption. In other words, they assume that all attributes (e.g., the frequency of each word, the presence or absence of a word) of each text are independent of each other given the context of the class BIBREF11 . In the MVB model, a document is represented by a vector generated by a bag-of-words with binary weights. In this case, a document can be considered an event, and the presence or the absence of the words to be the attributes of the event. On the other hand, the MNB model represents each document as a vector generated by a bag-of-words with TF weights. Here, the individual word occurrences are considered as events and the document is a collection of word events. Both these models use the Bayes rule to classify a document. Consider that each document should be classified into one of the classes in $C=\lbrace c_j\rbrace _{j=1}^{|C|}$ . The probability of each class given the document is defined as: $$P(c_j|d_i) = \frac{P(d_i|c_j)P(c_j)}{P(d_i)}.$$   (Eq. 16) The prior $P(d_i)$ is the same for all classes, so to determine the class to which $d_i$ belongs to, the following equation can be used: $$prediction(d_i) = argmax_{c_j}P(d_i|c_j)P(c_j)$$   (Eq. 17) The prior $P(c_j)$ can be obtained by the following equation: $$P(c_j) = \frac{1+|D_j|}{|C|+|D|},$$   (Eq. 18) where $|D_j|$ is the number of documents in class $c_j$ . As for the posterior $P(d_i|c_j)$ , different calculations are performed for each model. For MVB, it is defined as: $$P(d_i|c_j) = \prod _{k=1}^{|V|}P(w_k|c_j)^{t_i^k}(1-P(w_k|c_j))^{1-t_i^k},$$   (Eq. 19) where $w_k$ is the k-th word in the vocabulary $V$ , and $t_i^k$ is the value (0 or 1) of the k-th element of the vector of document $d_i$ . For the MNB, it is defined as: $$P(d_i|c_j) = P(|d_i|)|d_i|!\prod _{k=1}^{|V|}\frac{P(w_k|c_j)^{n_i^k}}{n_i^k!},$$   (Eq. 20) where $|d_i|$ is the number of words in document $d_i$ and $n_i^k$ is the k-th element of the vector of document $d_i$ and it represents how many times word $w_k$ occurs in $d_i$ . Finally, the posterior $P(w_k|c_j)$ can be obtained by the following equation: $$P(w_k|c_j) = \frac{1+|D_j^k|}{|C|+|D|},$$   (Eq. 21) where $|D_j^k|$ is the number of documents in class $c_j$ that contain the word $w_k$ . In general, MVB tends to perform better than MNB at small vocabulary sizes whereas MNB is more efficient on large vocabularies. Despite being robust tools for text classification, both these models depend directly on the bag-of-words features and do not naturally work with representations such as word2vec. Latent semantic analysis (LSA), or latent semantic indexing (LSI), was proposed in BIBREF12 , and it extends the vector space model by using singular value decomposition (SVD) to find a set of underlying latent variables which spans the meaning of texts. It is built from a term-document matrix, in which each row represents a term, and each column represents a document. This matrix can be built by concatenating the vectors of all documents in a corpus, obtained using the bag-of-words model, that is, $ {X} = [ {v}_1, {v}_2, ..., {v}_{|D|}]$ , where ${v}_i$ is the vector representation obtained using the bag-of-words model. In this method, the term-document matrix is decomposed using the singular value decomposition, $${X} = {U\Sigma V}^\top ,$$   (Eq. 23) where $U$ and $V$ are orthogonal matrices and correspond to the left singular vectors and right singular vectors of $X$ , respectively. $\Sigma $ is a diagonal matrix, and it contains the square roots of the eigenvalues of $X^TX$ and $XX^T$ . LSA finds a low-rank approximation of $X$ by selecting only the $k$ largest singular values and its respective singular vectors, $${X}_k = {U}_k{\Sigma }_k {V}_k^{\top }.$$   (Eq. 24) To compare two documents, we project both of them into this lower dimension space and calculate the cosine distance between them. The projection ${\hat{d}}$ of document ${d}$ is obtained by the following equation: $${\hat{d}} = {\Sigma }_k^{-1} {U}_k^\top {d}.$$   (Eq. 25) Despite its extensive application on text classification BIBREF13 , BIBREF14 , BIBREF15 , this method was initially proposed for document indexing and, therefore, does not encode any class information when modeling the low-rank approximation. To perform classification, 1-nearest neighbor is usually performed, placing a query document into the class of the nearest training document. The support vector machine (SVM) was first presented in BIBREF16 and performs the separation between samples of two different classes by projecting them onto a higher dimensionality space. It was first applied in text classification by BIBREF17 and have since been successfully applied in many tasks related to natural language processing BIBREF18 , BIBREF19 . Consider a training data set $D$ , with $n$ samples $$D = \lbrace ({x}_i,c_i)|{x}_i\in \mathbb {R}^p, c_i \in \lbrace -1,1\rbrace  \rbrace _{i=1}^{n},$$   (Eq. 27) where $c_i$ represents the class to which ${x}_i$ belongs to. Each ${x}_i$ is a $p$ -dimensional vector. The goal is to find the hyperplane that divides the points from $c_i = 1$ from the points from $c_i = -1$ . This hyperplane can be written as a set of points $x$ satisfying: $${w} \cdot {x} - b = 0,$$   (Eq. 28) where $\cdot $ denotes the dot product. The vector ${w}$ is perpendicular to the hyperplane. The parameter $\frac{b}{\Vert {w}\Vert }$ determines the offset of the hyperplane from the origin along the normal vector ${w}$ . We wish to choose ${w}$ and $b$ , so they maximize the distance between the parallel hyperplanes that are as far apart as possible, while still separating the data. If the training data is linearly separable, we can select two hyperplanes in a way that there are no points between them and then try to maximize the distance. In other words, minimize $\Vert {w}\Vert $ subject to $c_i({w}\cdot {x}_u-b) \ge 1, i=\lbrace 1,2,...,n\rbrace $ . If the training data is not linearly separable, the kernel trick can be applied, where every dot product is replaced by a non-linear kernel function. Word subspace All methods mentioned above utilize the BOW features to represent a document. Although this representation is simple and powerful, its main problem lies on disregarding the word semantics within a document, where the context and meaning could offer many benefits to the model such as identification of synonyms. In our formulation, words are represented as vectors in a real-valued feature vector space $\mathbb {R}^{p}$ , by using word2vec BIBREF8 . Through this representation, it is possible to calculate the distance between two words, where words from similar contexts are represented by vectors close to each other, while words from different contexts are represented as far apart vectors. Also, this representation brings the new concept of arithmetic operations between words, where operations such as addition and subtraction carry meaning (eg., “king”-“man”+“woman”=“queen”) BIBREF20 . Consider a set of documents which belong to the same context $D_c = \lbrace d_i\rbrace _{i=1}^{|D_c|}$ . Each document $d_i$ is represented by a set of $N_i$ words, $d_i = \lbrace w_k\rbrace _{k=1}^{N_i}$ . By considering that all words from documents of the same context belong to the same distribution, a set of words $W_c = \lbrace w_k\rbrace _{k=1}^{N_c}$ with the words in the context $c$ is obtained. We then translate these words into word vectors using word2vec, resulting in a set of word vectors $X_c = \lbrace {x}^k_c\rbrace _{k=1}^{N_c} \in \mathbb {R}^p$ . This set of word vectors is modeled into a word subspace, which is a compact, scalable and meaningful representation of the whole set. Such a word subspace is generated by applying PCA to the set of word vectors. First, we compute an autocorrelation matrix, ${R}_c$ : $${R}_c = \frac{1}{N_c}\sum _{i=1}^{N_c}{x}^{i}_c{x}_c^{i^{\top }}.$$   (Eq. 29) The orthonormal basis vectors of $m_c$ -dimensional subspace ${Y}_c$ are obtained as the eigenvectors with the $m_c$ largest eigenvalues of the matrix ${R}_c$ . We represent a subspace ${Y}_c$ by the matrix ${Y}_c \in \mathbb {R}^{p \times m_c}$ , which has the corresponding orthonormal basis vectors as its column vectors. Text classification based on word subspace We formulate our problem as a single label classification problem. Given a set of training documents, which we will refer as corpus, $D = \lbrace d_i\rbrace _{i=1}^{|D|}$ , with known classes $C = \lbrace c_j\rbrace _{j=1}^{|C|}$ , we wish to classify a query document $d_q$ into one of the classes in $C$ . Text classification based on word subspace can be performed under the framework of mutual subspace method (MSM). This task involves two different stages: A learning stage, where the word subspace for each class is modeled, and a classification stage, where the word subspace for a query is modeled and compared to the word subspaces of the classes. In the learning stage, it is assumed that all documents of the same class belong to the same context, resulting in a set of words $W_c = \lbrace w_c^k\rbrace _{k=1}^{N_c}$ . This set assumes that each word appears only once in each class. Each set $\lbrace W_c\rbrace _{c=1}^{|C|}$ is then modeled into a word subspace ${Y}_c$ , as explained in Section "Word subspace" . As the number of words in each class may vary largely, the dimension $m_c$ of each class word subspace is not set to the same value. In the classification stage, for a query document $d_q$ , it is also assumed that each word occurs only once, generating a subspace ${Y}_q$ . To measure the similarity between a class word subspace ${Y}_c$ and a query word subspace ${Y}_q$ , the canonical angles between the two word subspaces are used BIBREF21 . There are several methods for calculating canonical angles BIBREF22 , BIBREF23 , and BIBREF24 , among which the simplest and most practical is the singular value decomposition (SVD). Consider, for example, two subspaces, one from the training data and another from the query, represented as matrices of bases, ${Y}_{c} = [{\Phi }_{1} \ldots {\Phi }_{m_c}] \in \mathbb {R}^{p \times m_c}$ and ${Y}_{q} = [{\Psi }_{1} \ldots {\Psi }_{m_q}] \in \mathbb {R}^{p \times m_q}$ , where ${\Phi }_{i}$ are the bases for ${Y}_c$ and ${\Psi }_{i}$ are the bases for ${Y}_q$ . Let the SVD of ${Y}_c^{\top }{Y}_q \in \mathbb {R}^{m_c \times m_q}$ be ${Y}_c^{\top }{Y}_q = {U \Sigma V}^{\top }$ , where ${Y}_q$0 , ${Y}_q$1 represents the set of singular values. The canonical angles ${Y}_q$2 can be obtained as ${Y}_q$3 ${Y}_q$4 . The similarity between the two subspaces is measured by ${Y}_q$5 angles as follows: $$S_{({Y}_c,{Y}_q)}[t] = \frac{1}{t}\sum _{i = 1}^{t} \cos ^{2} \theta _{i},\; 1 \le t \le m_q, \; m_q \le m_c.$$   (Eq. 30) Fig. 1 shows the modeling and comparison of sets of words by MSM. This method can compare sets of different sizes, and naturally encodes proximity between sets with related words. Finally, the class with the highest similarity with $d_q$ is assigned as the class of $d_q$ : $$prediction(d_q) = argmax_c(S_{({Y}_c,{Y}_q)}).$$   (Eq. 32) TF weighted word subspace The word subspace formulation presented in Section "Word subspace" is a practical and compact way to represent sets of word vectors, retaining most of the variability of features. However, as seen in the BOW features, the frequency of words is relevant information that can improve the characterization of a text. To incorporate this information into the word subspace modeling, we propose an extension of the word subspace, called the term-frequency (TF) weighted word subspace. Like the word subspace, the TF weighted word subspace is mathematically defined as a low-dimensional linear subspace in a word vector space with high dimensionality. However, a weighted version of the PCA BIBREF25 , BIBREF26 is utilized to incorporate the information given by the frequencies of words (term-frequencies). This TF weighted word subspace is equivalent to the word subspace if we consider all occurrences of the words. Consider the set of word vectors $\lbrace {x}_c^k\rbrace _{k=1}^{N_c} \in \mathbb {R}^{p}$ , which represents each word in the context $c$ , and the set of weights $\lbrace \omega _i\rbrace _{i=1}^{N_c}$ , which represent the frequencies of the words in the context $c$ . We incorporate these frequencies into the subspace calculation by weighting the data matrix ${X}$ as follows: $${\widetilde{X}}={X}{\Omega }^{1/2},$$   (Eq. 33) where ${X} \in \mathbb {R}^{p \times N_c}$ is a matrix containing the word vectors $\lbrace {x}_c^k\rbrace _{k=1}^{N_c}$ and ${\Omega }$ is a diagonal matrix containing the weights $\lbrace \omega _i\rbrace _{i=1}^{N_c}$ . We then perform PCA by solving the SVD of the matrix ${\widetilde{X}}$ : $${\widetilde{X}}={AMB}^{\top },$$   (Eq. 34) where the columns of the orthogonal matrices ${A}$ and ${B}$ are, respectively, the left-singular vectors and right-singular vectors of the matrix ${\widetilde{X}}$ , and the diagonal matrix ${M}$ contains singular values of ${\widetilde{X}}$ . Finally, the orthonormal basis vectors of the $m_c$ -dimensional TF weighted subspace ${W}$ are the column vectors in ${A}$ corresponding to the $m_c$ largest singular values in ${M}$ . Text classification with TF weighted word subspace can also be performed under the framework of MSM. In this paper, we will refer to MSM with TF weighted word subspace as TF-MSM. Experimental Evaluation In this section we describe the experiments performed to demonstrate the validity of our proposed method and its extension. We used the Reuters-8 dataset without stop words from BIBREF27 aiming at single-label classification, which is a preprocessed format of the Reuters-21578. Words in the texts were considered as they appeared, without performing stemming or typo correction. This database has eight different classes with the number of samples varying from 51 to over 3000 documents, as can be seen in Table 1 . To obtain the vector representation of words, we used a freely available word2vec model, trained by BIBREF8 , on approximately 100 billion words, which encodes the vector representation in $\mathbb {R}^{300}$ of over 3 million words from several different languages. Since we decided to focus on English words only, we filtered these vectors to about 800 thousand words, excluding all words with non-roman characters. To show the validity of our word subspace representation for text classification and the proposed extension, we divided our experiment section into two parts: The first one aims to verify if sets of word vectors are suitable for subspace representation, and the second one puts our methods in practice in a text classification test, comparing our results with the conventional methods described in Section "Related Work" . Evaluation of the word subspace representation In this experiment, we modeled the word vectors from each class in the Reuters-8 database into a word subspace. The primary goal is to visualize how much of the text data can be represented by a lower dimensional subspace. Subspace representations are very efficient in compactly represent data that is close to a normal distribution. This characteristic is due to the application of the PCA, that is optimal to find the direction with the highest variation within the data. In PCA, the principal components give the directions of maximum variance, while their corresponding eigenvalues give the variance of the data in each of them. Therefore, by observing the distribution of the eigenvalues computed when performing PCA in the modeling of the subspaces, we can suggest if the data is suitable or not for subspace representation. For each class, we normalized the eigenvalues by the largest one of the class. Fig. 2 shows the mean of the eigenvalues and the standard deviation among classes. It is possible to see that the first largest eigenvalues retain larger variance than the smallest ones. In fact, looking at the first 150 largest eigenvalues, we can see that they retain, on average, 86.37% of the data variance. Also, by observing the standard deviation, we can understand that the eigenvalues distribution among classes follows the same pattern, that is, most of the variance is in the first dimensions. This plot indicates that text data represented by vectors generated with word2vec is suitable for subspace representation. Text classification experiment In this experiment, we performed text classification among the classes in the Reuters-8 database. We compared the classification using the word subspace, and its weighted extension, based on MSM (to which we will refer as MSM and TF-MSM, respectively) with the baselines presented in Section "Related Work" : MVB, MNB, LSA, and SVM. Since none of the baseline methods work with vector set classification, we also compared to a simple baseline for comparing sets of vectors, defined as the average of similarities between all vector pair combinations of two given sets. For two matrices ${A}$ and ${B}$ , containing the sets of vectors $\lbrace  {x}^{i}_a \rbrace _{i = 1}^{N_A}$ and $\lbrace  {x}^{i}_b \rbrace _{i = 1}^{N_B}$ , respectively, where $N_A$ and $N_B$ are the number of main words in each set, the similarity is defined as: $$Sim_{(A,B)} = \frac{1}{N_A N_B}\sum _{i}^{N_A}\sum _{j}^{N_B}{{x}_a^i}^{\top }{x}_b^j.$$   (Eq. 41) We refer to this baseline as similarity average (SA). For this method, we only considered one occurrence of each word in each set. Different features were used, depending on the method. Classification with SA, MSM, and TF-MSM was performed using word2vec features, to which we refer as w2v. For MVB, due to its nature, only bag-of-words features with binary weights were used (binBOW). For the same reason, we only used bag-of-words features with term-frequency weights (tfBOW) with MNB. Classification with LSA and SVM is usually performed using bag-of-words features and, therefore, we tested with binBOW, tfBOW, and with the term-frequency inverse document-frequency weight, tfidfBOW. We also tested them using word2vec vectors. In this case, we considered each word vector from all documents in each class to be a single sample. To determine the dimensions of the class subspaces and query subspace of MSM and TF-MSM, and the dimension of the approximation performed by LSA, we performed a 10-fold cross validation, wherein each fold, the data were randomly divided into train (60%), validation (20%) and test set (20%). The results can be seen in Table 2 . The simplest baseline, SA with w2v, achieved an accuracy rate of 78.73%. This result is important because it shows the validity of the word2vec representation, performing better than more elaborate methods based on BOW, such as MVB with binBOW. LSA with BOW features was almost 10% more accurate than SA, where the best results with binary weights were achieved with an approximation with 130 dimensions, with TF weights were achieved with 50 dimensions, and with TF-IDF weights were achieved with 30 dimensions. SVM with BOW features was about 3% more accurate than LSA, with binary weights leading to a higher accuracy rate. It is interesting to note that despite the reasonably high accuracy rates achieved using LSA and SVM with BOW features, they poorly performed when using w2v features. Among the baselines, the best method was MNB with tfBOW features, with an accuracy of 91.47%, being the only conventional method to outperform MSM. MSM with w2v had an accuracy rate of 90.62%, with the best results achieved with word subspace dimensions for the training classes ranging from 150 to 181, and for the query ranging from 3 to 217. Incorporating the frequency information in the subspace modeling resulted in higher accuracy, with TF-MSM achieving 92.01%, with dimensions of word subspaces for training classes ranging from 150 to 172, and for the query, ranging from 2 to 109. To confirm that TF-MSM is significantly more accurate than MNB, we performed a t-test to compare their results. It resulted in a p-value of 0.031, which shows that at a 95% significance level, TF-MSM has produced better results. Discussion Given the observation of the eigenvalues distribution of word vectors, we could see that word vectors that belong to the same context, i.e., same class, are suitable for subspace representation. Our analysis showed that half of the word vector space dimensions suffice to represent most of the variability of the data in each class of the Reuters-8 database. The results from the text classification experiment showed that subspace-based methods performed better than the text classification methods discussed in this work. Ultimately, our proposed TF weighted word subspace with MSM surpassed all the other methods. word2vec features are reliable tools to represent the semantic meaning of the words and when treated as sets of word vectors, they are capable of representing the content of texts. However, despite the fact that word vectors can be treated separately, conventional methods such as SVM and LSA may not be suitable for text classification using word vectors. Among the conventional methods, LSA and SVM achieved about 86% and 89%, respectively, when using bag-of-words features. Interestingly, both methods had better performance when using binary weights. For LSA, we can see that despite the slight differences in the performance, tfidfBOW required approximations with smaller dimensions. SVM had the lowest accuracy rate when using the tfidfBOW features. One possible explanation for this is that TF-IDF weights are useful when rare words and very frequent words exist in the corpus, giving higher weights for rare words and lower weights for common words. Since we removed the stop words, the most frequent words among the training documents were not considered and, therefore, using TF-IDF weights did not improve the results. Only MNB with tfBOW performed better than MSM. This result may be because tfBOW features encode the word frequencies, while MSM only considers a single occurrence of words. When incorporating the word frequencies with our TF weighted word subspace, we achieved a higher accuracy of 92.01%, performing better than MNB at a significance level of 95%. Conclusions and Future Work In this paper, we proposed a new method for text classification, based on the novel concept of word subspace under the MSM framework. We also proposed the term-frequency weighted word subspace which can incorporate the frequency of words directly in the modeling of the subspace by using a weighted version of PCA. Most of the conventional text classification methods are based on the bag-of-words features, which are very simple to compute and had been proved to produce positive results. However, bag-of-words are commonly high dimensional models, with a sparse representation, which is computationally heavy to model. Also, bag-of-words fail to convey the semantic meaning of words inside a text. Due to these problems, neural networks started to be applied to generate a vector representation of words. Despite the fact that these representations can encode the semantic meaning of words, conventional methods do not work well when considering word vectors separately. In our work, we focused on the word2vec representation, which can embed the semantic structure of words, rendering vector angles as a useful metric to show meaningful similarities between words. Our experiments showed that our word subspace modeling along with the MSM outperforms most of the conventional methods. Ultimately, our TF weighted subspace formulation resulted in significantly higher accuracy when compared to all conventional text classification methods discussed in this work. It is important to note that our method does not consider the order of the words in a text, resulting in a loss of context information. As a future work, we wish to extend our word subspace concept further in mainly two directions. First, we seek to encode word order, which may enrich the representation of context information. Second, we wish to model dynamic context change, enabling analysis of large documents, by having a long-short memory to interpret information using cues from different parts of a text. Acknowledgment This work is supported by JSPS KAKENHI Grant Number JP16H02842 and the Japanese Ministry of Education, Culture, Sports, Science, and Technology (MEXT) scholarship.
Which dataset has been used in this work?
440faf8d0af8291d324977ad0f68c8d661fe365e
Which Ġdataset Ġhas Ġbeen Ġused Ġin Ġthis Ġwork ?
Reuters - 8 Ġdataset Ġwithout Ġstop Ġwords
<s> Which Ġdataset Ġhas Ġbeen Ġused Ġin Ġthis Ġwork ? </s> Introduction </s> Text Ġclassification Ġhas Ġbecome Ġan Ġindispensable Ġtask Ġdue Ġto Ġthe Ġrapid Ġgrowth Ġin Ġthe Ġnumber Ġof Ġtexts Ġin Ġdigital Ġform Ġavailable Ġonline . ĠIt Ġaims Ġto Ġclassify Ġdifferent Ġtexts , Ġalso Ġcalled Ġdocuments , Ġinto Ġa Ġfixed Ġnumber Ġof Ġpred efined Ġcategories , Ġhelping Ġto Ġorganize Ġdata , Ġand Ġmaking Ġeasier Ġfor Ġusers Ġto Ġfind Ġthe Ġdesired Ġinformation . ĠOver Ġthe Ġpast Ġthree Ġdecades , Ġmany Ġmethods Ġbased Ġon Ġmachine Ġlearning Ġand Ġstatistical Ġmodels Ġhave Ġbeen Ġapplied Ġto Ġperform Ġthis Ġtask , Ġsuch Ġas Ġlatent Ġsemantic Ġanalysis Ġ( L SA ), Ġsupport Ġvector Ġmachines Ġ( S VM ), Ġand Ġmult in omial Ġnaive ĠBay es Ġ( M NB ). </s> The Ġfirst Ġstep Ġin Ġutilizing Ġsuch Ġmethods Ġto Ġcategor ize Ġtextual Ġdata Ġis Ġto Ġconvert Ġthe Ġtexts Ġinto Ġa Ġvector Ġrepresentation . ĠOne Ġof Ġthe Ġmost Ġpopular Ġtext Ġrepresentation Ġmodels Ġis Ġthe Ġbag - of - words Ġmodel ĠB IB REF 0 Ġ, Ġwhich Ġrepresents Ġeach Ġdocument Ġin Ġa Ġcollection Ġas Ġa Ġvector Ġin Ġa Ġvector Ġspace . ĠEach Ġdimension Ġof Ġthe Ġvectors Ġrepresents Ġa Ġterm Ġ( e . g ., Ġa Ġword , Ġa Ġsequence Ġof Ġwords ), Ġand Ġits Ġvalue Ġenc odes Ġa Ġweight , Ġwhich Ġcan Ġbe Ġhow Ġmany Ġtimes Ġthe Ġterm Ġoccurs Ġin Ġthe Ġdocument . </s> Despite Ġshowing Ġpositive Ġresults Ġin Ġtasks Ġsuch Ġas Ġlanguage Ġmodeling Ġand Ġclassification ĠB IB REF 1 Ġ, ĠB IB REF 2 Ġ, ĠB IB REF 3 Ġ, Ġthe ĠB OW Ġrepresentation Ġhas Ġlimitations : Ġfirst , Ġfeature Ġvectors Ġare Ġcommonly Ġvery Ġhigh - dimensional , Ġresulting Ġin Ġsparse Ġdocument Ġrepresentations , Ġwhich Ġare Ġhard Ġto Ġmodel Ġdue Ġto Ġspace Ġand Ġtime Ġcomplexity . ĠSecond , ĠB OW Ġdoes Ġnot Ġconsider Ġthe Ġproximity Ġof Ġwords Ġand Ġtheir Ġposition Ġin Ġthe Ġtext Ġand Ġconsequently Ġcannot Ġencode Ġthe Ġwords Ġsemantic Ġmeanings . </s> To Ġsolve Ġthese Ġproblems , Ġneural Ġnetworks Ġhave Ġbeen Ġemployed Ġto Ġlearn Ġvector Ġrepresentations Ġof Ġwords ĠB IB REF 4 Ġ, ĠB IB REF 5 Ġ, ĠB IB REF 6 Ġ, ĠB IB REF 7 Ġ. ĠIn Ġparticular , Ġthe Ġword 2 vec Ġrepresentation ĠB IB REF 8 Ġhas Ġgained Ġattention . ĠGiven Ġa Ġtraining Ġcorpus , Ġword 2 vec Ġcan Ġgenerate Ġa Ġvector Ġfor Ġeach Ġword Ġin Ġthe Ġcorpus Ġthat Ġenc odes Ġits Ġsemantic Ġinformation . ĠThese Ġword Ġvectors Ġare Ġdistributed Ġin Ġsuch Ġa Ġway Ġthat Ġwords Ġfrom Ġsimilar Ġcontexts Ġare Ġrepresented Ġby Ġword Ġvectors Ġwith Ġhigh Ġcorrelation , Ġwhile Ġwords Ġfrom Ġdifferent Ġcontexts Ġare Ġrepresented Ġby Ġword Ġvectors Ġwith Ġlow Ġcorrelation . </s> One Ġcrucial Ġaspect Ġof Ġthe Ġword 2 vec Ġrepresentation Ġis Ġthat Ġarithmetic Ġand Ġdistance Ġcalculation Ġbetween Ġtwo Ġword Ġvectors Ġcan Ġbe Ġperformed , Ġgiving Ġinformation Ġabout Ġtheir Ġsemantic Ġrelationship . ĠHowever , Ġrather Ġthan Ġlooking Ġat Ġpairs Ġof Ġword Ġvectors , Ġwe Ġare Ġinterested Ġin Ġstudying Ġthe Ġrelationship Ġbetween Ġsets Ġof Ġvectors Ġas Ġa Ġwhole Ġand , Ġtherefore , Ġit Ġis Ġdesirable Ġto Ġhave Ġa Ġtext Ġrepresentation Ġbased Ġon Ġa Ġset Ġof Ġthese Ġword Ġvectors . </s> To Ġtackle Ġthis Ġproblem , Ġwe Ġintroduce Ġthe Ġnovel Ġconcept Ġof Ġword Ġsub space . ĠIt Ġis Ġmat hematically Ġdefined Ġas Ġa Ġlow Ġdimensional Ġlinear Ġsub space Ġin Ġa Ġword Ġvector Ġspace Ġwith Ġhigh Ġdimension ality . ĠGiven Ġthat Ġwords Ġfrom Ġtexts Ġof Ġthe Ġsame Ġclass Ġbelong Ġto Ġthe Ġsame Ġcontext , Ġit Ġis Ġpossible Ġto Ġmodel Ġword Ġvectors Ġof Ġeach Ġclass Ġas Ġword Ġsub sp aces Ġand Ġefficiently Ġcompare Ġthem Ġin Ġterms Ġof Ġsimilarity Ġby Ġusing Ġcanonical Ġangles Ġbetween Ġthe Ġword Ġsub sp aces . ĠThrough Ġthis Ġrepresentation , Ġmost Ġof Ġthe Ġvariability Ġof Ġthe Ġclass Ġis Ġretained . ĠConsequently , Ġa Ġword Ġsub space Ġcan Ġeffectively Ġand Ġcompact ly Ġrepresent Ġthe Ġcontext Ġof Ġthe Ġcorresponding Ġtext . ĠWe Ġachieve Ġthis Ġframework Ġthrough Ġthe Ġmutual Ġsub space Ġmethod Ġ( MS M ) ĠB IB REF 9 Ġ. </s> The Ġword Ġsub space Ġof Ġeach Ġtext Ġclass Ġis Ġmodeled Ġby Ġapplying ĠPC A Ġwithout Ġdata Ġcent ering Ġto Ġthe Ġset Ġof Ġword Ġvectors Ġof Ġthe Ġclass . ĠWhen Ġmodeling Ġthe Ġword Ġsub sp aces , Ġwe Ġassume Ġonly Ġone Ġoccurrence Ġof Ġeach Ġword Ġinside Ġthe Ġclass . </s> However , Ġas Ġseen Ġin Ġthe ĠB OW Ġapproach , Ġthe Ġfrequency Ġof Ġwords Ġinside Ġa Ġtext Ġis Ġan Ġinformative Ġfeature Ġthat Ġshould Ġbe Ġconsidered . ĠIn Ġorder Ġto Ġintroduce Ġthis Ġfeature Ġin Ġthe Ġword Ġsub space Ġmodeling Ġand Ġenhance Ġits Ġperformance , Ġwe Ġfurther Ġextend Ġthe Ġconcept Ġof Ġword Ġsub space Ġto Ġthe Ġterm - frequency Ġ( TF ) Ġweighted Ġword Ġsub space . </s> In Ġthis Ġextension , Ġwe Ġconsider Ġa Ġset Ġof Ġweights , Ġwhich Ġenc odes Ġthe Ġwords Ġfrequencies , Ġwhen Ġperforming Ġthe ĠPC A . ĠText Ġclassification Ġwith ĠTF Ġweighted Ġword Ġsub space Ġcan Ġalso Ġbe Ġperformed Ġunder Ġthe Ġframework Ġof ĠMSM . ĠWe Ġshow Ġthe Ġvalidity Ġof Ġour Ġmodeling Ġthrough Ġexperiments Ġon Ġthe ĠReuters Ġdatabase , Ġan Ġestablished Ġdatabase Ġfor Ġnatural Ġlanguage Ġprocessing Ġtasks . ĠWe Ġdemonstrate Ġthe Ġeffectiveness Ġof Ġthe Ġword Ġsub space Ġformulation Ġand Ġits Ġextension , Ġcomparing Ġour Ġmethods ' Ġperformance Ġto Ġvarious Ġstate - of - art Ġmethods . </s> The Ġmain Ġcontributions Ġof Ġour Ġwork Ġare : </s> The Ġremainder Ġof Ġthis Ġpaper Ġis Ġorganized Ġas Ġfollows . ĠIn ĠSection Ġ" Related ĠWork " Ġ, Ġwe Ġdescribe Ġthe Ġmain Ġworks Ġrelated Ġto Ġtext Ġclassification . ĠIn ĠSection Ġ" Word Ġsub space " Ġ, Ġwe Ġpresent Ġthe Ġformulation Ġof Ġour Ġproposed Ġword Ġsub space . ĠIn ĠSection Ġ" Con ventional Ġtext Ġclassification Ġmethods " Ġ, Ġwe Ġexplain Ġhow Ġtext Ġclassification Ġwith Ġword Ġsub sp aces Ġis Ġperformed Ġunder Ġthe ĠMSM Ġframework . ĠThen , Ġwe Ġpresent Ġthe ĠTF Ġweighted Ġword Ġsub space Ġextension Ġin ĠSection Ġ" TF Ġweighted Ġword Ġsub space " Ġ. ĠEvaluation Ġexperiments Ġand Ġtheir Ġresults Ġare Ġdescribed Ġin ĠSection Ġ" Exper imental ĠEvaluation " Ġ. ĠFurther Ġdiscussion Ġis Ġthen Ġpresented Ġin ĠSection Ġ" Discussion " Ġ, Ġand Ġour Ġconclusions Ġare Ġdescribed Ġin ĠSection Ġ" Con clusions Ġand ĠFuture ĠWork " Ġ. </s> Related ĠWork </s> In Ġthis Ġsection , Ġwe Ġoutline Ġrelevant Ġwork Ġtowards Ġtext Ġclassification . ĠWe Ġstart Ġby Ġdescribing Ġhow Ġtext Ġdata Ġis Ġconvention ally Ġrepresented Ġusing Ġthe Ġbag - of - words Ġmodel Ġand Ġthen Ġfollow Ġto Ġdescribe Ġthe Ġconventional Ġmethods Ġutilized Ġin Ġtext Ġclassification . </s> Text ĠRepresent ation Ġwith Ġbag - of - words </s> The Ġbag - of - words Ġrepresentation Ġcomes Ġfrom Ġthe Ġhypothesis Ġthat Ġfrequencies Ġof Ġwords Ġin Ġa Ġdocument Ġcan Ġindicate Ġthe Ġrelevance Ġof Ġthe Ġdocument Ġto Ġa Ġquery ĠB IB REF 0 Ġ, Ġthat Ġis , Ġif Ġdocuments Ġand Ġa Ġquery Ġhave Ġsimilar Ġfrequencies Ġfor Ġthe Ġsame Ġwords , Ġthey Ġmight Ġhave Ġa Ġsimilar Ġmeaning . ĠThis Ġrepresentation Ġis Ġbased Ġon Ġthe Ġvector Ġspace Ġmodel Ġ( V SM ), Ġthat Ġwas Ġdeveloped Ġfor Ġthe ĠSM ART Ġinformation Ġretrieval Ġsystem ĠB IB REF 10 Ġ. ĠIn Ġthe ĠV SM , Ġthe Ġmain Ġidea Ġis Ġthat Ġdocuments Ġin Ġa Ġcollection Ġcan Ġbe Ġrepresented Ġas Ġa Ġvector Ġin Ġa Ġvector Ġspace , Ġwhere Ġvectors Ġclose Ġto Ġeach Ġother Ġrepresent Ġsem antically Ġsimilar Ġdocuments . </s> More Ġformally , Ġa Ġdocument Ġ$ d $ Ġcan Ġbe Ġrepresented Ġby Ġa Ġvector Ġin Ġ$\ math bb Ġ{ R } ^{ n } $ Ġ, Ġwhere Ġeach Ġdimension Ġrepresents Ġa Ġdifferent Ġterm . ĠA Ġterm Ġcan Ġbe Ġa Ġsingle Ġword , Ġconstit uting Ġthe Ġconventional Ġbag - of - words , Ġor Ġcombinations Ġof Ġ$ N $ Ġwords , Ġconstit uting Ġthe Ġbag - of - N - gram s . ĠIf Ġa Ġterm Ġoccurs Ġin Ġthe Ġdocument , Ġits Ġposition Ġin Ġthe Ġvector Ġwill Ġhave Ġa Ġnon - zero Ġvalue , Ġalso Ġknown Ġas Ġterm Ġweight . ĠTwo Ġdocuments Ġin Ġthe ĠV SM Ġcan Ġbe Ġcompared Ġto Ġeach Ġother Ġby Ġtaking Ġthe Ġcos ine Ġdistance Ġbetween Ġthem ĠB IB REF 1 Ġ. </s> There Ġare Ġseveral Ġways Ġto Ġcompute Ġthe Ġterm Ġweights . ĠAmong Ġthem , Ġwe Ġcan Ġhighlight Ġsome : ĠBinary Ġweights , Ġterm - frequency Ġ( TF ) Ġweights , Ġand Ġterm - frequency Ġinverse Ġdocument - frequency Ġ( TF - ID F ) Ġweights . </s> Consider Ġa Ġcorpus Ġwith Ġdocuments Ġ$ D Ġ= Ġ\ l brace Ġd _ i \ r brace Ġ_ { i = 1 } ^{ | D | } $ Ġand Ġa Ġvocabulary Ġwith Ġall Ġterms Ġin Ġthe Ġcorpus Ġ$ V Ġ= Ġ\ l brace Ġw _ i \ r brace Ġ_ { i = 1 } ^{ | V | } $ Ġ. ĠThe Ġterm Ġweights Ġcan Ġbe Ġdefined Ġas : </s> B inary Ġweight : ĠIf Ġa Ġterm Ġoccurs Ġin Ġthe Ġdocument , Ġits Ġweight Ġis Ġ1 . ĠOtherwise , Ġit Ġis Ġzero . </s> Term - frequency Ġweight Ġ( TF ): ĠThe Ġweight Ġof Ġa Ġterm Ġ$ w $ Ġis Ġdefined Ġby Ġthe Ġnumber Ġof Ġtimes Ġit Ġoccurs Ġin Ġthe Ġdocument Ġ$ d $ Ġ. </s> $$ TF ( w , d ) Ġ= Ġn _ d ^ w $$ Ġ Ġ Ġ( E q . Ġ8 ) </s> In verse Ġdocument - frequency : ĠThe Ġweight Ġof Ġa Ġterm Ġ$ w $ Ġ, Ġgiven Ġthe Ġcorpus Ġ$ D $ Ġ, Ġis Ġdefined Ġas Ġthe Ġtotal Ġnumber Ġof Ġdocuments Ġ$ | D | $ Ġdivided Ġby Ġthe Ġnumber Ġof Ġdocuments Ġthat Ġhave Ġthe Ġterm Ġ$ w $ Ġ, Ġ$ | D ^ w | $ Ġ. </s> $$ ID F ( w Ġ| ĠD ) Ġ= Ġ\ frac { | D | }{ | D ^ w | } $$ Ġ Ġ Ġ( E q . Ġ10 ) </s> Term - frequency Ġinverse Ġdocument - frequency Ġ( TF - ID F ): ĠThe Ġweight Ġof Ġa Ġterm Ġ$ w $ Ġis Ġdefined Ġby Ġthe Ġmultiplication Ġof Ġits Ġterm - frequency Ġand Ġits Ġinverse Ġdocument - frequency . ĠWhen Ġconsidering Ġonly Ġthe ĠTF Ġweights , Ġall Ġterms Ġhave Ġthe Ġsame Ġimportance Ġamong Ġthe Ġcorpus . ĠBy Ġusing Ġthe ĠIDF Ġweight , Ġwords Ġthat Ġare Ġmore Ġcommon Ġacross Ġall Ġdocuments Ġin Ġ$ D $ Ġreceive Ġa Ġsmaller Ġweight , Ġgiving Ġmore Ġimportance Ġto Ġrare Ġterms Ġin Ġthe Ġcorpus . </s> $$ TF ID F ( w , d Ġ| ĠD )= TF Ġ\ times ĠIDF $$ Ġ Ġ Ġ( E q . Ġ12 ) </s> In Ġvery Ġlarge Ġcorpus , Ġit Ġis Ġcommon Ġto Ġconsider Ġthe Ġlog ar ith m Ġof Ġthe ĠIDF Ġin Ġorder Ġto Ġdamp en Ġits Ġeffect . </s> $$ TF ID F ( w , d Ġ| ĠD )= TF Ġ\ times Ġlog _{ 10 } ( ID F ) $$ Ġ Ġ Ġ( E q . Ġ13 ) </s> Con ventional Ġtext Ġclassification Ġmethods </s> Multi - vari ate ĠBern ou lli Ġ( M VB ) Ġand Ġmult in omial Ġnaive ĠBay es Ġ( M NB ) Ġare Ġtwo Ġgener ative Ġmodels Ġbased Ġon Ġthe Ġnaive ĠBay es Ġassumption . ĠIn Ġother Ġwords , Ġthey Ġassume Ġthat Ġall Ġattributes Ġ( e . g ., Ġthe Ġfrequency Ġof Ġeach Ġword , Ġthe Ġpresence Ġor Ġabsence Ġof Ġa Ġword ) Ġof Ġeach Ġtext Ġare Ġindependent Ġof Ġeach Ġother Ġgiven Ġthe Ġcontext Ġof Ġthe Ġclass ĠB IB REF 11 Ġ. </s> In Ġthe ĠMV B Ġmodel , Ġa Ġdocument Ġis Ġrepresented Ġby Ġa Ġvector Ġgenerated Ġby Ġa Ġbag - of - words Ġwith Ġbinary Ġweights . ĠIn Ġthis Ġcase , Ġa Ġdocument Ġcan Ġbe Ġconsidered Ġan Ġevent , Ġand Ġthe Ġpresence Ġor Ġthe Ġabsence Ġof Ġthe Ġwords Ġto Ġbe Ġthe Ġattributes Ġof Ġthe Ġevent . ĠOn Ġthe Ġother Ġhand , Ġthe ĠMN B Ġmodel Ġrepresents Ġeach Ġdocument Ġas Ġa Ġvector Ġgenerated Ġby Ġa Ġbag - of - words Ġwith ĠTF Ġweights . ĠHere , Ġthe Ġindividual Ġword Ġoccurrences Ġare Ġconsidered Ġas Ġevents Ġand Ġthe Ġdocument Ġis Ġa Ġcollection Ġof Ġword Ġevents . </s> Both Ġthese Ġmodels Ġuse Ġthe ĠBay es Ġrule Ġto Ġclassify Ġa Ġdocument . ĠConsider Ġthat Ġeach Ġdocument Ġshould Ġbe Ġclassified Ġinto Ġone Ġof Ġthe Ġclasses Ġin Ġ$ C = \ l brace Ġc _ j \ r brace Ġ_ { j = 1 } ^{ | C | } $ Ġ. ĠThe Ġprobability Ġof Ġeach Ġclass Ġgiven Ġthe Ġdocument Ġis Ġdefined Ġas : </s> $$ P ( c _ j | d _ i ) Ġ= Ġ\ frac { P ( d _ i | c _ j ) P ( c _ j ) }{ P ( d _ i ) }. $$ Ġ Ġ Ġ( E q . Ġ16 ) </s> The Ġprior Ġ$ P ( d _ i ) $ Ġis Ġthe Ġsame Ġfor Ġall Ġclasses , Ġso Ġto Ġdetermine Ġthe Ġclass Ġto Ġwhich Ġ$ d _ i $ Ġbelongs Ġto , Ġthe Ġfollowing Ġequation Ġcan Ġbe Ġused : </s> $$ pred iction ( d _ i ) Ġ= Ġarg max _{ c _ j } P ( d _ i | c _ j ) P ( c _ j ) $$ Ġ Ġ Ġ( E q . Ġ17 ) </s> The Ġprior Ġ$ P ( c _ j ) $ Ġcan Ġbe Ġobtained Ġby Ġthe Ġfollowing Ġequation : </s> $$ P ( c _ j ) Ġ= Ġ\ frac { 1 + | D _ j | }{ | C | + | D | }, $$ Ġ Ġ Ġ( E q . Ġ18 ) </s> where Ġ$ | D _ j | $ Ġis Ġthe Ġnumber Ġof Ġdocuments Ġin Ġclass Ġ$ c _ j $ Ġ. </s> As Ġfor Ġthe Ġposterior Ġ$ P ( d _ i | c _ j ) $ Ġ, Ġdifferent Ġcalculations Ġare Ġperformed Ġfor Ġeach Ġmodel . ĠFor ĠMV B , Ġit Ġis Ġdefined Ġas : </s> $$ P ( d _ i | c _ j ) Ġ= Ġ\ pro d Ġ_ { k = 1 } ^{ | V | } P ( w _ k | c _ j ) ^{ t _ i ^ k } ( 1 - P ( w _ k | c _ j )) ^{ 1 - t _ i ^ k }, $$ Ġ Ġ Ġ( E q . Ġ19 ) </s> where Ġ$ w _ k $ Ġis Ġthe Ġk - th Ġword Ġin Ġthe Ġvocabulary Ġ$ V $ Ġ, Ġand Ġ$ t _ i ^ k $ Ġis Ġthe Ġvalue Ġ( 0 Ġor Ġ1 ) Ġof Ġthe Ġk - th Ġelement Ġof Ġthe Ġvector Ġof Ġdocument Ġ$ d _ i $ Ġ. </s> For Ġthe ĠMN B , Ġit Ġis Ġdefined Ġas : </s> $$ P ( d _ i | c _ j ) Ġ= ĠP ( | d _ i | )| d _ i | ! \ pro d Ġ_ { k = 1 } ^{ | V | }\ frac { P ( w _ k | c _ j ) ^{ n _ i ^ k }} { n _ i ^ k ! }, $$ Ġ Ġ Ġ( E q . Ġ20 ) </s> where Ġ$ | d _ i | $ Ġis Ġthe Ġnumber Ġof Ġwords Ġin Ġdocument Ġ$ d _ i $ Ġand Ġ$ n _ i ^ k $ Ġis Ġthe Ġk - th Ġelement Ġof Ġthe Ġvector Ġof Ġdocument Ġ$ d _ i $ Ġand Ġit Ġrepresents Ġhow Ġmany Ġtimes Ġword Ġ$ w _ k $ Ġoccurs Ġin Ġ$ d _ i $ Ġ. </s> Finally , Ġthe Ġposterior Ġ$ P ( w _ k | c _ j ) $ Ġcan Ġbe Ġobtained Ġby Ġthe Ġfollowing Ġequation : </s> $$ P ( w _ k | c _ j ) Ġ= Ġ\ frac { 1 + | D _ j ^ k | }{ | C | + | D | }, $$ Ġ Ġ Ġ( E q . Ġ21 ) </s> where Ġ$ | D _ j ^ k | $ Ġis Ġthe Ġnumber Ġof Ġdocuments Ġin Ġclass Ġ$ c _ j $ Ġthat Ġcontain Ġthe Ġword Ġ$ w _ k $ Ġ. </s> In Ġgeneral , ĠMV B Ġtends Ġto Ġperform Ġbetter Ġthan ĠMN B Ġat Ġsmall Ġvocabulary Ġsizes Ġwhereas ĠMN B Ġis Ġmore Ġefficient Ġon Ġlarge Ġvoc ab ular ies . </s> Despite Ġbeing Ġrobust Ġtools Ġfor Ġtext Ġclassification , Ġboth Ġthese Ġmodels Ġdepend Ġdirectly Ġon Ġthe Ġbag - of - words Ġfeatures Ġand Ġdo Ġnot Ġnaturally Ġwork Ġwith Ġrepresentations Ġsuch Ġas Ġword 2 vec . </s> Lat ent Ġsemantic Ġanalysis Ġ( L SA ), Ġor Ġlatent Ġsemantic Ġindex ing Ġ( LS I ), Ġwas Ġproposed Ġin ĠB IB REF 12 Ġ, Ġand Ġit Ġextends Ġthe Ġvector Ġspace Ġmodel Ġby Ġusing Ġsingular Ġvalue Ġdecom position Ġ( S VD ) Ġto Ġfind Ġa Ġset Ġof Ġunderlying Ġlatent Ġvariables Ġwhich Ġspans Ġthe Ġmeaning Ġof Ġtexts . </s> It Ġis Ġbuilt Ġfrom Ġa Ġterm - document Ġmatrix , Ġin Ġwhich Ġeach Ġrow Ġrepresents Ġa Ġterm , Ġand Ġeach Ġcolumn Ġrepresents Ġa Ġdocument . ĠThis Ġmatrix Ġcan Ġbe Ġbuilt Ġby Ġconc aten ating Ġthe Ġvectors Ġof Ġall Ġdocuments Ġin Ġa Ġcorpus , Ġobtained Ġusing Ġthe Ġbag - of - words Ġmodel , Ġthat Ġis , Ġ$ Ġ{ X } Ġ= Ġ[ Ġ{ v } _ 1 , Ġ{ v } _ 2 , Ġ... , Ġ{ v } _{ | D | } ] $ Ġ, Ġwhere Ġ${ v } _ i $ Ġis Ġthe Ġvector Ġrepresentation Ġobtained Ġusing Ġthe Ġbag - of - words Ġmodel . </s> In Ġthis Ġmethod , Ġthe Ġterm - document Ġmatrix Ġis Ġdecomp osed Ġusing Ġthe Ġsingular Ġvalue Ġdecom position , </s> $$ { X } Ġ= Ġ{ U \ S igma ĠV } ^ \ top Ġ, $$ Ġ Ġ Ġ( E q . Ġ23 ) </s> where Ġ$ U $ Ġand Ġ$ V $ Ġare Ġorth og onal Ġmat rices Ġand Ġcorrespond Ġto Ġthe Ġleft Ġsingular Ġvectors Ġand Ġright Ġsingular Ġvectors Ġof Ġ$ X $ Ġ, Ġrespectively . Ġ$\ S igma Ġ$ Ġis Ġa Ġdiagonal Ġmatrix , Ġand Ġit Ġcontains Ġthe Ġsquare Ġroots Ġof Ġthe Ġe igen values Ġof Ġ$ X ^ TX $ Ġand Ġ$ XX ^ T $ Ġ. ĠL SA Ġfinds Ġa Ġlow - rank Ġapproximation Ġof Ġ$ X $ Ġby Ġselecting Ġonly Ġthe Ġ$ k $ Ġlargest Ġsingular Ġvalues Ġand Ġits Ġrespective Ġsingular Ġvectors , </s> $$ { X } _ k Ġ= Ġ{ U } _ k {\ S igma Ġ} _ k Ġ{ V } _ k ^ {\ top Ġ} . $$ Ġ Ġ Ġ( E q . Ġ24 ) </s> To Ġcompare Ġtwo Ġdocuments , Ġwe Ġproject Ġboth Ġof Ġthem Ġinto Ġthis Ġlower Ġdimension Ġspace Ġand Ġcalculate Ġthe Ġcos ine Ġdistance Ġbetween Ġthem . ĠThe Ġprojection Ġ${ \ hat { d }} $ Ġof Ġdocument Ġ${ d } $ Ġis Ġobtained Ġby Ġthe Ġfollowing Ġequation : </s> $$ {\ hat { d }} Ġ= Ġ{\ S igma Ġ} _ k ^{ - 1 } Ġ{ U } _ k ^ \ top Ġ{ d }. $$ Ġ Ġ Ġ( E q . Ġ25 ) </s> Despite Ġits Ġextensive Ġapplication Ġon Ġtext Ġclassification ĠB IB REF 13 Ġ, ĠB IB REF 14 Ġ, ĠB IB REF 15 Ġ, Ġthis Ġmethod Ġwas Ġinitially Ġproposed Ġfor Ġdocument Ġindex ing Ġand , Ġtherefore , Ġdoes Ġnot Ġencode Ġany Ġclass Ġinformation Ġwhen Ġmodeling Ġthe Ġlow - rank Ġapproximation . ĠTo Ġperform Ġclassification , Ġ1 - ne arest Ġneighbor Ġis Ġusually Ġperformed , Ġplacing Ġa Ġquery Ġdocument Ġinto Ġthe Ġclass Ġof Ġthe Ġnearest Ġtraining Ġdocument . </s> The Ġsupport Ġvector Ġmachine Ġ( S VM ) Ġwas Ġfirst Ġpresented Ġin ĠB IB REF 16 Ġand Ġperforms Ġthe Ġseparation Ġbetween Ġsamples Ġof Ġtwo Ġdifferent Ġclasses Ġby Ġprojecting Ġthem Ġonto Ġa Ġhigher Ġdimension ality Ġspace . ĠIt Ġwas Ġfirst Ġapplied Ġin Ġtext Ġclassification Ġby ĠB IB REF 17 Ġand Ġhave Ġsince Ġbeen Ġsuccessfully Ġapplied Ġin Ġmany Ġtasks Ġrelated Ġto Ġnatural Ġlanguage Ġprocessing ĠB IB REF 18 Ġ, ĠB IB REF 19 Ġ. </s> Consider Ġa Ġtraining Ġdata Ġset Ġ$ D $ Ġ, Ġwith Ġ$ n $ Ġsamples </s> $$ D Ġ= Ġ\ l brace Ġ({ x } _ i , c _ i )| { x } _ i \ in Ġ\ math bb Ġ{ R } ^ p , Ġc _ i Ġ\ in Ġ\ l brace Ġ- 1 , 1 \ r brace Ġ Ġ\ r brace Ġ_ { i = 1 } ^{ n }, $$ Ġ Ġ Ġ( E q . Ġ27 ) </s> where Ġ$ c _ i $ Ġrepresents Ġthe Ġclass Ġto Ġwhich Ġ${ x } _ i $ Ġbelongs Ġto . ĠEach Ġ${ x } _ i $ Ġis Ġa Ġ$ p $ Ġ- dimensional Ġvector . ĠThe Ġgoal Ġis Ġto Ġfind Ġthe Ġhyper plane Ġthat Ġdivides Ġthe Ġpoints Ġfrom Ġ$ c _ i Ġ= Ġ1 $ Ġfrom Ġthe Ġpoints Ġfrom Ġ$ c _ i Ġ= Ġ- 1 $ Ġ. ĠThis Ġhyper plane Ġcan Ġbe Ġwritten Ġas Ġa Ġset Ġof Ġpoints Ġ$ x $ Ġsatisfying : </s> $$ { w } Ġ\ cd ot Ġ{ x } Ġ- Ġb Ġ= Ġ0 , $$ Ġ Ġ Ġ( E q . Ġ28 ) </s> where Ġ$\ cd ot Ġ$ Ġdenotes Ġthe Ġdot Ġproduct . ĠThe Ġvector Ġ${ w } $ Ġis Ġperpendicular Ġto Ġthe Ġhyper plane . ĠThe Ġparameter Ġ$\ frac { b }{ \ Vert Ġ{ w }\ Vert Ġ} $ Ġdetermines Ġthe Ġoffset Ġof Ġthe Ġhyper plane Ġfrom Ġthe Ġorigin Ġalong Ġthe Ġnormal Ġvector Ġ${ w } $ Ġ. </s> We Ġwish Ġto Ġchoose Ġ${ w } $ Ġand Ġ$ b $ Ġ, Ġso Ġthey Ġmaximize Ġthe Ġdistance Ġbetween Ġthe Ġparallel Ġhyper planes Ġthat Ġare Ġas Ġfar Ġapart Ġas Ġpossible , Ġwhile Ġstill Ġseparating Ġthe Ġdata . </s> If Ġthe Ġtraining Ġdata Ġis Ġlin early Ġsepar able , Ġwe Ġcan Ġselect Ġtwo Ġhyper planes Ġin Ġa Ġway Ġthat Ġthere Ġare Ġno Ġpoints Ġbetween Ġthem Ġand Ġthen Ġtry Ġto Ġmaximize Ġthe Ġdistance . ĠIn Ġother Ġwords , Ġminimize Ġ$\ Vert Ġ{ w }\ Vert Ġ$ Ġsubject Ġto Ġ$ c _ i ({ w }\ cd ot Ġ{ x } _ u - b ) Ġ\ ge Ġ1 , Ġi = \ l brace Ġ1 , 2 ,... , n \ r brace Ġ$ Ġ. ĠIf Ġthe Ġtraining Ġdata Ġis Ġnot Ġlin early Ġsepar able , Ġthe Ġkernel Ġtrick Ġcan Ġbe Ġapplied , Ġwhere Ġevery Ġdot Ġproduct Ġis Ġreplaced Ġby Ġa Ġnon - linear Ġkernel Ġfunction . </s> Word Ġsub space </s> All Ġmethods Ġmentioned Ġabove Ġutilize Ġthe ĠB OW Ġfeatures Ġto Ġrepresent Ġa Ġdocument . ĠAlthough Ġthis Ġrepresentation Ġis Ġsimple Ġand Ġpowerful , Ġits Ġmain Ġproblem Ġlies Ġon Ġdisreg arding Ġthe Ġword Ġsemantics Ġwithin Ġa Ġdocument , Ġwhere Ġthe Ġcontext Ġand Ġmeaning Ġcould Ġoffer Ġmany Ġbenefits Ġto Ġthe Ġmodel Ġsuch Ġas Ġidentification Ġof Ġsyn onyms . </s> In Ġour Ġformulation , Ġwords Ġare Ġrepresented Ġas Ġvectors Ġin Ġa Ġreal - valued Ġfeature Ġvector Ġspace Ġ$\ math bb Ġ{ R } ^{ p } $ Ġ, Ġby Ġusing Ġword 2 vec ĠB IB REF 8 Ġ. ĠThrough Ġthis Ġrepresentation , Ġit Ġis Ġpossible Ġto Ġcalculate Ġthe Ġdistance Ġbetween Ġtwo Ġwords , Ġwhere Ġwords Ġfrom Ġsimilar Ġcontexts Ġare Ġrepresented Ġby Ġvectors Ġclose Ġto Ġeach Ġother , Ġwhile Ġwords Ġfrom Ġdifferent Ġcontexts Ġare Ġrepresented Ġas Ġfar Ġapart Ġvectors . ĠAlso , Ġthis Ġrepresentation Ġbrings Ġthe Ġnew Ġconcept Ġof Ġarithmetic Ġoperations Ġbetween Ġwords , Ġwhere Ġoperations Ġsuch Ġas Ġaddition Ġand Ġsubt raction Ġcarry Ġmeaning Ġ( eg ., ĠâĢ ľ king âĢ Ŀ - âĢ ľ man âĢ Ŀ + âĢ ľ woman âĢ Ŀ = âĢ ľ que en âĢ Ŀ ) ĠB IB REF 20 Ġ. </s> Consider Ġa Ġset Ġof Ġdocuments Ġwhich Ġbelong Ġto Ġthe Ġsame Ġcontext Ġ$ D _ c Ġ= Ġ\ l brace Ġd _ i \ r brace Ġ_ { i = 1 } ^{ | D _ c | } $ Ġ. ĠEach Ġdocument Ġ$ d _ i $ Ġis Ġrepresented Ġby Ġa Ġset Ġof Ġ$ N _ i $ Ġwords , Ġ$ d _ i Ġ= Ġ\ l brace Ġw _ k \ r brace Ġ_ { k = 1 } ^{ N _ i } $ Ġ. ĠBy Ġconsidering Ġthat Ġall Ġwords Ġfrom Ġdocuments Ġof Ġthe Ġsame Ġcontext Ġbelong Ġto Ġthe Ġsame Ġdistribution , Ġa Ġset Ġof Ġwords Ġ$ W _ c Ġ= Ġ\ l brace Ġw _ k \ r brace Ġ_ { k = 1 } ^{ N _ c } $ Ġwith Ġthe Ġwords Ġin Ġthe Ġcontext Ġ$ c $ Ġis Ġobtained . </s> We Ġthen Ġtranslate Ġthese Ġwords Ġinto Ġword Ġvectors Ġusing Ġword 2 vec , Ġresulting Ġin Ġa Ġset Ġof Ġword Ġvectors Ġ$ X _ c Ġ= Ġ\ l brace Ġ{ x } ^ k _ c \ r brace Ġ_ { k = 1 } ^{ N _ c } Ġ\ in Ġ\ math bb Ġ{ R } ^ p $ Ġ. ĠThis Ġset Ġof Ġword Ġvectors Ġis Ġmodeled Ġinto Ġa Ġword Ġsub space , Ġwhich Ġis Ġa Ġcompact , Ġscalable Ġand Ġmeaningful Ġrepresentation Ġof Ġthe Ġwhole Ġset . ĠSuch Ġa Ġword Ġsub space Ġis Ġgenerated Ġby Ġapplying ĠPC A Ġto Ġthe Ġset Ġof Ġword Ġvectors . </s> First , Ġwe Ġcompute Ġan Ġaut oc or relation Ġmatrix , Ġ${ R } _ c $ Ġ: </s> $$ { R } _ c Ġ= Ġ\ frac { 1 }{ N _ c }\ sum Ġ_ { i = 1 } ^{ N _ c }{ x } ^{ i } _ c { x } _ c ^{ i ^ {\ top Ġ} }. $$ Ġ Ġ Ġ( E q . Ġ29 ) </s> The Ġorth on ormal Ġbasis Ġvectors Ġof Ġ$ m _ c $ Ġ- dimensional Ġsub space Ġ${ Y } _ c $ Ġare Ġobtained Ġas Ġthe Ġe igen ve ctors Ġwith Ġthe Ġ$ m _ c $ Ġlargest Ġe igen values Ġof Ġthe Ġmatrix Ġ${ R } _ c $ Ġ. ĠWe Ġrepresent Ġa Ġsub space Ġ${ Y } _ c $ Ġby Ġthe Ġmatrix Ġ${ Y } _ c Ġ\ in Ġ\ math bb Ġ{ R } ^{ p Ġ\ times Ġm _ c } $ Ġ, Ġwhich Ġhas Ġthe Ġcorresponding Ġorth on ormal Ġbasis Ġvectors Ġas Ġits Ġcolumn Ġvectors . </s> Text Ġclassification Ġbased Ġon Ġword Ġsub space </s> We Ġformulate Ġour Ġproblem Ġas Ġa Ġsingle Ġlabel Ġclassification Ġproblem . ĠGiven Ġa Ġset Ġof Ġtraining Ġdocuments , Ġwhich Ġwe Ġwill Ġrefer Ġas Ġcorpus , Ġ$ D Ġ= Ġ\ l brace Ġd _ i \ r brace Ġ_ { i = 1 } ^{ | D | } $ Ġ, Ġwith Ġknown Ġclasses Ġ$ C Ġ= Ġ\ l brace Ġc _ j \ r brace Ġ_ { j = 1 } ^{ | C | } $ Ġ, Ġwe Ġwish Ġto Ġclassify Ġa Ġquery Ġdocument Ġ$ d _ q $ Ġinto Ġone Ġof Ġthe Ġclasses Ġin Ġ$ C $ Ġ. </s> Text Ġclassification Ġbased Ġon Ġword Ġsub space Ġcan Ġbe Ġperformed Ġunder Ġthe Ġframework Ġof Ġmutual Ġsub space Ġmethod Ġ( MS M ). ĠThis Ġtask Ġinvolves Ġtwo Ġdifferent Ġstages : ĠA Ġlearning Ġstage , Ġwhere Ġthe Ġword Ġsub space Ġfor Ġeach Ġclass Ġis Ġmodeled , Ġand Ġa Ġclassification Ġstage , Ġwhere Ġthe Ġword Ġsub space Ġfor Ġa Ġquery Ġis Ġmodeled Ġand Ġcompared Ġto Ġthe Ġword Ġsub sp aces Ġof Ġthe Ġclasses . </s> In Ġthe Ġlearning Ġstage , Ġit Ġis Ġassumed Ġthat Ġall Ġdocuments Ġof Ġthe Ġsame Ġclass Ġbelong Ġto Ġthe Ġsame Ġcontext , Ġresulting Ġin Ġa Ġset Ġof Ġwords Ġ$ W _ c Ġ= Ġ\ l brace Ġw _ c ^ k \ r brace Ġ_ { k = 1 } ^{ N _ c } $ Ġ. ĠThis Ġset Ġassumes Ġthat Ġeach Ġword Ġappears Ġonly Ġonce Ġin Ġeach Ġclass . ĠEach Ġset Ġ$\ l brace ĠW _ c \ r brace Ġ_ { c = 1 } ^{ | C | } $ Ġis Ġthen Ġmodeled Ġinto Ġa Ġword Ġsub space Ġ${ Y } _ c $ Ġ, Ġas Ġexplained Ġin ĠSection Ġ" Word Ġsub space " Ġ. ĠAs Ġthe Ġnumber Ġof Ġwords Ġin Ġeach Ġclass Ġmay Ġvary Ġlargely , Ġthe Ġdimension Ġ$ m _ c $ Ġof Ġeach Ġclass Ġword Ġsub space Ġis Ġnot Ġset Ġto Ġthe Ġsame Ġvalue . </s> In Ġthe Ġclassification Ġstage , Ġfor Ġa Ġquery Ġdocument Ġ$ d _ q $ Ġ, Ġit Ġis Ġalso Ġassumed Ġthat Ġeach Ġword Ġoccurs Ġonly Ġonce , Ġgenerating Ġa Ġsub space Ġ${ Y } _ q $ Ġ. </s> To Ġmeasure Ġthe Ġsimilarity Ġbetween Ġa Ġclass Ġword Ġsub space Ġ${ Y } _ c $ Ġand Ġa Ġquery Ġword Ġsub space Ġ${ Y } _ q $ Ġ, Ġthe Ġcanonical Ġangles Ġbetween Ġthe Ġtwo Ġword Ġsub sp aces Ġare Ġused ĠB IB REF 21 Ġ. ĠThere Ġare Ġseveral Ġmethods Ġfor Ġcalculating Ġcanonical Ġangles ĠB IB REF 22 Ġ, ĠB IB REF 23 Ġ, Ġand ĠB IB REF 24 Ġ, Ġamong Ġwhich Ġthe Ġsimplest Ġand Ġmost Ġpractical Ġis Ġthe Ġsingular Ġvalue Ġdecom position Ġ( S VD ). ĠConsider , Ġfor Ġexample , Ġtwo Ġsub sp aces , Ġone Ġfrom Ġthe Ġtraining Ġdata Ġand Ġanother Ġfrom Ġthe Ġquery , Ġrepresented Ġas Ġmat rices Ġof Ġbases , Ġ${ Y } _{ c } Ġ= Ġ[ {\ Ph i Ġ} _{ 1 } Ġ\ ld ots Ġ{\ Ph i Ġ} _{ m _ c } ] Ġ\ in Ġ\ math bb Ġ{ R } ^{ p Ġ\ times Ġm _ c } $ Ġand Ġ${ Y } _{ q } Ġ= Ġ[ {\ Ps i Ġ} _{ 1 } Ġ\ ld ots Ġ{\ Ps i Ġ} _{ m _ q } ] Ġ\ in Ġ\ math bb Ġ{ R } ^{ p Ġ\ times Ġm _ q } $ Ġ, Ġwhere Ġ${ \ Ph i Ġ} _{ i } $ Ġare Ġthe Ġbases Ġfor Ġ${ Y } _ c $ Ġand Ġ${ \ Ps i Ġ} _{ i } $ Ġare Ġthe Ġbases Ġfor Ġ${ Y } _ q $ Ġ. ĠLet Ġthe ĠS VD Ġof Ġ${ Y } _ c ^ {\ top Ġ} { Y } _ q Ġ\ in Ġ\ math bb Ġ{ R } ^{ m _ c Ġ\ times Ġm _ q } $ Ġbe Ġ${ Y } _ c ^ {\ top Ġ} { Y } _ q Ġ= Ġ{ U Ġ\ S igma ĠV } ^ {\ top Ġ} $ Ġ, Ġwhere Ġ${ Y } _ q $ 0 Ġ, Ġ${ Y } _ q $ 1 Ġrepresents Ġthe Ġset Ġof Ġsingular Ġvalues . ĠThe Ġcanonical Ġangles Ġ${ Y } _ q $ 2 Ġcan Ġbe Ġobtained Ġas Ġ${ Y } _ q $ 3 Ġ${ Y } _ q $ 4 Ġ. ĠThe Ġsimilarity Ġbetween Ġthe Ġtwo Ġsub sp aces Ġis Ġmeasured Ġby Ġ${ Y } _ q $ 5 Ġangles Ġas Ġfollows : </s> $$ S _{ ({ Y } _ c , { Y } _ q )} [ t ] Ġ= Ġ\ frac { 1 }{ t }\ sum Ġ_ { i Ġ= Ġ1 } ^{ t } Ġ\ cos Ġ^ { 2 } Ġ\ the ta Ġ_ { i }, \ ; Ġ1 Ġ\ le Ġt Ġ\ le Ġm _ q , Ġ\ ; Ġm _ q Ġ\ le Ġm _ c . $$ Ġ Ġ Ġ( E q . Ġ30 ) </s> Fig . Ġ1 Ġshows Ġthe Ġmodeling Ġand Ġcomparison Ġof Ġsets Ġof Ġwords Ġby ĠMSM . ĠThis Ġmethod Ġcan Ġcompare Ġsets Ġof Ġdifferent Ġsizes , Ġand Ġnaturally Ġenc odes Ġproximity Ġbetween Ġsets Ġwith Ġrelated Ġwords . </s> Finally , Ġthe Ġclass Ġwith Ġthe Ġhighest Ġsimilarity Ġwith Ġ$ d _ q $ Ġis Ġassigned Ġas Ġthe Ġclass Ġof Ġ$ d _ q $ Ġ: </s> $$ pred iction ( d _ q ) Ġ= Ġarg max _ c ( S _{ ({ Y } _ c , { Y } _ q )} ). $$ Ġ Ġ Ġ( E q . Ġ32 ) </s> TF Ġweighted Ġword Ġsub space </s> The Ġword Ġsub space Ġformulation Ġpresented Ġin ĠSection Ġ" Word Ġsub space " Ġis Ġa Ġpractical Ġand Ġcompact Ġway Ġto Ġrepresent Ġsets Ġof Ġword Ġvectors , Ġretaining Ġmost Ġof Ġthe Ġvariability Ġof Ġfeatures . ĠHowever , Ġas Ġseen Ġin Ġthe ĠB OW Ġfeatures , Ġthe Ġfrequency Ġof Ġwords Ġis Ġrelevant Ġinformation Ġthat Ġcan Ġimprove Ġthe Ġcharacterization Ġof Ġa Ġtext . ĠTo Ġincorporate Ġthis Ġinformation Ġinto Ġthe Ġword Ġsub space Ġmodeling , Ġwe Ġpropose Ġan Ġextension Ġof Ġthe Ġword Ġsub space , Ġcalled Ġthe Ġterm - frequency Ġ( TF ) Ġweighted Ġword Ġsub space . </s> Like Ġthe Ġword Ġsub space , Ġthe ĠTF Ġweighted Ġword Ġsub space Ġis Ġmat hematically Ġdefined Ġas Ġa Ġlow - dimensional Ġlinear Ġsub space Ġin Ġa Ġword Ġvector Ġspace Ġwith Ġhigh Ġdimension ality . ĠHowever , Ġa Ġweighted Ġversion Ġof Ġthe ĠPC A ĠB IB REF 25 Ġ, ĠB IB REF 26 Ġis Ġutilized Ġto Ġincorporate Ġthe Ġinformation Ġgiven Ġby Ġthe Ġfrequencies Ġof Ġwords Ġ( term - f requ encies ). ĠThis ĠTF Ġweighted Ġword Ġsub space Ġis Ġequivalent Ġto Ġthe Ġword Ġsub space Ġif Ġwe Ġconsider Ġall Ġoccurrences Ġof Ġthe Ġwords . </s> Consider Ġthe Ġset Ġof Ġword Ġvectors Ġ$\ l brace Ġ{ x } _ c ^ k \ r brace Ġ_ { k = 1 } ^{ N _ c } Ġ\ in Ġ\ math bb Ġ{ R } ^{ p } $ Ġ, Ġwhich Ġrepresents Ġeach Ġword Ġin Ġthe Ġcontext Ġ$ c $ Ġ, Ġand Ġthe Ġset Ġof Ġweights Ġ$\ l brace Ġ\ ome ga Ġ_ i \ r brace Ġ_ { i = 1 } ^{ N _ c } $ Ġ, Ġwhich Ġrepresent Ġthe Ġfrequencies Ġof Ġthe Ġwords Ġin Ġthe Ġcontext Ġ$ c $ Ġ. </s> We Ġincorporate Ġthese Ġfrequencies Ġinto Ġthe Ġsub space Ġcalculation Ġby Ġweight ing Ġthe Ġdata Ġmatrix Ġ${ X } $ Ġas Ġfollows : </s> $$ {\ wid et ilde { X }} ={ X }{ \ O mega Ġ} ^{ 1 / 2 }, $$ Ġ Ġ Ġ( E q . Ġ33 ) </s> where Ġ${ X } Ġ\ in Ġ\ math bb Ġ{ R } ^{ p Ġ\ times ĠN _ c } $ Ġis Ġa Ġmatrix Ġcontaining Ġthe Ġword Ġvectors Ġ$\ l brace Ġ{ x } _ c ^ k \ r brace Ġ_ { k = 1 } ^{ N _ c } $ Ġand Ġ${ \ O mega Ġ} $ Ġis Ġa Ġdiagonal Ġmatrix Ġcontaining Ġthe Ġweights Ġ$\ l brace Ġ\ ome ga Ġ_ i \ r brace Ġ_ { i = 1 } ^{ N _ c } $ Ġ. </s> We Ġthen Ġperform ĠPC A Ġby Ġsolving Ġthe ĠS VD Ġof Ġthe Ġmatrix Ġ${ \ wid et ilde { X }} $ Ġ: </s> $$ {\ wid et ilde { X }} ={ AM B } ^ {\ top Ġ}, $$ Ġ Ġ Ġ( E q . Ġ34 ) </s> where Ġthe Ġcolumns Ġof Ġthe Ġorth og onal Ġmat rices Ġ${ A } $ Ġand Ġ${ B } $ Ġare , Ġrespectively , Ġthe Ġleft - sing ular Ġvectors Ġand Ġright - sing ular Ġvectors Ġof Ġthe Ġmatrix Ġ${ \ wid et ilde { X }} $ Ġ, Ġand Ġthe Ġdiagonal Ġmatrix Ġ${ M } $ Ġcontains Ġsingular Ġvalues Ġof Ġ${ \ wid et ilde { X }} $ Ġ. </s> Finally , Ġthe Ġorth on ormal Ġbasis Ġvectors Ġof Ġthe Ġ$ m _ c $ Ġ- dimensional ĠTF Ġweighted Ġsub space Ġ${ W } $ Ġare Ġthe Ġcolumn Ġvectors Ġin Ġ${ A } $ Ġcorresponding Ġto Ġthe Ġ$ m _ c $ Ġlargest Ġsingular Ġvalues Ġin Ġ${ M } $ Ġ. </s> Text Ġclassification Ġwith ĠTF Ġweighted Ġword Ġsub space Ġcan Ġalso Ġbe Ġperformed Ġunder Ġthe Ġframework Ġof ĠMSM . ĠIn Ġthis Ġpaper , Ġwe Ġwill Ġrefer Ġto ĠMSM Ġwith ĠTF Ġweighted Ġword Ġsub space Ġas ĠTF - MS M . </s> Exper imental ĠEvaluation </s> In Ġthis Ġsection Ġwe Ġdescribe Ġthe Ġexperiments Ġperformed Ġto Ġdemonstrate Ġthe Ġvalidity Ġof Ġour Ġproposed Ġmethod Ġand Ġits Ġextension . ĠWe Ġused Ġthe ĠReuters - 8 Ġdataset Ġwithout Ġstop Ġwords Ġfrom ĠB IB REF 27 Ġaiming Ġat Ġsingle - label Ġclassification , Ġwhich Ġis Ġa Ġpre process ed Ġformat Ġof Ġthe ĠReuters - 215 78 . ĠWords Ġin Ġthe Ġtexts Ġwere Ġconsidered Ġas Ġthey Ġappeared , Ġwithout Ġperforming Ġstemming Ġor Ġtypo Ġcorrection . ĠThis Ġdatabase Ġhas Ġeight Ġdifferent Ġclasses Ġwith Ġthe Ġnumber Ġof Ġsamples Ġvarying Ġfrom Ġ51 Ġto Ġover Ġ3000 Ġdocuments , Ġas Ġcan Ġbe Ġseen Ġin ĠTable Ġ1 Ġ. </s> To Ġobtain Ġthe Ġvector Ġrepresentation Ġof Ġwords , Ġwe Ġused Ġa Ġfreely Ġavailable Ġword 2 vec Ġmodel , Ġtrained Ġby ĠB IB REF 8 Ġ, Ġon Ġapproximately Ġ100 Ġbillion Ġwords , Ġwhich Ġenc odes Ġthe Ġvector Ġrepresentation Ġin Ġ$\ math bb Ġ{ R } ^{ 300 } $ Ġof Ġover Ġ3 Ġmillion Ġwords Ġfrom Ġseveral Ġdifferent Ġlanguages . ĠSince Ġwe Ġdecided Ġto Ġfocus Ġon ĠEnglish Ġwords Ġonly , Ġwe Ġfiltered Ġthese Ġvectors Ġto Ġabout Ġ800 Ġthousand Ġwords , Ġexcluding Ġall Ġwords Ġwith Ġnon - roman Ġcharacters . </s> To Ġshow Ġthe Ġvalidity Ġof Ġour Ġword Ġsub space Ġrepresentation Ġfor Ġtext Ġclassification Ġand Ġthe Ġproposed Ġextension , Ġwe Ġdivided Ġour Ġexperiment Ġsection Ġinto Ġtwo Ġparts : ĠThe Ġfirst Ġone Ġaims Ġto Ġverify Ġif Ġsets Ġof Ġword Ġvectors Ġare Ġsuitable Ġfor Ġsub space Ġrepresentation , Ġand Ġthe Ġsecond Ġone Ġputs Ġour Ġmethods Ġin Ġpractice Ġin Ġa Ġtext Ġclassification Ġtest , Ġcomparing Ġour Ġresults Ġwith Ġthe Ġconventional Ġmethods Ġdescribed Ġin ĠSection Ġ" Related ĠWork " Ġ. </s> E val uation Ġof Ġthe Ġword Ġsub space Ġrepresentation </s> In Ġthis Ġexperiment , Ġwe Ġmodeled Ġthe Ġword Ġvectors Ġfrom Ġeach Ġclass Ġin Ġthe ĠReuters - 8 Ġdatabase Ġinto Ġa Ġword Ġsub space . ĠThe Ġprimary Ġgoal Ġis Ġto Ġvisualize Ġhow Ġmuch Ġof Ġthe Ġtext Ġdata Ġcan Ġbe Ġrepresented Ġby Ġa Ġlower Ġdimensional Ġsub space . </s> Sub space Ġrepresentations Ġare Ġvery Ġefficient Ġin Ġcompact ly Ġrepresent Ġdata Ġthat Ġis Ġclose Ġto Ġa Ġnormal Ġdistribution . ĠThis Ġcharacteristic Ġis Ġdue Ġto Ġthe Ġapplication Ġof Ġthe ĠPC A , Ġthat Ġis Ġoptimal Ġto Ġfind Ġthe Ġdirection Ġwith Ġthe Ġhighest Ġvariation Ġwithin Ġthe Ġdata . </s> In ĠPC A , Ġthe Ġprincipal Ġcomponents Ġgive Ġthe Ġdirections Ġof Ġmaximum Ġvariance , Ġwhile Ġtheir Ġcorresponding Ġe igen values Ġgive Ġthe Ġvariance Ġof Ġthe Ġdata Ġin Ġeach Ġof Ġthem . ĠTherefore , Ġby Ġobserving Ġthe Ġdistribution Ġof Ġthe Ġe igen values Ġcomputed Ġwhen Ġperforming ĠPC A Ġin Ġthe Ġmodeling Ġof Ġthe Ġsub sp aces , Ġwe Ġcan Ġsuggest Ġif Ġthe Ġdata Ġis Ġsuitable Ġor Ġnot Ġfor Ġsub space Ġrepresentation . </s> For Ġeach Ġclass , Ġwe Ġnormalized Ġthe Ġe igen values Ġby Ġthe Ġlargest Ġone Ġof Ġthe Ġclass . ĠFig . Ġ2 Ġshows Ġthe Ġmean Ġof Ġthe Ġe igen values Ġand Ġthe Ġstandard Ġdeviation Ġamong Ġclasses . ĠIt Ġis Ġpossible Ġto Ġsee Ġthat Ġthe Ġfirst Ġlargest Ġe igen values Ġretain Ġlarger Ġvariance Ġthan Ġthe Ġsmallest Ġones . ĠIn Ġfact , Ġlooking Ġat Ġthe Ġfirst Ġ150 Ġlargest Ġe igen values , Ġwe Ġcan Ġsee Ġthat Ġthey Ġretain , Ġon Ġaverage , Ġ86 . 37 % Ġof Ġthe Ġdata Ġvariance . ĠAlso , Ġby Ġobserving Ġthe Ġstandard Ġdeviation , Ġwe Ġcan Ġunderstand Ġthat Ġthe Ġe igen values Ġdistribution Ġamong Ġclasses Ġfollows Ġthe Ġsame Ġpattern , Ġthat Ġis , Ġmost Ġof Ġthe Ġvariance Ġis Ġin Ġthe Ġfirst Ġdimensions . ĠThis Ġplot Ġindicates Ġthat Ġtext Ġdata Ġrepresented Ġby Ġvectors Ġgenerated Ġwith Ġword 2 vec Ġis Ġsuitable Ġfor Ġsub space Ġrepresentation . </s> Text Ġclassification Ġexperiment </s> In Ġthis Ġexperiment , Ġwe Ġperformed Ġtext Ġclassification Ġamong Ġthe Ġclasses Ġin Ġthe ĠReuters - 8 Ġdatabase . ĠWe Ġcompared Ġthe Ġclassification Ġusing Ġthe Ġword Ġsub space , Ġand Ġits Ġweighted Ġextension , Ġbased Ġon ĠMSM Ġ( to Ġwhich Ġwe Ġwill Ġrefer Ġas ĠMSM Ġand ĠTF - MS M , Ġrespectively ) Ġwith Ġthe Ġbas elines Ġpresented Ġin ĠSection Ġ" Related ĠWork " Ġ: ĠMV B , ĠMN B , ĠL SA , Ġand ĠS VM . ĠSince Ġnone Ġof Ġthe Ġbaseline Ġmethods Ġwork Ġwith Ġvector Ġset Ġclassification , Ġwe Ġalso Ġcompared Ġto Ġa Ġsimple Ġbaseline Ġfor Ġcomparing Ġsets Ġof Ġvectors , Ġdefined Ġas Ġthe Ġaverage Ġof Ġsimilarities Ġbetween Ġall Ġvector Ġpair Ġcombinations Ġof Ġtwo Ġgiven Ġsets . ĠFor Ġtwo Ġmat rices Ġ${ A } $ Ġand Ġ${ B } $ Ġ, Ġcontaining Ġthe Ġsets Ġof Ġvectors Ġ$\ l brace Ġ Ġ{ x } ^{ i } _ a Ġ\ r brace Ġ_ { i Ġ= Ġ1 } ^{ N _ A } $ Ġand Ġ$\ l brace Ġ Ġ{ x } ^{ i } _ b Ġ\ r brace Ġ_ { i Ġ= Ġ1 } ^{ N _ B } $ Ġ, Ġrespectively , Ġwhere Ġ$ N _ A $ Ġand Ġ$ N _ B $ Ġare Ġthe Ġnumber Ġof Ġmain Ġwords Ġin Ġeach Ġset , Ġthe Ġsimilarity Ġis Ġdefined Ġas : </s> $$ Sim _{ ( A , B )} Ġ= Ġ\ frac { 1 }{ N _ A ĠN _ B }\ sum Ġ_ { i } ^{ N _ A }\ sum Ġ_ { j } ^{ N _ B }{ { x } _ a ^ i } ^ {\ top Ġ} { x } _ b ^ j . $$ Ġ Ġ Ġ( E q . Ġ41 ) </s> We Ġrefer Ġto Ġthis Ġbaseline Ġas Ġsimilarity Ġaverage Ġ( SA ). ĠFor Ġthis Ġmethod , Ġwe Ġonly Ġconsidered Ġone Ġoccurrence Ġof Ġeach Ġword Ġin Ġeach Ġset . </s> Different Ġfeatures Ġwere Ġused , Ġdepending Ġon Ġthe Ġmethod . ĠClassification Ġwith ĠSA , ĠMSM , Ġand ĠTF - MS M Ġwas Ġperformed Ġusing Ġword 2 vec Ġfeatures , Ġto Ġwhich Ġwe Ġrefer Ġas Ġw 2 v . ĠFor ĠMV B , Ġdue Ġto Ġits Ġnature , Ġonly Ġbag - of - words Ġfeatures Ġwith Ġbinary Ġweights Ġwere Ġused Ġ( bin B OW ). ĠFor Ġthe Ġsame Ġreason , Ġwe Ġonly Ġused Ġbag - of - words Ġfeatures Ġwith Ġterm - frequency Ġweights Ġ( tf B OW ) Ġwith ĠMN B . ĠClassification Ġwith ĠL SA Ġand ĠS VM Ġis Ġusually Ġperformed Ġusing Ġbag - of - words Ġfeatures Ġand , Ġtherefore , Ġwe Ġtested Ġwith Ġbin B OW , Ġtf B OW , Ġand Ġwith Ġthe Ġterm - frequency Ġinverse Ġdocument - frequency Ġweight , Ġtf id f B OW . ĠWe Ġalso Ġtested Ġthem Ġusing Ġword 2 vec Ġvectors . ĠIn Ġthis Ġcase , Ġwe Ġconsidered Ġeach Ġword Ġvector Ġfrom Ġall Ġdocuments Ġin Ġeach Ġclass Ġto Ġbe Ġa Ġsingle Ġsample . </s> To Ġdetermine Ġthe Ġdimensions Ġof Ġthe Ġclass Ġsub sp aces Ġand Ġquery Ġsub space Ġof ĠMSM Ġand ĠTF - MS M , Ġand Ġthe Ġdimension Ġof Ġthe Ġapproximation Ġperformed Ġby ĠL SA , Ġwe Ġperformed Ġa Ġ10 - fold Ġcross Ġvalidation , Ġwherein Ġeach Ġfold , Ġthe Ġdata Ġwere Ġrandomly Ġdivided Ġinto Ġtrain Ġ( 60 %), Ġvalidation Ġ( 20 %) Ġand Ġtest Ġset Ġ( 20 %). </s> The Ġresults Ġcan Ġbe Ġseen Ġin ĠTable Ġ2 Ġ. ĠThe Ġsimplest Ġbaseline , ĠSA Ġwith Ġw 2 v , Ġachieved Ġan Ġaccuracy Ġrate Ġof Ġ78 . 73 %. ĠThis Ġresult Ġis Ġimportant Ġbecause Ġit Ġshows Ġthe Ġvalidity Ġof Ġthe Ġword 2 vec Ġrepresentation , Ġperforming Ġbetter Ġthan Ġmore Ġelaborate Ġmethods Ġbased Ġon ĠB OW , Ġsuch Ġas ĠMV B Ġwith Ġbin B OW . </s> L SA Ġwith ĠB OW Ġfeatures Ġwas Ġalmost Ġ10 % Ġmore Ġaccurate Ġthan ĠSA , Ġwhere Ġthe Ġbest Ġresults Ġwith Ġbinary Ġweights Ġwere Ġachieved Ġwith Ġan Ġapproximation Ġwith Ġ130 Ġdimensions , Ġwith ĠTF Ġweights Ġwere Ġachieved Ġwith Ġ50 Ġdimensions , Ġand Ġwith ĠTF - ID F Ġweights Ġwere Ġachieved Ġwith Ġ30 Ġdimensions . ĠS VM Ġwith ĠB OW Ġfeatures Ġwas Ġabout Ġ3 % Ġmore Ġaccurate Ġthan ĠL SA , Ġwith Ġbinary Ġweights Ġleading Ġto Ġa Ġhigher Ġaccuracy Ġrate . </s> It Ġis Ġinteresting Ġto Ġnote Ġthat Ġdespite Ġthe Ġreasonably Ġhigh Ġaccuracy Ġrates Ġachieved Ġusing ĠL SA Ġand ĠS VM Ġwith ĠB OW Ġfeatures , Ġthey Ġpoorly Ġperformed Ġwhen Ġusing Ġw 2 v Ġfeatures . </s> Among Ġthe Ġbas elines , Ġthe Ġbest Ġmethod Ġwas ĠMN B Ġwith Ġtf B OW Ġfeatures , Ġwith Ġan Ġaccuracy Ġof Ġ91 . 47 %, Ġbeing Ġthe Ġonly Ġconventional Ġmethod Ġto Ġoutper form ĠMSM . ĠMSM Ġwith Ġw 2 v Ġhad Ġan Ġaccuracy Ġrate Ġof Ġ90 . 62 %, Ġwith Ġthe Ġbest Ġresults Ġachieved Ġwith Ġword Ġsub space Ġdimensions Ġfor Ġthe Ġtraining Ġclasses Ġranging Ġfrom Ġ150 Ġto Ġ181 , Ġand Ġfor Ġthe Ġquery Ġranging Ġfrom Ġ3 Ġto Ġ217 . ĠInc orpor ating Ġthe Ġfrequency Ġinformation Ġin Ġthe Ġsub space Ġmodeling Ġresulted Ġin Ġhigher Ġaccuracy , Ġwith ĠTF - MS M Ġachieving Ġ92 . 01 %, Ġwith Ġdimensions Ġof Ġword Ġsub sp aces Ġfor Ġtraining Ġclasses Ġranging Ġfrom Ġ150 Ġto Ġ172 , Ġand Ġfor Ġthe Ġquery , Ġranging Ġfrom Ġ2 Ġto Ġ109 . ĠTo Ġconfirm Ġthat ĠTF - MS M Ġis Ġsignificantly Ġmore Ġaccurate Ġthan ĠMN B , Ġwe Ġperformed Ġa Ġt - test Ġto Ġcompare Ġtheir Ġresults . ĠIt Ġresulted Ġin Ġa Ġp - value Ġof Ġ0 . 031 , Ġwhich Ġshows Ġthat Ġat Ġa Ġ95 % Ġsignificance Ġlevel , ĠTF - MS M Ġhas Ġproduced Ġbetter Ġresults . </s> Discussion </s> Given Ġthe Ġobservation Ġof Ġthe Ġe igen values Ġdistribution Ġof Ġword Ġvectors , Ġwe Ġcould Ġsee Ġthat Ġword Ġvectors Ġthat Ġbelong Ġto Ġthe Ġsame Ġcontext , Ġi . e ., Ġsame Ġclass , Ġare Ġsuitable Ġfor Ġsub space Ġrepresentation . ĠOur Ġanalysis Ġshowed Ġthat Ġhalf Ġof Ġthe Ġword Ġvector Ġspace Ġdimensions Ġsuffice Ġto Ġrepresent Ġmost Ġof Ġthe Ġvariability Ġof Ġthe Ġdata Ġin Ġeach Ġclass Ġof Ġthe ĠReuters - 8 Ġdatabase . </s> The Ġresults Ġfrom Ġthe Ġtext Ġclassification Ġexperiment Ġshowed Ġthat Ġsub space - based Ġmethods Ġperformed Ġbetter Ġthan Ġthe Ġtext Ġclassification Ġmethods Ġdiscussed Ġin Ġthis Ġwork . ĠUltimately , Ġour Ġproposed ĠTF Ġweighted Ġword Ġsub space Ġwith ĠMSM Ġsurpassed Ġall Ġthe Ġother Ġmethods . Ġword 2 vec Ġfeatures Ġare Ġreliable Ġtools Ġto Ġrepresent Ġthe Ġsemantic Ġmeaning Ġof Ġthe Ġwords Ġand Ġwhen Ġtreated Ġas Ġsets Ġof Ġword Ġvectors , Ġthey Ġare Ġcapable Ġof Ġrepresenting Ġthe Ġcontent Ġof Ġtexts . ĠHowever , Ġdespite Ġthe Ġfact Ġthat Ġword Ġvectors Ġcan Ġbe Ġtreated Ġseparately , Ġconventional Ġmethods Ġsuch Ġas ĠS VM Ġand ĠL SA Ġmay Ġnot Ġbe Ġsuitable Ġfor Ġtext Ġclassification Ġusing Ġword Ġvectors . </s> Among Ġthe Ġconventional Ġmethods , ĠL SA Ġand ĠS VM Ġachieved Ġabout Ġ86 % Ġand Ġ89 %, Ġrespectively , Ġwhen Ġusing Ġbag - of - words Ġfeatures . ĠInterestingly , Ġboth Ġmethods Ġhad Ġbetter Ġperformance Ġwhen Ġusing Ġbinary Ġweights . ĠFor ĠL SA , Ġwe Ġcan Ġsee Ġthat Ġdespite Ġthe Ġslight Ġdifferences Ġin Ġthe Ġperformance , Ġtf id f B OW Ġrequired Ġapprox im ations Ġwith Ġsmaller Ġdimensions . ĠS VM Ġhad Ġthe Ġlowest Ġaccuracy Ġrate Ġwhen Ġusing Ġthe Ġtf id f B OW Ġfeatures . ĠOne Ġpossible Ġexplanation Ġfor Ġthis Ġis Ġthat ĠTF - ID F Ġweights Ġare Ġuseful Ġwhen Ġrare Ġwords Ġand Ġvery Ġfrequent Ġwords Ġexist Ġin Ġthe Ġcorpus , Ġgiving Ġhigher Ġweights Ġfor Ġrare Ġwords Ġand Ġlower Ġweights Ġfor Ġcommon Ġwords . ĠSince Ġwe Ġremoved Ġthe Ġstop Ġwords , Ġthe Ġmost Ġfrequent Ġwords Ġamong Ġthe Ġtraining Ġdocuments Ġwere Ġnot Ġconsidered Ġand , Ġtherefore , Ġusing ĠTF - ID F Ġweights Ġdid Ġnot Ġimprove Ġthe Ġresults . </s> Only ĠMN B Ġwith Ġtf B OW Ġperformed Ġbetter Ġthan ĠMSM . ĠThis Ġresult Ġmay Ġbe Ġbecause Ġtf B OW Ġfeatures Ġencode Ġthe Ġword Ġfrequencies , Ġwhile ĠMSM Ġonly Ġconsiders Ġa Ġsingle Ġoccurrence Ġof Ġwords . ĠWhen Ġincorporating Ġthe Ġword Ġfrequencies Ġwith Ġour ĠTF Ġweighted Ġword Ġsub space , Ġwe Ġachieved Ġa Ġhigher Ġaccuracy Ġof Ġ92 . 01 %, Ġperforming Ġbetter Ġthan ĠMN B Ġat Ġa Ġsignificance Ġlevel Ġof Ġ95 %. </s> Con clusions Ġand ĠFuture ĠWork </s> In Ġthis Ġpaper , Ġwe Ġproposed Ġa Ġnew Ġmethod Ġfor Ġtext Ġclassification , Ġbased Ġon Ġthe Ġnovel Ġconcept Ġof Ġword Ġsub space Ġunder Ġthe ĠMSM Ġframework . ĠWe Ġalso Ġproposed Ġthe Ġterm - frequency Ġweighted Ġword Ġsub space Ġwhich Ġcan Ġincorporate Ġthe Ġfrequency Ġof Ġwords Ġdirectly Ġin Ġthe Ġmodeling Ġof Ġthe Ġsub space Ġby Ġusing Ġa Ġweighted Ġversion Ġof ĠPC A . </s> Most Ġof Ġthe Ġconventional Ġtext Ġclassification Ġmethods Ġare Ġbased Ġon Ġthe Ġbag - of - words Ġfeatures , Ġwhich Ġare Ġvery Ġsimple Ġto Ġcompute Ġand Ġhad Ġbeen Ġproved Ġto Ġproduce Ġpositive Ġresults . ĠHowever , Ġbag - of - words Ġare Ġcommonly Ġhigh Ġdimensional Ġmodels , Ġwith Ġa Ġsparse Ġrepresentation , Ġwhich Ġis Ġcomput ationally Ġheavy Ġto Ġmodel . ĠAlso , Ġbag - of - words Ġfail Ġto Ġconvey Ġthe Ġsemantic Ġmeaning Ġof Ġwords Ġinside Ġa Ġtext . ĠDue Ġto Ġthese Ġproblems , Ġneural Ġnetworks Ġstarted Ġto Ġbe Ġapplied Ġto Ġgenerate Ġa Ġvector Ġrepresentation Ġof Ġwords . ĠDespite Ġthe Ġfact Ġthat Ġthese Ġrepresentations Ġcan Ġencode Ġthe Ġsemantic Ġmeaning Ġof Ġwords , Ġconventional Ġmethods Ġdo Ġnot Ġwork Ġwell Ġwhen Ġconsidering Ġword Ġvectors Ġseparately . </s> In Ġour Ġwork , Ġwe Ġfocused Ġon Ġthe Ġword 2 vec Ġrepresentation , Ġwhich Ġcan Ġembed Ġthe Ġsemantic Ġstructure Ġof Ġwords , Ġrendering Ġvector Ġangles Ġas Ġa Ġuseful Ġmetric Ġto Ġshow Ġmeaningful Ġsimilarities Ġbetween Ġwords . ĠOur Ġexperiments Ġshowed Ġthat Ġour Ġword Ġsub space Ġmodeling Ġalong Ġwith Ġthe ĠMSM Ġoutper forms Ġmost Ġof Ġthe Ġconventional Ġmethods . ĠUltimately , Ġour ĠTF Ġweighted Ġsub space Ġformulation Ġresulted Ġin Ġsignificantly Ġhigher Ġaccuracy Ġwhen Ġcompared Ġto Ġall Ġconventional Ġtext Ġclassification Ġmethods Ġdiscussed Ġin Ġthis Ġwork . ĠIt Ġis Ġimportant Ġto Ġnote Ġthat Ġour Ġmethod Ġdoes Ġnot Ġconsider Ġthe Ġorder Ġof Ġthe Ġwords Ġin Ġa Ġtext , Ġresulting Ġin Ġa Ġloss Ġof Ġcontext Ġinformation . ĠAs Ġa Ġfuture Ġwork , Ġwe Ġwish Ġto Ġextend Ġour Ġword Ġsub space Ġconcept Ġfurther Ġin Ġmainly Ġtwo Ġdirections . ĠFirst , Ġwe Ġseek Ġto Ġencode Ġword Ġorder , Ġwhich Ġmay Ġenrich Ġthe Ġrepresentation Ġof Ġcontext Ġinformation . ĠSecond , Ġwe Ġwish Ġto Ġmodel Ġdynamic Ġcontext Ġchange , Ġenabling Ġanalysis Ġof Ġlarge Ġdocuments , Ġby Ġhaving Ġa Ġlong - short Ġmemory Ġto Ġinterpret Ġinformation Ġusing Ġcues Ġfrom Ġdifferent Ġparts Ġof Ġa Ġtext . </s> Acknowled gment </s> This Ġwork Ġis Ġsupported Ġby ĠJS PS ĠK AK EN HI ĠGrant ĠNumber ĠJP 16 H 028 42 Ġand Ġthe ĠJapanese ĠMinistry Ġof ĠEducation , ĠCulture , ĠSports , ĠScience , Ġand ĠTechnology Ġ( M EXT ) Ġscholarship .