text classification for themes including sentiment, web-page, science, medical and healthcare
ABSTRACTIVE
In this section, we first justify the approach when there exists unbalance in the number of labeled features or in class distribution. Then, to test the influence of $\lambda $ , we conduct some experiments with the method which incorporates the KL divergence of class distribution. Last, we evaluate our approaches in 9 commonly used text classification datasets. We set $\lambda = 5|K|$ by default in all experiments unless there is explicit declaration. The baseline we choose here is GE-FL BIBREF0 , a method based on generalization expectation criteria. We evaluate our methods on several commonly used datasets whose themes range from sentiment, web-page, science to medical and healthcare. We use bag-of-words feature and remove stopwords in the preprocess stage. Though we have labels of all documents, we do not use them during the learning process, instead, we use the label of features.
1503.00841
Introduction </s> We Ġposs es Ġa Ġwealth Ġof Ġprior Ġknowledge Ġabout Ġmany Ġnatural Ġlanguage Ġprocessing Ġtasks . ĠFor Ġexample , Ġin Ġtext Ġcategor ization , Ġwe Ġknow Ġthat Ġwords Ġsuch Ġas ĠNBA , Ġplayer , Ġand Ġbasketball Ġare Ġstrong Ġindicators Ġof Ġthe Ġsports Ġcategory ĠB IB REF 0 Ġ, Ġand Ġwords Ġlike Ġterrible , Ġboring , Ġand Ġmessing Ġindicate Ġa Ġnegative Ġpol arity Ġwhile Ġwords Ġlike Ġperfect , Ġexciting , Ġand Ġmoving Ġsuggest Ġa Ġpositive Ġpol arity Ġin Ġsentiment Ġclassification . </s> A Ġkey Ġproblem Ġarisen Ġhere , Ġis Ġhow Ġto Ġleverage Ġsuch Ġknowledge Ġto Ġguide Ġthe Ġlearning Ġprocess , Ġan Ġinteresting Ġproblem Ġfor Ġboth ĠN LP Ġand Ġmachine Ġlearning Ġcommunities . ĠPrevious Ġstudies Ġaddressing Ġthe Ġproblem Ġfall Ġinto Ġseveral Ġlines . ĠFirst , Ġto Ġleverage Ġprior Ġknowledge Ġto Ġlabel Ġdata ĠB IB REF 1 Ġ, ĠB IB REF 2 Ġ. ĠSecond , Ġto Ġencode Ġprior Ġknowledge Ġwith Ġa Ġprior Ġon Ġparameters , Ġwhich Ġcan Ġbe Ġcommonly Ġseen Ġin Ġmany ĠBay esian Ġapproaches ĠB IB REF 3 Ġ, ĠB IB REF 4 Ġ. ĠThird , Ġto Ġformal ise Ġprior Ġknowledge Ġwith Ġadditional Ġvariables Ġand Ġdependencies ĠB IB REF 5 Ġ. ĠLast , Ġto Ġuse Ġprior Ġknowledge Ġto Ġcontrol Ġthe Ġdistributions Ġover Ġlatent Ġoutput Ġvariables ĠB IB REF 6 Ġ, ĠB IB REF 7 Ġ, ĠB IB REF 8 Ġ, Ġwhich Ġmakes Ġthe Ġoutput Ġvariables Ġeasily Ġinterpret able . </s> However , Ġa Ġcrucial Ġproblem , Ġwhich Ġhas Ġrarely Ġbeen Ġaddressed , Ġis Ġthe Ġbias Ġin Ġthe Ġprior Ġknowledge Ġthat Ġwe Ġsupply Ġto Ġthe Ġlearning Ġmodel . ĠWould Ġthe Ġmodel Ġbe Ġrobust Ġor Ġsensitive Ġto Ġthe Ġprior Ġknowledge ? ĠOr , Ġwhich Ġkind Ġof Ġknowledge Ġis Ġappropriate Ġfor Ġthe Ġtask ? ĠLet 's Ġsee Ġan Ġexample : Ġwe Ġmay Ġbe Ġa Ġbaseball Ġfan Ġbut Ġunfamiliar Ġwith Ġhockey Ġso Ġthat Ġwe Ġcan Ġprovide Ġa Ġfew Ġnumber Ġof Ġfeature Ġwords Ġof Ġbaseball , Ġbut Ġmuch Ġless Ġof Ġhockey Ġfor Ġa Ġbaseball - h ockey Ġclassification Ġtask . ĠSuch Ġprior Ġknowledge Ġmay Ġmislead Ġthe Ġmodel Ġwith Ġheavy Ġbias Ġto Ġbaseball . ĠIf Ġthe Ġmodel Ġcannot Ġhandle Ġthis Ġsituation Ġappropriately , Ġthe Ġperformance Ġmay Ġbe Ġundesirable . </s> In Ġthis Ġpaper , Ġwe Ġinvestigate Ġinto Ġthe Ġproblem Ġin Ġthe Ġframework Ġof ĠGeneral ized ĠExpect ation ĠCrit eria ĠB IB REF 7 Ġ. ĠThe Ġstudy Ġaims Ġto Ġreveal Ġthe Ġfactors Ġof Ġreducing Ġthe Ġsens ibility Ġof Ġthe Ġprior Ġknowledge Ġand Ġtherefore Ġto Ġmake Ġthe Ġmodel Ġmore Ġrobust Ġand Ġpractical . ĠTo Ġthis Ġend , Ġwe Ġintroduce Ġauxiliary Ġregular ization Ġterms Ġin Ġwhich Ġour Ġprior Ġknowledge Ġis Ġformal ized Ġas Ġdistribution Ġover Ġoutput Ġvariables . ĠRecall Ġthe Ġexample Ġjust Ġmentioned , Ġthough Ġwe Ġdo Ġnot Ġhave Ġenough Ġknowledge Ġto Ġprovide Ġfeatures Ġfor Ġclass Ġhockey , Ġit Ġis Ġeasy Ġfor Ġus Ġto Ġprovide Ġsome Ġneutral Ġwords , Ġnamely Ġwords Ġthat Ġare Ġnot Ġstrong Ġindicators Ġof Ġany Ġclass , Ġlike Ġplayer Ġhere . ĠAs Ġone Ġof Ġthe Ġfactors Ġrevealed Ġin Ġthis Ġpaper , Ġsupplying Ġneutral Ġfeature Ġwords Ġcan Ġboost Ġthe Ġperformance Ġremarkably , Ġmaking Ġthe Ġmodel Ġmore Ġrobust . </s> More Ġattract ively , Ġwe Ġdo Ġnot Ġneed Ġmanual Ġannotation Ġto Ġlabel Ġthese Ġneutral Ġfeature Ġwords Ġin Ġour Ġproposed Ġapproach . </s> More Ġspecifically , Ġwe Ġexplore Ġthree Ġregular ization Ġterms Ġto Ġaddress Ġthe Ġproblem : Ġ( 1 ) Ġa Ġregular ization Ġterm Ġassociated Ġwith Ġneutral Ġfeatures ; Ġ( 2 ) Ġthe Ġmaximum Ġentropy Ġof Ġclass Ġdistribution Ġregular ization Ġterm ; Ġand Ġ( 3 ) Ġthe ĠKL Ġdivergence Ġbetween Ġreference Ġand Ġpredicted Ġclass Ġdistribution . ĠFor Ġthe Ġfirst Ġmanner , Ġwe Ġsimply Ġuse Ġthe Ġmost Ġcommon Ġfeatures Ġas Ġneutral Ġfeatures Ġand Ġassume Ġthe Ġneutral Ġfeatures Ġare Ġdistributed Ġuniformly Ġover Ġclass Ġlabels . ĠFor Ġthe Ġsecond Ġand Ġthird Ġone , Ġwe Ġassume Ġwe Ġhave Ġsome Ġknowledge Ġabout Ġthe Ġclass Ġdistribution Ġwhich Ġwill Ġbe Ġdetailed Ġsoon Ġlater . </s> To Ġsummarize , Ġthe Ġmain Ġcontributions Ġof Ġthis Ġwork Ġare Ġas Ġfollows : </s> The Ġrest Ġof Ġthe Ġpaper Ġis Ġstructured Ġas Ġfollows : ĠIn ĠSection Ġ2 , Ġwe Ġbriefly Ġdescribe Ġthe Ġgeneralized Ġexpectation Ġcriteria Ġand Ġpresent Ġthe Ġproposed Ġregular ization Ġterms . ĠIn ĠSection Ġ3 , Ġwe Ġconduct Ġextensive Ġexperiments Ġto Ġjustify Ġthe Ġproposed Ġmethods . ĠWe Ġsurvey Ġrelated Ġwork Ġin ĠSection Ġ4 , Ġand Ġsummarize Ġour Ġwork Ġin ĠSection Ġ5 . </s> Method </s> We Ġaddress Ġthe Ġrobust ness Ġproblem Ġon Ġtop Ġof ĠGE - FL ĠB IB REF 0 Ġ, Ġa ĠGE Ġmethod Ġwhich Ġlever ages Ġlabeled Ġfeatures Ġas Ġprior Ġknowledge . ĠA Ġlabeled Ġfeature Ġis Ġa Ġstrong Ġindicator Ġof Ġa Ġspecific Ġclass Ġand Ġis Ġmanually Ġprovided Ġto Ġthe Ġclass ifier . ĠFor Ġexample , Ġwords Ġlike Ġamazing , Ġexciting Ġcan Ġbe Ġlabeled Ġfeatures Ġfor Ġclass Ġpositive Ġin Ġsentiment Ġclassification . </s> General ized ĠExpect ation ĠCrit eria </s> General ized Ġexpectation Ġ( GE ) Ġcriteria ĠB IB REF 7 Ġprovides Ġus Ġa Ġnatural Ġway Ġto Ġdirectly Ġconst rain Ġthe Ġmodel Ġin Ġthe Ġpreferred Ġdirection . ĠFor Ġexample , Ġwhen Ġwe Ġknow Ġthe Ġproportion Ġof Ġeach Ġclass Ġof Ġthe Ġdataset Ġin Ġa Ġclassification Ġtask , Ġwe Ġcan Ġguide Ġthe Ġmodel Ġto Ġpredict Ġout Ġa Ġpre - specified Ġclass Ġdistribution . </s> Form ally , Ġin Ġa Ġparameter Ġestimation Ġobjective Ġfunction , Ġa ĠGE Ġterm Ġexpresses Ġpreferences Ġon Ġthe Ġvalue Ġof Ġsome Ġconstraint Ġfunctions Ġabout Ġthe Ġmodel 's Ġexpectation . ĠGiven Ġa Ġconstraint Ġfunction Ġ$ G ({ \ rm Ġx }, Ġy ) $ Ġ, Ġa Ġconditional Ġmodel Ġdistribution Ġ$ p _ \ the ta Ġ( y | \ rm Ġx ) $ Ġ, Ġan Ġempirical Ġdistribution Ġ$\ t ilde { p } ({ \ rm Ġx }) $ Ġover Ġinput Ġsamples Ġand Ġa Ġscore Ġfunction Ġ$ S $ Ġ, Ġa ĠGE Ġterm Ġcan Ġbe Ġexpressed Ġas Ġfollows : </s> $$ S ( E _{ \ t ilde { p } ({ \ rm Ġx }) } [ E _{ p _ \ the ta Ġ( y | {\ rm Ġx }) } [ G ({ \ rm Ġx }, Ġy ) ]] ) $$ Ġ Ġ Ġ( E q . Ġ4 ) </s> Learning Ġfrom ĠLab eled ĠFeatures </s> D ruck Ġet Ġal . Ġge - fl Ġproposed ĠGE - FL Ġto Ġlearn Ġfrom Ġlabeled Ġfeatures Ġusing Ġgeneralized Ġexpectation Ġcriteria . ĠWhen Ġgiven Ġa Ġset Ġof Ġlabeled Ġfeatures Ġ$ K $ Ġ, Ġthe Ġreference Ġdistribution Ġover Ġclasses Ġof Ġthese Ġfeatures Ġis Ġden oted Ġby Ġ$\ hat { p } ( y | Ġx _ k ), Ġk Ġ\ in ĠK $ Ġ. ĠGE - FL Ġintroduces Ġthe Ġdivergence Ġbetween Ġthis Ġreference Ġdistribution Ġand Ġthe Ġmodel Ġpredicted Ġdistribution Ġ$ p _ \ the ta Ġ( y Ġ| Ġx _ k ) $ Ġ, Ġas Ġa Ġterm Ġof Ġthe Ġobjective Ġfunction : </s> $$ \ math cal Ġ{ O } Ġ= Ġ\ sum Ġ_ { k Ġ\ in ĠK } ĠKL (\ hat { p } ( y | x _ k ) Ġ|| Ġp _ \ the ta Ġ( y Ġ| Ġx _ k )) Ġ+ Ġ\ sum Ġ_ { y , i } Ġ\ frac {\ the ta Ġ_ { yi } ^ 2 }{ 2 Ġ\ s igma Ġ^ 2 } $$ Ġ Ġ Ġ( E q . Ġ6 ) </s> where Ġ$\ the ta Ġ_ { yi } $ Ġis Ġthe Ġmodel Ġparameter Ġwhich Ġindicates Ġthe Ġimportance Ġof Ġword Ġ$ i $ Ġto Ġclass Ġ$ y $ Ġ. ĠThe Ġpredicted Ġdistribution Ġ$ p _ \ the ta Ġ( y Ġ| Ġx _ k ) $ Ġcan Ġbe Ġexpressed Ġas Ġfollows : Ġ$ Ġp _ \ the ta Ġ( y Ġ| Ġx _ k ) Ġ= Ġ\ frac { 1 }{ C _ k } Ġ\ sum Ġ_ {\ rm Ġx } Ġp _ \ the ta Ġ( y | {\ rm Ġx }) I ( x _ k ) Ġ$ </s> in Ġwhich Ġ$ I ( x _ k ) $ Ġis Ġ1 Ġif Ġfeature Ġ$ k $ Ġoccurs Ġin Ġinstance Ġ${ \ rm Ġx } $ Ġand Ġ0 Ġotherwise , Ġ$ C _ k Ġ= Ġ\ sum Ġ_ {\ rm Ġx } ĠI ( x _ k ) $ Ġis Ġthe Ġnumber Ġof Ġinstances Ġwith Ġa Ġnon - zero Ġvalue Ġof Ġfeature Ġ$ k $ Ġ, Ġand Ġ$ p _ \ the ta Ġ( y | {\ rm Ġx }) $ Ġtakes Ġa Ġsoft max Ġform Ġas Ġfollows : Ġ$ Ġp _ \ the ta Ġ( y | {\ rm Ġx }) Ġ= Ġ\ frac { 1 }{ Z (\ rm Ġx ) }\ exp Ġ( \ sum Ġ_ i Ġ\ the ta Ġ_ { yi } x _ i ). Ġ$ </s> To Ġsolve Ġthe Ġoptimization Ġproblem , ĠL - BF GS Ġcan Ġbe Ġused Ġfor Ġparameter Ġestimation . </s> In Ġthe Ġframework Ġof ĠGE , Ġthis Ġterm Ġcan Ġbe Ġobtained Ġby Ġsetting Ġthe Ġconstraint Ġfunction Ġ$ G ({ \ rm Ġx }, Ġy ) Ġ= Ġ\ frac { 1 }{ C _ k } Ġ\ vec { I } Ġ( y ) I ( x _ k ) $ Ġ, Ġwhere Ġ$\ vec { I } ( y ) $ Ġis Ġan Ġindicator Ġvector Ġwith Ġ1 Ġat Ġthe Ġindex Ġcorresponding Ġto Ġlabel Ġ$ y $ Ġand Ġ0 Ġelsewhere . </s> Regular ization ĠTerms </s> GE - FL Ġreduces Ġthe Ġheavy Ġload Ġof Ġinstance Ġannotation Ġand Ġperforms Ġwell Ġwhen Ġwe Ġprovide Ġprior Ġknowledge Ġwith Ġno Ġbias . ĠIn Ġour Ġexperiments , Ġwe Ġobserve Ġthat Ġcomparable Ġnumbers Ġof Ġlabeled Ġfeatures Ġfor Ġeach Ġclass Ġhave Ġto Ġbe Ġsupplied . ĠBut Ġas Ġmentioned Ġbefore , Ġit Ġis Ġoften Ġthe Ġcase Ġthat Ġwe Ġare Ġnot Ġable Ġto Ġprovide Ġenough Ġknowledge Ġfor Ġsome Ġof Ġthe Ġclasses . ĠFor Ġthe Ġbaseball - h ockey Ġclassification Ġtask , Ġas Ġshown Ġbefore , ĠGE - FL Ġwill Ġpredict Ġmost Ġof Ġthe Ġinstances Ġas Ġbaseball . ĠIn Ġthis Ġsection , Ġwe Ġwill Ġshow Ġthree Ġterms Ġto Ġmake Ġthe Ġmodel Ġmore Ġrobust . </s> Ne utral Ġfeatures Ġare Ġfeatures Ġthat Ġare Ġnot Ġinformative Ġindicator Ġof Ġany Ġclasses , Ġfor Ġinstance , Ġword Ġplayer Ġto Ġthe Ġbaseball - h ockey Ġclassification Ġtask . ĠSuch Ġfeatures Ġare Ġusually Ġfrequent Ġwords Ġacross Ġall Ġcategories . ĠWhen Ġwe Ġset Ġthe Ġpreference Ġdistribution Ġof Ġthe Ġneutral Ġfeatures Ġto Ġbe Ġuniform Ġdistributed , Ġthese Ġneutral Ġfeatures Ġwill Ġprevent Ġthe Ġmodel Ġfrom Ġbi asing Ġto Ġthe Ġclass Ġthat Ġhas Ġa Ġdominate Ġnumber Ġof Ġlabeled Ġfeatures . </s> Form ally , Ġgiven Ġa Ġset Ġof Ġneutral Ġfeatures Ġ$ K ^{ ^ {\ prime Ġ}} $ Ġ, Ġthe Ġuniform Ġdistribution Ġis Ġ$\ hat { p } _ u ( y | x _ k ) Ġ= Ġ\ frac { 1 }{ | C | }, Ġk Ġ\ in ĠK ^{ ^ {\ prime Ġ}} $ Ġ, Ġwhere Ġ$ | C | $ Ġis Ġthe Ġnumber Ġof Ġclasses . ĠThe Ġobjective Ġfunction Ġwith Ġthe Ġnew Ġterm Ġbecomes </s> $$ \ math cal Ġ{ O } _{ NE } Ġ= Ġ\ math cal Ġ{ O } Ġ+ Ġ\ sum Ġ_ { k Ġ\ in ĠK ^{ ^ {\ prime Ġ} }} ĠKL (\ hat { p } _ u ( y | x _ k ) Ġ|| Ġp _ \ the ta Ġ( y Ġ| Ġx _ k )). $$ Ġ Ġ Ġ( E q . Ġ9 ) </s> Note Ġthat Ġwe Ġdo Ġnot Ġneed Ġmanual Ġannotation Ġto Ġprovide Ġneutral Ġfeatures . ĠOne Ġsimple Ġway Ġis Ġto Ġtake Ġthe Ġmost Ġcommon Ġfeatures Ġas Ġneutral Ġfeatures . ĠExperimental Ġresults Ġshow Ġthat Ġthis Ġstrategy Ġworks Ġsuccessfully . </s> Another Ġway Ġto Ġprevent Ġthe Ġmodel Ġfrom Ġdrifting Ġfrom Ġthe Ġdesired Ġdirection Ġis Ġto Ġconst rain Ġthe Ġpredicted Ġclass Ġdistribution Ġon Ġunl abel ed Ġdata . ĠWhen Ġlacking Ġknowledge Ġabout Ġthe Ġclass Ġdistribution Ġof Ġthe Ġdata , Ġone Ġfeasible Ġway Ġis Ġto Ġtake Ġmaximum Ġentropy Ġprinciple , Ġas Ġbelow : </s> $$ \ math cal Ġ{ O } _{ ME } Ġ= Ġ\ math cal Ġ{ O } Ġ+ Ġ\ lambda Ġ\ sum Ġ_ { y } Ġp ( y ) Ġ\ log Ġp ( y ) $$ Ġ Ġ Ġ( E q . Ġ11 ) </s> where Ġ$ p ( y ) $ Ġis Ġthe Ġpredicted Ġclass Ġdistribution , Ġgiven Ġby Ġ$ Ġp ( y ) Ġ= Ġ\ frac { 1 }{ | X | } Ġ\ sum Ġ_ {\ rm Ġx } Ġp _ \ the ta Ġ( y Ġ| Ġ\ rm Ġx ). Ġ$ ĠTo Ġcontrol Ġthe Ġinfluence Ġof Ġthis Ġterm Ġon Ġthe Ġoverall Ġobjective Ġfunction , Ġwe Ġcan Ġtune Ġ$\ lambda Ġ$ Ġaccording Ġto Ġthe Ġdifference Ġin Ġthe Ġnumber Ġof Ġlabeled Ġfeatures Ġof Ġeach Ġclass . ĠIn Ġthis Ġpaper , Ġwe Ġsimply Ġset Ġ$\ lambda Ġ$ Ġto Ġbe Ġproportional Ġto Ġthe Ġtotal Ġnumber Ġof Ġlabeled Ġfeatures , Ġsay Ġ$\ lambda Ġ= Ġ\ beta Ġ| K | $ Ġ. </s> This Ġmaximum Ġentropy Ġterm Ġcan Ġbe Ġderived Ġby Ġsetting Ġthe Ġconstraint Ġfunction Ġto Ġ$ G ({ \ rm Ġx }, Ġy ) Ġ= Ġ\ vec { I } ( y ) $ Ġ. ĠTherefore , Ġ$ E _{ p _ \ the ta Ġ( y | {\ rm Ġx }) } [ G ({ \ rm Ġx }, Ġy )] $ Ġis Ġjust Ġthe Ġmodel Ġdistribution Ġ$ p _ \ the ta Ġ( y | {\ rm Ġx }) $ Ġand Ġits Ġexpectation Ġwith Ġthe Ġempirical Ġdistribution Ġ$\ t ilde { p } (\ rm Ġx ) $ Ġis Ġsimply Ġthe Ġaverage Ġover Ġinput Ġsamples , Ġnamely Ġ$ p ( y ) $ Ġ. ĠWhen Ġ$ S $ Ġtakes Ġthe Ġmaximum Ġentropy Ġform , Ġwe Ġcan Ġderive Ġthe Ġobjective Ġfunction Ġas Ġabove . </s> Sometimes , Ġwe Ġhave Ġalready Ġhad Ġmuch Ġknowledge Ġabout Ġthe Ġcorpus , Ġand Ġcan Ġestimate Ġthe Ġclass Ġdistribution Ġroughly Ġwithout Ġlabeling Ġinstances . ĠTherefore , Ġwe Ġintroduce Ġthe ĠKL Ġdivergence Ġbetween Ġthe Ġpredicted Ġand Ġreference Ġclass Ġdistributions Ġinto Ġthe Ġobjective Ġfunction . </s> Given Ġthe Ġpreference Ġclass Ġdistribution Ġ$\ hat { p } ( y ) $ Ġ, Ġwe Ġmodify Ġthe Ġobjective Ġfunction Ġas Ġfollows : </s> $$ \ math cal Ġ{ O } _{ K L } Ġ& = Ġ\ math cal Ġ{ O } Ġ+ Ġ\ lambda ĠKL (\ hat { p } ( y ) Ġ|| Ġp ( y )) $$ Ġ Ġ Ġ( E q . Ġ13 ) </s> Similarly , Ġwe Ġset Ġ$\ lambda Ġ= Ġ\ beta Ġ| K | $ Ġ. </s> This Ġdivergence Ġterm Ġcan Ġbe Ġderived Ġby Ġsetting Ġthe Ġconstraint Ġfunction Ġto Ġ$ G ({ \ rm Ġx }, Ġy ) Ġ= Ġ\ vec { I } ( y ) $ Ġand Ġsetting Ġthe Ġscore Ġfunction Ġto Ġ$ S (\ hat { p }, Ġp ) Ġ= Ġ\ sum Ġ_ i Ġ\ hat { p } _ i Ġ\ log Ġ\ frac {\ hat { p } _ i }{ p _ i } $ Ġ, Ġwhere Ġ$ p $ Ġand Ġ$\ hat { p } $ Ġare Ġdistributions . ĠNote Ġthat Ġthis Ġregular ization Ġterm Ġinvolves Ġthe Ġreference Ġclass Ġdistribution Ġwhich Ġwill Ġbe Ġdiscussed Ġlater . </s> Exper iments </s> In Ġthis Ġsection , Ġwe Ġfirst Ġjustify Ġthe Ġapproach Ġwhen Ġthere Ġexists Ġun balance Ġin Ġthe Ġnumber Ġof Ġlabeled Ġfeatures Ġor Ġin Ġclass Ġdistribution . ĠThen , Ġto Ġtest Ġthe Ġinfluence Ġof Ġ$\ lambda Ġ$ Ġ, Ġwe Ġconduct Ġsome Ġexperiments Ġwith Ġthe Ġmethod Ġwhich Ġincorporates Ġthe ĠKL Ġdivergence Ġof Ġclass Ġdistribution . ĠLast , Ġwe Ġevaluate Ġour Ġapproaches Ġin Ġ9 Ġcommonly Ġused Ġtext Ġclassification Ġdatasets . ĠWe Ġset Ġ$\ lambda Ġ= Ġ5 | K | $ Ġby Ġdefault Ġin Ġall Ġexperiments Ġunless Ġthere Ġis Ġexplicit Ġdeclaration . ĠThe Ġbaseline Ġwe Ġchoose Ġhere Ġis ĠGE - FL ĠB IB REF 0 Ġ, Ġa Ġmethod Ġbased Ġon Ġgeneral ization Ġexpectation Ġcriteria . </s> Data ĠPrepar ation </s> We Ġevaluate Ġour Ġmethods Ġon Ġseveral Ġcommonly Ġused Ġdatasets Ġwhose Ġthemes Ġrange Ġfrom Ġsentiment , Ġweb - page , Ġscience Ġto Ġmedical Ġand Ġhealthcare . ĠWe Ġuse Ġbag - of - words Ġfeature Ġand Ġremove Ġstop words Ġin Ġthe Ġpre process Ġstage . ĠThough Ġwe Ġhave Ġlabels Ġof Ġall Ġdocuments , Ġwe Ġdo Ġnot Ġuse Ġthem Ġduring Ġthe Ġlearning Ġprocess , Ġinstead , Ġwe Ġuse Ġthe Ġlabel Ġof Ġfeatures . </s> The Ġmovie Ġdataset , Ġin Ġwhich Ġthe Ġtask Ġis Ġto Ġclassify Ġthe Ġmovie Ġreviews Ġas Ġpositive Ġor Ġneg t ive , Ġis Ġused Ġfor Ġtesting Ġthe Ġproposed Ġapproaches Ġwith Ġun balanced Ġlabeled Ġfeatures , Ġun balanced Ġdatasets Ġor Ġdifferent Ġ$\ lambda Ġ$ Ġparameters . ĠAll Ġun balanced Ġdatasets Ġare Ġconstructed Ġbased Ġon Ġthe Ġmovie Ġdataset Ġby Ġrandomly Ġremoving Ġdocuments Ġof Ġthe Ġpositive Ġclass . ĠFor Ġeach Ġexperiment , Ġwe Ġconduct Ġ10 - fold Ġcross Ġvalidation . </s> As Ġdescribed Ġin ĠB IB REF 0 Ġ, Ġthere Ġare Ġtwo Ġways Ġto Ġobtain Ġlabeled Ġfeatures . ĠThe Ġfirst Ġway Ġis Ġto Ġuse Ġinformation Ġgain . ĠWe Ġfirst Ġcalculate Ġthe Ġmutual Ġinformation Ġof Ġall Ġfeatures Ġaccording Ġto Ġthe Ġlabels Ġof Ġthe Ġdocuments Ġand Ġselect Ġthe Ġtop Ġ20 Ġas Ġlabeled Ġfeatures Ġfor Ġeach Ġclass Ġas Ġa Ġfeature Ġpool . ĠNote Ġthat Ġusing Ġinformation Ġgain Ġrequires Ġthe Ġdocument Ġlabel , Ġbut Ġthis Ġis Ġonly Ġto Ġsimulate Ġhow Ġwe Ġhuman Ġprovide Ġprior Ġknowledge Ġto Ġthe Ġmodel . ĠThe Ġsecond Ġway Ġis Ġto Ġuse ĠL DA ĠB IB REF 9 Ġto Ġselect Ġfeatures . ĠWe Ġuse Ġthe Ġsame Ġselection Ġprocess Ġas ĠB IB REF 0 Ġ, Ġwhere Ġthey Ġfirst Ġtrain Ġa ĠL DA Ġon Ġthe Ġdataset , Ġand Ġthen Ġselect Ġthe Ġmost Ġprobable Ġfeatures Ġof Ġeach Ġtopic Ġ( s orted Ġby Ġ$ P ( w _ i | t _ j ) $ Ġ, Ġthe Ġprobability Ġof Ġword Ġ$ w _ i $ Ġgiven Ġtopic Ġ$ t _ j $ Ġ). </s> Similar Ġto ĠB IB REF 10 Ġ, ĠB IB REF 0 Ġ, Ġwe Ġestimate Ġthe Ġreference Ġdistribution Ġof Ġthe Ġlabeled Ġfeatures Ġusing Ġa Ġhe uristic Ġstrategy . ĠIf Ġthere Ġare Ġ$ | C | $ Ġclasses Ġin Ġtotal , Ġand Ġ$ n $ Ġclasses Ġare Ġassociated Ġwith Ġa Ġfeature Ġ$ k $ Ġ, Ġthe Ġprobability Ġthat Ġfeature Ġ$ k $ Ġis Ġrelated Ġwith Ġany Ġone Ġof Ġthe Ġ$ n $ Ġclasses Ġis Ġ$\ frac { 0 . 9 }{ n } $ Ġand Ġwith Ġany Ġother Ġclass Ġis Ġ$\ frac { 0 . 1 }{ | C | Ġ- Ġn } $ Ġ. </s> Ne utral Ġfeatures Ġare Ġthe Ġmost Ġfrequent Ġwords Ġafter Ġremoving Ġstop Ġwords , Ġand Ġtheir Ġreference Ġdistributions Ġare Ġuniformly Ġdistributed . ĠWe Ġuse Ġthe Ġtop Ġ10 Ġfrequent Ġwords Ġas Ġneutral Ġfeatures Ġin Ġall Ġexperiments . </s> With ĠUn balanced ĠLab eled ĠFeatures </s> In Ġthis Ġsection , Ġwe Ġevaluate Ġour Ġapproach Ġwhen Ġthere Ġis Ġun balanced Ġknowledge Ġon Ġthe Ġcategories Ġto Ġbe Ġclassified . ĠThe Ġlabeled Ġfeatures Ġare Ġobtained Ġthrough Ġinformation Ġgain . ĠTwo Ġsettings Ġare Ġchosen : </s> ( a ) ĠWe Ġrandomly Ġselect Ġ$ t Ġ\ in Ġ[ 1 , Ġ20 ] $ Ġfeatures Ġfrom Ġthe Ġfeature Ġpool Ġfor Ġone Ġclass , Ġand Ġonly Ġone Ġfeature Ġfor Ġthe Ġother . ĠThe Ġoriginal Ġbalanced Ġmovie Ġdataset Ġis Ġused Ġ( positive : negative = 1 : 1 ). </s> ( b ) ĠSimilar Ġto Ġ( a ), Ġbut Ġthe Ġdataset Ġis Ġun balanced , Ġobtained Ġby Ġrandomly Ġremoving Ġ75 % Ġpositive Ġdocuments Ġ( positive : negative = 1 : 4 ). </s> As Ġshown Ġin ĠFigure Ġ1 Ġ, ĠMaximum Ġentropy Ġprinciple Ġshows Ġimprovement Ġonly Ġon Ġthe Ġbalanced Ġcase . ĠAn Ġobvious Ġreason Ġis Ġthat Ġmaximum Ġentropy Ġonly Ġfavors Ġuniform Ġdistribution . </s> In cor por ating ĠNeutral Ġfeatures Ġperforms Ġsimilarly Ġto Ġmaximum Ġentropy Ġsince Ġwe Ġassume Ġthat Ġneutral Ġwords Ġare Ġuniformly Ġdistributed . ĠIts Ġaccuracy Ġdecreases Ġslowly Ġwhen Ġthe Ġnumber Ġof Ġlabeled Ġfeatures Ġbecomes Ġlarger Ġ( Ġ$ t > 4 $ Ġ) Ġ( Figure Ġ1 Ġ( a )), Ġsuggesting Ġthat Ġthe Ġmodel Ġgradually Ġbiases Ġto Ġthe Ġclass Ġwith Ġmore Ġlabeled Ġfeatures , Ġjust Ġlike ĠGE - FL . </s> In cor por ating Ġthe ĠKL Ġdivergence Ġof Ġclass Ġdistribution Ġperforms Ġmuch Ġbetter Ġthan ĠGE - FL Ġon Ġboth Ġbalanced Ġand Ġun balanced Ġdatasets . ĠThis Ġshows Ġthat Ġit Ġis Ġeffective Ġto Ġcontrol Ġthe Ġun balance Ġin Ġlabeled Ġfeatures Ġand Ġin Ġthe Ġdataset . </s> With ĠBalanced ĠLab eled ĠFeatures </s> We Ġalso Ġcompare Ġwith Ġthe Ġbaseline Ġwhen Ġthe Ġlabeled Ġfeatures Ġare Ġbalanced . ĠSimilar Ġto Ġthe Ġexperiment Ġabove , Ġthe Ġlabeled Ġfeatures Ġare Ġobtained Ġby Ġinformation Ġgain . ĠTwo Ġsettings Ġare Ġexperimented Ġwith : </s> ( a ) ĠWe Ġrandomly Ġselect Ġ$ t Ġ\ in Ġ[ 1 , Ġ20 ] $ Ġfeatures Ġfrom Ġthe Ġfeature Ġpool Ġfor Ġeach Ġclass , Ġand Ġconduct Ġcomparisons Ġon Ġthe Ġoriginal Ġbalanced Ġmovie Ġdataset Ġ( positive : neg t ive = 1 : 1 ). </s> ( b ) ĠSimilar Ġto Ġ( a ), Ġbut Ġthe Ġclass Ġdistribution Ġis Ġun balanced , Ġby Ġrandomly Ġremoving Ġ75 % Ġpositive Ġdocuments Ġ( positive : negative = 1 : 4 ). </s> Results Ġare Ġshown Ġin ĠFigure Ġ2 Ġ. ĠWhen Ġthe Ġdataset Ġis Ġbalanced Ġ( Figure Ġ2 Ġ( a )), Ġthere Ġis Ġlittle Ġdifference Ġbetween ĠGE - FL Ġand Ġour Ġmethods . ĠThe Ġreason Ġis Ġthat Ġthe Ġproposed Ġregular ization Ġterms Ġprovide Ġno Ġadditional Ġknowledge Ġto Ġthe Ġmodel Ġand Ġthere Ġis Ġno Ġbias Ġin Ġthe Ġlabeled Ġfeatures . ĠOn Ġthe Ġun balanced Ġdataset Ġ( Figure Ġ2 Ġ( b )), Ġincorporating ĠKL Ġdivergence Ġis Ġmuch Ġbetter Ġthan ĠGE - FL Ġsince Ġwe Ġprovide Ġadditional Ġknowledge ( the Ġtrue Ġclass Ġdistribution ), Ġbut Ġmaximum Ġentropy Ġand Ġneutral Ġfeatures Ġare Ġmuch Ġworse Ġbecause Ġforcing Ġthe Ġmodel Ġto Ġapproach Ġthe Ġuniform Ġdistribution Ġmis le ads Ġit . </s> With ĠUn balanced ĠClass ĠDistribut ions </s> Our Ġmethods Ġare Ġalso Ġevaluated Ġon Ġdatasets Ġwith Ġdifferent Ġun balanced Ġclass Ġdistributions . ĠWe Ġmanually Ġconstruct Ġseveral Ġmovie Ġdatasets Ġwith Ġclass Ġdistributions Ġof Ġ1 : 2 , Ġ1 : 3 , Ġ1 : 4 Ġby Ġrandomly Ġremoving Ġ50 %, Ġ67 %, Ġ75 % Ġpositive Ġdocuments . ĠThe Ġoriginal Ġbalanced Ġmovie Ġdataset Ġis Ġused Ġas Ġa Ġcontrol Ġgroup . ĠWe Ġtest Ġwith Ġboth Ġbalanced Ġand Ġun balanced Ġlabeled Ġfeatures . ĠFor Ġthe Ġbalanced Ġcase , Ġwe Ġrandomly Ġselect Ġ10 Ġfeatures Ġfrom Ġthe Ġfeature Ġpool Ġfor Ġeach Ġclass , Ġand Ġfor Ġthe Ġun balanced Ġcase , Ġwe Ġselect Ġ10 Ġfeatures Ġfor Ġone Ġclass , Ġand Ġ1 Ġfeature Ġfor Ġthe Ġother . ĠResults Ġare Ġshown Ġin ĠFigure Ġ3 Ġ. </s> Figure Ġ3 Ġ( a ) Ġshows Ġthat Ġwhen Ġthe Ġdataset Ġand Ġthe Ġlabeled Ġfeatures Ġare Ġboth Ġbalanced , Ġthere Ġis Ġlittle Ġdifference Ġbetween Ġour Ġmethods Ġand ĠGE - FL ( also Ġsee ĠFigure Ġ2 Ġ( a )). ĠBut Ġwhen Ġthe Ġclass Ġdistribution Ġbecomes Ġmore Ġun balanced , Ġthe Ġdifference Ġbecomes Ġmore Ġremarkable . ĠPerformance Ġof Ġneutral Ġfeatures Ġand Ġmaximum Ġentropy Ġdecrease Ġsignificantly Ġbut Ġincorporating ĠKL Ġdivergence Ġincreases Ġremarkably . ĠThis Ġsuggests Ġif Ġwe Ġhave Ġmore Ġaccurate Ġknowledge Ġabout Ġclass Ġdistribution , ĠKL Ġdivergence Ġcan Ġguide Ġthe Ġmodel Ġto Ġthe Ġright Ġdirection . </s> Figure Ġ3 Ġ( b ) Ġshows Ġthat Ġwhen Ġthe Ġlabeled Ġfeatures Ġare Ġun balanced , Ġour Ġmethods Ġsignificantly Ġoutper forms ĠGE - FL . ĠInc orpor ating ĠKL Ġdivergence Ġis Ġrobust Ġenough Ġto Ġcontrol Ġun balance Ġboth Ġin Ġthe Ġdataset Ġand Ġin Ġlabeled Ġfeatures Ġwhile Ġthe Ġother Ġthree Ġmethods Ġare Ġnot Ġso Ġcompetitive . </s> The ĠInfluence Ġof ĠÎ » \ lambda Ġ </s> We Ġpresent Ġthe Ġinfluence Ġof Ġ$\ lambda Ġ$ Ġon Ġthe Ġmethod Ġthat Ġincorporates ĠKL Ġdivergence Ġin Ġthis Ġsection . ĠSince Ġwe Ġsimply Ġset Ġ$\ lambda Ġ= Ġ\ beta Ġ| K | $ Ġ, Ġwe Ġjust Ġtune Ġ$\ beta Ġ$ Ġhere . ĠNote Ġthat Ġwhen Ġ$\ beta Ġ= Ġ0 $ Ġ, Ġthe Ġnewly Ġintroduced Ġregular ization Ġterm Ġis Ġdisappeared , Ġand Ġthus Ġthe Ġmodel Ġis Ġactually ĠGE - FL . ĠAgain , Ġwe Ġtest Ġthe Ġmethod Ġwith Ġdifferent Ġ$\ lambda Ġ$ Ġin Ġtwo Ġsettings : </s> ( a ) ĠWe Ġrandomly Ġselect Ġ$ t Ġ\ in Ġ[ 1 , Ġ20 ] $ Ġfeatures Ġfrom Ġthe Ġfeature Ġpool Ġfor Ġone Ġclass , Ġand Ġonly Ġone Ġfeature Ġfor Ġthe Ġother Ġclass . ĠThe Ġoriginal Ġbalanced Ġmovie Ġdataset Ġis Ġused Ġ( positive : negative = 1 : 1 ). </s> ( b ) ĠSimilar Ġto Ġ( a ), Ġbut Ġthe Ġdataset Ġis Ġun balanced , Ġobtained Ġby Ġrandomly Ġremoving Ġ75 % Ġpositive Ġdocuments Ġ( positive : negative = 1 : 4 ). </s> Results Ġare Ġshown Ġin ĠFigure Ġ4 Ġ. ĠAs Ġexpected , Ġ$\ lambda Ġ$ Ġreflects Ġhow Ġstrong Ġthe Ġregular ization Ġis . ĠThe Ġmodel Ġtends Ġto Ġbe Ġcloser Ġto Ġour Ġpreferences Ġwith Ġthe Ġincreasing Ġof Ġ$\ lambda Ġ$ Ġon Ġboth Ġcases . </s> Using ĠL DA ĠSelected ĠFeatures </s> We Ġcompare Ġour Ġmethods Ġwith ĠGE - FL Ġon Ġall Ġthe Ġ9 Ġdatasets Ġin Ġthis Ġsection . ĠInstead Ġof Ġusing Ġfeatures Ġobtained Ġby Ġinformation Ġgain , Ġwe Ġuse ĠL DA Ġto Ġselect Ġlabeled Ġfeatures . ĠUnlike Ġinformation Ġgain , ĠL DA Ġdoes Ġnot Ġemploy Ġany Ġinstance Ġlabels Ġto Ġfind Ġlabeled Ġfeatures . ĠIn Ġthis Ġsetting , Ġwe Ġcan Ġbuild Ġclassification Ġmodels Ġwithout Ġany Ġinstance Ġannotation , Ġbut Ġjust Ġwith Ġlabeled Ġfeatures . </s> Table Ġ1 Ġshows Ġthat Ġour Ġthree Ġmethods Ġsignificantly Ġoutper form ĠGE - FL . ĠInc orpor ating Ġneutral Ġfeatures Ġperforms Ġbetter Ġthan ĠGE - FL Ġon Ġ7 Ġof Ġthe Ġ9 Ġdatasets , Ġmaximum Ġentropy Ġis Ġbetter Ġon Ġ8 Ġdatasets , Ġand ĠKL Ġdivergence Ġbetter Ġon Ġ7 Ġdatasets . </s> L DA Ġselects Ġout Ġthe Ġmost Ġpredictive Ġfeatures Ġas Ġlabeled Ġfeatures Ġwithout Ġconsidering Ġthe Ġbalance Ġamong Ġclasses . ĠGE - FL Ġdoes Ġnot Ġexert Ġany Ġcontrol Ġon Ġsuch Ġan Ġissue , Ġso Ġthe Ġperformance Ġis Ġseverely Ġsuffered . ĠOur Ġmethods Ġintroduce Ġauxiliary Ġregular ization Ġterms Ġto Ġcontrol Ġsuch Ġa Ġbias Ġproblem Ġand Ġthus Ġpromote Ġthe Ġmodel Ġsignificantly . </s> Related ĠWork </s> There Ġhave Ġbeen Ġmuch Ġwork Ġthat Ġincorporate Ġprior Ġknowledge Ġinto Ġlearning , Ġand Ġtwo Ġrelated Ġlines Ġare Ġsurveyed Ġhere . ĠOne Ġis Ġto Ġuse Ġprior Ġknowledge Ġto Ġlabel Ġunl abel ed Ġinstances Ġand Ġthen Ġapply Ġa Ġstandard Ġlearning Ġalgorithm . ĠThe Ġother Ġis Ġto Ġconst rain Ġthe Ġmodel Ġdirectly Ġwith Ġprior Ġknowledge . </s> L iu Ġet Ġal . text Ġmanually Ġlabeled Ġfeatures Ġwhich Ġare Ġhighly Ġpredictive Ġto Ġun super vised Ġclust ering Ġassignments Ġand Ġuse Ġthem Ġto Ġlabel Ġunl abel ed Ġdata . ĠChang Ġet Ġal . gu iding Ġproposed Ġconstraint Ġdriven Ġlearning . ĠThey Ġfirst Ġused Ġconstraints Ġand Ġthe Ġlearned Ġmodel Ġto Ġannot ate Ġunl abel ed Ġinstances , Ġand Ġthen Ġupdated Ġthe Ġmodel Ġwith Ġthe Ġnewly Ġlabeled Ġdata . ĠDa um Ã© Ġda ume 2008 cross Ġproposed Ġa Ġself Ġtraining Ġmethod Ġin Ġwhich Ġseveral Ġmodels Ġare Ġtrained Ġon Ġthe Ġsame Ġdataset , Ġand Ġonly Ġunl abel ed Ġinstances Ġthat Ġsatisfy Ġthe Ġcross Ġtask Ġknowledge Ġconstraints Ġare Ġused Ġin Ġthe Ġself Ġtraining Ġprocess . </s> Ma Call um Ġet Ġal . ge c Ġproposed Ġgeneralized Ġexpectation ( GE ) Ġcriteria Ġwhich Ġformal ised Ġthe Ġknowledge Ġas Ġconstraint Ġterms Ġabout Ġthe Ġexpectation Ġof Ġthe Ġmodel Ġinto Ġthe Ġobjective Ġfunction . Gra Ã§a Ġet Ġal . pr Ġproposed Ġposterior Ġregular ization ( PR ) Ġframework Ġwhich Ġprojects Ġthe Ġmodel 's Ġposterior Ġonto Ġa Ġset Ġof Ġdistributions Ġthat Ġsatisfy Ġthe Ġauxiliary Ġconstraints . ĠDru ck Ġet Ġal . ge - fl Ġexplored Ġconstraints Ġof Ġlabeled Ġfeatures Ġin Ġthe Ġframework Ġof ĠGE Ġby Ġforcing Ġthe Ġmodel 's Ġpredicted Ġfeature Ġdistribution Ġto Ġapproach Ġthe Ġreference Ġdistribution . ĠAnd r ze j ewski Ġet Ġal . andr ze j ewski 2011 framework Ġproposed Ġa Ġframework Ġin Ġwhich Ġgeneral Ġdomain Ġknowledge Ġcan Ġbe Ġeasily Ġincorporated Ġinto ĠL DA . ĠAlt endor f Ġet Ġal . alt endor f 2012 learning Ġexplored Ġmon oton icity Ġconstraints Ġto Ġimprove Ġthe Ġaccuracy Ġwhile Ġlearning Ġfrom Ġsparse Ġdata . ĠChen Ġet Ġal . chen 2013 le ver aging Ġtried Ġto Ġlearn Ġcomprehens ible Ġtopic Ġmodels Ġby Ġleveraging Ġmulti - domain Ġknowledge . </s> M ann Ġand ĠMcC all um Ġsimple , general ized Ġincorporated Ġnot Ġonly Ġlabeled Ġfeatures Ġbut Ġalso Ġother Ġknowledge Ġlike Ġclass Ġdistribution Ġinto Ġthe Ġobjective Ġfunction Ġof ĠGE - FL . ĠBut Ġthey Ġdiscussed Ġonly Ġfrom Ġthe Ġsemi - super vised Ġperspective Ġand Ġdid Ġnot Ġinvestigate Ġinto Ġthe Ġrobust ness Ġproblem , Ġunlike Ġwhat Ġwe Ġaddressed Ġin Ġthis Ġpaper . </s> There Ġare Ġalso Ġsome Ġactive Ġlearning Ġmethods Ġtrying Ġto Ġuse Ġprior Ġknowledge . ĠR agh avan Ġet Ġal . feed back Ġproposed Ġto Ġuse Ġfeedback Ġon Ġinstances Ġand Ġfeatures Ġinter l aced ly , Ġand Ġdemonstrated Ġthat Ġfeedback Ġon Ġfeatures Ġboosts Ġthe Ġmodel Ġmuch . ĠDru ck Ġet Ġal . active Ġproposed Ġan Ġactive Ġlearning Ġmethod Ġwhich Ġsolic its Ġlabels Ġon Ġfeatures Ġrather Ġthan Ġon Ġinstances Ġand Ġthen Ġused ĠGE - FL Ġto Ġtrain Ġthe Ġmodel . </s> Conclusion Ġand ĠDisc ussions </s> This Ġpaper Ġinvestigates Ġinto Ġthe Ġproblem Ġof Ġhow Ġto Ġleverage Ġprior Ġknowledge Ġrobust ly Ġin Ġlearning Ġmodels . ĠWe Ġpropose Ġthree Ġregular ization Ġterms Ġon Ġtop Ġof Ġgeneralized Ġexpectation Ġcriteria . ĠAs Ġdemonstrated Ġby Ġthe Ġexperimental Ġresults , Ġthe Ġperformance Ġcan Ġbe Ġconsiderably Ġimproved Ġwhen Ġtaking Ġinto Ġaccount Ġthese Ġfactors . ĠCompar ative Ġresults Ġshow Ġthat Ġour Ġproposed Ġmethods Ġis Ġmore Ġeffective Ġand Ġworks Ġmore Ġrobust ly Ġagainst Ġbas elines . ĠTo Ġthe Ġbest Ġof Ġour Ġknowledge , Ġthis Ġis Ġthe Ġfirst Ġwork Ġto Ġaddress Ġthe Ġrobust ness Ġproblem Ġof Ġleveraging Ġknowledge , Ġand Ġmay Ġinspire Ġother Ġresearch . </s> We Ġthen Ġpresent Ġmore Ġdetailed Ġdiscussions Ġabout Ġthe Ġthree Ġregular ization Ġmethods . ĠInc orpor ating Ġneutral Ġfeatures Ġis Ġthe Ġsimplest Ġway Ġof Ġregular ization , Ġwhich Ġdoesn 't Ġrequire Ġany Ġmodification Ġof ĠGE - FL Ġbut Ġjust Ġfinding Ġout Ġsome Ġcommon Ġfeatures . ĠBut Ġas ĠFigure Ġ1 Ġ( a ) Ġshows , Ġonly Ġusing Ġneutral Ġfeatures Ġare Ġnot Ġstrong Ġenough Ġto Ġhandle Ġextremely Ġun balanced Ġlabeled Ġfeatures . </s> The Ġmaximum Ġentropy Ġregular ization Ġterm Ġshows Ġthe Ġstrong Ġability Ġof Ġcontrolling Ġun balance . </s> This Ġmethod Ġdoesn 't Ġneed Ġany Ġextra Ġknowledge , Ġand Ġis Ġthus Ġsuitable Ġwhen Ġwe Ġknow Ġnothing Ġabout Ġthe Ġcorpus . ĠBut Ġthis Ġmethod Ġassumes Ġthat Ġthe Ġcategories Ġare Ġuniformly Ġdistributed , Ġwhich Ġmay Ġnot Ġbe Ġthe Ġcase Ġin Ġpractice , Ġand Ġit Ġwill Ġhave Ġa Ġdegraded Ġperformance Ġif Ġthe Ġassumption Ġis Ġviolated Ġ( see ĠFigure Ġ1 Ġ( b ), ĠFigure Ġ2 Ġ( b ), ĠFigure Ġ3 Ġ( a )). </s> The ĠKL Ġdivergence Ġperforms Ġmuch Ġbetter Ġon Ġun balanced Ġcorpor a Ġthan Ġother Ġmethods . ĠThe Ġreason Ġis Ġthat ĠKL Ġdivergence Ġutilizes Ġthe Ġreference Ġclass Ġdistribution Ġand Ġdoesn 't Ġmake Ġany Ġassumptions . ĠThe Ġfact Ġsuggests Ġthat Ġadditional Ġknowledge Ġdoes Ġbenefit Ġthe Ġmodel . </s> However , Ġthe ĠKL Ġdivergence Ġterm Ġrequires Ġproviding Ġthe Ġtrue Ġclass Ġdistribution . ĠSometimes , Ġwe Ġmay Ġhave Ġthe Ġexact Ġknowledge Ġabout Ġthe Ġtrue Ġdistribution , Ġbut Ġsometimes Ġwe Ġmay Ġnot . ĠFortunately , Ġthe Ġmodel Ġis Ġinsensitive Ġto Ġthe Ġtrue Ġdistribution Ġand Ġtherefore Ġa Ġrough Ġestimation Ġof Ġthe Ġtrue Ġdistribution Ġis Ġsufficient . ĠIn Ġour Ġexperiments , Ġwhen Ġthe Ġtrue Ġclass Ġdistribution Ġis Ġ1 : 2 , Ġwhere Ġthe Ġreference Ġclass Ġdistribution Ġis Ġset Ġto Ġ1 : 1 . 5 / 1 : 2 / 1 : 2 . 5 , Ġthe Ġaccuracy Ġis Ġ0 . 755 / 0 . 756 / 0 . 760 Ġrespectively . ĠThis Ġprovides Ġus Ġthe Ġpossibility Ġto Ġperform Ġsimple Ġcomputing Ġon Ġthe Ġcorpus Ġto Ġobtain Ġthe Ġdistribution Ġin Ġreality . ĠOr , Ġwe Ġcan Ġset Ġthe Ġdistribution Ġroughly Ġwith Ġdomain Ġexpertise .
Introduction We posses a wealth of prior knowledge about many natural language processing tasks. For example, in text categorization, we know that words such as NBA, player, and basketball are strong indicators of the sports category BIBREF0 , and words like terrible, boring, and messing indicate a negative polarity while words like perfect, exciting, and moving suggest a positive polarity in sentiment classification. A key problem arisen here, is how to leverage such knowledge to guide the learning process, an interesting problem for both NLP and machine learning communities. Previous studies addressing the problem fall into several lines. First, to leverage prior knowledge to label data BIBREF1 , BIBREF2 . Second, to encode prior knowledge with a prior on parameters, which can be commonly seen in many Bayesian approaches BIBREF3 , BIBREF4 . Third, to formalise prior knowledge with additional variables and dependencies BIBREF5 . Last, to use prior knowledge to control the distributions over latent output variables BIBREF6 , BIBREF7 , BIBREF8 , which makes the output variables easily interpretable. However, a crucial problem, which has rarely been addressed, is the bias in the prior knowledge that we supply to the learning model. Would the model be robust or sensitive to the prior knowledge? Or, which kind of knowledge is appropriate for the task? Let's see an example: we may be a baseball fan but unfamiliar with hockey so that we can provide a few number of feature words of baseball, but much less of hockey for a baseball-hockey classification task. Such prior knowledge may mislead the model with heavy bias to baseball. If the model cannot handle this situation appropriately, the performance may be undesirable. In this paper, we investigate into the problem in the framework of Generalized Expectation Criteria BIBREF7 . The study aims to reveal the factors of reducing the sensibility of the prior knowledge and therefore to make the model more robust and practical. To this end, we introduce auxiliary regularization terms in which our prior knowledge is formalized as distribution over output variables. Recall the example just mentioned, though we do not have enough knowledge to provide features for class hockey, it is easy for us to provide some neutral words, namely words that are not strong indicators of any class, like player here. As one of the factors revealed in this paper, supplying neutral feature words can boost the performance remarkably, making the model more robust. More attractively, we do not need manual annotation to label these neutral feature words in our proposed approach. More specifically, we explore three regularization terms to address the problem: (1) a regularization term associated with neutral features; (2) the maximum entropy of class distribution regularization term; and (3) the KL divergence between reference and predicted class distribution. For the first manner, we simply use the most common features as neutral features and assume the neutral features are distributed uniformly over class labels. For the second and third one, we assume we have some knowledge about the class distribution which will be detailed soon later. To summarize, the main contributions of this work are as follows: The rest of the paper is structured as follows: In Section 2, we briefly describe the generalized expectation criteria and present the proposed regularization terms. In Section 3, we conduct extensive experiments to justify the proposed methods. We survey related work in Section 4, and summarize our work in Section 5. Method We address the robustness problem on top of GE-FL BIBREF0 , a GE method which leverages labeled features as prior knowledge. A labeled feature is a strong indicator of a specific class and is manually provided to the classifier. For example, words like amazing, exciting can be labeled features for class positive in sentiment classification. Generalized Expectation Criteria Generalized expectation (GE) criteria BIBREF7 provides us a natural way to directly constrain the model in the preferred direction. For example, when we know the proportion of each class of the dataset in a classification task, we can guide the model to predict out a pre-specified class distribution. Formally, in a parameter estimation objective function, a GE term expresses preferences on the value of some constraint functions about the model's expectation. Given a constraint function $G({\rm x}, y)$ , a conditional model distribution $p_\theta (y|\rm x)$ , an empirical distribution $\tilde{p}({\rm x})$ over input samples and a score function $S$ , a GE term can be expressed as follows: $$S(E_{\tilde{p}({\rm x})}[E_{p_\theta (y|{\rm x})}[G({\rm x}, y)]])$$   (Eq. 4) Learning from Labeled Features Druck et al. ge-fl proposed GE-FL to learn from labeled features using generalized expectation criteria. When given a set of labeled features $K$ , the reference distribution over classes of these features is denoted by $\hat{p}(y| x_k), k \in K$ . GE-FL introduces the divergence between this reference distribution and the model predicted distribution $p_\theta (y | x_k)$ , as a term of the objective function: $$\mathcal {O} = \sum _{k \in K} KL(\hat{p}(y|x_k) || p_\theta (y | x_k)) + \sum _{y,i} \frac{\theta _{yi}^2}{2 \sigma ^2}$$   (Eq. 6) where $\theta _{yi}$ is the model parameter which indicates the importance of word $i$ to class $y$ . The predicted distribution $p_\theta (y | x_k)$ can be expressed as follows: $ p_\theta (y | x_k) = \frac{1}{C_k} \sum _{\rm x} p_\theta (y|{\rm x})I(x_k) $ in which $I(x_k)$ is 1 if feature $k$ occurs in instance ${\rm x}$ and 0 otherwise, $C_k = \sum _{\rm x} I(x_k)$ is the number of instances with a non-zero value of feature $k$ , and $p_\theta (y|{\rm x})$ takes a softmax form as follows: $ p_\theta (y|{\rm x}) = \frac{1}{Z(\rm x)}\exp (\sum _i \theta _{yi}x_i). $ To solve the optimization problem, L-BFGS can be used for parameter estimation. In the framework of GE, this term can be obtained by setting the constraint function $G({\rm x}, y) = \frac{1}{C_k} \vec{I} (y)I(x_k)$ , where $\vec{I}(y)$ is an indicator vector with 1 at the index corresponding to label $y$ and 0 elsewhere. Regularization Terms GE-FL reduces the heavy load of instance annotation and performs well when we provide prior knowledge with no bias. In our experiments, we observe that comparable numbers of labeled features for each class have to be supplied. But as mentioned before, it is often the case that we are not able to provide enough knowledge for some of the classes. For the baseball-hockey classification task, as shown before, GE-FL will predict most of the instances as baseball. In this section, we will show three terms to make the model more robust. Neutral features are features that are not informative indicator of any classes, for instance, word player to the baseball-hockey classification task. Such features are usually frequent words across all categories. When we set the preference distribution of the neutral features to be uniform distributed, these neutral features will prevent the model from biasing to the class that has a dominate number of labeled features. Formally, given a set of neutral features $K^{^{\prime }}$ , the uniform distribution is $\hat{p}_u(y|x_k) = \frac{1}{|C|}, k \in K^{^{\prime }}$ , where $|C|$ is the number of classes. The objective function with the new term becomes $$\mathcal {O}_{NE} = \mathcal {O} + \sum _{k \in K^{^{\prime }}} KL(\hat{p}_u(y|x_k) || p_\theta (y | x_k)).$$   (Eq. 9) Note that we do not need manual annotation to provide neutral features. One simple way is to take the most common features as neutral features. Experimental results show that this strategy works successfully. Another way to prevent the model from drifting from the desired direction is to constrain the predicted class distribution on unlabeled data. When lacking knowledge about the class distribution of the data, one feasible way is to take maximum entropy principle, as below: $$\mathcal {O}_{ME} = \mathcal {O} + \lambda \sum _{y} p(y) \log p(y)$$   (Eq. 11) where $p(y)$ is the predicted class distribution, given by $ p(y) = \frac{1}{|X|} \sum _{\rm x} p_\theta (y | \rm x). $ To control the influence of this term on the overall objective function, we can tune $\lambda $ according to the difference in the number of labeled features of each class. In this paper, we simply set $\lambda $ to be proportional to the total number of labeled features, say $\lambda = \beta |K|$ . This maximum entropy term can be derived by setting the constraint function to $G({\rm x}, y) = \vec{I}(y)$ . Therefore, $E_{p_\theta (y|{\rm x})}[G({\rm x}, y)]$ is just the model distribution $p_\theta (y|{\rm x})$ and its expectation with the empirical distribution $\tilde{p}(\rm x)$ is simply the average over input samples, namely $p(y)$ . When $S$ takes the maximum entropy form, we can derive the objective function as above. Sometimes, we have already had much knowledge about the corpus, and can estimate the class distribution roughly without labeling instances. Therefore, we introduce the KL divergence between the predicted and reference class distributions into the objective function. Given the preference class distribution $\hat{p}(y)$ , we modify the objective function as follows: $$\mathcal {O}_{KL} &= \mathcal {O} + \lambda KL(\hat{p}(y) || p(y))$$   (Eq. 13) Similarly, we set $\lambda = \beta |K|$ . This divergence term can be derived by setting the constraint function to $G({\rm x}, y) = \vec{I}(y)$ and setting the score function to $S(\hat{p}, p) = \sum _i \hat{p}_i \log \frac{\hat{p}_i}{p_i}$ , where $p$ and $\hat{p}$ are distributions. Note that this regularization term involves the reference class distribution which will be discussed later. Experiments In this section, we first justify the approach when there exists unbalance in the number of labeled features or in class distribution. Then, to test the influence of $\lambda $ , we conduct some experiments with the method which incorporates the KL divergence of class distribution. Last, we evaluate our approaches in 9 commonly used text classification datasets. We set $\lambda = 5|K|$ by default in all experiments unless there is explicit declaration. The baseline we choose here is GE-FL BIBREF0 , a method based on generalization expectation criteria. Data Preparation We evaluate our methods on several commonly used datasets whose themes range from sentiment, web-page, science to medical and healthcare. We use bag-of-words feature and remove stopwords in the preprocess stage. Though we have labels of all documents, we do not use them during the learning process, instead, we use the label of features. The movie dataset, in which the task is to classify the movie reviews as positive or negtive, is used for testing the proposed approaches with unbalanced labeled features, unbalanced datasets or different $\lambda $ parameters. All unbalanced datasets are constructed based on the movie dataset by randomly removing documents of the positive class. For each experiment, we conduct 10-fold cross validation. As described in BIBREF0 , there are two ways to obtain labeled features. The first way is to use information gain. We first calculate the mutual information of all features according to the labels of the documents and select the top 20 as labeled features for each class as a feature pool. Note that using information gain requires the document label, but this is only to simulate how we human provide prior knowledge to the model. The second way is to use LDA BIBREF9 to select features. We use the same selection process as BIBREF0 , where they first train a LDA on the dataset, and then select the most probable features of each topic (sorted by $P(w_i|t_j)$ , the probability of word $w_i$ given topic $t_j$ ). Similar to BIBREF10 , BIBREF0 , we estimate the reference distribution of the labeled features using a heuristic strategy. If there are $|C|$ classes in total, and $n$ classes are associated with a feature $k$ , the probability that feature $k$ is related with any one of the $n$ classes is $\frac{0.9}{n}$ and with any other class is $\frac{0.1}{|C| - n}$ . Neutral features are the most frequent words after removing stop words, and their reference distributions are uniformly distributed. We use the top 10 frequent words as neutral features in all experiments. With Unbalanced Labeled Features In this section, we evaluate our approach when there is unbalanced knowledge on the categories to be classified. The labeled features are obtained through information gain. Two settings are chosen: (a) We randomly select $t \in [1, 20]$ features from the feature pool for one class, and only one feature for the other. The original balanced movie dataset is used (positive:negative=1:1). (b) Similar to (a), but the dataset is unbalanced, obtained by randomly removing 75% positive documents (positive:negative=1:4). As shown in Figure 1 , Maximum entropy principle shows improvement only on the balanced case. An obvious reason is that maximum entropy only favors uniform distribution. Incorporating Neutral features performs similarly to maximum entropy since we assume that neutral words are uniformly distributed. Its accuracy decreases slowly when the number of labeled features becomes larger ( $t>4$ ) (Figure 1 (a)), suggesting that the model gradually biases to the class with more labeled features, just like GE-FL. Incorporating the KL divergence of class distribution performs much better than GE-FL on both balanced and unbalanced datasets. This shows that it is effective to control the unbalance in labeled features and in the dataset. With Balanced Labeled Features We also compare with the baseline when the labeled features are balanced. Similar to the experiment above, the labeled features are obtained by information gain. Two settings are experimented with: (a) We randomly select $t \in [1, 20]$ features from the feature pool for each class, and conduct comparisons on the original balanced movie dataset (positive:negtive=1:1). (b) Similar to (a), but the class distribution is unbalanced, by randomly removing 75% positive documents (positive:negative=1:4). Results are shown in Figure 2 . When the dataset is balanced (Figure 2 (a)), there is little difference between GE-FL and our methods. The reason is that the proposed regularization terms provide no additional knowledge to the model and there is no bias in the labeled features. On the unbalanced dataset (Figure 2 (b)), incorporating KL divergence is much better than GE-FL since we provide additional knowledge(the true class distribution), but maximum entropy and neutral features are much worse because forcing the model to approach the uniform distribution misleads it. With Unbalanced Class Distributions Our methods are also evaluated on datasets with different unbalanced class distributions. We manually construct several movie datasets with class distributions of 1:2, 1:3, 1:4 by randomly removing 50%, 67%, 75% positive documents. The original balanced movie dataset is used as a control group. We test with both balanced and unbalanced labeled features. For the balanced case, we randomly select 10 features from the feature pool for each class, and for the unbalanced case, we select 10 features for one class, and 1 feature for the other. Results are shown in Figure 3 . Figure 3 (a) shows that when the dataset and the labeled features are both balanced, there is little difference between our methods and GE-FL(also see Figure 2 (a)). But when the class distribution becomes more unbalanced, the difference becomes more remarkable. Performance of neutral features and maximum entropy decrease significantly but incorporating KL divergence increases remarkably. This suggests if we have more accurate knowledge about class distribution, KL divergence can guide the model to the right direction. Figure 3 (b) shows that when the labeled features are unbalanced, our methods significantly outperforms GE-FL. Incorporating KL divergence is robust enough to control unbalance both in the dataset and in labeled features while the other three methods are not so competitive. The Influence of λ\lambda  We present the influence of $\lambda $ on the method that incorporates KL divergence in this section. Since we simply set $\lambda = \beta |K|$ , we just tune $\beta $ here. Note that when $\beta = 0$ , the newly introduced regularization term is disappeared, and thus the model is actually GE-FL. Again, we test the method with different $\lambda $ in two settings: (a) We randomly select $t \in [1, 20]$ features from the feature pool for one class, and only one feature for the other class. The original balanced movie dataset is used (positive:negative=1:1). (b) Similar to (a), but the dataset is unbalanced, obtained by randomly removing 75% positive documents (positive:negative=1:4). Results are shown in Figure 4 . As expected, $\lambda $ reflects how strong the regularization is. The model tends to be closer to our preferences with the increasing of $\lambda $ on both cases. Using LDA Selected Features We compare our methods with GE-FL on all the 9 datasets in this section. Instead of using features obtained by information gain, we use LDA to select labeled features. Unlike information gain, LDA does not employ any instance labels to find labeled features. In this setting, we can build classification models without any instance annotation, but just with labeled features. Table 1 shows that our three methods significantly outperform GE-FL. Incorporating neutral features performs better than GE-FL on 7 of the 9 datasets, maximum entropy is better on 8 datasets, and KL divergence better on 7 datasets. LDA selects out the most predictive features as labeled features without considering the balance among classes. GE-FL does not exert any control on such an issue, so the performance is severely suffered. Our methods introduce auxiliary regularization terms to control such a bias problem and thus promote the model significantly. Related Work There have been much work that incorporate prior knowledge into learning, and two related lines are surveyed here. One is to use prior knowledge to label unlabeled instances and then apply a standard learning algorithm. The other is to constrain the model directly with prior knowledge. Liu et al.text manually labeled features which are highly predictive to unsupervised clustering assignments and use them to label unlabeled data. Chang et al.guiding proposed constraint driven learning. They first used constraints and the learned model to annotate unlabeled instances, and then updated the model with the newly labeled data. Daumé daume2008cross proposed a self training method in which several models are trained on the same dataset, and only unlabeled instances that satisfy the cross task knowledge constraints are used in the self training process. MaCallum et al.gec proposed generalized expectation(GE) criteria which formalised the knowledge as constraint terms about the expectation of the model into the objective function.Graça et al.pr proposed posterior regularization(PR) framework which projects the model's posterior onto a set of distributions that satisfy the auxiliary constraints. Druck et al.ge-fl explored constraints of labeled features in the framework of GE by forcing the model's predicted feature distribution to approach the reference distribution. Andrzejewski et al.andrzejewski2011framework proposed a framework in which general domain knowledge can be easily incorporated into LDA. Altendorf et al.altendorf2012learning explored monotonicity constraints to improve the accuracy while learning from sparse data. Chen et al.chen2013leveraging tried to learn comprehensible topic models by leveraging multi-domain knowledge. Mann and McCallum simple,generalized incorporated not only labeled features but also other knowledge like class distribution into the objective function of GE-FL. But they discussed only from the semi-supervised perspective and did not investigate into the robustness problem, unlike what we addressed in this paper. There are also some active learning methods trying to use prior knowledge. Raghavan et al.feedback proposed to use feedback on instances and features interlacedly, and demonstrated that feedback on features boosts the model much. Druck et al.active proposed an active learning method which solicits labels on features rather than on instances and then used GE-FL to train the model. Conclusion and Discussions This paper investigates into the problem of how to leverage prior knowledge robustly in learning models. We propose three regularization terms on top of generalized expectation criteria. As demonstrated by the experimental results, the performance can be considerably improved when taking into account these factors. Comparative results show that our proposed methods is more effective and works more robustly against baselines. To the best of our knowledge, this is the first work to address the robustness problem of leveraging knowledge, and may inspire other research. We then present more detailed discussions about the three regularization methods. Incorporating neutral features is the simplest way of regularization, which doesn't require any modification of GE-FL but just finding out some common features. But as Figure 1 (a) shows, only using neutral features are not strong enough to handle extremely unbalanced labeled features. The maximum entropy regularization term shows the strong ability of controlling unbalance. This method doesn't need any extra knowledge, and is thus suitable when we know nothing about the corpus. But this method assumes that the categories are uniformly distributed, which may not be the case in practice, and it will have a degraded performance if the assumption is violated (see Figure 1 (b), Figure 2 (b), Figure 3 (a)). The KL divergence performs much better on unbalanced corpora than other methods. The reason is that KL divergence utilizes the reference class distribution and doesn't make any assumptions. The fact suggests that additional knowledge does benefit the model. However, the KL divergence term requires providing the true class distribution. Sometimes, we may have the exact knowledge about the true distribution, but sometimes we may not. Fortunately, the model is insensitive to the true distribution and therefore a rough estimation of the true distribution is sufficient. In our experiments, when the true class distribution is 1:2, where the reference class distribution is set to 1:1.5/1:2/1:2.5, the accuracy is 0.755/0.756/0.760 respectively. This provides us the possibility to perform simple computing on the corpus to obtain the distribution in reality. Or, we can set the distribution roughly with domain expertise.
What NLP tasks do they consider?
76121e359dfe3f16c2a352bd35f28005f2a40da3
What ĠN LP Ġtasks Ġdo Ġthey Ġconsider ?
text Ġclassification Ġfor Ġthemes Ġincluding Ġsentiment , Ġweb - page , Ġscience , Ġmedical Ġand Ġhealthcare
<s> What ĠN LP Ġtasks Ġdo Ġthey Ġconsider ? </s> Introduction </s> We Ġposs es Ġa Ġwealth Ġof Ġprior Ġknowledge Ġabout Ġmany Ġnatural Ġlanguage Ġprocessing Ġtasks . ĠFor Ġexample , Ġin Ġtext Ġcategor ization , Ġwe Ġknow Ġthat Ġwords Ġsuch Ġas ĠNBA , Ġplayer , Ġand Ġbasketball Ġare Ġstrong Ġindicators Ġof Ġthe Ġsports Ġcategory ĠB IB REF 0 Ġ, Ġand Ġwords Ġlike Ġterrible , Ġboring , Ġand Ġmessing Ġindicate Ġa Ġnegative Ġpol arity Ġwhile Ġwords Ġlike Ġperfect , Ġexciting , Ġand Ġmoving Ġsuggest Ġa Ġpositive Ġpol arity Ġin Ġsentiment Ġclassification . </s> A Ġkey Ġproblem Ġarisen Ġhere , Ġis Ġhow Ġto Ġleverage Ġsuch Ġknowledge Ġto Ġguide Ġthe Ġlearning Ġprocess , Ġan Ġinteresting Ġproblem Ġfor Ġboth ĠN LP Ġand Ġmachine Ġlearning Ġcommunities . ĠPrevious Ġstudies Ġaddressing Ġthe Ġproblem Ġfall Ġinto Ġseveral Ġlines . ĠFirst , Ġto Ġleverage Ġprior Ġknowledge Ġto Ġlabel Ġdata ĠB IB REF 1 Ġ, ĠB IB REF 2 Ġ. ĠSecond , Ġto Ġencode Ġprior Ġknowledge Ġwith Ġa Ġprior Ġon Ġparameters , Ġwhich Ġcan Ġbe Ġcommonly Ġseen Ġin Ġmany ĠBay esian Ġapproaches ĠB IB REF 3 Ġ, ĠB IB REF 4 Ġ. ĠThird , Ġto Ġformal ise Ġprior Ġknowledge Ġwith Ġadditional Ġvariables Ġand Ġdependencies ĠB IB REF 5 Ġ. ĠLast , Ġto Ġuse Ġprior Ġknowledge Ġto Ġcontrol Ġthe Ġdistributions Ġover Ġlatent Ġoutput Ġvariables ĠB IB REF 6 Ġ, ĠB IB REF 7 Ġ, ĠB IB REF 8 Ġ, Ġwhich Ġmakes Ġthe Ġoutput Ġvariables Ġeasily Ġinterpret able . </s> However , Ġa Ġcrucial Ġproblem , Ġwhich Ġhas Ġrarely Ġbeen Ġaddressed , Ġis Ġthe Ġbias Ġin Ġthe Ġprior Ġknowledge Ġthat Ġwe Ġsupply Ġto Ġthe Ġlearning Ġmodel . ĠWould Ġthe Ġmodel Ġbe Ġrobust Ġor Ġsensitive Ġto Ġthe Ġprior Ġknowledge ? ĠOr , Ġwhich Ġkind Ġof Ġknowledge Ġis Ġappropriate Ġfor Ġthe Ġtask ? ĠLet 's Ġsee Ġan Ġexample : Ġwe Ġmay Ġbe Ġa Ġbaseball Ġfan Ġbut Ġunfamiliar Ġwith Ġhockey Ġso Ġthat Ġwe Ġcan Ġprovide Ġa Ġfew Ġnumber Ġof Ġfeature Ġwords Ġof Ġbaseball , Ġbut Ġmuch Ġless Ġof Ġhockey Ġfor Ġa Ġbaseball - h ockey Ġclassification Ġtask . ĠSuch Ġprior Ġknowledge Ġmay Ġmislead Ġthe Ġmodel Ġwith Ġheavy Ġbias Ġto Ġbaseball . ĠIf Ġthe Ġmodel Ġcannot Ġhandle Ġthis Ġsituation Ġappropriately , Ġthe Ġperformance Ġmay Ġbe Ġundesirable . </s> In Ġthis Ġpaper , Ġwe Ġinvestigate Ġinto Ġthe Ġproblem Ġin Ġthe Ġframework Ġof ĠGeneral ized ĠExpect ation ĠCrit eria ĠB IB REF 7 Ġ. ĠThe Ġstudy Ġaims Ġto Ġreveal Ġthe Ġfactors Ġof Ġreducing Ġthe Ġsens ibility Ġof Ġthe Ġprior Ġknowledge Ġand Ġtherefore Ġto Ġmake Ġthe Ġmodel Ġmore Ġrobust Ġand Ġpractical . ĠTo Ġthis Ġend , Ġwe Ġintroduce Ġauxiliary Ġregular ization Ġterms Ġin Ġwhich Ġour Ġprior Ġknowledge Ġis Ġformal ized Ġas Ġdistribution Ġover Ġoutput Ġvariables . ĠRecall Ġthe Ġexample Ġjust Ġmentioned , Ġthough Ġwe Ġdo Ġnot Ġhave Ġenough Ġknowledge Ġto Ġprovide Ġfeatures Ġfor Ġclass Ġhockey , Ġit Ġis Ġeasy Ġfor Ġus Ġto Ġprovide Ġsome Ġneutral Ġwords , Ġnamely Ġwords Ġthat Ġare Ġnot Ġstrong Ġindicators Ġof Ġany Ġclass , Ġlike Ġplayer Ġhere . ĠAs Ġone Ġof Ġthe Ġfactors Ġrevealed Ġin Ġthis Ġpaper , Ġsupplying Ġneutral Ġfeature Ġwords Ġcan Ġboost Ġthe Ġperformance Ġremarkably , Ġmaking Ġthe Ġmodel Ġmore Ġrobust . </s> More Ġattract ively , Ġwe Ġdo Ġnot Ġneed Ġmanual Ġannotation Ġto Ġlabel Ġthese Ġneutral Ġfeature Ġwords Ġin Ġour Ġproposed Ġapproach . </s> More Ġspecifically , Ġwe Ġexplore Ġthree Ġregular ization Ġterms Ġto Ġaddress Ġthe Ġproblem : Ġ( 1 ) Ġa Ġregular ization Ġterm Ġassociated Ġwith Ġneutral Ġfeatures ; Ġ( 2 ) Ġthe Ġmaximum Ġentropy Ġof Ġclass Ġdistribution Ġregular ization Ġterm ; Ġand Ġ( 3 ) Ġthe ĠKL Ġdivergence Ġbetween Ġreference Ġand Ġpredicted Ġclass Ġdistribution . ĠFor Ġthe Ġfirst Ġmanner , Ġwe Ġsimply Ġuse Ġthe Ġmost Ġcommon Ġfeatures Ġas Ġneutral Ġfeatures Ġand Ġassume Ġthe Ġneutral Ġfeatures Ġare Ġdistributed Ġuniformly Ġover Ġclass Ġlabels . ĠFor Ġthe Ġsecond Ġand Ġthird Ġone , Ġwe Ġassume Ġwe Ġhave Ġsome Ġknowledge Ġabout Ġthe Ġclass Ġdistribution Ġwhich Ġwill Ġbe Ġdetailed Ġsoon Ġlater . </s> To Ġsummarize , Ġthe Ġmain Ġcontributions Ġof Ġthis Ġwork Ġare Ġas Ġfollows : </s> The Ġrest Ġof Ġthe Ġpaper Ġis Ġstructured Ġas Ġfollows : ĠIn ĠSection Ġ2 , Ġwe Ġbriefly Ġdescribe Ġthe Ġgeneralized Ġexpectation Ġcriteria Ġand Ġpresent Ġthe Ġproposed Ġregular ization Ġterms . ĠIn ĠSection Ġ3 , Ġwe Ġconduct Ġextensive Ġexperiments Ġto Ġjustify Ġthe Ġproposed Ġmethods . ĠWe Ġsurvey Ġrelated Ġwork Ġin ĠSection Ġ4 , Ġand Ġsummarize Ġour Ġwork Ġin ĠSection Ġ5 . </s> Method </s> We Ġaddress Ġthe Ġrobust ness Ġproblem Ġon Ġtop Ġof ĠGE - FL ĠB IB REF 0 Ġ, Ġa ĠGE Ġmethod Ġwhich Ġlever ages Ġlabeled Ġfeatures Ġas Ġprior Ġknowledge . ĠA Ġlabeled Ġfeature Ġis Ġa Ġstrong Ġindicator Ġof Ġa Ġspecific Ġclass Ġand Ġis Ġmanually Ġprovided Ġto Ġthe Ġclass ifier . ĠFor Ġexample , Ġwords Ġlike Ġamazing , Ġexciting Ġcan Ġbe Ġlabeled Ġfeatures Ġfor Ġclass Ġpositive Ġin Ġsentiment Ġclassification . </s> General ized ĠExpect ation ĠCrit eria </s> General ized Ġexpectation Ġ( GE ) Ġcriteria ĠB IB REF 7 Ġprovides Ġus Ġa Ġnatural Ġway Ġto Ġdirectly Ġconst rain Ġthe Ġmodel Ġin Ġthe Ġpreferred Ġdirection . ĠFor Ġexample , Ġwhen Ġwe Ġknow Ġthe Ġproportion Ġof Ġeach Ġclass Ġof Ġthe Ġdataset Ġin Ġa Ġclassification Ġtask , Ġwe Ġcan Ġguide Ġthe Ġmodel Ġto Ġpredict Ġout Ġa Ġpre - specified Ġclass Ġdistribution . </s> Form ally , Ġin Ġa Ġparameter Ġestimation Ġobjective Ġfunction , Ġa ĠGE Ġterm Ġexpresses Ġpreferences Ġon Ġthe Ġvalue Ġof Ġsome Ġconstraint Ġfunctions Ġabout Ġthe Ġmodel 's Ġexpectation . ĠGiven Ġa Ġconstraint Ġfunction Ġ$ G ({ \ rm Ġx }, Ġy ) $ Ġ, Ġa Ġconditional Ġmodel Ġdistribution Ġ$ p _ \ the ta Ġ( y | \ rm Ġx ) $ Ġ, Ġan Ġempirical Ġdistribution Ġ$\ t ilde { p } ({ \ rm Ġx }) $ Ġover Ġinput Ġsamples Ġand Ġa Ġscore Ġfunction Ġ$ S $ Ġ, Ġa ĠGE Ġterm Ġcan Ġbe Ġexpressed Ġas Ġfollows : </s> $$ S ( E _{ \ t ilde { p } ({ \ rm Ġx }) } [ E _{ p _ \ the ta Ġ( y | {\ rm Ġx }) } [ G ({ \ rm Ġx }, Ġy ) ]] ) $$ Ġ Ġ Ġ( E q . Ġ4 ) </s> Learning Ġfrom ĠLab eled ĠFeatures </s> D ruck Ġet Ġal . Ġge - fl Ġproposed ĠGE - FL Ġto Ġlearn Ġfrom Ġlabeled Ġfeatures Ġusing Ġgeneralized Ġexpectation Ġcriteria . ĠWhen Ġgiven Ġa Ġset Ġof Ġlabeled Ġfeatures Ġ$ K $ Ġ, Ġthe Ġreference Ġdistribution Ġover Ġclasses Ġof Ġthese Ġfeatures Ġis Ġden oted Ġby Ġ$\ hat { p } ( y | Ġx _ k ), Ġk Ġ\ in ĠK $ Ġ. ĠGE - FL Ġintroduces Ġthe Ġdivergence Ġbetween Ġthis Ġreference Ġdistribution Ġand Ġthe Ġmodel Ġpredicted Ġdistribution Ġ$ p _ \ the ta Ġ( y Ġ| Ġx _ k ) $ Ġ, Ġas Ġa Ġterm Ġof Ġthe Ġobjective Ġfunction : </s> $$ \ math cal Ġ{ O } Ġ= Ġ\ sum Ġ_ { k Ġ\ in ĠK } ĠKL (\ hat { p } ( y | x _ k ) Ġ|| Ġp _ \ the ta Ġ( y Ġ| Ġx _ k )) Ġ+ Ġ\ sum Ġ_ { y , i } Ġ\ frac {\ the ta Ġ_ { yi } ^ 2 }{ 2 Ġ\ s igma Ġ^ 2 } $$ Ġ Ġ Ġ( E q . Ġ6 ) </s> where Ġ$\ the ta Ġ_ { yi } $ Ġis Ġthe Ġmodel Ġparameter Ġwhich Ġindicates Ġthe Ġimportance Ġof Ġword Ġ$ i $ Ġto Ġclass Ġ$ y $ Ġ. ĠThe Ġpredicted Ġdistribution Ġ$ p _ \ the ta Ġ( y Ġ| Ġx _ k ) $ Ġcan Ġbe Ġexpressed Ġas Ġfollows : Ġ$ Ġp _ \ the ta Ġ( y Ġ| Ġx _ k ) Ġ= Ġ\ frac { 1 }{ C _ k } Ġ\ sum Ġ_ {\ rm Ġx } Ġp _ \ the ta Ġ( y | {\ rm Ġx }) I ( x _ k ) Ġ$ </s> in Ġwhich Ġ$ I ( x _ k ) $ Ġis Ġ1 Ġif Ġfeature Ġ$ k $ Ġoccurs Ġin Ġinstance Ġ${ \ rm Ġx } $ Ġand Ġ0 Ġotherwise , Ġ$ C _ k Ġ= Ġ\ sum Ġ_ {\ rm Ġx } ĠI ( x _ k ) $ Ġis Ġthe Ġnumber Ġof Ġinstances Ġwith Ġa Ġnon - zero Ġvalue Ġof Ġfeature Ġ$ k $ Ġ, Ġand Ġ$ p _ \ the ta Ġ( y | {\ rm Ġx }) $ Ġtakes Ġa Ġsoft max Ġform Ġas Ġfollows : Ġ$ Ġp _ \ the ta Ġ( y | {\ rm Ġx }) Ġ= Ġ\ frac { 1 }{ Z (\ rm Ġx ) }\ exp Ġ( \ sum Ġ_ i Ġ\ the ta Ġ_ { yi } x _ i ). Ġ$ </s> To Ġsolve Ġthe Ġoptimization Ġproblem , ĠL - BF GS Ġcan Ġbe Ġused Ġfor Ġparameter Ġestimation . </s> In Ġthe Ġframework Ġof ĠGE , Ġthis Ġterm Ġcan Ġbe Ġobtained Ġby Ġsetting Ġthe Ġconstraint Ġfunction Ġ$ G ({ \ rm Ġx }, Ġy ) Ġ= Ġ\ frac { 1 }{ C _ k } Ġ\ vec { I } Ġ( y ) I ( x _ k ) $ Ġ, Ġwhere Ġ$\ vec { I } ( y ) $ Ġis Ġan Ġindicator Ġvector Ġwith Ġ1 Ġat Ġthe Ġindex Ġcorresponding Ġto Ġlabel Ġ$ y $ Ġand Ġ0 Ġelsewhere . </s> Regular ization ĠTerms </s> GE - FL Ġreduces Ġthe Ġheavy Ġload Ġof Ġinstance Ġannotation Ġand Ġperforms Ġwell Ġwhen Ġwe Ġprovide Ġprior Ġknowledge Ġwith Ġno Ġbias . ĠIn Ġour Ġexperiments , Ġwe Ġobserve Ġthat Ġcomparable Ġnumbers Ġof Ġlabeled Ġfeatures Ġfor Ġeach Ġclass Ġhave Ġto Ġbe Ġsupplied . ĠBut Ġas Ġmentioned Ġbefore , Ġit Ġis Ġoften Ġthe Ġcase Ġthat Ġwe Ġare Ġnot Ġable Ġto Ġprovide Ġenough Ġknowledge Ġfor Ġsome Ġof Ġthe Ġclasses . ĠFor Ġthe Ġbaseball - h ockey Ġclassification Ġtask , Ġas Ġshown Ġbefore , ĠGE - FL Ġwill Ġpredict Ġmost Ġof Ġthe Ġinstances Ġas Ġbaseball . ĠIn Ġthis Ġsection , Ġwe Ġwill Ġshow Ġthree Ġterms Ġto Ġmake Ġthe Ġmodel Ġmore Ġrobust . </s> Ne utral Ġfeatures Ġare Ġfeatures Ġthat Ġare Ġnot Ġinformative Ġindicator Ġof Ġany Ġclasses , Ġfor Ġinstance , Ġword Ġplayer Ġto Ġthe Ġbaseball - h ockey Ġclassification Ġtask . ĠSuch Ġfeatures Ġare Ġusually Ġfrequent Ġwords Ġacross Ġall Ġcategories . ĠWhen Ġwe Ġset Ġthe Ġpreference Ġdistribution Ġof Ġthe Ġneutral Ġfeatures Ġto Ġbe Ġuniform Ġdistributed , Ġthese Ġneutral Ġfeatures Ġwill Ġprevent Ġthe Ġmodel Ġfrom Ġbi asing Ġto Ġthe Ġclass Ġthat Ġhas Ġa Ġdominate Ġnumber Ġof Ġlabeled Ġfeatures . </s> Form ally , Ġgiven Ġa Ġset Ġof Ġneutral Ġfeatures Ġ$ K ^{ ^ {\ prime Ġ}} $ Ġ, Ġthe Ġuniform Ġdistribution Ġis Ġ$\ hat { p } _ u ( y | x _ k ) Ġ= Ġ\ frac { 1 }{ | C | }, Ġk Ġ\ in ĠK ^{ ^ {\ prime Ġ}} $ Ġ, Ġwhere Ġ$ | C | $ Ġis Ġthe Ġnumber Ġof Ġclasses . ĠThe Ġobjective Ġfunction Ġwith Ġthe Ġnew Ġterm Ġbecomes </s> $$ \ math cal Ġ{ O } _{ NE } Ġ= Ġ\ math cal Ġ{ O } Ġ+ Ġ\ sum Ġ_ { k Ġ\ in ĠK ^{ ^ {\ prime Ġ} }} ĠKL (\ hat { p } _ u ( y | x _ k ) Ġ|| Ġp _ \ the ta Ġ( y Ġ| Ġx _ k )). $$ Ġ Ġ Ġ( E q . Ġ9 ) </s> Note Ġthat Ġwe Ġdo Ġnot Ġneed Ġmanual Ġannotation Ġto Ġprovide Ġneutral Ġfeatures . ĠOne Ġsimple Ġway Ġis Ġto Ġtake Ġthe Ġmost Ġcommon Ġfeatures Ġas Ġneutral Ġfeatures . ĠExperimental Ġresults Ġshow Ġthat Ġthis Ġstrategy Ġworks Ġsuccessfully . </s> Another Ġway Ġto Ġprevent Ġthe Ġmodel Ġfrom Ġdrifting Ġfrom Ġthe Ġdesired Ġdirection Ġis Ġto Ġconst rain Ġthe Ġpredicted Ġclass Ġdistribution Ġon Ġunl abel ed Ġdata . ĠWhen Ġlacking Ġknowledge Ġabout Ġthe Ġclass Ġdistribution Ġof Ġthe Ġdata , Ġone Ġfeasible Ġway Ġis Ġto Ġtake Ġmaximum Ġentropy Ġprinciple , Ġas Ġbelow : </s> $$ \ math cal Ġ{ O } _{ ME } Ġ= Ġ\ math cal Ġ{ O } Ġ+ Ġ\ lambda Ġ\ sum Ġ_ { y } Ġp ( y ) Ġ\ log Ġp ( y ) $$ Ġ Ġ Ġ( E q . Ġ11 ) </s> where Ġ$ p ( y ) $ Ġis Ġthe Ġpredicted Ġclass Ġdistribution , Ġgiven Ġby Ġ$ Ġp ( y ) Ġ= Ġ\ frac { 1 }{ | X | } Ġ\ sum Ġ_ {\ rm Ġx } Ġp _ \ the ta Ġ( y Ġ| Ġ\ rm Ġx ). Ġ$ ĠTo Ġcontrol Ġthe Ġinfluence Ġof Ġthis Ġterm Ġon Ġthe Ġoverall Ġobjective Ġfunction , Ġwe Ġcan Ġtune Ġ$\ lambda Ġ$ Ġaccording Ġto Ġthe Ġdifference Ġin Ġthe Ġnumber Ġof Ġlabeled Ġfeatures Ġof Ġeach Ġclass . ĠIn Ġthis Ġpaper , Ġwe Ġsimply Ġset Ġ$\ lambda Ġ$ Ġto Ġbe Ġproportional Ġto Ġthe Ġtotal Ġnumber Ġof Ġlabeled Ġfeatures , Ġsay Ġ$\ lambda Ġ= Ġ\ beta Ġ| K | $ Ġ. </s> This Ġmaximum Ġentropy Ġterm Ġcan Ġbe Ġderived Ġby Ġsetting Ġthe Ġconstraint Ġfunction Ġto Ġ$ G ({ \ rm Ġx }, Ġy ) Ġ= Ġ\ vec { I } ( y ) $ Ġ. ĠTherefore , Ġ$ E _{ p _ \ the ta Ġ( y | {\ rm Ġx }) } [ G ({ \ rm Ġx }, Ġy )] $ Ġis Ġjust Ġthe Ġmodel Ġdistribution Ġ$ p _ \ the ta Ġ( y | {\ rm Ġx }) $ Ġand Ġits Ġexpectation Ġwith Ġthe Ġempirical Ġdistribution Ġ$\ t ilde { p } (\ rm Ġx ) $ Ġis Ġsimply Ġthe Ġaverage Ġover Ġinput Ġsamples , Ġnamely Ġ$ p ( y ) $ Ġ. ĠWhen Ġ$ S $ Ġtakes Ġthe Ġmaximum Ġentropy Ġform , Ġwe Ġcan Ġderive Ġthe Ġobjective Ġfunction Ġas Ġabove . </s> Sometimes , Ġwe Ġhave Ġalready Ġhad Ġmuch Ġknowledge Ġabout Ġthe Ġcorpus , Ġand Ġcan Ġestimate Ġthe Ġclass Ġdistribution Ġroughly Ġwithout Ġlabeling Ġinstances . ĠTherefore , Ġwe Ġintroduce Ġthe ĠKL Ġdivergence Ġbetween Ġthe Ġpredicted Ġand Ġreference Ġclass Ġdistributions Ġinto Ġthe Ġobjective Ġfunction . </s> Given Ġthe Ġpreference Ġclass Ġdistribution Ġ$\ hat { p } ( y ) $ Ġ, Ġwe Ġmodify Ġthe Ġobjective Ġfunction Ġas Ġfollows : </s> $$ \ math cal Ġ{ O } _{ K L } Ġ& = Ġ\ math cal Ġ{ O } Ġ+ Ġ\ lambda ĠKL (\ hat { p } ( y ) Ġ|| Ġp ( y )) $$ Ġ Ġ Ġ( E q . Ġ13 ) </s> Similarly , Ġwe Ġset Ġ$\ lambda Ġ= Ġ\ beta Ġ| K | $ Ġ. </s> This Ġdivergence Ġterm Ġcan Ġbe Ġderived Ġby Ġsetting Ġthe Ġconstraint Ġfunction Ġto Ġ$ G ({ \ rm Ġx }, Ġy ) Ġ= Ġ\ vec { I } ( y ) $ Ġand Ġsetting Ġthe Ġscore Ġfunction Ġto Ġ$ S (\ hat { p }, Ġp ) Ġ= Ġ\ sum Ġ_ i Ġ\ hat { p } _ i Ġ\ log Ġ\ frac {\ hat { p } _ i }{ p _ i } $ Ġ, Ġwhere Ġ$ p $ Ġand Ġ$\ hat { p } $ Ġare Ġdistributions . ĠNote Ġthat Ġthis Ġregular ization Ġterm Ġinvolves Ġthe Ġreference Ġclass Ġdistribution Ġwhich Ġwill Ġbe Ġdiscussed Ġlater . </s> Exper iments </s> In Ġthis Ġsection , Ġwe Ġfirst Ġjustify Ġthe Ġapproach Ġwhen Ġthere Ġexists Ġun balance Ġin Ġthe Ġnumber Ġof Ġlabeled Ġfeatures Ġor Ġin Ġclass Ġdistribution . ĠThen , Ġto Ġtest Ġthe Ġinfluence Ġof Ġ$\ lambda Ġ$ Ġ, Ġwe Ġconduct Ġsome Ġexperiments Ġwith Ġthe Ġmethod Ġwhich Ġincorporates Ġthe ĠKL Ġdivergence Ġof Ġclass Ġdistribution . ĠLast , Ġwe Ġevaluate Ġour Ġapproaches Ġin Ġ9 Ġcommonly Ġused Ġtext Ġclassification Ġdatasets . ĠWe Ġset Ġ$\ lambda Ġ= Ġ5 | K | $ Ġby Ġdefault Ġin Ġall Ġexperiments Ġunless Ġthere Ġis Ġexplicit Ġdeclaration . ĠThe Ġbaseline Ġwe Ġchoose Ġhere Ġis ĠGE - FL ĠB IB REF 0 Ġ, Ġa Ġmethod Ġbased Ġon Ġgeneral ization Ġexpectation Ġcriteria . </s> Data ĠPrepar ation </s> We Ġevaluate Ġour Ġmethods Ġon Ġseveral Ġcommonly Ġused Ġdatasets Ġwhose Ġthemes Ġrange Ġfrom Ġsentiment , Ġweb - page , Ġscience Ġto Ġmedical Ġand Ġhealthcare . ĠWe Ġuse Ġbag - of - words Ġfeature Ġand Ġremove Ġstop words Ġin Ġthe Ġpre process Ġstage . ĠThough Ġwe Ġhave Ġlabels Ġof Ġall Ġdocuments , Ġwe Ġdo Ġnot Ġuse Ġthem Ġduring Ġthe Ġlearning Ġprocess , Ġinstead , Ġwe Ġuse Ġthe Ġlabel Ġof Ġfeatures . </s> The Ġmovie Ġdataset , Ġin Ġwhich Ġthe Ġtask Ġis Ġto Ġclassify Ġthe Ġmovie Ġreviews Ġas Ġpositive Ġor Ġneg t ive , Ġis Ġused Ġfor Ġtesting Ġthe Ġproposed Ġapproaches Ġwith Ġun balanced Ġlabeled Ġfeatures , Ġun balanced Ġdatasets Ġor Ġdifferent Ġ$\ lambda Ġ$ Ġparameters . ĠAll Ġun balanced Ġdatasets Ġare Ġconstructed Ġbased Ġon Ġthe Ġmovie Ġdataset Ġby Ġrandomly Ġremoving Ġdocuments Ġof Ġthe Ġpositive Ġclass . ĠFor Ġeach Ġexperiment , Ġwe Ġconduct Ġ10 - fold Ġcross Ġvalidation . </s> As Ġdescribed Ġin ĠB IB REF 0 Ġ, Ġthere Ġare Ġtwo Ġways Ġto Ġobtain Ġlabeled Ġfeatures . ĠThe Ġfirst Ġway Ġis Ġto Ġuse Ġinformation Ġgain . ĠWe Ġfirst Ġcalculate Ġthe Ġmutual Ġinformation Ġof Ġall Ġfeatures Ġaccording Ġto Ġthe Ġlabels Ġof Ġthe Ġdocuments Ġand Ġselect Ġthe Ġtop Ġ20 Ġas Ġlabeled Ġfeatures Ġfor Ġeach Ġclass Ġas Ġa Ġfeature Ġpool . ĠNote Ġthat Ġusing Ġinformation Ġgain Ġrequires Ġthe Ġdocument Ġlabel , Ġbut Ġthis Ġis Ġonly Ġto Ġsimulate Ġhow Ġwe Ġhuman Ġprovide Ġprior Ġknowledge Ġto Ġthe Ġmodel . ĠThe Ġsecond Ġway Ġis Ġto Ġuse ĠL DA ĠB IB REF 9 Ġto Ġselect Ġfeatures . ĠWe Ġuse Ġthe Ġsame Ġselection Ġprocess Ġas ĠB IB REF 0 Ġ, Ġwhere Ġthey Ġfirst Ġtrain Ġa ĠL DA Ġon Ġthe Ġdataset , Ġand Ġthen Ġselect Ġthe Ġmost Ġprobable Ġfeatures Ġof Ġeach Ġtopic Ġ( s orted Ġby Ġ$ P ( w _ i | t _ j ) $ Ġ, Ġthe Ġprobability Ġof Ġword Ġ$ w _ i $ Ġgiven Ġtopic Ġ$ t _ j $ Ġ). </s> Similar Ġto ĠB IB REF 10 Ġ, ĠB IB REF 0 Ġ, Ġwe Ġestimate Ġthe Ġreference Ġdistribution Ġof Ġthe Ġlabeled Ġfeatures Ġusing Ġa Ġhe uristic Ġstrategy . ĠIf Ġthere Ġare Ġ$ | C | $ Ġclasses Ġin Ġtotal , Ġand Ġ$ n $ Ġclasses Ġare Ġassociated Ġwith Ġa Ġfeature Ġ$ k $ Ġ, Ġthe Ġprobability Ġthat Ġfeature Ġ$ k $ Ġis Ġrelated Ġwith Ġany Ġone Ġof Ġthe Ġ$ n $ Ġclasses Ġis Ġ$\ frac { 0 . 9 }{ n } $ Ġand Ġwith Ġany Ġother Ġclass Ġis Ġ$\ frac { 0 . 1 }{ | C | Ġ- Ġn } $ Ġ. </s> Ne utral Ġfeatures Ġare Ġthe Ġmost Ġfrequent Ġwords Ġafter Ġremoving Ġstop Ġwords , Ġand Ġtheir Ġreference Ġdistributions Ġare Ġuniformly Ġdistributed . ĠWe Ġuse Ġthe Ġtop Ġ10 Ġfrequent Ġwords Ġas Ġneutral Ġfeatures Ġin Ġall Ġexperiments . </s> With ĠUn balanced ĠLab eled ĠFeatures </s> In Ġthis Ġsection , Ġwe Ġevaluate Ġour Ġapproach Ġwhen Ġthere Ġis Ġun balanced Ġknowledge Ġon Ġthe Ġcategories Ġto Ġbe Ġclassified . ĠThe Ġlabeled Ġfeatures Ġare Ġobtained Ġthrough Ġinformation Ġgain . ĠTwo Ġsettings Ġare Ġchosen : </s> ( a ) ĠWe Ġrandomly Ġselect Ġ$ t Ġ\ in Ġ[ 1 , Ġ20 ] $ Ġfeatures Ġfrom Ġthe Ġfeature Ġpool Ġfor Ġone Ġclass , Ġand Ġonly Ġone Ġfeature Ġfor Ġthe Ġother . ĠThe Ġoriginal Ġbalanced Ġmovie Ġdataset Ġis Ġused Ġ( positive : negative = 1 : 1 ). </s> ( b ) ĠSimilar Ġto Ġ( a ), Ġbut Ġthe Ġdataset Ġis Ġun balanced , Ġobtained Ġby Ġrandomly Ġremoving Ġ75 % Ġpositive Ġdocuments Ġ( positive : negative = 1 : 4 ). </s> As Ġshown Ġin ĠFigure Ġ1 Ġ, ĠMaximum Ġentropy Ġprinciple Ġshows Ġimprovement Ġonly Ġon Ġthe Ġbalanced Ġcase . ĠAn Ġobvious Ġreason Ġis Ġthat Ġmaximum Ġentropy Ġonly Ġfavors Ġuniform Ġdistribution . </s> In cor por ating ĠNeutral Ġfeatures Ġperforms Ġsimilarly Ġto Ġmaximum Ġentropy Ġsince Ġwe Ġassume Ġthat Ġneutral Ġwords Ġare Ġuniformly Ġdistributed . ĠIts Ġaccuracy Ġdecreases Ġslowly Ġwhen Ġthe Ġnumber Ġof Ġlabeled Ġfeatures Ġbecomes Ġlarger Ġ( Ġ$ t > 4 $ Ġ) Ġ( Figure Ġ1 Ġ( a )), Ġsuggesting Ġthat Ġthe Ġmodel Ġgradually Ġbiases Ġto Ġthe Ġclass Ġwith Ġmore Ġlabeled Ġfeatures , Ġjust Ġlike ĠGE - FL . </s> In cor por ating Ġthe ĠKL Ġdivergence Ġof Ġclass Ġdistribution Ġperforms Ġmuch Ġbetter Ġthan ĠGE - FL Ġon Ġboth Ġbalanced Ġand Ġun balanced Ġdatasets . ĠThis Ġshows Ġthat Ġit Ġis Ġeffective Ġto Ġcontrol Ġthe Ġun balance Ġin Ġlabeled Ġfeatures Ġand Ġin Ġthe Ġdataset . </s> With ĠBalanced ĠLab eled ĠFeatures </s> We Ġalso Ġcompare Ġwith Ġthe Ġbaseline Ġwhen Ġthe Ġlabeled Ġfeatures Ġare Ġbalanced . ĠSimilar Ġto Ġthe Ġexperiment Ġabove , Ġthe Ġlabeled Ġfeatures Ġare Ġobtained Ġby Ġinformation Ġgain . ĠTwo Ġsettings Ġare Ġexperimented Ġwith : </s> ( a ) ĠWe Ġrandomly Ġselect Ġ$ t Ġ\ in Ġ[ 1 , Ġ20 ] $ Ġfeatures Ġfrom Ġthe Ġfeature Ġpool Ġfor Ġeach Ġclass , Ġand Ġconduct Ġcomparisons Ġon Ġthe Ġoriginal Ġbalanced Ġmovie Ġdataset Ġ( positive : neg t ive = 1 : 1 ). </s> ( b ) ĠSimilar Ġto Ġ( a ), Ġbut Ġthe Ġclass Ġdistribution Ġis Ġun balanced , Ġby Ġrandomly Ġremoving Ġ75 % Ġpositive Ġdocuments Ġ( positive : negative = 1 : 4 ). </s> Results Ġare Ġshown Ġin ĠFigure Ġ2 Ġ. ĠWhen Ġthe Ġdataset Ġis Ġbalanced Ġ( Figure Ġ2 Ġ( a )), Ġthere Ġis Ġlittle Ġdifference Ġbetween ĠGE - FL Ġand Ġour Ġmethods . ĠThe Ġreason Ġis Ġthat Ġthe Ġproposed Ġregular ization Ġterms Ġprovide Ġno Ġadditional Ġknowledge Ġto Ġthe Ġmodel Ġand Ġthere Ġis Ġno Ġbias Ġin Ġthe Ġlabeled Ġfeatures . ĠOn Ġthe Ġun balanced Ġdataset Ġ( Figure Ġ2 Ġ( b )), Ġincorporating ĠKL Ġdivergence Ġis Ġmuch Ġbetter Ġthan ĠGE - FL Ġsince Ġwe Ġprovide Ġadditional Ġknowledge ( the Ġtrue Ġclass Ġdistribution ), Ġbut Ġmaximum Ġentropy Ġand Ġneutral Ġfeatures Ġare Ġmuch Ġworse Ġbecause Ġforcing Ġthe Ġmodel Ġto Ġapproach Ġthe Ġuniform Ġdistribution Ġmis le ads Ġit . </s> With ĠUn balanced ĠClass ĠDistribut ions </s> Our Ġmethods Ġare Ġalso Ġevaluated Ġon Ġdatasets Ġwith Ġdifferent Ġun balanced Ġclass Ġdistributions . ĠWe Ġmanually Ġconstruct Ġseveral Ġmovie Ġdatasets Ġwith Ġclass Ġdistributions Ġof Ġ1 : 2 , Ġ1 : 3 , Ġ1 : 4 Ġby Ġrandomly Ġremoving Ġ50 %, Ġ67 %, Ġ75 % Ġpositive Ġdocuments . ĠThe Ġoriginal Ġbalanced Ġmovie Ġdataset Ġis Ġused Ġas Ġa Ġcontrol Ġgroup . ĠWe Ġtest Ġwith Ġboth Ġbalanced Ġand Ġun balanced Ġlabeled Ġfeatures . ĠFor Ġthe Ġbalanced Ġcase , Ġwe Ġrandomly Ġselect Ġ10 Ġfeatures Ġfrom Ġthe Ġfeature Ġpool Ġfor Ġeach Ġclass , Ġand Ġfor Ġthe Ġun balanced Ġcase , Ġwe Ġselect Ġ10 Ġfeatures Ġfor Ġone Ġclass , Ġand Ġ1 Ġfeature Ġfor Ġthe Ġother . ĠResults Ġare Ġshown Ġin ĠFigure Ġ3 Ġ. </s> Figure Ġ3 Ġ( a ) Ġshows Ġthat Ġwhen Ġthe Ġdataset Ġand Ġthe Ġlabeled Ġfeatures Ġare Ġboth Ġbalanced , Ġthere Ġis Ġlittle Ġdifference Ġbetween Ġour Ġmethods Ġand ĠGE - FL ( also Ġsee ĠFigure Ġ2 Ġ( a )). ĠBut Ġwhen Ġthe Ġclass Ġdistribution Ġbecomes Ġmore Ġun balanced , Ġthe Ġdifference Ġbecomes Ġmore Ġremarkable . ĠPerformance Ġof Ġneutral Ġfeatures Ġand Ġmaximum Ġentropy Ġdecrease Ġsignificantly Ġbut Ġincorporating ĠKL Ġdivergence Ġincreases Ġremarkably . ĠThis Ġsuggests Ġif Ġwe Ġhave Ġmore Ġaccurate Ġknowledge Ġabout Ġclass Ġdistribution , ĠKL Ġdivergence Ġcan Ġguide Ġthe Ġmodel Ġto Ġthe Ġright Ġdirection . </s> Figure Ġ3 Ġ( b ) Ġshows Ġthat Ġwhen Ġthe Ġlabeled Ġfeatures Ġare Ġun balanced , Ġour Ġmethods Ġsignificantly Ġoutper forms ĠGE - FL . ĠInc orpor ating ĠKL Ġdivergence Ġis Ġrobust Ġenough Ġto Ġcontrol Ġun balance Ġboth Ġin Ġthe Ġdataset Ġand Ġin Ġlabeled Ġfeatures Ġwhile Ġthe Ġother Ġthree Ġmethods Ġare Ġnot Ġso Ġcompetitive . </s> The ĠInfluence Ġof ĠÎ » \ lambda Ġ </s> We Ġpresent Ġthe Ġinfluence Ġof Ġ$\ lambda Ġ$ Ġon Ġthe Ġmethod Ġthat Ġincorporates ĠKL Ġdivergence Ġin Ġthis Ġsection . ĠSince Ġwe Ġsimply Ġset Ġ$\ lambda Ġ= Ġ\ beta Ġ| K | $ Ġ, Ġwe Ġjust Ġtune Ġ$\ beta Ġ$ Ġhere . ĠNote Ġthat Ġwhen Ġ$\ beta Ġ= Ġ0 $ Ġ, Ġthe Ġnewly Ġintroduced Ġregular ization Ġterm Ġis Ġdisappeared , Ġand Ġthus Ġthe Ġmodel Ġis Ġactually ĠGE - FL . ĠAgain , Ġwe Ġtest Ġthe Ġmethod Ġwith Ġdifferent Ġ$\ lambda Ġ$ Ġin Ġtwo Ġsettings : </s> ( a ) ĠWe Ġrandomly Ġselect Ġ$ t Ġ\ in Ġ[ 1 , Ġ20 ] $ Ġfeatures Ġfrom Ġthe Ġfeature Ġpool Ġfor Ġone Ġclass , Ġand Ġonly Ġone Ġfeature Ġfor Ġthe Ġother Ġclass . ĠThe Ġoriginal Ġbalanced Ġmovie Ġdataset Ġis Ġused Ġ( positive : negative = 1 : 1 ). </s> ( b ) ĠSimilar Ġto Ġ( a ), Ġbut Ġthe Ġdataset Ġis Ġun balanced , Ġobtained Ġby Ġrandomly Ġremoving Ġ75 % Ġpositive Ġdocuments Ġ( positive : negative = 1 : 4 ). </s> Results Ġare Ġshown Ġin ĠFigure Ġ4 Ġ. ĠAs Ġexpected , Ġ$\ lambda Ġ$ Ġreflects Ġhow Ġstrong Ġthe Ġregular ization Ġis . ĠThe Ġmodel Ġtends Ġto Ġbe Ġcloser Ġto Ġour Ġpreferences Ġwith Ġthe Ġincreasing Ġof Ġ$\ lambda Ġ$ Ġon Ġboth Ġcases . </s> Using ĠL DA ĠSelected ĠFeatures </s> We Ġcompare Ġour Ġmethods Ġwith ĠGE - FL Ġon Ġall Ġthe Ġ9 Ġdatasets Ġin Ġthis Ġsection . ĠInstead Ġof Ġusing Ġfeatures Ġobtained Ġby Ġinformation Ġgain , Ġwe Ġuse ĠL DA Ġto Ġselect Ġlabeled Ġfeatures . ĠUnlike Ġinformation Ġgain , ĠL DA Ġdoes Ġnot Ġemploy Ġany Ġinstance Ġlabels Ġto Ġfind Ġlabeled Ġfeatures . ĠIn Ġthis Ġsetting , Ġwe Ġcan Ġbuild Ġclassification Ġmodels Ġwithout Ġany Ġinstance Ġannotation , Ġbut Ġjust Ġwith Ġlabeled Ġfeatures . </s> Table Ġ1 Ġshows Ġthat Ġour Ġthree Ġmethods Ġsignificantly Ġoutper form ĠGE - FL . ĠInc orpor ating Ġneutral Ġfeatures Ġperforms Ġbetter Ġthan ĠGE - FL Ġon Ġ7 Ġof Ġthe Ġ9 Ġdatasets , Ġmaximum Ġentropy Ġis Ġbetter Ġon Ġ8 Ġdatasets , Ġand ĠKL Ġdivergence Ġbetter Ġon Ġ7 Ġdatasets . </s> L DA Ġselects Ġout Ġthe Ġmost Ġpredictive Ġfeatures Ġas Ġlabeled Ġfeatures Ġwithout Ġconsidering Ġthe Ġbalance Ġamong Ġclasses . ĠGE - FL Ġdoes Ġnot Ġexert Ġany Ġcontrol Ġon Ġsuch Ġan Ġissue , Ġso Ġthe Ġperformance Ġis Ġseverely Ġsuffered . ĠOur Ġmethods Ġintroduce Ġauxiliary Ġregular ization Ġterms Ġto Ġcontrol Ġsuch Ġa Ġbias Ġproblem Ġand Ġthus Ġpromote Ġthe Ġmodel Ġsignificantly . </s> Related ĠWork </s> There Ġhave Ġbeen Ġmuch Ġwork Ġthat Ġincorporate Ġprior Ġknowledge Ġinto Ġlearning , Ġand Ġtwo Ġrelated Ġlines Ġare Ġsurveyed Ġhere . ĠOne Ġis Ġto Ġuse Ġprior Ġknowledge Ġto Ġlabel Ġunl abel ed Ġinstances Ġand Ġthen Ġapply Ġa Ġstandard Ġlearning Ġalgorithm . ĠThe Ġother Ġis Ġto Ġconst rain Ġthe Ġmodel Ġdirectly Ġwith Ġprior Ġknowledge . </s> L iu Ġet Ġal . text Ġmanually Ġlabeled Ġfeatures Ġwhich Ġare Ġhighly Ġpredictive Ġto Ġun super vised Ġclust ering Ġassignments Ġand Ġuse Ġthem Ġto Ġlabel Ġunl abel ed Ġdata . ĠChang Ġet Ġal . gu iding Ġproposed Ġconstraint Ġdriven Ġlearning . ĠThey Ġfirst Ġused Ġconstraints Ġand Ġthe Ġlearned Ġmodel Ġto Ġannot ate Ġunl abel ed Ġinstances , Ġand Ġthen Ġupdated Ġthe Ġmodel Ġwith Ġthe Ġnewly Ġlabeled Ġdata . ĠDa um Ã© Ġda ume 2008 cross Ġproposed Ġa Ġself Ġtraining Ġmethod Ġin Ġwhich Ġseveral Ġmodels Ġare Ġtrained Ġon Ġthe Ġsame Ġdataset , Ġand Ġonly Ġunl abel ed Ġinstances Ġthat Ġsatisfy Ġthe Ġcross Ġtask Ġknowledge Ġconstraints Ġare Ġused Ġin Ġthe Ġself Ġtraining Ġprocess . </s> Ma Call um Ġet Ġal . ge c Ġproposed Ġgeneralized Ġexpectation ( GE ) Ġcriteria Ġwhich Ġformal ised Ġthe Ġknowledge Ġas Ġconstraint Ġterms Ġabout Ġthe Ġexpectation Ġof Ġthe Ġmodel Ġinto Ġthe Ġobjective Ġfunction . Gra Ã§a Ġet Ġal . pr Ġproposed Ġposterior Ġregular ization ( PR ) Ġframework Ġwhich Ġprojects Ġthe Ġmodel 's Ġposterior Ġonto Ġa Ġset Ġof Ġdistributions Ġthat Ġsatisfy Ġthe Ġauxiliary Ġconstraints . ĠDru ck Ġet Ġal . ge - fl Ġexplored Ġconstraints Ġof Ġlabeled Ġfeatures Ġin Ġthe Ġframework Ġof ĠGE Ġby Ġforcing Ġthe Ġmodel 's Ġpredicted Ġfeature Ġdistribution Ġto Ġapproach Ġthe Ġreference Ġdistribution . ĠAnd r ze j ewski Ġet Ġal . andr ze j ewski 2011 framework Ġproposed Ġa Ġframework Ġin Ġwhich Ġgeneral Ġdomain Ġknowledge Ġcan Ġbe Ġeasily Ġincorporated Ġinto ĠL DA . ĠAlt endor f Ġet Ġal . alt endor f 2012 learning Ġexplored Ġmon oton icity Ġconstraints Ġto Ġimprove Ġthe Ġaccuracy Ġwhile Ġlearning Ġfrom Ġsparse Ġdata . ĠChen Ġet Ġal . chen 2013 le ver aging Ġtried Ġto Ġlearn Ġcomprehens ible Ġtopic Ġmodels Ġby Ġleveraging Ġmulti - domain Ġknowledge . </s> M ann Ġand ĠMcC all um Ġsimple , general ized Ġincorporated Ġnot Ġonly Ġlabeled Ġfeatures Ġbut Ġalso Ġother Ġknowledge Ġlike Ġclass Ġdistribution Ġinto Ġthe Ġobjective Ġfunction Ġof ĠGE - FL . ĠBut Ġthey Ġdiscussed Ġonly Ġfrom Ġthe Ġsemi - super vised Ġperspective Ġand Ġdid Ġnot Ġinvestigate Ġinto Ġthe Ġrobust ness Ġproblem , Ġunlike Ġwhat Ġwe Ġaddressed Ġin Ġthis Ġpaper . </s> There Ġare Ġalso Ġsome Ġactive Ġlearning Ġmethods Ġtrying Ġto Ġuse Ġprior Ġknowledge . ĠR agh avan Ġet Ġal . feed back Ġproposed Ġto Ġuse Ġfeedback Ġon Ġinstances Ġand Ġfeatures Ġinter l aced ly , Ġand Ġdemonstrated Ġthat Ġfeedback Ġon Ġfeatures Ġboosts Ġthe Ġmodel Ġmuch . ĠDru ck Ġet Ġal . active Ġproposed Ġan Ġactive Ġlearning Ġmethod Ġwhich Ġsolic its Ġlabels Ġon Ġfeatures Ġrather Ġthan Ġon Ġinstances Ġand Ġthen Ġused ĠGE - FL Ġto Ġtrain Ġthe Ġmodel . </s> Conclusion Ġand ĠDisc ussions </s> This Ġpaper Ġinvestigates Ġinto Ġthe Ġproblem Ġof Ġhow Ġto Ġleverage Ġprior Ġknowledge Ġrobust ly Ġin Ġlearning Ġmodels . ĠWe Ġpropose Ġthree Ġregular ization Ġterms Ġon Ġtop Ġof Ġgeneralized Ġexpectation Ġcriteria . ĠAs Ġdemonstrated Ġby Ġthe Ġexperimental Ġresults , Ġthe Ġperformance Ġcan Ġbe Ġconsiderably Ġimproved Ġwhen Ġtaking Ġinto Ġaccount Ġthese Ġfactors . ĠCompar ative Ġresults Ġshow Ġthat Ġour Ġproposed Ġmethods Ġis Ġmore Ġeffective Ġand Ġworks Ġmore Ġrobust ly Ġagainst Ġbas elines . ĠTo Ġthe Ġbest Ġof Ġour Ġknowledge , Ġthis Ġis Ġthe Ġfirst Ġwork Ġto Ġaddress Ġthe Ġrobust ness Ġproblem Ġof Ġleveraging Ġknowledge , Ġand Ġmay Ġinspire Ġother Ġresearch . </s> We Ġthen Ġpresent Ġmore Ġdetailed Ġdiscussions Ġabout Ġthe Ġthree Ġregular ization Ġmethods . ĠInc orpor ating Ġneutral Ġfeatures Ġis Ġthe Ġsimplest Ġway Ġof Ġregular ization , Ġwhich Ġdoesn 't Ġrequire Ġany Ġmodification Ġof ĠGE - FL Ġbut Ġjust Ġfinding Ġout Ġsome Ġcommon Ġfeatures . ĠBut Ġas ĠFigure Ġ1 Ġ( a ) Ġshows , Ġonly Ġusing Ġneutral Ġfeatures Ġare Ġnot Ġstrong Ġenough Ġto Ġhandle Ġextremely Ġun balanced Ġlabeled Ġfeatures . </s> The Ġmaximum Ġentropy Ġregular ization Ġterm Ġshows Ġthe Ġstrong Ġability Ġof Ġcontrolling Ġun balance . </s> This Ġmethod Ġdoesn 't Ġneed Ġany Ġextra Ġknowledge , Ġand Ġis Ġthus Ġsuitable Ġwhen Ġwe Ġknow Ġnothing Ġabout Ġthe Ġcorpus . ĠBut Ġthis Ġmethod Ġassumes Ġthat Ġthe Ġcategories Ġare Ġuniformly Ġdistributed , Ġwhich Ġmay Ġnot Ġbe Ġthe Ġcase Ġin Ġpractice , Ġand Ġit Ġwill Ġhave Ġa Ġdegraded Ġperformance Ġif Ġthe Ġassumption Ġis Ġviolated Ġ( see ĠFigure Ġ1 Ġ( b ), ĠFigure Ġ2 Ġ( b ), ĠFigure Ġ3 Ġ( a )). </s> The ĠKL Ġdivergence Ġperforms Ġmuch Ġbetter Ġon Ġun balanced Ġcorpor a Ġthan Ġother Ġmethods . ĠThe Ġreason Ġis Ġthat ĠKL Ġdivergence Ġutilizes Ġthe Ġreference Ġclass Ġdistribution Ġand Ġdoesn 't Ġmake Ġany Ġassumptions . ĠThe Ġfact Ġsuggests Ġthat Ġadditional Ġknowledge Ġdoes Ġbenefit Ġthe Ġmodel . </s> However , Ġthe ĠKL Ġdivergence Ġterm Ġrequires Ġproviding Ġthe Ġtrue Ġclass Ġdistribution . ĠSometimes , Ġwe Ġmay Ġhave Ġthe Ġexact Ġknowledge Ġabout Ġthe Ġtrue Ġdistribution , Ġbut Ġsometimes Ġwe Ġmay Ġnot . ĠFortunately , Ġthe Ġmodel Ġis Ġinsensitive Ġto Ġthe Ġtrue Ġdistribution Ġand Ġtherefore Ġa Ġrough Ġestimation Ġof Ġthe Ġtrue Ġdistribution Ġis Ġsufficient . ĠIn Ġour Ġexperiments , Ġwhen Ġthe Ġtrue Ġclass Ġdistribution Ġis Ġ1 : 2 , Ġwhere Ġthe Ġreference Ġclass Ġdistribution Ġis Ġset Ġto Ġ1 : 1 . 5 / 1 : 2 / 1 : 2 . 5 , Ġthe Ġaccuracy Ġis Ġ0 . 755 / 0 . 756 / 0 . 760 Ġrespectively . ĠThis Ġprovides Ġus Ġthe Ġpossibility Ġto Ġperform Ġsimple Ġcomputing Ġon Ġthe Ġcorpus Ġto Ġobtain Ġthe Ġdistribution Ġin Ġreality . ĠOr , Ġwe Ġcan Ġset Ġthe Ġdistribution Ġroughly Ġwith Ġdomain Ġexpertise .