three hand-crafted polarity scores, which are minimum, mean, and maximum polarity scores
EXTRACTIVE
polarity scores, which are minimum, mean, and maximum polarity scores, from each review
EXTRACTIVE
After creating several embeddings as mentioned above, we create document (review or tweet) vectors. For each document, we sum all the vectors of words occurring in that document and take their average. In addition to it, we extract three hand-crafted polarity scores, which are minimum, mean, and maximum polarity scores, from each review. These polarity scores of words are computed as in (DISPLAY_FORM4). For example, if a review consists of five words, it would have five polarity scores and we utilise only three of these sentiment scores as mentioned. Lastly, we concatenate these three scores to the averaged word vector per review. That is, each review is represented by the average word vector of its constituent word embeddings and three supervised scores. We then feed these inputs into the SVM approach. The flowchart of our framework is given in Figure FIGREF11. When combining the unsupervised features, which are word vectors created on a word basis, with supervised three scores extracted on a review basis, we have better state-of-the-art results.
After creating several embeddings as mentioned above, we create document (review or tweet) vectors. For each document, we sum all the vectors of words occurring in that document and take their average. In addition to it, we extract three hand-crafted polarity scores, which are minimum, mean, and maximum polarity scores, from each review. These polarity scores of words are computed as in (DISPLAY_FORM4). For example, if a review consists of five words, it would have five polarity scores and we utilise only three of these sentiment scores as mentioned. Lastly, we concatenate these three scores to the averaged word vector per review.
2001.01269
Introduction </s> Sent iment Ġanalysis Ġhas Ġrecently Ġbeen Ġone Ġof Ġthe Ġhottest Ġtopics Ġin Ġnatural Ġlanguage Ġprocessing Ġ( N LP ). ĠIt Ġis Ġused Ġto Ġidentify Ġand Ġcategor ise Ġopinions Ġexpressed Ġby Ġreviewers Ġon Ġa Ġtopic Ġor Ġan Ġentity . ĠSent iment Ġanalysis Ġcan Ġbe Ġlever aged Ġin Ġmarketing , Ġsocial Ġmedia Ġanalysis , Ġand Ġcustomer Ġservice . ĠAlthough Ġmany Ġstudies Ġhave Ġbeen Ġconducted Ġfor Ġsentiment Ġanalysis Ġin Ġwidely Ġspoken Ġlanguages , Ġthis Ġtopic Ġis Ġstill Ġimmature Ġfor ĠTurkish Ġand Ġmany Ġother Ġlanguages . </s> Ne ural Ġnetworks Ġoutper form Ġthe Ġconventional Ġmachine Ġlearning Ġalgorithms Ġin Ġmost Ġclassification Ġtasks , Ġincluding Ġsentiment Ġanalysis ĠB IB REF 0 . ĠIn Ġthese Ġnetworks , Ġword Ġembed ding Ġvectors Ġare Ġfed Ġas Ġinput Ġto Ġovercome Ġthe Ġdata Ġsp arsity Ġproblem Ġand Ġmake Ġthe Ġrepresentations Ġof Ġwords Ġmore ĠâĢ ľ meaning ful âĢ Ŀ Ġand Ġrobust . ĠThose Ġembed d ings Ġindicate Ġhow Ġclose Ġthe Ġwords Ġare Ġto Ġeach Ġother Ġin Ġthe Ġvector Ġspace Ġmodel Ġ( V SM ). </s> Most Ġof Ġthe Ġstudies Ġutil ise Ġembed d ings , Ġsuch Ġas Ġword 2 vec ĠB IB REF 1 , Ġwhich Ġtake Ġinto Ġaccount Ġthe Ġsynt actic Ġand Ġsemantic Ġrepresentations Ġof Ġthe Ġwords Ġonly . ĠDisc arding Ġthe Ġsentimental Ġaspects Ġof Ġwords Ġmay Ġlead Ġto Ġwords Ġof Ġdifferent Ġpolar ities Ġbeing Ġclose Ġto Ġeach Ġother Ġin Ġthe ĠV SM , Ġif Ġthey Ġshare Ġsimilar Ġsemantic Ġand Ġsynt actic Ġfeatures . </s> For ĠTurkish , Ġthere Ġare Ġonly Ġa Ġfew Ġstudies Ġwhich Ġleverage Ġsentimental Ġinformation Ġin Ġgenerating Ġthe Ġword Ġand Ġdocument Ġembed d ings . ĠUnlike Ġthe Ġstudies Ġconducted Ġfor ĠEnglish Ġand Ġother Ġwidely - spoken Ġlanguages , Ġin Ġthis Ġpaper , Ġwe Ġuse Ġthe Ġofficial Ġdiction aries Ġfor Ġthis Ġlanguage Ġand Ġcombine Ġthe Ġun super vised Ġand Ġsupervised Ġscores Ġto Ġgenerate Ġa Ġunified Ġscore Ġfor Ġeach Ġdimension Ġof Ġthe Ġword Ġembed d ings Ġin Ġthis Ġtask . </s> Our Ġmain Ġcontribution Ġis Ġto Ġcreate Ġoriginal Ġand Ġeffective Ġword Ġvectors Ġthat Ġcapture Ġsynt actic , Ġsemantic , Ġand Ġsentimental Ġcharacteristics Ġof Ġwords , Ġand Ġuse Ġall Ġof Ġthis Ġknowledge Ġin Ġgenerating Ġembed d ings . ĠWe Ġalso Ġutil ise Ġthe Ġword 2 vec Ġembed d ings Ġtrained Ġon Ġa Ġlarge Ġcorpus . ĠBesides Ġusing Ġthese Ġword Ġembed d ings , Ġwe Ġalso Ġgenerate Ġhand - crafted Ġfeatures Ġon Ġa Ġreview Ġbasis Ġand Ġcreate Ġdocument Ġvectors . ĠWe Ġevaluate Ġthose Ġembed d ings Ġon Ġtwo Ġdatasets . ĠThe Ġresults Ġshow Ġthat Ġwe Ġoutper form Ġthe Ġapproaches Ġwhich Ġdo Ġnot Ġtake Ġinto Ġaccount Ġthe Ġsentimental Ġinformation . ĠWe Ġalso Ġhad Ġbetter Ġperformances Ġthan Ġother Ġstudies Ġcarried Ġout Ġon Ġsentiment Ġanalysis Ġin ĠTurkish Ġmedia . ĠWe Ġalso Ġevaluated Ġour Ġnovel Ġembed ding Ġapproaches Ġon Ġtwo ĠEnglish Ġcorpor a Ġof Ġdifferent Ġgenres . ĠWe Ġoutper formed Ġthe Ġbaseline Ġapproaches Ġfor Ġthis Ġlanguage Ġas Ġwell . ĠThe Ġsource Ġcode Ġand Ġdatasets Ġare Ġpublicly Ġavailable . </s> The Ġpaper Ġis Ġorganised Ġas Ġfollows . ĠIn ĠSection Ġ2 , Ġwe Ġpresent Ġthe Ġexisting Ġworks Ġon Ġsentiment Ġclassification . ĠIn ĠSection Ġ3 , Ġwe Ġdescribe Ġthe Ġmethods Ġproposed Ġin Ġthis Ġwork . ĠThe Ġexperimental Ġresults Ġare Ġshown Ġand Ġthe Ġmain Ġcontributions Ġof Ġour Ġproposed Ġapproach Ġare Ġdiscussed Ġin ĠSection Ġ4 . ĠIn ĠSection Ġ5 , Ġwe Ġconclude Ġthe Ġpaper . </s> Related ĠWork </s> In Ġthe Ġliterature , Ġthe Ġmain Ġconsensus Ġis Ġthat Ġthe Ġuse Ġof Ġdense Ġword Ġembed d ings Ġoutper form Ġthe Ġsparse Ġembed d ings Ġin Ġmany Ġtasks . ĠLat ent Ġsemantic Ġanalysis Ġ( L SA ) Ġused Ġto Ġbe Ġthe Ġmost Ġpopular Ġmethod Ġin Ġgenerating Ġword Ġembed d ings Ġbefore Ġthe Ġinvention Ġof Ġthe Ġword 2 vec Ġand Ġother Ġword Ġvector Ġalgorithms Ġwhich Ġare Ġmostly Ġcreated Ġby Ġshallow Ġneural Ġnetwork Ġmodels . ĠAlthough Ġmany Ġstudies Ġhave Ġbeen Ġemployed Ġon Ġgenerating Ġword Ġvectors Ġincluding Ġboth Ġsemantic Ġand Ġsentimental Ġcomponents , Ġgenerating Ġand Ġanalys ing Ġthe Ġeffects Ġof Ġdifferent Ġtypes Ġof Ġembed d ings Ġon Ġdifferent Ġtasks Ġis Ġan Ġemerging Ġfield Ġfor ĠTurkish . </s> Lat ent ĠDir ich let Ġallocation Ġ( L DA ) Ġis Ġused Ġin ĠB IB REF 2 Ġto Ġextract Ġmixture Ġof Ġlatent Ġtopics . ĠHowever , Ġit Ġfoc uss es Ġon Ġfinding Ġthe Ġlatent Ġtopics Ġof Ġa Ġdocument , Ġnot Ġthe Ġword Ġmeanings Ġthemselves . ĠIn ĠB IB REF 3 , ĠL SA Ġis Ġutil ised Ġto Ġgenerate Ġword Ġvectors , Ġleveraging Ġindirect Ġco - occ urrence Ġstatistics . ĠThese Ġoutper form Ġthe Ġuse Ġof Ġsparse Ġvectors ĠB IB REF 4 . </s> Some Ġof Ġthe Ġprior Ġstudies Ġhave Ġalso Ġtaken Ġinto Ġaccount Ġthe Ġsentimental Ġcharacteristics Ġof Ġa Ġword Ġwhen Ġcreating Ġword Ġvectors ĠB IB REF 5 , ĠB IB REF 6 , ĠB IB REF 7 . ĠA Ġmodel Ġwith Ġsemantic Ġand Ġsentiment Ġcomponents Ġis Ġbuilt Ġin ĠB IB REF 8 , Ġmaking Ġuse Ġof Ġstar - rat ings Ġof Ġreviews . ĠIn ĠB IB REF 9 , Ġa Ġsentiment Ġlex icon Ġis Ġinduced Ġpreferring Ġthe Ġuse Ġof Ġdomain - specific Ġco - occ urrence Ġstatistics Ġover Ġthe Ġword 2 vec Ġmethod Ġand Ġthey Ġoutper form Ġthe Ġlatter . </s> In Ġa Ġrecent Ġwork Ġon Ġsentiment Ġanalysis Ġin ĠTurkish ĠB IB REF 10 , Ġthey Ġlearn Ġembed d ings Ġusing ĠTurkish Ġsocial Ġmedia . ĠThey Ġuse Ġthe Ġword 2 vec Ġalgorithm , Ġcreate Ġseveral Ġun super vised Ġhand - crafted Ġfeatures , Ġgenerate Ġdocument Ġvectors Ġand Ġfeed Ġthem Ġas Ġinput Ġinto Ġthe Ġsupport Ġvector Ġmachines Ġ( S VM ) Ġapproach . ĠWe Ġoutper form Ġthis Ġbaseline Ġapproach Ġusing Ġmore Ġeffective Ġword Ġembed d ings Ġand Ġsupervised Ġhand - crafted Ġfeatures . </s> In ĠEnglish , Ġmuch Ġof Ġthe Ġrecent Ġwork Ġon Ġlearning Ġsentiment - specific Ġembed d ings Ġrelies Ġonly Ġon Ġdistant Ġsupervision . ĠIn ĠB IB REF 11 , Ġem oj is Ġare Ġused Ġas Ġfeatures Ġand Ġa Ġbi - L ST M Ġ( bid irection al Ġlong Ġshort - term Ġmemory ) Ġneural Ġnetwork Ġmodel Ġis Ġbuilt Ġto Ġlearn Ġsentiment - aware Ġword Ġembed d ings . ĠIn ĠB IB REF 12 , Ġa Ġneural Ġnetwork Ġthat Ġlearns Ġword Ġembed d ings Ġis Ġbuilt Ġby Ġusing Ġcontextual Ġinformation Ġabout Ġthe Ġdata Ġand Ġsupervised Ġscores Ġof Ġthe Ġwords . ĠThis Ġwork Ġcaptures Ġthe Ġsupervised Ġinformation Ġby Ġutil ising Ġemot icons Ġas Ġfeatures . ĠMost Ġof Ġour Ġapproaches Ġdo Ġnot Ġrely Ġon Ġa Ġneural Ġnetwork Ġmodel Ġin Ġlearning Ġembed d ings . ĠHowever , Ġthey Ġproduce Ġstate - of - the - art Ġresults . </s> Method ology </s> We Ġgenerate Ġseveral Ġword Ġvectors , Ġwhich Ġcapture Ġthe Ġsentimental , Ġlex ical , Ġand Ġcontextual Ġcharacteristics Ġof Ġwords . ĠIn Ġaddition Ġto Ġthese Ġmostly Ġoriginal Ġvectors , Ġwe Ġalso Ġcreate Ġword 2 vec Ġembed d ings Ġto Ġrepresent Ġthe Ġcorpus Ġwords Ġby Ġtraining Ġthe Ġembed ding Ġmodel Ġon Ġthese Ġdatasets . ĠAfter Ġgenerating Ġthese , Ġwe Ġcombine Ġthem Ġwith Ġhand - crafted Ġfeatures Ġto Ġcreate Ġdocument Ġvectors Ġand Ġperform Ġclassification , Ġas Ġwill Ġbe Ġexplained Ġin ĠSection Ġ3 . 5 . </s> Method ology Ġ: :: ĠCorpus - based ĠApproach </s> Context ual Ġinformation Ġis Ġinformative Ġin Ġthe Ġsense Ġthat , Ġin Ġgeneral , Ġsimilar Ġwords Ġtend Ġto Ġappear Ġin Ġthe Ġsame Ġcontexts . ĠFor Ġexample , Ġthe Ġword Ġsmart Ġis Ġmore Ġlikely Ġto Ġco occ ur Ġwith Ġthe Ġword Ġhard working Ġthan Ġwith Ġthe Ġword Ġlazy . ĠThis Ġsimilarity Ġcan Ġbe Ġdefined Ġsem antically Ġand Ġsentiment ally . ĠIn Ġthe Ġcorpus - based Ġapproach , Ġwe Ġcapture Ġboth Ġof Ġthese Ġcharacteristics Ġand Ġgenerate Ġword Ġembed d ings Ġspecific Ġto Ġa Ġdomain . </s> Firstly , Ġwe Ġconstruct Ġa Ġmatrix Ġwhose Ġentries Ġcorrespond Ġto Ġthe Ġnumber Ġof Ġco occ ur rences Ġof Ġthe Ġrow Ġand Ġcolumn Ġwords Ġin Ġsliding Ġwindows . ĠDi agonal Ġentries Ġare Ġassigned Ġthe Ġnumber Ġof Ġsliding Ġwindows Ġthat Ġthe Ġcorresponding Ġrow Ġword Ġappears Ġin Ġthe Ġwhole Ġcorpus . ĠWe Ġthen Ġnormal ise Ġeach Ġrow Ġby Ġdividing Ġentries Ġin Ġthe Ġrow Ġby Ġthe Ġmaximum Ġscore Ġin Ġit . </s> Secondly , Ġwe Ġperform Ġthe Ġprincipal Ġcomponent Ġanalysis Ġ( PC A ) Ġmethod Ġto Ġreduce Ġthe Ġdimension ality . ĠIt Ġcaptures Ġlatent Ġmeanings Ġand Ġtakes Ġinto Ġaccount Ġhigh - order Ġco occ urrence Ġremoving Ġnoise . ĠThe Ġattribute Ġ( column ) Ġnumber Ġof Ġthe Ġmatrix Ġis Ġreduced Ġto Ġ200 . ĠWe Ġthen Ġcompute Ġcos ine Ġsimilarity Ġbetween Ġeach Ġrow Ġpair Ġ$ w _ i $ Ġand Ġ$ w _ j $ Ġas Ġin Ġ( DIS PLAY _ FORM 3 ) Ġto Ġfind Ġout Ġhow Ġsimilar Ġtwo Ġword Ġvectors Ġ( rows ) Ġare . </s> Third ly , Ġall Ġthe Ġvalues Ġin Ġthe Ġmatrix Ġare Ġsubt racted Ġfrom Ġ1 Ġto Ġcreate Ġa Ġdiss imilar ity Ġmatrix . ĠThen , Ġwe Ġfeed Ġthe Ġmatrix Ġas Ġinput Ġinto Ġthe Ġfuzzy Ġc - me ans Ġclust ering Ġalgorithm . ĠWe Ġchose Ġthe Ġnumber Ġof Ġclusters Ġas Ġ200 , Ġas Ġit Ġis Ġconsidered Ġa Ġstandard Ġfor Ġword Ġembed d ings Ġin Ġthe Ġliterature . ĠAfter Ġclust ering , Ġthe Ġdimension Ġi Ġfor Ġa Ġcorresponding Ġword Ġindicates Ġthe Ġdegree Ġto Ġwhich Ġthis Ġword Ġbelongs Ġto Ġcluster Ġi . ĠThe Ġintuition Ġbehind Ġthis Ġidea Ġis Ġthat Ġif Ġtwo Ġwords Ġare Ġsimilar Ġin Ġthe ĠV SM , Ġthey Ġare Ġmore Ġlikely Ġto Ġbelong Ġto Ġthe Ġsame Ġclusters Ġwith Ġakin Ġprobabilities . ĠIn Ġthe Ġend , Ġeach Ġword Ġin Ġthe Ġcorpus Ġis Ġrepresented Ġby Ġa Ġ200 - dimensional Ġvector . </s> In Ġaddition Ġto Ġthis Ġmethod , Ġwe Ġalso Ġperform Ġsingular Ġvalue Ġdecom position Ġ( S VD ) Ġon Ġthe Ġco occ urrence Ġmat rices , Ġwhere Ġwe Ġcompute Ġthe Ġmatrix Ġ$ M ^{ P PM I } Ġ= ĠU \ S igma ĠV ^{ T } $. ĠPositive Ġpoint wise Ġmutual Ġinformation Ġ( P PM I ) Ġscores Ġbetween Ġwords Ġare Ġcalculated Ġand Ġthe Ġtrunc ated Ġsingular Ġvalue Ġdecom position Ġis Ġcomputed . ĠWe Ġtake Ġinto Ġaccount Ġthe ĠU Ġmatrix Ġonly Ġfor Ġeach Ġword . ĠWe Ġhave Ġchosen Ġthe Ġsingular Ġvalue Ġnumber Ġas Ġ200 . ĠThat Ġis , Ġeach Ġword Ġin Ġthe Ġcorpus Ġis Ġrepresented Ġby Ġa Ġ200 - dimensional Ġvector Ġas Ġfollows . </s> Method ology Ġ: :: ĠDictionary - based ĠApproach </s> In ĠTurkish , Ġthere Ġdo Ġnot Ġexist Ġwell - established Ġsentiment Ġlex icons Ġas Ġin ĠEnglish . ĠIn Ġthis Ġapproach , Ġwe Ġmade Ġuse Ġof Ġthe ĠTD K Ġ( T Ã¼r k ĠDil ĠKur um u Ġ- ĠâĢ ľ Turkish ĠLanguage ĠInstitution âĢ Ŀ ) Ġdictionary Ġto Ġobtain Ġword Ġpolar ities . ĠAlthough Ġit Ġis Ġnot Ġa Ġsentiment Ġlex icon , Ġcombining Ġit Ġwith Ġdomain - specific Ġpol arity Ġscores Ġobtained Ġfrom Ġthe Ġcorpus Ġled Ġus Ġto Ġhave Ġstate - of - the - art Ġresults . </s> We Ġfirst Ġconstruct Ġa Ġmatrix Ġwhose Ġrow Ġentries Ġare Ġcorpus Ġwords Ġand Ġcolumn Ġentries Ġare Ġthe Ġwords Ġin Ġtheir Ġdictionary Ġdefinitions . ĠWe Ġfollowed Ġthe Ġboolean Ġapproach . ĠFor Ġinstance , Ġfor Ġthe Ġword Ġcat , Ġthe Ġcolumn Ġwords Ġoccurring Ġin Ġits Ġdictionary Ġdefinition Ġare Ġgiven Ġa Ġscore Ġof Ġ1 . ĠThose Ġcolumn Ġwords Ġnot Ġappearing Ġin Ġthe Ġdefinition Ġof Ġcat Ġare Ġassigned Ġa Ġscore Ġof Ġ0 Ġfor Ġthat Ġcorresponding Ġrow Ġentry . </s> When Ġwe Ġperformed Ġclust ering Ġon Ġthis Ġmatrix , Ġwe Ġobserved Ġthat Ġthose Ġwords Ġhaving Ġsimilar Ġmeanings Ġare , Ġin Ġgeneral , Ġassigned Ġto Ġthe Ġsame Ġclusters . ĠHowever , Ġthis Ġsimilarity Ġfails Ġin Ġcapturing Ġthe Ġsentimental Ġcharacteristics . ĠFor Ġinstance , Ġthe Ġwords Ġhappy Ġand Ġunhappy Ġare Ġassigned Ġto Ġthe Ġsame Ġcluster , Ġsince Ġthey Ġhave Ġthe Ġsame Ġwords , Ġsuch Ġas Ġfeeling , Ġin Ġtheir Ġdictionary Ġdefinitions . ĠHowever , Ġthey Ġare Ġof Ġopposite Ġpolar ities Ġand Ġshould Ġbe Ġdiscern ed Ġfrom Ġeach Ġother . </s> Therefore , Ġwe Ġutil ise Ġa Ġmetric Ġto Ġmove Ġsuch Ġwords Ġaway Ġfrom Ġeach Ġother Ġin Ġthe ĠV SM , Ġeven Ġthough Ġthey Ġhave Ġcommon Ġwords Ġin Ġtheir Ġdictionary Ġdefinitions . ĠWe Ġmultiply Ġeach Ġvalue Ġin Ġa Ġrow Ġwith Ġthe Ġcorresponding Ġrow Ġword 's Ġraw Ġsupervised Ġscore , Ġthereby Ġhaving Ġmore Ġmeaningful Ġclusters . ĠUsing Ġthe Ġtraining Ġdata Ġonly , Ġthe Ġsupervised Ġpol arity Ġscore Ġper Ġword Ġis Ġcalculated Ġas Ġin Ġ( DIS PLAY _ FORM 4 ). </s> Here , Ġ$ Ġw _{ t } $ Ġdenotes Ġthe Ġsentiment Ġscore Ġof Ġword Ġ$ t $, Ġ$ N _{ t } $ Ġis Ġthe Ġnumber Ġof Ġdocuments Ġ( review s Ġor Ġtweets ) Ġin Ġwhich Ġ$ t $ Ġoccurs Ġin Ġthe Ġdataset Ġof Ġpositive Ġpol arity , Ġ$ N $ Ġis Ġthe Ġnumber Ġof Ġall Ġthe Ġwords Ġin Ġthe Ġcorpus Ġof Ġpositive Ġpol arity . Ġ$ N ^ {\ prime Ġ} $ Ġdenotes Ġthe Ġcorpus Ġof Ġnegative Ġpol arity . Ġ$ N ^ {\ prime Ġ} _{ t } $ Ġand Ġ$ N ^ {\ prime Ġ} $ Ġdenote Ġsimilar Ġvalues Ġfor Ġthe Ġnegative Ġpol arity Ġcorpus . ĠWe Ġperform Ġnormal isation Ġto Ġprevent Ġthe Ġimbalance Ġproblem Ġand Ġadd Ġa Ġsmall Ġnumber Ġto Ġboth Ġnumer ator Ġand Ġdenomin ator Ġfor Ġsmoot hing . </s> As Ġan Ġalternative Ġto Ġmultiplying Ġwith Ġthe Ġsupervised Ġpol arity Ġscores , Ġwe Ġalso Ġseparately Ġmultiplied Ġall Ġthe Ġrow Ġscores Ġwith Ġonly Ġ+ 1 Ġif Ġthe Ġrow Ġword Ġis Ġa Ġpositive Ġword , Ġand Ġwith Ġ- 1 Ġif Ġit Ġis Ġa Ġnegative Ġword . ĠWe Ġhave Ġobserved Ġit Ġboosts Ġthe Ġperformance Ġmore Ġcompared Ġto Ġusing Ġraw Ġscores . </s> The Ġeffect Ġof Ġthis Ġmultiplication Ġis Ġexempl ified Ġin ĠFigure ĠFIG REF 7 , Ġshowing Ġthe Ġpositions Ġof Ġword Ġvectors Ġin Ġthe ĠV SM . ĠThose ĠâĢ ľ x " Ġwords Ġare Ġsentiment ally Ġnegative Ġwords , Ġthose ĠâĢ ľ o " Ġwords Ġare Ġsentiment ally Ġpositive Ġones . ĠOn Ġthe Ġtop Ġcoordinate Ġplane , Ġthe Ġwords Ġof Ġopposite Ġpolar ities Ġare Ġfound Ġto Ġbe Ġclose Ġto Ġeach Ġother , Ġsince Ġthey Ġhave Ġcommon Ġwords Ġin Ġtheir Ġdictionary Ġdefinitions . ĠOnly Ġthe Ġinformation Ġconcerned Ġwith Ġthe Ġdictionary Ġdefinitions Ġare Ġused Ġthere , Ġdisc arding Ġthe Ġpol arity Ġscores . ĠHowever , Ġwhen Ġwe Ġutil ise Ġthe Ġsupervised Ġscore Ġ(+ 1 Ġor Ġ- 1 ), Ġwords Ġof Ġopposite Ġpolar ities Ġ( e . g . ĠâĢ ľ happy " Ġand ĠâĢ ľ un happy ") Ġget Ġfar Ġaway Ġfrom Ġeach Ġother Ġas Ġthey Ġare Ġtranslated Ġacross Ġcoordinate Ġregions . ĠPositive Ġwords Ġnow Ġappear Ġin Ġquad rant Ġ1 , Ġwhereas Ġnegative Ġwords Ġappear Ġin Ġquad rant Ġ3 . ĠThus , Ġin Ġthe ĠV SM , Ġwords Ġthat Ġare Ġsentiment ally Ġsimilar Ġto Ġeach Ġother Ġcould Ġbe Ġclustered Ġmore Ġaccurately . ĠBesides Ġclust ering , Ġwe Ġalso Ġemployed Ġthe ĠS VD Ġmethod Ġto Ġperform Ġdimension ality Ġreduction Ġon Ġthe Ġun super vised Ġdictionary Ġalgorithm Ġand Ġused Ġthe Ġnewly Ġgenerated Ġmatrix Ġby Ġcombining Ġit Ġwith Ġother Ġsub appro aches . ĠThe Ġnumber Ġof Ġdimensions Ġis Ġchosen Ġas Ġ200 Ġagain Ġaccording Ġto Ġthe Ġ$ U $ Ġmatrix . ĠThe Ġdetails Ġare Ġgiven Ġin ĠSection Ġ3 . 4 . ĠWhen Ġusing Ġand Ġevaluating Ġthis Ġsub appro ach Ġon Ġthe ĠEnglish Ġcorpor a , Ġwe Ġused Ġthe ĠSent i Word Net Ġlex icon ĠB IB REF 13 . ĠWe Ġhave Ġachieved Ġbetter Ġresults Ġfor Ġthe Ġdictionary - based Ġalgorithm Ġwhen Ġwe Ġemployed Ġthe ĠS VD Ġreduction Ġmethod Ġcompared Ġto Ġthe Ġuse Ġof Ġclust ering . </s> Method ology Ġ: :: ĠSuper vised ĠContext ual Ġ4 - sc ores </s> Our Ġlast Ġcomponent Ġis Ġa Ġsimple Ġmetric Ġthat Ġuses Ġfour Ġsupervised Ġscores Ġfor Ġeach Ġword Ġin Ġthe Ġcorpus . ĠWe Ġextract Ġthese Ġscores Ġas Ġfollows . ĠFor Ġa Ġtarget Ġword Ġin Ġthe Ġcorpus , Ġwe Ġscan Ġthrough Ġall Ġof Ġits Ġcontexts . ĠIn Ġaddition Ġto Ġthe Ġtarget Ġword 's Ġpol arity Ġscore Ġ( the Ġself Ġscore ), Ġout Ġof Ġall Ġthe Ġpol arity Ġscores Ġof Ġwords Ġoccurring Ġin Ġthe Ġsame Ġcontexts Ġas Ġthe Ġtarget Ġword , Ġminimum , Ġmaximum , Ġand Ġaverage Ġscores Ġare Ġtaken Ġinto Ġconsideration . ĠThe Ġword Ġpol arity Ġscores Ġare Ġcomputed Ġusing Ġ( DIS PLAY _ FORM 4 ). ĠHere , Ġwe Ġobtain Ġthose Ġscores Ġfrom Ġthe Ġtraining Ġdata . </s> The Ġintuition Ġbehind Ġthis Ġmethod Ġis Ġthat Ġthose Ġfour Ġscores Ġare Ġmore Ġindicative Ġof Ġa Ġword 's Ġpol arity Ġrather Ġthan Ġonly Ġone Ġ( the Ġself Ġscore ). ĠThis Ġapproach Ġis Ġfully Ġsupervised Ġunlike Ġthe Ġprevious Ġtwo Ġapproaches . </s> Method ology Ġ: :: ĠComb ination Ġof Ġthe ĠWord ĠEmb edd ings </s> In Ġaddition Ġto Ġusing Ġthe Ġthree Ġapproaches Ġindependently , Ġwe Ġalso Ġcombined Ġall Ġthe Ġmat rices Ġgenerated Ġin Ġthe Ġprevious Ġapproaches . ĠThat Ġis , Ġwe Ġconc aten ate Ġthe Ġreduced Ġforms Ġ( S VD Ġ- ĠU ) Ġof Ġcorpus - based , Ġdictionary - based , Ġand Ġthe Ġwhole Ġof Ġ4 - score Ġvectors Ġof Ġeach Ġword , Ġhorizontally . ĠAccordingly , Ġeach Ġcorpus Ġword Ġis Ġrepresented Ġby Ġa Ġ404 - dimensional Ġvector , Ġsince Ġcorpus - based Ġand Ġdictionary - based Ġvector Ġcomponents Ġare Ġeach Ġcomposed Ġof Ġ200 Ġdimensions , Ġwhereas Ġthe Ġ4 - score Ġvector Ġcomponent Ġis Ġformed Ġby Ġfour Ġvalues . </s> The Ġmain Ġintuition Ġbehind Ġthe Ġensemble Ġmethod Ġis Ġthat Ġsome Ġapproaches Ġcompensate Ġfor Ġwhat Ġthe Ġothers Ġmay Ġlack . ĠFor Ġexample , Ġthe Ġcorpus - based Ġapproach Ġcaptures Ġthe Ġdomain - specific , Ġsemantic , Ġand Ġsynt actic Ġcharacteristics . ĠOn Ġthe Ġother Ġhand , Ġthe Ġ4 - sc ores Ġmethod Ġcaptures Ġsupervised Ġfeatures , Ġand Ġthe Ġdictionary - based Ġapproach Ġis Ġhelpful Ġin Ġcapturing Ġthe Ġgeneral Ġsemantic Ġcharacteristics . ĠThat Ġis , Ġcombining Ġthose Ġthree Ġapproaches Ġmakes Ġword Ġvectors Ġmore Ġrepresentative . </s> Method ology Ġ: :: ĠGener ating ĠDocument ĠV ect ors </s> After Ġcreating Ġseveral Ġembed d ings Ġas Ġmentioned Ġabove , Ġwe Ġcreate Ġdocument Ġ( review Ġor Ġtweet ) Ġvectors . ĠFor Ġeach Ġdocument , Ġwe Ġsum Ġall Ġthe Ġvectors Ġof Ġwords Ġoccurring Ġin Ġthat Ġdocument Ġand Ġtake Ġtheir Ġaverage . ĠIn Ġaddition Ġto Ġit , Ġwe Ġextract Ġthree Ġhand - crafted Ġpol arity Ġscores , Ġwhich Ġare Ġminimum , Ġmean , Ġand Ġmaximum Ġpol arity Ġscores , Ġfrom Ġeach Ġreview . ĠThese Ġpol arity Ġscores Ġof Ġwords Ġare Ġcomputed Ġas Ġin Ġ( DIS PLAY _ FORM 4 ). ĠFor Ġexample , Ġif Ġa Ġreview Ġconsists Ġof Ġfive Ġwords , Ġit Ġwould Ġhave Ġfive Ġpol arity Ġscores Ġand Ġwe Ġutil ise Ġonly Ġthree Ġof Ġthese Ġsentiment Ġscores Ġas Ġmentioned . ĠLastly , Ġwe Ġconc aten ate Ġthese Ġthree Ġscores Ġto Ġthe Ġaveraged Ġword Ġvector Ġper Ġreview . </s> That Ġis , Ġeach Ġreview Ġis Ġrepresented Ġby Ġthe Ġaverage Ġword Ġvector Ġof Ġits Ġconstituent Ġword Ġembed d ings Ġand Ġthree Ġsupervised Ġscores . ĠWe Ġthen Ġfeed Ġthese Ġinputs Ġinto Ġthe ĠS VM Ġapproach . ĠThe Ġflow chart Ġof Ġour Ġframework Ġis Ġgiven Ġin ĠFigure ĠFIG REF 11 . ĠWhen Ġcombining Ġthe Ġun super vised Ġfeatures , Ġwhich Ġare Ġword Ġvectors Ġcreated Ġon Ġa Ġword Ġbasis , Ġwith Ġsupervised Ġthree Ġscores Ġextracted Ġon Ġa Ġreview Ġbasis , Ġwe Ġhave Ġbetter Ġstate - of - the - art Ġresults . </s> Dat as ets </s> We Ġutil ised Ġtwo Ġdatasets Ġfor Ġboth ĠTurkish Ġand ĠEnglish Ġto Ġevaluate Ġour Ġmethods . </s> For ĠTurkish , Ġas Ġthe Ġfirst Ġdataset , Ġwe Ġutil ised Ġthe Ġmovie Ġreviews Ġwhich Ġare Ġcollected Ġfrom Ġa Ġpopular Ġwebsite . ĠThe Ġnumber Ġof Ġreviews Ġin Ġthis Ġmovie Ġcorpus Ġis Ġ20 , 244 Ġand Ġthe Ġaverage Ġnumber Ġof Ġwords Ġin Ġreviews Ġis Ġ39 . ĠEach Ġof Ġthese Ġreviews Ġhas Ġa Ġstar - rating Ġscore Ġwhich Ġis Ġindicative Ġof Ġsentiment . ĠThese Ġpol arity Ġscores Ġare Ġbetween Ġthe Ġvalues Ġ0 . 5 Ġand Ġ5 , Ġat Ġintervals Ġof Ġ0 . 5 . ĠWe Ġconsider Ġa Ġreview Ġto Ġbe Ġnegative Ġit Ġthe Ġscore Ġis Ġequal Ġto Ġor Ġlower Ġthan Ġ2 . 5 . ĠOn Ġthe Ġother Ġhand , Ġif Ġit Ġis Ġequal Ġto Ġor Ġhigher Ġthan Ġ4 , Ġit Ġis Ġassumed Ġto Ġbe Ġpositive . ĠWe Ġhave Ġrandomly Ġselected Ġ7 , 020 Ġnegative Ġand Ġ7 , 020 Ġpositive Ġreviews Ġand Ġprocessed Ġonly Ġthem . </s> The Ġsecond ĠTurkish Ġdataset Ġis Ġthe ĠTwitter Ġcorpus Ġwhich Ġis Ġformed Ġof Ġtweets Ġabout ĠTurkish Ġmobile Ġnetwork Ġoperators . ĠThose Ġtweets Ġare Ġmostly Ġmuch Ġno is ier Ġand Ġshorter Ġcompared Ġto Ġthe Ġreviews Ġin Ġthe Ġmovie Ġcorpus . ĠIn Ġtotal , Ġthere Ġare Ġ1 , 7 16 Ġtweets . Ġ9 73 Ġof Ġthem Ġare Ġnegative Ġand Ġ7 43 Ġof Ġthem Ġare Ġpositive . ĠThese Ġtweets Ġare Ġmanually Ġannot ated Ġby Ġtwo Ġhumans , Ġwhere Ġthe Ġlabels Ġare Ġeither Ġpositive Ġor Ġnegative . ĠWe Ġmeasured Ġthe ĠCohen 's ĠKappa Ġinter - annot ator Ġagreement Ġscore Ġto Ġbe Ġ0 . 82 . ĠIf Ġthere Ġwas Ġa Ġdisagreement Ġon Ġthe Ġpol arity Ġof Ġa Ġtweet , Ġwe Ġremoved Ġit . </s> We Ġalso Ġutil ised Ġtwo Ġother Ġdatasets Ġin ĠEnglish Ġto Ġtest Ġthe Ġcross - ling uality Ġof Ġour Ġapproaches . ĠOne Ġof Ġthem Ġis Ġa Ġmovie Ġcorpus Ġcollected Ġfrom Ġthe Ġweb . ĠThere Ġare Ġ5 , 331 Ġpositive Ġreviews Ġand Ġ5 , 331 Ġnegative Ġreviews Ġin Ġthis Ġcorpus . ĠThe Ġother Ġis Ġa ĠTwitter Ġdataset , Ġwhich Ġhas Ġnearly Ġ1 . 6 Ġmillion Ġtweets Ġannot ated Ġthrough Ġa Ġdistant Ġsupervised Ġmethod ĠB IB REF 14 . ĠThese Ġtweets Ġhave Ġpositive , Ġneutral , Ġand Ġnegative Ġlabels . ĠWe Ġhave Ġselected Ġ7 , 020 Ġpositive Ġtweets Ġand Ġ7 , 020 Ġnegative Ġtweets Ġrandomly Ġto Ġgenerate Ġa Ġbalanced Ġdataset . </s> Exper iments Ġ: :: ĠPre processing </s> In ĠTurkish , Ġpeople Ġsometimes Ġprefer Ġto Ġspell ĠEnglish Ġcharacters Ġfor Ġthe Ġcorresponding ĠTurkish Ġcharacters Ġ( e . g . Ġi Ġfor Ġ Ä± , Ġc Ġfor ĠÃ § ) Ġwhen Ġwriting Ġin Ġelectronic Ġformat . ĠTo Ġnormal ise Ġsuch Ġwords , Ġwe Ġused Ġthe ĠZ emb erek Ġtool ĠB IB REF 15 . ĠAll Ġpunct uation Ġmarks Ġexcept ĠâĢ ľ !" Ġand ĠâĢ ľ ?" Ġare Ġremoved , Ġsince Ġthey Ġdo Ġnot Ġcontribute Ġmuch Ġto Ġthe Ġpol arity Ġof Ġa Ġdocument . ĠWe Ġtook Ġinto Ġaccount Ġemot icons , Ġsuch Ġas ĠâĢ ľ : )) ", Ġand Ġidi oms , Ġsuch Ġas ĠâĢ ľ k af ay Ä± Ġy em ek âĢ Ŀ Ġ( l ose Ġone 's Ġmind ), Ġsince Ġtwo Ġor Ġmore Ġwords Ġcan Ġexpress Ġa Ġsentiment Ġtogether , Ġirrespective Ġof Ġthe Ġindividual Ġwords Ġthereof . ĠSince ĠTurkish Ġis Ġan Ġagg l ut in ative Ġlanguage , Ġwe Ġused Ġthe Ġmorph ological Ġparser Ġand Ġdis amb ig u ator Ġtools ĠB IB REF 16 , ĠB IB REF 17 . ĠWe Ġalso Ġperformed Ġneg ation Ġhandling Ġand Ġstop - word Ġelimination . ĠIn Ġneg ation Ġhandling , Ġwe Ġappend Ġan Ġunderscore Ġto Ġthe Ġend Ġof Ġa Ġword Ġif Ġit Ġis Ġneg ated . ĠFor Ġexample , ĠâĢ ľ g Ã¼ zel Ġde ÄŁ il " Ġ( not Ġbeautiful ) Ġis Ġred efined Ġas ĠâĢ ľ g Ã¼ zel _ " Ġ( beaut iful _ ) Ġin Ġthe Ġfeature Ġselection Ġstage Ġwhen Ġsupervised Ġscores Ġare Ġbeing Ġcomputed . </s> Exper iments Ġ: :: ĠHyper param eters </s> We Ġused Ġthe ĠLib S VM Ġutility Ġof Ġthe ĠWE KA Ġtool . ĠWe Ġchose Ġthe Ġlinear Ġkernel Ġoption Ġto Ġclassify Ġthe Ġreviews . ĠWe Ġtrained Ġword 2 vec Ġembed d ings Ġon Ġall Ġthe Ġfour Ġcorpor a Ġusing Ġthe ĠG ens im Ġlibrary ĠB IB REF 18 Ġwith Ġthe Ġskip - gram Ġmethod . ĠThe Ġdimension Ġsize Ġof Ġthese Ġembed d ings Ġare Ġset Ġat Ġ200 . ĠAs Ġmentioned , Ġother Ġembed d ings , Ġwhich Ġare Ġgenerated Ġutil ising Ġthe Ġclust ering Ġand Ġthe ĠS VD Ġapproach , Ġare Ġalso Ġof Ġsize Ġ200 . ĠFor Ġc - mean Ġclust ering , Ġwe Ġset Ġthe Ġmaximum Ġnumber Ġof Ġiterations Ġat Ġ25 , Ġunless Ġit Ġconver ges . </s> Exper iments Ġ: :: ĠResults </s> We Ġevaluated Ġour Ġmodels Ġon Ġfour Ġcorpor a , Ġwhich Ġare Ġthe Ġmovie Ġand Ġthe ĠTwitter Ġdatasets Ġin ĠTurkish Ġand ĠEnglish . ĠAll Ġof Ġthe Ġembed d ings Ġare Ġlearnt Ġon Ġfour Ġcorpor a Ġseparately . ĠWe Ġhave Ġused Ġthe Ġaccuracy Ġmetric Ġsince Ġall Ġthe Ġdatasets Ġare Ġcompletely Ġor Ġnearly Ġcompletely Ġbalanced . ĠWe Ġperformed Ġ10 - fold Ġcross - valid ation Ġfor Ġboth Ġof Ġthe Ġdatasets . ĠWe Ġused Ġthe Ġapproximate Ġrandom isation Ġtechnique Ġto Ġtest Ġwhether Ġour Ġresults Ġare Ġstatistically Ġsignificant . ĠHere , Ġwe Ġtried Ġto Ġpredict Ġthe Ġlabels Ġof Ġreviews Ġand Ġassess Ġthe Ġperformance . </s> We Ġobtained Ġvarying Ġaccur acies Ġas Ġshown Ġin ĠTable ĠT AB REF 17 . ĠâĢ ľ 3 Ġfeats " Ġfeatures Ġare Ġthose Ġhand - crafted Ġfeatures Ġwe Ġextracted , Ġwhich Ġare Ġthe Ġminimum , Ġmean , Ġand Ġmaximum Ġpol arity Ġscores Ġof Ġthe Ġreviews Ġas Ġexplained Ġin ĠSection Ġ3 . 5 . ĠAs Ġcan Ġbe Ġseen , Ġat Ġleast Ġone Ġof Ġour Ġmethods Ġoutper forms Ġthe Ġbaseline Ġword 2 vec Ġapproach Ġfor Ġall Ġthe ĠTurkish Ġand ĠEnglish Ġcorpor a , Ġand Ġall Ġcategories . ĠAll Ġof Ġour Ġapproaches Ġperformed Ġbetter Ġwhen Ġwe Ġused Ġthe Ġsupervised Ġscores , Ġwhich Ġare Ġextracted Ġon Ġa Ġreview Ġbasis , Ġand Ġconc aten ated Ġthem Ġto Ġword Ġvectors . ĠMostly , Ġthe Ġsupervised Ġ4 - sc ores Ġfeature Ġleads Ġto Ġthe Ġhighest Ġaccur acies , Ġsince Ġit Ġemploys Ġthe Ġannot ational Ġinformation Ġconcerned Ġwith Ġpolar ities Ġon Ġa Ġword Ġbasis . </s> As Ġcan Ġbe Ġseen Ġin ĠTable ĠT AB REF 17 , Ġthe Ġclust ering Ġmethod , Ġin Ġgeneral , Ġyields Ġthe Ġlowest Ġscores . ĠWe Ġfound Ġout Ġthat Ġthe Ġcorpus Ġ- ĠS VD Ġmetric Ġdoes Ġalways Ġperform Ġbetter Ġthan Ġthe Ġclust ering Ġmethod . ĠWe Ġattribute Ġit Ġto Ġthat Ġin ĠS VD Ġthe Ġmost Ġimportant Ġsingular Ġvalues Ġare Ġtaken Ġinto Ġaccount . ĠThe Ġcorpus Ġ- ĠS VD Ġtechnique Ġoutper forms Ġthe Ġword 2 vec Ġalgorithm Ġfor Ġsome Ġcorpor a . ĠWhen Ġwe Ġdo Ġnot Ġtake Ġinto Ġaccount Ġthe Ġ3 - fe ats Ġtechnique , Ġthe Ġcorpus - based ĠS VD Ġmethod Ġyields Ġthe Ġhighest Ġaccur acies Ġfor Ġthe ĠEnglish ĠTwitter Ġdataset . ĠWe Ġshow Ġthat Ġsimple Ġmodels Ġcan Ġoutper form Ġmore Ġcomplex Ġmodels , Ġsuch Ġas Ġthe Ġconc aten ation Ġof Ġthe Ġthree Ġsub appro aches Ġor Ġthe Ġword 2 vec Ġalgorithm . ĠAnother Ġinteresting Ġfinding Ġis Ġthat Ġfor Ġsome Ġcases Ġthe Ġaccuracy Ġdecreases Ġwhen Ġwe Ġutil ise Ġthe Ġpol arity Ġlabels , Ġas Ġin Ġthe Ġcase Ġfor Ġthe ĠEnglish ĠTwitter Ġdataset . </s> Since Ġthe ĠTD K Ġdictionary Ġcovers Ġmost Ġof Ġthe Ġdomain - specific Ġvocabulary Ġused Ġin Ġthe Ġmovie Ġreviews , Ġthe Ġdictionary Ġmethod Ġperforms Ġwell . ĠHowever , Ġthe Ġdictionary Ġlacks Ġmany Ġof Ġthe Ġwords , Ġoccurring Ġin Ġthe Ġtweets ; Ġtherefore , Ġits Ġperformance Ġis Ġnot Ġthe Ġbest Ġof Ġall . ĠWhen Ġthe ĠTD K Ġmethod Ġis Ġcombined Ġwith Ġthe Ġ3 - fe ats Ġtechnique , Ġwe Ġobserved Ġa Ġgreat Ġimprovement , Ġas Ġcan Ġbe Ġexpected . </s> Success Ġrates Ġobtained Ġfor Ġthe Ġmovie Ġcorpus Ġare Ġmuch Ġbetter Ġthan Ġthose Ġfor Ġthe ĠTwitter Ġdataset Ġfor Ġmost Ġof Ġour Ġapproaches , Ġsince Ġtweets Ġare , Ġin Ġgeneral , Ġmuch Ġshorter Ġand Ġno is ier . ĠWe Ġalso Ġfound Ġout Ġthat , Ġwhen Ġchoosing Ġthe Ġp Ġvalue Ġas Ġ0 . 05 , Ġour Ġresults Ġare Ġstatistically Ġsignificant Ġcompared Ġto Ġthe Ġbaseline Ġapproach Ġin ĠTurkish ĠB IB REF 10 . ĠSome Ġof Ġour Ġsub appro aches Ġalso Ġproduce Ġbetter Ġsuccess Ġrates Ġthan Ġthose Ġsentiment Ġanalysis Ġmodels Ġemployed Ġin ĠEnglish ĠB IB REF 11 , ĠB IB REF 12 . ĠWe Ġhave Ġachieved Ġstate - of - the - art Ġresults Ġfor Ġthe Ġsentiment Ġclassification Ġtask Ġfor Ġboth ĠTurkish Ġand ĠEnglish . ĠAs Ġmentioned , Ġour Ġapproaches , Ġin Ġgeneral , Ġperform Ġbest Ġin Ġpredicting Ġthe Ġlabels Ġof Ġreviews Ġwhen Ġthree Ġsupervised Ġscores Ġare Ġaddition ality Ġutil ised . </s> We Ġalso Ġemployed Ġthe Ġconv olution al Ġneural Ġnetwork Ġmodel Ġ( CNN ). ĠHowever , Ġthe ĠS VM Ġclass ifier , Ġwhich Ġis Ġa Ġconventional Ġmachine Ġlearning Ġalgorithm , Ġperformed Ġbetter . ĠWe Ġdid Ġnot Ġinclude Ġthe Ġperformances Ġof ĠCNN Ġfor Ġembed ding Ġtypes Ġhere Ġdue Ġto Ġthe Ġpage Ġlimit Ġof Ġthe Ġpaper . </s> As Ġa Ġqualitative Ġassessment Ġof Ġthe Ġword Ġrepresentations , Ġgiven Ġsome Ġquery Ġwords Ġwe Ġvisual ised Ġthe Ġmost Ġsimilar Ġwords Ġto Ġthose Ġwords Ġusing Ġthe Ġcos ine Ġsimilarity Ġmetric . ĠBy Ġassessing Ġthe Ġsimilarities Ġbetween Ġa Ġword Ġand Ġall Ġthe Ġother Ġcorpus Ġwords , Ġwe Ġcan Ġfind Ġthe Ġmost Ġakin Ġwords Ġaccording Ġto Ġdifferent Ġapproaches . ĠTable ĠT AB REF 18 Ġshows Ġthe Ġmost Ġsimilar Ġwords Ġto Ġgiven Ġquery Ġwords . ĠThose Ġwords Ġwhich Ġare Ġindicative Ġof Ġsentiment Ġare , Ġin Ġgeneral , Ġfound Ġto Ġbe Ġmost Ġsimilar Ġto Ġthose Ġwords Ġof Ġthe Ġsame Ġpol arity . ĠFor Ġexample , Ġthe Ġmost Ġakin Ġword Ġto Ġm uh te ÅŁ em Ġ( g orge ous ) Ġis Ġ10 / 10 , Ġboth Ġof Ġwhich Ġhave Ġpositive Ġpol arity . ĠAs Ġcan Ġbe Ġseen Ġin ĠTable ĠT AB REF 18 , Ġour Ġcorpus - based Ġapproach Ġis Ġmore Ġadept Ġat Ġcapturing Ġdomain - specific Ġfeatures Ġas Ġcompared Ġto Ġword 2 vec , Ġwhich Ġgenerally Ġcaptures Ġgeneral Ġsemantic Ġand Ġsynt actic Ġcharacteristics , Ġbut Ġnot Ġthe Ġsentimental Ġones . </s> Conclusion </s> We Ġhave Ġdemonstrated Ġthat Ġusing Ġword Ġvectors Ġthat Ġcapture Ġonly Ġsemantic Ġand Ġsynt actic Ġcharacteristics Ġmay Ġbe Ġimproved Ġby Ġtaking Ġinto Ġaccount Ġtheir Ġsentimental Ġaspects Ġas Ġwell . ĠOur Ġapproaches Ġare Ġcross - ling ual Ġand Ġcross - domain . ĠThey Ġcan Ġbe Ġapplied Ġto Ġother Ġdomains Ġand Ġother Ġlanguages Ġthan ĠTurkish Ġand ĠEnglish Ġwith Ġminor Ġchanges . </s> Our Ġstudy Ġis Ġone Ġof Ġthe Ġfew Ġones Ġthat Ġperform Ġsentiment Ġanalysis Ġin ĠTurkish Ġand Ġlever ages Ġsentimental Ġcharacteristics Ġof Ġwords Ġin Ġgenerating Ġword Ġvectors Ġand Ġoutper forms Ġall Ġthe Ġothers . ĠAny Ġof Ġthe Ġapproaches Ġwe Ġpropose Ġcan Ġbe Ġused Ġindependently Ġof Ġthe Ġothers . ĠOur Ġapproaches Ġwithout Ġusing Ġsentiment Ġlabels Ġcan Ġbe Ġapplied Ġto Ġother Ġclassification Ġtasks , Ġsuch Ġas Ġtopic Ġclassification Ġand Ġconcept Ġmining . </s> The Ġexperiments Ġshow Ġthat Ġeven Ġun super vised Ġapproaches , Ġas Ġin Ġthe Ġcorpus - based Ġapproach , Ġcan Ġoutper form Ġsupervised Ġapproaches Ġin Ġclassification Ġtasks . ĠComb ining Ġsome Ġapproaches , Ġwhich Ġcan Ġcompensate Ġfor Ġwhat Ġothers Ġlack , Ġcan Ġhelp Ġus Ġbuild Ġbetter Ġvectors . ĠOur Ġword Ġvectors Ġare Ġcreated Ġby Ġconventional Ġmachine Ġlearning Ġalgorithms ; Ġhowever , Ġthey , Ġas Ġin Ġthe Ġcorpus - based Ġmodel , Ġproduce Ġstate - of - the - art Ġresults . ĠAlthough Ġwe Ġpreferred Ġto Ġuse Ġa Ġclassical Ġmachine Ġlearning Ġalgorithm , Ġwhich Ġis ĠS VM , Ġover Ġa Ġneural Ġnetwork Ġclass ifier Ġto Ġpredict Ġthe Ġlabels Ġof Ġreviews , Ġwe Ġachieved Ġaccur acies Ġof Ġover Ġ90 Ġper Ġcent Ġfor Ġthe ĠTurkish Ġmovie Ġcorpus Ġand Ġabout Ġ88 Ġper Ġcent Ġfor Ġthe ĠEnglish ĠTwitter Ġdataset . </s> We Ġperformed Ġonly Ġbinary Ġsentiment Ġclassification Ġin Ġthis Ġstudy Ġas Ġmost Ġof Ġthe Ġstudies Ġin Ġthe Ġliterature Ġdo . ĠWe Ġwill Ġextend Ġour Ġsystem Ġin Ġfuture Ġby Ġusing Ġneutral Ġreviews Ġas Ġwell . ĠWe Ġalso Ġplan Ġto Ġemploy ĠTurkish ĠWord Net Ġto Ġenhance Ġthe Ġgeneral is ability Ġof Ġour Ġembed d ings Ġas Ġanother Ġfuture Ġwork . </s> Acknowled gments </s> This Ġwork Ġwas Ġsupported Ġby ĠBo ÄŁ azi Ã§ i ĠUniversity ĠResearch ĠFund ĠGrant ĠNumber Ġ69 80 D , Ġand Ġby ĠTurkish ĠMinistry Ġof ĠDevelopment Ġunder Ġthe ĠTAM ĠProject Ġnumber ĠD PT 2007 K 12 - 06 10 . ĠC em ĠR Ä± f k Ä± ĠA yd Ä± n Ġis Ġsupported Ġby ĠT Ã ľ B Ä ° TA K ĠB IDE B Ġ22 11 E .
Introduction Sentiment analysis has recently been one of the hottest topics in natural language processing (NLP). It is used to identify and categorise opinions expressed by reviewers on a topic or an entity. Sentiment analysis can be leveraged in marketing, social media analysis, and customer service. Although many studies have been conducted for sentiment analysis in widely spoken languages, this topic is still immature for Turkish and many other languages. Neural networks outperform the conventional machine learning algorithms in most classification tasks, including sentiment analysis BIBREF0. In these networks, word embedding vectors are fed as input to overcome the data sparsity problem and make the representations of words more “meaningful” and robust. Those embeddings indicate how close the words are to each other in the vector space model (VSM). Most of the studies utilise embeddings, such as word2vec BIBREF1, which take into account the syntactic and semantic representations of the words only. Discarding the sentimental aspects of words may lead to words of different polarities being close to each other in the VSM, if they share similar semantic and syntactic features. For Turkish, there are only a few studies which leverage sentimental information in generating the word and document embeddings. Unlike the studies conducted for English and other widely-spoken languages, in this paper, we use the official dictionaries for this language and combine the unsupervised and supervised scores to generate a unified score for each dimension of the word embeddings in this task. Our main contribution is to create original and effective word vectors that capture syntactic, semantic, and sentimental characteristics of words, and use all of this knowledge in generating embeddings. We also utilise the word2vec embeddings trained on a large corpus. Besides using these word embeddings, we also generate hand-crafted features on a review basis and create document vectors. We evaluate those embeddings on two datasets. The results show that we outperform the approaches which do not take into account the sentimental information. We also had better performances than other studies carried out on sentiment analysis in Turkish media. We also evaluated our novel embedding approaches on two English corpora of different genres. We outperformed the baseline approaches for this language as well. The source code and datasets are publicly available. The paper is organised as follows. In Section 2, we present the existing works on sentiment classification. In Section 3, we describe the methods proposed in this work. The experimental results are shown and the main contributions of our proposed approach are discussed in Section 4. In Section 5, we conclude the paper. Related Work In the literature, the main consensus is that the use of dense word embeddings outperform the sparse embeddings in many tasks. Latent semantic analysis (LSA) used to be the most popular method in generating word embeddings before the invention of the word2vec and other word vector algorithms which are mostly created by shallow neural network models. Although many studies have been employed on generating word vectors including both semantic and sentimental components, generating and analysing the effects of different types of embeddings on different tasks is an emerging field for Turkish. Latent Dirichlet allocation (LDA) is used in BIBREF2 to extract mixture of latent topics. However, it focusses on finding the latent topics of a document, not the word meanings themselves. In BIBREF3, LSA is utilised to generate word vectors, leveraging indirect co-occurrence statistics. These outperform the use of sparse vectors BIBREF4. Some of the prior studies have also taken into account the sentimental characteristics of a word when creating word vectors BIBREF5, BIBREF6, BIBREF7. A model with semantic and sentiment components is built in BIBREF8, making use of star-ratings of reviews. In BIBREF9, a sentiment lexicon is induced preferring the use of domain-specific co-occurrence statistics over the word2vec method and they outperform the latter. In a recent work on sentiment analysis in Turkish BIBREF10, they learn embeddings using Turkish social media. They use the word2vec algorithm, create several unsupervised hand-crafted features, generate document vectors and feed them as input into the support vector machines (SVM) approach. We outperform this baseline approach using more effective word embeddings and supervised hand-crafted features. In English, much of the recent work on learning sentiment-specific embeddings relies only on distant supervision. In BIBREF11, emojis are used as features and a bi-LSTM (bidirectional long short-term memory) neural network model is built to learn sentiment-aware word embeddings. In BIBREF12, a neural network that learns word embeddings is built by using contextual information about the data and supervised scores of the words. This work captures the supervised information by utilising emoticons as features. Most of our approaches do not rely on a neural network model in learning embeddings. However, they produce state-of-the-art results. Methodology We generate several word vectors, which capture the sentimental, lexical, and contextual characteristics of words. In addition to these mostly original vectors, we also create word2vec embeddings to represent the corpus words by training the embedding model on these datasets. After generating these, we combine them with hand-crafted features to create document vectors and perform classification, as will be explained in Section 3.5. Methodology ::: Corpus-based Approach Contextual information is informative in the sense that, in general, similar words tend to appear in the same contexts. For example, the word smart is more likely to cooccur with the word hardworking than with the word lazy. This similarity can be defined semantically and sentimentally. In the corpus-based approach, we capture both of these characteristics and generate word embeddings specific to a domain. Firstly, we construct a matrix whose entries correspond to the number of cooccurrences of the row and column words in sliding windows. Diagonal entries are assigned the number of sliding windows that the corresponding row word appears in the whole corpus. We then normalise each row by dividing entries in the row by the maximum score in it. Secondly, we perform the principal component analysis (PCA) method to reduce the dimensionality. It captures latent meanings and takes into account high-order cooccurrence removing noise. The attribute (column) number of the matrix is reduced to 200. We then compute cosine similarity between each row pair $w_i$ and $w_j$ as in (DISPLAY_FORM3) to find out how similar two word vectors (rows) are. Thirdly, all the values in the matrix are subtracted from 1 to create a dissimilarity matrix. Then, we feed the matrix as input into the fuzzy c-means clustering algorithm. We chose the number of clusters as 200, as it is considered a standard for word embeddings in the literature. After clustering, the dimension i for a corresponding word indicates the degree to which this word belongs to cluster i. The intuition behind this idea is that if two words are similar in the VSM, they are more likely to belong to the same clusters with akin probabilities. In the end, each word in the corpus is represented by a 200-dimensional vector. In addition to this method, we also perform singular value decomposition (SVD) on the cooccurrence matrices, where we compute the matrix $M^{PPMI} = U\Sigma V^{T}$. Positive pointwise mutual information (PPMI) scores between words are calculated and the truncated singular value decomposition is computed. We take into account the U matrix only for each word. We have chosen the singular value number as 200. That is, each word in the corpus is represented by a 200-dimensional vector as follows. Methodology ::: Dictionary-based Approach In Turkish, there do not exist well-established sentiment lexicons as in English. In this approach, we made use of the TDK (Türk Dil Kurumu - “Turkish Language Institution”) dictionary to obtain word polarities. Although it is not a sentiment lexicon, combining it with domain-specific polarity scores obtained from the corpus led us to have state-of-the-art results. We first construct a matrix whose row entries are corpus words and column entries are the words in their dictionary definitions. We followed the boolean approach. For instance, for the word cat, the column words occurring in its dictionary definition are given a score of 1. Those column words not appearing in the definition of cat are assigned a score of 0 for that corresponding row entry. When we performed clustering on this matrix, we observed that those words having similar meanings are, in general, assigned to the same clusters. However, this similarity fails in capturing the sentimental characteristics. For instance, the words happy and unhappy are assigned to the same cluster, since they have the same words, such as feeling, in their dictionary definitions. However, they are of opposite polarities and should be discerned from each other. Therefore, we utilise a metric to move such words away from each other in the VSM, even though they have common words in their dictionary definitions. We multiply each value in a row with the corresponding row word's raw supervised score, thereby having more meaningful clusters. Using the training data only, the supervised polarity score per word is calculated as in (DISPLAY_FORM4). Here, $ w_{t}$ denotes the sentiment score of word $t$, $N_{t}$ is the number of documents (reviews or tweets) in which $t$ occurs in the dataset of positive polarity, $N$ is the number of all the words in the corpus of positive polarity. $N^{\prime }$ denotes the corpus of negative polarity. $N^{\prime }_{t}$ and $N^{\prime }$ denote similar values for the negative polarity corpus. We perform normalisation to prevent the imbalance problem and add a small number to both numerator and denominator for smoothing. As an alternative to multiplying with the supervised polarity scores, we also separately multiplied all the row scores with only +1 if the row word is a positive word, and with -1 if it is a negative word. We have observed it boosts the performance more compared to using raw scores. The effect of this multiplication is exemplified in Figure FIGREF7, showing the positions of word vectors in the VSM. Those “x" words are sentimentally negative words, those “o" words are sentimentally positive ones. On the top coordinate plane, the words of opposite polarities are found to be close to each other, since they have common words in their dictionary definitions. Only the information concerned with the dictionary definitions are used there, discarding the polarity scores. However, when we utilise the supervised score (+1 or -1), words of opposite polarities (e.g. “happy" and “unhappy") get far away from each other as they are translated across coordinate regions. Positive words now appear in quadrant 1, whereas negative words appear in quadrant 3. Thus, in the VSM, words that are sentimentally similar to each other could be clustered more accurately. Besides clustering, we also employed the SVD method to perform dimensionality reduction on the unsupervised dictionary algorithm and used the newly generated matrix by combining it with other subapproaches. The number of dimensions is chosen as 200 again according to the $U$ matrix. The details are given in Section 3.4. When using and evaluating this subapproach on the English corpora, we used the SentiWordNet lexicon BIBREF13. We have achieved better results for the dictionary-based algorithm when we employed the SVD reduction method compared to the use of clustering. Methodology ::: Supervised Contextual 4-scores Our last component is a simple metric that uses four supervised scores for each word in the corpus. We extract these scores as follows. For a target word in the corpus, we scan through all of its contexts. In addition to the target word's polarity score (the self score), out of all the polarity scores of words occurring in the same contexts as the target word, minimum, maximum, and average scores are taken into consideration. The word polarity scores are computed using (DISPLAY_FORM4). Here, we obtain those scores from the training data. The intuition behind this method is that those four scores are more indicative of a word's polarity rather than only one (the self score). This approach is fully supervised unlike the previous two approaches. Methodology ::: Combination of the Word Embeddings In addition to using the three approaches independently, we also combined all the matrices generated in the previous approaches. That is, we concatenate the reduced forms (SVD - U) of corpus-based, dictionary-based, and the whole of 4-score vectors of each word, horizontally. Accordingly, each corpus word is represented by a 404-dimensional vector, since corpus-based and dictionary-based vector components are each composed of 200 dimensions, whereas the 4-score vector component is formed by four values. The main intuition behind the ensemble method is that some approaches compensate for what the others may lack. For example, the corpus-based approach captures the domain-specific, semantic, and syntactic characteristics. On the other hand, the 4-scores method captures supervised features, and the dictionary-based approach is helpful in capturing the general semantic characteristics. That is, combining those three approaches makes word vectors more representative. Methodology ::: Generating Document Vectors After creating several embeddings as mentioned above, we create document (review or tweet) vectors. For each document, we sum all the vectors of words occurring in that document and take their average. In addition to it, we extract three hand-crafted polarity scores, which are minimum, mean, and maximum polarity scores, from each review. These polarity scores of words are computed as in (DISPLAY_FORM4). For example, if a review consists of five words, it would have five polarity scores and we utilise only three of these sentiment scores as mentioned. Lastly, we concatenate these three scores to the averaged word vector per review. That is, each review is represented by the average word vector of its constituent word embeddings and three supervised scores. We then feed these inputs into the SVM approach. The flowchart of our framework is given in Figure FIGREF11. When combining the unsupervised features, which are word vectors created on a word basis, with supervised three scores extracted on a review basis, we have better state-of-the-art results. Datasets We utilised two datasets for both Turkish and English to evaluate our methods. For Turkish, as the first dataset, we utilised the movie reviews which are collected from a popular website. The number of reviews in this movie corpus is 20,244 and the average number of words in reviews is 39. Each of these reviews has a star-rating score which is indicative of sentiment. These polarity scores are between the values 0.5 and 5, at intervals of 0.5. We consider a review to be negative it the score is equal to or lower than 2.5. On the other hand, if it is equal to or higher than 4, it is assumed to be positive. We have randomly selected 7,020 negative and 7,020 positive reviews and processed only them. The second Turkish dataset is the Twitter corpus which is formed of tweets about Turkish mobile network operators. Those tweets are mostly much noisier and shorter compared to the reviews in the movie corpus. In total, there are 1,716 tweets. 973 of them are negative and 743 of them are positive. These tweets are manually annotated by two humans, where the labels are either positive or negative. We measured the Cohen's Kappa inter-annotator agreement score to be 0.82. If there was a disagreement on the polarity of a tweet, we removed it. We also utilised two other datasets in English to test the cross-linguality of our approaches. One of them is a movie corpus collected from the web. There are 5,331 positive reviews and 5,331 negative reviews in this corpus. The other is a Twitter dataset, which has nearly 1.6 million tweets annotated through a distant supervised method BIBREF14. These tweets have positive, neutral, and negative labels. We have selected 7,020 positive tweets and 7,020 negative tweets randomly to generate a balanced dataset. Experiments ::: Preprocessing In Turkish, people sometimes prefer to spell English characters for the corresponding Turkish characters (e.g. i for ı, c for ç) when writing in electronic format. To normalise such words, we used the Zemberek tool BIBREF15. All punctuation marks except “!" and “?" are removed, since they do not contribute much to the polarity of a document. We took into account emoticons, such as “:))", and idioms, such as “kafayı yemek” (lose one's mind), since two or more words can express a sentiment together, irrespective of the individual words thereof. Since Turkish is an agglutinative language, we used the morphological parser and disambiguator tools BIBREF16, BIBREF17. We also performed negation handling and stop-word elimination. In negation handling, we append an underscore to the end of a word if it is negated. For example, “güzel değil" (not beautiful) is redefined as “güzel_" (beautiful_) in the feature selection stage when supervised scores are being computed. Experiments ::: Hyperparameters We used the LibSVM utility of the WEKA tool. We chose the linear kernel option to classify the reviews. We trained word2vec embeddings on all the four corpora using the Gensim library BIBREF18 with the skip-gram method. The dimension size of these embeddings are set at 200. As mentioned, other embeddings, which are generated utilising the clustering and the SVD approach, are also of size 200. For c-mean clustering, we set the maximum number of iterations at 25, unless it converges. Experiments ::: Results We evaluated our models on four corpora, which are the movie and the Twitter datasets in Turkish and English. All of the embeddings are learnt on four corpora separately. We have used the accuracy metric since all the datasets are completely or nearly completely balanced. We performed 10-fold cross-validation for both of the datasets. We used the approximate randomisation technique to test whether our results are statistically significant. Here, we tried to predict the labels of reviews and assess the performance. We obtained varying accuracies as shown in Table TABREF17. “3 feats" features are those hand-crafted features we extracted, which are the minimum, mean, and maximum polarity scores of the reviews as explained in Section 3.5. As can be seen, at least one of our methods outperforms the baseline word2vec approach for all the Turkish and English corpora, and all categories. All of our approaches performed better when we used the supervised scores, which are extracted on a review basis, and concatenated them to word vectors. Mostly, the supervised 4-scores feature leads to the highest accuracies, since it employs the annotational information concerned with polarities on a word basis. As can be seen in Table TABREF17, the clustering method, in general, yields the lowest scores. We found out that the corpus - SVD metric does always perform better than the clustering method. We attribute it to that in SVD the most important singular values are taken into account. The corpus - SVD technique outperforms the word2vec algorithm for some corpora. When we do not take into account the 3-feats technique, the corpus-based SVD method yields the highest accuracies for the English Twitter dataset. We show that simple models can outperform more complex models, such as the concatenation of the three subapproaches or the word2vec algorithm. Another interesting finding is that for some cases the accuracy decreases when we utilise the polarity labels, as in the case for the English Twitter dataset. Since the TDK dictionary covers most of the domain-specific vocabulary used in the movie reviews, the dictionary method performs well. However, the dictionary lacks many of the words, occurring in the tweets; therefore, its performance is not the best of all. When the TDK method is combined with the 3-feats technique, we observed a great improvement, as can be expected. Success rates obtained for the movie corpus are much better than those for the Twitter dataset for most of our approaches, since tweets are, in general, much shorter and noisier. We also found out that, when choosing the p value as 0.05, our results are statistically significant compared to the baseline approach in Turkish BIBREF10. Some of our subapproaches also produce better success rates than those sentiment analysis models employed in English BIBREF11, BIBREF12. We have achieved state-of-the-art results for the sentiment classification task for both Turkish and English. As mentioned, our approaches, in general, perform best in predicting the labels of reviews when three supervised scores are additionality utilised. We also employed the convolutional neural network model (CNN). However, the SVM classifier, which is a conventional machine learning algorithm, performed better. We did not include the performances of CNN for embedding types here due to the page limit of the paper. As a qualitative assessment of the word representations, given some query words we visualised the most similar words to those words using the cosine similarity metric. By assessing the similarities between a word and all the other corpus words, we can find the most akin words according to different approaches. Table TABREF18 shows the most similar words to given query words. Those words which are indicative of sentiment are, in general, found to be most similar to those words of the same polarity. For example, the most akin word to muhteşem (gorgeous) is 10/10, both of which have positive polarity. As can be seen in Table TABREF18, our corpus-based approach is more adept at capturing domain-specific features as compared to word2vec, which generally captures general semantic and syntactic characteristics, but not the sentimental ones. Conclusion We have demonstrated that using word vectors that capture only semantic and syntactic characteristics may be improved by taking into account their sentimental aspects as well. Our approaches are cross-lingual and cross-domain. They can be applied to other domains and other languages than Turkish and English with minor changes. Our study is one of the few ones that perform sentiment analysis in Turkish and leverages sentimental characteristics of words in generating word vectors and outperforms all the others. Any of the approaches we propose can be used independently of the others. Our approaches without using sentiment labels can be applied to other classification tasks, such as topic classification and concept mining. The experiments show that even unsupervised approaches, as in the corpus-based approach, can outperform supervised approaches in classification tasks. Combining some approaches, which can compensate for what others lack, can help us build better vectors. Our word vectors are created by conventional machine learning algorithms; however, they, as in the corpus-based model, produce state-of-the-art results. Although we preferred to use a classical machine learning algorithm, which is SVM, over a neural network classifier to predict the labels of reviews, we achieved accuracies of over 90 per cent for the Turkish movie corpus and about 88 per cent for the English Twitter dataset. We performed only binary sentiment classification in this study as most of the studies in the literature do. We will extend our system in future by using neutral reviews as well. We also plan to employ Turkish WordNet to enhance the generalisability of our embeddings as another future work. Acknowledgments This work was supported by Boğaziçi University Research Fund Grant Number 6980D, and by Turkish Ministry of Development under the TAM Project number DPT2007K12-0610. Cem Rıfkı Aydın is supported by TÜBİTAK BIDEB 2211E.
Which hand-crafted features are combined with word2vec?
0d34c0812f1e69ea33f76ca8c24c23b0415ebc8d
Which Ġhand - crafted Ġfeatures Ġare Ġcombined Ġwith Ġword 2 vec ?
three Ġhand - crafted Ġpol arity Ġscores , Ġwhich Ġare Ġminimum , Ġmean , Ġand Ġmaximum Ġpol arity Ġscores
<s> Which Ġhand - crafted Ġfeatures Ġare Ġcombined Ġwith Ġword 2 vec ? </s> Introduction </s> Sent iment Ġanalysis Ġhas Ġrecently Ġbeen Ġone Ġof Ġthe Ġhottest Ġtopics Ġin Ġnatural Ġlanguage Ġprocessing Ġ( N LP ). ĠIt Ġis Ġused Ġto Ġidentify Ġand Ġcategor ise Ġopinions Ġexpressed Ġby Ġreviewers Ġon Ġa Ġtopic Ġor Ġan Ġentity . ĠSent iment Ġanalysis Ġcan Ġbe Ġlever aged Ġin Ġmarketing , Ġsocial Ġmedia Ġanalysis , Ġand Ġcustomer Ġservice . ĠAlthough Ġmany Ġstudies Ġhave Ġbeen Ġconducted Ġfor Ġsentiment Ġanalysis Ġin Ġwidely Ġspoken Ġlanguages , Ġthis Ġtopic Ġis Ġstill Ġimmature Ġfor ĠTurkish Ġand Ġmany Ġother Ġlanguages . </s> Ne ural Ġnetworks Ġoutper form Ġthe Ġconventional Ġmachine Ġlearning Ġalgorithms Ġin Ġmost Ġclassification Ġtasks , Ġincluding Ġsentiment Ġanalysis ĠB IB REF 0 . ĠIn Ġthese Ġnetworks , Ġword Ġembed ding Ġvectors Ġare Ġfed Ġas Ġinput Ġto Ġovercome Ġthe Ġdata Ġsp arsity Ġproblem Ġand Ġmake Ġthe Ġrepresentations Ġof Ġwords Ġmore ĠâĢ ľ meaning ful âĢ Ŀ Ġand Ġrobust . ĠThose Ġembed d ings Ġindicate Ġhow Ġclose Ġthe Ġwords Ġare Ġto Ġeach Ġother Ġin Ġthe Ġvector Ġspace Ġmodel Ġ( V SM ). </s> Most Ġof Ġthe Ġstudies Ġutil ise Ġembed d ings , Ġsuch Ġas Ġword 2 vec ĠB IB REF 1 , Ġwhich Ġtake Ġinto Ġaccount Ġthe Ġsynt actic Ġand Ġsemantic Ġrepresentations Ġof Ġthe Ġwords Ġonly . ĠDisc arding Ġthe Ġsentimental Ġaspects Ġof Ġwords Ġmay Ġlead Ġto Ġwords Ġof Ġdifferent Ġpolar ities Ġbeing Ġclose Ġto Ġeach Ġother Ġin Ġthe ĠV SM , Ġif Ġthey Ġshare Ġsimilar Ġsemantic Ġand Ġsynt actic Ġfeatures . </s> For ĠTurkish , Ġthere Ġare Ġonly Ġa Ġfew Ġstudies Ġwhich Ġleverage Ġsentimental Ġinformation Ġin Ġgenerating Ġthe Ġword Ġand Ġdocument Ġembed d ings . ĠUnlike Ġthe Ġstudies Ġconducted Ġfor ĠEnglish Ġand Ġother Ġwidely - spoken Ġlanguages , Ġin Ġthis Ġpaper , Ġwe Ġuse Ġthe Ġofficial Ġdiction aries Ġfor Ġthis Ġlanguage Ġand Ġcombine Ġthe Ġun super vised Ġand Ġsupervised Ġscores Ġto Ġgenerate Ġa Ġunified Ġscore Ġfor Ġeach Ġdimension Ġof Ġthe Ġword Ġembed d ings Ġin Ġthis Ġtask . </s> Our Ġmain Ġcontribution Ġis Ġto Ġcreate Ġoriginal Ġand Ġeffective Ġword Ġvectors Ġthat Ġcapture Ġsynt actic , Ġsemantic , Ġand Ġsentimental Ġcharacteristics Ġof Ġwords , Ġand Ġuse Ġall Ġof Ġthis Ġknowledge Ġin Ġgenerating Ġembed d ings . ĠWe Ġalso Ġutil ise Ġthe Ġword 2 vec Ġembed d ings Ġtrained Ġon Ġa Ġlarge Ġcorpus . ĠBesides Ġusing Ġthese Ġword Ġembed d ings , Ġwe Ġalso Ġgenerate Ġhand - crafted Ġfeatures Ġon Ġa Ġreview Ġbasis Ġand Ġcreate Ġdocument Ġvectors . ĠWe Ġevaluate Ġthose Ġembed d ings Ġon Ġtwo Ġdatasets . ĠThe Ġresults Ġshow Ġthat Ġwe Ġoutper form Ġthe Ġapproaches Ġwhich Ġdo Ġnot Ġtake Ġinto Ġaccount Ġthe Ġsentimental Ġinformation . ĠWe Ġalso Ġhad Ġbetter Ġperformances Ġthan Ġother Ġstudies Ġcarried Ġout Ġon Ġsentiment Ġanalysis Ġin ĠTurkish Ġmedia . ĠWe Ġalso Ġevaluated Ġour Ġnovel Ġembed ding Ġapproaches Ġon Ġtwo ĠEnglish Ġcorpor a Ġof Ġdifferent Ġgenres . ĠWe Ġoutper formed Ġthe Ġbaseline Ġapproaches Ġfor Ġthis Ġlanguage Ġas Ġwell . ĠThe Ġsource Ġcode Ġand Ġdatasets Ġare Ġpublicly Ġavailable . </s> The Ġpaper Ġis Ġorganised Ġas Ġfollows . ĠIn ĠSection Ġ2 , Ġwe Ġpresent Ġthe Ġexisting Ġworks Ġon Ġsentiment Ġclassification . ĠIn ĠSection Ġ3 , Ġwe Ġdescribe Ġthe Ġmethods Ġproposed Ġin Ġthis Ġwork . ĠThe Ġexperimental Ġresults Ġare Ġshown Ġand Ġthe Ġmain Ġcontributions Ġof Ġour Ġproposed Ġapproach Ġare Ġdiscussed Ġin ĠSection Ġ4 . ĠIn ĠSection Ġ5 , Ġwe Ġconclude Ġthe Ġpaper . </s> Related ĠWork </s> In Ġthe Ġliterature , Ġthe Ġmain Ġconsensus Ġis Ġthat Ġthe Ġuse Ġof Ġdense Ġword Ġembed d ings Ġoutper form Ġthe Ġsparse Ġembed d ings Ġin Ġmany Ġtasks . ĠLat ent Ġsemantic Ġanalysis Ġ( L SA ) Ġused Ġto Ġbe Ġthe Ġmost Ġpopular Ġmethod Ġin Ġgenerating Ġword Ġembed d ings Ġbefore Ġthe Ġinvention Ġof Ġthe Ġword 2 vec Ġand Ġother Ġword Ġvector Ġalgorithms Ġwhich Ġare Ġmostly Ġcreated Ġby Ġshallow Ġneural Ġnetwork Ġmodels . ĠAlthough Ġmany Ġstudies Ġhave Ġbeen Ġemployed Ġon Ġgenerating Ġword Ġvectors Ġincluding Ġboth Ġsemantic Ġand Ġsentimental Ġcomponents , Ġgenerating Ġand Ġanalys ing Ġthe Ġeffects Ġof Ġdifferent Ġtypes Ġof Ġembed d ings Ġon Ġdifferent Ġtasks Ġis Ġan Ġemerging Ġfield Ġfor ĠTurkish . </s> Lat ent ĠDir ich let Ġallocation Ġ( L DA ) Ġis Ġused Ġin ĠB IB REF 2 Ġto Ġextract Ġmixture Ġof Ġlatent Ġtopics . ĠHowever , Ġit Ġfoc uss es Ġon Ġfinding Ġthe Ġlatent Ġtopics Ġof Ġa Ġdocument , Ġnot Ġthe Ġword Ġmeanings Ġthemselves . ĠIn ĠB IB REF 3 , ĠL SA Ġis Ġutil ised Ġto Ġgenerate Ġword Ġvectors , Ġleveraging Ġindirect Ġco - occ urrence Ġstatistics . ĠThese Ġoutper form Ġthe Ġuse Ġof Ġsparse Ġvectors ĠB IB REF 4 . </s> Some Ġof Ġthe Ġprior Ġstudies Ġhave Ġalso Ġtaken Ġinto Ġaccount Ġthe Ġsentimental Ġcharacteristics Ġof Ġa Ġword Ġwhen Ġcreating Ġword Ġvectors ĠB IB REF 5 , ĠB IB REF 6 , ĠB IB REF 7 . ĠA Ġmodel Ġwith Ġsemantic Ġand Ġsentiment Ġcomponents Ġis Ġbuilt Ġin ĠB IB REF 8 , Ġmaking Ġuse Ġof Ġstar - rat ings Ġof Ġreviews . ĠIn ĠB IB REF 9 , Ġa Ġsentiment Ġlex icon Ġis Ġinduced Ġpreferring Ġthe Ġuse Ġof Ġdomain - specific Ġco - occ urrence Ġstatistics Ġover Ġthe Ġword 2 vec Ġmethod Ġand Ġthey Ġoutper form Ġthe Ġlatter . </s> In Ġa Ġrecent Ġwork Ġon Ġsentiment Ġanalysis Ġin ĠTurkish ĠB IB REF 10 , Ġthey Ġlearn Ġembed d ings Ġusing ĠTurkish Ġsocial Ġmedia . ĠThey Ġuse Ġthe Ġword 2 vec Ġalgorithm , Ġcreate Ġseveral Ġun super vised Ġhand - crafted Ġfeatures , Ġgenerate Ġdocument Ġvectors Ġand Ġfeed Ġthem Ġas Ġinput Ġinto Ġthe Ġsupport Ġvector Ġmachines Ġ( S VM ) Ġapproach . ĠWe Ġoutper form Ġthis Ġbaseline Ġapproach Ġusing Ġmore Ġeffective Ġword Ġembed d ings Ġand Ġsupervised Ġhand - crafted Ġfeatures . </s> In ĠEnglish , Ġmuch Ġof Ġthe Ġrecent Ġwork Ġon Ġlearning Ġsentiment - specific Ġembed d ings Ġrelies Ġonly Ġon Ġdistant Ġsupervision . ĠIn ĠB IB REF 11 , Ġem oj is Ġare Ġused Ġas Ġfeatures Ġand Ġa Ġbi - L ST M Ġ( bid irection al Ġlong Ġshort - term Ġmemory ) Ġneural Ġnetwork Ġmodel Ġis Ġbuilt Ġto Ġlearn Ġsentiment - aware Ġword Ġembed d ings . ĠIn ĠB IB REF 12 , Ġa Ġneural Ġnetwork Ġthat Ġlearns Ġword Ġembed d ings Ġis Ġbuilt Ġby Ġusing Ġcontextual Ġinformation Ġabout Ġthe Ġdata Ġand Ġsupervised Ġscores Ġof Ġthe Ġwords . ĠThis Ġwork Ġcaptures Ġthe Ġsupervised Ġinformation Ġby Ġutil ising Ġemot icons Ġas Ġfeatures . ĠMost Ġof Ġour Ġapproaches Ġdo Ġnot Ġrely Ġon Ġa Ġneural Ġnetwork Ġmodel Ġin Ġlearning Ġembed d ings . ĠHowever , Ġthey Ġproduce Ġstate - of - the - art Ġresults . </s> Method ology </s> We Ġgenerate Ġseveral Ġword Ġvectors , Ġwhich Ġcapture Ġthe Ġsentimental , Ġlex ical , Ġand Ġcontextual Ġcharacteristics Ġof Ġwords . ĠIn Ġaddition Ġto Ġthese Ġmostly Ġoriginal Ġvectors , Ġwe Ġalso Ġcreate Ġword 2 vec Ġembed d ings Ġto Ġrepresent Ġthe Ġcorpus Ġwords Ġby Ġtraining Ġthe Ġembed ding Ġmodel Ġon Ġthese Ġdatasets . ĠAfter Ġgenerating Ġthese , Ġwe Ġcombine Ġthem Ġwith Ġhand - crafted Ġfeatures Ġto Ġcreate Ġdocument Ġvectors Ġand Ġperform Ġclassification , Ġas Ġwill Ġbe Ġexplained Ġin ĠSection Ġ3 . 5 . </s> Method ology Ġ: :: ĠCorpus - based ĠApproach </s> Context ual Ġinformation Ġis Ġinformative Ġin Ġthe Ġsense Ġthat , Ġin Ġgeneral , Ġsimilar Ġwords Ġtend Ġto Ġappear Ġin Ġthe Ġsame Ġcontexts . ĠFor Ġexample , Ġthe Ġword Ġsmart Ġis Ġmore Ġlikely Ġto Ġco occ ur Ġwith Ġthe Ġword Ġhard working Ġthan Ġwith Ġthe Ġword Ġlazy . ĠThis Ġsimilarity Ġcan Ġbe Ġdefined Ġsem antically Ġand Ġsentiment ally . ĠIn Ġthe Ġcorpus - based Ġapproach , Ġwe Ġcapture Ġboth Ġof Ġthese Ġcharacteristics Ġand Ġgenerate Ġword Ġembed d ings Ġspecific Ġto Ġa Ġdomain . </s> Firstly , Ġwe Ġconstruct Ġa Ġmatrix Ġwhose Ġentries Ġcorrespond Ġto Ġthe Ġnumber Ġof Ġco occ ur rences Ġof Ġthe Ġrow Ġand Ġcolumn Ġwords Ġin Ġsliding Ġwindows . ĠDi agonal Ġentries Ġare Ġassigned Ġthe Ġnumber Ġof Ġsliding Ġwindows Ġthat Ġthe Ġcorresponding Ġrow Ġword Ġappears Ġin Ġthe Ġwhole Ġcorpus . ĠWe Ġthen Ġnormal ise Ġeach Ġrow Ġby Ġdividing Ġentries Ġin Ġthe Ġrow Ġby Ġthe Ġmaximum Ġscore Ġin Ġit . </s> Secondly , Ġwe Ġperform Ġthe Ġprincipal Ġcomponent Ġanalysis Ġ( PC A ) Ġmethod Ġto Ġreduce Ġthe Ġdimension ality . ĠIt Ġcaptures Ġlatent Ġmeanings Ġand Ġtakes Ġinto Ġaccount Ġhigh - order Ġco occ urrence Ġremoving Ġnoise . ĠThe Ġattribute Ġ( column ) Ġnumber Ġof Ġthe Ġmatrix Ġis Ġreduced Ġto Ġ200 . ĠWe Ġthen Ġcompute Ġcos ine Ġsimilarity Ġbetween Ġeach Ġrow Ġpair Ġ$ w _ i $ Ġand Ġ$ w _ j $ Ġas Ġin Ġ( DIS PLAY _ FORM 3 ) Ġto Ġfind Ġout Ġhow Ġsimilar Ġtwo Ġword Ġvectors Ġ( rows ) Ġare . </s> Third ly , Ġall Ġthe Ġvalues Ġin Ġthe Ġmatrix Ġare Ġsubt racted Ġfrom Ġ1 Ġto Ġcreate Ġa Ġdiss imilar ity Ġmatrix . ĠThen , Ġwe Ġfeed Ġthe Ġmatrix Ġas Ġinput Ġinto Ġthe Ġfuzzy Ġc - me ans Ġclust ering Ġalgorithm . ĠWe Ġchose Ġthe Ġnumber Ġof Ġclusters Ġas Ġ200 , Ġas Ġit Ġis Ġconsidered Ġa Ġstandard Ġfor Ġword Ġembed d ings Ġin Ġthe Ġliterature . ĠAfter Ġclust ering , Ġthe Ġdimension Ġi Ġfor Ġa Ġcorresponding Ġword Ġindicates Ġthe Ġdegree Ġto Ġwhich Ġthis Ġword Ġbelongs Ġto Ġcluster Ġi . ĠThe Ġintuition Ġbehind Ġthis Ġidea Ġis Ġthat Ġif Ġtwo Ġwords Ġare Ġsimilar Ġin Ġthe ĠV SM , Ġthey Ġare Ġmore Ġlikely Ġto Ġbelong Ġto Ġthe Ġsame Ġclusters Ġwith Ġakin Ġprobabilities . ĠIn Ġthe Ġend , Ġeach Ġword Ġin Ġthe Ġcorpus Ġis Ġrepresented Ġby Ġa Ġ200 - dimensional Ġvector . </s> In Ġaddition Ġto Ġthis Ġmethod , Ġwe Ġalso Ġperform Ġsingular Ġvalue Ġdecom position Ġ( S VD ) Ġon Ġthe Ġco occ urrence Ġmat rices , Ġwhere Ġwe Ġcompute Ġthe Ġmatrix Ġ$ M ^{ P PM I } Ġ= ĠU \ S igma ĠV ^{ T } $. ĠPositive Ġpoint wise Ġmutual Ġinformation Ġ( P PM I ) Ġscores Ġbetween Ġwords Ġare Ġcalculated Ġand Ġthe Ġtrunc ated Ġsingular Ġvalue Ġdecom position Ġis Ġcomputed . ĠWe Ġtake Ġinto Ġaccount Ġthe ĠU Ġmatrix Ġonly Ġfor Ġeach Ġword . ĠWe Ġhave Ġchosen Ġthe Ġsingular Ġvalue Ġnumber Ġas Ġ200 . ĠThat Ġis , Ġeach Ġword Ġin Ġthe Ġcorpus Ġis Ġrepresented Ġby Ġa Ġ200 - dimensional Ġvector Ġas Ġfollows . </s> Method ology Ġ: :: ĠDictionary - based ĠApproach </s> In ĠTurkish , Ġthere Ġdo Ġnot Ġexist Ġwell - established Ġsentiment Ġlex icons Ġas Ġin ĠEnglish . ĠIn Ġthis Ġapproach , Ġwe Ġmade Ġuse Ġof Ġthe ĠTD K Ġ( T Ã¼r k ĠDil ĠKur um u Ġ- ĠâĢ ľ Turkish ĠLanguage ĠInstitution âĢ Ŀ ) Ġdictionary Ġto Ġobtain Ġword Ġpolar ities . ĠAlthough Ġit Ġis Ġnot Ġa Ġsentiment Ġlex icon , Ġcombining Ġit Ġwith Ġdomain - specific Ġpol arity Ġscores Ġobtained Ġfrom Ġthe Ġcorpus Ġled Ġus Ġto Ġhave Ġstate - of - the - art Ġresults . </s> We Ġfirst Ġconstruct Ġa Ġmatrix Ġwhose Ġrow Ġentries Ġare Ġcorpus Ġwords Ġand Ġcolumn Ġentries Ġare Ġthe Ġwords Ġin Ġtheir Ġdictionary Ġdefinitions . ĠWe Ġfollowed Ġthe Ġboolean Ġapproach . ĠFor Ġinstance , Ġfor Ġthe Ġword Ġcat , Ġthe Ġcolumn Ġwords Ġoccurring Ġin Ġits Ġdictionary Ġdefinition Ġare Ġgiven Ġa Ġscore Ġof Ġ1 . ĠThose Ġcolumn Ġwords Ġnot Ġappearing Ġin Ġthe Ġdefinition Ġof Ġcat Ġare Ġassigned Ġa Ġscore Ġof Ġ0 Ġfor Ġthat Ġcorresponding Ġrow Ġentry . </s> When Ġwe Ġperformed Ġclust ering Ġon Ġthis Ġmatrix , Ġwe Ġobserved Ġthat Ġthose Ġwords Ġhaving Ġsimilar Ġmeanings Ġare , Ġin Ġgeneral , Ġassigned Ġto Ġthe Ġsame Ġclusters . ĠHowever , Ġthis Ġsimilarity Ġfails Ġin Ġcapturing Ġthe Ġsentimental Ġcharacteristics . ĠFor Ġinstance , Ġthe Ġwords Ġhappy Ġand Ġunhappy Ġare Ġassigned Ġto Ġthe Ġsame Ġcluster , Ġsince Ġthey Ġhave Ġthe Ġsame Ġwords , Ġsuch Ġas Ġfeeling , Ġin Ġtheir Ġdictionary Ġdefinitions . ĠHowever , Ġthey Ġare Ġof Ġopposite Ġpolar ities Ġand Ġshould Ġbe Ġdiscern ed Ġfrom Ġeach Ġother . </s> Therefore , Ġwe Ġutil ise Ġa Ġmetric Ġto Ġmove Ġsuch Ġwords Ġaway Ġfrom Ġeach Ġother Ġin Ġthe ĠV SM , Ġeven Ġthough Ġthey Ġhave Ġcommon Ġwords Ġin Ġtheir Ġdictionary Ġdefinitions . ĠWe Ġmultiply Ġeach Ġvalue Ġin Ġa Ġrow Ġwith Ġthe Ġcorresponding Ġrow Ġword 's Ġraw Ġsupervised Ġscore , Ġthereby Ġhaving Ġmore Ġmeaningful Ġclusters . ĠUsing Ġthe Ġtraining Ġdata Ġonly , Ġthe Ġsupervised Ġpol arity Ġscore Ġper Ġword Ġis Ġcalculated Ġas Ġin Ġ( DIS PLAY _ FORM 4 ). </s> Here , Ġ$ Ġw _{ t } $ Ġdenotes Ġthe Ġsentiment Ġscore Ġof Ġword Ġ$ t $, Ġ$ N _{ t } $ Ġis Ġthe Ġnumber Ġof Ġdocuments Ġ( review s Ġor Ġtweets ) Ġin Ġwhich Ġ$ t $ Ġoccurs Ġin Ġthe Ġdataset Ġof Ġpositive Ġpol arity , Ġ$ N $ Ġis Ġthe Ġnumber Ġof Ġall Ġthe Ġwords Ġin Ġthe Ġcorpus Ġof Ġpositive Ġpol arity . Ġ$ N ^ {\ prime Ġ} $ Ġdenotes Ġthe Ġcorpus Ġof Ġnegative Ġpol arity . Ġ$ N ^ {\ prime Ġ} _{ t } $ Ġand Ġ$ N ^ {\ prime Ġ} $ Ġdenote Ġsimilar Ġvalues Ġfor Ġthe Ġnegative Ġpol arity Ġcorpus . ĠWe Ġperform Ġnormal isation Ġto Ġprevent Ġthe Ġimbalance Ġproblem Ġand Ġadd Ġa Ġsmall Ġnumber Ġto Ġboth Ġnumer ator Ġand Ġdenomin ator Ġfor Ġsmoot hing . </s> As Ġan Ġalternative Ġto Ġmultiplying Ġwith Ġthe Ġsupervised Ġpol arity Ġscores , Ġwe Ġalso Ġseparately Ġmultiplied Ġall Ġthe Ġrow Ġscores Ġwith Ġonly Ġ+ 1 Ġif Ġthe Ġrow Ġword Ġis Ġa Ġpositive Ġword , Ġand Ġwith Ġ- 1 Ġif Ġit Ġis Ġa Ġnegative Ġword . ĠWe Ġhave Ġobserved Ġit Ġboosts Ġthe Ġperformance Ġmore Ġcompared Ġto Ġusing Ġraw Ġscores . </s> The Ġeffect Ġof Ġthis Ġmultiplication Ġis Ġexempl ified Ġin ĠFigure ĠFIG REF 7 , Ġshowing Ġthe Ġpositions Ġof Ġword Ġvectors Ġin Ġthe ĠV SM . ĠThose ĠâĢ ľ x " Ġwords Ġare Ġsentiment ally Ġnegative Ġwords , Ġthose ĠâĢ ľ o " Ġwords Ġare Ġsentiment ally Ġpositive Ġones . ĠOn Ġthe Ġtop Ġcoordinate Ġplane , Ġthe Ġwords Ġof Ġopposite Ġpolar ities Ġare Ġfound Ġto Ġbe Ġclose Ġto Ġeach Ġother , Ġsince Ġthey Ġhave Ġcommon Ġwords Ġin Ġtheir Ġdictionary Ġdefinitions . ĠOnly Ġthe Ġinformation Ġconcerned Ġwith Ġthe Ġdictionary Ġdefinitions Ġare Ġused Ġthere , Ġdisc arding Ġthe Ġpol arity Ġscores . ĠHowever , Ġwhen Ġwe Ġutil ise Ġthe Ġsupervised Ġscore Ġ(+ 1 Ġor Ġ- 1 ), Ġwords Ġof Ġopposite Ġpolar ities Ġ( e . g . ĠâĢ ľ happy " Ġand ĠâĢ ľ un happy ") Ġget Ġfar Ġaway Ġfrom Ġeach Ġother Ġas Ġthey Ġare Ġtranslated Ġacross Ġcoordinate Ġregions . ĠPositive Ġwords Ġnow Ġappear Ġin Ġquad rant Ġ1 , Ġwhereas Ġnegative Ġwords Ġappear Ġin Ġquad rant Ġ3 . ĠThus , Ġin Ġthe ĠV SM , Ġwords Ġthat Ġare Ġsentiment ally Ġsimilar Ġto Ġeach Ġother Ġcould Ġbe Ġclustered Ġmore Ġaccurately . ĠBesides Ġclust ering , Ġwe Ġalso Ġemployed Ġthe ĠS VD Ġmethod Ġto Ġperform Ġdimension ality Ġreduction Ġon Ġthe Ġun super vised Ġdictionary Ġalgorithm Ġand Ġused Ġthe Ġnewly Ġgenerated Ġmatrix Ġby Ġcombining Ġit Ġwith Ġother Ġsub appro aches . ĠThe Ġnumber Ġof Ġdimensions Ġis Ġchosen Ġas Ġ200 Ġagain Ġaccording Ġto Ġthe Ġ$ U $ Ġmatrix . ĠThe Ġdetails Ġare Ġgiven Ġin ĠSection Ġ3 . 4 . ĠWhen Ġusing Ġand Ġevaluating Ġthis Ġsub appro ach Ġon Ġthe ĠEnglish Ġcorpor a , Ġwe Ġused Ġthe ĠSent i Word Net Ġlex icon ĠB IB REF 13 . ĠWe Ġhave Ġachieved Ġbetter Ġresults Ġfor Ġthe Ġdictionary - based Ġalgorithm Ġwhen Ġwe Ġemployed Ġthe ĠS VD Ġreduction Ġmethod Ġcompared Ġto Ġthe Ġuse Ġof Ġclust ering . </s> Method ology Ġ: :: ĠSuper vised ĠContext ual Ġ4 - sc ores </s> Our Ġlast Ġcomponent Ġis Ġa Ġsimple Ġmetric Ġthat Ġuses Ġfour Ġsupervised Ġscores Ġfor Ġeach Ġword Ġin Ġthe Ġcorpus . ĠWe Ġextract Ġthese Ġscores Ġas Ġfollows . ĠFor Ġa Ġtarget Ġword Ġin Ġthe Ġcorpus , Ġwe Ġscan Ġthrough Ġall Ġof Ġits Ġcontexts . ĠIn Ġaddition Ġto Ġthe Ġtarget Ġword 's Ġpol arity Ġscore Ġ( the Ġself Ġscore ), Ġout Ġof Ġall Ġthe Ġpol arity Ġscores Ġof Ġwords Ġoccurring Ġin Ġthe Ġsame Ġcontexts Ġas Ġthe Ġtarget Ġword , Ġminimum , Ġmaximum , Ġand Ġaverage Ġscores Ġare Ġtaken Ġinto Ġconsideration . ĠThe Ġword Ġpol arity Ġscores Ġare Ġcomputed Ġusing Ġ( DIS PLAY _ FORM 4 ). ĠHere , Ġwe Ġobtain Ġthose Ġscores Ġfrom Ġthe Ġtraining Ġdata . </s> The Ġintuition Ġbehind Ġthis Ġmethod Ġis Ġthat Ġthose Ġfour Ġscores Ġare Ġmore Ġindicative Ġof Ġa Ġword 's Ġpol arity Ġrather Ġthan Ġonly Ġone Ġ( the Ġself Ġscore ). ĠThis Ġapproach Ġis Ġfully Ġsupervised Ġunlike Ġthe Ġprevious Ġtwo Ġapproaches . </s> Method ology Ġ: :: ĠComb ination Ġof Ġthe ĠWord ĠEmb edd ings </s> In Ġaddition Ġto Ġusing Ġthe Ġthree Ġapproaches Ġindependently , Ġwe Ġalso Ġcombined Ġall Ġthe Ġmat rices Ġgenerated Ġin Ġthe Ġprevious Ġapproaches . ĠThat Ġis , Ġwe Ġconc aten ate Ġthe Ġreduced Ġforms Ġ( S VD Ġ- ĠU ) Ġof Ġcorpus - based , Ġdictionary - based , Ġand Ġthe Ġwhole Ġof Ġ4 - score Ġvectors Ġof Ġeach Ġword , Ġhorizontally . ĠAccordingly , Ġeach Ġcorpus Ġword Ġis Ġrepresented Ġby Ġa Ġ404 - dimensional Ġvector , Ġsince Ġcorpus - based Ġand Ġdictionary - based Ġvector Ġcomponents Ġare Ġeach Ġcomposed Ġof Ġ200 Ġdimensions , Ġwhereas Ġthe Ġ4 - score Ġvector Ġcomponent Ġis Ġformed Ġby Ġfour Ġvalues . </s> The Ġmain Ġintuition Ġbehind Ġthe Ġensemble Ġmethod Ġis Ġthat Ġsome Ġapproaches Ġcompensate Ġfor Ġwhat Ġthe Ġothers Ġmay Ġlack . ĠFor Ġexample , Ġthe Ġcorpus - based Ġapproach Ġcaptures Ġthe Ġdomain - specific , Ġsemantic , Ġand Ġsynt actic Ġcharacteristics . ĠOn Ġthe Ġother Ġhand , Ġthe Ġ4 - sc ores Ġmethod Ġcaptures Ġsupervised Ġfeatures , Ġand Ġthe Ġdictionary - based Ġapproach Ġis Ġhelpful Ġin Ġcapturing Ġthe Ġgeneral Ġsemantic Ġcharacteristics . ĠThat Ġis , Ġcombining Ġthose Ġthree Ġapproaches Ġmakes Ġword Ġvectors Ġmore Ġrepresentative . </s> Method ology Ġ: :: ĠGener ating ĠDocument ĠV ect ors </s> After Ġcreating Ġseveral Ġembed d ings Ġas Ġmentioned Ġabove , Ġwe Ġcreate Ġdocument Ġ( review Ġor Ġtweet ) Ġvectors . ĠFor Ġeach Ġdocument , Ġwe Ġsum Ġall Ġthe Ġvectors Ġof Ġwords Ġoccurring Ġin Ġthat Ġdocument Ġand Ġtake Ġtheir Ġaverage . ĠIn Ġaddition Ġto Ġit , Ġwe Ġextract Ġthree Ġhand - crafted Ġpol arity Ġscores , Ġwhich Ġare Ġminimum , Ġmean , Ġand Ġmaximum Ġpol arity Ġscores , Ġfrom Ġeach Ġreview . ĠThese Ġpol arity Ġscores Ġof Ġwords Ġare Ġcomputed Ġas Ġin Ġ( DIS PLAY _ FORM 4 ). ĠFor Ġexample , Ġif Ġa Ġreview Ġconsists Ġof Ġfive Ġwords , Ġit Ġwould Ġhave Ġfive Ġpol arity Ġscores Ġand Ġwe Ġutil ise Ġonly Ġthree Ġof Ġthese Ġsentiment Ġscores Ġas Ġmentioned . ĠLastly , Ġwe Ġconc aten ate Ġthese Ġthree Ġscores Ġto Ġthe Ġaveraged Ġword Ġvector Ġper Ġreview . </s> That Ġis , Ġeach Ġreview Ġis Ġrepresented Ġby Ġthe Ġaverage Ġword Ġvector Ġof Ġits Ġconstituent Ġword Ġembed d ings Ġand Ġthree Ġsupervised Ġscores . ĠWe Ġthen Ġfeed Ġthese Ġinputs Ġinto Ġthe ĠS VM Ġapproach . ĠThe Ġflow chart Ġof Ġour Ġframework Ġis Ġgiven Ġin ĠFigure ĠFIG REF 11 . ĠWhen Ġcombining Ġthe Ġun super vised Ġfeatures , Ġwhich Ġare Ġword Ġvectors Ġcreated Ġon Ġa Ġword Ġbasis , Ġwith Ġsupervised Ġthree Ġscores Ġextracted Ġon Ġa Ġreview Ġbasis , Ġwe Ġhave Ġbetter Ġstate - of - the - art Ġresults . </s> Dat as ets </s> We Ġutil ised Ġtwo Ġdatasets Ġfor Ġboth ĠTurkish Ġand ĠEnglish Ġto Ġevaluate Ġour Ġmethods . </s> For ĠTurkish , Ġas Ġthe Ġfirst Ġdataset , Ġwe Ġutil ised Ġthe Ġmovie Ġreviews Ġwhich Ġare Ġcollected Ġfrom Ġa Ġpopular Ġwebsite . ĠThe Ġnumber Ġof Ġreviews Ġin Ġthis Ġmovie Ġcorpus Ġis Ġ20 , 244 Ġand Ġthe Ġaverage Ġnumber Ġof Ġwords Ġin Ġreviews Ġis Ġ39 . ĠEach Ġof Ġthese Ġreviews Ġhas Ġa Ġstar - rating Ġscore Ġwhich Ġis Ġindicative Ġof Ġsentiment . ĠThese Ġpol arity Ġscores Ġare Ġbetween Ġthe Ġvalues Ġ0 . 5 Ġand Ġ5 , Ġat Ġintervals Ġof Ġ0 . 5 . ĠWe Ġconsider Ġa Ġreview Ġto Ġbe Ġnegative Ġit Ġthe Ġscore Ġis Ġequal Ġto Ġor Ġlower Ġthan Ġ2 . 5 . ĠOn Ġthe Ġother Ġhand , Ġif Ġit Ġis Ġequal Ġto Ġor Ġhigher Ġthan Ġ4 , Ġit Ġis Ġassumed Ġto Ġbe Ġpositive . ĠWe Ġhave Ġrandomly Ġselected Ġ7 , 020 Ġnegative Ġand Ġ7 , 020 Ġpositive Ġreviews Ġand Ġprocessed Ġonly Ġthem . </s> The Ġsecond ĠTurkish Ġdataset Ġis Ġthe ĠTwitter Ġcorpus Ġwhich Ġis Ġformed Ġof Ġtweets Ġabout ĠTurkish Ġmobile Ġnetwork Ġoperators . ĠThose Ġtweets Ġare Ġmostly Ġmuch Ġno is ier Ġand Ġshorter Ġcompared Ġto Ġthe Ġreviews Ġin Ġthe Ġmovie Ġcorpus . ĠIn Ġtotal , Ġthere Ġare Ġ1 , 7 16 Ġtweets . Ġ9 73 Ġof Ġthem Ġare Ġnegative Ġand Ġ7 43 Ġof Ġthem Ġare Ġpositive . ĠThese Ġtweets Ġare Ġmanually Ġannot ated Ġby Ġtwo Ġhumans , Ġwhere Ġthe Ġlabels Ġare Ġeither Ġpositive Ġor Ġnegative . ĠWe Ġmeasured Ġthe ĠCohen 's ĠKappa Ġinter - annot ator Ġagreement Ġscore Ġto Ġbe Ġ0 . 82 . ĠIf Ġthere Ġwas Ġa Ġdisagreement Ġon Ġthe Ġpol arity Ġof Ġa Ġtweet , Ġwe Ġremoved Ġit . </s> We Ġalso Ġutil ised Ġtwo Ġother Ġdatasets Ġin ĠEnglish Ġto Ġtest Ġthe Ġcross - ling uality Ġof Ġour Ġapproaches . ĠOne Ġof Ġthem Ġis Ġa Ġmovie Ġcorpus Ġcollected Ġfrom Ġthe Ġweb . ĠThere Ġare Ġ5 , 331 Ġpositive Ġreviews Ġand Ġ5 , 331 Ġnegative Ġreviews Ġin Ġthis Ġcorpus . ĠThe Ġother Ġis Ġa ĠTwitter Ġdataset , Ġwhich Ġhas Ġnearly Ġ1 . 6 Ġmillion Ġtweets Ġannot ated Ġthrough Ġa Ġdistant Ġsupervised Ġmethod ĠB IB REF 14 . ĠThese Ġtweets Ġhave Ġpositive , Ġneutral , Ġand Ġnegative Ġlabels . ĠWe Ġhave Ġselected Ġ7 , 020 Ġpositive Ġtweets Ġand Ġ7 , 020 Ġnegative Ġtweets Ġrandomly Ġto Ġgenerate Ġa Ġbalanced Ġdataset . </s> Exper iments Ġ: :: ĠPre processing </s> In ĠTurkish , Ġpeople Ġsometimes Ġprefer Ġto Ġspell ĠEnglish Ġcharacters Ġfor Ġthe Ġcorresponding ĠTurkish Ġcharacters Ġ( e . g . Ġi Ġfor Ġ Ä± , Ġc Ġfor ĠÃ § ) Ġwhen Ġwriting Ġin Ġelectronic Ġformat . ĠTo Ġnormal ise Ġsuch Ġwords , Ġwe Ġused Ġthe ĠZ emb erek Ġtool ĠB IB REF 15 . ĠAll Ġpunct uation Ġmarks Ġexcept ĠâĢ ľ !" Ġand ĠâĢ ľ ?" Ġare Ġremoved , Ġsince Ġthey Ġdo Ġnot Ġcontribute Ġmuch Ġto Ġthe Ġpol arity Ġof Ġa Ġdocument . ĠWe Ġtook Ġinto Ġaccount Ġemot icons , Ġsuch Ġas ĠâĢ ľ : )) ", Ġand Ġidi oms , Ġsuch Ġas ĠâĢ ľ k af ay Ä± Ġy em ek âĢ Ŀ Ġ( l ose Ġone 's Ġmind ), Ġsince Ġtwo Ġor Ġmore Ġwords Ġcan Ġexpress Ġa Ġsentiment Ġtogether , Ġirrespective Ġof Ġthe Ġindividual Ġwords Ġthereof . ĠSince ĠTurkish Ġis Ġan Ġagg l ut in ative Ġlanguage , Ġwe Ġused Ġthe Ġmorph ological Ġparser Ġand Ġdis amb ig u ator Ġtools ĠB IB REF 16 , ĠB IB REF 17 . ĠWe Ġalso Ġperformed Ġneg ation Ġhandling Ġand Ġstop - word Ġelimination . ĠIn Ġneg ation Ġhandling , Ġwe Ġappend Ġan Ġunderscore Ġto Ġthe Ġend Ġof Ġa Ġword Ġif Ġit Ġis Ġneg ated . ĠFor Ġexample , ĠâĢ ľ g Ã¼ zel Ġde ÄŁ il " Ġ( not Ġbeautiful ) Ġis Ġred efined Ġas ĠâĢ ľ g Ã¼ zel _ " Ġ( beaut iful _ ) Ġin Ġthe Ġfeature Ġselection Ġstage Ġwhen Ġsupervised Ġscores Ġare Ġbeing Ġcomputed . </s> Exper iments Ġ: :: ĠHyper param eters </s> We Ġused Ġthe ĠLib S VM Ġutility Ġof Ġthe ĠWE KA Ġtool . ĠWe Ġchose Ġthe Ġlinear Ġkernel Ġoption Ġto Ġclassify Ġthe Ġreviews . ĠWe Ġtrained Ġword 2 vec Ġembed d ings Ġon Ġall Ġthe Ġfour Ġcorpor a Ġusing Ġthe ĠG ens im Ġlibrary ĠB IB REF 18 Ġwith Ġthe Ġskip - gram Ġmethod . ĠThe Ġdimension Ġsize Ġof Ġthese Ġembed d ings Ġare Ġset Ġat Ġ200 . ĠAs Ġmentioned , Ġother Ġembed d ings , Ġwhich Ġare Ġgenerated Ġutil ising Ġthe Ġclust ering Ġand Ġthe ĠS VD Ġapproach , Ġare Ġalso Ġof Ġsize Ġ200 . ĠFor Ġc - mean Ġclust ering , Ġwe Ġset Ġthe Ġmaximum Ġnumber Ġof Ġiterations Ġat Ġ25 , Ġunless Ġit Ġconver ges . </s> Exper iments Ġ: :: ĠResults </s> We Ġevaluated Ġour Ġmodels Ġon Ġfour Ġcorpor a , Ġwhich Ġare Ġthe Ġmovie Ġand Ġthe ĠTwitter Ġdatasets Ġin ĠTurkish Ġand ĠEnglish . ĠAll Ġof Ġthe Ġembed d ings Ġare Ġlearnt Ġon Ġfour Ġcorpor a Ġseparately . ĠWe Ġhave Ġused Ġthe Ġaccuracy Ġmetric Ġsince Ġall Ġthe Ġdatasets Ġare Ġcompletely Ġor Ġnearly Ġcompletely Ġbalanced . ĠWe Ġperformed Ġ10 - fold Ġcross - valid ation Ġfor Ġboth Ġof Ġthe Ġdatasets . ĠWe Ġused Ġthe Ġapproximate Ġrandom isation Ġtechnique Ġto Ġtest Ġwhether Ġour Ġresults Ġare Ġstatistically Ġsignificant . ĠHere , Ġwe Ġtried Ġto Ġpredict Ġthe Ġlabels Ġof Ġreviews Ġand Ġassess Ġthe Ġperformance . </s> We Ġobtained Ġvarying Ġaccur acies Ġas Ġshown Ġin ĠTable ĠT AB REF 17 . ĠâĢ ľ 3 Ġfeats " Ġfeatures Ġare Ġthose Ġhand - crafted Ġfeatures Ġwe Ġextracted , Ġwhich Ġare Ġthe Ġminimum , Ġmean , Ġand Ġmaximum Ġpol arity Ġscores Ġof Ġthe Ġreviews Ġas Ġexplained Ġin ĠSection Ġ3 . 5 . ĠAs Ġcan Ġbe Ġseen , Ġat Ġleast Ġone Ġof Ġour Ġmethods Ġoutper forms Ġthe Ġbaseline Ġword 2 vec Ġapproach Ġfor Ġall Ġthe ĠTurkish Ġand ĠEnglish Ġcorpor a , Ġand Ġall Ġcategories . ĠAll Ġof Ġour Ġapproaches Ġperformed Ġbetter Ġwhen Ġwe Ġused Ġthe Ġsupervised Ġscores , Ġwhich Ġare Ġextracted Ġon Ġa Ġreview Ġbasis , Ġand Ġconc aten ated Ġthem Ġto Ġword Ġvectors . ĠMostly , Ġthe Ġsupervised Ġ4 - sc ores Ġfeature Ġleads Ġto Ġthe Ġhighest Ġaccur acies , Ġsince Ġit Ġemploys Ġthe Ġannot ational Ġinformation Ġconcerned Ġwith Ġpolar ities Ġon Ġa Ġword Ġbasis . </s> As Ġcan Ġbe Ġseen Ġin ĠTable ĠT AB REF 17 , Ġthe Ġclust ering Ġmethod , Ġin Ġgeneral , Ġyields Ġthe Ġlowest Ġscores . ĠWe Ġfound Ġout Ġthat Ġthe Ġcorpus Ġ- ĠS VD Ġmetric Ġdoes Ġalways Ġperform Ġbetter Ġthan Ġthe Ġclust ering Ġmethod . ĠWe Ġattribute Ġit Ġto Ġthat Ġin ĠS VD Ġthe Ġmost Ġimportant Ġsingular Ġvalues Ġare Ġtaken Ġinto Ġaccount . ĠThe Ġcorpus Ġ- ĠS VD Ġtechnique Ġoutper forms Ġthe Ġword 2 vec Ġalgorithm Ġfor Ġsome Ġcorpor a . ĠWhen Ġwe Ġdo Ġnot Ġtake Ġinto Ġaccount Ġthe Ġ3 - fe ats Ġtechnique , Ġthe Ġcorpus - based ĠS VD Ġmethod Ġyields Ġthe Ġhighest Ġaccur acies Ġfor Ġthe ĠEnglish ĠTwitter Ġdataset . ĠWe Ġshow Ġthat Ġsimple Ġmodels Ġcan Ġoutper form Ġmore Ġcomplex Ġmodels , Ġsuch Ġas Ġthe Ġconc aten ation Ġof Ġthe Ġthree Ġsub appro aches Ġor Ġthe Ġword 2 vec Ġalgorithm . ĠAnother Ġinteresting Ġfinding Ġis Ġthat Ġfor Ġsome Ġcases Ġthe Ġaccuracy Ġdecreases Ġwhen Ġwe Ġutil ise Ġthe Ġpol arity Ġlabels , Ġas Ġin Ġthe Ġcase Ġfor Ġthe ĠEnglish ĠTwitter Ġdataset . </s> Since Ġthe ĠTD K Ġdictionary Ġcovers Ġmost Ġof Ġthe Ġdomain - specific Ġvocabulary Ġused Ġin Ġthe Ġmovie Ġreviews , Ġthe Ġdictionary Ġmethod Ġperforms Ġwell . ĠHowever , Ġthe Ġdictionary Ġlacks Ġmany Ġof Ġthe Ġwords , Ġoccurring Ġin Ġthe Ġtweets ; Ġtherefore , Ġits Ġperformance Ġis Ġnot Ġthe Ġbest Ġof Ġall . ĠWhen Ġthe ĠTD K Ġmethod Ġis Ġcombined Ġwith Ġthe Ġ3 - fe ats Ġtechnique , Ġwe Ġobserved Ġa Ġgreat Ġimprovement , Ġas Ġcan Ġbe Ġexpected . </s> Success Ġrates Ġobtained Ġfor Ġthe Ġmovie Ġcorpus Ġare Ġmuch Ġbetter Ġthan Ġthose Ġfor Ġthe ĠTwitter Ġdataset Ġfor Ġmost Ġof Ġour Ġapproaches , Ġsince Ġtweets Ġare , Ġin Ġgeneral , Ġmuch Ġshorter Ġand Ġno is ier . ĠWe Ġalso Ġfound Ġout Ġthat , Ġwhen Ġchoosing Ġthe Ġp Ġvalue Ġas Ġ0 . 05 , Ġour Ġresults Ġare Ġstatistically Ġsignificant Ġcompared Ġto Ġthe Ġbaseline Ġapproach Ġin ĠTurkish ĠB IB REF 10 . ĠSome Ġof Ġour Ġsub appro aches Ġalso Ġproduce Ġbetter Ġsuccess Ġrates Ġthan Ġthose Ġsentiment Ġanalysis Ġmodels Ġemployed Ġin ĠEnglish ĠB IB REF 11 , ĠB IB REF 12 . ĠWe Ġhave Ġachieved Ġstate - of - the - art Ġresults Ġfor Ġthe Ġsentiment Ġclassification Ġtask Ġfor Ġboth ĠTurkish Ġand ĠEnglish . ĠAs Ġmentioned , Ġour Ġapproaches , Ġin Ġgeneral , Ġperform Ġbest Ġin Ġpredicting Ġthe Ġlabels Ġof Ġreviews Ġwhen Ġthree Ġsupervised Ġscores Ġare Ġaddition ality Ġutil ised . </s> We Ġalso Ġemployed Ġthe Ġconv olution al Ġneural Ġnetwork Ġmodel Ġ( CNN ). ĠHowever , Ġthe ĠS VM Ġclass ifier , Ġwhich Ġis Ġa Ġconventional Ġmachine Ġlearning Ġalgorithm , Ġperformed Ġbetter . ĠWe Ġdid Ġnot Ġinclude Ġthe Ġperformances Ġof ĠCNN Ġfor Ġembed ding Ġtypes Ġhere Ġdue Ġto Ġthe Ġpage Ġlimit Ġof Ġthe Ġpaper . </s> As Ġa Ġqualitative Ġassessment Ġof Ġthe Ġword Ġrepresentations , Ġgiven Ġsome Ġquery Ġwords Ġwe Ġvisual ised Ġthe Ġmost Ġsimilar Ġwords Ġto Ġthose Ġwords Ġusing Ġthe Ġcos ine Ġsimilarity Ġmetric . ĠBy Ġassessing Ġthe Ġsimilarities Ġbetween Ġa Ġword Ġand Ġall Ġthe Ġother Ġcorpus Ġwords , Ġwe Ġcan Ġfind Ġthe Ġmost Ġakin Ġwords Ġaccording Ġto Ġdifferent Ġapproaches . ĠTable ĠT AB REF 18 Ġshows Ġthe Ġmost Ġsimilar Ġwords Ġto Ġgiven Ġquery Ġwords . ĠThose Ġwords Ġwhich Ġare Ġindicative Ġof Ġsentiment Ġare , Ġin Ġgeneral , Ġfound Ġto Ġbe Ġmost Ġsimilar Ġto Ġthose Ġwords Ġof Ġthe Ġsame Ġpol arity . ĠFor Ġexample , Ġthe Ġmost Ġakin Ġword Ġto Ġm uh te ÅŁ em Ġ( g orge ous ) Ġis Ġ10 / 10 , Ġboth Ġof Ġwhich Ġhave Ġpositive Ġpol arity . ĠAs Ġcan Ġbe Ġseen Ġin ĠTable ĠT AB REF 18 , Ġour Ġcorpus - based Ġapproach Ġis Ġmore Ġadept Ġat Ġcapturing Ġdomain - specific Ġfeatures Ġas Ġcompared Ġto Ġword 2 vec , Ġwhich Ġgenerally Ġcaptures Ġgeneral Ġsemantic Ġand Ġsynt actic Ġcharacteristics , Ġbut Ġnot Ġthe Ġsentimental Ġones . </s> Conclusion </s> We Ġhave Ġdemonstrated Ġthat Ġusing Ġword Ġvectors Ġthat Ġcapture Ġonly Ġsemantic Ġand Ġsynt actic Ġcharacteristics Ġmay Ġbe Ġimproved Ġby Ġtaking Ġinto Ġaccount Ġtheir Ġsentimental Ġaspects Ġas Ġwell . ĠOur Ġapproaches Ġare Ġcross - ling ual Ġand Ġcross - domain . ĠThey Ġcan Ġbe Ġapplied Ġto Ġother Ġdomains Ġand Ġother Ġlanguages Ġthan ĠTurkish Ġand ĠEnglish Ġwith Ġminor Ġchanges . </s> Our Ġstudy Ġis Ġone Ġof Ġthe Ġfew Ġones Ġthat Ġperform Ġsentiment Ġanalysis Ġin ĠTurkish Ġand Ġlever ages Ġsentimental Ġcharacteristics Ġof Ġwords Ġin Ġgenerating Ġword Ġvectors Ġand Ġoutper forms Ġall Ġthe Ġothers . ĠAny Ġof Ġthe Ġapproaches Ġwe Ġpropose Ġcan Ġbe Ġused Ġindependently Ġof Ġthe Ġothers . ĠOur Ġapproaches Ġwithout Ġusing Ġsentiment Ġlabels Ġcan Ġbe Ġapplied Ġto Ġother Ġclassification Ġtasks , Ġsuch Ġas Ġtopic Ġclassification Ġand Ġconcept Ġmining . </s> The Ġexperiments Ġshow Ġthat Ġeven Ġun super vised Ġapproaches , Ġas Ġin Ġthe Ġcorpus - based Ġapproach , Ġcan Ġoutper form Ġsupervised Ġapproaches Ġin Ġclassification Ġtasks . ĠComb ining Ġsome Ġapproaches , Ġwhich Ġcan Ġcompensate Ġfor Ġwhat Ġothers Ġlack , Ġcan Ġhelp Ġus Ġbuild Ġbetter Ġvectors . ĠOur Ġword Ġvectors Ġare Ġcreated Ġby Ġconventional Ġmachine Ġlearning Ġalgorithms ; Ġhowever , Ġthey , Ġas Ġin Ġthe Ġcorpus - based Ġmodel , Ġproduce Ġstate - of - the - art Ġresults . ĠAlthough Ġwe Ġpreferred Ġto Ġuse Ġa Ġclassical Ġmachine Ġlearning Ġalgorithm , Ġwhich Ġis ĠS VM , Ġover Ġa Ġneural Ġnetwork Ġclass ifier Ġto Ġpredict Ġthe Ġlabels Ġof Ġreviews , Ġwe Ġachieved Ġaccur acies Ġof Ġover Ġ90 Ġper Ġcent Ġfor Ġthe ĠTurkish Ġmovie Ġcorpus Ġand Ġabout Ġ88 Ġper Ġcent Ġfor Ġthe ĠEnglish ĠTwitter Ġdataset . </s> We Ġperformed Ġonly Ġbinary Ġsentiment Ġclassification Ġin Ġthis Ġstudy Ġas Ġmost Ġof Ġthe Ġstudies Ġin Ġthe Ġliterature Ġdo . ĠWe Ġwill Ġextend Ġour Ġsystem Ġin Ġfuture Ġby Ġusing Ġneutral Ġreviews Ġas Ġwell . ĠWe Ġalso Ġplan Ġto Ġemploy ĠTurkish ĠWord Net Ġto Ġenhance Ġthe Ġgeneral is ability Ġof Ġour Ġembed d ings Ġas Ġanother Ġfuture Ġwork . </s> Acknowled gments </s> This Ġwork Ġwas Ġsupported Ġby ĠBo ÄŁ azi Ã§ i ĠUniversity ĠResearch ĠFund ĠGrant ĠNumber Ġ69 80 D , Ġand Ġby ĠTurkish ĠMinistry Ġof ĠDevelopment Ġunder Ġthe ĠTAM ĠProject Ġnumber ĠD PT 2007 K 12 - 06 10 . ĠC em ĠR Ä± f k Ä± ĠA yd Ä± n Ġis Ġsupported Ġby ĠT Ã ľ B Ä ° TA K ĠB IDE B Ġ22 11 E .