Accuracy
ABSTRACTIVE
ratio of correct `translations'
EXTRACTIVE
To test all the possible combinations of parameters, we divided the bilingual dictionary into 4500 noun pairs used as a training set and 500 noun pairs used as a test set. We then learned transformation matrices on the training set using both training algorithms (CBOW and SkipGram) and several values of regularization $\lambda $ from 0 to 5, with a step of 0.5. The resulting matrices were applied to the Ukrainian vectors from the test set and the corresponding Russian `translations' were calculated. The ratio of correct `translations' (matches) was used as an evaluation measure. It came out that regularization only worsened the results for both algorithms, so in the Table 1 we report the results without regularization.
To test all the possible combinations of parameters, we divided the bilingual dictionary into 4500 noun pairs used as a training set and 500 noun pairs used as a test set. We then learned transformation matrices on the training set using both training algorithms (CBOW and SkipGram) and several values of regularization $\lambda $ from 0 to 5, with a step of 0.5. The resulting matrices were applied to the Ukrainian vectors from the test set and the corresponding Russian `translations' were calculated. The ratio of correct `translations' (matches) was used as an evaluation measure. It came out that regularization only worsened the results for both algorithms, so in the Table 1 we report the results without regularization.
1604.05372
Introduction </s> This Ġresearch Ġaddresses Ġthe Ġproblem Ġof Ġrepresenting Ġthe Ġsemantics Ġof Ġtext Ġdocuments Ġin Ġmulti - ling ual Ġcomparable Ġcorpor a . ĠWe Ġpresent Ġa Ġnew Ġapproach Ġto Ġthis Ġproblem , Ġbased Ġon Ġneural Ġembed d ings , Ġand Ġtest Ġit Ġon Ġthe Ġtask Ġof Ġclust ering Ġtexts Ġinto Ġmeaningful Ġclasses Ġdepending Ġon Ġtheir Ġtopics . ĠThe Ġsetting Ġis Ġun super vised , Ġmeaning Ġthat Ġone Ġeither Ġdoes Ġnot Ġhave Ġenough Ġannot ated Ġdata Ġto Ġtrain Ġa Ġsupervised Ġclass ifier Ġor Ġdoes Ġnot Ġwant Ġto Ġbe Ġlimited Ġwith Ġa Ġpre - defined Ġset Ġof Ġclasses . ĠThere Ġis Ġa Ġlot Ġof Ġsufficiently Ġgood Ġapproaches Ġto Ġthis Ġproblem Ġin Ġthe Ġcase Ġof Ġmono - ling ual Ġtext Ġcollections , Ġbut Ġthe Ġpresence Ġof Ġmultiple Ġlanguages Ġintroduces Ġcomplications . </s> When Ġa Ġtext Ġcollection Ġcontains Ġdocuments Ġin Ġseveral Ġlanguages , Ġit Ġbecomes Ġimpractical Ġto Ġsimply Ġrepresent Ġthe Ġdocuments Ġas Ġvectors Ġof Ġwords Ġoccurring Ġin Ġthem Ġ(" bag - of - words "), Ġas Ġthe Ġwords Ġsurface Ġforms Ġare Ġdifferent , Ġeven Ġin Ġclosely - related Ġlanguages . ĠThus , Ġone Ġhas Ġto Ġinvent Ġmeans Ġto Ġcross Ġthe Ġinter - ling ual Ġgap Ġand Ġbring Ġall Ġdocuments Ġto Ġsome Ġsort Ġof Ġshared Ġrepresentation , Ġwithout Ġlosing Ġinformation Ġabout Ġtheir Ġtopics Ġor Ġcategories . </s> Of Ġcourse , Ġone Ġobvious Ġway Ġto Ġsolve Ġthis Ġproblem Ġis Ġto Ġtranslate Ġall Ġdocuments Ġinto Ġone Ġlanguage , Ġand Ġthen Ġapply Ġany Ġclust ering Ġalgorithm . ĠHowever , Ġthis Ġrequires Ġeither Ġbuying Ġhuman / machine Ġtranslation Ġservices Ġ( which Ġcan Ġbe Ġexpensive Ġif Ġyou Ġdeal Ġwith Ġlarge Ġtext Ġcollection ) Ġor Ġtraining Ġown Ġstatistical Ġmachine Ġtranslation Ġmodel Ġ( which Ġas Ġa Ġrule Ġrequires Ġbig Ġparallel Ġcorpus ). ĠThis Ġis Ġthe Ġreason Ġto Ġsearch Ġfor Ġother Ġsolutions . </s> In Ġthis Ġpaper , Ġa Ġnovel Ġway Ġof Ġreducing Ġthe Ġproblem Ġof Ġcross - ling ual Ġdocument Ġrepresentation Ġto Ġa Ġmon oling ual Ġsetting Ġis Ġproposed . ĠEssentially , Ġwe Ġtrain ĠContinuous ĠBag - of - Words Ġmodels ĠB IB REF 0 Ġon Ġlarge Ġcomparable Ġmon oling ual Ġcorpor a Ġfor Ġtwo Ġlanguages Ġour Ġdataset Ġconsists Ġof . ĠThis Ġprovides Ġus Ġwith Ġvector Ġrepresentations Ġof Ġwords , Ġallowing Ġto Ġmeasure Ġtheir Ġsemantic Ġsimilarity . ĠThen , Ġa Ġlinear Ġtransformation Ġmatrix Ġfrom Ġvectors Ġof Ġlanguage ĠA Ġto Ġvectors Ġof Ġlanguage ĠB Ġis Ġlearned , Ġusing Ġa Ġsmall Ġbilingual Ġdictionary Ġas Ġtraining Ġdata . ĠThis Ġmatrix Ġis Ġthen Ġemployed Ġto Ġ` project ' Ġword Ġand Ġdocument Ġrepresentations Ġfrom Ġsemantic Ġspace Ġof Ġlanguage ĠA Ġto Ġsemantic Ġspace Ġof Ġlanguage ĠB . ĠIt Ġallows Ġnot Ġonly Ġquite Ġaccurate Ġ` translation ' Ġof Ġwords , Ġbut Ġalso Ġof Ġdocument Ġ` sem antic Ġfingerprints ' Ġ( d ense Ġrepresentations Ġof Ġdocument Ġsemantics , Ġcalculated Ġas Ġan Ġaverage Ġof Ġthe Ġtrained Ġdistribution al Ġvectors Ġfor Ġall Ġthe Ġwords Ġin Ġdocument ). </s> This Ġapproach Ġis Ġevaluated Ġin Ġa Ġsetting , Ġwhere Ġthe Ġinput Ġis Ġa Ġcollection Ġof Ġdocuments Ġin Ġseveral Ġlanguages Ġand Ġsome Ġnumber Ġof Ġtopics Ġto Ġwhich Ġthese Ġdocuments Ġbelong Ġ( we Ġalso Ġhave Ġlarge Ġmon oling ual Ġcorpor a Ġto Ġtrain Ġdistribution al Ġmodels Ġon ). ĠFor Ġeach Ġdocument , Ġwe Ġare Ġgiven Ġits Ġlanguage , Ġbut Ġnot Ġits Ġtopic . ĠThe Ġtask Ġis Ġto Ġcluster Ġthis Ġcollection Ġso Ġthat Ġdocuments Ġbelonging Ġto Ġone Ġtopic Ġwere Ġclustered Ġtogether , Ġindependent Ġof Ġtheir Ġlanguage . ĠNote Ġthat Ġwe Ġare Ġinterested Ġin Ġclust ering Ġthe Ġcollection Ġas Ġa Ġwhole , Ġnot Ġeach Ġlanguage Ġseparately Ġ( which Ġis Ġtrivial ). </s> Our Ġevaluation Ġdata Ġconsists Ġof Ġcomparable Ġcorpor a Ġof ĠRussian Ġand ĠUkrainian Ġacademic Ġtexts . ĠOn Ġthis Ġmaterial , Ġwe Ġshow Ġthat Ġthe Ġ` trans lated Ġsemantic Ġfingerprints ' Ġmethod Ġrepresents Ġdocuments Ġin Ġdifferent Ġlanguages Ġprecisely Ġenough Ġto Ġallow Ġalmost Ġexact Ġclust ering Ġaccording Ġto Ġdocument Ġtopics , Ġwith Ġonly Ġ5 % Ġof Ġincorrect Ġassignments . ĠIt Ġsignificantly Ġoutper forms Ġboth Ġnaive Ġbag - of - words Ġbaseline Ġand Ġthe Ġnot - so - na ive Ġmethod Ġof Ġ` orth ographic Ġtranslation ' Ġbased Ġon ĠD amer au - Le ven sh tein Ġdistance , Ġeven Ġenriched Ġwith Ġdictionary Ġm appings . ĠAt Ġthe Ġsame Ġtime , Ġit Ġdoes Ġnot Ġrequire Ġlarge Ġparallel Ġcorpor a Ġor Ġa Ġready - made Ġstatistical Ġmachine Ġtranslation Ġmodel . </s> The Ġrest Ġof Ġthe Ġpaper Ġis Ġstructured Ġas Ġfollows . ĠIn ĠSection Ġ" Related ĠWork " Ġwe Ġdescribe Ġthe Ġfoundations Ġof Ġour Ġapproach Ġand Ġthe Ġrelated Ġwork . ĠSection Ġ" Ac ademic Ġtexts Ġas ĠCompar able ĠCorpor a " Ġintroduces Ġthe Ġemployed Ġcorpor a Ġand Ġthe Ġstory Ġbehind Ġthem . ĠSection Ġ" Learning Ġto ĠTrans late : ĠUkrainian - to - Russian Ġtransformations " Ġis Ġdedicated Ġto Ġlearning Ġthe Ġtransformation Ġmatrix , Ġand ĠSection Ġ" Exper iment ĠDesign Ġand ĠEvaluation " Ġdescribes Ġour Ġexperimental Ġsetting Ġand Ġevaluation Ġresults . ĠWe Ġdiscuss Ġthe Ġfindings Ġin ĠSection Ġ" Discussion " Ġand Ġconclude Ġin ĠSection Ġ" Conclusion Ġand ĠFuture ĠWork " Ġ, Ġalso Ġsuggesting Ġdirections Ġfor Ġfuture Ġwork . </s> Related ĠWork </s> Cl ust ering Ġmulti - ling ual Ġdocuments Ġhas Ġreceived Ġmuch Ġattention Ġin Ġnatural Ġlanguage Ġprocessing . ĠAmong Ġapproaches Ġnot Ġusing Ġsome Ġform Ġof Ġmachine Ġtranslation , Ġone Ġcan Ġmention ĠB IB REF 1 Ġ, Ġwho Ġessentially Ġemploy Ġa Ġbilingual Ġdictionary Ġto Ġbring Ġsome Ġwords Ġin Ġthe Ġdocuments Ġto Ġa Ġlanguage - independent Ġform Ġand Ġthen Ġto Ġperform Ġclust ering . ĠIn Ġthe Ġsection Ġ" Exper iment ĠDesign Ġand ĠEvaluation " Ġwe Ġshow Ġthat Ġour Ġapproach Ġbased Ġon Ġneural Ġembed d ings Ġsignificantly Ġoutper forms Ġtheir Ġreported Ġresults . </s> BI B REF 2 Ġproposed Ġtraining Ġjoint Ġmulti - ling ual Ġneural Ġembed ding Ġmodels . ĠThe oret ically , Ġthis Ġcan Ġbe Ġused Ġto Ġachieve Ġour Ġaim Ġof Ġlanguage - independent Ġsemantic Ġrepresentations Ġfor Ġdocuments . ĠUnfortunately , Ġit Ġdemands Ġa Ġlarge Ġword - aligned Ġparallel Ġcorpus . ĠThis Ġis Ġnot Ġthe Ġcase Ġwith Ġthe Ġmore Ġrecent ĠTrans - gram Ġapproach Ġintroduced Ġin ĠB IB REF 3 Ġ, Ġalso Ġable Ġto Ġlearn Ġmulti - ling ual Ġmodels . ĠHowever , Ġit Ġstill Ġneeds Ġsentence - aligned Ġcorpor a Ġto Ġtrain Ġon Ġ( in Ġthe Ġsize Ġof Ġmillions Ġof Ġpaired Ġsentences ). ĠLarge Ġparallel Ġcorpor a Ġ( whether Ġword - Ġor Ġsentence - aligned ) Ġare Ġoften Ġa Ġscarce Ġresource , Ġespecially Ġin Ġthe Ġcase Ġof Ġunder - represented Ġlanguages . </s> The Ġapproach Ġdescribed Ġin Ġthis Ġpaper Ġtakes Ġas Ġan Ġinput Ġonly Ġcomparable Ġmon oling ual Ġcorpor a Ġand Ġbilingual Ġdiction aries Ġin Ġthe Ġsize Ġof Ġseveral Ġthousand Ġword Ġpairs . ĠSuch Ġresources Ġare Ġmuch Ġeasier Ġto Ġfind Ġand Ġevaluate . ĠWe Ġemploy Ġthe Ġidea Ġof Ġlearning Ġa Ġlinear Ġtransformation Ġmatrix Ġto Ġmap Ġor Ġproject Ġword Ġembed d ings Ġfrom Ġthe Ġsemantic Ġspace Ġof Ġone Ġlanguage Ġto Ġthat Ġof Ġanother . ĠThis Ġidea Ġwas Ġfirst Ġproposed Ġin ĠB IB REF 4 Ġ, Ġwho Ġapplied Ġit Ġto Ġlex ical Ġtranslation Ġbetween ĠEnglish , ĠSpanish , ĠCzech Ġand ĠVietnamese . ĠWe Ġextend Ġit Ġfrom Ġcontinuous Ġrepresentations Ġof Ġsingle Ġwords Ġor Ġcoll ocations Ġto Ġ` sem antic Ġfingerprints ' Ġof Ġdocuments Ġas Ġa Ġwhole . </s> Ac ademic Ġtexts Ġas ĠCompar able ĠCorpor a </s> The ĠRussian Ġand ĠUkrainian Ġlanguages Ġare Ġmainly Ġspoken Ġin ĠRussian ĠFederation Ġand Ġthe ĠUkraine Ġand Ġbelong Ġto Ġthe ĠEast - Sl av ic Ġgroup Ġof Ġthe ĠIndo - European Ġlanguage Ġfamily . ĠThey Ġshare Ġmany Ġcommon Ġmorph os ynt actic Ġfeatures : Ġboth Ġare ĠSV O Ġlanguages Ġwith Ġfree Ġword Ġorder Ġand Ġrich Ġmorphology , Ġboth Ġuse Ġthe ĠCyr ill ic Ġalphabet Ġand Ġshare Ġmany Ġcommon Ġcogn ates . </s> Both ĠRussia Ġand Ġthe ĠUkraine Ġhave Ġcommon Ġacademic Ġtradition Ġthat Ġmakes Ġit Ġeasier Ġto Ġcollect Ġcorpor a , Ġwhich Ġare Ġcomparable Ġin Ġterms Ġof Ġboth Ġgenre Ġand Ġstrictly Ġdefined Ġacademic Ġfields . ĠWe Ġwork Ġwith Ġsuch Ġa Ġcorpus Ġof ĠRussian Ġand ĠUkrainian Ġacademic Ġtexts , Ġinitially Ġcollected Ġfor Ġthe Ġpurposes Ġof Ġcross - ling ual Ġplagiar ism Ġdetection . ĠThis Ġdata Ġis Ġavailable Ġonline Ġthrough Ġa Ġnumber Ġof Ġlibrary Ġservices , Ġbut Ġunfortunately Ġcannot Ġbe Ġrep ublished Ġdue Ġto Ġcopyright Ġlimitations . </s> The ĠUkrainian Ġsub cor p us Ġcontains Ġabout Ġ60 Ġthousand Ġextended Ġsumm aries Ġ( Russian Ġand ĠUkrainian Ġr ussian ` Ð° Ð² ÑĤ Ð¾ ÑĢ Ðµ Ñ Ħ Ðµ ÑĢ Ð° ÑĤ ', Ġ` av t ore fer at ') Ġof Ġthe ses Ġsubmitted Ġbetween Ġ1998 Ġand Ġ2011 . ĠThe ĠRussian Ġsub cor p us Ġis Ġsmaller Ġin Ġthe Ġnumber Ġof Ġdocuments Ġ( about Ġ16 Ġthousand , Ġapproximately Ġthe Ġsame Ġtime Ġperiod ), Ġbut Ġthe Ġdocuments Ġare Ġfull Ġtexts Ġof Ġthe ses , Ġthus Ġthe Ġtotal Ġvolume Ġof Ġthe ĠRussian Ġsub cor p us Ġis Ġnotably Ġlarger : Ġ8 30 Ġmillion Ġtokens Ġversus Ġ250 Ġmillion Ġtokens Ġin Ġthe ĠUkrainian Ġone . ĠGenerally , Ġthe Ġtexts Ġbelong Ġto Ġone Ġgenre Ġthat Ġcan Ġbe Ġdefined Ġas Ġpost - Soviet Ġexp ository Ġacademic Ġprose , Ġsubmitted Ġfor Ġacademic Ġdegree Ġaward Ġprocess . </s> The Ġdocuments Ġwere Ġconverted Ġto Ġplain Ġtext Ġfiles Ġfrom ĠMS ĠWord Ġformat Ġin Ġthe Ġcase Ġof Ġthe ĠUkrainian Ġsub cor p us Ġand Ġmainly Ġfrom ĠOC Red ĠPDF Ġfiles Ġin Ġthe Ġcase Ġof Ġthe ĠRussian Ġsub cor p us . ĠBecause Ġof Ġthis , Ġthe ĠRussian Ġdocuments Ġoften Ġsuffer Ġfrom ĠO CR Ġartifacts , Ġsuch Ġas Ġwords Ġsplit Ġwith Ġline Ġbreaks , Ġincorrectly Ġrecognized Ġcharacters Ġand Ġso Ġon . ĠHowever , Ġit Ġdoes Ġnot Ġinfluence Ġthe Ġresulting Ġmodel Ġmuch , Ġas Ġwe Ġshow Ġbelow . </s> Both ĠUkrainian Ġand ĠRussian Ġdocuments Ġcome Ġwith Ġmeta Ġdata Ġallowing Ġto Ġseparate Ġthem Ġinto Ġacademic Ġfields , Ġwith Ġeconomics , Ġmedicine Ġand Ġlaw Ġbeing Ġmost Ġfrequent Ġtopics Ġfor Ġthe ĠUkrainian Ġdata Ġand Ġeconomics , Ġhistory Ġand Ġped ag ogy Ġdominating Ġthe ĠRussian Ġdata . </s> For Ġevaluation , Ġ3 Ġtopics Ġwere Ġused , Ġdistant Ġenough Ġfrom Ġeach Ġother Ġand Ġabundantly Ġpresented Ġin Ġboth Ġsub cor pora : Ġeconomics , Ġlaw Ġand Ġhistory . ĠWe Ġrandomly Ġselected Ġ100 Ġtexts Ġin Ġeach Ġlanguage Ġfor Ġeach Ġtopic . ĠAs Ġan Ġaverage Ġlength Ġof ĠRussian Ġtexts Ġis Ġsignificantly Ġhigher Ġ( them Ġbeing Ġfull Ġthe ses ), Ġwe Ġcropped Ġthem , Ġleaving Ġonly Ġthe Ġfirst Ġ5 Ġthousand Ġwords , Ġto Ġmimic Ġthe Ġsize Ġof Ġthe ĠUkrainian Ġsumm aries . ĠThese Ġ600 Ġdocuments Ġin Ġ3 Ġclasses Ġare Ġused Ġas Ġa Ġtest Ġset Ġ( see ĠSection Ġ" Exper iment ĠDesign Ġand ĠEvaluation " Ġfor Ġthe Ġdescription Ġof Ġthe Ġconducted Ġexperiments ). </s> The Ġcorpor a Ġ( including Ġtest Ġset ) Ġwere ĠPo S - tag ged . ĠEach Ġword Ġwas Ġreplaced Ġwith Ġits Ġle mma Ġfollowed Ġby Ġa ĠPo S - tag Ġ( ` r ussian Ð´ Ð¸ Ñģ Ñģ Ðµ ÑĢ ÑĤ Ð° Ñ Ĩ Ð¸ Ñı _ S ', Ġ` r ussian Ð´ Ð¸ Ñģ Ñģ Ðµ ÑĢ ÑĤ Ð° Ñ Ĩ i Ñı _ N '). ĠFunctional Ġparts Ġof Ġspeech Ġ( con jun ctions , Ġpronouns , Ġpre pos itions , Ġetc .) Ġand Ġnumer als Ġwere Ġremoved Ġfrom Ġthe Ġtexts . </s> Learning Ġto ĠTrans late : ĠUkrainian - to - Russian Ġtransformations </s> As Ġalready Ġstated , Ġour Ġmain Ġproposal Ġis Ġusing Ġneural Ġembed ding Ġmodels Ġto Ġ` project ' Ġdocuments Ġin Ġone Ġlanguage Ġinto Ġthe Ġsemantic Ġspace Ġof Ġanother Ġlanguage . ĠFor Ġthis , Ġwe Ġfirst Ġtrained Ġa ĠContinuous ĠBag - of - Words Ġ( CB OW ) Ġand Ġa ĠContinuous ĠSkip G ram Ġmodel ĠB IB REF 0 Ġfor Ġeach Ġof Ġour Ġmon oling ual Ġsub cor pora . ĠThe Ġmodels Ġwere Ġtrained Ġwith Ġidentical Ġhyper param eters : Ġvector Ġsize Ġof Ġ300 Ġcomponents , Ġsymm etric Ġwindow Ġof Ġ2 Ġwords , Ġnegative Ġsampling Ġwith Ġ10 Ġsamples , Ġ5 Ġiterations Ġover Ġthe Ġcorpus , Ġno Ġdown - sam pling . ĠThe Ġonly Ġlanguage - dependent Ġdifference Ġwas Ġthat Ġfor Ġthe ĠUkrainian Ġmodel Ġwe Ġignored Ġwords Ġwith Ġthe Ġcorpus Ġfrequency Ġless Ġthan Ġ10 Ġand Ġfor Ġthe ĠRussian Ġmodel Ġthis Ġthreshold Ġwas Ġset Ġto Ġ15 Ġ( as Ġthe ĠRussian Ġcorpus Ġis Ġ3 Ġtimes Ġlarger ). ĠAll Ġin Ġall , Ġthe Ġfinal ĠUkrainian Ġmodel Ġrecognizes Ġ429 Ġ215 Ġwords Ġand Ġthe ĠRussian Ġone Ġ271 Ġ720 Ġwords . ĠTraining Ġwas Ġperformed Ġusing ĠCB OW Ġand ĠSkip G ram Ġimplementation Ġin ĠG ens im Ġlibrary ĠB IB REF 7 Ġ. </s> After Ġthe Ġmodels Ġwere Ġtrained , Ġwe Ġfollowed Ġthe Ġpath Ġoutlined Ġin ĠB IB REF 4 Ġto Ġlearn Ġa Ġlinear Ġtransformation Ġmatrix Ġfrom ĠUkrainian Ġto ĠRussian . ĠFirst , Ġwe Ġextracted Ġall Ġnoun Ġpairs Ġfrom ĠRussian - Uk rain ian Ġbilingual Ġdictionary ĠB IB REF 8 Ġ, Ġwith Ġthe Ġconstraint Ġthat Ġtheir Ġfrequency Ġin Ġour Ġcorpor a Ġwas Ġabove Ġthe Ġalready Ġmentioned Ġthresholds Ġ15 Ġand Ġ10 Ġfor ĠRussian Ġand ĠUkrainian Ġwords Ġcorrespond ingly . ĠThat Ġmade Ġit Ġa Ġlist Ġof Ġabout Ġ5 Ġthousand Ġpairs Ġof Ġnoun s Ġbeing Ġtranslations Ġof Ġeach Ġother . </s> For Ġall Ġthese Ġwords , Ġtheir Ġvectors Ġwere Ġfound Ġin Ġthe Ġmodels Ġcorresponding Ġto Ġthe Ġwords ' Ġlanguages . ĠIt Ġprovided Ġus Ġwith Ġa Ġmatrix Ġof Ġ5 Ġthousand Ġof Ġ300 - dimensional ĠUkrainian Ġvectors Ġand Ġthe Ġmatrix Ġof Ġcorresponding Ġ5 Ġthousand Ġof Ġ300 - dimensional ĠRussian Ġvectors . ĠThis Ġdata Ġserved Ġas Ġa Ġtraining Ġset Ġto Ġlearn Ġan Ġoptimal Ġtransformation Ġmatrix . ĠThe Ġlatter Ġis Ġactually Ġa Ġ300 x 301 Ġmatrix Ġof Ġcoefficients , Ġsuch Ġthat Ġwhen Ġthe Ġinitial ĠUkrainian Ġmatrix Ġis Ġmultiplied Ġby Ġthis Ġtransformation Ġmatrix , Ġthe Ġresult Ġis Ġmaxim ally Ġclose Ġto Ġthe Ġcorresponding ĠRussian Ġmatrix . ĠThis Ġtransformation Ġmatrix Ġhas Ġ301 Ġ( not Ġ300 ) Ġcolumns , Ġbecause Ġwe Ġadd Ġone Ġcomponent Ġequal Ġto Ġ1 Ġto Ġeach Ġvector , Ġas Ġa Ġbias Ġterm . </s> Produ cing Ġthe Ġtransformation Ġmatrix Ġis Ġa Ġlinear Ġregression Ġproblem : Ġthe Ġinput Ġis Ġ301 Ġcomponents Ġof ĠUkrainian Ġvectors Ġ( including Ġthe Ġbias Ġterm ) Ġand Ġthe Ġoutput Ġis Ġ300 Ġcomponents Ġof ĠRussian Ġvectors . ĠAs Ġwe Ġneed Ġ300 Ġvalues Ġas Ġan Ġoutput , Ġthere Ġare Ġactually Ġ300 Ġlinear Ġregression Ġproblems Ġand Ġthat 's Ġwhy Ġthe Ġresulting Ġmatrix Ġsize Ġis Ġ300 x 301 Ġ( 301 Ġweights Ġfor Ġeach Ġof Ġ300 Ġcomponents ). </s> There Ġare Ġtwo Ġmain Ġways Ġto Ġsolve Ġa Ġlinear Ġregression Ġproblem : Ġone Ġcan Ġeither Ġlearn Ġthe Ġoptimal Ġweights Ġin Ġan Ġiter ative Ġway Ġusing Ġsome Ġvariant Ġof Ġgradient Ġdescent , Ġor Ġone Ġcan Ġsolve Ġit Ġnumer ically Ġwithout Ġiteration , Ġusing Ġnormal Ġequation . ĠFor ĠEnglish Ġand ĠSpanish , ĠB IB REF 4 Ġused Ġsto ch astic Ġgradient Ġdescent . ĠHowever , Ġnormal Ġequation Ġis Ġactually Ġless Ġerror - prone Ġand Ġis Ġguaranteed Ġto Ġfind Ġthe Ġglobal Ġoptimum . ĠIts Ġonly Ġdisadvantage Ġis Ġthat Ġit Ġbecomes Ġvery Ġcomput ationally Ġexpensive Ġwhen Ġthe Ġnumber Ġof Ġfeatures Ġis Ġlarge Ġ( th ousands Ġand Ġmore ). ĠHowever , Ġin Ġour Ġcase Ġthe Ġnumber Ġof Ġfeatures Ġis Ġonly Ġ301 , Ġso Ġcomputational Ġcomplexity Ġis Ġnot Ġan Ġissue . </s> Thus , Ġwe Ġuse Ġnormal Ġequation Ġto Ġfind Ġthe Ġoptimal Ġtransformation Ġmatrix . ĠThe Ġalgebra ic Ġsolution Ġto Ġeach Ġof Ġ300 Ġnormal Ġequations Ġ( one Ġfor Ġeach Ġvector Ġcomponent Ġ$ i $ Ġ) Ġis Ġshown Ġin Ġthe ĠEqu ation Ġ3 Ġ: </s> $$ \ beta Ġ_ i Ġ= Ġ( \ text bf Ġ{ X } ^ \ inter cal Ġ* Ġ\ text bf Ġ{ X }) ^{ - 1 } Ġ* Ġ\ text bf Ġ{ X } ^ \ inter cal Ġ* Ġy _ i $$ Ġ Ġ Ġ( E q . Ġ3 ) </s> where Ġ$\ text bf Ġ{ X } $ Ġis Ġthe Ġmatrix Ġof Ġ5 Ġthousand ĠUkrainian Ġword Ġvectors Ġ( input ), Ġ$ y _ i $ Ġis Ġthe Ġvector Ġof Ġthe Ġ$ i $ Ġth Ġcomponents Ġof Ġ5 Ġthousand Ġcorresponding ĠRussian Ġwords Ġ( correct Ġpredictions ), Ġand Ġ$\ beta Ġ_ i $ Ġis Ġour Ġaim : Ġthe Ġvector Ġof Ġ301 Ġoptimal Ġcoefficients Ġwhich Ġtransform Ġthe ĠUkrainian Ġvectors Ġinto Ġthe Ġ$ i $ Ġth Ġcomponent Ġof Ġthe ĠRussian Ġvectors . </s> After Ġsolving Ġsuch Ġnormal Ġequations Ġfor Ġall Ġthe Ġ300 Ġcomponents Ġ$ i $ Ġ, Ġwe Ġhave Ġthe Ġ300 x 301 Ġlinear Ġtransformation Ġmatrix Ġwhich Ġfits Ġthe Ġdata Ġbest . </s> This Ġmatrix Ġbasically Ġmaps Ġthe ĠUkrainian Ġvectors Ġinto Ġthe ĠRussian Ġones . ĠIt Ġis Ġbased Ġon Ġthe Ġassumption Ġthat Ġthe Ġrelations Ġbetween Ġsemantic Ġconcepts Ġin Ġdifferent Ġlanguages Ġare Ġin Ġfact Ġvery Ġsimilar Ġ( stud ents Ġare Ġclose Ġto Ġteachers , Ġwhile Ġpirates Ġare Ġclose Ġto Ġc ors airs , Ġand Ġso Ġon ). ĠIn Ġcontinuous Ġdistribution al Ġmodels Ġwhich Ġstrive Ġto Ġrepresent Ġthese Ġsemantic Ġspaces , Ġmutual Ġ` ge omet rical ' Ġrelations Ġbetween Ġvectors Ġrepresenting Ġparticular Ġwords Ġare Ġalso Ġsimilar Ġacross Ġmodels Ġ( if Ġthey Ġare Ġtrained Ġon Ġcomparable Ġcorpor a ), Ġbut Ġthe Ġexact Ġvectors Ġfor Ġwords Ġden oting Ġone Ġand Ġthe Ġsame Ġnotion Ġare Ġdifferent . ĠThis Ġis Ġbecause Ġthe Ġmodels Ġthemselves Ġare Ġsto ch astic Ġand Ġthe Ġparticular Ġvalues Ġof Ġvectors Ġ( un like Ġtheir Ġpositions Ġin Ġrelation Ġto Ġeach Ġother ) Ġdepend Ġa Ġlot Ġon Ġtechnical Ġfactors , Ġincluding Ġthe Ġrandom Ġseed Ġused Ġto Ġinitialize Ġvectors Ġprior Ġto Ġtraining . ĠIn Ġorder Ġto Ġmigrate Ġfrom Ġa Ġmodel ĠA Ġto Ġanother Ġmodel ĠB , Ġone Ġhas Ġto Ġ` rot ate Ġand Ġscale ' ĠA Ġvectors Ġin Ġa Ġuniform Ġlinear Ġway . ĠTo Ġlearn Ġthe Ġoptimal Ġtransformation Ġmatrix Ġmeans Ġto Ġfind Ġout Ġthe Ġexact Ġdirections Ġof Ġrotating Ġand Ġscaling , Ġwhich Ġminimize Ġprediction Ġerrors . </s> L ingu istically Ġspeaking , Ġonce Ġwe Ġlearned Ġthe Ġtransformation Ġmatrix , Ġwe Ġcan Ġpredict Ġwhat Ġa ĠRussian Ġvector Ġwould Ġmost Ġprobably Ġbe , Ġgiven Ġa ĠUkrainian Ġone . ĠThis Ġessentially Ġmeans Ġwe Ġare Ġable Ġto Ġ` trans late ' ĠUkrainian Ġwords Ġinto ĠRussian , Ġby Ġcalculating Ġthe Ġword Ġin Ġthe ĠRussian Ġmodel Ġwith Ġthe Ġvector Ġclosest Ġto Ġthe Ġpredicted Ġone . </s> We Ġhad Ġto Ġchoose Ġbetween ĠCB OW Ġor ĠContinuous ĠSkip G ram Ġmodels Ġto Ġuse Ġwhen Ġlearning Ġthe Ġtransformation Ġmatrix . ĠAlso , Ġthere Ġwas Ġa Ġquestion Ġof Ġwhether Ġto Ġemploy Ġregular ized Ġor Ġstandard Ġnormal Ġequations . ĠRegular ization Ġis Ġan Ġattempt Ġto Ġavoid Ġover - fitting Ġby Ġtrying Ġto Ġsomehow Ġdecrease Ġthe Ġvalues Ġof Ġlearned Ġweights . ĠThe Ġregular ized Ġnormal Ġequation Ġis Ġshown Ġin Ġ4 Ġ: </s> $$ \ beta Ġ_ i Ġ= Ġ( \ text bf Ġ{ X } ^ \ inter cal Ġ* Ġ\ text bf Ġ{ X } Ġ+ Ġ\ lambda Ġ* ĠL ) ^{ - 1 } Ġ* Ġ\ text bf Ġ{ X } ^ \ inter cal Ġ* Ġy _ i $$ Ġ Ġ Ġ( E q . Ġ4 ) </s> Comp aring Ġto Ġ3 Ġ, Ġit Ġadds Ġthe Ġterm Ġ$\ lambda Ġ* ĠL $ Ġ, Ġwhere Ġ$ L $ Ġis Ġthe Ġidentity Ġmatrix Ġof Ġthe Ġsize Ġequal Ġto Ġthe Ġnumber Ġof Ġfeatures , Ġwith Ġ0 Ġat Ġthe Ġtop Ġleft Ġcell , Ġand Ġ$\ lambda Ġ$ Ġis Ġa Ġreal Ġnumber Ġused Ġto Ġtune Ġthe Ġinfluence Ġof Ġregular ization Ġterm Ġ( if Ġ$\ lambda Ġ= Ġ0 $ Ġ, Ġthere Ġis Ġno Ġregular ization ). </s> To Ġtest Ġall Ġthe Ġpossible Ġcombinations Ġof Ġparameters , Ġwe Ġdivided Ġthe Ġbilingual Ġdictionary Ġinto Ġ4 500 Ġnoun Ġpairs Ġused Ġas Ġa Ġtraining Ġset Ġand Ġ500 Ġnoun Ġpairs Ġused Ġas Ġa Ġtest Ġset . ĠWe Ġthen Ġlearned Ġtransformation Ġmat rices Ġon Ġthe Ġtraining Ġset Ġusing Ġboth Ġtraining Ġalgorithms Ġ( CB OW Ġand ĠSkip G ram ) Ġand Ġseveral Ġvalues Ġof Ġregular ization Ġ$\ lambda Ġ$ Ġfrom Ġ0 Ġto Ġ5 , Ġwith Ġa Ġstep Ġof Ġ0 . 5 . ĠThe Ġresulting Ġmat rices Ġwere Ġapplied Ġto Ġthe ĠUkrainian Ġvectors Ġfrom Ġthe Ġtest Ġset Ġand Ġthe Ġcorresponding ĠRussian Ġ` trans lations ' Ġwere Ġcalculated . ĠThe Ġratio Ġof Ġcorrect Ġ` trans lations ' Ġ( mat ches ) Ġwas Ġused Ġas Ġan Ġevaluation Ġmeasure . ĠIt Ġcame Ġout Ġthat Ġregular ization Ġonly Ġworsened Ġthe Ġresults Ġfor Ġboth Ġalgorithms , Ġso Ġin Ġthe ĠTable Ġ1 Ġwe Ġreport Ġthe Ġresults Ġwithout Ġregular ization . </s> For Ġreference , Ġwe Ġalso Ġreport Ġthe Ġaccuracy Ġof Ġ` qu azi - translation ' Ġvia ĠD amer au - Le ven sh tein Ġedit Ġdistance ĠB IB REF 9 Ġ, Ġas Ġa Ġsort Ġof Ġa Ġbaseline . ĠAs Ġalready Ġstated , Ġthe Ġtwo Ġlanguages Ġshare Ġmany Ġcogn ates , Ġand Ġa Ġlot Ġof ĠUkrainian Ġwords Ġcan Ġbe Ġorth ographically Ġtransformed Ġinto Ġtheir ĠRussian Ġtranslations Ġ( and Ġvice Ġversa ) Ġby Ġone Ġor Ġtwo Ġcharacter Ġreplacements . ĠThus , Ġwe Ġextracted Ġ50 , 000 Ġmost Ġfrequent Ġnoun s Ġfrom Ġour ĠRussian Ġcorpor a ; Ġthen Ġfor Ġeach ĠUkrainian Ġnoun Ġin Ġthe Ġbilingual Ġdictionary Ġwe Ġfound Ġthe Ġclosest ĠRussian Ġnoun Ġ( or Ġ5 Ġclosest Ġnoun s Ġfor Ġ@ 5 Ġmetric ) Ġby Ġedit Ġdistance Ġand Ġcalculated Ġhow Ġoften Ġit Ġturned Ġout Ġto Ġbe Ġthe Ġcorrect Ġtranslation . ĠAs Ġthe ĠTable Ġ1 Ġshows , Ġnotwithstanding Ġthe Ġorth ographic Ġsimilarity Ġof Ġthe Ġtwo Ġlanguages , ĠCB OW Ġconsistently Ġoutper forms Ġthis Ġapproach Ġeven Ġon Ġthe Ġtest Ġset . ĠOn Ġthe Ġtraining Ġset , Ġits Ġsuperiority Ġis Ġeven Ġmore Ġobvious .
Introduction This research addresses the problem of representing the semantics of text documents in multi-lingual comparable corpora. We present a new approach to this problem, based on neural embeddings, and test it on the task of clustering texts into meaningful classes depending on their topics. The setting is unsupervised, meaning that one either does not have enough annotated data to train a supervised classifier or does not want to be limited with a pre-defined set of classes. There is a lot of sufficiently good approaches to this problem in the case of mono-lingual text collections, but the presence of multiple languages introduces complications. When a text collection contains documents in several languages, it becomes impractical to simply represent the documents as vectors of words occurring in them ("bag-of-words"), as the words surface forms are different, even in closely-related languages. Thus, one has to invent means to cross the inter-lingual gap and bring all documents to some sort of shared representation, without losing information about their topics or categories. Of course, one obvious way to solve this problem is to translate all documents into one language, and then apply any clustering algorithm. However, this requires either buying human/machine translation services (which can be expensive if you deal with large text collection) or training own statistical machine translation model (which as a rule requires big parallel corpus). This is the reason to search for other solutions. In this paper, a novel way of reducing the problem of cross-lingual document representation to a monolingual setting is proposed. Essentially, we train Continuous Bag-of-Words models BIBREF0 on large comparable monolingual corpora for two languages our dataset consists of. This provides us with vector representations of words, allowing to measure their semantic similarity. Then, a linear transformation matrix from vectors of language A to vectors of language B is learned, using a small bilingual dictionary as training data. This matrix is then employed to `project' word and document representations from semantic space of language A to semantic space of language B. It allows not only quite accurate `translation' of words, but also of document `semantic fingerprints' (dense representations of document semantics, calculated as an average of the trained distributional vectors for all the words in document). This approach is evaluated in a setting, where the input is a collection of documents in several languages and some number of topics to which these documents belong (we also have large monolingual corpora to train distributional models on). For each document, we are given its language, but not its topic. The task is to cluster this collection so that documents belonging to one topic were clustered together, independent of their language. Note that we are interested in clustering the collection as a whole, not each language separately (which is trivial). Our evaluation data consists of comparable corpora of Russian and Ukrainian academic texts. On this material, we show that the `translated semantic fingerprints' method represents documents in different languages precisely enough to allow almost exact clustering according to document topics, with only 5% of incorrect assignments. It significantly outperforms both naive bag-of-words baseline and the not-so-naive method of `orthographic translation' based on Damerau-Levenshtein distance, even enriched with dictionary mappings. At the same time, it does not require large parallel corpora or a ready-made statistical machine translation model. The rest of the paper is structured as follows. In Section "Related Work" we describe the foundations of our approach and the related work. Section "Academic texts as Comparable Corpora" introduces the employed corpora and the story behind them. Section "Learning to Translate: Ukrainian-to-Russian transformations" is dedicated to learning the transformation matrix, and Section "Experiment Design and Evaluation" describes our experimental setting and evaluation results. We discuss the findings in Section "Discussion" and conclude in Section "Conclusion and Future Work" , also suggesting directions for future work. Related Work Clustering multi-lingual documents has received much attention in natural language processing. Among approaches not using some form of machine translation, one can mention BIBREF1 , who essentially employ a bilingual dictionary to bring some words in the documents to a language-independent form and then to perform clustering. In the section "Experiment Design and Evaluation" we show that our approach based on neural embeddings significantly outperforms their reported results. BIBREF2 proposed training joint multi-lingual neural embedding models. Theoretically, this can be used to achieve our aim of language-independent semantic representations for documents. Unfortunately, it demands a large word-aligned parallel corpus. This is not the case with the more recent Trans-gram approach introduced in BIBREF3 , also able to learn multi-lingual models. However, it still needs sentence-aligned corpora to train on (in the size of millions of paired sentences). Large parallel corpora (whether word- or sentence-aligned) are often a scarce resource, especially in the case of under-represented languages. The approach described in this paper takes as an input only comparable monolingual corpora and bilingual dictionaries in the size of several thousand word pairs. Such resources are much easier to find and evaluate. We employ the idea of learning a linear transformation matrix to map or project word embeddings from the semantic space of one language to that of another. This idea was first proposed in BIBREF4 , who applied it to lexical translation between English, Spanish, Czech and Vietnamese. We extend it from continuous representations of single words or collocations to `semantic fingerprints' of documents as a whole. Academic texts as Comparable Corpora The Russian and Ukrainian languages are mainly spoken in Russian Federation and the Ukraine and belong to the East-Slavic group of the Indo-European language family. They share many common morphosyntactic features: both are SVO languages with free word order and rich morphology, both use the Cyrillic alphabet and share many common cognates. Both Russia and the Ukraine have common academic tradition that makes it easier to collect corpora, which are comparable in terms of both genre and strictly defined academic fields. We work with such a corpus of Russian and Ukrainian academic texts, initially collected for the purposes of cross-lingual plagiarism detection. This data is available online through a number of library services, but unfortunately cannot be republished due to copyright limitations. The Ukrainian subcorpus contains about 60 thousand extended summaries (Russian and Ukrainian russian`автореферат', `avtoreferat') of theses submitted between 1998 and 2011. The Russian subcorpus is smaller in the number of documents (about 16 thousand, approximately the same time period), but the documents are full texts of theses, thus the total volume of the Russian subcorpus is notably larger: 830 million tokens versus 250 million tokens in the Ukrainian one. Generally, the texts belong to one genre that can be defined as post-Soviet expository academic prose, submitted for academic degree award process. The documents were converted to plain text files from MS Word format in the case of the Ukrainian subcorpus and mainly from OCRed PDF files in the case of the Russian subcorpus. Because of this, the Russian documents often suffer from OCR artifacts, such as words split with line breaks, incorrectly recognized characters and so on. However, it does not influence the resulting model much, as we show below. Both Ukrainian and Russian documents come with meta data allowing to separate them into academic fields, with economics, medicine and law being most frequent topics for the Ukrainian data and economics, history and pedagogy dominating the Russian data. For evaluation, 3 topics were used, distant enough from each other and abundantly presented in both subcorpora: economics, law and history. We randomly selected 100 texts in each language for each topic. As an average length of Russian texts is significantly higher (them being full theses), we cropped them, leaving only the first 5 thousand words, to mimic the size of the Ukrainian summaries. These 600 documents in 3 classes are used as a test set (see Section "Experiment Design and Evaluation" for the description of the conducted experiments). The corpora (including test set) were PoS-tagged. Each word was replaced with its lemma followed by a PoS-tag (`russianдиссертация_S', `russianдиссертацiя_N'). Functional parts of speech (conjunctions, pronouns, prepositions, etc.) and numerals were removed from the texts. Learning to Translate: Ukrainian-to-Russian transformations As already stated, our main proposal is using neural embedding models to `project' documents in one language into the semantic space of another language. For this, we first trained a Continuous Bag-of-Words (CBOW) and a Continuous SkipGram model BIBREF0 for each of our monolingual subcorpora. The models were trained with identical hyperparameters: vector size of 300 components, symmetric window of 2 words, negative sampling with 10 samples, 5 iterations over the corpus, no down-sampling. The only language-dependent difference was that for the Ukrainian model we ignored words with the corpus frequency less than 10 and for the Russian model this threshold was set to 15 (as the Russian corpus is 3 times larger). All in all, the final Ukrainian model recognizes 429 215 words and the Russian one 271 720 words. Training was performed using CBOW and SkipGram implementation in Gensim library BIBREF7 . After the models were trained, we followed the path outlined in BIBREF4 to learn a linear transformation matrix from Ukrainian to Russian. First, we extracted all noun pairs from Russian-Ukrainian bilingual dictionary BIBREF8 , with the constraint that their frequency in our corpora was above the already mentioned thresholds 15 and 10 for Russian and Ukrainian words correspondingly. That made it a list of about 5 thousand pairs of nouns being translations of each other. For all these words, their vectors were found in the models corresponding to the words' languages. It provided us with a matrix of 5 thousand of 300-dimensional Ukrainian vectors and the matrix of corresponding 5 thousand of 300-dimensional Russian vectors. This data served as a training set to learn an optimal transformation matrix. The latter is actually a 300x301 matrix of coefficients, such that when the initial Ukrainian matrix is multiplied by this transformation matrix, the result is maximally close to the corresponding Russian matrix. This transformation matrix has 301 (not 300) columns, because we add one component equal to 1 to each vector, as a bias term. Producing the transformation matrix is a linear regression problem: the input is 301 components of Ukrainian vectors (including the bias term) and the output is 300 components of Russian vectors. As we need 300 values as an output, there are actually 300 linear regression problems and that's why the resulting matrix size is 300x301 (301 weights for each of 300 components). There are two main ways to solve a linear regression problem: one can either learn the optimal weights in an iterative way using some variant of gradient descent, or one can solve it numerically without iteration, using normal equation. For English and Spanish, BIBREF4 used stochastic gradient descent. However, normal equation is actually less error-prone and is guaranteed to find the global optimum. Its only disadvantage is that it becomes very computationally expensive when the number of features is large (thousands and more). However, in our case the number of features is only 301, so computational complexity is not an issue. Thus, we use normal equation to find the optimal transformation matrix. The algebraic solution to each of 300 normal equations (one for each vector component $i$ ) is shown in the Equation 3 : $$\beta _i = (\textbf {X}^\intercal * \textbf {X})^{-1} * \textbf {X}^\intercal * y_i$$   (Eq. 3) where $\textbf {X}$ is the matrix of 5 thousand Ukrainian word vectors (input), $y_i$ is the vector of the $i$ th components of 5 thousand corresponding Russian words (correct predictions), and $\beta _i$ is our aim: the vector of 301 optimal coefficients which transform the Ukrainian vectors into the $i$ th component of the Russian vectors. After solving such normal equations for all the 300 components $i$ , we have the 300x301 linear transformation matrix which fits the data best. This matrix basically maps the Ukrainian vectors into the Russian ones. It is based on the assumption that the relations between semantic concepts in different languages are in fact very similar (students are close to teachers, while pirates are close to corsairs, and so on). In continuous distributional models which strive to represent these semantic spaces, mutual `geometrical' relations between vectors representing particular words are also similar across models (if they are trained on comparable corpora), but the exact vectors for words denoting one and the same notion are different. This is because the models themselves are stochastic and the particular values of vectors (unlike their positions in relation to each other) depend a lot on technical factors, including the random seed used to initialize vectors prior to training. In order to migrate from a model A to another model B, one has to `rotate and scale' A vectors in a uniform linear way. To learn the optimal transformation matrix means to find out the exact directions of rotating and scaling, which minimize prediction errors. Linguistically speaking, once we learned the transformation matrix, we can predict what a Russian vector would most probably be, given a Ukrainian one. This essentially means we are able to `translate' Ukrainian words into Russian, by calculating the word in the Russian model with the vector closest to the predicted one. We had to choose between CBOW or Continuous SkipGram models to use when learning the transformation matrix. Also, there was a question of whether to employ regularized or standard normal equations. Regularization is an attempt to avoid over-fitting by trying to somehow decrease the values of learned weights. The regularized normal equation is shown in 4 : $$\beta _i = (\textbf {X}^\intercal * \textbf {X} + \lambda * L)^{-1} * \textbf {X}^\intercal * y_i$$   (Eq. 4) Comparing to 3 , it adds the term $\lambda * L$ , where $L$ is the identity matrix of the size equal to the number of features, with 0 at the top left cell, and $\lambda $ is a real number used to tune the influence of regularization term (if $\lambda = 0$ , there is no regularization). To test all the possible combinations of parameters, we divided the bilingual dictionary into 4500 noun pairs used as a training set and 500 noun pairs used as a test set. We then learned transformation matrices on the training set using both training algorithms (CBOW and SkipGram) and several values of regularization $\lambda $ from 0 to 5, with a step of 0.5. The resulting matrices were applied to the Ukrainian vectors from the test set and the corresponding Russian `translations' were calculated. The ratio of correct `translations' (matches) was used as an evaluation measure. It came out that regularization only worsened the results for both algorithms, so in the Table 1 we report the results without regularization. For reference, we also report the accuracy of `quazi-translation' via Damerau-Levenshtein edit distance BIBREF9 , as a sort of a baseline. As already stated, the two languages share many cognates, and a lot of Ukrainian words can be orthographically transformed into their Russian translations (and vice versa) by one or two character replacements. Thus, we extracted 50,000 most frequent nouns from our Russian corpora; then for each Ukrainian noun in the bilingual dictionary we found the closest Russian noun (or 5 closest nouns for @5 metric) by edit distance and calculated how often it turned out to be the correct translation. As the Table 1 shows, notwithstanding the orthographic similarity of the two languages, CBOW consistently outperforms this approach even on the test set. On the training set, its superiority is even more obvious.
What evaluation metric do they use?
aefa333b2cf0a4000cd40566149816f5b36135e7
What Ġevaluation Ġmetric Ġdo Ġthey Ġuse ?
Acc uracy
<s> What Ġevaluation Ġmetric Ġdo Ġthey Ġuse ? </s> Introduction </s> This Ġresearch Ġaddresses Ġthe Ġproblem Ġof Ġrepresenting Ġthe Ġsemantics Ġof Ġtext Ġdocuments Ġin Ġmulti - ling ual Ġcomparable Ġcorpor a . ĠWe Ġpresent Ġa Ġnew Ġapproach Ġto Ġthis Ġproblem , Ġbased Ġon Ġneural Ġembed d ings , Ġand Ġtest Ġit Ġon Ġthe Ġtask Ġof Ġclust ering Ġtexts Ġinto Ġmeaningful Ġclasses Ġdepending Ġon Ġtheir Ġtopics . ĠThe Ġsetting Ġis Ġun super vised , Ġmeaning Ġthat Ġone Ġeither Ġdoes Ġnot Ġhave Ġenough Ġannot ated Ġdata Ġto Ġtrain Ġa Ġsupervised Ġclass ifier Ġor Ġdoes Ġnot Ġwant Ġto Ġbe Ġlimited Ġwith Ġa Ġpre - defined Ġset Ġof Ġclasses . ĠThere Ġis Ġa Ġlot Ġof Ġsufficiently Ġgood Ġapproaches Ġto Ġthis Ġproblem Ġin Ġthe Ġcase Ġof Ġmono - ling ual Ġtext Ġcollections , Ġbut Ġthe Ġpresence Ġof Ġmultiple Ġlanguages Ġintroduces Ġcomplications . </s> When Ġa Ġtext Ġcollection Ġcontains Ġdocuments Ġin Ġseveral Ġlanguages , Ġit Ġbecomes Ġimpractical Ġto Ġsimply Ġrepresent Ġthe Ġdocuments Ġas Ġvectors Ġof Ġwords Ġoccurring Ġin Ġthem Ġ(" bag - of - words "), Ġas Ġthe Ġwords Ġsurface Ġforms Ġare Ġdifferent , Ġeven Ġin Ġclosely - related Ġlanguages . ĠThus , Ġone Ġhas Ġto Ġinvent Ġmeans Ġto Ġcross Ġthe Ġinter - ling ual Ġgap Ġand Ġbring Ġall Ġdocuments Ġto Ġsome Ġsort Ġof Ġshared Ġrepresentation , Ġwithout Ġlosing Ġinformation Ġabout Ġtheir Ġtopics Ġor Ġcategories . </s> Of Ġcourse , Ġone Ġobvious Ġway Ġto Ġsolve Ġthis Ġproblem Ġis Ġto Ġtranslate Ġall Ġdocuments Ġinto Ġone Ġlanguage , Ġand Ġthen Ġapply Ġany Ġclust ering Ġalgorithm . ĠHowever , Ġthis Ġrequires Ġeither Ġbuying Ġhuman / machine Ġtranslation Ġservices Ġ( which Ġcan Ġbe Ġexpensive Ġif Ġyou Ġdeal Ġwith Ġlarge Ġtext Ġcollection ) Ġor Ġtraining Ġown Ġstatistical Ġmachine Ġtranslation Ġmodel Ġ( which Ġas Ġa Ġrule Ġrequires Ġbig Ġparallel Ġcorpus ). ĠThis Ġis Ġthe Ġreason Ġto Ġsearch Ġfor Ġother Ġsolutions . </s> In Ġthis Ġpaper , Ġa Ġnovel Ġway Ġof Ġreducing Ġthe Ġproblem Ġof Ġcross - ling ual Ġdocument Ġrepresentation Ġto Ġa Ġmon oling ual Ġsetting Ġis Ġproposed . ĠEssentially , Ġwe Ġtrain ĠContinuous ĠBag - of - Words Ġmodels ĠB IB REF 0 Ġon Ġlarge Ġcomparable Ġmon oling ual Ġcorpor a Ġfor Ġtwo Ġlanguages Ġour Ġdataset Ġconsists Ġof . ĠThis Ġprovides Ġus Ġwith Ġvector Ġrepresentations Ġof Ġwords , Ġallowing Ġto Ġmeasure Ġtheir Ġsemantic Ġsimilarity . ĠThen , Ġa Ġlinear Ġtransformation Ġmatrix Ġfrom Ġvectors Ġof Ġlanguage ĠA Ġto Ġvectors Ġof Ġlanguage ĠB Ġis Ġlearned , Ġusing Ġa Ġsmall Ġbilingual Ġdictionary Ġas Ġtraining Ġdata . ĠThis Ġmatrix Ġis Ġthen Ġemployed Ġto Ġ` project ' Ġword Ġand Ġdocument Ġrepresentations Ġfrom Ġsemantic Ġspace Ġof Ġlanguage ĠA Ġto Ġsemantic Ġspace Ġof Ġlanguage ĠB . ĠIt Ġallows Ġnot Ġonly Ġquite Ġaccurate Ġ` translation ' Ġof Ġwords , Ġbut Ġalso Ġof Ġdocument Ġ` sem antic Ġfingerprints ' Ġ( d ense Ġrepresentations Ġof Ġdocument Ġsemantics , Ġcalculated Ġas Ġan Ġaverage Ġof Ġthe Ġtrained Ġdistribution al Ġvectors Ġfor Ġall Ġthe Ġwords Ġin Ġdocument ). </s> This Ġapproach Ġis Ġevaluated Ġin Ġa Ġsetting , Ġwhere Ġthe Ġinput Ġis Ġa Ġcollection Ġof Ġdocuments Ġin Ġseveral Ġlanguages Ġand Ġsome Ġnumber Ġof Ġtopics Ġto Ġwhich Ġthese Ġdocuments Ġbelong Ġ( we Ġalso Ġhave Ġlarge Ġmon oling ual Ġcorpor a Ġto Ġtrain Ġdistribution al Ġmodels Ġon ). ĠFor Ġeach Ġdocument , Ġwe Ġare Ġgiven Ġits Ġlanguage , Ġbut Ġnot Ġits Ġtopic . ĠThe Ġtask Ġis Ġto Ġcluster Ġthis Ġcollection Ġso Ġthat Ġdocuments Ġbelonging Ġto Ġone Ġtopic Ġwere Ġclustered Ġtogether , Ġindependent Ġof Ġtheir Ġlanguage . ĠNote Ġthat Ġwe Ġare Ġinterested Ġin Ġclust ering Ġthe Ġcollection Ġas Ġa Ġwhole , Ġnot Ġeach Ġlanguage Ġseparately Ġ( which Ġis Ġtrivial ). </s> Our Ġevaluation Ġdata Ġconsists Ġof Ġcomparable Ġcorpor a Ġof ĠRussian Ġand ĠUkrainian Ġacademic Ġtexts . ĠOn Ġthis Ġmaterial , Ġwe Ġshow Ġthat Ġthe Ġ` trans lated Ġsemantic Ġfingerprints ' Ġmethod Ġrepresents Ġdocuments Ġin Ġdifferent Ġlanguages Ġprecisely Ġenough Ġto Ġallow Ġalmost Ġexact Ġclust ering Ġaccording Ġto Ġdocument Ġtopics , Ġwith Ġonly Ġ5 % Ġof Ġincorrect Ġassignments . ĠIt Ġsignificantly Ġoutper forms Ġboth Ġnaive Ġbag - of - words Ġbaseline Ġand Ġthe Ġnot - so - na ive Ġmethod Ġof Ġ` orth ographic Ġtranslation ' Ġbased Ġon ĠD amer au - Le ven sh tein Ġdistance , Ġeven Ġenriched Ġwith Ġdictionary Ġm appings . ĠAt Ġthe Ġsame Ġtime , Ġit Ġdoes Ġnot Ġrequire Ġlarge Ġparallel Ġcorpor a Ġor Ġa Ġready - made Ġstatistical Ġmachine Ġtranslation Ġmodel . </s> The Ġrest Ġof Ġthe Ġpaper Ġis Ġstructured Ġas Ġfollows . ĠIn ĠSection Ġ" Related ĠWork " Ġwe Ġdescribe Ġthe Ġfoundations Ġof Ġour Ġapproach Ġand Ġthe Ġrelated Ġwork . ĠSection Ġ" Ac ademic Ġtexts Ġas ĠCompar able ĠCorpor a " Ġintroduces Ġthe Ġemployed Ġcorpor a Ġand Ġthe Ġstory Ġbehind Ġthem . ĠSection Ġ" Learning Ġto ĠTrans late : ĠUkrainian - to - Russian Ġtransformations " Ġis Ġdedicated Ġto Ġlearning Ġthe Ġtransformation Ġmatrix , Ġand ĠSection Ġ" Exper iment ĠDesign Ġand ĠEvaluation " Ġdescribes Ġour Ġexperimental Ġsetting Ġand Ġevaluation Ġresults . ĠWe Ġdiscuss Ġthe Ġfindings Ġin ĠSection Ġ" Discussion " Ġand Ġconclude Ġin ĠSection Ġ" Conclusion Ġand ĠFuture ĠWork " Ġ, Ġalso Ġsuggesting Ġdirections Ġfor Ġfuture Ġwork . </s> Related ĠWork </s> Cl ust ering Ġmulti - ling ual Ġdocuments Ġhas Ġreceived Ġmuch Ġattention Ġin Ġnatural Ġlanguage Ġprocessing . ĠAmong Ġapproaches Ġnot Ġusing Ġsome Ġform Ġof Ġmachine Ġtranslation , Ġone Ġcan Ġmention ĠB IB REF 1 Ġ, Ġwho Ġessentially Ġemploy Ġa Ġbilingual Ġdictionary Ġto Ġbring Ġsome Ġwords Ġin Ġthe Ġdocuments Ġto Ġa Ġlanguage - independent Ġform Ġand Ġthen Ġto Ġperform Ġclust ering . ĠIn Ġthe Ġsection Ġ" Exper iment ĠDesign Ġand ĠEvaluation " Ġwe Ġshow Ġthat Ġour Ġapproach Ġbased Ġon Ġneural Ġembed d ings Ġsignificantly Ġoutper forms Ġtheir Ġreported Ġresults . </s> BI B REF 2 Ġproposed Ġtraining Ġjoint Ġmulti - ling ual Ġneural Ġembed ding Ġmodels . ĠThe oret ically , Ġthis Ġcan Ġbe Ġused Ġto Ġachieve Ġour Ġaim Ġof Ġlanguage - independent Ġsemantic Ġrepresentations Ġfor Ġdocuments . ĠUnfortunately , Ġit Ġdemands Ġa Ġlarge Ġword - aligned Ġparallel Ġcorpus . ĠThis Ġis Ġnot Ġthe Ġcase Ġwith Ġthe Ġmore Ġrecent ĠTrans - gram Ġapproach Ġintroduced Ġin ĠB IB REF 3 Ġ, Ġalso Ġable Ġto Ġlearn Ġmulti - ling ual Ġmodels . ĠHowever , Ġit Ġstill Ġneeds Ġsentence - aligned Ġcorpor a Ġto Ġtrain Ġon Ġ( in Ġthe Ġsize Ġof Ġmillions Ġof Ġpaired Ġsentences ). ĠLarge Ġparallel Ġcorpor a Ġ( whether Ġword - Ġor Ġsentence - aligned ) Ġare Ġoften Ġa Ġscarce Ġresource , Ġespecially Ġin Ġthe Ġcase Ġof Ġunder - represented Ġlanguages . </s> The Ġapproach Ġdescribed Ġin Ġthis Ġpaper Ġtakes Ġas Ġan Ġinput Ġonly Ġcomparable Ġmon oling ual Ġcorpor a Ġand Ġbilingual Ġdiction aries Ġin Ġthe Ġsize Ġof Ġseveral Ġthousand Ġword Ġpairs . ĠSuch Ġresources Ġare Ġmuch Ġeasier Ġto Ġfind Ġand Ġevaluate . ĠWe Ġemploy Ġthe Ġidea Ġof Ġlearning Ġa Ġlinear Ġtransformation Ġmatrix Ġto Ġmap Ġor Ġproject Ġword Ġembed d ings Ġfrom Ġthe Ġsemantic Ġspace Ġof Ġone Ġlanguage Ġto Ġthat Ġof Ġanother . ĠThis Ġidea Ġwas Ġfirst Ġproposed Ġin ĠB IB REF 4 Ġ, Ġwho Ġapplied Ġit Ġto Ġlex ical Ġtranslation Ġbetween ĠEnglish , ĠSpanish , ĠCzech Ġand ĠVietnamese . ĠWe Ġextend Ġit Ġfrom Ġcontinuous Ġrepresentations Ġof Ġsingle Ġwords Ġor Ġcoll ocations Ġto Ġ` sem antic Ġfingerprints ' Ġof Ġdocuments Ġas Ġa Ġwhole . </s> Ac ademic Ġtexts Ġas ĠCompar able ĠCorpor a </s> The ĠRussian Ġand ĠUkrainian Ġlanguages Ġare Ġmainly Ġspoken Ġin ĠRussian ĠFederation Ġand Ġthe ĠUkraine Ġand Ġbelong Ġto Ġthe ĠEast - Sl av ic Ġgroup Ġof Ġthe ĠIndo - European Ġlanguage Ġfamily . ĠThey Ġshare Ġmany Ġcommon Ġmorph os ynt actic Ġfeatures : Ġboth Ġare ĠSV O Ġlanguages Ġwith Ġfree Ġword Ġorder Ġand Ġrich Ġmorphology , Ġboth Ġuse Ġthe ĠCyr ill ic Ġalphabet Ġand Ġshare Ġmany Ġcommon Ġcogn ates . </s> Both ĠRussia Ġand Ġthe ĠUkraine Ġhave Ġcommon Ġacademic Ġtradition Ġthat Ġmakes Ġit Ġeasier Ġto Ġcollect Ġcorpor a , Ġwhich Ġare Ġcomparable Ġin Ġterms Ġof Ġboth Ġgenre Ġand Ġstrictly Ġdefined Ġacademic Ġfields . ĠWe Ġwork Ġwith Ġsuch Ġa Ġcorpus Ġof ĠRussian Ġand ĠUkrainian Ġacademic Ġtexts , Ġinitially Ġcollected Ġfor Ġthe Ġpurposes Ġof Ġcross - ling ual Ġplagiar ism Ġdetection . ĠThis Ġdata Ġis Ġavailable Ġonline Ġthrough Ġa Ġnumber Ġof Ġlibrary Ġservices , Ġbut Ġunfortunately Ġcannot Ġbe Ġrep ublished Ġdue Ġto Ġcopyright Ġlimitations . </s> The ĠUkrainian Ġsub cor p us Ġcontains Ġabout Ġ60 Ġthousand Ġextended Ġsumm aries Ġ( Russian Ġand ĠUkrainian Ġr ussian ` Ð° Ð² ÑĤ Ð¾ ÑĢ Ðµ Ñ Ħ Ðµ ÑĢ Ð° ÑĤ ', Ġ` av t ore fer at ') Ġof Ġthe ses Ġsubmitted Ġbetween Ġ1998 Ġand Ġ2011 . ĠThe ĠRussian Ġsub cor p us Ġis Ġsmaller Ġin Ġthe Ġnumber Ġof Ġdocuments Ġ( about Ġ16 Ġthousand , Ġapproximately Ġthe Ġsame Ġtime Ġperiod ), Ġbut Ġthe Ġdocuments Ġare Ġfull Ġtexts Ġof Ġthe ses , Ġthus Ġthe Ġtotal Ġvolume Ġof Ġthe ĠRussian Ġsub cor p us Ġis Ġnotably Ġlarger : Ġ8 30 Ġmillion Ġtokens Ġversus Ġ250 Ġmillion Ġtokens Ġin Ġthe ĠUkrainian Ġone . ĠGenerally , Ġthe Ġtexts Ġbelong Ġto Ġone Ġgenre Ġthat Ġcan Ġbe Ġdefined Ġas Ġpost - Soviet Ġexp ository Ġacademic Ġprose , Ġsubmitted Ġfor Ġacademic Ġdegree Ġaward Ġprocess . </s> The Ġdocuments Ġwere Ġconverted Ġto Ġplain Ġtext Ġfiles Ġfrom ĠMS ĠWord Ġformat Ġin Ġthe Ġcase Ġof Ġthe ĠUkrainian Ġsub cor p us Ġand Ġmainly Ġfrom ĠOC Red ĠPDF Ġfiles Ġin Ġthe Ġcase Ġof Ġthe ĠRussian Ġsub cor p us . ĠBecause Ġof Ġthis , Ġthe ĠRussian Ġdocuments Ġoften Ġsuffer Ġfrom ĠO CR Ġartifacts , Ġsuch Ġas Ġwords Ġsplit Ġwith Ġline Ġbreaks , Ġincorrectly Ġrecognized Ġcharacters Ġand Ġso Ġon . ĠHowever , Ġit Ġdoes Ġnot Ġinfluence Ġthe Ġresulting Ġmodel Ġmuch , Ġas Ġwe Ġshow Ġbelow . </s> Both ĠUkrainian Ġand ĠRussian Ġdocuments Ġcome Ġwith Ġmeta Ġdata Ġallowing Ġto Ġseparate Ġthem Ġinto Ġacademic Ġfields , Ġwith Ġeconomics , Ġmedicine Ġand Ġlaw Ġbeing Ġmost Ġfrequent Ġtopics Ġfor Ġthe ĠUkrainian Ġdata Ġand Ġeconomics , Ġhistory Ġand Ġped ag ogy Ġdominating Ġthe ĠRussian Ġdata . </s> For Ġevaluation , Ġ3 Ġtopics Ġwere Ġused , Ġdistant Ġenough Ġfrom Ġeach Ġother Ġand Ġabundantly Ġpresented Ġin Ġboth Ġsub cor pora : Ġeconomics , Ġlaw Ġand Ġhistory . ĠWe Ġrandomly Ġselected Ġ100 Ġtexts Ġin Ġeach Ġlanguage Ġfor Ġeach Ġtopic . ĠAs Ġan Ġaverage Ġlength Ġof ĠRussian Ġtexts Ġis Ġsignificantly Ġhigher Ġ( them Ġbeing Ġfull Ġthe ses ), Ġwe Ġcropped Ġthem , Ġleaving Ġonly Ġthe Ġfirst Ġ5 Ġthousand Ġwords , Ġto Ġmimic Ġthe Ġsize Ġof Ġthe ĠUkrainian Ġsumm aries . ĠThese Ġ600 Ġdocuments Ġin Ġ3 Ġclasses Ġare Ġused Ġas Ġa Ġtest Ġset Ġ( see ĠSection Ġ" Exper iment ĠDesign Ġand ĠEvaluation " Ġfor Ġthe Ġdescription Ġof Ġthe Ġconducted Ġexperiments ). </s> The Ġcorpor a Ġ( including Ġtest Ġset ) Ġwere ĠPo S - tag ged . ĠEach Ġword Ġwas Ġreplaced Ġwith Ġits Ġle mma Ġfollowed Ġby Ġa ĠPo S - tag Ġ( ` r ussian Ð´ Ð¸ Ñģ Ñģ Ðµ ÑĢ ÑĤ Ð° Ñ Ĩ Ð¸ Ñı _ S ', Ġ` r ussian Ð´ Ð¸ Ñģ Ñģ Ðµ ÑĢ ÑĤ Ð° Ñ Ĩ i Ñı _ N '). ĠFunctional Ġparts Ġof Ġspeech Ġ( con jun ctions , Ġpronouns , Ġpre pos itions , Ġetc .) Ġand Ġnumer als Ġwere Ġremoved Ġfrom Ġthe Ġtexts . </s> Learning Ġto ĠTrans late : ĠUkrainian - to - Russian Ġtransformations </s> As Ġalready Ġstated , Ġour Ġmain Ġproposal Ġis Ġusing Ġneural Ġembed ding Ġmodels Ġto Ġ` project ' Ġdocuments Ġin Ġone Ġlanguage Ġinto Ġthe Ġsemantic Ġspace Ġof Ġanother Ġlanguage . ĠFor Ġthis , Ġwe Ġfirst Ġtrained Ġa ĠContinuous ĠBag - of - Words Ġ( CB OW ) Ġand Ġa ĠContinuous ĠSkip G ram Ġmodel ĠB IB REF 0 Ġfor Ġeach Ġof Ġour Ġmon oling ual Ġsub cor pora . ĠThe Ġmodels Ġwere Ġtrained Ġwith Ġidentical Ġhyper param eters : Ġvector Ġsize Ġof Ġ300 Ġcomponents , Ġsymm etric Ġwindow Ġof Ġ2 Ġwords , Ġnegative Ġsampling Ġwith Ġ10 Ġsamples , Ġ5 Ġiterations Ġover Ġthe Ġcorpus , Ġno Ġdown - sam pling . ĠThe Ġonly Ġlanguage - dependent Ġdifference Ġwas Ġthat Ġfor Ġthe ĠUkrainian Ġmodel Ġwe Ġignored Ġwords Ġwith Ġthe Ġcorpus Ġfrequency Ġless Ġthan Ġ10 Ġand Ġfor Ġthe ĠRussian Ġmodel Ġthis Ġthreshold Ġwas Ġset Ġto Ġ15 Ġ( as Ġthe ĠRussian Ġcorpus Ġis Ġ3 Ġtimes Ġlarger ). ĠAll Ġin Ġall , Ġthe Ġfinal ĠUkrainian Ġmodel Ġrecognizes Ġ429 Ġ215 Ġwords Ġand Ġthe ĠRussian Ġone Ġ271 Ġ720 Ġwords . ĠTraining Ġwas Ġperformed Ġusing ĠCB OW Ġand ĠSkip G ram Ġimplementation Ġin ĠG ens im Ġlibrary ĠB IB REF 7 Ġ. </s> After Ġthe Ġmodels Ġwere Ġtrained , Ġwe Ġfollowed Ġthe Ġpath Ġoutlined Ġin ĠB IB REF 4 Ġto Ġlearn Ġa Ġlinear Ġtransformation Ġmatrix Ġfrom ĠUkrainian Ġto ĠRussian . ĠFirst , Ġwe Ġextracted Ġall Ġnoun Ġpairs Ġfrom ĠRussian - Uk rain ian Ġbilingual Ġdictionary ĠB IB REF 8 Ġ, Ġwith Ġthe Ġconstraint Ġthat Ġtheir Ġfrequency Ġin Ġour Ġcorpor a Ġwas Ġabove Ġthe Ġalready Ġmentioned Ġthresholds Ġ15 Ġand Ġ10 Ġfor ĠRussian Ġand ĠUkrainian Ġwords Ġcorrespond ingly . ĠThat Ġmade Ġit Ġa Ġlist Ġof Ġabout Ġ5 Ġthousand Ġpairs Ġof Ġnoun s Ġbeing Ġtranslations Ġof Ġeach Ġother . </s> For Ġall Ġthese Ġwords , Ġtheir Ġvectors Ġwere Ġfound Ġin Ġthe Ġmodels Ġcorresponding Ġto Ġthe Ġwords ' Ġlanguages . ĠIt Ġprovided Ġus Ġwith Ġa Ġmatrix Ġof Ġ5 Ġthousand Ġof Ġ300 - dimensional ĠUkrainian Ġvectors Ġand Ġthe Ġmatrix Ġof Ġcorresponding Ġ5 Ġthousand Ġof Ġ300 - dimensional ĠRussian Ġvectors . ĠThis Ġdata Ġserved Ġas Ġa Ġtraining Ġset Ġto Ġlearn Ġan Ġoptimal Ġtransformation Ġmatrix . ĠThe Ġlatter Ġis Ġactually Ġa Ġ300 x 301 Ġmatrix Ġof Ġcoefficients , Ġsuch Ġthat Ġwhen Ġthe Ġinitial ĠUkrainian Ġmatrix Ġis Ġmultiplied Ġby Ġthis Ġtransformation Ġmatrix , Ġthe Ġresult Ġis Ġmaxim ally Ġclose Ġto Ġthe Ġcorresponding ĠRussian Ġmatrix . ĠThis Ġtransformation Ġmatrix Ġhas Ġ301 Ġ( not Ġ300 ) Ġcolumns , Ġbecause Ġwe Ġadd Ġone Ġcomponent Ġequal Ġto Ġ1 Ġto Ġeach Ġvector , Ġas Ġa Ġbias Ġterm . </s> Produ cing Ġthe Ġtransformation Ġmatrix Ġis Ġa Ġlinear Ġregression Ġproblem : Ġthe Ġinput Ġis Ġ301 Ġcomponents Ġof ĠUkrainian Ġvectors Ġ( including Ġthe Ġbias Ġterm ) Ġand Ġthe Ġoutput Ġis Ġ300 Ġcomponents Ġof ĠRussian Ġvectors . ĠAs Ġwe Ġneed Ġ300 Ġvalues Ġas Ġan Ġoutput , Ġthere Ġare Ġactually Ġ300 Ġlinear Ġregression Ġproblems Ġand Ġthat 's Ġwhy Ġthe Ġresulting Ġmatrix Ġsize Ġis Ġ300 x 301 Ġ( 301 Ġweights Ġfor Ġeach Ġof Ġ300 Ġcomponents ). </s> There Ġare Ġtwo Ġmain Ġways Ġto Ġsolve Ġa Ġlinear Ġregression Ġproblem : Ġone Ġcan Ġeither Ġlearn Ġthe Ġoptimal Ġweights Ġin Ġan Ġiter ative Ġway Ġusing Ġsome Ġvariant Ġof Ġgradient Ġdescent , Ġor Ġone Ġcan Ġsolve Ġit Ġnumer ically Ġwithout Ġiteration , Ġusing Ġnormal Ġequation . ĠFor ĠEnglish Ġand ĠSpanish , ĠB IB REF 4 Ġused Ġsto ch astic Ġgradient Ġdescent . ĠHowever , Ġnormal Ġequation Ġis Ġactually Ġless Ġerror - prone Ġand Ġis Ġguaranteed Ġto Ġfind Ġthe Ġglobal Ġoptimum . ĠIts Ġonly Ġdisadvantage Ġis Ġthat Ġit Ġbecomes Ġvery Ġcomput ationally Ġexpensive Ġwhen Ġthe Ġnumber Ġof Ġfeatures Ġis Ġlarge Ġ( th ousands Ġand Ġmore ). ĠHowever , Ġin Ġour Ġcase Ġthe Ġnumber Ġof Ġfeatures Ġis Ġonly Ġ301 , Ġso Ġcomputational Ġcomplexity Ġis Ġnot Ġan Ġissue . </s> Thus , Ġwe Ġuse Ġnormal Ġequation Ġto Ġfind Ġthe Ġoptimal Ġtransformation Ġmatrix . ĠThe Ġalgebra ic Ġsolution Ġto Ġeach Ġof Ġ300 Ġnormal Ġequations Ġ( one Ġfor Ġeach Ġvector Ġcomponent Ġ$ i $ Ġ) Ġis Ġshown Ġin Ġthe ĠEqu ation Ġ3 Ġ: </s> $$ \ beta Ġ_ i Ġ= Ġ( \ text bf Ġ{ X } ^ \ inter cal Ġ* Ġ\ text bf Ġ{ X }) ^{ - 1 } Ġ* Ġ\ text bf Ġ{ X } ^ \ inter cal Ġ* Ġy _ i $$ Ġ Ġ Ġ( E q . Ġ3 ) </s> where Ġ$\ text bf Ġ{ X } $ Ġis Ġthe Ġmatrix Ġof Ġ5 Ġthousand ĠUkrainian Ġword Ġvectors Ġ( input ), Ġ$ y _ i $ Ġis Ġthe Ġvector Ġof Ġthe Ġ$ i $ Ġth Ġcomponents Ġof Ġ5 Ġthousand Ġcorresponding ĠRussian Ġwords Ġ( correct Ġpredictions ), Ġand Ġ$\ beta Ġ_ i $ Ġis Ġour Ġaim : Ġthe Ġvector Ġof Ġ301 Ġoptimal Ġcoefficients Ġwhich Ġtransform Ġthe ĠUkrainian Ġvectors Ġinto Ġthe Ġ$ i $ Ġth Ġcomponent Ġof Ġthe ĠRussian Ġvectors . </s> After Ġsolving Ġsuch Ġnormal Ġequations Ġfor Ġall Ġthe Ġ300 Ġcomponents Ġ$ i $ Ġ, Ġwe Ġhave Ġthe Ġ300 x 301 Ġlinear Ġtransformation Ġmatrix Ġwhich Ġfits Ġthe Ġdata Ġbest . </s> This Ġmatrix Ġbasically Ġmaps Ġthe ĠUkrainian Ġvectors Ġinto Ġthe ĠRussian Ġones . ĠIt Ġis Ġbased Ġon Ġthe Ġassumption Ġthat Ġthe Ġrelations Ġbetween Ġsemantic Ġconcepts Ġin Ġdifferent Ġlanguages Ġare Ġin Ġfact Ġvery Ġsimilar Ġ( stud ents Ġare Ġclose Ġto Ġteachers , Ġwhile Ġpirates Ġare Ġclose Ġto Ġc ors airs , Ġand Ġso Ġon ). ĠIn Ġcontinuous Ġdistribution al Ġmodels Ġwhich Ġstrive Ġto Ġrepresent Ġthese Ġsemantic Ġspaces , Ġmutual Ġ` ge omet rical ' Ġrelations Ġbetween Ġvectors Ġrepresenting Ġparticular Ġwords Ġare Ġalso Ġsimilar Ġacross Ġmodels Ġ( if Ġthey Ġare Ġtrained Ġon Ġcomparable Ġcorpor a ), Ġbut Ġthe Ġexact Ġvectors Ġfor Ġwords Ġden oting Ġone Ġand Ġthe Ġsame Ġnotion Ġare Ġdifferent . ĠThis Ġis Ġbecause Ġthe Ġmodels Ġthemselves Ġare Ġsto ch astic Ġand Ġthe Ġparticular Ġvalues Ġof Ġvectors Ġ( un like Ġtheir Ġpositions Ġin Ġrelation Ġto Ġeach Ġother ) Ġdepend Ġa Ġlot Ġon Ġtechnical Ġfactors , Ġincluding Ġthe Ġrandom Ġseed Ġused Ġto Ġinitialize Ġvectors Ġprior Ġto Ġtraining . ĠIn Ġorder Ġto Ġmigrate Ġfrom Ġa Ġmodel ĠA Ġto Ġanother Ġmodel ĠB , Ġone Ġhas Ġto Ġ` rot ate Ġand Ġscale ' ĠA Ġvectors Ġin Ġa Ġuniform Ġlinear Ġway . ĠTo Ġlearn Ġthe Ġoptimal Ġtransformation Ġmatrix Ġmeans Ġto Ġfind Ġout Ġthe Ġexact Ġdirections Ġof Ġrotating Ġand Ġscaling , Ġwhich Ġminimize Ġprediction Ġerrors . </s> L ingu istically Ġspeaking , Ġonce Ġwe Ġlearned Ġthe Ġtransformation Ġmatrix , Ġwe Ġcan Ġpredict Ġwhat Ġa ĠRussian Ġvector Ġwould Ġmost Ġprobably Ġbe , Ġgiven Ġa ĠUkrainian Ġone . ĠThis Ġessentially Ġmeans Ġwe Ġare Ġable Ġto Ġ` trans late ' ĠUkrainian Ġwords Ġinto ĠRussian , Ġby Ġcalculating Ġthe Ġword Ġin Ġthe ĠRussian Ġmodel Ġwith Ġthe Ġvector Ġclosest Ġto Ġthe Ġpredicted Ġone . </s> We Ġhad Ġto Ġchoose Ġbetween ĠCB OW Ġor ĠContinuous ĠSkip G ram Ġmodels Ġto Ġuse Ġwhen Ġlearning Ġthe Ġtransformation Ġmatrix . ĠAlso , Ġthere Ġwas Ġa Ġquestion Ġof Ġwhether Ġto Ġemploy Ġregular ized Ġor Ġstandard Ġnormal Ġequations . ĠRegular ization Ġis Ġan Ġattempt Ġto Ġavoid Ġover - fitting Ġby Ġtrying Ġto Ġsomehow Ġdecrease Ġthe Ġvalues Ġof Ġlearned Ġweights . ĠThe Ġregular ized Ġnormal Ġequation Ġis Ġshown Ġin Ġ4 Ġ: </s> $$ \ beta Ġ_ i Ġ= Ġ( \ text bf Ġ{ X } ^ \ inter cal Ġ* Ġ\ text bf Ġ{ X } Ġ+ Ġ\ lambda Ġ* ĠL ) ^{ - 1 } Ġ* Ġ\ text bf Ġ{ X } ^ \ inter cal Ġ* Ġy _ i $$ Ġ Ġ Ġ( E q . Ġ4 ) </s> Comp aring Ġto Ġ3 Ġ, Ġit Ġadds Ġthe Ġterm Ġ$\ lambda Ġ* ĠL $ Ġ, Ġwhere Ġ$ L $ Ġis Ġthe Ġidentity Ġmatrix Ġof Ġthe Ġsize Ġequal Ġto Ġthe Ġnumber Ġof Ġfeatures , Ġwith Ġ0 Ġat Ġthe Ġtop Ġleft Ġcell , Ġand Ġ$\ lambda Ġ$ Ġis Ġa Ġreal Ġnumber Ġused Ġto Ġtune Ġthe Ġinfluence Ġof Ġregular ization Ġterm Ġ( if Ġ$\ lambda Ġ= Ġ0 $ Ġ, Ġthere Ġis Ġno Ġregular ization ). </s> To Ġtest Ġall Ġthe Ġpossible Ġcombinations Ġof Ġparameters , Ġwe Ġdivided Ġthe Ġbilingual Ġdictionary Ġinto Ġ4 500 Ġnoun Ġpairs Ġused Ġas Ġa Ġtraining Ġset Ġand Ġ500 Ġnoun Ġpairs Ġused Ġas Ġa Ġtest Ġset . ĠWe Ġthen Ġlearned Ġtransformation Ġmat rices Ġon Ġthe Ġtraining Ġset Ġusing Ġboth Ġtraining Ġalgorithms Ġ( CB OW Ġand ĠSkip G ram ) Ġand Ġseveral Ġvalues Ġof Ġregular ization Ġ$\ lambda Ġ$ Ġfrom Ġ0 Ġto Ġ5 , Ġwith Ġa Ġstep Ġof Ġ0 . 5 . ĠThe Ġresulting Ġmat rices Ġwere Ġapplied Ġto Ġthe ĠUkrainian Ġvectors Ġfrom Ġthe Ġtest Ġset Ġand Ġthe Ġcorresponding ĠRussian Ġ` trans lations ' Ġwere Ġcalculated . ĠThe Ġratio Ġof Ġcorrect Ġ` trans lations ' Ġ( mat ches ) Ġwas Ġused Ġas Ġan Ġevaluation Ġmeasure . ĠIt Ġcame Ġout Ġthat Ġregular ization Ġonly Ġworsened Ġthe Ġresults Ġfor Ġboth Ġalgorithms , Ġso Ġin Ġthe ĠTable Ġ1 Ġwe Ġreport Ġthe Ġresults Ġwithout Ġregular ization . </s> For Ġreference , Ġwe Ġalso Ġreport Ġthe Ġaccuracy Ġof Ġ` qu azi - translation ' Ġvia ĠD amer au - Le ven sh tein Ġedit Ġdistance ĠB IB REF 9 Ġ, Ġas Ġa Ġsort Ġof Ġa Ġbaseline . ĠAs Ġalready Ġstated , Ġthe Ġtwo Ġlanguages Ġshare Ġmany Ġcogn ates , Ġand Ġa Ġlot Ġof ĠUkrainian Ġwords Ġcan Ġbe Ġorth ographically Ġtransformed Ġinto Ġtheir ĠRussian Ġtranslations Ġ( and Ġvice Ġversa ) Ġby Ġone Ġor Ġtwo Ġcharacter Ġreplacements . ĠThus , Ġwe Ġextracted Ġ50 , 000 Ġmost Ġfrequent Ġnoun s Ġfrom Ġour ĠRussian Ġcorpor a ; Ġthen Ġfor Ġeach ĠUkrainian Ġnoun Ġin Ġthe Ġbilingual Ġdictionary Ġwe Ġfound Ġthe Ġclosest ĠRussian Ġnoun Ġ( or Ġ5 Ġclosest Ġnoun s Ġfor Ġ@ 5 Ġmetric ) Ġby Ġedit Ġdistance Ġand Ġcalculated Ġhow Ġoften Ġit Ġturned Ġout Ġto Ġbe Ġthe Ġcorrect Ġtranslation . ĠAs Ġthe ĠTable Ġ1 Ġshows , Ġnotwithstanding Ġthe Ġorth ographic Ġsimilarity Ġof Ġthe Ġtwo Ġlanguages , ĠCB OW Ġconsistently Ġoutper forms Ġthis Ġapproach Ġeven Ġon Ġthe Ġtest Ġset . ĠOn Ġthe Ġtraining Ġset , Ġits Ġsuperiority Ġis Ġeven Ġmore Ġobvious .