Book, electronics, beauty, music, IMDB, Yelp, cell phone, baby, DVDs, kitchen
ABSTRACTIVE
we use set 1 of the source domain as the only source with sentiment label information during training, and we evaluate the trained model on set 1 of the target domain, Book (BK), Electronics (E), Beauty (BT), and Music (M)
EXTRACTIVE
FLOAT SELECTED: Table 1: Summary of datasets. Most previous works BIBREF0 , BIBREF1 , BIBREF6 , BIBREF7 , BIBREF29 carried out experiments on the Amazon benchmark released by Blitzer et al. ( BIBREF0 ). The dataset contains 4 different domains: Book (B), DVDs (D), Electronics (E), and Kitchen (K). Following their experimental settings, we consider the binary classification task to predict whether a review is positive or negative on the target domain. Each domain consists of 1000 positive and 1000 negative reviews respectively. We also allow 4000 unlabeled reviews to be used for both the source and the target domains, of which the positive and negative reviews are balanced as well, following the settings in previous works. We construct 12 cross-domain sentiment classification tasks and split the labeled data in each domain into a training set of 1600 reviews and a test set of 400 reviews. The classifier is trained on the training set of the source domain and is evaluated on the test set of the target domain. The comparison results are shown in Table TABREF37 .
Small-scale datasets: Our new dataset was derived from the large-scale Amazon datasets released by McAuley et al. ( BIBREF24 ). It contains four domains: Book (BK), Electronics (E), Beauty (BT), and Music (M). Each domain contains two datasets. Set 1 contains 6000 instances with exactly balanced class labels, and set 2 contains 6000 instances that are randomly sampled from the large dataset, preserving the original label distribution, which we believe better reflects the label distribution in real life. The examples in these two sets do not overlap. Detailed statistics of the generated datasets are given in Table TABREF9 . In all our experiments on the small-scale datasets, we use set 1 of the source domain as the only source with sentiment label information during training, and we evaluate the trained model on set 1 of the target domain. Since we cannot control the label distribution of unlabeled data during training, we consider two different settings:
1809.00530
Introduction </s> In Ġpractice , Ġit Ġis Ġoften Ġdifficult Ġand Ġcostly Ġto Ġannot ate Ġsufficient Ġtraining Ġdata Ġfor Ġdiverse Ġapplication Ġdomains Ġon - the - fly . ĠWe Ġmay Ġhave Ġsufficient Ġlabeled Ġdata Ġin Ġan Ġexisting Ġdomain Ġ( called Ġthe Ġsource Ġdomain ), Ġbut Ġvery Ġfew Ġor Ġno Ġlabeled Ġdata Ġin Ġa Ġnew Ġdomain Ġ( called Ġthe Ġtarget Ġdomain ). ĠThis Ġissue Ġhas Ġmotivated Ġresearch Ġon Ġcross - domain Ġsentiment Ġclassification , Ġwhere Ġknowledge Ġin Ġthe Ġsource Ġdomain Ġis Ġtransferred Ġto Ġthe Ġtarget Ġdomain Ġin Ġorder Ġto Ġalleviate Ġthe Ġrequired Ġlabeling Ġeffort . </s> One Ġkey Ġchallenge Ġof Ġdomain Ġadaptation Ġis Ġthat Ġdata Ġin Ġthe Ġsource Ġand Ġtarget Ġdomains Ġare Ġdrawn Ġfrom Ġdifferent Ġdistributions . ĠThus , Ġadaptation Ġperformance Ġwill Ġdecline Ġwith Ġan Ġincrease Ġin Ġdistribution Ġdifference . ĠSpecifically , Ġin Ġsentiment Ġanalysis , Ġreviews Ġof Ġdifferent Ġproducts Ġhave Ġdifferent Ġvocabulary . ĠFor Ġinstance , Ġrestaurants Ġreviews Ġwould Ġcontain Ġopinion Ġwords Ġsuch Ġas ĠâĢ ľ t ender âĢ Ŀ , ĠâĢ ľ t asty âĢ Ŀ , Ġor ĠâĢ ľ under cooked âĢ Ŀ Ġand Ġmovie Ġreviews Ġwould Ġcontain ĠâĢ ľ th r illing âĢ Ŀ , ĠâĢ ľ hor r ific âĢ Ŀ , Ġor ĠâĢ ľ h ilar ious âĢ Ŀ . ĠThe Ġintersection Ġbetween Ġthese Ġtwo Ġsets Ġof Ġopinion Ġwords Ġcould Ġbe Ġsmall Ġwhich Ġmakes Ġdomain Ġadaptation Ġdifficult . </s> Several Ġtechniques Ġhave Ġbeen Ġproposed Ġfor Ġaddressing Ġthe Ġproblem Ġof Ġdomain Ġshifting . ĠThe Ġaim Ġis Ġto Ġbridge Ġthe Ġsource Ġand Ġtarget Ġdomains Ġby Ġlearning Ġdomain - inv ari ant Ġfeature Ġrepresentations Ġso Ġthat Ġa Ġclass ifier Ġtrained Ġon Ġa Ġsource Ġdomain Ġcan Ġbe Ġadapted Ġto Ġanother Ġtarget Ġdomain . ĠIn Ġcross - domain Ġsentiment Ġclassification , Ġmany Ġworks ĠB IB REF 0 Ġ, ĠB IB REF 1 Ġ, ĠB IB REF 2 Ġ, ĠB IB REF 3 Ġ, ĠB IB REF 4 Ġutilize Ġa Ġkey Ġintuition Ġthat Ġdomain - specific Ġfeatures Ġcould Ġbe Ġaligned Ġwith Ġthe Ġhelp Ġof Ġdomain - inv ari ant Ġfeatures Ġ( p ivot Ġfeatures ). ĠFor Ġinstance , ĠâĢ ľ h ilar ious âĢ Ŀ Ġand ĠâĢ ľ t asty âĢ Ŀ Ġcould Ġbe Ġaligned Ġas Ġboth Ġof Ġthem Ġare Ġrelevant Ġto ĠâĢ ľ good âĢ Ŀ . </s> Despite Ġtheir Ġpromising Ġresults , Ġthese Ġworks Ġshare Ġtwo Ġmajor Ġlimitations . ĠFirst , Ġthey Ġhighly Ġdepend Ġon Ġthe Ġhe uristic Ġselection Ġof Ġpivot Ġfeatures , Ġwhich Ġmay Ġbe Ġsensitive Ġto Ġdifferent Ġapplications . ĠThus Ġthe Ġlearned Ġnew Ġrepresentations Ġmay Ġnot Ġeffectively Ġreduce Ġthe Ġdomain Ġdifference . ĠFurthermore , Ġthese Ġworks Ġonly Ġutilize Ġthe Ġunl abel ed Ġtarget Ġdata Ġfor Ġrepresentation Ġlearning Ġwhile Ġthe Ġsentiment Ġclass ifier Ġwas Ġsolely Ġtrained Ġon Ġthe Ġsource Ġdomain . ĠThere Ġhave Ġnot Ġbeen Ġmany Ġstudies Ġon Ġexploiting Ġunl abel ed Ġtarget Ġdata Ġfor Ġrefining Ġthe Ġclass ifier , Ġeven Ġthough Ġit Ġmay Ġcontain Ġbeneficial Ġinformation . ĠHow Ġto Ġeffectively Ġleverage Ġunl abel ed Ġtarget Ġdata Ġstill Ġremains Ġan Ġimportant Ġchallenge Ġfor Ġdomain Ġadaptation . </s> In Ġthis Ġwork , Ġwe Ġargue Ġthat Ġthe Ġinformation Ġfrom Ġunl abel ed Ġtarget Ġdata Ġis Ġbeneficial Ġfor Ġdomain Ġadaptation Ġand Ġwe Ġpropose Ġa Ġnovel ĠDomain ĠAdapt ive ĠSemi - super vised Ġlearning Ġframework Ġ( D AS ) Ġto Ġbetter Ġexploit Ġit . ĠOur Ġmain Ġintuition Ġis Ġto Ġtreat Ġthe Ġproblem Ġas Ġa Ġsemi - super vised Ġlearning Ġtask Ġby Ġconsidering Ġtarget Ġinstances Ġas Ġunl abel ed Ġdata , Ġassuming Ġthe Ġdomain Ġdistance Ġcan Ġbe Ġeffectively Ġreduced Ġthrough Ġdomain - inv ari ant Ġrepresentation Ġlearning . ĠSpecifically , Ġthe Ġproposed Ġapproach Ġjointly Ġperforms Ġfeature Ġadaptation Ġand Ġsemi - super vised Ġlearning Ġin Ġa Ġmulti - task Ġlearning Ġsetting . ĠFor Ġfeature Ġadaptation , Ġit Ġexplicitly Ġminim izes Ġthe Ġdistance Ġbetween Ġthe Ġencoded Ġrepresentations Ġof Ġthe Ġtwo Ġdomains . ĠOn Ġthis Ġbasis , Ġtwo Ġsemi - super vised Ġregular izations ĠâĢĵ Ġentropy Ġminim ization Ġand Ġself - ense mble Ġboot stra pping ĠâĢĵ Ġare Ġjointly Ġemployed Ġto Ġexploit Ġunl abel ed Ġtarget Ġdata Ġfor Ġclass ifier Ġrefinement . </s> We Ġevaluate Ġour Ġmethod Ġrig orously Ġunder Ġmultiple Ġexperimental Ġsettings Ġby Ġtaking Ġlabel Ġdistribution Ġand Ġcorpus Ġsize Ġinto Ġconsideration . ĠThe Ġresults Ġshow Ġthat Ġour Ġmodel Ġis Ġable Ġto Ġobtain Ġsignificant Ġimprovements Ġover Ġstrong Ġbas elines . ĠWe Ġalso Ġdemonstrate Ġthrough Ġa Ġseries Ġof Ġanalysis Ġthat Ġthe Ġproposed Ġmethod Ġbenefits Ġgreatly Ġfrom Ġincorporating Ġunl abel ed Ġtarget Ġdata Ġvia Ġsemi - super vised Ġlearning , Ġwhich Ġis Ġconsistent Ġwith Ġour Ġmotivation . ĠOur Ġdatasets Ġand Ġsource Ġcode Ġcan Ġbe Ġobtained Ġfrom Ġhttps :// github . com / ru idan / D AS . </s> Related ĠWork </s> Domain ĠAdapt ation : ĠThe Ġmajority Ġof Ġfeature Ġadaptation Ġmethods Ġfor Ġsentiment Ġanalysis Ġrely Ġon Ġa Ġkey Ġintuition Ġthat Ġeven Ġthough Ġcertain Ġopinion Ġwords Ġare Ġcompletely Ġdistinct Ġfor Ġeach Ġdomain , Ġthey Ġcan Ġbe Ġaligned Ġif Ġthey Ġhave Ġhigh Ġcorrelation Ġwith Ġsome Ġdomain - inv ari ant Ġopinion Ġwords Ġ( p ivot Ġwords ) Ġsuch Ġas ĠâĢ ľ ex cellent âĢ Ŀ Ġor ĠâĢ ľ ter rible âĢ Ŀ . ĠBl itzer Ġet Ġal . Ġ( ĠB IB REF 0 Ġ) Ġproposed Ġa Ġmethod Ġbased Ġon Ġstructural Ġcorrespondence Ġlearning Ġ( S CL ), Ġwhich Ġuses Ġpivot Ġfeature Ġprediction Ġto Ġinduce Ġa Ġprojected Ġfeature Ġspace Ġthat Ġworks Ġwell Ġfor Ġboth Ġthe Ġsource Ġand Ġthe Ġtarget Ġdomains . ĠThe Ġpivot Ġwords Ġare Ġselected Ġin Ġa Ġway Ġto Ġcover Ġcommon Ġdomain - inv ari ant Ġopinion Ġwords . ĠSub sequent Ġresearch Ġaims Ġto Ġbetter Ġalign Ġthe Ġdomain - specific Ġwords ĠB IB REF 1 Ġ, ĠB IB REF 5 Ġ, ĠB IB REF 3 Ġsuch Ġthat Ġthe Ġdomain Ġdiscrepancy Ġcould Ġbe Ġreduced . ĠMore Ġrecently , ĠYu Ġand ĠJiang Ġ( ĠB IB REF 4 Ġ) Ġborrow Ġthe Ġidea Ġof Ġpivot Ġfeature Ġprediction Ġfrom ĠS CL Ġand Ġextend Ġit Ġto Ġa Ġneural Ġnetwork - based Ġsolution Ġwith Ġauxiliary Ġtasks . ĠIn Ġtheir Ġexperiment , Ġsubstantial Ġimprovement Ġover ĠS CL Ġhas Ġbeen Ġobserved Ġdue Ġto Ġthe Ġuse Ġof Ġreal - valued Ġword Ġembed d ings . ĠUn super vised Ġrepresentation Ġlearning Ġwith Ġdeep Ġneural Ġnetworks Ġ( D NN ) Ġsuch Ġas Ġden o ising Ġaut oen cod ers Ġhas Ġalso Ġbeen Ġexplored Ġfor Ġfeature Ġadaptation ĠB IB REF 6 Ġ, ĠB IB REF 7 Ġ, ĠB IB REF 8 Ġ. ĠIt Ġhas Ġbeen Ġshown Ġthat ĠD NN s Ġcould Ġlearn Ġtransfer able Ġrepresentations Ġthat Ġdis ent angle Ġthe Ġunderlying Ġfactors Ġof Ġvariation Ġbehind Ġdata Ġsamples . </s> Although Ġthe Ġaforementioned Ġmethods Ġaim Ġto Ġreduce Ġthe Ġdomain Ġdiscrepancy , Ġthey Ġdo Ġnot Ġexplicitly Ġminimize Ġthe Ġdistance Ġbetween Ġdistributions , Ġand Ġsome Ġof Ġthem Ġhighly Ġrely Ġon Ġthe Ġselection Ġof Ġpivot Ġfeatures . ĠIn Ġour Ġmethod , Ġwe Ġformally Ġconstruct Ġan Ġobjective Ġfor Ġthis Ġpurpose . ĠSimilar Ġideas Ġhave Ġbeen Ġexplored Ġin Ġmany Ġcomputer Ġvision Ġproblems , Ġwhere Ġthe Ġrepresentations Ġof Ġthe Ġunderlying Ġdomains Ġare Ġencouraged Ġto Ġbe Ġsimilar Ġthrough Ġexplicit Ġobjectives ĠB IB REF 9 Ġ, ĠB IB REF 10 Ġ, ĠB IB REF 11 Ġ, ĠB IB REF 12 Ġ, ĠB IB REF 13 Ġsuch Ġas Ġmaximum Ġmean Ġdiscrepancy Ġ( MM D ) ĠB IB REF 14 Ġ. ĠIn ĠN LP Ġtasks , ĠLi Ġet Ġal . Ġ( ĠB IB REF 15 Ġ) Ġand ĠChen Ġet Ġal . Ġ( ĠB IB REF 16 Ġ) Ġboth Ġproposed Ġusing Ġadvers arial Ġtraining Ġframework Ġfor Ġreducing Ġdomain Ġdifference . ĠIn Ġtheir Ġmodel , Ġa Ġsub - network Ġis Ġadded Ġas Ġa Ġdomain Ġdiscrim inator Ġwhile Ġdeep Ġfeatures Ġare Ġlearned Ġto Ġconfuse Ġthe Ġdiscrim inator . ĠThe Ġfeature Ġadaptation Ġcomponent Ġin Ġour Ġmodel Ġshares Ġsimilar Ġintuition Ġwith ĠM MD Ġand Ġadversary Ġtraining . ĠWe Ġwill Ġshow Ġa Ġdetailed Ġcomparison Ġwith Ġthem Ġin Ġour Ġexperiments . </s> Sem i - super vised ĠLearning : ĠWe Ġattempt Ġto Ġtreat Ġdomain Ġadaptation Ġas Ġa Ġsemi - super vised Ġlearning Ġtask Ġby Ġconsidering Ġthe Ġtarget Ġinstances Ġas Ġunl abel ed Ġdata . ĠSome Ġefforts Ġhave Ġbeen Ġinitiated Ġon Ġtransfer Ġlearning Ġfrom Ġunl abel ed Ġdata ĠB IB REF 17 Ġ, ĠB IB REF 18 Ġ, ĠB IB REF 19 Ġ. ĠIn Ġour Ġmodel , Ġwe Ġreduce Ġthe Ġdomain Ġdiscrepancy Ġby Ġfeature Ġadaptation , Ġand Ġthereafter Ġadopt Ġsemi - super vised Ġlearning Ġtechniques Ġto Ġlearn Ġfrom Ġunl abel ed Ġdata . ĠPrim arily Ġmotivated Ġby ĠB IB REF 20 Ġand ĠB IB REF 21 Ġ, Ġwe Ġemployed Ġentropy Ġminim ization Ġand Ġself - ense mble Ġboot stra pping Ġas Ġregular izations Ġto Ġincorporate Ġunl abel ed Ġdata . ĠOur Ġexperimental Ġresults Ġshow Ġthat Ġboth Ġmethods Ġare Ġeffective Ġwhen Ġjointly Ġtrained Ġwith Ġthe Ġfeature Ġadaptation Ġobjective , Ġwhich Ġconfirms Ġto Ġour Ġmotivation . </s> Not ations Ġand ĠModel ĠOverview </s> We Ġconduct Ġmost Ġof Ġour Ġexperiments Ġunder Ġan Ġun super vised Ġdomain Ġadaptation Ġsetting , Ġwhere Ġwe Ġhave Ġno Ġlabeled Ġdata Ġfrom Ġthe Ġtarget Ġdomain . ĠConsider Ġtwo Ġsets ĠIN LINE FORM 0 Ġand ĠIN LINE FORM 1 Ġ. ĠIN LINE FORM 2 Ġis Ġfrom Ġthe Ġsource Ġdomain Ġwith ĠIN LINE FORM 3 Ġlabeled Ġexamples , Ġwhere ĠIN LINE FORM 4 Ġis Ġa Ġone - hot Ġvector Ġrepresentation Ġof Ġsentiment Ġlabel Ġand ĠIN LINE FORM 5 Ġdenotes Ġthe Ġnumber Ġof Ġclasses . ĠIN LINE FORM 6 Ġis Ġfrom Ġthe Ġtarget Ġdomain Ġwith ĠIN LINE FORM 7 Ġunl abel ed Ġexamples . ĠIN LINE FORM 8 Ġdenotes Ġthe Ġtotal Ġnumber Ġof Ġtraining Ġdocuments Ġincluding Ġboth Ġlabeled Ġand Ġunl abel ed . ĠWe Ġaim Ġto Ġlearn Ġa Ġsentiment Ġclass ifier Ġfrom ĠIN LINE FORM 13 Ġand ĠIN LINE FORM 14 Ġsuch Ġthat Ġthe Ġclass ifier Ġwould Ġwork Ġwell Ġon Ġthe Ġtarget Ġdomain . ĠWe Ġalso Ġpresent Ġsome Ġresults Ġunder Ġa Ġsetting Ġwhere Ġwe Ġassume Ġthat Ġa Ġsmall Ġnumber Ġof Ġlabeled Ġtarget Ġexamples Ġare Ġavailable Ġ( see ĠFigure ĠFIG REF 27 Ġ). </s> For Ġthe Ġproposed Ġmodel , Ġwe Ġdenote ĠIN LINE FORM 0 Ġparameter ized Ġby ĠIN LINE FORM 1 Ġas Ġa Ġneural - based Ġfeature Ġenc oder Ġthat Ġmaps Ġdocuments Ġfrom Ġboth Ġdomains Ġto Ġa Ġshared Ġfeature Ġspace , Ġand ĠIN LINE FORM 2 Ġparameter ized Ġby ĠIN LINE FORM 3 Ġas Ġa Ġfully Ġconnected Ġlayer Ġwith Ġsoft max Ġactivation Ġserving Ġas Ġthe Ġsentiment Ġclass ifier . ĠWe Ġaim Ġto Ġlearn Ġfeature Ġrepresentations Ġthat Ġare Ġdomain - inv ari ant Ġand Ġat Ġthe Ġsame Ġtime Ġdiscrim in ative Ġon Ġboth Ġdomains , Ġthus Ġwe Ġsimultaneously Ġconsider Ġthree Ġfactors Ġin Ġour Ġobjective : Ġ( 1 ) Ġminimize Ġthe Ġclassification Ġerror Ġon Ġthe Ġlabeled Ġsource Ġexamples ; Ġ( 2 ) Ġminimize Ġthe Ġdomain Ġdiscrepancy ; Ġand Ġ( 3 ) Ġleverage Ġunl abel ed Ġdata Ġvia Ġsemi - super vised Ġlearning . </s> Supp ose Ġwe Ġalready Ġhave Ġthe Ġencoded Ġfeatures Ġof Ġdocuments ĠIN LINE FORM 0 Ġ( see ĠSection ĠSEC REF 10 Ġ), Ġthe Ġobjective Ġfunction Ġfor Ġpurpose Ġ( 1 ) Ġis Ġthus Ġthe Ġcross Ġentropy Ġloss Ġon Ġthe Ġlabeled Ġsource Ġexamples ĠDIS PLAY FORM 0 </s> where ĠIN LINE FORM 0 Ġdenotes Ġthe Ġpredicted Ġlabel Ġdistribution . ĠIn Ġthe Ġfollowing Ġsubsections , Ġwe Ġwill Ġexplain Ġhow Ġto Ġperform Ġfeature Ġadaptation Ġand Ġdomain Ġadaptive Ġsemi - super vised Ġlearning Ġin Ġdetails Ġfor Ġpurpose Ġ( 2 ) Ġand Ġ( 3 ) Ġrespectively . </s> Feature ĠAdapt ation </s> Unlike Ġprior Ġworks ĠB IB REF 0 Ġ, ĠB IB REF 4 Ġ, Ġour Ġmethod Ġdoes Ġnot Ġattempt Ġto Ġalign Ġdomain - specific Ġwords Ġthrough Ġpivot Ġwords . ĠIn Ġour Ġpreliminary Ġexperiments , Ġwe Ġfound Ġthat Ġword Ġembed d ings Ġpre - trained Ġon Ġa Ġlarge Ġcorpus Ġare Ġable Ġto Ġadequately Ġcapture Ġthis Ġinformation . ĠAs Ġwe Ġwill Ġlater Ġshow Ġin Ġour Ġexperiments , Ġeven Ġwithout Ġadaptation , Ġa Ġnaive Ġneural Ġnetwork Ġclass ifier Ġwith Ġpre - trained Ġword Ġembed d ings Ġcan Ġalready Ġachieve Ġreasonably Ġgood Ġresults . </s> We Ġattempt Ġto Ġexplicitly Ġminimize Ġthe Ġdistance Ġbetween Ġthe Ġsource Ġand Ġtarget Ġfeature Ġrepresentations Ġ( ĠIN LINE FORM 0 Ġand ĠIN LINE FORM 1 Ġ). ĠA Ġfew Ġmethods Ġfrom Ġliterature Ġcan Ġbe Ġapplied Ġsuch Ġas ĠMaximum ĠMean ĠDisc rep ancy Ġ( MM D ) ĠB IB REF 14 Ġor Ġadversary Ġtraining ĠB IB REF 15 Ġ, ĠB IB REF 16 Ġ. ĠThe Ġmain Ġidea Ġof ĠM MD Ġis Ġto Ġestimate Ġthe Ġdistance Ġbetween Ġtwo Ġdistributions Ġas Ġthe Ġdistance Ġbetween Ġsample Ġmeans Ġof Ġthe Ġprojected Ġembed d ings Ġin ĠHilbert Ġspace . ĠM MD Ġis Ġimplicitly Ġcomputed Ġthrough Ġa Ġcharacteristic Ġkernel , Ġwhich Ġis Ġused Ġto Ġensure Ġthat Ġthe Ġsample Ġmean Ġis Ġinject ive , Ġleading Ġto Ġthe ĠM MD Ġbeing Ġzero Ġif Ġand Ġonly Ġif Ġthe Ġdistributions Ġare Ġidentical . ĠIn Ġour Ġimplementation , Ġwe Ġskip Ġthe Ġmapping Ġprocedure Ġinduced Ġby Ġa Ġcharacteristic Ġkernel Ġfor Ġsimpl ifying Ġthe Ġcomputation Ġand Ġlearning . ĠWe Ġsimply Ġestimate Ġthe Ġdistribution Ġdistance Ġas Ġthe Ġdistance Ġbetween Ġthe Ġsample Ġmeans Ġin Ġthe Ġcurrent Ġembed ding Ġspace . ĠAlthough Ġthis Ġapproximation Ġcannot Ġpreserve Ġall Ġstatistical Ġfeatures Ġof Ġthe Ġunderlying Ġdistributions , Ġwe Ġfind Ġit Ġperforms Ġcompar ably Ġto ĠM MD Ġon Ġour Ġproblem . ĠThe Ġfollowing Ġequations Ġformally Ġdescribe Ġthe Ġfeature Ġadaptation Ġloss ĠIN LINE FORM 2 Ġ: ĠDIS PLAY FORM 0 </s> IN LINE FORM 0 Ġnormal ization Ġis Ġapplied Ġon Ġthe Ġmean Ġrepresentations ĠIN LINE FORM 1 Ġand ĠIN LINE FORM 2 Ġ, Ġresc aling Ġthe Ġvectors Ġsuch Ġthat Ġall Ġentries Ġsum Ġto Ġ1 . ĠWe Ġadopt Ġa Ġsymm etric Ġversion Ġof ĠKL Ġdivergence ĠB IB REF 12 Ġas Ġthe Ġdistance Ġfunction . ĠGiven Ġtwo Ġdistribution Ġvectors ĠIN LINE FORM 3 Ġ, ĠIN LINE FORM 4 Ġ. </s> Domain ĠAdapt ive ĠSemi - super vised ĠLearning Ġ( D AS ) </s> We Ġattempt Ġto Ġexploit Ġthe Ġinformation Ġin Ġtarget Ġdata Ġthrough Ġsemi - super vised Ġlearning Ġobjectives , Ġwhich Ġare Ġjointly Ġtrained Ġwith ĠIN LINE FORM 0 Ġand ĠIN LINE FORM 1 Ġ. ĠNormally , Ġto Ġincorporate Ġtarget Ġdata , Ġwe Ġcan Ġminimize Ġthe Ġcross Ġentropy Ġloss Ġbetween Ġthe Ġtrue Ġlabel Ġdistributions ĠIN LINE FORM 2 Ġand Ġthe Ġpredicted Ġlabel Ġdistributions ĠIN LINE FORM 3 Ġover Ġtarget Ġsamples . ĠThe Ġchallenge Ġhere Ġis Ġthat ĠIN LINE FORM 4 Ġis Ġunknown , Ġand Ġthus Ġwe Ġattempt Ġto Ġestimate Ġit Ġvia Ġsemi - super vised Ġlearning . ĠWe Ġuse Ġentropy Ġminim ization Ġand Ġboot stra pping Ġfor Ġthis Ġpurpose . ĠWe Ġwill Ġlater Ġshow Ġin Ġour Ġexperiments Ġthat Ġboth Ġmethods Ġare Ġeffective , Ġand Ġjointly Ġemploying Ġthem Ġoverall Ġyields Ġthe Ġbest Ġresults . </s> Ent ropy ĠMin im ization : ĠIn Ġthis Ġmethod , ĠIN LINE FORM 0 Ġis Ġestimated Ġas Ġthe Ġpredicted Ġlabel Ġdistribution ĠIN LINE FORM 1 Ġ, Ġwhich Ġis Ġa Ġfunction Ġof ĠIN LINE FORM 2 Ġand ĠIN LINE FORM 3 Ġ. ĠThe Ġloss Ġcan Ġthus Ġbe Ġwritten Ġas ĠDIS PLAY FORM 0 </s> Ass ume Ġthe Ġdomain Ġdiscrepancy Ġcan Ġbe Ġeffectively Ġreduced Ġthrough Ġfeature Ġadaptation , Ġby Ġminimizing Ġthe Ġentropy Ġpenalty , Ġtraining Ġof Ġthe Ġclass ifier Ġis Ġinfluenced Ġby Ġthe Ġunl abel ed Ġtarget Ġdata Ġand Ġwill Ġgenerally Ġmaximize Ġthe Ġmargins Ġbetween Ġthe Ġtarget Ġexamples Ġand Ġthe Ġdecision Ġboundaries , Ġincreasing Ġthe Ġprediction Ġconfidence Ġon Ġthe Ġtarget Ġdomain . </s> Self - ense mble ĠBoot stra pping : ĠAnother Ġway Ġto Ġestimate ĠIN LINE FORM 0 Ġcorresponds Ġto Ġboot stra pping . ĠThe Ġidea Ġis Ġto Ġestimate Ġthe Ġunknown Ġlabels Ġas Ġthe Ġpredictions Ġof Ġthe Ġmodel Ġlearned Ġfrom Ġthe Ġprevious Ġround Ġof Ġtraining . ĠBoot stra pping Ġhas Ġbeen Ġexplored Ġfor Ġdomain Ġadaptation Ġin Ġprevious Ġworks ĠB IB REF 18 Ġ, ĠB IB REF 19 Ġ. ĠHowever , Ġin Ġtheir Ġmethods , Ġdomain Ġdiscrepancy Ġwas Ġnot Ġexplicitly Ġminimized Ġvia Ġfeature Ġadaptation . ĠApp lying Ġboot stra pping Ġor Ġother Ġsemi - super vised Ġlearning Ġtechniques Ġin Ġthis Ġcase Ġmay Ġworsen Ġthe Ġresults Ġas Ġthe Ġclass ifier Ġcan Ġperform Ġquite Ġbad Ġon Ġthe Ġtarget Ġdata . </s> [ t ] ĠPse ud oc ode Ġfor Ġtraining ĠD AS ĠIN LINE FORM 0 Ġ, ĠIN LINE FORM 1 Ġ, ĠIN LINE FORM 2 Ġ, ĠIN LINE FORM 3 ĠIN LINE FORM 4 Ġ= Ġen semb ling Ġmomentum , ĠIN LINE FORM 5 ĠIN LINE FORM 6 Ġ= Ġweight Ġramp - up Ġfunction ĠIN LINE FORM 7 ĠIN LINE FORM 8 ĠIN LINE FORM 9 Ġeach Ġmin ib atch ĠIN LINE FORM 10 Ġ, ĠIN LINE FORM 11 Ġ, ĠIN LINE FORM 12 Ġin </s> IN LINE FORM 0 Ġ, ĠIN LINE FORM 1 Ġ, ĠIN LINE FORM 2 Ġcompute Ġloss ĠIN LINE FORM 3 Ġon ĠIN LINE FORM 4 Ġcompute Ġloss ĠIN LINE FORM 5 Ġon ĠIN LINE FORM 6 Ġcompute Ġloss ĠIN LINE FORM 7 Ġon ĠIN LINE FORM 8 Ġcompute Ġloss ĠIN LINE FORM 9 Ġon ĠIN LINE FORM 10 ĠIN LINE FORM 11 </s> update Ġnetwork Ġparameters ĠIN LINE FORM 0 Ġ, Ġfor ĠIN LINE FORM 1 ĠIN LINE FORM 2 ĠIN LINE FORM 3 </s> Insp ired Ġby Ġthe Ġen semb ling Ġmethod Ġproposed Ġin ĠB IB REF 21 Ġ, Ġwe Ġestimate ĠIN LINE FORM 0 Ġby Ġforming Ġensemble Ġpredictions Ġof Ġlabels Ġduring Ġtraining , Ġusing Ġthe Ġoutputs Ġon Ġdifferent Ġtraining Ġepoch s . ĠThe Ġloss Ġis Ġformulated Ġas Ġfollows : ĠDIS PLAY FORM 0 </s> where ĠIN LINE FORM 0 Ġdenotes Ġthe Ġestimated Ġlabels Ġcomputed Ġon Ġthe Ġensemble Ġpredictions Ġfrom Ġdifferent Ġepoch s . ĠThe Ġloss Ġis Ġapplied Ġon Ġall Ġdocuments . ĠIt Ġserves Ġfor Ġboot stra pping Ġon Ġthe Ġunl abel ed Ġtarget Ġdata , Ġand Ġit Ġalso Ġserves Ġas Ġa Ġregular ization Ġthat Ġencourages Ġthe Ġnetwork Ġpredictions Ġto Ġbe Ġconsistent Ġin Ġdifferent Ġtraining Ġepoch s . ĠIN LINE FORM 1 Ġis Ġjointly Ġtrained Ġwith ĠIN LINE FORM 2 Ġ, ĠIN LINE FORM 3 Ġ, Ġand ĠIN LINE FORM 4 Ġ. ĠAl gorithm ĠSEC REF 6 Ġillustrates Ġthe Ġoverall Ġtraining Ġprocess Ġof Ġthe Ġproposed Ġdomain Ġadaptive Ġsemi - super vised Ġlearning Ġ( D AS ) Ġframework . </s> In ĠAl gorithm ĠSEC REF 6 Ġ, ĠIN LINE FORM 0 Ġ, ĠIN LINE FORM 1 Ġ, Ġand ĠIN LINE FORM 2 Ġare Ġweights Ġto Ġbalance Ġthe Ġeffects Ġof ĠIN LINE FORM 3 Ġ, ĠIN LINE FORM 4 Ġ, Ġand ĠIN LINE FORM 5 Ġrespectively . ĠIN LINE FORM 6 Ġand ĠIN LINE FORM 7 Ġare Ġconstant Ġhyper - param eters . ĠWe Ġset ĠIN LINE FORM 8 Ġas Ġa ĠGa ussian Ġcurve Ġto Ġramp Ġup Ġthe Ġweight Ġfrom Ġ0 Ġto ĠIN LINE FORM 9 Ġ. ĠThis Ġis Ġto Ġensure Ġthe Ġramp - up Ġof Ġthe Ġboot stra pping Ġloss Ġcomponent Ġis Ġslow Ġenough Ġin Ġthe Ġbeginning Ġof Ġthe Ġtraining . ĠAfter Ġeach Ġtraining Ġepoch , Ġwe Ġcompute ĠIN LINE FORM 10 Ġwhich Ġdenotes Ġthe Ġpredictions Ġmade Ġby Ġthe Ġnetwork Ġin Ġcurrent Ġepoch , Ġand Ġthen Ġthe Ġensemble Ġprediction ĠIN LINE FORM 11 Ġis Ġupdated Ġas Ġa Ġweighted Ġaverage Ġof Ġthe Ġoutputs Ġfrom Ġprevious Ġepoch s Ġand Ġthe Ġcurrent Ġepoch , Ġwith Ġrecent Ġepoch s Ġhaving Ġlarger Ġweight . ĠFor Ġgenerating Ġestimated Ġlabels ĠIN LINE FORM 12 Ġ, ĠIN LINE FORM 13 Ġis Ġconverted Ġto Ġa Ġone - hot Ġvector Ġwhere Ġthe Ġentry Ġwith Ġthe Ġmaximum Ġvalue Ġis Ġset Ġto Ġone Ġand Ġother Ġentries Ġare Ġset Ġto Ġz eros . ĠThe Ġself - ense mble Ġboot stra pping Ġis Ġa Ġgeneralized Ġversion Ġof Ġboot stra ppings Ġthat Ġonly Ġuse Ġthe Ġoutputs Ġfrom Ġthe Ġprevious Ġround Ġof Ġtraining ĠB IB REF 18 Ġ, ĠB IB REF 19 Ġ. ĠThe Ġensemble Ġprediction Ġis Ġlikely Ġto Ġbe Ġcloser Ġto Ġthe Ġcorrect , Ġunknown Ġlabels Ġof Ġthe Ġtarget Ġdata . </s> CNN ĠEnc oder ĠImplementation </s> We Ġhave Ġleft Ġthe Ġfeature Ġenc oder ĠIN LINE FORM 0 Ġunspecified , Ġfor Ġwhich , Ġa Ġfew Ġoptions Ġcan Ġbe Ġconsidered . ĠIn Ġour Ġimplementation , Ġwe Ġadopt Ġa Ġone - layer ĠCNN Ġstructure Ġfrom Ġprevious Ġworks ĠB IB REF 22 Ġ, ĠB IB REF 4 Ġ, Ġas Ġit Ġhas Ġbeen Ġdemonstrated Ġto Ġwork Ġwell Ġfor Ġsentiment Ġclassification Ġtasks . ĠGiven Ġa Ġreview Ġdocument ĠIN LINE FORM 1 Ġconsisting Ġof ĠIN LINE FORM 2 Ġwords , Ġwe Ġbegin Ġby Ġassoci ating Ġeach Ġword Ġwith Ġa Ġcontinuous Ġword Ġembed ding ĠB IB REF 23 ĠIN LINE FORM 3 Ġfrom Ġan Ġembed ding Ġmatrix ĠIN LINE FORM 4 Ġ, Ġwhere ĠIN LINE FORM 5 Ġis Ġthe Ġvocabulary Ġsize Ġand ĠIN LINE FORM 6 Ġis Ġthe Ġembed ding Ġdimension . ĠIN LINE FORM 7 Ġis Ġjointly Ġupdated Ġwith Ġother Ġnetwork Ġparameters Ġduring Ġtraining . ĠGiven Ġa Ġwindow Ġof Ġdense Ġword Ġembed d ings ĠIN LINE FORM 8 Ġ, Ġthe Ġconv olution Ġlayer Ġfirst Ġconc aten ates Ġthese Ġvectors Ġto Ġform Ġa Ġvector ĠIN LINE FORM 9 Ġof Ġlength ĠIN LINE FORM 10 Ġand Ġthen Ġthe Ġoutput Ġvector Ġis Ġcomputed Ġby ĠEqu ation Ġ( ĠEQ REF 11 Ġ): ĠDIS PLAY FORM 0 </s> IN LINE FORM 0 Ġ, ĠIN LINE FORM 1 Ġis Ġthe Ġparameter Ġset Ġof Ġthe Ġenc oder ĠIN LINE FORM 2 Ġand Ġis Ġshared Ġacross Ġall Ġwindows Ġof Ġthe Ġsequence . ĠIN LINE FORM 3 Ġis Ġan Ġelement - wise Ġnon - linear Ġactivation Ġfunction . ĠThe Ġconv olution Ġoperation Ġcan Ġcapture Ġlocal Ġcontextual Ġdependencies Ġof Ġthe Ġinput Ġsequence Ġand Ġthe Ġextracted Ġfeature Ġvectors Ġare Ġsimilar Ġto ĠIN LINE FORM 4 Ġ- gram s . ĠAfter Ġthe Ġconv olution Ġoperation Ġis Ġapplied Ġto Ġthe Ġwhole Ġsequence , Ġwe Ġobtain Ġa Ġlist Ġof Ġhidden Ġvectors ĠIN LINE FORM 5 Ġ. ĠA Ġmax - over - time Ġpool ing Ġlayer Ġis Ġapplied Ġto Ġobtain Ġthe Ġfinal Ġvector Ġrepresentation ĠIN LINE FORM 6 Ġof Ġthe Ġinput Ġdocument . </s> Dat as ets Ġand ĠExperimental ĠSettings </s> Ex isting Ġbenchmark Ġdatasets Ġsuch Ġas Ġthe ĠAmazon Ġbenchmark ĠB IB REF 0 Ġtypically Ġremove Ġreviews Ġwith Ġneutral Ġlabels Ġin Ġboth Ġdomains . ĠThis Ġis Ġproblematic Ġas Ġthe Ġlabel Ġinformation Ġof Ġthe Ġtarget Ġdomain Ġis Ġnot Ġaccessible Ġin Ġan Ġun super vised Ġdomain Ġadaptation Ġsetting . ĠFurthermore , Ġremoving Ġneutral Ġinstances Ġmay Ġbias Ġthe Ġdataset Ġfavorably Ġfor Ġmax - margin - based Ġalgorithms Ġlike Ġours , Ġsince Ġthe Ġresulting Ġdataset Ġhas Ġall Ġuncertain Ġlabels Ġremoved , Ġleaving Ġonly Ġhigh Ġconfidence Ġexamples . ĠTherefore , Ġwe Ġconstruct Ġnew Ġdatasets Ġby Ġourselves . ĠThe Ġresults Ġon Ġthe Ġoriginal ĠAmazon Ġbenchmark Ġis Ġqual itatively Ġsimilar , Ġand Ġwe Ġpresent Ġthem Ġin ĠAppendix ĠSEC REF 6 Ġfor Ġcomple teness Ġsince Ġmost Ġof Ġprevious Ġworks Ġreported Ġresults Ġon Ġit . </s> Small - scale Ġdatasets : ĠOur Ġnew Ġdataset Ġwas Ġderived Ġfrom Ġthe Ġlarge - scale ĠAmazon Ġdatasets Ġreleased Ġby ĠMcA u ley Ġet Ġal . Ġ( ĠB IB REF 24 Ġ). ĠIt Ġcontains Ġfour Ġdomains : ĠBook Ġ( B K ), ĠElectronics Ġ( E ), ĠBeauty Ġ( BT ), Ġand ĠMusic Ġ( M ). ĠEach Ġdomain Ġcontains Ġtwo Ġdatasets . ĠSet Ġ1 Ġcontains Ġ6000 Ġinstances Ġwith Ġexactly Ġbalanced Ġclass Ġlabels , Ġand Ġset Ġ2 Ġcontains Ġ6000 Ġinstances Ġthat Ġare Ġrandomly Ġsampled Ġfrom Ġthe Ġlarge Ġdataset , Ġpreserving Ġthe Ġoriginal Ġlabel Ġdistribution , Ġwhich Ġwe Ġbelieve Ġbetter Ġreflects Ġthe Ġlabel Ġdistribution Ġin Ġreal Ġlife . ĠThe Ġexamples Ġin Ġthese Ġtwo Ġsets Ġdo Ġnot Ġoverlap . ĠDet ailed Ġstatistics Ġof Ġthe Ġgenerated Ġdatasets Ġare Ġgiven Ġin ĠTable ĠT AB REF 9 Ġ. </s> In Ġall Ġour Ġexperiments Ġon Ġthe Ġsmall - scale Ġdatasets , Ġwe Ġuse Ġset Ġ1 Ġof Ġthe Ġsource Ġdomain Ġas Ġthe Ġonly Ġsource Ġwith Ġsentiment Ġlabel Ġinformation Ġduring Ġtraining , Ġand Ġwe Ġevaluate Ġthe Ġtrained Ġmodel Ġon Ġset Ġ1 Ġof Ġthe Ġtarget Ġdomain . ĠSince Ġwe Ġcannot Ġcontrol Ġthe Ġlabel Ġdistribution Ġof Ġunl abel ed Ġdata Ġduring Ġtraining , Ġwe Ġconsider Ġtwo Ġdifferent Ġsettings : </s> Setting Ġ( 1 ): ĠOnly Ġset Ġ1 Ġof Ġthe Ġtarget Ġdomain Ġis Ġused Ġas Ġthe Ġunl abel ed Ġset . ĠThis Ġtells Ġus Ġhow Ġthe Ġmethod Ġperforms Ġin Ġa Ġcondition Ġwhen Ġthe Ġtarget Ġdomain Ġhas Ġa Ġclose - to - balanced Ġlabel Ġdistribution . ĠAs Ġwe Ġalso Ġevaluate Ġon Ġset Ġ1 Ġof Ġthe Ġtarget Ġdomain , Ġthis Ġis Ġalso Ġconsidered Ġas Ġa Ġtrans duct ive Ġsetting . </s> Setting Ġ( 2 ): ĠSet Ġ2 Ġfrom Ġboth Ġthe Ġsource Ġand Ġtarget Ġdomains Ġare Ġused Ġas Ġunl abel ed Ġsets . ĠSince Ġset Ġ2 Ġis Ġdirectly Ġsampled Ġfrom Ġmillions Ġof Ġreviews , Ġit Ġbetter Ġreflects Ġreal - life Ġsentiment Ġdistribution . </s> Large - scale Ġdatasets : ĠWe Ġfurther Ġconduct Ġexperiments Ġon Ġfour Ġmuch Ġlarger Ġdatasets : ĠIM DB Ġ( I ), ĠYelp 2014 Ġ( Y ), ĠCell ĠPhone Ġ( C ), Ġand ĠBaby Ġ( B ). ĠIM DB Ġand ĠYelp 2014 Ġwere Ġpreviously Ġused Ġin ĠB IB REF 25 Ġ, ĠB IB REF 26 Ġ. ĠCell Ġphone Ġand ĠBaby Ġare Ġfrom Ġthe Ġlarge - scale ĠAmazon Ġdataset ĠB IB REF 24 Ġ, ĠB IB REF 27 Ġ. ĠDet ailed Ġstatistics Ġare Ġsummarized Ġin ĠTable ĠT AB REF 9 Ġ. ĠWe Ġkeep Ġall Ġreviews Ġin Ġthe Ġoriginal Ġdatasets Ġand Ġconsider Ġa Ġtrans duct ive Ġsetting Ġwhere Ġall Ġtarget Ġexamples Ġare Ġused Ġfor Ġboth Ġtraining Ġ( without Ġlabel Ġinformation ) Ġand Ġevaluation . ĠWe Ġperform Ġsampling Ġto Ġbalance Ġthe Ġclasses Ġof Ġlabeled Ġsource Ġdata Ġin Ġeach Ġmin ib atch ĠIN LINE FORM 3 Ġduring Ġtraining . </s> Se lection Ġof ĠDevelopment ĠSet </s> Ide ally , Ġthe Ġdevelopment Ġset Ġshould Ġbe Ġdrawn Ġfrom Ġthe Ġsame Ġdistribution Ġas Ġthe Ġtest Ġset . ĠHowever , Ġunder Ġthe Ġun super vised Ġdomain Ġadaptation Ġsetting , Ġwe Ġdo Ġnot Ġhave Ġany Ġlabeled Ġtarget Ġdata Ġat Ġtraining Ġphase Ġwhich Ġcould Ġbe Ġused Ġas Ġdevelopment Ġset . ĠIn Ġall Ġof Ġour Ġexperiments , Ġfor Ġeach Ġpair Ġof Ġdomains , Ġwe Ġinstead Ġsample Ġ1000 Ġexamples Ġfrom Ġthe Ġtraining Ġset Ġof Ġthe Ġsource Ġdomain Ġas Ġdevelopment Ġset . ĠWe Ġtrain Ġthe Ġnetwork Ġfor Ġa Ġfixed Ġnumber Ġof Ġepoch s , Ġand Ġthe Ġmodel Ġwith Ġthe Ġminimum Ġclassification Ġerror Ġon Ġthis Ġdevelopment Ġset Ġis Ġsaved Ġfor Ġevaluation . ĠThis Ġapproach Ġworks Ġwell Ġon Ġmost Ġof Ġthe Ġproblems Ġsince Ġthe Ġtarget Ġdomain Ġis Ġsupposed Ġto Ġbehave Ġlike Ġthe Ġsource Ġdomain Ġif Ġthe Ġdomain Ġdifference Ġis Ġeffectively Ġreduced . </s> Another Ġproblem Ġis Ġhow Ġto Ġselect Ġthe Ġvalues Ġfor Ġhyper - param eters . ĠIf Ġwe Ġtune ĠIN LINE FORM 0 Ġand ĠIN LINE FORM 1 Ġdirectly Ġon Ġthe Ġdevelopment Ġset Ġfrom Ġthe Ġsource Ġdomain , Ġmost Ġlikely Ġboth Ġof Ġthem Ġwill Ġbe Ġset Ġto Ġ0 , Ġas Ġunl abel ed Ġtarget Ġdata Ġis Ġnot Ġhelpful Ġfor Ġimproving Ġin - domain Ġaccuracy Ġof Ġthe Ġsource Ġdomain . ĠOther Ġneural Ġnetwork Ġmodels Ġalso Ġhave Ġthe Ġsame Ġproblem Ġfor Ġhyper - param eter Ġtuning . ĠTherefore , Ġour Ġstrategy Ġis Ġto Ġuse Ġthe Ġdevelopment Ġset Ġfrom Ġthe Ġtarget Ġdomain Ġto Ġoptimize ĠIN LINE FORM 2 Ġand ĠIN LINE FORM 3 Ġfor Ġone Ġproblem Ġ( e . g ., Ġwe Ġonly Ġdo Ġthis Ġon ĠE ĠIN LINE FORM 4 ĠB K ), Ġand Ġfix Ġtheir Ġvalues Ġon Ġthe Ġother Ġproblems . ĠThis Ġsetting Ġassumes Ġthat Ġwe Ġhave Ġat Ġleast Ġtwo Ġlabeled Ġdomains Ġsuch Ġthat Ġwe Ġcan Ġoptimize Ġthe Ġhyper - param eters , Ġand Ġthen Ġwe Ġfix Ġthem Ġfor Ġother Ġnew Ġunl abel ed Ġdomains Ġto Ġtransfer Ġto . </s> Training ĠDetails Ġand ĠHyper - param eters </s> We Ġinitialize Ġword Ġembed d ings Ġusing Ġthe Ġ300 - dimension ĠGl o Ve Ġvectors Ġsupplied Ġby ĠPen nington Ġet Ġal ., Ġ( ĠB IB REF 28 Ġ), Ġwhich Ġwere Ġtrained Ġon Ġ840 Ġbillion Ġtokens Ġfrom Ġthe ĠCommon ĠC rawl . ĠFor Ġeach Ġpair Ġof Ġdomains , Ġthe Ġvocabulary Ġconsists Ġof Ġthe Ġtop Ġ10000 Ġmost Ġfrequent Ġwords . ĠFor Ġwords Ġin Ġthe Ġvocabulary Ġbut Ġnot Ġpresent Ġin Ġthe Ġpre - trained Ġembed d ings , Ġwe Ġrandomly Ġinitialize Ġthem . </s> We Ġset Ġhyper - param eters Ġof Ġthe ĠCNN Ġenc oder Ġfollowing Ġprevious Ġworks ĠB IB REF 22 Ġ, ĠB IB REF 4 Ġwithout Ġspecific Ġtuning Ġon Ġour Ġdatasets . ĠThe Ġwindow Ġsize Ġis Ġset Ġto Ġ3 Ġand Ġthe Ġsize Ġof Ġthe Ġhidden Ġlayer Ġis Ġset Ġto Ġ300 . ĠThe Ġnon linear Ġactivation Ġfunction Ġis ĠRel u . ĠFor Ġregular ization , Ġwe Ġalso Ġfollow Ġtheir Ġsettings Ġand Ġemploy Ġdrop out Ġwith Ġprobability Ġset Ġto Ġ0 . 5 Ġon ĠIN LINE FORM 0 Ġbefore Ġfeeding Ġit Ġto Ġthe Ġoutput Ġlayer ĠIN LINE FORM 1 Ġ, Ġand Ġconst rain Ġthe ĠIN LINE FORM 2 Ġ- norm Ġof Ġthe Ġweight Ġvector ĠIN LINE FORM 3 Ġ, Ġsetting Ġits Ġmax Ġnorm Ġto Ġ3 . </s> On Ġthe Ġsmall - scale Ġdatasets Ġand Ġthe ĠA am zon Ġbenchmark , ĠIN LINE FORM 0 Ġand ĠIN LINE FORM 1 Ġare Ġset Ġto Ġ200 Ġand Ġ1 , Ġrespectively , Ġtuned Ġon Ġthe Ġdevelopment Ġset Ġof Ġtask ĠE ĠIN LINE FORM 2 ĠB K Ġunder Ġsetting Ġ1 . ĠOn Ġthe Ġlarge - scale Ġdatasets , ĠIN LINE FORM 3 Ġand ĠIN LINE FORM 4 Ġare Ġset Ġto Ġ500 Ġand Ġ0 . 2 , Ġrespectively , Ġtuned Ġon ĠI ĠIN LINE FORM 5 ĠY . ĠWe Ġuse Ġa ĠGa ussian Ġcurve ĠIN LINE FORM 6 Ġto Ġramp Ġup Ġthe Ġweight Ġof Ġthe Ġboot stra pping Ġloss ĠIN LINE FORM 7 Ġfrom Ġ0 Ġto ĠIN LINE FORM 8 Ġ, Ġwhere ĠIN LINE FORM 9 Ġdenotes Ġthe Ġmaximum Ġnumber Ġof Ġtraining Ġepoch s . ĠWe Ġtrain Ġ30 Ġepoch s Ġfor Ġall Ġexperiments . ĠWe Ġset ĠIN LINE FORM 10 Ġto Ġ3 Ġand ĠIN LINE FORM 11 Ġto Ġ0 . 5 Ġfor Ġall Ġexperiments . </s> The Ġbatch Ġsize Ġis Ġset Ġto Ġ50 Ġon Ġthe Ġsmall - scale Ġdatasets Ġand Ġthe ĠAmazon Ġbenchmark . ĠWe Ġincrease Ġthe Ġbatch Ġsize Ġto Ġ250 Ġon Ġthe Ġlarge - scale Ġdatasets Ġto Ġreduce Ġthe Ġnumber Ġof Ġiterations . ĠRM SP rop Ġoptim izer Ġwith Ġlearning Ġrate Ġset Ġto Ġ0 . 000 5 Ġis Ġused Ġfor Ġall Ġexperiments . </s> Mod els Ġfor ĠComparison </s> We Ġcompare Ġwith Ġthe Ġfollowing Ġbas elines : </s> ( 1 ) ĠNa ive : ĠA Ġnon - domain - adapt ive Ġbaseline Ġwith Ġbag - of - words Ġrepresentations Ġand ĠS VM Ġclass ifier Ġtrained Ġon Ġthe Ġsource Ġdomain . </s> ( 2 ) Ġm S DA ĠB IB REF 7 Ġ: ĠThis Ġis Ġthe Ġstate - of - the - art Ġmethod Ġbased Ġon Ġdiscrete Ġinput Ġfeatures . ĠTop Ġ1000 Ġbag - of - words Ġfeatures Ġare Ġkept Ġas Ġpivot Ġfeatures . ĠWe Ġset Ġthe Ġnumber Ġof Ġstacked Ġlayers Ġto Ġ3 Ġand Ġthe Ġcorruption Ġprobability Ġto Ġ0 . 5 . </s> ( 3 ) ĠNa ive NN : ĠThis Ġis Ġa Ġnon - domain - adapt ive ĠCNN Ġtrained Ġon Ġsource Ġdomain , Ġwhich Ġis Ġa Ġvariant Ġof Ġour Ġmodel Ġby Ġsetting ĠIN LINE FORM 0 Ġ, ĠIN LINE FORM 1 Ġ, Ġand ĠIN LINE FORM 2 Ġto Ġz eros . </s> ( 4 ) ĠAux NN ĠB IB REF 4 Ġ: ĠThis Ġis Ġa Ġneural Ġmodel Ġthat Ġexploits Ġauxiliary Ġtasks , Ġwhich Ġhas Ġachieved Ġstate - of - the - art Ġresults Ġon Ġcross - domain Ġsentiment Ġclassification . ĠThe Ġsentence Ġenc oder Ġused Ġin Ġthis Ġmodel Ġis Ġthe Ġsame Ġas Ġours . </s> ( 5 ) ĠAD AN ĠB IB REF 16 Ġ: ĠThis Ġmethod Ġexploits Ġadvers arial Ġtraining Ġto Ġreduce Ġrepresentation Ġdifference Ġbetween Ġdomains . ĠThe Ġoriginal Ġpaper Ġuses Ġa Ġsimple Ġfeed forward Ġnetwork Ġas Ġenc oder . ĠFor Ġfair Ġcomparison , Ġwe Ġreplace Ġit Ġwith Ġour ĠCNN - based Ġenc oder . ĠWe Ġtrain Ġ5 Ġiterations Ġon Ġthe Ġdiscrim inator Ġper Ġiteration Ġon Ġthe Ġenc oder Ġand Ġsentiment Ġclass ifier Ġas Ġsuggested Ġin Ġtheir Ġpaper . </s> ( 6 ) ĠM MD : ĠM MD Ġhas Ġbeen Ġwidely Ġused Ġfor Ġminimizing Ġdomain Ġdiscrepancy Ġon Ġimages . ĠIn Ġthose Ġworks ĠB IB REF 9 Ġ, ĠB IB REF 13 Ġ, Ġvariants Ġof Ġdeep ĠCNN s Ġare Ġused Ġfor Ġencoding Ġimages Ġand Ġthe ĠM MD s Ġof Ġmultiple Ġlayers Ġare Ġjointly Ġminimized . ĠIn ĠN LP , Ġadding Ġmore Ġlayers Ġof ĠCNN s Ġmay Ġnot Ġbe Ġvery Ġhelpful Ġand Ġthus Ġthose Ġmodels Ġfrom Ġimage - related Ġtasks Ġcan Ġnot Ġbe Ġdirectly Ġapplied Ġto Ġour Ġproblem . ĠTo Ġcompare Ġwith ĠM MD - based Ġmethod , Ġwe Ġtrain Ġa Ġmodel Ġthat Ġjointly Ġminimize Ġthe Ġclassification Ġloss ĠIN LINE FORM 0 Ġon Ġthe Ġsource Ġdomain Ġand ĠM MD Ġbetween ĠIN LINE FORM 1 Ġand ĠIN LINE FORM 2 Ġ. ĠFor Ġcomputing ĠM MD , Ġwe Ġuse Ġa ĠGa ussian ĠRB F Ġwhich Ġis Ġa Ġcommon Ġchoice Ġfor Ġcharacteristic Ġkernel . </s> In Ġaddition Ġto Ġthe Ġabove Ġbas elines , Ġwe Ġalso Ġshow Ġresults Ġof Ġdifferent Ġvariants Ġof Ġour Ġmodel . ĠD AS Ġas Ġshown Ġin ĠAl gorithm ĠSEC REF 6 Ġdenotes Ġour Ġfull Ġmodel . ĠD AS - EM Ġdenotes Ġthe Ġmodel Ġwith Ġonly Ġentropy Ġminim ization Ġfor Ġsemi - super vised Ġlearning Ġ( set ĠIN LINE FORM 0 Ġ). ĠD AS - SE Ġdenotes Ġthe Ġmodel Ġwith Ġonly Ġself - ense mble Ġboot stra pping Ġfor Ġsemi - super vised Ġlearning Ġ( set ĠIN LINE FORM 1 Ġ). ĠF ANN Ġ( feature - adapt ation Ġneural Ġnetwork ) Ġdenotes Ġthe Ġmodel Ġwithout Ġsemi - super vised Ġlearning Ġperformed Ġ( set Ġboth ĠIN LINE FORM 2 Ġand ĠIN LINE FORM 3 Ġto Ġz eros ). </s> Main ĠResults </s> Figure ĠFIG REF 17 Ġshows Ġthe Ġcomparison Ġof Ġadaptation Ġresults Ġ( see ĠAppendix ĠSEC REF 7 Ġfor Ġthe Ġexact Ġnumerical Ġnumbers ). ĠWe Ġreport Ġclassification Ġaccuracy Ġon Ġthe Ġsmall - scale Ġdataset . ĠFor Ġthe Ġlarge - scale Ġdataset , Ġmacro - F 1 Ġis Ġinstead Ġused Ġsince Ġthe Ġlabel Ġdistribution Ġin Ġthe Ġtest Ġset Ġis Ġextremely Ġun balanced . ĠKey Ġobservations Ġare Ġsummarized Ġas Ġfollows . Ġ( 1 ) ĠBoth ĠD AS - EM Ġand ĠD AS - SE Ġperform Ġbetter Ġin Ġmost Ġcases Ġcompared Ġwith ĠAD AN , ĠMD D , Ġand ĠF ANN , Ġin Ġwhich Ġonly Ġfeature Ġadaptation Ġis Ġperformed . ĠThis Ġdemonstrates Ġthe Ġeffectiveness Ġof Ġthe Ġproposed Ġdomain Ġadaptive Ġsemi - super vised Ġlearning Ġframework . ĠD AS - EM Ġis Ġmore Ġeffective Ġthan ĠD AS - SE Ġin Ġmost Ġcases , Ġand Ġthe Ġfull Ġmodel ĠD AS Ġwith Ġboth Ġtechniques Ġjointly Ġemployed Ġoverall Ġhas Ġthe Ġbest Ġperformance . Ġ( 2 ) ĠWhen Ġcomparing Ġthe Ġtwo Ġsettings Ġon Ġthe Ġsmall - scale Ġdataset , Ġall Ġdomain - adapt ive Ġmethods Ġgenerally Ġperform Ġbetter Ġunder Ġsetting Ġ1 . ĠIn Ġsetting Ġ1 , Ġthe Ġtarget Ġexamples Ġare Ġbalanced Ġin Ġclasses , Ġwhich Ġcan Ġprovide Ġmore Ġdiverse Ġopinion - related Ġfeatures . ĠHowever , Ġwhen Ġconsidering Ġun super vised Ġdomain Ġadaptation , Ġwe Ġshould Ġnot Ġpresume Ġthe Ġlabel Ġdistribution Ġof Ġthe Ġunl abel ed Ġdata . ĠThus , Ġit Ġis Ġnecessary Ġto Ġconduct Ġexperiments Ġusing Ġdatasets Ġthat Ġreflect Ġreal - life Ġsentiment Ġdistribution Ġas Ġwhat Ġwe Ġdid Ġon Ġsetting 2 Ġand Ġthe Ġlarge - scale Ġdataset . ĠUnfortunately , Ġthis Ġis Ġignored Ġby Ġmost Ġof Ġprevious Ġworks . Ġ( 3 ) ĠWord - embed d ings Ġare Ġvery Ġhelpful , Ġas Ġwe Ġcan Ġsee Ġeven ĠNa ive NN Ġcan Ġsubstantially Ġoutper form Ġm S DA Ġon Ġmost Ġtasks . </s> To Ġsee Ġthe Ġeffect Ġof Ġsemi - super vised Ġlearning Ġalone , Ġwe Ġalso Ġconduct Ġexperiments Ġby Ġsetting ĠIN LINE FORM 0 Ġto Ġeliminate Ġthe Ġeffect Ġof Ġfeature Ġadaptation . ĠBoth Ġentropy Ġminim ization Ġand Ġboot stra pping Ġperform Ġvery Ġbadly Ġin Ġthis Ġsetting . ĠEnt ropy Ġminim ization Ġgives Ġalmost Ġrandom Ġpredictions Ġwith Ġaccuracy Ġbelow Ġ0 . 4 , Ġand Ġthe Ġresults Ġof Ġboot stra pping Ġare Ġalso Ġmuch Ġlower Ġcompared Ġto ĠNa ive NN . ĠThis Ġsuggests Ġthat Ġthe Ġfeature Ġadaptation Ġcomponent Ġis Ġessential . ĠWithout Ġit , Ġthe Ġlearned Ġtarget Ġrepresentations Ġare Ġless Ġmeaningful Ġand Ġdiscrim in ative . ĠApp lying Ġsemi - super vised Ġlearning Ġin Ġthis Ġcase Ġis Ġlikely Ġto Ġworsen Ġthe Ġresults . </s> Further ĠAnalysis </s> In ĠFigure ĠFIG REF 23 Ġ, Ġwe Ġshow Ġthe Ġchange Ġof Ġaccuracy Ġwith Ġrespect Ġto Ġthe Ġpercentage Ġof Ġunl abel ed Ġdata Ġused Ġfor Ġtraining Ġon Ġthree Ġparticular Ġproblems Ġunder Ġsetting Ġ1 . ĠThe Ġvalue Ġat ĠIN LINE FORM 0 Ġdenotes Ġthe Ġaccur acies Ġof ĠNa ive NN Ġwhich Ġdoes Ġnot Ġutilize Ġany Ġtarget Ġdata . ĠFor ĠD AS , Ġwe Ġobserve Ġa Ġnon linear Ġincreasing Ġtrend Ġwhere Ġthe Ġaccuracy Ġquickly Ġimproves Ġat Ġthe Ġbeginning , Ġand Ġthen Ġgradually Ġstabil izes . ĠFor Ġother Ġmethods , Ġthis Ġtrend Ġis Ġless Ġobvious , Ġand Ġadding Ġmore Ġunl abel ed Ġdata Ġsometimes Ġeven Ġworsen Ġthe Ġresults . ĠThis Ġfinding Ġagain Ġsuggests Ġthat Ġthe Ġproposed Ġapproach Ġcan Ġbetter Ġexploit Ġthe Ġinformation Ġfrom Ġunl abel ed Ġdata . </s> We Ġalso Ġconduct Ġexperiments Ġunder Ġa Ġsetting Ġwith Ġa Ġsmall Ġnumber Ġof Ġlabeled Ġtarget Ġexamples Ġavailable . ĠFigure ĠFIG REF 27 Ġshows Ġthe Ġchange Ġof Ġaccuracy Ġwith Ġrespect Ġto Ġthe Ġnumber Ġof Ġlabeled Ġtarget Ġexamples Ġadded Ġfor Ġtraining . ĠWe Ġcan Ġobserve Ġthat ĠD AS Ġis Ġstill Ġmore Ġeffective Ġunder Ġthis Ġsetting , Ġwhile Ġthe Ġperformance Ġdifferences Ġto Ġother Ġmethods Ġgradually Ġdecrease Ġwith Ġthe Ġincreasing Ġnumber Ġof Ġlabeled Ġtarget Ġexamples . </s> CNN ĠFilter ĠAnalysis </s> In Ġthis Ġsubsection , Ġwe Ġaim Ġto Ġbetter Ġunderstand ĠD AS Ġby Ġanalyzing Ġsentiment - related ĠCNN Ġfilters . ĠTo Ġdo Ġthat , Ġ1 ) Ġwe Ġfirst Ġselect Ġa Ġlist Ġof Ġthe Ġmost Ġrelated ĠCNN Ġfilters Ġfor Ġpredicting Ġeach Ġsentiment Ġlabel Ġ( positive , Ġnegative Ġneutral ). ĠThose Ġfilters Ġcan Ġbe Ġidentified Ġaccording Ġto Ġthe Ġlearned Ġweights ĠIN LINE FORM 0 Ġof Ġthe Ġoutput Ġlayer ĠIN LINE FORM 1 Ġ. ĠHigher Ġweight Ġindicates Ġstronger Ġrelated ness . Ġ2 ) ĠRecall Ġthat Ġin Ġour Ġimplementation , Ġeach ĠCNN Ġfilter Ġhas Ġa Ġwindow Ġsize Ġof Ġ3 Ġwith ĠRel u Ġactivation . ĠWe Ġcan Ġthus Ġrepresent Ġeach Ġselected Ġfilter Ġas Ġa Ġranked Ġlist Ġof Ġtrig rams Ġwith Ġhighest Ġactivation Ġvalues . </s> We Ġanalyze Ġthe ĠCNN Ġfilters Ġlearned Ġby ĠNa ive NN , ĠF ANN Ġand ĠD AS Ġrespectively Ġon Ġtask ĠE ĠIN LINE FORM 0 ĠBT Ġunder Ġsetting Ġ1 . ĠWe Ġfocus Ġon ĠE ĠIN LINE FORM 1 ĠBT Ġfor Ġstudy Ġbecause Ġelectronics Ġand Ġbeauty Ġare Ġvery Ġdifferent Ġdomains Ġand Ġeach Ġof Ġthem Ġhas Ġa Ġdiverse Ġset Ġof Ġdomain - specific Ġsentiment Ġexpressions . ĠFor Ġeach Ġmethod , Ġwe Ġidentify Ġthe Ġtop Ġ10 Ġmost Ġrelated Ġfilters Ġfor Ġeach Ġsentiment Ġlabel , Ġand Ġextract Ġthe Ġtop Ġtrig rams Ġof Ġeach Ġselected Ġfilter Ġon Ġboth Ġsource Ġand Ġtarget Ġdomains . ĠSince Ġlabeled Ġsource Ġexamples Ġare Ġused Ġfor Ġtraining , Ġwe Ġfind Ġthe Ġfilters Ġlearned Ġby Ġthe Ġthree Ġmethods Ġcapture Ġsimilar Ġexpressions Ġon Ġthe Ġsource Ġdomain , Ġcontaining Ġboth Ġdomain - inv ari ant Ġand Ġdomain - specific Ġtrig rams . ĠOn Ġthe Ġtarget Ġdomain , ĠD AS Ġcaptures Ġmore Ġtarget - specific Ġexpressions Ġcompared Ġto Ġthe Ġother Ġtwo Ġmethods . ĠDue Ġto Ġspace Ġlimitation , Ġwe Ġonly Ġpresent Ġa Ġsmall Ġsubset Ġof Ġpositive - sent iment - related Ġfilters Ġin ĠTable ĠT AB REF 34 Ġ. ĠThe Ġcomplete Ġresults Ġare Ġprovided Ġin ĠAppendix ĠSEC REF 8 Ġ. ĠFrom ĠTable ĠT AB REF 34 Ġ, Ġwe Ġcan Ġobserve Ġthat Ġthe Ġfilters Ġlearned Ġby ĠNa ive NN Ġare Ġalmost Ġunable Ġto Ġcapture Ġtarget - specific Ġsentiment Ġexpressions , Ġwhile ĠF ANN Ġis Ġable Ġto Ġcapture Ġlimited Ġtarget - specific Ġwords Ġsuch Ġas ĠâĢ ľ clean âĢ Ŀ Ġand ĠâĢ ľ s cent âĢ Ŀ . ĠThe Ġfilters Ġlearned Ġby ĠD AS Ġare Ġmore Ġdomain - adapt ive , Ġcapturing Ġdiverse Ġsentiment Ġexpressions Ġin Ġthe Ġtarget Ġdomain . </s> Conclusion </s> In Ġthis Ġwork , Ġwe Ġpropose ĠD AS , Ġa Ġnovel Ġframework Ġthat Ġjointly Ġperforms Ġfeature Ġadaptation Ġand Ġsemi - super vised Ġlearning . ĠWe Ġhave Ġdemonstrated Ġthrough Ġmultiple Ġexperiments Ġthat ĠD AS Ġcan Ġbetter Ġleverage Ġunl abel ed Ġdata , Ġand Ġachieve Ġsubstantial Ġimprovements Ġover Ġbaseline Ġmethods . ĠWe Ġhave Ġalso Ġshown Ġthat Ġfeature Ġadaptation Ġis Ġan Ġessential Ġcomponent , Ġwithout Ġwhich , Ġsemi - super vised Ġlearning Ġis Ġnot Ġable Ġto Ġfunction Ġproperly . ĠThe Ġproposed Ġframework Ġcould Ġbe Ġpotentially Ġadapted Ġto Ġother Ġdomain Ġadaptation Ġtasks , Ġwhich Ġis Ġthe Ġfocus Ġof Ġour Ġfuture Ġstudies . </s> Results Ġon ĠAmazon ĠBench mark </s> Most Ġprevious Ġworks ĠB IB REF 0 Ġ, ĠB IB REF 1 Ġ, ĠB IB REF 6 Ġ, ĠB IB REF 7 Ġ, ĠB IB REF 29 Ġcarried Ġout Ġexperiments Ġon Ġthe ĠAmazon Ġbenchmark Ġreleased Ġby ĠBl itzer Ġet Ġal . Ġ( ĠB IB REF 0 Ġ). ĠThe Ġdataset Ġcontains Ġ4 Ġdifferent Ġdomains : ĠBook Ġ( B ), ĠDVDs Ġ( D ), ĠElectronics Ġ( E ), Ġand ĠKitchen Ġ( K ). ĠFollowing Ġtheir Ġexperimental Ġsettings , Ġwe Ġconsider Ġthe Ġbinary Ġclassification Ġtask Ġto Ġpredict Ġwhether Ġa Ġreview Ġis Ġpositive Ġor Ġnegative Ġon Ġthe Ġtarget Ġdomain . ĠEach Ġdomain Ġconsists Ġof Ġ1000 Ġpositive Ġand Ġ1000 Ġnegative Ġreviews Ġrespectively . ĠWe Ġalso Ġallow Ġ4000 Ġunl abel ed Ġreviews Ġto Ġbe Ġused Ġfor Ġboth Ġthe Ġsource Ġand Ġthe Ġtarget Ġdomains , Ġof Ġwhich Ġthe Ġpositive Ġand Ġnegative Ġreviews Ġare Ġbalanced Ġas Ġwell , Ġfollowing Ġthe Ġsettings Ġin Ġprevious Ġworks . ĠWe Ġconstruct Ġ12 Ġcross - domain Ġsentiment Ġclassification Ġtasks Ġand Ġsplit Ġthe Ġlabeled Ġdata Ġin Ġeach Ġdomain Ġinto Ġa Ġtraining Ġset Ġof Ġ1600 Ġreviews Ġand Ġa Ġtest Ġset Ġof Ġ400 Ġreviews . ĠThe Ġclass ifier Ġis Ġtrained Ġon Ġthe Ġtraining Ġset Ġof Ġthe Ġsource Ġdomain Ġand Ġis Ġevaluated Ġon Ġthe Ġtest Ġset Ġof Ġthe Ġtarget Ġdomain . ĠThe Ġcomparison Ġresults Ġare Ġshown Ġin ĠTable ĠT AB REF 37 Ġ. </s> N umer ical ĠResults Ġof ĠFigure Ġ </s> Due Ġto Ġspace Ġlimitation , Ġwe Ġonly Ġshow Ġresults Ġin Ġfigures Ġin Ġthe Ġpaper . ĠAll Ġnumerical Ġnumbers Ġused Ġfor Ġplotting ĠFigure ĠFIG REF 17 Ġare Ġpresented Ġin ĠTable ĠT AB REF 38 Ġ. ĠWe Ġcan Ġobserve Ġthat ĠD AS - EM , ĠD AS - SE , Ġand ĠD AS Ġall Ġachieve Ġsubstantial Ġimprovements Ġover Ġbaseline Ġmethods Ġunder Ġdifferent Ġsettings . </s> CNN ĠFilter ĠAnalysis ĠFull ĠResults </s> As Ġmentioned Ġin ĠSection ĠSEC REF 36 Ġ, Ġwe Ġconduct ĠCNN Ġfilter Ġanalysis Ġon ĠNa ive NN , ĠF ANN , Ġand ĠD AS . ĠFor Ġeach Ġmethod , Ġwe Ġidentify Ġthe Ġtop Ġ10 Ġmost Ġrelated Ġfilters Ġfor Ġpositive , Ġnegative , Ġneutral Ġsentiment Ġlabels Ġrespectively , Ġand Ġthen Ġrepresent Ġeach Ġselected Ġfilter Ġas Ġa Ġranked Ġlist Ġof Ġtrig rams Ġwith Ġthe Ġhighest Ġactivation Ġvalues Ġon Ġit . ĠTable ĠT AB REF 39 Ġ, ĠT AB REF 40 Ġ, ĠT AB REF 41 Ġin Ġthe Ġfollowing Ġpages Ġillustrate Ġthe Ġtrig rams Ġfrom Ġthe Ġtarget Ġdomain Ġ( beaut y ) Ġcaptured Ġby Ġthe Ġselected Ġfilters Ġlearned Ġon ĠE ĠIN LINE FORM 0 ĠBT Ġfor Ġeach Ġmethod . </s> We Ġcan Ġobserve Ġthat Ġcompared Ġto ĠNa ive NN Ġand ĠF ANN , ĠD AS Ġis Ġable Ġto Ġcapture Ġa Ġmore Ġdiverse Ġset Ġof Ġrelevant Ġsentiment Ġexpressions Ġon Ġthe Ġtarget Ġdomain Ġfor Ġeach Ġsentiment Ġlabel . ĠThis Ġobservation Ġis Ġconsistent Ġwith Ġour Ġmotivation . ĠSince ĠNa ive NN , ĠF ANN Ġand Ġother Ġbaseline Ġmethods Ġsolely Ġtrain Ġthe Ġsentiment Ġclass ifier Ġon Ġthe Ġsource Ġdomain , Ġthe Ġlearned Ġenc oder Ġis Ġnot Ġable Ġto Ġproduce Ġdiscrim in ative Ġfeatures Ġon Ġthe Ġtarget Ġdomain . ĠD AS Ġaddresses Ġthis Ġproblem Ġby Ġrefining Ġthe Ġclass ifier Ġon Ġthe Ġtarget Ġdomain Ġwith Ġsemi - super vised Ġlearning , Ġand Ġthe Ġoverall Ġobjective Ġforces Ġthe Ġenc oder Ġto Ġlearn Ġfeature Ġrepresentations Ġthat Ġare Ġnot Ġonly Ġdomain - inv ari ant Ġbut Ġalso Ġdiscrim in ative Ġon Ġboth Ġdomains .
Introduction In practice, it is often difficult and costly to annotate sufficient training data for diverse application domains on-the-fly. We may have sufficient labeled data in an existing domain (called the source domain), but very few or no labeled data in a new domain (called the target domain). This issue has motivated research on cross-domain sentiment classification, where knowledge in the source domain is transferred to the target domain in order to alleviate the required labeling effort. One key challenge of domain adaptation is that data in the source and target domains are drawn from different distributions. Thus, adaptation performance will decline with an increase in distribution difference. Specifically, in sentiment analysis, reviews of different products have different vocabulary. For instance, restaurants reviews would contain opinion words such as “tender”, “tasty”, or “undercooked” and movie reviews would contain “thrilling”, “horrific”, or “hilarious”. The intersection between these two sets of opinion words could be small which makes domain adaptation difficult. Several techniques have been proposed for addressing the problem of domain shifting. The aim is to bridge the source and target domains by learning domain-invariant feature representations so that a classifier trained on a source domain can be adapted to another target domain. In cross-domain sentiment classification, many works BIBREF0 , BIBREF1 , BIBREF2 , BIBREF3 , BIBREF4 utilize a key intuition that domain-specific features could be aligned with the help of domain-invariant features (pivot features). For instance, “hilarious” and “tasty” could be aligned as both of them are relevant to “good”. Despite their promising results, these works share two major limitations. First, they highly depend on the heuristic selection of pivot features, which may be sensitive to different applications. Thus the learned new representations may not effectively reduce the domain difference. Furthermore, these works only utilize the unlabeled target data for representation learning while the sentiment classifier was solely trained on the source domain. There have not been many studies on exploiting unlabeled target data for refining the classifier, even though it may contain beneficial information. How to effectively leverage unlabeled target data still remains an important challenge for domain adaptation. In this work, we argue that the information from unlabeled target data is beneficial for domain adaptation and we propose a novel Domain Adaptive Semi-supervised learning framework (DAS) to better exploit it. Our main intuition is to treat the problem as a semi-supervised learning task by considering target instances as unlabeled data, assuming the domain distance can be effectively reduced through domain-invariant representation learning. Specifically, the proposed approach jointly performs feature adaptation and semi-supervised learning in a multi-task learning setting. For feature adaptation, it explicitly minimizes the distance between the encoded representations of the two domains. On this basis, two semi-supervised regularizations – entropy minimization and self-ensemble bootstrapping – are jointly employed to exploit unlabeled target data for classifier refinement. We evaluate our method rigorously under multiple experimental settings by taking label distribution and corpus size into consideration. The results show that our model is able to obtain significant improvements over strong baselines. We also demonstrate through a series of analysis that the proposed method benefits greatly from incorporating unlabeled target data via semi-supervised learning, which is consistent with our motivation. Our datasets and source code can be obtained from https://github.com/ruidan/DAS. Related Work Domain Adaptation: The majority of feature adaptation methods for sentiment analysis rely on a key intuition that even though certain opinion words are completely distinct for each domain, they can be aligned if they have high correlation with some domain-invariant opinion words (pivot words) such as “excellent” or “terrible”. Blitzer et al. ( BIBREF0 ) proposed a method based on structural correspondence learning (SCL), which uses pivot feature prediction to induce a projected feature space that works well for both the source and the target domains. The pivot words are selected in a way to cover common domain-invariant opinion words. Subsequent research aims to better align the domain-specific words BIBREF1 , BIBREF5 , BIBREF3 such that the domain discrepancy could be reduced. More recently, Yu and Jiang ( BIBREF4 ) borrow the idea of pivot feature prediction from SCL and extend it to a neural network-based solution with auxiliary tasks. In their experiment, substantial improvement over SCL has been observed due to the use of real-valued word embeddings. Unsupervised representation learning with deep neural networks (DNN) such as denoising autoencoders has also been explored for feature adaptation BIBREF6 , BIBREF7 , BIBREF8 . It has been shown that DNNs could learn transferable representations that disentangle the underlying factors of variation behind data samples. Although the aforementioned methods aim to reduce the domain discrepancy, they do not explicitly minimize the distance between distributions, and some of them highly rely on the selection of pivot features. In our method, we formally construct an objective for this purpose. Similar ideas have been explored in many computer vision problems, where the representations of the underlying domains are encouraged to be similar through explicit objectives BIBREF9 , BIBREF10 , BIBREF11 , BIBREF12 , BIBREF13 such as maximum mean discrepancy (MMD) BIBREF14 . In NLP tasks, Li et al. ( BIBREF15 ) and Chen et al. ( BIBREF16 ) both proposed using adversarial training framework for reducing domain difference. In their model, a sub-network is added as a domain discriminator while deep features are learned to confuse the discriminator. The feature adaptation component in our model shares similar intuition with MMD and adversary training. We will show a detailed comparison with them in our experiments. Semi-supervised Learning: We attempt to treat domain adaptation as a semi-supervised learning task by considering the target instances as unlabeled data. Some efforts have been initiated on transfer learning from unlabeled data BIBREF17 , BIBREF18 , BIBREF19 . In our model, we reduce the domain discrepancy by feature adaptation, and thereafter adopt semi-supervised learning techniques to learn from unlabeled data. Primarily motivated by BIBREF20 and BIBREF21 , we employed entropy minimization and self-ensemble bootstrapping as regularizations to incorporate unlabeled data. Our experimental results show that both methods are effective when jointly trained with the feature adaptation objective, which confirms to our motivation. Notations and Model Overview We conduct most of our experiments under an unsupervised domain adaptation setting, where we have no labeled data from the target domain. Consider two sets INLINEFORM0 and INLINEFORM1 . INLINEFORM2 is from the source domain with INLINEFORM3 labeled examples, where INLINEFORM4 is a one-hot vector representation of sentiment label and INLINEFORM5 denotes the number of classes. INLINEFORM6 is from the target domain with INLINEFORM7 unlabeled examples. INLINEFORM8 denotes the total number of training documents including both labeled and unlabeled. We aim to learn a sentiment classifier from INLINEFORM13 and INLINEFORM14 such that the classifier would work well on the target domain. We also present some results under a setting where we assume that a small number of labeled target examples are available (see Figure FIGREF27 ). For the proposed model, we denote INLINEFORM0 parameterized by INLINEFORM1 as a neural-based feature encoder that maps documents from both domains to a shared feature space, and INLINEFORM2 parameterized by INLINEFORM3 as a fully connected layer with softmax activation serving as the sentiment classifier. We aim to learn feature representations that are domain-invariant and at the same time discriminative on both domains, thus we simultaneously consider three factors in our objective: (1) minimize the classification error on the labeled source examples; (2) minimize the domain discrepancy; and (3) leverage unlabeled data via semi-supervised learning. Suppose we already have the encoded features of documents INLINEFORM0 (see Section SECREF10 ), the objective function for purpose (1) is thus the cross entropy loss on the labeled source examples DISPLAYFORM0 where INLINEFORM0 denotes the predicted label distribution. In the following subsections, we will explain how to perform feature adaptation and domain adaptive semi-supervised learning in details for purpose (2) and (3) respectively. Feature Adaptation Unlike prior works BIBREF0 , BIBREF4 , our method does not attempt to align domain-specific words through pivot words. In our preliminary experiments, we found that word embeddings pre-trained on a large corpus are able to adequately capture this information. As we will later show in our experiments, even without adaptation, a naive neural network classifier with pre-trained word embeddings can already achieve reasonably good results. We attempt to explicitly minimize the distance between the source and target feature representations ( INLINEFORM0 and INLINEFORM1 ). A few methods from literature can be applied such as Maximum Mean Discrepancy (MMD) BIBREF14 or adversary training BIBREF15 , BIBREF16 . The main idea of MMD is to estimate the distance between two distributions as the distance between sample means of the projected embeddings in Hilbert space. MMD is implicitly computed through a characteristic kernel, which is used to ensure that the sample mean is injective, leading to the MMD being zero if and only if the distributions are identical. In our implementation, we skip the mapping procedure induced by a characteristic kernel for simplifying the computation and learning. We simply estimate the distribution distance as the distance between the sample means in the current embedding space. Although this approximation cannot preserve all statistical features of the underlying distributions, we find it performs comparably to MMD on our problem. The following equations formally describe the feature adaptation loss INLINEFORM2 : DISPLAYFORM0 INLINEFORM0 normalization is applied on the mean representations INLINEFORM1 and INLINEFORM2 , rescaling the vectors such that all entries sum to 1. We adopt a symmetric version of KL divergence BIBREF12 as the distance function. Given two distribution vectors INLINEFORM3 , INLINEFORM4 . Domain Adaptive Semi-supervised Learning (DAS) We attempt to exploit the information in target data through semi-supervised learning objectives, which are jointly trained with INLINEFORM0 and INLINEFORM1 . Normally, to incorporate target data, we can minimize the cross entropy loss between the true label distributions INLINEFORM2 and the predicted label distributions INLINEFORM3 over target samples. The challenge here is that INLINEFORM4 is unknown, and thus we attempt to estimate it via semi-supervised learning. We use entropy minimization and bootstrapping for this purpose. We will later show in our experiments that both methods are effective, and jointly employing them overall yields the best results. Entropy Minimization: In this method, INLINEFORM0 is estimated as the predicted label distribution INLINEFORM1 , which is a function of INLINEFORM2 and INLINEFORM3 . The loss can thus be written as DISPLAYFORM0 Assume the domain discrepancy can be effectively reduced through feature adaptation, by minimizing the entropy penalty, training of the classifier is influenced by the unlabeled target data and will generally maximize the margins between the target examples and the decision boundaries, increasing the prediction confidence on the target domain. Self-ensemble Bootstrapping: Another way to estimate INLINEFORM0 corresponds to bootstrapping. The idea is to estimate the unknown labels as the predictions of the model learned from the previous round of training. Bootstrapping has been explored for domain adaptation in previous works BIBREF18 , BIBREF19 . However, in their methods, domain discrepancy was not explicitly minimized via feature adaptation. Applying bootstrapping or other semi-supervised learning techniques in this case may worsen the results as the classifier can perform quite bad on the target data. [t] Pseudocode for training DAS INLINEFORM0 , INLINEFORM1 , INLINEFORM2 , INLINEFORM3 INLINEFORM4 = ensembling momentum, INLINEFORM5 INLINEFORM6 = weight ramp-up function INLINEFORM7 INLINEFORM8 INLINEFORM9 each minibatch INLINEFORM10 , INLINEFORM11 , INLINEFORM12 in INLINEFORM0 , INLINEFORM1 , INLINEFORM2 compute loss INLINEFORM3 on INLINEFORM4 compute loss INLINEFORM5 on INLINEFORM6 compute loss INLINEFORM7 on INLINEFORM8 compute loss INLINEFORM9 on INLINEFORM10 INLINEFORM11 update network parameters INLINEFORM0 , for INLINEFORM1 INLINEFORM2 INLINEFORM3 Inspired by the ensembling method proposed in BIBREF21 , we estimate INLINEFORM0 by forming ensemble predictions of labels during training, using the outputs on different training epochs. The loss is formulated as follows: DISPLAYFORM0 where INLINEFORM0 denotes the estimated labels computed on the ensemble predictions from different epochs. The loss is applied on all documents. It serves for bootstrapping on the unlabeled target data, and it also serves as a regularization that encourages the network predictions to be consistent in different training epochs. INLINEFORM1 is jointly trained with INLINEFORM2 , INLINEFORM3 , and INLINEFORM4 . Algorithm SECREF6 illustrates the overall training process of the proposed domain adaptive semi-supervised learning (DAS) framework. In Algorithm SECREF6 , INLINEFORM0 , INLINEFORM1 , and INLINEFORM2 are weights to balance the effects of INLINEFORM3 , INLINEFORM4 , and INLINEFORM5 respectively. INLINEFORM6 and INLINEFORM7 are constant hyper-parameters. We set INLINEFORM8 as a Gaussian curve to ramp up the weight from 0 to INLINEFORM9 . This is to ensure the ramp-up of the bootstrapping loss component is slow enough in the beginning of the training. After each training epoch, we compute INLINEFORM10 which denotes the predictions made by the network in current epoch, and then the ensemble prediction INLINEFORM11 is updated as a weighted average of the outputs from previous epochs and the current epoch, with recent epochs having larger weight. For generating estimated labels INLINEFORM12 , INLINEFORM13 is converted to a one-hot vector where the entry with the maximum value is set to one and other entries are set to zeros. The self-ensemble bootstrapping is a generalized version of bootstrappings that only use the outputs from the previous round of training BIBREF18 , BIBREF19 . The ensemble prediction is likely to be closer to the correct, unknown labels of the target data. CNN Encoder Implementation We have left the feature encoder INLINEFORM0 unspecified, for which, a few options can be considered. In our implementation, we adopt a one-layer CNN structure from previous works BIBREF22 , BIBREF4 , as it has been demonstrated to work well for sentiment classification tasks. Given a review document INLINEFORM1 consisting of INLINEFORM2 words, we begin by associating each word with a continuous word embedding BIBREF23 INLINEFORM3 from an embedding matrix INLINEFORM4 , where INLINEFORM5 is the vocabulary size and INLINEFORM6 is the embedding dimension. INLINEFORM7 is jointly updated with other network parameters during training. Given a window of dense word embeddings INLINEFORM8 , the convolution layer first concatenates these vectors to form a vector INLINEFORM9 of length INLINEFORM10 and then the output vector is computed by Equation ( EQREF11 ): DISPLAYFORM0 INLINEFORM0 , INLINEFORM1 is the parameter set of the encoder INLINEFORM2 and is shared across all windows of the sequence. INLINEFORM3 is an element-wise non-linear activation function. The convolution operation can capture local contextual dependencies of the input sequence and the extracted feature vectors are similar to INLINEFORM4 -grams. After the convolution operation is applied to the whole sequence, we obtain a list of hidden vectors INLINEFORM5 . A max-over-time pooling layer is applied to obtain the final vector representation INLINEFORM6 of the input document. Datasets and Experimental Settings Existing benchmark datasets such as the Amazon benchmark BIBREF0 typically remove reviews with neutral labels in both domains. This is problematic as the label information of the target domain is not accessible in an unsupervised domain adaptation setting. Furthermore, removing neutral instances may bias the dataset favorably for max-margin-based algorithms like ours, since the resulting dataset has all uncertain labels removed, leaving only high confidence examples. Therefore, we construct new datasets by ourselves. The results on the original Amazon benchmark is qualitatively similar, and we present them in Appendix SECREF6 for completeness since most of previous works reported results on it. Small-scale datasets: Our new dataset was derived from the large-scale Amazon datasets released by McAuley et al. ( BIBREF24 ). It contains four domains: Book (BK), Electronics (E), Beauty (BT), and Music (M). Each domain contains two datasets. Set 1 contains 6000 instances with exactly balanced class labels, and set 2 contains 6000 instances that are randomly sampled from the large dataset, preserving the original label distribution, which we believe better reflects the label distribution in real life. The examples in these two sets do not overlap. Detailed statistics of the generated datasets are given in Table TABREF9 . In all our experiments on the small-scale datasets, we use set 1 of the source domain as the only source with sentiment label information during training, and we evaluate the trained model on set 1 of the target domain. Since we cannot control the label distribution of unlabeled data during training, we consider two different settings: Setting (1): Only set 1 of the target domain is used as the unlabeled set. This tells us how the method performs in a condition when the target domain has a close-to-balanced label distribution. As we also evaluate on set 1 of the target domain, this is also considered as a transductive setting. Setting (2): Set 2 from both the source and target domains are used as unlabeled sets. Since set 2 is directly sampled from millions of reviews, it better reflects real-life sentiment distribution. Large-scale datasets: We further conduct experiments on four much larger datasets: IMDB (I), Yelp2014 (Y), Cell Phone (C), and Baby (B). IMDB and Yelp2014 were previously used in BIBREF25 , BIBREF26 . Cell phone and Baby are from the large-scale Amazon dataset BIBREF24 , BIBREF27 . Detailed statistics are summarized in Table TABREF9 . We keep all reviews in the original datasets and consider a transductive setting where all target examples are used for both training (without label information) and evaluation. We perform sampling to balance the classes of labeled source data in each minibatch INLINEFORM3 during training. Selection of Development Set Ideally, the development set should be drawn from the same distribution as the test set. However, under the unsupervised domain adaptation setting, we do not have any labeled target data at training phase which could be used as development set. In all of our experiments, for each pair of domains, we instead sample 1000 examples from the training set of the source domain as development set. We train the network for a fixed number of epochs, and the model with the minimum classification error on this development set is saved for evaluation. This approach works well on most of the problems since the target domain is supposed to behave like the source domain if the domain difference is effectively reduced. Another problem is how to select the values for hyper-parameters. If we tune INLINEFORM0 and INLINEFORM1 directly on the development set from the source domain, most likely both of them will be set to 0, as unlabeled target data is not helpful for improving in-domain accuracy of the source domain. Other neural network models also have the same problem for hyper-parameter tuning. Therefore, our strategy is to use the development set from the target domain to optimize INLINEFORM2 and INLINEFORM3 for one problem (e.g., we only do this on E INLINEFORM4 BK), and fix their values on the other problems. This setting assumes that we have at least two labeled domains such that we can optimize the hyper-parameters, and then we fix them for other new unlabeled domains to transfer to. Training Details and Hyper-parameters We initialize word embeddings using the 300-dimension GloVe vectors supplied by Pennington et al., ( BIBREF28 ), which were trained on 840 billion tokens from the Common Crawl. For each pair of domains, the vocabulary consists of the top 10000 most frequent words. For words in the vocabulary but not present in the pre-trained embeddings, we randomly initialize them. We set hyper-parameters of the CNN encoder following previous works BIBREF22 , BIBREF4 without specific tuning on our datasets. The window size is set to 3 and the size of the hidden layer is set to 300. The nonlinear activation function is Relu. For regularization, we also follow their settings and employ dropout with probability set to 0.5 on INLINEFORM0 before feeding it to the output layer INLINEFORM1 , and constrain the INLINEFORM2 -norm of the weight vector INLINEFORM3 , setting its max norm to 3. On the small-scale datasets and the Aamzon benchmark, INLINEFORM0 and INLINEFORM1 are set to 200 and 1, respectively, tuned on the development set of task E INLINEFORM2 BK under setting 1. On the large-scale datasets, INLINEFORM3 and INLINEFORM4 are set to 500 and 0.2, respectively, tuned on I INLINEFORM5 Y. We use a Gaussian curve INLINEFORM6 to ramp up the weight of the bootstrapping loss INLINEFORM7 from 0 to INLINEFORM8 , where INLINEFORM9 denotes the maximum number of training epochs. We train 30 epochs for all experiments. We set INLINEFORM10 to 3 and INLINEFORM11 to 0.5 for all experiments. The batch size is set to 50 on the small-scale datasets and the Amazon benchmark. We increase the batch size to 250 on the large-scale datasets to reduce the number of iterations. RMSProp optimizer with learning rate set to 0.0005 is used for all experiments. Models for Comparison We compare with the following baselines: (1) Naive: A non-domain-adaptive baseline with bag-of-words representations and SVM classifier trained on the source domain. (2) mSDA BIBREF7 : This is the state-of-the-art method based on discrete input features. Top 1000 bag-of-words features are kept as pivot features. We set the number of stacked layers to 3 and the corruption probability to 0.5. (3) NaiveNN: This is a non-domain-adaptive CNN trained on source domain, which is a variant of our model by setting INLINEFORM0 , INLINEFORM1 , and INLINEFORM2 to zeros. (4) AuxNN BIBREF4 : This is a neural model that exploits auxiliary tasks, which has achieved state-of-the-art results on cross-domain sentiment classification. The sentence encoder used in this model is the same as ours. (5) ADAN BIBREF16 : This method exploits adversarial training to reduce representation difference between domains. The original paper uses a simple feedforward network as encoder. For fair comparison, we replace it with our CNN-based encoder. We train 5 iterations on the discriminator per iteration on the encoder and sentiment classifier as suggested in their paper. (6) MMD: MMD has been widely used for minimizing domain discrepancy on images. In those works BIBREF9 , BIBREF13 , variants of deep CNNs are used for encoding images and the MMDs of multiple layers are jointly minimized. In NLP, adding more layers of CNNs may not be very helpful and thus those models from image-related tasks can not be directly applied to our problem. To compare with MMD-based method, we train a model that jointly minimize the classification loss INLINEFORM0 on the source domain and MMD between INLINEFORM1 and INLINEFORM2 . For computing MMD, we use a Gaussian RBF which is a common choice for characteristic kernel. In addition to the above baselines, we also show results of different variants of our model. DAS as shown in Algorithm SECREF6 denotes our full model. DAS-EM denotes the model with only entropy minimization for semi-supervised learning (set INLINEFORM0 ). DAS-SE denotes the model with only self-ensemble bootstrapping for semi-supervised learning (set INLINEFORM1 ). FANN (feature-adaptation neural network) denotes the model without semi-supervised learning performed (set both INLINEFORM2 and INLINEFORM3 to zeros). Main Results Figure FIGREF17 shows the comparison of adaptation results (see Appendix SECREF7 for the exact numerical numbers). We report classification accuracy on the small-scale dataset. For the large-scale dataset, macro-F1 is instead used since the label distribution in the test set is extremely unbalanced. Key observations are summarized as follows. (1) Both DAS-EM and DAS-SE perform better in most cases compared with ADAN, MDD, and FANN, in which only feature adaptation is performed. This demonstrates the effectiveness of the proposed domain adaptive semi-supervised learning framework. DAS-EM is more effective than DAS-SE in most cases, and the full model DAS with both techniques jointly employed overall has the best performance. (2) When comparing the two settings on the small-scale dataset, all domain-adaptive methods generally perform better under setting 1. In setting 1, the target examples are balanced in classes, which can provide more diverse opinion-related features. However, when considering unsupervised domain adaptation, we should not presume the label distribution of the unlabeled data. Thus, it is necessary to conduct experiments using datasets that reflect real-life sentiment distribution as what we did on setting2 and the large-scale dataset. Unfortunately, this is ignored by most of previous works. (3) Word-embeddings are very helpful, as we can see even NaiveNN can substantially outperform mSDA on most tasks. To see the effect of semi-supervised learning alone, we also conduct experiments by setting INLINEFORM0 to eliminate the effect of feature adaptation. Both entropy minimization and bootstrapping perform very badly in this setting. Entropy minimization gives almost random predictions with accuracy below 0.4, and the results of bootstrapping are also much lower compared to NaiveNN. This suggests that the feature adaptation component is essential. Without it, the learned target representations are less meaningful and discriminative. Applying semi-supervised learning in this case is likely to worsen the results. Further Analysis In Figure FIGREF23 , we show the change of accuracy with respect to the percentage of unlabeled data used for training on three particular problems under setting 1. The value at INLINEFORM0 denotes the accuracies of NaiveNN which does not utilize any target data. For DAS, we observe a nonlinear increasing trend where the accuracy quickly improves at the beginning, and then gradually stabilizes. For other methods, this trend is less obvious, and adding more unlabeled data sometimes even worsen the results. This finding again suggests that the proposed approach can better exploit the information from unlabeled data. We also conduct experiments under a setting with a small number of labeled target examples available. Figure FIGREF27 shows the change of accuracy with respect to the number of labeled target examples added for training. We can observe that DAS is still more effective under this setting, while the performance differences to other methods gradually decrease with the increasing number of labeled target examples. CNN Filter Analysis In this subsection, we aim to better understand DAS by analyzing sentiment-related CNN filters. To do that, 1) we first select a list of the most related CNN filters for predicting each sentiment label (positive, negative neutral). Those filters can be identified according to the learned weights INLINEFORM0 of the output layer INLINEFORM1 . Higher weight indicates stronger relatedness. 2) Recall that in our implementation, each CNN filter has a window size of 3 with Relu activation. We can thus represent each selected filter as a ranked list of trigrams with highest activation values. We analyze the CNN filters learned by NaiveNN, FANN and DAS respectively on task E INLINEFORM0 BT under setting 1. We focus on E INLINEFORM1 BT for study because electronics and beauty are very different domains and each of them has a diverse set of domain-specific sentiment expressions. For each method, we identify the top 10 most related filters for each sentiment label, and extract the top trigrams of each selected filter on both source and target domains. Since labeled source examples are used for training, we find the filters learned by the three methods capture similar expressions on the source domain, containing both domain-invariant and domain-specific trigrams. On the target domain, DAS captures more target-specific expressions compared to the other two methods. Due to space limitation, we only present a small subset of positive-sentiment-related filters in Table TABREF34 . The complete results are provided in Appendix SECREF8 . From Table TABREF34 , we can observe that the filters learned by NaiveNN are almost unable to capture target-specific sentiment expressions, while FANN is able to capture limited target-specific words such as “clean” and “scent”. The filters learned by DAS are more domain-adaptive, capturing diverse sentiment expressions in the target domain. Conclusion In this work, we propose DAS, a novel framework that jointly performs feature adaptation and semi-supervised learning. We have demonstrated through multiple experiments that DAS can better leverage unlabeled data, and achieve substantial improvements over baseline methods. We have also shown that feature adaptation is an essential component, without which, semi-supervised learning is not able to function properly. The proposed framework could be potentially adapted to other domain adaptation tasks, which is the focus of our future studies. Results on Amazon Benchmark Most previous works BIBREF0 , BIBREF1 , BIBREF6 , BIBREF7 , BIBREF29 carried out experiments on the Amazon benchmark released by Blitzer et al. ( BIBREF0 ). The dataset contains 4 different domains: Book (B), DVDs (D), Electronics (E), and Kitchen (K). Following their experimental settings, we consider the binary classification task to predict whether a review is positive or negative on the target domain. Each domain consists of 1000 positive and 1000 negative reviews respectively. We also allow 4000 unlabeled reviews to be used for both the source and the target domains, of which the positive and negative reviews are balanced as well, following the settings in previous works. We construct 12 cross-domain sentiment classification tasks and split the labeled data in each domain into a training set of 1600 reviews and a test set of 400 reviews. The classifier is trained on the training set of the source domain and is evaluated on the test set of the target domain. The comparison results are shown in Table TABREF37 . Numerical Results of Figure  Due to space limitation, we only show results in figures in the paper. All numerical numbers used for plotting Figure FIGREF17 are presented in Table TABREF38 . We can observe that DAS-EM, DAS-SE, and DAS all achieve substantial improvements over baseline methods under different settings. CNN Filter Analysis Full Results As mentioned in Section SECREF36 , we conduct CNN filter analysis on NaiveNN, FANN, and DAS. For each method, we identify the top 10 most related filters for positive, negative, neutral sentiment labels respectively, and then represent each selected filter as a ranked list of trigrams with the highest activation values on it. Table TABREF39 , TABREF40 , TABREF41 in the following pages illustrate the trigrams from the target domain (beauty) captured by the selected filters learned on E INLINEFORM0 BT for each method. We can observe that compared to NaiveNN and FANN, DAS is able to capture a more diverse set of relevant sentiment expressions on the target domain for each sentiment label. This observation is consistent with our motivation. Since NaiveNN, FANN and other baseline methods solely train the sentiment classifier on the source domain, the learned encoder is not able to produce discriminative features on the target domain. DAS addresses this problem by refining the classifier on the target domain with semi-supervised learning, and the overall objective forces the encoder to learn feature representations that are not only domain-invariant but also discriminative on both domains.
What are the source and target domains?
9176d2ba1c638cdec334971c4c7f1bb959495a8e
What Ġare Ġthe Ġsource Ġand Ġtarget Ġdomains ?
Book , Ġelectronics , Ġbeauty , Ġmusic , ĠIM DB , ĠYelp , Ġcell Ġphone , Ġbaby , ĠDVDs , Ġkitchen
<s> What Ġare Ġthe Ġsource Ġand Ġtarget Ġdomains ? </s> Introduction </s> In Ġpractice , Ġit Ġis Ġoften Ġdifficult Ġand Ġcostly Ġto Ġannot ate Ġsufficient Ġtraining Ġdata Ġfor Ġdiverse Ġapplication Ġdomains Ġon - the - fly . ĠWe Ġmay Ġhave Ġsufficient Ġlabeled Ġdata Ġin Ġan Ġexisting Ġdomain Ġ( called Ġthe Ġsource Ġdomain ), Ġbut Ġvery Ġfew Ġor Ġno Ġlabeled Ġdata Ġin Ġa Ġnew Ġdomain Ġ( called Ġthe Ġtarget Ġdomain ). ĠThis Ġissue Ġhas Ġmotivated Ġresearch Ġon Ġcross - domain Ġsentiment Ġclassification , Ġwhere Ġknowledge Ġin Ġthe Ġsource Ġdomain Ġis Ġtransferred Ġto Ġthe Ġtarget Ġdomain Ġin Ġorder Ġto Ġalleviate Ġthe Ġrequired Ġlabeling Ġeffort . </s> One Ġkey Ġchallenge Ġof Ġdomain Ġadaptation Ġis Ġthat Ġdata Ġin Ġthe Ġsource Ġand Ġtarget Ġdomains Ġare Ġdrawn Ġfrom Ġdifferent Ġdistributions . ĠThus , Ġadaptation Ġperformance Ġwill Ġdecline Ġwith Ġan Ġincrease Ġin Ġdistribution Ġdifference . ĠSpecifically , Ġin Ġsentiment Ġanalysis , Ġreviews Ġof Ġdifferent Ġproducts Ġhave Ġdifferent Ġvocabulary . ĠFor Ġinstance , Ġrestaurants Ġreviews Ġwould Ġcontain Ġopinion Ġwords Ġsuch Ġas ĠâĢ ľ t ender âĢ Ŀ , ĠâĢ ľ t asty âĢ Ŀ , Ġor ĠâĢ ľ under cooked âĢ Ŀ Ġand Ġmovie Ġreviews Ġwould Ġcontain ĠâĢ ľ th r illing âĢ Ŀ , ĠâĢ ľ hor r ific âĢ Ŀ , Ġor ĠâĢ ľ h ilar ious âĢ Ŀ . ĠThe Ġintersection Ġbetween Ġthese Ġtwo Ġsets Ġof Ġopinion Ġwords Ġcould Ġbe Ġsmall Ġwhich Ġmakes Ġdomain Ġadaptation Ġdifficult . </s> Several Ġtechniques Ġhave Ġbeen Ġproposed Ġfor Ġaddressing Ġthe Ġproblem Ġof Ġdomain Ġshifting . ĠThe Ġaim Ġis Ġto Ġbridge Ġthe Ġsource Ġand Ġtarget Ġdomains Ġby Ġlearning Ġdomain - inv ari ant Ġfeature Ġrepresentations Ġso Ġthat Ġa Ġclass ifier Ġtrained Ġon Ġa Ġsource Ġdomain Ġcan Ġbe Ġadapted Ġto Ġanother Ġtarget Ġdomain . ĠIn Ġcross - domain Ġsentiment Ġclassification , Ġmany Ġworks ĠB IB REF 0 Ġ, ĠB IB REF 1 Ġ, ĠB IB REF 2 Ġ, ĠB IB REF 3 Ġ, ĠB IB REF 4 Ġutilize Ġa Ġkey Ġintuition Ġthat Ġdomain - specific Ġfeatures Ġcould Ġbe Ġaligned Ġwith Ġthe Ġhelp Ġof Ġdomain - inv ari ant Ġfeatures Ġ( p ivot Ġfeatures ). ĠFor Ġinstance , ĠâĢ ľ h ilar ious âĢ Ŀ Ġand ĠâĢ ľ t asty âĢ Ŀ Ġcould Ġbe Ġaligned Ġas Ġboth Ġof Ġthem Ġare Ġrelevant Ġto ĠâĢ ľ good âĢ Ŀ . </s> Despite Ġtheir Ġpromising Ġresults , Ġthese Ġworks Ġshare Ġtwo Ġmajor Ġlimitations . ĠFirst , Ġthey Ġhighly Ġdepend Ġon Ġthe Ġhe uristic Ġselection Ġof Ġpivot Ġfeatures , Ġwhich Ġmay Ġbe Ġsensitive Ġto Ġdifferent Ġapplications . ĠThus Ġthe Ġlearned Ġnew Ġrepresentations Ġmay Ġnot Ġeffectively Ġreduce Ġthe Ġdomain Ġdifference . ĠFurthermore , Ġthese Ġworks Ġonly Ġutilize Ġthe Ġunl abel ed Ġtarget Ġdata Ġfor Ġrepresentation Ġlearning Ġwhile Ġthe Ġsentiment Ġclass ifier Ġwas Ġsolely Ġtrained Ġon Ġthe Ġsource Ġdomain . ĠThere Ġhave Ġnot Ġbeen Ġmany Ġstudies Ġon Ġexploiting Ġunl abel ed Ġtarget Ġdata Ġfor Ġrefining Ġthe Ġclass ifier , Ġeven Ġthough Ġit Ġmay Ġcontain Ġbeneficial Ġinformation . ĠHow Ġto Ġeffectively Ġleverage Ġunl abel ed Ġtarget Ġdata Ġstill Ġremains Ġan Ġimportant Ġchallenge Ġfor Ġdomain Ġadaptation . </s> In Ġthis Ġwork , Ġwe Ġargue Ġthat Ġthe Ġinformation Ġfrom Ġunl abel ed Ġtarget Ġdata Ġis Ġbeneficial Ġfor Ġdomain Ġadaptation Ġand Ġwe Ġpropose Ġa Ġnovel ĠDomain ĠAdapt ive ĠSemi - super vised Ġlearning Ġframework Ġ( D AS ) Ġto Ġbetter Ġexploit Ġit . ĠOur Ġmain Ġintuition Ġis Ġto Ġtreat Ġthe Ġproblem Ġas Ġa Ġsemi - super vised Ġlearning Ġtask Ġby Ġconsidering Ġtarget Ġinstances Ġas Ġunl abel ed Ġdata , Ġassuming Ġthe Ġdomain Ġdistance Ġcan Ġbe Ġeffectively Ġreduced Ġthrough Ġdomain - inv ari ant Ġrepresentation Ġlearning . ĠSpecifically , Ġthe Ġproposed Ġapproach Ġjointly Ġperforms Ġfeature Ġadaptation Ġand Ġsemi - super vised Ġlearning Ġin Ġa Ġmulti - task Ġlearning Ġsetting . ĠFor Ġfeature Ġadaptation , Ġit Ġexplicitly Ġminim izes Ġthe Ġdistance Ġbetween Ġthe Ġencoded Ġrepresentations Ġof Ġthe Ġtwo Ġdomains . ĠOn Ġthis Ġbasis , Ġtwo Ġsemi - super vised Ġregular izations ĠâĢĵ Ġentropy Ġminim ization Ġand Ġself - ense mble Ġboot stra pping ĠâĢĵ Ġare Ġjointly Ġemployed Ġto Ġexploit Ġunl abel ed Ġtarget Ġdata Ġfor Ġclass ifier Ġrefinement . </s> We Ġevaluate Ġour Ġmethod Ġrig orously Ġunder Ġmultiple Ġexperimental Ġsettings Ġby Ġtaking Ġlabel Ġdistribution Ġand Ġcorpus Ġsize Ġinto Ġconsideration . ĠThe Ġresults Ġshow Ġthat Ġour Ġmodel Ġis Ġable Ġto Ġobtain Ġsignificant Ġimprovements Ġover Ġstrong Ġbas elines . ĠWe Ġalso Ġdemonstrate Ġthrough Ġa Ġseries Ġof Ġanalysis Ġthat Ġthe Ġproposed Ġmethod Ġbenefits Ġgreatly Ġfrom Ġincorporating Ġunl abel ed Ġtarget Ġdata Ġvia Ġsemi - super vised Ġlearning , Ġwhich Ġis Ġconsistent Ġwith Ġour Ġmotivation . ĠOur Ġdatasets Ġand Ġsource Ġcode Ġcan Ġbe Ġobtained Ġfrom Ġhttps :// github . com / ru idan / D AS . </s> Related ĠWork </s> Domain ĠAdapt ation : ĠThe Ġmajority Ġof Ġfeature Ġadaptation Ġmethods Ġfor Ġsentiment Ġanalysis Ġrely Ġon Ġa Ġkey Ġintuition Ġthat Ġeven Ġthough Ġcertain Ġopinion Ġwords Ġare Ġcompletely Ġdistinct Ġfor Ġeach Ġdomain , Ġthey Ġcan Ġbe Ġaligned Ġif Ġthey Ġhave Ġhigh Ġcorrelation Ġwith Ġsome Ġdomain - inv ari ant Ġopinion Ġwords Ġ( p ivot Ġwords ) Ġsuch Ġas ĠâĢ ľ ex cellent âĢ Ŀ Ġor ĠâĢ ľ ter rible âĢ Ŀ . ĠBl itzer Ġet Ġal . Ġ( ĠB IB REF 0 Ġ) Ġproposed Ġa Ġmethod Ġbased Ġon Ġstructural Ġcorrespondence Ġlearning Ġ( S CL ), Ġwhich Ġuses Ġpivot Ġfeature Ġprediction Ġto Ġinduce Ġa Ġprojected Ġfeature Ġspace Ġthat Ġworks Ġwell Ġfor Ġboth Ġthe Ġsource Ġand Ġthe Ġtarget Ġdomains . ĠThe Ġpivot Ġwords Ġare Ġselected Ġin Ġa Ġway Ġto Ġcover Ġcommon Ġdomain - inv ari ant Ġopinion Ġwords . ĠSub sequent Ġresearch Ġaims Ġto Ġbetter Ġalign Ġthe Ġdomain - specific Ġwords ĠB IB REF 1 Ġ, ĠB IB REF 5 Ġ, ĠB IB REF 3 Ġsuch Ġthat Ġthe Ġdomain Ġdiscrepancy Ġcould Ġbe Ġreduced . ĠMore Ġrecently , ĠYu Ġand ĠJiang Ġ( ĠB IB REF 4 Ġ) Ġborrow Ġthe Ġidea Ġof Ġpivot Ġfeature Ġprediction Ġfrom ĠS CL Ġand Ġextend Ġit Ġto Ġa Ġneural Ġnetwork - based Ġsolution Ġwith Ġauxiliary Ġtasks . ĠIn Ġtheir Ġexperiment , Ġsubstantial Ġimprovement Ġover ĠS CL Ġhas Ġbeen Ġobserved Ġdue Ġto Ġthe Ġuse Ġof Ġreal - valued Ġword Ġembed d ings . ĠUn super vised Ġrepresentation Ġlearning Ġwith Ġdeep Ġneural Ġnetworks Ġ( D NN ) Ġsuch Ġas Ġden o ising Ġaut oen cod ers Ġhas Ġalso Ġbeen Ġexplored Ġfor Ġfeature Ġadaptation ĠB IB REF 6 Ġ, ĠB IB REF 7 Ġ, ĠB IB REF 8 Ġ. ĠIt Ġhas Ġbeen Ġshown Ġthat ĠD NN s Ġcould Ġlearn Ġtransfer able Ġrepresentations Ġthat Ġdis ent angle Ġthe Ġunderlying Ġfactors Ġof Ġvariation Ġbehind Ġdata Ġsamples . </s> Although Ġthe Ġaforementioned Ġmethods Ġaim Ġto Ġreduce Ġthe Ġdomain Ġdiscrepancy , Ġthey Ġdo Ġnot Ġexplicitly Ġminimize Ġthe Ġdistance Ġbetween Ġdistributions , Ġand Ġsome Ġof Ġthem Ġhighly Ġrely Ġon Ġthe Ġselection Ġof Ġpivot Ġfeatures . ĠIn Ġour Ġmethod , Ġwe Ġformally Ġconstruct Ġan Ġobjective Ġfor Ġthis Ġpurpose . ĠSimilar Ġideas Ġhave Ġbeen Ġexplored Ġin Ġmany Ġcomputer Ġvision Ġproblems , Ġwhere Ġthe Ġrepresentations Ġof Ġthe Ġunderlying Ġdomains Ġare Ġencouraged Ġto Ġbe Ġsimilar Ġthrough Ġexplicit Ġobjectives ĠB IB REF 9 Ġ, ĠB IB REF 10 Ġ, ĠB IB REF 11 Ġ, ĠB IB REF 12 Ġ, ĠB IB REF 13 Ġsuch Ġas Ġmaximum Ġmean Ġdiscrepancy Ġ( MM D ) ĠB IB REF 14 Ġ. ĠIn ĠN LP Ġtasks , ĠLi Ġet Ġal . Ġ( ĠB IB REF 15 Ġ) Ġand ĠChen Ġet Ġal . Ġ( ĠB IB REF 16 Ġ) Ġboth Ġproposed Ġusing Ġadvers arial Ġtraining Ġframework Ġfor Ġreducing Ġdomain Ġdifference . ĠIn Ġtheir Ġmodel , Ġa Ġsub - network Ġis Ġadded Ġas Ġa Ġdomain Ġdiscrim inator Ġwhile Ġdeep Ġfeatures Ġare Ġlearned Ġto Ġconfuse Ġthe Ġdiscrim inator . ĠThe Ġfeature Ġadaptation Ġcomponent Ġin Ġour Ġmodel Ġshares Ġsimilar Ġintuition Ġwith ĠM MD Ġand Ġadversary Ġtraining . ĠWe Ġwill Ġshow Ġa Ġdetailed Ġcomparison Ġwith Ġthem Ġin Ġour Ġexperiments . </s> Sem i - super vised ĠLearning : ĠWe Ġattempt Ġto Ġtreat Ġdomain Ġadaptation Ġas Ġa Ġsemi - super vised Ġlearning Ġtask Ġby Ġconsidering Ġthe Ġtarget Ġinstances Ġas Ġunl abel ed Ġdata . ĠSome Ġefforts Ġhave Ġbeen Ġinitiated Ġon Ġtransfer Ġlearning Ġfrom Ġunl abel ed Ġdata ĠB IB REF 17 Ġ, ĠB IB REF 18 Ġ, ĠB IB REF 19 Ġ. ĠIn Ġour Ġmodel , Ġwe Ġreduce Ġthe Ġdomain Ġdiscrepancy Ġby Ġfeature Ġadaptation , Ġand Ġthereafter Ġadopt Ġsemi - super vised Ġlearning Ġtechniques Ġto Ġlearn Ġfrom Ġunl abel ed Ġdata . ĠPrim arily Ġmotivated Ġby ĠB IB REF 20 Ġand ĠB IB REF 21 Ġ, Ġwe Ġemployed Ġentropy Ġminim ization Ġand Ġself - ense mble Ġboot stra pping Ġas Ġregular izations Ġto Ġincorporate Ġunl abel ed Ġdata . ĠOur Ġexperimental Ġresults Ġshow Ġthat Ġboth Ġmethods Ġare Ġeffective Ġwhen Ġjointly Ġtrained Ġwith Ġthe Ġfeature Ġadaptation Ġobjective , Ġwhich Ġconfirms Ġto Ġour Ġmotivation . </s> Not ations Ġand ĠModel ĠOverview </s> We Ġconduct Ġmost Ġof Ġour Ġexperiments Ġunder Ġan Ġun super vised Ġdomain Ġadaptation Ġsetting , Ġwhere Ġwe Ġhave Ġno Ġlabeled Ġdata Ġfrom Ġthe Ġtarget Ġdomain . ĠConsider Ġtwo Ġsets ĠIN LINE FORM 0 Ġand ĠIN LINE FORM 1 Ġ. ĠIN LINE FORM 2 Ġis Ġfrom Ġthe Ġsource Ġdomain Ġwith ĠIN LINE FORM 3 Ġlabeled Ġexamples , Ġwhere ĠIN LINE FORM 4 Ġis Ġa Ġone - hot Ġvector Ġrepresentation Ġof Ġsentiment Ġlabel Ġand ĠIN LINE FORM 5 Ġdenotes Ġthe Ġnumber Ġof Ġclasses . ĠIN LINE FORM 6 Ġis Ġfrom Ġthe Ġtarget Ġdomain Ġwith ĠIN LINE FORM 7 Ġunl abel ed Ġexamples . ĠIN LINE FORM 8 Ġdenotes Ġthe Ġtotal Ġnumber Ġof Ġtraining Ġdocuments Ġincluding Ġboth Ġlabeled Ġand Ġunl abel ed . ĠWe Ġaim Ġto Ġlearn Ġa Ġsentiment Ġclass ifier Ġfrom ĠIN LINE FORM 13 Ġand ĠIN LINE FORM 14 Ġsuch Ġthat Ġthe Ġclass ifier Ġwould Ġwork Ġwell Ġon Ġthe Ġtarget Ġdomain . ĠWe Ġalso Ġpresent Ġsome Ġresults Ġunder Ġa Ġsetting Ġwhere Ġwe Ġassume Ġthat Ġa Ġsmall Ġnumber Ġof Ġlabeled Ġtarget Ġexamples Ġare Ġavailable Ġ( see ĠFigure ĠFIG REF 27 Ġ). </s> For Ġthe Ġproposed Ġmodel , Ġwe Ġdenote ĠIN LINE FORM 0 Ġparameter ized Ġby ĠIN LINE FORM 1 Ġas Ġa Ġneural - based Ġfeature Ġenc oder Ġthat Ġmaps Ġdocuments Ġfrom Ġboth Ġdomains Ġto Ġa Ġshared Ġfeature Ġspace , Ġand ĠIN LINE FORM 2 Ġparameter ized Ġby ĠIN LINE FORM 3 Ġas Ġa Ġfully Ġconnected Ġlayer Ġwith Ġsoft max Ġactivation Ġserving Ġas Ġthe Ġsentiment Ġclass ifier . ĠWe Ġaim Ġto Ġlearn Ġfeature Ġrepresentations Ġthat Ġare Ġdomain - inv ari ant Ġand Ġat Ġthe Ġsame Ġtime Ġdiscrim in ative Ġon Ġboth Ġdomains , Ġthus Ġwe Ġsimultaneously Ġconsider Ġthree Ġfactors Ġin Ġour Ġobjective : Ġ( 1 ) Ġminimize Ġthe Ġclassification Ġerror Ġon Ġthe Ġlabeled Ġsource Ġexamples ; Ġ( 2 ) Ġminimize Ġthe Ġdomain Ġdiscrepancy ; Ġand Ġ( 3 ) Ġleverage Ġunl abel ed Ġdata Ġvia Ġsemi - super vised Ġlearning . </s> Supp ose Ġwe Ġalready Ġhave Ġthe Ġencoded Ġfeatures Ġof Ġdocuments ĠIN LINE FORM 0 Ġ( see ĠSection ĠSEC REF 10 Ġ), Ġthe Ġobjective Ġfunction Ġfor Ġpurpose Ġ( 1 ) Ġis Ġthus Ġthe Ġcross Ġentropy Ġloss Ġon Ġthe Ġlabeled Ġsource Ġexamples ĠDIS PLAY FORM 0 </s> where ĠIN LINE FORM 0 Ġdenotes Ġthe Ġpredicted Ġlabel Ġdistribution . ĠIn Ġthe Ġfollowing Ġsubsections , Ġwe Ġwill Ġexplain Ġhow Ġto Ġperform Ġfeature Ġadaptation Ġand Ġdomain Ġadaptive Ġsemi - super vised Ġlearning Ġin Ġdetails Ġfor Ġpurpose Ġ( 2 ) Ġand Ġ( 3 ) Ġrespectively . </s> Feature ĠAdapt ation </s> Unlike Ġprior Ġworks ĠB IB REF 0 Ġ, ĠB IB REF 4 Ġ, Ġour Ġmethod Ġdoes Ġnot Ġattempt Ġto Ġalign Ġdomain - specific Ġwords Ġthrough Ġpivot Ġwords . ĠIn Ġour Ġpreliminary Ġexperiments , Ġwe Ġfound Ġthat Ġword Ġembed d ings Ġpre - trained Ġon Ġa Ġlarge Ġcorpus Ġare Ġable Ġto Ġadequately Ġcapture Ġthis Ġinformation . ĠAs Ġwe Ġwill Ġlater Ġshow Ġin Ġour Ġexperiments , Ġeven Ġwithout Ġadaptation , Ġa Ġnaive Ġneural Ġnetwork Ġclass ifier Ġwith Ġpre - trained Ġword Ġembed d ings Ġcan Ġalready Ġachieve Ġreasonably Ġgood Ġresults . </s> We Ġattempt Ġto Ġexplicitly Ġminimize Ġthe Ġdistance Ġbetween Ġthe Ġsource Ġand Ġtarget Ġfeature Ġrepresentations Ġ( ĠIN LINE FORM 0 Ġand ĠIN LINE FORM 1 Ġ). ĠA Ġfew Ġmethods Ġfrom Ġliterature Ġcan Ġbe Ġapplied Ġsuch Ġas ĠMaximum ĠMean ĠDisc rep ancy Ġ( MM D ) ĠB IB REF 14 Ġor Ġadversary Ġtraining ĠB IB REF 15 Ġ, ĠB IB REF 16 Ġ. ĠThe Ġmain Ġidea Ġof ĠM MD Ġis Ġto Ġestimate Ġthe Ġdistance Ġbetween Ġtwo Ġdistributions Ġas Ġthe Ġdistance Ġbetween Ġsample Ġmeans Ġof Ġthe Ġprojected Ġembed d ings Ġin ĠHilbert Ġspace . ĠM MD Ġis Ġimplicitly Ġcomputed Ġthrough Ġa Ġcharacteristic Ġkernel , Ġwhich Ġis Ġused Ġto Ġensure Ġthat Ġthe Ġsample Ġmean Ġis Ġinject ive , Ġleading Ġto Ġthe ĠM MD Ġbeing Ġzero Ġif Ġand Ġonly Ġif Ġthe Ġdistributions Ġare Ġidentical . ĠIn Ġour Ġimplementation , Ġwe Ġskip Ġthe Ġmapping Ġprocedure Ġinduced Ġby Ġa Ġcharacteristic Ġkernel Ġfor Ġsimpl ifying Ġthe Ġcomputation Ġand Ġlearning . ĠWe Ġsimply Ġestimate Ġthe Ġdistribution Ġdistance Ġas Ġthe Ġdistance Ġbetween Ġthe Ġsample Ġmeans Ġin Ġthe Ġcurrent Ġembed ding Ġspace . ĠAlthough Ġthis Ġapproximation Ġcannot Ġpreserve Ġall Ġstatistical Ġfeatures Ġof Ġthe Ġunderlying Ġdistributions , Ġwe Ġfind Ġit Ġperforms Ġcompar ably Ġto ĠM MD Ġon Ġour Ġproblem . ĠThe Ġfollowing Ġequations Ġformally Ġdescribe Ġthe Ġfeature Ġadaptation Ġloss ĠIN LINE FORM 2 Ġ: ĠDIS PLAY FORM 0 </s> IN LINE FORM 0 Ġnormal ization Ġis Ġapplied Ġon Ġthe Ġmean Ġrepresentations ĠIN LINE FORM 1 Ġand ĠIN LINE FORM 2 Ġ, Ġresc aling Ġthe Ġvectors Ġsuch Ġthat Ġall Ġentries Ġsum Ġto Ġ1 . ĠWe Ġadopt Ġa Ġsymm etric Ġversion Ġof ĠKL Ġdivergence ĠB IB REF 12 Ġas Ġthe Ġdistance Ġfunction . ĠGiven Ġtwo Ġdistribution Ġvectors ĠIN LINE FORM 3 Ġ, ĠIN LINE FORM 4 Ġ. </s> Domain ĠAdapt ive ĠSemi - super vised ĠLearning Ġ( D AS ) </s> We Ġattempt Ġto Ġexploit Ġthe Ġinformation Ġin Ġtarget Ġdata Ġthrough Ġsemi - super vised Ġlearning Ġobjectives , Ġwhich Ġare Ġjointly Ġtrained Ġwith ĠIN LINE FORM 0 Ġand ĠIN LINE FORM 1 Ġ. ĠNormally , Ġto Ġincorporate Ġtarget Ġdata , Ġwe Ġcan Ġminimize Ġthe Ġcross Ġentropy Ġloss Ġbetween Ġthe Ġtrue Ġlabel Ġdistributions ĠIN LINE FORM 2 Ġand Ġthe Ġpredicted Ġlabel Ġdistributions ĠIN LINE FORM 3 Ġover Ġtarget Ġsamples . ĠThe Ġchallenge Ġhere Ġis Ġthat ĠIN LINE FORM 4 Ġis Ġunknown , Ġand Ġthus Ġwe Ġattempt Ġto Ġestimate Ġit Ġvia Ġsemi - super vised Ġlearning . ĠWe Ġuse Ġentropy Ġminim ization Ġand Ġboot stra pping Ġfor Ġthis Ġpurpose . ĠWe Ġwill Ġlater Ġshow Ġin Ġour Ġexperiments Ġthat Ġboth Ġmethods Ġare Ġeffective , Ġand Ġjointly Ġemploying Ġthem Ġoverall Ġyields Ġthe Ġbest Ġresults . </s> Ent ropy ĠMin im ization : ĠIn Ġthis Ġmethod , ĠIN LINE FORM 0 Ġis Ġestimated Ġas Ġthe Ġpredicted Ġlabel Ġdistribution ĠIN LINE FORM 1 Ġ, Ġwhich Ġis Ġa Ġfunction Ġof ĠIN LINE FORM 2 Ġand ĠIN LINE FORM 3 Ġ. ĠThe Ġloss Ġcan Ġthus Ġbe Ġwritten Ġas ĠDIS PLAY FORM 0 </s> Ass ume Ġthe Ġdomain Ġdiscrepancy Ġcan Ġbe Ġeffectively Ġreduced Ġthrough Ġfeature Ġadaptation , Ġby Ġminimizing Ġthe Ġentropy Ġpenalty , Ġtraining Ġof Ġthe Ġclass ifier Ġis Ġinfluenced Ġby Ġthe Ġunl abel ed Ġtarget Ġdata Ġand Ġwill Ġgenerally Ġmaximize Ġthe Ġmargins Ġbetween Ġthe Ġtarget Ġexamples Ġand Ġthe Ġdecision Ġboundaries , Ġincreasing Ġthe Ġprediction Ġconfidence Ġon Ġthe Ġtarget Ġdomain . </s> Self - ense mble ĠBoot stra pping : ĠAnother Ġway Ġto Ġestimate ĠIN LINE FORM 0 Ġcorresponds Ġto Ġboot stra pping . ĠThe Ġidea Ġis Ġto Ġestimate Ġthe Ġunknown Ġlabels Ġas Ġthe Ġpredictions Ġof Ġthe Ġmodel Ġlearned Ġfrom Ġthe Ġprevious Ġround Ġof Ġtraining . ĠBoot stra pping Ġhas Ġbeen Ġexplored Ġfor Ġdomain Ġadaptation Ġin Ġprevious Ġworks ĠB IB REF 18 Ġ, ĠB IB REF 19 Ġ. ĠHowever , Ġin Ġtheir Ġmethods , Ġdomain Ġdiscrepancy Ġwas Ġnot Ġexplicitly Ġminimized Ġvia Ġfeature Ġadaptation . ĠApp lying Ġboot stra pping Ġor Ġother Ġsemi - super vised Ġlearning Ġtechniques Ġin Ġthis Ġcase Ġmay Ġworsen Ġthe Ġresults Ġas Ġthe Ġclass ifier Ġcan Ġperform Ġquite Ġbad Ġon Ġthe Ġtarget Ġdata . </s> [ t ] ĠPse ud oc ode Ġfor Ġtraining ĠD AS ĠIN LINE FORM 0 Ġ, ĠIN LINE FORM 1 Ġ, ĠIN LINE FORM 2 Ġ, ĠIN LINE FORM 3 ĠIN LINE FORM 4 Ġ= Ġen semb ling Ġmomentum , ĠIN LINE FORM 5 ĠIN LINE FORM 6 Ġ= Ġweight Ġramp - up Ġfunction ĠIN LINE FORM 7 ĠIN LINE FORM 8 ĠIN LINE FORM 9 Ġeach Ġmin ib atch ĠIN LINE FORM 10 Ġ, ĠIN LINE FORM 11 Ġ, ĠIN LINE FORM 12 Ġin </s> IN LINE FORM 0 Ġ, ĠIN LINE FORM 1 Ġ, ĠIN LINE FORM 2 Ġcompute Ġloss ĠIN LINE FORM 3 Ġon ĠIN LINE FORM 4 Ġcompute Ġloss ĠIN LINE FORM 5 Ġon ĠIN LINE FORM 6 Ġcompute Ġloss ĠIN LINE FORM 7 Ġon ĠIN LINE FORM 8 Ġcompute Ġloss ĠIN LINE FORM 9 Ġon ĠIN LINE FORM 10 ĠIN LINE FORM 11 </s> update Ġnetwork Ġparameters ĠIN LINE FORM 0 Ġ, Ġfor ĠIN LINE FORM 1 ĠIN LINE FORM 2 ĠIN LINE FORM 3 </s> Insp ired Ġby Ġthe Ġen semb ling Ġmethod Ġproposed Ġin ĠB IB REF 21 Ġ, Ġwe Ġestimate ĠIN LINE FORM 0 Ġby Ġforming Ġensemble Ġpredictions Ġof Ġlabels Ġduring Ġtraining , Ġusing Ġthe Ġoutputs Ġon Ġdifferent Ġtraining Ġepoch s . ĠThe Ġloss Ġis Ġformulated Ġas Ġfollows : ĠDIS PLAY FORM 0 </s> where ĠIN LINE FORM 0 Ġdenotes Ġthe Ġestimated Ġlabels Ġcomputed Ġon Ġthe Ġensemble Ġpredictions Ġfrom Ġdifferent Ġepoch s . ĠThe Ġloss Ġis Ġapplied Ġon Ġall Ġdocuments . ĠIt Ġserves Ġfor Ġboot stra pping Ġon Ġthe Ġunl abel ed Ġtarget Ġdata , Ġand Ġit Ġalso Ġserves Ġas Ġa Ġregular ization Ġthat Ġencourages Ġthe Ġnetwork Ġpredictions Ġto Ġbe Ġconsistent Ġin Ġdifferent Ġtraining Ġepoch s . ĠIN LINE FORM 1 Ġis Ġjointly Ġtrained Ġwith ĠIN LINE FORM 2 Ġ, ĠIN LINE FORM 3 Ġ, Ġand ĠIN LINE FORM 4 Ġ. ĠAl gorithm ĠSEC REF 6 Ġillustrates Ġthe Ġoverall Ġtraining Ġprocess Ġof Ġthe Ġproposed Ġdomain Ġadaptive Ġsemi - super vised Ġlearning Ġ( D AS ) Ġframework . </s> In ĠAl gorithm ĠSEC REF 6 Ġ, ĠIN LINE FORM 0 Ġ, ĠIN LINE FORM 1 Ġ, Ġand ĠIN LINE FORM 2 Ġare Ġweights Ġto Ġbalance Ġthe Ġeffects Ġof ĠIN LINE FORM 3 Ġ, ĠIN LINE FORM 4 Ġ, Ġand ĠIN LINE FORM 5 Ġrespectively . ĠIN LINE FORM 6 Ġand ĠIN LINE FORM 7 Ġare Ġconstant Ġhyper - param eters . ĠWe Ġset ĠIN LINE FORM 8 Ġas Ġa ĠGa ussian Ġcurve Ġto Ġramp Ġup Ġthe Ġweight Ġfrom Ġ0 Ġto ĠIN LINE FORM 9 Ġ. ĠThis Ġis Ġto Ġensure Ġthe Ġramp - up Ġof Ġthe Ġboot stra pping Ġloss Ġcomponent Ġis Ġslow Ġenough Ġin Ġthe Ġbeginning Ġof Ġthe Ġtraining . ĠAfter Ġeach Ġtraining Ġepoch , Ġwe Ġcompute ĠIN LINE FORM 10 Ġwhich Ġdenotes Ġthe Ġpredictions Ġmade Ġby Ġthe Ġnetwork Ġin Ġcurrent Ġepoch , Ġand Ġthen Ġthe Ġensemble Ġprediction ĠIN LINE FORM 11 Ġis Ġupdated Ġas Ġa Ġweighted Ġaverage Ġof Ġthe Ġoutputs Ġfrom Ġprevious Ġepoch s Ġand Ġthe Ġcurrent Ġepoch , Ġwith Ġrecent Ġepoch s Ġhaving Ġlarger Ġweight . ĠFor Ġgenerating Ġestimated Ġlabels ĠIN LINE FORM 12 Ġ, ĠIN LINE FORM 13 Ġis Ġconverted Ġto Ġa Ġone - hot Ġvector Ġwhere Ġthe Ġentry Ġwith Ġthe Ġmaximum Ġvalue Ġis Ġset Ġto Ġone Ġand Ġother Ġentries Ġare Ġset Ġto Ġz eros . ĠThe Ġself - ense mble Ġboot stra pping Ġis Ġa Ġgeneralized Ġversion Ġof Ġboot stra ppings Ġthat Ġonly Ġuse Ġthe Ġoutputs Ġfrom Ġthe Ġprevious Ġround Ġof Ġtraining ĠB IB REF 18 Ġ, ĠB IB REF 19 Ġ. ĠThe Ġensemble Ġprediction Ġis Ġlikely Ġto Ġbe Ġcloser Ġto Ġthe Ġcorrect , Ġunknown Ġlabels Ġof Ġthe Ġtarget Ġdata . </s> CNN ĠEnc oder ĠImplementation </s> We Ġhave Ġleft Ġthe Ġfeature Ġenc oder ĠIN LINE FORM 0 Ġunspecified , Ġfor Ġwhich , Ġa Ġfew Ġoptions Ġcan Ġbe Ġconsidered . ĠIn Ġour Ġimplementation , Ġwe Ġadopt Ġa Ġone - layer ĠCNN Ġstructure Ġfrom Ġprevious Ġworks ĠB IB REF 22 Ġ, ĠB IB REF 4 Ġ, Ġas Ġit Ġhas Ġbeen Ġdemonstrated Ġto Ġwork Ġwell Ġfor Ġsentiment Ġclassification Ġtasks . ĠGiven Ġa Ġreview Ġdocument ĠIN LINE FORM 1 Ġconsisting Ġof ĠIN LINE FORM 2 Ġwords , Ġwe Ġbegin Ġby Ġassoci ating Ġeach Ġword Ġwith Ġa Ġcontinuous Ġword Ġembed ding ĠB IB REF 23 ĠIN LINE FORM 3 Ġfrom Ġan Ġembed ding Ġmatrix ĠIN LINE FORM 4 Ġ, Ġwhere ĠIN LINE FORM 5 Ġis Ġthe Ġvocabulary Ġsize Ġand ĠIN LINE FORM 6 Ġis Ġthe Ġembed ding Ġdimension . ĠIN LINE FORM 7 Ġis Ġjointly Ġupdated Ġwith Ġother Ġnetwork Ġparameters Ġduring Ġtraining . ĠGiven Ġa Ġwindow Ġof Ġdense Ġword Ġembed d ings ĠIN LINE FORM 8 Ġ, Ġthe Ġconv olution Ġlayer Ġfirst Ġconc aten ates Ġthese Ġvectors Ġto Ġform Ġa Ġvector ĠIN LINE FORM 9 Ġof Ġlength ĠIN LINE FORM 10 Ġand Ġthen Ġthe Ġoutput Ġvector Ġis Ġcomputed Ġby ĠEqu ation Ġ( ĠEQ REF 11 Ġ): ĠDIS PLAY FORM 0 </s> IN LINE FORM 0 Ġ, ĠIN LINE FORM 1 Ġis Ġthe Ġparameter Ġset Ġof Ġthe Ġenc oder ĠIN LINE FORM 2 Ġand Ġis Ġshared Ġacross Ġall Ġwindows Ġof Ġthe Ġsequence . ĠIN LINE FORM 3 Ġis Ġan Ġelement - wise Ġnon - linear Ġactivation Ġfunction . ĠThe Ġconv olution Ġoperation Ġcan Ġcapture Ġlocal Ġcontextual Ġdependencies Ġof Ġthe Ġinput Ġsequence Ġand Ġthe Ġextracted Ġfeature Ġvectors Ġare Ġsimilar Ġto ĠIN LINE FORM 4 Ġ- gram s . ĠAfter Ġthe Ġconv olution Ġoperation Ġis Ġapplied Ġto Ġthe Ġwhole Ġsequence , Ġwe Ġobtain Ġa Ġlist Ġof Ġhidden Ġvectors ĠIN LINE FORM 5 Ġ. ĠA Ġmax - over - time Ġpool ing Ġlayer Ġis Ġapplied Ġto Ġobtain Ġthe Ġfinal Ġvector Ġrepresentation ĠIN LINE FORM 6 Ġof Ġthe Ġinput Ġdocument . </s> Dat as ets Ġand ĠExperimental ĠSettings </s> Ex isting Ġbenchmark Ġdatasets Ġsuch Ġas Ġthe ĠAmazon Ġbenchmark ĠB IB REF 0 Ġtypically Ġremove Ġreviews Ġwith Ġneutral Ġlabels Ġin Ġboth Ġdomains . ĠThis Ġis Ġproblematic Ġas Ġthe Ġlabel Ġinformation Ġof Ġthe Ġtarget Ġdomain Ġis Ġnot Ġaccessible Ġin Ġan Ġun super vised Ġdomain Ġadaptation Ġsetting . ĠFurthermore , Ġremoving Ġneutral Ġinstances Ġmay Ġbias Ġthe Ġdataset Ġfavorably Ġfor Ġmax - margin - based Ġalgorithms Ġlike Ġours , Ġsince Ġthe Ġresulting Ġdataset Ġhas Ġall Ġuncertain Ġlabels Ġremoved , Ġleaving Ġonly Ġhigh Ġconfidence Ġexamples . ĠTherefore , Ġwe Ġconstruct Ġnew Ġdatasets Ġby Ġourselves . ĠThe Ġresults Ġon Ġthe Ġoriginal ĠAmazon Ġbenchmark Ġis Ġqual itatively Ġsimilar , Ġand Ġwe Ġpresent Ġthem Ġin ĠAppendix ĠSEC REF 6 Ġfor Ġcomple teness Ġsince Ġmost Ġof Ġprevious Ġworks Ġreported Ġresults Ġon Ġit . </s> Small - scale Ġdatasets : ĠOur Ġnew Ġdataset Ġwas Ġderived Ġfrom Ġthe Ġlarge - scale ĠAmazon Ġdatasets Ġreleased Ġby ĠMcA u ley Ġet Ġal . Ġ( ĠB IB REF 24 Ġ). ĠIt Ġcontains Ġfour Ġdomains : ĠBook Ġ( B K ), ĠElectronics Ġ( E ), ĠBeauty Ġ( BT ), Ġand ĠMusic Ġ( M ). ĠEach Ġdomain Ġcontains Ġtwo Ġdatasets . ĠSet Ġ1 Ġcontains Ġ6000 Ġinstances Ġwith Ġexactly Ġbalanced Ġclass Ġlabels , Ġand Ġset Ġ2 Ġcontains Ġ6000 Ġinstances Ġthat Ġare Ġrandomly Ġsampled Ġfrom Ġthe Ġlarge Ġdataset , Ġpreserving Ġthe Ġoriginal Ġlabel Ġdistribution , Ġwhich Ġwe Ġbelieve Ġbetter Ġreflects Ġthe Ġlabel Ġdistribution Ġin Ġreal Ġlife . ĠThe Ġexamples Ġin Ġthese Ġtwo Ġsets Ġdo Ġnot Ġoverlap . ĠDet ailed Ġstatistics Ġof Ġthe Ġgenerated Ġdatasets Ġare Ġgiven Ġin ĠTable ĠT AB REF 9 Ġ. </s> In Ġall Ġour Ġexperiments Ġon Ġthe Ġsmall - scale Ġdatasets , Ġwe Ġuse Ġset Ġ1 Ġof Ġthe Ġsource Ġdomain Ġas Ġthe Ġonly Ġsource Ġwith Ġsentiment Ġlabel Ġinformation Ġduring Ġtraining , Ġand Ġwe Ġevaluate Ġthe Ġtrained Ġmodel Ġon Ġset Ġ1 Ġof Ġthe Ġtarget Ġdomain . ĠSince Ġwe Ġcannot Ġcontrol Ġthe Ġlabel Ġdistribution Ġof Ġunl abel ed Ġdata Ġduring Ġtraining , Ġwe Ġconsider Ġtwo Ġdifferent Ġsettings : </s> Setting Ġ( 1 ): ĠOnly Ġset Ġ1 Ġof Ġthe Ġtarget Ġdomain Ġis Ġused Ġas Ġthe Ġunl abel ed Ġset . ĠThis Ġtells Ġus Ġhow Ġthe Ġmethod Ġperforms Ġin Ġa Ġcondition Ġwhen Ġthe Ġtarget Ġdomain Ġhas Ġa Ġclose - to - balanced Ġlabel Ġdistribution . ĠAs Ġwe Ġalso Ġevaluate Ġon Ġset Ġ1 Ġof Ġthe Ġtarget Ġdomain , Ġthis Ġis Ġalso Ġconsidered Ġas Ġa Ġtrans duct ive Ġsetting . </s> Setting Ġ( 2 ): ĠSet Ġ2 Ġfrom Ġboth Ġthe Ġsource Ġand Ġtarget Ġdomains Ġare Ġused Ġas Ġunl abel ed Ġsets . ĠSince Ġset Ġ2 Ġis Ġdirectly Ġsampled Ġfrom Ġmillions Ġof Ġreviews , Ġit Ġbetter Ġreflects Ġreal - life Ġsentiment Ġdistribution . </s> Large - scale Ġdatasets : ĠWe Ġfurther Ġconduct Ġexperiments Ġon Ġfour Ġmuch Ġlarger Ġdatasets : ĠIM DB Ġ( I ), ĠYelp 2014 Ġ( Y ), ĠCell ĠPhone Ġ( C ), Ġand ĠBaby Ġ( B ). ĠIM DB Ġand ĠYelp 2014 Ġwere Ġpreviously Ġused Ġin ĠB IB REF 25 Ġ, ĠB IB REF 26 Ġ. ĠCell Ġphone Ġand ĠBaby Ġare Ġfrom Ġthe Ġlarge - scale ĠAmazon Ġdataset ĠB IB REF 24 Ġ, ĠB IB REF 27 Ġ. ĠDet ailed Ġstatistics Ġare Ġsummarized Ġin ĠTable ĠT AB REF 9 Ġ. ĠWe Ġkeep Ġall Ġreviews Ġin Ġthe Ġoriginal Ġdatasets Ġand Ġconsider Ġa Ġtrans duct ive Ġsetting Ġwhere Ġall Ġtarget Ġexamples Ġare Ġused Ġfor Ġboth Ġtraining Ġ( without Ġlabel Ġinformation ) Ġand Ġevaluation . ĠWe Ġperform Ġsampling Ġto Ġbalance Ġthe Ġclasses Ġof Ġlabeled Ġsource Ġdata Ġin Ġeach Ġmin ib atch ĠIN LINE FORM 3 Ġduring Ġtraining . </s> Se lection Ġof ĠDevelopment ĠSet </s> Ide ally , Ġthe Ġdevelopment Ġset Ġshould Ġbe Ġdrawn Ġfrom Ġthe Ġsame Ġdistribution Ġas Ġthe Ġtest Ġset . ĠHowever , Ġunder Ġthe Ġun super vised Ġdomain Ġadaptation Ġsetting , Ġwe Ġdo Ġnot Ġhave Ġany Ġlabeled Ġtarget Ġdata Ġat Ġtraining Ġphase Ġwhich Ġcould Ġbe Ġused Ġas Ġdevelopment Ġset . ĠIn Ġall Ġof Ġour Ġexperiments , Ġfor Ġeach Ġpair Ġof Ġdomains , Ġwe Ġinstead Ġsample Ġ1000 Ġexamples Ġfrom Ġthe Ġtraining Ġset Ġof Ġthe Ġsource Ġdomain Ġas Ġdevelopment Ġset . ĠWe Ġtrain Ġthe Ġnetwork Ġfor Ġa Ġfixed Ġnumber Ġof Ġepoch s , Ġand Ġthe Ġmodel Ġwith Ġthe Ġminimum Ġclassification Ġerror Ġon Ġthis Ġdevelopment Ġset Ġis Ġsaved Ġfor Ġevaluation . ĠThis Ġapproach Ġworks Ġwell Ġon Ġmost Ġof Ġthe Ġproblems Ġsince Ġthe Ġtarget Ġdomain Ġis Ġsupposed Ġto Ġbehave Ġlike Ġthe Ġsource Ġdomain Ġif Ġthe Ġdomain Ġdifference Ġis Ġeffectively Ġreduced . </s> Another Ġproblem Ġis Ġhow Ġto Ġselect Ġthe Ġvalues Ġfor Ġhyper - param eters . ĠIf Ġwe Ġtune ĠIN LINE FORM 0 Ġand ĠIN LINE FORM 1 Ġdirectly Ġon Ġthe Ġdevelopment Ġset Ġfrom Ġthe Ġsource Ġdomain , Ġmost Ġlikely Ġboth Ġof Ġthem Ġwill Ġbe Ġset Ġto Ġ0 , Ġas Ġunl abel ed Ġtarget Ġdata Ġis Ġnot Ġhelpful Ġfor Ġimproving Ġin - domain Ġaccuracy Ġof Ġthe Ġsource Ġdomain . ĠOther Ġneural Ġnetwork Ġmodels Ġalso Ġhave Ġthe Ġsame Ġproblem Ġfor Ġhyper - param eter Ġtuning . ĠTherefore , Ġour Ġstrategy Ġis Ġto Ġuse Ġthe Ġdevelopment Ġset Ġfrom Ġthe Ġtarget Ġdomain Ġto Ġoptimize ĠIN LINE FORM 2 Ġand ĠIN LINE FORM 3 Ġfor Ġone Ġproblem Ġ( e . g ., Ġwe Ġonly Ġdo Ġthis Ġon ĠE ĠIN LINE FORM 4 ĠB K ), Ġand Ġfix Ġtheir Ġvalues Ġon Ġthe Ġother Ġproblems . ĠThis Ġsetting Ġassumes Ġthat Ġwe Ġhave Ġat Ġleast Ġtwo Ġlabeled Ġdomains Ġsuch Ġthat Ġwe Ġcan Ġoptimize Ġthe Ġhyper - param eters , Ġand Ġthen Ġwe Ġfix Ġthem Ġfor Ġother Ġnew Ġunl abel ed Ġdomains Ġto Ġtransfer Ġto . </s> Training ĠDetails Ġand ĠHyper - param eters </s> We Ġinitialize Ġword Ġembed d ings Ġusing Ġthe Ġ300 - dimension ĠGl o Ve Ġvectors Ġsupplied Ġby ĠPen nington Ġet Ġal ., Ġ( ĠB IB REF 28 Ġ), Ġwhich Ġwere Ġtrained Ġon Ġ840 Ġbillion Ġtokens Ġfrom Ġthe ĠCommon ĠC rawl . ĠFor Ġeach Ġpair Ġof Ġdomains , Ġthe Ġvocabulary Ġconsists Ġof Ġthe Ġtop Ġ10000 Ġmost Ġfrequent Ġwords . ĠFor Ġwords Ġin Ġthe Ġvocabulary Ġbut Ġnot Ġpresent Ġin Ġthe Ġpre - trained Ġembed d ings , Ġwe Ġrandomly Ġinitialize Ġthem . </s> We Ġset Ġhyper - param eters Ġof Ġthe ĠCNN Ġenc oder Ġfollowing Ġprevious Ġworks ĠB IB REF 22 Ġ, ĠB IB REF 4 Ġwithout Ġspecific Ġtuning Ġon Ġour Ġdatasets . ĠThe Ġwindow Ġsize Ġis Ġset Ġto Ġ3 Ġand Ġthe Ġsize Ġof Ġthe Ġhidden Ġlayer Ġis Ġset Ġto Ġ300 . ĠThe Ġnon linear Ġactivation Ġfunction Ġis ĠRel u . ĠFor Ġregular ization , Ġwe Ġalso Ġfollow Ġtheir Ġsettings Ġand Ġemploy Ġdrop out Ġwith Ġprobability Ġset Ġto Ġ0 . 5 Ġon ĠIN LINE FORM 0 Ġbefore Ġfeeding Ġit Ġto Ġthe Ġoutput Ġlayer ĠIN LINE FORM 1 Ġ, Ġand Ġconst rain Ġthe ĠIN LINE FORM 2 Ġ- norm Ġof Ġthe Ġweight Ġvector ĠIN LINE FORM 3 Ġ, Ġsetting Ġits Ġmax Ġnorm Ġto Ġ3 . </s> On Ġthe Ġsmall - scale Ġdatasets Ġand Ġthe ĠA am zon Ġbenchmark , ĠIN LINE FORM 0 Ġand ĠIN LINE FORM 1 Ġare Ġset Ġto Ġ200 Ġand Ġ1 , Ġrespectively , Ġtuned Ġon Ġthe Ġdevelopment Ġset Ġof Ġtask ĠE ĠIN LINE FORM 2 ĠB K Ġunder Ġsetting Ġ1 . ĠOn Ġthe Ġlarge - scale Ġdatasets , ĠIN LINE FORM 3 Ġand ĠIN LINE FORM 4 Ġare Ġset Ġto Ġ500 Ġand Ġ0 . 2 , Ġrespectively , Ġtuned Ġon ĠI ĠIN LINE FORM 5 ĠY . ĠWe Ġuse Ġa ĠGa ussian Ġcurve ĠIN LINE FORM 6 Ġto Ġramp Ġup Ġthe Ġweight Ġof Ġthe Ġboot stra pping Ġloss ĠIN LINE FORM 7 Ġfrom Ġ0 Ġto ĠIN LINE FORM 8 Ġ, Ġwhere ĠIN LINE FORM 9 Ġdenotes Ġthe Ġmaximum Ġnumber Ġof Ġtraining Ġepoch s . ĠWe Ġtrain Ġ30 Ġepoch s Ġfor Ġall Ġexperiments . ĠWe Ġset ĠIN LINE FORM 10 Ġto Ġ3 Ġand ĠIN LINE FORM 11 Ġto Ġ0 . 5 Ġfor Ġall Ġexperiments . </s> The Ġbatch Ġsize Ġis Ġset Ġto Ġ50 Ġon Ġthe Ġsmall - scale Ġdatasets Ġand Ġthe ĠAmazon Ġbenchmark . ĠWe Ġincrease Ġthe Ġbatch Ġsize Ġto Ġ250 Ġon Ġthe Ġlarge - scale Ġdatasets Ġto Ġreduce Ġthe Ġnumber Ġof Ġiterations . ĠRM SP rop Ġoptim izer Ġwith Ġlearning Ġrate Ġset Ġto Ġ0 . 000 5 Ġis Ġused Ġfor Ġall Ġexperiments . </s> Mod els Ġfor ĠComparison </s> We Ġcompare Ġwith Ġthe Ġfollowing Ġbas elines : </s> ( 1 ) ĠNa ive : ĠA Ġnon - domain - adapt ive Ġbaseline Ġwith Ġbag - of - words Ġrepresentations Ġand ĠS VM Ġclass ifier Ġtrained Ġon Ġthe Ġsource Ġdomain . </s> ( 2 ) Ġm S DA ĠB IB REF 7 Ġ: ĠThis Ġis Ġthe Ġstate - of - the - art Ġmethod Ġbased Ġon Ġdiscrete Ġinput Ġfeatures . ĠTop Ġ1000 Ġbag - of - words Ġfeatures Ġare Ġkept Ġas Ġpivot Ġfeatures . ĠWe Ġset Ġthe Ġnumber Ġof Ġstacked Ġlayers Ġto Ġ3 Ġand Ġthe Ġcorruption Ġprobability Ġto Ġ0 . 5 . </s> ( 3 ) ĠNa ive NN : ĠThis Ġis Ġa Ġnon - domain - adapt ive ĠCNN Ġtrained Ġon Ġsource Ġdomain , Ġwhich Ġis Ġa Ġvariant Ġof Ġour Ġmodel Ġby Ġsetting ĠIN LINE FORM 0 Ġ, ĠIN LINE FORM 1 Ġ, Ġand ĠIN LINE FORM 2 Ġto Ġz eros . </s> ( 4 ) ĠAux NN ĠB IB REF 4 Ġ: ĠThis Ġis Ġa Ġneural Ġmodel Ġthat Ġexploits Ġauxiliary Ġtasks , Ġwhich Ġhas Ġachieved Ġstate - of - the - art Ġresults Ġon Ġcross - domain Ġsentiment Ġclassification . ĠThe Ġsentence Ġenc oder Ġused Ġin Ġthis Ġmodel Ġis Ġthe Ġsame Ġas Ġours . </s> ( 5 ) ĠAD AN ĠB IB REF 16 Ġ: ĠThis Ġmethod Ġexploits Ġadvers arial Ġtraining Ġto Ġreduce Ġrepresentation Ġdifference Ġbetween Ġdomains . ĠThe Ġoriginal Ġpaper Ġuses Ġa Ġsimple Ġfeed forward Ġnetwork Ġas Ġenc oder . ĠFor Ġfair Ġcomparison , Ġwe Ġreplace Ġit Ġwith Ġour ĠCNN - based Ġenc oder . ĠWe Ġtrain Ġ5 Ġiterations Ġon Ġthe Ġdiscrim inator Ġper Ġiteration Ġon Ġthe Ġenc oder Ġand Ġsentiment Ġclass ifier Ġas Ġsuggested Ġin Ġtheir Ġpaper . </s> ( 6 ) ĠM MD : ĠM MD Ġhas Ġbeen Ġwidely Ġused Ġfor Ġminimizing Ġdomain Ġdiscrepancy Ġon Ġimages . ĠIn Ġthose Ġworks ĠB IB REF 9 Ġ, ĠB IB REF 13 Ġ, Ġvariants Ġof Ġdeep ĠCNN s Ġare Ġused Ġfor Ġencoding Ġimages Ġand Ġthe ĠM MD s Ġof Ġmultiple Ġlayers Ġare Ġjointly Ġminimized . ĠIn ĠN LP , Ġadding Ġmore Ġlayers Ġof ĠCNN s Ġmay Ġnot Ġbe Ġvery Ġhelpful Ġand Ġthus Ġthose Ġmodels Ġfrom Ġimage - related Ġtasks Ġcan Ġnot Ġbe Ġdirectly Ġapplied Ġto Ġour Ġproblem . ĠTo Ġcompare Ġwith ĠM MD - based Ġmethod , Ġwe Ġtrain Ġa Ġmodel Ġthat Ġjointly Ġminimize Ġthe Ġclassification Ġloss ĠIN LINE FORM 0 Ġon Ġthe Ġsource Ġdomain Ġand ĠM MD Ġbetween ĠIN LINE FORM 1 Ġand ĠIN LINE FORM 2 Ġ. ĠFor Ġcomputing ĠM MD , Ġwe Ġuse Ġa ĠGa ussian ĠRB F Ġwhich Ġis Ġa Ġcommon Ġchoice Ġfor Ġcharacteristic Ġkernel . </s> In Ġaddition Ġto Ġthe Ġabove Ġbas elines , Ġwe Ġalso Ġshow Ġresults Ġof Ġdifferent Ġvariants Ġof Ġour Ġmodel . ĠD AS Ġas Ġshown Ġin ĠAl gorithm ĠSEC REF 6 Ġdenotes Ġour Ġfull Ġmodel . ĠD AS - EM Ġdenotes Ġthe Ġmodel Ġwith Ġonly Ġentropy Ġminim ization Ġfor Ġsemi - super vised Ġlearning Ġ( set ĠIN LINE FORM 0 Ġ). ĠD AS - SE Ġdenotes Ġthe Ġmodel Ġwith Ġonly Ġself - ense mble Ġboot stra pping Ġfor Ġsemi - super vised Ġlearning Ġ( set ĠIN LINE FORM 1 Ġ). ĠF ANN Ġ( feature - adapt ation Ġneural Ġnetwork ) Ġdenotes Ġthe Ġmodel Ġwithout Ġsemi - super vised Ġlearning Ġperformed Ġ( set Ġboth ĠIN LINE FORM 2 Ġand ĠIN LINE FORM 3 Ġto Ġz eros ). </s> Main ĠResults </s> Figure ĠFIG REF 17 Ġshows Ġthe Ġcomparison Ġof Ġadaptation Ġresults Ġ( see ĠAppendix ĠSEC REF 7 Ġfor Ġthe Ġexact Ġnumerical Ġnumbers ). ĠWe Ġreport Ġclassification Ġaccuracy Ġon Ġthe Ġsmall - scale Ġdataset . ĠFor Ġthe Ġlarge - scale Ġdataset , Ġmacro - F 1 Ġis Ġinstead Ġused Ġsince Ġthe Ġlabel Ġdistribution Ġin Ġthe Ġtest Ġset Ġis Ġextremely Ġun balanced . ĠKey Ġobservations Ġare Ġsummarized Ġas Ġfollows . Ġ( 1 ) ĠBoth ĠD AS - EM Ġand ĠD AS - SE Ġperform Ġbetter Ġin Ġmost Ġcases Ġcompared Ġwith ĠAD AN , ĠMD D , Ġand ĠF ANN , Ġin Ġwhich Ġonly Ġfeature Ġadaptation Ġis Ġperformed . ĠThis Ġdemonstrates Ġthe Ġeffectiveness Ġof Ġthe Ġproposed Ġdomain Ġadaptive Ġsemi - super vised Ġlearning Ġframework . ĠD AS - EM Ġis Ġmore Ġeffective Ġthan ĠD AS - SE Ġin Ġmost Ġcases , Ġand Ġthe Ġfull Ġmodel ĠD AS Ġwith Ġboth Ġtechniques Ġjointly Ġemployed Ġoverall Ġhas Ġthe Ġbest Ġperformance . Ġ( 2 ) ĠWhen Ġcomparing Ġthe Ġtwo Ġsettings Ġon Ġthe Ġsmall - scale Ġdataset , Ġall Ġdomain - adapt ive Ġmethods Ġgenerally Ġperform Ġbetter Ġunder Ġsetting Ġ1 . ĠIn Ġsetting Ġ1 , Ġthe Ġtarget Ġexamples Ġare Ġbalanced Ġin Ġclasses , Ġwhich Ġcan Ġprovide Ġmore Ġdiverse Ġopinion - related Ġfeatures . ĠHowever , Ġwhen Ġconsidering Ġun super vised Ġdomain Ġadaptation , Ġwe Ġshould Ġnot Ġpresume Ġthe Ġlabel Ġdistribution Ġof Ġthe Ġunl abel ed Ġdata . ĠThus , Ġit Ġis Ġnecessary Ġto Ġconduct Ġexperiments Ġusing Ġdatasets Ġthat Ġreflect Ġreal - life Ġsentiment Ġdistribution Ġas Ġwhat Ġwe Ġdid Ġon Ġsetting 2 Ġand Ġthe Ġlarge - scale Ġdataset . ĠUnfortunately , Ġthis Ġis Ġignored Ġby Ġmost Ġof Ġprevious Ġworks . Ġ( 3 ) ĠWord - embed d ings Ġare Ġvery Ġhelpful , Ġas Ġwe Ġcan Ġsee Ġeven ĠNa ive NN Ġcan Ġsubstantially Ġoutper form Ġm S DA Ġon Ġmost Ġtasks . </s> To Ġsee Ġthe Ġeffect Ġof Ġsemi - super vised Ġlearning Ġalone , Ġwe Ġalso Ġconduct Ġexperiments Ġby Ġsetting ĠIN LINE FORM 0 Ġto Ġeliminate Ġthe Ġeffect Ġof Ġfeature Ġadaptation . ĠBoth Ġentropy Ġminim ization Ġand Ġboot stra pping Ġperform Ġvery Ġbadly Ġin Ġthis Ġsetting . ĠEnt ropy Ġminim ization Ġgives Ġalmost Ġrandom Ġpredictions Ġwith Ġaccuracy Ġbelow Ġ0 . 4 , Ġand Ġthe Ġresults Ġof Ġboot stra pping Ġare Ġalso Ġmuch Ġlower Ġcompared Ġto ĠNa ive NN . ĠThis Ġsuggests Ġthat Ġthe Ġfeature Ġadaptation Ġcomponent Ġis Ġessential . ĠWithout Ġit , Ġthe Ġlearned Ġtarget Ġrepresentations Ġare Ġless Ġmeaningful Ġand Ġdiscrim in ative . ĠApp lying Ġsemi - super vised Ġlearning Ġin Ġthis Ġcase Ġis Ġlikely Ġto Ġworsen Ġthe Ġresults . </s> Further ĠAnalysis </s> In ĠFigure ĠFIG REF 23 Ġ, Ġwe Ġshow Ġthe Ġchange Ġof Ġaccuracy Ġwith Ġrespect Ġto Ġthe Ġpercentage Ġof Ġunl abel ed Ġdata Ġused Ġfor Ġtraining Ġon Ġthree Ġparticular Ġproblems Ġunder Ġsetting Ġ1 . ĠThe Ġvalue Ġat ĠIN LINE FORM 0 Ġdenotes Ġthe Ġaccur acies Ġof ĠNa ive NN Ġwhich Ġdoes Ġnot Ġutilize Ġany Ġtarget Ġdata . ĠFor ĠD AS , Ġwe Ġobserve Ġa Ġnon linear Ġincreasing Ġtrend Ġwhere Ġthe Ġaccuracy Ġquickly Ġimproves Ġat Ġthe Ġbeginning , Ġand Ġthen Ġgradually Ġstabil izes . ĠFor Ġother Ġmethods , Ġthis Ġtrend Ġis Ġless Ġobvious , Ġand Ġadding Ġmore Ġunl abel ed Ġdata Ġsometimes Ġeven Ġworsen Ġthe Ġresults . ĠThis Ġfinding Ġagain Ġsuggests Ġthat Ġthe Ġproposed Ġapproach Ġcan Ġbetter Ġexploit Ġthe Ġinformation Ġfrom Ġunl abel ed Ġdata . </s> We Ġalso Ġconduct Ġexperiments Ġunder Ġa Ġsetting Ġwith Ġa Ġsmall Ġnumber Ġof Ġlabeled Ġtarget Ġexamples Ġavailable . ĠFigure ĠFIG REF 27 Ġshows Ġthe Ġchange Ġof Ġaccuracy Ġwith Ġrespect Ġto Ġthe Ġnumber Ġof Ġlabeled Ġtarget Ġexamples Ġadded Ġfor Ġtraining . ĠWe Ġcan Ġobserve Ġthat ĠD AS Ġis Ġstill Ġmore Ġeffective Ġunder Ġthis Ġsetting , Ġwhile Ġthe Ġperformance Ġdifferences Ġto Ġother Ġmethods Ġgradually Ġdecrease Ġwith Ġthe Ġincreasing Ġnumber Ġof Ġlabeled Ġtarget Ġexamples . </s> CNN ĠFilter ĠAnalysis </s> In Ġthis Ġsubsection , Ġwe Ġaim Ġto Ġbetter Ġunderstand ĠD AS Ġby Ġanalyzing Ġsentiment - related ĠCNN Ġfilters . ĠTo Ġdo Ġthat , Ġ1 ) Ġwe Ġfirst Ġselect Ġa Ġlist Ġof Ġthe Ġmost Ġrelated ĠCNN Ġfilters Ġfor Ġpredicting Ġeach Ġsentiment Ġlabel Ġ( positive , Ġnegative Ġneutral ). ĠThose Ġfilters Ġcan Ġbe Ġidentified Ġaccording Ġto Ġthe Ġlearned Ġweights ĠIN LINE FORM 0 Ġof Ġthe Ġoutput Ġlayer ĠIN LINE FORM 1 Ġ. ĠHigher Ġweight Ġindicates Ġstronger Ġrelated ness . Ġ2 ) ĠRecall Ġthat Ġin Ġour Ġimplementation , Ġeach ĠCNN Ġfilter Ġhas Ġa Ġwindow Ġsize Ġof Ġ3 Ġwith ĠRel u Ġactivation . ĠWe Ġcan Ġthus Ġrepresent Ġeach Ġselected Ġfilter Ġas Ġa Ġranked Ġlist Ġof Ġtrig rams Ġwith Ġhighest Ġactivation Ġvalues . </s> We Ġanalyze Ġthe ĠCNN Ġfilters Ġlearned Ġby ĠNa ive NN , ĠF ANN Ġand ĠD AS Ġrespectively Ġon Ġtask ĠE ĠIN LINE FORM 0 ĠBT Ġunder Ġsetting Ġ1 . ĠWe Ġfocus Ġon ĠE ĠIN LINE FORM 1 ĠBT Ġfor Ġstudy Ġbecause Ġelectronics Ġand Ġbeauty Ġare Ġvery Ġdifferent Ġdomains Ġand Ġeach Ġof Ġthem Ġhas Ġa Ġdiverse Ġset Ġof Ġdomain - specific Ġsentiment Ġexpressions . ĠFor Ġeach Ġmethod , Ġwe Ġidentify Ġthe Ġtop Ġ10 Ġmost Ġrelated Ġfilters Ġfor Ġeach Ġsentiment Ġlabel , Ġand Ġextract Ġthe Ġtop Ġtrig rams Ġof Ġeach Ġselected Ġfilter Ġon Ġboth Ġsource Ġand Ġtarget Ġdomains . ĠSince Ġlabeled Ġsource Ġexamples Ġare Ġused Ġfor Ġtraining , Ġwe Ġfind Ġthe Ġfilters Ġlearned Ġby Ġthe Ġthree Ġmethods Ġcapture Ġsimilar Ġexpressions Ġon Ġthe Ġsource Ġdomain , Ġcontaining Ġboth Ġdomain - inv ari ant Ġand Ġdomain - specific Ġtrig rams . ĠOn Ġthe Ġtarget Ġdomain , ĠD AS Ġcaptures Ġmore Ġtarget - specific Ġexpressions Ġcompared Ġto Ġthe Ġother Ġtwo Ġmethods . ĠDue Ġto Ġspace Ġlimitation , Ġwe Ġonly Ġpresent Ġa Ġsmall Ġsubset Ġof Ġpositive - sent iment - related Ġfilters Ġin ĠTable ĠT AB REF 34 Ġ. ĠThe Ġcomplete Ġresults Ġare Ġprovided Ġin ĠAppendix ĠSEC REF 8 Ġ. ĠFrom ĠTable ĠT AB REF 34 Ġ, Ġwe Ġcan Ġobserve Ġthat Ġthe Ġfilters Ġlearned Ġby ĠNa ive NN Ġare Ġalmost Ġunable Ġto Ġcapture Ġtarget - specific Ġsentiment Ġexpressions , Ġwhile ĠF ANN Ġis Ġable Ġto Ġcapture Ġlimited Ġtarget - specific Ġwords Ġsuch Ġas ĠâĢ ľ clean âĢ Ŀ Ġand ĠâĢ ľ s cent âĢ Ŀ . ĠThe Ġfilters Ġlearned Ġby ĠD AS Ġare Ġmore Ġdomain - adapt ive , Ġcapturing Ġdiverse Ġsentiment Ġexpressions Ġin Ġthe Ġtarget Ġdomain . </s> Conclusion </s> In Ġthis Ġwork , Ġwe Ġpropose ĠD AS , Ġa Ġnovel Ġframework Ġthat Ġjointly Ġperforms Ġfeature Ġadaptation Ġand Ġsemi - super vised Ġlearning . ĠWe Ġhave Ġdemonstrated Ġthrough Ġmultiple Ġexperiments Ġthat ĠD AS Ġcan Ġbetter Ġleverage Ġunl abel ed Ġdata , Ġand Ġachieve Ġsubstantial Ġimprovements Ġover Ġbaseline Ġmethods . ĠWe Ġhave Ġalso Ġshown Ġthat Ġfeature Ġadaptation Ġis Ġan Ġessential Ġcomponent , Ġwithout Ġwhich , Ġsemi - super vised Ġlearning Ġis Ġnot Ġable Ġto Ġfunction Ġproperly . ĠThe Ġproposed Ġframework Ġcould Ġbe Ġpotentially Ġadapted Ġto Ġother Ġdomain Ġadaptation Ġtasks , Ġwhich Ġis Ġthe Ġfocus Ġof Ġour Ġfuture Ġstudies . </s> Results Ġon ĠAmazon ĠBench mark </s> Most Ġprevious Ġworks ĠB IB REF 0 Ġ, ĠB IB REF 1 Ġ, ĠB IB REF 6 Ġ, ĠB IB REF 7 Ġ, ĠB IB REF 29 Ġcarried Ġout Ġexperiments Ġon Ġthe ĠAmazon Ġbenchmark Ġreleased Ġby ĠBl itzer Ġet Ġal . Ġ( ĠB IB REF 0 Ġ). ĠThe Ġdataset Ġcontains Ġ4 Ġdifferent Ġdomains : ĠBook Ġ( B ), ĠDVDs Ġ( D ), ĠElectronics Ġ( E ), Ġand ĠKitchen Ġ( K ). ĠFollowing Ġtheir Ġexperimental Ġsettings , Ġwe Ġconsider Ġthe Ġbinary Ġclassification Ġtask Ġto Ġpredict Ġwhether Ġa Ġreview Ġis Ġpositive Ġor Ġnegative Ġon Ġthe Ġtarget Ġdomain . ĠEach Ġdomain Ġconsists Ġof Ġ1000 Ġpositive Ġand Ġ1000 Ġnegative Ġreviews Ġrespectively . ĠWe Ġalso Ġallow Ġ4000 Ġunl abel ed Ġreviews Ġto Ġbe Ġused Ġfor Ġboth Ġthe Ġsource Ġand Ġthe Ġtarget Ġdomains , Ġof Ġwhich Ġthe Ġpositive Ġand Ġnegative Ġreviews Ġare Ġbalanced Ġas Ġwell , Ġfollowing Ġthe Ġsettings Ġin Ġprevious Ġworks . ĠWe Ġconstruct Ġ12 Ġcross - domain Ġsentiment Ġclassification Ġtasks Ġand Ġsplit Ġthe Ġlabeled Ġdata Ġin Ġeach Ġdomain Ġinto Ġa Ġtraining Ġset Ġof Ġ1600 Ġreviews Ġand Ġa Ġtest Ġset Ġof Ġ400 Ġreviews . ĠThe Ġclass ifier Ġis Ġtrained Ġon Ġthe Ġtraining Ġset Ġof Ġthe Ġsource Ġdomain Ġand Ġis Ġevaluated Ġon Ġthe Ġtest Ġset Ġof Ġthe Ġtarget Ġdomain . ĠThe Ġcomparison Ġresults Ġare Ġshown Ġin ĠTable ĠT AB REF 37 Ġ. </s> N umer ical ĠResults Ġof ĠFigure Ġ </s> Due Ġto Ġspace Ġlimitation , Ġwe Ġonly Ġshow Ġresults Ġin Ġfigures Ġin Ġthe Ġpaper . ĠAll Ġnumerical Ġnumbers Ġused Ġfor Ġplotting ĠFigure ĠFIG REF 17 Ġare Ġpresented Ġin ĠTable ĠT AB REF 38 Ġ. ĠWe Ġcan Ġobserve Ġthat ĠD AS - EM , ĠD AS - SE , Ġand ĠD AS Ġall Ġachieve Ġsubstantial Ġimprovements Ġover Ġbaseline Ġmethods Ġunder Ġdifferent Ġsettings . </s> CNN ĠFilter ĠAnalysis ĠFull ĠResults </s> As Ġmentioned Ġin ĠSection ĠSEC REF 36 Ġ, Ġwe Ġconduct ĠCNN Ġfilter Ġanalysis Ġon ĠNa ive NN , ĠF ANN , Ġand ĠD AS . ĠFor Ġeach Ġmethod , Ġwe Ġidentify Ġthe Ġtop Ġ10 Ġmost Ġrelated Ġfilters Ġfor Ġpositive , Ġnegative , Ġneutral Ġsentiment Ġlabels Ġrespectively , Ġand Ġthen Ġrepresent Ġeach Ġselected Ġfilter Ġas Ġa Ġranked Ġlist Ġof Ġtrig rams Ġwith Ġthe Ġhighest Ġactivation Ġvalues Ġon Ġit . ĠTable ĠT AB REF 39 Ġ, ĠT AB REF 40 Ġ, ĠT AB REF 41 Ġin Ġthe Ġfollowing Ġpages Ġillustrate Ġthe Ġtrig rams Ġfrom Ġthe Ġtarget Ġdomain Ġ( beaut y ) Ġcaptured Ġby Ġthe Ġselected Ġfilters Ġlearned Ġon ĠE ĠIN LINE FORM 0 ĠBT Ġfor Ġeach Ġmethod . </s> We Ġcan Ġobserve Ġthat Ġcompared Ġto ĠNa ive NN Ġand ĠF ANN , ĠD AS Ġis Ġable Ġto Ġcapture Ġa Ġmore Ġdiverse Ġset Ġof Ġrelevant Ġsentiment Ġexpressions Ġon Ġthe Ġtarget Ġdomain Ġfor Ġeach Ġsentiment Ġlabel . ĠThis Ġobservation Ġis Ġconsistent Ġwith Ġour Ġmotivation . ĠSince ĠNa ive NN , ĠF ANN Ġand Ġother Ġbaseline Ġmethods Ġsolely Ġtrain Ġthe Ġsentiment Ġclass ifier Ġon Ġthe Ġsource Ġdomain , Ġthe Ġlearned Ġenc oder Ġis Ġnot Ġable Ġto Ġproduce Ġdiscrim in ative Ġfeatures Ġon Ġthe Ġtarget Ġdomain . ĠD AS Ġaddresses Ġthis Ġproblem Ġby Ġrefining Ġthe Ġclass ifier Ġon Ġthe Ġtarget Ġdomain Ġwith Ġsemi - super vised Ġlearning , Ġand Ġthe Ġoverall Ġobjective Ġforces Ġthe Ġenc oder Ġto Ġlearn Ġfeature Ġrepresentations Ġthat Ġare Ġnot Ġonly Ġdomain - inv ari ant Ġbut Ġalso Ġdiscrim in ative Ġon Ġboth Ġdomains .