33,663
EXTRACTIVE
33,663 distinct review keywords 
EXTRACTIVE
Data used to evaluate recommendations. For evaluation, we use a third set of e-book annotations, namely Amazon review keywords. These review keywords are extracted from the Amazon review texts and are typically provided in the review section of books on Amazon. Our idea is to not favor one or the other data source (i.e., editor tags and Amazon search terms) when evaluating our approaches against expected tags. At the same time, we consider Amazon review keywords to be a good mixture of editor tags and search terms as they describe both the content and the users' opinions on the e-books (i.e., the readers' vocabulary). As shown in Table TABREF3 , we collect Amazon review keywords for 2,896 e-books (publishers: Kiwi, Rowohlt, Fischer, and Droemer), which leads to 33,663 distinct review keywords and on average 30 keyword assignments per e-book.
Data used to evaluate recommendations. For evaluation, we use a third set of e-book annotations, namely Amazon review keywords. These review keywords are extracted from the Amazon review texts and are typically provided in the review section of books on Amazon. Our idea is to not favor one or the other data source (i.e., editor tags and Amazon search terms) when evaluating our approaches against expected tags. At the same time, we consider Amazon review keywords to be a good mixture of editor tags and search terms as they describe both the content and the users' opinions on the e-books (i.e., the readers' vocabulary). As shown in Table TABREF3 , we collect Amazon review keywords for 2,896 e-books (publishers: Kiwi, Rowohlt, Fischer, and Droemer), which leads to 33,663 distinct review keywords and on average 30 keyword assignments per e-book.
1908.04042
Introduction </s> When Ġpeople Ġshop Ġfor Ġbooks Ġonline Ġin Ġe - book Ġstores Ġsuch Ġas , Ġe . g ., Ġthe ĠAmazon ĠKindle Ġstore , Ġthey Ġenter Ġsearch Ġterms Ġwith Ġthe Ġgoal Ġto Ġfind Ġe - books Ġthat Ġmeet Ġtheir Ġpreferences . ĠSuch Ġe - books Ġhave Ġa Ġvariety Ġof Ġmetadata Ġsuch Ġas , Ġe . g ., Ġtitle , Ġauthor Ġor Ġkeywords , Ġwhich Ġcan Ġbe Ġused Ġto Ġretrieve Ġe - books Ġthat Ġare Ġrelevant Ġto Ġthe Ġquery . ĠAs Ġa Ġconsequence , Ġfrom Ġthe Ġperspective Ġof Ġe - book Ġpublishers Ġand Ġeditors , Ġannot ating Ġe - books Ġwith Ġtags Ġthat Ġbest Ġdescribe Ġthe Ġcontent Ġand Ġwhich Ġmeet Ġthe Ġvocabulary Ġof Ġusers Ġ( e . g ., Ġwhen Ġsearching Ġand Ġreviewing Ġe - books ) Ġis Ġan Ġessential Ġtask ĠB IB REF 0 Ġ. </s> Problem Ġand Ġaim Ġof Ġthis Ġwork . ĠAn not ating Ġe - books Ġwith Ġsuitable Ġtags Ġis , Ġhowever , Ġa Ġcomplex Ġtask Ġas Ġusers ' Ġvocabulary Ġmay Ġdiffer Ġfrom Ġthe Ġone Ġof Ġeditors . ĠSuch Ġa Ġvocabulary Ġmismatch Ġyet Ġhind ers Ġeffective Ġorganization Ġand Ġretrieval ĠB IB REF 1 Ġof Ġe - books . ĠFor Ġexample , Ġwhile Ġeditors Ġmostly Ġannot ate Ġe - books Ġwith Ġdescriptive Ġtags Ġthat Ġreflect Ġthe Ġbook 's Ġcontent , ĠAmazon Ġusers Ġoften Ġsearch Ġfor Ġparts Ġof Ġthe Ġbook Ġtitle . ĠIn Ġthe Ġdata Ġwe Ġuse Ġfor Ġthe Ġpresent Ġstudy Ġ( see ĠSection ĠSEC REF 2 Ġ), Ġwe Ġfind Ġthat Ġaround Ġ30 % Ġof Ġthe ĠAmazon Ġsearch Ġterms Ġcontain Ġparts Ġof Ġe - book Ġtitles . </s> In Ġthis Ġpaper , Ġwe Ġpresent Ġour Ġwork Ġto Ġsupport Ġeditors Ġin Ġthe Ġe - book Ġannotation Ġprocess Ġwith Ġtag Ġrecommendations ĠB IB REF 2 Ġ, ĠB IB REF 3 Ġ. ĠOur Ġidea Ġis Ġto Ġexploit Ġuser - generated Ġsearch Ġquery Ġterms Ġin ĠAmazon Ġto Ġmimic Ġthe Ġvocabulary Ġof Ġusers Ġin ĠAmazon , Ġwho Ġsearch Ġfor Ġe - books . ĠWe Ġcombine Ġthese Ġsearch Ġterms Ġwith Ġtags Ġassigned Ġby Ġeditors Ġin Ġa Ġhybrid Ġtag Ġrecommendation Ġapproach . ĠThus , Ġour Ġaim Ġis Ġto Ġshow Ġthat Ġwe Ġcan Ġimprove Ġthe Ġperformance Ġof Ġtag Ġrecomm ender Ġsystems Ġfor Ġe - books Ġboth Ġconcerning Ġrecommendation Ġaccuracy Ġas Ġwell Ġas Ġsemantic Ġsimilarity Ġand Ġtag Ġrecommendation Ġdiversity . </s> Related Ġwork . ĠIn Ġtag Ġrecomm ender Ġsystems , Ġmostly Ġcontent - based Ġalgorithms Ġ( e . g ., ĠB IB REF 4 Ġ, ĠB IB REF 5 Ġ) Ġare Ġused Ġto Ġrecommend Ġtags Ġto Ġannot ate Ġresources Ġsuch Ġas Ġe - books . ĠIn Ġour Ġwork , Ġwe Ġincorporate Ġboth Ġcontent Ġfeatures Ġof Ġe - books Ġ( i . e ., Ġtitle Ġand Ġdescription Ġtext ) Ġas Ġwell Ġas ĠAmazon Ġsearch Ġterms Ġto Ġaccount Ġfor Ġthe Ġvocabulary Ġof Ġe - book Ġreaders . </s> Con cerning Ġthe Ġevaluation Ġof Ġtag Ġrecommendation Ġsystems , Ġmost Ġstudies Ġfocus Ġon Ġmeasuring Ġthe Ġaccuracy Ġof Ġtag Ġrecommendations Ġ( e . g ., ĠB IB REF 2 Ġ). ĠHowever , Ġthe Ġauthors Ġof ĠB IB REF 6 Ġsuggest Ġalso Ġto Ġuse Ġbeyond - acc uracy Ġmetrics Ġsuch Ġas Ġdiversity Ġto Ġevaluate Ġthe Ġquality Ġof Ġtag Ġrecommendations . ĠIn Ġour Ġwork , Ġwe Ġmeasure Ġrecommendation Ġdiversity Ġin Ġaddition Ġto Ġrecommendation Ġaccuracy Ġand Ġpropose Ġa Ġnovel Ġmetric Ġtermed Ġsemantic Ġsimilarity Ġto Ġvalidate Ġsemantic Ġmatches Ġof Ġtag Ġrecommendations . </s> App roach Ġand Ġfindings . ĠWe Ġexploit Ġeditor Ġtags Ġand Ġuser - generated Ġsearch Ġterms Ġas Ġinput Ġfor Ġtag Ġrecommendation Ġapproaches . ĠOur Ġevaluation Ġcomprises Ġof Ġa Ġrich Ġset Ġof Ġ19 Ġdifferent Ġalgorithms Ġto Ġrecommend Ġtags Ġfor Ġe - books , Ġwhich Ġwe Ġgroup Ġinto Ġ( i ) Ġpopularity - based , Ġ( ii ) Ġsimilarity - based Ġ( i . e ., Ġusing Ġcontent Ġinformation ), Ġand Ġ( iii ) Ġhybrid Ġapproaches . ĠWe Ġevaluate Ġour Ġapproaches Ġin Ġterms Ġof Ġaccuracy , Ġsemantic Ġsimilarity Ġand Ġdiversity Ġon Ġthe Ġreview Ġcontent Ġof ĠAmazon Ġusers , Ġwhich Ġreflects Ġthe Ġreaders ' Ġvocabulary . ĠWith Ġsemantic Ġsimilarity , Ġwe Ġmeasure Ġhow Ġsem antically Ġsimilar Ġ( based Ġon Ġlearned ĠDoc 2 V ec ĠB IB REF 7 Ġembed d ings ) Ġthe Ġlist Ġof Ġrecommended Ġtags Ġis Ġto Ġthe Ġlist Ġof Ġrelevant Ġtags . ĠWe Ġuse Ġthis Ġadditional Ġmetric Ġto Ġmeasure Ġnot Ġonly Ġexact ĠâĢ ľ h its âĢ Ŀ Ġof Ġour Ġrecommendations Ġbut Ġalso Ġsemantic Ġmatches . </s> Our Ġevaluation Ġresults Ġshow Ġthat Ġcombining Ġboth Ġdata Ġsources Ġenhances Ġthe Ġquality Ġof Ġtag Ġrecommendations Ġfor Ġannot ating Ġe - books . ĠFurthermore , Ġapproaches Ġthat Ġsolely Ġtrain Ġon ĠAmazon Ġsearch Ġterms Ġprovide Ġpoor Ġperformance Ġin Ġterms Ġof Ġaccuracy Ġbut Ġdeliver Ġgood Ġresults Ġin Ġterms Ġof Ġsemantic Ġsimilarity Ġand Ġrecommendation Ġdiversity . </s> Method </s> In Ġthis Ġsection , Ġwe Ġdescribe Ġour Ġdataset Ġas Ġwell Ġas Ġour Ġtag Ġrecommendation Ġapproaches Ġwe Ġpropose Ġto Ġannot ate Ġe - books . </s> Dat as et </s> Our Ġdataset Ġcontains Ġtwo Ġsources Ġof Ġdata , Ġone Ġto Ġgenerate Ġtag Ġrecommendations Ġand Ġanother Ġone Ġto Ġevaluate Ġtag Ġrecommendations . ĠH GV ĠG mb H Ġhas Ġcollected Ġall Ġdata Ġsources Ġand Ġwe Ġprovide Ġthe Ġdataset Ġstatistics Ġin ĠTable ĠT AB REF 3 Ġ. </s> Data Ġused Ġto Ġgenerate Ġrecommendations . ĠWe Ġemploy Ġtwo Ġsources Ġof Ġe - book Ġannotation Ġdata : Ġ( i ) Ġeditor Ġtags , Ġand Ġ( ii ) ĠAmazon Ġsearch Ġterms . ĠFor Ġeditor Ġtags , Ġwe Ġcollect Ġdata Ġof Ġ48 , 705 Ġe - books Ġfrom Ġ13 Ġpublishers , Ġnamely ĠKun st mann , ĠDel ius - K las n ig , ĠV UR , ĠH JR , ĠDi ogen es , ĠCampus , ĠKiw i , ĠBelt z , ĠCh beck , ĠRow oh lt , ĠDro emer , ĠFischer Ġand ĠNe op ub li . ĠApart Ġfrom Ġthe Ġeditor Ġtags , Ġthis Ġdata Ġcontains Ġmetadata Ġfields Ġof Ġe - books Ġsuch Ġas Ġthe ĠISBN , Ġthe Ġtitle , Ġa Ġdescription Ġtext , Ġthe Ġauthor Ġand Ġa Ġlist Ġof ĠB IS AC s , Ġwhich Ġare Ġidentifiers Ġfor Ġbook Ġcategories . </s> For Ġthe ĠAmazon Ġsearch Ġterms , Ġwe Ġcollect Ġsearch Ġquery Ġlogs Ġof Ġ21 , 243 Ġe - books Ġfor Ġ12 Ġmonths Ġ( i . e ., ĠNovember Ġ2017 Ġto ĠOctober Ġ2018 ). ĠApart Ġfrom Ġthe Ġsearch Ġterms , Ġthis Ġdata Ġcontains Ġthe Ġe - books ' ĠISBN s , Ġtitles Ġand Ġdescription Ġtexts . </s> Table ĠT AB REF 3 Ġshows Ġthat Ġthe Ġoverlap Ġof Ġe - books Ġthat Ġhave Ġeditor Ġtags Ġand ĠAmazon Ġsearch Ġterms Ġis Ġsmall Ġ( i . e ., Ġonly Ġ49 7 ). ĠFurthermore , Ġauthor Ġand ĠB IS AC Ġ( i . e ., Ġthe Ġbook Ġcategory Ġidentifier ) Ġinformation Ġare Ġprimarily Ġavailable Ġfor Ġe - books Ġthat Ġcontain Ġeditor Ġtags . ĠConsequently , Ġboth Ġdata Ġsources Ġprovide Ġcomplementary Ġinformation , Ġwhich Ġunder pins Ġthe Ġintention Ġof Ġthis Ġwork , Ġi . e ., Ġto Ġevaluate Ġtag Ġrecommendation Ġapproaches Ġusing Ġannotation Ġsources Ġfrom Ġdifferent Ġcontexts . </s> Data Ġused Ġto Ġevaluate Ġrecommendations . ĠFor Ġevaluation , Ġwe Ġuse Ġa Ġthird Ġset Ġof Ġe - book Ġannotations , Ġnamely ĠAmazon Ġreview Ġkeywords . ĠThese Ġreview Ġkeywords Ġare Ġextracted Ġfrom Ġthe ĠAmazon Ġreview Ġtexts Ġand Ġare Ġtypically Ġprovided Ġin Ġthe Ġreview Ġsection Ġof Ġbooks Ġon ĠAmazon . ĠOur Ġidea Ġis Ġto Ġnot Ġfavor Ġone Ġor Ġthe Ġother Ġdata Ġsource Ġ( i . e ., Ġeditor Ġtags Ġand ĠAmazon Ġsearch Ġterms ) Ġwhen Ġevaluating Ġour Ġapproaches Ġagainst Ġexpected Ġtags . ĠAt Ġthe Ġsame Ġtime , Ġwe Ġconsider ĠAmazon Ġreview Ġkeywords Ġto Ġbe Ġa Ġgood Ġmixture Ġof Ġeditor Ġtags Ġand Ġsearch Ġterms Ġas Ġthey Ġdescribe Ġboth Ġthe Ġcontent Ġand Ġthe Ġusers ' Ġopinions Ġon Ġthe Ġe - books Ġ( i . e ., Ġthe Ġreaders ' Ġvocabulary ). ĠAs Ġshown Ġin ĠTable ĠT AB REF 3 Ġ, Ġwe Ġcollect ĠAmazon Ġreview Ġkeywords Ġfor Ġ2 , 896 Ġe - books Ġ( pub lisher s : ĠKiw i , ĠRow oh lt , ĠFischer , Ġand ĠDro emer ), Ġwhich Ġleads Ġto Ġ33 , 663 Ġdistinct Ġreview Ġkeywords Ġand Ġon Ġaverage Ġ30 Ġkeyword Ġassignments Ġper Ġe - book . </s> Tag ĠRecommend ation ĠAppro aches </s> We Ġimplement Ġthree Ġtypes Ġof Ġtag Ġrecommendation Ġapproaches , Ġi . e ., Ġ( i ) Ġpopularity - based , Ġ( ii ) Ġsimilarity - based Ġ( i . e ., Ġusing Ġcontent Ġinformation ), Ġand Ġ( iii ) Ġhybrid Ġapproaches . ĠDue Ġto Ġthe Ġlack Ġof Ġpersonalized Ġtags Ġ( i . e ., Ġwe Ġdo Ġnot Ġknow Ġwhich Ġuser Ġhas Ġassigned Ġa Ġtag ), Ġwe Ġdo Ġnot Ġimplement Ġother Ġtypes Ġof Ġalgorithms Ġsuch Ġas Ġcollaborative Ġfiltering ĠB IB REF 8 Ġ. ĠIn Ġtotal , Ġwe Ġevaluate Ġ19 Ġdifferent Ġalgorithms Ġto Ġrecommend Ġtags Ġfor Ġannot ating Ġe - books . </s> Pop ularity - based Ġapproaches . ĠWe Ġrecommend Ġthe Ġmost Ġfrequently Ġused Ġtags Ġin Ġthe Ġdataset , Ġwhich Ġis Ġa Ġcommon Ġstrategy Ġfor Ġtag Ġrecommendations ĠB IB REF 9 Ġ. ĠThat Ġis , Ġa Ġmost Ġpopular ĠIN LINE FORM 0 Ġapproach Ġfor Ġeditor Ġtags Ġand Ġa Ġmost Ġpopular ĠIN LINE FORM 1 Ġapproach Ġfor ĠAmazon Ġsearch Ġterms . ĠFor Ġe - books , Ġfor Ġwhich Ġwe Ġalso Ġhave Ġauthor Ġ(= ĠIN LINE FORM 2 Ġand ĠIN LINE FORM 3 Ġ) Ġor ĠB IS AC Ġ(= ĠIN LINE FORM 4 Ġand ĠIN LINE FORM 5 Ġ) Ġinformation , Ġwe Ġuse Ġthese Ġfeatures Ġto Ġfurther Ġfilter Ġthe Ġrecommended Ġtags , Ġi . e ., Ġto Ġonly Ġrecommend Ġtags Ġthat Ġwere Ġused Ġto Ġannot ate Ġe - books Ġof Ġa Ġspecific Ġauthor Ġor Ġa Ġspecific ĠB IS AC . </s> We Ġcombine Ġboth Ġdata Ġsources Ġ( i . e ., Ġeditor Ġtags Ġand ĠAmazon Ġsearch Ġterms ) Ġusing Ġa Ġround - ro bin Ġcombination Ġstrategy , Ġwhich Ġensures Ġan Ġequal Ġweight Ġfor Ġboth Ġsources . ĠThis Ġgives Ġus Ġthree Ġadditional Ġpopularity - based Ġalgorithms Ġ(= ĠIN LINE FORM 0 Ġ, ĠIN LINE FORM 1 Ġand ĠIN LINE FORM 2 Ġ). </s> Similar ity - based Ġapproaches . ĠWe Ġexploit Ġthe Ġtextual Ġcontent Ġof Ġe - books Ġ( i . e ., Ġdescription Ġor Ġtitle ) Ġto Ġrecommend Ġrelevant Ġtags ĠB IB REF 10 Ġ. ĠFor Ġthis , Ġwe Ġfirst Ġemploy Ġa Ġcontent - based Ġfiltering Ġapproach ĠB IB REF 11 Ġbased Ġon ĠTF - ID F ĠB IB REF 12 Ġto Ġfind Ġtop - ĠIN LINE FORM 0 Ġsimilar Ġe - books . ĠFor Ġeach Ġof Ġthe Ġsimilar Ġe - books , Ġwe Ġthen Ġeither Ġextract Ġthe Ġassigned Ġeditor Ġtags Ġ(= ĠIN LINE FORM 2 Ġand ĠIN LINE FORM 3 Ġ) Ġor Ġthe ĠAmazon Ġsearch Ġterms Ġ(= ĠIN LINE FORM 4 Ġand ĠIN LINE FORM 5 Ġ). ĠTo Ġcombine Ġthe Ġtags Ġof Ġthe Ġtop - ĠIN LINE FORM 6 Ġsimilar Ġe - books , Ġwe Ġuse Ġthe Ġcross - source Ġalgorithm ĠB IB REF 13 Ġ, Ġwhich Ġfavors Ġtags Ġthat Ġwere Ġused Ġto Ġannot ate Ġmore Ġthan Ġone Ġsimilar Ġe - book Ġ( i . e ., Ġtags Ġthat Ġcome Ġfrom Ġmultiple Ġrecommendation Ġsources ). ĠThe Ġfinal Ġtag Ġrele v ancy Ġis Ġcalculated Ġas : ĠDIS PLAY FORM 0 </s> where ĠIN LINE FORM 0 Ġdenotes Ġthe Ġnumber Ġof Ġdistinct Ġe - books , Ġwhich Ġyielded Ġthe Ġrecommendation Ġof Ġtag ĠIN LINE FORM 1 Ġ, Ġto Ġfavor Ġtags Ġthat Ġcome Ġfrom Ġmultiple Ġsources Ġand ĠIN LINE FORM 2 Ġis Ġthe Ġsimilarity Ġscore Ġof Ġthe Ġcorresponding Ġe - book . ĠWe Ġagain Ġuse Ġa Ġround - ro bin Ġstrategy Ġto Ġcombine Ġboth Ġdata Ġsources Ġ(= ĠIN LINE FORM 3 Ġand ĠIN LINE FORM 4 Ġ). </s> Hy brid Ġapproaches . ĠWe Ġuse Ġthe Ġpreviously Ġmentioned Ġcross - source Ġalgorithm ĠB IB REF 13 Ġto Ġconstruct Ġfour Ġhybrid Ġrecommendation Ġapproaches . ĠIn Ġthis Ġcase , Ġtags Ġare Ġfavored Ġthat Ġare Ġrecommended Ġby Ġmore Ġthan Ġone Ġalgorithm . </s> H ence , Ġto Ġcreate Ġa Ġpopularity - based Ġhybrid Ġ(= ĠIN LINE FORM 0 Ġ), Ġwe Ġcombine Ġthe Ġbest Ġthree Ġperforming Ġpopularity - based Ġapproaches Ġfrom Ġthe Ġones Ġ( i ) Ġwithout Ġany Ġcontextual Ġsignal , Ġ( ii ) Ġwith Ġthe Ġauthor Ġas Ġcontext , Ġand Ġ( iii ) Ġwith ĠB IS AC Ġas Ġcontext . ĠIn Ġthe Ġcase Ġof Ġthe Ġsimilarity - based Ġhybrid Ġ(= ĠIN LINE FORM 1 Ġ), Ġwe Ġutilize Ġthe Ġtwo Ġbest Ġperforming Ġsimilarity - based Ġapproaches Ġfrom Ġthe Ġones Ġ( i ) Ġwhich Ġuse Ġthe Ġtitle , Ġand Ġ( ii ) Ġwhich Ġuse Ġthe Ġdescription Ġtext . ĠWe Ġfurther Ġdefine ĠIN LINE FORM 2 Ġ, Ġa Ġhybrid Ġapproach Ġthat Ġcombines Ġthe Ġthree Ġpopularity - based Ġmethods Ġof ĠIN LINE FORM 3 Ġand Ġthe Ġtwo Ġsimilarity - based Ġapproaches Ġof ĠIN LINE FORM 4 Ġ. ĠFinally , Ġwe Ġdefine ĠIN LINE FORM 5 Ġas Ġa Ġhybrid Ġapproach Ġthat Ġuses Ġthe Ġbest Ġperforming Ġpopularity - based Ġand Ġthe Ġbest Ġperforming Ġsimilarity - based Ġapproach Ġ( see ĠFigure ĠFIG REF 11 Ġin ĠSection ĠSEC REF 4 Ġfor Ġmore Ġdetails Ġabout Ġthe Ġparticular Ġalgorithm Ġcombinations ). </s> Exper imental ĠSetup </s> In Ġthis Ġsection , Ġwe Ġdescribe Ġour Ġevaluation Ġprotocol Ġas Ġwell Ġas Ġthe Ġmeasures Ġwe Ġuse Ġto Ġevaluate Ġand Ġcompare Ġour Ġtag Ġrecommendation Ġapproaches . </s> E val uation ĠProtocol </s> For Ġevaluation , Ġwe Ġuse Ġthe Ġthird Ġset Ġof Ġe - book Ġannotations , Ġnamely ĠAmazon Ġreview Ġkeywords . ĠAs Ġdescribed Ġin ĠSection ĠSEC REF 1 Ġ, Ġthese Ġreview Ġkeywords Ġare Ġextracted Ġfrom Ġthe ĠAmazon Ġreview Ġtexts Ġand Ġthus , Ġreflect Ġthe Ġusers ' Ġvocabulary . ĠWe Ġevaluate Ġour Ġapproaches Ġfor Ġthe Ġ2 , 896 Ġe - books , Ġfor Ġwhom Ġwe Ġgot Ġreview Ġkeywords . ĠTo Ġfollow Ġcommon Ġpractice Ġfor Ġtag Ġrecommendation Ġevaluation ĠB IB REF 14 Ġ, Ġwe Ġpredict Ġthe Ġassigned Ġreview Ġkeywords Ġ(= Ġour Ġtest Ġset ) Ġfor Ġrespective Ġe - books . </s> E val uation ĠMet rics </s> In Ġthis Ġwork , Ġwe Ġmeasure Ġ( i ) Ġrecommendation Ġaccuracy , Ġ( ii ) Ġsemantic Ġsimilarity , Ġand Ġ( iii ) Ġrecommendation Ġdiversity Ġto Ġevaluate Ġthe Ġquality Ġof Ġour Ġapproaches Ġfrom Ġdifferent Ġperspectives . </s> Recommend ation Ġaccuracy . ĠWe Ġuse ĠNormal ized ĠDiscount ed ĠCum ulative ĠGain Ġ( n DC G ) ĠB IB REF 15 Ġto Ġmeasure Ġthe Ġaccuracy Ġof Ġthe Ġtag Ġrecommendation Ġapproaches . ĠThe Ġn DC G Ġmeasure Ġis Ġa Ġstandard Ġranking - dependent Ġmetric Ġthat Ġnot Ġonly Ġmeasures Ġhow Ġmany Ġtags Ġcan Ġbe Ġcorrectly Ġpredicted Ġbut Ġalso Ġtakes Ġinto Ġaccount Ġtheir Ġposition Ġin Ġthe Ġrecommendation Ġlist Ġwith Ġlength Ġof ĠIN LINE FORM 0 Ġ. ĠIt Ġis Ġbased Ġon Ġthe ĠDiscount ed ĠCumm ulative ĠGain , Ġwhich Ġis Ġgiven Ġby : ĠDIS PLAY FORM 0 </s> where ĠIN LINE FORM 0 Ġis Ġa Ġfunction Ġthat Ġreturns Ġ1 Ġif Ġthe Ġrecommended Ġtag Ġat Ġposition ĠIN LINE FORM 1 Ġin Ġthe Ġrecommended Ġlist Ġis Ġrelevant . ĠWe Ġthen Ġcalculate ĠDC G @ ĠIN LINE FORM 2 Ġfor Ġevery Ġevaluated Ġe - book Ġby Ġdividing ĠDC G @ ĠIN LINE FORM 3 Ġwith Ġthe Ġideal ĠDC G Ġvalue Ġi DC G @ ĠIN LINE FORM 4 Ġ, Ġwhich Ġis Ġthe Ġhighest Ġpossible ĠDC G Ġvalue Ġthat Ġcan Ġbe Ġachieved Ġif Ġall Ġthe Ġrelevant Ġtags Ġwould Ġbe Ġrecommended Ġin Ġthe Ġcorrect Ġorder . ĠIt Ġis Ġgiven Ġby Ġthe Ġfollowing Ġformula ĠB IB REF 15 Ġ: ĠDIS PLAY FORM 0 </s> Sem antic Ġsimilarity . ĠOne Ġprec ond ition Ġof Ġstandard Ġrecommendation Ġaccuracy Ġmeasures Ġis Ġthat Ġto Ġgenerate Ġa ĠâĢ ľ hit âĢ Ŀ , Ġthe Ġrecommended Ġtag Ġneeds Ġto Ġbe Ġan Ġexact Ġsynt act ical Ġmatch Ġto Ġthe Ġone Ġfrom Ġthe Ġtest Ġset . ĠWhen Ġtags Ġare Ġrecommended Ġfrom Ġone Ġdata Ġsource Ġand Ġcompared Ġto Ġtags Ġfrom Ġanother Ġsource , Ġthis Ġcan Ġbe Ġproblematic . ĠFor Ġexample , Ġif Ġwe Ġrecommend Ġthe Ġtag ĠâĢ ľ vict im âĢ Ŀ Ġbut Ġexpect Ġthe Ġtag ĠâĢ ľ pre y âĢ Ŀ , Ġwe Ġwould Ġmark Ġthis Ġas Ġa Ġmismatch , Ġtherefore Ġbeing Ġa Ġbad Ġrecommendation . ĠBut Ġif Ġwe Ġknow Ġthat Ġthe Ġcorresponding Ġe - book Ġis Ġa Ġcrime Ġnovel , Ġthe Ġrecommended Ġtag Ġwould Ġbe Ġ( sem antically ) Ġdescriptive Ġto Ġreflect Ġthe Ġbook 's Ġcontent . ĠHence , Ġin Ġthis Ġpaper , Ġwe Ġpropose Ġto Ġadditionally Ġmeasure Ġthe Ġsemantic Ġsimilarity Ġbetween Ġrecommended Ġtags Ġand Ġtags Ġfrom Ġthe Ġtest Ġset Ġ( i . e ., Ġthe ĠAmazon Ġreview Ġkeywords ). </s> Over Ġthe Ġlast Ġfour Ġyears , Ġthere Ġhave Ġbeen Ġseveral Ġnotable Ġpublications Ġin Ġthe Ġarea Ġof Ġapplying Ġdeep Ġlearning Ġto Ġuncover Ġsemantic Ġrelationships Ġbetween Ġtextual Ġcontent Ġ( e . g ., Ġby Ġlearning Ġword Ġembed d ings Ġwith ĠWord 2 V ec ĠB IB REF 16 Ġ, ĠB IB REF 17 Ġ). ĠBased Ġon Ġthis , Ġwe Ġpropose Ġan Ġalternative Ġmeasure Ġof Ġrecommendation Ġquality Ġby Ġlearning Ġthe Ġsemantic Ġrelationships Ġfrom Ġboth Ġvoc ab ular ies Ġand Ġthen Ġusing Ġit Ġto Ġcompare Ġhow Ġsem antically Ġsimilar Ġthe Ġrecommended Ġtags Ġare Ġto Ġthe Ġexpected Ġreview Ġkeywords . ĠFor Ġthis , Ġwe Ġfirst Ġextract Ġthe Ġtextual Ġcontent Ġin Ġthe Ġform Ġof Ġthe Ġdescription Ġtext , Ġtitle , Ġeditor Ġtags Ġand ĠAmazon Ġsearch Ġterms Ġof Ġe - books Ġfrom Ġour Ġdataset . ĠWe Ġthen Ġtrain Ġa ĠDoc 2 V ec ĠB IB REF 7 Ġmodel Ġon Ġthe Ġcontent . ĠThen , Ġwe Ġuse Ġthe Ġmodel Ġto Ġinfer Ġthe Ġlatent Ġrepresentation Ġfor Ġboth Ġthe Ġcomplete Ġlist Ġof Ġrecommended Ġtags Ġas Ġwell Ġas Ġthe Ġlist Ġof Ġexpected Ġtags Ġfrom Ġthe Ġtest Ġset . ĠFinally , Ġwe Ġuse Ġthe Ġcos ine Ġsimilarity Ġmeasure Ġto Ġcalculate Ġhow Ġsem antically Ġsimilar Ġthese Ġtwo Ġlists Ġare . </s> Recommend ation Ġdiversity . ĠAs Ġdefined Ġin ĠB IB REF 18 Ġ, Ġwe Ġcalculate Ġrecommendation Ġdiversity Ġas Ġthe Ġaverage Ġdiss imilar ity Ġof Ġall Ġpairs Ġof Ġtags Ġin Ġthe Ġlist Ġof Ġrecommended Ġtags . ĠThus , Ġgiven Ġa Ġdistance Ġfunction ĠIN LINE FORM 0 Ġthat Ġcorresponds Ġto Ġthe Ġdiss imilar ity Ġbetween Ġtwo Ġtags ĠIN LINE FORM 1 Ġand ĠIN LINE FORM 2 Ġin Ġthe Ġlist Ġof Ġrecommended Ġtags , ĠIN LINE FORM 3 Ġis Ġgiven Ġas Ġthe Ġaverage Ġdiss imilar ity Ġof Ġall Ġpairs Ġof Ġtags : ĠDIS PLAY FORM 0 </s> where ĠIN LINE FORM 0 Ġis Ġthe Ġnumber Ġof Ġevaluated Ġe - books Ġand Ġthe Ġdiss imilar ity Ġfunction Ġis Ġdefined Ġas ĠIN LINE FORM 1 Ġ. ĠIn Ġour Ġexperiments , Ġwe Ġuse Ġthe Ġpreviously Ġtrained ĠDoc 2 V ec Ġmodel Ġto Ġextract Ġthe Ġlatent Ġrepresentation Ġof Ġa Ġspecific Ġtag . ĠThe Ġsimilarity Ġof Ġtwo Ġtags ĠIN LINE FORM 2 Ġis Ġthen Ġcalculated Ġwith Ġthe ĠCos ine Ġsimilarity Ġmeasure Ġusing Ġthe Ġlatent Ġvector Ġrepresentations Ġof Ġrespective Ġtags ĠIN LINE FORM 3 Ġand ĠIN LINE FORM 4 Ġ. </s> Results </s> Con cerning Ġtag Ġrecommendation Ġaccuracy , Ġin Ġthis Ġsection , Ġwe Ġreport Ġresults Ġfor Ġdifferent Ġvalues Ġof ĠIN LINE FORM 0 Ġ( i . e ., Ġnumber Ġof Ġrecommended Ġtags ). ĠFor Ġthe Ġbeyond - acc uracy Ġexperiment , Ġwe Ġuse Ġthe Ġfull Ġlist Ġof Ġrecommended Ġtags Ġ( i . e ., ĠIN LINE FORM 1 Ġ). </s> Recommend ation ĠAccuracy ĠEvaluation </s> Figure ĠFIG REF 11 Ġshows Ġthe Ġresults Ġof Ġthe Ġaccuracy Ġexperiment Ġfor Ġthe Ġ( i ) Ġpopularity - based , Ġ( ii ) Ġsimilarity - based , Ġand Ġ( iii ) Ġhybrid Ġtag Ġrecommendation Ġapproaches . </s> Pop ularity - based Ġapproaches . ĠIn ĠFigure ĠFIG REF 11 Ġ, Ġwe Ġsee Ġthat Ġpopularity - based Ġapproaches Ġbased Ġon Ġeditor Ġtags Ġtend Ġto Ġperform Ġbetter Ġthan Ġif Ġtrained Ġon ĠAmazon Ġsearch Ġterms . ĠIf Ġwe Ġtake Ġinto Ġaccount Ġcontextual Ġinformation Ġlike ĠB IS AC Ġor Ġauthor , Ġwe Ġcan Ġfurther Ġimprove Ġaccuracy Ġin Ġterms Ġof ĠIN LINE FORM 0 Ġ. ĠThat Ġis , Ġwe Ġfind Ġthat Ġusing Ġpopular Ġtags Ġfrom Ġe - books Ġof Ġa Ġspecific Ġauthor Ġleads Ġto Ġthe Ġbest Ġaccuracy Ġof Ġthe Ġpopularity - based Ġapproaches . ĠThis Ġsuggests Ġthat Ġeditors Ġand Ġreaders Ġdo Ġseem Ġto Ġreuse Ġtags Ġfor Ġe - books Ġof Ġsame Ġauthors . ĠIf Ġwe Ġuse Ġboth Ġeditor Ġtags Ġand ĠAmazon Ġsearch Ġterms , Ġwe Ġcan Ġfurther Ġincrease Ġaccuracy , Ġespecially Ġfor Ġhigher Ġvalues Ġof ĠIN LINE FORM 1 Ġlike Ġin Ġthe Ġcase Ġof ĠIN LINE FORM 2 Ġ. ĠThis Ġis , Ġhowever , Ġnot Ġthe Ġcase Ġfor ĠIN LINE FORM 3 Ġas Ġthe Ġaccuracy Ġof Ġthe Ġintegrated ĠIN LINE FORM 4 Ġapproach Ġis Ġlow . ĠThe Ġreason Ġfor Ġthis Ġis Ġthe Ġlimited Ġamount Ġof Ġe - books Ġfrom Ġwithin Ġthe ĠAmazon Ġsearch Ġquery Ġlogs Ġthat Ġhave ĠB IS AC Ġinformation Ġ( i . e ., Ġonly ĠIN LINE FORM 5 Ġ). </s> Similar ity - based Ġapproaches . ĠWe Ġfurther Ġimprove Ġaccuracy Ġif Ġwe Ġfirst Ġfind Ġsimilar Ġe - books Ġand Ġthen Ġextract Ġtheir Ġtop - ĠIN LINE FORM 0 Ġtags Ġin Ġa Ġcross - source Ġmanner Ġas Ġdescribed Ġin ĠSection ĠSEC REF 4 Ġ. </s> As Ġshown Ġin ĠFigure ĠFIG REF 11 Ġ, Ġusing Ġthe Ġdescription Ġtext Ġto Ġfind Ġsimilar Ġe - books Ġresults Ġin Ġmore Ġaccurate Ġtag Ġrecommendations Ġthan Ġusing Ġthe Ġtitle Ġ( i . e ., ĠIN LINE FORM 0 Ġfor ĠIN LINE FORM 1 Ġ). ĠThis Ġis Ġsomehow Ġexpected Ġas Ġthe Ġdescription Ġtext Ġconsists Ġof Ġa Ġbigger Ġcorpus Ġof Ġwords Ġ( i . e ., Ġmultiple Ġsentences ) Ġthan Ġthe Ġtitle . ĠCon cerning Ġthe Ġcollected ĠAmazon Ġsearch Ġquery Ġlogs , Ġextracting Ġand Ġthen Ġrecommending Ġtags Ġfrom Ġthis Ġsource Ġresults Ġin Ġa Ġmuch Ġlower Ġaccuracy Ġperformance . ĠThus , Ġthese Ġresults Ġalso Ġsuggest Ġto Ġinvestigate Ġbeyond - acc uracy Ġmetrics Ġas Ġdone Ġin ĠSection ĠSEC REF 17 Ġ. </s> Hy brid Ġapproaches . ĠFigure ĠFIG REF 11 Ġshows Ġthe Ġaccuracy Ġresults Ġof Ġthe Ġfour Ġhybrid Ġapproaches . ĠBy Ġcombining Ġthe Ġbest Ġthree Ġpopularity - based Ġapproaches , Ġwe Ġoutper form Ġall Ġof Ġthe Ġinitially Ġevaluated Ġpopularity Ġalgorithms Ġ( i . e ., ĠIN LINE FORM 0 Ġfor ĠIN LINE FORM 1 Ġ). ĠOn Ġthe Ġcontrary , Ġthe Ġcombination Ġof Ġthe Ġtwo Ġbest Ġperforming Ġsimilarity - based Ġapproaches ĠIN LINE FORM 2 Ġand ĠIN LINE FORM 3 Ġdoes Ġnot Ġyield Ġbetter Ġaccuracy . ĠThe Ġnegative Ġimpact Ġof Ġusing Ġa Ġlower - performing Ġapproach Ġsuch Ġas ĠIN LINE FORM 4 Ġwithin Ġa Ġhybrid Ġcombination Ġcan Ġalso Ġbe Ġobserved Ġin ĠIN LINE FORM 5 Ġfor Ġlower Ġvalues Ġof ĠIN LINE FORM 6 Ġ. ĠOverall , Ġthis Ġconfirms Ġour Ġinitial Ġintuition Ġthat Ġcombining Ġthe Ġbest Ġperforming Ġpopularity - based Ġapproach Ġwith Ġthe Ġbest Ġsimilarity - based Ġapproach Ġshould Ġresult Ġin Ġthe Ġhighest Ġaccuracy Ġ( i . e ., ĠIN LINE FORM 7 Ġfor ĠIN LINE FORM 8 Ġ). ĠMoreover , Ġour Ġgoal , Ġnamely Ġto Ġexploit Ġeditor Ġtags Ġin Ġcombination Ġwith Ġsearch Ġterms Ġused Ġby Ġreaders Ġto Ġincrease Ġthe Ġmetadata Ġquality Ġof Ġe - books , Ġis Ġshown Ġto Ġbe Ġbest Ġsupported Ġby Ġapplying Ġhybrid Ġapproaches Ġas Ġthey Ġprovide Ġthe Ġbest Ġprediction Ġresults . </s> Beyond - Acc uracy ĠEvaluation </s> Figure ĠFIG REF 16 Ġillustrates Ġthe Ġresults Ġof Ġthe Ġexperiments , Ġwhich Ġmeasure Ġthe Ġrecommendation Ġimpact Ġbeyond - acc uracy . </s> Sem antic Ġsimilarity . ĠFigure ĠFIG REF 16 Ġillustrates Ġthe Ġresults Ġof Ġour Ġproposed Ġsemantic Ġsimilarity Ġmeasure . ĠTo Ġcompare Ġour Ġproposed Ġmeasure Ġto Ġstandard Ġaccuracy Ġmeasures Ġsuch Ġas ĠIN LINE FORM 0 Ġ, Ġwe Ġuse ĠKendall 's ĠTau Ġrank Ġcorrelation ĠB IB REF 19 Ġas Ġsuggested Ġby ĠB IB REF 20 Ġfor Ġautomatic Ġevaluation Ġof Ġinformation - ordering Ġtasks . ĠFrom Ġthat , Ġwe Ġrank Ġour Ġrecommendation Ġapproaches Ġaccording Ġto Ġboth Ġaccuracy Ġand Ġsemantic Ġsimilarity Ġand Ġcalculate Ġthe Ġrelation Ġbetween Ġboth Ġrankings . ĠThis Ġresults Ġin ĠIN LINE FORM 1 Ġwith Ġa Ġp - value Ġ< ĠIN LINE FORM 2 Ġ, Ġwhich Ġsuggests Ġa Ġhigh Ġcorrelation Ġbetween Ġthe Ġsemantic Ġsimilarity Ġand Ġthe Ġstandard Ġaccuracy Ġmeasure . </s> Therefore , Ġthe Ġsemantic Ġsimilarity Ġmeasure Ġhelps Ġus Ġinterpret Ġthe Ġrecommendation Ġquality . ĠFor Ġinstance , Ġwe Ġachieve Ġthe Ġlowest ĠIN LINE FORM 0 Ġvalues Ġwith Ġthe Ġsimilarity - based Ġapproaches Ġthat Ġrecommend ĠAmazon Ġsearch Ġterms Ġ( i . e ., ĠIN LINE FORM 1 Ġand ĠIN LINE FORM 2 Ġ). ĠWhen Ġcomparing Ġthese Ġresults Ġwith Ġothers Ġfrom ĠFigure ĠFIG REF 11 Ġ, Ġa Ġconclusion Ġcould Ġbe Ġquickly Ġdrawn Ġthat Ġthe Ġrecommended Ġtags Ġare Ġmerely Ġunus able . ĠHowever , Ġby Ġlooking Ġat ĠFigure ĠFIG REF 16 Ġ, Ġwe Ġsee Ġthat , Ġalthough Ġthese Ġapproaches Ġdo Ġnot Ġprovide Ġthe Ġhighest Ġrecommendation Ġaccuracy , Ġthey Ġstill Ġresult Ġin Ġtag Ġrecommendations Ġthat Ġare Ġsem antically Ġrelated Ġat Ġa Ġhigh Ġdegree Ġto Ġthe Ġexpected Ġannotations Ġfrom Ġthe Ġtest Ġset . ĠOverall , Ġthis Ġsuggests Ġthat Ġapproaches , Ġwhich Ġprovide Ġa Ġpoor Ġaccuracy Ġperformance Ġconcerning ĠIN LINE FORM 4 Ġbut Ġprovide Ġa Ġgood Ġperformance Ġregarding Ġsemantic Ġsimilarity Ġcould Ġstill Ġbe Ġhelpful Ġfor Ġannot ating Ġe - books . </s> Recommend ation Ġdiversity . ĠFigure ĠFIG REF 16 Ġshows Ġthe Ġdiversity Ġof Ġthe Ġtag Ġrecommendation Ġapproaches . ĠWe Ġachieve Ġthe Ġhighest Ġdiversity Ġwith Ġthe Ġsimilarity - based Ġapproaches , Ġwhich Ġextract ĠAmazon Ġsearch Ġterms . ĠTheir Ġaccuracy Ġis , Ġhowever , Ġvery Ġlow . ĠThus , Ġthe Ġcombination Ġof Ġthe Ġtwo Ġvoc ab ular ies Ġcan Ġprovide Ġa Ġgood Ġtrade - off Ġbetween Ġrecommendation Ġaccuracy Ġand Ġdiversity . </s> Conclusion Ġand ĠFuture ĠWork </s> In Ġthis Ġpaper , Ġwe Ġpresent Ġour Ġwork Ġto Ġsupport Ġeditors Ġin Ġthe Ġe - book Ġannotation Ġprocess . ĠSpecifically , Ġwe Ġaim Ġto Ġprovide Ġtag Ġrecommendations Ġthat Ġincorporate Ġboth Ġthe Ġvocabulary Ġof Ġthe Ġeditors Ġand Ġe - book Ġreaders . ĠTherefore , Ġwe Ġtrain Ġvarious Ġconfigurations Ġof Ġtag Ġrecomm ender Ġapproaches Ġon Ġeditors ' Ġtags Ġand ĠAmazon Ġsearch Ġterms Ġand Ġevaluate Ġthem Ġon Ġa Ġdataset Ġcontaining ĠAmazon Ġreview Ġkeywords . ĠWe Ġfind Ġthat Ġcombining Ġboth Ġdata Ġsources Ġenhances Ġthe Ġquality Ġof Ġtag Ġrecommendations Ġfor Ġannot ating Ġe - books . ĠFurthermore , Ġwhile Ġapproaches Ġthat Ġtrain Ġonly Ġon ĠAmazon Ġsearch Ġterms Ġprovide Ġpoor Ġperformance Ġconcerning Ġrecommendation Ġaccuracy , Ġwe Ġshow Ġthat Ġthey Ġstill Ġoffer Ġhelpful Ġannotations Ġconcerning Ġrecommendation Ġdiversity Ġas Ġwell Ġas Ġour Ġnovel Ġsemantic Ġsimilarity Ġmetric . </s> Future Ġwork . ĠFor Ġfuture Ġwork , Ġwe Ġplan Ġto Ġvalidate Ġour Ġfindings Ġusing Ġanother Ġdataset , Ġe . g ., Ġby Ġrecommending Ġtags Ġfor Ġscientific Ġarticles Ġand Ġbooks Ġin ĠBib S onomy . ĠWith Ġthis , Ġwe Ġaim Ġto Ġdemonstrate Ġthe Ġusefulness Ġof Ġthe Ġproposed Ġapproach Ġin Ġa Ġsimilar Ġdomain Ġand Ġto Ġenhance Ġthe Ġreprodu c ibility Ġof Ġour Ġresults Ġby Ġusing Ġan Ġopen Ġdataset . </s> Moreover , Ġwe Ġplan Ġto Ġevaluate Ġour Ġtag Ġrecommendation Ġapproaches Ġin Ġa Ġstudy Ġwith Ġdomain Ġusers . ĠAlso , Ġwe Ġwant Ġto Ġimprove Ġour Ġsimilarity - based Ġapproaches Ġby Ġintegrating Ġnovel Ġembed ding Ġapproaches ĠB IB REF 16 Ġ, ĠB IB REF 17 Ġas Ġwe Ġdid , Ġfor Ġexample , Ġwith Ġour Ġproposed Ġsemantic Ġsimilarity Ġevaluation Ġmetric . ĠFinally , Ġwe Ġaim Ġto Ġincorporate Ġexplanations Ġfor Ġrecommended Ġtags Ġso Ġthat Ġeditors Ġof Ġe - book Ġannotations Ġreceive Ġadditional Ġsupport Ġin Ġannot ating Ġe - books ĠB IB REF 21 Ġ. ĠBy Ġmaking Ġthe Ġunderlying Ġ( sem antic ) Ġreasoning Ġvisible Ġto Ġthe Ġeditor Ġwho Ġis Ġin Ġcharge Ġof Ġtail oring Ġannotations , Ġwe Ġaim Ġto Ġsupport Ġtwo Ġgoals : Ġ( i ) Ġallowing Ġreaders Ġto Ġdiscover Ġe - books Ġmore Ġefficiently , Ġand Ġ( ii ) Ġenabling Ġpublishers Ġto Ġleverage Ġsemi - automatic Ġcategor ization Ġprocesses Ġfor Ġe - books . ĠIn Ġturn , Ġproviding Ġexplanations Ġfost ers Ġcontrol Ġover Ġwhich Ġvocabulary Ġto Ġchoose Ġwhen Ġtagging Ġe - books Ġfor Ġdifferent Ġapplication Ġcontexts . </s> Acknowled gments . ĠThe Ġauthors Ġwould Ġlike Ġto Ġthank ĠPeter ĠLang s , ĠJan - Philipp ĠWolf Ġand ĠAly ona ĠSch ra a Ġfrom ĠH GV ĠG mb H Ġfor Ġproviding Ġthe Ġe - book Ġannotation Ġdata . ĠThis Ġwork Ġwas Ġfunded Ġby Ġthe ĠKnow - Center ĠG mb H Ġ( FF G ĠCOM ET ĠProgram ), Ġthe ĠFF G ĠData ĠMarket ĠAustria Ġproject Ġand Ġthe ĠAI 4 EU Ġproject Ġ( EU Ġgrant Ġ8 256 19 ). ĠThe ĠKnow - Center ĠG mb H Ġis Ġfunded Ġwithin Ġthe ĠAustrian ĠCOM ET ĠProgram Ġ- ĠCompet ence ĠCenters Ġfor ĠExcellent ĠTechnologies Ġ- Ġunder Ġthe Ġausp ices Ġof Ġthe ĠAustrian ĠMinistry Ġof ĠTransport , ĠInnovation Ġand ĠTechnology , Ġthe ĠAustrian ĠMinistry Ġof ĠEconomics Ġand ĠLabor Ġand Ġby Ġthe ĠState Ġof ĠSty ria . ĠCOM ET Ġis Ġmanaged Ġby Ġthe ĠAustrian ĠResearch ĠPromotion ĠAgency Ġ( FF G ).
Introduction When people shop for books online in e-book stores such as, e.g., the Amazon Kindle store, they enter search terms with the goal to find e-books that meet their preferences. Such e-books have a variety of metadata such as, e.g., title, author or keywords, which can be used to retrieve e-books that are relevant to the query. As a consequence, from the perspective of e-book publishers and editors, annotating e-books with tags that best describe the content and which meet the vocabulary of users (e.g., when searching and reviewing e-books) is an essential task BIBREF0 . Problem and aim of this work. Annotating e-books with suitable tags is, however, a complex task as users' vocabulary may differ from the one of editors. Such a vocabulary mismatch yet hinders effective organization and retrieval BIBREF1 of e-books. For example, while editors mostly annotate e-books with descriptive tags that reflect the book's content, Amazon users often search for parts of the book title. In the data we use for the present study (see Section SECREF2 ), we find that around 30% of the Amazon search terms contain parts of e-book titles. In this paper, we present our work to support editors in the e-book annotation process with tag recommendations BIBREF2 , BIBREF3 . Our idea is to exploit user-generated search query terms in Amazon to mimic the vocabulary of users in Amazon, who search for e-books. We combine these search terms with tags assigned by editors in a hybrid tag recommendation approach. Thus, our aim is to show that we can improve the performance of tag recommender systems for e-books both concerning recommendation accuracy as well as semantic similarity and tag recommendation diversity. Related work. In tag recommender systems, mostly content-based algorithms (e.g., BIBREF4 , BIBREF5 ) are used to recommend tags to annotate resources such as e-books. In our work, we incorporate both content features of e-books (i.e., title and description text) as well as Amazon search terms to account for the vocabulary of e-book readers. Concerning the evaluation of tag recommendation systems, most studies focus on measuring the accuracy of tag recommendations (e.g., BIBREF2 ). However, the authors of BIBREF6 suggest also to use beyond-accuracy metrics such as diversity to evaluate the quality of tag recommendations. In our work, we measure recommendation diversity in addition to recommendation accuracy and propose a novel metric termed semantic similarity to validate semantic matches of tag recommendations. Approach and findings. We exploit editor tags and user-generated search terms as input for tag recommendation approaches. Our evaluation comprises of a rich set of 19 different algorithms to recommend tags for e-books, which we group into (i) popularity-based, (ii) similarity-based (i.e., using content information), and (iii) hybrid approaches. We evaluate our approaches in terms of accuracy, semantic similarity and diversity on the review content of Amazon users, which reflects the readers' vocabulary. With semantic similarity, we measure how semantically similar (based on learned Doc2Vec BIBREF7 embeddings) the list of recommended tags is to the list of relevant tags. We use this additional metric to measure not only exact “hits” of our recommendations but also semantic matches. Our evaluation results show that combining both data sources enhances the quality of tag recommendations for annotating e-books. Furthermore, approaches that solely train on Amazon search terms provide poor performance in terms of accuracy but deliver good results in terms of semantic similarity and recommendation diversity. Method In this section, we describe our dataset as well as our tag recommendation approaches we propose to annotate e-books. Dataset Our dataset contains two sources of data, one to generate tag recommendations and another one to evaluate tag recommendations. HGV GmbH has collected all data sources and we provide the dataset statistics in Table TABREF3 . Data used to generate recommendations. We employ two sources of e-book annotation data: (i) editor tags, and (ii) Amazon search terms. For editor tags, we collect data of 48,705 e-books from 13 publishers, namely Kunstmann, Delius-Klasnig, VUR, HJR, Diogenes, Campus, Kiwi, Beltz, Chbeck, Rowohlt, Droemer, Fischer and Neopubli. Apart from the editor tags, this data contains metadata fields of e-books such as the ISBN, the title, a description text, the author and a list of BISACs, which are identifiers for book categories. For the Amazon search terms, we collect search query logs of 21,243 e-books for 12 months (i.e., November 2017 to October 2018). Apart from the search terms, this data contains the e-books' ISBNs, titles and description texts. Table TABREF3 shows that the overlap of e-books that have editor tags and Amazon search terms is small (i.e., only 497). Furthermore, author and BISAC (i.e., the book category identifier) information are primarily available for e-books that contain editor tags. Consequently, both data sources provide complementary information, which underpins the intention of this work, i.e., to evaluate tag recommendation approaches using annotation sources from different contexts. Data used to evaluate recommendations. For evaluation, we use a third set of e-book annotations, namely Amazon review keywords. These review keywords are extracted from the Amazon review texts and are typically provided in the review section of books on Amazon. Our idea is to not favor one or the other data source (i.e., editor tags and Amazon search terms) when evaluating our approaches against expected tags. At the same time, we consider Amazon review keywords to be a good mixture of editor tags and search terms as they describe both the content and the users' opinions on the e-books (i.e., the readers' vocabulary). As shown in Table TABREF3 , we collect Amazon review keywords for 2,896 e-books (publishers: Kiwi, Rowohlt, Fischer, and Droemer), which leads to 33,663 distinct review keywords and on average 30 keyword assignments per e-book. Tag Recommendation Approaches We implement three types of tag recommendation approaches, i.e., (i) popularity-based, (ii) similarity-based (i.e., using content information), and (iii) hybrid approaches. Due to the lack of personalized tags (i.e., we do not know which user has assigned a tag), we do not implement other types of algorithms such as collaborative filtering BIBREF8 . In total, we evaluate 19 different algorithms to recommend tags for annotating e-books. Popularity-based approaches. We recommend the most frequently used tags in the dataset, which is a common strategy for tag recommendations BIBREF9 . That is, a most popular INLINEFORM0 approach for editor tags and a most popular INLINEFORM1 approach for Amazon search terms. For e-books, for which we also have author (= INLINEFORM2 and INLINEFORM3 ) or BISAC (= INLINEFORM4 and INLINEFORM5 ) information, we use these features to further filter the recommended tags, i.e., to only recommend tags that were used to annotate e-books of a specific author or a specific BISAC. We combine both data sources (i.e., editor tags and Amazon search terms) using a round-robin combination strategy, which ensures an equal weight for both sources. This gives us three additional popularity-based algorithms (= INLINEFORM0 , INLINEFORM1 and INLINEFORM2 ). Similarity-based approaches. We exploit the textual content of e-books (i.e., description or title) to recommend relevant tags BIBREF10 . For this, we first employ a content-based filtering approach BIBREF11 based on TF-IDF BIBREF12 to find top- INLINEFORM0 similar e-books. For each of the similar e-books, we then either extract the assigned editor tags (= INLINEFORM2 and INLINEFORM3 ) or the Amazon search terms (= INLINEFORM4 and INLINEFORM5 ). To combine the tags of the top- INLINEFORM6 similar e-books, we use the cross-source algorithm BIBREF13 , which favors tags that were used to annotate more than one similar e-book (i.e., tags that come from multiple recommendation sources). The final tag relevancy is calculated as: DISPLAYFORM0 where INLINEFORM0 denotes the number of distinct e-books, which yielded the recommendation of tag INLINEFORM1 , to favor tags that come from multiple sources and INLINEFORM2 is the similarity score of the corresponding e-book. We again use a round-robin strategy to combine both data sources (= INLINEFORM3 and INLINEFORM4 ). Hybrid approaches. We use the previously mentioned cross-source algorithm BIBREF13 to construct four hybrid recommendation approaches. In this case, tags are favored that are recommended by more than one algorithm. Hence, to create a popularity-based hybrid (= INLINEFORM0 ), we combine the best three performing popularity-based approaches from the ones (i) without any contextual signal, (ii) with the author as context, and (iii) with BISAC as context. In the case of the similarity-based hybrid (= INLINEFORM1 ), we utilize the two best performing similarity-based approaches from the ones (i) which use the title, and (ii) which use the description text. We further define INLINEFORM2 , a hybrid approach that combines the three popularity-based methods of INLINEFORM3 and the two similarity-based approaches of INLINEFORM4 . Finally, we define INLINEFORM5 as a hybrid approach that uses the best performing popularity-based and the best performing similarity-based approach (see Figure FIGREF11 in Section SECREF4 for more details about the particular algorithm combinations). Experimental Setup In this section, we describe our evaluation protocol as well as the measures we use to evaluate and compare our tag recommendation approaches. Evaluation Protocol For evaluation, we use the third set of e-book annotations, namely Amazon review keywords. As described in Section SECREF1 , these review keywords are extracted from the Amazon review texts and thus, reflect the users' vocabulary. We evaluate our approaches for the 2,896 e-books, for whom we got review keywords. To follow common practice for tag recommendation evaluation BIBREF14 , we predict the assigned review keywords (= our test set) for respective e-books. Evaluation Metrics In this work, we measure (i) recommendation accuracy, (ii) semantic similarity, and (iii) recommendation diversity to evaluate the quality of our approaches from different perspectives. Recommendation accuracy. We use Normalized Discounted Cumulative Gain (nDCG) BIBREF15 to measure the accuracy of the tag recommendation approaches. The nDCG measure is a standard ranking-dependent metric that not only measures how many tags can be correctly predicted but also takes into account their position in the recommendation list with length of INLINEFORM0 . It is based on the Discounted Cummulative Gain, which is given by: DISPLAYFORM0 where INLINEFORM0 is a function that returns 1 if the recommended tag at position INLINEFORM1 in the recommended list is relevant. We then calculate DCG@ INLINEFORM2 for every evaluated e-book by dividing DCG@ INLINEFORM3 with the ideal DCG value iDCG@ INLINEFORM4 , which is the highest possible DCG value that can be achieved if all the relevant tags would be recommended in the correct order. It is given by the following formula BIBREF15 : DISPLAYFORM0 Semantic similarity. One precondition of standard recommendation accuracy measures is that to generate a “hit”, the recommended tag needs to be an exact syntactical match to the one from the test set. When tags are recommended from one data source and compared to tags from another source, this can be problematic. For example, if we recommend the tag “victim” but expect the tag “prey”, we would mark this as a mismatch, therefore being a bad recommendation. But if we know that the corresponding e-book is a crime novel, the recommended tag would be (semantically) descriptive to reflect the book's content. Hence, in this paper, we propose to additionally measure the semantic similarity between recommended tags and tags from the test set (i.e., the Amazon review keywords). Over the last four years, there have been several notable publications in the area of applying deep learning to uncover semantic relationships between textual content (e.g., by learning word embeddings with Word2Vec BIBREF16 , BIBREF17 ). Based on this, we propose an alternative measure of recommendation quality by learning the semantic relationships from both vocabularies and then using it to compare how semantically similar the recommended tags are to the expected review keywords. For this, we first extract the textual content in the form of the description text, title, editor tags and Amazon search terms of e-books from our dataset. We then train a Doc2Vec BIBREF7 model on the content. Then, we use the model to infer the latent representation for both the complete list of recommended tags as well as the list of expected tags from the test set. Finally, we use the cosine similarity measure to calculate how semantically similar these two lists are. Recommendation diversity. As defined in BIBREF18 , we calculate recommendation diversity as the average dissimilarity of all pairs of tags in the list of recommended tags. Thus, given a distance function INLINEFORM0 that corresponds to the dissimilarity between two tags INLINEFORM1 and INLINEFORM2 in the list of recommended tags, INLINEFORM3 is given as the average dissimilarity of all pairs of tags: DISPLAYFORM0 where INLINEFORM0 is the number of evaluated e-books and the dissimilarity function is defined as INLINEFORM1 . In our experiments, we use the previously trained Doc2Vec model to extract the latent representation of a specific tag. The similarity of two tags INLINEFORM2 is then calculated with the Cosine similarity measure using the latent vector representations of respective tags INLINEFORM3 and INLINEFORM4 . Results Concerning tag recommendation accuracy, in this section, we report results for different values of INLINEFORM0 (i.e., number of recommended tags). For the beyond-accuracy experiment, we use the full list of recommended tags (i.e., INLINEFORM1 ). Recommendation Accuracy Evaluation Figure FIGREF11 shows the results of the accuracy experiment for the (i) popularity-based, (ii) similarity-based, and (iii) hybrid tag recommendation approaches. Popularity-based approaches. In Figure FIGREF11 , we see that popularity-based approaches based on editor tags tend to perform better than if trained on Amazon search terms. If we take into account contextual information like BISAC or author, we can further improve accuracy in terms of INLINEFORM0 . That is, we find that using popular tags from e-books of a specific author leads to the best accuracy of the popularity-based approaches. This suggests that editors and readers do seem to reuse tags for e-books of same authors. If we use both editor tags and Amazon search terms, we can further increase accuracy, especially for higher values of INLINEFORM1 like in the case of INLINEFORM2 . This is, however, not the case for INLINEFORM3 as the accuracy of the integrated INLINEFORM4 approach is low. The reason for this is the limited amount of e-books from within the Amazon search query logs that have BISAC information (i.e., only INLINEFORM5 ). Similarity-based approaches. We further improve accuracy if we first find similar e-books and then extract their top- INLINEFORM0 tags in a cross-source manner as described in Section SECREF4 . As shown in Figure FIGREF11 , using the description text to find similar e-books results in more accurate tag recommendations than using the title (i.e., INLINEFORM0 for INLINEFORM1 ). This is somehow expected as the description text consists of a bigger corpus of words (i.e., multiple sentences) than the title. Concerning the collected Amazon search query logs, extracting and then recommending tags from this source results in a much lower accuracy performance. Thus, these results also suggest to investigate beyond-accuracy metrics as done in Section SECREF17 . Hybrid approaches. Figure FIGREF11 shows the accuracy results of the four hybrid approaches. By combining the best three popularity-based approaches, we outperform all of the initially evaluated popularity algorithms (i.e., INLINEFORM0 for INLINEFORM1 ). On the contrary, the combination of the two best performing similarity-based approaches INLINEFORM2 and INLINEFORM3 does not yield better accuracy. The negative impact of using a lower-performing approach such as INLINEFORM4 within a hybrid combination can also be observed in INLINEFORM5 for lower values of INLINEFORM6 . Overall, this confirms our initial intuition that combining the best performing popularity-based approach with the best similarity-based approach should result in the highest accuracy (i.e., INLINEFORM7 for INLINEFORM8 ). Moreover, our goal, namely to exploit editor tags in combination with search terms used by readers to increase the metadata quality of e-books, is shown to be best supported by applying hybrid approaches as they provide the best prediction results. Beyond-Accuracy Evaluation Figure FIGREF16 illustrates the results of the experiments, which measure the recommendation impact beyond-accuracy. Semantic similarity. Figure FIGREF16 illustrates the results of our proposed semantic similarity measure. To compare our proposed measure to standard accuracy measures such as INLINEFORM0 , we use Kendall's Tau rank correlation BIBREF19 as suggested by BIBREF20 for automatic evaluation of information-ordering tasks. From that, we rank our recommendation approaches according to both accuracy and semantic similarity and calculate the relation between both rankings. This results in INLINEFORM1 with a p-value < INLINEFORM2 , which suggests a high correlation between the semantic similarity and the standard accuracy measure. Therefore, the semantic similarity measure helps us interpret the recommendation quality. For instance, we achieve the lowest INLINEFORM0 values with the similarity-based approaches that recommend Amazon search terms (i.e., INLINEFORM1 and INLINEFORM2 ). When comparing these results with others from Figure FIGREF11 , a conclusion could be quickly drawn that the recommended tags are merely unusable. However, by looking at Figure FIGREF16 , we see that, although these approaches do not provide the highest recommendation accuracy, they still result in tag recommendations that are semantically related at a high degree to the expected annotations from the test set. Overall, this suggests that approaches, which provide a poor accuracy performance concerning INLINEFORM4 but provide a good performance regarding semantic similarity could still be helpful for annotating e-books. Recommendation diversity. Figure FIGREF16 shows the diversity of the tag recommendation approaches. We achieve the highest diversity with the similarity-based approaches, which extract Amazon search terms. Their accuracy is, however, very low. Thus, the combination of the two vocabularies can provide a good trade-off between recommendation accuracy and diversity. Conclusion and Future Work In this paper, we present our work to support editors in the e-book annotation process. Specifically, we aim to provide tag recommendations that incorporate both the vocabulary of the editors and e-book readers. Therefore, we train various configurations of tag recommender approaches on editors' tags and Amazon search terms and evaluate them on a dataset containing Amazon review keywords. We find that combining both data sources enhances the quality of tag recommendations for annotating e-books. Furthermore, while approaches that train only on Amazon search terms provide poor performance concerning recommendation accuracy, we show that they still offer helpful annotations concerning recommendation diversity as well as our novel semantic similarity metric. Future work. For future work, we plan to validate our findings using another dataset, e.g., by recommending tags for scientific articles and books in BibSonomy. With this, we aim to demonstrate the usefulness of the proposed approach in a similar domain and to enhance the reproducibility of our results by using an open dataset. Moreover, we plan to evaluate our tag recommendation approaches in a study with domain users. Also, we want to improve our similarity-based approaches by integrating novel embedding approaches BIBREF16 , BIBREF17 as we did, for example, with our proposed semantic similarity evaluation metric. Finally, we aim to incorporate explanations for recommended tags so that editors of e-book annotations receive additional support in annotating e-books BIBREF21 . By making the underlying (semantic) reasoning visible to the editor who is in charge of tailoring annotations, we aim to support two goals: (i) allowing readers to discover e-books more efficiently, and (ii) enabling publishers to leverage semi-automatic categorization processes for e-books. In turn, providing explanations fosters control over which vocabulary to choose when tagging e-books for different application contexts. Acknowledgments. The authors would like to thank Peter Langs, Jan-Philipp Wolf and Alyona Schraa from HGV GmbH for providing the e-book annotation data. This work was funded by the Know-Center GmbH (FFG COMET Program), the FFG Data Market Austria project and the AI4EU project (EU grant 825619). The Know-Center GmbH is funded within the Austrian COMET Program - Competence Centers for Excellent Technologies - under the auspices of the Austrian Ministry of Transport, Innovation and Technology, the Austrian Ministry of Economics and Labor and by the State of Styria. COMET is managed by the Austrian Research Promotion Agency (FFG).
how large is the vocabulary?
b4f881331b975e6e4cab1868267211ed729d782d
how Ġlarge Ġis Ġthe Ġvocabulary ?
33 , 663
<s> how Ġlarge Ġis Ġthe Ġvocabulary ? </s> Introduction </s> When Ġpeople Ġshop Ġfor Ġbooks Ġonline Ġin Ġe - book Ġstores Ġsuch Ġas , Ġe . g ., Ġthe ĠAmazon ĠKindle Ġstore , Ġthey Ġenter Ġsearch Ġterms Ġwith Ġthe Ġgoal Ġto Ġfind Ġe - books Ġthat Ġmeet Ġtheir Ġpreferences . ĠSuch Ġe - books Ġhave Ġa Ġvariety Ġof Ġmetadata Ġsuch Ġas , Ġe . g ., Ġtitle , Ġauthor Ġor Ġkeywords , Ġwhich Ġcan Ġbe Ġused Ġto Ġretrieve Ġe - books Ġthat Ġare Ġrelevant Ġto Ġthe Ġquery . ĠAs Ġa Ġconsequence , Ġfrom Ġthe Ġperspective Ġof Ġe - book Ġpublishers Ġand Ġeditors , Ġannot ating Ġe - books Ġwith Ġtags Ġthat Ġbest Ġdescribe Ġthe Ġcontent Ġand Ġwhich Ġmeet Ġthe Ġvocabulary Ġof Ġusers Ġ( e . g ., Ġwhen Ġsearching Ġand Ġreviewing Ġe - books ) Ġis Ġan Ġessential Ġtask ĠB IB REF 0 Ġ. </s> Problem Ġand Ġaim Ġof Ġthis Ġwork . ĠAn not ating Ġe - books Ġwith Ġsuitable Ġtags Ġis , Ġhowever , Ġa Ġcomplex Ġtask Ġas Ġusers ' Ġvocabulary Ġmay Ġdiffer Ġfrom Ġthe Ġone Ġof Ġeditors . ĠSuch Ġa Ġvocabulary Ġmismatch Ġyet Ġhind ers Ġeffective Ġorganization Ġand Ġretrieval ĠB IB REF 1 Ġof Ġe - books . ĠFor Ġexample , Ġwhile Ġeditors Ġmostly Ġannot ate Ġe - books Ġwith Ġdescriptive Ġtags Ġthat Ġreflect Ġthe Ġbook 's Ġcontent , ĠAmazon Ġusers Ġoften Ġsearch Ġfor Ġparts Ġof Ġthe Ġbook Ġtitle . ĠIn Ġthe Ġdata Ġwe Ġuse Ġfor Ġthe Ġpresent Ġstudy Ġ( see ĠSection ĠSEC REF 2 Ġ), Ġwe Ġfind Ġthat Ġaround Ġ30 % Ġof Ġthe ĠAmazon Ġsearch Ġterms Ġcontain Ġparts Ġof Ġe - book Ġtitles . </s> In Ġthis Ġpaper , Ġwe Ġpresent Ġour Ġwork Ġto Ġsupport Ġeditors Ġin Ġthe Ġe - book Ġannotation Ġprocess Ġwith Ġtag Ġrecommendations ĠB IB REF 2 Ġ, ĠB IB REF 3 Ġ. ĠOur Ġidea Ġis Ġto Ġexploit Ġuser - generated Ġsearch Ġquery Ġterms Ġin ĠAmazon Ġto Ġmimic Ġthe Ġvocabulary Ġof Ġusers Ġin ĠAmazon , Ġwho Ġsearch Ġfor Ġe - books . ĠWe Ġcombine Ġthese Ġsearch Ġterms Ġwith Ġtags Ġassigned Ġby Ġeditors Ġin Ġa Ġhybrid Ġtag Ġrecommendation Ġapproach . ĠThus , Ġour Ġaim Ġis Ġto Ġshow Ġthat Ġwe Ġcan Ġimprove Ġthe Ġperformance Ġof Ġtag Ġrecomm ender Ġsystems Ġfor Ġe - books Ġboth Ġconcerning Ġrecommendation Ġaccuracy Ġas Ġwell Ġas Ġsemantic Ġsimilarity Ġand Ġtag Ġrecommendation Ġdiversity . </s> Related Ġwork . ĠIn Ġtag Ġrecomm ender Ġsystems , Ġmostly Ġcontent - based Ġalgorithms Ġ( e . g ., ĠB IB REF 4 Ġ, ĠB IB REF 5 Ġ) Ġare Ġused Ġto Ġrecommend Ġtags Ġto Ġannot ate Ġresources Ġsuch Ġas Ġe - books . ĠIn Ġour Ġwork , Ġwe Ġincorporate Ġboth Ġcontent Ġfeatures Ġof Ġe - books Ġ( i . e ., Ġtitle Ġand Ġdescription Ġtext ) Ġas Ġwell Ġas ĠAmazon Ġsearch Ġterms Ġto Ġaccount Ġfor Ġthe Ġvocabulary Ġof Ġe - book Ġreaders . </s> Con cerning Ġthe Ġevaluation Ġof Ġtag Ġrecommendation Ġsystems , Ġmost Ġstudies Ġfocus Ġon Ġmeasuring Ġthe Ġaccuracy Ġof Ġtag Ġrecommendations Ġ( e . g ., ĠB IB REF 2 Ġ). ĠHowever , Ġthe Ġauthors Ġof ĠB IB REF 6 Ġsuggest Ġalso Ġto Ġuse Ġbeyond - acc uracy Ġmetrics Ġsuch Ġas Ġdiversity Ġto Ġevaluate Ġthe Ġquality Ġof Ġtag Ġrecommendations . ĠIn Ġour Ġwork , Ġwe Ġmeasure Ġrecommendation Ġdiversity Ġin Ġaddition Ġto Ġrecommendation Ġaccuracy Ġand Ġpropose Ġa Ġnovel Ġmetric Ġtermed Ġsemantic Ġsimilarity Ġto Ġvalidate Ġsemantic Ġmatches Ġof Ġtag Ġrecommendations . </s> App roach Ġand Ġfindings . ĠWe Ġexploit Ġeditor Ġtags Ġand Ġuser - generated Ġsearch Ġterms Ġas Ġinput Ġfor Ġtag Ġrecommendation Ġapproaches . ĠOur Ġevaluation Ġcomprises Ġof Ġa Ġrich Ġset Ġof Ġ19 Ġdifferent Ġalgorithms Ġto Ġrecommend Ġtags Ġfor Ġe - books , Ġwhich Ġwe Ġgroup Ġinto Ġ( i ) Ġpopularity - based , Ġ( ii ) Ġsimilarity - based Ġ( i . e ., Ġusing Ġcontent Ġinformation ), Ġand Ġ( iii ) Ġhybrid Ġapproaches . ĠWe Ġevaluate Ġour Ġapproaches Ġin Ġterms Ġof Ġaccuracy , Ġsemantic Ġsimilarity Ġand Ġdiversity Ġon Ġthe Ġreview Ġcontent Ġof ĠAmazon Ġusers , Ġwhich Ġreflects Ġthe Ġreaders ' Ġvocabulary . ĠWith Ġsemantic Ġsimilarity , Ġwe Ġmeasure Ġhow Ġsem antically Ġsimilar Ġ( based Ġon Ġlearned ĠDoc 2 V ec ĠB IB REF 7 Ġembed d ings ) Ġthe Ġlist Ġof Ġrecommended Ġtags Ġis Ġto Ġthe Ġlist Ġof Ġrelevant Ġtags . ĠWe Ġuse Ġthis Ġadditional Ġmetric Ġto Ġmeasure Ġnot Ġonly Ġexact ĠâĢ ľ h its âĢ Ŀ Ġof Ġour Ġrecommendations Ġbut Ġalso Ġsemantic Ġmatches . </s> Our Ġevaluation Ġresults Ġshow Ġthat Ġcombining Ġboth Ġdata Ġsources Ġenhances Ġthe Ġquality Ġof Ġtag Ġrecommendations Ġfor Ġannot ating Ġe - books . ĠFurthermore , Ġapproaches Ġthat Ġsolely Ġtrain Ġon ĠAmazon Ġsearch Ġterms Ġprovide Ġpoor Ġperformance Ġin Ġterms Ġof Ġaccuracy Ġbut Ġdeliver Ġgood Ġresults Ġin Ġterms Ġof Ġsemantic Ġsimilarity Ġand Ġrecommendation Ġdiversity . </s> Method </s> In Ġthis Ġsection , Ġwe Ġdescribe Ġour Ġdataset Ġas Ġwell Ġas Ġour Ġtag Ġrecommendation Ġapproaches Ġwe Ġpropose Ġto Ġannot ate Ġe - books . </s> Dat as et </s> Our Ġdataset Ġcontains Ġtwo Ġsources Ġof Ġdata , Ġone Ġto Ġgenerate Ġtag Ġrecommendations Ġand Ġanother Ġone Ġto Ġevaluate Ġtag Ġrecommendations . ĠH GV ĠG mb H Ġhas Ġcollected Ġall Ġdata Ġsources Ġand Ġwe Ġprovide Ġthe Ġdataset Ġstatistics Ġin ĠTable ĠT AB REF 3 Ġ. </s> Data Ġused Ġto Ġgenerate Ġrecommendations . ĠWe Ġemploy Ġtwo Ġsources Ġof Ġe - book Ġannotation Ġdata : Ġ( i ) Ġeditor Ġtags , Ġand Ġ( ii ) ĠAmazon Ġsearch Ġterms . ĠFor Ġeditor Ġtags , Ġwe Ġcollect Ġdata Ġof Ġ48 , 705 Ġe - books Ġfrom Ġ13 Ġpublishers , Ġnamely ĠKun st mann , ĠDel ius - K las n ig , ĠV UR , ĠH JR , ĠDi ogen es , ĠCampus , ĠKiw i , ĠBelt z , ĠCh beck , ĠRow oh lt , ĠDro emer , ĠFischer Ġand ĠNe op ub li . ĠApart Ġfrom Ġthe Ġeditor Ġtags , Ġthis Ġdata Ġcontains Ġmetadata Ġfields Ġof Ġe - books Ġsuch Ġas Ġthe ĠISBN , Ġthe Ġtitle , Ġa Ġdescription Ġtext , Ġthe Ġauthor Ġand Ġa Ġlist Ġof ĠB IS AC s , Ġwhich Ġare Ġidentifiers Ġfor Ġbook Ġcategories . </s> For Ġthe ĠAmazon Ġsearch Ġterms , Ġwe Ġcollect Ġsearch Ġquery Ġlogs Ġof Ġ21 , 243 Ġe - books Ġfor Ġ12 Ġmonths Ġ( i . e ., ĠNovember Ġ2017 Ġto ĠOctober Ġ2018 ). ĠApart Ġfrom Ġthe Ġsearch Ġterms , Ġthis Ġdata Ġcontains Ġthe Ġe - books ' ĠISBN s , Ġtitles Ġand Ġdescription Ġtexts . </s> Table ĠT AB REF 3 Ġshows Ġthat Ġthe Ġoverlap Ġof Ġe - books Ġthat Ġhave Ġeditor Ġtags Ġand ĠAmazon Ġsearch Ġterms Ġis Ġsmall Ġ( i . e ., Ġonly Ġ49 7 ). ĠFurthermore , Ġauthor Ġand ĠB IS AC Ġ( i . e ., Ġthe Ġbook Ġcategory Ġidentifier ) Ġinformation Ġare Ġprimarily Ġavailable Ġfor Ġe - books Ġthat Ġcontain Ġeditor Ġtags . ĠConsequently , Ġboth Ġdata Ġsources Ġprovide Ġcomplementary Ġinformation , Ġwhich Ġunder pins Ġthe Ġintention Ġof Ġthis Ġwork , Ġi . e ., Ġto Ġevaluate Ġtag Ġrecommendation Ġapproaches Ġusing Ġannotation Ġsources Ġfrom Ġdifferent Ġcontexts . </s> Data Ġused Ġto Ġevaluate Ġrecommendations . ĠFor Ġevaluation , Ġwe Ġuse Ġa Ġthird Ġset Ġof Ġe - book Ġannotations , Ġnamely ĠAmazon Ġreview Ġkeywords . ĠThese Ġreview Ġkeywords Ġare Ġextracted Ġfrom Ġthe ĠAmazon Ġreview Ġtexts Ġand Ġare Ġtypically Ġprovided Ġin Ġthe Ġreview Ġsection Ġof Ġbooks Ġon ĠAmazon . ĠOur Ġidea Ġis Ġto Ġnot Ġfavor Ġone Ġor Ġthe Ġother Ġdata Ġsource Ġ( i . e ., Ġeditor Ġtags Ġand ĠAmazon Ġsearch Ġterms ) Ġwhen Ġevaluating Ġour Ġapproaches Ġagainst Ġexpected Ġtags . ĠAt Ġthe Ġsame Ġtime , Ġwe Ġconsider ĠAmazon Ġreview Ġkeywords Ġto Ġbe Ġa Ġgood Ġmixture Ġof Ġeditor Ġtags Ġand Ġsearch Ġterms Ġas Ġthey Ġdescribe Ġboth Ġthe Ġcontent Ġand Ġthe Ġusers ' Ġopinions Ġon Ġthe Ġe - books Ġ( i . e ., Ġthe Ġreaders ' Ġvocabulary ). ĠAs Ġshown Ġin ĠTable ĠT AB REF 3 Ġ, Ġwe Ġcollect ĠAmazon Ġreview Ġkeywords Ġfor Ġ2 , 896 Ġe - books Ġ( pub lisher s : ĠKiw i , ĠRow oh lt , ĠFischer , Ġand ĠDro emer ), Ġwhich Ġleads Ġto Ġ33 , 663 Ġdistinct Ġreview Ġkeywords Ġand Ġon Ġaverage Ġ30 Ġkeyword Ġassignments Ġper Ġe - book . </s> Tag ĠRecommend ation ĠAppro aches </s> We Ġimplement Ġthree Ġtypes Ġof Ġtag Ġrecommendation Ġapproaches , Ġi . e ., Ġ( i ) Ġpopularity - based , Ġ( ii ) Ġsimilarity - based Ġ( i . e ., Ġusing Ġcontent Ġinformation ), Ġand Ġ( iii ) Ġhybrid Ġapproaches . ĠDue Ġto Ġthe Ġlack Ġof Ġpersonalized Ġtags Ġ( i . e ., Ġwe Ġdo Ġnot Ġknow Ġwhich Ġuser Ġhas Ġassigned Ġa Ġtag ), Ġwe Ġdo Ġnot Ġimplement Ġother Ġtypes Ġof Ġalgorithms Ġsuch Ġas Ġcollaborative Ġfiltering ĠB IB REF 8 Ġ. ĠIn Ġtotal , Ġwe Ġevaluate Ġ19 Ġdifferent Ġalgorithms Ġto Ġrecommend Ġtags Ġfor Ġannot ating Ġe - books . </s> Pop ularity - based Ġapproaches . ĠWe Ġrecommend Ġthe Ġmost Ġfrequently Ġused Ġtags Ġin Ġthe Ġdataset , Ġwhich Ġis Ġa Ġcommon Ġstrategy Ġfor Ġtag Ġrecommendations ĠB IB REF 9 Ġ. ĠThat Ġis , Ġa Ġmost Ġpopular ĠIN LINE FORM 0 Ġapproach Ġfor Ġeditor Ġtags Ġand Ġa Ġmost Ġpopular ĠIN LINE FORM 1 Ġapproach Ġfor ĠAmazon Ġsearch Ġterms . ĠFor Ġe - books , Ġfor Ġwhich Ġwe Ġalso Ġhave Ġauthor Ġ(= ĠIN LINE FORM 2 Ġand ĠIN LINE FORM 3 Ġ) Ġor ĠB IS AC Ġ(= ĠIN LINE FORM 4 Ġand ĠIN LINE FORM 5 Ġ) Ġinformation , Ġwe Ġuse Ġthese Ġfeatures Ġto Ġfurther Ġfilter Ġthe Ġrecommended Ġtags , Ġi . e ., Ġto Ġonly Ġrecommend Ġtags Ġthat Ġwere Ġused Ġto Ġannot ate Ġe - books Ġof Ġa Ġspecific Ġauthor Ġor Ġa Ġspecific ĠB IS AC . </s> We Ġcombine Ġboth Ġdata Ġsources Ġ( i . e ., Ġeditor Ġtags Ġand ĠAmazon Ġsearch Ġterms ) Ġusing Ġa Ġround - ro bin Ġcombination Ġstrategy , Ġwhich Ġensures Ġan Ġequal Ġweight Ġfor Ġboth Ġsources . ĠThis Ġgives Ġus Ġthree Ġadditional Ġpopularity - based Ġalgorithms Ġ(= ĠIN LINE FORM 0 Ġ, ĠIN LINE FORM 1 Ġand ĠIN LINE FORM 2 Ġ). </s> Similar ity - based Ġapproaches . ĠWe Ġexploit Ġthe Ġtextual Ġcontent Ġof Ġe - books Ġ( i . e ., Ġdescription Ġor Ġtitle ) Ġto Ġrecommend Ġrelevant Ġtags ĠB IB REF 10 Ġ. ĠFor Ġthis , Ġwe Ġfirst Ġemploy Ġa Ġcontent - based Ġfiltering Ġapproach ĠB IB REF 11 Ġbased Ġon ĠTF - ID F ĠB IB REF 12 Ġto Ġfind Ġtop - ĠIN LINE FORM 0 Ġsimilar Ġe - books . ĠFor Ġeach Ġof Ġthe Ġsimilar Ġe - books , Ġwe Ġthen Ġeither Ġextract Ġthe Ġassigned Ġeditor Ġtags Ġ(= ĠIN LINE FORM 2 Ġand ĠIN LINE FORM 3 Ġ) Ġor Ġthe ĠAmazon Ġsearch Ġterms Ġ(= ĠIN LINE FORM 4 Ġand ĠIN LINE FORM 5 Ġ). ĠTo Ġcombine Ġthe Ġtags Ġof Ġthe Ġtop - ĠIN LINE FORM 6 Ġsimilar Ġe - books , Ġwe Ġuse Ġthe Ġcross - source Ġalgorithm ĠB IB REF 13 Ġ, Ġwhich Ġfavors Ġtags Ġthat Ġwere Ġused Ġto Ġannot ate Ġmore Ġthan Ġone Ġsimilar Ġe - book Ġ( i . e ., Ġtags Ġthat Ġcome Ġfrom Ġmultiple Ġrecommendation Ġsources ). ĠThe Ġfinal Ġtag Ġrele v ancy Ġis Ġcalculated Ġas : ĠDIS PLAY FORM 0 </s> where ĠIN LINE FORM 0 Ġdenotes Ġthe Ġnumber Ġof Ġdistinct Ġe - books , Ġwhich Ġyielded Ġthe Ġrecommendation Ġof Ġtag ĠIN LINE FORM 1 Ġ, Ġto Ġfavor Ġtags Ġthat Ġcome Ġfrom Ġmultiple Ġsources Ġand ĠIN LINE FORM 2 Ġis Ġthe Ġsimilarity Ġscore Ġof Ġthe Ġcorresponding Ġe - book . ĠWe Ġagain Ġuse Ġa Ġround - ro bin Ġstrategy Ġto Ġcombine Ġboth Ġdata Ġsources Ġ(= ĠIN LINE FORM 3 Ġand ĠIN LINE FORM 4 Ġ). </s> Hy brid Ġapproaches . ĠWe Ġuse Ġthe Ġpreviously Ġmentioned Ġcross - source Ġalgorithm ĠB IB REF 13 Ġto Ġconstruct Ġfour Ġhybrid Ġrecommendation Ġapproaches . ĠIn Ġthis Ġcase , Ġtags Ġare Ġfavored Ġthat Ġare Ġrecommended Ġby Ġmore Ġthan Ġone Ġalgorithm . </s> H ence , Ġto Ġcreate Ġa Ġpopularity - based Ġhybrid Ġ(= ĠIN LINE FORM 0 Ġ), Ġwe Ġcombine Ġthe Ġbest Ġthree Ġperforming Ġpopularity - based Ġapproaches Ġfrom Ġthe Ġones Ġ( i ) Ġwithout Ġany Ġcontextual Ġsignal , Ġ( ii ) Ġwith Ġthe Ġauthor Ġas Ġcontext , Ġand Ġ( iii ) Ġwith ĠB IS AC Ġas Ġcontext . ĠIn Ġthe Ġcase Ġof Ġthe Ġsimilarity - based Ġhybrid Ġ(= ĠIN LINE FORM 1 Ġ), Ġwe Ġutilize Ġthe Ġtwo Ġbest Ġperforming Ġsimilarity - based Ġapproaches Ġfrom Ġthe Ġones Ġ( i ) Ġwhich Ġuse Ġthe Ġtitle , Ġand Ġ( ii ) Ġwhich Ġuse Ġthe Ġdescription Ġtext . ĠWe Ġfurther Ġdefine ĠIN LINE FORM 2 Ġ, Ġa Ġhybrid Ġapproach Ġthat Ġcombines Ġthe Ġthree Ġpopularity - based Ġmethods Ġof ĠIN LINE FORM 3 Ġand Ġthe Ġtwo Ġsimilarity - based Ġapproaches Ġof ĠIN LINE FORM 4 Ġ. ĠFinally , Ġwe Ġdefine ĠIN LINE FORM 5 Ġas Ġa Ġhybrid Ġapproach Ġthat Ġuses Ġthe Ġbest Ġperforming Ġpopularity - based Ġand Ġthe Ġbest Ġperforming Ġsimilarity - based Ġapproach Ġ( see ĠFigure ĠFIG REF 11 Ġin ĠSection ĠSEC REF 4 Ġfor Ġmore Ġdetails Ġabout Ġthe Ġparticular Ġalgorithm Ġcombinations ). </s> Exper imental ĠSetup </s> In Ġthis Ġsection , Ġwe Ġdescribe Ġour Ġevaluation Ġprotocol Ġas Ġwell Ġas Ġthe Ġmeasures Ġwe Ġuse Ġto Ġevaluate Ġand Ġcompare Ġour Ġtag Ġrecommendation Ġapproaches . </s> E val uation ĠProtocol </s> For Ġevaluation , Ġwe Ġuse Ġthe Ġthird Ġset Ġof Ġe - book Ġannotations , Ġnamely ĠAmazon Ġreview Ġkeywords . ĠAs Ġdescribed Ġin ĠSection ĠSEC REF 1 Ġ, Ġthese Ġreview Ġkeywords Ġare Ġextracted Ġfrom Ġthe ĠAmazon Ġreview Ġtexts Ġand Ġthus , Ġreflect Ġthe Ġusers ' Ġvocabulary . ĠWe Ġevaluate Ġour Ġapproaches Ġfor Ġthe Ġ2 , 896 Ġe - books , Ġfor Ġwhom Ġwe Ġgot Ġreview Ġkeywords . ĠTo Ġfollow Ġcommon Ġpractice Ġfor Ġtag Ġrecommendation Ġevaluation ĠB IB REF 14 Ġ, Ġwe Ġpredict Ġthe Ġassigned Ġreview Ġkeywords Ġ(= Ġour Ġtest Ġset ) Ġfor Ġrespective Ġe - books . </s> E val uation ĠMet rics </s> In Ġthis Ġwork , Ġwe Ġmeasure Ġ( i ) Ġrecommendation Ġaccuracy , Ġ( ii ) Ġsemantic Ġsimilarity , Ġand Ġ( iii ) Ġrecommendation Ġdiversity Ġto Ġevaluate Ġthe Ġquality Ġof Ġour Ġapproaches Ġfrom Ġdifferent Ġperspectives . </s> Recommend ation Ġaccuracy . ĠWe Ġuse ĠNormal ized ĠDiscount ed ĠCum ulative ĠGain Ġ( n DC G ) ĠB IB REF 15 Ġto Ġmeasure Ġthe Ġaccuracy Ġof Ġthe Ġtag Ġrecommendation Ġapproaches . ĠThe Ġn DC G Ġmeasure Ġis Ġa Ġstandard Ġranking - dependent Ġmetric Ġthat Ġnot Ġonly Ġmeasures Ġhow Ġmany Ġtags Ġcan Ġbe Ġcorrectly Ġpredicted Ġbut Ġalso Ġtakes Ġinto Ġaccount Ġtheir Ġposition Ġin Ġthe Ġrecommendation Ġlist Ġwith Ġlength Ġof ĠIN LINE FORM 0 Ġ. ĠIt Ġis Ġbased Ġon Ġthe ĠDiscount ed ĠCumm ulative ĠGain , Ġwhich Ġis Ġgiven Ġby : ĠDIS PLAY FORM 0 </s> where ĠIN LINE FORM 0 Ġis Ġa Ġfunction Ġthat Ġreturns Ġ1 Ġif Ġthe Ġrecommended Ġtag Ġat Ġposition ĠIN LINE FORM 1 Ġin Ġthe Ġrecommended Ġlist Ġis Ġrelevant . ĠWe Ġthen Ġcalculate ĠDC G @ ĠIN LINE FORM 2 Ġfor Ġevery Ġevaluated Ġe - book Ġby Ġdividing ĠDC G @ ĠIN LINE FORM 3 Ġwith Ġthe Ġideal ĠDC G Ġvalue Ġi DC G @ ĠIN LINE FORM 4 Ġ, Ġwhich Ġis Ġthe Ġhighest Ġpossible ĠDC G Ġvalue Ġthat Ġcan Ġbe Ġachieved Ġif Ġall Ġthe Ġrelevant Ġtags Ġwould Ġbe Ġrecommended Ġin Ġthe Ġcorrect Ġorder . ĠIt Ġis Ġgiven Ġby Ġthe Ġfollowing Ġformula ĠB IB REF 15 Ġ: ĠDIS PLAY FORM 0 </s> Sem antic Ġsimilarity . ĠOne Ġprec ond ition Ġof Ġstandard Ġrecommendation Ġaccuracy Ġmeasures Ġis Ġthat Ġto Ġgenerate Ġa ĠâĢ ľ hit âĢ Ŀ , Ġthe Ġrecommended Ġtag Ġneeds Ġto Ġbe Ġan Ġexact Ġsynt act ical Ġmatch Ġto Ġthe Ġone Ġfrom Ġthe Ġtest Ġset . ĠWhen Ġtags Ġare Ġrecommended Ġfrom Ġone Ġdata Ġsource Ġand Ġcompared Ġto Ġtags Ġfrom Ġanother Ġsource , Ġthis Ġcan Ġbe Ġproblematic . ĠFor Ġexample , Ġif Ġwe Ġrecommend Ġthe Ġtag ĠâĢ ľ vict im âĢ Ŀ Ġbut Ġexpect Ġthe Ġtag ĠâĢ ľ pre y âĢ Ŀ , Ġwe Ġwould Ġmark Ġthis Ġas Ġa Ġmismatch , Ġtherefore Ġbeing Ġa Ġbad Ġrecommendation . ĠBut Ġif Ġwe Ġknow Ġthat Ġthe Ġcorresponding Ġe - book Ġis Ġa Ġcrime Ġnovel , Ġthe Ġrecommended Ġtag Ġwould Ġbe Ġ( sem antically ) Ġdescriptive Ġto Ġreflect Ġthe Ġbook 's Ġcontent . ĠHence , Ġin Ġthis Ġpaper , Ġwe Ġpropose Ġto Ġadditionally Ġmeasure Ġthe Ġsemantic Ġsimilarity Ġbetween Ġrecommended Ġtags Ġand Ġtags Ġfrom Ġthe Ġtest Ġset Ġ( i . e ., Ġthe ĠAmazon Ġreview Ġkeywords ). </s> Over Ġthe Ġlast Ġfour Ġyears , Ġthere Ġhave Ġbeen Ġseveral Ġnotable Ġpublications Ġin Ġthe Ġarea Ġof Ġapplying Ġdeep Ġlearning Ġto Ġuncover Ġsemantic Ġrelationships Ġbetween Ġtextual Ġcontent Ġ( e . g ., Ġby Ġlearning Ġword Ġembed d ings Ġwith ĠWord 2 V ec ĠB IB REF 16 Ġ, ĠB IB REF 17 Ġ). ĠBased Ġon Ġthis , Ġwe Ġpropose Ġan Ġalternative Ġmeasure Ġof Ġrecommendation Ġquality Ġby Ġlearning Ġthe Ġsemantic Ġrelationships Ġfrom Ġboth Ġvoc ab ular ies Ġand Ġthen Ġusing Ġit Ġto Ġcompare Ġhow Ġsem antically Ġsimilar Ġthe Ġrecommended Ġtags Ġare Ġto Ġthe Ġexpected Ġreview Ġkeywords . ĠFor Ġthis , Ġwe Ġfirst Ġextract Ġthe Ġtextual Ġcontent Ġin Ġthe Ġform Ġof Ġthe Ġdescription Ġtext , Ġtitle , Ġeditor Ġtags Ġand ĠAmazon Ġsearch Ġterms Ġof Ġe - books Ġfrom Ġour Ġdataset . ĠWe Ġthen Ġtrain Ġa ĠDoc 2 V ec ĠB IB REF 7 Ġmodel Ġon Ġthe Ġcontent . ĠThen , Ġwe Ġuse Ġthe Ġmodel Ġto Ġinfer Ġthe Ġlatent Ġrepresentation Ġfor Ġboth Ġthe Ġcomplete Ġlist Ġof Ġrecommended Ġtags Ġas Ġwell Ġas Ġthe Ġlist Ġof Ġexpected Ġtags Ġfrom Ġthe Ġtest Ġset . ĠFinally , Ġwe Ġuse Ġthe Ġcos ine Ġsimilarity Ġmeasure Ġto Ġcalculate Ġhow Ġsem antically Ġsimilar Ġthese Ġtwo Ġlists Ġare . </s> Recommend ation Ġdiversity . ĠAs Ġdefined Ġin ĠB IB REF 18 Ġ, Ġwe Ġcalculate Ġrecommendation Ġdiversity Ġas Ġthe Ġaverage Ġdiss imilar ity Ġof Ġall Ġpairs Ġof Ġtags Ġin Ġthe Ġlist Ġof Ġrecommended Ġtags . ĠThus , Ġgiven Ġa Ġdistance Ġfunction ĠIN LINE FORM 0 Ġthat Ġcorresponds Ġto Ġthe Ġdiss imilar ity Ġbetween Ġtwo Ġtags ĠIN LINE FORM 1 Ġand ĠIN LINE FORM 2 Ġin Ġthe Ġlist Ġof Ġrecommended Ġtags , ĠIN LINE FORM 3 Ġis Ġgiven Ġas Ġthe Ġaverage Ġdiss imilar ity Ġof Ġall Ġpairs Ġof Ġtags : ĠDIS PLAY FORM 0 </s> where ĠIN LINE FORM 0 Ġis Ġthe Ġnumber Ġof Ġevaluated Ġe - books Ġand Ġthe Ġdiss imilar ity Ġfunction Ġis Ġdefined Ġas ĠIN LINE FORM 1 Ġ. ĠIn Ġour Ġexperiments , Ġwe Ġuse Ġthe Ġpreviously Ġtrained ĠDoc 2 V ec Ġmodel Ġto Ġextract Ġthe Ġlatent Ġrepresentation Ġof Ġa Ġspecific Ġtag . ĠThe Ġsimilarity Ġof Ġtwo Ġtags ĠIN LINE FORM 2 Ġis Ġthen Ġcalculated Ġwith Ġthe ĠCos ine Ġsimilarity Ġmeasure Ġusing Ġthe Ġlatent Ġvector Ġrepresentations Ġof Ġrespective Ġtags ĠIN LINE FORM 3 Ġand ĠIN LINE FORM 4 Ġ. </s> Results </s> Con cerning Ġtag Ġrecommendation Ġaccuracy , Ġin Ġthis Ġsection , Ġwe Ġreport Ġresults Ġfor Ġdifferent Ġvalues Ġof ĠIN LINE FORM 0 Ġ( i . e ., Ġnumber Ġof Ġrecommended Ġtags ). ĠFor Ġthe Ġbeyond - acc uracy Ġexperiment , Ġwe Ġuse Ġthe Ġfull Ġlist Ġof Ġrecommended Ġtags Ġ( i . e ., ĠIN LINE FORM 1 Ġ). </s> Recommend ation ĠAccuracy ĠEvaluation </s> Figure ĠFIG REF 11 Ġshows Ġthe Ġresults Ġof Ġthe Ġaccuracy Ġexperiment Ġfor Ġthe Ġ( i ) Ġpopularity - based , Ġ( ii ) Ġsimilarity - based , Ġand Ġ( iii ) Ġhybrid Ġtag Ġrecommendation Ġapproaches . </s> Pop ularity - based Ġapproaches . ĠIn ĠFigure ĠFIG REF 11 Ġ, Ġwe Ġsee Ġthat Ġpopularity - based Ġapproaches Ġbased Ġon Ġeditor Ġtags Ġtend Ġto Ġperform Ġbetter Ġthan Ġif Ġtrained Ġon ĠAmazon Ġsearch Ġterms . ĠIf Ġwe Ġtake Ġinto Ġaccount Ġcontextual Ġinformation Ġlike ĠB IS AC Ġor Ġauthor , Ġwe Ġcan Ġfurther Ġimprove Ġaccuracy Ġin Ġterms Ġof ĠIN LINE FORM 0 Ġ. ĠThat Ġis , Ġwe Ġfind Ġthat Ġusing Ġpopular Ġtags Ġfrom Ġe - books Ġof Ġa Ġspecific Ġauthor Ġleads Ġto Ġthe Ġbest Ġaccuracy Ġof Ġthe Ġpopularity - based Ġapproaches . ĠThis Ġsuggests Ġthat Ġeditors Ġand Ġreaders Ġdo Ġseem Ġto Ġreuse Ġtags Ġfor Ġe - books Ġof Ġsame Ġauthors . ĠIf Ġwe Ġuse Ġboth Ġeditor Ġtags Ġand ĠAmazon Ġsearch Ġterms , Ġwe Ġcan Ġfurther Ġincrease Ġaccuracy , Ġespecially Ġfor Ġhigher Ġvalues Ġof ĠIN LINE FORM 1 Ġlike Ġin Ġthe Ġcase Ġof ĠIN LINE FORM 2 Ġ. ĠThis Ġis , Ġhowever , Ġnot Ġthe Ġcase Ġfor ĠIN LINE FORM 3 Ġas Ġthe Ġaccuracy Ġof Ġthe Ġintegrated ĠIN LINE FORM 4 Ġapproach Ġis Ġlow . ĠThe Ġreason Ġfor Ġthis Ġis Ġthe Ġlimited Ġamount Ġof Ġe - books Ġfrom Ġwithin Ġthe ĠAmazon Ġsearch Ġquery Ġlogs Ġthat Ġhave ĠB IS AC Ġinformation Ġ( i . e ., Ġonly ĠIN LINE FORM 5 Ġ). </s> Similar ity - based Ġapproaches . ĠWe Ġfurther Ġimprove Ġaccuracy Ġif Ġwe Ġfirst Ġfind Ġsimilar Ġe - books Ġand Ġthen Ġextract Ġtheir Ġtop - ĠIN LINE FORM 0 Ġtags Ġin Ġa Ġcross - source Ġmanner Ġas Ġdescribed Ġin ĠSection ĠSEC REF 4 Ġ. </s> As Ġshown Ġin ĠFigure ĠFIG REF 11 Ġ, Ġusing Ġthe Ġdescription Ġtext Ġto Ġfind Ġsimilar Ġe - books Ġresults Ġin Ġmore Ġaccurate Ġtag Ġrecommendations Ġthan Ġusing Ġthe Ġtitle Ġ( i . e ., ĠIN LINE FORM 0 Ġfor ĠIN LINE FORM 1 Ġ). ĠThis Ġis Ġsomehow Ġexpected Ġas Ġthe Ġdescription Ġtext Ġconsists Ġof Ġa Ġbigger Ġcorpus Ġof Ġwords Ġ( i . e ., Ġmultiple Ġsentences ) Ġthan Ġthe Ġtitle . ĠCon cerning Ġthe Ġcollected ĠAmazon Ġsearch Ġquery Ġlogs , Ġextracting Ġand Ġthen Ġrecommending Ġtags Ġfrom Ġthis Ġsource Ġresults Ġin Ġa Ġmuch Ġlower Ġaccuracy Ġperformance . ĠThus , Ġthese Ġresults Ġalso Ġsuggest Ġto Ġinvestigate Ġbeyond - acc uracy Ġmetrics Ġas Ġdone Ġin ĠSection ĠSEC REF 17 Ġ. </s> Hy brid Ġapproaches . ĠFigure ĠFIG REF 11 Ġshows Ġthe Ġaccuracy Ġresults Ġof Ġthe Ġfour Ġhybrid Ġapproaches . ĠBy Ġcombining Ġthe Ġbest Ġthree Ġpopularity - based Ġapproaches , Ġwe Ġoutper form Ġall Ġof Ġthe Ġinitially Ġevaluated Ġpopularity Ġalgorithms Ġ( i . e ., ĠIN LINE FORM 0 Ġfor ĠIN LINE FORM 1 Ġ). ĠOn Ġthe Ġcontrary , Ġthe Ġcombination Ġof Ġthe Ġtwo Ġbest Ġperforming Ġsimilarity - based Ġapproaches ĠIN LINE FORM 2 Ġand ĠIN LINE FORM 3 Ġdoes Ġnot Ġyield Ġbetter Ġaccuracy . ĠThe Ġnegative Ġimpact Ġof Ġusing Ġa Ġlower - performing Ġapproach Ġsuch Ġas ĠIN LINE FORM 4 Ġwithin Ġa Ġhybrid Ġcombination Ġcan Ġalso Ġbe Ġobserved Ġin ĠIN LINE FORM 5 Ġfor Ġlower Ġvalues Ġof ĠIN LINE FORM 6 Ġ. ĠOverall , Ġthis Ġconfirms Ġour Ġinitial Ġintuition Ġthat Ġcombining Ġthe Ġbest Ġperforming Ġpopularity - based Ġapproach Ġwith Ġthe Ġbest Ġsimilarity - based Ġapproach Ġshould Ġresult Ġin Ġthe Ġhighest Ġaccuracy Ġ( i . e ., ĠIN LINE FORM 7 Ġfor ĠIN LINE FORM 8 Ġ). ĠMoreover , Ġour Ġgoal , Ġnamely Ġto Ġexploit Ġeditor Ġtags Ġin Ġcombination Ġwith Ġsearch Ġterms Ġused Ġby Ġreaders Ġto Ġincrease Ġthe Ġmetadata Ġquality Ġof Ġe - books , Ġis Ġshown Ġto Ġbe Ġbest Ġsupported Ġby Ġapplying Ġhybrid Ġapproaches Ġas Ġthey Ġprovide Ġthe Ġbest Ġprediction Ġresults . </s> Beyond - Acc uracy ĠEvaluation </s> Figure ĠFIG REF 16 Ġillustrates Ġthe Ġresults Ġof Ġthe Ġexperiments , Ġwhich Ġmeasure Ġthe Ġrecommendation Ġimpact Ġbeyond - acc uracy . </s> Sem antic Ġsimilarity . ĠFigure ĠFIG REF 16 Ġillustrates Ġthe Ġresults Ġof Ġour Ġproposed Ġsemantic Ġsimilarity Ġmeasure . ĠTo Ġcompare Ġour Ġproposed Ġmeasure Ġto Ġstandard Ġaccuracy Ġmeasures Ġsuch Ġas ĠIN LINE FORM 0 Ġ, Ġwe Ġuse ĠKendall 's ĠTau Ġrank Ġcorrelation ĠB IB REF 19 Ġas Ġsuggested Ġby ĠB IB REF 20 Ġfor Ġautomatic Ġevaluation Ġof Ġinformation - ordering Ġtasks . ĠFrom Ġthat , Ġwe Ġrank Ġour Ġrecommendation Ġapproaches Ġaccording Ġto Ġboth Ġaccuracy Ġand Ġsemantic Ġsimilarity Ġand Ġcalculate Ġthe Ġrelation Ġbetween Ġboth Ġrankings . ĠThis Ġresults Ġin ĠIN LINE FORM 1 Ġwith Ġa Ġp - value Ġ< ĠIN LINE FORM 2 Ġ, Ġwhich Ġsuggests Ġa Ġhigh Ġcorrelation Ġbetween Ġthe Ġsemantic Ġsimilarity Ġand Ġthe Ġstandard Ġaccuracy Ġmeasure . </s> Therefore , Ġthe Ġsemantic Ġsimilarity Ġmeasure Ġhelps Ġus Ġinterpret Ġthe Ġrecommendation Ġquality . ĠFor Ġinstance , Ġwe Ġachieve Ġthe Ġlowest ĠIN LINE FORM 0 Ġvalues Ġwith Ġthe Ġsimilarity - based Ġapproaches Ġthat Ġrecommend ĠAmazon Ġsearch Ġterms Ġ( i . e ., ĠIN LINE FORM 1 Ġand ĠIN LINE FORM 2 Ġ). ĠWhen Ġcomparing Ġthese Ġresults Ġwith Ġothers Ġfrom ĠFigure ĠFIG REF 11 Ġ, Ġa Ġconclusion Ġcould Ġbe Ġquickly Ġdrawn Ġthat Ġthe Ġrecommended Ġtags Ġare Ġmerely Ġunus able . ĠHowever , Ġby Ġlooking Ġat ĠFigure ĠFIG REF 16 Ġ, Ġwe Ġsee Ġthat , Ġalthough Ġthese Ġapproaches Ġdo Ġnot Ġprovide Ġthe Ġhighest Ġrecommendation Ġaccuracy , Ġthey Ġstill Ġresult Ġin Ġtag Ġrecommendations Ġthat Ġare Ġsem antically Ġrelated Ġat Ġa Ġhigh Ġdegree Ġto Ġthe Ġexpected Ġannotations Ġfrom Ġthe Ġtest Ġset . ĠOverall , Ġthis Ġsuggests Ġthat Ġapproaches , Ġwhich Ġprovide Ġa Ġpoor Ġaccuracy Ġperformance Ġconcerning ĠIN LINE FORM 4 Ġbut Ġprovide Ġa Ġgood Ġperformance Ġregarding Ġsemantic Ġsimilarity Ġcould Ġstill Ġbe Ġhelpful Ġfor Ġannot ating Ġe - books . </s> Recommend ation Ġdiversity . ĠFigure ĠFIG REF 16 Ġshows Ġthe Ġdiversity Ġof Ġthe Ġtag Ġrecommendation Ġapproaches . ĠWe Ġachieve Ġthe Ġhighest Ġdiversity Ġwith Ġthe Ġsimilarity - based Ġapproaches , Ġwhich Ġextract ĠAmazon Ġsearch Ġterms . ĠTheir Ġaccuracy Ġis , Ġhowever , Ġvery Ġlow . ĠThus , Ġthe Ġcombination Ġof Ġthe Ġtwo Ġvoc ab ular ies Ġcan Ġprovide Ġa Ġgood Ġtrade - off Ġbetween Ġrecommendation Ġaccuracy Ġand Ġdiversity . </s> Conclusion Ġand ĠFuture ĠWork </s> In Ġthis Ġpaper , Ġwe Ġpresent Ġour Ġwork Ġto Ġsupport Ġeditors Ġin Ġthe Ġe - book Ġannotation Ġprocess . ĠSpecifically , Ġwe Ġaim Ġto Ġprovide Ġtag Ġrecommendations Ġthat Ġincorporate Ġboth Ġthe Ġvocabulary Ġof Ġthe Ġeditors Ġand Ġe - book Ġreaders . ĠTherefore , Ġwe Ġtrain Ġvarious Ġconfigurations Ġof Ġtag Ġrecomm ender Ġapproaches Ġon Ġeditors ' Ġtags Ġand ĠAmazon Ġsearch Ġterms Ġand Ġevaluate Ġthem Ġon Ġa Ġdataset Ġcontaining ĠAmazon Ġreview Ġkeywords . ĠWe Ġfind Ġthat Ġcombining Ġboth Ġdata Ġsources Ġenhances Ġthe Ġquality Ġof Ġtag Ġrecommendations Ġfor Ġannot ating Ġe - books . ĠFurthermore , Ġwhile Ġapproaches Ġthat Ġtrain Ġonly Ġon ĠAmazon Ġsearch Ġterms Ġprovide Ġpoor Ġperformance Ġconcerning Ġrecommendation Ġaccuracy , Ġwe Ġshow Ġthat Ġthey Ġstill Ġoffer Ġhelpful Ġannotations Ġconcerning Ġrecommendation Ġdiversity Ġas Ġwell Ġas Ġour Ġnovel Ġsemantic Ġsimilarity Ġmetric . </s> Future Ġwork . ĠFor Ġfuture Ġwork , Ġwe Ġplan Ġto Ġvalidate Ġour Ġfindings Ġusing Ġanother Ġdataset , Ġe . g ., Ġby Ġrecommending Ġtags Ġfor Ġscientific Ġarticles Ġand Ġbooks Ġin ĠBib S onomy . ĠWith Ġthis , Ġwe Ġaim Ġto Ġdemonstrate Ġthe Ġusefulness Ġof Ġthe Ġproposed Ġapproach Ġin Ġa Ġsimilar Ġdomain Ġand Ġto Ġenhance Ġthe Ġreprodu c ibility Ġof Ġour Ġresults Ġby Ġusing Ġan Ġopen Ġdataset . </s> Moreover , Ġwe Ġplan Ġto Ġevaluate Ġour Ġtag Ġrecommendation Ġapproaches Ġin Ġa Ġstudy Ġwith Ġdomain Ġusers . ĠAlso , Ġwe Ġwant Ġto Ġimprove Ġour Ġsimilarity - based Ġapproaches Ġby Ġintegrating Ġnovel Ġembed ding Ġapproaches ĠB IB REF 16 Ġ, ĠB IB REF 17 Ġas Ġwe Ġdid , Ġfor Ġexample , Ġwith Ġour Ġproposed Ġsemantic Ġsimilarity Ġevaluation Ġmetric . ĠFinally , Ġwe Ġaim Ġto Ġincorporate Ġexplanations Ġfor Ġrecommended Ġtags Ġso Ġthat Ġeditors Ġof Ġe - book Ġannotations Ġreceive Ġadditional Ġsupport Ġin Ġannot ating Ġe - books ĠB IB REF 21 Ġ. ĠBy Ġmaking Ġthe Ġunderlying Ġ( sem antic ) Ġreasoning Ġvisible Ġto Ġthe Ġeditor Ġwho Ġis Ġin Ġcharge Ġof Ġtail oring Ġannotations , Ġwe Ġaim Ġto Ġsupport Ġtwo Ġgoals : Ġ( i ) Ġallowing Ġreaders Ġto Ġdiscover Ġe - books Ġmore Ġefficiently , Ġand Ġ( ii ) Ġenabling Ġpublishers Ġto Ġleverage Ġsemi - automatic Ġcategor ization Ġprocesses Ġfor Ġe - books . ĠIn Ġturn , Ġproviding Ġexplanations Ġfost ers Ġcontrol Ġover Ġwhich Ġvocabulary Ġto Ġchoose Ġwhen Ġtagging Ġe - books Ġfor Ġdifferent Ġapplication Ġcontexts . </s> Acknowled gments . ĠThe Ġauthors Ġwould Ġlike Ġto Ġthank ĠPeter ĠLang s , ĠJan - Philipp ĠWolf Ġand ĠAly ona ĠSch ra a Ġfrom ĠH GV ĠG mb H Ġfor Ġproviding Ġthe Ġe - book Ġannotation Ġdata . ĠThis Ġwork Ġwas Ġfunded Ġby Ġthe ĠKnow - Center ĠG mb H Ġ( FF G ĠCOM ET ĠProgram ), Ġthe ĠFF G ĠData ĠMarket ĠAustria Ġproject Ġand Ġthe ĠAI 4 EU Ġproject Ġ( EU Ġgrant Ġ8 256 19 ). ĠThe ĠKnow - Center ĠG mb H Ġis Ġfunded Ġwithin Ġthe ĠAustrian ĠCOM ET ĠProgram Ġ- ĠCompet ence ĠCenters Ġfor ĠExcellent ĠTechnologies Ġ- Ġunder Ġthe Ġausp ices Ġof Ġthe ĠAustrian ĠMinistry Ġof ĠTransport , ĠInnovation Ġand ĠTechnology , Ġthe ĠAustrian ĠMinistry Ġof ĠEconomics Ġand ĠLabor Ġand Ġby Ġthe ĠState Ġof ĠSty ria . ĠCOM ET Ġis Ġmanaged Ġby Ġthe ĠAustrian ĠResearch ĠPromotion ĠAgency Ġ( FF G ).