generating a poem from images we use an existing actor-critic architecture, various types of sequence to sequence models
EXTRACTIVE
For generating a poem from images we use an existing actor-critic architecture BIBREF1. This involves 3 parallel CNNs: an object CNN, sentiment CNN, and scene CNN, for feature extraction. These features are combined with a skip-thought model which provides poetic clues, which are then fed into a sequence-to-sequence model trained by policy gradient with 2 discriminator networks for rewards. This as a whole forms a pipeline that takes in an image and outputs a poem as shown on the top left of Figure FIGREF4. A CNN-RNN generative model acts as an agent. The parameters of this agent define a policy whose execution determines which word is selected as an action. When the agent selects all words in a poem, it receives a reward. Two discriminative networks, shown on the top right of Figure FIGREF4, are defined to serve as rewards concerning whether the generated poem properly describes the input image and whether the generated poem is poetic. The goal of the poem generation model is to generate a sequence of words as a poem for an image to maximize the expected return. For Shakespearizing modern English texts, we experimented with various types of sequence to sequence models. Since the size of the parallel translation data available is small, we leverage a dictionary providing a mapping between Shakespearean words and modern English words to retrofit pre-trained word embeddings. Incorporating this extra information improves the translation task. The large number of shared word types between the source and target sentences indicates that sharing the representation between them is beneficial. We use a sequence-to-sequence model which consists of a single layer unidrectional LSTM encoder and a single layer LSTM decoder and pre-trained retrofitted word embeddings shared between source and target sentences. We experimented with two different types of attention: global attention BIBREF9, in which the model makes use of the output from the encoder and decoder for the current time step only, and Bahdanau attention BIBREF10, where computing attention requires the output of the decoder from the prior time step. We found that global attention performs better in practice for our task of text style transfer. Since a pair of corresponding Shakespeare and modern English sentences have significant vocabulary overlap we extend the sequence-to-sequence model mentioned above using pointer networks BIBREF11 that provide location based attention and have been used to enable copying of tokens directly from the input. Moreover, there are lot of proper nouns and rare words which might not be predicted by a vanilla sequence to sequence model.
1910.03634
Introduction </s> Ne ural Ġnetworks Ġhave Ġbeen Ġsuccessfully Ġused Ġto Ġdescribe Ġimages Ġwith Ġtext Ġusing Ġsequence - to - sequence Ġmodels ĠB IB REF 0 . ĠHowever , Ġthe Ġresults Ġare Ġsimple Ġand Ġdry Ġcapt ions Ġwhich Ġare Ġone Ġor Ġtwo Ġphrases Ġlong . ĠHumans Ġlooking Ġat Ġa Ġpainting Ġsee Ġmore Ġthan Ġjust Ġobjects . ĠPaint ings Ġstimulate Ġsentiments , Ġmetaphors Ġand Ġstories Ġas Ġwell . ĠTherefore , Ġour Ġgoal Ġis Ġto Ġhave Ġa Ġneural Ġnetwork Ġdescribe Ġthe Ġpainting Ġartist ically Ġin Ġa Ġstyle Ġof Ġchoice . ĠAs Ġa Ġproof Ġof Ġconcept , Ġwe Ġpresent Ġa Ġmodel Ġwhich Ġgenerates ĠShakespeare an Ġprose Ġfor Ġa Ġgiven Ġpainting Ġas Ġshown Ġin ĠFigure ĠFIG REF 1 . ĠAcc ompl ishing Ġthis Ġtask Ġis Ġdifficult Ġwith Ġtraditional Ġsequence Ġto Ġsequence Ġmodels Ġsince Ġthere Ġdoes Ġnot Ġexist Ġa Ġlarge Ġcollection Ġof ĠShakespeare an Ġprose Ġwhich Ġdescribes Ġpaintings : ĠShakespeare 's Ġworks Ġdescribes Ġa Ġsingle Ġpainting Ġshown Ġin ĠFigure ĠFIG REF 3 . ĠFortunately Ġwe Ġhave Ġa Ġdataset Ġof Ġmodern ĠEnglish Ġpoems Ġwhich Ġdescribe Ġimages ĠB IB REF 1 Ġand Ġline - by - line Ġmodern Ġparaph r ases Ġof ĠShakespeare 's Ġplays ĠB IB REF 2 . ĠOur Ġsolution Ġis Ġtherefore Ġto Ġcombine Ġtwo Ġseparately Ġtrained Ġmodels Ġto Ġsynthes ize ĠShakespeare an Ġprose Ġfor Ġa Ġgiven Ġpainting . </s> Introduction Ġ: :: ĠRelated Ġwork </s> A Ġgeneral Ġend - to - end Ġapproach Ġto Ġsequence Ġlearning ĠB IB REF 3 Ġmakes Ġminimal Ġassumptions Ġon Ġthe Ġsequence Ġstructure . ĠThis Ġmodel Ġis Ġwidely Ġused Ġin Ġtasks Ġsuch Ġas Ġmachine Ġtranslation , Ġtext Ġsummar ization , Ġconvers ational Ġmodeling , Ġand Ġimage Ġcaption ing . ĠA Ġgener ative Ġmodel Ġusing Ġa Ġdeep Ġrecurrent Ġarchitecture ĠB IB REF 0 Ġhas Ġalso Ġbe een Ġused Ġfor Ġgenerating Ġphrases Ġdescribing Ġan Ġimage . ĠThe Ġtask Ġof Ġsynthes izing Ġmultiple Ġlines Ġof Ġpoetry Ġfor Ġa Ġgiven Ġimage ĠB IB REF 1 Ġis Ġaccomplished Ġby Ġextracting Ġpoetic Ġclues Ġfrom Ġimages . ĠGiven Ġthe Ġcontext Ġimage , Ġthe Ġnetwork Ġassociates Ġimage Ġattributes Ġwith Ġpoetic Ġdescriptions Ġusing Ġa Ġconv olution al Ġneural Ġnet . ĠThe Ġpoem Ġis Ġgenerated Ġusing Ġa Ġrecurrent Ġneural Ġnet Ġwhich Ġis Ġtrained Ġusing Ġmulti - ad vers arial Ġtraining Ġvia Ġpolicy Ġgradient . </s> Trans forming Ġtext Ġfrom Ġmodern ĠEnglish Ġto ĠShakespeare an ĠEnglish Ġusing Ġtext Ġ" style Ġtransfer " Ġis Ġchallenging . ĠAn Ġend Ġto Ġend Ġapproach Ġusing Ġa Ġsequence - to - sequence Ġmodel Ġover Ġa Ġparallel Ġtext Ġcorpus ĠB IB REF 2 Ġhas Ġbeen Ġproposed Ġbased Ġon Ġmachine Ġtranslation . ĠIn Ġthe Ġabsence Ġof Ġa Ġparallel Ġtext Ġcorpus , Ġgener ative Ġadvers arial Ġnetworks Ġ( GAN s ) Ġhave Ġbeen Ġused , Ġwhich Ġsimultaneously Ġtrain Ġtwo Ġmodels : Ġa Ġgener ative Ġmodel Ġwhich Ġcaptures Ġthe Ġdata Ġdistribution , Ġand Ġa Ġdiscrim in ative Ġmodel Ġwhich Ġevaluates Ġthe Ġperformance Ġof Ġthe Ġgenerator . ĠUsing Ġa Ġtarget Ġdomain Ġlanguage Ġmodel Ġas Ġa Ġdiscrim inator Ġhas Ġalso Ġbeen Ġemployed ĠB IB REF 4 , Ġproviding Ġricher Ġand Ġmore Ġstable Ġtoken - level Ġfeedback Ġduring Ġthe Ġlearning Ġprocess . ĠA Ġkey Ġchallenge Ġin Ġboth Ġimage Ġand Ġtext Ġstyle Ġtransfer Ġis Ġseparating Ġcontent Ġfrom Ġstyle ĠB IB REF 5 , ĠB IB REF 6 , ĠB IB REF 7 . ĠCross - aligned Ġauto - enc oder Ġmodels Ġhave Ġfocused Ġon Ġstyle Ġtransfer Ġusing Ġnon - par allel Ġtext ĠB IB REF 7 . ĠRecently , Ġa Ġfine Ġgr ained Ġmodel Ġfor Ġtext Ġstyle Ġtransfer Ġhas Ġbeen Ġproposed ĠB IB REF 8 Ġwhich Ġcontrols Ġseveral Ġfactors Ġof Ġvariation Ġin Ġtextual Ġdata Ġby Ġusing Ġback - translation . ĠThis Ġallows Ġcontrol Ġover Ġmultiple Ġattributes , Ġsuch Ġas Ġgender Ġand Ġsentiment , Ġand Ġfine - gr ained Ġcontrol Ġover Ġthe Ġtrade - off Ġbetween Ġcontent Ġand Ġstyle . </s> Methods </s> We Ġuse Ġa Ġtotal Ġthree Ġdatasets : Ġtwo Ġdatasets Ġfor Ġgenerating Ġan ĠEnglish Ġpoem Ġfrom Ġan Ġimage , Ġand ĠShakespeare Ġplays Ġand Ġtheir ĠEnglish Ġtranslations Ġfor Ġtext Ġstyle Ġtransfer . </s> We Ġtrain Ġa Ġmodel Ġfor Ġgenerating Ġpoems Ġfrom Ġimages Ġbased Ġon Ġtwo Ġdatasets ĠB IB REF 1 . ĠThe Ġfirst Ġdataset Ġconsists Ġof Ġimage Ġand Ġpoem Ġpairs , Ġnamely Ġa Ġmulti - mod al Ġpoem Ġdataset Ġ( Multi M - Po em ), Ġand Ġthe Ġsecond Ġdataset Ġis Ġa Ġlarge Ġpoem Ġcorpus , Ġnamely Ġa Ġun i - mod al Ġpoem Ġdataset Ġ( Un i M - Po em ). ĠThe Ġimage Ġand Ġpoem Ġpairs Ġare Ġextended Ġby Ġadding Ġthe Ġnearest Ġthree Ġneighbor Ġpoems Ġfrom Ġthe Ġpoem Ġcorpus Ġwithout Ġredundancy , Ġand Ġan Ġextended Ġimage Ġand Ġpoem Ġpair Ġdataset Ġis Ġconstructed Ġand Ġden oted Ġas ĠMulti M - Po em ( Ex ) BI B REF 1 . </s> We Ġuse Ġa Ġcollection Ġof Ġline - by - line Ġmodern Ġparaph r ases Ġfor Ġ16 Ġof ĠShakespeare âĢ Ļ s Ġplays ĠB IB REF 2 , Ġfor Ġtraining Ġa Ġstyle Ġtransfer Ġnetwork Ġfrom ĠEnglish Ġpoems Ġto ĠShakespeare an Ġprose . ĠWe Ġuse Ġ18 , 395 Ġsentences Ġfrom Ġthe Ġtraining Ġdata Ġsplit . ĠWe Ġkeep Ġ1 , 218 Ġsentences Ġin Ġthe Ġvalidation Ġdata Ġset Ġand Ġ1 , 462 Ġsentences Ġin Ġour Ġtest Ġset . </s> Methods Ġ: :: ĠImage ĠTo ĠPo em ĠActor - Crit ic ĠModel </s> For Ġgenerating Ġa Ġpoem Ġfrom Ġimages Ġwe Ġuse Ġan Ġexisting Ġactor - crit ic Ġarchitecture ĠB IB REF 1 . ĠThis Ġinvolves Ġ3 Ġparallel ĠCNN s : Ġan Ġobject ĠCNN , Ġsentiment ĠCNN , Ġand Ġscene ĠCNN , Ġfor Ġfeature Ġextraction . ĠThese Ġfeatures Ġare Ġcombined Ġwith Ġa Ġskip - thought Ġmodel Ġwhich Ġprovides Ġpoetic Ġclues , Ġwhich Ġare Ġthen Ġfed Ġinto Ġa Ġsequence - to - sequence Ġmodel Ġtrained Ġby Ġpolicy Ġgradient Ġwith Ġ2 Ġdiscrim inator Ġnetworks Ġfor Ġrewards . ĠThis Ġas Ġa Ġwhole Ġforms Ġa Ġpipeline Ġthat Ġtakes Ġin Ġan Ġimage Ġand Ġoutputs Ġa Ġpoem Ġas Ġshown Ġon Ġthe Ġtop Ġleft Ġof ĠFigure ĠFIG REF 4 . ĠA ĠCNN - R NN Ġgener ative Ġmodel Ġacts Ġas Ġan Ġagent . ĠThe Ġparameters Ġof Ġthis Ġagent Ġdefine Ġa Ġpolicy Ġwhose Ġexecution Ġdetermines Ġwhich Ġword Ġis Ġselected Ġas Ġan Ġaction . ĠWhen Ġthe Ġagent Ġselects Ġall Ġwords Ġin Ġa Ġpoem , Ġit Ġreceives Ġa Ġreward . ĠTwo Ġdiscrim in ative Ġnetworks , Ġshown Ġon Ġthe Ġtop Ġright Ġof ĠFigure ĠFIG REF 4 , Ġare Ġdefined Ġto Ġserve Ġas Ġrewards Ġconcerning Ġwhether Ġthe Ġgenerated Ġpoem Ġproperly Ġdescribes Ġthe Ġinput Ġimage Ġand Ġwhether Ġthe Ġgenerated Ġpoem Ġis Ġpoetic . ĠThe Ġgoal Ġof Ġthe Ġpoem Ġgeneration Ġmodel Ġis Ġto Ġgenerate Ġa Ġsequence Ġof Ġwords Ġas Ġa Ġpoem Ġfor Ġan Ġimage Ġto Ġmaximize Ġthe Ġexpected Ġreturn . </s> Methods Ġ: :: ĠSh akespe ar izing ĠPo etic ĠCapt ions </s> For ĠSh akespe ar izing Ġmodern ĠEnglish Ġtexts , Ġwe Ġexperimented Ġwith Ġvarious Ġtypes Ġof Ġsequence Ġto Ġsequence Ġmodels . ĠSince Ġthe Ġsize Ġof Ġthe Ġparallel Ġtranslation Ġdata Ġavailable Ġis Ġsmall , Ġwe Ġleverage Ġa Ġdictionary Ġproviding Ġa Ġmapping Ġbetween ĠShakespeare an Ġwords Ġand Ġmodern ĠEnglish Ġwords Ġto Ġretro fit Ġpre - trained Ġword Ġembed d ings . ĠInc orpor ating Ġthis Ġextra Ġinformation Ġimproves Ġthe Ġtranslation Ġtask . ĠThe Ġlarge Ġnumber Ġof Ġshared Ġword Ġtypes Ġbetween Ġthe Ġsource Ġand Ġtarget Ġsentences Ġindicates Ġthat Ġsharing Ġthe Ġrepresentation Ġbetween Ġthem Ġis Ġbeneficial . </s> Methods Ġ: :: ĠSh akespe ar izing ĠPo etic ĠCapt ions Ġ: :: ĠSe q 2 Se q Ġwith ĠAttention </s> We Ġuse Ġa Ġsequence - to - sequence Ġmodel Ġwhich Ġconsists Ġof Ġa Ġsingle Ġlayer Ġun id rection al ĠL ST M Ġenc oder Ġand Ġa Ġsingle Ġlayer ĠL ST M Ġdec oder Ġand Ġpre - trained Ġretro fitted Ġword Ġembed d ings Ġshared Ġbetween Ġsource Ġand Ġtarget Ġsentences . ĠWe Ġexperimented Ġwith Ġtwo Ġdifferent Ġtypes Ġof Ġattention : Ġglobal Ġattention ĠB IB REF 9 , Ġin Ġwhich Ġthe Ġmodel Ġmakes Ġuse Ġof Ġthe Ġoutput Ġfrom Ġthe Ġenc oder Ġand Ġdec oder Ġfor Ġthe Ġcurrent Ġtime Ġstep Ġonly , Ġand ĠBah dan au Ġattention ĠB IB REF 10 , Ġwhere Ġcomputing Ġattention Ġrequires Ġthe Ġoutput Ġof Ġthe Ġdec oder Ġfrom Ġthe Ġprior Ġtime Ġstep . ĠWe Ġfound Ġthat Ġglobal Ġattention Ġperforms Ġbetter Ġin Ġpractice Ġfor Ġour Ġtask Ġof Ġtext Ġstyle Ġtransfer . </s> Methods Ġ: :: ĠSh akespe ar izing ĠPo etic ĠCapt ions Ġ: :: ĠSe q 2 Se q Ġwith Ġa ĠPo inter ĠNetwork </s> Since Ġa Ġpair Ġof Ġcorresponding ĠShakespeare Ġand Ġmodern ĠEnglish Ġsentences Ġhave Ġsignificant Ġvocabulary Ġoverlap Ġwe Ġextend Ġthe Ġsequence - to - sequence Ġmodel Ġmentioned Ġabove Ġusing Ġpointer Ġnetworks ĠB IB REF 11 Ġthat Ġprovide Ġlocation Ġbased Ġattention Ġand Ġhave Ġbeen Ġused Ġto Ġenable Ġcopying Ġof Ġtokens Ġdirectly Ġfrom Ġthe Ġinput . ĠMoreover , Ġthere Ġare Ġlot Ġof Ġproper Ġnoun s Ġand Ġrare Ġwords Ġwhich Ġmight Ġnot Ġbe Ġpredicted Ġby Ġa Ġvanilla Ġsequence Ġto Ġsequence Ġmodel . </s> Methods Ġ: :: ĠSh akespe ar izing ĠPo etic ĠCapt ions Ġ: :: ĠPrediction </s> For Ġboth Ġseq 2 seq Ġmodels , Ġwe Ġuse Ġthe Ġattention Ġmat rices Ġreturned Ġat Ġeach Ġdec oder Ġtime Ġstep Ġduring Ġinference , Ġto Ġcompute Ġthe Ġnext Ġword Ġin Ġthe Ġtranslated Ġsequence Ġif Ġthe Ġdec oder Ġoutput Ġat Ġthe Ġcurrent Ġtime Ġstep Ġis Ġthe ĠUN K Ġtoken . ĠWe Ġreplace Ġthe ĠUN K s Ġin Ġthe Ġtarget Ġoutput Ġwith Ġthe Ġhighest Ġaligned , Ġmaximum Ġattention , Ġsource Ġword . ĠThe Ġseq 2 seq Ġmodel Ġwith Ġglobal Ġattention Ġgives Ġthe Ġbest Ġresults Ġwith Ġan Ġaverage Ġtarget ĠB LE U Ġscore Ġof Ġ29 . 65 Ġon Ġthe Ġstyle Ġtransfer Ġdataset , Ġcompared Ġwith Ġan Ġaverage Ġtarget ĠB LE U Ġscore Ġof Ġ26 . 97 Ġusing Ġthe Ġseq 2 seq Ġmodel Ġwith Ġpointer Ġnetworks . </s> Results </s> We Ġperform Ġa Ġqualitative Ġanalysis Ġof Ġthe ĠShakespeare an Ġprose Ġgenerated Ġfor Ġthe Ġinput Ġpaintings . ĠWe Ġconducted Ġa Ġsurvey , Ġin Ġwhich Ġwe Ġpresented Ġfamous Ġpaintings Ġincluding Ġthose Ġshown Ġin ĠFigures ĠFIG REF 1 Ġand ĠFIG REF 10 Ġand Ġthe Ġcorresponding ĠShakespeare an Ġprose Ġgenerated Ġby Ġthe Ġmodel , Ġand Ġasked Ġ32 Ġstudents Ġto Ġrate Ġthem Ġon Ġthe Ġbasis Ġof Ġcontent , Ġcreativity Ġand Ġsimilarity Ġto ĠShakespeare an Ġstyle Ġon Ġa ĠL ik ert Ġscale Ġof Ġ1 - 5 . ĠFigure ĠFIG REF 12 Ġshows Ġthe Ġresult Ġof Ġour Ġhuman Ġevaluation . </s> The Ġaverage Ġcontent Ġscore Ġacross Ġthe Ġpaintings Ġis Ġ3 . 7 Ġwhich Ġdemonstrates Ġthat Ġthe Ġprose Ġgenerated Ġis Ġrelevant Ġto Ġthe Ġpainting . ĠThe Ġaverage Ġcreativity Ġscore Ġis Ġ3 . 9 Ġwhich Ġdemonstrates Ġthat Ġthe Ġmodel Ġcaptures Ġmore Ġthan Ġbasic Ġobjects Ġin Ġthe Ġpainting Ġsuccessfully Ġusing Ġpoetic Ġclues Ġin Ġthe Ġscene . ĠThe Ġaverage Ġstyle Ġscore Ġis Ġ3 . 9 Ġwhich Ġdemonstrates Ġthat Ġthe Ġprose Ġgenerated Ġis Ġperceived Ġto Ġbe Ġin Ġthe Ġstyle Ġof ĠShakespeare . </s> We Ġalso Ġperform Ġa Ġquantitative Ġanalysis Ġof Ġstyle Ġtransfer Ġby Ġgenerating ĠB LE U Ġscores Ġfor Ġthe Ġmodel Ġoutput Ġusing Ġthe Ġstyle Ġtransfer Ġdataset . ĠThe Ġvariation Ġof Ġthe ĠB LE U Ġscores Ġwith Ġthe Ġsource Ġsentence Ġlengths Ġis Ġshown Ġin ĠFigure ĠFIG REF 11 . ĠAs Ġexpected , Ġthe ĠB LE U Ġscores Ġdecrease Ġwith Ġincrease Ġin Ġsource Ġsentence Ġlengths . </s> Results Ġ: :: ĠImplementation </s> All Ġmodels Ġwere Ġtrained Ġon ĠGoogle ĠCol ab Ġwith Ġa Ġsingle ĠGPU Ġusing ĠPython Ġ3 . 6 Ġand ĠT ensor flow Ġ2 . 0 . ĠThe Ġnumber Ġof Ġhidden Ġunits Ġfor Ġthe Ġenc oder Ġand Ġdec oder Ġis Ġ1 , 576 Ġand Ġ256 Ġfor Ġseq 2 seq Ġwith Ġglobal Ġattention Ġand Ġseq 2 seq Ġwith Ġpointer Ġnetworks Ġrespectively . ĠAdam Ġoptim izer Ġwas Ġused Ġwith Ġthe Ġdefault Ġlearning Ġrate Ġof Ġ0 . 001 . ĠThe Ġmodel Ġwas Ġtrained Ġfor Ġ25 Ġepoch s . ĠWe Ġuse Ġpre - trained Ġretro fitted Ġword Ġembed d ings Ġof Ġdimension Ġ192 . </s> Results Ġ: :: ĠLim itations </s> Since Ġwe Ġdo Ġnot Ġhave Ġan Ġend - to - end Ġdataset , Ġthe Ġgenerated ĠEnglish Ġpoem Ġmay Ġnot Ġwork Ġwell Ġwith ĠShakespeare Ġstyle Ġtransfer Ġas Ġshown Ġin ĠFigure ĠFIG REF 12 Ġfor Ġ" St arry ĠNight " Ġwith Ġa Ġlow Ġaverage Ġcontent Ġscore . ĠThis Ġhappens Ġwhen Ġthe Ġstyle Ġtransfer Ġdataset Ġdoes Ġnot Ġhave Ġsimilar Ġwords Ġin Ġthe Ġtraining Ġset Ġof Ġsentences . ĠA Ġsolution Ġwould Ġbe Ġto Ġexpand Ġthe Ġstyle Ġtransfer Ġdataset , Ġfor Ġa Ġbetter Ġrepresentation Ġof Ġthe Ġpoem Ġdata . </s> Results Ġ: :: ĠCon clusions Ġand ĠFuture ĠWork </s> In Ġconclusion , Ġcombining Ġtwo Ġpipelines Ġwith Ġan Ġintermediate Ġrepresentation Ġworks Ġwell Ġin Ġpractice . ĠWe Ġobserve Ġthat Ġa ĠCNN - R NN Ġbased Ġimage - to - po em Ġnet Ġcombined Ġwith Ġa Ġseq 2 seq Ġmodel Ġwith Ġparallel Ġtext Ġcorpus Ġfor Ġtext Ġstyle Ġtransfer Ġsynthes izes ĠShakespeare - style Ġprose Ġfor Ġa Ġgiven Ġpainting . ĠFor Ġthe Ġseq 2 seq Ġmodel Ġused , Ġwe Ġobserve Ġthat Ġit Ġperforms Ġbetter Ġin Ġpractice Ġusing Ġglobal Ġattention Ġas Ġcompared Ġwith Ġlocal Ġattention . ĠWe Ġmake Ġour Ġmodels Ġand Ġcode Ġpublicly Ġavailable ĠB IB REF 12 . ĠIn Ġfuture Ġwork Ġwe Ġwould Ġlike Ġto Ġexperiment Ġwith ĠG AN s Ġin Ġthe Ġabsence Ġof Ġnon - par allel Ġdatasets , Ġso Ġthat Ġwe Ġcan Ġuse Ġvaried Ġstyles Ġfor Ġtext Ġstyle Ġtransfer . ĠWe Ġwould Ġalso Ġlike Ġto Ġexperiment Ġwith Ġcross Ġaligned Ġauto - enc od ers , Ġwhich Ġform Ġa Ġlatent Ġcontent Ġrepresentation , Ġto Ġefficiently Ġseparate Ġstyle Ġand Ġcontent .
Introduction Neural networks have been successfully used to describe images with text using sequence-to-sequence models BIBREF0. However, the results are simple and dry captions which are one or two phrases long. Humans looking at a painting see more than just objects. Paintings stimulate sentiments, metaphors and stories as well. Therefore, our goal is to have a neural network describe the painting artistically in a style of choice. As a proof of concept, we present a model which generates Shakespearean prose for a given painting as shown in Figure FIGREF1. Accomplishing this task is difficult with traditional sequence to sequence models since there does not exist a large collection of Shakespearean prose which describes paintings: Shakespeare's works describes a single painting shown in Figure FIGREF3. Fortunately we have a dataset of modern English poems which describe images BIBREF1 and line-by-line modern paraphrases of Shakespeare's plays BIBREF2. Our solution is therefore to combine two separately trained models to synthesize Shakespearean prose for a given painting. Introduction ::: Related work A general end-to-end approach to sequence learning BIBREF3 makes minimal assumptions on the sequence structure. This model is widely used in tasks such as machine translation, text summarization, conversational modeling, and image captioning. A generative model using a deep recurrent architecture BIBREF0 has also beeen used for generating phrases describing an image. The task of synthesizing multiple lines of poetry for a given image BIBREF1 is accomplished by extracting poetic clues from images. Given the context image, the network associates image attributes with poetic descriptions using a convolutional neural net. The poem is generated using a recurrent neural net which is trained using multi-adversarial training via policy gradient. Transforming text from modern English to Shakespearean English using text "style transfer" is challenging. An end to end approach using a sequence-to-sequence model over a parallel text corpus BIBREF2 has been proposed based on machine translation. In the absence of a parallel text corpus, generative adversarial networks (GANs) have been used, which simultaneously train two models: a generative model which captures the data distribution, and a discriminative model which evaluates the performance of the generator. Using a target domain language model as a discriminator has also been employed BIBREF4, providing richer and more stable token-level feedback during the learning process. A key challenge in both image and text style transfer is separating content from style BIBREF5, BIBREF6, BIBREF7. Cross-aligned auto-encoder models have focused on style transfer using non-parallel text BIBREF7. Recently, a fine grained model for text style transfer has been proposed BIBREF8 which controls several factors of variation in textual data by using back-translation. This allows control over multiple attributes, such as gender and sentiment, and fine-grained control over the trade-off between content and style. Methods We use a total three datasets: two datasets for generating an English poem from an image, and Shakespeare plays and their English translations for text style transfer. We train a model for generating poems from images based on two datasets BIBREF1. The first dataset consists of image and poem pairs, namely a multi-modal poem dataset (MultiM-Poem), and the second dataset is a large poem corpus, namely a uni-modal poem dataset (UniM-Poem). The image and poem pairs are extended by adding the nearest three neighbor poems from the poem corpus without redundancy, and an extended image and poem pair dataset is constructed and denoted as MultiM-Poem(Ex)BIBREF1. We use a collection of line-by-line modern paraphrases for 16 of Shakespeare’s plays BIBREF2, for training a style transfer network from English poems to Shakespearean prose. We use 18,395 sentences from the training data split. We keep 1,218 sentences in the validation data set and 1,462 sentences in our test set. Methods ::: Image To Poem Actor-Critic Model For generating a poem from images we use an existing actor-critic architecture BIBREF1. This involves 3 parallel CNNs: an object CNN, sentiment CNN, and scene CNN, for feature extraction. These features are combined with a skip-thought model which provides poetic clues, which are then fed into a sequence-to-sequence model trained by policy gradient with 2 discriminator networks for rewards. This as a whole forms a pipeline that takes in an image and outputs a poem as shown on the top left of Figure FIGREF4. A CNN-RNN generative model acts as an agent. The parameters of this agent define a policy whose execution determines which word is selected as an action. When the agent selects all words in a poem, it receives a reward. Two discriminative networks, shown on the top right of Figure FIGREF4, are defined to serve as rewards concerning whether the generated poem properly describes the input image and whether the generated poem is poetic. The goal of the poem generation model is to generate a sequence of words as a poem for an image to maximize the expected return. Methods ::: Shakespearizing Poetic Captions For Shakespearizing modern English texts, we experimented with various types of sequence to sequence models. Since the size of the parallel translation data available is small, we leverage a dictionary providing a mapping between Shakespearean words and modern English words to retrofit pre-trained word embeddings. Incorporating this extra information improves the translation task. The large number of shared word types between the source and target sentences indicates that sharing the representation between them is beneficial. Methods ::: Shakespearizing Poetic Captions ::: Seq2Seq with Attention We use a sequence-to-sequence model which consists of a single layer unidrectional LSTM encoder and a single layer LSTM decoder and pre-trained retrofitted word embeddings shared between source and target sentences. We experimented with two different types of attention: global attention BIBREF9, in which the model makes use of the output from the encoder and decoder for the current time step only, and Bahdanau attention BIBREF10, where computing attention requires the output of the decoder from the prior time step. We found that global attention performs better in practice for our task of text style transfer. Methods ::: Shakespearizing Poetic Captions ::: Seq2Seq with a Pointer Network Since a pair of corresponding Shakespeare and modern English sentences have significant vocabulary overlap we extend the sequence-to-sequence model mentioned above using pointer networks BIBREF11 that provide location based attention and have been used to enable copying of tokens directly from the input. Moreover, there are lot of proper nouns and rare words which might not be predicted by a vanilla sequence to sequence model. Methods ::: Shakespearizing Poetic Captions ::: Prediction For both seq2seq models, we use the attention matrices returned at each decoder time step during inference, to compute the next word in the translated sequence if the decoder output at the current time step is the UNK token. We replace the UNKs in the target output with the highest aligned, maximum attention, source word. The seq2seq model with global attention gives the best results with an average target BLEU score of 29.65 on the style transfer dataset, compared with an average target BLEU score of 26.97 using the seq2seq model with pointer networks. Results We perform a qualitative analysis of the Shakespearean prose generated for the input paintings. We conducted a survey, in which we presented famous paintings including those shown in Figures FIGREF1 and FIGREF10 and the corresponding Shakespearean prose generated by the model, and asked 32 students to rate them on the basis of content, creativity and similarity to Shakespearean style on a Likert scale of 1-5. Figure FIGREF12 shows the result of our human evaluation. The average content score across the paintings is 3.7 which demonstrates that the prose generated is relevant to the painting. The average creativity score is 3.9 which demonstrates that the model captures more than basic objects in the painting successfully using poetic clues in the scene. The average style score is 3.9 which demonstrates that the prose generated is perceived to be in the style of Shakespeare. We also perform a quantitative analysis of style transfer by generating BLEU scores for the model output using the style transfer dataset. The variation of the BLEU scores with the source sentence lengths is shown in Figure FIGREF11. As expected, the BLEU scores decrease with increase in source sentence lengths. Results ::: Implementation All models were trained on Google Colab with a single GPU using Python 3.6 and Tensorflow 2.0. The number of hidden units for the encoder and decoder is 1,576 and 256 for seq2seq with global attention and seq2seq with pointer networks respectively. Adam optimizer was used with the default learning rate of 0.001. The model was trained for 25 epochs. We use pre-trained retrofitted word embeddings of dimension 192. Results ::: Limitations Since we do not have an end-to-end dataset, the generated English poem may not work well with Shakespeare style transfer as shown in Figure FIGREF12 for "Starry Night" with a low average content score. This happens when the style transfer dataset does not have similar words in the training set of sentences. A solution would be to expand the style transfer dataset, for a better representation of the poem data. Results ::: Conclusions and Future Work In conclusion, combining two pipelines with an intermediate representation works well in practice. We observe that a CNN-RNN based image-to-poem net combined with a seq2seq model with parallel text corpus for text style transfer synthesizes Shakespeare-style prose for a given painting. For the seq2seq model used, we observe that it performs better in practice using global attention as compared with local attention. We make our models and code publicly available BIBREF12. In future work we would like to experiment with GANs in the absence of non-parallel datasets, so that we can use varied styles for text style transfer. We would also like to experiment with cross aligned auto-encoders, which form a latent content representation, to efficiently separate style and content.
What models are used for painting embedding and what for language style transfer?
72e4e26d0dd79c590c28b10938952a9f9497ff1e
What Ġmodels Ġare Ġused Ġfor Ġpainting Ġembed ding Ġand Ġwhat Ġfor Ġlanguage Ġstyle Ġtransfer ?
gener ating Ġa Ġpoem Ġfrom Ġimages Ġwe Ġuse Ġan Ġexisting Ġactor - crit ic Ġarchitecture , Ġvarious Ġtypes Ġof Ġsequence Ġto Ġsequence Ġmodels
<s> What Ġmodels Ġare Ġused Ġfor Ġpainting Ġembed ding Ġand Ġwhat Ġfor Ġlanguage Ġstyle Ġtransfer ? </s> Introduction </s> Ne ural Ġnetworks Ġhave Ġbeen Ġsuccessfully Ġused Ġto Ġdescribe Ġimages Ġwith Ġtext Ġusing Ġsequence - to - sequence Ġmodels ĠB IB REF 0 . ĠHowever , Ġthe Ġresults Ġare Ġsimple Ġand Ġdry Ġcapt ions Ġwhich Ġare Ġone Ġor Ġtwo Ġphrases Ġlong . ĠHumans Ġlooking Ġat Ġa Ġpainting Ġsee Ġmore Ġthan Ġjust Ġobjects . ĠPaint ings Ġstimulate Ġsentiments , Ġmetaphors Ġand Ġstories Ġas Ġwell . ĠTherefore , Ġour Ġgoal Ġis Ġto Ġhave Ġa Ġneural Ġnetwork Ġdescribe Ġthe Ġpainting Ġartist ically Ġin Ġa Ġstyle Ġof Ġchoice . ĠAs Ġa Ġproof Ġof Ġconcept , Ġwe Ġpresent Ġa Ġmodel Ġwhich Ġgenerates ĠShakespeare an Ġprose Ġfor Ġa Ġgiven Ġpainting Ġas Ġshown Ġin ĠFigure ĠFIG REF 1 . ĠAcc ompl ishing Ġthis Ġtask Ġis Ġdifficult Ġwith Ġtraditional Ġsequence Ġto Ġsequence Ġmodels Ġsince Ġthere Ġdoes Ġnot Ġexist Ġa Ġlarge Ġcollection Ġof ĠShakespeare an Ġprose Ġwhich Ġdescribes Ġpaintings : ĠShakespeare 's Ġworks Ġdescribes Ġa Ġsingle Ġpainting Ġshown Ġin ĠFigure ĠFIG REF 3 . ĠFortunately Ġwe Ġhave Ġa Ġdataset Ġof Ġmodern ĠEnglish Ġpoems Ġwhich Ġdescribe Ġimages ĠB IB REF 1 Ġand Ġline - by - line Ġmodern Ġparaph r ases Ġof ĠShakespeare 's Ġplays ĠB IB REF 2 . ĠOur Ġsolution Ġis Ġtherefore Ġto Ġcombine Ġtwo Ġseparately Ġtrained Ġmodels Ġto Ġsynthes ize ĠShakespeare an Ġprose Ġfor Ġa Ġgiven Ġpainting . </s> Introduction Ġ: :: ĠRelated Ġwork </s> A Ġgeneral Ġend - to - end Ġapproach Ġto Ġsequence Ġlearning ĠB IB REF 3 Ġmakes Ġminimal Ġassumptions Ġon Ġthe Ġsequence Ġstructure . ĠThis Ġmodel Ġis Ġwidely Ġused Ġin Ġtasks Ġsuch Ġas Ġmachine Ġtranslation , Ġtext Ġsummar ization , Ġconvers ational Ġmodeling , Ġand Ġimage Ġcaption ing . ĠA Ġgener ative Ġmodel Ġusing Ġa Ġdeep Ġrecurrent Ġarchitecture ĠB IB REF 0 Ġhas Ġalso Ġbe een Ġused Ġfor Ġgenerating Ġphrases Ġdescribing Ġan Ġimage . ĠThe Ġtask Ġof Ġsynthes izing Ġmultiple Ġlines Ġof Ġpoetry Ġfor Ġa Ġgiven Ġimage ĠB IB REF 1 Ġis Ġaccomplished Ġby Ġextracting Ġpoetic Ġclues Ġfrom Ġimages . ĠGiven Ġthe Ġcontext Ġimage , Ġthe Ġnetwork Ġassociates Ġimage Ġattributes Ġwith Ġpoetic Ġdescriptions Ġusing Ġa Ġconv olution al Ġneural Ġnet . ĠThe Ġpoem Ġis Ġgenerated Ġusing Ġa Ġrecurrent Ġneural Ġnet Ġwhich Ġis Ġtrained Ġusing Ġmulti - ad vers arial Ġtraining Ġvia Ġpolicy Ġgradient . </s> Trans forming Ġtext Ġfrom Ġmodern ĠEnglish Ġto ĠShakespeare an ĠEnglish Ġusing Ġtext Ġ" style Ġtransfer " Ġis Ġchallenging . ĠAn Ġend Ġto Ġend Ġapproach Ġusing Ġa Ġsequence - to - sequence Ġmodel Ġover Ġa Ġparallel Ġtext Ġcorpus ĠB IB REF 2 Ġhas Ġbeen Ġproposed Ġbased Ġon Ġmachine Ġtranslation . ĠIn Ġthe Ġabsence Ġof Ġa Ġparallel Ġtext Ġcorpus , Ġgener ative Ġadvers arial Ġnetworks Ġ( GAN s ) Ġhave Ġbeen Ġused , Ġwhich Ġsimultaneously Ġtrain Ġtwo Ġmodels : Ġa Ġgener ative Ġmodel Ġwhich Ġcaptures Ġthe Ġdata Ġdistribution , Ġand Ġa Ġdiscrim in ative Ġmodel Ġwhich Ġevaluates Ġthe Ġperformance Ġof Ġthe Ġgenerator . ĠUsing Ġa Ġtarget Ġdomain Ġlanguage Ġmodel Ġas Ġa Ġdiscrim inator Ġhas Ġalso Ġbeen Ġemployed ĠB IB REF 4 , Ġproviding Ġricher Ġand Ġmore Ġstable Ġtoken - level Ġfeedback Ġduring Ġthe Ġlearning Ġprocess . ĠA Ġkey Ġchallenge Ġin Ġboth Ġimage Ġand Ġtext Ġstyle Ġtransfer Ġis Ġseparating Ġcontent Ġfrom Ġstyle ĠB IB REF 5 , ĠB IB REF 6 , ĠB IB REF 7 . ĠCross - aligned Ġauto - enc oder Ġmodels Ġhave Ġfocused Ġon Ġstyle Ġtransfer Ġusing Ġnon - par allel Ġtext ĠB IB REF 7 . ĠRecently , Ġa Ġfine Ġgr ained Ġmodel Ġfor Ġtext Ġstyle Ġtransfer Ġhas Ġbeen Ġproposed ĠB IB REF 8 Ġwhich Ġcontrols Ġseveral Ġfactors Ġof Ġvariation Ġin Ġtextual Ġdata Ġby Ġusing Ġback - translation . ĠThis Ġallows Ġcontrol Ġover Ġmultiple Ġattributes , Ġsuch Ġas Ġgender Ġand Ġsentiment , Ġand Ġfine - gr ained Ġcontrol Ġover Ġthe Ġtrade - off Ġbetween Ġcontent Ġand Ġstyle . </s> Methods </s> We Ġuse Ġa Ġtotal Ġthree Ġdatasets : Ġtwo Ġdatasets Ġfor Ġgenerating Ġan ĠEnglish Ġpoem Ġfrom Ġan Ġimage , Ġand ĠShakespeare Ġplays Ġand Ġtheir ĠEnglish Ġtranslations Ġfor Ġtext Ġstyle Ġtransfer . </s> We Ġtrain Ġa Ġmodel Ġfor Ġgenerating Ġpoems Ġfrom Ġimages Ġbased Ġon Ġtwo Ġdatasets ĠB IB REF 1 . ĠThe Ġfirst Ġdataset Ġconsists Ġof Ġimage Ġand Ġpoem Ġpairs , Ġnamely Ġa Ġmulti - mod al Ġpoem Ġdataset Ġ( Multi M - Po em ), Ġand Ġthe Ġsecond Ġdataset Ġis Ġa Ġlarge Ġpoem Ġcorpus , Ġnamely Ġa Ġun i - mod al Ġpoem Ġdataset Ġ( Un i M - Po em ). ĠThe Ġimage Ġand Ġpoem Ġpairs Ġare Ġextended Ġby Ġadding Ġthe Ġnearest Ġthree Ġneighbor Ġpoems Ġfrom Ġthe Ġpoem Ġcorpus Ġwithout Ġredundancy , Ġand Ġan Ġextended Ġimage Ġand Ġpoem Ġpair Ġdataset Ġis Ġconstructed Ġand Ġden oted Ġas ĠMulti M - Po em ( Ex ) BI B REF 1 . </s> We Ġuse Ġa Ġcollection Ġof Ġline - by - line Ġmodern Ġparaph r ases Ġfor Ġ16 Ġof ĠShakespeare âĢ Ļ s Ġplays ĠB IB REF 2 , Ġfor Ġtraining Ġa Ġstyle Ġtransfer Ġnetwork Ġfrom ĠEnglish Ġpoems Ġto ĠShakespeare an Ġprose . ĠWe Ġuse Ġ18 , 395 Ġsentences Ġfrom Ġthe Ġtraining Ġdata Ġsplit . ĠWe Ġkeep Ġ1 , 218 Ġsentences Ġin Ġthe Ġvalidation Ġdata Ġset Ġand Ġ1 , 462 Ġsentences Ġin Ġour Ġtest Ġset . </s> Methods Ġ: :: ĠImage ĠTo ĠPo em ĠActor - Crit ic ĠModel </s> For Ġgenerating Ġa Ġpoem Ġfrom Ġimages Ġwe Ġuse Ġan Ġexisting Ġactor - crit ic Ġarchitecture ĠB IB REF 1 . ĠThis Ġinvolves Ġ3 Ġparallel ĠCNN s : Ġan Ġobject ĠCNN , Ġsentiment ĠCNN , Ġand Ġscene ĠCNN , Ġfor Ġfeature Ġextraction . ĠThese Ġfeatures Ġare Ġcombined Ġwith Ġa Ġskip - thought Ġmodel Ġwhich Ġprovides Ġpoetic Ġclues , Ġwhich Ġare Ġthen Ġfed Ġinto Ġa Ġsequence - to - sequence Ġmodel Ġtrained Ġby Ġpolicy Ġgradient Ġwith Ġ2 Ġdiscrim inator Ġnetworks Ġfor Ġrewards . ĠThis Ġas Ġa Ġwhole Ġforms Ġa Ġpipeline Ġthat Ġtakes Ġin Ġan Ġimage Ġand Ġoutputs Ġa Ġpoem Ġas Ġshown Ġon Ġthe Ġtop Ġleft Ġof ĠFigure ĠFIG REF 4 . ĠA ĠCNN - R NN Ġgener ative Ġmodel Ġacts Ġas Ġan Ġagent . ĠThe Ġparameters Ġof Ġthis Ġagent Ġdefine Ġa Ġpolicy Ġwhose Ġexecution Ġdetermines Ġwhich Ġword Ġis Ġselected Ġas Ġan Ġaction . ĠWhen Ġthe Ġagent Ġselects Ġall Ġwords Ġin Ġa Ġpoem , Ġit Ġreceives Ġa Ġreward . ĠTwo Ġdiscrim in ative Ġnetworks , Ġshown Ġon Ġthe Ġtop Ġright Ġof ĠFigure ĠFIG REF 4 , Ġare Ġdefined Ġto Ġserve Ġas Ġrewards Ġconcerning Ġwhether Ġthe Ġgenerated Ġpoem Ġproperly Ġdescribes Ġthe Ġinput Ġimage Ġand Ġwhether Ġthe Ġgenerated Ġpoem Ġis Ġpoetic . ĠThe Ġgoal Ġof Ġthe Ġpoem Ġgeneration Ġmodel Ġis Ġto Ġgenerate Ġa Ġsequence Ġof Ġwords Ġas Ġa Ġpoem Ġfor Ġan Ġimage Ġto Ġmaximize Ġthe Ġexpected Ġreturn . </s> Methods Ġ: :: ĠSh akespe ar izing ĠPo etic ĠCapt ions </s> For ĠSh akespe ar izing Ġmodern ĠEnglish Ġtexts , Ġwe Ġexperimented Ġwith Ġvarious Ġtypes Ġof Ġsequence Ġto Ġsequence Ġmodels . ĠSince Ġthe Ġsize Ġof Ġthe Ġparallel Ġtranslation Ġdata Ġavailable Ġis Ġsmall , Ġwe Ġleverage Ġa Ġdictionary Ġproviding Ġa Ġmapping Ġbetween ĠShakespeare an Ġwords Ġand Ġmodern ĠEnglish Ġwords Ġto Ġretro fit Ġpre - trained Ġword Ġembed d ings . ĠInc orpor ating Ġthis Ġextra Ġinformation Ġimproves Ġthe Ġtranslation Ġtask . ĠThe Ġlarge Ġnumber Ġof Ġshared Ġword Ġtypes Ġbetween Ġthe Ġsource Ġand Ġtarget Ġsentences Ġindicates Ġthat Ġsharing Ġthe Ġrepresentation Ġbetween Ġthem Ġis Ġbeneficial . </s> Methods Ġ: :: ĠSh akespe ar izing ĠPo etic ĠCapt ions Ġ: :: ĠSe q 2 Se q Ġwith ĠAttention </s> We Ġuse Ġa Ġsequence - to - sequence Ġmodel Ġwhich Ġconsists Ġof Ġa Ġsingle Ġlayer Ġun id rection al ĠL ST M Ġenc oder Ġand Ġa Ġsingle Ġlayer ĠL ST M Ġdec oder Ġand Ġpre - trained Ġretro fitted Ġword Ġembed d ings Ġshared Ġbetween Ġsource Ġand Ġtarget Ġsentences . ĠWe Ġexperimented Ġwith Ġtwo Ġdifferent Ġtypes Ġof Ġattention : Ġglobal Ġattention ĠB IB REF 9 , Ġin Ġwhich Ġthe Ġmodel Ġmakes Ġuse Ġof Ġthe Ġoutput Ġfrom Ġthe Ġenc oder Ġand Ġdec oder Ġfor Ġthe Ġcurrent Ġtime Ġstep Ġonly , Ġand ĠBah dan au Ġattention ĠB IB REF 10 , Ġwhere Ġcomputing Ġattention Ġrequires Ġthe Ġoutput Ġof Ġthe Ġdec oder Ġfrom Ġthe Ġprior Ġtime Ġstep . ĠWe Ġfound Ġthat Ġglobal Ġattention Ġperforms Ġbetter Ġin Ġpractice Ġfor Ġour Ġtask Ġof Ġtext Ġstyle Ġtransfer . </s> Methods Ġ: :: ĠSh akespe ar izing ĠPo etic ĠCapt ions Ġ: :: ĠSe q 2 Se q Ġwith Ġa ĠPo inter ĠNetwork </s> Since Ġa Ġpair Ġof Ġcorresponding ĠShakespeare Ġand Ġmodern ĠEnglish Ġsentences Ġhave Ġsignificant Ġvocabulary Ġoverlap Ġwe Ġextend Ġthe Ġsequence - to - sequence Ġmodel Ġmentioned Ġabove Ġusing Ġpointer Ġnetworks ĠB IB REF 11 Ġthat Ġprovide Ġlocation Ġbased Ġattention Ġand Ġhave Ġbeen Ġused Ġto Ġenable Ġcopying Ġof Ġtokens Ġdirectly Ġfrom Ġthe Ġinput . ĠMoreover , Ġthere Ġare Ġlot Ġof Ġproper Ġnoun s Ġand Ġrare Ġwords Ġwhich Ġmight Ġnot Ġbe Ġpredicted Ġby Ġa Ġvanilla Ġsequence Ġto Ġsequence Ġmodel . </s> Methods Ġ: :: ĠSh akespe ar izing ĠPo etic ĠCapt ions Ġ: :: ĠPrediction </s> For Ġboth Ġseq 2 seq Ġmodels , Ġwe Ġuse Ġthe Ġattention Ġmat rices Ġreturned Ġat Ġeach Ġdec oder Ġtime Ġstep Ġduring Ġinference , Ġto Ġcompute Ġthe Ġnext Ġword Ġin Ġthe Ġtranslated Ġsequence Ġif Ġthe Ġdec oder Ġoutput Ġat Ġthe Ġcurrent Ġtime Ġstep Ġis Ġthe ĠUN K Ġtoken . ĠWe Ġreplace Ġthe ĠUN K s Ġin Ġthe Ġtarget Ġoutput Ġwith Ġthe Ġhighest Ġaligned , Ġmaximum Ġattention , Ġsource Ġword . ĠThe Ġseq 2 seq Ġmodel Ġwith Ġglobal Ġattention Ġgives Ġthe Ġbest Ġresults Ġwith Ġan Ġaverage Ġtarget ĠB LE U Ġscore Ġof Ġ29 . 65 Ġon Ġthe Ġstyle Ġtransfer Ġdataset , Ġcompared Ġwith Ġan Ġaverage Ġtarget ĠB LE U Ġscore Ġof Ġ26 . 97 Ġusing Ġthe Ġseq 2 seq Ġmodel Ġwith Ġpointer Ġnetworks . </s> Results </s> We Ġperform Ġa Ġqualitative Ġanalysis Ġof Ġthe ĠShakespeare an Ġprose Ġgenerated Ġfor Ġthe Ġinput Ġpaintings . ĠWe Ġconducted Ġa Ġsurvey , Ġin Ġwhich Ġwe Ġpresented Ġfamous Ġpaintings Ġincluding Ġthose Ġshown Ġin ĠFigures ĠFIG REF 1 Ġand ĠFIG REF 10 Ġand Ġthe Ġcorresponding ĠShakespeare an Ġprose Ġgenerated Ġby Ġthe Ġmodel , Ġand Ġasked Ġ32 Ġstudents Ġto Ġrate Ġthem Ġon Ġthe Ġbasis Ġof Ġcontent , Ġcreativity Ġand Ġsimilarity Ġto ĠShakespeare an Ġstyle Ġon Ġa ĠL ik ert Ġscale Ġof Ġ1 - 5 . ĠFigure ĠFIG REF 12 Ġshows Ġthe Ġresult Ġof Ġour Ġhuman Ġevaluation . </s> The Ġaverage Ġcontent Ġscore Ġacross Ġthe Ġpaintings Ġis Ġ3 . 7 Ġwhich Ġdemonstrates Ġthat Ġthe Ġprose Ġgenerated Ġis Ġrelevant Ġto Ġthe Ġpainting . ĠThe Ġaverage Ġcreativity Ġscore Ġis Ġ3 . 9 Ġwhich Ġdemonstrates Ġthat Ġthe Ġmodel Ġcaptures Ġmore Ġthan Ġbasic Ġobjects Ġin Ġthe Ġpainting Ġsuccessfully Ġusing Ġpoetic Ġclues Ġin Ġthe Ġscene . ĠThe Ġaverage Ġstyle Ġscore Ġis Ġ3 . 9 Ġwhich Ġdemonstrates Ġthat Ġthe Ġprose Ġgenerated Ġis Ġperceived Ġto Ġbe Ġin Ġthe Ġstyle Ġof ĠShakespeare . </s> We Ġalso Ġperform Ġa Ġquantitative Ġanalysis Ġof Ġstyle Ġtransfer Ġby Ġgenerating ĠB LE U Ġscores Ġfor Ġthe Ġmodel Ġoutput Ġusing Ġthe Ġstyle Ġtransfer Ġdataset . ĠThe Ġvariation Ġof Ġthe ĠB LE U Ġscores Ġwith Ġthe Ġsource Ġsentence Ġlengths Ġis Ġshown Ġin ĠFigure ĠFIG REF 11 . ĠAs Ġexpected , Ġthe ĠB LE U Ġscores Ġdecrease Ġwith Ġincrease Ġin Ġsource Ġsentence Ġlengths . </s> Results Ġ: :: ĠImplementation </s> All Ġmodels Ġwere Ġtrained Ġon ĠGoogle ĠCol ab Ġwith Ġa Ġsingle ĠGPU Ġusing ĠPython Ġ3 . 6 Ġand ĠT ensor flow Ġ2 . 0 . ĠThe Ġnumber Ġof Ġhidden Ġunits Ġfor Ġthe Ġenc oder Ġand Ġdec oder Ġis Ġ1 , 576 Ġand Ġ256 Ġfor Ġseq 2 seq Ġwith Ġglobal Ġattention Ġand Ġseq 2 seq Ġwith Ġpointer Ġnetworks Ġrespectively . ĠAdam Ġoptim izer Ġwas Ġused Ġwith Ġthe Ġdefault Ġlearning Ġrate Ġof Ġ0 . 001 . ĠThe Ġmodel Ġwas Ġtrained Ġfor Ġ25 Ġepoch s . ĠWe Ġuse Ġpre - trained Ġretro fitted Ġword Ġembed d ings Ġof Ġdimension Ġ192 . </s> Results Ġ: :: ĠLim itations </s> Since Ġwe Ġdo Ġnot Ġhave Ġan Ġend - to - end Ġdataset , Ġthe Ġgenerated ĠEnglish Ġpoem Ġmay Ġnot Ġwork Ġwell Ġwith ĠShakespeare Ġstyle Ġtransfer Ġas Ġshown Ġin ĠFigure ĠFIG REF 12 Ġfor Ġ" St arry ĠNight " Ġwith Ġa Ġlow Ġaverage Ġcontent Ġscore . ĠThis Ġhappens Ġwhen Ġthe Ġstyle Ġtransfer Ġdataset Ġdoes Ġnot Ġhave Ġsimilar Ġwords Ġin Ġthe Ġtraining Ġset Ġof Ġsentences . ĠA Ġsolution Ġwould Ġbe Ġto Ġexpand Ġthe Ġstyle Ġtransfer Ġdataset , Ġfor Ġa Ġbetter Ġrepresentation Ġof Ġthe Ġpoem Ġdata . </s> Results Ġ: :: ĠCon clusions Ġand ĠFuture ĠWork </s> In Ġconclusion , Ġcombining Ġtwo Ġpipelines Ġwith Ġan Ġintermediate Ġrepresentation Ġworks Ġwell Ġin Ġpractice . ĠWe Ġobserve Ġthat Ġa ĠCNN - R NN Ġbased Ġimage - to - po em Ġnet Ġcombined Ġwith Ġa Ġseq 2 seq Ġmodel Ġwith Ġparallel Ġtext Ġcorpus Ġfor Ġtext Ġstyle Ġtransfer Ġsynthes izes ĠShakespeare - style Ġprose Ġfor Ġa Ġgiven Ġpainting . ĠFor Ġthe Ġseq 2 seq Ġmodel Ġused , Ġwe Ġobserve Ġthat Ġit Ġperforms Ġbetter Ġin Ġpractice Ġusing Ġglobal Ġattention Ġas Ġcompared Ġwith Ġlocal Ġattention . ĠWe Ġmake Ġour Ġmodels Ġand Ġcode Ġpublicly Ġavailable ĠB IB REF 12 . ĠIn Ġfuture Ġwork Ġwe Ġwould Ġlike Ġto Ġexperiment Ġwith ĠG AN s Ġin Ġthe Ġabsence Ġof Ġnon - par allel Ġdatasets , Ġso Ġthat Ġwe Ġcan Ġuse Ġvaried Ġstyles Ġfor Ġtext Ġstyle Ġtransfer . ĠWe Ġwould Ġalso Ġlike Ġto Ġexperiment Ġwith Ġcross Ġaligned Ġauto - enc od ers , Ġwhich Ġform Ġa Ġlatent Ġcontent Ġrepresentation , Ġto Ġefficiently Ġseparate Ġstyle Ġand Ġcontent .