To compare the conversions between USVC and PitchNet, we employed an automatic evaluation score and a human evaluation score.
EXTRACTIVE
Automatic: Normalized cross correlation (NCC)
Manual: Mean Opinion Score (MOS)
ABSTRACTIVE
To compare the conversions between USVC and PitchNet, we employed an automatic evaluation score and a human evaluation score. The automatic score roughly followed the design in BIBREF13. The pitch tracker of librosa package BIBREF18 was employed to extract pitch information of the input and output audio. Then the output pitch was compared to the input pitch using the normalized cross correlation (NCC) which would give a score between 0 and 1. The higher the score is, the better the output pitch matches the input pitch. We conducted the evaluation on USVC (our) and PitchNet. The evaluated automatic scores on conversion and reconstruction tasks are shown in Tab. TABREF14. Our method performed better both on conversion and reconstruction. The scores of reconstruction are higher than conversion since both models were trained using a reconstruction loss. However, the score of our method on conversion is even higher than the score of USVC (Our) on reconstruction. Mean Opinion Score (MOS) was used as a subjective metric to evaluate the quality of the converted audio. Two questions were asked: (1) what is quality of the audio? (naturalness) (2) How well does the converted version match the original? (similarity) A score of 1-5 would be given to answer the questions. The evaluation was conducted on USVC (Our) and PitchNet. Besides, the converted samples provided by BIBREF0 was also included to give a more convincing evaluation. As shown by Tab. TABREF15, the naturalness and similarity of our method are both higher than the other two ones. Our implementation of USVC performed slightly lower than the original author's because we cannot fully reproduce the results of them. Next we qualitatively analyze the influence of input pitch in our method. We used different pitch as input to observe how the output pitch would change along with the input pitch. The input pitch was multiplied by 0.7, 1.0 and 1.3 respectively. And the output pitch was also extracted by the pitch tracker of the librosa package. Fig. FIGREF16 plots the pitch of input audio and output audio with different pitch as input while keeping the target singer the same. As shown by Fig. FIGREF16, the output pitch changes significantly along with the input pitch. The examples are also presented at our website.
To compare the conversions between USVC and PitchNet, we employed an automatic evaluation score and a human evaluation score. The automatic score roughly followed the design in BIBREF13. The pitch tracker of librosa package BIBREF18 was employed to extract pitch information of the input and output audio. Then the output pitch was compared to the input pitch using the normalized cross correlation (NCC) which would give a score between 0 and 1. The higher the score is, the better the output pitch matches the input pitch. We conducted the evaluation on USVC (our) and PitchNet. The evaluated automatic scores on conversion and reconstruction tasks are shown in Tab. TABREF14. Our method performed better both on conversion and reconstruction. The scores of reconstruction are higher than conversion since both models were trained using a reconstruction loss. However, the score of our method on conversion is even higher than the score of USVC (Our) on reconstruction. Mean Opinion Score (MOS) was used as a subjective metric to evaluate the quality of the converted audio. Two questions were asked: (1) what is quality of the audio? (naturalness) (2) How well does the converted version match the original? (similarity) A score of 1-5 would be given to answer the questions. The evaluation was conducted on USVC (Our) and PitchNet. Besides, the converted samples provided by BIBREF0 was also included to give a more convincing evaluation. As shown by Tab. TABREF15, the naturalness and similarity of our method are both higher than the other two ones. Our implementation of USVC performed slightly lower than the original author's because we cannot fully reproduce the results of them.
1912.01852
Introduction </s> S inging Ġis Ġan Ġimportant Ġway Ġof Ġhuman Ġexpression Ġand Ġthe Ġtechniques Ġof Ġsinging Ġsynthesis Ġhave Ġbroad Ġapplications Ġin Ġdifferent Ġprospects Ġincluding Ġvirtual Ġhuman , Ġmovie Ġdub bing Ġand Ġso Ġon . ĠTraditional Ġsinging Ġsynthes ize Ġsystems Ġare Ġbased Ġon Ġconc aten ative ĠB IB REF 1 Ġor ĠH MM ĠB IB REF 2 Ġbased Ġapproaches . ĠWith Ġthe Ġsuccess Ġof Ġdeep Ġlearning Ġin ĠText - to - Spe ech , Ġsome Ġneural Ġsinging Ġsynthesis Ġmethods Ġhave Ġalso Ġbeen Ġproposed Ġrecently . ĠFor Ġexample , ĠB IB REF 3 Ġintroduces Ġa Ġsinging Ġsynthesis Ġmethod Ġusing Ġan Ġarchitecture Ġsimilar Ġto ĠWave Net ĠB IB REF 4 . ĠIt Ġadop ts Ġlyrics Ġand Ġnotes Ġas Ġinput Ġand Ġgenerates Ġvoc oder Ġfeatures Ġaut ore gress ively Ġfor Ġfinal Ġsinging Ġvoice Ġsynthesis . </s> S inging Ġvoice Ġconversion Ġis Ġanother Ġway Ġof Ġsinging Ġsynthesis Ġwhich Ġextracts Ġmusical Ġexpression Ġwithin Ġexisting Ġsinging Ġand Ġreproduced Ġthem Ġwith Ġanother Ġsinger 's Ġvoice . ĠIt Ġis Ġvery Ġsimilar Ġto Ġspeech Ġbased Ġvoice Ġconversion ĠB IB REF 5 , ĠB IB REF 6 , ĠB IB REF 7 , ĠB IB REF 8 , Ġbut Ġcompared Ġwith Ġspeech Ġvoice Ġconversion , Ġsinging Ġvoice Ġconversion Ġneeds Ġto Ġdeal Ġwith Ġa Ġwider Ġrange Ġof Ġfrequency Ġvariations Ġas Ġwell Ġas Ġthe Ġsharper Ġchange Ġof Ġvolume Ġand Ġpitch Ġwithin Ġsinging Ġvoice . ĠThe Ġperformance Ġof Ġsinging Ġconversion Ġis Ġhighly Ġdependent Ġon Ġthe Ġmusical Ġexpression Ġof Ġconverted Ġsinging Ġand Ġthe Ġsimilarity Ġof Ġthe Ġconverted Ġvoice Ġtim bre Ġcompared Ġto Ġtarget Ġsinger 's Ġvoice . ĠThere Ġare Ġseveral Ġsinging Ġvoice Ġconversion Ġmethods Ġto Ġconvert Ġone 's Ġsinging Ġvoice Ġto Ġanother ĠB IB REF 9 , ĠB IB REF 10 , ĠB IB REF 11 . ĠThey Ġgenerally Ġrequire Ġparallel Ġdata Ġto Ġtrain Ġthe Ġconversion Ġmodel . ĠTo Ġovercome Ġthe Ġlimitation Ġof Ġthe Ġparallel Ġtraining Ġdata Ġfor Ġsinging Ġvoice Ġconversion , Ġan Ġun super vised Ġmethod ĠB IB REF 0 Ġhas Ġbeen Ġproposed Ġto Ġutilize Ġnon - par allel Ġdata . ĠThis Ġmethod Ġemploys Ġan Ġaut oen c oder Ġarchitecture Ġcomposed Ġof Ġa ĠWave Net - like Ġenc oder , Ġa ĠWave Net ĠB IB REF 4 Ġaut ore gressive Ġdec oder , Ġand Ġa Ġlearn able Ġsinger Ġembed ding Ġtable . ĠVoice Ġwave form Ġis Ġpassed Ġinto Ġthe Ġenc oder Ġand Ġthe Ġoutput Ġof Ġthe Ġenc oder Ġwill Ġbe Ġconc aten ated Ġwith Ġthe Ġembed ding Ġvector Ġassociated Ġwith Ġthe Ġsinger . ĠThe Ġconc aten ated Ġfeatures Ġwill Ġbe Ġused Ġto Ġcondition Ġthe ĠWave Net Ġdec oder Ġto Ġreconstruct Ġthe Ġinput Ġaudio . ĠA Ġconfusion Ġloss ĠB IB REF 12 Ġis Ġalso Ġintroduced Ġto Ġforce Ġthe Ġenc oder Ġto Ġlearn Ġa Ġsinger - inv ari ant Ġrepresentation . ĠBy Ġswitching Ġbetween Ġembed d ings Ġof Ġdifferent Ġsingers Ġduring Ġgeneration , Ġthe Ġsinging Ġvoice Ġconversion Ġcan Ġbe Ġachieved . ĠWhile Ġthis Ġapproach Ġcould Ġgenerates Ġsinging Ġvoice Ġpercept ually Ġsimilar Ġto Ġtarget Ġsinger , Ġthe Ġquality Ġof Ġgenerated Ġsinging Ġoften Ġsuffers Ġdue Ġto Ġthe Ġdifficulty Ġon Ġlearning Ġjoint Ġrepresentation Ġof Ġphon etic Ġand Ġpitch Ġrepresentation . </s> To Ġaddress Ġthe Ġdifficulty Ġof Ġlearning Ġjoin Ġphon etic Ġand Ġpitch Ġrepresentation Ġin ĠB IB REF 0 , Ġwe Ġpropose Ġto Ġuse Ġadvers ar ially Ġtrained Ġpitch Ġregression Ġnetwork Ġto Ġencourage Ġthe Ġenc oder Ġnetwork Ġto Ġlearn Ġnot Ġonly Ġsinger - inv ari ant Ġbut Ġalso Ġpitch - inv ari ant Ġrepresentation , Ġat Ġthe Ġsame Ġtime Ġextract Ġthe Ġpitch Ġfrom Ġsource Ġaudio Ġto Ġbe Ġused Ġas Ġan Ġadditional Ġinput Ġto Ġthe Ġdec oder . ĠThe Ġproposed Ġmethod Ġcan Ġgreatly Ġimprove Ġthe Ġquality Ġof Ġthe Ġconverted Ġvoice Ġand Ġachieve Ġflexible Ġpitch Ġmanipulation Ġat Ġthe Ġsame Ġtime . </s> In Ġthe Ġfollowing Ġsections , Ġwe Ġwill Ġintroduce Ġour Ġproposed Ġmethod Ġin Ġsection ĠSEC REF 2 . ĠAnd Ġthen Ġsection ĠSEC REF 3 Ġwill Ġshow Ġthat Ġour Ġmethod Ġis Ġeffective Ġby Ġquantitative Ġand Ġqualitative Ġexperiments . ĠFinally , Ġwe Ġwill Ġconclude Ġin Ġsection ĠSEC REF 4 Ġand Ġacknowled gements Ġare Ġin ĠSEC REF 5 . </s> Method </s> Our Ġmethod Ġfollows Ġthe Ġaut oen c oder Ġarchitecture Ġin ĠB IB REF 0 Ġexcept Ġthat Ġthere Ġis Ġan Ġadditional Ġpitch Ġregression Ġnetwork Ġto Ġseparate Ġpitch Ġinformation Ġout Ġof Ġthe Ġlatent Ġspace . ĠThe Ġarchitecture Ġof ĠPitch Net Ġis Ġillustrated Ġin ĠFig . ĠFIG REF 1 . ĠIt Ġconsists Ġof Ġfive Ġparts , Ġan Ġenc oder , Ġa Ġdec oder , Ġa ĠLook ĠUp ĠTable Ġ( L UT ) Ġof Ġspeaker Ġembed ding Ġvectors , Ġa Ġsinger Ġclassification Ġnetwork , Ġand Ġa Ġpitch Ġregression Ġnetwork . </s> First , Ġthe Ġinput Ġwave form Ġis Ġpassed Ġthrough Ġthe Ġenc oder Ġto Ġextract Ġhigh - level Ġsemantic Ġfeatures . ĠAn Ġaverage Ġpool ing Ġof Ġstride Ġ800 Ġis Ġthen Ġapplied Ġto Ġthe Ġfeatures , Ġforming Ġa Ġbottleneck Ġto Ġlimit Ġthe Ġinformation Ġpassing Ġthrough Ġthe Ġenc oder . ĠAfter Ġthat , Ġa Ġsinger Ġid Ġis Ġused Ġto Ġretrieve Ġthe Ġtarget Ġsinger 's Ġembed ding Ġvector Ġfrom ĠL UT , Ġand Ġconc aten ated Ġwith Ġthe Ġoutput Ġof Ġthe Ġenc oder Ġat Ġeach Ġtime Ġstep Ġto Ġbe Ġa Ġsequence Ġof Ġcondition Ġvectors . ĠThe Ġpitch Ġof Ġthe Ġinput Ġaudio , Ġextracted Ġseparately Ġfrom Ġthe Ġnetwork , Ġis Ġfed Ġinto Ġthe Ġdec oder Ġafter Ġa Ġlinear Ġinterpol ation Ġas Ġa Ġcompensation Ġsignal Ġtogether Ġwith Ġthe Ġcondition Ġvector . ĠFinally , Ġthe Ġdec oder Ġis Ġconditioned Ġon Ġthe Ġcondition Ġvector Ġand Ġpitch Ġto Ġgenerate Ġaudio Ġsamples . ĠSince Ġthe Ġdec oder Ġis Ġan Ġaut ore gressive Ġmodel , Ġthe Ġoutput Ġwill Ġbe Ġfed Ġback Ġto Ġthe Ġdec oder Ġat Ġthe Ġnext Ġtime Ġstep . ĠThe Ġmodel Ġis Ġtrained Ġon Ġa Ġsoft max - based Ġloss Ġto Ġminimize Ġthe Ġreconstruction Ġerror Ġwith Ġteacher - forcing . </s> In Ġorder Ġto Ġproject Ġthe Ġoutput Ġfeatures Ġof Ġthe Ġenc oder Ġinto Ġa Ġsinger Ġand Ġpitch Ġinvari ant Ġlatent Ġspace , Ġa Ġsinger Ġclassification Ġnetwork Ġand Ġa Ġpitch Ġregression Ġnetwork Ġare Ġemployed Ġto Ġforce Ġthe Ġenc oder Ġnot Ġto Ġencode Ġsinger Ġand Ġpitch Ġinformation . ĠThe Ġsinger Ġclassification Ġloss Ġand Ġpitch Ġregression Ġloss Ġare Ġadded Ġadvers ar ially Ġto Ġthe Ġreconstruction Ġloss Ġto Ġtrain Ġthe Ġentire Ġmodel Ġend Ġto Ġend . </s> Method Ġ: :: ĠTraining ĠLoss </s> To Ġformally Ġdescribe Ġthe Ġmodel , Ġlet Ġ$ E $ Ġbe Ġthe Ġenc oder Ġnetwork , Ġ$ D $ Ġbe Ġthe Ġdec oder Ġnetwork , Ġ$ C _ s $ Ġbe Ġthe Ġsinger Ġclassification Ġnetwork Ġand Ġ$ C _ p $ Ġbe Ġthe Ġpitch Ġregression Ġnetwork . ĠLet Ġ$ v _ j $ Ġdenote Ġthe Ġembed ding Ġvector Ġof Ġsinger Ġ$ j $, Ġ$ s ^ j $ Ġdenote Ġan Ġinput Ġaudio Ġof Ġsinger Ġ$ j $ Ġand Ġ$ p ( s ^ j ) $ Ġdenote Ġthe Ġextracted Ġpitch Ġof Ġ$ s ^ j $. ĠNow Ġgiven Ġan Ġinput Ġaudio Ġsequence Ġ$ s ^ j $ Ġand Ġa Ġtarget Ġsinger Ġ$ k $ Ġwhere Ġ$ j , k Ġ= Ġ1 , 2 ,... , N $ Ġand Ġ$ N $ Ġis Ġthe Ġnumber Ġof Ġsingers , Ġthe Ġoutput Ġof Ġthe Ġmodel Ġis </s> Note Ġthat Ġ$ D $ Ġis Ġan Ġaut ore gressive Ġmodel Ġwhich Ġwould Ġfeed Ġthe Ġoutput Ġback Ġto Ġitself . ĠThe Ġreconstruction Ġloss Ġis </s> where Ġ$\ math cal Ġ{ L } _{ ce } ( o , Ġy ) $ Ġis Ġthe Ġcross Ġentropy Ġloss Ġapplied Ġto Ġeach Ġelement Ġof Ġ$ o $ Ġand Ġ$ y $. ĠHowever , Ġonly Ġreconstruction Ġloss Ġis Ġnot Ġenough Ġto Ġtrain Ġthe Ġmodel Ġto Ġlearn Ġto Ġconvert Ġsinging Ġvoice Ġbetween Ġdifferent Ġsingers Ġbecause Ġit Ġjust Ġforces Ġthe Ġmodel Ġto Ġreconstruct Ġthe Ġinput Ġvoice . ĠTherefore , Ġa Ġsinger Ġclassification Ġloss ( also Ġnamed Ġdomain Ġconfusion Ġloss ĠB IB REF 0 ) Ġis Ġapplied Ġto Ġmake Ġthe Ġenc oder Ġto Ġlearn Ġa Ġsinger Ġinvari ant Ġrepresentation </s> Furthermore , Ġa Ġpitch Ġregression Ġloss Ġis Ġintroduced Ġto Ġforce Ġthe Ġenc oder Ġto Ġlearn Ġa Ġpitch - independent Ġrepresentation Ġand Ġmake Ġthe Ġwhole Ġmodel Ġobtain Ġthe Ġpitch Ġinformation Ġfrom Ġ$ p ( s ^ j ) $ Ġrather Ġthan Ġdirectly Ġfrom Ġthe Ġinput Ġaudio </s> where Ġ$\ math cal Ġ{ L } _{ m se } ( a , Ġb ) $ Ġis Ġthe Ġmean Ġsquare Ġerror Ġfunction Ġ$\ frac { 1 }{ m } || a - b || _ 2 ^ 2 $ Ġand Ġm Ġis Ġthe Ġnumber Ġof Ġelements Ġin Ġ$ a $. ĠThe Ġoverall Ġloss Ġwe Ġminimize Ġto Ġtrain Ġthe Ġmodel Ġis </s> where Ġ$\ lambda Ġ$ Ġand Ġ$\ mu Ġ$ Ġare Ġtwo Ġweight Ġfactors . ĠFurthermore , Ġthe Ġadvers arial Ġloss Ġused Ġto Ġtrain Ġthe Ġsinger Ġclass ifier Ġand Ġpitch Ġregression Ġnetwork Ġis </s> In Ġthe Ġtraining Ġprocess , Ġwe Ġminimize Ġ$\ math cal Ġ{ L } _{ ad } $ Ġand Ġ$\ math cal Ġ{ L } _{ total } $ Ġaltern ately , Ġthat Ġis </s> Opt imize Ġ$ C _ s $ Ġand Ġ$ C _ p $ Ġone Ġstep Ġusing Ġ$\ math cal Ġ{ L } _{ ad } $ Ġas Ġthe Ġobjective Ġfunction . </s> Opt imize Ġthe Ġwhole Ġmodel Ġone Ġstep Ġusing Ġ$\ math cal Ġ{ L } _{ total } $ Ġas Ġthe Ġobjective Ġfunction . </s> Go Ġback Ġto Ġstep Ġ1 . </s> Furthermore , Ġback translation Ġand Ġmix up Ġtechniques ĠB IB REF 0 Ġare Ġalso Ġused Ġto Ġimprove Ġthe Ġquality Ġof Ġthe Ġconverted Ġsinging Ġvoice . </s> Method Ġ: :: ĠThe Ġarchitecture Ġof Ġthe ĠSub - Net works </s> The Ġenc oder Ġand Ġdec oder Ġnetworks Ġfollow Ġthe Ġdesign Ġin ĠB IB REF 13 . ĠThe Ġenc oder Ġis Ġa Ġfully Ġconv olution al Ġnetwork Ġwith Ġthree Ġblocks Ġof Ġten Ġresidual - l ayers Ġwhich Ġconsists Ġof Ġa ĠRe LU Ġactivation , Ġa Ġdil ated Ġconv olution , Ġa ĠRe LU Ġactivation , Ġa Ġ1 x 1 Ġconv olution , Ġand Ġa Ġresidual Ġsumm ation Ġin Ġorder . ĠAfter Ġthree Ġresidual Ġblocks , Ġa Ġ1 x 1 Ġconv olution Ġand Ġan Ġaverage Ġpool ing Ġwith Ġa Ġkernel Ġsize Ġof Ġ800 Ġare Ġapplied Ġto Ġget Ġthe Ġfinal Ġoutput . ĠThe Ġdec oder Ġis Ġa ĠWave Net ĠB IB REF 4 Ġvoc oder Ġwhich Ġconsists Ġof Ġfour Ġblocks Ġof Ġten Ġresidual Ġlayers . ĠThe Ġlinear Ġinterpol ation Ġand Ġnearest - ne igh bor Ġinterpol ation Ġare Ġapplied Ġto Ġthe Ġinput Ġpitch Ġand Ġenc oder Ġoutput Ġrespectively , Ġups am pling Ġthem Ġto Ġbe Ġof Ġthe Ġsame Ġsample Ġrate Ġas Ġthe Ġinput Ġaudio Ġwave form . </s> As Ġshown Ġby ĠFig . ĠFIG REF 2 , Ġthe Ġsinger Ġclassification Ġnetwork Ġand Ġpitch Ġregression Ġnetwork Ġhave Ġthe Ġsame Ġarchitecture Ġof Ġa Ġstack Ġof Ġtwo Ġconv olution al Ġneural Ġnetworks Ġwith Ġa Ġkernel Ġsize Ġof Ġ3 Ġand Ġchannels Ġof Ġ100 . ĠExcept Ġthat Ġthe Ġpitch Ġregression Ġnetwork Ġdoes Ġnot Ġaverage Ġthe Ġoutput Ġof Ġthe Ġtwo Ġconv olution Ġnetworks Ġbefore Ġpassing Ġit Ġinto Ġthe Ġfinal Ġfully Ġconnected Ġnetwork . ĠA Ġdrop out Ġlayer Ġis Ġalso Ġemployed Ġat Ġthe Ġbeginning Ġof Ġthe Ġnetwork Ġto Ġmake Ġthe Ġtraining Ġprocess Ġmore Ġstable . </s> Exper iments </s> Here Ġwe Ġcompare Ġthe Ġaudio Ġquality Ġbetween Ġour Ġmethod Ġand ĠB IB REF 0 's Ġmethod Ġ( Below Ġwe Ġcall ĠUS VC ), Ġand Ġshow Ġthat Ġthe Ġinput Ġpitch Ġcan Ġaffect Ġthe Ġoutput Ġsinging Ġvoice Ġby Ġqualitative Ġanalysis . ĠSince Ġthe Ġauthors Ġof ĠB IB REF 0 Ġdo Ġnot Ġrelease Ġtheir Ġsource Ġcode Ġand Ġonly Ġprovide Ġpart Ġof Ġthe Ġconverted Ġresults Ġat Ġtheir Ġwebsite , Ġwe Ġimplemented ĠUS VC Ġby Ġourselves , Ġden oted Ġby ĠUS VC ( our ) Ġbelow , Ġto Ġgive Ġa Ġmore Ġcomprehensive Ġcomparison . ĠAudio Ġsamples Ġare Ġavailable Ġat Ġour Ġwebsite Ġ. </s> Exper iments Ġ: :: ĠDat as et Ġand ĠPre processing </s> N US - 48 E ĠB IB REF 14 Ġdataset , Ġsung Ġby Ġ6 Ġmale Ġsingers Ġand Ġ6 Ġfemale Ġsingers , Ġwas Ġused Ġto Ġtrain Ġthe Ġmodels . ĠIt Ġcontains Ġ48 Ġsongs Ġeach Ġwith Ġlength Ġof Ġseveral Ġminutes . ĠEvery Ġsinger Ġprovided Ġ4 Ġsongs . ĠThe Ġmale Ġpart Ġof Ġthe Ġdataset Ġwas Ġselected Ġto Ġtrain Ġthe Ġmodels . ĠDuring Ġtesting , ĠWe Ġconverted Ġeach Ġone 's Ġsinging Ġvoice Ġto Ġthe Ġother Ġfive Ġsinger 's Ġvoice . ĠBefore Ġtraining , ĠWe Ġconverted Ġthe Ġsongs Ġto Ġmon oph onic Ġaudio Ġof Ġ16 kHz Ġsample Ġrate Ġand ĠPC M - 16 Ġbit Ġformat . ĠBesides , Ġ8 - bit Ġmu - law Ġencoding Ġwas Ġemployed Ġto Ġreduce Ġthe Ġinput Ġspace Ġto Ġspeed Ġup Ġthe Ġtraining Ġprocess , Ġalthough Ġit Ġwill Ġdegrade Ġthe Ġaudio Ġquality . ĠK aldi Ġtool kit ĠB IB REF 15 Ġwas Ġused Ġto Ġextract Ġpitch Ġfrom Ġthe Ġsongs Ġwith Ġhop Ġlength Ġof Ġ100 Ġwhich Ġmeans Ġthat Ġwe Ġcould Ġget Ġ1600 Ġpitch Ġsamples Ġin Ġan Ġaudio Ġsegment Ġof Ġone Ġsecond . ĠBefore Ġfeeding Ġthem Ġinto Ġthe Ġmodel , Ġwe Ġnormalized Ġthe Ġvalue Ġof Ġpitch Ġbetween Ġ0 Ġand Ġ1 . </s> Exper iments Ġ: :: ĠTraining </s> We Ġimplemented ĠUS VC Ġand ĠPitch Net Ġusing ĠPy Tor ch ĠB IB REF 16 Ġframework . ĠBoth Ġmodels Ġwere Ġtrained Ġon Ġtwo ĠTesla ĠP 40 ĠGPUs Ġfor Ġfour Ġdays . ĠAdam Ġoptim izer ĠB IB REF 17 Ġwas Ġused Ġwith Ġa Ġlearning Ġrate Ġof Ġ$ 10 ^{ - 3 } $ Ġand Ġa Ġdecay Ġfactor Ġof Ġ0 . 98 Ġevery Ġ1000 Ġsteps . ĠThe Ġbatch Ġsize Ġwas Ġset Ġto Ġ4 Ġand Ġfinally Ġthe Ġmodels Ġwere Ġtrained Ġfor Ġ30 k Ġsteps . Ġ$\ lambda Ġ$ Ġand Ġ$\ mu Ġ$ Ġin Ġthe Ġtraining Ġloss Ġ( DIS PLAY _ FORM 8 )( DIS PLAY _ FORM 9 ) Ġwere Ġset Ġto Ġ$ 0 . 01 $ Ġand Ġ$ 0 . 1 $ Ġrespectively . ĠThe Ġdrop out Ġprobability Ġin Ġthe Ġsinger Ġclassification Ġnetwork Ġand Ġpitch Ġregression Ġnetwork Ġwere Ġboth Ġ$ 0 . 2 $. </s> During Ġthe Ġtraining Ġprocess , Ġback translation Ġand Ġmix up ĠB IB REF 0 Ġwere Ġemployed Ġto Ġimprove Ġthe Ġconversion . ĠNew Ġtraining Ġsamples Ġwere Ġgenerated Ġby Ġmixing Ġembed ding Ġvectors Ġof Ġtwo Ġdifferent Ġsingers ĠA Ġand ĠB Ġwith Ġa Ġuniform Ġrandom Ġweight Ġfactor . ĠThen Ġthese Ġsamples Ġwere Ġfed Ġto Ġthe Ġmodel Ġto Ġreconstruct ĠA 's Ġvoice Ġwith Ġthe Ġembed ding Ġvector Ġof ĠA . ĠThe Ġreconstructed Ġvoice Ġand Ġoriginal Ġvoice Ġwere Ġused Ġto Ġcalculate Ġthe Ġreconstruction Ġloss . ĠAfter Ġtraining Ġfor Ġ200 k Ġsteps Ġwithout Ġback translation Ġand Ġmix up , Ġwe Ġgenerated Ġ96 Ġnew Ġaudio Ġsegments Ġevery Ġ2 k Ġsteps Ġand Ġused Ġthem Ġto Ġtrain Ġfor Ġ24 Ġsteps Ġwithout Ġthe Ġadvers arial Ġloss Ġ( DIS PLAY _ FORM 9 ). </s> Besides , Ġaudio Ġtime Ġreversal Ġand Ġphase Ġin version ĠB IB REF 0 Ġwere Ġalso Ġemployed Ġto Ġaugment Ġthe Ġtraining Ġdata Ġby Ġ4 Ġtimes . </s> Exper iments Ġ: :: ĠEvaluation </s> To Ġcompare Ġthe Ġconversions Ġbetween ĠUS VC Ġand ĠPitch Net , Ġwe Ġemployed Ġan Ġautomatic Ġevaluation Ġscore Ġand Ġa Ġhuman Ġevaluation Ġscore . </s> The Ġautomatic Ġscore Ġroughly Ġfollowed Ġthe Ġdesign Ġin ĠB IB REF 13 . ĠThe Ġpitch Ġtracker Ġof Ġlib ros a Ġpackage ĠB IB REF 18 Ġwas Ġemployed Ġto Ġextract Ġpitch Ġinformation Ġof Ġthe Ġinput Ġand Ġoutput Ġaudio . ĠThen Ġthe Ġoutput Ġpitch Ġwas Ġcompared Ġto Ġthe Ġinput Ġpitch Ġusing Ġthe Ġnormalized Ġcross Ġcorrelation Ġ( N CC ) Ġwhich Ġwould Ġgive Ġa Ġscore Ġbetween Ġ0 Ġand Ġ1 . ĠThe Ġhigher Ġthe Ġscore Ġis , Ġthe Ġbetter Ġthe Ġoutput Ġpitch Ġmatches Ġthe Ġinput Ġpitch . ĠWe Ġconducted Ġthe Ġevaluation Ġon ĠUS VC Ġ( our ) Ġand ĠPitch Net . ĠThe Ġevaluated Ġautomatic Ġscores Ġon Ġconversion Ġand Ġreconstruction Ġtasks Ġare Ġshown Ġin ĠTab . ĠT AB REF 14 . ĠOur Ġmethod Ġperformed Ġbetter Ġboth Ġon Ġconversion Ġand Ġreconstruction . ĠThe Ġscores Ġof Ġreconstruction Ġare Ġhigher Ġthan Ġconversion Ġsince Ġboth Ġmodels Ġwere Ġtrained Ġusing Ġa Ġreconstruction Ġloss . ĠHowever , Ġthe Ġscore Ġof Ġour Ġmethod Ġon Ġconversion Ġis Ġeven Ġhigher Ġthan Ġthe Ġscore Ġof ĠUS VC Ġ( Our ) Ġon Ġreconstruction . </s> Me an ĠOpinion ĠScore Ġ( M OS ) Ġwas Ġused Ġas Ġa Ġsubjective Ġmetric Ġto Ġevaluate Ġthe Ġquality Ġof Ġthe Ġconverted Ġaudio . ĠTwo Ġquestions Ġwere Ġasked : Ġ( 1 ) Ġwhat Ġis Ġquality Ġof Ġthe Ġaudio ? Ġ( natural ness ) Ġ( 2 ) ĠHow Ġwell Ġdoes Ġthe Ġconverted Ġversion Ġmatch Ġthe Ġoriginal ? Ġ( similar ity ) ĠA Ġscore Ġof Ġ1 - 5 Ġwould Ġbe Ġgiven Ġto Ġanswer Ġthe Ġquestions . ĠThe Ġevaluation Ġwas Ġconducted Ġon ĠUS VC Ġ( Our ) Ġand ĠPitch Net . ĠBesides , Ġthe Ġconverted Ġsamples Ġprovided Ġby ĠB IB REF 0 Ġwas Ġalso Ġincluded Ġto Ġgive Ġa Ġmore Ġconvincing Ġevaluation . ĠAs Ġshown Ġby ĠTab . ĠT AB REF 15 , Ġthe Ġnatural ness Ġand Ġsimilarity Ġof Ġour Ġmethod Ġare Ġboth Ġhigher Ġthan Ġthe Ġother Ġtwo Ġones . ĠOur Ġimplementation Ġof ĠUS VC Ġperformed Ġslightly Ġlower Ġthan Ġthe Ġoriginal Ġauthor 's Ġbecause Ġwe Ġcannot Ġfully Ġreproduce Ġthe Ġresults Ġof Ġthem . </s> Next Ġwe Ġqual itatively Ġanalyze Ġthe Ġinfluence Ġof Ġinput Ġpitch Ġin Ġour Ġmethod . ĠWe Ġused Ġdifferent Ġpitch Ġas Ġinput Ġto Ġobserve Ġhow Ġthe Ġoutput Ġpitch Ġwould Ġchange Ġalong Ġwith Ġthe Ġinput Ġpitch . ĠThe Ġinput Ġpitch Ġwas Ġmultiplied Ġby Ġ0 . 7 , Ġ1 . 0 Ġand Ġ1 . 3 Ġrespectively . ĠAnd Ġthe Ġoutput Ġpitch Ġwas Ġalso Ġextracted Ġby Ġthe Ġpitch Ġtracker Ġof Ġthe Ġlib ros a Ġpackage . ĠFig . ĠFIG REF 16 Ġplots Ġthe Ġpitch Ġof Ġinput Ġaudio Ġand Ġoutput Ġaudio Ġwith Ġdifferent Ġpitch Ġas Ġinput Ġwhile Ġkeeping Ġthe Ġtarget Ġsinger Ġthe Ġsame . ĠAs Ġshown Ġby ĠFig . ĠFIG REF 16 , Ġthe Ġoutput Ġpitch Ġchanges Ġsignificantly Ġalong Ġwith Ġthe Ġinput Ġpitch . ĠThe Ġexamples Ġare Ġalso Ġpresented Ġat Ġour Ġwebsite . </s> Conclusion </s> In Ġthis Ġpaper , Ġa Ġnovel Ġun super vised Ġsinging Ġvoice Ġconversion Ġmethod Ġnamed ĠPitch Net Ġis Ġproposed . ĠA Ġpitch Ġregression Ġnetwork Ġis Ġemployed Ġto Ġrender Ġan Ġadvers arial Ġloss Ġseparating Ġpitch Ġrelated Ġinformation Ġfrom Ġthe Ġlatent Ġspace Ġin Ġaut oen c oder . ĠAfter Ġthe ĠWave Net - like Ġenc oder , Ġa Ġsinger Ġand Ġpitch Ġinvari ant Ġrepresentation Ġis Ġgenerated Ġand Ġthen Ġfed Ġinto Ġthe ĠWave Net Ġdec oder Ġconditioning Ġon Ġthe Ġsinger Ġembed ding Ġand Ġthe Ġextracted Ġpitch Ġto Ġreconstruct Ġthe Ġtarget Ġsinging Ġvoice . ĠOur Ġmethod Ġoutper forms Ġthe Ġexisting Ġun super vised Ġsinging Ġvoice Ġconversion Ġmethod Ġand Ġachieves Ġflexible Ġpitch Ġmanipulation . </s> Acknowled gements </s> The Ġauthors Ġwould Ġlike Ġto Ġthanks ĠKun ĠXu Ġand Ġother Ġmembers Ġin Ġthe ĠTen cent ĠAI ĠLab Ġteam Ġfor Ġdiscussions Ġand Ġsuggestions .
Introduction Singing is an important way of human expression and the techniques of singing synthesis have broad applications in different prospects including virtual human, movie dubbing and so on. Traditional singing synthesize systems are based on concatenative BIBREF1 or HMM BIBREF2 based approaches. With the success of deep learning in Text-to-Speech, some neural singing synthesis methods have also been proposed recently. For example, BIBREF3 introduces a singing synthesis method using an architecture similar to WaveNet BIBREF4. It adopts lyrics and notes as input and generates vocoder features autoregressively for final singing voice synthesis. Singing voice conversion is another way of singing synthesis which extracts musical expression within existing singing and reproduced them with another singer's voice. It is very similar to speech based voice conversion BIBREF5, BIBREF6, BIBREF7, BIBREF8, but compared with speech voice conversion, singing voice conversion needs to deal with a wider range of frequency variations as well as the sharper change of volume and pitch within singing voice. The performance of singing conversion is highly dependent on the musical expression of converted singing and the similarity of the converted voice timbre compared to target singer's voice. There are several singing voice conversion methods to convert one's singing voice to another BIBREF9, BIBREF10, BIBREF11. They generally require parallel data to train the conversion model. To overcome the limitation of the parallel training data for singing voice conversion, an unsupervised method BIBREF0 has been proposed to utilize non-parallel data. This method employs an autoencoder architecture composed of a WaveNet-like encoder, a WaveNet BIBREF4 autoregressive decoder, and a learnable singer embedding table. Voice waveform is passed into the encoder and the output of the encoder will be concatenated with the embedding vector associated with the singer. The concatenated features will be used to condition the WaveNet decoder to reconstruct the input audio. A confusion loss BIBREF12 is also introduced to force the encoder to learn a singer-invariant representation. By switching between embeddings of different singers during generation, the singing voice conversion can be achieved. While this approach could generates singing voice perceptually similar to target singer, the quality of generated singing often suffers due to the difficulty on learning joint representation of phonetic and pitch representation. To address the difficulty of learning join phonetic and pitch representation in BIBREF0, we propose to use adversarially trained pitch regression network to encourage the encoder network to learn not only singer-invariant but also pitch-invariant representation, at the same time extract the pitch from source audio to be used as an additional input to the decoder. The proposed method can greatly improve the quality of the converted voice and achieve flexible pitch manipulation at the same time. In the following sections, we will introduce our proposed method in section SECREF2. And then section SECREF3 will show that our method is effective by quantitative and qualitative experiments. Finally, we will conclude in section SECREF4 and acknowledgements are in SECREF5. Method Our method follows the autoencoder architecture in BIBREF0 except that there is an additional pitch regression network to separate pitch information out of the latent space. The architecture of PitchNet is illustrated in Fig. FIGREF1. It consists of five parts, an encoder, a decoder, a Look Up Table (LUT) of speaker embedding vectors, a singer classification network, and a pitch regression network. First, the input waveform is passed through the encoder to extract high-level semantic features. An average pooling of stride 800 is then applied to the features, forming a bottleneck to limit the information passing through the encoder. After that, a singer id is used to retrieve the target singer's embedding vector from LUT, and concatenated with the output of the encoder at each time step to be a sequence of condition vectors. The pitch of the input audio, extracted separately from the network, is fed into the decoder after a linear interpolation as a compensation signal together with the condition vector. Finally, the decoder is conditioned on the condition vector and pitch to generate audio samples. Since the decoder is an autoregressive model, the output will be fed back to the decoder at the next time step. The model is trained on a softmax-based loss to minimize the reconstruction error with teacher-forcing. In order to project the output features of the encoder into a singer and pitch invariant latent space, a singer classification network and a pitch regression network are employed to force the encoder not to encode singer and pitch information. The singer classification loss and pitch regression loss are added adversarially to the reconstruction loss to train the entire model end to end. Method ::: Training Loss To formally describe the model, let $E$ be the encoder network, $D$ be the decoder network, $C_s$ be the singer classification network and $C_p$ be the pitch regression network. Let $v_j$ denote the embedding vector of singer $j$, $s^j$ denote an input audio of singer $j$ and $p(s^j)$ denote the extracted pitch of $s^j$. Now given an input audio sequence $s^j$ and a target singer $k$ where $j,k = 1,2,...,N$ and $N$ is the number of singers, the output of the model is Note that $D$ is an autoregressive model which would feed the output back to itself. The reconstruction loss is where $\mathcal {L}_{ce}(o, y)$ is the cross entropy loss applied to each element of $o$ and $y$. However, only reconstruction loss is not enough to train the model to learn to convert singing voice between different singers because it just forces the model to reconstruct the input voice. Therefore, a singer classification loss(also named domain confusion loss BIBREF0) is applied to make the encoder to learn a singer invariant representation Furthermore, a pitch regression loss is introduced to force the encoder to learn a pitch-independent representation and make the whole model obtain the pitch information from $p(s^j)$ rather than directly from the input audio where $\mathcal {L}_{mse}(a, b)$ is the mean square error function $\frac{1}{m}||a-b||_2^2$ and m is the number of elements in $a$. The overall loss we minimize to train the model is where $\lambda $ and $\mu $ are two weight factors. Furthermore, the adversarial loss used to train the singer classifier and pitch regression network is In the training process, we minimize $\mathcal {L}_{ad}$ and $\mathcal {L}_{total}$ alternately, that is Optimize $C_s$ and $C_p$ one step using $\mathcal {L}_{ad}$ as the objective function. Optimize the whole model one step using $\mathcal {L}_{total}$ as the objective function. Go back to step 1. Furthermore, backtranslation and mixup techniques BIBREF0 are also used to improve the quality of the converted singing voice. Method ::: The architecture of the Sub-Networks The encoder and decoder networks follow the design in BIBREF13. The encoder is a fully convolutional network with three blocks of ten residual-layers which consists of a ReLU activation, a dilated convolution, a ReLU activation, a 1x1 convolution, and a residual summation in order. After three residual blocks, a 1x1 convolution and an average pooling with a kernel size of 800 are applied to get the final output. The decoder is a WaveNet BIBREF4 vocoder which consists of four blocks of ten residual layers. The linear interpolation and nearest-neighbor interpolation are applied to the input pitch and encoder output respectively, upsampling them to be of the same sample rate as the input audio waveform. As shown by Fig. FIGREF2, the singer classification network and pitch regression network have the same architecture of a stack of two convolutional neural networks with a kernel size of 3 and channels of 100. Except that the pitch regression network does not average the output of the two convolution networks before passing it into the final fully connected network. A dropout layer is also employed at the beginning of the network to make the training process more stable. Experiments Here we compare the audio quality between our method and BIBREF0's method (Below we call USVC), and show that the input pitch can affect the output singing voice by qualitative analysis. Since the authors of BIBREF0 do not release their source code and only provide part of the converted results at their website, we implemented USVC by ourselves, denoted by USVC(our) below, to give a more comprehensive comparison. Audio samples are available at our website . Experiments ::: Dataset and Preprocessing NUS-48E BIBREF14 dataset, sung by 6 male singers and 6 female singers, was used to train the models. It contains 48 songs each with length of several minutes. Every singer provided 4 songs. The male part of the dataset was selected to train the models. During testing, We converted each one's singing voice to the other five singer's voice. Before training, We converted the songs to monophonic audio of 16kHz sample rate and PCM-16 bit format. Besides, 8-bit mu-law encoding was employed to reduce the input space to speed up the training process, although it will degrade the audio quality. Kaldi toolkit BIBREF15 was used to extract pitch from the songs with hop length of 100 which means that we could get 1600 pitch samples in an audio segment of one second. Before feeding them into the model, we normalized the value of pitch between 0 and 1. Experiments ::: Training We implemented USVC and PitchNet using PyTorch BIBREF16 framework. Both models were trained on two Tesla P40 GPUs for four days. Adam optimizer BIBREF17 was used with a learning rate of $10^{-3}$ and a decay factor of 0.98 every 1000 steps. The batch size was set to 4 and finally the models were trained for 30k steps. $\lambda $ and $\mu $ in the training loss (DISPLAY_FORM8)(DISPLAY_FORM9) were set to $0.01$ and $0.1$ respectively. The dropout probability in the singer classification network and pitch regression network were both $0.2$. During the training process, backtranslation and mixup BIBREF0 were employed to improve the conversion. New training samples were generated by mixing embedding vectors of two different singers A and B with a uniform random weight factor. Then these samples were fed to the model to reconstruct A's voice with the embedding vector of A. The reconstructed voice and original voice were used to calculate the reconstruction loss. After training for 200k steps without backtranslation and mixup, we generated 96 new audio segments every 2k steps and used them to train for 24 steps without the adversarial loss (DISPLAY_FORM9). Besides, audio time reversal and phase inversion BIBREF0 were also employed to augment the training data by 4 times. Experiments ::: Evaluation To compare the conversions between USVC and PitchNet, we employed an automatic evaluation score and a human evaluation score. The automatic score roughly followed the design in BIBREF13. The pitch tracker of librosa package BIBREF18 was employed to extract pitch information of the input and output audio. Then the output pitch was compared to the input pitch using the normalized cross correlation (NCC) which would give a score between 0 and 1. The higher the score is, the better the output pitch matches the input pitch. We conducted the evaluation on USVC (our) and PitchNet. The evaluated automatic scores on conversion and reconstruction tasks are shown in Tab. TABREF14. Our method performed better both on conversion and reconstruction. The scores of reconstruction are higher than conversion since both models were trained using a reconstruction loss. However, the score of our method on conversion is even higher than the score of USVC (Our) on reconstruction. Mean Opinion Score (MOS) was used as a subjective metric to evaluate the quality of the converted audio. Two questions were asked: (1) what is quality of the audio? (naturalness) (2) How well does the converted version match the original? (similarity) A score of 1-5 would be given to answer the questions. The evaluation was conducted on USVC (Our) and PitchNet. Besides, the converted samples provided by BIBREF0 was also included to give a more convincing evaluation. As shown by Tab. TABREF15, the naturalness and similarity of our method are both higher than the other two ones. Our implementation of USVC performed slightly lower than the original author's because we cannot fully reproduce the results of them. Next we qualitatively analyze the influence of input pitch in our method. We used different pitch as input to observe how the output pitch would change along with the input pitch. The input pitch was multiplied by 0.7, 1.0 and 1.3 respectively. And the output pitch was also extracted by the pitch tracker of the librosa package. Fig. FIGREF16 plots the pitch of input audio and output audio with different pitch as input while keeping the target singer the same. As shown by Fig. FIGREF16, the output pitch changes significantly along with the input pitch. The examples are also presented at our website. Conclusion In this paper, a novel unsupervised singing voice conversion method named PitchNet is proposed. A pitch regression network is employed to render an adversarial loss separating pitch related information from the latent space in autoencoder. After the WaveNet-like encoder, a singer and pitch invariant representation is generated and then fed into the WaveNet decoder conditioning on the singer embedding and the extracted pitch to reconstruct the target singing voice. Our method outperforms the existing unsupervised singing voice conversion method and achieves flexible pitch manipulation. Acknowledgements The authors would like to thanks Kun Xu and other members in the Tencent AI Lab team for discussions and suggestions.
How is the quality of singing voice measured?
d6e353e0231d09fd5dcba493544d53706f3fe1ab
How Ġis Ġthe Ġquality Ġof Ġsinging Ġvoice Ġmeasured ?
To Ġcompare Ġthe Ġconversions Ġbetween ĠUS VC Ġand ĠPitch Net , Ġwe Ġemployed Ġan Ġautomatic Ġevaluation Ġscore Ġand Ġa Ġhuman Ġevaluation Ġscore .
<s> How Ġis Ġthe Ġquality Ġof Ġsinging Ġvoice Ġmeasured ? </s> Introduction </s> S inging Ġis Ġan Ġimportant Ġway Ġof Ġhuman Ġexpression Ġand Ġthe Ġtechniques Ġof Ġsinging Ġsynthesis Ġhave Ġbroad Ġapplications Ġin Ġdifferent Ġprospects Ġincluding Ġvirtual Ġhuman , Ġmovie Ġdub bing Ġand Ġso Ġon . ĠTraditional Ġsinging Ġsynthes ize Ġsystems Ġare Ġbased Ġon Ġconc aten ative ĠB IB REF 1 Ġor ĠH MM ĠB IB REF 2 Ġbased Ġapproaches . ĠWith Ġthe Ġsuccess Ġof Ġdeep Ġlearning Ġin ĠText - to - Spe ech , Ġsome Ġneural Ġsinging Ġsynthesis Ġmethods Ġhave Ġalso Ġbeen Ġproposed Ġrecently . ĠFor Ġexample , ĠB IB REF 3 Ġintroduces Ġa Ġsinging Ġsynthesis Ġmethod Ġusing Ġan Ġarchitecture Ġsimilar Ġto ĠWave Net ĠB IB REF 4 . ĠIt Ġadop ts Ġlyrics Ġand Ġnotes Ġas Ġinput Ġand Ġgenerates Ġvoc oder Ġfeatures Ġaut ore gress ively Ġfor Ġfinal Ġsinging Ġvoice Ġsynthesis . </s> S inging Ġvoice Ġconversion Ġis Ġanother Ġway Ġof Ġsinging Ġsynthesis Ġwhich Ġextracts Ġmusical Ġexpression Ġwithin Ġexisting Ġsinging Ġand Ġreproduced Ġthem Ġwith Ġanother Ġsinger 's Ġvoice . ĠIt Ġis Ġvery Ġsimilar Ġto Ġspeech Ġbased Ġvoice Ġconversion ĠB IB REF 5 , ĠB IB REF 6 , ĠB IB REF 7 , ĠB IB REF 8 , Ġbut Ġcompared Ġwith Ġspeech Ġvoice Ġconversion , Ġsinging Ġvoice Ġconversion Ġneeds Ġto Ġdeal Ġwith Ġa Ġwider Ġrange Ġof Ġfrequency Ġvariations Ġas Ġwell Ġas Ġthe Ġsharper Ġchange Ġof Ġvolume Ġand Ġpitch Ġwithin Ġsinging Ġvoice . ĠThe Ġperformance Ġof Ġsinging Ġconversion Ġis Ġhighly Ġdependent Ġon Ġthe Ġmusical Ġexpression Ġof Ġconverted Ġsinging Ġand Ġthe Ġsimilarity Ġof Ġthe Ġconverted Ġvoice Ġtim bre Ġcompared Ġto Ġtarget Ġsinger 's Ġvoice . ĠThere Ġare Ġseveral Ġsinging Ġvoice Ġconversion Ġmethods Ġto Ġconvert Ġone 's Ġsinging Ġvoice Ġto Ġanother ĠB IB REF 9 , ĠB IB REF 10 , ĠB IB REF 11 . ĠThey Ġgenerally Ġrequire Ġparallel Ġdata Ġto Ġtrain Ġthe Ġconversion Ġmodel . ĠTo Ġovercome Ġthe Ġlimitation Ġof Ġthe Ġparallel Ġtraining Ġdata Ġfor Ġsinging Ġvoice Ġconversion , Ġan Ġun super vised Ġmethod ĠB IB REF 0 Ġhas Ġbeen Ġproposed Ġto Ġutilize Ġnon - par allel Ġdata . ĠThis Ġmethod Ġemploys Ġan Ġaut oen c oder Ġarchitecture Ġcomposed Ġof Ġa ĠWave Net - like Ġenc oder , Ġa ĠWave Net ĠB IB REF 4 Ġaut ore gressive Ġdec oder , Ġand Ġa Ġlearn able Ġsinger Ġembed ding Ġtable . ĠVoice Ġwave form Ġis Ġpassed Ġinto Ġthe Ġenc oder Ġand Ġthe Ġoutput Ġof Ġthe Ġenc oder Ġwill Ġbe Ġconc aten ated Ġwith Ġthe Ġembed ding Ġvector Ġassociated Ġwith Ġthe Ġsinger . ĠThe Ġconc aten ated Ġfeatures Ġwill Ġbe Ġused Ġto Ġcondition Ġthe ĠWave Net Ġdec oder Ġto Ġreconstruct Ġthe Ġinput Ġaudio . ĠA Ġconfusion Ġloss ĠB IB REF 12 Ġis Ġalso Ġintroduced Ġto Ġforce Ġthe Ġenc oder Ġto Ġlearn Ġa Ġsinger - inv ari ant Ġrepresentation . ĠBy Ġswitching Ġbetween Ġembed d ings Ġof Ġdifferent Ġsingers Ġduring Ġgeneration , Ġthe Ġsinging Ġvoice Ġconversion Ġcan Ġbe Ġachieved . ĠWhile Ġthis Ġapproach Ġcould Ġgenerates Ġsinging Ġvoice Ġpercept ually Ġsimilar Ġto Ġtarget Ġsinger , Ġthe Ġquality Ġof Ġgenerated Ġsinging Ġoften Ġsuffers Ġdue Ġto Ġthe Ġdifficulty Ġon Ġlearning Ġjoint Ġrepresentation Ġof Ġphon etic Ġand Ġpitch Ġrepresentation . </s> To Ġaddress Ġthe Ġdifficulty Ġof Ġlearning Ġjoin Ġphon etic Ġand Ġpitch Ġrepresentation Ġin ĠB IB REF 0 , Ġwe Ġpropose Ġto Ġuse Ġadvers ar ially Ġtrained Ġpitch Ġregression Ġnetwork Ġto Ġencourage Ġthe Ġenc oder Ġnetwork Ġto Ġlearn Ġnot Ġonly Ġsinger - inv ari ant Ġbut Ġalso Ġpitch - inv ari ant Ġrepresentation , Ġat Ġthe Ġsame Ġtime Ġextract Ġthe Ġpitch Ġfrom Ġsource Ġaudio Ġto Ġbe Ġused Ġas Ġan Ġadditional Ġinput Ġto Ġthe Ġdec oder . ĠThe Ġproposed Ġmethod Ġcan Ġgreatly Ġimprove Ġthe Ġquality Ġof Ġthe Ġconverted Ġvoice Ġand Ġachieve Ġflexible Ġpitch Ġmanipulation Ġat Ġthe Ġsame Ġtime . </s> In Ġthe Ġfollowing Ġsections , Ġwe Ġwill Ġintroduce Ġour Ġproposed Ġmethod Ġin Ġsection ĠSEC REF 2 . ĠAnd Ġthen Ġsection ĠSEC REF 3 Ġwill Ġshow Ġthat Ġour Ġmethod Ġis Ġeffective Ġby Ġquantitative Ġand Ġqualitative Ġexperiments . ĠFinally , Ġwe Ġwill Ġconclude Ġin Ġsection ĠSEC REF 4 Ġand Ġacknowled gements Ġare Ġin ĠSEC REF 5 . </s> Method </s> Our Ġmethod Ġfollows Ġthe Ġaut oen c oder Ġarchitecture Ġin ĠB IB REF 0 Ġexcept Ġthat Ġthere Ġis Ġan Ġadditional Ġpitch Ġregression Ġnetwork Ġto Ġseparate Ġpitch Ġinformation Ġout Ġof Ġthe Ġlatent Ġspace . ĠThe Ġarchitecture Ġof ĠPitch Net Ġis Ġillustrated Ġin ĠFig . ĠFIG REF 1 . ĠIt Ġconsists Ġof Ġfive Ġparts , Ġan Ġenc oder , Ġa Ġdec oder , Ġa ĠLook ĠUp ĠTable Ġ( L UT ) Ġof Ġspeaker Ġembed ding Ġvectors , Ġa Ġsinger Ġclassification Ġnetwork , Ġand Ġa Ġpitch Ġregression Ġnetwork . </s> First , Ġthe Ġinput Ġwave form Ġis Ġpassed Ġthrough Ġthe Ġenc oder Ġto Ġextract Ġhigh - level Ġsemantic Ġfeatures . ĠAn Ġaverage Ġpool ing Ġof Ġstride Ġ800 Ġis Ġthen Ġapplied Ġto Ġthe Ġfeatures , Ġforming Ġa Ġbottleneck Ġto Ġlimit Ġthe Ġinformation Ġpassing Ġthrough Ġthe Ġenc oder . ĠAfter Ġthat , Ġa Ġsinger Ġid Ġis Ġused Ġto Ġretrieve Ġthe Ġtarget Ġsinger 's Ġembed ding Ġvector Ġfrom ĠL UT , Ġand Ġconc aten ated Ġwith Ġthe Ġoutput Ġof Ġthe Ġenc oder Ġat Ġeach Ġtime Ġstep Ġto Ġbe Ġa Ġsequence Ġof Ġcondition Ġvectors . ĠThe Ġpitch Ġof Ġthe Ġinput Ġaudio , Ġextracted Ġseparately Ġfrom Ġthe Ġnetwork , Ġis Ġfed Ġinto Ġthe Ġdec oder Ġafter Ġa Ġlinear Ġinterpol ation Ġas Ġa Ġcompensation Ġsignal Ġtogether Ġwith Ġthe Ġcondition Ġvector . ĠFinally , Ġthe Ġdec oder Ġis Ġconditioned Ġon Ġthe Ġcondition Ġvector Ġand Ġpitch Ġto Ġgenerate Ġaudio Ġsamples . ĠSince Ġthe Ġdec oder Ġis Ġan Ġaut ore gressive Ġmodel , Ġthe Ġoutput Ġwill Ġbe Ġfed Ġback Ġto Ġthe Ġdec oder Ġat Ġthe Ġnext Ġtime Ġstep . ĠThe Ġmodel Ġis Ġtrained Ġon Ġa Ġsoft max - based Ġloss Ġto Ġminimize Ġthe Ġreconstruction Ġerror Ġwith Ġteacher - forcing . </s> In Ġorder Ġto Ġproject Ġthe Ġoutput Ġfeatures Ġof Ġthe Ġenc oder Ġinto Ġa Ġsinger Ġand Ġpitch Ġinvari ant Ġlatent Ġspace , Ġa Ġsinger Ġclassification Ġnetwork Ġand Ġa Ġpitch Ġregression Ġnetwork Ġare Ġemployed Ġto Ġforce Ġthe Ġenc oder Ġnot Ġto Ġencode Ġsinger Ġand Ġpitch Ġinformation . ĠThe Ġsinger Ġclassification Ġloss Ġand Ġpitch Ġregression Ġloss Ġare Ġadded Ġadvers ar ially Ġto Ġthe Ġreconstruction Ġloss Ġto Ġtrain Ġthe Ġentire Ġmodel Ġend Ġto Ġend . </s> Method Ġ: :: ĠTraining ĠLoss </s> To Ġformally Ġdescribe Ġthe Ġmodel , Ġlet Ġ$ E $ Ġbe Ġthe Ġenc oder Ġnetwork , Ġ$ D $ Ġbe Ġthe Ġdec oder Ġnetwork , Ġ$ C _ s $ Ġbe Ġthe Ġsinger Ġclassification Ġnetwork Ġand Ġ$ C _ p $ Ġbe Ġthe Ġpitch Ġregression Ġnetwork . ĠLet Ġ$ v _ j $ Ġdenote Ġthe Ġembed ding Ġvector Ġof Ġsinger Ġ$ j $, Ġ$ s ^ j $ Ġdenote Ġan Ġinput Ġaudio Ġof Ġsinger Ġ$ j $ Ġand Ġ$ p ( s ^ j ) $ Ġdenote Ġthe Ġextracted Ġpitch Ġof Ġ$ s ^ j $. ĠNow Ġgiven Ġan Ġinput Ġaudio Ġsequence Ġ$ s ^ j $ Ġand Ġa Ġtarget Ġsinger Ġ$ k $ Ġwhere Ġ$ j , k Ġ= Ġ1 , 2 ,... , N $ Ġand Ġ$ N $ Ġis Ġthe Ġnumber Ġof Ġsingers , Ġthe Ġoutput Ġof Ġthe Ġmodel Ġis </s> Note Ġthat Ġ$ D $ Ġis Ġan Ġaut ore gressive Ġmodel Ġwhich Ġwould Ġfeed Ġthe Ġoutput Ġback Ġto Ġitself . ĠThe Ġreconstruction Ġloss Ġis </s> where Ġ$\ math cal Ġ{ L } _{ ce } ( o , Ġy ) $ Ġis Ġthe Ġcross Ġentropy Ġloss Ġapplied Ġto Ġeach Ġelement Ġof Ġ$ o $ Ġand Ġ$ y $. ĠHowever , Ġonly Ġreconstruction Ġloss Ġis Ġnot Ġenough Ġto Ġtrain Ġthe Ġmodel Ġto Ġlearn Ġto Ġconvert Ġsinging Ġvoice Ġbetween Ġdifferent Ġsingers Ġbecause Ġit Ġjust Ġforces Ġthe Ġmodel Ġto Ġreconstruct Ġthe Ġinput Ġvoice . ĠTherefore , Ġa Ġsinger Ġclassification Ġloss ( also Ġnamed Ġdomain Ġconfusion Ġloss ĠB IB REF 0 ) Ġis Ġapplied Ġto Ġmake Ġthe Ġenc oder Ġto Ġlearn Ġa Ġsinger Ġinvari ant Ġrepresentation </s> Furthermore , Ġa Ġpitch Ġregression Ġloss Ġis Ġintroduced Ġto Ġforce Ġthe Ġenc oder Ġto Ġlearn Ġa Ġpitch - independent Ġrepresentation Ġand Ġmake Ġthe Ġwhole Ġmodel Ġobtain Ġthe Ġpitch Ġinformation Ġfrom Ġ$ p ( s ^ j ) $ Ġrather Ġthan Ġdirectly Ġfrom Ġthe Ġinput Ġaudio </s> where Ġ$\ math cal Ġ{ L } _{ m se } ( a , Ġb ) $ Ġis Ġthe Ġmean Ġsquare Ġerror Ġfunction Ġ$\ frac { 1 }{ m } || a - b || _ 2 ^ 2 $ Ġand Ġm Ġis Ġthe Ġnumber Ġof Ġelements Ġin Ġ$ a $. ĠThe Ġoverall Ġloss Ġwe Ġminimize Ġto Ġtrain Ġthe Ġmodel Ġis </s> where Ġ$\ lambda Ġ$ Ġand Ġ$\ mu Ġ$ Ġare Ġtwo Ġweight Ġfactors . ĠFurthermore , Ġthe Ġadvers arial Ġloss Ġused Ġto Ġtrain Ġthe Ġsinger Ġclass ifier Ġand Ġpitch Ġregression Ġnetwork Ġis </s> In Ġthe Ġtraining Ġprocess , Ġwe Ġminimize Ġ$\ math cal Ġ{ L } _{ ad } $ Ġand Ġ$\ math cal Ġ{ L } _{ total } $ Ġaltern ately , Ġthat Ġis </s> Opt imize Ġ$ C _ s $ Ġand Ġ$ C _ p $ Ġone Ġstep Ġusing Ġ$\ math cal Ġ{ L } _{ ad } $ Ġas Ġthe Ġobjective Ġfunction . </s> Opt imize Ġthe Ġwhole Ġmodel Ġone Ġstep Ġusing Ġ$\ math cal Ġ{ L } _{ total } $ Ġas Ġthe Ġobjective Ġfunction . </s> Go Ġback Ġto Ġstep Ġ1 . </s> Furthermore , Ġback translation Ġand Ġmix up Ġtechniques ĠB IB REF 0 Ġare Ġalso Ġused Ġto Ġimprove Ġthe Ġquality Ġof Ġthe Ġconverted Ġsinging Ġvoice . </s> Method Ġ: :: ĠThe Ġarchitecture Ġof Ġthe ĠSub - Net works </s> The Ġenc oder Ġand Ġdec oder Ġnetworks Ġfollow Ġthe Ġdesign Ġin ĠB IB REF 13 . ĠThe Ġenc oder Ġis Ġa Ġfully Ġconv olution al Ġnetwork Ġwith Ġthree Ġblocks Ġof Ġten Ġresidual - l ayers Ġwhich Ġconsists Ġof Ġa ĠRe LU Ġactivation , Ġa Ġdil ated Ġconv olution , Ġa ĠRe LU Ġactivation , Ġa Ġ1 x 1 Ġconv olution , Ġand Ġa Ġresidual Ġsumm ation Ġin Ġorder . ĠAfter Ġthree Ġresidual Ġblocks , Ġa Ġ1 x 1 Ġconv olution Ġand Ġan Ġaverage Ġpool ing Ġwith Ġa Ġkernel Ġsize Ġof Ġ800 Ġare Ġapplied Ġto Ġget Ġthe Ġfinal Ġoutput . ĠThe Ġdec oder Ġis Ġa ĠWave Net ĠB IB REF 4 Ġvoc oder Ġwhich Ġconsists Ġof Ġfour Ġblocks Ġof Ġten Ġresidual Ġlayers . ĠThe Ġlinear Ġinterpol ation Ġand Ġnearest - ne igh bor Ġinterpol ation Ġare Ġapplied Ġto Ġthe Ġinput Ġpitch Ġand Ġenc oder Ġoutput Ġrespectively , Ġups am pling Ġthem Ġto Ġbe Ġof Ġthe Ġsame Ġsample Ġrate Ġas Ġthe Ġinput Ġaudio Ġwave form . </s> As Ġshown Ġby ĠFig . ĠFIG REF 2 , Ġthe Ġsinger Ġclassification Ġnetwork Ġand Ġpitch Ġregression Ġnetwork Ġhave Ġthe Ġsame Ġarchitecture Ġof Ġa Ġstack Ġof Ġtwo Ġconv olution al Ġneural Ġnetworks Ġwith Ġa Ġkernel Ġsize Ġof Ġ3 Ġand Ġchannels Ġof Ġ100 . ĠExcept Ġthat Ġthe Ġpitch Ġregression Ġnetwork Ġdoes Ġnot Ġaverage Ġthe Ġoutput Ġof Ġthe Ġtwo Ġconv olution Ġnetworks Ġbefore Ġpassing Ġit Ġinto Ġthe Ġfinal Ġfully Ġconnected Ġnetwork . ĠA Ġdrop out Ġlayer Ġis Ġalso Ġemployed Ġat Ġthe Ġbeginning Ġof Ġthe Ġnetwork Ġto Ġmake Ġthe Ġtraining Ġprocess Ġmore Ġstable . </s> Exper iments </s> Here Ġwe Ġcompare Ġthe Ġaudio Ġquality Ġbetween Ġour Ġmethod Ġand ĠB IB REF 0 's Ġmethod Ġ( Below Ġwe Ġcall ĠUS VC ), Ġand Ġshow Ġthat Ġthe Ġinput Ġpitch Ġcan Ġaffect Ġthe Ġoutput Ġsinging Ġvoice Ġby Ġqualitative Ġanalysis . ĠSince Ġthe Ġauthors Ġof ĠB IB REF 0 Ġdo Ġnot Ġrelease Ġtheir Ġsource Ġcode Ġand Ġonly Ġprovide Ġpart Ġof Ġthe Ġconverted Ġresults Ġat Ġtheir Ġwebsite , Ġwe Ġimplemented ĠUS VC Ġby Ġourselves , Ġden oted Ġby ĠUS VC ( our ) Ġbelow , Ġto Ġgive Ġa Ġmore Ġcomprehensive Ġcomparison . ĠAudio Ġsamples Ġare Ġavailable Ġat Ġour Ġwebsite Ġ. </s> Exper iments Ġ: :: ĠDat as et Ġand ĠPre processing </s> N US - 48 E ĠB IB REF 14 Ġdataset , Ġsung Ġby Ġ6 Ġmale Ġsingers Ġand Ġ6 Ġfemale Ġsingers , Ġwas Ġused Ġto Ġtrain Ġthe Ġmodels . ĠIt Ġcontains Ġ48 Ġsongs Ġeach Ġwith Ġlength Ġof Ġseveral Ġminutes . ĠEvery Ġsinger Ġprovided Ġ4 Ġsongs . ĠThe Ġmale Ġpart Ġof Ġthe Ġdataset Ġwas Ġselected Ġto Ġtrain Ġthe Ġmodels . ĠDuring Ġtesting , ĠWe Ġconverted Ġeach Ġone 's Ġsinging Ġvoice Ġto Ġthe Ġother Ġfive Ġsinger 's Ġvoice . ĠBefore Ġtraining , ĠWe Ġconverted Ġthe Ġsongs Ġto Ġmon oph onic Ġaudio Ġof Ġ16 kHz Ġsample Ġrate Ġand ĠPC M - 16 Ġbit Ġformat . ĠBesides , Ġ8 - bit Ġmu - law Ġencoding Ġwas Ġemployed Ġto Ġreduce Ġthe Ġinput Ġspace Ġto Ġspeed Ġup Ġthe Ġtraining Ġprocess , Ġalthough Ġit Ġwill Ġdegrade Ġthe Ġaudio Ġquality . ĠK aldi Ġtool kit ĠB IB REF 15 Ġwas Ġused Ġto Ġextract Ġpitch Ġfrom Ġthe Ġsongs Ġwith Ġhop Ġlength Ġof Ġ100 Ġwhich Ġmeans Ġthat Ġwe Ġcould Ġget Ġ1600 Ġpitch Ġsamples Ġin Ġan Ġaudio Ġsegment Ġof Ġone Ġsecond . ĠBefore Ġfeeding Ġthem Ġinto Ġthe Ġmodel , Ġwe Ġnormalized Ġthe Ġvalue Ġof Ġpitch Ġbetween Ġ0 Ġand Ġ1 . </s> Exper iments Ġ: :: ĠTraining </s> We Ġimplemented ĠUS VC Ġand ĠPitch Net Ġusing ĠPy Tor ch ĠB IB REF 16 Ġframework . ĠBoth Ġmodels Ġwere Ġtrained Ġon Ġtwo ĠTesla ĠP 40 ĠGPUs Ġfor Ġfour Ġdays . ĠAdam Ġoptim izer ĠB IB REF 17 Ġwas Ġused Ġwith Ġa Ġlearning Ġrate Ġof Ġ$ 10 ^{ - 3 } $ Ġand Ġa Ġdecay Ġfactor Ġof Ġ0 . 98 Ġevery Ġ1000 Ġsteps . ĠThe Ġbatch Ġsize Ġwas Ġset Ġto Ġ4 Ġand Ġfinally Ġthe Ġmodels Ġwere Ġtrained Ġfor Ġ30 k Ġsteps . Ġ$\ lambda Ġ$ Ġand Ġ$\ mu Ġ$ Ġin Ġthe Ġtraining Ġloss Ġ( DIS PLAY _ FORM 8 )( DIS PLAY _ FORM 9 ) Ġwere Ġset Ġto Ġ$ 0 . 01 $ Ġand Ġ$ 0 . 1 $ Ġrespectively . ĠThe Ġdrop out Ġprobability Ġin Ġthe Ġsinger Ġclassification Ġnetwork Ġand Ġpitch Ġregression Ġnetwork Ġwere Ġboth Ġ$ 0 . 2 $. </s> During Ġthe Ġtraining Ġprocess , Ġback translation Ġand Ġmix up ĠB IB REF 0 Ġwere Ġemployed Ġto Ġimprove Ġthe Ġconversion . ĠNew Ġtraining Ġsamples Ġwere Ġgenerated Ġby Ġmixing Ġembed ding Ġvectors Ġof Ġtwo Ġdifferent Ġsingers ĠA Ġand ĠB Ġwith Ġa Ġuniform Ġrandom Ġweight Ġfactor . ĠThen Ġthese Ġsamples Ġwere Ġfed Ġto Ġthe Ġmodel Ġto Ġreconstruct ĠA 's Ġvoice Ġwith Ġthe Ġembed ding Ġvector Ġof ĠA . ĠThe Ġreconstructed Ġvoice Ġand Ġoriginal Ġvoice Ġwere Ġused Ġto Ġcalculate Ġthe Ġreconstruction Ġloss . ĠAfter Ġtraining Ġfor Ġ200 k Ġsteps Ġwithout Ġback translation Ġand Ġmix up , Ġwe Ġgenerated Ġ96 Ġnew Ġaudio Ġsegments Ġevery Ġ2 k Ġsteps Ġand Ġused Ġthem Ġto Ġtrain Ġfor Ġ24 Ġsteps Ġwithout Ġthe Ġadvers arial Ġloss Ġ( DIS PLAY _ FORM 9 ). </s> Besides , Ġaudio Ġtime Ġreversal Ġand Ġphase Ġin version ĠB IB REF 0 Ġwere Ġalso Ġemployed Ġto Ġaugment Ġthe Ġtraining Ġdata Ġby Ġ4 Ġtimes . </s> Exper iments Ġ: :: ĠEvaluation </s> To Ġcompare Ġthe Ġconversions Ġbetween ĠUS VC Ġand ĠPitch Net , Ġwe Ġemployed Ġan Ġautomatic Ġevaluation Ġscore Ġand Ġa Ġhuman Ġevaluation Ġscore . </s> The Ġautomatic Ġscore Ġroughly Ġfollowed Ġthe Ġdesign Ġin ĠB IB REF 13 . ĠThe Ġpitch Ġtracker Ġof Ġlib ros a Ġpackage ĠB IB REF 18 Ġwas Ġemployed Ġto Ġextract Ġpitch Ġinformation Ġof Ġthe Ġinput Ġand Ġoutput Ġaudio . ĠThen Ġthe Ġoutput Ġpitch Ġwas Ġcompared Ġto Ġthe Ġinput Ġpitch Ġusing Ġthe Ġnormalized Ġcross Ġcorrelation Ġ( N CC ) Ġwhich Ġwould Ġgive Ġa Ġscore Ġbetween Ġ0 Ġand Ġ1 . ĠThe Ġhigher Ġthe Ġscore Ġis , Ġthe Ġbetter Ġthe Ġoutput Ġpitch Ġmatches Ġthe Ġinput Ġpitch . ĠWe Ġconducted Ġthe Ġevaluation Ġon ĠUS VC Ġ( our ) Ġand ĠPitch Net . ĠThe Ġevaluated Ġautomatic Ġscores Ġon Ġconversion Ġand Ġreconstruction Ġtasks Ġare Ġshown Ġin ĠTab . ĠT AB REF 14 . ĠOur Ġmethod Ġperformed Ġbetter Ġboth Ġon Ġconversion Ġand Ġreconstruction . ĠThe Ġscores Ġof Ġreconstruction Ġare Ġhigher Ġthan Ġconversion Ġsince Ġboth Ġmodels Ġwere Ġtrained Ġusing Ġa Ġreconstruction Ġloss . ĠHowever , Ġthe Ġscore Ġof Ġour Ġmethod Ġon Ġconversion Ġis Ġeven Ġhigher Ġthan Ġthe Ġscore Ġof ĠUS VC Ġ( Our ) Ġon Ġreconstruction . </s> Me an ĠOpinion ĠScore Ġ( M OS ) Ġwas Ġused Ġas Ġa Ġsubjective Ġmetric Ġto Ġevaluate Ġthe Ġquality Ġof Ġthe Ġconverted Ġaudio . ĠTwo Ġquestions Ġwere Ġasked : Ġ( 1 ) Ġwhat Ġis Ġquality Ġof Ġthe Ġaudio ? Ġ( natural ness ) Ġ( 2 ) ĠHow Ġwell Ġdoes Ġthe Ġconverted Ġversion Ġmatch Ġthe Ġoriginal ? Ġ( similar ity ) ĠA Ġscore Ġof Ġ1 - 5 Ġwould Ġbe Ġgiven Ġto Ġanswer Ġthe Ġquestions . ĠThe Ġevaluation Ġwas Ġconducted Ġon ĠUS VC Ġ( Our ) Ġand ĠPitch Net . ĠBesides , Ġthe Ġconverted Ġsamples Ġprovided Ġby ĠB IB REF 0 Ġwas Ġalso Ġincluded Ġto Ġgive Ġa Ġmore Ġconvincing Ġevaluation . ĠAs Ġshown Ġby ĠTab . ĠT AB REF 15 , Ġthe Ġnatural ness Ġand Ġsimilarity Ġof Ġour Ġmethod Ġare Ġboth Ġhigher Ġthan Ġthe Ġother Ġtwo Ġones . ĠOur Ġimplementation Ġof ĠUS VC Ġperformed Ġslightly Ġlower Ġthan Ġthe Ġoriginal Ġauthor 's Ġbecause Ġwe Ġcannot Ġfully Ġreproduce Ġthe Ġresults Ġof Ġthem . </s> Next Ġwe Ġqual itatively Ġanalyze Ġthe Ġinfluence Ġof Ġinput Ġpitch Ġin Ġour Ġmethod . ĠWe Ġused Ġdifferent Ġpitch Ġas Ġinput Ġto Ġobserve Ġhow Ġthe Ġoutput Ġpitch Ġwould Ġchange Ġalong Ġwith Ġthe Ġinput Ġpitch . ĠThe Ġinput Ġpitch Ġwas Ġmultiplied Ġby Ġ0 . 7 , Ġ1 . 0 Ġand Ġ1 . 3 Ġrespectively . ĠAnd Ġthe Ġoutput Ġpitch Ġwas Ġalso Ġextracted Ġby Ġthe Ġpitch Ġtracker Ġof Ġthe Ġlib ros a Ġpackage . ĠFig . ĠFIG REF 16 Ġplots Ġthe Ġpitch Ġof Ġinput Ġaudio Ġand Ġoutput Ġaudio Ġwith Ġdifferent Ġpitch Ġas Ġinput Ġwhile Ġkeeping Ġthe Ġtarget Ġsinger Ġthe Ġsame . ĠAs Ġshown Ġby ĠFig . ĠFIG REF 16 , Ġthe Ġoutput Ġpitch Ġchanges Ġsignificantly Ġalong Ġwith Ġthe Ġinput Ġpitch . ĠThe Ġexamples Ġare Ġalso Ġpresented Ġat Ġour Ġwebsite . </s> Conclusion </s> In Ġthis Ġpaper , Ġa Ġnovel Ġun super vised Ġsinging Ġvoice Ġconversion Ġmethod Ġnamed ĠPitch Net Ġis Ġproposed . ĠA Ġpitch Ġregression Ġnetwork Ġis Ġemployed Ġto Ġrender Ġan Ġadvers arial Ġloss Ġseparating Ġpitch Ġrelated Ġinformation Ġfrom Ġthe Ġlatent Ġspace Ġin Ġaut oen c oder . ĠAfter Ġthe ĠWave Net - like Ġenc oder , Ġa Ġsinger Ġand Ġpitch Ġinvari ant Ġrepresentation Ġis Ġgenerated Ġand Ġthen Ġfed Ġinto Ġthe ĠWave Net Ġdec oder Ġconditioning Ġon Ġthe Ġsinger Ġembed ding Ġand Ġthe Ġextracted Ġpitch Ġto Ġreconstruct Ġthe Ġtarget Ġsinging Ġvoice . ĠOur Ġmethod Ġoutper forms Ġthe Ġexisting Ġun super vised Ġsinging Ġvoice Ġconversion Ġmethod Ġand Ġachieves Ġflexible Ġpitch Ġmanipulation . </s> Acknowled gements </s> The Ġauthors Ġwould Ġlike Ġto Ġthanks ĠKun ĠXu Ġand Ġother Ġmembers Ġin Ġthe ĠTen cent ĠAI ĠLab Ġteam Ġfor Ġdiscussions Ġand Ġsuggestions .