Transformer over BERT (ToBERT)
EXTRACTIVE
The transformer layer
ABSTRACTIVE
In this paper, we propose a method that builds upon BERT's architecture. We split the input text sequence into shorter segments in order to obtain a representation for each of them using BERT. Then, we use either a recurrent LSTM BIBREF10 network, or another Transformer, to perform the actual classification. We call these techniques Recurrence over BERT (RoBERT) and Transformer over BERT (ToBERT). Given that these models introduce a hierarchy of representations (segment-wise and document-wise), we refer to them as Hierarchical Transformers. To the best of our knowledge, no attempt has been done before to use the Transformer architecture for classification of such long sequences. In this paper, we presented two methods for long documents using BERT model: RoBERT and ToBERT. We evaluated our experiments on two classification tasks - customer satisfaction prediction and topic identification - using 3 datasets: CSAT, 20newsgroups and Fisher. We observed that ToBERT outperforms RoBERT on pre-trained BERT features and fine-tuned BERT features for all our tasks. Also, we noticed that fine-tuned BERT performs better than pre-trained BERT. We have shown that both RoBERT and ToBERT improved the simple baselines of taking an average (or the most frequent) of segment-wise predictions for long documents to obtain final prediction. Position embeddings did not significantly affect our models performance, but slightly improved the accuracy on the CSAT task. We obtained the best results on Fisher dataset and good improvements for CSAT task compared to the CNN baseline. It is interesting to note that the longer the average input in a given task, the bigger improvement we observe w.r.t. the baseline for that task. Our results confirm that both RoBERT and ToBERT can be used for long sequences with competitive performance and quick fine-tuning procedure. For future work, we shall focus on training models on long documents directly (i.e. in an end-to-end manner).
In this paper, we propose a method that builds upon BERT's architecture. We split the input text sequence into shorter segments in order to obtain a representation for each of them using BERT. Then, we use either a recurrent LSTM BIBREF10 network, or another Transformer, to perform the actual classification. We call these techniques Recurrence over BERT (RoBERT) and Transformer over BERT (ToBERT). Given that these models introduce a hierarchy of representations (segment-wise and document-wise), we refer to them as Hierarchical Transformers. To the best of our knowledge, no attempt has been done before to use the Transformer architecture for classification of such long sequences. Table TABREF25 presents results using pre-trained BERT features. We extracted features from the pooled output of final transformer block as these were shown to be working well for most of the tasks BIBREF1. The features extracted from a pre-trained BERT model without any fine-tuning lead to a sub-par performance. However, We also notice that ToBERT model exploited the pre-trained BERT features better than RoBERT. It also converged faster than RoBERT. Table TABREF26 shows results using features extracted after fine-tuning BERT model with our datasets. Significant improvements can be observed compared to using pre-trained BERT features. Also, it can be noticed that ToBERT outperforms RoBERT on Fisher and 20newsgroups dataset by 13.63% and 0.81% respectively. On CSAT, ToBERT performs slightly worse than RoBERT but it is not statistically significant as this dataset is small.
1910.10781
Introduction </s> B id irection al ĠEnc oder ĠRepresent ations Ġfrom ĠTransformers Ġ( BER T ) Ġis Ġa Ġnovel ĠTrans former ĠB IB REF 0 Ġmodel , Ġwhich Ġrecently Ġachieved Ġstate - of - the - art Ġperformance Ġin Ġseveral Ġlanguage Ġunderstanding Ġtasks , Ġsuch Ġas Ġquestion Ġanswering , Ġnatural Ġlanguage Ġinference , Ġsemantic Ġsimilarity , Ġsentiment Ġanalysis , Ġand Ġothers ĠB IB REF 1 . ĠWhile Ġwell - su ited Ġto Ġdealing Ġwith Ġrelatively Ġshort Ġsequences , ĠTransformers Ġsuffer Ġfrom Ġa Ġmajor Ġissue Ġthat Ġhind ers Ġtheir Ġapplic ability Ġin Ġclassification Ġof Ġlong Ġsequences , Ġi . e . Ġthey Ġare Ġable Ġto Ġconsume Ġonly Ġa Ġlimited Ġcontext Ġof Ġsymbols Ġas Ġtheir Ġinput ĠB IB REF 2 . </s> There Ġare Ġseveral Ġnatural Ġlanguage Ġ( N LP ) Ġprocessing Ġtasks Ġthat Ġinvolve Ġsuch Ġlong Ġsequences . ĠOf Ġparticular Ġinterest Ġare Ġtopic Ġidentification Ġof Ġspoken Ġconversations ĠB IB REF 3 , ĠB IB REF 4 , ĠB IB REF 5 Ġand Ġcall Ġcenter Ġcustomer Ġsatisfaction Ġprediction ĠB IB REF 6 , ĠB IB REF 7 , ĠB IB REF 8 , ĠB IB REF 9 . ĠCall Ġcenter Ġconversations , Ġwhile Ġusually Ġquite Ġshort Ġand Ġto Ġthe Ġpoint , Ġoften Ġinvolve Ġagents Ġtrying Ġto Ġsolve Ġvery Ġcomplex Ġissues Ġthat Ġthe Ġcustomers Ġexperience , Ġresulting Ġin Ġsome Ġcalls Ġtaking Ġeven Ġan Ġhour Ġor Ġmore . ĠFor Ġspeech Ġanalytics Ġpurposes , Ġthese Ġcalls Ġare Ġtypically Ġtransc ribed Ġusing Ġan Ġautomatic Ġspeech Ġrecognition Ġ( AS R ) Ġsystem , Ġand Ġprocessed Ġin Ġtextual Ġrepresentations Ġfurther Ġdown Ġthe ĠN LP Ġpipeline . ĠThese Ġtranscripts Ġsometimes Ġexceed Ġthe Ġlength Ġof Ġ5000 Ġwords . ĠFurthermore , Ġtemporal Ġinformation Ġmight Ġplay Ġan Ġimportant Ġrole Ġin Ġtasks Ġlike ĠCS AT . ĠFor Ġexample , Ġa Ġcustomer Ġmay Ġbe Ġangry Ġat Ġthe Ġbeginning Ġof Ġthe Ġcall , Ġbut Ġafter Ġher Ġissue Ġis Ġresolved , Ġshe Ġwould Ġbe Ġvery Ġsatisfied Ġwith Ġthe Ġway Ġit Ġwas Ġhandled . ĠTherefore , Ġsimple Ġbag Ġof Ġwords Ġmodels , Ġor Ġany Ġmodel Ġthat Ġdoes Ġnot Ġinclude Ġtemporal Ġdependencies Ġbetween Ġthe Ġinputs , Ġmay Ġnot Ġbe Ġwell - su ited Ġto Ġhandle Ġthis Ġcategory Ġof Ġtasks . ĠThis Ġmotiv ates Ġus Ġto Ġemploy Ġmodel Ġsuch Ġas ĠB ERT Ġin Ġthis Ġtask . </s> In Ġthis Ġpaper , Ġwe Ġpropose Ġa Ġmethod Ġthat Ġbuilds Ġupon ĠB ERT 's Ġarchitecture . ĠWe Ġsplit Ġthe Ġinput Ġtext Ġsequence Ġinto Ġshorter Ġsegments Ġin Ġorder Ġto Ġobtain Ġa Ġrepresentation Ġfor Ġeach Ġof Ġthem Ġusing ĠB ERT . ĠThen , Ġwe Ġuse Ġeither Ġa Ġrecurrent ĠL ST M ĠB IB REF 10 Ġnetwork , Ġor Ġanother ĠTrans former , Ġto Ġperform Ġthe Ġactual Ġclassification . ĠWe Ġcall Ġthese Ġtechniques ĠRec urrence Ġover ĠB ERT Ġ( Ro BER T ) Ġand ĠTrans former Ġover ĠB ERT Ġ( To BER T ). ĠGiven Ġthat Ġthese Ġmodels Ġintroduce Ġa Ġhierarchy Ġof Ġrepresentations Ġ( se gment - wise Ġand Ġdocument - wise ), Ġwe Ġrefer Ġto Ġthem Ġas ĠHier arch ical ĠTransformers . ĠTo Ġthe Ġbest Ġof Ġour Ġknowledge , Ġno Ġattempt Ġhas Ġbeen Ġdone Ġbefore Ġto Ġuse Ġthe ĠTrans former Ġarchitecture Ġfor Ġclassification Ġof Ġsuch Ġlong Ġsequences . </s> Our Ġnovel Ġcontributions Ġare : </s> Two Ġextensions Ġ- ĠRo BER T Ġand ĠTo BER T Ġ- Ġto Ġthe ĠB ERT Ġmodel , Ġwhich Ġenable Ġits Ġapplication Ġin Ġclassification Ġof Ġlong Ġtexts Ġby Ġperforming Ġsegment ation Ġand Ġusing Ġanother Ġlayer Ġon Ġtop Ġof Ġthe Ġsegment Ġrepresentations . </s> State - of - the - art Ġresults Ġon Ġthe ĠFisher Ġtopic Ġclassification Ġtask . </s> Sign ificant Ġimprovement Ġon Ġthe ĠCS AT Ġprediction Ġtask Ġover Ġthe ĠMS - CNN Ġmodel . </s> Related Ġwork </s> Several Ġdimension ality Ġreduction Ġalgorithms Ġsuch Ġas ĠR BM , Ġaut oen cod ers , Ġsub space Ġmult in omial Ġmodels Ġ( S MM ) Ġare Ġused Ġto Ġobtain Ġa Ġlow Ġdimensional Ġrepresentation Ġof Ġdocuments Ġfrom Ġa Ġsimple ĠB OW Ġrepresentation Ġand Ġthen Ġclassify Ġit Ġusing Ġa Ġsimple Ġlinear Ġclass ifiers ĠB IB REF 11 , ĠB IB REF 12 , ĠB IB REF 13 , ĠB IB REF 4 . ĠIn ĠB IB REF 14 Ġhierarchical Ġattention Ġnetworks Ġare Ġused Ġfor Ġdocument Ġclassification . ĠThey Ġevaluate Ġtheir Ġmodel Ġon Ġseveral Ġdatasets Ġwith Ġaverage Ġnumber Ġof Ġwords Ġaround Ġ150 . ĠCharacter - level ĠCNN Ġare Ġexplored Ġin ĠB IB REF 15 Ġbut Ġit Ġis Ġprohib itive Ġfor Ġvery Ġlong Ġdocuments . ĠIn ĠB IB REF 16 , Ġdataset Ġcollected Ġfrom Ġar X iv Ġpapers Ġis Ġused Ġfor Ġclassification . ĠFor Ġclassification , Ġthey Ġsample Ġrandom Ġblocks Ġof Ġwords Ġand Ġuse Ġthem Ġtogether Ġfor Ġclassification Ġinstead Ġof Ġusing Ġfull Ġdocument Ġwhich Ġmay Ġwork Ġwell Ġas Ġar X iv Ġpapers Ġare Ġusually Ġcoherent Ġand Ġwell Ġwritten Ġon Ġa Ġwell Ġdefined Ġtopic . ĠTheir Ġmethod Ġmay Ġnot Ġwork Ġwell Ġon Ġspoken Ġconversations Ġas Ġrandom Ġblock Ġof Ġwords Ġusually Ġdo Ġnot Ġrepresent Ġtopic Ġof Ġfull Ġconversation . </s> Several Ġresearchers Ġaddressed Ġthe Ġproblem Ġof Ġpredicting Ġcustomer Ġsatisfaction ĠB IB REF 6 , ĠB IB REF 7 , ĠB IB REF 8 , ĠB IB REF 9 . ĠIn Ġmost Ġof Ġthese Ġworks , Ġlog istic Ġregression , ĠS VM , ĠCNN Ġare Ġapplied Ġon Ġdifferent Ġkinds Ġof Ġrepresentations . </s> In ĠB IB REF 17 , Ġauthors Ġuse ĠB ERT Ġfor Ġdocument Ġclassification Ġbut Ġthe Ġaverage Ġdocument Ġlength Ġis Ġless Ġthan ĠB ERT Ġmaximum Ġlength Ġ512 . ĠTrans former XL ĠB IB REF 2 Ġis Ġan Ġextension Ġto Ġthe ĠTrans former Ġarchitecture Ġthat Ġallows Ġit Ġto Ġbetter Ġdeal Ġwith Ġlong Ġinputs Ġfor Ġthe Ġlanguage Ġmodelling Ġtask . ĠIt Ġrelies Ġon Ġthe Ġauto - reg ressive Ġproperty Ġof Ġthe Ġmodel , Ġwhich Ġis Ġnot Ġthe Ġcase Ġin Ġour Ġtasks . </s> Method Ġ: :: ĠB ERT </s> Because Ġour Ġwork Ġbuilds Ġheavily Ġupon ĠB ERT , Ġwe Ġprovide Ġa Ġbrief Ġsummary Ġof Ġits Ġfeatures . ĠB ERT Ġis Ġbuilt Ġupon Ġthe ĠTrans former Ġarchitecture ĠB IB REF 0 , Ġwhich Ġuses Ġself - att ention , Ġfeed - forward Ġlayers , Ġresidual Ġconnections Ġand Ġlayer Ġnormal ization Ġas Ġthe Ġmain Ġbuilding Ġblocks . ĠIt Ġhas Ġtwo Ġpre - training Ġobjectives : </s> Mask ed Ġlanguage Ġmodelling Ġ- Ġsome Ġof Ġthe Ġwords Ġin Ġa Ġsentence Ġare Ġbeing Ġmasked Ġand Ġthe Ġmodel Ġhas Ġto Ġpredict Ġthem Ġbased Ġon Ġthe Ġcontext Ġ( note Ġthe Ġdifference Ġfrom Ġthe Ġtypical Ġaut ore gressive Ġlanguage Ġmodel Ġtraining Ġobjective ); </s> Next Ġsentence Ġprediction Ġ- Ġgiven Ġtwo Ġinput Ġsequences , Ġdecide Ġwhether Ġthe Ġsecond Ġone Ġis Ġthe Ġnext Ġsentence Ġor Ġnot . </s> BER T Ġhas Ġbeen Ġshown Ġto Ġbeat Ġthe Ġstate - of - the - art Ġperformance Ġon Ġ11 Ġtasks Ġwith Ġno Ġmodifications Ġto Ġthe Ġmodel Ġarchitecture , Ġbesides Ġadding Ġa Ġtask - specific Ġoutput Ġlayer ĠB IB REF 1 . ĠWe Ġfollow Ġsame Ġprocedure Ġsuggested Ġin ĠB IB REF 1 Ġfor Ġour Ġtasks . ĠFig . ĠFIG REF 8 Ġshows Ġthe ĠB ERT Ġmodel Ġfor Ġclassification . ĠWe Ġobtain Ġtwo Ġkinds Ġof Ġrepresentation Ġfrom ĠB ERT : Ġpooled Ġoutput Ġfrom Ġlast Ġtransformer Ġblock , Ġden oted Ġby ĠH , Ġand Ġposterior Ġprobabilities , Ġden oted Ġby ĠP . ĠThere Ġare Ġtwo Ġvariants Ġof ĠB ERT Ġ- ĠB ERT - Base Ġand ĠB ERT - Large . ĠIn Ġthis Ġwork Ġwe Ġare Ġusing ĠB ERT - Base Ġfor Ġfaster Ġtraining Ġand Ġexperimentation , Ġhowever , Ġour Ġmethods Ġare Ġapplicable Ġto ĠB ERT - Large Ġas Ġwell . ĠB ERT - Base Ġand ĠB ERT - Large Ġare Ġdifferent Ġin Ġmodel Ġparameters Ġsuch Ġas Ġnumber Ġof Ġtransformer Ġblocks , Ġnumber Ġof Ġself - att ention Ġheads . ĠTotal Ġnumber Ġof Ġparameters Ġin ĠB ERT - Base Ġare Ġ110 M Ġand Ġ340 M Ġin ĠB ERT - Large . </s> BER T Ġsuffers Ġfrom Ġmajor Ġlimitations Ġin Ġterms Ġof Ġhandling Ġlong Ġsequences . ĠFirstly , Ġthe Ġself - att ention Ġlayer Ġhas Ġa Ġquad r atic Ġcomplexity Ġ$ O ( n ^ 2 ) $ Ġin Ġterms Ġof Ġthe Ġsequence Ġlength Ġ$ n $ ĠB IB REF 0 . ĠSecondly , ĠB ERT Ġuses Ġa Ġlearned Ġpositional Ġembed d ings Ġscheme ĠB IB REF 1 , Ġwhich Ġmeans Ġthat Ġit Ġwon 't Ġlikely Ġbe Ġable Ġto Ġgeneral ize Ġto Ġpositions Ġbeyond Ġthose Ġseen Ġin Ġthe Ġtraining Ġdata . </s> To Ġinvestigate Ġthe Ġeffect Ġof Ġfine - tun ing ĠB ERT Ġon Ġtask Ġperformance , Ġwe Ġuse Ġeither Ġthe Ġpre - trained ĠB ERT Ġweights , Ġor Ġthe Ġweights Ġfrom Ġa ĠB ERT Ġfine - tun ed Ġon Ġthe Ġtask - specific Ġdataset Ġon Ġa Ġsegment - level Ġ( i . e . Ġwe Ġpreserve Ġthe Ġoriginal Ġlabel Ġbut Ġfine - t une Ġon Ġeach Ġsegment Ġseparately Ġinstead Ġof Ġon Ġthe Ġwhole Ġtext Ġsequence ). ĠWe Ġcompare Ġthese Ġresults Ġto Ġusing Ġthe Ġfine - tun ed Ġsegment - level ĠB ERT Ġpredictions Ġdirectly Ġas Ġinputs Ġto Ġthe Ġnext Ġlayer . </s> Method Ġ: :: ĠRec urrence Ġover ĠB ERT </s> Given Ġthat ĠB ERT Ġis Ġlimited Ġto Ġa Ġparticular Ġinput Ġlength , Ġwe Ġsplit Ġthe Ġinput Ġsequence Ġinto Ġsegments Ġof Ġa Ġfixed Ġsize Ġwith Ġoverlap . ĠFor Ġeach Ġof Ġthese Ġsegments , Ġwe Ġobtain ĠH Ġor ĠP Ġfrom ĠB ERT Ġmodel . ĠWe Ġthen Ġstack Ġthese Ġsegment - level Ġrepresentations Ġinto Ġa Ġsequence , Ġwhich Ġserves Ġas Ġinput Ġto Ġa Ġsmall Ġ( 100 - dimensional ) ĠL ST M Ġlayer . ĠIts Ġoutput Ġserves Ġas Ġa Ġdocument Ġembed ding . ĠFinally , Ġwe Ġuse Ġtwo Ġfully Ġconnected Ġlayers Ġwith ĠRe LU Ġ( 30 - dimensional ) Ġand Ġsoft max Ġ( the Ġsame Ġdimension ality Ġas Ġthe Ġnumber Ġof Ġclasses ) Ġactiv ations Ġto Ġobtain Ġthe Ġfinal Ġpredictions . </s> With Ġthis Ġapproach , Ġwe Ġovercome ĠB ERT 's Ġcomputational Ġcomplexity , Ġreducing Ġit Ġto Ġ$ O ( n / k Ġ* Ġk ^ 2 ) Ġ= ĠO ( n k ) $ Ġfor ĠRo BER T , Ġwith Ġ$ k $ Ġden oting Ġthe Ġsegment Ġsize Ġ( the ĠL ST M Ġcomponent Ġhas Ġnegligible Ġlinear Ġcomplexity Ġ$ O ( k ) $ ). ĠThe Ġpositional Ġembed d ings Ġare Ġalso Ġno Ġlonger Ġan Ġissue . </s> Method Ġ: :: ĠTrans former Ġover ĠB ERT </s> Given Ġthat ĠTransformers ' Ġedge Ġover Ġrecurrent Ġnetworks Ġis Ġtheir Ġability Ġto Ġeffectively Ġcapture Ġlong Ġdistance Ġrelationships Ġbetween Ġwords Ġin Ġa Ġsequence ĠB IB REF 0 , Ġwe Ġexperiment Ġwith Ġreplacing Ġthe ĠL ST M Ġrecurrent Ġlayer Ġin Ġfavor Ġof Ġa Ġsmall ĠTrans former Ġmodel Ġ( 2 Ġlayers Ġof Ġtransformer Ġbuilding Ġblock Ġcontaining Ġself - att ention , Ġfully Ġconnected , Ġetc .). ĠTo Ġinvestigate Ġif Ġpreserving Ġthe Ġinformation Ġabout Ġthe Ġinput Ġsequence Ġorder Ġis Ġimportant , Ġwe Ġalso Ġbuild Ġa Ġvariant Ġof ĠTo BER T Ġwhich Ġlearns Ġpositional Ġembed d ings Ġat Ġthe Ġsegment - level Ġrepresentations Ġ( but Ġis Ġlimited Ġto Ġsequences Ġof Ġlength Ġseen Ġduring Ġthe Ġtraining ). </s> To BER T 's Ġcomputational Ġcomplexity Ġ$ O (\ frac { n ^ 2 }{ k ^ 2 }) $ Ġis Ġas ym pt ot ically Ġinferior Ġto ĠRo BER T , Ġas Ġthe Ġtop - level ĠTrans former Ġmodel Ġagain Ġsuffers Ġfrom Ġquad r atic Ġcomplexity Ġin Ġthe Ġnumber Ġof Ġsegments . ĠHowever , Ġin Ġpractice Ġthis Ġnumber Ġis Ġmuch Ġsmaller Ġthan Ġthe Ġinput Ġsequence Ġlength Ġ($ {\ frac { n }{ k }} Ġ<< Ġn $ ), Ġso Ġwe Ġhaven 't Ġobserved Ġperformance Ġor Ġmemory Ġissues Ġwith Ġour Ġdatasets . </s> Exper iments </s> We Ġevaluated Ġour Ġmodels Ġon Ġ3 Ġdifferent Ġdatasets : </s> CS AT Ġdataset Ġfor ĠCS AT Ġprediction , Ġconsisting Ġof Ġspoken Ġtranscripts Ġ( automatic Ġvia ĠAS R ). </s> 20 Ġnews groups Ġfor Ġtopic Ġidentification Ġtask , Ġconsisting Ġof Ġwritten Ġtext ; </s> F isher ĠPhase Ġ1 Ġcorpus Ġfor Ġtopic Ġidentification Ġtask , Ġconsisting Ġof Ġspoken Ġtranscripts Ġ( man ual ); </s> Exper iments Ġ: :: ĠCS AT </s> CS AT Ġdataset Ġconsists Ġof ĠUS ĠEnglish Ġtelephone Ġspeech Ġfrom Ġcall Ġcenters . ĠFor Ġeach Ġcall Ġin Ġthis Ġdataset , Ġcustomers Ġparticipated Ġin Ġthat Ġcall Ġgave Ġa Ġrating Ġon Ġhis Ġexperience Ġwith Ġagent . ĠOriginally , Ġthis Ġdataset Ġhas Ġlabels Ġrated Ġon Ġa Ġscale Ġ1 - 9 Ġwith Ġ9 Ġbeing Ġextremely Ġsatisfied Ġand Ġ1 Ġbeing Ġextremely Ġdissatisfied . ĠFig . ĠFIG REF 16 Ġshows Ġthe Ġhist ogram Ġof Ġratings Ġfor Ġour Ġdataset . ĠAs Ġthe Ġdistribution Ġis Ġskewed Ġtowards Ġextremes , Ġwe Ġchoose Ġto Ġdo Ġbinary Ġclassification Ġwith Ġratings Ġabove Ġ4 . 5 Ġas Ġsatisfied Ġand Ġbelow Ġ4 . 5 Ġas Ġdissatisfied . ĠQuant ization Ġof Ġratings Ġalso Ġhelped Ġus Ġto Ġcreate Ġa Ġbalanced Ġdataset . ĠThis Ġdataset Ġcontains Ġ4 331 Ġcalls Ġand Ġwe Ġsplit Ġthem Ġinto Ġ3 Ġsets Ġfor Ġour Ġexperiments : Ġ28 66 Ġcalls Ġfor Ġtraining , Ġ36 2 Ġcalls Ġfor Ġvalidation Ġand , Ġfinally , Ġ110 3 Ġcalls Ġfor Ġtesting . </s> We Ġobtained Ġthe Ġtranscripts Ġby Ġemploying Ġan ĠAS R Ġsystem . ĠThe ĠAS R Ġsystem Ġuses ĠTD NN - L ST M Ġacoustic Ġmodel Ġtrained Ġon ĠFisher Ġand ĠSwitch board Ġdatasets Ġwith Ġlatt ice - free Ġmaximum Ġmutual Ġinformation Ġcriterion ĠB IB REF 18 . ĠThe Ġword Ġerror Ġrates Ġusing Ġfour - gram Ġlanguage Ġmodels Ġwere Ġ9 . 2 % Ġand Ġ17 . 3 % Ġrespectively Ġon ĠSwitch board Ġand ĠCall Home Ġportions Ġof ĠEval 2000 Ġdataset . </s> Exper iments Ġ: :: Ġ20 Ġnews groups </s> 20 Ġnews groups Ġdata Ġset Ġis Ġone Ġof Ġthe Ġfrequently Ġused Ġdatasets Ġin Ġthe Ġtext Ġprocessing Ġcommunity Ġfor Ġtext Ġclassification Ġand Ġtext Ġclust ering . ĠThis Ġdata Ġset Ġcontains Ġapproximately Ġ20 , 000 ĠEnglish Ġdocuments Ġfrom Ġ20 Ġtopics Ġto Ġbe Ġidentified , Ġwith Ġ113 14 Ġdocuments Ġfor Ġtraining Ġand Ġ75 32 Ġfor Ġtesting . ĠIn Ġthis Ġwork , Ġwe Ġused Ġonly Ġ90 % Ġof Ġdocuments Ġfor Ġtraining Ġand Ġthe Ġremaining Ġ10 % Ġfor Ġvalidation . ĠFor Ġfair Ġcomparison Ġwith Ġother Ġpublications , Ġwe Ġused Ġ53 160 Ġwords Ġvocabulary Ġset Ġavailable Ġin Ġthe Ġdatasets Ġwebsite . </s> Exper iments Ġ: :: ĠFisher </s> F isher ĠPhase Ġ1 ĠUS ĠEnglish Ġcorpus Ġis Ġoften Ġused Ġfor Ġautomatic Ġspeech Ġrecognition Ġin Ġspeech Ġcommunity . ĠIn Ġthis Ġwork , Ġwe Ġused Ġit Ġfor Ġtopic Ġidentification Ġas Ġin ĠB IB REF 3 . ĠThe Ġdocuments Ġare Ġ10 - minute Ġlong Ġtelephone Ġconversations Ġbetween Ġtwo Ġpeople Ġdiscussing Ġa Ġgiven Ġtopic . ĠWe Ġused Ġsame Ġtraining Ġand Ġtest Ġsplits Ġas ĠB IB REF 3 Ġin Ġwhich Ġ13 74 Ġand Ġ13 72 Ġdocuments Ġare Ġused Ġfor Ġtraining Ġand Ġtesting Ġrespectively . ĠFor Ġvalidation Ġof Ġour Ġmodel , Ġwe Ġused Ġ10 % Ġof Ġtraining Ġdataset Ġand Ġthe Ġremaining Ġ90 % Ġwas Ġused Ġfor Ġactual Ġmodel Ġtraining . ĠThe Ġnumber Ġof Ġtopics Ġin Ġthis Ġdata Ġset Ġis Ġ40 . </s> Exper iments Ġ: :: ĠDat as et ĠStatistics </s> Table ĠT AB REF 22 Ġshows Ġstatistics Ġof Ġour Ġdatasets . ĠIt Ġcan Ġbe Ġobserved Ġthat Ġaverage Ġlength Ġof ĠFisher Ġis Ġmuch Ġhigher Ġthan Ġ20 Ġnews groups Ġand ĠCS AT . ĠCum ulative Ġdistribution Ġof Ġdocument Ġlengths Ġfor Ġeach Ġdataset Ġis Ġshown Ġin ĠFig . ĠFIG REF 21 . ĠIt Ġcan Ġbe Ġobserved Ġthat Ġalmost Ġall Ġof Ġthe Ġdocuments Ġin ĠFisher Ġdataset Ġhave Ġlength Ġmore Ġthan Ġ1000 Ġwords . ĠFor ĠCS AT , Ġmore Ġthan Ġ50 % Ġof Ġthe Ġdocuments Ġhave Ġlength Ġgreater Ġthan Ġ500 Ġand Ġfor Ġ20 news groups Ġonly Ġ10 % Ġof Ġthe Ġdocuments Ġhave Ġlength Ġgreater Ġthan Ġ500 . ĠNote Ġthat , Ġfor ĠCS AT Ġand Ġ20 news groups , Ġthere Ġare Ġfew Ġdocuments Ġwith Ġlength Ġmore Ġthan Ġ5000 . </s> Exper iments Ġ: :: ĠArchitecture Ġand ĠTraining ĠDetails </s> In Ġthis Ġwork , Ġwe Ġsplit Ġdocument Ġinto Ġsegments Ġof Ġ200 Ġtokens Ġwith Ġa Ġshift Ġof Ġ50 Ġtokens Ġto Ġextract Ġfeatures Ġfrom ĠB ERT Ġmodel . ĠFor ĠRo BER T , ĠL ST M Ġmodel Ġis Ġtrained Ġto Ġminimize Ġcross - ent ropy Ġloss Ġwith ĠAdam Ġoptim izer ĠB IB REF 19 . ĠThe Ġinitial Ġlearning Ġrate Ġis Ġset Ġto Ġ$ 0 . 001 $ Ġand Ġis Ġreduced Ġby Ġa Ġfactor Ġof Ġ$ 0 . 95 $ Ġif Ġvalidation Ġloss Ġdoes Ġnot Ġdecrease Ġfor Ġ3 - ep och s . ĠFor ĠTo BER T , Ġthe ĠTrans former Ġis Ġtrained Ġwith Ġthe Ġdefault ĠB ERT Ġversion Ġof ĠAdam Ġoptim izer ĠB IB REF 1 Ġwith Ġan Ġinitial Ġlearning Ġrate Ġof Ġ$ 5 e $ - 5 . ĠWe Ġreport Ġaccuracy Ġin Ġall Ġof Ġour Ġexperiments . ĠWe Ġchose Ġa Ġmodel Ġwith Ġthe Ġbest Ġvalidation Ġaccuracy Ġto Ġcalculate Ġaccuracy Ġon Ġthe Ġtest Ġset . ĠTo Ġacc om od ate Ġfor Ġnon - d etermin ism Ġof Ġsome ĠT ensor Flow ĠGPU Ġoperations , Ġwe Ġreport Ġaccuracy Ġaveraged Ġover Ġ5 Ġruns . </s> Results </s> Table ĠT AB REF 25 Ġpresents Ġresults Ġusing Ġpre - trained ĠB ERT Ġfeatures . ĠWe Ġextracted Ġfeatures Ġfrom Ġthe Ġpooled Ġoutput Ġof Ġfinal Ġtransformer Ġblock Ġas Ġthese Ġwere Ġshown Ġto Ġbe Ġworking Ġwell Ġfor Ġmost Ġof Ġthe Ġtasks ĠB IB REF 1 . ĠThe Ġfeatures Ġextracted Ġfrom Ġa Ġpre - trained ĠB ERT Ġmodel Ġwithout Ġany Ġfine - tun ing Ġlead Ġto Ġa Ġsub - par Ġperformance . ĠHowever , ĠWe Ġalso Ġnotice Ġthat ĠTo BER T Ġmodel Ġexploited Ġthe Ġpre - trained ĠB ERT Ġfeatures Ġbetter Ġthan ĠRo BER T . ĠIt Ġalso Ġconver ged Ġfaster Ġthan ĠRo BER T . ĠTable ĠT AB REF 26 Ġshows Ġresults Ġusing Ġfeatures Ġextracted Ġafter Ġfine - tun ing ĠB ERT Ġmodel Ġwith Ġour Ġdatasets . ĠSignificant Ġimprovements Ġcan Ġbe Ġobserved Ġcompared Ġto Ġusing Ġpre - trained ĠB ERT Ġfeatures . ĠAlso , Ġit Ġcan Ġbe Ġnoticed Ġthat ĠTo BER T Ġoutper forms ĠRo BER T Ġon ĠFisher Ġand Ġ20 news groups Ġdataset Ġby Ġ13 . 63 % Ġand Ġ0 . 81 % Ġrespectively . ĠOn ĠCS AT , ĠTo BER T Ġperforms Ġslightly Ġworse Ġthan ĠRo BER T Ġbut Ġit Ġis Ġnot Ġstatistically Ġsignificant Ġas Ġthis Ġdataset Ġis Ġsmall . </s> Table ĠT AB REF 27 Ġpresents Ġresults Ġusing Ġfine - tun ed ĠB ERT Ġpredictions Ġinstead Ġof Ġthe Ġpooled Ġoutput Ġfrom Ġfinal Ġtransformer Ġblock . ĠFor Ġeach Ġdocument , Ġhaving Ġobtained Ġsegment - wise Ġpredictions Ġwe Ġcan Ġobtain Ġfinal Ġprediction Ġfor Ġthe Ġwhole Ġdocument Ġin Ġthree Ġways : </s> Comp ute Ġthe Ġaverage Ġof Ġall Ġsegment - wise Ġpredictions Ġand Ġfind Ġthe Ġmost Ġprobable Ġclass ; </s> Find Ġthe Ġmost Ġfrequently Ġpredicted Ġclass ; </s> Train Ġa Ġclassification Ġmodel . </s> It Ġcan Ġbe Ġobserved Ġfrom ĠTable ĠT AB REF 27 Ġthat Ġa Ġsimple Ġaveraging Ġoperation Ġor Ġtaking Ġmost Ġfrequent Ġpredicted Ġclass Ġworks Ġcompetitive ly Ġfor ĠCS AT Ġand Ġ20 news groups Ġbut Ġnot Ġfor Ġthe ĠFisher Ġdataset . ĠWe Ġbelieve Ġthe Ġimprovements Ġfrom Ġusing ĠRo BER T Ġor ĠTo BER T , Ġcompared Ġto Ġsimple Ġaveraging Ġor Ġmost Ġfrequent Ġoperations , Ġare Ġproportional Ġto Ġthe Ġfraction Ġof Ġlong Ġdocuments Ġin Ġthe Ġdataset . ĠCS AT Ġand Ġ20 news groups Ġhave Ġ( on Ġaverage ) Ġsignificantly Ġshorter Ġdocuments Ġthan ĠFisher , Ġas Ġseen Ġin ĠFig . ĠFIG REF 21 . ĠAlso , Ġsignificant Ġimprovements Ġfor ĠFisher Ġcould Ġbe Ġbecause Ġof Ġless Ġconfident Ġpredictions Ġfrom ĠB ERT Ġmodel Ġas Ġthis Ġdataset Ġhas Ġ40 Ġclasses . ĠFig . ĠFIG REF 31 Ġpresents Ġthe Ġcomparison Ġof Ġaverage Ġvoting Ġand ĠTo BER T Ġfor Ġvarious Ġdocument Ġlength Ġranges Ġfor ĠFisher Ġdataset . ĠWe Ġused Ġfine - tun ed ĠB ERT Ġsegment - level Ġpredictions Ġ( P ) Ġfor Ġthis Ġanalysis . ĠIt Ġcan Ġbe Ġobserved Ġthat ĠTo BER T Ġoutper forms Ġaverage Ġvoting Ġin Ġevery Ġinterval . ĠTo Ġthe Ġbest Ġof Ġour Ġknowledge , Ġthis Ġis Ġa Ġstate - of - the - art Ġresult Ġreported Ġon Ġthe ĠFisher Ġdataset . </s> Table ĠT AB REF 32 Ġpresents Ġthe Ġeffect Ġof Ġposition Ġembed d ings Ġon Ġthe Ġmodel Ġperformance . ĠIt Ġcan Ġbe Ġobserved Ġthat Ġposition Ġembed d ings Ġdid Ġnot Ġsignificantly Ġaffect Ġthe Ġmodel Ġperformance Ġfor ĠFisher Ġand Ġ20 news groups , Ġbut Ġthey Ġhelped Ġslightly Ġin ĠCS AT Ġprediction Ġ( an Ġabsolute Ġimprovement Ġof Ġ0 . 64 % ĠF 1 - score ). ĠWe Ġthink Ġthat Ġthis Ġis Ġexplained Ġby Ġthe Ġfact Ġthat ĠFisher Ġand Ġ20 news groups Ġare Ġtopic Ġidentification Ġtasks , Ġand Ġthe Ġtopic Ġdoes Ġnot Ġchange Ġmuch Ġthroughout Ġthese Ġdocuments . ĠHowever , ĠCS AT Ġmay Ġvary Ġduring Ġthe Ġcall , Ġand Ġin Ġsome Ġcases Ġa Ġnaive Ġassumption Ġthat Ġthe Ġsequential Ġnature Ġof Ġthe Ġtranscripts Ġis Ġirrelevant Ġmay Ġlead Ġto Ġwrong Ġconclusions . </s> Table ĠT AB REF 33 Ġcompares Ġour Ġresults Ġwith Ġprevious Ġworks . ĠIt Ġcan Ġbe Ġseen Ġthat Ġour Ġmodel ĠTo BER T Ġoutper forms ĠCNN Ġbased Ġexperiments Ġby Ġsignificant Ġmargin Ġon ĠCS AT Ġand ĠFisher Ġdatasets . ĠFor ĠCS AT Ġdataset , Ġwe Ġused Ġmulti - scale ĠCNN Ġ( MS - CNN ) Ġas Ġthe Ġbaseline , Ġgiven Ġits Ġstrong Ġresults Ġon ĠFisher Ġand Ġ20 news groups . ĠThe Ġsetup Ġwas Ġreplicated Ġfrom ĠB IB REF 5 Ġfor Ġcomparison . ĠWe Ġalso Ġsee Ġthat Ġour Ġresult Ġon Ġ20 Ġnews groups Ġis Ġ0 . 6 % Ġworse Ġthan Ġthe Ġstate - of - the - art . </s> Con clusions </s> In Ġthis Ġpaper , Ġwe Ġpresented Ġtwo Ġmethods Ġfor Ġlong Ġdocuments Ġusing ĠB ERT Ġmodel : ĠRo BER T Ġand ĠTo BER T . ĠWe Ġevaluated Ġour Ġexperiments Ġon Ġtwo Ġclassification Ġtasks Ġ- Ġcustomer Ġsatisfaction Ġprediction Ġand Ġtopic Ġidentification Ġ- Ġusing Ġ3 Ġdatasets : ĠCS AT , Ġ20 news groups Ġand ĠFisher . ĠWe Ġobserved Ġthat ĠTo BER T Ġoutper forms ĠRo BER T Ġon Ġpre - trained ĠB ERT Ġfeatures Ġand Ġfine - tun ed ĠB ERT Ġfeatures Ġfor Ġall Ġour Ġtasks . ĠAlso , Ġwe Ġnoticed Ġthat Ġfine - tun ed ĠB ERT Ġperforms Ġbetter Ġthan Ġpre - trained ĠB ERT . ĠWe Ġhave Ġshown Ġthat Ġboth ĠRo BER T Ġand ĠTo BER T Ġimproved Ġthe Ġsimple Ġbas elines Ġof Ġtaking Ġan Ġaverage Ġ( or Ġthe Ġmost Ġfrequent ) Ġof Ġsegment - wise Ġpredictions Ġfor Ġlong Ġdocuments Ġto Ġobtain Ġfinal Ġprediction . ĠPosition Ġembed d ings Ġdid Ġnot Ġsignificantly Ġaffect Ġour Ġmodels Ġperformance , Ġbut Ġslightly Ġimproved Ġthe Ġaccuracy Ġon Ġthe ĠCS AT Ġtask . ĠWe Ġobtained Ġthe Ġbest Ġresults Ġon ĠFisher Ġdataset Ġand Ġgood Ġimprovements Ġfor ĠCS AT Ġtask Ġcompared Ġto Ġthe ĠCNN Ġbaseline . ĠIt Ġis Ġinteresting Ġto Ġnote Ġthat Ġthe Ġlonger Ġthe Ġaverage Ġinput Ġin Ġa Ġgiven Ġtask , Ġthe Ġbigger Ġimprovement Ġwe Ġobserve Ġw . r . t . Ġthe Ġbaseline Ġfor Ġthat Ġtask . ĠOur Ġresults Ġconfirm Ġthat Ġboth ĠRo BER T Ġand ĠTo BER T Ġcan Ġbe Ġused Ġfor Ġlong Ġsequences Ġwith Ġcompetitive Ġperformance Ġand Ġquick Ġfine - tun ing Ġprocedure . ĠFor Ġfuture Ġwork , Ġwe Ġshall Ġfocus Ġon Ġtraining Ġmodels Ġon Ġlong Ġdocuments Ġdirectly Ġ( i . e . Ġin Ġan Ġend - to - end Ġmanner ).
Introduction Bidirectional Encoder Representations from Transformers (BERT) is a novel Transformer BIBREF0 model, which recently achieved state-of-the-art performance in several language understanding tasks, such as question answering, natural language inference, semantic similarity, sentiment analysis, and others BIBREF1. While well-suited to dealing with relatively short sequences, Transformers suffer from a major issue that hinders their applicability in classification of long sequences, i.e. they are able to consume only a limited context of symbols as their input BIBREF2. There are several natural language (NLP) processing tasks that involve such long sequences. Of particular interest are topic identification of spoken conversations BIBREF3, BIBREF4, BIBREF5 and call center customer satisfaction prediction BIBREF6, BIBREF7, BIBREF8, BIBREF9. Call center conversations, while usually quite short and to the point, often involve agents trying to solve very complex issues that the customers experience, resulting in some calls taking even an hour or more. For speech analytics purposes, these calls are typically transcribed using an automatic speech recognition (ASR) system, and processed in textual representations further down the NLP pipeline. These transcripts sometimes exceed the length of 5000 words. Furthermore, temporal information might play an important role in tasks like CSAT. For example, a customer may be angry at the beginning of the call, but after her issue is resolved, she would be very satisfied with the way it was handled. Therefore, simple bag of words models, or any model that does not include temporal dependencies between the inputs, may not be well-suited to handle this category of tasks. This motivates us to employ model such as BERT in this task. In this paper, we propose a method that builds upon BERT's architecture. We split the input text sequence into shorter segments in order to obtain a representation for each of them using BERT. Then, we use either a recurrent LSTM BIBREF10 network, or another Transformer, to perform the actual classification. We call these techniques Recurrence over BERT (RoBERT) and Transformer over BERT (ToBERT). Given that these models introduce a hierarchy of representations (segment-wise and document-wise), we refer to them as Hierarchical Transformers. To the best of our knowledge, no attempt has been done before to use the Transformer architecture for classification of such long sequences. Our novel contributions are: Two extensions - RoBERT and ToBERT - to the BERT model, which enable its application in classification of long texts by performing segmentation and using another layer on top of the segment representations. State-of-the-art results on the Fisher topic classification task. Significant improvement on the CSAT prediction task over the MS-CNN model. Related work Several dimensionality reduction algorithms such as RBM, autoencoders, subspace multinomial models (SMM) are used to obtain a low dimensional representation of documents from a simple BOW representation and then classify it using a simple linear classifiers BIBREF11, BIBREF12, BIBREF13, BIBREF4. In BIBREF14 hierarchical attention networks are used for document classification. They evaluate their model on several datasets with average number of words around 150. Character-level CNN are explored in BIBREF15 but it is prohibitive for very long documents. In BIBREF16, dataset collected from arXiv papers is used for classification. For classification, they sample random blocks of words and use them together for classification instead of using full document which may work well as arXiv papers are usually coherent and well written on a well defined topic. Their method may not work well on spoken conversations as random block of words usually do not represent topic of full conversation. Several researchers addressed the problem of predicting customer satisfaction BIBREF6, BIBREF7, BIBREF8, BIBREF9. In most of these works, logistic regression, SVM, CNN are applied on different kinds of representations. In BIBREF17, authors use BERT for document classification but the average document length is less than BERT maximum length 512. TransformerXL BIBREF2 is an extension to the Transformer architecture that allows it to better deal with long inputs for the language modelling task. It relies on the auto-regressive property of the model, which is not the case in our tasks. Method ::: BERT Because our work builds heavily upon BERT, we provide a brief summary of its features. BERT is built upon the Transformer architecture BIBREF0, which uses self-attention, feed-forward layers, residual connections and layer normalization as the main building blocks. It has two pre-training objectives: Masked language modelling - some of the words in a sentence are being masked and the model has to predict them based on the context (note the difference from the typical autoregressive language model training objective); Next sentence prediction - given two input sequences, decide whether the second one is the next sentence or not. BERT has been shown to beat the state-of-the-art performance on 11 tasks with no modifications to the model architecture, besides adding a task-specific output layer BIBREF1. We follow same procedure suggested in BIBREF1 for our tasks. Fig. FIGREF8 shows the BERT model for classification. We obtain two kinds of representation from BERT: pooled output from last transformer block, denoted by H, and posterior probabilities, denoted by P. There are two variants of BERT - BERT-Base and BERT-Large. In this work we are using BERT-Base for faster training and experimentation, however, our methods are applicable to BERT-Large as well. BERT-Base and BERT-Large are different in model parameters such as number of transformer blocks, number of self-attention heads. Total number of parameters in BERT-Base are 110M and 340M in BERT-Large. BERT suffers from major limitations in terms of handling long sequences. Firstly, the self-attention layer has a quadratic complexity $O(n^2)$ in terms of the sequence length $n$ BIBREF0. Secondly, BERT uses a learned positional embeddings scheme BIBREF1, which means that it won't likely be able to generalize to positions beyond those seen in the training data. To investigate the effect of fine-tuning BERT on task performance, we use either the pre-trained BERT weights, or the weights from a BERT fine-tuned on the task-specific dataset on a segment-level (i.e. we preserve the original label but fine-tune on each segment separately instead of on the whole text sequence). We compare these results to using the fine-tuned segment-level BERT predictions directly as inputs to the next layer. Method ::: Recurrence over BERT Given that BERT is limited to a particular input length, we split the input sequence into segments of a fixed size with overlap. For each of these segments, we obtain H or P from BERT model. We then stack these segment-level representations into a sequence, which serves as input to a small (100-dimensional) LSTM layer. Its output serves as a document embedding. Finally, we use two fully connected layers with ReLU (30-dimensional) and softmax (the same dimensionality as the number of classes) activations to obtain the final predictions. With this approach, we overcome BERT's computational complexity, reducing it to $O(n/k * k^2) = O(nk)$ for RoBERT, with $k$ denoting the segment size (the LSTM component has negligible linear complexity $O(k)$). The positional embeddings are also no longer an issue. Method ::: Transformer over BERT Given that Transformers' edge over recurrent networks is their ability to effectively capture long distance relationships between words in a sequence BIBREF0, we experiment with replacing the LSTM recurrent layer in favor of a small Transformer model (2 layers of transformer building block containing self-attention, fully connected, etc.). To investigate if preserving the information about the input sequence order is important, we also build a variant of ToBERT which learns positional embeddings at the segment-level representations (but is limited to sequences of length seen during the training). ToBERT's computational complexity $O(\frac{n^2}{k^2})$ is asymptotically inferior to RoBERT, as the top-level Transformer model again suffers from quadratic complexity in the number of segments. However, in practice this number is much smaller than the input sequence length (${\frac{n}{k}} << n$), so we haven't observed performance or memory issues with our datasets. Experiments We evaluated our models on 3 different datasets: CSAT dataset for CSAT prediction, consisting of spoken transcripts (automatic via ASR). 20 newsgroups for topic identification task, consisting of written text; Fisher Phase 1 corpus for topic identification task, consisting of spoken transcripts (manual); Experiments ::: CSAT CSAT dataset consists of US English telephone speech from call centers. For each call in this dataset, customers participated in that call gave a rating on his experience with agent. Originally, this dataset has labels rated on a scale 1-9 with 9 being extremely satisfied and 1 being extremely dissatisfied. Fig. FIGREF16 shows the histogram of ratings for our dataset. As the distribution is skewed towards extremes, we choose to do binary classification with ratings above 4.5 as satisfied and below 4.5 as dissatisfied. Quantization of ratings also helped us to create a balanced dataset. This dataset contains 4331 calls and we split them into 3 sets for our experiments: 2866 calls for training, 362 calls for validation and, finally, 1103 calls for testing. We obtained the transcripts by employing an ASR system. The ASR system uses TDNN-LSTM acoustic model trained on Fisher and Switchboard datasets with lattice-free maximum mutual information criterion BIBREF18. The word error rates using four-gram language models were 9.2% and 17.3% respectively on Switchboard and CallHome portions of Eval2000 dataset. Experiments ::: 20 newsgroups 20 newsgroups data set is one of the frequently used datasets in the text processing community for text classification and text clustering. This data set contains approximately 20,000 English documents from 20 topics to be identified, with 11314 documents for training and 7532 for testing. In this work, we used only 90% of documents for training and the remaining 10% for validation. For fair comparison with other publications, we used 53160 words vocabulary set available in the datasets website. Experiments ::: Fisher Fisher Phase 1 US English corpus is often used for automatic speech recognition in speech community. In this work, we used it for topic identification as in BIBREF3. The documents are 10-minute long telephone conversations between two people discussing a given topic. We used same training and test splits as BIBREF3 in which 1374 and 1372 documents are used for training and testing respectively. For validation of our model, we used 10% of training dataset and the remaining 90% was used for actual model training. The number of topics in this data set is 40. Experiments ::: Dataset Statistics Table TABREF22 shows statistics of our datasets. It can be observed that average length of Fisher is much higher than 20 newsgroups and CSAT. Cumulative distribution of document lengths for each dataset is shown in Fig. FIGREF21. It can be observed that almost all of the documents in Fisher dataset have length more than 1000 words. For CSAT, more than 50% of the documents have length greater than 500 and for 20newsgroups only 10% of the documents have length greater than 500. Note that, for CSAT and 20newsgroups, there are few documents with length more than 5000. Experiments ::: Architecture and Training Details In this work, we split document into segments of 200 tokens with a shift of 50 tokens to extract features from BERT model. For RoBERT, LSTM model is trained to minimize cross-entropy loss with Adam optimizer BIBREF19. The initial learning rate is set to $0.001$ and is reduced by a factor of $0.95$ if validation loss does not decrease for 3-epochs. For ToBERT, the Transformer is trained with the default BERT version of Adam optimizer BIBREF1 with an initial learning rate of $5e$-5. We report accuracy in all of our experiments. We chose a model with the best validation accuracy to calculate accuracy on the test set. To accomodate for non-determinism of some TensorFlow GPU operations, we report accuracy averaged over 5 runs. Results Table TABREF25 presents results using pre-trained BERT features. We extracted features from the pooled output of final transformer block as these were shown to be working well for most of the tasks BIBREF1. The features extracted from a pre-trained BERT model without any fine-tuning lead to a sub-par performance. However, We also notice that ToBERT model exploited the pre-trained BERT features better than RoBERT. It also converged faster than RoBERT. Table TABREF26 shows results using features extracted after fine-tuning BERT model with our datasets. Significant improvements can be observed compared to using pre-trained BERT features. Also, it can be noticed that ToBERT outperforms RoBERT on Fisher and 20newsgroups dataset by 13.63% and 0.81% respectively. On CSAT, ToBERT performs slightly worse than RoBERT but it is not statistically significant as this dataset is small. Table TABREF27 presents results using fine-tuned BERT predictions instead of the pooled output from final transformer block. For each document, having obtained segment-wise predictions we can obtain final prediction for the whole document in three ways: Compute the average of all segment-wise predictions and find the most probable class; Find the most frequently predicted class; Train a classification model. It can be observed from Table TABREF27 that a simple averaging operation or taking most frequent predicted class works competitively for CSAT and 20newsgroups but not for the Fisher dataset. We believe the improvements from using RoBERT or ToBERT, compared to simple averaging or most frequent operations, are proportional to the fraction of long documents in the dataset. CSAT and 20newsgroups have (on average) significantly shorter documents than Fisher, as seen in Fig. FIGREF21. Also, significant improvements for Fisher could be because of less confident predictions from BERT model as this dataset has 40 classes. Fig. FIGREF31 presents the comparison of average voting and ToBERT for various document length ranges for Fisher dataset. We used fine-tuned BERT segment-level predictions (P) for this analysis. It can be observed that ToBERT outperforms average voting in every interval. To the best of our knowledge, this is a state-of-the-art result reported on the Fisher dataset. Table TABREF32 presents the effect of position embeddings on the model performance. It can be observed that position embeddings did not significantly affect the model performance for Fisher and 20newsgroups, but they helped slightly in CSAT prediction (an absolute improvement of 0.64% F1-score). We think that this is explained by the fact that Fisher and 20newsgroups are topic identification tasks, and the topic does not change much throughout these documents. However, CSAT may vary during the call, and in some cases a naive assumption that the sequential nature of the transcripts is irrelevant may lead to wrong conclusions. Table TABREF33 compares our results with previous works. It can be seen that our model ToBERT outperforms CNN based experiments by significant margin on CSAT and Fisher datasets. For CSAT dataset, we used multi-scale CNN (MS-CNN) as the baseline, given its strong results on Fisher and 20newsgroups. The setup was replicated from BIBREF5 for comparison. We also see that our result on 20 newsgroups is 0.6% worse than the state-of-the-art. Conclusions In this paper, we presented two methods for long documents using BERT model: RoBERT and ToBERT. We evaluated our experiments on two classification tasks - customer satisfaction prediction and topic identification - using 3 datasets: CSAT, 20newsgroups and Fisher. We observed that ToBERT outperforms RoBERT on pre-trained BERT features and fine-tuned BERT features for all our tasks. Also, we noticed that fine-tuned BERT performs better than pre-trained BERT. We have shown that both RoBERT and ToBERT improved the simple baselines of taking an average (or the most frequent) of segment-wise predictions for long documents to obtain final prediction. Position embeddings did not significantly affect our models performance, but slightly improved the accuracy on the CSAT task. We obtained the best results on Fisher dataset and good improvements for CSAT task compared to the CNN baseline. It is interesting to note that the longer the average input in a given task, the bigger improvement we observe w.r.t. the baseline for that task. Our results confirm that both RoBERT and ToBERT can be used for long sequences with competitive performance and quick fine-tuning procedure. For future work, we shall focus on training models on long documents directly (i.e. in an end-to-end manner).
On top of BERT does the RNN layer work better or the transformer layer?
0810b43404686ddfe4ca84783477ae300fdd2ea4
On Ġtop Ġof ĠB ERT Ġdoes Ġthe ĠR NN Ġlayer Ġwork Ġbetter Ġor Ġthe Ġtransformer Ġlayer ?
Trans former Ġover ĠB ERT Ġ( To BER T )
<s> On Ġtop Ġof ĠB ERT Ġdoes Ġthe ĠR NN Ġlayer Ġwork Ġbetter Ġor Ġthe Ġtransformer Ġlayer ? </s> Introduction </s> B id irection al ĠEnc oder ĠRepresent ations Ġfrom ĠTransformers Ġ( BER T ) Ġis Ġa Ġnovel ĠTrans former ĠB IB REF 0 Ġmodel , Ġwhich Ġrecently Ġachieved Ġstate - of - the - art Ġperformance Ġin Ġseveral Ġlanguage Ġunderstanding Ġtasks , Ġsuch Ġas Ġquestion Ġanswering , Ġnatural Ġlanguage Ġinference , Ġsemantic Ġsimilarity , Ġsentiment Ġanalysis , Ġand Ġothers ĠB IB REF 1 . ĠWhile Ġwell - su ited Ġto Ġdealing Ġwith Ġrelatively Ġshort Ġsequences , ĠTransformers Ġsuffer Ġfrom Ġa Ġmajor Ġissue Ġthat Ġhind ers Ġtheir Ġapplic ability Ġin Ġclassification Ġof Ġlong Ġsequences , Ġi . e . Ġthey Ġare Ġable Ġto Ġconsume Ġonly Ġa Ġlimited Ġcontext Ġof Ġsymbols Ġas Ġtheir Ġinput ĠB IB REF 2 . </s> There Ġare Ġseveral Ġnatural Ġlanguage Ġ( N LP ) Ġprocessing Ġtasks Ġthat Ġinvolve Ġsuch Ġlong Ġsequences . ĠOf Ġparticular Ġinterest Ġare Ġtopic Ġidentification Ġof Ġspoken Ġconversations ĠB IB REF 3 , ĠB IB REF 4 , ĠB IB REF 5 Ġand Ġcall Ġcenter Ġcustomer Ġsatisfaction Ġprediction ĠB IB REF 6 , ĠB IB REF 7 , ĠB IB REF 8 , ĠB IB REF 9 . ĠCall Ġcenter Ġconversations , Ġwhile Ġusually Ġquite Ġshort Ġand Ġto Ġthe Ġpoint , Ġoften Ġinvolve Ġagents Ġtrying Ġto Ġsolve Ġvery Ġcomplex Ġissues Ġthat Ġthe Ġcustomers Ġexperience , Ġresulting Ġin Ġsome Ġcalls Ġtaking Ġeven Ġan Ġhour Ġor Ġmore . ĠFor Ġspeech Ġanalytics Ġpurposes , Ġthese Ġcalls Ġare Ġtypically Ġtransc ribed Ġusing Ġan Ġautomatic Ġspeech Ġrecognition Ġ( AS R ) Ġsystem , Ġand Ġprocessed Ġin Ġtextual Ġrepresentations Ġfurther Ġdown Ġthe ĠN LP Ġpipeline . ĠThese Ġtranscripts Ġsometimes Ġexceed Ġthe Ġlength Ġof Ġ5000 Ġwords . ĠFurthermore , Ġtemporal Ġinformation Ġmight Ġplay Ġan Ġimportant Ġrole Ġin Ġtasks Ġlike ĠCS AT . ĠFor Ġexample , Ġa Ġcustomer Ġmay Ġbe Ġangry Ġat Ġthe Ġbeginning Ġof Ġthe Ġcall , Ġbut Ġafter Ġher Ġissue Ġis Ġresolved , Ġshe Ġwould Ġbe Ġvery Ġsatisfied Ġwith Ġthe Ġway Ġit Ġwas Ġhandled . ĠTherefore , Ġsimple Ġbag Ġof Ġwords Ġmodels , Ġor Ġany Ġmodel Ġthat Ġdoes Ġnot Ġinclude Ġtemporal Ġdependencies Ġbetween Ġthe Ġinputs , Ġmay Ġnot Ġbe Ġwell - su ited Ġto Ġhandle Ġthis Ġcategory Ġof Ġtasks . ĠThis Ġmotiv ates Ġus Ġto Ġemploy Ġmodel Ġsuch Ġas ĠB ERT Ġin Ġthis Ġtask . </s> In Ġthis Ġpaper , Ġwe Ġpropose Ġa Ġmethod Ġthat Ġbuilds Ġupon ĠB ERT 's Ġarchitecture . ĠWe Ġsplit Ġthe Ġinput Ġtext Ġsequence Ġinto Ġshorter Ġsegments Ġin Ġorder Ġto Ġobtain Ġa Ġrepresentation Ġfor Ġeach Ġof Ġthem Ġusing ĠB ERT . ĠThen , Ġwe Ġuse Ġeither Ġa Ġrecurrent ĠL ST M ĠB IB REF 10 Ġnetwork , Ġor Ġanother ĠTrans former , Ġto Ġperform Ġthe Ġactual Ġclassification . ĠWe Ġcall Ġthese Ġtechniques ĠRec urrence Ġover ĠB ERT Ġ( Ro BER T ) Ġand ĠTrans former Ġover ĠB ERT Ġ( To BER T ). ĠGiven Ġthat Ġthese Ġmodels Ġintroduce Ġa Ġhierarchy Ġof Ġrepresentations Ġ( se gment - wise Ġand Ġdocument - wise ), Ġwe Ġrefer Ġto Ġthem Ġas ĠHier arch ical ĠTransformers . ĠTo Ġthe Ġbest Ġof Ġour Ġknowledge , Ġno Ġattempt Ġhas Ġbeen Ġdone Ġbefore Ġto Ġuse Ġthe ĠTrans former Ġarchitecture Ġfor Ġclassification Ġof Ġsuch Ġlong Ġsequences . </s> Our Ġnovel Ġcontributions Ġare : </s> Two Ġextensions Ġ- ĠRo BER T Ġand ĠTo BER T Ġ- Ġto Ġthe ĠB ERT Ġmodel , Ġwhich Ġenable Ġits Ġapplication Ġin Ġclassification Ġof Ġlong Ġtexts Ġby Ġperforming Ġsegment ation Ġand Ġusing Ġanother Ġlayer Ġon Ġtop Ġof Ġthe Ġsegment Ġrepresentations . </s> State - of - the - art Ġresults Ġon Ġthe ĠFisher Ġtopic Ġclassification Ġtask . </s> Sign ificant Ġimprovement Ġon Ġthe ĠCS AT Ġprediction Ġtask Ġover Ġthe ĠMS - CNN Ġmodel . </s> Related Ġwork </s> Several Ġdimension ality Ġreduction Ġalgorithms Ġsuch Ġas ĠR BM , Ġaut oen cod ers , Ġsub space Ġmult in omial Ġmodels Ġ( S MM ) Ġare Ġused Ġto Ġobtain Ġa Ġlow Ġdimensional Ġrepresentation Ġof Ġdocuments Ġfrom Ġa Ġsimple ĠB OW Ġrepresentation Ġand Ġthen Ġclassify Ġit Ġusing Ġa Ġsimple Ġlinear Ġclass ifiers ĠB IB REF 11 , ĠB IB REF 12 , ĠB IB REF 13 , ĠB IB REF 4 . ĠIn ĠB IB REF 14 Ġhierarchical Ġattention Ġnetworks Ġare Ġused Ġfor Ġdocument Ġclassification . ĠThey Ġevaluate Ġtheir Ġmodel Ġon Ġseveral Ġdatasets Ġwith Ġaverage Ġnumber Ġof Ġwords Ġaround Ġ150 . ĠCharacter - level ĠCNN Ġare Ġexplored Ġin ĠB IB REF 15 Ġbut Ġit Ġis Ġprohib itive Ġfor Ġvery Ġlong Ġdocuments . ĠIn ĠB IB REF 16 , Ġdataset Ġcollected Ġfrom Ġar X iv Ġpapers Ġis Ġused Ġfor Ġclassification . ĠFor Ġclassification , Ġthey Ġsample Ġrandom Ġblocks Ġof Ġwords Ġand Ġuse Ġthem Ġtogether Ġfor Ġclassification Ġinstead Ġof Ġusing Ġfull Ġdocument Ġwhich Ġmay Ġwork Ġwell Ġas Ġar X iv Ġpapers Ġare Ġusually Ġcoherent Ġand Ġwell Ġwritten Ġon Ġa Ġwell Ġdefined Ġtopic . ĠTheir Ġmethod Ġmay Ġnot Ġwork Ġwell Ġon Ġspoken Ġconversations Ġas Ġrandom Ġblock Ġof Ġwords Ġusually Ġdo Ġnot Ġrepresent Ġtopic Ġof Ġfull Ġconversation . </s> Several Ġresearchers Ġaddressed Ġthe Ġproblem Ġof Ġpredicting Ġcustomer Ġsatisfaction ĠB IB REF 6 , ĠB IB REF 7 , ĠB IB REF 8 , ĠB IB REF 9 . ĠIn Ġmost Ġof Ġthese Ġworks , Ġlog istic Ġregression , ĠS VM , ĠCNN Ġare Ġapplied Ġon Ġdifferent Ġkinds Ġof Ġrepresentations . </s> In ĠB IB REF 17 , Ġauthors Ġuse ĠB ERT Ġfor Ġdocument Ġclassification Ġbut Ġthe Ġaverage Ġdocument Ġlength Ġis Ġless Ġthan ĠB ERT Ġmaximum Ġlength Ġ512 . ĠTrans former XL ĠB IB REF 2 Ġis Ġan Ġextension Ġto Ġthe ĠTrans former Ġarchitecture Ġthat Ġallows Ġit Ġto Ġbetter Ġdeal Ġwith Ġlong Ġinputs Ġfor Ġthe Ġlanguage Ġmodelling Ġtask . ĠIt Ġrelies Ġon Ġthe Ġauto - reg ressive Ġproperty Ġof Ġthe Ġmodel , Ġwhich Ġis Ġnot Ġthe Ġcase Ġin Ġour Ġtasks . </s> Method Ġ: :: ĠB ERT </s> Because Ġour Ġwork Ġbuilds Ġheavily Ġupon ĠB ERT , Ġwe Ġprovide Ġa Ġbrief Ġsummary Ġof Ġits Ġfeatures . ĠB ERT Ġis Ġbuilt Ġupon Ġthe ĠTrans former Ġarchitecture ĠB IB REF 0 , Ġwhich Ġuses Ġself - att ention , Ġfeed - forward Ġlayers , Ġresidual Ġconnections Ġand Ġlayer Ġnormal ization Ġas Ġthe Ġmain Ġbuilding Ġblocks . ĠIt Ġhas Ġtwo Ġpre - training Ġobjectives : </s> Mask ed Ġlanguage Ġmodelling Ġ- Ġsome Ġof Ġthe Ġwords Ġin Ġa Ġsentence Ġare Ġbeing Ġmasked Ġand Ġthe Ġmodel Ġhas Ġto Ġpredict Ġthem Ġbased Ġon Ġthe Ġcontext Ġ( note Ġthe Ġdifference Ġfrom Ġthe Ġtypical Ġaut ore gressive Ġlanguage Ġmodel Ġtraining Ġobjective ); </s> Next Ġsentence Ġprediction Ġ- Ġgiven Ġtwo Ġinput Ġsequences , Ġdecide Ġwhether Ġthe Ġsecond Ġone Ġis Ġthe Ġnext Ġsentence Ġor Ġnot . </s> BER T Ġhas Ġbeen Ġshown Ġto Ġbeat Ġthe Ġstate - of - the - art Ġperformance Ġon Ġ11 Ġtasks Ġwith Ġno Ġmodifications Ġto Ġthe Ġmodel Ġarchitecture , Ġbesides Ġadding Ġa Ġtask - specific Ġoutput Ġlayer ĠB IB REF 1 . ĠWe Ġfollow Ġsame Ġprocedure Ġsuggested Ġin ĠB IB REF 1 Ġfor Ġour Ġtasks . ĠFig . ĠFIG REF 8 Ġshows Ġthe ĠB ERT Ġmodel Ġfor Ġclassification . ĠWe Ġobtain Ġtwo Ġkinds Ġof Ġrepresentation Ġfrom ĠB ERT : Ġpooled Ġoutput Ġfrom Ġlast Ġtransformer Ġblock , Ġden oted Ġby ĠH , Ġand Ġposterior Ġprobabilities , Ġden oted Ġby ĠP . ĠThere Ġare Ġtwo Ġvariants Ġof ĠB ERT Ġ- ĠB ERT - Base Ġand ĠB ERT - Large . ĠIn Ġthis Ġwork Ġwe Ġare Ġusing ĠB ERT - Base Ġfor Ġfaster Ġtraining Ġand Ġexperimentation , Ġhowever , Ġour Ġmethods Ġare Ġapplicable Ġto ĠB ERT - Large Ġas Ġwell . ĠB ERT - Base Ġand ĠB ERT - Large Ġare Ġdifferent Ġin Ġmodel Ġparameters Ġsuch Ġas Ġnumber Ġof Ġtransformer Ġblocks , Ġnumber Ġof Ġself - att ention Ġheads . ĠTotal Ġnumber Ġof Ġparameters Ġin ĠB ERT - Base Ġare Ġ110 M Ġand Ġ340 M Ġin ĠB ERT - Large . </s> BER T Ġsuffers Ġfrom Ġmajor Ġlimitations Ġin Ġterms Ġof Ġhandling Ġlong Ġsequences . ĠFirstly , Ġthe Ġself - att ention Ġlayer Ġhas Ġa Ġquad r atic Ġcomplexity Ġ$ O ( n ^ 2 ) $ Ġin Ġterms Ġof Ġthe Ġsequence Ġlength Ġ$ n $ ĠB IB REF 0 . ĠSecondly , ĠB ERT Ġuses Ġa Ġlearned Ġpositional Ġembed d ings Ġscheme ĠB IB REF 1 , Ġwhich Ġmeans Ġthat Ġit Ġwon 't Ġlikely Ġbe Ġable Ġto Ġgeneral ize Ġto Ġpositions Ġbeyond Ġthose Ġseen Ġin Ġthe Ġtraining Ġdata . </s> To Ġinvestigate Ġthe Ġeffect Ġof Ġfine - tun ing ĠB ERT Ġon Ġtask Ġperformance , Ġwe Ġuse Ġeither Ġthe Ġpre - trained ĠB ERT Ġweights , Ġor Ġthe Ġweights Ġfrom Ġa ĠB ERT Ġfine - tun ed Ġon Ġthe Ġtask - specific Ġdataset Ġon Ġa Ġsegment - level Ġ( i . e . Ġwe Ġpreserve Ġthe Ġoriginal Ġlabel Ġbut Ġfine - t une Ġon Ġeach Ġsegment Ġseparately Ġinstead Ġof Ġon Ġthe Ġwhole Ġtext Ġsequence ). ĠWe Ġcompare Ġthese Ġresults Ġto Ġusing Ġthe Ġfine - tun ed Ġsegment - level ĠB ERT Ġpredictions Ġdirectly Ġas Ġinputs Ġto Ġthe Ġnext Ġlayer . </s> Method Ġ: :: ĠRec urrence Ġover ĠB ERT </s> Given Ġthat ĠB ERT Ġis Ġlimited Ġto Ġa Ġparticular Ġinput Ġlength , Ġwe Ġsplit Ġthe Ġinput Ġsequence Ġinto Ġsegments Ġof Ġa Ġfixed Ġsize Ġwith Ġoverlap . ĠFor Ġeach Ġof Ġthese Ġsegments , Ġwe Ġobtain ĠH Ġor ĠP Ġfrom ĠB ERT Ġmodel . ĠWe Ġthen Ġstack Ġthese Ġsegment - level Ġrepresentations Ġinto Ġa Ġsequence , Ġwhich Ġserves Ġas Ġinput Ġto Ġa Ġsmall Ġ( 100 - dimensional ) ĠL ST M Ġlayer . ĠIts Ġoutput Ġserves Ġas Ġa Ġdocument Ġembed ding . ĠFinally , Ġwe Ġuse Ġtwo Ġfully Ġconnected Ġlayers Ġwith ĠRe LU Ġ( 30 - dimensional ) Ġand Ġsoft max Ġ( the Ġsame Ġdimension ality Ġas Ġthe Ġnumber Ġof Ġclasses ) Ġactiv ations Ġto Ġobtain Ġthe Ġfinal Ġpredictions . </s> With Ġthis Ġapproach , Ġwe Ġovercome ĠB ERT 's Ġcomputational Ġcomplexity , Ġreducing Ġit Ġto Ġ$ O ( n / k Ġ* Ġk ^ 2 ) Ġ= ĠO ( n k ) $ Ġfor ĠRo BER T , Ġwith Ġ$ k $ Ġden oting Ġthe Ġsegment Ġsize Ġ( the ĠL ST M Ġcomponent Ġhas Ġnegligible Ġlinear Ġcomplexity Ġ$ O ( k ) $ ). ĠThe Ġpositional Ġembed d ings Ġare Ġalso Ġno Ġlonger Ġan Ġissue . </s> Method Ġ: :: ĠTrans former Ġover ĠB ERT </s> Given Ġthat ĠTransformers ' Ġedge Ġover Ġrecurrent Ġnetworks Ġis Ġtheir Ġability Ġto Ġeffectively Ġcapture Ġlong Ġdistance Ġrelationships Ġbetween Ġwords Ġin Ġa Ġsequence ĠB IB REF 0 , Ġwe Ġexperiment Ġwith Ġreplacing Ġthe ĠL ST M Ġrecurrent Ġlayer Ġin Ġfavor Ġof Ġa Ġsmall ĠTrans former Ġmodel Ġ( 2 Ġlayers Ġof Ġtransformer Ġbuilding Ġblock Ġcontaining Ġself - att ention , Ġfully Ġconnected , Ġetc .). ĠTo Ġinvestigate Ġif Ġpreserving Ġthe Ġinformation Ġabout Ġthe Ġinput Ġsequence Ġorder Ġis Ġimportant , Ġwe Ġalso Ġbuild Ġa Ġvariant Ġof ĠTo BER T Ġwhich Ġlearns Ġpositional Ġembed d ings Ġat Ġthe Ġsegment - level Ġrepresentations Ġ( but Ġis Ġlimited Ġto Ġsequences Ġof Ġlength Ġseen Ġduring Ġthe Ġtraining ). </s> To BER T 's Ġcomputational Ġcomplexity Ġ$ O (\ frac { n ^ 2 }{ k ^ 2 }) $ Ġis Ġas ym pt ot ically Ġinferior Ġto ĠRo BER T , Ġas Ġthe Ġtop - level ĠTrans former Ġmodel Ġagain Ġsuffers Ġfrom Ġquad r atic Ġcomplexity Ġin Ġthe Ġnumber Ġof Ġsegments . ĠHowever , Ġin Ġpractice Ġthis Ġnumber Ġis Ġmuch Ġsmaller Ġthan Ġthe Ġinput Ġsequence Ġlength Ġ($ {\ frac { n }{ k }} Ġ<< Ġn $ ), Ġso Ġwe Ġhaven 't Ġobserved Ġperformance Ġor Ġmemory Ġissues Ġwith Ġour Ġdatasets . </s> Exper iments </s> We Ġevaluated Ġour Ġmodels Ġon Ġ3 Ġdifferent Ġdatasets : </s> CS AT Ġdataset Ġfor ĠCS AT Ġprediction , Ġconsisting Ġof Ġspoken Ġtranscripts Ġ( automatic Ġvia ĠAS R ). </s> 20 Ġnews groups Ġfor Ġtopic Ġidentification Ġtask , Ġconsisting Ġof Ġwritten Ġtext ; </s> F isher ĠPhase Ġ1 Ġcorpus Ġfor Ġtopic Ġidentification Ġtask , Ġconsisting Ġof Ġspoken Ġtranscripts Ġ( man ual ); </s> Exper iments Ġ: :: ĠCS AT </s> CS AT Ġdataset Ġconsists Ġof ĠUS ĠEnglish Ġtelephone Ġspeech Ġfrom Ġcall Ġcenters . ĠFor Ġeach Ġcall Ġin Ġthis Ġdataset , Ġcustomers Ġparticipated Ġin Ġthat Ġcall Ġgave Ġa Ġrating Ġon Ġhis Ġexperience Ġwith Ġagent . ĠOriginally , Ġthis Ġdataset Ġhas Ġlabels Ġrated Ġon Ġa Ġscale Ġ1 - 9 Ġwith Ġ9 Ġbeing Ġextremely Ġsatisfied Ġand Ġ1 Ġbeing Ġextremely Ġdissatisfied . ĠFig . ĠFIG REF 16 Ġshows Ġthe Ġhist ogram Ġof Ġratings Ġfor Ġour Ġdataset . ĠAs Ġthe Ġdistribution Ġis Ġskewed Ġtowards Ġextremes , Ġwe Ġchoose Ġto Ġdo Ġbinary Ġclassification Ġwith Ġratings Ġabove Ġ4 . 5 Ġas Ġsatisfied Ġand Ġbelow Ġ4 . 5 Ġas Ġdissatisfied . ĠQuant ization Ġof Ġratings Ġalso Ġhelped Ġus Ġto Ġcreate Ġa Ġbalanced Ġdataset . ĠThis Ġdataset Ġcontains Ġ4 331 Ġcalls Ġand Ġwe Ġsplit Ġthem Ġinto Ġ3 Ġsets Ġfor Ġour Ġexperiments : Ġ28 66 Ġcalls Ġfor Ġtraining , Ġ36 2 Ġcalls Ġfor Ġvalidation Ġand , Ġfinally , Ġ110 3 Ġcalls Ġfor Ġtesting . </s> We Ġobtained Ġthe Ġtranscripts Ġby Ġemploying Ġan ĠAS R Ġsystem . ĠThe ĠAS R Ġsystem Ġuses ĠTD NN - L ST M Ġacoustic Ġmodel Ġtrained Ġon ĠFisher Ġand ĠSwitch board Ġdatasets Ġwith Ġlatt ice - free Ġmaximum Ġmutual Ġinformation Ġcriterion ĠB IB REF 18 . ĠThe Ġword Ġerror Ġrates Ġusing Ġfour - gram Ġlanguage Ġmodels Ġwere Ġ9 . 2 % Ġand Ġ17 . 3 % Ġrespectively Ġon ĠSwitch board Ġand ĠCall Home Ġportions Ġof ĠEval 2000 Ġdataset . </s> Exper iments Ġ: :: Ġ20 Ġnews groups </s> 20 Ġnews groups Ġdata Ġset Ġis Ġone Ġof Ġthe Ġfrequently Ġused Ġdatasets Ġin Ġthe Ġtext Ġprocessing Ġcommunity Ġfor Ġtext Ġclassification Ġand Ġtext Ġclust ering . ĠThis Ġdata Ġset Ġcontains Ġapproximately Ġ20 , 000 ĠEnglish Ġdocuments Ġfrom Ġ20 Ġtopics Ġto Ġbe Ġidentified , Ġwith Ġ113 14 Ġdocuments Ġfor Ġtraining Ġand Ġ75 32 Ġfor Ġtesting . ĠIn Ġthis Ġwork , Ġwe Ġused Ġonly Ġ90 % Ġof Ġdocuments Ġfor Ġtraining Ġand Ġthe Ġremaining Ġ10 % Ġfor Ġvalidation . ĠFor Ġfair Ġcomparison Ġwith Ġother Ġpublications , Ġwe Ġused Ġ53 160 Ġwords Ġvocabulary Ġset Ġavailable Ġin Ġthe Ġdatasets Ġwebsite . </s> Exper iments Ġ: :: ĠFisher </s> F isher ĠPhase Ġ1 ĠUS ĠEnglish Ġcorpus Ġis Ġoften Ġused Ġfor Ġautomatic Ġspeech Ġrecognition Ġin Ġspeech Ġcommunity . ĠIn Ġthis Ġwork , Ġwe Ġused Ġit Ġfor Ġtopic Ġidentification Ġas Ġin ĠB IB REF 3 . ĠThe Ġdocuments Ġare Ġ10 - minute Ġlong Ġtelephone Ġconversations Ġbetween Ġtwo Ġpeople Ġdiscussing Ġa Ġgiven Ġtopic . ĠWe Ġused Ġsame Ġtraining Ġand Ġtest Ġsplits Ġas ĠB IB REF 3 Ġin Ġwhich Ġ13 74 Ġand Ġ13 72 Ġdocuments Ġare Ġused Ġfor Ġtraining Ġand Ġtesting Ġrespectively . ĠFor Ġvalidation Ġof Ġour Ġmodel , Ġwe Ġused Ġ10 % Ġof Ġtraining Ġdataset Ġand Ġthe Ġremaining Ġ90 % Ġwas Ġused Ġfor Ġactual Ġmodel Ġtraining . ĠThe Ġnumber Ġof Ġtopics Ġin Ġthis Ġdata Ġset Ġis Ġ40 . </s> Exper iments Ġ: :: ĠDat as et ĠStatistics </s> Table ĠT AB REF 22 Ġshows Ġstatistics Ġof Ġour Ġdatasets . ĠIt Ġcan Ġbe Ġobserved Ġthat Ġaverage Ġlength Ġof ĠFisher Ġis Ġmuch Ġhigher Ġthan Ġ20 Ġnews groups Ġand ĠCS AT . ĠCum ulative Ġdistribution Ġof Ġdocument Ġlengths Ġfor Ġeach Ġdataset Ġis Ġshown Ġin ĠFig . ĠFIG REF 21 . ĠIt Ġcan Ġbe Ġobserved Ġthat Ġalmost Ġall Ġof Ġthe Ġdocuments Ġin ĠFisher Ġdataset Ġhave Ġlength Ġmore Ġthan Ġ1000 Ġwords . ĠFor ĠCS AT , Ġmore Ġthan Ġ50 % Ġof Ġthe Ġdocuments Ġhave Ġlength Ġgreater Ġthan Ġ500 Ġand Ġfor Ġ20 news groups Ġonly Ġ10 % Ġof Ġthe Ġdocuments Ġhave Ġlength Ġgreater Ġthan Ġ500 . ĠNote Ġthat , Ġfor ĠCS AT Ġand Ġ20 news groups , Ġthere Ġare Ġfew Ġdocuments Ġwith Ġlength Ġmore Ġthan Ġ5000 . </s> Exper iments Ġ: :: ĠArchitecture Ġand ĠTraining ĠDetails </s> In Ġthis Ġwork , Ġwe Ġsplit Ġdocument Ġinto Ġsegments Ġof Ġ200 Ġtokens Ġwith Ġa Ġshift Ġof Ġ50 Ġtokens Ġto Ġextract Ġfeatures Ġfrom ĠB ERT Ġmodel . ĠFor ĠRo BER T , ĠL ST M Ġmodel Ġis Ġtrained Ġto Ġminimize Ġcross - ent ropy Ġloss Ġwith ĠAdam Ġoptim izer ĠB IB REF 19 . ĠThe Ġinitial Ġlearning Ġrate Ġis Ġset Ġto Ġ$ 0 . 001 $ Ġand Ġis Ġreduced Ġby Ġa Ġfactor Ġof Ġ$ 0 . 95 $ Ġif Ġvalidation Ġloss Ġdoes Ġnot Ġdecrease Ġfor Ġ3 - ep och s . ĠFor ĠTo BER T , Ġthe ĠTrans former Ġis Ġtrained Ġwith Ġthe Ġdefault ĠB ERT Ġversion Ġof ĠAdam Ġoptim izer ĠB IB REF 1 Ġwith Ġan Ġinitial Ġlearning Ġrate Ġof Ġ$ 5 e $ - 5 . ĠWe Ġreport Ġaccuracy Ġin Ġall Ġof Ġour Ġexperiments . ĠWe Ġchose Ġa Ġmodel Ġwith Ġthe Ġbest Ġvalidation Ġaccuracy Ġto Ġcalculate Ġaccuracy Ġon Ġthe Ġtest Ġset . ĠTo Ġacc om od ate Ġfor Ġnon - d etermin ism Ġof Ġsome ĠT ensor Flow ĠGPU Ġoperations , Ġwe Ġreport Ġaccuracy Ġaveraged Ġover Ġ5 Ġruns . </s> Results </s> Table ĠT AB REF 25 Ġpresents Ġresults Ġusing Ġpre - trained ĠB ERT Ġfeatures . ĠWe Ġextracted Ġfeatures Ġfrom Ġthe Ġpooled Ġoutput Ġof Ġfinal Ġtransformer Ġblock Ġas Ġthese Ġwere Ġshown Ġto Ġbe Ġworking Ġwell Ġfor Ġmost Ġof Ġthe Ġtasks ĠB IB REF 1 . ĠThe Ġfeatures Ġextracted Ġfrom Ġa Ġpre - trained ĠB ERT Ġmodel Ġwithout Ġany Ġfine - tun ing Ġlead Ġto Ġa Ġsub - par Ġperformance . ĠHowever , ĠWe Ġalso Ġnotice Ġthat ĠTo BER T Ġmodel Ġexploited Ġthe Ġpre - trained ĠB ERT Ġfeatures Ġbetter Ġthan ĠRo BER T . ĠIt Ġalso Ġconver ged Ġfaster Ġthan ĠRo BER T . ĠTable ĠT AB REF 26 Ġshows Ġresults Ġusing Ġfeatures Ġextracted Ġafter Ġfine - tun ing ĠB ERT Ġmodel Ġwith Ġour Ġdatasets . ĠSignificant Ġimprovements Ġcan Ġbe Ġobserved Ġcompared Ġto Ġusing Ġpre - trained ĠB ERT Ġfeatures . ĠAlso , Ġit Ġcan Ġbe Ġnoticed Ġthat ĠTo BER T Ġoutper forms ĠRo BER T Ġon ĠFisher Ġand Ġ20 news groups Ġdataset Ġby Ġ13 . 63 % Ġand Ġ0 . 81 % Ġrespectively . ĠOn ĠCS AT , ĠTo BER T Ġperforms Ġslightly Ġworse Ġthan ĠRo BER T Ġbut Ġit Ġis Ġnot Ġstatistically Ġsignificant Ġas Ġthis Ġdataset Ġis Ġsmall . </s> Table ĠT AB REF 27 Ġpresents Ġresults Ġusing Ġfine - tun ed ĠB ERT Ġpredictions Ġinstead Ġof Ġthe Ġpooled Ġoutput Ġfrom Ġfinal Ġtransformer Ġblock . ĠFor Ġeach Ġdocument , Ġhaving Ġobtained Ġsegment - wise Ġpredictions Ġwe Ġcan Ġobtain Ġfinal Ġprediction Ġfor Ġthe Ġwhole Ġdocument Ġin Ġthree Ġways : </s> Comp ute Ġthe Ġaverage Ġof Ġall Ġsegment - wise Ġpredictions Ġand Ġfind Ġthe Ġmost Ġprobable Ġclass ; </s> Find Ġthe Ġmost Ġfrequently Ġpredicted Ġclass ; </s> Train Ġa Ġclassification Ġmodel . </s> It Ġcan Ġbe Ġobserved Ġfrom ĠTable ĠT AB REF 27 Ġthat Ġa Ġsimple Ġaveraging Ġoperation Ġor Ġtaking Ġmost Ġfrequent Ġpredicted Ġclass Ġworks Ġcompetitive ly Ġfor ĠCS AT Ġand Ġ20 news groups Ġbut Ġnot Ġfor Ġthe ĠFisher Ġdataset . ĠWe Ġbelieve Ġthe Ġimprovements Ġfrom Ġusing ĠRo BER T Ġor ĠTo BER T , Ġcompared Ġto Ġsimple Ġaveraging Ġor Ġmost Ġfrequent Ġoperations , Ġare Ġproportional Ġto Ġthe Ġfraction Ġof Ġlong Ġdocuments Ġin Ġthe Ġdataset . ĠCS AT Ġand Ġ20 news groups Ġhave Ġ( on Ġaverage ) Ġsignificantly Ġshorter Ġdocuments Ġthan ĠFisher , Ġas Ġseen Ġin ĠFig . ĠFIG REF 21 . ĠAlso , Ġsignificant Ġimprovements Ġfor ĠFisher Ġcould Ġbe Ġbecause Ġof Ġless Ġconfident Ġpredictions Ġfrom ĠB ERT Ġmodel Ġas Ġthis Ġdataset Ġhas Ġ40 Ġclasses . ĠFig . ĠFIG REF 31 Ġpresents Ġthe Ġcomparison Ġof Ġaverage Ġvoting Ġand ĠTo BER T Ġfor Ġvarious Ġdocument Ġlength Ġranges Ġfor ĠFisher Ġdataset . ĠWe Ġused Ġfine - tun ed ĠB ERT Ġsegment - level Ġpredictions Ġ( P ) Ġfor Ġthis Ġanalysis . ĠIt Ġcan Ġbe Ġobserved Ġthat ĠTo BER T Ġoutper forms Ġaverage Ġvoting Ġin Ġevery Ġinterval . ĠTo Ġthe Ġbest Ġof Ġour Ġknowledge , Ġthis Ġis Ġa Ġstate - of - the - art Ġresult Ġreported Ġon Ġthe ĠFisher Ġdataset . </s> Table ĠT AB REF 32 Ġpresents Ġthe Ġeffect Ġof Ġposition Ġembed d ings Ġon Ġthe Ġmodel Ġperformance . ĠIt Ġcan Ġbe Ġobserved Ġthat Ġposition Ġembed d ings Ġdid Ġnot Ġsignificantly Ġaffect Ġthe Ġmodel Ġperformance Ġfor ĠFisher Ġand Ġ20 news groups , Ġbut Ġthey Ġhelped Ġslightly Ġin ĠCS AT Ġprediction Ġ( an Ġabsolute Ġimprovement Ġof Ġ0 . 64 % ĠF 1 - score ). ĠWe Ġthink Ġthat Ġthis Ġis Ġexplained Ġby Ġthe Ġfact Ġthat ĠFisher Ġand Ġ20 news groups Ġare Ġtopic Ġidentification Ġtasks , Ġand Ġthe Ġtopic Ġdoes Ġnot Ġchange Ġmuch Ġthroughout Ġthese Ġdocuments . ĠHowever , ĠCS AT Ġmay Ġvary Ġduring Ġthe Ġcall , Ġand Ġin Ġsome Ġcases Ġa Ġnaive Ġassumption Ġthat Ġthe Ġsequential Ġnature Ġof Ġthe Ġtranscripts Ġis Ġirrelevant Ġmay Ġlead Ġto Ġwrong Ġconclusions . </s> Table ĠT AB REF 33 Ġcompares Ġour Ġresults Ġwith Ġprevious Ġworks . ĠIt Ġcan Ġbe Ġseen Ġthat Ġour Ġmodel ĠTo BER T Ġoutper forms ĠCNN Ġbased Ġexperiments Ġby Ġsignificant Ġmargin Ġon ĠCS AT Ġand ĠFisher Ġdatasets . ĠFor ĠCS AT Ġdataset , Ġwe Ġused Ġmulti - scale ĠCNN Ġ( MS - CNN ) Ġas Ġthe Ġbaseline , Ġgiven Ġits Ġstrong Ġresults Ġon ĠFisher Ġand Ġ20 news groups . ĠThe Ġsetup Ġwas Ġreplicated Ġfrom ĠB IB REF 5 Ġfor Ġcomparison . ĠWe Ġalso Ġsee Ġthat Ġour Ġresult Ġon Ġ20 Ġnews groups Ġis Ġ0 . 6 % Ġworse Ġthan Ġthe Ġstate - of - the - art . </s> Con clusions </s> In Ġthis Ġpaper , Ġwe Ġpresented Ġtwo Ġmethods Ġfor Ġlong Ġdocuments Ġusing ĠB ERT Ġmodel : ĠRo BER T Ġand ĠTo BER T . ĠWe Ġevaluated Ġour Ġexperiments Ġon Ġtwo Ġclassification Ġtasks Ġ- Ġcustomer Ġsatisfaction Ġprediction Ġand Ġtopic Ġidentification Ġ- Ġusing Ġ3 Ġdatasets : ĠCS AT , Ġ20 news groups Ġand ĠFisher . ĠWe Ġobserved Ġthat ĠTo BER T Ġoutper forms ĠRo BER T Ġon Ġpre - trained ĠB ERT Ġfeatures Ġand Ġfine - tun ed ĠB ERT Ġfeatures Ġfor Ġall Ġour Ġtasks . ĠAlso , Ġwe Ġnoticed Ġthat Ġfine - tun ed ĠB ERT Ġperforms Ġbetter Ġthan Ġpre - trained ĠB ERT . ĠWe Ġhave Ġshown Ġthat Ġboth ĠRo BER T Ġand ĠTo BER T Ġimproved Ġthe Ġsimple Ġbas elines Ġof Ġtaking Ġan Ġaverage Ġ( or Ġthe Ġmost Ġfrequent ) Ġof Ġsegment - wise Ġpredictions Ġfor Ġlong Ġdocuments Ġto Ġobtain Ġfinal Ġprediction . ĠPosition Ġembed d ings Ġdid Ġnot Ġsignificantly Ġaffect Ġour Ġmodels Ġperformance , Ġbut Ġslightly Ġimproved Ġthe Ġaccuracy Ġon Ġthe ĠCS AT Ġtask . ĠWe Ġobtained Ġthe Ġbest Ġresults Ġon ĠFisher Ġdataset Ġand Ġgood Ġimprovements Ġfor ĠCS AT Ġtask Ġcompared Ġto Ġthe ĠCNN Ġbaseline . ĠIt Ġis Ġinteresting Ġto Ġnote Ġthat Ġthe Ġlonger Ġthe Ġaverage Ġinput Ġin Ġa Ġgiven Ġtask , Ġthe Ġbigger Ġimprovement Ġwe Ġobserve Ġw . r . t . Ġthe Ġbaseline Ġfor Ġthat Ġtask . ĠOur Ġresults Ġconfirm Ġthat Ġboth ĠRo BER T Ġand ĠTo BER T Ġcan Ġbe Ġused Ġfor Ġlong Ġsequences Ġwith Ġcompetitive Ġperformance Ġand Ġquick Ġfine - tun ing Ġprocedure . ĠFor Ġfuture Ġwork , Ġwe Ġshall Ġfocus Ġon Ġtraining Ġmodels Ġon Ġlong Ġdocuments Ġdirectly Ġ( i . e . Ġin Ġan Ġend - to - end Ġmanner ).