Unanswerable
NONE
Yes
BOOLEAN
To verify the performance of NeuronBlocks, we conducted extensive experiments for common NLP tasks on public data sets including CoNLL-2003 BIBREF14 , GLUE benchmark BIBREF13 , and WikiQA corpus BIBREF15 . The experimental results showed that the models built with NeuronBlocks can achieve reliable and competitive results on various tasks, with productivity greatly improved. For sequence labeling task, we evaluated NeuronBlocks on CoNLL-2003 BIBREF14 English NER dataset, following most works on the same task. This dataset includes four types of named entities, namely, PERSON, LOCATION, ORGANIZATION, and MISC. We adopted the BIOES tagging scheme instead of IOB, as many previous works indicated meaningful improvement with BIOES scheme BIBREF16 , BIBREF17 . Table TABREF28 shows the results on CoNLL-2003 Englist testb dataset, with 12 different combinations of network layers/blocks, such as word/character embedding, CNN/LSTM and CRF. The results suggest that the flexible combination of layers/blocks in NeuronBlocks can easily reproduce the performance of original models, with comparative or slightly better performance.
1904.09535
Introduction </s> Deep ĠNeural ĠNetworks Ġ( D NN ) Ġhave Ġbeen Ġwidely Ġemployed Ġin Ġindustry Ġfor Ġsolving Ġvarious ĠNatural ĠLanguage ĠProcessing Ġ( N LP ) Ġtasks , Ġsuch Ġas Ġtext Ġclassification , Ġsequence Ġlabeling , Ġquestion Ġanswering , Ġetc . ĠHowever , Ġwhen Ġengineers Ġapply ĠD NN Ġmodels Ġto Ġaddress Ġspecific ĠN LP Ġtasks , Ġthey Ġoften Ġface Ġthe Ġfollowing Ġchallenges . </s> The Ġabove Ġchallenges Ġoften Ġhinder Ġthe Ġproductivity Ġof Ġengineers , Ġand Ġresult Ġin Ġless Ġoptimal Ġsolutions Ġto Ġtheir Ġgiven Ġtasks . ĠThis Ġmotiv ates Ġus Ġto Ġdevelop Ġan ĠN LP Ġtool kit Ġfor ĠD NN Ġmodels , Ġwhich Ġfacilitates Ġengineers Ġto Ġdevelop ĠD NN Ġapproaches . ĠBefore Ġdesigning Ġthis ĠN LP Ġtool kit , Ġwe Ġconducted Ġa Ġsurvey Ġamong Ġengineers Ġand Ġidentified Ġa Ġspectrum Ġof Ġthree Ġtypical Ġperson as . </s> To Ġsatisfy Ġthe Ġrequirements Ġof Ġall Ġthe Ġabove Ġthree Ġperson as , Ġthe ĠN LP Ġtool kit Ġhas Ġto Ġbe Ġgeneric Ġenough Ġto Ġcover Ġas Ġmany Ġtasks Ġas Ġpossible . ĠAt Ġthe Ġsame Ġtime , Ġit Ġalso Ġneeds Ġto Ġbe Ġflexible Ġenough Ġto Ġallow Ġalternative Ġnetwork Ġarchitectures Ġas Ġwell Ġas Ġcustomized Ġmodules . ĠTherefore , Ġwe Ġanalyzed Ġthe ĠN LP Ġjobs Ġsubmitted Ġto Ġa Ġcommercial Ġcentralized ĠGPU Ġcluster . ĠTable ĠT AB REF 11 Ġshowed Ġthat Ġabout Ġ87 . 5 % ĠN LP Ġrelated Ġjobs Ġbelong Ġto Ġa Ġfew Ġcommon Ġtasks , Ġincluding Ġsentence Ġclassification , Ġtext Ġmatching , Ġsequence Ġlabeling , ĠM RC , Ġetc . ĠIt Ġfurther Ġsuggested Ġthat Ġmore Ġthan Ġ90 % Ġof Ġthe Ġnetworks Ġwere Ġcomposed Ġof Ġseveral Ġcommon Ġcomponents , Ġsuch Ġas Ġembed ding , ĠCNN / R NN , ĠTrans former Ġand Ġso Ġon . </s> Based Ġon Ġthe Ġabove Ġobservations , Ġwe Ġdeveloped ĠNe uron Blocks , Ġa ĠD NN Ġtool kit Ġfor ĠN LP Ġtasks . ĠThe Ġbasic Ġidea Ġis Ġto Ġprovide Ġtwo Ġlayers Ġof Ġsupport Ġto Ġthe Ġengineers . ĠThe Ġupper Ġlayer Ġtargets Ġcommon ĠN LP Ġtasks . ĠFor Ġeach Ġtask , Ġthe Ġtool kit Ġcontains Ġseveral Ġend - to - end Ġnetwork Ġtemplates , Ġwhich Ġcan Ġbe Ġimmediately Ġinstant iated Ġwith Ġsimple Ġconfiguration . ĠThe Ġbottom Ġlayer Ġconsists Ġof Ġa Ġsuite Ġof Ġreusable Ġand Ġstandard Ġcomponents , Ġwhich Ġcan Ġbe Ġadopted Ġas Ġbuilding Ġblocks Ġto Ġconstruct Ġnetworks Ġwith Ġcomplex Ġarchitecture . ĠBy Ġfollowing Ġthe Ġinterface Ġguidelines , Ġusers Ġcan Ġalso Ġcontribute Ġto Ġthis Ġgallery Ġof Ġcomponents Ġwith Ġtheir Ġown Ġmodules . </s> The Ġtechnical Ġcontributions Ġof ĠNe uron Blocks Ġare Ġsummarized Ġinto Ġthe Ġfollowing Ġthree Ġaspects . </s> Related ĠWork </s> There Ġare Ġseveral Ġgeneral - purpose Ġdeep Ġlearning Ġframeworks , Ġsuch Ġas ĠT ensor Flow , ĠPy Tor ch Ġand ĠKer as , Ġwhich Ġhave Ġgained Ġpopularity Ġin ĠN LP Ġcommunity . ĠThese Ġframeworks Ġoffer Ġhuge Ġflexibility Ġin ĠD NN Ġmodel Ġdesign Ġand Ġsupport Ġvarious ĠN LP Ġtasks . ĠHowever , Ġbuilding Ġmodels Ġunder Ġthese Ġframeworks Ġrequires Ġa Ġlarge Ġoverhead Ġof Ġmastering Ġthese Ġframework Ġdetails . ĠTherefore , Ġhigher Ġlevel Ġabstraction Ġto Ġhide Ġthe Ġframework Ġdetails Ġis Ġfavored Ġby Ġmany Ġengineers . </s> There Ġare Ġalso Ġseveral Ġpopular Ġdeep Ġlearning Ġtool k its Ġin ĠN LP , Ġincluding ĠOpen N MT ĠB IB REF 0 Ġ, ĠAllen N LP ĠB IB REF 1 Ġetc . ĠOpen N MT Ġis Ġan Ġopen - source Ġtool kit Ġmainly Ġtargeting Ġneural Ġmachine Ġtranslation Ġor Ġother Ġnatural Ġlanguage Ġgeneration Ġtasks . ĠAllen N LP Ġprovides Ġseveral Ġpre - built Ġmodels Ġfor ĠN LP Ġtasks , Ġsuch Ġas Ġsemantic Ġrole Ġlabeling , Ġmachine Ġcomprehension , Ġtextual Ġentail ment , Ġetc . ĠAlthough Ġthese Ġtool k its Ġreduce Ġthe Ġdevelopment Ġcost , Ġthey Ġare Ġlimited Ġto Ġcertain Ġtasks , Ġand Ġthus Ġnot Ġflexible Ġenough Ġto Ġsupport Ġnew Ġnetwork Ġarchitectures Ġor Ġnew Ġcomponents . </s> Design </s> The ĠNe uron blocks Ġis Ġbuilt Ġon ĠPy Tor ch . ĠThe Ġoverall Ġframework Ġis Ġillustrated Ġin ĠFigure ĠFIG REF 16 Ġ. ĠIt Ġconsists Ġof Ġtwo Ġlayers : Ġthe ĠBlock ĠZoo Ġand Ġthe ĠModel ĠZoo . ĠIn ĠBlock ĠZoo , Ġthe Ġmost Ġcommonly Ġused Ġcomponents Ġof Ġdeep Ġneural Ġnetworks Ġare Ġcategorized Ġinto Ġseveral Ġgroups Ġaccording Ġto Ġtheir Ġfunctions . ĠWithin Ġeach Ġcategory , Ġseveral Ġalternative Ġcomponents Ġare Ġencaps ulated Ġinto Ġstandard Ġand Ġreusable Ġblocks Ġwith Ġa Ġconsistent Ġinterface . ĠThese Ġblocks Ġserve Ġas Ġbasic Ġand Ġexchange able Ġunits Ġto Ġconstruct Ġcomplex Ġnetwork Ġarchitectures Ġfor Ġdifferent ĠN LP Ġtasks . ĠIn ĠModel ĠZoo , Ġthe Ġmost Ġpopular ĠN LP Ġtasks Ġare Ġidentified . ĠFor Ġeach Ġtask , Ġseveral Ġend - to - end Ġnetwork Ġtemplates Ġare Ġprovided Ġin Ġthe Ġform Ġof ĠJSON Ġconfiguration Ġfiles . ĠUsers Ġcan Ġsimply Ġbrowse Ġthese Ġconfigurations Ġand Ġchoose Ġone Ġor Ġmore Ġto Ġinstant iate . ĠThe Ġwhole Ġtask Ġcan Ġbe Ġcompleted Ġwithout Ġany Ġcoding Ġefforts . </s> Block ĠZoo </s> We Ġrecognize Ġthe Ġfollowing Ġmajor Ġfunctional Ġcategories Ġof Ġneural Ġnetwork Ġcomponents . ĠEach Ġcategory Ġcovers Ġas Ġmany Ġcommonly Ġused Ġmodules Ġas Ġpossible . ĠThe ĠBlock ĠZoo Ġis Ġan Ġopen Ġframework , Ġand Ġmore Ġmodules Ġcan Ġbe Ġadded Ġin Ġthe Ġfuture . </s> [ item se p = Ġ- 0 . 4 em , top se p Ġ= Ġ0 . 3 em , Ġalign = left , Ġlabel se p =- 0 . 6 em , Ġleft margin = 1 . 2 em ] </s> Emb edd ing ĠLayer : ĠWord / character Ġembed ding Ġand Ġextra Ġhand crafted Ġfeature Ġembed ding Ġsuch Ġas Ġpos - tag ging Ġare Ġsupported . </s> Ne ural ĠNetwork ĠL ayers : ĠBlock Ġzoo Ġprovides Ġcommon Ġlayers Ġlike ĠR NN , ĠCNN , ĠQR NN ĠB IB REF 2 Ġ, ĠTrans former ĠB IB REF 3 Ġ, ĠHighway Ġnetwork , ĠEnc oder ĠDecoder Ġarchitecture , Ġetc . ĠFurthermore , Ġattention Ġmechanisms Ġare Ġwidely Ġused Ġin Ġneural Ġnetworks . ĠThus Ġwe Ġalso Ġsupport Ġmultiple Ġattention Ġlayers , Ġsuch Ġas ĠLinear / Bi - linear ĠAttention , ĠFull ĠAttention ĠB IB REF 4 Ġ, ĠBid irection al Ġattention Ġflow ĠB IB REF 5 Ġ, Ġetc . ĠMeanwhile , Ġregular ization Ġlayers Ġsuch Ġas ĠDrop out , ĠLayer ĠNorm , ĠB atch ĠNorm , Ġetc Ġare Ġalso Ġsupported Ġfor Ġimproving Ġgeneral ization Ġability . </s> L oss ĠFunction : ĠBesides Ġof Ġthe Ġloss Ġfunctions Ġbuilt Ġin ĠPy Tor ch , Ġwe Ġoffer Ġmore Ġoptions Ġsuch Ġas ĠF ocal ĠLoss ĠB IB REF 6 Ġ. </s> Met rics : ĠFor Ġclassification Ġtask , ĠA UC , ĠAccuracy , ĠPrecision / Rec all , ĠF 1 Ġmetrics Ġare Ġsupported . ĠFor Ġsequence Ġlabeling Ġtask , ĠF 1 / Acc uracy Ġare Ġsupported . ĠFor Ġknowledge Ġdist illation Ġtask , ĠM SE / RM SE Ġare Ġsupported . ĠFor ĠM RC Ġtask , ĠEx act Match / F 1 Ġare Ġsupported . </s> Model ĠZoo </s> In ĠNe uron Blocks , Ġwe Ġidentify Ġfour Ġtypes Ġof Ġmost Ġpopular ĠN LP Ġtasks . ĠFor Ġeach Ġtask , Ġwe Ġprovide Ġvarious Ġend - to - end Ġnetwork Ġtemplates . </s> [ item se p = Ġ- 0 . 4 em , top se p Ġ= Ġ0 . 3 em , Ġalign = left , Ġlabel se p =- 0 . 6 em , Ġleft margin = 1 . 2 em ] </s> Text ĠClassification Ġand ĠMatch ing . ĠT asks Ġsuch Ġas Ġdomain / intent Ġclassification , Ġquestion Ġanswer Ġmatching Ġare Ġsupported . </s> Sequ ence ĠLab eling . ĠPredict Ġeach Ġtoken Ġin Ġa Ġsequence Ġinto Ġpred efined Ġtypes . ĠCommon Ġtasks Ġinclude ĠN ER , ĠPOS Ġtagging , ĠSlot Ġtagging , Ġetc . </s> Know ledge ĠDist illation ĠB IB REF 7 Ġ. ĠTeacher - Student Ġbased Ġknowledge Ġdist illation Ġis Ġone Ġcommon Ġapproach Ġfor Ġmodel Ġcompression . ĠNe uron Blocks Ġprovides Ġknowledge Ġdist illation Ġtemplate Ġto Ġimprove Ġthe Ġinference Ġspeed Ġof Ġheavy ĠD NN Ġmodels Ġlike ĠB ERT / G PT . </s> Ext ractive ĠMachine ĠReading ĠComp rehens ion . ĠGiven Ġa Ġpair Ġof Ġquestion Ġand Ġpassage , Ġpredict Ġthe Ġstart Ġand Ġend Ġpositions Ġof Ġthe Ġanswer Ġspans Ġin Ġthe Ġpassage . </s> User ĠInterface </s> Ne uron Blocks Ġprovides Ġconvenient Ġuser Ġinterface Ġfor Ġusers Ġto Ġbuild , Ġtrain , Ġand Ġtest ĠD NN Ġmodels . ĠThe Ġdetails Ġare Ġdescribed Ġin Ġthe Ġfollowing . </s> [ item se p = Ġ- 0 . 4 em , top se p Ġ= Ġ0 . 3 em , Ġalign = left , Ġlabel se p =- 0 . 6 em , Ġleft margin = 1 . 2 em ] </s> I / O Ġinterface . ĠThis Ġpart Ġdefines Ġmodel Ġinput / output , Ġsuch Ġas Ġtraining Ġdata , Ġpre - trained Ġmodels / embed d ings , Ġmodel Ġsaving Ġpath , Ġetc . </s> Model ĠArchitecture Ġinterface . ĠThis Ġis Ġthe Ġkey Ġpart Ġof Ġthe Ġconfiguration Ġfile , Ġwhich Ġdefines Ġthe Ġwhole Ġmodel Ġarchitecture . ĠFigure ĠFIG REF 19 Ġshows Ġan Ġexample Ġof Ġhow Ġto Ġspecify Ġa Ġmodel Ġarchitecture Ġusing Ġthe Ġblocks Ġin ĠNe uron Blocks . ĠTo Ġbe Ġmore Ġspecific , Ġit Ġconsists Ġof Ġa Ġlist Ġof Ġlayers / blocks Ġto Ġconstruct Ġthe Ġarchitecture , Ġwhere Ġthe Ġblocks Ġare Ġsupplied Ġin Ġthe Ġgallery Ġof ĠBlock ĠZoo . </s> Training ĠParameters Ġinterface . ĠIn Ġthis Ġpart , Ġthe Ġmodel Ġoptim izer Ġas Ġwell Ġas Ġall Ġother Ġtraining Ġhyper Ġparameters Ġare Ġindicated . </s> Work flow </s> Figure ĠFIG REF 34 Ġshows Ġthe Ġworkflow Ġof Ġbuilding ĠD NN Ġmodels Ġin ĠNe uron Blocks . ĠUsers Ġonly Ġneed Ġto Ġwrite Ġa ĠJSON Ġconfiguration Ġfile . ĠThey Ġcan Ġeither Ġinstant iate Ġan Ġexisting Ġtemplate Ġfrom ĠModel ĠZoo , Ġor Ġconstruct Ġa Ġnew Ġarchitecture Ġbased Ġon Ġthe Ġblocks Ġfrom ĠBlock ĠZoo . ĠThis Ġconfiguration Ġfile Ġis Ġshared Ġacross Ġtraining , Ġtest , Ġand Ġprediction . ĠFor Ġmodel Ġhyper - param eter Ġtuning Ġor Ġarchitecture Ġmodification , Ġusers Ġjust Ġneed Ġto Ġchange Ġthe ĠJSON Ġconfiguration Ġfile . ĠAdvanced Ġusers Ġcan Ġalso Ġcontribute Ġnovel Ġcustomized Ġblocks Ġinto ĠBlock ĠZoo , Ġas Ġlong Ġas Ġthey Ġfollow Ġthe Ġsame Ġinterface Ġguidelines Ġwith Ġthe Ġexisting Ġblocks . ĠThese Ġnew Ġblocks Ġcan Ġbe Ġfurther Ġshared Ġacross Ġall Ġusers Ġfor Ġmodel Ġarchitecture Ġdesign . ĠMoreover , ĠNe uron Blocks Ġhas Ġflexible Ġplatform Ġsupport , Ġsuch Ġas ĠGPU / CPU , ĠGPU Ġmanagement Ġplatforms Ġlike ĠPA I . </s> Exper iments </s> To Ġverify Ġthe Ġperformance Ġof ĠNe uron Blocks , Ġwe Ġconducted Ġextensive Ġexperiments Ġfor Ġcommon ĠN LP Ġtasks Ġon Ġpublic Ġdata Ġsets Ġincluding ĠCo N LL - 2003 ĠB IB REF 14 Ġ, ĠGL UE Ġbenchmark ĠB IB REF 13 Ġ, Ġand ĠWiki Q A Ġcorpus ĠB IB REF 15 Ġ. ĠThe Ġexperimental Ġresults Ġshowed Ġthat Ġthe Ġmodels Ġbuilt Ġwith ĠNe uron Blocks Ġcan Ġachieve Ġreliable Ġand Ġcompetitive Ġresults Ġon Ġvarious Ġtasks , Ġwith Ġproductivity Ġgreatly Ġimproved . </s> Sequ ence ĠLab eling </s> For Ġsequence Ġlabeling Ġtask , Ġwe Ġevaluated ĠNe uron Blocks Ġon ĠCo N LL - 2003 ĠB IB REF 14 ĠEnglish ĠN ER Ġdataset , Ġfollowing Ġmost Ġworks Ġon Ġthe Ġsame Ġtask . ĠThis Ġdataset Ġincludes Ġfour Ġtypes Ġof Ġnamed Ġentities , Ġnamely , ĠPERSON , ĠLOC ATION , ĠOR GAN IZ ATION , Ġand ĠM ISC . ĠWe Ġadopted Ġthe ĠB IO ES Ġtagging Ġscheme Ġinstead Ġof ĠI OB , Ġas Ġmany Ġprevious Ġworks Ġindicated Ġmeaningful Ġimprovement Ġwith ĠB IO ES Ġscheme ĠB IB REF 16 Ġ, ĠB IB REF 17 Ġ. ĠTable ĠT AB REF 28 Ġshows Ġthe Ġresults Ġon ĠCo N LL - 2003 ĠEng list Ġtest b Ġdataset , Ġwith Ġ12 Ġdifferent Ġcombinations Ġof Ġnetwork Ġlayers / blocks , Ġsuch Ġas Ġword / character Ġembed ding , ĠCNN / L ST M Ġand ĠCR F . ĠThe Ġresults Ġsuggest Ġthat Ġthe Ġflexible Ġcombination Ġof Ġlayers / blocks Ġin ĠNe uron Blocks Ġcan Ġeasily Ġreproduce Ġthe Ġperformance Ġof Ġoriginal Ġmodels , Ġwith Ġcomparative Ġor Ġslightly Ġbetter Ġperformance . </s> GL UE ĠBench mark </s> The ĠGeneral ĠLanguage ĠUnderstanding ĠEvaluation Ġ( GL UE ) Ġbenchmark ĠB IB REF 13 Ġis Ġa Ġcollection Ġof Ġnatural Ġlanguage Ġunderstanding Ġtasks . ĠWe Ġexperimented Ġon Ġthe ĠGL UE Ġbenchmark Ġtasks Ġusing ĠBi L ST M Ġand ĠAttention Ġbased Ġmodels . ĠAs Ġshown Ġin ĠTable ĠT AB REF 29 Ġ, Ġthe Ġmodels Ġbuilt Ġby ĠNe uron Blocks Ġcan Ġachieve Ġcompetitive Ġor Ġeven Ġbetter Ġresults Ġon ĠGL UE Ġtasks Ġwith Ġminimal Ġcoding Ġefforts . </s> Know ledge ĠDist illation </s> We Ġevaluated ĠKnowledge ĠDist illation Ġtask Ġin ĠNe uron Blocks Ġon Ġa Ġdataset Ġcollected Ġfrom Ġone Ġcommercial Ġsearch Ġengine . ĠWe Ġrefer Ġto Ġthis Ġdataset Ġas ĠDomain ĠClassification ĠDat as et . ĠEach Ġsample Ġin Ġthis Ġdataset Ġconsists Ġof Ġtwo Ġparts , Ġi . e ., Ġa Ġquestion Ġand Ġa Ġbinary Ġlabel Ġindicating Ġwhether Ġthe Ġquestion Ġbelongs Ġto Ġa Ġspecific Ġdomain . ĠTable ĠT AB REF 36 Ġshows Ġthe Ġresults , Ġwhere ĠArea ĠUnder ĠCurve Ġ( A UC ) Ġmetric Ġis Ġused Ġas Ġthe Ġperformance Ġevaluation Ġcriteria Ġand ĠQu eries Ġper ĠSecond Ġ( Q PS ) Ġis Ġused Ġto Ġmeasure Ġinference Ġspeed . ĠBy Ġknowledge Ġdist illation Ġtraining Ġapproach , Ġthe Ġstudent Ġmodel Ġby ĠNe uron Blocks Ġmanaged Ġto Ġget Ġ23 - 27 Ġtimes Ġinference Ġspeed up Ġwith Ġonly Ġsmall Ġperformance Ġregression Ġcompared Ġwith ĠB ERT base Ġfine - tun ed Ġclass ifier . </s> Wiki Q A </s> The ĠWiki Q A Ġcorpus ĠB IB REF 15 Ġis Ġa Ġpublicly Ġavailable Ġdataset Ġfor Ġopen - domain Ġquestion Ġanswering . ĠThis Ġdataset Ġcontains Ġ3 , 047 Ġquestions Ġfrom ĠBing Ġquery Ġlogs , Ġeach Ġassociated Ġwith Ġsome Ġcandidate Ġanswer Ġsentences Ġfrom ĠWikipedia . ĠWe Ġconducted Ġexperiments Ġon ĠWiki Q A Ġdataset Ġusing ĠCNN , ĠBi L ST M , Ġand ĠAttention Ġbased Ġmodels . ĠThe Ġresults Ġare Ġshown Ġin ĠTable ĠT AB REF 41 Ġ. ĠThe Ġmodels Ġbuilt Ġin ĠNe uron Blocks Ġachieved Ġcompetitive Ġor Ġeven Ġbetter Ġresults Ġwith Ġsimple Ġmodel Ġconfigurations . </s> Conclusion Ġand ĠFuture ĠWork </s> In Ġthis Ġpaper , Ġwe Ġintroduce ĠNe uron Blocks , Ġa ĠD NN Ġtool kit Ġfor ĠN LP Ġtasks Ġbuilt Ġon ĠPy Tor ch . ĠNe uron Blocks Ġtargets Ġthree Ġtypes Ġof Ġengineers , Ġand Ġprovides Ġa Ġtwo - layer Ġsolution Ġto Ġsatisfy Ġthe Ġrequirements Ġfrom Ġall Ġthree Ġtypes Ġof Ġusers . ĠTo Ġbe Ġmore Ġspecific , Ġthe ĠModel ĠZoo Ġconsists Ġof Ġvarious Ġtemplates Ġfor Ġthe Ġmost Ġcommon ĠN LP Ġtasks , Ġwhile Ġthe ĠBlock ĠZoo Ġsupplies Ġa Ġgallery Ġof Ġalternative Ġlayers / modules Ġfor Ġthe Ġnetworks . ĠSuch Ġdesign Ġachieves Ġa Ġbalance Ġbetween Ġgener ality Ġand Ġflexibility . ĠExt ensive Ġexperiments Ġhave Ġverified Ġthe Ġeffectiveness Ġof Ġthis Ġapproach . ĠNe uron Blocks Ġhas Ġbeen Ġwidely Ġused Ġin Ġa Ġproduct Ġteam Ġof Ġa Ġcommercial Ġsearch Ġengine , Ġand Ġsignificantly Ġimproved Ġthe Ġproductivity Ġfor Ġdeveloping ĠN LP ĠD NN Ġapproaches . </s> As Ġan Ġopen - source Ġtool kit , Ġwe Ġwill Ġfurther Ġextend Ġit Ġin Ġvarious Ġdirections . ĠThe Ġfollowing Ġnames Ġa Ġfew Ġexamples .
Introduction Deep Neural Networks (DNN) have been widely employed in industry for solving various Natural Language Processing (NLP) tasks, such as text classification, sequence labeling, question answering, etc. However, when engineers apply DNN models to address specific NLP tasks, they often face the following challenges. The above challenges often hinder the productivity of engineers, and result in less optimal solutions to their given tasks. This motivates us to develop an NLP toolkit for DNN models, which facilitates engineers to develop DNN approaches. Before designing this NLP toolkit, we conducted a survey among engineers and identified a spectrum of three typical personas. To satisfy the requirements of all the above three personas, the NLP toolkit has to be generic enough to cover as many tasks as possible. At the same time, it also needs to be flexible enough to allow alternative network architectures as well as customized modules. Therefore, we analyzed the NLP jobs submitted to a commercial centralized GPU cluster. Table TABREF11 showed that about 87.5% NLP related jobs belong to a few common tasks, including sentence classification, text matching, sequence labeling, MRC, etc. It further suggested that more than 90% of the networks were composed of several common components, such as embedding, CNN/RNN, Transformer and so on. Based on the above observations, we developed NeuronBlocks, a DNN toolkit for NLP tasks. The basic idea is to provide two layers of support to the engineers. The upper layer targets common NLP tasks. For each task, the toolkit contains several end-to-end network templates, which can be immediately instantiated with simple configuration. The bottom layer consists of a suite of reusable and standard components, which can be adopted as building blocks to construct networks with complex architecture. By following the interface guidelines, users can also contribute to this gallery of components with their own modules. The technical contributions of NeuronBlocks are summarized into the following three aspects. Related Work There are several general-purpose deep learning frameworks, such as TensorFlow, PyTorch and Keras, which have gained popularity in NLP community. These frameworks offer huge flexibility in DNN model design and support various NLP tasks. However, building models under these frameworks requires a large overhead of mastering these framework details. Therefore, higher level abstraction to hide the framework details is favored by many engineers. There are also several popular deep learning toolkits in NLP, including OpenNMT BIBREF0 , AllenNLP BIBREF1 etc. OpenNMT is an open-source toolkit mainly targeting neural machine translation or other natural language generation tasks. AllenNLP provides several pre-built models for NLP tasks, such as semantic role labeling, machine comprehension, textual entailment, etc. Although these toolkits reduce the development cost, they are limited to certain tasks, and thus not flexible enough to support new network architectures or new components. Design The Neuronblocks is built on PyTorch. The overall framework is illustrated in Figure FIGREF16 . It consists of two layers: the Block Zoo and the Model Zoo. In Block Zoo, the most commonly used components of deep neural networks are categorized into several groups according to their functions. Within each category, several alternative components are encapsulated into standard and reusable blocks with a consistent interface. These blocks serve as basic and exchangeable units to construct complex network architectures for different NLP tasks. In Model Zoo, the most popular NLP tasks are identified. For each task, several end-to-end network templates are provided in the form of JSON configuration files. Users can simply browse these configurations and choose one or more to instantiate. The whole task can be completed without any coding efforts. Block Zoo We recognize the following major functional categories of neural network components. Each category covers as many commonly used modules as possible. The Block Zoo is an open framework, and more modules can be added in the future. [itemsep= -0.4em,topsep = 0.3em, align=left, labelsep=-0.6em, leftmargin=1.2em] Embedding Layer: Word/character embedding and extra handcrafted feature embedding such as pos-tagging are supported. Neural Network Layers: Block zoo provides common layers like RNN, CNN, QRNN BIBREF2 , Transformer BIBREF3 , Highway network, Encoder Decoder architecture, etc. Furthermore, attention mechanisms are widely used in neural networks. Thus we also support multiple attention layers, such as Linear/Bi-linear Attention, Full Attention BIBREF4 , Bidirectional attention flow BIBREF5 , etc. Meanwhile, regularization layers such as Dropout, Layer Norm, Batch Norm, etc are also supported for improving generalization ability. Loss Function: Besides of the loss functions built in PyTorch, we offer more options such as Focal Loss BIBREF6 . Metrics: For classification task, AUC, Accuracy, Precision/Recall, F1 metrics are supported. For sequence labeling task, F1/Accuracy are supported. For knowledge distillation task, MSE/RMSE are supported. For MRC task, ExactMatch/F1 are supported. Model Zoo In NeuronBlocks, we identify four types of most popular NLP tasks. For each task, we provide various end-to-end network templates. [itemsep= -0.4em,topsep = 0.3em, align=left, labelsep=-0.6em, leftmargin=1.2em] Text Classification and Matching. Tasks such as domain/intent classification, question answer matching are supported. Sequence Labeling. Predict each token in a sequence into predefined types. Common tasks include NER, POS tagging, Slot tagging, etc. Knowledge Distillation BIBREF7 . Teacher-Student based knowledge distillation is one common approach for model compression. NeuronBlocks provides knowledge distillation template to improve the inference speed of heavy DNN models like BERT/GPT. Extractive Machine Reading Comprehension. Given a pair of question and passage, predict the start and end positions of the answer spans in the passage. User Interface NeuronBlocks provides convenient user interface for users to build, train, and test DNN models. The details are described in the following. [itemsep= -0.4em,topsep = 0.3em, align=left, labelsep=-0.6em, leftmargin=1.2em] I/O interface. This part defines model input/output, such as training data, pre-trained models/embeddings, model saving path, etc. Model Architecture interface. This is the key part of the configuration file, which defines the whole model architecture. Figure FIGREF19 shows an example of how to specify a model architecture using the blocks in NeuronBlocks. To be more specific, it consists of a list of layers/blocks to construct the architecture, where the blocks are supplied in the gallery of Block Zoo. Training Parameters interface. In this part, the model optimizer as well as all other training hyper parameters are indicated. Workflow Figure FIGREF34 shows the workflow of building DNN models in NeuronBlocks. Users only need to write a JSON configuration file. They can either instantiate an existing template from Model Zoo, or construct a new architecture based on the blocks from Block Zoo. This configuration file is shared across training, test, and prediction. For model hyper-parameter tuning or architecture modification, users just need to change the JSON configuration file. Advanced users can also contribute novel customized blocks into Block Zoo, as long as they follow the same interface guidelines with the existing blocks. These new blocks can be further shared across all users for model architecture design. Moreover, NeuronBlocks has flexible platform support, such as GPU/CPU, GPU management platforms like PAI. Experiments To verify the performance of NeuronBlocks, we conducted extensive experiments for common NLP tasks on public data sets including CoNLL-2003 BIBREF14 , GLUE benchmark BIBREF13 , and WikiQA corpus BIBREF15 . The experimental results showed that the models built with NeuronBlocks can achieve reliable and competitive results on various tasks, with productivity greatly improved. Sequence Labeling For sequence labeling task, we evaluated NeuronBlocks on CoNLL-2003 BIBREF14 English NER dataset, following most works on the same task. This dataset includes four types of named entities, namely, PERSON, LOCATION, ORGANIZATION, and MISC. We adopted the BIOES tagging scheme instead of IOB, as many previous works indicated meaningful improvement with BIOES scheme BIBREF16 , BIBREF17 . Table TABREF28 shows the results on CoNLL-2003 Englist testb dataset, with 12 different combinations of network layers/blocks, such as word/character embedding, CNN/LSTM and CRF. The results suggest that the flexible combination of layers/blocks in NeuronBlocks can easily reproduce the performance of original models, with comparative or slightly better performance. GLUE Benchmark The General Language Understanding Evaluation (GLUE) benchmark BIBREF13 is a collection of natural language understanding tasks. We experimented on the GLUE benchmark tasks using BiLSTM and Attention based models. As shown in Table TABREF29 , the models built by NeuronBlocks can achieve competitive or even better results on GLUE tasks with minimal coding efforts. Knowledge Distillation We evaluated Knowledge Distillation task in NeuronBlocks on a dataset collected from one commercial search engine. We refer to this dataset as Domain Classification Dataset. Each sample in this dataset consists of two parts, i.e., a question and a binary label indicating whether the question belongs to a specific domain. Table TABREF36 shows the results, where Area Under Curve (AUC) metric is used as the performance evaluation criteria and Queries per Second (QPS) is used to measure inference speed. By knowledge distillation training approach, the student model by NeuronBlocks managed to get 23-27 times inference speedup with only small performance regression compared with BERTbase fine-tuned classifier. WikiQA The WikiQA corpus BIBREF15 is a publicly available dataset for open-domain question answering. This dataset contains 3,047 questions from Bing query logs, each associated with some candidate answer sentences from Wikipedia. We conducted experiments on WikiQA dataset using CNN, BiLSTM, and Attention based models. The results are shown in Table TABREF41 . The models built in NeuronBlocks achieved competitive or even better results with simple model configurations. Conclusion and Future Work In this paper, we introduce NeuronBlocks, a DNN toolkit for NLP tasks built on PyTorch. NeuronBlocks targets three types of engineers, and provides a two-layer solution to satisfy the requirements from all three types of users. To be more specific, the Model Zoo consists of various templates for the most common NLP tasks, while the Block Zoo supplies a gallery of alternative layers/modules for the networks. Such design achieves a balance between generality and flexibility. Extensive experiments have verified the effectiveness of this approach. NeuronBlocks has been widely used in a product team of a commercial search engine, and significantly improved the productivity for developing NLP DNN approaches. As an open-source toolkit, we will further extend it in various directions. The following names a few examples.
Do they report results only on English?
b1cf5739467ba90059add58d11b73d075a11ec86
Do Ġthey Ġreport Ġresults Ġonly Ġon ĠEnglish ?
Un answer able
<s> Do Ġthey Ġreport Ġresults Ġonly Ġon ĠEnglish ? </s> Introduction </s> Deep ĠNeural ĠNetworks Ġ( D NN ) Ġhave Ġbeen Ġwidely Ġemployed Ġin Ġindustry Ġfor Ġsolving Ġvarious ĠNatural ĠLanguage ĠProcessing Ġ( N LP ) Ġtasks , Ġsuch Ġas Ġtext Ġclassification , Ġsequence Ġlabeling , Ġquestion Ġanswering , Ġetc . ĠHowever , Ġwhen Ġengineers Ġapply ĠD NN Ġmodels Ġto Ġaddress Ġspecific ĠN LP Ġtasks , Ġthey Ġoften Ġface Ġthe Ġfollowing Ġchallenges . </s> The Ġabove Ġchallenges Ġoften Ġhinder Ġthe Ġproductivity Ġof Ġengineers , Ġand Ġresult Ġin Ġless Ġoptimal Ġsolutions Ġto Ġtheir Ġgiven Ġtasks . ĠThis Ġmotiv ates Ġus Ġto Ġdevelop Ġan ĠN LP Ġtool kit Ġfor ĠD NN Ġmodels , Ġwhich Ġfacilitates Ġengineers Ġto Ġdevelop ĠD NN Ġapproaches . ĠBefore Ġdesigning Ġthis ĠN LP Ġtool kit , Ġwe Ġconducted Ġa Ġsurvey Ġamong Ġengineers Ġand Ġidentified Ġa Ġspectrum Ġof Ġthree Ġtypical Ġperson as . </s> To Ġsatisfy Ġthe Ġrequirements Ġof Ġall Ġthe Ġabove Ġthree Ġperson as , Ġthe ĠN LP Ġtool kit Ġhas Ġto Ġbe Ġgeneric Ġenough Ġto Ġcover Ġas Ġmany Ġtasks Ġas Ġpossible . ĠAt Ġthe Ġsame Ġtime , Ġit Ġalso Ġneeds Ġto Ġbe Ġflexible Ġenough Ġto Ġallow Ġalternative Ġnetwork Ġarchitectures Ġas Ġwell Ġas Ġcustomized Ġmodules . ĠTherefore , Ġwe Ġanalyzed Ġthe ĠN LP Ġjobs Ġsubmitted Ġto Ġa Ġcommercial Ġcentralized ĠGPU Ġcluster . ĠTable ĠT AB REF 11 Ġshowed Ġthat Ġabout Ġ87 . 5 % ĠN LP Ġrelated Ġjobs Ġbelong Ġto Ġa Ġfew Ġcommon Ġtasks , Ġincluding Ġsentence Ġclassification , Ġtext Ġmatching , Ġsequence Ġlabeling , ĠM RC , Ġetc . ĠIt Ġfurther Ġsuggested Ġthat Ġmore Ġthan Ġ90 % Ġof Ġthe Ġnetworks Ġwere Ġcomposed Ġof Ġseveral Ġcommon Ġcomponents , Ġsuch Ġas Ġembed ding , ĠCNN / R NN , ĠTrans former Ġand Ġso Ġon . </s> Based Ġon Ġthe Ġabove Ġobservations , Ġwe Ġdeveloped ĠNe uron Blocks , Ġa ĠD NN Ġtool kit Ġfor ĠN LP Ġtasks . ĠThe Ġbasic Ġidea Ġis Ġto Ġprovide Ġtwo Ġlayers Ġof Ġsupport Ġto Ġthe Ġengineers . ĠThe Ġupper Ġlayer Ġtargets Ġcommon ĠN LP Ġtasks . ĠFor Ġeach Ġtask , Ġthe Ġtool kit Ġcontains Ġseveral Ġend - to - end Ġnetwork Ġtemplates , Ġwhich Ġcan Ġbe Ġimmediately Ġinstant iated Ġwith Ġsimple Ġconfiguration . ĠThe Ġbottom Ġlayer Ġconsists Ġof Ġa Ġsuite Ġof Ġreusable Ġand Ġstandard Ġcomponents , Ġwhich Ġcan Ġbe Ġadopted Ġas Ġbuilding Ġblocks Ġto Ġconstruct Ġnetworks Ġwith Ġcomplex Ġarchitecture . ĠBy Ġfollowing Ġthe Ġinterface Ġguidelines , Ġusers Ġcan Ġalso Ġcontribute Ġto Ġthis Ġgallery Ġof Ġcomponents Ġwith Ġtheir Ġown Ġmodules . </s> The Ġtechnical Ġcontributions Ġof ĠNe uron Blocks Ġare Ġsummarized Ġinto Ġthe Ġfollowing Ġthree Ġaspects . </s> Related ĠWork </s> There Ġare Ġseveral Ġgeneral - purpose Ġdeep Ġlearning Ġframeworks , Ġsuch Ġas ĠT ensor Flow , ĠPy Tor ch Ġand ĠKer as , Ġwhich Ġhave Ġgained Ġpopularity Ġin ĠN LP Ġcommunity . ĠThese Ġframeworks Ġoffer Ġhuge Ġflexibility Ġin ĠD NN Ġmodel Ġdesign Ġand Ġsupport Ġvarious ĠN LP Ġtasks . ĠHowever , Ġbuilding Ġmodels Ġunder Ġthese Ġframeworks Ġrequires Ġa Ġlarge Ġoverhead Ġof Ġmastering Ġthese Ġframework Ġdetails . ĠTherefore , Ġhigher Ġlevel Ġabstraction Ġto Ġhide Ġthe Ġframework Ġdetails Ġis Ġfavored Ġby Ġmany Ġengineers . </s> There Ġare Ġalso Ġseveral Ġpopular Ġdeep Ġlearning Ġtool k its Ġin ĠN LP , Ġincluding ĠOpen N MT ĠB IB REF 0 Ġ, ĠAllen N LP ĠB IB REF 1 Ġetc . ĠOpen N MT Ġis Ġan Ġopen - source Ġtool kit Ġmainly Ġtargeting Ġneural Ġmachine Ġtranslation Ġor Ġother Ġnatural Ġlanguage Ġgeneration Ġtasks . ĠAllen N LP Ġprovides Ġseveral Ġpre - built Ġmodels Ġfor ĠN LP Ġtasks , Ġsuch Ġas Ġsemantic Ġrole Ġlabeling , Ġmachine Ġcomprehension , Ġtextual Ġentail ment , Ġetc . ĠAlthough Ġthese Ġtool k its Ġreduce Ġthe Ġdevelopment Ġcost , Ġthey Ġare Ġlimited Ġto Ġcertain Ġtasks , Ġand Ġthus Ġnot Ġflexible Ġenough Ġto Ġsupport Ġnew Ġnetwork Ġarchitectures Ġor Ġnew Ġcomponents . </s> Design </s> The ĠNe uron blocks Ġis Ġbuilt Ġon ĠPy Tor ch . ĠThe Ġoverall Ġframework Ġis Ġillustrated Ġin ĠFigure ĠFIG REF 16 Ġ. ĠIt Ġconsists Ġof Ġtwo Ġlayers : Ġthe ĠBlock ĠZoo Ġand Ġthe ĠModel ĠZoo . ĠIn ĠBlock ĠZoo , Ġthe Ġmost Ġcommonly Ġused Ġcomponents Ġof Ġdeep Ġneural Ġnetworks Ġare Ġcategorized Ġinto Ġseveral Ġgroups Ġaccording Ġto Ġtheir Ġfunctions . ĠWithin Ġeach Ġcategory , Ġseveral Ġalternative Ġcomponents Ġare Ġencaps ulated Ġinto Ġstandard Ġand Ġreusable Ġblocks Ġwith Ġa Ġconsistent Ġinterface . ĠThese Ġblocks Ġserve Ġas Ġbasic Ġand Ġexchange able Ġunits Ġto Ġconstruct Ġcomplex Ġnetwork Ġarchitectures Ġfor Ġdifferent ĠN LP Ġtasks . ĠIn ĠModel ĠZoo , Ġthe Ġmost Ġpopular ĠN LP Ġtasks Ġare Ġidentified . ĠFor Ġeach Ġtask , Ġseveral Ġend - to - end Ġnetwork Ġtemplates Ġare Ġprovided Ġin Ġthe Ġform Ġof ĠJSON Ġconfiguration Ġfiles . ĠUsers Ġcan Ġsimply Ġbrowse Ġthese Ġconfigurations Ġand Ġchoose Ġone Ġor Ġmore Ġto Ġinstant iate . ĠThe Ġwhole Ġtask Ġcan Ġbe Ġcompleted Ġwithout Ġany Ġcoding Ġefforts . </s> Block ĠZoo </s> We Ġrecognize Ġthe Ġfollowing Ġmajor Ġfunctional Ġcategories Ġof Ġneural Ġnetwork Ġcomponents . ĠEach Ġcategory Ġcovers Ġas Ġmany Ġcommonly Ġused Ġmodules Ġas Ġpossible . ĠThe ĠBlock ĠZoo Ġis Ġan Ġopen Ġframework , Ġand Ġmore Ġmodules Ġcan Ġbe Ġadded Ġin Ġthe Ġfuture . </s> [ item se p = Ġ- 0 . 4 em , top se p Ġ= Ġ0 . 3 em , Ġalign = left , Ġlabel se p =- 0 . 6 em , Ġleft margin = 1 . 2 em ] </s> Emb edd ing ĠLayer : ĠWord / character Ġembed ding Ġand Ġextra Ġhand crafted Ġfeature Ġembed ding Ġsuch Ġas Ġpos - tag ging Ġare Ġsupported . </s> Ne ural ĠNetwork ĠL ayers : ĠBlock Ġzoo Ġprovides Ġcommon Ġlayers Ġlike ĠR NN , ĠCNN , ĠQR NN ĠB IB REF 2 Ġ, ĠTrans former ĠB IB REF 3 Ġ, ĠHighway Ġnetwork , ĠEnc oder ĠDecoder Ġarchitecture , Ġetc . ĠFurthermore , Ġattention Ġmechanisms Ġare Ġwidely Ġused Ġin Ġneural Ġnetworks . ĠThus Ġwe Ġalso Ġsupport Ġmultiple Ġattention Ġlayers , Ġsuch Ġas ĠLinear / Bi - linear ĠAttention , ĠFull ĠAttention ĠB IB REF 4 Ġ, ĠBid irection al Ġattention Ġflow ĠB IB REF 5 Ġ, Ġetc . ĠMeanwhile , Ġregular ization Ġlayers Ġsuch Ġas ĠDrop out , ĠLayer ĠNorm , ĠB atch ĠNorm , Ġetc Ġare Ġalso Ġsupported Ġfor Ġimproving Ġgeneral ization Ġability . </s> L oss ĠFunction : ĠBesides Ġof Ġthe Ġloss Ġfunctions Ġbuilt Ġin ĠPy Tor ch , Ġwe Ġoffer Ġmore Ġoptions Ġsuch Ġas ĠF ocal ĠLoss ĠB IB REF 6 Ġ. </s> Met rics : ĠFor Ġclassification Ġtask , ĠA UC , ĠAccuracy , ĠPrecision / Rec all , ĠF 1 Ġmetrics Ġare Ġsupported . ĠFor Ġsequence Ġlabeling Ġtask , ĠF 1 / Acc uracy Ġare Ġsupported . ĠFor Ġknowledge Ġdist illation Ġtask , ĠM SE / RM SE Ġare Ġsupported . ĠFor ĠM RC Ġtask , ĠEx act Match / F 1 Ġare Ġsupported . </s> Model ĠZoo </s> In ĠNe uron Blocks , Ġwe Ġidentify Ġfour Ġtypes Ġof Ġmost Ġpopular ĠN LP Ġtasks . ĠFor Ġeach Ġtask , Ġwe Ġprovide Ġvarious Ġend - to - end Ġnetwork Ġtemplates . </s> [ item se p = Ġ- 0 . 4 em , top se p Ġ= Ġ0 . 3 em , Ġalign = left , Ġlabel se p =- 0 . 6 em , Ġleft margin = 1 . 2 em ] </s> Text ĠClassification Ġand ĠMatch ing . ĠT asks Ġsuch Ġas Ġdomain / intent Ġclassification , Ġquestion Ġanswer Ġmatching Ġare Ġsupported . </s> Sequ ence ĠLab eling . ĠPredict Ġeach Ġtoken Ġin Ġa Ġsequence Ġinto Ġpred efined Ġtypes . ĠCommon Ġtasks Ġinclude ĠN ER , ĠPOS Ġtagging , ĠSlot Ġtagging , Ġetc . </s> Know ledge ĠDist illation ĠB IB REF 7 Ġ. ĠTeacher - Student Ġbased Ġknowledge Ġdist illation Ġis Ġone Ġcommon Ġapproach Ġfor Ġmodel Ġcompression . ĠNe uron Blocks Ġprovides Ġknowledge Ġdist illation Ġtemplate Ġto Ġimprove Ġthe Ġinference Ġspeed Ġof Ġheavy ĠD NN Ġmodels Ġlike ĠB ERT / G PT . </s> Ext ractive ĠMachine ĠReading ĠComp rehens ion . ĠGiven Ġa Ġpair Ġof Ġquestion Ġand Ġpassage , Ġpredict Ġthe Ġstart Ġand Ġend Ġpositions Ġof Ġthe Ġanswer Ġspans Ġin Ġthe Ġpassage . </s> User ĠInterface </s> Ne uron Blocks Ġprovides Ġconvenient Ġuser Ġinterface Ġfor Ġusers Ġto Ġbuild , Ġtrain , Ġand Ġtest ĠD NN Ġmodels . ĠThe Ġdetails Ġare Ġdescribed Ġin Ġthe Ġfollowing . </s> [ item se p = Ġ- 0 . 4 em , top se p Ġ= Ġ0 . 3 em , Ġalign = left , Ġlabel se p =- 0 . 6 em , Ġleft margin = 1 . 2 em ] </s> I / O Ġinterface . ĠThis Ġpart Ġdefines Ġmodel Ġinput / output , Ġsuch Ġas Ġtraining Ġdata , Ġpre - trained Ġmodels / embed d ings , Ġmodel Ġsaving Ġpath , Ġetc . </s> Model ĠArchitecture Ġinterface . ĠThis Ġis Ġthe Ġkey Ġpart Ġof Ġthe Ġconfiguration Ġfile , Ġwhich Ġdefines Ġthe Ġwhole Ġmodel Ġarchitecture . ĠFigure ĠFIG REF 19 Ġshows Ġan Ġexample Ġof Ġhow Ġto Ġspecify Ġa Ġmodel Ġarchitecture Ġusing Ġthe Ġblocks Ġin ĠNe uron Blocks . ĠTo Ġbe Ġmore Ġspecific , Ġit Ġconsists Ġof Ġa Ġlist Ġof Ġlayers / blocks Ġto Ġconstruct Ġthe Ġarchitecture , Ġwhere Ġthe Ġblocks Ġare Ġsupplied Ġin Ġthe Ġgallery Ġof ĠBlock ĠZoo . </s> Training ĠParameters Ġinterface . ĠIn Ġthis Ġpart , Ġthe Ġmodel Ġoptim izer Ġas Ġwell Ġas Ġall Ġother Ġtraining Ġhyper Ġparameters Ġare Ġindicated . </s> Work flow </s> Figure ĠFIG REF 34 Ġshows Ġthe Ġworkflow Ġof Ġbuilding ĠD NN Ġmodels Ġin ĠNe uron Blocks . ĠUsers Ġonly Ġneed Ġto Ġwrite Ġa ĠJSON Ġconfiguration Ġfile . ĠThey Ġcan Ġeither Ġinstant iate Ġan Ġexisting Ġtemplate Ġfrom ĠModel ĠZoo , Ġor Ġconstruct Ġa Ġnew Ġarchitecture Ġbased Ġon Ġthe Ġblocks Ġfrom ĠBlock ĠZoo . ĠThis Ġconfiguration Ġfile Ġis Ġshared Ġacross Ġtraining , Ġtest , Ġand Ġprediction . ĠFor Ġmodel Ġhyper - param eter Ġtuning Ġor Ġarchitecture Ġmodification , Ġusers Ġjust Ġneed Ġto Ġchange Ġthe ĠJSON Ġconfiguration Ġfile . ĠAdvanced Ġusers Ġcan Ġalso Ġcontribute Ġnovel Ġcustomized Ġblocks Ġinto ĠBlock ĠZoo , Ġas Ġlong Ġas Ġthey Ġfollow Ġthe Ġsame Ġinterface Ġguidelines Ġwith Ġthe Ġexisting Ġblocks . ĠThese Ġnew Ġblocks Ġcan Ġbe Ġfurther Ġshared Ġacross Ġall Ġusers Ġfor Ġmodel Ġarchitecture Ġdesign . ĠMoreover , ĠNe uron Blocks Ġhas Ġflexible Ġplatform Ġsupport , Ġsuch Ġas ĠGPU / CPU , ĠGPU Ġmanagement Ġplatforms Ġlike ĠPA I . </s> Exper iments </s> To Ġverify Ġthe Ġperformance Ġof ĠNe uron Blocks , Ġwe Ġconducted Ġextensive Ġexperiments Ġfor Ġcommon ĠN LP Ġtasks Ġon Ġpublic Ġdata Ġsets Ġincluding ĠCo N LL - 2003 ĠB IB REF 14 Ġ, ĠGL UE Ġbenchmark ĠB IB REF 13 Ġ, Ġand ĠWiki Q A Ġcorpus ĠB IB REF 15 Ġ. ĠThe Ġexperimental Ġresults Ġshowed Ġthat Ġthe Ġmodels Ġbuilt Ġwith ĠNe uron Blocks Ġcan Ġachieve Ġreliable Ġand Ġcompetitive Ġresults Ġon Ġvarious Ġtasks , Ġwith Ġproductivity Ġgreatly Ġimproved . </s> Sequ ence ĠLab eling </s> For Ġsequence Ġlabeling Ġtask , Ġwe Ġevaluated ĠNe uron Blocks Ġon ĠCo N LL - 2003 ĠB IB REF 14 ĠEnglish ĠN ER Ġdataset , Ġfollowing Ġmost Ġworks Ġon Ġthe Ġsame Ġtask . ĠThis Ġdataset Ġincludes Ġfour Ġtypes Ġof Ġnamed Ġentities , Ġnamely , ĠPERSON , ĠLOC ATION , ĠOR GAN IZ ATION , Ġand ĠM ISC . ĠWe Ġadopted Ġthe ĠB IO ES Ġtagging Ġscheme Ġinstead Ġof ĠI OB , Ġas Ġmany Ġprevious Ġworks Ġindicated Ġmeaningful Ġimprovement Ġwith ĠB IO ES Ġscheme ĠB IB REF 16 Ġ, ĠB IB REF 17 Ġ. ĠTable ĠT AB REF 28 Ġshows Ġthe Ġresults Ġon ĠCo N LL - 2003 ĠEng list Ġtest b Ġdataset , Ġwith Ġ12 Ġdifferent Ġcombinations Ġof Ġnetwork Ġlayers / blocks , Ġsuch Ġas Ġword / character Ġembed ding , ĠCNN / L ST M Ġand ĠCR F . ĠThe Ġresults Ġsuggest Ġthat Ġthe Ġflexible Ġcombination Ġof Ġlayers / blocks Ġin ĠNe uron Blocks Ġcan Ġeasily Ġreproduce Ġthe Ġperformance Ġof Ġoriginal Ġmodels , Ġwith Ġcomparative Ġor Ġslightly Ġbetter Ġperformance . </s> GL UE ĠBench mark </s> The ĠGeneral ĠLanguage ĠUnderstanding ĠEvaluation Ġ( GL UE ) Ġbenchmark ĠB IB REF 13 Ġis Ġa Ġcollection Ġof Ġnatural Ġlanguage Ġunderstanding Ġtasks . ĠWe Ġexperimented Ġon Ġthe ĠGL UE Ġbenchmark Ġtasks Ġusing ĠBi L ST M Ġand ĠAttention Ġbased Ġmodels . ĠAs Ġshown Ġin ĠTable ĠT AB REF 29 Ġ, Ġthe Ġmodels Ġbuilt Ġby ĠNe uron Blocks Ġcan Ġachieve Ġcompetitive Ġor Ġeven Ġbetter Ġresults Ġon ĠGL UE Ġtasks Ġwith Ġminimal Ġcoding Ġefforts . </s> Know ledge ĠDist illation </s> We Ġevaluated ĠKnowledge ĠDist illation Ġtask Ġin ĠNe uron Blocks Ġon Ġa Ġdataset Ġcollected Ġfrom Ġone Ġcommercial Ġsearch Ġengine . ĠWe Ġrefer Ġto Ġthis Ġdataset Ġas ĠDomain ĠClassification ĠDat as et . ĠEach Ġsample Ġin Ġthis Ġdataset Ġconsists Ġof Ġtwo Ġparts , Ġi . e ., Ġa Ġquestion Ġand Ġa Ġbinary Ġlabel Ġindicating Ġwhether Ġthe Ġquestion Ġbelongs Ġto Ġa Ġspecific Ġdomain . ĠTable ĠT AB REF 36 Ġshows Ġthe Ġresults , Ġwhere ĠArea ĠUnder ĠCurve Ġ( A UC ) Ġmetric Ġis Ġused Ġas Ġthe Ġperformance Ġevaluation Ġcriteria Ġand ĠQu eries Ġper ĠSecond Ġ( Q PS ) Ġis Ġused Ġto Ġmeasure Ġinference Ġspeed . ĠBy Ġknowledge Ġdist illation Ġtraining Ġapproach , Ġthe Ġstudent Ġmodel Ġby ĠNe uron Blocks Ġmanaged Ġto Ġget Ġ23 - 27 Ġtimes Ġinference Ġspeed up Ġwith Ġonly Ġsmall Ġperformance Ġregression Ġcompared Ġwith ĠB ERT base Ġfine - tun ed Ġclass ifier . </s> Wiki Q A </s> The ĠWiki Q A Ġcorpus ĠB IB REF 15 Ġis Ġa Ġpublicly Ġavailable Ġdataset Ġfor Ġopen - domain Ġquestion Ġanswering . ĠThis Ġdataset Ġcontains Ġ3 , 047 Ġquestions Ġfrom ĠBing Ġquery Ġlogs , Ġeach Ġassociated Ġwith Ġsome Ġcandidate Ġanswer Ġsentences Ġfrom ĠWikipedia . ĠWe Ġconducted Ġexperiments Ġon ĠWiki Q A Ġdataset Ġusing ĠCNN , ĠBi L ST M , Ġand ĠAttention Ġbased Ġmodels . ĠThe Ġresults Ġare Ġshown Ġin ĠTable ĠT AB REF 41 Ġ. ĠThe Ġmodels Ġbuilt Ġin ĠNe uron Blocks Ġachieved Ġcompetitive Ġor Ġeven Ġbetter Ġresults Ġwith Ġsimple Ġmodel Ġconfigurations . </s> Conclusion Ġand ĠFuture ĠWork </s> In Ġthis Ġpaper , Ġwe Ġintroduce ĠNe uron Blocks , Ġa ĠD NN Ġtool kit Ġfor ĠN LP Ġtasks Ġbuilt Ġon ĠPy Tor ch . ĠNe uron Blocks Ġtargets Ġthree Ġtypes Ġof Ġengineers , Ġand Ġprovides Ġa Ġtwo - layer Ġsolution Ġto Ġsatisfy Ġthe Ġrequirements Ġfrom Ġall Ġthree Ġtypes Ġof Ġusers . ĠTo Ġbe Ġmore Ġspecific , Ġthe ĠModel ĠZoo Ġconsists Ġof Ġvarious Ġtemplates Ġfor Ġthe Ġmost Ġcommon ĠN LP Ġtasks , Ġwhile Ġthe ĠBlock ĠZoo Ġsupplies Ġa Ġgallery Ġof Ġalternative Ġlayers / modules Ġfor Ġthe Ġnetworks . ĠSuch Ġdesign Ġachieves Ġa Ġbalance Ġbetween Ġgener ality Ġand Ġflexibility . ĠExt ensive Ġexperiments Ġhave Ġverified Ġthe Ġeffectiveness Ġof Ġthis Ġapproach . ĠNe uron Blocks Ġhas Ġbeen Ġwidely Ġused Ġin Ġa Ġproduct Ġteam Ġof Ġa Ġcommercial Ġsearch Ġengine , Ġand Ġsignificantly Ġimproved Ġthe Ġproductivity Ġfor Ġdeveloping ĠN LP ĠD NN Ġapproaches . </s> As Ġan Ġopen - source Ġtool kit , Ġwe Ġwill Ġfurther Ġextend Ġit Ġin Ġvarious Ġdirections . ĠThe Ġfollowing Ġnames Ġa Ġfew Ġexamples .