PubMed+PMC
EXTRACTIVE
PubMed+PMC (the data used for BioBERTv1.0) and/or CORD-19 (Covid-19 Open Research Dataset)
EXTRACTIVE
We train Word2Vec with vector size $d_\mathrm {W2V} = d_\mathrm {LM} = 768$ on PubMed+PMC (see Appendix for details). Then, we follow the procedure described in Section SECREF3 to update the wordpiece embedding layer and tokenizer of general-domain BERT.
In Section SECREF4, we use the proposed method to domain-adapt BERT on PubMed+PMC (the data used for BioBERTv1.0) and/or CORD-19 (Covid-19 Open Research Dataset). We improve over general-domain BERT on eight out of eight biomedical NER tasks, using a fraction of the compute cost associated with BioBERT. In Section SECREF5, we show how to quickly adapt an existing Question Answering model to text about the Covid-19 pandemic, without any target-domain Language Model pretraining or finetuning.
2004.03354
Introduction </s> P ret rained ĠLanguage ĠModels Ġ( PT L Ms ) Ġsuch Ġas ĠB ERT ĠB IB REF 1 Ġhave Ġspearheaded Ġadvances Ġon Ġmany ĠN LP Ġtasks . ĠUsually , ĠP TL Ms Ġare Ġpret rained Ġon Ġunl abel ed Ġgeneral - domain Ġand / or Ġmixed - domain Ġtext , Ġsuch Ġas ĠWikipedia , Ġdigital Ġbooks Ġor Ġthe ĠCommon ĠC rawl Ġcorpus . </s> When Ġapplying ĠP TL Ms Ġto Ġspecific Ġdomains , Ġit Ġcan Ġbe Ġuseful Ġto Ġdomain - adapt Ġthem . ĠDomain Ġadaptation Ġof ĠP TL Ms Ġhas Ġtypically Ġbeen Ġachieved Ġby Ġpret raining Ġon Ġtarget - domain Ġtext . ĠOne Ġsuch Ġmodel Ġis ĠBio BER T ĠB IB REF 2 , Ġwhich Ġwas Ġinitialized Ġfrom Ġgeneral - domain ĠB ERT Ġand Ġthen Ġpret rained Ġon Ġbiomedical Ġscientific Ġpublications . ĠThe Ġdomain Ġadaptation Ġis Ġshown Ġto Ġbe Ġhelpful Ġfor Ġtarget - domain Ġtasks Ġsuch Ġas Ġbiomedical ĠNamed ĠEntity ĠRecogn ition Ġ( NER ) Ġor ĠQuestion ĠAn sw ering Ġ( Q A ). ĠOn Ġthe Ġdownside , Ġthe Ġcomputational Ġcost Ġof Ġpret raining Ġcan Ġbe Ġconsiderable : ĠBio BER T v 1 . 0 Ġwas Ġadapted Ġfor Ġten Ġdays Ġon Ġeight Ġlarge ĠGPUs Ġ( see ĠTable ĠT AB REF 1 ), Ġwhich Ġis Ġexpensive , Ġenvironmentally Ġunf riend ly , Ġprohib itive Ġfor Ġsmall Ġresearch Ġlabs Ġand Ġstudents , Ġand Ġmay Ġdelay Ġprototyp ing Ġon Ġemerging Ġdomains . </s> We Ġtherefore Ġpropose Ġa Ġfast , ĠCPU - only Ġdomain - adapt ation Ġmethod Ġfor ĠP TL Ms : ĠWe Ġtrain ĠWord 2 V ec ĠB IB REF 3 Ġon Ġtarget - domain Ġtext Ġand Ġalign Ġthe Ġresulting Ġword Ġvectors Ġwith Ġthe Ġword piece Ġvectors Ġof Ġan Ġexisting Ġgeneral - domain ĠP TL M . ĠThe ĠP TL M Ġthus Ġgains Ġdomain - specific Ġlex ical Ġknowledge Ġin Ġthe Ġform Ġof Ġadditional Ġword Ġvectors , Ġbut Ġits Ġdeeper Ġlayers Ġremain Ġunchanged . ĠSince ĠWord 2 V ec Ġand Ġthe Ġvector Ġspace Ġalignment Ġare Ġefficient Ġmodels , Ġthe Ġprocess Ġrequires Ġa Ġfraction Ġof Ġthe Ġresources Ġassociated Ġwith Ġpret raining Ġthe ĠP TL M Ġitself , Ġand Ġit Ġcan Ġbe Ġdone Ġon ĠCPU . </s> In ĠSection ĠSEC REF 4 , Ġwe Ġuse Ġthe Ġproposed Ġmethod Ġto Ġdomain - adapt ĠB ERT Ġon ĠPubMed + PM C Ġ( the Ġdata Ġused Ġfor ĠBio BER T v 1 . 0 ) Ġand / or ĠC ORD - 19 Ġ( C ov id - 19 ĠOpen ĠResearch ĠDat as et ). ĠWe Ġimprove Ġover Ġgeneral - domain ĠB ERT Ġon Ġeight Ġout Ġof Ġeight Ġbiomedical ĠN ER Ġtasks , Ġusing Ġa Ġfraction Ġof Ġthe Ġcompute Ġcost Ġassociated Ġwith ĠBio BER T . ĠIn ĠSection ĠSEC REF 5 , Ġwe Ġshow Ġhow Ġto Ġquickly Ġadapt Ġan Ġexisting ĠQuestion ĠAn sw ering Ġmodel Ġto Ġtext Ġabout Ġthe ĠCov id - 19 Ġpand emic , Ġwithout Ġany Ġtarget - domain ĠLanguage ĠModel Ġpret raining Ġor Ġfin et uning . </s> Related Ġwork Ġ: :: ĠThe ĠB ERT ĠP TL M </s> For Ġour Ġpurpose , Ġa ĠP TL M Ġconsists Ġof Ġthree Ġparts : ĠA Ġtoken izer Ġ$\ math cal Ġ{ T } _ \ math rm Ġ{ LM } Ġ: Ġ\ math bb Ġ{ L } ^ + Ġ\ right arrow Ġ\ math bb Ġ{ L } _ \ math rm Ġ{ LM } ^ + $, Ġa Ġword piece Ġembed ding Ġfunction Ġ$\ math cal Ġ{ E } _ \ math rm Ġ{ LM }: Ġ\ math bb Ġ{ L } _ \ math rm Ġ{ LM } Ġ\ right arrow Ġ\ math bb Ġ{ R } ^{ d _ \ math rm Ġ{ LM }} $ Ġand Ġan Ġenc oder Ġfunction Ġ$\ math cal Ġ{ F } _ \ math rm Ġ{ LM } $. Ġ$\ math bb Ġ{ L } _ \ math rm Ġ{ LM } $ Ġis Ġa Ġlimited Ġvocabulary Ġof Ġword pieces . ĠAll Ġwords Ġthat Ġare Ġnot Ġin Ġ$\ math bb Ġ{ L } _ \ math rm Ġ{ LM } $ Ġare Ġtoken ized Ġinto Ġsequences Ġof Ġshorter Ġword pieces , Ġe . g ., Ġt achy card ia Ġbecomes Ġta Ġ## chy Ġ## card Ġ## ia . ĠGiven Ġa Ġsentence Ġ$ S Ġ= Ġ[ w _ 1 , Ġ\ ld ots Ġ, Ġw _ T ] $, Ġtoken ized Ġas Ġ$\ math cal Ġ{ T } _ \ math rm Ġ{ LM } ( S ) Ġ= Ġ[ \ math cal Ġ{ T } _ \ math rm Ġ{ LM } ( w _ 1 ); Ġ\ ld ots Ġ; Ġ\ math cal Ġ{ T } _ \ math rm Ġ{ LM } ( w _ T )] $, Ġ$\ math cal Ġ{ E } _ \ math rm Ġ{ LM } $ Ġembed s Ġevery Ġword piece Ġin Ġ$\ math cal Ġ{ T } _ \ math rm Ġ{ LM } ( S ) $ Ġinto Ġa Ġreal - valued , Ġtrain able Ġword piece Ġvector . ĠThe Ġword piece Ġvectors Ġof Ġthe Ġentire Ġsequence Ġare Ġstacked Ġand Ġfed Ġinto Ġ$\ math cal Ġ{ F } _ \ math rm Ġ{ LM } $. ĠNote Ġthat Ġwe Ġconsider Ġposition Ġand Ġsegment Ġembed d ings Ġto Ġbe Ġa Ġpart Ġof Ġ$\ math cal Ġ{ F } _ \ math rm Ġ{ LM } $ Ġrather Ġthan Ġ$\ math cal Ġ{ E } _ \ math rm Ġ{ LM } $. </s> In Ġthe Ġcase Ġof ĠB ERT , Ġ$\ math cal Ġ{ F } _ \ math rm Ġ{ LM } $ Ġis Ġa ĠTrans former ĠB IB REF 4 , Ġfollowed Ġby Ġa Ġfinal ĠFeed - Forward ĠNet . ĠDuring Ġpret raining , Ġthe ĠFeed - Forward ĠNet Ġpredicts Ġthe Ġidentity Ġof Ġmasked Ġword pieces . ĠWhen Ġfin et uning Ġon Ġa Ġsupervised Ġtask , Ġit Ġis Ġusually Ġreplaced Ġwith Ġa Ġrandomly Ġinitialized Ġtask - specific Ġlayer . </s> Related Ġwork Ġ: :: ĠDomain - adapt ed ĠP TL Ms </s> Domain Ġadaptation Ġof ĠP TL Ms Ġis Ġtypically Ġachieved Ġby Ġpret raining Ġon Ġunl abel ed Ġtarget - domain Ġtext . ĠSome Ġexamples Ġof Ġsuch Ġmodels Ġare ĠBio BER T ĠB IB REF 2 , Ġwhich Ġwas Ġpret rained Ġon Ġthe ĠPubMed Ġand / or ĠPubMed ĠCentral Ġ( PM C ) Ġcorpor a , ĠSci BER T ĠB IB REF 5 , Ġwhich Ġwas Ġpret rained Ġon Ġpapers Ġfrom ĠSem antic Sch olar , ĠClinical BER T ĠB IB REF 6 , ĠB IB REF 7 Ġand ĠClinical XL Net ĠB IB REF 8 , Ġwhich Ġwere Ġpret rained Ġon Ġclinical Ġpatient Ġnotes , Ġand ĠAdapt a BER T ĠB IB REF 9 , Ġwhich Ġwas Ġpret rained Ġon ĠEarly ĠModern ĠEnglish Ġtext . ĠIn Ġmost Ġcases , Ġa Ġdomain - adapt ed ĠP TL M Ġis Ġinitialized Ġfrom Ġa Ġgeneral - domain ĠP TL M Ġ( e . g ., Ġstandard ĠB ERT ), Ġthough ĠB IB REF 5 Ġreport Ġbetter Ġresults Ġwith Ġa Ġmodel Ġthat Ġwas Ġpret rained Ġfrom Ġscratch Ġwith Ġa Ġcustom Ġword piece Ġvocabulary . ĠIn Ġthis Ġpaper , Ġwe Ġfocus Ġon ĠBio BER T , Ġas Ġits Ġdomain Ġadaptation Ġcorpor a Ġare Ġpublicly Ġavailable . </s> Related Ġwork Ġ: :: ĠWord Ġvectors </s> Word Ġvectors Ġare Ġdistributed Ġrepresentations Ġof Ġwords Ġthat Ġare Ġtrained Ġon Ġunl abel ed Ġtext . ĠContrary Ġto ĠP TL Ms , Ġword Ġvectors Ġare Ġnon - context ual , Ġi . e ., Ġa Ġword Ġtype Ġis Ġalways Ġassigned Ġthe Ġsame Ġvector , Ġregardless Ġof Ġcontext . ĠIn Ġthis Ġpaper , Ġwe Ġuse ĠWord 2 V ec ĠB IB REF 3 Ġto Ġtrain Ġword Ġvectors . ĠWe Ġwill Ġdenote Ġthe ĠWord 2 V ec Ġlookup Ġfunction Ġas Ġ$\ math cal Ġ{ E } _ \ math rm Ġ{ W 2 V } Ġ: Ġ\ math bb Ġ{ L } _ \ math rm Ġ{ W 2 V } Ġ\ right arrow Ġ\ math bb Ġ{ R } ^{ d _ \ math rm Ġ{ W 2 V }} $. </s> Related Ġwork Ġ: :: ĠWord Ġvector Ġspace Ġalignment </s> Word Ġvector Ġspace Ġalignment Ġhas Ġmost Ġfrequently Ġbeen Ġexplored Ġin Ġthe Ġcontext Ġof Ġcross - ling ual Ġword Ġembed d ings . ĠFor Ġinstance , ĠB IB REF 10 Ġalign ĠEnglish Ġand ĠSpanish ĠWord 2 V ec Ġspaces Ġby Ġa Ġsimple Ġlinear Ġtransformation . ĠB IB REF 11 Ġuse Ġa Ġrelated Ġmethod Ġto Ġalign Ġcross - ling ual Ġword Ġvectors Ġand Ġmult ilingual ĠB ERT Ġword piece Ġvectors . </s> Method </s> In Ġthe Ġfollowing , Ġwe Ġassume Ġaccess Ġto Ġa Ġgeneral - domain ĠP TL M , Ġas Ġdescribed Ġin ĠSection ĠSEC REF 2 , Ġand Ġa Ġcorpus Ġof Ġunl abel ed Ġtarget - domain Ġtext . </s> Method Ġ: :: ĠCreating Ġnew Ġinput Ġvectors </s> In Ġa Ġfirst Ġstep , Ġwe Ġtrain ĠWord 2 V ec Ġon Ġthe Ġtarget - domain Ġcorpus . ĠIn Ġa Ġsecond Ġstep , Ġwe Ġtake Ġthe Ġintersection Ġof Ġ$\ math bb Ġ{ L } _ \ math rm Ġ{ LM } $ Ġand Ġ$\ math bb Ġ{ L } _ \ math rm Ġ{ W 2 V } $. ĠIn Ġpractice , Ġthe Ġintersection Ġmostly Ġcontains Ġword pieces Ġfrom Ġ$\ math bb Ġ{ L } _ \ math rm Ġ{ LM } $ Ġthat Ġcorrespond Ġto Ġstandalone Ġwords . ĠIt Ġalso Ġcontains Ġsingle Ġcharacters Ġand Ġother Ġnoise , Ġhowever , Ġwe Ġfound Ġthat Ġfiltering Ġthem Ġdoes Ġnot Ġimprove Ġalignment Ġquality . ĠIn Ġa Ġthird Ġstep , Ġwe Ġuse Ġthe Ġintersection Ġto Ġfit Ġan Ġuncon str ained Ġlinear Ġtransformation Ġ$\ math bf Ġ{ W } Ġ\ in Ġ\ math bb Ġ{ R } ^{ d _ \ math rm Ġ{ LM } Ġ\ times Ġd _ \ math rm Ġ{ W 2 V }} $ Ġvia Ġleast Ġsquares : </s> Int uitive ly , Ġ$\ math bf Ġ{ W } $ Ġmakes ĠWord 2 V ec Ġvectors ĠâĢ ľ look Ġlike âĢ Ŀ Ġthe ĠP TL M 's Ġnative Ġword piece Ġvectors , Ġjust Ġlike Ġcross - ling ual Ġalignment Ġmakes Ġword Ġvectors Ġfrom Ġone Ġlanguage ĠâĢ ľ look Ġlike âĢ Ŀ Ġword Ġvectors Ġfrom Ġanother Ġlanguage . ĠIn ĠTable ĠT AB REF 7 Ġ( top ), Ġwe Ġshow Ġexamples Ġof Ġwithin - space Ġand Ġcross - space Ġnearest Ġneighbors Ġafter Ġalignment . </s> Method Ġ: :: ĠUp dating Ġthe Ġword piece Ġembed ding Ġlayer </s> Next , Ġwe Ġredef ine Ġthe Ġword piece Ġembed ding Ġlayer Ġof Ġthe ĠP TL M . ĠThe Ġmost Ġradical Ġstrategy Ġwould Ġbe Ġto Ġreplace Ġthe Ġentire Ġlayer Ġwith Ġthe Ġaligned ĠWord 2 V ec Ġvectors : </s> In Ġinitial Ġexperiments , Ġthis Ġstrategy Ġled Ġto Ġa Ġdrop Ġin Ġperformance , Ġpresumably Ġbecause Ġfunction Ġwords Ġare Ġnot Ġwell Ġrepresented Ġby ĠWord 2 V ec , Ġand Ġreplacing Ġthem Ġdisrupt s ĠB ERT 's Ġsynt actic Ġabilities . ĠTo Ġprevent Ġthis Ġproblem , Ġwe Ġleave Ġexisting Ġword piece Ġvectors Ġintact Ġand Ġonly Ġadd Ġnew Ġones : </s> Method Ġ: :: ĠUp dating Ġthe Ġtoken izer </s> In Ġa Ġfinal Ġstep , Ġwe Ġupdate Ġthe Ġtoken izer Ġto Ġaccount Ġfor Ġthe Ġadded Ġwords . ĠLet Ġ$\ math cal Ġ{ T } _ \ math rm Ġ{ LM } $ Ġbe Ġthe Ġstandard ĠB ERT Ġtoken izer , Ġand Ġlet Ġ$\ hat {\ math cal Ġ{ T }} _ \ math rm Ġ{ LM } $ Ġbe Ġthe Ġtoken izer Ġthat Ġtreats Ġall Ġwords Ġin Ġ$\ math bb Ġ{ L } _ \ math rm Ġ{ LM } Ġ\ cup Ġ\ math bb Ġ{ L } _ \ math rm Ġ{ W 2 V } $ Ġas Ġone - word piece Ġtokens , Ġwhile Ġtoken izing Ġany Ġother Ġwords Ġas Ġusual . </s> In Ġpractice , Ġa Ġgiven Ġword Ġmay Ġor Ġmay Ġnot Ġbenefit Ġfrom Ġbeing Ġtoken ized Ġby Ġ$\ hat {\ math cal Ġ{ T }} _ \ math rm Ġ{ LM } $ Ġinstead Ġof Ġ$\ math cal Ġ{ T } _ \ math rm Ġ{ LM } $. ĠTo Ġgive Ġa Ġconcrete Ġexample , Ġ82 % Ġof Ġthe Ġwords Ġin Ġthe ĠBC 5 C DR ĠN ER Ġdataset Ġthat Ġend Ġin Ġthe Ġsuffix Ġ- ia Ġare Ġinside Ġa Ġdisease Ġentity Ġ( e . g ., Ġt achy card ia ). Ġ$\ math cal Ġ{ T } _ \ math rm Ġ{ LM } $ Ġtoken izes Ġthis Ġword Ġas Ġta Ġ## chy Ġ## card Ġ## ia , Ġthereby Ġexposing Ġthe Ġorth ographic Ġcue Ġto Ġthe Ġmodel . ĠAs Ġa Ġresult , Ġ$\ math cal Ġ{ T } _ \ math rm Ġ{ LM } $ Ġleads Ġto Ġhigher Ġrecall Ġon Ġ- ia Ġdiseases . ĠBut Ġthere Ġare Ġmany Ġcases Ġwhere Ġword piece Ġtoken ization Ġis Ġmeaningless Ġor Ġmisleading . ĠFor Ġinstance Ġeuth ym ia Ġ( not Ġa Ġdisease ) Ġis Ġtoken ized Ġby Ġ$\ math cal Ġ{ T } _ \ math rm Ġ{ LM } $ Ġas Ġe Ġ## uth Ġ## ym Ġ## ia , Ġmaking Ġit Ġlikely Ġto Ġbe Ġclassified Ġas Ġa Ġdisease . ĠBy Ġcontrast , Ġ$\ hat {\ math cal Ġ{ T }} _ \ math rm Ġ{ LM } $ Ġgives Ġeuth ym ia Ġa Ġone - word piece Ġrepresentation Ġthat Ġdepends Ġonly Ġon Ġdistribution al Ġsemantics . ĠWe Ġfind Ġthat Ġusing Ġ$\ hat {\ math cal Ġ{ T }} _ \ math rm Ġ{ LM } $ Ġimproves Ġprecision Ġon Ġ- ia Ġdiseases . </s> To Ġcombine Ġthese Ġcomplementary Ġstrengths , Ġwe Ġuse Ġa Ġ50 / 50 Ġmixture Ġof Ġ$\ math cal Ġ{ T } _ \ math rm Ġ{ LM } $ - token ization Ġand Ġ$\ hat {\ math cal Ġ{ T }} _ \ math rm Ġ{ LM } $ - token ization Ġwhen Ġfin et uning Ġthe ĠP TL M Ġon Ġa Ġtask . ĠAt Ġtest Ġtime , Ġwe Ġuse Ġboth Ġtoken izers Ġand Ġmean - pool Ġthe Ġoutputs . ĠLet Ġ$ o (\ math cal Ġ{ T } ( S )) $ Ġbe Ġsome Ġoutput Ġof Ġinterest Ġ( e . g ., Ġa Ġlog it ), Ġgiven Ġsentence Ġ$ S $ Ġtoken ized Ġby Ġ$\ math cal Ġ{ T } $. ĠWe Ġpredict : </s> Exper iment Ġ1 : ĠBi omedical ĠN ER </s> In Ġthis Ġsection , Ġwe Ġuse Ġthe Ġproposed Ġmethod Ġto Ġcreate ĠGreen Bio BER T , Ġan Ġinexpensive Ġand Ġenvironmentally Ġfriendly Ġalternative Ġto ĠBio BER T . ĠRecall Ġthat ĠBio BER T v 1 . 0 Ġ( b io bert _ v 1 . 0 _ pub med _ pm c ) Ġwas Ġinitialized Ġfrom Ġgeneral - domain ĠB ERT Ġ( bert - base - c ased ) Ġand Ġpret rained Ġon ĠPubMed + PM C . </s> Exper iment Ġ1 : ĠBi omedical ĠN ER Ġ: :: ĠDomain Ġadaptation </s> We Ġtrain ĠWord 2 V ec Ġwith Ġvector Ġsize Ġ$ d _ \ math rm Ġ{ W 2 V } Ġ= Ġd _ \ math rm Ġ{ LM } Ġ= Ġ768 $ Ġon ĠPubMed + PM C Ġ( see ĠAppendix Ġfor Ġdetails ). ĠThen , Ġwe Ġfollow Ġthe Ġprocedure Ġdescribed Ġin ĠSection ĠSEC REF 3 Ġto Ġupdate Ġthe Ġword piece Ġembed ding Ġlayer Ġand Ġtoken izer Ġof Ġgeneral - domain ĠB ERT . </s> Exper iment Ġ1 : ĠBi omedical ĠN ER Ġ: :: ĠFin et uning </s> We Ġfin et une ĠGreen Bio BER T Ġon Ġthe Ġeight Ġpublicly Ġavailable ĠN ER Ġtasks Ġused Ġin ĠB IB REF 2 . ĠWe Ġalso Ġdo Ġreproduction Ġexperiments Ġwith Ġgeneral - domain ĠB ERT Ġand ĠBio BER T v 1 . 0 , Ġusing Ġthe Ġsame Ġsetup Ġas Ġour Ġmodel . ĠWe Ġaverage Ġresults Ġover Ġeight Ġrandom Ġseeds . ĠSee ĠAppendix Ġfor Ġdetails Ġon Ġpre processing , Ġtraining Ġand Ġhyper param eters . </s> Exper iment Ġ1 : ĠBi omedical ĠN ER Ġ: :: ĠResults Ġand Ġdiscussion </s> Table ĠT AB REF 7 Ġ( bottom ) Ġshows Ġentity - level Ġprecision , Ġrecall Ġand ĠF 1 . ĠFor Ġease Ġof Ġvisualization , ĠFigure ĠFIG REF 13 Ġshows Ġwhat Ġportion Ġof Ġthe ĠBio BER T ĠâĢĵ ĠB ERT ĠF 1 Ġdelta Ġis Ġcovered . ĠWe Ġimprove Ġover Ġgeneral - domain ĠB ERT Ġon Ġall Ġtasks Ġwith Ġvarying Ġeffect Ġsizes . ĠDepending Ġon Ġthe Ġpoints Ġof Ġreference , Ġwe Ġcover Ġan Ġaverage Ġ52 % Ġto Ġ60 % Ġof Ġthe ĠBio BER T ĠâĢĵ ĠB ERT ĠF 1 Ġdelta Ġ( 54 % Ġfor ĠBio BER T v 1 . 0 , Ġ60 % Ġfor ĠBio BER T v 1 . 1 Ġand Ġ52 % Ġfor Ġour Ġreproduction Ġexperiments ). ĠTable ĠT AB REF 17 Ġ( top ) Ġshows Ġthe Ġimportance Ġof Ġvector Ġspace Ġalignment : ĠIf Ġwe Ġreplace Ġthe Ġaligned ĠWord 2 V ec Ġvectors Ġwith Ġtheir Ġnon - aligned Ġcounterparts Ġ( by Ġsetting Ġ$\ math bf Ġ{ W } Ġ= Ġ\ math bf Ġ{ 1 } $ ) Ġor Ġwith Ġrandomly Ġinitialized Ġvectors , ĠF 1 Ġdrops Ġon Ġall Ġtasks . </s> Exper iment Ġ2 : ĠCov id - 19 ĠQ A </s> In Ġthis Ġsection , Ġwe Ġuse Ġthe Ġproposed Ġmethod Ġto Ġquickly Ġadapt Ġan Ġexisting Ġgeneral - domain ĠQ A Ġmodel Ġto Ġan Ġemerging Ġtarget Ġdomain : ĠCov id - 19 . ĠOur Ġbaseline Ġmodel Ġis ĠS Qu AD BER T Ġ( bert - large - unc ased - wh ole - word - mask ing - fin et uned - squ ad ), Ġa Ġversion Ġof ĠB ERT Ġthat Ġwas Ġfin et uned Ġon Ġgeneral - domain ĠS Qu AD ĠB IB REF 19 . ĠWe Ġevaluate Ġon ĠDeep set - AI ĠCov id - Q A , Ġa ĠS Qu AD - style Ġdataset Ġwith Ġ13 80 Ġquestions Ġ( see ĠAppendix Ġfor Ġdetails Ġon Ġdata Ġand Ġpre processing ). ĠWe Ġassume Ġthat Ġthere Ġis Ġno Ġtarget - domain Ġfin et uning Ġdata , Ġwhich Ġis Ġa Ġrealistic Ġsetup Ġfor Ġa Ġnew Ġdomain . </s> Exper iment Ġ2 : ĠCov id - 19 ĠQ A Ġ: :: ĠDomain Ġadaptation </s> We Ġtrain ĠWord 2 V ec Ġwith Ġvector Ġsize Ġ$ d _ \ math rm Ġ{ W 2 V } Ġ= Ġd _ \ math rm Ġ{ LM } Ġ= Ġ1024 $ Ġon ĠC ORD - 19 Ġ( C ov id - 19 ĠOpen ĠResearch ĠDat as et ) Ġand / or ĠPubMed + PM C . ĠThe Ġprocess Ġtakes Ġless Ġthan Ġan Ġhour Ġon ĠC ORD - 19 Ġand Ġabout Ġone Ġday Ġon Ġthe Ġcombined Ġcorpus , Ġagain Ġwithout Ġthe Ġneed Ġfor Ġa ĠGPU . ĠThen , Ġwe Ġupdate ĠS Qu AD BER T 's Ġword piece Ġembed ding Ġlayer Ġand Ġtoken izer , Ġas Ġdescribed Ġin ĠSection ĠSEC REF 3 . ĠWe Ġrefer Ġto Ġthe Ġresulting Ġmodel Ġas ĠGreen C ov id S Qu AD BER T . </s> Exper iment Ġ2 : ĠCov id - 19 ĠQ A Ġ: :: ĠResults Ġand Ġdiscussion </s> Table ĠT AB REF 17 Ġ( bottom ) Ġshows Ġthat ĠGreen C ov id S Qu AD BER T Ġoutper forms Ġgeneral - domain ĠS Qu AD BER T Ġin Ġall Ġmetrics . ĠMost Ġof Ġthe Ġimprovement Ġcan Ġbe Ġachieved Ġwith Ġjust Ġthe Ġsmall ĠC ORD - 19 Ġcorpus , Ġwhich Ġis Ġmore Ġspecific Ġto Ġthe Ġtarget Ġdomain Ġ( comp are ĠâĢ ľ C ord - 19 Ġonly âĢ Ŀ Ġand ĠâĢ ľ C ord - 19 + Pub Med + PM C âĢ Ŀ ). </s> Conclusion </s> As Ġa Ġreaction Ġto Ġthe Ġtrend Ġtowards Ġhigh - resource Ġmodels , Ġwe Ġhave Ġproposed Ġan Ġinexpensive , ĠCPU - only Ġmethod Ġfor Ġdomain - adapt ing ĠPret rained ĠLanguage ĠModels : ĠWe Ġtrain ĠWord 2 V ec Ġvectors Ġon Ġtarget - domain Ġdata Ġand Ġalign Ġthem Ġwith Ġthe Ġword piece Ġvector Ġspace Ġof Ġa Ġgeneral - domain ĠP TL M . </s> On Ġeight Ġbiomedical ĠN ER Ġtasks , Ġwe Ġcover Ġover Ġ50 % Ġof Ġthe ĠBio BER T ĠâĢĵ ĠB ERT ĠF 1 Ġdelta , Ġat Ġ5 % Ġof ĠBio BER T 's Ġdomain Ġadaptation ĠCO $ _ 2 $ Ġfootprint Ġand Ġ2 % Ġof Ġits Ġcloud Ġcompute Ġcost . ĠWe Ġhave Ġalso Ġshown Ġhow Ġto Ġrapidly Ġadapt Ġan Ġexisting ĠB ERT ĠQ A Ġmodel Ġto Ġan Ġemerging Ġdomain ĠâĢĵ Ġthe ĠCov id - 19 Ġpand emic ĠâĢĵ Ġwithout Ġthe Ġneed Ġfor Ġtarget - domain ĠLanguage ĠModel Ġpret raining Ġor Ġfin et uning . </s> We Ġhope Ġthat Ġour Ġapproach Ġwill Ġbenefit Ġpractitioners Ġwith Ġlimited Ġtime Ġor Ġresources , Ġand Ġthat Ġit Ġwill Ġencourage Ġenvironmentally Ġfriend lier ĠN LP . </s> I nex p ensive ĠDomain ĠAdapt ation Ġof ĠPret rained ĠLanguage ĠModels Ġ( App endix ) Ġ: :: ĠWord 2 V ec Ġtraining </s> We Ġdownloaded Ġthe ĠPubMed , ĠPM C Ġand ĠC ORD - 19 Ġcorpor a Ġfrom : </s> https :// ft p . nc bi . nl m . nih . gov / pub / pm c / oa _ b ulk / Ġ[ 20 ĠJanuary Ġ2020 , Ġ68 GB Ġraw Ġtext ] </s> https :// ft p . nc bi . nl m . nih . gov / pub med / bas eline / Ġ[ 20 ĠJanuary Ġ2020 , Ġ24 GB Ġraw Ġtext ] </s> https :// pages . sem antics ch olar . org / cor on av irus - research Ġ[ 17 ĠApril Ġ2020 , Ġ2 GB Ġraw Ġtext ] </s> We Ġextract Ġall Ġabstract s Ġand Ġtext Ġbodies Ġand Ġapply Ġthe ĠB ERT Ġbasic Ġtoken izer Ġ( a Ġword Ġtoken izer Ġthat Ġstandard ĠB ERT Ġuses Ġbefore Ġword piece Ġtoken ization ). ĠThen , Ġwe Ġtrain ĠCB OW ĠWord 2 V ec Ġwith Ġnegative Ġsampling . ĠWe Ġuse Ġdefault Ġparameters Ġexcept Ġfor Ġthe Ġvector Ġsize Ġ( which Ġwe Ġset Ġto Ġ$ d _ \ math rm Ġ{ W 2 V } Ġ= Ġd _ \ math rm Ġ{ LM } $ ). </s> I nex p ensive ĠDomain ĠAdapt ation Ġof ĠPret rained ĠLanguage ĠModels Ġ( App endix ) Ġ: :: ĠExperiment Ġ1 : ĠBi omedical ĠN ER Ġ: :: ĠPret rained Ġmodels </s> General - domain ĠB ERT Ġand ĠBio BER T v 1 . 0 Ġwere Ġdownloaded Ġfrom : </s> https :// storage . google ap is . com / bert _ models / 2018 _ 10 _ 18 / c ased _ L - 12 _ H - 768 _ A - 12 . zip </s> https :// github . com / na ver / b io bert - pret rained </s> I nex p ensive ĠDomain ĠAdapt ation Ġof ĠPret rained ĠLanguage ĠModels Ġ( App endix ) Ġ: :: ĠExperiment Ġ1 : ĠBi omedical ĠN ER Ġ: :: ĠData </s> We Ġdownloaded Ġthe ĠN ER Ġdatasets Ġby Ġfollowing Ġinstructions Ġon Ġhttps :// github . com / d mis - lab / b io bert # Dat as ets . ĠFor Ġdetailed Ġdataset Ġstatistics , Ġsee ĠB IB REF 2 . </s> I nex p ensive ĠDomain ĠAdapt ation Ġof ĠPret rained ĠLanguage ĠModels Ġ( App endix ) Ġ: :: ĠExperiment Ġ1 : ĠBi omedical ĠN ER Ġ: :: ĠPre processing </s> We Ġcut Ġall Ġsentences Ġinto Ġchunks Ġof Ġ30 Ġor Ġfewer Ġwhites pace - token ized Ġwords Ġ( without Ġsplitting Ġinside Ġlabeled Ġspans ). ĠThen , Ġwe Ġtoken ize Ġevery Ġchunk Ġ$ S $ Ġwith Ġ$\ math cal Ġ{ T } Ġ= Ġ\ math cal Ġ{ T } _ \ math rm Ġ{ LM } $ Ġor Ġ$\ math cal Ġ{ T } Ġ= Ġ\ hat {\ math cal Ġ{ T }} _ \ math rm Ġ{ LM } $ Ġand Ġadd Ġspecial Ġtokens : </s> Word - initial Ġword pieces Ġin Ġ$\ math cal Ġ{ T } ( S ) $ Ġare Ġlabeled Ġas ĠB ( eg in ), ĠI ( n side ) Ġor ĠO ( ut side ), Ġwhile Ġnon - word - initial Ġword pieces Ġare Ġlabeled Ġas ĠX ( ignore ). </s> I nex p ensive ĠDomain ĠAdapt ation Ġof ĠPret rained ĠLanguage ĠModels Ġ( App endix ) Ġ: :: ĠExperiment Ġ1 : ĠBi omedical ĠN ER Ġ: :: ĠModel ing , Ġtraining Ġand Ġinference </s> We Ġfollow ĠB IB REF 2 's Ġimplementation Ġ( https :// github . com / d mis - lab / b io bert ): ĠWe Ġadd Ġa Ġrandomly Ġinitialized Ġsoft max Ġclass ifier Ġon Ġtop Ġof Ġthe Ġlast ĠB ERT Ġlayer Ġto Ġpredict Ġthe Ġlabels . ĠWe Ġfin et une Ġthe Ġentire Ġmodel Ġto Ġminimize Ġnegative Ġlog Ġlikelihood , Ġwith Ġthe Ġstandard ĠAdam Ġoptim izer ĠB IB REF 20 Ġand Ġa Ġlinear Ġlearning Ġrate Ġsched uler Ġ( 10 % Ġwarm up ). ĠLike ĠB IB REF 2 , Ġwe Ġfin et une Ġon Ġthe Ġconc aten ation Ġof Ġthe Ġtraining Ġand Ġdevelopment Ġset . ĠAll Ġfin et uning Ġruns Ġwere Ġdone Ġon Ġa ĠGeForce ĠTitan ĠX ĠGPU Ġ( 12 GB ). </s> Since Ġwe Ġdo Ġnot Ġhave Ġthe Ġresources Ġfor Ġan Ġextensive Ġhyper param eter Ġsearch , Ġwe Ġuse Ġdefaults Ġand Ġrecommendations Ġfrom Ġthe ĠBio BER T Ġrepository : ĠB atch Ġsize Ġof Ġ32 , Ġpeak Ġlearning Ġrate Ġof Ġ$ 1 Ġ\ cd ot Ġ10 ^{ - 5 } $, Ġand Ġ100 Ġepoch s . </s> At Ġinference Ġtime , Ġwe Ġgather Ġthe Ġoutput Ġlog its Ġof Ġword - initial Ġword pieces Ġonly . ĠSince Ġthe Ġnumber Ġof Ġword - initial Ġword pieces Ġis Ġthe Ġsame Ġfor Ġ$\ math cal Ġ{ T } _ \ math rm Ġ{ LM } ( S ) $ Ġand Ġ$\ hat {\ math cal Ġ{ T }} _ \ math rm Ġ{ LM } ( S ) $, Ġthis Ġmakes Ġmean - pool ing Ġthe Ġlog its Ġstraightforward . </s> I nex p ensive ĠDomain ĠAdapt ation Ġof ĠPret rained ĠLanguage ĠModels Ġ( App endix ) Ġ: :: ĠExperiment Ġ1 : ĠBi omedical ĠN ER Ġ: :: ĠNote Ġon Ġour Ġreproduction Ġexperiments </s> We Ġfound Ġit Ġeasier Ġto Ġreproduce Ġor Ġexceed ĠB IB REF 2 's Ġresults Ġfor Ġgeneral - domain ĠB ERT , Ġcompared Ġto Ġtheir Ġresults Ġfor ĠBio BER T v 1 . 0 Ġ( see ĠFigure ĠFIG REF 13 , Ġmain Ġpaper ). ĠWhile Ġthis Ġmay Ġbe Ġdue Ġto Ġhyper param eters , Ġit Ġsuggests Ġthat ĠBio BER T v 1 . 0 Ġwas Ġmore Ġstrongly Ġtuned Ġthan ĠB ERT Ġin Ġthe Ġoriginal ĠBio BER T Ġpaper . ĠThis Ġobservation Ġdoes Ġnot Ġaffect Ġour Ġconclusions , Ġas ĠGreen Bio BER T Ġperforms Ġbetter Ġthan Ġreproduced ĠB ERT Ġas Ġwell . </s> I nex p ensive ĠDomain ĠAdapt ation Ġof ĠPret rained ĠLanguage ĠModels Ġ( App endix ) Ġ: :: ĠExperiment Ġ2 : ĠCov id - 19 ĠQ A Ġ: :: ĠPret rained Ġmodel </s> We Ġdownloaded Ġthe ĠS Qu AD BER T Ġbaseline Ġfrom : </s> https :// h ug ging face . co / bert - large - unc ased - wh ole - word - mask ing - fin et uned - squ ad </s> I nex p ensive ĠDomain ĠAdapt ation Ġof ĠPret rained ĠLanguage ĠModels Ġ( App endix ) Ġ: :: ĠExperiment Ġ2 : ĠCov id - 19 ĠQ A Ġ: :: ĠData </s> We Ġdownloaded Ġthe ĠDeep set - AI ĠCov id - Q A Ġdataset Ġfrom : </s> https :// github . com / deep set - ai / CO VID - Q A / bl ob / master / data / question - ans w ering / 2004 23 _ c ov id Q A . json Ġ[ 24 ĠApril Ġ2020 ] </s> At Ġthe Ġtime Ġof Ġwriting , Ġthe Ġdataset Ġcontains Ġ13 80 Ġquestions Ġand Ġgold Ġanswer Ġspans . ĠEvery Ġquestion Ġis Ġassociated Ġwith Ġone Ġof Ġ98 Ġresearch Ġpapers Ġ( context s ). ĠWe Ġtreat Ġthe Ġentire Ġdataset Ġas Ġa Ġtest Ġset . </s> Note Ġthat Ġthere Ġare Ġsome Ġimportant Ġdifferences Ġbetween Ġthe Ġdataset Ġand ĠS Qu AD , Ġwhich Ġmake Ġthe Ġtask Ġchallenging : </s> The Ġcontexts Ġare Ġfull Ġdocuments Ġrather Ġthan Ġsingle Ġparagraphs . ĠThus , Ġthe Ġcorrect Ġanswer Ġmay Ġappear Ġseveral Ġtimes , Ġoften Ġwith Ġslightly Ġdifferent Ġword ings . ĠOnly Ġa Ġsingle Ġone Ġof Ġthe Ġoccurrences Ġis Ġannot ated Ġas Ġcorrect , Ġe . g .: </s> What Ġwas Ġthe Ġprevalence Ġof ĠCor on av irus ĠOC 43 Ġin Ġcommunity Ġsamples Ġin ĠI lor in , ĠNigeria ? </s> 13 . 3 % Ġ( 95 % ĠCI Ġ6 . 9 - 23 . 6 %) Ġ# Ġfrom Ġmain Ġtext </s> ( 13 . 3 %, Ġ10 / 75 ). Ġ# Ġfrom Ġabstract </s> S Qu AD Ġgold Ġanswers Ġare Ġdefined Ġas Ġthe ĠâĢ ľ short est Ġspan Ġin Ġthe Ġparagraph Ġthat Ġanswered Ġthe Ġquestion âĢ Ŀ ĠB IB REF 19 , Ġbut Ġmany ĠCov id - Q A Ġgold Ġanswers Ġare Ġlonger Ġand Ġcontain Ġnon - essential Ġcontext , Ġe . g .: </s> When Ġwas Ġthe ĠMiddle ĠEast ĠRes pir atory ĠSyndrome ĠCor on av irus Ġisolated Ġfirst ? </s> ( M ERS - Co V ) Ġwas Ġfirst Ġisolated Ġin Ġ2012 , Ġin Ġa Ġ60 - year - old Ġman Ġwho Ġdied Ġin ĠJ edd ah , ĠK SA Ġdue Ġto Ġsevere Ġacute Ġpneumonia Ġand Ġmultiple Ġorgan Ġfailure </s> 2012 , </s> I nex p ensive ĠDomain ĠAdapt ation Ġof ĠPret rained ĠLanguage ĠModels Ġ( App endix ) Ġ: :: ĠExperiment Ġ2 : ĠCov id - 19 ĠQ A Ġ: :: ĠPre processing </s> We Ġtoken ize Ġevery Ġquestion - context Ġpair Ġ$( Q , ĠC ) $ Ġwith Ġ$\ math cal Ġ{ T } Ġ= Ġ\ math cal Ġ{ T } _ \ math rm Ġ{ LM } $ Ġor Ġ$\ math cal Ġ{ T } Ġ= Ġ\ hat {\ math cal Ġ{ T }} _ \ math rm Ġ{ LM } $, Ġwhich Ġyields Ġ$( \ math cal Ġ{ T } ( Q ), Ġ\ math cal Ġ{ T } ( C )) $. ĠSince Ġ$\ math cal Ġ{ T } ( C ) $ Ġis Ġusually Ġtoo Ġlong Ġto Ġbe Ġdig ested Ġin Ġa Ġsingle Ġforward Ġpass , Ġwe Ġdefine Ġa Ġsliding Ġwindow Ġwith Ġwidth Ġand Ġstride Ġ$ N Ġ= Ġ\ math rm Ġ{ floor } (\ frac { 509 Ġ- Ġ| \ math cal Ġ{ T } ( Q )| }{ 2 }) $. ĠAt Ġstep Ġ$ n $, Ġthe ĠâĢ ľ active âĢ Ŀ Ġwindow Ġis Ġbetween Ġ$ a ^{ ( l )} _ n Ġ= Ġn N $ Ġand Ġ$ a ^{ ( r )} _ n Ġ= Ġ\ math rm Ġ{ min } ( | C | , Ġn N + N ) $. ĠThe Ġinput Ġis Ġdefined Ġas : </s> $ p ^{ ( l )} _ n $ Ġand Ġ$ p ^{ ( r )} _ n $ Ġare Ġchosen Ġsuch Ġthat Ġ$ | X ^{ ( n )} | Ġ= Ġ512 $, Ġand Ġsuch Ġthat Ġthe Ġactive Ġwindow Ġis Ġin Ġthe Ġcenter Ġof Ġthe Ġinput Ġ( if Ġpossible ). </s> I nex p ensive ĠDomain ĠAdapt ation Ġof ĠPret rained ĠLanguage ĠModels Ġ( App endix ) Ġ: :: ĠExperiment Ġ2 : ĠCov id - 19 ĠQ A Ġ: :: ĠModel ing Ġand Ġinference </s> Feed ing Ġ$ X ^{ ( n )} $ Ġinto Ġthe Ġpret rained ĠQ A Ġmodel Ġyields Ġstart Ġlog its Ġ$\ math bf Ġ{ h ^ {\ prime Ġ}} ^{ (\ math rm Ġ{ start }, Ġn )} Ġ\ in Ġ\ math bb Ġ{ R } ^{ | X ^{ ( n )} | } $ Ġand Ġend Ġlog its Ġ$\ math bf Ġ{ h ^ {\ prime Ġ}} ^{ (\ math rm Ġ{ end }, n )} Ġ\ in Ġ\ math bb Ġ{ R } ^{ | X ^{ ( n )} | } $. ĠWe Ġextract Ġand Ġconc aten ate Ġthe Ġslices Ġthat Ġcorrespond Ġto Ġthe Ġactive Ġwindows Ġof Ġall Ġsteps : </s> Next , Ġwe Ġmap Ġthe Ġlog its Ġfrom Ġthe Ġword piece Ġlevel Ġto Ġthe Ġword Ġlevel . ĠThis Ġallows Ġus Ġto Ġmean - pool Ġthe Ġoutputs Ġof Ġ$\ math cal Ġ{ T } _ \ math rm Ġ{ LM } $ Ġand Ġ$\ hat {\ math cal Ġ{ T }} _ \ math rm Ġ{ LM } $ Ġeven Ġwhen Ġ$ | \ math cal Ġ{ T } _ \ math rm Ġ{ LM } ( C )| Ġ\ ne Ġ| \ hat {\ math cal Ġ{ T }} _ \ math rm Ġ{ LM } ( C )| $. </s> Let Ġ$ c _ i $ Ġbe Ġa Ġwhites pace - del im ited Ġword Ġin Ġ$ C $. ĠLet Ġ$\ math cal Ġ{ T } ( C ) _{ j : j + | \ math cal Ġ{ T } ( c _ i )| } $ Ġbe Ġthe Ġcorresponding Ġword pieces . ĠThe Ġstart Ġand Ġend Ġlog its Ġof Ġ$ c _ i $ Ġare Ġderived Ġas : </s> Finally , Ġwe Ġreturn Ġthe Ġanswer Ġspan Ġ$ C _{ k : k ^ {\ prime Ġ}} $ Ġthat Ġmaxim izes Ġ$ o ^{ (\ math rm Ġ{ start }) } _ k Ġ+ Ġo ^{ (\ math rm Ġ{ end }) } _{ k ^ {\ prime Ġ}} $, Ġsubject Ġto Ġthe Ġconstraints Ġthat Ġ$ k ^ {\ prime Ġ} $ Ġdoes Ġnot Ġpreced e Ġ$ k $ Ġand Ġthe Ġanswer Ġspan Ġis Ġnot Ġlonger Ġthan Ġ500 Ġcharacters .
Introduction Pretrained Language Models (PTLMs) such as BERT BIBREF1 have spearheaded advances on many NLP tasks. Usually, PTLMs are pretrained on unlabeled general-domain and/or mixed-domain text, such as Wikipedia, digital books or the Common Crawl corpus. When applying PTLMs to specific domains, it can be useful to domain-adapt them. Domain adaptation of PTLMs has typically been achieved by pretraining on target-domain text. One such model is BioBERT BIBREF2, which was initialized from general-domain BERT and then pretrained on biomedical scientific publications. The domain adaptation is shown to be helpful for target-domain tasks such as biomedical Named Entity Recognition (NER) or Question Answering (QA). On the downside, the computational cost of pretraining can be considerable: BioBERTv1.0 was adapted for ten days on eight large GPUs (see Table TABREF1), which is expensive, environmentally unfriendly, prohibitive for small research labs and students, and may delay prototyping on emerging domains. We therefore propose a fast, CPU-only domain-adaptation method for PTLMs: We train Word2Vec BIBREF3 on target-domain text and align the resulting word vectors with the wordpiece vectors of an existing general-domain PTLM. The PTLM thus gains domain-specific lexical knowledge in the form of additional word vectors, but its deeper layers remain unchanged. Since Word2Vec and the vector space alignment are efficient models, the process requires a fraction of the resources associated with pretraining the PTLM itself, and it can be done on CPU. In Section SECREF4, we use the proposed method to domain-adapt BERT on PubMed+PMC (the data used for BioBERTv1.0) and/or CORD-19 (Covid-19 Open Research Dataset). We improve over general-domain BERT on eight out of eight biomedical NER tasks, using a fraction of the compute cost associated with BioBERT. In Section SECREF5, we show how to quickly adapt an existing Question Answering model to text about the Covid-19 pandemic, without any target-domain Language Model pretraining or finetuning. Related work ::: The BERT PTLM For our purpose, a PTLM consists of three parts: A tokenizer $\mathcal {T}_\mathrm {LM} : \mathbb {L}^+ \rightarrow \mathbb {L}_\mathrm {LM}^+$, a wordpiece embedding function $\mathcal {E}_\mathrm {LM}: \mathbb {L}_\mathrm {LM} \rightarrow \mathbb {R}^{d_\mathrm {LM}}$ and an encoder function $\mathcal {F}_\mathrm {LM}$. $\mathbb {L}_\mathrm {LM}$ is a limited vocabulary of wordpieces. All words that are not in $\mathbb {L}_\mathrm {LM}$ are tokenized into sequences of shorter wordpieces, e.g., tachycardia becomes ta ##chy ##card ##ia. Given a sentence $S = [w_1, \ldots , w_T]$, tokenized as $\mathcal {T}_\mathrm {LM}(S) = [\mathcal {T}_\mathrm {LM}(w_1); \ldots ; \mathcal {T}_\mathrm {LM}(w_T)]$, $\mathcal {E}_\mathrm {LM}$ embeds every wordpiece in $\mathcal {T}_\mathrm {LM}(S)$ into a real-valued, trainable wordpiece vector. The wordpiece vectors of the entire sequence are stacked and fed into $\mathcal {F}_\mathrm {LM}$. Note that we consider position and segment embeddings to be a part of $\mathcal {F}_\mathrm {LM}$ rather than $\mathcal {E}_\mathrm {LM}$. In the case of BERT, $\mathcal {F}_\mathrm {LM}$ is a Transformer BIBREF4, followed by a final Feed-Forward Net. During pretraining, the Feed-Forward Net predicts the identity of masked wordpieces. When finetuning on a supervised task, it is usually replaced with a randomly initialized task-specific layer. Related work ::: Domain-adapted PTLMs Domain adaptation of PTLMs is typically achieved by pretraining on unlabeled target-domain text. Some examples of such models are BioBERT BIBREF2, which was pretrained on the PubMed and/or PubMed Central (PMC) corpora, SciBERT BIBREF5, which was pretrained on papers from SemanticScholar, ClinicalBERT BIBREF6, BIBREF7 and ClinicalXLNet BIBREF8, which were pretrained on clinical patient notes, and AdaptaBERT BIBREF9, which was pretrained on Early Modern English text. In most cases, a domain-adapted PTLM is initialized from a general-domain PTLM (e.g., standard BERT), though BIBREF5 report better results with a model that was pretrained from scratch with a custom wordpiece vocabulary. In this paper, we focus on BioBERT, as its domain adaptation corpora are publicly available. Related work ::: Word vectors Word vectors are distributed representations of words that are trained on unlabeled text. Contrary to PTLMs, word vectors are non-contextual, i.e., a word type is always assigned the same vector, regardless of context. In this paper, we use Word2Vec BIBREF3 to train word vectors. We will denote the Word2Vec lookup function as $\mathcal {E}_\mathrm {W2V} : \mathbb {L}_\mathrm {W2V} \rightarrow \mathbb {R}^{d_\mathrm {W2V}}$. Related work ::: Word vector space alignment Word vector space alignment has most frequently been explored in the context of cross-lingual word embeddings. For instance, BIBREF10 align English and Spanish Word2Vec spaces by a simple linear transformation. BIBREF11 use a related method to align cross-lingual word vectors and multilingual BERT wordpiece vectors. Method In the following, we assume access to a general-domain PTLM, as described in Section SECREF2, and a corpus of unlabeled target-domain text. Method ::: Creating new input vectors In a first step, we train Word2Vec on the target-domain corpus. In a second step, we take the intersection of $\mathbb {L}_\mathrm {LM}$ and $\mathbb {L}_\mathrm {W2V}$. In practice, the intersection mostly contains wordpieces from $\mathbb {L}_\mathrm {LM}$ that correspond to standalone words. It also contains single characters and other noise, however, we found that filtering them does not improve alignment quality. In a third step, we use the intersection to fit an unconstrained linear transformation $\mathbf {W} \in \mathbb {R}^{d_\mathrm {LM} \times d_\mathrm {W2V}}$ via least squares: Intuitively, $\mathbf {W}$ makes Word2Vec vectors “look like” the PTLM's native wordpiece vectors, just like cross-lingual alignment makes word vectors from one language “look like” word vectors from another language. In Table TABREF7 (top), we show examples of within-space and cross-space nearest neighbors after alignment. Method ::: Updating the wordpiece embedding layer Next, we redefine the wordpiece embedding layer of the PTLM. The most radical strategy would be to replace the entire layer with the aligned Word2Vec vectors: In initial experiments, this strategy led to a drop in performance, presumably because function words are not well represented by Word2Vec, and replacing them disrupts BERT's syntactic abilities. To prevent this problem, we leave existing wordpiece vectors intact and only add new ones: Method ::: Updating the tokenizer In a final step, we update the tokenizer to account for the added words. Let $\mathcal {T}_\mathrm {LM}$ be the standard BERT tokenizer, and let $\hat{\mathcal {T}}_\mathrm {LM}$ be the tokenizer that treats all words in $\mathbb {L}_\mathrm {LM} \cup \mathbb {L}_\mathrm {W2V}$ as one-wordpiece tokens, while tokenizing any other words as usual. In practice, a given word may or may not benefit from being tokenized by $\hat{\mathcal {T}}_\mathrm {LM}$ instead of $\mathcal {T}_\mathrm {LM}$. To give a concrete example, 82% of the words in the BC5CDR NER dataset that end in the suffix -ia are inside a disease entity (e.g., tachycardia). $\mathcal {T}_\mathrm {LM}$ tokenizes this word as ta ##chy ##card ##ia, thereby exposing the orthographic cue to the model. As a result, $\mathcal {T}_\mathrm {LM}$ leads to higher recall on -ia diseases. But there are many cases where wordpiece tokenization is meaningless or misleading. For instance euthymia (not a disease) is tokenized by $\mathcal {T}_\mathrm {LM}$ as e ##uth ##ym ##ia, making it likely to be classified as a disease. By contrast, $\hat{\mathcal {T}}_\mathrm {LM}$ gives euthymia a one-wordpiece representation that depends only on distributional semantics. We find that using $\hat{\mathcal {T}}_\mathrm {LM}$ improves precision on -ia diseases. To combine these complementary strengths, we use a 50/50 mixture of $\mathcal {T}_\mathrm {LM}$-tokenization and $\hat{\mathcal {T}}_\mathrm {LM}$-tokenization when finetuning the PTLM on a task. At test time, we use both tokenizers and mean-pool the outputs. Let $o(\mathcal {T}(S))$ be some output of interest (e.g., a logit), given sentence $S$ tokenized by $\mathcal {T}$. We predict: Experiment 1: Biomedical NER In this section, we use the proposed method to create GreenBioBERT, an inexpensive and environmentally friendly alternative to BioBERT. Recall that BioBERTv1.0 (biobert_v1.0_pubmed_pmc) was initialized from general-domain BERT (bert-base-cased) and pretrained on PubMed+PMC. Experiment 1: Biomedical NER ::: Domain adaptation We train Word2Vec with vector size $d_\mathrm {W2V} = d_\mathrm {LM} = 768$ on PubMed+PMC (see Appendix for details). Then, we follow the procedure described in Section SECREF3 to update the wordpiece embedding layer and tokenizer of general-domain BERT. Experiment 1: Biomedical NER ::: Finetuning We finetune GreenBioBERT on the eight publicly available NER tasks used in BIBREF2. We also do reproduction experiments with general-domain BERT and BioBERTv1.0, using the same setup as our model. We average results over eight random seeds. See Appendix for details on preprocessing, training and hyperparameters. Experiment 1: Biomedical NER ::: Results and discussion Table TABREF7 (bottom) shows entity-level precision, recall and F1. For ease of visualization, Figure FIGREF13 shows what portion of the BioBERT – BERT F1 delta is covered. We improve over general-domain BERT on all tasks with varying effect sizes. Depending on the points of reference, we cover an average 52% to 60% of the BioBERT – BERT F1 delta (54% for BioBERTv1.0, 60% for BioBERTv1.1 and 52% for our reproduction experiments). Table TABREF17 (top) shows the importance of vector space alignment: If we replace the aligned Word2Vec vectors with their non-aligned counterparts (by setting $\mathbf {W} = \mathbf {1}$) or with randomly initialized vectors, F1 drops on all tasks. Experiment 2: Covid-19 QA In this section, we use the proposed method to quickly adapt an existing general-domain QA model to an emerging target domain: Covid-19. Our baseline model is SQuADBERT (bert-large-uncased-whole-word-masking-finetuned-squad), a version of BERT that was finetuned on general-domain SQuAD BIBREF19. We evaluate on Deepset-AI Covid-QA, a SQuAD-style dataset with 1380 questions (see Appendix for details on data and preprocessing). We assume that there is no target-domain finetuning data, which is a realistic setup for a new domain. Experiment 2: Covid-19 QA ::: Domain adaptation We train Word2Vec with vector size $d_\mathrm {W2V} = d_\mathrm {LM} = 1024$ on CORD-19 (Covid-19 Open Research Dataset) and/or PubMed+PMC. The process takes less than an hour on CORD-19 and about one day on the combined corpus, again without the need for a GPU. Then, we update SQuADBERT's wordpiece embedding layer and tokenizer, as described in Section SECREF3. We refer to the resulting model as GreenCovidSQuADBERT. Experiment 2: Covid-19 QA ::: Results and discussion Table TABREF17 (bottom) shows that GreenCovidSQuADBERT outperforms general-domain SQuADBERT in all metrics. Most of the improvement can be achieved with just the small CORD-19 corpus, which is more specific to the target domain (compare “Cord-19 only” and “Cord-19+PubMed+PMC”). Conclusion As a reaction to the trend towards high-resource models, we have proposed an inexpensive, CPU-only method for domain-adapting Pretrained Language Models: We train Word2Vec vectors on target-domain data and align them with the wordpiece vector space of a general-domain PTLM. On eight biomedical NER tasks, we cover over 50% of the BioBERT – BERT F1 delta, at 5% of BioBERT's domain adaptation CO$_2$ footprint and 2% of its cloud compute cost. We have also shown how to rapidly adapt an existing BERT QA model to an emerging domain – the Covid-19 pandemic – without the need for target-domain Language Model pretraining or finetuning. We hope that our approach will benefit practitioners with limited time or resources, and that it will encourage environmentally friendlier NLP. Inexpensive Domain Adaptation of Pretrained Language Models (Appendix) ::: Word2Vec training We downloaded the PubMed, PMC and CORD-19 corpora from: https://ftp.ncbi.nlm.nih.gov/pub/pmc/oa_bulk/ [20 January 2020, 68GB raw text] https://ftp.ncbi.nlm.nih.gov/pubmed/baseline/ [20 January 2020, 24GB raw text] https://pages.semanticscholar.org/coronavirus-research [17 April 2020, 2GB raw text] We extract all abstracts and text bodies and apply the BERT basic tokenizer (a word tokenizer that standard BERT uses before wordpiece tokenization). Then, we train CBOW Word2Vec with negative sampling. We use default parameters except for the vector size (which we set to $d_\mathrm {W2V} = d_\mathrm {LM}$). Inexpensive Domain Adaptation of Pretrained Language Models (Appendix) ::: Experiment 1: Biomedical NER ::: Pretrained models General-domain BERT and BioBERTv1.0 were downloaded from: https://storage.googleapis.com/bert_models/2018_10_18/cased_L-12_H-768_A-12.zip https://github.com/naver/biobert-pretrained Inexpensive Domain Adaptation of Pretrained Language Models (Appendix) ::: Experiment 1: Biomedical NER ::: Data We downloaded the NER datasets by following instructions on https://github.com/dmis-lab/biobert#Datasets. For detailed dataset statistics, see BIBREF2. Inexpensive Domain Adaptation of Pretrained Language Models (Appendix) ::: Experiment 1: Biomedical NER ::: Preprocessing We cut all sentences into chunks of 30 or fewer whitespace-tokenized words (without splitting inside labeled spans). Then, we tokenize every chunk $S$ with $\mathcal {T} = \mathcal {T}_\mathrm {LM}$ or $\mathcal {T} = \hat{\mathcal {T}}_\mathrm {LM}$ and add special tokens: Word-initial wordpieces in $\mathcal {T}(S)$ are labeled as B(egin), I(nside) or O(utside), while non-word-initial wordpieces are labeled as X(ignore). Inexpensive Domain Adaptation of Pretrained Language Models (Appendix) ::: Experiment 1: Biomedical NER ::: Modeling, training and inference We follow BIBREF2's implementation (https://github.com/dmis-lab/biobert): We add a randomly initialized softmax classifier on top of the last BERT layer to predict the labels. We finetune the entire model to minimize negative log likelihood, with the standard Adam optimizer BIBREF20 and a linear learning rate scheduler (10% warmup). Like BIBREF2, we finetune on the concatenation of the training and development set. All finetuning runs were done on a GeForce Titan X GPU (12GB). Since we do not have the resources for an extensive hyperparameter search, we use defaults and recommendations from the BioBERT repository: Batch size of 32, peak learning rate of $1 \cdot 10^{-5}$, and 100 epochs. At inference time, we gather the output logits of word-initial wordpieces only. Since the number of word-initial wordpieces is the same for $\mathcal {T}_\mathrm {LM}(S)$ and $\hat{\mathcal {T}}_\mathrm {LM}(S)$, this makes mean-pooling the logits straightforward. Inexpensive Domain Adaptation of Pretrained Language Models (Appendix) ::: Experiment 1: Biomedical NER ::: Note on our reproduction experiments We found it easier to reproduce or exceed BIBREF2's results for general-domain BERT, compared to their results for BioBERTv1.0 (see Figure FIGREF13, main paper). While this may be due to hyperparameters, it suggests that BioBERTv1.0 was more strongly tuned than BERT in the original BioBERT paper. This observation does not affect our conclusions, as GreenBioBERT performs better than reproduced BERT as well. Inexpensive Domain Adaptation of Pretrained Language Models (Appendix) ::: Experiment 2: Covid-19 QA ::: Pretrained model We downloaded the SQuADBERT baseline from: https://huggingface.co/bert-large-uncased-whole-word-masking-finetuned-squad Inexpensive Domain Adaptation of Pretrained Language Models (Appendix) ::: Experiment 2: Covid-19 QA ::: Data We downloaded the Deepset-AI Covid-QA dataset from: https://github.com/deepset-ai/COVID-QA/blob/master/data/question-answering/200423_covidQA.json [24 April 2020] At the time of writing, the dataset contains 1380 questions and gold answer spans. Every question is associated with one of 98 research papers (contexts). We treat the entire dataset as a test set. Note that there are some important differences between the dataset and SQuAD, which make the task challenging: The contexts are full documents rather than single paragraphs. Thus, the correct answer may appear several times, often with slightly different wordings. Only a single one of the occurrences is annotated as correct, e.g.: What was the prevalence of Coronavirus OC43 in community samples in Ilorin, Nigeria? 13.3% (95% CI 6.9-23.6%) # from main text (13.3%, 10/75). # from abstract SQuAD gold answers are defined as the “shortest span in the paragraph that answered the question” BIBREF19, but many Covid-QA gold answers are longer and contain non-essential context, e.g.: When was the Middle East Respiratory Syndrome Coronavirus isolated first? (MERS-CoV) was first isolated in 2012, in a 60-year-old man who died in Jeddah, KSA due to severe acute pneumonia and multiple organ failure 2012, Inexpensive Domain Adaptation of Pretrained Language Models (Appendix) ::: Experiment 2: Covid-19 QA ::: Preprocessing We tokenize every question-context pair $(Q, C)$ with $\mathcal {T} = \mathcal {T}_\mathrm {LM}$ or $\mathcal {T} = \hat{\mathcal {T}}_\mathrm {LM}$, which yields $(\mathcal {T}(Q), \mathcal {T}(C))$. Since $\mathcal {T}(C)$ is usually too long to be digested in a single forward pass, we define a sliding window with width and stride $N = \mathrm {floor}(\frac{509 - |\mathcal {T}(Q)|}{2})$. At step $n$, the “active” window is between $a^{(l)}_n = nN$ and $a^{(r)}_n = \mathrm {min}(|C|, nN+N)$. The input is defined as: $p^{(l)}_n$ and $p^{(r)}_n$ are chosen such that $|X^{(n)}| = 512$, and such that the active window is in the center of the input (if possible). Inexpensive Domain Adaptation of Pretrained Language Models (Appendix) ::: Experiment 2: Covid-19 QA ::: Modeling and inference Feeding $X^{(n)}$ into the pretrained QA model yields start logits $\mathbf {h^{\prime }}^{(\mathrm {start}, n)} \in \mathbb {R}^{|X^{(n)}|}$ and end logits $\mathbf {h^{\prime }}^{(\mathrm {end},n)} \in \mathbb {R}^{|X^{(n)}|}$. We extract and concatenate the slices that correspond to the active windows of all steps: Next, we map the logits from the wordpiece level to the word level. This allows us to mean-pool the outputs of $\mathcal {T}_\mathrm {LM}$ and $\hat{\mathcal {T}}_\mathrm {LM}$ even when $|\mathcal {T}_\mathrm {LM}(C)| \ne |\hat{\mathcal {T}}_\mathrm {LM}(C)|$. Let $c_i$ be a whitespace-delimited word in $C$. Let $\mathcal {T}(C)_{j:j+|\mathcal {T}(c_i)|}$ be the corresponding wordpieces. The start and end logits of $c_i$ are derived as: Finally, we return the answer span $C_{k:k^{\prime }}$ that maximizes $o^{(\mathrm {start})}_k + o^{(\mathrm {end})}_{k^{\prime }}$, subject to the constraints that $k^{\prime }$ does not precede $k$ and the answer span is not longer than 500 characters.
What in-domain text did they use?
16535db1d73a9373ffe9d6eedaa2369cefd91ac4
What Ġin - domain Ġtext Ġdid Ġthey Ġuse ?
Pub Med + PM C
<s> What Ġin - domain Ġtext Ġdid Ġthey Ġuse ? </s> Introduction </s> P ret rained ĠLanguage ĠModels Ġ( PT L Ms ) Ġsuch Ġas ĠB ERT ĠB IB REF 1 Ġhave Ġspearheaded Ġadvances Ġon Ġmany ĠN LP Ġtasks . ĠUsually , ĠP TL Ms Ġare Ġpret rained Ġon Ġunl abel ed Ġgeneral - domain Ġand / or Ġmixed - domain Ġtext , Ġsuch Ġas ĠWikipedia , Ġdigital Ġbooks Ġor Ġthe ĠCommon ĠC rawl Ġcorpus . </s> When Ġapplying ĠP TL Ms Ġto Ġspecific Ġdomains , Ġit Ġcan Ġbe Ġuseful Ġto Ġdomain - adapt Ġthem . ĠDomain Ġadaptation Ġof ĠP TL Ms Ġhas Ġtypically Ġbeen Ġachieved Ġby Ġpret raining Ġon Ġtarget - domain Ġtext . ĠOne Ġsuch Ġmodel Ġis ĠBio BER T ĠB IB REF 2 , Ġwhich Ġwas Ġinitialized Ġfrom Ġgeneral - domain ĠB ERT Ġand Ġthen Ġpret rained Ġon Ġbiomedical Ġscientific Ġpublications . ĠThe Ġdomain Ġadaptation Ġis Ġshown Ġto Ġbe Ġhelpful Ġfor Ġtarget - domain Ġtasks Ġsuch Ġas Ġbiomedical ĠNamed ĠEntity ĠRecogn ition Ġ( NER ) Ġor ĠQuestion ĠAn sw ering Ġ( Q A ). ĠOn Ġthe Ġdownside , Ġthe Ġcomputational Ġcost Ġof Ġpret raining Ġcan Ġbe Ġconsiderable : ĠBio BER T v 1 . 0 Ġwas Ġadapted Ġfor Ġten Ġdays Ġon Ġeight Ġlarge ĠGPUs Ġ( see ĠTable ĠT AB REF 1 ), Ġwhich Ġis Ġexpensive , Ġenvironmentally Ġunf riend ly , Ġprohib itive Ġfor Ġsmall Ġresearch Ġlabs Ġand Ġstudents , Ġand Ġmay Ġdelay Ġprototyp ing Ġon Ġemerging Ġdomains . </s> We Ġtherefore Ġpropose Ġa Ġfast , ĠCPU - only Ġdomain - adapt ation Ġmethod Ġfor ĠP TL Ms : ĠWe Ġtrain ĠWord 2 V ec ĠB IB REF 3 Ġon Ġtarget - domain Ġtext Ġand Ġalign Ġthe Ġresulting Ġword Ġvectors Ġwith Ġthe Ġword piece Ġvectors Ġof Ġan Ġexisting Ġgeneral - domain ĠP TL M . ĠThe ĠP TL M Ġthus Ġgains Ġdomain - specific Ġlex ical Ġknowledge Ġin Ġthe Ġform Ġof Ġadditional Ġword Ġvectors , Ġbut Ġits Ġdeeper Ġlayers Ġremain Ġunchanged . ĠSince ĠWord 2 V ec Ġand Ġthe Ġvector Ġspace Ġalignment Ġare Ġefficient Ġmodels , Ġthe Ġprocess Ġrequires Ġa Ġfraction Ġof Ġthe Ġresources Ġassociated Ġwith Ġpret raining Ġthe ĠP TL M Ġitself , Ġand Ġit Ġcan Ġbe Ġdone Ġon ĠCPU . </s> In ĠSection ĠSEC REF 4 , Ġwe Ġuse Ġthe Ġproposed Ġmethod Ġto Ġdomain - adapt ĠB ERT Ġon ĠPubMed + PM C Ġ( the Ġdata Ġused Ġfor ĠBio BER T v 1 . 0 ) Ġand / or ĠC ORD - 19 Ġ( C ov id - 19 ĠOpen ĠResearch ĠDat as et ). ĠWe Ġimprove Ġover Ġgeneral - domain ĠB ERT Ġon Ġeight Ġout Ġof Ġeight Ġbiomedical ĠN ER Ġtasks , Ġusing Ġa Ġfraction Ġof Ġthe Ġcompute Ġcost Ġassociated Ġwith ĠBio BER T . ĠIn ĠSection ĠSEC REF 5 , Ġwe Ġshow Ġhow Ġto Ġquickly Ġadapt Ġan Ġexisting ĠQuestion ĠAn sw ering Ġmodel Ġto Ġtext Ġabout Ġthe ĠCov id - 19 Ġpand emic , Ġwithout Ġany Ġtarget - domain ĠLanguage ĠModel Ġpret raining Ġor Ġfin et uning . </s> Related Ġwork Ġ: :: ĠThe ĠB ERT ĠP TL M </s> For Ġour Ġpurpose , Ġa ĠP TL M Ġconsists Ġof Ġthree Ġparts : ĠA Ġtoken izer Ġ$\ math cal Ġ{ T } _ \ math rm Ġ{ LM } Ġ: Ġ\ math bb Ġ{ L } ^ + Ġ\ right arrow Ġ\ math bb Ġ{ L } _ \ math rm Ġ{ LM } ^ + $, Ġa Ġword piece Ġembed ding Ġfunction Ġ$\ math cal Ġ{ E } _ \ math rm Ġ{ LM }: Ġ\ math bb Ġ{ L } _ \ math rm Ġ{ LM } Ġ\ right arrow Ġ\ math bb Ġ{ R } ^{ d _ \ math rm Ġ{ LM }} $ Ġand Ġan Ġenc oder Ġfunction Ġ$\ math cal Ġ{ F } _ \ math rm Ġ{ LM } $. Ġ$\ math bb Ġ{ L } _ \ math rm Ġ{ LM } $ Ġis Ġa Ġlimited Ġvocabulary Ġof Ġword pieces . ĠAll Ġwords Ġthat Ġare Ġnot Ġin Ġ$\ math bb Ġ{ L } _ \ math rm Ġ{ LM } $ Ġare Ġtoken ized Ġinto Ġsequences Ġof Ġshorter Ġword pieces , Ġe . g ., Ġt achy card ia Ġbecomes Ġta Ġ## chy Ġ## card Ġ## ia . ĠGiven Ġa Ġsentence Ġ$ S Ġ= Ġ[ w _ 1 , Ġ\ ld ots Ġ, Ġw _ T ] $, Ġtoken ized Ġas Ġ$\ math cal Ġ{ T } _ \ math rm Ġ{ LM } ( S ) Ġ= Ġ[ \ math cal Ġ{ T } _ \ math rm Ġ{ LM } ( w _ 1 ); Ġ\ ld ots Ġ; Ġ\ math cal Ġ{ T } _ \ math rm Ġ{ LM } ( w _ T )] $, Ġ$\ math cal Ġ{ E } _ \ math rm Ġ{ LM } $ Ġembed s Ġevery Ġword piece Ġin Ġ$\ math cal Ġ{ T } _ \ math rm Ġ{ LM } ( S ) $ Ġinto Ġa Ġreal - valued , Ġtrain able Ġword piece Ġvector . ĠThe Ġword piece Ġvectors Ġof Ġthe Ġentire Ġsequence Ġare Ġstacked Ġand Ġfed Ġinto Ġ$\ math cal Ġ{ F } _ \ math rm Ġ{ LM } $. ĠNote Ġthat Ġwe Ġconsider Ġposition Ġand Ġsegment Ġembed d ings Ġto Ġbe Ġa Ġpart Ġof Ġ$\ math cal Ġ{ F } _ \ math rm Ġ{ LM } $ Ġrather Ġthan Ġ$\ math cal Ġ{ E } _ \ math rm Ġ{ LM } $. </s> In Ġthe Ġcase Ġof ĠB ERT , Ġ$\ math cal Ġ{ F } _ \ math rm Ġ{ LM } $ Ġis Ġa ĠTrans former ĠB IB REF 4 , Ġfollowed Ġby Ġa Ġfinal ĠFeed - Forward ĠNet . ĠDuring Ġpret raining , Ġthe ĠFeed - Forward ĠNet Ġpredicts Ġthe Ġidentity Ġof Ġmasked Ġword pieces . ĠWhen Ġfin et uning Ġon Ġa Ġsupervised Ġtask , Ġit Ġis Ġusually Ġreplaced Ġwith Ġa Ġrandomly Ġinitialized Ġtask - specific Ġlayer . </s> Related Ġwork Ġ: :: ĠDomain - adapt ed ĠP TL Ms </s> Domain Ġadaptation Ġof ĠP TL Ms Ġis Ġtypically Ġachieved Ġby Ġpret raining Ġon Ġunl abel ed Ġtarget - domain Ġtext . ĠSome Ġexamples Ġof Ġsuch Ġmodels Ġare ĠBio BER T ĠB IB REF 2 , Ġwhich Ġwas Ġpret rained Ġon Ġthe ĠPubMed Ġand / or ĠPubMed ĠCentral Ġ( PM C ) Ġcorpor a , ĠSci BER T ĠB IB REF 5 , Ġwhich Ġwas Ġpret rained Ġon Ġpapers Ġfrom ĠSem antic Sch olar , ĠClinical BER T ĠB IB REF 6 , ĠB IB REF 7 Ġand ĠClinical XL Net ĠB IB REF 8 , Ġwhich Ġwere Ġpret rained Ġon Ġclinical Ġpatient Ġnotes , Ġand ĠAdapt a BER T ĠB IB REF 9 , Ġwhich Ġwas Ġpret rained Ġon ĠEarly ĠModern ĠEnglish Ġtext . ĠIn Ġmost Ġcases , Ġa Ġdomain - adapt ed ĠP TL M Ġis Ġinitialized Ġfrom Ġa Ġgeneral - domain ĠP TL M Ġ( e . g ., Ġstandard ĠB ERT ), Ġthough ĠB IB REF 5 Ġreport Ġbetter Ġresults Ġwith Ġa Ġmodel Ġthat Ġwas Ġpret rained Ġfrom Ġscratch Ġwith Ġa Ġcustom Ġword piece Ġvocabulary . ĠIn Ġthis Ġpaper , Ġwe Ġfocus Ġon ĠBio BER T , Ġas Ġits Ġdomain Ġadaptation Ġcorpor a Ġare Ġpublicly Ġavailable . </s> Related Ġwork Ġ: :: ĠWord Ġvectors </s> Word Ġvectors Ġare Ġdistributed Ġrepresentations Ġof Ġwords Ġthat Ġare Ġtrained Ġon Ġunl abel ed Ġtext . ĠContrary Ġto ĠP TL Ms , Ġword Ġvectors Ġare Ġnon - context ual , Ġi . e ., Ġa Ġword Ġtype Ġis Ġalways Ġassigned Ġthe Ġsame Ġvector , Ġregardless Ġof Ġcontext . ĠIn Ġthis Ġpaper , Ġwe Ġuse ĠWord 2 V ec ĠB IB REF 3 Ġto Ġtrain Ġword Ġvectors . ĠWe Ġwill Ġdenote Ġthe ĠWord 2 V ec Ġlookup Ġfunction Ġas Ġ$\ math cal Ġ{ E } _ \ math rm Ġ{ W 2 V } Ġ: Ġ\ math bb Ġ{ L } _ \ math rm Ġ{ W 2 V } Ġ\ right arrow Ġ\ math bb Ġ{ R } ^{ d _ \ math rm Ġ{ W 2 V }} $. </s> Related Ġwork Ġ: :: ĠWord Ġvector Ġspace Ġalignment </s> Word Ġvector Ġspace Ġalignment Ġhas Ġmost Ġfrequently Ġbeen Ġexplored Ġin Ġthe Ġcontext Ġof Ġcross - ling ual Ġword Ġembed d ings . ĠFor Ġinstance , ĠB IB REF 10 Ġalign ĠEnglish Ġand ĠSpanish ĠWord 2 V ec Ġspaces Ġby Ġa Ġsimple Ġlinear Ġtransformation . ĠB IB REF 11 Ġuse Ġa Ġrelated Ġmethod Ġto Ġalign Ġcross - ling ual Ġword Ġvectors Ġand Ġmult ilingual ĠB ERT Ġword piece Ġvectors . </s> Method </s> In Ġthe Ġfollowing , Ġwe Ġassume Ġaccess Ġto Ġa Ġgeneral - domain ĠP TL M , Ġas Ġdescribed Ġin ĠSection ĠSEC REF 2 , Ġand Ġa Ġcorpus Ġof Ġunl abel ed Ġtarget - domain Ġtext . </s> Method Ġ: :: ĠCreating Ġnew Ġinput Ġvectors </s> In Ġa Ġfirst Ġstep , Ġwe Ġtrain ĠWord 2 V ec Ġon Ġthe Ġtarget - domain Ġcorpus . ĠIn Ġa Ġsecond Ġstep , Ġwe Ġtake Ġthe Ġintersection Ġof Ġ$\ math bb Ġ{ L } _ \ math rm Ġ{ LM } $ Ġand Ġ$\ math bb Ġ{ L } _ \ math rm Ġ{ W 2 V } $. ĠIn Ġpractice , Ġthe Ġintersection Ġmostly Ġcontains Ġword pieces Ġfrom Ġ$\ math bb Ġ{ L } _ \ math rm Ġ{ LM } $ Ġthat Ġcorrespond Ġto Ġstandalone Ġwords . ĠIt Ġalso Ġcontains Ġsingle Ġcharacters Ġand Ġother Ġnoise , Ġhowever , Ġwe Ġfound Ġthat Ġfiltering Ġthem Ġdoes Ġnot Ġimprove Ġalignment Ġquality . ĠIn Ġa Ġthird Ġstep , Ġwe Ġuse Ġthe Ġintersection Ġto Ġfit Ġan Ġuncon str ained Ġlinear Ġtransformation Ġ$\ math bf Ġ{ W } Ġ\ in Ġ\ math bb Ġ{ R } ^{ d _ \ math rm Ġ{ LM } Ġ\ times Ġd _ \ math rm Ġ{ W 2 V }} $ Ġvia Ġleast Ġsquares : </s> Int uitive ly , Ġ$\ math bf Ġ{ W } $ Ġmakes ĠWord 2 V ec Ġvectors ĠâĢ ľ look Ġlike âĢ Ŀ Ġthe ĠP TL M 's Ġnative Ġword piece Ġvectors , Ġjust Ġlike Ġcross - ling ual Ġalignment Ġmakes Ġword Ġvectors Ġfrom Ġone Ġlanguage ĠâĢ ľ look Ġlike âĢ Ŀ Ġword Ġvectors Ġfrom Ġanother Ġlanguage . ĠIn ĠTable ĠT AB REF 7 Ġ( top ), Ġwe Ġshow Ġexamples Ġof Ġwithin - space Ġand Ġcross - space Ġnearest Ġneighbors Ġafter Ġalignment . </s> Method Ġ: :: ĠUp dating Ġthe Ġword piece Ġembed ding Ġlayer </s> Next , Ġwe Ġredef ine Ġthe Ġword piece Ġembed ding Ġlayer Ġof Ġthe ĠP TL M . ĠThe Ġmost Ġradical Ġstrategy Ġwould Ġbe Ġto Ġreplace Ġthe Ġentire Ġlayer Ġwith Ġthe Ġaligned ĠWord 2 V ec Ġvectors : </s> In Ġinitial Ġexperiments , Ġthis Ġstrategy Ġled Ġto Ġa Ġdrop Ġin Ġperformance , Ġpresumably Ġbecause Ġfunction Ġwords Ġare Ġnot Ġwell Ġrepresented Ġby ĠWord 2 V ec , Ġand Ġreplacing Ġthem Ġdisrupt s ĠB ERT 's Ġsynt actic Ġabilities . ĠTo Ġprevent Ġthis Ġproblem , Ġwe Ġleave Ġexisting Ġword piece Ġvectors Ġintact Ġand Ġonly Ġadd Ġnew Ġones : </s> Method Ġ: :: ĠUp dating Ġthe Ġtoken izer </s> In Ġa Ġfinal Ġstep , Ġwe Ġupdate Ġthe Ġtoken izer Ġto Ġaccount Ġfor Ġthe Ġadded Ġwords . ĠLet Ġ$\ math cal Ġ{ T } _ \ math rm Ġ{ LM } $ Ġbe Ġthe Ġstandard ĠB ERT Ġtoken izer , Ġand Ġlet Ġ$\ hat {\ math cal Ġ{ T }} _ \ math rm Ġ{ LM } $ Ġbe Ġthe Ġtoken izer Ġthat Ġtreats Ġall Ġwords Ġin Ġ$\ math bb Ġ{ L } _ \ math rm Ġ{ LM } Ġ\ cup Ġ\ math bb Ġ{ L } _ \ math rm Ġ{ W 2 V } $ Ġas Ġone - word piece Ġtokens , Ġwhile Ġtoken izing Ġany Ġother Ġwords Ġas Ġusual . </s> In Ġpractice , Ġa Ġgiven Ġword Ġmay Ġor Ġmay Ġnot Ġbenefit Ġfrom Ġbeing Ġtoken ized Ġby Ġ$\ hat {\ math cal Ġ{ T }} _ \ math rm Ġ{ LM } $ Ġinstead Ġof Ġ$\ math cal Ġ{ T } _ \ math rm Ġ{ LM } $. ĠTo Ġgive Ġa Ġconcrete Ġexample , Ġ82 % Ġof Ġthe Ġwords Ġin Ġthe ĠBC 5 C DR ĠN ER Ġdataset Ġthat Ġend Ġin Ġthe Ġsuffix Ġ- ia Ġare Ġinside Ġa Ġdisease Ġentity Ġ( e . g ., Ġt achy card ia ). Ġ$\ math cal Ġ{ T } _ \ math rm Ġ{ LM } $ Ġtoken izes Ġthis Ġword Ġas Ġta Ġ## chy Ġ## card Ġ## ia , Ġthereby Ġexposing Ġthe Ġorth ographic Ġcue Ġto Ġthe Ġmodel . ĠAs Ġa Ġresult , Ġ$\ math cal Ġ{ T } _ \ math rm Ġ{ LM } $ Ġleads Ġto Ġhigher Ġrecall Ġon Ġ- ia Ġdiseases . ĠBut Ġthere Ġare Ġmany Ġcases Ġwhere Ġword piece Ġtoken ization Ġis Ġmeaningless Ġor Ġmisleading . ĠFor Ġinstance Ġeuth ym ia Ġ( not Ġa Ġdisease ) Ġis Ġtoken ized Ġby Ġ$\ math cal Ġ{ T } _ \ math rm Ġ{ LM } $ Ġas Ġe Ġ## uth Ġ## ym Ġ## ia , Ġmaking Ġit Ġlikely Ġto Ġbe Ġclassified Ġas Ġa Ġdisease . ĠBy Ġcontrast , Ġ$\ hat {\ math cal Ġ{ T }} _ \ math rm Ġ{ LM } $ Ġgives Ġeuth ym ia Ġa Ġone - word piece Ġrepresentation Ġthat Ġdepends Ġonly Ġon Ġdistribution al Ġsemantics . ĠWe Ġfind Ġthat Ġusing Ġ$\ hat {\ math cal Ġ{ T }} _ \ math rm Ġ{ LM } $ Ġimproves Ġprecision Ġon Ġ- ia Ġdiseases . </s> To Ġcombine Ġthese Ġcomplementary Ġstrengths , Ġwe Ġuse Ġa Ġ50 / 50 Ġmixture Ġof Ġ$\ math cal Ġ{ T } _ \ math rm Ġ{ LM } $ - token ization Ġand Ġ$\ hat {\ math cal Ġ{ T }} _ \ math rm Ġ{ LM } $ - token ization Ġwhen Ġfin et uning Ġthe ĠP TL M Ġon Ġa Ġtask . ĠAt Ġtest Ġtime , Ġwe Ġuse Ġboth Ġtoken izers Ġand Ġmean - pool Ġthe Ġoutputs . ĠLet Ġ$ o (\ math cal Ġ{ T } ( S )) $ Ġbe Ġsome Ġoutput Ġof Ġinterest Ġ( e . g ., Ġa Ġlog it ), Ġgiven Ġsentence Ġ$ S $ Ġtoken ized Ġby Ġ$\ math cal Ġ{ T } $. ĠWe Ġpredict : </s> Exper iment Ġ1 : ĠBi omedical ĠN ER </s> In Ġthis Ġsection , Ġwe Ġuse Ġthe Ġproposed Ġmethod Ġto Ġcreate ĠGreen Bio BER T , Ġan Ġinexpensive Ġand Ġenvironmentally Ġfriendly Ġalternative Ġto ĠBio BER T . ĠRecall Ġthat ĠBio BER T v 1 . 0 Ġ( b io bert _ v 1 . 0 _ pub med _ pm c ) Ġwas Ġinitialized Ġfrom Ġgeneral - domain ĠB ERT Ġ( bert - base - c ased ) Ġand Ġpret rained Ġon ĠPubMed + PM C . </s> Exper iment Ġ1 : ĠBi omedical ĠN ER Ġ: :: ĠDomain Ġadaptation </s> We Ġtrain ĠWord 2 V ec Ġwith Ġvector Ġsize Ġ$ d _ \ math rm Ġ{ W 2 V } Ġ= Ġd _ \ math rm Ġ{ LM } Ġ= Ġ768 $ Ġon ĠPubMed + PM C Ġ( see ĠAppendix Ġfor Ġdetails ). ĠThen , Ġwe Ġfollow Ġthe Ġprocedure Ġdescribed Ġin ĠSection ĠSEC REF 3 Ġto Ġupdate Ġthe Ġword piece Ġembed ding Ġlayer Ġand Ġtoken izer Ġof Ġgeneral - domain ĠB ERT . </s> Exper iment Ġ1 : ĠBi omedical ĠN ER Ġ: :: ĠFin et uning </s> We Ġfin et une ĠGreen Bio BER T Ġon Ġthe Ġeight Ġpublicly Ġavailable ĠN ER Ġtasks Ġused Ġin ĠB IB REF 2 . ĠWe Ġalso Ġdo Ġreproduction Ġexperiments Ġwith Ġgeneral - domain ĠB ERT Ġand ĠBio BER T v 1 . 0 , Ġusing Ġthe Ġsame Ġsetup Ġas Ġour Ġmodel . ĠWe Ġaverage Ġresults Ġover Ġeight Ġrandom Ġseeds . ĠSee ĠAppendix Ġfor Ġdetails Ġon Ġpre processing , Ġtraining Ġand Ġhyper param eters . </s> Exper iment Ġ1 : ĠBi omedical ĠN ER Ġ: :: ĠResults Ġand Ġdiscussion </s> Table ĠT AB REF 7 Ġ( bottom ) Ġshows Ġentity - level Ġprecision , Ġrecall Ġand ĠF 1 . ĠFor Ġease Ġof Ġvisualization , ĠFigure ĠFIG REF 13 Ġshows Ġwhat Ġportion Ġof Ġthe ĠBio BER T ĠâĢĵ ĠB ERT ĠF 1 Ġdelta Ġis Ġcovered . ĠWe Ġimprove Ġover Ġgeneral - domain ĠB ERT Ġon Ġall Ġtasks Ġwith Ġvarying Ġeffect Ġsizes . ĠDepending Ġon Ġthe Ġpoints Ġof Ġreference , Ġwe Ġcover Ġan Ġaverage Ġ52 % Ġto Ġ60 % Ġof Ġthe ĠBio BER T ĠâĢĵ ĠB ERT ĠF 1 Ġdelta Ġ( 54 % Ġfor ĠBio BER T v 1 . 0 , Ġ60 % Ġfor ĠBio BER T v 1 . 1 Ġand Ġ52 % Ġfor Ġour Ġreproduction Ġexperiments ). ĠTable ĠT AB REF 17 Ġ( top ) Ġshows Ġthe Ġimportance Ġof Ġvector Ġspace Ġalignment : ĠIf Ġwe Ġreplace Ġthe Ġaligned ĠWord 2 V ec Ġvectors Ġwith Ġtheir Ġnon - aligned Ġcounterparts Ġ( by Ġsetting Ġ$\ math bf Ġ{ W } Ġ= Ġ\ math bf Ġ{ 1 } $ ) Ġor Ġwith Ġrandomly Ġinitialized Ġvectors , ĠF 1 Ġdrops Ġon Ġall Ġtasks . </s> Exper iment Ġ2 : ĠCov id - 19 ĠQ A </s> In Ġthis Ġsection , Ġwe Ġuse Ġthe Ġproposed Ġmethod Ġto Ġquickly Ġadapt Ġan Ġexisting Ġgeneral - domain ĠQ A Ġmodel Ġto Ġan Ġemerging Ġtarget Ġdomain : ĠCov id - 19 . ĠOur Ġbaseline Ġmodel Ġis ĠS Qu AD BER T Ġ( bert - large - unc ased - wh ole - word - mask ing - fin et uned - squ ad ), Ġa Ġversion Ġof ĠB ERT Ġthat Ġwas Ġfin et uned Ġon Ġgeneral - domain ĠS Qu AD ĠB IB REF 19 . ĠWe Ġevaluate Ġon ĠDeep set - AI ĠCov id - Q A , Ġa ĠS Qu AD - style Ġdataset Ġwith Ġ13 80 Ġquestions Ġ( see ĠAppendix Ġfor Ġdetails Ġon Ġdata Ġand Ġpre processing ). ĠWe Ġassume Ġthat Ġthere Ġis Ġno Ġtarget - domain Ġfin et uning Ġdata , Ġwhich Ġis Ġa Ġrealistic Ġsetup Ġfor Ġa Ġnew Ġdomain . </s> Exper iment Ġ2 : ĠCov id - 19 ĠQ A Ġ: :: ĠDomain Ġadaptation </s> We Ġtrain ĠWord 2 V ec Ġwith Ġvector Ġsize Ġ$ d _ \ math rm Ġ{ W 2 V } Ġ= Ġd _ \ math rm Ġ{ LM } Ġ= Ġ1024 $ Ġon ĠC ORD - 19 Ġ( C ov id - 19 ĠOpen ĠResearch ĠDat as et ) Ġand / or ĠPubMed + PM C . ĠThe Ġprocess Ġtakes Ġless Ġthan Ġan Ġhour Ġon ĠC ORD - 19 Ġand Ġabout Ġone Ġday Ġon Ġthe Ġcombined Ġcorpus , Ġagain Ġwithout Ġthe Ġneed Ġfor Ġa ĠGPU . ĠThen , Ġwe Ġupdate ĠS Qu AD BER T 's Ġword piece Ġembed ding Ġlayer Ġand Ġtoken izer , Ġas Ġdescribed Ġin ĠSection ĠSEC REF 3 . ĠWe Ġrefer Ġto Ġthe Ġresulting Ġmodel Ġas ĠGreen C ov id S Qu AD BER T . </s> Exper iment Ġ2 : ĠCov id - 19 ĠQ A Ġ: :: ĠResults Ġand Ġdiscussion </s> Table ĠT AB REF 17 Ġ( bottom ) Ġshows Ġthat ĠGreen C ov id S Qu AD BER T Ġoutper forms Ġgeneral - domain ĠS Qu AD BER T Ġin Ġall Ġmetrics . ĠMost Ġof Ġthe Ġimprovement Ġcan Ġbe Ġachieved Ġwith Ġjust Ġthe Ġsmall ĠC ORD - 19 Ġcorpus , Ġwhich Ġis Ġmore Ġspecific Ġto Ġthe Ġtarget Ġdomain Ġ( comp are ĠâĢ ľ C ord - 19 Ġonly âĢ Ŀ Ġand ĠâĢ ľ C ord - 19 + Pub Med + PM C âĢ Ŀ ). </s> Conclusion </s> As Ġa Ġreaction Ġto Ġthe Ġtrend Ġtowards Ġhigh - resource Ġmodels , Ġwe Ġhave Ġproposed Ġan Ġinexpensive , ĠCPU - only Ġmethod Ġfor Ġdomain - adapt ing ĠPret rained ĠLanguage ĠModels : ĠWe Ġtrain ĠWord 2 V ec Ġvectors Ġon Ġtarget - domain Ġdata Ġand Ġalign Ġthem Ġwith Ġthe Ġword piece Ġvector Ġspace Ġof Ġa Ġgeneral - domain ĠP TL M . </s> On Ġeight Ġbiomedical ĠN ER Ġtasks , Ġwe Ġcover Ġover Ġ50 % Ġof Ġthe ĠBio BER T ĠâĢĵ ĠB ERT ĠF 1 Ġdelta , Ġat Ġ5 % Ġof ĠBio BER T 's Ġdomain Ġadaptation ĠCO $ _ 2 $ Ġfootprint Ġand Ġ2 % Ġof Ġits Ġcloud Ġcompute Ġcost . ĠWe Ġhave Ġalso Ġshown Ġhow Ġto Ġrapidly Ġadapt Ġan Ġexisting ĠB ERT ĠQ A Ġmodel Ġto Ġan Ġemerging Ġdomain ĠâĢĵ Ġthe ĠCov id - 19 Ġpand emic ĠâĢĵ Ġwithout Ġthe Ġneed Ġfor Ġtarget - domain ĠLanguage ĠModel Ġpret raining Ġor Ġfin et uning . </s> We Ġhope Ġthat Ġour Ġapproach Ġwill Ġbenefit Ġpractitioners Ġwith Ġlimited Ġtime Ġor Ġresources , Ġand Ġthat Ġit Ġwill Ġencourage Ġenvironmentally Ġfriend lier ĠN LP . </s> I nex p ensive ĠDomain ĠAdapt ation Ġof ĠPret rained ĠLanguage ĠModels Ġ( App endix ) Ġ: :: ĠWord 2 V ec Ġtraining </s> We Ġdownloaded Ġthe ĠPubMed , ĠPM C Ġand ĠC ORD - 19 Ġcorpor a Ġfrom : </s> https :// ft p . nc bi . nl m . nih . gov / pub / pm c / oa _ b ulk / Ġ[ 20 ĠJanuary Ġ2020 , Ġ68 GB Ġraw Ġtext ] </s> https :// ft p . nc bi . nl m . nih . gov / pub med / bas eline / Ġ[ 20 ĠJanuary Ġ2020 , Ġ24 GB Ġraw Ġtext ] </s> https :// pages . sem antics ch olar . org / cor on av irus - research Ġ[ 17 ĠApril Ġ2020 , Ġ2 GB Ġraw Ġtext ] </s> We Ġextract Ġall Ġabstract s Ġand Ġtext Ġbodies Ġand Ġapply Ġthe ĠB ERT Ġbasic Ġtoken izer Ġ( a Ġword Ġtoken izer Ġthat Ġstandard ĠB ERT Ġuses Ġbefore Ġword piece Ġtoken ization ). ĠThen , Ġwe Ġtrain ĠCB OW ĠWord 2 V ec Ġwith Ġnegative Ġsampling . ĠWe Ġuse Ġdefault Ġparameters Ġexcept Ġfor Ġthe Ġvector Ġsize Ġ( which Ġwe Ġset Ġto Ġ$ d _ \ math rm Ġ{ W 2 V } Ġ= Ġd _ \ math rm Ġ{ LM } $ ). </s> I nex p ensive ĠDomain ĠAdapt ation Ġof ĠPret rained ĠLanguage ĠModels Ġ( App endix ) Ġ: :: ĠExperiment Ġ1 : ĠBi omedical ĠN ER Ġ: :: ĠPret rained Ġmodels </s> General - domain ĠB ERT Ġand ĠBio BER T v 1 . 0 Ġwere Ġdownloaded Ġfrom : </s> https :// storage . google ap is . com / bert _ models / 2018 _ 10 _ 18 / c ased _ L - 12 _ H - 768 _ A - 12 . zip </s> https :// github . com / na ver / b io bert - pret rained </s> I nex p ensive ĠDomain ĠAdapt ation Ġof ĠPret rained ĠLanguage ĠModels Ġ( App endix ) Ġ: :: ĠExperiment Ġ1 : ĠBi omedical ĠN ER Ġ: :: ĠData </s> We Ġdownloaded Ġthe ĠN ER Ġdatasets Ġby Ġfollowing Ġinstructions Ġon Ġhttps :// github . com / d mis - lab / b io bert # Dat as ets . ĠFor Ġdetailed Ġdataset Ġstatistics , Ġsee ĠB IB REF 2 . </s> I nex p ensive ĠDomain ĠAdapt ation Ġof ĠPret rained ĠLanguage ĠModels Ġ( App endix ) Ġ: :: ĠExperiment Ġ1 : ĠBi omedical ĠN ER Ġ: :: ĠPre processing </s> We Ġcut Ġall Ġsentences Ġinto Ġchunks Ġof Ġ30 Ġor Ġfewer Ġwhites pace - token ized Ġwords Ġ( without Ġsplitting Ġinside Ġlabeled Ġspans ). ĠThen , Ġwe Ġtoken ize Ġevery Ġchunk Ġ$ S $ Ġwith Ġ$\ math cal Ġ{ T } Ġ= Ġ\ math cal Ġ{ T } _ \ math rm Ġ{ LM } $ Ġor Ġ$\ math cal Ġ{ T } Ġ= Ġ\ hat {\ math cal Ġ{ T }} _ \ math rm Ġ{ LM } $ Ġand Ġadd Ġspecial Ġtokens : </s> Word - initial Ġword pieces Ġin Ġ$\ math cal Ġ{ T } ( S ) $ Ġare Ġlabeled Ġas ĠB ( eg in ), ĠI ( n side ) Ġor ĠO ( ut side ), Ġwhile Ġnon - word - initial Ġword pieces Ġare Ġlabeled Ġas ĠX ( ignore ). </s> I nex p ensive ĠDomain ĠAdapt ation Ġof ĠPret rained ĠLanguage ĠModels Ġ( App endix ) Ġ: :: ĠExperiment Ġ1 : ĠBi omedical ĠN ER Ġ: :: ĠModel ing , Ġtraining Ġand Ġinference </s> We Ġfollow ĠB IB REF 2 's Ġimplementation Ġ( https :// github . com / d mis - lab / b io bert ): ĠWe Ġadd Ġa Ġrandomly Ġinitialized Ġsoft max Ġclass ifier Ġon Ġtop Ġof Ġthe Ġlast ĠB ERT Ġlayer Ġto Ġpredict Ġthe Ġlabels . ĠWe Ġfin et une Ġthe Ġentire Ġmodel Ġto Ġminimize Ġnegative Ġlog Ġlikelihood , Ġwith Ġthe Ġstandard ĠAdam Ġoptim izer ĠB IB REF 20 Ġand Ġa Ġlinear Ġlearning Ġrate Ġsched uler Ġ( 10 % Ġwarm up ). ĠLike ĠB IB REF 2 , Ġwe Ġfin et une Ġon Ġthe Ġconc aten ation Ġof Ġthe Ġtraining Ġand Ġdevelopment Ġset . ĠAll Ġfin et uning Ġruns Ġwere Ġdone Ġon Ġa ĠGeForce ĠTitan ĠX ĠGPU Ġ( 12 GB ). </s> Since Ġwe Ġdo Ġnot Ġhave Ġthe Ġresources Ġfor Ġan Ġextensive Ġhyper param eter Ġsearch , Ġwe Ġuse Ġdefaults Ġand Ġrecommendations Ġfrom Ġthe ĠBio BER T Ġrepository : ĠB atch Ġsize Ġof Ġ32 , Ġpeak Ġlearning Ġrate Ġof Ġ$ 1 Ġ\ cd ot Ġ10 ^{ - 5 } $, Ġand Ġ100 Ġepoch s . </s> At Ġinference Ġtime , Ġwe Ġgather Ġthe Ġoutput Ġlog its Ġof Ġword - initial Ġword pieces Ġonly . ĠSince Ġthe Ġnumber Ġof Ġword - initial Ġword pieces Ġis Ġthe Ġsame Ġfor Ġ$\ math cal Ġ{ T } _ \ math rm Ġ{ LM } ( S ) $ Ġand Ġ$\ hat {\ math cal Ġ{ T }} _ \ math rm Ġ{ LM } ( S ) $, Ġthis Ġmakes Ġmean - pool ing Ġthe Ġlog its Ġstraightforward . </s> I nex p ensive ĠDomain ĠAdapt ation Ġof ĠPret rained ĠLanguage ĠModels Ġ( App endix ) Ġ: :: ĠExperiment Ġ1 : ĠBi omedical ĠN ER Ġ: :: ĠNote Ġon Ġour Ġreproduction Ġexperiments </s> We Ġfound Ġit Ġeasier Ġto Ġreproduce Ġor Ġexceed ĠB IB REF 2 's Ġresults Ġfor Ġgeneral - domain ĠB ERT , Ġcompared Ġto Ġtheir Ġresults Ġfor ĠBio BER T v 1 . 0 Ġ( see ĠFigure ĠFIG REF 13 , Ġmain Ġpaper ). ĠWhile Ġthis Ġmay Ġbe Ġdue Ġto Ġhyper param eters , Ġit Ġsuggests Ġthat ĠBio BER T v 1 . 0 Ġwas Ġmore Ġstrongly Ġtuned Ġthan ĠB ERT Ġin Ġthe Ġoriginal ĠBio BER T Ġpaper . ĠThis Ġobservation Ġdoes Ġnot Ġaffect Ġour Ġconclusions , Ġas ĠGreen Bio BER T Ġperforms Ġbetter Ġthan Ġreproduced ĠB ERT Ġas Ġwell . </s> I nex p ensive ĠDomain ĠAdapt ation Ġof ĠPret rained ĠLanguage ĠModels Ġ( App endix ) Ġ: :: ĠExperiment Ġ2 : ĠCov id - 19 ĠQ A Ġ: :: ĠPret rained Ġmodel </s> We Ġdownloaded Ġthe ĠS Qu AD BER T Ġbaseline Ġfrom : </s> https :// h ug ging face . co / bert - large - unc ased - wh ole - word - mask ing - fin et uned - squ ad </s> I nex p ensive ĠDomain ĠAdapt ation Ġof ĠPret rained ĠLanguage ĠModels Ġ( App endix ) Ġ: :: ĠExperiment Ġ2 : ĠCov id - 19 ĠQ A Ġ: :: ĠData </s> We Ġdownloaded Ġthe ĠDeep set - AI ĠCov id - Q A Ġdataset Ġfrom : </s> https :// github . com / deep set - ai / CO VID - Q A / bl ob / master / data / question - ans w ering / 2004 23 _ c ov id Q A . json Ġ[ 24 ĠApril Ġ2020 ] </s> At Ġthe Ġtime Ġof Ġwriting , Ġthe Ġdataset Ġcontains Ġ13 80 Ġquestions Ġand Ġgold Ġanswer Ġspans . ĠEvery Ġquestion Ġis Ġassociated Ġwith Ġone Ġof Ġ98 Ġresearch Ġpapers Ġ( context s ). ĠWe Ġtreat Ġthe Ġentire Ġdataset Ġas Ġa Ġtest Ġset . </s> Note Ġthat Ġthere Ġare Ġsome Ġimportant Ġdifferences Ġbetween Ġthe Ġdataset Ġand ĠS Qu AD , Ġwhich Ġmake Ġthe Ġtask Ġchallenging : </s> The Ġcontexts Ġare Ġfull Ġdocuments Ġrather Ġthan Ġsingle Ġparagraphs . ĠThus , Ġthe Ġcorrect Ġanswer Ġmay Ġappear Ġseveral Ġtimes , Ġoften Ġwith Ġslightly Ġdifferent Ġword ings . ĠOnly Ġa Ġsingle Ġone Ġof Ġthe Ġoccurrences Ġis Ġannot ated Ġas Ġcorrect , Ġe . g .: </s> What Ġwas Ġthe Ġprevalence Ġof ĠCor on av irus ĠOC 43 Ġin Ġcommunity Ġsamples Ġin ĠI lor in , ĠNigeria ? </s> 13 . 3 % Ġ( 95 % ĠCI Ġ6 . 9 - 23 . 6 %) Ġ# Ġfrom Ġmain Ġtext </s> ( 13 . 3 %, Ġ10 / 75 ). Ġ# Ġfrom Ġabstract </s> S Qu AD Ġgold Ġanswers Ġare Ġdefined Ġas Ġthe ĠâĢ ľ short est Ġspan Ġin Ġthe Ġparagraph Ġthat Ġanswered Ġthe Ġquestion âĢ Ŀ ĠB IB REF 19 , Ġbut Ġmany ĠCov id - Q A Ġgold Ġanswers Ġare Ġlonger Ġand Ġcontain Ġnon - essential Ġcontext , Ġe . g .: </s> When Ġwas Ġthe ĠMiddle ĠEast ĠRes pir atory ĠSyndrome ĠCor on av irus Ġisolated Ġfirst ? </s> ( M ERS - Co V ) Ġwas Ġfirst Ġisolated Ġin Ġ2012 , Ġin Ġa Ġ60 - year - old Ġman Ġwho Ġdied Ġin ĠJ edd ah , ĠK SA Ġdue Ġto Ġsevere Ġacute Ġpneumonia Ġand Ġmultiple Ġorgan Ġfailure </s> 2012 , </s> I nex p ensive ĠDomain ĠAdapt ation Ġof ĠPret rained ĠLanguage ĠModels Ġ( App endix ) Ġ: :: ĠExperiment Ġ2 : ĠCov id - 19 ĠQ A Ġ: :: ĠPre processing </s> We Ġtoken ize Ġevery Ġquestion - context Ġpair Ġ$( Q , ĠC ) $ Ġwith Ġ$\ math cal Ġ{ T } Ġ= Ġ\ math cal Ġ{ T } _ \ math rm Ġ{ LM } $ Ġor Ġ$\ math cal Ġ{ T } Ġ= Ġ\ hat {\ math cal Ġ{ T }} _ \ math rm Ġ{ LM } $, Ġwhich Ġyields Ġ$( \ math cal Ġ{ T } ( Q ), Ġ\ math cal Ġ{ T } ( C )) $. ĠSince Ġ$\ math cal Ġ{ T } ( C ) $ Ġis Ġusually Ġtoo Ġlong Ġto Ġbe Ġdig ested Ġin Ġa Ġsingle Ġforward Ġpass , Ġwe Ġdefine Ġa Ġsliding Ġwindow Ġwith Ġwidth Ġand Ġstride Ġ$ N Ġ= Ġ\ math rm Ġ{ floor } (\ frac { 509 Ġ- Ġ| \ math cal Ġ{ T } ( Q )| }{ 2 }) $. ĠAt Ġstep Ġ$ n $, Ġthe ĠâĢ ľ active âĢ Ŀ Ġwindow Ġis Ġbetween Ġ$ a ^{ ( l )} _ n Ġ= Ġn N $ Ġand Ġ$ a ^{ ( r )} _ n Ġ= Ġ\ math rm Ġ{ min } ( | C | , Ġn N + N ) $. ĠThe Ġinput Ġis Ġdefined Ġas : </s> $ p ^{ ( l )} _ n $ Ġand Ġ$ p ^{ ( r )} _ n $ Ġare Ġchosen Ġsuch Ġthat Ġ$ | X ^{ ( n )} | Ġ= Ġ512 $, Ġand Ġsuch Ġthat Ġthe Ġactive Ġwindow Ġis Ġin Ġthe Ġcenter Ġof Ġthe Ġinput Ġ( if Ġpossible ). </s> I nex p ensive ĠDomain ĠAdapt ation Ġof ĠPret rained ĠLanguage ĠModels Ġ( App endix ) Ġ: :: ĠExperiment Ġ2 : ĠCov id - 19 ĠQ A Ġ: :: ĠModel ing Ġand Ġinference </s> Feed ing Ġ$ X ^{ ( n )} $ Ġinto Ġthe Ġpret rained ĠQ A Ġmodel Ġyields Ġstart Ġlog its Ġ$\ math bf Ġ{ h ^ {\ prime Ġ}} ^{ (\ math rm Ġ{ start }, Ġn )} Ġ\ in Ġ\ math bb Ġ{ R } ^{ | X ^{ ( n )} | } $ Ġand Ġend Ġlog its Ġ$\ math bf Ġ{ h ^ {\ prime Ġ}} ^{ (\ math rm Ġ{ end }, n )} Ġ\ in Ġ\ math bb Ġ{ R } ^{ | X ^{ ( n )} | } $. ĠWe Ġextract Ġand Ġconc aten ate Ġthe Ġslices Ġthat Ġcorrespond Ġto Ġthe Ġactive Ġwindows Ġof Ġall Ġsteps : </s> Next , Ġwe Ġmap Ġthe Ġlog its Ġfrom Ġthe Ġword piece Ġlevel Ġto Ġthe Ġword Ġlevel . ĠThis Ġallows Ġus Ġto Ġmean - pool Ġthe Ġoutputs Ġof Ġ$\ math cal Ġ{ T } _ \ math rm Ġ{ LM } $ Ġand Ġ$\ hat {\ math cal Ġ{ T }} _ \ math rm Ġ{ LM } $ Ġeven Ġwhen Ġ$ | \ math cal Ġ{ T } _ \ math rm Ġ{ LM } ( C )| Ġ\ ne Ġ| \ hat {\ math cal Ġ{ T }} _ \ math rm Ġ{ LM } ( C )| $. </s> Let Ġ$ c _ i $ Ġbe Ġa Ġwhites pace - del im ited Ġword Ġin Ġ$ C $. ĠLet Ġ$\ math cal Ġ{ T } ( C ) _{ j : j + | \ math cal Ġ{ T } ( c _ i )| } $ Ġbe Ġthe Ġcorresponding Ġword pieces . ĠThe Ġstart Ġand Ġend Ġlog its Ġof Ġ$ c _ i $ Ġare Ġderived Ġas : </s> Finally , Ġwe Ġreturn Ġthe Ġanswer Ġspan Ġ$ C _{ k : k ^ {\ prime Ġ}} $ Ġthat Ġmaxim izes Ġ$ o ^{ (\ math rm Ġ{ start }) } _ k Ġ+ Ġo ^{ (\ math rm Ġ{ end }) } _{ k ^ {\ prime Ġ}} $, Ġsubject Ġto Ġthe Ġconstraints Ġthat Ġ$ k ^ {\ prime Ġ} $ Ġdoes Ġnot Ġpreced e Ġ$ k $ Ġand Ġthe Ġanswer Ġspan Ġis Ġnot Ġlonger Ġthan Ġ500 Ġcharacters .