No
BOOLEAN
No
BOOLEAN
In this experiment set, our BERT implementation is compared to several systems that participated in the MEDDOCAN challenge: a CRF classifier BIBREF18, a spaCy entity recogniser BIBREF18, and NLNDE BIBREF12, the winner of the shared task and current state of the art for sensitive information detection and classification in Spanish clinical text. Specifically, we include the results of a domain-independent NLNDE model (S2), and the results of a model enriched with domain-specific embeddings (S3). Finally, we include the results obtained by mao2019hadoken with a CRF output layer on top of BERT embeddings. MEDDOCAN consists of two scenarios: With regard to the winner of the MEDDOCAN shared task, the BERT-based model has not improved the scores obtained by neither the domain-dependent (S3) nor the domain-independent (S2) NLNDE model. However, attending to the obtained results, BERT remains only 0.3 F1-score points behind, and would have achieved the second position among all the MEDDOCAN shared task competitors. Taking into account that only 3% of the gold labels remain incorrectly annotated, the task can be considered almost solved, and it is not clear if the differences among the systems are actually significant, or whether they stem from minor variations in initialisation or a long-tail of minor labelling inconsistencies.
The results of the experiments show that, in NUBes-PHI, the BERT-based model outperforms the other systems without requiring any adaptation or domain-specific feature engineering, just by being trained on the provided labelled data. Interestingly, the BERT-based model obtains a remarkably higher recall than the other systems. High recall is a desirable outcome because, when anonymising sensible documents, the accidental leak of sensible data is likely to be more dangerous than the unintended over-obfuscation of non-sensitive text. The experiments with the MEDDOCAN 2019 shared task dataset follow the same pattern. In this case, the BERT-based model falls 0.3 F1-score points behind the shared task winning system, but it would have achieved the second position in the competition with no further refinement.
2003.03106
Introduction </s> During Ġthe Ġfirst Ġtwo Ġdecades Ġof Ġthe Ġ21 st Ġcentury , Ġthe Ġsharing Ġand Ġprocessing Ġof Ġvast Ġamounts Ġof Ġdata Ġhas Ġbecome Ġpervasive . ĠThis Ġexpansion Ġof Ġdata Ġsharing Ġand Ġprocessing Ġcapabilities Ġis Ġboth Ġa Ġblessing Ġand Ġa Ġcurse . ĠData Ġhelps Ġbuild Ġbetter Ġinformation Ġsystems Ġfor Ġthe Ġdigital Ġera Ġand Ġenables Ġfurther Ġresearch Ġfor Ġadvanced Ġdata Ġmanagement Ġthat Ġbenefits Ġthe Ġsociety Ġin Ġgeneral . ĠBut Ġthe Ġuse Ġof Ġthis Ġvery Ġdata Ġcontaining Ġsensitive Ġinformation Ġconflicts Ġwith Ġprivate Ġdata Ġprotection , Ġboth Ġfrom Ġan Ġethical Ġand Ġa Ġlegal Ġperspective . </s> There Ġare Ġseveral Ġapplication Ġdomains Ġon Ġwhich Ġthis Ġsituation Ġis Ġparticularly Ġacute . ĠThis Ġis Ġthe Ġcase Ġof Ġthe Ġmedical Ġdomain ĠB IB REF 0 . ĠThere Ġare Ġplenty Ġof Ġpotential Ġapplications Ġfor Ġadvanced Ġmedical Ġdata Ġmanagement Ġthat Ġcan Ġonly Ġbe Ġresearched Ġand Ġdeveloped Ġusing Ġreal Ġdata ; Ġyet , Ġthe Ġuse Ġof Ġmedical Ġdata Ġis Ġseverely Ġlimited ĠâĢĵ when Ġnot Ġentirely Ġprohibited âĢĵ Ġdue Ġto Ġdata Ġprivacy Ġprotection Ġpolicies . </s> One Ġway Ġof Ġcircumvent ing Ġthis Ġproblem Ġis Ġto Ġanonym ise Ġthe Ġdata Ġby Ġremoving , Ġreplacing Ġor Ġobfusc ating Ġthe Ġpersonal Ġinformation Ġmentioned , Ġas Ġexempl ified Ġin ĠTable ĠT AB REF 1 . ĠThis Ġtask Ġcan Ġbe Ġdone Ġby Ġhand , Ġhaving Ġpeople Ġread Ġand Ġanonym ise Ġthe Ġdocuments Ġone Ġby Ġone . ĠDespite Ġbeing Ġa Ġreliable Ġand Ġsimple Ġsolution , Ġthis Ġapproach Ġis Ġtedious , Ġexpensive , Ġtime Ġconsuming Ġand Ġdifficult Ġto Ġscale Ġto Ġthe Ġpotentially Ġthousands Ġor Ġmillions Ġof Ġdocuments Ġthat Ġneed Ġto Ġbe Ġanonym ised . </s> For Ġthis Ġreason , Ġnumerous Ġof Ġsystems Ġand Ġapproaches Ġhave Ġbeen Ġdeveloped Ġduring Ġthe Ġlast Ġdecades Ġto Ġattempt Ġto Ġautomate Ġthe Ġanonym isation Ġof Ġsensitive Ġcontent , Ġstarting Ġwith Ġthe Ġautomatic Ġdetection Ġand Ġclassification Ġof Ġsensitive Ġinformation . ĠSome Ġof Ġthese Ġsystems Ġrely Ġon Ġrules , Ġpatterns Ġand Ġdiction aries , Ġwhile Ġothers Ġuse Ġmore Ġadvanced Ġtechniques Ġrelated Ġto Ġmachine Ġlearning Ġand , Ġmore Ġrecently , Ġdeep Ġlearning . </s> Given Ġthat Ġthis Ġpaper Ġis Ġconcerned Ġwith Ġtext Ġdocuments Ġ( e . g . Ġmedical Ġrecords ), Ġthe Ġinvolved Ġtechniques Ġare Ġrelated Ġto ĠNatural ĠLanguage ĠProcessing Ġ( N LP ). ĠWhen Ġusing ĠN LP Ġapproaches , Ġit Ġis Ġcommon Ġto Ġpose Ġthe Ġproblem Ġof Ġdocument Ġanonym isation Ġas Ġa Ġsequence Ġlab elling Ġproblem , Ġi . e . Ġclass ifying Ġeach Ġtoken Ġwithin Ġa Ġsequence Ġas Ġbeing Ġsensitive Ġinformation Ġor Ġnot . ĠFurther , Ġdepending Ġon Ġthe Ġobjective Ġof Ġthe Ġanonym isation Ġtask , Ġit Ġis Ġalso Ġimportant Ġto Ġdetermine Ġthe Ġtype Ġof Ġsensitive Ġinformation Ġ( names Ġof Ġindividuals , Ġaddresses , Ġage , Ġsex , Ġetc .). </s> The Ġanonym isation Ġsystems Ġbased Ġon ĠN LP Ġtechniques Ġperform Ġreasonably Ġwell , Ġbut Ġare Ġfar Ġfrom Ġperfect . ĠDepending Ġon Ġthe Ġdifficulty Ġposed Ġby Ġeach Ġdataset Ġor Ġthe Ġamount Ġof Ġavailable Ġdata Ġfor Ġtraining Ġmachine Ġlearning Ġmodels , Ġthe Ġperformance Ġachieved Ġby Ġthese Ġmethods Ġis Ġnot Ġenough Ġto Ġfully Ġrely Ġon Ġthem Ġin Ġcertain Ġsituations ĠB IB REF 0 . ĠHowever , Ġin Ġthe Ġlast Ġtwo Ġyears , Ġthe ĠN LP Ġcommunity Ġhas Ġreached Ġan Ġimportant Ġmilestone Ġthanks Ġto Ġthe Ġappearance Ġof Ġthe Ġso - called ĠTransformers Ġneural Ġnetwork Ġarchitectures ĠB IB REF 1 . ĠIn Ġthis Ġpaper , Ġwe Ġconduct Ġseveral Ġexperiments Ġin Ġsensitive Ġinformation Ġdetection Ġand Ġclassification Ġon ĠSpanish Ġclinical Ġtext Ġusing ĠB ERT Ġ( from Ġ` B id irection al ĠEnc oder ĠRepresent ations Ġfrom ĠTransformers ') ĠB IB REF 2 Ġas Ġthe Ġbase Ġfor Ġa Ġsequence Ġlab elling Ġapproach . ĠThe Ġexperiments Ġare Ġcarried Ġout Ġon Ġtwo Ġdatasets : Ġthe ĠMED DOC AN : ĠMedical ĠDocument ĠAn onym ization Ġshared Ġtask Ġdataset ĠB IB REF 3 , Ġand ĠN UB es ĠB IB REF 4 , Ġa Ġcorpus Ġof Ġreal Ġmedical Ġreports Ġin ĠSpanish . ĠIn Ġthese Ġexperiments , Ġwe Ġcompare Ġthe Ġperformance Ġof ĠB ERT Ġwith Ġother Ġmachine - learning - based Ġsystems , Ġsome Ġof Ġwhich Ġuse Ġlanguage - specific Ġfeatures . ĠOur Ġaim Ġis Ġto Ġevaluate Ġhow Ġgood Ġa ĠB ERT - based Ġmodel Ġperforms Ġwithout Ġlanguage Ġnor Ġdomain Ġspecial isation Ġapart Ġfrom Ġthe Ġtraining Ġdata Ġlabelled Ġfor Ġthe Ġtask Ġat Ġhand . </s> The Ġrest Ġof Ġthe Ġpaper Ġis Ġstructured Ġas Ġfollows : Ġthe Ġnext Ġsection Ġdescribes Ġrelated Ġwork Ġabout Ġdata Ġanonym isation Ġin Ġgeneral Ġand Ġclinical Ġdata Ġanonym isation Ġin Ġparticular ; Ġit Ġalso Ġprovides Ġa Ġmore Ġdetailed Ġexplanation Ġand Ġbackground Ġabout Ġthe ĠTransformers Ġarchitecture Ġand ĠB ERT . ĠSection ĠSEC REF 3 Ġdescribes Ġthe Ġdata Ġinvolved Ġin Ġthe Ġexperiments Ġand Ġthe Ġsystems Ġevaluated Ġin Ġthis Ġpaper , Ġincluding Ġthe ĠB ERT - based Ġsystem ; Ġfinally , Ġit Ġdetails Ġthe Ġexperimental Ġdesign . ĠSection ĠSEC REF 4 Ġintroduces Ġthe Ġresults Ġfor Ġeach Ġset Ġof Ġexperiments . ĠFinally , ĠSection ĠSEC REF 5 Ġcontains Ġthe Ġconclusions Ġand Ġfuture Ġlines Ġof Ġwork . </s> Related ĠWork </s> The Ġstate Ġof Ġthe Ġart Ġin Ġthe Ġfield Ġof ĠNatural ĠLanguage ĠProcessing Ġ( N LP ) Ġhas Ġreached Ġan Ġimportant Ġmilestone Ġin Ġthe Ġlast Ġcouple Ġof Ġyears Ġthanks Ġto Ġdeep - learning Ġarchitectures , Ġincreasing Ġin Ġseveral Ġpoints Ġthe Ġperformance Ġof Ġnew Ġmodels Ġfor Ġalmost Ġany Ġtext Ġprocessing Ġtask . </s> The Ġmajor Ġchange Ġstarted Ġwith Ġthe ĠTransformers Ġmodel Ġproposed Ġby Ġvas w ani 2017 att ention . ĠIt Ġsubstituted Ġthe Ġwidely Ġused Ġrecurrent Ġand Ġconv olution al Ġneural Ġnetwork Ġarchitectures Ġby Ġanother Ġapproach Ġbased Ġsolely Ġon Ġself - att ention , Ġobtaining Ġan Ġimpressive Ġperformance Ġgain . ĠThe Ġoriginal Ġproposal Ġwas Ġfocused Ġon Ġan Ġenc oder - dec oder Ġarchitecture Ġfor Ġmachine Ġtranslation , Ġbut Ġsoon Ġthe Ġuse Ġof ĠTransformers Ġwas Ġmade Ġmore Ġgeneral ĠB IB REF 1 . ĠThere Ġare Ġseveral Ġother Ġpopular Ġmodels Ġthat Ġuse ĠTransformers , Ġsuch Ġas ĠOpen ĠAI 's ĠG PT Ġand ĠG PT 2 ĠB IB REF 5 , ĠRo BER Ta ĠB IB REF 6 Ġand Ġthe Ġmost Ġrecent ĠXL Net ĠB IB REF 7 ; Ġstill , ĠB ERT ĠB IB REF 2 Ġis Ġone Ġof Ġthe Ġmost Ġwidespread ĠTrans former - based Ġmodels . </s> BER T Ġtrains Ġits Ġun super vised Ġlanguage Ġmodel Ġusing Ġa ĠMask ed ĠLanguage ĠModel Ġand ĠNext ĠSent ence ĠPrediction . ĠA Ġcommon Ġproblem Ġin ĠN LP Ġis Ġthe Ġlack Ġof Ġenough Ġtraining Ġdata . ĠB ERT Ġcan Ġbe Ġpre - trained Ġto Ġlearn Ġgeneral Ġor Ġspecific Ġlanguage Ġmodels Ġusing Ġvery Ġlarge Ġamounts Ġof Ġunl ab elled Ġtext Ġ( e . g . Ġweb Ġcontent , ĠWikipedia , Ġetc .), Ġand Ġthis Ġknowledge Ġcan Ġbe Ġtransferred Ġto Ġa Ġdifferent Ġdownstream Ġtask Ġin Ġa Ġprocess Ġthat Ġreceives Ġthe Ġname Ġfine - tun ing . </s> dev lin 2018 bert Ġhave Ġused Ġfine - tun ing Ġto Ġachieve Ġstate - of - the - art Ġresults Ġon Ġa Ġwide Ġvariety Ġof Ġchallenging Ġnatural Ġlanguage Ġtasks , Ġsuch Ġas Ġtext Ġclassification , ĠQuestion ĠAn sw ering Ġ( Q A ) Ġand ĠNamed ĠEntity ĠRecogn ition Ġand ĠClassification Ġ( NER C ). ĠB ERT Ġhas Ġalso Ġbeen Ġused Ġsuccessfully Ġby Ġother Ġcommunity Ġpractitioners Ġfor Ġa Ġwide Ġrange Ġof ĠN LP - related Ġtasks ĠB IB REF 8 , ĠB IB REF 9 . </s> Regarding Ġthe Ġtask Ġof Ġdata Ġanonym isation Ġin Ġparticular , Ġanonym isation Ġsystems Ġmay Ġfollow Ġdifferent Ġapproaches Ġand Ġpursue Ġdifferent Ġobjectives Ġ( C orm ode Ġand ĠS riv ast ava , Ġ2009 ). ĠThe Ġfirst Ġobjective Ġof Ġthese Ġsystems Ġis Ġto Ġdetect Ġand Ġclassify Ġthe Ġsensitive Ġinformation Ġcontained Ġin Ġthe Ġdocuments Ġto Ġbe Ġanonym ised . ĠIn Ġorder Ġto Ġachieve Ġthat , Ġthey Ġuse Ġrule - based Ġapproaches , ĠMachine ĠLearning Ġ( ML ) Ġapproaches , Ġor Ġa Ġcombination Ġof Ġboth . </s> Although Ġmost Ġof Ġthese Ġefforts Ġare Ġfor ĠEnglish Ġtexts ĠâĢĵ see , Ġamong Ġothers , Ġthe Ġi 2 b 2 Ġde - ident ification Ġchallenges ĠB IB REF 10 , ĠB IB REF 11 , Ġd ern on 2016 deep , Ġor Ġk hin 2018 deep âĢĵ , Ġother Ġlanguages Ġare Ġalso Ġattracting Ġgrowing Ġinterest . ĠSome Ġexamples Ġare Ġm amed e 2016 aut om ated Ġfor ĠPortuguese Ġand Ġt ve it 2004 an onym ization Ġfor ĠNorwegian . ĠWith Ġrespect Ġto Ġthe Ġanonym isation Ġof Ġtext Ġwritten Ġin ĠSpanish , Ġrecent Ġstudies Ġinclude Ġmed ina 2018 building , Ġhas san 2018 anon im iz acion Ġand Ġgar cia 2018 aut om ating . ĠMost Ġnotably , Ġin Ġ2019 Ġthe Ġfirst Ġcommunity Ġchallenge Ġabout Ġanonym isation Ġof Ġmedical Ġdocuments Ġin ĠSpanish , ĠMED DOC AN ĠB IB REF 3 , Ġwas Ġheld Ġas Ġpart Ġof Ġthe ĠI ber LE F Ġinitiative . ĠThe Ġwinners Ġof Ġthe Ġchallenge ĠâĢĵ the ĠNeither - Language - nor - Domain - Experts Ġ( NL N DE ) ĠB IB REF 12 âĢĵ Ġachieved ĠF 1 - sc ores Ġas Ġhigh Ġas Ġ0 . 975 Ġin Ġthe Ġtask Ġof Ġsensitive Ġinformation Ġdetection Ġand Ġcategor isation Ġby Ġusing Ġrecurrent Ġneural Ġnetworks Ġwith ĠCond itional ĠRandom ĠField Ġ( CR F ) Ġoutput Ġlayers . </s> At Ġthe Ġsame Ġchallenge , Ġm ao 2019 had oken Ġoccupied Ġthe Ġ8 th Ġposition Ġamong Ġ18 Ġparticipants Ġusing ĠB ERT . ĠAccording Ġto Ġthe Ġdescription Ġof Ġthe Ġsystem , Ġthe Ġauthors Ġused ĠB ERT - Base ĠMult ilingual ĠC ased Ġand Ġan Ġoutput ĠCR F Ġlayer . ĠHowever , Ġtheir Ġsystem Ġis Ġ$\ sim Ġ$ 3 ĠF 1 - score Ġpoints Ġbelow Ġour Ġimplementation Ġwithout Ġthe ĠCR F Ġlayer . </s> Materials Ġand ĠMethods </s> The Ġaim Ġof Ġthis Ġpaper Ġis Ġto Ġevaluate ĠB ERT 's Ġmult ilingual Ġmodel Ġand Ġcompare Ġit Ġto Ġother Ġestablished Ġmachine - learning Ġalgorithms Ġin Ġa Ġspecific Ġtask : Ġsensitive Ġdata Ġdetection Ġand Ġclassification Ġin ĠSpanish Ġclinical Ġfree Ġtext . ĠThis Ġsection Ġdescribes Ġthe Ġdata Ġinvolved Ġin Ġthe Ġexperiments Ġand Ġthe Ġsystems Ġevaluated . ĠFinally , Ġwe Ġintroduce Ġthe Ġexperimental Ġsetup . </s> Materials Ġand ĠMethods Ġ: :: ĠData </s> Two Ġdatasets Ġare Ġexploited Ġin Ġthis Ġarticle . ĠBoth Ġdatasets Ġconsist Ġof Ġplain Ġtext Ġcontaining Ġclinical Ġnarrative Ġwritten Ġin ĠSpanish , Ġand Ġtheir Ġrespective Ġmanual Ġannotations Ġof Ġsensitive Ġinformation Ġin ĠBR AT ĠB IB REF 13 Ġstandoff Ġformat . ĠIn Ġorder Ġto Ġfeed Ġthe Ġdata Ġto Ġthe Ġdifferent Ġalgorithms Ġpresented Ġin ĠSection ĠSEC REF 7 , Ġthese Ġdatasets Ġwere Ġtransformed Ġto Ġcomply Ġwith Ġthe Ġcommonly Ġused ĠB IO Ġsequence Ġrepresentation Ġscheme ĠB IB REF 14 . </s> Materials Ġand ĠMethods Ġ: :: ĠData Ġ: :: ĠN UB es - PH I </s> N UB es ĠB IB REF 4 Ġis Ġa Ġcorpus Ġof Ġaround Ġ7 , 000 Ġreal Ġmedical Ġreports Ġwritten Ġin ĠSpanish Ġand Ġannot ated Ġwith Ġneg ation Ġand Ġuncertainty Ġinformation . ĠBefore Ġbeing Ġpublished , Ġsensitive Ġinformation Ġhad Ġto Ġbe Ġmanually Ġannot ated Ġand Ġreplaced Ġfor Ġthe Ġcorpus Ġto Ġbe Ġsafely Ġshared . ĠIn Ġthis Ġarticle , Ġwe Ġwork Ġwith Ġthe ĠN UB es Ġversion Ġprior Ġto Ġits Ġanonym isation , Ġthat Ġis , Ġwith Ġthe Ġmanual Ġannotations Ġof Ġsensitive Ġinformation . ĠIt Ġfollows Ġthat Ġthe Ġversion Ġwe Ġwork Ġwith Ġis Ġnot Ġpublicly Ġavailable Ġand , Ġdue Ġto Ġcontractual Ġrestrictions , Ġwe Ġcannot Ġreveal Ġthe Ġproven ance Ġof Ġthe Ġdata . ĠIn Ġorder Ġto Ġavoid Ġconfusion Ġbetween Ġthe Ġtwo Ġcorpus Ġversions , Ġwe Ġhence forth Ġrefer Ġto Ġthe Ġversion Ġrelevant Ġin Ġthis Ġpaper Ġas ĠN UB es - PH I Ġ( from Ġ` N UB es Ġwith ĠPersonal ĠHealth ĠInformation '). </s> N UB es - PH I Ġconsists Ġof Ġ32 , 055 Ġsentences Ġannot ated Ġfor Ġ11 Ġdifferent Ġsensitive Ġinformation Ġcategories . ĠOverall , Ġit Ġcontains Ġ7 , 8 18 Ġannotations . ĠThe Ġcorpus Ġhas Ġbeen Ġrandomly Ġsplit Ġinto Ġtrain Ġ( 72 %), Ġdevelopment Ġ( 8 %) Ġand Ġtest Ġ( 20 %) Ġsets Ġto Ġconduct Ġthe Ġexperiments Ġdescribed Ġin Ġthis Ġpaper . ĠThe Ġsize Ġof Ġeach Ġsplit Ġand Ġthe Ġdistribution Ġof Ġthe Ġannotations Ġcan Ġbe Ġconsulted Ġin ĠTables Ġand Ġ, Ġrespectively . </s> The Ġmajority Ġof Ġsensitive Ġinformation Ġin ĠN UB es - PH I Ġare Ġtemporal Ġexpressions Ġ( ` Date ' Ġand Ġ` Time '), Ġfollowed Ġby Ġhealthcare Ġfacility Ġmentions Ġ( ` H ospital '), Ġand Ġthe Ġage Ġof Ġthe Ġpatient . ĠMent ions Ġof Ġpeople Ġare Ġnot Ġthat Ġfrequent , Ġwith Ġphysician Ġnames Ġ( ` Doctor ') Ġoccurring Ġmuch Ġmore Ġoften Ġthan Ġpatient Ġnames Ġ( ` Pat ient '). ĠThe Ġleast Ġfrequent Ġsensitive Ġinformation Ġtypes , Ġwhich Ġaccount Ġfor Ġ$\ sim Ġ$ 10 % Ġof Ġthe Ġremaining Ġannotations , Ġconsist Ġof Ġthe Ġpatient 's Ġsex , Ġjob , Ġand Ġkins hip , Ġand Ġlocations Ġother Ġthan Ġhealthcare Ġfacilities Ġ( ` Location '). ĠFinally , Ġthe Ġtag Ġ` Other ' Ġincludes , Ġfor Ġinstance , Ġmentions Ġto Ġinstitutions Ġunrelated Ġto Ġhealthcare Ġand Ġwhether Ġthe Ġpatient Ġis Ġright - Ġor Ġleft - handed . ĠIt Ġoccurs Ġjust Ġ36 Ġtimes . </s> Materials Ġand ĠMethods Ġ: :: ĠData Ġ: :: ĠThe ĠMED DOC AN Ġcorpus </s> The Ġorganisers Ġof Ġthe ĠMED DOC AN Ġshared Ġtask ĠB IB REF 3 Ġcurated Ġa Ġsynthetic Ġcorpus Ġof Ġclinical Ġcases Ġenriched Ġwith Ġsensitive Ġinformation Ġby Ġhealth Ġdocument al ists . ĠIn Ġthis Ġregard , Ġthe ĠMED DOC AN Ġevaluation Ġscenario Ġcould Ġbe Ġsaid Ġto Ġbe Ġsomewhat Ġfar Ġfrom Ġthe Ġreal Ġuse Ġcase Ġthe Ġtechnology Ġdeveloped Ġfor Ġthe Ġshared Ġtask Ġis Ġsupposed Ġto Ġbe Ġapplied Ġin . ĠHowever , Ġat Ġthe Ġmoment Ġit Ġalso Ġprovides Ġthe Ġonly Ġpublic Ġmeans Ġfor Ġa Ġrigorous Ġcomparison Ġbetween Ġsystems Ġfor Ġsensitive Ġhealth Ġinformation Ġdetection Ġin ĠSpanish Ġtexts . </s> The Ġsize Ġof Ġthe ĠMED DOC AN Ġcorpus Ġis Ġshown Ġin ĠTable Ġ. ĠCompared Ġto ĠN UB es - PH I Ġ( Table Ġ), Ġthis Ġcorpus Ġcontains Ġmore Ġsensitive Ġinformation Ġannotations , Ġboth Ġin Ġabsolute Ġand Ġrelative Ġterms . </s> The Ġsensitive Ġannotation Ġcategories Ġconsidered Ġin ĠMED DOC AN Ġdiffer Ġin Ġpart Ġfrom Ġthose Ġin ĠN UB es - PH I . ĠMost Ġnotably , Ġit Ġcontains Ġfiner - gr ained Ġlabels Ġfor Ġlocation - related Ġmentions ĠâĢĵ name ly , Ġ` Address ', Ġ` Ter rit ory ', Ġand Ġ` Country ' âĢĵ , Ġand Ġother Ġsensitive Ġinformation Ġcategories Ġthat Ġwe Ġdid Ġnot Ġencounter Ġin ĠN UB es - PH I Ġ( e . g ., Ġidentifiers , Ġphone Ġnumbers , Ġe - mail Ġaddresses , Ġetc .). ĠIn Ġtotal , Ġthe ĠMED DOC AN Ġcorpus Ġhas Ġ21 Ġsensitive Ġinformation Ġcategories . ĠWe Ġrefer Ġthe Ġreader Ġto Ġthe Ġorganisers ' Ġarticle ĠB IB REF 3 Ġfor Ġmore Ġdetailed Ġinformation Ġabout Ġthis Ġcorpus . </s> Materials Ġand ĠMethods Ġ: :: ĠSystems </s> Apart Ġfrom Ġexperimenting Ġwith Ġa Ġpre - trained ĠB ERT Ġmodel , Ġwe Ġhave Ġrun Ġexperiments Ġwith Ġother Ġsystems Ġand Ġbas elines , Ġto Ġcompare Ġthem Ġand Ġobtain Ġa Ġbetter Ġperspective Ġabout ĠB ERT 's Ġperformance Ġin Ġthese Ġdatasets . </s> Materials Ġand ĠMethods Ġ: :: ĠSystems Ġ: :: ĠBas eline </s> As Ġthe Ġsimplest Ġbaseline , Ġa Ġsensitive Ġdata Ġrecogn iser Ġand Ġclass ifier Ġhas Ġbeen Ġdeveloped Ġthat Ġconsists Ġof Ġregular - express ions Ġand Ġdictionary Ġlook - ups . ĠFor Ġeach Ġcategory Ġto Ġdetect Ġa Ġspecific Ġmethod Ġhas Ġbeen Ġimplemented . ĠFor Ġinstance , Ġthe ĠDate , ĠAge , ĠTime Ġand ĠDoctor Ġdetectors Ġare Ġbased Ġon Ġregular - express ions ; ĠHospital , ĠSex , ĠK ins hip , ĠLocation , ĠPatient Ġand ĠJob Ġare Ġlooked Ġup Ġin Ġdiction aries . ĠThe Ġdiction aries Ġare Ġhand - crafted Ġfrom Ġthe Ġtraining Ġdata Ġavailable , Ġexcept Ġfor Ġthe ĠPatient 's Ġcase , Ġfor Ġwhich Ġthe Ġpossible Ġcandidates Ġconsidered Ġare Ġthe Ġ100 Ġmost Ġcommon Ġfemale Ġand Ġmale Ġnames Ġin ĠSpain Ġaccording Ġto Ġthe ĠInstit uto ĠN ac ional Ġde ĠEst ad ÃŃ st ica Ġ( INE ; ĠSpanish ĠStatistical ĠOffice ). </s> Materials Ġand ĠMethods Ġ: :: ĠSystems Ġ: :: ĠCR F </s> Cond itional ĠRandom ĠFields Ġ( CR F ) ĠB IB REF 15 Ġhave Ġbeen Ġextensively Ġused Ġfor Ġtasks Ġof Ġsequential Ġnature . ĠIn Ġthis Ġpaper , Ġwe Ġpropose Ġas Ġone Ġof Ġthe Ġcompetitive Ġbas elines Ġa ĠCR F Ġclass ifier Ġtrained Ġwith Ġsk learn - cr f su ite Ġfor ĠPython Ġ3 . 5 Ġand Ġthe Ġfollowing Ġconfiguration : Ġalgorithm Ġ= Ġlb f gs ; Ġmaximum Ġiterations Ġ= Ġ100 ; Ġc 1 Ġ= Ġc 2 Ġ= Ġ0 . 1 ; Ġall Ġtransitions Ġ= Ġtrue ; Ġoptim ise Ġ= Ġfalse . ĠThe Ġfeatures Ġextracted Ġfrom Ġeach Ġtoken Ġare Ġas Ġfollows : </s> [ no item se p ] </s> prefix es Ġand Ġsuffix es Ġof Ġ2 Ġand Ġ3 Ġcharacters ; </s> the Ġlength Ġof Ġthe Ġtoken Ġin Ġcharacters Ġand Ġthe Ġlength Ġof Ġthe Ġsentence Ġin Ġtokens ; </s> whether Ġthe Ġtoken Ġis Ġall - letters , Ġa Ġnumber , Ġor Ġa Ġsequence Ġof Ġpunct uation Ġmarks ; </s> whether Ġthe Ġtoken Ġcontains Ġthe Ġcharacter Ġ` @ '; </s> whether Ġthe Ġtoken Ġis Ġthe Ġstart Ġor Ġend Ġof Ġthe Ġsentence ; </s> the Ġtoken 's Ġcasing Ġand Ġthe Ġratio Ġof Ġu pperc ase Ġcharacters , Ġdigits , Ġand Ġpunct uation Ġmarks Ġto Ġits Ġlength ; </s> and , Ġthe Ġle mma , Ġpart - of - speech Ġtag , Ġand Ġnamed - entity Ġtag Ġgiven Ġby Ġ ix a - p ipes ĠB IB REF 16 Ġupon Ġanalys ing Ġthe Ġsentence Ġthe Ġtoken Ġbelongs Ġto . </s> Notice ably , Ġnone Ġof Ġthe Ġfeatures Ġused Ġto Ġtrain Ġthe ĠCR F Ġclass ifier Ġis Ġdomain - dependent . ĠHowever , Ġthe Ġlatter Ġgroup Ġof Ġfeatures Ġis Ġlanguage Ġdependent . </s> Materials Ġand ĠMethods Ġ: :: ĠSystems Ġ: :: Ġspa Cy </s> sp a Cy Ġis Ġa Ġwidely Ġused ĠN LP Ġlibrary Ġthat Ġimplements Ġstate - of - the - art Ġtext Ġprocessing Ġpipelines , Ġincluding Ġa Ġsequence - lab elling Ġpipeline Ġsimilar Ġto Ġthe Ġone Ġdescribed Ġby Ġstr ub ell 2017 fast . Ġspa Cy Ġoffers Ġseveral Ġpre - trained Ġmodels Ġin ĠSpanish , Ġwhich Ġperform Ġbasic ĠN LP Ġtasks Ġsuch Ġas ĠNamed ĠEntity ĠRecogn ition Ġ( NER ). ĠIn Ġthis Ġpaper , Ġwe Ġhave Ġtrained Ġa Ġnew ĠN ER Ġmodel Ġto Ġdetect ĠN UB es - PH I Ġlabels . ĠFor Ġthis Ġpurpose , Ġthe Ġnew Ġmodel Ġuses Ġall Ġthe Ġlabels Ġof Ġthe Ġtraining Ġcorpus Ġcoded Ġwith Ġits Ġcontext Ġat Ġsentence Ġlevel . ĠThe Ġnetwork Ġoptim isation Ġparameters Ġand Ġdrop out Ġvalues Ġare Ġthe Ġones Ġrecommended Ġin Ġthe Ġdocumentation Ġfor Ġsmall Ġdatasets . ĠFinally , Ġthe Ġmodel Ġis Ġtrained Ġusing Ġbatches Ġof Ġsize Ġ64 . ĠNo Ġmore Ġfeatures Ġare Ġincluded , Ġso Ġthe Ġclass ifier Ġis Ġlanguage - dependent Ġbut Ġnot Ġdomain - dependent . </s> Materials Ġand ĠMethods Ġ: :: ĠSystems Ġ: :: ĠB ERT </s> As Ġintroduced Ġearlier , ĠB ERT Ġhas Ġshown Ġan Ġoutstanding Ġperformance Ġin ĠN ERC - like Ġtasks , Ġimproving Ġthe Ġstart - of - the - art Ġresults Ġfor Ġalmost Ġevery Ġdataset Ġand Ġlanguage . ĠWe Ġtake Ġthe Ġsame Ġapproach Ġhere , Ġby Ġusing Ġthe Ġmodel ĠB ERT - Base ĠMult ilingual ĠC ased Ġwith Ġa ĠFully ĠConnect ed Ġ( FC ) Ġlayer Ġon Ġtop Ġto Ġperform Ġa Ġfine - tun ing Ġof Ġthe Ġwhole Ġmodel Ġfor Ġan Ġanonym isation Ġtask Ġin ĠSpanish Ġclinical Ġdata . ĠOur Ġimplementation Ġis Ġbuilt Ġon ĠPy Tor ch Ġand Ġthe ĠPy Tor ch - Transform ers Ġlibrary ĠB IB REF 1 . ĠThe Ġtraining Ġphase Ġconsists Ġin Ġthe Ġfollowing Ġsteps Ġ( rough ly Ġdepicted Ġin ĠFigure Ġ): </s> Pre - processing : Ġsince Ġwe Ġare Ġrelying Ġon Ġa Ġpre - trained ĠB ERT Ġmodel , Ġwe Ġmust Ġmatch Ġthe Ġsame Ġconfiguration Ġby Ġusing Ġa Ġspecific Ġtoken isation Ġand Ġvocabulary . ĠB ERT Ġalso Ġneeds Ġthat Ġthe Ġinputs Ġcontains Ġspecial Ġtokens Ġto Ġsignal Ġthe Ġbeginning Ġand Ġthe Ġend Ġof Ġeach Ġsequence . </s> Fine - tun ing : Ġthe Ġpre - process ed Ġsequence Ġis Ġfed Ġinto Ġthe Ġmodel . ĠB ERT Ġoutputs Ġthe Ġcontextual Ġembed d ings Ġthat Ġencode Ġeach Ġof Ġthe Ġinput ted Ġtokens . ĠThis Ġembed ding Ġrepresentation Ġfor Ġeach Ġtoken Ġis Ġfed Ġinto Ġthe ĠFC Ġlinear Ġlayer Ġafter Ġa Ġdrop out Ġlayer Ġ( with Ġa Ġ0 . 1 Ġdrop out Ġprobability ), Ġwhich Ġin Ġturn Ġoutputs Ġthe Ġlog its Ġfor Ġeach Ġpossible Ġclass . ĠThe Ġcross - ent ropy Ġloss Ġfunction Ġis Ġcalculated Ġcomparing Ġthe Ġlog its Ġand Ġthe Ġgold Ġlabels , Ġand Ġthe Ġerror Ġis Ġback - prop ag ated Ġto Ġadjust Ġthe Ġmodel Ġparameters . </s> We Ġhave Ġtrained Ġthe Ġmodel Ġusing Ġan ĠAdam W Ġoptim iser ĠB IB REF 17 Ġwith Ġthe Ġlearning Ġrate Ġset Ġto Ġ3 e - 5 , Ġas Ġrecommended Ġby Ġdev lin 2018 bert , Ġand Ġwith Ġa Ġgradient Ġclipping Ġof Ġ1 . 0 . ĠWe Ġalso Ġapplied Ġa Ġlearning - rate Ġsched uler Ġthat Ġwar ms Ġup Ġthe Ġlearning Ġrate Ġfrom Ġzero Ġto Ġits Ġmaximum Ġvalue Ġas Ġthe Ġtraining Ġprogresses , Ġwhich Ġis Ġalso Ġa Ġcommon Ġpractice . ĠFor Ġeach Ġexperiment Ġset Ġproposed Ġbelow , Ġthe Ġtraining Ġwas Ġrun Ġwith Ġan Ġearly - st opping Ġpatience Ġof Ġ15 Ġepoch s . ĠThen , Ġthe Ġmodel Ġthat Ġperformed Ġbest Ġagainst Ġthe Ġdevelopment Ġset Ġwas Ġused Ġto Ġproduce Ġthe Ġreported Ġresults . </s> The Ġexperiments Ġwere Ġrun Ġon Ġa Ġ64 - core Ġserver Ġwith Ġoperating Ġsystem ĠUbuntu Ġ16 . 04 , Ġ250 GB Ġof ĠRAM Ġmemory , Ġand Ġ4 ĠGeForce ĠRTX Ġ20 80 ĠGPUs Ġwith Ġ11 GB Ġof Ġmemory . ĠThe Ġmaximum Ġsequence Ġlength Ġwas Ġset Ġat Ġ500 Ġand Ġthe Ġbatch Ġsize Ġat Ġ12 . ĠIn Ġthis Ġsetting , Ġeach Ġepoch ĠâĢĵ a Ġfull Ġpass Ġthrough Ġall Ġthe Ġtraining Ġdata âĢĵ Ġrequired Ġabout Ġ10 Ġminutes Ġto Ġcomplete . </s> Materials Ġand ĠMethods Ġ: :: ĠExperimental Ġdesign </s> We Ġhave Ġconducted Ġexperiments Ġwith ĠB ERT Ġin Ġthe Ġtwo Ġdatasets Ġof ĠSpanish Ġclinical Ġnarrative Ġpresented Ġin ĠSection ĠSEC REF 3 ĠThe Ġfirst Ġexperiment Ġset Ġuses ĠN UB es - PH I , Ġa Ġcorpus Ġof Ġreal Ġmedical Ġreports Ġmanually Ġannot ated Ġwith Ġsensitive Ġinformation . ĠBecause Ġthis Ġcorpus Ġis Ġnot Ġpublicly Ġavailable , Ġand Ġin Ġorder Ġto Ġcompare Ġthe ĠB ERT - based Ġmodel Ġto Ġother Ġrelated Ġpublished Ġsystems , Ġthe Ġsecond Ġset Ġof Ġexperiments Ġuses Ġthe ĠMED DOC AN Ġ2019 Ġshared Ġtask Ġcompetition Ġdataset . ĠThe Ġfollowing Ġsections Ġprovide Ġgreater Ġdetail Ġabout Ġthe Ġtwo Ġexperimental Ġsetups . </s> Materials Ġand ĠMethods Ġ: :: ĠExperimental Ġdesign Ġ: :: ĠExperiment ĠA : ĠN UB es - PH I </s> In Ġthis Ġexperiment Ġset , Ġwe Ġevaluate Ġall Ġthe Ġsystems Ġpresented Ġin ĠSection ĠSEC REF 7 , Ġnamely , Ġthe Ġrule - based Ġbaseline , Ġthe ĠCR F Ġclass ifier , Ġthe Ġspa Cy Ġentity Ġtag ger , Ġand ĠB ERT . ĠThe Ġevaluation Ġcomprises Ġthree Ġscenarios Ġof Ġincreasing Ġdifficulty : </s> [ no item se p ] </s> - ĠEval uates Ġthe Ġperformance Ġof Ġthe Ġsystems Ġat Ġpredicting Ġwhether Ġeach Ġtoken Ġis Ġsensitive Ġor Ġnon - sensitive ; Ġthat Ġis , Ġthe Ġmeasurements Ġonly Ġtake Ġinto Ġaccount Ġwhether Ġa Ġsensitive Ġtoken Ġhas Ġbeen Ġrecognised Ġor Ġnot , Ġregardless Ġof Ġthe ĠB IO Ġlabel Ġand Ġthe Ġcategory Ġassigned . ĠThis Ġscenario Ġshows Ġhow Ġgood Ġa Ġsystem Ġwould Ġbe Ġat Ġobfusc ating Ġsensitive Ġdata Ġ( e . g ., Ġby Ġreplacing Ġsensitive Ġtokens Ġwith Ġaster isks ). </s> - ĠWe Ġmeasure Ġthe Ġperformance Ġof Ġthe Ġsystems Ġat Ġpredicting Ġthe Ġsensitive Ġinformation Ġtype Ġof Ġeach Ġtoken ĠâĢĵ i . e ., Ġthe Ġ11 Ġcategories Ġpresented Ġin ĠSection ĠSEC REF 5 Ġor Ġ` out '. ĠDetect ing Ġentity Ġtypes Ġcorrectly Ġis Ġimportant Ġif Ġa Ġsystem Ġis Ġgoing Ġto Ġbe Ġused Ġto Ġreplace Ġsensitive Ġdata Ġby Ġfake Ġdata Ġof Ġthe Ġsame Ġtype Ġ( e . g ., Ġrandom Ġpeople Ġnames ). </s> - ĠThis Ġis Ġthe Ġstrict est Ġevaluation , Ġas Ġit Ġtakes Ġinto Ġaccount Ġboth Ġthe ĠB IO Ġlabel Ġand Ġthe Ġcategory Ġassigned Ġto Ġeach Ġindividual Ġtoken . ĠBeing Ġable Ġto Ġdiscern Ġbetween Ġtwo Ġcontiguous Ġsensitive Ġentities Ġof Ġthe Ġsame Ġtype Ġis Ġrelevant Ġnot Ġonly Ġbecause Ġit Ġis Ġhelpful Ġwhen Ġproducing Ġfake Ġreplacements , Ġbut Ġbecause Ġit Ġalso Ġyields Ġmore Ġaccurate Ġstatistics Ġof Ġthe Ġsensitive Ġinformation Ġpresent Ġin Ġa Ġgiven Ġdocument Ġcollection . </s> The Ġsystems Ġare Ġevaluated Ġin Ġterms Ġof Ġmicro - average Ġprecision , Ġrecall Ġand ĠF 1 - score Ġin Ġall Ġthe Ġscenarios . </s> In Ġaddition Ġto Ġthe Ġscenarios Ġproposed , Ġa Ġsubject Ġworth Ġbeing Ġstudied Ġis Ġthe Ġneed Ġof Ġlabelled Ġdata . ĠMan ually Ġlabelled Ġdata Ġis Ġan Ġscarce Ġand Ġexpensive Ġresource , Ġwhich Ġfor Ġsome Ġapplication Ġdomains Ġor Ġlanguages Ġis Ġdifficult Ġto Ġcome Ġby . ĠIn Ġorder Ġto Ġobtain Ġan Ġestimation Ġof Ġthe Ġdependency Ġof Ġeach Ġsystem Ġon Ġthe Ġavailable Ġamount Ġof Ġtraining Ġdata , Ġwe Ġhave Ġret rained Ġall Ġthe Ġcompared Ġmodels Ġusing Ġdecreasing Ġamounts Ġof Ġdata ĠâĢĵ from Ġ100 % Ġof Ġthe Ġavailable Ġtraining Ġinstances Ġto Ġjust Ġ1 %. ĠThe Ġsame Ġdata Ġsubs ets Ġhave Ġbeen Ġused Ġto Ġtrain Ġall Ġthe Ġsystems . ĠDue Ġto Ġthe Ġknowledge Ġtransferred Ġfrom Ġthe Ġpre - trained ĠB ERT Ġmodel , Ġthe ĠB ERT - based Ġmodel Ġis Ġexpected Ġto Ġbe Ġmore Ġrobust Ġto Ġdata Ġscarcity Ġthan Ġthose Ġthat Ġstart Ġtheir Ġtraining Ġfrom Ġscratch . </s> Materials Ġand ĠMethods Ġ: :: ĠExperimental Ġdesign Ġ: :: ĠExperiment ĠB : ĠMED DOC AN </s> In Ġthis Ġexperiment Ġset , Ġour ĠB ERT Ġimplementation Ġis Ġcompared Ġto Ġseveral Ġsystems Ġthat Ġparticipated Ġin Ġthe ĠMED DOC AN Ġchallenge : Ġa ĠCR F Ġclass ifier ĠB IB REF 18 , Ġa Ġspa Cy Ġentity Ġrecogn iser ĠB IB REF 18 , Ġand ĠNL N DE ĠB IB REF 12 , Ġthe Ġwinner Ġof Ġthe Ġshared Ġtask Ġand Ġcurrent Ġstate Ġof Ġthe Ġart Ġfor Ġsensitive Ġinformation Ġdetection Ġand Ġclassification Ġin ĠSpanish Ġclinical Ġtext . ĠSpecifically , Ġwe Ġinclude Ġthe Ġresults Ġof Ġa Ġdomain - independent ĠNL N DE Ġmodel Ġ( S 2 ), Ġand Ġthe Ġresults Ġof Ġa Ġmodel Ġenriched Ġwith Ġdomain - specific Ġembed d ings Ġ( S 3 ). ĠFinally , Ġwe Ġinclude Ġthe Ġresults Ġobtained Ġby Ġm ao 2019 had oken Ġwith Ġa ĠCR F Ġoutput Ġlayer Ġon Ġtop Ġof ĠB ERT Ġembed d ings . ĠMED DOC AN Ġconsists Ġof Ġtwo Ġscenarios : </s> [ no item se p ] </s> - ĠThis Ġevaluation Ġmeasures Ġhow Ġgood Ġa Ġsystem Ġis Ġat Ġdetecting Ġsensitive Ġtext Ġspans , Ġregardless Ġof Ġthe Ġcategory Ġassigned Ġto Ġthem . </s> - ĠIn Ġthis Ġscenario , Ġsystems Ġare Ġrequired Ġto Ġmatch Ġexactly Ġnot Ġonly Ġthe Ġboundaries Ġof Ġeach Ġsensitive Ġspan , Ġbut Ġalso Ġthe Ġcategory Ġassigned . </s> The Ġsystems Ġare Ġevaluated Ġin Ġterms Ġof Ġmicro - aver aged Ġprecision , Ġrecall Ġand ĠF - 1 Ġscore . ĠNote Ġthat , Ġin Ġcontrast Ġto Ġthe Ġevaluation Ġin ĠExperiment ĠA , ĠMED DOC AN Ġmeasurements Ġare Ġentity - based Ġinstead Ġof Ġtoken wise . ĠAn Ġexhaustive Ġexplanation Ġof Ġthe ĠMED DOC AN Ġevaluation Ġprocedure Ġis Ġavailable Ġonline , Ġas Ġwell Ġas Ġthe Ġofficial Ġevaluation Ġscript , Ġwhich Ġwe Ġused Ġto Ġobtain Ġthe Ġreported Ġresults . </s> Results </s> This Ġsection Ġdescribes Ġthe Ġresults Ġobtained Ġin Ġthe Ġtwo Ġsets Ġof Ġexperiments : ĠN UB es - PH I Ġand ĠMED DOC AN . </s> Results Ġ: :: ĠExperiment ĠA : ĠN UB es - PH I </s> Table Ġshows Ġthe Ġresults Ġof Ġthe Ġconducted Ġexperiments Ġin ĠN UB es - PH I Ġfor Ġall Ġthe Ġcompared Ġsystems . ĠThe Ġincluded Ġbaseline Ġserves Ġto Ġgive Ġa Ġquick Ġinsight Ġabout Ġhow Ġchallenging Ġthe Ġdata Ġis . ĠWith Ġsimple Ġregular Ġexpressions Ġand Ġg az ette ers Ġa Ġprecision Ġof Ġ0 . 8 53 Ġis Ġobtained . ĠOn Ġthe Ġother Ġhand , Ġthe Ġrecall , Ġwhich Ġdirectly Ġdepends Ġon Ġthe Ġcoverage Ġprovided Ġby Ġthe Ġrules Ġand Ġresources , Ġdrops Ġto Ġ0 . 469 . ĠHence , Ġthis Ġtask Ġis Ġunlikely Ġto Ġbe Ġsolved Ġwithout Ġthe Ġgeneral isation Ġcapabilities Ġprovided Ġby Ġmachine - learning Ġand Ġdeep - learning Ġmodels . </s> Regarding Ġthe Ġdetection Ġscenario ĠâĢĵ that Ġis , Ġthe Ġscenario Ġconcerned Ġwith Ġa Ġbinary Ġclassification Ġto Ġdetermine Ġwhether Ġeach Ġindividual Ġtoken Ġconve ys Ġsensitive Ġinformation Ġor Ġnot âĢĵ , Ġit Ġcan Ġbe Ġobserved Ġthat ĠB ERT Ġoutper forms Ġits Ġcompetitors . ĠA Ġfact Ġworth Ġhighlighting Ġis Ġthat , Ġaccording Ġto Ġthese Ġresults , ĠB ERT Ġachieves Ġa Ġprecision Ġlower Ġthan Ġthe Ġrest Ġof Ġthe Ġsystems Ġ( i . e ., Ġit Ġmakes Ġmore Ġfalse Ġpositive Ġpredictions ); Ġin Ġexchange , Ġit Ġob tains Ġa Ġremarkably Ġhigher Ġrecall . ĠNotice ably , Ġit Ġreaches Ġa Ġrecall Ġof Ġ0 . 9 79 , Ġimproving Ġby Ġmore Ġthan Ġ4 Ġpoints Ġthe Ġsecond - best Ġsystem , Ġspa Cy . </s> The Ġtable Ġalso Ġshows Ġthe Ġresults Ġfor Ġthe Ġrelaxed Ġmetric Ġthat Ġonly Ġtakes Ġinto Ġaccount Ġthe Ġentity Ġtype Ġdetected , Ġregardless Ġof Ġthe ĠB IO Ġlabel Ġ( i . e ., Ġignoring Ġwhether Ġthe Ġtoken Ġis Ġat Ġthe Ġbeginning Ġor Ġin Ġthe Ġmiddle Ġof Ġa Ġsensitive Ġsequence Ġof Ġtokens ). ĠThe Ġconclusions Ġare Ġvery Ġsimilar Ġto Ġthose Ġextracted Ġpreviously , Ġwith ĠB ERT Ġgaining Ġ2 . 1 Ġpoints Ġof ĠF 1 - score Ġover Ġthe ĠCR F Ġbased Ġapproach . ĠThe Ġconfusion Ġmat rices Ġof Ġthe Ġpredictions Ġmade Ġby ĠCR F , Ġspa Cy , Ġand ĠB ERT Ġin Ġthis Ġscenario Ġare Ġshown Ġin ĠTable Ġ. ĠAs Ġcan Ġbee Ġseen , ĠB ERT Ġhas Ġless Ġdifficulty Ġin Ġpredicting Ġcorrectly Ġless Ġfrequent Ġcategories , Ġsuch Ġas Ġ` Location ', Ġ` Job ', Ġand Ġ` Pat ient '. ĠOne Ġof Ġthe Ġmost Ġcommon Ġmistakes Ġaccording Ġto Ġthe Ġconfusion Ġmat rices Ġis Ġclass ifying Ġhospital Ġnames Ġas Ġ` Location ' Ġinstead Ġof Ġthe Ġmore Ġaccurate Ġ` H ospital '; Ġthis Ġis Ġhardly Ġa Ġharmful Ġerror , Ġgiven Ġthat Ġa Ġhospital Ġis Ġactually Ġa Ġlocation . ĠLast , Ġthe Ġcategory Ġ` Other ' Ġis Ġcompletely Ġleaked Ġby Ġall Ġthe Ġcompared Ġsystems , Ġmost Ġlikely Ġdue Ġto Ġits Ġalmost Ġtotal Ġlack Ġof Ġsupport Ġin Ġboth Ġtraining Ġand Ġevaluation Ġdatasets . </s> To Ġfinish Ġwith Ġthis Ġexperiment Ġset , ĠTable Ġalso Ġshows Ġthe Ġstrict Ġclassification Ġprecision , Ġrecall Ġand ĠF 1 - score Ġfor Ġthe Ġcompared Ġsystems . ĠDespite Ġthe Ġfact Ġthat , Ġin Ġgeneral , Ġthe Ġsystems Ġobtain Ġhigh Ġvalues , ĠB ERT Ġoutper forms Ġthem Ġagain . ĠB ERT 's ĠF 1 - score Ġis Ġ1 . 9 Ġpoints Ġhigher Ġthan Ġthe Ġnext Ġmost Ġcompetitive Ġresult Ġin Ġthe Ġcomparison . ĠMore Ġremarkably , Ġthe Ġrecall Ġobtained Ġby ĠB ERT Ġis Ġabout Ġ5 Ġpoints Ġabove . </s> Upon Ġmanual Ġinspection Ġof Ġthe Ġerrors Ġcommitted Ġby Ġthe ĠB ERT - based Ġmodel , Ġwe Ġdiscovered Ġthat Ġit Ġhas Ġa Ġslight Ġtendency Ġtowards Ġproducing Ġill - formed ĠB IO Ġsequences Ġ( e . g , Ġstarting Ġa Ġsensitive Ġspan Ġwith Ġ` Inside ' Ġinstead Ġof Ġ` Begin '; Ġsee ĠTable Ġ). ĠWe Ġcould Ġexpect Ġthat Ġcomplement ing Ġthe ĠB ERT - based Ġmodel Ġwith Ġa ĠCR F Ġlayer Ġon Ġtop Ġwould Ġhelp Ġenforce Ġthe Ġemission Ġof Ġvalid Ġsequences , Ġallev iating Ġthis Ġkind Ġof Ġerrors Ġand Ġfurther Ġimproving Ġits Ġresults . </s> Finally , ĠFigure Ġshows Ġthe Ġimpact Ġof Ġdecreasing Ġthe Ġamount Ġof Ġtraining Ġdata Ġin Ġthe Ġdetection Ġscenario . ĠIt Ġshows Ġthe Ġdifference Ġin Ġprecision , Ġrecall , Ġand ĠF 1 - score Ġwith Ġrespect Ġto Ġthat Ġobtained Ġusing Ġ100 % Ġof Ġthe Ġtraining Ġdata . ĠA Ġgeneral Ġdownward Ġtrend Ġcan Ġbe Ġobserved , Ġas Ġone Ġwould Ġexpect : Ġless Ġtraining Ġdata Ġleads Ġto Ġless Ġaccurate Ġpredictions . ĠHowever , Ġthe ĠB ERT - based Ġmodel Ġis Ġthe Ġmost Ġrobust Ġto Ġtraining - data Ġreduction , Ġshowing Ġan Ġsteadily Ġlow Ġperformance Ġloss . ĠWith Ġ1 % Ġof Ġthe Ġdataset Ġ( 230 Ġtraining Ġinstances ), Ġthe ĠB ERT - based Ġmodel Ġonly Ġsuffers Ġa Ġstriking Ġ7 - point ĠF 1 - score Ġloss , Ġin Ġcontrast Ġto Ġthe Ġ32 Ġand Ġ39 Ġpoints Ġlost Ġby Ġthe ĠCR F Ġand Ġspa Cy Ġmodels , Ġrespectively . ĠThis Ġsteep Ġperformance Ġdrop Ġstems Ġto Ġa Ġlarger Ġextent Ġfrom Ġrecall Ġdecline , Ġwhich Ġis Ġnot Ġthat Ġmarked Ġin Ġthe Ġcase Ġof ĠB ERT . ĠOverall , Ġthese Ġresults Ġindicate Ġthat Ġthe Ġtransfer - learning Ġachieved Ġthrough Ġthe ĠB ERT Ġmult ilingual Ġpre - trained Ġmodel Ġnot Ġonly Ġhelps Ġobtain Ġbetter Ġresults , Ġbut Ġalso Ġlowers Ġthe Ġneed Ġof Ġmanually Ġlabelled Ġdata Ġfor Ġthis Ġapplication Ġdomain . </s> Results Ġ: :: ĠExperiment ĠB : ĠMED DOC AN </s> The Ġresults Ġof Ġthe Ġtwo ĠMED DOC AN Ġscenarios ĠâĢĵ det ection Ġand Ġclassification âĢĵ Ġare Ġshown Ġin ĠTable Ġ. ĠThese Ġresults Ġfollow Ġthe Ġsame Ġpattern Ġas Ġin Ġthe Ġprevious Ġexperiments , Ġwith Ġthe ĠCR F Ġclass ifier Ġbeing Ġthe Ġmost Ġprecise Ġof Ġall , Ġand ĠB ERT Ġoutper forming Ġboth Ġthe ĠCR F Ġand Ġspa Cy Ġclass ifiers Ġthanks Ġto Ġits Ġgreater Ġrecall . ĠWe Ġalso Ġshow Ġthe Ġresults Ġof Ġm ao 2019 had oken Ġwho , Ġdespite Ġof Ġhaving Ġused Ġa ĠB ERT - based Ġsystem , Ġachieve Ġlower Ġscores Ġthan Ġour Ġmodels . ĠThe Ġreason Ġwhy Ġit Ġshould Ġbe Ġso Ġremain Ġunclear . </s> With Ġregard Ġto Ġthe Ġwinner Ġof Ġthe ĠMED DOC AN Ġshared Ġtask , Ġthe ĠB ERT - based Ġmodel Ġhas Ġnot Ġimproved Ġthe Ġscores Ġobtained Ġby Ġneither Ġthe Ġdomain - dependent Ġ( S 3 ) Ġnor Ġthe Ġdomain - independent Ġ( S 2 ) ĠNL N DE Ġmodel . ĠHowever , Ġattending Ġto Ġthe Ġobtained Ġresults , ĠB ERT Ġremains Ġonly Ġ0 . 3 ĠF 1 - score Ġpoints Ġbehind , Ġand Ġwould Ġhave Ġachieved Ġthe Ġsecond Ġposition Ġamong Ġall Ġthe ĠMED DOC AN Ġshared Ġtask Ġcompetitors . ĠTaking Ġinto Ġaccount Ġthat Ġonly Ġ3 % Ġof Ġthe Ġgold Ġlabels Ġremain Ġincorrectly Ġannot ated , Ġthe Ġtask Ġcan Ġbe Ġconsidered Ġalmost Ġsolved , Ġand Ġit Ġis Ġnot Ġclear Ġif Ġthe Ġdifferences Ġamong Ġthe Ġsystems Ġare Ġactually Ġsignificant , Ġor Ġwhether Ġthey Ġstem Ġfrom Ġminor Ġvariations Ġin Ġinitial isation Ġor Ġa Ġlong - tail Ġof Ġminor Ġlab elling Ġinconsistencies . </s> Con clusions Ġand ĠFuture ĠWork </s> In Ġthis Ġwork Ġwe Ġhave Ġbriefly Ġintroduced Ġthe Ġproblems Ġrelated Ġto Ġdata Ġprivacy Ġprotection Ġin Ġclinical Ġdomain . ĠWe Ġhave Ġalso Ġdescribed Ġsome Ġof Ġthe Ġgroundbreaking Ġadvances Ġon Ġthe ĠNatural ĠLanguage ĠProcessing Ġfield Ġdue Ġto Ġthe Ġappearance Ġof ĠTransformers - based Ġdeep - learning Ġarchitectures Ġand Ġtransfer Ġlearning Ġfrom Ġvery Ġlarge Ġgeneral - domain Ġmult ilingual Ġcorpor a , Ġfocusing Ġour Ġattention Ġin Ġone Ġof Ġits Ġmost Ġrepresentative Ġexamples , ĠGoogle 's ĠB ERT Ġmodel . </s> In Ġorder Ġto Ġassess Ġthe Ġperformance Ġof ĠB ERT Ġfor ĠSpanish Ġclinical Ġdata Ġanonym isation , Ġwe Ġhave Ġconducted Ġseveral Ġexperiments Ġwith Ġa ĠB ERT - based Ġsequence Ġlab elling Ġapproach Ġusing Ġthe Ġpre - trained Ġmult ilingual ĠB ERT Ġmodel Ġshared Ġby ĠGoogle Ġas Ġthe Ġstarting Ġpoint Ġfor Ġthe Ġmodel Ġtraining . ĠWe Ġhave Ġcompared Ġthis ĠB ERT - based Ġsequence Ġlab elling Ġagainst Ġother Ġmethods Ġand Ġsystems . ĠOne Ġof Ġthe Ġexperiments Ġuses Ġthe ĠMED DOC AN Ġ2019 Ġshared Ġtask Ġdataset , Ġwhile Ġthe Ġother Ġuses Ġa Ġnovel ĠSpanish Ġclinical Ġreports Ġdataset Ġcalled ĠN UB es - PH I . </s> The Ġresults Ġof Ġthe Ġexperiments Ġshow Ġthat , Ġin ĠN UB es - PH I , Ġthe ĠB ERT - based Ġmodel Ġoutper forms Ġthe Ġother Ġsystems Ġwithout Ġrequiring Ġany Ġadaptation Ġor Ġdomain - specific Ġfeature Ġengineering , Ġjust Ġby Ġbeing Ġtrained Ġon Ġthe Ġprovided Ġlabelled Ġdata . ĠInterestingly , Ġthe ĠB ERT - based Ġmodel Ġob tains Ġa Ġremarkably Ġhigher Ġrecall Ġthan Ġthe Ġother Ġsystems . ĠHigh Ġrecall Ġis Ġa Ġdesirable Ġoutcome Ġbecause , Ġwhen Ġanonym ising Ġsensible Ġdocuments , Ġthe Ġaccidental Ġleak Ġof Ġsensible Ġdata Ġis Ġlikely Ġto Ġbe Ġmore Ġdangerous Ġthan Ġthe Ġunintended Ġover - ob fusc ation Ġof Ġnon - sensitive Ġtext . </s> Further , Ġwe Ġhave Ġconducted Ġan Ġadditional Ġexperiment Ġon Ġthis Ġdataset Ġby Ġprogressively Ġreducing Ġthe Ġtraining Ġdata Ġfor Ġall Ġthe Ġcompared Ġsystems . ĠThe ĠB ERT - based Ġmodel Ġshows Ġthe Ġhighest Ġrobust ness Ġto Ġtraining - data Ġscarcity , Ġlo osing Ġonly Ġ7 Ġpoints Ġof ĠF 1 - score Ġwhen Ġtrained Ġon Ġ230 Ġinstances Ġinstead Ġof Ġ21 , 371 . ĠThese Ġobservation Ġare Ġin Ġline Ġwith Ġthe Ġresults Ġobtained Ġby Ġthe ĠN LP Ġcommunity Ġusing ĠB ERT Ġfor Ġother Ġtasks . </s> The Ġexperiments Ġwith Ġthe ĠMED DOC AN Ġ2019 Ġshared Ġtask Ġdataset Ġfollow Ġthe Ġsame Ġpattern . ĠIn Ġthis Ġcase , Ġthe ĠB ERT - based Ġmodel Ġfalls Ġ0 . 3 ĠF 1 - score Ġpoints Ġbehind Ġthe Ġshared Ġtask Ġwinning Ġsystem , Ġbut Ġit Ġwould Ġhave Ġachieved Ġthe Ġsecond Ġposition Ġin Ġthe Ġcompetition Ġwith Ġno Ġfurther Ġrefinement . </s> Since Ġwe Ġhave Ġused Ġa Ġpre - trained Ġmult ilingual ĠB ERT Ġmodel , Ġthe Ġsame Ġapproach Ġis Ġlikely Ġto Ġwork Ġfor Ġother Ġlanguages Ġjust Ġby Ġproviding Ġsome Ġlabelled Ġtraining Ġdata . ĠFurther , Ġthis Ġis Ġthe Ġsimplest Ġfine - tun ing Ġthat Ġcan Ġbe Ġperformed Ġbased Ġon ĠB ERT . ĠMore Ġsophisticated Ġfine - tun ing Ġlayers Ġcould Ġhelp Ġimprove Ġthe Ġresults . ĠFor Ġexample , Ġit Ġcould Ġbe Ġexpected Ġthat Ġa ĠCR F Ġlayer Ġhelped Ġenforce Ġbetter ĠB IO Ġtagging Ġsequence Ġpredictions . ĠPrec ise ly , Ġm ao 2019 had oken Ġparticipated Ġin Ġthe ĠMED DOC AN Ġcompetition Ġusing Ġa ĠB ERT + CR F Ġarchitecture , Ġbut Ġtheir Ġreported Ġscores Ġare Ġabout Ġ3 Ġpoints Ġlower Ġthan Ġour Ġimplementation . ĠFrom Ġthe Ġdescription Ġof Ġtheir Ġwork , Ġit Ġis Ġunclear Ġwhat Ġthe Ġsource Ġof Ġthis Ġscore Ġdifference Ġcould Ġbe . </s> Further , Ġat Ġthe Ġtime Ġof Ġwriting Ġthis Ġpaper , Ġnew Ġmult ilingual Ġpre - trained Ġmodels Ġand ĠTrans former Ġarchitectures Ġhave Ġbecome Ġavailable . ĠIt Ġwould Ġnot Ġcome Ġas Ġa Ġsurprise Ġthat Ġthese Ġnew Ġresources Ġand Ġsystems ĠâĢĵ e . g ., ĠXL M - Ro BER Ta ĠB IB REF 19 Ġor ĠBET O ĠB IB REF 20 , Ġa ĠB ERT Ġmodel Ġfully Ġpre - trained Ġon ĠSpanish Ġtexts âĢĵ Ġfurther Ġadvanced Ġthe Ġstate Ġof Ġthe Ġart Ġin Ġthis Ġtask . </s> Acknowled gements </s> This Ġwork Ġhas Ġbeen Ġsupported Ġby ĠVic om tech Ġand Ġpartially Ġfunded Ġby Ġthe Ġproject ĠDeep Reading Ġ( RT I 2018 - 09 68 46 - B - C 21 , ĠMC IU / AE I / F ED ER , UE ).
Introduction During the first two decades of the 21st century, the sharing and processing of vast amounts of data has become pervasive. This expansion of data sharing and processing capabilities is both a blessing and a curse. Data helps build better information systems for the digital era and enables further research for advanced data management that benefits the society in general. But the use of this very data containing sensitive information conflicts with private data protection, both from an ethical and a legal perspective. There are several application domains on which this situation is particularly acute. This is the case of the medical domain BIBREF0. There are plenty of potential applications for advanced medical data management that can only be researched and developed using real data; yet, the use of medical data is severely limited –when not entirely prohibited– due to data privacy protection policies. One way of circumventing this problem is to anonymise the data by removing, replacing or obfuscating the personal information mentioned, as exemplified in Table TABREF1. This task can be done by hand, having people read and anonymise the documents one by one. Despite being a reliable and simple solution, this approach is tedious, expensive, time consuming and difficult to scale to the potentially thousands or millions of documents that need to be anonymised. For this reason, numerous of systems and approaches have been developed during the last decades to attempt to automate the anonymisation of sensitive content, starting with the automatic detection and classification of sensitive information. Some of these systems rely on rules, patterns and dictionaries, while others use more advanced techniques related to machine learning and, more recently, deep learning. Given that this paper is concerned with text documents (e.g. medical records), the involved techniques are related to Natural Language Processing (NLP). When using NLP approaches, it is common to pose the problem of document anonymisation as a sequence labelling problem, i.e. classifying each token within a sequence as being sensitive information or not. Further, depending on the objective of the anonymisation task, it is also important to determine the type of sensitive information (names of individuals, addresses, age, sex, etc.). The anonymisation systems based on NLP techniques perform reasonably well, but are far from perfect. Depending on the difficulty posed by each dataset or the amount of available data for training machine learning models, the performance achieved by these methods is not enough to fully rely on them in certain situations BIBREF0. However, in the last two years, the NLP community has reached an important milestone thanks to the appearance of the so-called Transformers neural network architectures BIBREF1. In this paper, we conduct several experiments in sensitive information detection and classification on Spanish clinical text using BERT (from `Bidirectional Encoder Representations from Transformers') BIBREF2 as the base for a sequence labelling approach. The experiments are carried out on two datasets: the MEDDOCAN: Medical Document Anonymization shared task dataset BIBREF3, and NUBes BIBREF4, a corpus of real medical reports in Spanish. In these experiments, we compare the performance of BERT with other machine-learning-based systems, some of which use language-specific features. Our aim is to evaluate how good a BERT-based model performs without language nor domain specialisation apart from the training data labelled for the task at hand. The rest of the paper is structured as follows: the next section describes related work about data anonymisation in general and clinical data anonymisation in particular; it also provides a more detailed explanation and background about the Transformers architecture and BERT. Section SECREF3 describes the data involved in the experiments and the systems evaluated in this paper, including the BERT-based system; finally, it details the experimental design. Section SECREF4 introduces the results for each set of experiments. Finally, Section SECREF5 contains the conclusions and future lines of work. Related Work The state of the art in the field of Natural Language Processing (NLP) has reached an important milestone in the last couple of years thanks to deep-learning architectures, increasing in several points the performance of new models for almost any text processing task. The major change started with the Transformers model proposed by vaswani2017attention. It substituted the widely used recurrent and convolutional neural network architectures by another approach based solely on self-attention, obtaining an impressive performance gain. The original proposal was focused on an encoder-decoder architecture for machine translation, but soon the use of Transformers was made more general BIBREF1. There are several other popular models that use Transformers, such as Open AI's GPT and GPT2 BIBREF5, RoBERTa BIBREF6 and the most recent XLNet BIBREF7; still, BERT BIBREF2 is one of the most widespread Transformer-based models. BERT trains its unsupervised language model using a Masked Language Model and Next Sentence Prediction. A common problem in NLP is the lack of enough training data. BERT can be pre-trained to learn general or specific language models using very large amounts of unlabelled text (e.g. web content, Wikipedia, etc.), and this knowledge can be transferred to a different downstream task in a process that receives the name fine-tuning. devlin2018bert have used fine-tuning to achieve state-of-the-art results on a wide variety of challenging natural language tasks, such as text classification, Question Answering (QA) and Named Entity Recognition and Classification (NERC). BERT has also been used successfully by other community practitioners for a wide range of NLP-related tasks BIBREF8, BIBREF9. Regarding the task of data anonymisation in particular, anonymisation systems may follow different approaches and pursue different objectives (Cormode and Srivastava, 2009). The first objective of these systems is to detect and classify the sensitive information contained in the documents to be anonymised. In order to achieve that, they use rule-based approaches, Machine Learning (ML) approaches, or a combination of both. Although most of these efforts are for English texts –see, among others, the i2b2 de-identification challenges BIBREF10, BIBREF11, dernon2016deep, or khin2018deep–, other languages are also attracting growing interest. Some examples are mamede2016automated for Portuguese and tveit2004anonymization for Norwegian. With respect to the anonymisation of text written in Spanish, recent studies include medina2018building, hassan2018anonimizacion and garcia2018automating. Most notably, in 2019 the first community challenge about anonymisation of medical documents in Spanish, MEDDOCAN BIBREF3, was held as part of the IberLEF initiative. The winners of the challenge –the Neither-Language-nor-Domain-Experts (NLNDE) BIBREF12– achieved F1-scores as high as 0.975 in the task of sensitive information detection and categorisation by using recurrent neural networks with Conditional Random Field (CRF) output layers. At the same challenge, mao2019hadoken occupied the 8th position among 18 participants using BERT. According to the description of the system, the authors used BERT-Base Multilingual Cased and an output CRF layer. However, their system is $\sim $3 F1-score points below our implementation without the CRF layer. Materials and Methods The aim of this paper is to evaluate BERT's multilingual model and compare it to other established machine-learning algorithms in a specific task: sensitive data detection and classification in Spanish clinical free text. This section describes the data involved in the experiments and the systems evaluated. Finally, we introduce the experimental setup. Materials and Methods ::: Data Two datasets are exploited in this article. Both datasets consist of plain text containing clinical narrative written in Spanish, and their respective manual annotations of sensitive information in BRAT BIBREF13 standoff format. In order to feed the data to the different algorithms presented in Section SECREF7, these datasets were transformed to comply with the commonly used BIO sequence representation scheme BIBREF14. Materials and Methods ::: Data ::: NUBes-PHI NUBes BIBREF4 is a corpus of around 7,000 real medical reports written in Spanish and annotated with negation and uncertainty information. Before being published, sensitive information had to be manually annotated and replaced for the corpus to be safely shared. In this article, we work with the NUBes version prior to its anonymisation, that is, with the manual annotations of sensitive information. It follows that the version we work with is not publicly available and, due to contractual restrictions, we cannot reveal the provenance of the data. In order to avoid confusion between the two corpus versions, we henceforth refer to the version relevant in this paper as NUBes-PHI (from `NUBes with Personal Health Information'). NUBes-PHI consists of 32,055 sentences annotated for 11 different sensitive information categories. Overall, it contains 7,818 annotations. The corpus has been randomly split into train (72%), development (8%) and test (20%) sets to conduct the experiments described in this paper. The size of each split and the distribution of the annotations can be consulted in Tables and , respectively. The majority of sensitive information in NUBes-PHI are temporal expressions (`Date' and `Time'), followed by healthcare facility mentions (`Hospital'), and the age of the patient. Mentions of people are not that frequent, with physician names (`Doctor') occurring much more often than patient names (`Patient'). The least frequent sensitive information types, which account for $\sim $10% of the remaining annotations, consist of the patient's sex, job, and kinship, and locations other than healthcare facilities (`Location'). Finally, the tag `Other' includes, for instance, mentions to institutions unrelated to healthcare and whether the patient is right- or left-handed. It occurs just 36 times. Materials and Methods ::: Data ::: The MEDDOCAN corpus The organisers of the MEDDOCAN shared task BIBREF3 curated a synthetic corpus of clinical cases enriched with sensitive information by health documentalists. In this regard, the MEDDOCAN evaluation scenario could be said to be somewhat far from the real use case the technology developed for the shared task is supposed to be applied in. However, at the moment it also provides the only public means for a rigorous comparison between systems for sensitive health information detection in Spanish texts. The size of the MEDDOCAN corpus is shown in Table . Compared to NUBes-PHI (Table ), this corpus contains more sensitive information annotations, both in absolute and relative terms. The sensitive annotation categories considered in MEDDOCAN differ in part from those in NUBes-PHI. Most notably, it contains finer-grained labels for location-related mentions –namely, `Address', `Territory', and `Country'–, and other sensitive information categories that we did not encounter in NUBes-PHI (e.g., identifiers, phone numbers, e-mail addresses, etc.). In total, the MEDDOCAN corpus has 21 sensitive information categories. We refer the reader to the organisers' article BIBREF3 for more detailed information about this corpus. Materials and Methods ::: Systems Apart from experimenting with a pre-trained BERT model, we have run experiments with other systems and baselines, to compare them and obtain a better perspective about BERT's performance in these datasets. Materials and Methods ::: Systems ::: Baseline As the simplest baseline, a sensitive data recogniser and classifier has been developed that consists of regular-expressions and dictionary look-ups. For each category to detect a specific method has been implemented. For instance, the Date, Age, Time and Doctor detectors are based on regular-expressions; Hospital, Sex, Kinship, Location, Patient and Job are looked up in dictionaries. The dictionaries are hand-crafted from the training data available, except for the Patient's case, for which the possible candidates considered are the 100 most common female and male names in Spain according to the Instituto Nacional de Estadística (INE; Spanish Statistical Office). Materials and Methods ::: Systems ::: CRF Conditional Random Fields (CRF) BIBREF15 have been extensively used for tasks of sequential nature. In this paper, we propose as one of the competitive baselines a CRF classifier trained with sklearn-crfsuite for Python 3.5 and the following configuration: algorithm = lbfgs; maximum iterations = 100; c1 = c2 = 0.1; all transitions = true; optimise = false. The features extracted from each token are as follows: [noitemsep] prefixes and suffixes of 2 and 3 characters; the length of the token in characters and the length of the sentence in tokens; whether the token is all-letters, a number, or a sequence of punctuation marks; whether the token contains the character `@'; whether the token is the start or end of the sentence; the token's casing and the ratio of uppercase characters, digits, and punctuation marks to its length; and, the lemma, part-of-speech tag, and named-entity tag given by ixa-pipes BIBREF16 upon analysing the sentence the token belongs to. Noticeably, none of the features used to train the CRF classifier is domain-dependent. However, the latter group of features is language dependent. Materials and Methods ::: Systems ::: spaCy spaCy is a widely used NLP library that implements state-of-the-art text processing pipelines, including a sequence-labelling pipeline similar to the one described by strubell2017fast. spaCy offers several pre-trained models in Spanish, which perform basic NLP tasks such as Named Entity Recognition (NER). In this paper, we have trained a new NER model to detect NUBes-PHI labels. For this purpose, the new model uses all the labels of the training corpus coded with its context at sentence level. The network optimisation parameters and dropout values are the ones recommended in the documentation for small datasets. Finally, the model is trained using batches of size 64. No more features are included, so the classifier is language-dependent but not domain-dependent. Materials and Methods ::: Systems ::: BERT As introduced earlier, BERT has shown an outstanding performance in NERC-like tasks, improving the start-of-the-art results for almost every dataset and language. We take the same approach here, by using the model BERT-Base Multilingual Cased with a Fully Connected (FC) layer on top to perform a fine-tuning of the whole model for an anonymisation task in Spanish clinical data. Our implementation is built on PyTorch and the PyTorch-Transformers library BIBREF1. The training phase consists in the following steps (roughly depicted in Figure ): Pre-processing: since we are relying on a pre-trained BERT model, we must match the same configuration by using a specific tokenisation and vocabulary. BERT also needs that the inputs contains special tokens to signal the beginning and the end of each sequence. Fine-tuning: the pre-processed sequence is fed into the model. BERT outputs the contextual embeddings that encode each of the inputted tokens. This embedding representation for each token is fed into the FC linear layer after a dropout layer (with a 0.1 dropout probability), which in turn outputs the logits for each possible class. The cross-entropy loss function is calculated comparing the logits and the gold labels, and the error is back-propagated to adjust the model parameters. We have trained the model using an AdamW optimiser BIBREF17 with the learning rate set to 3e-5, as recommended by devlin2018bert, and with a gradient clipping of 1.0. We also applied a learning-rate scheduler that warms up the learning rate from zero to its maximum value as the training progresses, which is also a common practice. For each experiment set proposed below, the training was run with an early-stopping patience of 15 epochs. Then, the model that performed best against the development set was used to produce the reported results. The experiments were run on a 64-core server with operating system Ubuntu 16.04, 250GB of RAM memory, and 4 GeForce RTX 2080 GPUs with 11GB of memory. The maximum sequence length was set at 500 and the batch size at 12. In this setting, each epoch –a full pass through all the training data– required about 10 minutes to complete. Materials and Methods ::: Experimental design We have conducted experiments with BERT in the two datasets of Spanish clinical narrative presented in Section SECREF3 The first experiment set uses NUBes-PHI, a corpus of real medical reports manually annotated with sensitive information. Because this corpus is not publicly available, and in order to compare the BERT-based model to other related published systems, the second set of experiments uses the MEDDOCAN 2019 shared task competition dataset. The following sections provide greater detail about the two experimental setups. Materials and Methods ::: Experimental design ::: Experiment A: NUBes-PHI In this experiment set, we evaluate all the systems presented in Section SECREF7, namely, the rule-based baseline, the CRF classifier, the spaCy entity tagger, and BERT. The evaluation comprises three scenarios of increasing difficulty: [noitemsep] - Evaluates the performance of the systems at predicting whether each token is sensitive or non-sensitive; that is, the measurements only take into account whether a sensitive token has been recognised or not, regardless of the BIO label and the category assigned. This scenario shows how good a system would be at obfuscating sensitive data (e.g., by replacing sensitive tokens with asterisks). - We measure the performance of the systems at predicting the sensitive information type of each token –i.e., the 11 categories presented in Section SECREF5 or `out'. Detecting entity types correctly is important if a system is going to be used to replace sensitive data by fake data of the same type (e.g., random people names). - This is the strictest evaluation, as it takes into account both the BIO label and the category assigned to each individual token. Being able to discern between two contiguous sensitive entities of the same type is relevant not only because it is helpful when producing fake replacements, but because it also yields more accurate statistics of the sensitive information present in a given document collection. The systems are evaluated in terms of micro-average precision, recall and F1-score in all the scenarios. In addition to the scenarios proposed, a subject worth being studied is the need of labelled data. Manually labelled data is an scarce and expensive resource, which for some application domains or languages is difficult to come by. In order to obtain an estimation of the dependency of each system on the available amount of training data, we have retrained all the compared models using decreasing amounts of data –from 100% of the available training instances to just 1%. The same data subsets have been used to train all the systems. Due to the knowledge transferred from the pre-trained BERT model, the BERT-based model is expected to be more robust to data scarcity than those that start their training from scratch. Materials and Methods ::: Experimental design ::: Experiment B: MEDDOCAN In this experiment set, our BERT implementation is compared to several systems that participated in the MEDDOCAN challenge: a CRF classifier BIBREF18, a spaCy entity recogniser BIBREF18, and NLNDE BIBREF12, the winner of the shared task and current state of the art for sensitive information detection and classification in Spanish clinical text. Specifically, we include the results of a domain-independent NLNDE model (S2), and the results of a model enriched with domain-specific embeddings (S3). Finally, we include the results obtained by mao2019hadoken with a CRF output layer on top of BERT embeddings. MEDDOCAN consists of two scenarios: [noitemsep] - This evaluation measures how good a system is at detecting sensitive text spans, regardless of the category assigned to them. - In this scenario, systems are required to match exactly not only the boundaries of each sensitive span, but also the category assigned. The systems are evaluated in terms of micro-averaged precision, recall and F-1 score. Note that, in contrast to the evaluation in Experiment A, MEDDOCAN measurements are entity-based instead of tokenwise. An exhaustive explanation of the MEDDOCAN evaluation procedure is available online, as well as the official evaluation script, which we used to obtain the reported results. Results This section describes the results obtained in the two sets of experiments: NUBes-PHI and MEDDOCAN. Results ::: Experiment A: NUBes-PHI Table shows the results of the conducted experiments in NUBes-PHI for all the compared systems. The included baseline serves to give a quick insight about how challenging the data is. With simple regular expressions and gazetteers a precision of 0.853 is obtained. On the other hand, the recall, which directly depends on the coverage provided by the rules and resources, drops to 0.469. Hence, this task is unlikely to be solved without the generalisation capabilities provided by machine-learning and deep-learning models. Regarding the detection scenario –that is, the scenario concerned with a binary classification to determine whether each individual token conveys sensitive information or not–, it can be observed that BERT outperforms its competitors. A fact worth highlighting is that, according to these results, BERT achieves a precision lower than the rest of the systems (i.e., it makes more false positive predictions); in exchange, it obtains a remarkably higher recall. Noticeably, it reaches a recall of 0.979, improving by more than 4 points the second-best system, spaCy. The table also shows the results for the relaxed metric that only takes into account the entity type detected, regardless of the BIO label (i.e., ignoring whether the token is at the beginning or in the middle of a sensitive sequence of tokens). The conclusions are very similar to those extracted previously, with BERT gaining 2.1 points of F1-score over the CRF based approach. The confusion matrices of the predictions made by CRF, spaCy, and BERT in this scenario are shown in Table . As can bee seen, BERT has less difficulty in predicting correctly less frequent categories, such as `Location', `Job', and `Patient'. One of the most common mistakes according to the confusion matrices is classifying hospital names as `Location' instead of the more accurate `Hospital'; this is hardly a harmful error, given that a hospital is actually a location. Last, the category `Other' is completely leaked by all the compared systems, most likely due to its almost total lack of support in both training and evaluation datasets. To finish with this experiment set, Table also shows the strict classification precision, recall and F1-score for the compared systems. Despite the fact that, in general, the systems obtain high values, BERT outperforms them again. BERT's F1-score is 1.9 points higher than the next most competitive result in the comparison. More remarkably, the recall obtained by BERT is about 5 points above. Upon manual inspection of the errors committed by the BERT-based model, we discovered that it has a slight tendency towards producing ill-formed BIO sequences (e.g, starting a sensitive span with `Inside' instead of `Begin'; see Table ). We could expect that complementing the BERT-based model with a CRF layer on top would help enforce the emission of valid sequences, alleviating this kind of errors and further improving its results. Finally, Figure shows the impact of decreasing the amount of training data in the detection scenario. It shows the difference in precision, recall, and F1-score with respect to that obtained using 100% of the training data. A general downward trend can be observed, as one would expect: less training data leads to less accurate predictions. However, the BERT-based model is the most robust to training-data reduction, showing an steadily low performance loss. With 1% of the dataset (230 training instances), the BERT-based model only suffers a striking 7-point F1-score loss, in contrast to the 32 and 39 points lost by the CRF and spaCy models, respectively. This steep performance drop stems to a larger extent from recall decline, which is not that marked in the case of BERT. Overall, these results indicate that the transfer-learning achieved through the BERT multilingual pre-trained model not only helps obtain better results, but also lowers the need of manually labelled data for this application domain. Results ::: Experiment B: MEDDOCAN The results of the two MEDDOCAN scenarios –detection and classification– are shown in Table . These results follow the same pattern as in the previous experiments, with the CRF classifier being the most precise of all, and BERT outperforming both the CRF and spaCy classifiers thanks to its greater recall. We also show the results of mao2019hadoken who, despite of having used a BERT-based system, achieve lower scores than our models. The reason why it should be so remain unclear. With regard to the winner of the MEDDOCAN shared task, the BERT-based model has not improved the scores obtained by neither the domain-dependent (S3) nor the domain-independent (S2) NLNDE model. However, attending to the obtained results, BERT remains only 0.3 F1-score points behind, and would have achieved the second position among all the MEDDOCAN shared task competitors. Taking into account that only 3% of the gold labels remain incorrectly annotated, the task can be considered almost solved, and it is not clear if the differences among the systems are actually significant, or whether they stem from minor variations in initialisation or a long-tail of minor labelling inconsistencies. Conclusions and Future Work In this work we have briefly introduced the problems related to data privacy protection in clinical domain. We have also described some of the groundbreaking advances on the Natural Language Processing field due to the appearance of Transformers-based deep-learning architectures and transfer learning from very large general-domain multilingual corpora, focusing our attention in one of its most representative examples, Google's BERT model. In order to assess the performance of BERT for Spanish clinical data anonymisation, we have conducted several experiments with a BERT-based sequence labelling approach using the pre-trained multilingual BERT model shared by Google as the starting point for the model training. We have compared this BERT-based sequence labelling against other methods and systems. One of the experiments uses the MEDDOCAN 2019 shared task dataset, while the other uses a novel Spanish clinical reports dataset called NUBes-PHI. The results of the experiments show that, in NUBes-PHI, the BERT-based model outperforms the other systems without requiring any adaptation or domain-specific feature engineering, just by being trained on the provided labelled data. Interestingly, the BERT-based model obtains a remarkably higher recall than the other systems. High recall is a desirable outcome because, when anonymising sensible documents, the accidental leak of sensible data is likely to be more dangerous than the unintended over-obfuscation of non-sensitive text. Further, we have conducted an additional experiment on this dataset by progressively reducing the training data for all the compared systems. The BERT-based model shows the highest robustness to training-data scarcity, loosing only 7 points of F1-score when trained on 230 instances instead of 21,371. These observation are in line with the results obtained by the NLP community using BERT for other tasks. The experiments with the MEDDOCAN 2019 shared task dataset follow the same pattern. In this case, the BERT-based model falls 0.3 F1-score points behind the shared task winning system, but it would have achieved the second position in the competition with no further refinement. Since we have used a pre-trained multilingual BERT model, the same approach is likely to work for other languages just by providing some labelled training data. Further, this is the simplest fine-tuning that can be performed based on BERT. More sophisticated fine-tuning layers could help improve the results. For example, it could be expected that a CRF layer helped enforce better BIO tagging sequence predictions. Precisely, mao2019hadoken participated in the MEDDOCAN competition using a BERT+CRF architecture, but their reported scores are about 3 points lower than our implementation. From the description of their work, it is unclear what the source of this score difference could be. Further, at the time of writing this paper, new multilingual pre-trained models and Transformer architectures have become available. It would not come as a surprise that these new resources and systems –e.g., XLM-RoBERTa BIBREF19 or BETO BIBREF20, a BERT model fully pre-trained on Spanish texts– further advanced the state of the art in this task. Acknowledgements This work has been supported by Vicomtech and partially funded by the project DeepReading (RTI2018-096846-B-C21, MCIU/AEI/FEDER,UE).
Does BERT reach the best performance among all the algorithms compared?
c0bee6539eb6956a7347daa9d2419b367bd02064
Does ĠB ERT Ġreach Ġthe Ġbest Ġperformance Ġamong Ġall Ġthe Ġalgorithms Ġcompared ?
No
<s> Does ĠB ERT Ġreach Ġthe Ġbest Ġperformance Ġamong Ġall Ġthe Ġalgorithms Ġcompared ? </s> Introduction </s> During Ġthe Ġfirst Ġtwo Ġdecades Ġof Ġthe Ġ21 st Ġcentury , Ġthe Ġsharing Ġand Ġprocessing Ġof Ġvast Ġamounts Ġof Ġdata Ġhas Ġbecome Ġpervasive . ĠThis Ġexpansion Ġof Ġdata Ġsharing Ġand Ġprocessing Ġcapabilities Ġis Ġboth Ġa Ġblessing Ġand Ġa Ġcurse . ĠData Ġhelps Ġbuild Ġbetter Ġinformation Ġsystems Ġfor Ġthe Ġdigital Ġera Ġand Ġenables Ġfurther Ġresearch Ġfor Ġadvanced Ġdata Ġmanagement Ġthat Ġbenefits Ġthe Ġsociety Ġin Ġgeneral . ĠBut Ġthe Ġuse Ġof Ġthis Ġvery Ġdata Ġcontaining Ġsensitive Ġinformation Ġconflicts Ġwith Ġprivate Ġdata Ġprotection , Ġboth Ġfrom Ġan Ġethical Ġand Ġa Ġlegal Ġperspective . </s> There Ġare Ġseveral Ġapplication Ġdomains Ġon Ġwhich Ġthis Ġsituation Ġis Ġparticularly Ġacute . ĠThis Ġis Ġthe Ġcase Ġof Ġthe Ġmedical Ġdomain ĠB IB REF 0 . ĠThere Ġare Ġplenty Ġof Ġpotential Ġapplications Ġfor Ġadvanced Ġmedical Ġdata Ġmanagement Ġthat Ġcan Ġonly Ġbe Ġresearched Ġand Ġdeveloped Ġusing Ġreal Ġdata ; Ġyet , Ġthe Ġuse Ġof Ġmedical Ġdata Ġis Ġseverely Ġlimited ĠâĢĵ when Ġnot Ġentirely Ġprohibited âĢĵ Ġdue Ġto Ġdata Ġprivacy Ġprotection Ġpolicies . </s> One Ġway Ġof Ġcircumvent ing Ġthis Ġproblem Ġis Ġto Ġanonym ise Ġthe Ġdata Ġby Ġremoving , Ġreplacing Ġor Ġobfusc ating Ġthe Ġpersonal Ġinformation Ġmentioned , Ġas Ġexempl ified Ġin ĠTable ĠT AB REF 1 . ĠThis Ġtask Ġcan Ġbe Ġdone Ġby Ġhand , Ġhaving Ġpeople Ġread Ġand Ġanonym ise Ġthe Ġdocuments Ġone Ġby Ġone . ĠDespite Ġbeing Ġa Ġreliable Ġand Ġsimple Ġsolution , Ġthis Ġapproach Ġis Ġtedious , Ġexpensive , Ġtime Ġconsuming Ġand Ġdifficult Ġto Ġscale Ġto Ġthe Ġpotentially Ġthousands Ġor Ġmillions Ġof Ġdocuments Ġthat Ġneed Ġto Ġbe Ġanonym ised . </s> For Ġthis Ġreason , Ġnumerous Ġof Ġsystems Ġand Ġapproaches Ġhave Ġbeen Ġdeveloped Ġduring Ġthe Ġlast Ġdecades Ġto Ġattempt Ġto Ġautomate Ġthe Ġanonym isation Ġof Ġsensitive Ġcontent , Ġstarting Ġwith Ġthe Ġautomatic Ġdetection Ġand Ġclassification Ġof Ġsensitive Ġinformation . ĠSome Ġof Ġthese Ġsystems Ġrely Ġon Ġrules , Ġpatterns Ġand Ġdiction aries , Ġwhile Ġothers Ġuse Ġmore Ġadvanced Ġtechniques Ġrelated Ġto Ġmachine Ġlearning Ġand , Ġmore Ġrecently , Ġdeep Ġlearning . </s> Given Ġthat Ġthis Ġpaper Ġis Ġconcerned Ġwith Ġtext Ġdocuments Ġ( e . g . Ġmedical Ġrecords ), Ġthe Ġinvolved Ġtechniques Ġare Ġrelated Ġto ĠNatural ĠLanguage ĠProcessing Ġ( N LP ). ĠWhen Ġusing ĠN LP Ġapproaches , Ġit Ġis Ġcommon Ġto Ġpose Ġthe Ġproblem Ġof Ġdocument Ġanonym isation Ġas Ġa Ġsequence Ġlab elling Ġproblem , Ġi . e . Ġclass ifying Ġeach Ġtoken Ġwithin Ġa Ġsequence Ġas Ġbeing Ġsensitive Ġinformation Ġor Ġnot . ĠFurther , Ġdepending Ġon Ġthe Ġobjective Ġof Ġthe Ġanonym isation Ġtask , Ġit Ġis Ġalso Ġimportant Ġto Ġdetermine Ġthe Ġtype Ġof Ġsensitive Ġinformation Ġ( names Ġof Ġindividuals , Ġaddresses , Ġage , Ġsex , Ġetc .). </s> The Ġanonym isation Ġsystems Ġbased Ġon ĠN LP Ġtechniques Ġperform Ġreasonably Ġwell , Ġbut Ġare Ġfar Ġfrom Ġperfect . ĠDepending Ġon Ġthe Ġdifficulty Ġposed Ġby Ġeach Ġdataset Ġor Ġthe Ġamount Ġof Ġavailable Ġdata Ġfor Ġtraining Ġmachine Ġlearning Ġmodels , Ġthe Ġperformance Ġachieved Ġby Ġthese Ġmethods Ġis Ġnot Ġenough Ġto Ġfully Ġrely Ġon Ġthem Ġin Ġcertain Ġsituations ĠB IB REF 0 . ĠHowever , Ġin Ġthe Ġlast Ġtwo Ġyears , Ġthe ĠN LP Ġcommunity Ġhas Ġreached Ġan Ġimportant Ġmilestone Ġthanks Ġto Ġthe Ġappearance Ġof Ġthe Ġso - called ĠTransformers Ġneural Ġnetwork Ġarchitectures ĠB IB REF 1 . ĠIn Ġthis Ġpaper , Ġwe Ġconduct Ġseveral Ġexperiments Ġin Ġsensitive Ġinformation Ġdetection Ġand Ġclassification Ġon ĠSpanish Ġclinical Ġtext Ġusing ĠB ERT Ġ( from Ġ` B id irection al ĠEnc oder ĠRepresent ations Ġfrom ĠTransformers ') ĠB IB REF 2 Ġas Ġthe Ġbase Ġfor Ġa Ġsequence Ġlab elling Ġapproach . ĠThe Ġexperiments Ġare Ġcarried Ġout Ġon Ġtwo Ġdatasets : Ġthe ĠMED DOC AN : ĠMedical ĠDocument ĠAn onym ization Ġshared Ġtask Ġdataset ĠB IB REF 3 , Ġand ĠN UB es ĠB IB REF 4 , Ġa Ġcorpus Ġof Ġreal Ġmedical Ġreports Ġin ĠSpanish . ĠIn Ġthese Ġexperiments , Ġwe Ġcompare Ġthe Ġperformance Ġof ĠB ERT Ġwith Ġother Ġmachine - learning - based Ġsystems , Ġsome Ġof Ġwhich Ġuse Ġlanguage - specific Ġfeatures . ĠOur Ġaim Ġis Ġto Ġevaluate Ġhow Ġgood Ġa ĠB ERT - based Ġmodel Ġperforms Ġwithout Ġlanguage Ġnor Ġdomain Ġspecial isation Ġapart Ġfrom Ġthe Ġtraining Ġdata Ġlabelled Ġfor Ġthe Ġtask Ġat Ġhand . </s> The Ġrest Ġof Ġthe Ġpaper Ġis Ġstructured Ġas Ġfollows : Ġthe Ġnext Ġsection Ġdescribes Ġrelated Ġwork Ġabout Ġdata Ġanonym isation Ġin Ġgeneral Ġand Ġclinical Ġdata Ġanonym isation Ġin Ġparticular ; Ġit Ġalso Ġprovides Ġa Ġmore Ġdetailed Ġexplanation Ġand Ġbackground Ġabout Ġthe ĠTransformers Ġarchitecture Ġand ĠB ERT . ĠSection ĠSEC REF 3 Ġdescribes Ġthe Ġdata Ġinvolved Ġin Ġthe Ġexperiments Ġand Ġthe Ġsystems Ġevaluated Ġin Ġthis Ġpaper , Ġincluding Ġthe ĠB ERT - based Ġsystem ; Ġfinally , Ġit Ġdetails Ġthe Ġexperimental Ġdesign . ĠSection ĠSEC REF 4 Ġintroduces Ġthe Ġresults Ġfor Ġeach Ġset Ġof Ġexperiments . ĠFinally , ĠSection ĠSEC REF 5 Ġcontains Ġthe Ġconclusions Ġand Ġfuture Ġlines Ġof Ġwork . </s> Related ĠWork </s> The Ġstate Ġof Ġthe Ġart Ġin Ġthe Ġfield Ġof ĠNatural ĠLanguage ĠProcessing Ġ( N LP ) Ġhas Ġreached Ġan Ġimportant Ġmilestone Ġin Ġthe Ġlast Ġcouple Ġof Ġyears Ġthanks Ġto Ġdeep - learning Ġarchitectures , Ġincreasing Ġin Ġseveral Ġpoints Ġthe Ġperformance Ġof Ġnew Ġmodels Ġfor Ġalmost Ġany Ġtext Ġprocessing Ġtask . </s> The Ġmajor Ġchange Ġstarted Ġwith Ġthe ĠTransformers Ġmodel Ġproposed Ġby Ġvas w ani 2017 att ention . ĠIt Ġsubstituted Ġthe Ġwidely Ġused Ġrecurrent Ġand Ġconv olution al Ġneural Ġnetwork Ġarchitectures Ġby Ġanother Ġapproach Ġbased Ġsolely Ġon Ġself - att ention , Ġobtaining Ġan Ġimpressive Ġperformance Ġgain . ĠThe Ġoriginal Ġproposal Ġwas Ġfocused Ġon Ġan Ġenc oder - dec oder Ġarchitecture Ġfor Ġmachine Ġtranslation , Ġbut Ġsoon Ġthe Ġuse Ġof ĠTransformers Ġwas Ġmade Ġmore Ġgeneral ĠB IB REF 1 . ĠThere Ġare Ġseveral Ġother Ġpopular Ġmodels Ġthat Ġuse ĠTransformers , Ġsuch Ġas ĠOpen ĠAI 's ĠG PT Ġand ĠG PT 2 ĠB IB REF 5 , ĠRo BER Ta ĠB IB REF 6 Ġand Ġthe Ġmost Ġrecent ĠXL Net ĠB IB REF 7 ; Ġstill , ĠB ERT ĠB IB REF 2 Ġis Ġone Ġof Ġthe Ġmost Ġwidespread ĠTrans former - based Ġmodels . </s> BER T Ġtrains Ġits Ġun super vised Ġlanguage Ġmodel Ġusing Ġa ĠMask ed ĠLanguage ĠModel Ġand ĠNext ĠSent ence ĠPrediction . ĠA Ġcommon Ġproblem Ġin ĠN LP Ġis Ġthe Ġlack Ġof Ġenough Ġtraining Ġdata . ĠB ERT Ġcan Ġbe Ġpre - trained Ġto Ġlearn Ġgeneral Ġor Ġspecific Ġlanguage Ġmodels Ġusing Ġvery Ġlarge Ġamounts Ġof Ġunl ab elled Ġtext Ġ( e . g . Ġweb Ġcontent , ĠWikipedia , Ġetc .), Ġand Ġthis Ġknowledge Ġcan Ġbe Ġtransferred Ġto Ġa Ġdifferent Ġdownstream Ġtask Ġin Ġa Ġprocess Ġthat Ġreceives Ġthe Ġname Ġfine - tun ing . </s> dev lin 2018 bert Ġhave Ġused Ġfine - tun ing Ġto Ġachieve Ġstate - of - the - art Ġresults Ġon Ġa Ġwide Ġvariety Ġof Ġchallenging Ġnatural Ġlanguage Ġtasks , Ġsuch Ġas Ġtext Ġclassification , ĠQuestion ĠAn sw ering Ġ( Q A ) Ġand ĠNamed ĠEntity ĠRecogn ition Ġand ĠClassification Ġ( NER C ). ĠB ERT Ġhas Ġalso Ġbeen Ġused Ġsuccessfully Ġby Ġother Ġcommunity Ġpractitioners Ġfor Ġa Ġwide Ġrange Ġof ĠN LP - related Ġtasks ĠB IB REF 8 , ĠB IB REF 9 . </s> Regarding Ġthe Ġtask Ġof Ġdata Ġanonym isation Ġin Ġparticular , Ġanonym isation Ġsystems Ġmay Ġfollow Ġdifferent Ġapproaches Ġand Ġpursue Ġdifferent Ġobjectives Ġ( C orm ode Ġand ĠS riv ast ava , Ġ2009 ). ĠThe Ġfirst Ġobjective Ġof Ġthese Ġsystems Ġis Ġto Ġdetect Ġand Ġclassify Ġthe Ġsensitive Ġinformation Ġcontained Ġin Ġthe Ġdocuments Ġto Ġbe Ġanonym ised . ĠIn Ġorder Ġto Ġachieve Ġthat , Ġthey Ġuse Ġrule - based Ġapproaches , ĠMachine ĠLearning Ġ( ML ) Ġapproaches , Ġor Ġa Ġcombination Ġof Ġboth . </s> Although Ġmost Ġof Ġthese Ġefforts Ġare Ġfor ĠEnglish Ġtexts ĠâĢĵ see , Ġamong Ġothers , Ġthe Ġi 2 b 2 Ġde - ident ification Ġchallenges ĠB IB REF 10 , ĠB IB REF 11 , Ġd ern on 2016 deep , Ġor Ġk hin 2018 deep âĢĵ , Ġother Ġlanguages Ġare Ġalso Ġattracting Ġgrowing Ġinterest . ĠSome Ġexamples Ġare Ġm amed e 2016 aut om ated Ġfor ĠPortuguese Ġand Ġt ve it 2004 an onym ization Ġfor ĠNorwegian . ĠWith Ġrespect Ġto Ġthe Ġanonym isation Ġof Ġtext Ġwritten Ġin ĠSpanish , Ġrecent Ġstudies Ġinclude Ġmed ina 2018 building , Ġhas san 2018 anon im iz acion Ġand Ġgar cia 2018 aut om ating . ĠMost Ġnotably , Ġin Ġ2019 Ġthe Ġfirst Ġcommunity Ġchallenge Ġabout Ġanonym isation Ġof Ġmedical Ġdocuments Ġin ĠSpanish , ĠMED DOC AN ĠB IB REF 3 , Ġwas Ġheld Ġas Ġpart Ġof Ġthe ĠI ber LE F Ġinitiative . ĠThe Ġwinners Ġof Ġthe Ġchallenge ĠâĢĵ the ĠNeither - Language - nor - Domain - Experts Ġ( NL N DE ) ĠB IB REF 12 âĢĵ Ġachieved ĠF 1 - sc ores Ġas Ġhigh Ġas Ġ0 . 975 Ġin Ġthe Ġtask Ġof Ġsensitive Ġinformation Ġdetection Ġand Ġcategor isation Ġby Ġusing Ġrecurrent Ġneural Ġnetworks Ġwith ĠCond itional ĠRandom ĠField Ġ( CR F ) Ġoutput Ġlayers . </s> At Ġthe Ġsame Ġchallenge , Ġm ao 2019 had oken Ġoccupied Ġthe Ġ8 th Ġposition Ġamong Ġ18 Ġparticipants Ġusing ĠB ERT . ĠAccording Ġto Ġthe Ġdescription Ġof Ġthe Ġsystem , Ġthe Ġauthors Ġused ĠB ERT - Base ĠMult ilingual ĠC ased Ġand Ġan Ġoutput ĠCR F Ġlayer . ĠHowever , Ġtheir Ġsystem Ġis Ġ$\ sim Ġ$ 3 ĠF 1 - score Ġpoints Ġbelow Ġour Ġimplementation Ġwithout Ġthe ĠCR F Ġlayer . </s> Materials Ġand ĠMethods </s> The Ġaim Ġof Ġthis Ġpaper Ġis Ġto Ġevaluate ĠB ERT 's Ġmult ilingual Ġmodel Ġand Ġcompare Ġit Ġto Ġother Ġestablished Ġmachine - learning Ġalgorithms Ġin Ġa Ġspecific Ġtask : Ġsensitive Ġdata Ġdetection Ġand Ġclassification Ġin ĠSpanish Ġclinical Ġfree Ġtext . ĠThis Ġsection Ġdescribes Ġthe Ġdata Ġinvolved Ġin Ġthe Ġexperiments Ġand Ġthe Ġsystems Ġevaluated . ĠFinally , Ġwe Ġintroduce Ġthe Ġexperimental Ġsetup . </s> Materials Ġand ĠMethods Ġ: :: ĠData </s> Two Ġdatasets Ġare Ġexploited Ġin Ġthis Ġarticle . ĠBoth Ġdatasets Ġconsist Ġof Ġplain Ġtext Ġcontaining Ġclinical Ġnarrative Ġwritten Ġin ĠSpanish , Ġand Ġtheir Ġrespective Ġmanual Ġannotations Ġof Ġsensitive Ġinformation Ġin ĠBR AT ĠB IB REF 13 Ġstandoff Ġformat . ĠIn Ġorder Ġto Ġfeed Ġthe Ġdata Ġto Ġthe Ġdifferent Ġalgorithms Ġpresented Ġin ĠSection ĠSEC REF 7 , Ġthese Ġdatasets Ġwere Ġtransformed Ġto Ġcomply Ġwith Ġthe Ġcommonly Ġused ĠB IO Ġsequence Ġrepresentation Ġscheme ĠB IB REF 14 . </s> Materials Ġand ĠMethods Ġ: :: ĠData Ġ: :: ĠN UB es - PH I </s> N UB es ĠB IB REF 4 Ġis Ġa Ġcorpus Ġof Ġaround Ġ7 , 000 Ġreal Ġmedical Ġreports Ġwritten Ġin ĠSpanish Ġand Ġannot ated Ġwith Ġneg ation Ġand Ġuncertainty Ġinformation . ĠBefore Ġbeing Ġpublished , Ġsensitive Ġinformation Ġhad Ġto Ġbe Ġmanually Ġannot ated Ġand Ġreplaced Ġfor Ġthe Ġcorpus Ġto Ġbe Ġsafely Ġshared . ĠIn Ġthis Ġarticle , Ġwe Ġwork Ġwith Ġthe ĠN UB es Ġversion Ġprior Ġto Ġits Ġanonym isation , Ġthat Ġis , Ġwith Ġthe Ġmanual Ġannotations Ġof Ġsensitive Ġinformation . ĠIt Ġfollows Ġthat Ġthe Ġversion Ġwe Ġwork Ġwith Ġis Ġnot Ġpublicly Ġavailable Ġand , Ġdue Ġto Ġcontractual Ġrestrictions , Ġwe Ġcannot Ġreveal Ġthe Ġproven ance Ġof Ġthe Ġdata . ĠIn Ġorder Ġto Ġavoid Ġconfusion Ġbetween Ġthe Ġtwo Ġcorpus Ġversions , Ġwe Ġhence forth Ġrefer Ġto Ġthe Ġversion Ġrelevant Ġin Ġthis Ġpaper Ġas ĠN UB es - PH I Ġ( from Ġ` N UB es Ġwith ĠPersonal ĠHealth ĠInformation '). </s> N UB es - PH I Ġconsists Ġof Ġ32 , 055 Ġsentences Ġannot ated Ġfor Ġ11 Ġdifferent Ġsensitive Ġinformation Ġcategories . ĠOverall , Ġit Ġcontains Ġ7 , 8 18 Ġannotations . ĠThe Ġcorpus Ġhas Ġbeen Ġrandomly Ġsplit Ġinto Ġtrain Ġ( 72 %), Ġdevelopment Ġ( 8 %) Ġand Ġtest Ġ( 20 %) Ġsets Ġto Ġconduct Ġthe Ġexperiments Ġdescribed Ġin Ġthis Ġpaper . ĠThe Ġsize Ġof Ġeach Ġsplit Ġand Ġthe Ġdistribution Ġof Ġthe Ġannotations Ġcan Ġbe Ġconsulted Ġin ĠTables Ġand Ġ, Ġrespectively . </s> The Ġmajority Ġof Ġsensitive Ġinformation Ġin ĠN UB es - PH I Ġare Ġtemporal Ġexpressions Ġ( ` Date ' Ġand Ġ` Time '), Ġfollowed Ġby Ġhealthcare Ġfacility Ġmentions Ġ( ` H ospital '), Ġand Ġthe Ġage Ġof Ġthe Ġpatient . ĠMent ions Ġof Ġpeople Ġare Ġnot Ġthat Ġfrequent , Ġwith Ġphysician Ġnames Ġ( ` Doctor ') Ġoccurring Ġmuch Ġmore Ġoften Ġthan Ġpatient Ġnames Ġ( ` Pat ient '). ĠThe Ġleast Ġfrequent Ġsensitive Ġinformation Ġtypes , Ġwhich Ġaccount Ġfor Ġ$\ sim Ġ$ 10 % Ġof Ġthe Ġremaining Ġannotations , Ġconsist Ġof Ġthe Ġpatient 's Ġsex , Ġjob , Ġand Ġkins hip , Ġand Ġlocations Ġother Ġthan Ġhealthcare Ġfacilities Ġ( ` Location '). ĠFinally , Ġthe Ġtag Ġ` Other ' Ġincludes , Ġfor Ġinstance , Ġmentions Ġto Ġinstitutions Ġunrelated Ġto Ġhealthcare Ġand Ġwhether Ġthe Ġpatient Ġis Ġright - Ġor Ġleft - handed . ĠIt Ġoccurs Ġjust Ġ36 Ġtimes . </s> Materials Ġand ĠMethods Ġ: :: ĠData Ġ: :: ĠThe ĠMED DOC AN Ġcorpus </s> The Ġorganisers Ġof Ġthe ĠMED DOC AN Ġshared Ġtask ĠB IB REF 3 Ġcurated Ġa Ġsynthetic Ġcorpus Ġof Ġclinical Ġcases Ġenriched Ġwith Ġsensitive Ġinformation Ġby Ġhealth Ġdocument al ists . ĠIn Ġthis Ġregard , Ġthe ĠMED DOC AN Ġevaluation Ġscenario Ġcould Ġbe Ġsaid Ġto Ġbe Ġsomewhat Ġfar Ġfrom Ġthe Ġreal Ġuse Ġcase Ġthe Ġtechnology Ġdeveloped Ġfor Ġthe Ġshared Ġtask Ġis Ġsupposed Ġto Ġbe Ġapplied Ġin . ĠHowever , Ġat Ġthe Ġmoment Ġit Ġalso Ġprovides Ġthe Ġonly Ġpublic Ġmeans Ġfor Ġa Ġrigorous Ġcomparison Ġbetween Ġsystems Ġfor Ġsensitive Ġhealth Ġinformation Ġdetection Ġin ĠSpanish Ġtexts . </s> The Ġsize Ġof Ġthe ĠMED DOC AN Ġcorpus Ġis Ġshown Ġin ĠTable Ġ. ĠCompared Ġto ĠN UB es - PH I Ġ( Table Ġ), Ġthis Ġcorpus Ġcontains Ġmore Ġsensitive Ġinformation Ġannotations , Ġboth Ġin Ġabsolute Ġand Ġrelative Ġterms . </s> The Ġsensitive Ġannotation Ġcategories Ġconsidered Ġin ĠMED DOC AN Ġdiffer Ġin Ġpart Ġfrom Ġthose Ġin ĠN UB es - PH I . ĠMost Ġnotably , Ġit Ġcontains Ġfiner - gr ained Ġlabels Ġfor Ġlocation - related Ġmentions ĠâĢĵ name ly , Ġ` Address ', Ġ` Ter rit ory ', Ġand Ġ` Country ' âĢĵ , Ġand Ġother Ġsensitive Ġinformation Ġcategories Ġthat Ġwe Ġdid Ġnot Ġencounter Ġin ĠN UB es - PH I Ġ( e . g ., Ġidentifiers , Ġphone Ġnumbers , Ġe - mail Ġaddresses , Ġetc .). ĠIn Ġtotal , Ġthe ĠMED DOC AN Ġcorpus Ġhas Ġ21 Ġsensitive Ġinformation Ġcategories . ĠWe Ġrefer Ġthe Ġreader Ġto Ġthe Ġorganisers ' Ġarticle ĠB IB REF 3 Ġfor Ġmore Ġdetailed Ġinformation Ġabout Ġthis Ġcorpus . </s> Materials Ġand ĠMethods Ġ: :: ĠSystems </s> Apart Ġfrom Ġexperimenting Ġwith Ġa Ġpre - trained ĠB ERT Ġmodel , Ġwe Ġhave Ġrun Ġexperiments Ġwith Ġother Ġsystems Ġand Ġbas elines , Ġto Ġcompare Ġthem Ġand Ġobtain Ġa Ġbetter Ġperspective Ġabout ĠB ERT 's Ġperformance Ġin Ġthese Ġdatasets . </s> Materials Ġand ĠMethods Ġ: :: ĠSystems Ġ: :: ĠBas eline </s> As Ġthe Ġsimplest Ġbaseline , Ġa Ġsensitive Ġdata Ġrecogn iser Ġand Ġclass ifier Ġhas Ġbeen Ġdeveloped Ġthat Ġconsists Ġof Ġregular - express ions Ġand Ġdictionary Ġlook - ups . ĠFor Ġeach Ġcategory Ġto Ġdetect Ġa Ġspecific Ġmethod Ġhas Ġbeen Ġimplemented . ĠFor Ġinstance , Ġthe ĠDate , ĠAge , ĠTime Ġand ĠDoctor Ġdetectors Ġare Ġbased Ġon Ġregular - express ions ; ĠHospital , ĠSex , ĠK ins hip , ĠLocation , ĠPatient Ġand ĠJob Ġare Ġlooked Ġup Ġin Ġdiction aries . ĠThe Ġdiction aries Ġare Ġhand - crafted Ġfrom Ġthe Ġtraining Ġdata Ġavailable , Ġexcept Ġfor Ġthe ĠPatient 's Ġcase , Ġfor Ġwhich Ġthe Ġpossible Ġcandidates Ġconsidered Ġare Ġthe Ġ100 Ġmost Ġcommon Ġfemale Ġand Ġmale Ġnames Ġin ĠSpain Ġaccording Ġto Ġthe ĠInstit uto ĠN ac ional Ġde ĠEst ad ÃŃ st ica Ġ( INE ; ĠSpanish ĠStatistical ĠOffice ). </s> Materials Ġand ĠMethods Ġ: :: ĠSystems Ġ: :: ĠCR F </s> Cond itional ĠRandom ĠFields Ġ( CR F ) ĠB IB REF 15 Ġhave Ġbeen Ġextensively Ġused Ġfor Ġtasks Ġof Ġsequential Ġnature . ĠIn Ġthis Ġpaper , Ġwe Ġpropose Ġas Ġone Ġof Ġthe Ġcompetitive Ġbas elines Ġa ĠCR F Ġclass ifier Ġtrained Ġwith Ġsk learn - cr f su ite Ġfor ĠPython Ġ3 . 5 Ġand Ġthe Ġfollowing Ġconfiguration : Ġalgorithm Ġ= Ġlb f gs ; Ġmaximum Ġiterations Ġ= Ġ100 ; Ġc 1 Ġ= Ġc 2 Ġ= Ġ0 . 1 ; Ġall Ġtransitions Ġ= Ġtrue ; Ġoptim ise Ġ= Ġfalse . ĠThe Ġfeatures Ġextracted Ġfrom Ġeach Ġtoken Ġare Ġas Ġfollows : </s> [ no item se p ] </s> prefix es Ġand Ġsuffix es Ġof Ġ2 Ġand Ġ3 Ġcharacters ; </s> the Ġlength Ġof Ġthe Ġtoken Ġin Ġcharacters Ġand Ġthe Ġlength Ġof Ġthe Ġsentence Ġin Ġtokens ; </s> whether Ġthe Ġtoken Ġis Ġall - letters , Ġa Ġnumber , Ġor Ġa Ġsequence Ġof Ġpunct uation Ġmarks ; </s> whether Ġthe Ġtoken Ġcontains Ġthe Ġcharacter Ġ` @ '; </s> whether Ġthe Ġtoken Ġis Ġthe Ġstart Ġor Ġend Ġof Ġthe Ġsentence ; </s> the Ġtoken 's Ġcasing Ġand Ġthe Ġratio Ġof Ġu pperc ase Ġcharacters , Ġdigits , Ġand Ġpunct uation Ġmarks Ġto Ġits Ġlength ; </s> and , Ġthe Ġle mma , Ġpart - of - speech Ġtag , Ġand Ġnamed - entity Ġtag Ġgiven Ġby Ġ ix a - p ipes ĠB IB REF 16 Ġupon Ġanalys ing Ġthe Ġsentence Ġthe Ġtoken Ġbelongs Ġto . </s> Notice ably , Ġnone Ġof Ġthe Ġfeatures Ġused Ġto Ġtrain Ġthe ĠCR F Ġclass ifier Ġis Ġdomain - dependent . ĠHowever , Ġthe Ġlatter Ġgroup Ġof Ġfeatures Ġis Ġlanguage Ġdependent . </s> Materials Ġand ĠMethods Ġ: :: ĠSystems Ġ: :: Ġspa Cy </s> sp a Cy Ġis Ġa Ġwidely Ġused ĠN LP Ġlibrary Ġthat Ġimplements Ġstate - of - the - art Ġtext Ġprocessing Ġpipelines , Ġincluding Ġa Ġsequence - lab elling Ġpipeline Ġsimilar Ġto Ġthe Ġone Ġdescribed Ġby Ġstr ub ell 2017 fast . Ġspa Cy Ġoffers Ġseveral Ġpre - trained Ġmodels Ġin ĠSpanish , Ġwhich Ġperform Ġbasic ĠN LP Ġtasks Ġsuch Ġas ĠNamed ĠEntity ĠRecogn ition Ġ( NER ). ĠIn Ġthis Ġpaper , Ġwe Ġhave Ġtrained Ġa Ġnew ĠN ER Ġmodel Ġto Ġdetect ĠN UB es - PH I Ġlabels . ĠFor Ġthis Ġpurpose , Ġthe Ġnew Ġmodel Ġuses Ġall Ġthe Ġlabels Ġof Ġthe Ġtraining Ġcorpus Ġcoded Ġwith Ġits Ġcontext Ġat Ġsentence Ġlevel . ĠThe Ġnetwork Ġoptim isation Ġparameters Ġand Ġdrop out Ġvalues Ġare Ġthe Ġones Ġrecommended Ġin Ġthe Ġdocumentation Ġfor Ġsmall Ġdatasets . ĠFinally , Ġthe Ġmodel Ġis Ġtrained Ġusing Ġbatches Ġof Ġsize Ġ64 . ĠNo Ġmore Ġfeatures Ġare Ġincluded , Ġso Ġthe Ġclass ifier Ġis Ġlanguage - dependent Ġbut Ġnot Ġdomain - dependent . </s> Materials Ġand ĠMethods Ġ: :: ĠSystems Ġ: :: ĠB ERT </s> As Ġintroduced Ġearlier , ĠB ERT Ġhas Ġshown Ġan Ġoutstanding Ġperformance Ġin ĠN ERC - like Ġtasks , Ġimproving Ġthe Ġstart - of - the - art Ġresults Ġfor Ġalmost Ġevery Ġdataset Ġand Ġlanguage . ĠWe Ġtake Ġthe Ġsame Ġapproach Ġhere , Ġby Ġusing Ġthe Ġmodel ĠB ERT - Base ĠMult ilingual ĠC ased Ġwith Ġa ĠFully ĠConnect ed Ġ( FC ) Ġlayer Ġon Ġtop Ġto Ġperform Ġa Ġfine - tun ing Ġof Ġthe Ġwhole Ġmodel Ġfor Ġan Ġanonym isation Ġtask Ġin ĠSpanish Ġclinical Ġdata . ĠOur Ġimplementation Ġis Ġbuilt Ġon ĠPy Tor ch Ġand Ġthe ĠPy Tor ch - Transform ers Ġlibrary ĠB IB REF 1 . ĠThe Ġtraining Ġphase Ġconsists Ġin Ġthe Ġfollowing Ġsteps Ġ( rough ly Ġdepicted Ġin ĠFigure Ġ): </s> Pre - processing : Ġsince Ġwe Ġare Ġrelying Ġon Ġa Ġpre - trained ĠB ERT Ġmodel , Ġwe Ġmust Ġmatch Ġthe Ġsame Ġconfiguration Ġby Ġusing Ġa Ġspecific Ġtoken isation Ġand Ġvocabulary . ĠB ERT Ġalso Ġneeds Ġthat Ġthe Ġinputs Ġcontains Ġspecial Ġtokens Ġto Ġsignal Ġthe Ġbeginning Ġand Ġthe Ġend Ġof Ġeach Ġsequence . </s> Fine - tun ing : Ġthe Ġpre - process ed Ġsequence Ġis Ġfed Ġinto Ġthe Ġmodel . ĠB ERT Ġoutputs Ġthe Ġcontextual Ġembed d ings Ġthat Ġencode Ġeach Ġof Ġthe Ġinput ted Ġtokens . ĠThis Ġembed ding Ġrepresentation Ġfor Ġeach Ġtoken Ġis Ġfed Ġinto Ġthe ĠFC Ġlinear Ġlayer Ġafter Ġa Ġdrop out Ġlayer Ġ( with Ġa Ġ0 . 1 Ġdrop out Ġprobability ), Ġwhich Ġin Ġturn Ġoutputs Ġthe Ġlog its Ġfor Ġeach Ġpossible Ġclass . ĠThe Ġcross - ent ropy Ġloss Ġfunction Ġis Ġcalculated Ġcomparing Ġthe Ġlog its Ġand Ġthe Ġgold Ġlabels , Ġand Ġthe Ġerror Ġis Ġback - prop ag ated Ġto Ġadjust Ġthe Ġmodel Ġparameters . </s> We Ġhave Ġtrained Ġthe Ġmodel Ġusing Ġan ĠAdam W Ġoptim iser ĠB IB REF 17 Ġwith Ġthe Ġlearning Ġrate Ġset Ġto Ġ3 e - 5 , Ġas Ġrecommended Ġby Ġdev lin 2018 bert , Ġand Ġwith Ġa Ġgradient Ġclipping Ġof Ġ1 . 0 . ĠWe Ġalso Ġapplied Ġa Ġlearning - rate Ġsched uler Ġthat Ġwar ms Ġup Ġthe Ġlearning Ġrate Ġfrom Ġzero Ġto Ġits Ġmaximum Ġvalue Ġas Ġthe Ġtraining Ġprogresses , Ġwhich Ġis Ġalso Ġa Ġcommon Ġpractice . ĠFor Ġeach Ġexperiment Ġset Ġproposed Ġbelow , Ġthe Ġtraining Ġwas Ġrun Ġwith Ġan Ġearly - st opping Ġpatience Ġof Ġ15 Ġepoch s . ĠThen , Ġthe Ġmodel Ġthat Ġperformed Ġbest Ġagainst Ġthe Ġdevelopment Ġset Ġwas Ġused Ġto Ġproduce Ġthe Ġreported Ġresults . </s> The Ġexperiments Ġwere Ġrun Ġon Ġa Ġ64 - core Ġserver Ġwith Ġoperating Ġsystem ĠUbuntu Ġ16 . 04 , Ġ250 GB Ġof ĠRAM Ġmemory , Ġand Ġ4 ĠGeForce ĠRTX Ġ20 80 ĠGPUs Ġwith Ġ11 GB Ġof Ġmemory . ĠThe Ġmaximum Ġsequence Ġlength Ġwas Ġset Ġat Ġ500 Ġand Ġthe Ġbatch Ġsize Ġat Ġ12 . ĠIn Ġthis Ġsetting , Ġeach Ġepoch ĠâĢĵ a Ġfull Ġpass Ġthrough Ġall Ġthe Ġtraining Ġdata âĢĵ Ġrequired Ġabout Ġ10 Ġminutes Ġto Ġcomplete . </s> Materials Ġand ĠMethods Ġ: :: ĠExperimental Ġdesign </s> We Ġhave Ġconducted Ġexperiments Ġwith ĠB ERT Ġin Ġthe Ġtwo Ġdatasets Ġof ĠSpanish Ġclinical Ġnarrative Ġpresented Ġin ĠSection ĠSEC REF 3 ĠThe Ġfirst Ġexperiment Ġset Ġuses ĠN UB es - PH I , Ġa Ġcorpus Ġof Ġreal Ġmedical Ġreports Ġmanually Ġannot ated Ġwith Ġsensitive Ġinformation . ĠBecause Ġthis Ġcorpus Ġis Ġnot Ġpublicly Ġavailable , Ġand Ġin Ġorder Ġto Ġcompare Ġthe ĠB ERT - based Ġmodel Ġto Ġother Ġrelated Ġpublished Ġsystems , Ġthe Ġsecond Ġset Ġof Ġexperiments Ġuses Ġthe ĠMED DOC AN Ġ2019 Ġshared Ġtask Ġcompetition Ġdataset . ĠThe Ġfollowing Ġsections Ġprovide Ġgreater Ġdetail Ġabout Ġthe Ġtwo Ġexperimental Ġsetups . </s> Materials Ġand ĠMethods Ġ: :: ĠExperimental Ġdesign Ġ: :: ĠExperiment ĠA : ĠN UB es - PH I </s> In Ġthis Ġexperiment Ġset , Ġwe Ġevaluate Ġall Ġthe Ġsystems Ġpresented Ġin ĠSection ĠSEC REF 7 , Ġnamely , Ġthe Ġrule - based Ġbaseline , Ġthe ĠCR F Ġclass ifier , Ġthe Ġspa Cy Ġentity Ġtag ger , Ġand ĠB ERT . ĠThe Ġevaluation Ġcomprises Ġthree Ġscenarios Ġof Ġincreasing Ġdifficulty : </s> [ no item se p ] </s> - ĠEval uates Ġthe Ġperformance Ġof Ġthe Ġsystems Ġat Ġpredicting Ġwhether Ġeach Ġtoken Ġis Ġsensitive Ġor Ġnon - sensitive ; Ġthat Ġis , Ġthe Ġmeasurements Ġonly Ġtake Ġinto Ġaccount Ġwhether Ġa Ġsensitive Ġtoken Ġhas Ġbeen Ġrecognised Ġor Ġnot , Ġregardless Ġof Ġthe ĠB IO Ġlabel Ġand Ġthe Ġcategory Ġassigned . ĠThis Ġscenario Ġshows Ġhow Ġgood Ġa Ġsystem Ġwould Ġbe Ġat Ġobfusc ating Ġsensitive Ġdata Ġ( e . g ., Ġby Ġreplacing Ġsensitive Ġtokens Ġwith Ġaster isks ). </s> - ĠWe Ġmeasure Ġthe Ġperformance Ġof Ġthe Ġsystems Ġat Ġpredicting Ġthe Ġsensitive Ġinformation Ġtype Ġof Ġeach Ġtoken ĠâĢĵ i . e ., Ġthe Ġ11 Ġcategories Ġpresented Ġin ĠSection ĠSEC REF 5 Ġor Ġ` out '. ĠDetect ing Ġentity Ġtypes Ġcorrectly Ġis Ġimportant Ġif Ġa Ġsystem Ġis Ġgoing Ġto Ġbe Ġused Ġto Ġreplace Ġsensitive Ġdata Ġby Ġfake Ġdata Ġof Ġthe Ġsame Ġtype Ġ( e . g ., Ġrandom Ġpeople Ġnames ). </s> - ĠThis Ġis Ġthe Ġstrict est Ġevaluation , Ġas Ġit Ġtakes Ġinto Ġaccount Ġboth Ġthe ĠB IO Ġlabel Ġand Ġthe Ġcategory Ġassigned Ġto Ġeach Ġindividual Ġtoken . ĠBeing Ġable Ġto Ġdiscern Ġbetween Ġtwo Ġcontiguous Ġsensitive Ġentities Ġof Ġthe Ġsame Ġtype Ġis Ġrelevant Ġnot Ġonly Ġbecause Ġit Ġis Ġhelpful Ġwhen Ġproducing Ġfake Ġreplacements , Ġbut Ġbecause Ġit Ġalso Ġyields Ġmore Ġaccurate Ġstatistics Ġof Ġthe Ġsensitive Ġinformation Ġpresent Ġin Ġa Ġgiven Ġdocument Ġcollection . </s> The Ġsystems Ġare Ġevaluated Ġin Ġterms Ġof Ġmicro - average Ġprecision , Ġrecall Ġand ĠF 1 - score Ġin Ġall Ġthe Ġscenarios . </s> In Ġaddition Ġto Ġthe Ġscenarios Ġproposed , Ġa Ġsubject Ġworth Ġbeing Ġstudied Ġis Ġthe Ġneed Ġof Ġlabelled Ġdata . ĠMan ually Ġlabelled Ġdata Ġis Ġan Ġscarce Ġand Ġexpensive Ġresource , Ġwhich Ġfor Ġsome Ġapplication Ġdomains Ġor Ġlanguages Ġis Ġdifficult Ġto Ġcome Ġby . ĠIn Ġorder Ġto Ġobtain Ġan Ġestimation Ġof Ġthe Ġdependency Ġof Ġeach Ġsystem Ġon Ġthe Ġavailable Ġamount Ġof Ġtraining Ġdata , Ġwe Ġhave Ġret rained Ġall Ġthe Ġcompared Ġmodels Ġusing Ġdecreasing Ġamounts Ġof Ġdata ĠâĢĵ from Ġ100 % Ġof Ġthe Ġavailable Ġtraining Ġinstances Ġto Ġjust Ġ1 %. ĠThe Ġsame Ġdata Ġsubs ets Ġhave Ġbeen Ġused Ġto Ġtrain Ġall Ġthe Ġsystems . ĠDue Ġto Ġthe Ġknowledge Ġtransferred Ġfrom Ġthe Ġpre - trained ĠB ERT Ġmodel , Ġthe ĠB ERT - based Ġmodel Ġis Ġexpected Ġto Ġbe Ġmore Ġrobust Ġto Ġdata Ġscarcity Ġthan Ġthose Ġthat Ġstart Ġtheir Ġtraining Ġfrom Ġscratch . </s> Materials Ġand ĠMethods Ġ: :: ĠExperimental Ġdesign Ġ: :: ĠExperiment ĠB : ĠMED DOC AN </s> In Ġthis Ġexperiment Ġset , Ġour ĠB ERT Ġimplementation Ġis Ġcompared Ġto Ġseveral Ġsystems Ġthat Ġparticipated Ġin Ġthe ĠMED DOC AN Ġchallenge : Ġa ĠCR F Ġclass ifier ĠB IB REF 18 , Ġa Ġspa Cy Ġentity Ġrecogn iser ĠB IB REF 18 , Ġand ĠNL N DE ĠB IB REF 12 , Ġthe Ġwinner Ġof Ġthe Ġshared Ġtask Ġand Ġcurrent Ġstate Ġof Ġthe Ġart Ġfor Ġsensitive Ġinformation Ġdetection Ġand Ġclassification Ġin ĠSpanish Ġclinical Ġtext . ĠSpecifically , Ġwe Ġinclude Ġthe Ġresults Ġof Ġa Ġdomain - independent ĠNL N DE Ġmodel Ġ( S 2 ), Ġand Ġthe Ġresults Ġof Ġa Ġmodel Ġenriched Ġwith Ġdomain - specific Ġembed d ings Ġ( S 3 ). ĠFinally , Ġwe Ġinclude Ġthe Ġresults Ġobtained Ġby Ġm ao 2019 had oken Ġwith Ġa ĠCR F Ġoutput Ġlayer Ġon Ġtop Ġof ĠB ERT Ġembed d ings . ĠMED DOC AN Ġconsists Ġof Ġtwo Ġscenarios : </s> [ no item se p ] </s> - ĠThis Ġevaluation Ġmeasures Ġhow Ġgood Ġa Ġsystem Ġis Ġat Ġdetecting Ġsensitive Ġtext Ġspans , Ġregardless Ġof Ġthe Ġcategory Ġassigned Ġto Ġthem . </s> - ĠIn Ġthis Ġscenario , Ġsystems Ġare Ġrequired Ġto Ġmatch Ġexactly Ġnot Ġonly Ġthe Ġboundaries Ġof Ġeach Ġsensitive Ġspan , Ġbut Ġalso Ġthe Ġcategory Ġassigned . </s> The Ġsystems Ġare Ġevaluated Ġin Ġterms Ġof Ġmicro - aver aged Ġprecision , Ġrecall Ġand ĠF - 1 Ġscore . ĠNote Ġthat , Ġin Ġcontrast Ġto Ġthe Ġevaluation Ġin ĠExperiment ĠA , ĠMED DOC AN Ġmeasurements Ġare Ġentity - based Ġinstead Ġof Ġtoken wise . ĠAn Ġexhaustive Ġexplanation Ġof Ġthe ĠMED DOC AN Ġevaluation Ġprocedure Ġis Ġavailable Ġonline , Ġas Ġwell Ġas Ġthe Ġofficial Ġevaluation Ġscript , Ġwhich Ġwe Ġused Ġto Ġobtain Ġthe Ġreported Ġresults . </s> Results </s> This Ġsection Ġdescribes Ġthe Ġresults Ġobtained Ġin Ġthe Ġtwo Ġsets Ġof Ġexperiments : ĠN UB es - PH I Ġand ĠMED DOC AN . </s> Results Ġ: :: ĠExperiment ĠA : ĠN UB es - PH I </s> Table Ġshows Ġthe Ġresults Ġof Ġthe Ġconducted Ġexperiments Ġin ĠN UB es - PH I Ġfor Ġall Ġthe Ġcompared Ġsystems . ĠThe Ġincluded Ġbaseline Ġserves Ġto Ġgive Ġa Ġquick Ġinsight Ġabout Ġhow Ġchallenging Ġthe Ġdata Ġis . ĠWith Ġsimple Ġregular Ġexpressions Ġand Ġg az ette ers Ġa Ġprecision Ġof Ġ0 . 8 53 Ġis Ġobtained . ĠOn Ġthe Ġother Ġhand , Ġthe Ġrecall , Ġwhich Ġdirectly Ġdepends Ġon Ġthe Ġcoverage Ġprovided Ġby Ġthe Ġrules Ġand Ġresources , Ġdrops Ġto Ġ0 . 469 . ĠHence , Ġthis Ġtask Ġis Ġunlikely Ġto Ġbe Ġsolved Ġwithout Ġthe Ġgeneral isation Ġcapabilities Ġprovided Ġby Ġmachine - learning Ġand Ġdeep - learning Ġmodels . </s> Regarding Ġthe Ġdetection Ġscenario ĠâĢĵ that Ġis , Ġthe Ġscenario Ġconcerned Ġwith Ġa Ġbinary Ġclassification Ġto Ġdetermine Ġwhether Ġeach Ġindividual Ġtoken Ġconve ys Ġsensitive Ġinformation Ġor Ġnot âĢĵ , Ġit Ġcan Ġbe Ġobserved Ġthat ĠB ERT Ġoutper forms Ġits Ġcompetitors . ĠA Ġfact Ġworth Ġhighlighting Ġis Ġthat , Ġaccording Ġto Ġthese Ġresults , ĠB ERT Ġachieves Ġa Ġprecision Ġlower Ġthan Ġthe Ġrest Ġof Ġthe Ġsystems Ġ( i . e ., Ġit Ġmakes Ġmore Ġfalse Ġpositive Ġpredictions ); Ġin Ġexchange , Ġit Ġob tains Ġa Ġremarkably Ġhigher Ġrecall . ĠNotice ably , Ġit Ġreaches Ġa Ġrecall Ġof Ġ0 . 9 79 , Ġimproving Ġby Ġmore Ġthan Ġ4 Ġpoints Ġthe Ġsecond - best Ġsystem , Ġspa Cy . </s> The Ġtable Ġalso Ġshows Ġthe Ġresults Ġfor Ġthe Ġrelaxed Ġmetric Ġthat Ġonly Ġtakes Ġinto Ġaccount Ġthe Ġentity Ġtype Ġdetected , Ġregardless Ġof Ġthe ĠB IO Ġlabel Ġ( i . e ., Ġignoring Ġwhether Ġthe Ġtoken Ġis Ġat Ġthe Ġbeginning Ġor Ġin Ġthe Ġmiddle Ġof Ġa Ġsensitive Ġsequence Ġof Ġtokens ). ĠThe Ġconclusions Ġare Ġvery Ġsimilar Ġto Ġthose Ġextracted Ġpreviously , Ġwith ĠB ERT Ġgaining Ġ2 . 1 Ġpoints Ġof ĠF 1 - score Ġover Ġthe ĠCR F Ġbased Ġapproach . ĠThe Ġconfusion Ġmat rices Ġof Ġthe Ġpredictions Ġmade Ġby ĠCR F , Ġspa Cy , Ġand ĠB ERT Ġin Ġthis Ġscenario Ġare Ġshown Ġin ĠTable Ġ. ĠAs Ġcan Ġbee Ġseen , ĠB ERT Ġhas Ġless Ġdifficulty Ġin Ġpredicting Ġcorrectly Ġless Ġfrequent Ġcategories , Ġsuch Ġas Ġ` Location ', Ġ` Job ', Ġand Ġ` Pat ient '. ĠOne Ġof Ġthe Ġmost Ġcommon Ġmistakes Ġaccording Ġto Ġthe Ġconfusion Ġmat rices Ġis Ġclass ifying Ġhospital Ġnames Ġas Ġ` Location ' Ġinstead Ġof Ġthe Ġmore Ġaccurate Ġ` H ospital '; Ġthis Ġis Ġhardly Ġa Ġharmful Ġerror , Ġgiven Ġthat Ġa Ġhospital Ġis Ġactually Ġa Ġlocation . ĠLast , Ġthe Ġcategory Ġ` Other ' Ġis Ġcompletely Ġleaked Ġby Ġall Ġthe Ġcompared Ġsystems , Ġmost Ġlikely Ġdue Ġto Ġits Ġalmost Ġtotal Ġlack Ġof Ġsupport Ġin Ġboth Ġtraining Ġand Ġevaluation Ġdatasets . </s> To Ġfinish Ġwith Ġthis Ġexperiment Ġset , ĠTable Ġalso Ġshows Ġthe Ġstrict Ġclassification Ġprecision , Ġrecall Ġand ĠF 1 - score Ġfor Ġthe Ġcompared Ġsystems . ĠDespite Ġthe Ġfact Ġthat , Ġin Ġgeneral , Ġthe Ġsystems Ġobtain Ġhigh Ġvalues , ĠB ERT Ġoutper forms Ġthem Ġagain . ĠB ERT 's ĠF 1 - score Ġis Ġ1 . 9 Ġpoints Ġhigher Ġthan Ġthe Ġnext Ġmost Ġcompetitive Ġresult Ġin Ġthe Ġcomparison . ĠMore Ġremarkably , Ġthe Ġrecall Ġobtained Ġby ĠB ERT Ġis Ġabout Ġ5 Ġpoints Ġabove . </s> Upon Ġmanual Ġinspection Ġof Ġthe Ġerrors Ġcommitted Ġby Ġthe ĠB ERT - based Ġmodel , Ġwe Ġdiscovered Ġthat Ġit Ġhas Ġa Ġslight Ġtendency Ġtowards Ġproducing Ġill - formed ĠB IO Ġsequences Ġ( e . g , Ġstarting Ġa Ġsensitive Ġspan Ġwith Ġ` Inside ' Ġinstead Ġof Ġ` Begin '; Ġsee ĠTable Ġ). ĠWe Ġcould Ġexpect Ġthat Ġcomplement ing Ġthe ĠB ERT - based Ġmodel Ġwith Ġa ĠCR F Ġlayer Ġon Ġtop Ġwould Ġhelp Ġenforce Ġthe Ġemission Ġof Ġvalid Ġsequences , Ġallev iating Ġthis Ġkind Ġof Ġerrors Ġand Ġfurther Ġimproving Ġits Ġresults . </s> Finally , ĠFigure Ġshows Ġthe Ġimpact Ġof Ġdecreasing Ġthe Ġamount Ġof Ġtraining Ġdata Ġin Ġthe Ġdetection Ġscenario . ĠIt Ġshows Ġthe Ġdifference Ġin Ġprecision , Ġrecall , Ġand ĠF 1 - score Ġwith Ġrespect Ġto Ġthat Ġobtained Ġusing Ġ100 % Ġof Ġthe Ġtraining Ġdata . ĠA Ġgeneral Ġdownward Ġtrend Ġcan Ġbe Ġobserved , Ġas Ġone Ġwould Ġexpect : Ġless Ġtraining Ġdata Ġleads Ġto Ġless Ġaccurate Ġpredictions . ĠHowever , Ġthe ĠB ERT - based Ġmodel Ġis Ġthe Ġmost Ġrobust Ġto Ġtraining - data Ġreduction , Ġshowing Ġan Ġsteadily Ġlow Ġperformance Ġloss . ĠWith Ġ1 % Ġof Ġthe Ġdataset Ġ( 230 Ġtraining Ġinstances ), Ġthe ĠB ERT - based Ġmodel Ġonly Ġsuffers Ġa Ġstriking Ġ7 - point ĠF 1 - score Ġloss , Ġin Ġcontrast Ġto Ġthe Ġ32 Ġand Ġ39 Ġpoints Ġlost Ġby Ġthe ĠCR F Ġand Ġspa Cy Ġmodels , Ġrespectively . ĠThis Ġsteep Ġperformance Ġdrop Ġstems Ġto Ġa Ġlarger Ġextent Ġfrom Ġrecall Ġdecline , Ġwhich Ġis Ġnot Ġthat Ġmarked Ġin Ġthe Ġcase Ġof ĠB ERT . ĠOverall , Ġthese Ġresults Ġindicate Ġthat Ġthe Ġtransfer - learning Ġachieved Ġthrough Ġthe ĠB ERT Ġmult ilingual Ġpre - trained Ġmodel Ġnot Ġonly Ġhelps Ġobtain Ġbetter Ġresults , Ġbut Ġalso Ġlowers Ġthe Ġneed Ġof Ġmanually Ġlabelled Ġdata Ġfor Ġthis Ġapplication Ġdomain . </s> Results Ġ: :: ĠExperiment ĠB : ĠMED DOC AN </s> The Ġresults Ġof Ġthe Ġtwo ĠMED DOC AN Ġscenarios ĠâĢĵ det ection Ġand Ġclassification âĢĵ Ġare Ġshown Ġin ĠTable Ġ. ĠThese Ġresults Ġfollow Ġthe Ġsame Ġpattern Ġas Ġin Ġthe Ġprevious Ġexperiments , Ġwith Ġthe ĠCR F Ġclass ifier Ġbeing Ġthe Ġmost Ġprecise Ġof Ġall , Ġand ĠB ERT Ġoutper forming Ġboth Ġthe ĠCR F Ġand Ġspa Cy Ġclass ifiers Ġthanks Ġto Ġits Ġgreater Ġrecall . ĠWe Ġalso Ġshow Ġthe Ġresults Ġof Ġm ao 2019 had oken Ġwho , Ġdespite Ġof Ġhaving Ġused Ġa ĠB ERT - based Ġsystem , Ġachieve Ġlower Ġscores Ġthan Ġour Ġmodels . ĠThe Ġreason Ġwhy Ġit Ġshould Ġbe Ġso Ġremain Ġunclear . </s> With Ġregard Ġto Ġthe Ġwinner Ġof Ġthe ĠMED DOC AN Ġshared Ġtask , Ġthe ĠB ERT - based Ġmodel Ġhas Ġnot Ġimproved Ġthe Ġscores Ġobtained Ġby Ġneither Ġthe Ġdomain - dependent Ġ( S 3 ) Ġnor Ġthe Ġdomain - independent Ġ( S 2 ) ĠNL N DE Ġmodel . ĠHowever , Ġattending Ġto Ġthe Ġobtained Ġresults , ĠB ERT Ġremains Ġonly Ġ0 . 3 ĠF 1 - score Ġpoints Ġbehind , Ġand Ġwould Ġhave Ġachieved Ġthe Ġsecond Ġposition Ġamong Ġall Ġthe ĠMED DOC AN Ġshared Ġtask Ġcompetitors . ĠTaking Ġinto Ġaccount Ġthat Ġonly Ġ3 % Ġof Ġthe Ġgold Ġlabels Ġremain Ġincorrectly Ġannot ated , Ġthe Ġtask Ġcan Ġbe Ġconsidered Ġalmost Ġsolved , Ġand Ġit Ġis Ġnot Ġclear Ġif Ġthe Ġdifferences Ġamong Ġthe Ġsystems Ġare Ġactually Ġsignificant , Ġor Ġwhether Ġthey Ġstem Ġfrom Ġminor Ġvariations Ġin Ġinitial isation Ġor Ġa Ġlong - tail Ġof Ġminor Ġlab elling Ġinconsistencies . </s> Con clusions Ġand ĠFuture ĠWork </s> In Ġthis Ġwork Ġwe Ġhave Ġbriefly Ġintroduced Ġthe Ġproblems Ġrelated Ġto Ġdata Ġprivacy Ġprotection Ġin Ġclinical Ġdomain . ĠWe Ġhave Ġalso Ġdescribed Ġsome Ġof Ġthe Ġgroundbreaking Ġadvances Ġon Ġthe ĠNatural ĠLanguage ĠProcessing Ġfield Ġdue Ġto Ġthe Ġappearance Ġof ĠTransformers - based Ġdeep - learning Ġarchitectures Ġand Ġtransfer Ġlearning Ġfrom Ġvery Ġlarge Ġgeneral - domain Ġmult ilingual Ġcorpor a , Ġfocusing Ġour Ġattention Ġin Ġone Ġof Ġits Ġmost Ġrepresentative Ġexamples , ĠGoogle 's ĠB ERT Ġmodel . </s> In Ġorder Ġto Ġassess Ġthe Ġperformance Ġof ĠB ERT Ġfor ĠSpanish Ġclinical Ġdata Ġanonym isation , Ġwe Ġhave Ġconducted Ġseveral Ġexperiments Ġwith Ġa ĠB ERT - based Ġsequence Ġlab elling Ġapproach Ġusing Ġthe Ġpre - trained Ġmult ilingual ĠB ERT Ġmodel Ġshared Ġby ĠGoogle Ġas Ġthe Ġstarting Ġpoint Ġfor Ġthe Ġmodel Ġtraining . ĠWe Ġhave Ġcompared Ġthis ĠB ERT - based Ġsequence Ġlab elling Ġagainst Ġother Ġmethods Ġand Ġsystems . ĠOne Ġof Ġthe Ġexperiments Ġuses Ġthe ĠMED DOC AN Ġ2019 Ġshared Ġtask Ġdataset , Ġwhile Ġthe Ġother Ġuses Ġa Ġnovel ĠSpanish Ġclinical Ġreports Ġdataset Ġcalled ĠN UB es - PH I . </s> The Ġresults Ġof Ġthe Ġexperiments Ġshow Ġthat , Ġin ĠN UB es - PH I , Ġthe ĠB ERT - based Ġmodel Ġoutper forms Ġthe Ġother Ġsystems Ġwithout Ġrequiring Ġany Ġadaptation Ġor Ġdomain - specific Ġfeature Ġengineering , Ġjust Ġby Ġbeing Ġtrained Ġon Ġthe Ġprovided Ġlabelled Ġdata . ĠInterestingly , Ġthe ĠB ERT - based Ġmodel Ġob tains Ġa Ġremarkably Ġhigher Ġrecall Ġthan Ġthe Ġother Ġsystems . ĠHigh Ġrecall Ġis Ġa Ġdesirable Ġoutcome Ġbecause , Ġwhen Ġanonym ising Ġsensible Ġdocuments , Ġthe Ġaccidental Ġleak Ġof Ġsensible Ġdata Ġis Ġlikely Ġto Ġbe Ġmore Ġdangerous Ġthan Ġthe Ġunintended Ġover - ob fusc ation Ġof Ġnon - sensitive Ġtext . </s> Further , Ġwe Ġhave Ġconducted Ġan Ġadditional Ġexperiment Ġon Ġthis Ġdataset Ġby Ġprogressively Ġreducing Ġthe Ġtraining Ġdata Ġfor Ġall Ġthe Ġcompared Ġsystems . ĠThe ĠB ERT - based Ġmodel Ġshows Ġthe Ġhighest Ġrobust ness Ġto Ġtraining - data Ġscarcity , Ġlo osing Ġonly Ġ7 Ġpoints Ġof ĠF 1 - score Ġwhen Ġtrained Ġon Ġ230 Ġinstances Ġinstead Ġof Ġ21 , 371 . ĠThese Ġobservation Ġare Ġin Ġline Ġwith Ġthe Ġresults Ġobtained Ġby Ġthe ĠN LP Ġcommunity Ġusing ĠB ERT Ġfor Ġother Ġtasks . </s> The Ġexperiments Ġwith Ġthe ĠMED DOC AN Ġ2019 Ġshared Ġtask Ġdataset Ġfollow Ġthe Ġsame Ġpattern . ĠIn Ġthis Ġcase , Ġthe ĠB ERT - based Ġmodel Ġfalls Ġ0 . 3 ĠF 1 - score Ġpoints Ġbehind Ġthe Ġshared Ġtask Ġwinning Ġsystem , Ġbut Ġit Ġwould Ġhave Ġachieved Ġthe Ġsecond Ġposition Ġin Ġthe Ġcompetition Ġwith Ġno Ġfurther Ġrefinement . </s> Since Ġwe Ġhave Ġused Ġa Ġpre - trained Ġmult ilingual ĠB ERT Ġmodel , Ġthe Ġsame Ġapproach Ġis Ġlikely Ġto Ġwork Ġfor Ġother Ġlanguages Ġjust Ġby Ġproviding Ġsome Ġlabelled Ġtraining Ġdata . ĠFurther , Ġthis Ġis Ġthe Ġsimplest Ġfine - tun ing Ġthat Ġcan Ġbe Ġperformed Ġbased Ġon ĠB ERT . ĠMore Ġsophisticated Ġfine - tun ing Ġlayers Ġcould Ġhelp Ġimprove Ġthe Ġresults . ĠFor Ġexample , Ġit Ġcould Ġbe Ġexpected Ġthat Ġa ĠCR F Ġlayer Ġhelped Ġenforce Ġbetter ĠB IO Ġtagging Ġsequence Ġpredictions . ĠPrec ise ly , Ġm ao 2019 had oken Ġparticipated Ġin Ġthe ĠMED DOC AN Ġcompetition Ġusing Ġa ĠB ERT + CR F Ġarchitecture , Ġbut Ġtheir Ġreported Ġscores Ġare Ġabout Ġ3 Ġpoints Ġlower Ġthan Ġour Ġimplementation . ĠFrom Ġthe Ġdescription Ġof Ġtheir Ġwork , Ġit Ġis Ġunclear Ġwhat Ġthe Ġsource Ġof Ġthis Ġscore Ġdifference Ġcould Ġbe . </s> Further , Ġat Ġthe Ġtime Ġof Ġwriting Ġthis Ġpaper , Ġnew Ġmult ilingual Ġpre - trained Ġmodels Ġand ĠTrans former Ġarchitectures Ġhave Ġbecome Ġavailable . ĠIt Ġwould Ġnot Ġcome Ġas Ġa Ġsurprise Ġthat Ġthese Ġnew Ġresources Ġand Ġsystems ĠâĢĵ e . g ., ĠXL M - Ro BER Ta ĠB IB REF 19 Ġor ĠBET O ĠB IB REF 20 , Ġa ĠB ERT Ġmodel Ġfully Ġpre - trained Ġon ĠSpanish Ġtexts âĢĵ Ġfurther Ġadvanced Ġthe Ġstate Ġof Ġthe Ġart Ġin Ġthis Ġtask . </s> Acknowled gements </s> This Ġwork Ġhas Ġbeen Ġsupported Ġby ĠVic om tech Ġand Ġpartially Ġfunded Ġby Ġthe Ġproject ĠDeep Reading Ġ( RT I 2018 - 09 68 46 - B - C 21 , ĠMC IU / AE I / F ED ER , UE ).