0.5115
ABSTRACTIVE
0.6103
EXTRACTIVE
FLOAT SELECTED: Table 1: Factoid Questions. In Batch 3 we obtained the highest score. Also the relative distance between our best system and the top performing system shrunk between Batch 4 and 5.
Sharma et al. BIBREF3 describe a system with two stage process for factoid and list type question answering. Their system extracts relevant entities and then runs supervised classifier to rank the entities. Wiese et al. BIBREF4 propose neural network based model for Factoid and List-type question answering task. The model is based on Fast QA and predicts the answer span in the passage for a given question. The model is trained on SQuAD data set and fine tuned on the BioASQ data. Dimitriadis et al. BIBREF5 proposed two stage process for Factoid question answering task. Their system uses general purpose tools such as Metamap, BeCas to identify candidate sentences. These candidate sentences are represented in the form of features, and are then ranked by the binary classifier. Classifier is trained on candidate sentences extracted from relevant questions, snippets and correct answers from BioASQ challenge. For factoid question answering task highest ‘MRR’ achieved in the 6th edition of BioASQ competition is ‘0.4325’. Our system is a neural network model based on contextual word embeddings BIBREF1 and achieved a ‘MRR’ score ‘0.6103’ in one of the test batches for Factoid Question Answering task.
2002.01984
Introduction </s> Bio AS Q Ġis Ġa Ġbiomedical Ġdocument Ġclassification , Ġdocument Ġretrieval , Ġand Ġquestion Ġanswering Ġcompetition , Ġcurrently Ġin Ġits Ġseventh Ġyear . ĠWe Ġprovide Ġan Ġoverview Ġof Ġour Ġsubmissions Ġto Ġsemantic Ġquestion Ġanswering Ġtask Ġ( 7 b , ĠPhase ĠB ) Ġof ĠBio AS Q Ġ7 Ġ( except Ġfor Ġ' ide al Ġanswer ' Ġtest , Ġin Ġwhich Ġwe Ġdid Ġnot Ġparticipate Ġthis Ġyear ). ĠIn Ġthis Ġtask Ġsystems Ġare Ġprovided Ġwith Ġbiomedical Ġquestions Ġand Ġare Ġrequired Ġto Ġsubmit Ġideal Ġand Ġexact Ġanswers Ġto Ġthose Ġquestions . ĠWe Ġhave Ġused ĠBio BER T ĠB IB REF 0 Ġbased Ġsystem Ġ, Ġsee Ġalso ĠBid irection al ĠEnc oder ĠRepresent ations Ġfrom ĠTransformers ( BER T ) ĠB IB REF 1 , Ġand Ġwe Ġfine Ġtuned Ġit Ġfor Ġthe Ġbiomedical Ġquestion Ġanswering Ġtask . ĠOur Ġsystem Ġscored Ġnear Ġthe Ġtop Ġfor Ġfact oid Ġquestions Ġfor Ġall Ġthe Ġbatches Ġof Ġthe Ġchallenge . ĠMore Ġspec if ially , Ġin Ġthe Ġthird Ġtest Ġbatch Ġset , Ġour Ġsystem Ġachieved Ġhighest ĠâĢ ĺ MR R âĢ Ļ Ġscore Ġfor ĠFact oid ĠQuestion ĠAn sw ering Ġtask . ĠAlso , Ġfor ĠList - type Ġquestion Ġanswering Ġtask Ġour Ġsystem Ġachieved Ġhighest Ġrecall Ġscore Ġin Ġthe Ġfourth Ġtest Ġbatch Ġset . ĠAlong Ġwith Ġour Ġdetailed Ġapproach , Ġwe Ġpresent Ġthe Ġresults Ġfor Ġour Ġsubmissions Ġand Ġalso Ġhighlight Ġidentified Ġdowns ides Ġfor Ġour Ġcurrent Ġapproach Ġand Ġways Ġto Ġimprove Ġthem Ġin Ġour Ġfuture Ġexperiments . ĠIn Ġlast Ġtest Ġbatch Ġresults Ġwe Ġplaced Ġ4 th Ġfor ĠList - type Ġquestions Ġand Ġ3 rd Ġfor ĠFact oid - type Ġquestions .) </s> The ĠQ A Ġtask Ġis Ġorganized Ġin Ġtwo Ġphases . ĠPhase ĠA Ġdeals Ġwith Ġretrieval Ġof Ġthe Ġrelevant Ġdocument , Ġsnippets , Ġconcepts , Ġand ĠR DF Ġtri ples , Ġand Ġphase ĠB Ġdeals Ġwith Ġexact Ġand Ġideal Ġanswer Ġgenerations Ġ( which Ġis Ġa Ġparagraph Ġsize Ġsummary Ġof Ġsnippets ). ĠEx act Ġanswer Ġgeneration Ġis Ġrequired Ġfor Ġfact oid , Ġlist , Ġand Ġyes / no Ġtype Ġquestion . </s> Bio AS Q Ġorganizers Ġprovide Ġthe Ġtraining Ġand Ġtesting Ġdata . ĠThe Ġtraining Ġdata Ġconsists Ġof Ġquestions , Ġgold Ġstandard Ġdocuments , Ġsnippets , Ġconcepts , Ġand Ġideal Ġanswers Ġ( which Ġwe Ġdid Ġnot Ġuse Ġin Ġthis Ġpaper , Ġbut Ġwe Ġused Ġlast Ġyear ĠB IB REF 2 ). ĠThe Ġtest Ġdata Ġis Ġsplit Ġbetween Ġphases ĠA Ġand ĠB . ĠThe ĠPhase ĠA Ġdataset Ġconsists Ġof Ġthe Ġquestions , Ġunique Ġ ids , Ġquestion Ġtypes . ĠThe ĠPhase ĠB Ġdataset Ġconsists Ġof Ġthe Ġquestions , Ġgolden Ġstandard Ġdocuments , Ġsnippets , Ġunique Ġ ids Ġand Ġquestion Ġtypes . ĠEx act Ġanswers Ġfor Ġfact oid Ġtype Ġquestions Ġare Ġevaluated Ġusing Ġstrict Ġaccuracy Ġ( the Ġtop Ġanswer ), Ġlen ient Ġaccuracy Ġ( the Ġtop Ġ5 Ġanswers ), Ġand ĠMR R Ġ( Me an ĠRec ip rocal ĠRank ) Ġwhich Ġtakes Ġinto Ġaccount Ġthe Ġranks Ġof Ġreturned Ġanswers . ĠAnswers Ġfor Ġthe Ġlist Ġtype Ġquestion Ġare Ġevaluated Ġusing Ġprecision , Ġrecall , Ġand ĠF - me asure . </s> Related ĠWork Ġ: :: ĠBio As q </s> Sh arma Ġet Ġal . ĠB IB REF 3 Ġdescribe Ġa Ġsystem Ġwith Ġtwo Ġstage Ġprocess Ġfor Ġfact oid Ġand Ġlist Ġtype Ġquestion Ġanswering . ĠTheir Ġsystem Ġextracts Ġrelevant Ġentities Ġand Ġthen Ġruns Ġsupervised Ġclass ifier Ġto Ġrank Ġthe Ġentities . ĠW ies e Ġet Ġal . ĠB IB REF 4 Ġpropose Ġneural Ġnetwork Ġbased Ġmodel Ġfor ĠFact oid Ġand ĠList - type Ġquestion Ġanswering Ġtask . ĠThe Ġmodel Ġis Ġbased Ġon ĠFast ĠQ A Ġand Ġpredicts Ġthe Ġanswer Ġspan Ġin Ġthe Ġpassage Ġfor Ġa Ġgiven Ġquestion . ĠThe Ġmodel Ġis Ġtrained Ġon ĠS Qu AD Ġdata Ġset Ġand Ġfine Ġtuned Ġon Ġthe ĠBio AS Q Ġdata . ĠDim it riad is Ġet Ġal . ĠB IB REF 5 Ġproposed Ġtwo Ġstage Ġprocess Ġfor ĠFact oid Ġquestion Ġanswering Ġtask . ĠTheir Ġsystem Ġuses Ġgeneral Ġpurpose Ġtools Ġsuch Ġas ĠMet am ap , ĠBe Cas Ġto Ġidentify Ġcandidate Ġsentences . ĠThese Ġcandidate Ġsentences Ġare Ġrepresented Ġin Ġthe Ġform Ġof Ġfeatures , Ġand Ġare Ġthen Ġranked Ġby Ġthe Ġbinary Ġclass ifier . ĠClass ifier Ġis Ġtrained Ġon Ġcandidate Ġsentences Ġextracted Ġfrom Ġrelevant Ġquestions , Ġsnippets Ġand Ġcorrect Ġanswers Ġfrom ĠBio AS Q Ġchallenge . ĠFor Ġfact oid Ġquestion Ġanswering Ġtask Ġhighest ĠâĢ ĺ MR R âĢ Ļ Ġachieved Ġin Ġthe Ġ6 th Ġedition Ġof ĠBio AS Q Ġcompetition Ġis ĠâĢ ĺ 0 . 43 25 âĢ Ļ . ĠOur Ġsystem Ġis Ġa Ġneural Ġnetwork Ġmodel Ġbased Ġon Ġcontextual Ġword Ġembed d ings ĠB IB REF 1 Ġand Ġachieved Ġa ĠâĢ ĺ MR R âĢ Ļ Ġscore ĠâĢ ĺ 0 . 6 103 âĢ Ļ Ġin Ġone Ġof Ġthe Ġtest Ġbatches Ġfor ĠFact oid ĠQuestion ĠAn sw ering Ġtask . </s> Related ĠWork Ġ: :: ĠA Ġminimum Ġbackground Ġon ĠB ERT </s> BER T Ġstands Ġfor Ġ" B id irection al ĠEnc oder ĠRepresent ations Ġfrom ĠTransformers " ĠB IB REF 1 Ġis Ġa Ġcontextual Ġword Ġembed ding Ġmodel . ĠGiven Ġa Ġsentence Ġas Ġan Ġinput , Ġcontextual Ġembed ding Ġfor Ġthe Ġwords Ġare Ġreturned . ĠThe ĠB ERT Ġmodel Ġwas Ġdesigned Ġso Ġit Ġcan Ġbe Ġfine Ġtuned Ġfor Ġ11 Ġdifferent Ġtasks ĠB IB REF 1 , Ġincluding Ġquestion Ġanswering Ġtasks . ĠFor Ġa Ġquestion Ġanswering Ġtask , Ġquestion Ġand Ġparagraph Ġ( context ) Ġare Ġgiven Ġas Ġan Ġinput . ĠA ĠB ERT Ġstandard Ġis Ġthat Ġquestion Ġtext Ġand Ġparagraph Ġtext Ġare Ġseparated Ġby Ġa Ġsepar ator Ġ[ Sep ]. ĠB ERT Ġquestion - ans w ering Ġfine Ġtuning Ġinvolves Ġadding Ġsoft max Ġlayer . ĠSoft max Ġlayer Ġtakes Ġcontextual Ġword Ġembed d ings Ġfrom ĠB ERT Ġas Ġinput Ġand Ġlearns Ġto Ġidentity Ġanswer Ġspan Ġpresent Ġin Ġthe Ġparagraph Ġ( context ). ĠThis Ġprocess Ġis Ġrepresented Ġin ĠFigure ĠFIG REF 4 . </s> BER T Ġwas Ġoriginally Ġtrained Ġto Ġperform Ġtasks Ġsuch Ġas Ġlanguage Ġmodel Ġcreation Ġusing Ġmasked Ġwords Ġand Ġnext - sent ence - pred iction . ĠIn Ġother Ġwords ĠB ERT Ġweights Ġare Ġlearned Ġsuch Ġthat Ġcontext Ġis Ġused Ġin Ġbuilding Ġthe Ġrepresentation Ġof Ġthe Ġword , Ġnot Ġjust Ġas Ġa Ġloss Ġfunction Ġto Ġhelp Ġlearn Ġa Ġcontext - independent Ġrepresentation . ĠFor Ġdetailed Ġunderstanding Ġof ĠB ERT ĠArchitecture , Ġplease Ġrefer Ġto Ġthe Ġoriginal ĠB ERT Ġpaper ĠB IB REF 1 . </s> Related ĠWork Ġ: :: ĠA Ġminimum Ġbackground Ġon ĠB ERT Ġ: :: ĠComparison Ġof ĠWord ĠEmb edd ings Ġand ĠContext ual ĠWord ĠEmb edd ings </s> A ĠâĢ ĺ word Ġembed ding âĢ Ļ Ġis Ġa Ġlearned Ġrepresentation . ĠIt Ġis Ġrepresented Ġin Ġthe Ġform Ġof Ġvector Ġwhere Ġwords Ġthat Ġhave Ġthe Ġsame Ġmeaning Ġhave Ġa Ġsimilar Ġvector Ġrepresentation . ĠConsider Ġa Ġword Ġembed ding Ġmodel Ġ' word 2 vec ' ĠB IB REF 6 Ġtrained Ġon Ġa Ġcorpus . ĠWord Ġembed d ings Ġgenerated Ġfrom Ġthe Ġmodel Ġare Ġcontext Ġindependent Ġthat Ġis , Ġword Ġembed d ings Ġare Ġreturned Ġregardless Ġof Ġwhere Ġthe Ġwords Ġappear Ġin Ġa Ġsentence Ġand Ġregardless Ġof Ġe . g . Ġthe Ġsentiment Ġof Ġthe Ġsentence . ĠHowever , Ġcontextual Ġword Ġembed ding Ġmodels Ġlike ĠB ERT Ġalso Ġtakes Ġcontext Ġof Ġthe Ġword Ġinto Ġconsideration . </s> Related ĠWork Ġ: :: ĠComparison Ġof ĠB ERT Ġand ĠBio - BER T </s> âĢ ĺ BER T âĢ Ļ Ġand ĠBio BER T Ġare Ġvery Ġsimilar Ġin Ġterms Ġof Ġarchitecture . ĠDifference Ġis Ġthat ĠâĢ ĺ BER T âĢ Ļ Ġis Ġpret rained Ġon ĠWikipedia Ġarticles , Ġwhereas ĠBio BER T Ġversion Ġused Ġin Ġour Ġexperiments Ġis Ġpret rained Ġon ĠWikipedia , ĠPM C Ġand ĠPubMed Ġarticles . ĠTherefore ĠBio BER T Ġmodel Ġis Ġexpected Ġto Ġperform Ġwell Ġwith Ġbiomedical Ġtext , Ġin Ġterms Ġof Ġgenerating Ġcontextual Ġword Ġembed d ings . </s> Bio BER T Ġmodel Ġused Ġin Ġour Ġexperiments Ġis Ġbased Ġon ĠB ERT - Base ĠArchitecture ; ĠB ERT - Base Ġhas Ġ12 Ġtransformer ĠL ayers Ġwhere Ġas ĠB ERT - Large Ġhas Ġ24 Ġtransformer Ġlayers . ĠMoreover Ġcontextual Ġword Ġembed ding Ġvector Ġsize Ġis Ġ768 Ġfor ĠB ERT - Base Ġand Ġmore Ġfor ĠB ERT - large . ĠAccording Ġto ĠB IB REF 1 ĠBert - Large , Ġfine - tun ed Ġon ĠS Qu AD Ġ1 . 1 Ġquestion Ġanswering Ġdata ĠB IB REF 7 Ġcan Ġachieve ĠF 1 ĠScore Ġof Ġ90 . 9 Ġfor ĠQuestion ĠAn sw ering Ġtask Ġwhere Ġas ĠB ERT - Base ĠFine - tun ed Ġon Ġthe Ġsame ĠS Qu AD Ġquestion Ġanswering ĠB IB REF 7 Ġdata Ġcould Ġachieve ĠF 1 Ġscore Ġof Ġ88 . 5 . ĠOne Ġdownside Ġof Ġthe Ġcurrent Ġversion ĠBio BER T Ġis Ġthat Ġword - piece Ġvocabulary Ġis Ġthe Ġsame Ġas Ġthat Ġof Ġoriginal ĠB ERT ĠModel , Ġas Ġa Ġresult Ġword - piece Ġvocabulary Ġdoes Ġnot Ġinclude Ġbiomedical Ġjargon . ĠLee Ġet Ġal . ĠB IB REF 0 Ġcreated ĠBio BER T , Ġusing Ġthe Ġsame Ġpre - trained ĠB ERT Ġreleased Ġby ĠGoogle , Ġand Ġhence Ġin Ġthe Ġword - piece Ġvocabulary Ġ( voc ab . txt ), Ġas Ġa Ġresult Ġbiomedical Ġjargon Ġis Ġnot Ġincluded Ġin Ġword - piece Ġvocabulary . ĠMod ifying Ġword - piece Ġvocabulary Ġ( voc ab . txt ) Ġat Ġthis Ġstage Ġwould Ġloose Ġoriginal Ġcompatibility Ġwith ĠâĢ ĺ BER T âĢ Ļ , Ġhence Ġit Ġis Ġleft Ġun modified . </s> In Ġour Ġfuture Ġwork Ġwe Ġwould Ġlike Ġto Ġbuild Ġpre - trained ĠâĢ ĺ BER T âĢ Ļ Ġmodel Ġfrom Ġscratch . ĠWe Ġwould Ġpret rain Ġthe Ġmodel Ġwith Ġbiomedical Ġcorpus Ġ( Pub Med , ĠâĢ ĺ PM C âĢ Ļ ) Ġand ĠWikipedia . ĠDoing Ġso Ġwould Ġgive Ġus Ġscope Ġto Ġcreate Ġword Ġpiece Ġvocabulary Ġto Ġinclude Ġbiomedical Ġjargon Ġand Ġthere Ġare Ġchances Ġof Ġmodel Ġperforming Ġbetter Ġwith Ġbiomedical Ġjargon Ġbeing Ġincluded Ġin Ġthe Ġword Ġpiece Ġvocabulary . ĠWe Ġwill Ġconsider Ġthis Ġscenario Ġin Ġthe Ġfuture , Ġor Ġwait Ġfor Ġthe Ġnext Ġversion Ġof ĠBio BER T . </s> Exper iments : ĠFact oid ĠQuestion ĠAn sw ering ĠTask </s> For ĠFact oid ĠQuestion ĠAn sw ering Ġtask , Ġwe Ġfine Ġtuned ĠBio BER T ĠB IB REF 0 Ġwith Ġquestion Ġanswering Ġdata Ġand Ġadded Ġnew Ġfeatures . ĠFig . ĠFIG REF 4 Ġshows Ġthe Ġarchitecture Ġof ĠBio BER T Ġfine Ġtuned Ġfor Ġquestion Ġanswering Ġtasks : ĠInput Ġto ĠBio BER T Ġis Ġword Ġtoken ized Ġembed d ings Ġfor Ġquestion Ġand Ġthe Ġparagraph Ġ( Context ). ĠAs Ġper Ġthe ĠâĢ ĺ BER T âĢ Ļ ĠB IB REF 1 Ġstandards , Ġtokens ĠâĢ ĺ [ CL S ] âĢ Ļ Ġand ĠâĢ ĺ [ SE P ] âĢ Ļ Ġare Ġapp ended Ġto Ġthe Ġtoken ized Ġinput Ġas Ġillustrated Ġin Ġthe Ġfigure . ĠThe Ġresulting Ġmodel Ġhas Ġa Ġsoft max Ġlayer Ġformed Ġfor Ġpredicting Ġanswer Ġspan Ġindices Ġin Ġthe Ġgiven Ġparagraph Ġ( Context ). ĠOn Ġtest Ġdata , Ġthe Ġfine Ġtuned Ġmodel Ġgenerates Ġ$ n $ - best Ġpredictions Ġfor Ġeach Ġquestion . ĠFor Ġa Ġquestion , Ġ$ n $ - best Ġcorresponds Ġthat Ġ$ n $ Ġanswers Ġare Ġreturned Ġas Ġpossible Ġanswers Ġin Ġthe Ġdecreasing Ġorder Ġof Ġconfidence . ĠVariable Ġ$ n $ Ġis Ġconfig urable . ĠIn Ġour Ġpaper , Ġany Ġfurther Ġmentions Ġof ĠâĢ ĺ answer Ġreturned Ġby Ġthe Ġmodel âĢ Ļ Ġcorrespond Ġto Ġthe Ġtop Ġanswer Ġreturned Ġby Ġthe Ġmodel . </s> Exper iments : ĠFact oid ĠQuestion ĠAn sw ering ĠTask Ġ: :: ĠSetup </s> Bio AS Q Ġprovides Ġthe Ġtraining Ġdata . ĠThis Ġdata Ġis Ġbased Ġon Ġprevious ĠBio AS Q Ġcompetitions . ĠTrain Ġdata Ġwe Ġhave Ġconsidered Ġis Ġaggregate Ġof Ġall Ġtrain Ġdata Ġsets Ġtill Ġthe Ġ5 th Ġversion Ġof ĠBio AS Q Ġcompetition . ĠWe Ġcleaned Ġthe Ġdata , Ġthat Ġis , Ġquestion - ans w ering Ġdata Ġwithout Ġanswers Ġare Ġremoved Ġand Ġleft Ġwith Ġa Ġtotal Ġcount Ġof ĠâĢ ĺ 530 âĢ Ļ Ġquestion Ġanswers . ĠThe Ġdata Ġis Ġsplit Ġinto Ġtrain Ġand Ġtest Ġdata Ġin Ġthe Ġratio Ġof Ġ94 Ġto Ġ6 ; Ġthat Ġis , Ġcount Ġof Ġ' 495 ' Ġfor Ġtraining Ġand Ġ' 35 ' Ġfor Ġtesting . </s> The Ġoriginal Ġdata Ġformat Ġis Ġconverted Ġto Ġthe ĠB ERT / Bio BER T Ġformat , Ġwhere ĠBio BER T Ġexpects ĠâĢ ĺ start _ index âĢ Ļ Ġof Ġthe Ġactual Ġanswer . ĠThe ĠâĢ ĺ start _ index Ġcorresponds Ġto Ġthe Ġindex Ġof Ġthe Ġanswer Ġtext Ġpresent Ġin Ġthe Ġparagraph / ĠContext . ĠFor Ġfinding ĠâĢ ĺ start _ index âĢ Ļ Ġwe Ġused Ġbuilt - in Ġpython Ġfunction Ġfind (). ĠThe Ġfunction Ġreturns Ġthe Ġlowest Ġindex Ġof Ġthe Ġactual Ġanswer Ġpresent Ġin Ġthe Ġcontext ( paragraph ). ĠIf Ġthe Ġanswer Ġis Ġnot Ġfound ĠâĢ ĺ - 1 âĢ Ļ Ġis Ġreturned Ġas Ġthe Ġindex . ĠThe Ġefficient Ġway Ġof Ġfinding Ġstart _ index Ġis , Ġif Ġthe Ġparagraph Ġ( Context ) Ġhas Ġmultiple Ġinstances Ġof Ġanswer Ġtext , Ġthen ĠâĢ ĺ start _ index âĢ Ļ Ġof Ġthe Ġanswer Ġshould Ġbe Ġthat Ġinstance Ġof Ġanswer Ġtext Ġwhose Ġcontext Ġactually Ġmatches Ġwith Ġwhat âĢ Ļ s Ġbeen Ġasked Ġin Ġthe Ġquestion . </s> Example Ġ( Question , ĠAnswer Ġand ĠPar agraph Ġfrom ĠB IB REF 8 ): </s> Question : ĠWhich Ġdrug Ġshould Ġbe Ġused Ġas Ġan Ġantidote Ġin Ġbenz od iazep ine Ġoverdose ? </s> Answer : Ġ' Fl um az en il ' </s> Par agraph ( context ): </s> " Fl um az en il Ġuse Ġin Ġbenz od iazep ine Ġoverdose Ġin Ġthe ĠUK : Ġa Ġretrospective Ġsurvey Ġof ĠNP IS Ġdata . ĠOB JECT IVE : ĠBenz od iazep ine Ġ( B Z D ) Ġoverdose Ġ( OD ) Ġcontinues Ġto Ġcause Ġsignificant Ġmorbid ity Ġand Ġmortality Ġin Ġthe ĠUK . ĠFl um az en il Ġis Ġan Ġeffective Ġantidote Ġbut Ġthere Ġis Ġa Ġrisk Ġof Ġseizures , Ġparticularly Ġin Ġthose Ġwho Ġhave Ġco - ing ested Ġtr icy cl ic Ġantidepressants . ĠA Ġstudy Ġwas Ġundertaken Ġto Ġexamine Ġthe Ġfrequency Ġof Ġuse , Ġsafety Ġand Ġefficacy Ġof Ġfl um az en il Ġin Ġthe Ġmanagement Ġof ĠB Z D ĠOD Ġin Ġthe ĠUK . ĠM ETHOD S : ĠA Ġ2 - year Ġretrospective Ġcohort Ġstudy Ġwas Ġperformed Ġof Ġall Ġenqu iries Ġto Ġthe ĠUK ĠNational ĠPo isons ĠInformation ĠService Ġinvolving ĠB Z D ĠOD . ĠRES ULTS : ĠFl um az en il Ġwas Ġadministered Ġto Ġ80 Ġpatients Ġin Ġ450 4 ĠB Z D - related Ġenqu iries , Ġ68 Ġof Ġwhom Ġdid Ġnot Ġhave Ġvent il atory Ġfailure Ġor Ġhad Ġrecognised Ġcont rain d ications Ġto Ġfl um az en il . ĠFactors Ġassociated Ġwith Ġfl um az en il Ġuse Ġwere Ġincreased Ġage , Ġsevere Ġpoisoning Ġand Ġvent il atory Ġfailure . ĠCo - ing est ion Ġof Ġtr icy cl ic Ġantidepressants Ġand Ġchronic Ġobstruct ive Ġpulmonary Ġdisease Ġdid Ġnot Ġinfluence Ġfl um az en il Ġadministration . ĠSe iz ure Ġfrequency Ġin Ġpatients Ġnot Ġtreated Ġwith Ġfl um az en il Ġwas Ġ0 . 3 % ". </s> Act ual Ġanswer Ġis Ġ' Fl um az en il ', Ġbut Ġthere Ġare Ġmultiple Ġinstances Ġof Ġword Ġ' F lu - m az en il '. ĠE fficient Ġway Ġto Ġidentify Ġthe Ġstart - index Ġfor Ġ' Fl um az en il ' ( answer ) Ġis Ġto Ġfind Ġthat Ġparticular Ġinstance Ġof Ġthe Ġword Ġ' Fl um az en il ' Ġwhich Ġmatches Ġthe Ġcontext Ġfor Ġthe Ġquestion . ĠIn Ġthe Ġabove Ġexample Ġ' Fl um az en il ' Ġhighlighted Ġin Ġbold Ġis Ġthe Ġactual Ġinstance Ġthat Ġmatches Ġquestion 's Ġcontext . ĠUnfortunately , Ġwe Ġcould Ġnot Ġidentify Ġreadily Ġavailable Ġtools Ġthat Ġcan Ġachieve Ġthis Ġgoal . ĠIn Ġour Ġfuture Ġwork , Ġwe Ġlook Ġforward Ġto Ġhandling Ġthese Ġscenarios Ġeffectively . </s> Note : ĠThe Ġcreators Ġof Ġ' S Qu AD ' ĠB IB REF 7 Ġhave Ġhandled Ġthe Ġtask Ġof Ġidentifying Ġanswer 's Ġstart _ index Ġeffectively . ĠBut Ġ' S Qu AD ' Ġdata Ġset Ġis Ġmuch Ġmore Ġgeneral Ġand Ġdoes Ġnot Ġinclude Ġbiomedical Ġquestion Ġanswering Ġdata . </s> Exper iments : ĠFact oid ĠQuestion ĠAn sw ering ĠTask Ġ: :: ĠTraining Ġand Ġerror Ġanalysis </s> During Ġour Ġtraining Ġwith Ġthe ĠBio AS Q Ġdata , Ġlearning Ġrate Ġis Ġset Ġto Ġ3 e - 5 , Ġas Ġmentioned Ġin Ġthe ĠBio BER T Ġpaper ĠB IB REF 0 . ĠWe Ġstarted Ġtraining Ġthe Ġmodel Ġwith Ġ4 95 Ġavailable Ġtrain Ġdata Ġand Ġ35 Ġtest Ġdata Ġby Ġsetting Ġnumber Ġof Ġepoch s Ġto Ġ50 . ĠAfter Ġtraining Ġwith Ġthese Ġhyper - param eters Ġtraining Ġaccuracy ( ex act Ġmatch ) Ġwas Ġ99 . 3 % ( over fitting ) Ġand Ġtesting Ġaccuracy Ġis Ġonly Ġ4 %. ĠIn Ġthe Ġnext Ġiteration Ġwe Ġreduced Ġthe Ġnumber Ġof Ġepoch s Ġto Ġ25 Ġthen Ġtraining Ġaccuracy Ġis Ġreduced Ġto Ġ98 . 5 % Ġand Ġtest Ġaccuracy Ġmoved Ġto Ġ5 %. ĠWe Ġfurther Ġreduced Ġnumber Ġof Ġepoch s Ġto Ġ15 , Ġand Ġthe Ġresulting Ġtraining Ġaccuracy Ġwas Ġ70 % Ġand Ġtest Ġaccuracy Ġ15 %. ĠIn Ġthe Ġnext Ġiteration Ġset Ġnumber Ġof Ġepoch s Ġto Ġ12 Ġand Ġachieved Ġtrain Ġaccuracy Ġof Ġ57 . 7 % Ġand Ġtest Ġaccuracy Ġ23 . 3 %. ĠRepe ated Ġthe Ġexperiment Ġwith Ġ11 Ġepoch s Ġand Ġfound Ġtraining Ġaccuracy Ġto Ġbe Ġ57 . 7 % Ġand Ġtest Ġaccuracy Ġto Ġbe Ġsame Ġ22 %. ĠIn Ġthe Ġnext Ġiteration Ġwe Ġset Ġnumber Ġof Ġepoch s Ġto Ġ' 9 ' Ġand Ġfound Ġtraining Ġaccuracy Ġof Ġ48 % Ġand Ġtest Ġaccuracy Ġof Ġ15 %. ĠHence Ġoptimum Ġnumber Ġof Ġepoch s Ġis Ġtaken Ġas Ġ12 Ġepoch s . </s> During Ġour Ġerror Ġanalysis Ġwe Ġfound Ġthat Ġon Ġtest Ġdata , Ġmodel Ġtends Ġto Ġreturn Ġtext Ġin Ġthe Ġbeginning Ġof Ġthe Ġcontext ( paragraph ) Ġas Ġthe Ġanswer . ĠOn Ġanalys ing Ġtrain Ġdata , Ġwe Ġfound Ġthat Ġthere Ġare Ġ' 120 ' ( out Ġof Ġ' 495 ') Ġquestion Ġanswering Ġdata Ġinstances Ġhaving Ġstart _ index : 0 , Ġmeaning Ġ120 ( Ġ25 %) Ġquestion Ġanswering Ġdata Ġhas Ġfirst Ġword ( s ) Ġin Ġthe Ġcontext ( paragraph ) Ġas Ġthe Ġanswer . ĠWe Ġremoved Ġ70 % Ġof Ġthose Ġinstances Ġin Ġorder Ġto Ġmake Ġtrain Ġdata Ġmore Ġbalanced . ĠIn Ġthe Ġnew Ġtrain Ġdata Ġset Ġwe Ġare Ġleft Ġwith Ġ' 411 ' Ġquestion Ġanswering Ġdata Ġinstances . ĠThis Ġtime Ġwe Ġgot Ġthe Ġhighest Ġtest Ġaccuracy Ġof Ġ26 % Ġat Ġ11 Ġepoch s . ĠWe Ġhave Ġsubmitted Ġour Ġresults Ġfor ĠBio AS Q Ġtest Ġbatch - 2 , Ġgot Ġstrict Ġaccuracy Ġof Ġ32 % Ġand Ġour Ġsystem Ġstood Ġin Ġ2 nd Ġplace . ĠInitially , Ġhyper - param eter - Ġ' batch Ġsize ' Ġis Ġset Ġto ĠâĢ ĺ 400 âĢ Ļ . ĠLater Ġit Ġis Ġtuned Ġto Ġ' 32 '. ĠAlthough Ġaccuracy ( ex act Ġanswer Ġmatch ) Ġremained Ġat Ġ26 %, Ġmodel Ġgenerated Ġconcise Ġand Ġbetter Ġanswers Ġat Ġbatch Ġsize ĠâĢ ĺ 32 âĢ Ļ , Ġthat Ġis Ġwrong Ġanswers Ġare Ġclose Ġto Ġthe Ġexpected Ġanswer Ġin Ġgood Ġnumber Ġof Ġcases . </s> Example .( from ĠB IB REF 8 ) </s> Question : ĠWhich Ġmutated Ġgene Ġcauses ĠC hed iak ĠHig ashi ĠSyndrome ? </s> Ex act ĠAnswer : ĠâĢ ĺ lys os omal Ġtrafficking Ġregulator Ġgene âĢ Ļ . </s> The Ġanswer Ġreturned Ġby Ġa Ġmodel Ġtrained Ġat ĠâĢ ĺ 400 âĢ Ļ Ġbatch Ġsize Ġis ĠâĢ ĺ Aut os omal - re cessive Ġcomplicated Ġsp astic Ġparap leg ia Ġwith Ġa Ġnovel Ġl ys os omal Ġtrafficking Ġregulator âĢ Ļ , Ġand Ġfrom Ġthe Ġone Ġtrained Ġat ĠâĢ ĺ 32 âĢ Ļ Ġbatch Ġsize Ġis ĠâĢ ĺ lys os omal Ġtrafficking Ġregulator âĢ Ļ . </s> In Ġfurther Ġexperiments , Ġwe Ġhave Ġfine Ġtuned Ġthe ĠBio BER T Ġmodel Ġwith Ġboth ĠâĢ ĺ S Qu AD âĢ Ļ Ġdataset Ġ( version Ġ2 . 0 ) Ġand ĠBio As q Ġtrain Ġdata . ĠFor Ġtraining Ġon ĠâĢ ĺ S Qu AD âĢ Ļ , Ġhyper Ġparameters - ĠLearning Ġrate Ġand Ġnumber Ġof Ġepoch s Ġare Ġset Ġto ĠâĢ ĺ 3 e - 3 âĢ Ļ Ġand ĠâĢ ĺ 3 âĢ Ļ Ġrespectively Ġas Ġmentioned Ġin Ġthe Ġpaper ĠB IB REF 1 . ĠTest Ġaccuracy Ġof Ġthe Ġmodel Ġboosted Ġto Ġ44 %. ĠIn Ġone Ġmore Ġexperiment Ġwe Ġtrained Ġmodel Ġonly Ġon ĠâĢ ĺ S Qu AD âĢ Ļ Ġdataset , Ġthis Ġtime Ġtest Ġaccuracy Ġof Ġthe Ġmodel Ġmoved Ġto Ġ47 %. ĠThe Ġreason Ġmodel Ġdid Ġnot Ġperform Ġup Ġto Ġthe Ġmark Ġwhen Ġtrained Ġwith ĠâĢ ĺ S Qu AD âĢ Ļ Ġalongside ĠBio AS Q Ġdata Ġcould Ġbe Ġthat Ġin Ġformatted ĠBio AS Q Ġdata , Ġstart _ index Ġfor Ġthe Ġanswer Ġis Ġnot Ġaccurate , Ġand Ġaffected Ġthe Ġoverall Ġaccuracy . </s> Our ĠSystems Ġand ĠTheir ĠPerformance Ġon ĠFact oid ĠQuestions </s> We Ġhave Ġexperimented Ġwith Ġseveral Ġsystems Ġand Ġtheir Ġvariations , Ġe . g . Ġcreated Ġby Ġtraining Ġwith Ġspecific Ġadditional Ġfeatures Ġ( see Ġnext Ġsubsection ). ĠHere Ġis Ġtheir Ġlist Ġand Ġshort Ġdescriptions . ĠUnfortunately Ġwe Ġdid Ġnot Ġpay Ġattention Ġto Ġnaming , Ġand Ġthe Ġsystems Ġevolved Ġbetween Ġtest Ġbatches , Ġso Ġthe Ġoverall Ġpicture Ġcan Ġonly Ġbe Ġunderstood Ġby Ġlooking Ġat Ġthe Ġdetails . </s> When Ġwe Ġstarted Ġthe Ġexperiments Ġour Ġobjective Ġwas Ġto Ġsee Ġwhether ĠBio BER T Ġand Ġentail ment - based Ġtechniques Ġcan Ġprovide Ġvalue Ġfor Ġin Ġthe Ġcontext Ġof Ġbiomedical Ġquestion Ġanswering . ĠThe Ġanswer Ġto Ġboth Ġquestions Ġwas Ġa Ġyes , Ġqualified Ġby Ġmany Ġexamples Ġclearly Ġshowing Ġthe Ġlimitations Ġof Ġboth Ġmethods . ĠTherefore Ġwe Ġtried Ġto Ġaddress Ġsome Ġof Ġthese Ġlimitations Ġusing Ġfeature Ġengineering Ġwith Ġmixed Ġresults : Ġsome Ġclear Ġerrors Ġgot Ġcorrected Ġand Ġnew Ġerrors Ġgot Ġintroduced , Ġwithout Ġoverall Ġimprovement Ġbut Ġconvincing Ġus Ġthat Ġin Ġfuture Ġexperiments Ġit Ġmight Ġbe Ġworth Ġtrying Ġfeature Ġengineering Ġagain Ġespecially Ġif Ġmore Ġtraining Ġdata Ġwere Ġavailable . </s> Overall Ġwe Ġexperimented Ġwith Ġseveral Ġapproaches Ġwith Ġthe Ġfollowing Ġaspects Ġof Ġthe Ġsystems Ġchanging Ġbetween Ġbatches , Ġthat Ġis Ġbeing Ġabsent Ġor Ġpresent : </s> training Ġon ĠBio As q Ġdata Ġvs . Ġtraining Ġon ĠS Qu AD </s> using Ġthe ĠBio As q Ġsnippets Ġfor Ġcontext Ġvs . Ġusing Ġthe Ġdocuments Ġfrom Ġthe Ġprovided ĠURLs Ġfor Ġcontext </s> adding Ġor Ġnot Ġthe ĠLAT , Ġi . e . Ġlex ical Ġanswer Ġtype , Ġfeature Ġ( see ĠB IB REF 9 , ĠB IB REF 10 Ġand Ġan Ġexplanation Ġin Ġthe Ġsubsection Ġjust Ġbelow ). </s> For ĠYes / No Ġquestions Ġ( only ) Ġwe Ġexperimented Ġwith Ġthe Ġentail ment Ġmethods . </s> We Ġwill Ġdiscuss Ġthe Ġperformance Ġof Ġthese Ġmodels Ġbelow Ġand Ġin ĠSection Ġ6 . ĠBut Ġbefore Ġwe Ġdo Ġthat , Ġlet Ġus Ġdiscuss Ġa Ġfeature Ġengineering Ġexperiment Ġwhich Ġeventually Ġproduced Ġmixed Ġresults , Ġbut Ġwhere Ġwe Ġfeel Ġit Ġis Ġpotentially Ġuseful Ġin Ġfuture Ġexperiments . </s> Our ĠSystems Ġand ĠTheir ĠPerformance Ġon ĠFact oid ĠQuestions Ġ: :: ĠLAT ĠFeature Ġconsidered Ġand Ġits Ġimpact Ġ( s lightly Ġnegative ) </s> During Ġerror Ġanalysis Ġwe Ġfound Ġthat Ġfor Ġsome Ġcases , Ġanswer Ġbeing Ġreturned Ġby Ġthe Ġmodel Ġis Ġfar Ġaway Ġfrom Ġwhat Ġit Ġis Ġbeing Ġasked Ġin Ġthe ĠQuestion . </s> Example : Ġ( from ĠB IB REF 8 ) </s> Question : ĠHy 's Ġlaw Ġmeasures Ġfailure Ġof Ġwhich Ġorgan ? </s> Act ual ĠAnswer : ĠâĢ ĺ L iver âĢ Ļ . </s> The Ġanswer Ġreturned Ġby Ġone Ġof Ġour Ġmodels Ġwas ĠâĢ ĺ alan ine Ġam in ot rans fer ase âĢ Ļ , Ġwhich Ġis Ġan Ġenzyme . ĠThe Ġmodel Ġreturns Ġan Ġenzyme , Ġwhen Ġthe Ġquestion Ġasked Ġfor Ġthe Ġorgan Ġname . ĠTo Ġaddress Ġthis Ġtype Ġof Ġerrors , Ġwe Ġdecided Ġto Ġtry Ġthe Ġconcepts Ġof ĠâĢ ĺ Lex ical ĠAnswer ĠType âĢ Ļ Ġ( L AT ) Ġand ĠFocus ĠWord , Ġwhich Ġwas Ġused Ġin ĠIBM ĠWatson , Ġsee ĠB IB REF 11 Ġfor Ġoverview ; ĠB IB REF 10 Ġfor Ġtechnical Ġdetails , Ġand ĠB IB REF 9 Ġfor Ġdetails Ġon Ġquestion Ġanalysis . ĠIn Ġan Ġexample Ġgiven Ġin Ġthe Ġlast Ġsource Ġwe Ġread : </s> PO ETS Ġ& ĠPO ET RY : ĠHe Ġwas Ġa Ġbank Ġclerk Ġin Ġthe ĠYuk on Ġbefore Ġhe Ġpublished Ġ" S ongs Ġof Ġa ĠSour d ough " Ġin Ġ1907 . </s> The Ġfocus Ġis Ġthe Ġpart Ġof Ġthe Ġquestion Ġthat Ġis Ġa Ġreference Ġto Ġthe Ġanswer . ĠIn Ġthe Ġexample Ġabove , Ġthe Ġfocus Ġis Ġ" he ". </s> L AT s Ġare Ġterms Ġin Ġthe Ġquestion Ġthat Ġindicate Ġwhat Ġtype Ġof Ġentity Ġis Ġbeing Ġasked Ġfor . </s> The Ġhead word Ġof Ġthe Ġfocus Ġis Ġgenerally Ġa ĠLAT , Ġbut Ġquestions Ġoften Ġcontain Ġadditional ĠLAT s , Ġand Ġin Ġthe ĠJe opard y ! Ġdomain , Ġcategories Ġare Ġan Ġadditional Ġsource Ġof ĠLAT s . </s> ( ...) ĠIn Ġthe Ġexample , ĠLAT s Ġare Ġ" he ", Ġ" cl erk ", Ġand Ġ" po et ". </s> For Ġexample Ġin Ġthe Ġquestion Ġ" Which Ġplant Ġdoes Ġo le uro pe in Ġoriginate Ġfrom ?" Ġ( BI B REF 8 ). ĠThe ĠLAT Ġhere Ġis ĠâĢ ĺ plant âĢ Ļ . ĠFor Ġthe ĠBio As q Ġtask Ġwe Ġdid Ġnot Ġneed Ġto Ġexplicitly Ġdistinguish Ġbetween Ġthe Ġfocus Ġand Ġthe ĠLAT Ġconcepts . ĠIn Ġthis Ġexample , Ġthe Ġexpectation Ġis Ġthat Ġanswer Ġreturned Ġby Ġthe Ġmodel Ġis Ġa Ġplant . ĠThus Ġit Ġis Ġconceivable Ġthat Ġthe Ġcos ine Ġdistance Ġbetween Ġcontextual Ġembed ding Ġof Ġword Ġ' plant ' Ġin Ġthe Ġquestion Ġand Ġcontextual Ġembed ding Ġfor Ġthe Ġanswer Ġpresent Ġin Ġthe Ġparagraph ( context ) Ġis Ġcomparatively Ġlow . ĠAs Ġa Ġresult Ġmodel Ġlearns Ġto Ġadjust Ġits Ġweights Ġduring Ġtraining Ġphase Ġand Ġreturns Ġanswers Ġwith Ġlow Ġcos ine Ġdistance Ġwith Ġthe ĠLAT . </s> We Ġused ĠStanford ĠCore N LP ĠB IB REF 12 Ġlibrary Ġto Ġwrite Ġrules Ġfor Ġextracting Ġlex ical Ġanswer Ġtype Ġpresent Ġin Ġthe Ġquestion , Ġboth Ġ' parts Ġof Ġspeech ' ( POS ) Ġand Ġdependency Ġparsing Ġfunctionality Ġwas Ġused . ĠWe Ġincorporated Ġthe ĠLex ical ĠAnswer ĠType Ġinto Ġone Ġof Ġour Ġsystems , ĠUN CC _ Q A 1 Ġin ĠB atch Ġ4 . ĠThis Ġsystem Ġunder per formed Ġour Ġsystem ĠFAC TO IDS Ġby Ġabout Ġ3 % Ġin Ġthe ĠMR R Ġmeasure , Ġbut Ġcorrected Ġerrors Ġsuch Ġas Ġin Ġthe Ġexample Ġabove . </s> Our ĠSystems Ġand ĠTheir ĠPerformance Ġon ĠFact oid ĠQuestions Ġ: :: ĠLAT ĠFeature Ġconsidered Ġand Ġits Ġimpact Ġ( s lightly Ġnegative ) Ġ: :: ĠAss um ptions Ġand Ġrules Ġfor Ġder iving Ġlex ical Ġanswer Ġtype . </s> There Ġare Ġdifferent Ġquestion Ġtypes : ĠâĢ ĺ Which âĢ Ļ , ĠâĢ ĺ What âĢ Ļ , ĠâĢ ĺ When âĢ Ļ , ĠâĢ ĺ How âĢ Ļ Ġetc . ĠEach Ġtype Ġof Ġquestion Ġis Ġbeing Ġhandled Ġdifferently Ġand Ġthere Ġare Ġcommon alities Ġamong Ġthe Ġrules Ġwritten Ġfor Ġdifferent Ġquestion Ġtypes . ĠQuestion Ġwords Ġare Ġidentified Ġthrough Ġparts Ġof Ġspeech Ġtags : Ġ' WD T ', Ġ' WR B ' Ġ, ' WP '. ĠWe Ġassumed Ġthat ĠLAT Ġis Ġa ĠâĢ ĺ N oun âĢ Ļ Ġand Ġfollows Ġthe Ġquestion Ġword . ĠOften Ġit Ġwas Ġalso Ġa Ġsubject Ġ( ns ub j ). ĠThis Ġprocess Ġis Ġillustrated Ġin ĠFig . FIG REF 15 . </s> L AT Ġcomputation Ġwas Ġgoverned Ġby Ġa Ġfew Ġsimple Ġrules , Ġe . g . Ġwhen Ġa Ġquestion Ġhas Ġmultiple Ġwords Ġthat Ġare Ġ' Subject s âĢ Ļ Ġ( and ĠâĢ ĺ N oun âĢ Ļ ), Ġa Ġword Ġthat Ġis Ġin Ġproximity Ġto Ġthe Ġquestion Ġword Ġis Ġconsidered Ġas ĠâĢ ĺ L AT âĢ Ļ . ĠThese Ġrules Ġare Ġdifferent Ġfor Ġeach Ġ" Wh " Ġword . </s> Name ly , Ġwhen Ġthe Ġword Ġimmediately Ġfollowing Ġthe Ġquestion Ġword Ġis Ġa ĠN oun , Ġwindow Ġsize Ġis Ġset Ġto ĠâĢ ĺ 3 âĢ Ļ . ĠThe Ġwindow Ġsize ĠâĢ ĺ 3 âĢ Ļ Ġmeans Ġwe Ġiter ate Ġthrough Ġthe Ġnext ĠâĢ ĺ 3 âĢ Ļ Ġwords Ġto Ġcheck Ġif Ġany Ġof Ġthe Ġword Ġis Ġboth ĠN oun Ġand ĠSubject , ĠIf Ġso , Ġsuch Ġword Ġis Ġconsidered Ġthe ĠâĢ ĺ L AT âĢ Ļ ; Ġelse Ġthe Ġword Ġthat Ġis Ġpresent Ġvery Ġnext Ġto Ġthe Ġquestion Ġword Ġis Ġconsidered Ġas Ġthe ĠâĢ ĺ L AT âĢ Ļ . </s> For Ġquestions Ġwith Ġwords ĠâĢ ĺ Which âĢ Ļ Ġ, ĠâĢ ĺ What âĢ Ļ , ĠâĢ ĺ When âĢ Ļ ; Ġa ĠN oun Ġimmediately Ġfollowing Ġthe Ġquestion Ġword Ġis Ġvery Ġoften Ġthe ĠLAT , Ġe . g . Ġ' en zyme ' Ġin ĠWhich Ġenzyme Ġis Ġtargeted Ġby ĠEv ol oc um ab ? . ĠWhen Ġthe Ġword Ġimmediately Ġfollowing Ġthe Ġquestion Ġword Ġis Ġnot Ġa ĠN oun , Ġe . g . Ġin ĠWhat Ġis Ġthe Ġfunction Ġof Ġthe Ġprotein ĠMag t 1 ? Ġthe Ġwindow Ġsize Ġis Ġset Ġto ĠâĢ ĺ 5 âĢ Ļ , Ġand Ġwe Ġiter ate Ġthrough Ġthe Ġnext ĠâĢ ĺ 5 âĢ Ļ Ġwords Ġ( if Ġpresent ) Ġand Ġsearch Ġfor Ġthe Ġword Ġthat Ġis Ġboth ĠN oun Ġand ĠSubject . ĠIf Ġpresent , Ġthe Ġword Ġis Ġconsidered Ġas Ġthe ĠâĢ ĺ L AT âĢ Ļ ; Ġelse , Ġthe ĠN oun Ġin Ġclose Ġproximity Ġto Ġthe Ġquestion Ġword Ġand Ġfollowing Ġit Ġis Ġreturned Ġas Ġthe ĠâĢ ĺ L AT âĢ Ļ . </s> For Ġquestions Ġwith Ġquestion Ġwords : ĠâĢ ĺ When âĢ Ļ , ĠâĢ ĺ Who âĢ Ļ , ĠâĢ ĺ Why âĢ Ļ , Ġthe ĠâĢ Ļ L AT âĢ Ļ Ġis Ġa Ġquestion Ġword Ġitself . ĠFor Ġthe Ġword ĠâĢ ĺ How ', Ġe . g . Ġin ĠHow Ġmany Ġse len op rote ins Ġare Ġencoded Ġin Ġthe Ġhuman Ġgenome ?, Ġwe Ġlook Ġat Ġthe Ġadjective Ġand Ġif Ġwe Ġfind Ġone , Ġwe Ġtake Ġit Ġto Ġbe Ġthe ĠLAT , Ġotherwise Ġthe Ġword Ġ' How ' Ġis Ġconsidered Ġas Ġthe ĠâĢ ĺ L AT âĢ Ļ . </s> Perhaps Ġbecause Ġof Ġusing Ġonly Ġvery Ġsimple Ġrules , Ġthe Ġaccuracy Ġfor ĠâĢ ĺ L AT âĢ Ļ Ġderiv ation Ġis Ġ75 %; Ġthat Ġis , Ġin Ġthe Ġremaining Ġ25 % Ġof Ġthe Ġcases Ġthe ĠLAT Ġword Ġis Ġidentified Ġincorrectly . ĠWorth Ġnoting Ġis Ġthat Ġthe Ġoverall Ġperformance Ġthe Ġsystem Ġthat Ġused ĠLAT s Ġwas Ġslightly Ġinferior Ġto Ġthe Ġsystem Ġwithout ĠLAT s , Ġbut Ġthe Ġtypes Ġof Ġerrors Ġchanged . ĠThe Ġtraining Ġused ĠBio BER T Ġwith Ġthe ĠLAT Ġfeature Ġas Ġpart Ġof Ġthe Ġinput Ġstring . ĠThe Ġerrors Ġit Ġintroduces Ġusually Ġinvolve Ġfinding Ġthe Ġwrong Ġelement Ġof Ġthe Ġcorrect Ġtype Ġe . g . Ġwrong Ġenzyme Ġwhen Ġtwo Ġsimilar Ġenzymes Ġare Ġdescribed Ġin Ġthe Ġtext , Ġor Ġ' ne uron ' Ġwhen Ġasked Ġabout Ġa Ġtype Ġof Ġcell Ġwith Ġa Ġcertain Ġfunction , Ġwhen Ġthe Ġanswer Ġcalls Ġfor Ġa Ġdifferent Ġcell Ġcategory , Ġadip ocytes , Ġand Ġboth Ġare Ġmentioned Ġin Ġthe Ġtext . ĠWe Ġfeel Ġwith Ġmore Ġdata Ġand Ġadditional Ġtuning Ġor Ġperhaps Ġusing Ġan Ġensemble Ġmodel , Ġwe Ġmight Ġbe Ġable Ġto Ġkeep Ġthe Ġcorrect Ġanswers , Ġand Ġimprove Ġthe Ġresults Ġon Ġthe Ġconfusing Ġexamples Ġlike Ġthe Ġone Ġmentioned Ġabove . ĠTherefore Ġif Ġwe Ġimprove Ġour ĠâĢ ĺ L AT âĢ Ļ Ġderiv ation Ġlogic , Ġor Ġhave Ġlarger Ġdatasets , Ġthen Ġperhaps Ġthe Ġneural Ġnetwork Ġtechniques Ġthey Ġwill Ġyield Ġbetter Ġresults . </s> Our ĠSystems Ġand ĠTheir ĠPerformance Ġon ĠFact oid ĠQuestions Ġ: :: ĠImpact Ġof ĠTraining Ġusing ĠBio As q Ġdata Ġ( s lightly Ġnegative ) </s> Training Ġon ĠBio As q Ġdata Ġin Ġour Ġentry Ġin ĠB atch Ġ1 Ġand ĠB atch Ġ2 Ġunder Ġthe Ġname ĠQ A 1 Ġshowed Ġit Ġmight Ġlead Ġto Ġover fitting . ĠThis Ġhappened Ġboth Ġwith Ġ( B atch Ġ2 ) Ġand Ġwithout Ġ( B atch Ġ1 ) Ġhyper param eters Ġtuning : Ġab ys mal Ġ18 % ĠMR R Ġin ĠB atch Ġ1 , Ġand Ġsl igh ly Ġbetter Ġone , Ġ40 % Ġin ĠB atch Ġ2 Ġ( although Ġin ĠB atch Ġ2 Ġit Ġwas Ġoverall Ġthe Ġsecond Ġbest Ġresult Ġin ĠMR R Ġbut Ġ16 % Ġlower Ġthan Ġthe Ġhighest Ġscore ). </s> In ĠB atch Ġ3 Ġ( only ), Ġour ĠUN CC _ Q A 3 Ġsystem Ġwas Ġfine Ġtuned Ġon ĠBio As q Ġand ĠS Qu AD Ġ2 . 0 ĠB IB REF 7 , Ġand Ġfor Ġdata Ġpre processing ĠContext Ġparagraph Ġis Ġgenerated Ġfrom Ġrelevant Ġsnippets Ġprovided Ġin Ġthe Ġtest Ġdata . ĠThis Ġsystem Ġunder per formed , Ġby Ġabout Ġ2 % Ġin ĠMR R , Ġour Ġother Ġentry ĠUN CC _ Q A 1 , Ġwhich Ġwas Ġalso Ġan Ġoverall Ġcategory Ġwinner Ġfor Ġthis Ġbatch . ĠThe Ġlatter Ġwas Ġalso Ġtrained Ġon ĠS Qu AD , Ġbut Ġnot Ġon ĠBio As q . ĠWe Ġsuspect Ġthat Ġthe Ġreason Ġcould Ġbe Ġthe Ġsimplistic Ġnature Ġof Ġthe Ġfind () Ġfunction Ġdescribed Ġin ĠSection Ġ3 . 1 . ĠSo , Ġthis Ġcould Ġbe Ġan Ġarea Ġwhere Ġa Ġbetter Ġalgorithm Ġfor Ġfinding Ġthe Ġbest Ġoccurrence Ġof Ġan Ġentity Ġcould Ġimprove Ġperformance . </s> Our ĠSystems Ġand ĠTheir ĠPerformance Ġon ĠFact oid ĠQuestions Ġ: :: ĠImpact Ġof ĠUsing ĠContext Ġfrom ĠURLs Ġ( negative ) </s> In Ġsome Ġexperiments , Ġfor Ġcontext Ġin Ġtesting , Ġwe Ġused Ġdocuments Ġfor Ġwhich ĠURL Ġpointers Ġare Ġprovided Ġin ĠBio As q . ĠHowever , Ġour Ġsystem ĠUN CC _ Q A 3 Ġunder per formed Ġour Ġother Ġsystem Ġtested Ġonly Ġon Ġthe Ġprovided Ġsnippets . </s> In ĠB atch Ġ5 Ġthe Ġunder performance Ġwas Ġabout Ġ6 % Ġof ĠMR R , Ġcompared Ġto Ġour Ġbest Ġsystem ĠUN CC _ Q A 1 , Ġand Ġby Ġ9 % Ġto Ġthe Ġtop Ġperformer . </s> Performance Ġon ĠYes / No Ġand ĠList Ġquestions </s> Our Ġwork Ġfocused Ġon ĠFact oid Ġquestions . ĠBut Ġwe Ġalso Ġhave Ġdone Ġexperiments Ġon ĠList - type Ġand ĠYes / No Ġquestions . </s> Performance Ġon ĠYes / No Ġand ĠList Ġquestions Ġ: :: ĠEnt ail ment Ġimproves ĠYes / No Ġaccuracy </s> We Ġstarted Ġby Ġanswering Ġalways ĠYES Ġ( in Ġbatch Ġ2 Ġand Ġ3 ) Ġto Ġget Ġthe Ġbaseline Ġperformance . ĠFor Ġbatch Ġ4 Ġwe Ġused Ġentail ment . ĠOur Ġalgorithm Ġwas Ġvery Ġsimple : ĠGiven Ġa Ġquestion Ġwe Ġiter ate Ġthrough Ġthe Ġcandidate Ġsentences Ġand Ġtry Ġto Ġfind Ġany Ġcandidate Ġsentence Ġis Ġcontradict ing Ġthe Ġquestion Ġ( with Ġconfidence Ġover Ġ50 %), Ġif Ġso Ġ' No ' Ġis Ġreturned Ġas Ġanswer , Ġelse Ġ' Yes ' Ġis Ġreturned . ĠIn Ġbatch Ġ4 Ġthis Ġstrategy Ġproduced Ġbetter Ġthan Ġthe ĠBio As q Ġbaseline Ġperformance , Ġand Ġcompared Ġto Ġour Ġother Ġsystems , Ġthe Ġuse Ġof Ġentail ment Ġincreased Ġthe Ġperformance Ġby Ġabout Ġ13 % Ġ( mac ro ĠF 1 Ġscore ). ĠWe Ġused Ġ' Allen N lp ' ĠB IB REF 13 Ġentail ment Ġlibrary Ġto Ġfind Ġentail ment Ġof Ġthe Ġcandidate Ġsentences Ġwith Ġquestion . </s> Performance Ġon ĠYes / No Ġand ĠList Ġquestions Ġ: :: ĠFor ĠList - type Ġthe ĠURLs Ġhave Ġnegative Ġimpact </s> Overall , Ġwe Ġfollowed Ġthe Ġsimilar Ġstrategy Ġthat 's Ġbeen Ġfollowed Ġfor ĠFact oid ĠQuestion ĠAn sw ering Ġtask . ĠWe Ġstarted Ġour Ġexperiment Ġwith Ġbatch Ġ2 , Ġwhere Ġwe Ġsubmitted Ġ20 Ġbest Ġanswers Ġ( with Ġcontext Ġfrom Ġsnippets ). ĠStarting Ġwith Ġbatch Ġ3 , Ġwe Ġperformed Ġpost Ġprocessing : Ġonce Ġmodels Ġgenerate Ġanswer Ġpredictions Ġ( n - best Ġpredictions ), Ġwe Ġdo Ġpost - processing Ġon Ġthe Ġpredicted Ġanswers . ĠIn Ġtest Ġbatch Ġ4 , Ġour Ġsystem Ġ( called ĠFAC TO IDS ) Ġachieved Ġhighest Ġrecall Ġscore Ġof ĠâĢ ĺ 0 . 70 33 âĢ Ļ Ġbut Ġlow Ġprecision Ġof Ġ0 . 11 19 , Ġleaving Ġopen Ġthe Ġquestion Ġof Ġhow Ġcould Ġwe Ġhave Ġbetter Ġbalanced Ġthe Ġtwo Ġmeasures . </s> In Ġthe Ġpost - processing Ġphase , Ġwe Ġtake Ġthe Ġtop ĠâĢ ĺ 20 âĢ Ļ Ġ( batch Ġ3 ) Ġand Ġtop Ġ5 Ġ( batch Ġ4 Ġand Ġ5 ), Ġpredicted Ġanswers , Ġtoken ize Ġthem Ġusing Ġcommon Ġsepar ators : Ġ' com ma ' Ġ, Ġ' and ', Ġ' also ', Ġ' as Ġwell Ġas '. ĠTokens Ġwith Ġcharacters Ġcount Ġmore Ġthan ĠâĢ ĺ 100 âĢ Ļ Ġare Ġeliminated Ġand Ġrest Ġof Ġthe Ġtokens Ġare Ġadded Ġto Ġthe Ġlist Ġof Ġpossible Ġanswers . ĠBio AS Q Ġevaluation Ġmechanism Ġdoes Ġnot Ġconsider Ġsnippets Ġwith Ġmore Ġthan ĠâĢ ĺ 100 âĢ Ļ Ġcharacters Ġas Ġa Ġvalid Ġanswer . ĠConsidering Ġlengthy Ġsnippets Ġin Ġto Ġthe Ġlist Ġof Ġanswers Ġwould Ġreduce Ġthe Ġmean Ġprecision Ġscore . ĠAs Ġa Ġfinal Ġstep , Ġduplicate Ġsnippets Ġin Ġthe Ġanswer Ġpool Ġare Ġremoved . ĠFor Ġexample , Ġconsider Ġthese Ġtop Ġ3 Ġanswers Ġpredicted Ġby Ġthe Ġsystem Ġ( before Ġpost - processing ): </s> { </s> " text ": Ġ" d end ritic Ġcells ", </s> " pro b ability ": Ġ0 . 755 45 407 334 26 441 , </s> " start _ log it ": Ġ8 . 46 60 46 3333 129 88 , </s> " end _ log it ": Ġ9 . 536 355 01 86 157 23 </s> }, </s> { </s> " text ": Ġ" ne ut roph ils , Ġmac roph ages Ġand </s> dist inct Ġsub types Ġof Ġd end ritic Ġcells ", </s> " pro b ability ": Ġ0 . 13 80 68 67 348 30 42 14 , </s> " start _ log it ": Ġ6 . 76 64 785 385 13 184 , </s> " end _ log it ": Ġ9 . 536 355 01 86 157 23 </s> }, </s> { </s> " text ": Ġ" mac roph ages Ġand Ġdistinct Ġsub types Ġof Ġd end ritic ", </s> " pro b ability ": Ġ0 . 01 397 34 75 27 11 78 242 , </s> " start _ log it ": Ġ6 . 76 64 785 385 13 184 , </s> " end _ log it ": Ġ7 . 24 57 64 73 23 60 84 </s> }, </s> After Ġexecution Ġof Ġpost - processing Ġhe ur istics , Ġthe Ġlist Ġof Ġanswers Ġreturned Ġis Ġas Ġfollows : </s> [" d end ritic Ġcells "], </s> [" ne ut roph ils "], </s> [" mac roph ages "], </s> [" dist inct Ġsub types Ġof Ġd end ritic Ġcells "] </s> Summary Ġof Ġour Ġresults </s> The Ġtables Ġbelow Ġsummarize Ġall Ġour Ġresults . ĠThey Ġshow Ġthat Ġthe Ġperformance Ġof Ġour Ġsystems Ġwas Ġmixed . ĠThe Ġsimple Ġarchitectures Ġand Ġalgorithm Ġwe Ġused Ġworked Ġvery Ġwell Ġonly Ġin ĠB atch Ġ3 . ĠHowever , Ġwe Ġfeel Ġwe Ġcan Ġbuilt Ġa Ġbetter Ġsystem Ġbased Ġon Ġthis Ġexperience . ĠIn Ġparticular Ġwe Ġobserved Ġboth Ġthe Ġvalue Ġof Ġcontextual Ġembed d ings Ġand Ġof Ġfeature Ġengineering Ġ( L AT ), Ġhowever Ġwe Ġfailed Ġto Ġcombine Ġthem Ġproperly . </s> Summary Ġof Ġour Ġresults Ġ: :: ĠFact oid Ġquestions Ġ: :: ĠSystems Ġused Ġin ĠB atch Ġ5 Ġexperiments </s> System Ġdescription Ġfor ĠâĢ ĺ UN CC _ Q A 1 âĢ Ļ : ĠThe Ġsystem Ġwas Ġfin et uned Ġon Ġthe ĠS Qu AD Ġ2 . 0 . ĠFor Ġdata Ġpre processing ĠContext Ġ/ Ġparagraph Ġwas Ġgenerated Ġfrom Ġrelevant Ġsnippets Ġprovided Ġin Ġthe Ġtest Ġdata . </s> System Ġdescription Ġfor ĠâĢ ĺ Q A 1 âĢ Ļ Ġ: ĠâĢ ĺ L AT âĢ Ļ Ġfeature Ġwas Ġadded Ġand Ġfin et uned Ġwith ĠS Qu AD Ġ2 . 0 . ĠFor Ġdata Ġpre processing ĠContext Ġ/ Ġparagraph Ġwas Ġgenerated Ġfrom Ġrelevant Ġsnippets Ġprovided Ġin Ġthe Ġtest Ġdata . </s> System Ġdescription Ġfor ĠâĢ ĺ UN CC _ Q A 3 âĢ Ļ Ġ: ĠFine Ġtuning Ġprocess Ġis Ġsame Ġas Ġit Ġis Ġdone Ġfor Ġthe Ġsystem ĠâĢ ĺ UN CC _ Q A 1 âĢ Ļ Ġin Ġtest Ġbatch - 5 . ĠDifference Ġis Ġduring Ġdata Ġpre processing , ĠContext / paragraph Ġis Ġgenerated Ġfrom Ġthe Ġrelevant Ġdocuments Ġfor Ġwhich ĠUR LS Ġare Ġincluded Ġin Ġthe Ġtest Ġdata . </s> Summary Ġof Ġour Ġresults Ġ: :: ĠList ĠQuestions </s> For ĠList - type Ġquestions , Ġalthough Ġpost Ġprocessing Ġhelped Ġin Ġthe Ġlater Ġbatches , Ġwe Ġnever Ġmanaged Ġto Ġobtain Ġcompetitive Ġprecision , Ġalthough Ġour Ġrecall Ġwas Ġgood . </s> Summary Ġof Ġour Ġresults Ġ: :: ĠYes / No Ġquestions </s> The Ġonly Ġthing Ġworth Ġremembering Ġfrom Ġour Ġperformance Ġis Ġthat Ġusing Ġentail ment Ġcan Ġhave Ġa Ġmeasurable Ġimpact Ġ( at Ġleast Ġwith Ġrespect Ġto Ġa Ġweak Ġbaseline ). ĠThe Ġresults Ġ( weak ) Ġare Ġin ĠTable Ġ3 . </s> Discussion , ĠFuture ĠExper iments , Ġand ĠCon clusions Ġ: :: ĠSummary : </s> In Ġcontrast Ġto Ġ2018 , Ġwhen Ġwe Ġsubmitted ĠB IB REF 2 Ġto ĠBio AS Q Ġa Ġsystem Ġbased Ġon Ġextract ive Ġsummar ization Ġ( and Ġscored Ġvery Ġhigh Ġin Ġthe Ġideal Ġanswer Ġcategory ), Ġthis Ġyear Ġwe Ġmainly Ġtargeted Ġfact oid Ġquestion Ġanswering Ġtask Ġand Ġfocused Ġon Ġexperimenting Ġwith ĠBio BER T . ĠAfter Ġthese Ġexperiments Ġwe Ġsee Ġthe Ġpromise Ġof ĠBio BER T Ġin ĠQ A Ġtasks , Ġbut Ġwe Ġalso Ġsee Ġits Ġlimitations . ĠThe Ġlatter Ġwe Ġtried Ġto Ġaddress Ġwith Ġmixed Ġresults Ġusing Ġfeature Ġengineering . ĠOverall Ġthese Ġexperiments Ġallowed Ġus Ġto Ġsecure Ġa Ġbest Ġand Ġa Ġsecond Ġbest Ġscore Ġin Ġdifferent Ġtest Ġbatches . ĠAlong Ġwith ĠFact oid - type Ġquestion , Ġwe Ġalso Ġtried ĠâĢ ĺ Yes / No âĢ Ļ Ġand ĠâĢ ĺ List âĢ Ļ - type Ġquestions , Ġand Ġdid Ġreasonably Ġwell Ġwith Ġour Ġvery Ġsimple Ġapproach . </s> For ĠYes / No Ġthe Ġmoral Ġworth Ġremembering Ġis Ġthat Ġreasoning Ġhas Ġa Ġpotential Ġto Ġinfluence Ġresults , Ġas Ġevidenced Ġby Ġour Ġadding Ġthe ĠAllen N LP Ġentail ment ĠB IB REF 13 Ġsystem Ġincreased Ġits Ġperformance . </s> All Ġour Ġdata Ġand Ġsoftware Ġis Ġavailable Ġat ĠGithub , Ġin Ġthe Ġpreviously Ġreferenced ĠURL Ġ( end Ġof ĠSection Ġ2 ). </s> Discussion , ĠFuture ĠExper iments , Ġand ĠCon clusions Ġ: :: ĠFuture Ġexperiments </s> In Ġthe Ġcurrent Ġmodel , Ġwe Ġhave Ġa Ġshallow Ġneural Ġnetwork Ġwith Ġa Ġsoft max Ġlayer Ġfor Ġpredicting Ġanswer Ġspan . ĠSh allow Ġnetworks Ġhowever Ġare Ġnot Ġgood Ġat Ġgeneral izations . ĠIn Ġour Ġfuture Ġexperiments Ġwe Ġwould Ġlike Ġto Ġcreate Ġdense Ġquestion Ġanswering Ġneural Ġnetwork Ġwith Ġa Ġsoft max Ġlayer Ġfor Ġpredicting Ġanswer Ġspan . ĠThe Ġmain Ġidea Ġis Ġto Ġget Ġcontextual Ġword Ġembed ding Ġfor Ġthe Ġwords Ġpresent Ġin Ġthe Ġquestion Ġand Ġparagraph Ġ( Context ) Ġand Ġfeed Ġthe Ġcontextual Ġword Ġembed d ings Ġretrieved Ġfrom Ġthe Ġlast Ġlayer Ġof ĠBio BER T Ġto Ġthe Ġdense Ġquestion Ġanswering Ġnetwork . ĠThe Ġmentioned Ġdense Ġlayered Ġquestion Ġanswering Ġneural Ġnetwork Ġneed Ġto Ġbe Ġtuned Ġfor Ġfinding Ġright Ġhyper Ġparameters . ĠAn Ġexample Ġof Ġsuch Ġarchitecture Ġis Ġshown Ġin ĠFig . FIG REF 30 . </s> In Ġone Ġmore Ġexperiment , Ġwe Ġwould Ġlike Ġto Ġadd Ġa Ġbetter Ġversion Ġof ĠâĢ ĺ L AT âĢ Ļ Ġcontextual Ġword Ġembed ding Ġas Ġa Ġfeature , Ġalong Ġwith Ġthe Ġactual Ġcontextual Ġword Ġembed d ings Ġfor Ġquestion Ġtext , Ġand ĠContext Ġand Ġfeed Ġthem Ġas Ġinput Ġto Ġthe Ġdense Ġquestion Ġanswering Ġneural Ġnetwork . ĠBy Ġthis Ġexperiment , Ġwe Ġwould Ġlike Ġto Ġfind Ġif ĠâĢ ĺ L AT âĢ Ļ Ġfeature Ġis Ġimproving Ġoverall Ġanswer Ġprediction Ġaccuracy . ĠAdding ĠâĢ ĺ L AT âĢ Ļ Ġfeature Ġthis Ġway Ġinstead Ġof Ġfeeding Ġthis Ġword Ġpiece Ġembed ding Ġdirectly Ġto Ġthe ĠBio BER T Ġ( as Ġwe Ġdid Ġin Ġour Ġabove Ġexperiments ) Ġwould Ġnot Ġdowngrade Ġthe Ġquality Ġof Ġcontextual Ġword Ġembed d ings Ġgenerated Ġform ĠâĢ ĺ Bio BER T '. ĠQuality Ġcontextual Ġword Ġembed d ings Ġwould Ġlead Ġto Ġefficient Ġtransfer Ġlearning Ġand Ġchances Ġare Ġthat Ġit Ġwould Ġimprove Ġthe Ġmodel 's Ġanswer Ġprediction Ġaccuracy . </s> We Ġalso Ġsee Ġpotential Ġfor Ġincorporating Ġdomain Ġspecific Ġinference Ġinto Ġthe Ġtask Ġe . g . Ġusing Ġthe ĠMed N LI Ġdataset ĠB IB REF 14 . ĠFor Ġall Ġtypes Ġof Ġexperiments Ġit Ġmight Ġbe Ġworth Ġexploring Ġclinical ĠB ERT Ġembed d ings ĠB IB REF 15 , Ġexplicitly Ġincorporating Ġdomain Ġknowledge Ġ( e . g . ĠB IB REF 16 ) Ġand Ġpossibly Ġdeeper Ġdiscourse Ġrepresentations Ġ( e . g . ĠB IB REF 17 ). </s> APP END IX </s> In Ġthis Ġappendix Ġwe Ġprovide Ġadditional Ġdetails Ġabout Ġthe Ġimplementations . </s> APP END IX Ġ: :: ĠSystems Ġand Ġtheir Ġdescriptions : </s> We Ġused Ġseveral Ġvariants Ġof Ġour Ġsystems Ġwhen Ġexperimenting Ġwith Ġthe ĠBio AS Q Ġproblems . ĠIn Ġretrospect , Ġit Ġwould Ġbe Ġmuch Ġeasier Ġto Ġunderstand Ġthe Ġchanges Ġif Ġwe Ġadopted Ġsome Ġm n emonic Ġconventions Ġin Ġnaming Ġthe Ġsystems . ĠSo , Ġwe Ġapologize Ġfor Ġthe Ġnames Ġthat Ġdo Ġnot Ġreflect Ġthe Ġmodifications , Ġand Ġnecess itate Ġthis Ġlist . </s> APP END IX Ġ: :: ĠSystems Ġand Ġtheir Ġdescriptions : Ġ: :: ĠFact oid ĠType ĠQuestion ĠAn sw ering : </s> We Ġpre process ed Ġthe Ġtest Ġdata Ġto Ġconvert Ġtest Ġdata Ġto ĠBio BER T Ġformat , ĠWe Ġgenerated ĠContext / paragraph Ġby Ġeither Ġaggreg ating Ġrelevant Ġsnippets Ġprovided Ġor Ġby Ġaggreg ating Ġdocuments Ġfor Ġwhich ĠUR LS Ġare Ġprovided Ġin Ġthe ĠBio AS Q Ġtest Ġdata . </s> APP END IX Ġ: :: ĠSystems Ġand Ġtheir Ġdescriptions : Ġ: :: ĠSystem Ġdescription Ġfor ĠQ A 1 : </s> We Ġgenerated ĠContext / paragraph Ġby Ġaggreg ating Ġrelevant Ġsnippets Ġavailable Ġin Ġthe Ġtest Ġdata Ġand Ġmapped Ġit Ġagainst Ġthe Ġquestion Ġtext Ġand Ġquestion Ġid . ĠWe Ġignored Ġthe Ġcontent Ġpresent Ġin Ġthe Ġdocuments Ġ( document ĠUR LS Ġwere Ġprovided Ġin Ġthe Ġoriginal Ġtest Ġdata ). ĠThe Ġmodel Ġis Ġfin et uned Ġwith ĠBio AS Q Ġdata . </s> data Ġpre processing Ġis Ġdone Ġin Ġthe Ġsame Ġway Ġas Ġit Ġis Ġdone Ġfor Ġtest Ġbatch - 1 . ĠModel Ġfine Ġtuned Ġon ĠBio AS Q Ġdata . </s> âĢ ĺ L AT âĢ Ļ / ĠFocus Ġword Ġfeature Ġadded Ġand Ġfine Ġtuned Ġwith ĠS Qu AD Ġ2 . 0 Ġ[ reference ]. ĠFor Ġdata Ġpre processing ĠContext Ġ/ Ġparagraph Ġis Ġgenerated Ġfrom Ġrelevant Ġsnippets Ġprovided Ġin Ġthe Ġtest Ġdata . </s> APP END IX Ġ: :: ĠSystems Ġand Ġtheir Ġdescriptions : Ġ: :: ĠSystem Ġdescription Ġfor ĠUN CC _ Q A _ 1 : </s> System Ġis Ġfin et uned Ġon Ġthe ĠS Qu AD Ġ2 . 0 Ġ[ reference ]. ĠFor Ġdata Ġpre processing ĠContext Ġ/ Ġparagraph Ġis Ġgenerated Ġfrom Ġrelevant Ġsnippets Ġprovided Ġin Ġthe Ġtest Ġdata . </s> âĢ ĺ L AT âĢ Ļ / ĠFocus Ġword Ġfeature Ġadded Ġand Ġfine Ġtuned Ġwith ĠS Qu AD Ġ2 . 0 Ġ[ reference ]. ĠFor Ġdata Ġpre processing ĠContext Ġ/ Ġparagraph Ġis Ġgenerated Ġfrom Ġrelevant Ġsnippets Ġprovided Ġin Ġthe Ġtest Ġdata . </s> The ĠSystem Ġis Ġfin et uned Ġon Ġthe ĠS Qu AD Ġ2 . 0 . ĠFor Ġdata Ġpre processing ĠContext Ġ/ Ġparagraph Ġis Ġgenerated Ġfrom Ġrelevant Ġsnippets Ġprovided Ġin Ġthe Ġtest Ġdata . </s> APP END IX Ġ: :: ĠSystems Ġand Ġtheir Ġdescriptions : Ġ: :: ĠSystem Ġdescription Ġfor ĠUN CC _ Q A 3 : </s> System Ġis Ġfin et uned Ġon Ġthe ĠS Qu AD Ġ2 . 0 Ġ[ reference ] Ġand ĠBio AS Q Ġdataset [ ]. For Ġdata Ġpre processing ĠContext Ġ/ Ġparagraph Ġis Ġgenerated Ġfrom Ġrelevant Ġsnippets Ġprovided Ġin Ġthe Ġtest Ġdata . </s> Fine Ġtuning Ġprocess Ġis Ġsame Ġas Ġit Ġis Ġdone Ġfor Ġthe Ġsystem ĠâĢ ĺ UN CC _ Q A _ 1 âĢ Ļ Ġin Ġtest Ġbatch - 5 . ĠDifference Ġis Ġduring Ġdata Ġpre processing , ĠContext / paragraph Ġis Ġgenerated Ġform Ġfrom Ġthe Ġrelevant Ġdocuments Ġfor Ġwhich ĠUR LS Ġare Ġincluded Ġin Ġthe Ġtest Ġdata . </s> APP END IX Ġ: :: ĠSystems Ġand Ġtheir Ġdescriptions : Ġ: :: ĠSystem Ġdescription Ġfor ĠUN CC _ Q A 2 : </s> Fine Ġtuning Ġprocess Ġis Ġsame Ġas Ġfor ĠâĢ ĺ UN CC _ Q A _ 1 ĠâĢ Ļ . ĠDifference Ġis ĠContext / paragraph Ġis Ġgenerated Ġform Ġfrom Ġthe Ġrelevant Ġdocuments Ġfor Ġwhich ĠUR LS Ġare Ġincluded Ġin Ġthe Ġtest Ġdata . ĠSystem ĠâĢ ĺ UN CC _ Q A _ 1 âĢ Ļ Ġgot Ġthe Ġhighest ĠâĢ ĺ MR R âĢ Ļ Ġscore Ġin Ġthe Ġ3 rd Ġtest Ġbatch Ġset . </s> APP END IX Ġ: :: ĠSystems Ġand Ġtheir Ġdescriptions : Ġ: :: ĠSystem Ġdescription Ġfor ĠFAC TO IDS : </s> The ĠSystem Ġis Ġfin et uned Ġon Ġthe ĠS Qu AD Ġ2 . 0 . ĠFor Ġdata Ġpre processing ĠContext Ġ/ Ġparagraph Ġis Ġgenerated Ġfrom Ġrelevant Ġsnippets Ġprovided Ġin Ġthe Ġtest Ġdata . </s> APP END IX Ġ: :: ĠSystems Ġand Ġtheir Ġdescriptions : Ġ: :: ĠList ĠType ĠQuestions : </s> We Ġattempted ĠList Ġtype Ġquestions Ġstarting Ġfrom Ġtest Ġbatch ĠâĢ ĺ 2 âĢ Ļ . ĠUsed Ġsimilar Ġapproach Ġthat 's Ġbeen Ġfollowed Ġfor ĠFact oid ĠQuestion Ġanswering Ġtask . ĠFor Ġall Ġthe Ġtest Ġbatch Ġsets , Ġin Ġthe Ġdata Ġpre Ġprocessing Ġphase ĠContext / Ġparagraph Ġis Ġgenerated Ġeither Ġby Ġaggreg ating Ġrelevant Ġsnippets Ġor Ġby Ġaggreg ating Ġdocuments ( UR LS ) Ġprovided Ġin Ġthe ĠBio AS Q Ġtest Ġdata . </s> For Ġtest Ġbatch - 2 , Ġmodel Ġ( System : ĠQ A 1 ) Ġis Ġfin et uned Ġon ĠBio AS Q Ġdata Ġand Ġsubmitted Ġtop ĠâĢ ĺ 20 âĢ Ļ Ġanswers Ġpredicted Ġby Ġthe Ġmodel Ġas Ġthe Ġlist Ġof Ġanswers . Ġsystem ĠâĢ ĺ Q A 1 âĢ Ļ Ġachieved Ġlow ĠF - Measure Ġscore : âĢ ĺ 0 . 07 86 âĢ Ļ Ġin Ġthe Ġsecond Ġtest Ġbatch . ĠIn Ġthe Ġfurther Ġtest Ġbatches Ġfor ĠList Ġtype Ġquestions , Ġwe Ġfin et uned Ġthe Ġmodel Ġon ĠSquad Ġdata Ġset Ġ[ reference ], Ġimplemented Ġpost Ġprocessing Ġtechniques Ġ( re fer Ġsection Ġ5 . 2 ) Ġand Ġachieved Ġa Ġbetter ĠF - me asure Ġscore : ĠâĢ ĺ 0 . 28 62 âĢ Ļ Ġin Ġthe Ġfinal Ġtest Ġbatch Ġset . </s> In Ġtest Ġbatch - 3 Ġ( System s Ġ: ĠâĢ ĺ Q A 1 âĢ Ļ / âĢ Ļ âĢ Ļ UN CC _ Q A _ 1 âĢ Ļ / âĢ Ļ UN CC _ Q A 3 âĢ Ļ / âĢ Ļ UN CC _ Q A 2 âĢ Ļ ) Ġtop Ġ20 Ġanswers Ġreturned Ġby Ġthe Ġmodel Ġis Ġsent Ġfor Ġpost Ġprocessing Ġand Ġin Ġtest Ġbatch Ġ4 Ġand Ġ5 Ġonly Ġtop Ġ5 Ġanswers Ġare Ġsent Ġfor Ġpost Ġprocessing . ĠSystem ĠUN CC _ Q A 2 ( in Ġbatch Ġ3 ) Ġfor ĠList Ġtype Ġquestion Ġanswering , ĠContext Ġis Ġgenerated Ġfrom Ġdocuments Ġfor Ġwhich ĠUR LS Ġare Ġprovided Ġin Ġthe ĠBio AS Q Ġtest Ġdata . Ġfor Ġthe Ġrest Ġof Ġthe Ġsystems Ġ( in Ġtest Ġbatch - 3 ) Ġfor ĠList ĠType Ġquestion Ġanswering Ġtask Ġsnippets Ġpresent Ġin Ġthe ĠBio a S Q Ġtest Ġdata Ġare Ġused Ġto Ġgenerate Ġcontext . </s> In Ġtest Ġbatch - 4 Ġ( System Ġ: ĠâĢ ĺ F AC TO IDS âĢ Ļ / âĢ Ļ UN CC _ Q A _ 1 âĢ Ļ / âĢ Ļ UN CC _ Q A 3 âĢ Ļ âĢ Ļ ) Ġtop Ġ5 Ġanswers Ġreturned Ġby Ġthe Ġmodel Ġis Ġsent Ġfor Ġpost Ġprocessing . ĠIn Ġcase Ġof Ġsystem ĠâĢ ĺ F AC TO IDS âĢ Ļ Ġsnippets Ġin Ġthe Ġtest Ġdata Ġwere Ġused Ġto Ġgenerate Ġcontext . Ġfor Ġsystems ĠâĢ Ļ UN CC _ Q A _ 1 âĢ Ļ Ġand ĠâĢ Ļ UN CC _ Q A 3 âĢ Ļ Ġcontext Ġis Ġgenerated Ġfrom Ġthe Ġdocuments Ġfor Ġwhich ĠUR LS Ġare Ġprovided Ġin Ġthe ĠBio AS Q Ġtest Ġdata . </s> In Ġtest Ġbatch - 5 Ġ( ĠSystems : ĠâĢ ĺ Q A 1 âĢ Ļ / âĢ Ļ UN CC _ Q A _ 1 âĢ Ļ / âĢ Ļ UN CC _ Q A 3 âĢ Ļ / âĢ Ļ UN CC _ Q A 2 âĢ Ļ Ġ) Ġour Ġapproach Ġis Ġthe Ġsame Ġas Ġthat Ġof Ġtest Ġbatch - 4 Ġwhere Ġtop Ġ5 Ġanswers Ġreturned Ġby Ġthe Ġmodel Ġis Ġsent Ġfor Ġpost Ġprocessing . Ġfor Ġall Ġthe Ġsystems Ġ( in Ġtest Ġbatch - 5 ) Ġcontext Ġis Ġgenerated Ġfrom Ġthe Ġsnippets Ġprovided Ġin Ġthe ĠBio AS Q Ġtest Ġdata . </s> APP END IX Ġ: :: ĠSystems Ġand Ġtheir Ġdescriptions : Ġ: :: ĠYes / No ĠType ĠQuestions : </s> For Ġthe Ġfirst Ġ3 Ġtest Ġbatches , ĠWe Ġhave Ġsubmitted Ġanswer ĠâĢ ĺ Yes âĢ Ļ Ġto Ġall Ġthe Ġquestions . ĠLater , Ġwe Ġemployed ĠâĢ ĺ Sent ence ĠEnt ail ment âĢ Ļ Ġtechniques ( re fer Ġsection Ġ6 . 0 ) Ġfor Ġthe Ġfourth Ġand Ġfifth Ġtest Ġbatch Ġsets . ĠOur ĠSystems Ġwith ĠâĢ ĺ Sent ence ĠEnt ail ment âĢ Ļ Ġapproach Ġ( for ĠâĢ ĺ Yes âĢ Ļ / ĠâĢ ĺ No âĢ Ļ Ġquestion Ġanswering ): ĠâĢ ĺ UN CC _ Q A _ 1 âĢ Ļ ( test Ġbatch - 4 ), ĠUN CC _ Q A 3 ( test Ġbatch - 5 ). </s> APP END IX Ġ: :: ĠAdditional Ġdetails Ġfor ĠYes / No ĠType ĠQuestions </s> We Ġused ĠText ual ĠEnt ail ment Ġin ĠB atch Ġ4 Ġand Ġ5 Ġfor ĠâĢ ĺ Yes âĢ Ļ / âĢ ĺ No âĢ Ļ Ġquestion Ġtype . ĠThe Ġalgorithm Ġwas Ġvery Ġsimple : ĠGiven Ġa Ġquestion Ġwe Ġiter ate Ġthrough Ġthe Ġcandidate Ġsentences , Ġand Ġlook Ġfor Ġany Ġcandidate Ġsentences Ġcontradict ing Ġthe Ġquestion . ĠIf Ġwe Ġfind Ġone Ġ' No ' Ġis Ġreturned Ġas Ġanswer , Ġelse Ġ' Yes ' Ġis Ġreturned . Ġ( The Ġconfidence Ġfor Ġcontradiction Ġwas Ġset Ġat Ġ50 %) ĠWe Ġused ĠAllen N LP ĠB IB REF 13 Ġentail ment Ġlibrary Ġto Ġfind Ġentail ment Ġof Ġthe Ġcandidate Ġsentences Ġwith Ġquestion . </s> Flow ĠChart Ġfor ĠYes / No ĠQuestion Ġanswer Ġprocessing Ġis Ġshown Ġin ĠFig . FIG REF 51 </s> APP END IX Ġ: :: ĠAss um ptions , Ġrules Ġand Ġlogic Ġflow Ġfor Ġder iving ĠLex ical ĠAnswer ĠTypes Ġfrom Ġquestions </s> There Ġare Ġdifferent Ġquestion Ġtypes , Ġand Ġwe Ġdistinguished Ġthem Ġbased Ġon Ġthe Ġquestion Ġwords : ĠâĢ ĺ Which âĢ Ļ , ĠâĢ ĺ What âĢ Ļ , ĠâĢ ĺ When âĢ Ļ , ĠâĢ ĺ How âĢ Ļ Ġetc . ĠEach Ġtype Ġof Ġquestion Ġis Ġbeing Ġhandled Ġdifferently Ġand Ġthere Ġare Ġcommon alities Ġamong Ġthe Ġrules Ġwritten Ġfor Ġdifferent Ġquestion Ġtypes . ĠHow Ġare Ġquestion Ġwords Ġidentified ? Ġquestion Ġwords Ġhave Ġparts Ġof Ġspeech ( POS ): Ġ' WD T ', Ġ' WR B ', Ġ' WP '. </s> Ass um ptions : </s> 1 ) ĠLex ical Ġanswer Ġtype Ġ( âĢ ĺ L AT âĢ Ļ ) Ġor Ġfocus Ġword Ġis Ġof Ġtype ĠN oun Ġand Ġfollows Ġthe Ġquestion Ġword . </s> 2 ) ĠThe ĠLAT Ġword Ġis Ġa ĠSubject . Ġ( This Ġclearly Ġnot Ġalways Ġtrue , Ġbut Ġwe Ġused Ġa Ġvery Ġsimple Ġmethod ). ĠNote : ĠâĢ ĺ Stan ford N LP âĢ Ļ Ġdependency Ġparsing Ġtag Ġfor Ġidentifying Ġsubject Ġis Ġ' ns ub j ' Ġor Ġ' ns ub j pass '. </s> 3 ) ĠWhen Ġa Ġquestion Ġhas Ġmultiple Ġwords Ġthat Ġare Ġof Ġtype ĠSubject Ġ( and ĠN oun ), Ġa Ġword Ġthat Ġis Ġin Ġproximity Ġto Ġthe Ġquestion Ġword Ġis Ġconsidered Ġas ĠâĢ ĺ L AT âĢ Ļ . </s> 4 ) ĠFor Ġquestions Ġwith Ġquestion Ġwords : ĠâĢ ĺ When âĢ Ļ , ĠâĢ ĺ Who âĢ Ļ , ĠâĢ ĺ Why âĢ Ļ , Ġthe ĠâĢ Ļ L AT âĢ Ļ Ġis Ġa Ġquestion Ġword Ġitself Ġthat Ġis , ĠâĢ ĺ When âĢ Ļ , ĠâĢ ĺ Who âĢ Ļ , ĠâĢ ĺ Why âĢ Ļ Ġrespectively . </s> Rules Ġand Ġlogic Ġflow Ġto Ġtraverse Ġa Ġquestion : ĠThe Ġthree Ġcases Ġbelow Ġdescribe Ġthe Ġlogic Ġflow Ġof Ġfinding ĠLAT s . ĠThe Ġfigures Ġshow Ġthe Ġgram matical Ġstructures Ġused Ġfor Ġthis Ġpurpose . </s> APP END IX Ġ: :: ĠAss um ptions , Ġrules Ġand Ġlogic Ġflow Ġfor Ġder iving ĠLex ical ĠAnswer ĠTypes Ġfrom Ġquestions Ġ: :: ĠCase - 1 : </s> Question Ġwith Ġquestion Ġword ĠâĢ ĺ How âĢ Ļ . </s> For Ġquestions Ġwith Ġquestion Ġword Ġ' How ', Ġthe Ġadjective Ġthat Ġfollows Ġthe Ġquestion Ġword Ġis Ġconsidered Ġas ĠâĢ ĺ L AT âĢ Ļ Ġ( need Ġnot Ġfollow Ġimmediately ). ĠIf Ġan Ġadjective Ġis Ġabsent , Ġword Ġ' How ' Ġis Ġconsidered Ġas ĠâĢ ĺ L AT âĢ Ļ . ĠWhen Ġthere Ġare Ġmultiple Ġwords Ġthat Ġare Ġadject ives , Ġa Ġword Ġin Ġclose Ġproximity Ġto Ġthe Ġquestion Ġword Ġand Ġfollows Ġit Ġis Ġreturned Ġas ĠâĢ ĺ L AT âĢ Ļ . ĠNote : ĠThe Ġpart Ġof Ġspeech Ġtag Ġto Ġidentify Ġadject ives Ġis Ġ' JJ '. ĠFor ĠOther Ġpossible Ġquestion Ġwords Ġlike ĠâĢ ĺ whose âĢ Ļ . ĠâĢ ĺ L AT âĢ Ļ / Focus Ġword Ġis Ġquestion Ġwords Ġitself . </s> Example ĠQuestion : ĠHow Ġmany Ġse len op rote ins Ġare Ġencoded Ġin Ġthe Ġhuman Ġgenome ? </s> APP END IX Ġ: :: ĠAss um ptions , Ġrules Ġand Ġlogic Ġflow Ġfor Ġder iving ĠLex ical ĠAnswer ĠTypes Ġfrom Ġquestions Ġ: :: ĠCase - 2 : </s> Questions Ġwith Ġquestion Ġwords ĠâĢ ĺ Which âĢ Ļ Ġ, ĠâĢ ĺ What âĢ Ļ Ġand Ġall Ġother Ġpossible Ġquestion Ġwords ; Ġa Ġ' N oun ' Ġimmediately Ġfollowing Ġthe Ġquestion Ġword . </s> Example ĠQuestion : ĠWhich Ġenzyme Ġis Ġtargeted Ġby ĠEv ol oc um ab ? </s> Here , ĠFocus Ġword / L AT Ġis ĠâĢ ĺ en zyme âĢ Ļ Ġwhich Ġis Ġboth ĠN oun Ġand ĠSubject Ġand Ġimmediately Ġfollows Ġthe Ġquestion Ġword . </s> When Ġthe Ġword Ġimmediately Ġfollowing Ġthe Ġquestion Ġword Ġis Ġa Ġnoun , Ġthe Ġwindow Ġsize Ġis Ġset Ġto ĠâĢ ĺ 3 âĢ Ļ . ĠThis Ġsize ĠâĢ ĺ 3 âĢ Ļ Ġmeans Ġthat Ġwe Ġiter ate Ġthrough Ġthe Ġnext ĠâĢ ĺ 3 âĢ Ļ Ġwords Ġ( if Ġpresent ) Ġto Ġcheck Ġif Ġany Ġof Ġthe Ġword Ġis Ġboth Ġ' N oun ' Ġand Ġ' Subject ', ĠIf Ġso , Ġthe Ġword Ġis Ġconsidered Ġas ĠâĢ ĺ L AT âĢ Ļ / Focus ĠWord . ĠElse Ġthe Ġword Ġthat Ġis Ġpresent Ġvery Ġnext Ġto Ġthe Ġquestion Ġword Ġis Ġconsidered Ġas ĠâĢ ĺ L AT âĢ Ļ . </s> APP END IX Ġ: :: ĠAss um ptions , Ġrules Ġand Ġlogic Ġflow Ġfor Ġder iving ĠLex ical ĠAnswer ĠTypes Ġfrom Ġquestions Ġ: :: ĠCase - 3 : </s> Questions Ġwith Ġquestion Ġwords ĠâĢ ĺ Which âĢ Ļ Ġ, ĠâĢ ĺ What âĢ Ļ Ġand Ġall Ġother Ġpossible Ġquestion Ġwords ; Ġword Ġimmediately Ġfollowing Ġthe Ġquestion Ġword Ġis Ġnot Ġa Ġ' N oun '. </s> Example ĠQuestion : ĠWhat Ġis Ġthe Ġfunction Ġof Ġthe Ġprotein ĠMag t 1 ? </s> Here , ĠFocus Ġword / L AT Ġis ĠâĢ ĺ function ĠâĢ Ļ Ġwhich Ġis Ġboth ĠN oun Ġand ĠSubject Ġand Ġdoes Ġnot Ġimmediately Ġfollow Ġthe Ġquestion Ġword . </s> When Ġthe Ġvery Ġnext Ġword Ġfollowing Ġthe Ġquestion Ġword Ġis Ġnot Ġa ĠN oun , Ġwindow Ġsize Ġis Ġset Ġto ĠâĢ ĺ 5 âĢ Ļ . ĠWindow Ġsize ĠâĢ ĺ 5 âĢ Ļ Ġcorresponds Ġthat Ġwe Ġiter ate Ġthrough Ġthe Ġnext ĠâĢ ĺ 5 âĢ Ļ Ġwords Ġ( if Ġpresent ) Ġand Ġsearch Ġfor Ġthe Ġword Ġthat Ġis Ġboth ĠN oun Ġand ĠSubject . ĠIf Ġpresent , Ġthe Ġword Ġis Ġconsidered Ġas ĠâĢ ĺ L AT âĢ Ļ . ĠElse , Ġthe Ġ' N oun ' Ġclose Ġproximity Ġto Ġthe Ġquestion Ġword Ġand Ġfollows Ġit Ġis Ġreturned Ġas ĠâĢ ĺ L AT âĢ Ļ . </s> Ad Ġwe Ġmentioned Ġearlier , Ġthe Ġaccuracy Ġfor ĠâĢ ĺ L AT âĢ Ļ Ġderiv ation Ġis Ġ75 Ġpercent . ĠBut Ġclearly Ġthe Ġsimple Ġlogic Ġdescribed Ġabove Ġcan Ġbe Ġimproved , Ġas Ġshown Ġin ĠB IB REF 9 , ĠB IB REF 10 . ĠWhether Ġthis Ġin Ġturn Ġproduces Ġimprovements Ġin Ġthis Ġparticular Ġtask Ġis Ġan Ġopen Ġquestion . </s> APP END IX Ġ: :: ĠPro posing ĠFuture ĠExper iments </s> In Ġthe Ġcurrent Ġmodel , Ġwe Ġhave Ġa Ġshallow Ġneural Ġnetwork Ġwith Ġa Ġsoft max Ġlayer Ġfor Ġpredicting Ġanswer Ġspan . ĠSh allow Ġnetworks Ġhowever Ġare Ġnot Ġgood Ġat Ġgeneral izations . ĠIn Ġour Ġfuture Ġexperiments Ġwe Ġwould Ġlike Ġto Ġcreate Ġdense Ġquestion Ġanswering Ġneural Ġnetwork Ġwith Ġa Ġsoft max Ġlayer Ġfor Ġpredicting Ġanswer Ġspan . ĠThe Ġmain Ġidea Ġis Ġto Ġget Ġcontextual Ġword Ġembed ding Ġfor Ġthe Ġwords Ġpresent Ġin Ġthe Ġquestion Ġand Ġparagraph Ġ( Context ) Ġand Ġfeed Ġthe Ġcontextual Ġword Ġembed d ings Ġretrieved Ġfrom Ġthe Ġlast Ġlayer Ġof ĠBio BER T Ġto Ġthe Ġdense Ġquestion Ġanswering Ġnetwork . ĠThe Ġmentioned Ġdense Ġlayered Ġquestion Ġanswering ĠNeural Ġnetwork Ġneed Ġto Ġbe Ġtuned Ġfor Ġfinding Ġright Ġhyper Ġparameters . ĠAn Ġexample Ġof Ġsuch Ġarchitecture Ġis Ġshown Ġin ĠFig . FIG REF 30 . </s> In Ġanother Ġexperiment Ġwe Ġwould Ġlike Ġto Ġonly Ġfeed Ġcontextual Ġword Ġembed d ings Ġfor ĠFocus Ġword / ĠâĢ ĺ L AT âĢ Ļ , Ġparagraph / ĠContext Ġas Ġinput Ġto Ġthe Ġquestion Ġanswering Ġneural Ġnetwork . ĠIn Ġthis Ġexperiment Ġwe Ġwould Ġneglect Ġall Ġembed d ings Ġfor Ġthe Ġquestion Ġtext Ġexcept Ġthat Ġof ĠFocus Ġword / ĠâĢ ĺ L AT âĢ Ļ . ĠOur Ġassumption Ġand Ġidea Ġfor Ġconsidering Ġfocus Ġword Ġand Ġneglect ing Ġremaining Ġwords Ġin Ġthe Ġquestion Ġis Ġthat Ġduring Ġtraining Ġphase Ġit Ġwould Ġmake Ġmore Ġprecise Ġfor Ġthe Ġmodel Ġto Ġidentify Ġthe Ġfocus Ġof Ġthe Ġquestion Ġand Ġmap Ġanswers Ġagainst Ġthe Ġquestion âĢ Ļ s Ġfocus . ĠTo Ġvalidate Ġour Ġassumption , Ġwe Ġwould Ġlike Ġto Ġtake Ġsample Ġquestion Ġanswering Ġdata Ġand Ġfind Ġthe Ġcos ine Ġdistance Ġbetween Ġcontextual Ġembed ding Ġof ĠFocus Ġword Ġand Ġthat Ġof Ġthe Ġactual Ġanswer Ġand Ġverify Ġif Ġthe Ġcos ine Ġdistance Ġis Ġcomparatively Ġlow Ġin Ġmost Ġof Ġthe Ġcases . </s> In Ġone Ġmore Ġexperiment , Ġwe Ġwould Ġlike Ġto Ġadd Ġa Ġbetter Ġversion Ġof ĠâĢ ĺ L AT âĢ Ļ Ġcontextual Ġword Ġembed ding Ġas Ġa Ġfeature , Ġalong Ġwith Ġthe Ġactual Ġcontextual Ġword Ġembed d ings Ġfor Ġquestion Ġtext , Ġand ĠContext Ġand Ġfeed Ġthem Ġas Ġinput Ġto Ġthe Ġdense Ġquestion Ġanswering Ġneural Ġnetwork . ĠBy Ġthis Ġexperiment , Ġwe Ġwould Ġlike Ġto Ġfind Ġif ĠâĢ ĺ L AT âĢ Ļ Ġfeature Ġis Ġimproving Ġoverall Ġanswer Ġprediction Ġaccuracy . ĠAdding ĠâĢ ĺ L AT âĢ Ļ Ġfeature Ġthis Ġway Ġinstead Ġof Ġfeeding ĠFocus Ġword âĢ Ļ s Ġword Ġpiece Ġembed ding Ġdirectly Ġ( as Ġwe Ġdid Ġin Ġour Ġabove Ġexperiments ) Ġto Ġthe ĠBio BER T Ġwould Ġnot Ġdowngrade Ġthe Ġquality Ġof Ġcontextual Ġword Ġembed d ings Ġgenerated Ġform ĠâĢ ĺ Bio BER T '. ĠQuality Ġcontextual Ġword Ġembed d ings Ġwould Ġlead Ġto Ġefficient Ġtransfer Ġlearning Ġand Ġchances Ġare Ġthat Ġit Ġwould Ġimprove Ġthe Ġmodel 's Ġanswer Ġprediction Ġaccuracy .
Introduction BioASQ is a biomedical document classification, document retrieval, and question answering competition, currently in its seventh year. We provide an overview of our submissions to semantic question answering task (7b, Phase B) of BioASQ 7 (except for 'ideal answer' test, in which we did not participate this year). In this task systems are provided with biomedical questions and are required to submit ideal and exact answers to those questions. We have used BioBERT BIBREF0 based system , see also Bidirectional Encoder Representations from Transformers(BERT) BIBREF1, and we fine tuned it for the biomedical question answering task. Our system scored near the top for factoid questions for all the batches of the challenge. More specifially, in the third test batch set, our system achieved highest ‘MRR’ score for Factoid Question Answering task. Also, for List-type question answering task our system achieved highest recall score in the fourth test batch set. Along with our detailed approach, we present the results for our submissions and also highlight identified downsides for our current approach and ways to improve them in our future experiments. In last test batch results we placed 4th for List-type questions and 3rd for Factoid-type questions.) The QA task is organized in two phases. Phase A deals with retrieval of the relevant document, snippets, concepts, and RDF triples, and phase B deals with exact and ideal answer generations (which is a paragraph size summary of snippets). Exact answer generation is required for factoid, list, and yes/no type question. BioASQ organizers provide the training and testing data. The training data consists of questions, gold standard documents, snippets, concepts, and ideal answers (which we did not use in this paper, but we used last year BIBREF2). The test data is split between phases A and B. The Phase A dataset consists of the questions, unique ids, question types. The Phase B dataset consists of the questions, golden standard documents, snippets, unique ids and question types. Exact answers for factoid type questions are evaluated using strict accuracy (the top answer), lenient accuracy (the top 5 answers), and MRR (Mean Reciprocal Rank) which takes into account the ranks of returned answers. Answers for the list type question are evaluated using precision, recall, and F-measure. Related Work ::: BioAsq Sharma et al. BIBREF3 describe a system with two stage process for factoid and list type question answering. Their system extracts relevant entities and then runs supervised classifier to rank the entities. Wiese et al. BIBREF4 propose neural network based model for Factoid and List-type question answering task. The model is based on Fast QA and predicts the answer span in the passage for a given question. The model is trained on SQuAD data set and fine tuned on the BioASQ data. Dimitriadis et al. BIBREF5 proposed two stage process for Factoid question answering task. Their system uses general purpose tools such as Metamap, BeCas to identify candidate sentences. These candidate sentences are represented in the form of features, and are then ranked by the binary classifier. Classifier is trained on candidate sentences extracted from relevant questions, snippets and correct answers from BioASQ challenge. For factoid question answering task highest ‘MRR’ achieved in the 6th edition of BioASQ competition is ‘0.4325’. Our system is a neural network model based on contextual word embeddings BIBREF1 and achieved a ‘MRR’ score ‘0.6103’ in one of the test batches for Factoid Question Answering task. Related Work ::: A minimum background on BERT BERT stands for "Bidirectional Encoder Representations from Transformers" BIBREF1 is a contextual word embedding model. Given a sentence as an input, contextual embedding for the words are returned. The BERT model was designed so it can be fine tuned for 11 different tasks BIBREF1, including question answering tasks. For a question answering task, question and paragraph (context) are given as an input. A BERT standard is that question text and paragraph text are separated by a separator [Sep]. BERT question-answering fine tuning involves adding softmax layer. Softmax layer takes contextual word embeddings from BERT as input and learns to identity answer span present in the paragraph (context). This process is represented in Figure FIGREF4. BERT was originally trained to perform tasks such as language model creation using masked words and next-sentence-prediction. In other words BERT weights are learned such that context is used in building the representation of the word, not just as a loss function to help learn a context-independent representation. For detailed understanding of BERT Architecture, please refer to the original BERT paper BIBREF1. Related Work ::: A minimum background on BERT ::: Comparison of Word Embeddings and Contextual Word Embeddings A ‘word embedding’ is a learned representation. It is represented in the form of vector where words that have the same meaning have a similar vector representation. Consider a word embedding model 'word2vec' BIBREF6 trained on a corpus. Word embeddings generated from the model are context independent that is, word embeddings are returned regardless of where the words appear in a sentence and regardless of e.g. the sentiment of the sentence. However, contextual word embedding models like BERT also takes context of the word into consideration. Related Work ::: Comparison of BERT and Bio-BERT ‘BERT’ and BioBERT are very similar in terms of architecture. Difference is that ‘BERT’ is pretrained on Wikipedia articles, whereas BioBERT version used in our experiments is pretrained on Wikipedia, PMC and PubMed articles. Therefore BioBERT model is expected to perform well with biomedical text, in terms of generating contextual word embeddings. BioBERT model used in our experiments is based on BERT-Base Architecture; BERT-Base has 12 transformer Layers where as BERT-Large has 24 transformer layers. Moreover contextual word embedding vector size is 768 for BERT-Base and more for BERT-large. According to BIBREF1 Bert-Large, fine-tuned on SQuAD 1.1 question answering data BIBREF7 can achieve F1 Score of 90.9 for Question Answering task where as BERT-Base Fine-tuned on the same SQuAD question answering BIBREF7 data could achieve F1 score of 88.5. One downside of the current version BioBERT is that word-piece vocabulary is the same as that of original BERT Model, as a result word-piece vocabulary does not include biomedical jargon. Lee et al. BIBREF0 created BioBERT, using the same pre-trained BERT released by Google, and hence in the word-piece vocabulary (vocab.txt), as a result biomedical jargon is not included in word-piece vocabulary. Modifying word-piece vocabulary (vocab.txt) at this stage would loose original compatibility with ‘BERT’, hence it is left unmodified. In our future work we would like to build pre-trained ‘BERT’ model from scratch. We would pretrain the model with biomedical corpus (PubMed, ‘PMC’) and Wikipedia. Doing so would give us scope to create word piece vocabulary to include biomedical jargon and there are chances of model performing better with biomedical jargon being included in the word piece vocabulary. We will consider this scenario in the future, or wait for the next version of BioBERT. Experiments: Factoid Question Answering Task For Factoid Question Answering task, we fine tuned BioBERT BIBREF0 with question answering data and added new features. Fig. FIGREF4 shows the architecture of BioBERT fine tuned for question answering tasks: Input to BioBERT is word tokenized embeddings for question and the paragraph (Context). As per the ‘BERT’ BIBREF1 standards, tokens ‘[CLS]’ and ‘[SEP]’ are appended to the tokenized input as illustrated in the figure. The resulting model has a softmax layer formed for predicting answer span indices in the given paragraph (Context). On test data, the fine tuned model generates $n$-best predictions for each question. For a question, $n$-best corresponds that $n$ answers are returned as possible answers in the decreasing order of confidence. Variable $n$ is configurable. In our paper, any further mentions of ‘answer returned by the model’ correspond to the top answer returned by the model. Experiments: Factoid Question Answering Task ::: Setup BioASQ provides the training data. This data is based on previous BioASQ competitions. Train data we have considered is aggregate of all train data sets till the 5th version of BioASQ competition. We cleaned the data, that is, question-answering data without answers are removed and left with a total count of ‘530’ question answers. The data is split into train and test data in the ratio of 94 to 6; that is, count of '495' for training and '35' for testing. The original data format is converted to the BERT/BioBERT format, where BioBERT expects ‘start_index’ of the actual answer. The ‘start_index corresponds to the index of the answer text present in the paragraph/ Context. For finding ‘start_index’ we used built-in python function find(). The function returns the lowest index of the actual answer present in the context(paragraph). If the answer is not found ‘-1’ is returned as the index. The efficient way of finding start_index is, if the paragraph (Context) has multiple instances of answer text, then ‘start_index’ of the answer should be that instance of answer text whose context actually matches with what’s been asked in the question. Example (Question, Answer and Paragraph from BIBREF8): Question: Which drug should be used as an antidote in benzodiazepine overdose? Answer: 'Flumazenil' Paragraph(context): "Flumazenil use in benzodiazepine overdose in the UK: a retrospective survey of NPIS data. OBJECTIVE: Benzodiazepine (BZD) overdose (OD) continues to cause significant morbidity and mortality in the UK. Flumazenil is an effective antidote but there is a risk of seizures, particularly in those who have co-ingested tricyclic antidepressants. A study was undertaken to examine the frequency of use, safety and efficacy of flumazenil in the management of BZD OD in the UK. METHODS: A 2-year retrospective cohort study was performed of all enquiries to the UK National Poisons Information Service involving BZD OD. RESULTS: Flumazenil was administered to 80 patients in 4504 BZD-related enquiries, 68 of whom did not have ventilatory failure or had recognised contraindications to flumazenil. Factors associated with flumazenil use were increased age, severe poisoning and ventilatory failure. Co-ingestion of tricyclic antidepressants and chronic obstructive pulmonary disease did not influence flumazenil administration. Seizure frequency in patients not treated with flumazenil was 0.3%". Actual answer is 'Flumazenil', but there are multiple instances of word 'Flu-mazenil'. Efficient way to identify the start-index for 'Flumazenil'(answer) is to find that particular instance of the word 'Flumazenil' which matches the context for the question. In the above example 'Flumazenil' highlighted in bold is the actual instance that matches question's context. Unfortunately, we could not identify readily available tools that can achieve this goal. In our future work, we look forward to handling these scenarios effectively. Note: The creators of 'SQuAD' BIBREF7 have handled the task of identifying answer's start_index effectively. But 'SQuAD' data set is much more general and does not include biomedical question answering data. Experiments: Factoid Question Answering Task ::: Training and error analysis During our training with the BioASQ data, learning rate is set to 3e-5, as mentioned in the BioBERT paper BIBREF0. We started training the model with 495 available train data and 35 test data by setting number of epochs to 50. After training with these hyper-parameters training accuracy(exact match) was 99.3%(overfitting) and testing accuracy is only 4%. In the next iteration we reduced the number of epochs to 25 then training accuracy is reduced to 98.5% and test accuracy moved to 5%. We further reduced number of epochs to 15, and the resulting training accuracy was 70% and test accuracy 15%. In the next iteration set number of epochs to 12 and achieved train accuracy of 57.7% and test accuracy 23.3%. Repeated the experiment with 11 epochs and found training accuracy to be 57.7% and test accuracy to be same 22%. In the next iteration we set number of epochs to '9' and found training accuracy of 48% and test accuracy of 15%. Hence optimum number of epochs is taken as 12 epochs. During our error analysis we found that on test data, model tends to return text in the beginning of the context(paragraph) as the answer. On analysing train data, we found that there are '120'(out of '495') question answering data instances having start_index:0, meaning 120( 25%) question answering data has first word(s) in the context(paragraph) as the answer. We removed 70% of those instances in order to make train data more balanced. In the new train data set we are left with '411' question answering data instances. This time we got the highest test accuracy of 26% at 11 epochs. We have submitted our results for BioASQ test batch-2, got strict accuracy of 32% and our system stood in 2nd place. Initially, hyper-parameter- 'batch size' is set to ‘400’. Later it is tuned to '32'. Although accuracy(exact answer match) remained at 26%, model generated concise and better answers at batch size ‘32’, that is wrong answers are close to the expected answer in good number of cases. Example.(from BIBREF8) Question: Which mutated gene causes Chediak Higashi Syndrome? Exact Answer: ‘lysosomal trafficking regulator gene’. The answer returned by a model trained at ‘400’ batch size is ‘Autosomal-recessive complicated spastic paraplegia with a novel lysosomal trafficking regulator’, and from the one trained at ‘32’ batch size is ‘lysosomal trafficking regulator’. In further experiments, we have fine tuned the BioBERT model with both ‘SQuAD’ dataset (version 2.0) and BioAsq train data. For training on ‘SQuAD’, hyper parameters- Learning rate and number of epochs are set to ‘3e-3’ and ‘3’ respectively as mentioned in the paper BIBREF1. Test accuracy of the model boosted to 44%. In one more experiment we trained model only on ‘SQuAD’ dataset, this time test accuracy of the model moved to 47%. The reason model did not perform up to the mark when trained with ‘SQuAD’ alongside BioASQ data could be that in formatted BioASQ data, start_index for the answer is not accurate, and affected the overall accuracy. Our Systems and Their Performance on Factoid Questions We have experimented with several systems and their variations, e.g. created by training with specific additional features (see next subsection). Here is their list and short descriptions. Unfortunately we did not pay attention to naming, and the systems evolved between test batches, so the overall picture can only be understood by looking at the details. When we started the experiments our objective was to see whether BioBERT and entailment-based techniques can provide value for in the context of biomedical question answering. The answer to both questions was a yes, qualified by many examples clearly showing the limitations of both methods. Therefore we tried to address some of these limitations using feature engineering with mixed results: some clear errors got corrected and new errors got introduced, without overall improvement but convincing us that in future experiments it might be worth trying feature engineering again especially if more training data were available. Overall we experimented with several approaches with the following aspects of the systems changing between batches, that is being absent or present: training on BioAsq data vs. training on SQuAD using the BioAsq snippets for context vs. using the documents from the provided URLs for context adding or not the LAT, i.e. lexical answer type, feature (see BIBREF9, BIBREF10 and an explanation in the subsection just below). For Yes/No questions (only) we experimented with the entailment methods. We will discuss the performance of these models below and in Section 6. But before we do that, let us discuss a feature engineering experiment which eventually produced mixed results, but where we feel it is potentially useful in future experiments. Our Systems and Their Performance on Factoid Questions ::: LAT Feature considered and its impact (slightly negative) During error analysis we found that for some cases, answer being returned by the model is far away from what it is being asked in the Question. Example: (from BIBREF8) Question: Hy's law measures failure of which organ? Actual Answer: ‘Liver’. The answer returned by one of our models was ‘alanine aminotransferase’, which is an enzyme. The model returns an enzyme, when the question asked for the organ name. To address this type of errors, we decided to try the concepts of ‘Lexical Answer Type’ (LAT) and Focus Word, which was used in IBM Watson, see BIBREF11 for overview; BIBREF10 for technical details, and BIBREF9 for details on question analysis. In an example given in the last source we read: POETS & POETRY: He was a bank clerk in the Yukon before he published "Songs of a Sourdough" in 1907. The focus is the part of the question that is a reference to the answer. In the example above, the focus is "he". LATs are terms in the question that indicate what type of entity is being asked for. The headword of the focus is generally a LAT, but questions often contain additional LATs, and in the Jeopardy! domain, categories are an additional source of LATs. (...) In the example, LATs are "he", "clerk", and "poet". For example in the question "Which plant does oleuropein originate from?" (BIBREF8). The LAT here is ‘plant’. For the BioAsq task we did not need to explicitly distinguish between the focus and the LAT concepts. In this example, the expectation is that answer returned by the model is a plant. Thus it is conceivable that the cosine distance between contextual embedding of word 'plant' in the question and contextual embedding for the answer present in the paragraph(context) is comparatively low. As a result model learns to adjust its weights during training phase and returns answers with low cosine distance with the LAT. We used Stanford CoreNLP BIBREF12 library to write rules for extracting lexical answer type present in the question, both 'parts of speech'(POS) and dependency parsing functionality was used. We incorporated the Lexical Answer Type into one of our systems, UNCC_QA1 in Batch 4. This system underperformed our system FACTOIDS by about 3% in the MRR measure, but corrected errors such as in the example above. Our Systems and Their Performance on Factoid Questions ::: LAT Feature considered and its impact (slightly negative) ::: Assumptions and rules for deriving lexical answer type. There are different question types: ‘Which’, ‘What’, ‘When’, ‘How’ etc. Each type of question is being handled differently and there are commonalities among the rules written for different question types. Question words are identified through parts of speech tags: 'WDT', 'WRB' ,'WP'. We assumed that LAT is a ‘Noun’ and follows the question word. Often it was also a subject (nsubj). This process is illustrated in Fig.FIGREF15. LAT computation was governed by a few simple rules, e.g. when a question has multiple words that are 'Subjects’ (and ‘Noun’), a word that is in proximity to the question word is considered as ‘LAT’. These rules are different for each "Wh" word. Namely, when the word immediately following the question word is a Noun, window size is set to ‘3’. The window size ‘3’ means we iterate through the next ‘3’ words to check if any of the word is both Noun and Subject, If so, such word is considered the ‘LAT’; else the word that is present very next to the question word is considered as the ‘LAT’. For questions with words ‘Which’ , ‘What’, ‘When’; a Noun immediately following the question word is very often the LAT, e.g. 'enzyme' in Which enzyme is targeted by Evolocumab?. When the word immediately following the question word is not a Noun, e.g. in What is the function of the protein Magt1? the window size is set to ‘5’, and we iterate through the next ‘5’ words (if present) and search for the word that is both Noun and Subject. If present, the word is considered as the ‘LAT’; else, the Noun in close proximity to the question word and following it is returned as the ‘LAT’. For questions with question words: ‘When’, ‘Who’, ‘Why’, the ’LAT’ is a question word itself. For the word ‘How', e.g. in How many selenoproteins are encoded in the human genome?, we look at the adjective and if we find one, we take it to be the LAT, otherwise the word 'How' is considered as the ‘LAT’. Perhaps because of using only very simple rules, the accuracy for ‘LAT’ derivation is 75%; that is, in the remaining 25% of the cases the LAT word is identified incorrectly. Worth noting is that the overall performance the system that used LATs was slightly inferior to the system without LATs, but the types of errors changed. The training used BioBERT with the LAT feature as part of the input string. The errors it introduces usually involve finding the wrong element of the correct type e.g. wrong enzyme when two similar enzymes are described in the text, or 'neuron' when asked about a type of cell with a certain function, when the answer calls for a different cell category, adipocytes, and both are mentioned in the text. We feel with more data and additional tuning or perhaps using an ensemble model, we might be able to keep the correct answers, and improve the results on the confusing examples like the one mentioned above. Therefore if we improve our ‘LAT’ derivation logic, or have larger datasets, then perhaps the neural network techniques they will yield better results. Our Systems and Their Performance on Factoid Questions ::: Impact of Training using BioAsq data (slightly negative) Training on BioAsq data in our entry in Batch 1 and Batch 2 under the name QA1 showed it might lead to overfitting. This happened both with (Batch 2) and without (Batch 1) hyperparameters tuning: abysmal 18% MRR in Batch 1, and slighly better one, 40% in Batch 2 (although in Batch 2 it was overall the second best result in MRR but 16% lower than the highest score). In Batch 3 (only), our UNCC_QA3 system was fine tuned on BioAsq and SQuAD 2.0 BIBREF7, and for data preprocessing Context paragraph is generated from relevant snippets provided in the test data. This system underperformed, by about 2% in MRR, our other entry UNCC_QA1, which was also an overall category winner for this batch. The latter was also trained on SQuAD, but not on BioAsq. We suspect that the reason could be the simplistic nature of the find() function described in Section 3.1. So, this could be an area where a better algorithm for finding the best occurrence of an entity could improve performance. Our Systems and Their Performance on Factoid Questions ::: Impact of Using Context from URLs (negative) In some experiments, for context in testing, we used documents for which URL pointers are provided in BioAsq. However, our system UNCC_QA3 underperformed our other system tested only on the provided snippets. In Batch 5 the underperformance was about 6% of MRR, compared to our best system UNCC_QA1, and by 9% to the top performer. Performance on Yes/No and List questions Our work focused on Factoid questions. But we also have done experiments on List-type and Yes/No questions. Performance on Yes/No and List questions ::: Entailment improves Yes/No accuracy We started by answering always YES (in batch 2 and 3) to get the baseline performance. For batch 4 we used entailment. Our algorithm was very simple: Given a question we iterate through the candidate sentences and try to find any candidate sentence is contradicting the question (with confidence over 50%), if so 'No' is returned as answer, else 'Yes' is returned. In batch 4 this strategy produced better than the BioAsq baseline performance, and compared to our other systems, the use of entailment increased the performance by about 13% (macro F1 score). We used 'AllenNlp' BIBREF13 entailment library to find entailment of the candidate sentences with question. Performance on Yes/No and List questions ::: For List-type the URLs have negative impact Overall, we followed the similar strategy that's been followed for Factoid Question Answering task. We started our experiment with batch 2, where we submitted 20 best answers (with context from snippets). Starting with batch 3, we performed post processing: once models generate answer predictions (n-best predictions), we do post-processing on the predicted answers. In test batch 4, our system (called FACTOIDS) achieved highest recall score of ‘0.7033’ but low precision of 0.1119, leaving open the question of how could we have better balanced the two measures. In the post-processing phase, we take the top ‘20’ (batch 3) and top 5 (batch 4 and 5), predicted answers, tokenize them using common separators: 'comma' , 'and', 'also', 'as well as'. Tokens with characters count more than ‘100’ are eliminated and rest of the tokens are added to the list of possible answers. BioASQ evaluation mechanism does not consider snippets with more than ‘100’ characters as a valid answer. Considering lengthy snippets in to the list of answers would reduce the mean precision score. As a final step, duplicate snippets in the answer pool are removed. For example, consider these top 3 answers predicted by the system (before post-processing): { "text": "dendritic cells", "probability": 0.7554540733426441, "start_logit": 8.466046333312988, "end_logit": 9.536355018615723 }, { "text": "neutrophils, macrophages and distinct subtypes of dendritic cells", "probability": 0.13806867348304214, "start_logit": 6.766478538513184, "end_logit": 9.536355018615723 }, { "text": "macrophages and distinct subtypes of dendritic", "probability": 0.013973475271178242, "start_logit": 6.766478538513184, "end_logit": 7.24576473236084 }, After execution of post-processing heuristics, the list of answers returned is as follows: ["dendritic cells"], ["neutrophils"], ["macrophages"], ["distinct subtypes of dendritic cells"] Summary of our results The tables below summarize all our results. They show that the performance of our systems was mixed. The simple architectures and algorithm we used worked very well only in Batch 3. However, we feel we can built a better system based on this experience. In particular we observed both the value of contextual embeddings and of feature engineering (LAT), however we failed to combine them properly. Summary of our results ::: Factoid questions ::: Systems used in Batch 5 experiments System description for ‘UNCC_QA1’: The system was finetuned on the SQuAD 2.0. For data preprocessing Context / paragraph was generated from relevant snippets provided in the test data. System description for ‘QA1’ : ‘LAT’ feature was added and finetuned with SQuAD 2.0. For data preprocessing Context / paragraph was generated from relevant snippets provided in the test data. System description for ‘UNCC_QA3’ : Fine tuning process is same as it is done for the system ‘UNCC_QA1’ in test batch-5. Difference is during data preprocessing, Context/paragraph is generated from the relevant documents for which URLS are included in the test data. Summary of our results ::: List Questions For List-type questions, although post processing helped in the later batches, we never managed to obtain competitive precision, although our recall was good. Summary of our results ::: Yes/No questions The only thing worth remembering from our performance is that using entailment can have a measurable impact (at least with respect to a weak baseline). The results (weak) are in Table 3. Discussion, Future Experiments, and Conclusions ::: Summary: In contrast to 2018, when we submitted BIBREF2 to BioASQ a system based on extractive summarization (and scored very high in the ideal answer category), this year we mainly targeted factoid question answering task and focused on experimenting with BioBERT. After these experiments we see the promise of BioBERT in QA tasks, but we also see its limitations. The latter we tried to address with mixed results using feature engineering. Overall these experiments allowed us to secure a best and a second best score in different test batches. Along with Factoid-type question, we also tried ‘Yes/No’ and ‘List’-type questions, and did reasonably well with our very simple approach. For Yes/No the moral worth remembering is that reasoning has a potential to influence results, as evidenced by our adding the AllenNLP entailment BIBREF13 system increased its performance. All our data and software is available at Github, in the previously referenced URL (end of Section 2). Discussion, Future Experiments, and Conclusions ::: Future experiments In the current model, we have a shallow neural network with a softmax layer for predicting answer span. Shallow networks however are not good at generalizations. In our future experiments we would like to create dense question answering neural network with a softmax layer for predicting answer span. The main idea is to get contextual word embedding for the words present in the question and paragraph (Context) and feed the contextual word embeddings retrieved from the last layer of BioBERT to the dense question answering network. The mentioned dense layered question answering neural network need to be tuned for finding right hyper parameters. An example of such architecture is shown in Fig.FIGREF30. In one more experiment, we would like to add a better version of ‘LAT’ contextual word embedding as a feature, along with the actual contextual word embeddings for question text, and Context and feed them as input to the dense question answering neural network. By this experiment, we would like to find if ‘LAT’ feature is improving overall answer prediction accuracy. Adding ‘LAT’ feature this way instead of feeding this word piece embedding directly to the BioBERT (as we did in our above experiments) would not downgrade the quality of contextual word embeddings generated form ‘BioBERT'. Quality contextual word embeddings would lead to efficient transfer learning and chances are that it would improve the model's answer prediction accuracy. We also see potential for incorporating domain specific inference into the task e.g. using the MedNLI dataset BIBREF14. For all types of experiments it might be worth exploring clinical BERT embeddings BIBREF15, explicitly incorporating domain knowledge (e.g. BIBREF16) and possibly deeper discourse representations (e.g. BIBREF17). APPENDIX In this appendix we provide additional details about the implementations. APPENDIX ::: Systems and their descriptions: We used several variants of our systems when experimenting with the BioASQ problems. In retrospect, it would be much easier to understand the changes if we adopted some mnemonic conventions in naming the systems. So, we apologize for the names that do not reflect the modifications, and necessitate this list. APPENDIX ::: Systems and their descriptions: ::: Factoid Type Question Answering: We preprocessed the test data to convert test data to BioBERT format, We generated Context/paragraph by either aggregating relevant snippets provided or by aggregating documents for which URLS are provided in the BioASQ test data. APPENDIX ::: Systems and their descriptions: ::: System description for QA1: We generated Context/paragraph by aggregating relevant snippets available in the test data and mapped it against the question text and question id. We ignored the content present in the documents (document URLS were provided in the original test data). The model is finetuned with BioASQ data. data preprocessing is done in the same way as it is done for test batch-1. Model fine tuned on BioASQ data. ‘LAT’/ Focus word feature added and fine tuned with SQuAD 2.0 [reference]. For data preprocessing Context / paragraph is generated from relevant snippets provided in the test data. APPENDIX ::: Systems and their descriptions: ::: System description for UNCC_QA_1: System is finetuned on the SQuAD 2.0 [reference]. For data preprocessing Context / paragraph is generated from relevant snippets provided in the test data. ‘LAT’/ Focus word feature added and fine tuned with SQuAD 2.0 [reference]. For data preprocessing Context / paragraph is generated from relevant snippets provided in the test data. The System is finetuned on the SQuAD 2.0. For data preprocessing Context / paragraph is generated from relevant snippets provided in the test data. APPENDIX ::: Systems and their descriptions: ::: System description for UNCC_QA3: System is finetuned on the SQuAD 2.0 [reference] and BioASQ dataset[].For data preprocessing Context / paragraph is generated from relevant snippets provided in the test data. Fine tuning process is same as it is done for the system ‘UNCC_QA_1’ in test batch-5. Difference is during data preprocessing, Context/paragraph is generated form from the relevant documents for which URLS are included in the test data. APPENDIX ::: Systems and their descriptions: ::: System description for UNCC_QA2: Fine tuning process is same as for ‘UNCC_QA_1 ’. Difference is Context/paragraph is generated form from the relevant documents for which URLS are included in the test data. System ‘UNCC_QA_1’ got the highest ‘MRR’ score in the 3rd test batch set. APPENDIX ::: Systems and their descriptions: ::: System description for FACTOIDS: The System is finetuned on the SQuAD 2.0. For data preprocessing Context / paragraph is generated from relevant snippets provided in the test data. APPENDIX ::: Systems and their descriptions: ::: List Type Questions: We attempted List type questions starting from test batch ‘2’. Used similar approach that's been followed for Factoid Question answering task. For all the test batch sets, in the data pre processing phase Context/ paragraph is generated either by aggregating relevant snippets or by aggregating documents(URLS) provided in the BioASQ test data. For test batch-2, model (System: QA1) is finetuned on BioASQ data and submitted top ‘20’ answers predicted by the model as the list of answers. system ‘QA1’ achieved low F-Measure score:‘0.0786’ in the second test batch. In the further test batches for List type questions, we finetuned the model on Squad data set [reference], implemented post processing techniques (refer section 5.2) and achieved a better F-measure score: ‘0.2862’ in the final test batch set. In test batch-3 (Systems : ‘QA1’/’’UNCC_QA_1’/’UNCC_QA3’/’UNCC_QA2’) top 20 answers returned by the model is sent for post processing and in test batch 4 and 5 only top 5 answers are sent for post processing. System UNCC_QA2(in batch 3) for List type question answering, Context is generated from documents for which URLS are provided in the BioASQ test data. for the rest of the systems (in test batch-3) for List Type question answering task snippets present in the BioaSQ test data are used to generate context. In test batch-4 (System : ‘FACTOIDS’/’UNCC_QA_1’/’UNCC_QA3’’) top 5 answers returned by the model is sent for post processing. In case of system ‘FACTOIDS’ snippets in the test data were used to generate context. for systems ’UNCC_QA_1’ and ’UNCC_QA3’ context is generated from the documents for which URLS are provided in the BioASQ test data. In test batch-5 ( Systems: ‘QA1’/’UNCC_QA_1’/’UNCC_QA3’/’UNCC_QA2’ ) our approach is the same as that of test batch-4 where top 5 answers returned by the model is sent for post processing. for all the systems (in test batch-5) context is generated from the snippets provided in the BioASQ test data. APPENDIX ::: Systems and their descriptions: ::: Yes/No Type Questions: For the first 3 test batches, We have submitted answer ‘Yes’ to all the questions. Later, we employed ‘Sentence Entailment’ techniques(refer section 6.0) for the fourth and fifth test batch sets. Our Systems with ‘Sentence Entailment’ approach (for ‘Yes’/ ‘No’ question answering): ‘UNCC_QA_1’(test batch-4), UNCC_QA3(test batch-5). APPENDIX ::: Additional details for Yes/No Type Questions We used Textual Entailment in Batch 4 and 5 for ‘Yes’/‘No’ question type. The algorithm was very simple: Given a question we iterate through the candidate sentences, and look for any candidate sentences contradicting the question. If we find one 'No' is returned as answer, else 'Yes' is returned. (The confidence for contradiction was set at 50%) We used AllenNLP BIBREF13 entailment library to find entailment of the candidate sentences with question. Flow Chart for Yes/No Question answer processing is shown in Fig.FIGREF51 APPENDIX ::: Assumptions, rules and logic flow for deriving Lexical Answer Types from questions There are different question types, and we distinguished them based on the question words: ‘Which’, ‘What’, ‘When’, ‘How’ etc. Each type of question is being handled differently and there are commonalities among the rules written for different question types. How are question words identified? question words have parts of speech(POS): 'WDT', 'WRB', 'WP'. Assumptions: 1) Lexical answer type (‘LAT’) or focus word is of type Noun and follows the question word. 2) The LAT word is a Subject. (This clearly not always true, but we used a very simple method). Note: ‘StanfordNLP’ dependency parsing tag for identifying subject is 'nsubj' or 'nsubjpass'. 3) When a question has multiple words that are of type Subject (and Noun), a word that is in proximity to the question word is considered as ‘LAT’. 4) For questions with question words: ‘When’, ‘Who’, ‘Why’, the ’LAT’ is a question word itself that is, ‘When’, ‘Who’, ‘Why’ respectively. Rules and logic flow to traverse a question: The three cases below describe the logic flow of finding LATs. The figures show the grammatical structures used for this purpose. APPENDIX ::: Assumptions, rules and logic flow for deriving Lexical Answer Types from questions ::: Case-1: Question with question word ‘How’. For questions with question word 'How', the adjective that follows the question word is considered as ‘LAT’ (need not follow immediately). If an adjective is absent, word 'How' is considered as ‘LAT’. When there are multiple words that are adjectives, a word in close proximity to the question word and follows it is returned as ‘LAT’. Note: The part of speech tag to identify adjectives is 'JJ'. For Other possible question words like ‘whose’. ‘LAT’/Focus word is question words itself. Example Question: How many selenoproteins are encoded in the human genome? APPENDIX ::: Assumptions, rules and logic flow for deriving Lexical Answer Types from questions ::: Case-2: Questions with question words ‘Which’ , ‘What’ and all other possible question words; a 'Noun' immediately following the question word. Example Question: Which enzyme is targeted by Evolocumab? Here, Focus word/LAT is ‘enzyme’ which is both Noun and Subject and immediately follows the question word. When the word immediately following the question word is a noun, the window size is set to ‘3’. This size ‘3’ means that we iterate through the next ‘3’ words (if present) to check if any of the word is both 'Noun' and 'Subject', If so, the word is considered as ‘LAT’/Focus Word. Else the word that is present very next to the question word is considered as ‘LAT’. APPENDIX ::: Assumptions, rules and logic flow for deriving Lexical Answer Types from questions ::: Case-3: Questions with question words ‘Which’ , ‘What’ and all other possible question words; word immediately following the question word is not a 'Noun'. Example Question: What is the function of the protein Magt1? Here, Focus word/LAT is ‘function ’ which is both Noun and Subject and does not immediately follow the question word. When the very next word following the question word is not a Noun, window size is set to ‘5’. Window size ‘5’ corresponds that we iterate through the next ‘5’ words (if present) and search for the word that is both Noun and Subject. If present, the word is considered as ‘LAT’. Else, the 'Noun' close proximity to the question word and follows it is returned as ‘LAT’. Ad we mentioned earlier, the accuracy for ‘LAT’ derivation is 75 percent. But clearly the simple logic described above can be improved, as shown in BIBREF9, BIBREF10. Whether this in turn produces improvements in this particular task is an open question. APPENDIX ::: Proposing Future Experiments In the current model, we have a shallow neural network with a softmax layer for predicting answer span. Shallow networks however are not good at generalizations. In our future experiments we would like to create dense question answering neural network with a softmax layer for predicting answer span. The main idea is to get contextual word embedding for the words present in the question and paragraph (Context) and feed the contextual word embeddings retrieved from the last layer of BioBERT to the dense question answering network. The mentioned dense layered question answering Neural network need to be tuned for finding right hyper parameters. An example of such architecture is shown in Fig.FIGREF30. In another experiment we would like to only feed contextual word embeddings for Focus word/ ‘LAT’, paragraph/ Context as input to the question answering neural network. In this experiment we would neglect all embeddings for the question text except that of Focus word/ ‘LAT’. Our assumption and idea for considering focus word and neglecting remaining words in the question is that during training phase it would make more precise for the model to identify the focus of the question and map answers against the question’s focus. To validate our assumption, we would like to take sample question answering data and find the cosine distance between contextual embedding of Focus word and that of the actual answer and verify if the cosine distance is comparatively low in most of the cases. In one more experiment, we would like to add a better version of ‘LAT’ contextual word embedding as a feature, along with the actual contextual word embeddings for question text, and Context and feed them as input to the dense question answering neural network. By this experiment, we would like to find if ‘LAT’ feature is improving overall answer prediction accuracy. Adding ‘LAT’ feature this way instead of feeding Focus word’s word piece embedding directly (as we did in our above experiments) to the BioBERT would not downgrade the quality of contextual word embeddings generated form ‘BioBERT'. Quality contextual word embeddings would lead to efficient transfer learning and chances are that it would improve the model's answer prediction accuracy.
What was their highest MRR score?
9ec1f88ceec84a10dc070ba70e90a792fba8ce71
What Ġwas Ġtheir Ġhighest ĠMR R Ġscore ?
0 . 5 115
<s> What Ġwas Ġtheir Ġhighest ĠMR R Ġscore ? </s> Introduction </s> Bio AS Q Ġis Ġa Ġbiomedical Ġdocument Ġclassification , Ġdocument Ġretrieval , Ġand Ġquestion Ġanswering Ġcompetition , Ġcurrently Ġin Ġits Ġseventh Ġyear . ĠWe Ġprovide Ġan Ġoverview Ġof Ġour Ġsubmissions Ġto Ġsemantic Ġquestion Ġanswering Ġtask Ġ( 7 b , ĠPhase ĠB ) Ġof ĠBio AS Q Ġ7 Ġ( except Ġfor Ġ' ide al Ġanswer ' Ġtest , Ġin Ġwhich Ġwe Ġdid Ġnot Ġparticipate Ġthis Ġyear ). ĠIn Ġthis Ġtask Ġsystems Ġare Ġprovided Ġwith Ġbiomedical Ġquestions Ġand Ġare Ġrequired Ġto Ġsubmit Ġideal Ġand Ġexact Ġanswers Ġto Ġthose Ġquestions . ĠWe Ġhave Ġused ĠBio BER T ĠB IB REF 0 Ġbased Ġsystem Ġ, Ġsee Ġalso ĠBid irection al ĠEnc oder ĠRepresent ations Ġfrom ĠTransformers ( BER T ) ĠB IB REF 1 , Ġand Ġwe Ġfine Ġtuned Ġit Ġfor Ġthe Ġbiomedical Ġquestion Ġanswering Ġtask . ĠOur Ġsystem Ġscored Ġnear Ġthe Ġtop Ġfor Ġfact oid Ġquestions Ġfor Ġall Ġthe Ġbatches Ġof Ġthe Ġchallenge . ĠMore Ġspec if ially , Ġin Ġthe Ġthird Ġtest Ġbatch Ġset , Ġour Ġsystem Ġachieved Ġhighest ĠâĢ ĺ MR R âĢ Ļ Ġscore Ġfor ĠFact oid ĠQuestion ĠAn sw ering Ġtask . ĠAlso , Ġfor ĠList - type Ġquestion Ġanswering Ġtask Ġour Ġsystem Ġachieved Ġhighest Ġrecall Ġscore Ġin Ġthe Ġfourth Ġtest Ġbatch Ġset . ĠAlong Ġwith Ġour Ġdetailed Ġapproach , Ġwe Ġpresent Ġthe Ġresults Ġfor Ġour Ġsubmissions Ġand Ġalso Ġhighlight Ġidentified Ġdowns ides Ġfor Ġour Ġcurrent Ġapproach Ġand Ġways Ġto Ġimprove Ġthem Ġin Ġour Ġfuture Ġexperiments . ĠIn Ġlast Ġtest Ġbatch Ġresults Ġwe Ġplaced Ġ4 th Ġfor ĠList - type Ġquestions Ġand Ġ3 rd Ġfor ĠFact oid - type Ġquestions .) </s> The ĠQ A Ġtask Ġis Ġorganized Ġin Ġtwo Ġphases . ĠPhase ĠA Ġdeals Ġwith Ġretrieval Ġof Ġthe Ġrelevant Ġdocument , Ġsnippets , Ġconcepts , Ġand ĠR DF Ġtri ples , Ġand Ġphase ĠB Ġdeals Ġwith Ġexact Ġand Ġideal Ġanswer Ġgenerations Ġ( which Ġis Ġa Ġparagraph Ġsize Ġsummary Ġof Ġsnippets ). ĠEx act Ġanswer Ġgeneration Ġis Ġrequired Ġfor Ġfact oid , Ġlist , Ġand Ġyes / no Ġtype Ġquestion . </s> Bio AS Q Ġorganizers Ġprovide Ġthe Ġtraining Ġand Ġtesting Ġdata . ĠThe Ġtraining Ġdata Ġconsists Ġof Ġquestions , Ġgold Ġstandard Ġdocuments , Ġsnippets , Ġconcepts , Ġand Ġideal Ġanswers Ġ( which Ġwe Ġdid Ġnot Ġuse Ġin Ġthis Ġpaper , Ġbut Ġwe Ġused Ġlast Ġyear ĠB IB REF 2 ). ĠThe Ġtest Ġdata Ġis Ġsplit Ġbetween Ġphases ĠA Ġand ĠB . ĠThe ĠPhase ĠA Ġdataset Ġconsists Ġof Ġthe Ġquestions , Ġunique Ġ ids , Ġquestion Ġtypes . ĠThe ĠPhase ĠB Ġdataset Ġconsists Ġof Ġthe Ġquestions , Ġgolden Ġstandard Ġdocuments , Ġsnippets , Ġunique Ġ ids Ġand Ġquestion Ġtypes . ĠEx act Ġanswers Ġfor Ġfact oid Ġtype Ġquestions Ġare Ġevaluated Ġusing Ġstrict Ġaccuracy Ġ( the Ġtop Ġanswer ), Ġlen ient Ġaccuracy Ġ( the Ġtop Ġ5 Ġanswers ), Ġand ĠMR R Ġ( Me an ĠRec ip rocal ĠRank ) Ġwhich Ġtakes Ġinto Ġaccount Ġthe Ġranks Ġof Ġreturned Ġanswers . ĠAnswers Ġfor Ġthe Ġlist Ġtype Ġquestion Ġare Ġevaluated Ġusing Ġprecision , Ġrecall , Ġand ĠF - me asure . </s> Related ĠWork Ġ: :: ĠBio As q </s> Sh arma Ġet Ġal . ĠB IB REF 3 Ġdescribe Ġa Ġsystem Ġwith Ġtwo Ġstage Ġprocess Ġfor Ġfact oid Ġand Ġlist Ġtype Ġquestion Ġanswering . ĠTheir Ġsystem Ġextracts Ġrelevant Ġentities Ġand Ġthen Ġruns Ġsupervised Ġclass ifier Ġto Ġrank Ġthe Ġentities . ĠW ies e Ġet Ġal . ĠB IB REF 4 Ġpropose Ġneural Ġnetwork Ġbased Ġmodel Ġfor ĠFact oid Ġand ĠList - type Ġquestion Ġanswering Ġtask . ĠThe Ġmodel Ġis Ġbased Ġon ĠFast ĠQ A Ġand Ġpredicts Ġthe Ġanswer Ġspan Ġin Ġthe Ġpassage Ġfor Ġa Ġgiven Ġquestion . ĠThe Ġmodel Ġis Ġtrained Ġon ĠS Qu AD Ġdata Ġset Ġand Ġfine Ġtuned Ġon Ġthe ĠBio AS Q Ġdata . ĠDim it riad is Ġet Ġal . ĠB IB REF 5 Ġproposed Ġtwo Ġstage Ġprocess Ġfor ĠFact oid Ġquestion Ġanswering Ġtask . ĠTheir Ġsystem Ġuses Ġgeneral Ġpurpose Ġtools Ġsuch Ġas ĠMet am ap , ĠBe Cas Ġto Ġidentify Ġcandidate Ġsentences . ĠThese Ġcandidate Ġsentences Ġare Ġrepresented Ġin Ġthe Ġform Ġof Ġfeatures , Ġand Ġare Ġthen Ġranked Ġby Ġthe Ġbinary Ġclass ifier . ĠClass ifier Ġis Ġtrained Ġon Ġcandidate Ġsentences Ġextracted Ġfrom Ġrelevant Ġquestions , Ġsnippets Ġand Ġcorrect Ġanswers Ġfrom ĠBio AS Q Ġchallenge . ĠFor Ġfact oid Ġquestion Ġanswering Ġtask Ġhighest ĠâĢ ĺ MR R âĢ Ļ Ġachieved Ġin Ġthe Ġ6 th Ġedition Ġof ĠBio AS Q Ġcompetition Ġis ĠâĢ ĺ 0 . 43 25 âĢ Ļ . ĠOur Ġsystem Ġis Ġa Ġneural Ġnetwork Ġmodel Ġbased Ġon Ġcontextual Ġword Ġembed d ings ĠB IB REF 1 Ġand Ġachieved Ġa ĠâĢ ĺ MR R âĢ Ļ Ġscore ĠâĢ ĺ 0 . 6 103 âĢ Ļ Ġin Ġone Ġof Ġthe Ġtest Ġbatches Ġfor ĠFact oid ĠQuestion ĠAn sw ering Ġtask . </s> Related ĠWork Ġ: :: ĠA Ġminimum Ġbackground Ġon ĠB ERT </s> BER T Ġstands Ġfor Ġ" B id irection al ĠEnc oder ĠRepresent ations Ġfrom ĠTransformers " ĠB IB REF 1 Ġis Ġa Ġcontextual Ġword Ġembed ding Ġmodel . ĠGiven Ġa Ġsentence Ġas Ġan Ġinput , Ġcontextual Ġembed ding Ġfor Ġthe Ġwords Ġare Ġreturned . ĠThe ĠB ERT Ġmodel Ġwas Ġdesigned Ġso Ġit Ġcan Ġbe Ġfine Ġtuned Ġfor Ġ11 Ġdifferent Ġtasks ĠB IB REF 1 , Ġincluding Ġquestion Ġanswering Ġtasks . ĠFor Ġa Ġquestion Ġanswering Ġtask , Ġquestion Ġand Ġparagraph Ġ( context ) Ġare Ġgiven Ġas Ġan Ġinput . ĠA ĠB ERT Ġstandard Ġis Ġthat Ġquestion Ġtext Ġand Ġparagraph Ġtext Ġare Ġseparated Ġby Ġa Ġsepar ator Ġ[ Sep ]. ĠB ERT Ġquestion - ans w ering Ġfine Ġtuning Ġinvolves Ġadding Ġsoft max Ġlayer . ĠSoft max Ġlayer Ġtakes Ġcontextual Ġword Ġembed d ings Ġfrom ĠB ERT Ġas Ġinput Ġand Ġlearns Ġto Ġidentity Ġanswer Ġspan Ġpresent Ġin Ġthe Ġparagraph Ġ( context ). ĠThis Ġprocess Ġis Ġrepresented Ġin ĠFigure ĠFIG REF 4 . </s> BER T Ġwas Ġoriginally Ġtrained Ġto Ġperform Ġtasks Ġsuch Ġas Ġlanguage Ġmodel Ġcreation Ġusing Ġmasked Ġwords Ġand Ġnext - sent ence - pred iction . ĠIn Ġother Ġwords ĠB ERT Ġweights Ġare Ġlearned Ġsuch Ġthat Ġcontext Ġis Ġused Ġin Ġbuilding Ġthe Ġrepresentation Ġof Ġthe Ġword , Ġnot Ġjust Ġas Ġa Ġloss Ġfunction Ġto Ġhelp Ġlearn Ġa Ġcontext - independent Ġrepresentation . ĠFor Ġdetailed Ġunderstanding Ġof ĠB ERT ĠArchitecture , Ġplease Ġrefer Ġto Ġthe Ġoriginal ĠB ERT Ġpaper ĠB IB REF 1 . </s> Related ĠWork Ġ: :: ĠA Ġminimum Ġbackground Ġon ĠB ERT Ġ: :: ĠComparison Ġof ĠWord ĠEmb edd ings Ġand ĠContext ual ĠWord ĠEmb edd ings </s> A ĠâĢ ĺ word Ġembed ding âĢ Ļ Ġis Ġa Ġlearned Ġrepresentation . ĠIt Ġis Ġrepresented Ġin Ġthe Ġform Ġof Ġvector Ġwhere Ġwords Ġthat Ġhave Ġthe Ġsame Ġmeaning Ġhave Ġa Ġsimilar Ġvector Ġrepresentation . ĠConsider Ġa Ġword Ġembed ding Ġmodel Ġ' word 2 vec ' ĠB IB REF 6 Ġtrained Ġon Ġa Ġcorpus . ĠWord Ġembed d ings Ġgenerated Ġfrom Ġthe Ġmodel Ġare Ġcontext Ġindependent Ġthat Ġis , Ġword Ġembed d ings Ġare Ġreturned Ġregardless Ġof Ġwhere Ġthe Ġwords Ġappear Ġin Ġa Ġsentence Ġand Ġregardless Ġof Ġe . g . Ġthe Ġsentiment Ġof Ġthe Ġsentence . ĠHowever , Ġcontextual Ġword Ġembed ding Ġmodels Ġlike ĠB ERT Ġalso Ġtakes Ġcontext Ġof Ġthe Ġword Ġinto Ġconsideration . </s> Related ĠWork Ġ: :: ĠComparison Ġof ĠB ERT Ġand ĠBio - BER T </s> âĢ ĺ BER T âĢ Ļ Ġand ĠBio BER T Ġare Ġvery Ġsimilar Ġin Ġterms Ġof Ġarchitecture . ĠDifference Ġis Ġthat ĠâĢ ĺ BER T âĢ Ļ Ġis Ġpret rained Ġon ĠWikipedia Ġarticles , Ġwhereas ĠBio BER T Ġversion Ġused Ġin Ġour Ġexperiments Ġis Ġpret rained Ġon ĠWikipedia , ĠPM C Ġand ĠPubMed Ġarticles . ĠTherefore ĠBio BER T Ġmodel Ġis Ġexpected Ġto Ġperform Ġwell Ġwith Ġbiomedical Ġtext , Ġin Ġterms Ġof Ġgenerating Ġcontextual Ġword Ġembed d ings . </s> Bio BER T Ġmodel Ġused Ġin Ġour Ġexperiments Ġis Ġbased Ġon ĠB ERT - Base ĠArchitecture ; ĠB ERT - Base Ġhas Ġ12 Ġtransformer ĠL ayers Ġwhere Ġas ĠB ERT - Large Ġhas Ġ24 Ġtransformer Ġlayers . ĠMoreover Ġcontextual Ġword Ġembed ding Ġvector Ġsize Ġis Ġ768 Ġfor ĠB ERT - Base Ġand Ġmore Ġfor ĠB ERT - large . ĠAccording Ġto ĠB IB REF 1 ĠBert - Large , Ġfine - tun ed Ġon ĠS Qu AD Ġ1 . 1 Ġquestion Ġanswering Ġdata ĠB IB REF 7 Ġcan Ġachieve ĠF 1 ĠScore Ġof Ġ90 . 9 Ġfor ĠQuestion ĠAn sw ering Ġtask Ġwhere Ġas ĠB ERT - Base ĠFine - tun ed Ġon Ġthe Ġsame ĠS Qu AD Ġquestion Ġanswering ĠB IB REF 7 Ġdata Ġcould Ġachieve ĠF 1 Ġscore Ġof Ġ88 . 5 . ĠOne Ġdownside Ġof Ġthe Ġcurrent Ġversion ĠBio BER T Ġis Ġthat Ġword - piece Ġvocabulary Ġis Ġthe Ġsame Ġas Ġthat Ġof Ġoriginal ĠB ERT ĠModel , Ġas Ġa Ġresult Ġword - piece Ġvocabulary Ġdoes Ġnot Ġinclude Ġbiomedical Ġjargon . ĠLee Ġet Ġal . ĠB IB REF 0 Ġcreated ĠBio BER T , Ġusing Ġthe Ġsame Ġpre - trained ĠB ERT Ġreleased Ġby ĠGoogle , Ġand Ġhence Ġin Ġthe Ġword - piece Ġvocabulary Ġ( voc ab . txt ), Ġas Ġa Ġresult Ġbiomedical Ġjargon Ġis Ġnot Ġincluded Ġin Ġword - piece Ġvocabulary . ĠMod ifying Ġword - piece Ġvocabulary Ġ( voc ab . txt ) Ġat Ġthis Ġstage Ġwould Ġloose Ġoriginal Ġcompatibility Ġwith ĠâĢ ĺ BER T âĢ Ļ , Ġhence Ġit Ġis Ġleft Ġun modified . </s> In Ġour Ġfuture Ġwork Ġwe Ġwould Ġlike Ġto Ġbuild Ġpre - trained ĠâĢ ĺ BER T âĢ Ļ Ġmodel Ġfrom Ġscratch . ĠWe Ġwould Ġpret rain Ġthe Ġmodel Ġwith Ġbiomedical Ġcorpus Ġ( Pub Med , ĠâĢ ĺ PM C âĢ Ļ ) Ġand ĠWikipedia . ĠDoing Ġso Ġwould Ġgive Ġus Ġscope Ġto Ġcreate Ġword Ġpiece Ġvocabulary Ġto Ġinclude Ġbiomedical Ġjargon Ġand Ġthere Ġare Ġchances Ġof Ġmodel Ġperforming Ġbetter Ġwith Ġbiomedical Ġjargon Ġbeing Ġincluded Ġin Ġthe Ġword Ġpiece Ġvocabulary . ĠWe Ġwill Ġconsider Ġthis Ġscenario Ġin Ġthe Ġfuture , Ġor Ġwait Ġfor Ġthe Ġnext Ġversion Ġof ĠBio BER T . </s> Exper iments : ĠFact oid ĠQuestion ĠAn sw ering ĠTask </s> For ĠFact oid ĠQuestion ĠAn sw ering Ġtask , Ġwe Ġfine Ġtuned ĠBio BER T ĠB IB REF 0 Ġwith Ġquestion Ġanswering Ġdata Ġand Ġadded Ġnew Ġfeatures . ĠFig . ĠFIG REF 4 Ġshows Ġthe Ġarchitecture Ġof ĠBio BER T Ġfine Ġtuned Ġfor Ġquestion Ġanswering Ġtasks : ĠInput Ġto ĠBio BER T Ġis Ġword Ġtoken ized Ġembed d ings Ġfor Ġquestion Ġand Ġthe Ġparagraph Ġ( Context ). ĠAs Ġper Ġthe ĠâĢ ĺ BER T âĢ Ļ ĠB IB REF 1 Ġstandards , Ġtokens ĠâĢ ĺ [ CL S ] âĢ Ļ Ġand ĠâĢ ĺ [ SE P ] âĢ Ļ Ġare Ġapp ended Ġto Ġthe Ġtoken ized Ġinput Ġas Ġillustrated Ġin Ġthe Ġfigure . ĠThe Ġresulting Ġmodel Ġhas Ġa Ġsoft max Ġlayer Ġformed Ġfor Ġpredicting Ġanswer Ġspan Ġindices Ġin Ġthe Ġgiven Ġparagraph Ġ( Context ). ĠOn Ġtest Ġdata , Ġthe Ġfine Ġtuned Ġmodel Ġgenerates Ġ$ n $ - best Ġpredictions Ġfor Ġeach Ġquestion . ĠFor Ġa Ġquestion , Ġ$ n $ - best Ġcorresponds Ġthat Ġ$ n $ Ġanswers Ġare Ġreturned Ġas Ġpossible Ġanswers Ġin Ġthe Ġdecreasing Ġorder Ġof Ġconfidence . ĠVariable Ġ$ n $ Ġis Ġconfig urable . ĠIn Ġour Ġpaper , Ġany Ġfurther Ġmentions Ġof ĠâĢ ĺ answer Ġreturned Ġby Ġthe Ġmodel âĢ Ļ Ġcorrespond Ġto Ġthe Ġtop Ġanswer Ġreturned Ġby Ġthe Ġmodel . </s> Exper iments : ĠFact oid ĠQuestion ĠAn sw ering ĠTask Ġ: :: ĠSetup </s> Bio AS Q Ġprovides Ġthe Ġtraining Ġdata . ĠThis Ġdata Ġis Ġbased Ġon Ġprevious ĠBio AS Q Ġcompetitions . ĠTrain Ġdata Ġwe Ġhave Ġconsidered Ġis Ġaggregate Ġof Ġall Ġtrain Ġdata Ġsets Ġtill Ġthe Ġ5 th Ġversion Ġof ĠBio AS Q Ġcompetition . ĠWe Ġcleaned Ġthe Ġdata , Ġthat Ġis , Ġquestion - ans w ering Ġdata Ġwithout Ġanswers Ġare Ġremoved Ġand Ġleft Ġwith Ġa Ġtotal Ġcount Ġof ĠâĢ ĺ 530 âĢ Ļ Ġquestion Ġanswers . ĠThe Ġdata Ġis Ġsplit Ġinto Ġtrain Ġand Ġtest Ġdata Ġin Ġthe Ġratio Ġof Ġ94 Ġto Ġ6 ; Ġthat Ġis , Ġcount Ġof Ġ' 495 ' Ġfor Ġtraining Ġand Ġ' 35 ' Ġfor Ġtesting . </s> The Ġoriginal Ġdata Ġformat Ġis Ġconverted Ġto Ġthe ĠB ERT / Bio BER T Ġformat , Ġwhere ĠBio BER T Ġexpects ĠâĢ ĺ start _ index âĢ Ļ Ġof Ġthe Ġactual Ġanswer . ĠThe ĠâĢ ĺ start _ index Ġcorresponds Ġto Ġthe Ġindex Ġof Ġthe Ġanswer Ġtext Ġpresent Ġin Ġthe Ġparagraph / ĠContext . ĠFor Ġfinding ĠâĢ ĺ start _ index âĢ Ļ Ġwe Ġused Ġbuilt - in Ġpython Ġfunction Ġfind (). ĠThe Ġfunction Ġreturns Ġthe Ġlowest Ġindex Ġof Ġthe Ġactual Ġanswer Ġpresent Ġin Ġthe Ġcontext ( paragraph ). ĠIf Ġthe Ġanswer Ġis Ġnot Ġfound ĠâĢ ĺ - 1 âĢ Ļ Ġis Ġreturned Ġas Ġthe Ġindex . ĠThe Ġefficient Ġway Ġof Ġfinding Ġstart _ index Ġis , Ġif Ġthe Ġparagraph Ġ( Context ) Ġhas Ġmultiple Ġinstances Ġof Ġanswer Ġtext , Ġthen ĠâĢ ĺ start _ index âĢ Ļ Ġof Ġthe Ġanswer Ġshould Ġbe Ġthat Ġinstance Ġof Ġanswer Ġtext Ġwhose Ġcontext Ġactually Ġmatches Ġwith Ġwhat âĢ Ļ s Ġbeen Ġasked Ġin Ġthe Ġquestion . </s> Example Ġ( Question , ĠAnswer Ġand ĠPar agraph Ġfrom ĠB IB REF 8 ): </s> Question : ĠWhich Ġdrug Ġshould Ġbe Ġused Ġas Ġan Ġantidote Ġin Ġbenz od iazep ine Ġoverdose ? </s> Answer : Ġ' Fl um az en il ' </s> Par agraph ( context ): </s> " Fl um az en il Ġuse Ġin Ġbenz od iazep ine Ġoverdose Ġin Ġthe ĠUK : Ġa Ġretrospective Ġsurvey Ġof ĠNP IS Ġdata . ĠOB JECT IVE : ĠBenz od iazep ine Ġ( B Z D ) Ġoverdose Ġ( OD ) Ġcontinues Ġto Ġcause Ġsignificant Ġmorbid ity Ġand Ġmortality Ġin Ġthe ĠUK . ĠFl um az en il Ġis Ġan Ġeffective Ġantidote Ġbut Ġthere Ġis Ġa Ġrisk Ġof Ġseizures , Ġparticularly Ġin Ġthose Ġwho Ġhave Ġco - ing ested Ġtr icy cl ic Ġantidepressants . ĠA Ġstudy Ġwas Ġundertaken Ġto Ġexamine Ġthe Ġfrequency Ġof Ġuse , Ġsafety Ġand Ġefficacy Ġof Ġfl um az en il Ġin Ġthe Ġmanagement Ġof ĠB Z D ĠOD Ġin Ġthe ĠUK . ĠM ETHOD S : ĠA Ġ2 - year Ġretrospective Ġcohort Ġstudy Ġwas Ġperformed Ġof Ġall Ġenqu iries Ġto Ġthe ĠUK ĠNational ĠPo isons ĠInformation ĠService Ġinvolving ĠB Z D ĠOD . ĠRES ULTS : ĠFl um az en il Ġwas Ġadministered Ġto Ġ80 Ġpatients Ġin Ġ450 4 ĠB Z D - related Ġenqu iries , Ġ68 Ġof Ġwhom Ġdid Ġnot Ġhave Ġvent il atory Ġfailure Ġor Ġhad Ġrecognised Ġcont rain d ications Ġto Ġfl um az en il . ĠFactors Ġassociated Ġwith Ġfl um az en il Ġuse Ġwere Ġincreased Ġage , Ġsevere Ġpoisoning Ġand Ġvent il atory Ġfailure . ĠCo - ing est ion Ġof Ġtr icy cl ic Ġantidepressants Ġand Ġchronic Ġobstruct ive Ġpulmonary Ġdisease Ġdid Ġnot Ġinfluence Ġfl um az en il Ġadministration . ĠSe iz ure Ġfrequency Ġin Ġpatients Ġnot Ġtreated Ġwith Ġfl um az en il Ġwas Ġ0 . 3 % ". </s> Act ual Ġanswer Ġis Ġ' Fl um az en il ', Ġbut Ġthere Ġare Ġmultiple Ġinstances Ġof Ġword Ġ' F lu - m az en il '. ĠE fficient Ġway Ġto Ġidentify Ġthe Ġstart - index Ġfor Ġ' Fl um az en il ' ( answer ) Ġis Ġto Ġfind Ġthat Ġparticular Ġinstance Ġof Ġthe Ġword Ġ' Fl um az en il ' Ġwhich Ġmatches Ġthe Ġcontext Ġfor Ġthe Ġquestion . ĠIn Ġthe Ġabove Ġexample Ġ' Fl um az en il ' Ġhighlighted Ġin Ġbold Ġis Ġthe Ġactual Ġinstance Ġthat Ġmatches Ġquestion 's Ġcontext . ĠUnfortunately , Ġwe Ġcould Ġnot Ġidentify Ġreadily Ġavailable Ġtools Ġthat Ġcan Ġachieve Ġthis Ġgoal . ĠIn Ġour Ġfuture Ġwork , Ġwe Ġlook Ġforward Ġto Ġhandling Ġthese Ġscenarios Ġeffectively . </s> Note : ĠThe Ġcreators Ġof Ġ' S Qu AD ' ĠB IB REF 7 Ġhave Ġhandled Ġthe Ġtask Ġof Ġidentifying Ġanswer 's Ġstart _ index Ġeffectively . ĠBut Ġ' S Qu AD ' Ġdata Ġset Ġis Ġmuch Ġmore Ġgeneral Ġand Ġdoes Ġnot Ġinclude Ġbiomedical Ġquestion Ġanswering Ġdata . </s> Exper iments : ĠFact oid ĠQuestion ĠAn sw ering ĠTask Ġ: :: ĠTraining Ġand Ġerror Ġanalysis </s> During Ġour Ġtraining Ġwith Ġthe ĠBio AS Q Ġdata , Ġlearning Ġrate Ġis Ġset Ġto Ġ3 e - 5 , Ġas Ġmentioned Ġin Ġthe ĠBio BER T Ġpaper ĠB IB REF 0 . ĠWe Ġstarted Ġtraining Ġthe Ġmodel Ġwith Ġ4 95 Ġavailable Ġtrain Ġdata Ġand Ġ35 Ġtest Ġdata Ġby Ġsetting Ġnumber Ġof Ġepoch s Ġto Ġ50 . ĠAfter Ġtraining Ġwith Ġthese Ġhyper - param eters Ġtraining Ġaccuracy ( ex act Ġmatch ) Ġwas Ġ99 . 3 % ( over fitting ) Ġand Ġtesting Ġaccuracy Ġis Ġonly Ġ4 %. ĠIn Ġthe Ġnext Ġiteration Ġwe Ġreduced Ġthe Ġnumber Ġof Ġepoch s Ġto Ġ25 Ġthen Ġtraining Ġaccuracy Ġis Ġreduced Ġto Ġ98 . 5 % Ġand Ġtest Ġaccuracy Ġmoved Ġto Ġ5 %. ĠWe Ġfurther Ġreduced Ġnumber Ġof Ġepoch s Ġto Ġ15 , Ġand Ġthe Ġresulting Ġtraining Ġaccuracy Ġwas Ġ70 % Ġand Ġtest Ġaccuracy Ġ15 %. ĠIn Ġthe Ġnext Ġiteration Ġset Ġnumber Ġof Ġepoch s Ġto Ġ12 Ġand Ġachieved Ġtrain Ġaccuracy Ġof Ġ57 . 7 % Ġand Ġtest Ġaccuracy Ġ23 . 3 %. ĠRepe ated Ġthe Ġexperiment Ġwith Ġ11 Ġepoch s Ġand Ġfound Ġtraining Ġaccuracy Ġto Ġbe Ġ57 . 7 % Ġand Ġtest Ġaccuracy Ġto Ġbe Ġsame Ġ22 %. ĠIn Ġthe Ġnext Ġiteration Ġwe Ġset Ġnumber Ġof Ġepoch s Ġto Ġ' 9 ' Ġand Ġfound Ġtraining Ġaccuracy Ġof Ġ48 % Ġand Ġtest Ġaccuracy Ġof Ġ15 %. ĠHence Ġoptimum Ġnumber Ġof Ġepoch s Ġis Ġtaken Ġas Ġ12 Ġepoch s . </s> During Ġour Ġerror Ġanalysis Ġwe Ġfound Ġthat Ġon Ġtest Ġdata , Ġmodel Ġtends Ġto Ġreturn Ġtext Ġin Ġthe Ġbeginning Ġof Ġthe Ġcontext ( paragraph ) Ġas Ġthe Ġanswer . ĠOn Ġanalys ing Ġtrain Ġdata , Ġwe Ġfound Ġthat Ġthere Ġare Ġ' 120 ' ( out Ġof Ġ' 495 ') Ġquestion Ġanswering Ġdata Ġinstances Ġhaving Ġstart _ index : 0 , Ġmeaning Ġ120 ( Ġ25 %) Ġquestion Ġanswering Ġdata Ġhas Ġfirst Ġword ( s ) Ġin Ġthe Ġcontext ( paragraph ) Ġas Ġthe Ġanswer . ĠWe Ġremoved Ġ70 % Ġof Ġthose Ġinstances Ġin Ġorder Ġto Ġmake Ġtrain Ġdata Ġmore Ġbalanced . ĠIn Ġthe Ġnew Ġtrain Ġdata Ġset Ġwe Ġare Ġleft Ġwith Ġ' 411 ' Ġquestion Ġanswering Ġdata Ġinstances . ĠThis Ġtime Ġwe Ġgot Ġthe Ġhighest Ġtest Ġaccuracy Ġof Ġ26 % Ġat Ġ11 Ġepoch s . ĠWe Ġhave Ġsubmitted Ġour Ġresults Ġfor ĠBio AS Q Ġtest Ġbatch - 2 , Ġgot Ġstrict Ġaccuracy Ġof Ġ32 % Ġand Ġour Ġsystem Ġstood Ġin Ġ2 nd Ġplace . ĠInitially , Ġhyper - param eter - Ġ' batch Ġsize ' Ġis Ġset Ġto ĠâĢ ĺ 400 âĢ Ļ . ĠLater Ġit Ġis Ġtuned Ġto Ġ' 32 '. ĠAlthough Ġaccuracy ( ex act Ġanswer Ġmatch ) Ġremained Ġat Ġ26 %, Ġmodel Ġgenerated Ġconcise Ġand Ġbetter Ġanswers Ġat Ġbatch Ġsize ĠâĢ ĺ 32 âĢ Ļ , Ġthat Ġis Ġwrong Ġanswers Ġare Ġclose Ġto Ġthe Ġexpected Ġanswer Ġin Ġgood Ġnumber Ġof Ġcases . </s> Example .( from ĠB IB REF 8 ) </s> Question : ĠWhich Ġmutated Ġgene Ġcauses ĠC hed iak ĠHig ashi ĠSyndrome ? </s> Ex act ĠAnswer : ĠâĢ ĺ lys os omal Ġtrafficking Ġregulator Ġgene âĢ Ļ . </s> The Ġanswer Ġreturned Ġby Ġa Ġmodel Ġtrained Ġat ĠâĢ ĺ 400 âĢ Ļ Ġbatch Ġsize Ġis ĠâĢ ĺ Aut os omal - re cessive Ġcomplicated Ġsp astic Ġparap leg ia Ġwith Ġa Ġnovel Ġl ys os omal Ġtrafficking Ġregulator âĢ Ļ , Ġand Ġfrom Ġthe Ġone Ġtrained Ġat ĠâĢ ĺ 32 âĢ Ļ Ġbatch Ġsize Ġis ĠâĢ ĺ lys os omal Ġtrafficking Ġregulator âĢ Ļ . </s> In Ġfurther Ġexperiments , Ġwe Ġhave Ġfine Ġtuned Ġthe ĠBio BER T Ġmodel Ġwith Ġboth ĠâĢ ĺ S Qu AD âĢ Ļ Ġdataset Ġ( version Ġ2 . 0 ) Ġand ĠBio As q Ġtrain Ġdata . ĠFor Ġtraining Ġon ĠâĢ ĺ S Qu AD âĢ Ļ , Ġhyper Ġparameters - ĠLearning Ġrate Ġand Ġnumber Ġof Ġepoch s Ġare Ġset Ġto ĠâĢ ĺ 3 e - 3 âĢ Ļ Ġand ĠâĢ ĺ 3 âĢ Ļ Ġrespectively Ġas Ġmentioned Ġin Ġthe Ġpaper ĠB IB REF 1 . ĠTest Ġaccuracy Ġof Ġthe Ġmodel Ġboosted Ġto Ġ44 %. ĠIn Ġone Ġmore Ġexperiment Ġwe Ġtrained Ġmodel Ġonly Ġon ĠâĢ ĺ S Qu AD âĢ Ļ Ġdataset , Ġthis Ġtime Ġtest Ġaccuracy Ġof Ġthe Ġmodel Ġmoved Ġto Ġ47 %. ĠThe Ġreason Ġmodel Ġdid Ġnot Ġperform Ġup Ġto Ġthe Ġmark Ġwhen Ġtrained Ġwith ĠâĢ ĺ S Qu AD âĢ Ļ Ġalongside ĠBio AS Q Ġdata Ġcould Ġbe Ġthat Ġin Ġformatted ĠBio AS Q Ġdata , Ġstart _ index Ġfor Ġthe Ġanswer Ġis Ġnot Ġaccurate , Ġand Ġaffected Ġthe Ġoverall Ġaccuracy . </s> Our ĠSystems Ġand ĠTheir ĠPerformance Ġon ĠFact oid ĠQuestions </s> We Ġhave Ġexperimented Ġwith Ġseveral Ġsystems Ġand Ġtheir Ġvariations , Ġe . g . Ġcreated Ġby Ġtraining Ġwith Ġspecific Ġadditional Ġfeatures Ġ( see Ġnext Ġsubsection ). ĠHere Ġis Ġtheir Ġlist Ġand Ġshort Ġdescriptions . ĠUnfortunately Ġwe Ġdid Ġnot Ġpay Ġattention Ġto Ġnaming , Ġand Ġthe Ġsystems Ġevolved Ġbetween Ġtest Ġbatches , Ġso Ġthe Ġoverall Ġpicture Ġcan Ġonly Ġbe Ġunderstood Ġby Ġlooking Ġat Ġthe Ġdetails . </s> When Ġwe Ġstarted Ġthe Ġexperiments Ġour Ġobjective Ġwas Ġto Ġsee Ġwhether ĠBio BER T Ġand Ġentail ment - based Ġtechniques Ġcan Ġprovide Ġvalue Ġfor Ġin Ġthe Ġcontext Ġof Ġbiomedical Ġquestion Ġanswering . ĠThe Ġanswer Ġto Ġboth Ġquestions Ġwas Ġa Ġyes , Ġqualified Ġby Ġmany Ġexamples Ġclearly Ġshowing Ġthe Ġlimitations Ġof Ġboth Ġmethods . ĠTherefore Ġwe Ġtried Ġto Ġaddress Ġsome Ġof Ġthese Ġlimitations Ġusing Ġfeature Ġengineering Ġwith Ġmixed Ġresults : Ġsome Ġclear Ġerrors Ġgot Ġcorrected Ġand Ġnew Ġerrors Ġgot Ġintroduced , Ġwithout Ġoverall Ġimprovement Ġbut Ġconvincing Ġus Ġthat Ġin Ġfuture Ġexperiments Ġit Ġmight Ġbe Ġworth Ġtrying Ġfeature Ġengineering Ġagain Ġespecially Ġif Ġmore Ġtraining Ġdata Ġwere Ġavailable . </s> Overall Ġwe Ġexperimented Ġwith Ġseveral Ġapproaches Ġwith Ġthe Ġfollowing Ġaspects Ġof Ġthe Ġsystems Ġchanging Ġbetween Ġbatches , Ġthat Ġis Ġbeing Ġabsent Ġor Ġpresent : </s> training Ġon ĠBio As q Ġdata Ġvs . Ġtraining Ġon ĠS Qu AD </s> using Ġthe ĠBio As q Ġsnippets Ġfor Ġcontext Ġvs . Ġusing Ġthe Ġdocuments Ġfrom Ġthe Ġprovided ĠURLs Ġfor Ġcontext </s> adding Ġor Ġnot Ġthe ĠLAT , Ġi . e . Ġlex ical Ġanswer Ġtype , Ġfeature Ġ( see ĠB IB REF 9 , ĠB IB REF 10 Ġand Ġan Ġexplanation Ġin Ġthe Ġsubsection Ġjust Ġbelow ). </s> For ĠYes / No Ġquestions Ġ( only ) Ġwe Ġexperimented Ġwith Ġthe Ġentail ment Ġmethods . </s> We Ġwill Ġdiscuss Ġthe Ġperformance Ġof Ġthese Ġmodels Ġbelow Ġand Ġin ĠSection Ġ6 . ĠBut Ġbefore Ġwe Ġdo Ġthat , Ġlet Ġus Ġdiscuss Ġa Ġfeature Ġengineering Ġexperiment Ġwhich Ġeventually Ġproduced Ġmixed Ġresults , Ġbut Ġwhere Ġwe Ġfeel Ġit Ġis Ġpotentially Ġuseful Ġin Ġfuture Ġexperiments . </s> Our ĠSystems Ġand ĠTheir ĠPerformance Ġon ĠFact oid ĠQuestions Ġ: :: ĠLAT ĠFeature Ġconsidered Ġand Ġits Ġimpact Ġ( s lightly Ġnegative ) </s> During Ġerror Ġanalysis Ġwe Ġfound Ġthat Ġfor Ġsome Ġcases , Ġanswer Ġbeing Ġreturned Ġby Ġthe Ġmodel Ġis Ġfar Ġaway Ġfrom Ġwhat Ġit Ġis Ġbeing Ġasked Ġin Ġthe ĠQuestion . </s> Example : Ġ( from ĠB IB REF 8 ) </s> Question : ĠHy 's Ġlaw Ġmeasures Ġfailure Ġof Ġwhich Ġorgan ? </s> Act ual ĠAnswer : ĠâĢ ĺ L iver âĢ Ļ . </s> The Ġanswer Ġreturned Ġby Ġone Ġof Ġour Ġmodels Ġwas ĠâĢ ĺ alan ine Ġam in ot rans fer ase âĢ Ļ , Ġwhich Ġis Ġan Ġenzyme . ĠThe Ġmodel Ġreturns Ġan Ġenzyme , Ġwhen Ġthe Ġquestion Ġasked Ġfor Ġthe Ġorgan Ġname . ĠTo Ġaddress Ġthis Ġtype Ġof Ġerrors , Ġwe Ġdecided Ġto Ġtry Ġthe Ġconcepts Ġof ĠâĢ ĺ Lex ical ĠAnswer ĠType âĢ Ļ Ġ( L AT ) Ġand ĠFocus ĠWord , Ġwhich Ġwas Ġused Ġin ĠIBM ĠWatson , Ġsee ĠB IB REF 11 Ġfor Ġoverview ; ĠB IB REF 10 Ġfor Ġtechnical Ġdetails , Ġand ĠB IB REF 9 Ġfor Ġdetails Ġon Ġquestion Ġanalysis . ĠIn Ġan Ġexample Ġgiven Ġin Ġthe Ġlast Ġsource Ġwe Ġread : </s> PO ETS Ġ& ĠPO ET RY : ĠHe Ġwas Ġa Ġbank Ġclerk Ġin Ġthe ĠYuk on Ġbefore Ġhe Ġpublished Ġ" S ongs Ġof Ġa ĠSour d ough " Ġin Ġ1907 . </s> The Ġfocus Ġis Ġthe Ġpart Ġof Ġthe Ġquestion Ġthat Ġis Ġa Ġreference Ġto Ġthe Ġanswer . ĠIn Ġthe Ġexample Ġabove , Ġthe Ġfocus Ġis Ġ" he ". </s> L AT s Ġare Ġterms Ġin Ġthe Ġquestion Ġthat Ġindicate Ġwhat Ġtype Ġof Ġentity Ġis Ġbeing Ġasked Ġfor . </s> The Ġhead word Ġof Ġthe Ġfocus Ġis Ġgenerally Ġa ĠLAT , Ġbut Ġquestions Ġoften Ġcontain Ġadditional ĠLAT s , Ġand Ġin Ġthe ĠJe opard y ! Ġdomain , Ġcategories Ġare Ġan Ġadditional Ġsource Ġof ĠLAT s . </s> ( ...) ĠIn Ġthe Ġexample , ĠLAT s Ġare Ġ" he ", Ġ" cl erk ", Ġand Ġ" po et ". </s> For Ġexample Ġin Ġthe Ġquestion Ġ" Which Ġplant Ġdoes Ġo le uro pe in Ġoriginate Ġfrom ?" Ġ( BI B REF 8 ). ĠThe ĠLAT Ġhere Ġis ĠâĢ ĺ plant âĢ Ļ . ĠFor Ġthe ĠBio As q Ġtask Ġwe Ġdid Ġnot Ġneed Ġto Ġexplicitly Ġdistinguish Ġbetween Ġthe Ġfocus Ġand Ġthe ĠLAT Ġconcepts . ĠIn Ġthis Ġexample , Ġthe Ġexpectation Ġis Ġthat Ġanswer Ġreturned Ġby Ġthe Ġmodel Ġis Ġa Ġplant . ĠThus Ġit Ġis Ġconceivable Ġthat Ġthe Ġcos ine Ġdistance Ġbetween Ġcontextual Ġembed ding Ġof Ġword Ġ' plant ' Ġin Ġthe Ġquestion Ġand Ġcontextual Ġembed ding Ġfor Ġthe Ġanswer Ġpresent Ġin Ġthe Ġparagraph ( context ) Ġis Ġcomparatively Ġlow . ĠAs Ġa Ġresult Ġmodel Ġlearns Ġto Ġadjust Ġits Ġweights Ġduring Ġtraining Ġphase Ġand Ġreturns Ġanswers Ġwith Ġlow Ġcos ine Ġdistance Ġwith Ġthe ĠLAT . </s> We Ġused ĠStanford ĠCore N LP ĠB IB REF 12 Ġlibrary Ġto Ġwrite Ġrules Ġfor Ġextracting Ġlex ical Ġanswer Ġtype Ġpresent Ġin Ġthe Ġquestion , Ġboth Ġ' parts Ġof Ġspeech ' ( POS ) Ġand Ġdependency Ġparsing Ġfunctionality Ġwas Ġused . ĠWe Ġincorporated Ġthe ĠLex ical ĠAnswer ĠType Ġinto Ġone Ġof Ġour Ġsystems , ĠUN CC _ Q A 1 Ġin ĠB atch Ġ4 . ĠThis Ġsystem Ġunder per formed Ġour Ġsystem ĠFAC TO IDS Ġby Ġabout Ġ3 % Ġin Ġthe ĠMR R Ġmeasure , Ġbut Ġcorrected Ġerrors Ġsuch Ġas Ġin Ġthe Ġexample Ġabove . </s> Our ĠSystems Ġand ĠTheir ĠPerformance Ġon ĠFact oid ĠQuestions Ġ: :: ĠLAT ĠFeature Ġconsidered Ġand Ġits Ġimpact Ġ( s lightly Ġnegative ) Ġ: :: ĠAss um ptions Ġand Ġrules Ġfor Ġder iving Ġlex ical Ġanswer Ġtype . </s> There Ġare Ġdifferent Ġquestion Ġtypes : ĠâĢ ĺ Which âĢ Ļ , ĠâĢ ĺ What âĢ Ļ , ĠâĢ ĺ When âĢ Ļ , ĠâĢ ĺ How âĢ Ļ Ġetc . ĠEach Ġtype Ġof Ġquestion Ġis Ġbeing Ġhandled Ġdifferently Ġand Ġthere Ġare Ġcommon alities Ġamong Ġthe Ġrules Ġwritten Ġfor Ġdifferent Ġquestion Ġtypes . ĠQuestion Ġwords Ġare Ġidentified Ġthrough Ġparts Ġof Ġspeech Ġtags : Ġ' WD T ', Ġ' WR B ' Ġ, ' WP '. ĠWe Ġassumed Ġthat ĠLAT Ġis Ġa ĠâĢ ĺ N oun âĢ Ļ Ġand Ġfollows Ġthe Ġquestion Ġword . ĠOften Ġit Ġwas Ġalso Ġa Ġsubject Ġ( ns ub j ). ĠThis Ġprocess Ġis Ġillustrated Ġin ĠFig . FIG REF 15 . </s> L AT Ġcomputation Ġwas Ġgoverned Ġby Ġa Ġfew Ġsimple Ġrules , Ġe . g . Ġwhen Ġa Ġquestion Ġhas Ġmultiple Ġwords Ġthat Ġare Ġ' Subject s âĢ Ļ Ġ( and ĠâĢ ĺ N oun âĢ Ļ ), Ġa Ġword Ġthat Ġis Ġin Ġproximity Ġto Ġthe Ġquestion Ġword Ġis Ġconsidered Ġas ĠâĢ ĺ L AT âĢ Ļ . ĠThese Ġrules Ġare Ġdifferent Ġfor Ġeach Ġ" Wh " Ġword . </s> Name ly , Ġwhen Ġthe Ġword Ġimmediately Ġfollowing Ġthe Ġquestion Ġword Ġis Ġa ĠN oun , Ġwindow Ġsize Ġis Ġset Ġto ĠâĢ ĺ 3 âĢ Ļ . ĠThe Ġwindow Ġsize ĠâĢ ĺ 3 âĢ Ļ Ġmeans Ġwe Ġiter ate Ġthrough Ġthe Ġnext ĠâĢ ĺ 3 âĢ Ļ Ġwords Ġto Ġcheck Ġif Ġany Ġof Ġthe Ġword Ġis Ġboth ĠN oun Ġand ĠSubject , ĠIf Ġso , Ġsuch Ġword Ġis Ġconsidered Ġthe ĠâĢ ĺ L AT âĢ Ļ ; Ġelse Ġthe Ġword Ġthat Ġis Ġpresent Ġvery Ġnext Ġto Ġthe Ġquestion Ġword Ġis Ġconsidered Ġas Ġthe ĠâĢ ĺ L AT âĢ Ļ . </s> For Ġquestions Ġwith Ġwords ĠâĢ ĺ Which âĢ Ļ Ġ, ĠâĢ ĺ What âĢ Ļ , ĠâĢ ĺ When âĢ Ļ ; Ġa ĠN oun Ġimmediately Ġfollowing Ġthe Ġquestion Ġword Ġis Ġvery Ġoften Ġthe ĠLAT , Ġe . g . Ġ' en zyme ' Ġin ĠWhich Ġenzyme Ġis Ġtargeted Ġby ĠEv ol oc um ab ? . ĠWhen Ġthe Ġword Ġimmediately Ġfollowing Ġthe Ġquestion Ġword Ġis Ġnot Ġa ĠN oun , Ġe . g . Ġin ĠWhat Ġis Ġthe Ġfunction Ġof Ġthe Ġprotein ĠMag t 1 ? Ġthe Ġwindow Ġsize Ġis Ġset Ġto ĠâĢ ĺ 5 âĢ Ļ , Ġand Ġwe Ġiter ate Ġthrough Ġthe Ġnext ĠâĢ ĺ 5 âĢ Ļ Ġwords Ġ( if Ġpresent ) Ġand Ġsearch Ġfor Ġthe Ġword Ġthat Ġis Ġboth ĠN oun Ġand ĠSubject . ĠIf Ġpresent , Ġthe Ġword Ġis Ġconsidered Ġas Ġthe ĠâĢ ĺ L AT âĢ Ļ ; Ġelse , Ġthe ĠN oun Ġin Ġclose Ġproximity Ġto Ġthe Ġquestion Ġword Ġand Ġfollowing Ġit Ġis Ġreturned Ġas Ġthe ĠâĢ ĺ L AT âĢ Ļ . </s> For Ġquestions Ġwith Ġquestion Ġwords : ĠâĢ ĺ When âĢ Ļ , ĠâĢ ĺ Who âĢ Ļ , ĠâĢ ĺ Why âĢ Ļ , Ġthe ĠâĢ Ļ L AT âĢ Ļ Ġis Ġa Ġquestion Ġword Ġitself . ĠFor Ġthe Ġword ĠâĢ ĺ How ', Ġe . g . Ġin ĠHow Ġmany Ġse len op rote ins Ġare Ġencoded Ġin Ġthe Ġhuman Ġgenome ?, Ġwe Ġlook Ġat Ġthe Ġadjective Ġand Ġif Ġwe Ġfind Ġone , Ġwe Ġtake Ġit Ġto Ġbe Ġthe ĠLAT , Ġotherwise Ġthe Ġword Ġ' How ' Ġis Ġconsidered Ġas Ġthe ĠâĢ ĺ L AT âĢ Ļ . </s> Perhaps Ġbecause Ġof Ġusing Ġonly Ġvery Ġsimple Ġrules , Ġthe Ġaccuracy Ġfor ĠâĢ ĺ L AT âĢ Ļ Ġderiv ation Ġis Ġ75 %; Ġthat Ġis , Ġin Ġthe Ġremaining Ġ25 % Ġof Ġthe Ġcases Ġthe ĠLAT Ġword Ġis Ġidentified Ġincorrectly . ĠWorth Ġnoting Ġis Ġthat Ġthe Ġoverall Ġperformance Ġthe Ġsystem Ġthat Ġused ĠLAT s Ġwas Ġslightly Ġinferior Ġto Ġthe Ġsystem Ġwithout ĠLAT s , Ġbut Ġthe Ġtypes Ġof Ġerrors Ġchanged . ĠThe Ġtraining Ġused ĠBio BER T Ġwith Ġthe ĠLAT Ġfeature Ġas Ġpart Ġof Ġthe Ġinput Ġstring . ĠThe Ġerrors Ġit Ġintroduces Ġusually Ġinvolve Ġfinding Ġthe Ġwrong Ġelement Ġof Ġthe Ġcorrect Ġtype Ġe . g . Ġwrong Ġenzyme Ġwhen Ġtwo Ġsimilar Ġenzymes Ġare Ġdescribed Ġin Ġthe Ġtext , Ġor Ġ' ne uron ' Ġwhen Ġasked Ġabout Ġa Ġtype Ġof Ġcell Ġwith Ġa Ġcertain Ġfunction , Ġwhen Ġthe Ġanswer Ġcalls Ġfor Ġa Ġdifferent Ġcell Ġcategory , Ġadip ocytes , Ġand Ġboth Ġare Ġmentioned Ġin Ġthe Ġtext . ĠWe Ġfeel Ġwith Ġmore Ġdata Ġand Ġadditional Ġtuning Ġor Ġperhaps Ġusing Ġan Ġensemble Ġmodel , Ġwe Ġmight Ġbe Ġable Ġto Ġkeep Ġthe Ġcorrect Ġanswers , Ġand Ġimprove Ġthe Ġresults Ġon Ġthe Ġconfusing Ġexamples Ġlike Ġthe Ġone Ġmentioned Ġabove . ĠTherefore Ġif Ġwe Ġimprove Ġour ĠâĢ ĺ L AT âĢ Ļ Ġderiv ation Ġlogic , Ġor Ġhave Ġlarger Ġdatasets , Ġthen Ġperhaps Ġthe Ġneural Ġnetwork Ġtechniques Ġthey Ġwill Ġyield Ġbetter Ġresults . </s> Our ĠSystems Ġand ĠTheir ĠPerformance Ġon ĠFact oid ĠQuestions Ġ: :: ĠImpact Ġof ĠTraining Ġusing ĠBio As q Ġdata Ġ( s lightly Ġnegative ) </s> Training Ġon ĠBio As q Ġdata Ġin Ġour Ġentry Ġin ĠB atch Ġ1 Ġand ĠB atch Ġ2 Ġunder Ġthe Ġname ĠQ A 1 Ġshowed Ġit Ġmight Ġlead Ġto Ġover fitting . ĠThis Ġhappened Ġboth Ġwith Ġ( B atch Ġ2 ) Ġand Ġwithout Ġ( B atch Ġ1 ) Ġhyper param eters Ġtuning : Ġab ys mal Ġ18 % ĠMR R Ġin ĠB atch Ġ1 , Ġand Ġsl igh ly Ġbetter Ġone , Ġ40 % Ġin ĠB atch Ġ2 Ġ( although Ġin ĠB atch Ġ2 Ġit Ġwas Ġoverall Ġthe Ġsecond Ġbest Ġresult Ġin ĠMR R Ġbut Ġ16 % Ġlower Ġthan Ġthe Ġhighest Ġscore ). </s> In ĠB atch Ġ3 Ġ( only ), Ġour ĠUN CC _ Q A 3 Ġsystem Ġwas Ġfine Ġtuned Ġon ĠBio As q Ġand ĠS Qu AD Ġ2 . 0 ĠB IB REF 7 , Ġand Ġfor Ġdata Ġpre processing ĠContext Ġparagraph Ġis Ġgenerated Ġfrom Ġrelevant Ġsnippets Ġprovided Ġin Ġthe Ġtest Ġdata . ĠThis Ġsystem Ġunder per formed , Ġby Ġabout Ġ2 % Ġin ĠMR R , Ġour Ġother Ġentry ĠUN CC _ Q A 1 , Ġwhich Ġwas Ġalso Ġan Ġoverall Ġcategory Ġwinner Ġfor Ġthis Ġbatch . ĠThe Ġlatter Ġwas Ġalso Ġtrained Ġon ĠS Qu AD , Ġbut Ġnot Ġon ĠBio As q . ĠWe Ġsuspect Ġthat Ġthe Ġreason Ġcould Ġbe Ġthe Ġsimplistic Ġnature Ġof Ġthe Ġfind () Ġfunction Ġdescribed Ġin ĠSection Ġ3 . 1 . ĠSo , Ġthis Ġcould Ġbe Ġan Ġarea Ġwhere Ġa Ġbetter Ġalgorithm Ġfor Ġfinding Ġthe Ġbest Ġoccurrence Ġof Ġan Ġentity Ġcould Ġimprove Ġperformance . </s> Our ĠSystems Ġand ĠTheir ĠPerformance Ġon ĠFact oid ĠQuestions Ġ: :: ĠImpact Ġof ĠUsing ĠContext Ġfrom ĠURLs Ġ( negative ) </s> In Ġsome Ġexperiments , Ġfor Ġcontext Ġin Ġtesting , Ġwe Ġused Ġdocuments Ġfor Ġwhich ĠURL Ġpointers Ġare Ġprovided Ġin ĠBio As q . ĠHowever , Ġour Ġsystem ĠUN CC _ Q A 3 Ġunder per formed Ġour Ġother Ġsystem Ġtested Ġonly Ġon Ġthe Ġprovided Ġsnippets . </s> In ĠB atch Ġ5 Ġthe Ġunder performance Ġwas Ġabout Ġ6 % Ġof ĠMR R , Ġcompared Ġto Ġour Ġbest Ġsystem ĠUN CC _ Q A 1 , Ġand Ġby Ġ9 % Ġto Ġthe Ġtop Ġperformer . </s> Performance Ġon ĠYes / No Ġand ĠList Ġquestions </s> Our Ġwork Ġfocused Ġon ĠFact oid Ġquestions . ĠBut Ġwe Ġalso Ġhave Ġdone Ġexperiments Ġon ĠList - type Ġand ĠYes / No Ġquestions . </s> Performance Ġon ĠYes / No Ġand ĠList Ġquestions Ġ: :: ĠEnt ail ment Ġimproves ĠYes / No Ġaccuracy </s> We Ġstarted Ġby Ġanswering Ġalways ĠYES Ġ( in Ġbatch Ġ2 Ġand Ġ3 ) Ġto Ġget Ġthe Ġbaseline Ġperformance . ĠFor Ġbatch Ġ4 Ġwe Ġused Ġentail ment . ĠOur Ġalgorithm Ġwas Ġvery Ġsimple : ĠGiven Ġa Ġquestion Ġwe Ġiter ate Ġthrough Ġthe Ġcandidate Ġsentences Ġand Ġtry Ġto Ġfind Ġany Ġcandidate Ġsentence Ġis Ġcontradict ing Ġthe Ġquestion Ġ( with Ġconfidence Ġover Ġ50 %), Ġif Ġso Ġ' No ' Ġis Ġreturned Ġas Ġanswer , Ġelse Ġ' Yes ' Ġis Ġreturned . ĠIn Ġbatch Ġ4 Ġthis Ġstrategy Ġproduced Ġbetter Ġthan Ġthe ĠBio As q Ġbaseline Ġperformance , Ġand Ġcompared Ġto Ġour Ġother Ġsystems , Ġthe Ġuse Ġof Ġentail ment Ġincreased Ġthe Ġperformance Ġby Ġabout Ġ13 % Ġ( mac ro ĠF 1 Ġscore ). ĠWe Ġused Ġ' Allen N lp ' ĠB IB REF 13 Ġentail ment Ġlibrary Ġto Ġfind Ġentail ment Ġof Ġthe Ġcandidate Ġsentences Ġwith Ġquestion . </s> Performance Ġon ĠYes / No Ġand ĠList Ġquestions Ġ: :: ĠFor ĠList - type Ġthe ĠURLs Ġhave Ġnegative Ġimpact </s> Overall , Ġwe Ġfollowed Ġthe Ġsimilar Ġstrategy Ġthat 's Ġbeen Ġfollowed Ġfor ĠFact oid ĠQuestion ĠAn sw ering Ġtask . ĠWe Ġstarted Ġour Ġexperiment Ġwith Ġbatch Ġ2 , Ġwhere Ġwe Ġsubmitted Ġ20 Ġbest Ġanswers Ġ( with Ġcontext Ġfrom Ġsnippets ). ĠStarting Ġwith Ġbatch Ġ3 , Ġwe Ġperformed Ġpost Ġprocessing : Ġonce Ġmodels Ġgenerate Ġanswer Ġpredictions Ġ( n - best Ġpredictions ), Ġwe Ġdo Ġpost - processing Ġon Ġthe Ġpredicted Ġanswers . ĠIn Ġtest Ġbatch Ġ4 , Ġour Ġsystem Ġ( called ĠFAC TO IDS ) Ġachieved Ġhighest Ġrecall Ġscore Ġof ĠâĢ ĺ 0 . 70 33 âĢ Ļ Ġbut Ġlow Ġprecision Ġof Ġ0 . 11 19 , Ġleaving Ġopen Ġthe Ġquestion Ġof Ġhow Ġcould Ġwe Ġhave Ġbetter Ġbalanced Ġthe Ġtwo Ġmeasures . </s> In Ġthe Ġpost - processing Ġphase , Ġwe Ġtake Ġthe Ġtop ĠâĢ ĺ 20 âĢ Ļ Ġ( batch Ġ3 ) Ġand Ġtop Ġ5 Ġ( batch Ġ4 Ġand Ġ5 ), Ġpredicted Ġanswers , Ġtoken ize Ġthem Ġusing Ġcommon Ġsepar ators : Ġ' com ma ' Ġ, Ġ' and ', Ġ' also ', Ġ' as Ġwell Ġas '. ĠTokens Ġwith Ġcharacters Ġcount Ġmore Ġthan ĠâĢ ĺ 100 âĢ Ļ Ġare Ġeliminated Ġand Ġrest Ġof Ġthe Ġtokens Ġare Ġadded Ġto Ġthe Ġlist Ġof Ġpossible Ġanswers . ĠBio AS Q Ġevaluation Ġmechanism Ġdoes Ġnot Ġconsider Ġsnippets Ġwith Ġmore Ġthan ĠâĢ ĺ 100 âĢ Ļ Ġcharacters Ġas Ġa Ġvalid Ġanswer . ĠConsidering Ġlengthy Ġsnippets Ġin Ġto Ġthe Ġlist Ġof Ġanswers Ġwould Ġreduce Ġthe Ġmean Ġprecision Ġscore . ĠAs Ġa Ġfinal Ġstep , Ġduplicate Ġsnippets Ġin Ġthe Ġanswer Ġpool Ġare Ġremoved . ĠFor Ġexample , Ġconsider Ġthese Ġtop Ġ3 Ġanswers Ġpredicted Ġby Ġthe Ġsystem Ġ( before Ġpost - processing ): </s> { </s> " text ": Ġ" d end ritic Ġcells ", </s> " pro b ability ": Ġ0 . 755 45 407 334 26 441 , </s> " start _ log it ": Ġ8 . 46 60 46 3333 129 88 , </s> " end _ log it ": Ġ9 . 536 355 01 86 157 23 </s> }, </s> { </s> " text ": Ġ" ne ut roph ils , Ġmac roph ages Ġand </s> dist inct Ġsub types Ġof Ġd end ritic Ġcells ", </s> " pro b ability ": Ġ0 . 13 80 68 67 348 30 42 14 , </s> " start _ log it ": Ġ6 . 76 64 785 385 13 184 , </s> " end _ log it ": Ġ9 . 536 355 01 86 157 23 </s> }, </s> { </s> " text ": Ġ" mac roph ages Ġand Ġdistinct Ġsub types Ġof Ġd end ritic ", </s> " pro b ability ": Ġ0 . 01 397 34 75 27 11 78 242 , </s> " start _ log it ": Ġ6 . 76 64 785 385 13 184 , </s> " end _ log it ": Ġ7 . 24 57 64 73 23 60 84 </s> }, </s> After Ġexecution Ġof Ġpost - processing Ġhe ur istics , Ġthe Ġlist Ġof Ġanswers Ġreturned Ġis Ġas Ġfollows : </s> [" d end ritic Ġcells "], </s> [" ne ut roph ils "], </s> [" mac roph ages "], </s> [" dist inct Ġsub types Ġof Ġd end ritic Ġcells "] </s> Summary Ġof Ġour Ġresults </s> The Ġtables Ġbelow Ġsummarize Ġall Ġour Ġresults . ĠThey Ġshow Ġthat Ġthe Ġperformance Ġof Ġour Ġsystems Ġwas Ġmixed . ĠThe Ġsimple Ġarchitectures Ġand Ġalgorithm Ġwe Ġused Ġworked Ġvery Ġwell Ġonly Ġin ĠB atch Ġ3 . ĠHowever , Ġwe Ġfeel Ġwe Ġcan Ġbuilt Ġa Ġbetter Ġsystem Ġbased Ġon Ġthis Ġexperience . ĠIn Ġparticular Ġwe Ġobserved Ġboth Ġthe Ġvalue Ġof Ġcontextual Ġembed d ings Ġand Ġof Ġfeature Ġengineering Ġ( L AT ), Ġhowever Ġwe Ġfailed Ġto Ġcombine Ġthem Ġproperly . </s> Summary Ġof Ġour Ġresults Ġ: :: ĠFact oid Ġquestions Ġ: :: ĠSystems Ġused Ġin ĠB atch Ġ5 Ġexperiments </s> System Ġdescription Ġfor ĠâĢ ĺ UN CC _ Q A 1 âĢ Ļ : ĠThe Ġsystem Ġwas Ġfin et uned Ġon Ġthe ĠS Qu AD Ġ2 . 0 . ĠFor Ġdata Ġpre processing ĠContext Ġ/ Ġparagraph Ġwas Ġgenerated Ġfrom Ġrelevant Ġsnippets Ġprovided Ġin Ġthe Ġtest Ġdata . </s> System Ġdescription Ġfor ĠâĢ ĺ Q A 1 âĢ Ļ Ġ: ĠâĢ ĺ L AT âĢ Ļ Ġfeature Ġwas Ġadded Ġand Ġfin et uned Ġwith ĠS Qu AD Ġ2 . 0 . ĠFor Ġdata Ġpre processing ĠContext Ġ/ Ġparagraph Ġwas Ġgenerated Ġfrom Ġrelevant Ġsnippets Ġprovided Ġin Ġthe Ġtest Ġdata . </s> System Ġdescription Ġfor ĠâĢ ĺ UN CC _ Q A 3 âĢ Ļ Ġ: ĠFine Ġtuning Ġprocess Ġis Ġsame Ġas Ġit Ġis Ġdone Ġfor Ġthe Ġsystem ĠâĢ ĺ UN CC _ Q A 1 âĢ Ļ Ġin Ġtest Ġbatch - 5 . ĠDifference Ġis Ġduring Ġdata Ġpre processing , ĠContext / paragraph Ġis Ġgenerated Ġfrom Ġthe Ġrelevant Ġdocuments Ġfor Ġwhich ĠUR LS Ġare Ġincluded Ġin Ġthe Ġtest Ġdata . </s> Summary Ġof Ġour Ġresults Ġ: :: ĠList ĠQuestions </s> For ĠList - type Ġquestions , Ġalthough Ġpost Ġprocessing Ġhelped Ġin Ġthe Ġlater Ġbatches , Ġwe Ġnever Ġmanaged Ġto Ġobtain Ġcompetitive Ġprecision , Ġalthough Ġour Ġrecall Ġwas Ġgood . </s> Summary Ġof Ġour Ġresults Ġ: :: ĠYes / No Ġquestions </s> The Ġonly Ġthing Ġworth Ġremembering Ġfrom Ġour Ġperformance Ġis Ġthat Ġusing Ġentail ment Ġcan Ġhave Ġa Ġmeasurable Ġimpact Ġ( at Ġleast Ġwith Ġrespect Ġto Ġa Ġweak Ġbaseline ). ĠThe Ġresults Ġ( weak ) Ġare Ġin ĠTable Ġ3 . </s> Discussion , ĠFuture ĠExper iments , Ġand ĠCon clusions Ġ: :: ĠSummary : </s> In Ġcontrast Ġto Ġ2018 , Ġwhen Ġwe Ġsubmitted ĠB IB REF 2 Ġto ĠBio AS Q Ġa Ġsystem Ġbased Ġon Ġextract ive Ġsummar ization Ġ( and Ġscored Ġvery Ġhigh Ġin Ġthe Ġideal Ġanswer Ġcategory ), Ġthis Ġyear Ġwe Ġmainly Ġtargeted Ġfact oid Ġquestion Ġanswering Ġtask Ġand Ġfocused Ġon Ġexperimenting Ġwith ĠBio BER T . ĠAfter Ġthese Ġexperiments Ġwe Ġsee Ġthe Ġpromise Ġof ĠBio BER T Ġin ĠQ A Ġtasks , Ġbut Ġwe Ġalso Ġsee Ġits Ġlimitations . ĠThe Ġlatter Ġwe Ġtried Ġto Ġaddress Ġwith Ġmixed Ġresults Ġusing Ġfeature Ġengineering . ĠOverall Ġthese Ġexperiments Ġallowed Ġus Ġto Ġsecure Ġa Ġbest Ġand Ġa Ġsecond Ġbest Ġscore Ġin Ġdifferent Ġtest Ġbatches . ĠAlong Ġwith ĠFact oid - type Ġquestion , Ġwe Ġalso Ġtried ĠâĢ ĺ Yes / No âĢ Ļ Ġand ĠâĢ ĺ List âĢ Ļ - type Ġquestions , Ġand Ġdid Ġreasonably Ġwell Ġwith Ġour Ġvery Ġsimple Ġapproach . </s> For ĠYes / No Ġthe Ġmoral Ġworth Ġremembering Ġis Ġthat Ġreasoning Ġhas Ġa Ġpotential Ġto Ġinfluence Ġresults , Ġas Ġevidenced Ġby Ġour Ġadding Ġthe ĠAllen N LP Ġentail ment ĠB IB REF 13 Ġsystem Ġincreased Ġits Ġperformance . </s> All Ġour Ġdata Ġand Ġsoftware Ġis Ġavailable Ġat ĠGithub , Ġin Ġthe Ġpreviously Ġreferenced ĠURL Ġ( end Ġof ĠSection Ġ2 ). </s> Discussion , ĠFuture ĠExper iments , Ġand ĠCon clusions Ġ: :: ĠFuture Ġexperiments </s> In Ġthe Ġcurrent Ġmodel , Ġwe Ġhave Ġa Ġshallow Ġneural Ġnetwork Ġwith Ġa Ġsoft max Ġlayer Ġfor Ġpredicting Ġanswer Ġspan . ĠSh allow Ġnetworks Ġhowever Ġare Ġnot Ġgood Ġat Ġgeneral izations . ĠIn Ġour Ġfuture Ġexperiments Ġwe Ġwould Ġlike Ġto Ġcreate Ġdense Ġquestion Ġanswering Ġneural Ġnetwork Ġwith Ġa Ġsoft max Ġlayer Ġfor Ġpredicting Ġanswer Ġspan . ĠThe Ġmain Ġidea Ġis Ġto Ġget Ġcontextual Ġword Ġembed ding Ġfor Ġthe Ġwords Ġpresent Ġin Ġthe Ġquestion Ġand Ġparagraph Ġ( Context ) Ġand Ġfeed Ġthe Ġcontextual Ġword Ġembed d ings Ġretrieved Ġfrom Ġthe Ġlast Ġlayer Ġof ĠBio BER T Ġto Ġthe Ġdense Ġquestion Ġanswering Ġnetwork . ĠThe Ġmentioned Ġdense Ġlayered Ġquestion Ġanswering Ġneural Ġnetwork Ġneed Ġto Ġbe Ġtuned Ġfor Ġfinding Ġright Ġhyper Ġparameters . ĠAn Ġexample Ġof Ġsuch Ġarchitecture Ġis Ġshown Ġin ĠFig . FIG REF 30 . </s> In Ġone Ġmore Ġexperiment , Ġwe Ġwould Ġlike Ġto Ġadd Ġa Ġbetter Ġversion Ġof ĠâĢ ĺ L AT âĢ Ļ Ġcontextual Ġword Ġembed ding Ġas Ġa Ġfeature , Ġalong Ġwith Ġthe Ġactual Ġcontextual Ġword Ġembed d ings Ġfor Ġquestion Ġtext , Ġand ĠContext Ġand Ġfeed Ġthem Ġas Ġinput Ġto Ġthe Ġdense Ġquestion Ġanswering Ġneural Ġnetwork . ĠBy Ġthis Ġexperiment , Ġwe Ġwould Ġlike Ġto Ġfind Ġif ĠâĢ ĺ L AT âĢ Ļ Ġfeature Ġis Ġimproving Ġoverall Ġanswer Ġprediction Ġaccuracy . ĠAdding ĠâĢ ĺ L AT âĢ Ļ Ġfeature Ġthis Ġway Ġinstead Ġof Ġfeeding Ġthis Ġword Ġpiece Ġembed ding Ġdirectly Ġto Ġthe ĠBio BER T Ġ( as Ġwe Ġdid Ġin Ġour Ġabove Ġexperiments ) Ġwould Ġnot Ġdowngrade Ġthe Ġquality Ġof Ġcontextual Ġword Ġembed d ings Ġgenerated Ġform ĠâĢ ĺ Bio BER T '. ĠQuality Ġcontextual Ġword Ġembed d ings Ġwould Ġlead Ġto Ġefficient Ġtransfer Ġlearning Ġand Ġchances Ġare Ġthat Ġit Ġwould Ġimprove Ġthe Ġmodel 's Ġanswer Ġprediction Ġaccuracy . </s> We Ġalso Ġsee Ġpotential Ġfor Ġincorporating Ġdomain Ġspecific Ġinference Ġinto Ġthe Ġtask Ġe . g . Ġusing Ġthe ĠMed N LI Ġdataset ĠB IB REF 14 . ĠFor Ġall Ġtypes Ġof Ġexperiments Ġit Ġmight Ġbe Ġworth Ġexploring Ġclinical ĠB ERT Ġembed d ings ĠB IB REF 15 , Ġexplicitly Ġincorporating Ġdomain Ġknowledge Ġ( e . g . ĠB IB REF 16 ) Ġand Ġpossibly Ġdeeper Ġdiscourse Ġrepresentations Ġ( e . g . ĠB IB REF 17 ). </s> APP END IX </s> In Ġthis Ġappendix Ġwe Ġprovide Ġadditional Ġdetails Ġabout Ġthe Ġimplementations . </s> APP END IX Ġ: :: ĠSystems Ġand Ġtheir Ġdescriptions : </s> We Ġused Ġseveral Ġvariants Ġof Ġour Ġsystems Ġwhen Ġexperimenting Ġwith Ġthe ĠBio AS Q Ġproblems . ĠIn Ġretrospect , Ġit Ġwould Ġbe Ġmuch Ġeasier Ġto Ġunderstand Ġthe Ġchanges Ġif Ġwe Ġadopted Ġsome Ġm n emonic Ġconventions Ġin Ġnaming Ġthe Ġsystems . ĠSo , Ġwe Ġapologize Ġfor Ġthe Ġnames Ġthat Ġdo Ġnot Ġreflect Ġthe Ġmodifications , Ġand Ġnecess itate Ġthis Ġlist . </s> APP END IX Ġ: :: ĠSystems Ġand Ġtheir Ġdescriptions : Ġ: :: ĠFact oid ĠType ĠQuestion ĠAn sw ering : </s> We Ġpre process ed Ġthe Ġtest Ġdata Ġto Ġconvert Ġtest Ġdata Ġto ĠBio BER T Ġformat , ĠWe Ġgenerated ĠContext / paragraph Ġby Ġeither Ġaggreg ating Ġrelevant Ġsnippets Ġprovided Ġor Ġby Ġaggreg ating Ġdocuments Ġfor Ġwhich ĠUR LS Ġare Ġprovided Ġin Ġthe ĠBio AS Q Ġtest Ġdata . </s> APP END IX Ġ: :: ĠSystems Ġand Ġtheir Ġdescriptions : Ġ: :: ĠSystem Ġdescription Ġfor ĠQ A 1 : </s> We Ġgenerated ĠContext / paragraph Ġby Ġaggreg ating Ġrelevant Ġsnippets Ġavailable Ġin Ġthe Ġtest Ġdata Ġand Ġmapped Ġit Ġagainst Ġthe Ġquestion Ġtext Ġand Ġquestion Ġid . ĠWe Ġignored Ġthe Ġcontent Ġpresent Ġin Ġthe Ġdocuments Ġ( document ĠUR LS Ġwere Ġprovided Ġin Ġthe Ġoriginal Ġtest Ġdata ). ĠThe Ġmodel Ġis Ġfin et uned Ġwith ĠBio AS Q Ġdata . </s> data Ġpre processing Ġis Ġdone Ġin Ġthe Ġsame Ġway Ġas Ġit Ġis Ġdone Ġfor Ġtest Ġbatch - 1 . ĠModel Ġfine Ġtuned Ġon ĠBio AS Q Ġdata . </s> âĢ ĺ L AT âĢ Ļ / ĠFocus Ġword Ġfeature Ġadded Ġand Ġfine Ġtuned Ġwith ĠS Qu AD Ġ2 . 0 Ġ[ reference ]. ĠFor Ġdata Ġpre processing ĠContext Ġ/ Ġparagraph Ġis Ġgenerated Ġfrom Ġrelevant Ġsnippets Ġprovided Ġin Ġthe Ġtest Ġdata . </s> APP END IX Ġ: :: ĠSystems Ġand Ġtheir Ġdescriptions : Ġ: :: ĠSystem Ġdescription Ġfor ĠUN CC _ Q A _ 1 : </s> System Ġis Ġfin et uned Ġon Ġthe ĠS Qu AD Ġ2 . 0 Ġ[ reference ]. ĠFor Ġdata Ġpre processing ĠContext Ġ/ Ġparagraph Ġis Ġgenerated Ġfrom Ġrelevant Ġsnippets Ġprovided Ġin Ġthe Ġtest Ġdata . </s> âĢ ĺ L AT âĢ Ļ / ĠFocus Ġword Ġfeature Ġadded Ġand Ġfine Ġtuned Ġwith ĠS Qu AD Ġ2 . 0 Ġ[ reference ]. ĠFor Ġdata Ġpre processing ĠContext Ġ/ Ġparagraph Ġis Ġgenerated Ġfrom Ġrelevant Ġsnippets Ġprovided Ġin Ġthe Ġtest Ġdata . </s> The ĠSystem Ġis Ġfin et uned Ġon Ġthe ĠS Qu AD Ġ2 . 0 . ĠFor Ġdata Ġpre processing ĠContext Ġ/ Ġparagraph Ġis Ġgenerated Ġfrom Ġrelevant Ġsnippets Ġprovided Ġin Ġthe Ġtest Ġdata . </s> APP END IX Ġ: :: ĠSystems Ġand Ġtheir Ġdescriptions : Ġ: :: ĠSystem Ġdescription Ġfor ĠUN CC _ Q A 3 : </s> System Ġis Ġfin et uned Ġon Ġthe ĠS Qu AD Ġ2 . 0 Ġ[ reference ] Ġand ĠBio AS Q Ġdataset [ ]. For Ġdata Ġpre processing ĠContext Ġ/ Ġparagraph Ġis Ġgenerated Ġfrom Ġrelevant Ġsnippets Ġprovided Ġin Ġthe Ġtest Ġdata . </s> Fine Ġtuning Ġprocess Ġis Ġsame Ġas Ġit Ġis Ġdone Ġfor Ġthe Ġsystem ĠâĢ ĺ UN CC _ Q A _ 1 âĢ Ļ Ġin Ġtest Ġbatch - 5 . ĠDifference Ġis Ġduring Ġdata Ġpre processing , ĠContext / paragraph Ġis Ġgenerated Ġform Ġfrom Ġthe Ġrelevant Ġdocuments Ġfor Ġwhich ĠUR LS Ġare Ġincluded Ġin Ġthe Ġtest Ġdata . </s> APP END IX Ġ: :: ĠSystems Ġand Ġtheir Ġdescriptions : Ġ: :: ĠSystem Ġdescription Ġfor ĠUN CC _ Q A 2 : </s> Fine Ġtuning Ġprocess Ġis Ġsame Ġas Ġfor ĠâĢ ĺ UN CC _ Q A _ 1 ĠâĢ Ļ . ĠDifference Ġis ĠContext / paragraph Ġis Ġgenerated Ġform Ġfrom Ġthe Ġrelevant Ġdocuments Ġfor Ġwhich ĠUR LS Ġare Ġincluded Ġin Ġthe Ġtest Ġdata . ĠSystem ĠâĢ ĺ UN CC _ Q A _ 1 âĢ Ļ Ġgot Ġthe Ġhighest ĠâĢ ĺ MR R âĢ Ļ Ġscore Ġin Ġthe Ġ3 rd Ġtest Ġbatch Ġset . </s> APP END IX Ġ: :: ĠSystems Ġand Ġtheir Ġdescriptions : Ġ: :: ĠSystem Ġdescription Ġfor ĠFAC TO IDS : </s> The ĠSystem Ġis Ġfin et uned Ġon Ġthe ĠS Qu AD Ġ2 . 0 . ĠFor Ġdata Ġpre processing ĠContext Ġ/ Ġparagraph Ġis Ġgenerated Ġfrom Ġrelevant Ġsnippets Ġprovided Ġin Ġthe Ġtest Ġdata . </s> APP END IX Ġ: :: ĠSystems Ġand Ġtheir Ġdescriptions : Ġ: :: ĠList ĠType ĠQuestions : </s> We Ġattempted ĠList Ġtype Ġquestions Ġstarting Ġfrom Ġtest Ġbatch ĠâĢ ĺ 2 âĢ Ļ . ĠUsed Ġsimilar Ġapproach Ġthat 's Ġbeen Ġfollowed Ġfor ĠFact oid ĠQuestion Ġanswering Ġtask . ĠFor Ġall Ġthe Ġtest Ġbatch Ġsets , Ġin Ġthe Ġdata Ġpre Ġprocessing Ġphase ĠContext / Ġparagraph Ġis Ġgenerated Ġeither Ġby Ġaggreg ating Ġrelevant Ġsnippets Ġor Ġby Ġaggreg ating Ġdocuments ( UR LS ) Ġprovided Ġin Ġthe ĠBio AS Q Ġtest Ġdata . </s> For Ġtest Ġbatch - 2 , Ġmodel Ġ( System : ĠQ A 1 ) Ġis Ġfin et uned Ġon ĠBio AS Q Ġdata Ġand Ġsubmitted Ġtop ĠâĢ ĺ 20 âĢ Ļ Ġanswers Ġpredicted Ġby Ġthe Ġmodel Ġas Ġthe Ġlist Ġof Ġanswers . Ġsystem ĠâĢ ĺ Q A 1 âĢ Ļ Ġachieved Ġlow ĠF - Measure Ġscore : âĢ ĺ 0 . 07 86 âĢ Ļ Ġin Ġthe Ġsecond Ġtest Ġbatch . ĠIn Ġthe Ġfurther Ġtest Ġbatches Ġfor ĠList Ġtype Ġquestions , Ġwe Ġfin et uned Ġthe Ġmodel Ġon ĠSquad Ġdata Ġset Ġ[ reference ], Ġimplemented Ġpost Ġprocessing Ġtechniques Ġ( re fer Ġsection Ġ5 . 2 ) Ġand Ġachieved Ġa Ġbetter ĠF - me asure Ġscore : ĠâĢ ĺ 0 . 28 62 âĢ Ļ Ġin Ġthe Ġfinal Ġtest Ġbatch Ġset . </s> In Ġtest Ġbatch - 3 Ġ( System s Ġ: ĠâĢ ĺ Q A 1 âĢ Ļ / âĢ Ļ âĢ Ļ UN CC _ Q A _ 1 âĢ Ļ / âĢ Ļ UN CC _ Q A 3 âĢ Ļ / âĢ Ļ UN CC _ Q A 2 âĢ Ļ ) Ġtop Ġ20 Ġanswers Ġreturned Ġby Ġthe Ġmodel Ġis Ġsent Ġfor Ġpost Ġprocessing Ġand Ġin Ġtest Ġbatch Ġ4 Ġand Ġ5 Ġonly Ġtop Ġ5 Ġanswers Ġare Ġsent Ġfor Ġpost Ġprocessing . ĠSystem ĠUN CC _ Q A 2 ( in Ġbatch Ġ3 ) Ġfor ĠList Ġtype Ġquestion Ġanswering , ĠContext Ġis Ġgenerated Ġfrom Ġdocuments Ġfor Ġwhich ĠUR LS Ġare Ġprovided Ġin Ġthe ĠBio AS Q Ġtest Ġdata . Ġfor Ġthe Ġrest Ġof Ġthe Ġsystems Ġ( in Ġtest Ġbatch - 3 ) Ġfor ĠList ĠType Ġquestion Ġanswering Ġtask Ġsnippets Ġpresent Ġin Ġthe ĠBio a S Q Ġtest Ġdata Ġare Ġused Ġto Ġgenerate Ġcontext . </s> In Ġtest Ġbatch - 4 Ġ( System Ġ: ĠâĢ ĺ F AC TO IDS âĢ Ļ / âĢ Ļ UN CC _ Q A _ 1 âĢ Ļ / âĢ Ļ UN CC _ Q A 3 âĢ Ļ âĢ Ļ ) Ġtop Ġ5 Ġanswers Ġreturned Ġby Ġthe Ġmodel Ġis Ġsent Ġfor Ġpost Ġprocessing . ĠIn Ġcase Ġof Ġsystem ĠâĢ ĺ F AC TO IDS âĢ Ļ Ġsnippets Ġin Ġthe Ġtest Ġdata Ġwere Ġused Ġto Ġgenerate Ġcontext . Ġfor Ġsystems ĠâĢ Ļ UN CC _ Q A _ 1 âĢ Ļ Ġand ĠâĢ Ļ UN CC _ Q A 3 âĢ Ļ Ġcontext Ġis Ġgenerated Ġfrom Ġthe Ġdocuments Ġfor Ġwhich ĠUR LS Ġare Ġprovided Ġin Ġthe ĠBio AS Q Ġtest Ġdata . </s> In Ġtest Ġbatch - 5 Ġ( ĠSystems : ĠâĢ ĺ Q A 1 âĢ Ļ / âĢ Ļ UN CC _ Q A _ 1 âĢ Ļ / âĢ Ļ UN CC _ Q A 3 âĢ Ļ / âĢ Ļ UN CC _ Q A 2 âĢ Ļ Ġ) Ġour Ġapproach Ġis Ġthe Ġsame Ġas Ġthat Ġof Ġtest Ġbatch - 4 Ġwhere Ġtop Ġ5 Ġanswers Ġreturned Ġby Ġthe Ġmodel Ġis Ġsent Ġfor Ġpost Ġprocessing . Ġfor Ġall Ġthe Ġsystems Ġ( in Ġtest Ġbatch - 5 ) Ġcontext Ġis Ġgenerated Ġfrom Ġthe Ġsnippets Ġprovided Ġin Ġthe ĠBio AS Q Ġtest Ġdata . </s> APP END IX Ġ: :: ĠSystems Ġand Ġtheir Ġdescriptions : Ġ: :: ĠYes / No ĠType ĠQuestions : </s> For Ġthe Ġfirst Ġ3 Ġtest Ġbatches , ĠWe Ġhave Ġsubmitted Ġanswer ĠâĢ ĺ Yes âĢ Ļ Ġto Ġall Ġthe Ġquestions . ĠLater , Ġwe Ġemployed ĠâĢ ĺ Sent ence ĠEnt ail ment âĢ Ļ Ġtechniques ( re fer Ġsection Ġ6 . 0 ) Ġfor Ġthe Ġfourth Ġand Ġfifth Ġtest Ġbatch Ġsets . ĠOur ĠSystems Ġwith ĠâĢ ĺ Sent ence ĠEnt ail ment âĢ Ļ Ġapproach Ġ( for ĠâĢ ĺ Yes âĢ Ļ / ĠâĢ ĺ No âĢ Ļ Ġquestion Ġanswering ): ĠâĢ ĺ UN CC _ Q A _ 1 âĢ Ļ ( test Ġbatch - 4 ), ĠUN CC _ Q A 3 ( test Ġbatch - 5 ). </s> APP END IX Ġ: :: ĠAdditional Ġdetails Ġfor ĠYes / No ĠType ĠQuestions </s> We Ġused ĠText ual ĠEnt ail ment Ġin ĠB atch Ġ4 Ġand Ġ5 Ġfor ĠâĢ ĺ Yes âĢ Ļ / âĢ ĺ No âĢ Ļ Ġquestion Ġtype . ĠThe Ġalgorithm Ġwas Ġvery Ġsimple : ĠGiven Ġa Ġquestion Ġwe Ġiter ate Ġthrough Ġthe Ġcandidate Ġsentences , Ġand Ġlook Ġfor Ġany Ġcandidate Ġsentences Ġcontradict ing Ġthe Ġquestion . ĠIf Ġwe Ġfind Ġone Ġ' No ' Ġis Ġreturned Ġas Ġanswer , Ġelse Ġ' Yes ' Ġis Ġreturned . Ġ( The Ġconfidence Ġfor Ġcontradiction Ġwas Ġset Ġat Ġ50 %) ĠWe Ġused ĠAllen N LP ĠB IB REF 13 Ġentail ment Ġlibrary Ġto Ġfind Ġentail ment Ġof Ġthe Ġcandidate Ġsentences Ġwith Ġquestion . </s> Flow ĠChart Ġfor ĠYes / No ĠQuestion Ġanswer Ġprocessing Ġis Ġshown Ġin ĠFig . FIG REF 51 </s> APP END IX Ġ: :: ĠAss um ptions , Ġrules Ġand Ġlogic Ġflow Ġfor Ġder iving ĠLex ical ĠAnswer ĠTypes Ġfrom Ġquestions </s> There Ġare Ġdifferent Ġquestion Ġtypes , Ġand Ġwe Ġdistinguished Ġthem Ġbased Ġon Ġthe Ġquestion Ġwords : ĠâĢ ĺ Which âĢ Ļ , ĠâĢ ĺ What âĢ Ļ , ĠâĢ ĺ When âĢ Ļ , ĠâĢ ĺ How âĢ Ļ Ġetc . ĠEach Ġtype Ġof Ġquestion Ġis Ġbeing Ġhandled Ġdifferently Ġand Ġthere Ġare Ġcommon alities Ġamong Ġthe Ġrules Ġwritten Ġfor Ġdifferent Ġquestion Ġtypes . ĠHow Ġare Ġquestion Ġwords Ġidentified ? Ġquestion Ġwords Ġhave Ġparts Ġof Ġspeech ( POS ): Ġ' WD T ', Ġ' WR B ', Ġ' WP '. </s> Ass um ptions : </s> 1 ) ĠLex ical Ġanswer Ġtype Ġ( âĢ ĺ L AT âĢ Ļ ) Ġor Ġfocus Ġword Ġis Ġof Ġtype ĠN oun Ġand Ġfollows Ġthe Ġquestion Ġword . </s> 2 ) ĠThe ĠLAT Ġword Ġis Ġa ĠSubject . Ġ( This Ġclearly Ġnot Ġalways Ġtrue , Ġbut Ġwe Ġused Ġa Ġvery Ġsimple Ġmethod ). ĠNote : ĠâĢ ĺ Stan ford N LP âĢ Ļ Ġdependency Ġparsing Ġtag Ġfor Ġidentifying Ġsubject Ġis Ġ' ns ub j ' Ġor Ġ' ns ub j pass '. </s> 3 ) ĠWhen Ġa Ġquestion Ġhas Ġmultiple Ġwords Ġthat Ġare Ġof Ġtype ĠSubject Ġ( and ĠN oun ), Ġa Ġword Ġthat Ġis Ġin Ġproximity Ġto Ġthe Ġquestion Ġword Ġis Ġconsidered Ġas ĠâĢ ĺ L AT âĢ Ļ . </s> 4 ) ĠFor Ġquestions Ġwith Ġquestion Ġwords : ĠâĢ ĺ When âĢ Ļ , ĠâĢ ĺ Who âĢ Ļ , ĠâĢ ĺ Why âĢ Ļ , Ġthe ĠâĢ Ļ L AT âĢ Ļ Ġis Ġa Ġquestion Ġword Ġitself Ġthat Ġis , ĠâĢ ĺ When âĢ Ļ , ĠâĢ ĺ Who âĢ Ļ , ĠâĢ ĺ Why âĢ Ļ Ġrespectively . </s> Rules Ġand Ġlogic Ġflow Ġto Ġtraverse Ġa Ġquestion : ĠThe Ġthree Ġcases Ġbelow Ġdescribe Ġthe Ġlogic Ġflow Ġof Ġfinding ĠLAT s . ĠThe Ġfigures Ġshow Ġthe Ġgram matical Ġstructures Ġused Ġfor Ġthis Ġpurpose . </s> APP END IX Ġ: :: ĠAss um ptions , Ġrules Ġand Ġlogic Ġflow Ġfor Ġder iving ĠLex ical ĠAnswer ĠTypes Ġfrom Ġquestions Ġ: :: ĠCase - 1 : </s> Question Ġwith Ġquestion Ġword ĠâĢ ĺ How âĢ Ļ . </s> For Ġquestions Ġwith Ġquestion Ġword Ġ' How ', Ġthe Ġadjective Ġthat Ġfollows Ġthe Ġquestion Ġword Ġis Ġconsidered Ġas ĠâĢ ĺ L AT âĢ Ļ Ġ( need Ġnot Ġfollow Ġimmediately ). ĠIf Ġan Ġadjective Ġis Ġabsent , Ġword Ġ' How ' Ġis Ġconsidered Ġas ĠâĢ ĺ L AT âĢ Ļ . ĠWhen Ġthere Ġare Ġmultiple Ġwords Ġthat Ġare Ġadject ives , Ġa Ġword Ġin Ġclose Ġproximity Ġto Ġthe Ġquestion Ġword Ġand Ġfollows Ġit Ġis Ġreturned Ġas ĠâĢ ĺ L AT âĢ Ļ . ĠNote : ĠThe Ġpart Ġof Ġspeech Ġtag Ġto Ġidentify Ġadject ives Ġis Ġ' JJ '. ĠFor ĠOther Ġpossible Ġquestion Ġwords Ġlike ĠâĢ ĺ whose âĢ Ļ . ĠâĢ ĺ L AT âĢ Ļ / Focus Ġword Ġis Ġquestion Ġwords Ġitself . </s> Example ĠQuestion : ĠHow Ġmany Ġse len op rote ins Ġare Ġencoded Ġin Ġthe Ġhuman Ġgenome ? </s> APP END IX Ġ: :: ĠAss um ptions , Ġrules Ġand Ġlogic Ġflow Ġfor Ġder iving ĠLex ical ĠAnswer ĠTypes Ġfrom Ġquestions Ġ: :: ĠCase - 2 : </s> Questions Ġwith Ġquestion Ġwords ĠâĢ ĺ Which âĢ Ļ Ġ, ĠâĢ ĺ What âĢ Ļ Ġand Ġall Ġother Ġpossible Ġquestion Ġwords ; Ġa Ġ' N oun ' Ġimmediately Ġfollowing Ġthe Ġquestion Ġword . </s> Example ĠQuestion : ĠWhich Ġenzyme Ġis Ġtargeted Ġby ĠEv ol oc um ab ? </s> Here , ĠFocus Ġword / L AT Ġis ĠâĢ ĺ en zyme âĢ Ļ Ġwhich Ġis Ġboth ĠN oun Ġand ĠSubject Ġand Ġimmediately Ġfollows Ġthe Ġquestion Ġword . </s> When Ġthe Ġword Ġimmediately Ġfollowing Ġthe Ġquestion Ġword Ġis Ġa Ġnoun , Ġthe Ġwindow Ġsize Ġis Ġset Ġto ĠâĢ ĺ 3 âĢ Ļ . ĠThis Ġsize ĠâĢ ĺ 3 âĢ Ļ Ġmeans Ġthat Ġwe Ġiter ate Ġthrough Ġthe Ġnext ĠâĢ ĺ 3 âĢ Ļ Ġwords Ġ( if Ġpresent ) Ġto Ġcheck Ġif Ġany Ġof Ġthe Ġword Ġis Ġboth Ġ' N oun ' Ġand Ġ' Subject ', ĠIf Ġso , Ġthe Ġword Ġis Ġconsidered Ġas ĠâĢ ĺ L AT âĢ Ļ / Focus ĠWord . ĠElse Ġthe Ġword Ġthat Ġis Ġpresent Ġvery Ġnext Ġto Ġthe Ġquestion Ġword Ġis Ġconsidered Ġas ĠâĢ ĺ L AT âĢ Ļ . </s> APP END IX Ġ: :: ĠAss um ptions , Ġrules Ġand Ġlogic Ġflow Ġfor Ġder iving ĠLex ical ĠAnswer ĠTypes Ġfrom Ġquestions Ġ: :: ĠCase - 3 : </s> Questions Ġwith Ġquestion Ġwords ĠâĢ ĺ Which âĢ Ļ Ġ, ĠâĢ ĺ What âĢ Ļ Ġand Ġall Ġother Ġpossible Ġquestion Ġwords ; Ġword Ġimmediately Ġfollowing Ġthe Ġquestion Ġword Ġis Ġnot Ġa Ġ' N oun '. </s> Example ĠQuestion : ĠWhat Ġis Ġthe Ġfunction Ġof Ġthe Ġprotein ĠMag t 1 ? </s> Here , ĠFocus Ġword / L AT Ġis ĠâĢ ĺ function ĠâĢ Ļ Ġwhich Ġis Ġboth ĠN oun Ġand ĠSubject Ġand Ġdoes Ġnot Ġimmediately Ġfollow Ġthe Ġquestion Ġword . </s> When Ġthe Ġvery Ġnext Ġword Ġfollowing Ġthe Ġquestion Ġword Ġis Ġnot Ġa ĠN oun , Ġwindow Ġsize Ġis Ġset Ġto ĠâĢ ĺ 5 âĢ Ļ . ĠWindow Ġsize ĠâĢ ĺ 5 âĢ Ļ Ġcorresponds Ġthat Ġwe Ġiter ate Ġthrough Ġthe Ġnext ĠâĢ ĺ 5 âĢ Ļ Ġwords Ġ( if Ġpresent ) Ġand Ġsearch Ġfor Ġthe Ġword Ġthat Ġis Ġboth ĠN oun Ġand ĠSubject . ĠIf Ġpresent , Ġthe Ġword Ġis Ġconsidered Ġas ĠâĢ ĺ L AT âĢ Ļ . ĠElse , Ġthe Ġ' N oun ' Ġclose Ġproximity Ġto Ġthe Ġquestion Ġword Ġand Ġfollows Ġit Ġis Ġreturned Ġas ĠâĢ ĺ L AT âĢ Ļ . </s> Ad Ġwe Ġmentioned Ġearlier , Ġthe Ġaccuracy Ġfor ĠâĢ ĺ L AT âĢ Ļ Ġderiv ation Ġis Ġ75 Ġpercent . ĠBut Ġclearly Ġthe Ġsimple Ġlogic Ġdescribed Ġabove Ġcan Ġbe Ġimproved , Ġas Ġshown Ġin ĠB IB REF 9 , ĠB IB REF 10 . ĠWhether Ġthis Ġin Ġturn Ġproduces Ġimprovements Ġin Ġthis Ġparticular Ġtask Ġis Ġan Ġopen Ġquestion . </s> APP END IX Ġ: :: ĠPro posing ĠFuture ĠExper iments </s> In Ġthe Ġcurrent Ġmodel , Ġwe Ġhave Ġa Ġshallow Ġneural Ġnetwork Ġwith Ġa Ġsoft max Ġlayer Ġfor Ġpredicting Ġanswer Ġspan . ĠSh allow Ġnetworks Ġhowever Ġare Ġnot Ġgood Ġat Ġgeneral izations . ĠIn Ġour Ġfuture Ġexperiments Ġwe Ġwould Ġlike Ġto Ġcreate Ġdense Ġquestion Ġanswering Ġneural Ġnetwork Ġwith Ġa Ġsoft max Ġlayer Ġfor Ġpredicting Ġanswer Ġspan . ĠThe Ġmain Ġidea Ġis Ġto Ġget Ġcontextual Ġword Ġembed ding Ġfor Ġthe Ġwords Ġpresent Ġin Ġthe Ġquestion Ġand Ġparagraph Ġ( Context ) Ġand Ġfeed Ġthe Ġcontextual Ġword Ġembed d ings Ġretrieved Ġfrom Ġthe Ġlast Ġlayer Ġof ĠBio BER T Ġto Ġthe Ġdense Ġquestion Ġanswering Ġnetwork . ĠThe Ġmentioned Ġdense Ġlayered Ġquestion Ġanswering ĠNeural Ġnetwork Ġneed Ġto Ġbe Ġtuned Ġfor Ġfinding Ġright Ġhyper Ġparameters . ĠAn Ġexample Ġof Ġsuch Ġarchitecture Ġis Ġshown Ġin ĠFig . FIG REF 30 . </s> In Ġanother Ġexperiment Ġwe Ġwould Ġlike Ġto Ġonly Ġfeed Ġcontextual Ġword Ġembed d ings Ġfor ĠFocus Ġword / ĠâĢ ĺ L AT âĢ Ļ , Ġparagraph / ĠContext Ġas Ġinput Ġto Ġthe Ġquestion Ġanswering Ġneural Ġnetwork . ĠIn Ġthis Ġexperiment Ġwe Ġwould Ġneglect Ġall Ġembed d ings Ġfor Ġthe Ġquestion Ġtext Ġexcept Ġthat Ġof ĠFocus Ġword / ĠâĢ ĺ L AT âĢ Ļ . ĠOur Ġassumption Ġand Ġidea Ġfor Ġconsidering Ġfocus Ġword Ġand Ġneglect ing Ġremaining Ġwords Ġin Ġthe Ġquestion Ġis Ġthat Ġduring Ġtraining Ġphase Ġit Ġwould Ġmake Ġmore Ġprecise Ġfor Ġthe Ġmodel Ġto Ġidentify Ġthe Ġfocus Ġof Ġthe Ġquestion Ġand Ġmap Ġanswers Ġagainst Ġthe Ġquestion âĢ Ļ s Ġfocus . ĠTo Ġvalidate Ġour Ġassumption , Ġwe Ġwould Ġlike Ġto Ġtake Ġsample Ġquestion Ġanswering Ġdata Ġand Ġfind Ġthe Ġcos ine Ġdistance Ġbetween Ġcontextual Ġembed ding Ġof ĠFocus Ġword Ġand Ġthat Ġof Ġthe Ġactual Ġanswer Ġand Ġverify Ġif Ġthe Ġcos ine Ġdistance Ġis Ġcomparatively Ġlow Ġin Ġmost Ġof Ġthe Ġcases . </s> In Ġone Ġmore Ġexperiment , Ġwe Ġwould Ġlike Ġto Ġadd Ġa Ġbetter Ġversion Ġof ĠâĢ ĺ L AT âĢ Ļ Ġcontextual Ġword Ġembed ding Ġas Ġa Ġfeature , Ġalong Ġwith Ġthe Ġactual Ġcontextual Ġword Ġembed d ings Ġfor Ġquestion Ġtext , Ġand ĠContext Ġand Ġfeed Ġthem Ġas Ġinput Ġto Ġthe Ġdense Ġquestion Ġanswering Ġneural Ġnetwork . ĠBy Ġthis Ġexperiment , Ġwe Ġwould Ġlike Ġto Ġfind Ġif ĠâĢ ĺ L AT âĢ Ļ Ġfeature Ġis Ġimproving Ġoverall Ġanswer Ġprediction Ġaccuracy . ĠAdding ĠâĢ ĺ L AT âĢ Ļ Ġfeature Ġthis Ġway Ġinstead Ġof Ġfeeding ĠFocus Ġword âĢ Ļ s Ġword Ġpiece Ġembed ding Ġdirectly Ġ( as Ġwe Ġdid Ġin Ġour Ġabove Ġexperiments ) Ġto Ġthe ĠBio BER T Ġwould Ġnot Ġdowngrade Ġthe Ġquality Ġof Ġcontextual Ġword Ġembed d ings Ġgenerated Ġform ĠâĢ ĺ Bio BER T '. ĠQuality Ġcontextual Ġword Ġembed d ings Ġwould Ġlead Ġto Ġefficient Ġtransfer Ġlearning Ġand Ġchances Ġare Ġthat Ġit Ġwould Ġimprove Ġthe Ġmodel 's Ġanswer Ġprediction Ġaccuracy .