Logistic Regression, neural networks
EXTRACTIVE
The choice of two methods for our empirical study is motivated by the best performance achieved by Logistic Regression in question-question similarity at SemEval 2017 (best system BIBREF37 and second best system BIBREF38 ), and the high performance achieved by neural networks on larger datasets such as SNLI BIBREF13 , BIBREF39 , BIBREF40 , BIBREF41 . We first define the RQE task, then present the two approaches, and evaluate their performance on five different datasets.
1901.08079
Introduction </s> With Ġthe Ġavailability Ġof Ġrich Ġdata Ġon Ġusers ' Ġlocations , Ġprofiles Ġand Ġsearch Ġhistory , Ġpersonal ization Ġhas Ġbecome Ġthe Ġleading Ġtrend Ġin Ġlarge - scale Ġinformation Ġretrieval . ĠHowever , Ġefficiency Ġthrough Ġpersonal ization Ġis Ġnot Ġyet Ġthe Ġmost Ġsuitable Ġmodel Ġwhen Ġtackling Ġdomain - specific Ġsearches . ĠThis Ġis Ġdue Ġto Ġseveral Ġfactors , Ġsuch Ġas Ġthe Ġlex ical Ġand Ġsemantic Ġchallenges Ġof Ġdomain - specific Ġdata Ġthat Ġoften Ġinclude Ġadvanced Ġargument ation Ġand Ġcomplex Ġcontextual Ġinformation , Ġthe Ġhigher Ġsp ars eness Ġof Ġrelevant Ġinformation Ġsources , Ġand Ġthe Ġmore Ġpronounced Ġlack Ġof Ġsimilarities Ġbetween Ġusers ' Ġsearches . </s> A Ġrecent Ġstudy Ġon Ġexpert Ġsearch Ġstrategies Ġamong Ġhealthcare Ġinformation Ġprofessionals ĠB IB REF 0 Ġshowed Ġthat , Ġfor Ġa Ġgiven Ġsearch Ġtask , Ġthey Ġspend Ġan Ġaverage Ġof Ġ60 Ġminutes Ġper Ġcollection Ġor Ġdatabase , Ġ3 Ġminutes Ġto Ġexamine Ġthe Ġrelevance Ġof Ġeach Ġdocument , Ġand Ġ4 Ġhours Ġof Ġtotal Ġsearch Ġtime . ĠWhen Ġwritten Ġin Ġsteps , Ġtheir Ġsearch Ġstrategy Ġspans Ġover Ġ15 Ġlines Ġand Ġcan Ġreach Ġup Ġto Ġ105 Ġlines . </s> With Ġthe Ġabundance Ġof Ġinformation Ġsources Ġin Ġthe Ġmedical Ġdomain , Ġconsumers Ġare Ġmore Ġand Ġmore Ġfaced Ġwith Ġa Ġsimilar Ġchallenge , Ġone Ġthat Ġneeds Ġdedicated Ġsolutions Ġthat Ġcan Ġadapt Ġto Ġthe Ġheterogeneity Ġand Ġspecifics Ġof Ġhealth - related Ġinformation . </s> D ed icated ĠQuestion ĠAn sw ering Ġ( Q A ) Ġsystems Ġare Ġone Ġof Ġthe Ġviable Ġsolutions Ġto Ġthis Ġproblem Ġas Ġthey Ġare Ġdesigned Ġto Ġunderstand Ġnatural Ġlanguage Ġquestions Ġwithout Ġrelying Ġon Ġexternal Ġinformation Ġon Ġthe Ġusers . </s> In Ġthe Ġcontext Ġof ĠQ A , Ġthe Ġgoal Ġof ĠRecogn izing ĠQuestion ĠEnt ail ment Ġ( R Q E ) Ġis Ġto Ġretrieve Ġanswers Ġto Ġa Ġpremise Ġquestion Ġ( ĠIN LINE FORM 0 Ġ) Ġby Ġretrieving Ġinferred Ġor Ġent ailed Ġquestions , Ġcalled Ġhypothesis Ġquestions Ġ( ĠIN LINE FORM 1 Ġ) Ġthat Ġalready Ġhave Ġassociated Ġanswers . ĠTherefore , Ġwe Ġdefine Ġthe Ġentail ment Ġrelation Ġbetween Ġtwo Ġquestions Ġas : Ġa Ġquestion ĠIN LINE FORM 2 Ġentails Ġa Ġquestion ĠIN LINE FORM 3 Ġif Ġevery Ġanswer Ġto ĠIN LINE FORM 4 Ġis Ġalso Ġa Ġcorrect Ġanswer Ġto ĠIN LINE FORM 5 ĠB IB REF 1 Ġ. </s> R Q E Ġis Ġparticularly Ġrelevant Ġdue Ġto Ġthe Ġincreasing Ġnumbers Ġof Ġsimilar Ġquestions Ġposted Ġonline ĠB IB REF 2 Ġand Ġits Ġability Ġto Ġsolve Ġdifferently Ġthe Ġchallenging Ġissues Ġof Ġquestion Ġunderstanding Ġand Ġanswer Ġextraction . ĠIn Ġaddition Ġto Ġbeing Ġused Ġto Ġfind Ġrelevant Ġanswers , Ġthese Ġresources Ġcan Ġalso Ġbe Ġused Ġin Ġtraining Ġmodels Ġable Ġto Ġrecognize Ġinference Ġrelations Ġand Ġsimilarity Ġbetween Ġquestions . </s> Question Ġsimilarity Ġhas Ġrecently Ġattracted Ġinternational Ġchallenges ĠB IB REF 3 Ġ, ĠB IB REF 4 Ġand Ġseveral Ġresearch Ġefforts Ġproposing Ġa Ġwide Ġrange Ġof Ġapproaches , Ġincluding ĠLog istic ĠReg ression , ĠRec urrent ĠNeural ĠNetworks Ġ( R NN s ), ĠLong ĠShort ĠTerm ĠMemory Ġcells Ġ( L ST Ms ), Ġand ĠConv olution al ĠNeural ĠNetworks Ġ( CNN s ) ĠB IB REF 5 Ġ, ĠB IB REF 6 Ġ, ĠB IB REF 1 Ġ, ĠB IB REF 7 Ġ. </s> In Ġthis Ġpaper , Ġwe Ġstudy Ġquestion Ġentail ment Ġin Ġthe Ġmedical Ġdomain Ġand Ġthe Ġeffectiveness Ġof Ġthe Ġend - to - end ĠR Q E - based ĠQ A Ġapproach Ġby Ġevaluating Ġthe Ġrelevance Ġof Ġthe Ġretrieved Ġanswers . ĠAlthough Ġentail ment Ġwas Ġattempted Ġin ĠQ A Ġbefore ĠB IB REF 8 Ġ, ĠB IB REF 9 Ġ, ĠB IB REF 10 Ġ, Ġas Ġfar Ġas Ġwe Ġknow , Ġwe Ġare Ġthe Ġfirst Ġto Ġintroduce Ġand Ġevaluate Ġa Ġfull Ġmedical Ġquestion Ġanswering Ġapproach Ġbased Ġon Ġquestion Ġentail ment Ġfor Ġfree - text Ġquestions . ĠOur Ġcontributions Ġare : </s> The Ġnext Ġsection Ġis Ġdedicated Ġto Ġrelated Ġwork Ġon Ġquestion Ġanswering , Ġquestion Ġsimilarity Ġand Ġentail ment . ĠIn ĠSection ĠSEC REF 3 Ġ, Ġwe Ġpresent Ġtwo Ġmachine Ġlearning Ġ( ML ) Ġand Ġdeep Ġlearning Ġ( DL ) Ġmethods Ġfor ĠR Q E Ġand Ġcompare Ġtheir Ġperformance Ġusing Ġopen - domain Ġand Ġclinical Ġdatasets . ĠSection ĠSEC REF 4 Ġdescribes Ġthe Ġnew Ġcollection Ġof Ġmedical Ġquestion - answer Ġpairs . ĠIn ĠSection ĠSEC REF 5 Ġ, Ġwe Ġdescribe Ġour ĠR Q E - based Ġapproach Ġfor ĠQ A . ĠSection ĠSEC REF 6 Ġpresents Ġour Ġevaluation Ġof Ġthe Ġretrieved Ġanswers Ġand Ġthe Ġresults Ġobtained Ġon ĠT REC Ġ2017 ĠLive Q A Ġmedical Ġquestions . </s> Background </s> In Ġthis Ġsection Ġwe Ġdefine Ġthe ĠR Q E Ġtask Ġand Ġdescribe Ġrelated Ġwork Ġat Ġthe Ġintersection Ġof Ġquestion Ġanswering , Ġquestion Ġsimilarity Ġand Ġtextual Ġinference . </s> Task ĠDefinition </s> The Ġdefinition Ġof ĠRecogn izing ĠQuestion ĠEnt ail ment Ġ( R Q E ) Ġcan Ġhave Ġa Ġsignificant Ġimpact Ġon ĠQ A Ġresults . ĠIn Ġrelated Ġwork , Ġthe Ġmeaning Ġassociated Ġwith ĠNatural ĠLanguage ĠIn ference Ġ( N LI ) Ġvaries Ġamong Ġdifferent Ġtasks Ġand Ġevents . ĠFor Ġinstance , ĠRecogn izing ĠText ual ĠEnt ail ment Ġ( R TE ) Ġwas Ġaddressed Ġby Ġthe ĠP ASC AL Ġchallenge ĠB IB REF 12 Ġ, Ġwhere Ġthe Ġentail ment Ġrelation Ġhas Ġbeen Ġassessed Ġmanually Ġby Ġhuman Ġjudges Ġwho Ġselected Ġrelevant Ġsentences Ġ" ent ailing " Ġa Ġset Ġof Ġhypotheses Ġfrom Ġa Ġlist Ġof Ġdocuments Ġreturned Ġby Ġdifferent ĠInformation ĠRet ri eval Ġ( IR ) Ġmethods . ĠIn Ġanother Ġdefinition , Ġthe ĠStanford ĠNatural ĠLanguage ĠIn ference Ġcorpus ĠSN LI ĠB IB REF 13 Ġ, Ġused Ġthree Ġclassification Ġlabels Ġfor Ġthe Ġrelations Ġbetween Ġtwo Ġsentences : Ġentail ment , Ġneutral Ġand Ġcontradiction . ĠFor Ġthe Ġentail ment Ġlabel , Ġthe Ġannot ators Ġwho Ġbuilt Ġthe Ġcorpus Ġwere Ġpresented Ġwith Ġan Ġimage Ġand Ġasked Ġto Ġwrite Ġa Ġcaption ĠâĢ ľ that Ġis Ġa Ġdefinitely Ġtrue Ġdescription Ġof Ġthe Ġphoto âĢ Ŀ . ĠFor Ġthe Ġneutral Ġlabel , Ġthey Ġwere Ġasked Ġto Ġprovide Ġa Ġcaption ĠâĢ ľ that Ġmight Ġbe Ġa Ġtrue Ġdescription Ġof Ġthe Ġlabel âĢ Ŀ . ĠThey Ġwere Ġasked Ġfor Ġa Ġcaption Ġthat ĠâĢ ľ is Ġdefinitely Ġa Ġfalse Ġdescription Ġof Ġthe Ġphoto âĢ Ŀ Ġfor Ġthe Ġcontradiction Ġlabel . </s> More Ġrecently , Ġthe Ġmulti N LI Ġcorpus ĠB IB REF 14 Ġwas Ġshared Ġin Ġthe Ġscope Ġof Ġthe ĠRep E val Ġ2017 Ġshared Ġtask ĠB IB REF 15 Ġ. ĠTo Ġbuild Ġthe Ġcorpus , Ġannot ators Ġwere Ġpresented Ġwith Ġa Ġpremise Ġtext Ġand Ġasked Ġto Ġwrite Ġthree Ġsentences . ĠOne Ġnovel Ġsentence , Ġwhich Ġis ĠâĢ ľ necess arily Ġtrue Ġor Ġappropriate Ġin Ġthe Ġsame Ġsituations Ġas Ġthe Ġpremise , âĢ Ŀ Ġfor Ġthe Ġentail ment Ġlabel , Ġa Ġsentence , Ġwhich Ġis ĠâĢ ľ necess arily Ġfalse Ġor Ġinappropriate Ġwhenever Ġthe Ġpremise Ġis Ġtrue , âĢ Ŀ Ġfor Ġthe Ġcontradiction Ġlabel , Ġand Ġa Ġlast Ġsentence , ĠâĢ ľ where Ġneither Ġcondition Ġapplies , âĢ Ŀ Ġfor Ġthe Ġneutral Ġlabel . </s> Whereas Ġthese ĠNL I Ġdefinitions Ġmight Ġbe Ġsuitable Ġfor Ġthe Ġbroad Ġtopic Ġof Ġtext Ġunderstanding , Ġtheir Ġrelation Ġto Ġpractical Ġinformation Ġretrieval Ġor Ġquestion Ġanswering Ġsystems Ġis Ġnot Ġstraightforward . </s> In Ġcontrast , ĠR Q E Ġhas Ġto Ġbe Ġtailored Ġto Ġthe Ġquestion Ġanswering Ġtask . ĠFor Ġinstance , Ġif Ġthe Ġpremise Ġquestion Ġis Ġ" looking Ġfor Ġcold Ġmedications Ġfor Ġa Ġ30 Ġyo Ġwoman ", Ġa ĠR Q E Ġapproach Ġshould Ġbe Ġable Ġto Ġconsider Ġthe Ġmore Ġgeneral Ġ( less Ġrestricted ) Ġquestion Ġ" looking Ġfor Ġcold Ġmedications " Ġas Ġrelevant , Ġsince Ġits Ġanswers Ġare Ġrelevant Ġfor Ġthe Ġinitial Ġquestion , Ġwhereas Ġ" looking Ġfor Ġmedications Ġfor Ġa Ġ30 Ġyo Ġwoman " Ġis Ġa Ġuseless Ġcontextual ization . ĠThe Ġentail ment Ġrelation Ġwe Ġare Ġseeking Ġin Ġthe ĠQ A Ġcontext Ġshould Ġinclude Ġrelevant Ġand Ġmeaningful Ġrelax ations Ġof Ġcontextual Ġand Ġsemantic Ġconstraints Ġ( cf . ĠSection ĠSEC REF 13 Ġ). </s> Related ĠWork Ġon ĠQuestion ĠAn sw ering </s> Class ical ĠQ A Ġsystems Ġface Ġtwo Ġmain Ġchallenges Ġrelated Ġto Ġquestion Ġanalysis Ġand Ġanswer Ġextraction . ĠSeveral ĠQ A Ġapproaches Ġwere Ġproposed Ġin Ġthe Ġliterature Ġfor Ġthe Ġopen Ġdomain ĠB IB REF 16 Ġ, ĠB IB REF 17 Ġand Ġthe Ġmedical Ġdomain ĠB IB REF 18 Ġ, ĠB IB REF 19 Ġ, ĠB IB REF 20 Ġ. ĠA Ġvariety Ġof Ġmethods Ġwere Ġdeveloped Ġfor Ġquestion Ġanalysis , Ġfocus Ġ( topic ) Ġrecognition Ġand Ġquestion Ġtype Ġidentification ĠB IB REF 21 Ġ, ĠB IB REF 22 Ġ, ĠB IB REF 23 Ġ, ĠB IB REF 24 Ġ. ĠSimilarly , Ġmany Ġdifferent Ġapproaches Ġtackled Ġdocument Ġor Ġpassage Ġretrieval Ġand Ġanswer Ġselection Ġand Ġ( re ) ranking ĠB IB REF 25 Ġ, ĠB IB REF 26 Ġ, ĠB IB REF 27 Ġ. </s> An Ġalternative Ġapproach Ġconsists Ġin Ġfinding Ġsimilar Ġquestions Ġor ĠFAQ s Ġthat Ġare Ġalready Ġanswered ĠB IB REF 28 Ġ, ĠB IB REF 29 Ġ. ĠOne Ġof Ġthe Ġearliest Ġquestion Ġanswering Ġsystems Ġbased Ġon Ġfinding Ġsimilar Ġquestions Ġand Ġre - using Ġthe Ġexisting Ġanswers Ġwas ĠFAQ ĠF IND ER ĠB IB REF 30 Ġ. ĠAnother Ġsystem Ġthat Ġcomple ments Ġthe Ġexisting ĠQ & A Ġservices Ġof ĠNet Well ness Ġis ĠSim Q ĠB IB REF 2 Ġ, Ġwhich Ġallows Ġretrieval Ġof Ġsimilar Ġweb - based Ġconsumer Ġhealth Ġquestions . ĠSim Q Ġuses Ġsynt actic Ġand Ġsemantic Ġfeatures Ġto Ġcompute Ġsimilarity Ġbetween Ġquestions , Ġand ĠU ML S ĠB IB REF 31 Ġas Ġa Ġstandardized Ġsemantic Ġknowledge Ġsource . ĠThe Ġsystem Ġachieves Ġ72 . 2 % Ġprecision , Ġ78 . 0 % Ġrecall Ġand Ġ75 . 0 % ĠF - score Ġon ĠNet Well ness Ġquestions . ĠHowever , Ġthe Ġmethod Ġwas Ġevaluated Ġonly Ġon Ġone Ġquestion Ġsimilarity Ġdataset , Ġand Ġthe Ġretrieved Ġanswers Ġwere Ġnot Ġevaluated . </s> The Ġaim Ġof Ġthe Ġmedical Ġtask Ġat ĠT REC Ġ2017 ĠLive Q A Ġwas Ġto Ġdevelop Ġtechniques Ġfor Ġanswering Ġcomplex Ġquestions Ġsuch Ġas Ġconsumer Ġhealth Ġquestions , Ġas Ġwell Ġas Ġto Ġidentify Ġrelevant Ġanswer Ġsources Ġthat Ġcan Ġcomply Ġwith Ġthe Ġsensitivity Ġof Ġmedical Ġinformation Ġretrieval . </s> The ĠCM U - OA Q A Ġsystem ĠB IB REF 32 Ġachieved Ġthe Ġbest Ġperformance Ġof Ġ0 . 6 37 Ġaverage Ġscore Ġon Ġthe Ġmedical Ġtask Ġby Ġusing Ġan Ġattention al Ġenc oder - dec oder Ġmodel Ġfor Ġparaph rase Ġidentification Ġand Ġanswer Ġranking . ĠThe ĠQu ora Ġquestion - similar ity Ġdataset Ġwas Ġused Ġfor Ġtraining . ĠThe ĠPR NA Ġsystem ĠB IB REF 33 Ġachieved Ġthe Ġsecond Ġbest Ġperformance Ġin Ġthe Ġmedical Ġtask Ġwith Ġ0 . 49 Ġaverage Ġscore Ġusing ĠWikipedia Ġas Ġthe Ġfirst Ġanswer Ġsource Ġand ĠYahoo Ġand ĠGoogle Ġsearches Ġas Ġsecondary Ġanswer Ġsources . ĠEach Ġmedical Ġquestion Ġwas Ġdecomp osed Ġinto Ġseveral Ġsub quest ions . ĠTo Ġextract Ġthe Ġanswer Ġfrom Ġthe Ġselected Ġtext Ġpassage , Ġa Ġbi - direction al Ġattention Ġmodel Ġtrained Ġon Ġthe ĠSQU AD Ġdataset Ġwas Ġused . </s> Deep Ġneural Ġnetwork Ġmodels Ġhave Ġbeen Ġpushing Ġthe Ġlimits Ġof Ġperformance Ġachieved Ġin ĠQ A Ġrelated Ġtasks Ġusing Ġlarge Ġtraining Ġdatasets . ĠThe Ġresults Ġobtained Ġby ĠCM U - OA Q A Ġand ĠPR NA Ġshowed Ġthat Ġlarge Ġopen - domain Ġdatasets Ġwere Ġbeneficial Ġfor Ġthe Ġmedical Ġdomain . ĠHowever , Ġthe Ġbest Ġsystem Ġ( CM U - OA Q A ) Ġrelying Ġon Ġthe Ġsame Ġtraining Ġdata Ġobtained Ġa Ġscore Ġof Ġ1 . 139 Ġon Ġthe ĠLive Q A Ġopen - domain Ġtask . </s> While Ġthis Ġgap Ġin Ġperformance Ġcan Ġbe Ġexplained Ġin Ġpart Ġby Ġthe Ġdiscrepancies Ġbetween Ġthe Ġmedical Ġtest Ġquestions Ġand Ġthe Ġopen - domain Ġquestions , Ġit Ġalso Ġhighlights Ġthe Ġneed Ġfor Ġlarger Ġmedical Ġdatasets Ġto Ġsupport Ġdeep Ġlearning Ġapproaches Ġin Ġdealing Ġwith Ġthe Ġlinguistic Ġcomplexity Ġof Ġconsumer Ġhealth Ġquestions Ġand Ġthe Ġchallenge Ġof Ġfinding Ġcorrect Ġand Ġcomplete Ġanswers . </s> Another Ġtechnique Ġwas Ġused Ġby ĠEC N U - ICA Ġteam ĠB IB REF 34 Ġbased Ġon Ġlearning Ġquestion Ġsimilarity Ġvia Ġtwo Ġlong Ġshort - term Ġmemory Ġ( L ST M ) Ġnetworks Ġapplied Ġto Ġobtain Ġthe Ġsemantic Ġrepresentations Ġof Ġthe Ġquestions . ĠTo Ġconstruct Ġa Ġcollection Ġof Ġsimilar Ġquestion Ġpairs , Ġthey Ġsearched Ġcommunity Ġquestion Ġanswering Ġsites Ġsuch Ġas ĠYahoo ! Ġand ĠAnswers . com . ĠIn Ġcontrast , Ġthe ĠEC N U - ICA Ġsystem Ġachieved Ġthe Ġbest Ġperformance Ġof Ġ1 . 8 95 Ġin Ġthe Ġopen - domain Ġtask Ġbut Ġan Ġaverage Ġscore Ġof Ġonly Ġ0 . 402 Ġin Ġthe Ġmedical Ġtask . ĠAs Ġthe ĠEC N U - ICA Ġapproach Ġalso Ġrelied Ġon Ġa Ġneural Ġnetwork Ġfor Ġquestion Ġmatching , Ġthis Ġresult Ġshows Ġthat Ġtraining Ġattention - based Ġdec oder - enc oder Ġnetworks Ġon Ġthe ĠQu ora Ġdataset Ġgeneralized Ġbetter Ġto Ġthe Ġmedical Ġdomain Ġthan Ġtraining ĠL ST Ms Ġon Ġsimilar Ġquestions Ġfrom ĠYahoo ! Ġand ĠAnswers . com . </s> The ĠCM U - Live Med Q A Ġteam ĠB IB REF 20 Ġdesigned Ġa Ġspecific Ġsystem Ġfor Ġthe Ġmedical Ġtask . ĠUsing Ġonly Ġthe Ġprovided Ġtraining Ġdatasets Ġand Ġthe Ġassumption Ġthat Ġeach Ġquestion Ġcontains Ġonly Ġone Ġfocus , Ġthe ĠCM U - Live Med Q A Ġsystem Ġobtained Ġan Ġaverage Ġscore Ġof Ġ0 . 353 . ĠThey Ġused Ġa Ġconv olution al Ġneural Ġnetwork Ġ( CNN ) Ġmodel Ġto Ġclassify Ġa Ġquestion Ġinto Ġa Ġrestricted Ġset Ġof Ġ10 Ġquestion Ġtypes Ġand Ġcrawled Ġ" relevant " Ġonline Ġweb Ġpages Ġto Ġfind Ġthe Ġanswers . ĠHowever , Ġthe Ġresults Ġwere Ġlower Ġthan Ġthose Ġachieved Ġby Ġthe Ġsystems Ġrelying Ġon Ġfinding Ġsimilar Ġanswered Ġquestions . ĠThese Ġresults Ġsupport Ġthe Ġrelevance Ġof Ġsimilar Ġquestion Ġmatching Ġfor Ġthe Ġend - to - end ĠQ A Ġtask Ġas Ġa Ġnew Ġway Ġof Ġapproaching ĠQ A Ġinstead Ġof Ġthe Ġclassical ĠQ A Ġapproaches Ġbased Ġon ĠQuestion ĠAnalysis Ġand ĠAnswer ĠRet ri eval . </s> Related ĠWork Ġon ĠQuestion ĠSimilar ity Ġand ĠEnt ail ment </s> Several Ġefforts Ġfocused Ġon Ġrecognizing Ġsimilar Ġquestions . ĠJe on Ġet Ġal . ĠB IB REF 35 Ġshowed Ġthat Ġa Ġretrieval Ġmodel Ġbased Ġon Ġtranslation Ġprobabilities Ġlearned Ġfrom Ġa Ġquestion Ġand Ġanswer Ġarchive Ġcan Ġrecognize Ġsem antically Ġsimilar Ġquestions . ĠD uan Ġet Ġal . ĠB IB REF 36 Ġproposed Ġa Ġdedicated Ġlanguage Ġmodeling Ġapproach Ġfor Ġquestion Ġsearch , Ġusing Ġquestion Ġtopic Ġ( user 's Ġinterest ) Ġand Ġquestion Ġfocus Ġ( certain Ġaspect Ġof Ġthe Ġtopic ). </s> L ately , Ġthese Ġefforts Ġwere Ġsupported Ġby Ġa Ġtask Ġon ĠQuestion - Question Ġsimilarity Ġintroduced Ġin Ġthe Ġcommunity ĠQ A Ġchallenge Ġat ĠSem E val Ġ( task Ġ3 B ) ĠB IB REF 3 Ġ. ĠGiven Ġa Ġnew Ġquestion , Ġthe Ġtask Ġfocused Ġon Ġre ranking Ġall Ġsimilar Ġquestions Ġretrieved Ġby Ġa Ġsearch Ġengine , Ġassuming Ġthat Ġthe Ġanswers Ġto Ġthe Ġsimilar Ġquestions Ġwill Ġbe Ġcorrect Ġanswers Ġfor Ġthe Ġnew Ġquestion . ĠDifferent Ġmachine Ġlearning Ġand Ġdeep Ġlearning Ġapproaches Ġwere Ġtested Ġin Ġthe Ġscope Ġof ĠSem E val Ġ2016 ĠB IB REF 3 Ġand Ġ2017 ĠB IB REF 4 Ġtask Ġ3 B . ĠThe Ġbest Ġperforming Ġsystem Ġin Ġ2017 Ġachieved Ġa ĠMAP Ġof Ġ47 . 22 % Ġusing Ġsupervised ĠLog istic ĠReg ression Ġthat Ġcombined Ġdifferent Ġun super vised Ġsimilarity Ġmeasures Ġsuch Ġas ĠCos ine Ġand ĠSoft - Cos ine ĠB IB REF 37 Ġ. ĠThe Ġsecond Ġbest Ġsystem Ġachieved Ġ46 . 93 % ĠMAP Ġwith Ġa Ġlearning - to - rank Ġmethod Ġusing ĠLog istic ĠReg ression Ġand Ġa Ġrich Ġset Ġof Ġfeatures Ġincluding Ġlex ical Ġand Ġsemantic Ġfeatures Ġas Ġwell Ġas Ġembed d ings Ġgenerated Ġby Ġdifferent Ġneural Ġnetworks Ġ( si ames e , ĠBi - L ST M , ĠGR U Ġand ĠCNN s ) ĠB IB REF 38 Ġ. ĠIn Ġthe Ġscope Ġof Ġthis Ġchallenge , Ġa Ġdataset Ġwas Ġcollected Ġfrom ĠQatar ĠLiving Ġforum Ġfor Ġtraining . ĠWe Ġrefer Ġto Ġthis Ġdataset Ġas ĠSem E val - c Q A . </s> In Ġanother Ġeffort , Ġan Ġanswer - based Ġdefinition Ġof ĠR Q E Ġwas Ġproposed Ġand Ġtested ĠB IB REF 1 Ġ. ĠThe Ġauthors Ġintroduced Ġa Ġdataset Ġof Ġclinical Ġquestions Ġand Ġused Ġa Ġfeature - based Ġmethod Ġthat Ġprovided Ġan ĠAccuracy Ġof Ġ75 % Ġon Ġconsumer Ġhealth Ġquestions . ĠWe Ġwill Ġcall Ġthis Ġdataset ĠClinical - Q E . ĠDos ĠSantos Ġet Ġal . ĠB IB REF 5 Ġproposed Ġa Ġnew Ġapproach Ġto Ġretrieve Ġsem antically Ġequivalent Ġquestions Ġcombining Ġa Ġbag - of - words Ġrepresentation Ġwith Ġa Ġdistributed Ġvector Ġrepresentation Ġcreated Ġby Ġa ĠCNN Ġand Ġuser Ġdata Ġcollected Ġfrom Ġtwo ĠStack ĠExchange Ġcommunities . ĠLei Ġet Ġal . ĠB IB REF 7 Ġproposed Ġa Ġrecurrent Ġand Ġconv olution al Ġmodel Ġ( g ated Ġconv olution ) Ġto Ġmap Ġquestions Ġto Ġtheir Ġsemantic Ġrepresentations . ĠThe Ġmodels Ġwere Ġpre - trained Ġwithin Ġan Ġenc oder - dec oder Ġframework . </s> R Q E ĠAppro aches Ġand ĠExper iments </s> The Ġchoice Ġof Ġtwo Ġmethods Ġfor Ġour Ġempirical Ġstudy Ġis Ġmotivated Ġby Ġthe Ġbest Ġperformance Ġachieved Ġby ĠLog istic ĠReg ression Ġin Ġquestion - question Ġsimilarity Ġat ĠSem E val Ġ2017 Ġ( best Ġsystem ĠB IB REF 37 Ġand Ġsecond Ġbest Ġsystem ĠB IB REF 38 Ġ), Ġand Ġthe Ġhigh Ġperformance Ġachieved Ġby Ġneural Ġnetworks Ġon Ġlarger Ġdatasets Ġsuch Ġas ĠSN LI ĠB IB REF 13 Ġ, ĠB IB REF 39 Ġ, ĠB IB REF 40 Ġ, ĠB IB REF 41 Ġ. ĠWe Ġfirst Ġdefine Ġthe ĠR Q E Ġtask , Ġthen Ġpresent Ġthe Ġtwo Ġapproaches , Ġand Ġevaluate Ġtheir Ġperformance Ġon Ġfive Ġdifferent Ġdatasets . </s> Definition </s> In Ġthe Ġcontext Ġof ĠQ A , Ġthe Ġgoal Ġof ĠR Q E Ġis Ġto Ġretrieve Ġanswers Ġto Ġa Ġnew Ġquestion Ġby Ġretrieving Ġent ailed Ġquestions Ġwith Ġassociated Ġanswers . ĠWe Ġtherefore Ġdefine Ġquestion Ġentail ment Ġas : </s> a Ġquestion ĠIN LINE FORM 0 Ġentails Ġa Ġquestion ĠIN LINE FORM 1 Ġif Ġevery Ġanswer Ġto ĠIN LINE FORM 2 Ġis Ġalso Ġa Ġcomplete Ġor Ġpartial Ġanswer Ġto ĠIN LINE FORM 3 Ġ. </s> We Ġpresent Ġbelow Ġtwo Ġexamples Ġof Ġconsumer Ġhealth Ġquestions ĠIN LINE FORM 0 Ġand Ġent ailed Ġquestions ĠIN LINE FORM 1 Ġ: </s> Example Ġ1 Ġ( each Ġanswer Ġto Ġthe Ġent ailed Ġquestion ĠB 1 Ġis Ġa Ġcomplete Ġanswer Ġto ĠA 1 ): </s> A 1 : ĠWhat Ġis Ġthe Ġlatest Ġnews Ġon Ġt enn itis , Ġor Ġringing Ġin Ġthe Ġear , ĠI Ġam Ġ75 Ġyears Ġold Ġand Ġhave Ġhad Ġringing Ġin Ġthe Ġear Ġsince Ġmy Ġmid Ġ5 os . ĠThank Ġyou . </s> B 1 : ĠWhat Ġis Ġthe Ġlatest Ġresearch Ġon ĠT inn itus ? </s> Example Ġ2 Ġ( each Ġanswer Ġto Ġthe Ġent ailed Ġquestion ĠB 2 Ġis Ġa Ġpartial Ġanswer Ġto ĠA 2 ): </s> A 2 : ĠMy Ġmother Ġhas Ġbeen Ġdiagnosed Ġwith ĠAlzheimer 's , Ġmy Ġfather Ġis Ġnot Ġof Ġthe Ġgreatest Ġhealth Ġeither Ġand Ġis Ġthe Ġmain Ġcareg iver Ġfor Ġmy Ġmother . ĠMy Ġquestion Ġis Ġwhere Ġdo Ġwe Ġstart Ġwith Ġattempting Ġto Ġhelp Ġour Ġparents Ġw / Ġthe Ġcare Ġgiving Ġand Ġwhat Ġsort Ġof Ġfinancial Ġoptions Ġare Ġthere Ġout Ġthere Ġfor Ġpeople Ġon Ġfixed Ġincomes . </s> B 2 : ĠWhat Ġresources Ġare Ġavailable Ġfor ĠAlzheimer 's Ġcaregivers ? </s> The Ġinclusion Ġof Ġpartial Ġanswers Ġin Ġthe Ġdefinition Ġof Ġquestion Ġentail ment Ġalso Ġallows Ġefficient Ġrelaxation Ġof Ġthe Ġcontextual Ġconstraints Ġof Ġthe Ġoriginal Ġquestion ĠIN LINE FORM 0 Ġto Ġretrieve Ġrelevant Ġanswers Ġfrom Ġent ailed , Ġbut Ġless Ġrestricted , Ġquestions . </s> Deep ĠLearning ĠModel </s> To Ġrecognize Ġentail ment Ġbetween Ġtwo Ġquestions ĠIN LINE FORM 0 Ġ( prem ise ) Ġand ĠIN LINE FORM 1 Ġ( hyp ot hesis ), Ġwe Ġadapted Ġthe Ġneural Ġnetwork Ġproposed Ġby ĠBowman Ġet Ġal . ĠB IB REF 13 Ġ. ĠOur ĠDL Ġmodel , Ġpresented Ġin ĠFigure ĠFIG REF 20 Ġ, Ġconsists Ġof Ġthree Ġ600 d ĠRe LU Ġlayers , Ġwith Ġa Ġbottom Ġlayer Ġtaking Ġthe Ġconc aten ated Ġsentence Ġrepresentations Ġas Ġinput Ġand Ġa Ġtop Ġlayer Ġfeeding Ġa Ġsoft max Ġclass ifier . ĠThe Ġsentence Ġembed ding Ġmodel Ġsums Ġthe ĠRec urrent Ġneural Ġnetwork Ġ( R NN ) Ġembed d ings Ġof Ġits Ġwords . ĠThe Ġword Ġembed d ings Ġare Ġfirst Ġinitialized Ġwith Ġpret rained ĠGl o Ve Ġvectors . ĠThis Ġadaptation Ġprovided Ġthe Ġbest Ġperformance Ġin Ġprevious Ġexperiments Ġwith ĠR Q E Ġdata . </s> G lo Ve Ġis Ġan Ġun super vised Ġlearning Ġalgorithm Ġto Ġgenerate Ġvector Ġrepresentations Ġfor Ġwords ĠB IB REF 42 Ġ. ĠTraining Ġis Ġperformed Ġon Ġaggreg ated Ġword Ġco - occ urrence Ġstatistics Ġfrom Ġa Ġlarge Ġcorpus , Ġand Ġthe Ġresulting Ġrepresentations Ġshow Ġinteresting Ġlinear Ġsubst ruct ures Ġof Ġthe Ġword Ġvector Ġspace . ĠWe Ġuse Ġthe Ġpret rained Ġcommon Ġcrawl Ġversion Ġwith Ġ840 B Ġtokens Ġand Ġ300 d Ġvectors , Ġwhich Ġare Ġnot Ġupdated Ġduring Ġtraining . </s> Log istic ĠReg ression ĠClass ifier </s> In Ġthis Ġfeature - based Ġapproach , Ġwe Ġuse ĠLog istic ĠReg ression Ġto Ġclassify Ġquestion Ġpairs Ġinto Ġentail ment Ġor Ġno - ent ail ment . ĠLog istic ĠReg ression Ġachieved Ġgood Ġresults Ġon Ġthis Ġspecific Ġtask Ġand Ġoutper formed Ġother Ġstatistical Ġlearning Ġalgorithms Ġsuch Ġas ĠS VM Ġand ĠNa ive ĠBay es . ĠIn Ġa Ġpre processing Ġstep , Ġwe Ġremove Ġstop Ġwords Ġand Ġperform Ġword Ġstemming Ġusing Ġthe ĠPorter Ġalgorithm ĠB IB REF 43 Ġfor Ġall Ġ( ĠIN LINE FORM 0 Ġ, ĠIN LINE FORM 1 Ġ) Ġpairs . </s> We Ġuse Ġa Ġlist Ġof Ġnine Ġfeatures , Ġselected Ġafter Ġseveral Ġexperiments Ġon ĠR TE Ġdatasets ĠB IB REF 12 Ġ. ĠWe Ġcompute Ġfive Ġsimilarity Ġmeasures Ġbetween Ġthe Ġpre - process ed Ġquestions Ġand Ġuse Ġtheir Ġvalues Ġas Ġfeatures . ĠWe Ġuse ĠWord ĠOver lap , Ġthe ĠDice Ġcoefficient Ġbased Ġon Ġthe Ġnumber Ġof Ġcommon Ġbig rams , ĠCos ine , ĠLe ven sh tein , Ġand Ġthe ĠJ acc ard Ġsimilarities . ĠOur Ġfeature Ġlist Ġalso Ġincludes Ġthe Ġmaximum Ġand Ġaverage Ġvalues Ġobtained Ġwith Ġthese Ġmeasures Ġand Ġthe Ġquestion Ġlength Ġratio Ġ( length ( ĠIN LINE FORM 0 Ġ) / length ( ĠIN LINE FORM 1 Ġ) ). ĠWe Ġcompute Ġa Ġmorph os ynt actic Ġfeature Ġindicating Ġthe Ġnumber Ġof Ġcommon Ġnoun s Ġand Ġverbs Ġbetween ĠIN LINE FORM 2 Ġand ĠIN LINE FORM 3 Ġ. ĠTree T agger ĠB IB REF 44 Ġwas Ġused Ġfor ĠPOS Ġtagging . </s> For ĠR Q E , Ġwe Ġadd Ġan Ġadditional Ġfeature Ġspecific Ġto Ġthe Ġquestion Ġtype . ĠWe Ġuse Ġa Ġdictionary Ġlookup Ġto Ġmap Ġtriggers Ġto Ġthe Ġquestion Ġtype Ġ( e . g . ĠTreatment , ĠPro gn osis , ĠInher itance ). ĠTr ig gers Ġare Ġidentified Ġfor Ġeach Ġquestion Ġtype Ġbased Ġon Ġa Ġmanual Ġannotation Ġof Ġa Ġset Ġof Ġmedical Ġquestions Ġ( cf . ĠSection ĠSEC REF 36 Ġ). ĠThis Ġfeature Ġhas Ġthree Ġpossible Ġvalues : Ġ2 Ġ( Perfect Ġmatch Ġbetween ĠIN LINE FORM 0 Ġtype ( s ) Ġand ĠIN LINE FORM 1 Ġtype ( s )), Ġ1 Ġ( Over lap Ġbetween ĠIN LINE FORM 2 Ġtype ( s ) Ġand ĠIN LINE FORM 3 Ġtype ( s )) Ġand Ġ0 Ġ( No Ġcommon Ġtypes ). </s> Dat as ets ĠUsed Ġfor Ġthe ĠR Q E ĠStudy </s> We Ġevaluate Ġthe ĠR Q E Ġmethods Ġ( i . e . Ġdeep Ġlearning Ġmodel Ġand Ġlog istic Ġregression Ġclass ifier ) Ġusing Ġtwo Ġdatasets Ġof Ġsentence Ġpairs Ġ( SN LI Ġand Ġmulti N LI ), Ġand Ġthree Ġdatasets Ġof Ġquestion Ġpairs Ġ( Qu ora , ĠClinical - Q E , Ġand ĠSem E val - c Q A ). </s> The ĠStanford ĠNatural ĠLanguage ĠIn ference Ġcorpus Ġ( SN LI ) ĠB IB REF 13 Ġcontains Ġ5 69 , 0 37 Ġsentence Ġpairs Ġwritten Ġby Ġhumans Ġbased Ġon Ġimage Ġcaption ing . ĠThe Ġtraining Ġset Ġof Ġthe ĠMulti N LI Ġcorpus ĠB IB REF 14 Ġconsists Ġof Ġ39 3 , 000 Ġpairs Ġof Ġsentences Ġfrom Ġfive Ġgenres Ġof Ġwritten Ġand Ġspoken ĠEnglish Ġ( e . g . ĠTravel , ĠGovernment ). ĠTwo Ġother Ġ" matched " Ġand Ġ" m ism atched " Ġsets Ġare Ġalso Ġavailable Ġfor Ġdevelopment Ġ( 20 , 000 Ġpairs ). ĠBoth ĠSN LI Ġand Ġmulti N LI Ġconsider Ġthree Ġtypes Ġof Ġrelationships Ġbetween Ġsentences : Ġentail ment , Ġneutral Ġand Ġcontradiction . ĠWe Ġconverted Ġthe Ġcontradiction Ġand Ġneutral Ġlabels Ġto Ġthe Ġsame Ġnon - ent ail ment Ġclass . </s> The ĠQU OR A Ġdataset Ġof Ġsimilar Ġquestions Ġwas Ġrecently Ġpublished Ġwith Ġ404 , 279 Ġquestion Ġpairs . ĠWe Ġrandomly Ġselected Ġthree Ġdistinct Ġsubs ets Ġ( 80 % / 10 % / 10 %) Ġfor Ġtraining Ġ( 323 , 423 Ġpairs ), Ġdevelopment Ġ( 40 , 428 Ġpairs ) Ġand Ġtest Ġ( 40 , 428 Ġpairs ). </s> The Ġclinical - Q E Ġdataset ĠB IB REF 1 Ġcontains Ġ8 , 588 Ġquestion Ġpairs Ġand Ġwas Ġconstructed Ġusing Ġ4 , 655 Ġclinical Ġquestions Ġasked Ġby Ġfamily Ġdoctors ĠB IB REF 45 Ġ. ĠWe Ġrandomly Ġselected Ġthree Ġdistinct Ġsubs ets Ġ( 80 % / 10 % / 10 %) Ġfor Ġtraining Ġ( 6 , 870 Ġpairs ), Ġdevelopment Ġ( 8 59 Ġpairs ) Ġand Ġtest Ġ( 8 59 Ġpairs ). </s> The Ġquestion Ġsimilarity Ġdataset Ġof ĠSem E val Ġ2016 ĠTask Ġ3 B Ġ( Sem E val - c Q A ) ĠB IB REF 3 Ġcontains Ġ3 , 8 69 Ġquestion Ġpairs Ġand Ġaims Ġto Ġre - rank Ġa Ġlist Ġof Ġrelated Ġquestions Ġaccording Ġto Ġtheir Ġsimilarity Ġto Ġthe Ġoriginal Ġquestion . ĠThe Ġsame Ġdataset Ġwas Ġused Ġfor ĠSem E val Ġ2017 ĠTask Ġ3 ĠB IB REF 4 Ġ. </s> To Ġconstruct Ġour Ġtest Ġdataset , Ġwe Ġused Ġa Ġpublicly Ġshared Ġset Ġof ĠConsumer ĠHealth ĠQuestions Ġ( CHQ s ) Ġreceived Ġby Ġthe ĠU . S . ĠNational ĠLibrary Ġof ĠMedicine Ġ( N LM ), Ġand Ġannot ated Ġwith Ġnamed Ġentities , Ġquestion Ġtypes , Ġand Ġfocus ĠB IB REF 46 Ġ, ĠB IB REF 47 Ġ. ĠThe ĠCH Q Ġdataset Ġconsists Ġof Ġ1 , 7 21 Ġconsumer Ġinformation Ġrequests Ġmanually Ġannot ated Ġwith Ġsub quest ions , Ġeach Ġidentified Ġby Ġa Ġquestion Ġtype Ġand Ġa Ġfocus . </s> First , Ġwe Ġselected Ġautomatically Ġharvested ĠFAQ s , Ġfrom ĠU . S . ĠNational ĠInstitutes Ġof ĠHealth Ġ( NI H ) Ġwebsites , Ġthat Ġshare Ġboth Ġthe Ġsame Ġfocus Ġand Ġthe Ġsame Ġquestion Ġtype Ġwith Ġthe ĠCH Q s . ĠAs ĠFAQ s Ġare Ġmost Ġoften Ġvery Ġshort , Ġwe Ġfirst Ġassume Ġthat Ġthe ĠCH Q Ġentails Ġthe ĠFAQ . ĠTwo Ġsets Ġof Ġpairs Ġwere Ġconstructed : Ġ( i ) Ġpositive Ġpairs Ġof ĠCH Q s Ġand ĠFAQ s Ġsharing Ġat Ġleast Ġone Ġcommon Ġquestion Ġtype Ġand Ġthe Ġquestion Ġfocus , Ġand Ġ( ii ) Ġnegative Ġpairs Ġcorresponding Ġto Ġa Ġfocus Ġmismatch Ġor Ġtype Ġmismatch . ĠFor Ġeach Ġcategory Ġof Ġnegative Ġexamples , Ġwe Ġrandomly Ġselected Ġthe Ġsame Ġnumber Ġof Ġpairs Ġfor Ġa Ġbalanced Ġdataset . ĠThen , Ġwe Ġmanually Ġvalidated Ġthe Ġconstructed Ġpairs Ġand Ġcorrected Ġthe Ġpositive Ġand Ġnegative Ġlabels Ġwhen Ġneeded . ĠThe Ġfinal ĠR Q E Ġdataset Ġcontains Ġ850 ĠCH Q - FAQ Ġpairs Ġwith Ġ405 Ġpositive Ġand Ġ445 Ġnegative Ġpairs . ĠTable ĠT AB REF 26 Ġpresents Ġexamples Ġfrom Ġthe Ġfive Ġtraining Ġdatasets Ġ( SN LI , ĠMulti N LI , ĠSem E val - c Q A , ĠClinical - Q E Ġand ĠQu ora ) Ġand Ġthe Ġnew Ġtest Ġdataset Ġof Ġmedical ĠCH Q - FAQ Ġpairs . </s> Results Ġof ĠR Q E ĠAppro aches </s> In Ġthe Ġfirst Ġexperiment , Ġwe Ġevaluated Ġthe ĠDL Ġand ĠML Ġmethods Ġon ĠSN LI , Ġmulti - N LI , ĠQu ora , Ġand ĠClinical - Q E . ĠFor Ġthe Ġdatasets Ġthat Ġdid Ġnot Ġhave Ġa Ġdevelopment Ġand Ġtest Ġsets , Ġwe Ġrandomly Ġselected Ġtwo Ġsets , Ġeach Ġamount ing Ġto Ġ10 % Ġof Ġthe Ġdata , Ġfor Ġtest Ġand Ġdevelopment , Ġand Ġused Ġthe Ġremaining Ġ80 % Ġfor Ġtraining . ĠFor ĠMulti N LI , Ġwe Ġused Ġthe Ġdev 1 - matched Ġset Ġfor Ġvalidation Ġand Ġthe Ġdev 2 - m ism atched Ġset Ġfor Ġtesting . </s> Table ĠT AB REF 28 Ġpresents Ġthe Ġresults Ġof Ġthe Ġfirst Ġexperiment . ĠThe ĠDL Ġmodel Ġwith ĠGl o Ve Ġword Ġembed d ings Ġachieved Ġbetter Ġresults Ġon Ġthree Ġdatasets , Ġwith Ġ82 . 80 % ĠAccuracy Ġon ĠSN LI , Ġ78 . 52 % ĠAccuracy Ġon ĠMulti N LI , Ġand Ġ83 . 62 % ĠAccuracy Ġon ĠQu ora . ĠLog istic ĠReg ression Ġachieved Ġthe Ġbest ĠAccuracy Ġof Ġ98 . 60 % Ġon ĠClinical - R Q E . ĠWe Ġalso Ġperformed Ġa Ġ10 - fold Ġcross - valid ation Ġon Ġthe Ġfull ĠClinical - Q E Ġdata Ġof Ġ8 , 588 Ġquestion Ġpairs , Ġwhich Ġgave Ġ98 . 61 % ĠAccuracy . </s> In Ġthe Ġsecond Ġexperiment , Ġwe Ġused Ġthese Ġdatasets Ġfor Ġtraining Ġonly Ġand Ġcompared Ġtheir Ġperformance Ġon Ġour Ġtest Ġset Ġof Ġ850 Ġconsumer Ġhealth Ġquestions . ĠTable ĠT AB REF 29 Ġpresents Ġthe Ġresults Ġof Ġthis Ġexperiment . ĠLog istic ĠReg ression Ġtrained Ġon Ġthe Ġclinical - R Q E Ġdata Ġoutper formed ĠDL Ġmodels Ġtrained Ġon Ġall Ġdatasets , Ġwith Ġ73 . 18 % ĠAccuracy . </s> To Ġvalidate Ġfurther Ġthe Ġperformance Ġof Ġthe ĠLR Ġmethod , Ġwe Ġevaluated Ġit Ġon Ġquestion Ġsimilarity Ġdetection . ĠA Ġtypical Ġapproach Ġto Ġthis Ġtask Ġis Ġto Ġuse Ġan ĠIR Ġmethod Ġto Ġfind Ġsimilar Ġquestion Ġcandidates , Ġthen Ġa Ġmore Ġsophisticated Ġmethod Ġto Ġselect Ġand Ġre - rank Ġthe Ġsimilar Ġquestions . ĠWe Ġfollowed Ġa Ġsimilar Ġapproach Ġfor Ġthis Ġevaluation Ġby Ġcombining Ġthe ĠLR Ġmethod Ġwith Ġthe ĠIR Ġbaseline Ġprovided Ġin Ġthe Ġcontext Ġof ĠSem E val - c Q A . ĠThe Ġhybrid Ġmethod Ġcombines Ġthe Ġscore Ġprovided Ġby Ġthe ĠLog istic ĠReg ression Ġmodel Ġand Ġthe Ġreciprocal Ġrank Ġfrom Ġthe ĠIR Ġbaseline Ġusing Ġa Ġweight - based Ġcombination : </s> IN LINE FORM 0 </s> The Ġweight ĠIN LINE FORM 0 Ġwas Ġset Ġempir ically Ġthrough Ġseveral Ġtests Ġon Ġthe Ġc Q A - 2016 Ġdevelopment Ġset Ġ( ĠIN LINE FORM 1 Ġ). ĠTable ĠT AB REF 30 Ġpresents Ġthe Ġresults Ġon Ġthe Ġc Q A - 2016 Ġand Ġc Q A - 2017 Ġtest Ġdatasets . ĠThe Ġhybrid Ġmethod Ġ( LR + IR ) Ġprovided Ġthe Ġbest Ġresults Ġon Ġboth Ġdatasets . ĠOn Ġthe Ġ2016 Ġtest Ġdata , Ġthe ĠLR + IR Ġmethod Ġoutper formed Ġthe Ġbest Ġsystem Ġin Ġall Ġmeasures , Ġwith Ġ80 . 57 % ĠAccuracy Ġand Ġ77 . 47 % ĠMAP Ġ( official Ġsystem Ġranking Ġmeasure Ġin ĠSem E val - c Q A ). ĠOn Ġthe Ġc Q A - 2017 Ġtest Ġdata , Ġthe ĠLR + IR Ġmethod Ġobtained Ġ44 . 66 % ĠMAP Ġand Ġoutper formed Ġthe Ġc Q A - 2017 Ġbest Ġsystem Ġin ĠAccuracy Ġwith Ġ67 . 27 %. </s> Discussion Ġof ĠR Q E ĠResults </s> When Ġtrained Ġand Ġtested Ġon Ġthe Ġsame Ġcorpus , Ġthe ĠDL Ġmodel Ġwith ĠGl o Ve Ġembed d ings Ġgave Ġthe Ġbest Ġresults Ġon Ġthree Ġdatasets Ġ( SN LI , ĠMulti N LI Ġand ĠQu ora ). ĠLog istic ĠReg ression Ġgave Ġthe Ġbest ĠAccuracy Ġon Ġthe ĠClinical - R Q E Ġdataset Ġwith Ġ98 . 60 %. ĠWhen Ġtested Ġon Ġour Ġtest Ġset Ġ( 850 Ġmedical ĠCH Q s - FAQ s Ġpairs ), ĠLog istic ĠReg ression Ġtrained Ġon ĠClinical - Q E Ġgave Ġthe Ġbest Ġperformance Ġwith Ġ73 . 18 % ĠAccuracy . </s> The ĠSN LI Ġand Ġmulti - N LI Ġmodels Ġdid Ġnot Ġperform Ġwell Ġwhen Ġtested Ġon Ġmedical ĠR Q E Ġdata . ĠWe Ġperformed Ġadditional Ġevaluations Ġusing Ġthe ĠR TE - 1 , ĠR TE - 2 Ġand ĠR TE - 3 Ġopen - domain Ġdatasets Ġprovided Ġby Ġthe ĠP ASC AL Ġchallenge Ġand Ġthe Ġresults Ġwere Ġsimilar . ĠWe Ġhave Ġalso Ġtested Ġthe ĠSem E val - c Q A - 2016 Ġmodel Ġand Ġhad Ġa Ġsimilar Ġdrop Ġin Ġperformance Ġon ĠR Q E Ġdata . ĠThis Ġcould Ġbe Ġexplained Ġby Ġthe Ġdifferent Ġtypes Ġof Ġdata Ġleading Ġto Ġwrong Ġinternal Ġconceptual izations Ġof Ġmedical Ġterms Ġand Ġquestions Ġin Ġthe Ġdeep Ġneural Ġlayers . ĠThis Ġperformance Ġdrop Ġcould Ġalso Ġbe Ġcaused Ġby Ġthe Ġcomplexity Ġof Ġthe Ġtest Ġconsumer Ġhealth Ġquestions Ġthat Ġare Ġoften Ġcomposed Ġof Ġseveral Ġsub quest ions , Ġcontain Ġcontextual Ġinformation , Ġand Ġmay Ġcontain Ġmiss pell ings Ġand Ġun gram matical Ġsentences , Ġwhich Ġmakes Ġthem Ġmore Ġdifficult Ġto Ġprocess ĠB IB REF 48 Ġ. ĠAnother Ġaspect Ġis Ġthe Ġsemantics Ġof Ġthe Ġtask Ġas Ġdiscussed Ġin ĠSection ĠSEC REF 6 Ġ. ĠThe Ġdefinition Ġof Ġtextual Ġentail ment Ġin Ġopen - domain Ġmay Ġnot Ġquite Ġapply Ġto Ġquestion Ġentail ment Ġdue Ġto Ġthe Ġstrict Ġsemantics . ĠAlso Ġthe Ġgeneral Ġtextual Ġentail ment Ġdefinitions Ġrefer Ġonly Ġto Ġthe Ġpremise Ġand Ġhypothesis , Ġwhile Ġthe Ġdefinition Ġof ĠR Q E Ġfor Ġquestion Ġanswering Ġrelies Ġon Ġthe Ġrelationship Ġbetween Ġthe Ġsets Ġof Ġanswers Ġof Ġthe Ġcompared Ġquestions . </s> Building Ġa ĠMedical ĠQ A ĠCollection Ġfrom ĠTr usted ĠResources </s> A ĠR Q E - based ĠQ A Ġsystem Ġrequires Ġa Ġcollection Ġof Ġquestion - answer Ġpairs Ġto Ġmap Ġnew Ġuser Ġquestions Ġto Ġthe Ġexisting Ġquestions Ġwith Ġan ĠR Q E Ġapproach , Ġrank Ġthe Ġretrieved Ġquestions , Ġand Ġpresent Ġtheir Ġanswers Ġto Ġthe Ġuser . </s> Method </s> To Ġconstruct Ġtrusted Ġmedical Ġquestion - answer Ġpairs , Ġwe Ġcrawled Ġwebsites Ġfrom Ġthe ĠNational ĠInstitutes Ġof ĠHealth Ġ( cf . ĠSection ĠSEC REF 56 Ġ). ĠEach Ġweb Ġpage Ġdescribes Ġa Ġspecific Ġtopic Ġ( e . g . Ġname Ġof Ġa Ġdisease Ġor Ġa Ġdrug ), Ġand Ġoften Ġincludes Ġsyn onyms Ġof Ġthe Ġmain Ġtopic Ġthat Ġwe Ġextracted Ġduring Ġthe Ġcrawl . </s> We Ġconstructed Ġhand - crafted Ġpatterns Ġfor Ġeach Ġwebsite Ġto Ġautomatically Ġgenerate Ġthe Ġquestion - answer Ġpairs Ġbased Ġon Ġthe Ġdocument Ġstructure Ġand Ġthe Ġsection Ġtitles . ĠWe Ġalso Ġannot ated Ġeach Ġquestion Ġwith Ġthe Ġassociated Ġfocus Ġ( topic Ġof Ġthe Ġweb Ġpage ) Ġas Ġwell Ġas Ġthe Ġquestion Ġtype Ġidentified Ġwith Ġthe Ġdesigned Ġpatterns Ġ( cf . ĠSection ĠSEC REF 36 Ġ). </s> To Ġprovide Ġadditional Ġinformation Ġabout Ġthe Ġquestions Ġthat Ġcould Ġbe Ġused Ġfor Ġdiverse ĠIR Ġand ĠN LP Ġtasks , Ġwe Ġautomatically Ġannot ated Ġthe Ġquestions Ġwith Ġthe Ġfocus , Ġits ĠU ML S ĠConcept ĠUnique ĠIdent ifier Ġ( C UI ) Ġand ĠSem antic ĠType . ĠWe Ġcombined Ġtwo Ġmethods Ġto Ġrecognize Ġnamed Ġentities Ġfrom Ġthe Ġtitles Ġof Ġthe Ġcrawled Ġarticles Ġand Ġtheir Ġassociated ĠU ML S ĠCU Is : Ġ( i ) Ġexact Ġstring Ġmatching Ġto Ġthe ĠU ML S ĠMet at hes aurus , Ġand Ġ( ii ) ĠMeta Map ĠLite ĠB IB REF 49 Ġ. ĠWe Ġthen Ġused Ġthe ĠU ML S ĠSem antic ĠNetwork Ġto Ġretrieve Ġthe Ġassociated Ġsemantic Ġtypes Ġand Ġgroups . </s> Question ĠTypes </s> The Ġquestion Ġtypes Ġwere Ġderived Ġafter Ġthe Ġmanual Ġevaluation Ġof Ġ1 , 7 21 Ġconsumer Ġhealth Ġquestions . ĠOur Ġtax onomy Ġincludes Ġ16 Ġtypes Ġabout ĠDiseases , Ġ20 Ġtypes Ġabout ĠDrugs Ġand Ġone Ġtype Ġ( Information ) Ġfor Ġthe Ġother Ġnamed Ġentities Ġsuch Ġas ĠProcedures , ĠMedical Ġexams Ġand ĠTreat ments . ĠWe Ġdescribe Ġbelow Ġthe Ġconsidered Ġquestion Ġtypes Ġand Ġexamples Ġof Ġassociated Ġquestion Ġpatterns . </s> Question ĠTypes Ġabout ĠDiseases Ġ( 16 ): ĠInformation , ĠResearch Ġ( or ĠClinical ĠTrial ), ĠCauses , ĠTreatment , ĠPrevention , ĠDiagn osis Ġ( Ex ams Ġand ĠTests ), ĠPro gn osis , ĠCompl ications , ĠSymptoms , ĠInher itance , ĠSus cept ibility , ĠGenetic Ġchanges , ĠFrequency , ĠConsider ations , ĠContact Ġa Ġmedical Ġprofessional , ĠSupport ĠGroups . </s> Examples : </s> What Ġresearch Ġ( or Ġclinical Ġtrial ) Ġis Ġbeing Ġdone Ġfor ĠDIS E ASE ? </s> What Ġis Ġthe Ġoutlook Ġfor ĠDIS E ASE ? </s> How Ġmany Ġpeople Ġare Ġaffected Ġby ĠDIS E ASE ? </s> When Ġto Ġcontact Ġa Ġmedical Ġprofessional Ġabout ĠDIS E ASE ? </s> Who Ġis Ġat Ġrisk Ġfor ĠDIS E ASE ? </s> Where Ġto Ġfind Ġsupport Ġfor Ġpeople Ġwith ĠDIS E ASE ? </s> Question ĠTypes ĠAbout ĠDrugs Ġ( 20 ): ĠInformation , ĠInter action Ġwith Ġmedications , ĠInter action Ġwith Ġfood , ĠInter action Ġwith Ġherbs Ġand Ġsupplements , ĠImportant Ġwarning , ĠSpecial Ġinstructions , ĠBrand Ġnames , ĠHow Ġdoes Ġit Ġwork , ĠHow Ġeffective Ġis Ġit , ĠInd ication , ĠCont rain d ication , ĠLearn Ġmore , ĠSide Ġeffects , ĠEmergency Ġor Ġoverdose , ĠSe vere Ġreaction , ĠForget Ġa Ġdose , ĠDietary , ĠWhy Ġget Ġvaccinated , ĠStorage Ġand Ġdisposal , ĠUsage , ĠD ose . </s> Examples : </s> Are Ġthere Ġinteractions Ġbetween ĠDR UG Ġand Ġherbs Ġand Ġsupplements ? </s> What Ġimportant Ġwarning Ġor Ġinformation Ġshould ĠI Ġknow Ġabout ĠDR UG ? </s> Are Ġthere Ġsafety Ġconcerns Ġor Ġspecial Ġprecautions Ġabout ĠDR UG ? </s> What Ġis Ġthe Ġaction Ġof ĠDR UG Ġand Ġhow Ġdoes Ġit Ġwork ? </s> Who Ġshould Ġget ĠDR UG Ġand Ġwhy Ġis Ġit Ġprescribed ? </s> What Ġto Ġdo Ġin Ġcase Ġof Ġa Ġsevere Ġreaction Ġto ĠDR UG ? </s> Question ĠType Ġfor Ġother Ġmedical Ġentities Ġ( e . g . ĠProcedure , ĠExam , ĠTreatment ): ĠInformation . </s> What Ġis ĠCor on ary ĠAr tery ĠBy pass ĠSurgery ? </s> What Ġare ĠLiver ĠFunction ĠTests ? </s> Medical ĠResources </s> We Ġused Ġ12 Ġtrusted Ġwebsites Ġto Ġconstruct Ġa Ġcollection Ġof Ġquestion - answer Ġpairs . ĠFor Ġeach Ġwebsite , Ġwe Ġextracted Ġthe Ġfree Ġtext Ġof Ġeach Ġarticle Ġas Ġwell Ġas Ġthe Ġsyn onyms Ġof Ġthe Ġarticle Ġfocus Ġ( topic ). ĠThese Ġresources Ġand Ġtheir Ġbrief Ġdescriptions Ġare Ġprovided Ġbelow : </s> National ĠCancer ĠInstitute Ġ( NC I ) Ġ: ĠWe Ġextracted Ġfree Ġtext Ġfrom Ġ116 Ġarticles Ġon Ġvarious Ġcancer Ġtypes Ġ( 729 ĠQ A Ġpairs ). ĠWe Ġmanually Ġrestruct ured Ġthe Ġcontent Ġof Ġthe Ġarticles Ġto Ġgenerate Ġcomplete Ġanswers Ġ( e . g . Ġa Ġfull Ġanswer Ġabout Ġthe Ġtreatment Ġof Ġall Ġstages Ġof Ġa Ġspecific Ġtype Ġof Ġcancer ). ĠFigure ĠFIG REF 54 Ġpresents Ġexamples Ġof ĠQ A Ġpairs Ġgenerated Ġfrom Ġa ĠNC I Ġarticle . </s> Gen etic Ġand ĠRare ĠDiseases ĠInformation ĠCenter Ġ( G ARD ): ĠThis Ġresource Ġcontains Ġinformation Ġabout Ġvarious Ġaspects Ġof Ġgenetic / ra re Ġdiseases . ĠWe Ġextracted Ġall Ġdisease Ġquestion / answer Ġpairs Ġfrom Ġ4 , 278 Ġtopics Ġ( 5 , 394 ĠQ A Ġpairs ). </s> Gen etics ĠHome ĠReference Ġ( G HR ): ĠThis ĠNL M Ġresource Ġcontains Ġconsumer - oriented Ġinformation Ġabout Ġthe Ġeffects Ġof Ġgenetic Ġvariation Ġon Ġhuman Ġhealth . ĠWe Ġextracted Ġ1 , 0 99 Ġarticles Ġabout Ġdiseases Ġfrom Ġthis Ġresource Ġ( 5 , 430 ĠQ A Ġpairs ). </s> Med line Plus ĠHealth ĠTopics : ĠThis Ġportion Ġof ĠMedline Plus Ġcontains Ġinformation Ġon Ġsymptoms , Ġcauses , Ġtreatment Ġand Ġprevention Ġfor Ġdiseases , Ġhealth Ġconditions Ġand Ġwellness Ġissues . ĠWe Ġextracted Ġthe Ġfree Ġtexts Ġin Ġsummary Ġsections Ġof Ġ9 81 Ġarticles Ġ( 98 1 ĠQ A Ġpairs ). </s> National ĠInstitute Ġof ĠDiabetes Ġand ĠDigest ive Ġand ĠKid ney ĠDiseases Ġ( N ID DK ) Ġ: ĠWe Ġextracted Ġtext Ġfrom Ġ174 Ġhealth Ġinformation Ġpages Ġon Ġdiseases Ġstudied Ġby Ġthis Ġinstitute Ġ( 1 , 192 ĠQ A Ġpairs ). </s> National ĠInstitute Ġof ĠNeurolog ical ĠDisorders Ġand ĠStro ke Ġ( N IN DS ): ĠWe Ġextracted Ġfree Ġtext Ġfrom Ġ277 Ġinformation Ġpages Ġon Ġneurological Ġand Ġstroke - related Ġdiseases Ġfrom Ġthis Ġresource Ġ( 1 , 104 ĠQ A Ġpairs ). </s> NI HS en ior Health Ġ: ĠThis Ġwebsite Ġcontains Ġhealth Ġand Ġwellness Ġinformation Ġfor Ġolder Ġadults . ĠWe Ġextracted Ġ71 Ġarticles Ġfrom Ġthis Ġresource Ġ( 7 69 ĠQ A Ġpairs ). </s> National ĠHeart , ĠLung , Ġand ĠBlood ĠInstitute Ġ( N HL BI ) Ġ: ĠWe Ġextracted Ġtext Ġfrom Ġ135 Ġarticles Ġon Ġdiseases , Ġtests , Ġprocedures , Ġand Ġother Ġrelevant Ġtopics Ġon Ġdisorders Ġof Ġheart , Ġlung , Ġblood , Ġand Ġsleep Ġ( 559 ĠQ A Ġpairs ). </s> Cent ers Ġfor ĠDisease ĠControl Ġand ĠPrevention Ġ( CDC ) Ġ: ĠWe Ġextracted Ġtext Ġfrom Ġ152 Ġarticles Ġon Ġdiseases Ġand Ġconditions Ġ( 270 ĠQ A Ġpairs ). </s> Med line Plus ĠA . D . A . M . ĠMedical ĠEncyclopedia : ĠThis Ġresource Ġcontains Ġ4 , 366 Ġarticles Ġabout Ġconditions , Ġtests , Ġand Ġprocedures . Ġ17 , 348 ĠQ A Ġpairs Ġwere Ġextracted Ġfrom Ġthis Ġresource . ĠFigure ĠFIG REF 55 Ġpresents Ġexamples Ġof ĠQ A Ġpairs Ġgenerated Ġfrom ĠA . D . A . M Ġencyclopedia . </s> Med line Plus ĠDrugs : ĠWe Ġextracted Ġfree Ġtext Ġfrom Ġ1 , 316 Ġarticles Ġabout ĠDrugs Ġand Ġgenerated Ġ12 , 889 ĠQ A Ġpairs . </s> Med line Plus ĠHer bs Ġand ĠSupp lements : ĠWe Ġextracted Ġfree Ġtext Ġfrom Ġ99 Ġarticles Ġand Ġgenerated Ġ7 92 ĠQ A Ġpairs . </s> The Ġfinal Ġcollection Ġcontains Ġ47 , 457 Ġannot ated Ġquestion - answer Ġpairs Ġabout ĠDiseases , ĠDrugs Ġand Ġother Ġnamed Ġentities Ġ( e . g . ĠTests ) Ġextracted Ġfrom Ġthese Ġ12 Ġtrusted Ġresources . </s> The ĠProp osed ĠEnt ail ment - based ĠQ A ĠSystem </s> Our Ġgoal Ġis Ġto Ġgenerate Ġa Ġranked Ġlist Ġof Ġanswers Ġfor Ġa Ġgiven ĠPrem ise ĠQuestion ĠIN LINE FORM 0 Ġby Ġranking Ġthe Ġrecognized ĠHyp ot hesis ĠQuestions ĠIN LINE FORM 1 Ġ. ĠBased Ġon Ġthe ĠR Q E Ġexperiments Ġabove Ġ( Section ĠSEC REF 27 Ġ), Ġwe Ġselected ĠLog istic ĠReg ression Ġtrained Ġon Ġthe Ġclinical - R Q E Ġdataset Ġto Ġrecognize Ġent ailed Ġquestions Ġand Ġrank Ġthem Ġwith Ġtheir Ġclassification Ġscores . </s> R Q E - based ĠQ A ĠApproach </s> Class ifying Ġthe Ġfull ĠQ A Ġcollection Ġfor Ġeach Ġtest Ġquestion Ġis Ġnot Ġfeasible Ġfor Ġreal - time Ġapplications . ĠTherefore , Ġwe Ġfirst Ġfilter Ġthe Ġquestions Ġwith Ġan ĠIR Ġmethod Ġto Ġretrieve Ġcandidate Ġquestions , Ġthen Ġclassify Ġthem Ġas Ġent ailed Ġ( or Ġnot ) Ġby Ġthe Ġuser / test Ġquestion . ĠBased Ġon Ġthe Ġpositive Ġresults Ġof Ġthe Ġcombination Ġmethod Ġtested Ġon ĠSem E val - c Q A Ġdata Ġ( Section ĠSEC REF 27 Ġ), Ġwe Ġadopted Ġa Ġcombination Ġmethod Ġto Ġmerge Ġthe Ġresults Ġobtained Ġby Ġthe Ġsearch Ġengine Ġand Ġthe ĠR Q E Ġscores . ĠThe Ġanswers Ġare Ġthen Ġcombined Ġfrom Ġboth Ġmethods Ġand Ġranked Ġusing Ġan Ġaggregate Ġscore . ĠFigure ĠFIG REF 82 Ġpresents Ġthe Ġoverall Ġarchitecture Ġof Ġthe Ġproposed ĠQ A Ġsystem . ĠWe Ġdescribe Ġeach Ġmodule Ġin Ġmore Ġdetails Ġnext . </s> Finding ĠSimilar ĠQuestion ĠCand idates </s> For Ġeach Ġpremise Ġquestion ĠIN LINE FORM 0 Ġ, Ġwe Ġuse Ġthe ĠTer rier Ġsearch Ġengine Ġto Ġretrieve ĠIN LINE FORM 1 Ġrelevant Ġquestion Ġcandidates ĠIN LINE FORM 2 Ġand Ġthen Ġapply Ġthe ĠR Q E Ġclass ifier Ġto Ġpredict Ġthe Ġlabels Ġfor Ġthe Ġpairs Ġ( ĠIN LINE FORM 3 Ġ, ĠIN LINE FORM 4 Ġ). </s> We Ġindexed Ġthe Ġquestions Ġof Ġour ĠQ A Ġcollection Ġwithout Ġthe Ġassociated Ġanswers . ĠIn Ġorder Ġto Ġimprove Ġthe Ġindex ing Ġand Ġthe Ġperformance Ġof Ġquestion Ġretrieval , Ġwe Ġalso Ġindexed Ġthe Ġsyn onyms Ġof Ġthe Ġquestion Ġfocus Ġand Ġthe Ġtriggers Ġof Ġthe Ġquestion Ġtype Ġwith Ġeach Ġquestion . ĠThis Ġchoice Ġallowed Ġus Ġto Ġavoid Ġthe Ġshortcomings Ġof Ġquery Ġexpansion , Ġincluding Ġincorrect Ġor Ġirrelevant Ġsyn onyms Ġand Ġthe Ġincreased Ġexecution Ġtime . ĠThe Ġsyn onyms Ġof Ġthe Ġquestion Ġfocus Ġ( topic ) Ġwere Ġextracted Ġautomatically Ġfrom Ġthe ĠQ A Ġcollection . ĠThe Ġtriggers Ġof Ġeach Ġquestion Ġtype Ġwere Ġdefined Ġmanually Ġin Ġthe Ġquestion Ġtypes Ġtax onomy . ĠBelow Ġare Ġtwo Ġexamples Ġof Ġindexed Ġquestions Ġfrom Ġour ĠQ A Ġcollection , Ġwith Ġthe Ġautomatically Ġadded Ġfocus Ġsyn onyms Ġand Ġquestion Ġtype Ġtriggers : </s> What Ġare Ġthe Ġtreatments Ġfor ĠTort ic oll is ? </s> Focus : ĠTort ic oll is . ĠQuestion Ġtype : ĠTreatment . </s> Added Ġfocus Ġsyn onyms : Ġ" Sp as mod ic Ġtort ic oll is , ĠW ry Ġneck , ĠL ox ia , ĠC erv ical Ġdy ston ia ". ĠAdded Ġquestion Ġtype Ġtriggers : Ġ" rel ieve , Ġmanage , Ġcure , Ġremedy , Ġtherapy ". </s> What Ġis Ġthe Ġoutlook Ġfor ĠLegion naire Ġdisease ? </s> Focus : ĠLegion naire Ġdisease . ĠQuestion ĠType : ĠPro gn osis . </s> Added Ġfocus Ġsyn onyms : Ġ" Leg ion ella Ġpneumonia , ĠPont iac Ġfever , ĠLegion ell osis ". ĠAdded Ġquestion Ġtype Ġtriggers : Ġ" pro gn osis , Ġlife Ġexpectancy ". </s> The ĠIR Ġtask Ġconsists Ġof Ġretrieving Ġhypothesis Ġquestions ĠIN LINE FORM 0 Ġrelevant Ġto Ġthe Ġsubmitted Ġquestion ĠIN LINE FORM 1 Ġ. ĠAs Ġfusion Ġof ĠIR Ġresult Ġhas Ġshown Ġgood Ġperformance Ġin Ġdifferent Ġtracks Ġin ĠT REC , Ġwe Ġmerge Ġthe Ġresults Ġof Ġthe ĠTF - ID F Ġweight ing Ġfunction Ġand Ġthe ĠIn - exp B 2 ĠD FR Ġmodel ĠB IB REF 50 Ġ. </s> Let ĠIN LINE FORM 0 Ġ= ĠIN LINE FORM 1 Ġ, ĠIN LINE FORM 2 Ġ, Ġ... , ĠIN LINE FORM 3 Ġbe Ġthe Ġset Ġof ĠIN LINE FORM 4 Ġquestions Ġretrieved Ġby Ġthe Ġfirst ĠIR Ġmodel ĠIN LINE FORM 5 Ġand ĠIN LINE FORM 6 Ġ= ĠIN LINE FORM 7 Ġ, ĠIN LINE FORM 8 Ġ, Ġ... , ĠIN LINE FORM 9 Ġbe Ġthe Ġset Ġof ĠIN LINE FORM 10 Ġquestions Ġretrieved Ġby Ġthe Ġsecond ĠIR Ġmodel ĠIN LINE FORM 11 Ġ. ĠWe Ġmerge Ġboth Ġsets Ġby Ġsum ming Ġthe Ġscores Ġof Ġeach Ġretrieved Ġquestion ĠIN LINE FORM 12 Ġin Ġboth ĠIN LINE FORM 13 Ġand ĠIN LINE FORM 14 Ġlists , Ġthen Ġwe Ġre - rank Ġthe Ġhypothesis Ġquestions ĠIN LINE FORM 15 Ġ. </s> Comb ining ĠIR Ġand ĠR Q E ĠMethods </s> The ĠIR Ġmodels Ġand Ġthe ĠR Q E ĠLog istic ĠReg ression Ġmodel Ġbring Ġdifferent Ġperspectives Ġto Ġthe Ġsearch Ġfor Ġrelevant Ġcandidate Ġquestions . ĠIn Ġparticular , Ġquestion Ġentail ment Ġallows Ġunderstanding Ġthe Ġrelations Ġbetween Ġthe Ġimportant Ġterms , Ġwhereas Ġthe Ġtraditional ĠIR Ġmethods Ġidentify Ġthe Ġimportant Ġterms , Ġbut Ġwill Ġnot Ġnotice Ġif Ġthe Ġrelations Ġare Ġopposite . ĠMoreover , Ġsome Ġof Ġthe Ġquestion Ġtypes Ġthat Ġthe ĠR Q E Ġclass ifier Ġlearns Ġwill Ġnot Ġbe Ġdeemed Ġimportant Ġterms Ġby Ġtraditional ĠIR Ġand Ġthe Ġmost Ġrelevant Ġquestions Ġwill Ġnot Ġbe Ġranked Ġat Ġthe Ġtop Ġof Ġthe Ġlist . </s> Therefore , Ġin Ġour Ġapproach , Ġwhen Ġa Ġquestion Ġis Ġsubmitted Ġto Ġthe Ġsystem , Ġcandidate Ġquestions Ġare Ġfet ched Ġusing Ġthe ĠIR Ġmodels , Ġthen Ġthe ĠR Q E Ġclass ifier Ġis Ġapplied Ġto Ġfilter Ġout Ġthe Ġnon - ent ailed Ġquestions Ġand Ġre - rank Ġthe Ġremaining Ġcandidates . </s> Specifically , Ġwe Ġdenote ĠIN LINE FORM 0 Ġthe Ġlist Ġof Ġquestion Ġcandidates ĠIN LINE FORM 1 Ġreturned Ġby Ġthe ĠIR Ġsystem . ĠThe Ġpremise Ġquestion ĠIN LINE FORM 2 Ġis Ġthen Ġused Ġto Ġconstruct ĠN Ġquestion Ġpairs ĠIN LINE FORM 3 Ġ. ĠThe ĠR Q E Ġclass ifier Ġis Ġthen Ġapplied Ġto Ġfilter Ġout Ġthe Ġquestion Ġpairs Ġthat Ġare Ġnot Ġent ailed Ġand Ġre - rank Ġthe Ġremaining Ġpairs . </s> More Ġprecisely , Ġlet ĠIN LINE FORM 0 Ġ= ĠIN LINE FORM 1 Ġin ĠIN LINE FORM 2 Ġbe Ġthe Ġlist Ġof Ġselected Ġcandidate Ġquestions Ġthat Ġhave Ġa Ġpositive Ġentail ment Ġrelation Ġwith Ġa Ġgiven Ġpremise Ġquestion ĠIN LINE FORM 3 Ġ. ĠWe Ġrank ĠIN LINE FORM 4 Ġby Ġcomputing Ġa Ġhybrid Ġscore ĠIN LINE FORM 5 Ġfor Ġeach Ġcandidate Ġquestion ĠIN LINE FORM 6 Ġtaking Ġinto Ġaccount Ġthe Ġscore Ġof Ġthe ĠIR Ġsystem ĠIN LINE FORM 7 Ġand Ġthe Ġscore Ġof Ġthe ĠR Q E Ġsystem ĠIN LINE FORM 8 Ġ. </s> For Ġeach Ġsystem ĠIN LINE FORM 0 ĠIN LINE FORM 1 Ġ, Ġwe Ġnormal ize Ġthe Ġassociated Ġscore Ġby Ġdividing Ġit Ġby Ġthe Ġmaximum Ġscore Ġamong Ġthe ĠIN LINE FORM 2 Ġcandidate Ġquestions Ġretrieved Ġby ĠIN LINE FORM 3 Ġfor ĠIN LINE FORM 4 Ġ: </s> IN LINE FORM 0 </s> IN LINE FORM 0 ĠIN LINE FORM 1 </s> In Ġour Ġexperiments , Ġwe Ġfixed Ġthe Ġvalue Ġof ĠIN LINE FORM 0 Ġto Ġ100 . ĠThis Ġthreshold Ġvalue Ġwas Ġselected Ġas Ġa Ġsafe Ġvalue Ġfor Ġthis Ġtask Ġfor Ġthe Ġfollowing Ġreasons : </s> Our Ġcollection Ġof Ġ47 , 457 Ġquestion - answer Ġpairs Ġwas Ġcollected Ġfrom Ġonly Ġ12 ĠNIH Ġinstit utes Ġand Ġis Ġunlikely Ġto Ġcontain Ġmore Ġthan Ġ100 Ġoccurrences Ġof Ġthe Ġsame Ġfocus - type Ġpair . </s> Each Ġquestion Ġwas Ġindexed Ġwith Ġadditional Ġannotations Ġfor Ġthe Ġquestion Ġfocus , Ġits Ġsyn onyms Ġand Ġthe Ġquestion Ġtype Ġsyn onyms . </s> E val uating ĠR Q E Ġfor ĠMedical ĠQuestion ĠAn sw ering </s> The Ġobjective Ġof Ġthis Ġevaluation Ġis Ġto Ġstudy Ġthe Ġeffectiveness Ġof ĠR Q E Ġfor ĠMedical ĠQuestion ĠAn sw ering , Ġby Ġcomparing Ġthe Ġanswers Ġretrieved Ġby Ġthe Ġhybrid Ġentail ment - based Ġapproach , Ġthe ĠIR Ġmethod Ġand Ġthe Ġother ĠQ A Ġsystems Ġparticipating Ġto Ġthe Ġmedical Ġtask Ġat ĠT REC Ġ2017 ĠLive Q A Ġchallenge Ġ( Live Q A - Med ). </s> E val uation ĠMethod </s> We Ġdeveloped Ġan Ġinterface Ġto Ġperform Ġthe Ġmanual Ġevaluation Ġof Ġthe Ġretrieved Ġanswers . ĠFigure Ġ5 Ġpresents Ġthe Ġevaluation Ġinterface Ġshowing , Ġfor Ġeach Ġtest Ġquestion , Ġthe Ġtop - 10 Ġanswers Ġof Ġthe Ġevaluated ĠQ A Ġmethod Ġand Ġthe Ġreference Ġanswer ( s ) Ġused Ġby ĠLive Q A Ġassess ors Ġto Ġhelp Ġjudging Ġthe Ġretrieved Ġanswers Ġby Ġthe Ġparticipating Ġsystems . </s> We Ġused Ġthe Ġtest Ġquestions Ġof Ġthe Ġmedical Ġtask Ġat ĠT REC - 2017 ĠLive Q A ĠB IB REF 11 Ġ. ĠThese Ġquestions Ġare Ġrandomly Ġselected Ġfrom Ġthe Ġconsumer Ġhealth Ġquestions Ġthat Ġthe ĠNL M Ġreceives Ġdaily Ġfrom Ġall Ġover Ġthe Ġworld . ĠThe Ġtest Ġquestions Ġcover Ġdifferent Ġmedical Ġentities Ġand Ġhave Ġa Ġwide Ġlist Ġof Ġquestion Ġtypes Ġsuch Ġas ĠComparison , ĠDiagn osis , ĠIng red ient , ĠSide Ġeffects Ġand ĠTap ering . </s> For Ġa Ġrelevant Ġcomparison , Ġwe Ġused Ġthe Ġsame Ġjudgment Ġscores Ġas Ġthe ĠLive Q A ĠTrack : </s> Correct Ġand ĠComplete ĠAnswer Ġ( 4 ) </s> Correct Ġbut ĠIn complete Ġ( 3 ) </s> In correct Ġbut ĠRelated Ġ( 2 ) </s> In correct Ġ( 1 ) </s> We Ġevaluated Ġthe Ġanswers Ġreturned Ġby Ġthe ĠIR - based Ġmethod Ġand Ġthe Ġhybrid ĠQ A Ġmethod Ġ( IR + R Q E ) Ġaccording Ġto Ġthe Ġsame Ġreference Ġanswers Ġused Ġin ĠLive Q A - Med . ĠThe Ġanswers Ġwere Ġanonym ized Ġ( the Ġmethod Ġnames Ġwere Ġblinded ) Ġand Ġpresented Ġto Ġ3 Ġassess ors : Ġa Ġmedical Ġdoctor Ġ( Ass essor ĠA ), Ġa Ġmedical Ġl ibrarian Ġ( B ) Ġand Ġa Ġresearcher Ġin Ġmedical Ġinform atics Ġ( C ). ĠNone Ġof Ġthe Ġassess ors Ġparticipated Ġin Ġthe Ġdevelopment Ġof Ġthe ĠQ A Ġmethods . ĠAss essors ĠB Ġand ĠC Ġevaluated Ġ1 , 000 Ġanswers Ġretrieved Ġby Ġeach Ġof Ġthe Ġmethods Ġ( IR Ġand ĠIR + R Q E ). ĠAss essor ĠA Ġevaluated Ġ2 , 000 Ġanswers Ġfrom Ġboth Ġmethods . </s> Table ĠT AB REF 103 Ġpresents Ġthe Ġinter - annot ator Ġagreement Ġ( IA A ) Ġthrough ĠF 1 Ġscore Ġcomputed Ġby Ġconsidering Ġone Ġof Ġthe Ġassess ors Ġas Ġreference . ĠIn Ġthe Ġfirst Ġevaluation , Ġwe Ġcomputed Ġthe ĠTrue ĠP osit ives Ġ( TP ) Ġand ĠFalse ĠP osit ives Ġ( FP ) Ġover Ġall Ġratings Ġand Ġthe ĠPrecision Ġand ĠF 1 Ġscore . ĠAs Ġthere Ġare Ġno Ġnegative Ġlabels Ġ( only Ġtrue Ġor Ġfalse Ġpositives Ġfor Ġeach Ġcategory ), ĠRecall Ġis Ġ100 %. ĠWe Ġalso Ġcomputed Ġa Ġpartial ĠI AA Ġby Ġgrouping Ġthe Ġ" Correct Ġand ĠComplete ĠAnswer " Ġand Ġ" Correct Ġbut ĠIn complete " Ġratings Ġ( as ĠCorrect ), Ġand Ġthe Ġ" In correct Ġbut ĠRelated " Ġand Ġ" In correct " Ġratings Ġ( as ĠInc orrect ). ĠThe Ġaverage Ġagreement Ġon Ġdistinguishing Ġthe ĠCorrect Ġand ĠInc orrect Ġanswers Ġis Ġ94 . 33 % ĠF 1 Ġscore . ĠTherefore , Ġwe Ġused Ġthe Ġevaluations Ġperformed Ġby Ġassess or ĠA Ġfor Ġboth Ġmethods . ĠThe Ġofficial Ġresults Ġof Ġthe ĠT REC ĠLive Q A Ġtrack Ġrelied Ġon Ġone Ġassess or Ġper Ġquestion Ġas Ġwell . </s> E val uation Ġof Ġthe Ġfirst Ġretrieved Ġanswer </s> We Ġcomputed Ġthe Ġmeasures Ġused Ġby ĠT REC ĠLive Q A Ġchallenges ĠB IB REF 51 Ġ, ĠB IB REF 11 Ġto Ġevaluate Ġthe Ġfirst Ġretrieved Ġanswer Ġfor Ġeach Ġtest Ġquestion : </s> av g Score ( 0 - 3 ): Ġthe Ġaverage Ġscore Ġover Ġall Ġquestions , Ġtransferring Ġ1 - 4 Ġlevel Ġgrades Ġto Ġ0 - 3 Ġscores . ĠThis Ġis Ġthe Ġmain Ġscore Ġused Ġto Ġrank ĠLive Q A Ġruns . </s> su cc @ i + : Ġthe Ġnumber Ġof Ġquestions Ġwith Ġscore Ġi Ġor Ġabove Ġ( i ĠIN LINE FORM 0 Ġ{ 2 .. 4 }) Ġdivided Ġby Ġthe Ġtotal Ġnumber Ġof Ġquestions . </s> pre c @ i + : Ġthe Ġnumber Ġof Ġquestions Ġwith Ġscore Ġi Ġor Ġabove Ġ( i ĠIN LINE FORM 0 Ġ{ 2 .. 4 }) Ġdivided Ġby Ġnumber Ġof Ġquestions Ġanswered Ġby Ġthe Ġsystem . </s> Table ĠT AB REF 108 Ġpresents Ġthe Ġaverage Ġscores , Ġsuccess Ġand Ġprecision Ġresults . ĠThe Ġhybrid ĠIR + R Q E ĠQ A Ġsystem Ġachieved Ġbetter Ġresults Ġthan Ġthe ĠIR - based Ġsystem Ġwith Ġ0 . 8 27 Ġaverage Ġscore . ĠIt Ġalso Ġachieved Ġa Ġhigher Ġscore Ġthan Ġthe Ġbest Ġresults Ġachieved Ġin Ġthe Ġmedical Ġchallenge Ġat ĠLive Q A ' 17 . ĠEval uating Ġthe ĠR Q E Ġsystem Ġalone Ġis Ġnot Ġrelevant , Ġas Ġapplying ĠR Q E Ġon Ġthe Ġfull Ġcollection Ġfor Ġeach Ġuser Ġquestion Ġis Ġnot Ġfeasible Ġfor Ġa Ġreal - time Ġsystem Ġbecause Ġof Ġthe Ġextended Ġexecution Ġtime . </s> E val uation Ġof Ġthe Ġtop Ġten Ġanswers </s> In Ġthis Ġevaluation , Ġwe Ġused ĠMean ĠAverage ĠPrecision Ġ( MAP ) Ġand ĠMean ĠRec ip rocal ĠRank Ġ( MR R ) Ġwhich Ġare Ġcommonly Ġused Ġin ĠQ A Ġto Ġevaluate Ġthe Ġtop - 10 Ġanswers Ġfor Ġeach Ġquestion . ĠWe Ġconsider Ġanswers Ġrated Ġas ĠâĢ ľ Correct Ġand ĠComplete ĠAnswer âĢ Ŀ Ġor ĠâĢ ľ Correct Ġbut ĠIn complete âĢ Ŀ Ġas Ġcorrect Ġanswers , Ġas Ġthe Ġtest Ġquestions Ġcontain Ġmultiple Ġsub quest ions Ġwhile Ġeach Ġanswer Ġin Ġour ĠQ A Ġcollection Ġcan Ġcover Ġonly Ġone Ġsub question . </s> MAP Ġis Ġthe Ġmean Ġof Ġthe ĠAverage ĠPrecision Ġ( Avg P ) Ġscores Ġover Ġall Ġquestions . </s> ( 1 ) ĠIN LINE FORM 0 </s> Q Ġis Ġthe Ġnumber Ġof Ġquestions . ĠIN LINE FORM 0 Ġis Ġthe ĠAvg P Ġof Ġthe ĠIN LINE FORM 1 Ġquestion . </s> IN LINE FORM 0 </s> K Ġis Ġthe Ġnumber Ġof Ġcorrect Ġanswers . ĠIN LINE FORM 0 Ġis Ġthe Ġrank Ġof ĠIN LINE FORM 1 Ġcorrect Ġanswer . </s> MR R Ġis Ġthe Ġaverage Ġof Ġthe Ġreciprocal Ġranks Ġfor Ġeach Ġquestion . ĠThe Ġreciprocal Ġrank Ġof Ġa Ġquestion Ġis Ġthe Ġmultipl icative Ġinverse Ġof Ġthe Ġrank Ġof Ġthe Ġfirst Ġcorrect Ġanswer . </s> ( 2 ) ĠIN LINE FORM 0 </s> Q Ġis Ġthe Ġnumber Ġof Ġquestions . ĠIN LINE FORM 0 Ġis Ġthe Ġrank Ġof Ġthe Ġfirst Ġcorrect Ġanswer Ġfor Ġthe ĠIN LINE FORM 1 Ġquestion . </s> Table ĠT AB REF 113 Ġpresents Ġthe ĠMAP @ 10 Ġand ĠMR R @ 10 Ġof Ġour ĠQ A Ġmethods . ĠThe ĠIR + R Q E Ġsystem Ġoutper forms Ġthe ĠIR - based ĠQ A Ġsystem Ġwith Ġ0 . 311 ĠMAP @ 10 Ġand Ġ0 . 333 ĠMR R @ 10 . </s> Discussion Ġof Ġentail ment - based ĠQ A Ġfor Ġthe Ġmedical Ġdomain </s> In Ġour Ġevaluation , Ġwe Ġfollowed Ġthe Ġsame ĠLive Q A Ġguidelines Ġwith Ġthe Ġhighest Ġpossible Ġrig or . ĠIn Ġparticular , Ġwe Ġconsulted Ġwith ĠN IST Ġassess ors Ġwho Ġprovided Ġus Ġwith Ġthe Ġparaph r ases Ġof Ġthe Ġtest Ġquestions Ġthat Ġthey Ġused Ġto Ġjudge Ġthe Ġanswers . ĠOur ĠI AA Ġon Ġthe Ġanswers Ġrating Ġwas Ġalso Ġhigh Ġcompared Ġto Ġrelated Ġtasks , Ġwith Ġan Ġ88 . 5 % ĠF 1 Ġagreement Ġwith Ġthe Ġexact Ġfour Ġcategories Ġand Ġa Ġ94 . 3 % Ġagreement Ġwhen Ġreducing Ġthe Ġcategories Ġto Ġtwo : ĠâĢ ľ Correct âĢ Ŀ Ġand ĠâĢ ľ In correct âĢ Ŀ Ġanswers . ĠOur Ġresults Ġshow Ġthat ĠR Q E Ġimproves Ġthe Ġoverall Ġperformance Ġand Ġexceeds Ġthe Ġbest Ġresults Ġin Ġthe Ġmedical ĠLive Q A ' 17 Ġchallenge Ġby Ġa Ġfactor Ġof Ġ29 . 8 %. ĠThis Ġperformance Ġimprovement Ġis Ġparticularly Ġinteresting Ġas : </s> Our Ġanswer Ġsource Ġhas Ġonly Ġ47 K Ġquestion - answer Ġpairs Ġwhen ĠLive Q A Ġparticipating Ġsystems Ġrelied Ġon Ġmuch Ġlarger Ġcollections , Ġincluding Ġthe ĠWorld ĠWide ĠWeb . </s> Our Ġsystem Ġanswered Ġone Ġsub question Ġat Ġmost Ġwhen Ġmany ĠLive Q A Ġtest Ġquestions Ġhad Ġseveral Ġsub quest ions . </s> The Ġlatter Ġobservation , Ġ( b ), Ġmakes Ġthe Ġhybrid ĠIR + R Q E Ġapproach Ġeven Ġmore Ġpromising Ġas Ġit Ġgives Ġit Ġa Ġlarge Ġpotential Ġfor Ġthe Ġimprovement Ġof Ġanswer Ġcomple teness . </s> The Ġformer Ġobservation , Ġ( a ), Ġprovides Ġanother Ġinteresting Ġinsight : Ġrestricting Ġthe Ġanswer Ġsource Ġto Ġonly Ġreliable Ġcollections Ġcan Ġactually Ġimprove Ġthe ĠQ A Ġperformance Ġwithout Ġlosing Ġcoverage Ġ( i . e ., Ġour ĠQ A Ġapproach Ġprovided Ġat Ġleast Ġone Ġanswer Ġto Ġeach Ġtest Ġquestion Ġand Ġobtained Ġthe Ġbest Ġrelevance Ġscore ). </s> In Ġanother Ġobservation , Ġthe Ġassess ors Ġreported Ġthat Ġmany Ġof Ġthe Ġreturned Ġanswers Ġhad Ġa Ġcorrect Ġquestion Ġtype Ġbut Ġa Ġwrong Ġfocus , Ġwhich Ġindicates Ġthat Ġincluding Ġa Ġfocus Ġrecognition Ġmodule Ġto Ġfilter Ġsuch Ġwrong Ġanswers Ġcan Ġimprove Ġfurther Ġthe ĠQ A Ġperformance Ġin Ġterms Ġof Ġprecision . ĠAnother Ġaspect Ġthat Ġwas Ġreported Ġis Ġthe Ġrepetition Ġof Ġthe Ġsame Ġ( or Ġsimilar ) Ġanswer Ġfrom Ġdifferent Ġwebsites , Ġwhich Ġcould Ġbe Ġaddressed Ġby Ġimproving Ġanswer Ġselection Ġwith Ġinter - answer Ġcomparisons Ġand Ġremoval Ġof Ġnear - du pl icates . ĠAlso , Ġhalf Ġof Ġthe ĠLive Q A Ġtest Ġquestions Ġare Ġabout ĠDrugs , Ġwhen Ġonly Ġtwo Ġof Ġour Ġresources Ġare Ġspecialized Ġin ĠDrugs , Ġamong Ġ12 Ġsub - col lections Ġoverall . ĠAccordingly , Ġthe Ġassess ors Ġnoticed Ġthat Ġthe Ġperformance Ġof Ġthe ĠQ A Ġsystems Ġwas Ġbetter Ġon Ġquestions Ġabout Ġdiseases Ġthan Ġon Ġquestions Ġabout Ġdrugs , Ġwhich Ġsuggests Ġa Ġneed Ġfor Ġextending Ġour Ġmedical ĠQ A Ġcollection Ġwith Ġmore Ġinformation Ġabout Ġdrugs Ġand Ġassociated Ġquestion Ġtypes . </s> We Ġalso Ġlooked Ġclosely Ġat Ġthe Ġprivate Ġwebsites Ġused Ġby Ġthe ĠLive Q A - Med Ġannot ators Ġto Ġprovide Ġsome Ġof Ġthe Ġreference Ġanswers Ġfor Ġthe Ġtest Ġquestions . ĠFor Ġinstance , Ġthe ĠConsumer Lab Ġwebsite Ġwas Ġuseful Ġto Ġanswer Ġa Ġquestion Ġabout Ġthe Ġingredients Ġof Ġa ĠDrug Ġ( CO EN Z Y ME ĠQ 10 ). ĠSimilarly , Ġthe Ġe Health Me Ġwebsite Ġwas Ġused Ġto Ġanswer Ġa Ġtest Ġquestion Ġasking Ġabout Ġinteractions Ġbetween Ġtwo Ġdrugs Ġ( P hen ter mine Ġand ĠD icy cl om ine ) Ġwhen Ġno Ġinformation Ġwas Ġfound Ġin ĠDaily Med . Ġe Health Me Ġprovides Ġhealthcare Ġbig Ġdata Ġanalysis Ġand Ġprivate Ġresearch Ġand Ġstudies Ġincluding Ġself - reported Ġadverse Ġdrug Ġeffects Ġby Ġpatients . </s> But Ġthe Ġquestion Ġremains Ġon Ġthe Ġextent Ġto Ġwhich Ġsuch Ġbig Ġdata Ġand Ġother Ġprivate Ġwebsites Ġcould Ġbe Ġused Ġto Ġautomatically Ġanswer Ġmedical Ġquestions Ġif Ġinformation Ġis Ġotherwise Ġunavailable . ĠUnlike Ġmedical Ġprofessionals , Ġpatients Ġdo Ġnot Ġnecessarily Ġhave Ġthe Ġknowledge Ġand Ġtools Ġto Ġvalidate Ġsuch Ġinformation . ĠAn Ġalternative Ġapproach Ġcould Ġbe Ġto Ġput Ġlimitations Ġon Ġmedical ĠQ A Ġsystems Ġin Ġterms Ġof Ġthe Ġquestions Ġthat Ġcan Ġbe Ġanswered Ġ( e . g . Ġ" What Ġis Ġmy Ġdiagnosis Ġfor Ġsuch Ġsymptoms ") Ġand Ġbuild Ġclass ifiers Ġto Ġdetect Ġsuch Ġquestions Ġand Ġwarn Ġthe Ġusers Ġabout Ġthe Ġdangers Ġof Ġlooking Ġfor Ġtheir Ġanswers Ġonline . </s> More Ġgenerally , Ġmedical ĠQ A Ġsystems Ġshould Ġfollow Ġsome Ġstrict Ġguidelines Ġregarding Ġthe Ġgoal Ġand Ġbackground Ġknowledge Ġand Ġresources Ġof Ġeach Ġsystem Ġin Ġorder Ġto Ġprotect Ġthe Ġconsumers Ġfrom Ġmisleading Ġor Ġharmful Ġinformation . ĠSuch Ġguidelines Ġcould Ġbe Ġbased Ġ( i ) Ġon Ġthe Ġsource Ġof Ġthe Ġinformation Ġsuch Ġas Ġhealth Ġand Ġmedical Ġinformation Ġwebsites Ġsponsored Ġby Ġthe ĠU . S . Ġgovernment , Ġnot - for - profit Ġhealth Ġor Ġmedical Ġorganizations , Ġand Ġmedical Ġuniversity Ġcenters , Ġor Ġ( ii ) Ġon Ġconventions Ġsuch Ġas Ġthe Ġcode Ġof Ġconduct Ġof Ġthe ĠH ON ĠFoundation Ġ( H ON code ) Ġthat Ġaddresses Ġthe Ġreliability Ġand Ġusefulness Ġof Ġmedical Ġinformation Ġon Ġthe ĠInternet . </s> Our Ġexperiments Ġshow Ġthat Ġlimiting Ġthe Ġnumber Ġof Ġanswer Ġsources Ġwith Ġsuch Ġguidelines Ġis Ġnot Ġonly Ġfeasible , Ġbut Ġit Ġcould Ġalso Ġenhance Ġthe Ġperformance Ġof Ġthe ĠQ A Ġsystem Ġfrom Ġan Ġinformation Ġretrieval Ġperspective . </s> Conclusion </s> In Ġthis Ġpaper , Ġwe Ġcarried Ġout Ġan Ġempirical Ġstudy Ġof Ġmachine Ġlearning Ġand Ġdeep Ġlearning Ġmethods Ġfor ĠRecogn izing ĠQuestion ĠEnt ail ment Ġin Ġthe Ġmedical Ġdomain Ġusing Ġseveral Ġdatasets . ĠWe Ġdeveloped Ġa ĠR Q E - based ĠQ A Ġsystem Ġto Ġanswer Ġnew Ġmedical Ġquestions Ġusing Ġexisting Ġquestion - answer Ġpairs . ĠWe Ġbuilt Ġand Ġshared Ġa Ġcollection Ġof Ġ47 K Ġmedical Ġquestion - answer Ġpairs . ĠOur ĠQ A Ġapproach Ġoutper formed Ġthe Ġbest Ġresults Ġon ĠT REC - 2017 ĠLive Q A Ġmedical Ġtest Ġquestions . ĠThe Ġproposed Ġapproach Ġcan Ġbe Ġapplied Ġand Ġadapted Ġto Ġopen - domain Ġas Ġwell Ġas Ġspecific - domain ĠQ A . ĠDeep Ġlearning Ġmodels Ġachieved Ġinteresting Ġresults Ġon Ġopen - domain Ġand Ġclinical Ġdatasets , Ġbut Ġobtained Ġa Ġlower Ġperformance Ġon Ġconsumer Ġhealth Ġquestions . ĠWe Ġwill Ġcontinue Ġinvestigating Ġother Ġnetwork Ġarchitectures Ġincluding Ġtransfer Ġlearning , Ġas Ġwell Ġas Ġcreation Ġof Ġa Ġlarge Ġcollection Ġof Ġconsumer Ġhealth Ġquestions Ġfor Ġtraining Ġto Ġimprove Ġthe Ġperformance Ġof ĠDL Ġmodels . ĠFuture Ġwork Ġalso Ġincludes Ġexploring Ġintegration Ġof Ġa ĠQuestion ĠFocus ĠRecogn ition Ġmodule Ġto Ġenhance Ġcandidate Ġquestion Ġretrieval , Ġand Ġexpanding Ġour Ġquestion - answer Ġcollection . </s> Acknowled gements </s> We Ġthank ĠHal il ĠKil ic oglu Ġ( N LM / NI H ) Ġfor Ġhis Ġhelp Ġwith Ġthe Ġcrawling Ġand Ġthe Ġmanual Ġevaluation Ġand ĠSony a ĠE . ĠSho osh an Ġ( N LM / NI H ) Ġfor Ġher Ġhelp Ġwith Ġthe Ġjudgment Ġof Ġthe Ġretrieved Ġanswers . ĠWe Ġalso Ġthank ĠEllen ĠV oor he es Ġ( N IST ) Ġfor Ġher Ġvaluable Ġsupport Ġwith Ġthe ĠT REC ĠLive Q A Ġevaluation . </s> We Ġconsider Ġthe Ġcase Ġof Ġthe Ġquestion Ġnumber Ġ36 Ġin Ġthe ĠT REC - 2017 ĠLive Q A Ġmedical Ġtest Ġdataset : </s> 36 . Ġcongen ital Ġdi aph rag matic Ġher nia . Ġwhat Ġare Ġthe Ġcauses Ġof Ġcongen ital Ġdi aph rag matic Ġher nia ? ĠCan Ġcousin Ġmarriage Ġcause Ġthis ? ĠWhat Ġkind Ġof Ġlung Ġdisease Ġthe Ġbaby Ġmight Ġexperience Ġlife Ġlong ? </s> This Ġquestion Ġwas Ġanswered Ġby Ġ5 Ġparticipating Ġruns Ġ( vs . Ġ8 Ġruns Ġfor Ġother Ġquestions ), Ġand Ġall Ġsubmitted Ġanswers Ġwere Ġwrong Ġ( sc ores Ġof Ġ1 Ġor Ġ2 ). ĠHowever , Ġour ĠIR - based ĠQ A Ġsystem Ġretrieved Ġone Ġexcellent Ġanswer Ġ( score Ġ4 ) Ġand Ġour Ġhybrid ĠIR + R Q E Ġsystem Ġprovided Ġ3 Ġexcellent Ġanswers . </s> A ) ĠT REC Ġ2017 ĠLive Q A - Med ĠParticipants ' ĠResults : </s> B ) ĠOur ĠIR - based ĠQ A ĠSystem : </s> C ) ĠOur ĠIR + R Q E ĠQ A ĠSystem :
Introduction With the availability of rich data on users' locations, profiles and search history, personalization has become the leading trend in large-scale information retrieval. However, efficiency through personalization is not yet the most suitable model when tackling domain-specific searches. This is due to several factors, such as the lexical and semantic challenges of domain-specific data that often include advanced argumentation and complex contextual information, the higher sparseness of relevant information sources, and the more pronounced lack of similarities between users' searches. A recent study on expert search strategies among healthcare information professionals BIBREF0 showed that, for a given search task, they spend an average of 60 minutes per collection or database, 3 minutes to examine the relevance of each document, and 4 hours of total search time. When written in steps, their search strategy spans over 15 lines and can reach up to 105 lines. With the abundance of information sources in the medical domain, consumers are more and more faced with a similar challenge, one that needs dedicated solutions that can adapt to the heterogeneity and specifics of health-related information. Dedicated Question Answering (QA) systems are one of the viable solutions to this problem as they are designed to understand natural language questions without relying on external information on the users. In the context of QA, the goal of Recognizing Question Entailment (RQE) is to retrieve answers to a premise question ( INLINEFORM0 ) by retrieving inferred or entailed questions, called hypothesis questions ( INLINEFORM1 ) that already have associated answers. Therefore, we define the entailment relation between two questions as: a question INLINEFORM2 entails a question INLINEFORM3 if every answer to INLINEFORM4 is also a correct answer to INLINEFORM5 BIBREF1 . RQE is particularly relevant due to the increasing numbers of similar questions posted online BIBREF2 and its ability to solve differently the challenging issues of question understanding and answer extraction. In addition to being used to find relevant answers, these resources can also be used in training models able to recognize inference relations and similarity between questions. Question similarity has recently attracted international challenges BIBREF3 , BIBREF4 and several research efforts proposing a wide range of approaches, including Logistic Regression, Recurrent Neural Networks (RNNs), Long Short Term Memory cells (LSTMs), and Convolutional Neural Networks (CNNs) BIBREF5 , BIBREF6 , BIBREF1 , BIBREF7 . In this paper, we study question entailment in the medical domain and the effectiveness of the end-to-end RQE-based QA approach by evaluating the relevance of the retrieved answers. Although entailment was attempted in QA before BIBREF8 , BIBREF9 , BIBREF10 , as far as we know, we are the first to introduce and evaluate a full medical question answering approach based on question entailment for free-text questions. Our contributions are: The next section is dedicated to related work on question answering, question similarity and entailment. In Section SECREF3 , we present two machine learning (ML) and deep learning (DL) methods for RQE and compare their performance using open-domain and clinical datasets. Section SECREF4 describes the new collection of medical question-answer pairs. In Section SECREF5 , we describe our RQE-based approach for QA. Section SECREF6 presents our evaluation of the retrieved answers and the results obtained on TREC 2017 LiveQA medical questions. Background In this section we define the RQE task and describe related work at the intersection of question answering, question similarity and textual inference. Task Definition The definition of Recognizing Question Entailment (RQE) can have a significant impact on QA results. In related work, the meaning associated with Natural Language Inference (NLI) varies among different tasks and events. For instance, Recognizing Textual Entailment (RTE) was addressed by the PASCAL challenge BIBREF12 , where the entailment relation has been assessed manually by human judges who selected relevant sentences "entailing" a set of hypotheses from a list of documents returned by different Information Retrieval (IR) methods. In another definition, the Stanford Natural Language Inference corpus SNLI BIBREF13 , used three classification labels for the relations between two sentences: entailment, neutral and contradiction. For the entailment label, the annotators who built the corpus were presented with an image and asked to write a caption “that is a definitely true description of the photo”. For the neutral label, they were asked to provide a caption “that might be a true description of the label”. They were asked for a caption that “is definitely a false description of the photo” for the contradiction label. More recently, the multiNLI corpus BIBREF14 was shared in the scope of the RepEval 2017 shared task BIBREF15 . To build the corpus, annotators were presented with a premise text and asked to write three sentences. One novel sentence, which is “necessarily true or appropriate in the same situations as the premise,” for the entailment label, a sentence, which is “necessarily false or inappropriate whenever the premise is true,” for the contradiction label, and a last sentence, “where neither condition applies,” for the neutral label. Whereas these NLI definitions might be suitable for the broad topic of text understanding, their relation to practical information retrieval or question answering systems is not straightforward. In contrast, RQE has to be tailored to the question answering task. For instance, if the premise question is "looking for cold medications for a 30 yo woman", a RQE approach should be able to consider the more general (less restricted) question "looking for cold medications" as relevant, since its answers are relevant for the initial question, whereas "looking for medications for a 30 yo woman" is a useless contextualization. The entailment relation we are seeking in the QA context should include relevant and meaningful relaxations of contextual and semantic constraints (cf. Section SECREF13 ). Related Work on Question Answering Classical QA systems face two main challenges related to question analysis and answer extraction. Several QA approaches were proposed in the literature for the open domain BIBREF16 , BIBREF17 and the medical domain BIBREF18 , BIBREF19 , BIBREF20 . A variety of methods were developed for question analysis, focus (topic) recognition and question type identification BIBREF21 , BIBREF22 , BIBREF23 , BIBREF24 . Similarly, many different approaches tackled document or passage retrieval and answer selection and (re)ranking BIBREF25 , BIBREF26 , BIBREF27 . An alternative approach consists in finding similar questions or FAQs that are already answered BIBREF28 , BIBREF29 . One of the earliest question answering systems based on finding similar questions and re-using the existing answers was FAQ FINDER BIBREF30 . Another system that complements the existing Q&A services of NetWellness is SimQ BIBREF2 , which allows retrieval of similar web-based consumer health questions. SimQ uses syntactic and semantic features to compute similarity between questions, and UMLS BIBREF31 as a standardized semantic knowledge source. The system achieves 72.2% precision, 78.0% recall and 75.0% F-score on NetWellness questions. However, the method was evaluated only on one question similarity dataset, and the retrieved answers were not evaluated. The aim of the medical task at TREC 2017 LiveQA was to develop techniques for answering complex questions such as consumer health questions, as well as to identify relevant answer sources that can comply with the sensitivity of medical information retrieval. The CMU-OAQA system BIBREF32 achieved the best performance of 0.637 average score on the medical task by using an attentional encoder-decoder model for paraphrase identification and answer ranking. The Quora question-similarity dataset was used for training. The PRNA system BIBREF33 achieved the second best performance in the medical task with 0.49 average score using Wikipedia as the first answer source and Yahoo and Google searches as secondary answer sources. Each medical question was decomposed into several subquestions. To extract the answer from the selected text passage, a bi-directional attention model trained on the SQUAD dataset was used. Deep neural network models have been pushing the limits of performance achieved in QA related tasks using large training datasets. The results obtained by CMU-OAQA and PRNA showed that large open-domain datasets were beneficial for the medical domain. However, the best system (CMU-OAQA) relying on the same training data obtained a score of 1.139 on the LiveQA open-domain task. While this gap in performance can be explained in part by the discrepancies between the medical test questions and the open-domain questions, it also highlights the need for larger medical datasets to support deep learning approaches in dealing with the linguistic complexity of consumer health questions and the challenge of finding correct and complete answers. Another technique was used by ECNU-ICA team BIBREF34 based on learning question similarity via two long short-term memory (LSTM) networks applied to obtain the semantic representations of the questions. To construct a collection of similar question pairs, they searched community question answering sites such as Yahoo! and Answers.com. In contrast, the ECNU-ICA system achieved the best performance of 1.895 in the open-domain task but an average score of only 0.402 in the medical task. As the ECNU-ICA approach also relied on a neural network for question matching, this result shows that training attention-based decoder-encoder networks on the Quora dataset generalized better to the medical domain than training LSTMs on similar questions from Yahoo! and Answers.com. The CMU-LiveMedQA team BIBREF20 designed a specific system for the medical task. Using only the provided training datasets and the assumption that each question contains only one focus, the CMU-LiveMedQA system obtained an average score of 0.353. They used a convolutional neural network (CNN) model to classify a question into a restricted set of 10 question types and crawled "relevant" online web pages to find the answers. However, the results were lower than those achieved by the systems relying on finding similar answered questions. These results support the relevance of similar question matching for the end-to-end QA task as a new way of approaching QA instead of the classical QA approaches based on Question Analysis and Answer Retrieval. Related Work on Question Similarity and Entailment Several efforts focused on recognizing similar questions. Jeon et al. BIBREF35 showed that a retrieval model based on translation probabilities learned from a question and answer archive can recognize semantically similar questions. Duan et al. BIBREF36 proposed a dedicated language modeling approach for question search, using question topic (user's interest) and question focus (certain aspect of the topic). Lately, these efforts were supported by a task on Question-Question similarity introduced in the community QA challenge at SemEval (task 3B) BIBREF3 . Given a new question, the task focused on reranking all similar questions retrieved by a search engine, assuming that the answers to the similar questions will be correct answers for the new question. Different machine learning and deep learning approaches were tested in the scope of SemEval 2016 BIBREF3 and 2017 BIBREF4 task 3B. The best performing system in 2017 achieved a MAP of 47.22% using supervised Logistic Regression that combined different unsupervised similarity measures such as Cosine and Soft-Cosine BIBREF37 . The second best system achieved 46.93% MAP with a learning-to-rank method using Logistic Regression and a rich set of features including lexical and semantic features as well as embeddings generated by different neural networks (siamese, Bi-LSTM, GRU and CNNs) BIBREF38 . In the scope of this challenge, a dataset was collected from Qatar Living forum for training. We refer to this dataset as SemEval-cQA. In another effort, an answer-based definition of RQE was proposed and tested BIBREF1 . The authors introduced a dataset of clinical questions and used a feature-based method that provided an Accuracy of 75% on consumer health questions. We will call this dataset Clinical-QE. Dos Santos et al. BIBREF5 proposed a new approach to retrieve semantically equivalent questions combining a bag-of-words representation with a distributed vector representation created by a CNN and user data collected from two Stack Exchange communities. Lei et al. BIBREF7 proposed a recurrent and convolutional model (gated convolution) to map questions to their semantic representations. The models were pre-trained within an encoder-decoder framework. RQE Approaches and Experiments The choice of two methods for our empirical study is motivated by the best performance achieved by Logistic Regression in question-question similarity at SemEval 2017 (best system BIBREF37 and second best system BIBREF38 ), and the high performance achieved by neural networks on larger datasets such as SNLI BIBREF13 , BIBREF39 , BIBREF40 , BIBREF41 . We first define the RQE task, then present the two approaches, and evaluate their performance on five different datasets. Definition In the context of QA, the goal of RQE is to retrieve answers to a new question by retrieving entailed questions with associated answers. We therefore define question entailment as: a question INLINEFORM0 entails a question INLINEFORM1 if every answer to INLINEFORM2 is also a complete or partial answer to INLINEFORM3 . We present below two examples of consumer health questions INLINEFORM0 and entailed questions INLINEFORM1 : Example 1 (each answer to the entailed question B1 is a complete answer to A1): A1: What is the latest news on tennitis, or ringing in the ear, I am 75 years old and have had ringing in the ear since my mid 5os. Thank you. B1: What is the latest research on Tinnitus? Example 2 (each answer to the entailed question B2 is a partial answer to A2): A2: My mother has been diagnosed with Alzheimer's, my father is not of the greatest health either and is the main caregiver for my mother. My question is where do we start with attempting to help our parents w/ the care giving and what sort of financial options are there out there for people on fixed incomes. B2: What resources are available for Alzheimer's caregivers? The inclusion of partial answers in the definition of question entailment also allows efficient relaxation of the contextual constraints of the original question INLINEFORM0 to retrieve relevant answers from entailed, but less restricted, questions. Deep Learning Model To recognize entailment between two questions INLINEFORM0 (premise) and INLINEFORM1 (hypothesis), we adapted the neural network proposed by Bowman et al. BIBREF13 . Our DL model, presented in Figure FIGREF20 , consists of three 600d ReLU layers, with a bottom layer taking the concatenated sentence representations as input and a top layer feeding a softmax classifier. The sentence embedding model sums the Recurrent neural network (RNN) embeddings of its words. The word embeddings are first initialized with pretrained GloVe vectors. This adaptation provided the best performance in previous experiments with RQE data. GloVe is an unsupervised learning algorithm to generate vector representations for words BIBREF42 . Training is performed on aggregated word co-occurrence statistics from a large corpus, and the resulting representations show interesting linear substructures of the word vector space. We use the pretrained common crawl version with 840B tokens and 300d vectors, which are not updated during training. Logistic Regression Classifier In this feature-based approach, we use Logistic Regression to classify question pairs into entailment or no-entailment. Logistic Regression achieved good results on this specific task and outperformed other statistical learning algorithms such as SVM and Naive Bayes. In a preprocessing step, we remove stop words and perform word stemming using the Porter algorithm BIBREF43 for all ( INLINEFORM0 , INLINEFORM1 ) pairs. We use a list of nine features, selected after several experiments on RTE datasets BIBREF12 . We compute five similarity measures between the pre-processed questions and use their values as features. We use Word Overlap, the Dice coefficient based on the number of common bigrams, Cosine, Levenshtein, and the Jaccard similarities. Our feature list also includes the maximum and average values obtained with these measures and the question length ratio (length( INLINEFORM0 )/length( INLINEFORM1 )). We compute a morphosyntactic feature indicating the number of common nouns and verbs between INLINEFORM2 and INLINEFORM3 . TreeTagger BIBREF44 was used for POS tagging. For RQE, we add an additional feature specific to the question type. We use a dictionary lookup to map triggers to the question type (e.g. Treatment, Prognosis, Inheritance). Triggers are identified for each question type based on a manual annotation of a set of medical questions (cf. Section SECREF36 ). This feature has three possible values: 2 (Perfect match between INLINEFORM0 type(s) and INLINEFORM1 type(s)), 1 (Overlap between INLINEFORM2 type(s) and INLINEFORM3 type(s)) and 0 (No common types). Datasets Used for the RQE Study We evaluate the RQE methods (i.e. deep learning model and logistic regression classifier) using two datasets of sentence pairs (SNLI and multiNLI), and three datasets of question pairs (Quora, Clinical-QE, and SemEval-cQA). The Stanford Natural Language Inference corpus (SNLI) BIBREF13 contains 569,037 sentence pairs written by humans based on image captioning. The training set of the MultiNLI corpus BIBREF14 consists of 393,000 pairs of sentences from five genres of written and spoken English (e.g. Travel, Government). Two other "matched" and "mismatched" sets are also available for development (20,000 pairs). Both SNLI and multiNLI consider three types of relationships between sentences: entailment, neutral and contradiction. We converted the contradiction and neutral labels to the same non-entailment class. The QUORA dataset of similar questions was recently published with 404,279 question pairs. We randomly selected three distinct subsets (80%/10%/10%) for training (323,423 pairs), development (40,428 pairs) and test (40,428 pairs). The clinical-QE dataset BIBREF1 contains 8,588 question pairs and was constructed using 4,655 clinical questions asked by family doctors BIBREF45 . We randomly selected three distinct subsets (80%/10%/10%) for training (6,870 pairs), development (859 pairs) and test (859 pairs). The question similarity dataset of SemEval 2016 Task 3B (SemEval-cQA) BIBREF3 contains 3,869 question pairs and aims to re-rank a list of related questions according to their similarity to the original question. The same dataset was used for SemEval 2017 Task 3 BIBREF4 . To construct our test dataset, we used a publicly shared set of Consumer Health Questions (CHQs) received by the U.S. National Library of Medicine (NLM), and annotated with named entities, question types, and focus BIBREF46 , BIBREF47 . The CHQ dataset consists of 1,721 consumer information requests manually annotated with subquestions, each identified by a question type and a focus. First, we selected automatically harvested FAQs, from U.S. National Institutes of Health (NIH) websites, that share both the same focus and the same question type with the CHQs. As FAQs are most often very short, we first assume that the CHQ entails the FAQ. Two sets of pairs were constructed: (i) positive pairs of CHQs and FAQs sharing at least one common question type and the question focus, and (ii) negative pairs corresponding to a focus mismatch or type mismatch. For each category of negative examples, we randomly selected the same number of pairs for a balanced dataset. Then, we manually validated the constructed pairs and corrected the positive and negative labels when needed. The final RQE dataset contains 850 CHQ-FAQ pairs with 405 positive and 445 negative pairs. Table TABREF26 presents examples from the five training datasets (SNLI, MultiNLI, SemEval-cQA, Clinical-QE and Quora) and the new test dataset of medical CHQ-FAQ pairs. Results of RQE Approaches In the first experiment, we evaluated the DL and ML methods on SNLI, multi-NLI, Quora, and Clinical-QE. For the datasets that did not have a development and test sets, we randomly selected two sets, each amounting to 10% of the data, for test and development, and used the remaining 80% for training. For MultiNLI, we used the dev1-matched set for validation and the dev2-mismatched set for testing. Table TABREF28 presents the results of the first experiment. The DL model with GloVe word embeddings achieved better results on three datasets, with 82.80% Accuracy on SNLI, 78.52% Accuracy on MultiNLI, and 83.62% Accuracy on Quora. Logistic Regression achieved the best Accuracy of 98.60% on Clinical-RQE. We also performed a 10-fold cross-validation on the full Clinical-QE data of 8,588 question pairs, which gave 98.61% Accuracy. In the second experiment, we used these datasets for training only and compared their performance on our test set of 850 consumer health questions. Table TABREF29 presents the results of this experiment. Logistic Regression trained on the clinical-RQE data outperformed DL models trained on all datasets, with 73.18% Accuracy. To validate further the performance of the LR method, we evaluated it on question similarity detection. A typical approach to this task is to use an IR method to find similar question candidates, then a more sophisticated method to select and re-rank the similar questions. We followed a similar approach for this evaluation by combining the LR method with the IR baseline provided in the context of SemEval-cQA. The hybrid method combines the score provided by the Logistic Regression model and the reciprocal rank from the IR baseline using a weight-based combination: INLINEFORM0 The weight INLINEFORM0 was set empirically through several tests on the cQA-2016 development set ( INLINEFORM1 ). Table TABREF30 presents the results on the cQA-2016 and cQA-2017 test datasets. The hybrid method (LR+IR) provided the best results on both datasets. On the 2016 test data, the LR+IR method outperformed the best system in all measures, with 80.57% Accuracy and 77.47% MAP (official system ranking measure in SemEval-cQA). On the cQA-2017 test data, the LR+IR method obtained 44.66% MAP and outperformed the cQA-2017 best system in Accuracy with 67.27%. Discussion of RQE Results When trained and tested on the same corpus, the DL model with GloVe embeddings gave the best results on three datasets (SNLI, MultiNLI and Quora). Logistic Regression gave the best Accuracy on the Clinical-RQE dataset with 98.60%. When tested on our test set (850 medical CHQs-FAQs pairs), Logistic Regression trained on Clinical-QE gave the best performance with 73.18% Accuracy. The SNLI and multi-NLI models did not perform well when tested on medical RQE data. We performed additional evaluations using the RTE-1, RTE-2 and RTE-3 open-domain datasets provided by the PASCAL challenge and the results were similar. We have also tested the SemEval-cQA-2016 model and had a similar drop in performance on RQE data. This could be explained by the different types of data leading to wrong internal conceptualizations of medical terms and questions in the deep neural layers. This performance drop could also be caused by the complexity of the test consumer health questions that are often composed of several subquestions, contain contextual information, and may contain misspellings and ungrammatical sentences, which makes them more difficult to process BIBREF48 . Another aspect is the semantics of the task as discussed in Section SECREF6 . The definition of textual entailment in open-domain may not quite apply to question entailment due to the strict semantics. Also the general textual entailment definitions refer only to the premise and hypothesis, while the definition of RQE for question answering relies on the relationship between the sets of answers of the compared questions. Building a Medical QA Collection from Trusted Resources A RQE-based QA system requires a collection of question-answer pairs to map new user questions to the existing questions with an RQE approach, rank the retrieved questions, and present their answers to the user. Method To construct trusted medical question-answer pairs, we crawled websites from the National Institutes of Health (cf. Section SECREF56 ). Each web page describes a specific topic (e.g. name of a disease or a drug), and often includes synonyms of the main topic that we extracted during the crawl. We constructed hand-crafted patterns for each website to automatically generate the question-answer pairs based on the document structure and the section titles. We also annotated each question with the associated focus (topic of the web page) as well as the question type identified with the designed patterns (cf. Section SECREF36 ). To provide additional information about the questions that could be used for diverse IR and NLP tasks, we automatically annotated the questions with the focus, its UMLS Concept Unique Identifier (CUI) and Semantic Type. We combined two methods to recognize named entities from the titles of the crawled articles and their associated UMLS CUIs: (i) exact string matching to the UMLS Metathesaurus, and (ii) MetaMap Lite BIBREF49 . We then used the UMLS Semantic Network to retrieve the associated semantic types and groups. Question Types The question types were derived after the manual evaluation of 1,721 consumer health questions. Our taxonomy includes 16 types about Diseases, 20 types about Drugs and one type (Information) for the other named entities such as Procedures, Medical exams and Treatments. We describe below the considered question types and examples of associated question patterns. Question Types about Diseases (16): Information, Research (or Clinical Trial), Causes, Treatment, Prevention, Diagnosis (Exams and Tests), Prognosis, Complications, Symptoms, Inheritance, Susceptibility, Genetic changes, Frequency, Considerations, Contact a medical professional, Support Groups. Examples: What research (or clinical trial) is being done for DISEASE? What is the outlook for DISEASE? How many people are affected by DISEASE? When to contact a medical professional about DISEASE? Who is at risk for DISEASE? Where to find support for people with DISEASE? Question Types About Drugs (20): Information, Interaction with medications, Interaction with food, Interaction with herbs and supplements, Important warning, Special instructions, Brand names, How does it work, How effective is it, Indication, Contraindication, Learn more, Side effects, Emergency or overdose, Severe reaction, Forget a dose, Dietary, Why get vaccinated, Storage and disposal, Usage, Dose. Examples: Are there interactions between DRUG and herbs and supplements? What important warning or information should I know about DRUG? Are there safety concerns or special precautions about DRUG? What is the action of DRUG and how does it work? Who should get DRUG and why is it prescribed? What to do in case of a severe reaction to DRUG? Question Type for other medical entities (e.g. Procedure, Exam, Treatment): Information. What is Coronary Artery Bypass Surgery? What are Liver Function Tests? Medical Resources We used 12 trusted websites to construct a collection of question-answer pairs. For each website, we extracted the free text of each article as well as the synonyms of the article focus (topic). These resources and their brief descriptions are provided below: National Cancer Institute (NCI) : We extracted free text from 116 articles on various cancer types (729 QA pairs). We manually restructured the content of the articles to generate complete answers (e.g. a full answer about the treatment of all stages of a specific type of cancer). Figure FIGREF54 presents examples of QA pairs generated from a NCI article. Genetic and Rare Diseases Information Center (GARD): This resource contains information about various aspects of genetic/rare diseases. We extracted all disease question/answer pairs from 4,278 topics (5,394 QA pairs). Genetics Home Reference (GHR): This NLM resource contains consumer-oriented information about the effects of genetic variation on human health. We extracted 1,099 articles about diseases from this resource (5,430 QA pairs). MedlinePlus Health Topics: This portion of MedlinePlus contains information on symptoms, causes, treatment and prevention for diseases, health conditions and wellness issues. We extracted the free texts in summary sections of 981 articles (981 QA pairs). National Institute of Diabetes and Digestive and Kidney Diseases (NIDDK) : We extracted text from 174 health information pages on diseases studied by this institute (1,192 QA pairs). National Institute of Neurological Disorders and Stroke (NINDS): We extracted free text from 277 information pages on neurological and stroke-related diseases from this resource (1,104 QA pairs). NIHSeniorHealth : This website contains health and wellness information for older adults. We extracted 71 articles from this resource (769 QA pairs). National Heart, Lung, and Blood Institute (NHLBI) : We extracted text from 135 articles on diseases, tests, procedures, and other relevant topics on disorders of heart, lung, blood, and sleep (559 QA pairs). Centers for Disease Control and Prevention (CDC) : We extracted text from 152 articles on diseases and conditions (270 QA pairs). MedlinePlus A.D.A.M. Medical Encyclopedia: This resource contains 4,366 articles about conditions, tests, and procedures. 17,348 QA pairs were extracted from this resource. Figure FIGREF55 presents examples of QA pairs generated from A.D.A.M encyclopedia. MedlinePlus Drugs: We extracted free text from 1,316 articles about Drugs and generated 12,889 QA pairs. MedlinePlus Herbs and Supplements: We extracted free text from 99 articles and generated 792 QA pairs. The final collection contains 47,457 annotated question-answer pairs about Diseases, Drugs and other named entities (e.g. Tests) extracted from these 12 trusted resources. The Proposed Entailment-based QA System Our goal is to generate a ranked list of answers for a given Premise Question INLINEFORM0 by ranking the recognized Hypothesis Questions INLINEFORM1 . Based on the RQE experiments above (Section SECREF27 ), we selected Logistic Regression trained on the clinical-RQE dataset to recognize entailed questions and rank them with their classification scores. RQE-based QA Approach Classifying the full QA collection for each test question is not feasible for real-time applications. Therefore, we first filter the questions with an IR method to retrieve candidate questions, then classify them as entailed (or not) by the user/test question. Based on the positive results of the combination method tested on SemEval-cQA data (Section SECREF27 ), we adopted a combination method to merge the results obtained by the search engine and the RQE scores. The answers are then combined from both methods and ranked using an aggregate score. Figure FIGREF82 presents the overall architecture of the proposed QA system. We describe each module in more details next. Finding Similar Question Candidates For each premise question INLINEFORM0 , we use the Terrier search engine to retrieve INLINEFORM1 relevant question candidates INLINEFORM2 and then apply the RQE classifier to predict the labels for the pairs ( INLINEFORM3 , INLINEFORM4 ). We indexed the questions of our QA collection without the associated answers. In order to improve the indexing and the performance of question retrieval, we also indexed the synonyms of the question focus and the triggers of the question type with each question. This choice allowed us to avoid the shortcomings of query expansion, including incorrect or irrelevant synonyms and the increased execution time. The synonyms of the question focus (topic) were extracted automatically from the QA collection. The triggers of each question type were defined manually in the question types taxonomy. Below are two examples of indexed questions from our QA collection, with the automatically added focus synonyms and question type triggers: What are the treatments for Torticollis? Focus: Torticollis. Question type: Treatment. Added focus synonyms: "Spasmodic torticollis, Wry neck, Loxia, Cervical dystonia". Added question type triggers: "relieve, manage, cure, remedy, therapy". What is the outlook for Legionnaire disease? Focus: Legionnaire disease. Question Type: Prognosis. Added focus synonyms: "Legionella pneumonia, Pontiac fever, Legionellosis". Added question type triggers: "prognosis, life expectancy". The IR task consists of retrieving hypothesis questions INLINEFORM0 relevant to the submitted question INLINEFORM1 . As fusion of IR result has shown good performance in different tracks in TREC, we merge the results of the TF-IDF weighting function and the In-expB2 DFR model BIBREF50 . Let INLINEFORM0 = INLINEFORM1 , INLINEFORM2 , ..., INLINEFORM3 be the set of INLINEFORM4 questions retrieved by the first IR model INLINEFORM5 and INLINEFORM6 = INLINEFORM7 , INLINEFORM8 , ..., INLINEFORM9 be the set of INLINEFORM10 questions retrieved by the second IR model INLINEFORM11 . We merge both sets by summing the scores of each retrieved question INLINEFORM12 in both INLINEFORM13 and INLINEFORM14 lists, then we re-rank the hypothesis questions INLINEFORM15 . Combining IR and RQE Methods The IR models and the RQE Logistic Regression model bring different perspectives to the search for relevant candidate questions. In particular, question entailment allows understanding the relations between the important terms, whereas the traditional IR methods identify the important terms, but will not notice if the relations are opposite. Moreover, some of the question types that the RQE classifier learns will not be deemed important terms by traditional IR and the most relevant questions will not be ranked at the top of the list. Therefore, in our approach, when a question is submitted to the system, candidate questions are fetched using the IR models, then the RQE classifier is applied to filter out the non-entailed questions and re-rank the remaining candidates. Specifically, we denote INLINEFORM0 the list of question candidates INLINEFORM1 returned by the IR system. The premise question INLINEFORM2 is then used to construct N question pairs INLINEFORM3 . The RQE classifier is then applied to filter out the question pairs that are not entailed and re-rank the remaining pairs. More precisely, let INLINEFORM0 = INLINEFORM1 in INLINEFORM2 be the list of selected candidate questions that have a positive entailment relation with a given premise question INLINEFORM3 . We rank INLINEFORM4 by computing a hybrid score INLINEFORM5 for each candidate question INLINEFORM6 taking into account the score of the IR system INLINEFORM7 and the score of the RQE system INLINEFORM8 . For each system INLINEFORM0 INLINEFORM1 , we normalize the associated score by dividing it by the maximum score among the INLINEFORM2 candidate questions retrieved by INLINEFORM3 for INLINEFORM4 : INLINEFORM0 INLINEFORM0 INLINEFORM1 In our experiments, we fixed the value of INLINEFORM0 to 100. This threshold value was selected as a safe value for this task for the following reasons: Our collection of 47,457 question-answer pairs was collected from only 12 NIH institutes and is unlikely to contain more than 100 occurrences of the same focus-type pair. Each question was indexed with additional annotations for the question focus, its synonyms and the question type synonyms. Evaluating RQE for Medical Question Answering The objective of this evaluation is to study the effectiveness of RQE for Medical Question Answering, by comparing the answers retrieved by the hybrid entailment-based approach, the IR method and the other QA systems participating to the medical task at TREC 2017 LiveQA challenge (LiveQA-Med). Evaluation Method We developed an interface to perform the manual evaluation of the retrieved answers. Figure 5 presents the evaluation interface showing, for each test question, the top-10 answers of the evaluated QA method and the reference answer(s) used by LiveQA assessors to help judging the retrieved answers by the participating systems. We used the test questions of the medical task at TREC-2017 LiveQA BIBREF11 . These questions are randomly selected from the consumer health questions that the NLM receives daily from all over the world. The test questions cover different medical entities and have a wide list of question types such as Comparison, Diagnosis, Ingredient, Side effects and Tapering. For a relevant comparison, we used the same judgment scores as the LiveQA Track: Correct and Complete Answer (4) Correct but Incomplete (3) Incorrect but Related (2) Incorrect (1) We evaluated the answers returned by the IR-based method and the hybrid QA method (IR+RQE) according to the same reference answers used in LiveQA-Med. The answers were anonymized (the method names were blinded) and presented to 3 assessors: a medical doctor (Assessor A), a medical librarian (B) and a researcher in medical informatics (C). None of the assessors participated in the development of the QA methods. Assessors B and C evaluated 1,000 answers retrieved by each of the methods (IR and IR+RQE). Assessor A evaluated 2,000 answers from both methods. Table TABREF103 presents the inter-annotator agreement (IAA) through F1 score computed by considering one of the assessors as reference. In the first evaluation, we computed the True Positives (TP) and False Positives (FP) over all ratings and the Precision and F1 score. As there are no negative labels (only true or false positives for each category), Recall is 100%. We also computed a partial IAA by grouping the "Correct and Complete Answer" and "Correct but Incomplete" ratings (as Correct), and the "Incorrect but Related" and "Incorrect" ratings (as Incorrect). The average agreement on distinguishing the Correct and Incorrect answers is 94.33% F1 score. Therefore, we used the evaluations performed by assessor A for both methods. The official results of the TREC LiveQA track relied on one assessor per question as well. Evaluation of the first retrieved answer We computed the measures used by TREC LiveQA challenges BIBREF51 , BIBREF11 to evaluate the first retrieved answer for each test question: avgScore(0-3): the average score over all questions, transferring 1-4 level grades to 0-3 scores. This is the main score used to rank LiveQA runs. succ@i+: the number of questions with score i or above (i INLINEFORM0 {2..4}) divided by the total number of questions. prec@i+: the number of questions with score i or above (i INLINEFORM0 {2..4}) divided by number of questions answered by the system. Table TABREF108 presents the average scores, success and precision results. The hybrid IR+RQE QA system achieved better results than the IR-based system with 0.827 average score. It also achieved a higher score than the best results achieved in the medical challenge at LiveQA'17. Evaluating the RQE system alone is not relevant, as applying RQE on the full collection for each user question is not feasible for a real-time system because of the extended execution time. Evaluation of the top ten answers In this evaluation, we used Mean Average Precision (MAP) and Mean Reciprocal Rank (MRR) which are commonly used in QA to evaluate the top-10 answers for each question. We consider answers rated as “Correct and Complete Answer” or “Correct but Incomplete” as correct answers, as the test questions contain multiple subquestions while each answer in our QA collection can cover only one subquestion. MAP is the mean of the Average Precision (AvgP) scores over all questions. (1) INLINEFORM0 Q is the number of questions. INLINEFORM0 is the AvgP of the INLINEFORM1 question. INLINEFORM0 K is the number of correct answers. INLINEFORM0 is the rank of INLINEFORM1 correct answer. MRR is the average of the reciprocal ranks for each question. The reciprocal rank of a question is the multiplicative inverse of the rank of the first correct answer. (2) INLINEFORM0 Q is the number of questions. INLINEFORM0 is the rank of the first correct answer for the INLINEFORM1 question. Table TABREF113 presents the MAP@10 and MRR@10 of our QA methods. The IR+RQE system outperforms the IR-based QA system with 0.311 MAP@10 and 0.333 MRR@10. Discussion of entailment-based QA for the medical domain In our evaluation, we followed the same LiveQA guidelines with the highest possible rigor. In particular, we consulted with NIST assessors who provided us with the paraphrases of the test questions that they used to judge the answers. Our IAA on the answers rating was also high compared to related tasks, with an 88.5% F1 agreement with the exact four categories and a 94.3% agreement when reducing the categories to two: “Correct” and “Incorrect” answers. Our results show that RQE improves the overall performance and exceeds the best results in the medical LiveQA'17 challenge by a factor of 29.8%. This performance improvement is particularly interesting as: Our answer source has only 47K question-answer pairs when LiveQA participating systems relied on much larger collections, including the World Wide Web. Our system answered one subquestion at most when many LiveQA test questions had several subquestions. The latter observation, (b), makes the hybrid IR+RQE approach even more promising as it gives it a large potential for the improvement of answer completeness. The former observation, (a), provides another interesting insight: restricting the answer source to only reliable collections can actually improve the QA performance without losing coverage (i.e., our QA approach provided at least one answer to each test question and obtained the best relevance score). In another observation, the assessors reported that many of the returned answers had a correct question type but a wrong focus, which indicates that including a focus recognition module to filter such wrong answers can improve further the QA performance in terms of precision. Another aspect that was reported is the repetition of the same (or similar) answer from different websites, which could be addressed by improving answer selection with inter-answer comparisons and removal of near-duplicates. Also, half of the LiveQA test questions are about Drugs, when only two of our resources are specialized in Drugs, among 12 sub-collections overall. Accordingly, the assessors noticed that the performance of the QA systems was better on questions about diseases than on questions about drugs, which suggests a need for extending our medical QA collection with more information about drugs and associated question types. We also looked closely at the private websites used by the LiveQA-Med annotators to provide some of the reference answers for the test questions. For instance, the ConsumerLab website was useful to answer a question about the ingredients of a Drug (COENZYME Q10). Similarly, the eHealthMe website was used to answer a test question asking about interactions between two drugs (Phentermine and Dicyclomine) when no information was found in DailyMed. eHealthMe provides healthcare big data analysis and private research and studies including self-reported adverse drug effects by patients. But the question remains on the extent to which such big data and other private websites could be used to automatically answer medical questions if information is otherwise unavailable. Unlike medical professionals, patients do not necessarily have the knowledge and tools to validate such information. An alternative approach could be to put limitations on medical QA systems in terms of the questions that can be answered (e.g. "What is my diagnosis for such symptoms") and build classifiers to detect such questions and warn the users about the dangers of looking for their answers online. More generally, medical QA systems should follow some strict guidelines regarding the goal and background knowledge and resources of each system in order to protect the consumers from misleading or harmful information. Such guidelines could be based (i) on the source of the information such as health and medical information websites sponsored by the U.S. government, not-for-profit health or medical organizations, and medical university centers, or (ii) on conventions such as the code of conduct of the HON Foundation (HONcode) that addresses the reliability and usefulness of medical information on the Internet. Our experiments show that limiting the number of answer sources with such guidelines is not only feasible, but it could also enhance the performance of the QA system from an information retrieval perspective. Conclusion In this paper, we carried out an empirical study of machine learning and deep learning methods for Recognizing Question Entailment in the medical domain using several datasets. We developed a RQE-based QA system to answer new medical questions using existing question-answer pairs. We built and shared a collection of 47K medical question-answer pairs. Our QA approach outperformed the best results on TREC-2017 LiveQA medical test questions. The proposed approach can be applied and adapted to open-domain as well as specific-domain QA. Deep learning models achieved interesting results on open-domain and clinical datasets, but obtained a lower performance on consumer health questions. We will continue investigating other network architectures including transfer learning, as well as creation of a large collection of consumer health questions for training to improve the performance of DL models. Future work also includes exploring integration of a Question Focus Recognition module to enhance candidate question retrieval, and expanding our question-answer collection. Acknowledgements We thank Halil Kilicoglu (NLM/NIH) for his help with the crawling and the manual evaluation and Sonya E. Shooshan (NLM/NIH) for her help with the judgment of the retrieved answers. We also thank Ellen Voorhees (NIST) for her valuable support with the TREC LiveQA evaluation. We consider the case of the question number 36 in the TREC-2017 LiveQA medical test dataset: 36. congenital diaphragmatic hernia. what are the causes of congenital diaphragmatic hernia? Can cousin marriage cause this? What kind of lung disease the baby might experience life long? This question was answered by 5 participating runs (vs. 8 runs for other questions), and all submitted answers were wrong (scores of 1 or 2). However, our IR-based QA system retrieved one excellent answer (score 4) and our hybrid IR+RQE system provided 3 excellent answers. A) TREC 2017 LiveQA-Med Participants' Results: B) Our IR-based QA System: C) Our IR+RQE QA System:
What machine learning and deep learning methods are used for RQE?
d98847340e46ffe381992f1a594e75d3fb8d385e
What Ġmachine Ġlearning Ġand Ġdeep Ġlearning Ġmethods Ġare Ġused Ġfor ĠR Q E ?
Log istic ĠReg ression , Ġneural Ġnetworks
<s> What Ġmachine Ġlearning Ġand Ġdeep Ġlearning Ġmethods Ġare Ġused Ġfor ĠR Q E ? </s> Introduction </s> With Ġthe Ġavailability Ġof Ġrich Ġdata Ġon Ġusers ' Ġlocations , Ġprofiles Ġand Ġsearch Ġhistory , Ġpersonal ization Ġhas Ġbecome Ġthe Ġleading Ġtrend Ġin Ġlarge - scale Ġinformation Ġretrieval . ĠHowever , Ġefficiency Ġthrough Ġpersonal ization Ġis Ġnot Ġyet Ġthe Ġmost Ġsuitable Ġmodel Ġwhen Ġtackling Ġdomain - specific Ġsearches . ĠThis Ġis Ġdue Ġto Ġseveral Ġfactors , Ġsuch Ġas Ġthe Ġlex ical Ġand Ġsemantic Ġchallenges Ġof Ġdomain - specific Ġdata Ġthat Ġoften Ġinclude Ġadvanced Ġargument ation Ġand Ġcomplex Ġcontextual Ġinformation , Ġthe Ġhigher Ġsp ars eness Ġof Ġrelevant Ġinformation Ġsources , Ġand Ġthe Ġmore Ġpronounced Ġlack Ġof Ġsimilarities Ġbetween Ġusers ' Ġsearches . </s> A Ġrecent Ġstudy Ġon Ġexpert Ġsearch Ġstrategies Ġamong Ġhealthcare Ġinformation Ġprofessionals ĠB IB REF 0 Ġshowed Ġthat , Ġfor Ġa Ġgiven Ġsearch Ġtask , Ġthey Ġspend Ġan Ġaverage Ġof Ġ60 Ġminutes Ġper Ġcollection Ġor Ġdatabase , Ġ3 Ġminutes Ġto Ġexamine Ġthe Ġrelevance Ġof Ġeach Ġdocument , Ġand Ġ4 Ġhours Ġof Ġtotal Ġsearch Ġtime . ĠWhen Ġwritten Ġin Ġsteps , Ġtheir Ġsearch Ġstrategy Ġspans Ġover Ġ15 Ġlines Ġand Ġcan Ġreach Ġup Ġto Ġ105 Ġlines . </s> With Ġthe Ġabundance Ġof Ġinformation Ġsources Ġin Ġthe Ġmedical Ġdomain , Ġconsumers Ġare Ġmore Ġand Ġmore Ġfaced Ġwith Ġa Ġsimilar Ġchallenge , Ġone Ġthat Ġneeds Ġdedicated Ġsolutions Ġthat Ġcan Ġadapt Ġto Ġthe Ġheterogeneity Ġand Ġspecifics Ġof Ġhealth - related Ġinformation . </s> D ed icated ĠQuestion ĠAn sw ering Ġ( Q A ) Ġsystems Ġare Ġone Ġof Ġthe Ġviable Ġsolutions Ġto Ġthis Ġproblem Ġas Ġthey Ġare Ġdesigned Ġto Ġunderstand Ġnatural Ġlanguage Ġquestions Ġwithout Ġrelying Ġon Ġexternal Ġinformation Ġon Ġthe Ġusers . </s> In Ġthe Ġcontext Ġof ĠQ A , Ġthe Ġgoal Ġof ĠRecogn izing ĠQuestion ĠEnt ail ment Ġ( R Q E ) Ġis Ġto Ġretrieve Ġanswers Ġto Ġa Ġpremise Ġquestion Ġ( ĠIN LINE FORM 0 Ġ) Ġby Ġretrieving Ġinferred Ġor Ġent ailed Ġquestions , Ġcalled Ġhypothesis Ġquestions Ġ( ĠIN LINE FORM 1 Ġ) Ġthat Ġalready Ġhave Ġassociated Ġanswers . ĠTherefore , Ġwe Ġdefine Ġthe Ġentail ment Ġrelation Ġbetween Ġtwo Ġquestions Ġas : Ġa Ġquestion ĠIN LINE FORM 2 Ġentails Ġa Ġquestion ĠIN LINE FORM 3 Ġif Ġevery Ġanswer Ġto ĠIN LINE FORM 4 Ġis Ġalso Ġa Ġcorrect Ġanswer Ġto ĠIN LINE FORM 5 ĠB IB REF 1 Ġ. </s> R Q E Ġis Ġparticularly Ġrelevant Ġdue Ġto Ġthe Ġincreasing Ġnumbers Ġof Ġsimilar Ġquestions Ġposted Ġonline ĠB IB REF 2 Ġand Ġits Ġability Ġto Ġsolve Ġdifferently Ġthe Ġchallenging Ġissues Ġof Ġquestion Ġunderstanding Ġand Ġanswer Ġextraction . ĠIn Ġaddition Ġto Ġbeing Ġused Ġto Ġfind Ġrelevant Ġanswers , Ġthese Ġresources Ġcan Ġalso Ġbe Ġused Ġin Ġtraining Ġmodels Ġable Ġto Ġrecognize Ġinference Ġrelations Ġand Ġsimilarity Ġbetween Ġquestions . </s> Question Ġsimilarity Ġhas Ġrecently Ġattracted Ġinternational Ġchallenges ĠB IB REF 3 Ġ, ĠB IB REF 4 Ġand Ġseveral Ġresearch Ġefforts Ġproposing Ġa Ġwide Ġrange Ġof Ġapproaches , Ġincluding ĠLog istic ĠReg ression , ĠRec urrent ĠNeural ĠNetworks Ġ( R NN s ), ĠLong ĠShort ĠTerm ĠMemory Ġcells Ġ( L ST Ms ), Ġand ĠConv olution al ĠNeural ĠNetworks Ġ( CNN s ) ĠB IB REF 5 Ġ, ĠB IB REF 6 Ġ, ĠB IB REF 1 Ġ, ĠB IB REF 7 Ġ. </s> In Ġthis Ġpaper , Ġwe Ġstudy Ġquestion Ġentail ment Ġin Ġthe Ġmedical Ġdomain Ġand Ġthe Ġeffectiveness Ġof Ġthe Ġend - to - end ĠR Q E - based ĠQ A Ġapproach Ġby Ġevaluating Ġthe Ġrelevance Ġof Ġthe Ġretrieved Ġanswers . ĠAlthough Ġentail ment Ġwas Ġattempted Ġin ĠQ A Ġbefore ĠB IB REF 8 Ġ, ĠB IB REF 9 Ġ, ĠB IB REF 10 Ġ, Ġas Ġfar Ġas Ġwe Ġknow , Ġwe Ġare Ġthe Ġfirst Ġto Ġintroduce Ġand Ġevaluate Ġa Ġfull Ġmedical Ġquestion Ġanswering Ġapproach Ġbased Ġon Ġquestion Ġentail ment Ġfor Ġfree - text Ġquestions . ĠOur Ġcontributions Ġare : </s> The Ġnext Ġsection Ġis Ġdedicated Ġto Ġrelated Ġwork Ġon Ġquestion Ġanswering , Ġquestion Ġsimilarity Ġand Ġentail ment . ĠIn ĠSection ĠSEC REF 3 Ġ, Ġwe Ġpresent Ġtwo Ġmachine Ġlearning Ġ( ML ) Ġand Ġdeep Ġlearning Ġ( DL ) Ġmethods Ġfor ĠR Q E Ġand Ġcompare Ġtheir Ġperformance Ġusing Ġopen - domain Ġand Ġclinical Ġdatasets . ĠSection ĠSEC REF 4 Ġdescribes Ġthe Ġnew Ġcollection Ġof Ġmedical Ġquestion - answer Ġpairs . ĠIn ĠSection ĠSEC REF 5 Ġ, Ġwe Ġdescribe Ġour ĠR Q E - based Ġapproach Ġfor ĠQ A . ĠSection ĠSEC REF 6 Ġpresents Ġour Ġevaluation Ġof Ġthe Ġretrieved Ġanswers Ġand Ġthe Ġresults Ġobtained Ġon ĠT REC Ġ2017 ĠLive Q A Ġmedical Ġquestions . </s> Background </s> In Ġthis Ġsection Ġwe Ġdefine Ġthe ĠR Q E Ġtask Ġand Ġdescribe Ġrelated Ġwork Ġat Ġthe Ġintersection Ġof Ġquestion Ġanswering , Ġquestion Ġsimilarity Ġand Ġtextual Ġinference . </s> Task ĠDefinition </s> The Ġdefinition Ġof ĠRecogn izing ĠQuestion ĠEnt ail ment Ġ( R Q E ) Ġcan Ġhave Ġa Ġsignificant Ġimpact Ġon ĠQ A Ġresults . ĠIn Ġrelated Ġwork , Ġthe Ġmeaning Ġassociated Ġwith ĠNatural ĠLanguage ĠIn ference Ġ( N LI ) Ġvaries Ġamong Ġdifferent Ġtasks Ġand Ġevents . ĠFor Ġinstance , ĠRecogn izing ĠText ual ĠEnt ail ment Ġ( R TE ) Ġwas Ġaddressed Ġby Ġthe ĠP ASC AL Ġchallenge ĠB IB REF 12 Ġ, Ġwhere Ġthe Ġentail ment Ġrelation Ġhas Ġbeen Ġassessed Ġmanually Ġby Ġhuman Ġjudges Ġwho Ġselected Ġrelevant Ġsentences Ġ" ent ailing " Ġa Ġset Ġof Ġhypotheses Ġfrom Ġa Ġlist Ġof Ġdocuments Ġreturned Ġby Ġdifferent ĠInformation ĠRet ri eval Ġ( IR ) Ġmethods . ĠIn Ġanother Ġdefinition , Ġthe ĠStanford ĠNatural ĠLanguage ĠIn ference Ġcorpus ĠSN LI ĠB IB REF 13 Ġ, Ġused Ġthree Ġclassification Ġlabels Ġfor Ġthe Ġrelations Ġbetween Ġtwo Ġsentences : Ġentail ment , Ġneutral Ġand Ġcontradiction . ĠFor Ġthe Ġentail ment Ġlabel , Ġthe Ġannot ators Ġwho Ġbuilt Ġthe Ġcorpus Ġwere Ġpresented Ġwith Ġan Ġimage Ġand Ġasked Ġto Ġwrite Ġa Ġcaption ĠâĢ ľ that Ġis Ġa Ġdefinitely Ġtrue Ġdescription Ġof Ġthe Ġphoto âĢ Ŀ . ĠFor Ġthe Ġneutral Ġlabel , Ġthey Ġwere Ġasked Ġto Ġprovide Ġa Ġcaption ĠâĢ ľ that Ġmight Ġbe Ġa Ġtrue Ġdescription Ġof Ġthe Ġlabel âĢ Ŀ . ĠThey Ġwere Ġasked Ġfor Ġa Ġcaption Ġthat ĠâĢ ľ is Ġdefinitely Ġa Ġfalse Ġdescription Ġof Ġthe Ġphoto âĢ Ŀ Ġfor Ġthe Ġcontradiction Ġlabel . </s> More Ġrecently , Ġthe Ġmulti N LI Ġcorpus ĠB IB REF 14 Ġwas Ġshared Ġin Ġthe Ġscope Ġof Ġthe ĠRep E val Ġ2017 Ġshared Ġtask ĠB IB REF 15 Ġ. ĠTo Ġbuild Ġthe Ġcorpus , Ġannot ators Ġwere Ġpresented Ġwith Ġa Ġpremise Ġtext Ġand Ġasked Ġto Ġwrite Ġthree Ġsentences . ĠOne Ġnovel Ġsentence , Ġwhich Ġis ĠâĢ ľ necess arily Ġtrue Ġor Ġappropriate Ġin Ġthe Ġsame Ġsituations Ġas Ġthe Ġpremise , âĢ Ŀ Ġfor Ġthe Ġentail ment Ġlabel , Ġa Ġsentence , Ġwhich Ġis ĠâĢ ľ necess arily Ġfalse Ġor Ġinappropriate Ġwhenever Ġthe Ġpremise Ġis Ġtrue , âĢ Ŀ Ġfor Ġthe Ġcontradiction Ġlabel , Ġand Ġa Ġlast Ġsentence , ĠâĢ ľ where Ġneither Ġcondition Ġapplies , âĢ Ŀ Ġfor Ġthe Ġneutral Ġlabel . </s> Whereas Ġthese ĠNL I Ġdefinitions Ġmight Ġbe Ġsuitable Ġfor Ġthe Ġbroad Ġtopic Ġof Ġtext Ġunderstanding , Ġtheir Ġrelation Ġto Ġpractical Ġinformation Ġretrieval Ġor Ġquestion Ġanswering Ġsystems Ġis Ġnot Ġstraightforward . </s> In Ġcontrast , ĠR Q E Ġhas Ġto Ġbe Ġtailored Ġto Ġthe Ġquestion Ġanswering Ġtask . ĠFor Ġinstance , Ġif Ġthe Ġpremise Ġquestion Ġis Ġ" looking Ġfor Ġcold Ġmedications Ġfor Ġa Ġ30 Ġyo Ġwoman ", Ġa ĠR Q E Ġapproach Ġshould Ġbe Ġable Ġto Ġconsider Ġthe Ġmore Ġgeneral Ġ( less Ġrestricted ) Ġquestion Ġ" looking Ġfor Ġcold Ġmedications " Ġas Ġrelevant , Ġsince Ġits Ġanswers Ġare Ġrelevant Ġfor Ġthe Ġinitial Ġquestion , Ġwhereas Ġ" looking Ġfor Ġmedications Ġfor Ġa Ġ30 Ġyo Ġwoman " Ġis Ġa Ġuseless Ġcontextual ization . ĠThe Ġentail ment Ġrelation Ġwe Ġare Ġseeking Ġin Ġthe ĠQ A Ġcontext Ġshould Ġinclude Ġrelevant Ġand Ġmeaningful Ġrelax ations Ġof Ġcontextual Ġand Ġsemantic Ġconstraints Ġ( cf . ĠSection ĠSEC REF 13 Ġ). </s> Related ĠWork Ġon ĠQuestion ĠAn sw ering </s> Class ical ĠQ A Ġsystems Ġface Ġtwo Ġmain Ġchallenges Ġrelated Ġto Ġquestion Ġanalysis Ġand Ġanswer Ġextraction . ĠSeveral ĠQ A Ġapproaches Ġwere Ġproposed Ġin Ġthe Ġliterature Ġfor Ġthe Ġopen Ġdomain ĠB IB REF 16 Ġ, ĠB IB REF 17 Ġand Ġthe Ġmedical Ġdomain ĠB IB REF 18 Ġ, ĠB IB REF 19 Ġ, ĠB IB REF 20 Ġ. ĠA Ġvariety Ġof Ġmethods Ġwere Ġdeveloped Ġfor Ġquestion Ġanalysis , Ġfocus Ġ( topic ) Ġrecognition Ġand Ġquestion Ġtype Ġidentification ĠB IB REF 21 Ġ, ĠB IB REF 22 Ġ, ĠB IB REF 23 Ġ, ĠB IB REF 24 Ġ. ĠSimilarly , Ġmany Ġdifferent Ġapproaches Ġtackled Ġdocument Ġor Ġpassage Ġretrieval Ġand Ġanswer Ġselection Ġand Ġ( re ) ranking ĠB IB REF 25 Ġ, ĠB IB REF 26 Ġ, ĠB IB REF 27 Ġ. </s> An Ġalternative Ġapproach Ġconsists Ġin Ġfinding Ġsimilar Ġquestions Ġor ĠFAQ s Ġthat Ġare Ġalready Ġanswered ĠB IB REF 28 Ġ, ĠB IB REF 29 Ġ. ĠOne Ġof Ġthe Ġearliest Ġquestion Ġanswering Ġsystems Ġbased Ġon Ġfinding Ġsimilar Ġquestions Ġand Ġre - using Ġthe Ġexisting Ġanswers Ġwas ĠFAQ ĠF IND ER ĠB IB REF 30 Ġ. ĠAnother Ġsystem Ġthat Ġcomple ments Ġthe Ġexisting ĠQ & A Ġservices Ġof ĠNet Well ness Ġis ĠSim Q ĠB IB REF 2 Ġ, Ġwhich Ġallows Ġretrieval Ġof Ġsimilar Ġweb - based Ġconsumer Ġhealth Ġquestions . ĠSim Q Ġuses Ġsynt actic Ġand Ġsemantic Ġfeatures Ġto Ġcompute Ġsimilarity Ġbetween Ġquestions , Ġand ĠU ML S ĠB IB REF 31 Ġas Ġa Ġstandardized Ġsemantic Ġknowledge Ġsource . ĠThe Ġsystem Ġachieves Ġ72 . 2 % Ġprecision , Ġ78 . 0 % Ġrecall Ġand Ġ75 . 0 % ĠF - score Ġon ĠNet Well ness Ġquestions . ĠHowever , Ġthe Ġmethod Ġwas Ġevaluated Ġonly Ġon Ġone Ġquestion Ġsimilarity Ġdataset , Ġand Ġthe Ġretrieved Ġanswers Ġwere Ġnot Ġevaluated . </s> The Ġaim Ġof Ġthe Ġmedical Ġtask Ġat ĠT REC Ġ2017 ĠLive Q A Ġwas Ġto Ġdevelop Ġtechniques Ġfor Ġanswering Ġcomplex Ġquestions Ġsuch Ġas Ġconsumer Ġhealth Ġquestions , Ġas Ġwell Ġas Ġto Ġidentify Ġrelevant Ġanswer Ġsources Ġthat Ġcan Ġcomply Ġwith Ġthe Ġsensitivity Ġof Ġmedical Ġinformation Ġretrieval . </s> The ĠCM U - OA Q A Ġsystem ĠB IB REF 32 Ġachieved Ġthe Ġbest Ġperformance Ġof Ġ0 . 6 37 Ġaverage Ġscore Ġon Ġthe Ġmedical Ġtask Ġby Ġusing Ġan Ġattention al Ġenc oder - dec oder Ġmodel Ġfor Ġparaph rase Ġidentification Ġand Ġanswer Ġranking . ĠThe ĠQu ora Ġquestion - similar ity Ġdataset Ġwas Ġused Ġfor Ġtraining . ĠThe ĠPR NA Ġsystem ĠB IB REF 33 Ġachieved Ġthe Ġsecond Ġbest Ġperformance Ġin Ġthe Ġmedical Ġtask Ġwith Ġ0 . 49 Ġaverage Ġscore Ġusing ĠWikipedia Ġas Ġthe Ġfirst Ġanswer Ġsource Ġand ĠYahoo Ġand ĠGoogle Ġsearches Ġas Ġsecondary Ġanswer Ġsources . ĠEach Ġmedical Ġquestion Ġwas Ġdecomp osed Ġinto Ġseveral Ġsub quest ions . ĠTo Ġextract Ġthe Ġanswer Ġfrom Ġthe Ġselected Ġtext Ġpassage , Ġa Ġbi - direction al Ġattention Ġmodel Ġtrained Ġon Ġthe ĠSQU AD Ġdataset Ġwas Ġused . </s> Deep Ġneural Ġnetwork Ġmodels Ġhave Ġbeen Ġpushing Ġthe Ġlimits Ġof Ġperformance Ġachieved Ġin ĠQ A Ġrelated Ġtasks Ġusing Ġlarge Ġtraining Ġdatasets . ĠThe Ġresults Ġobtained Ġby ĠCM U - OA Q A Ġand ĠPR NA Ġshowed Ġthat Ġlarge Ġopen - domain Ġdatasets Ġwere Ġbeneficial Ġfor Ġthe Ġmedical Ġdomain . ĠHowever , Ġthe Ġbest Ġsystem Ġ( CM U - OA Q A ) Ġrelying Ġon Ġthe Ġsame Ġtraining Ġdata Ġobtained Ġa Ġscore Ġof Ġ1 . 139 Ġon Ġthe ĠLive Q A Ġopen - domain Ġtask . </s> While Ġthis Ġgap Ġin Ġperformance Ġcan Ġbe Ġexplained Ġin Ġpart Ġby Ġthe Ġdiscrepancies Ġbetween Ġthe Ġmedical Ġtest Ġquestions Ġand Ġthe Ġopen - domain Ġquestions , Ġit Ġalso Ġhighlights Ġthe Ġneed Ġfor Ġlarger Ġmedical Ġdatasets Ġto Ġsupport Ġdeep Ġlearning Ġapproaches Ġin Ġdealing Ġwith Ġthe Ġlinguistic Ġcomplexity Ġof Ġconsumer Ġhealth Ġquestions Ġand Ġthe Ġchallenge Ġof Ġfinding Ġcorrect Ġand Ġcomplete Ġanswers . </s> Another Ġtechnique Ġwas Ġused Ġby ĠEC N U - ICA Ġteam ĠB IB REF 34 Ġbased Ġon Ġlearning Ġquestion Ġsimilarity Ġvia Ġtwo Ġlong Ġshort - term Ġmemory Ġ( L ST M ) Ġnetworks Ġapplied Ġto Ġobtain Ġthe Ġsemantic Ġrepresentations Ġof Ġthe Ġquestions . ĠTo Ġconstruct Ġa Ġcollection Ġof Ġsimilar Ġquestion Ġpairs , Ġthey Ġsearched Ġcommunity Ġquestion Ġanswering Ġsites Ġsuch Ġas ĠYahoo ! Ġand ĠAnswers . com . ĠIn Ġcontrast , Ġthe ĠEC N U - ICA Ġsystem Ġachieved Ġthe Ġbest Ġperformance Ġof Ġ1 . 8 95 Ġin Ġthe Ġopen - domain Ġtask Ġbut Ġan Ġaverage Ġscore Ġof Ġonly Ġ0 . 402 Ġin Ġthe Ġmedical Ġtask . ĠAs Ġthe ĠEC N U - ICA Ġapproach Ġalso Ġrelied Ġon Ġa Ġneural Ġnetwork Ġfor Ġquestion Ġmatching , Ġthis Ġresult Ġshows Ġthat Ġtraining Ġattention - based Ġdec oder - enc oder Ġnetworks Ġon Ġthe ĠQu ora Ġdataset Ġgeneralized Ġbetter Ġto Ġthe Ġmedical Ġdomain Ġthan Ġtraining ĠL ST Ms Ġon Ġsimilar Ġquestions Ġfrom ĠYahoo ! Ġand ĠAnswers . com . </s> The ĠCM U - Live Med Q A Ġteam ĠB IB REF 20 Ġdesigned Ġa Ġspecific Ġsystem Ġfor Ġthe Ġmedical Ġtask . ĠUsing Ġonly Ġthe Ġprovided Ġtraining Ġdatasets Ġand Ġthe Ġassumption Ġthat Ġeach Ġquestion Ġcontains Ġonly Ġone Ġfocus , Ġthe ĠCM U - Live Med Q A Ġsystem Ġobtained Ġan Ġaverage Ġscore Ġof Ġ0 . 353 . ĠThey Ġused Ġa Ġconv olution al Ġneural Ġnetwork Ġ( CNN ) Ġmodel Ġto Ġclassify Ġa Ġquestion Ġinto Ġa Ġrestricted Ġset Ġof Ġ10 Ġquestion Ġtypes Ġand Ġcrawled Ġ" relevant " Ġonline Ġweb Ġpages Ġto Ġfind Ġthe Ġanswers . ĠHowever , Ġthe Ġresults Ġwere Ġlower Ġthan Ġthose Ġachieved Ġby Ġthe Ġsystems Ġrelying Ġon Ġfinding Ġsimilar Ġanswered Ġquestions . ĠThese Ġresults Ġsupport Ġthe Ġrelevance Ġof Ġsimilar Ġquestion Ġmatching Ġfor Ġthe Ġend - to - end ĠQ A Ġtask Ġas Ġa Ġnew Ġway Ġof Ġapproaching ĠQ A Ġinstead Ġof Ġthe Ġclassical ĠQ A Ġapproaches Ġbased Ġon ĠQuestion ĠAnalysis Ġand ĠAnswer ĠRet ri eval . </s> Related ĠWork Ġon ĠQuestion ĠSimilar ity Ġand ĠEnt ail ment </s> Several Ġefforts Ġfocused Ġon Ġrecognizing Ġsimilar Ġquestions . ĠJe on Ġet Ġal . ĠB IB REF 35 Ġshowed Ġthat Ġa Ġretrieval Ġmodel Ġbased Ġon Ġtranslation Ġprobabilities Ġlearned Ġfrom Ġa Ġquestion Ġand Ġanswer Ġarchive Ġcan Ġrecognize Ġsem antically Ġsimilar Ġquestions . ĠD uan Ġet Ġal . ĠB IB REF 36 Ġproposed Ġa Ġdedicated Ġlanguage Ġmodeling Ġapproach Ġfor Ġquestion Ġsearch , Ġusing Ġquestion Ġtopic Ġ( user 's Ġinterest ) Ġand Ġquestion Ġfocus Ġ( certain Ġaspect Ġof Ġthe Ġtopic ). </s> L ately , Ġthese Ġefforts Ġwere Ġsupported Ġby Ġa Ġtask Ġon ĠQuestion - Question Ġsimilarity Ġintroduced Ġin Ġthe Ġcommunity ĠQ A Ġchallenge Ġat ĠSem E val Ġ( task Ġ3 B ) ĠB IB REF 3 Ġ. ĠGiven Ġa Ġnew Ġquestion , Ġthe Ġtask Ġfocused Ġon Ġre ranking Ġall Ġsimilar Ġquestions Ġretrieved Ġby Ġa Ġsearch Ġengine , Ġassuming Ġthat Ġthe Ġanswers Ġto Ġthe Ġsimilar Ġquestions Ġwill Ġbe Ġcorrect Ġanswers Ġfor Ġthe Ġnew Ġquestion . ĠDifferent Ġmachine Ġlearning Ġand Ġdeep Ġlearning Ġapproaches Ġwere Ġtested Ġin Ġthe Ġscope Ġof ĠSem E val Ġ2016 ĠB IB REF 3 Ġand Ġ2017 ĠB IB REF 4 Ġtask Ġ3 B . ĠThe Ġbest Ġperforming Ġsystem Ġin Ġ2017 Ġachieved Ġa ĠMAP Ġof Ġ47 . 22 % Ġusing Ġsupervised ĠLog istic ĠReg ression Ġthat Ġcombined Ġdifferent Ġun super vised Ġsimilarity Ġmeasures Ġsuch Ġas ĠCos ine Ġand ĠSoft - Cos ine ĠB IB REF 37 Ġ. ĠThe Ġsecond Ġbest Ġsystem Ġachieved Ġ46 . 93 % ĠMAP Ġwith Ġa Ġlearning - to - rank Ġmethod Ġusing ĠLog istic ĠReg ression Ġand Ġa Ġrich Ġset Ġof Ġfeatures Ġincluding Ġlex ical Ġand Ġsemantic Ġfeatures Ġas Ġwell Ġas Ġembed d ings Ġgenerated Ġby Ġdifferent Ġneural Ġnetworks Ġ( si ames e , ĠBi - L ST M , ĠGR U Ġand ĠCNN s ) ĠB IB REF 38 Ġ. ĠIn Ġthe Ġscope Ġof Ġthis Ġchallenge , Ġa Ġdataset Ġwas Ġcollected Ġfrom ĠQatar ĠLiving Ġforum Ġfor Ġtraining . ĠWe Ġrefer Ġto Ġthis Ġdataset Ġas ĠSem E val - c Q A . </s> In Ġanother Ġeffort , Ġan Ġanswer - based Ġdefinition Ġof ĠR Q E Ġwas Ġproposed Ġand Ġtested ĠB IB REF 1 Ġ. ĠThe Ġauthors Ġintroduced Ġa Ġdataset Ġof Ġclinical Ġquestions Ġand Ġused Ġa Ġfeature - based Ġmethod Ġthat Ġprovided Ġan ĠAccuracy Ġof Ġ75 % Ġon Ġconsumer Ġhealth Ġquestions . ĠWe Ġwill Ġcall Ġthis Ġdataset ĠClinical - Q E . ĠDos ĠSantos Ġet Ġal . ĠB IB REF 5 Ġproposed Ġa Ġnew Ġapproach Ġto Ġretrieve Ġsem antically Ġequivalent Ġquestions Ġcombining Ġa Ġbag - of - words Ġrepresentation Ġwith Ġa Ġdistributed Ġvector Ġrepresentation Ġcreated Ġby Ġa ĠCNN Ġand Ġuser Ġdata Ġcollected Ġfrom Ġtwo ĠStack ĠExchange Ġcommunities . ĠLei Ġet Ġal . ĠB IB REF 7 Ġproposed Ġa Ġrecurrent Ġand Ġconv olution al Ġmodel Ġ( g ated Ġconv olution ) Ġto Ġmap Ġquestions Ġto Ġtheir Ġsemantic Ġrepresentations . ĠThe Ġmodels Ġwere Ġpre - trained Ġwithin Ġan Ġenc oder - dec oder Ġframework . </s> R Q E ĠAppro aches Ġand ĠExper iments </s> The Ġchoice Ġof Ġtwo Ġmethods Ġfor Ġour Ġempirical Ġstudy Ġis Ġmotivated Ġby Ġthe Ġbest Ġperformance Ġachieved Ġby ĠLog istic ĠReg ression Ġin Ġquestion - question Ġsimilarity Ġat ĠSem E val Ġ2017 Ġ( best Ġsystem ĠB IB REF 37 Ġand Ġsecond Ġbest Ġsystem ĠB IB REF 38 Ġ), Ġand Ġthe Ġhigh Ġperformance Ġachieved Ġby Ġneural Ġnetworks Ġon Ġlarger Ġdatasets Ġsuch Ġas ĠSN LI ĠB IB REF 13 Ġ, ĠB IB REF 39 Ġ, ĠB IB REF 40 Ġ, ĠB IB REF 41 Ġ. ĠWe Ġfirst Ġdefine Ġthe ĠR Q E Ġtask , Ġthen Ġpresent Ġthe Ġtwo Ġapproaches , Ġand Ġevaluate Ġtheir Ġperformance Ġon Ġfive Ġdifferent Ġdatasets . </s> Definition </s> In Ġthe Ġcontext Ġof ĠQ A , Ġthe Ġgoal Ġof ĠR Q E Ġis Ġto Ġretrieve Ġanswers Ġto Ġa Ġnew Ġquestion Ġby Ġretrieving Ġent ailed Ġquestions Ġwith Ġassociated Ġanswers . ĠWe Ġtherefore Ġdefine Ġquestion Ġentail ment Ġas : </s> a Ġquestion ĠIN LINE FORM 0 Ġentails Ġa Ġquestion ĠIN LINE FORM 1 Ġif Ġevery Ġanswer Ġto ĠIN LINE FORM 2 Ġis Ġalso Ġa Ġcomplete Ġor Ġpartial Ġanswer Ġto ĠIN LINE FORM 3 Ġ. </s> We Ġpresent Ġbelow Ġtwo Ġexamples Ġof Ġconsumer Ġhealth Ġquestions ĠIN LINE FORM 0 Ġand Ġent ailed Ġquestions ĠIN LINE FORM 1 Ġ: </s> Example Ġ1 Ġ( each Ġanswer Ġto Ġthe Ġent ailed Ġquestion ĠB 1 Ġis Ġa Ġcomplete Ġanswer Ġto ĠA 1 ): </s> A 1 : ĠWhat Ġis Ġthe Ġlatest Ġnews Ġon Ġt enn itis , Ġor Ġringing Ġin Ġthe Ġear , ĠI Ġam Ġ75 Ġyears Ġold Ġand Ġhave Ġhad Ġringing Ġin Ġthe Ġear Ġsince Ġmy Ġmid Ġ5 os . ĠThank Ġyou . </s> B 1 : ĠWhat Ġis Ġthe Ġlatest Ġresearch Ġon ĠT inn itus ? </s> Example Ġ2 Ġ( each Ġanswer Ġto Ġthe Ġent ailed Ġquestion ĠB 2 Ġis Ġa Ġpartial Ġanswer Ġto ĠA 2 ): </s> A 2 : ĠMy Ġmother Ġhas Ġbeen Ġdiagnosed Ġwith ĠAlzheimer 's , Ġmy Ġfather Ġis Ġnot Ġof Ġthe Ġgreatest Ġhealth Ġeither Ġand Ġis Ġthe Ġmain Ġcareg iver Ġfor Ġmy Ġmother . ĠMy Ġquestion Ġis Ġwhere Ġdo Ġwe Ġstart Ġwith Ġattempting Ġto Ġhelp Ġour Ġparents Ġw / Ġthe Ġcare Ġgiving Ġand Ġwhat Ġsort Ġof Ġfinancial Ġoptions Ġare Ġthere Ġout Ġthere Ġfor Ġpeople Ġon Ġfixed Ġincomes . </s> B 2 : ĠWhat Ġresources Ġare Ġavailable Ġfor ĠAlzheimer 's Ġcaregivers ? </s> The Ġinclusion Ġof Ġpartial Ġanswers Ġin Ġthe Ġdefinition Ġof Ġquestion Ġentail ment Ġalso Ġallows Ġefficient Ġrelaxation Ġof Ġthe Ġcontextual Ġconstraints Ġof Ġthe Ġoriginal Ġquestion ĠIN LINE FORM 0 Ġto Ġretrieve Ġrelevant Ġanswers Ġfrom Ġent ailed , Ġbut Ġless Ġrestricted , Ġquestions . </s> Deep ĠLearning ĠModel </s> To Ġrecognize Ġentail ment Ġbetween Ġtwo Ġquestions ĠIN LINE FORM 0 Ġ( prem ise ) Ġand ĠIN LINE FORM 1 Ġ( hyp ot hesis ), Ġwe Ġadapted Ġthe Ġneural Ġnetwork Ġproposed Ġby ĠBowman Ġet Ġal . ĠB IB REF 13 Ġ. ĠOur ĠDL Ġmodel , Ġpresented Ġin ĠFigure ĠFIG REF 20 Ġ, Ġconsists Ġof Ġthree Ġ600 d ĠRe LU Ġlayers , Ġwith Ġa Ġbottom Ġlayer Ġtaking Ġthe Ġconc aten ated Ġsentence Ġrepresentations Ġas Ġinput Ġand Ġa Ġtop Ġlayer Ġfeeding Ġa Ġsoft max Ġclass ifier . ĠThe Ġsentence Ġembed ding Ġmodel Ġsums Ġthe ĠRec urrent Ġneural Ġnetwork Ġ( R NN ) Ġembed d ings Ġof Ġits Ġwords . ĠThe Ġword Ġembed d ings Ġare Ġfirst Ġinitialized Ġwith Ġpret rained ĠGl o Ve Ġvectors . ĠThis Ġadaptation Ġprovided Ġthe Ġbest Ġperformance Ġin Ġprevious Ġexperiments Ġwith ĠR Q E Ġdata . </s> G lo Ve Ġis Ġan Ġun super vised Ġlearning Ġalgorithm Ġto Ġgenerate Ġvector Ġrepresentations Ġfor Ġwords ĠB IB REF 42 Ġ. ĠTraining Ġis Ġperformed Ġon Ġaggreg ated Ġword Ġco - occ urrence Ġstatistics Ġfrom Ġa Ġlarge Ġcorpus , Ġand Ġthe Ġresulting Ġrepresentations Ġshow Ġinteresting Ġlinear Ġsubst ruct ures Ġof Ġthe Ġword Ġvector Ġspace . ĠWe Ġuse Ġthe Ġpret rained Ġcommon Ġcrawl Ġversion Ġwith Ġ840 B Ġtokens Ġand Ġ300 d Ġvectors , Ġwhich Ġare Ġnot Ġupdated Ġduring Ġtraining . </s> Log istic ĠReg ression ĠClass ifier </s> In Ġthis Ġfeature - based Ġapproach , Ġwe Ġuse ĠLog istic ĠReg ression Ġto Ġclassify Ġquestion Ġpairs Ġinto Ġentail ment Ġor Ġno - ent ail ment . ĠLog istic ĠReg ression Ġachieved Ġgood Ġresults Ġon Ġthis Ġspecific Ġtask Ġand Ġoutper formed Ġother Ġstatistical Ġlearning Ġalgorithms Ġsuch Ġas ĠS VM Ġand ĠNa ive ĠBay es . ĠIn Ġa Ġpre processing Ġstep , Ġwe Ġremove Ġstop Ġwords Ġand Ġperform Ġword Ġstemming Ġusing Ġthe ĠPorter Ġalgorithm ĠB IB REF 43 Ġfor Ġall Ġ( ĠIN LINE FORM 0 Ġ, ĠIN LINE FORM 1 Ġ) Ġpairs . </s> We Ġuse Ġa Ġlist Ġof Ġnine Ġfeatures , Ġselected Ġafter Ġseveral Ġexperiments Ġon ĠR TE Ġdatasets ĠB IB REF 12 Ġ. ĠWe Ġcompute Ġfive Ġsimilarity Ġmeasures Ġbetween Ġthe Ġpre - process ed Ġquestions Ġand Ġuse Ġtheir Ġvalues Ġas Ġfeatures . ĠWe Ġuse ĠWord ĠOver lap , Ġthe ĠDice Ġcoefficient Ġbased Ġon Ġthe Ġnumber Ġof Ġcommon Ġbig rams , ĠCos ine , ĠLe ven sh tein , Ġand Ġthe ĠJ acc ard Ġsimilarities . ĠOur Ġfeature Ġlist Ġalso Ġincludes Ġthe Ġmaximum Ġand Ġaverage Ġvalues Ġobtained Ġwith Ġthese Ġmeasures Ġand Ġthe Ġquestion Ġlength Ġratio Ġ( length ( ĠIN LINE FORM 0 Ġ) / length ( ĠIN LINE FORM 1 Ġ) ). ĠWe Ġcompute Ġa Ġmorph os ynt actic Ġfeature Ġindicating Ġthe Ġnumber Ġof Ġcommon Ġnoun s Ġand Ġverbs Ġbetween ĠIN LINE FORM 2 Ġand ĠIN LINE FORM 3 Ġ. ĠTree T agger ĠB IB REF 44 Ġwas Ġused Ġfor ĠPOS Ġtagging . </s> For ĠR Q E , Ġwe Ġadd Ġan Ġadditional Ġfeature Ġspecific Ġto Ġthe Ġquestion Ġtype . ĠWe Ġuse Ġa Ġdictionary Ġlookup Ġto Ġmap Ġtriggers Ġto Ġthe Ġquestion Ġtype Ġ( e . g . ĠTreatment , ĠPro gn osis , ĠInher itance ). ĠTr ig gers Ġare Ġidentified Ġfor Ġeach Ġquestion Ġtype Ġbased Ġon Ġa Ġmanual Ġannotation Ġof Ġa Ġset Ġof Ġmedical Ġquestions Ġ( cf . ĠSection ĠSEC REF 36 Ġ). ĠThis Ġfeature Ġhas Ġthree Ġpossible Ġvalues : Ġ2 Ġ( Perfect Ġmatch Ġbetween ĠIN LINE FORM 0 Ġtype ( s ) Ġand ĠIN LINE FORM 1 Ġtype ( s )), Ġ1 Ġ( Over lap Ġbetween ĠIN LINE FORM 2 Ġtype ( s ) Ġand ĠIN LINE FORM 3 Ġtype ( s )) Ġand Ġ0 Ġ( No Ġcommon Ġtypes ). </s> Dat as ets ĠUsed Ġfor Ġthe ĠR Q E ĠStudy </s> We Ġevaluate Ġthe ĠR Q E Ġmethods Ġ( i . e . Ġdeep Ġlearning Ġmodel Ġand Ġlog istic Ġregression Ġclass ifier ) Ġusing Ġtwo Ġdatasets Ġof Ġsentence Ġpairs Ġ( SN LI Ġand Ġmulti N LI ), Ġand Ġthree Ġdatasets Ġof Ġquestion Ġpairs Ġ( Qu ora , ĠClinical - Q E , Ġand ĠSem E val - c Q A ). </s> The ĠStanford ĠNatural ĠLanguage ĠIn ference Ġcorpus Ġ( SN LI ) ĠB IB REF 13 Ġcontains Ġ5 69 , 0 37 Ġsentence Ġpairs Ġwritten Ġby Ġhumans Ġbased Ġon Ġimage Ġcaption ing . ĠThe Ġtraining Ġset Ġof Ġthe ĠMulti N LI Ġcorpus ĠB IB REF 14 Ġconsists Ġof Ġ39 3 , 000 Ġpairs Ġof Ġsentences Ġfrom Ġfive Ġgenres Ġof Ġwritten Ġand Ġspoken ĠEnglish Ġ( e . g . ĠTravel , ĠGovernment ). ĠTwo Ġother Ġ" matched " Ġand Ġ" m ism atched " Ġsets Ġare Ġalso Ġavailable Ġfor Ġdevelopment Ġ( 20 , 000 Ġpairs ). ĠBoth ĠSN LI Ġand Ġmulti N LI Ġconsider Ġthree Ġtypes Ġof Ġrelationships Ġbetween Ġsentences : Ġentail ment , Ġneutral Ġand Ġcontradiction . ĠWe Ġconverted Ġthe Ġcontradiction Ġand Ġneutral Ġlabels Ġto Ġthe Ġsame Ġnon - ent ail ment Ġclass . </s> The ĠQU OR A Ġdataset Ġof Ġsimilar Ġquestions Ġwas Ġrecently Ġpublished Ġwith Ġ404 , 279 Ġquestion Ġpairs . ĠWe Ġrandomly Ġselected Ġthree Ġdistinct Ġsubs ets Ġ( 80 % / 10 % / 10 %) Ġfor Ġtraining Ġ( 323 , 423 Ġpairs ), Ġdevelopment Ġ( 40 , 428 Ġpairs ) Ġand Ġtest Ġ( 40 , 428 Ġpairs ). </s> The Ġclinical - Q E Ġdataset ĠB IB REF 1 Ġcontains Ġ8 , 588 Ġquestion Ġpairs Ġand Ġwas Ġconstructed Ġusing Ġ4 , 655 Ġclinical Ġquestions Ġasked Ġby Ġfamily Ġdoctors ĠB IB REF 45 Ġ. ĠWe Ġrandomly Ġselected Ġthree Ġdistinct Ġsubs ets Ġ( 80 % / 10 % / 10 %) Ġfor Ġtraining Ġ( 6 , 870 Ġpairs ), Ġdevelopment Ġ( 8 59 Ġpairs ) Ġand Ġtest Ġ( 8 59 Ġpairs ). </s> The Ġquestion Ġsimilarity Ġdataset Ġof ĠSem E val Ġ2016 ĠTask Ġ3 B Ġ( Sem E val - c Q A ) ĠB IB REF 3 Ġcontains Ġ3 , 8 69 Ġquestion Ġpairs Ġand Ġaims Ġto Ġre - rank Ġa Ġlist Ġof Ġrelated Ġquestions Ġaccording Ġto Ġtheir Ġsimilarity Ġto Ġthe Ġoriginal Ġquestion . ĠThe Ġsame Ġdataset Ġwas Ġused Ġfor ĠSem E val Ġ2017 ĠTask Ġ3 ĠB IB REF 4 Ġ. </s> To Ġconstruct Ġour Ġtest Ġdataset , Ġwe Ġused Ġa Ġpublicly Ġshared Ġset Ġof ĠConsumer ĠHealth ĠQuestions Ġ( CHQ s ) Ġreceived Ġby Ġthe ĠU . S . ĠNational ĠLibrary Ġof ĠMedicine Ġ( N LM ), Ġand Ġannot ated Ġwith Ġnamed Ġentities , Ġquestion Ġtypes , Ġand Ġfocus ĠB IB REF 46 Ġ, ĠB IB REF 47 Ġ. ĠThe ĠCH Q Ġdataset Ġconsists Ġof Ġ1 , 7 21 Ġconsumer Ġinformation Ġrequests Ġmanually Ġannot ated Ġwith Ġsub quest ions , Ġeach Ġidentified Ġby Ġa Ġquestion Ġtype Ġand Ġa Ġfocus . </s> First , Ġwe Ġselected Ġautomatically Ġharvested ĠFAQ s , Ġfrom ĠU . S . ĠNational ĠInstitutes Ġof ĠHealth Ġ( NI H ) Ġwebsites , Ġthat Ġshare Ġboth Ġthe Ġsame Ġfocus Ġand Ġthe Ġsame Ġquestion Ġtype Ġwith Ġthe ĠCH Q s . ĠAs ĠFAQ s Ġare Ġmost Ġoften Ġvery Ġshort , Ġwe Ġfirst Ġassume Ġthat Ġthe ĠCH Q Ġentails Ġthe ĠFAQ . ĠTwo Ġsets Ġof Ġpairs Ġwere Ġconstructed : Ġ( i ) Ġpositive Ġpairs Ġof ĠCH Q s Ġand ĠFAQ s Ġsharing Ġat Ġleast Ġone Ġcommon Ġquestion Ġtype Ġand Ġthe Ġquestion Ġfocus , Ġand Ġ( ii ) Ġnegative Ġpairs Ġcorresponding Ġto Ġa Ġfocus Ġmismatch Ġor Ġtype Ġmismatch . ĠFor Ġeach Ġcategory Ġof Ġnegative Ġexamples , Ġwe Ġrandomly Ġselected Ġthe Ġsame Ġnumber Ġof Ġpairs Ġfor Ġa Ġbalanced Ġdataset . ĠThen , Ġwe Ġmanually Ġvalidated Ġthe Ġconstructed Ġpairs Ġand Ġcorrected Ġthe Ġpositive Ġand Ġnegative Ġlabels Ġwhen Ġneeded . ĠThe Ġfinal ĠR Q E Ġdataset Ġcontains Ġ850 ĠCH Q - FAQ Ġpairs Ġwith Ġ405 Ġpositive Ġand Ġ445 Ġnegative Ġpairs . ĠTable ĠT AB REF 26 Ġpresents Ġexamples Ġfrom Ġthe Ġfive Ġtraining Ġdatasets Ġ( SN LI , ĠMulti N LI , ĠSem E val - c Q A , ĠClinical - Q E Ġand ĠQu ora ) Ġand Ġthe Ġnew Ġtest Ġdataset Ġof Ġmedical ĠCH Q - FAQ Ġpairs . </s> Results Ġof ĠR Q E ĠAppro aches </s> In Ġthe Ġfirst Ġexperiment , Ġwe Ġevaluated Ġthe ĠDL Ġand ĠML Ġmethods Ġon ĠSN LI , Ġmulti - N LI , ĠQu ora , Ġand ĠClinical - Q E . ĠFor Ġthe Ġdatasets Ġthat Ġdid Ġnot Ġhave Ġa Ġdevelopment Ġand Ġtest Ġsets , Ġwe Ġrandomly Ġselected Ġtwo Ġsets , Ġeach Ġamount ing Ġto Ġ10 % Ġof Ġthe Ġdata , Ġfor Ġtest Ġand Ġdevelopment , Ġand Ġused Ġthe Ġremaining Ġ80 % Ġfor Ġtraining . ĠFor ĠMulti N LI , Ġwe Ġused Ġthe Ġdev 1 - matched Ġset Ġfor Ġvalidation Ġand Ġthe Ġdev 2 - m ism atched Ġset Ġfor Ġtesting . </s> Table ĠT AB REF 28 Ġpresents Ġthe Ġresults Ġof Ġthe Ġfirst Ġexperiment . ĠThe ĠDL Ġmodel Ġwith ĠGl o Ve Ġword Ġembed d ings Ġachieved Ġbetter Ġresults Ġon Ġthree Ġdatasets , Ġwith Ġ82 . 80 % ĠAccuracy Ġon ĠSN LI , Ġ78 . 52 % ĠAccuracy Ġon ĠMulti N LI , Ġand Ġ83 . 62 % ĠAccuracy Ġon ĠQu ora . ĠLog istic ĠReg ression Ġachieved Ġthe Ġbest ĠAccuracy Ġof Ġ98 . 60 % Ġon ĠClinical - R Q E . ĠWe Ġalso Ġperformed Ġa Ġ10 - fold Ġcross - valid ation Ġon Ġthe Ġfull ĠClinical - Q E Ġdata Ġof Ġ8 , 588 Ġquestion Ġpairs , Ġwhich Ġgave Ġ98 . 61 % ĠAccuracy . </s> In Ġthe Ġsecond Ġexperiment , Ġwe Ġused Ġthese Ġdatasets Ġfor Ġtraining Ġonly Ġand Ġcompared Ġtheir Ġperformance Ġon Ġour Ġtest Ġset Ġof Ġ850 Ġconsumer Ġhealth Ġquestions . ĠTable ĠT AB REF 29 Ġpresents Ġthe Ġresults Ġof Ġthis Ġexperiment . ĠLog istic ĠReg ression Ġtrained Ġon Ġthe Ġclinical - R Q E Ġdata Ġoutper formed ĠDL Ġmodels Ġtrained Ġon Ġall Ġdatasets , Ġwith Ġ73 . 18 % ĠAccuracy . </s> To Ġvalidate Ġfurther Ġthe Ġperformance Ġof Ġthe ĠLR Ġmethod , Ġwe Ġevaluated Ġit Ġon Ġquestion Ġsimilarity Ġdetection . ĠA Ġtypical Ġapproach Ġto Ġthis Ġtask Ġis Ġto Ġuse Ġan ĠIR Ġmethod Ġto Ġfind Ġsimilar Ġquestion Ġcandidates , Ġthen Ġa Ġmore Ġsophisticated Ġmethod Ġto Ġselect Ġand Ġre - rank Ġthe Ġsimilar Ġquestions . ĠWe Ġfollowed Ġa Ġsimilar Ġapproach Ġfor Ġthis Ġevaluation Ġby Ġcombining Ġthe ĠLR Ġmethod Ġwith Ġthe ĠIR Ġbaseline Ġprovided Ġin Ġthe Ġcontext Ġof ĠSem E val - c Q A . ĠThe Ġhybrid Ġmethod Ġcombines Ġthe Ġscore Ġprovided Ġby Ġthe ĠLog istic ĠReg ression Ġmodel Ġand Ġthe Ġreciprocal Ġrank Ġfrom Ġthe ĠIR Ġbaseline Ġusing Ġa Ġweight - based Ġcombination : </s> IN LINE FORM 0 </s> The Ġweight ĠIN LINE FORM 0 Ġwas Ġset Ġempir ically Ġthrough Ġseveral Ġtests Ġon Ġthe Ġc Q A - 2016 Ġdevelopment Ġset Ġ( ĠIN LINE FORM 1 Ġ). ĠTable ĠT AB REF 30 Ġpresents Ġthe Ġresults Ġon Ġthe Ġc Q A - 2016 Ġand Ġc Q A - 2017 Ġtest Ġdatasets . ĠThe Ġhybrid Ġmethod Ġ( LR + IR ) Ġprovided Ġthe Ġbest Ġresults Ġon Ġboth Ġdatasets . ĠOn Ġthe Ġ2016 Ġtest Ġdata , Ġthe ĠLR + IR Ġmethod Ġoutper formed Ġthe Ġbest Ġsystem Ġin Ġall Ġmeasures , Ġwith Ġ80 . 57 % ĠAccuracy Ġand Ġ77 . 47 % ĠMAP Ġ( official Ġsystem Ġranking Ġmeasure Ġin ĠSem E val - c Q A ). ĠOn Ġthe Ġc Q A - 2017 Ġtest Ġdata , Ġthe ĠLR + IR Ġmethod Ġobtained Ġ44 . 66 % ĠMAP Ġand Ġoutper formed Ġthe Ġc Q A - 2017 Ġbest Ġsystem Ġin ĠAccuracy Ġwith Ġ67 . 27 %. </s> Discussion Ġof ĠR Q E ĠResults </s> When Ġtrained Ġand Ġtested Ġon Ġthe Ġsame Ġcorpus , Ġthe ĠDL Ġmodel Ġwith ĠGl o Ve Ġembed d ings Ġgave Ġthe Ġbest Ġresults Ġon Ġthree Ġdatasets Ġ( SN LI , ĠMulti N LI Ġand ĠQu ora ). ĠLog istic ĠReg ression Ġgave Ġthe Ġbest ĠAccuracy Ġon Ġthe ĠClinical - R Q E Ġdataset Ġwith Ġ98 . 60 %. ĠWhen Ġtested Ġon Ġour Ġtest Ġset Ġ( 850 Ġmedical ĠCH Q s - FAQ s Ġpairs ), ĠLog istic ĠReg ression Ġtrained Ġon ĠClinical - Q E Ġgave Ġthe Ġbest Ġperformance Ġwith Ġ73 . 18 % ĠAccuracy . </s> The ĠSN LI Ġand Ġmulti - N LI Ġmodels Ġdid Ġnot Ġperform Ġwell Ġwhen Ġtested Ġon Ġmedical ĠR Q E Ġdata . ĠWe Ġperformed Ġadditional Ġevaluations Ġusing Ġthe ĠR TE - 1 , ĠR TE - 2 Ġand ĠR TE - 3 Ġopen - domain Ġdatasets Ġprovided Ġby Ġthe ĠP ASC AL Ġchallenge Ġand Ġthe Ġresults Ġwere Ġsimilar . ĠWe Ġhave Ġalso Ġtested Ġthe ĠSem E val - c Q A - 2016 Ġmodel Ġand Ġhad Ġa Ġsimilar Ġdrop Ġin Ġperformance Ġon ĠR Q E Ġdata . ĠThis Ġcould Ġbe Ġexplained Ġby Ġthe Ġdifferent Ġtypes Ġof Ġdata Ġleading Ġto Ġwrong Ġinternal Ġconceptual izations Ġof Ġmedical Ġterms Ġand Ġquestions Ġin Ġthe Ġdeep Ġneural Ġlayers . ĠThis Ġperformance Ġdrop Ġcould Ġalso Ġbe Ġcaused Ġby Ġthe Ġcomplexity Ġof Ġthe Ġtest Ġconsumer Ġhealth Ġquestions Ġthat Ġare Ġoften Ġcomposed Ġof Ġseveral Ġsub quest ions , Ġcontain Ġcontextual Ġinformation , Ġand Ġmay Ġcontain Ġmiss pell ings Ġand Ġun gram matical Ġsentences , Ġwhich Ġmakes Ġthem Ġmore Ġdifficult Ġto Ġprocess ĠB IB REF 48 Ġ. ĠAnother Ġaspect Ġis Ġthe Ġsemantics Ġof Ġthe Ġtask Ġas Ġdiscussed Ġin ĠSection ĠSEC REF 6 Ġ. ĠThe Ġdefinition Ġof Ġtextual Ġentail ment Ġin Ġopen - domain Ġmay Ġnot Ġquite Ġapply Ġto Ġquestion Ġentail ment Ġdue Ġto Ġthe Ġstrict Ġsemantics . ĠAlso Ġthe Ġgeneral Ġtextual Ġentail ment Ġdefinitions Ġrefer Ġonly Ġto Ġthe Ġpremise Ġand Ġhypothesis , Ġwhile Ġthe Ġdefinition Ġof ĠR Q E Ġfor Ġquestion Ġanswering Ġrelies Ġon Ġthe Ġrelationship Ġbetween Ġthe Ġsets Ġof Ġanswers Ġof Ġthe Ġcompared Ġquestions . </s> Building Ġa ĠMedical ĠQ A ĠCollection Ġfrom ĠTr usted ĠResources </s> A ĠR Q E - based ĠQ A Ġsystem Ġrequires Ġa Ġcollection Ġof Ġquestion - answer Ġpairs Ġto Ġmap Ġnew Ġuser Ġquestions Ġto Ġthe Ġexisting Ġquestions Ġwith Ġan ĠR Q E Ġapproach , Ġrank Ġthe Ġretrieved Ġquestions , Ġand Ġpresent Ġtheir Ġanswers Ġto Ġthe Ġuser . </s> Method </s> To Ġconstruct Ġtrusted Ġmedical Ġquestion - answer Ġpairs , Ġwe Ġcrawled Ġwebsites Ġfrom Ġthe ĠNational ĠInstitutes Ġof ĠHealth Ġ( cf . ĠSection ĠSEC REF 56 Ġ). ĠEach Ġweb Ġpage Ġdescribes Ġa Ġspecific Ġtopic Ġ( e . g . Ġname Ġof Ġa Ġdisease Ġor Ġa Ġdrug ), Ġand Ġoften Ġincludes Ġsyn onyms Ġof Ġthe Ġmain Ġtopic Ġthat Ġwe Ġextracted Ġduring Ġthe Ġcrawl . </s> We Ġconstructed Ġhand - crafted Ġpatterns Ġfor Ġeach Ġwebsite Ġto Ġautomatically Ġgenerate Ġthe Ġquestion - answer Ġpairs Ġbased Ġon Ġthe Ġdocument Ġstructure Ġand Ġthe Ġsection Ġtitles . ĠWe Ġalso Ġannot ated Ġeach Ġquestion Ġwith Ġthe Ġassociated Ġfocus Ġ( topic Ġof Ġthe Ġweb Ġpage ) Ġas Ġwell Ġas Ġthe Ġquestion Ġtype Ġidentified Ġwith Ġthe Ġdesigned Ġpatterns Ġ( cf . ĠSection ĠSEC REF 36 Ġ). </s> To Ġprovide Ġadditional Ġinformation Ġabout Ġthe Ġquestions Ġthat Ġcould Ġbe Ġused Ġfor Ġdiverse ĠIR Ġand ĠN LP Ġtasks , Ġwe Ġautomatically Ġannot ated Ġthe Ġquestions Ġwith Ġthe Ġfocus , Ġits ĠU ML S ĠConcept ĠUnique ĠIdent ifier Ġ( C UI ) Ġand ĠSem antic ĠType . ĠWe Ġcombined Ġtwo Ġmethods Ġto Ġrecognize Ġnamed Ġentities Ġfrom Ġthe Ġtitles Ġof Ġthe Ġcrawled Ġarticles Ġand Ġtheir Ġassociated ĠU ML S ĠCU Is : Ġ( i ) Ġexact Ġstring Ġmatching Ġto Ġthe ĠU ML S ĠMet at hes aurus , Ġand Ġ( ii ) ĠMeta Map ĠLite ĠB IB REF 49 Ġ. ĠWe Ġthen Ġused Ġthe ĠU ML S ĠSem antic ĠNetwork Ġto Ġretrieve Ġthe Ġassociated Ġsemantic Ġtypes Ġand Ġgroups . </s> Question ĠTypes </s> The Ġquestion Ġtypes Ġwere Ġderived Ġafter Ġthe Ġmanual Ġevaluation Ġof Ġ1 , 7 21 Ġconsumer Ġhealth Ġquestions . ĠOur Ġtax onomy Ġincludes Ġ16 Ġtypes Ġabout ĠDiseases , Ġ20 Ġtypes Ġabout ĠDrugs Ġand Ġone Ġtype Ġ( Information ) Ġfor Ġthe Ġother Ġnamed Ġentities Ġsuch Ġas ĠProcedures , ĠMedical Ġexams Ġand ĠTreat ments . ĠWe Ġdescribe Ġbelow Ġthe Ġconsidered Ġquestion Ġtypes Ġand Ġexamples Ġof Ġassociated Ġquestion Ġpatterns . </s> Question ĠTypes Ġabout ĠDiseases Ġ( 16 ): ĠInformation , ĠResearch Ġ( or ĠClinical ĠTrial ), ĠCauses , ĠTreatment , ĠPrevention , ĠDiagn osis Ġ( Ex ams Ġand ĠTests ), ĠPro gn osis , ĠCompl ications , ĠSymptoms , ĠInher itance , ĠSus cept ibility , ĠGenetic Ġchanges , ĠFrequency , ĠConsider ations , ĠContact Ġa Ġmedical Ġprofessional , ĠSupport ĠGroups . </s> Examples : </s> What Ġresearch Ġ( or Ġclinical Ġtrial ) Ġis Ġbeing Ġdone Ġfor ĠDIS E ASE ? </s> What Ġis Ġthe Ġoutlook Ġfor ĠDIS E ASE ? </s> How Ġmany Ġpeople Ġare Ġaffected Ġby ĠDIS E ASE ? </s> When Ġto Ġcontact Ġa Ġmedical Ġprofessional Ġabout ĠDIS E ASE ? </s> Who Ġis Ġat Ġrisk Ġfor ĠDIS E ASE ? </s> Where Ġto Ġfind Ġsupport Ġfor Ġpeople Ġwith ĠDIS E ASE ? </s> Question ĠTypes ĠAbout ĠDrugs Ġ( 20 ): ĠInformation , ĠInter action Ġwith Ġmedications , ĠInter action Ġwith Ġfood , ĠInter action Ġwith Ġherbs Ġand Ġsupplements , ĠImportant Ġwarning , ĠSpecial Ġinstructions , ĠBrand Ġnames , ĠHow Ġdoes Ġit Ġwork , ĠHow Ġeffective Ġis Ġit , ĠInd ication , ĠCont rain d ication , ĠLearn Ġmore , ĠSide Ġeffects , ĠEmergency Ġor Ġoverdose , ĠSe vere Ġreaction , ĠForget Ġa Ġdose , ĠDietary , ĠWhy Ġget Ġvaccinated , ĠStorage Ġand Ġdisposal , ĠUsage , ĠD ose . </s> Examples : </s> Are Ġthere Ġinteractions Ġbetween ĠDR UG Ġand Ġherbs Ġand Ġsupplements ? </s> What Ġimportant Ġwarning Ġor Ġinformation Ġshould ĠI Ġknow Ġabout ĠDR UG ? </s> Are Ġthere Ġsafety Ġconcerns Ġor Ġspecial Ġprecautions Ġabout ĠDR UG ? </s> What Ġis Ġthe Ġaction Ġof ĠDR UG Ġand Ġhow Ġdoes Ġit Ġwork ? </s> Who Ġshould Ġget ĠDR UG Ġand Ġwhy Ġis Ġit Ġprescribed ? </s> What Ġto Ġdo Ġin Ġcase Ġof Ġa Ġsevere Ġreaction Ġto ĠDR UG ? </s> Question ĠType Ġfor Ġother Ġmedical Ġentities Ġ( e . g . ĠProcedure , ĠExam , ĠTreatment ): ĠInformation . </s> What Ġis ĠCor on ary ĠAr tery ĠBy pass ĠSurgery ? </s> What Ġare ĠLiver ĠFunction ĠTests ? </s> Medical ĠResources </s> We Ġused Ġ12 Ġtrusted Ġwebsites Ġto Ġconstruct Ġa Ġcollection Ġof Ġquestion - answer Ġpairs . ĠFor Ġeach Ġwebsite , Ġwe Ġextracted Ġthe Ġfree Ġtext Ġof Ġeach Ġarticle Ġas Ġwell Ġas Ġthe Ġsyn onyms Ġof Ġthe Ġarticle Ġfocus Ġ( topic ). ĠThese Ġresources Ġand Ġtheir Ġbrief Ġdescriptions Ġare Ġprovided Ġbelow : </s> National ĠCancer ĠInstitute Ġ( NC I ) Ġ: ĠWe Ġextracted Ġfree Ġtext Ġfrom Ġ116 Ġarticles Ġon Ġvarious Ġcancer Ġtypes Ġ( 729 ĠQ A Ġpairs ). ĠWe Ġmanually Ġrestruct ured Ġthe Ġcontent Ġof Ġthe Ġarticles Ġto Ġgenerate Ġcomplete Ġanswers Ġ( e . g . Ġa Ġfull Ġanswer Ġabout Ġthe Ġtreatment Ġof Ġall Ġstages Ġof Ġa Ġspecific Ġtype Ġof Ġcancer ). ĠFigure ĠFIG REF 54 Ġpresents Ġexamples Ġof ĠQ A Ġpairs Ġgenerated Ġfrom Ġa ĠNC I Ġarticle . </s> Gen etic Ġand ĠRare ĠDiseases ĠInformation ĠCenter Ġ( G ARD ): ĠThis Ġresource Ġcontains Ġinformation Ġabout Ġvarious Ġaspects Ġof Ġgenetic / ra re Ġdiseases . ĠWe Ġextracted Ġall Ġdisease Ġquestion / answer Ġpairs Ġfrom Ġ4 , 278 Ġtopics Ġ( 5 , 394 ĠQ A Ġpairs ). </s> Gen etics ĠHome ĠReference Ġ( G HR ): ĠThis ĠNL M Ġresource Ġcontains Ġconsumer - oriented Ġinformation Ġabout Ġthe Ġeffects Ġof Ġgenetic Ġvariation Ġon Ġhuman Ġhealth . ĠWe Ġextracted Ġ1 , 0 99 Ġarticles Ġabout Ġdiseases Ġfrom Ġthis Ġresource Ġ( 5 , 430 ĠQ A Ġpairs ). </s> Med line Plus ĠHealth ĠTopics : ĠThis Ġportion Ġof ĠMedline Plus Ġcontains Ġinformation Ġon Ġsymptoms , Ġcauses , Ġtreatment Ġand Ġprevention Ġfor Ġdiseases , Ġhealth Ġconditions Ġand Ġwellness Ġissues . ĠWe Ġextracted Ġthe Ġfree Ġtexts Ġin Ġsummary Ġsections Ġof Ġ9 81 Ġarticles Ġ( 98 1 ĠQ A Ġpairs ). </s> National ĠInstitute Ġof ĠDiabetes Ġand ĠDigest ive Ġand ĠKid ney ĠDiseases Ġ( N ID DK ) Ġ: ĠWe Ġextracted Ġtext Ġfrom Ġ174 Ġhealth Ġinformation Ġpages Ġon Ġdiseases Ġstudied Ġby Ġthis Ġinstitute Ġ( 1 , 192 ĠQ A Ġpairs ). </s> National ĠInstitute Ġof ĠNeurolog ical ĠDisorders Ġand ĠStro ke Ġ( N IN DS ): ĠWe Ġextracted Ġfree Ġtext Ġfrom Ġ277 Ġinformation Ġpages Ġon Ġneurological Ġand Ġstroke - related Ġdiseases Ġfrom Ġthis Ġresource Ġ( 1 , 104 ĠQ A Ġpairs ). </s> NI HS en ior Health Ġ: ĠThis Ġwebsite Ġcontains Ġhealth Ġand Ġwellness Ġinformation Ġfor Ġolder Ġadults . ĠWe Ġextracted Ġ71 Ġarticles Ġfrom Ġthis Ġresource Ġ( 7 69 ĠQ A Ġpairs ). </s> National ĠHeart , ĠLung , Ġand ĠBlood ĠInstitute Ġ( N HL BI ) Ġ: ĠWe Ġextracted Ġtext Ġfrom Ġ135 Ġarticles Ġon Ġdiseases , Ġtests , Ġprocedures , Ġand Ġother Ġrelevant Ġtopics Ġon Ġdisorders Ġof Ġheart , Ġlung , Ġblood , Ġand Ġsleep Ġ( 559 ĠQ A Ġpairs ). </s> Cent ers Ġfor ĠDisease ĠControl Ġand ĠPrevention Ġ( CDC ) Ġ: ĠWe Ġextracted Ġtext Ġfrom Ġ152 Ġarticles Ġon Ġdiseases Ġand Ġconditions Ġ( 270 ĠQ A Ġpairs ). </s> Med line Plus ĠA . D . A . M . ĠMedical ĠEncyclopedia : ĠThis Ġresource Ġcontains Ġ4 , 366 Ġarticles Ġabout Ġconditions , Ġtests , Ġand Ġprocedures . Ġ17 , 348 ĠQ A Ġpairs Ġwere Ġextracted Ġfrom Ġthis Ġresource . ĠFigure ĠFIG REF 55 Ġpresents Ġexamples Ġof ĠQ A Ġpairs Ġgenerated Ġfrom ĠA . D . A . M Ġencyclopedia . </s> Med line Plus ĠDrugs : ĠWe Ġextracted Ġfree Ġtext Ġfrom Ġ1 , 316 Ġarticles Ġabout ĠDrugs Ġand Ġgenerated Ġ12 , 889 ĠQ A Ġpairs . </s> Med line Plus ĠHer bs Ġand ĠSupp lements : ĠWe Ġextracted Ġfree Ġtext Ġfrom Ġ99 Ġarticles Ġand Ġgenerated Ġ7 92 ĠQ A Ġpairs . </s> The Ġfinal Ġcollection Ġcontains Ġ47 , 457 Ġannot ated Ġquestion - answer Ġpairs Ġabout ĠDiseases , ĠDrugs Ġand Ġother Ġnamed Ġentities Ġ( e . g . ĠTests ) Ġextracted Ġfrom Ġthese Ġ12 Ġtrusted Ġresources . </s> The ĠProp osed ĠEnt ail ment - based ĠQ A ĠSystem </s> Our Ġgoal Ġis Ġto Ġgenerate Ġa Ġranked Ġlist Ġof Ġanswers Ġfor Ġa Ġgiven ĠPrem ise ĠQuestion ĠIN LINE FORM 0 Ġby Ġranking Ġthe Ġrecognized ĠHyp ot hesis ĠQuestions ĠIN LINE FORM 1 Ġ. ĠBased Ġon Ġthe ĠR Q E Ġexperiments Ġabove Ġ( Section ĠSEC REF 27 Ġ), Ġwe Ġselected ĠLog istic ĠReg ression Ġtrained Ġon Ġthe Ġclinical - R Q E Ġdataset Ġto Ġrecognize Ġent ailed Ġquestions Ġand Ġrank Ġthem Ġwith Ġtheir Ġclassification Ġscores . </s> R Q E - based ĠQ A ĠApproach </s> Class ifying Ġthe Ġfull ĠQ A Ġcollection Ġfor Ġeach Ġtest Ġquestion Ġis Ġnot Ġfeasible Ġfor Ġreal - time Ġapplications . ĠTherefore , Ġwe Ġfirst Ġfilter Ġthe Ġquestions Ġwith Ġan ĠIR Ġmethod Ġto Ġretrieve Ġcandidate Ġquestions , Ġthen Ġclassify Ġthem Ġas Ġent ailed Ġ( or Ġnot ) Ġby Ġthe Ġuser / test Ġquestion . ĠBased Ġon Ġthe Ġpositive Ġresults Ġof Ġthe Ġcombination Ġmethod Ġtested Ġon ĠSem E val - c Q A Ġdata Ġ( Section ĠSEC REF 27 Ġ), Ġwe Ġadopted Ġa Ġcombination Ġmethod Ġto Ġmerge Ġthe Ġresults Ġobtained Ġby Ġthe Ġsearch Ġengine Ġand Ġthe ĠR Q E Ġscores . ĠThe Ġanswers Ġare Ġthen Ġcombined Ġfrom Ġboth Ġmethods Ġand Ġranked Ġusing Ġan Ġaggregate Ġscore . ĠFigure ĠFIG REF 82 Ġpresents Ġthe Ġoverall Ġarchitecture Ġof Ġthe Ġproposed ĠQ A Ġsystem . ĠWe Ġdescribe Ġeach Ġmodule Ġin Ġmore Ġdetails Ġnext . </s> Finding ĠSimilar ĠQuestion ĠCand idates </s> For Ġeach Ġpremise Ġquestion ĠIN LINE FORM 0 Ġ, Ġwe Ġuse Ġthe ĠTer rier Ġsearch Ġengine Ġto Ġretrieve ĠIN LINE FORM 1 Ġrelevant Ġquestion Ġcandidates ĠIN LINE FORM 2 Ġand Ġthen Ġapply Ġthe ĠR Q E Ġclass ifier Ġto Ġpredict Ġthe Ġlabels Ġfor Ġthe Ġpairs Ġ( ĠIN LINE FORM 3 Ġ, ĠIN LINE FORM 4 Ġ). </s> We Ġindexed Ġthe Ġquestions Ġof Ġour ĠQ A Ġcollection Ġwithout Ġthe Ġassociated Ġanswers . ĠIn Ġorder Ġto Ġimprove Ġthe Ġindex ing Ġand Ġthe Ġperformance Ġof Ġquestion Ġretrieval , Ġwe Ġalso Ġindexed Ġthe Ġsyn onyms Ġof Ġthe Ġquestion Ġfocus Ġand Ġthe Ġtriggers Ġof Ġthe Ġquestion Ġtype Ġwith Ġeach Ġquestion . ĠThis Ġchoice Ġallowed Ġus Ġto Ġavoid Ġthe Ġshortcomings Ġof Ġquery Ġexpansion , Ġincluding Ġincorrect Ġor Ġirrelevant Ġsyn onyms Ġand Ġthe Ġincreased Ġexecution Ġtime . ĠThe Ġsyn onyms Ġof Ġthe Ġquestion Ġfocus Ġ( topic ) Ġwere Ġextracted Ġautomatically Ġfrom Ġthe ĠQ A Ġcollection . ĠThe Ġtriggers Ġof Ġeach Ġquestion Ġtype Ġwere Ġdefined Ġmanually Ġin Ġthe Ġquestion Ġtypes Ġtax onomy . ĠBelow Ġare Ġtwo Ġexamples Ġof Ġindexed Ġquestions Ġfrom Ġour ĠQ A Ġcollection , Ġwith Ġthe Ġautomatically Ġadded Ġfocus Ġsyn onyms Ġand Ġquestion Ġtype Ġtriggers : </s> What Ġare Ġthe Ġtreatments Ġfor ĠTort ic oll is ? </s> Focus : ĠTort ic oll is . ĠQuestion Ġtype : ĠTreatment . </s> Added Ġfocus Ġsyn onyms : Ġ" Sp as mod ic Ġtort ic oll is , ĠW ry Ġneck , ĠL ox ia , ĠC erv ical Ġdy ston ia ". ĠAdded Ġquestion Ġtype Ġtriggers : Ġ" rel ieve , Ġmanage , Ġcure , Ġremedy , Ġtherapy ". </s> What Ġis Ġthe Ġoutlook Ġfor ĠLegion naire Ġdisease ? </s> Focus : ĠLegion naire Ġdisease . ĠQuestion ĠType : ĠPro gn osis . </s> Added Ġfocus Ġsyn onyms : Ġ" Leg ion ella Ġpneumonia , ĠPont iac Ġfever , ĠLegion ell osis ". ĠAdded Ġquestion Ġtype Ġtriggers : Ġ" pro gn osis , Ġlife Ġexpectancy ". </s> The ĠIR Ġtask Ġconsists Ġof Ġretrieving Ġhypothesis Ġquestions ĠIN LINE FORM 0 Ġrelevant Ġto Ġthe Ġsubmitted Ġquestion ĠIN LINE FORM 1 Ġ. ĠAs Ġfusion Ġof ĠIR Ġresult Ġhas Ġshown Ġgood Ġperformance Ġin Ġdifferent Ġtracks Ġin ĠT REC , Ġwe Ġmerge Ġthe Ġresults Ġof Ġthe ĠTF - ID F Ġweight ing Ġfunction Ġand Ġthe ĠIn - exp B 2 ĠD FR Ġmodel ĠB IB REF 50 Ġ. </s> Let ĠIN LINE FORM 0 Ġ= ĠIN LINE FORM 1 Ġ, ĠIN LINE FORM 2 Ġ, Ġ... , ĠIN LINE FORM 3 Ġbe Ġthe Ġset Ġof ĠIN LINE FORM 4 Ġquestions Ġretrieved Ġby Ġthe Ġfirst ĠIR Ġmodel ĠIN LINE FORM 5 Ġand ĠIN LINE FORM 6 Ġ= ĠIN LINE FORM 7 Ġ, ĠIN LINE FORM 8 Ġ, Ġ... , ĠIN LINE FORM 9 Ġbe Ġthe Ġset Ġof ĠIN LINE FORM 10 Ġquestions Ġretrieved Ġby Ġthe Ġsecond ĠIR Ġmodel ĠIN LINE FORM 11 Ġ. ĠWe Ġmerge Ġboth Ġsets Ġby Ġsum ming Ġthe Ġscores Ġof Ġeach Ġretrieved Ġquestion ĠIN LINE FORM 12 Ġin Ġboth ĠIN LINE FORM 13 Ġand ĠIN LINE FORM 14 Ġlists , Ġthen Ġwe Ġre - rank Ġthe Ġhypothesis Ġquestions ĠIN LINE FORM 15 Ġ. </s> Comb ining ĠIR Ġand ĠR Q E ĠMethods </s> The ĠIR Ġmodels Ġand Ġthe ĠR Q E ĠLog istic ĠReg ression Ġmodel Ġbring Ġdifferent Ġperspectives Ġto Ġthe Ġsearch Ġfor Ġrelevant Ġcandidate Ġquestions . ĠIn Ġparticular , Ġquestion Ġentail ment Ġallows Ġunderstanding Ġthe Ġrelations Ġbetween Ġthe Ġimportant Ġterms , Ġwhereas Ġthe Ġtraditional ĠIR Ġmethods Ġidentify Ġthe Ġimportant Ġterms , Ġbut Ġwill Ġnot Ġnotice Ġif Ġthe Ġrelations Ġare Ġopposite . ĠMoreover , Ġsome Ġof Ġthe Ġquestion Ġtypes Ġthat Ġthe ĠR Q E Ġclass ifier Ġlearns Ġwill Ġnot Ġbe Ġdeemed Ġimportant Ġterms Ġby Ġtraditional ĠIR Ġand Ġthe Ġmost Ġrelevant Ġquestions Ġwill Ġnot Ġbe Ġranked Ġat Ġthe Ġtop Ġof Ġthe Ġlist . </s> Therefore , Ġin Ġour Ġapproach , Ġwhen Ġa Ġquestion Ġis Ġsubmitted Ġto Ġthe Ġsystem , Ġcandidate Ġquestions Ġare Ġfet ched Ġusing Ġthe ĠIR Ġmodels , Ġthen Ġthe ĠR Q E Ġclass ifier Ġis Ġapplied Ġto Ġfilter Ġout Ġthe Ġnon - ent ailed Ġquestions Ġand Ġre - rank Ġthe Ġremaining Ġcandidates . </s> Specifically , Ġwe Ġdenote ĠIN LINE FORM 0 Ġthe Ġlist Ġof Ġquestion Ġcandidates ĠIN LINE FORM 1 Ġreturned Ġby Ġthe ĠIR Ġsystem . ĠThe Ġpremise Ġquestion ĠIN LINE FORM 2 Ġis Ġthen Ġused Ġto Ġconstruct ĠN Ġquestion Ġpairs ĠIN LINE FORM 3 Ġ. ĠThe ĠR Q E Ġclass ifier Ġis Ġthen Ġapplied Ġto Ġfilter Ġout Ġthe Ġquestion Ġpairs Ġthat Ġare Ġnot Ġent ailed Ġand Ġre - rank Ġthe Ġremaining Ġpairs . </s> More Ġprecisely , Ġlet ĠIN LINE FORM 0 Ġ= ĠIN LINE FORM 1 Ġin ĠIN LINE FORM 2 Ġbe Ġthe Ġlist Ġof Ġselected Ġcandidate Ġquestions Ġthat Ġhave Ġa Ġpositive Ġentail ment Ġrelation Ġwith Ġa Ġgiven Ġpremise Ġquestion ĠIN LINE FORM 3 Ġ. ĠWe Ġrank ĠIN LINE FORM 4 Ġby Ġcomputing Ġa Ġhybrid Ġscore ĠIN LINE FORM 5 Ġfor Ġeach Ġcandidate Ġquestion ĠIN LINE FORM 6 Ġtaking Ġinto Ġaccount Ġthe Ġscore Ġof Ġthe ĠIR Ġsystem ĠIN LINE FORM 7 Ġand Ġthe Ġscore Ġof Ġthe ĠR Q E Ġsystem ĠIN LINE FORM 8 Ġ. </s> For Ġeach Ġsystem ĠIN LINE FORM 0 ĠIN LINE FORM 1 Ġ, Ġwe Ġnormal ize Ġthe Ġassociated Ġscore Ġby Ġdividing Ġit Ġby Ġthe Ġmaximum Ġscore Ġamong Ġthe ĠIN LINE FORM 2 Ġcandidate Ġquestions Ġretrieved Ġby ĠIN LINE FORM 3 Ġfor ĠIN LINE FORM 4 Ġ: </s> IN LINE FORM 0 </s> IN LINE FORM 0 ĠIN LINE FORM 1 </s> In Ġour Ġexperiments , Ġwe Ġfixed Ġthe Ġvalue Ġof ĠIN LINE FORM 0 Ġto Ġ100 . ĠThis Ġthreshold Ġvalue Ġwas Ġselected Ġas Ġa Ġsafe Ġvalue Ġfor Ġthis Ġtask Ġfor Ġthe Ġfollowing Ġreasons : </s> Our Ġcollection Ġof Ġ47 , 457 Ġquestion - answer Ġpairs Ġwas Ġcollected Ġfrom Ġonly Ġ12 ĠNIH Ġinstit utes Ġand Ġis Ġunlikely Ġto Ġcontain Ġmore Ġthan Ġ100 Ġoccurrences Ġof Ġthe Ġsame Ġfocus - type Ġpair . </s> Each Ġquestion Ġwas Ġindexed Ġwith Ġadditional Ġannotations Ġfor Ġthe Ġquestion Ġfocus , Ġits Ġsyn onyms Ġand Ġthe Ġquestion Ġtype Ġsyn onyms . </s> E val uating ĠR Q E Ġfor ĠMedical ĠQuestion ĠAn sw ering </s> The Ġobjective Ġof Ġthis Ġevaluation Ġis Ġto Ġstudy Ġthe Ġeffectiveness Ġof ĠR Q E Ġfor ĠMedical ĠQuestion ĠAn sw ering , Ġby Ġcomparing Ġthe Ġanswers Ġretrieved Ġby Ġthe Ġhybrid Ġentail ment - based Ġapproach , Ġthe ĠIR Ġmethod Ġand Ġthe Ġother ĠQ A Ġsystems Ġparticipating Ġto Ġthe Ġmedical Ġtask Ġat ĠT REC Ġ2017 ĠLive Q A Ġchallenge Ġ( Live Q A - Med ). </s> E val uation ĠMethod </s> We Ġdeveloped Ġan Ġinterface Ġto Ġperform Ġthe Ġmanual Ġevaluation Ġof Ġthe Ġretrieved Ġanswers . ĠFigure Ġ5 Ġpresents Ġthe Ġevaluation Ġinterface Ġshowing , Ġfor Ġeach Ġtest Ġquestion , Ġthe Ġtop - 10 Ġanswers Ġof Ġthe Ġevaluated ĠQ A Ġmethod Ġand Ġthe Ġreference Ġanswer ( s ) Ġused Ġby ĠLive Q A Ġassess ors Ġto Ġhelp Ġjudging Ġthe Ġretrieved Ġanswers Ġby Ġthe Ġparticipating Ġsystems . </s> We Ġused Ġthe Ġtest Ġquestions Ġof Ġthe Ġmedical Ġtask Ġat ĠT REC - 2017 ĠLive Q A ĠB IB REF 11 Ġ. ĠThese Ġquestions Ġare Ġrandomly Ġselected Ġfrom Ġthe Ġconsumer Ġhealth Ġquestions Ġthat Ġthe ĠNL M Ġreceives Ġdaily Ġfrom Ġall Ġover Ġthe Ġworld . ĠThe Ġtest Ġquestions Ġcover Ġdifferent Ġmedical Ġentities Ġand Ġhave Ġa Ġwide Ġlist Ġof Ġquestion Ġtypes Ġsuch Ġas ĠComparison , ĠDiagn osis , ĠIng red ient , ĠSide Ġeffects Ġand ĠTap ering . </s> For Ġa Ġrelevant Ġcomparison , Ġwe Ġused Ġthe Ġsame Ġjudgment Ġscores Ġas Ġthe ĠLive Q A ĠTrack : </s> Correct Ġand ĠComplete ĠAnswer Ġ( 4 ) </s> Correct Ġbut ĠIn complete Ġ( 3 ) </s> In correct Ġbut ĠRelated Ġ( 2 ) </s> In correct Ġ( 1 ) </s> We Ġevaluated Ġthe Ġanswers Ġreturned Ġby Ġthe ĠIR - based Ġmethod Ġand Ġthe Ġhybrid ĠQ A Ġmethod Ġ( IR + R Q E ) Ġaccording Ġto Ġthe Ġsame Ġreference Ġanswers Ġused Ġin ĠLive Q A - Med . ĠThe Ġanswers Ġwere Ġanonym ized Ġ( the Ġmethod Ġnames Ġwere Ġblinded ) Ġand Ġpresented Ġto Ġ3 Ġassess ors : Ġa Ġmedical Ġdoctor Ġ( Ass essor ĠA ), Ġa Ġmedical Ġl ibrarian Ġ( B ) Ġand Ġa Ġresearcher Ġin Ġmedical Ġinform atics Ġ( C ). ĠNone Ġof Ġthe Ġassess ors Ġparticipated Ġin Ġthe Ġdevelopment Ġof Ġthe ĠQ A Ġmethods . ĠAss essors ĠB Ġand ĠC Ġevaluated Ġ1 , 000 Ġanswers Ġretrieved Ġby Ġeach Ġof Ġthe Ġmethods Ġ( IR Ġand ĠIR + R Q E ). ĠAss essor ĠA Ġevaluated Ġ2 , 000 Ġanswers Ġfrom Ġboth Ġmethods . </s> Table ĠT AB REF 103 Ġpresents Ġthe Ġinter - annot ator Ġagreement Ġ( IA A ) Ġthrough ĠF 1 Ġscore Ġcomputed Ġby Ġconsidering Ġone Ġof Ġthe Ġassess ors Ġas Ġreference . ĠIn Ġthe Ġfirst Ġevaluation , Ġwe Ġcomputed Ġthe ĠTrue ĠP osit ives Ġ( TP ) Ġand ĠFalse ĠP osit ives Ġ( FP ) Ġover Ġall Ġratings Ġand Ġthe ĠPrecision Ġand ĠF 1 Ġscore . ĠAs Ġthere Ġare Ġno Ġnegative Ġlabels Ġ( only Ġtrue Ġor Ġfalse Ġpositives Ġfor Ġeach Ġcategory ), ĠRecall Ġis Ġ100 %. ĠWe Ġalso Ġcomputed Ġa Ġpartial ĠI AA Ġby Ġgrouping Ġthe Ġ" Correct Ġand ĠComplete ĠAnswer " Ġand Ġ" Correct Ġbut ĠIn complete " Ġratings Ġ( as ĠCorrect ), Ġand Ġthe Ġ" In correct Ġbut ĠRelated " Ġand Ġ" In correct " Ġratings Ġ( as ĠInc orrect ). ĠThe Ġaverage Ġagreement Ġon Ġdistinguishing Ġthe ĠCorrect Ġand ĠInc orrect Ġanswers Ġis Ġ94 . 33 % ĠF 1 Ġscore . ĠTherefore , Ġwe Ġused Ġthe Ġevaluations Ġperformed Ġby Ġassess or ĠA Ġfor Ġboth Ġmethods . ĠThe Ġofficial Ġresults Ġof Ġthe ĠT REC ĠLive Q A Ġtrack Ġrelied Ġon Ġone Ġassess or Ġper Ġquestion Ġas Ġwell . </s> E val uation Ġof Ġthe Ġfirst Ġretrieved Ġanswer </s> We Ġcomputed Ġthe Ġmeasures Ġused Ġby ĠT REC ĠLive Q A Ġchallenges ĠB IB REF 51 Ġ, ĠB IB REF 11 Ġto Ġevaluate Ġthe Ġfirst Ġretrieved Ġanswer Ġfor Ġeach Ġtest Ġquestion : </s> av g Score ( 0 - 3 ): Ġthe Ġaverage Ġscore Ġover Ġall Ġquestions , Ġtransferring Ġ1 - 4 Ġlevel Ġgrades Ġto Ġ0 - 3 Ġscores . ĠThis Ġis Ġthe Ġmain Ġscore Ġused Ġto Ġrank ĠLive Q A Ġruns . </s> su cc @ i + : Ġthe Ġnumber Ġof Ġquestions Ġwith Ġscore Ġi Ġor Ġabove Ġ( i ĠIN LINE FORM 0 Ġ{ 2 .. 4 }) Ġdivided Ġby Ġthe Ġtotal Ġnumber Ġof Ġquestions . </s> pre c @ i + : Ġthe Ġnumber Ġof Ġquestions Ġwith Ġscore Ġi Ġor Ġabove Ġ( i ĠIN LINE FORM 0 Ġ{ 2 .. 4 }) Ġdivided Ġby Ġnumber Ġof Ġquestions Ġanswered Ġby Ġthe Ġsystem . </s> Table ĠT AB REF 108 Ġpresents Ġthe Ġaverage Ġscores , Ġsuccess Ġand Ġprecision Ġresults . ĠThe Ġhybrid ĠIR + R Q E ĠQ A Ġsystem Ġachieved Ġbetter Ġresults Ġthan Ġthe ĠIR - based Ġsystem Ġwith Ġ0 . 8 27 Ġaverage Ġscore . ĠIt Ġalso Ġachieved Ġa Ġhigher Ġscore Ġthan Ġthe Ġbest Ġresults Ġachieved Ġin Ġthe Ġmedical Ġchallenge Ġat ĠLive Q A ' 17 . ĠEval uating Ġthe ĠR Q E Ġsystem Ġalone Ġis Ġnot Ġrelevant , Ġas Ġapplying ĠR Q E Ġon Ġthe Ġfull Ġcollection Ġfor Ġeach Ġuser Ġquestion Ġis Ġnot Ġfeasible Ġfor Ġa Ġreal - time Ġsystem Ġbecause Ġof Ġthe Ġextended Ġexecution Ġtime . </s> E val uation Ġof Ġthe Ġtop Ġten Ġanswers </s> In Ġthis Ġevaluation , Ġwe Ġused ĠMean ĠAverage ĠPrecision Ġ( MAP ) Ġand ĠMean ĠRec ip rocal ĠRank Ġ( MR R ) Ġwhich Ġare Ġcommonly Ġused Ġin ĠQ A Ġto Ġevaluate Ġthe Ġtop - 10 Ġanswers Ġfor Ġeach Ġquestion . ĠWe Ġconsider Ġanswers Ġrated Ġas ĠâĢ ľ Correct Ġand ĠComplete ĠAnswer âĢ Ŀ Ġor ĠâĢ ľ Correct Ġbut ĠIn complete âĢ Ŀ Ġas Ġcorrect Ġanswers , Ġas Ġthe Ġtest Ġquestions Ġcontain Ġmultiple Ġsub quest ions Ġwhile Ġeach Ġanswer Ġin Ġour ĠQ A Ġcollection Ġcan Ġcover Ġonly Ġone Ġsub question . </s> MAP Ġis Ġthe Ġmean Ġof Ġthe ĠAverage ĠPrecision Ġ( Avg P ) Ġscores Ġover Ġall Ġquestions . </s> ( 1 ) ĠIN LINE FORM 0 </s> Q Ġis Ġthe Ġnumber Ġof Ġquestions . ĠIN LINE FORM 0 Ġis Ġthe ĠAvg P Ġof Ġthe ĠIN LINE FORM 1 Ġquestion . </s> IN LINE FORM 0 </s> K Ġis Ġthe Ġnumber Ġof Ġcorrect Ġanswers . ĠIN LINE FORM 0 Ġis Ġthe Ġrank Ġof ĠIN LINE FORM 1 Ġcorrect Ġanswer . </s> MR R Ġis Ġthe Ġaverage Ġof Ġthe Ġreciprocal Ġranks Ġfor Ġeach Ġquestion . ĠThe Ġreciprocal Ġrank Ġof Ġa Ġquestion Ġis Ġthe Ġmultipl icative Ġinverse Ġof Ġthe Ġrank Ġof Ġthe Ġfirst Ġcorrect Ġanswer . </s> ( 2 ) ĠIN LINE FORM 0 </s> Q Ġis Ġthe Ġnumber Ġof Ġquestions . ĠIN LINE FORM 0 Ġis Ġthe Ġrank Ġof Ġthe Ġfirst Ġcorrect Ġanswer Ġfor Ġthe ĠIN LINE FORM 1 Ġquestion . </s> Table ĠT AB REF 113 Ġpresents Ġthe ĠMAP @ 10 Ġand ĠMR R @ 10 Ġof Ġour ĠQ A Ġmethods . ĠThe ĠIR + R Q E Ġsystem Ġoutper forms Ġthe ĠIR - based ĠQ A Ġsystem Ġwith Ġ0 . 311 ĠMAP @ 10 Ġand Ġ0 . 333 ĠMR R @ 10 . </s> Discussion Ġof Ġentail ment - based ĠQ A Ġfor Ġthe Ġmedical Ġdomain </s> In Ġour Ġevaluation , Ġwe Ġfollowed Ġthe Ġsame ĠLive Q A Ġguidelines Ġwith Ġthe Ġhighest Ġpossible Ġrig or . ĠIn Ġparticular , Ġwe Ġconsulted Ġwith ĠN IST Ġassess ors Ġwho Ġprovided Ġus Ġwith Ġthe Ġparaph r ases Ġof Ġthe Ġtest Ġquestions Ġthat Ġthey Ġused Ġto Ġjudge Ġthe Ġanswers . ĠOur ĠI AA Ġon Ġthe Ġanswers Ġrating Ġwas Ġalso Ġhigh Ġcompared Ġto Ġrelated Ġtasks , Ġwith Ġan Ġ88 . 5 % ĠF 1 Ġagreement Ġwith Ġthe Ġexact Ġfour Ġcategories Ġand Ġa Ġ94 . 3 % Ġagreement Ġwhen Ġreducing Ġthe Ġcategories Ġto Ġtwo : ĠâĢ ľ Correct âĢ Ŀ Ġand ĠâĢ ľ In correct âĢ Ŀ Ġanswers . ĠOur Ġresults Ġshow Ġthat ĠR Q E Ġimproves Ġthe Ġoverall Ġperformance Ġand Ġexceeds Ġthe Ġbest Ġresults Ġin Ġthe Ġmedical ĠLive Q A ' 17 Ġchallenge Ġby Ġa Ġfactor Ġof Ġ29 . 8 %. ĠThis Ġperformance Ġimprovement Ġis Ġparticularly Ġinteresting Ġas : </s> Our Ġanswer Ġsource Ġhas Ġonly Ġ47 K Ġquestion - answer Ġpairs Ġwhen ĠLive Q A Ġparticipating Ġsystems Ġrelied Ġon Ġmuch Ġlarger Ġcollections , Ġincluding Ġthe ĠWorld ĠWide ĠWeb . </s> Our Ġsystem Ġanswered Ġone Ġsub question Ġat Ġmost Ġwhen Ġmany ĠLive Q A Ġtest Ġquestions Ġhad Ġseveral Ġsub quest ions . </s> The Ġlatter Ġobservation , Ġ( b ), Ġmakes Ġthe Ġhybrid ĠIR + R Q E Ġapproach Ġeven Ġmore Ġpromising Ġas Ġit Ġgives Ġit Ġa Ġlarge Ġpotential Ġfor Ġthe Ġimprovement Ġof Ġanswer Ġcomple teness . </s> The Ġformer Ġobservation , Ġ( a ), Ġprovides Ġanother Ġinteresting Ġinsight : Ġrestricting Ġthe Ġanswer Ġsource Ġto Ġonly Ġreliable Ġcollections Ġcan Ġactually Ġimprove Ġthe ĠQ A Ġperformance Ġwithout Ġlosing Ġcoverage Ġ( i . e ., Ġour ĠQ A Ġapproach Ġprovided Ġat Ġleast Ġone Ġanswer Ġto Ġeach Ġtest Ġquestion Ġand Ġobtained Ġthe Ġbest Ġrelevance Ġscore ). </s> In Ġanother Ġobservation , Ġthe Ġassess ors Ġreported Ġthat Ġmany Ġof Ġthe Ġreturned Ġanswers Ġhad Ġa Ġcorrect Ġquestion Ġtype Ġbut Ġa Ġwrong Ġfocus , Ġwhich Ġindicates Ġthat Ġincluding Ġa Ġfocus Ġrecognition Ġmodule Ġto Ġfilter Ġsuch Ġwrong Ġanswers Ġcan Ġimprove Ġfurther Ġthe ĠQ A Ġperformance Ġin Ġterms Ġof Ġprecision . ĠAnother Ġaspect Ġthat Ġwas Ġreported Ġis Ġthe Ġrepetition Ġof Ġthe Ġsame Ġ( or Ġsimilar ) Ġanswer Ġfrom Ġdifferent Ġwebsites , Ġwhich Ġcould Ġbe Ġaddressed Ġby Ġimproving Ġanswer Ġselection Ġwith Ġinter - answer Ġcomparisons Ġand Ġremoval Ġof Ġnear - du pl icates . ĠAlso , Ġhalf Ġof Ġthe ĠLive Q A Ġtest Ġquestions Ġare Ġabout ĠDrugs , Ġwhen Ġonly Ġtwo Ġof Ġour Ġresources Ġare Ġspecialized Ġin ĠDrugs , Ġamong Ġ12 Ġsub - col lections Ġoverall . ĠAccordingly , Ġthe Ġassess ors Ġnoticed Ġthat Ġthe Ġperformance Ġof Ġthe ĠQ A Ġsystems Ġwas Ġbetter Ġon Ġquestions Ġabout Ġdiseases Ġthan Ġon Ġquestions Ġabout Ġdrugs , Ġwhich Ġsuggests Ġa Ġneed Ġfor Ġextending Ġour Ġmedical ĠQ A Ġcollection Ġwith Ġmore Ġinformation Ġabout Ġdrugs Ġand Ġassociated Ġquestion Ġtypes . </s> We Ġalso Ġlooked Ġclosely Ġat Ġthe Ġprivate Ġwebsites Ġused Ġby Ġthe ĠLive Q A - Med Ġannot ators Ġto Ġprovide Ġsome Ġof Ġthe Ġreference Ġanswers Ġfor Ġthe Ġtest Ġquestions . ĠFor Ġinstance , Ġthe ĠConsumer Lab Ġwebsite Ġwas Ġuseful Ġto Ġanswer Ġa Ġquestion Ġabout Ġthe Ġingredients Ġof Ġa ĠDrug Ġ( CO EN Z Y ME ĠQ 10 ). ĠSimilarly , Ġthe Ġe Health Me Ġwebsite Ġwas Ġused Ġto Ġanswer Ġa Ġtest Ġquestion Ġasking Ġabout Ġinteractions Ġbetween Ġtwo Ġdrugs Ġ( P hen ter mine Ġand ĠD icy cl om ine ) Ġwhen Ġno Ġinformation Ġwas Ġfound Ġin ĠDaily Med . Ġe Health Me Ġprovides Ġhealthcare Ġbig Ġdata Ġanalysis Ġand Ġprivate Ġresearch Ġand Ġstudies Ġincluding Ġself - reported Ġadverse Ġdrug Ġeffects Ġby Ġpatients . </s> But Ġthe Ġquestion Ġremains Ġon Ġthe Ġextent Ġto Ġwhich Ġsuch Ġbig Ġdata Ġand Ġother Ġprivate Ġwebsites Ġcould Ġbe Ġused Ġto Ġautomatically Ġanswer Ġmedical Ġquestions Ġif Ġinformation Ġis Ġotherwise Ġunavailable . ĠUnlike Ġmedical Ġprofessionals , Ġpatients Ġdo Ġnot Ġnecessarily Ġhave Ġthe Ġknowledge Ġand Ġtools Ġto Ġvalidate Ġsuch Ġinformation . ĠAn Ġalternative Ġapproach Ġcould Ġbe Ġto Ġput Ġlimitations Ġon Ġmedical ĠQ A Ġsystems Ġin Ġterms Ġof Ġthe Ġquestions Ġthat Ġcan Ġbe Ġanswered Ġ( e . g . Ġ" What Ġis Ġmy Ġdiagnosis Ġfor Ġsuch Ġsymptoms ") Ġand Ġbuild Ġclass ifiers Ġto Ġdetect Ġsuch Ġquestions Ġand Ġwarn Ġthe Ġusers Ġabout Ġthe Ġdangers Ġof Ġlooking Ġfor Ġtheir Ġanswers Ġonline . </s> More Ġgenerally , Ġmedical ĠQ A Ġsystems Ġshould Ġfollow Ġsome Ġstrict Ġguidelines Ġregarding Ġthe Ġgoal Ġand Ġbackground Ġknowledge Ġand Ġresources Ġof Ġeach Ġsystem Ġin Ġorder Ġto Ġprotect Ġthe Ġconsumers Ġfrom Ġmisleading Ġor Ġharmful Ġinformation . ĠSuch Ġguidelines Ġcould Ġbe Ġbased Ġ( i ) Ġon Ġthe Ġsource Ġof Ġthe Ġinformation Ġsuch Ġas Ġhealth Ġand Ġmedical Ġinformation Ġwebsites Ġsponsored Ġby Ġthe ĠU . S . Ġgovernment , Ġnot - for - profit Ġhealth Ġor Ġmedical Ġorganizations , Ġand Ġmedical Ġuniversity Ġcenters , Ġor Ġ( ii ) Ġon Ġconventions Ġsuch Ġas Ġthe Ġcode Ġof Ġconduct Ġof Ġthe ĠH ON ĠFoundation Ġ( H ON code ) Ġthat Ġaddresses Ġthe Ġreliability Ġand Ġusefulness Ġof Ġmedical Ġinformation Ġon Ġthe ĠInternet . </s> Our Ġexperiments Ġshow Ġthat Ġlimiting Ġthe Ġnumber Ġof Ġanswer Ġsources Ġwith Ġsuch Ġguidelines Ġis Ġnot Ġonly Ġfeasible , Ġbut Ġit Ġcould Ġalso Ġenhance Ġthe Ġperformance Ġof Ġthe ĠQ A Ġsystem Ġfrom Ġan Ġinformation Ġretrieval Ġperspective . </s> Conclusion </s> In Ġthis Ġpaper , Ġwe Ġcarried Ġout Ġan Ġempirical Ġstudy Ġof Ġmachine Ġlearning Ġand Ġdeep Ġlearning Ġmethods Ġfor ĠRecogn izing ĠQuestion ĠEnt ail ment Ġin Ġthe Ġmedical Ġdomain Ġusing Ġseveral Ġdatasets . ĠWe Ġdeveloped Ġa ĠR Q E - based ĠQ A Ġsystem Ġto Ġanswer Ġnew Ġmedical Ġquestions Ġusing Ġexisting Ġquestion - answer Ġpairs . ĠWe Ġbuilt Ġand Ġshared Ġa Ġcollection Ġof Ġ47 K Ġmedical Ġquestion - answer Ġpairs . ĠOur ĠQ A Ġapproach Ġoutper formed Ġthe Ġbest Ġresults Ġon ĠT REC - 2017 ĠLive Q A Ġmedical Ġtest Ġquestions . ĠThe Ġproposed Ġapproach Ġcan Ġbe Ġapplied Ġand Ġadapted Ġto Ġopen - domain Ġas Ġwell Ġas Ġspecific - domain ĠQ A . ĠDeep Ġlearning Ġmodels Ġachieved Ġinteresting Ġresults Ġon Ġopen - domain Ġand Ġclinical Ġdatasets , Ġbut Ġobtained Ġa Ġlower Ġperformance Ġon Ġconsumer Ġhealth Ġquestions . ĠWe Ġwill Ġcontinue Ġinvestigating Ġother Ġnetwork Ġarchitectures Ġincluding Ġtransfer Ġlearning , Ġas Ġwell Ġas Ġcreation Ġof Ġa Ġlarge Ġcollection Ġof Ġconsumer Ġhealth Ġquestions Ġfor Ġtraining Ġto Ġimprove Ġthe Ġperformance Ġof ĠDL Ġmodels . ĠFuture Ġwork Ġalso Ġincludes Ġexploring Ġintegration Ġof Ġa ĠQuestion ĠFocus ĠRecogn ition Ġmodule Ġto Ġenhance Ġcandidate Ġquestion Ġretrieval , Ġand Ġexpanding Ġour Ġquestion - answer Ġcollection . </s> Acknowled gements </s> We Ġthank ĠHal il ĠKil ic oglu Ġ( N LM / NI H ) Ġfor Ġhis Ġhelp Ġwith Ġthe Ġcrawling Ġand Ġthe Ġmanual Ġevaluation Ġand ĠSony a ĠE . ĠSho osh an Ġ( N LM / NI H ) Ġfor Ġher Ġhelp Ġwith Ġthe Ġjudgment Ġof Ġthe Ġretrieved Ġanswers . ĠWe Ġalso Ġthank ĠEllen ĠV oor he es Ġ( N IST ) Ġfor Ġher Ġvaluable Ġsupport Ġwith Ġthe ĠT REC ĠLive Q A Ġevaluation . </s> We Ġconsider Ġthe Ġcase Ġof Ġthe Ġquestion Ġnumber Ġ36 Ġin Ġthe ĠT REC - 2017 ĠLive Q A Ġmedical Ġtest Ġdataset : </s> 36 . Ġcongen ital Ġdi aph rag matic Ġher nia . Ġwhat Ġare Ġthe Ġcauses Ġof Ġcongen ital Ġdi aph rag matic Ġher nia ? ĠCan Ġcousin Ġmarriage Ġcause Ġthis ? ĠWhat Ġkind Ġof Ġlung Ġdisease Ġthe Ġbaby Ġmight Ġexperience Ġlife Ġlong ? </s> This Ġquestion Ġwas Ġanswered Ġby Ġ5 Ġparticipating Ġruns Ġ( vs . Ġ8 Ġruns Ġfor Ġother Ġquestions ), Ġand Ġall Ġsubmitted Ġanswers Ġwere Ġwrong Ġ( sc ores Ġof Ġ1 Ġor Ġ2 ). ĠHowever , Ġour ĠIR - based ĠQ A Ġsystem Ġretrieved Ġone Ġexcellent Ġanswer Ġ( score Ġ4 ) Ġand Ġour Ġhybrid ĠIR + R Q E Ġsystem Ġprovided Ġ3 Ġexcellent Ġanswers . </s> A ) ĠT REC Ġ2017 ĠLive Q A - Med ĠParticipants ' ĠResults : </s> B ) ĠOur ĠIR - based ĠQ A ĠSystem : </s> C ) ĠOur ĠIR + R Q E ĠQ A ĠSystem :