precision, recall, F1 and accuracy
EXTRACTIVE
precision, recall, F1 , accuracy 
EXTRACTIVE
We employ Jieba for Chinese word segmentation and keep the top INLINEFORM0 frequent words. The word embedding size is set to 128 and the other low-frequency words are replaced with the mark <UNK>. The hidden size of GRU is set to 128 for each direction in Bi-GRU. In the pair-wise attentive reader, the hidden state is set to 256 for mGRu. In the CNN layer, filter windows are set to 1, 3, 4, and 5 with each filter containing 200 feature maps. We add a dropout layer BIBREF38 after the CNN layer with a dropout rate of INLINEFORM1 . We use Adam BIBREF32 for training and set learning rate to INLINEFORM2 , INLINEFORM3 to INLINEFORM4 , INLINEFORM5 to INLINEFORM6 , INLINEFORM7 to INLINEFORM8 , batch size to 64. We employ precision, recall, F1 and accuracy for evaluation metrics. We repeat all the experiments for 10 times, and report the average results.
We employ Jieba for Chinese word segmentation and keep the top INLINEFORM0 frequent words. The word embedding size is set to 128 and the other low-frequency words are replaced with the mark <UNK>. The hidden size of GRU is set to 128 for each direction in Bi-GRU. In the pair-wise attentive reader, the hidden state is set to 256 for mGRu. In the CNN layer, filter windows are set to 1, 3, 4, and 5 with each filter containing 200 feature maps. We add a dropout layer BIBREF38 after the CNN layer with a dropout rate of INLINEFORM1 . We use Adam BIBREF32 for training and set learning rate to INLINEFORM2 , INLINEFORM3 to INLINEFORM4 , INLINEFORM5 to INLINEFORM6 , INLINEFORM7 to INLINEFORM8 , batch size to 64. We employ precision, recall, F1 and accuracy for evaluation metrics. We repeat all the experiments for 10 times, and report the average results.
1809.06537
Introduction </s> Aut omatic Ġjudgment Ġprediction Ġis Ġto Ġtrain Ġa Ġmachine Ġjudge Ġto Ġdetermine Ġwhether Ġa Ġcertain Ġplea Ġin Ġa Ġgiven Ġcivil Ġcase Ġwould Ġbe Ġsupported Ġor Ġrejected . ĠIn Ġcountries Ġwith Ġcivil Ġlaw Ġsystem , Ġe . g . Ġmainland ĠChina , Ġsuch Ġprocess Ġshould Ġbe Ġdone Ġwith Ġreference Ġto Ġrelated Ġlaw Ġarticles Ġand Ġthe Ġfact Ġdescription , Ġas Ġis Ġperformed Ġby Ġa Ġhuman Ġjudge . ĠThe Ġintuition Ġcomes Ġfrom Ġthe Ġfact Ġthat Ġunder Ġcivil Ġlaw Ġsystem , Ġlaw Ġarticles Ġact Ġas Ġprinciples Ġfor Ġjur id ical Ġjudgments . ĠSuch Ġtechniques Ġwould Ġhave Ġa Ġwide Ġrange Ġof Ġpromising Ġapplications . ĠOn Ġthe Ġone Ġhand , Ġlegal Ġconsulting Ġsystems Ġcould Ġprovide Ġbetter Ġaccess Ġto Ġhigh - quality Ġlegal Ġresources Ġin Ġa Ġlow - cost Ġway Ġto Ġlegal Ġoutsiders , Ġwho Ġsuffer Ġfrom Ġthe Ġcomplicated Ġtermin ologies . ĠOn Ġthe Ġother Ġhand , Ġmachine Ġjudge Ġassistants Ġfor Ġprofessionals Ġwould Ġhelp Ġimprove Ġthe Ġefficiency Ġof Ġthe Ġjudicial Ġsystem . ĠBesides , Ġautomated Ġjudgment Ġsystem Ġcan Ġhelp Ġin Ġimproving Ġjur id ical Ġequality Ġand Ġtransparency . ĠFrom Ġanother Ġperspective , Ġthere Ġare Ġcurrently Ġ7 Ġtimes Ġmuch Ġmore Ġcivil Ġcases Ġthan Ġcriminal Ġcases Ġin Ġmainland ĠChina , Ġwith Ġannual Ġrates Ġof Ġincrease Ġof ĠIN LINE FORM 0 Ġand ĠIN LINE FORM 1 Ġrespectively , Ġmaking Ġjudgment Ġprediction Ġin Ġcivil Ġcases Ġa Ġpromising Ġapplication ĠB IB REF 0 Ġ. </s> Previous Ġworks ĠB IB REF 1 Ġ, ĠB IB REF 2 Ġ, ĠB IB REF 3 Ġ, ĠB IB REF 4 Ġformal ize Ġjudgment Ġprediction Ġas Ġthe Ġtext Ġclassification Ġtask , Ġregarding Ġeither Ġcharge Ġnames Ġor Ġbinary Ġjudgments , Ġi . e ., Ġsupport Ġor Ġreject , Ġas Ġthe Ġtarget Ġclasses . ĠThese Ġworks Ġfocus Ġon Ġthe Ġsituation Ġwhere Ġonly Ġone Ġresult Ġis Ġexpected , Ġe . g ., Ġthe ĠUS ĠSupreme ĠCourt 's Ġdecisions ĠB IB REF 2 Ġ, Ġand Ġthe Ġcharge Ġname Ġprediction Ġfor Ġcriminal Ġcases ĠB IB REF 3 Ġ. ĠDespite Ġthese Ġrecent Ġefforts Ġand Ġtheir Ġprogress , Ġautomatic Ġjudgment Ġprediction Ġin Ġcivil Ġlaw Ġsystem Ġis Ġstill Ġconfronted Ġwith Ġtwo Ġmain Ġchallenges : </s> One - to - Many ĠRel ation Ġbetween ĠCase Ġand ĠPle a . ĠEvery Ġsingle Ġcivil Ġcase Ġmay Ġcontain Ġmultiple Ġpleas Ġand Ġthe Ġresult Ġof Ġeach Ġplea Ġis Ġco - d etermined Ġby Ġrelated Ġlaw Ġarticles Ġand Ġspecific Ġaspects Ġof Ġthe Ġinvolved Ġcase . ĠFor Ġexample , Ġin Ġdivorce Ġproceedings , Ġjudgment Ġof Ġalienation Ġof Ġmutual Ġaffection Ġis Ġthe Ġkey Ġfactor Ġfor Ġgranting Ġdivorce Ġbut Ġcustody Ġof Ġchildren Ġdepends Ġon Ġwhich Ġside Ġcan Ġprovide Ġbetter Ġan Ġenvironment Ġfor Ġchildren 's Ġgrowth Ġas Ġwell Ġas Ġparents ' Ġfinancial Ġcondition . ĠHere , Ġdifferent Ġpleas Ġare Ġindependent . </s> H eter ogeneity Ġof ĠInput ĠTriple . ĠInput s Ġto Ġa Ġjudgment Ġprediction Ġsystem Ġconsist Ġof Ġthree Ġheter ogeneous Ġyet Ġcomplementary Ġparts , Ġi . e ., Ġfact Ġdescription , Ġplaintiff 's Ġplea , Ġand Ġrelated Ġlaw Ġarticles . ĠCon cat en ating Ġthem Ġtogether Ġand Ġtreating Ġthem Ġsimply Ġas Ġa Ġsequence Ġof Ġwords Ġas Ġin Ġprevious Ġworks ĠB IB REF 2 Ġ, ĠB IB REF 1 Ġwould Ġcause Ġa Ġgreat Ġloss Ġof Ġinformation . ĠThis Ġis Ġthe Ġsame Ġin Ġquestion - ans w ering Ġwhere Ġthe Ġdual Ġinputs , Ġi . e ., Ġquery Ġand Ġpassage , Ġshould Ġbe Ġmodeled Ġseparately . </s> Despite Ġthe Ġintroduction Ġof Ġthe Ġneural Ġnetworks Ġthat Ġcan Ġlearn Ġbetter Ġsemantic Ġrepresentations Ġof Ġinput Ġtext , Ġit Ġremains Ġunsolved Ġto Ġincorporate Ġproper Ġmechanisms Ġto Ġintegrate Ġthe Ġcomplementary Ġtriple Ġof Ġpleas , Ġfact Ġdescriptions , Ġand Ġlaw Ġarticles Ġtogether . </s> Insp ired Ġby Ġrecent Ġadvances Ġin Ġquestion Ġanswering Ġ( Q A ) Ġbased Ġreading Ġcomprehension Ġ( RC ) ĠB IB REF 5 Ġ, ĠB IB REF 6 Ġ, ĠB IB REF 7 Ġ, ĠB IB REF 8 Ġ, Ġwe Ġpropose Ġthe ĠLegal ĠReading ĠComp rehens ion Ġ( L RC ) Ġframework Ġfor Ġautomatic Ġjudgment Ġprediction . ĠL RC Ġincorporates Ġthe Ġreading Ġmechanism Ġfor Ġbetter Ġmodeling Ġof Ġthe Ġcomplementary Ġinputs Ġabove - mentioned , Ġas Ġis Ġdone Ġby Ġhuman Ġjudges Ġwhen Ġreferring Ġto Ġlegal Ġmaterials Ġin Ġsearch Ġof Ġsupporting Ġlaw Ġarticles . ĠReading Ġmechanism , Ġby Ġsim ulating Ġhow Ġhuman Ġconnects Ġand Ġintegrates Ġmultiple Ġtext , Ġhas Ġproven Ġan Ġeffective Ġmodule Ġin ĠRC Ġtasks . ĠWe Ġargue Ġthat Ġapplying Ġthe Ġreading Ġmechanism Ġin Ġa Ġproper Ġway Ġamong Ġthe Ġtriple ts Ġcan Ġobtain Ġa Ġbetter Ġunderstanding Ġand Ġmore Ġinformative Ġrepresentation Ġof Ġthe Ġoriginal Ġtext , Ġand Ġfurther Ġimprove Ġperformance Ġ. ĠTo Ġinstant iate Ġthe Ġframework , Ġwe Ġpropose Ġan Ġend - to - end Ġneural Ġnetwork Ġmodel Ġnamed ĠAuto Judge . </s> For Ġexperiments , Ġwe Ġtrain Ġand Ġevaluate Ġour Ġmodels Ġin Ġthe Ġcivil Ġlaw Ġsystem Ġof Ġmainland ĠChina . ĠWe Ġcollect Ġand Ġconstruct Ġa Ġlarge - scale Ġreal - world Ġdata Ġset Ġof ĠIN LINE FORM 0 Ġcase Ġdocuments Ġthat Ġthe ĠSupreme ĠPeople 's ĠCourt Ġof ĠPeople 's ĠRepublic Ġof ĠChina Ġhas Ġmade Ġpublicly Ġavailable . ĠFact Ġdescription , Ġpleas , Ġand Ġresults Ġcan Ġbe Ġextracted Ġeasily Ġfrom Ġthese Ġcase Ġdocuments Ġwith Ġregular Ġexpressions , Ġsince Ġthe Ġoriginal Ġdocuments Ġhave Ġspecial Ġtyp ographical Ġcharacteristics Ġindicating Ġthe Ġdiscourse Ġstructure . ĠWe Ġalso Ġtake Ġinto Ġaccount Ġlaw Ġarticles Ġand Ġtheir Ġcorresponding Ġjur id ical Ġinterpretations . ĠWe Ġalso Ġimplement Ġand Ġevaluate Ġprevious Ġmethods Ġon Ġour Ġdataset , Ġwhich Ġprove Ġto Ġbe Ġstrong Ġbas elines . </s> Our Ġexperiment Ġresults Ġshow Ġsignificant Ġimprovements Ġover Ġprevious Ġmethods . ĠFurther Ġexperiments Ġdemonstrate Ġthat Ġour Ġmodel Ġalso Ġachieves Ġconsiderable Ġimprovement Ġover Ġother Ġoff - the - she lf Ġstate - of - the - art Ġmodels Ġunder Ġclassification Ġand Ġquestion Ġanswering Ġframework Ġrespectively . ĠAb lation Ġtests Ġcarried Ġout Ġby Ġtaking Ġoff Ġsome Ġcomponents Ġof Ġour Ġmodel Ġfurther Ġprove Ġits Ġrobust ness Ġand Ġeffectiveness . </s> To Ġsum Ġup , Ġour Ġcontributions Ġare Ġas Ġfollows : </s> ( 1 ) ĠWe Ġintroduce Ġreading Ġmechanism Ġand Ġre - form al ize Ġjudgment Ġprediction Ġas ĠLegal ĠReading ĠComp rehens ion Ġto Ġbetter Ġmodel Ġthe Ġcomplementary Ġinputs . </s> ( 2 ) ĠWe Ġconstruct Ġa Ġreal - world Ġdataset Ġfor Ġexperiments , Ġand Ġplan Ġto Ġpublish Ġit Ġfor Ġfurther Ġresearch . </s> ( 3 ) ĠBesides Ġbas elines Ġfrom Ġprevious Ġworks , Ġwe Ġalso Ġcarry Ġout Ġcomprehensive Ġexperiments Ġcomparing Ġdifferent Ġexisting Ġdeep Ġneural Ġnetwork Ġmethods Ġon Ġour Ġdataset . ĠSupported Ġby Ġthese Ġexperiments , Ġimprovements Ġachieved Ġby ĠL RC Ġprove Ġto Ġbe Ġrobust . </s> Jud gment ĠPrediction </s> Aut omatic Ġjudgment Ġprediction Ġhas Ġbeen Ġstudied Ġfor Ġdecades . ĠAt Ġthe Ġvery Ġfirst Ġstage Ġof Ġjudgment Ġprediction Ġstudies , Ġresearchers Ġfocus Ġon Ġmathematical Ġand Ġstatistical Ġanalysis Ġof Ġexisting Ġcases , Ġwithout Ġany Ġconclusions Ġor Ġmethod ologies Ġon Ġhow Ġto Ġpredict Ġthem ĠB IB REF 9 Ġ, ĠB IB REF 10 Ġ, ĠB IB REF 11 Ġ, ĠB IB REF 12 Ġ, ĠB IB REF 13 Ġ, ĠB IB REF 14 Ġ. ĠRecent Ġattempts Ġconsider Ġjudgment Ġprediction Ġunder Ġthe Ġtext Ġclassification Ġframework . ĠMost Ġof Ġthese Ġworks Ġextract Ġefficient Ġfeatures Ġfrom Ġtext Ġ( e . g ., ĠN - gram s ) ĠB IB REF 15 Ġ, ĠB IB REF 4 Ġ, ĠB IB REF 1 Ġ, ĠB IB REF 16 Ġ, ĠB IB REF 17 Ġor Ġcase Ġprofiles Ġ( e . g ., Ġdates , Ġterms , Ġlocations Ġand Ġtypes ) ĠB IB REF 2 Ġ. ĠAll Ġthese Ġmethods Ġrequire Ġa Ġlarge Ġamount Ġof Ġhuman Ġeffort Ġto Ġdesign Ġfeatures Ġor Ġannot ate Ġcases . ĠBesides , Ġthey Ġalso Ġsuffer Ġfrom Ġgeneral ization Ġissue Ġwhen Ġapplied Ġto Ġother Ġscenarios . </s> Mot ivated Ġby Ġthe Ġsuccessful Ġapplication Ġof Ġdeep Ġneural Ġnetworks , ĠLuo Ġet Ġal . ĠB IB REF 3 Ġintroduce Ġan Ġattention - based Ġneural Ġmodel Ġto Ġpredict Ġcharges Ġof Ġcriminal Ġcases , Ġand Ġverify Ġthe Ġeffectiveness Ġof Ġtaking Ġlaw Ġarticles Ġinto Ġconsideration . ĠNevertheless , Ġthey Ġstill Ġfall Ġinto Ġthe Ġtext Ġclassification Ġframework Ġand Ġlack Ġthe Ġability Ġto Ġhandle Ġmultiple Ġinputs Ġwith Ġmore Ġcomplicated Ġstructures . </s> Text ĠClassification </s> As Ġthe Ġbasis Ġof Ġprevious Ġjudgment Ġprediction Ġworks , Ġtypical Ġtext Ġclassification Ġtask Ġtakes Ġa Ġsingle Ġtext Ġcontent Ġas Ġinput Ġand Ġpredicts Ġthe Ġcategory Ġit Ġbelongs Ġto . ĠRecent Ġworks Ġusually Ġemploy Ġneural Ġnetworks Ġto Ġmodel Ġthe Ġinternal Ġstructure Ġof Ġa Ġsingle Ġinput ĠB IB REF 18 Ġ, ĠB IB REF 19 Ġ, ĠB IB REF 20 Ġ, ĠB IB REF 21 Ġ. </s> There Ġalso Ġexists Ġanother Ġthread Ġof Ġtext Ġclassification Ġcalled Ġentail ment Ġprediction . ĠMethods Ġproposed Ġin ĠB IB REF 22 Ġ, ĠB IB REF 23 Ġare Ġintended Ġfor Ġcomplementary Ġinputs , Ġbut Ġthe Ġmechanisms Ġcan Ġbe Ġconsidered Ġas Ġa Ġsimplified Ġversion Ġof Ġreading Ġcomprehension . </s> Reading ĠComp rehens ion </s> Reading Ġcomprehension Ġis Ġa Ġrelevant Ġtask Ġto Ġmodel Ġheter ogeneous Ġand Ġcomplementary Ġinputs , Ġwhere Ġan Ġanswer Ġis Ġpredicted Ġgiven Ġtwo Ġchannels Ġof Ġinputs , Ġi . e . Ġa Ġtextual Ġpassage Ġand Ġa Ġquery . ĠConsider able Ġprogress Ġhas Ġbeen Ġmade ĠB IB REF 6 Ġ, ĠB IB REF 24 Ġ, ĠB IB REF 5 Ġ. ĠThese Ġmodels Ġemploy Ġvarious Ġattention Ġmechanism Ġto Ġmodel Ġthe Ġinteraction Ġbetween Ġpassage Ġand Ġquery . ĠInspired Ġby Ġthe Ġadvantage Ġof Ġreading Ġcomprehension Ġmodels Ġon Ġmodeling Ġmultiple Ġinputs , Ġwe Ġapply Ġthis Ġidea Ġinto Ġthe Ġlegal Ġarea Ġand Ġpropose Ġlegal Ġreading Ġcomprehension Ġfor Ġjudgment Ġprediction . </s> Con ventional ĠReading ĠComp rehens ion </s> Con ventional Ġreading Ġcomprehension ĠB IB REF 25 Ġ, ĠB IB REF 26 Ġ, ĠB IB REF 7 Ġ, ĠB IB REF 8 Ġusually Ġconsiders Ġreading Ġcomprehension Ġas Ġpredicting Ġthe Ġanswer Ġgiven Ġa Ġpassage Ġand Ġa Ġquery , Ġwhere Ġthe Ġanswer Ġcould Ġbe Ġa Ġsingle Ġword , Ġa Ġtext Ġspan Ġof Ġthe Ġoriginal Ġpassage , Ġchosen Ġfrom Ġanswer Ġcandidates , Ġor Ġgenerated Ġby Ġhuman Ġannot ators . </s> Generally , Ġan Ġinstance Ġin ĠRC Ġis Ġrepresented Ġas Ġa Ġtriple ĠIN LINE FORM 0 Ġ, Ġwhere ĠIN LINE FORM 1 Ġ, ĠIN LINE FORM 2 Ġand ĠIN LINE FORM 3 Ġcorrespond Ġto ĠIN LINE FORM 4 Ġ, ĠIN LINE FORM 5 Ġand ĠIN LINE FORM 6 Ġrespectively . ĠGiven Ġa Ġtriple ĠIN LINE FORM 7 Ġ, ĠRC Ġtakes Ġthe Ġpair ĠIN LINE FORM 8 Ġas Ġthe Ġinput Ġand Ġemploys Ġattention - based Ġneural Ġmodels Ġto Ġconstruct Ġan Ġefficient Ġrepresentation . ĠAfterwards , Ġthe Ġrepresentation Ġis Ġfed Ġinto Ġthe Ġoutput Ġlayer Ġto Ġselect Ġor Ġgenerate Ġan ĠIN LINE FORM 9 Ġ. </s> Legal ĠReading ĠComp rehens ion </s> Ex isting Ġworks Ġusually Ġformal ize Ġjudgment Ġprediction Ġas Ġa Ġtext Ġclassification Ġtask Ġand Ġfocus Ġon Ġextracting Ġwell - designed Ġfeatures Ġof Ġspecific Ġcases . ĠSuch Ġsimpl ification Ġignores Ġthat Ġthe Ġjudgment Ġof Ġa Ġcase Ġis Ġdetermined Ġby Ġits Ġfact Ġdescription Ġand Ġmultiple Ġpleas . ĠMoreover , Ġthe Ġfinal Ġjudgment Ġshould Ġact Ġup Ġto Ġthe Ġlegal Ġprovisions , Ġespecially Ġin Ġcivil Ġlaw Ġsystems . ĠTherefore , Ġhow Ġto Ġintegrate Ġthe Ġinformation Ġ( i . e ., Ġfact Ġdescriptions , Ġpleas , Ġand Ġlaw Ġarticles ) Ġin Ġa Ġreasonable Ġway Ġis Ġcritical Ġfor Ġjudgment Ġprediction . </s> Insp ired Ġby Ġthe Ġsuccessful Ġapplication Ġof ĠRC , Ġwe Ġpropose Ġa Ġframework Ġof ĠLegal ĠReading ĠComp rehens ion ( L RC ) Ġfor Ġjudgment Ġprediction Ġin Ġthe Ġlegal Ġarea . ĠAs Ġillustrated Ġin ĠFig . ĠFIG REF 1 Ġ, Ġfor Ġeach Ġplea Ġin Ġa Ġgiven Ġcase , Ġthe Ġprediction Ġof Ġjudgment Ġresult Ġis Ġmade Ġbased Ġthe Ġfact Ġdescription Ġand Ġthe Ġpotentially Ġrelevant Ġlaw Ġarticles . </s> In Ġa Ġnutshell , ĠL RC Ġcan Ġbe Ġformal ized Ġas Ġthe Ġfollowing Ġquadru plet Ġtask : ĠDIS PLAY FORM 0 </s> where ĠIN LINE FORM 0 Ġis Ġthe Ġfact Ġdescription , ĠIN LINE FORM 1 Ġis Ġthe Ġplea , ĠIN LINE FORM 2 Ġis Ġthe Ġlaw Ġarticles Ġand ĠIN LINE FORM 3 Ġis Ġthe Ġresult . ĠGiven ĠIN LINE FORM 4 Ġ, ĠL RC Ġaims Ġto Ġpredict Ġthe Ġjudgment Ġresult Ġas ĠDIS PLAY FORM 0 </s> The Ġprobability Ġis Ġcalculated Ġwith Ġrespect Ġto Ġthe Ġinteraction Ġamong Ġthe Ġtriple ĠIN LINE FORM 0 Ġ, Ġwhich Ġwill Ġdraw Ġon Ġthe Ġexperience Ġof Ġthe Ġinteraction Ġbetween ĠIN LINE FORM 1 Ġpairs Ġin ĠRC . </s> To Ġsummarize , ĠL RC Ġis Ġinnovative Ġin Ġthe Ġfollowing Ġaspects : </s> ( 1 ) ĠWhile Ġprevious Ġworks Ġfit Ġthe Ġproblem Ġinto Ġtext Ġclassification Ġframework , ĠL RC Ġre - form al izes Ġthe Ġway Ġto Ġapproach Ġsuch Ġproblems . ĠThis Ġnew Ġframework Ġprovides Ġthe Ġability Ġto Ġdeal Ġwith Ġthe Ġheterogeneity Ġof Ġthe Ġcomplementary Ġinputs . </s> ( 2 ) ĠRather Ġthan Ġemploying Ġconventional ĠRC Ġmodels Ġto Ġhandle Ġpair - wise Ġtext Ġinformation Ġin Ġthe Ġlegal Ġarea , ĠL RC Ġtakes Ġthe Ġcritical Ġlaw Ġarticles Ġinto Ġconsideration Ġand Ġmodels Ġthe Ġfacts , Ġpleas , Ġand Ġlaw Ġarticles Ġjointly Ġfor Ġjudgment Ġprediction , Ġwhich Ġis Ġmore Ġsuitable Ġto Ġsimulate Ġthe Ġhuman Ġmode Ġof Ġdealing Ġwith Ġcases . </s> Methods </s> We Ġpropose Ġa Ġnovel Ġjudgment Ġprediction Ġmodel ĠAuto Judge Ġto Ġinstant iate Ġthe ĠL RC Ġframework . ĠAs Ġshown Ġin ĠFig . ĠFIG REF 6 Ġ, ĠAuto Judge Ġconsists Ġof Ġthree Ġflexible Ġmodules , Ġincluding Ġa Ġtext Ġenc oder , Ġa Ġpair - wise Ġattentive Ġreader , Ġand Ġan Ġoutput Ġmodule . </s> In Ġthe Ġfollowing Ġparts , Ġwe Ġgive Ġa Ġdetailed Ġintroduction Ġto Ġthese Ġthree Ġmodules . </s> Text ĠEnc oder </s> As Ġillustrated Ġin ĠFig . ĠFIG REF 6 Ġ, ĠText ĠEnc oder Ġaims Ġto Ġencode Ġthe Ġword Ġsequences Ġof Ġinputs Ġinto Ġcontinuous Ġrepresentation Ġsequences . </s> Form ally , Ġconsider Ġa Ġfact Ġdescription ĠIN LINE FORM 0 Ġ, Ġa Ġplea ĠIN LINE FORM 1 Ġ, Ġand Ġthe Ġrelevant Ġlaw Ġarticles ĠIN LINE FORM 2 Ġ, Ġwhere ĠIN LINE FORM 3 Ġdenotes Ġthe ĠIN LINE FORM 4 Ġ- th Ġword Ġin Ġthe Ġsequence Ġand ĠIN LINE FORM 5 Ġare Ġthe Ġlengths Ġof Ġword Ġsequences ĠIN LINE FORM 6 Ġrespectively . ĠFirst , Ġwe Ġconvert Ġthe Ġwords Ġto Ġtheir Ġrespective Ġword Ġembed d ings Ġto Ġobtain ĠIN LINE FORM 7 Ġ, ĠIN LINE FORM 8 Ġand ĠIN LINE FORM 9 Ġ, Ġwhere ĠIN LINE FORM 10 Ġ. ĠAfterwards , Ġwe Ġemploy Ġbi - direction al ĠGR U ĠB IB REF 27 Ġ, ĠB IB REF 28 Ġ, ĠB IB REF 29 Ġto Ġproduce Ġthe Ġencoded Ġrepresentation ĠIN LINE FORM 11 Ġof Ġall Ġwords Ġas Ġfollows : ĠDIS PLAY FORM 0 </s> Note Ġthat , Ġwe Ġadopt Ġdifferent Ġbi - direction al ĠGR Us Ġto Ġencode Ġfact Ġdescriptions , Ġpleas , Ġand Ġlaw Ġarticles Ġrespectively ( den oted Ġas ĠIN LINE FORM 0 Ġ, ĠIN LINE FORM 1 Ġ, Ġand ĠIN LINE FORM 2 Ġ). ĠWith Ġthese Ġtext Ġenc od ers , ĠIN LINE FORM 3 Ġ, ĠIN LINE FORM 4 Ġ, Ġand ĠIN LINE FORM 5 Ġare Ġconverting Ġinto ĠIN LINE FORM 6 Ġ, ĠIN LINE FORM 7 Ġ, Ġand ĠIN LINE FORM 8 Ġ. </s> P air - W ise ĠAtt ent ive ĠReader </s> How Ġto Ġmodel Ġthe Ġinteractions Ġamong Ġthe Ġinput Ġtext Ġis Ġthe Ġmost Ġimportant Ġproblem Ġin Ġreading Ġcomprehension . ĠIn ĠAuto Judge , Ġwe Ġemploy Ġa Ġpair - wise Ġattentive Ġreader Ġto Ġprocess ĠIN LINE FORM 0 Ġand ĠIN LINE FORM 1 Ġrespectively . ĠMore Ġspecifically , Ġwe Ġpropose Ġto Ġuse Ġpair - wise Ġmutual Ġattention Ġmechanism Ġto Ġcapture Ġthe Ġcomplex Ġsemantic Ġinteraction Ġbetween Ġtext Ġpairs , Ġas Ġwell Ġas Ġincreasing Ġthe Ġinterpret ability Ġof ĠAuto Judge . </s> For Ġeach Ġinput Ġpair ĠIN LINE FORM 0 Ġor ĠIN LINE FORM 1 Ġ, Ġwe Ġemploy Ġpair - wise Ġmutual Ġattention Ġto Ġselect Ġrelevant Ġinformation Ġfrom Ġfact Ġdescriptions ĠIN LINE FORM 2 Ġand Ġproduce Ġmore Ġinformative Ġrepresentation Ġsequences . </s> As Ġa Ġvariant Ġof Ġthe Ġoriginal Ġattention Ġmechanism ĠB IB REF 28 Ġ, Ġwe Ġdesign Ġthe Ġpair - wise Ġmutual Ġattention Ġunit Ġas Ġa ĠGR U Ġwith Ġinternal Ġmemories Ġden oted Ġas Ġm GR U . </s> Taking Ġthe Ġrepresentation Ġsequence Ġpair ĠIN LINE FORM 0 Ġfor Ġinstance , Ġm GR U Ġstores Ġthe Ġfact Ġsequence ĠIN LINE FORM 1 Ġinto Ġits Ġmemories . ĠFor Ġeach Ġtimestamp ĠIN LINE FORM 2 Ġ, Ġit Ġselects Ġrelevant Ġfact Ġinformation ĠIN LINE FORM 3 Ġfrom Ġthe Ġmemories Ġas Ġfollows , ĠDIS PLAY FORM 0 </s> Here , Ġthe Ġweight ĠIN LINE FORM 0 Ġis Ġthe Ġsoft max Ġvalue Ġas ĠDIS PLAY FORM 0 </s> Note Ġthat , ĠIN LINE FORM 0 Ġrepresents Ġthe Ġrelevance Ġbetween ĠIN LINE FORM 1 Ġand ĠIN LINE FORM 2 Ġ. ĠIt Ġis Ġcalculated Ġas Ġfollows , ĠDIS PLAY FORM 0 </s> Here , ĠIN LINE FORM 0 Ġis Ġthe Ġlast Ġhidden Ġstate Ġin Ġthe ĠGR U , Ġwhich Ġwill Ġbe Ġintroduced Ġin Ġthe Ġfollowing Ġpart . ĠIN LINE FORM 1 Ġis Ġa Ġweight Ġvector , Ġand ĠIN LINE FORM 2 Ġ, ĠIN LINE FORM 3 Ġ, ĠIN LINE FORM 4 Ġare Ġattention Ġmetrics Ġof Ġour Ġproposed Ġpair - wise Ġattention Ġmechanism . </s> With Ġthe Ġrelevant Ġfact Ġinformation ĠIN LINE FORM 0 Ġand ĠIN LINE FORM 1 Ġ, Ġwe Ġget Ġthe ĠIN LINE FORM 2 Ġ- th Ġinput Ġof Ġm GR U Ġas ĠDIS PLAY FORM 0 </s> where ĠIN LINE FORM 0 Ġindicates Ġthe Ġconc aten ation Ġoperation . </s> Then , Ġwe Ġfeed ĠIN LINE FORM 0 Ġinto ĠGR U Ġto Ġget Ġmore Ġinformative Ġrepresentation Ġsequence ĠIN LINE FORM 1 Ġas Ġfollows , ĠDIS PLAY FORM 0 </s> For Ġthe Ġinput Ġpair ĠIN LINE FORM 0 Ġ, Ġwe Ġcan Ġget ĠIN LINE FORM 1 Ġin Ġthe Ġsame Ġway . ĠTherefore , Ġwe Ġomit Ġthe Ġimplementation Ġdetails ĠHere . </s> Similar Ġstructures Ġwith Ġattention Ġmechanism Ġare Ġalso Ġapplied Ġin ĠB IB REF 5 Ġ, ĠB IB REF 30 Ġ, ĠB IB REF 31 Ġ, ĠB IB REF 28 Ġto Ġobtain Ġmutually Ġaware Ġrepresentations Ġin Ġreading Ġcomprehension Ġmodels , Ġwhich Ġsignificantly Ġimprove Ġthe Ġperformance Ġof Ġthis Ġtask . </s> Output ĠLayer </s> Using Ġtext Ġenc oder Ġand Ġpair - wise Ġattentive Ġreader , Ġthe Ġinitial Ġinput Ġtriple ĠIN LINE FORM 0 Ġhas Ġbeen Ġconverted Ġinto Ġtwo Ġsequences , Ġi . e ., ĠIN LINE FORM 1 Ġand ĠIN LINE FORM 2 Ġ, Ġwhere ĠIN LINE FORM 3 Ġis Ġdefined Ġsimilarly Ġto ĠIN LINE FORM 4 Ġ. ĠThese Ġsequences Ġreserve Ġcomplex Ġsemantic Ġinformation Ġabout Ġthe Ġpleas Ġand Ġlaw Ġarticles , Ġand Ġfilter Ġout Ġirrelevant Ġinformation Ġin Ġfact Ġdescriptions . </s> With Ġthese Ġtwo Ġsequences , Ġwe Ġconc aten ate ĠIN LINE FORM 0 Ġand ĠIN LINE FORM 1 Ġalong Ġthe Ġsequence Ġlength Ġdimension Ġto Ġgenerate Ġthe Ġsequence ĠIN LINE FORM 2 Ġ. ĠSince Ġwe Ġhave Ġemployed Ġseveral ĠGR U Ġlayers Ġto Ġencode Ġthe Ġsequential Ġinputs , Ġanother Ġrecurrent Ġlayer Ġmay Ġbe Ġredundant . ĠTherefore , Ġwe Ġutilize Ġa Ġ1 - layer ĠCNN ĠB IB REF 18 Ġto Ġcapture Ġthe Ġlocal Ġstructure Ġand Ġgenerate Ġthe Ġrepresentation Ġvector Ġfor Ġthe Ġfinal Ġprediction . </s> Assuming ĠIN LINE FORM 0 Ġis Ġthe Ġpredicted Ġprobability Ġthat Ġthe Ġplea Ġin Ġthe Ġcase Ġsample Ġwould Ġbe Ġsupported Ġand ĠIN LINE FORM 1 Ġis Ġthe Ġgold Ġstandard , ĠAuto Judge Ġaims Ġto Ġminimize Ġthe Ġcross - ent ropy Ġas Ġfollows , ĠDIS PLAY FORM 0 </s> where ĠIN LINE FORM 0 Ġis Ġthe Ġnumber Ġof Ġtraining Ġdata . ĠAs Ġall Ġthe Ġcalculation Ġin Ġour Ġmodel Ġis Ġdifferent iable , Ġwe Ġemploy ĠAdam ĠB IB REF 32 Ġfor Ġoptimization . </s> Exper iments </s> To Ġevaluate Ġthe Ġproposed ĠL RC Ġframework Ġand Ġthe ĠAuto Judge Ġmodel , Ġwe Ġcarry Ġout Ġa Ġseries Ġof Ġexperiments Ġon Ġthe Ġdivorce Ġproceedings , Ġa Ġtypical Ġyet Ġcomplex Ġfield Ġof Ġcivil Ġcases . ĠDiv orce Ġproceedings Ġoften Ġcome Ġwith Ġseveral Ġkinds Ġof Ġpleas , Ġe . g . Ġseeking Ġdivorce , Ġcustody Ġof Ġchildren , Ġcompensation , Ġand Ġmaintenance , Ġwhich Ġfocuses Ġon Ġdifferent Ġaspects Ġand Ġthus Ġmakes Ġit Ġa Ġchallenge Ġfor Ġjudgment Ġprediction . </s> Dat as et ĠConstruction Ġfor ĠEvaluation </s> Since Ġnone Ġof Ġthe Ġdatasets Ġfrom Ġprevious Ġworks Ġhave Ġbeen Ġpublished , Ġwe Ġdecide Ġto Ġbuild Ġa Ġnew Ġone . ĠWe Ġrandomly Ġcollect ĠIN LINE FORM 0 Ġcases Ġfrom ĠChina ĠJud gments ĠOnline , Ġamong Ġwhich ĠIN LINE FORM 1 Ġcases Ġare Ġfor Ġtraining , ĠIN LINE FORM 2 Ġeach Ġfor Ġvalidation Ġand Ġtesting . ĠAmong Ġthe Ġoriginal Ġcases , ĠIN LINE FORM 3 Ġare Ġgranted Ġdivorce Ġand Ġothers Ġnot . ĠThere Ġare ĠIN LINE FORM 4 Ġvalid Ġpleas Ġin Ġtotal , Ġwith ĠIN LINE FORM 5 Ġsupported Ġand ĠIN LINE FORM 6 Ġrejected . ĠNote Ġthat , Ġif Ġthe Ġdivorce Ġplea Ġin Ġa Ġcase Ġis Ġnot Ġgranted , Ġthe Ġother Ġpleas Ġof Ġthis Ġcase Ġwill Ġnot Ġbe Ġconsidered Ġby Ġthe Ġjudge . ĠCase Ġmaterials Ġare Ġall Ġnatural Ġlanguage Ġsentences , Ġwith Ġaverage ly ĠIN LINE FORM 7 Ġtokens Ġper Ġfact Ġdescription Ġand ĠIN LINE FORM 8 Ġper Ġplea . ĠThere Ġare Ġ62 Ġrelevant Ġlaw Ġarticles Ġin Ġtotal , Ġeach Ġwith ĠIN LINE FORM 9 Ġtokens Ġaverage ly . ĠNote Ġthat Ġthe Ġcase Ġdocuments Ġinclude Ġspecial Ġtyp ographical Ġsignals , Ġmaking Ġit Ġeasy Ġto Ġextract Ġlabeled Ġdata Ġwith Ġregular Ġexpression . </s> We Ġapply Ġsome Ġrules Ġwith Ġlegal Ġprior Ġto Ġpre process Ġthe Ġdataset Ġaccording Ġto Ġprevious Ġworks ĠB IB REF 33 Ġ, ĠB IB REF 34 Ġ, ĠB IB REF 35 Ġ, Ġwhich Ġhave Ġproved Ġeffective Ġin Ġour Ġexperiments . </s> Name ĠReplacement : ĠAll Ġnames Ġin Ġcase Ġdocuments Ġare Ġreplaced Ġwith Ġmarks Ġindicating Ġtheir Ġroles , Ġinstead Ġof Ġsimply Ġanonym izing Ġthem , Ġe . g . Ġ< Pl ant iff >, Ġ< Def endant >, Ġ< D aughter _ x > Ġand Ġso Ġon . ĠSince ĠâĢ ľ all Ġare Ġequal Ġbefore Ġthe Ġlaw âĢ Ŀ , Ġnames Ġshould Ġmake Ġno Ġmore Ġdifference Ġthan Ġwhat Ġrole Ġthey Ġtake . </s> Law ĠArticle ĠF iltration Ġ: ĠSince Ġmost Ġaccessible Ġdivorce Ġproceeding Ġdocuments Ġdo Ġnot Ġcontain Ġground - truth Ġfine - gr ained Ġarticles , Ġwe Ġuse Ġan Ġun super vised Ġmethod Ġinstead . ĠFirst , Ġwe Ġextract Ġall Ġthe Ġarticles Ġfrom Ġthe Ġlaw Ġtext Ġwith Ġregular Ġexpression . ĠAfterwards , Ġwe Ġselect Ġthe Ġmost Ġrelevant Ġ10 Ġarticles Ġaccording Ġto Ġthe Ġfact Ġdescriptions Ġas Ġfollows . ĠWe Ġobtain Ġsentence Ġrepresentation Ġwith ĠCB OW ĠB IB REF 36 Ġ, ĠB IB REF 37 Ġweighted Ġby Ġinverse Ġdocument Ġfrequency , Ġand Ġcalculate Ġcos ine Ġdistance Ġbetween Ġcases Ġand Ġlaw Ġarticles . ĠWord Ġembed d ings Ġare Ġpre - trained Ġwith ĠChinese ĠWikipedia Ġpages . ĠAs Ġthe Ġfinal Ġstep , Ġwe Ġextract Ġtop Ġ5 Ġrelevant Ġarticles Ġfor Ġeach Ġsample Ġrespectively Ġfrom Ġthe Ġmain Ġmarriage Ġlaw Ġarticles Ġand Ġtheir Ġinterpretations , Ġwhich Ġare Ġequally Ġimportant . ĠWe Ġmanually Ġcheck Ġthe Ġextracted Ġarticles Ġfor Ġ100 Ġcases Ġto Ġensure Ġthat Ġthe Ġextraction Ġquality Ġis Ġfairly Ġgood Ġand Ġacceptable . </s> The Ġfil t ration Ġprocess Ġis Ġautomatic Ġand Ġfully Ġun super vised Ġsince Ġthe Ġoriginal Ġdocuments Ġhave Ġno Ġground - truth Ġlabels Ġfor Ġfine - gr ained Ġlaw Ġarticles , Ġand Ġcoarse - gr ained Ġlaw - articles Ġonly Ġprovide Ġlimited Ġinformation . ĠWe Ġalso Ġexperiment Ġwith Ġthe Ġground - truth Ġarticles , Ġbut Ġonly Ġa Ġsmall Ġfraction Ġof Ġthem Ġhas Ġfine - gr ained Ġones , Ġand Ġthey Ġare Ġusually Ġnot Ġavailable Ġin Ġreal - world Ġscenarios . </s> Im plementation ĠDetails </s> We Ġemploy ĠJ ie ba Ġfor ĠChinese Ġword Ġsegment ation Ġand Ġkeep Ġthe Ġtop ĠIN LINE FORM 0 Ġfrequent Ġwords . ĠThe Ġword Ġembed ding Ġsize Ġis Ġset Ġto Ġ128 Ġand Ġthe Ġother Ġlow - frequency Ġwords Ġare Ġreplaced Ġwith Ġthe Ġmark Ġ< UN K >. ĠThe Ġhidden Ġsize Ġof ĠGR U Ġis Ġset Ġto Ġ128 Ġfor Ġeach Ġdirection Ġin ĠBi - GR U . ĠIn Ġthe Ġpair - wise Ġattentive Ġreader , Ġthe Ġhidden Ġstate Ġis Ġset Ġto Ġ256 Ġfor Ġm GR u . ĠIn Ġthe ĠCNN Ġlayer , Ġfilter Ġwindows Ġare Ġset Ġto Ġ1 , Ġ3 , Ġ4 , Ġand Ġ5 Ġwith Ġeach Ġfilter Ġcontaining Ġ200 Ġfeature Ġmaps . ĠWe Ġadd Ġa Ġdrop out Ġlayer ĠB IB REF 38 Ġafter Ġthe ĠCNN Ġlayer Ġwith Ġa Ġdrop out Ġrate Ġof ĠIN LINE FORM 1 Ġ. ĠWe Ġuse ĠAdam ĠB IB REF 32 Ġfor Ġtraining Ġand Ġset Ġlearning Ġrate Ġto ĠIN LINE FORM 2 Ġ, ĠIN LINE FORM 3 Ġto ĠIN LINE FORM 4 Ġ, ĠIN LINE FORM 5 Ġto ĠIN LINE FORM 6 Ġ, ĠIN LINE FORM 7 Ġto ĠIN LINE FORM 8 Ġ, Ġbatch Ġsize Ġto Ġ64 . ĠWe Ġemploy Ġprecision , Ġrecall , ĠF 1 Ġand Ġaccuracy Ġfor Ġevaluation Ġmetrics . ĠWe Ġrepeat Ġall Ġthe Ġexperiments Ġfor Ġ10 Ġtimes , Ġand Ġreport Ġthe Ġaverage Ġresults . </s> Bas elines </s> For Ġcomparison , Ġwe Ġadopt Ġand Ġre - im plement Ġthree Ġkinds Ġof Ġbas elines Ġas Ġfollows : </s> We Ġimplement Ġan ĠS VM Ġwith Ġlex ical Ġfeatures Ġin Ġaccordance Ġwith Ġprevious Ġworks ĠB IB REF 16 Ġ, ĠB IB REF 17 Ġ, ĠB IB REF 1 Ġ, ĠB IB REF 15 Ġ, ĠB IB REF 4 Ġand Ġselect Ġthe Ġbest Ġfeature Ġset Ġon Ġthe Ġdevelopment Ġset . </s> We Ġimplement Ġand Ġfine - t une Ġa Ġseries Ġof Ġneural Ġtext Ġclass ifiers , Ġincluding Ġattention - based Ġmethod ĠB IB REF 3 Ġand Ġother Ġmethods Ġwe Ġdeem Ġimportant . ĠCNN ĠB IB REF 18 Ġand ĠGR U ĠB IB REF 27 Ġ, ĠB IB REF 21 Ġtake Ġas Ġinput Ġthe Ġconc aten ation Ġof Ġfact Ġdescription Ġand Ġplea . ĠSimilarly , ĠCNN / GR U + law Ġrefers Ġto Ġusing Ġthe Ġconc aten ation Ġof Ġfact Ġdescription , Ġplea Ġand Ġlaw Ġarticles Ġas Ġinputs . </s> We Ġimplement Ġand Ġtrain Ġsome Ġoff - the - she lf ĠRC Ġmodels , Ġincluding Ġr - net ĠB IB REF 5 Ġand ĠAo A ĠB IB REF 6 Ġ, Ġwhich Ġare Ġthe Ġleading Ġmodels Ġon ĠS Qu AD Ġleader board . ĠIn Ġour Ġinitial Ġexperiments , Ġthese Ġmodels Ġtake Ġfact Ġdescription Ġas Ġpassage Ġand Ġplea Ġas Ġquery . ĠFurther , ĠLaw Ġarticles Ġare Ġadded Ġto Ġthe Ġfact Ġdescription Ġas Ġa Ġpart Ġof Ġthe Ġreading Ġmaterials , Ġwhich Ġis Ġa Ġsimple Ġway Ġto Ġconsider Ġthem Ġas Ġwell . </s> Results Ġand ĠAnalysis </s> From ĠTable ĠT AB REF 37 Ġ, Ġwe Ġhave Ġthe Ġfollowing Ġobservations : </s> ( 1 ) ĠAuto Judge Ġconsistently Ġand Ġsignificantly Ġoutper forms Ġall Ġthe Ġbas elines , Ġincluding ĠRC Ġmodels Ġand Ġother Ġneural Ġtext Ġclassification Ġmodels , Ġwhich Ġshows Ġthe Ġeffectiveness Ġand Ġrobust ness Ġof Ġour Ġmodel . </s> ( 2 ) ĠRC Ġmodels Ġachieve Ġbetter Ġperformance Ġthan Ġmost Ġtext Ġclassification Ġmodels Ġ( excluding ĠGR U + Att ention ), Ġwhich Ġindicates Ġthat Ġreading Ġmechanism Ġis Ġa Ġbetter Ġway Ġto Ġintegrate Ġinformation Ġfrom Ġheter ogeneous Ġyet Ġcomplementary Ġinputs . ĠOn Ġthe Ġcontrary , Ġsimply Ġadding Ġlaw Ġarticles Ġas Ġa Ġpart Ġof Ġthe Ġreading Ġmaterials Ġmakes Ġno Ġdifference Ġin Ġperformance . ĠNote Ġthat , ĠGR U + Att ention Ġemploy Ġsimilar Ġattention Ġmechanism Ġas ĠRC Ġdoes Ġand Ġtakes Ġadditional Ġlaw Ġarticles Ġinto Ġconsideration , Ġthus Ġachieves Ġcomparable Ġperformance Ġwith ĠRC Ġmodels . </s> ( 3 ) ĠComp aring Ġwith Ġconventional ĠRC Ġmodels , ĠAuto Judge Ġachieves Ġsignificant Ġimprovement Ġwith Ġthe Ġconsideration Ġof Ġadditional Ġlaw Ġarticles . ĠIt Ġreflects Ġthe Ġdifference Ġbetween ĠL RC Ġand Ġconventional ĠRC Ġmodels . ĠWe Ġre - form al ize ĠL RC Ġin Ġlegal Ġarea Ġto Ġincorporate Ġlaw Ġarticles Ġvia Ġthe Ġreading Ġmechanism , Ġwhich Ġcan Ġenhance Ġjudgment Ġprediction . ĠMoreover , ĠCNN / GR U + law Ġdecrease Ġthe Ġperformance Ġby Ġsimply Ġconc aten ating Ġoriginal Ġtext Ġwith Ġlaw Ġarticles , Ġwhile ĠGR U + Att ention / Auto Judge Ġincrease Ġthe Ġperformance Ġby Ġintegrating Ġlaw Ġarticles Ġwith Ġattention Ġmechanism . ĠIt Ġshows Ġthe Ġimportance Ġand Ġrationality Ġof Ġusing Ġattention Ġmechanism Ġto Ġcapture Ġthe Ġinteraction Ġbetween Ġmultiple Ġinputs . </s> The Ġexperiments Ġsupport Ġour Ġhypothesis Ġas Ġproposed Ġin Ġthe ĠIntroduction Ġpart Ġthat Ġin Ġcivil Ġcases , Ġit 's Ġimportant Ġto Ġmodel Ġthe Ġinteractions Ġamong Ġcase Ġmaterials . ĠReading Ġmechanism Ġcan Ġwell Ġperform Ġthe Ġmatching Ġamong Ġthem . </s> A bl ation ĠTest </s> Auto Judge Ġis Ġcharacterized Ġby Ġthe Ġincorporation Ġof Ġpair - wise Ġattentive Ġreader , Ġlaw Ġarticles , Ġand Ġa ĠCNN Ġoutput Ġlayer , Ġas Ġwell Ġas Ġsome Ġpre - processing Ġwith Ġlegal Ġprior . ĠWe Ġdesign Ġab lation Ġtests Ġrespectively Ġto Ġevaluate Ġthe Ġeffectiveness Ġof Ġthese Ġmodules . ĠWhen Ġtaken Ġoff Ġthe Ġattention Ġmechanism , ĠAuto Judge Ġde grades Ġinto Ġa ĠGR U Ġon Ġwhich Ġa ĠCNN Ġis Ġstacked . ĠWhen Ġtaken Ġoff Ġlaw Ġarticles , Ġthe ĠCNN Ġoutput Ġlayer Ġonly Ġtakes ĠIN LINE FORM 0 Ġas Ġinput . ĠBesides , Ġour Ġmodel Ġis Ġtested Ġrespectively Ġwithout Ġname - repl acement Ġor Ġun super vised Ġselection Ġof Ġlaw Ġarticles Ġ( i . e . Ġpassing Ġthe Ġwhole Ġlaw Ġtext ). ĠAs Ġmentioned Ġabove , Ġwe Ġsystem Ġuse Ġlaw Ġarticles Ġextracted Ġwith Ġun super vised Ġmethod , Ġso Ġwe Ġalso Ġexperiment Ġwith Ġground - truth Ġlaw Ġarticles . </s> Results Ġare Ġshown Ġin ĠTable ĠT AB REF 38 Ġ. ĠWe Ġcan Ġinfer Ġthat : </s> ( 1 ) ĠThe Ġperformance Ġdrops Ġsignificantly Ġafter Ġremoving Ġthe Ġattention Ġlayer Ġor Ġexcluding Ġthe Ġlaw Ġarticles , Ġwhich Ġis Ġconsistent Ġwith Ġthe Ġcomparison Ġbetween ĠAuto Judge Ġand Ġbas elines . ĠThe Ġresult Ġver ifies Ġthat Ġboth Ġthe Ġreading Ġmechanism Ġand Ġincorporation Ġof Ġlaw Ġarticles Ġare Ġimportant Ġand Ġeffective . </s> ( 2 ) ĠAfter Ġreplacing ĠCNN Ġwith Ġan ĠL ST M Ġlayer , Ġperformance Ġdrops Ġas Ġmuch Ġas ĠIN LINE FORM 0 Ġin Ġaccuracy Ġand ĠIN LINE FORM 1 Ġin ĠF 1 Ġscore . ĠThe Ġreason Ġmay Ġbe Ġthe Ġredundancy Ġof ĠR NN s . ĠAuto Judge Ġhas Ġemployed Ġseveral ĠGR U Ġlayers Ġto Ġencode Ġtext Ġsequences . ĠAnother ĠR NN Ġlayer Ġmay Ġbe Ġuseless Ġto Ġcapture Ġsequential Ġdependencies , Ġwhile ĠCNN Ġcan Ġcatch Ġthe Ġlocal Ġstructure Ġin Ġconv olution Ġwindows . </s> ( 3 ) ĠMot ivated Ġby Ġexisting Ġrule - based Ġworks , Ġwe Ġconduct Ġdata Ġpre - processing Ġon Ġcases , Ġincluding Ġname Ġreplacement Ġand Ġlaw Ġarticle Ġfil t ration . ĠIf Ġwe Ġremove Ġthe Ġpre - processing Ġoperations , Ġthe Ġperformance Ġdrops Ġconsiderably . ĠIt Ġdemonstrates Ġthat Ġapplying Ġthe Ġprior Ġknowledge Ġin Ġlegal Ġfiled Ġwould Ġbenefit Ġthe Ġunderstanding Ġof Ġlegal Ġcases . </s> It 's Ġintuitive Ġthat Ġthe Ġquality Ġof Ġthe Ġretrieved Ġlaw Ġarticles Ġwould Ġaffect Ġthe Ġfinal Ġperformance . ĠAs Ġis Ġshown Ġin ĠTable ĠT AB REF 38 Ġ, Ġfeeding Ġthe Ġwhole Ġlaw Ġtext Ġwithout Ġfil t ration Ġresults Ġin Ġworse Ġperformance . ĠHowever , Ġwhen Ġwe Ġtrain Ġand Ġevaluate Ġour Ġmodel Ġwith Ġground Ġtruth Ġarticles , Ġthe Ġperformance Ġis Ġboosted Ġby Ġnearly ĠIN LINE FORM 0 Ġin Ġboth ĠF 1 Ġand ĠAcc . ĠThe Ġperformance Ġimprovement Ġis Ġquite Ġlimited Ġcompared Ġto Ġthat Ġin Ġprevious Ġwork ĠB IB REF 3 Ġfor Ġthe Ġfollowing Ġreasons : Ġ( 1 ) ĠAs Ġmentioned Ġabove , Ġmost Ġcase Ġdocuments Ġonly Ġcontain Ġcoarse - gr ained Ġarticles , Ġand Ġonly Ġa Ġsmall Ġnumber Ġof Ġthem Ġcontain Ġfine - gr ained Ġones , Ġwhich Ġhas Ġlimited Ġinformation Ġin Ġthemselves . Ġ( 2 ) ĠUnlike Ġin Ġcriminal Ġcases Ġwhere Ġthe Ġapplication Ġof Ġan Ġarticle Ġindicates Ġthe Ġcorresponding Ġcrime , Ġlaw Ġarticles Ġin Ġcivil Ġcases Ġwork Ġas Ġreference , Ġand Ġcan Ġbe Ġapplied Ġin Ġboth Ġthe Ġcases Ġof Ġsupports Ġand Ġrejects . ĠAs Ġlaw Ġarticles Ġcut Ġboth Ġways Ġfor Ġthe Ġjudgment Ġresult , Ġthis Ġis Ġone Ġof Ġthe Ġcharacteristics Ġthat Ġdistinguishes Ġcivil Ġcases Ġfrom Ġcriminal Ġones . ĠWe Ġalso Ġneed Ġto Ġremember Ġthat , Ġthe Ġperformance Ġof ĠIN LINE FORM 1 Ġin Ġaccuracy Ġor ĠIN LINE FORM 2 Ġin ĠF 1 Ġscore Ġis Ġunatt ain able Ġin Ġreal - world Ġsetting Ġfor Ġautomatic Ġprediction Ġwhere Ġground - truth Ġarticles Ġare Ġnot Ġavailable . </s> In Ġthe Ġarea Ġof Ġcivil Ġcases , Ġthe Ġunderstanding Ġof Ġthe Ġcase Ġmaterials Ġand Ġhow Ġthey Ġinteract Ġis Ġa Ġcritical Ġfactor . ĠThe Ġinclusion Ġof Ġlaw Ġarticles Ġis Ġnot Ġenough . ĠAs Ġis Ġshown Ġin ĠTable ĠT AB REF 38 Ġ, Ġcompared Ġto Ġfeeding Ġthe Ġmodel Ġwith Ġan Ġun - selected Ġset Ġof Ġlaw Ġarticles , Ġtaking Ġaway Ġthe Ġreading Ġmechanism Ġresults Ġin Ġgreater Ġperformance Ġdrop . ĠTherefore , Ġthe Ġability Ġto Ġread , Ġunderstand Ġand Ġselect Ġrelevant Ġinformation Ġfrom Ġthe Ġcomplex Ġmulti - s ourced Ġcase Ġmaterials Ġis Ġnecessary . ĠIt 's Ġeven Ġmore Ġimportant Ġin Ġreal Ġworld Ġsince Ġwe Ġdon 't Ġhave Ġaccess Ġto Ġground - truth Ġlaw Ġarticles Ġto Ġmake Ġpredictions . </s> Case ĠStudy </s> We Ġvisualize Ġthe Ġheat Ġmaps Ġof Ġattention Ġresults . ĠAs Ġshown Ġin ĠFig . ĠFIG REF 47 Ġ, Ġdeeper Ġbackground Ġcolor Ġrepresents Ġlarger Ġattention Ġscore . ĠThe Ġattention Ġscore Ġis Ġcalculated Ġwith ĠE q . Ġ( ĠEQ REF 15 Ġ). ĠWe Ġtake Ġthe Ġaverage Ġof Ġthe Ġresulting ĠIN LINE FORM 0 Ġattention Ġmatrix Ġover Ġthe Ġtime Ġdimension Ġto Ġobtain Ġattention Ġvalues Ġfor Ġeach Ġword . </s> The Ġvisualization Ġdemonstrates Ġthat Ġthe Ġattention Ġmechanism Ġcan Ġcapture Ġrelevant Ġpatterns Ġand Ġsemantics Ġin Ġaccordance Ġwith Ġdifferent Ġpleas Ġin Ġdifferent Ġcases . </s> As Ġfor Ġthe Ġfailed Ġsamples , Ġthe Ġmost Ġcommon Ġreason Ġcomes Ġfrom Ġthe Ġanonymity Ġissue , Ġwhich Ġis Ġalso Ġshown Ġin ĠFig . ĠFIG REF 47 Ġ. ĠAs Ġmentioned Ġabove , Ġwe Ġconduct Ġname Ġreplacement . ĠHowever , Ġsome Ġcritical Ġelements Ġare Ġalso Ġanonym ized Ġby Ġthe Ġgovernment , Ġdue Ġto Ġthe Ġprivacy Ġissue . ĠThese Ġelements Ġare Ġsometimes Ġimportant Ġto Ġjudgment Ġprediction . ĠFor Ġexample , Ġdetermination Ġof Ġthe Ġkey Ġfactor Ġlong - time Ġseparation Ġis Ġrelevant Ġto Ġthe Ġexplicit Ġdates , Ġwhich Ġare Ġanonym ized . </s> Conclusion </s> In Ġthis Ġpaper , Ġwe Ġexplore Ġthe Ġtask Ġof Ġpredicting Ġjudgments Ġof Ġcivil Ġcases . ĠComp aring Ġwith Ġconventional Ġtext Ġclassification Ġframework , Ġwe Ġpropose ĠLegal ĠReading ĠComp rehens ion Ġframework Ġto Ġhandle Ġmultiple Ġand Ġcomplex Ġtextual Ġinputs . ĠMoreover , Ġwe Ġpresent Ġa Ġnovel Ġneural Ġmodel , ĠAuto Judge , Ġto Ġincorporate Ġlaw Ġarticles Ġfor Ġjudgment Ġprediction . ĠIn Ġexperiments , Ġwe Ġcompare Ġour Ġmodel Ġon Ġdivorce Ġproceedings Ġwith Ġvarious Ġstate - of - the - art Ġbas elines Ġof Ġvarious Ġframeworks . ĠExperimental Ġresults Ġshow Ġthat Ġour Ġmodel Ġachieves Ġconsiderable Ġimprovement Ġthan Ġall Ġthe Ġbas elines . ĠBesides , Ġvisualization Ġresults Ġalso Ġdemonstrate Ġthe Ġeffectiveness Ġand Ġinterpret ability Ġof Ġour Ġproposed Ġmodel . </s> In Ġthe Ġfuture , Ġwe Ġcan Ġexplore Ġthe Ġfollowing Ġdirections : Ġ( 1 ) ĠLimited Ġby Ġthe Ġdatasets , Ġwe Ġcan Ġonly Ġverify Ġour Ġproposed Ġmodel Ġon Ġdivorce Ġproceedings . ĠA Ġmore Ġgeneral Ġand Ġlarger Ġdataset Ġwill Ġbenefit Ġthe Ġresearch Ġon Ġjudgment Ġprediction . Ġ( 2 ) ĠJudicial Ġdecisions Ġin Ġsome Ġcivil Ġcases Ġare Ġnot Ġalways Ġbinary , Ġbut Ġmore Ġdiverse Ġand Ġflexible Ġones , Ġe . g . Ġcompensation Ġamount . ĠThus , Ġit Ġis Ġcritical Ġfor Ġjudgment Ġprediction Ġto Ġmanage Ġvarious Ġjudgment Ġforms .
Introduction Automatic judgment prediction is to train a machine judge to determine whether a certain plea in a given civil case would be supported or rejected. In countries with civil law system, e.g. mainland China, such process should be done with reference to related law articles and the fact description, as is performed by a human judge. The intuition comes from the fact that under civil law system, law articles act as principles for juridical judgments. Such techniques would have a wide range of promising applications. On the one hand, legal consulting systems could provide better access to high-quality legal resources in a low-cost way to legal outsiders, who suffer from the complicated terminologies. On the other hand, machine judge assistants for professionals would help improve the efficiency of the judicial system. Besides, automated judgment system can help in improving juridical equality and transparency. From another perspective, there are currently 7 times much more civil cases than criminal cases in mainland China, with annual rates of increase of INLINEFORM0 and INLINEFORM1 respectively, making judgment prediction in civil cases a promising application BIBREF0 . Previous works BIBREF1 , BIBREF2 , BIBREF3 , BIBREF4 formalize judgment prediction as the text classification task, regarding either charge names or binary judgments, i.e., support or reject, as the target classes. These works focus on the situation where only one result is expected, e.g., the US Supreme Court's decisions BIBREF2 , and the charge name prediction for criminal cases BIBREF3 . Despite these recent efforts and their progress, automatic judgment prediction in civil law system is still confronted with two main challenges: One-to-Many Relation between Case and Plea. Every single civil case may contain multiple pleas and the result of each plea is co-determined by related law articles and specific aspects of the involved case. For example, in divorce proceedings, judgment of alienation of mutual affection is the key factor for granting divorce but custody of children depends on which side can provide better an environment for children's growth as well as parents' financial condition. Here, different pleas are independent. Heterogeneity of Input Triple. Inputs to a judgment prediction system consist of three heterogeneous yet complementary parts, i.e., fact description, plaintiff's plea, and related law articles. Concatenating them together and treating them simply as a sequence of words as in previous works BIBREF2 , BIBREF1 would cause a great loss of information. This is the same in question-answering where the dual inputs, i.e., query and passage, should be modeled separately. Despite the introduction of the neural networks that can learn better semantic representations of input text, it remains unsolved to incorporate proper mechanisms to integrate the complementary triple of pleas, fact descriptions, and law articles together. Inspired by recent advances in question answering (QA) based reading comprehension (RC) BIBREF5 , BIBREF6 , BIBREF7 , BIBREF8 , we propose the Legal Reading Comprehension (LRC) framework for automatic judgment prediction. LRC incorporates the reading mechanism for better modeling of the complementary inputs above-mentioned, as is done by human judges when referring to legal materials in search of supporting law articles. Reading mechanism, by simulating how human connects and integrates multiple text, has proven an effective module in RC tasks. We argue that applying the reading mechanism in a proper way among the triplets can obtain a better understanding and more informative representation of the original text, and further improve performance . To instantiate the framework, we propose an end-to-end neural network model named AutoJudge. For experiments, we train and evaluate our models in the civil law system of mainland China. We collect and construct a large-scale real-world data set of INLINEFORM0 case documents that the Supreme People's Court of People's Republic of China has made publicly available. Fact description, pleas, and results can be extracted easily from these case documents with regular expressions, since the original documents have special typographical characteristics indicating the discourse structure. We also take into account law articles and their corresponding juridical interpretations. We also implement and evaluate previous methods on our dataset, which prove to be strong baselines. Our experiment results show significant improvements over previous methods. Further experiments demonstrate that our model also achieves considerable improvement over other off-the-shelf state-of-the-art models under classification and question answering framework respectively. Ablation tests carried out by taking off some components of our model further prove its robustness and effectiveness. To sum up, our contributions are as follows: (1) We introduce reading mechanism and re-formalize judgment prediction as Legal Reading Comprehension to better model the complementary inputs. (2) We construct a real-world dataset for experiments, and plan to publish it for further research. (3) Besides baselines from previous works, we also carry out comprehensive experiments comparing different existing deep neural network methods on our dataset. Supported by these experiments, improvements achieved by LRC prove to be robust. Judgment Prediction Automatic judgment prediction has been studied for decades. At the very first stage of judgment prediction studies, researchers focus on mathematical and statistical analysis of existing cases, without any conclusions or methodologies on how to predict them BIBREF9 , BIBREF10 , BIBREF11 , BIBREF12 , BIBREF13 , BIBREF14 . Recent attempts consider judgment prediction under the text classification framework. Most of these works extract efficient features from text (e.g., N-grams) BIBREF15 , BIBREF4 , BIBREF1 , BIBREF16 , BIBREF17 or case profiles (e.g., dates, terms, locations and types) BIBREF2 . All these methods require a large amount of human effort to design features or annotate cases. Besides, they also suffer from generalization issue when applied to other scenarios. Motivated by the successful application of deep neural networks, Luo et al. BIBREF3 introduce an attention-based neural model to predict charges of criminal cases, and verify the effectiveness of taking law articles into consideration. Nevertheless, they still fall into the text classification framework and lack the ability to handle multiple inputs with more complicated structures. Text Classification As the basis of previous judgment prediction works, typical text classification task takes a single text content as input and predicts the category it belongs to. Recent works usually employ neural networks to model the internal structure of a single input BIBREF18 , BIBREF19 , BIBREF20 , BIBREF21 . There also exists another thread of text classification called entailment prediction. Methods proposed in BIBREF22 , BIBREF23 are intended for complementary inputs, but the mechanisms can be considered as a simplified version of reading comprehension. Reading Comprehension Reading comprehension is a relevant task to model heterogeneous and complementary inputs, where an answer is predicted given two channels of inputs, i.e. a textual passage and a query. Considerable progress has been made BIBREF6 , BIBREF24 , BIBREF5 . These models employ various attention mechanism to model the interaction between passage and query. Inspired by the advantage of reading comprehension models on modeling multiple inputs, we apply this idea into the legal area and propose legal reading comprehension for judgment prediction. Conventional Reading Comprehension Conventional reading comprehension BIBREF25 , BIBREF26 , BIBREF7 , BIBREF8 usually considers reading comprehension as predicting the answer given a passage and a query, where the answer could be a single word, a text span of the original passage, chosen from answer candidates, or generated by human annotators. Generally, an instance in RC is represented as a triple INLINEFORM0 , where INLINEFORM1 , INLINEFORM2 and INLINEFORM3 correspond to INLINEFORM4 , INLINEFORM5 and INLINEFORM6 respectively. Given a triple INLINEFORM7 , RC takes the pair INLINEFORM8 as the input and employs attention-based neural models to construct an efficient representation. Afterwards, the representation is fed into the output layer to select or generate an INLINEFORM9 . Legal Reading Comprehension Existing works usually formalize judgment prediction as a text classification task and focus on extracting well-designed features of specific cases. Such simplification ignores that the judgment of a case is determined by its fact description and multiple pleas. Moreover, the final judgment should act up to the legal provisions, especially in civil law systems. Therefore, how to integrate the information (i.e., fact descriptions, pleas, and law articles) in a reasonable way is critical for judgment prediction. Inspired by the successful application of RC, we propose a framework of Legal Reading Comprehension(LRC) for judgment prediction in the legal area. As illustrated in Fig. FIGREF1 , for each plea in a given case, the prediction of judgment result is made based the fact description and the potentially relevant law articles. In a nutshell, LRC can be formalized as the following quadruplet task: DISPLAYFORM0 where INLINEFORM0 is the fact description, INLINEFORM1 is the plea, INLINEFORM2 is the law articles and INLINEFORM3 is the result. Given INLINEFORM4 , LRC aims to predict the judgment result as DISPLAYFORM0 The probability is calculated with respect to the interaction among the triple INLINEFORM0 , which will draw on the experience of the interaction between INLINEFORM1 pairs in RC. To summarize, LRC is innovative in the following aspects: (1) While previous works fit the problem into text classification framework, LRC re-formalizes the way to approach such problems. This new framework provides the ability to deal with the heterogeneity of the complementary inputs. (2) Rather than employing conventional RC models to handle pair-wise text information in the legal area, LRC takes the critical law articles into consideration and models the facts, pleas, and law articles jointly for judgment prediction, which is more suitable to simulate the human mode of dealing with cases. Methods We propose a novel judgment prediction model AutoJudge to instantiate the LRC framework. As shown in Fig. FIGREF6 , AutoJudge consists of three flexible modules, including a text encoder, a pair-wise attentive reader, and an output module. In the following parts, we give a detailed introduction to these three modules. Text Encoder As illustrated in Fig. FIGREF6 , Text Encoder aims to encode the word sequences of inputs into continuous representation sequences. Formally, consider a fact description INLINEFORM0 , a plea INLINEFORM1 , and the relevant law articles INLINEFORM2 , where INLINEFORM3 denotes the INLINEFORM4 -th word in the sequence and INLINEFORM5 are the lengths of word sequences INLINEFORM6 respectively. First, we convert the words to their respective word embeddings to obtain INLINEFORM7 , INLINEFORM8 and INLINEFORM9 , where INLINEFORM10 . Afterwards, we employ bi-directional GRU BIBREF27 , BIBREF28 , BIBREF29 to produce the encoded representation INLINEFORM11 of all words as follows: DISPLAYFORM0 Note that, we adopt different bi-directional GRUs to encode fact descriptions, pleas, and law articles respectively(denoted as INLINEFORM0 , INLINEFORM1 , and INLINEFORM2 ). With these text encoders, INLINEFORM3 , INLINEFORM4 , and INLINEFORM5 are converting into INLINEFORM6 , INLINEFORM7 , and INLINEFORM8 . Pair-Wise Attentive Reader How to model the interactions among the input text is the most important problem in reading comprehension. In AutoJudge, we employ a pair-wise attentive reader to process INLINEFORM0 and INLINEFORM1 respectively. More specifically, we propose to use pair-wise mutual attention mechanism to capture the complex semantic interaction between text pairs, as well as increasing the interpretability of AutoJudge. For each input pair INLINEFORM0 or INLINEFORM1 , we employ pair-wise mutual attention to select relevant information from fact descriptions INLINEFORM2 and produce more informative representation sequences. As a variant of the original attention mechanism BIBREF28 , we design the pair-wise mutual attention unit as a GRU with internal memories denoted as mGRU. Taking the representation sequence pair INLINEFORM0 for instance, mGRU stores the fact sequence INLINEFORM1 into its memories. For each timestamp INLINEFORM2 , it selects relevant fact information INLINEFORM3 from the memories as follows, DISPLAYFORM0 Here, the weight INLINEFORM0 is the softmax value as DISPLAYFORM0 Note that, INLINEFORM0 represents the relevance between INLINEFORM1 and INLINEFORM2 . It is calculated as follows, DISPLAYFORM0 Here, INLINEFORM0 is the last hidden state in the GRU, which will be introduced in the following part. INLINEFORM1 is a weight vector, and INLINEFORM2 , INLINEFORM3 , INLINEFORM4 are attention metrics of our proposed pair-wise attention mechanism. With the relevant fact information INLINEFORM0 and INLINEFORM1 , we get the INLINEFORM2 -th input of mGRU as DISPLAYFORM0 where INLINEFORM0 indicates the concatenation operation. Then, we feed INLINEFORM0 into GRU to get more informative representation sequence INLINEFORM1 as follows, DISPLAYFORM0 For the input pair INLINEFORM0 , we can get INLINEFORM1 in the same way. Therefore, we omit the implementation details Here. Similar structures with attention mechanism are also applied in BIBREF5 , BIBREF30 , BIBREF31 , BIBREF28 to obtain mutually aware representations in reading comprehension models, which significantly improve the performance of this task. Output Layer Using text encoder and pair-wise attentive reader, the initial input triple INLINEFORM0 has been converted into two sequences, i.e., INLINEFORM1 and INLINEFORM2 , where INLINEFORM3 is defined similarly to INLINEFORM4 . These sequences reserve complex semantic information about the pleas and law articles, and filter out irrelevant information in fact descriptions. With these two sequences, we concatenate INLINEFORM0 and INLINEFORM1 along the sequence length dimension to generate the sequence INLINEFORM2 . Since we have employed several GRU layers to encode the sequential inputs, another recurrent layer may be redundant. Therefore, we utilize a 1-layer CNN BIBREF18 to capture the local structure and generate the representation vector for the final prediction. Assuming INLINEFORM0 is the predicted probability that the plea in the case sample would be supported and INLINEFORM1 is the gold standard, AutoJudge aims to minimize the cross-entropy as follows, DISPLAYFORM0 where INLINEFORM0 is the number of training data. As all the calculation in our model is differentiable, we employ Adam BIBREF32 for optimization. Experiments To evaluate the proposed LRC framework and the AutoJudge model, we carry out a series of experiments on the divorce proceedings, a typical yet complex field of civil cases. Divorce proceedings often come with several kinds of pleas, e.g. seeking divorce, custody of children, compensation, and maintenance, which focuses on different aspects and thus makes it a challenge for judgment prediction. Dataset Construction for Evaluation Since none of the datasets from previous works have been published, we decide to build a new one. We randomly collect INLINEFORM0 cases from China Judgments Online, among which INLINEFORM1 cases are for training, INLINEFORM2 each for validation and testing. Among the original cases, INLINEFORM3 are granted divorce and others not. There are INLINEFORM4 valid pleas in total, with INLINEFORM5 supported and INLINEFORM6 rejected. Note that, if the divorce plea in a case is not granted, the other pleas of this case will not be considered by the judge. Case materials are all natural language sentences, with averagely INLINEFORM7 tokens per fact description and INLINEFORM8 per plea. There are 62 relevant law articles in total, each with INLINEFORM9 tokens averagely. Note that the case documents include special typographical signals, making it easy to extract labeled data with regular expression. We apply some rules with legal prior to preprocess the dataset according to previous works BIBREF33 , BIBREF34 , BIBREF35 , which have proved effective in our experiments. Name Replacement: All names in case documents are replaced with marks indicating their roles, instead of simply anonymizing them, e.g. <Plantiff>, <Defendant>, <Daughter_x> and so on. Since “all are equal before the law”, names should make no more difference than what role they take. Law Article Filtration : Since most accessible divorce proceeding documents do not contain ground-truth fine-grained articles, we use an unsupervised method instead. First, we extract all the articles from the law text with regular expression. Afterwards, we select the most relevant 10 articles according to the fact descriptions as follows. We obtain sentence representation with CBOW BIBREF36 , BIBREF37 weighted by inverse document frequency, and calculate cosine distance between cases and law articles. Word embeddings are pre-trained with Chinese Wikipedia pages. As the final step, we extract top 5 relevant articles for each sample respectively from the main marriage law articles and their interpretations, which are equally important. We manually check the extracted articles for 100 cases to ensure that the extraction quality is fairly good and acceptable. The filtration process is automatic and fully unsupervised since the original documents have no ground-truth labels for fine-grained law articles, and coarse-grained law-articles only provide limited information. We also experiment with the ground-truth articles, but only a small fraction of them has fine-grained ones, and they are usually not available in real-world scenarios. Implementation Details We employ Jieba for Chinese word segmentation and keep the top INLINEFORM0 frequent words. The word embedding size is set to 128 and the other low-frequency words are replaced with the mark <UNK>. The hidden size of GRU is set to 128 for each direction in Bi-GRU. In the pair-wise attentive reader, the hidden state is set to 256 for mGRu. In the CNN layer, filter windows are set to 1, 3, 4, and 5 with each filter containing 200 feature maps. We add a dropout layer BIBREF38 after the CNN layer with a dropout rate of INLINEFORM1 . We use Adam BIBREF32 for training and set learning rate to INLINEFORM2 , INLINEFORM3 to INLINEFORM4 , INLINEFORM5 to INLINEFORM6 , INLINEFORM7 to INLINEFORM8 , batch size to 64. We employ precision, recall, F1 and accuracy for evaluation metrics. We repeat all the experiments for 10 times, and report the average results. Baselines For comparison, we adopt and re-implement three kinds of baselines as follows: We implement an SVM with lexical features in accordance with previous works BIBREF16 , BIBREF17 , BIBREF1 , BIBREF15 , BIBREF4 and select the best feature set on the development set. We implement and fine-tune a series of neural text classifiers, including attention-based method BIBREF3 and other methods we deem important. CNN BIBREF18 and GRU BIBREF27 , BIBREF21 take as input the concatenation of fact description and plea. Similarly, CNN/GRU+law refers to using the concatenation of fact description, plea and law articles as inputs. We implement and train some off-the-shelf RC models, including r-net BIBREF5 and AoA BIBREF6 , which are the leading models on SQuAD leaderboard. In our initial experiments, these models take fact description as passage and plea as query. Further, Law articles are added to the fact description as a part of the reading materials, which is a simple way to consider them as well. Results and Analysis From Table TABREF37 , we have the following observations: (1) AutoJudge consistently and significantly outperforms all the baselines, including RC models and other neural text classification models, which shows the effectiveness and robustness of our model. (2) RC models achieve better performance than most text classification models (excluding GRU+Attention), which indicates that reading mechanism is a better way to integrate information from heterogeneous yet complementary inputs. On the contrary, simply adding law articles as a part of the reading materials makes no difference in performance. Note that, GRU+Attention employ similar attention mechanism as RC does and takes additional law articles into consideration, thus achieves comparable performance with RC models. (3) Comparing with conventional RC models, AutoJudge achieves significant improvement with the consideration of additional law articles. It reflects the difference between LRC and conventional RC models. We re-formalize LRC in legal area to incorporate law articles via the reading mechanism, which can enhance judgment prediction. Moreover, CNN/GRU+law decrease the performance by simply concatenating original text with law articles, while GRU+Attention/AutoJudge increase the performance by integrating law articles with attention mechanism. It shows the importance and rationality of using attention mechanism to capture the interaction between multiple inputs. The experiments support our hypothesis as proposed in the Introduction part that in civil cases, it's important to model the interactions among case materials. Reading mechanism can well perform the matching among them. Ablation Test AutoJudge is characterized by the incorporation of pair-wise attentive reader, law articles, and a CNN output layer, as well as some pre-processing with legal prior. We design ablation tests respectively to evaluate the effectiveness of these modules. When taken off the attention mechanism, AutoJudge degrades into a GRU on which a CNN is stacked. When taken off law articles, the CNN output layer only takes INLINEFORM0 as input. Besides, our model is tested respectively without name-replacement or unsupervised selection of law articles (i.e. passing the whole law text). As mentioned above, we system use law articles extracted with unsupervised method, so we also experiment with ground-truth law articles. Results are shown in Table TABREF38 . We can infer that: (1) The performance drops significantly after removing the attention layer or excluding the law articles, which is consistent with the comparison between AutoJudge and baselines. The result verifies that both the reading mechanism and incorporation of law articles are important and effective. (2) After replacing CNN with an LSTM layer, performance drops as much as INLINEFORM0 in accuracy and INLINEFORM1 in F1 score. The reason may be the redundancy of RNNs. AutoJudge has employed several GRU layers to encode text sequences. Another RNN layer may be useless to capture sequential dependencies, while CNN can catch the local structure in convolution windows. (3) Motivated by existing rule-based works, we conduct data pre-processing on cases, including name replacement and law article filtration. If we remove the pre-processing operations, the performance drops considerably. It demonstrates that applying the prior knowledge in legal filed would benefit the understanding of legal cases. It's intuitive that the quality of the retrieved law articles would affect the final performance. As is shown in Table TABREF38 , feeding the whole law text without filtration results in worse performance. However, when we train and evaluate our model with ground truth articles, the performance is boosted by nearly INLINEFORM0 in both F1 and Acc. The performance improvement is quite limited compared to that in previous work BIBREF3 for the following reasons: (1) As mentioned above, most case documents only contain coarse-grained articles, and only a small number of them contain fine-grained ones, which has limited information in themselves. (2) Unlike in criminal cases where the application of an article indicates the corresponding crime, law articles in civil cases work as reference, and can be applied in both the cases of supports and rejects. As law articles cut both ways for the judgment result, this is one of the characteristics that distinguishes civil cases from criminal ones. We also need to remember that, the performance of INLINEFORM1 in accuracy or INLINEFORM2 in F1 score is unattainable in real-world setting for automatic prediction where ground-truth articles are not available. In the area of civil cases, the understanding of the case materials and how they interact is a critical factor. The inclusion of law articles is not enough. As is shown in Table TABREF38 , compared to feeding the model with an un-selected set of law articles, taking away the reading mechanism results in greater performance drop. Therefore, the ability to read, understand and select relevant information from the complex multi-sourced case materials is necessary. It's even more important in real world since we don't have access to ground-truth law articles to make predictions. Case Study We visualize the heat maps of attention results. As shown in Fig. FIGREF47 , deeper background color represents larger attention score. The attention score is calculated with Eq. ( EQREF15 ). We take the average of the resulting INLINEFORM0 attention matrix over the time dimension to obtain attention values for each word. The visualization demonstrates that the attention mechanism can capture relevant patterns and semantics in accordance with different pleas in different cases. As for the failed samples, the most common reason comes from the anonymity issue, which is also shown in Fig. FIGREF47 . As mentioned above, we conduct name replacement. However, some critical elements are also anonymized by the government, due to the privacy issue. These elements are sometimes important to judgment prediction. For example, determination of the key factor long-time separation is relevant to the explicit dates, which are anonymized. Conclusion In this paper, we explore the task of predicting judgments of civil cases. Comparing with conventional text classification framework, we propose Legal Reading Comprehension framework to handle multiple and complex textual inputs. Moreover, we present a novel neural model, AutoJudge, to incorporate law articles for judgment prediction. In experiments, we compare our model on divorce proceedings with various state-of-the-art baselines of various frameworks. Experimental results show that our model achieves considerable improvement than all the baselines. Besides, visualization results also demonstrate the effectiveness and interpretability of our proposed model. In the future, we can explore the following directions: (1) Limited by the datasets, we can only verify our proposed model on divorce proceedings. A more general and larger dataset will benefit the research on judgment prediction. (2) Judicial decisions in some civil cases are not always binary, but more diverse and flexible ones, e.g. compensation amount. Thus, it is critical for judgment prediction to manage various judgment forms.
what evaluation metrics are reported?
d9412dda3279729e95fcb35cbed09e61577a896e
what Ġevaluation Ġmetrics Ġare Ġreported ?
pre cision , Ġrecall , ĠF 1 Ġand Ġaccuracy
<s> what Ġevaluation Ġmetrics Ġare Ġreported ? </s> Introduction </s> Aut omatic Ġjudgment Ġprediction Ġis Ġto Ġtrain Ġa Ġmachine Ġjudge Ġto Ġdetermine Ġwhether Ġa Ġcertain Ġplea Ġin Ġa Ġgiven Ġcivil Ġcase Ġwould Ġbe Ġsupported Ġor Ġrejected . ĠIn Ġcountries Ġwith Ġcivil Ġlaw Ġsystem , Ġe . g . Ġmainland ĠChina , Ġsuch Ġprocess Ġshould Ġbe Ġdone Ġwith Ġreference Ġto Ġrelated Ġlaw Ġarticles Ġand Ġthe Ġfact Ġdescription , Ġas Ġis Ġperformed Ġby Ġa Ġhuman Ġjudge . ĠThe Ġintuition Ġcomes Ġfrom Ġthe Ġfact Ġthat Ġunder Ġcivil Ġlaw Ġsystem , Ġlaw Ġarticles Ġact Ġas Ġprinciples Ġfor Ġjur id ical Ġjudgments . ĠSuch Ġtechniques Ġwould Ġhave Ġa Ġwide Ġrange Ġof Ġpromising Ġapplications . ĠOn Ġthe Ġone Ġhand , Ġlegal Ġconsulting Ġsystems Ġcould Ġprovide Ġbetter Ġaccess Ġto Ġhigh - quality Ġlegal Ġresources Ġin Ġa Ġlow - cost Ġway Ġto Ġlegal Ġoutsiders , Ġwho Ġsuffer Ġfrom Ġthe Ġcomplicated Ġtermin ologies . ĠOn Ġthe Ġother Ġhand , Ġmachine Ġjudge Ġassistants Ġfor Ġprofessionals Ġwould Ġhelp Ġimprove Ġthe Ġefficiency Ġof Ġthe Ġjudicial Ġsystem . ĠBesides , Ġautomated Ġjudgment Ġsystem Ġcan Ġhelp Ġin Ġimproving Ġjur id ical Ġequality Ġand Ġtransparency . ĠFrom Ġanother Ġperspective , Ġthere Ġare Ġcurrently Ġ7 Ġtimes Ġmuch Ġmore Ġcivil Ġcases Ġthan Ġcriminal Ġcases Ġin Ġmainland ĠChina , Ġwith Ġannual Ġrates Ġof Ġincrease Ġof ĠIN LINE FORM 0 Ġand ĠIN LINE FORM 1 Ġrespectively , Ġmaking Ġjudgment Ġprediction Ġin Ġcivil Ġcases Ġa Ġpromising Ġapplication ĠB IB REF 0 Ġ. </s> Previous Ġworks ĠB IB REF 1 Ġ, ĠB IB REF 2 Ġ, ĠB IB REF 3 Ġ, ĠB IB REF 4 Ġformal ize Ġjudgment Ġprediction Ġas Ġthe Ġtext Ġclassification Ġtask , Ġregarding Ġeither Ġcharge Ġnames Ġor Ġbinary Ġjudgments , Ġi . e ., Ġsupport Ġor Ġreject , Ġas Ġthe Ġtarget Ġclasses . ĠThese Ġworks Ġfocus Ġon Ġthe Ġsituation Ġwhere Ġonly Ġone Ġresult Ġis Ġexpected , Ġe . g ., Ġthe ĠUS ĠSupreme ĠCourt 's Ġdecisions ĠB IB REF 2 Ġ, Ġand Ġthe Ġcharge Ġname Ġprediction Ġfor Ġcriminal Ġcases ĠB IB REF 3 Ġ. ĠDespite Ġthese Ġrecent Ġefforts Ġand Ġtheir Ġprogress , Ġautomatic Ġjudgment Ġprediction Ġin Ġcivil Ġlaw Ġsystem Ġis Ġstill Ġconfronted Ġwith Ġtwo Ġmain Ġchallenges : </s> One - to - Many ĠRel ation Ġbetween ĠCase Ġand ĠPle a . ĠEvery Ġsingle Ġcivil Ġcase Ġmay Ġcontain Ġmultiple Ġpleas Ġand Ġthe Ġresult Ġof Ġeach Ġplea Ġis Ġco - d etermined Ġby Ġrelated Ġlaw Ġarticles Ġand Ġspecific Ġaspects Ġof Ġthe Ġinvolved Ġcase . ĠFor Ġexample , Ġin Ġdivorce Ġproceedings , Ġjudgment Ġof Ġalienation Ġof Ġmutual Ġaffection Ġis Ġthe Ġkey Ġfactor Ġfor Ġgranting Ġdivorce Ġbut Ġcustody Ġof Ġchildren Ġdepends Ġon Ġwhich Ġside Ġcan Ġprovide Ġbetter Ġan Ġenvironment Ġfor Ġchildren 's Ġgrowth Ġas Ġwell Ġas Ġparents ' Ġfinancial Ġcondition . ĠHere , Ġdifferent Ġpleas Ġare Ġindependent . </s> H eter ogeneity Ġof ĠInput ĠTriple . ĠInput s Ġto Ġa Ġjudgment Ġprediction Ġsystem Ġconsist Ġof Ġthree Ġheter ogeneous Ġyet Ġcomplementary Ġparts , Ġi . e ., Ġfact Ġdescription , Ġplaintiff 's Ġplea , Ġand Ġrelated Ġlaw Ġarticles . ĠCon cat en ating Ġthem Ġtogether Ġand Ġtreating Ġthem Ġsimply Ġas Ġa Ġsequence Ġof Ġwords Ġas Ġin Ġprevious Ġworks ĠB IB REF 2 Ġ, ĠB IB REF 1 Ġwould Ġcause Ġa Ġgreat Ġloss Ġof Ġinformation . ĠThis Ġis Ġthe Ġsame Ġin Ġquestion - ans w ering Ġwhere Ġthe Ġdual Ġinputs , Ġi . e ., Ġquery Ġand Ġpassage , Ġshould Ġbe Ġmodeled Ġseparately . </s> Despite Ġthe Ġintroduction Ġof Ġthe Ġneural Ġnetworks Ġthat Ġcan Ġlearn Ġbetter Ġsemantic Ġrepresentations Ġof Ġinput Ġtext , Ġit Ġremains Ġunsolved Ġto Ġincorporate Ġproper Ġmechanisms Ġto Ġintegrate Ġthe Ġcomplementary Ġtriple Ġof Ġpleas , Ġfact Ġdescriptions , Ġand Ġlaw Ġarticles Ġtogether . </s> Insp ired Ġby Ġrecent Ġadvances Ġin Ġquestion Ġanswering Ġ( Q A ) Ġbased Ġreading Ġcomprehension Ġ( RC ) ĠB IB REF 5 Ġ, ĠB IB REF 6 Ġ, ĠB IB REF 7 Ġ, ĠB IB REF 8 Ġ, Ġwe Ġpropose Ġthe ĠLegal ĠReading ĠComp rehens ion Ġ( L RC ) Ġframework Ġfor Ġautomatic Ġjudgment Ġprediction . ĠL RC Ġincorporates Ġthe Ġreading Ġmechanism Ġfor Ġbetter Ġmodeling Ġof Ġthe Ġcomplementary Ġinputs Ġabove - mentioned , Ġas Ġis Ġdone Ġby Ġhuman Ġjudges Ġwhen Ġreferring Ġto Ġlegal Ġmaterials Ġin Ġsearch Ġof Ġsupporting Ġlaw Ġarticles . ĠReading Ġmechanism , Ġby Ġsim ulating Ġhow Ġhuman Ġconnects Ġand Ġintegrates Ġmultiple Ġtext , Ġhas Ġproven Ġan Ġeffective Ġmodule Ġin ĠRC Ġtasks . ĠWe Ġargue Ġthat Ġapplying Ġthe Ġreading Ġmechanism Ġin Ġa Ġproper Ġway Ġamong Ġthe Ġtriple ts Ġcan Ġobtain Ġa Ġbetter Ġunderstanding Ġand Ġmore Ġinformative Ġrepresentation Ġof Ġthe Ġoriginal Ġtext , Ġand Ġfurther Ġimprove Ġperformance Ġ. ĠTo Ġinstant iate Ġthe Ġframework , Ġwe Ġpropose Ġan Ġend - to - end Ġneural Ġnetwork Ġmodel Ġnamed ĠAuto Judge . </s> For Ġexperiments , Ġwe Ġtrain Ġand Ġevaluate Ġour Ġmodels Ġin Ġthe Ġcivil Ġlaw Ġsystem Ġof Ġmainland ĠChina . ĠWe Ġcollect Ġand Ġconstruct Ġa Ġlarge - scale Ġreal - world Ġdata Ġset Ġof ĠIN LINE FORM 0 Ġcase Ġdocuments Ġthat Ġthe ĠSupreme ĠPeople 's ĠCourt Ġof ĠPeople 's ĠRepublic Ġof ĠChina Ġhas Ġmade Ġpublicly Ġavailable . ĠFact Ġdescription , Ġpleas , Ġand Ġresults Ġcan Ġbe Ġextracted Ġeasily Ġfrom Ġthese Ġcase Ġdocuments Ġwith Ġregular Ġexpressions , Ġsince Ġthe Ġoriginal Ġdocuments Ġhave Ġspecial Ġtyp ographical Ġcharacteristics Ġindicating Ġthe Ġdiscourse Ġstructure . ĠWe Ġalso Ġtake Ġinto Ġaccount Ġlaw Ġarticles Ġand Ġtheir Ġcorresponding Ġjur id ical Ġinterpretations . ĠWe Ġalso Ġimplement Ġand Ġevaluate Ġprevious Ġmethods Ġon Ġour Ġdataset , Ġwhich Ġprove Ġto Ġbe Ġstrong Ġbas elines . </s> Our Ġexperiment Ġresults Ġshow Ġsignificant Ġimprovements Ġover Ġprevious Ġmethods . ĠFurther Ġexperiments Ġdemonstrate Ġthat Ġour Ġmodel Ġalso Ġachieves Ġconsiderable Ġimprovement Ġover Ġother Ġoff - the - she lf Ġstate - of - the - art Ġmodels Ġunder Ġclassification Ġand Ġquestion Ġanswering Ġframework Ġrespectively . ĠAb lation Ġtests Ġcarried Ġout Ġby Ġtaking Ġoff Ġsome Ġcomponents Ġof Ġour Ġmodel Ġfurther Ġprove Ġits Ġrobust ness Ġand Ġeffectiveness . </s> To Ġsum Ġup , Ġour Ġcontributions Ġare Ġas Ġfollows : </s> ( 1 ) ĠWe Ġintroduce Ġreading Ġmechanism Ġand Ġre - form al ize Ġjudgment Ġprediction Ġas ĠLegal ĠReading ĠComp rehens ion Ġto Ġbetter Ġmodel Ġthe Ġcomplementary Ġinputs . </s> ( 2 ) ĠWe Ġconstruct Ġa Ġreal - world Ġdataset Ġfor Ġexperiments , Ġand Ġplan Ġto Ġpublish Ġit Ġfor Ġfurther Ġresearch . </s> ( 3 ) ĠBesides Ġbas elines Ġfrom Ġprevious Ġworks , Ġwe Ġalso Ġcarry Ġout Ġcomprehensive Ġexperiments Ġcomparing Ġdifferent Ġexisting Ġdeep Ġneural Ġnetwork Ġmethods Ġon Ġour Ġdataset . ĠSupported Ġby Ġthese Ġexperiments , Ġimprovements Ġachieved Ġby ĠL RC Ġprove Ġto Ġbe Ġrobust . </s> Jud gment ĠPrediction </s> Aut omatic Ġjudgment Ġprediction Ġhas Ġbeen Ġstudied Ġfor Ġdecades . ĠAt Ġthe Ġvery Ġfirst Ġstage Ġof Ġjudgment Ġprediction Ġstudies , Ġresearchers Ġfocus Ġon Ġmathematical Ġand Ġstatistical Ġanalysis Ġof Ġexisting Ġcases , Ġwithout Ġany Ġconclusions Ġor Ġmethod ologies Ġon Ġhow Ġto Ġpredict Ġthem ĠB IB REF 9 Ġ, ĠB IB REF 10 Ġ, ĠB IB REF 11 Ġ, ĠB IB REF 12 Ġ, ĠB IB REF 13 Ġ, ĠB IB REF 14 Ġ. ĠRecent Ġattempts Ġconsider Ġjudgment Ġprediction Ġunder Ġthe Ġtext Ġclassification Ġframework . ĠMost Ġof Ġthese Ġworks Ġextract Ġefficient Ġfeatures Ġfrom Ġtext Ġ( e . g ., ĠN - gram s ) ĠB IB REF 15 Ġ, ĠB IB REF 4 Ġ, ĠB IB REF 1 Ġ, ĠB IB REF 16 Ġ, ĠB IB REF 17 Ġor Ġcase Ġprofiles Ġ( e . g ., Ġdates , Ġterms , Ġlocations Ġand Ġtypes ) ĠB IB REF 2 Ġ. ĠAll Ġthese Ġmethods Ġrequire Ġa Ġlarge Ġamount Ġof Ġhuman Ġeffort Ġto Ġdesign Ġfeatures Ġor Ġannot ate Ġcases . ĠBesides , Ġthey Ġalso Ġsuffer Ġfrom Ġgeneral ization Ġissue Ġwhen Ġapplied Ġto Ġother Ġscenarios . </s> Mot ivated Ġby Ġthe Ġsuccessful Ġapplication Ġof Ġdeep Ġneural Ġnetworks , ĠLuo Ġet Ġal . ĠB IB REF 3 Ġintroduce Ġan Ġattention - based Ġneural Ġmodel Ġto Ġpredict Ġcharges Ġof Ġcriminal Ġcases , Ġand Ġverify Ġthe Ġeffectiveness Ġof Ġtaking Ġlaw Ġarticles Ġinto Ġconsideration . ĠNevertheless , Ġthey Ġstill Ġfall Ġinto Ġthe Ġtext Ġclassification Ġframework Ġand Ġlack Ġthe Ġability Ġto Ġhandle Ġmultiple Ġinputs Ġwith Ġmore Ġcomplicated Ġstructures . </s> Text ĠClassification </s> As Ġthe Ġbasis Ġof Ġprevious Ġjudgment Ġprediction Ġworks , Ġtypical Ġtext Ġclassification Ġtask Ġtakes Ġa Ġsingle Ġtext Ġcontent Ġas Ġinput Ġand Ġpredicts Ġthe Ġcategory Ġit Ġbelongs Ġto . ĠRecent Ġworks Ġusually Ġemploy Ġneural Ġnetworks Ġto Ġmodel Ġthe Ġinternal Ġstructure Ġof Ġa Ġsingle Ġinput ĠB IB REF 18 Ġ, ĠB IB REF 19 Ġ, ĠB IB REF 20 Ġ, ĠB IB REF 21 Ġ. </s> There Ġalso Ġexists Ġanother Ġthread Ġof Ġtext Ġclassification Ġcalled Ġentail ment Ġprediction . ĠMethods Ġproposed Ġin ĠB IB REF 22 Ġ, ĠB IB REF 23 Ġare Ġintended Ġfor Ġcomplementary Ġinputs , Ġbut Ġthe Ġmechanisms Ġcan Ġbe Ġconsidered Ġas Ġa Ġsimplified Ġversion Ġof Ġreading Ġcomprehension . </s> Reading ĠComp rehens ion </s> Reading Ġcomprehension Ġis Ġa Ġrelevant Ġtask Ġto Ġmodel Ġheter ogeneous Ġand Ġcomplementary Ġinputs , Ġwhere Ġan Ġanswer Ġis Ġpredicted Ġgiven Ġtwo Ġchannels Ġof Ġinputs , Ġi . e . Ġa Ġtextual Ġpassage Ġand Ġa Ġquery . ĠConsider able Ġprogress Ġhas Ġbeen Ġmade ĠB IB REF 6 Ġ, ĠB IB REF 24 Ġ, ĠB IB REF 5 Ġ. ĠThese Ġmodels Ġemploy Ġvarious Ġattention Ġmechanism Ġto Ġmodel Ġthe Ġinteraction Ġbetween Ġpassage Ġand Ġquery . ĠInspired Ġby Ġthe Ġadvantage Ġof Ġreading Ġcomprehension Ġmodels Ġon Ġmodeling Ġmultiple Ġinputs , Ġwe Ġapply Ġthis Ġidea Ġinto Ġthe Ġlegal Ġarea Ġand Ġpropose Ġlegal Ġreading Ġcomprehension Ġfor Ġjudgment Ġprediction . </s> Con ventional ĠReading ĠComp rehens ion </s> Con ventional Ġreading Ġcomprehension ĠB IB REF 25 Ġ, ĠB IB REF 26 Ġ, ĠB IB REF 7 Ġ, ĠB IB REF 8 Ġusually Ġconsiders Ġreading Ġcomprehension Ġas Ġpredicting Ġthe Ġanswer Ġgiven Ġa Ġpassage Ġand Ġa Ġquery , Ġwhere Ġthe Ġanswer Ġcould Ġbe Ġa Ġsingle Ġword , Ġa Ġtext Ġspan Ġof Ġthe Ġoriginal Ġpassage , Ġchosen Ġfrom Ġanswer Ġcandidates , Ġor Ġgenerated Ġby Ġhuman Ġannot ators . </s> Generally , Ġan Ġinstance Ġin ĠRC Ġis Ġrepresented Ġas Ġa Ġtriple ĠIN LINE FORM 0 Ġ, Ġwhere ĠIN LINE FORM 1 Ġ, ĠIN LINE FORM 2 Ġand ĠIN LINE FORM 3 Ġcorrespond Ġto ĠIN LINE FORM 4 Ġ, ĠIN LINE FORM 5 Ġand ĠIN LINE FORM 6 Ġrespectively . ĠGiven Ġa Ġtriple ĠIN LINE FORM 7 Ġ, ĠRC Ġtakes Ġthe Ġpair ĠIN LINE FORM 8 Ġas Ġthe Ġinput Ġand Ġemploys Ġattention - based Ġneural Ġmodels Ġto Ġconstruct Ġan Ġefficient Ġrepresentation . ĠAfterwards , Ġthe Ġrepresentation Ġis Ġfed Ġinto Ġthe Ġoutput Ġlayer Ġto Ġselect Ġor Ġgenerate Ġan ĠIN LINE FORM 9 Ġ. </s> Legal ĠReading ĠComp rehens ion </s> Ex isting Ġworks Ġusually Ġformal ize Ġjudgment Ġprediction Ġas Ġa Ġtext Ġclassification Ġtask Ġand Ġfocus Ġon Ġextracting Ġwell - designed Ġfeatures Ġof Ġspecific Ġcases . ĠSuch Ġsimpl ification Ġignores Ġthat Ġthe Ġjudgment Ġof Ġa Ġcase Ġis Ġdetermined Ġby Ġits Ġfact Ġdescription Ġand Ġmultiple Ġpleas . ĠMoreover , Ġthe Ġfinal Ġjudgment Ġshould Ġact Ġup Ġto Ġthe Ġlegal Ġprovisions , Ġespecially Ġin Ġcivil Ġlaw Ġsystems . ĠTherefore , Ġhow Ġto Ġintegrate Ġthe Ġinformation Ġ( i . e ., Ġfact Ġdescriptions , Ġpleas , Ġand Ġlaw Ġarticles ) Ġin Ġa Ġreasonable Ġway Ġis Ġcritical Ġfor Ġjudgment Ġprediction . </s> Insp ired Ġby Ġthe Ġsuccessful Ġapplication Ġof ĠRC , Ġwe Ġpropose Ġa Ġframework Ġof ĠLegal ĠReading ĠComp rehens ion ( L RC ) Ġfor Ġjudgment Ġprediction Ġin Ġthe Ġlegal Ġarea . ĠAs Ġillustrated Ġin ĠFig . ĠFIG REF 1 Ġ, Ġfor Ġeach Ġplea Ġin Ġa Ġgiven Ġcase , Ġthe Ġprediction Ġof Ġjudgment Ġresult Ġis Ġmade Ġbased Ġthe Ġfact Ġdescription Ġand Ġthe Ġpotentially Ġrelevant Ġlaw Ġarticles . </s> In Ġa Ġnutshell , ĠL RC Ġcan Ġbe Ġformal ized Ġas Ġthe Ġfollowing Ġquadru plet Ġtask : ĠDIS PLAY FORM 0 </s> where ĠIN LINE FORM 0 Ġis Ġthe Ġfact Ġdescription , ĠIN LINE FORM 1 Ġis Ġthe Ġplea , ĠIN LINE FORM 2 Ġis Ġthe Ġlaw Ġarticles Ġand ĠIN LINE FORM 3 Ġis Ġthe Ġresult . ĠGiven ĠIN LINE FORM 4 Ġ, ĠL RC Ġaims Ġto Ġpredict Ġthe Ġjudgment Ġresult Ġas ĠDIS PLAY FORM 0 </s> The Ġprobability Ġis Ġcalculated Ġwith Ġrespect Ġto Ġthe Ġinteraction Ġamong Ġthe Ġtriple ĠIN LINE FORM 0 Ġ, Ġwhich Ġwill Ġdraw Ġon Ġthe Ġexperience Ġof Ġthe Ġinteraction Ġbetween ĠIN LINE FORM 1 Ġpairs Ġin ĠRC . </s> To Ġsummarize , ĠL RC Ġis Ġinnovative Ġin Ġthe Ġfollowing Ġaspects : </s> ( 1 ) ĠWhile Ġprevious Ġworks Ġfit Ġthe Ġproblem Ġinto Ġtext Ġclassification Ġframework , ĠL RC Ġre - form al izes Ġthe Ġway Ġto Ġapproach Ġsuch Ġproblems . ĠThis Ġnew Ġframework Ġprovides Ġthe Ġability Ġto Ġdeal Ġwith Ġthe Ġheterogeneity Ġof Ġthe Ġcomplementary Ġinputs . </s> ( 2 ) ĠRather Ġthan Ġemploying Ġconventional ĠRC Ġmodels Ġto Ġhandle Ġpair - wise Ġtext Ġinformation Ġin Ġthe Ġlegal Ġarea , ĠL RC Ġtakes Ġthe Ġcritical Ġlaw Ġarticles Ġinto Ġconsideration Ġand Ġmodels Ġthe Ġfacts , Ġpleas , Ġand Ġlaw Ġarticles Ġjointly Ġfor Ġjudgment Ġprediction , Ġwhich Ġis Ġmore Ġsuitable Ġto Ġsimulate Ġthe Ġhuman Ġmode Ġof Ġdealing Ġwith Ġcases . </s> Methods </s> We Ġpropose Ġa Ġnovel Ġjudgment Ġprediction Ġmodel ĠAuto Judge Ġto Ġinstant iate Ġthe ĠL RC Ġframework . ĠAs Ġshown Ġin ĠFig . ĠFIG REF 6 Ġ, ĠAuto Judge Ġconsists Ġof Ġthree Ġflexible Ġmodules , Ġincluding Ġa Ġtext Ġenc oder , Ġa Ġpair - wise Ġattentive Ġreader , Ġand Ġan Ġoutput Ġmodule . </s> In Ġthe Ġfollowing Ġparts , Ġwe Ġgive Ġa Ġdetailed Ġintroduction Ġto Ġthese Ġthree Ġmodules . </s> Text ĠEnc oder </s> As Ġillustrated Ġin ĠFig . ĠFIG REF 6 Ġ, ĠText ĠEnc oder Ġaims Ġto Ġencode Ġthe Ġword Ġsequences Ġof Ġinputs Ġinto Ġcontinuous Ġrepresentation Ġsequences . </s> Form ally , Ġconsider Ġa Ġfact Ġdescription ĠIN LINE FORM 0 Ġ, Ġa Ġplea ĠIN LINE FORM 1 Ġ, Ġand Ġthe Ġrelevant Ġlaw Ġarticles ĠIN LINE FORM 2 Ġ, Ġwhere ĠIN LINE FORM 3 Ġdenotes Ġthe ĠIN LINE FORM 4 Ġ- th Ġword Ġin Ġthe Ġsequence Ġand ĠIN LINE FORM 5 Ġare Ġthe Ġlengths Ġof Ġword Ġsequences ĠIN LINE FORM 6 Ġrespectively . ĠFirst , Ġwe Ġconvert Ġthe Ġwords Ġto Ġtheir Ġrespective Ġword Ġembed d ings Ġto Ġobtain ĠIN LINE FORM 7 Ġ, ĠIN LINE FORM 8 Ġand ĠIN LINE FORM 9 Ġ, Ġwhere ĠIN LINE FORM 10 Ġ. ĠAfterwards , Ġwe Ġemploy Ġbi - direction al ĠGR U ĠB IB REF 27 Ġ, ĠB IB REF 28 Ġ, ĠB IB REF 29 Ġto Ġproduce Ġthe Ġencoded Ġrepresentation ĠIN LINE FORM 11 Ġof Ġall Ġwords Ġas Ġfollows : ĠDIS PLAY FORM 0 </s> Note Ġthat , Ġwe Ġadopt Ġdifferent Ġbi - direction al ĠGR Us Ġto Ġencode Ġfact Ġdescriptions , Ġpleas , Ġand Ġlaw Ġarticles Ġrespectively ( den oted Ġas ĠIN LINE FORM 0 Ġ, ĠIN LINE FORM 1 Ġ, Ġand ĠIN LINE FORM 2 Ġ). ĠWith Ġthese Ġtext Ġenc od ers , ĠIN LINE FORM 3 Ġ, ĠIN LINE FORM 4 Ġ, Ġand ĠIN LINE FORM 5 Ġare Ġconverting Ġinto ĠIN LINE FORM 6 Ġ, ĠIN LINE FORM 7 Ġ, Ġand ĠIN LINE FORM 8 Ġ. </s> P air - W ise ĠAtt ent ive ĠReader </s> How Ġto Ġmodel Ġthe Ġinteractions Ġamong Ġthe Ġinput Ġtext Ġis Ġthe Ġmost Ġimportant Ġproblem Ġin Ġreading Ġcomprehension . ĠIn ĠAuto Judge , Ġwe Ġemploy Ġa Ġpair - wise Ġattentive Ġreader Ġto Ġprocess ĠIN LINE FORM 0 Ġand ĠIN LINE FORM 1 Ġrespectively . ĠMore Ġspecifically , Ġwe Ġpropose Ġto Ġuse Ġpair - wise Ġmutual Ġattention Ġmechanism Ġto Ġcapture Ġthe Ġcomplex Ġsemantic Ġinteraction Ġbetween Ġtext Ġpairs , Ġas Ġwell Ġas Ġincreasing Ġthe Ġinterpret ability Ġof ĠAuto Judge . </s> For Ġeach Ġinput Ġpair ĠIN LINE FORM 0 Ġor ĠIN LINE FORM 1 Ġ, Ġwe Ġemploy Ġpair - wise Ġmutual Ġattention Ġto Ġselect Ġrelevant Ġinformation Ġfrom Ġfact Ġdescriptions ĠIN LINE FORM 2 Ġand Ġproduce Ġmore Ġinformative Ġrepresentation Ġsequences . </s> As Ġa Ġvariant Ġof Ġthe Ġoriginal Ġattention Ġmechanism ĠB IB REF 28 Ġ, Ġwe Ġdesign Ġthe Ġpair - wise Ġmutual Ġattention Ġunit Ġas Ġa ĠGR U Ġwith Ġinternal Ġmemories Ġden oted Ġas Ġm GR U . </s> Taking Ġthe Ġrepresentation Ġsequence Ġpair ĠIN LINE FORM 0 Ġfor Ġinstance , Ġm GR U Ġstores Ġthe Ġfact Ġsequence ĠIN LINE FORM 1 Ġinto Ġits Ġmemories . ĠFor Ġeach Ġtimestamp ĠIN LINE FORM 2 Ġ, Ġit Ġselects Ġrelevant Ġfact Ġinformation ĠIN LINE FORM 3 Ġfrom Ġthe Ġmemories Ġas Ġfollows , ĠDIS PLAY FORM 0 </s> Here , Ġthe Ġweight ĠIN LINE FORM 0 Ġis Ġthe Ġsoft max Ġvalue Ġas ĠDIS PLAY FORM 0 </s> Note Ġthat , ĠIN LINE FORM 0 Ġrepresents Ġthe Ġrelevance Ġbetween ĠIN LINE FORM 1 Ġand ĠIN LINE FORM 2 Ġ. ĠIt Ġis Ġcalculated Ġas Ġfollows , ĠDIS PLAY FORM 0 </s> Here , ĠIN LINE FORM 0 Ġis Ġthe Ġlast Ġhidden Ġstate Ġin Ġthe ĠGR U , Ġwhich Ġwill Ġbe Ġintroduced Ġin Ġthe Ġfollowing Ġpart . ĠIN LINE FORM 1 Ġis Ġa Ġweight Ġvector , Ġand ĠIN LINE FORM 2 Ġ, ĠIN LINE FORM 3 Ġ, ĠIN LINE FORM 4 Ġare Ġattention Ġmetrics Ġof Ġour Ġproposed Ġpair - wise Ġattention Ġmechanism . </s> With Ġthe Ġrelevant Ġfact Ġinformation ĠIN LINE FORM 0 Ġand ĠIN LINE FORM 1 Ġ, Ġwe Ġget Ġthe ĠIN LINE FORM 2 Ġ- th Ġinput Ġof Ġm GR U Ġas ĠDIS PLAY FORM 0 </s> where ĠIN LINE FORM 0 Ġindicates Ġthe Ġconc aten ation Ġoperation . </s> Then , Ġwe Ġfeed ĠIN LINE FORM 0 Ġinto ĠGR U Ġto Ġget Ġmore Ġinformative Ġrepresentation Ġsequence ĠIN LINE FORM 1 Ġas Ġfollows , ĠDIS PLAY FORM 0 </s> For Ġthe Ġinput Ġpair ĠIN LINE FORM 0 Ġ, Ġwe Ġcan Ġget ĠIN LINE FORM 1 Ġin Ġthe Ġsame Ġway . ĠTherefore , Ġwe Ġomit Ġthe Ġimplementation Ġdetails ĠHere . </s> Similar Ġstructures Ġwith Ġattention Ġmechanism Ġare Ġalso Ġapplied Ġin ĠB IB REF 5 Ġ, ĠB IB REF 30 Ġ, ĠB IB REF 31 Ġ, ĠB IB REF 28 Ġto Ġobtain Ġmutually Ġaware Ġrepresentations Ġin Ġreading Ġcomprehension Ġmodels , Ġwhich Ġsignificantly Ġimprove Ġthe Ġperformance Ġof Ġthis Ġtask . </s> Output ĠLayer </s> Using Ġtext Ġenc oder Ġand Ġpair - wise Ġattentive Ġreader , Ġthe Ġinitial Ġinput Ġtriple ĠIN LINE FORM 0 Ġhas Ġbeen Ġconverted Ġinto Ġtwo Ġsequences , Ġi . e ., ĠIN LINE FORM 1 Ġand ĠIN LINE FORM 2 Ġ, Ġwhere ĠIN LINE FORM 3 Ġis Ġdefined Ġsimilarly Ġto ĠIN LINE FORM 4 Ġ. ĠThese Ġsequences Ġreserve Ġcomplex Ġsemantic Ġinformation Ġabout Ġthe Ġpleas Ġand Ġlaw Ġarticles , Ġand Ġfilter Ġout Ġirrelevant Ġinformation Ġin Ġfact Ġdescriptions . </s> With Ġthese Ġtwo Ġsequences , Ġwe Ġconc aten ate ĠIN LINE FORM 0 Ġand ĠIN LINE FORM 1 Ġalong Ġthe Ġsequence Ġlength Ġdimension Ġto Ġgenerate Ġthe Ġsequence ĠIN LINE FORM 2 Ġ. ĠSince Ġwe Ġhave Ġemployed Ġseveral ĠGR U Ġlayers Ġto Ġencode Ġthe Ġsequential Ġinputs , Ġanother Ġrecurrent Ġlayer Ġmay Ġbe Ġredundant . ĠTherefore , Ġwe Ġutilize Ġa Ġ1 - layer ĠCNN ĠB IB REF 18 Ġto Ġcapture Ġthe Ġlocal Ġstructure Ġand Ġgenerate Ġthe Ġrepresentation Ġvector Ġfor Ġthe Ġfinal Ġprediction . </s> Assuming ĠIN LINE FORM 0 Ġis Ġthe Ġpredicted Ġprobability Ġthat Ġthe Ġplea Ġin Ġthe Ġcase Ġsample Ġwould Ġbe Ġsupported Ġand ĠIN LINE FORM 1 Ġis Ġthe Ġgold Ġstandard , ĠAuto Judge Ġaims Ġto Ġminimize Ġthe Ġcross - ent ropy Ġas Ġfollows , ĠDIS PLAY FORM 0 </s> where ĠIN LINE FORM 0 Ġis Ġthe Ġnumber Ġof Ġtraining Ġdata . ĠAs Ġall Ġthe Ġcalculation Ġin Ġour Ġmodel Ġis Ġdifferent iable , Ġwe Ġemploy ĠAdam ĠB IB REF 32 Ġfor Ġoptimization . </s> Exper iments </s> To Ġevaluate Ġthe Ġproposed ĠL RC Ġframework Ġand Ġthe ĠAuto Judge Ġmodel , Ġwe Ġcarry Ġout Ġa Ġseries Ġof Ġexperiments Ġon Ġthe Ġdivorce Ġproceedings , Ġa Ġtypical Ġyet Ġcomplex Ġfield Ġof Ġcivil Ġcases . ĠDiv orce Ġproceedings Ġoften Ġcome Ġwith Ġseveral Ġkinds Ġof Ġpleas , Ġe . g . Ġseeking Ġdivorce , Ġcustody Ġof Ġchildren , Ġcompensation , Ġand Ġmaintenance , Ġwhich Ġfocuses Ġon Ġdifferent Ġaspects Ġand Ġthus Ġmakes Ġit Ġa Ġchallenge Ġfor Ġjudgment Ġprediction . </s> Dat as et ĠConstruction Ġfor ĠEvaluation </s> Since Ġnone Ġof Ġthe Ġdatasets Ġfrom Ġprevious Ġworks Ġhave Ġbeen Ġpublished , Ġwe Ġdecide Ġto Ġbuild Ġa Ġnew Ġone . ĠWe Ġrandomly Ġcollect ĠIN LINE FORM 0 Ġcases Ġfrom ĠChina ĠJud gments ĠOnline , Ġamong Ġwhich ĠIN LINE FORM 1 Ġcases Ġare Ġfor Ġtraining , ĠIN LINE FORM 2 Ġeach Ġfor Ġvalidation Ġand Ġtesting . ĠAmong Ġthe Ġoriginal Ġcases , ĠIN LINE FORM 3 Ġare Ġgranted Ġdivorce Ġand Ġothers Ġnot . ĠThere Ġare ĠIN LINE FORM 4 Ġvalid Ġpleas Ġin Ġtotal , Ġwith ĠIN LINE FORM 5 Ġsupported Ġand ĠIN LINE FORM 6 Ġrejected . ĠNote Ġthat , Ġif Ġthe Ġdivorce Ġplea Ġin Ġa Ġcase Ġis Ġnot Ġgranted , Ġthe Ġother Ġpleas Ġof Ġthis Ġcase Ġwill Ġnot Ġbe Ġconsidered Ġby Ġthe Ġjudge . ĠCase Ġmaterials Ġare Ġall Ġnatural Ġlanguage Ġsentences , Ġwith Ġaverage ly ĠIN LINE FORM 7 Ġtokens Ġper Ġfact Ġdescription Ġand ĠIN LINE FORM 8 Ġper Ġplea . ĠThere Ġare Ġ62 Ġrelevant Ġlaw Ġarticles Ġin Ġtotal , Ġeach Ġwith ĠIN LINE FORM 9 Ġtokens Ġaverage ly . ĠNote Ġthat Ġthe Ġcase Ġdocuments Ġinclude Ġspecial Ġtyp ographical Ġsignals , Ġmaking Ġit Ġeasy Ġto Ġextract Ġlabeled Ġdata Ġwith Ġregular Ġexpression . </s> We Ġapply Ġsome Ġrules Ġwith Ġlegal Ġprior Ġto Ġpre process Ġthe Ġdataset Ġaccording Ġto Ġprevious Ġworks ĠB IB REF 33 Ġ, ĠB IB REF 34 Ġ, ĠB IB REF 35 Ġ, Ġwhich Ġhave Ġproved Ġeffective Ġin Ġour Ġexperiments . </s> Name ĠReplacement : ĠAll Ġnames Ġin Ġcase Ġdocuments Ġare Ġreplaced Ġwith Ġmarks Ġindicating Ġtheir Ġroles , Ġinstead Ġof Ġsimply Ġanonym izing Ġthem , Ġe . g . Ġ< Pl ant iff >, Ġ< Def endant >, Ġ< D aughter _ x > Ġand Ġso Ġon . ĠSince ĠâĢ ľ all Ġare Ġequal Ġbefore Ġthe Ġlaw âĢ Ŀ , Ġnames Ġshould Ġmake Ġno Ġmore Ġdifference Ġthan Ġwhat Ġrole Ġthey Ġtake . </s> Law ĠArticle ĠF iltration Ġ: ĠSince Ġmost Ġaccessible Ġdivorce Ġproceeding Ġdocuments Ġdo Ġnot Ġcontain Ġground - truth Ġfine - gr ained Ġarticles , Ġwe Ġuse Ġan Ġun super vised Ġmethod Ġinstead . ĠFirst , Ġwe Ġextract Ġall Ġthe Ġarticles Ġfrom Ġthe Ġlaw Ġtext Ġwith Ġregular Ġexpression . ĠAfterwards , Ġwe Ġselect Ġthe Ġmost Ġrelevant Ġ10 Ġarticles Ġaccording Ġto Ġthe Ġfact Ġdescriptions Ġas Ġfollows . ĠWe Ġobtain Ġsentence Ġrepresentation Ġwith ĠCB OW ĠB IB REF 36 Ġ, ĠB IB REF 37 Ġweighted Ġby Ġinverse Ġdocument Ġfrequency , Ġand Ġcalculate Ġcos ine Ġdistance Ġbetween Ġcases Ġand Ġlaw Ġarticles . ĠWord Ġembed d ings Ġare Ġpre - trained Ġwith ĠChinese ĠWikipedia Ġpages . ĠAs Ġthe Ġfinal Ġstep , Ġwe Ġextract Ġtop Ġ5 Ġrelevant Ġarticles Ġfor Ġeach Ġsample Ġrespectively Ġfrom Ġthe Ġmain Ġmarriage Ġlaw Ġarticles Ġand Ġtheir Ġinterpretations , Ġwhich Ġare Ġequally Ġimportant . ĠWe Ġmanually Ġcheck Ġthe Ġextracted Ġarticles Ġfor Ġ100 Ġcases Ġto Ġensure Ġthat Ġthe Ġextraction Ġquality Ġis Ġfairly Ġgood Ġand Ġacceptable . </s> The Ġfil t ration Ġprocess Ġis Ġautomatic Ġand Ġfully Ġun super vised Ġsince Ġthe Ġoriginal Ġdocuments Ġhave Ġno Ġground - truth Ġlabels Ġfor Ġfine - gr ained Ġlaw Ġarticles , Ġand Ġcoarse - gr ained Ġlaw - articles Ġonly Ġprovide Ġlimited Ġinformation . ĠWe Ġalso Ġexperiment Ġwith Ġthe Ġground - truth Ġarticles , Ġbut Ġonly Ġa Ġsmall Ġfraction Ġof Ġthem Ġhas Ġfine - gr ained Ġones , Ġand Ġthey Ġare Ġusually Ġnot Ġavailable Ġin Ġreal - world Ġscenarios . </s> Im plementation ĠDetails </s> We Ġemploy ĠJ ie ba Ġfor ĠChinese Ġword Ġsegment ation Ġand Ġkeep Ġthe Ġtop ĠIN LINE FORM 0 Ġfrequent Ġwords . ĠThe Ġword Ġembed ding Ġsize Ġis Ġset Ġto Ġ128 Ġand Ġthe Ġother Ġlow - frequency Ġwords Ġare Ġreplaced Ġwith Ġthe Ġmark Ġ< UN K >. ĠThe Ġhidden Ġsize Ġof ĠGR U Ġis Ġset Ġto Ġ128 Ġfor Ġeach Ġdirection Ġin ĠBi - GR U . ĠIn Ġthe Ġpair - wise Ġattentive Ġreader , Ġthe Ġhidden Ġstate Ġis Ġset Ġto Ġ256 Ġfor Ġm GR u . ĠIn Ġthe ĠCNN Ġlayer , Ġfilter Ġwindows Ġare Ġset Ġto Ġ1 , Ġ3 , Ġ4 , Ġand Ġ5 Ġwith Ġeach Ġfilter Ġcontaining Ġ200 Ġfeature Ġmaps . ĠWe Ġadd Ġa Ġdrop out Ġlayer ĠB IB REF 38 Ġafter Ġthe ĠCNN Ġlayer Ġwith Ġa Ġdrop out Ġrate Ġof ĠIN LINE FORM 1 Ġ. ĠWe Ġuse ĠAdam ĠB IB REF 32 Ġfor Ġtraining Ġand Ġset Ġlearning Ġrate Ġto ĠIN LINE FORM 2 Ġ, ĠIN LINE FORM 3 Ġto ĠIN LINE FORM 4 Ġ, ĠIN LINE FORM 5 Ġto ĠIN LINE FORM 6 Ġ, ĠIN LINE FORM 7 Ġto ĠIN LINE FORM 8 Ġ, Ġbatch Ġsize Ġto Ġ64 . ĠWe Ġemploy Ġprecision , Ġrecall , ĠF 1 Ġand Ġaccuracy Ġfor Ġevaluation Ġmetrics . ĠWe Ġrepeat Ġall Ġthe Ġexperiments Ġfor Ġ10 Ġtimes , Ġand Ġreport Ġthe Ġaverage Ġresults . </s> Bas elines </s> For Ġcomparison , Ġwe Ġadopt Ġand Ġre - im plement Ġthree Ġkinds Ġof Ġbas elines Ġas Ġfollows : </s> We Ġimplement Ġan ĠS VM Ġwith Ġlex ical Ġfeatures Ġin Ġaccordance Ġwith Ġprevious Ġworks ĠB IB REF 16 Ġ, ĠB IB REF 17 Ġ, ĠB IB REF 1 Ġ, ĠB IB REF 15 Ġ, ĠB IB REF 4 Ġand Ġselect Ġthe Ġbest Ġfeature Ġset Ġon Ġthe Ġdevelopment Ġset . </s> We Ġimplement Ġand Ġfine - t une Ġa Ġseries Ġof Ġneural Ġtext Ġclass ifiers , Ġincluding Ġattention - based Ġmethod ĠB IB REF 3 Ġand Ġother Ġmethods Ġwe Ġdeem Ġimportant . ĠCNN ĠB IB REF 18 Ġand ĠGR U ĠB IB REF 27 Ġ, ĠB IB REF 21 Ġtake Ġas Ġinput Ġthe Ġconc aten ation Ġof Ġfact Ġdescription Ġand Ġplea . ĠSimilarly , ĠCNN / GR U + law Ġrefers Ġto Ġusing Ġthe Ġconc aten ation Ġof Ġfact Ġdescription , Ġplea Ġand Ġlaw Ġarticles Ġas Ġinputs . </s> We Ġimplement Ġand Ġtrain Ġsome Ġoff - the - she lf ĠRC Ġmodels , Ġincluding Ġr - net ĠB IB REF 5 Ġand ĠAo A ĠB IB REF 6 Ġ, Ġwhich Ġare Ġthe Ġleading Ġmodels Ġon ĠS Qu AD Ġleader board . ĠIn Ġour Ġinitial Ġexperiments , Ġthese Ġmodels Ġtake Ġfact Ġdescription Ġas Ġpassage Ġand Ġplea Ġas Ġquery . ĠFurther , ĠLaw Ġarticles Ġare Ġadded Ġto Ġthe Ġfact Ġdescription Ġas Ġa Ġpart Ġof Ġthe Ġreading Ġmaterials , Ġwhich Ġis Ġa Ġsimple Ġway Ġto Ġconsider Ġthem Ġas Ġwell . </s> Results Ġand ĠAnalysis </s> From ĠTable ĠT AB REF 37 Ġ, Ġwe Ġhave Ġthe Ġfollowing Ġobservations : </s> ( 1 ) ĠAuto Judge Ġconsistently Ġand Ġsignificantly Ġoutper forms Ġall Ġthe Ġbas elines , Ġincluding ĠRC Ġmodels Ġand Ġother Ġneural Ġtext Ġclassification Ġmodels , Ġwhich Ġshows Ġthe Ġeffectiveness Ġand Ġrobust ness Ġof Ġour Ġmodel . </s> ( 2 ) ĠRC Ġmodels Ġachieve Ġbetter Ġperformance Ġthan Ġmost Ġtext Ġclassification Ġmodels Ġ( excluding ĠGR U + Att ention ), Ġwhich Ġindicates Ġthat Ġreading Ġmechanism Ġis Ġa Ġbetter Ġway Ġto Ġintegrate Ġinformation Ġfrom Ġheter ogeneous Ġyet Ġcomplementary Ġinputs . ĠOn Ġthe Ġcontrary , Ġsimply Ġadding Ġlaw Ġarticles Ġas Ġa Ġpart Ġof Ġthe Ġreading Ġmaterials Ġmakes Ġno Ġdifference Ġin Ġperformance . ĠNote Ġthat , ĠGR U + Att ention Ġemploy Ġsimilar Ġattention Ġmechanism Ġas ĠRC Ġdoes Ġand Ġtakes Ġadditional Ġlaw Ġarticles Ġinto Ġconsideration , Ġthus Ġachieves Ġcomparable Ġperformance Ġwith ĠRC Ġmodels . </s> ( 3 ) ĠComp aring Ġwith Ġconventional ĠRC Ġmodels , ĠAuto Judge Ġachieves Ġsignificant Ġimprovement Ġwith Ġthe Ġconsideration Ġof Ġadditional Ġlaw Ġarticles . ĠIt Ġreflects Ġthe Ġdifference Ġbetween ĠL RC Ġand Ġconventional ĠRC Ġmodels . ĠWe Ġre - form al ize ĠL RC Ġin Ġlegal Ġarea Ġto Ġincorporate Ġlaw Ġarticles Ġvia Ġthe Ġreading Ġmechanism , Ġwhich Ġcan Ġenhance Ġjudgment Ġprediction . ĠMoreover , ĠCNN / GR U + law Ġdecrease Ġthe Ġperformance Ġby Ġsimply Ġconc aten ating Ġoriginal Ġtext Ġwith Ġlaw Ġarticles , Ġwhile ĠGR U + Att ention / Auto Judge Ġincrease Ġthe Ġperformance Ġby Ġintegrating Ġlaw Ġarticles Ġwith Ġattention Ġmechanism . ĠIt Ġshows Ġthe Ġimportance Ġand Ġrationality Ġof Ġusing Ġattention Ġmechanism Ġto Ġcapture Ġthe Ġinteraction Ġbetween Ġmultiple Ġinputs . </s> The Ġexperiments Ġsupport Ġour Ġhypothesis Ġas Ġproposed Ġin Ġthe ĠIntroduction Ġpart Ġthat Ġin Ġcivil Ġcases , Ġit 's Ġimportant Ġto Ġmodel Ġthe Ġinteractions Ġamong Ġcase Ġmaterials . ĠReading Ġmechanism Ġcan Ġwell Ġperform Ġthe Ġmatching Ġamong Ġthem . </s> A bl ation ĠTest </s> Auto Judge Ġis Ġcharacterized Ġby Ġthe Ġincorporation Ġof Ġpair - wise Ġattentive Ġreader , Ġlaw Ġarticles , Ġand Ġa ĠCNN Ġoutput Ġlayer , Ġas Ġwell Ġas Ġsome Ġpre - processing Ġwith Ġlegal Ġprior . ĠWe Ġdesign Ġab lation Ġtests Ġrespectively Ġto Ġevaluate Ġthe Ġeffectiveness Ġof Ġthese Ġmodules . ĠWhen Ġtaken Ġoff Ġthe Ġattention Ġmechanism , ĠAuto Judge Ġde grades Ġinto Ġa ĠGR U Ġon Ġwhich Ġa ĠCNN Ġis Ġstacked . ĠWhen Ġtaken Ġoff Ġlaw Ġarticles , Ġthe ĠCNN Ġoutput Ġlayer Ġonly Ġtakes ĠIN LINE FORM 0 Ġas Ġinput . ĠBesides , Ġour Ġmodel Ġis Ġtested Ġrespectively Ġwithout Ġname - repl acement Ġor Ġun super vised Ġselection Ġof Ġlaw Ġarticles Ġ( i . e . Ġpassing Ġthe Ġwhole Ġlaw Ġtext ). ĠAs Ġmentioned Ġabove , Ġwe Ġsystem Ġuse Ġlaw Ġarticles Ġextracted Ġwith Ġun super vised Ġmethod , Ġso Ġwe Ġalso Ġexperiment Ġwith Ġground - truth Ġlaw Ġarticles . </s> Results Ġare Ġshown Ġin ĠTable ĠT AB REF 38 Ġ. ĠWe Ġcan Ġinfer Ġthat : </s> ( 1 ) ĠThe Ġperformance Ġdrops Ġsignificantly Ġafter Ġremoving Ġthe Ġattention Ġlayer Ġor Ġexcluding Ġthe Ġlaw Ġarticles , Ġwhich Ġis Ġconsistent Ġwith Ġthe Ġcomparison Ġbetween ĠAuto Judge Ġand Ġbas elines . ĠThe Ġresult Ġver ifies Ġthat Ġboth Ġthe Ġreading Ġmechanism Ġand Ġincorporation Ġof Ġlaw Ġarticles Ġare Ġimportant Ġand Ġeffective . </s> ( 2 ) ĠAfter Ġreplacing ĠCNN Ġwith Ġan ĠL ST M Ġlayer , Ġperformance Ġdrops Ġas Ġmuch Ġas ĠIN LINE FORM 0 Ġin Ġaccuracy Ġand ĠIN LINE FORM 1 Ġin ĠF 1 Ġscore . ĠThe Ġreason Ġmay Ġbe Ġthe Ġredundancy Ġof ĠR NN s . ĠAuto Judge Ġhas Ġemployed Ġseveral ĠGR U Ġlayers Ġto Ġencode Ġtext Ġsequences . ĠAnother ĠR NN Ġlayer Ġmay Ġbe Ġuseless Ġto Ġcapture Ġsequential Ġdependencies , Ġwhile ĠCNN Ġcan Ġcatch Ġthe Ġlocal Ġstructure Ġin Ġconv olution Ġwindows . </s> ( 3 ) ĠMot ivated Ġby Ġexisting Ġrule - based Ġworks , Ġwe Ġconduct Ġdata Ġpre - processing Ġon Ġcases , Ġincluding Ġname Ġreplacement Ġand Ġlaw Ġarticle Ġfil t ration . ĠIf Ġwe Ġremove Ġthe Ġpre - processing Ġoperations , Ġthe Ġperformance Ġdrops Ġconsiderably . ĠIt Ġdemonstrates Ġthat Ġapplying Ġthe Ġprior Ġknowledge Ġin Ġlegal Ġfiled Ġwould Ġbenefit Ġthe Ġunderstanding Ġof Ġlegal Ġcases . </s> It 's Ġintuitive Ġthat Ġthe Ġquality Ġof Ġthe Ġretrieved Ġlaw Ġarticles Ġwould Ġaffect Ġthe Ġfinal Ġperformance . ĠAs Ġis Ġshown Ġin ĠTable ĠT AB REF 38 Ġ, Ġfeeding Ġthe Ġwhole Ġlaw Ġtext Ġwithout Ġfil t ration Ġresults Ġin Ġworse Ġperformance . ĠHowever , Ġwhen Ġwe Ġtrain Ġand Ġevaluate Ġour Ġmodel Ġwith Ġground Ġtruth Ġarticles , Ġthe Ġperformance Ġis Ġboosted Ġby Ġnearly ĠIN LINE FORM 0 Ġin Ġboth ĠF 1 Ġand ĠAcc . ĠThe Ġperformance Ġimprovement Ġis Ġquite Ġlimited Ġcompared Ġto Ġthat Ġin Ġprevious Ġwork ĠB IB REF 3 Ġfor Ġthe Ġfollowing Ġreasons : Ġ( 1 ) ĠAs Ġmentioned Ġabove , Ġmost Ġcase Ġdocuments Ġonly Ġcontain Ġcoarse - gr ained Ġarticles , Ġand Ġonly Ġa Ġsmall Ġnumber Ġof Ġthem Ġcontain Ġfine - gr ained Ġones , Ġwhich Ġhas Ġlimited Ġinformation Ġin Ġthemselves . Ġ( 2 ) ĠUnlike Ġin Ġcriminal Ġcases Ġwhere Ġthe Ġapplication Ġof Ġan Ġarticle Ġindicates Ġthe Ġcorresponding Ġcrime , Ġlaw Ġarticles Ġin Ġcivil Ġcases Ġwork Ġas Ġreference , Ġand Ġcan Ġbe Ġapplied Ġin Ġboth Ġthe Ġcases Ġof Ġsupports Ġand Ġrejects . ĠAs Ġlaw Ġarticles Ġcut Ġboth Ġways Ġfor Ġthe Ġjudgment Ġresult , Ġthis Ġis Ġone Ġof Ġthe Ġcharacteristics Ġthat Ġdistinguishes Ġcivil Ġcases Ġfrom Ġcriminal Ġones . ĠWe Ġalso Ġneed Ġto Ġremember Ġthat , Ġthe Ġperformance Ġof ĠIN LINE FORM 1 Ġin Ġaccuracy Ġor ĠIN LINE FORM 2 Ġin ĠF 1 Ġscore Ġis Ġunatt ain able Ġin Ġreal - world Ġsetting Ġfor Ġautomatic Ġprediction Ġwhere Ġground - truth Ġarticles Ġare Ġnot Ġavailable . </s> In Ġthe Ġarea Ġof Ġcivil Ġcases , Ġthe Ġunderstanding Ġof Ġthe Ġcase Ġmaterials Ġand Ġhow Ġthey Ġinteract Ġis Ġa Ġcritical Ġfactor . ĠThe Ġinclusion Ġof Ġlaw Ġarticles Ġis Ġnot Ġenough . ĠAs Ġis Ġshown Ġin ĠTable ĠT AB REF 38 Ġ, Ġcompared Ġto Ġfeeding Ġthe Ġmodel Ġwith Ġan Ġun - selected Ġset Ġof Ġlaw Ġarticles , Ġtaking Ġaway Ġthe Ġreading Ġmechanism Ġresults Ġin Ġgreater Ġperformance Ġdrop . ĠTherefore , Ġthe Ġability Ġto Ġread , Ġunderstand Ġand Ġselect Ġrelevant Ġinformation Ġfrom Ġthe Ġcomplex Ġmulti - s ourced Ġcase Ġmaterials Ġis Ġnecessary . ĠIt 's Ġeven Ġmore Ġimportant Ġin Ġreal Ġworld Ġsince Ġwe Ġdon 't Ġhave Ġaccess Ġto Ġground - truth Ġlaw Ġarticles Ġto Ġmake Ġpredictions . </s> Case ĠStudy </s> We Ġvisualize Ġthe Ġheat Ġmaps Ġof Ġattention Ġresults . ĠAs Ġshown Ġin ĠFig . ĠFIG REF 47 Ġ, Ġdeeper Ġbackground Ġcolor Ġrepresents Ġlarger Ġattention Ġscore . ĠThe Ġattention Ġscore Ġis Ġcalculated Ġwith ĠE q . Ġ( ĠEQ REF 15 Ġ). ĠWe Ġtake Ġthe Ġaverage Ġof Ġthe Ġresulting ĠIN LINE FORM 0 Ġattention Ġmatrix Ġover Ġthe Ġtime Ġdimension Ġto Ġobtain Ġattention Ġvalues Ġfor Ġeach Ġword . </s> The Ġvisualization Ġdemonstrates Ġthat Ġthe Ġattention Ġmechanism Ġcan Ġcapture Ġrelevant Ġpatterns Ġand Ġsemantics Ġin Ġaccordance Ġwith Ġdifferent Ġpleas Ġin Ġdifferent Ġcases . </s> As Ġfor Ġthe Ġfailed Ġsamples , Ġthe Ġmost Ġcommon Ġreason Ġcomes Ġfrom Ġthe Ġanonymity Ġissue , Ġwhich Ġis Ġalso Ġshown Ġin ĠFig . ĠFIG REF 47 Ġ. ĠAs Ġmentioned Ġabove , Ġwe Ġconduct Ġname Ġreplacement . ĠHowever , Ġsome Ġcritical Ġelements Ġare Ġalso Ġanonym ized Ġby Ġthe Ġgovernment , Ġdue Ġto Ġthe Ġprivacy Ġissue . ĠThese Ġelements Ġare Ġsometimes Ġimportant Ġto Ġjudgment Ġprediction . ĠFor Ġexample , Ġdetermination Ġof Ġthe Ġkey Ġfactor Ġlong - time Ġseparation Ġis Ġrelevant Ġto Ġthe Ġexplicit Ġdates , Ġwhich Ġare Ġanonym ized . </s> Conclusion </s> In Ġthis Ġpaper , Ġwe Ġexplore Ġthe Ġtask Ġof Ġpredicting Ġjudgments Ġof Ġcivil Ġcases . ĠComp aring Ġwith Ġconventional Ġtext Ġclassification Ġframework , Ġwe Ġpropose ĠLegal ĠReading ĠComp rehens ion Ġframework Ġto Ġhandle Ġmultiple Ġand Ġcomplex Ġtextual Ġinputs . ĠMoreover , Ġwe Ġpresent Ġa Ġnovel Ġneural Ġmodel , ĠAuto Judge , Ġto Ġincorporate Ġlaw Ġarticles Ġfor Ġjudgment Ġprediction . ĠIn Ġexperiments , Ġwe Ġcompare Ġour Ġmodel Ġon Ġdivorce Ġproceedings Ġwith Ġvarious Ġstate - of - the - art Ġbas elines Ġof Ġvarious Ġframeworks . ĠExperimental Ġresults Ġshow Ġthat Ġour Ġmodel Ġachieves Ġconsiderable Ġimprovement Ġthan Ġall Ġthe Ġbas elines . ĠBesides , Ġvisualization Ġresults Ġalso Ġdemonstrate Ġthe Ġeffectiveness Ġand Ġinterpret ability Ġof Ġour Ġproposed Ġmodel . </s> In Ġthe Ġfuture , Ġwe Ġcan Ġexplore Ġthe Ġfollowing Ġdirections : Ġ( 1 ) ĠLimited Ġby Ġthe Ġdatasets , Ġwe Ġcan Ġonly Ġverify Ġour Ġproposed Ġmodel Ġon Ġdivorce Ġproceedings . ĠA Ġmore Ġgeneral Ġand Ġlarger Ġdataset Ġwill Ġbenefit Ġthe Ġresearch Ġon Ġjudgment Ġprediction . Ġ( 2 ) ĠJudicial Ġdecisions Ġin Ġsome Ġcivil Ġcases Ġare Ġnot Ġalways Ġbinary , Ġbut Ġmore Ġdiverse Ġand Ġflexible Ġones , Ġe . g . Ġcompensation Ġamount . ĠThus , Ġit Ġis Ġcritical Ġfor Ġjudgment Ġprediction Ġto Ġmanage Ġvarious Ġjudgment Ġforms .