Segmentation quality is evaluated by calculating the precision, recall, and F-score of the automatic segmentations in comparison to the segmentations made by expert annotators from the ANNODIS subcorpus.
ABSTRACTIVE
we compare the Annodis segmentation with the automatically produced segmentation
EXTRACTIVE
Two batch of tests were performed. The first on the $D$ set of documents common to the two subcorpus “specialist” $E$ and “naive” $N$ from Annodis. $D$ contains 38 documents with 13 364 words. This first test allowed to measure the distance between the human markers. In fact, in order to get an idea of the quality of the human segmentations, the cuts in the texts made by the specialists were measured it versus the so-called “naifs” note takers and vice versa. The second series of tests consisted of using all the documents of the subcorpus “specialist” $E$, because the documents of the subcorpus of Annodis are not identical. Then we benchmarked the performance of the three systems automatically. We have found that segmentation by experts and naive produces two subcorpus $E$ and $N$ with very similar characteristics. This surprised us, as we expected a more important difference between them. In any case, we deduced that, at least in this corpus, it is not necessary to be an expert in linguistics to discursively segment the documents. As far as system evaluations are concerned, we use the 78 $E$ documents as reference. Table TABREF26 shows the results. We calculate the precision $P$, the recall $R$ and the $F$-score on the text corpus used in our tests, as follow:
In this first exploratory work, only documents in French were considered, but the system can be adapted to other languages. The evaluation is based on the correspondence of word pairs representing a border. In this way we compare the Annodis segmentation with the automatically produced segmentation. For each pair of reference segments, a $L_r$ list of word pairs is provided: the last word of the first segment and the first word of the second.
2002.04095
Introduction </s> R het orical ĠStructure ĠTheory Ġ( R ST ) ĠB IB REF 0 Ġis Ġa Ġtechnique Ġof ĠNatural ĠLanguage ĠProcessing Ġ( N LP ), Ġin Ġwhich Ġa Ġdocument Ġcan Ġbe Ġstructured Ġhierarch ically Ġaccording Ġto Ġits Ġdiscourse . ĠThe Ġgenerated Ġhierarchy , Ġa Ġtree , Ġprovides Ġinformation Ġassociated Ġwith Ġthe Ġboundaries Ġof Ġthe Ġdiscourse Ġsegments Ġand Ġrelated Ġto Ġtheir Ġimportance Ġand Ġdependencies . ĠThe Ġfigure ĠFIG REF 1 Ġshows Ġan Ġexample Ġof Ġsuch Ġa Ġre th orical Ġtree . ĠIn Ġthe Ġre th orical Ġparsing Ġprocess , Ġthe Ġtext Ġhas Ġbeen Ġdivided Ġinto Ġfive Ġunits . ĠIn Ġthe Ġfigure ĠFIG REF 1 , Ġthe Ġarrow Ġthat Ġleaves Ġthe Ġunit Ġ( 2 ) Ġtowards Ġthe Ġunit Ġ( 1 ) Ġsymbol izes Ġthat Ġthe Ġunit Ġ( 2 ) Ġis Ġthe Ġsatellite Ġof Ġthe Ġunit Ġ( 1 ), Ġwhich Ġis Ġthe Ġcore Ġin Ġa ĠâĢ ľ Con cession âĢ Ŀ Ġrelationship . ĠIn Ġturn , Ġthe Ġunits Ġ( 1 ) Ġand Ġ( 2 ) Ġcomprise Ġthe Ġnucleus Ġof Ġthree ĠâĢ ľ Demon stration âĢ Ŀ Ġrelationships . </s> The Ġdisc ursive Ġanalysis Ġof Ġa Ġdocument Ġnormally Ġincludes Ġthree Ġconsecutive Ġsteps : Ġ1 ) Ġdisc ursive Ġsegment ation ; Ġ2 ) Ġdetection Ġof Ġthe Ġdisc ursive Ġrelations ; Ġ3 ) Ġconstruction Ġof Ġthe Ġhierarchical Ġrhetorical Ġtree . ĠRegarding Ġthe Ġdisc ursive Ġsegment ation , Ġthere Ġare Ġsegment ers Ġin Ġseveral Ġlanguages . ĠHowever , Ġeach Ġpiece Ġdepends Ġon Ġso f istic ated Ġlinguistic Ġresources , Ġwhich Ġcompl icates Ġthe Ġreproduction Ġof Ġthe Ġexperiments Ġin Ġother Ġlanguages . ĠConsequently , Ġthe Ġdevelopment Ġof Ġmult ilingual Ġsystems Ġusing Ġdisc ursive Ġanalysis Ġare Ġyet Ġto Ġbe Ġdeveloped . ĠD iverse Ġapplications Ġbased Ġon Ġthe Ġlatest Ġtechnologies Ġrequire Ġat Ġleast Ġone Ġof Ġthe Ġthree Ġsteps Ġmentioned Ġabove ĠB IB REF 1 , ĠB IB REF 2 , ĠB IB REF 3 . ĠIn Ġthis Ġcontext , Ġthe Ġidea Ġof Ġexploring Ġthe Ġarchitecture Ġof Ġa Ġgeneric Ġsystem Ġthat Ġis Ġable Ġnot Ġonly Ġof Ġsegment ing Ġa Ġtext Ġcorrectly Ġbut Ġalso Ġof Ġadapting Ġit Ġto Ġany Ġlanguage , Ġwas Ġa Ġgreat Ġmotivation Ġof Ġthis Ġresearch Ġwork . </s> In Ġthis Ġarticle Ġwe Ġshow Ġthe Ġpreliminary Ġresults Ġof Ġa Ġgeneric Ġsegment er Ġcomposed Ġof Ġseveral Ġsystems Ġ( different Ġsegment ation Ġstrategies ). ĠIn Ġaddition , Ġwe Ġdescribe Ġan Ġautomatic Ġevaluation Ġprotocol Ġof Ġdisc ursive Ġsegment ation . ĠThe Ġarticle Ġis Ġcomposed Ġby Ġthe Ġfollowing Ġsections : Ġstate Ġof Ġthe Ġart Ġ( SEC REF 2 ), Ġwhich Ġpresents Ġa Ġbrief Ġb ibli ographic Ġreview ; ĠDescription Ġof Ġthe ĠAnn od is Ġ( SEC REF 3 ) Ġcorpus Ġused Ġin Ġour Ġtests Ġand Ġof Ġthe Ġgeneral Ġarchitecture Ġof Ġthe Ġproposed Ġsystems Ġ( SEC REF 4 ); ĠSe gment ation Ġstrategies Ġ( SEC REF 5 ), Ġwhich Ġcharacter izes Ġthe Ġdifferent Ġmethods Ġimplemented Ġto Ġsegment Ġthe Ġtext ; Ġresults Ġof Ġour Ġnumerical Ġexperiments Ġ( sec : exper iments ); Ġand Ġwe Ġconclude Ġwith Ġour Ġconclusions Ġand Ġperspectives Ġ( SEC REF 7 ). </s> State - of - the - art </s> In ĠR ST , Ġthere Ġare Ġtow Ġdisc ursive Ġunits : Ġnucle i Ġand Ġsatellites . ĠThe Ġnucleus Ġprovide Ġinformation Ġpertinent Ġto Ġthe Ġpurposes Ġof Ġthe Ġauthor Ġof Ġthe Ġtext Ġand Ġthe Ġsatellites Ġadd Ġadditional Ġinformation Ġto Ġthe Ġnucle u , Ġon Ġwhich Ġthey Ġare Ġdependent Ġon . ĠIn Ġthe Ġcontext Ġof ĠR ST , Ġpossible Ġdisc ursive Ġrelationships Ġmay Ġbe Ġnucleus - s atellite Ġand Ġmult in uclear . ĠIn Ġnucleus - s atellite Ġrelationships , Ġa Ġsatellite Ġdepends Ġon Ġone Ġnucleus , Ġwhereas Ġin Ġmult in uclear Ġrelationships , Ġseveral Ġnucle i Ġ( at Ġleast Ġtwo ) Ġare Ġreg roup ed Ġat Ġthe Ġsame Ġlevel Ġof Ġimportance Ġ( tree Ġhierarchy ). ĠThus , Ġin Ġthe Ġdisc ursive Ġsegment ation Ġproposes Ġto Ġreduce Ġthe Ġtext Ġinto Ġthe Ġminimal Ġdisc ursive Ġunits Ġcalled ĠElementary ĠDisc ursive ĠUnits Ġ( ED U ), Ġthrough Ġthe Ġuse Ġof Ġexplicit Ġdisc ursive Ġmarkers . ĠAs Ġan Ġexample , Ġwe Ġcan Ġquote Ġsome Ġmarkers Ġin ĠFrench : </s> af in Ġde , Ġpour Ġque , Ġdon c , Ġqu and Ġb ien Ġm Ãª me Ġque , Ġensu ite , Ġde Ġf ois Ġque , Ġglob at iv ament e , Ġpar Ġcont re , Ġsin on , ĠÃł Ġce Ġmoment - l Ãł , Ġc epend ant , Ġsubs Ã© qu em ment , Ġpu is que , Ġau Ġfur Ġet ĠÃł Ġmes ure Ġque , Ġsi , Ġfinal ement , Ġetc .. </s> Mark ers Ġor Ġparticles Ġare Ġoften Ġused Ġto Ġconnect Ġideas . ĠLet 's Ġconsider Ġthe Ġsentence Ġbelow : </s> La Ġv ille Ġd ' Av ign on Ġest Ġla Ġcapit ale Ġdu ĠV au cl use , Ġqui Ġest Ġun ĠdÃ© part ement Ġdu Ġsu d Ġde Ġla ĠFrance . </s> qu i Ġ( which ) Ġis Ġa Ġdisc ursive Ġmarker Ġbecause Ġit Ġconnects Ġtwo Ġideas . ĠThe Ġfirst Ġone , ĠâĢ ľ Av ign on ĠCity Ġis Ġthe Ġcapital Ġof ĠV au cl use âĢ Ŀ Ġ( La Ġv ille Ġd ' Av ign on Ġest Ġla Ġcapit ale Ġdu ĠV au cl use ), Ġand Ġthe Ġsecond Ġone Ġ( s atellite ), ĠâĢ ľ [ V au cl use ] Ġis Ġa Ġdepartment Ġin Ġthe Ġsouth Ġof ĠFrance âĢ Ŀ Ġ([ V au cl use ] Ġest Ġun ĠdÃ© part ement Ġdu Ġsu d Ġde Ġla ĠFrance ). ĠSeveral Ġresearch Ġhas Ġaddressed Ġautomatic Ġsegment ation Ġin Ġseveral Ġlanguages , Ġsuch Ġas : ĠFrench ĠB IB REF 4 , ĠEnglish ĠB IB REF 5 , ĠPortuguese ĠB IB REF 6 , ĠSpanish ĠB IB REF 7 , ĠB IB REF 8 Ġand ĠTah i . ĠB IB REF 9 . ĠAll Ġconverge Ġto Ġthe Ġidea Ġof Ġusing Ġan Ġexplicit Ġlist Ġof Ġmarks Ġin Ġorder Ġto Ġsegment Ġtexts . </s> Ann od is ĠCorpus </s> In Ġthis Ġfirst Ġexplor atory Ġwork , Ġour Ġtests Ġconsidered Ġonly Ġdocuments Ġin ĠFrench Ġfrom Ġthe ĠAnn od is Ġcorpus . ĠAnn od is Ġ( AN NO t ation ĠDIS c ursive ) Ġis Ġa Ġset Ġof Ġdocuments Ġin ĠFrench Ġthat Ġwere Ġmanually Ġenriched Ġwith Ġnotes Ġof Ġdisc ursive Ġstructures . ĠIts Ġmain Ġcharacteristics Ġare : </s> Two Ġannotations : ĠR het orical Ġrelations Ġand Ġmult ile vel Ġstructures . </s> Documents Ġ( 687 Ġ000 Ġwords ) Ġtaken Ġfrom Ġfour Ġsources : Ġthe ĠEst ĠRÃ© public ain Ġnewspaper Ġ( 39 Ġarticles , Ġ10 Ġ000 Ġwords ); ĠWikipedia Ġ( 30 Ġarticles Ġ+ Ġ30 Ġsumm aries , Ġ242 Ġ000 Ġwords ); ĠProceedings Ġof Ġthe Ġconference ĠTra itement ĠAutom at ique Ġdes ĠLang ues ĠNature ll es Ġ( T AL N ) Ġ2008 Ġ( 25 Ġarticles , Ġ169 Ġ000 Ġwords ); ĠReports Ġfrom ĠInstit ut ĠFranÃ§ ais Ġde ĠRelations ĠInternational es Ġ( 32 Ġrap orts , Ġ266 Ġ000 Ġwords ). </s> The Ġcorpor a Ġwere Ġnoted Ġusing ĠGl o zz . </s> Ann od is Ġaims Ġat Ġbuilding Ġan Ġannot ated Ġcorpus . ĠThe Ġproposed Ġannotations Ġare Ġon Ġtwo Ġlevels Ġof Ġanalysis , Ġthat Ġis , Ġtwo Ġperspectives : </s> A sc endant : Ġpart Ġof ĠED U Ġare Ġused Ġin Ġthe Ġconstruction Ġof Ġmore Ġcomplex Ġstructures , Ġthrough Ġthe Ġrelations Ġof Ġdiscourse ; </s> Desc ending : Ġapproaches Ġthe Ġtext Ġin Ġits Ġentirety Ġand Ġrelies Ġon Ġthe Ġvarious Ġshallow Ġindices Ġto Ġidentify Ġhigh - level Ġdisc ursive Ġstructures Ġ( mac ro Ġstructures ). </s> Two Ġtypes Ġof Ġpersons Ġannot ated ĠAnn od is : Ġlinguistic Ġexperts Ġand Ġstudents . ĠThe Ġfirst Ġgroup Ġconstituted Ġa Ġ$ E $ Ġsub cor p us Ġcalled ĠâĢ ľ special ist âĢ Ŀ Ġand Ġthe Ġsecond Ġgroup Ġresulted Ġin Ġa Ġ$ N $ Ġsub cor p us Ġcalled ĠâĢ ľ na ive âĢ Ŀ . ĠThese Ġrhet orically Ġannot ated Ġsub cor ps Ġwere Ġused Ġas Ġreferences Ġin Ġour Ġexperiments . Ġ( c . f . ĠÂ§ SEC REF 6 ). </s> Disc ourse ĠSe gment er ĠOverall ĠDescription </s> The ĠFigure ĠFIG REF 12 Ġshows Ġthe Ġgeneral Ġarchitecture Ġof Ġthe Ġproposed Ġdiscourse Ġsegment er Ġsystem . ĠThe Ġinitial Ġinput Ġis Ġthe Ġraw Ġtext Ġencoded Ġin ĠUTF - 8 . ĠThe Ġtwo Ġinitial Ġprocesses Ġare ĠPart Ġof ĠSpeech Ġmorph os ynt actic ĠT agging Ġ( POS ) Ġand Ġthe Ġsegment ation Ġat Ġthe Ġlevel Ġof Ġthe Ġsentences . ĠThis Ġlast Ġis Ġjust Ġa Ġpre processing Ġstep Ġthat Ġsplits Ġsentences . ĠIn Ġthe Ġlast Ġprocess Ġthe Ġsystem Ġuses Ġa Ġbank Ġof Ġexplicit Ġmarkers Ġin Ġro der Ġto Ġapply Ġthe Ġrules Ġfor Ġthe Ġfinal Ġdiscourse Ġsegment ation . </s> For Ġthe Ġexperiments , Ġwe Ġused Ġlists Ġof Ġmarkers Ġin ĠFrench , ĠSpanish , ĠEnglish Ġand ĠPortuguese . ĠWe Ġalso Ġused Ġthe ĠLex icon n ĠB IB REF 10 Ġproject Ġlist , Ġwhich Ġreg roups Ġ328 ĠFrench - language Ġmarkers . ĠAnother Ġimportant Ġparameter Ġspecifies Ġwhich Ġsegment ation Ġstrategy Ġshould Ġbe Ġapplied , Ġaccording Ġto Ġthe ĠPOS Ġlab elling Ġof Ġthe Ġdocument . </s> Description Ġof Ġsegment ation Ġstrategies Ġ: :: ĠSe gment ation Ġwith Ġexplicit Ġuse Ġof Ġa Ġmarker </s> The Ġelementary Ġsystem ĠSe gment er $ _{ \ mu Ġ} $ Ġ( bas eline ) Ġrelies Ġsolely Ġon Ġa Ġlist Ġof Ġdisc ursive Ġmarkers Ġto Ġperform Ġthe Ġsegment ation . ĠIt Ġreplaces Ġthe Ġappearance Ġof Ġa Ġmarker Ġin Ġthe Ġlist Ġwith Ġa Ġspecial Ġsymbol , Ġfor Ġexample Ġ$\ mu Ġ$ , Ġwhich Ġindicates Ġa Ġboundary Ġbetween Ġthe Ġright Ġand Ġleft Ġsegment . ĠBe Ġthe Ġsentence Ġof Ġthe Ġpreceding Ġexample : ĠLa Ġv ille Ġd ' Av ign on Ġest Ġla Ġcapit ale Ġdu ĠV au cl use , Ġqui Ġest Ġun ĠdÃ© part ement Ġdu ĠSud Ġde Ġla ĠFrance .. ĠThe ĠSe gment er Ġsplit Ġthe Ġsentence Ġin Ġtwo Ġparts : Ġthe Ġleft Ġsegment Ġ( SE ), ĠLa Ġv ille Ġd ' Av ign on Ġest Ġla Ġcapit ale Ġdu ĠV au cl use , Ġand Ġthe Ġright Ġsegment Ġ( SD ), Ġest Ġun ĠdÃ© part ement Ġdu Ġsu d Ġde Ġla ĠFrance . </s> Description Ġof Ġsegment ation Ġstrategies Ġ: :: ĠSe gment ation Ġwith Ġexplicit Ġuse Ġof Ġa Ġmarker Ġand ĠPOS Ġlabels </s> The ĠSe gment er $ _{ \ mu Ġ+ } $ Ġsystem Ġpresents Ġan Ġimprovement Ġto Ġthe ĠSe gment er $ _{ \ mu Ġ} $ : Ġinclusion Ġof Ġgrammar Ġcategories Ġwith Ġthe ĠTree T agger Ġtool . ĠThe Ġadvantage Ġof Ġthis Ġsystem Ġis Ġthe Ġdetection Ġof Ġcertain Ġgram matical Ġforms Ġin Ġorder Ġto Ġcondition Ġthe Ġsegment ation . ĠSince Ġit Ġis Ġbased Ġon Ġthe ĠSe gment er $ _{ \ mu Ġ} $, Ġwe Ġtry Ġto Ġrecognise Ġthe Ġopp ort une Ġconditions Ġto Ġgather Ġtwo Ġsegments Ġwhen Ġboth Ġare Ġpart Ġof Ġthe Ġsame Ġdisc ursive Ġsegment . ĠWe Ġtry Ġto Ġidentify Ġmore Ġsubtly Ġwhen Ġit Ġis Ġpertinent Ġto Ġleave Ġthe Ġtwo Ġsegments Ġseparate . ĠThe ĠSe gment er $ _{ \ mu Ġ} $ Ġhas Ġtwo Ġdistinct Ġstrategies : </s> Seg ment ador $ _{ \ mu Ġ+ V } $ Ġ( verbal Ġversion , ĠV ): Ġit Ġrelies Ġsolely Ġon Ġthe Ġpresence Ġof Ġverbal Ġforms Ġto Ġthe Ġright Ġand Ġleft Ġof Ġthe Ġdisc ursive Ġmarker . ĠThe Ġtwo Ġgram matical Ġrules Ġof Ġthis Ġstrategy Ġare : </s> If Ġthere Ġare Ġno Ġverbs Ġin Ġthe Ġleft Ġand Ġright Ġsegments , Ġreg roup Ġthem . </s> If Ġthere Ġis Ġat Ġleast Ġone Ġverb Ġin Ġthe Ġleft Ġor Ġright Ġsegment , Ġthe Ġsegments Ġwill Ġremain Ġseparate . </s> Seg ment er $ _{ \ mu Ġ+ ( V - N )} $ Ġ( verb - n oun Ġversion , ĠV - N ): Ġit Ġrelies Ġon Ġthe Ġpresence Ġof Ġverbs Ġand Ġnoun s . ĠFor Ġthis Ġversion , Ġfour Ġrules Ġare Ġconsidered : </s> If Ġthere Ġis Ġno Ġnoun Ġin Ġeither Ġthe Ġleft Ġor Ġright Ġsegment , Ġwe Ġreg roup Ġthe Ġsegments . </s> We Ġreg roup Ġthe Ġsegments Ġif Ġat Ġleast Ġone Ġof Ġthem Ġhas Ġno Ġnoun . </s> If Ġat Ġleast Ġone Ġnoun Ġis Ġpresent Ġin Ġboth Ġsegments , Ġthey Ġremain Ġindependent . </s> If Ġthere Ġis Ġno Ġverb - nom inal Ġform , Ġthe Ġsegments Ġremain Ġindependent . </s> Exper iments </s> In Ġthis Ġfirst Ġexplor atory Ġwork , Ġonly Ġdocuments Ġin ĠFrench Ġwere Ġconsidered , Ġbut Ġthe Ġsystem Ġcan Ġbe Ġadapted Ġto Ġother Ġlanguages . ĠThe Ġevaluation Ġis Ġbased Ġon Ġthe Ġcorrespondence Ġof Ġword Ġpairs Ġrepresenting Ġa Ġborder . ĠIn Ġthis Ġway Ġwe Ġcompare Ġthe ĠAnn od is Ġsegment ation Ġwith Ġthe Ġautomatically Ġproduced Ġsegment ation . ĠFor Ġeach Ġpair Ġof Ġreference Ġsegments , Ġa Ġ$ L _ r $ Ġlist Ġof Ġword Ġpairs Ġis Ġprovided : Ġthe Ġlast Ġword Ġof Ġthe Ġfirst Ġsegment Ġand Ġthe Ġfirst Ġword Ġof Ġthe Ġsecond . </s> For Ġexample , Ġconsidering Ġthe Ġreference Ġtext Ġwik 1 _ 01 _ 02 - 04 - 2006 . se g , Ġfrom ĠAnn od is Ġcorpus : </s> [ Le ĠBan ĠAmendment ] $ _ 1 $ Ġ[ Apr Ã¨ s Ġav oir Ġadopt Ã© Ġla ĠConvention , ] _ 2 Ġ[ un Ġcertain Ġn omb re Ġde ĠP ED Ġet Ġd ' ass oci ations Ġde ĠdÃ© f ense Ġde Ġl ' en viron n ement Ġs out in rent ] _ 3 Ġ[ que Ġle Ġdocument Ġn ' all ait Ġpas Ġas se z Ġlo in .] _ 4 Ġ[ De Ġn omb re ux Ġpays Ġet ĠON G Ġmilit Ã¨ rent ] _ 5 Ġ[ en Ġf ave ur Ġd ' une Ġinter d iction Ġtot ale Ġde Ġl ' exp Ã© d ition Ġde ĠdÃ© che ts Ġd ange re ux ĠÃł Ġdestinations Ġdes ĠP ED .] _ 6 Ġ[ Plus Ġexact ement , ] _ 7 Ġ[ la ĠConvention Ġorigin ale Ġn ' inter dis ait Ġpas Ġl ' ex portation Ġde ĠdÃ© che ts , ] _ 8 Ġ[ except Ã© Ġvers Ġl ' Ant ar ct ique .] _ 9 Ġ[ El le Ġn ' ex ige ait ] _ 10 Ġ[ qu ' une Ġproc Ã© d ure Ġde Ġconsent ement Ġpr Ã© al able Ġen Ġcon naissance Ġde Ġcause ] _ 11 Ġ[( P IC , ĠPrior ĠIn formed ĠConsent ). ] _ 12 </s> Here Ġare Ġthe Ġword Ġpairs Ġof Ġthe Ġcreated Ġreference Ġlist Ġ( p unct uation Ġmarks Ġare Ġdisreg arded ): </s> $ L _ r $ ={ [ Con vention ĠâĢĵ Ġun ], Ġ[ s out in rent ĠâĢĵ Ġque ], Ġ[ lo in ĠâĢĵ Ġde ], Ġ[ m ilit Ã¨ rent ĠâĢĵ Ġen ], Ġ[ ex act ement ĠâĢĵ Ġla ], Ġ[ P ED ĠâĢĵ Ġplus ], Ġ[ ex act ement ĠâĢĵ Ġla ], Ġ[ d Ã© che ts ĠâĢĵ Ġexcept Ã© ], Ġ[ Ant art ique ĠâĢĵ ĠEl le ], Ġ[ ex ige ait ĠâĢĵ Ġqu ' une ], Ġ[ cause ĠâĢĵ ĠP IC ] Ġ} </s> We Ġdecided Ġto Ġcount Ġthe Ġword Ġpairs Ġinstead Ġof Ġthe Ġsegments , Ġas Ġthis Ġis Ġa Ġfirst Ġversion Ġof Ġthe Ġevaluation Ġprotocol . ĠIn Ġfact , Ġthe Ġsegments Ġmay Ġbe Ġnested , Ġwhich Ġcompl icates Ġthe Ġevaluation Ġprocess . ĠAlthough Ġthere Ġare Ġsome Ġerrors , Ġword Ġboundaries Ġallow Ġus Ġto Ġdetect Ġsegments Ġmore Ġeasily . </s> We Ġhave Ġbuilt Ġa Ġsecond Ġ$ L _ c $ Ġlist Ġfor Ġthe Ġautomatically Ġidentified Ġsegments , Ġfollowing Ġthe Ġsame Ġcriteria Ġof Ġ$ L _ r $. ĠThe Ġ$ L _ r $ Ġand Ġ$ L _ c $ Ġlists Ġreg roup , Ġpair Ġby Ġpair , Ġthe Ġsegment Ġborder . ĠWe Ġthen Ġcount Ġthe Ġcommon Ġpair Ġintersection Ġof Ġthe Ġtwo Ġlists . ĠEach Ġpair Ġin Ġthe Ġ$ L _ c $ Ġlist Ġis Ġalso Ġpresent Ġin Ġthe Ġ$ L _ r $ Ġreference Ġlist Ġand Ġis Ġa Ġcorrectly Ġassigned Ġto Ġthe Ġclass Ġpair . ĠA Ġword Ġpair Ġbelonging Ġto Ġthe Ġ$ L _ c $ Ġlist Ġbut Ġnot Ġbelonging Ġto Ġthe Ġ$ L _ r $ Ġreference Ġlist , Ġwill Ġbe Ġa Ġpair Ġassigned Ġto Ġthe Ġclass . </s> For Ġthat Ġsame Ġtext , Ġthe Ġ$ L _ c $ Ġlist Ġof Ġcandidate Ġpairs Ġobtained Ġwith Ġthe ĠSe gment ator $ _{ \ mu Ġ} $ Ġis : </s> $ L _ r $ ={ [ lo in âĢĵ De ], Ġ[ p ays âĢĵ et ], Ġ[ m ilit Ã¨ rent âĢĵ en ], Ġ[ d ange re ux âĢĵ Ãł ], Ġ[ P ED âĢĵ Plus ], Ġ[ Ant ar ct ique âĢĵ El le ], Ġ[ pr Ã© al able âĢĵ en ], Ġ[ cause âĢĵ P IC ] Ġ} </s> We Ġcalculate Ġthe Ġprecision Ġ$ P $, Ġthe Ġrecall Ġ$ R $ Ġand Ġthe Ġ$ F $ - score Ġon Ġthe Ġtext Ġcorpus Ġused Ġin Ġour Ġtests , Ġas Ġfollow : </s> The Ġprecision , Ġthe Ġrecall Ġand Ġthe Ġ$ F $ - score Ġfor Ġthis Ġexample Ġis : Ġ$ P $ Ġ= Ġ5 Ġ/ Ġ11 Ġ= Ġ0 . 45 ; Ġ$ R $ Ġ= Ġ5 Ġ/ Ġ8 Ġ= Ġ0 . 625 ; ĠF - score Ġ= Ġ2 Ġ$\ times Ġ\ frac { Ġ0 . 45 Ġ\ times Ġ0 . 625 }{ Ġ0 . 45 Ġ+ Ġ0 . 625 } Ġ= Ġ0 . 523 $. ĠWe Ġused Ġthe Ġdocuments Ġin Ġthe ĠAnn od is Ġcorpus Ġwithout Ġsegment ation , Ġbecause Ġthey Ġhad Ġbeen Ġsegment ed Ġwith Ġthe ĠSe gment er $ _{ \ mu Ġ} $ Ġand Ġwith Ġthe Ġgrammar Ġsegment ers . </s> Two Ġbatch Ġof Ġtests Ġwere Ġperformed . ĠThe Ġfirst Ġon Ġthe Ġ$ D $ Ġset Ġof Ġdocuments Ġcommon Ġto Ġthe Ġtwo Ġsub cor p us ĠâĢ ľ special ist âĢ Ŀ Ġ$ E $ Ġand ĠâĢ ľ na ive âĢ Ŀ Ġ$ N $ Ġfrom ĠAnn od is . Ġ$ D $ Ġcontains Ġ38 Ġdocuments Ġwith Ġ13 Ġ364 Ġwords . ĠThis Ġfirst Ġtest Ġallowed Ġto Ġmeasure Ġthe Ġdistance Ġbetween Ġthe Ġhuman Ġmarkers . ĠIn Ġfact , Ġin Ġorder Ġto Ġget Ġan Ġidea Ġof Ġthe Ġquality Ġof Ġthe Ġhuman Ġsegment ations , Ġthe Ġcuts Ġin Ġthe Ġtexts Ġmade Ġby Ġthe Ġspecialists Ġwere Ġmeasured Ġit Ġversus Ġthe Ġso - called ĠâĢ ľ na if s âĢ Ŀ Ġnote Ġt akers Ġand Ġvice Ġversa . ĠThe Ġsecond Ġseries Ġof Ġtests Ġconsisted Ġof Ġusing Ġall Ġthe Ġdocuments Ġof Ġthe Ġsub cor p us ĠâĢ ľ special ist âĢ Ŀ Ġ$ E $, Ġbecause Ġthe Ġdocuments Ġof Ġthe Ġsub cor p us Ġof ĠAnn od is Ġare Ġnot Ġidentical . ĠThen Ġwe Ġbenchmark ed Ġthe Ġperformance Ġof Ġthe Ġthree Ġsystems Ġautomatically . </s> Exper iments Ġ: :: ĠResults </s> In Ġthis Ġsection Ġwe Ġwill Ġcompare Ġthe Ġresults Ġof Ġthe Ġdifferent Ġsegment ation Ġsystems Ġthrough Ġautomatic Ġevaluations . ĠFirst Ġof Ġall , Ġthe Ġhuman Ġsegment ation , Ġfrom Ġthe Ġsub cor p us Ġ$ D $ Ġcomposed Ġof Ġcommon Ġdocuments . ĠThe Ġresults Ġare Ġpresented Ġin Ġthe Ġtable Ġtab : hum ains . ĠThe Ġfirst Ġrow Ġshows Ġthe Ġperformance Ġof Ġthe Ġ$ I $ Ġsegments , Ġtaking Ġthe Ġexperts Ġas Ġa Ġreference , Ġwhile Ġthe Ġsecond Ġpresents Ġthe Ġprocess Ġin Ġthe Ġopposite Ġdirection . </s> We Ġhave Ġfound Ġthat Ġsegment ation Ġby Ġexperts Ġand Ġnaive Ġproduces Ġtwo Ġsub cor p us Ġ$ E $ Ġand Ġ$ N $ Ġwith Ġvery Ġsimilar Ġcharacteristics . ĠThis Ġsurprised Ġus , Ġas Ġwe Ġexpected Ġa Ġmore Ġimportant Ġdifference Ġbetween Ġthem . ĠIn Ġany Ġcase , Ġwe Ġded uced Ġthat , Ġat Ġleast Ġin Ġthis Ġcorpus , Ġit Ġis Ġnot Ġnecessary Ġto Ġbe Ġan Ġexpert Ġin Ġlingu istics Ġto Ġdisc urs ively Ġsegment Ġthe Ġdocuments . ĠAs Ġfar Ġas Ġsystem Ġevaluations Ġare Ġconcerned , Ġwe Ġuse Ġthe Ġ78 Ġ$ E $ Ġdocuments Ġas Ġreference . ĠTable ĠT AB REF 26 Ġshows Ġthe Ġresults . </s> In Ġthe Ġcase Ġof Ġthe ĠExperts , Ġthe Ġgram matical Ġverb - nom inal Ġversion Ġ( V - N ) Ġhad Ġbetter ĠF - score Ġperformance . ĠThe Ġverbal Ġversion Ġ( V ) Ġobtained Ġa Ġbetter Ġaccuracy Ġ$ P $ Ġthan Ġthe Ġverb - nom inal Ġ( V - N ). ĠIn Ġthe Ġcase Ġof Ġthe ĠNa ive , Ġthe Ġperformance ĠF - score , Ġ$ P $ Ġand Ġ$ R $ Ġis Ġvery Ġsimilar Ġfrom Ġthe ĠExperts . </s> Con clusions , Ġdiscussion Ġand Ġperspectives </s> The Ġaim Ġof Ġthis Ġwork Ġwas Ġtwo fold : Ġto Ġdesign Ġa Ġdisc ursive Ġsegment er Ġusing Ġa Ġminimum Ġof Ġresources Ġand Ġto Ġestablish Ġan Ġevaluation Ġprotocol Ġto Ġmeasure Ġthe Ġperformance Ġof Ġsegment ers . ĠThe Ġresults Ġshow Ġthat Ġwe Ġcan Ġbuild Ġa Ġsimple Ġversion Ġof Ġthe Ġbaseline , Ġwhich Ġemploys Ġonly Ġa Ġlist Ġof Ġmarkers Ġand Ġpresents Ġa Ġvery Ġencouraging Ġperformance . ĠOf Ġcourse , Ġthe Ġquality Ġof Ġthe Ġlist Ġis Ġa Ġprep onder ant Ġfactor Ġfor Ġa Ġcorrect Ġsegment ation . </s> We Ġhave Ġstudied Ġthe Ġimpact Ġof Ġthe Ġmarker Ġwhich , Ġeven Ġthough Ġit Ġmay Ġseem Ġfringe - worthy , Ġcontributes Ġto Ġimproving Ġthe Ġperformance Ġof Ġour Ġsegment ers . ĠThus , Ġit Ġis Ġan Ġinteresting Ġmarker Ġthat Ġwe Ġcan Ġconsider Ġas Ġa Ġdisc ursive Ġmarker . ĠThe ĠSe gment ator $ _{ \ mu Ġ} $ Ġversion Ġprovides Ġthe Ġbest Ġresults Ġin Ġterms Ġof ĠF - score Ġand Ġrecall , Ġfollowed Ġby Ġthe ĠSe gment ator $ _{ \ mu Ġ+ V } $ Ġversion , Ġwhich Ġpasses Ġit Ġin Ġprecision . ĠRegarding Ġevaluation , Ġwe Ġdeveloped Ġa Ġsimple Ġprotocol Ġto Ġcompare Ġthe Ġperformance Ġof Ġthe Ġsystems . ĠThis Ġis , Ġto Ġour Ġknowledge , Ġthe Ġfirst Ġautomatic Ġevaluation Ġin ĠFrench . </s> It Ġis Ġnecessary Ġto Ġintensify Ġour Ġresearch Ġin Ġorder Ġto Ġpropose Ġimprovements Ġto Ġour Ġsegment ers , Ġas Ġwell Ġas Ġto Ġstudy Ġfurther Ġthe Ġimpact Ġof Ġgrammar Ġtag Ġrules Ġon Ġsegment ation . ĠSince Ġwe Ġhave Ġa Ġstandard Ġevaluation Ġprotocol , Ġwe Ġintend Ġto Ġcarry Ġout Ġtests Ġwith ĠPortuguese , ĠSpanish Ġ( see ĠB IB REF 11 ), ĠEnglish , Ġetc . ĠFor Ġthat , Ġwe Ġwill Ġonly Ġneed Ġa Ġlist Ġof Ġmarkers Ġfor Ġeach Ġlanguage . </s> The Ġperformance Ġof Ġthe Ġsystems Ġremains Ġmodest , Ġof Ġcourse , Ġbut Ġwe Ġmust Ġnot Ġforget Ġthat Ġthis Ġis Ġa Ġbaseline Ġand Ġits Ġprimary Ġobjective Ġis Ġto Ġprovide Ġstandard Ġsystems Ġthat Ġcan Ġbe Ġused Ġin Ġtesting Ġprotocols Ġsuch Ġas Ġthe Ġone Ġwe Ġproposed . ĠDespite Ġthis Ġevolution , Ġthese Ġbas elines Ġ( or Ġtheir Ġimproved Ġversions ) Ġcan Ġbe Ġused Ġin Ġapplications Ġsuch Ġas Ġautomatic Ġdocument Ġsummar isation Ġ( e . g ., ĠB IB REF 12 , ĠB IB REF 13 ), Ġor Ġsentences Ġcompression ĠB IB REF 14 . </s> The Ġmain Ġfeature Ġof Ġthe Ġproposed Ġbaseline Ġsystem Ġis Ġits Ġflexibility Ġwith Ġrespect Ġto Ġthe Ġlanguage Ġconsidered . ĠIn Ġfact , Ġit Ġonly Ġuses Ġa Ġlist Ġof Ġlanguage Ġmarkers Ġand Ġthe Ġgram matical Ġcategory Ġof Ġwords . ĠThe Ġfirst Ġresource , Ġalthough Ġdependent Ġon Ġeach Ġlanguage , Ġis Ġrelatively Ġeasy Ġto Ġobtain . ĠWe Ġhave Ġfound Ġthat , Ġeven Ġwith Ġlists Ġof Ġmoderate Ġsize , Ġthe Ġresults Ġare Ġquite Ġsignificant . ĠThe Ġgram matical Ġcategories Ġwere Ġobtained Ġwith Ġthe Ġhelp Ġof Ġthe ĠTree T agger Ġstatistics Ġtool . ĠHowever , ĠTree T agger Ġcould Ġbe Ġreplaced Ġby Ġany Ġother Ġtool Ġproducing Ġsimilar Ġresults . </s> App endix </s> In Ġthis Ġappendix , Ġwe Ġpresent Ġthe Ġlist Ġof Ġrhetorical Ġconnectors Ġin ĠFrench Ġthat Ġconstitute Ġour Ġlist Ġof Ġmarkers . ĠWe Ġpoint Ġout Ġthat Ġthe Ġmarkers Ġending Ġin Ġapost rophe Ġsuch Ġas : </s> pr Ã¨ s Ġqu ', ĠÃł Ġcondition Ġd ', Ġetc . </s> are Ġdeleted Ġfrom Ġa Ġregular Ġexpression Ġimplying Ġ' and ': Ġpr Ã¨ s Ġqu ' Ġ+ Ġpr Ã¨ s Ġque , ĠÃł Ġcondition Ġd ' Ġ+ ĠÃł Ġcondition Ġde , Ġetc . </s> 3 </s> , Ġ/ ĠÃł Ġ/ ĠÃł ĠÃ § a Ġpr Ã¨ s Ġqu ' Ġ/ ĠÃł Ġc ec i Ġpr Ã¨ s Ġqu ' Ġ/ ĠÃł Ġc ela Ġpr Ã¨ s Ġqu ' Ġ/ ĠÃł Ġce Ġmoment - l Ãł Ġ/ ĠÃł Ġce Ġpoint Ġqu ' Ġ/ ĠÃł Ġce Ġpropos Ġ/ ĠÃł Ġc et ĠÃ© gard Ġ/ ĠÃł Ġcondition Ġd ' Ġ/ ĠÃł Ġcondition Ġqu ' Ġ/ ĠÃł ĠdÃ© f aut Ġd ' Ġ/ ĠÃł ĠdÃ© f aut Ġde Ġ/ ĠÃł Ġdire Ġv ra i Ġ/ ĠÃł ĠÃ© lab orer Ġ/ ĠÃł Ġen Ġ/ Ġaf in Ġd ' Ġ/ Ġaf in Ġqu ' Ġ/ Ġaf in Ġque Ġ/ ĠÃł Ġforce Ġ/ ĠÃł Ġforce Ġd ' Ġ/ Ġa ins i Ġ/ ĠÃł Ġla Ġplace Ġ/ ĠÃł Ġla ĠrÃ© flex ion Ġ/ ĠÃł Ġl ' Ã© po que Ġo Ã ¹ Ġ/ ĠÃł Ġl ' he ure Ġo Ã ¹ Ġ/ ĠÃł Ġl ' inst ant Ġo Ã ¹ Ġ/ ĠÃł Ġl ' in verse Ġ/ Ġal ors Ġ/ Ġal ors Ġm Ãª me Ġqu ' Ġ/ Ġal ors Ġqu ' Ġ/ ĠÃł Ġmes ure Ġqu ' Ġ/ ĠÃł Ġmo ins Ġd ' Ġ/ ĠÃł Ġmo ins Ġqu ' Ġ/ ĠÃł Ġpart ĠÃ § a Ġ/ ĠÃł Ġpart ir Ġdu Ġmoment Ġo Ã ¹ Ġ/ ĠÃł Ġpart Ġqu ' Ġ/ Ġapr Ã¨ s Ġ/ ĠÃł Ġpr Ã©s ent Ġqu ' Ġ/ Ġapr Ã¨ s Ġqu ' Ġ/ Ġapr Ã¨ s Ġquo i Ġ/ Ġapr Ã¨ s Ġt out Ġ/ ĠÃł Ġpre uve Ġ/ ĠÃł Ġpropos Ġ/ ĠÃł Ġse ule Ġfin Ġd ' Ġ/ ĠÃł Ġse ule Ġfin Ġqu ' Ġ/ ĠÃł Ġsupp oser Ġqu ' Ġ/ ĠÃł Ġt elle Ġen se igne Ġqu ' Ġ/ ĠÃł Ġtel Ġpoint Ġqu ' Ġ/ Ġattend u Ġqu ' Ġ/ Ġau Ġbout Ġdu Ġcom pt e Ġ/ Ġau Ġcas Ġo Ã ¹ Ġ/ Ġau Ġcontra ire Ġ/ Ġau Ġf ait Ġ/ Ġau Ġfur Ġet ĠÃł Ġmes ure Ġqu ' Ġ/ Ġau Ġlieu Ġ/ Ġau Ġlieu Ġd ' Ġ/ Ġau Ġm Ãª me Ġtit re Ġqu ' Ġ/ Ġau Ġmo ins Ġ/ Ġau Ġmoment Ġd ' Ġ/ Ġau Ġmoment Ġo Ã ¹ Ġa up ar av ant Ġ/ Ġau Ġpoint Ġd ' Ġ/ Ġau Ġpoint Ġqu ' Ġ/ Ġa uss i Ġ/ Ġa uss i Ġlong tem ps Ġqu ' Ġ/ Ġa uss it Ã´ t Ġ/ Ġa uss it Ã´ t Ġqu ' Ġ/ Ġaut ant Ġ/ Ġaut ant Ġdire Ġqu ' Ġ/ Ġau Ġtotal Ġ/ Ġaut re ment Ġ/ Ġaut re ment Ġd it Ġ/ Ġav ant Ġ/ Ġav ant Ġd ' Ġ/ Ġav ant Ġm Ãª me Ġd ' Ġ/ Ġav ant Ġm Ãª me Ġqu ' Ġ/ Ġav ant Ġqu ' Ġ/ ĠÃł Ġv ra i Ġdire Ġ/ Ġb ien Ġqu ' Ġ/ Ġb ient Ã´ t Ġ/ Ġbre f Ġ/ Ġcar Ġ/ Ġc ec i Ġd it Ġ/ Ġc ec i Ġ Ã©t ant Ġd it Ġ/ Ġc ela Ġd it Ġ/ Ġc epend ant Ġ/ Ġc epend ant Ġqu ' Ġ/ Ġc ' est ĠÃł Ġdire Ġqu ' Ġ/ Ġc ' est Ġpour qu oi Ġ/ Ġc ette Ġf ois Ġqu ' Ġ/ Ġcomm e Ġ/ Ġcomm e ĠÃ § a Ġ/ Ġcomm e Ġquo i Ġ/ Ġcomm e Ġsi Ġ/ Ġcomparative ment Ġ/ Ġcons Ã© qu em ment Ġ/ Ġcons id Ã© rant Ġqu ' Ġ/ Ġcons id Ã© rÃ© Ġqu ' Ġ/ Ġcor rÃ© l ative ment Ġ/ Ġd ' ab ord Ġ/ Ġd ' ail le urs Ġ/ Ġd ans Ġce Ġcas Ġ/ Ġd ans Ġce Ġcas - l Ãł Ġ/ Ġd ans Ġla Ġmes ure Ġo Ã ¹ Ġ/ Ġd ans Ġle Ġbut Ġd ' Ġ/ Ġd ans Ġle Ġbut Ġqu ' Ġ/ Ġd ans Ġle Ġcas Ġo Ã ¹ Ġd ans Ġle Ġcoup Ġ/ Ġd ans Ġle Ġsens Ġo Ã ¹ Ġ/ Ġd ans Ġle Ġsens Ġqu ' Ġ/ Ġd ans Ġl ' es po ir Ġd ' Ġ/ Ġd ans Ġl ' es po ir Ġqu ' Ġ/ Ġd ans Ġl ' hyp oth Ã¨ se Ġo Ã ¹ Ġ/ Ġd ans Ġl ' intention Ġd ' Ġ/ Ġd ans Ġl ' intention Ġqu ' Ġ/ Ġd ans Ġt ous Ġles Ġcas Ġ/ Ġd ' aut ant Ġplus Ġqu ' Ġ/ Ġd ' aut ant Ġqu ' Ġ/ Ġd ' aut re Ġpart Ġ/ Ġde Ġce Ġf ait Ġ/ ĠdÃ© c id Ã© ment Ġ/ Ġde Ġfa Ã§ on ĠÃł Ġ/ Ġde Ġfa Ã§ on ĠÃł Ġce Ġqu ' Ġ/ Ġde Ġfa Ã§ on Ġqu ' Ġ/ Ġde Ġf ait Ġ/ ĠdÃ© j Ãł Ġ/ ĠdÃ© j Ãł Ġqu ' Ġ/ Ġde Ġla Ġm Ãª me Ġfa Ã§ on Ġ/ Ġde Ġla Ġm Ãª me Ġfa Ã§ on Ġqu ' Ġ/ Ġde Ġla Ġm Ãª me Ġman i Ã¨re Ġ/ Ġde Ġla Ġm Ãª me Ġman i Ã¨re Ġqu ' Ġ/ Ġde Ġman i Ã¨re ĠÃł Ġ/ Ġde Ġman i Ã¨re ĠÃł Ġce Ġqu ' Ġ/ Ġde Ġman i Ã¨re Ġqu ' Ġ/ Ġde Ġm Ãª me Ġ/ Ġde Ġm Ãª me Ġqu ' Ġ/ Ġde Ġplus Ġ/ Ġdep u is Ġ/ Ġdep u is Ġqu ' Ġ/ Ġdes Ġf ois Ġqu ' Ġ/ Ġd Ã¨ s Ġl ors Ġ/ Ġd Ã¨ s Ġl ors Ġqu ' Ġ/ Ġde Ġsort e Ġqu ' Ġ/ Ġd Ã¨ s Ġqu ' Ġ/ Ġde Ġt elle Ġfa Ã§ on Ġqu ' Ġ/ Ġde Ġt elle Ġman i Ã¨re Ġqu ' Ġ/ Ġde Ġt oute Ġfa Ã§ on Ġ/ Ġde Ġt oute Ġman i Ã¨re Ġ/ Ġde Ġt out es Ġfa Ã§ ons Ġ/ Ġde Ġt out es Ġman i Ã¨ res Ġ/ Ġd ' ici Ġqu ' Ġ/ Ġdire Ġqu ' Ġ/ Ġdon c Ġ/ Ġd ' o Ã ¹ Ġ/ Ġd ' o Ã ¹ Ġqu ' Ġ/ Ġdu Ġcoup Ġ/ Ġdu Ġf ait Ġqu ' Ġ/ Ġdu Ġmo ins Ġ/ Ġdu Ġmoment Ġqu ' Ġ/ Ġd ' un Ġaut re Ġc Ã´ t Ã© Ġd ' un Ġc Ã´ t Ã© Ġ/ Ġd ' un Ġcoup Ġ/ Ġd ' une Ġpart Ġ/ Ġd ' un Ġse ul Ġcoup Ġ/ Ġdu Ġrest e Ġ/ Ġdu Ġtem ps Ġo Ã ¹ Ġ/ Ġeffective ment Ġ/ ĠÃ© gal ement Ġ/ Ġen Ġ/ Ġen Ġadm ett ant Ġqu ' Ġ/ Ġen Ġattendant Ġ/ Ġen Ġbre f Ġ/ Ġen Ġce Ġcas Ġ/ Ġen Ġce Ġsens Ġqu ' Ġ/ Ġen Ġcomp ara ison Ġ/ Ġen Ġcons Ã© qu ence Ġ/ Ġenc ore Ġ/ Ġenc ore Ġqu ' Ġ/ Ġen Ġd ' aut res Ġterm es Ġ/ Ġen ĠdÃ© fin itive Ġ/ Ġen ĠdÃ© pit Ġdu Ġf ait Ġqu ' Ġ/ Ġen ĠdÃ© pit Ġqu ' Ġ/ Ġen Ġeff et Ġ/ Ġen Ġf ait Ġ/ Ġen fin Ġ/ Ġen Ġg ros Ġ/ Ġen Ġm Ãª me Ġtem ps Ġ/ Ġen Ġm Ãª me Ġtem ps Ġqu ' Ġ/ Ġen Ġout re Ġ/ Ġen Ġpartic ul ier Ġ/ Ġen Ġplus Ġ/ Ġen Ġplus Ġd ' Ġ/ Ġen Ġplus Ġde Ġ/ Ġen ĠrÃ© al itÃ© Ġ/ Ġen ĠrÃ© sum Ã© Ġ/ Ġen Ġrev anche Ġ/ Ġen Ġsom me Ġ/ Ġensu ite Ġ/ Ġen Ġsupp os ant Ġqu ' Ġ/ Ġen Ġt ous Ġcas Ġen Ġt ous Ġles Ġcas Ġ/ Ġen Ġt out Ġcas Ġ/ Ġen Ġt out Ġ Ã©t at Ġde Ġcause Ġ/ Ġen Ġv Ã© rit Ã© Ġ/ Ġen Ġv ue Ġd ' Ġ/ Ġet Ġ/ Ġ Ã©t ant Ġdon n Ã© Ġqu ' Ġ/ Ġet Ġdire Ġqu ' Ġ/ Ġet Ġpu is Ġ/ Ġexcept Ã© Ġqu ' Ġ/ Ġfa ute Ġd ' Ġ/ Ġfinal ement Ġ/ Ġglobal ement Ġ/ Ġhist oire Ġd ' Ġ/ Ġhorm is Ġle Ġf ait Ġqu ' Ġ/ Ġhorm is Ġqu ' Ġ/ Ġinstant an Ã© ment Ġ/ Ġinverse ment Ġ/ Ġj us qu ' Ãł Ġ/ Ġj us qu ' Ãł Ġce Ġqu ' Ġ/ Ġla Ġpre uve Ġ/ Ġle Ġf ait Ġest Ġqu ' Ġ/ Ġle Ġjour Ġo Ã ¹ Ġ/ Ġle Ġtem ps Ġqu ' Ġ/ Ġl ors qu ' Ġ/ Ġmain ten ant Ġ/ Ġmain ten ant Ġqu ' Ġ/ Ġm ais Ġ/ Ġmal gr Ã© Ġle Ġf ait Ġqu ' Ġ/ Ġmal gr Ã© Ġqu ' Ġ/ Ġmal gr Ã© Ġt out Ġ/ Ġmal he ure use ment Ġ/ Ġm Ãª me Ġ/ Ġm Ãª me Ġqu ' Ġ/ Ġm Ãª me Ġsi Ġ/ Ġm ie ux Ġ/ Ġmis ĠÃł Ġpart Ġle Ġf ait Ġqu ' Ġ/ Ġmis ĠÃł Ġpart Ġqu ' Ġ/ Ġn Ã© an mo ins Ġ/ Ġnon ob stant Ġ/ Ġnon ob stant Ġqu ' Ġ/ Ġor Ġ/ Ġo u Ġ/ Ġo u Ġb ien Ġ/ Ġout re Ġqu ' Ġ/ Ġpar Ġa ille urs Ġ/ Ġpar all Ã¨ lement Ġ/ Ġpar ce Ġqu ' Ġ/ Ġpar Ġcomp ara ison Ġ/ Ġpar Ġcons Ã© qu ent Ġ/ Ġpar Ġcont re Ġ/ Ġpar - d ess us Ġt out Ġ/ Ġpar Ġex em ple Ġ/ Ġpar Ġle Ġf ait Ġqu ' Ġ/ Ġpar Ġsuite Ġ/ Ġp endant Ġqu ' Ġ/ Ġpe u Ġimport e Ġplus Ġqu ' Ġ/ Ġplus Ġt ard Ġpl ut Ã´ t Ġ/ Ġpl ut Ã´ t Ġqu ' Ġ/ Ġpl ut Ã´ t Ġque Ġd ' Ġ/ Ġpour Ġ/ Ġpour Ġaut ant Ġpour Ġaut ant Ġqu ' Ġ/ Ġpour Ġcomm encer Ġ/ Ġpour Ġcon cl ure Ġ/ Ġpour Ġfin ir Ġ/ Ġpour Ġle Ġcoup Ġ/ Ġpour Ġpe u Ġqu ' Ġ/ Ġpour Ġpre uve Ġ/ Ġpour Ġqu ' Ġ/ Ġpour Ġr Ã©s umer Ġ/ Ġpour t ant Ġ/ Ġpour Ġtermin er Ġ/ Ġpour Ġune Ġf ois Ġqu ' Ġ/ Ġpour vu Ġqu ' Ġ/ Ġprem i Ã¨re ment Ġ/ Ġpre uve Ġqu ' Ġ/ Ġpu is Ġ/ Ġpu is qu ' Ġ/ Ġqu and Ġ/ Ġqu and Ġb ien Ġm Ãª me Ġ/ Ġqu and Ġb ien Ġm Ãª me Ġqu ' Ġ/ Ġqu and Ġm Ãª me Ġ/ Ġquant ĠÃł Ġ/ Ġqu itte ĠÃł Ġ/ Ġqu itte ĠÃł Ġce Ġqu ' Ġ/ Ġquo iqu ' Ġ/ Ġquo i Ġqu ' il Ġen Ġso it Ġ/ ĠrÃ© c ip ro qu ement Ġ/ ĠrÃ© flex ion Ġfa ite Ġ/ Ġrem ar que Ġ/ Ġr Ã©s ult at Ġ/ Ġs ' Ġ/ Ġs ach ant Ġqu ' Ġ/ Ġsans Ġ/ Ġsans Ġcomp ter Ġqu ' Ġ/ Ġsans Ġo ub lier Ġqu ' Ġ/ Ġsans Ġqu ' Ġ/ Ġsa uf ĠÃł Ġ/ Ġsa uf Ġqu ' Ġ/ Ġse lon Ġqu ' Ġ/ Ġsi Ġ/ Ġsi Ġb ien Ġqu ' Ġ/ Ġsi Ġce Ġn ' est Ġqu ' Ġ/ Ġsim ultan Ã© ment Ġ/ Ġsin on Ġ/ Ġsin on Ġqu ' Ġ/ Ġsi Ġtant Ġest Ġqu ' Ġ/ Ġsit Ã´ t Ġqu ' Ġ/ Ġso it Ġ/ Ġso it Ġd it Ġen Ġpass ant Ġ/ Ġsom me Ġt oute Ġ/ Ġs oud ain Ġ/ Ġsubs Ã© qu em ment Ġ/ Ġsu iv ant Ġqu ' Ġ/ Ġsur t out Ġ/ Ġsur t out Ġqu ' Ġ/ Ġt and is Ġqu ' Ġ/ Ġtant Ġet Ġsi Ġb ien Ġqu ' Ġ/ Ġtant Ġqu ' Ġ/ Ġtotal Ġ/ Ġt out ĠÃł Ġcoup Ġ/ Ġt out Ġau Ġmo ins Ġ/ Ġt out Ġb ien Ġcons id Ã© rÃ© Ġ/ Ġt out Ġcom pt e Ġf ait Ġ/ Ġt out Ġd ' ab ord Ġ/ Ġt out Ġde Ġm Ãª me Ġ/ Ġt out Ġen Ġ/ Ġune Ġf ois Ġqu ' Ġ/ Ġun Ġjour Ġ/ Ġun Ġjour Ġqu ' Ġ/ Ġun Ġpe u Ġplus Ġt ard Ġ/ Ġv u Ġqu ' Ġ/
Introduction Rhetorical Structure Theory (RST) BIBREF0 is a technique of Natural Language Processing (NLP), in which a document can be structured hierarchically according to its discourse. The generated hierarchy, a tree, provides information associated with the boundaries of the discourse segments and related to their importance and dependencies. The figure FIGREF1 shows an example of such a rethorical tree. In the rethorical parsing process, the text has been divided into five units. In the figure FIGREF1, the arrow that leaves the unit (2) towards the unit (1) symbolizes that the unit (2) is the satellite of the unit (1), which is the core in a “Concession” relationship. In turn, the units (1) and (2) comprise the nucleus of three “Demonstration” relationships. The discursive analysis of a document normally includes three consecutive steps: 1) discursive segmentation; 2) detection of the discursive relations; 3) construction of the hierarchical rhetorical tree. Regarding the discursive segmentation, there are segmenters in several languages. However, each piece depends on sofisticated linguistic resources, which complicates the reproduction of the experiments in other languages. Consequently, the development of multilingual systems using discursive analysis are yet to be developed. Diverse applications based on the latest technologies require at least one of the three steps mentioned above BIBREF1, BIBREF2, BIBREF3. In this context, the idea of exploring the architecture of a generic system that is able not only of segmenting a text correctly but also of adapting it to any language, was a great motivation of this research work. In this article we show the preliminary results of a generic segmenter composed of several systems (different segmentation strategies). In addition, we describe an automatic evaluation protocol of discursive segmentation. The article is composed by the following sections: state of the art (SECREF2), which presents a brief bibliographic review; Description of the Annodis (SECREF3) corpus used in our tests and of the general architecture of the proposed systems (SECREF4); Segmentation strategies (SECREF5), which characterizes the different methods implemented to segment the text; results of our numerical experiments (sec:experiments); and we conclude with our conclusions and perspectives (SECREF7). State-of-the-art In RST, there are tow discursive units: nuclei and satellites. The nucleus provide information pertinent to the purposes of the author of the text and the satellites add additional information to the nucleu, on which they are dependent on. In the context of RST, possible discursive relationships may be nucleus-satellite and multinuclear. In nucleus-satellite relationships, a satellite depends on one nucleus, whereas in multinuclear relationships, several nuclei (at least two) are regrouped at the same level of importance (tree hierarchy). Thus, in the discursive segmentation proposes to reduce the text into the minimal discursive units called Elementary Discursive Units (EDU), through the use of explicit discursive markers. As an example, we can quote some markers in French: afin de, pour que, donc, quand bien même que, ensuite, de fois que, globativamente, par contre, sinon, à ce moment-là, cependant, subséquemment, puisque, au fur et à mesure que, si, finalement, etc.. Markers or particles are often used to connect ideas. Let's consider the sentence below: La ville d'Avignon est la capitale du Vaucluse, qui est un département du sud de la France. qui (which) is a discursive marker because it connects two ideas. The first one, “Avignon City is the capital of Vaucluse” (La ville d'Avignon est la capitale du Vaucluse), and the second one (satellite), “[Vaucluse] is a department in the south of France” ([Vaucluse] est un département du sud de la France). Several research has addressed automatic segmentation in several languages, such as: French BIBREF4, English BIBREF5, Portuguese BIBREF6, Spanish BIBREF7, BIBREF8 and Tahi. BIBREF9. All converge to the idea of using an explicit list of marks in order to segment texts. Annodis Corpus In this first exploratory work, our tests considered only documents in French from the Annodis corpus. Annodis (ANNOtation DIScursive) is a set of documents in French that were manually enriched with notes of discursive structures. Its main characteristics are: Two annotations: Rhetorical relations and multilevel structures. Documents (687 000 words) taken from four sources: the Est Républicain newspaper (39 articles, 10 000 words); Wikipedia (30 articles + 30 summaries, 242 000 words); Proceedings of the conference Traitement Automatique des Langues Naturelles (TALN) 2008 (25 articles, 169 000 words); Reports from Institut Français de Relations Internationales (32 raports, 266 000 words). The corpora were noted using Glozz. Annodis aims at building an annotated corpus. The proposed annotations are on two levels of analysis, that is, two perspectives: Ascendant: part of EDU are used in the construction of more complex structures, through the relations of discourse; Descending: approaches the text in its entirety and relies on the various shallow indices to identify high-level discursive structures (macro structures). Two types of persons annotated Annodis: linguistic experts and students. The first group constituted a $E$ subcorpus called “specialist” and the second group resulted in a $N$ subcorpus called “naive”. These rhetorically annotated subcorps were used as references in our experiments. (c.f. §SECREF6). Discourse Segmenter Overall Description The Figure FIGREF12 shows the general architecture of the proposed discourse segmenter system. The initial input is the raw text encoded in UTF-8. The two initial processes are Part of Speech morphosyntactic Tagging (POS) and the segmentation at the level of the sentences. This last is just a preprocessing step that splits sentences. In the last process the system uses a bank of explicit markers in roder to apply the rules for the final discourse segmentation. For the experiments, we used lists of markers in French, Spanish, English and Portuguese. We also used the Lexiconn BIBREF10 project list, which regroups 328 French-language markers. Another important parameter specifies which segmentation strategy should be applied, according to the POS labelling of the document. Description of segmentation strategies ::: Segmentation with explicit use of a marker The elementary system Segmenter$_{\mu }$ (baseline) relies solely on a list of discursive markers to perform the segmentation. It replaces the appearance of a marker in the list with a special symbol, for example $\mu $, which indicates a boundary between the right and left segment. Be the sentence of the preceding example: La ville d'Avignon est la capitale du Vaucluse, qui est un département du Sud de la France.. The Segmenter split the sentence in two parts: the left segment (SE), La ville d'Avignon est la capitale du Vaucluse, and the right segment (SD), est un département du sud de la France. Description of segmentation strategies ::: Segmentation with explicit use of a marker and POS labels The Segmenter$_{\mu +}$ system presents an improvement to the Segmenter$_{\mu }$: inclusion of grammar categories with the TreeTagger tool. The advantage of this system is the detection of certain grammatical forms in order to condition the segmentation. Since it is based on the Segmenter$_{\mu }$, we try to recognise the opportune conditions to gather two segments when both are part of the same discursive segment. We try to identify more subtly when it is pertinent to leave the two segments separate. The Segmenter$_{\mu }$ has two distinct strategies: Segmentador$_{\mu +V}$ (verbal version, V): it relies solely on the presence of verbal forms to the right and left of the discursive marker. The two grammatical rules of this strategy are: If there are no verbs in the left and right segments, regroup them. If there is at least one verb in the left or right segment, the segments will remain separate. Segmenter$_{\mu +(V-N)}$ (verb-noun version, V-N): it relies on the presence of verbs and nouns. For this version, four rules are considered: If there is no noun in either the left or right segment, we regroup the segments. We regroup the segments if at least one of them has no noun. If at least one noun is present in both segments, they remain independent. If there is no verb-nominal form, the segments remain independent. Experiments In this first exploratory work, only documents in French were considered, but the system can be adapted to other languages. The evaluation is based on the correspondence of word pairs representing a border. In this way we compare the Annodis segmentation with the automatically produced segmentation. For each pair of reference segments, a $L_r$ list of word pairs is provided: the last word of the first segment and the first word of the second. For example, considering the reference text wik1_01_02-04-2006.seg, from Annodis corpus: [Le Ban Amendment]$_1$ [Après avoir adopté la Convention,]_2 [un certain nombre de PED et d'associations de défense de l'environnement soutinrent]_3 [que le document n'allait pas assez loin.]_4 [De nombreux pays et ONG militèrent]_5 [en faveur d'une interdiction totale de l'expédition de déchets dangereux à destinations des PED.]_6 [Plus exactement,]_7 [la Convention originale n'interdisait pas l'exportation de déchets,]_8 [excepté vers l'Antarctique.]_9 [Elle n'exigeait]_10 [qu'une procédure de consentement préalable en connaissance de cause]_11 [(PIC, Prior Informed Consent).]_12 Here are the word pairs of the created reference list (punctuation marks are disregarded): $L_r$={[Convention – un], [soutinrent – que], [loin – de], [militèrent – en], [exactement – la], [PED – plus], [exactement – la], [déchets – excepté], [Antartique – Elle], [exigeait – qu'une], [cause – PIC] } We decided to count the word pairs instead of the segments, as this is a first version of the evaluation protocol. In fact, the segments may be nested, which complicates the evaluation process. Although there are some errors, word boundaries allow us to detect segments more easily. We have built a second $L_c$ list for the automatically identified segments, following the same criteria of $L_r$. The $L_r$ and $L_c$ lists regroup, pair by pair, the segment border. We then count the common pair intersection of the two lists. Each pair in the $L_c$ list is also present in the $L_r$ reference list and is a correctly assigned to the class pair. A word pair belonging to the $L_c$ list but not belonging to the $L_r$ reference list, will be a pair assigned to the class. For that same text, the $L_c$ list of candidate pairs obtained with the Segmentator$_{\mu }$ is: $L_r$={[loin–De], [pays–et], [militèrent–en], [dangereux–à], [PED–Plus], [Antarctique–Elle], [préalable–en], [cause–PIC] } We calculate the precision $P$, the recall $R$ and the $F$-score on the text corpus used in our tests, as follow: The precision, the recall and the $F$-score for this example is: $P$ = 5 / 11 = 0.45; $R$ = 5 / 8 = 0.625; F-score = 2 $\times \frac{ 0.45 \times 0.625}{ 0.45 + 0.625} = 0.523$. We used the documents in the Annodis corpus without segmentation, because they had been segmented with the Segmenter$_{\mu }$ and with the grammar segmenters. Two batch of tests were performed. The first on the $D$ set of documents common to the two subcorpus “specialist” $E$ and “naive” $N$ from Annodis. $D$ contains 38 documents with 13 364 words. This first test allowed to measure the distance between the human markers. In fact, in order to get an idea of the quality of the human segmentations, the cuts in the texts made by the specialists were measured it versus the so-called “naifs” note takers and vice versa. The second series of tests consisted of using all the documents of the subcorpus “specialist” $E$, because the documents of the subcorpus of Annodis are not identical. Then we benchmarked the performance of the three systems automatically. Experiments ::: Results In this section we will compare the results of the different segmentation systems through automatic evaluations. First of all, the human segmentation, from the subcorpus $D$ composed of common documents. The results are presented in the table tab:humains. The first row shows the performance of the $I$ segments, taking the experts as a reference, while the second presents the process in the opposite direction. We have found that segmentation by experts and naive produces two subcorpus $E$ and $N$ with very similar characteristics. This surprised us, as we expected a more important difference between them. In any case, we deduced that, at least in this corpus, it is not necessary to be an expert in linguistics to discursively segment the documents. As far as system evaluations are concerned, we use the 78 $E$ documents as reference. Table TABREF26 shows the results. In the case of the Experts, the grammatical verb-nominal version (V-N) had better F-score performance. The verbal version (V) obtained a better accuracy $P$ than the verb-nominal (V-N). In the case of the Naive, the performance F-score, $P$ and $R$ is very similar from the Experts. Conclusions, discussion and perspectives The aim of this work was twofold: to design a discursive segmenter using a minimum of resources and to establish an evaluation protocol to measure the performance of segmenters. The results show that we can build a simple version of the baseline, which employs only a list of markers and presents a very encouraging performance. Of course, the quality of the list is a preponderant factor for a correct segmentation. We have studied the impact of the marker which, even though it may seem fringe-worthy, contributes to improving the performance of our segmenters. Thus, it is an interesting marker that we can consider as a discursive marker. The Segmentator$_{\mu }$ version provides the best results in terms of F-score and recall, followed by the Segmentator$_{\mu +V}$ version, which passes it in precision. Regarding evaluation, we developed a simple protocol to compare the performance of the systems. This is, to our knowledge, the first automatic evaluation in French. It is necessary to intensify our research in order to propose improvements to our segmenters, as well as to study further the impact of grammar tag rules on segmentation. Since we have a standard evaluation protocol, we intend to carry out tests with Portuguese, Spanish (see BIBREF11), English, etc. For that, we will only need a list of markers for each language. The performance of the systems remains modest, of course, but we must not forget that this is a baseline and its primary objective is to provide standard systems that can be used in testing protocols such as the one we proposed. Despite this evolution, these baselines (or their improved versions) can be used in applications such as automatic document summarisation (e.g., BIBREF12, BIBREF13), or sentences compression BIBREF14. The main feature of the proposed baseline system is its flexibility with respect to the language considered. In fact, it only uses a list of language markers and the grammatical category of words. The first resource, although dependent on each language, is relatively easy to obtain. We have found that, even with lists of moderate size, the results are quite significant. The grammatical categories were obtained with the help of the TreeTagger statistics tool. However, TreeTagger could be replaced by any other tool producing similar results. Appendix In this appendix, we present the list of rhetorical connectors in French that constitute our list of markers. We point out that the markers ending in apostrophe such as: près qu', à condition d', etc. are deleted from a regular expression implying 'and': près qu' + près que, à condition d' + à condition de, etc. 3 , / à / à ça près qu' / à ceci près qu' / à cela près qu' / à ce moment-là / à ce point qu' / à ce propos / à cet égard / à condition d' / à condition qu' / à défaut d' / à défaut de / à dire vrai / à élaborer / à en / afin d' / afin qu' / afin que / à force / à force d' / ainsi / à la place / à la réflexion / à l'époque où / à l'heure où / à l'instant où / à l'inverse / alors / alors même qu' / alors qu' / à mesure qu' / à moins d' / à moins qu' / à part ça / à partir du moment où / à part qu' / après / à présent qu' / après qu' / après quoi / après tout / à preuve / à propos / à seule fin d' / à seule fin qu' / à supposer qu' / à telle enseigne qu' / à tel point qu' / attendu qu' / au bout du compte / au cas où / au contraire / au fait / au fur et à mesure qu' / au lieu / au lieu d' / au même titre qu' / au moins / au moment d' / au moment où auparavant / au point d' / au point qu' / aussi / aussi longtemps qu' / aussitôt / aussitôt qu' / autant / autant dire qu' / au total / autrement / autrement dit / avant / avant d' / avant même d' / avant même qu' / avant qu' / à vrai dire / bien qu' / bientôt / bref / car / ceci dit / ceci étant dit / cela dit / cependant / cependant qu' / c'est à dire qu' / c'est pourquoi / cette fois qu' / comme / comme ça / comme quoi / comme si / comparativement / conséquemment / considérant qu' / considéré qu' / corrélativement / d'abord / d'ailleurs / dans ce cas / dans ce cas-là / dans la mesure où / dans le but d' / dans le but qu' / dans le cas où dans le coup / dans le sens où / dans le sens qu' / dans l'espoir d' / dans l'espoir qu' / dans l'hypothèse où / dans l'intention d' / dans l'intention qu' / dans tous les cas / d'autant plus qu' / d'autant qu' / d'autre part / de ce fait / décidément / de façon à / de façon à ce qu' / de façon qu' / de fait / déjà / déjà qu' / de la même façon / de la même façon qu' / de la même manière / de la même manière qu' / de manière à / de manière à ce qu' / de manière qu' / de même / de même qu' / de plus / depuis / depuis qu' / des fois qu' / dès lors / dès lors qu' / de sorte qu' / dès qu' / de telle façon qu' / de telle manière qu' / de toute façon / de toute manière / de toutes façons / de toutes manières / d'ici qu' / dire qu' / donc / d'où / d'où qu' / du coup / du fait qu' / du moins / du moment qu' / d'un autre côté d'un côté / d'un coup / d'une part / d'un seul coup / du reste / du temps où / effectivement / également / en / en admettant qu' / en attendant / en bref / en ce cas / en ce sens qu' / en comparaison / en conséquence / encore / encore qu' / en d'autres termes / en définitive / en dépit du fait qu' / en dépit qu' / en effet / en fait / enfin / en gros / en même temps / en même temps qu' / en outre / en particulier / en plus / en plus d' / en plus de / en réalité / en résumé / en revanche / en somme / ensuite / en supposant qu' / en tous cas en tous les cas / en tout cas / en tout état de cause / en vérité / en vue d' / et / étant donné qu' / et dire qu' / et puis / excepté qu' / faute d' / finalement / globalement / histoire d' / hormis le fait qu' / hormis qu' / instantanément / inversement / jusqu'à / jusqu'à ce qu' / la preuve / le fait est qu' / le jour où / le temps qu' / lorsqu' / maintenant / maintenant qu' / mais / malgré le fait qu' / malgré qu' / malgré tout / malheureusement / même / même qu' / même si / mieux / mis à part le fait qu' / mis à part qu' / néanmoins / nonobstant / nonobstant qu' / or / ou / ou bien / outre qu' / par ailleurs / parallèlement / parce qu' / par comparaison / par conséquent / par contre / par-dessus tout / par exemple / par le fait qu' / par suite / pendant qu' / peu importe plus qu' / plus tard plutôt / plutôt qu' / plutôt que d' / pour / pour autant pour autant qu' / pour commencer / pour conclure / pour finir / pour le coup / pour peu qu' / pour preuve / pour qu' / pour résumer / pourtant / pour terminer / pour une fois qu' / pourvu qu' / premièrement / preuve qu' / puis / puisqu' / quand / quand bien même / quand bien même qu' / quand même / quant à / quitte à / quitte à ce qu' / quoiqu' / quoi qu'il en soit / réciproquement / réflexion faite / remarque / résultat / s' / sachant qu' / sans / sans compter qu' / sans oublier qu' / sans qu' / sauf à / sauf qu' / selon qu' / si / si bien qu' / si ce n'est qu' / simultanément / sinon / sinon qu' / si tant est qu' / sitôt qu' / soit / soit dit en passant / somme toute / soudain / subséquemment / suivant qu' / surtout / surtout qu' / tandis qu' / tant et si bien qu' / tant qu' / total / tout à coup / tout au moins / tout bien considéré / tout compte fait / tout d'abord / tout de même / tout en / une fois qu' / un jour / un jour qu' / un peu plus tard / vu qu' /
How is segmentation quality evaluated?
f8da63df16c4c42093e5778c01a8e7e9b270142e
How Ġis Ġsegment ation Ġquality Ġevaluated ?
Seg mentation Ġquality Ġis Ġevaluated Ġby Ġcalculating Ġthe Ġprecision , Ġrecall , Ġand ĠF - score Ġof Ġthe Ġautomatic Ġsegment ations Ġin Ġcomparison Ġto Ġthe Ġsegment ations Ġmade Ġby Ġexpert Ġannot ators Ġfrom Ġthe ĠANN OD IS Ġsub cor p us .
<s> How Ġis Ġsegment ation Ġquality Ġevaluated ? </s> Introduction </s> R het orical ĠStructure ĠTheory Ġ( R ST ) ĠB IB REF 0 Ġis Ġa Ġtechnique Ġof ĠNatural ĠLanguage ĠProcessing Ġ( N LP ), Ġin Ġwhich Ġa Ġdocument Ġcan Ġbe Ġstructured Ġhierarch ically Ġaccording Ġto Ġits Ġdiscourse . ĠThe Ġgenerated Ġhierarchy , Ġa Ġtree , Ġprovides Ġinformation Ġassociated Ġwith Ġthe Ġboundaries Ġof Ġthe Ġdiscourse Ġsegments Ġand Ġrelated Ġto Ġtheir Ġimportance Ġand Ġdependencies . ĠThe Ġfigure ĠFIG REF 1 Ġshows Ġan Ġexample Ġof Ġsuch Ġa Ġre th orical Ġtree . ĠIn Ġthe Ġre th orical Ġparsing Ġprocess , Ġthe Ġtext Ġhas Ġbeen Ġdivided Ġinto Ġfive Ġunits . ĠIn Ġthe Ġfigure ĠFIG REF 1 , Ġthe Ġarrow Ġthat Ġleaves Ġthe Ġunit Ġ( 2 ) Ġtowards Ġthe Ġunit Ġ( 1 ) Ġsymbol izes Ġthat Ġthe Ġunit Ġ( 2 ) Ġis Ġthe Ġsatellite Ġof Ġthe Ġunit Ġ( 1 ), Ġwhich Ġis Ġthe Ġcore Ġin Ġa ĠâĢ ľ Con cession âĢ Ŀ Ġrelationship . ĠIn Ġturn , Ġthe Ġunits Ġ( 1 ) Ġand Ġ( 2 ) Ġcomprise Ġthe Ġnucleus Ġof Ġthree ĠâĢ ľ Demon stration âĢ Ŀ Ġrelationships . </s> The Ġdisc ursive Ġanalysis Ġof Ġa Ġdocument Ġnormally Ġincludes Ġthree Ġconsecutive Ġsteps : Ġ1 ) Ġdisc ursive Ġsegment ation ; Ġ2 ) Ġdetection Ġof Ġthe Ġdisc ursive Ġrelations ; Ġ3 ) Ġconstruction Ġof Ġthe Ġhierarchical Ġrhetorical Ġtree . ĠRegarding Ġthe Ġdisc ursive Ġsegment ation , Ġthere Ġare Ġsegment ers Ġin Ġseveral Ġlanguages . ĠHowever , Ġeach Ġpiece Ġdepends Ġon Ġso f istic ated Ġlinguistic Ġresources , Ġwhich Ġcompl icates Ġthe Ġreproduction Ġof Ġthe Ġexperiments Ġin Ġother Ġlanguages . ĠConsequently , Ġthe Ġdevelopment Ġof Ġmult ilingual Ġsystems Ġusing Ġdisc ursive Ġanalysis Ġare Ġyet Ġto Ġbe Ġdeveloped . ĠD iverse Ġapplications Ġbased Ġon Ġthe Ġlatest Ġtechnologies Ġrequire Ġat Ġleast Ġone Ġof Ġthe Ġthree Ġsteps Ġmentioned Ġabove ĠB IB REF 1 , ĠB IB REF 2 , ĠB IB REF 3 . ĠIn Ġthis Ġcontext , Ġthe Ġidea Ġof Ġexploring Ġthe Ġarchitecture Ġof Ġa Ġgeneric Ġsystem Ġthat Ġis Ġable Ġnot Ġonly Ġof Ġsegment ing Ġa Ġtext Ġcorrectly Ġbut Ġalso Ġof Ġadapting Ġit Ġto Ġany Ġlanguage , Ġwas Ġa Ġgreat Ġmotivation Ġof Ġthis Ġresearch Ġwork . </s> In Ġthis Ġarticle Ġwe Ġshow Ġthe Ġpreliminary Ġresults Ġof Ġa Ġgeneric Ġsegment er Ġcomposed Ġof Ġseveral Ġsystems Ġ( different Ġsegment ation Ġstrategies ). ĠIn Ġaddition , Ġwe Ġdescribe Ġan Ġautomatic Ġevaluation Ġprotocol Ġof Ġdisc ursive Ġsegment ation . ĠThe Ġarticle Ġis Ġcomposed Ġby Ġthe Ġfollowing Ġsections : Ġstate Ġof Ġthe Ġart Ġ( SEC REF 2 ), Ġwhich Ġpresents Ġa Ġbrief Ġb ibli ographic Ġreview ; ĠDescription Ġof Ġthe ĠAnn od is Ġ( SEC REF 3 ) Ġcorpus Ġused Ġin Ġour Ġtests Ġand Ġof Ġthe Ġgeneral Ġarchitecture Ġof Ġthe Ġproposed Ġsystems Ġ( SEC REF 4 ); ĠSe gment ation Ġstrategies Ġ( SEC REF 5 ), Ġwhich Ġcharacter izes Ġthe Ġdifferent Ġmethods Ġimplemented Ġto Ġsegment Ġthe Ġtext ; Ġresults Ġof Ġour Ġnumerical Ġexperiments Ġ( sec : exper iments ); Ġand Ġwe Ġconclude Ġwith Ġour Ġconclusions Ġand Ġperspectives Ġ( SEC REF 7 ). </s> State - of - the - art </s> In ĠR ST , Ġthere Ġare Ġtow Ġdisc ursive Ġunits : Ġnucle i Ġand Ġsatellites . ĠThe Ġnucleus Ġprovide Ġinformation Ġpertinent Ġto Ġthe Ġpurposes Ġof Ġthe Ġauthor Ġof Ġthe Ġtext Ġand Ġthe Ġsatellites Ġadd Ġadditional Ġinformation Ġto Ġthe Ġnucle u , Ġon Ġwhich Ġthey Ġare Ġdependent Ġon . ĠIn Ġthe Ġcontext Ġof ĠR ST , Ġpossible Ġdisc ursive Ġrelationships Ġmay Ġbe Ġnucleus - s atellite Ġand Ġmult in uclear . ĠIn Ġnucleus - s atellite Ġrelationships , Ġa Ġsatellite Ġdepends Ġon Ġone Ġnucleus , Ġwhereas Ġin Ġmult in uclear Ġrelationships , Ġseveral Ġnucle i Ġ( at Ġleast Ġtwo ) Ġare Ġreg roup ed Ġat Ġthe Ġsame Ġlevel Ġof Ġimportance Ġ( tree Ġhierarchy ). ĠThus , Ġin Ġthe Ġdisc ursive Ġsegment ation Ġproposes Ġto Ġreduce Ġthe Ġtext Ġinto Ġthe Ġminimal Ġdisc ursive Ġunits Ġcalled ĠElementary ĠDisc ursive ĠUnits Ġ( ED U ), Ġthrough Ġthe Ġuse Ġof Ġexplicit Ġdisc ursive Ġmarkers . ĠAs Ġan Ġexample , Ġwe Ġcan Ġquote Ġsome Ġmarkers Ġin ĠFrench : </s> af in Ġde , Ġpour Ġque , Ġdon c , Ġqu and Ġb ien Ġm Ãª me Ġque , Ġensu ite , Ġde Ġf ois Ġque , Ġglob at iv ament e , Ġpar Ġcont re , Ġsin on , ĠÃł Ġce Ġmoment - l Ãł , Ġc epend ant , Ġsubs Ã© qu em ment , Ġpu is que , Ġau Ġfur Ġet ĠÃł Ġmes ure Ġque , Ġsi , Ġfinal ement , Ġetc .. </s> Mark ers Ġor Ġparticles Ġare Ġoften Ġused Ġto Ġconnect Ġideas . ĠLet 's Ġconsider Ġthe Ġsentence Ġbelow : </s> La Ġv ille Ġd ' Av ign on Ġest Ġla Ġcapit ale Ġdu ĠV au cl use , Ġqui Ġest Ġun ĠdÃ© part ement Ġdu Ġsu d Ġde Ġla ĠFrance . </s> qu i Ġ( which ) Ġis Ġa Ġdisc ursive Ġmarker Ġbecause Ġit Ġconnects Ġtwo Ġideas . ĠThe Ġfirst Ġone , ĠâĢ ľ Av ign on ĠCity Ġis Ġthe Ġcapital Ġof ĠV au cl use âĢ Ŀ Ġ( La Ġv ille Ġd ' Av ign on Ġest Ġla Ġcapit ale Ġdu ĠV au cl use ), Ġand Ġthe Ġsecond Ġone Ġ( s atellite ), ĠâĢ ľ [ V au cl use ] Ġis Ġa Ġdepartment Ġin Ġthe Ġsouth Ġof ĠFrance âĢ Ŀ Ġ([ V au cl use ] Ġest Ġun ĠdÃ© part ement Ġdu Ġsu d Ġde Ġla ĠFrance ). ĠSeveral Ġresearch Ġhas Ġaddressed Ġautomatic Ġsegment ation Ġin Ġseveral Ġlanguages , Ġsuch Ġas : ĠFrench ĠB IB REF 4 , ĠEnglish ĠB IB REF 5 , ĠPortuguese ĠB IB REF 6 , ĠSpanish ĠB IB REF 7 , ĠB IB REF 8 Ġand ĠTah i . ĠB IB REF 9 . ĠAll Ġconverge Ġto Ġthe Ġidea Ġof Ġusing Ġan Ġexplicit Ġlist Ġof Ġmarks Ġin Ġorder Ġto Ġsegment Ġtexts . </s> Ann od is ĠCorpus </s> In Ġthis Ġfirst Ġexplor atory Ġwork , Ġour Ġtests Ġconsidered Ġonly Ġdocuments Ġin ĠFrench Ġfrom Ġthe ĠAnn od is Ġcorpus . ĠAnn od is Ġ( AN NO t ation ĠDIS c ursive ) Ġis Ġa Ġset Ġof Ġdocuments Ġin ĠFrench Ġthat Ġwere Ġmanually Ġenriched Ġwith Ġnotes Ġof Ġdisc ursive Ġstructures . ĠIts Ġmain Ġcharacteristics Ġare : </s> Two Ġannotations : ĠR het orical Ġrelations Ġand Ġmult ile vel Ġstructures . </s> Documents Ġ( 687 Ġ000 Ġwords ) Ġtaken Ġfrom Ġfour Ġsources : Ġthe ĠEst ĠRÃ© public ain Ġnewspaper Ġ( 39 Ġarticles , Ġ10 Ġ000 Ġwords ); ĠWikipedia Ġ( 30 Ġarticles Ġ+ Ġ30 Ġsumm aries , Ġ242 Ġ000 Ġwords ); ĠProceedings Ġof Ġthe Ġconference ĠTra itement ĠAutom at ique Ġdes ĠLang ues ĠNature ll es Ġ( T AL N ) Ġ2008 Ġ( 25 Ġarticles , Ġ169 Ġ000 Ġwords ); ĠReports Ġfrom ĠInstit ut ĠFranÃ§ ais Ġde ĠRelations ĠInternational es Ġ( 32 Ġrap orts , Ġ266 Ġ000 Ġwords ). </s> The Ġcorpor a Ġwere Ġnoted Ġusing ĠGl o zz . </s> Ann od is Ġaims Ġat Ġbuilding Ġan Ġannot ated Ġcorpus . ĠThe Ġproposed Ġannotations Ġare Ġon Ġtwo Ġlevels Ġof Ġanalysis , Ġthat Ġis , Ġtwo Ġperspectives : </s> A sc endant : Ġpart Ġof ĠED U Ġare Ġused Ġin Ġthe Ġconstruction Ġof Ġmore Ġcomplex Ġstructures , Ġthrough Ġthe Ġrelations Ġof Ġdiscourse ; </s> Desc ending : Ġapproaches Ġthe Ġtext Ġin Ġits Ġentirety Ġand Ġrelies Ġon Ġthe Ġvarious Ġshallow Ġindices Ġto Ġidentify Ġhigh - level Ġdisc ursive Ġstructures Ġ( mac ro Ġstructures ). </s> Two Ġtypes Ġof Ġpersons Ġannot ated ĠAnn od is : Ġlinguistic Ġexperts Ġand Ġstudents . ĠThe Ġfirst Ġgroup Ġconstituted Ġa Ġ$ E $ Ġsub cor p us Ġcalled ĠâĢ ľ special ist âĢ Ŀ Ġand Ġthe Ġsecond Ġgroup Ġresulted Ġin Ġa Ġ$ N $ Ġsub cor p us Ġcalled ĠâĢ ľ na ive âĢ Ŀ . ĠThese Ġrhet orically Ġannot ated Ġsub cor ps Ġwere Ġused Ġas Ġreferences Ġin Ġour Ġexperiments . Ġ( c . f . ĠÂ§ SEC REF 6 ). </s> Disc ourse ĠSe gment er ĠOverall ĠDescription </s> The ĠFigure ĠFIG REF 12 Ġshows Ġthe Ġgeneral Ġarchitecture Ġof Ġthe Ġproposed Ġdiscourse Ġsegment er Ġsystem . ĠThe Ġinitial Ġinput Ġis Ġthe Ġraw Ġtext Ġencoded Ġin ĠUTF - 8 . ĠThe Ġtwo Ġinitial Ġprocesses Ġare ĠPart Ġof ĠSpeech Ġmorph os ynt actic ĠT agging Ġ( POS ) Ġand Ġthe Ġsegment ation Ġat Ġthe Ġlevel Ġof Ġthe Ġsentences . ĠThis Ġlast Ġis Ġjust Ġa Ġpre processing Ġstep Ġthat Ġsplits Ġsentences . ĠIn Ġthe Ġlast Ġprocess Ġthe Ġsystem Ġuses Ġa Ġbank Ġof Ġexplicit Ġmarkers Ġin Ġro der Ġto Ġapply Ġthe Ġrules Ġfor Ġthe Ġfinal Ġdiscourse Ġsegment ation . </s> For Ġthe Ġexperiments , Ġwe Ġused Ġlists Ġof Ġmarkers Ġin ĠFrench , ĠSpanish , ĠEnglish Ġand ĠPortuguese . ĠWe Ġalso Ġused Ġthe ĠLex icon n ĠB IB REF 10 Ġproject Ġlist , Ġwhich Ġreg roups Ġ328 ĠFrench - language Ġmarkers . ĠAnother Ġimportant Ġparameter Ġspecifies Ġwhich Ġsegment ation Ġstrategy Ġshould Ġbe Ġapplied , Ġaccording Ġto Ġthe ĠPOS Ġlab elling Ġof Ġthe Ġdocument . </s> Description Ġof Ġsegment ation Ġstrategies Ġ: :: ĠSe gment ation Ġwith Ġexplicit Ġuse Ġof Ġa Ġmarker </s> The Ġelementary Ġsystem ĠSe gment er $ _{ \ mu Ġ} $ Ġ( bas eline ) Ġrelies Ġsolely Ġon Ġa Ġlist Ġof Ġdisc ursive Ġmarkers Ġto Ġperform Ġthe Ġsegment ation . ĠIt Ġreplaces Ġthe Ġappearance Ġof Ġa Ġmarker Ġin Ġthe Ġlist Ġwith Ġa Ġspecial Ġsymbol , Ġfor Ġexample Ġ$\ mu Ġ$ , Ġwhich Ġindicates Ġa Ġboundary Ġbetween Ġthe Ġright Ġand Ġleft Ġsegment . ĠBe Ġthe Ġsentence Ġof Ġthe Ġpreceding Ġexample : ĠLa Ġv ille Ġd ' Av ign on Ġest Ġla Ġcapit ale Ġdu ĠV au cl use , Ġqui Ġest Ġun ĠdÃ© part ement Ġdu ĠSud Ġde Ġla ĠFrance .. ĠThe ĠSe gment er Ġsplit Ġthe Ġsentence Ġin Ġtwo Ġparts : Ġthe Ġleft Ġsegment Ġ( SE ), ĠLa Ġv ille Ġd ' Av ign on Ġest Ġla Ġcapit ale Ġdu ĠV au cl use , Ġand Ġthe Ġright Ġsegment Ġ( SD ), Ġest Ġun ĠdÃ© part ement Ġdu Ġsu d Ġde Ġla ĠFrance . </s> Description Ġof Ġsegment ation Ġstrategies Ġ: :: ĠSe gment ation Ġwith Ġexplicit Ġuse Ġof Ġa Ġmarker Ġand ĠPOS Ġlabels </s> The ĠSe gment er $ _{ \ mu Ġ+ } $ Ġsystem Ġpresents Ġan Ġimprovement Ġto Ġthe ĠSe gment er $ _{ \ mu Ġ} $ : Ġinclusion Ġof Ġgrammar Ġcategories Ġwith Ġthe ĠTree T agger Ġtool . ĠThe Ġadvantage Ġof Ġthis Ġsystem Ġis Ġthe Ġdetection Ġof Ġcertain Ġgram matical Ġforms Ġin Ġorder Ġto Ġcondition Ġthe Ġsegment ation . ĠSince Ġit Ġis Ġbased Ġon Ġthe ĠSe gment er $ _{ \ mu Ġ} $, Ġwe Ġtry Ġto Ġrecognise Ġthe Ġopp ort une Ġconditions Ġto Ġgather Ġtwo Ġsegments Ġwhen Ġboth Ġare Ġpart Ġof Ġthe Ġsame Ġdisc ursive Ġsegment . ĠWe Ġtry Ġto Ġidentify Ġmore Ġsubtly Ġwhen Ġit Ġis Ġpertinent Ġto Ġleave Ġthe Ġtwo Ġsegments Ġseparate . ĠThe ĠSe gment er $ _{ \ mu Ġ} $ Ġhas Ġtwo Ġdistinct Ġstrategies : </s> Seg ment ador $ _{ \ mu Ġ+ V } $ Ġ( verbal Ġversion , ĠV ): Ġit Ġrelies Ġsolely Ġon Ġthe Ġpresence Ġof Ġverbal Ġforms Ġto Ġthe Ġright Ġand Ġleft Ġof Ġthe Ġdisc ursive Ġmarker . ĠThe Ġtwo Ġgram matical Ġrules Ġof Ġthis Ġstrategy Ġare : </s> If Ġthere Ġare Ġno Ġverbs Ġin Ġthe Ġleft Ġand Ġright Ġsegments , Ġreg roup Ġthem . </s> If Ġthere Ġis Ġat Ġleast Ġone Ġverb Ġin Ġthe Ġleft Ġor Ġright Ġsegment , Ġthe Ġsegments Ġwill Ġremain Ġseparate . </s> Seg ment er $ _{ \ mu Ġ+ ( V - N )} $ Ġ( verb - n oun Ġversion , ĠV - N ): Ġit Ġrelies Ġon Ġthe Ġpresence Ġof Ġverbs Ġand Ġnoun s . ĠFor Ġthis Ġversion , Ġfour Ġrules Ġare Ġconsidered : </s> If Ġthere Ġis Ġno Ġnoun Ġin Ġeither Ġthe Ġleft Ġor Ġright Ġsegment , Ġwe Ġreg roup Ġthe Ġsegments . </s> We Ġreg roup Ġthe Ġsegments Ġif Ġat Ġleast Ġone Ġof Ġthem Ġhas Ġno Ġnoun . </s> If Ġat Ġleast Ġone Ġnoun Ġis Ġpresent Ġin Ġboth Ġsegments , Ġthey Ġremain Ġindependent . </s> If Ġthere Ġis Ġno Ġverb - nom inal Ġform , Ġthe Ġsegments Ġremain Ġindependent . </s> Exper iments </s> In Ġthis Ġfirst Ġexplor atory Ġwork , Ġonly Ġdocuments Ġin ĠFrench Ġwere Ġconsidered , Ġbut Ġthe Ġsystem Ġcan Ġbe Ġadapted Ġto Ġother Ġlanguages . ĠThe Ġevaluation Ġis Ġbased Ġon Ġthe Ġcorrespondence Ġof Ġword Ġpairs Ġrepresenting Ġa Ġborder . ĠIn Ġthis Ġway Ġwe Ġcompare Ġthe ĠAnn od is Ġsegment ation Ġwith Ġthe Ġautomatically Ġproduced Ġsegment ation . ĠFor Ġeach Ġpair Ġof Ġreference Ġsegments , Ġa Ġ$ L _ r $ Ġlist Ġof Ġword Ġpairs Ġis Ġprovided : Ġthe Ġlast Ġword Ġof Ġthe Ġfirst Ġsegment Ġand Ġthe Ġfirst Ġword Ġof Ġthe Ġsecond . </s> For Ġexample , Ġconsidering Ġthe Ġreference Ġtext Ġwik 1 _ 01 _ 02 - 04 - 2006 . se g , Ġfrom ĠAnn od is Ġcorpus : </s> [ Le ĠBan ĠAmendment ] $ _ 1 $ Ġ[ Apr Ã¨ s Ġav oir Ġadopt Ã© Ġla ĠConvention , ] _ 2 Ġ[ un Ġcertain Ġn omb re Ġde ĠP ED Ġet Ġd ' ass oci ations Ġde ĠdÃ© f ense Ġde Ġl ' en viron n ement Ġs out in rent ] _ 3 Ġ[ que Ġle Ġdocument Ġn ' all ait Ġpas Ġas se z Ġlo in .] _ 4 Ġ[ De Ġn omb re ux Ġpays Ġet ĠON G Ġmilit Ã¨ rent ] _ 5 Ġ[ en Ġf ave ur Ġd ' une Ġinter d iction Ġtot ale Ġde Ġl ' exp Ã© d ition Ġde ĠdÃ© che ts Ġd ange re ux ĠÃł Ġdestinations Ġdes ĠP ED .] _ 6 Ġ[ Plus Ġexact ement , ] _ 7 Ġ[ la ĠConvention Ġorigin ale Ġn ' inter dis ait Ġpas Ġl ' ex portation Ġde ĠdÃ© che ts , ] _ 8 Ġ[ except Ã© Ġvers Ġl ' Ant ar ct ique .] _ 9 Ġ[ El le Ġn ' ex ige ait ] _ 10 Ġ[ qu ' une Ġproc Ã© d ure Ġde Ġconsent ement Ġpr Ã© al able Ġen Ġcon naissance Ġde Ġcause ] _ 11 Ġ[( P IC , ĠPrior ĠIn formed ĠConsent ). ] _ 12 </s> Here Ġare Ġthe Ġword Ġpairs Ġof Ġthe Ġcreated Ġreference Ġlist Ġ( p unct uation Ġmarks Ġare Ġdisreg arded ): </s> $ L _ r $ ={ [ Con vention ĠâĢĵ Ġun ], Ġ[ s out in rent ĠâĢĵ Ġque ], Ġ[ lo in ĠâĢĵ Ġde ], Ġ[ m ilit Ã¨ rent ĠâĢĵ Ġen ], Ġ[ ex act ement ĠâĢĵ Ġla ], Ġ[ P ED ĠâĢĵ Ġplus ], Ġ[ ex act ement ĠâĢĵ Ġla ], Ġ[ d Ã© che ts ĠâĢĵ Ġexcept Ã© ], Ġ[ Ant art ique ĠâĢĵ ĠEl le ], Ġ[ ex ige ait ĠâĢĵ Ġqu ' une ], Ġ[ cause ĠâĢĵ ĠP IC ] Ġ} </s> We Ġdecided Ġto Ġcount Ġthe Ġword Ġpairs Ġinstead Ġof Ġthe Ġsegments , Ġas Ġthis Ġis Ġa Ġfirst Ġversion Ġof Ġthe Ġevaluation Ġprotocol . ĠIn Ġfact , Ġthe Ġsegments Ġmay Ġbe Ġnested , Ġwhich Ġcompl icates Ġthe Ġevaluation Ġprocess . ĠAlthough Ġthere Ġare Ġsome Ġerrors , Ġword Ġboundaries Ġallow Ġus Ġto Ġdetect Ġsegments Ġmore Ġeasily . </s> We Ġhave Ġbuilt Ġa Ġsecond Ġ$ L _ c $ Ġlist Ġfor Ġthe Ġautomatically Ġidentified Ġsegments , Ġfollowing Ġthe Ġsame Ġcriteria Ġof Ġ$ L _ r $. ĠThe Ġ$ L _ r $ Ġand Ġ$ L _ c $ Ġlists Ġreg roup , Ġpair Ġby Ġpair , Ġthe Ġsegment Ġborder . ĠWe Ġthen Ġcount Ġthe Ġcommon Ġpair Ġintersection Ġof Ġthe Ġtwo Ġlists . ĠEach Ġpair Ġin Ġthe Ġ$ L _ c $ Ġlist Ġis Ġalso Ġpresent Ġin Ġthe Ġ$ L _ r $ Ġreference Ġlist Ġand Ġis Ġa Ġcorrectly Ġassigned Ġto Ġthe Ġclass Ġpair . ĠA Ġword Ġpair Ġbelonging Ġto Ġthe Ġ$ L _ c $ Ġlist Ġbut Ġnot Ġbelonging Ġto Ġthe Ġ$ L _ r $ Ġreference Ġlist , Ġwill Ġbe Ġa Ġpair Ġassigned Ġto Ġthe Ġclass . </s> For Ġthat Ġsame Ġtext , Ġthe Ġ$ L _ c $ Ġlist Ġof Ġcandidate Ġpairs Ġobtained Ġwith Ġthe ĠSe gment ator $ _{ \ mu Ġ} $ Ġis : </s> $ L _ r $ ={ [ lo in âĢĵ De ], Ġ[ p ays âĢĵ et ], Ġ[ m ilit Ã¨ rent âĢĵ en ], Ġ[ d ange re ux âĢĵ Ãł ], Ġ[ P ED âĢĵ Plus ], Ġ[ Ant ar ct ique âĢĵ El le ], Ġ[ pr Ã© al able âĢĵ en ], Ġ[ cause âĢĵ P IC ] Ġ} </s> We Ġcalculate Ġthe Ġprecision Ġ$ P $, Ġthe Ġrecall Ġ$ R $ Ġand Ġthe Ġ$ F $ - score Ġon Ġthe Ġtext Ġcorpus Ġused Ġin Ġour Ġtests , Ġas Ġfollow : </s> The Ġprecision , Ġthe Ġrecall Ġand Ġthe Ġ$ F $ - score Ġfor Ġthis Ġexample Ġis : Ġ$ P $ Ġ= Ġ5 Ġ/ Ġ11 Ġ= Ġ0 . 45 ; Ġ$ R $ Ġ= Ġ5 Ġ/ Ġ8 Ġ= Ġ0 . 625 ; ĠF - score Ġ= Ġ2 Ġ$\ times Ġ\ frac { Ġ0 . 45 Ġ\ times Ġ0 . 625 }{ Ġ0 . 45 Ġ+ Ġ0 . 625 } Ġ= Ġ0 . 523 $. ĠWe Ġused Ġthe Ġdocuments Ġin Ġthe ĠAnn od is Ġcorpus Ġwithout Ġsegment ation , Ġbecause Ġthey Ġhad Ġbeen Ġsegment ed Ġwith Ġthe ĠSe gment er $ _{ \ mu Ġ} $ Ġand Ġwith Ġthe Ġgrammar Ġsegment ers . </s> Two Ġbatch Ġof Ġtests Ġwere Ġperformed . ĠThe Ġfirst Ġon Ġthe Ġ$ D $ Ġset Ġof Ġdocuments Ġcommon Ġto Ġthe Ġtwo Ġsub cor p us ĠâĢ ľ special ist âĢ Ŀ Ġ$ E $ Ġand ĠâĢ ľ na ive âĢ Ŀ Ġ$ N $ Ġfrom ĠAnn od is . Ġ$ D $ Ġcontains Ġ38 Ġdocuments Ġwith Ġ13 Ġ364 Ġwords . ĠThis Ġfirst Ġtest Ġallowed Ġto Ġmeasure Ġthe Ġdistance Ġbetween Ġthe Ġhuman Ġmarkers . ĠIn Ġfact , Ġin Ġorder Ġto Ġget Ġan Ġidea Ġof Ġthe Ġquality Ġof Ġthe Ġhuman Ġsegment ations , Ġthe Ġcuts Ġin Ġthe Ġtexts Ġmade Ġby Ġthe Ġspecialists Ġwere Ġmeasured Ġit Ġversus Ġthe Ġso - called ĠâĢ ľ na if s âĢ Ŀ Ġnote Ġt akers Ġand Ġvice Ġversa . ĠThe Ġsecond Ġseries Ġof Ġtests Ġconsisted Ġof Ġusing Ġall Ġthe Ġdocuments Ġof Ġthe Ġsub cor p us ĠâĢ ľ special ist âĢ Ŀ Ġ$ E $, Ġbecause Ġthe Ġdocuments Ġof Ġthe Ġsub cor p us Ġof ĠAnn od is Ġare Ġnot Ġidentical . ĠThen Ġwe Ġbenchmark ed Ġthe Ġperformance Ġof Ġthe Ġthree Ġsystems Ġautomatically . </s> Exper iments Ġ: :: ĠResults </s> In Ġthis Ġsection Ġwe Ġwill Ġcompare Ġthe Ġresults Ġof Ġthe Ġdifferent Ġsegment ation Ġsystems Ġthrough Ġautomatic Ġevaluations . ĠFirst Ġof Ġall , Ġthe Ġhuman Ġsegment ation , Ġfrom Ġthe Ġsub cor p us Ġ$ D $ Ġcomposed Ġof Ġcommon Ġdocuments . ĠThe Ġresults Ġare Ġpresented Ġin Ġthe Ġtable Ġtab : hum ains . ĠThe Ġfirst Ġrow Ġshows Ġthe Ġperformance Ġof Ġthe Ġ$ I $ Ġsegments , Ġtaking Ġthe Ġexperts Ġas Ġa Ġreference , Ġwhile Ġthe Ġsecond Ġpresents Ġthe Ġprocess Ġin Ġthe Ġopposite Ġdirection . </s> We Ġhave Ġfound Ġthat Ġsegment ation Ġby Ġexperts Ġand Ġnaive Ġproduces Ġtwo Ġsub cor p us Ġ$ E $ Ġand Ġ$ N $ Ġwith Ġvery Ġsimilar Ġcharacteristics . ĠThis Ġsurprised Ġus , Ġas Ġwe Ġexpected Ġa Ġmore Ġimportant Ġdifference Ġbetween Ġthem . ĠIn Ġany Ġcase , Ġwe Ġded uced Ġthat , Ġat Ġleast Ġin Ġthis Ġcorpus , Ġit Ġis Ġnot Ġnecessary Ġto Ġbe Ġan Ġexpert Ġin Ġlingu istics Ġto Ġdisc urs ively Ġsegment Ġthe Ġdocuments . ĠAs Ġfar Ġas Ġsystem Ġevaluations Ġare Ġconcerned , Ġwe Ġuse Ġthe Ġ78 Ġ$ E $ Ġdocuments Ġas Ġreference . ĠTable ĠT AB REF 26 Ġshows Ġthe Ġresults . </s> In Ġthe Ġcase Ġof Ġthe ĠExperts , Ġthe Ġgram matical Ġverb - nom inal Ġversion Ġ( V - N ) Ġhad Ġbetter ĠF - score Ġperformance . ĠThe Ġverbal Ġversion Ġ( V ) Ġobtained Ġa Ġbetter Ġaccuracy Ġ$ P $ Ġthan Ġthe Ġverb - nom inal Ġ( V - N ). ĠIn Ġthe Ġcase Ġof Ġthe ĠNa ive , Ġthe Ġperformance ĠF - score , Ġ$ P $ Ġand Ġ$ R $ Ġis Ġvery Ġsimilar Ġfrom Ġthe ĠExperts . </s> Con clusions , Ġdiscussion Ġand Ġperspectives </s> The Ġaim Ġof Ġthis Ġwork Ġwas Ġtwo fold : Ġto Ġdesign Ġa Ġdisc ursive Ġsegment er Ġusing Ġa Ġminimum Ġof Ġresources Ġand Ġto Ġestablish Ġan Ġevaluation Ġprotocol Ġto Ġmeasure Ġthe Ġperformance Ġof Ġsegment ers . ĠThe Ġresults Ġshow Ġthat Ġwe Ġcan Ġbuild Ġa Ġsimple Ġversion Ġof Ġthe Ġbaseline , Ġwhich Ġemploys Ġonly Ġa Ġlist Ġof Ġmarkers Ġand Ġpresents Ġa Ġvery Ġencouraging Ġperformance . ĠOf Ġcourse , Ġthe Ġquality Ġof Ġthe Ġlist Ġis Ġa Ġprep onder ant Ġfactor Ġfor Ġa Ġcorrect Ġsegment ation . </s> We Ġhave Ġstudied Ġthe Ġimpact Ġof Ġthe Ġmarker Ġwhich , Ġeven Ġthough Ġit Ġmay Ġseem Ġfringe - worthy , Ġcontributes Ġto Ġimproving Ġthe Ġperformance Ġof Ġour Ġsegment ers . ĠThus , Ġit Ġis Ġan Ġinteresting Ġmarker Ġthat Ġwe Ġcan Ġconsider Ġas Ġa Ġdisc ursive Ġmarker . ĠThe ĠSe gment ator $ _{ \ mu Ġ} $ Ġversion Ġprovides Ġthe Ġbest Ġresults Ġin Ġterms Ġof ĠF - score Ġand Ġrecall , Ġfollowed Ġby Ġthe ĠSe gment ator $ _{ \ mu Ġ+ V } $ Ġversion , Ġwhich Ġpasses Ġit Ġin Ġprecision . ĠRegarding Ġevaluation , Ġwe Ġdeveloped Ġa Ġsimple Ġprotocol Ġto Ġcompare Ġthe Ġperformance Ġof Ġthe Ġsystems . ĠThis Ġis , Ġto Ġour Ġknowledge , Ġthe Ġfirst Ġautomatic Ġevaluation Ġin ĠFrench . </s> It Ġis Ġnecessary Ġto Ġintensify Ġour Ġresearch Ġin Ġorder Ġto Ġpropose Ġimprovements Ġto Ġour Ġsegment ers , Ġas Ġwell Ġas Ġto Ġstudy Ġfurther Ġthe Ġimpact Ġof Ġgrammar Ġtag Ġrules Ġon Ġsegment ation . ĠSince Ġwe Ġhave Ġa Ġstandard Ġevaluation Ġprotocol , Ġwe Ġintend Ġto Ġcarry Ġout Ġtests Ġwith ĠPortuguese , ĠSpanish Ġ( see ĠB IB REF 11 ), ĠEnglish , Ġetc . ĠFor Ġthat , Ġwe Ġwill Ġonly Ġneed Ġa Ġlist Ġof Ġmarkers Ġfor Ġeach Ġlanguage . </s> The Ġperformance Ġof Ġthe Ġsystems Ġremains Ġmodest , Ġof Ġcourse , Ġbut Ġwe Ġmust Ġnot Ġforget Ġthat Ġthis Ġis Ġa Ġbaseline Ġand Ġits Ġprimary Ġobjective Ġis Ġto Ġprovide Ġstandard Ġsystems Ġthat Ġcan Ġbe Ġused Ġin Ġtesting Ġprotocols Ġsuch Ġas Ġthe Ġone Ġwe Ġproposed . ĠDespite Ġthis Ġevolution , Ġthese Ġbas elines Ġ( or Ġtheir Ġimproved Ġversions ) Ġcan Ġbe Ġused Ġin Ġapplications Ġsuch Ġas Ġautomatic Ġdocument Ġsummar isation Ġ( e . g ., ĠB IB REF 12 , ĠB IB REF 13 ), Ġor Ġsentences Ġcompression ĠB IB REF 14 . </s> The Ġmain Ġfeature Ġof Ġthe Ġproposed Ġbaseline Ġsystem Ġis Ġits Ġflexibility Ġwith Ġrespect Ġto Ġthe Ġlanguage Ġconsidered . ĠIn Ġfact , Ġit Ġonly Ġuses Ġa Ġlist Ġof Ġlanguage Ġmarkers Ġand Ġthe Ġgram matical Ġcategory Ġof Ġwords . ĠThe Ġfirst Ġresource , Ġalthough Ġdependent Ġon Ġeach Ġlanguage , Ġis Ġrelatively Ġeasy Ġto Ġobtain . ĠWe Ġhave Ġfound Ġthat , Ġeven Ġwith Ġlists Ġof Ġmoderate Ġsize , Ġthe Ġresults Ġare Ġquite Ġsignificant . ĠThe Ġgram matical Ġcategories Ġwere Ġobtained Ġwith Ġthe Ġhelp Ġof Ġthe ĠTree T agger Ġstatistics Ġtool . ĠHowever , ĠTree T agger Ġcould Ġbe Ġreplaced Ġby Ġany Ġother Ġtool Ġproducing Ġsimilar Ġresults . </s> App endix </s> In Ġthis Ġappendix , Ġwe Ġpresent Ġthe Ġlist Ġof Ġrhetorical Ġconnectors Ġin ĠFrench Ġthat Ġconstitute Ġour Ġlist Ġof Ġmarkers . ĠWe Ġpoint Ġout Ġthat Ġthe Ġmarkers Ġending Ġin Ġapost rophe Ġsuch Ġas : </s> pr Ã¨ s Ġqu ', ĠÃł Ġcondition Ġd ', Ġetc . </s> are Ġdeleted Ġfrom Ġa Ġregular Ġexpression Ġimplying Ġ' and ': Ġpr Ã¨ s Ġqu ' Ġ+ Ġpr Ã¨ s Ġque , ĠÃł Ġcondition Ġd ' Ġ+ ĠÃł Ġcondition Ġde , Ġetc . </s> 3 </s> , Ġ/ ĠÃł Ġ/ ĠÃł ĠÃ § a Ġpr Ã¨ s Ġqu ' Ġ/ ĠÃł Ġc ec i Ġpr Ã¨ s Ġqu ' Ġ/ ĠÃł Ġc ela Ġpr Ã¨ s Ġqu ' Ġ/ ĠÃł Ġce Ġmoment - l Ãł Ġ/ ĠÃł Ġce Ġpoint Ġqu ' Ġ/ ĠÃł Ġce Ġpropos Ġ/ ĠÃł Ġc et ĠÃ© gard Ġ/ ĠÃł Ġcondition Ġd ' Ġ/ ĠÃł Ġcondition Ġqu ' Ġ/ ĠÃł ĠdÃ© f aut Ġd ' Ġ/ ĠÃł ĠdÃ© f aut Ġde Ġ/ ĠÃł Ġdire Ġv ra i Ġ/ ĠÃł ĠÃ© lab orer Ġ/ ĠÃł Ġen Ġ/ Ġaf in Ġd ' Ġ/ Ġaf in Ġqu ' Ġ/ Ġaf in Ġque Ġ/ ĠÃł Ġforce Ġ/ ĠÃł Ġforce Ġd ' Ġ/ Ġa ins i Ġ/ ĠÃł Ġla Ġplace Ġ/ ĠÃł Ġla ĠrÃ© flex ion Ġ/ ĠÃł Ġl ' Ã© po que Ġo Ã ¹ Ġ/ ĠÃł Ġl ' he ure Ġo Ã ¹ Ġ/ ĠÃł Ġl ' inst ant Ġo Ã ¹ Ġ/ ĠÃł Ġl ' in verse Ġ/ Ġal ors Ġ/ Ġal ors Ġm Ãª me Ġqu ' Ġ/ Ġal ors Ġqu ' Ġ/ ĠÃł Ġmes ure Ġqu ' Ġ/ ĠÃł Ġmo ins Ġd ' Ġ/ ĠÃł Ġmo ins Ġqu ' Ġ/ ĠÃł Ġpart ĠÃ § a Ġ/ ĠÃł Ġpart ir Ġdu Ġmoment Ġo Ã ¹ Ġ/ ĠÃł Ġpart Ġqu ' Ġ/ Ġapr Ã¨ s Ġ/ ĠÃł Ġpr Ã©s ent Ġqu ' Ġ/ Ġapr Ã¨ s Ġqu ' Ġ/ Ġapr Ã¨ s Ġquo i Ġ/ Ġapr Ã¨ s Ġt out Ġ/ ĠÃł Ġpre uve Ġ/ ĠÃł Ġpropos Ġ/ ĠÃł Ġse ule Ġfin Ġd ' Ġ/ ĠÃł Ġse ule Ġfin Ġqu ' Ġ/ ĠÃł Ġsupp oser Ġqu ' Ġ/ ĠÃł Ġt elle Ġen se igne Ġqu ' Ġ/ ĠÃł Ġtel Ġpoint Ġqu ' Ġ/ Ġattend u Ġqu ' Ġ/ Ġau Ġbout Ġdu Ġcom pt e Ġ/ Ġau Ġcas Ġo Ã ¹ Ġ/ Ġau Ġcontra ire Ġ/ Ġau Ġf ait Ġ/ Ġau Ġfur Ġet ĠÃł Ġmes ure Ġqu ' Ġ/ Ġau Ġlieu Ġ/ Ġau Ġlieu Ġd ' Ġ/ Ġau Ġm Ãª me Ġtit re Ġqu ' Ġ/ Ġau Ġmo ins Ġ/ Ġau Ġmoment Ġd ' Ġ/ Ġau Ġmoment Ġo Ã ¹ Ġa up ar av ant Ġ/ Ġau Ġpoint Ġd ' Ġ/ Ġau Ġpoint Ġqu ' Ġ/ Ġa uss i Ġ/ Ġa uss i Ġlong tem ps Ġqu ' Ġ/ Ġa uss it Ã´ t Ġ/ Ġa uss it Ã´ t Ġqu ' Ġ/ Ġaut ant Ġ/ Ġaut ant Ġdire Ġqu ' Ġ/ Ġau Ġtotal Ġ/ Ġaut re ment Ġ/ Ġaut re ment Ġd it Ġ/ Ġav ant Ġ/ Ġav ant Ġd ' Ġ/ Ġav ant Ġm Ãª me Ġd ' Ġ/ Ġav ant Ġm Ãª me Ġqu ' Ġ/ Ġav ant Ġqu ' Ġ/ ĠÃł Ġv ra i Ġdire Ġ/ Ġb ien Ġqu ' Ġ/ Ġb ient Ã´ t Ġ/ Ġbre f Ġ/ Ġcar Ġ/ Ġc ec i Ġd it Ġ/ Ġc ec i Ġ Ã©t ant Ġd it Ġ/ Ġc ela Ġd it Ġ/ Ġc epend ant Ġ/ Ġc epend ant Ġqu ' Ġ/ Ġc ' est ĠÃł Ġdire Ġqu ' Ġ/ Ġc ' est Ġpour qu oi Ġ/ Ġc ette Ġf ois Ġqu ' Ġ/ Ġcomm e Ġ/ Ġcomm e ĠÃ § a Ġ/ Ġcomm e Ġquo i Ġ/ Ġcomm e Ġsi Ġ/ Ġcomparative ment Ġ/ Ġcons Ã© qu em ment Ġ/ Ġcons id Ã© rant Ġqu ' Ġ/ Ġcons id Ã© rÃ© Ġqu ' Ġ/ Ġcor rÃ© l ative ment Ġ/ Ġd ' ab ord Ġ/ Ġd ' ail le urs Ġ/ Ġd ans Ġce Ġcas Ġ/ Ġd ans Ġce Ġcas - l Ãł Ġ/ Ġd ans Ġla Ġmes ure Ġo Ã ¹ Ġ/ Ġd ans Ġle Ġbut Ġd ' Ġ/ Ġd ans Ġle Ġbut Ġqu ' Ġ/ Ġd ans Ġle Ġcas Ġo Ã ¹ Ġd ans Ġle Ġcoup Ġ/ Ġd ans Ġle Ġsens Ġo Ã ¹ Ġ/ Ġd ans Ġle Ġsens Ġqu ' Ġ/ Ġd ans Ġl ' es po ir Ġd ' Ġ/ Ġd ans Ġl ' es po ir Ġqu ' Ġ/ Ġd ans Ġl ' hyp oth Ã¨ se Ġo Ã ¹ Ġ/ Ġd ans Ġl ' intention Ġd ' Ġ/ Ġd ans Ġl ' intention Ġqu ' Ġ/ Ġd ans Ġt ous Ġles Ġcas Ġ/ Ġd ' aut ant Ġplus Ġqu ' Ġ/ Ġd ' aut ant Ġqu ' Ġ/ Ġd ' aut re Ġpart Ġ/ Ġde Ġce Ġf ait Ġ/ ĠdÃ© c id Ã© ment Ġ/ Ġde Ġfa Ã§ on ĠÃł Ġ/ Ġde Ġfa Ã§ on ĠÃł Ġce Ġqu ' Ġ/ Ġde Ġfa Ã§ on Ġqu ' Ġ/ Ġde Ġf ait Ġ/ ĠdÃ© j Ãł Ġ/ ĠdÃ© j Ãł Ġqu ' Ġ/ Ġde Ġla Ġm Ãª me Ġfa Ã§ on Ġ/ Ġde Ġla Ġm Ãª me Ġfa Ã§ on Ġqu ' Ġ/ Ġde Ġla Ġm Ãª me Ġman i Ã¨re Ġ/ Ġde Ġla Ġm Ãª me Ġman i Ã¨re Ġqu ' Ġ/ Ġde Ġman i Ã¨re ĠÃł Ġ/ Ġde Ġman i Ã¨re ĠÃł Ġce Ġqu ' Ġ/ Ġde Ġman i Ã¨re Ġqu ' Ġ/ Ġde Ġm Ãª me Ġ/ Ġde Ġm Ãª me Ġqu ' Ġ/ Ġde Ġplus Ġ/ Ġdep u is Ġ/ Ġdep u is Ġqu ' Ġ/ Ġdes Ġf ois Ġqu ' Ġ/ Ġd Ã¨ s Ġl ors Ġ/ Ġd Ã¨ s Ġl ors Ġqu ' Ġ/ Ġde Ġsort e Ġqu ' Ġ/ Ġd Ã¨ s Ġqu ' Ġ/ Ġde Ġt elle Ġfa Ã§ on Ġqu ' Ġ/ Ġde Ġt elle Ġman i Ã¨re Ġqu ' Ġ/ Ġde Ġt oute Ġfa Ã§ on Ġ/ Ġde Ġt oute Ġman i Ã¨re Ġ/ Ġde Ġt out es Ġfa Ã§ ons Ġ/ Ġde Ġt out es Ġman i Ã¨ res Ġ/ Ġd ' ici Ġqu ' Ġ/ Ġdire Ġqu ' Ġ/ Ġdon c Ġ/ Ġd ' o Ã ¹ Ġ/ Ġd ' o Ã ¹ Ġqu ' Ġ/ Ġdu Ġcoup Ġ/ Ġdu Ġf ait Ġqu ' Ġ/ Ġdu Ġmo ins Ġ/ Ġdu Ġmoment Ġqu ' Ġ/ Ġd ' un Ġaut re Ġc Ã´ t Ã© Ġd ' un Ġc Ã´ t Ã© Ġ/ Ġd ' un Ġcoup Ġ/ Ġd ' une Ġpart Ġ/ Ġd ' un Ġse ul Ġcoup Ġ/ Ġdu Ġrest e Ġ/ Ġdu Ġtem ps Ġo Ã ¹ Ġ/ Ġeffective ment Ġ/ ĠÃ© gal ement Ġ/ Ġen Ġ/ Ġen Ġadm ett ant Ġqu ' Ġ/ Ġen Ġattendant Ġ/ Ġen Ġbre f Ġ/ Ġen Ġce Ġcas Ġ/ Ġen Ġce Ġsens Ġqu ' Ġ/ Ġen Ġcomp ara ison Ġ/ Ġen Ġcons Ã© qu ence Ġ/ Ġenc ore Ġ/ Ġenc ore Ġqu ' Ġ/ Ġen Ġd ' aut res Ġterm es Ġ/ Ġen ĠdÃ© fin itive Ġ/ Ġen ĠdÃ© pit Ġdu Ġf ait Ġqu ' Ġ/ Ġen ĠdÃ© pit Ġqu ' Ġ/ Ġen Ġeff et Ġ/ Ġen Ġf ait Ġ/ Ġen fin Ġ/ Ġen Ġg ros Ġ/ Ġen Ġm Ãª me Ġtem ps Ġ/ Ġen Ġm Ãª me Ġtem ps Ġqu ' Ġ/ Ġen Ġout re Ġ/ Ġen Ġpartic ul ier Ġ/ Ġen Ġplus Ġ/ Ġen Ġplus Ġd ' Ġ/ Ġen Ġplus Ġde Ġ/ Ġen ĠrÃ© al itÃ© Ġ/ Ġen ĠrÃ© sum Ã© Ġ/ Ġen Ġrev anche Ġ/ Ġen Ġsom me Ġ/ Ġensu ite Ġ/ Ġen Ġsupp os ant Ġqu ' Ġ/ Ġen Ġt ous Ġcas Ġen Ġt ous Ġles Ġcas Ġ/ Ġen Ġt out Ġcas Ġ/ Ġen Ġt out Ġ Ã©t at Ġde Ġcause Ġ/ Ġen Ġv Ã© rit Ã© Ġ/ Ġen Ġv ue Ġd ' Ġ/ Ġet Ġ/ Ġ Ã©t ant Ġdon n Ã© Ġqu ' Ġ/ Ġet Ġdire Ġqu ' Ġ/ Ġet Ġpu is Ġ/ Ġexcept Ã© Ġqu ' Ġ/ Ġfa ute Ġd ' Ġ/ Ġfinal ement Ġ/ Ġglobal ement Ġ/ Ġhist oire Ġd ' Ġ/ Ġhorm is Ġle Ġf ait Ġqu ' Ġ/ Ġhorm is Ġqu ' Ġ/ Ġinstant an Ã© ment Ġ/ Ġinverse ment Ġ/ Ġj us qu ' Ãł Ġ/ Ġj us qu ' Ãł Ġce Ġqu ' Ġ/ Ġla Ġpre uve Ġ/ Ġle Ġf ait Ġest Ġqu ' Ġ/ Ġle Ġjour Ġo Ã ¹ Ġ/ Ġle Ġtem ps Ġqu ' Ġ/ Ġl ors qu ' Ġ/ Ġmain ten ant Ġ/ Ġmain ten ant Ġqu ' Ġ/ Ġm ais Ġ/ Ġmal gr Ã© Ġle Ġf ait Ġqu ' Ġ/ Ġmal gr Ã© Ġqu ' Ġ/ Ġmal gr Ã© Ġt out Ġ/ Ġmal he ure use ment Ġ/ Ġm Ãª me Ġ/ Ġm Ãª me Ġqu ' Ġ/ Ġm Ãª me Ġsi Ġ/ Ġm ie ux Ġ/ Ġmis ĠÃł Ġpart Ġle Ġf ait Ġqu ' Ġ/ Ġmis ĠÃł Ġpart Ġqu ' Ġ/ Ġn Ã© an mo ins Ġ/ Ġnon ob stant Ġ/ Ġnon ob stant Ġqu ' Ġ/ Ġor Ġ/ Ġo u Ġ/ Ġo u Ġb ien Ġ/ Ġout re Ġqu ' Ġ/ Ġpar Ġa ille urs Ġ/ Ġpar all Ã¨ lement Ġ/ Ġpar ce Ġqu ' Ġ/ Ġpar Ġcomp ara ison Ġ/ Ġpar Ġcons Ã© qu ent Ġ/ Ġpar Ġcont re Ġ/ Ġpar - d ess us Ġt out Ġ/ Ġpar Ġex em ple Ġ/ Ġpar Ġle Ġf ait Ġqu ' Ġ/ Ġpar Ġsuite Ġ/ Ġp endant Ġqu ' Ġ/ Ġpe u Ġimport e Ġplus Ġqu ' Ġ/ Ġplus Ġt ard Ġpl ut Ã´ t Ġ/ Ġpl ut Ã´ t Ġqu ' Ġ/ Ġpl ut Ã´ t Ġque Ġd ' Ġ/ Ġpour Ġ/ Ġpour Ġaut ant Ġpour Ġaut ant Ġqu ' Ġ/ Ġpour Ġcomm encer Ġ/ Ġpour Ġcon cl ure Ġ/ Ġpour Ġfin ir Ġ/ Ġpour Ġle Ġcoup Ġ/ Ġpour Ġpe u Ġqu ' Ġ/ Ġpour Ġpre uve Ġ/ Ġpour Ġqu ' Ġ/ Ġpour Ġr Ã©s umer Ġ/ Ġpour t ant Ġ/ Ġpour Ġtermin er Ġ/ Ġpour Ġune Ġf ois Ġqu ' Ġ/ Ġpour vu Ġqu ' Ġ/ Ġprem i Ã¨re ment Ġ/ Ġpre uve Ġqu ' Ġ/ Ġpu is Ġ/ Ġpu is qu ' Ġ/ Ġqu and Ġ/ Ġqu and Ġb ien Ġm Ãª me Ġ/ Ġqu and Ġb ien Ġm Ãª me Ġqu ' Ġ/ Ġqu and Ġm Ãª me Ġ/ Ġquant ĠÃł Ġ/ Ġqu itte ĠÃł Ġ/ Ġqu itte ĠÃł Ġce Ġqu ' Ġ/ Ġquo iqu ' Ġ/ Ġquo i Ġqu ' il Ġen Ġso it Ġ/ ĠrÃ© c ip ro qu ement Ġ/ ĠrÃ© flex ion Ġfa ite Ġ/ Ġrem ar que Ġ/ Ġr Ã©s ult at Ġ/ Ġs ' Ġ/ Ġs ach ant Ġqu ' Ġ/ Ġsans Ġ/ Ġsans Ġcomp ter Ġqu ' Ġ/ Ġsans Ġo ub lier Ġqu ' Ġ/ Ġsans Ġqu ' Ġ/ Ġsa uf ĠÃł Ġ/ Ġsa uf Ġqu ' Ġ/ Ġse lon Ġqu ' Ġ/ Ġsi Ġ/ Ġsi Ġb ien Ġqu ' Ġ/ Ġsi Ġce Ġn ' est Ġqu ' Ġ/ Ġsim ultan Ã© ment Ġ/ Ġsin on Ġ/ Ġsin on Ġqu ' Ġ/ Ġsi Ġtant Ġest Ġqu ' Ġ/ Ġsit Ã´ t Ġqu ' Ġ/ Ġso it Ġ/ Ġso it Ġd it Ġen Ġpass ant Ġ/ Ġsom me Ġt oute Ġ/ Ġs oud ain Ġ/ Ġsubs Ã© qu em ment Ġ/ Ġsu iv ant Ġqu ' Ġ/ Ġsur t out Ġ/ Ġsur t out Ġqu ' Ġ/ Ġt and is Ġqu ' Ġ/ Ġtant Ġet Ġsi Ġb ien Ġqu ' Ġ/ Ġtant Ġqu ' Ġ/ Ġtotal Ġ/ Ġt out ĠÃł Ġcoup Ġ/ Ġt out Ġau Ġmo ins Ġ/ Ġt out Ġb ien Ġcons id Ã© rÃ© Ġ/ Ġt out Ġcom pt e Ġf ait Ġ/ Ġt out Ġd ' ab ord Ġ/ Ġt out Ġde Ġm Ãª me Ġ/ Ġt out Ġen Ġ/ Ġune Ġf ois Ġqu ' Ġ/ Ġun Ġjour Ġ/ Ġun Ġjour Ġqu ' Ġ/ Ġun Ġpe u Ġplus Ġt ard Ġ/ Ġv u Ġqu ' Ġ/