Document to Vector (Doc2Vec)
EXTRACTIVE
Doc2Vec, PV-DBOW model
EXTRACTIVE
Figure 1 shows a summary of our pipeline. First, we pre-process each document to transform text into consecutive word tokens, where words are in their most normalised forms, and some words are removed if they have no distinctive meaning when used out of context BIBREF5 , BIBREF6 . We then train a paragraph vector model using the Document to Vector (Doc2Vec) framework BIBREF7 on the whole set (13 million) of preprocessed text records, although training on smaller sets (1 million) also produces good results. This training step is only done once. This Doc2Vec model is subsequently used to infer high-dimensional vector descriptions for the text of each of the 3229 documents in our target analysis set. We then compute a matrix containing pairwise similarities between any pair of document vectors, as inferred with Doc2Vec. This matrix can be thought of as a full, weighted graph with documents as nodes and edges weighted by their similarity. We sparsify this graph to the union of a minimum spanning tree and a k-Nearest Neighbors (MST-kNN) graph BIBREF8 , a geometric construction that removes less important similarities but preserves global connectivity for the graph and, hence, for the dataset. The derived MST-kNN graph is analysed with Markov Stability BIBREF9 , BIBREF10 , BIBREF11 , BIBREF12 , a multi-resolution dynamics-based graph partitioning method that identifies relevant subgraphs (i.e., clusters of documents) at different levels of granularity. MS uses a diffusive process on the graph to reveal the multiscale organisation at different resolutions without the need for choosing a priori the number of clusters, scale or organisation. To analyse a posteriori the different partitions across levels of resolution, we use both visualisations and quantitative scores. The visualisations include word clouds to summarise the main content, graph layouts, as well as Sankey diagrams and contingency tables that capture the correspondences across levels of resolution and relationships to the hand-coded classifications. The partitions are also evaluated quantitatively to score: (i) their intrinsic topic coherence (using pairwise mutual information BIBREF13 , BIBREF14 ), and (ii) their similarity to the operator hand-coded categories (using normalised mutual information BIBREF15 ). We now expand on the steps of the computational framework.
Here, we use the Gensim Python library BIBREF23 to train the PV-DBOW model. The Doc2Vec training was repeated several times with a variety of training hyper-parameters to optimise the output based on our own numerical experiments and the general guidelines provided by BIBREF24 . We trained Doc2Vec models using text corpora of different sizes and content with different sets of hyper-parameters, in order to characterise the usability and quality of models. Specifically, we checked the effect of corpus size on model quality by training Doc2Vec models on the full 13 million NRLS records and on subsets of 1 million and 2 million randomly sampled records. (We note that our target subset of 3229 records has been excluded from these samples.) Furthermore, we checked the importance of the specificity of the text corpus by obtaining a Doc2Vec model from a generic, non-specific set of 5 million articles from Wikipedia representing standard English usage across a variety of topics.
1811.05711
Introduction </s> The Ġvast Ġamounts Ġof Ġdata Ġcollected Ġby Ġhealthcare Ġproviders Ġin Ġconjunction Ġwith Ġmodern Ġdata Ġanalytics Ġtechniques Ġpresent Ġa Ġunique Ġopportunity Ġto Ġimprove Ġhealth Ġservice Ġprovision Ġand Ġthe Ġquality Ġand Ġsafety Ġof Ġmedical Ġcare Ġfor Ġpatient Ġbenefit ĠB IB REF 0 Ġ. ĠMuch Ġof Ġthe Ġrecent Ġresearch Ġin Ġthis Ġarea Ġhas Ġbeen Ġon Ġpersonal ised Ġmedicine Ġand Ġits Ġaim Ġto Ġdeliver Ġbetter Ġdiagn ostics Ġaided Ġby Ġthe Ġintegration Ġof Ġdiverse Ġdatasets Ġproviding Ġcomplementary Ġinformation . ĠAnother Ġlarge Ġsource Ġof Ġhealthcare Ġdata Ġis Ġorganis ational . ĠIn Ġthe ĠUnited ĠKingdom , Ġthe ĠNational ĠHealth ĠService Ġ( N HS ) Ġhas Ġa Ġlong Ġhistory Ġof Ġdocumenting Ġextensively Ġthe Ġdifferent Ġaspects Ġof Ġhealthcare Ġprovision . ĠThe ĠNHS Ġis Ġcurrently Ġin Ġthe Ġprocess Ġof Ġincreasing Ġthe Ġavailability Ġof Ġseveral Ġdatabases , Ġproperly Ġanonym ised , Ġwith Ġthe Ġaim Ġof Ġleveraging Ġadvanced Ġanalytics Ġto Ġidentify Ġareas Ġof Ġimprovement Ġin ĠNHS Ġservices . </s> One Ġsuch Ġdatabase Ġis Ġthe ĠNational ĠReporting Ġand ĠLearning ĠSystem Ġ( NR LS ), Ġa Ġcentral Ġrepository Ġof Ġpatient Ġsafety Ġincident Ġreports Ġfrom Ġthe ĠNHS Ġin ĠEngland Ġand ĠWales . ĠSet Ġup Ġin Ġ2003 , Ġthe ĠNR LS Ġnow Ġcontains Ġmore Ġthan Ġ13 Ġmillion Ġdetailed Ġrecords . ĠThe Ġincidents Ġare Ġreported Ġusing Ġa Ġset Ġof Ġstandard ised Ġcategories Ġand Ġcontain Ġa Ġwealth Ġof Ġorganis ational Ġand Ġspat io - tem poral Ġinformation Ġ( struct ured Ġdata ), Ġas Ġwell Ġas , Ġcru cially , Ġa Ġsubstantial Ġcomponent Ġof Ġfree Ġtext Ġ( un struct ured Ġdata ) Ġwhere Ġincidents Ġare Ġdescribed Ġin Ġthe Ġ` voice ' Ġof Ġthe Ġperson Ġreporting . ĠThe Ġincidents Ġare Ġwide Ġranging : Ġfrom Ġpatient Ġaccidents Ġto Ġlost Ġforms Ġor Ġreferrals ; Ġfrom Ġdelays Ġin Ġadmission Ġand Ġdischarge Ġto Ġserious Ġunt oward Ġincidents , Ġsuch Ġas Ġretained Ġforeign Ġobjects Ġafter Ġoperations . ĠThe Ġreview Ġand Ġanalysis Ġof Ġsuch Ġdata Ġprovides Ġcritical Ġinsight Ġinto Ġthe Ġcomplex Ġfunctioning Ġof Ġdifferent Ġprocesses Ġand Ġprocedures Ġin Ġhealthcare Ġtowards Ġservice Ġimprovement Ġfor Ġsafer Ġcare r . </s> Although Ġstatistical Ġanalyses Ġare Ġroutinely Ġperformed Ġon Ġthe Ġstructured Ġcomponent Ġof Ġthe Ġdata Ġ( dates , Ġlocations , Ġassigned Ġcategories , Ġetc ), Ġthe Ġfree Ġtext Ġremains Ġlargely Ġunused Ġin Ġsystematic Ġprocesses . ĠFree Ġtext Ġis Ġusually Ġread Ġmanually Ġbut Ġthis Ġis Ġtime - consuming , Ġmeaning Ġthat Ġit Ġis Ġoften Ġignored Ġin Ġpractice , Ġunless Ġa Ġdetailed Ġreview Ġof Ġa Ġcase Ġis Ġundertaken Ġbecause Ġof Ġthe Ġseverity Ġof Ġharm Ġthat Ġresulted . ĠThere Ġis Ġa Ġlack Ġof Ġmethod ologies Ġthat Ġcan Ġsummar ise Ġcontent Ġand Ġprovide Ġcontent - based Ġgroup ings Ġacross Ġthe Ġlarge Ġvolume Ġof Ġreports Ġsubmitted Ġnationally Ġfor Ġorganis ational Ġlearning . ĠMethods Ġthat Ġcould Ġprovide Ġautomatic Ġcategor isation Ġof Ġincidents Ġfrom Ġthe Ġfree Ġtext Ġwould Ġsid est ep Ġproblems Ġsuch Ġas Ġdifficulties Ġin Ġassigning Ġan Ġincident Ġcategory Ġby Ġvirtue Ġof Ġa Ġprior i Ġpre - defined Ġlists Ġin Ġthe Ġreporting Ġsystem Ġor Ġhuman Ġerror , Ġas Ġwell Ġas Ġoffering Ġa Ġunique Ġinsight Ġinto Ġthe Ġroot Ġcause Ġanalysis Ġof Ġincidents Ġthat Ġcould Ġimprove Ġthe Ġsafety Ġand Ġquality Ġof Ġcare Ġand Ġefficiency Ġof Ġhealthcare Ġservices . </s> Our Ġgoal Ġin Ġthis Ġwork Ġis Ġto Ġshowcase Ġan Ġalgorith mic Ġmethodology Ġthat Ġdetects Ġcontent - based Ġgroups Ġof Ġrecords Ġin Ġa Ġgiven Ġdataset Ġin Ġan Ġun super vised Ġmanner , Ġbased Ġonly Ġon Ġthe Ġfree Ġand Ġun struct ured Ġtextual Ġdescription Ġof Ġthe Ġincidents . ĠTo Ġdo Ġso , Ġwe Ġcombine Ġrecently Ġdeveloped Ġdeep Ġneural - network Ġhigh - dimensional Ġtext - embed ding Ġalgorithms Ġwith Ġnetwork - the oret ical Ġmethods . ĠIn Ġparticular , Ġwe Ġapply Ġmult isc ale ĠMark ov ĠStability Ġ( MS ) Ġcommunity Ġdetection Ġto Ġa Ġsp ars ified Ġgeometric Ġsimilarity Ġgraph Ġof Ġdocuments Ġobtained Ġfrom Ġtext Ġvector Ġsimilarities . ĠOur Ġmethod Ġdep arts Ġfrom Ġtraditional Ġnatural Ġlanguage Ġprocessing Ġtools , Ġwhich Ġhave Ġgenerally Ġused Ġbag - of - words Ġ( Bo W ) Ġrepresentation Ġof Ġdocuments Ġand Ġstatistical Ġmethods Ġbased Ġon ĠLat ent ĠDir ich let ĠAll ocation Ġ( L DA ) Ġto Ġcluster Ġdocuments ĠB IB REF 1 Ġ. ĠMore Ġrecent Ġapproaches Ġhave Ġused Ġdeep Ġneural Ġnetwork Ġbased Ġlanguage Ġmodels Ġclustered Ġwith Ġk - me ans , Ġwithout Ġa Ġfull Ġmult isc ale Ġgraph Ġanalysis ĠB IB REF 2 Ġ. ĠThere Ġhave Ġbeen Ġsome Ġprevious Ġapplications Ġof Ġnetwork Ġtheory Ġto Ġtext Ġanalysis . ĠFor Ġexample , ĠL anch ich in etti Ġand Ġco - workers ĠB IB REF 3 Ġused Ġa Ġprob abil istic Ġgraph Ġconstruction Ġanalysed Ġwith Ġthe ĠInfo Map Ġalgorithm ĠB IB REF 4 Ġ; Ġhowever , Ġtheir Ġcommunity Ġdetection Ġwas Ġcarried Ġout Ġat Ġa Ġsingle - scale Ġand Ġthe Ġrepresentation Ġof Ġtext Ġas ĠBo W Ġarrays Ġlacks Ġthe Ġpower Ġof Ġneural Ġnetwork Ġtext Ġembed d ings . ĠThe Ġapplication Ġof Ġmult isc ale Ġcommunity Ġdetection Ġallows Ġus Ġto Ġfind Ġgroups Ġof Ġrecords Ġwith Ġconsistent Ġcontent Ġat Ġdifferent Ġlevels Ġof Ġresolution ; Ġhence Ġthe Ġcontent Ġcategories Ġemerge Ġfrom Ġthe Ġtextual Ġdata , Ġrather Ġthan Ġfitting Ġwith Ġpre - designed Ġclass ifications . ĠThe Ġobtained Ġresults Ġcould Ġthus Ġhelp Ġmitigate Ġpossible Ġhuman Ġerror Ġor Ġeffort Ġin Ġfinding Ġthe Ġright Ġcategory Ġin Ġcomplex Ġcategory Ġclassification Ġtrees . </s> We Ġshowcase Ġthe Ġmethodology Ġthrough Ġthe Ġanalysis Ġof Ġa Ġdataset Ġof Ġpatient Ġincidents Ġreported Ġto Ġthe ĠNR LS . ĠFirst , Ġwe Ġuse Ġthe Ġ13 Ġmillion Ġrecords Ġcollected Ġby Ġthe ĠNR LS Ġsince Ġ2004 Ġto Ġtrain Ġour Ġtext Ġembed ding Ġ( although Ġa Ġmuch Ġsmaller Ġcorpus Ġcan Ġbe Ġused ). ĠWe Ġthen Ġanalyse Ġa Ġsubset Ġof Ġ3 229 Ġrecords Ġreported Ġfrom ĠSt ĠMary 's ĠHospital , ĠLondon Ġ( Im perial ĠCollege ĠHealthcare ĠNHS ĠTrust ) Ġover Ġthree Ġmonths Ġin Ġ2014 Ġto Ġextract Ġclusters Ġof Ġincidents Ġat Ġdifferent Ġlevels Ġof Ġresolution Ġin Ġterms Ġof Ġcontent . ĠOur Ġmethod Ġreveals Ġmultiple Ġlevels Ġof Ġintrinsic Ġstructure Ġin Ġthe Ġtopics Ġof Ġthe Ġdataset , Ġas Ġshown Ġby Ġthe Ġextraction Ġof Ġrelevant Ġword Ġdescript ors Ġfrom Ġthe Ġgrouped Ġrecords Ġand Ġa Ġhigh Ġlevel Ġof Ġtopic Ġco herence . ĠOriginally , Ġthe Ġrecords Ġhad Ġbeen Ġmanually Ġcoded Ġby Ġthe Ġoperator Ġupon Ġreporting Ġwith Ġup Ġto Ġ170 Ġfeatures Ġper Ġcase , Ġincluding Ġa Ġtwo - level Ġmanual Ġclassification Ġof Ġthe Ġincidents . ĠTherefore , Ġwe Ġalso Ġcarried Ġout Ġan Ġa Ġposterior i Ġcomparison Ġagainst Ġthe Ġhand - coded Ġcategories Ġassigned Ġby Ġthe Ġreporter Ġ( health care Ġpersonnel ) Ġat Ġthe Ġtime Ġof Ġthe Ġreport Ġsubmission . ĠOur Ġresults Ġshow Ġgood Ġoverall Ġcorrespondence Ġwith Ġthe Ġhand - coded Ġcategories Ġacross Ġresolutions Ġand , Ġspecifically , Ġat Ġthe Ġmedium Ġlevel Ġof Ġgran ularity . ĠSeveral Ġof Ġour Ġclusters Ġof Ġcontent Ġcorrespond Ġstrongly Ġto Ġwell - defined Ġcategories , Ġyet Ġour Ġresults Ġalso Ġreveal Ġcomplementary Ġcategories Ġof Ġincidents Ġnot Ġdefined Ġin Ġthe Ġexternal Ġclassification . ĠIn Ġaddition , Ġthe Ġtuning Ġof Ġthe Ġgran ularity Ġafforded Ġby Ġthe Ġmethod Ġcan Ġbe Ġused Ġto Ġprovide Ġa Ġdistinct Ġlevel Ġof Ġresolution Ġin Ġcertain Ġareas Ġcorresponding Ġto Ġspecial ise Ġor Ġparticular Ġsub - the mes . </s> Mult isc ale Ġgraph Ġpartition ing Ġfor Ġtext Ġanalysis : Ġdescription Ġof Ġthe Ġframework </s> Our Ġframework Ġcombines Ġtext - embed ding , Ġgeometric Ġgraph Ġconstruction Ġand Ġmulti - resolution Ġcommunity Ġdetection Ġto Ġidentify , Ġrather Ġthan Ġimpose , Ġcontent - based Ġclusters Ġfrom Ġfree , Ġun struct ured Ġtext Ġin Ġan Ġun super vised Ġmanner . </s> Figure Ġ1 Ġshows Ġa Ġsummary Ġof Ġour Ġpipeline . ĠFirst , Ġwe Ġpre - process Ġeach Ġdocument Ġto Ġtransform Ġtext Ġinto Ġconsecutive Ġword Ġtokens , Ġwhere Ġwords Ġare Ġin Ġtheir Ġmost Ġnormal ised Ġforms , Ġand Ġsome Ġwords Ġare Ġremoved Ġif Ġthey Ġhave Ġno Ġdistinctive Ġmeaning Ġwhen Ġused Ġout Ġof Ġcontext ĠB IB REF 5 Ġ, ĠB IB REF 6 Ġ. ĠWe Ġthen Ġtrain Ġa Ġparagraph Ġvector Ġmodel Ġusing Ġthe ĠDocument Ġto ĠVector Ġ( Doc 2 V ec ) Ġframework ĠB IB REF 7 Ġon Ġthe Ġwhole Ġset Ġ( 13 Ġmillion ) Ġof Ġpre process ed Ġtext Ġrecords , Ġalthough Ġtraining Ġon Ġsmaller Ġsets Ġ( 1 Ġmillion ) Ġalso Ġproduces Ġgood Ġresults . ĠThis Ġtraining Ġstep Ġis Ġonly Ġdone Ġonce . ĠThis ĠDoc 2 V ec Ġmodel Ġis Ġsubsequently Ġused Ġto Ġinfer Ġhigh - dimensional Ġvector Ġdescriptions Ġfor Ġthe Ġtext Ġof Ġeach Ġof Ġthe Ġ3 229 Ġdocuments Ġin Ġour Ġtarget Ġanalysis Ġset . ĠWe Ġthen Ġcompute Ġa Ġmatrix Ġcontaining Ġpair wise Ġsimilarities Ġbetween Ġany Ġpair Ġof Ġdocument Ġvectors , Ġas Ġinferred Ġwith ĠDoc 2 V ec . ĠThis Ġmatrix Ġcan Ġbe Ġthought Ġof Ġas Ġa Ġfull , Ġweighted Ġgraph Ġwith Ġdocuments Ġas Ġnodes Ġand Ġedges Ġweighted Ġby Ġtheir Ġsimilarity . ĠWe Ġsp ars ify Ġthis Ġgraph Ġto Ġthe Ġunion Ġof Ġa Ġminimum Ġspanning Ġtree Ġand Ġa Ġk - Ne arest ĠNeigh bors Ġ( M ST - k NN ) Ġgraph ĠB IB REF 8 Ġ, Ġa Ġgeometric Ġconstruction Ġthat Ġremoves Ġless Ġimportant Ġsimilarities Ġbut Ġpreserves Ġglobal Ġconnectivity Ġfor Ġthe Ġgraph Ġand , Ġhence , Ġfor Ġthe Ġdataset . ĠThe Ġderived ĠM ST - k NN Ġgraph Ġis Ġanalysed Ġwith ĠMark ov ĠStability ĠB IB REF 9 Ġ, ĠB IB REF 10 Ġ, ĠB IB REF 11 Ġ, ĠB IB REF 12 Ġ, Ġa Ġmulti - resolution Ġdynamics - based Ġgraph Ġpartition ing Ġmethod Ġthat Ġidentifies Ġrelevant Ġsub graph s Ġ( i . e ., Ġclusters Ġof Ġdocuments ) Ġat Ġdifferent Ġlevels Ġof Ġgran ularity . ĠMS Ġuses Ġa Ġdiff usive Ġprocess Ġon Ġthe Ġgraph Ġto Ġreveal Ġthe Ġmult isc ale Ġorganisation Ġat Ġdifferent Ġresolutions Ġwithout Ġthe Ġneed Ġfor Ġchoosing Ġa Ġprior i Ġthe Ġnumber Ġof Ġclusters , Ġscale Ġor Ġorganisation . ĠTo Ġanalyse Ġa Ġposterior i Ġthe Ġdifferent Ġpartitions Ġacross Ġlevels Ġof Ġresolution , Ġwe Ġuse Ġboth Ġvisual isations Ġand Ġquantitative Ġscores . ĠThe Ġvisual isations Ġinclude Ġword Ġclouds Ġto Ġsummar ise Ġthe Ġmain Ġcontent , Ġgraph Ġlayouts , Ġas Ġwell Ġas ĠSan key Ġdiagrams Ġand Ġcontingency Ġtables Ġthat Ġcapture Ġthe Ġcorrespond ences Ġacross Ġlevels Ġof Ġresolution Ġand Ġrelationships Ġto Ġthe Ġhand - coded Ġclass ifications . ĠThe Ġpartitions Ġare Ġalso Ġevaluated Ġquant itatively Ġto Ġscore : Ġ( i ) Ġtheir Ġintrinsic Ġtopic Ġco herence Ġ( using Ġpair wise Ġmutual Ġinformation ĠB IB REF 13 Ġ, ĠB IB REF 14 Ġ), Ġand Ġ( ii ) Ġtheir Ġsimilarity Ġto Ġthe Ġoperator Ġhand - coded Ġcategories Ġ( using Ġnormal ised Ġmutual Ġinformation ĠB IB REF 15 Ġ). ĠWe Ġnow Ġexpand Ġon Ġthe Ġsteps Ġof Ġthe Ġcomputational Ġframework . </s> Data Ġdescription </s> The Ġfull Ġdataset Ġincludes Ġmore Ġthan Ġ13 Ġmillion Ġconfidential Ġreports Ġof Ġpatient Ġsafety Ġincidents Ġreported Ġto Ġthe ĠNational ĠReporting Ġand ĠLearning ĠSystem Ġ( NR LS ) Ġbetween Ġ2004 Ġand Ġ2016 Ġfrom ĠNHS Ġtrusts Ġand Ġhospitals Ġin ĠEngland Ġand ĠWales . ĠEach Ġrecord Ġhas Ġmore Ġthan Ġ170 Ġfeatures , Ġincluding Ġorganis ational Ġdetails Ġ( e . g ., Ġtime , Ġtrust Ġcode Ġand Ġlocation ), Ġanonym ised Ġpatient Ġinformation , Ġmedication Ġand Ġmedical Ġdevices , Ġamong Ġother Ġdetails . ĠThe Ġrecords Ġare Ġmanually Ġclassified Ġby Ġoperators Ġto Ġa Ġtwo - level Ġsystem Ġof Ġcategories Ġof Ġincident Ġtype . ĠIn Ġparticular , Ġthe Ġtop Ġlevel Ġcontains Ġ15 Ġcategories Ġincluding Ġgeneral Ġgroups Ġsuch Ġas Ġ` Pat ient Ġaccident ', Ġ` Med ication ', Ġ` Cl inical Ġassessment ', Ġ` Document ation ', Ġ` Ad missions / Transfer ' Ġor Ġ` Inf rastructure ' Ġalongside Ġmore Ġspecific Ġgroups Ġsuch Ġas Ġ` Agg ressive Ġbehaviour ', Ġ` Pat ient Ġabuse ', Ġ` Self - harm ' Ġor Ġ` In fect ion Ġcontrol '. ĠIn Ġmost Ġrecords , Ġthere Ġis Ġalso Ġa Ġdetailed Ġdescription Ġof Ġthe Ġincident Ġin Ġfree Ġtext , Ġalthough Ġthe Ġquality Ġof Ġthe Ġtext Ġis Ġhighly Ġvariable . ĠOur Ġanalysis Ġset Ġfor Ġclust ering Ġis Ġthe Ġgroup Ġof Ġ3 229 Ġrecords Ġreported Ġduring Ġthe Ġfirst Ġquarter Ġof Ġ2014 Ġat ĠSt . ĠMary 's ĠHospital Ġin ĠLondon Ġ( Im perial ĠCollege ĠHealthcare ĠNHS ĠTrust ). </s> Text ĠPre processing </s> Text Ġpre processing Ġis Ġimportant Ġto Ġenhance Ġthe Ġperformance Ġof Ġtext Ġembed ding . ĠWe Ġapplied Ġstandard Ġpre processing Ġtechniques Ġin Ġnatural Ġlanguage Ġprocessing Ġto Ġthe Ġraw Ġtext Ġof Ġall Ġ13 Ġmillion Ġrecords Ġin Ġour Ġcorpus . ĠWe Ġnormal ise Ġwords Ġinto Ġa Ġsingle Ġform Ġand Ġremove Ġwords Ġthat Ġdo Ġnot Ġcarry Ġsignificant Ġmeaning . ĠSpecifically , Ġwe Ġdivide Ġour Ġdocuments Ġinto Ġiter ative Ġword Ġtokens Ġusing Ġthe ĠNL T K Ġlibrary ĠB IB REF 5 Ġand Ġremove Ġpunct uation Ġand Ġdigit - only Ġtokens . ĠWe Ġthen Ġapply Ġword Ġstemming Ġusing Ġthe ĠPorter Ġalgorithm ĠB IB REF 6 Ġ, ĠB IB REF 16 Ġ. ĠIf Ġthe ĠPorter Ġmethod Ġcannot Ġfind Ġa Ġstemmed Ġversion Ġfor Ġa Ġtoken , Ġwe Ġapply Ġthe ĠSnow ball Ġalgorithm ĠB IB REF 17 Ġ. ĠFinally , Ġwe Ġremove Ġany Ġstop - words Ġ( repeat Ġwords Ġwith Ġlow Ġcontent ) Ġusing ĠNL T K 's Ġstop - word Ġlist . ĠAlthough Ġsome Ġof Ġthe Ġsynt actic Ġinformation Ġis Ġreduced Ġdue Ġto Ġtext Ġpre processing , Ġthis Ġprocess Ġpreserves Ġand Ġconsolid ates Ġthe Ġsemantic Ġinformation Ġof Ġthe Ġvocabulary , Ġwhich Ġis Ġof Ġrelevance Ġto Ġour Ġstudy . </s> Text ĠEmb edd ing </s> Com put ational Ġmethods Ġfor Ġtext Ġanalysis Ġrely Ġon Ġa Ġchoice Ġof Ġa Ġmathematical Ġrepresentation Ġof Ġthe Ġbase Ġunits , Ġsuch Ġas Ġcharacter Ġ$ n $ Ġ- gram s , Ġwords Ġor Ġdocuments Ġof Ġany Ġlength . ĠAn Ġimportant Ġconsideration Ġfor Ġour Ġmethodology Ġis Ġan Ġattempt Ġto Ġavoid Ġthe Ġuse Ġof Ġlabelled Ġdata Ġat Ġthe Ġcore Ġof Ġmany Ġsupervised Ġor Ġsemi - super vised Ġclassification Ġmethods ĠB IB REF 18 Ġ, ĠB IB REF 19 Ġ. ĠIn Ġthis Ġwork , Ġwe Ġuse Ġa Ġrepresentation Ġof Ġtext Ġdocuments Ġin Ġvector Ġform Ġfollowing Ġrecent Ġdevelopments Ġin Ġthe Ġfield . </s> Class ically , Ġbag - of - words Ġ( Bow ) Ġmethods Ġwere Ġused Ġto Ġobtain Ġrepresentations Ġof Ġthe Ġdocuments Ġin Ġa Ġcorpus Ġin Ġterms Ġof Ġvectors Ġof Ġterm Ġfrequencies Ġweighted Ġby Ġinverse Ġdocument Ġfrequency Ġ( TF - i DF ). ĠWhile Ġsuch Ġmethods Ġprovide Ġa Ġstatistical Ġdescription Ġof Ġdocuments , Ġthey Ġdo Ġnot Ġcarry Ġinformation Ġabout Ġthe Ġorder Ġor Ġproximity Ġof Ġwords Ġto Ġeach Ġother Ġsince Ġthey Ġregard Ġword Ġtokens Ġin Ġan Ġindependent Ġmanner Ġwith Ġno Ġsemantic Ġor Ġsynt actic Ġrelationships Ġconsidered . ĠFurthermore , ĠBo W Ġrepresentations Ġtend Ġto Ġbe Ġhigh - dimensional Ġand Ġsparse , Ġdue Ġto Ġlarge Ġsizes Ġof Ġword Ġdiction aries Ġand Ġlow Ġfrequencies Ġof Ġmany Ġterms . </s> Recently , Ġdeep Ġneural Ġnetwork Ġlanguage Ġmodels Ġhave Ġsuccessfully Ġovercome Ġcertain Ġlimitations Ġof ĠBo W Ġmethods Ġby Ġincorporating Ġword Ġneighbourhoods Ġin Ġthe Ġmathematical Ġdescription Ġof Ġeach Ġterm . ĠDist ributed ĠBag Ġof ĠWords Ġ( DB OW ) Ġis Ġa Ġform Ġof ĠPar agraph ĠV ect ors Ġ( P V ), Ġalso Ġknown Ġas ĠDoc 2 V ec ĠB IB REF 7 Ġ. ĠThis Ġmethod Ġcreates Ġa Ġmodel Ġwhich Ġrepresents Ġany Ġlength Ġof Ġword Ġsequences Ġ( i . e . Ġsentences , Ġparagraphs , Ġdocuments ) Ġas Ġ$ d $ Ġ- dimensional Ġvectors , Ġwhere Ġ$ d $ Ġis Ġa Ġuser - defined Ġparameter Ġ( typically Ġ$ d = 500 $ Ġ). ĠTraining Ġa ĠDoc 2 V ec Ġmodel Ġstarts Ġwith Ġa Ġrandom Ġ$ d $ Ġ- dimensional Ġvector Ġassignment Ġfor Ġeach Ġdocument Ġin Ġthe Ġcorpus . ĠA Ġsto ch astic Ġgradient Ġdescent Ġalgorithm Ġiter ates Ġover Ġthe Ġcorpus Ġwith Ġthe Ġobjective Ġof Ġpredicting Ġa Ġrandomly Ġsampled Ġset Ġof Ġwords Ġfrom Ġeach Ġdocument Ġby Ġusing Ġonly Ġthe Ġdocument 's Ġ$ d $ Ġ- dimensional Ġvector ĠB IB REF 7 Ġ. ĠThe Ġobjective Ġfunction Ġbeing Ġoptim ised Ġby ĠPV - DB OW Ġis Ġsimilar Ġto Ġthe Ġskip - gram Ġmodel Ġin ĠRef s . ĠB IB REF 20 Ġ, ĠB IB REF 21 Ġ. ĠDoc 2 V ec Ġhas Ġbeen Ġshown ĠB IB REF 22 Ġto Ġcapture Ġboth Ġsemantic Ġand Ġsynt actic Ġcharacter isations Ġof Ġthe Ġinput Ġtext Ġoutper forming ĠBo W Ġmodels , Ġsuch Ġas ĠL DA ĠB IB REF 1 Ġ. </s> Here , Ġwe Ġuse Ġthe ĠG ens im ĠPython Ġlibrary ĠB IB REF 23 Ġto Ġtrain Ġthe ĠPV - DB OW Ġmodel . ĠThe ĠDoc 2 V ec Ġtraining Ġwas Ġrepeated Ġseveral Ġtimes Ġwith Ġa Ġvariety Ġof Ġtraining Ġhyper - param eters Ġto Ġoptim ise Ġthe Ġoutput Ġbased Ġon Ġour Ġown Ġnumerical Ġexperiments Ġand Ġthe Ġgeneral Ġguidelines Ġprovided Ġby ĠB IB REF 24 Ġ. ĠWe Ġtrained ĠDoc 2 V ec Ġmodels Ġusing Ġtext Ġcorpor a Ġof Ġdifferent Ġsizes Ġand Ġcontent Ġwith Ġdifferent Ġsets Ġof Ġhyper - param eters , Ġin Ġorder Ġto Ġcharacter ise Ġthe Ġusability Ġand Ġquality Ġof Ġmodels . ĠSpecifically , Ġwe Ġchecked Ġthe Ġeffect Ġof Ġcorpus Ġsize Ġon Ġmodel Ġquality Ġby Ġtraining ĠDoc 2 V ec Ġmodels Ġon Ġthe Ġfull Ġ13 Ġmillion ĠNR LS Ġrecords Ġand Ġon Ġsubs ets Ġof Ġ1 Ġmillion Ġand Ġ2 Ġmillion Ġrandomly Ġsampled Ġrecords . Ġ( We Ġnote Ġthat Ġour Ġtarget Ġsubset Ġof Ġ3 229 Ġrecords Ġhas Ġbeen Ġexcluded Ġfrom Ġthese Ġsamples .) ĠFurthermore , Ġwe Ġchecked Ġthe Ġimportance Ġof Ġthe Ġspecificity Ġof Ġthe Ġtext Ġcorpus Ġby Ġobtaining Ġa ĠDoc 2 V ec Ġmodel Ġfrom Ġa Ġgeneric , Ġnon - specific Ġset Ġof Ġ5 Ġmillion Ġarticles Ġfrom ĠWikipedia Ġrepresenting Ġstandard ĠEnglish Ġusage Ġacross Ġa Ġvariety Ġof Ġtopics . </s> We Ġbenchmark ed Ġthe ĠDoc 2 V ec Ġmodels Ġby Ġscoring Ġhow Ġwell Ġthe Ġdocument Ġvectors Ġrepresent Ġthe Ġsemantic Ġtopic Ġstructure : Ġ( i ) Ġcalculating Ġcent ro ids Ġfor Ġthe Ġ15 Ġexternally Ġhand - coded Ġcategories ; Ġ( ii ) Ġselecting Ġthe Ġ100 Ġnearest Ġreports Ġfor Ġeach Ġcent roid ; Ġ( iii ) Ġcounting Ġthe Ġnumber Ġof Ġincident Ġreports Ġ( out Ġof Ġ1500 ) Ġcorrectly Ġassigned Ġto Ġtheir Ġcent roid . ĠThe Ġresults Ġin ĠTable Ġ1 Ġshow Ġthat Ġtraining Ġon Ġthe Ġhighly Ġspecific Ġtext Ġin Ġthe ĠNR LS Ġrecords Ġis Ġan Ġimportant Ġingredient Ġin Ġthe Ġsuccessful Ġvector isation Ġof Ġthe Ġdocuments , Ġas Ġshown Ġby Ġthe Ġdegraded Ġperformance Ġfor Ġthe ĠWikipedia Ġmodel Ġacross Ġa Ġvariety Ġof Ġtraining Ġhyper - param eters . ĠOur Ġresults Ġalso Ġshow Ġthat Ġreducing Ġthe Ġsize Ġof Ġthe Ġcorpus Ġfrom Ġ13 Ġmillion Ġto Ġ1 Ġmillion Ġrecords Ġdid Ġnot Ġaffect Ġthe Ġbenchmark ing Ġdramatically . ĠThis Ġrobust ness Ġof Ġthe Ġresults Ġto Ġthe Ġsize Ġof Ġthe Ġtraining Ġcorpus Ġwas Ġconfirmed Ġfurther Ġwith Ġthe Ġuse Ġof Ġmore Ġdetailed Ġmetrics , Ġas Ġdiscussed Ġbelow Ġin ĠSection Ġsec : com par isons . </s> Based Ġon Ġour Ġbenchmark ing , Ġwe Ġuse Ġhence forth Ġ( unless Ġotherwise Ġnoted ) Ġthe Ġoptim ised ĠDoc 2 V ec Ġmodel Ġobtained Ġfrom Ġthe Ġ13 + Ġmillion ĠNR LS Ġrecords Ġwith Ġthe Ġfollowing Ġhyper - param eters : Ġ{ training Ġmethod Ġ= Ġd bow , Ġnumber Ġof Ġdimensions Ġfor Ġfeature Ġvectors Ġsize Ġ= Ġ300 , Ġnumber Ġof Ġepoch s Ġ= Ġ10 , Ġwindow Ġsize Ġ= Ġ15 , Ġminimum Ġcount Ġ= Ġ5 , Ġnumber Ġof Ġnegative Ġsamples Ġ= Ġ5 , Ġrandom Ġdown - sam pling Ġthreshold Ġfor Ġfrequent Ġwords Ġ= Ġ0 . 001 Ġ} . ĠAs Ġan Ġindication Ġof Ġcomputational Ġcost , Ġthe Ġtraining Ġof Ġthe Ġmodel Ġon Ġthe Ġ13 Ġmillion Ġrecords Ġtakes Ġapproximately Ġ11 Ġhours Ġ( run Ġin Ġparallel Ġwith Ġ7 Ġthreads ) Ġon Ġshared Ġservers . </s> Graph ĠConstruction </s> Once Ġthe ĠDoc 2 V ec Ġmodel Ġis Ġtrained , Ġwe Ġuse Ġit Ġto Ġinfer Ġa Ġvector Ġfor Ġeach Ġof Ġthe Ġ$ N = 3 229 $ Ġrecords Ġin Ġour Ġanalysis Ġset . ĠWe Ġthen Ġconstruct Ġa Ġnormal ised Ġcos ine Ġsimilarity Ġmatrix Ġbetween Ġthe Ġvectors Ġby : Ġcomputing Ġthe Ġmatrix Ġof Ġcos ine Ġsimilarities Ġbetween Ġall Ġpairs Ġof Ġrecords , Ġ$ S _ \ text { cos } $ Ġ; Ġtransforming Ġit Ġinto Ġa Ġdistance Ġmatrix Ġ$ D _{ cos } Ġ= Ġ1 - S _{ cos } $ Ġ; Ġapplying Ġelement - wise Ġmax Ġnorm Ġto Ġobtain Ġ$\ hat { D } = \ Vert ĠD _{ cos }\ Vert Ġ_ { max } $ Ġ; Ġand Ġnormal ising Ġthe Ġsimilarity Ġmatrix Ġ$\ hat { S } Ġ= Ġ1 - \ hat { D } $ Ġwhich Ġhas Ġelements Ġin Ġthe Ġinterval Ġ$ [ 0 , 1 ] $ Ġ. </s> The Ġsimilarity Ġmatrix Ġcan Ġbe Ġthought Ġof Ġas Ġthe Ġadj ac ency Ġmatrix Ġof Ġa Ġfully Ġconnected Ġweighted Ġgraph . ĠHowever , Ġsuch Ġa Ġgraph Ġcontains Ġmany Ġedges Ġwith Ġsmall Ġweights Ġreflecting Ġweak Ġsimilarities Ġin Ġhigh - dimensional Ġnoisy Ġdatasets Ġeven Ġthe Ġleast Ġsimilar Ġnodes Ġpresent Ġa Ġsubstantial Ġdegree Ġof Ġsimilarity . ĠSuch Ġweak Ġsimilarities Ġare Ġin Ġmost Ġcases Ġredundant , Ġas Ġthey Ġcan Ġbe Ġexplained Ġthrough Ġstronger Ġpair wise Ġsimilarities Ġpresent Ġin Ġthe Ġgraph . ĠThese Ġweak , Ġredundant Ġedges Ġobscure Ġthe Ġgraph Ġstructure , Ġas Ġshown Ġby Ġthe Ġdiffuse , Ġspherical Ġvisual isation Ġof Ġthe Ġfull Ġgraph Ġlayout Ġin ĠFigure Ġ2 ĠA . </s> To Ġreveal Ġthe Ġgraph Ġstructure , Ġwe Ġobtain Ġa ĠM ST - k NN Ġgraph Ġfrom Ġthe Ġnormal ised Ġsimilarity Ġmatrix ĠB IB REF 8 Ġ. ĠThis Ġis Ġa Ġsimple Ġsp ars ification Ġbased Ġon Ġa Ġgeometric Ġhe uristic Ġthat Ġpreserves Ġthe Ġglobal Ġconnectivity Ġof Ġthe Ġgraph Ġwhile Ġretaining Ġdetails Ġabout Ġthe Ġlocal Ġgeometry Ġof Ġthe Ġdataset . ĠThe ĠM ST - k NN Ġalgorithm Ġstarts Ġby Ġcomputing Ġthe Ġminimum Ġspanning Ġtree Ġ( M ST ) Ġof Ġthe Ġfull Ġmatrix Ġ$\ hat { D } $ Ġ, Ġi . e ., Ġthe Ġtree Ġwith Ġ$( N - 1 ) $ Ġedges Ġconnecting Ġall Ġnodes Ġin Ġthe Ġgraph Ġwith Ġminimal Ġsum Ġof Ġedge Ġweights Ġ( dist ances ). ĠThe ĠM ST Ġis Ġcomputed Ġusing Ġthe ĠKr usk al Ġalgorithm Ġimplemented Ġin ĠSci Py ĠB IB REF 25 Ġ. ĠTo Ġthis ĠM ST , Ġwe Ġadd Ġedges Ġconnecting Ġeach Ġnode Ġto Ġits Ġ$ k $ Ġnearest Ġnodes Ġ( k NN ) Ġif Ġthey Ġare Ġnot Ġalready Ġin Ġthe ĠM ST . ĠHere Ġ$ k $ Ġis Ġan Ġuser - defined Ġparameter . ĠThe Ġbinary Ġadj ac ency Ġmatrix Ġof Ġthe ĠM ST - k NN Ġgraphs , Ġ$ E _ \ text { M ST - k NN } $ Ġ, Ġis ĠHad am ard - multipl ied Ġwith Ġ$\ hat { S } $ Ġto Ġgive Ġthe Ġadj ac ency Ġmatrix Ġ$ A $ Ġof Ġthe Ġweighted , Ġund irect ed Ġsp ars ified Ġgraph . ĠThe ĠM ST - k NN Ġmethod Ġavoids Ġa Ġdirect Ġthreshold ing Ġof Ġthe Ġweights Ġin Ġ$\ hat { S } $ Ġ, Ġand Ġob tains Ġa Ġgraph Ġdescription Ġthat Ġpreserves Ġlocal Ġgeometric Ġinformation Ġtogether Ġwith Ġa Ġglobal Ġsub graph Ġ( the ĠM ST ) Ġthat Ġcaptures Ġproperties Ġof Ġthe Ġfull Ġdataset . </s> The Ġnetwork Ġlayout Ġvisual isations Ġin ĠFigure Ġ2 ĠB âĢĵ E Ġgive Ġan Ġintuitive Ġpicture Ġof Ġthe Ġeffect Ġof Ġthe Ġsp ars ification . ĠThe Ġhighly Ġsparse Ġgraphs Ġobtained Ġwhen Ġthe Ġnumber Ġof Ġneighbours Ġ$ k $ Ġis Ġvery Ġsmall Ġare Ġnot Ġrobust . ĠAs Ġ$ k $ Ġis Ġincreased , Ġthe Ġlocal Ġsimilarities Ġbetween Ġdocuments Ġinduce Ġthe Ġformation Ġof Ġdense Ġsub graph s Ġ( which Ġappear Ġcloser Ġin Ġthe Ġgraph Ġvisual isation Ġlayout ). ĠWhen Ġthe Ġnumber Ġof Ġneighbours Ġbecomes Ġtoo Ġlarge , Ġthe Ġlocal Ġstructure Ġbecomes Ġdiffuse Ġand Ġthe Ġsub graph s Ġlose Ġco herence , Ġsignalling Ġthe Ġdegradation Ġof Ġthe Ġlocal Ġgraph Ġstructure . ĠFigure Ġ2 Ġshows Ġthat Ġthe ĠM ST - k NN Ġgraph Ġwith Ġ$ k = 13 $ Ġpresents Ġa Ġreasonable Ġbalance Ġbetween Ġlocal Ġand Ġglobal Ġstructure . ĠRel atively Ġsparse Ġgraphs Ġthat Ġpreserve Ġimportant Ġedges Ġand Ġglobal Ġconnectivity Ġof Ġthe Ġdataset Ġ( gu arant eed Ġhere Ġby Ġthe ĠM ST ) Ġhave Ġcomputational Ġadvantages Ġwhen Ġusing Ġcommunity Ġdetection Ġalgorithms . </s> The ĠM ST - k NN Ġconstruction Ġhas Ġbeen Ġreported Ġto Ġbe Ġrobust Ġto Ġthe Ġselection Ġof Ġthe Ġparameter Ġ$ k $ Ġdue Ġto Ġthe Ġguaranteed Ġconnectivity Ġprovided Ġby Ġthe ĠM ST ĠB IB REF 8 Ġ. ĠIn Ġthe Ġfollowing , Ġwe Ġfix Ġ$ k = 13 $ Ġfor Ġour Ġanalysis Ġwith Ġthe Ġmulti - scale Ġgraph Ġpartition ing Ġframework , Ġbut Ġwe Ġhave Ġscanned Ġvalues Ġof Ġ$ k Ġ\ in Ġ[ 1 , 50 ] $ Ġin Ġthe Ġgraph Ġconstruction Ġfrom Ġour Ġdata Ġand Ġhave Ġfound Ġthat Ġthe Ġconstruction Ġis Ġrobust Ġas Ġlong Ġas Ġ$ k $ Ġis Ġnor Ġtoo Ġsmall Ġ( i . e ., Ġ$ k Ġ> Ġ13 $ Ġ). ĠThe Ġdetailed Ġcomparisons Ġare Ġshown Ġin ĠSection Ġsec : com par isons . </s> The ĠM ST - k NN Ġconstruction Ġhas Ġthe Ġadvantage Ġof Ġits Ġsimplicity Ġand Ġrobust ness , Ġand Ġthe Ġfact Ġthat Ġit Ġbalances Ġthe Ġlocal Ġand Ġglobal Ġstructure Ġof Ġthe Ġdata . ĠHowever , Ġthe Ġarea Ġof Ġnetwork Ġinference Ġand Ġgraph Ġconstruction Ġfrom Ġdata , Ġand Ġgraph Ġsp ars ification Ġis Ġvery Ġactive , Ġand Ġseveral Ġalternative Ġapproaches Ġexist Ġbased Ġon Ġdifferent Ġhe ur istics , Ġe . g ., ĠGraph ical ĠL asso ĠB IB REF 26 Ġ, ĠPlan ar ĠMaxim ally ĠFil tered ĠGraph ĠB IB REF 27 Ġ, Ġspectral Ġsp ars ification ĠB IB REF 28 Ġ, Ġor Ġthe ĠRelax ed ĠMinimum ĠSpan ning ĠTree Ġ( RM ST ) ĠB IB REF 29 Ġ. ĠWe Ġhave Ġexperimented Ġwith Ġsome Ġof Ġthose Ġmethods Ġand Ġobtained Ġcomparable Ġresults . ĠA Ġdetailed Ġcomparison Ġof Ġsp ars ification Ġmethods Ġas Ġwell Ġas Ġthe Ġchoice Ġof Ġdistance Ġin Ġdefining Ġthe Ġsimilarity Ġmatrix Ġ$\ hat { S } $ Ġis Ġleft Ġfor Ġfuture Ġwork . </s> Mult isc ale ĠGraph ĠPart ition ing </s> The Ġarea Ġof Ġcommunity Ġdetection Ġencompasses Ġa Ġvariety Ġof Ġgraph Ġpartition ing Ġapproaches Ġwhich Ġaim Ġto Ġfind Ġ` good ' Ġpartitions Ġinto Ġsub graph s Ġ( or Ġcommunities ) Ġaccording Ġto Ġdifferent Ġcost Ġfunctions , Ġwithout Ġimposing Ġthe Ġnumber Ġof Ġcommunities Ġa Ġprior i ĠB IB REF 31 Ġ. ĠThe Ġnotion Ġof Ġcommunity Ġthus Ġdepends Ġon Ġthe Ġchoice Ġof Ġcost Ġfunction . ĠCommon ly , Ġcommunities Ġare Ġtaken Ġto Ġbe Ġsub graph s Ġwhose Ġnodes Ġare Ġconnected Ġstrongly Ġwithin Ġthe Ġcommunity Ġwith Ġrelatively Ġweak Ġinter - community Ġedges . ĠSuch Ġstructural Ġnotion Ġis Ġrelated Ġto Ġbalanced Ġcuts . ĠOther Ġcost Ġfunctions Ġare Ġposed Ġin Ġterms Ġof Ġtransitions Ġinside Ġand Ġoutside Ġof Ġthe Ġcommunities , Ġusually Ġas Ġone - step Ġprocesses ĠB IB REF 4 Ġ. ĠWhen Ġtransition Ġpaths Ġof Ġrandom Ġwalks Ġof Ġall Ġlengths Ġare Ġconsidered , Ġthe Ġconcept Ġof Ġcommunity Ġbecomes Ġintrinsically Ġmulti - scale , Ġi . e ., Ġdifferent Ġpartitions Ġcan Ġbe Ġfound Ġto Ġbe Ġrelevant Ġat Ġdifferent Ġtime Ġscales Ġleading Ġto Ġa Ġmulti - level Ġdescription Ġdictated Ġby Ġthe Ġtransition Ġdynamics ĠB IB REF 9 Ġ, ĠB IB REF 32 Ġ, ĠB IB REF 10 Ġ. ĠThis Ġleads Ġto Ġthe Ġframework Ġof ĠMark ov ĠStability , Ġa Ġdynamics - based , Ġmulti - scale Ġcommunity Ġdetection Ġmethodology , Ġwhich Ġcan Ġbe Ġshown Ġto Ġrecover Ġseamlessly Ġseveral Ġwell - known Ġhe ur istics Ġas Ġparticular Ġcases ĠB IB REF 9 Ġ, ĠB IB REF 11 Ġ, ĠB IB REF 12 Ġ. </s> Here , Ġwe Ġapply ĠMS Ġto Ġfind Ġpartitions Ġof Ġthe Ġsimilarity Ġgraph Ġ$ A $ Ġat Ġdifferent Ġlevels Ġof Ġresolution . ĠThe Ġsub graph s Ġdetected Ġcorrespond Ġto Ġclusters Ġof Ġdocuments Ġwith Ġsimilar Ġcontent . ĠMS Ġis Ġan Ġun super vised Ġcommunity Ġdetection Ġmethod Ġthat Ġfinds Ġrobust Ġand Ġstable Ġpartitions Ġunder Ġthe Ġevolution Ġof Ġa Ġcontinuous - time Ġdiffusion Ġprocess Ġwithout Ġa Ġprior i Ġchoice Ġof Ġthe Ġnumber Ġor Ġtype Ġof Ġcommunities Ġor Ġtheir Ġorganisation ĠB IB REF 9 Ġ, ĠB IB REF 32 Ġ, ĠB IB REF 10 Ġ, ĠB IB REF 33 ĠThe Ġcode Ġfor ĠMark ov ĠStability Ġis Ġopen Ġand Ġaccessible Ġat Ġhttps :// github . com / m ich a els cha ub / Part ition St ability Ġand Ġhttp :// www f . im perial . ac . uk / ~ mp bara / Part ition _ St ability /, Ġlast Ġaccessed Ġon ĠMarch Ġ24 , Ġ2018 . ĠIn Ġsimple Ġterms , ĠMS Ġcan Ġbe Ġunderstood Ġby Ġanalogy Ġto Ġa Ġdrop Ġof Ġink Ġdiff using Ġon Ġthe Ġgraph Ġunder Ġa Ġdiff usive ĠMark ov Ġprocess . ĠThe Ġink Ġdiff uses Ġhom ogene ously Ġunless Ġthe Ġgraph Ġhas Ġsome Ġintrinsic Ġstructural Ġorganisation , Ġin Ġwhich Ġcase Ġthe Ġink Ġgets Ġtransient ly Ġcontained , Ġover Ġparticular Ġtime Ġscales , Ġwithin Ġgroups Ġof Ġnodes Ġ( i . e ., Ġsub graph s Ġor Ġcommunities ). ĠThe Ġexistence Ġof Ġthis Ġtransient Ġcontainment Ġsignals Ġthe Ġpresence Ġof Ġa Ġnatural Ġpartition Ġof Ġthe Ġgraph . ĠAs Ġthe Ġprocess Ġevolves , Ġthe Ġink Ġdiff uses Ġout Ġof Ġthose Ġinitial Ġcommunities Ġbut Ġmight Ġget Ġtransient ly Ġcontained Ġin Ġother , Ġlarger Ġsub graph s . ĠBy Ġanalys ing Ġthis ĠMark ov Ġdynamics Ġover Ġtime , ĠMS Ġdetects Ġthe Ġstructure Ġof Ġthe Ġgraph Ġacross Ġscales . ĠThe ĠMark ov Ġtime Ġ$ t $ Ġthus Ġacts Ġas Ġa Ġresolution Ġparameter Ġthat Ġallows Ġus Ġto Ġextract Ġrobust Ġpartitions Ġthat Ġpersist Ġover Ġparticular Ġtime Ġscales , Ġin Ġan Ġun super vised Ġmanner . </s> Given Ġthe Ġadj ac ency Ġmatrix Ġ$ A _{ N Ġ\ times ĠN } $ Ġof Ġthe Ġgraph Ġobtained Ġas Ġdescribed Ġpreviously , Ġlet Ġus Ġdefine Ġthe Ġdiagonal Ġmatrix Ġ$ D = \ text { di ag } (\ math bf Ġ{ d }) $ Ġ, Ġwhere Ġ$\ math bf Ġ{ d } = A Ġ\ math bf Ġ{ 1 } $ Ġis Ġthe Ġdegree Ġvector . ĠThe Ġrandom Ġwalk ĠLa pl ac ian Ġmatrix Ġis Ġdefined Ġas Ġ$ L _ \ text { RW } = I _ N - D ^{ - 1 } A $ Ġwhere Ġ$ I _ N $ Ġis Ġthe Ġidentity Ġmatrix Ġof Ġsize Ġ$ N $ Ġ, Ġand Ġthe Ġtransition Ġmatrix Ġ( or Ġkernel ) Ġof Ġthe Ġassociated Ġcontinuous - time ĠMark ov Ġprocess Ġis Ġ$ P ( t )= e ^{ - t ĠL _ \ text { RW } }, Ġ\ , Ġt > 0 $ ĠB IB REF 10 Ġ. ĠFor Ġeach Ġpartition , Ġa Ġbinary Ġmembership Ġmatrix Ġ$ H _{ N Ġ\ times ĠC } $ Ġmaps Ġthe Ġ$ N $ Ġnodes Ġinto Ġ$ C $ Ġclusters . ĠWe Ġcan Ġthen Ġdefine Ġthe Ġ$ D = \ text { di ag } (\ math bf Ġ{ d }) $ 0 Ġclustered Ġaut oc ov ari ance Ġmatrix : </s> $$ R ( t , H ) Ġ= ĠH ^ T [ \ Pi ĠP ( t )- \ pi Ġ\ pi Ġ^ T ] H Ġ$$ Ġ Ġ Ġ( E q . Ġ4 ) </s> where Ġ$\ pi Ġ$ Ġis Ġthe Ġsteady - state Ġdistribution Ġof Ġthe Ġprocess Ġand Ġ$\ Pi Ġ= \ text { di ag } (\ pi Ġ) $ Ġ. ĠThe Ġelement Ġ$ [ R ( t , H )] _{ \ alpha Ġ\ beta Ġ} $ Ġquant ifies Ġthe Ġprobability Ġthat Ġa Ġrandom Ġwalk er Ġstarting Ġfrom Ġcommunity Ġ$\ alpha Ġ$ Ġwill Ġend Ġin Ġcommunity Ġ$\ beta Ġ$ Ġat Ġtime Ġ$ t $ Ġ, Ġsubtract ing Ġthe Ġprobability Ġthat Ġthe Ġsame Ġevent Ġoccurs Ġby Ġchance Ġat Ġstation arity . </s> We Ġthen Ġdefine Ġour Ġcost Ġfunction Ġmeasuring Ġthe Ġgoodness Ġof Ġa Ġpartition Ġover Ġtime Ġ$ t $ Ġ, Ġtermed Ġthe ĠMark ov ĠStability Ġof Ġpartition Ġ$ H $ Ġ: </s> $$ r ( t , H ) Ġ= Ġ\ text { trace } Ġ\ left [ R ( t , H )\ right ]. Ġ$$ Ġ Ġ Ġ( E q . Ġ5 ) </s> A Ġpartition Ġ$ H $ Ġthat Ġmaxim ises Ġ$ r ( t , H ) $ Ġis Ġcomprised Ġof Ġcommunities Ġthat Ġpreserve Ġthe Ġflow Ġwithin Ġthemselves Ġover Ġtime Ġ$ t $ Ġ, Ġsince Ġin Ġthat Ġcase Ġthe Ġdiagonal Ġelements Ġof Ġ$ R ( t , H ) $ Ġwill Ġbe Ġlarge Ġand Ġthe Ġoff - di agonal Ġelements Ġwill Ġbe Ġsmall . ĠFor Ġdetails , Ġsee ĠB IB REF 9 Ġ, ĠB IB REF 32 Ġ, ĠB IB REF 10 Ġ, ĠB IB REF 34 Ġ. </s> MS Ġsearches Ġfor Ġpartitions Ġat Ġeach ĠMark ov Ġtime Ġthat Ġmaxim ise Ġ$ r ( t , H ) $ Ġ. ĠAlthough Ġthe Ġmaxim isation Ġof Ġ( Ġ5 Ġ) Ġis Ġan ĠNP - hard Ġproblem Ġ( hen ce Ġwith Ġno Ġguarantees Ġfor Ġglobal Ġoptim ality ), Ġthere Ġare Ġefficient Ġoptim isation Ġmethods Ġthat Ġwork Ġwell Ġin Ġpractice . ĠOur Ġimplementation Ġhere Ġuses Ġthe ĠLou v ain ĠAl gorithm ĠB IB REF 35 Ġ, ĠB IB REF 12 Ġwhich Ġis Ġefficient Ġand Ġknown Ġto Ġgive Ġgood Ġresults Ġwhen Ġapplied Ġto Ġbenchmarks ĠB IB REF 36 Ġ. ĠTo Ġobtain Ġrobust Ġpartitions , Ġwe Ġrun Ġthe ĠLou v ain Ġalgorithm Ġ500 Ġtimes Ġwith Ġdifferent Ġinitial isations Ġat Ġeach ĠMark ov Ġtime Ġand Ġpick Ġthe Ġbest Ġ50 Ġwith Ġthe Ġhighest ĠMark ov ĠStability Ġvalue Ġ$ r ( t , H ) $ Ġ. ĠWe Ġthen Ġcompute Ġthe Ġvariation Ġof Ġinformation ĠB IB REF 37 Ġof Ġthis Ġensemble Ġof Ġsolutions Ġ$ VI ( t ) $ Ġ. Ġas Ġa Ġmeasure Ġof Ġthe Ġreprodu c ibility Ġof Ġthe Ġresult Ġunder Ġthe Ġoptim isation . ĠIn Ġaddition , Ġthe Ġrelevant Ġpartitions Ġare Ġrequired Ġto Ġbe Ġpersistent Ġacross Ġtime , Ġas Ġgiven Ġby Ġlow Ġvalues Ġof Ġthe Ġvariation Ġof Ġinformation Ġbetween Ġoptim ised Ġpartitions Ġacross Ġtime Ġ$ VI ( t , t ^ {\ prime Ġ}) $ Ġ. ĠRob ust Ġpartitions Ġare Ġthus Ġindicated Ġby ĠMark ov Ġtimes Ġwhere Ġ$ VI ( t ) $ Ġshows Ġa Ġdip Ġand Ġ$ VI ( t , t ^ {\ prime Ġ}) $ Ġhas Ġan Ġextended Ġplateau , Ġindicating Ġconsistent Ġresults Ġfrom Ġdifferent ĠLou v ain Ġruns Ġand Ġvalidity Ġover Ġextended Ġscales ĠB IB REF 34 Ġ, ĠB IB REF 10 Ġ. </s> Visual isation Ġand Ġinterpretation Ġof Ġthe Ġresults </s> We Ġuse Ġthe ĠForce At las 2 ĠB IB REF 30 Ġlayout Ġto Ġrepresent Ġthe Ġgraph Ġof Ġ3 229 ĠNR LS ĠPatient ĠIncident Ġreports . ĠThis Ġlayout Ġfollows Ġa Ġforce - directed Ġiter ative Ġmethod Ġto Ġfind Ġnode Ġpositions Ġthat Ġbalance Ġattractive Ġand Ġrep ulsive Ġforces . ĠHence Ġsimilar Ġnodes Ġtend Ġto Ġbe Ġgrouped Ġtogether Ġon Ġthe Ġplan ar Ġlayout . ĠWe Ġcolour Ġthe Ġnodes Ġby Ġeither Ġhand - coded Ġcategories Ġ( Figure Ġ2 Ġ) Ġor Ġmult isc ale ĠMS Ġcommunities Ġ( Figure Ġ3 Ġ). ĠSp at ially Ġconsistent Ġcolour ings Ġon Ġthis Ġlayout Ġimply Ġgood Ġclusters Ġof Ġdocuments Ġin Ġterms Ġof Ġthe Ġsimilarity Ġgraph . </s> San key Ġdiagrams Ġallow Ġus Ġto Ġvisual ise Ġthe Ġrelationship Ġof Ġnode Ġmembers hips Ġacross Ġdifferent Ġpartitions Ġand Ġwith Ġrespect Ġto Ġthe Ġhand - coded Ġcategories . ĠIn Ġparticular , Ġtwo - layer ĠSan key Ġdiagrams Ġ( e . g ., ĠFig . Ġ4 Ġ) Ġreflect Ġthe Ġcorrespondence Ġbetween ĠMS Ġclusters Ġand Ġthe Ġhand - coded Ġexternal Ġcategories , Ġwhereas Ġthe Ġmult il ayer ĠSan key Ġdiagram Ġin ĠFig . Ġ3 Ġrepresents Ġthe Ġresults Ġof Ġthe Ġmulti - resolution ĠMS Ġcommunity Ġdetection Ġacross Ġscales . </s> In Ġaddition Ġto ĠSan key Ġdiagrams Ġbetween Ġour ĠMS Ġclusters Ġand Ġthe Ġhand - coded Ġcategories , Ġwe Ġalso Ġprovide Ġa Ġcomplementary Ġvisual isation Ġas Ġheat maps Ġof Ġnormal ised Ġcontingency Ġ( z - score ) Ġtables , Ġe . g ., ĠFig . Ġ4 Ġ. ĠThis Ġallows Ġus Ġto Ġcompare Ġthe Ġrelative Ġassociation Ġof Ġcontent Ġclusters Ġto Ġthe Ġexternal Ġcategories Ġat Ġdifferent Ġresolution Ġlevels . ĠA Ġquant ification Ġof Ġthis Ġcorrespondence Ġis Ġalso Ġprovided Ġby Ġthe Ġ$ N MI $ Ġscore Ġintroduced Ġin ĠE q . Ġ( Ġ8 Ġ). </s> Our Ġmethod Ġclusters Ġtext Ġdocuments Ġaccording Ġto Ġtheir Ġintrinsic Ġcontent . ĠThis Ġcan Ġbe Ġunderstood Ġas Ġa Ġtype Ġof Ġtopic Ġdetection . ĠTo Ġunderstand Ġthe Ġcontent Ġof Ġthe Ġclusters , Ġwe Ġuse ĠWord ĠClouds Ġas Ġbasic , Ġyet Ġintuitive , Ġtools Ġthat Ġsummar ise Ġinformation Ġfrom Ġa Ġgroup Ġof Ġdocuments . ĠWord Ġclouds Ġallow Ġus Ġto Ġevaluate Ġthe Ġresults Ġand Ġextract Ġinsights Ġwhen Ġcomparing Ġa Ġposterior i Ġwith Ġhand - coded Ġcategories . ĠThey Ġcan Ġalso Ġprovide Ġan Ġaid Ġfor Ġmonitoring Ġresults Ġwhen Ġused Ġby Ġpractitioners . </s> The Ġstemming Ġmethods Ġdescribed Ġin Ġthe ĠText ĠPre processing Ġsubsection Ġtrunc ate Ġwords Ġseverely . ĠSuch Ġtrunc ation Ġenhances Ġthe Ġpower Ġof Ġthe Ġlanguage Ġprocessing Ġcomputational Ġmethods , Ġas Ġit Ġreduces Ġthe Ġredundancy Ġin Ġthe Ġword Ġcorpus . ĠYet Ġwhen Ġpresenting Ġthe Ġresults Ġback Ġto Ġa Ġhuman Ġobserver , Ġit Ġis Ġdesirable Ġto Ġreport Ġthe Ġcontent Ġof Ġthe Ġclusters Ġwith Ġwords Ġthat Ġare Ġreadily Ġcomprehens ible . ĠTo Ġgenerate Ġcomprehens ible Ġword Ġclouds Ġin Ġour Ġa Ġposterior i Ġanalyses , Ġwe Ġuse Ġa Ġtext Ġprocessing Ġmethod Ġsimilar Ġto Ġthe Ġone Ġdescribed Ġin ĠB IB REF 38 Ġ. ĠSpecifically , Ġwe Ġuse Ġthe Ġpart Ġof Ġspeech Ġ( POS ) Ġtagging Ġmodule Ġfrom ĠNL T K Ġto Ġleave Ġout Ġsentence Ġparts Ġexcept Ġthe Ġadject ives , Ġnoun s , Ġand Ġverbs . ĠWe Ġalso Ġremove Ġless Ġmeaningful Ġcommon Ġverbs Ġsuch Ġas Ġ` be ', Ġ` have ', Ġand Ġ` do ' Ġand Ġtheir Ġvariations . ĠThe Ġresidual Ġwords Ġare Ġthen Ġle mm at ised Ġand Ġrepresented Ġwith Ġtheir Ġle mm as Ġin Ġorder Ġto Ġnormal ise Ġvariations Ġof Ġthe Ġsame Ġword . ĠOnce Ġthe Ġtext Ġis Ġprocessed Ġin Ġthis Ġmanner , Ġwe Ġuse Ġthe ĠPython Ġlibrary Ġword cloud The Ġword Ġcloud Ġgenerator Ġlibrary Ġfor ĠPython Ġis Ġopen Ġand Ġaccessible Ġat Ġhttps :// github . com / am ueller / word _ cloud , Ġlast Ġaccessed Ġon ĠMarch Ġ25 , Ġ2018 Ġto Ġcreate Ġword Ġclouds Ġwith Ġ2 Ġor Ġ3 - gram Ġfrequency Ġlist Ġof Ġcommon Ġword Ġgroups . ĠThe Ġresults Ġpresent Ġdistinct , Ġunderstandable Ġword Ġtopics . </s> Quant itative Ġbenchmark ing Ġof Ġtopic Ġclusters </s> Although Ġour Ġdataset Ġhas Ġattached Ġa Ġhand - coded Ġclassification Ġby Ġa Ġhuman Ġoperator , Ġwe Ġdo Ġnot Ġuse Ġit Ġin Ġour Ġanalysis Ġand Ġwe Ġdo Ġnot Ġconsider Ġit Ġas Ġa Ġ` ground Ġtruth '. ĠIndeed , Ġone Ġof Ġour Ġaims Ġis Ġto Ġexplore Ġthe Ġrelevance Ġof Ġthe Ġfixed Ġexternal Ġclasses Ġas Ġcompared Ġto Ġthe Ġcontent - driven Ġgroup ings Ġobtained Ġin Ġan Ġun super vised Ġmanner . ĠHence Ġwe Ġprovide Ġa Ġdouble Ġroute Ġto Ġquantify Ġthe Ġquality Ġof Ġthe Ġclusters Ġby Ġcomputing Ġtwo Ġcomplementary Ġmeasures : Ġan Ġintrinsic Ġmeasure Ġof Ġtopic Ġco herence Ġand Ġa Ġmeasure Ġof Ġsimilarity Ġto Ġthe Ġexternal Ġhand - coded Ġcategories , Ġdefined Ġas Ġfollows . </s> As Ġan Ġintrinsic Ġmeasure Ġof Ġconsistency Ġof Ġword Ġassociation Ġwithout Ġany Ġreference Ġto Ġan Ġexternal Ġ` ground Ġtruth ', Ġwe Ġuse Ġthe Ġpoint wise Ġmutual Ġinformation Ġ( Ġ$ PM I $ Ġ) ĠB IB REF 13 Ġ, ĠB IB REF 14 Ġ. ĠThe Ġ$ PM I $ Ġis Ġan Ġinformation - the oret ical Ġscore Ġthat Ġcaptures Ġthe Ġprobability Ġof Ġbeing Ġused Ġtogether Ġin Ġthe Ġsame Ġgroup Ġof Ġdocuments . ĠThe Ġ$ PM I $ Ġscore Ġfor Ġa Ġpair Ġof Ġwords Ġ$( w _ 1 , w _ 2 ) $ Ġis : </s> $$ PM I ( w _ 1 , w _ 2 )= \ log Ġ{\ frac { P ( w _ 1 Ġw _ 2 ) }{ P ( w _ 1 ) P ( w _ 2 ) }} $$ Ġ Ġ Ġ( E q . Ġ6 ) </s> where Ġthe Ġprobabilities Ġof Ġthe Ġwords Ġ$ P ( w _ 1 ) $ Ġ, Ġ$ P ( w _ 2 ) $ Ġ, Ġand Ġof Ġtheir Ġco - occ urrence Ġ$ P ( w _ 1 Ġw _ 2 ) $ Ġare Ġobtained Ġfrom Ġthe Ġcorpus . ĠTo Ġobtain Ġthe Ġaggregate Ġ$\ wide hat { PM I } $ Ġfor Ġthe Ġgraph Ġpartition Ġ$ C = \ l brace Ġc _ i \ r brace Ġ$ Ġwe Ġcompute Ġthe Ġ$ PM I $ Ġfor Ġeach Ġcluster , Ġas Ġthe Ġmedian Ġ$ PM I $ Ġbetween Ġits Ġ10 Ġmost Ġcommon Ġwords Ġ( changing Ġthe Ġnumber Ġof Ġwords Ġgives Ġsimilar Ġresults ), Ġand Ġwe Ġobtain Ġthe Ġweighted Ġaverage Ġof Ġthe Ġ$ PM I $ Ġcluster Ġscores : </s> $$ \ wide hat { PM I } Ġ( C ) Ġ= Ġ\ sum Ġ_ { c _ i Ġ\ in ĠC } Ġ\ frac { n _ i }{ N } Ġ\ , Ġ\ math op Ġ{\ oper at orn ame { med ian }} _{ \ begin { array }{ c } w _ k , Ġw _ \ ell Ġ\ in ĠS _ i Ġ\\ Ġk < \ ell Ġ\ end { array }} ĠPM I ( w _ k , w _ \ ell Ġ), $$ Ġ Ġ Ġ( E q . Ġ7 ) </s> where Ġ$ c _ i $ Ġdenotes Ġthe Ġclusters Ġin Ġpartition Ġ$ C $ Ġ, Ġeach Ġwith Ġsize Ġ$ n _ i $ Ġ; Ġ$ N = \ sum Ġ_ { c _ i Ġ\ in ĠC } Ġn _ i $ Ġis Ġthe Ġtotal Ġnumber Ġof Ġnodes ; Ġand Ġ$ S _ i $ Ġdenotes Ġthe Ġset Ġof Ġtop Ġ10 Ġwords Ġfor Ġcluster Ġ$ c _ i $ Ġ. </s> We Ġuse Ġthis Ġ$\ wide hat { PM I } $ Ġscore Ġto Ġevaluate Ġpartitions Ġwithout Ġrequiring Ġa Ġlabelled Ġground Ġtruth . The Ġ$ PM I $ Ġscore Ġhas Ġbeen Ġshown Ġto Ġperform Ġwell ĠB IB REF 13 Ġ, ĠB IB REF 14 Ġwhen Ġcompared Ġto Ġhuman Ġinterpretation Ġof Ġtopics Ġon Ġdifferent Ġcorpor a ĠB IB REF 39 Ġ, ĠB IB REF 40 Ġ, Ġand Ġis Ġdesigned Ġto Ġevaluate Ġtopical Ġco herence Ġfor Ġgroups Ġof Ġdocuments , Ġin Ġcontrast Ġto Ġother Ġtools Ġaimed Ġat Ġshort Ġforms Ġof Ġtext . ĠSee ĠB IB REF 18 Ġ, ĠB IB REF 19 Ġ, ĠB IB REF 41 Ġ, ĠB IB REF 42 Ġfor Ġother Ġexamples . </s> To Ġcompare Ġagainst Ġthe Ġexternal Ġclassification Ġa Ġposterior i , Ġwe Ġuse Ġthe Ġnormal ised Ġmutual Ġinformation Ġ( Ġ$ N MI $ Ġ), Ġa Ġwell - used Ġinformation - the oret ical Ġscore Ġthat Ġquant ifies Ġthe Ġsimilarity Ġbetween Ġcluster ings Ġconsidering Ġboth Ġthe Ġcorrect Ġand Ġincorrect Ġassignments Ġin Ġterms Ġof Ġthe Ġinformation Ġ( or Ġpredict ability ) Ġbetween Ġthe Ġcluster ings . ĠThe ĠN MI Ġbetween Ġtwo Ġpartitions Ġ$ C $ Ġand Ġ$ D $ Ġof Ġthe Ġsame Ġgraph Ġis : </s> $$ N MI ( C , D )= \ frac { I ( C , D ) }{ \ sq rt { H ( C ) H ( D ) }} = \ frac {\ sum Ġ\ limits Ġ_ { c Ġ\ in ĠC } Ġ\ sum Ġ\ limits Ġ_ { d Ġ\ in ĠD } Ġp ( c , d ) Ġ\ , Ġ\ log Ġ\ df rac { p ( c , d ) }{ p ( c ) p ( d ) }} {\ sq rt { H ( C ) H ( D ) }} $$ Ġ Ġ Ġ( E q . Ġ8 ) </s> where Ġ$ I ( C , D ) $ Ġis Ġthe ĠMutual ĠInformation Ġand Ġ$ H ( C ) $ Ġand Ġ$ H ( D ) $ Ġare Ġthe Ġent rop ies Ġof Ġthe Ġtwo Ġpartitions . </s> The Ġ$ N MI $ Ġis Ġbounded Ġ( Ġ$ 0 Ġ\ le ĠN MI Ġ\ le Ġ1 $ Ġ) Ġwith Ġa Ġhigher Ġvalue Ġcorresponding Ġto Ġhigher Ġsimilarity Ġof Ġthe Ġpartitions Ġ( i . e ., Ġ$ N MI = 1 $ Ġwhen Ġthere Ġis Ġperfect Ġagreement Ġbetween Ġpartitions Ġ$ C $ Ġand Ġ$ D $ Ġ). ĠThe Ġ$ N MI $ Ġscore Ġis Ġdirectly Ġrelated http :// sci kit - learn . org / stable / modules / generated / sk learn . met rics . v _ me asure _ score . html Ġto Ġthe ĠV - me asure Ġused Ġin Ġthe Ġcomputer Ġscience Ġliterature ĠB IB REF 43 Ġ. ĠWe Ġuse Ġthe Ġ$ N MI $ Ġto Ġcompare Ġthe Ġpartitions Ġobtained Ġby ĠMS Ġ( and Ġother Ġmethods ) Ġagainst Ġthe Ġhand - coded Ġclassification Ġassigned Ġby Ġthe Ġoperator . </s> Multi - resolution Ġcommunity Ġdetection Ġextracts Ġcontent Ġclusters Ġat Ġdifferent Ġlevels Ġof Ġgran ularity </s> We Ġapplied ĠMS Ġacross Ġa Ġbroad Ġspan Ġof ĠMark ov Ġtimes Ġ( Ġ$ t Ġ\ in Ġ[ 0 . 01 , Ġ100 ] $ Ġin Ġsteps Ġof Ġ0 . 01 ) Ġto Ġthe ĠM ST - k NN Ġsimilarity Ġgraph Ġof Ġ$ N = 3 229 $ Ġincident Ġrecords . ĠAt Ġeach ĠMark ov Ġtime , Ġwe Ġran Ġ500 Ġindependent Ġoptim isations Ġof Ġthe ĠLou v ain Ġalgorithm Ġand Ġselected Ġthe Ġoptimal Ġpartition Ġat Ġeach Ġtime . ĠRepe ating Ġthe Ġoptim isation Ġfrom Ġ500 Ġdifferent Ġinitial Ġstarting Ġpoints Ġenhances Ġthe Ġrobust ness Ġof Ġthe Ġoutcome Ġand Ġallows Ġus Ġto Ġquantify Ġthe Ġrobust ness Ġof Ġthe Ġpartition Ġto Ġthe Ġoptim isation Ġprocedure . ĠTo Ġquantify Ġthis Ġrobust ness , Ġwe Ġcomputed Ġthe Ġaverage Ġvariation Ġof Ġinformation Ġ$ VI ( t ) $ Ġ( a Ġmeasure Ġof Ġdiss imilar ity ) Ġbetween Ġthe Ġtop Ġ50 Ġpartitions Ġfor Ġeach Ġ$ t $ Ġ. ĠOnce Ġthe Ġfull Ġscan Ġacross ĠMark ov Ġtime Ġwas Ġfinal ised , Ġa Ġfinal Ġcomparison Ġof Ġall Ġthe Ġoptimal Ġpartitions Ġobtained Ġwas Ġcarried Ġout , Ġso Ġas Ġto Ġassess Ġif Ġany Ġof Ġthe Ġoptim ised Ġpartitions Ġwas Ġoptimal Ġat Ġany Ġother ĠMark ov Ġtime , Ġin Ġwhich Ġcase Ġit Ġwas Ġselected . ĠWe Ġthen Ġobtained Ġthe Ġ$ VI ( t , t ^ {\ prime Ġ}) $ Ġacross Ġall Ġoptimal Ġpartitions Ġfound Ġacross ĠMark ov Ġtimes Ġto Ġascertain Ġwhen Ġpartitions Ġare Ġrobust Ġacross Ġlevels Ġof Ġresolution . ĠThis Ġlayered Ġprocess Ġof Ġoptim isation Ġenhances Ġthe Ġrobust ness Ġof Ġthe Ġoutcome Ġgiven Ġthe ĠNP - hard Ġnature Ġof ĠMS Ġoptim isation , Ġwhich Ġprevents Ġguaranteed Ġglobal Ġoptim ality . </s> Figure Ġ3 Ġpresents Ġa Ġsummary Ġof Ġour Ġanalysis . ĠWe Ġplot Ġthe Ġnumber Ġof Ġclusters Ġof Ġthe Ġoptimal Ġpartition Ġand Ġthe Ġtwo Ġmetrics Ġof Ġvariation Ġof Ġinformation Ġacross Ġall ĠMark ov Ġtimes . ĠThe Ġexistence Ġof Ġa Ġlong Ġplateau Ġin Ġ$ VI ( t , t ^ {\ prime Ġ}) $ Ġcoupled Ġto Ġa Ġdip Ġin Ġ$ VI ( t ) $ Ġimplies Ġthe Ġpresence Ġof Ġa Ġpartition Ġthat Ġis Ġrobust Ġboth Ġto Ġthe Ġoptim isation Ġand Ġacross ĠMark ov Ġtime . ĠTo Ġillustrate Ġthe Ġmulti - scale Ġfeatures Ġof Ġthe Ġmethod , Ġwe Ġchoose Ġseveral Ġof Ġthese Ġrobust Ġpartitions , Ġfrom Ġfiner Ġ( 44 Ġcommunities ) Ġto Ġco arser Ġ( 3 Ġcommunities ), Ġobtained Ġat Ġfive ĠMark ov Ġtimes Ġand Ġexamine Ġtheir Ġstructure Ġand Ġcontent . ĠWe Ġalso Ġpresent Ġa Ġmulti - level ĠSan key Ġdiagram Ġto Ġsummar ise Ġthe Ġrelationships Ġand Ġrelative Ġnode Ġmembership Ġacross Ġthe Ġlevels . </s> The ĠMS Ġanalysis Ġof Ġthe Ġgraph Ġof Ġincident Ġreports Ġreveals Ġa Ġrich Ġmulti - level Ġstructure Ġof Ġpartitions , Ġwith Ġa Ġstrong Ġquasi - h ier arch ical Ġorganisation , Ġas Ġseen Ġin Ġthe Ġgraph Ġlayouts Ġand Ġthe Ġmulti - level ĠSan key Ġdiagram . ĠIt Ġis Ġimportant Ġto Ġremark Ġthat , Ġalthough Ġthe ĠMark ov Ġtime Ġacts Ġas Ġa Ġnatural Ġresolution Ġparameter Ġfrom Ġfiner Ġto Ġco arser Ġpartitions , Ġour Ġprocess Ġof Ġoptim isation Ġdoes Ġnot Ġimpose Ġany Ġhierarchical Ġstructure Ġa Ġprior i . ĠHence Ġthe Ġobserved Ġconsistency Ġof Ġcommunities Ġacross Ġlevel Ġis Ġintrinsic Ġto Ġthe Ġdata Ġand Ġsuggests Ġthe Ġexistence Ġof Ġcontent Ġclusters Ġthat Ġnaturally Ġintegrate Ġwith Ġeach Ġother Ġas Ġsub - the mes Ġof Ġlarger Ġthem atic Ġcategories . ĠThe Ġdetection Ġof Ġintrinsic Ġscales Ġwithin Ġthe Ġgraph Ġprovided Ġby ĠMS Ġthus Ġenables Ġus Ġto Ġobtain Ġclusters Ġof Ġrecords Ġwith Ġhigh Ġcontent Ġsimilarity Ġat Ġdifferent Ġlevels Ġof Ġgran ularity . ĠThis Ġcapability Ġcan Ġbe Ġused Ġby Ġpractitioners Ġto Ġtune Ġthe Ġlevel Ġof Ġdescription Ġto Ġtheir Ġspecific Ġneeds . </s> Inter pret ation Ġof ĠMS Ġcommunities : Ġcontent Ġand Ġa Ġposterior i Ġcomparison Ġwith Ġhand - coded Ġcategories </s> To Ġascertain Ġthe Ġrelevance Ġof Ġthe Ġdifferent Ġlayers Ġof Ġcontent Ġclusters Ġfound Ġin Ġthe ĠMS Ġanalysis , Ġwe Ġexamined Ġin Ġdetail Ġthe Ġfive Ġlevels Ġof Ġresolution Ġpresented Ġin ĠFigure Ġ3 Ġ. ĠFor Ġeach Ġlevel , Ġwe Ġprepared Ġword Ġclouds Ġ( le mm at ised Ġfor Ġincreased Ġintellig ibility ), Ġas Ġwell Ġas Ġa ĠSan key Ġdiagram Ġand Ġa Ġcontingency Ġtable Ġlinking Ġcontent Ġclusters Ġ( i . e ., Ġgraph Ġcommunities ) Ġwith Ġthe Ġhand - coded Ġcategories Ġexternally Ġassigned Ġby Ġan Ġoperator . ĠWe Ġnote Ġagain Ġthat Ġthis Ġcomparison Ġwas Ġonly Ġdone Ġa Ġposterior i , Ġi . e ., Ġthe Ġexternal Ġcategories Ġwere Ġnot Ġused Ġin Ġour Ġtext Ġanalysis . ĠThe Ġresults Ġare Ġshown Ġin ĠFigures Ġ4 ĠâĢĵ Ġ6 Ġ( and ĠSupplementary ĠFigures ĠS 1 ĠâĢĵ ĠS 2 Ġ) Ġfor Ġall Ġlevels . </s> The Ġpartition Ġinto Ġ44 Ġcommunities Ġpresents Ġcontent Ġclusters Ġwith Ġwell - defined Ġcharacter isations , Ġas Ġshown Ġby Ġthe ĠSan key Ġdiagram Ġand Ġthe Ġhighly Ġclustered Ġstructure Ġof Ġthe Ġcontingency Ġtable Ġ( Figure Ġ4 Ġ). ĠThe Ġcontent Ġlabels Ġfor Ġthe Ġcommunities Ġwere Ġderived Ġby Ġus Ġfrom Ġthe Ġword Ġclouds Ġpresented Ġin Ġdetail Ġin Ġthe ĠSupplementary ĠInformation Ġ( Fig . ĠS 1 Ġin Ġthe ĠSI ). ĠCompared Ġto Ġthe Ġ15 Ġhand - coded Ġcategories , Ġthis Ġ44 - community Ġpartition Ġprovides Ġfiner Ġgroup ings Ġof Ġrecords Ġwith Ġseveral Ġclusters Ġcorresponding Ġto Ġsub - the mes Ġor Ġmore Ġspecific Ġsub - classes Ġwithin Ġlarge , Ġgeneric Ġhand - coded Ġcategories . ĠThis Ġis Ġapparent Ġin Ġthe Ġexternal Ġclasses Ġ` Acc idents ', Ġ` Med ication ', Ġ` Cl inical Ġassessment ', Ġ` Document ation ' Ġand Ġ` Inf rastructure ', Ġwhere Ġa Ġvariety Ġof Ġsub top ics Ġare Ġidentified Ġcorresponding Ġto Ġmeaningful Ġsub classes Ġ( see ĠFig . ĠS 1 Ġfor Ġdetails ). ĠIn Ġother Ġcases , Ġhowever , Ġthe Ġcontent Ġclusters Ġcut Ġacross Ġthe Ġexternal Ġcategories , Ġor Ġcorrespond Ġto Ġhighly Ġspecific Ġcontent . ĠExamples Ġof Ġthe Ġformer Ġare Ġthe Ġcontent Ġcommunities Ġof Ġrecords Ġfrom Ġlabour Ġward , Ġchemotherapy , Ġradi otherapy Ġand Ġinfection Ġcontrol , Ġwhose Ġreports Ġare Ġgrouped Ġcoherent ly Ġbased Ġon Ġcontent Ġby Ġour Ġalgorithm , Ġyet Ġbelong Ġto Ġhighly Ġdiverse Ġexternal Ġclasses . ĠAt Ġthis Ġlevel Ġof Ġresolution , Ġour Ġalgorithm Ġalso Ġidentified Ġhighly Ġspecific Ġtopics Ġas Ġseparate Ġcontent Ġclusters . ĠThese Ġinclude Ġblood Ġtransf usions , Ġpressure Ġul cer , Ġconsent , Ġmental Ġhealth , Ġand Ġchild Ġprotection . </s> We Ġhave Ġstudied Ġtwo Ġlevels Ġof Ġresolution Ġwhere Ġthe Ġnumber Ġof Ġcommunities Ġ( 12 Ġand Ġ17 ) Ġis Ġclose Ġto Ġthat Ġof Ġhand - coded Ġcategories Ġ( 15 ). ĠThe Ġresults Ġof Ġthe Ġ12 - community Ġpartition Ġare Ġpresented Ġin ĠFigure Ġ5 Ġ( see ĠFigure ĠS 2 Ġin Ġthe ĠSI Ġfor Ġthe Ġslightly Ġfiner Ġ17 - community Ġpartition ). ĠAs Ġexpected Ġfrom Ġthe Ġquasi - h ier arch ical Ġnature Ġof Ġour Ġmulti - resolution Ġanalysis , Ġwe Ġfind Ġthat Ġsome Ġof Ġthe Ġcommunities Ġin Ġthe Ġ12 - way Ġpartition Ġemerge Ġfrom Ġconsistent Ġaggregation Ġof Ġsmaller Ġcommunities Ġin Ġthe Ġ44 - way Ġpartition . ĠIn Ġterms Ġof Ġtopics , Ġthis Ġmeans Ġthat Ġsome Ġof Ġthe Ġsub - the mes Ġobserved Ġin ĠFigure Ġ4 Ġare Ġmerged Ġinto Ġa Ġmore Ġgeneral Ġtopic . ĠThis Ġis Ġapparent Ġin Ġthe Ġcase Ġof ĠAcc idents : Ġseven Ġof Ġthe Ġcommunities Ġin Ġthe Ġ44 - way Ġpartition Ġbecome Ġone Ġlarger Ġcommunity Ġ( community Ġ2 Ġin ĠFig . Ġ5 Ġ), Ġwhich Ġhas Ġa Ġspecific Ġand Ġcomplete Ġidentification Ġwith Ġthe Ġexternal Ġcategory Ġ` Pat ient Ġaccidents '. ĠA Ġsimilar Ġphenomenon Ġis Ġseen Ġfor Ġthe ĠNursing Ġcommunity Ġ( community Ġ1 ) Ġwhich Ġfalls Ġcompletely Ġunder Ġthe Ġexternal Ġcategory Ġ` Inf rastructure '. ĠThe Ġclusters Ġrelated Ġto Ġ` Med ication ' Ġsimilarly Ġaggregate Ġinto Ġa Ġlarger Ġcommunity Ġ( community Ġ3 ), Ġyet Ġthere Ġstill Ġremains Ġa Ġsmaller , Ġspecific Ġcommunity Ġrelated Ġto ĠHome care Ġmedication Ġ( community Ġ12 ) Ġwith Ġdistinct Ġcontent . </s> Other Ġcommunities Ġstrand Ġacross Ġa Ġfew Ġexternal Ġcategories . ĠThis Ġis Ġclearly Ġobservable Ġin Ġcommunities Ġ10 Ġand Ġ11 Ġ( S amples / Ġlab Ġtests / forms Ġand ĠRefer rals / app oint ments ), Ġwhich Ġfall Ġnaturally Ġacross Ġthe Ġexternal Ġcategories Ġ` Document ation ' Ġand Ġ` Cl inical ĠAssessment '. ĠSimilarly , Ġcommunity Ġ9 Ġ( Pat ient Ġtransfers ) Ġsits Ġacross Ġthe Ġ` Ad mission / Transfer ' Ġand Ġ` Inf rastructure ' Ġexternal Ġcategories , Ġdue Ġto Ġits Ġrelation Ġto Ġnursing Ġand Ġother Ġphysical Ġconstraints . ĠThe Ġrest Ġof Ġthe Ġcommunities Ġcontain Ġa Ġsubstantial Ġproportion Ġof Ġrecords Ġthat Ġhave Ġbeen Ġhand - classified Ġunder Ġthe Ġgeneric Ġ` T reatment / Pro ced ure ' Ġclass ; Ġyet Ġhere Ġthey Ġare Ġseparated Ġinto Ġgroups Ġthat Ġretain Ġmedical Ġco herence , Ġi . e ., Ġthey Ġrefer Ġto Ġmedical Ġprocedures Ġor Ġprocesses , Ġsuch Ġas ĠRad i otherapy Ġ( Comm . Ġ4 ), ĠBlood Ġtransf usions Ġ( Comm . Ġ7 ), ĠIV / c ann ula Ġ( Comm . Ġ5 ), ĠPressure Ġul cer Ġ( Comm . Ġ8 ), Ġand Ġthe Ġlarge Ġcommunity ĠLabour Ġward Ġ( Comm . Ġ6 ). </s> The Ġhigh Ġspecificity Ġof Ġthe ĠRad i otherapy , ĠPressure Ġul cer Ġand ĠLabour Ġward Ġcommunities Ġmeans Ġthat Ġthey Ġare Ġstill Ġpreserved Ġas Ġseparate Ġgroups Ġon Ġthe Ġnext Ġlevel Ġof Ġco ars eness Ġgiven Ġby Ġthe Ġ7 - way Ġpartition Ġ( Figure Ġ6 ĠA ). ĠThe Ġmer gers Ġin Ġthis Ġcase Ġlead Ġto Ġa Ġlarger Ġcommunities Ġreferring Ġto ĠMed ication , ĠRefer rals / Form s Ġand ĠStaff ing / Pat ient Ġtransfers . ĠFigure Ġ6 ĠB Ġshows Ġthe Ġfinal Ġlevel Ġof Ġagg l omer ation Ġinto Ġ3 Ġcommunities : Ġa Ġcommunity Ġof Ġrecords Ġreferring Ġto Ġaccidents ; Ġanother Ġcommunity Ġbroadly Ġreferring Ġto Ġprocedural Ġmatters Ġ( re fer rals , Ġforms , Ġstaffing , Ġmedical Ġprocedures ) Ġcutting Ġacross Ġmany Ġof Ġthe Ġexternal Ġcategories ; Ġand Ġthe Ġlabour Ġward Ġcommunity Ġstill Ġon Ġits Ġown Ġas Ġa Ġsub group Ġof Ġincidents Ġwith Ġdistinctive Ġcontent . </s> This Ġprocess Ġof Ġagg l omer ation Ġof Ġcontent , Ġfrom Ġsub - the mes Ġinto Ġlarger Ġthemes , Ġas Ġa Ġresult Ġof Ġthe Ġmulti - scale Ġhierarchy Ġof Ġgraph Ġpartitions Ġobtained Ġwith ĠMS Ġis Ġshown Ġexplicitly Ġwith Ġword Ġclouds Ġin ĠFigure Ġ8 Ġfor Ġthe Ġ17 , Ġ12 Ġand Ġ7 - way Ġpartitions . </s> Rob ust ness Ġof Ġthe Ġresults Ġand Ġcomparison Ġwith Ġother Ġmethods </s> Our Ġframework Ġconsists Ġof Ġa Ġseries Ġof Ġsteps Ġfor Ġwhich Ġthere Ġare Ġchoices Ġand Ġalternatives . ĠAlthough Ġit Ġis Ġnot Ġpossible Ġto Ġprovide Ġcomparisons Ġto Ġthe Ġmyriad Ġof Ġmethods Ġand Ġpossibilities Ġavailable , Ġwe Ġhave Ġexamined Ġquant itatively Ġthe Ġrobust ness Ġof Ġthe Ġresults Ġto Ġparam etric Ġand Ġmethodological Ġchoices Ġin Ġdifferent Ġsteps Ġof Ġthe Ġframework : Ġ( i ) Ġthe Ġimportance Ġof Ġusing ĠDoc 2 V ec Ġembed d ings Ġinstead Ġof ĠBo W Ġvectors , Ġ( ii ) Ġthe Ġsize Ġof Ġtraining Ġcorpus Ġfor ĠDoc 2 V ec ; Ġ( iii ) Ġthe Ġsp arsity Ġof Ġthe ĠM ST - k NN Ġsimilarity Ġgraph Ġconstruction . ĠWe Ġhave Ġalso Ġcarried Ġout Ġquantitative Ġcomparisons Ġto Ġother Ġmethods , Ġincluding : Ġ( i ) ĠL DA - Bo W , Ġand Ġ( ii ) Ġclust ering Ġwith Ġother Ġcommunity Ġdetection Ġmethods . ĠWe Ġprovide Ġa Ġbrief Ġsummary Ġhere Ġand Ġadditional Ġmaterial Ġin Ġthe ĠSI . </s> The Ġuse Ġof Ġfixed - sized Ġvector Ġembed d ings Ġ( Doc 2 V ec ) Ġinstead Ġof Ġstandard Ġbag Ġof Ġwords Ġ( Bo W ) Ġis Ġan Ġintegral Ġpart Ġof Ġour Ġpipeline . ĠDoc 2 V ec Ġproduces Ġlower Ġdimensional Ġvector Ġrepresentations Ġ( as Ġcompared Ġto ĠBo W ) Ġwith Ġhigher Ġsemantic Ġand Ġsynt actic Ġcontent . ĠIt Ġhas Ġbeen Ġreported Ġthat ĠDoc 2 V ec Ġoutper forms ĠBo W Ġrepresentations Ġin Ġpractical Ġbenchmarks Ġof Ġsemantic Ġsimilarity , Ġas Ġwell Ġas Ġbeing Ġless Ġsensitive Ġto Ġhyper - param eters ĠB IB REF 22 Ġ. </s> To Ġquantify Ġthe Ġimprovement Ġprovided Ġby ĠDoc 2 V ec Ġin Ġour Ġframework , Ġwe Ġconstructed Ġa ĠM ST - k NN Ġgraph Ġfollowing Ġthe Ġsame Ġsteps Ġbut Ġstarting Ġwith ĠTF - i DF Ġvectors Ġfor Ġeach Ġdocument . ĠWe Ġthen Ġran ĠMS Ġon Ġthis ĠTF - i DF Ġsimilarity Ġgraph , Ġand Ġcompared Ġthe Ġresults Ġto Ġthose Ġobtained Ġfrom Ġthe ĠDoc 2 V ec Ġsimilarity Ġgraph . ĠFigure Ġ7 Ġshows Ġthat Ġthe ĠDoc 2 V ec Ġversion Ġoutper forms Ġthe ĠBo W Ġversion Ġacross Ġall Ġresolutions Ġin Ġterms Ġof Ġboth Ġ$ N MI $ Ġand Ġ$\ wide hat { PM I } $ Ġscores . </s> As Ġshown Ġin ĠTable Ġ1 Ġ, Ġwe Ġhave Ġtested Ġthe Ġeffect Ġof Ġthe Ġsize Ġof Ġthe Ġtraining Ġcorpus Ġon Ġthe ĠDoc 2 V ec Ġmodel . ĠWe Ġtrained ĠDoc 2 V ec Ġon Ġtwo Ġadditional Ġtraining Ġsets Ġof Ġ1 Ġmillion Ġand Ġ2 Ġmillion Ġrecords Ġ( random ly Ġchosen Ġfrom Ġthe Ġfull Ġset Ġof Ġ$\ sim Ġ$ Ġ13 Ġmillion Ġrecords ). ĠWe Ġthen Ġfollowed Ġthe Ġsame Ġprocedure Ġto Ġconstruct Ġthe ĠM ST - k NN Ġsimilarity Ġgraph Ġand Ġcarried Ġout Ġthe ĠMS Ġanalysis . ĠThe Ġresults , Ġpresented Ġin ĠFigure ĠS 3 Ġin Ġthe ĠSI , Ġshow Ġthat Ġthe Ġperformance Ġis Ġaffected Ġonly Ġmildly Ġby Ġthe Ġsize Ġof Ġthe ĠDoc 2 V ec Ġtraining Ġset . </s> To Ġexamine Ġthe Ġeffect Ġof Ġsp ars ification Ġin Ġthe Ġgraph Ġconstruction , Ġwe Ġhave Ġstudied Ġthe Ġdependence Ġof Ġquality Ġof Ġthe Ġpartitions Ġagainst Ġthe Ġnumber Ġof Ġneighbours , Ġ$ k $ Ġ, Ġin Ġthe ĠM ST - k NN Ġgraph . ĠOur Ġnumer ics , Ġshown Ġin ĠFigure ĠS 4 Ġin Ġthe ĠSI , Ġindicate Ġthat Ġboth Ġthe Ġ$ N MI $ Ġand Ġ$\ wide hat { PM I } $ Ġscores Ġof Ġthe ĠMS Ġcluster ings Ġreach Ġa Ġsimilar Ġlevel Ġof Ġquality Ġfor Ġvalues Ġof Ġ$ k $ Ġabove Ġ13 - 16 , Ġwith Ġminor Ġimprovement Ġafter Ġthat . ĠHence Ġour Ġresults Ġare Ġrobust Ġto Ġthe Ġchoice Ġof Ġ$ k $ Ġ, Ġprovided Ġit Ġis Ġnot Ġtoo Ġsmall . ĠDue Ġto Ġcomputational Ġefficiency , Ġwe Ġthus Ġfavour Ġa Ġrelatively Ġsmall Ġ$ k $ Ġ, Ġbut Ġnot Ġtoo Ġsmall . </s> We Ġcarried Ġout Ġa Ġcomparison Ġwith ĠL DA , Ġa Ġwidely Ġused Ġmethodology Ġfor Ġtext Ġanalysis . ĠA Ġkey Ġdifference Ġbetween Ġstandard ĠL DA Ġand Ġour ĠMS Ġmethod Ġis Ġthe Ġfact Ġthat Ġa Ġdifferent ĠL DA Ġmodel Ġneeds Ġto Ġbe Ġtrained Ġseparately Ġfor Ġeach Ġnumber Ġof Ġtopics Ġpre - d etermined Ġby Ġthe Ġuser . ĠTo Ġoffer Ġa Ġcomparison Ġacross Ġthe Ġmethods , ĠWe Ġobtained Ġfive ĠL DA Ġmodels Ġcorresponding Ġto Ġthe Ġfive ĠMS Ġlevels Ġwe Ġconsidered Ġin Ġdetail . ĠThe Ġresults Ġin ĠTable Ġ2 Ġshow Ġthat ĠMS Ġand ĠL DA Ġgive Ġpartitions Ġthat Ġare Ġcompar ably Ġsimilar Ġto Ġthe Ġhand - coded Ġcategories Ġ( as Ġmeasured Ġwith Ġ$ N MI $ Ġ), Ġwith Ġsome Ġdifferences Ġdepending Ġon Ġthe Ġscale , Ġwhereas Ġthe ĠMS Ġcluster ings Ġhave Ġhigher Ġtopic Ġco herence Ġ( as Ġgiven Ġby Ġ$\ wide hat { PM I } $ Ġ) Ġacross Ġall Ġscales . </s> To Ġgive Ġan Ġindication Ġof Ġthe Ġcomputational Ġcost , Ġwe Ġran Ġboth Ġmethods Ġon Ġthe Ġsame Ġservers . ĠOur Ġmethod Ġtakes Ġapproximately Ġ13 Ġhours Ġin Ġtotal Ġto Ġcompute Ġboth Ġthe ĠDoc 2 V ec Ġmodel Ġon Ġ13 Ġmillion Ġrecords Ġ( 11 Ġhours ) Ġand Ġthe Ġfull ĠMS Ġscan Ġwith Ġ400 Ġpartitions Ġacross Ġall Ġresolutions Ġ( 2 Ġhours ). ĠThe Ġtime Ġrequired Ġto Ġtrain Ġjust Ġthe Ġ5 ĠL DA Ġmodels Ġon Ġthe Ġsame Ġcorpus Ġamounts Ġto Ġ30 Ġhours Ġ( with Ġtim ings Ġranging Ġfrom Ġ$\ sim Ġ$ Ġ2 Ġhours Ġfor Ġthe Ġ3 Ġtopic ĠL DA Ġmodel Ġto Ġ12 . 5 Ġhours Ġfor Ġthe Ġ44 Ġtopic ĠL DA Ġmodel ). </s> This Ġcomparison Ġalso Ġhighlights Ġthe Ġconceptual Ġdifference Ġbetween Ġour Ġmulti - scale Ġmethodology Ġand ĠL DA Ġtopic Ġmodelling . ĠWhile ĠL DA Ġcomp utes Ġtopics Ġat Ġa Ġpre - d etermined Ġlevel Ġof Ġresolution , Ġour Ġmethod Ġob tains Ġpartitions Ġat Ġall Ġresolutions Ġin Ġone Ġsweep Ġof Ġthe ĠMark ov Ġtime , Ġfrom Ġwhich Ġrelevant Ġpartitions Ġare Ġchosen Ġbased Ġon Ġtheir Ġrobust ness . ĠHowever , Ġthe ĠMS Ġpartitions Ġat Ġall Ġresolutions Ġare Ġavailable Ġfor Ġfurther Ġinvestigation Ġif Ġso Ġneeded . </s> We Ġhave Ġused Ġseveral Ġalgorithms Ġreadily Ġavailable Ġin Ġcode Ġlibraries Ġ( i . e ., Ġthe Ġi Graph Ġmodule Ġfor ĠPython ) Ġto Ġcluster / part ition Ġthe Ġsame Ġk NN - M ST Ġgraph . ĠFigure ĠS 5 Ġin Ġthe ĠSI Ġshows Ġthe Ġcomparison Ġagainst Ġseveral Ġwell - known Ġpartition ing Ġmethods Ġ( Mod ularity ĠOptim isation ĠB IB REF 44 Ġ, ĠInfo Map ĠB IB REF 4 Ġ, ĠWalk trap ĠB IB REF 45 Ġ, ĠLabel ĠProp ag ation ĠB IB REF 46 Ġ, Ġand ĠMulti - resolution ĠLou v ain ĠB IB REF 35 Ġ) Ġwhich Ġgive Ġjust Ġone Ġpartition Ġ( or Ġtwo Ġin Ġthe Ġcase Ġof Ġthe ĠLou v ain Ġimplementation Ġin Ġi Graph ) Ġinto Ġa Ġparticular Ġnumber Ġof Ġclusters , Ġin Ġcontrast Ġwith Ġour Ġmult isc ale ĠMS Ġanalysis . ĠOur Ġresults Ġshow Ġthat ĠMS Ġprovides Ġimproved Ġor Ġequal Ġresults Ġto Ġother Ġgraph Ġpartition ing Ġmethods Ġfor Ġboth Ġ$ N MI $ Ġand Ġ$\ wide hat { PM I } $ Ġacross Ġall Ġscales . ĠOnly Ġfor Ġvery Ġfine Ġresolution Ġwith Ġmore Ġthan Ġ50 Ġclusters , ĠInf om ap , Ġwhich Ġpartitions Ġgraphs Ġinto Ġsmall Ġcl ique - like Ġsub graph s ĠB IB REF 32 Ġ, ĠB IB REF 47 Ġ, Ġprovides Ġa Ġslightly Ġimproved Ġ$ N MI $ Ġfor Ġthat Ġparticular Ġscale . ĠTherefore , ĠMS Ġallows Ġus Ġto Ġfind Ġrelevant , Ġyet Ġhigh Ġquality Ġcluster ings Ġacross Ġall Ġscales Ġby Ġsweeping Ġthe ĠMark ov Ġtime Ġparameter . </s> Discussion </s> This Ġwork Ġhas Ġapplied Ġa Ġmult isc ale Ġgraph Ġpartition ing Ġalgorithm Ġ( Mark ov ĠStability ) Ġto Ġextract Ġcontent - based Ġclusters Ġof Ġdocuments Ġfrom Ġa Ġtextual Ġdataset Ġof Ġhealthcare Ġsafety Ġincident Ġreports Ġin Ġan Ġun super vised Ġmanner Ġat Ġdifferent Ġlevels Ġof Ġresolution . ĠThe Ġmethod Ġuses Ġparagraph Ġvectors Ġto Ġrepresent Ġthe Ġrecords Ġand Ġob tains Ġan Ġensuing Ġsimilarity Ġgraph Ġof Ġdocuments Ġconstructed Ġfrom Ġtheir Ġcontent . ĠThe Ġframework Ġbrings Ġthe Ġadvantage Ġof Ġmulti - resolution Ġalgorithms Ġcapable Ġof Ġcapturing Ġclusters Ġwithout Ġimposing Ġa Ġprior i Ġtheir Ġnumber Ġor Ġstructure . ĠSince Ġdifferent Ġlevels Ġof Ġresolution Ġof Ġthe Ġclust ering Ġcan Ġbe Ġfound Ġto Ġbe Ġrelevant , Ġthe Ġpractitioner Ġcan Ġchoose Ġthe Ġlevel Ġof Ġdescription Ġand Ġdetail Ġto Ġsuit Ġthe Ġrequirements Ġof Ġa Ġspecific Ġtask . </s> Our Ġa Ġposterior i Ġanalysis Ġevaluating Ġthe Ġsimilarity Ġagainst Ġthe Ġhand - coded Ġcategories Ġand Ġthe Ġintrinsic Ġtopic Ġco herence Ġof Ġthe Ġclusters Ġshowed Ġthat Ġthe Ġmethod Ġperformed Ġwell Ġin Ġrecovering Ġmeaningful Ġcategories . ĠThe Ġclusters Ġof Ġcontent Ġcapture Ġtopics Ġof Ġmedical Ġpractice , Ġthus Ġproviding Ġcomplementary Ġinformation Ġto Ġthe Ġexternally Ġimposed Ġclassification Ġcategories . ĠOur Ġanalysis Ġshows Ġthat Ġsome Ġof Ġthe Ġmost Ġrelevant Ġand Ġpersistent Ġcommunities Ġemerge Ġbecause Ġof Ġtheir Ġhighly Ġhom ogeneous Ġmedical Ġcontent , Ġalthough Ġthey Ġare Ġnot Ġeasily Ġmapped Ġto Ġthe Ġstandard ised Ġexternal Ġcategories . ĠThis Ġis Ġapparent Ġin Ġthe Ġmedically - based Ġcontent Ġclusters Ġassociated Ġwith ĠLabour Ġward , ĠPressure Ġul cer , ĠChem otherapy , ĠRad i otherapy , Ġamong Ġothers , Ġwhich Ġexempl ify Ġthe Ġalternative Ġgroup ings Ġthat Ġemerge Ġfrom Ġfree Ġtext Ġcontent . </s> The Ġcategories Ġin Ġthe Ġtop Ġlevel Ġ( Level Ġ1 ) Ġof Ġthe Ġpre - defined Ġclassification Ġhierarchy Ġare Ġhighly Ġdiverse Ġin Ġsize Ġ( as Ġshown Ġby Ġtheir Ġnumber Ġof Ġassigned Ġrecords ), Ġwith Ġlarge Ġgroups Ġsuch Ġas Ġ` Pat ient Ġaccident ', Ġ` Med ication ', Ġ` Cl inical Ġassessment ', Ġ` Document ation ', Ġ` Ad missions / Transfer ' Ġor Ġ` Inf rastructure ' Ġalongside Ġsmall , Ġspecific Ġgroups Ġsuch Ġas Ġ` Agg ressive Ġbehaviour ', Ġ` Pat ient Ġabuse ', Ġ` Self - harm ' Ġor Ġ` In fect ion Ġcontrol '. ĠOur Ġmulti - scale Ġpartition ing Ġfinds Ġcorresponding Ġgroups Ġin Ġcontent Ġacross Ġdifferent Ġlevels Ġof Ġresolution , Ġproviding Ġadditional Ġsub c ategories Ġwith Ġmedical Ġdetail Ġwithin Ġsome Ġof Ġthe Ġlarge Ġcategories Ġ( as Ġshown Ġin ĠFig . Ġ4 Ġand ĠS 1 Ġ). ĠAn Ġarea Ġof Ġfuture Ġresearch Ġwill Ġbe Ġto Ġconfirm Ġif Ġthe Ġcategories Ġfound Ġby Ġour Ġanalysis Ġare Ġconsistent Ġwith Ġa Ġsecond Ġlevel Ġin Ġthe Ġhierarchy Ġof Ġexternal Ġcategories Ġ( Level Ġ2 , Ġaround Ġ100 Ġcategories ) Ġthat Ġis Ġused Ġless Ġconsistently Ġin Ġhospital Ġsettings . ĠThe Ġuse Ġof Ġcontent - driven Ġclassification Ġof Ġreports Ġcould Ġalso Ġbe Ġimportant Ġwithin Ġcurrent Ġefforts Ġby Ġthe ĠWorld ĠHealth ĠOrganisation Ġ( WHO ) Ġunder Ġthe Ġframework Ġfor Ġthe ĠInternational ĠClassification Ġfor ĠPatient ĠSafety Ġ( IC PS ) ĠB IB REF 48 Ġto Ġestablish Ġa Ġset Ġof Ġconceptual Ġcategories Ġto Ġmonitor , Ġanalyse Ġand Ġinterpret Ġinformation Ġto Ġimprove Ġpatient Ġcare . </s> One Ġof Ġthe Ġadvantages Ġof Ġa Ġfree Ġtext Ġanalytical Ġapproach Ġis Ġthe Ġprovision , Ġin Ġa Ġtimely Ġmanner , Ġof Ġan Ġintellig ible Ġdescription Ġof Ġincident Ġreport Ġcategories Ġderived Ġdirectly Ġfrom Ġthe Ġrich Ġdescription Ġin Ġthe Ġ' words ' Ġof Ġthe Ġreporter Ġthemselves . ĠThe Ġinsight Ġfrom Ġanalys ing Ġthe Ġfree Ġtext Ġentry Ġof Ġthe Ġperson Ġreporting Ġcould Ġplay Ġa Ġvaluable Ġrole Ġand Ġadd Ġrich Ġinformation Ġthan Ġwould Ġhave Ġotherwise Ġbeen Ġobtained Ġfrom Ġthe Ġexisting Ġapproach Ġof Ġpre - defined Ġclasses . ĠNot Ġonly Ġcould Ġthis Ġimprove Ġthe Ġcurrent Ġstate Ġof Ġplay Ġwhere Ġmuch Ġof Ġthe Ġfree Ġtext Ġof Ġthese Ġreports Ġgoes Ġunused , Ġbut Ġit Ġavoids Ġthe Ġfallacy Ġof Ġassigning Ġincidents Ġto Ġa Ġpre - defined Ġcategory Ġthat , Ġthrough Ġa Ġlack Ġof Ġgran ularity , Ġcan Ġmiss Ġan Ġimportant Ġopportunity Ġfor Ġfeedback Ġand Ġlearning . ĠThe Ġnuanced Ġinformation Ġand Ġclass ifications Ġextracted Ġfrom Ġfree Ġtext Ġanalysis Ġthus Ġsuggest Ġa Ġcomplementary Ġaxis Ġto Ġexisting Ġapproaches Ġto Ġcharacter ise Ġpatient Ġsafety Ġincident Ġreports . </s> Currently , Ġlocal Ġincident Ġreporting Ġsystem Ġare Ġused Ġby Ġhospitals Ġto Ġsubmit Ġreports Ġto Ġthe ĠNR LS Ġand Ġrequire Ġrisk Ġmanagers Ġto Ġimprove Ġdata Ġquality Ġof Ġreports , Ġdue Ġto Ġerrors Ġor Ġuncertainty Ġin Ġcategor isation Ġfrom Ġreporters , Ġbefore Ġsubmission . ĠThe Ġapplication Ġof Ġfree Ġtext Ġanalytical Ġapproaches , Ġlike Ġthe Ġone Ġwe Ġhave Ġpresented Ġhere , Ġhas Ġthe Ġpotential Ġto Ġfree Ġup Ġrisk Ġmanagers Ġtime Ġfrom Ġlabour - intensive Ġtasks Ġof Ġclassification Ġand Ġcorrection Ġby Ġhuman Ġoperators , Ġinstead Ġfor Ġquality Ġimprovement Ġactivities Ġderived Ġfrom Ġthe Ġintelligence Ġof Ġthe Ġdata Ġitself . ĠAdditionally , Ġthe Ġmethod Ġallows Ġfor Ġthe Ġdiscovery Ġof Ġemerging Ġtopics Ġor Ġclasses Ġof Ġincidents Ġdirectly Ġfrom Ġthe Ġdata Ġwhen Ġsuch Ġevents Ġdo Ġnot Ġfit Ġthe Ġpre - ass igned Ġcategories Ġby Ġusing Ġprojection Ġtechniques Ġalongside Ġmethods Ġfor Ġanomaly Ġand Ġinnovation Ġdetection . </s> In Ġongoing Ġwork , Ġwe Ġare Ġcurrently Ġexamining Ġthe Ġuse Ġof Ġour Ġcharacter isation Ġof Ġincident Ġreports Ġto Ġenable Ġcomparisons Ġacross Ġhealthcare Ġorganisations Ġand Ġalso Ġto Ġmonitor Ġtheir Ġchange Ġover Ġtime . ĠThis Ġpart Ġof Ġongoing Ġresearch Ġrequires Ġthe Ġquant ification Ġof Ġin - class Ġtext Ġsimilarities Ġand Ġto Ġdynamically Ġmanage Ġthe Ġembed ding Ġof Ġthe Ġreports Ġthrough Ġupdates Ġand Ġrecal culation Ġof Ġthe Ġvector Ġembed ding . ĠImprovements Ġin Ġthe Ġprocess Ġof Ġrobust Ġgraph Ġconstruction Ġare Ġalso Ġpart Ġof Ġour Ġfuture Ġwork . ĠDetect ing Ġanomalies Ġin Ġthe Ġdata Ġto Ġdecide Ġwhether Ġnewer Ġtopic Ġclusters Ġshould Ġbe Ġcreated , Ġor Ġproviding Ġonline Ġclassification Ġsuggestions Ġto Ġusers Ġbased Ġon Ġthe Ġtext Ġthey Ġinput Ġare Ġsome Ġof Ġthe Ġimprovements Ġwe Ġaim Ġto Ġadd Ġin Ġthe Ġfuture Ġto Ġaid Ġwith Ġdecision Ġsupport Ġand Ġdata Ġcollection , Ġand Ġto Ġpotentially Ġhelp Ġfine - t une Ġsome Ġof Ġthe Ġpred efined Ġcategories Ġof Ġthe Ġexternal Ġclassification . </s> Availability Ġof ĠData Ġand ĠMaterials </s> The Ġdataset Ġin Ġthis Ġwork Ġis Ġmanaged Ġby Ġthe ĠBig ĠData Ġand ĠAnalytics ĠUnit Ġ( B DA U ), ĠImperial ĠCollege ĠLondon , Ġand Ġconsists Ġof Ġincident Ġreports Ġsubmitted Ġto Ġthe ĠNR LS . ĠAnalysis Ġof Ġthe Ġdata Ġwas Ġundertaken Ġwithin Ġthe ĠSecure ĠEnvironment Ġof Ġthe ĠB DA U . ĠDue Ġto Ġits Ġnature , Ġwe Ġcannot Ġpublic ise Ġany Ġpart Ġof Ġthe Ġdataset , Ġbeyond Ġthat Ġalready Ġprovided Ġwithin Ġthis Ġmanuscript . ĠNo Ġindividual Ġidentifiable Ġpatient Ġinformation Ġis Ġdisclosed Ġin Ġthis Ġwork . ĠOnly Ġaggreg ated Ġinformation Ġis Ġused Ġto Ġdescribe Ġthe Ġclusters . </s> Comp eting Ġinterests </s> The Ġauthors Ġdeclare Ġthat Ġthey Ġhave Ġno Ġcompeting Ġinterests . </s> List Ġof Ġabbrevi ations </s> N HS : ĠNational ĠHealth ĠService ; ĠNR LS : ĠNational ĠReporting Ġand ĠLearning ĠSystem ; ĠBo W : ĠBag Ġof ĠWords ; ĠL DA : ĠLat ent ĠDir ich let ĠAll ocation ; ĠDoc 2 V ec : ĠDocument Ġto ĠVector ; ĠM ST : ĠMinimum ĠSpan ning ĠTree ; Ġk NN : Ġk - Ne arest ĠNeigh b ours ; ĠMS : ĠMark ov ĠStability ; ĠNL T K : ĠNatural ĠLanguage ĠTool kit ; ĠTF - i DF : ĠTerm ĠFrequency Ġ- Ġinverse ĠDocument ĠFrequency ; ĠPV : ĠPar agraph ĠV ect ors ; ĠDB OW : ĠDist ributed ĠBag Ġof ĠWords ; ĠVI : ĠVari ation Ġof ĠInformation ; ĠN MI : ĠNormal ised ĠMutual ĠInformation ; ĠPM I : ĠPair wise ĠMutual ĠInformation . </s> Auth ors ' Ġcontributions </s> M TA Ġconducted Ġthe Ġcomputational Ġresearch . ĠMTA Ġand ĠMB Ġanalysed Ġthe Ġdata Ġand Ġdesigned Ġthe Ġcomputational Ġframework . ĠMB , ĠEM Ġand ĠSN Y Ġconceived Ġthe Ġstudy . ĠAll Ġauthors Ġwrote Ġthe Ġmanuscript . </s> Acknowled gements </s> We Ġthank ĠJoshua ĠSy mons Ġfor Ġhelp Ġwith Ġaccessing Ġthe Ġdata . ĠWe Ġalso Ġthank ĠElias ĠBam is , ĠZ ijing ĠLiu Ġand ĠMichael ĠScha ub Ġfor Ġhelpful Ġdiscussions . ĠThis Ġresearch Ġwas Ġsupported Ġby Ġthe ĠNational ĠInstitute Ġfor ĠHealth ĠResearch Ġ( NI HR ) ĠImperial ĠPatient ĠSafety ĠTrans l ational ĠResearch ĠCentre Ġand ĠNI HR ĠImperial ĠBi omedical ĠResearch ĠCentre . ĠThe Ġviews Ġexpressed Ġare Ġthose Ġof Ġthe Ġauthors Ġand Ġnot Ġnecessarily Ġthose Ġof Ġthe ĠNHS , Ġthe ĠNI HR , Ġor Ġthe ĠDepartment Ġof ĠHealth . ĠAll Ġauthors Ġacknowledge Ġfunding Ġfrom Ġthe ĠEPS RC Ġthrough Ġaward ĠEP / N 01 45 29 / 1 Ġfunding Ġthe ĠEPS RC ĠCentre Ġfor ĠMathematics Ġof ĠPrecision ĠHealthcare . </s> Auth ors ' ĠInformation </s> M TA Ġis Ġa ĠPhD Ġstudent Ġat ĠImperial ĠCollege ĠLondon , ĠDepartment Ġof ĠMathematics . ĠHe Ġholds Ġan ĠM Sc Ġdegree Ġin Ġfinance Ġfrom ĠSaban ci ĠUniversity Ġand Ġa ĠB Sc Ġin ĠElectrical Ġand ĠElectronics ĠEngineering Ġfrom ĠBog az ici ĠUniversity . ĠEM Ġis Ġa ĠClinical ĠSenior ĠLect urer Ġin Ġthe ĠDepartment Ġof ĠSurgery Ġand ĠCancer Ġand ĠCentre Ġfor ĠHealth ĠPolicy Ġat ĠImperial ĠCollege ĠLondon Ġand ĠTransformation ĠChief ĠClinical ĠInformation ĠOfficer Ġ( Cl inical ĠAnalytics Ġand ĠIn format ics ), ĠI CH NT . ĠSN Y Ġis Ġa ĠProfessor Ġof ĠThe oret ical ĠChemistry Ġin Ġthe ĠDepartment Ġof ĠChemistry Ġat ĠImperial ĠCollege ĠLondon Ġand Ġalso Ġwith Ġthe ĠEPS RC ĠCentre Ġfor ĠMathematics Ġof ĠPrecision ĠHealthcare . ĠMB Ġis ĠProfessor Ġof ĠMathematics Ġand ĠChair Ġin ĠBi om at hemat ics Ġin Ġthe ĠDepartment Ġof ĠMathematics Ġat ĠImperial ĠCollege ĠLondon , Ġand ĠDirector Ġof Ġthe ĠEPS RC ĠCentre Ġfor ĠMathematics Ġof ĠPrecision ĠHealthcare Ġat ĠImperial .
Introduction The vast amounts of data collected by healthcare providers in conjunction with modern data analytics techniques present a unique opportunity to improve health service provision and the quality and safety of medical care for patient benefit BIBREF0 . Much of the recent research in this area has been on personalised medicine and its aim to deliver better diagnostics aided by the integration of diverse datasets providing complementary information. Another large source of healthcare data is organisational. In the United Kingdom, the National Health Service (NHS) has a long history of documenting extensively the different aspects of healthcare provision. The NHS is currently in the process of increasing the availability of several databases, properly anonymised, with the aim of leveraging advanced analytics to identify areas of improvement in NHS services. One such database is the National Reporting and Learning System (NRLS), a central repository of patient safety incident reports from the NHS in England and Wales. Set up in 2003, the NRLS now contains more than 13 million detailed records. The incidents are reported using a set of standardised categories and contain a wealth of organisational and spatio-temporal information (structured data), as well as, crucially, a substantial component of free text (unstructured data) where incidents are described in the `voice' of the person reporting. The incidents are wide ranging: from patient accidents to lost forms or referrals; from delays in admission and discharge to serious untoward incidents, such as retained foreign objects after operations. The review and analysis of such data provides critical insight into the complex functioning of different processes and procedures in healthcare towards service improvement for safer carer. Although statistical analyses are routinely performed on the structured component of the data (dates, locations, assigned categories, etc), the free text remains largely unused in systematic processes. Free text is usually read manually but this is time-consuming, meaning that it is often ignored in practice, unless a detailed review of a case is undertaken because of the severity of harm that resulted. There is a lack of methodologies that can summarise content and provide content-based groupings across the large volume of reports submitted nationally for organisational learning. Methods that could provide automatic categorisation of incidents from the free text would sidestep problems such as difficulties in assigning an incident category by virtue of a priori pre-defined lists in the reporting system or human error, as well as offering a unique insight into the root cause analysis of incidents that could improve the safety and quality of care and efficiency of healthcare services. Our goal in this work is to showcase an algorithmic methodology that detects content-based groups of records in a given dataset in an unsupervised manner, based only on the free and unstructured textual description of the incidents. To do so, we combine recently developed deep neural-network high-dimensional text-embedding algorithms with network-theoretical methods. In particular, we apply multiscale Markov Stability (MS) community detection to a sparsified geometric similarity graph of documents obtained from text vector similarities. Our method departs from traditional natural language processing tools, which have generally used bag-of-words (BoW) representation of documents and statistical methods based on Latent Dirichlet Allocation (LDA) to cluster documents BIBREF1 . More recent approaches have used deep neural network based language models clustered with k-means, without a full multiscale graph analysis BIBREF2 . There have been some previous applications of network theory to text analysis. For example, Lanchichinetti and co-workers BIBREF3 used a probabilistic graph construction analysed with the InfoMap algorithm BIBREF4 ; however, their community detection was carried out at a single-scale and the representation of text as BoW arrays lacks the power of neural network text embeddings. The application of multiscale community detection allows us to find groups of records with consistent content at different levels of resolution; hence the content categories emerge from the textual data, rather than fitting with pre-designed classifications. The obtained results could thus help mitigate possible human error or effort in finding the right category in complex category classification trees. We showcase the methodology through the analysis of a dataset of patient incidents reported to the NRLS. First, we use the 13 million records collected by the NRLS since 2004 to train our text embedding (although a much smaller corpus can be used). We then analyse a subset of 3229 records reported from St Mary's Hospital, London (Imperial College Healthcare NHS Trust) over three months in 2014 to extract clusters of incidents at different levels of resolution in terms of content. Our method reveals multiple levels of intrinsic structure in the topics of the dataset, as shown by the extraction of relevant word descriptors from the grouped records and a high level of topic coherence. Originally, the records had been manually coded by the operator upon reporting with up to 170 features per case, including a two-level manual classification of the incidents. Therefore, we also carried out an a posteriori comparison against the hand-coded categories assigned by the reporter (healthcare personnel) at the time of the report submission. Our results show good overall correspondence with the hand-coded categories across resolutions and, specifically, at the medium level of granularity. Several of our clusters of content correspond strongly to well-defined categories, yet our results also reveal complementary categories of incidents not defined in the external classification. In addition, the tuning of the granularity afforded by the method can be used to provide a distinct level of resolution in certain areas corresponding to specialise or particular sub-themes. Multiscale graph partitioning for text analysis: description of the framework Our framework combines text-embedding, geometric graph construction and multi-resolution community detection to identify, rather than impose, content-based clusters from free, unstructured text in an unsupervised manner. Figure 1 shows a summary of our pipeline. First, we pre-process each document to transform text into consecutive word tokens, where words are in their most normalised forms, and some words are removed if they have no distinctive meaning when used out of context BIBREF5 , BIBREF6 . We then train a paragraph vector model using the Document to Vector (Doc2Vec) framework BIBREF7 on the whole set (13 million) of preprocessed text records, although training on smaller sets (1 million) also produces good results. This training step is only done once. This Doc2Vec model is subsequently used to infer high-dimensional vector descriptions for the text of each of the 3229 documents in our target analysis set. We then compute a matrix containing pairwise similarities between any pair of document vectors, as inferred with Doc2Vec. This matrix can be thought of as a full, weighted graph with documents as nodes and edges weighted by their similarity. We sparsify this graph to the union of a minimum spanning tree and a k-Nearest Neighbors (MST-kNN) graph BIBREF8 , a geometric construction that removes less important similarities but preserves global connectivity for the graph and, hence, for the dataset. The derived MST-kNN graph is analysed with Markov Stability BIBREF9 , BIBREF10 , BIBREF11 , BIBREF12 , a multi-resolution dynamics-based graph partitioning method that identifies relevant subgraphs (i.e., clusters of documents) at different levels of granularity. MS uses a diffusive process on the graph to reveal the multiscale organisation at different resolutions without the need for choosing a priori the number of clusters, scale or organisation. To analyse a posteriori the different partitions across levels of resolution, we use both visualisations and quantitative scores. The visualisations include word clouds to summarise the main content, graph layouts, as well as Sankey diagrams and contingency tables that capture the correspondences across levels of resolution and relationships to the hand-coded classifications. The partitions are also evaluated quantitatively to score: (i) their intrinsic topic coherence (using pairwise mutual information BIBREF13 , BIBREF14 ), and (ii) their similarity to the operator hand-coded categories (using normalised mutual information BIBREF15 ). We now expand on the steps of the computational framework. Data description The full dataset includes more than 13 million confidential reports of patient safety incidents reported to the National Reporting and Learning System (NRLS) between 2004 and 2016 from NHS trusts and hospitals in England and Wales. Each record has more than 170 features, including organisational details (e.g., time, trust code and location), anonymised patient information, medication and medical devices, among other details. The records are manually classified by operators to a two-level system of categories of incident type. In particular, the top level contains 15 categories including general groups such as `Patient accident', `Medication', `Clinical assessment', `Documentation', `Admissions/Transfer' or `Infrastructure' alongside more specific groups such as `Aggressive behaviour', `Patient abuse', `Self-harm' or `Infection control'. In most records, there is also a detailed description of the incident in free text, although the quality of the text is highly variable. Our analysis set for clustering is the group of 3229 records reported during the first quarter of 2014 at St. Mary's Hospital in London (Imperial College Healthcare NHS Trust). Text Preprocessing Text preprocessing is important to enhance the performance of text embedding. We applied standard preprocessing techniques in natural language processing to the raw text of all 13 million records in our corpus. We normalise words into a single form and remove words that do not carry significant meaning. Specifically, we divide our documents into iterative word tokens using the NLTK library BIBREF5 and remove punctuation and digit-only tokens. We then apply word stemming using the Porter algorithm BIBREF6 , BIBREF16 . If the Porter method cannot find a stemmed version for a token, we apply the Snowball algorithm BIBREF17 . Finally, we remove any stop-words (repeat words with low content) using NLTK's stop-word list. Although some of the syntactic information is reduced due to text preprocessing, this process preserves and consolidates the semantic information of the vocabulary, which is of relevance to our study. Text Embedding Computational methods for text analysis rely on a choice of a mathematical representation of the base units, such as character $n$ -grams, words or documents of any length. An important consideration for our methodology is an attempt to avoid the use of labelled data at the core of many supervised or semi-supervised classification methods BIBREF18 , BIBREF19 . In this work, we use a representation of text documents in vector form following recent developments in the field. Classically, bag-of-words (Bow) methods were used to obtain representations of the documents in a corpus in terms of vectors of term frequencies weighted by inverse document frequency (TF-iDF). While such methods provide a statistical description of documents, they do not carry information about the order or proximity of words to each other since they regard word tokens in an independent manner with no semantic or syntactic relationships considered. Furthermore, BoW representations tend to be high-dimensional and sparse, due to large sizes of word dictionaries and low frequencies of many terms. Recently, deep neural network language models have successfully overcome certain limitations of BoW methods by incorporating word neighbourhoods in the mathematical description of each term. Distributed Bag of Words (DBOW) is a form of Paragraph Vectors (PV), also known as Doc2Vec BIBREF7 . This method creates a model which represents any length of word sequences (i.e. sentences, paragraphs, documents) as $d$ -dimensional vectors, where $d$ is a user-defined parameter (typically $d=500$ ). Training a Doc2Vec model starts with a random $d$ -dimensional vector assignment for each document in the corpus. A stochastic gradient descent algorithm iterates over the corpus with the objective of predicting a randomly sampled set of words from each document by using only the document's $d$ -dimensional vector BIBREF7 . The objective function being optimised by PV-DBOW is similar to the skip-gram model in Refs. BIBREF20 , BIBREF21 . Doc2Vec has been shown BIBREF22 to capture both semantic and syntactic characterisations of the input text outperforming BoW models, such as LDA BIBREF1 . Here, we use the Gensim Python library BIBREF23 to train the PV-DBOW model. The Doc2Vec training was repeated several times with a variety of training hyper-parameters to optimise the output based on our own numerical experiments and the general guidelines provided by BIBREF24 . We trained Doc2Vec models using text corpora of different sizes and content with different sets of hyper-parameters, in order to characterise the usability and quality of models. Specifically, we checked the effect of corpus size on model quality by training Doc2Vec models on the full 13 million NRLS records and on subsets of 1 million and 2 million randomly sampled records. (We note that our target subset of 3229 records has been excluded from these samples.) Furthermore, we checked the importance of the specificity of the text corpus by obtaining a Doc2Vec model from a generic, non-specific set of 5 million articles from Wikipedia representing standard English usage across a variety of topics. We benchmarked the Doc2Vec models by scoring how well the document vectors represent the semantic topic structure: (i) calculating centroids for the 15 externally hand-coded categories; (ii) selecting the 100 nearest reports for each centroid; (iii) counting the number of incident reports (out of 1500) correctly assigned to their centroid. The results in Table 1 show that training on the highly specific text in the NRLS records is an important ingredient in the successful vectorisation of the documents, as shown by the degraded performance for the Wikipedia model across a variety of training hyper-parameters. Our results also show that reducing the size of the corpus from 13 million to 1 million records did not affect the benchmarking dramatically. This robustness of the results to the size of the training corpus was confirmed further with the use of more detailed metrics, as discussed below in Section sec:comparisons. Based on our benchmarking, we use henceforth (unless otherwise noted) the optimised Doc2Vec model obtained from the 13+ million NRLS records with the following hyper-parameters: {training method = dbow, number of dimensions for feature vectors size = 300, number of epochs = 10, window size = 15, minimum count = 5, number of negative samples = 5, random down-sampling threshold for frequent words = 0.001 }. As an indication of computational cost, the training of the model on the 13 million records takes approximately 11 hours (run in parallel with 7 threads) on shared servers. Graph Construction Once the Doc2Vec model is trained, we use it to infer a vector for each of the $N=3229$ records in our analysis set. We then construct a normalised cosine similarity matrix between the vectors by: computing the matrix of cosine similarities between all pairs of records, $S_\text{cos}$ ; transforming it into a distance matrix $D_{cos} = 1-S_{cos}$ ; applying element-wise max norm to obtain $\hat{D}=\Vert D_{cos}\Vert _{max}$ ; and normalising the similarity matrix $\hat{S} = 1-\hat{D}$ which has elements in the interval $[0,1]$ . The similarity matrix can be thought of as the adjacency matrix of a fully connected weighted graph. However, such a graph contains many edges with small weights reflecting weak similarities in high-dimensional noisy datasets even the least similar nodes present a substantial degree of similarity. Such weak similarities are in most cases redundant, as they can be explained through stronger pairwise similarities present in the graph. These weak, redundant edges obscure the graph structure, as shown by the diffuse, spherical visualisation of the full graph layout in Figure 2 A. To reveal the graph structure, we obtain a MST-kNN graph from the normalised similarity matrix BIBREF8 . This is a simple sparsification based on a geometric heuristic that preserves the global connectivity of the graph while retaining details about the local geometry of the dataset. The MST-kNN algorithm starts by computing the minimum spanning tree (MST) of the full matrix $\hat{D}$ , i.e., the tree with $(N-1)$ edges connecting all nodes in the graph with minimal sum of edge weights (distances). The MST is computed using the Kruskal algorithm implemented in SciPy BIBREF25 . To this MST, we add edges connecting each node to its $k$ nearest nodes (kNN) if they are not already in the MST. Here $k$ is an user-defined parameter. The binary adjacency matrix of the MST-kNN graphs, $E_\text{MST-kNN}$ , is Hadamard-multiplied with $\hat{S}$ to give the adjacency matrix $A$ of the weighted, undirected sparsified graph. The MST-kNN method avoids a direct thresholding of the weights in $\hat{S}$ , and obtains a graph description that preserves local geometric information together with a global subgraph (the MST) that captures properties of the full dataset. The network layout visualisations in Figure 2 B–E give an intuitive picture of the effect of the sparsification. The highly sparse graphs obtained when the number of neighbours $k$ is very small are not robust. As $k$ is increased, the local similarities between documents induce the formation of dense subgraphs (which appear closer in the graph visualisation layout). When the number of neighbours becomes too large, the local structure becomes diffuse and the subgraphs lose coherence, signalling the degradation of the local graph structure. Figure 2 shows that the MST-kNN graph with $k=13$ presents a reasonable balance between local and global structure. Relatively sparse graphs that preserve important edges and global connectivity of the dataset (guaranteed here by the MST) have computational advantages when using community detection algorithms. The MST-kNN construction has been reported to be robust to the selection of the parameter $k$ due to the guaranteed connectivity provided by the MST BIBREF8 . In the following, we fix $k=13$ for our analysis with the multi-scale graph partitioning framework, but we have scanned values of $k \in [1,50]$ in the graph construction from our data and have found that the construction is robust as long as $k$ is nor too small (i.e., $k > 13$ ). The detailed comparisons are shown in Section sec:comparisons. The MST-kNN construction has the advantage of its simplicity and robustness, and the fact that it balances the local and global structure of the data. However, the area of network inference and graph construction from data, and graph sparsification is very active, and several alternative approaches exist based on different heuristics, e.g., Graphical Lasso BIBREF26 , Planar Maximally Filtered Graph BIBREF27 , spectral sparsification BIBREF28 , or the Relaxed Minimum Spanning Tree (RMST) BIBREF29 . We have experimented with some of those methods and obtained comparable results. A detailed comparison of sparsification methods as well as the choice of distance in defining the similarity matrix $\hat{S}$ is left for future work. Multiscale Graph Partitioning The area of community detection encompasses a variety of graph partitioning approaches which aim to find `good' partitions into subgraphs (or communities) according to different cost functions, without imposing the number of communities a priori BIBREF31 . The notion of community thus depends on the choice of cost function. Commonly, communities are taken to be subgraphs whose nodes are connected strongly within the community with relatively weak inter-community edges. Such structural notion is related to balanced cuts. Other cost functions are posed in terms of transitions inside and outside of the communities, usually as one-step processes BIBREF4 . When transition paths of random walks of all lengths are considered, the concept of community becomes intrinsically multi-scale, i.e., different partitions can be found to be relevant at different time scales leading to a multi-level description dictated by the transition dynamics BIBREF9 , BIBREF32 , BIBREF10 . This leads to the framework of Markov Stability, a dynamics-based, multi-scale community detection methodology, which can be shown to recover seamlessly several well-known heuristics as particular cases BIBREF9 , BIBREF11 , BIBREF12 . Here, we apply MS to find partitions of the similarity graph $A$ at different levels of resolution. The subgraphs detected correspond to clusters of documents with similar content. MS is an unsupervised community detection method that finds robust and stable partitions under the evolution of a continuous-time diffusion process without a priori choice of the number or type of communities or their organisation BIBREF9 , BIBREF32 , BIBREF10 , BIBREF33 The code for Markov Stability is open and accessible at https://github.com/michaelschaub/PartitionStability and http://wwwf.imperial.ac.uk/~mpbara/Partition_Stability/, last accessed on March 24, 2018. In simple terms, MS can be understood by analogy to a drop of ink diffusing on the graph under a diffusive Markov process. The ink diffuses homogeneously unless the graph has some intrinsic structural organisation, in which case the ink gets transiently contained, over particular time scales, within groups of nodes (i.e., subgraphs or communities). The existence of this transient containment signals the presence of a natural partition of the graph. As the process evolves, the ink diffuses out of those initial communities but might get transiently contained in other, larger subgraphs. By analysing this Markov dynamics over time, MS detects the structure of the graph across scales. The Markov time $t$ thus acts as a resolution parameter that allows us to extract robust partitions that persist over particular time scales, in an unsupervised manner. Given the adjacency matrix $A_{N \times N}$ of the graph obtained as described previously, let us define the diagonal matrix $D=\text{diag}(\mathbf {d})$ , where $\mathbf {d}=A \mathbf {1}$ is the degree vector. The random walk Laplacian matrix is defined as $L_\text{RW}=I_N-D^{-1}A$ where $I_N$ is the identity matrix of size $N$ , and the transition matrix (or kernel) of the associated continuous-time Markov process is $P(t)=e^{-t L_\text{RW}}, \, t>0$ BIBREF10 . For each partition, a binary membership matrix $H_{N \times C}$ maps the $N$ nodes into $C$ clusters. We can then define the $D=\text{diag}(\mathbf {d})$0 clustered autocovariance matrix: $$R(t,H) = H^T[\Pi P(t)-\pi \pi ^T]H $$   (Eq. 4) where $\pi $ is the steady-state distribution of the process and $\Pi =\text{diag}(\pi )$ . The element $[R(t,H)]_{\alpha \beta }$ quantifies the probability that a random walker starting from community $\alpha $ will end in community $\beta $ at time $t$ , subtracting the probability that the same event occurs by chance at stationarity. We then define our cost function measuring the goodness of a partition over time $t$ , termed the Markov Stability of partition $H$ : $$r(t,H) = \text{trace} \left[R(t,H)\right]. $$   (Eq. 5) A partition $H$ that maximises $r(t,H)$ is comprised of communities that preserve the flow within themselves over time $t$ , since in that case the diagonal elements of $R(t,H)$ will be large and the off-diagonal elements will be small. For details, see BIBREF9 , BIBREF32 , BIBREF10 , BIBREF34 . MS searches for partitions at each Markov time that maximise $r(t,H)$ . Although the maximisation of ( 5 ) is an NP-hard problem (hence with no guarantees for global optimality), there are efficient optimisation methods that work well in practice. Our implementation here uses the Louvain Algorithm BIBREF35 , BIBREF12 which is efficient and known to give good results when applied to benchmarks BIBREF36 . To obtain robust partitions, we run the Louvain algorithm 500 times with different initialisations at each Markov time and pick the best 50 with the highest Markov Stability value $r(t,H)$ . We then compute the variation of information BIBREF37 of this ensemble of solutions $VI(t)$ . as a measure of the reproducibility of the result under the optimisation. In addition, the relevant partitions are required to be persistent across time, as given by low values of the variation of information between optimised partitions across time $VI(t,t^{\prime })$ . Robust partitions are thus indicated by Markov times where $VI(t)$ shows a dip and $VI(t,t^{\prime })$ has an extended plateau, indicating consistent results from different Louvain runs and validity over extended scales BIBREF34 , BIBREF10 . Visualisation and interpretation of the results We use the ForceAtlas2 BIBREF30 layout to represent the graph of 3229 NRLS Patient Incident reports. This layout follows a force-directed iterative method to find node positions that balance attractive and repulsive forces. Hence similar nodes tend to be grouped together on the planar layout. We colour the nodes by either hand-coded categories (Figure 2 ) or multiscale MS communities (Figure 3 ). Spatially consistent colourings on this layout imply good clusters of documents in terms of the similarity graph. Sankey diagrams allow us to visualise the relationship of node memberships across different partitions and with respect to the hand-coded categories. In particular, two-layer Sankey diagrams (e.g., Fig. 4 ) reflect the correspondence between MS clusters and the hand-coded external categories, whereas the multilayer Sankey diagram in Fig. 3 represents the results of the multi-resolution MS community detection across scales. In addition to Sankey diagrams between our MS clusters and the hand-coded categories, we also provide a complementary visualisation as heatmaps of normalised contingency (z-score) tables, e.g., Fig. 4 . This allows us to compare the relative association of content clusters to the external categories at different resolution levels. A quantification of this correspondence is also provided by the $NMI$ score introduced in Eq. ( 8 ). Our method clusters text documents according to their intrinsic content. This can be understood as a type of topic detection. To understand the content of the clusters, we use Word Clouds as basic, yet intuitive, tools that summarise information from a group of documents. Word clouds allow us to evaluate the results and extract insights when comparing a posteriori with hand-coded categories. They can also provide an aid for monitoring results when used by practitioners. The stemming methods described in the Text Preprocessing subsection truncate words severely. Such truncation enhances the power of the language processing computational methods, as it reduces the redundancy in the word corpus. Yet when presenting the results back to a human observer, it is desirable to report the content of the clusters with words that are readily comprehensible. To generate comprehensible word clouds in our a posteriori analyses, we use a text processing method similar to the one described in BIBREF38 . Specifically, we use the part of speech (POS) tagging module from NLTK to leave out sentence parts except the adjectives, nouns, and verbs. We also remove less meaningful common verbs such as `be', `have', and `do' and their variations. The residual words are then lemmatised and represented with their lemmas in order to normalise variations of the same word. Once the text is processed in this manner, we use the Python library wordcloudThe word cloud generator library for Python is open and accessible at https://github.com/amueller/word_cloud, last accessed on March 25, 2018 to create word clouds with 2 or 3-gram frequency list of common word groups. The results present distinct, understandable word topics. Quantitative benchmarking of topic clusters Although our dataset has attached a hand-coded classification by a human operator, we do not use it in our analysis and we do not consider it as a `ground truth'. Indeed, one of our aims is to explore the relevance of the fixed external classes as compared to the content-driven groupings obtained in an unsupervised manner. Hence we provide a double route to quantify the quality of the clusters by computing two complementary measures: an intrinsic measure of topic coherence and a measure of similarity to the external hand-coded categories, defined as follows. As an intrinsic measure of consistency of word association without any reference to an external `ground truth', we use the pointwise mutual information ( $PMI$ ) BIBREF13 , BIBREF14 . The $PMI$ is an information-theoretical score that captures the probability of being used together in the same group of documents. The $PMI$ score for a pair of words $(w_1,w_2)$ is: $$PMI(w_1,w_2)=\log {\frac{P(w_1 w_2)}{P(w_1)P(w_2)}}$$   (Eq. 6) where the probabilities of the words $P(w_1)$ , $P(w_2)$ , and of their co-occurrence $P(w_1 w_2)$ are obtained from the corpus. To obtain the aggregate $\widehat{PMI}$ for the graph partition $C=\lbrace c_i\rbrace $ we compute the $PMI$ for each cluster, as the median $PMI$ between its 10 most common words (changing the number of words gives similar results), and we obtain the weighted average of the $PMI$ cluster scores: $$\widehat{PMI} (C) = \sum _{c_i \in C} \frac{n_i}{N} \, \mathop {\operatorname{median}}_{\begin{array}{c}w_k, w_\ell \in S_i \\ k<\ell \end{array}} PMI(w_k,w_\ell ),$$   (Eq. 7) where $c_i$ denotes the clusters in partition $C$ , each with size $n_i$ ; $N=\sum _{c_i \in C} n_i$ is the total number of nodes; and $S_i$ denotes the set of top 10 words for cluster $c_i$ . We use this $\widehat{PMI}$ score to evaluate partitions without requiring a labelled ground truth.The $PMI$ score has been shown to perform well BIBREF13 , BIBREF14 when compared to human interpretation of topics on different corpora BIBREF39 , BIBREF40 , and is designed to evaluate topical coherence for groups of documents, in contrast to other tools aimed at short forms of text. See BIBREF18 , BIBREF19 , BIBREF41 , BIBREF42 for other examples. To compare against the external classification a posteriori, we use the normalised mutual information ( $NMI$ ), a well-used information-theoretical score that quantifies the similarity between clusterings considering both the correct and incorrect assignments in terms of the information (or predictability) between the clusterings. The NMI between two partitions $C$ and $D$ of the same graph is: $$NMI(C,D)=\frac{I(C,D)}{\sqrt{H(C)H(D)}}=\frac{\sum \limits _{c \in C} \sum \limits _{d \in D} p(c,d) \, \log \dfrac{p(c,d)}{p(c)p(d)}}{\sqrt{H(C)H(D)}}$$   (Eq. 8) where $I(C,D)$ is the Mutual Information and $H(C)$ and $H(D)$ are the entropies of the two partitions. The $NMI$ is bounded ( $0 \le NMI \le 1$ ) with a higher value corresponding to higher similarity of the partitions (i.e., $NMI=1$ when there is perfect agreement between partitions $C$ and $D$ ). The $NMI$ score is directly relatedhttp://scikit-learn.org/stable/modules/generated/sklearn.metrics.v_measure_score.html to the V-measure used in the computer science literature BIBREF43 . We use the $NMI$ to compare the partitions obtained by MS (and other methods) against the hand-coded classification assigned by the operator. Multi-resolution community detection extracts content clusters at different levels of granularity We applied MS across a broad span of Markov times ( $t \in [0.01, 100]$ in steps of 0.01) to the MST-kNN similarity graph of $N=3229$ incident records. At each Markov time, we ran 500 independent optimisations of the Louvain algorithm and selected the optimal partition at each time. Repeating the optimisation from 500 different initial starting points enhances the robustness of the outcome and allows us to quantify the robustness of the partition to the optimisation procedure. To quantify this robustness, we computed the average variation of information $VI(t)$ (a measure of dissimilarity) between the top 50 partitions for each $t$ . Once the full scan across Markov time was finalised, a final comparison of all the optimal partitions obtained was carried out, so as to assess if any of the optimised partitions was optimal at any other Markov time, in which case it was selected. We then obtained the $VI(t,t^{\prime })$ across all optimal partitions found across Markov times to ascertain when partitions are robust across levels of resolution. This layered process of optimisation enhances the robustness of the outcome given the NP-hard nature of MS optimisation, which prevents guaranteed global optimality. Figure 3 presents a summary of our analysis. We plot the number of clusters of the optimal partition and the two metrics of variation of information across all Markov times. The existence of a long plateau in $VI(t,t^{\prime })$ coupled to a dip in $VI(t)$ implies the presence of a partition that is robust both to the optimisation and across Markov time. To illustrate the multi-scale features of the method, we choose several of these robust partitions, from finer (44 communities) to coarser (3 communities), obtained at five Markov times and examine their structure and content. We also present a multi-level Sankey diagram to summarise the relationships and relative node membership across the levels. The MS analysis of the graph of incident reports reveals a rich multi-level structure of partitions, with a strong quasi-hierarchical organisation, as seen in the graph layouts and the multi-level Sankey diagram. It is important to remark that, although the Markov time acts as a natural resolution parameter from finer to coarser partitions, our process of optimisation does not impose any hierarchical structure a priori. Hence the observed consistency of communities across level is intrinsic to the data and suggests the existence of content clusters that naturally integrate with each other as sub-themes of larger thematic categories. The detection of intrinsic scales within the graph provided by MS thus enables us to obtain clusters of records with high content similarity at different levels of granularity. This capability can be used by practitioners to tune the level of description to their specific needs. Interpretation of MS communities: content and a posteriori comparison with hand-coded categories To ascertain the relevance of the different layers of content clusters found in the MS analysis, we examined in detail the five levels of resolution presented in Figure 3 . For each level, we prepared word clouds (lemmatised for increased intelligibility), as well as a Sankey diagram and a contingency table linking content clusters (i.e., graph communities) with the hand-coded categories externally assigned by an operator. We note again that this comparison was only done a posteriori, i.e., the external categories were not used in our text analysis. The results are shown in Figures 4 – 6 (and Supplementary Figures S1 – S2 ) for all levels. The partition into 44 communities presents content clusters with well-defined characterisations, as shown by the Sankey diagram and the highly clustered structure of the contingency table (Figure 4 ). The content labels for the communities were derived by us from the word clouds presented in detail in the Supplementary Information (Fig. S1 in the SI). Compared to the 15 hand-coded categories, this 44-community partition provides finer groupings of records with several clusters corresponding to sub-themes or more specific sub-classes within large, generic hand-coded categories. This is apparent in the external classes `Accidents', `Medication', `Clinical assessment', `Documentation' and `Infrastructure', where a variety of subtopics are identified corresponding to meaningful subclasses (see Fig. S1 for details). In other cases, however, the content clusters cut across the external categories, or correspond to highly specific content. Examples of the former are the content communities of records from labour ward, chemotherapy, radiotherapy and infection control, whose reports are grouped coherently based on content by our algorithm, yet belong to highly diverse external classes. At this level of resolution, our algorithm also identified highly specific topics as separate content clusters. These include blood transfusions, pressure ulcer, consent, mental health, and child protection. We have studied two levels of resolution where the number of communities (12 and 17) is close to that of hand-coded categories (15). The results of the 12-community partition are presented in Figure 5 (see Figure S2 in the SI for the slightly finer 17-community partition). As expected from the quasi-hierarchical nature of our multi-resolution analysis, we find that some of the communities in the 12-way partition emerge from consistent aggregation of smaller communities in the 44-way partition. In terms of topics, this means that some of the sub-themes observed in Figure 4 are merged into a more general topic. This is apparent in the case of Accidents: seven of the communities in the 44-way partition become one larger community (community 2 in Fig. 5 ), which has a specific and complete identification with the external category `Patient accidents'. A similar phenomenon is seen for the Nursing community (community 1) which falls completely under the external category `Infrastructure'. The clusters related to `Medication' similarly aggregate into a larger community (community 3), yet there still remains a smaller, specific community related to Homecare medication (community 12) with distinct content. Other communities strand across a few external categories. This is clearly observable in communities 10 and 11 (Samples/ lab tests/forms and Referrals/appointments), which fall naturally across the external categories `Documentation' and `Clinical Assessment'. Similarly, community 9 (Patient transfers) sits across the `Admission/Transfer' and `Infrastructure' external categories, due to its relation to nursing and other physical constraints. The rest of the communities contain a substantial proportion of records that have been hand-classified under the generic `Treatment/Procedure' class; yet here they are separated into groups that retain medical coherence, i.e., they refer to medical procedures or processes, such as Radiotherapy (Comm. 4), Blood transfusions (Comm. 7), IV/cannula (Comm. 5), Pressure ulcer (Comm. 8), and the large community Labour ward (Comm. 6). The high specificity of the Radiotherapy, Pressure ulcer and Labour ward communities means that they are still preserved as separate groups on the next level of coarseness given by the 7-way partition (Figure 6 A). The mergers in this case lead to a larger communities referring to Medication, Referrals/Forms and Staffing/Patient transfers. Figure 6 B shows the final level of agglomeration into 3 communities: a community of records referring to accidents; another community broadly referring to procedural matters (referrals, forms, staffing, medical procedures) cutting across many of the external categories; and the labour ward community still on its own as a subgroup of incidents with distinctive content. This process of agglomeration of content, from sub-themes into larger themes, as a result of the multi-scale hierarchy of graph partitions obtained with MS is shown explicitly with word clouds in Figure 8 for the 17, 12 and 7-way partitions. Robustness of the results and comparison with other methods Our framework consists of a series of steps for which there are choices and alternatives. Although it is not possible to provide comparisons to the myriad of methods and possibilities available, we have examined quantitatively the robustness of the results to parametric and methodological choices in different steps of the framework: (i) the importance of using Doc2Vec embeddings instead of BoW vectors, (ii) the size of training corpus for Doc2Vec; (iii) the sparsity of the MST-kNN similarity graph construction. We have also carried out quantitative comparisons to other methods, including: (i) LDA-BoW, and (ii) clustering with other community detection methods. We provide a brief summary here and additional material in the SI. The use of fixed-sized vector embeddings (Doc2Vec) instead of standard bag of words (BoW) is an integral part of our pipeline. Doc2Vec produces lower dimensional vector representations (as compared to BoW) with higher semantic and syntactic content. It has been reported that Doc2Vec outperforms BoW representations in practical benchmarks of semantic similarity, as well as being less sensitive to hyper-parameters BIBREF22 . To quantify the improvement provided by Doc2Vec in our framework, we constructed a MST-kNN graph following the same steps but starting with TF-iDF vectors for each document. We then ran MS on this TF-iDF similarity graph, and compared the results to those obtained from the Doc2Vec similarity graph. Figure 7 shows that the Doc2Vec version outperforms the BoW version across all resolutions in terms of both $NMI$ and $\widehat{PMI}$ scores. As shown in Table 1 , we have tested the effect of the size of the training corpus on the Doc2Vec model. We trained Doc2Vec on two additional training sets of 1 million and 2 million records (randomly chosen from the full set of $\sim $ 13 million records). We then followed the same procedure to construct the MST-kNN similarity graph and carried out the MS analysis. The results, presented in Figure S3 in the SI, show that the performance is affected only mildly by the size of the Doc2Vec training set. To examine the effect of sparsification in the graph construction, we have studied the dependence of quality of the partitions against the number of neighbours, $k$ , in the MST-kNN graph. Our numerics, shown in Figure S4 in the SI, indicate that both the $NMI$ and $\widehat{PMI}$ scores of the MS clusterings reach a similar level of quality for values of $k$ above 13-16, with minor improvement after that. Hence our results are robust to the choice of $k$ , provided it is not too small. Due to computational efficiency, we thus favour a relatively small $k$ , but not too small. We carried out a comparison with LDA, a widely used methodology for text analysis. A key difference between standard LDA and our MS method is the fact that a different LDA model needs to be trained separately for each number of topics pre-determined by the user. To offer a comparison across the methods, We obtained five LDA models corresponding to the five MS levels we considered in detail. The results in Table 2 show that MS and LDA give partitions that are comparably similar to the hand-coded categories (as measured with $NMI$ ), with some differences depending on the scale, whereas the MS clusterings have higher topic coherence (as given by $\widehat{PMI}$ ) across all scales. To give an indication of the computational cost, we ran both methods on the same servers. Our method takes approximately 13 hours in total to compute both the Doc2Vec model on 13 million records (11 hours) and the full MS scan with 400 partitions across all resolutions (2 hours). The time required to train just the 5 LDA models on the same corpus amounts to 30 hours (with timings ranging from $\sim $ 2 hours for the 3 topic LDA model to 12.5 hours for the 44 topic LDA model). This comparison also highlights the conceptual difference between our multi-scale methodology and LDA topic modelling. While LDA computes topics at a pre-determined level of resolution, our method obtains partitions at all resolutions in one sweep of the Markov time, from which relevant partitions are chosen based on their robustness. However, the MS partitions at all resolutions are available for further investigation if so needed. We have used several algorithms readily available in code libraries (i.e., the iGraph module for Python) to cluster/partition the same kNN-MST graph. Figure S5 in the SI shows the comparison against several well-known partitioning methods (Modularity Optimisation BIBREF44 , InfoMap BIBREF4 , Walktrap BIBREF45 , Label Propagation BIBREF46 , and Multi-resolution Louvain BIBREF35 ) which give just one partition (or two in the case of the Louvain implementation in iGraph) into a particular number of clusters, in contrast with our multiscale MS analysis. Our results show that MS provides improved or equal results to other graph partitioning methods for both $NMI$ and $\widehat{PMI}$ across all scales. Only for very fine resolution with more than 50 clusters, Infomap, which partitions graphs into small clique-like subgraphs BIBREF32 , BIBREF47 , provides a slightly improved $NMI$ for that particular scale. Therefore, MS allows us to find relevant, yet high quality clusterings across all scales by sweeping the Markov time parameter. Discussion This work has applied a multiscale graph partitioning algorithm (Markov Stability) to extract content-based clusters of documents from a textual dataset of healthcare safety incident reports in an unsupervised manner at different levels of resolution. The method uses paragraph vectors to represent the records and obtains an ensuing similarity graph of documents constructed from their content. The framework brings the advantage of multi-resolution algorithms capable of capturing clusters without imposing a priori their number or structure. Since different levels of resolution of the clustering can be found to be relevant, the practitioner can choose the level of description and detail to suit the requirements of a specific task. Our a posteriori analysis evaluating the similarity against the hand-coded categories and the intrinsic topic coherence of the clusters showed that the method performed well in recovering meaningful categories. The clusters of content capture topics of medical practice, thus providing complementary information to the externally imposed classification categories. Our analysis shows that some of the most relevant and persistent communities emerge because of their highly homogeneous medical content, although they are not easily mapped to the standardised external categories. This is apparent in the medically-based content clusters associated with Labour ward, Pressure ulcer, Chemotherapy, Radiotherapy, among others, which exemplify the alternative groupings that emerge from free text content. The categories in the top level (Level 1) of the pre-defined classification hierarchy are highly diverse in size (as shown by their number of assigned records), with large groups such as `Patient accident', `Medication', `Clinical assessment', `Documentation', `Admissions/Transfer' or `Infrastructure' alongside small, specific groups such as `Aggressive behaviour', `Patient abuse', `Self-harm' or `Infection control'. Our multi-scale partitioning finds corresponding groups in content across different levels of resolution, providing additional subcategories with medical detail within some of the large categories (as shown in Fig. 4 and S1 ). An area of future research will be to confirm if the categories found by our analysis are consistent with a second level in the hierarchy of external categories (Level 2, around 100 categories) that is used less consistently in hospital settings. The use of content-driven classification of reports could also be important within current efforts by the World Health Organisation (WHO) under the framework for the International Classification for Patient Safety (ICPS) BIBREF48 to establish a set of conceptual categories to monitor, analyse and interpret information to improve patient care. One of the advantages of a free text analytical approach is the provision, in a timely manner, of an intelligible description of incident report categories derived directly from the rich description in the 'words' of the reporter themselves. The insight from analysing the free text entry of the person reporting could play a valuable role and add rich information than would have otherwise been obtained from the existing approach of pre-defined classes. Not only could this improve the current state of play where much of the free text of these reports goes unused, but it avoids the fallacy of assigning incidents to a pre-defined category that, through a lack of granularity, can miss an important opportunity for feedback and learning. The nuanced information and classifications extracted from free text analysis thus suggest a complementary axis to existing approaches to characterise patient safety incident reports. Currently, local incident reporting system are used by hospitals to submit reports to the NRLS and require risk managers to improve data quality of reports, due to errors or uncertainty in categorisation from reporters, before submission. The application of free text analytical approaches, like the one we have presented here, has the potential to free up risk managers time from labour-intensive tasks of classification and correction by human operators, instead for quality improvement activities derived from the intelligence of the data itself. Additionally, the method allows for the discovery of emerging topics or classes of incidents directly from the data when such events do not fit the pre-assigned categories by using projection techniques alongside methods for anomaly and innovation detection. In ongoing work, we are currently examining the use of our characterisation of incident reports to enable comparisons across healthcare organisations and also to monitor their change over time. This part of ongoing research requires the quantification of in-class text similarities and to dynamically manage the embedding of the reports through updates and recalculation of the vector embedding. Improvements in the process of robust graph construction are also part of our future work. Detecting anomalies in the data to decide whether newer topic clusters should be created, or providing online classification suggestions to users based on the text they input are some of the improvements we aim to add in the future to aid with decision support and data collection, and to potentially help fine-tune some of the predefined categories of the external classification. Availability of Data and Materials The dataset in this work is managed by the Big Data and Analytics Unit (BDAU), Imperial College London, and consists of incident reports submitted to the NRLS. Analysis of the data was undertaken within the Secure Environment of the BDAU. Due to its nature, we cannot publicise any part of the dataset, beyond that already provided within this manuscript. No individual identifiable patient information is disclosed in this work. Only aggregated information is used to describe the clusters. Competing interests The authors declare that they have no competing interests. List of abbreviations NHS: National Health Service; NRLS: National Reporting and Learning System; BoW: Bag of Words; LDA: Latent Dirichlet Allocation; Doc2Vec: Document to Vector; MST: Minimum Spanning Tree; kNN: k-Nearest Neighbours; MS: Markov Stability; NLTK: Natural Language Toolkit; TF-iDF: Term Frequency - inverse Document Frequency; PV: Paragraph Vectors; DBOW: Distributed Bag of Words; VI: Variation of Information; NMI: Normalised Mutual Information; PMI: Pairwise Mutual Information. Authors' contributions MTA conducted the computational research. MTA and MB analysed the data and designed the computational framework. MB, EM and SNY conceived the study. All authors wrote the manuscript. Acknowledgements We thank Joshua Symons for help with accessing the data. We also thank Elias Bamis, Zijing Liu and Michael Schaub for helpful discussions. This research was supported by the National Institute for Health Research (NIHR) Imperial Patient Safety Translational Research Centre and NIHR Imperial Biomedical Research Centre. The views expressed are those of the authors and not necessarily those of the NHS, the NIHR, or the Department of Health. All authors acknowledge funding from the EPSRC through award EP/N014529/1 funding the EPSRC Centre for Mathematics of Precision Healthcare. Authors' Information MTA is a PhD student at Imperial College London, Department of Mathematics. He holds an MSc degree in finance from Sabanci University and a BSc in Electrical and Electronics Engineering from Bogazici University. EM is a Clinical Senior Lecturer in the Department of Surgery and Cancer and Centre for Health Policy at Imperial College London and Transformation Chief Clinical Information Officer (Clinical Analytics and Informatics), ICHNT. SNY is a Professor of Theoretical Chemistry in the Department of Chemistry at Imperial College London and also with the EPSRC Centre for Mathematics of Precision Healthcare. MB is Professor of Mathematics and Chair in Biomathematics in the Department of Mathematics at Imperial College London, and Director of the EPSRC Centre for Mathematics of Precision Healthcare at Imperial.
Which text embedding methodologies are used?
0689904db9b00a814e3109fb1698086370a28fa2
Which Ġtext Ġembed ding Ġmethod ologies Ġare Ġused ?
Document Ġto ĠVector Ġ( Doc 2 V ec )
<s> Which Ġtext Ġembed ding Ġmethod ologies Ġare Ġused ? </s> Introduction </s> The Ġvast Ġamounts Ġof Ġdata Ġcollected Ġby Ġhealthcare Ġproviders Ġin Ġconjunction Ġwith Ġmodern Ġdata Ġanalytics Ġtechniques Ġpresent Ġa Ġunique Ġopportunity Ġto Ġimprove Ġhealth Ġservice Ġprovision Ġand Ġthe Ġquality Ġand Ġsafety Ġof Ġmedical Ġcare Ġfor Ġpatient Ġbenefit ĠB IB REF 0 Ġ. ĠMuch Ġof Ġthe Ġrecent Ġresearch Ġin Ġthis Ġarea Ġhas Ġbeen Ġon Ġpersonal ised Ġmedicine Ġand Ġits Ġaim Ġto Ġdeliver Ġbetter Ġdiagn ostics Ġaided Ġby Ġthe Ġintegration Ġof Ġdiverse Ġdatasets Ġproviding Ġcomplementary Ġinformation . ĠAnother Ġlarge Ġsource Ġof Ġhealthcare Ġdata Ġis Ġorganis ational . ĠIn Ġthe ĠUnited ĠKingdom , Ġthe ĠNational ĠHealth ĠService Ġ( N HS ) Ġhas Ġa Ġlong Ġhistory Ġof Ġdocumenting Ġextensively Ġthe Ġdifferent Ġaspects Ġof Ġhealthcare Ġprovision . ĠThe ĠNHS Ġis Ġcurrently Ġin Ġthe Ġprocess Ġof Ġincreasing Ġthe Ġavailability Ġof Ġseveral Ġdatabases , Ġproperly Ġanonym ised , Ġwith Ġthe Ġaim Ġof Ġleveraging Ġadvanced Ġanalytics Ġto Ġidentify Ġareas Ġof Ġimprovement Ġin ĠNHS Ġservices . </s> One Ġsuch Ġdatabase Ġis Ġthe ĠNational ĠReporting Ġand ĠLearning ĠSystem Ġ( NR LS ), Ġa Ġcentral Ġrepository Ġof Ġpatient Ġsafety Ġincident Ġreports Ġfrom Ġthe ĠNHS Ġin ĠEngland Ġand ĠWales . ĠSet Ġup Ġin Ġ2003 , Ġthe ĠNR LS Ġnow Ġcontains Ġmore Ġthan Ġ13 Ġmillion Ġdetailed Ġrecords . ĠThe Ġincidents Ġare Ġreported Ġusing Ġa Ġset Ġof Ġstandard ised Ġcategories Ġand Ġcontain Ġa Ġwealth Ġof Ġorganis ational Ġand Ġspat io - tem poral Ġinformation Ġ( struct ured Ġdata ), Ġas Ġwell Ġas , Ġcru cially , Ġa Ġsubstantial Ġcomponent Ġof Ġfree Ġtext Ġ( un struct ured Ġdata ) Ġwhere Ġincidents Ġare Ġdescribed Ġin Ġthe Ġ` voice ' Ġof Ġthe Ġperson Ġreporting . ĠThe Ġincidents Ġare Ġwide Ġranging : Ġfrom Ġpatient Ġaccidents Ġto Ġlost Ġforms Ġor Ġreferrals ; Ġfrom Ġdelays Ġin Ġadmission Ġand Ġdischarge Ġto Ġserious Ġunt oward Ġincidents , Ġsuch Ġas Ġretained Ġforeign Ġobjects Ġafter Ġoperations . ĠThe Ġreview Ġand Ġanalysis Ġof Ġsuch Ġdata Ġprovides Ġcritical Ġinsight Ġinto Ġthe Ġcomplex Ġfunctioning Ġof Ġdifferent Ġprocesses Ġand Ġprocedures Ġin Ġhealthcare Ġtowards Ġservice Ġimprovement Ġfor Ġsafer Ġcare r . </s> Although Ġstatistical Ġanalyses Ġare Ġroutinely Ġperformed Ġon Ġthe Ġstructured Ġcomponent Ġof Ġthe Ġdata Ġ( dates , Ġlocations , Ġassigned Ġcategories , Ġetc ), Ġthe Ġfree Ġtext Ġremains Ġlargely Ġunused Ġin Ġsystematic Ġprocesses . ĠFree Ġtext Ġis Ġusually Ġread Ġmanually Ġbut Ġthis Ġis Ġtime - consuming , Ġmeaning Ġthat Ġit Ġis Ġoften Ġignored Ġin Ġpractice , Ġunless Ġa Ġdetailed Ġreview Ġof Ġa Ġcase Ġis Ġundertaken Ġbecause Ġof Ġthe Ġseverity Ġof Ġharm Ġthat Ġresulted . ĠThere Ġis Ġa Ġlack Ġof Ġmethod ologies Ġthat Ġcan Ġsummar ise Ġcontent Ġand Ġprovide Ġcontent - based Ġgroup ings Ġacross Ġthe Ġlarge Ġvolume Ġof Ġreports Ġsubmitted Ġnationally Ġfor Ġorganis ational Ġlearning . ĠMethods Ġthat Ġcould Ġprovide Ġautomatic Ġcategor isation Ġof Ġincidents Ġfrom Ġthe Ġfree Ġtext Ġwould Ġsid est ep Ġproblems Ġsuch Ġas Ġdifficulties Ġin Ġassigning Ġan Ġincident Ġcategory Ġby Ġvirtue Ġof Ġa Ġprior i Ġpre - defined Ġlists Ġin Ġthe Ġreporting Ġsystem Ġor Ġhuman Ġerror , Ġas Ġwell Ġas Ġoffering Ġa Ġunique Ġinsight Ġinto Ġthe Ġroot Ġcause Ġanalysis Ġof Ġincidents Ġthat Ġcould Ġimprove Ġthe Ġsafety Ġand Ġquality Ġof Ġcare Ġand Ġefficiency Ġof Ġhealthcare Ġservices . </s> Our Ġgoal Ġin Ġthis Ġwork Ġis Ġto Ġshowcase Ġan Ġalgorith mic Ġmethodology Ġthat Ġdetects Ġcontent - based Ġgroups Ġof Ġrecords Ġin Ġa Ġgiven Ġdataset Ġin Ġan Ġun super vised Ġmanner , Ġbased Ġonly Ġon Ġthe Ġfree Ġand Ġun struct ured Ġtextual Ġdescription Ġof Ġthe Ġincidents . ĠTo Ġdo Ġso , Ġwe Ġcombine Ġrecently Ġdeveloped Ġdeep Ġneural - network Ġhigh - dimensional Ġtext - embed ding Ġalgorithms Ġwith Ġnetwork - the oret ical Ġmethods . ĠIn Ġparticular , Ġwe Ġapply Ġmult isc ale ĠMark ov ĠStability Ġ( MS ) Ġcommunity Ġdetection Ġto Ġa Ġsp ars ified Ġgeometric Ġsimilarity Ġgraph Ġof Ġdocuments Ġobtained Ġfrom Ġtext Ġvector Ġsimilarities . ĠOur Ġmethod Ġdep arts Ġfrom Ġtraditional Ġnatural Ġlanguage Ġprocessing Ġtools , Ġwhich Ġhave Ġgenerally Ġused Ġbag - of - words Ġ( Bo W ) Ġrepresentation Ġof Ġdocuments Ġand Ġstatistical Ġmethods Ġbased Ġon ĠLat ent ĠDir ich let ĠAll ocation Ġ( L DA ) Ġto Ġcluster Ġdocuments ĠB IB REF 1 Ġ. ĠMore Ġrecent Ġapproaches Ġhave Ġused Ġdeep Ġneural Ġnetwork Ġbased Ġlanguage Ġmodels Ġclustered Ġwith Ġk - me ans , Ġwithout Ġa Ġfull Ġmult isc ale Ġgraph Ġanalysis ĠB IB REF 2 Ġ. ĠThere Ġhave Ġbeen Ġsome Ġprevious Ġapplications Ġof Ġnetwork Ġtheory Ġto Ġtext Ġanalysis . ĠFor Ġexample , ĠL anch ich in etti Ġand Ġco - workers ĠB IB REF 3 Ġused Ġa Ġprob abil istic Ġgraph Ġconstruction Ġanalysed Ġwith Ġthe ĠInfo Map Ġalgorithm ĠB IB REF 4 Ġ; Ġhowever , Ġtheir Ġcommunity Ġdetection Ġwas Ġcarried Ġout Ġat Ġa Ġsingle - scale Ġand Ġthe Ġrepresentation Ġof Ġtext Ġas ĠBo W Ġarrays Ġlacks Ġthe Ġpower Ġof Ġneural Ġnetwork Ġtext Ġembed d ings . ĠThe Ġapplication Ġof Ġmult isc ale Ġcommunity Ġdetection Ġallows Ġus Ġto Ġfind Ġgroups Ġof Ġrecords Ġwith Ġconsistent Ġcontent Ġat Ġdifferent Ġlevels Ġof Ġresolution ; Ġhence Ġthe Ġcontent Ġcategories Ġemerge Ġfrom Ġthe Ġtextual Ġdata , Ġrather Ġthan Ġfitting Ġwith Ġpre - designed Ġclass ifications . ĠThe Ġobtained Ġresults Ġcould Ġthus Ġhelp Ġmitigate Ġpossible Ġhuman Ġerror Ġor Ġeffort Ġin Ġfinding Ġthe Ġright Ġcategory Ġin Ġcomplex Ġcategory Ġclassification Ġtrees . </s> We Ġshowcase Ġthe Ġmethodology Ġthrough Ġthe Ġanalysis Ġof Ġa Ġdataset Ġof Ġpatient Ġincidents Ġreported Ġto Ġthe ĠNR LS . ĠFirst , Ġwe Ġuse Ġthe Ġ13 Ġmillion Ġrecords Ġcollected Ġby Ġthe ĠNR LS Ġsince Ġ2004 Ġto Ġtrain Ġour Ġtext Ġembed ding Ġ( although Ġa Ġmuch Ġsmaller Ġcorpus Ġcan Ġbe Ġused ). ĠWe Ġthen Ġanalyse Ġa Ġsubset Ġof Ġ3 229 Ġrecords Ġreported Ġfrom ĠSt ĠMary 's ĠHospital , ĠLondon Ġ( Im perial ĠCollege ĠHealthcare ĠNHS ĠTrust ) Ġover Ġthree Ġmonths Ġin Ġ2014 Ġto Ġextract Ġclusters Ġof Ġincidents Ġat Ġdifferent Ġlevels Ġof Ġresolution Ġin Ġterms Ġof Ġcontent . ĠOur Ġmethod Ġreveals Ġmultiple Ġlevels Ġof Ġintrinsic Ġstructure Ġin Ġthe Ġtopics Ġof Ġthe Ġdataset , Ġas Ġshown Ġby Ġthe Ġextraction Ġof Ġrelevant Ġword Ġdescript ors Ġfrom Ġthe Ġgrouped Ġrecords Ġand Ġa Ġhigh Ġlevel Ġof Ġtopic Ġco herence . ĠOriginally , Ġthe Ġrecords Ġhad Ġbeen Ġmanually Ġcoded Ġby Ġthe Ġoperator Ġupon Ġreporting Ġwith Ġup Ġto Ġ170 Ġfeatures Ġper Ġcase , Ġincluding Ġa Ġtwo - level Ġmanual Ġclassification Ġof Ġthe Ġincidents . ĠTherefore , Ġwe Ġalso Ġcarried Ġout Ġan Ġa Ġposterior i Ġcomparison Ġagainst Ġthe Ġhand - coded Ġcategories Ġassigned Ġby Ġthe Ġreporter Ġ( health care Ġpersonnel ) Ġat Ġthe Ġtime Ġof Ġthe Ġreport Ġsubmission . ĠOur Ġresults Ġshow Ġgood Ġoverall Ġcorrespondence Ġwith Ġthe Ġhand - coded Ġcategories Ġacross Ġresolutions Ġand , Ġspecifically , Ġat Ġthe Ġmedium Ġlevel Ġof Ġgran ularity . ĠSeveral Ġof Ġour Ġclusters Ġof Ġcontent Ġcorrespond Ġstrongly Ġto Ġwell - defined Ġcategories , Ġyet Ġour Ġresults Ġalso Ġreveal Ġcomplementary Ġcategories Ġof Ġincidents Ġnot Ġdefined Ġin Ġthe Ġexternal Ġclassification . ĠIn Ġaddition , Ġthe Ġtuning Ġof Ġthe Ġgran ularity Ġafforded Ġby Ġthe Ġmethod Ġcan Ġbe Ġused Ġto Ġprovide Ġa Ġdistinct Ġlevel Ġof Ġresolution Ġin Ġcertain Ġareas Ġcorresponding Ġto Ġspecial ise Ġor Ġparticular Ġsub - the mes . </s> Mult isc ale Ġgraph Ġpartition ing Ġfor Ġtext Ġanalysis : Ġdescription Ġof Ġthe Ġframework </s> Our Ġframework Ġcombines Ġtext - embed ding , Ġgeometric Ġgraph Ġconstruction Ġand Ġmulti - resolution Ġcommunity Ġdetection Ġto Ġidentify , Ġrather Ġthan Ġimpose , Ġcontent - based Ġclusters Ġfrom Ġfree , Ġun struct ured Ġtext Ġin Ġan Ġun super vised Ġmanner . </s> Figure Ġ1 Ġshows Ġa Ġsummary Ġof Ġour Ġpipeline . ĠFirst , Ġwe Ġpre - process Ġeach Ġdocument Ġto Ġtransform Ġtext Ġinto Ġconsecutive Ġword Ġtokens , Ġwhere Ġwords Ġare Ġin Ġtheir Ġmost Ġnormal ised Ġforms , Ġand Ġsome Ġwords Ġare Ġremoved Ġif Ġthey Ġhave Ġno Ġdistinctive Ġmeaning Ġwhen Ġused Ġout Ġof Ġcontext ĠB IB REF 5 Ġ, ĠB IB REF 6 Ġ. ĠWe Ġthen Ġtrain Ġa Ġparagraph Ġvector Ġmodel Ġusing Ġthe ĠDocument Ġto ĠVector Ġ( Doc 2 V ec ) Ġframework ĠB IB REF 7 Ġon Ġthe Ġwhole Ġset Ġ( 13 Ġmillion ) Ġof Ġpre process ed Ġtext Ġrecords , Ġalthough Ġtraining Ġon Ġsmaller Ġsets Ġ( 1 Ġmillion ) Ġalso Ġproduces Ġgood Ġresults . ĠThis Ġtraining Ġstep Ġis Ġonly Ġdone Ġonce . ĠThis ĠDoc 2 V ec Ġmodel Ġis Ġsubsequently Ġused Ġto Ġinfer Ġhigh - dimensional Ġvector Ġdescriptions Ġfor Ġthe Ġtext Ġof Ġeach Ġof Ġthe Ġ3 229 Ġdocuments Ġin Ġour Ġtarget Ġanalysis Ġset . ĠWe Ġthen Ġcompute Ġa Ġmatrix Ġcontaining Ġpair wise Ġsimilarities Ġbetween Ġany Ġpair Ġof Ġdocument Ġvectors , Ġas Ġinferred Ġwith ĠDoc 2 V ec . ĠThis Ġmatrix Ġcan Ġbe Ġthought Ġof Ġas Ġa Ġfull , Ġweighted Ġgraph Ġwith Ġdocuments Ġas Ġnodes Ġand Ġedges Ġweighted Ġby Ġtheir Ġsimilarity . ĠWe Ġsp ars ify Ġthis Ġgraph Ġto Ġthe Ġunion Ġof Ġa Ġminimum Ġspanning Ġtree Ġand Ġa Ġk - Ne arest ĠNeigh bors Ġ( M ST - k NN ) Ġgraph ĠB IB REF 8 Ġ, Ġa Ġgeometric Ġconstruction Ġthat Ġremoves Ġless Ġimportant Ġsimilarities Ġbut Ġpreserves Ġglobal Ġconnectivity Ġfor Ġthe Ġgraph Ġand , Ġhence , Ġfor Ġthe Ġdataset . ĠThe Ġderived ĠM ST - k NN Ġgraph Ġis Ġanalysed Ġwith ĠMark ov ĠStability ĠB IB REF 9 Ġ, ĠB IB REF 10 Ġ, ĠB IB REF 11 Ġ, ĠB IB REF 12 Ġ, Ġa Ġmulti - resolution Ġdynamics - based Ġgraph Ġpartition ing Ġmethod Ġthat Ġidentifies Ġrelevant Ġsub graph s Ġ( i . e ., Ġclusters Ġof Ġdocuments ) Ġat Ġdifferent Ġlevels Ġof Ġgran ularity . ĠMS Ġuses Ġa Ġdiff usive Ġprocess Ġon Ġthe Ġgraph Ġto Ġreveal Ġthe Ġmult isc ale Ġorganisation Ġat Ġdifferent Ġresolutions Ġwithout Ġthe Ġneed Ġfor Ġchoosing Ġa Ġprior i Ġthe Ġnumber Ġof Ġclusters , Ġscale Ġor Ġorganisation . ĠTo Ġanalyse Ġa Ġposterior i Ġthe Ġdifferent Ġpartitions Ġacross Ġlevels Ġof Ġresolution , Ġwe Ġuse Ġboth Ġvisual isations Ġand Ġquantitative Ġscores . ĠThe Ġvisual isations Ġinclude Ġword Ġclouds Ġto Ġsummar ise Ġthe Ġmain Ġcontent , Ġgraph Ġlayouts , Ġas Ġwell Ġas ĠSan key Ġdiagrams Ġand Ġcontingency Ġtables Ġthat Ġcapture Ġthe Ġcorrespond ences Ġacross Ġlevels Ġof Ġresolution Ġand Ġrelationships Ġto Ġthe Ġhand - coded Ġclass ifications . ĠThe Ġpartitions Ġare Ġalso Ġevaluated Ġquant itatively Ġto Ġscore : Ġ( i ) Ġtheir Ġintrinsic Ġtopic Ġco herence Ġ( using Ġpair wise Ġmutual Ġinformation ĠB IB REF 13 Ġ, ĠB IB REF 14 Ġ), Ġand Ġ( ii ) Ġtheir Ġsimilarity Ġto Ġthe Ġoperator Ġhand - coded Ġcategories Ġ( using Ġnormal ised Ġmutual Ġinformation ĠB IB REF 15 Ġ). ĠWe Ġnow Ġexpand Ġon Ġthe Ġsteps Ġof Ġthe Ġcomputational Ġframework . </s> Data Ġdescription </s> The Ġfull Ġdataset Ġincludes Ġmore Ġthan Ġ13 Ġmillion Ġconfidential Ġreports Ġof Ġpatient Ġsafety Ġincidents Ġreported Ġto Ġthe ĠNational ĠReporting Ġand ĠLearning ĠSystem Ġ( NR LS ) Ġbetween Ġ2004 Ġand Ġ2016 Ġfrom ĠNHS Ġtrusts Ġand Ġhospitals Ġin ĠEngland Ġand ĠWales . ĠEach Ġrecord Ġhas Ġmore Ġthan Ġ170 Ġfeatures , Ġincluding Ġorganis ational Ġdetails Ġ( e . g ., Ġtime , Ġtrust Ġcode Ġand Ġlocation ), Ġanonym ised Ġpatient Ġinformation , Ġmedication Ġand Ġmedical Ġdevices , Ġamong Ġother Ġdetails . ĠThe Ġrecords Ġare Ġmanually Ġclassified Ġby Ġoperators Ġto Ġa Ġtwo - level Ġsystem Ġof Ġcategories Ġof Ġincident Ġtype . ĠIn Ġparticular , Ġthe Ġtop Ġlevel Ġcontains Ġ15 Ġcategories Ġincluding Ġgeneral Ġgroups Ġsuch Ġas Ġ` Pat ient Ġaccident ', Ġ` Med ication ', Ġ` Cl inical Ġassessment ', Ġ` Document ation ', Ġ` Ad missions / Transfer ' Ġor Ġ` Inf rastructure ' Ġalongside Ġmore Ġspecific Ġgroups Ġsuch Ġas Ġ` Agg ressive Ġbehaviour ', Ġ` Pat ient Ġabuse ', Ġ` Self - harm ' Ġor Ġ` In fect ion Ġcontrol '. ĠIn Ġmost Ġrecords , Ġthere Ġis Ġalso Ġa Ġdetailed Ġdescription Ġof Ġthe Ġincident Ġin Ġfree Ġtext , Ġalthough Ġthe Ġquality Ġof Ġthe Ġtext Ġis Ġhighly Ġvariable . ĠOur Ġanalysis Ġset Ġfor Ġclust ering Ġis Ġthe Ġgroup Ġof Ġ3 229 Ġrecords Ġreported Ġduring Ġthe Ġfirst Ġquarter Ġof Ġ2014 Ġat ĠSt . ĠMary 's ĠHospital Ġin ĠLondon Ġ( Im perial ĠCollege ĠHealthcare ĠNHS ĠTrust ). </s> Text ĠPre processing </s> Text Ġpre processing Ġis Ġimportant Ġto Ġenhance Ġthe Ġperformance Ġof Ġtext Ġembed ding . ĠWe Ġapplied Ġstandard Ġpre processing Ġtechniques Ġin Ġnatural Ġlanguage Ġprocessing Ġto Ġthe Ġraw Ġtext Ġof Ġall Ġ13 Ġmillion Ġrecords Ġin Ġour Ġcorpus . ĠWe Ġnormal ise Ġwords Ġinto Ġa Ġsingle Ġform Ġand Ġremove Ġwords Ġthat Ġdo Ġnot Ġcarry Ġsignificant Ġmeaning . ĠSpecifically , Ġwe Ġdivide Ġour Ġdocuments Ġinto Ġiter ative Ġword Ġtokens Ġusing Ġthe ĠNL T K Ġlibrary ĠB IB REF 5 Ġand Ġremove Ġpunct uation Ġand Ġdigit - only Ġtokens . ĠWe Ġthen Ġapply Ġword Ġstemming Ġusing Ġthe ĠPorter Ġalgorithm ĠB IB REF 6 Ġ, ĠB IB REF 16 Ġ. ĠIf Ġthe ĠPorter Ġmethod Ġcannot Ġfind Ġa Ġstemmed Ġversion Ġfor Ġa Ġtoken , Ġwe Ġapply Ġthe ĠSnow ball Ġalgorithm ĠB IB REF 17 Ġ. ĠFinally , Ġwe Ġremove Ġany Ġstop - words Ġ( repeat Ġwords Ġwith Ġlow Ġcontent ) Ġusing ĠNL T K 's Ġstop - word Ġlist . ĠAlthough Ġsome Ġof Ġthe Ġsynt actic Ġinformation Ġis Ġreduced Ġdue Ġto Ġtext Ġpre processing , Ġthis Ġprocess Ġpreserves Ġand Ġconsolid ates Ġthe Ġsemantic Ġinformation Ġof Ġthe Ġvocabulary , Ġwhich Ġis Ġof Ġrelevance Ġto Ġour Ġstudy . </s> Text ĠEmb edd ing </s> Com put ational Ġmethods Ġfor Ġtext Ġanalysis Ġrely Ġon Ġa Ġchoice Ġof Ġa Ġmathematical Ġrepresentation Ġof Ġthe Ġbase Ġunits , Ġsuch Ġas Ġcharacter Ġ$ n $ Ġ- gram s , Ġwords Ġor Ġdocuments Ġof Ġany Ġlength . ĠAn Ġimportant Ġconsideration Ġfor Ġour Ġmethodology Ġis Ġan Ġattempt Ġto Ġavoid Ġthe Ġuse Ġof Ġlabelled Ġdata Ġat Ġthe Ġcore Ġof Ġmany Ġsupervised Ġor Ġsemi - super vised Ġclassification Ġmethods ĠB IB REF 18 Ġ, ĠB IB REF 19 Ġ. ĠIn Ġthis Ġwork , Ġwe Ġuse Ġa Ġrepresentation Ġof Ġtext Ġdocuments Ġin Ġvector Ġform Ġfollowing Ġrecent Ġdevelopments Ġin Ġthe Ġfield . </s> Class ically , Ġbag - of - words Ġ( Bow ) Ġmethods Ġwere Ġused Ġto Ġobtain Ġrepresentations Ġof Ġthe Ġdocuments Ġin Ġa Ġcorpus Ġin Ġterms Ġof Ġvectors Ġof Ġterm Ġfrequencies Ġweighted Ġby Ġinverse Ġdocument Ġfrequency Ġ( TF - i DF ). ĠWhile Ġsuch Ġmethods Ġprovide Ġa Ġstatistical Ġdescription Ġof Ġdocuments , Ġthey Ġdo Ġnot Ġcarry Ġinformation Ġabout Ġthe Ġorder Ġor Ġproximity Ġof Ġwords Ġto Ġeach Ġother Ġsince Ġthey Ġregard Ġword Ġtokens Ġin Ġan Ġindependent Ġmanner Ġwith Ġno Ġsemantic Ġor Ġsynt actic Ġrelationships Ġconsidered . ĠFurthermore , ĠBo W Ġrepresentations Ġtend Ġto Ġbe Ġhigh - dimensional Ġand Ġsparse , Ġdue Ġto Ġlarge Ġsizes Ġof Ġword Ġdiction aries Ġand Ġlow Ġfrequencies Ġof Ġmany Ġterms . </s> Recently , Ġdeep Ġneural Ġnetwork Ġlanguage Ġmodels Ġhave Ġsuccessfully Ġovercome Ġcertain Ġlimitations Ġof ĠBo W Ġmethods Ġby Ġincorporating Ġword Ġneighbourhoods Ġin Ġthe Ġmathematical Ġdescription Ġof Ġeach Ġterm . ĠDist ributed ĠBag Ġof ĠWords Ġ( DB OW ) Ġis Ġa Ġform Ġof ĠPar agraph ĠV ect ors Ġ( P V ), Ġalso Ġknown Ġas ĠDoc 2 V ec ĠB IB REF 7 Ġ. ĠThis Ġmethod Ġcreates Ġa Ġmodel Ġwhich Ġrepresents Ġany Ġlength Ġof Ġword Ġsequences Ġ( i . e . Ġsentences , Ġparagraphs , Ġdocuments ) Ġas Ġ$ d $ Ġ- dimensional Ġvectors , Ġwhere Ġ$ d $ Ġis Ġa Ġuser - defined Ġparameter Ġ( typically Ġ$ d = 500 $ Ġ). ĠTraining Ġa ĠDoc 2 V ec Ġmodel Ġstarts Ġwith Ġa Ġrandom Ġ$ d $ Ġ- dimensional Ġvector Ġassignment Ġfor Ġeach Ġdocument Ġin Ġthe Ġcorpus . ĠA Ġsto ch astic Ġgradient Ġdescent Ġalgorithm Ġiter ates Ġover Ġthe Ġcorpus Ġwith Ġthe Ġobjective Ġof Ġpredicting Ġa Ġrandomly Ġsampled Ġset Ġof Ġwords Ġfrom Ġeach Ġdocument Ġby Ġusing Ġonly Ġthe Ġdocument 's Ġ$ d $ Ġ- dimensional Ġvector ĠB IB REF 7 Ġ. ĠThe Ġobjective Ġfunction Ġbeing Ġoptim ised Ġby ĠPV - DB OW Ġis Ġsimilar Ġto Ġthe Ġskip - gram Ġmodel Ġin ĠRef s . ĠB IB REF 20 Ġ, ĠB IB REF 21 Ġ. ĠDoc 2 V ec Ġhas Ġbeen Ġshown ĠB IB REF 22 Ġto Ġcapture Ġboth Ġsemantic Ġand Ġsynt actic Ġcharacter isations Ġof Ġthe Ġinput Ġtext Ġoutper forming ĠBo W Ġmodels , Ġsuch Ġas ĠL DA ĠB IB REF 1 Ġ. </s> Here , Ġwe Ġuse Ġthe ĠG ens im ĠPython Ġlibrary ĠB IB REF 23 Ġto Ġtrain Ġthe ĠPV - DB OW Ġmodel . ĠThe ĠDoc 2 V ec Ġtraining Ġwas Ġrepeated Ġseveral Ġtimes Ġwith Ġa Ġvariety Ġof Ġtraining Ġhyper - param eters Ġto Ġoptim ise Ġthe Ġoutput Ġbased Ġon Ġour Ġown Ġnumerical Ġexperiments Ġand Ġthe Ġgeneral Ġguidelines Ġprovided Ġby ĠB IB REF 24 Ġ. ĠWe Ġtrained ĠDoc 2 V ec Ġmodels Ġusing Ġtext Ġcorpor a Ġof Ġdifferent Ġsizes Ġand Ġcontent Ġwith Ġdifferent Ġsets Ġof Ġhyper - param eters , Ġin Ġorder Ġto Ġcharacter ise Ġthe Ġusability Ġand Ġquality Ġof Ġmodels . ĠSpecifically , Ġwe Ġchecked Ġthe Ġeffect Ġof Ġcorpus Ġsize Ġon Ġmodel Ġquality Ġby Ġtraining ĠDoc 2 V ec Ġmodels Ġon Ġthe Ġfull Ġ13 Ġmillion ĠNR LS Ġrecords Ġand Ġon Ġsubs ets Ġof Ġ1 Ġmillion Ġand Ġ2 Ġmillion Ġrandomly Ġsampled Ġrecords . Ġ( We Ġnote Ġthat Ġour Ġtarget Ġsubset Ġof Ġ3 229 Ġrecords Ġhas Ġbeen Ġexcluded Ġfrom Ġthese Ġsamples .) ĠFurthermore , Ġwe Ġchecked Ġthe Ġimportance Ġof Ġthe Ġspecificity Ġof Ġthe Ġtext Ġcorpus Ġby Ġobtaining Ġa ĠDoc 2 V ec Ġmodel Ġfrom Ġa Ġgeneric , Ġnon - specific Ġset Ġof Ġ5 Ġmillion Ġarticles Ġfrom ĠWikipedia Ġrepresenting Ġstandard ĠEnglish Ġusage Ġacross Ġa Ġvariety Ġof Ġtopics . </s> We Ġbenchmark ed Ġthe ĠDoc 2 V ec Ġmodels Ġby Ġscoring Ġhow Ġwell Ġthe Ġdocument Ġvectors Ġrepresent Ġthe Ġsemantic Ġtopic Ġstructure : Ġ( i ) Ġcalculating Ġcent ro ids Ġfor Ġthe Ġ15 Ġexternally Ġhand - coded Ġcategories ; Ġ( ii ) Ġselecting Ġthe Ġ100 Ġnearest Ġreports Ġfor Ġeach Ġcent roid ; Ġ( iii ) Ġcounting Ġthe Ġnumber Ġof Ġincident Ġreports Ġ( out Ġof Ġ1500 ) Ġcorrectly Ġassigned Ġto Ġtheir Ġcent roid . ĠThe Ġresults Ġin ĠTable Ġ1 Ġshow Ġthat Ġtraining Ġon Ġthe Ġhighly Ġspecific Ġtext Ġin Ġthe ĠNR LS Ġrecords Ġis Ġan Ġimportant Ġingredient Ġin Ġthe Ġsuccessful Ġvector isation Ġof Ġthe Ġdocuments , Ġas Ġshown Ġby Ġthe Ġdegraded Ġperformance Ġfor Ġthe ĠWikipedia Ġmodel Ġacross Ġa Ġvariety Ġof Ġtraining Ġhyper - param eters . ĠOur Ġresults Ġalso Ġshow Ġthat Ġreducing Ġthe Ġsize Ġof Ġthe Ġcorpus Ġfrom Ġ13 Ġmillion Ġto Ġ1 Ġmillion Ġrecords Ġdid Ġnot Ġaffect Ġthe Ġbenchmark ing Ġdramatically . ĠThis Ġrobust ness Ġof Ġthe Ġresults Ġto Ġthe Ġsize Ġof Ġthe Ġtraining Ġcorpus Ġwas Ġconfirmed Ġfurther Ġwith Ġthe Ġuse Ġof Ġmore Ġdetailed Ġmetrics , Ġas Ġdiscussed Ġbelow Ġin ĠSection Ġsec : com par isons . </s> Based Ġon Ġour Ġbenchmark ing , Ġwe Ġuse Ġhence forth Ġ( unless Ġotherwise Ġnoted ) Ġthe Ġoptim ised ĠDoc 2 V ec Ġmodel Ġobtained Ġfrom Ġthe Ġ13 + Ġmillion ĠNR LS Ġrecords Ġwith Ġthe Ġfollowing Ġhyper - param eters : Ġ{ training Ġmethod Ġ= Ġd bow , Ġnumber Ġof Ġdimensions Ġfor Ġfeature Ġvectors Ġsize Ġ= Ġ300 , Ġnumber Ġof Ġepoch s Ġ= Ġ10 , Ġwindow Ġsize Ġ= Ġ15 , Ġminimum Ġcount Ġ= Ġ5 , Ġnumber Ġof Ġnegative Ġsamples Ġ= Ġ5 , Ġrandom Ġdown - sam pling Ġthreshold Ġfor Ġfrequent Ġwords Ġ= Ġ0 . 001 Ġ} . ĠAs Ġan Ġindication Ġof Ġcomputational Ġcost , Ġthe Ġtraining Ġof Ġthe Ġmodel Ġon Ġthe Ġ13 Ġmillion Ġrecords Ġtakes Ġapproximately Ġ11 Ġhours Ġ( run Ġin Ġparallel Ġwith Ġ7 Ġthreads ) Ġon Ġshared Ġservers . </s> Graph ĠConstruction </s> Once Ġthe ĠDoc 2 V ec Ġmodel Ġis Ġtrained , Ġwe Ġuse Ġit Ġto Ġinfer Ġa Ġvector Ġfor Ġeach Ġof Ġthe Ġ$ N = 3 229 $ Ġrecords Ġin Ġour Ġanalysis Ġset . ĠWe Ġthen Ġconstruct Ġa Ġnormal ised Ġcos ine Ġsimilarity Ġmatrix Ġbetween Ġthe Ġvectors Ġby : Ġcomputing Ġthe Ġmatrix Ġof Ġcos ine Ġsimilarities Ġbetween Ġall Ġpairs Ġof Ġrecords , Ġ$ S _ \ text { cos } $ Ġ; Ġtransforming Ġit Ġinto Ġa Ġdistance Ġmatrix Ġ$ D _{ cos } Ġ= Ġ1 - S _{ cos } $ Ġ; Ġapplying Ġelement - wise Ġmax Ġnorm Ġto Ġobtain Ġ$\ hat { D } = \ Vert ĠD _{ cos }\ Vert Ġ_ { max } $ Ġ; Ġand Ġnormal ising Ġthe Ġsimilarity Ġmatrix Ġ$\ hat { S } Ġ= Ġ1 - \ hat { D } $ Ġwhich Ġhas Ġelements Ġin Ġthe Ġinterval Ġ$ [ 0 , 1 ] $ Ġ. </s> The Ġsimilarity Ġmatrix Ġcan Ġbe Ġthought Ġof Ġas Ġthe Ġadj ac ency Ġmatrix Ġof Ġa Ġfully Ġconnected Ġweighted Ġgraph . ĠHowever , Ġsuch Ġa Ġgraph Ġcontains Ġmany Ġedges Ġwith Ġsmall Ġweights Ġreflecting Ġweak Ġsimilarities Ġin Ġhigh - dimensional Ġnoisy Ġdatasets Ġeven Ġthe Ġleast Ġsimilar Ġnodes Ġpresent Ġa Ġsubstantial Ġdegree Ġof Ġsimilarity . ĠSuch Ġweak Ġsimilarities Ġare Ġin Ġmost Ġcases Ġredundant , Ġas Ġthey Ġcan Ġbe Ġexplained Ġthrough Ġstronger Ġpair wise Ġsimilarities Ġpresent Ġin Ġthe Ġgraph . ĠThese Ġweak , Ġredundant Ġedges Ġobscure Ġthe Ġgraph Ġstructure , Ġas Ġshown Ġby Ġthe Ġdiffuse , Ġspherical Ġvisual isation Ġof Ġthe Ġfull Ġgraph Ġlayout Ġin ĠFigure Ġ2 ĠA . </s> To Ġreveal Ġthe Ġgraph Ġstructure , Ġwe Ġobtain Ġa ĠM ST - k NN Ġgraph Ġfrom Ġthe Ġnormal ised Ġsimilarity Ġmatrix ĠB IB REF 8 Ġ. ĠThis Ġis Ġa Ġsimple Ġsp ars ification Ġbased Ġon Ġa Ġgeometric Ġhe uristic Ġthat Ġpreserves Ġthe Ġglobal Ġconnectivity Ġof Ġthe Ġgraph Ġwhile Ġretaining Ġdetails Ġabout Ġthe Ġlocal Ġgeometry Ġof Ġthe Ġdataset . ĠThe ĠM ST - k NN Ġalgorithm Ġstarts Ġby Ġcomputing Ġthe Ġminimum Ġspanning Ġtree Ġ( M ST ) Ġof Ġthe Ġfull Ġmatrix Ġ$\ hat { D } $ Ġ, Ġi . e ., Ġthe Ġtree Ġwith Ġ$( N - 1 ) $ Ġedges Ġconnecting Ġall Ġnodes Ġin Ġthe Ġgraph Ġwith Ġminimal Ġsum Ġof Ġedge Ġweights Ġ( dist ances ). ĠThe ĠM ST Ġis Ġcomputed Ġusing Ġthe ĠKr usk al Ġalgorithm Ġimplemented Ġin ĠSci Py ĠB IB REF 25 Ġ. ĠTo Ġthis ĠM ST , Ġwe Ġadd Ġedges Ġconnecting Ġeach Ġnode Ġto Ġits Ġ$ k $ Ġnearest Ġnodes Ġ( k NN ) Ġif Ġthey Ġare Ġnot Ġalready Ġin Ġthe ĠM ST . ĠHere Ġ$ k $ Ġis Ġan Ġuser - defined Ġparameter . ĠThe Ġbinary Ġadj ac ency Ġmatrix Ġof Ġthe ĠM ST - k NN Ġgraphs , Ġ$ E _ \ text { M ST - k NN } $ Ġ, Ġis ĠHad am ard - multipl ied Ġwith Ġ$\ hat { S } $ Ġto Ġgive Ġthe Ġadj ac ency Ġmatrix Ġ$ A $ Ġof Ġthe Ġweighted , Ġund irect ed Ġsp ars ified Ġgraph . ĠThe ĠM ST - k NN Ġmethod Ġavoids Ġa Ġdirect Ġthreshold ing Ġof Ġthe Ġweights Ġin Ġ$\ hat { S } $ Ġ, Ġand Ġob tains Ġa Ġgraph Ġdescription Ġthat Ġpreserves Ġlocal Ġgeometric Ġinformation Ġtogether Ġwith Ġa Ġglobal Ġsub graph Ġ( the ĠM ST ) Ġthat Ġcaptures Ġproperties Ġof Ġthe Ġfull Ġdataset . </s> The Ġnetwork Ġlayout Ġvisual isations Ġin ĠFigure Ġ2 ĠB âĢĵ E Ġgive Ġan Ġintuitive Ġpicture Ġof Ġthe Ġeffect Ġof Ġthe Ġsp ars ification . ĠThe Ġhighly Ġsparse Ġgraphs Ġobtained Ġwhen Ġthe Ġnumber Ġof Ġneighbours Ġ$ k $ Ġis Ġvery Ġsmall Ġare Ġnot Ġrobust . ĠAs Ġ$ k $ Ġis Ġincreased , Ġthe Ġlocal Ġsimilarities Ġbetween Ġdocuments Ġinduce Ġthe Ġformation Ġof Ġdense Ġsub graph s Ġ( which Ġappear Ġcloser Ġin Ġthe Ġgraph Ġvisual isation Ġlayout ). ĠWhen Ġthe Ġnumber Ġof Ġneighbours Ġbecomes Ġtoo Ġlarge , Ġthe Ġlocal Ġstructure Ġbecomes Ġdiffuse Ġand Ġthe Ġsub graph s Ġlose Ġco herence , Ġsignalling Ġthe Ġdegradation Ġof Ġthe Ġlocal Ġgraph Ġstructure . ĠFigure Ġ2 Ġshows Ġthat Ġthe ĠM ST - k NN Ġgraph Ġwith Ġ$ k = 13 $ Ġpresents Ġa Ġreasonable Ġbalance Ġbetween Ġlocal Ġand Ġglobal Ġstructure . ĠRel atively Ġsparse Ġgraphs Ġthat Ġpreserve Ġimportant Ġedges Ġand Ġglobal Ġconnectivity Ġof Ġthe Ġdataset Ġ( gu arant eed Ġhere Ġby Ġthe ĠM ST ) Ġhave Ġcomputational Ġadvantages Ġwhen Ġusing Ġcommunity Ġdetection Ġalgorithms . </s> The ĠM ST - k NN Ġconstruction Ġhas Ġbeen Ġreported Ġto Ġbe Ġrobust Ġto Ġthe Ġselection Ġof Ġthe Ġparameter Ġ$ k $ Ġdue Ġto Ġthe Ġguaranteed Ġconnectivity Ġprovided Ġby Ġthe ĠM ST ĠB IB REF 8 Ġ. ĠIn Ġthe Ġfollowing , Ġwe Ġfix Ġ$ k = 13 $ Ġfor Ġour Ġanalysis Ġwith Ġthe Ġmulti - scale Ġgraph Ġpartition ing Ġframework , Ġbut Ġwe Ġhave Ġscanned Ġvalues Ġof Ġ$ k Ġ\ in Ġ[ 1 , 50 ] $ Ġin Ġthe Ġgraph Ġconstruction Ġfrom Ġour Ġdata Ġand Ġhave Ġfound Ġthat Ġthe Ġconstruction Ġis Ġrobust Ġas Ġlong Ġas Ġ$ k $ Ġis Ġnor Ġtoo Ġsmall Ġ( i . e ., Ġ$ k Ġ> Ġ13 $ Ġ). ĠThe Ġdetailed Ġcomparisons Ġare Ġshown Ġin ĠSection Ġsec : com par isons . </s> The ĠM ST - k NN Ġconstruction Ġhas Ġthe Ġadvantage Ġof Ġits Ġsimplicity Ġand Ġrobust ness , Ġand Ġthe Ġfact Ġthat Ġit Ġbalances Ġthe Ġlocal Ġand Ġglobal Ġstructure Ġof Ġthe Ġdata . ĠHowever , Ġthe Ġarea Ġof Ġnetwork Ġinference Ġand Ġgraph Ġconstruction Ġfrom Ġdata , Ġand Ġgraph Ġsp ars ification Ġis Ġvery Ġactive , Ġand Ġseveral Ġalternative Ġapproaches Ġexist Ġbased Ġon Ġdifferent Ġhe ur istics , Ġe . g ., ĠGraph ical ĠL asso ĠB IB REF 26 Ġ, ĠPlan ar ĠMaxim ally ĠFil tered ĠGraph ĠB IB REF 27 Ġ, Ġspectral Ġsp ars ification ĠB IB REF 28 Ġ, Ġor Ġthe ĠRelax ed ĠMinimum ĠSpan ning ĠTree Ġ( RM ST ) ĠB IB REF 29 Ġ. ĠWe Ġhave Ġexperimented Ġwith Ġsome Ġof Ġthose Ġmethods Ġand Ġobtained Ġcomparable Ġresults . ĠA Ġdetailed Ġcomparison Ġof Ġsp ars ification Ġmethods Ġas Ġwell Ġas Ġthe Ġchoice Ġof Ġdistance Ġin Ġdefining Ġthe Ġsimilarity Ġmatrix Ġ$\ hat { S } $ Ġis Ġleft Ġfor Ġfuture Ġwork . </s> Mult isc ale ĠGraph ĠPart ition ing </s> The Ġarea Ġof Ġcommunity Ġdetection Ġencompasses Ġa Ġvariety Ġof Ġgraph Ġpartition ing Ġapproaches Ġwhich Ġaim Ġto Ġfind Ġ` good ' Ġpartitions Ġinto Ġsub graph s Ġ( or Ġcommunities ) Ġaccording Ġto Ġdifferent Ġcost Ġfunctions , Ġwithout Ġimposing Ġthe Ġnumber Ġof Ġcommunities Ġa Ġprior i ĠB IB REF 31 Ġ. ĠThe Ġnotion Ġof Ġcommunity Ġthus Ġdepends Ġon Ġthe Ġchoice Ġof Ġcost Ġfunction . ĠCommon ly , Ġcommunities Ġare Ġtaken Ġto Ġbe Ġsub graph s Ġwhose Ġnodes Ġare Ġconnected Ġstrongly Ġwithin Ġthe Ġcommunity Ġwith Ġrelatively Ġweak Ġinter - community Ġedges . ĠSuch Ġstructural Ġnotion Ġis Ġrelated Ġto Ġbalanced Ġcuts . ĠOther Ġcost Ġfunctions Ġare Ġposed Ġin Ġterms Ġof Ġtransitions Ġinside Ġand Ġoutside Ġof Ġthe Ġcommunities , Ġusually Ġas Ġone - step Ġprocesses ĠB IB REF 4 Ġ. ĠWhen Ġtransition Ġpaths Ġof Ġrandom Ġwalks Ġof Ġall Ġlengths Ġare Ġconsidered , Ġthe Ġconcept Ġof Ġcommunity Ġbecomes Ġintrinsically Ġmulti - scale , Ġi . e ., Ġdifferent Ġpartitions Ġcan Ġbe Ġfound Ġto Ġbe Ġrelevant Ġat Ġdifferent Ġtime Ġscales Ġleading Ġto Ġa Ġmulti - level Ġdescription Ġdictated Ġby Ġthe Ġtransition Ġdynamics ĠB IB REF 9 Ġ, ĠB IB REF 32 Ġ, ĠB IB REF 10 Ġ. ĠThis Ġleads Ġto Ġthe Ġframework Ġof ĠMark ov ĠStability , Ġa Ġdynamics - based , Ġmulti - scale Ġcommunity Ġdetection Ġmethodology , Ġwhich Ġcan Ġbe Ġshown Ġto Ġrecover Ġseamlessly Ġseveral Ġwell - known Ġhe ur istics Ġas Ġparticular Ġcases ĠB IB REF 9 Ġ, ĠB IB REF 11 Ġ, ĠB IB REF 12 Ġ. </s> Here , Ġwe Ġapply ĠMS Ġto Ġfind Ġpartitions Ġof Ġthe Ġsimilarity Ġgraph Ġ$ A $ Ġat Ġdifferent Ġlevels Ġof Ġresolution . ĠThe Ġsub graph s Ġdetected Ġcorrespond Ġto Ġclusters Ġof Ġdocuments Ġwith Ġsimilar Ġcontent . ĠMS Ġis Ġan Ġun super vised Ġcommunity Ġdetection Ġmethod Ġthat Ġfinds Ġrobust Ġand Ġstable Ġpartitions Ġunder Ġthe Ġevolution Ġof Ġa Ġcontinuous - time Ġdiffusion Ġprocess Ġwithout Ġa Ġprior i Ġchoice Ġof Ġthe Ġnumber Ġor Ġtype Ġof Ġcommunities Ġor Ġtheir Ġorganisation ĠB IB REF 9 Ġ, ĠB IB REF 32 Ġ, ĠB IB REF 10 Ġ, ĠB IB REF 33 ĠThe Ġcode Ġfor ĠMark ov ĠStability Ġis Ġopen Ġand Ġaccessible Ġat Ġhttps :// github . com / m ich a els cha ub / Part ition St ability Ġand Ġhttp :// www f . im perial . ac . uk / ~ mp bara / Part ition _ St ability /, Ġlast Ġaccessed Ġon ĠMarch Ġ24 , Ġ2018 . ĠIn Ġsimple Ġterms , ĠMS Ġcan Ġbe Ġunderstood Ġby Ġanalogy Ġto Ġa Ġdrop Ġof Ġink Ġdiff using Ġon Ġthe Ġgraph Ġunder Ġa Ġdiff usive ĠMark ov Ġprocess . ĠThe Ġink Ġdiff uses Ġhom ogene ously Ġunless Ġthe Ġgraph Ġhas Ġsome Ġintrinsic Ġstructural Ġorganisation , Ġin Ġwhich Ġcase Ġthe Ġink Ġgets Ġtransient ly Ġcontained , Ġover Ġparticular Ġtime Ġscales , Ġwithin Ġgroups Ġof Ġnodes Ġ( i . e ., Ġsub graph s Ġor Ġcommunities ). ĠThe Ġexistence Ġof Ġthis Ġtransient Ġcontainment Ġsignals Ġthe Ġpresence Ġof Ġa Ġnatural Ġpartition Ġof Ġthe Ġgraph . ĠAs Ġthe Ġprocess Ġevolves , Ġthe Ġink Ġdiff uses Ġout Ġof Ġthose Ġinitial Ġcommunities Ġbut Ġmight Ġget Ġtransient ly Ġcontained Ġin Ġother , Ġlarger Ġsub graph s . ĠBy Ġanalys ing Ġthis ĠMark ov Ġdynamics Ġover Ġtime , ĠMS Ġdetects Ġthe Ġstructure Ġof Ġthe Ġgraph Ġacross Ġscales . ĠThe ĠMark ov Ġtime Ġ$ t $ Ġthus Ġacts Ġas Ġa Ġresolution Ġparameter Ġthat Ġallows Ġus Ġto Ġextract Ġrobust Ġpartitions Ġthat Ġpersist Ġover Ġparticular Ġtime Ġscales , Ġin Ġan Ġun super vised Ġmanner . </s> Given Ġthe Ġadj ac ency Ġmatrix Ġ$ A _{ N Ġ\ times ĠN } $ Ġof Ġthe Ġgraph Ġobtained Ġas Ġdescribed Ġpreviously , Ġlet Ġus Ġdefine Ġthe Ġdiagonal Ġmatrix Ġ$ D = \ text { di ag } (\ math bf Ġ{ d }) $ Ġ, Ġwhere Ġ$\ math bf Ġ{ d } = A Ġ\ math bf Ġ{ 1 } $ Ġis Ġthe Ġdegree Ġvector . ĠThe Ġrandom Ġwalk ĠLa pl ac ian Ġmatrix Ġis Ġdefined Ġas Ġ$ L _ \ text { RW } = I _ N - D ^{ - 1 } A $ Ġwhere Ġ$ I _ N $ Ġis Ġthe Ġidentity Ġmatrix Ġof Ġsize Ġ$ N $ Ġ, Ġand Ġthe Ġtransition Ġmatrix Ġ( or Ġkernel ) Ġof Ġthe Ġassociated Ġcontinuous - time ĠMark ov Ġprocess Ġis Ġ$ P ( t )= e ^{ - t ĠL _ \ text { RW } }, Ġ\ , Ġt > 0 $ ĠB IB REF 10 Ġ. ĠFor Ġeach Ġpartition , Ġa Ġbinary Ġmembership Ġmatrix Ġ$ H _{ N Ġ\ times ĠC } $ Ġmaps Ġthe Ġ$ N $ Ġnodes Ġinto Ġ$ C $ Ġclusters . ĠWe Ġcan Ġthen Ġdefine Ġthe Ġ$ D = \ text { di ag } (\ math bf Ġ{ d }) $ 0 Ġclustered Ġaut oc ov ari ance Ġmatrix : </s> $$ R ( t , H ) Ġ= ĠH ^ T [ \ Pi ĠP ( t )- \ pi Ġ\ pi Ġ^ T ] H Ġ$$ Ġ Ġ Ġ( E q . Ġ4 ) </s> where Ġ$\ pi Ġ$ Ġis Ġthe Ġsteady - state Ġdistribution Ġof Ġthe Ġprocess Ġand Ġ$\ Pi Ġ= \ text { di ag } (\ pi Ġ) $ Ġ. ĠThe Ġelement Ġ$ [ R ( t , H )] _{ \ alpha Ġ\ beta Ġ} $ Ġquant ifies Ġthe Ġprobability Ġthat Ġa Ġrandom Ġwalk er Ġstarting Ġfrom Ġcommunity Ġ$\ alpha Ġ$ Ġwill Ġend Ġin Ġcommunity Ġ$\ beta Ġ$ Ġat Ġtime Ġ$ t $ Ġ, Ġsubtract ing Ġthe Ġprobability Ġthat Ġthe Ġsame Ġevent Ġoccurs Ġby Ġchance Ġat Ġstation arity . </s> We Ġthen Ġdefine Ġour Ġcost Ġfunction Ġmeasuring Ġthe Ġgoodness Ġof Ġa Ġpartition Ġover Ġtime Ġ$ t $ Ġ, Ġtermed Ġthe ĠMark ov ĠStability Ġof Ġpartition Ġ$ H $ Ġ: </s> $$ r ( t , H ) Ġ= Ġ\ text { trace } Ġ\ left [ R ( t , H )\ right ]. Ġ$$ Ġ Ġ Ġ( E q . Ġ5 ) </s> A Ġpartition Ġ$ H $ Ġthat Ġmaxim ises Ġ$ r ( t , H ) $ Ġis Ġcomprised Ġof Ġcommunities Ġthat Ġpreserve Ġthe Ġflow Ġwithin Ġthemselves Ġover Ġtime Ġ$ t $ Ġ, Ġsince Ġin Ġthat Ġcase Ġthe Ġdiagonal Ġelements Ġof Ġ$ R ( t , H ) $ Ġwill Ġbe Ġlarge Ġand Ġthe Ġoff - di agonal Ġelements Ġwill Ġbe Ġsmall . ĠFor Ġdetails , Ġsee ĠB IB REF 9 Ġ, ĠB IB REF 32 Ġ, ĠB IB REF 10 Ġ, ĠB IB REF 34 Ġ. </s> MS Ġsearches Ġfor Ġpartitions Ġat Ġeach ĠMark ov Ġtime Ġthat Ġmaxim ise Ġ$ r ( t , H ) $ Ġ. ĠAlthough Ġthe Ġmaxim isation Ġof Ġ( Ġ5 Ġ) Ġis Ġan ĠNP - hard Ġproblem Ġ( hen ce Ġwith Ġno Ġguarantees Ġfor Ġglobal Ġoptim ality ), Ġthere Ġare Ġefficient Ġoptim isation Ġmethods Ġthat Ġwork Ġwell Ġin Ġpractice . ĠOur Ġimplementation Ġhere Ġuses Ġthe ĠLou v ain ĠAl gorithm ĠB IB REF 35 Ġ, ĠB IB REF 12 Ġwhich Ġis Ġefficient Ġand Ġknown Ġto Ġgive Ġgood Ġresults Ġwhen Ġapplied Ġto Ġbenchmarks ĠB IB REF 36 Ġ. ĠTo Ġobtain Ġrobust Ġpartitions , Ġwe Ġrun Ġthe ĠLou v ain Ġalgorithm Ġ500 Ġtimes Ġwith Ġdifferent Ġinitial isations Ġat Ġeach ĠMark ov Ġtime Ġand Ġpick Ġthe Ġbest Ġ50 Ġwith Ġthe Ġhighest ĠMark ov ĠStability Ġvalue Ġ$ r ( t , H ) $ Ġ. ĠWe Ġthen Ġcompute Ġthe Ġvariation Ġof Ġinformation ĠB IB REF 37 Ġof Ġthis Ġensemble Ġof Ġsolutions Ġ$ VI ( t ) $ Ġ. Ġas Ġa Ġmeasure Ġof Ġthe Ġreprodu c ibility Ġof Ġthe Ġresult Ġunder Ġthe Ġoptim isation . ĠIn Ġaddition , Ġthe Ġrelevant Ġpartitions Ġare Ġrequired Ġto Ġbe Ġpersistent Ġacross Ġtime , Ġas Ġgiven Ġby Ġlow Ġvalues Ġof Ġthe Ġvariation Ġof Ġinformation Ġbetween Ġoptim ised Ġpartitions Ġacross Ġtime Ġ$ VI ( t , t ^ {\ prime Ġ}) $ Ġ. ĠRob ust Ġpartitions Ġare Ġthus Ġindicated Ġby ĠMark ov Ġtimes Ġwhere Ġ$ VI ( t ) $ Ġshows Ġa Ġdip Ġand Ġ$ VI ( t , t ^ {\ prime Ġ}) $ Ġhas Ġan Ġextended Ġplateau , Ġindicating Ġconsistent Ġresults Ġfrom Ġdifferent ĠLou v ain Ġruns Ġand Ġvalidity Ġover Ġextended Ġscales ĠB IB REF 34 Ġ, ĠB IB REF 10 Ġ. </s> Visual isation Ġand Ġinterpretation Ġof Ġthe Ġresults </s> We Ġuse Ġthe ĠForce At las 2 ĠB IB REF 30 Ġlayout Ġto Ġrepresent Ġthe Ġgraph Ġof Ġ3 229 ĠNR LS ĠPatient ĠIncident Ġreports . ĠThis Ġlayout Ġfollows Ġa Ġforce - directed Ġiter ative Ġmethod Ġto Ġfind Ġnode Ġpositions Ġthat Ġbalance Ġattractive Ġand Ġrep ulsive Ġforces . ĠHence Ġsimilar Ġnodes Ġtend Ġto Ġbe Ġgrouped Ġtogether Ġon Ġthe Ġplan ar Ġlayout . ĠWe Ġcolour Ġthe Ġnodes Ġby Ġeither Ġhand - coded Ġcategories Ġ( Figure Ġ2 Ġ) Ġor Ġmult isc ale ĠMS Ġcommunities Ġ( Figure Ġ3 Ġ). ĠSp at ially Ġconsistent Ġcolour ings Ġon Ġthis Ġlayout Ġimply Ġgood Ġclusters Ġof Ġdocuments Ġin Ġterms Ġof Ġthe Ġsimilarity Ġgraph . </s> San key Ġdiagrams Ġallow Ġus Ġto Ġvisual ise Ġthe Ġrelationship Ġof Ġnode Ġmembers hips Ġacross Ġdifferent Ġpartitions Ġand Ġwith Ġrespect Ġto Ġthe Ġhand - coded Ġcategories . ĠIn Ġparticular , Ġtwo - layer ĠSan key Ġdiagrams Ġ( e . g ., ĠFig . Ġ4 Ġ) Ġreflect Ġthe Ġcorrespondence Ġbetween ĠMS Ġclusters Ġand Ġthe Ġhand - coded Ġexternal Ġcategories , Ġwhereas Ġthe Ġmult il ayer ĠSan key Ġdiagram Ġin ĠFig . Ġ3 Ġrepresents Ġthe Ġresults Ġof Ġthe Ġmulti - resolution ĠMS Ġcommunity Ġdetection Ġacross Ġscales . </s> In Ġaddition Ġto ĠSan key Ġdiagrams Ġbetween Ġour ĠMS Ġclusters Ġand Ġthe Ġhand - coded Ġcategories , Ġwe Ġalso Ġprovide Ġa Ġcomplementary Ġvisual isation Ġas Ġheat maps Ġof Ġnormal ised Ġcontingency Ġ( z - score ) Ġtables , Ġe . g ., ĠFig . Ġ4 Ġ. ĠThis Ġallows Ġus Ġto Ġcompare Ġthe Ġrelative Ġassociation Ġof Ġcontent Ġclusters Ġto Ġthe Ġexternal Ġcategories Ġat Ġdifferent Ġresolution Ġlevels . ĠA Ġquant ification Ġof Ġthis Ġcorrespondence Ġis Ġalso Ġprovided Ġby Ġthe Ġ$ N MI $ Ġscore Ġintroduced Ġin ĠE q . Ġ( Ġ8 Ġ). </s> Our Ġmethod Ġclusters Ġtext Ġdocuments Ġaccording Ġto Ġtheir Ġintrinsic Ġcontent . ĠThis Ġcan Ġbe Ġunderstood Ġas Ġa Ġtype Ġof Ġtopic Ġdetection . ĠTo Ġunderstand Ġthe Ġcontent Ġof Ġthe Ġclusters , Ġwe Ġuse ĠWord ĠClouds Ġas Ġbasic , Ġyet Ġintuitive , Ġtools Ġthat Ġsummar ise Ġinformation Ġfrom Ġa Ġgroup Ġof Ġdocuments . ĠWord Ġclouds Ġallow Ġus Ġto Ġevaluate Ġthe Ġresults Ġand Ġextract Ġinsights Ġwhen Ġcomparing Ġa Ġposterior i Ġwith Ġhand - coded Ġcategories . ĠThey Ġcan Ġalso Ġprovide Ġan Ġaid Ġfor Ġmonitoring Ġresults Ġwhen Ġused Ġby Ġpractitioners . </s> The Ġstemming Ġmethods Ġdescribed Ġin Ġthe ĠText ĠPre processing Ġsubsection Ġtrunc ate Ġwords Ġseverely . ĠSuch Ġtrunc ation Ġenhances Ġthe Ġpower Ġof Ġthe Ġlanguage Ġprocessing Ġcomputational Ġmethods , Ġas Ġit Ġreduces Ġthe Ġredundancy Ġin Ġthe Ġword Ġcorpus . ĠYet Ġwhen Ġpresenting Ġthe Ġresults Ġback Ġto Ġa Ġhuman Ġobserver , Ġit Ġis Ġdesirable Ġto Ġreport Ġthe Ġcontent Ġof Ġthe Ġclusters Ġwith Ġwords Ġthat Ġare Ġreadily Ġcomprehens ible . ĠTo Ġgenerate Ġcomprehens ible Ġword Ġclouds Ġin Ġour Ġa Ġposterior i Ġanalyses , Ġwe Ġuse Ġa Ġtext Ġprocessing Ġmethod Ġsimilar Ġto Ġthe Ġone Ġdescribed Ġin ĠB IB REF 38 Ġ. ĠSpecifically , Ġwe Ġuse Ġthe Ġpart Ġof Ġspeech Ġ( POS ) Ġtagging Ġmodule Ġfrom ĠNL T K Ġto Ġleave Ġout Ġsentence Ġparts Ġexcept Ġthe Ġadject ives , Ġnoun s , Ġand Ġverbs . ĠWe Ġalso Ġremove Ġless Ġmeaningful Ġcommon Ġverbs Ġsuch Ġas Ġ` be ', Ġ` have ', Ġand Ġ` do ' Ġand Ġtheir Ġvariations . ĠThe Ġresidual Ġwords Ġare Ġthen Ġle mm at ised Ġand Ġrepresented Ġwith Ġtheir Ġle mm as Ġin Ġorder Ġto Ġnormal ise Ġvariations Ġof Ġthe Ġsame Ġword . ĠOnce Ġthe Ġtext Ġis Ġprocessed Ġin Ġthis Ġmanner , Ġwe Ġuse Ġthe ĠPython Ġlibrary Ġword cloud The Ġword Ġcloud Ġgenerator Ġlibrary Ġfor ĠPython Ġis Ġopen Ġand Ġaccessible Ġat Ġhttps :// github . com / am ueller / word _ cloud , Ġlast Ġaccessed Ġon ĠMarch Ġ25 , Ġ2018 Ġto Ġcreate Ġword Ġclouds Ġwith Ġ2 Ġor Ġ3 - gram Ġfrequency Ġlist Ġof Ġcommon Ġword Ġgroups . ĠThe Ġresults Ġpresent Ġdistinct , Ġunderstandable Ġword Ġtopics . </s> Quant itative Ġbenchmark ing Ġof Ġtopic Ġclusters </s> Although Ġour Ġdataset Ġhas Ġattached Ġa Ġhand - coded Ġclassification Ġby Ġa Ġhuman Ġoperator , Ġwe Ġdo Ġnot Ġuse Ġit Ġin Ġour Ġanalysis Ġand Ġwe Ġdo Ġnot Ġconsider Ġit Ġas Ġa Ġ` ground Ġtruth '. ĠIndeed , Ġone Ġof Ġour Ġaims Ġis Ġto Ġexplore Ġthe Ġrelevance Ġof Ġthe Ġfixed Ġexternal Ġclasses Ġas Ġcompared Ġto Ġthe Ġcontent - driven Ġgroup ings Ġobtained Ġin Ġan Ġun super vised Ġmanner . ĠHence Ġwe Ġprovide Ġa Ġdouble Ġroute Ġto Ġquantify Ġthe Ġquality Ġof Ġthe Ġclusters Ġby Ġcomputing Ġtwo Ġcomplementary Ġmeasures : Ġan Ġintrinsic Ġmeasure Ġof Ġtopic Ġco herence Ġand Ġa Ġmeasure Ġof Ġsimilarity Ġto Ġthe Ġexternal Ġhand - coded Ġcategories , Ġdefined Ġas Ġfollows . </s> As Ġan Ġintrinsic Ġmeasure Ġof Ġconsistency Ġof Ġword Ġassociation Ġwithout Ġany Ġreference Ġto Ġan Ġexternal Ġ` ground Ġtruth ', Ġwe Ġuse Ġthe Ġpoint wise Ġmutual Ġinformation Ġ( Ġ$ PM I $ Ġ) ĠB IB REF 13 Ġ, ĠB IB REF 14 Ġ. ĠThe Ġ$ PM I $ Ġis Ġan Ġinformation - the oret ical Ġscore Ġthat Ġcaptures Ġthe Ġprobability Ġof Ġbeing Ġused Ġtogether Ġin Ġthe Ġsame Ġgroup Ġof Ġdocuments . ĠThe Ġ$ PM I $ Ġscore Ġfor Ġa Ġpair Ġof Ġwords Ġ$( w _ 1 , w _ 2 ) $ Ġis : </s> $$ PM I ( w _ 1 , w _ 2 )= \ log Ġ{\ frac { P ( w _ 1 Ġw _ 2 ) }{ P ( w _ 1 ) P ( w _ 2 ) }} $$ Ġ Ġ Ġ( E q . Ġ6 ) </s> where Ġthe Ġprobabilities Ġof Ġthe Ġwords Ġ$ P ( w _ 1 ) $ Ġ, Ġ$ P ( w _ 2 ) $ Ġ, Ġand Ġof Ġtheir Ġco - occ urrence Ġ$ P ( w _ 1 Ġw _ 2 ) $ Ġare Ġobtained Ġfrom Ġthe Ġcorpus . ĠTo Ġobtain Ġthe Ġaggregate Ġ$\ wide hat { PM I } $ Ġfor Ġthe Ġgraph Ġpartition Ġ$ C = \ l brace Ġc _ i \ r brace Ġ$ Ġwe Ġcompute Ġthe Ġ$ PM I $ Ġfor Ġeach Ġcluster , Ġas Ġthe Ġmedian Ġ$ PM I $ Ġbetween Ġits Ġ10 Ġmost Ġcommon Ġwords Ġ( changing Ġthe Ġnumber Ġof Ġwords Ġgives Ġsimilar Ġresults ), Ġand Ġwe Ġobtain Ġthe Ġweighted Ġaverage Ġof Ġthe Ġ$ PM I $ Ġcluster Ġscores : </s> $$ \ wide hat { PM I } Ġ( C ) Ġ= Ġ\ sum Ġ_ { c _ i Ġ\ in ĠC } Ġ\ frac { n _ i }{ N } Ġ\ , Ġ\ math op Ġ{\ oper at orn ame { med ian }} _{ \ begin { array }{ c } w _ k , Ġw _ \ ell Ġ\ in ĠS _ i Ġ\\ Ġk < \ ell Ġ\ end { array }} ĠPM I ( w _ k , w _ \ ell Ġ), $$ Ġ Ġ Ġ( E q . Ġ7 ) </s> where Ġ$ c _ i $ Ġdenotes Ġthe Ġclusters Ġin Ġpartition Ġ$ C $ Ġ, Ġeach Ġwith Ġsize Ġ$ n _ i $ Ġ; Ġ$ N = \ sum Ġ_ { c _ i Ġ\ in ĠC } Ġn _ i $ Ġis Ġthe Ġtotal Ġnumber Ġof Ġnodes ; Ġand Ġ$ S _ i $ Ġdenotes Ġthe Ġset Ġof Ġtop Ġ10 Ġwords Ġfor Ġcluster Ġ$ c _ i $ Ġ. </s> We Ġuse Ġthis Ġ$\ wide hat { PM I } $ Ġscore Ġto Ġevaluate Ġpartitions Ġwithout Ġrequiring Ġa Ġlabelled Ġground Ġtruth . The Ġ$ PM I $ Ġscore Ġhas Ġbeen Ġshown Ġto Ġperform Ġwell ĠB IB REF 13 Ġ, ĠB IB REF 14 Ġwhen Ġcompared Ġto Ġhuman Ġinterpretation Ġof Ġtopics Ġon Ġdifferent Ġcorpor a ĠB IB REF 39 Ġ, ĠB IB REF 40 Ġ, Ġand Ġis Ġdesigned Ġto Ġevaluate Ġtopical Ġco herence Ġfor Ġgroups Ġof Ġdocuments , Ġin Ġcontrast Ġto Ġother Ġtools Ġaimed Ġat Ġshort Ġforms Ġof Ġtext . ĠSee ĠB IB REF 18 Ġ, ĠB IB REF 19 Ġ, ĠB IB REF 41 Ġ, ĠB IB REF 42 Ġfor Ġother Ġexamples . </s> To Ġcompare Ġagainst Ġthe Ġexternal Ġclassification Ġa Ġposterior i , Ġwe Ġuse Ġthe Ġnormal ised Ġmutual Ġinformation Ġ( Ġ$ N MI $ Ġ), Ġa Ġwell - used Ġinformation - the oret ical Ġscore Ġthat Ġquant ifies Ġthe Ġsimilarity Ġbetween Ġcluster ings Ġconsidering Ġboth Ġthe Ġcorrect Ġand Ġincorrect Ġassignments Ġin Ġterms Ġof Ġthe Ġinformation Ġ( or Ġpredict ability ) Ġbetween Ġthe Ġcluster ings . ĠThe ĠN MI Ġbetween Ġtwo Ġpartitions Ġ$ C $ Ġand Ġ$ D $ Ġof Ġthe Ġsame Ġgraph Ġis : </s> $$ N MI ( C , D )= \ frac { I ( C , D ) }{ \ sq rt { H ( C ) H ( D ) }} = \ frac {\ sum Ġ\ limits Ġ_ { c Ġ\ in ĠC } Ġ\ sum Ġ\ limits Ġ_ { d Ġ\ in ĠD } Ġp ( c , d ) Ġ\ , Ġ\ log Ġ\ df rac { p ( c , d ) }{ p ( c ) p ( d ) }} {\ sq rt { H ( C ) H ( D ) }} $$ Ġ Ġ Ġ( E q . Ġ8 ) </s> where Ġ$ I ( C , D ) $ Ġis Ġthe ĠMutual ĠInformation Ġand Ġ$ H ( C ) $ Ġand Ġ$ H ( D ) $ Ġare Ġthe Ġent rop ies Ġof Ġthe Ġtwo Ġpartitions . </s> The Ġ$ N MI $ Ġis Ġbounded Ġ( Ġ$ 0 Ġ\ le ĠN MI Ġ\ le Ġ1 $ Ġ) Ġwith Ġa Ġhigher Ġvalue Ġcorresponding Ġto Ġhigher Ġsimilarity Ġof Ġthe Ġpartitions Ġ( i . e ., Ġ$ N MI = 1 $ Ġwhen Ġthere Ġis Ġperfect Ġagreement Ġbetween Ġpartitions Ġ$ C $ Ġand Ġ$ D $ Ġ). ĠThe Ġ$ N MI $ Ġscore Ġis Ġdirectly Ġrelated http :// sci kit - learn . org / stable / modules / generated / sk learn . met rics . v _ me asure _ score . html Ġto Ġthe ĠV - me asure Ġused Ġin Ġthe Ġcomputer Ġscience Ġliterature ĠB IB REF 43 Ġ. ĠWe Ġuse Ġthe Ġ$ N MI $ Ġto Ġcompare Ġthe Ġpartitions Ġobtained Ġby ĠMS Ġ( and Ġother Ġmethods ) Ġagainst Ġthe Ġhand - coded Ġclassification Ġassigned Ġby Ġthe Ġoperator . </s> Multi - resolution Ġcommunity Ġdetection Ġextracts Ġcontent Ġclusters Ġat Ġdifferent Ġlevels Ġof Ġgran ularity </s> We Ġapplied ĠMS Ġacross Ġa Ġbroad Ġspan Ġof ĠMark ov Ġtimes Ġ( Ġ$ t Ġ\ in Ġ[ 0 . 01 , Ġ100 ] $ Ġin Ġsteps Ġof Ġ0 . 01 ) Ġto Ġthe ĠM ST - k NN Ġsimilarity Ġgraph Ġof Ġ$ N = 3 229 $ Ġincident Ġrecords . ĠAt Ġeach ĠMark ov Ġtime , Ġwe Ġran Ġ500 Ġindependent Ġoptim isations Ġof Ġthe ĠLou v ain Ġalgorithm Ġand Ġselected Ġthe Ġoptimal Ġpartition Ġat Ġeach Ġtime . ĠRepe ating Ġthe Ġoptim isation Ġfrom Ġ500 Ġdifferent Ġinitial Ġstarting Ġpoints Ġenhances Ġthe Ġrobust ness Ġof Ġthe Ġoutcome Ġand Ġallows Ġus Ġto Ġquantify Ġthe Ġrobust ness Ġof Ġthe Ġpartition Ġto Ġthe Ġoptim isation Ġprocedure . ĠTo Ġquantify Ġthis Ġrobust ness , Ġwe Ġcomputed Ġthe Ġaverage Ġvariation Ġof Ġinformation Ġ$ VI ( t ) $ Ġ( a Ġmeasure Ġof Ġdiss imilar ity ) Ġbetween Ġthe Ġtop Ġ50 Ġpartitions Ġfor Ġeach Ġ$ t $ Ġ. ĠOnce Ġthe Ġfull Ġscan Ġacross ĠMark ov Ġtime Ġwas Ġfinal ised , Ġa Ġfinal Ġcomparison Ġof Ġall Ġthe Ġoptimal Ġpartitions Ġobtained Ġwas Ġcarried Ġout , Ġso Ġas Ġto Ġassess Ġif Ġany Ġof Ġthe Ġoptim ised Ġpartitions Ġwas Ġoptimal Ġat Ġany Ġother ĠMark ov Ġtime , Ġin Ġwhich Ġcase Ġit Ġwas Ġselected . ĠWe Ġthen Ġobtained Ġthe Ġ$ VI ( t , t ^ {\ prime Ġ}) $ Ġacross Ġall Ġoptimal Ġpartitions Ġfound Ġacross ĠMark ov Ġtimes Ġto Ġascertain Ġwhen Ġpartitions Ġare Ġrobust Ġacross Ġlevels Ġof Ġresolution . ĠThis Ġlayered Ġprocess Ġof Ġoptim isation Ġenhances Ġthe Ġrobust ness Ġof Ġthe Ġoutcome Ġgiven Ġthe ĠNP - hard Ġnature Ġof ĠMS Ġoptim isation , Ġwhich Ġprevents Ġguaranteed Ġglobal Ġoptim ality . </s> Figure Ġ3 Ġpresents Ġa Ġsummary Ġof Ġour Ġanalysis . ĠWe Ġplot Ġthe Ġnumber Ġof Ġclusters Ġof Ġthe Ġoptimal Ġpartition Ġand Ġthe Ġtwo Ġmetrics Ġof Ġvariation Ġof Ġinformation Ġacross Ġall ĠMark ov Ġtimes . ĠThe Ġexistence Ġof Ġa Ġlong Ġplateau Ġin Ġ$ VI ( t , t ^ {\ prime Ġ}) $ Ġcoupled Ġto Ġa Ġdip Ġin Ġ$ VI ( t ) $ Ġimplies Ġthe Ġpresence Ġof Ġa Ġpartition Ġthat Ġis Ġrobust Ġboth Ġto Ġthe Ġoptim isation Ġand Ġacross ĠMark ov Ġtime . ĠTo Ġillustrate Ġthe Ġmulti - scale Ġfeatures Ġof Ġthe Ġmethod , Ġwe Ġchoose Ġseveral Ġof Ġthese Ġrobust Ġpartitions , Ġfrom Ġfiner Ġ( 44 Ġcommunities ) Ġto Ġco arser Ġ( 3 Ġcommunities ), Ġobtained Ġat Ġfive ĠMark ov Ġtimes Ġand Ġexamine Ġtheir Ġstructure Ġand Ġcontent . ĠWe Ġalso Ġpresent Ġa Ġmulti - level ĠSan key Ġdiagram Ġto Ġsummar ise Ġthe Ġrelationships Ġand Ġrelative Ġnode Ġmembership Ġacross Ġthe Ġlevels . </s> The ĠMS Ġanalysis Ġof Ġthe Ġgraph Ġof Ġincident Ġreports Ġreveals Ġa Ġrich Ġmulti - level Ġstructure Ġof Ġpartitions , Ġwith Ġa Ġstrong Ġquasi - h ier arch ical Ġorganisation , Ġas Ġseen Ġin Ġthe Ġgraph Ġlayouts Ġand Ġthe Ġmulti - level ĠSan key Ġdiagram . ĠIt Ġis Ġimportant Ġto Ġremark Ġthat , Ġalthough Ġthe ĠMark ov Ġtime Ġacts Ġas Ġa Ġnatural Ġresolution Ġparameter Ġfrom Ġfiner Ġto Ġco arser Ġpartitions , Ġour Ġprocess Ġof Ġoptim isation Ġdoes Ġnot Ġimpose Ġany Ġhierarchical Ġstructure Ġa Ġprior i . ĠHence Ġthe Ġobserved Ġconsistency Ġof Ġcommunities Ġacross Ġlevel Ġis Ġintrinsic Ġto Ġthe Ġdata Ġand Ġsuggests Ġthe Ġexistence Ġof Ġcontent Ġclusters Ġthat Ġnaturally Ġintegrate Ġwith Ġeach Ġother Ġas Ġsub - the mes Ġof Ġlarger Ġthem atic Ġcategories . ĠThe Ġdetection Ġof Ġintrinsic Ġscales Ġwithin Ġthe Ġgraph Ġprovided Ġby ĠMS Ġthus Ġenables Ġus Ġto Ġobtain Ġclusters Ġof Ġrecords Ġwith Ġhigh Ġcontent Ġsimilarity Ġat Ġdifferent Ġlevels Ġof Ġgran ularity . ĠThis Ġcapability Ġcan Ġbe Ġused Ġby Ġpractitioners Ġto Ġtune Ġthe Ġlevel Ġof Ġdescription Ġto Ġtheir Ġspecific Ġneeds . </s> Inter pret ation Ġof ĠMS Ġcommunities : Ġcontent Ġand Ġa Ġposterior i Ġcomparison Ġwith Ġhand - coded Ġcategories </s> To Ġascertain Ġthe Ġrelevance Ġof Ġthe Ġdifferent Ġlayers Ġof Ġcontent Ġclusters Ġfound Ġin Ġthe ĠMS Ġanalysis , Ġwe Ġexamined Ġin Ġdetail Ġthe Ġfive Ġlevels Ġof Ġresolution Ġpresented Ġin ĠFigure Ġ3 Ġ. ĠFor Ġeach Ġlevel , Ġwe Ġprepared Ġword Ġclouds Ġ( le mm at ised Ġfor Ġincreased Ġintellig ibility ), Ġas Ġwell Ġas Ġa ĠSan key Ġdiagram Ġand Ġa Ġcontingency Ġtable Ġlinking Ġcontent Ġclusters Ġ( i . e ., Ġgraph Ġcommunities ) Ġwith Ġthe Ġhand - coded Ġcategories Ġexternally Ġassigned Ġby Ġan Ġoperator . ĠWe Ġnote Ġagain Ġthat Ġthis Ġcomparison Ġwas Ġonly Ġdone Ġa Ġposterior i , Ġi . e ., Ġthe Ġexternal Ġcategories Ġwere Ġnot Ġused Ġin Ġour Ġtext Ġanalysis . ĠThe Ġresults Ġare Ġshown Ġin ĠFigures Ġ4 ĠâĢĵ Ġ6 Ġ( and ĠSupplementary ĠFigures ĠS 1 ĠâĢĵ ĠS 2 Ġ) Ġfor Ġall Ġlevels . </s> The Ġpartition Ġinto Ġ44 Ġcommunities Ġpresents Ġcontent Ġclusters Ġwith Ġwell - defined Ġcharacter isations , Ġas Ġshown Ġby Ġthe ĠSan key Ġdiagram Ġand Ġthe Ġhighly Ġclustered Ġstructure Ġof Ġthe Ġcontingency Ġtable Ġ( Figure Ġ4 Ġ). ĠThe Ġcontent Ġlabels Ġfor Ġthe Ġcommunities Ġwere Ġderived Ġby Ġus Ġfrom Ġthe Ġword Ġclouds Ġpresented Ġin Ġdetail Ġin Ġthe ĠSupplementary ĠInformation Ġ( Fig . ĠS 1 Ġin Ġthe ĠSI ). ĠCompared Ġto Ġthe Ġ15 Ġhand - coded Ġcategories , Ġthis Ġ44 - community Ġpartition Ġprovides Ġfiner Ġgroup ings Ġof Ġrecords Ġwith Ġseveral Ġclusters Ġcorresponding Ġto Ġsub - the mes Ġor Ġmore Ġspecific Ġsub - classes Ġwithin Ġlarge , Ġgeneric Ġhand - coded Ġcategories . ĠThis Ġis Ġapparent Ġin Ġthe Ġexternal Ġclasses Ġ` Acc idents ', Ġ` Med ication ', Ġ` Cl inical Ġassessment ', Ġ` Document ation ' Ġand Ġ` Inf rastructure ', Ġwhere Ġa Ġvariety Ġof Ġsub top ics Ġare Ġidentified Ġcorresponding Ġto Ġmeaningful Ġsub classes Ġ( see ĠFig . ĠS 1 Ġfor Ġdetails ). ĠIn Ġother Ġcases , Ġhowever , Ġthe Ġcontent Ġclusters Ġcut Ġacross Ġthe Ġexternal Ġcategories , Ġor Ġcorrespond Ġto Ġhighly Ġspecific Ġcontent . ĠExamples Ġof Ġthe Ġformer Ġare Ġthe Ġcontent Ġcommunities Ġof Ġrecords Ġfrom Ġlabour Ġward , Ġchemotherapy , Ġradi otherapy Ġand Ġinfection Ġcontrol , Ġwhose Ġreports Ġare Ġgrouped Ġcoherent ly Ġbased Ġon Ġcontent Ġby Ġour Ġalgorithm , Ġyet Ġbelong Ġto Ġhighly Ġdiverse Ġexternal Ġclasses . ĠAt Ġthis Ġlevel Ġof Ġresolution , Ġour Ġalgorithm Ġalso Ġidentified Ġhighly Ġspecific Ġtopics Ġas Ġseparate Ġcontent Ġclusters . ĠThese Ġinclude Ġblood Ġtransf usions , Ġpressure Ġul cer , Ġconsent , Ġmental Ġhealth , Ġand Ġchild Ġprotection . </s> We Ġhave Ġstudied Ġtwo Ġlevels Ġof Ġresolution Ġwhere Ġthe Ġnumber Ġof Ġcommunities Ġ( 12 Ġand Ġ17 ) Ġis Ġclose Ġto Ġthat Ġof Ġhand - coded Ġcategories Ġ( 15 ). ĠThe Ġresults Ġof Ġthe Ġ12 - community Ġpartition Ġare Ġpresented Ġin ĠFigure Ġ5 Ġ( see ĠFigure ĠS 2 Ġin Ġthe ĠSI Ġfor Ġthe Ġslightly Ġfiner Ġ17 - community Ġpartition ). ĠAs Ġexpected Ġfrom Ġthe Ġquasi - h ier arch ical Ġnature Ġof Ġour Ġmulti - resolution Ġanalysis , Ġwe Ġfind Ġthat Ġsome Ġof Ġthe Ġcommunities Ġin Ġthe Ġ12 - way Ġpartition Ġemerge Ġfrom Ġconsistent Ġaggregation Ġof Ġsmaller Ġcommunities Ġin Ġthe Ġ44 - way Ġpartition . ĠIn Ġterms Ġof Ġtopics , Ġthis Ġmeans Ġthat Ġsome Ġof Ġthe Ġsub - the mes Ġobserved Ġin ĠFigure Ġ4 Ġare Ġmerged Ġinto Ġa Ġmore Ġgeneral Ġtopic . ĠThis Ġis Ġapparent Ġin Ġthe Ġcase Ġof ĠAcc idents : Ġseven Ġof Ġthe Ġcommunities Ġin Ġthe Ġ44 - way Ġpartition Ġbecome Ġone Ġlarger Ġcommunity Ġ( community Ġ2 Ġin ĠFig . Ġ5 Ġ), Ġwhich Ġhas Ġa Ġspecific Ġand Ġcomplete Ġidentification Ġwith Ġthe Ġexternal Ġcategory Ġ` Pat ient Ġaccidents '. ĠA Ġsimilar Ġphenomenon Ġis Ġseen Ġfor Ġthe ĠNursing Ġcommunity Ġ( community Ġ1 ) Ġwhich Ġfalls Ġcompletely Ġunder Ġthe Ġexternal Ġcategory Ġ` Inf rastructure '. ĠThe Ġclusters Ġrelated Ġto Ġ` Med ication ' Ġsimilarly Ġaggregate Ġinto Ġa Ġlarger Ġcommunity Ġ( community Ġ3 ), Ġyet Ġthere Ġstill Ġremains Ġa Ġsmaller , Ġspecific Ġcommunity Ġrelated Ġto ĠHome care Ġmedication Ġ( community Ġ12 ) Ġwith Ġdistinct Ġcontent . </s> Other Ġcommunities Ġstrand Ġacross Ġa Ġfew Ġexternal Ġcategories . ĠThis Ġis Ġclearly Ġobservable Ġin Ġcommunities Ġ10 Ġand Ġ11 Ġ( S amples / Ġlab Ġtests / forms Ġand ĠRefer rals / app oint ments ), Ġwhich Ġfall Ġnaturally Ġacross Ġthe Ġexternal Ġcategories Ġ` Document ation ' Ġand Ġ` Cl inical ĠAssessment '. ĠSimilarly , Ġcommunity Ġ9 Ġ( Pat ient Ġtransfers ) Ġsits Ġacross Ġthe Ġ` Ad mission / Transfer ' Ġand Ġ` Inf rastructure ' Ġexternal Ġcategories , Ġdue Ġto Ġits Ġrelation Ġto Ġnursing Ġand Ġother Ġphysical Ġconstraints . ĠThe Ġrest Ġof Ġthe Ġcommunities Ġcontain Ġa Ġsubstantial Ġproportion Ġof Ġrecords Ġthat Ġhave Ġbeen Ġhand - classified Ġunder Ġthe Ġgeneric Ġ` T reatment / Pro ced ure ' Ġclass ; Ġyet Ġhere Ġthey Ġare Ġseparated Ġinto Ġgroups Ġthat Ġretain Ġmedical Ġco herence , Ġi . e ., Ġthey Ġrefer Ġto Ġmedical Ġprocedures Ġor Ġprocesses , Ġsuch Ġas ĠRad i otherapy Ġ( Comm . Ġ4 ), ĠBlood Ġtransf usions Ġ( Comm . Ġ7 ), ĠIV / c ann ula Ġ( Comm . Ġ5 ), ĠPressure Ġul cer Ġ( Comm . Ġ8 ), Ġand Ġthe Ġlarge Ġcommunity ĠLabour Ġward Ġ( Comm . Ġ6 ). </s> The Ġhigh Ġspecificity Ġof Ġthe ĠRad i otherapy , ĠPressure Ġul cer Ġand ĠLabour Ġward Ġcommunities Ġmeans Ġthat Ġthey Ġare Ġstill Ġpreserved Ġas Ġseparate Ġgroups Ġon Ġthe Ġnext Ġlevel Ġof Ġco ars eness Ġgiven Ġby Ġthe Ġ7 - way Ġpartition Ġ( Figure Ġ6 ĠA ). ĠThe Ġmer gers Ġin Ġthis Ġcase Ġlead Ġto Ġa Ġlarger Ġcommunities Ġreferring Ġto ĠMed ication , ĠRefer rals / Form s Ġand ĠStaff ing / Pat ient Ġtransfers . ĠFigure Ġ6 ĠB Ġshows Ġthe Ġfinal Ġlevel Ġof Ġagg l omer ation Ġinto Ġ3 Ġcommunities : Ġa Ġcommunity Ġof Ġrecords Ġreferring Ġto Ġaccidents ; Ġanother Ġcommunity Ġbroadly Ġreferring Ġto Ġprocedural Ġmatters Ġ( re fer rals , Ġforms , Ġstaffing , Ġmedical Ġprocedures ) Ġcutting Ġacross Ġmany Ġof Ġthe Ġexternal Ġcategories ; Ġand Ġthe Ġlabour Ġward Ġcommunity Ġstill Ġon Ġits Ġown Ġas Ġa Ġsub group Ġof Ġincidents Ġwith Ġdistinctive Ġcontent . </s> This Ġprocess Ġof Ġagg l omer ation Ġof Ġcontent , Ġfrom Ġsub - the mes Ġinto Ġlarger Ġthemes , Ġas Ġa Ġresult Ġof Ġthe Ġmulti - scale Ġhierarchy Ġof Ġgraph Ġpartitions Ġobtained Ġwith ĠMS Ġis Ġshown Ġexplicitly Ġwith Ġword Ġclouds Ġin ĠFigure Ġ8 Ġfor Ġthe Ġ17 , Ġ12 Ġand Ġ7 - way Ġpartitions . </s> Rob ust ness Ġof Ġthe Ġresults Ġand Ġcomparison Ġwith Ġother Ġmethods </s> Our Ġframework Ġconsists Ġof Ġa Ġseries Ġof Ġsteps Ġfor Ġwhich Ġthere Ġare Ġchoices Ġand Ġalternatives . ĠAlthough Ġit Ġis Ġnot Ġpossible Ġto Ġprovide Ġcomparisons Ġto Ġthe Ġmyriad Ġof Ġmethods Ġand Ġpossibilities Ġavailable , Ġwe Ġhave Ġexamined Ġquant itatively Ġthe Ġrobust ness Ġof Ġthe Ġresults Ġto Ġparam etric Ġand Ġmethodological Ġchoices Ġin Ġdifferent Ġsteps Ġof Ġthe Ġframework : Ġ( i ) Ġthe Ġimportance Ġof Ġusing ĠDoc 2 V ec Ġembed d ings Ġinstead Ġof ĠBo W Ġvectors , Ġ( ii ) Ġthe Ġsize Ġof Ġtraining Ġcorpus Ġfor ĠDoc 2 V ec ; Ġ( iii ) Ġthe Ġsp arsity Ġof Ġthe ĠM ST - k NN Ġsimilarity Ġgraph Ġconstruction . ĠWe Ġhave Ġalso Ġcarried Ġout Ġquantitative Ġcomparisons Ġto Ġother Ġmethods , Ġincluding : Ġ( i ) ĠL DA - Bo W , Ġand Ġ( ii ) Ġclust ering Ġwith Ġother Ġcommunity Ġdetection Ġmethods . ĠWe Ġprovide Ġa Ġbrief Ġsummary Ġhere Ġand Ġadditional Ġmaterial Ġin Ġthe ĠSI . </s> The Ġuse Ġof Ġfixed - sized Ġvector Ġembed d ings Ġ( Doc 2 V ec ) Ġinstead Ġof Ġstandard Ġbag Ġof Ġwords Ġ( Bo W ) Ġis Ġan Ġintegral Ġpart Ġof Ġour Ġpipeline . ĠDoc 2 V ec Ġproduces Ġlower Ġdimensional Ġvector Ġrepresentations Ġ( as Ġcompared Ġto ĠBo W ) Ġwith Ġhigher Ġsemantic Ġand Ġsynt actic Ġcontent . ĠIt Ġhas Ġbeen Ġreported Ġthat ĠDoc 2 V ec Ġoutper forms ĠBo W Ġrepresentations Ġin Ġpractical Ġbenchmarks Ġof Ġsemantic Ġsimilarity , Ġas Ġwell Ġas Ġbeing Ġless Ġsensitive Ġto Ġhyper - param eters ĠB IB REF 22 Ġ. </s> To Ġquantify Ġthe Ġimprovement Ġprovided Ġby ĠDoc 2 V ec Ġin Ġour Ġframework , Ġwe Ġconstructed Ġa ĠM ST - k NN Ġgraph Ġfollowing Ġthe Ġsame Ġsteps Ġbut Ġstarting Ġwith ĠTF - i DF Ġvectors Ġfor Ġeach Ġdocument . ĠWe Ġthen Ġran ĠMS Ġon Ġthis ĠTF - i DF Ġsimilarity Ġgraph , Ġand Ġcompared Ġthe Ġresults Ġto Ġthose Ġobtained Ġfrom Ġthe ĠDoc 2 V ec Ġsimilarity Ġgraph . ĠFigure Ġ7 Ġshows Ġthat Ġthe ĠDoc 2 V ec Ġversion Ġoutper forms Ġthe ĠBo W Ġversion Ġacross Ġall Ġresolutions Ġin Ġterms Ġof Ġboth Ġ$ N MI $ Ġand Ġ$\ wide hat { PM I } $ Ġscores . </s> As Ġshown Ġin ĠTable Ġ1 Ġ, Ġwe Ġhave Ġtested Ġthe Ġeffect Ġof Ġthe Ġsize Ġof Ġthe Ġtraining Ġcorpus Ġon Ġthe ĠDoc 2 V ec Ġmodel . ĠWe Ġtrained ĠDoc 2 V ec Ġon Ġtwo Ġadditional Ġtraining Ġsets Ġof Ġ1 Ġmillion Ġand Ġ2 Ġmillion Ġrecords Ġ( random ly Ġchosen Ġfrom Ġthe Ġfull Ġset Ġof Ġ$\ sim Ġ$ Ġ13 Ġmillion Ġrecords ). ĠWe Ġthen Ġfollowed Ġthe Ġsame Ġprocedure Ġto Ġconstruct Ġthe ĠM ST - k NN Ġsimilarity Ġgraph Ġand Ġcarried Ġout Ġthe ĠMS Ġanalysis . ĠThe Ġresults , Ġpresented Ġin ĠFigure ĠS 3 Ġin Ġthe ĠSI , Ġshow Ġthat Ġthe Ġperformance Ġis Ġaffected Ġonly Ġmildly Ġby Ġthe Ġsize Ġof Ġthe ĠDoc 2 V ec Ġtraining Ġset . </s> To Ġexamine Ġthe Ġeffect Ġof Ġsp ars ification Ġin Ġthe Ġgraph Ġconstruction , Ġwe Ġhave Ġstudied Ġthe Ġdependence Ġof Ġquality Ġof Ġthe Ġpartitions Ġagainst Ġthe Ġnumber Ġof Ġneighbours , Ġ$ k $ Ġ, Ġin Ġthe ĠM ST - k NN Ġgraph . ĠOur Ġnumer ics , Ġshown Ġin ĠFigure ĠS 4 Ġin Ġthe ĠSI , Ġindicate Ġthat Ġboth Ġthe Ġ$ N MI $ Ġand Ġ$\ wide hat { PM I } $ Ġscores Ġof Ġthe ĠMS Ġcluster ings Ġreach Ġa Ġsimilar Ġlevel Ġof Ġquality Ġfor Ġvalues Ġof Ġ$ k $ Ġabove Ġ13 - 16 , Ġwith Ġminor Ġimprovement Ġafter Ġthat . ĠHence Ġour Ġresults Ġare Ġrobust Ġto Ġthe Ġchoice Ġof Ġ$ k $ Ġ, Ġprovided Ġit Ġis Ġnot Ġtoo Ġsmall . ĠDue Ġto Ġcomputational Ġefficiency , Ġwe Ġthus Ġfavour Ġa Ġrelatively Ġsmall Ġ$ k $ Ġ, Ġbut Ġnot Ġtoo Ġsmall . </s> We Ġcarried Ġout Ġa Ġcomparison Ġwith ĠL DA , Ġa Ġwidely Ġused Ġmethodology Ġfor Ġtext Ġanalysis . ĠA Ġkey Ġdifference Ġbetween Ġstandard ĠL DA Ġand Ġour ĠMS Ġmethod Ġis Ġthe Ġfact Ġthat Ġa Ġdifferent ĠL DA Ġmodel Ġneeds Ġto Ġbe Ġtrained Ġseparately Ġfor Ġeach Ġnumber Ġof Ġtopics Ġpre - d etermined Ġby Ġthe Ġuser . ĠTo Ġoffer Ġa Ġcomparison Ġacross Ġthe Ġmethods , ĠWe Ġobtained Ġfive ĠL DA Ġmodels Ġcorresponding Ġto Ġthe Ġfive ĠMS Ġlevels Ġwe Ġconsidered Ġin Ġdetail . ĠThe Ġresults Ġin ĠTable Ġ2 Ġshow Ġthat ĠMS Ġand ĠL DA Ġgive Ġpartitions Ġthat Ġare Ġcompar ably Ġsimilar Ġto Ġthe Ġhand - coded Ġcategories Ġ( as Ġmeasured Ġwith Ġ$ N MI $ Ġ), Ġwith Ġsome Ġdifferences Ġdepending Ġon Ġthe Ġscale , Ġwhereas Ġthe ĠMS Ġcluster ings Ġhave Ġhigher Ġtopic Ġco herence Ġ( as Ġgiven Ġby Ġ$\ wide hat { PM I } $ Ġ) Ġacross Ġall Ġscales . </s> To Ġgive Ġan Ġindication Ġof Ġthe Ġcomputational Ġcost , Ġwe Ġran Ġboth Ġmethods Ġon Ġthe Ġsame Ġservers . ĠOur Ġmethod Ġtakes Ġapproximately Ġ13 Ġhours Ġin Ġtotal Ġto Ġcompute Ġboth Ġthe ĠDoc 2 V ec Ġmodel Ġon Ġ13 Ġmillion Ġrecords Ġ( 11 Ġhours ) Ġand Ġthe Ġfull ĠMS Ġscan Ġwith Ġ400 Ġpartitions Ġacross Ġall Ġresolutions Ġ( 2 Ġhours ). ĠThe Ġtime Ġrequired Ġto Ġtrain Ġjust Ġthe Ġ5 ĠL DA Ġmodels Ġon Ġthe Ġsame Ġcorpus Ġamounts Ġto Ġ30 Ġhours Ġ( with Ġtim ings Ġranging Ġfrom Ġ$\ sim Ġ$ Ġ2 Ġhours Ġfor Ġthe Ġ3 Ġtopic ĠL DA Ġmodel Ġto Ġ12 . 5 Ġhours Ġfor Ġthe Ġ44 Ġtopic ĠL DA Ġmodel ). </s> This Ġcomparison Ġalso Ġhighlights Ġthe Ġconceptual Ġdifference Ġbetween Ġour Ġmulti - scale Ġmethodology Ġand ĠL DA Ġtopic Ġmodelling . ĠWhile ĠL DA Ġcomp utes Ġtopics Ġat Ġa Ġpre - d etermined Ġlevel Ġof Ġresolution , Ġour Ġmethod Ġob tains Ġpartitions Ġat Ġall Ġresolutions Ġin Ġone Ġsweep Ġof Ġthe ĠMark ov Ġtime , Ġfrom Ġwhich Ġrelevant Ġpartitions Ġare Ġchosen Ġbased Ġon Ġtheir Ġrobust ness . ĠHowever , Ġthe ĠMS Ġpartitions Ġat Ġall Ġresolutions Ġare Ġavailable Ġfor Ġfurther Ġinvestigation Ġif Ġso Ġneeded . </s> We Ġhave Ġused Ġseveral Ġalgorithms Ġreadily Ġavailable Ġin Ġcode Ġlibraries Ġ( i . e ., Ġthe Ġi Graph Ġmodule Ġfor ĠPython ) Ġto Ġcluster / part ition Ġthe Ġsame Ġk NN - M ST Ġgraph . ĠFigure ĠS 5 Ġin Ġthe ĠSI Ġshows Ġthe Ġcomparison Ġagainst Ġseveral Ġwell - known Ġpartition ing Ġmethods Ġ( Mod ularity ĠOptim isation ĠB IB REF 44 Ġ, ĠInfo Map ĠB IB REF 4 Ġ, ĠWalk trap ĠB IB REF 45 Ġ, ĠLabel ĠProp ag ation ĠB IB REF 46 Ġ, Ġand ĠMulti - resolution ĠLou v ain ĠB IB REF 35 Ġ) Ġwhich Ġgive Ġjust Ġone Ġpartition Ġ( or Ġtwo Ġin Ġthe Ġcase Ġof Ġthe ĠLou v ain Ġimplementation Ġin Ġi Graph ) Ġinto Ġa Ġparticular Ġnumber Ġof Ġclusters , Ġin Ġcontrast Ġwith Ġour Ġmult isc ale ĠMS Ġanalysis . ĠOur Ġresults Ġshow Ġthat ĠMS Ġprovides Ġimproved Ġor Ġequal Ġresults Ġto Ġother Ġgraph Ġpartition ing Ġmethods Ġfor Ġboth Ġ$ N MI $ Ġand Ġ$\ wide hat { PM I } $ Ġacross Ġall Ġscales . ĠOnly Ġfor Ġvery Ġfine Ġresolution Ġwith Ġmore Ġthan Ġ50 Ġclusters , ĠInf om ap , Ġwhich Ġpartitions Ġgraphs Ġinto Ġsmall Ġcl ique - like Ġsub graph s ĠB IB REF 32 Ġ, ĠB IB REF 47 Ġ, Ġprovides Ġa Ġslightly Ġimproved Ġ$ N MI $ Ġfor Ġthat Ġparticular Ġscale . ĠTherefore , ĠMS Ġallows Ġus Ġto Ġfind Ġrelevant , Ġyet Ġhigh Ġquality Ġcluster ings Ġacross Ġall Ġscales Ġby Ġsweeping Ġthe ĠMark ov Ġtime Ġparameter . </s> Discussion </s> This Ġwork Ġhas Ġapplied Ġa Ġmult isc ale Ġgraph Ġpartition ing Ġalgorithm Ġ( Mark ov ĠStability ) Ġto Ġextract Ġcontent - based Ġclusters Ġof Ġdocuments Ġfrom Ġa Ġtextual Ġdataset Ġof Ġhealthcare Ġsafety Ġincident Ġreports Ġin Ġan Ġun super vised Ġmanner Ġat Ġdifferent Ġlevels Ġof Ġresolution . ĠThe Ġmethod Ġuses Ġparagraph Ġvectors Ġto Ġrepresent Ġthe Ġrecords Ġand Ġob tains Ġan Ġensuing Ġsimilarity Ġgraph Ġof Ġdocuments Ġconstructed Ġfrom Ġtheir Ġcontent . ĠThe Ġframework Ġbrings Ġthe Ġadvantage Ġof Ġmulti - resolution Ġalgorithms Ġcapable Ġof Ġcapturing Ġclusters Ġwithout Ġimposing Ġa Ġprior i Ġtheir Ġnumber Ġor Ġstructure . ĠSince Ġdifferent Ġlevels Ġof Ġresolution Ġof Ġthe Ġclust ering Ġcan Ġbe Ġfound Ġto Ġbe Ġrelevant , Ġthe Ġpractitioner Ġcan Ġchoose Ġthe Ġlevel Ġof Ġdescription Ġand Ġdetail Ġto Ġsuit Ġthe Ġrequirements Ġof Ġa Ġspecific Ġtask . </s> Our Ġa Ġposterior i Ġanalysis Ġevaluating Ġthe Ġsimilarity Ġagainst Ġthe Ġhand - coded Ġcategories Ġand Ġthe Ġintrinsic Ġtopic Ġco herence Ġof Ġthe Ġclusters Ġshowed Ġthat Ġthe Ġmethod Ġperformed Ġwell Ġin Ġrecovering Ġmeaningful Ġcategories . ĠThe Ġclusters Ġof Ġcontent Ġcapture Ġtopics Ġof Ġmedical Ġpractice , Ġthus Ġproviding Ġcomplementary Ġinformation Ġto Ġthe Ġexternally Ġimposed Ġclassification Ġcategories . ĠOur Ġanalysis Ġshows Ġthat Ġsome Ġof Ġthe Ġmost Ġrelevant Ġand Ġpersistent Ġcommunities Ġemerge Ġbecause Ġof Ġtheir Ġhighly Ġhom ogeneous Ġmedical Ġcontent , Ġalthough Ġthey Ġare Ġnot Ġeasily Ġmapped Ġto Ġthe Ġstandard ised Ġexternal Ġcategories . ĠThis Ġis Ġapparent Ġin Ġthe Ġmedically - based Ġcontent Ġclusters Ġassociated Ġwith ĠLabour Ġward , ĠPressure Ġul cer , ĠChem otherapy , ĠRad i otherapy , Ġamong Ġothers , Ġwhich Ġexempl ify Ġthe Ġalternative Ġgroup ings Ġthat Ġemerge Ġfrom Ġfree Ġtext Ġcontent . </s> The Ġcategories Ġin Ġthe Ġtop Ġlevel Ġ( Level Ġ1 ) Ġof Ġthe Ġpre - defined Ġclassification Ġhierarchy Ġare Ġhighly Ġdiverse Ġin Ġsize Ġ( as Ġshown Ġby Ġtheir Ġnumber Ġof Ġassigned Ġrecords ), Ġwith Ġlarge Ġgroups Ġsuch Ġas Ġ` Pat ient Ġaccident ', Ġ` Med ication ', Ġ` Cl inical Ġassessment ', Ġ` Document ation ', Ġ` Ad missions / Transfer ' Ġor Ġ` Inf rastructure ' Ġalongside Ġsmall , Ġspecific Ġgroups Ġsuch Ġas Ġ` Agg ressive Ġbehaviour ', Ġ` Pat ient Ġabuse ', Ġ` Self - harm ' Ġor Ġ` In fect ion Ġcontrol '. ĠOur Ġmulti - scale Ġpartition ing Ġfinds Ġcorresponding Ġgroups Ġin Ġcontent Ġacross Ġdifferent Ġlevels Ġof Ġresolution , Ġproviding Ġadditional Ġsub c ategories Ġwith Ġmedical Ġdetail Ġwithin Ġsome Ġof Ġthe Ġlarge Ġcategories Ġ( as Ġshown Ġin ĠFig . Ġ4 Ġand ĠS 1 Ġ). ĠAn Ġarea Ġof Ġfuture Ġresearch Ġwill Ġbe Ġto Ġconfirm Ġif Ġthe Ġcategories Ġfound Ġby Ġour Ġanalysis Ġare Ġconsistent Ġwith Ġa Ġsecond Ġlevel Ġin Ġthe Ġhierarchy Ġof Ġexternal Ġcategories Ġ( Level Ġ2 , Ġaround Ġ100 Ġcategories ) Ġthat Ġis Ġused Ġless Ġconsistently Ġin Ġhospital Ġsettings . ĠThe Ġuse Ġof Ġcontent - driven Ġclassification Ġof Ġreports Ġcould Ġalso Ġbe Ġimportant Ġwithin Ġcurrent Ġefforts Ġby Ġthe ĠWorld ĠHealth ĠOrganisation Ġ( WHO ) Ġunder Ġthe Ġframework Ġfor Ġthe ĠInternational ĠClassification Ġfor ĠPatient ĠSafety Ġ( IC PS ) ĠB IB REF 48 Ġto Ġestablish Ġa Ġset Ġof Ġconceptual Ġcategories Ġto Ġmonitor , Ġanalyse Ġand Ġinterpret Ġinformation Ġto Ġimprove Ġpatient Ġcare . </s> One Ġof Ġthe Ġadvantages Ġof Ġa Ġfree Ġtext Ġanalytical Ġapproach Ġis Ġthe Ġprovision , Ġin Ġa Ġtimely Ġmanner , Ġof Ġan Ġintellig ible Ġdescription Ġof Ġincident Ġreport Ġcategories Ġderived Ġdirectly Ġfrom Ġthe Ġrich Ġdescription Ġin Ġthe Ġ' words ' Ġof Ġthe Ġreporter Ġthemselves . ĠThe Ġinsight Ġfrom Ġanalys ing Ġthe Ġfree Ġtext Ġentry Ġof Ġthe Ġperson Ġreporting Ġcould Ġplay Ġa Ġvaluable Ġrole Ġand Ġadd Ġrich Ġinformation Ġthan Ġwould Ġhave Ġotherwise Ġbeen Ġobtained Ġfrom Ġthe Ġexisting Ġapproach Ġof Ġpre - defined Ġclasses . ĠNot Ġonly Ġcould Ġthis Ġimprove Ġthe Ġcurrent Ġstate Ġof Ġplay Ġwhere Ġmuch Ġof Ġthe Ġfree Ġtext Ġof Ġthese Ġreports Ġgoes Ġunused , Ġbut Ġit Ġavoids Ġthe Ġfallacy Ġof Ġassigning Ġincidents Ġto Ġa Ġpre - defined Ġcategory Ġthat , Ġthrough Ġa Ġlack Ġof Ġgran ularity , Ġcan Ġmiss Ġan Ġimportant Ġopportunity Ġfor Ġfeedback Ġand Ġlearning . ĠThe Ġnuanced Ġinformation Ġand Ġclass ifications Ġextracted Ġfrom Ġfree Ġtext Ġanalysis Ġthus Ġsuggest Ġa Ġcomplementary Ġaxis Ġto Ġexisting Ġapproaches Ġto Ġcharacter ise Ġpatient Ġsafety Ġincident Ġreports . </s> Currently , Ġlocal Ġincident Ġreporting Ġsystem Ġare Ġused Ġby Ġhospitals Ġto Ġsubmit Ġreports Ġto Ġthe ĠNR LS Ġand Ġrequire Ġrisk Ġmanagers Ġto Ġimprove Ġdata Ġquality Ġof Ġreports , Ġdue Ġto Ġerrors Ġor Ġuncertainty Ġin Ġcategor isation Ġfrom Ġreporters , Ġbefore Ġsubmission . ĠThe Ġapplication Ġof Ġfree Ġtext Ġanalytical Ġapproaches , Ġlike Ġthe Ġone Ġwe Ġhave Ġpresented Ġhere , Ġhas Ġthe Ġpotential Ġto Ġfree Ġup Ġrisk Ġmanagers Ġtime Ġfrom Ġlabour - intensive Ġtasks Ġof Ġclassification Ġand Ġcorrection Ġby Ġhuman Ġoperators , Ġinstead Ġfor Ġquality Ġimprovement Ġactivities Ġderived Ġfrom Ġthe Ġintelligence Ġof Ġthe Ġdata Ġitself . ĠAdditionally , Ġthe Ġmethod Ġallows Ġfor Ġthe Ġdiscovery Ġof Ġemerging Ġtopics Ġor Ġclasses Ġof Ġincidents Ġdirectly Ġfrom Ġthe Ġdata Ġwhen Ġsuch Ġevents Ġdo Ġnot Ġfit Ġthe Ġpre - ass igned Ġcategories Ġby Ġusing Ġprojection Ġtechniques Ġalongside Ġmethods Ġfor Ġanomaly Ġand Ġinnovation Ġdetection . </s> In Ġongoing Ġwork , Ġwe Ġare Ġcurrently Ġexamining Ġthe Ġuse Ġof Ġour Ġcharacter isation Ġof Ġincident Ġreports Ġto Ġenable Ġcomparisons Ġacross Ġhealthcare Ġorganisations Ġand Ġalso Ġto Ġmonitor Ġtheir Ġchange Ġover Ġtime . ĠThis Ġpart Ġof Ġongoing Ġresearch Ġrequires Ġthe Ġquant ification Ġof Ġin - class Ġtext Ġsimilarities Ġand Ġto Ġdynamically Ġmanage Ġthe Ġembed ding Ġof Ġthe Ġreports Ġthrough Ġupdates Ġand Ġrecal culation Ġof Ġthe Ġvector Ġembed ding . ĠImprovements Ġin Ġthe Ġprocess Ġof Ġrobust Ġgraph Ġconstruction Ġare Ġalso Ġpart Ġof Ġour Ġfuture Ġwork . ĠDetect ing Ġanomalies Ġin Ġthe Ġdata Ġto Ġdecide Ġwhether Ġnewer Ġtopic Ġclusters Ġshould Ġbe Ġcreated , Ġor Ġproviding Ġonline Ġclassification Ġsuggestions Ġto Ġusers Ġbased Ġon Ġthe Ġtext Ġthey Ġinput Ġare Ġsome Ġof Ġthe Ġimprovements Ġwe Ġaim Ġto Ġadd Ġin Ġthe Ġfuture Ġto Ġaid Ġwith Ġdecision Ġsupport Ġand Ġdata Ġcollection , Ġand Ġto Ġpotentially Ġhelp Ġfine - t une Ġsome Ġof Ġthe Ġpred efined Ġcategories Ġof Ġthe Ġexternal Ġclassification . </s> Availability Ġof ĠData Ġand ĠMaterials </s> The Ġdataset Ġin Ġthis Ġwork Ġis Ġmanaged Ġby Ġthe ĠBig ĠData Ġand ĠAnalytics ĠUnit Ġ( B DA U ), ĠImperial ĠCollege ĠLondon , Ġand Ġconsists Ġof Ġincident Ġreports Ġsubmitted Ġto Ġthe ĠNR LS . ĠAnalysis Ġof Ġthe Ġdata Ġwas Ġundertaken Ġwithin Ġthe ĠSecure ĠEnvironment Ġof Ġthe ĠB DA U . ĠDue Ġto Ġits Ġnature , Ġwe Ġcannot Ġpublic ise Ġany Ġpart Ġof Ġthe Ġdataset , Ġbeyond Ġthat Ġalready Ġprovided Ġwithin Ġthis Ġmanuscript . ĠNo Ġindividual Ġidentifiable Ġpatient Ġinformation Ġis Ġdisclosed Ġin Ġthis Ġwork . ĠOnly Ġaggreg ated Ġinformation Ġis Ġused Ġto Ġdescribe Ġthe Ġclusters . </s> Comp eting Ġinterests </s> The Ġauthors Ġdeclare Ġthat Ġthey Ġhave Ġno Ġcompeting Ġinterests . </s> List Ġof Ġabbrevi ations </s> N HS : ĠNational ĠHealth ĠService ; ĠNR LS : ĠNational ĠReporting Ġand ĠLearning ĠSystem ; ĠBo W : ĠBag Ġof ĠWords ; ĠL DA : ĠLat ent ĠDir ich let ĠAll ocation ; ĠDoc 2 V ec : ĠDocument Ġto ĠVector ; ĠM ST : ĠMinimum ĠSpan ning ĠTree ; Ġk NN : Ġk - Ne arest ĠNeigh b ours ; ĠMS : ĠMark ov ĠStability ; ĠNL T K : ĠNatural ĠLanguage ĠTool kit ; ĠTF - i DF : ĠTerm ĠFrequency Ġ- Ġinverse ĠDocument ĠFrequency ; ĠPV : ĠPar agraph ĠV ect ors ; ĠDB OW : ĠDist ributed ĠBag Ġof ĠWords ; ĠVI : ĠVari ation Ġof ĠInformation ; ĠN MI : ĠNormal ised ĠMutual ĠInformation ; ĠPM I : ĠPair wise ĠMutual ĠInformation . </s> Auth ors ' Ġcontributions </s> M TA Ġconducted Ġthe Ġcomputational Ġresearch . ĠMTA Ġand ĠMB Ġanalysed Ġthe Ġdata Ġand Ġdesigned Ġthe Ġcomputational Ġframework . ĠMB , ĠEM Ġand ĠSN Y Ġconceived Ġthe Ġstudy . ĠAll Ġauthors Ġwrote Ġthe Ġmanuscript . </s> Acknowled gements </s> We Ġthank ĠJoshua ĠSy mons Ġfor Ġhelp Ġwith Ġaccessing Ġthe Ġdata . ĠWe Ġalso Ġthank ĠElias ĠBam is , ĠZ ijing ĠLiu Ġand ĠMichael ĠScha ub Ġfor Ġhelpful Ġdiscussions . ĠThis Ġresearch Ġwas Ġsupported Ġby Ġthe ĠNational ĠInstitute Ġfor ĠHealth ĠResearch Ġ( NI HR ) ĠImperial ĠPatient ĠSafety ĠTrans l ational ĠResearch ĠCentre Ġand ĠNI HR ĠImperial ĠBi omedical ĠResearch ĠCentre . ĠThe Ġviews Ġexpressed Ġare Ġthose Ġof Ġthe Ġauthors Ġand Ġnot Ġnecessarily Ġthose Ġof Ġthe ĠNHS , Ġthe ĠNI HR , Ġor Ġthe ĠDepartment Ġof ĠHealth . ĠAll Ġauthors Ġacknowledge Ġfunding Ġfrom Ġthe ĠEPS RC Ġthrough Ġaward ĠEP / N 01 45 29 / 1 Ġfunding Ġthe ĠEPS RC ĠCentre Ġfor ĠMathematics Ġof ĠPrecision ĠHealthcare . </s> Auth ors ' ĠInformation </s> M TA Ġis Ġa ĠPhD Ġstudent Ġat ĠImperial ĠCollege ĠLondon , ĠDepartment Ġof ĠMathematics . ĠHe Ġholds Ġan ĠM Sc Ġdegree Ġin Ġfinance Ġfrom ĠSaban ci ĠUniversity Ġand Ġa ĠB Sc Ġin ĠElectrical Ġand ĠElectronics ĠEngineering Ġfrom ĠBog az ici ĠUniversity . ĠEM Ġis Ġa ĠClinical ĠSenior ĠLect urer Ġin Ġthe ĠDepartment Ġof ĠSurgery Ġand ĠCancer Ġand ĠCentre Ġfor ĠHealth ĠPolicy Ġat ĠImperial ĠCollege ĠLondon Ġand ĠTransformation ĠChief ĠClinical ĠInformation ĠOfficer Ġ( Cl inical ĠAnalytics Ġand ĠIn format ics ), ĠI CH NT . ĠSN Y Ġis Ġa ĠProfessor Ġof ĠThe oret ical ĠChemistry Ġin Ġthe ĠDepartment Ġof ĠChemistry Ġat ĠImperial ĠCollege ĠLondon Ġand Ġalso Ġwith Ġthe ĠEPS RC ĠCentre Ġfor ĠMathematics Ġof ĠPrecision ĠHealthcare . ĠMB Ġis ĠProfessor Ġof ĠMathematics Ġand ĠChair Ġin ĠBi om at hemat ics Ġin Ġthe ĠDepartment Ġof ĠMathematics Ġat ĠImperial ĠCollege ĠLondon , Ġand ĠDirector Ġof Ġthe ĠEPS RC ĠCentre Ġfor ĠMathematics Ġof ĠPrecision ĠHealthcare Ġat ĠImperial .