LSA, TextRank, LexRank and ILP-based summary.
ABSTRACTIVE
LSA, TextRank, LexRank
ABSTRACTIVE
We considered a dataset of 100 employees, where for each employee multiple peer comments were recorded. Also, for each employee, a manual summary was generated by an HR personnel. The summaries generated by our ILP-based approach were compared with the corresponding manual summaries using the ROUGE BIBREF22 unigram score. For comparing performance of our ILP-based summarization algorithm, we explored a few summarization algorithms provided by the Sumy package. A common parameter which is required by all these algorithms is number of sentences keep in the final summary. ILP-based summarization requires a similar parameter K, which is automatically decided based on number of total candidate phrases. Assuming a sentence is equivalent to roughly 3 phrases, for Sumy algorithms, we set number of sentences parameter to the ceiling of K/3. Table TABREF51 shows average and standard deviation of ROUGE unigram f1 scores for each algorithm, over the 100 summaries. The performance of ILP-based summarization is comparable with the other algorithms, as the two sample t-test does not show statistically significant difference. Also, human evaluators preferred phrase-based summary generated by our approach to the other sentence-based summaries. FLOAT SELECTED: Table 9. Comparative performance of various summarization algorithms
FLOAT SELECTED: Table 9. Comparative performance of various summarization algorithms We considered a dataset of 100 employees, where for each employee multiple peer comments were recorded. Also, for each employee, a manual summary was generated by an HR personnel. The summaries generated by our ILP-based approach were compared with the corresponding manual summaries using the ROUGE BIBREF22 unigram score. For comparing performance of our ILP-based summarization algorithm, we explored a few summarization algorithms provided by the Sumy package. A common parameter which is required by all these algorithms is number of sentences keep in the final summary. ILP-based summarization requires a similar parameter K, which is automatically decided based on number of total candidate phrases. Assuming a sentence is equivalent to roughly 3 phrases, for Sumy algorithms, we set number of sentences parameter to the ceiling of K/3. Table TABREF51 shows average and standard deviation of ROUGE unigram f1 scores for each algorithm, over the 100 summaries. The performance of ILP-based summarization is comparable with the other algorithms, as the two sample t-test does not show statistically significant difference. Also, human evaluators preferred phrase-based summary generated by our approach to the other sentence-based summaries.
1712.00991
Introduction </s> Performance Ġappraisal Ġ( PA ) Ġis Ġan Ġimportant ĠHR Ġprocess , Ġparticularly Ġfor Ġmodern Ġorganizations Ġthat Ġcru cially Ġdepend Ġon Ġthe Ġskills Ġand Ġexpertise Ġof Ġtheir Ġworkforce . ĠThe ĠPA Ġprocess Ġenables Ġan Ġorganization Ġto Ġperiodically Ġmeasure Ġand Ġevaluate Ġevery Ġemployee 's Ġperformance . ĠIt Ġalso Ġprovides Ġa Ġmechanism Ġto Ġlink Ġthe Ġgoals Ġestablished Ġby Ġthe Ġorganization Ġto Ġits Ġeach Ġemployee 's Ġday - to - day Ġactivities Ġand Ġperformance . ĠDesign Ġand Ġanalysis Ġof ĠPA Ġprocesses Ġis Ġa Ġlively Ġarea Ġof Ġresearch Ġwithin Ġthe ĠHR Ġcommunity ĠB IB REF 0 Ġ, ĠB IB REF 1 Ġ, ĠB IB REF 2 Ġ, ĠB IB REF 3 Ġ. </s> The ĠPA Ġprocess Ġin Ġany Ġmodern Ġorganization Ġis Ġnowadays Ġimplemented Ġand Ġtracked Ġthrough Ġan ĠIT Ġsystem Ġ( the ĠPA Ġsystem ) Ġthat Ġrecords Ġthe Ġinteractions Ġthat Ġhappen Ġin Ġvarious Ġsteps . ĠAvailability Ġof Ġthis Ġdata Ġin Ġa Ġcomputer - readable Ġdatabase Ġopens Ġup Ġopportunities Ġto Ġanalyze Ġit Ġusing Ġautomated Ġstatistical , Ġdata - mining Ġand Ġtext - mining Ġtechniques , Ġto Ġgenerate Ġnovel Ġand Ġaction able Ġinsights Ġ/ Ġpatterns Ġand Ġto Ġhelp Ġin Ġimproving Ġthe Ġquality Ġand Ġeffectiveness Ġof Ġthe ĠPA Ġprocess ĠB IB REF 4 Ġ, ĠB IB REF 5 Ġ, ĠB IB REF 6 Ġ. ĠAutom ated Ġanalysis Ġof Ġlarge - scale ĠPA Ġdata Ġis Ġnow Ġfacilitated Ġby Ġtechnological Ġand Ġalgorith mic Ġadvances , Ġand Ġis Ġbecoming Ġessential Ġfor Ġlarge Ġorganizations Ġcontaining Ġthousands Ġof Ġgeographically Ġdistributed Ġemployees Ġhandling Ġa Ġwide Ġvariety Ġof Ġroles Ġand Ġtasks . </s> A Ġtypical ĠPA Ġprocess Ġinvolves Ġpurpose ful Ġmulti - step Ġmulti - mod al Ġcommunication Ġbetween Ġemployees , Ġtheir Ġsupervisors Ġand Ġtheir Ġpeers . ĠIn Ġmost ĠPA Ġprocesses , Ġthe Ġcommunication Ġincludes Ġthe Ġfollowing Ġsteps : Ġ( i ) Ġin Ġself - app ra isal , Ġan Ġemployee Ġrecords Ġhis / her Ġachievements , Ġactivities , Ġtasks Ġhandled Ġetc .; Ġ( ii ) Ġin Ġsupervisor Ġassessment , Ġthe Ġsupervisor Ġprovides Ġthe Ġcriticism , Ġevaluation Ġand Ġsuggestions Ġfor Ġimprovement Ġof Ġperformance Ġetc .; Ġand Ġ( iii ) Ġin Ġpeer Ġfeedback Ġ( aka ĠIN LINE FORM 0 Ġview ), Ġthe Ġpeers Ġof Ġthe Ġemployee Ġprovide Ġtheir Ġfeedback . ĠThere Ġare Ġseveral Ġbusiness Ġquestions Ġthat Ġmanagers Ġare Ġinterested Ġin . ĠExamples : </s> In Ġthis Ġpaper , Ġwe Ġdevelop Ġtext Ġmining Ġtechniques Ġthat Ġcan Ġautomatically Ġproduce Ġanswers Ġto Ġthese Ġquestions . ĠSince Ġthe Ġintended Ġusers Ġare ĠHR Ġexecutives , Ġideally , Ġthe Ġtechniques Ġshould Ġwork Ġwith Ġminimum Ġtraining Ġdata Ġand Ġexperimentation Ġwith Ġparameter Ġsetting . ĠThese Ġtechniques Ġhave Ġbeen Ġimplemented Ġand Ġare Ġbeing Ġused Ġin Ġa ĠPA Ġsystem Ġin Ġa Ġlarge Ġmulti - national ĠIT Ġcompany . </s> The Ġrest Ġof Ġthe Ġpaper Ġis Ġorganized Ġas Ġfollows . ĠSection ĠSEC REF 2 Ġsummarizes Ġrelated Ġwork . ĠSection ĠSEC REF 3 Ġsummarizes Ġthe ĠPA Ġdataset Ġused Ġin Ġthis Ġpaper . ĠSection ĠSEC REF 4 Ġapplies Ġsentence Ġclassification Ġalgorithms Ġto Ġautomatically Ġdiscover Ġthree Ġimportant Ġclasses Ġof Ġsentences Ġin Ġthe ĠPA Ġcorpus Ġviz ., Ġsentences Ġthat Ġdiscuss Ġstrengths , Ġweaknesses Ġof Ġemployees Ġand Ġcontain Ġsuggestions Ġfor Ġimproving Ġher Ġperformance . ĠSection ĠSEC REF 5 Ġconsiders Ġthe Ġproblem Ġof Ġmapping Ġthe Ġactual Ġtargets Ġmentioned Ġin Ġstrengths , Ġweaknesses Ġand Ġsuggestions Ġto Ġa Ġfixed Ġset Ġof Ġattributes . ĠIn ĠSection ĠSEC REF 6 Ġ, Ġwe Ġdiscuss Ġhow Ġthe Ġfeedback Ġfrom Ġpeers Ġfor Ġa Ġparticular Ġemployee Ġcan Ġbe Ġsummarized . ĠIn ĠSection ĠSEC REF 7 Ġwe Ġdraw Ġconclusions Ġand Ġidentify Ġsome Ġfurther Ġwork . </s> Related ĠWork </s> We Ġfirst Ġreview Ġsome Ġwork Ġrelated Ġto Ġsentence Ġclassification . ĠSem antically Ġclass ifying Ġsentences Ġ( based Ġon Ġthe Ġsentence 's Ġpurpose ) Ġis Ġa Ġmuch Ġharder Ġtask , Ġand Ġis Ġgaining Ġincreasing Ġattention Ġfrom Ġlingu ists Ġand ĠN LP Ġresearchers . ĠMcK night Ġand ĠS rin ivas an ĠB IB REF 7 Ġand ĠYam amoto Ġand ĠTak agi ĠB IB REF 8 Ġused ĠS VM Ġto Ġclassify Ġsentences Ġin Ġbiomedical Ġabstract s Ġinto Ġclasses Ġsuch Ġas ĠIN TR ODUCT ION , ĠBACK GROUND , ĠPUR POSE , ĠM ETHOD , ĠRES ULT , ĠCON CLUS ION . ĠCohen Ġet Ġal . ĠB IB REF 9 Ġapplied ĠS VM Ġand Ġother Ġtechniques Ġto Ġlearn Ġclass ifiers Ġfor Ġsentences Ġin Ġemails Ġinto Ġclasses , Ġwhich Ġare Ġspeech Ġacts Ġdefined Ġby Ġa Ġverb - n oun Ġpair , Ġwith Ġverbs Ġsuch Ġas Ġrequest , Ġpropose , Ġamend , Ġcommit , Ġdeliver Ġand Ġnoun s Ġsuch Ġas Ġmeeting , Ġdocument , Ġcommittee ; Ġsee Ġalso ĠB IB REF 10 Ġ. ĠKh oo Ġet Ġal . ĠB IB REF 11 Ġuses Ġvarious Ġclass ifiers Ġto Ġclassify Ġsentences Ġin Ġemails Ġinto Ġclasses Ġsuch Ġas ĠAP OLOGY , ĠIN STRUCT ION , ĠQUEST ION , ĠRE QUEST , ĠSAL UT ATION , ĠSTAT EMENT , ĠS UG G EST ION , ĠTHANK ING Ġetc . ĠQ ad ir Ġand ĠR il off ĠB IB REF 12 Ġproposes Ġseveral Ġfilters Ġand Ġclass ifiers Ġto Ġclassify Ġsentences Ġon Ġmessage Ġboards Ġ( community ĠQ A Ġsystems ) Ġinto Ġ4 Ġspeech Ġacts : ĠCOMM ISS IVE Ġ( spe aker Ġcommits Ġto Ġa Ġfuture Ġaction ), ĠDIRECT IVE Ġ( spe aker Ġexpects Ġlistener Ġto Ġtake Ġsome Ġaction ), ĠEX PRESS IVE Ġ( spe aker Ġexpresses Ġhis Ġor Ġher Ġpsychological Ġstate Ġto Ġthe Ġlistener ), ĠRE PRES ENT ATIVE Ġ( rep resents Ġthe Ġspeaker 's Ġbelief Ġof Ġsomething ). ĠH ache y Ġand ĠGro ver ĠB IB REF 13 Ġused ĠS VM Ġand Ġmaximum Ġentropy Ġclass ifiers Ġto Ġclassify Ġsentences Ġin Ġlegal Ġdocuments Ġinto Ġclasses Ġsuch Ġas ĠF ACT , ĠPROC EED INGS , ĠBACK GROUND , ĠFR AM ING , ĠDIS POS AL ; Ġsee Ġalso ĠB IB REF 14 Ġ. ĠDes hp ande Ġet Ġal . ĠB IB REF 15 Ġproposes Ġun super vised Ġlinguistic Ġpatterns Ġto Ġclassify Ġsentences Ġinto Ġclasses ĠS UG G EST ION , ĠCOM PLA INT . </s> There Ġis Ġmuch Ġwork Ġon Ġa Ġclosely Ġrelated Ġproblem Ġviz ., Ġclass ifying Ġsentences Ġin Ġdialog ues Ġthrough Ġdialogue - specific Ġcategories Ġcalled Ġdialogue Ġacts ĠB IB REF 16 Ġ, Ġwhich Ġwe Ġwill Ġnot Ġreview Ġhere . ĠJust Ġas Ġone Ġexample , ĠCot ter ill ĠB IB REF 17 Ġclass ifies Ġquestions Ġin Ġemails Ġinto Ġthe Ġdialogue Ġacts Ġof ĠYES _ NO _ QUEST ION , ĠWH _ QUEST ION , ĠACTION _ RE QUEST , ĠRH ET OR ICAL , ĠM ULT IP LE _ CHO ICE Ġetc . </s> We Ġcould Ġnot Ġfind Ġmuch Ġwork Ġrelated Ġto Ġmining Ġof Ġperformance Ġappra is als Ġdata . ĠPaw ar Ġet Ġal . ĠB IB REF 18 Ġuses Ġkernel - based Ġclassification Ġto Ġclassify Ġsentences Ġin Ġboth Ġperformance Ġappraisal Ġtext Ġand Ġproduct Ġreviews Ġinto Ġclasses ĠS UG G EST ION , ĠAPP RE CI ATION , ĠCOM PLA INT . ĠA pt e Ġet Ġal . ĠB IB REF 6 Ġprovides Ġtwo Ġalgorithms Ġfor Ġmatching Ġthe Ġdescriptions Ġof Ġgoals Ġor Ġtasks Ġassigned Ġto Ġemployees Ġto Ġa Ġstandard Ġtemplate Ġof Ġmodel Ġgoals . ĠOne Ġalgorithm Ġis Ġbased Ġon Ġthe Ġco - training Ġframework Ġand Ġuses Ġgoal Ġdescriptions Ġand Ġself - app ra isal Ġcomments Ġas Ġtwo Ġseparate Ġperspectives . ĠThe Ġsecond Ġapproach Ġuses Ġsemantic Ġsimilarity Ġunder Ġa Ġweak Ġsupervision Ġframework . ĠRam rak hi y ani Ġet Ġal . ĠB IB REF 5 Ġproposes Ġlabel Ġpropagation Ġalgorithms Ġto Ġdiscover Ġaspects Ġin Ġsupervisor Ġassessments Ġin Ġperformance Ġappra is als , Ġwhere Ġan Ġaspect Ġis Ġmod elled Ġas Ġa Ġverb - n oun Ġpair Ġ( e . g . Ġconduct Ġtraining , Ġimprove Ġcoding ). </s> Dat as et </s> In Ġthis Ġpaper , Ġwe Ġused Ġthe Ġsupervisor Ġassessment Ġand Ġpeer Ġfeedback Ġtext Ġproduced Ġduring Ġthe Ġperformance Ġappraisal Ġof Ġ45 28 Ġemployees Ġin Ġa Ġlarge Ġmulti - national ĠIT Ġcompany . ĠThe Ġcorpus Ġof Ġsupervisor Ġassessment Ġhas Ġ269 72 Ġsentences . ĠThe Ġsummary Ġstatistics Ġabout Ġthe Ġnumber Ġof Ġwords Ġin Ġa Ġsentence Ġis : Ġmin : 4 Ġmax : 217 Ġaverage : 15 . 5 ĠST DEV : 9 . 2 ĠQ 1 : 9 ĠQ 2 : 14 ĠQ 3 : 19 . </s> Sent ence ĠClassification </s> The ĠPA Ġcorpus Ġcontains Ġseveral Ġclasses Ġof Ġsentences Ġthat Ġare Ġof Ġinterest . ĠIn Ġthis Ġpaper , Ġwe Ġfocus Ġon Ġthree Ġimportant Ġclasses Ġof Ġsentences Ġviz ., Ġsentences Ġthat Ġdiscuss Ġstrengths Ġ( class ĠSTR ENGTH ), Ġweaknesses Ġof Ġemployees Ġ( class ĠWE AK NESS ) Ġand Ġsuggestions Ġfor Ġimproving Ġher Ġperformance Ġ( class ĠS UG G EST ION ). ĠThe Ġstrengths Ġor Ġweaknesses Ġare Ġmostly Ġabout Ġthe Ġperformance Ġin Ġwork Ġcarried Ġout , Ġbut Ġsometimes Ġthey Ġcan Ġbe Ġabout Ġthe Ġworking Ġstyle Ġor Ġother Ġpersonal Ġqualities . ĠThe Ġclasses ĠWE AK NESS Ġand ĠS UG G EST ION Ġare Ġsomewhat Ġoverlapping ; Ġe . g ., Ġa Ġsuggestion Ġmay Ġaddress Ġa Ġperceived Ġweakness . ĠFollowing Ġare Ġtwo Ġexample Ġsentences Ġin Ġeach Ġclass . </s> STR ENGTH : </s> WE AK NESS : </s> S UG G EST ION : </s> Several Ġlinguistic Ġaspects Ġof Ġthese Ġclasses Ġof Ġsentences Ġare Ġapparent . ĠThe Ġsubject Ġis Ġimplicit Ġin Ġmany Ġsentences . ĠThe Ġstrengths Ġare Ġoften Ġmentioned Ġas Ġeither Ġnoun Ġphrases Ġ( NP ) Ġwith Ġpositive Ġadject ives Ġ( Excellent Ġtechnology Ġleadership ) Ġor Ġpositive Ġnoun s Ġ( engineering Ġstrength ) Ġor Ġthrough Ġverbs Ġwith Ġpositive Ġpol arity Ġ( ded icated ) Ġor Ġas Ġverb Ġphrases Ġcontaining Ġpositive Ġadject ives Ġ( del ivers Ġinnovative Ġsolutions ). ĠSimilarly Ġfor Ġweaknesses , Ġwhere Ġneg ation Ġis Ġmore Ġfrequently Ġused Ġ( present ations Ġare Ġnot Ġhis Ġfor te ), Ġor Ġalternatively , Ġthe Ġpolar ities Ġof Ġverbs Ġ( avoid ) Ġor Ġadject ives Ġ( poor ) Ġtend Ġto Ġbe Ġnegative . ĠHowever , Ġsometimes Ġthe Ġform Ġof Ġboth Ġthe Ġstrengths Ġand Ġweaknesses Ġis Ġthe Ġsame , Ġtypically Ġa Ġstand - alone Ġsentiment - neutral ĠNP , Ġmaking Ġit Ġdifficult Ġto Ġdistinguish Ġbetween Ġthem ; Ġe . g ., Ġadherence Ġto Ġtiming Ġor Ġtimely Ġclosure . ĠSuggest ions Ġoften Ġhave Ġan Ġimperative Ġmood Ġand Ġcontain Ġsecondary Ġverbs Ġsuch Ġas Ġneed Ġto , Ġshould , Ġhas Ġto . ĠSuggest ions Ġare Ġsometimes Ġexpressed Ġusing Ġcompar atives Ġ( better Ġprocess Ġcompliance ). ĠWe Ġbuilt Ġa Ġsimple Ġset Ġof Ġpatterns Ġfor Ġeach Ġof Ġthe Ġ3 Ġclasses Ġon Ġthe ĠPOS - tag ged Ġform Ġof Ġthe Ġsentences . ĠWe Ġuse Ġeach Ġset Ġof Ġthese Ġpatterns Ġas Ġan Ġun super vised Ġsentence Ġclass ifier Ġfor Ġthat Ġclass . ĠIf Ġa Ġparticular Ġsentence Ġmatched Ġwith Ġpatterns Ġfor Ġmultiple Ġclasses , Ġthen Ġwe Ġhave Ġsimple Ġtie - breaking Ġrules Ġfor Ġpicking Ġthe Ġfinal Ġclass . ĠThe Ġpattern Ġfor Ġthe ĠSTR ENGTH Ġclass Ġlooks Ġfor Ġthe Ġpresence Ġof Ġpositive Ġwords Ġ/ Ġphrases Ġlike Ġtakes Ġownership , Ġexcellent , Ġhard Ġworking , Ġcommitment , Ġetc . ĠSimilarly , Ġthe Ġpattern Ġfor Ġthe ĠWE AK NESS Ġclass Ġlooks Ġfor Ġthe Ġpresence Ġof Ġnegative Ġwords Ġ/ Ġphrases Ġlike Ġlacking , Ġdiff ident , Ġslow Ġlear ner , Ġless Ġfocused , Ġetc . ĠThe ĠS UG G EST ION Ġpattern Ġnot Ġonly Ġlooks Ġfor Ġkeywords Ġlike Ġshould , Ġneeds Ġto Ġbut Ġalso Ġfor ĠPOS Ġbased Ġpattern Ġlike ĠâĢ ľ a Ġverb Ġin Ġthe Ġbase Ġform Ġ( VB ) Ġin Ġthe Ġbeginning Ġof Ġa Ġsentence âĢ Ŀ . </s> We Ġrandomly Ġselected Ġ2000 Ġsentences Ġfrom Ġthe Ġsupervisor Ġassessment Ġcorpus Ġand Ġmanually Ġtagged Ġthem Ġ( dat as et ĠD 1 ). ĠThis Ġlabelled Ġdataset Ġcontained Ġ7 05 , Ġ103 , Ġ8 22 Ġand Ġ370 Ġsentences Ġhaving Ġthe Ġclass Ġlabels ĠSTR ENGTH , ĠWE AK NESS , ĠS UG G EST ION Ġor ĠOTHER Ġrespectively . ĠWe Ġtrained Ġseveral Ġmulti - class Ġclass ifiers Ġon Ġthis Ġdataset . ĠTable ĠT AB REF 10 Ġshows Ġthe Ġresults Ġof Ġ5 - fold Ġcross - valid ation Ġexperiments Ġon Ġdataset ĠD 1 . ĠFor Ġthe Ġfirst Ġ5 Ġclass ifiers , Ġwe Ġused Ġtheir Ġimplementation Ġfrom Ġthe ĠSci Kit ĠLearn Ġlibrary Ġin ĠPython Ġ( sci kit - learn . org ). ĠThe Ġfeatures Ġused Ġfor Ġthese Ġclass ifiers Ġwere Ġsimply Ġthe Ġsentence Ġwords Ġalong Ġwith Ġtheir Ġfrequencies . ĠFor Ġthe Ġlast Ġ2 Ġclass ifiers Ġ( in ĠTable ĠT AB REF 10 Ġ), Ġwe Ġused Ġour Ġown Ġimplementation . ĠThe Ġoverall Ġaccuracy Ġfor Ġa Ġclass ifier Ġis Ġdefined Ġas ĠIN LINE FORM 0 Ġ, Ġwhere Ġthe Ġdenomin ator Ġis Ġ2000 Ġfor Ġdataset ĠD 1 . ĠNote Ġthat Ġthe Ġpattern - based Ġapproach Ġis Ġun super vised Ġi . e ., Ġit Ġdid Ġnot Ġuse Ġany Ġtraining Ġdata . ĠHence , Ġthe Ġresults Ġshown Ġfor Ġit Ġare Ġfor Ġthe Ġentire Ġdataset Ġand Ġnot Ġbased Ġon Ġcross - valid ation . </s> Compar ison Ġwith ĠSent iment ĠAnaly zer </s> We Ġalso Ġexplored Ġwhether Ġa Ġsentiment Ġanaly zer Ġcan Ġbe Ġused Ġas Ġa Ġbaseline Ġfor Ġidentifying Ġthe Ġclass Ġlabels ĠSTR ENGTH Ġand ĠWE AK NESS . ĠWe Ġused Ġan Ġimplementation Ġof Ġsentiment Ġanaly zer Ġfrom ĠText Bl ob Ġto Ġget Ġa Ġpol arity Ġscore Ġfor Ġeach Ġsentence . ĠTable ĠT AB REF 13 Ġshows Ġthe Ġdistribution Ġof Ġpositive , Ġnegative Ġand Ġneutral Ġsentiments Ġacross Ġthe Ġ3 Ġclass Ġlabels ĠSTR ENGTH , ĠWE AK NESS Ġand ĠS UG G EST ION . ĠIt Ġcan Ġbe Ġobserved Ġthat Ġdistribution Ġof Ġpositive Ġand Ġnegative Ġsentiments Ġis Ġalmost Ġsimilar Ġin ĠSTR ENGTH Ġas Ġwell Ġas ĠS UG G EST ION Ġsentences , Ġhence Ġwe Ġcan Ġconclude Ġthat Ġthe Ġinformation Ġabout Ġsentiments Ġis Ġnot Ġmuch Ġuseful Ġfor Ġour Ġclassification Ġproblem . </s> Discover ing ĠCl usters Ġwithin ĠSent ence ĠClasses </s> After Ġidentifying Ġsentences Ġin Ġeach Ġclass , Ġwe Ġcan Ġnow Ġanswer Ġquestion Ġ( 1 ) Ġin ĠSection ĠSEC REF 1 Ġ. ĠFrom Ġ127 42 Ġsentences Ġpredicted Ġto Ġhave Ġlabel ĠSTR ENGTH , Ġwe Ġextract Ġnoun s Ġthat Ġindicate Ġthe Ġactual Ġstrength , Ġand Ġcluster Ġthem Ġusing Ġa Ġsimple Ġclust ering Ġalgorithm Ġwhich Ġuses Ġthe Ġcos ine Ġsimilarity Ġbetween Ġword Ġembed d ings Ġof Ġthese Ġnoun s . ĠWe Ġrepeat Ġthis Ġfor Ġthe Ġ9 160 Ġsentences Ġwith Ġpredicted Ġlabel ĠWE AK NESS Ġor ĠS UG G EST ION Ġas Ġa Ġsingle Ġclass . ĠTables ĠT AB REF 15 Ġand ĠT AB REF 16 Ġshow Ġa Ġfew Ġrepresentative Ġclusters Ġin Ġstrengths Ġand Ġin Ġweaknesses , Ġrespectively . ĠWe Ġalso Ġexplored Ġclust ering Ġ127 42 ĠSTR ENGTH Ġsentences Ġdirectly Ġusing ĠCL UT O ĠB IB REF 19 Ġand ĠCar rot 2 ĠLing o ĠB IB REF 20 Ġclust ering Ġalgorithms . ĠCar rot 2 ĠLing o Ġdiscovered Ġ167 Ġclusters Ġand Ġalso Ġassigned Ġlabels Ġto Ġthese Ġclusters . ĠWe Ġthen Ġgenerated Ġ167 Ġclusters Ġusing ĠCL UT O Ġas Ġwell . ĠCL UT O Ġdoes Ġnot Ġgenerate Ġcluster Ġlabels Ġautomatically , Ġhence Ġwe Ġused Ġ5 Ġmost Ġfrequent Ġwords Ġwithin Ġthe Ġcluster Ġas Ġits Ġlabels . ĠTable ĠT AB REF 19 Ġshows Ġthe Ġlargest Ġ5 Ġclusters Ġby Ġboth Ġthe Ġalgorithms . ĠIt Ġwas Ġobserved Ġthat Ġthe Ġclusters Ġcreated Ġby ĠCL UT O Ġwere Ġmore Ġmeaningful Ġand Ġinformative Ġas Ġcompared Ġto Ġthose Ġby ĠCar rot 2 ĠLing o . ĠAlso , Ġit Ġwas Ġobserved Ġthat Ġthere Ġis Ġsome Ġcorrespondence Ġbetween Ġnoun Ġclusters Ġand Ġsentence Ġclusters . ĠE . g . Ġthe Ġnoun s Ġcluster Ġmotivation Ġexpertise Ġknowledge Ġtalent Ġskill Ġ( Table ĠT AB REF 15 Ġ) Ġcorresponds Ġto Ġthe ĠCL UT O Ġsentence Ġcluster Ġskill Ġcustomer Ġmanagement Ġknowledge Ġteam Ġ( Table ĠT AB REF 19 Ġ). ĠBut Ġoverall , Ġusers Ġfound Ġthe Ġnoun s Ġclusters Ġto Ġbe Ġmore Ġmeaningful Ġthan Ġthe Ġsentence Ġclusters . </s> PA Ġalong ĠAttributes </s> In Ġmany Ġorganizations , ĠPA Ġis Ġdone Ġfrom Ġa Ġpred efined Ġset Ġof Ġperspectives , Ġwhich Ġwe Ġcall Ġattributes . ĠEach Ġattribute Ġcovers Ġone Ġspecific Ġaspect Ġof Ġthe Ġwork Ġdone Ġby Ġthe Ġemployees . ĠThis Ġhas Ġthe Ġadvantage Ġthat Ġwe Ġcan Ġeasily Ġcompare Ġthe Ġperformance Ġof Ġany Ġtwo Ġemployees Ġ( or Ġgroups Ġof Ġemployees ) Ġalong Ġany Ġgiven Ġattribute . ĠWe Ġcan Ġcorrelate Ġvarious Ġperformance Ġattributes Ġand Ġfind Ġdependencies Ġamong Ġthem . ĠWe Ġcan Ġalso Ġcluster Ġemployees Ġin Ġthe Ġworkforce Ġusing Ġtheir Ġsupervisor Ġratings Ġfor Ġeach Ġattribute Ġto Ġdiscover Ġinteresting Ġinsights Ġinto Ġthe Ġworkforce . ĠThe ĠHR Ġmanagers Ġin Ġthe Ġorganization Ġconsidered Ġin Ġthis Ġpaper Ġhave Ġdefined Ġ15 Ġattributes Ġ( Table ĠT AB REF 20 Ġ). ĠEach Ġattribute Ġis Ġessentially Ġa Ġwork Ġitem Ġor Ġwork Ġcategory Ġdescribed Ġat Ġan Ġabstract Ġlevel . ĠFor Ġexample , ĠFUN CT ION AL _ EX CE LL ENCE Ġcovers Ġany Ġtasks , Ġgoals Ġor Ġactivities Ġrelated Ġto Ġthe Ġsoftware Ġengineering Ġlife - cycle Ġ( e . g ., Ġrequirements Ġanalysis , Ġdesign , Ġcoding , Ġtesting Ġetc .) Ġas Ġwell Ġas Ġtechnologies Ġsuch Ġas Ġdatabases , Ġweb Ġservices Ġand ĠGUI . </s> In Ġthe Ġexample Ġin ĠSection ĠSEC REF 4 Ġ, Ġthe Ġfirst Ġsentence Ġ( which Ġhas Ġclass ĠSTR ENGTH ) Ġcan Ġbe Ġmapped Ġto Ġtwo Ġattributes : ĠFUN CT ION AL _ EX CE LL ENCE Ġand ĠBU ILD ING _ EFF ECT IVE _ TE AMS . ĠSimilarly , Ġthe Ġthird Ġsentence Ġ( which Ġhas Ġclass ĠWE AK NESS ) Ġcan Ġbe Ġmapped Ġto Ġthe Ġattribute ĠINTER P ERSON AL _ EFF ECT IV EN ESS Ġand Ġso Ġforth . ĠThus , Ġin Ġorder Ġto Ġanswer Ġthe Ġsecond Ġquestion Ġin ĠSection ĠSEC REF 1 Ġ, Ġwe Ġneed Ġto Ġmap Ġeach Ġsentence Ġin Ġeach Ġof Ġthe Ġ3 Ġclasses Ġto Ġzero , Ġone , Ġtwo Ġor Ġmore Ġattributes , Ġwhich Ġis Ġa Ġmulti - class Ġmulti - label Ġclassification Ġproblem . </s> We Ġmanually Ġtagged Ġthe Ġsame Ġ2000 Ġsentences Ġin ĠDat as et ĠD 1 Ġwith Ġattributes , Ġwhere Ġeach Ġsentence Ġmay Ġget Ġ0 , Ġ1 , Ġ2 , Ġetc . Ġup Ġto Ġ15 Ġclass Ġlabels Ġ( this Ġis Ġdataset ĠD 2 ). ĠThis Ġlabelled Ġdataset Ġcontained Ġ7 49 , Ġ206 , Ġ289 , Ġ207 , Ġ91 , Ġ223 , Ġ191 , Ġ144 , Ġ103 , Ġ80 , Ġ82 , Ġ42 , Ġ29 , Ġ15 , Ġ24 Ġsentences Ġhaving Ġthe Ġclass Ġlabels Ġlisted Ġin ĠTable ĠT AB REF 20 Ġin Ġthe Ġsame Ġorder . ĠThe Ġnumber Ġof Ġsentences Ġhaving Ġ0 , Ġ1 , Ġ2 , Ġor Ġmore Ġthan Ġ2 Ġattributes Ġare : Ġ321 , Ġ1070 , Ġ470 Ġand Ġ139 Ġrespectively . ĠWe Ġtrained Ġseveral Ġmulti - class Ġmulti - label Ġclass ifiers Ġon Ġthis Ġdataset . ĠTable ĠT AB REF 21 Ġshows Ġthe Ġresults Ġof Ġ5 - fold Ġcross - valid ation Ġexperiments Ġon Ġdataset ĠD 2 . </s> Pre cision , ĠRecall Ġand ĠF - me asure Ġfor Ġthis Ġmulti - label Ġclassification Ġare Ġcomputed Ġusing Ġa Ġstrategy Ġsimilar Ġto Ġthe Ġone Ġdescribed Ġin ĠB IB REF 21 Ġ. ĠLet ĠIN LINE FORM 0 Ġbe Ġthe Ġset Ġof Ġpredicted Ġlabels Ġand ĠIN LINE FORM 1 Ġbe Ġthe Ġset Ġof Ġactual Ġlabels Ġfor Ġthe ĠIN LINE FORM 2 Ġinstance . ĠPrecision Ġand Ġrecall Ġfor Ġthis Ġinstance Ġare Ġcomputed Ġas Ġfollows : ĠIN LINE FORM 3 </s> It Ġcan Ġbe Ġobserved Ġthat ĠIN LINE FORM 0 Ġwould Ġbe Ġundefined Ġif ĠIN LINE FORM 1 Ġis Ġempty Ġand Ġsimilarly ĠIN LINE FORM 2 Ġwould Ġbe Ġundefined Ġwhen ĠIN LINE FORM 3 Ġis Ġempty . ĠHence , Ġoverall Ġprecision Ġand Ġrecall Ġare Ġcomputed Ġby Ġaveraging Ġover Ġall Ġthe Ġinstances Ġexcept Ġwhere Ġthey Ġare Ġundefined . ĠInst ance - level ĠF - me asure Ġcan Ġnot Ġbe Ġcomputed Ġfor Ġinstances Ġwhere Ġeither Ġprecision Ġor Ġrecall Ġare Ġundefined . ĠTherefore , Ġoverall ĠF - me asure Ġis Ġcomputed Ġusing Ġthe Ġoverall Ġprecision Ġand Ġrecall . </s> Sum mar ization Ġof ĠPeer ĠFeedback Ġusing ĠIL P </s> The ĠPA Ġsystem Ġincludes Ġa Ġset Ġof Ġpeer Ġfeedback Ġcomments Ġfor Ġeach Ġemployee . ĠTo Ġanswer Ġthe Ġthird Ġquestion Ġin ĠSection ĠSEC REF 1 Ġ, Ġwe Ġneed Ġto Ġcreate Ġa Ġsummary Ġof Ġall Ġthe Ġpeer Ġfeedback Ġcomments Ġabout Ġa Ġgiven Ġemployee . ĠAs Ġan Ġexample , Ġfollowing Ġare Ġthe Ġfeedback Ġcomments Ġfrom Ġ5 Ġpeers Ġof Ġan Ġemployee . </s> The Ġindividual Ġsentences Ġin Ġthe Ġcomments Ġwritten Ġby Ġeach Ġpeer Ġare Ġfirst Ġidentified Ġand Ġthen ĠPOS Ġtags Ġare Ġassigned Ġto Ġeach Ġsentence . ĠWe Ġhypothes ize Ġthat Ġa Ġgood Ġsummary Ġof Ġthese Ġmultiple Ġcomments Ġcan Ġbe Ġconstructed Ġby Ġidentifying Ġa Ġset Ġof Ġimportant Ġtext Ġfragments Ġor Ġphrases . ĠInitially , Ġa Ġset Ġof Ġcandidate Ġphrases Ġis Ġextracted Ġfrom Ġthese Ġcomments Ġand Ġa Ġsubset Ġof Ġthese Ġcandidate Ġphrases Ġis Ġchosen Ġas Ġthe Ġfinal Ġsummary , Ġusing ĠInteger ĠLinear ĠProgramming Ġ( IL P ). ĠThe Ġdetails Ġof Ġthe ĠIL P Ġformulation Ġare Ġshown Ġin ĠTable ĠT AB REF 36 Ġ. ĠAs Ġan Ġexample , Ġfollowing Ġis Ġthe Ġsummary Ġgenerated Ġfor Ġthe Ġabove Ġ5 Ġpeer Ġcomments . </s> h umble Ġnature , Ġeffective Ġcommunication , Ġtechnical Ġexpertise , Ġalways Ġsupportive , Ġvast Ġknowledge </s> Following Ġrules Ġare Ġused Ġto Ġidentify Ġcandidate Ġphrases : </s> Various Ġparameters Ġare Ġused Ġto Ġevaluate Ġa Ġcandidate Ġphrase Ġfor Ġits Ġimportance . ĠA Ġcandidate Ġphrase Ġis Ġmore Ġimportant : </s> A Ġcomplete Ġlist Ġof Ġparameters Ġis Ġdescribed Ġin Ġdetail Ġin ĠTable ĠT AB REF 36 Ġ. </s> There Ġis Ġa Ġtrivial Ġconstraint ĠIN LINE FORM 0 Ġwhich Ġmakes Ġsure Ġthat Ġonly ĠIN LINE FORM 1 Ġout Ġof ĠIN LINE FORM 2 Ġcandidate Ġphrases Ġare Ġchosen . ĠA Ġsuitable Ġvalue Ġof ĠIN LINE FORM 3 Ġis Ġused Ġfor Ġeach Ġemployee Ġdepending Ġon Ġnumber Ġof Ġcandidate Ġphrases Ġidentified Ġacross Ġall Ġpeers Ġ( see ĠAl gorithm ĠSEC REF 6 Ġ). ĠAnother Ġset Ġof Ġconstraints Ġ( ĠIN LINE FORM 4 Ġto ĠIN LINE FORM 5 Ġ) Ġmake Ġsure Ġthat Ġat Ġleast Ġone Ġphrase Ġis Ġselected Ġfor Ġeach Ġof Ġthe Ġleadership Ġattributes . ĠThe Ġconstraint ĠIN LINE FORM 6 Ġmakes Ġsure Ġthat Ġmultiple Ġphrases Ġsharing Ġthe Ġsame Ġhead word Ġare Ġnot Ġchosen Ġat Ġa Ġtime . ĠAlso , Ġsingle Ġword Ġcandidate Ġphrases Ġare Ġchosen Ġonly Ġif Ġthey Ġare Ġadject ives Ġor Ġnoun s Ġwith Ġlex ical Ġcategory Ġnoun . attribute . ĠThis Ġis Ġimposed Ġby Ġthe Ġconstraint ĠIN LINE FORM 7 Ġ. ĠIt Ġis Ġimportant Ġto Ġnote Ġthat Ġall Ġthe Ġconstraints Ġexcept ĠIN LINE FORM 8 Ġare Ġsoft Ġconstraints , Ġi . e . Ġthere Ġmay Ġbe Ġfeasible Ġsolutions Ġwhich Ġdo Ġnot Ġsatisfy Ġsome Ġof Ġthese Ġconstraints . ĠBut Ġeach Ġconstraint Ġwhich Ġis Ġnot Ġsatisfied , Ġresults Ġin Ġa Ġpenalty Ġthrough Ġthe Ġuse Ġof Ġslack Ġvariables . ĠThese Ġconstraints Ġare Ġdescribed Ġin Ġdetail Ġin ĠTable ĠT AB REF 36 Ġ. </s> The Ġobjective Ġfunction Ġmaxim izes Ġthe Ġtotal Ġimportance Ġscore Ġof Ġthe Ġselected Ġcandidate Ġphrases . ĠAt Ġthe Ġsame Ġtime , Ġit Ġalso Ġminim izes Ġthe Ġsum Ġof Ġall Ġslack Ġvariables Ġso Ġthat Ġthe Ġminimum Ġnumber Ġof Ġconstraints Ġare Ġbroken . </s> IN LINE FORM 0 Ġ: ĠNo . Ġof Ġcandidate Ġphrases ĠIN LINE FORM 1 Ġ: ĠNo . Ġof Ġphrases Ġto Ġselect Ġas Ġpart Ġof Ġsummary </s> IN LINE FORM 0 ĠIN LINE FORM 1 ĠIN LINE FORM 2 ĠIN LINE FORM 3 ĠIN LINE FORM 4 ĠIN LINE FORM 5 ĠIN LINE FORM 6 ĠIN LINE FORM 7 ĠIN LINE FORM 8 </s> IN LINE FORM 0 Ġand ĠIN LINE FORM 1 ĠIN LINE FORM 2 ĠIN LINE FORM 3 ĠIN LINE FORM 4 ĠIN LINE FORM 5 ĠIN LINE FORM 6 </s> IN LINE FORM 0 Ġ( For Ġdetermining Ġnumber Ġof Ġphrases Ġto Ġselect Ġto Ġinclude Ġin Ġsummary ) </s> E val uation Ġof Ġauto - generated Ġsumm aries </s> We Ġconsidered Ġa Ġdataset Ġof Ġ100 Ġemployees , Ġwhere Ġfor Ġeach Ġemployee Ġmultiple Ġpeer Ġcomments Ġwere Ġrecorded . ĠAlso , Ġfor Ġeach Ġemployee , Ġa Ġmanual Ġsummary Ġwas Ġgenerated Ġby Ġan ĠHR Ġpersonnel . ĠThe Ġsumm aries Ġgenerated Ġby Ġour ĠIL P - based Ġapproach Ġwere Ġcompared Ġwith Ġthe Ġcorresponding Ġmanual Ġsumm aries Ġusing Ġthe ĠR OU GE ĠB IB REF 22 Ġun ig ram Ġscore . ĠFor Ġcomparing Ġperformance Ġof Ġour ĠIL P - based Ġsummar ization Ġalgorithm , Ġwe Ġexplored Ġa Ġfew Ġsummar ization Ġalgorithms Ġprovided Ġby Ġthe ĠSum y Ġpackage . ĠA Ġcommon Ġparameter Ġwhich Ġis Ġrequired Ġby Ġall Ġthese Ġalgorithms Ġis Ġnumber Ġof Ġsentences Ġkeep Ġin Ġthe Ġfinal Ġsummary . ĠIL P - based Ġsummar ization Ġrequires Ġa Ġsimilar Ġparameter ĠK , Ġwhich Ġis Ġautomatically Ġdecided Ġbased Ġon Ġnumber Ġof Ġtotal Ġcandidate Ġphrases . ĠAssuming Ġa Ġsentence Ġis Ġequivalent Ġto Ġroughly Ġ3 Ġphrases , Ġfor ĠSum y Ġalgorithms , Ġwe Ġset Ġnumber Ġof Ġsentences Ġparameter Ġto Ġthe Ġceiling Ġof ĠK / 3 . ĠTable ĠT AB REF 51 Ġshows Ġaverage Ġand Ġstandard Ġdeviation Ġof ĠR OU GE Ġun ig ram Ġf 1 Ġscores Ġfor Ġeach Ġalgorithm , Ġover Ġthe Ġ100 Ġsumm aries . ĠThe Ġperformance Ġof ĠIL P - based Ġsummar ization Ġis Ġcomparable Ġwith Ġthe Ġother Ġalgorithms , Ġas Ġthe Ġtwo Ġsample Ġt - test Ġdoes Ġnot Ġshow Ġstatistically Ġsignificant Ġdifference . ĠAlso , Ġhuman Ġeval u ators Ġpreferred Ġphrase - based Ġsummary Ġgenerated Ġby Ġour Ġapproach Ġto Ġthe Ġother Ġsentence - based Ġsumm aries . </s> Con clusions Ġand ĠFurther ĠWork </s> In Ġthis Ġpaper , Ġwe Ġpresented Ġan Ġanalysis Ġof Ġthe Ġtext Ġgenerated Ġin ĠPerformance ĠApp ra isal Ġ( PA ) Ġprocess Ġin Ġa Ġlarge Ġmulti - national ĠIT Ġcompany . ĠWe Ġperformed Ġsentence Ġclassification Ġto Ġidentify Ġstrengths , Ġweaknesses Ġand Ġsuggestions Ġfor Ġimprovements Ġfound Ġin Ġthe Ġsupervisor Ġassessments Ġand Ġthen Ġused Ġclust ering Ġto Ġdiscover Ġbroad Ġcategories Ġamong Ġthem . ĠAs Ġthis Ġis Ġnon - top ical Ġclassification , Ġwe Ġfound Ġthat ĠS VM Ġwith ĠAD WS Ġkernel ĠB IB REF 18 Ġproduced Ġthe Ġbest Ġresults . ĠWe Ġalso Ġused Ġmulti - class Ġmulti - label Ġclassification Ġtechniques Ġto Ġmatch Ġsupervisor Ġassessments Ġto Ġpred efined Ġbroad Ġperspectives Ġon Ġperformance . ĠLog istic ĠReg ression Ġclass ifier Ġwas Ġobserved Ġto Ġproduce Ġthe Ġbest Ġresults Ġfor Ġthis Ġtopical Ġclassification . ĠFinally , Ġwe Ġproposed Ġan ĠIL P - based Ġsummar ization Ġtechnique Ġto Ġproduce Ġa Ġsummary Ġof Ġpeer Ġfeedback Ġcomments Ġfor Ġa Ġgiven Ġemployee Ġand Ġcompared Ġit Ġwith Ġmanual Ġsumm aries . </s> The ĠPA Ġprocess Ġalso Ġgenerates Ġmuch Ġstructured Ġdata , Ġsuch Ġas Ġsupervisor Ġratings . ĠIt Ġis Ġan Ġinteresting Ġproblem Ġto Ġcompare Ġand Ġcombine Ġthe Ġinsights Ġfrom Ġdiscovered Ġfrom Ġstructured Ġdata Ġand Ġun struct ured Ġtext . ĠAlso , Ġwe Ġare Ġplanning Ġto Ġautomatically Ġdiscover Ġany Ġadditional Ġperformance Ġattributes Ġto Ġthe Ġlist Ġof Ġ15 Ġattributes Ġcurrently Ġused Ġby ĠHR .
Introduction Performance appraisal (PA) is an important HR process, particularly for modern organizations that crucially depend on the skills and expertise of their workforce. The PA process enables an organization to periodically measure and evaluate every employee's performance. It also provides a mechanism to link the goals established by the organization to its each employee's day-to-day activities and performance. Design and analysis of PA processes is a lively area of research within the HR community BIBREF0 , BIBREF1 , BIBREF2 , BIBREF3 . The PA process in any modern organization is nowadays implemented and tracked through an IT system (the PA system) that records the interactions that happen in various steps. Availability of this data in a computer-readable database opens up opportunities to analyze it using automated statistical, data-mining and text-mining techniques, to generate novel and actionable insights / patterns and to help in improving the quality and effectiveness of the PA process BIBREF4 , BIBREF5 , BIBREF6 . Automated analysis of large-scale PA data is now facilitated by technological and algorithmic advances, and is becoming essential for large organizations containing thousands of geographically distributed employees handling a wide variety of roles and tasks. A typical PA process involves purposeful multi-step multi-modal communication between employees, their supervisors and their peers. In most PA processes, the communication includes the following steps: (i) in self-appraisal, an employee records his/her achievements, activities, tasks handled etc.; (ii) in supervisor assessment, the supervisor provides the criticism, evaluation and suggestions for improvement of performance etc.; and (iii) in peer feedback (aka INLINEFORM0 view), the peers of the employee provide their feedback. There are several business questions that managers are interested in. Examples: In this paper, we develop text mining techniques that can automatically produce answers to these questions. Since the intended users are HR executives, ideally, the techniques should work with minimum training data and experimentation with parameter setting. These techniques have been implemented and are being used in a PA system in a large multi-national IT company. The rest of the paper is organized as follows. Section SECREF2 summarizes related work. Section SECREF3 summarizes the PA dataset used in this paper. Section SECREF4 applies sentence classification algorithms to automatically discover three important classes of sentences in the PA corpus viz., sentences that discuss strengths, weaknesses of employees and contain suggestions for improving her performance. Section SECREF5 considers the problem of mapping the actual targets mentioned in strengths, weaknesses and suggestions to a fixed set of attributes. In Section SECREF6 , we discuss how the feedback from peers for a particular employee can be summarized. In Section SECREF7 we draw conclusions and identify some further work. Related Work We first review some work related to sentence classification. Semantically classifying sentences (based on the sentence's purpose) is a much harder task, and is gaining increasing attention from linguists and NLP researchers. McKnight and Srinivasan BIBREF7 and Yamamoto and Takagi BIBREF8 used SVM to classify sentences in biomedical abstracts into classes such as INTRODUCTION, BACKGROUND, PURPOSE, METHOD, RESULT, CONCLUSION. Cohen et al. BIBREF9 applied SVM and other techniques to learn classifiers for sentences in emails into classes, which are speech acts defined by a verb-noun pair, with verbs such as request, propose, amend, commit, deliver and nouns such as meeting, document, committee; see also BIBREF10 . Khoo et al. BIBREF11 uses various classifiers to classify sentences in emails into classes such as APOLOGY, INSTRUCTION, QUESTION, REQUEST, SALUTATION, STATEMENT, SUGGESTION, THANKING etc. Qadir and Riloff BIBREF12 proposes several filters and classifiers to classify sentences on message boards (community QA systems) into 4 speech acts: COMMISSIVE (speaker commits to a future action), DIRECTIVE (speaker expects listener to take some action), EXPRESSIVE (speaker expresses his or her psychological state to the listener), REPRESENTATIVE (represents the speaker's belief of something). Hachey and Grover BIBREF13 used SVM and maximum entropy classifiers to classify sentences in legal documents into classes such as FACT, PROCEEDINGS, BACKGROUND, FRAMING, DISPOSAL; see also BIBREF14 . Deshpande et al. BIBREF15 proposes unsupervised linguistic patterns to classify sentences into classes SUGGESTION, COMPLAINT. There is much work on a closely related problem viz., classifying sentences in dialogues through dialogue-specific categories called dialogue acts BIBREF16 , which we will not review here. Just as one example, Cotterill BIBREF17 classifies questions in emails into the dialogue acts of YES_NO_QUESTION, WH_QUESTION, ACTION_REQUEST, RHETORICAL, MULTIPLE_CHOICE etc. We could not find much work related to mining of performance appraisals data. Pawar et al. BIBREF18 uses kernel-based classification to classify sentences in both performance appraisal text and product reviews into classes SUGGESTION, APPRECIATION, COMPLAINT. Apte et al. BIBREF6 provides two algorithms for matching the descriptions of goals or tasks assigned to employees to a standard template of model goals. One algorithm is based on the co-training framework and uses goal descriptions and self-appraisal comments as two separate perspectives. The second approach uses semantic similarity under a weak supervision framework. Ramrakhiyani et al. BIBREF5 proposes label propagation algorithms to discover aspects in supervisor assessments in performance appraisals, where an aspect is modelled as a verb-noun pair (e.g. conduct training, improve coding). Dataset In this paper, we used the supervisor assessment and peer feedback text produced during the performance appraisal of 4528 employees in a large multi-national IT company. The corpus of supervisor assessment has 26972 sentences. The summary statistics about the number of words in a sentence is: min:4 max:217 average:15.5 STDEV:9.2 Q1:9 Q2:14 Q3:19. Sentence Classification The PA corpus contains several classes of sentences that are of interest. In this paper, we focus on three important classes of sentences viz., sentences that discuss strengths (class STRENGTH), weaknesses of employees (class WEAKNESS) and suggestions for improving her performance (class SUGGESTION). The strengths or weaknesses are mostly about the performance in work carried out, but sometimes they can be about the working style or other personal qualities. The classes WEAKNESS and SUGGESTION are somewhat overlapping; e.g., a suggestion may address a perceived weakness. Following are two example sentences in each class. STRENGTH: WEAKNESS: SUGGESTION: Several linguistic aspects of these classes of sentences are apparent. The subject is implicit in many sentences. The strengths are often mentioned as either noun phrases (NP) with positive adjectives (Excellent technology leadership) or positive nouns (engineering strength) or through verbs with positive polarity (dedicated) or as verb phrases containing positive adjectives (delivers innovative solutions). Similarly for weaknesses, where negation is more frequently used (presentations are not his forte), or alternatively, the polarities of verbs (avoid) or adjectives (poor) tend to be negative. However, sometimes the form of both the strengths and weaknesses is the same, typically a stand-alone sentiment-neutral NP, making it difficult to distinguish between them; e.g., adherence to timing or timely closure. Suggestions often have an imperative mood and contain secondary verbs such as need to, should, has to. Suggestions are sometimes expressed using comparatives (better process compliance). We built a simple set of patterns for each of the 3 classes on the POS-tagged form of the sentences. We use each set of these patterns as an unsupervised sentence classifier for that class. If a particular sentence matched with patterns for multiple classes, then we have simple tie-breaking rules for picking the final class. The pattern for the STRENGTH class looks for the presence of positive words / phrases like takes ownership, excellent, hard working, commitment, etc. Similarly, the pattern for the WEAKNESS class looks for the presence of negative words / phrases like lacking, diffident, slow learner, less focused, etc. The SUGGESTION pattern not only looks for keywords like should, needs to but also for POS based pattern like “a verb in the base form (VB) in the beginning of a sentence”. We randomly selected 2000 sentences from the supervisor assessment corpus and manually tagged them (dataset D1). This labelled dataset contained 705, 103, 822 and 370 sentences having the class labels STRENGTH, WEAKNESS, SUGGESTION or OTHER respectively. We trained several multi-class classifiers on this dataset. Table TABREF10 shows the results of 5-fold cross-validation experiments on dataset D1. For the first 5 classifiers, we used their implementation from the SciKit Learn library in Python (scikit-learn.org). The features used for these classifiers were simply the sentence words along with their frequencies. For the last 2 classifiers (in Table TABREF10 ), we used our own implementation. The overall accuracy for a classifier is defined as INLINEFORM0 , where the denominator is 2000 for dataset D1. Note that the pattern-based approach is unsupervised i.e., it did not use any training data. Hence, the results shown for it are for the entire dataset and not based on cross-validation. Comparison with Sentiment Analyzer We also explored whether a sentiment analyzer can be used as a baseline for identifying the class labels STRENGTH and WEAKNESS. We used an implementation of sentiment analyzer from TextBlob to get a polarity score for each sentence. Table TABREF13 shows the distribution of positive, negative and neutral sentiments across the 3 class labels STRENGTH, WEAKNESS and SUGGESTION. It can be observed that distribution of positive and negative sentiments is almost similar in STRENGTH as well as SUGGESTION sentences, hence we can conclude that the information about sentiments is not much useful for our classification problem. Discovering Clusters within Sentence Classes After identifying sentences in each class, we can now answer question (1) in Section SECREF1 . From 12742 sentences predicted to have label STRENGTH, we extract nouns that indicate the actual strength, and cluster them using a simple clustering algorithm which uses the cosine similarity between word embeddings of these nouns. We repeat this for the 9160 sentences with predicted label WEAKNESS or SUGGESTION as a single class. Tables TABREF15 and TABREF16 show a few representative clusters in strengths and in weaknesses, respectively. We also explored clustering 12742 STRENGTH sentences directly using CLUTO BIBREF19 and Carrot2 Lingo BIBREF20 clustering algorithms. Carrot2 Lingo discovered 167 clusters and also assigned labels to these clusters. We then generated 167 clusters using CLUTO as well. CLUTO does not generate cluster labels automatically, hence we used 5 most frequent words within the cluster as its labels. Table TABREF19 shows the largest 5 clusters by both the algorithms. It was observed that the clusters created by CLUTO were more meaningful and informative as compared to those by Carrot2 Lingo. Also, it was observed that there is some correspondence between noun clusters and sentence clusters. E.g. the nouns cluster motivation expertise knowledge talent skill (Table TABREF15 ) corresponds to the CLUTO sentence cluster skill customer management knowledge team (Table TABREF19 ). But overall, users found the nouns clusters to be more meaningful than the sentence clusters. PA along Attributes In many organizations, PA is done from a predefined set of perspectives, which we call attributes. Each attribute covers one specific aspect of the work done by the employees. This has the advantage that we can easily compare the performance of any two employees (or groups of employees) along any given attribute. We can correlate various performance attributes and find dependencies among them. We can also cluster employees in the workforce using their supervisor ratings for each attribute to discover interesting insights into the workforce. The HR managers in the organization considered in this paper have defined 15 attributes (Table TABREF20 ). Each attribute is essentially a work item or work category described at an abstract level. For example, FUNCTIONAL_EXCELLENCE covers any tasks, goals or activities related to the software engineering life-cycle (e.g., requirements analysis, design, coding, testing etc.) as well as technologies such as databases, web services and GUI. In the example in Section SECREF4 , the first sentence (which has class STRENGTH) can be mapped to two attributes: FUNCTIONAL_EXCELLENCE and BUILDING_EFFECTIVE_TEAMS. Similarly, the third sentence (which has class WEAKNESS) can be mapped to the attribute INTERPERSONAL_EFFECTIVENESS and so forth. Thus, in order to answer the second question in Section SECREF1 , we need to map each sentence in each of the 3 classes to zero, one, two or more attributes, which is a multi-class multi-label classification problem. We manually tagged the same 2000 sentences in Dataset D1 with attributes, where each sentence may get 0, 1, 2, etc. up to 15 class labels (this is dataset D2). This labelled dataset contained 749, 206, 289, 207, 91, 223, 191, 144, 103, 80, 82, 42, 29, 15, 24 sentences having the class labels listed in Table TABREF20 in the same order. The number of sentences having 0, 1, 2, or more than 2 attributes are: 321, 1070, 470 and 139 respectively. We trained several multi-class multi-label classifiers on this dataset. Table TABREF21 shows the results of 5-fold cross-validation experiments on dataset D2. Precision, Recall and F-measure for this multi-label classification are computed using a strategy similar to the one described in BIBREF21 . Let INLINEFORM0 be the set of predicted labels and INLINEFORM1 be the set of actual labels for the INLINEFORM2 instance. Precision and recall for this instance are computed as follows: INLINEFORM3 It can be observed that INLINEFORM0 would be undefined if INLINEFORM1 is empty and similarly INLINEFORM2 would be undefined when INLINEFORM3 is empty. Hence, overall precision and recall are computed by averaging over all the instances except where they are undefined. Instance-level F-measure can not be computed for instances where either precision or recall are undefined. Therefore, overall F-measure is computed using the overall precision and recall. Summarization of Peer Feedback using ILP The PA system includes a set of peer feedback comments for each employee. To answer the third question in Section SECREF1 , we need to create a summary of all the peer feedback comments about a given employee. As an example, following are the feedback comments from 5 peers of an employee. The individual sentences in the comments written by each peer are first identified and then POS tags are assigned to each sentence. We hypothesize that a good summary of these multiple comments can be constructed by identifying a set of important text fragments or phrases. Initially, a set of candidate phrases is extracted from these comments and a subset of these candidate phrases is chosen as the final summary, using Integer Linear Programming (ILP). The details of the ILP formulation are shown in Table TABREF36 . As an example, following is the summary generated for the above 5 peer comments. humble nature, effective communication, technical expertise, always supportive, vast knowledge Following rules are used to identify candidate phrases: Various parameters are used to evaluate a candidate phrase for its importance. A candidate phrase is more important: A complete list of parameters is described in detail in Table TABREF36 . There is a trivial constraint INLINEFORM0 which makes sure that only INLINEFORM1 out of INLINEFORM2 candidate phrases are chosen. A suitable value of INLINEFORM3 is used for each employee depending on number of candidate phrases identified across all peers (see Algorithm SECREF6 ). Another set of constraints ( INLINEFORM4 to INLINEFORM5 ) make sure that at least one phrase is selected for each of the leadership attributes. The constraint INLINEFORM6 makes sure that multiple phrases sharing the same headword are not chosen at a time. Also, single word candidate phrases are chosen only if they are adjectives or nouns with lexical category noun.attribute. This is imposed by the constraint INLINEFORM7 . It is important to note that all the constraints except INLINEFORM8 are soft constraints, i.e. there may be feasible solutions which do not satisfy some of these constraints. But each constraint which is not satisfied, results in a penalty through the use of slack variables. These constraints are described in detail in Table TABREF36 . The objective function maximizes the total importance score of the selected candidate phrases. At the same time, it also minimizes the sum of all slack variables so that the minimum number of constraints are broken. INLINEFORM0 : No. of candidate phrases INLINEFORM1 : No. of phrases to select as part of summary INLINEFORM0 INLINEFORM1 INLINEFORM2 INLINEFORM3 INLINEFORM4 INLINEFORM5 INLINEFORM6 INLINEFORM7 INLINEFORM8 INLINEFORM0 and INLINEFORM1 INLINEFORM2 INLINEFORM3 INLINEFORM4 INLINEFORM5 INLINEFORM6 INLINEFORM0 (For determining number of phrases to select to include in summary) Evaluation of auto-generated summaries We considered a dataset of 100 employees, where for each employee multiple peer comments were recorded. Also, for each employee, a manual summary was generated by an HR personnel. The summaries generated by our ILP-based approach were compared with the corresponding manual summaries using the ROUGE BIBREF22 unigram score. For comparing performance of our ILP-based summarization algorithm, we explored a few summarization algorithms provided by the Sumy package. A common parameter which is required by all these algorithms is number of sentences keep in the final summary. ILP-based summarization requires a similar parameter K, which is automatically decided based on number of total candidate phrases. Assuming a sentence is equivalent to roughly 3 phrases, for Sumy algorithms, we set number of sentences parameter to the ceiling of K/3. Table TABREF51 shows average and standard deviation of ROUGE unigram f1 scores for each algorithm, over the 100 summaries. The performance of ILP-based summarization is comparable with the other algorithms, as the two sample t-test does not show statistically significant difference. Also, human evaluators preferred phrase-based summary generated by our approach to the other sentence-based summaries. Conclusions and Further Work In this paper, we presented an analysis of the text generated in Performance Appraisal (PA) process in a large multi-national IT company. We performed sentence classification to identify strengths, weaknesses and suggestions for improvements found in the supervisor assessments and then used clustering to discover broad categories among them. As this is non-topical classification, we found that SVM with ADWS kernel BIBREF18 produced the best results. We also used multi-class multi-label classification techniques to match supervisor assessments to predefined broad perspectives on performance. Logistic Regression classifier was observed to produce the best results for this topical classification. Finally, we proposed an ILP-based summarization technique to produce a summary of peer feedback comments for a given employee and compared it with manual summaries. The PA process also generates much structured data, such as supervisor ratings. It is an interesting problem to compare and combine the insights from discovered from structured data and unstructured text. Also, we are planning to automatically discover any additional performance attributes to the list of 15 attributes currently used by HR.
What summarization algorithms did the authors experiment with?
443d2448136364235389039cbead07e80922ec5c
What Ġsummar ization Ġalgorithms Ġdid Ġthe Ġauthors Ġexperiment Ġwith ?
L SA , ĠText Rank , ĠLex Rank Ġand ĠIL P - based Ġsummary .
<s> What Ġsummar ization Ġalgorithms Ġdid Ġthe Ġauthors Ġexperiment Ġwith ? </s> Introduction </s> Performance Ġappraisal Ġ( PA ) Ġis Ġan Ġimportant ĠHR Ġprocess , Ġparticularly Ġfor Ġmodern Ġorganizations Ġthat Ġcru cially Ġdepend Ġon Ġthe Ġskills Ġand Ġexpertise Ġof Ġtheir Ġworkforce . ĠThe ĠPA Ġprocess Ġenables Ġan Ġorganization Ġto Ġperiodically Ġmeasure Ġand Ġevaluate Ġevery Ġemployee 's Ġperformance . ĠIt Ġalso Ġprovides Ġa Ġmechanism Ġto Ġlink Ġthe Ġgoals Ġestablished Ġby Ġthe Ġorganization Ġto Ġits Ġeach Ġemployee 's Ġday - to - day Ġactivities Ġand Ġperformance . ĠDesign Ġand Ġanalysis Ġof ĠPA Ġprocesses Ġis Ġa Ġlively Ġarea Ġof Ġresearch Ġwithin Ġthe ĠHR Ġcommunity ĠB IB REF 0 Ġ, ĠB IB REF 1 Ġ, ĠB IB REF 2 Ġ, ĠB IB REF 3 Ġ. </s> The ĠPA Ġprocess Ġin Ġany Ġmodern Ġorganization Ġis Ġnowadays Ġimplemented Ġand Ġtracked Ġthrough Ġan ĠIT Ġsystem Ġ( the ĠPA Ġsystem ) Ġthat Ġrecords Ġthe Ġinteractions Ġthat Ġhappen Ġin Ġvarious Ġsteps . ĠAvailability Ġof Ġthis Ġdata Ġin Ġa Ġcomputer - readable Ġdatabase Ġopens Ġup Ġopportunities Ġto Ġanalyze Ġit Ġusing Ġautomated Ġstatistical , Ġdata - mining Ġand Ġtext - mining Ġtechniques , Ġto Ġgenerate Ġnovel Ġand Ġaction able Ġinsights Ġ/ Ġpatterns Ġand Ġto Ġhelp Ġin Ġimproving Ġthe Ġquality Ġand Ġeffectiveness Ġof Ġthe ĠPA Ġprocess ĠB IB REF 4 Ġ, ĠB IB REF 5 Ġ, ĠB IB REF 6 Ġ. ĠAutom ated Ġanalysis Ġof Ġlarge - scale ĠPA Ġdata Ġis Ġnow Ġfacilitated Ġby Ġtechnological Ġand Ġalgorith mic Ġadvances , Ġand Ġis Ġbecoming Ġessential Ġfor Ġlarge Ġorganizations Ġcontaining Ġthousands Ġof Ġgeographically Ġdistributed Ġemployees Ġhandling Ġa Ġwide Ġvariety Ġof Ġroles Ġand Ġtasks . </s> A Ġtypical ĠPA Ġprocess Ġinvolves Ġpurpose ful Ġmulti - step Ġmulti - mod al Ġcommunication Ġbetween Ġemployees , Ġtheir Ġsupervisors Ġand Ġtheir Ġpeers . ĠIn Ġmost ĠPA Ġprocesses , Ġthe Ġcommunication Ġincludes Ġthe Ġfollowing Ġsteps : Ġ( i ) Ġin Ġself - app ra isal , Ġan Ġemployee Ġrecords Ġhis / her Ġachievements , Ġactivities , Ġtasks Ġhandled Ġetc .; Ġ( ii ) Ġin Ġsupervisor Ġassessment , Ġthe Ġsupervisor Ġprovides Ġthe Ġcriticism , Ġevaluation Ġand Ġsuggestions Ġfor Ġimprovement Ġof Ġperformance Ġetc .; Ġand Ġ( iii ) Ġin Ġpeer Ġfeedback Ġ( aka ĠIN LINE FORM 0 Ġview ), Ġthe Ġpeers Ġof Ġthe Ġemployee Ġprovide Ġtheir Ġfeedback . ĠThere Ġare Ġseveral Ġbusiness Ġquestions Ġthat Ġmanagers Ġare Ġinterested Ġin . ĠExamples : </s> In Ġthis Ġpaper , Ġwe Ġdevelop Ġtext Ġmining Ġtechniques Ġthat Ġcan Ġautomatically Ġproduce Ġanswers Ġto Ġthese Ġquestions . ĠSince Ġthe Ġintended Ġusers Ġare ĠHR Ġexecutives , Ġideally , Ġthe Ġtechniques Ġshould Ġwork Ġwith Ġminimum Ġtraining Ġdata Ġand Ġexperimentation Ġwith Ġparameter Ġsetting . ĠThese Ġtechniques Ġhave Ġbeen Ġimplemented Ġand Ġare Ġbeing Ġused Ġin Ġa ĠPA Ġsystem Ġin Ġa Ġlarge Ġmulti - national ĠIT Ġcompany . </s> The Ġrest Ġof Ġthe Ġpaper Ġis Ġorganized Ġas Ġfollows . ĠSection ĠSEC REF 2 Ġsummarizes Ġrelated Ġwork . ĠSection ĠSEC REF 3 Ġsummarizes Ġthe ĠPA Ġdataset Ġused Ġin Ġthis Ġpaper . ĠSection ĠSEC REF 4 Ġapplies Ġsentence Ġclassification Ġalgorithms Ġto Ġautomatically Ġdiscover Ġthree Ġimportant Ġclasses Ġof Ġsentences Ġin Ġthe ĠPA Ġcorpus Ġviz ., Ġsentences Ġthat Ġdiscuss Ġstrengths , Ġweaknesses Ġof Ġemployees Ġand Ġcontain Ġsuggestions Ġfor Ġimproving Ġher Ġperformance . ĠSection ĠSEC REF 5 Ġconsiders Ġthe Ġproblem Ġof Ġmapping Ġthe Ġactual Ġtargets Ġmentioned Ġin Ġstrengths , Ġweaknesses Ġand Ġsuggestions Ġto Ġa Ġfixed Ġset Ġof Ġattributes . ĠIn ĠSection ĠSEC REF 6 Ġ, Ġwe Ġdiscuss Ġhow Ġthe Ġfeedback Ġfrom Ġpeers Ġfor Ġa Ġparticular Ġemployee Ġcan Ġbe Ġsummarized . ĠIn ĠSection ĠSEC REF 7 Ġwe Ġdraw Ġconclusions Ġand Ġidentify Ġsome Ġfurther Ġwork . </s> Related ĠWork </s> We Ġfirst Ġreview Ġsome Ġwork Ġrelated Ġto Ġsentence Ġclassification . ĠSem antically Ġclass ifying Ġsentences Ġ( based Ġon Ġthe Ġsentence 's Ġpurpose ) Ġis Ġa Ġmuch Ġharder Ġtask , Ġand Ġis Ġgaining Ġincreasing Ġattention Ġfrom Ġlingu ists Ġand ĠN LP Ġresearchers . ĠMcK night Ġand ĠS rin ivas an ĠB IB REF 7 Ġand ĠYam amoto Ġand ĠTak agi ĠB IB REF 8 Ġused ĠS VM Ġto Ġclassify Ġsentences Ġin Ġbiomedical Ġabstract s Ġinto Ġclasses Ġsuch Ġas ĠIN TR ODUCT ION , ĠBACK GROUND , ĠPUR POSE , ĠM ETHOD , ĠRES ULT , ĠCON CLUS ION . ĠCohen Ġet Ġal . ĠB IB REF 9 Ġapplied ĠS VM Ġand Ġother Ġtechniques Ġto Ġlearn Ġclass ifiers Ġfor Ġsentences Ġin Ġemails Ġinto Ġclasses , Ġwhich Ġare Ġspeech Ġacts Ġdefined Ġby Ġa Ġverb - n oun Ġpair , Ġwith Ġverbs Ġsuch Ġas Ġrequest , Ġpropose , Ġamend , Ġcommit , Ġdeliver Ġand Ġnoun s Ġsuch Ġas Ġmeeting , Ġdocument , Ġcommittee ; Ġsee Ġalso ĠB IB REF 10 Ġ. ĠKh oo Ġet Ġal . ĠB IB REF 11 Ġuses Ġvarious Ġclass ifiers Ġto Ġclassify Ġsentences Ġin Ġemails Ġinto Ġclasses Ġsuch Ġas ĠAP OLOGY , ĠIN STRUCT ION , ĠQUEST ION , ĠRE QUEST , ĠSAL UT ATION , ĠSTAT EMENT , ĠS UG G EST ION , ĠTHANK ING Ġetc . ĠQ ad ir Ġand ĠR il off ĠB IB REF 12 Ġproposes Ġseveral Ġfilters Ġand Ġclass ifiers Ġto Ġclassify Ġsentences Ġon Ġmessage Ġboards Ġ( community ĠQ A Ġsystems ) Ġinto Ġ4 Ġspeech Ġacts : ĠCOMM ISS IVE Ġ( spe aker Ġcommits Ġto Ġa Ġfuture Ġaction ), ĠDIRECT IVE Ġ( spe aker Ġexpects Ġlistener Ġto Ġtake Ġsome Ġaction ), ĠEX PRESS IVE Ġ( spe aker Ġexpresses Ġhis Ġor Ġher Ġpsychological Ġstate Ġto Ġthe Ġlistener ), ĠRE PRES ENT ATIVE Ġ( rep resents Ġthe Ġspeaker 's Ġbelief Ġof Ġsomething ). ĠH ache y Ġand ĠGro ver ĠB IB REF 13 Ġused ĠS VM Ġand Ġmaximum Ġentropy Ġclass ifiers Ġto Ġclassify Ġsentences Ġin Ġlegal Ġdocuments Ġinto Ġclasses Ġsuch Ġas ĠF ACT , ĠPROC EED INGS , ĠBACK GROUND , ĠFR AM ING , ĠDIS POS AL ; Ġsee Ġalso ĠB IB REF 14 Ġ. ĠDes hp ande Ġet Ġal . ĠB IB REF 15 Ġproposes Ġun super vised Ġlinguistic Ġpatterns Ġto Ġclassify Ġsentences Ġinto Ġclasses ĠS UG G EST ION , ĠCOM PLA INT . </s> There Ġis Ġmuch Ġwork Ġon Ġa Ġclosely Ġrelated Ġproblem Ġviz ., Ġclass ifying Ġsentences Ġin Ġdialog ues Ġthrough Ġdialogue - specific Ġcategories Ġcalled Ġdialogue Ġacts ĠB IB REF 16 Ġ, Ġwhich Ġwe Ġwill Ġnot Ġreview Ġhere . ĠJust Ġas Ġone Ġexample , ĠCot ter ill ĠB IB REF 17 Ġclass ifies Ġquestions Ġin Ġemails Ġinto Ġthe Ġdialogue Ġacts Ġof ĠYES _ NO _ QUEST ION , ĠWH _ QUEST ION , ĠACTION _ RE QUEST , ĠRH ET OR ICAL , ĠM ULT IP LE _ CHO ICE Ġetc . </s> We Ġcould Ġnot Ġfind Ġmuch Ġwork Ġrelated Ġto Ġmining Ġof Ġperformance Ġappra is als Ġdata . ĠPaw ar Ġet Ġal . ĠB IB REF 18 Ġuses Ġkernel - based Ġclassification Ġto Ġclassify Ġsentences Ġin Ġboth Ġperformance Ġappraisal Ġtext Ġand Ġproduct Ġreviews Ġinto Ġclasses ĠS UG G EST ION , ĠAPP RE CI ATION , ĠCOM PLA INT . ĠA pt e Ġet Ġal . ĠB IB REF 6 Ġprovides Ġtwo Ġalgorithms Ġfor Ġmatching Ġthe Ġdescriptions Ġof Ġgoals Ġor Ġtasks Ġassigned Ġto Ġemployees Ġto Ġa Ġstandard Ġtemplate Ġof Ġmodel Ġgoals . ĠOne Ġalgorithm Ġis Ġbased Ġon Ġthe Ġco - training Ġframework Ġand Ġuses Ġgoal Ġdescriptions Ġand Ġself - app ra isal Ġcomments Ġas Ġtwo Ġseparate Ġperspectives . ĠThe Ġsecond Ġapproach Ġuses Ġsemantic Ġsimilarity Ġunder Ġa Ġweak Ġsupervision Ġframework . ĠRam rak hi y ani Ġet Ġal . ĠB IB REF 5 Ġproposes Ġlabel Ġpropagation Ġalgorithms Ġto Ġdiscover Ġaspects Ġin Ġsupervisor Ġassessments Ġin Ġperformance Ġappra is als , Ġwhere Ġan Ġaspect Ġis Ġmod elled Ġas Ġa Ġverb - n oun Ġpair Ġ( e . g . Ġconduct Ġtraining , Ġimprove Ġcoding ). </s> Dat as et </s> In Ġthis Ġpaper , Ġwe Ġused Ġthe Ġsupervisor Ġassessment Ġand Ġpeer Ġfeedback Ġtext Ġproduced Ġduring Ġthe Ġperformance Ġappraisal Ġof Ġ45 28 Ġemployees Ġin Ġa Ġlarge Ġmulti - national ĠIT Ġcompany . ĠThe Ġcorpus Ġof Ġsupervisor Ġassessment Ġhas Ġ269 72 Ġsentences . ĠThe Ġsummary Ġstatistics Ġabout Ġthe Ġnumber Ġof Ġwords Ġin Ġa Ġsentence Ġis : Ġmin : 4 Ġmax : 217 Ġaverage : 15 . 5 ĠST DEV : 9 . 2 ĠQ 1 : 9 ĠQ 2 : 14 ĠQ 3 : 19 . </s> Sent ence ĠClassification </s> The ĠPA Ġcorpus Ġcontains Ġseveral Ġclasses Ġof Ġsentences Ġthat Ġare Ġof Ġinterest . ĠIn Ġthis Ġpaper , Ġwe Ġfocus Ġon Ġthree Ġimportant Ġclasses Ġof Ġsentences Ġviz ., Ġsentences Ġthat Ġdiscuss Ġstrengths Ġ( class ĠSTR ENGTH ), Ġweaknesses Ġof Ġemployees Ġ( class ĠWE AK NESS ) Ġand Ġsuggestions Ġfor Ġimproving Ġher Ġperformance Ġ( class ĠS UG G EST ION ). ĠThe Ġstrengths Ġor Ġweaknesses Ġare Ġmostly Ġabout Ġthe Ġperformance Ġin Ġwork Ġcarried Ġout , Ġbut Ġsometimes Ġthey Ġcan Ġbe Ġabout Ġthe Ġworking Ġstyle Ġor Ġother Ġpersonal Ġqualities . ĠThe Ġclasses ĠWE AK NESS Ġand ĠS UG G EST ION Ġare Ġsomewhat Ġoverlapping ; Ġe . g ., Ġa Ġsuggestion Ġmay Ġaddress Ġa Ġperceived Ġweakness . ĠFollowing Ġare Ġtwo Ġexample Ġsentences Ġin Ġeach Ġclass . </s> STR ENGTH : </s> WE AK NESS : </s> S UG G EST ION : </s> Several Ġlinguistic Ġaspects Ġof Ġthese Ġclasses Ġof Ġsentences Ġare Ġapparent . ĠThe Ġsubject Ġis Ġimplicit Ġin Ġmany Ġsentences . ĠThe Ġstrengths Ġare Ġoften Ġmentioned Ġas Ġeither Ġnoun Ġphrases Ġ( NP ) Ġwith Ġpositive Ġadject ives Ġ( Excellent Ġtechnology Ġleadership ) Ġor Ġpositive Ġnoun s Ġ( engineering Ġstrength ) Ġor Ġthrough Ġverbs Ġwith Ġpositive Ġpol arity Ġ( ded icated ) Ġor Ġas Ġverb Ġphrases Ġcontaining Ġpositive Ġadject ives Ġ( del ivers Ġinnovative Ġsolutions ). ĠSimilarly Ġfor Ġweaknesses , Ġwhere Ġneg ation Ġis Ġmore Ġfrequently Ġused Ġ( present ations Ġare Ġnot Ġhis Ġfor te ), Ġor Ġalternatively , Ġthe Ġpolar ities Ġof Ġverbs Ġ( avoid ) Ġor Ġadject ives Ġ( poor ) Ġtend Ġto Ġbe Ġnegative . ĠHowever , Ġsometimes Ġthe Ġform Ġof Ġboth Ġthe Ġstrengths Ġand Ġweaknesses Ġis Ġthe Ġsame , Ġtypically Ġa Ġstand - alone Ġsentiment - neutral ĠNP , Ġmaking Ġit Ġdifficult Ġto Ġdistinguish Ġbetween Ġthem ; Ġe . g ., Ġadherence Ġto Ġtiming Ġor Ġtimely Ġclosure . ĠSuggest ions Ġoften Ġhave Ġan Ġimperative Ġmood Ġand Ġcontain Ġsecondary Ġverbs Ġsuch Ġas Ġneed Ġto , Ġshould , Ġhas Ġto . ĠSuggest ions Ġare Ġsometimes Ġexpressed Ġusing Ġcompar atives Ġ( better Ġprocess Ġcompliance ). ĠWe Ġbuilt Ġa Ġsimple Ġset Ġof Ġpatterns Ġfor Ġeach Ġof Ġthe Ġ3 Ġclasses Ġon Ġthe ĠPOS - tag ged Ġform Ġof Ġthe Ġsentences . ĠWe Ġuse Ġeach Ġset Ġof Ġthese Ġpatterns Ġas Ġan Ġun super vised Ġsentence Ġclass ifier Ġfor Ġthat Ġclass . ĠIf Ġa Ġparticular Ġsentence Ġmatched Ġwith Ġpatterns Ġfor Ġmultiple Ġclasses , Ġthen Ġwe Ġhave Ġsimple Ġtie - breaking Ġrules Ġfor Ġpicking Ġthe Ġfinal Ġclass . ĠThe Ġpattern Ġfor Ġthe ĠSTR ENGTH Ġclass Ġlooks Ġfor Ġthe Ġpresence Ġof Ġpositive Ġwords Ġ/ Ġphrases Ġlike Ġtakes Ġownership , Ġexcellent , Ġhard Ġworking , Ġcommitment , Ġetc . ĠSimilarly , Ġthe Ġpattern Ġfor Ġthe ĠWE AK NESS Ġclass Ġlooks Ġfor Ġthe Ġpresence Ġof Ġnegative Ġwords Ġ/ Ġphrases Ġlike Ġlacking , Ġdiff ident , Ġslow Ġlear ner , Ġless Ġfocused , Ġetc . ĠThe ĠS UG G EST ION Ġpattern Ġnot Ġonly Ġlooks Ġfor Ġkeywords Ġlike Ġshould , Ġneeds Ġto Ġbut Ġalso Ġfor ĠPOS Ġbased Ġpattern Ġlike ĠâĢ ľ a Ġverb Ġin Ġthe Ġbase Ġform Ġ( VB ) Ġin Ġthe Ġbeginning Ġof Ġa Ġsentence âĢ Ŀ . </s> We Ġrandomly Ġselected Ġ2000 Ġsentences Ġfrom Ġthe Ġsupervisor Ġassessment Ġcorpus Ġand Ġmanually Ġtagged Ġthem Ġ( dat as et ĠD 1 ). ĠThis Ġlabelled Ġdataset Ġcontained Ġ7 05 , Ġ103 , Ġ8 22 Ġand Ġ370 Ġsentences Ġhaving Ġthe Ġclass Ġlabels ĠSTR ENGTH , ĠWE AK NESS , ĠS UG G EST ION Ġor ĠOTHER Ġrespectively . ĠWe Ġtrained Ġseveral Ġmulti - class Ġclass ifiers Ġon Ġthis Ġdataset . ĠTable ĠT AB REF 10 Ġshows Ġthe Ġresults Ġof Ġ5 - fold Ġcross - valid ation Ġexperiments Ġon Ġdataset ĠD 1 . ĠFor Ġthe Ġfirst Ġ5 Ġclass ifiers , Ġwe Ġused Ġtheir Ġimplementation Ġfrom Ġthe ĠSci Kit ĠLearn Ġlibrary Ġin ĠPython Ġ( sci kit - learn . org ). ĠThe Ġfeatures Ġused Ġfor Ġthese Ġclass ifiers Ġwere Ġsimply Ġthe Ġsentence Ġwords Ġalong Ġwith Ġtheir Ġfrequencies . ĠFor Ġthe Ġlast Ġ2 Ġclass ifiers Ġ( in ĠTable ĠT AB REF 10 Ġ), Ġwe Ġused Ġour Ġown Ġimplementation . ĠThe Ġoverall Ġaccuracy Ġfor Ġa Ġclass ifier Ġis Ġdefined Ġas ĠIN LINE FORM 0 Ġ, Ġwhere Ġthe Ġdenomin ator Ġis Ġ2000 Ġfor Ġdataset ĠD 1 . ĠNote Ġthat Ġthe Ġpattern - based Ġapproach Ġis Ġun super vised Ġi . e ., Ġit Ġdid Ġnot Ġuse Ġany Ġtraining Ġdata . ĠHence , Ġthe Ġresults Ġshown Ġfor Ġit Ġare Ġfor Ġthe Ġentire Ġdataset Ġand Ġnot Ġbased Ġon Ġcross - valid ation . </s> Compar ison Ġwith ĠSent iment ĠAnaly zer </s> We Ġalso Ġexplored Ġwhether Ġa Ġsentiment Ġanaly zer Ġcan Ġbe Ġused Ġas Ġa Ġbaseline Ġfor Ġidentifying Ġthe Ġclass Ġlabels ĠSTR ENGTH Ġand ĠWE AK NESS . ĠWe Ġused Ġan Ġimplementation Ġof Ġsentiment Ġanaly zer Ġfrom ĠText Bl ob Ġto Ġget Ġa Ġpol arity Ġscore Ġfor Ġeach Ġsentence . ĠTable ĠT AB REF 13 Ġshows Ġthe Ġdistribution Ġof Ġpositive , Ġnegative Ġand Ġneutral Ġsentiments Ġacross Ġthe Ġ3 Ġclass Ġlabels ĠSTR ENGTH , ĠWE AK NESS Ġand ĠS UG G EST ION . ĠIt Ġcan Ġbe Ġobserved Ġthat Ġdistribution Ġof Ġpositive Ġand Ġnegative Ġsentiments Ġis Ġalmost Ġsimilar Ġin ĠSTR ENGTH Ġas Ġwell Ġas ĠS UG G EST ION Ġsentences , Ġhence Ġwe Ġcan Ġconclude Ġthat Ġthe Ġinformation Ġabout Ġsentiments Ġis Ġnot Ġmuch Ġuseful Ġfor Ġour Ġclassification Ġproblem . </s> Discover ing ĠCl usters Ġwithin ĠSent ence ĠClasses </s> After Ġidentifying Ġsentences Ġin Ġeach Ġclass , Ġwe Ġcan Ġnow Ġanswer Ġquestion Ġ( 1 ) Ġin ĠSection ĠSEC REF 1 Ġ. ĠFrom Ġ127 42 Ġsentences Ġpredicted Ġto Ġhave Ġlabel ĠSTR ENGTH , Ġwe Ġextract Ġnoun s Ġthat Ġindicate Ġthe Ġactual Ġstrength , Ġand Ġcluster Ġthem Ġusing Ġa Ġsimple Ġclust ering Ġalgorithm Ġwhich Ġuses Ġthe Ġcos ine Ġsimilarity Ġbetween Ġword Ġembed d ings Ġof Ġthese Ġnoun s . ĠWe Ġrepeat Ġthis Ġfor Ġthe Ġ9 160 Ġsentences Ġwith Ġpredicted Ġlabel ĠWE AK NESS Ġor ĠS UG G EST ION Ġas Ġa Ġsingle Ġclass . ĠTables ĠT AB REF 15 Ġand ĠT AB REF 16 Ġshow Ġa Ġfew Ġrepresentative Ġclusters Ġin Ġstrengths Ġand Ġin Ġweaknesses , Ġrespectively . ĠWe Ġalso Ġexplored Ġclust ering Ġ127 42 ĠSTR ENGTH Ġsentences Ġdirectly Ġusing ĠCL UT O ĠB IB REF 19 Ġand ĠCar rot 2 ĠLing o ĠB IB REF 20 Ġclust ering Ġalgorithms . ĠCar rot 2 ĠLing o Ġdiscovered Ġ167 Ġclusters Ġand Ġalso Ġassigned Ġlabels Ġto Ġthese Ġclusters . ĠWe Ġthen Ġgenerated Ġ167 Ġclusters Ġusing ĠCL UT O Ġas Ġwell . ĠCL UT O Ġdoes Ġnot Ġgenerate Ġcluster Ġlabels Ġautomatically , Ġhence Ġwe Ġused Ġ5 Ġmost Ġfrequent Ġwords Ġwithin Ġthe Ġcluster Ġas Ġits Ġlabels . ĠTable ĠT AB REF 19 Ġshows Ġthe Ġlargest Ġ5 Ġclusters Ġby Ġboth Ġthe Ġalgorithms . ĠIt Ġwas Ġobserved Ġthat Ġthe Ġclusters Ġcreated Ġby ĠCL UT O Ġwere Ġmore Ġmeaningful Ġand Ġinformative Ġas Ġcompared Ġto Ġthose Ġby ĠCar rot 2 ĠLing o . ĠAlso , Ġit Ġwas Ġobserved Ġthat Ġthere Ġis Ġsome Ġcorrespondence Ġbetween Ġnoun Ġclusters Ġand Ġsentence Ġclusters . ĠE . g . Ġthe Ġnoun s Ġcluster Ġmotivation Ġexpertise Ġknowledge Ġtalent Ġskill Ġ( Table ĠT AB REF 15 Ġ) Ġcorresponds Ġto Ġthe ĠCL UT O Ġsentence Ġcluster Ġskill Ġcustomer Ġmanagement Ġknowledge Ġteam Ġ( Table ĠT AB REF 19 Ġ). ĠBut Ġoverall , Ġusers Ġfound Ġthe Ġnoun s Ġclusters Ġto Ġbe Ġmore Ġmeaningful Ġthan Ġthe Ġsentence Ġclusters . </s> PA Ġalong ĠAttributes </s> In Ġmany Ġorganizations , ĠPA Ġis Ġdone Ġfrom Ġa Ġpred efined Ġset Ġof Ġperspectives , Ġwhich Ġwe Ġcall Ġattributes . ĠEach Ġattribute Ġcovers Ġone Ġspecific Ġaspect Ġof Ġthe Ġwork Ġdone Ġby Ġthe Ġemployees . ĠThis Ġhas Ġthe Ġadvantage Ġthat Ġwe Ġcan Ġeasily Ġcompare Ġthe Ġperformance Ġof Ġany Ġtwo Ġemployees Ġ( or Ġgroups Ġof Ġemployees ) Ġalong Ġany Ġgiven Ġattribute . ĠWe Ġcan Ġcorrelate Ġvarious Ġperformance Ġattributes Ġand Ġfind Ġdependencies Ġamong Ġthem . ĠWe Ġcan Ġalso Ġcluster Ġemployees Ġin Ġthe Ġworkforce Ġusing Ġtheir Ġsupervisor Ġratings Ġfor Ġeach Ġattribute Ġto Ġdiscover Ġinteresting Ġinsights Ġinto Ġthe Ġworkforce . ĠThe ĠHR Ġmanagers Ġin Ġthe Ġorganization Ġconsidered Ġin Ġthis Ġpaper Ġhave Ġdefined Ġ15 Ġattributes Ġ( Table ĠT AB REF 20 Ġ). ĠEach Ġattribute Ġis Ġessentially Ġa Ġwork Ġitem Ġor Ġwork Ġcategory Ġdescribed Ġat Ġan Ġabstract Ġlevel . ĠFor Ġexample , ĠFUN CT ION AL _ EX CE LL ENCE Ġcovers Ġany Ġtasks , Ġgoals Ġor Ġactivities Ġrelated Ġto Ġthe Ġsoftware Ġengineering Ġlife - cycle Ġ( e . g ., Ġrequirements Ġanalysis , Ġdesign , Ġcoding , Ġtesting Ġetc .) Ġas Ġwell Ġas Ġtechnologies Ġsuch Ġas Ġdatabases , Ġweb Ġservices Ġand ĠGUI . </s> In Ġthe Ġexample Ġin ĠSection ĠSEC REF 4 Ġ, Ġthe Ġfirst Ġsentence Ġ( which Ġhas Ġclass ĠSTR ENGTH ) Ġcan Ġbe Ġmapped Ġto Ġtwo Ġattributes : ĠFUN CT ION AL _ EX CE LL ENCE Ġand ĠBU ILD ING _ EFF ECT IVE _ TE AMS . ĠSimilarly , Ġthe Ġthird Ġsentence Ġ( which Ġhas Ġclass ĠWE AK NESS ) Ġcan Ġbe Ġmapped Ġto Ġthe Ġattribute ĠINTER P ERSON AL _ EFF ECT IV EN ESS Ġand Ġso Ġforth . ĠThus , Ġin Ġorder Ġto Ġanswer Ġthe Ġsecond Ġquestion Ġin ĠSection ĠSEC REF 1 Ġ, Ġwe Ġneed Ġto Ġmap Ġeach Ġsentence Ġin Ġeach Ġof Ġthe Ġ3 Ġclasses Ġto Ġzero , Ġone , Ġtwo Ġor Ġmore Ġattributes , Ġwhich Ġis Ġa Ġmulti - class Ġmulti - label Ġclassification Ġproblem . </s> We Ġmanually Ġtagged Ġthe Ġsame Ġ2000 Ġsentences Ġin ĠDat as et ĠD 1 Ġwith Ġattributes , Ġwhere Ġeach Ġsentence Ġmay Ġget Ġ0 , Ġ1 , Ġ2 , Ġetc . Ġup Ġto Ġ15 Ġclass Ġlabels Ġ( this Ġis Ġdataset ĠD 2 ). ĠThis Ġlabelled Ġdataset Ġcontained Ġ7 49 , Ġ206 , Ġ289 , Ġ207 , Ġ91 , Ġ223 , Ġ191 , Ġ144 , Ġ103 , Ġ80 , Ġ82 , Ġ42 , Ġ29 , Ġ15 , Ġ24 Ġsentences Ġhaving Ġthe Ġclass Ġlabels Ġlisted Ġin ĠTable ĠT AB REF 20 Ġin Ġthe Ġsame Ġorder . ĠThe Ġnumber Ġof Ġsentences Ġhaving Ġ0 , Ġ1 , Ġ2 , Ġor Ġmore Ġthan Ġ2 Ġattributes Ġare : Ġ321 , Ġ1070 , Ġ470 Ġand Ġ139 Ġrespectively . ĠWe Ġtrained Ġseveral Ġmulti - class Ġmulti - label Ġclass ifiers Ġon Ġthis Ġdataset . ĠTable ĠT AB REF 21 Ġshows Ġthe Ġresults Ġof Ġ5 - fold Ġcross - valid ation Ġexperiments Ġon Ġdataset ĠD 2 . </s> Pre cision , ĠRecall Ġand ĠF - me asure Ġfor Ġthis Ġmulti - label Ġclassification Ġare Ġcomputed Ġusing Ġa Ġstrategy Ġsimilar Ġto Ġthe Ġone Ġdescribed Ġin ĠB IB REF 21 Ġ. ĠLet ĠIN LINE FORM 0 Ġbe Ġthe Ġset Ġof Ġpredicted Ġlabels Ġand ĠIN LINE FORM 1 Ġbe Ġthe Ġset Ġof Ġactual Ġlabels Ġfor Ġthe ĠIN LINE FORM 2 Ġinstance . ĠPrecision Ġand Ġrecall Ġfor Ġthis Ġinstance Ġare Ġcomputed Ġas Ġfollows : ĠIN LINE FORM 3 </s> It Ġcan Ġbe Ġobserved Ġthat ĠIN LINE FORM 0 Ġwould Ġbe Ġundefined Ġif ĠIN LINE FORM 1 Ġis Ġempty Ġand Ġsimilarly ĠIN LINE FORM 2 Ġwould Ġbe Ġundefined Ġwhen ĠIN LINE FORM 3 Ġis Ġempty . ĠHence , Ġoverall Ġprecision Ġand Ġrecall Ġare Ġcomputed Ġby Ġaveraging Ġover Ġall Ġthe Ġinstances Ġexcept Ġwhere Ġthey Ġare Ġundefined . ĠInst ance - level ĠF - me asure Ġcan Ġnot Ġbe Ġcomputed Ġfor Ġinstances Ġwhere Ġeither Ġprecision Ġor Ġrecall Ġare Ġundefined . ĠTherefore , Ġoverall ĠF - me asure Ġis Ġcomputed Ġusing Ġthe Ġoverall Ġprecision Ġand Ġrecall . </s> Sum mar ization Ġof ĠPeer ĠFeedback Ġusing ĠIL P </s> The ĠPA Ġsystem Ġincludes Ġa Ġset Ġof Ġpeer Ġfeedback Ġcomments Ġfor Ġeach Ġemployee . ĠTo Ġanswer Ġthe Ġthird Ġquestion Ġin ĠSection ĠSEC REF 1 Ġ, Ġwe Ġneed Ġto Ġcreate Ġa Ġsummary Ġof Ġall Ġthe Ġpeer Ġfeedback Ġcomments Ġabout Ġa Ġgiven Ġemployee . ĠAs Ġan Ġexample , Ġfollowing Ġare Ġthe Ġfeedback Ġcomments Ġfrom Ġ5 Ġpeers Ġof Ġan Ġemployee . </s> The Ġindividual Ġsentences Ġin Ġthe Ġcomments Ġwritten Ġby Ġeach Ġpeer Ġare Ġfirst Ġidentified Ġand Ġthen ĠPOS Ġtags Ġare Ġassigned Ġto Ġeach Ġsentence . ĠWe Ġhypothes ize Ġthat Ġa Ġgood Ġsummary Ġof Ġthese Ġmultiple Ġcomments Ġcan Ġbe Ġconstructed Ġby Ġidentifying Ġa Ġset Ġof Ġimportant Ġtext Ġfragments Ġor Ġphrases . ĠInitially , Ġa Ġset Ġof Ġcandidate Ġphrases Ġis Ġextracted Ġfrom Ġthese Ġcomments Ġand Ġa Ġsubset Ġof Ġthese Ġcandidate Ġphrases Ġis Ġchosen Ġas Ġthe Ġfinal Ġsummary , Ġusing ĠInteger ĠLinear ĠProgramming Ġ( IL P ). ĠThe Ġdetails Ġof Ġthe ĠIL P Ġformulation Ġare Ġshown Ġin ĠTable ĠT AB REF 36 Ġ. ĠAs Ġan Ġexample , Ġfollowing Ġis Ġthe Ġsummary Ġgenerated Ġfor Ġthe Ġabove Ġ5 Ġpeer Ġcomments . </s> h umble Ġnature , Ġeffective Ġcommunication , Ġtechnical Ġexpertise , Ġalways Ġsupportive , Ġvast Ġknowledge </s> Following Ġrules Ġare Ġused Ġto Ġidentify Ġcandidate Ġphrases : </s> Various Ġparameters Ġare Ġused Ġto Ġevaluate Ġa Ġcandidate Ġphrase Ġfor Ġits Ġimportance . ĠA Ġcandidate Ġphrase Ġis Ġmore Ġimportant : </s> A Ġcomplete Ġlist Ġof Ġparameters Ġis Ġdescribed Ġin Ġdetail Ġin ĠTable ĠT AB REF 36 Ġ. </s> There Ġis Ġa Ġtrivial Ġconstraint ĠIN LINE FORM 0 Ġwhich Ġmakes Ġsure Ġthat Ġonly ĠIN LINE FORM 1 Ġout Ġof ĠIN LINE FORM 2 Ġcandidate Ġphrases Ġare Ġchosen . ĠA Ġsuitable Ġvalue Ġof ĠIN LINE FORM 3 Ġis Ġused Ġfor Ġeach Ġemployee Ġdepending Ġon Ġnumber Ġof Ġcandidate Ġphrases Ġidentified Ġacross Ġall Ġpeers Ġ( see ĠAl gorithm ĠSEC REF 6 Ġ). ĠAnother Ġset Ġof Ġconstraints Ġ( ĠIN LINE FORM 4 Ġto ĠIN LINE FORM 5 Ġ) Ġmake Ġsure Ġthat Ġat Ġleast Ġone Ġphrase Ġis Ġselected Ġfor Ġeach Ġof Ġthe Ġleadership Ġattributes . ĠThe Ġconstraint ĠIN LINE FORM 6 Ġmakes Ġsure Ġthat Ġmultiple Ġphrases Ġsharing Ġthe Ġsame Ġhead word Ġare Ġnot Ġchosen Ġat Ġa Ġtime . ĠAlso , Ġsingle Ġword Ġcandidate Ġphrases Ġare Ġchosen Ġonly Ġif Ġthey Ġare Ġadject ives Ġor Ġnoun s Ġwith Ġlex ical Ġcategory Ġnoun . attribute . ĠThis Ġis Ġimposed Ġby Ġthe Ġconstraint ĠIN LINE FORM 7 Ġ. ĠIt Ġis Ġimportant Ġto Ġnote Ġthat Ġall Ġthe Ġconstraints Ġexcept ĠIN LINE FORM 8 Ġare Ġsoft Ġconstraints , Ġi . e . Ġthere Ġmay Ġbe Ġfeasible Ġsolutions Ġwhich Ġdo Ġnot Ġsatisfy Ġsome Ġof Ġthese Ġconstraints . ĠBut Ġeach Ġconstraint Ġwhich Ġis Ġnot Ġsatisfied , Ġresults Ġin Ġa Ġpenalty Ġthrough Ġthe Ġuse Ġof Ġslack Ġvariables . ĠThese Ġconstraints Ġare Ġdescribed Ġin Ġdetail Ġin ĠTable ĠT AB REF 36 Ġ. </s> The Ġobjective Ġfunction Ġmaxim izes Ġthe Ġtotal Ġimportance Ġscore Ġof Ġthe Ġselected Ġcandidate Ġphrases . ĠAt Ġthe Ġsame Ġtime , Ġit Ġalso Ġminim izes Ġthe Ġsum Ġof Ġall Ġslack Ġvariables Ġso Ġthat Ġthe Ġminimum Ġnumber Ġof Ġconstraints Ġare Ġbroken . </s> IN LINE FORM 0 Ġ: ĠNo . Ġof Ġcandidate Ġphrases ĠIN LINE FORM 1 Ġ: ĠNo . Ġof Ġphrases Ġto Ġselect Ġas Ġpart Ġof Ġsummary </s> IN LINE FORM 0 ĠIN LINE FORM 1 ĠIN LINE FORM 2 ĠIN LINE FORM 3 ĠIN LINE FORM 4 ĠIN LINE FORM 5 ĠIN LINE FORM 6 ĠIN LINE FORM 7 ĠIN LINE FORM 8 </s> IN LINE FORM 0 Ġand ĠIN LINE FORM 1 ĠIN LINE FORM 2 ĠIN LINE FORM 3 ĠIN LINE FORM 4 ĠIN LINE FORM 5 ĠIN LINE FORM 6 </s> IN LINE FORM 0 Ġ( For Ġdetermining Ġnumber Ġof Ġphrases Ġto Ġselect Ġto Ġinclude Ġin Ġsummary ) </s> E val uation Ġof Ġauto - generated Ġsumm aries </s> We Ġconsidered Ġa Ġdataset Ġof Ġ100 Ġemployees , Ġwhere Ġfor Ġeach Ġemployee Ġmultiple Ġpeer Ġcomments Ġwere Ġrecorded . ĠAlso , Ġfor Ġeach Ġemployee , Ġa Ġmanual Ġsummary Ġwas Ġgenerated Ġby Ġan ĠHR Ġpersonnel . ĠThe Ġsumm aries Ġgenerated Ġby Ġour ĠIL P - based Ġapproach Ġwere Ġcompared Ġwith Ġthe Ġcorresponding Ġmanual Ġsumm aries Ġusing Ġthe ĠR OU GE ĠB IB REF 22 Ġun ig ram Ġscore . ĠFor Ġcomparing Ġperformance Ġof Ġour ĠIL P - based Ġsummar ization Ġalgorithm , Ġwe Ġexplored Ġa Ġfew Ġsummar ization Ġalgorithms Ġprovided Ġby Ġthe ĠSum y Ġpackage . ĠA Ġcommon Ġparameter Ġwhich Ġis Ġrequired Ġby Ġall Ġthese Ġalgorithms Ġis Ġnumber Ġof Ġsentences Ġkeep Ġin Ġthe Ġfinal Ġsummary . ĠIL P - based Ġsummar ization Ġrequires Ġa Ġsimilar Ġparameter ĠK , Ġwhich Ġis Ġautomatically Ġdecided Ġbased Ġon Ġnumber Ġof Ġtotal Ġcandidate Ġphrases . ĠAssuming Ġa Ġsentence Ġis Ġequivalent Ġto Ġroughly Ġ3 Ġphrases , Ġfor ĠSum y Ġalgorithms , Ġwe Ġset Ġnumber Ġof Ġsentences Ġparameter Ġto Ġthe Ġceiling Ġof ĠK / 3 . ĠTable ĠT AB REF 51 Ġshows Ġaverage Ġand Ġstandard Ġdeviation Ġof ĠR OU GE Ġun ig ram Ġf 1 Ġscores Ġfor Ġeach Ġalgorithm , Ġover Ġthe Ġ100 Ġsumm aries . ĠThe Ġperformance Ġof ĠIL P - based Ġsummar ization Ġis Ġcomparable Ġwith Ġthe Ġother Ġalgorithms , Ġas Ġthe Ġtwo Ġsample Ġt - test Ġdoes Ġnot Ġshow Ġstatistically Ġsignificant Ġdifference . ĠAlso , Ġhuman Ġeval u ators Ġpreferred Ġphrase - based Ġsummary Ġgenerated Ġby Ġour Ġapproach Ġto Ġthe Ġother Ġsentence - based Ġsumm aries . </s> Con clusions Ġand ĠFurther ĠWork </s> In Ġthis Ġpaper , Ġwe Ġpresented Ġan Ġanalysis Ġof Ġthe Ġtext Ġgenerated Ġin ĠPerformance ĠApp ra isal Ġ( PA ) Ġprocess Ġin Ġa Ġlarge Ġmulti - national ĠIT Ġcompany . ĠWe Ġperformed Ġsentence Ġclassification Ġto Ġidentify Ġstrengths , Ġweaknesses Ġand Ġsuggestions Ġfor Ġimprovements Ġfound Ġin Ġthe Ġsupervisor Ġassessments Ġand Ġthen Ġused Ġclust ering Ġto Ġdiscover Ġbroad Ġcategories Ġamong Ġthem . ĠAs Ġthis Ġis Ġnon - top ical Ġclassification , Ġwe Ġfound Ġthat ĠS VM Ġwith ĠAD WS Ġkernel ĠB IB REF 18 Ġproduced Ġthe Ġbest Ġresults . ĠWe Ġalso Ġused Ġmulti - class Ġmulti - label Ġclassification Ġtechniques Ġto Ġmatch Ġsupervisor Ġassessments Ġto Ġpred efined Ġbroad Ġperspectives Ġon Ġperformance . ĠLog istic ĠReg ression Ġclass ifier Ġwas Ġobserved Ġto Ġproduce Ġthe Ġbest Ġresults Ġfor Ġthis Ġtopical Ġclassification . ĠFinally , Ġwe Ġproposed Ġan ĠIL P - based Ġsummar ization Ġtechnique Ġto Ġproduce Ġa Ġsummary Ġof Ġpeer Ġfeedback Ġcomments Ġfor Ġa Ġgiven Ġemployee Ġand Ġcompared Ġit Ġwith Ġmanual Ġsumm aries . </s> The ĠPA Ġprocess Ġalso Ġgenerates Ġmuch Ġstructured Ġdata , Ġsuch Ġas Ġsupervisor Ġratings . ĠIt Ġis Ġan Ġinteresting Ġproblem Ġto Ġcompare Ġand Ġcombine Ġthe Ġinsights Ġfrom Ġdiscovered Ġfrom Ġstructured Ġdata Ġand Ġun struct ured Ġtext . ĠAlso , Ġwe Ġare Ġplanning Ġto Ġautomatically Ġdiscover Ġany Ġadditional Ġperformance Ġattributes Ġto Ġthe Ġlist Ġof Ġ15 Ġattributes Ġcurrently Ġused Ġby ĠHR .