$19,300$, added 2500 randomly sampled tweets
EXTRACTIVE
$19,300$ tweets
EXTRACTIVE
We aimed to mimic the presence of sparse/noisy content distribution, mandating the need to curate a novel dataset via specific lexicons. We scraped 500 random posts from recognized transport forum. A pool of 50 uni/bi-grams was created based on tf-idf representations, extracted from the posts, which was further pruned by annotators. Querying posts on Twitter with extracted lexicons led to a collection of $19,300$ tweets. In order to have lexical diversity, we added 2500 randomly sampled tweets to our dataset. In spite of the sparse nature of these posts, the lexical characteristics act as information cues.
We aimed to mimic the presence of sparse/noisy content distribution, mandating the need to curate a novel dataset via specific lexicons. We scraped 500 random posts from recognized transport forum. A pool of 50 uni/bi-grams was created based on tf-idf representations, extracted from the posts, which was further pruned by annotators. Querying posts on Twitter with extracted lexicons led to a collection of $19,300$ tweets. In order to have lexical diversity, we added 2500 randomly sampled tweets to our dataset. In spite of the sparse nature of these posts, the lexical characteristics act as information cues.
2001.09215
Introduction </s> With Ġthe Ġadvent Ġof Ġsocial Ġmedia Ġplatforms , Ġincreasing Ġuser Ġbase Ġaddress Ġtheir Ġgrievances Ġover Ġthese Ġplatforms , Ġin Ġthe Ġform Ġof Ġcomplaints . ĠAccording Ġto ĠB IB REF 0 , Ġcomplaint Ġis Ġconsidered Ġto Ġbe Ġa Ġbasic Ġspeech Ġact Ġused Ġto Ġexpress Ġnegative Ġmismatch Ġbetween Ġthe Ġexpectation Ġand Ġreality . ĠTransportation Ġand Ġits Ġrelated Ġlogistics Ġindustries Ġare Ġthe Ġback bones Ġof Ġevery Ġeconomy . ĠMany Ġtransport Ġorganizations Ġrely Ġon Ġcomplaints Ġgathered Ġvia Ġthese Ġplatforms Ġto Ġimprove Ġtheir Ġservices , Ġhence Ġunderstanding Ġthese Ġare Ġimportant Ġfor : Ġ( 1 ) Ġlingu ists Ġto Ġidentify Ġhuman Ġexpressions Ġof Ġcriticism Ġand Ġ( 2 ) Ġorganizations Ġto Ġimprove Ġtheir Ġquery Ġresponse Ġtime Ġand Ġaddress Ġconcerns Ġeffectively . </s> Pres ence Ġof Ġinevitable Ġnoise , Ġsparse Ġcontent Ġalong Ġwith Ġre ph r ased Ġand Ġstruct urally Ġmorphed Ġinstances Ġof Ġposts , Ġmake Ġthe Ġtask Ġat Ġhand Ġdifficult ĠB IB REF 1 . ĠPrevious Ġworks ĠB IB REF 2 Ġin Ġthe Ġdomain Ġof Ġcomplaint Ġextraction Ġhave Ġfocused Ġon Ġstatic Ġdatasets Ġonly . ĠThese Ġare Ġnot Ġrobust Ġto Ġchanges Ġin Ġthe Ġtrends Ġreflected , Ġinformation Ġflow Ġand Ġlinguistic Ġvariations . ĠWe Ġpropose Ġan Ġiter ative , Ġsemi - super vised Ġapproach Ġfor Ġidentification Ġof Ġcomplaint Ġbased Ġtweets , Ġhaving Ġthe Ġability Ġto Ġbe Ġreplicated Ġfor Ġstream Ġof Ġinformation Ġflow . ĠThe Ġpreference Ġof Ġa Ġsemi - super vised Ġapproach Ġover Ġsupervised Ġones Ġis Ġdue Ġto Ġthe Ġstated Ġreasons : Ġ( a ) Ġthe Ġtask Ġof Ġisol ating Ġthe Ġtraining Ġset , Ġmake Ġsupervised Ġtasks Ġless Ġattractive Ġand Ġimpractical Ġand Ġ( b ) Ġimbalance Ġbetween Ġthe Ġsubjective Ġand Ġobjective Ġclasses Ġlead Ġto Ġpoor Ġperformance . </s> Prop osed ĠMethod ology </s> We Ġaimed Ġto Ġmimic Ġthe Ġpresence Ġof Ġsparse / no isy Ġcontent Ġdistribution , Ġmand ating Ġthe Ġneed Ġto Ġcur ate Ġa Ġnovel Ġdataset Ġvia Ġspecific Ġlex icons . ĠWe Ġscrap ed Ġ500 Ġrandom Ġposts Ġfrom Ġrecognized Ġtransport Ġforum . ĠA Ġpool Ġof Ġ50 Ġun i / bi - gram s Ġwas Ġcreated Ġbased Ġon Ġtf - id f Ġrepresentations , Ġextracted Ġfrom Ġthe Ġposts , Ġwhich Ġwas Ġfurther Ġpr uned Ġby Ġannot ators . ĠQu er ying Ġposts Ġon ĠTwitter Ġwith Ġextracted Ġlex icons Ġled Ġto Ġa Ġcollection Ġof Ġ$ 19 , 300 $ Ġtweets . ĠIn Ġorder Ġto Ġhave Ġlex ical Ġdiversity , Ġwe Ġadded Ġ2500 Ġrandomly Ġsampled Ġtweets Ġto Ġour Ġdataset . ĠIn Ġspite Ġof Ġthe Ġsparse Ġnature Ġof Ġthese Ġposts , Ġthe Ġlex ical Ġcharacteristics Ġact Ġas Ġinformation Ġcues . </s> Figure ĠFIG REF 4 Ġpict or ially Ġrepresents Ġour Ġmethodology . ĠOur Ġapproach Ġrequired Ġan Ġinitial Ġset Ġof Ġinformative Ġtweets Ġfor Ġwhich Ġwe Ġemployed Ġtwo Ġhuman Ġannot ators Ġannot ating Ġa Ġrandom Ġsub - sample Ġof Ġthe Ġoriginal Ġdataset . ĠFrom Ġthe Ġ1500 Ġsamples , Ġ326 Ġwere Ġmarked Ġas Ġinformative Ġand Ġ11 74 Ġas Ġnon Ġinformative Ġ($ \ k appa Ġ= 0 . 81 $ ), Ġdiscriminated Ġon Ġthis Ġcriteria : ĠIs Ġthe Ġtweet Ġaddressing Ġany Ġcomplaint Ġor Ġraising Ġgrievances Ġabout Ġmodes Ġof Ġtransport Ġor Ġservices / Ġevents Ġassociated Ġwith Ġtransportation Ġsuch Ġas Ġtraffic ; Ġpublic Ġor Ġprivate Ġtransport ? . ĠAn Ġexample Ġtweet Ġmarked Ġas Ġinformative : ĠNo , Ġmetro Ġfares Ġwill Ġbe Ġreduced Ġ?? ?, Ġbut Ġproper Ġfare Ġstructure Ġneeds Ġto Ġpresented Ġright , Ġit 's Ġbad Ġ !!! . </s> We Ġutilized Ġtf - id f Ġfor Ġthe Ġidentification Ġof Ġinitial Ġseed Ġphrases Ġfrom Ġthe Ġcurated Ġset Ġof Ġinformative Ġtweets . Ġ50 Ġterms Ġhaving Ġthe Ġhighest Ġtf - id f Ġscores Ġwere Ġpassed Ġthrough Ġthe Ġcomplete Ġdataset Ġand Ġbased Ġon Ġsub - string Ġmatch , Ġthe Ġtransport Ġrelevant Ġtweets Ġwere Ġidentified . ĠThe Ġredundant Ġtweets Ġwere Ġfiltered Ġbased Ġon Ġthe Ġcos ine Ġsimilarity Ġscore . ĠImpl icit Ġinformation Ġindicators Ġwere Ġidentified Ġbased Ġon Ġdomain Ġrelevance Ġscore , Ġa Ġmetric Ġused Ġto Ġgauge Ġthe Ġcoverage Ġof Ġn - gram Ġ( 1 , 2 , 3 ) Ġwhen Ġevaluated Ġagainst Ġa Ġrandomly Ġcreated Ġpool Ġof Ġposts . </s> We Ġcollected Ġa Ġpool Ġof Ġ5000 Ġrandomly Ġsampled Ġtweets Ġdifferent Ġfrom Ġthe Ġdata Ġcollection Ġperiod . ĠThe Ġrationale Ġbehind Ġhaving Ġsuch Ġa Ġmetric Ġwas Ġto Ġdiscard Ġcommonly Ġoccurring Ġn - gram s Ġnormalized Ġby Ġrandom Ġnoise Ġand Ġinclude Ġones Ġwhich Ġare Ġof Ġlex ical Ġimportance . ĠWe Ġused Ġterms Ġassociated Ġwith Ġhigh Ġdomain Ġrelevance Ġscore Ġ( th reshold Ġdetermined Ġexperiment ally ) Ġas Ġseed Ġphrases Ġfor Ġthe Ġnext Ġset Ġof Ġiterations . ĠThe Ġgrowing Ġdictionary Ġaug ments Ġthe Ġcollection Ġprocess . ĠThe Ġprocess Ġran Ġfor Ġ4 Ġiterations Ġproviding Ġus Ġ7 200 Ġtransport Ġrelevant Ġtweets Ġas Ġno Ġnew Ġlex icons Ġwere Ġidentified . ĠIn Ġorder Ġto Ġidentify Ġlinguistic Ġsignals Ġassociated Ġwith Ġthe Ġcomplaint Ġposts , Ġwe Ġrandomly Ġsampled Ġa Ġset Ġof Ġ2000 Ġtweets Ġwhich Ġwas Ġused Ġas Ġtraining Ġset , Ġmanually Ġannot ated Ġinto Ġdistinct Ġlabels : Ġcomplaint Ġrelevant Ġ( 702 ) Ġand Ġcomplaint Ġnon - relevant Ġ( 12 98 ) Ġ($ \ k appa Ġ= 0 . 79 $ ). ĠWe Ġemployed Ġthese Ġfeatures Ġon Ġour Ġdataset . </s> L ingu istic Ġmarkers . ĠTo Ġcapture Ġlinguistic Ġaspects Ġof Ġcomplaints , Ġwe Ġutilized ĠBag Ġof ĠWords , Ġcount Ġof ĠPOS Ġtags Ġand ĠWord 2 vec Ġclusters . </s> Sent iment Ġmarkers . ĠWe Ġused Ġquant ified Ġscore Ġbased Ġon Ġthe Ġratio Ġof Ġtokens Ġmentioned Ġin Ġthe Ġfollowing Ġlex icons : ĠMP Q A , ĠN RC , ĠV AD ER Ġand ĠStanford . </s> Information Ġspecific Ġmarkers . ĠThese Ġaccount Ġfor Ġa Ġset Ġof Ġhand crafted Ġfeatures Ġassociated Ġwith Ġcomplaint , Ġwe Ġused Ġthe Ġstated Ġmarkers Ġ( a ) ĠText - Meta ĠData , Ġthis Ġincludes Ġthe Ġcount Ġof ĠURL 's , Ġhasht ags , Ġuser Ġmentions , Ġspecial Ġsymbols Ġand Ġuser Ġmentions , Ġused Ġto Ġenhance Ġretweet Ġimpact ; Ġ( b ) ĠRequest ĠIdentification , Ġwe Ġemployed Ġthe Ġmodel Ġpresented Ġin ĠB IB REF 3 Ġto Ġidentify Ġif Ġa Ġspecific Ġtweet Ġassertion Ġis Ġa Ġrequest ; Ġ( c ) ĠInt ens ifiers , Ġwe Ġmake Ġuse Ġof Ġfeature Ġset Ġderived Ġfrom Ġthe Ġnumber Ġof Ġwords Ġstarting Ġwith Ġcapital Ġletters Ġand Ġthe Ġrepetition Ġof Ġspecial Ġsymbols Ġ( ex clamation , Ġquestions Ġmarks ) Ġwithin Ġthe Ġsame Ġpost ; Ġ( d ) ĠPolit eness ĠMark ers , Ġwe Ġutilize Ġthe Ġpolit eness Ġscore Ġof Ġthe Ġtweet Ġextracted Ġfrom Ġthe Ġmodel Ġpresented Ġin ĠB IB REF 3 ; Ġ( e ) ĠP ron oun ĠVari ation , Ġthese Ġhave Ġthe Ġability Ġto Ġreveal Ġthe Ġpersonal Ġinvolvement Ġor Ġintensify Ġinvolvement . ĠWe Ġutilize Ġthe Ġfrequency Ġof Ġpronoun Ġtypes Ġ$\ l brace Ġ\ text it Ġ{ first , Ġsecond , Ġthird , Ġdemonstr ative Ġand Ġindefinite } $ } Ġusing Ġpre - defined Ġdiction aries . </s> From Ġthe Ġpool Ġof Ġ7 200 Ġtransport Ġrelevant Ġtweets , Ġwe Ġsampled Ġ35 00 Ġtweets Ġwhich Ġwere Ġused Ġas Ġthe Ġtesting Ġset . ĠThe Ġresults Ġare Ġreported Ġin ĠTable TA B REF 5 Ġwith Ġ10 Ġfold Ġcross - valid ation . ĠWith Ġincreasing Ġthe Ġnumber Ġof Ġiterations , Ġthe Ġpool Ġof Ġseed Ġphrases Ġgets Ġrefined Ġand Ġaug ments Ġthe Ġselection Ġof Ġtransport Ġrelevant Ġtweets . ĠThe Ġproposed Ġpipeline Ġis Ġtailored Ġto Ġidentify Ġcomplaint Ġrelevant Ġtweets Ġin Ġa Ġnoisy Ġscenario . </s> Results </s> Table ĠT AB REF 5 Ġreflects Ġthat Ġthe ĠB OW Ġmodel Ġprovided Ġthe Ġbest Ġresults , Ġboth Ġin Ġterms Ġof Ġaccuracy Ġand ĠF 1 - score . ĠThe Ġbest Ġresult Ġachieved Ġby Ġa Ġsentiment Ġmodel Ġwas Ġthe ĠStanford ĠSent iment Ġ($ 0 . 63 $ ĠF 1 - score ), Ġwith Ġothers Ġwithin Ġthe Ġsame Ġrange Ġand Ġlinguistic - based Ġfeatures Ġcollectively Ġgiving Ġthe Ġbest Ġperformance . </s> Conclusion Ġand ĠFuture ĠWork </s> In Ġthis Ġpaper , Ġwe Ġpresented Ġa Ġnovel Ġsemi - super vised Ġpipeline Ġalong Ġwith Ġa Ġnovel Ġdataset Ġfor Ġidentification Ġof Ġcomplaint Ġbased Ġposts Ġin Ġthe Ġtransport Ġdomain . ĠThe Ġproposed Ġmethodology Ġcan Ġbe Ġexpanded Ġfor Ġother Ġfields Ġby Ġaltering Ġthe Ġlex icons Ġused Ġfor Ġthe Ġcreation Ġof Ġinformation Ġcues . ĠThere Ġare Ġlimitations Ġto Ġthis Ġanalysis ; Ġwe Ġdo Ġnot Ġuse Ġneural Ġnetworks Ġwhich Ġmandate Ġa Ġlarge Ġvolume Ġof Ġdata . ĠIn Ġthe Ġfuture , Ġwe Ġaim Ġto Ġidentify Ġdemographic Ġfeatures Ġfor Ġidentification Ġof Ġcomplaint Ġbased Ġposts Ġon Ġsocial Ġmedia Ġplatforms .
Introduction With the advent of social media platforms, increasing user base address their grievances over these platforms, in the form of complaints. According to BIBREF0, complaint is considered to be a basic speech act used to express negative mismatch between the expectation and reality. Transportation and its related logistics industries are the backbones of every economy. Many transport organizations rely on complaints gathered via these platforms to improve their services, hence understanding these are important for: (1) linguists to identify human expressions of criticism and (2) organizations to improve their query response time and address concerns effectively. Presence of inevitable noise, sparse content along with rephrased and structurally morphed instances of posts, make the task at hand difficult BIBREF1. Previous works BIBREF2 in the domain of complaint extraction have focused on static datasets only. These are not robust to changes in the trends reflected, information flow and linguistic variations. We propose an iterative, semi-supervised approach for identification of complaint based tweets, having the ability to be replicated for stream of information flow. The preference of a semi-supervised approach over supervised ones is due to the stated reasons: (a) the task of isolating the training set, make supervised tasks less attractive and impractical and (b) imbalance between the subjective and objective classes lead to poor performance. Proposed Methodology We aimed to mimic the presence of sparse/noisy content distribution, mandating the need to curate a novel dataset via specific lexicons. We scraped 500 random posts from recognized transport forum. A pool of 50 uni/bi-grams was created based on tf-idf representations, extracted from the posts, which was further pruned by annotators. Querying posts on Twitter with extracted lexicons led to a collection of $19,300$ tweets. In order to have lexical diversity, we added 2500 randomly sampled tweets to our dataset. In spite of the sparse nature of these posts, the lexical characteristics act as information cues. Figure FIGREF4 pictorially represents our methodology. Our approach required an initial set of informative tweets for which we employed two human annotators annotating a random sub-sample of the original dataset. From the 1500 samples, 326 were marked as informative and 1174 as non informative ($\kappa =0.81$), discriminated on this criteria: Is the tweet addressing any complaint or raising grievances about modes of transport or services/ events associated with transportation such as traffic; public or private transport?. An example tweet marked as informative: No, metro fares will be reduced ???, but proper fare structure needs to presented right, it's bad !!!. We utilized tf-idf for the identification of initial seed phrases from the curated set of informative tweets. 50 terms having the highest tf-idf scores were passed through the complete dataset and based on sub-string match, the transport relevant tweets were identified. The redundant tweets were filtered based on the cosine similarity score. Implicit information indicators were identified based on domain relevance score, a metric used to gauge the coverage of n-gram (1,2,3) when evaluated against a randomly created pool of posts. We collected a pool of 5000 randomly sampled tweets different from the data collection period. The rationale behind having such a metric was to discard commonly occurring n-grams normalized by random noise and include ones which are of lexical importance. We used terms associated with high domain relevance score (threshold determined experimentally) as seed phrases for the next set of iterations. The growing dictionary augments the collection process. The process ran for 4 iterations providing us 7200 transport relevant tweets as no new lexicons were identified. In order to identify linguistic signals associated with the complaint posts, we randomly sampled a set of 2000 tweets which was used as training set, manually annotated into distinct labels: complaint relevant (702) and complaint non-relevant (1298) ($\kappa =0.79$). We employed these features on our dataset. Linguistic markers. To capture linguistic aspects of complaints, we utilized Bag of Words, count of POS tags and Word2vec clusters. Sentiment markers. We used quantified score based on the ratio of tokens mentioned in the following lexicons: MPQA, NRC, VADER and Stanford. Information specific markers. These account for a set of handcrafted features associated with complaint, we used the stated markers (a) Text-Meta Data, this includes the count of URL's, hashtags, user mentions, special symbols and user mentions, used to enhance retweet impact; (b) Request Identification, we employed the model presented in BIBREF3 to identify if a specific tweet assertion is a request; (c) Intensifiers, we make use of feature set derived from the number of words starting with capital letters and the repetition of special symbols (exclamation, questions marks) within the same post; (d) Politeness Markers, we utilize the politeness score of the tweet extracted from the model presented in BIBREF3; (e) Pronoun Variation, these have the ability to reveal the personal involvement or intensify involvement. We utilize the frequency of pronoun types $\lbrace \textit {first, second, third, demonstrative and indefinite}$} using pre-defined dictionaries. From the pool of 7200 transport relevant tweets, we sampled 3500 tweets which were used as the testing set. The results are reported in TableTABREF5 with 10 fold cross-validation. With increasing the number of iterations, the pool of seed phrases gets refined and augments the selection of transport relevant tweets. The proposed pipeline is tailored to identify complaint relevant tweets in a noisy scenario. Results Table TABREF5 reflects that the BOW model provided the best results, both in terms of accuracy and F1-score. The best result achieved by a sentiment model was the Stanford Sentiment ($0.63$ F1-score), with others within the same range and linguistic-based features collectively giving the best performance. Conclusion and Future Work In this paper, we presented a novel semi-supervised pipeline along with a novel dataset for identification of complaint based posts in the transport domain. The proposed methodology can be expanded for other fields by altering the lexicons used for the creation of information cues. There are limitations to this analysis; we do not use neural networks which mandate a large volume of data. In the future, we aim to identify demographic features for identification of complaint based posts on social media platforms.
How many tweets were collected?
bcc0cd4e262f2db4270429ab520971bcf39414cf
How Ġmany Ġtweets Ġwere Ġcollected ?
$ 19 , 300 $, Ġadded Ġ2500 Ġrandomly Ġsampled Ġtweets
<s> How Ġmany Ġtweets Ġwere Ġcollected ? </s> Introduction </s> With Ġthe Ġadvent Ġof Ġsocial Ġmedia Ġplatforms , Ġincreasing Ġuser Ġbase Ġaddress Ġtheir Ġgrievances Ġover Ġthese Ġplatforms , Ġin Ġthe Ġform Ġof Ġcomplaints . ĠAccording Ġto ĠB IB REF 0 , Ġcomplaint Ġis Ġconsidered Ġto Ġbe Ġa Ġbasic Ġspeech Ġact Ġused Ġto Ġexpress Ġnegative Ġmismatch Ġbetween Ġthe Ġexpectation Ġand Ġreality . ĠTransportation Ġand Ġits Ġrelated Ġlogistics Ġindustries Ġare Ġthe Ġback bones Ġof Ġevery Ġeconomy . ĠMany Ġtransport Ġorganizations Ġrely Ġon Ġcomplaints Ġgathered Ġvia Ġthese Ġplatforms Ġto Ġimprove Ġtheir Ġservices , Ġhence Ġunderstanding Ġthese Ġare Ġimportant Ġfor : Ġ( 1 ) Ġlingu ists Ġto Ġidentify Ġhuman Ġexpressions Ġof Ġcriticism Ġand Ġ( 2 ) Ġorganizations Ġto Ġimprove Ġtheir Ġquery Ġresponse Ġtime Ġand Ġaddress Ġconcerns Ġeffectively . </s> Pres ence Ġof Ġinevitable Ġnoise , Ġsparse Ġcontent Ġalong Ġwith Ġre ph r ased Ġand Ġstruct urally Ġmorphed Ġinstances Ġof Ġposts , Ġmake Ġthe Ġtask Ġat Ġhand Ġdifficult ĠB IB REF 1 . ĠPrevious Ġworks ĠB IB REF 2 Ġin Ġthe Ġdomain Ġof Ġcomplaint Ġextraction Ġhave Ġfocused Ġon Ġstatic Ġdatasets Ġonly . ĠThese Ġare Ġnot Ġrobust Ġto Ġchanges Ġin Ġthe Ġtrends Ġreflected , Ġinformation Ġflow Ġand Ġlinguistic Ġvariations . ĠWe Ġpropose Ġan Ġiter ative , Ġsemi - super vised Ġapproach Ġfor Ġidentification Ġof Ġcomplaint Ġbased Ġtweets , Ġhaving Ġthe Ġability Ġto Ġbe Ġreplicated Ġfor Ġstream Ġof Ġinformation Ġflow . ĠThe Ġpreference Ġof Ġa Ġsemi - super vised Ġapproach Ġover Ġsupervised Ġones Ġis Ġdue Ġto Ġthe Ġstated Ġreasons : Ġ( a ) Ġthe Ġtask Ġof Ġisol ating Ġthe Ġtraining Ġset , Ġmake Ġsupervised Ġtasks Ġless Ġattractive Ġand Ġimpractical Ġand Ġ( b ) Ġimbalance Ġbetween Ġthe Ġsubjective Ġand Ġobjective Ġclasses Ġlead Ġto Ġpoor Ġperformance . </s> Prop osed ĠMethod ology </s> We Ġaimed Ġto Ġmimic Ġthe Ġpresence Ġof Ġsparse / no isy Ġcontent Ġdistribution , Ġmand ating Ġthe Ġneed Ġto Ġcur ate Ġa Ġnovel Ġdataset Ġvia Ġspecific Ġlex icons . ĠWe Ġscrap ed Ġ500 Ġrandom Ġposts Ġfrom Ġrecognized Ġtransport Ġforum . ĠA Ġpool Ġof Ġ50 Ġun i / bi - gram s Ġwas Ġcreated Ġbased Ġon Ġtf - id f Ġrepresentations , Ġextracted Ġfrom Ġthe Ġposts , Ġwhich Ġwas Ġfurther Ġpr uned Ġby Ġannot ators . ĠQu er ying Ġposts Ġon ĠTwitter Ġwith Ġextracted Ġlex icons Ġled Ġto Ġa Ġcollection Ġof Ġ$ 19 , 300 $ Ġtweets . ĠIn Ġorder Ġto Ġhave Ġlex ical Ġdiversity , Ġwe Ġadded Ġ2500 Ġrandomly Ġsampled Ġtweets Ġto Ġour Ġdataset . ĠIn Ġspite Ġof Ġthe Ġsparse Ġnature Ġof Ġthese Ġposts , Ġthe Ġlex ical Ġcharacteristics Ġact Ġas Ġinformation Ġcues . </s> Figure ĠFIG REF 4 Ġpict or ially Ġrepresents Ġour Ġmethodology . ĠOur Ġapproach Ġrequired Ġan Ġinitial Ġset Ġof Ġinformative Ġtweets Ġfor Ġwhich Ġwe Ġemployed Ġtwo Ġhuman Ġannot ators Ġannot ating Ġa Ġrandom Ġsub - sample Ġof Ġthe Ġoriginal Ġdataset . ĠFrom Ġthe Ġ1500 Ġsamples , Ġ326 Ġwere Ġmarked Ġas Ġinformative Ġand Ġ11 74 Ġas Ġnon Ġinformative Ġ($ \ k appa Ġ= 0 . 81 $ ), Ġdiscriminated Ġon Ġthis Ġcriteria : ĠIs Ġthe Ġtweet Ġaddressing Ġany Ġcomplaint Ġor Ġraising Ġgrievances Ġabout Ġmodes Ġof Ġtransport Ġor Ġservices / Ġevents Ġassociated Ġwith Ġtransportation Ġsuch Ġas Ġtraffic ; Ġpublic Ġor Ġprivate Ġtransport ? . ĠAn Ġexample Ġtweet Ġmarked Ġas Ġinformative : ĠNo , Ġmetro Ġfares Ġwill Ġbe Ġreduced Ġ?? ?, Ġbut Ġproper Ġfare Ġstructure Ġneeds Ġto Ġpresented Ġright , Ġit 's Ġbad Ġ !!! . </s> We Ġutilized Ġtf - id f Ġfor Ġthe Ġidentification Ġof Ġinitial Ġseed Ġphrases Ġfrom Ġthe Ġcurated Ġset Ġof Ġinformative Ġtweets . Ġ50 Ġterms Ġhaving Ġthe Ġhighest Ġtf - id f Ġscores Ġwere Ġpassed Ġthrough Ġthe Ġcomplete Ġdataset Ġand Ġbased Ġon Ġsub - string Ġmatch , Ġthe Ġtransport Ġrelevant Ġtweets Ġwere Ġidentified . ĠThe Ġredundant Ġtweets Ġwere Ġfiltered Ġbased Ġon Ġthe Ġcos ine Ġsimilarity Ġscore . ĠImpl icit Ġinformation Ġindicators Ġwere Ġidentified Ġbased Ġon Ġdomain Ġrelevance Ġscore , Ġa Ġmetric Ġused Ġto Ġgauge Ġthe Ġcoverage Ġof Ġn - gram Ġ( 1 , 2 , 3 ) Ġwhen Ġevaluated Ġagainst Ġa Ġrandomly Ġcreated Ġpool Ġof Ġposts . </s> We Ġcollected Ġa Ġpool Ġof Ġ5000 Ġrandomly Ġsampled Ġtweets Ġdifferent Ġfrom Ġthe Ġdata Ġcollection Ġperiod . ĠThe Ġrationale Ġbehind Ġhaving Ġsuch Ġa Ġmetric Ġwas Ġto Ġdiscard Ġcommonly Ġoccurring Ġn - gram s Ġnormalized Ġby Ġrandom Ġnoise Ġand Ġinclude Ġones Ġwhich Ġare Ġof Ġlex ical Ġimportance . ĠWe Ġused Ġterms Ġassociated Ġwith Ġhigh Ġdomain Ġrelevance Ġscore Ġ( th reshold Ġdetermined Ġexperiment ally ) Ġas Ġseed Ġphrases Ġfor Ġthe Ġnext Ġset Ġof Ġiterations . ĠThe Ġgrowing Ġdictionary Ġaug ments Ġthe Ġcollection Ġprocess . ĠThe Ġprocess Ġran Ġfor Ġ4 Ġiterations Ġproviding Ġus Ġ7 200 Ġtransport Ġrelevant Ġtweets Ġas Ġno Ġnew Ġlex icons Ġwere Ġidentified . ĠIn Ġorder Ġto Ġidentify Ġlinguistic Ġsignals Ġassociated Ġwith Ġthe Ġcomplaint Ġposts , Ġwe Ġrandomly Ġsampled Ġa Ġset Ġof Ġ2000 Ġtweets Ġwhich Ġwas Ġused Ġas Ġtraining Ġset , Ġmanually Ġannot ated Ġinto Ġdistinct Ġlabels : Ġcomplaint Ġrelevant Ġ( 702 ) Ġand Ġcomplaint Ġnon - relevant Ġ( 12 98 ) Ġ($ \ k appa Ġ= 0 . 79 $ ). ĠWe Ġemployed Ġthese Ġfeatures Ġon Ġour Ġdataset . </s> L ingu istic Ġmarkers . ĠTo Ġcapture Ġlinguistic Ġaspects Ġof Ġcomplaints , Ġwe Ġutilized ĠBag Ġof ĠWords , Ġcount Ġof ĠPOS Ġtags Ġand ĠWord 2 vec Ġclusters . </s> Sent iment Ġmarkers . ĠWe Ġused Ġquant ified Ġscore Ġbased Ġon Ġthe Ġratio Ġof Ġtokens Ġmentioned Ġin Ġthe Ġfollowing Ġlex icons : ĠMP Q A , ĠN RC , ĠV AD ER Ġand ĠStanford . </s> Information Ġspecific Ġmarkers . ĠThese Ġaccount Ġfor Ġa Ġset Ġof Ġhand crafted Ġfeatures Ġassociated Ġwith Ġcomplaint , Ġwe Ġused Ġthe Ġstated Ġmarkers Ġ( a ) ĠText - Meta ĠData , Ġthis Ġincludes Ġthe Ġcount Ġof ĠURL 's , Ġhasht ags , Ġuser Ġmentions , Ġspecial Ġsymbols Ġand Ġuser Ġmentions , Ġused Ġto Ġenhance Ġretweet Ġimpact ; Ġ( b ) ĠRequest ĠIdentification , Ġwe Ġemployed Ġthe Ġmodel Ġpresented Ġin ĠB IB REF 3 Ġto Ġidentify Ġif Ġa Ġspecific Ġtweet Ġassertion Ġis Ġa Ġrequest ; Ġ( c ) ĠInt ens ifiers , Ġwe Ġmake Ġuse Ġof Ġfeature Ġset Ġderived Ġfrom Ġthe Ġnumber Ġof Ġwords Ġstarting Ġwith Ġcapital Ġletters Ġand Ġthe Ġrepetition Ġof Ġspecial Ġsymbols Ġ( ex clamation , Ġquestions Ġmarks ) Ġwithin Ġthe Ġsame Ġpost ; Ġ( d ) ĠPolit eness ĠMark ers , Ġwe Ġutilize Ġthe Ġpolit eness Ġscore Ġof Ġthe Ġtweet Ġextracted Ġfrom Ġthe Ġmodel Ġpresented Ġin ĠB IB REF 3 ; Ġ( e ) ĠP ron oun ĠVari ation , Ġthese Ġhave Ġthe Ġability Ġto Ġreveal Ġthe Ġpersonal Ġinvolvement Ġor Ġintensify Ġinvolvement . ĠWe Ġutilize Ġthe Ġfrequency Ġof Ġpronoun Ġtypes Ġ$\ l brace Ġ\ text it Ġ{ first , Ġsecond , Ġthird , Ġdemonstr ative Ġand Ġindefinite } $ } Ġusing Ġpre - defined Ġdiction aries . </s> From Ġthe Ġpool Ġof Ġ7 200 Ġtransport Ġrelevant Ġtweets , Ġwe Ġsampled Ġ35 00 Ġtweets Ġwhich Ġwere Ġused Ġas Ġthe Ġtesting Ġset . ĠThe Ġresults Ġare Ġreported Ġin ĠTable TA B REF 5 Ġwith Ġ10 Ġfold Ġcross - valid ation . ĠWith Ġincreasing Ġthe Ġnumber Ġof Ġiterations , Ġthe Ġpool Ġof Ġseed Ġphrases Ġgets Ġrefined Ġand Ġaug ments Ġthe Ġselection Ġof Ġtransport Ġrelevant Ġtweets . ĠThe Ġproposed Ġpipeline Ġis Ġtailored Ġto Ġidentify Ġcomplaint Ġrelevant Ġtweets Ġin Ġa Ġnoisy Ġscenario . </s> Results </s> Table ĠT AB REF 5 Ġreflects Ġthat Ġthe ĠB OW Ġmodel Ġprovided Ġthe Ġbest Ġresults , Ġboth Ġin Ġterms Ġof Ġaccuracy Ġand ĠF 1 - score . ĠThe Ġbest Ġresult Ġachieved Ġby Ġa Ġsentiment Ġmodel Ġwas Ġthe ĠStanford ĠSent iment Ġ($ 0 . 63 $ ĠF 1 - score ), Ġwith Ġothers Ġwithin Ġthe Ġsame Ġrange Ġand Ġlinguistic - based Ġfeatures Ġcollectively Ġgiving Ġthe Ġbest Ġperformance . </s> Conclusion Ġand ĠFuture ĠWork </s> In Ġthis Ġpaper , Ġwe Ġpresented Ġa Ġnovel Ġsemi - super vised Ġpipeline Ġalong Ġwith Ġa Ġnovel Ġdataset Ġfor Ġidentification Ġof Ġcomplaint Ġbased Ġposts Ġin Ġthe Ġtransport Ġdomain . ĠThe Ġproposed Ġmethodology Ġcan Ġbe Ġexpanded Ġfor Ġother Ġfields Ġby Ġaltering Ġthe Ġlex icons Ġused Ġfor Ġthe Ġcreation Ġof Ġinformation Ġcues . ĠThere Ġare Ġlimitations Ġto Ġthis Ġanalysis ; Ġwe Ġdo Ġnot Ġuse Ġneural Ġnetworks Ġwhich Ġmandate Ġa Ġlarge Ġvolume Ġof Ġdata . ĠIn Ġthe Ġfuture , Ġwe Ġaim Ġto Ġidentify Ġdemographic Ġfeatures Ġfor Ġidentification Ġof Ġcomplaint Ġbased Ġposts Ġon Ġsocial Ġmedia Ġplatforms .