{
  "name" : "1706.03335.pdf",
  "metadata" : {
    "source" : "CRF",
    "title" : "Exploring Automated Essay Scoring for Nonnative English Speakers",
    "authors" : [ "Amber Nigam" ],
    "emails" : [ ],
    "sections" : [ {
      "heading" : null,
      "text" : "Automated Essay Scoring (AES) has been quite popular and is being widely used. However, lack of appropriate methodology for rating nonnative English speakers’ essays has meant a lopsided advancement in this field. In this paper, we report initial results of our experiments with nonnative AES that learns from manual evaluation of nonnative essays. For this purpose, we conducted an exercise in which essays written by nonnative English speakers in test environment were rated both manually and by the automated system designed for the experiment. In the process, we experimented with a few features to learn about nuances linked to nonnative evaluation. The proposed methodology of automated essay evaluation has yielded a correlation coefficient of 0.750 with the manual evaluation."
    }, {
      "heading" : "1. Introduction",
      "text" : "There are different versions of Automated Essay Scoring (AES) and lack of generalizability across different analyses and corpuses prompts a question over the validity of one size fits all AES.\nFurthermore, nonnative analysis is differentiated from the native analysis on a few aspects. For example, it is difficult to detect context in essays which have errors specific to some nonnative usages. Moreover, a few nonnative usages like Qutub Minar and Karur are not a part of standard English. This, however, does not render the usage of these words incorrect.\nIn this paper, we have discussed our methodology for nonnative essay evaluation. First, we have described the feature set that we used for our experiments. Second, we have discussed various adjustments that were made to the system to make it learn and account for nonnative usages from manual evaluation. Finally, we have\ndiscussed the results of our experiments and the intent of larger analysis of which this experiment is a part of."
    }, {
      "heading" : "2. Related Work",
      "text" : "Although Automated Essay Scoring (AES) has been widely used in many of the real world applications, there is very limited published work on rating nonnative speakers. Following analyses deal with nonnative speakers in one way or another.\ne-rater system™, developed by Educational Testing Service (ETS), is one of the tools that automates scoring of English essays of native and nonnative speakers. In their analysis of e-rater, Jill Burstein, et al. (1999), reported that even when 75% of essays used for model building were written by nonnative English speakers, the features selected by the regression procedure were largely the same as those in models based on operational writing samples in which the majority of the sample were native English speakers. The correlations between e-rater scores and those of a single human reader were about .73. However, as mentioned in the paper, there were significant differences between final human reader score and e-rater score across language groups, and more data is needed to build individual models for different language groups to examine how this affects erater’s performance.\nAnother analysis in this field is by Sowmya Vajjala, (2016), which is the first multi-corpus study using TOEFL11SUBSET and First Certificate in English (FCE) datasets. For TOEFL11SUBSET, the best model achieved a prediction accuracy of 73% for classifying between three proficiencies (low, medium and high), using all the features. In general, shallow discourse features such as word overlap were more predictive for this dataset compared to deeper ones like reference chains. The native language of the author was an important predictor for some feature groups in this dataset. With FCE, the best model achieved a correlation of 0.64 and a Mean Absolute Error of 3.6, on the test data. In general, features that relied\non deeper linguistic modeling (such as reference chains) had more weight for this dataset compared to other features. The native language of the author was not an important predictor in the dataset. The study concludes by stating that the features do not seem to be completely generalizable across datasets.\nCurrent research differs from the existing research in its feature set, unique methodology, and an essay corpus composed of essays written by candidates whose native language is Hindi.\nWe are not referring to generic AES systems like by Dimitrios Alikaniotis, et al. (2016) and by Kaveh Taghipour, et al. (2016) because current paper is only concerned with nonnative AES."
    }, {
      "heading" : "3. Experiment",
      "text" : "The current experiment encompassed building a model through feature selection followed by manual and automated rating of essays (as shown in Figure 1). In all, there were more than 900 essays of length between 150 and 400 words across 7 unique topics. Essay topics were carefully chosen to be on commonly known issues so that they are easy to comprehend and write about. The test takers were all candidates whose native language is Hindi to keep the analysis independent of test taker’s native language. Each essay was manually scored on a scale of 1-10 by two raters, who had .81 Cohen’s kappa statistic (Cohen J, 1968) between their ratings. The split between training, validation, and testing sets was approximately 60:20:20.\nWe used LanguageTool (Daniel Naber, 2003; LanguageTool, 2012) for detecting grammatical errors. Besides, the system was designed to detect cheating attempts such as repeating content, writing out of context, and excessive usage of irrelevant words."
    }, {
      "heading" : "4. Feature Set Selection",
      "text" : "The top features (see Table 1 for correlation analysis of a few prominent features) are described below:\n4.1 Grammar error density (i.e. grammar errors per unit length)\nIn this paper, grammar error density refers to the grammatical error count per 100 words. We use density rather than simple error count to ensure that candidates who have written longer essays are not unduly penalized over those who have written shorter essays with the same grammar error density.\nA granular categorization of grammar errors has helped in classifying the errors into severity bands on the basis of their impact on manual evaluation. For example, it was observed that a subject-verb agreement error was generally more severely penalized than a style based error in the manual evaluation. Following are the buckets in which grammar errors have been classified:\nMajor errors like wrong form, incorrect tense, and agreement errors: This bucket includes severe grammar errors that degrade the quality of the essay and possibly cause serious comprehension issues. For instance, replacing “his” with “her” to form a sentence like “He is known for her intelligence.” would be detrimental to the semantics of the sentence.\nCapitalization errors: Such errors occur when case of the character is not what it ought to be. This might happen when the first word of a sentence begins with a letter in lowercase. Proper nouns like Paris, Shakespeare also need to be capitalized. Also, personal pronoun “I” is always written in capital when used. Other pronouns are capitalized only if they begin a sentence.\nTypography errors: These are also known as typographical errors and are not same as spelling errors. They result due to mechanical failure or slips of the hand or finger. For instance, “water” typed as “wster” due to S key being close to the A key. Typos generally involve duplication, omission, transposition or substitution of a small number of characters.\nStyle based errors: These errors include usage of informal language or shorthand like using “u” instead of “you”.\nCommon replacement errors: Such errors happen in case of a sound-alike or lookalike word pair when one of the words is replaced by the other word in the pair. An example of such word pair is “affect” and “effect”.\nPunctuation errors: These errors refer to incorrect usage of comma, semi-colon, colon, apostrophe, hyphen, etc. in a sentence or paragraph. Misplaced punctuation can sometimes alter the meaning of a sentence. For example, a sentence like “Jane finds inspiration in cooking, her family, and her dog.” without commas would be read as “Jane finds inspiration in cooking her family and her dog.”.\nMiscellaneous errors: All other grammar errors are put under miscellaneous bucket. These include errors such as repetition of words, improper white space usage, etc.\n4.2 Grammar error coverage\nIt is the count of type of grammar errors per 100 words in an essay. Grammar error coverage highlights the spread of errors across different grammar buckets.\n4.3 Spelling error density (i.e. spelling errors per unit length)\nSpelling error density of an essay is referred to as the number of spelling errors in the essay per 100 words.\nPenalty for a recurring error is based on error quantum, which is evaluated by damping the error frequency (as shown in Figure 2), to lower incremental penalty for a single recurring error.\n4.4 Spelling error coverage\nIt is the count of unique spelling errors per 100 words in an essay. The feature compliments spelling error density by accounting for cases where a difficult spelling might be repeated in the essay due to a difficult topic.\n4.5 Readability\nReadability attempts to estimate whether the reader can understand a written text. Why is it relevant in nonnative speakers’ analysis?\nIt helps distinguish between the candidates who can articulate their thoughts into a syntactically correct and, if required, complex structure, and those who cannot. Owing to the fact that some nonnative speakers lack even the basic ability of constructing appropriate text, it becomes a very important feature. Our data shows that readability alone has a strong correlation with manual scoring. We use Flesch–Kincaid grade level (Kincaid JP, 1975) that is evaluated on word count (WC), sentence count (SC), and syllable count (SyC) by the equation:\n0.39 ∗ ( WC\nSC ) + 11.8 ∗ (\nSyC WC ) − 15.59 (1)\n4.6 Lexical Density\nIt measures the ratio of content words to grammatical words (Ure, J 1971). Lexical words give a text its meaning and include nouns, adjectives, most verbs, and most adverbs. Grammatical words act as syntactic sugar and include pronouns, prepositions, conjunctions, etc. Lexical density gives a measure of the breadth of content in an essay. This is another factor that is strongly correlated with manual scoring.\nThe formula for lexical density (LD) is:\nNlex\nN ∗ 100 (2)\nwhere Nlex is number of lexical word tokens (nouns, adjectives, verbs, adverbs) and N is the total number of tokens in the text.\n4.7 Context/Relevance\nLatent semantic analysis (Foltz, et al., 1998) is used to detect the context of the topic using n-grams of phrases from word count 1 to 5. Our data shows that through grammatical error correction in the essays of nonnative speakers (Alla Rozovskaya and Dan Roth, 2016), we are better able to detect context of grammatically incorrect essays. Context is detected by checking if cosine similarity between vector of the evaluated essay and vector of at least one corpus essay is more than a specified threshold. This feature is only (negatively) scored when there is not sufficient context in the essay.\n4.8 Coherence\nWe use Grid model (Lapata and Barzilay, 2005; Barzilay and Lapata, 2008) to detect coherence. This attribute accounts for the flow and structuring of an essay."
    }, {
      "heading" : "5. Self-correction Mechanism",
      "text" : "One of the unique selling propositions of the engine is that it pops unknown spellings/phrases once their cumulative frequency is beyond a pre-defined threshold. This has helped us add many nonnative usages into our repository. For example, proper nouns such as Kaushambi and Ranganathan have been some of the recent additions."
    }, {
      "heading" : "6. Results",
      "text" : "Various machine learning algorithms were used to predict the score of the essays and the predictions by Random Forest algorithm were closest to the manual scoring as shown in the Table 2. The most\npredictive features of the analysis include a few grammar error densities, lexical density, readability, and spelling error density.\nWe used WEKA toolkit (Hall et al., 2009) for\nscore prediction."
    }, {
      "heading" : "7. Conclusion and Future Work",
      "text" : "In this paper, we presented a methodology of rating English essays of nonnative speakers that included but was not limited to selecting a relevant feature set for the evaluation, categorizing grammar errors into finer types to learn about their importance from their distinctive treatment in contribution to the manual evaluation in nonnative context, treating essays for typical nonnative grammar errors (Alla Rozovskaya and Dan Roth, 2016) that improved context matching for essays with nonnative errors, and devising a self-correction mechanism to learn and promptly address nonnative spellings and styles. Our results show that these incremental adjustments have cumulatively helped in better alignment of automated evaluation with manual evaluation.\nWe plan to use Long Short Term Memory (LSTM) network to explore this domain further. We also intend to extend our research by studying the impact of nativity on English learning and the errors that nativity induces. The idea is to come up with a region-wise plan for English improvement as we believe that when learning a language has a different path for different regions; course correction has to be different too. Lastly, we plan to test our evaluation system on an independent corpus.\nAcknowledgements We would like to thank Professor Dan Roth for his valuable feedback and guidance, especially in detecting context of grammatically incorrect essays."
    } ],
    "references" : [ {
      "title" : "Grammatical Error Correction: Machine Translation and Classifiers ACL (2016",
      "author" : [ "Alla Rozovskaya", "Dan Roth" ],
      "venue" : null,
      "citeRegEx" : "Rozovskaya and Roth.,? \\Q2016\\E",
      "shortCiteRegEx" : "Rozovskaya and Roth.",
      "year" : 2016
    }, {
      "title" : "Weighted Kappa: Nominal Scale Agreement with Provision for Scaled Disagreement or Partial Credit",
      "author" : [ "J. Cohen" ],
      "venue" : "Psychol. Bull. 70,",
      "citeRegEx" : "Cohen,? \\Q1968\\E",
      "shortCiteRegEx" : "Cohen",
      "year" : 1968
    }, {
      "title" : "A Rule-Based Style and Grammar Checker",
      "author" : [ "Daniel Naber" ],
      "venue" : "Diploma Thesis, University of Bielefeld,",
      "citeRegEx" : "Naber.,? \\Q2003\\E",
      "shortCiteRegEx" : "Naber.",
      "year" : 2003
    }, {
      "title" : "Automatic Text Scoring Using Neural Networks",
      "author" : [ "Dimitrios Alikaniotis", "Helen Yannakoudakis", "Marek Rei" ],
      "venue" : "In Proceedings of ACL,",
      "citeRegEx" : "Alikaniotis et al\\.,? \\Q2016\\E",
      "shortCiteRegEx" : "Alikaniotis et al\\.",
      "year" : 2016
    }, {
      "title" : "A Neural Approach to Automated Essay Scoring",
      "author" : [ "Kaveh Taghipour", "Hwee Tou Ng" ],
      "venue" : "In Proceedings of EMNLP,",
      "citeRegEx" : "Taghipour and Ng.,? \\Q2016\\E",
      "shortCiteRegEx" : "Taghipour and Ng.",
      "year" : 2016
    }, {
      "title" : "The measurement of textual coherence with Latent Semantic Analysis",
      "author" : [ "P.W. Foltz", "W. Kintsch", "T.K. Landauer" ],
      "venue" : "Discourse Processes,",
      "citeRegEx" : "Foltz et al\\.,? \\Q1998\\E",
      "shortCiteRegEx" : "Foltz et al\\.",
      "year" : 1998
    }, {
      "title" : "The weka data mining software: An update",
      "author" : [ "M. Hall", "E. Frank", "G. Holmes", "B. Pfahringer", "P. Reutemann", "I.H. Witten" ],
      "venue" : "In The SIGKDD Explorations,",
      "citeRegEx" : "Hall et al\\.,? \\Q2009\\E",
      "shortCiteRegEx" : "Hall et al\\.",
      "year" : 2009
    }, {
      "title" : "Derivation of New Readability Formulas (Automated Readability Index, Fog Count and Flesch Reading Ease Formula) for Navy Enlisted Personnel",
      "author" : [ "JP Kincaid", "RP Fishburne", "RL Rogers", "BS. Chissom" ],
      "venue" : "Memphis, Tenn: Naval Air Station;",
      "citeRegEx" : "Kincaid et al\\.,? \\Q1975\\E",
      "shortCiteRegEx" : "Kincaid et al\\.",
      "year" : 1975
    }, {
      "title" : "Automatic evaluation of text coherence: Models and representations",
      "author" : [ "Mirella Lapata", "Regina Barzilay." ],
      "venue" : "IJCAI, volume 5, pages 1085– 1090.",
      "citeRegEx" : "Lapata and Barzilay.,? 2005",
      "shortCiteRegEx" : "Lapata and Barzilay.",
      "year" : 2005
    }, {
      "title" : "Modeling local coherence: An entity-based approach",
      "author" : [ "Regina Barzilay", "Mirella Lapata." ],
      "venue" : "Computational Linguistics, 34(1):1–34.",
      "citeRegEx" : "Barzilay and Lapata.,? 2008",
      "shortCiteRegEx" : "Barzilay and Lapata.",
      "year" : 2008
    }, {
      "title" : "Lexical density and register differentiation",
      "author" : [ "J Ure" ],
      "venue" : "In G. Perren and J.L.M. Trim (eds), Applications of Linguistics,",
      "citeRegEx" : "Ure,? \\Q1971\\E",
      "shortCiteRegEx" : "Ure",
      "year" : 1971
    } ],
    "referenceMentions" : [ {
      "referenceID" : 8,
      "context" : "We use Grid model (Lapata and Barzilay, 2005; Barzilay and Lapata, 2008) to detect coherence.",
      "startOffset" : 18,
      "endOffset" : 72
    }, {
      "referenceID" : 9,
      "context" : "We use Grid model (Lapata and Barzilay, 2005; Barzilay and Lapata, 2008) to detect coherence.",
      "startOffset" : 18,
      "endOffset" : 72
    }, {
      "referenceID" : 6,
      "context" : "We used WEKA toolkit (Hall et al., 2009) for score prediction.",
      "startOffset" : 21,
      "endOffset" : 40
    } ],
    "year" : 2017,
    "abstractText" : "Automated Essay Scoring (AES) has been quite popular and is being widely used. However, lack of appropriate methodology for rating nonnative English speakers’ essays has meant a lopsided advancement in this field. In this paper, we report initial results of our experiments with nonnative AES that learns from manual evaluation of nonnative essays. For this purpose, we conducted an exercise in which essays written by nonnative English speakers in test environment were rated both manually and by the automated system designed for the experiment. In the process, we experimented with a few features to learn about nuances linked to nonnative evaluation. The proposed methodology of automated essay evaluation has yielded a correlation coefficient of 0.750 with the manual evaluation.",
    "creator" : "Microsoft® Word 2013"
  }
}