@inproceedings{da-corte-baptista-2024-enhancing,
    title = "Enhancing Writing Proficiency Classification in Developmental Education: The Quest for Accuracy",
    author = "Da Corte, Miguel  and
      Baptista, Jorge",
    editor = "Calzolari, Nicoletta  and
      Kan, Min-Yen  and
      Hoste, Veronique  and
      Lenci, Alessandro  and
      Sakti, Sakriani  and
      Xue, Nianwen",
    booktitle = "Proceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation (LREC-COLING 2024)",
    month = may,
    year = "2024",
    address = "Torino, Italia",
    publisher = "ELRA and ICCL",
    url = "https://aclanthology.org/2024.lrec-main.542",
    pages = "6134--6143",
    abstract = "Developmental Education (DevEd) courses align students{'} college-readiness skills with higher education literacy demands. These courses often use automated assessment tools like Accuplacer for student placement. Existing literature raises concerns about these exams{'} accuracy and placement precision due to their narrow representation of the writing process. These concerns warrant further attention within the domain of automatic placement systems, particularly in the establishment of a reference corpus of annotated essays for these systems{'} machine/deep learning. This study aims at an enhanced annotation procedure to assess college students{'} writing patterns more accurately. It examines the efficacy of machine-learning-based DevEd placement, contrasting Accuplacer{'}s classification of 100 college-intending students{'} essays into two levels (Level 1 and 2) against that of 6 human raters. The classification task encompassed the assessment of the 6 textual criteria currently used by Accuplacer: mechanical conventions, sentence variety {\&} style, idea development {\&} support, organization {\&} structure, purpose {\&} focus, and critical thinking. Results revealed low inter-rater agreement, both on the individual criteria and the overall classification, suggesting human assessment of writing proficiency can be inconsistent in this context. To achieve a more accurate determination of writing proficiency and improve DevEd placement, more robust classification methods are thus required.",
}

Developmental Education (DevEd) courses align students{'} college-readiness skills with higher education literacy demands. These courses often use automated assessment tools like Accuplacer for student placement. Existing literature raises concerns about these exams{'} accuracy and placement precision due to their narrow representation of the writing process. These concerns warrant further attention within the domain of automatic placement systems, particularly in the establishment of a reference corpus of annotated essays for these systems{'} machine/deep learning. This study aims at an enhanced annotation procedure to assess college students{'} writing patterns more accurately. It examines the efficacy of machine-learning-based DevEd placement, contrasting Accuplacer{'}s classification of 100 college-intending students{'} essays into two levels (Level 1 and 2) against that of 6 human raters. The classification task encompassed the assessment of the 6 textual criteria currently used by Accuplacer: mechanical conventions, sentence variety {\&} style, idea development {\&} support, organization {\&} structure, purpose {\&} focus, and critical thinking. Results revealed low inter-rater agreement, both on the individual criteria and the overall classification, suggesting human assessment of writing proficiency can be inconsistent in this context. To achieve a more accurate determination of writing proficiency and improve DevEd placement, more robust classification methods are thus required.
Da Corte, Miguel Baptista, Jorge
Enhancing Writing Proficiency Classification in Developmental Education: The Quest for Accuracy