@inproceedings{wilkens-etal-2024-paying,
    title = "Paying attention to the words: explaining readability prediction for {F}rench as a foreign language",
    author = "Wilkens, Rodrigo  and
      Watrin, Patrick  and
      Fran{\c{c}}ois, Thomas",
    editor = "Wilkens, Rodrigo  and
      Cardon, R{\'e}mi  and
      Todirascu, Amalia  and
      Gala, N{\'u}ria",
    booktitle = "Proceedings of the 3rd Workshop on Tools and Resources for People with REAding DIfficulties (READI) @ LREC-COLING 2024",
    month = may,
    year = "2024",
    address = "Torino, Italia",
    publisher = "ELRA and ICCL",
    url = "https://aclanthology.org/2024.readi-1.9",
    pages = "102--115",
    abstract = "Automatic text Readability Assessment (ARA) has been seen as a way of helping people with reading difficulties. Recent advancements in Natural Language Processing have shifted ARA from linguistic-based models to more precise black-box models. However, this shift has weakened the alignment between ARA models and the reading literature, potentially leading to inaccurate predictions based on unintended factors. In this paper, we investigate the explainability of ARA models, inspecting the relationship between attention mechanism scores, ARA features, and CEFR level predictions made by the model. We propose a method for identifying features associated with the predictions made by a model through the use of the attention mechanism. Exploring three feature families (i.e., psycho-linguistic, work frequency and graded lexicon), we associated features with the model{'}s attention heads. Finally, while not fully explanatory of the model{'}s performance, the correlations of these associations surpass those between features and text readability levels.",
}

Automatic text Readability Assessment (ARA) has been seen as a way of helping people with reading difficulties. Recent advancements in Natural Language Processing have shifted ARA from linguistic-based models to more precise black-box models. However, this shift has weakened the alignment between ARA models and the reading literature, potentially leading to inaccurate predictions based on unintended factors. In this paper, we investigate the explainability of ARA models, inspecting the relationship between attention mechanism scores, ARA features, and CEFR level predictions made by the model. We propose a method for identifying features associated with the predictions made by a model through the use of the attention mechanism. Exploring three feature families (i.e., psycho-linguistic, work frequency and graded lexicon), we associated features with the model{'}s attention heads. Finally, while not fully explanatory of the model{'}s performance, the correlations of these associations surpass those between features and text readability levels.
Wilkens, Rodrigo Watrin, Patrick Fran{\c{c}}ois, Thomas
Paying attention to the words: explaining readability prediction for French as a foreign language