@inproceedings{zhou-etal-2024-constructions,
    title = "Constructions Are So Difficult That {E}ven Large Language Models Get Them Right for the Wrong Reasons",
    author = {Zhou, Shijia  and
      Weissweiler, Leonie  and
      He, Taiqi  and
      Sch{\"u}tze, Hinrich  and
      Mortensen, David R.  and
      Levin, Lori},
    editor = "Calzolari, Nicoletta  and
      Kan, Min-Yen  and
      Hoste, Veronique  and
      Lenci, Alessandro  and
      Sakti, Sakriani  and
      Xue, Nianwen",
    booktitle = "Proceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation (LREC-COLING 2024)",
    month = may,
    year = "2024",
    address = "Torino, Italia",
    publisher = "ELRA and ICCL",
    url = "https://aclanthology.org/2024.lrec-main.336",
    pages = "3804--3811",
    abstract = "In this paper, we make a contribution that can be understood from two perspectives: from an NLP perspective, we introduce a small challenge dataset for NLI with large lexical overlap, which minimises the possibility of models discerning entailment solely based on token distinctions, and show that GPT-4 and Llama 2 fail it with strong bias. We then create further challenging sub-tasks in an effort to explain this failure. From a Computational Linguistics perspective, we identify a group of constructions with three classes of adjectives which cannot be distinguished by surface features. This enables us to probe for LLM{'}s understanding of these constructions in various ways, and we find that they fail in a variety of ways to distinguish between them, suggesting that they don{'}t adequately represent their meaning or capture the lexical properties of phrasal heads.",
}

In this paper, we make a contribution that can be understood from two perspectives: from an NLP perspective, we introduce a small challenge dataset for NLI with large lexical overlap, which minimises the possibility of models discerning entailment solely based on token distinctions, and show that GPT-4 and Llama 2 fail it with strong bias. We then create further challenging sub-tasks in an effort to explain this failure. From a Computational Linguistics perspective, we identify a group of constructions with three classes of adjectives which cannot be distinguished by surface features. This enables us to probe for LLM{'}s understanding of these constructions in various ways, and we find that they fail in a variety of ways to distinguish between them, suggesting that they don{'}t adequately represent their meaning or capture the lexical properties of phrasal heads.
Zhou, Shijia Weissweiler, Leonie He, Taiqi Sch{\"u}tze, Hinrich Mortensen, David R. Levin, Lori
Constructions Are So Difficult That Even Large Language Models Get Them Right for the Wrong Reasons