@inproceedings{xing-etal-2024-towards,
    title = "Towards Human-aligned Evaluation for Linear Programming Word Problems",
    author = "Xing, Linzi  and
      Wang, Xinglu  and
      Feng, Yuxi  and
      Fan, Zhenan  and
      Xiong, Jing  and
      Guo, Zhijiang  and
      Fu, Xiaojin  and
      Ramamonjison, Rindra  and
      Mostajabdaveh, Mahdi  and
      Han, Xiongwei  and
      Zhou, Zirui  and
      Zhang, Yong",
    editor = "Calzolari, Nicoletta  and
      Kan, Min-Yen  and
      Hoste, Veronique  and
      Lenci, Alessandro  and
      Sakti, Sakriani  and
      Xue, Nianwen",
    booktitle = "Proceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation (LREC-COLING 2024)",
    month = may,
    year = "2024",
    address = "Torino, Italia",
    publisher = "ELRA and ICCL",
    url = "https://aclanthology.org/2024.lrec-main.1438",
    pages = "16550--16556",
    abstract = "Math Word Problem (MWP) is a crucial NLP task aimed at providing solutions for given mathematical descriptions. A notable sub-category of MWP is the Linear Programming Word Problem (LPWP), which holds significant relevance in real-world decision-making and operations research. While the recent rise of generative large language models (LLMs) has brought more advanced solutions to LPWPs, existing evaluation methodologies for this task still diverge from human judgment and face challenges in recognizing mathematically equivalent answers. In this paper, we introduce a novel evaluation metric rooted in graph edit distance, featuring benefits such as permutation invariance and more accurate program equivalence identification. Human evaluations empirically validate the superior efficacy of our proposed metric when particularly assessing LLM-based solutions for LPWP.",
}

Math Word Problem (MWP) is a crucial NLP task aimed at providing solutions for given mathematical descriptions. A notable sub-category of MWP is the Linear Programming Word Problem (LPWP), which holds significant relevance in real-world decision-making and operations research. While the recent rise of generative large language models (LLMs) has brought more advanced solutions to LPWPs, existing evaluation methodologies for this task still diverge from human judgment and face challenges in recognizing mathematically equivalent answers. In this paper, we introduce a novel evaluation metric rooted in graph edit distance, featuring benefits such as permutation invariance and more accurate program equivalence identification. Human evaluations empirically validate the superior efficacy of our proposed metric when particularly assessing LLM-based solutions for LPWP.
Xing, Linzi Wang, Xinglu Feng, Yuxi Fan, Zhenan Xiong, Jing Guo, Zhijiang Fu, Xiaojin Ramamonjison, Rindra Mostajabdaveh, Mahdi Han, Xiongwei Zhou, Zirui Zhang, Yong
Towards Human-aligned Evaluation for Linear Programming Word Problems