@inproceedings{hertzberg-lokrantz-2024-medqa,
    title = "{M}ed{QA}-{SWE} - a Clinical Question {\&} Answer Dataset for {S}wedish",
    author = "Hertzberg, Niclas  and
      Lokrantz, Anna",
    editor = "Calzolari, Nicoletta  and
      Kan, Min-Yen  and
      Hoste, Veronique  and
      Lenci, Alessandro  and
      Sakti, Sakriani  and
      Xue, Nianwen",
    booktitle = "Proceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation (LREC-COLING 2024)",
    month = may,
    year = "2024",
    address = "Torino, Italia",
    publisher = "ELRA and ICCL",
    url = "https://aclanthology.org/2024.lrec-main.975",
    pages = "11178--11186",
    abstract = "Considering the rapid improvement of large generative language models, it is important to measure their ability to encode clinical domain knowledge in order to help determine their potential utility in a clinical setting. To this end we present MedQA-SWE {--} a novel multiple choice, clinical question {\&} answering (Q{\&}A) dataset in Swedish consisting of 3,180 questions. The dataset was created from a series of exams aimed at evaluating doctors{'} clinical understanding and decision making and is the first open-source clinical Q{\&}A dataset in Swedish. The exams {--} originally in PDF format {--} were parsed and each question manually checked and curated in order to limit errors in the dataset. We provide dataset statistics along with benchmark accuracy scores of seven large generative language models on a representative sample of questions in a zero-shot setting, with some models showing impressive performance given the difficulty of the exam the dataset is based on.",
}

Considering the rapid improvement of large generative language models, it is important to measure their ability to encode clinical domain knowledge in order to help determine their potential utility in a clinical setting. To this end we present MedQA-SWE {--} a novel multiple choice, clinical question {\&} answering (Q{\&}A) dataset in Swedish consisting of 3,180 questions. The dataset was created from a series of exams aimed at evaluating doctors{'} clinical understanding and decision making and is the first open-source clinical Q{\&}A dataset in Swedish. The exams {--} originally in PDF format {--} were parsed and each question manually checked and curated in order to limit errors in the dataset. We provide dataset statistics along with benchmark accuracy scores of seven large generative language models on a representative sample of questions in a zero-shot setting, with some models showing impressive performance given the difficulty of the exam the dataset is based on.
Hertzberg, Niclas Lokrantz, Anna
MedQA-SWE - a Clinical Question & Answer Dataset for Swedish