@inproceedings{jahan-etal-2024-finding,
    title = "Finding Spoken Identifications: Using {GPT}-4 Annotation for an Efficient and Fast Dataset Creation Pipeline",
    author = "Jahan, Maliha  and
      Wang, Helin  and
      Thebaud, Thomas  and
      Sun, Yinglun  and
      Le, Giang Ha  and
      Fagyal, Zsuzsanna  and
      Scharenborg, Odette  and
      Hasegawa-Johnson, Mark  and
      Moro Velazquez, Laureano  and
      Dehak, Najim",
    editor = "Calzolari, Nicoletta  and
      Kan, Min-Yen  and
      Hoste, Veronique  and
      Lenci, Alessandro  and
      Sakti, Sakriani  and
      Xue, Nianwen",
    booktitle = "Proceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation (LREC-COLING 2024)",
    month = may,
    year = "2024",
    address = "Torino, Italia",
    publisher = "ELRA and ICCL",
    url = "https://aclanthology.org/2024.lrec-main.641",
    pages = "7296--7306",
    abstract = "The growing emphasis on fairness in speech-processing tasks requires datasets with speakers from diverse subgroups that allow training and evaluating fair speech technology systems. However, creating such datasets through manual annotation can be costly. To address this challenge, we present a semi-automated dataset creation pipeline that leverages large language models. We use this pipeline to generate a dataset of speakers identifying themself or another speaker as belonging to a particular race, ethnicity, or national origin group. We use OpenaAI{'}s GPT-4 to perform two complex annotation tasks- separating files relevant to our intended dataset from the irrelevant ones (filtering) and finding and extracting information on identifications within a transcript (tagging). By evaluating GPT-4{'}s performance using human annotations as ground truths, we show that it can reduce resources required by dataset annotation while barely losing any important information. For the filtering task, GPT-4 had a very low miss rate of 6.93{\%}. GPT-4{'}s tagging performance showed a trade-off between precision and recall, where the latter got as high as 97{\%}, but precision never exceeded 45{\%}. Our approach reduces the time required for the filtering and tagging tasks by 95{\%} and 80{\%}, respectively. We also present an in-depth error analysis of GPT-4{'}s performance.",
}

The growing emphasis on fairness in speech-processing tasks requires datasets with speakers from diverse subgroups that allow training and evaluating fair speech technology systems. However, creating such datasets through manual annotation can be costly. To address this challenge, we present a semi-automated dataset creation pipeline that leverages large language models. We use this pipeline to generate a dataset of speakers identifying themself or another speaker as belonging to a particular race, ethnicity, or national origin group. We use OpenaAI{'}s GPT-4 to perform two complex annotation tasks- separating files relevant to our intended dataset from the irrelevant ones (filtering) and finding and extracting information on identifications within a transcript (tagging). By evaluating GPT-4{'}s performance using human annotations as ground truths, we show that it can reduce resources required by dataset annotation while barely losing any important information. For the filtering task, GPT-4 had a very low miss rate of 6.93{\%}. GPT-4{'}s tagging performance showed a trade-off between precision and recall, where the latter got as high as 97{\%}, but precision never exceeded 45{\%}. Our approach reduces the time required for the filtering and tagging tasks by 95{\%} and 80{\%}, respectively. We also present an in-depth error analysis of GPT-4{'}s performance.
Jahan, Maliha Wang, Helin Thebaud, Thomas Sun, Yinglun Le, Giang Ha Fagyal, Zsuzsanna Scharenborg, Odette Hasegawa-Johnson, Mark Moro Velazquez, Laureano Dehak, Najim
Finding Spoken Identifications: Using GPT-4 Annotation for an Efficient and Fast Dataset Creation Pipeline