2017 Best Paper | SIGACCESS SIGACCESS ACM's Special Interest Group on Accessible Computing SIGACCESS Mission Statement Leadership Join Local Chapters Contact RSS Feed Calls Call For Papers Call For Nominations Call For Award News Newsletter ASSETS (Conference) Current ASSETS Conference Most Recent Proceedings Past ASSETS Conferences Related Conferences Resources Accessible Conference Guide Accessible Virtual Conferences Accessible Writing Guide Describing Figures Accessibility FAQ Page Generator Accessible PDF Author Guide Accessible Presentation Guide Conflict of Interest Thesis Repository Download Logo ACM Diversity and Inclusion Council Access SIGCHI TACCESS (Journal) Anti-Harassment Policy Awards Outstanding Contributions Impact Award Best Paper Best Student Paper SIGACCESS Travel Scholarship 2017 Best Paper by pastchair_sigaccess on November 3, 2017, 0 comments Evaluating the Usability of Automatically Generated Captions for People who are Deaf or Hard of Hearing Author(s): Sushant Kafle and Matt Huenerfauth Institution: Rochester Institute of Technology Abstract: The accuracy of Automated Speech Recognition (ASR) technology has improved, but it is still imperfect in many settings. Researchers who evaluate ASR performance often focus on improving the Word Error Rate (WER) metric, but WER has been found to have little correlation with human-subject performance on many applications. We propose a new captioning-focused evaluation metric that better predicts the impact of ASR recognition errors on the usability of automatically generated captions for people who are Deaf or Hard of Hearing (DHH). Through a user study with 30 DHH users, we compared our new metric with the traditional WER metric on a caption usability evaluation task. In a side-by-side comparison of pairs of ASR text output (with identical WER), the texts preferred by our new metric were preferred by DHH participants. Further, our metric had significantly higher correlation with DHH participants’ subjective scores on the usability of a caption, as compared to the correlation between WER metric and participant subjective scores. This new metric could be used to select ASR systems for captioning applications, and it may be a better metric for ASR researchers to consider when optimizing ASR systems. Full Paper: Download Full Paper Here Categories: Best Paper pastchair_sigaccess Dr Shari Trewin is a researcher in human-computer interaction (HCI), specializing in accessibility. She has made research contributions to many areas of inclusive information and communication technology, including accessibility tools for developers, tools that enable users to customize the appearance and access mechanisms for Web pages, access to Virtual Worlds, abstract user interface models for accessibility, novel access features for input devices, and self-adaptive input devices. She holds seven patents for accessibility technology. Shari is an ACM Distinguished Scientist, and Associate Editor of the journal ACM Transactions on Accessible Computing. She has previously served as SIGACCESS Secretary/Treasurer, general chair of ASSETS 2009: The 11th International ACM SIGACCESS Conference on Computers and Accessibility, and program chair of ASSETS 2007. She has a Ph.D. in Artificial Intelligence from the University of Edinburgh, Scotland. View all posts → Comments are closed. Updates 17 Aug - June 2022 Newsletter 27 Jun - SIGACCESS Outstanding Contribution 26 Feb - Call for Papers: ASSETS 2022 26 Feb - January 2022 Newsletter Resources Accessible Conference Guide Accessible Writing Guide Describing Figures Accessible Presentation Guide Conflict of Interest Thesis Repository Accessibility FAQ Page Generator Accessible PDF Author Guide Download Logo ACM Diversity and Inclusion Council Access SIGCHI TACCESS (Journal) Copyright © 2022 SIGACCESS. All rights Reserved.