TITLE: Evaluating Software User Feedback Classifiers on Unseen Apps, Datasets, and Metadata 

ABSTRACT: Listening to user's requirements is crucial to building and maintaining high
quality software. Online software user feedback has been shown to contain large
amounts of information useful to requirements engineering (RE). Previous
studies have created machine learning classifiers for parsing this feedback for
development insight. While these classifiers report generally good performance
when evaluated on a test set, questions remain as to how well they extend to
unseen data in various forms.
  This study evaluates machine learning classifiers performance on feedback for
two common classification tasks (classifying bug reports and feature requests).
Using seven datasets from prior research studies, we investigate the
performance of classifiers when evaluated on feedback from different apps than
those contained in the training set and when evaluated on completely different
datasets (coming from different feedback platforms and/or labelled by different
researchers). We also measure the difference in performance of using
platform-specific metadata as a feature in classification.
  We demonstrate that classification performance is similar on feedback from
unseen apps compared to seen apps in the majority of cases tested. However, the
classifiers do not perform well on unseen datasets. We show that multi-dataset
training or zero shot classification approaches can somewhat mitigate this
performance decrease. Finally, we find that using metadata as features in
classifying bug reports and feature requests does not lead to a statistically
significant improvement in the majority of datasets tested. We discuss the
implications of these results on developing user feedback classification models
to analyse and extract software requirements.
Listening to user's requirements is crucial to building and maintaining high
quality software. Online software user feedback has been shown to contain large
amounts of information useful to requirements engineering (RE). Previous
studies have created machine learning classifiers for parsing this feedback for
development insight. While these classifiers report generally good performance
when evaluated on a test set, questions remain as to how well they extend to
unseen data in various forms.
  This study evaluates machine learning classifiers performance on feedback for
two common classification tasks (classifying bug reports and feature requests).
Using seven datasets from prior research studies, we investigate the
performance of classifiers when evaluated on feedback from different apps than
those contained in the training set and when evaluated on completely different
datasets (coming from different feedback platforms and/or labelled by different
researchers). We also measure the difference in performance of using
platform-specific metadata as a feature in classification.
  We demonstrate that classification performance is similar on feedback from
unseen apps compared to seen apps in the majority of cases tested. However, the
classifiers do not perform well on unseen datasets. We show that multi-dataset
training or zero shot classification approaches can somewhat mitigate this
performance decrease. Finally, we find that using metadata as features in
classifying bug reports and feature requests does not lead to a statistically
significant improvement in the majority of datasets tested. We discuss the
implications of these results on developing user feedback classification models
to analyse and extract software requirements.
Evaluating Software User Feedback Classifiers on Unseen Apps, Datasets, and Metadata
http://arxiv.org/abs/2112.13497v1