We perform two distinct studies to evaluate our approach synthetic and user study. In the synthetic study, we comparatively evaluate different approaches, namely, VSM (baseline), VSM+SA (our approach with TR disabled), VSM+TR (our approach with SA disabled), and VSM+SA+TR used by RetroRank. Results indicate that VSM+SA and VSM+TR both outperformed the baseline VSM, but found that VSM+SA+TR was the best performing approach overall. In the user study, we evaluated best performing approach (i.e., VSM+SA+TR) to the baseline VSM. Our choice for using VSM as a baseline is supported by four premises: i) VSM can determine term relevance in bug report queries [1]},
ii) each discussion comment can be represented into a vector of term frequency-inverse document frequency (TF\(\times \) IDF) weights that can be enhanced for relevance ranking [2]}, iii) VSM can be enhanced by NLP methods when VSM alone fails to capture textual matching between queries and comments [3]}, and iv) it is regarded as a suitable method for bug localization tasks [4]}, [5]}, [6]}. Results indicate that RetroRank when using combination VSM+SA+TR outperformed the baseline VSM.
