We repeated the experiments of the last subsection, with the same test sets,
but instead of using human raters, we used a LLM rater. The LLM rater receives the top 3 concepts,
just like the human rater, and makes a decision by computing cosine similarity
between its embedding of the textual representation of each class to its
embedding of the textual representation of the 3 concepts. The 3 LLMs used for rating are SBERT [1]} Model bert-base-nli-mean-tokens from Hugging Face, ST5 [2]} Model sentence-t5-large from Hugging Face and SRoBERTa.
Note that the two uses of SRoBERTa are quite different. The one used
for the original classification is based on a training set and a smart learning algorithm, while the model used for rating just computes similarity
between the class and the 3 concepts.
