Thus far, all approaches discussed have relied solely upon “importance” metrics extracted from a corpus and assigned to each word. However, modern machine learning models often operate in unclear, complicated ways. In some cases, they demonstrate an incredible understanding of language. Specifically, transformers often know which words are important and which are not for their specific NLP task via their attention mechanism [1]}. Here, we present a novel quantitative method for stopword generation that exploits the performance of large-scale language models by using them as a tool for stopword generation. Specifically, we employ two feature selection algorithms, iterative deletion and recursive deletion, to see which stopwords can be removed from a pre-trained DistilBERT transformer with the least degradation to its area under the curve (AUC) performance on the SST2 sentiment benchmark. We use these degradations as a proxy for word importance, crafting several stopword lists. We then test the effectiveness of these stopword lists by using them to train a sentiment analysis logistic regression model.
