GitHub issues classifier (using zero shot classification)

Predicts wether a statement is a feature request, issue/bug or question

This model was trained using the Zero-shot classifier distillation method with the BART-large-mnli model as teacher model, to train a classifier on Github issues from the Github Issues Prediction dataset

Labels

As per the dataset Kaggle competition, the classifier predicts wether an issue is a bug, feature or question. After playing around with different labels pre-training I've used a different mapping of labels that yielded better predictions (see notebook here for details), labels being

issue
feature request
question

Training data

15k of Github issues titles ("unlabeled_titles_simple.txt")
Hypothesis used: "This request is a {}"
Teacher model used: valhalla/distilbart-mnli-12-1
Studend model used: distilbert-base-uncased

Results

Agreement of student and teacher predictions: 94.82%

See this notebook for more info on feature engineering choice made

How to train using your own dataset

Download training dataset from https://www.kaggle.com/datasets/anmolkumar/github-bugs-prediction
Modify and run convert.py, updating the paths to convert to a CSV
Run distill.py with the csv file (see here for more info)

Acknowledgements

Joe Davison and his article on Zero-Shot Learning in Modern NLP
Jeremy Howard, fast.ai and his notebook Iterate like a grandmaster