|
--- |
|
license: apache-2.0 |
|
language: |
|
- en |
|
metrics: |
|
- accuracy |
|
library_name: transformers |
|
pipeline_tag: text-classification |
|
tags: |
|
- customer-service-tickets |
|
- github-issues |
|
- bart-large-mnli |
|
- zero-shot-classification |
|
- NLP |
|
widget: |
|
- text: "Sign up form is not working" |
|
example_title: "Example 1" |
|
- text: "json and yaml support" |
|
example_title: "Example 2" |
|
- text: "fullscreen and tabs media key don't do what they should" |
|
example_title: "Example 2" |
|
--- |
|
|
|
# GitHub issues classifier (using zero shot classification) |
|
|
|
Predicts wether a statement is a feature request, issue/bug or question |
|
|
|
This model was trained using the [**Zero-shot classifier distillation**](https://github.com/huggingface/transformers/tree/main/examples/research_projects/zero-shot-distillation) method |
|
with the [BART-large-mnli](https://huggingface.co/facebook/bart-large-mnli) model as teacher model, to train a classifier on Github issues from the [Github Issues Prediction dataset](https://www.kaggle.com/datasets/anmolkumar/github-bugs-prediction) |
|
|
|
## Labels |
|
|
|
As per the dataset Kaggle competition, the classifier predicts wether an issue is a bug, feature or question. After playing around with different labels pre-training I've used a different mapping |
|
of labels that yielded better predictions (see notebook [here](https://www.kaggle.com/code/antoinemacia/zero-shot-classifier-for-bug-analysis/edit) for details), labels being |
|
|
|
* issue |
|
* feature request |
|
* question |
|
|
|
|
|
## Training data |
|
|
|
* 15k of Github issues titles ("unlabeled_titles_simple.txt") |
|
* Hypothesis used: "This request is a {}" |
|
* Teacher model used: valhalla/distilbart-mnli-12-1 |
|
* Studend model used: distilbert-base-uncased |
|
|
|
## Results |
|
|
|
Agreement of student and teacher predictions: **94.82%** |
|
|
|
See [this notebook](https://www.kaggle.com/code/antoinemacia/zero-shot-classifier-for-bug-analysis/edit) for more info on feature engineering choice made |
|
|
|
|
|
## How to train using your own dataset |
|
* Download training dataset from https://www.kaggle.com/datasets/anmolkumar/github-bugs-prediction |
|
* Modify and run convert.py, updating the paths to convert to a CSV |
|
* Run distill.py with the csv file (see [here](https://github.com/huggingface/transformers/tree/main/examples/research_projects/zero-shot-distillation) for more info) |
|
|
|
## Acknowledgements |
|
|
|
* Joe Davison and his article on [Zero-Shot Learning in Modern NLP](https://joeddav.github.io/blog/2020/05/29/ZSL.html) |
|
* Jeremy Howard, fast.ai and his notebook [Iterate like a grandmaster](https://www.kaggle.com/code/antoinemacia/iterate-like-a-grandmaster) |
