Distil BERT Base Uncased Text Classification Model

This repository contains a Distil BERT Base Uncased model that has been fine-tuned for a custom text classification use case. The model is designed to classify text into nine different classes based on the following categories:

Neutral: This class is for any other sort of sentences that do not fall into the specific categories below.
Play: Use this class to classify the intent of the user to listen to music or audio.
Visit: This class is intended for classifying the user's intent to visit or open a website in a web browser.
ImgReco: Use this class to make the bot process an image for image-to-text conversion or any image recognition tasks.
Close: For classifying sentences that indicate the user's intent to close a specific application or software.
Web Search: This class is designed to identify the user's intent to search the internet for information.
Open: Use this class for classifying the intent of opening a specific app or software.
ImgGen: This class is for sentences related to text-to-image processing or image generation tasks.
Math: Classify sentences that include mathematical equations or expressions using this category.

Model Details

Model Architecture: Distil BERT Base Uncased
Number of Classes: 9
Training Data: The model was trained on a custom dataset for text classification tasks related to the mentioned categories.

Usage

You can use this fine-tuned Distil BERT model for text classification in your own applications or projects. Here's a simple example of how to use the model in Python:

from transformers import DistilBertTokenizer, DistilBertForSequenceClassification
import torch

# Load the pre-trained model and tokenizer
tokenizer = DistilBertTokenizer.from_pretrained("Meshwa/Distill-Bert-Automation-Command-Classification")
model = DistilBertForSequenceClassification.from_pretrained("Meshwa/Distill-Bert-Automation-Command-Classification")

# Prepare the input text
text = "Your input text goes here."

# Tokenize the text and classify it
input_ids = tokenizer(text, padding=True, truncation=True, max_length=512, return_tensors="pt")
output = model(**input_ids)
logits = outputs[0].softmax(1)
probabilities = probs.argmax()

# Get the class with the highest probability
pred_label = model.config.id2label[pred_label_idx.item()]
pred_score = probs.max().item()

# The pred_label variable will contain the predicted class label.
print(f"'label': {pred_label}, 'score': {pred_score}")

Remember to replace "your_model_directory_path" with the actual path to the fine-tuned model on your system.

Training

I used my custom labeled dataset for finetuning this model, the dataset contained 700k samples of labled sentences, containing 342k unique sentences and some little bit of synthetic data. I used Gretel AI for generating the synthetic data.

License

Please refer to the licenses associated with the Distil BERT Base Uncased model and any other relevant libraries.

If you have any questions or need further assistance, please feel free to reach out. Email Me

Model Stats after Finetuning

After Training is complete

Epoch	Training Loss	Validation Loss	Accuracy	F1	Precision	Recall
1	0.004100	0.002462	0.999403	0.999114	0.999298	0.998933
2	0.002000	0.002127	0.999372	0.999031	0.999202	0.998863
3	0.001200	0.001835	0.999478	0.999172	0.999349	0.998999

After Evaluation

	eval_loss	eval_Accuracy	eval_F1	eval_Precision	eval_Recall
train	0.000917	0.999567	0.999280	0.999474	0.999089
val	0.001835	0.999478	0.999172	0.999349	0.998999
test	0.002225	0.999369	0.999051	0.999203	0.998901

BibTeX entry and Citation info of the initial Distil-Bert-Base-Uncased

@article{Sanh2019DistilBERTAD,
  title={DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lighter},
  author={Victor Sanh and Lysandre Debut and Julien Chaumond and Thomas Wolf},
  journal={ArXiv},
  year={2019},
  volume={abs/1910.01108}
}

Citation

If you use the fine-tuned DistilBERT model provided in this repository, please cite it as follows:

@article{Meshwa2023,
  title={Fine-Tuned DistilBERT for Text Classification},
  author={Meshwa},
  journal={Hugging Face Model Hub},
  year={2023},
  url={https://huggingface.co/Meshwa/Distill-Bert-Automation-Command-Classification}

Meshwa. (2023). Fine-Tuned DistilBERT for Text Classification. Hugging Face Model Hub. URL: https://huggingface.co/Meshwa/Distill-Bert-Automation-Command-Classification

I appreciate your acknowledgment if you find this model useful in your work.