Hugging Face's logo

m-aliabbas1
/
idrakFE_Fewshot

Text Classification
setfit
Safetensors
sentence-transformers
mpnet
generated_from_setfit_trainer
Eval Results
Model card Files Community
idrakFE_Fewshot / README.md
m-aliabbas1's picture
m-aliabbas1
Add SetFit model
d0fb46b
preview code
|
raw
history blame contribute delete
17 kB
metadata
library_name: setfit
tags:
  - setfit
  - sentence-transformers
  - text-classification
  - generated_from_setfit_trainer
metrics:
  - accuracy
widget:
  - text: Please email the information to me.
  - text: Give me a second, please.
  - text: Is it possible to talk to a higher authority?
  - text: Sorry, too busy to chat right now.
  - text: I already own one, thanks.
pipeline_tag: text-classification
inference: true
base_model: sentence-transformers/paraphrase-mpnet-base-v2
model-index:
  - name: SetFit with sentence-transformers/paraphrase-mpnet-base-v2
    results:
      - task:
          type: text-classification
          name: Text Classification
        dataset:
          name: Unknown
          type: unknown
          split: test
        metrics:
          - type: accuracy
            value: 0.9333333333333333
            name: Accuracy

SetFit with sentence-transformers/paraphrase-mpnet-base-v2

This is a SetFit model that can be used for Text Classification. This SetFit model uses sentence-transformers/paraphrase-mpnet-base-v2 as the Sentence Transformer embedding model. A LogisticRegression instance is used for classification.

The model has been trained using an efficient few-shot learning technique that involves:

  1. Fine-tuning a Sentence Transformer with contrastive learning.
  2. Training a classification head with features from the fine-tuned Sentence Transformer.

Model Details

Model Description

Model Sources

Model Labels

Label Examples
do_not_qualify
  • "Your target age group doesn't include me."
  • "I'm outside the age range for this."
  • "I'm not in the age group you're looking for."
can_you_email
  • 'I prefer email, can you write to me?'
  • 'Email is more convenient for me, can you use that?'
  • 'Can you send me the details by email?'
say_again
  • 'Can you repeat that, please?'
  • 'I missed that, can you say it again?'
  • 'Could you please repeat what you just said?'
hold_a_sec
  • 'One moment, please hold.'
  • 'Hang on for a bit, please.'
  • 'Just a minute, please.'
language_barrier
  • 'English is hard for me, ¿puedo hablar en español?'
  • 'I struggle with English, ¿puede ser en español?'
  • "I'm more comfortable in Spanish, ¿podemos continuar en español?"
decline
  • 'wrong'
  • 'Never'
  • "I don't want this, thank you."
transfer_request
  • 'Can you transfer this call to your superior?'
  • 'I need to speak with someone in charge.'
  • 'Can I speak with your manager?'
scam
  • "I'm skeptical, this doesn't sound right."
  • "I'm wary, this feels like a scam."
  • "Are you sure this isn't a scam?"
who_are_you
  • "I would like to know who's calling."
  • "Who's calling, please?"
  • 'Who are you and why are you calling?'
where_did_you_get_my_info
  • 'Can you explain how you got my contact info?'
  • "What's the source of my details you have?"
  • "I didn't give you my number, where did you get it?"
do_not_call
  • "Stop calling me, it's annoying!"
  • "I don't want to be contacted again."
  • "Enough calls, I'm not interested!"
where_are_you_calling_from
  • 'Where are you calling from?'
  • 'From which city or country are you calling?'
  • 'Could you inform me of your current location?'
complain_calls
  • "Too many calls like this, it's irritating."
  • "I've had several calls like this, it's annoying."
  • "I keep getting these calls, it's too much."
busy
  • "Right now isn't good, I'm busy with something."
  • "I'm swamped at the moment, sorry."
  • "I'm busy right now, can't talk."
greetings
  • 'Hi, how can I help you?'
  • 'Hello, what can I help you with today?'
  • 'Hello, yes?'
sorry_greeting
  • "I'm not at my best, what do you need?"
  • "Sorry, it's a bad time, I'm sick."
  • "Not a great time, I'm dealing with a personal issue."
GreetBack
  • 'Doing well, how about yourself?'
  • 'Pretty good, what about you?'
  • "Not bad, and how's it going on your end?"
calling_about
  • 'Why are you calling me?'
  • "What's the matter, why the call?"
  • 'May I know the reason for your call?'
answering_machine
  • "Leave a message and I'll get back to you."
  • "You're speaking to an answering machine, leave a message."
  • "This is an answering machine, I'm not available."
weather
  • "Sunny skies here, what's it like where you are?"
  • "It's a bit cloudy here, is it the same there?"
  • "It's warm here, what about where you are?"
are_you_bot
  • 'Is this a bot calling me?'
  • 'Is this a recorded message or are you real?'
  • 'Are you a live person or a recording?'
affirmation
  • 'yes'
  • "That's true, yes."
  • "Precisely, that's right."
not_interested
  • "This doesn't interest me, sorry."
  • "This offer isn't relevant to my interests."
  • "Thanks, but this isn't something I need."
already
  • "I've made this purchase before."
  • "This isn't new to me, I have it already."
  • "I've been using this for a while now."
abusibve
  • "This is unacceptable, I won't tolerate this!"
  • 'I demand you stop this abusive calling!'
  • "Stop calling me, it's harassment!"

Evaluation

Metrics

Label Accuracy
all 0.9333

Uses

Direct Use for Inference

First install the SetFit library:

pip install setfit

Then you can load this model and run inference.

from setfit import SetFitModel

# Download from the 🤗 Hub
model = SetFitModel.from_pretrained("setfit_model_id")
# Run inference
preds = model("Give me a second, please.")

Training Details

Training Set Metrics

Training set Min Median Max
Word count 1 6.8375 13
Label Training Sample Count
GreetBack 9
abusibve 9
affirmation 10
already 10
answering_machine 8
are_you_bot 8
busy 9
calling_about 8
can_you_email 11
complain_calls 11
decline 10
do_not_call 12
do_not_qualify 9
greetings 8
hold_a_sec 8
language_barrier 10
not_interested 11
say_again 12
scam 9
sorry_greeting 9
transfer_request 8
weather 10
where_are_you_calling_from 9
where_did_you_get_my_info 11
who_are_you 11

Training Hyperparameters

  • batch_size: (8, 8)
  • num_epochs: (3, 3)
  • max_steps: -1
  • sampling_strategy: oversampling
  • num_iterations: 20
  • body_learning_rate: (2e-05, 2e-05)
  • head_learning_rate: 2e-05
  • loss: CosineSimilarityLoss
  • distance_metric: cosine_distance
  • margin: 0.25
  • end_to_end: False
  • use_amp: False
  • warmup_proportion: 0.1
  • seed: 42
  • eval_max_steps: -1
  • load_best_model_at_end: False

Training Results

Epoch Step Training Loss Validation Loss
0.0008 1 0.1054 -
0.0417 50 0.1111 -
0.0833 100 0.0798 -
0.125 150 0.0826 -
0.1667 200 0.0308 -
0.2083 250 0.0324 -
0.25 300 0.0607 -
0.2917 350 0.0042 -
0.3333 400 0.0116 -
0.375 450 0.0049 -
0.4167 500 0.0154 -
0.4583 550 0.0158 -
0.5 600 0.0036 -
0.5417 650 0.001 -
0.5833 700 0.0015 -
0.625 750 0.0012 -
0.6667 800 0.0009 -
0.7083 850 0.0008 -
0.75 900 0.0008 -
0.7917 950 0.0014 -
0.8333 1000 0.0005 -
0.875 1050 0.0027 -
0.9167 1100 0.0007 -
0.9583 1150 0.0008 -
1.0 1200 0.0012 -
1.0417 1250 0.0012 -
1.0833 1300 0.0006 -
1.125 1350 0.0005 -
1.1667 1400 0.0003 -
1.2083 1450 0.0012 -
1.25 1500 0.0006 -
1.2917 1550 0.0008 -
1.3333 1600 0.0008 -
1.375 1650 0.0003 -
1.4167 1700 0.0004 -
1.4583 1750 0.0005 -
1.5 1800 0.0004 -
1.5417 1850 0.0004 -
1.5833 1900 0.0008 -
1.625 1950 0.0004 -
1.6667 2000 0.0004 -
1.7083 2050 0.0021 -
1.75 2100 0.0004 -
1.7917 2150 0.0002 -
1.8333 2200 0.0006 -
1.875 2250 0.0004 -
1.9167 2300 0.0006 -
1.9583 2350 0.0006 -
2.0 2400 0.0003 -
2.0417 2450 0.0002 -
2.0833 2500 0.0002 -
2.125 2550 0.0003 -
2.1667 2600 0.0004 -
2.2083 2650 0.0004 -
2.25 2700 0.0005 -
2.2917 2750 0.0005 -
2.3333 2800 0.0005 -
2.375 2850 0.0007 -
2.4167 2900 0.0002 -
2.4583 2950 0.0003 -
2.5 3000 0.0004 -
2.5417 3050 0.0002 -
2.5833 3100 0.0004 -
2.625 3150 0.0002 -
2.6667 3200 0.0002 -
2.7083 3250 0.0003 -
2.75 3300 0.0002 -
2.7917 3350 0.0002 -
2.8333 3400 0.0003 -
2.875 3450 0.0002 -
2.9167 3500 0.0002 -
2.9583 3550 0.0002 -
3.0 3600 0.0002 -

Framework Versions

  • Python: 3.10.13
  • SetFit: 1.0.1
  • Sentence Transformers: 2.2.2
  • Transformers: 4.35.0
  • PyTorch: 2.1.0
  • Datasets: 2.14.6
  • Tokenizers: 0.14.1

Citation

BibTeX 

@article{https://doi.org/10.48550/arxiv.2209.11055,
    doi = {10.48550/ARXIV.2209.11055},
    url = {https://arxiv.org/abs/2209.11055},
    author = {Tunstall, Lewis and Reimers, Nils and Jo, Unso Eun Seo and Bates, Luke and Korat, Daniel and Wasserblat, Moshe and Pereg, Oren},
    keywords = {Computation and Language (cs.CL), FOS: Computer and information sciences, FOS: Computer and information sciences},
    title = {Efficient Few-Shot Learning Without Prompts},
    publisher = {arXiv},
    year = {2022},
    copyright = {Creative Commons Attribution 4.0 International}
}