Hugging Face's logo

twright8
/
setfit_lobbying_classifier

Text Classification
setfit
Safetensors
sentence-transformers
bert
generated_from_setfit_trainer
Eval Results
Model card Files Community
setfit_lobbying_classifier / README.md
twright8's picture
twright8
Update README.md
7924738 verified
preview code
|
raw
history blame contribute delete
36.1 kB
metadata
base_model: BAAI/bge-base-en-v1.5
library_name: setfit
metrics:
  - f1
  - accuracy
pipeline_tag: text-classification
tags:
  - setfit
  - sentence-transformers
  - text-classification
  - generated_from_setfit_trainer
widget:
  - text: Discussion on recent report publication
  - text: Growth
  - text: >-
      The roundtable was arranged in order to provide an overview of the work of
      Alliance members and promote international development policy positions to
      the Scottish Conservatives. During the meeting we presented the work of
      SCIAF and its campaign for a world leading climate change response. In
      particular SCIAF explained how climate change is already affecting some of
      the poorest communities in the world and is therefore a central concern
      for international development. We argued that Scotland needs to do what it
      can to mitigate climate change.
  - text: >-
      To introduce Energy UK discuss the energy industries contribution to
      tackling climate change and discuss stage 1 of theClimate Change
      (Emissions Reduction Targets) (Scotland) Bill. Also discussed the Scottish
      Government's ambition on electric vehicles and the role of the energy
      industry in a successful roll out.
  - text: >-
      To discuss our key asks on the Climate Change (Emissions Reduction
      Targets) (Scotland) Bill in advance of Stage 2 including support for
      amendments on regional land use partnerships and land use strategy as
      means to deliver climate mitigation for land.
inference: true
model-index:
  - name: SetFit with BAAI/bge-base-en-v1.5
    results:
      - task:
          type: text-classification
          name: Text Classification
        dataset:
          name: Unknown
          type: unknown
          split: test
        metrics:
          - type: f1
            value: 0.9667149059334297
            name: F1
          - type: accuracy
            value: 0.9420654911838791
            name: Accuracy

SetFit with BAAI/bge-base-en-v1.5

This is a SetFit model that can be used for Text Classification. This SetFit model uses BAAI/bge-base-en-v1.5 as the Sentence Transformer embedding model. A SetFitHead instance is used for classification.

The model has been trained using an efficient few-shot learning technique that involves:

  1. Fine-tuning a Sentence Transformer with contrastive learning.
  2. Training a classification head with features from the fine-tuned Sentence Transformer.

undefined = Health 1 = Housing 2 = Defence 3 = Climate

Model Details

Model Description

Model Sources

Evaluation

Metrics

Label F1 Accuracy
all 0.9667 0.9421

Uses

Direct Use for Inference

First install the SetFit library:

pip install setfit

Then you can load this model and run inference.

from setfit import SetFitModel

# Download from the 🤗 Hub
model = SetFitModel.from_pretrained("twright8/setfit_lobbying_classifier")
# Run inference
preds = model("Growth")

Training Details

Training Set Metrics

Training set Min Median Max
Word count 1 39.4538 282

Training Hyperparameters

  • batch_size: (16, 2)
  • num_epochs: (4, 9)
  • max_steps: -1
  • sampling_strategy: undersampling
  • body_learning_rate: (1.0797496673911536e-05, 3.457046714445997e-05)
  • head_learning_rate: 0.0004470582121407239
  • loss: CoSENTLoss
  • distance_metric: cosine_distance
  • margin: 0.25
  • end_to_end: True
  • use_amp: False
  • warmup_proportion: 0.1
  • seed: 42
  • eval_max_steps: -1
  • load_best_model_at_end: True

Training Results

Epoch Step Training Loss Validation Loss
0.0002 1 2.097 -
0.0077 50 8.5514 -
0.0155 100 3.5635 -
0.0232 150 2.9266 -
0.0310 200 2.1173 -
0.0387 250 3.1002 -
0.0465 300 3.6942 -
0.0542 350 3.4905 -
0.0620 400 4.0804 -
0.0697 450 1.6071 -
0.0774 500 2.3018 -
0.0852 550 2.3876 -
0.0929 600 0.2511 -
0.1007 650 0.2435 -
0.1084 700 2.2596 -
0.1162 750 1.121 -
0.1239 800 0.0907 -
0.1317 850 0.2172 -
0.1394 900 3.06 -
0.1471 950 0.0074 -
0.1549 1000 0.457 -
0.1626 1050 0.0575 -
0.1704 1100 0.0002 -
0.1781 1150 0.0003 -
0.1859 1200 0.0047 -
0.1936 1250 0.0004 -
0.2014 1300 0.0006 -
0.2091 1350 0.0027 -
0.2169 1400 0.0004 -
0.2246 1450 0.0009 -
0.2323 1500 0.0006 -
0.2401 1550 0.0003 -
0.2478 1600 0.0077 -
0.2556 1650 0.0004 -
0.2633 1700 0.0003 -
0.2711 1750 0.0005 -
0.2788 1800 0.0004 -
0.2866 1850 0.0007 -
0.2943 1900 0.0009 -
0.3020 1950 0.0062 -
0.3098 2000 0.0003 -
0.3175 2050 0.0001 -
0.3253 2100 0.0685 -
0.3330 2150 0.0008 -
0.3408 2200 0.0 -
0.3485 2250 0.0004 -
0.3563 2300 0.0004 -
0.3640 2350 0.0002 -
0.3717 2400 0.0001 -
0.3795 2450 0.0004 -
0.3872 2500 0.0004 -
0.3950 2550 0.0001 -
0.4027 2600 0.0001 -
0.4105 2650 0.0001 -
0.4182 2700 0.0005 -
0.4260 2750 0.0002 -
0.4337 2800 0.0001 -
0.4414 2850 0.0003 -
0.4492 2900 0.0005 -
0.4569 2950 0.0014 -
0.4647 3000 0.0001 -
0.4724 3050 0.0001 -
0.4802 3100 0.0002 -
0.4879 3150 0.0 -
0.4957 3200 0.0006 -
0.5034 3250 0.0 -
0.5112 3300 0.0 -
0.5189 3350 0.0002 -
0.5266 3400 0.0001 -
0.5344 3450 0.0006 -
0.5421 3500 0.0002 -
0.5499 3550 0.0001 -
0.5576 3600 0.0001 -
0.5654 3650 0.0001 -
0.5731 3700 0.0 -
0.5809 3750 0.0002 -
0.5886 3800 0.0 -
0.5963 3850 0.0044 -
0.6041 3900 0.0002 -
0.6118 3950 0.0001 -
0.6196 4000 0.0003 -
0.6273 4050 0.0005 -
0.6351 4100 0.0002 -
0.6428 4150 0.0 -
0.6506 4200 0.0003 -
0.6583 4250 0.0 -
0.6660 4300 0.0001 -
0.6738 4350 0.0 -
0.6815 4400 0.0008 -
0.6893 4450 0.0 -
0.6970 4500 0.0004 -
0.7048 4550 0.0001 -
0.7125 4600 0.0 -
0.7203 4650 0.0 -
0.7280 4700 0.0 -
0.7357 4750 0.0001 -
0.7435 4800 0.0001 -
0.7512 4850 0.001 -
0.7590 4900 0.0001 -
0.7667 4950 0.0 -
0.7745 5000 0.0001 -
0.7822 5050 0.0 -
0.7900 5100 0.0018 -
0.7977 5150 0.0001 -
0.8055 5200 0.0 -
0.8132 5250 0.0003 -
0.8209 5300 0.0003 -
0.8287 5350 0.0003 -
0.8364 5400 0.0001 -
0.8442 5450 0.0001 -
0.8519 5500 0.0001 -
0.8597 5550 0.0001 -
0.8674 5600 0.0001 -
0.8752 5650 0.0 -
0.8829 5700 0.0003 -
0.8906 5750 0.0003 -
0.8984 5800 0.0001 -
0.9061 5850 0.0001 -
0.9139 5900 0.0002 -
0.9216 5950 0.0 -
0.9294 6000 0.0001 -
0.9371 6050 0.0 -
0.9449 6100 0.0 -
0.9526 6150 0.0001 -
0.9603 6200 0.0 -
0.9681 6250 0.0001 -
0.9758 6300 0.0002 -
0.9836 6350 0.0 -
0.9913 6400 0.0 -
0.9991 6450 0.0002 -
1.0 6456 - 1.3837
1.0068 6500 0.0001 -
1.0146 6550 0.0001 -
1.0223 6600 0.0002 -
1.0300 6650 0.0001 -
1.0378 6700 0.0005 -
1.0455 6750 0.0001 -
1.0533 6800 0.0001 -
1.0610 6850 0.0 -
1.0688 6900 0.0 -
1.0765 6950 0.0009 -
1.0843 7000 0.0 -
1.0920 7050 0.0032 -
1.0998 7100 0.0001 -
1.1075 7150 0.0001 -
1.1152 7200 0.0001 -
1.1230 7250 0.0 -
1.1307 7300 0.0001 -
1.1385 7350 0.0 -
1.1462 7400 0.0 -
1.1540 7450 0.0002 -
1.1617 7500 0.0 -
1.1695 7550 0.0427 -
1.1772 7600 0.0 -
1.1849 7650 0.0 -
1.1927 7700 0.0 -
1.2004 7750 0.0002 -
1.2082 7800 0.0 -
1.2159 7850 0.0 -
1.2237 7900 0.0 -
1.2314 7950 0.0 -
1.2392 8000 0.0001 -
1.2469 8050 0.0 -
1.2546 8100 0.0001 -
1.2624 8150 0.0 -
1.2701 8200 0.0 -
1.2779 8250 0.0 -
1.2856 8300 0.0 -
1.2934 8350 0.0 -
1.3011 8400 0.0 -
1.3089 8450 0.0 -
1.3166 8500 0.0 -
1.3243 8550 0.0001 -
1.3321 8600 0.0 -
1.3398 8650 0.0002 -
1.3476 8700 0.0 -
1.3553 8750 0.0006 -
1.3631 8800 0.0 -
1.3708 8850 0.0 -
1.3786 8900 0.0001 -
1.3863 8950 0.0 -
1.3941 9000 0.0001 -
1.4018 9050 0.0 -
1.4095 9100 0.0002 -
1.4173 9150 0.0 -
1.4250 9200 0.0 -
1.4328 9250 0.0 -
1.4405 9300 0.0 -
1.4483 9350 0.0 -
1.4560 9400 0.0 -
1.4638 9450 0.0 -
1.4715 9500 0.0 -
1.4792 9550 0.0 -
1.4870 9600 0.0 -
1.4947 9650 0.0005 -
1.5025 9700 0.0 -
1.5102 9750 0.0001 -
1.5180 9800 0.0001 -
1.5257 9850 0.0001 -
1.5335 9900 0.0 -
1.5412 9950 0.0 -
1.5489 10000 0.0 -
1.5567 10050 0.0 -
1.5644 10100 0.0001 -
1.5722 10150 0.0 -
1.5799 10200 0.0002 -
1.5877 10250 0.0001 -
1.5954 10300 0.0005 -
1.6032 10350 0.0 -
1.6109 10400 0.0 -
1.6186 10450 0.0003 -
1.6264 10500 0.0002 -
1.6341 10550 0.0 -
1.6419 10600 0.0 -
1.6496 10650 0.0001 -
1.6574 10700 0.0002 -
1.6651 10750 0.0002 -
1.6729 10800 0.0054 -
1.6806 10850 0.0005 -
1.6884 10900 0.0001 -
1.6961 10950 0.0 -
1.7038 11000 0.0 -
1.7116 11050 0.0001 -
1.7193 11100 0.0001 -
1.7271 11150 0.0 -
1.7348 11200 0.0001 -
1.7426 11250 0.0 -
1.7503 11300 0.0001 -
1.7581 11350 0.0004 -
1.7658 11400 0.0 -
1.7735 11450 0.0001 -
1.7813 11500 0.0 -
1.7890 11550 0.0 -
1.7968 11600 0.0 -
1.8045 11650 0.0 -
1.8123 11700 0.0001 -
1.8200 11750 0.0002 -
1.8278 11800 0.0 -
1.8355 11850 0.0001 -
1.8432 11900 0.0 -
1.8510 11950 0.0001 -
1.8587 12000 0.0 -
1.8665 12050 0.0 -
1.8742 12100 0.0 -
1.8820 12150 0.0001 -
1.8897 12200 0.0 -
1.8975 12250 0.0 -
1.9052 12300 0.0 -
1.9129 12350 0.0 -
1.9207 12400 0.0 -
1.9284 12450 0.0 -
1.9362 12500 0.0 -
1.9439 12550 0.0003 -
1.9517 12600 0.0001 -
1.9594 12650 0.0 -
1.9672 12700 0.0001 -
1.9749 12750 0.0 -
1.9827 12800 0.0 -
1.9904 12850 0.0 -
1.9981 12900 0.0001 -
2.0 12912 - 2.611
2.0059 12950 0.0 -
2.0136 13000 0.0001 -
2.0214 13050 0.0001 -
2.0291 13100 0.0 -
2.0369 13150 0.0 -
2.0446 13200 0.0001 -
2.0524 13250 0.0 -
2.0601 13300 0.0002 -
2.0678 13350 0.0 -
2.0756 13400 0.0 -
2.0833 13450 0.0001 -
2.0911 13500 0.0001 -
2.0988 13550 0.0003 -
2.1066 13600 0.0 -
2.1143 13650 0.0001 -
2.1221 13700 0.0001 -
2.1298 13750 0.0001 -
2.1375 13800 0.0001 -
2.1453 13850 0.0 -
2.1530 13900 0.0 -
2.1608 13950 0.0 -
2.1685 14000 0.0 -
2.1763 14050 0.0 -
2.1840 14100 0.0001 -
2.1918 14150 0.0 -
2.1995 14200 0.0 -
2.2072 14250 0.0001 -
2.2150 14300 0.0 -
2.2227 14350 0.0 -
2.2305 14400 0.0004 -
2.2382 14450 0.0001 -
2.2460 14500 0.0 -
2.2537 14550 0.0003 -
2.2615 14600 0.0 -
2.2692 14650 0.0001 -
2.2770 14700 0.0001 -
2.2847 14750 0.0 -
2.2924 14800 0.0 -
2.3002 14850 0.0005 -
2.3079 14900 0.0 -
2.3157 14950 0.0002 -
2.3234 15000 0.0 -
2.3312 15050 0.0 -
2.3389 15100 0.0001 -
2.3467 15150 0.0001 -
2.3544 15200 0.0002 -
2.3621 15250 0.0001 -
2.3699 15300 0.0 -
2.3776 15350 0.0 -
2.3854 15400 0.0002 -
2.3931 15450 0.0003 -
2.4009 15500 0.0 -
2.4086 15550 0.0 -
2.4164 15600 0.0 -
2.4241 15650 0.0001 -
2.4318 15700 0.0 -
2.4396 15750 0.0 -
2.4473 15800 0.0002 -
2.4551 15850 0.0 -
2.4628 15900 0.0 -
2.4706 15950 0.0 -
2.4783 16000 0.0 -
2.4861 16050 0.0001 -
2.4938 16100 0.0 -
2.5015 16150 0.0 -
2.5093 16200 0.0 -
2.5170 16250 0.0 -
2.5248 16300 0.0 -
2.5325 16350 0.0 -
2.5403 16400 0.0 -
2.5480 16450 0.0 -
2.5558 16500 0.0 -
2.5635 16550 0.0001 -
2.5713 16600 0.0 -
2.5790 16650 0.0 -
2.5867 16700 0.0 -
2.5945 16750 0.0 -
2.6022 16800 0.0009 -
2.6100 16850 0.0001 -
2.6177 16900 0.0 -
2.6255 16950 0.0001 -
2.6332 17000 0.0 -
2.6410 17050 0.0 -
2.6487 17100 0.0001 -
2.6564 17150 0.0 -
2.6642 17200 0.0 -
2.6719 17250 0.0 -
2.6797 17300 0.0 -
2.6874 17350 0.0004 -
2.6952 17400 0.0 -
2.7029 17450 0.0 -
2.7107 17500 0.0 -
2.7184 17550 0.0 -
2.7261 17600 0.0 -
2.7339 17650 0.0 -
2.7416 17700 0.0001 -
2.7494 17750 0.0 -
2.7571 17800 0.0 -
2.7649 17850 0.0001 -
2.7726 17900 0.0 -
2.7804 17950 0.0001 -
2.7881 18000 0.0001 -
2.7958 18050 0.0 -
2.8036 18100 0.0 -
2.8113 18150 0.0 -
2.8191 18200 0.0 -
2.8268 18250 0.0 -
2.8346 18300 0.0001 -
2.8423 18350 0.0 -
2.8501 18400 0.0 -
2.8578 18450 0.0 -
2.8656 18500 0.0 -
2.8733 18550 0.0 -
2.8810 18600 0.0 -
2.8888 18650 0.0 -
2.8965 18700 0.0 -
2.9043 18750 0.0 -
2.9120 18800 0.0001 -
2.9198 18850 0.0 -
2.9275 18900 0.0 -
2.9353 18950 0.0 -
2.9430 19000 0.0 -
2.9507 19050 0.0 -
2.9585 19100 0.0 -
2.9662 19150 0.0 -
2.9740 19200 0.0 -
2.9817 19250 0.0003 -
2.9895 19300 0.0001 -
2.9972 19350 0.0 -
3.0 19368 - 2.0845
3.0050 19400 0.0 -
3.0127 19450 0.0001 -
3.0204 19500 0.0 -
3.0282 19550 0.0 -
3.0359 19600 0.0 -
3.0437 19650 0.0 -
3.0514 19700 0.0 -
3.0592 19750 0.0 -
3.0669 19800 0.0001 -
3.0747 19850 0.0 -
3.0824 19900 0.0 -
3.0901 19950 0.0001 -
3.0979 20000 0.0 -
3.1056 20050 0.0 -
3.1134 20100 0.0 -
3.1211 20150 0.0001 -
3.1289 20200 0.0 -
3.1366 20250 0.0 -
3.1444 20300 0.0 -
3.1521 20350 0.0 -
3.1599 20400 0.0 -
3.1676 20450 0.0001 -
3.1753 20500 0.0 -
3.1831 20550 0.0001 -
3.1908 20600 0.0 -
3.1986 20650 0.0 -
3.2063 20700 0.0 -
3.2141 20750 0.0 -
3.2218 20800 0.0 -
3.2296 20850 0.0003 -
3.2373 20900 0.0 -
3.2450 20950 0.0 -
3.2528 21000 0.0 -
3.2605 21050 0.0 -
3.2683 21100 0.0001 -
3.2760 21150 0.0001 -
3.2838 21200 0.0 -
3.2915 21250 0.0 -
3.2993 21300 0.0 -
3.3070 21350 0.0 -
3.3147 21400 0.0 -
3.3225 21450 0.0001 -
3.3302 21500 0.0 -
3.3380 21550 0.0 -
3.3457 21600 0.0 -
3.3535 21650 0.0 -
3.3612 21700 0.0 -
3.3690 21750 0.0 -
3.3767 21800 0.0 -
3.3844 21850 0.0 -
3.3922 21900 0.0001 -
3.3999 21950 0.0 -
3.4077 22000 0.0 -
3.4154 22050 0.0001 -
3.4232 22100 0.0 -
3.4309 22150 0.0001 -
3.4387 22200 0.0 -
3.4464 22250 0.0 -
3.4542 22300 0.0 -
3.4619 22350 0.0001 -
3.4696 22400 0.0 -
3.4774 22450 0.0 -
3.4851 22500 0.0 -
3.4929 22550 0.0001 -
3.5006 22600 0.0002 -
3.5084 22650 0.0001 -
3.5161 22700 0.0 -
3.5239 22750 0.0001 -
3.5316 22800 0.0 -
3.5393 22850 0.0 -
3.5471 22900 0.0001 -
3.5548 22950 0.0 -
3.5626 23000 0.0 -
3.5703 23050 0.0 -
3.5781 23100 0.0 -
3.5858 23150 0.0001 -
3.5936 23200 0.0 -
3.6013 23250 0.0001 -
3.6090 23300 0.0001 -
3.6168 23350 0.0 -
3.6245 23400 0.0003 -
3.6323 23450 0.0 -
3.6400 23500 0.0 -
3.6478 23550 0.0001 -
3.6555 23600 0.0 -
3.6633 23650 0.0 -
3.6710 23700 0.0 -
3.6787 23750 0.0001 -
3.6865 23800 0.0 -
3.6942 23850 0.0001 -
3.7020 23900 0.0002 -
3.7097 23950 0.0 -
3.7175 24000 0.0 -
3.7252 24050 0.0 -
3.7330 24100 0.0 -
3.7407 24150 0.0001 -
3.7485 24200 0.0 -
3.7562 24250 0.0 -
3.7639 24300 0.0 -
3.7717 24350 0.0 -
3.7794 24400 0.0 -
3.7872 24450 0.0 -
3.7949 24500 0.0001 -
3.8027 24550 0.0001 -
3.8104 24600 0.0 -
3.8182 24650 0.0 -
3.8259 24700 0.0 -
3.8336 24750 0.0 -
3.8414 24800 0.0001 -
3.8491 24850 0.0 -
3.8569 24900 0.0 -
3.8646 24950 0.0 -
3.8724 25000 0.0 -
3.8801 25050 0.0 -
3.8879 25100 0.0 -
3.8956 25150 0.0001 -
3.9033 25200 0.0 -
3.9111 25250 0.0002 -
3.9188 25300 0.0001 -
3.9266 25350 0.0 -
3.9343 25400 0.0 -
3.9421 25450 0.0 -
3.9498 25500 0.0001 -
3.9576 25550 0.0 -
3.9653 25600 0.0 -
3.9730 25650 0.0001 -
3.9808 25700 0.0 -
3.9885 25750 0.0 -
3.9963 25800 0.0 -
4.0 25824 - 2.3576
  • The bold row denotes the saved checkpoint.

Framework Versions

  • Python: 3.10.12
  • SetFit: 1.0.3
  • Sentence Transformers: 3.0.1
  • Transformers: 4.39.0
  • PyTorch: 2.3.0+cu121
  • Datasets: 2.20.0
  • Tokenizers: 0.15.2

Citation

BibTeX 

@article{https://doi.org/10.48550/arxiv.2209.11055,
    doi = {10.48550/ARXIV.2209.11055},
    url = {https://arxiv.org/abs/2209.11055},
    author = {Tunstall, Lewis and Reimers, Nils and Jo, Unso Eun Seo and Bates, Luke and Korat, Daniel and Wasserblat, Moshe and Pereg, Oren},
    keywords = {Computation and Language (cs.CL), FOS: Computer and information sciences, FOS: Computer and information sciences},
    title = {Efficient Few-Shot Learning Without Prompts},
    publisher = {arXiv},
    year = {2022},
    copyright = {Creative Commons Attribution 4.0 International}
}