GIZ
/

Edit model card

SECTOR-multilabel-bge

This model is a fine-tuned version of BAAI/bge-base-en-v1.5 on the Policy-Classification dataset.

The loss function BCEWithLogitsLoss is modified with pos_weight to focus on recall, therefore instead of loss the evaluation metrics are used to assess the model performance during training It achieves the following results on the evaluation set:

  • Loss: 0.6114
  • Precision-micro: 0.6428
  • Precision-samples: 0.7488
  • Precision-weighted: 0.6519
  • Recall-micro: 0.7855
  • Recall-samples: 0.8627
  • Recall-weighted: 0.7855
  • F1-micro: 0.7071
  • F1-samples: 0.7638
  • F1-weighted: 0.7109

Model description

The purpose of this model is to predict multiple labels simultaneously from a given input data. Specifically, the model will predict Sector labels - Agriculture,Buildings, Coastal Zone,Cross-Cutting Area,Disaster Risk Management (DRM),Economy-wide,Education,Energy,Environment,Health,Industries,LULUCF/Forestry,Social Development,Tourism, Transport,Urban,Waste,Water

Intended uses & limitations

More information needed

Training and evaluation data

  • Training Dataset: 10123

    Class Positive Count of Class
    Agriculture 2235
    Buildings 169
    Coastal Zone 698
    Cross-Cutting Area 1853
    Disaster Risk Management (DRM) 814
    Economy-wide 873
    Education 180
    Energy 2847
    Environment 905
    Health 662
    Industries 419
    LULUCF/Forestry 1861
    Social Development 507
    Tourism 192
    Transport 1173
    Urban 558
    Waste 714
    Water 1207
  • Validation Dataset: 936

    Class Positive Count of Class
    Agriculture 200
    Buildings 18
    Coastal Zone 71
    Cross-Cutting Area 180
    Disaster Risk Management (DRM) 85
    Economy-wide 85
    Education 23
    Energy 254
    Environment 91
    Health 68
    Industries 41
    LULUCF/Forestry 193
    Social Development 56
    Tourism 28
    Transport 107
    Urban 51
    Waste 59
    Water 106

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

  • learning_rate: 7.04e-05
  • train_batch_size: 16
  • eval_batch_size: 16
  • seed: 42
  • optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
  • lr_scheduler_type: cosine
  • lr_scheduler_warmup_steps: 300
  • num_epochs: 7

Training results

Training Loss Epoch Step Validation Loss Precision-micro Precision-samples Precision-weighted Recall-micro Recall-samples Recall-weighted F1-micro F1-samples F1-weighted
0.7077 1.0 633 0.5490 0.4226 0.5465 0.4954 0.8211 0.8908 0.8211 0.5580 0.6243 0.5977
0.4546 2.0 1266 0.5009 0.4899 0.6127 0.5202 0.8438 0.9023 0.8438 0.6199 0.6822 0.6366
0.3105 3.0 1899 0.4947 0.5005 0.6593 0.5317 0.8508 0.8970 0.8508 0.6303 0.7125 0.6474
0.2044 4.0 2532 0.5430 0.5757 0.7044 0.5970 0.8106 0.8801 0.8106 0.6733 0.7379 0.6834
0.1314 5.0 3165 0.5633 0.6132 0.7385 0.6271 0.8065 0.8772 0.8065 0.6967 0.7606 0.7032
0.0892 6.0 3798 0.6073 0.6425 0.7499 0.6545 0.7844 0.8610 0.7844 0.7064 0.7634 0.7113
0.0721 7.0 4431 0.6114 0.6428 0.7488 0.6519 0.7855 0.8627 0.7855 0.7071 0.7638 0.7109
label precision recall f1-score support
Agriculture 0.740 0.840 0.786 200
Buildings 0.535 0.833 0.652 18
Coastal Zone 0.579 0.718 0.641 71
Cross-Cutting Area 0.551 0.738 0.631 180
Disaster Risk Management (DRM) 0.642 0.717 0.67 85
Economy-wide 0.401 0.600 0.481 85
Education 0.652 0.652 0.652 23
Energy 0.771 0.862 0.814 254
Environment 0.539 0.747 0.626 91
Health 0.743 0.808 0.774 68
Industries 0.648 0.853 0.736 41
LULUCF/Forestry 0.728 0.849 0.784 193
Social Development 0.661 0.767 0.710 56
Tourism 0.586 0.607 0.596 28
Transport 0.715 0.822 0.765 107
Urban 0.414 0.568 0.479 51
Waste 0.662 0.898 0.762 59
Water 0.601 .783 0.680 106

Environmental Impact

Carbon emissions were measured using CodeCarbon.

  • Carbon Emitted: 0.05819 kg of CO2
  • Hours Used: 1.43 hours

Training Hardware

  • On Cloud: yes
  • GPU Model: 1 x Tesla T4
  • CPU Model: Intel(R) Xeon(R) CPU @ 2.00GHz
  • RAM Size: 12.67 GB

Framework versions

  • Transformers 4.38.1
  • Pytorch 2.1.0+cu121
  • Datasets 2.18.0
  • Tokenizers 0.15.2
Downloads last month
22
Safetensors
Model size
109M params
Tensor type
F32
·
Inference Examples
This model does not have enough activity to be deployed to Inference API (serverless) yet. Increase its social visibility and check back later, or deploy to Inference Endpoints (dedicated) instead.

Model tree for GIZ/SECTOR-multilabel-bge_f

Finetuned
(247)
this model

Dataset used to train GIZ/SECTOR-multilabel-bge_f