SECTOR-multilabel-bge
This model is a fine-tuned version of BAAI/bge-base-en-v1.5 on the Policy-Classification dataset.
The loss function BCEWithLogitsLoss is modified with pos_weight to focus on recall, therefore instead of loss the evaluation metrics are used to assess the model performance during training It achieves the following results on the evaluation set:
- Loss: 0.6114
- Precision-micro: 0.6428
- Precision-samples: 0.7488
- Precision-weighted: 0.6519
- Recall-micro: 0.7855
- Recall-samples: 0.8627
- Recall-weighted: 0.7855
- F1-micro: 0.7071
- F1-samples: 0.7638
- F1-weighted: 0.7109
Model description
The purpose of this model is to predict multiple labels simultaneously from a given input data. Specifically, the model will predict Sector labels - Agriculture,Buildings, Coastal Zone,Cross-Cutting Area,Disaster Risk Management (DRM),Economy-wide,Education,Energy,Environment,Health,Industries,LULUCF/Forestry,Social Development,Tourism, Transport,Urban,Waste,Water
Intended uses & limitations
More information needed
Training and evaluation data
Training Dataset: 10123
Class Positive Count of Class Agriculture 2235 Buildings 169 Coastal Zone 698 Cross-Cutting Area 1853 Disaster Risk Management (DRM) 814 Economy-wide 873 Education 180 Energy 2847 Environment 905 Health 662 Industries 419 LULUCF/Forestry 1861 Social Development 507 Tourism 192 Transport 1173 Urban 558 Waste 714 Water 1207 Validation Dataset: 936
Class Positive Count of Class Agriculture 200 Buildings 18 Coastal Zone 71 Cross-Cutting Area 180 Disaster Risk Management (DRM) 85 Economy-wide 85 Education 23 Energy 254 Environment 91 Health 68 Industries 41 LULUCF/Forestry 193 Social Development 56 Tourism 28 Transport 107 Urban 51 Waste 59 Water 106
Training procedure
Training hyperparameters
The following hyperparameters were used during training:
- learning_rate: 7.04e-05
- train_batch_size: 16
- eval_batch_size: 16
- seed: 42
- optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
- lr_scheduler_type: cosine
- lr_scheduler_warmup_steps: 300
- num_epochs: 7
Training results
Training Loss | Epoch | Step | Validation Loss | Precision-micro | Precision-samples | Precision-weighted | Recall-micro | Recall-samples | Recall-weighted | F1-micro | F1-samples | F1-weighted |
---|---|---|---|---|---|---|---|---|---|---|---|---|
0.7077 | 1.0 | 633 | 0.5490 | 0.4226 | 0.5465 | 0.4954 | 0.8211 | 0.8908 | 0.8211 | 0.5580 | 0.6243 | 0.5977 |
0.4546 | 2.0 | 1266 | 0.5009 | 0.4899 | 0.6127 | 0.5202 | 0.8438 | 0.9023 | 0.8438 | 0.6199 | 0.6822 | 0.6366 |
0.3105 | 3.0 | 1899 | 0.4947 | 0.5005 | 0.6593 | 0.5317 | 0.8508 | 0.8970 | 0.8508 | 0.6303 | 0.7125 | 0.6474 |
0.2044 | 4.0 | 2532 | 0.5430 | 0.5757 | 0.7044 | 0.5970 | 0.8106 | 0.8801 | 0.8106 | 0.6733 | 0.7379 | 0.6834 |
0.1314 | 5.0 | 3165 | 0.5633 | 0.6132 | 0.7385 | 0.6271 | 0.8065 | 0.8772 | 0.8065 | 0.6967 | 0.7606 | 0.7032 |
0.0892 | 6.0 | 3798 | 0.6073 | 0.6425 | 0.7499 | 0.6545 | 0.7844 | 0.8610 | 0.7844 | 0.7064 | 0.7634 | 0.7113 |
0.0721 | 7.0 | 4431 | 0.6114 | 0.6428 | 0.7488 | 0.6519 | 0.7855 | 0.8627 | 0.7855 | 0.7071 | 0.7638 | 0.7109 |
label | precision | recall | f1-score | support |
---|---|---|---|---|
Agriculture | 0.740 | 0.840 | 0.786 | 200 |
Buildings | 0.535 | 0.833 | 0.652 | 18 |
Coastal Zone | 0.579 | 0.718 | 0.641 | 71 |
Cross-Cutting Area | 0.551 | 0.738 | 0.631 | 180 |
Disaster Risk Management (DRM) | 0.642 | 0.717 | 0.67 | 85 |
Economy-wide | 0.401 | 0.600 | 0.481 | 85 |
Education | 0.652 | 0.652 | 0.652 | 23 |
Energy | 0.771 | 0.862 | 0.814 | 254 |
Environment | 0.539 | 0.747 | 0.626 | 91 |
Health | 0.743 | 0.808 | 0.774 | 68 |
Industries | 0.648 | 0.853 | 0.736 | 41 |
LULUCF/Forestry | 0.728 | 0.849 | 0.784 | 193 |
Social Development | 0.661 | 0.767 | 0.710 | 56 |
Tourism | 0.586 | 0.607 | 0.596 | 28 |
Transport | 0.715 | 0.822 | 0.765 | 107 |
Urban | 0.414 | 0.568 | 0.479 | 51 |
Waste | 0.662 | 0.898 | 0.762 | 59 |
Water | 0.601 | .783 | 0.680 | 106 |
Environmental Impact
Carbon emissions were measured using CodeCarbon.
- Carbon Emitted: 0.05819 kg of CO2
- Hours Used: 1.43 hours
Training Hardware
- On Cloud: yes
- GPU Model: 1 x Tesla T4
- CPU Model: Intel(R) Xeon(R) CPU @ 2.00GHz
- RAM Size: 12.67 GB
Framework versions
- Transformers 4.38.1
- Pytorch 2.1.0+cu121
- Datasets 2.18.0
- Tokenizers 0.15.2
- Downloads last month
- 22
Model tree for GIZ/SECTOR-multilabel-bge_f
Base model
BAAI/bge-base-en-v1.5