|
--- |
|
license: apache-2.0 |
|
base_model: allenai/longformer-base-4096 |
|
tags: |
|
- generated_from_trainer |
|
datasets: |
|
- essays_su_g |
|
metrics: |
|
- accuracy |
|
model-index: |
|
- name: longformer-sep_tok |
|
results: |
|
- task: |
|
name: Token Classification |
|
type: token-classification |
|
dataset: |
|
name: essays_su_g |
|
type: essays_su_g |
|
config: sep_tok |
|
split: train[80%:100%] |
|
args: sep_tok |
|
metrics: |
|
- name: Accuracy |
|
type: accuracy |
|
value: 0.8962127587948157 |
|
--- |
|
|
|
<!-- This model card has been generated automatically according to the information the Trainer had access to. You |
|
should probably proofread and complete it, then remove this comment. --> |
|
|
|
# longformer-sep_tok |
|
|
|
This model is a fine-tuned version of [allenai/longformer-base-4096](https://huggingface.co/allenai/longformer-base-4096) on the essays_su_g dataset. |
|
It achieves the following results on the evaluation set: |
|
- Loss: 0.3310 |
|
- Claim: {'precision': 0.6261209473442171, 'recall': 0.6533109404990403, 'f1-score': 0.6394270282963485, 'support': 4168.0} |
|
- Majorclaim: {'precision': 0.9034416826003824, 'recall': 0.8782527881040892, 'f1-score': 0.8906691800188501, 'support': 2152.0} |
|
- O: {'precision': 0.9998231340643792, 'recall': 0.9994695898161244, 'f1-score': 0.9996463306808134, 'support': 11312.0} |
|
- Premise: {'precision': 0.8951990632318502, 'recall': 0.8865236478091609, 'f1-score': 0.8908402347163843, 'support': 12073.0} |
|
- Accuracy: 0.8962 |
|
- Macro avg: {'precision': 0.8561462068102073, 'recall': 0.8543892415571037, 'f1-score': 0.8551456934280991, 'support': 29705.0} |
|
- Weighted avg: {'precision': 0.8978830564693184, 'recall': 0.8962127587948157, 'f1-score': 0.8969858736149474, 'support': 29705.0} |
|
|
|
## Model description |
|
|
|
More information needed |
|
|
|
## Intended uses & limitations |
|
|
|
More information needed |
|
|
|
## Training and evaluation data |
|
|
|
More information needed |
|
|
|
## Training procedure |
|
|
|
### Training hyperparameters |
|
|
|
The following hyperparameters were used during training: |
|
- learning_rate: 2e-05 |
|
- train_batch_size: 8 |
|
- eval_batch_size: 8 |
|
- seed: 42 |
|
- optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08 |
|
- lr_scheduler_type: linear |
|
- num_epochs: 10 |
|
|
|
### Training results |
|
|
|
| Training Loss | Epoch | Step | Validation Loss | Claim | Majorclaim | O | Premise | Accuracy | Macro avg | Weighted avg | |
|
|:-------------:|:-----:|:----:|:---------------:|:-------------------------------------------------------------------------------------------------------------------:|:------------------------------------------------------------------------------------------------------------------:|:-------------------------------------------------------------------------------------------------------------------:|:-------------------------------------------------------------------------------------------------------------------:|:--------:|:-------------------------------------------------------------------------------------------------------------------:|:-------------------------------------------------------------------------------------------------------------------:| |
|
| No log | 1.0 | 41 | 0.3618 | {'precision': 0.459572685379137, 'recall': 0.2631957773512476, 'f1-score': 0.33470633104500386, 'support': 4168.0} | {'precision': 0.7210526315789474, 'recall': 0.5729553903345725, 'f1-score': 0.6385292594510618, 'support': 2152.0} | {'precision': 0.9996423462088698, 'recall': 0.9883309759547383, 'f1-score': 0.9939544807965861, 'support': 11312.0} | {'precision': 0.8006100942872989, 'recall': 0.956514536569204, 'f1-score': 0.8716458466996264, 'support': 12073.0} | 0.8436 | {'precision': 0.7452194393635632, 'recall': 0.6952491700524406, 'f1-score': 0.7097089794980695, 'support': 29705.0} | {'precision': 0.8227882209885014, 'recall': 0.8435616899511866, 'f1-score': 0.8259944234340963, 'support': 29705.0} | |
|
| No log | 2.0 | 82 | 0.2635 | {'precision': 0.6204819277108434, 'recall': 0.42010556621880996, 'f1-score': 0.5010014306151646, 'support': 4168.0} | {'precision': 0.7734307824591573, 'recall': 0.8359665427509294, 'f1-score': 0.8034836980794997, 'support': 2152.0} | {'precision': 0.9996458923512748, 'recall': 0.9982319660537482, 'f1-score': 0.9989384288747346, 'support': 11312.0} | {'precision': 0.8523489932885906, 'recall': 0.9362213203014992, 'f1-score': 0.8923186231941264, 'support': 12073.0} | 0.8802 | {'precision': 0.8114768989524666, 'recall': 0.7976313488312466, 'f1-score': 0.7989355451908814, 'support': 29705.0} | {'precision': 0.8701900504562087, 'recall': 0.8801548560848342, 'f1-score': 0.8715780214214702, 'support': 29705.0} | |
|
| No log | 3.0 | 123 | 0.2378 | {'precision': 0.6241271370093908, 'recall': 0.6218809980806143, 'f1-score': 0.623002043023675, 'support': 4168.0} | {'precision': 0.8295557570262919, 'recall': 0.8503717472118959, 'f1-score': 0.8398347865993575, 'support': 2152.0} | {'precision': 0.9996462681287585, 'recall': 0.9992927864214993, 'f1-score': 0.9994694960212203, 'support': 11312.0} | {'precision': 0.8973251370659578, 'recall': 0.8947237637703968, 'f1-score': 0.8960225623159553, 'support': 12073.0} | 0.8930 | {'precision': 0.8376635748075998, 'recall': 0.8415673238711016, 'f1-score': 0.8395822219900521, 'support': 29705.0} | {'precision': 0.8930473274211594, 'recall': 0.893048308365595, 'f1-score': 0.8930374115440858, 'support': 29705.0} | |
|
| No log | 4.0 | 164 | 0.2418 | {'precision': 0.6509234828496042, 'recall': 0.591890595009597, 'f1-score': 0.6200050263885398, 'support': 4168.0} | {'precision': 0.8957732949087416, 'recall': 0.866635687732342, 'f1-score': 0.8809636277751535, 'support': 2152.0} | {'precision': 1.0, 'recall': 0.9993811881188119, 'f1-score': 0.9996904982977407, 'support': 11312.0} | {'precision': 0.8806673052362708, 'recall': 0.913857367679947, 'f1-score': 0.896955408316735, 'support': 12073.0} | 0.8978 | {'precision': 0.8568410207486541, 'recall': 0.8429412096351745, 'f1-score': 0.8494036401945422, 'support': 29705.0} | {'precision': 0.8949688464325286, 'recall': 0.8978286483756943, 'f1-score': 0.8960597959370997, 'support': 29705.0} | |
|
| No log | 5.0 | 205 | 0.2518 | {'precision': 0.633635729239358, 'recall': 0.6535508637236085, 'f1-score': 0.6434392346758001, 'support': 4168.0} | {'precision': 0.8811835413777162, 'recall': 0.8856877323420075, 'f1-score': 0.8834298957126304, 'support': 2152.0} | {'precision': 1.0, 'recall': 0.9994695898161244, 'f1-score': 0.9997347245556637, 'support': 11312.0} | {'precision': 0.8996397754879786, 'recall': 0.8895055081587012, 'f1-score': 0.8945439400249896, 'support': 12073.0} | 0.8980 | {'precision': 0.8536147615262633, 'recall': 0.8570534235101104, 'f1-score': 0.855286948742271, 'support': 29705.0} | {'precision': 0.8991971624299228, 'recall': 0.8979969702070358, 'f1-score': 0.8985633414508561, 'support': 29705.0} | |
|
| No log | 6.0 | 246 | 0.2668 | {'precision': 0.6150532759011562, 'recall': 0.650911708253359, 'f1-score': 0.6324746473948013, 'support': 4168.0} | {'precision': 0.9063266307013241, 'recall': 0.8587360594795539, 'f1-score': 0.8818897637795274, 'support': 2152.0} | {'precision': 1.0, 'recall': 0.9992043847241867, 'f1-score': 0.9996020340481981, 'support': 11312.0} | {'precision': 0.8931559571619813, 'recall': 0.8842044230928519, 'f1-score': 0.8886576482830385, 'support': 12073.0} | 0.8934 | {'precision': 0.8536339659411154, 'recall': 0.8482641438874878, 'f1-score': 0.8506560233763915, 'support': 29705.0} | {'precision': 0.8957760927130742, 'recall': 0.8934186163945463, 'f1-score': 0.8944703955182431, 'support': 29705.0} | |
|
| No log | 7.0 | 287 | 0.3051 | {'precision': 0.6341149188353518, 'recall': 0.5904510556621881, 'f1-score': 0.6115045347248106, 'support': 4168.0} | {'precision': 0.9206932773109243, 'recall': 0.8145910780669146, 'f1-score': 0.8643984220907298, 'support': 2152.0} | {'precision': 0.9998231497037757, 'recall': 0.999557991513437, 'f1-score': 0.9996905530259493, 'support': 11312.0} | {'precision': 0.875901990325906, 'recall': 0.9149341505839477, 'f1-score': 0.8949927078269324, 'support': 12073.0} | 0.8944 | {'precision': 0.8576333340439894, 'recall': 0.8298835689566219, 'f1-score': 0.8426465544171056, 'support': 29705.0} | {'precision': 0.8924116180149008, 'recall': 0.8943612186500589, 'f1-score': 0.8928693082813114, 'support': 29705.0} | |
|
| No log | 8.0 | 328 | 0.3102 | {'precision': 0.6281418839558375, 'recall': 0.6415547024952015, 'f1-score': 0.6347774480712166, 'support': 4168.0} | {'precision': 0.9114481409001957, 'recall': 0.8657063197026023, 'f1-score': 0.8879885605338417, 'support': 2152.0} | {'precision': 0.9998231340643792, 'recall': 0.9994695898161244, 'f1-score': 0.9996463306808134, 'support': 11312.0} | {'precision': 0.890625, 'recall': 0.8923217095999337, 'f1-score': 0.8914725474781745, 'support': 12073.0} | 0.8960 | {'precision': 0.8575095397301031, 'recall': 0.8497630804034655, 'f1-score': 0.8534712216910115, 'support': 29705.0} | {'precision': 0.8968876178785191, 'recall': 0.8960107725972059, 'f1-score': 0.8963962681095783, 'support': 29705.0} | |
|
| No log | 9.0 | 369 | 0.3262 | {'precision': 0.6178665496049166, 'recall': 0.675383877159309, 'f1-score': 0.645346171480972, 'support': 4168.0} | {'precision': 0.9207772795216741, 'recall': 0.8587360594795539, 'f1-score': 0.8886751622986293, 'support': 2152.0} | {'precision': 0.9998231340643792, 'recall': 0.9994695898161244, 'f1-score': 0.9996463306808134, 'support': 11312.0} | {'precision': 0.898259252999831, 'recall': 0.8804770976559264, 'f1-score': 0.8892792905843476, 'support': 12073.0} | 0.8954 | {'precision': 0.8591815540477001, 'recall': 0.8535166560277284, 'f1-score': 0.8557367387611907, 'support': 29705.0} | {'precision': 0.8992244988482462, 'recall': 0.8954384783706447, 'f1-score': 0.897037534417961, 'support': 29705.0} | |
|
| No log | 10.0 | 410 | 0.3310 | {'precision': 0.6261209473442171, 'recall': 0.6533109404990403, 'f1-score': 0.6394270282963485, 'support': 4168.0} | {'precision': 0.9034416826003824, 'recall': 0.8782527881040892, 'f1-score': 0.8906691800188501, 'support': 2152.0} | {'precision': 0.9998231340643792, 'recall': 0.9994695898161244, 'f1-score': 0.9996463306808134, 'support': 11312.0} | {'precision': 0.8951990632318502, 'recall': 0.8865236478091609, 'f1-score': 0.8908402347163843, 'support': 12073.0} | 0.8962 | {'precision': 0.8561462068102073, 'recall': 0.8543892415571037, 'f1-score': 0.8551456934280991, 'support': 29705.0} | {'precision': 0.8978830564693184, 'recall': 0.8962127587948157, 'f1-score': 0.8969858736149474, 'support': 29705.0} | |
|
|
|
|
|
### Framework versions |
|
|
|
- Transformers 4.37.2 |
|
- Pytorch 2.2.0+cu121 |
|
- Datasets 2.17.0 |
|
- Tokenizers 0.15.2 |
|
|
