metadata

license: apache-2.0
base_model: allenai/longformer-base-4096
tags:
  - generated_from_trainer
datasets:
  - essays_su_g
metrics:
  - accuracy
model-index:
  - name: longformer-sep_tok
    results:
      - task:
          name: Token Classification
          type: token-classification
        dataset:
          name: essays_su_g
          type: essays_su_g
          config: sep_tok
          split: train[80%:100%]
          args: sep_tok
        metrics:
          - name: Accuracy
            type: accuracy
            value: 0.8973236828816697

longformer-sep_tok

This model is a fine-tuned version of allenai/longformer-base-4096 on the essays_su_g dataset. It achieves the following results on the evaluation set:

Loss: 0.2957
Claim: {'precision': 0.6376988984088128, 'recall': 0.625, 'f1-score': 0.6312855931176542, 'support': 4168.0}
Majorclaim: {'precision': 0.9108095007270964, 'recall': 0.8731412639405205, 'f1-score': 0.8915776986951364, 'support': 2152.0}
O: {'precision': 0.9999115904871364, 'recall': 0.9998231966053748, 'f1-score': 0.9998673915926268, 'support': 11312.0}
Premise: {'precision': 0.8869018455005716, 'recall': 0.8996107015654767, 'f1-score': 0.8932110695341092, 'support': 12073.0}
Accuracy: 0.8973
Macro avg: {'precision': 0.8588304587809044, 'recall': 0.849393790527843, 'f1-score': 0.8539854382348816, 'support': 29705.0}
Weighted avg: {'precision': 0.896702809171908, 'recall': 0.8973236828816697, 'f1-score': 0.8969571027061913, 'support': 29705.0}

Model description

More information needed

Intended uses & limitations

More information needed

Training and evaluation data

More information needed

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

learning_rate: 2e-05
train_batch_size: 8
eval_batch_size: 8
seed: 42
optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
lr_scheduler_type: linear
num_epochs: 8

Training results

Training Loss	Epoch	Step	Validation Loss	Claim	Majorclaim	O	Premise	Accuracy	Macro avg	Weighted avg
No log	1.0	41	0.3532	{'precision': 0.46718648473034435, 'recall': 0.3450095969289827, 'f1-score': 0.39690863924924097, 'support': 4168.0}	{'precision': 0.72375, 'recall': 0.5381040892193308, 'f1-score': 0.6172707889125799, 'support': 2152.0}	{'precision': 0.999551489056333, 'recall': 0.9850601131541725, 'f1-score': 0.9922528940338379, 'support': 11312.0}	{'precision': 0.8174220044671806, 'recall': 0.9397001573759629, 'f1-score': 0.8743064118372379, 'support': 12073.0}	0.8444	{'precision': 0.7519774945634644, 'recall': 0.7019684891696123, 'f1-score': 0.7201846835082242, 'support': 29705.0}	{'precision': 0.8308502128427397, 'recall': 0.8444369634741626, 'f1-score': 0.8336154853914309, 'support': 29705.0}
No log	2.0	82	0.2649	{'precision': 0.6305418719211823, 'recall': 0.39923224568138194, 'f1-score': 0.48890847656823855, 'support': 4168.0}	{'precision': 0.7800252844500632, 'recall': 0.8601301115241635, 'f1-score': 0.818121546961326, 'support': 2152.0}	{'precision': 0.9998229775181448, 'recall': 0.9985855728429985, 'f1-score': 0.999203892083149, 'support': 11312.0}	{'precision': 0.8480029861888765, 'recall': 0.9408597697341174, 'f1-score': 0.8920213601382126, 'support': 12073.0}	0.8810	{'precision': 0.8145982800195667, 'recall': 0.7997019249456654, 'f1-score': 0.7995638189377315, 'support': 29705.0}	{'precision': 0.8703804244486646, 'recall': 0.8809964652415418, 'f1-score': 0.8709219460558968, 'support': 29705.0}
No log	3.0	123	0.2493	{'precision': 0.6082398829839103, 'recall': 0.5986084452975048, 'f1-score': 0.6033857315598549, 'support': 4168.0}	{'precision': 0.7881733495342244, 'recall': 0.904275092936803, 'f1-score': 0.8422419389742479, 'support': 2152.0}	{'precision': 0.9998230088495575, 'recall': 0.9987623762376238, 'f1-score': 0.9992924111091456, 'support': 11312.0}	{'precision': 0.8988507689707622, 'recall': 0.8810569038350037, 'f1-score': 0.8898648931275358, 'support': 12073.0}	0.8879	{'precision': 0.8237717525846135, 'recall': 0.8456757045767338, 'f1-score': 0.833696243692696, 'support': 29705.0}	{'precision': 0.8885075270279347, 'recall': 0.8879313246928127, 'f1-score': 0.8878892775966825, 'support': 29705.0}
No log	4.0	164	0.2464	{'precision': 0.6338532344668882, 'recall': 0.5947696737044146, 'f1-score': 0.6136898130956802, 'support': 4168.0}	{'precision': 0.8428571428571429, 'recall': 0.9047397769516728, 'f1-score': 0.8727028238458091, 'support': 2152.0}	{'precision': 0.9999115904871364, 'recall': 0.9998231966053748, 'f1-score': 0.9998673915926268, 'support': 11312.0}	{'precision': 0.8904132095621458, 'recall': 0.897788453574091, 'f1-score': 0.8940856223707004, 'support': 12073.0}	0.8946	{'precision': 0.8417587943433282, 'recall': 0.8492802752088883, 'f1-score': 0.8450864127262041, 'support': 29705.0}	{'precision': 0.8926674783141166, 'recall': 0.8946305335802054, 'f1-score': 0.8934762252306426, 'support': 29705.0}
No log	5.0	205	0.2565	{'precision': 0.6154180238870792, 'recall': 0.6799424184261037, 'f1-score': 0.6460731790721531, 'support': 4168.0}	{'precision': 0.9039196940726577, 'recall': 0.8787174721189591, 'f1-score': 0.8911404335532517, 'support': 2152.0}	{'precision': 1.0, 'recall': 0.9993811881188119, 'f1-score': 0.9996904982977407, 'support': 11312.0}	{'precision': 0.9022472870204221, 'recall': 0.8745962064109998, 'f1-score': 0.8882065948855989, 'support': 12073.0}	0.8951	{'precision': 0.8553962512450398, 'recall': 0.8581593212687186, 'f1-score': 0.8562776764521861, 'support': 29705.0}	{'precision': 0.8993478876082566, 'recall': 0.8951018347079617, 'f1-score': 0.8968989853619663, 'support': 29705.0}
No log	6.0	246	0.2794	{'precision': 0.6114372689715155, 'recall': 0.6746641074856046, 'f1-score': 0.6414965210448272, 'support': 4168.0}	{'precision': 0.9110212335692619, 'recall': 0.837360594795539, 'f1-score': 0.872639225181598, 'support': 2152.0}	{'precision': 0.9999115904871364, 'recall': 0.9998231966053748, 'f1-score': 0.9998673915926268, 'support': 11312.0}	{'precision': 0.8996361174578996, 'recall': 0.8805599271100804, 'f1-score': 0.8899958141481791, 'support': 12073.0}	0.8940	{'precision': 0.8555015526214533, 'recall': 0.8481019564991497, 'f1-score': 0.8509997379918077, 'support': 29705.0}	{'precision': 0.8982088870349785, 'recall': 0.8939572462548393, 'f1-score': 0.8957110422559299, 'support': 29705.0}
No log	7.0	287	0.2945	{'precision': 0.6273562107298212, 'recall': 0.6228406909788867, 'f1-score': 0.6250902961714422, 'support': 4168.0}	{'precision': 0.9130213631739573, 'recall': 0.8341078066914498, 'f1-score': 0.87178241864983, 'support': 2152.0}	{'precision': 1.0, 'recall': 0.9997347949080623, 'f1-score': 0.9998673798682641, 'support': 11312.0}	{'precision': 0.8854539537910836, 'recall': 0.9015157790110163, 'f1-score': 0.8934126821260003, 'support': 12073.0}	0.8949	{'precision': 0.8564578819237155, 'recall': 0.8395497678973538, 'f1-score': 0.8475381942038841, 'support': 29705.0}	{'precision': 0.8948570356502946, 'recall': 0.8949335128766202, 'f1-score': 0.8947356751979132, 'support': 29705.0}
No log	8.0	328	0.2957	{'precision': 0.6376988984088128, 'recall': 0.625, 'f1-score': 0.6312855931176542, 'support': 4168.0}	{'precision': 0.9108095007270964, 'recall': 0.8731412639405205, 'f1-score': 0.8915776986951364, 'support': 2152.0}	{'precision': 0.9999115904871364, 'recall': 0.9998231966053748, 'f1-score': 0.9998673915926268, 'support': 11312.0}	{'precision': 0.8869018455005716, 'recall': 0.8996107015654767, 'f1-score': 0.8932110695341092, 'support': 12073.0}	0.8973	{'precision': 0.8588304587809044, 'recall': 0.849393790527843, 'f1-score': 0.8539854382348816, 'support': 29705.0}	{'precision': 0.896702809171908, 'recall': 0.8973236828816697, 'f1-score': 0.8969571027061913, 'support': 29705.0}

Framework versions

Transformers 4.37.2
Pytorch 2.2.0+cu121
Datasets 2.17.0
Tokenizers 0.15.2