metadata
license: apache-2.0
base_model: allenai/longformer-base-4096
tags:
- generated_from_trainer
datasets:
- essays_su_g
metrics:
- accuracy
model-index:
- name: longformer-sep_tok
results:
- task:
name: Token Classification
type: token-classification
dataset:
name: essays_su_g
type: essays_su_g
config: sep_tok
split: train[80%:100%]
args: sep_tok
metrics:
- name: Accuracy
type: accuracy
value: 0.8962800875273523
longformer-sep_tok
This model is a fine-tuned version of allenai/longformer-base-4096 on the essays_su_g dataset. It achieves the following results on the evaluation set:
- Loss: 0.4178
- Claim: {'precision': 0.6449313621964097, 'recall': 0.5861324376199616, 'f1-score': 0.6141277023629965, 'support': 4168.0}
- Majorclaim: {'precision': 0.920619554695063, 'recall': 0.8838289962825279, 'f1-score': 0.9018492176386913, 'support': 2152.0}
- O: {'precision': 1.0, 'recall': 0.9996463932107497, 'f1-score': 0.9998231653404068, 'support': 11312.0}
- Premise: {'precision': 0.8746711313082994, 'recall': 0.9087219415224054, 'f1-score': 0.8913714657133571, 'support': 12073.0}
- Accuracy: 0.8963
- Macro avg: {'precision': 0.860055512049943, 'recall': 0.8445824421589111, 'f1-score': 0.851792887763863, 'support': 29705.0}
- Weighted avg: {'precision': 0.8934910542879485, 'recall': 0.8962800875273523, 'f1-score': 0.8945292419355487, 'support': 29705.0}
Model description
More information needed
Intended uses & limitations
More information needed
Training and evaluation data
More information needed
Training procedure
Training hyperparameters
The following hyperparameters were used during training:
- learning_rate: 2e-05
- train_batch_size: 8
- eval_batch_size: 8
- seed: 42
- optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
- lr_scheduler_type: linear
- num_epochs: 13
Training results
| Training Loss | Epoch | Step | Validation Loss | Claim | Majorclaim | O | Premise | Accuracy | Macro avg | Weighted avg |
|---|---|---|---|---|---|---|---|---|---|---|
| No log | 1.0 | 41 | 0.3959 | {'precision': 0.40099457504520797, 'recall': 0.21281190019193857, 'f1-score': 0.2780564263322884, 'support': 4168.0} | {'precision': 0.7553648068669528, 'recall': 0.40892193308550184, 'f1-score': 0.5305999397045523, 'support': 2152.0} | {'precision': 0.9995538900785154, 'recall': 0.9903642149929278, 'f1-score': 0.9949378330373002, 'support': 11312.0} | {'precision': 0.7791005291005291, 'recall': 0.9757309699329081, 'f1-score': 0.8663994410326187, 'support': 12073.0} | 0.8332 | {'precision': 0.7337534502728014, 'recall': 0.6469572545508191, 'f1-score': 0.6674984100266899, 'support': 29705.0} | {'precision': 0.8082789007091385, 'recall': 0.8331930651405487, 'f1-score': 0.8084688595893593, 'support': 29705.0} |
| No log | 2.0 | 82 | 0.2797 | {'precision': 0.656575682382134, 'recall': 0.31741842610364684, 'f1-score': 0.42794759825327516, 'support': 4168.0} | {'precision': 0.7055785123966942, 'recall': 0.9521375464684015, 'f1-score': 0.8105221518987342, 'support': 2152.0} | {'precision': 0.9998224274172068, 'recall': 0.995491513437058, 'f1-score': 0.9976522702104098, 'support': 11312.0} | {'precision': 0.8425645197071656, 'recall': 0.9437588006295039, 'f1-score': 0.890295358649789, 'support': 12073.0} | 0.8762 | {'precision': 0.8011352854758002, 'recall': 0.8022015716596526, 'f1-score': 0.7816043447530521, 'support': 29705.0} | {'precision': 0.8664293939812986, 'recall': 0.8761824608651743, 'f1-score': 0.8605254201651165, 'support': 29705.0} |
| No log | 3.0 | 123 | 0.2480 | {'precision': 0.6007310255858955, 'recall': 0.6703454894433781, 'f1-score': 0.633631931057943, 'support': 4168.0} | {'precision': 0.8402394106813996, 'recall': 0.8480483271375465, 'f1-score': 0.8441258094357077, 'support': 2152.0} | {'precision': 0.9988508795191373, 'recall': 0.998939179632249, 'f1-score': 0.9988950276243095, 'support': 11312.0} | {'precision': 0.9099317140634454, 'recall': 0.871945663878075, 'f1-score': 0.8905337957871585, 'support': 12073.0} | 0.8903 | {'precision': 0.8374382574624696, 'recall': 0.8473196650228122, 'f1-score': 0.8417966409762797, 'support': 29705.0} | {'precision': 0.8953593287135783, 'recall': 0.890287830331594, 'f1-score': 0.8923902272203232, 'support': 29705.0} |
| No log | 4.0 | 164 | 0.2510 | {'precision': 0.6090909090909091, 'recall': 0.6590690978886756, 'f1-score': 0.6330951832219405, 'support': 4168.0} | {'precision': 0.8471655328798186, 'recall': 0.8680297397769516, 'f1-score': 0.8574707367454671, 'support': 2152.0} | {'precision': 0.9999116061168567, 'recall': 1.0, 'f1-score': 0.9999558011049724, 'support': 11312.0} | {'precision': 0.9060546373212298, 'recall': 0.8763356249482316, 'f1-score': 0.8909473684210527, 'support': 12073.0} | 0.8923 | {'precision': 0.8405556713522035, 'recall': 0.8508586156534647, 'f1-score': 0.8453672723733583, 'support': 29705.0} | {'precision': 0.8958622743855031, 'recall': 0.8923413566739606, 'f1-score': 0.8938537401175597, 'support': 29705.0} |
| No log | 5.0 | 205 | 0.2514 | {'precision': 0.6300216502285302, 'recall': 0.628358925143954, 'f1-score': 0.6291891891891892, 'support': 4168.0} | {'precision': 0.8886300093196645, 'recall': 0.8861524163568774, 'f1-score': 0.8873894834806887, 'support': 2152.0} | {'precision': 0.9999115592111082, 'recall': 0.9994695898161244, 'f1-score': 0.9996905256642645, 'support': 11312.0} | {'precision': 0.8906986357999174, 'recall': 0.8923217095999337, 'f1-score': 0.8915094339622642, 'support': 12073.0} | 0.8956 | {'precision': 0.8523154636398051, 'recall': 0.8515756602292224, 'f1-score': 0.8519446580741017, 'support': 29705.0} | {'precision': 0.8955618988728123, 'recall': 0.8956404645682545, 'f1-score': 0.8956005834550264, 'support': 29705.0} |
| No log | 6.0 | 246 | 0.2718 | {'precision': 0.6259613143789327, 'recall': 0.6444337811900192, 'f1-score': 0.6350632462466013, 'support': 4168.0} | {'precision': 0.8953323903818954, 'recall': 0.8824349442379182, 'f1-score': 0.8888368827521648, 'support': 2152.0} | {'precision': 1.0, 'recall': 0.9994695898161244, 'f1-score': 0.9997347245556637, 'support': 11312.0} | {'precision': 0.8940518895470093, 'recall': 0.8876832601673155, 'f1-score': 0.8908561928512054, 'support': 12073.0} | 0.8957 | {'precision': 0.8538363985769593, 'recall': 0.8535053938528443, 'f1-score': 0.8536227616014088, 'support': 29705.0} | {'precision': 0.8968742812635675, 'recall': 0.8957414576670594, 'f1-score': 0.8962809830838163, 'support': 29705.0} |
| No log | 7.0 | 287 | 0.2978 | {'precision': 0.6061165845648604, 'recall': 0.7084932821497121, 'f1-score': 0.6533185840707965, 'support': 4168.0} | {'precision': 0.9254881808838643, 'recall': 0.8368959107806692, 'f1-score': 0.8789653489507077, 'support': 2152.0} | {'precision': 1.0, 'recall': 0.9994695898161244, 'f1-score': 0.9997347245556637, 'support': 11312.0} | {'precision': 0.9068301528365426, 'recall': 0.8698749275242276, 'f1-score': 0.8879682083368562, 'support': 12073.0} | 0.8942 | {'precision': 0.8596087295713168, 'recall': 0.8536834275676832, 'f1-score': 0.854996716478506, 'support': 29705.0} | {'precision': 0.9014679321637432, 'recall': 0.8941928968187174, 'f1-score': 0.8969535321586781, 'support': 29705.0} |
| No log | 8.0 | 328 | 0.3142 | {'precision': 0.6124081279723304, 'recall': 0.6797024952015355, 'f1-score': 0.6443029338185126, 'support': 4168.0} | {'precision': 0.8891476478807638, 'recall': 0.8870817843866171, 'f1-score': 0.888113514770877, 'support': 2152.0} | {'precision': 1.0, 'recall': 0.9988507779349364, 'f1-score': 0.9994250585997965, 'support': 11312.0} | {'precision': 0.9043239061291154, 'recall': 0.8713658576989978, 'f1-score': 0.8875390196574707, 'support': 12073.0} | 0.8942 | {'precision': 0.8514699204955525, 'recall': 0.8592502288055217, 'f1-score': 0.8548451317116642, 'support': 29705.0} | {'precision': 0.8986993884640595, 'recall': 0.8941592324524491, 'f1-score': 0.8960589045328405, 'support': 29705.0} |
| No log | 9.0 | 369 | 0.3483 | {'precision': 0.6076411247048723, 'recall': 0.6792226487523992, 'f1-score': 0.6414410331936106, 'support': 4168.0} | {'precision': 0.9248466257668712, 'recall': 0.8406133828996283, 'f1-score': 0.8807205452775072, 'support': 2152.0} | {'precision': 0.9999115748518879, 'recall': 0.9996463932107497, 'f1-score': 0.9997789664471067, 'support': 11312.0} | {'precision': 0.89831083948731, 'recall': 0.8765841133106933, 'f1-score': 0.8873144965204998, 'support': 12073.0} | 0.8931 | {'precision': 0.8576775412027353, 'recall': 0.8490166345433676, 'f1-score': 0.8523137603596811, 'support': 29705.0} | {'precision': 0.8981391902465936, 'recall': 0.8931493014644, 'f1-score': 0.8951652726722716, 'support': 29705.0} |
| No log | 10.0 | 410 | 0.3626 | {'precision': 0.6293302540415704, 'recall': 0.6537907869481766, 'f1-score': 0.641327371146152, 'support': 4168.0} | {'precision': 0.8859416445623343, 'recall': 0.9312267657992565, 'f1-score': 0.9080199365654735, 'support': 2152.0} | {'precision': 0.9999116061168567, 'recall': 1.0, 'f1-score': 0.9999558011049724, 'support': 11312.0} | {'precision': 0.8986440677966102, 'recall': 0.8783235318479251, 'f1-score': 0.8883676119465505, 'support': 12073.0} | 0.8970 | {'precision': 0.8534568931293429, 'recall': 0.8658352711488396, 'f1-score': 0.8594176801907871, 'support': 29705.0} | {'precision': 0.8984994053811064, 'recall': 0.8969870392189867, 'f1-score': 0.897622406583276, 'support': 29705.0} |
| No log | 11.0 | 451 | 0.3772 | {'precision': 0.6213443910955915, 'recall': 0.6830614203454894, 'f1-score': 0.6507428571428571, 'support': 4168.0} | {'precision': 0.8955895589558955, 'recall': 0.9247211895910781, 'f1-score': 0.9099222679469593, 'support': 2152.0} | {'precision': 1.0, 'recall': 0.9996463932107497, 'f1-score': 0.9998231653404068, 'support': 11312.0} | {'precision': 0.9045976020012076, 'recall': 0.8686324857119192, 'f1-score': 0.8862503169103356, 'support': 12073.0} | 0.8965 | {'precision': 0.8553828880131737, 'recall': 0.8690153722148091, 'f1-score': 0.8616846518351398, 'support': 29705.0} | {'precision': 0.9005311900999863, 'recall': 0.8965494024574987, 'f1-score': 0.8981702969729826, 'support': 29705.0} |
| No log | 12.0 | 492 | 0.4167 | {'precision': 0.6351280710925248, 'recall': 0.5830134357005758, 'f1-score': 0.6079559669752315, 'support': 4168.0} | {'precision': 0.9238427078148332, 'recall': 0.862453531598513, 'f1-score': 0.8920932468156694, 'support': 2152.0} | {'precision': 1.0, 'recall': 0.9994695898161244, 'f1-score': 0.9997347245556637, 'support': 11312.0} | {'precision': 0.8730499840815027, 'recall': 0.9085562826140976, 'f1-score': 0.8904493241871981, 'support': 12073.0} | 0.8942 | {'precision': 0.8580051907472153, 'recall': 0.8383732099323278, 'f1-score': 0.8475583156334406, 'support': 29705.0} | {'precision': 0.8916901452734269, 'recall': 0.8941592324524491, 'f1-score': 0.8925480233154618, 'support': 29705.0} |
| 0.175 | 13.0 | 533 | 0.4178 | {'precision': 0.6449313621964097, 'recall': 0.5861324376199616, 'f1-score': 0.6141277023629965, 'support': 4168.0} | {'precision': 0.920619554695063, 'recall': 0.8838289962825279, 'f1-score': 0.9018492176386913, 'support': 2152.0} | {'precision': 1.0, 'recall': 0.9996463932107497, 'f1-score': 0.9998231653404068, 'support': 11312.0} | {'precision': 0.8746711313082994, 'recall': 0.9087219415224054, 'f1-score': 0.8913714657133571, 'support': 12073.0} | 0.8963 | {'precision': 0.860055512049943, 'recall': 0.8445824421589111, 'f1-score': 0.851792887763863, 'support': 29705.0} | {'precision': 0.8934910542879485, 'recall': 0.8962800875273523, 'f1-score': 0.8945292419355487, 'support': 29705.0} |
Framework versions
- Transformers 4.37.2
- Pytorch 2.2.0+cu121
- Datasets 2.17.0
- Tokenizers 0.15.2