metadata

license: apache-2.0
base_model: allenai/longformer-base-4096
tags:
  - generated_from_trainer
datasets:
  - essays_su_g
metrics:
  - accuracy
model-index:
  - name: longformer-sep_tok
    results:
      - task:
          name: Token Classification
          type: token-classification
        dataset:
          name: essays_su_g
          type: essays_su_g
          config: sep_tok
          split: train[0%:20%]
          args: sep_tok
        metrics:
          - name: Accuracy
            type: accuracy
            value: 0.9033544303797468

longformer-sep_tok

This model is a fine-tuned version of allenai/longformer-base-4096 on the essays_su_g dataset. It achieves the following results on the evaluation set:

Loss: 0.4725
Claim: {'precision': 0.6448837209302326, 'recall': 0.6506335053965274, 'f1-score': 0.6477458537724831, 'support': 4262.0}
Majorclaim: {'precision': 0.9238191975622143, 'recall': 0.8401847575057737, 'f1-score': 0.8800193517174649, 'support': 2165.0}
O: {'precision': 0.9978613144690301, 'recall': 0.9997527608373167, 'f1-score': 0.9988061421925816, 'support': 12134.0}
Premise: {'precision': 0.8974495217853348, 'recall': 0.9067413145179845, 'f1-score': 0.9020714912448022, 'support': 13039.0}
Accuracy: 0.9034
Macro avg: {'precision': 0.866003438686703, 'recall': 0.8493280845644006, 'f1-score': 0.857160709731833, 'support': 31600.0}
Weighted avg: {'precision': 0.9037486229637037, 'recall': 0.9033544303797468, 'f1-score': 0.9034037540807719, 'support': 31600.0}

Model description

More information needed

Intended uses & limitations

More information needed

Training and evaluation data

More information needed

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

learning_rate: 2e-05
train_batch_size: 8
eval_batch_size: 8
seed: 42
optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
lr_scheduler_type: linear
num_epochs: 16

Training results

Training Loss	Epoch	Step	Validation Loss	Claim	Majorclaim	O	Premise	Accuracy	Macro avg	Weighted avg
No log	1.0	41	0.3643	{'precision': 0.4827682824320538, 'recall': 0.4042702956358517, 'f1-score': 0.44004597114033966, 'support': 4262.0}	{'precision': 0.7040155440414507, 'recall': 0.5020785219399538, 'f1-score': 0.586141817201402, 'support': 2165.0}	{'precision': 0.9970286102385602, 'recall': 0.967858908851162, 'f1-score': 0.9822272404131643, 'support': 12134.0}	{'precision': 0.8283247212401414, 'recall': 0.9343507937725286, 'f1-score': 0.8781489890799006, 'support': 13039.0}	0.8461	{'precision': 0.7530342894880515, 'recall': 0.7021396300498741, 'f1-score': 0.7216410044587016, 'support': 31600.0}	{'precision': 0.8379817490335457, 'recall': 0.8461075949367088, 'f1-score': 0.8390190812350419, 'support': 31600.0}
No log	2.0	82	0.2769	{'precision': 0.6401985111662531, 'recall': 0.4237447207883623, 'f1-score': 0.5099534095722152, 'support': 4262.0}	{'precision': 0.6977007161703732, 'recall': 0.8549653579676675, 'f1-score': 0.7683686176836863, 'support': 2165.0}	{'precision': 0.9958073002301875, 'recall': 0.9982693258612164, 'f1-score': 0.9970367931516998, 'support': 12134.0}	{'precision': 0.8682137229623264, 'recall': 0.9296725208988419, 'f1-score': 0.8978926706418281, 'support': 13039.0}	0.8827	{'precision': 0.800480062632285, 'recall': 0.801662981379022, 'f1-score': 0.7933128727623573, 'support': 31600.0}	{'precision': 0.8747725512594399, 'recall': 0.8826582278481012, 'f1-score': 0.8747660275153002, 'support': 31600.0}
No log	3.0	123	0.2435	{'precision': 0.6495901639344263, 'recall': 0.5950258094791178, 'f1-score': 0.621111927504286, 'support': 4262.0}	{'precision': 0.798990748528175, 'recall': 0.8775981524249422, 'f1-score': 0.836451683909311, 'support': 2165.0}	{'precision': 0.9976969896364534, 'recall': 0.9996703477830888, 'f1-score': 0.9986826938909928, 'support': 12134.0}	{'precision': 0.9022796352583586, 'recall': 0.9106526574123782, 'f1-score': 0.9064468109469827, 'support': 13039.0}	0.9	{'precision': 0.8371393843393533, 'recall': 0.8457367417748817, 'f1-score': 0.8406732790628931, 'support': 31600.0}	{'precision': 0.8977610027099522, 'recall': 0.9, 'f1-score': 0.8985845793132259, 'support': 31600.0}
No log	4.0	164	0.2710	{'precision': 0.5950994175537256, 'recall': 0.6952135147817926, 'f1-score': 0.6412725895465858, 'support': 4262.0}	{'precision': 0.781058282208589, 'recall': 0.9408775981524249, 'f1-score': 0.8535512256442489, 'support': 2165.0}	{'precision': 0.9986824769433466, 'recall': 0.9995055216746332, 'f1-score': 0.9990938298047615, 'support': 12134.0}	{'precision': 0.9320077512848597, 'recall': 0.8483779430937956, 'f1-score': 0.8882286815480969, 'support': 13039.0}	0.8921	{'precision': 0.8267119819976302, 'recall': 0.8709936444256616, 'f1-score': 0.8455365816359233, 'support': 31600.0}	{'precision': 0.9018280741401717, 'recall': 0.8920886075949367, 'f1-score': 0.8951158382824038, 'support': 31600.0}
No log	5.0	205	0.3124	{'precision': 0.6569706498951782, 'recall': 0.5882214922571563, 'f1-score': 0.6206981926219362, 'support': 4262.0}	{'precision': 0.9552845528455285, 'recall': 0.7598152424942263, 'f1-score': 0.8464111139696424, 'support': 2165.0}	{'precision': 0.9973686374475783, 'recall': 0.9995879347288611, 'f1-score': 0.9984770528915414, 'support': 12134.0}	{'precision': 0.877131141644486, 'recall': 0.9351177237518215, 'f1-score': 0.9051967334818115, 'support': 13039.0}	0.9011	{'precision': 0.8716887454581927, 'recall': 0.8206855983080162, 'f1-score': 0.8426957732412328, 'support': 31600.0}	{'precision': 0.8989615180207339, 'recall': 0.9010759493670886, 'f1-score': 0.8986163457707048, 'support': 31600.0}
No log	6.0	246	0.3000	{'precision': 0.6428049671292915, 'recall': 0.6194274988268419, 'f1-score': 0.6308997490739635, 'support': 4262.0}	{'precision': 0.9450247388675096, 'recall': 0.7939953810623557, 'f1-score': 0.8629518072289157, 'support': 2165.0}	{'precision': 0.996794608366894, 'recall': 0.9995055216746332, 'f1-score': 0.9981482243529073, 'support': 12134.0}	{'precision': 0.8865773302731916, 'recall': 0.9183986502032364, 'f1-score': 0.9022074888872147, 'support': 13039.0}	0.9007	{'precision': 0.8678004111592217, 'recall': 0.8328317629417669, 'f1-score': 0.8485518173857503, 'support': 31600.0}	{'precision': 0.9000253454718111, 'recall': 0.9006962025316456, 'f1-score': 0.8997658036424813, 'support': 31600.0}
No log	7.0	287	0.3219	{'precision': 0.5949652067130577, 'recall': 0.6820741435945565, 'f1-score': 0.6355487538259729, 'support': 4262.0}	{'precision': 0.9873817034700315, 'recall': 0.7228637413394919, 'f1-score': 0.8346666666666667, 'support': 2165.0}	{'precision': 0.9974464579901153, 'recall': 0.9979396736443052, 'f1-score': 0.9976930048611683, 'support': 12134.0}	{'precision': 0.8966048194626223, 'recall': 0.8931666538845003, 'f1-score': 0.8948824343015215, 'support': 13039.0}	0.8933	{'precision': 0.8690995469089566, 'recall': 0.8240110531157135, 'f1-score': 0.8406977149138323, 'support': 31600.0}	{'precision': 0.900862932318002, 'recall': 0.8932594936708861, 'f1-score': 0.8952576298728666, 'support': 31600.0}
No log	8.0	328	0.3370	{'precision': 0.6643180674383493, 'recall': 0.6194274988268419, 'f1-score': 0.6410879067508499, 'support': 4262.0}	{'precision': 0.9216867469879518, 'recall': 0.8480369515011548, 'f1-score': 0.8833293240317538, 'support': 2165.0}	{'precision': 0.9963854431939538, 'recall': 0.9995879347288611, 'f1-score': 0.9979841198008804, 'support': 12134.0}	{'precision': 0.889755590223609, 'recall': 0.918552036199095, 'f1-score': 0.9039245283018866, 'support': 13039.0}	0.9045	{'precision': 0.868036461960966, 'recall': 0.8464011053139882, 'f1-score': 0.8565814697213427, 'support': 31600.0}	{'precision': 0.9024822632687415, 'recall': 0.9044936708860759, 'f1-score': 0.9031815151675017, 'support': 31600.0}
No log	9.0	369	0.3583	{'precision': 0.6696588868940754, 'recall': 0.6126231816048804, 'f1-score': 0.6398725646366867, 'support': 4262.0}	{'precision': 0.923741738688358, 'recall': 0.8392609699769054, 'f1-score': 0.8794772507260408, 'support': 2165.0}	{'precision': 0.9980212713331684, 'recall': 0.9976100214273941, 'f1-score': 0.9978156040060999, 'support': 12134.0}	{'precision': 0.8865123116501287, 'recall': 0.9249942480251553, 'f1-score': 0.9053445428614323, 'support': 13039.0}	0.9049	{'precision': 0.8694835521414326, 'recall': 0.8436221052585838, 'f1-score': 0.8556274905575649, 'support': 31600.0}	{'precision': 0.9026332651318208, 'recall': 0.904873417721519, 'f1-score': 0.9032749098634072, 'support': 31600.0}
No log	10.0	410	0.4008	{'precision': 0.6432761242667825, 'recall': 0.6947442515251055, 'f1-score': 0.6680203045685279, 'support': 4262.0}	{'precision': 0.8905405405405405, 'recall': 0.9131639722863741, 'f1-score': 0.9017103762827823, 'support': 2165.0}	{'precision': 0.9983533673637411, 'recall': 0.9993406955661777, 'f1-score': 0.9988467874794069, 'support': 12134.0}	{'precision': 0.9138627187079408, 'recall': 0.8852672750977836, 'f1-score': 0.8993377483443709, 'support': 13039.0}	0.9053	{'precision': 0.8615081877197512, 'recall': 0.8731290486188602, 'f1-score': 0.866978804168772, 'support': 31600.0}	{'precision': 0.9082132550860689, 'recall': 0.9052848101265822, 'f1-score': 0.9065119405905274, 'support': 31600.0}
No log	11.0	451	0.4241	{'precision': 0.6283344541582605, 'recall': 0.6576724542468325, 'f1-score': 0.6426688066032329, 'support': 4262.0}	{'precision': 0.9258312020460358, 'recall': 0.836027713625866, 'f1-score': 0.8786407766990292, 'support': 2165.0}	{'precision': 0.9981066842278564, 'recall': 0.9992582825119499, 'f1-score': 0.9986821513878593, 'support': 12134.0}	{'precision': 0.8980515495550783, 'recall': 0.897844926758187, 'f1-score': 0.897948226270374, 'support': 13039.0}	0.9002	{'precision': 0.8625809724968077, 'recall': 0.8477008442857088, 'f1-score': 0.8544849902401238, 'support': 31600.0}	{'precision': 0.9019970461114446, 'recall': 0.9001582278481013, 'f1-score': 0.900875565904306, 'support': 31600.0}
No log	12.0	492	0.4143	{'precision': 0.6269706007669366, 'recall': 0.6905208822149226, 'f1-score': 0.6572130415364003, 'support': 4262.0}	{'precision': 0.9402756508422665, 'recall': 0.8508083140877598, 'f1-score': 0.8933074684772067, 'support': 2165.0}	{'precision': 0.998682042833608, 'recall': 0.9991758694577221, 'f1-score': 0.9989288951141139, 'support': 12134.0}	{'precision': 0.9047395955336925, 'recall': 0.8886417670066723, 'f1-score': 0.8966184322525729, 'support': 13039.0}	0.9018	{'precision': 0.8676669724941259, 'recall': 0.8572867081917691, 'f1-score': 0.8615169593450734, 'support': 31600.0}	{'precision': 0.9057833221028166, 'recall': 0.9017721518987342, 'f1-score': 0.9033880887258622, 'support': 31600.0}
0.1652	13.0	533	0.4456	{'precision': 0.6519438693351737, 'recall': 0.664946034725481, 'f1-score': 0.6583807643164131, 'support': 4262.0}	{'precision': 0.9468421052631579, 'recall': 0.8309468822170901, 'f1-score': 0.8851168511685117, 'support': 2165.0}	{'precision': 0.9975333004440059, 'recall': 0.9998351738915444, 'f1-score': 0.9986829107672046, 'support': 12134.0}	{'precision': 0.8987946327041164, 'recall': 0.909272183449651, 'f1-score': 0.9040030499428136, 'support': 13039.0}	0.9057	{'precision': 0.8737784769366135, 'recall': 0.8512500685709417, 'f1-score': 0.8615458940487357, 'support': 31600.0}	{'precision': 0.9067072852030946, 'recall': 0.9057278481012658, 'f1-score': 0.905937057207278, 'support': 31600.0}
0.1652	14.0	574	0.4601	{'precision': 0.6456975960458324, 'recall': 0.674331299859221, 'f1-score': 0.6597038907379777, 'support': 4262.0}	{'precision': 0.9215880893300248, 'recall': 0.8577367205542725, 'f1-score': 0.8885167464114833, 'support': 2165.0}	{'precision': 0.9972053263192504, 'recall': 0.9998351738915444, 'f1-score': 0.9985185185185186, 'support': 12134.0}	{'precision': 0.9043028994447871, 'recall': 0.8993787867167727, 'f1-score': 0.9018341215826509, 'support': 13039.0}	0.9047	{'precision': 0.8671984777849736, 'recall': 0.8578204952554527, 'f1-score': 0.8621433193126575, 'support': 31600.0}	{'precision': 0.9062815285811774, 'recall': 0.904746835443038, 'f1-score': 0.9053903656115826, 'support': 31600.0}
0.1652	15.0	615	0.4733	{'precision': 0.6527744124061061, 'recall': 0.6320976067573909, 'f1-score': 0.6422696388127309, 'support': 4262.0}	{'precision': 0.9225352112676056, 'recall': 0.8471131639722864, 'f1-score': 0.883216951601252, 'support': 2165.0}	{'precision': 0.9979432332373509, 'recall': 0.9996703477830888, 'f1-score': 0.9988060438881798, 'support': 12134.0}	{'precision': 0.8922730682670668, 'recall': 0.912186517370964, 'f1-score': 0.9021199135348325, 'support': 13039.0}	0.9035	{'precision': 0.8663814812945323, 'recall': 0.8477669089709325, 'f1-score': 0.8566031369592488, 'support': 31600.0}	{'precision': 0.9026204116235915, 'recall': 0.9035443037974683, 'f1-score': 0.9029041768973552, 'support': 31600.0}
0.1652	16.0	656	0.4725	{'precision': 0.6448837209302326, 'recall': 0.6506335053965274, 'f1-score': 0.6477458537724831, 'support': 4262.0}	{'precision': 0.9238191975622143, 'recall': 0.8401847575057737, 'f1-score': 0.8800193517174649, 'support': 2165.0}	{'precision': 0.9978613144690301, 'recall': 0.9997527608373167, 'f1-score': 0.9988061421925816, 'support': 12134.0}	{'precision': 0.8974495217853348, 'recall': 0.9067413145179845, 'f1-score': 0.9020714912448022, 'support': 13039.0}	0.9034	{'precision': 0.866003438686703, 'recall': 0.8493280845644006, 'f1-score': 0.857160709731833, 'support': 31600.0}	{'precision': 0.9037486229637037, 'recall': 0.9033544303797468, 'f1-score': 0.9034037540807719, 'support': 31600.0}

Framework versions

Transformers 4.37.2
Pytorch 2.2.0+cu121
Datasets 2.17.0
Tokenizers 0.15.2