metadata

license: apache-2.0
base_model: allenai/longformer-base-4096
tags:
  - generated_from_trainer
datasets:
  - essays_su_g
metrics:
  - accuracy
model-index:
  - name: longformer-sep_tok
    results:
      - task:
          name: Token Classification
          type: token-classification
        dataset:
          name: essays_su_g
          type: essays_su_g
          config: sep_tok
          split: train[80%:100%]
          args: sep_tok
        metrics:
          - name: Accuracy
            type: accuracy
            value: 0.8972900185154015

longformer-sep_tok

This model is a fine-tuned version of allenai/longformer-base-4096 on the essays_su_g dataset. It achieves the following results on the evaluation set:

Loss: 0.4170
Claim: {'precision': 0.6411675893306492, 'recall': 0.6113243761996161, 'f1-score': 0.6258904446082044, 'support': 4168.0}
Majorclaim: {'precision': 0.918905715681485, 'recall': 0.8740706319702602, 'f1-score': 0.8959276018099548, 'support': 2152.0}
O: {'precision': 0.9999116061168567, 'recall': 1.0, 'f1-score': 0.9999558011049724, 'support': 11312.0}
Premise: {'precision': 0.8821437232236683, 'recall': 0.9039178331814793, 'f1-score': 0.8928980526918672, 'support': 12073.0}
Accuracy: 0.8973
Macro avg: {'precision': 0.8605321585881648, 'recall': 0.8473282103378389, 'f1-score': 0.8536679750537497, 'support': 29705.0}
Weighted avg: {'precision': 0.8958422107843774, 'recall': 0.8972900185154015, 'f1-score': 0.8964216725962088, 'support': 29705.0}

Model description

More information needed

Intended uses & limitations

More information needed

Training and evaluation data

More information needed

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

learning_rate: 2e-05
train_batch_size: 8
eval_batch_size: 8
seed: 42
optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
lr_scheduler_type: linear
num_epochs: 14

Training results

Training Loss	Epoch	Step	Validation Loss	Claim	Majorclaim	O	Premise	Accuracy	Macro avg	Weighted avg
No log	1.0	41	0.3371	{'precision': 0.5051114023591088, 'recall': 0.4623320537428023, 'f1-score': 0.48277589878491795, 'support': 4168.0}	{'precision': 0.6749773345421578, 'recall': 0.6919144981412639, 'f1-score': 0.6833409821018817, 'support': 2152.0}	{'precision': 0.9997326917936381, 'recall': 0.9918670438472419, 'f1-score': 0.995784335478145, 'support': 11312.0}	{'precision': 0.8627718481662788, 'recall': 0.890499461608548, 'f1-score': 0.8764164017282138, 'support': 12073.0}	0.8546	{'precision': 0.760648319215296, 'recall': 0.759153264334964, 'f1-score': 0.7595794045232896, 'support': 29705.0}	{'precision': 0.8511387403281807, 'recall': 0.854637266453459, 'f1-score': 0.8526526632086279, 'support': 29705.0}
No log	2.0	82	0.2777	{'precision': 0.6244221879815101, 'recall': 0.388915547024952, 'f1-score': 0.4793021880544057, 'support': 4168.0}	{'precision': 0.7427921092564491, 'recall': 0.9098513011152416, 'f1-score': 0.8178780284043441, 'support': 2152.0}	{'precision': 0.9998232278592893, 'recall': 1.0, 'f1-score': 0.9999116061168567, 'support': 11312.0}	{'precision': 0.8542442434835473, 'recall': 0.9310858941439576, 'f1-score': 0.8910114140773621, 'support': 12073.0}	0.8797	{'precision': 0.8053204421451989, 'recall': 0.8074631855710378, 'f1-score': 0.7970258091632422, 'support': 29705.0}	{'precision': 0.869361097584513, 'recall': 0.8797172193233462, 'f1-score': 0.8694154495030058, 'support': 29705.0}
No log	3.0	123	0.2431	{'precision': 0.610574601641719, 'recall': 0.6067658349328215, 'f1-score': 0.6086642599277977, 'support': 4168.0}	{'precision': 0.8699186991869918, 'recall': 0.7955390334572491, 'f1-score': 0.8310679611650487, 'support': 2152.0}	{'precision': 0.9999115435647944, 'recall': 0.9992927864214993, 'f1-score': 0.9996020692399523, 'support': 11312.0}	{'precision': 0.8864117168429617, 'recall': 0.9023440735525553, 'f1-score': 0.8943069408529326, 'support': 12073.0}	0.8901	{'precision': 0.8417041403091168, 'recall': 0.8259854320910314, 'f1-score': 0.8334103077964329, 'support': 29705.0}	{'precision': 0.8897353313766411, 'recall': 0.8900521797677159, 'f1-score': 0.8897437196420146, 'support': 29705.0}
No log	4.0	164	0.2505	{'precision': 0.6349911190053286, 'recall': 0.5146353166986565, 'f1-score': 0.5685131195335278, 'support': 4168.0}	{'precision': 0.8684456928838952, 'recall': 0.8619888475836431, 'f1-score': 0.8652052238805971, 'support': 2152.0}	{'precision': 0.9999116061168567, 'recall': 1.0, 'f1-score': 0.9999558011049724, 'support': 11312.0}	{'precision': 0.8643422891753377, 'recall': 0.921974654187029, 'f1-score': 0.8922287683860366, 'support': 12073.0}	0.8902	{'precision': 0.8419226767953545, 'recall': 0.8246497046173321, 'f1-score': 0.8314757282262835, 'support': 29705.0}	{'precision': 0.8840849237740477, 'recall': 0.8901868372327891, 'f1-score': 0.8858731616505928, 'support': 29705.0}
No log	5.0	205	0.2614	{'precision': 0.6192818494835219, 'recall': 0.6041266794625719, 'f1-score': 0.6116103959193587, 'support': 4168.0}	{'precision': 0.8873449131513648, 'recall': 0.8308550185873605, 'f1-score': 0.8581713462922966, 'support': 2152.0}	{'precision': 0.9999116061168567, 'recall': 1.0, 'f1-score': 0.9999558011049724, 'support': 11312.0}	{'precision': 0.884087401510844, 'recall': 0.9015157790110163, 'f1-score': 0.892716535433071, 'support': 12073.0}	0.8922	{'precision': 0.8476564425656468, 'recall': 0.8341243692652373, 'f1-score': 0.8406135196874247, 'support': 29705.0}	{'precision': 0.8912748792655566, 'recall': 0.8921730348426191, 'f1-score': 0.8916089419894234, 'support': 29705.0}
No log	6.0	246	0.2767	{'precision': 0.6314779270633397, 'recall': 0.6314779270633397, 'f1-score': 0.6314779270633397, 'support': 4168.0}	{'precision': 0.8957934990439771, 'recall': 0.870817843866171, 'f1-score': 0.8831291234684261, 'support': 2152.0}	{'precision': 0.9999116061168567, 'recall': 1.0, 'f1-score': 0.9999558011049724, 'support': 11312.0}	{'precision': 0.8910319815364326, 'recall': 0.895386399403628, 'f1-score': 0.8932038834951457, 'support': 12073.0}	0.8964	{'precision': 0.8545537534401515, 'recall': 0.8494205425832846, 'f1-score': 0.851941683782971, 'support': 29705.0}	{'precision': 0.8964206972370266, 'recall': 0.8964147449924256, 'f1-score': 0.8964027733122504, 'support': 29705.0}
No log	7.0	287	0.2939	{'precision': 0.6163162705667276, 'recall': 0.6470729366602687, 'f1-score': 0.6313202247191011, 'support': 4168.0}	{'precision': 0.8980466888994759, 'recall': 0.8759293680297398, 'f1-score': 0.8868501529051988, 'support': 2152.0}	{'precision': 0.9999116061168567, 'recall': 1.0, 'f1-score': 0.9999558011049724, 'support': 11312.0}	{'precision': 0.8941008643114878, 'recall': 0.8825478340097739, 'f1-score': 0.8882867861609004, 'support': 12073.0}	0.8938	{'precision': 0.852093857473637, 'recall': 0.8513875346749455, 'f1-score': 0.8516032412225432, 'support': 29705.0}	{'precision': 0.895703838190886, 'recall': 0.8937552600572294, 'f1-score': 0.8946517629052753, 'support': 29705.0}
No log	8.0	328	0.3407	{'precision': 0.6260224948875256, 'recall': 0.5875719769673704, 'f1-score': 0.6061881188118811, 'support': 4168.0}	{'precision': 0.8996990972918756, 'recall': 0.8336431226765799, 'f1-score': 0.8654124457308249, 'support': 2152.0}	{'precision': 1.0, 'recall': 0.9994695898161244, 'f1-score': 0.9997347245556637, 'support': 11312.0}	{'precision': 0.8776114624189546, 'recall': 0.9081421353433281, 'f1-score': 0.8926158104697549, 'support': 12073.0}	0.8925	{'precision': 0.850833263649589, 'recall': 0.8322067062008507, 'f1-score': 0.8409877748920311, 'support': 29705.0}	{'precision': 0.8905173338443818, 'recall': 0.8925433428715704, 'f1-score': 0.8912475861436012, 'support': 29705.0}
No log	9.0	369	0.3397	{'precision': 0.6222800378429517, 'recall': 0.6312380038387716, 'f1-score': 0.6267270128632683, 'support': 4168.0}	{'precision': 0.8995675156174916, 'recall': 0.8698884758364313, 'f1-score': 0.8844790928419561, 'support': 2152.0}	{'precision': 0.9999115983026874, 'recall': 0.9999115983026874, 'f1-score': 0.9999115983026874, 'support': 11312.0}	{'precision': 0.8896888447533929, 'recall': 0.890499461608548, 'f1-score': 0.8900939686219315, 'support': 12073.0}	0.8943	{'precision': 0.852861999129131, 'recall': 0.8478843848966096, 'f1-score': 0.8503029181574608, 'support': 29705.0}	{'precision': 0.8948576305014636, 'recall': 0.8942938899175223, 'f1-score': 0.8945531621135354, 'support': 29705.0}
No log	10.0	410	0.3688	{'precision': 0.6277702397105382, 'recall': 0.6660268714011516, 'f1-score': 0.6463329452852155, 'support': 4168.0}	{'precision': 0.8976014760147601, 'recall': 0.904275092936803, 'f1-score': 0.9009259259259259, 'support': 2152.0}	{'precision': 0.9999116061168567, 'recall': 1.0, 'f1-score': 0.9999558011049724, 'support': 11312.0}	{'precision': 0.9005253346890357, 'recall': 0.8803114387476186, 'f1-score': 0.890303664921466, 'support': 12073.0}	0.8976	{'precision': 0.8564521641327977, 'recall': 0.8626533507713933, 'f1-score': 0.859379584309395, 'support': 29705.0}	{'precision': 0.8998898229116948, 'recall': 0.8975593334455478, 'f1-score': 0.8985976932246313, 'support': 29705.0}
No log	11.0	451	0.3741	{'precision': 0.6391675025075225, 'recall': 0.6115642994241842, 'f1-score': 0.6250613045610592, 'support': 4168.0}	{'precision': 0.8907678244972578, 'recall': 0.9056691449814126, 'f1-score': 0.8981566820276498, 'support': 2152.0}	{'precision': 0.9999116061168567, 'recall': 1.0, 'f1-score': 0.9999558011049724, 'support': 11312.0}	{'precision': 0.8859692206941716, 'recall': 0.8964631823076286, 'f1-score': 0.8911853102227345, 'support': 12073.0}	0.8966	{'precision': 0.8539540384539521, 'recall': 0.8534241566783063, 'f1-score': 0.8535897744791039, 'support': 29705.0}	{'precision': 0.8950778992965517, 'recall': 0.8965830668237671, 'f1-score': 0.8957707109763514, 'support': 29705.0}
No log	12.0	492	0.4274	{'precision': 0.6285550129273197, 'recall': 0.5249520153550864, 'f1-score': 0.572100928225912, 'support': 4168.0}	{'precision': 0.9079259610821072, 'recall': 0.8889405204460966, 'f1-score': 0.8983329420051654, 'support': 2152.0}	{'precision': 0.9999116061168567, 'recall': 1.0, 'f1-score': 0.9999558011049724, 'support': 11312.0}	{'precision': 0.8601999375195252, 'recall': 0.9122836080510229, 'f1-score': 0.8854765445994291, 'support': 12073.0}	0.8896	{'precision': 0.8491481294114522, 'recall': 0.8315440359630515, 'f1-score': 0.8389665539838698, 'support': 29705.0}	{'precision': 0.8843584546775584, 'recall': 0.8896482073724962, 'f1-score': 0.8860322338020225, 'support': 29705.0}
0.1761	13.0	533	0.4068	{'precision': 0.6263554926921263, 'recall': 0.6374760076775432, 'f1-score': 0.6318668252080857, 'support': 4168.0}	{'precision': 0.8843537414965986, 'recall': 0.9061338289962825, 'f1-score': 0.895111315125086, 'support': 2152.0}	{'precision': 0.9999116061168567, 'recall': 1.0, 'f1-score': 0.9999558011049724, 'support': 11312.0}	{'precision': 0.8937630807869401, 'recall': 0.8842872525470057, 'f1-score': 0.8889999167291198, 'support': 12073.0}	0.8953	{'precision': 0.8510959802731305, 'recall': 0.8569742723052078, 'f1-score': 0.853983464541816, 'support': 29705.0}	{'precision': 0.8959831916504318, 'recall': 0.8953038209055715, 'f1-score': 0.895616781497613, 'support': 29705.0}
0.1761	14.0	574	0.4170	{'precision': 0.6411675893306492, 'recall': 0.6113243761996161, 'f1-score': 0.6258904446082044, 'support': 4168.0}	{'precision': 0.918905715681485, 'recall': 0.8740706319702602, 'f1-score': 0.8959276018099548, 'support': 2152.0}	{'precision': 0.9999116061168567, 'recall': 1.0, 'f1-score': 0.9999558011049724, 'support': 11312.0}	{'precision': 0.8821437232236683, 'recall': 0.9039178331814793, 'f1-score': 0.8928980526918672, 'support': 12073.0}	0.8973	{'precision': 0.8605321585881648, 'recall': 0.8473282103378389, 'f1-score': 0.8536679750537497, 'support': 29705.0}	{'precision': 0.8958422107843774, 'recall': 0.8972900185154015, 'f1-score': 0.8964216725962088, 'support': 29705.0}

Framework versions

Transformers 4.37.2
Pytorch 2.2.0+cu121
Datasets 2.17.0
Tokenizers 0.15.2