metadata

license: apache-2.0
base_model: allenai/longformer-base-4096
tags:
  - generated_from_trainer
datasets:
  - essays_su_g
metrics:
  - accuracy
model-index:
  - name: longformer-sep_tok
    results:
      - task:
          name: Token Classification
          type: token-classification
        dataset:
          name: essays_su_g
          type: essays_su_g
          config: sep_tok
          split: train[80%:100%]
          args: sep_tok
        metrics:
          - name: Accuracy
            type: accuracy
            value: 0.8996128597879145

longformer-sep_tok

This model is a fine-tuned version of allenai/longformer-base-4096 on the essays_su_g dataset. It achieves the following results on the evaluation set:

Loss: 0.4440
Claim: {'precision': 0.6588266384778013, 'recall': 0.5981285988483686, 'f1-score': 0.6270120724346077, 'support': 4168.0}
Majorclaim: {'precision': 0.9058606368251039, 'recall': 0.9121747211895911, 'f1-score': 0.9090067145172495, 'support': 2152.0}
O: {'precision': 1.0, 'recall': 0.999557991513437, 'f1-score': 0.9997789469030461, 'support': 11312.0}
Premise: {'precision': 0.8805334618783642, 'recall': 0.9078108175267124, 'f1-score': 0.8939641109298531, 'support': 12073.0}
Accuracy: 0.8996
Macro avg: {'precision': 0.8613051842953173, 'recall': 0.8544180322695273, 'f1-score': 0.857440461196189, 'support': 29705.0}
Weighted avg: {'precision': 0.8967541492974446, 'recall': 0.8996128597879145, 'f1-score': 0.8978925071931304, 'support': 29705.0}

Model description

More information needed

Intended uses & limitations

More information needed

Training and evaluation data

More information needed

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

learning_rate: 2e-05
train_batch_size: 8
eval_batch_size: 8
seed: 42
optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
lr_scheduler_type: linear
num_epochs: 15

Training results

Training Loss	Epoch	Step	Validation Loss	Claim	Majorclaim	O	Premise	Accuracy	Macro avg	Weighted avg
No log	1.0	41	0.3459	{'precision': 0.5076716016150741, 'recall': 0.45249520153550865, 'f1-score': 0.4784980337434987, 'support': 4168.0}	{'precision': 0.6691762621789193, 'recall': 0.7021375464684015, 'f1-score': 0.6852607709750567, 'support': 2152.0}	{'precision': 0.9995503597122302, 'recall': 0.98258486562942, 'f1-score': 0.9909950071326675, 'support': 11312.0}	{'precision': 0.8565651760228354, 'recall': 0.8948065932245507, 'f1-score': 0.8752683816082643, 'support': 12073.0}	0.8522	{'precision': 0.7582408498822648, 'recall': 0.7580060517144703, 'f1-score': 0.7575055483648718, 'support': 29705.0}	{'precision': 0.8484856957054067, 'recall': 0.8522134320821411, 'f1-score': 0.8499010831719419, 'support': 29705.0}
No log	2.0	82	0.2620	{'precision': 0.6607431340872375, 'recall': 0.3925143953934741, 'f1-score': 0.49247441300421435, 'support': 4168.0}	{'precision': 0.7220035778175313, 'recall': 0.9377323420074349, 'f1-score': 0.815847988680008, 'support': 2152.0}	{'precision': 0.9998230714791224, 'recall': 0.9991159830268741, 'f1-score': 0.9994694021931375, 'support': 11312.0}	{'precision': 0.8597867479055598, 'recall': 0.9350617079433446, 'f1-score': 0.8958457326508749, 'support': 12073.0}	0.8835	{'precision': 0.8105891328223628, 'recall': 0.8161061070927819, 'f1-score': 0.8009093841320587, 'support': 29705.0}	{'precision': 0.8752039412346267, 'recall': 0.8835212927116647, 'f1-score': 0.8729130325852121, 'support': 29705.0}
No log	3.0	123	0.2374	{'precision': 0.6255685898970553, 'recall': 0.6269193857965452, 'f1-score': 0.6262432594367885, 'support': 4168.0}	{'precision': 0.8762836185819071, 'recall': 0.8327137546468402, 'f1-score': 0.8539432928282107, 'support': 2152.0}	{'precision': 1.0, 'recall': 0.9992927864214993, 'f1-score': 0.9996462681287585, 'support': 11312.0}	{'precision': 0.8920272600377699, 'recall': 0.8998591899279383, 'f1-score': 0.8959261091868711, 'support': 12073.0}	0.8946	{'precision': 0.848469867129183, 'recall': 0.8396962791982058, 'f1-score': 0.8439397323951572, 'support': 29705.0}	{'precision': 0.894616305009769, 'recall': 0.8945632048476687, 'f1-score': 0.8945424128188674, 'support': 29705.0}
No log	4.0	164	0.2502	{'precision': 0.6409472880061116, 'recall': 0.6038867562380038, 'f1-score': 0.6218653489808523, 'support': 4168.0}	{'precision': 0.8598290598290599, 'recall': 0.9349442379182156, 'f1-score': 0.8958147818343721, 'support': 2152.0}	{'precision': 0.9998231340643792, 'recall': 0.9994695898161244, 'f1-score': 0.9996463306808134, 'support': 11312.0}	{'precision': 0.8900247320692498, 'recall': 0.8942267870454734, 'f1-score': 0.8921208114696525, 'support': 12073.0}	0.8965	{'precision': 0.8476560534922002, 'recall': 0.8581318427544543, 'f1-score': 0.8523618182414225, 'support': 29705.0}	{'precision': 0.8947008354139008, 'recall': 0.8965157380912304, 'f1-score': 0.8954149818075824, 'support': 29705.0}
No log	5.0	205	0.2594	{'precision': 0.6565992865636148, 'recall': 0.6624280230326296, 'f1-score': 0.6595007763047892, 'support': 4168.0}	{'precision': 0.8777137793531237, 'recall': 0.9205390334572491, 'f1-score': 0.8986164663188932, 'support': 2152.0}	{'precision': 0.9999115670321896, 'recall': 0.999557991513437, 'f1-score': 0.9997347480106101, 'support': 11312.0}	{'precision': 0.9036447423544198, 'recall': 0.8933156630497805, 'f1-score': 0.8984505164945018, 'support': 12073.0}	0.9033	{'precision': 0.8594673438258369, 'recall': 0.868960177763274, 'f1-score': 0.8640756267821986, 'support': 29705.0}	{'precision': 0.903761942443296, 'recall': 0.9033496044436964, 'f1-score': 0.9035049461804666, 'support': 29705.0}
No log	6.0	246	0.2753	{'precision': 0.637114951164538, 'recall': 0.6103646833013435, 'f1-score': 0.6234530082097782, 'support': 4168.0}	{'precision': 0.9366306027820711, 'recall': 0.8447955390334573, 'f1-score': 0.8883459565111166, 'support': 2152.0}	{'precision': 1.0, 'recall': 0.9994695898161244, 'f1-score': 0.9997347245556637, 'support': 11312.0}	{'precision': 0.8789410348977136, 'recall': 0.9074794997100969, 'f1-score': 0.8929823131469556, 'support': 12073.0}	0.8963	{'precision': 0.8631716472110806, 'recall': 0.8405273279652555, 'f1-score': 0.8511290006058785, 'support': 29705.0}	{'precision': 0.8952896579013939, 'recall': 0.8962800875273523, 'f1-score': 0.895480468184721, 'support': 29705.0}
No log	7.0	287	0.2966	{'precision': 0.6248019914007694, 'recall': 0.6624280230326296, 'f1-score': 0.643065098404565, 'support': 4168.0}	{'precision': 0.8643994834266036, 'recall': 0.9330855018587361, 'f1-score': 0.8974301675977654, 'support': 2152.0}	{'precision': 1.0, 'recall': 0.9997347949080623, 'f1-score': 0.9998673798682641, 'support': 11312.0}	{'precision': 0.9038098506950404, 'recall': 0.8724426406029985, 'f1-score': 0.8878492856239727, 'support': 12073.0}	0.8958	{'precision': 0.8482528313806034, 'recall': 0.8669227401006067, 'f1-score': 0.8570529828736417, 'support': 29705.0}	{'precision': 0.898436583603221, 'recall': 0.8958424507658643, 'f1-score': 0.8968547139279125, 'support': 29705.0}
No log	8.0	328	0.3421	{'precision': 0.6529668636013357, 'recall': 0.6098848368522073, 'f1-score': 0.6306909812678327, 'support': 4168.0}	{'precision': 0.9046478198370868, 'recall': 0.8773234200743495, 'f1-score': 0.8907761264449163, 'support': 2152.0}	{'precision': 1.0, 'recall': 0.9999115983026874, 'f1-score': 0.9999557971975424, 'support': 11312.0}	{'precision': 0.8831158369582729, 'recall': 0.9080593058891742, 'f1-score': 0.8954138930861273, 'support': 12073.0}	0.8990	{'precision': 0.8601826300991738, 'recall': 0.8487947902796046, 'f1-score': 0.8542091994991047, 'support': 29705.0}	{'precision': 0.8968936372791452, 'recall': 0.8989732368288167, 'f1-score': 0.8977445596081872, 'support': 29705.0}
No log	9.0	369	0.3315	{'precision': 0.6249177451195438, 'recall': 0.6835412667946257, 'f1-score': 0.6529162369657385, 'support': 4168.0}	{'precision': 0.9221616261774913, 'recall': 0.8643122676579925, 'f1-score': 0.8923003118253778, 'support': 2152.0}	{'precision': 1.0, 'recall': 0.9990275813295615, 'f1-score': 0.9995135541502675, 'support': 11312.0}	{'precision': 0.9006594521474467, 'recall': 0.8823821751014661, 'f1-score': 0.8914271369398771, 'support': 12073.0}	0.8976	{'precision': 0.8619347058611204, 'recall': 0.8573158227209113, 'f1-score': 0.8590393099703152, 'support': 29705.0}	{'precision': 0.901357029017618, 'recall': 0.8975929978118162, 'f1-score': 0.8991847263270282, 'support': 29705.0}
No log	10.0	410	0.3580	{'precision': 0.6654402102496715, 'recall': 0.6074856046065259, 'f1-score': 0.635143609682679, 'support': 4168.0}	{'precision': 0.8756989247311828, 'recall': 0.9460966542750929, 'f1-score': 0.9095376368103641, 'support': 2152.0}	{'precision': 0.9999115592111082, 'recall': 0.9994695898161244, 'f1-score': 0.9996905256642645, 'support': 11312.0}	{'precision': 0.8878382784479948, 'recall': 0.9021784146442475, 'f1-score': 0.8949509058789696, 'support': 12073.0}	0.9011	{'precision': 0.8572222431599893, 'recall': 0.8638075658354977, 'f1-score': 0.8598306695090694, 'support': 29705.0}	{'precision': 0.898432249649582, 'recall': 0.9010604275374516, 'f1-score': 0.8994393224226316, 'support': 29705.0}
No log	11.0	451	0.3818	{'precision': 0.6605737496826606, 'recall': 0.6242802303262955, 'f1-score': 0.6419143949673121, 'support': 4168.0}	{'precision': 0.9070716228467816, 'recall': 0.9298327137546468, 'f1-score': 0.9183111519045434, 'support': 2152.0}	{'precision': 0.9999115826702034, 'recall': 0.9997347949080623, 'f1-score': 0.9998231809742728, 'support': 11312.0}	{'precision': 0.8877551020408163, 'recall': 0.9007703139236313, 'f1-score': 0.8942153517247049, 'support': 12073.0}	0.9018	{'precision': 0.8638280143101154, 'recall': 0.8636545132281589, 'f1-score': 0.8635660198927083, 'support': 29705.0}	{'precision': 0.8999884427250537, 'recall': 0.901767379229086, 'f1-score': 0.9007765211808001, 'support': 29705.0}
No log	12.0	492	0.4293	{'precision': 0.6597707576181158, 'recall': 0.5662188099808061, 'f1-score': 0.6094254357650096, 'support': 4168.0}	{'precision': 0.9170684667309547, 'recall': 0.8838289962825279, 'f1-score': 0.9001419782300046, 'support': 2152.0}	{'precision': 1.0, 'recall': 0.9994695898161244, 'f1-score': 0.9997347245556637, 'support': 11312.0}	{'precision': 0.8703326011923439, 'recall': 0.9189927938374887, 'f1-score': 0.8940010475001007, 'support': 12073.0}	0.8976	{'precision': 0.8617929563853536, 'recall': 0.8421275474792367, 'f1-score': 0.8508257965126946, 'support': 29705.0}	{'precision': 0.8935526460983838, 'recall': 0.8975929978118162, 'f1-score': 0.8947808316465885, 'support': 29705.0}
0.1673	13.0	533	0.4672	{'precision': 0.686539643515673, 'recall': 0.5359884836852208, 'f1-score': 0.6019940716787928, 'support': 4168.0}	{'precision': 0.9140989729225023, 'recall': 0.9098513011152416, 'f1-score': 0.911970190964136, 'support': 2152.0}	{'precision': 1.0, 'recall': 0.9993811881188119, 'f1-score': 0.9996904982977407, 'support': 11312.0}	{'precision': 0.8640418332820671, 'recall': 0.9306717468731881, 'f1-score': 0.896119950552299, 'support': 12073.0}	0.8999	{'precision': 0.8661701124300606, 'recall': 0.8439731799481155, 'f1-score': 0.852443677873242, 'support': 29705.0}	{'precision': 0.8945367876491145, 'recall': 0.8999495034505975, 'f1-score': 0.8954393610999487, 'support': 29705.0}
0.1673	14.0	574	0.4210	{'precision': 0.6521739130434783, 'recall': 0.6477927063339731, 'f1-score': 0.6499759268175253, 'support': 4168.0}	{'precision': 0.9036697247706422, 'recall': 0.9154275092936803, 'f1-score': 0.909510618651893, 'support': 2152.0}	{'precision': 1.0, 'recall': 0.9994695898161244, 'f1-score': 0.9997347245556637, 'support': 11312.0}	{'precision': 0.8944449043795016, 'recall': 0.8948894226787045, 'f1-score': 0.8946671083140114, 'support': 12073.0}	0.9015	{'precision': 0.8625721355484055, 'recall': 0.8643948070306205, 'f1-score': 0.8634720945847734, 'support': 29705.0}	{'precision': 0.9013159888182246, 'recall': 0.9015317286652079, 'f1-score': 0.9014200207764028, 'support': 29705.0}
0.1673	15.0	615	0.4440	{'precision': 0.6588266384778013, 'recall': 0.5981285988483686, 'f1-score': 0.6270120724346077, 'support': 4168.0}	{'precision': 0.9058606368251039, 'recall': 0.9121747211895911, 'f1-score': 0.9090067145172495, 'support': 2152.0}	{'precision': 1.0, 'recall': 0.999557991513437, 'f1-score': 0.9997789469030461, 'support': 11312.0}	{'precision': 0.8805334618783642, 'recall': 0.9078108175267124, 'f1-score': 0.8939641109298531, 'support': 12073.0}	0.8996	{'precision': 0.8613051842953173, 'recall': 0.8544180322695273, 'f1-score': 0.857440461196189, 'support': 29705.0}	{'precision': 0.8967541492974446, 'recall': 0.8996128597879145, 'f1-score': 0.8978925071931304, 'support': 29705.0}

Framework versions

Transformers 4.37.2
Pytorch 2.2.0+cu121
Datasets 2.17.0
Tokenizers 0.15.2