|
--- |
|
license: apache-2.0 |
|
base_model: allenai/longformer-base-4096 |
|
tags: |
|
- generated_from_trainer |
|
datasets: |
|
- essays_su_g |
|
metrics: |
|
- accuracy |
|
model-index: |
|
- name: longformer-sep_tok |
|
results: |
|
- task: |
|
name: Token Classification |
|
type: token-classification |
|
dataset: |
|
name: essays_su_g |
|
type: essays_su_g |
|
config: sep_tok |
|
split: train[80%:100%] |
|
args: sep_tok |
|
metrics: |
|
- name: Accuracy |
|
type: accuracy |
|
value: 0.9037199124726477 |
|
--- |
|
|
|
<!-- This model card has been generated automatically according to the information the Trainer had access to. You |
|
should probably proofread and complete it, then remove this comment. --> |
|
|
|
# longformer-sep_tok |
|
|
|
This model is a fine-tuned version of [allenai/longformer-base-4096](https://huggingface.co/allenai/longformer-base-4096) on the essays_su_g dataset. |
|
It achieves the following results on the evaluation set: |
|
- Loss: 0.4627 |
|
- Claim: {'precision': 0.6641901931649331, 'recall': 0.6434740882917467, 'f1-score': 0.6536680477699245, 'support': 4168.0} |
|
- Majorclaim: {'precision': 0.9209900047596382, 'recall': 0.8991635687732342, 'f1-score': 0.909945920526687, 'support': 2152.0} |
|
- O: {'precision': 1.0, 'recall': 0.9999115983026874, 'f1-score': 0.9999557971975424, 'support': 11312.0} |
|
- Premise: {'precision': 0.8908200734394125, 'recall': 0.9042491509980949, 'f1-score': 0.8974843801381124, 'support': 12073.0} |
|
- Accuracy: 0.9037 |
|
- Macro avg: {'precision': 0.8690000678409959, 'recall': 0.8616996015914409, 'f1-score': 0.8652635364080665, 'support': 29705.0} |
|
- Weighted avg: {'precision': 0.9027835705096182, 'recall': 0.9037199124726477, 'f1-score': 0.9031988198412557, 'support': 29705.0} |
|
|
|
## Model description |
|
|
|
More information needed |
|
|
|
## Intended uses & limitations |
|
|
|
More information needed |
|
|
|
## Training and evaluation data |
|
|
|
More information needed |
|
|
|
## Training procedure |
|
|
|
### Training hyperparameters |
|
|
|
The following hyperparameters were used during training: |
|
- learning_rate: 2e-05 |
|
- train_batch_size: 8 |
|
- eval_batch_size: 8 |
|
- seed: 42 |
|
- optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08 |
|
- lr_scheduler_type: linear |
|
- num_epochs: 16 |
|
|
|
### Training results |
|
|
|
| Training Loss | Epoch | Step | Validation Loss | Claim | Majorclaim | O | Premise | Accuracy | Macro avg | Weighted avg | |
|
|:-------------:|:-----:|:----:|:---------------:|:--------------------------------------------------------------------------------------------------------------------:|:------------------------------------------------------------------------------------------------------------------:|:-------------------------------------------------------------------------------------------------------------------:|:-------------------------------------------------------------------------------------------------------------------:|:--------:|:-------------------------------------------------------------------------------------------------------------------:|:-------------------------------------------------------------------------------------------------------------------:| |
|
| No log | 1.0 | 41 | 0.3480 | {'precision': 0.45800144822592326, 'recall': 0.30350287907869483, 'f1-score': 0.3650793650793651, 'support': 4168.0} | {'precision': 0.6792168674698795, 'recall': 0.6287174721189591, 'f1-score': 0.652992277992278, 'support': 2152.0} | {'precision': 0.9994626063591581, 'recall': 0.986474540311174, 'f1-score': 0.9929261022378432, 'support': 11312.0} | {'precision': 0.8190918322936313, 'recall': 0.9353101963058064, 'f1-score': 0.873351637727677, 'support': 12073.0} | 0.8439 | {'precision': 0.7389431885871479, 'recall': 0.7135012719536586, 'f1-score': 0.7210873457592908, 'support': 29705.0} | {'precision': 0.8269800178224755, 'recall': 0.8439319979801381, 'f1-score': 0.8316056073620907, 'support': 29705.0} | |
|
| No log | 2.0 | 82 | 0.2758 | {'precision': 0.6302521008403361, 'recall': 0.32389635316698656, 'f1-score': 0.427892234548336, 'support': 4168.0} | {'precision': 0.7593291404612159, 'recall': 0.841542750929368, 'f1-score': 0.7983248842847697, 'support': 2152.0} | {'precision': 0.9997344192634561, 'recall': 0.9983203677510608, 'f1-score': 0.9990268931351733, 'support': 11312.0} | {'precision': 0.833381357153148, 'recall': 0.9582539551064359, 'f1-score': 0.8914659988441533, 'support': 12073.0} | 0.8760 | {'precision': 0.805674254429539, 'recall': 0.7805033567384627, 'f1-score': 0.779177502703108, 'support': 29705.0} | {'precision': 0.8628640276786139, 'recall': 0.876047803400101, 'f1-score': 0.8606332672536217, 'support': 29705.0} | |
|
| No log | 3.0 | 123 | 0.2410 | {'precision': 0.620671283963772, 'recall': 0.559021113243762, 'f1-score': 0.5882352941176471, 'support': 4168.0} | {'precision': 0.8549924736578023, 'recall': 0.79182156133829, 'f1-score': 0.822195416164053, 'support': 2152.0} | {'precision': 0.9999115357395613, 'recall': 0.9992043847241867, 'f1-score': 0.999557835160948, 'support': 11312.0} | {'precision': 0.8765607712976135, 'recall': 0.918744305475027, 'f1-score': 0.897156953936992, 'support': 12073.0} | 0.8897 | {'precision': 0.8380340161646872, 'recall': 0.8171978411953165, 'f1-score': 0.8267863748449099, 'support': 29705.0} | {'precision': 0.8860669651248813, 'recall': 0.8897155361050328, 'f1-score': 0.8873759763571568, 'support': 29705.0} | |
|
| No log | 4.0 | 164 | 0.2487 | {'precision': 0.6344057431534167, 'recall': 0.5724568138195777, 'f1-score': 0.6018413419094463, 'support': 4168.0} | {'precision': 0.8230162027420025, 'recall': 0.9205390334572491, 'f1-score': 0.8690502303136654, 'support': 2152.0} | {'precision': 0.9998231027772864, 'recall': 0.9992927864214993, 'f1-score': 0.9995578742594394, 'support': 11312.0} | {'precision': 0.8858637887335459, 'recall': 0.8974571357574753, 'f1-score': 0.8916227781435153, 'support': 12073.0} | 0.8923 | {'precision': 0.8357772093515627, 'recall': 0.8474364423639503, 'f1-score': 0.8405180561565165, 'support': 29705.0} | {'precision': 0.8894248936462209, 'recall': 0.8923076923076924, 'f1-score': 0.8904302737876794, 'support': 29705.0} | |
|
| No log | 5.0 | 205 | 0.2594 | {'precision': 0.6126252038201724, 'recall': 0.6309980806142035, 'f1-score': 0.6216759248315801, 'support': 4168.0} | {'precision': 0.8722222222222222, 'recall': 0.8754646840148699, 'f1-score': 0.8738404452690167, 'support': 2152.0} | {'precision': 1.0, 'recall': 0.9992927864214993, 'f1-score': 0.9996462681287585, 'support': 11312.0} | {'precision': 0.8930364914630063, 'recall': 0.8837902758220824, 'f1-score': 0.8883893260064111, 'support': 12073.0} | 0.8917 | {'precision': 0.8444709793763502, 'recall': 0.8473864567181638, 'f1-score': 0.8458879910589416, 'support': 29705.0} | {'precision': 0.8929161297147813, 'recall': 0.8917017337148628, 'f1-score': 0.8922798455096741, 'support': 29705.0} | |
|
| No log | 6.0 | 246 | 0.2812 | {'precision': 0.5880121396054628, 'recall': 0.7437619961612284, 'f1-score': 0.6567796610169492, 'support': 4168.0} | {'precision': 0.8901355773726041, 'recall': 0.8847583643122676, 'f1-score': 0.8874388254486133, 'support': 2152.0} | {'precision': 1.0, 'recall': 0.999557991513437, 'f1-score': 0.9997789469030461, 'support': 11312.0} | {'precision': 0.9225448257031037, 'recall': 0.8395593473039012, 'f1-score': 0.8790980052038161, 'support': 12073.0} | 0.8903 | {'precision': 0.8501731356702926, 'recall': 0.8669094248227085, 'f1-score': 0.8557738596431061, 'support': 29705.0} | {'precision': 0.9027534099005212, 'recall': 0.8903214946978623, 'f1-score': 0.8944647582453118, 'support': 29705.0} | |
|
| No log | 7.0 | 287 | 0.3027 | {'precision': 0.6093205574912892, 'recall': 0.6713051823416507, 'f1-score': 0.6388127853881279, 'support': 4168.0} | {'precision': 0.905252822778596, 'recall': 0.8568773234200744, 'f1-score': 0.8804010503700167, 'support': 2152.0} | {'precision': 1.0, 'recall': 0.9998231966053748, 'f1-score': 0.9999115904871364, 'support': 11312.0} | {'precision': 0.8979262281149074, 'recall': 0.8750931831359231, 'f1-score': 0.886362682998448, 'support': 12073.0} | 0.8927 | {'precision': 0.8531249020961982, 'recall': 0.8507747213757557, 'f1-score': 0.8513720273109323, 'support': 29705.0} | {'precision': 0.8968327052777144, 'recall': 0.8926780003366437, 'f1-score': 0.894437008359695, 'support': 29705.0} | |
|
| No log | 8.0 | 328 | 0.3308 | {'precision': 0.6094457623463446, 'recall': 0.6780230326295585, 'f1-score': 0.64190800681431, 'support': 4168.0} | {'precision': 0.8877551020408163, 'recall': 0.8489776951672863, 'f1-score': 0.8679334916864608, 'support': 2152.0} | {'precision': 1.0, 'recall': 0.9993811881188119, 'f1-score': 0.9996904982977407, 'support': 11312.0} | {'precision': 0.9026911576249466, 'recall': 0.875176012590077, 'f1-score': 0.8887206661619985, 'support': 12073.0} | 0.8929 | {'precision': 0.8499730055030268, 'recall': 0.8503894821264335, 'f1-score': 0.8495631657401276, 'support': 29705.0} | {'precision': 0.8975192480409824, 'recall': 0.8929136509005218, 'f1-score': 0.8948422476293271, 'support': 29705.0} | |
|
| No log | 9.0 | 369 | 0.3408 | {'precision': 0.651685393258427, 'recall': 0.6261996161228407, 'f1-score': 0.63868836412578, 'support': 4168.0} | {'precision': 0.9157330735509012, 'recall': 0.8736059479553904, 'f1-score': 0.8941736028537456, 'support': 2152.0} | {'precision': 1.0, 'recall': 0.9993811881188119, 'f1-score': 0.9996904982977407, 'support': 11312.0} | {'precision': 0.8864851725814292, 'recall': 0.9062370578977884, 'f1-score': 0.8962523039115298, 'support': 12073.0} | 0.9001 | {'precision': 0.8634759098476893, 'recall': 0.8513559525237079, 'f1-score': 0.857201192297199, 'support': 29705.0} | {'precision': 0.8988863080948749, 'recall': 0.9000504965494025, 'f1-score': 0.8993525560304815, 'support': 29705.0} | |
|
| No log | 10.0 | 410 | 0.4050 | {'precision': 0.6122782446311859, 'recall': 0.6293186180422264, 'f1-score': 0.6206814955040227, 'support': 4168.0} | {'precision': 0.8178170144462279, 'recall': 0.9470260223048327, 'f1-score': 0.8776916451335055, 'support': 2152.0} | {'precision': 0.9999116061168567, 'recall': 1.0, 'f1-score': 0.9999558011049724, 'support': 11312.0} | {'precision': 0.9038395316804407, 'recall': 0.869626439161766, 'f1-score': 0.8864029718434716, 'support': 12073.0} | 0.8912 | {'precision': 0.8334615992186778, 'recall': 0.8614927698772062, 'f1-score': 0.8461829783964931, 'support': 29705.0} | {'precision': 0.8932830396594146, 'recall': 0.89116310385457, 'f1-score': 0.8917298769484514, 'support': 29705.0} | |
|
| No log | 11.0 | 451 | 0.4124 | {'precision': 0.5987719669701461, 'recall': 0.6785028790786948, 'f1-score': 0.6361489146327746, 'support': 4168.0} | {'precision': 0.9018375241779497, 'recall': 0.866635687732342, 'f1-score': 0.8838862559241706, 'support': 2152.0} | {'precision': 1.0, 'recall': 1.0, 'f1-score': 1.0, 'support': 11312.0} | {'precision': 0.90087915876573, 'recall': 0.8657334548165327, 'f1-score': 0.8829567053854277, 'support': 12073.0} | 0.8907 | {'precision': 0.8503721624784565, 'recall': 0.8527180054068924, 'f1-score': 0.8507479689855932, 'support': 29705.0} | {'precision': 0.8963053356048198, 'recall': 0.8906581383605454, 'f1-score': 0.8929650968879477, 'support': 29705.0} | |
|
| No log | 12.0 | 492 | 0.4421 | {'precision': 0.6705202312138728, 'recall': 0.5566218809980806, 'f1-score': 0.6082852648138437, 'support': 4168.0} | {'precision': 0.9058880308880309, 'recall': 0.8722118959107806, 'f1-score': 0.8887310606060606, 'support': 2152.0} | {'precision': 0.9999115513886432, 'recall': 0.9993811881188119, 'f1-score': 0.9996462994075515, 'support': 11312.0} | {'precision': 0.8699774617237895, 'recall': 0.9271929097987244, 'f1-score': 0.8976744186046511, 'support': 12073.0} | 0.8987 | {'precision': 0.861574318803584, 'recall': 0.8388519687065994, 'f1-score': 0.8485842608580267, 'support': 29705.0} | {'precision': 0.8940729416216161, 'recall': 0.8987039218986702, 'f1-score': 0.8952534731823099, 'support': 29705.0} | |
|
| 0.1687 | 13.0 | 533 | 0.4406 | {'precision': 0.6625574087503021, 'recall': 0.6576295585412668, 'f1-score': 0.6600842865743528, 'support': 4168.0} | {'precision': 0.8832599118942731, 'recall': 0.9316914498141264, 'f1-score': 0.9068294889190412, 'support': 2152.0} | {'precision': 0.9999115826702034, 'recall': 0.9997347949080623, 'f1-score': 0.9998231809742728, 'support': 11312.0} | {'precision': 0.9014848181514848, 'recall': 0.8951379110411662, 'f1-score': 0.8983001537758197, 'support': 12073.0} | 0.9043 | {'precision': 0.8618034303665659, 'recall': 0.8710484285761554, 'f1-score': 0.8662592775608715, 'support': 29705.0} | {'precision': 0.9041218866445364, 'recall': 0.9042922066992088, 'f1-score': 0.9041543829763381, 'support': 29705.0} | |
|
| 0.1687 | 14.0 | 574 | 0.4457 | {'precision': 0.6631526104417671, 'recall': 0.6338771593090211, 'f1-score': 0.6481844946025515, 'support': 4168.0} | {'precision': 0.9062062529164723, 'recall': 0.9024163568773235, 'f1-score': 0.9043073341094298, 'support': 2152.0} | {'precision': 0.9999115748518879, 'recall': 0.9996463932107497, 'f1-score': 0.9997789664471067, 'support': 11312.0} | {'precision': 0.8905371260901459, 'recall': 0.90499461608548, 'f1-score': 0.897707665762879, 'support': 12073.0} | 0.9028 | {'precision': 0.8649518910750683, 'recall': 0.8602336313706436, 'f1-score': 0.8624946152304918, 'support': 29705.0} | {'precision': 0.9014182930351262, 'recall': 0.9028109745834034, 'f1-score': 0.902044324986091, 'support': 29705.0} | |
|
| 0.1687 | 15.0 | 615 | 0.4688 | {'precision': 0.6694429984383133, 'recall': 0.6170825335892515, 'f1-score': 0.6421972534332085, 'support': 4168.0} | {'precision': 0.925692083535697, 'recall': 0.8856877323420075, 'f1-score': 0.905248159582047, 'support': 2152.0} | {'precision': 0.9999115826702034, 'recall': 0.9997347949080623, 'f1-score': 0.9998231809742728, 'support': 11312.0} | {'precision': 0.882983832239475, 'recall': 0.9137745382257931, 'f1-score': 0.8981153580005699, 'support': 12073.0} | 0.9028 | {'precision': 0.8695076242209221, 'recall': 0.8540698997662786, 'f1-score': 0.8613459879975246, 'support': 29705.0} | {'precision': 0.9006427002542411, 'recall': 0.9028446389496718, 'f1-score': 0.9014549312254513, 'support': 29705.0} | |
|
| 0.1687 | 16.0 | 656 | 0.4627 | {'precision': 0.6641901931649331, 'recall': 0.6434740882917467, 'f1-score': 0.6536680477699245, 'support': 4168.0} | {'precision': 0.9209900047596382, 'recall': 0.8991635687732342, 'f1-score': 0.909945920526687, 'support': 2152.0} | {'precision': 1.0, 'recall': 0.9999115983026874, 'f1-score': 0.9999557971975424, 'support': 11312.0} | {'precision': 0.8908200734394125, 'recall': 0.9042491509980949, 'f1-score': 0.8974843801381124, 'support': 12073.0} | 0.9037 | {'precision': 0.8690000678409959, 'recall': 0.8616996015914409, 'f1-score': 0.8652635364080665, 'support': 29705.0} | {'precision': 0.9027835705096182, 'recall': 0.9037199124726477, 'f1-score': 0.9031988198412557, 'support': 29705.0} | |
|
|
|
|
|
### Framework versions |
|
|
|
- Transformers 4.37.2 |
|
- Pytorch 2.2.0+cu121 |
|
- Datasets 2.17.0 |
|
- Tokenizers 0.15.2 |
|
|
