metadata

license: apache-2.0
base_model: allenai/longformer-base-4096
tags:
  - generated_from_trainer
datasets:
  - essays_su_g
metrics:
  - accuracy
model-index:
  - name: longformer-sep_tok
    results:
      - task:
          name: Token Classification
          type: token-classification
        dataset:
          name: essays_su_g
          type: essays_su_g
          config: sep_tok
          split: train[20%:40%]
          args: sep_tok
        metrics:
          - name: Accuracy
            type: accuracy
            value: 0.9141753288565909

longformer-sep_tok

This model is a fine-tuned version of allenai/longformer-base-4096 on the essays_su_g dataset. It achieves the following results on the evaluation set:

Loss: 0.4313
Claim: {'precision': 0.6927914852443154, 'recall': 0.6577859439595773, 'f1-score': 0.6748350612629594, 'support': 4354.0}
Majorclaim: {'precision': 0.9005037783375315, 'recall': 0.913154533844189, 'f1-score': 0.9067850348763474, 'support': 2349.0}
O: {'precision': 0.9998404722022812, 'recall': 0.9999202297383536, 'f1-score': 0.999880349379811, 'support': 12536.0}
Premise: {'precision': 0.9048672566371682, 'recall': 0.9174517720951099, 'f1-score': 0.9111160614836266, 'support': 13374.0}
Accuracy: 0.9142
Macro avg: {'precision': 0.8745007481053241, 'recall': 0.8720781199093074, 'f1-score': 0.873154126750686, 'support': 32613.0}
Weighted avg: {'precision': 0.9127462162898813, 'recall': 0.9141753288565909, 'f1-score': 0.9133792098172753, 'support': 32613.0}

Model description

More information needed

Intended uses & limitations

More information needed

Training and evaluation data

More information needed

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

learning_rate: 2e-05
train_batch_size: 8
eval_batch_size: 8
seed: 42
optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
lr_scheduler_type: linear
num_epochs: 16

Training results

Training Loss	Epoch	Step	Validation Loss	Claim	Majorclaim	O	Premise	Accuracy	Macro avg	Weighted avg
No log	1.0	41	0.3274	{'precision': 0.5337112171837709, 'recall': 0.4108865411116215, 'f1-score': 0.4643135219309629, 'support': 4354.0}	{'precision': 0.6779310344827586, 'recall': 0.8369518944231588, 'f1-score': 0.7490950657268052, 'support': 2349.0}	{'precision': 0.9961064759634486, 'recall': 1.0, 'f1-score': 0.9980494407069782, 'support': 12536.0}	{'precision': 0.868321718931475, 'recall': 0.8944220128607746, 'f1-score': 0.8811786372007365, 'support': 13374.0}	0.8663	{'precision': 0.7690176116403633, 'recall': 0.7855651120988888, 'f1-score': 0.7731591663913707, 'support': 32613.0}	{'precision': 0.8590551035257559, 'recall': 0.8663109802839358, 'f1-score': 0.8609350954068932, 'support': 32613.0}
No log	2.0	82	0.2487	{'precision': 0.6105208603265094, 'recall': 0.5411116214974736, 'f1-score': 0.5737245829782053, 'support': 4354.0}	{'precision': 0.8533737024221453, 'recall': 0.8399318859088974, 'f1-score': 0.8465994421797898, 'support': 2349.0}	{'precision': 0.9999201915403033, 'recall': 0.9994416081684748, 'f1-score': 0.9996808425756004, 'support': 12536.0}	{'precision': 0.8785221391604371, 'recall': 0.9138627187079408, 'f1-score': 0.8958440225756799, 'support': 13374.0}	0.8917	{'precision': 0.8355842233623487, 'recall': 0.8235869585706966, 'f1-score': 0.8289622225773189, 'support': 32613.0}	{'precision': 0.8875950468565349, 'recall': 0.8916689663631068, 'f1-score': 0.8892060198210008, 'support': 32613.0}
No log	3.0	123	0.2409	{'precision': 0.648193359375, 'recall': 0.6097841065686724, 'f1-score': 0.6284023668639053, 'support': 4354.0}	{'precision': 0.8872870249017037, 'recall': 0.8646232439335888, 'f1-score': 0.8758085381630012, 'support': 2349.0}	{'precision': 0.9997607464710104, 'recall': 1.0, 'f1-score': 0.9998803589232302, 'support': 12536.0}	{'precision': 0.8920300971583023, 'recall': 0.9130402273067145, 'f1-score': 0.9024128884454791, 'support': 13374.0}	0.9025	{'precision': 0.8568178069765041, 'recall': 0.8468618944522439, 'f1-score': 0.8516260380989039, 'support': 32613.0}	{'precision': 0.900545253284536, 'recall': 0.9024928709410358, 'f1-score': 0.9013800727011249, 'support': 32613.0}
No log	4.0	164	0.2301	{'precision': 0.6427889207258835, 'recall': 0.6182820395039045, 'f1-score': 0.6302973542495903, 'support': 4354.0}	{'precision': 0.8526522593320236, 'recall': 0.9237973605789698, 'f1-score': 0.8868001634654679, 'support': 2349.0}	{'precision': 0.9997606510292005, 'recall': 0.9996011486917677, 'f1-score': 0.999680893498205, 'support': 12536.0}	{'precision': 0.9004945301963135, 'recall': 0.8986092418124719, 'f1-score': 0.8995508982035928, 'support': 13374.0}	0.9018	{'precision': 0.8489240903208553, 'recall': 0.8600724476467785, 'f1-score': 0.854082327354214, 'support': 32613.0}	{'precision': 0.9008001866175751, 'recall': 0.9018182933186153, 'f1-score': 0.9011744291494633, 'support': 32613.0}
No log	5.0	205	0.2631	{'precision': 0.6787741203178207, 'recall': 0.5493798805695912, 'f1-score': 0.6072607260726073, 'support': 4354.0}	{'precision': 0.9056947608200455, 'recall': 0.8463175819497658, 'f1-score': 0.875, 'support': 2349.0}	{'precision': 0.9997607273887382, 'recall': 0.9999202297383536, 'f1-score': 0.9998404722022812, 'support': 12536.0}	{'precision': 0.8748955140707718, 'recall': 0.9391356363092568, 'f1-score': 0.9058781103498017, 'support': 13374.0}	0.9038	{'precision': 0.864781280649344, 'recall': 0.8336883321417418, 'f1-score': 0.8469948271561726, 'support': 32613.0}	{'precision': 0.8989271945775551, 'recall': 0.903780700947475, 'f1-score': 0.899905013603967, 'support': 32613.0}
No log	6.0	246	0.2843	{'precision': 0.7390988372093024, 'recall': 0.467156637574644, 'f1-score': 0.57247396566282, 'support': 4354.0}	{'precision': 0.886744966442953, 'recall': 0.8999574286930608, 'f1-score': 0.8933023452355799, 'support': 2349.0}	{'precision': 0.9996809952946806, 'recall': 0.9999202297383536, 'f1-score': 0.9998005982053838, 'support': 12536.0}	{'precision': 0.8590842147543178, 'recall': 0.9595483774487812, 'f1-score': 0.9065413958745407, 'support': 13374.0}	0.9050	{'precision': 0.8711522534253133, 'recall': 0.8316456683637099, 'f1-score': 0.8430295762445812, 'support': 32613.0}	{'precision': 0.8991013862116997, 'recall': 0.9050378683347131, 'f1-score': 0.896835733694634, 'support': 32613.0}
No log	7.0	287	0.2983	{'precision': 0.6895514223194749, 'recall': 0.5790078089113458, 'f1-score': 0.6294631710362047, 'support': 4354.0}	{'precision': 0.8248984115256742, 'recall': 0.9506172839506173, 'f1-score': 0.8833069620253163, 'support': 2349.0}	{'precision': 0.9992028061224489, 'recall': 0.9998404594767071, 'f1-score': 0.9995215311004785, 'support': 12536.0}	{'precision': 0.8964686998394864, 'recall': 0.9187228951697323, 'f1-score': 0.9074593796159526, 'support': 13374.0}	0.9068	{'precision': 0.8525303349517711, 'recall': 0.8620471118771007, 'f1-score': 0.854937760944488, 'support': 32613.0}	{'precision': 0.9031788559978264, 'recall': 0.9068469628675682, 'f1-score': 0.9039933265062536, 'support': 32613.0}
No log	8.0	328	0.3043	{'precision': 0.6570592208961945, 'recall': 0.6701883325677538, 'f1-score': 0.6635588402501421, 'support': 4354.0}	{'precision': 0.8852592895059208, 'recall': 0.9229459344401874, 'f1-score': 0.9037098791162985, 'support': 2349.0}	{'precision': 0.9996012441183507, 'recall': 0.9998404594767071, 'f1-score': 0.9997208374875374, 'support': 12536.0}	{'precision': 0.9098907766990292, 'recall': 0.8969642590100194, 'f1-score': 0.9033812787107464, 'support': 13374.0}	0.9081	{'precision': 0.8629526328048738, 'recall': 0.8724847463736669, 'f1-score': 0.8675927088911811, 'support': 32613.0}	{'precision': 0.9088458701337473, 'recall': 0.9081041302548064, 'f1-score': 0.9084190763411705, 'support': 32613.0}
No log	9.0	369	0.3104	{'precision': 0.6748837209302325, 'recall': 0.6665135507579237, 'f1-score': 0.6706725213773977, 'support': 4354.0}	{'precision': 0.9106310885218127, 'recall': 0.9152830991911451, 'f1-score': 0.9129511677282377, 'support': 2349.0}	{'precision': 0.9996012123145638, 'recall': 0.9997606892150607, 'f1-score': 0.9996809444045626, 'support': 12536.0}	{'precision': 0.9039063664827792, 'recall': 0.9066098399880365, 'f1-score': 0.9052560848140958, 'support': 13374.0}	0.9110	{'precision': 0.8722555970623471, 'recall': 0.8720417947880416, 'f1-score': 0.8721401795810735, 'support': 32613.0}	{'precision': 0.9105988621342419, 'recall': 0.910986416459694, 'f1-score': 0.9107878958829343, 'support': 32613.0}
No log	10.0	410	0.3317	{'precision': 0.657001414427157, 'recall': 0.6401010564997703, 'f1-score': 0.6484411354118194, 'support': 4354.0}	{'precision': 0.8625536899648575, 'recall': 0.9404001702852277, 'f1-score': 0.8997963340122199, 'support': 2349.0}	{'precision': 0.999680969851651, 'recall': 0.9998404594767071, 'f1-score': 0.9997607083034219, 'support': 12536.0}	{'precision': 0.905666063893912, 'recall': 0.898758785703604, 'f1-score': 0.9021992043833972, 'support': 13374.0}	0.9061	{'precision': 0.8562255345343944, 'recall': 0.8697751179913272, 'f1-score': 0.8625493455277146, 'support': 32613.0}	{'precision': 0.905500915362609, 'recall': 0.9060803973875449, 'f1-score': 0.9056494861218845, 'support': 32613.0}
No log	11.0	451	0.3783	{'precision': 0.6712632356562424, 'recall': 0.6260909508497933, 'f1-score': 0.6478906714200832, 'support': 4354.0}	{'precision': 0.8602825745682888, 'recall': 0.9331630481055768, 'f1-score': 0.8952419848887073, 'support': 2349.0}	{'precision': 0.999680969851651, 'recall': 0.9998404594767071, 'f1-score': 0.9997607083034219, 'support': 12536.0}	{'precision': 0.9027922174365067, 'recall': 0.9090025422461493, 'f1-score': 0.9058867362146051, 'support': 13374.0}	0.9079	{'precision': 0.8585047493781722, 'recall': 0.8670242501695566, 'f1-score': 0.8621950252067043, 'support': 32613.0}	{'precision': 0.9060628476302188, 'recall': 0.9078894919203998, 'f1-score': 0.9067601525554976, 'support': 32613.0}
No log	12.0	492	0.4069	{'precision': 0.6924342105263158, 'recall': 0.5801561782269178, 'f1-score': 0.6313421644588852, 'support': 4354.0}	{'precision': 0.8779304769603881, 'recall': 0.9246487867177522, 'f1-score': 0.9006842214389384, 'support': 2349.0}	{'precision': 0.9996809952946806, 'recall': 0.9999202297383536, 'f1-score': 0.9998005982053838, 'support': 12536.0}	{'precision': 0.8889048165137615, 'recall': 0.927321668909825, 'f1-score': 0.9077069457659372, 'support': 13374.0}	0.9087	{'precision': 0.8647376248237866, 'recall': 0.8580117158982121, 'f1-score': 0.8598834824672862, 'support': 32613.0}	{'precision': 0.9044654345224509, 'recall': 0.908686720019624, 'f1-score': 0.905704596694275, 'support': 32613.0}
0.1707	13.0	533	0.3896	{'precision': 0.6659044134461468, 'recall': 0.6688102893890675, 'f1-score': 0.667354188151713, 'support': 4354.0}	{'precision': 0.8783068783068783, 'recall': 0.918688803746275, 'f1-score': 0.8980441115272575, 'support': 2349.0}	{'precision': 0.9998404849258254, 'recall': 1.0, 'f1-score': 0.9999202361011406, 'support': 12536.0}	{'precision': 0.9120422801057002, 'recall': 0.9032451024375654, 'f1-score': 0.9076223749953041, 'support': 13374.0}	0.9103	{'precision': 0.8640235141961377, 'recall': 0.8726860488932269, 'f1-score': 0.8682352276938539, 'support': 32613.0}	{'precision': 0.9105002436590061, 'recall': 0.9102505135988717, 'f1-score': 0.9103335319087843, 'support': 32613.0}
0.1707	14.0	574	0.4518	{'precision': 0.6987791932059448, 'recall': 0.6047312815801562, 'f1-score': 0.6483624722974637, 'support': 4354.0}	{'precision': 0.8978132884777124, 'recall': 0.9088974031502767, 'f1-score': 0.9033213454622383, 'support': 2349.0}	{'precision': 0.9998404722022812, 'recall': 0.9999202297383536, 'f1-score': 0.999880349379811, 'support': 12536.0}	{'precision': 0.8916008614501076, 'recall': 0.9286675639300135, 'f1-score': 0.9097568121886903, 'support': 13374.0}	0.9114	{'precision': 0.8720084538340115, 'recall': 0.8605541195997, 'f1-score': 0.8653302448320508, 'support': 32613.0}	{'precision': 0.9079115108212787, 'recall': 0.9113850305093061, 'f1-score': 0.9090381047714349, 'support': 32613.0}
0.1707	15.0	615	0.4267	{'precision': 0.6925925925925925, 'recall': 0.6442351860358291, 'f1-score': 0.6675392670157068, 'support': 4354.0}	{'precision': 0.904277848369335, 'recall': 0.9088974031502767, 'f1-score': 0.9065817409766455, 'support': 2349.0}	{'precision': 0.9998404722022812, 'recall': 0.9999202297383536, 'f1-score': 0.999880349379811, 'support': 12536.0}	{'precision': 0.9008415660446396, 'recall': 0.9204426499177508, 'f1-score': 0.9105366322719035, 'support': 13374.0}	0.9133	{'precision': 0.8743881198022121, 'recall': 0.8683738672105525, 'f1-score': 0.8711344974110167, 'support': 32613.0}	{'precision': 0.911340633421535, 'recall': 0.9132861128997639, 'f1-score': 0.9121529285245232, 'support': 32613.0}
0.1707	16.0	656	0.4313	{'precision': 0.6927914852443154, 'recall': 0.6577859439595773, 'f1-score': 0.6748350612629594, 'support': 4354.0}	{'precision': 0.9005037783375315, 'recall': 0.913154533844189, 'f1-score': 0.9067850348763474, 'support': 2349.0}	{'precision': 0.9998404722022812, 'recall': 0.9999202297383536, 'f1-score': 0.999880349379811, 'support': 12536.0}	{'precision': 0.9048672566371682, 'recall': 0.9174517720951099, 'f1-score': 0.9111160614836266, 'support': 13374.0}	0.9142	{'precision': 0.8745007481053241, 'recall': 0.8720781199093074, 'f1-score': 0.873154126750686, 'support': 32613.0}	{'precision': 0.9127462162898813, 'recall': 0.9141753288565909, 'f1-score': 0.9133792098172753, 'support': 32613.0}

Framework versions

Transformers 4.37.2
Pytorch 2.2.0+cu121
Datasets 2.17.0
Tokenizers 0.15.2