|
--- |
|
license: apache-2.0 |
|
base_model: google/flan-t5-large |
|
tags: |
|
- generated_from_trainer |
|
metrics: |
|
- bleu |
|
model-index: |
|
- name: result |
|
results: [] |
|
--- |
|
|
|
<!-- This model card has been generated automatically according to the information the Trainer had access to. You |
|
should probably proofread and complete it, then remove this comment. --> |
|
|
|
# result |
|
|
|
This model is a fine-tuned version of [google/flan-t5-large](https://huggingface.co/google/flan-t5-large) on an unknown dataset. |
|
It achieves the following results on the evaluation set: |
|
- Loss: 0.5606 |
|
- Squad: {'exact_match': 31.547619047619047, 'f1': 65.97520968920112} |
|
- Bleu: {'bleu': 0.4478359370566898, 'precisions': [0.4970939125114714, 0.45436955820703, 0.43196470987444857, 0.4122681883024251], 'brevity_penalty': 1.0, 'length_ratio': 1.3752629364745477, 'translation_length': 3269, 'reference_length': 2377} |
|
|
|
## Model description |
|
|
|
More information needed |
|
|
|
## Intended uses & limitations |
|
|
|
More information needed |
|
|
|
## Training and evaluation data |
|
|
|
More information needed |
|
|
|
## Training procedure |
|
|
|
### Training hyperparameters |
|
|
|
The following hyperparameters were used during training: |
|
- learning_rate: 0.001 |
|
- train_batch_size: 8 |
|
- eval_batch_size: 8 |
|
- seed: 42 |
|
- optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08 |
|
- lr_scheduler_type: linear |
|
- num_epochs: 5 |
|
|
|
### Training results |
|
|
|
| Training Loss | Epoch | Step | Validation Loss | Squad | Bleu | |
|
|:-------------:|:-----:|:----:|:---------------:|:-------------------------------------------------------------:|:------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------:| |
|
| 0.3089 | 0.14 | 100 | 0.4816 | {'exact_match': 35.714285714285715, 'f1': 62.521389029334365} | {'bleu': 0.3454328094223805, 'precisions': [0.4022636892015907, 0.35407932924862945, 0.327451645741432, 0.30527817403708984], 'brevity_penalty': 1.0, 'length_ratio': 1.7017178552837064, 'translation_length': 3269, 'reference_length': 1921} | |
|
| 0.4283 | 0.27 | 200 | 0.4634 | {'exact_match': 19.047619047619047, 'f1': 58.11077854533359} | {'bleu': 0.34061189083646615, 'precisions': [0.598654022636892, 0.5469203482747501, 0.5269765863590091, 0.510342368045649], 'brevity_penalty': 0.6252757307345338, 'length_ratio': 0.6804746044962531, 'translation_length': 3269, 'reference_length': 4804} | |
|
| 0.4923 | 0.41 | 300 | 0.4645 | {'exact_match': 33.92857142857143, 'f1': 63.34981920724316} | {'bleu': 0.39658908018568106, 'precisions': [0.4472315692872438, 0.4040632054176072, 0.38004750593824227, 0.3601997146932953], 'brevity_penalty': 1.0, 'length_ratio': 1.5693710993759002, 'translation_length': 3269, 'reference_length': 2083} | |
|
| 0.5249 | 0.55 | 400 | 0.5079 | {'exact_match': 33.92857142857143, 'f1': 62.939136694434445} | {'bleu': 0.31207996448397607, 'precisions': [0.365861119608443, 0.3227990970654628, 0.29555480149304375, 0.2717546362339515], 'brevity_penalty': 1.0, 'length_ratio': 2.19543317662861, 'translation_length': 3269, 'reference_length': 1489} | |
|
| 0.4899 | 0.68 | 500 | 0.4844 | {'exact_match': 29.166666666666668, 'f1': 63.91710424811924} | {'bleu': 0.43831704036458624, 'precisions': [0.49036402569593146, 0.44437278297323446, 0.42144553783508654, 0.40192582025677603], 'brevity_penalty': 1.0, 'length_ratio': 1.307077169132347, 'translation_length': 3269, 'reference_length': 2501} | |
|
| 0.3923 | 0.82 | 600 | 0.4786 | {'exact_match': 29.166666666666668, 'f1': 62.65868074864891} | {'bleu': 0.4580633444938437, 'precisions': [0.5062710308962985, 0.4630764269590455, 0.44248388191381066, 0.4243937232524964], 'brevity_penalty': 1.0, 'length_ratio': 1.1974358974358974, 'translation_length': 3269, 'reference_length': 2730} | |
|
| 0.5221 | 0.96 | 700 | 0.4612 | {'exact_match': 30.952380952380953, 'f1': 62.84973727787755} | {'bleu': 0.5515506497257063, 'precisions': [0.6044661976139493, 0.563044179297001, 0.5446216491347132, 0.5281740370898717], 'brevity_penalty': 0.9860269604247005, 'length_ratio': 0.9861236802413273, 'translation_length': 3269, 'reference_length': 3315} | |
|
| 0.2892 | 1.1 | 800 | 0.4958 | {'exact_match': 30.952380952380953, 'f1': 63.021615827809455} | {'bleu': 0.4763133340972634, 'precisions': [0.5310492505353319, 0.48081264108352145, 0.4584323040380047, 0.4397289586305278], 'brevity_penalty': 1.0, 'length_ratio': 1.1742097701149425, 'translation_length': 3269, 'reference_length': 2784} | |
|
| 0.3726 | 1.23 | 900 | 0.4814 | {'exact_match': 28.571428571428573, 'f1': 65.07276724002743} | {'bleu': 0.4631031930079369, 'precisions': [0.514530437442643, 0.4669461464043857, 0.4462164913471327, 0.4290299572039943], 'brevity_penalty': 1.0, 'length_ratio': 1.3003182179793158, 'translation_length': 3269, 'reference_length': 2514} | |
|
| 0.3296 | 1.37 | 1000 | 0.4888 | {'exact_match': 30.952380952380953, 'f1': 63.75823569325163} | {'bleu': 0.3620195814265076, 'precisions': [0.4138880391557051, 0.37181554337310546, 0.34543603664743805, 0.3231098430813124], 'brevity_penalty': 1.0, 'length_ratio': 1.6885330578512396, 'translation_length': 3269, 'reference_length': 1936} | |
|
| 0.5811 | 1.51 | 1100 | 0.5143 | {'exact_match': 29.166666666666668, 'f1': 61.409131873413195} | {'bleu': 0.41718747361604114, 'precisions': [0.465891710003059, 0.4240567558851983, 0.4014251781472684, 0.3819543509272468], 'brevity_penalty': 1.0, 'length_ratio': 1.4541814946619218, 'translation_length': 3269, 'reference_length': 2248} | |
|
| 0.3257 | 1.64 | 1200 | 0.5088 | {'exact_match': 32.142857142857146, 'f1': 62.3134623709586} | {'bleu': 0.3381083910829391, 'precisions': [0.39186295503211993, 0.3473073202192841, 0.32168306752629794, 0.2985021398002853], 'brevity_penalty': 1.0, 'length_ratio': 1.8961716937354989, 'translation_length': 3269, 'reference_length': 1724} | |
|
| 0.3282 | 1.78 | 1300 | 0.4795 | {'exact_match': 35.714285714285715, 'f1': 64.30776389272819} | {'bleu': 0.37087590095858125, 'precisions': [0.4227592535943714, 0.38052241212512095, 0.3545978961655921, 0.33166904422253923], 'brevity_penalty': 1.0, 'length_ratio': 1.69730010384216, 'translation_length': 3269, 'reference_length': 1926} | |
|
| 0.3582 | 1.92 | 1400 | 0.5072 | {'exact_match': 31.547619047619047, 'f1': 64.26308637164007} | {'bleu': 0.3596752944859653, 'precisions': [0.4083817681248088, 0.36988068365043536, 0.34441805225653205, 0.32168330955777463], 'brevity_penalty': 1.0, 'length_ratio': 1.7951674903898958, 'translation_length': 3269, 'reference_length': 1821} | |
|
| 0.2635 | 2.05 | 1500 | 0.5226 | {'exact_match': 32.73809523809524, 'f1': 63.86332207970659} | {'bleu': 0.3937338044365284, 'precisions': [0.44447843377179563, 0.40148339245404707, 0.37801153715643027, 0.35627674750356636], 'brevity_penalty': 1.0, 'length_ratio': 1.4988537368179735, 'translation_length': 3269, 'reference_length': 2181} | |
|
| 0.2464 | 2.19 | 1600 | 0.5321 | {'exact_match': 33.333333333333336, 'f1': 67.38930420708819} | {'bleu': 0.45354685012711804, 'precisions': [0.5013765677577241, 0.4601741373750403, 0.43841194435018666, 0.41833095577746077], 'brevity_penalty': 1.0, 'length_ratio': 1.4225413402959095, 'translation_length': 3269, 'reference_length': 2298} | |
|
| 0.2393 | 2.33 | 1700 | 0.5099 | {'exact_match': 35.714285714285715, 'f1': 65.16842917580313} | {'bleu': 0.4114344617968027, 'precisions': [0.4588559192413582, 0.41921960657852303, 0.39667458432304037, 0.3755349500713267], 'brevity_penalty': 1.0, 'length_ratio': 1.5977517106549364, 'translation_length': 3269, 'reference_length': 2046} | |
|
| 0.2676 | 2.47 | 1800 | 0.5987 | {'exact_match': 29.761904761904763, 'f1': 63.78707251910303} | {'bleu': 0.3658122932518219, 'precisions': [0.42551238910981953, 0.37697516930022573, 0.34781133355955207, 0.3209700427960057], 'brevity_penalty': 1.0, 'length_ratio': 1.7187171398527865, 'translation_length': 3269, 'reference_length': 1902} | |
|
| 0.3071 | 2.6 | 1900 | 0.5240 | {'exact_match': 29.166666666666668, 'f1': 62.92129166698199} | {'bleu': 0.4832099557140429, 'precisions': [0.5377791373508718, 0.4872621734924218, 0.46521886664404477, 0.4472182596291013], 'brevity_penalty': 1.0, 'length_ratio': 1.0732107682206171, 'translation_length': 3269, 'reference_length': 3046} | |
|
| 0.2839 | 2.74 | 2000 | 0.5110 | {'exact_match': 35.714285714285715, 'f1': 65.45284344390186} | {'bleu': 0.43592975914902093, 'precisions': [0.4857754665035179, 0.4414704933892293, 0.4200882253138785, 0.4008559201141227], 'brevity_penalty': 1.0, 'length_ratio': 1.3502684840974803, 'translation_length': 3269, 'reference_length': 2421} | |
|
| 0.3259 | 2.88 | 2100 | 0.5020 | {'exact_match': 30.357142857142858, 'f1': 66.21243212925624} | {'bleu': 0.5131744912223959, 'precisions': [0.5646986846130315, 0.5169300225733634, 0.496776382762131, 0.4782453637660485], 'brevity_penalty': 1.0, 'length_ratio': 1.091121495327103, 'translation_length': 3269, 'reference_length': 2996} | |
|
| 0.3272 | 3.01 | 2200 | 0.5239 | {'exact_match': 30.952380952380953, 'f1': 67.09846522112723} | {'bleu': 0.5405154750897113, 'precisions': [0.5830529213826858, 0.5446630119316349, 0.5262979300984052, 0.5106990014265336], 'brevity_penalty': 1.0, 'length_ratio': 1.1176068376068375, 'translation_length': 3269, 'reference_length': 2925} | |
|
| 0.1856 | 3.15 | 2300 | 0.5524 | {'exact_match': 33.92857142857143, 'f1': 68.43656293435919} | {'bleu': 0.49285332133310633, 'precisions': [0.5396145610278372, 0.4982263785875524, 0.47777400746521886, 0.4593437945791726], 'brevity_penalty': 1.0, 'length_ratio': 1.2987683750496624, 'translation_length': 3269, 'reference_length': 2517} | |
|
| 0.233 | 3.29 | 2400 | 0.5277 | {'exact_match': 30.952380952380953, 'f1': 66.14618600499504} | {'bleu': 0.47457637309615686, 'precisions': [0.5221780360966657, 0.47887778136085135, 0.4594502884289108, 0.44151212553495006], 'brevity_penalty': 1.0, 'length_ratio': 1.2854895792371215, 'translation_length': 3269, 'reference_length': 2543} | |
|
| 0.2027 | 3.42 | 2500 | 0.5362 | {'exact_match': 30.952380952380953, 'f1': 67.73189400443187} | {'bleu': 0.4828086230052905, 'precisions': [0.5273784031814011, 0.4888745565946469, 0.46827281981676283, 0.4500713266761769], 'brevity_penalty': 1.0, 'length_ratio': 1.3672103722291928, 'translation_length': 3269, 'reference_length': 2391} | |
|
| 0.1462 | 3.56 | 2600 | 0.5681 | {'exact_match': 34.523809523809526, 'f1': 66.90869030594244} | {'bleu': 0.4470358204326154, 'precisions': [0.49434077699602325, 0.4524346984843599, 0.4316253817441466, 0.4136947218259629], 'brevity_penalty': 1.0, 'length_ratio': 1.4535349044019563, 'translation_length': 3269, 'reference_length': 2249} | |
|
| 0.2218 | 3.7 | 2700 | 0.5582 | {'exact_match': 30.952380952380953, 'f1': 64.88762964740765} | {'bleu': 0.3810698909088452, 'precisions': [0.4353013153869685, 0.38987423411802646, 0.36443841194435017, 0.340941512125535], 'brevity_penalty': 1.0, 'length_ratio': 1.5588936576061039, 'translation_length': 3269, 'reference_length': 2097} | |
|
| 0.2644 | 3.84 | 2800 | 0.5324 | {'exact_match': 33.92857142857143, 'f1': 66.27147689733899} | {'bleu': 0.46985264711666236, 'precisions': [0.5234016518813093, 0.4746855852950661, 0.4523243976925687, 0.43366619115549215], 'brevity_penalty': 1.0, 'length_ratio': 1.2926057730328193, 'translation_length': 3269, 'reference_length': 2529} | |
|
| 0.1883 | 3.97 | 2900 | 0.5218 | {'exact_match': 31.547619047619047, 'f1': 65.86132294563414} | {'bleu': 0.4691457249654658, 'precisions': [0.5151422453349648, 0.4756530151564012, 0.4540210383440787, 0.4354493580599144], 'brevity_penalty': 1.0, 'length_ratio': 1.3165525573902537, 'translation_length': 3269, 'reference_length': 2483} | |
|
| 0.0983 | 4.11 | 3000 | 0.5570 | {'exact_match': 34.523809523809526, 'f1': 66.88532922899014} | {'bleu': 0.4637264731606073, 'precisions': [0.5074946466809421, 0.4708158658497259, 0.4496097726501527, 0.4304564907275321], 'brevity_penalty': 1.0, 'length_ratio': 1.3834109183241643, 'translation_length': 3269, 'reference_length': 2363} | |
|
| 0.1298 | 4.25 | 3100 | 0.5555 | {'exact_match': 31.547619047619047, 'f1': 65.9162762624782} | {'bleu': 0.4642779990799932, 'precisions': [0.5126950137656776, 0.47049338922928086, 0.44859178825924667, 0.42938659058487877], 'brevity_penalty': 1.0, 'length_ratio': 1.3039489429597129, 'translation_length': 3269, 'reference_length': 2507} | |
|
| 0.1631 | 4.38 | 3200 | 0.5684 | {'exact_match': 32.73809523809524, 'f1': 67.06642684079529} | {'bleu': 0.5088183152910757, 'precisions': [0.5512389109819517, 0.5146726862302483, 0.494740413980319, 0.4775320970042796], 'brevity_penalty': 1.0, 'length_ratio': 1.2799530148786218, 'translation_length': 3269, 'reference_length': 2554} | |
|
| 0.1247 | 4.52 | 3300 | 0.5779 | {'exact_match': 32.142857142857146, 'f1': 66.7145922073288} | {'bleu': 0.4891310418038067, 'precisions': [0.5359437136739064, 0.4946791357626572, 0.47370206990159486, 0.4557774607703281], 'brevity_penalty': 1.0, 'length_ratio': 1.2592449922958397, 'translation_length': 3269, 'reference_length': 2596} | |
|
| 0.1375 | 4.66 | 3400 | 0.5710 | {'exact_match': 32.73809523809524, 'f1': 67.12997909883624} | {'bleu': 0.4788691854085839, 'precisions': [0.5240134597736311, 0.4853273137697517, 0.46420088225313877, 0.445435092724679], 'brevity_penalty': 1.0, 'length_ratio': 1.3485973597359735, 'translation_length': 3269, 'reference_length': 2424} | |
|
| 0.2406 | 4.79 | 3500 | 0.5565 | {'exact_match': 32.73809523809524, 'f1': 67.80127256596187} | {'bleu': 0.4744984435764912, 'precisions': [0.5206485163658611, 0.48113511770396644, 0.4594502884289108, 0.4404422253922967], 'brevity_penalty': 1.0, 'length_ratio': 1.4145391605365643, 'translation_length': 3269, 'reference_length': 2311} | |
|
| 0.1866 | 4.93 | 3600 | 0.5606 | {'exact_match': 31.547619047619047, 'f1': 65.97520968920112} | {'bleu': 0.4478359370566898, 'precisions': [0.4970939125114714, 0.45436955820703, 0.43196470987444857, 0.4122681883024251], 'brevity_penalty': 1.0, 'length_ratio': 1.3752629364745477, 'translation_length': 3269, 'reference_length': 2377} | |
|
|
|
|
|
### Framework versions |
|
|
|
- Transformers 4.34.0 |
|
- Pytorch 2.0.1+cu118 |
|
- Datasets 2.14.5 |
|
- Tokenizers 0.14.1 |
|
|
