Hugging Face's logo

silmi224
/
finetune-led-thousanddata

Summarization
Transformers
TensorBoard
Safetensors
led
text2text-generation
Generated from Trainer
Inference Endpoints
Model card Files Metrics Community
silmi224 commited on
Commit
6df9ce9
•
1 Parent(s): 8263e6a

Training complete

Browse files
Files changed (12) hide show
  1. README.md +185 -0
  2. config.json +59 -0
  3. generation_config.json +14 -0
  4. merges.txt +0 -0
  5. model.safetensors +3 -0
  6. runs/Jun23_04-59-24_ntfk28e35c/events.out.tfevents.1719118822.ntfk28e35c.5352.0 +3 -0
  7. runs/Jun23_04-59-24_ntfk28e35c/events.out.tfevents.1719119182.ntfk28e35c.5352.1 +3 -0
  8. special_tokens_map.json +51 -0
  9. tokenizer.json +0 -0
  10. tokenizer_config.json +57 -0
  11. training_args.bin +3 -0
  12. vocab.json +0 -0
README.md ADDED
@@ -0,0 +1,185 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ ---
2
+ tags:
3
+ - summarization
4
+ - generated_from_trainer
5
+ model-index:
6
+ - name: finetune-led-thousanddata
7
+ results: []
8
+ ---
9
+
10
+ <!-- This model card has been generated automatically according to the information the Trainer had access to. You
11
+ should probably proofread and complete it, then remove this comment. -->
12
+
13
+ # finetune-led-thousanddata
14
+
15
+ This model was trained from scratch on an unknown dataset.
16
+ It achieves the following results on the evaluation set:
17
+ - Loss: 1.9539
18
+ - Rouge1 Precision: 0.2722
19
+ - Rouge1 Recall: 0.3458
20
+ - Rouge1 Fmeasure: 0.3011
21
+
22
+ ## Model description
23
+
24
+ More information needed
25
+
26
+ ## Intended uses & limitations
27
+
28
+ More information needed
29
+
30
+ ## Training and evaluation data
31
+
32
+ More information needed
33
+
34
+ ## Training procedure
35
+
36
+ ### Training hyperparameters
37
+
38
+ The following hyperparameters were used during training:
39
+ - learning_rate: 5e-05
40
+ - train_batch_size: 8
41
+ - eval_batch_size: 8
42
+ - seed: 42
43
+ - gradient_accumulation_steps: 4
44
+ - total_train_batch_size: 32
45
+ - optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
46
+ - lr_scheduler_type: linear
47
+ - num_epochs: 4
48
+ - mixed_precision_training: Native AMP
49
+
50
+ ### Training results
51
+
52
+ | Training Loss | Epoch | Step | Validation Loss | Rouge1 Fmeasure | Rouge1 Precision | Rouge1 Recall |
53
+ |:-------------:|:-----:|:----:|:---------------:|:---------------:|:----------------:|:-------------:|
54
+ | 2.0529 | 0.13 | 10 | 2.6191 | 0.3014 | 0.2948 | 0.324 |
55
+ | 1.778 | 0.26 | 20 | 2.4690 | 0.2947 | 0.2802 | 0.3213 |
56
+ | 1.7425 | 0.38 | 30 | 2.3989 | 0.3037 | 0.2734 | 0.3524 |
57
+ | 1.7006 | 0.51 | 40 | 2.3216 | 0.2941 | 0.2665 | 0.3386 |
58
+ | 1.6751 | 0.64 | 50 | 2.3027 | 0.3101 | 0.282 | 0.3551 |
59
+ | 1.6887 | 0.77 | 60 | 2.2911 | 0.3058 | 0.2731 | 0.3577 |
60
+ | 1.6008 | 0.89 | 70 | 2.2476 | 0.3016 | 0.272 | 0.3487 |
61
+ | 1.5767 | 1.02 | 80 | 2.2167 | 0.3043 | 0.2775 | 0.3465 |
62
+ | 1.5046 | 1.15 | 90 | 2.2185 | 0.3004 | 0.2721 | 0.3458 |
63
+ | 1.5394 | 1.28 | 100 | 2.1977 | 0.2991 | 0.2696 | 0.3463 |
64
+ | 1.5449 | 1.41 | 110 | 2.1823 | 0.2978 | 0.2704 | 0.341 |
65
+ | 1.5073 | 1.53 | 120 | 2.1832 | 0.3057 | 0.276 | 0.3527 |
66
+ | 1.5232 | 0.42 | 130 | 2.2091 | 0.2955 | 0.2664 | 0.3424 |
67
+ | 1.4896 | 0.45 | 140 | 2.2069 | 0.2905 | 0.2574 | 0.3424 |
68
+ | 1.4848 | 0.48 | 150 | 2.1913 | 0.2868 | 0.2567 | 0.3356 |
69
+ | 1.5084 | 0.51 | 160 | 2.1826 | 0.3006 | 0.2755 | 0.3406 |
70
+ | 1.4322 | 0.54 | 170 | 2.2525 | 0.3049 | 0.2716 | 0.3582 |
71
+ | 1.4672 | 0.58 | 180 | 2.1890 | 0.2919 | 0.2663 | 0.3322 |
72
+ | 1.4543 | 0.61 | 190 | 2.1487 | 0.3022 | 0.276 | 0.344 |
73
+ | 1.5446 | 0.64 | 200 | 2.1496 | 0.2993 | 0.273 | 0.3418 |
74
+ | 1.412 | 0.67 | 210 | 2.1837 | 0.2976 | 0.268 | 0.3439 |
75
+ | 1.5241 | 0.7 | 220 | 2.1423 | 0.2913 | 0.2665 | 0.3305 |
76
+ | 1.4806 | 0.74 | 230 | 2.1303 | 0.2997 | 0.2736 | 0.3411 |
77
+ | 1.5405 | 0.77 | 240 | 2.1205 | 0.2966 | 0.2668 | 0.3428 |
78
+ | 1.4287 | 0.8 | 250 | 2.1322 | 0.2976 | 0.268 | 0.3442 |
79
+ | 1.4977 | 0.83 | 260 | 2.1334 | 0.2979 | 0.2665 | 0.3477 |
80
+ | 1.4171 | 0.86 | 270 | 2.1184 | 0.3043 | 0.2741 | 0.3509 |
81
+ | 1.4491 | 0.9 | 280 | 2.1038 | 0.2868 | 0.2628 | 0.3253 |
82
+ | 1.4316 | 0.93 | 290 | 2.1254 | 0.2958 | 0.2678 | 0.3393 |
83
+ | 1.4689 | 0.96 | 300 | 2.1052 | 0.299 | 0.2685 | 0.3471 |
84
+ | 1.4347 | 0.99 | 310 | 2.0815 | 0.3019 | 0.273 | 0.3476 |
85
+ | 1.3285 | 1.02 | 320 | 2.0877 | 0.2981 | 0.2695 | 0.3427 |
86
+ | 1.2636 | 1.06 | 330 | 2.0740 | 0.2933 | 0.2645 | 0.3382 |
87
+ | 1.32 | 1.09 | 340 | 2.0755 | 0.2997 | 0.2689 | 0.3487 |
88
+ | 1.357 | 1.12 | 350 | 2.0594 | 0.301 | 0.2743 | 0.3434 |
89
+ | 1.3412 | 1.15 | 360 | 2.0660 | 0.2961 | 0.2677 | 0.3405 |
90
+ | 1.327 | 1.18 | 370 | 2.0649 | 0.2912 | 0.263 | 0.335 |
91
+ | 1.3193 | 1.22 | 380 | 2.0842 | 0.2952 | 0.2673 | 0.3392 |
92
+ | 1.2961 | 1.25 | 390 | 2.0749 | 0.2957 | 0.2705 | 0.3342 |
93
+ | 1.3093 | 1.28 | 400 | 2.0715 | 0.2997 | 0.272 | 0.3441 |
94
+ | 1.3403 | 1.31 | 410 | 2.0671 | 0.3119 | 0.2823 | 0.3584 |
95
+ | 1.3685 | 1.34 | 420 | 2.0580 | 0.2973 | 0.2695 | 0.3409 |
96
+ | 1.2913 | 1.38 | 430 | 2.0685 | 0.2926 | 0.2632 | 0.339 |
97
+ | 1.3796 | 1.41 | 440 | 2.0339 | 0.2962 | 0.2697 | 0.3387 |
98
+ | 1.354 | 1.44 | 450 | 2.0371 | 0.2953 | 0.2665 | 0.3412 |
99
+ | 1.3268 | 1.47 | 460 | 2.0309 | 0.2957 | 0.2681 | 0.3395 |
100
+ | 1.3706 | 1.5 | 470 | 2.0215 | 0.2932 | 0.2685 | 0.3315 |
101
+ | 1.3288 | 1.54 | 480 | 2.0044 | 0.2948 | 0.2674 | 0.3374 |
102
+ | 1.4102 | 1.57 | 490 | 2.0046 | 0.2998 | 0.271 | 0.3446 |
103
+ | 1.3952 | 1.6 | 500 | 2.0044 | 0.3063 | 0.2794 | 0.3487 |
104
+ | 1.2994 | 1.63 | 510 | 1.9993 | 0.3052 | 0.2787 | 0.3461 |
105
+ | 1.2948 | 1.66 | 520 | 2.0168 | 0.3 | 0.2743 | 0.3406 |
106
+ | 1.2972 | 1.7 | 530 | 2.0290 | 0.3003 | 0.2734 | 0.342 |
107
+ | 1.3181 | 1.73 | 540 | 2.0234 | 0.2949 | 0.2676 | 0.338 |
108
+ | 1.3505 | 1.76 | 550 | 1.9942 | 0.301 | 0.2737 | 0.3436 |
109
+ | 1.3163 | 1.79 | 560 | 1.9983 | 0.2963 | 0.2705 | 0.3366 |
110
+ | 1.2876 | 1.82 | 570 | 2.0206 | 0.303 | 0.2739 | 0.3486 |
111
+ | 1.2895 | 1.86 | 580 | 2.0131 | 0.2958 | 0.2652 | 0.3443 |
112
+ | 1.3257 | 1.89 | 590 | 1.9888 | 0.3022 | 0.2743 | 0.3455 |
113
+ | 1.2891 | 1.92 | 600 | 1.9928 | 0.2972 | 0.2694 | 0.3408 |
114
+ | 1.3152 | 1.95 | 610 | 1.9785 | 0.292 | 0.2653 | 0.334 |
115
+ | 1.2834 | 1.98 | 620 | 2.0105 | 0.3039 | 0.2735 | 0.3511 |
116
+ | 1.2373 | 2.02 | 630 | 2.0023 | 0.3019 | 0.2735 | 0.346 |
117
+ | 1.2569 | 2.05 | 640 | 2.0006 | 0.3029 | 0.2753 | 0.3463 |
118
+ | 1.2337 | 2.08 | 650 | 1.9919 | 0.3006 | 0.2746 | 0.3416 |
119
+ | 1.1274 | 2.11 | 660 | 2.0095 | 0.3015 | 0.2732 | 0.3457 |
120
+ | 1.2178 | 2.14 | 670 | 1.9974 | 0.3031 | 0.275 | 0.3475 |
121
+ | 1.22 | 2.18 | 680 | 1.9924 | 0.3059 | 0.2777 | 0.3501 |
122
+ | 1.2913 | 2.21 | 690 | 1.9880 | 0.3044 | 0.2745 | 0.351 |
123
+ | 1.2441 | 2.24 | 700 | 1.9886 | 0.299 | 0.2721 | 0.3412 |
124
+ | 1.3258 | 2.27 | 710 | 1.9772 | 0.2956 | 0.2686 | 0.3377 |
125
+ | 1.158 | 2.3 | 720 | 2.0003 | 0.2983 | 0.2702 | 0.3424 |
126
+ | 1.1908 | 2.34 | 730 | 1.9845 | 0.2975 | 0.2705 | 0.3398 |
127
+ | 1.2411 | 2.37 | 740 | 1.9768 | 0.304 | 0.275 | 0.3493 |
128
+ | 1.1936 | 2.4 | 750 | 2.0065 | 0.293 | 0.2628 | 0.3403 |
129
+ | 1.1578 | 2.44 | 760 | 2.0199 | 0.301 | 0.2713 | 0.3473 |
130
+ | 1.2086 | 2.47 | 770 | 1.9949 | 0.2921 | 0.2664 | 0.3323 |
131
+ | 1.2574 | 2.5 | 780 | 1.9806 | 0.297 | 0.2693 | 0.3405 |
132
+ | 1.2331 | 2.53 | 790 | 2.0100 | 0.3012 | 0.2733 | 0.3446 |
133
+ | 1.2522 | 2.56 | 800 | 1.9969 | 0.301 | 0.2716 | 0.3468 |
134
+ | 1.2508 | 2.6 | 810 | 1.9931 | 0.3016 | 0.2719 | 0.3471 |
135
+ | 1.1558 | 2.63 | 820 | 1.9873 | 0.2986 | 0.2725 | 0.3402 |
136
+ | 1.2721 | 2.66 | 830 | 1.9763 | 0.2988 | 0.2671 | 0.348 |
137
+ | 1.2817 | 2.69 | 840 | 1.9713 | 0.2961 | 0.2688 | 0.3388 |
138
+ | 1.2183 | 2.72 | 850 | 1.9783 | 0.2985 | 0.2709 | 0.3416 |
139
+ | 1.2278 | 2.76 | 860 | 1.9757 | 0.2964 | 0.2681 | 0.3402 |
140
+ | 1.2087 | 2.79 | 870 | 1.9818 | 0.304 | 0.2735 | 0.3516 |
141
+ | 1.1838 | 2.82 | 880 | 1.9845 | 0.2916 | 0.2659 | 0.3312 |
142
+ | 1.1185 | 2.85 | 890 | 1.9912 | 0.3044 | 0.2759 | 0.3492 |
143
+ | 1.1214 | 2.88 | 900 | 1.9838 | 0.2995 | 0.2692 | 0.3468 |
144
+ | 1.2341 | 2.92 | 910 | 1.9685 | 0.296 | 0.2713 | 0.3344 |
145
+ | 1.1808 | 2.95 | 920 | 1.9803 | 0.3008 | 0.2725 | 0.345 |
146
+ | 1.2843 | 2.98 | 930 | 1.9645 | 0.3041 | 0.2745 | 0.3504 |
147
+ | 1.1824 | 3.01 | 940 | 1.9750 | 0.2985 | 0.2713 | 0.3412 |
148
+ | 1.1399 | 3.04 | 950 | 1.9762 | 0.2943 | 0.264 | 0.3416 |
149
+ | 1.1347 | 3.08 | 960 | 1.9841 | 0.2971 | 0.2685 | 0.3419 |
150
+ | 1.2298 | 3.11 | 970 | 1.9526 | 0.2993 | 0.2701 | 0.3448 |
151
+ | 1.1731 | 3.14 | 980 | 1.9787 | 0.304 | 0.2726 | 0.3531 |
152
+ | 1.1819 | 3.17 | 990 | 1.9570 | 0.2995 | 0.2715 | 0.3437 |
153
+ | 1.2072 | 3.2 | 1000 | 1.9613 | 0.3004 | 0.2705 | 0.3472 |
154
+ | 1.1214 | 3.24 | 1010 | 1.9670 | 0.3 | 0.2723 | 0.3432 |
155
+ | 1.226 | 3.27 | 1020 | 1.9676 | 0.2945 | 0.2639 | 0.3422 |
156
+ | 1.1956 | 3.3 | 1030 | 1.9721 | 0.2949 | 0.2657 | 0.3406 |
157
+ | 1.2286 | 3.33 | 1040 | 1.9572 | 0.3046 | 0.2759 | 0.3489 |
158
+ | 1.1786 | 3.36 | 1050 | 1.9549 | 0.3009 | 0.2728 | 0.3448 |
159
+ | 1.1512 | 3.4 | 1060 | 1.9609 | 0.2989 | 0.2699 | 0.3441 |
160
+ | 1.1897 | 3.43 | 1070 | 1.9626 | 0.2983 | 0.2697 | 0.3427 |
161
+ | 1.187 | 3.46 | 1080 | 1.9612 | 0.3016 | 0.2731 | 0.3457 |
162
+ | 1.1394 | 3.49 | 1090 | 1.9519 | 0.3015 | 0.2746 | 0.3431 |
163
+ | 1.1088 | 3.52 | 1100 | 1.9674 | 0.301 | 0.2709 | 0.3477 |
164
+ | 1.1787 | 3.56 | 1110 | 1.9549 | 0.3009 | 0.2728 | 0.3449 |
165
+ | 1.1961 | 3.59 | 1120 | 1.9545 | 0.3016 | 0.2722 | 0.3476 |
166
+ | 1.1194 | 3.62 | 1130 | 1.9693 | 0.3028 | 0.2735 | 0.3484 |
167
+ | 1.1991 | 3.65 | 1140 | 1.9538 | 0.3002 | 0.2706 | 0.3461 |
168
+ | 1.2109 | 3.68 | 1150 | 1.9428 | 0.3018 | 0.2729 | 0.3465 |
169
+ | 1.1389 | 3.72 | 1160 | 1.9578 | 0.3008 | 0.2723 | 0.3452 |
170
+ | 1.1922 | 3.75 | 1170 | 1.9576 | 0.2992 | 0.2701 | 0.3446 |
171
+ | 1.1002 | 3.78 | 1180 | 1.9571 | 0.299 | 0.2696 | 0.3445 |
172
+ | 1.1407 | 3.81 | 1190 | 1.9530 | 0.2979 | 0.2692 | 0.3422 |
173
+ | 1.1882 | 3.84 | 1200 | 1.9491 | 0.3009 | 0.2725 | 0.345 |
174
+ | 1.1755 | 3.88 | 1210 | 1.9562 | 0.3024 | 0.2735 | 0.3468 |
175
+ | 1.062 | 3.91 | 1220 | 1.9577 | 0.302 | 0.2722 | 0.3478 |
176
+ | 1.1965 | 3.94 | 1230 | 1.9575 | 0.3013 | 0.2716 | 0.3472 |
177
+ | 1.1255 | 3.97 | 1240 | 1.9550 | 0.3014 | 0.272 | 0.3466 |
178
+
179
+
180
+ ### Framework versions
181
+
182
+ - Transformers 4.35.2
183
+ - Pytorch 2.1.1+cu121
184
+ - Datasets 2.14.5
185
+ - Tokenizers 0.15.1
config.json ADDED
@@ -0,0 +1,59 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "_name_or_path": "./finetune/led/checkpoint-1240",
3
+ "activation_dropout": 0.0,
4
+ "activation_function": "gelu",
5
+ "architectures": [
6
+ "LEDForConditionalGeneration"
7
+ ],
8
+ "attention_dropout": 0.0,
9
+ "attention_window": [
10
+ 1024,
11
+ 1024,
12
+ 1024,
13
+ 1024,
14
+ 1024,
15
+ 1024
16
+ ],
17
+ "bos_token_id": 0,
18
+ "classif_dropout": 0.0,
19
+ "classifier_dropout": 0.0,
20
+ "d_model": 768,
21
+ "decoder_attention_heads": 12,
22
+ "decoder_ffn_dim": 3072,
23
+ "decoder_layerdrop": 0.0,
24
+ "decoder_layers": 6,
25
+ "decoder_start_token_id": 2,
26
+ "dropout": 0.1,
27
+ "early_stopping": true,
28
+ "encoder_attention_heads": 12,
29
+ "encoder_ffn_dim": 3072,
30
+ "encoder_layerdrop": 0.0,
31
+ "encoder_layers": 6,
32
+ "eos_token_id": 2,
33
+ "id2label": {
34
+ "0": "LABEL_0",
35
+ "1": "LABEL_1",
36
+ "2": "LABEL_2"
37
+ },
38
+ "init_std": 0.02,
39
+ "is_encoder_decoder": true,
40
+ "label2id": {
41
+ "LABEL_0": 0,
42
+ "LABEL_1": 1,
43
+ "LABEL_2": 2
44
+ },
45
+ "length_penalty": 2.0,
46
+ "max_decoder_position_embeddings": 1024,
47
+ "max_encoder_position_embeddings": 16384,
48
+ "max_length": 128,
49
+ "min_length": 40,
50
+ "model_type": "led",
51
+ "no_repeat_ngram_size": 3,
52
+ "num_beams": 2,
53
+ "num_hidden_layers": 6,
54
+ "pad_token_id": 1,
55
+ "torch_dtype": "float32",
56
+ "transformers_version": "4.35.2",
57
+ "use_cache": false,
58
+ "vocab_size": 50265
59
+ }
generation_config.json ADDED
@@ -0,0 +1,14 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "bos_token_id": 0,
3
+ "decoder_start_token_id": 2,
4
+ "early_stopping": true,
5
+ "eos_token_id": 2,
6
+ "length_penalty": 2.0,
7
+ "max_length": 128,
8
+ "min_length": 40,
9
+ "no_repeat_ngram_size": 3,
10
+ "num_beams": 2,
11
+ "pad_token_id": 1,
12
+ "transformers_version": "4.35.2",
13
+ "use_cache": false
14
+ }
merges.txt ADDED
The diff for this file is too large to render. See raw diff
 
model.safetensors ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:b46984c7b1d5e0b784a539c47841127ccbab8b9cee190d8fc0d9c7171548a87c
3
+ size 647614116
runs/Jun23_04-59-24_ntfk28e35c/events.out.tfevents.1719118822.ntfk28e35c.5352.0 ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:eb9418ca6797360c1b5e325aecc74415c5b770d9f40ef34b92972a3ed1591b69
3
+ size 5548
runs/Jun23_04-59-24_ntfk28e35c/events.out.tfevents.1719119182.ntfk28e35c.5352.1 ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:5faf632ffaf67bd2c01469cad411643e706c6da136db7dc19449ddf922e22af6
3
+ size 535
special_tokens_map.json ADDED
@@ -0,0 +1,51 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "bos_token": {
3
+ "content": "<s>",
4
+ "lstrip": false,
5
+ "normalized": true,
6
+ "rstrip": false,
7
+ "single_word": false
8
+ },
9
+ "cls_token": {
10
+ "content": "<s>",
11
+ "lstrip": false,
12
+ "normalized": true,
13
+ "rstrip": false,
14
+ "single_word": false
15
+ },
16
+ "eos_token": {
17
+ "content": "</s>",
18
+ "lstrip": false,
19
+ "normalized": true,
20
+ "rstrip": false,
21
+ "single_word": false
22
+ },
23
+ "mask_token": {
24
+ "content": "<mask>",
25
+ "lstrip": true,
26
+ "normalized": true,
27
+ "rstrip": false,
28
+ "single_word": false
29
+ },
30
+ "pad_token": {
31
+ "content": "<pad>",
32
+ "lstrip": false,
33
+ "normalized": true,
34
+ "rstrip": false,
35
+ "single_word": false
36
+ },
37
+ "sep_token": {
38
+ "content": "</s>",
39
+ "lstrip": false,
40
+ "normalized": true,
41
+ "rstrip": false,
42
+ "single_word": false
43
+ },
44
+ "unk_token": {
45
+ "content": "<unk>",
46
+ "lstrip": false,
47
+ "normalized": true,
48
+ "rstrip": false,
49
+ "single_word": false
50
+ }
51
+ }
tokenizer.json ADDED
The diff for this file is too large to render. See raw diff
 
tokenizer_config.json ADDED
@@ -0,0 +1,57 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "add_prefix_space": false,
3
+ "added_tokens_decoder": {
4
+ "0": {
5
+ "content": "<s>",
6
+ "lstrip": false,
7
+ "normalized": true,
8
+ "rstrip": false,
9
+ "single_word": false,
10
+ "special": true
11
+ },
12
+ "1": {
13
+ "content": "<pad>",
14
+ "lstrip": false,
15
+ "normalized": true,
16
+ "rstrip": false,
17
+ "single_word": false,
18
+ "special": true
19
+ },
20
+ "2": {
21
+ "content": "</s>",
22
+ "lstrip": false,
23
+ "normalized": true,
24
+ "rstrip": false,
25
+ "single_word": false,
26
+ "special": true
27
+ },
28
+ "3": {
29
+ "content": "<unk>",
30
+ "lstrip": false,
31
+ "normalized": true,
32
+ "rstrip": false,
33
+ "single_word": false,
34
+ "special": true
35
+ },
36
+ "50264": {
37
+ "content": "<mask>",
38
+ "lstrip": true,
39
+ "normalized": true,
40
+ "rstrip": false,
41
+ "single_word": false,
42
+ "special": true
43
+ }
44
+ },
45
+ "bos_token": "<s>",
46
+ "clean_up_tokenization_spaces": true,
47
+ "cls_token": "<s>",
48
+ "eos_token": "</s>",
49
+ "errors": "replace",
50
+ "mask_token": "<mask>",
51
+ "model_max_length": 16384,
52
+ "pad_token": "<pad>",
53
+ "sep_token": "</s>",
54
+ "tokenizer_class": "LEDTokenizer",
55
+ "trim_offsets": true,
56
+ "unk_token": "<unk>"
57
+ }
training_args.bin ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:fe46a1354b4d0462227a3d376d978ff5a959aa38698df130a651bf4dcc2a6376
3
+ size 4728
vocab.json ADDED
The diff for this file is too large to render. See raw diff