Hugging Face's logo

worldboss
/
ewe-tokenizer-v1

Transformers
Inference Endpoints
Model card Files Community
worldboss commited on
Commit
9f6884d
•
1 Parent(s): dd9d1f2

Upload tokenizer

Browse files
Files changed (6) hide show
  1. README.md +199 -0
  2. merges.txt +1 -0
  3. special_tokens_map.json +118 -0
  4. tokenizer.json +989 -0
  5. tokenizer_config.json +724 -0
  6. vocab.json +1 -0
README.md ADDED
@@ -0,0 +1,199 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ ---
2
+ library_name: transformers
3
+ tags: []
4
+ ---
5
+
6
+ # Model Card for Model ID
7
+
8
+ <!-- Provide a quick summary of what the model is/does. -->
9
+
10
+
11
+
12
+ ## Model Details
13
+
14
+ ### Model Description
15
+
16
+ <!-- Provide a longer summary of what this model is. -->
17
+
18
+ This is the model card of a 🤗 transformers model that has been pushed on the Hub. This model card has been automatically generated.
19
+
20
+ - **Developed by:** [More Information Needed]
21
+ - **Funded by [optional]:** [More Information Needed]
22
+ - **Shared by [optional]:** [More Information Needed]
23
+ - **Model type:** [More Information Needed]
24
+ - **Language(s) (NLP):** [More Information Needed]
25
+ - **License:** [More Information Needed]
26
+ - **Finetuned from model [optional]:** [More Information Needed]
27
+
28
+ ### Model Sources [optional]
29
+
30
+ <!-- Provide the basic links for the model. -->
31
+
32
+ - **Repository:** [More Information Needed]
33
+ - **Paper [optional]:** [More Information Needed]
34
+ - **Demo [optional]:** [More Information Needed]
35
+
36
+ ## Uses
37
+
38
+ <!-- Address questions around how the model is intended to be used, including the foreseeable users of the model and those affected by the model. -->
39
+
40
+ ### Direct Use
41
+
42
+ <!-- This section is for the model use without fine-tuning or plugging into a larger ecosystem/app. -->
43
+
44
+ [More Information Needed]
45
+
46
+ ### Downstream Use [optional]
47
+
48
+ <!-- This section is for the model use when fine-tuned for a task, or when plugged into a larger ecosystem/app -->
49
+
50
+ [More Information Needed]
51
+
52
+ ### Out-of-Scope Use
53
+
54
+ <!-- This section addresses misuse, malicious use, and uses that the model will not work well for. -->
55
+
56
+ [More Information Needed]
57
+
58
+ ## Bias, Risks, and Limitations
59
+
60
+ <!-- This section is meant to convey both technical and sociotechnical limitations. -->
61
+
62
+ [More Information Needed]
63
+
64
+ ### Recommendations
65
+
66
+ <!-- This section is meant to convey recommendations with respect to the bias, risk, and technical limitations. -->
67
+
68
+ Users (both direct and downstream) should be made aware of the risks, biases and limitations of the model. More information needed for further recommendations.
69
+
70
+ ## How to Get Started with the Model
71
+
72
+ Use the code below to get started with the model.
73
+
74
+ [More Information Needed]
75
+
76
+ ## Training Details
77
+
78
+ ### Training Data
79
+
80
+ <!-- This should link to a Dataset Card, perhaps with a short stub of information on what the training data is all about as well as documentation related to data pre-processing or additional filtering. -->
81
+
82
+ [More Information Needed]
83
+
84
+ ### Training Procedure
85
+
86
+ <!-- This relates heavily to the Technical Specifications. Content here should link to that section when it is relevant to the training procedure. -->
87
+
88
+ #### Preprocessing [optional]
89
+
90
+ [More Information Needed]
91
+
92
+
93
+ #### Training Hyperparameters
94
+
95
+ - **Training regime:** [More Information Needed] <!--fp32, fp16 mixed precision, bf16 mixed precision, bf16 non-mixed precision, fp16 non-mixed precision, fp8 mixed precision -->
96
+
97
+ #### Speeds, Sizes, Times [optional]
98
+
99
+ <!-- This section provides information about throughput, start/end time, checkpoint size if relevant, etc. -->
100
+
101
+ [More Information Needed]
102
+
103
+ ## Evaluation
104
+
105
+ <!-- This section describes the evaluation protocols and provides the results. -->
106
+
107
+ ### Testing Data, Factors & Metrics
108
+
109
+ #### Testing Data
110
+
111
+ <!-- This should link to a Dataset Card if possible. -->
112
+
113
+ [More Information Needed]
114
+
115
+ #### Factors
116
+
117
+ <!-- These are the things the evaluation is disaggregating by, e.g., subpopulations or domains. -->
118
+
119
+ [More Information Needed]
120
+
121
+ #### Metrics
122
+
123
+ <!-- These are the evaluation metrics being used, ideally with a description of why. -->
124
+
125
+ [More Information Needed]
126
+
127
+ ### Results
128
+
129
+ [More Information Needed]
130
+
131
+ #### Summary
132
+
133
+
134
+
135
+ ## Model Examination [optional]
136
+
137
+ <!-- Relevant interpretability work for the model goes here -->
138
+
139
+ [More Information Needed]
140
+
141
+ ## Environmental Impact
142
+
143
+ <!-- Total emissions (in grams of CO2eq) and additional considerations, such as electricity usage, go here. Edit the suggested text below accordingly -->
144
+
145
+ Carbon emissions can be estimated using the [Machine Learning Impact calculator](https://mlco2.github.io/impact#compute) presented in [Lacoste et al. (2019)](https://arxiv.org/abs/1910.09700).
146
+
147
+ - **Hardware Type:** [More Information Needed]
148
+ - **Hours used:** [More Information Needed]
149
+ - **Cloud Provider:** [More Information Needed]
150
+ - **Compute Region:** [More Information Needed]
151
+ - **Carbon Emitted:** [More Information Needed]
152
+
153
+ ## Technical Specifications [optional]
154
+
155
+ ### Model Architecture and Objective
156
+
157
+ [More Information Needed]
158
+
159
+ ### Compute Infrastructure
160
+
161
+ [More Information Needed]
162
+
163
+ #### Hardware
164
+
165
+ [More Information Needed]
166
+
167
+ #### Software
168
+
169
+ [More Information Needed]
170
+
171
+ ## Citation [optional]
172
+
173
+ <!-- If there is a paper or blog post introducing the model, the APA and Bibtex information for that should go in this section. -->
174
+
175
+ **BibTeX:**
176
+
177
+ [More Information Needed]
178
+
179
+ **APA:**
180
+
181
+ [More Information Needed]
182
+
183
+ ## Glossary [optional]
184
+
185
+ <!-- If relevant, include terms and calculations in this section that can help readers understand the model or model card. -->
186
+
187
+ [More Information Needed]
188
+
189
+ ## More Information [optional]
190
+
191
+ [More Information Needed]
192
+
193
+ ## Model Card Authors [optional]
194
+
195
+ [More Information Needed]
196
+
197
+ ## Model Card Contact
198
+
199
+ [More Information Needed]
merges.txt ADDED
@@ -0,0 +1 @@
 
 
1
+ #version: 0.2
special_tokens_map.json ADDED
@@ -0,0 +1,118 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "additional_special_tokens": [
3
+ "a",
4
+ "E",
5
+ "e",
6
+ "Ɛ",
7
+ "ɛ",
8
+ "Ə",
9
+ "ə",
10
+ "I",
11
+ "i",
12
+ "O",
13
+ "o",
14
+ "Ɔ",
15
+ "ɔ",
16
+ "U",
17
+ "u",
18
+ "ã",
19
+ "ẽ",
20
+ "ĩ",
21
+ "õ",
22
+ "ũ",
23
+ "B",
24
+ "b",
25
+ "D",
26
+ "d",
27
+ "Đ",
28
+ "ɖ",
29
+ "F",
30
+ "f",
31
+ "Ƒ",
32
+ "ƒ",
33
+ "G",
34
+ "g",
35
+ "Ɣ",
36
+ "ɣ",
37
+ "H",
38
+ "h",
39
+ "K",
40
+ "k",
41
+ "L",
42
+ "l",
43
+ "M",
44
+ "m",
45
+ "N",
46
+ "n",
47
+ "Ŋ",
48
+ "ŋ",
49
+ "P",
50
+ "p",
51
+ "R",
52
+ "r",
53
+ "S",
54
+ "s",
55
+ "T",
56
+ "t",
57
+ "V",
58
+ "v",
59
+ "Ʋ",
60
+ "ʋ",
61
+ "W",
62
+ "w",
63
+ "X",
64
+ "x",
65
+ "Y",
66
+ "y",
67
+ "Z",
68
+ "z",
69
+ "D",
70
+ "z",
71
+ "d",
72
+ "z",
73
+ "G",
74
+ "b",
75
+ "g",
76
+ "b",
77
+ "K",
78
+ "p",
79
+ "k",
80
+ "p",
81
+ "N",
82
+ "y",
83
+ "n",
84
+ "y",
85
+ "T",
86
+ "s",
87
+ "i",
88
+ "t",
89
+ "s",
90
+ "i",
91
+ "D",
92
+ "y",
93
+ "d",
94
+ "y",
95
+ "D",
96
+ "z",
97
+ "i",
98
+ "d",
99
+ "z",
100
+ "i",
101
+ "Ŋ",
102
+ "w",
103
+ "ŋ",
104
+ "w",
105
+ "ɔ",
106
+ "̃",
107
+ "ó",
108
+ "À",
109
+ "é",
110
+ "È",
111
+ "ò",
112
+ "à",
113
+ "í"
114
+ ],
115
+ "bos_token": "<|endoftext|>",
116
+ "eos_token": "<|endoftext|>",
117
+ "unk_token": "<|endoftext|>"
118
+ }
tokenizer.json ADDED
@@ -0,0 +1,989 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "version": "1.0",
3
+ "truncation": null,
4
+ "padding": null,
5
+ "added_tokens": [
6
+ {
7
+ "id": 0,
8
+ "content": "<|endoftext|>",
9
+ "single_word": false,
10
+ "lstrip": false,
11
+ "rstrip": false,
12
+ "normalized": true,
13
+ "special": true
14
+ },
15
+ {
16
+ "id": 1,
17
+ "content": "a",
18
+ "single_word": false,
19
+ "lstrip": false,
20
+ "rstrip": false,
21
+ "normalized": false,
22
+ "special": true
23
+ },
24
+ {
25
+ "id": 2,
26
+ "content": "E",
27
+ "single_word": false,
28
+ "lstrip": false,
29
+ "rstrip": false,
30
+ "normalized": false,
31
+ "special": true
32
+ },
33
+ {
34
+ "id": 3,
35
+ "content": "e",
36
+ "single_word": false,
37
+ "lstrip": false,
38
+ "rstrip": false,
39
+ "normalized": false,
40
+ "special": true
41
+ },
42
+ {
43
+ "id": 4,
44
+ "content": "Ɛ",
45
+ "single_word": false,
46
+ "lstrip": false,
47
+ "rstrip": false,
48
+ "normalized": false,
49
+ "special": true
50
+ },
51
+ {
52
+ "id": 5,
53
+ "content": "ɛ",
54
+ "single_word": false,
55
+ "lstrip": false,
56
+ "rstrip": false,
57
+ "normalized": false,
58
+ "special": true
59
+ },
60
+ {
61
+ "id": 6,
62
+ "content": "Ə",
63
+ "single_word": false,
64
+ "lstrip": false,
65
+ "rstrip": false,
66
+ "normalized": false,
67
+ "special": true
68
+ },
69
+ {
70
+ "id": 7,
71
+ "content": "ə",
72
+ "single_word": false,
73
+ "lstrip": false,
74
+ "rstrip": false,
75
+ "normalized": false,
76
+ "special": true
77
+ },
78
+ {
79
+ "id": 8,
80
+ "content": "I",
81
+ "single_word": false,
82
+ "lstrip": false,
83
+ "rstrip": false,
84
+ "normalized": false,
85
+ "special": true
86
+ },
87
+ {
88
+ "id": 9,
89
+ "content": "i",
90
+ "single_word": false,
91
+ "lstrip": false,
92
+ "rstrip": false,
93
+ "normalized": false,
94
+ "special": true
95
+ },
96
+ {
97
+ "id": 10,
98
+ "content": "O",
99
+ "single_word": false,
100
+ "lstrip": false,
101
+ "rstrip": false,
102
+ "normalized": false,
103
+ "special": true
104
+ },
105
+ {
106
+ "id": 11,
107
+ "content": "o",
108
+ "single_word": false,
109
+ "lstrip": false,
110
+ "rstrip": false,
111
+ "normalized": false,
112
+ "special": true
113
+ },
114
+ {
115
+ "id": 12,
116
+ "content": "Ɔ",
117
+ "single_word": false,
118
+ "lstrip": false,
119
+ "rstrip": false,
120
+ "normalized": false,
121
+ "special": true
122
+ },
123
+ {
124
+ "id": 13,
125
+ "content": "ɔ",
126
+ "single_word": false,
127
+ "lstrip": false,
128
+ "rstrip": false,
129
+ "normalized": false,
130
+ "special": true
131
+ },
132
+ {
133
+ "id": 14,
134
+ "content": "U",
135
+ "single_word": false,
136
+ "lstrip": false,
137
+ "rstrip": false,
138
+ "normalized": false,
139
+ "special": true
140
+ },
141
+ {
142
+ "id": 15,
143
+ "content": "u",
144
+ "single_word": false,
145
+ "lstrip": false,
146
+ "rstrip": false,
147
+ "normalized": false,
148
+ "special": true
149
+ },
150
+ {
151
+ "id": 16,
152
+ "content": "ã",
153
+ "single_word": false,
154
+ "lstrip": false,
155
+ "rstrip": false,
156
+ "normalized": false,
157
+ "special": true
158
+ },
159
+ {
160
+ "id": 17,
161
+ "content": "ẽ",
162
+ "single_word": false,
163
+ "lstrip": false,
164
+ "rstrip": false,
165
+ "normalized": false,
166
+ "special": true
167
+ },
168
+ {
169
+ "id": 18,
170
+ "content": "ĩ",
171
+ "single_word": false,
172
+ "lstrip": false,
173
+ "rstrip": false,
174
+ "normalized": false,
175
+ "special": true
176
+ },
177
+ {
178
+ "id": 19,
179
+ "content": "õ",
180
+ "single_word": false,
181
+ "lstrip": false,
182
+ "rstrip": false,
183
+ "normalized": false,
184
+ "special": true
185
+ },
186
+ {
187
+ "id": 20,
188
+ "content": "ũ",
189
+ "single_word": false,
190
+ "lstrip": false,
191
+ "rstrip": false,
192
+ "normalized": false,
193
+ "special": true
194
+ },
195
+ {
196
+ "id": 21,
197
+ "content": "B",
198
+ "single_word": false,
199
+ "lstrip": false,
200
+ "rstrip": false,
201
+ "normalized": false,
202
+ "special": true
203
+ },
204
+ {
205
+ "id": 22,
206
+ "content": "b",
207
+ "single_word": false,
208
+ "lstrip": false,
209
+ "rstrip": false,
210
+ "normalized": false,
211
+ "special": true
212
+ },
213
+ {
214
+ "id": 23,
215
+ "content": "D",
216
+ "single_word": false,
217
+ "lstrip": false,
218
+ "rstrip": false,
219
+ "normalized": false,
220
+ "special": true
221
+ },
222
+ {
223
+ "id": 24,
224
+ "content": "d",
225
+ "single_word": false,
226
+ "lstrip": false,
227
+ "rstrip": false,
228
+ "normalized": false,
229
+ "special": true
230
+ },
231
+ {
232
+ "id": 25,
233
+ "content": "Đ",
234
+ "single_word": false,
235
+ "lstrip": false,
236
+ "rstrip": false,
237
+ "normalized": false,
238
+ "special": true
239
+ },
240
+ {
241
+ "id": 26,
242
+ "content": "ɖ",
243
+ "single_word": false,
244
+ "lstrip": false,
245
+ "rstrip": false,
246
+ "normalized": false,
247
+ "special": true
248
+ },
249
+ {
250
+ "id": 27,
251
+ "content": "F",
252
+ "single_word": false,
253
+ "lstrip": false,
254
+ "rstrip": false,
255
+ "normalized": false,
256
+ "special": true
257
+ },
258
+ {
259
+ "id": 28,
260
+ "content": "f",
261
+ "single_word": false,
262
+ "lstrip": false,
263
+ "rstrip": false,
264
+ "normalized": false,
265
+ "special": true
266
+ },
267
+ {
268
+ "id": 29,
269
+ "content": "Ƒ",
270
+ "single_word": false,
271
+ "lstrip": false,
272
+ "rstrip": false,
273
+ "normalized": false,
274
+ "special": true
275
+ },
276
+ {
277
+ "id": 30,
278
+ "content": "ƒ",
279
+ "single_word": false,
280
+ "lstrip": false,
281
+ "rstrip": false,
282
+ "normalized": false,
283
+ "special": true
284
+ },
285
+ {
286
+ "id": 31,
287
+ "content": "G",
288
+ "single_word": false,
289
+ "lstrip": false,
290
+ "rstrip": false,
291
+ "normalized": false,
292
+ "special": true
293
+ },
294
+ {
295
+ "id": 32,
296
+ "content": "g",
297
+ "single_word": false,
298
+ "lstrip": false,
299
+ "rstrip": false,
300
+ "normalized": false,
301
+ "special": true
302
+ },
303
+ {
304
+ "id": 33,
305
+ "content": "Ɣ",
306
+ "single_word": false,
307
+ "lstrip": false,
308
+ "rstrip": false,
309
+ "normalized": false,
310
+ "special": true
311
+ },
312
+ {
313
+ "id": 34,
314
+ "content": "ɣ",
315
+ "single_word": false,
316
+ "lstrip": false,
317
+ "rstrip": false,
318
+ "normalized": false,
319
+ "special": true
320
+ },
321
+ {
322
+ "id": 35,
323
+ "content": "H",
324
+ "single_word": false,
325
+ "lstrip": false,
326
+ "rstrip": false,
327
+ "normalized": false,
328
+ "special": true
329
+ },
330
+ {
331
+ "id": 36,
332
+ "content": "h",
333
+ "single_word": false,
334
+ "lstrip": false,
335
+ "rstrip": false,
336
+ "normalized": false,
337
+ "special": true
338
+ },
339
+ {
340
+ "id": 37,
341
+ "content": "K",
342
+ "single_word": false,
343
+ "lstrip": false,
344
+ "rstrip": false,
345
+ "normalized": false,
346
+ "special": true
347
+ },
348
+ {
349
+ "id": 38,
350
+ "content": "k",
351
+ "single_word": false,
352
+ "lstrip": false,
353
+ "rstrip": false,
354
+ "normalized": false,
355
+ "special": true
356
+ },
357
+ {
358
+ "id": 39,
359
+ "content": "L",
360
+ "single_word": false,
361
+ "lstrip": false,
362
+ "rstrip": false,
363
+ "normalized": false,
364
+ "special": true
365
+ },
366
+ {
367
+ "id": 40,
368
+ "content": "l",
369
+ "single_word": false,
370
+ "lstrip": false,
371
+ "rstrip": false,
372
+ "normalized": false,
373
+ "special": true
374
+ },
375
+ {
376
+ "id": 41,
377
+ "content": "M",
378
+ "single_word": false,
379
+ "lstrip": false,
380
+ "rstrip": false,
381
+ "normalized": false,
382
+ "special": true
383
+ },
384
+ {
385
+ "id": 42,
386
+ "content": "m",
387
+ "single_word": false,
388
+ "lstrip": false,
389
+ "rstrip": false,
390
+ "normalized": false,
391
+ "special": true
392
+ },
393
+ {
394
+ "id": 43,
395
+ "content": "N",
396
+ "single_word": false,
397
+ "lstrip": false,
398
+ "rstrip": false,
399
+ "normalized": false,
400
+ "special": true
401
+ },
402
+ {
403
+ "id": 44,
404
+ "content": "n",
405
+ "single_word": false,
406
+ "lstrip": false,
407
+ "rstrip": false,
408
+ "normalized": false,
409
+ "special": true
410
+ },
411
+ {
412
+ "id": 45,
413
+ "content": "Ŋ",
414
+ "single_word": false,
415
+ "lstrip": false,
416
+ "rstrip": false,
417
+ "normalized": false,
418
+ "special": true
419
+ },
420
+ {
421
+ "id": 46,
422
+ "content": "ŋ",
423
+ "single_word": false,
424
+ "lstrip": false,
425
+ "rstrip": false,
426
+ "normalized": false,
427
+ "special": true
428
+ },
429
+ {
430
+ "id": 47,
431
+ "content": "P",
432
+ "single_word": false,
433
+ "lstrip": false,
434
+ "rstrip": false,
435
+ "normalized": false,
436
+ "special": true
437
+ },
438
+ {
439
+ "id": 48,
440
+ "content": "p",
441
+ "single_word": false,
442
+ "lstrip": false,
443
+ "rstrip": false,
444
+ "normalized": false,
445
+ "special": true
446
+ },
447
+ {
448
+ "id": 49,
449
+ "content": "R",
450
+ "single_word": false,
451
+ "lstrip": false,
452
+ "rstrip": false,
453
+ "normalized": false,
454
+ "special": true
455
+ },
456
+ {
457
+ "id": 50,
458
+ "content": "r",
459
+ "single_word": false,
460
+ "lstrip": false,
461
+ "rstrip": false,
462
+ "normalized": false,
463
+ "special": true
464
+ },
465
+ {
466
+ "id": 51,
467
+ "content": "S",
468
+ "single_word": false,
469
+ "lstrip": false,
470
+ "rstrip": false,
471
+ "normalized": false,
472
+ "special": true
473
+ },
474
+ {
475
+ "id": 52,
476
+ "content": "s",
477
+ "single_word": false,
478
+ "lstrip": false,
479
+ "rstrip": false,
480
+ "normalized": false,
481
+ "special": true
482
+ },
483
+ {
484
+ "id": 53,
485
+ "content": "T",
486
+ "single_word": false,
487
+ "lstrip": false,
488
+ "rstrip": false,
489
+ "normalized": false,
490
+ "special": true
491
+ },
492
+ {
493
+ "id": 54,
494
+ "content": "t",
495
+ "single_word": false,
496
+ "lstrip": false,
497
+ "rstrip": false,
498
+ "normalized": false,
499
+ "special": true
500
+ },
501
+ {
502
+ "id": 55,
503
+ "content": "V",
504
+ "single_word": false,
505
+ "lstrip": false,
506
+ "rstrip": false,
507
+ "normalized": false,
508
+ "special": true
509
+ },
510
+ {
511
+ "id": 56,
512
+ "content": "v",
513
+ "single_word": false,
514
+ "lstrip": false,
515
+ "rstrip": false,
516
+ "normalized": false,
517
+ "special": true
518
+ },
519
+ {
520
+ "id": 57,
521
+ "content": "Ʋ",
522
+ "single_word": false,
523
+ "lstrip": false,
524
+ "rstrip": false,
525
+ "normalized": false,
526
+ "special": true
527
+ },
528
+ {
529
+ "id": 58,
530
+ "content": "ʋ",
531
+ "single_word": false,
532
+ "lstrip": false,
533
+ "rstrip": false,
534
+ "normalized": false,
535
+ "special": true
536
+ },
537
+ {
538
+ "id": 59,
539
+ "content": "W",
540
+ "single_word": false,
541
+ "lstrip": false,
542
+ "rstrip": false,
543
+ "normalized": false,
544
+ "special": true
545
+ },
546
+ {
547
+ "id": 60,
548
+ "content": "w",
549
+ "single_word": false,
550
+ "lstrip": false,
551
+ "rstrip": false,
552
+ "normalized": false,
553
+ "special": true
554
+ },
555
+ {
556
+ "id": 61,
557
+ "content": "X",
558
+ "single_word": false,
559
+ "lstrip": false,
560
+ "rstrip": false,
561
+ "normalized": false,
562
+ "special": true
563
+ },
564
+ {
565
+ "id": 62,
566
+ "content": "x",
567
+ "single_word": false,
568
+ "lstrip": false,
569
+ "rstrip": false,
570
+ "normalized": false,
571
+ "special": true
572
+ },
573
+ {
574
+ "id": 63,
575
+ "content": "Y",
576
+ "single_word": false,
577
+ "lstrip": false,
578
+ "rstrip": false,
579
+ "normalized": false,
580
+ "special": true
581
+ },
582
+ {
583
+ "id": 64,
584
+ "content": "y",
585
+ "single_word": false,
586
+ "lstrip": false,
587
+ "rstrip": false,
588
+ "normalized": false,
589
+ "special": true
590
+ },
591
+ {
592
+ "id": 65,
593
+ "content": "Z",
594
+ "single_word": false,
595
+ "lstrip": false,
596
+ "rstrip": false,
597
+ "normalized": false,
598
+ "special": true
599
+ },
600
+ {
601
+ "id": 66,
602
+ "content": "z",
603
+ "single_word": false,
604
+ "lstrip": false,
605
+ "rstrip": false,
606
+ "normalized": false,
607
+ "special": true
608
+ },
609
+ {
610
+ "id": 67,
611
+ "content": "̃",
612
+ "single_word": false,
613
+ "lstrip": false,
614
+ "rstrip": false,
615
+ "normalized": false,
616
+ "special": true
617
+ },
618
+ {
619
+ "id": 68,
620
+ "content": "ó",
621
+ "single_word": false,
622
+ "lstrip": false,
623
+ "rstrip": false,
624
+ "normalized": false,
625
+ "special": true
626
+ },
627
+ {
628
+ "id": 69,
629
+ "content": "À",
630
+ "single_word": false,
631
+ "lstrip": false,
632
+ "rstrip": false,
633
+ "normalized": false,
634
+ "special": true
635
+ },
636
+ {
637
+ "id": 70,
638
+ "content": "é",
639
+ "single_word": false,
640
+ "lstrip": false,
641
+ "rstrip": false,
642
+ "normalized": false,
643
+ "special": true
644
+ },
645
+ {
646
+ "id": 71,
647
+ "content": "È",
648
+ "single_word": false,
649
+ "lstrip": false,
650
+ "rstrip": false,
651
+ "normalized": false,
652
+ "special": true
653
+ },
654
+ {
655
+ "id": 72,
656
+ "content": "ò",
657
+ "single_word": false,
658
+ "lstrip": false,
659
+ "rstrip": false,
660
+ "normalized": false,
661
+ "special": true
662
+ },
663
+ {
664
+ "id": 73,
665
+ "content": "à",
666
+ "single_word": false,
667
+ "lstrip": false,
668
+ "rstrip": false,
669
+ "normalized": false,
670
+ "special": true
671
+ },
672
+ {
673
+ "id": 74,
674
+ "content": "í",
675
+ "single_word": false,
676
+ "lstrip": false,
677
+ "rstrip": false,
678
+ "normalized": false,
679
+ "special": true
680
+ }
681
+ ],
682
+ "normalizer": null,
683
+ "pre_tokenizer": {
684
+ "type": "ByteLevel",
685
+ "add_prefix_space": false,
686
+ "trim_offsets": true,
687
+ "use_regex": true
688
+ },
689
+ "post_processor": {
690
+ "type": "ByteLevel",
691
+ "add_prefix_space": true,
692
+ "trim_offsets": false,
693
+ "use_regex": true
694
+ },
695
+ "decoder": {
696
+ "type": "ByteLevel",
697
+ "add_prefix_space": true,
698
+ "trim_offsets": true,
699
+ "use_regex": true
700
+ },
701
+ "model": {
702
+ "type": "BPE",
703
+ "dropout": null,
704
+ "unk_token": null,
705
+ "continuing_subword_prefix": "",
706
+ "end_of_word_suffix": "",
707
+ "fuse_unk": false,
708
+ "byte_fallback": false,
709
+ "ignore_merges": false,
710
+ "vocab": {
711
+ "<|endoftext|>": 0,
712
+ "a": 1,
713
+ "E": 2,
714
+ "e": 3,
715
+ "Ɛ": 4,
716
+ "ɛ": 5,
717
+ "Ə": 6,
718
+ "ə": 7,
719
+ "I": 8,
720
+ "i": 9,
721
+ "O": 10,
722
+ "o": 11,
723
+ "Ɔ": 12,
724
+ "ɔ": 13,
725
+ "U": 14,
726
+ "u": 15,
727
+ "ã": 16,
728
+ "ẽ": 17,
729
+ "ĩ": 18,
730
+ "õ": 19,
731
+ "ũ": 20,
732
+ "B": 21,
733
+ "b": 22,
734
+ "D": 23,
735
+ "d": 24,
736
+ "Đ": 25,
737
+ "ɖ": 26,
738
+ "F": 27,
739
+ "f": 28,
740
+ "Ƒ": 29,
741
+ "ƒ": 30,
742
+ "G": 31,
743
+ "g": 32,
744
+ "Ɣ": 33,
745
+ "ɣ": 34,
746
+ "H": 35,
747
+ "h": 36,
748
+ "K": 37,
749
+ "k": 38,
750
+ "L": 39,
751
+ "l": 40,
752
+ "M": 41,
753
+ "m": 42,
754
+ "N": 43,
755
+ "n": 44,
756
+ "Ŋ": 45,
757
+ "ŋ": 46,
758
+ "P": 47,
759
+ "p": 48,
760
+ "R": 49,
761
+ "r": 50,
762
+ "S": 51,
763
+ "s": 52,
764
+ "T": 53,
765
+ "t": 54,
766
+ "V": 55,
767
+ "v": 56,
768
+ "Ʋ": 57,
769
+ "ʋ": 58,
770
+ "W": 59,
771
+ "w": 60,
772
+ "X": 61,
773
+ "x": 62,
774
+ "Y": 63,
775
+ "y": 64,
776
+ "Z": 65,
777
+ "z": 66,
778
+ "̃": 67,
779
+ "ó": 68,
780
+ "À": 69,
781
+ "é": 70,
782
+ "È": 71,
783
+ "ò": 72,
784
+ "à": 73,
785
+ "í": 74,
786
+ "!": 75,
787
+ "\"": 76,
788
+ "#": 77,
789
+ "$": 78,
790
+ "%": 79,
791
+ "&": 80,
792
+ "'": 81,
793
+ "(": 82,
794
+ ")": 83,
795
+ "*": 84,
796
+ "+": 85,
797
+ ",": 86,
798
+ "-": 87,
799
+ ".": 88,
800
+ "/": 89,
801
+ "0": 90,
802
+ "1": 91,
803
+ "2": 92,
804
+ "3": 93,
805
+ "4": 94,
806
+ "5": 95,
807
+ "6": 96,
808
+ "7": 97,
809
+ "8": 98,
810
+ "9": 99,
811
+ ":": 100,
812
+ ";": 101,
813
+ "<": 102,
814
+ "=": 103,
815
+ ">": 104,
816
+ "?": 105,
817
+ "@": 106,
818
+ "A": 107,
819
+ "C": 108,
820
+ "J": 109,
821
+ "Q": 110,
822
+ "[": 111,
823
+ "\\": 112,
824
+ "]": 113,
825
+ "^": 114,
826
+ "_": 115,
827
+ "`": 116,
828
+ "c": 117,
829
+ "j": 118,
830
+ "q": 119,
831
+ "{": 120,
832
+ "|": 121,
833
+ "}": 122,
834
+ "~": 123,
835
+ "¡": 124,
836
+ "¢": 125,
837
+ "£": 126,
838
+ "¤": 127,
839
+ "¥": 128,
840
+ "¦": 129,
841
+ "§": 130,
842
+ "¨": 131,
843
+ "©": 132,
844
+ "ª": 133,
845
+ "«": 134,
846
+ "¬": 135,
847
+ "®": 136,
848
+ "¯": 137,
849
+ "°": 138,
850
+ "±": 139,
851
+ "²": 140,
852
+ "³": 141,
853
+ "´": 142,
854
+ "µ": 143,
855
+ "¶": 144,
856
+ "·": 145,
857
+ "¸": 146,
858
+ "¹": 147,
859
+ "º": 148,
860
+ "»": 149,
861
+ "¼": 150,
862
+ "½": 151,
863
+ "¾": 152,
864
+ "¿": 153,
865
+ "Á": 154,
866
+ "Â": 155,
867
+ "Ã": 156,
868
+ "Ä": 157,
869
+ "Å": 158,
870
+ "Æ": 159,
871
+ "Ç": 160,
872
+ "É": 161,
873
+ "Ê": 162,
874
+ "Ë": 163,
875
+ "Ì": 164,
876
+ "Í": 165,
877
+ "Î": 166,
878
+ "Ï": 167,
879
+ "Ð": 168,
880
+ "Ñ": 169,
881
+ "Ò": 170,
882
+ "Ó": 171,
883
+ "Ô": 172,
884
+ "Õ": 173,
885
+ "Ö": 174,
886
+ "×": 175,
887
+ "Ø": 176,
888
+ "Ù": 177,
889
+ "Ú": 178,
890
+ "Û": 179,
891
+ "Ü": 180,
892
+ "Ý": 181,
893
+ "Þ": 182,
894
+ "ß": 183,
895
+ "á": 184,
896
+ "â": 185,
897
+ "ä": 186,
898
+ "å": 187,
899
+ "æ": 188,
900
+ "ç": 189,
901
+ "è": 190,
902
+ "ê": 191,
903
+ "ë": 192,
904
+ "ì": 193,
905
+ "î": 194,
906
+ "ï": 195,
907
+ "ð": 196,
908
+ "ñ": 197,
909
+ "ô": 198,
910
+ "ö": 199,
911
+ "÷": 200,
912
+ "ø": 201,
913
+ "ù": 202,
914
+ "ú": 203,
915
+ "û": 204,
916
+ "ü": 205,
917
+ "ý": 206,
918
+ "þ": 207,
919
+ "ÿ": 208,
920
+ "Ā": 209,
921
+ "ā": 210,
922
+ "Ă": 211,
923
+ "ă": 212,
924
+ "Ą": 213,
925
+ "ą": 214,
926
+ "Ć": 215,
927
+ "ć": 216,
928
+ "Ĉ": 217,
929
+ "ĉ": 218,
930
+ "Ċ": 219,
931
+ "ċ": 220,
932
+ "Č": 221,
933
+ "č": 222,
934
+ "Ď": 223,
935
+ "ď": 224,
936
+ "đ": 225,
937
+ "Ē": 226,
938
+ "ē": 227,
939
+ "Ĕ": 228,
940
+ "ĕ": 229,
941
+ "Ė": 230,
942
+ "ė": 231,
943
+ "Ę": 232,
944
+ "ę": 233,
945
+ "Ě": 234,
946
+ "ě": 235,
947
+ "Ĝ": 236,
948
+ "ĝ": 237,
949
+ "Ğ": 238,
950
+ "ğ": 239,
951
+ "Ġ": 240,
952
+ "ġ": 241,
953
+ "Ģ": 242,
954
+ "ģ": 243,
955
+ "Ĥ": 244,
956
+ "ĥ": 245,
957
+ "Ħ": 246,
958
+ "ħ": 247,
959
+ "Ĩ": 248,
960
+ "Ī": 249,
961
+ "ī": 250,
962
+ "Ĭ": 251,
963
+ "ĭ": 252,
964
+ "Į": 253,
965
+ "į": 254,
966
+ "İ": 255,
967
+ "ı": 256,
968
+ "IJ": 257,
969
+ "ij": 258,
970
+ "Ĵ": 259,
971
+ "ĵ": 260,
972
+ "Ķ": 261,
973
+ "ķ": 262,
974
+ "ĸ": 263,
975
+ "Ĺ": 264,
976
+ "ĺ": 265,
977
+ "Ļ": 266,
978
+ "ļ": 267,
979
+ "Ľ": 268,
980
+ "ľ": 269,
981
+ "Ŀ": 270,
982
+ "ŀ": 271,
983
+ "Ł": 272,
984
+ "ł": 273,
985
+ "Ń": 274
986
+ },
987
+ "merges": []
988
+ }
989
+ }
tokenizer_config.json ADDED
@@ -0,0 +1,724 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "add_prefix_space": false,
3
+ "added_tokens_decoder": {
4
+ "0": {
5
+ "content": "<|endoftext|>",
6
+ "lstrip": false,
7
+ "normalized": true,
8
+ "rstrip": false,
9
+ "single_word": false,
10
+ "special": true
11
+ },
12
+ "1": {
13
+ "content": "a",
14
+ "lstrip": false,
15
+ "normalized": false,
16
+ "rstrip": false,
17
+ "single_word": false,
18
+ "special": true
19
+ },
20
+ "2": {
21
+ "content": "E",
22
+ "lstrip": false,
23
+ "normalized": false,
24
+ "rstrip": false,
25
+ "single_word": false,
26
+ "special": true
27
+ },
28
+ "3": {
29
+ "content": "e",
30
+ "lstrip": false,
31
+ "normalized": false,
32
+ "rstrip": false,
33
+ "single_word": false,
34
+ "special": true
35
+ },
36
+ "4": {
37
+ "content": "Ɛ",
38
+ "lstrip": false,
39
+ "normalized": false,
40
+ "rstrip": false,
41
+ "single_word": false,
42
+ "special": true
43
+ },
44
+ "5": {
45
+ "content": "ɛ",
46
+ "lstrip": false,
47
+ "normalized": false,
48
+ "rstrip": false,
49
+ "single_word": false,
50
+ "special": true
51
+ },
52
+ "6": {
53
+ "content": "Ə",
54
+ "lstrip": false,
55
+ "normalized": false,
56
+ "rstrip": false,
57
+ "single_word": false,
58
+ "special": true
59
+ },
60
+ "7": {
61
+ "content": "ə",
62
+ "lstrip": false,
63
+ "normalized": false,
64
+ "rstrip": false,
65
+ "single_word": false,
66
+ "special": true
67
+ },
68
+ "8": {
69
+ "content": "I",
70
+ "lstrip": false,
71
+ "normalized": false,
72
+ "rstrip": false,
73
+ "single_word": false,
74
+ "special": true
75
+ },
76
+ "9": {
77
+ "content": "i",
78
+ "lstrip": false,
79
+ "normalized": false,
80
+ "rstrip": false,
81
+ "single_word": false,
82
+ "special": true
83
+ },
84
+ "10": {
85
+ "content": "O",
86
+ "lstrip": false,
87
+ "normalized": false,
88
+ "rstrip": false,
89
+ "single_word": false,
90
+ "special": true
91
+ },
92
+ "11": {
93
+ "content": "o",
94
+ "lstrip": false,
95
+ "normalized": false,
96
+ "rstrip": false,
97
+ "single_word": false,
98
+ "special": true
99
+ },
100
+ "12": {
101
+ "content": "Ɔ",
102
+ "lstrip": false,
103
+ "normalized": false,
104
+ "rstrip": false,
105
+ "single_word": false,
106
+ "special": true
107
+ },
108
+ "13": {
109
+ "content": "ɔ",
110
+ "lstrip": false,
111
+ "normalized": false,
112
+ "rstrip": false,
113
+ "single_word": false,
114
+ "special": true
115
+ },
116
+ "14": {
117
+ "content": "U",
118
+ "lstrip": false,
119
+ "normalized": false,
120
+ "rstrip": false,
121
+ "single_word": false,
122
+ "special": true
123
+ },
124
+ "15": {
125
+ "content": "u",
126
+ "lstrip": false,
127
+ "normalized": false,
128
+ "rstrip": false,
129
+ "single_word": false,
130
+ "special": true
131
+ },
132
+ "16": {
133
+ "content": "ã",
134
+ "lstrip": false,
135
+ "normalized": false,
136
+ "rstrip": false,
137
+ "single_word": false,
138
+ "special": true
139
+ },
140
+ "17": {
141
+ "content": "ẽ",
142
+ "lstrip": false,
143
+ "normalized": false,
144
+ "rstrip": false,
145
+ "single_word": false,
146
+ "special": true
147
+ },
148
+ "18": {
149
+ "content": "ĩ",
150
+ "lstrip": false,
151
+ "normalized": false,
152
+ "rstrip": false,
153
+ "single_word": false,
154
+ "special": true
155
+ },
156
+ "19": {
157
+ "content": "õ",
158
+ "lstrip": false,
159
+ "normalized": false,
160
+ "rstrip": false,
161
+ "single_word": false,
162
+ "special": true
163
+ },
164
+ "20": {
165
+ "content": "ũ",
166
+ "lstrip": false,
167
+ "normalized": false,
168
+ "rstrip": false,
169
+ "single_word": false,
170
+ "special": true
171
+ },
172
+ "21": {
173
+ "content": "B",
174
+ "lstrip": false,
175
+ "normalized": false,
176
+ "rstrip": false,
177
+ "single_word": false,
178
+ "special": true
179
+ },
180
+ "22": {
181
+ "content": "b",
182
+ "lstrip": false,
183
+ "normalized": false,
184
+ "rstrip": false,
185
+ "single_word": false,
186
+ "special": true
187
+ },
188
+ "23": {
189
+ "content": "D",
190
+ "lstrip": false,
191
+ "normalized": false,
192
+ "rstrip": false,
193
+ "single_word": false,
194
+ "special": true
195
+ },
196
+ "24": {
197
+ "content": "d",
198
+ "lstrip": false,
199
+ "normalized": false,
200
+ "rstrip": false,
201
+ "single_word": false,
202
+ "special": true
203
+ },
204
+ "25": {
205
+ "content": "Đ",
206
+ "lstrip": false,
207
+ "normalized": false,
208
+ "rstrip": false,
209
+ "single_word": false,
210
+ "special": true
211
+ },
212
+ "26": {
213
+ "content": "ɖ",
214
+ "lstrip": false,
215
+ "normalized": false,
216
+ "rstrip": false,
217
+ "single_word": false,
218
+ "special": true
219
+ },
220
+ "27": {
221
+ "content": "F",
222
+ "lstrip": false,
223
+ "normalized": false,
224
+ "rstrip": false,
225
+ "single_word": false,
226
+ "special": true
227
+ },
228
+ "28": {
229
+ "content": "f",
230
+ "lstrip": false,
231
+ "normalized": false,
232
+ "rstrip": false,
233
+ "single_word": false,
234
+ "special": true
235
+ },
236
+ "29": {
237
+ "content": "Ƒ",
238
+ "lstrip": false,
239
+ "normalized": false,
240
+ "rstrip": false,
241
+ "single_word": false,
242
+ "special": true
243
+ },
244
+ "30": {
245
+ "content": "ƒ",
246
+ "lstrip": false,
247
+ "normalized": false,
248
+ "rstrip": false,
249
+ "single_word": false,
250
+ "special": true
251
+ },
252
+ "31": {
253
+ "content": "G",
254
+ "lstrip": false,
255
+ "normalized": false,
256
+ "rstrip": false,
257
+ "single_word": false,
258
+ "special": true
259
+ },
260
+ "32": {
261
+ "content": "g",
262
+ "lstrip": false,
263
+ "normalized": false,
264
+ "rstrip": false,
265
+ "single_word": false,
266
+ "special": true
267
+ },
268
+ "33": {
269
+ "content": "Ɣ",
270
+ "lstrip": false,
271
+ "normalized": false,
272
+ "rstrip": false,
273
+ "single_word": false,
274
+ "special": true
275
+ },
276
+ "34": {
277
+ "content": "ɣ",
278
+ "lstrip": false,
279
+ "normalized": false,
280
+ "rstrip": false,
281
+ "single_word": false,
282
+ "special": true
283
+ },
284
+ "35": {
285
+ "content": "H",
286
+ "lstrip": false,
287
+ "normalized": false,
288
+ "rstrip": false,
289
+ "single_word": false,
290
+ "special": true
291
+ },
292
+ "36": {
293
+ "content": "h",
294
+ "lstrip": false,
295
+ "normalized": false,
296
+ "rstrip": false,
297
+ "single_word": false,
298
+ "special": true
299
+ },
300
+ "37": {
301
+ "content": "K",
302
+ "lstrip": false,
303
+ "normalized": false,
304
+ "rstrip": false,
305
+ "single_word": false,
306
+ "special": true
307
+ },
308
+ "38": {
309
+ "content": "k",
310
+ "lstrip": false,
311
+ "normalized": false,
312
+ "rstrip": false,
313
+ "single_word": false,
314
+ "special": true
315
+ },
316
+ "39": {
317
+ "content": "L",
318
+ "lstrip": false,
319
+ "normalized": false,
320
+ "rstrip": false,
321
+ "single_word": false,
322
+ "special": true
323
+ },
324
+ "40": {
325
+ "content": "l",
326
+ "lstrip": false,
327
+ "normalized": false,
328
+ "rstrip": false,
329
+ "single_word": false,
330
+ "special": true
331
+ },
332
+ "41": {
333
+ "content": "M",
334
+ "lstrip": false,
335
+ "normalized": false,
336
+ "rstrip": false,
337
+ "single_word": false,
338
+ "special": true
339
+ },
340
+ "42": {
341
+ "content": "m",
342
+ "lstrip": false,
343
+ "normalized": false,
344
+ "rstrip": false,
345
+ "single_word": false,
346
+ "special": true
347
+ },
348
+ "43": {
349
+ "content": "N",
350
+ "lstrip": false,
351
+ "normalized": false,
352
+ "rstrip": false,
353
+ "single_word": false,
354
+ "special": true
355
+ },
356
+ "44": {
357
+ "content": "n",
358
+ "lstrip": false,
359
+ "normalized": false,
360
+ "rstrip": false,
361
+ "single_word": false,
362
+ "special": true
363
+ },
364
+ "45": {
365
+ "content": "Ŋ",
366
+ "lstrip": false,
367
+ "normalized": false,
368
+ "rstrip": false,
369
+ "single_word": false,
370
+ "special": true
371
+ },
372
+ "46": {
373
+ "content": "ŋ",
374
+ "lstrip": false,
375
+ "normalized": false,
376
+ "rstrip": false,
377
+ "single_word": false,
378
+ "special": true
379
+ },
380
+ "47": {
381
+ "content": "P",
382
+ "lstrip": false,
383
+ "normalized": false,
384
+ "rstrip": false,
385
+ "single_word": false,
386
+ "special": true
387
+ },
388
+ "48": {
389
+ "content": "p",
390
+ "lstrip": false,
391
+ "normalized": false,
392
+ "rstrip": false,
393
+ "single_word": false,
394
+ "special": true
395
+ },
396
+ "49": {
397
+ "content": "R",
398
+ "lstrip": false,
399
+ "normalized": false,
400
+ "rstrip": false,
401
+ "single_word": false,
402
+ "special": true
403
+ },
404
+ "50": {
405
+ "content": "r",
406
+ "lstrip": false,
407
+ "normalized": false,
408
+ "rstrip": false,
409
+ "single_word": false,
410
+ "special": true
411
+ },
412
+ "51": {
413
+ "content": "S",
414
+ "lstrip": false,
415
+ "normalized": false,
416
+ "rstrip": false,
417
+ "single_word": false,
418
+ "special": true
419
+ },
420
+ "52": {
421
+ "content": "s",
422
+ "lstrip": false,
423
+ "normalized": false,
424
+ "rstrip": false,
425
+ "single_word": false,
426
+ "special": true
427
+ },
428
+ "53": {
429
+ "content": "T",
430
+ "lstrip": false,
431
+ "normalized": false,
432
+ "rstrip": false,
433
+ "single_word": false,
434
+ "special": true
435
+ },
436
+ "54": {
437
+ "content": "t",
438
+ "lstrip": false,
439
+ "normalized": false,
440
+ "rstrip": false,
441
+ "single_word": false,
442
+ "special": true
443
+ },
444
+ "55": {
445
+ "content": "V",
446
+ "lstrip": false,
447
+ "normalized": false,
448
+ "rstrip": false,
449
+ "single_word": false,
450
+ "special": true
451
+ },
452
+ "56": {
453
+ "content": "v",
454
+ "lstrip": false,
455
+ "normalized": false,
456
+ "rstrip": false,
457
+ "single_word": false,
458
+ "special": true
459
+ },
460
+ "57": {
461
+ "content": "Ʋ",
462
+ "lstrip": false,
463
+ "normalized": false,
464
+ "rstrip": false,
465
+ "single_word": false,
466
+ "special": true
467
+ },
468
+ "58": {
469
+ "content": "ʋ",
470
+ "lstrip": false,
471
+ "normalized": false,
472
+ "rstrip": false,
473
+ "single_word": false,
474
+ "special": true
475
+ },
476
+ "59": {
477
+ "content": "W",
478
+ "lstrip": false,
479
+ "normalized": false,
480
+ "rstrip": false,
481
+ "single_word": false,
482
+ "special": true
483
+ },
484
+ "60": {
485
+ "content": "w",
486
+ "lstrip": false,
487
+ "normalized": false,
488
+ "rstrip": false,
489
+ "single_word": false,
490
+ "special": true
491
+ },
492
+ "61": {
493
+ "content": "X",
494
+ "lstrip": false,
495
+ "normalized": false,
496
+ "rstrip": false,
497
+ "single_word": false,
498
+ "special": true
499
+ },
500
+ "62": {
501
+ "content": "x",
502
+ "lstrip": false,
503
+ "normalized": false,
504
+ "rstrip": false,
505
+ "single_word": false,
506
+ "special": true
507
+ },
508
+ "63": {
509
+ "content": "Y",
510
+ "lstrip": false,
511
+ "normalized": false,
512
+ "rstrip": false,
513
+ "single_word": false,
514
+ "special": true
515
+ },
516
+ "64": {
517
+ "content": "y",
518
+ "lstrip": false,
519
+ "normalized": false,
520
+ "rstrip": false,
521
+ "single_word": false,
522
+ "special": true
523
+ },
524
+ "65": {
525
+ "content": "Z",
526
+ "lstrip": false,
527
+ "normalized": false,
528
+ "rstrip": false,
529
+ "single_word": false,
530
+ "special": true
531
+ },
532
+ "66": {
533
+ "content": "z",
534
+ "lstrip": false,
535
+ "normalized": false,
536
+ "rstrip": false,
537
+ "single_word": false,
538
+ "special": true
539
+ },
540
+ "67": {
541
+ "content": "̃",
542
+ "lstrip": false,
543
+ "normalized": false,
544
+ "rstrip": false,
545
+ "single_word": false,
546
+ "special": true
547
+ },
548
+ "68": {
549
+ "content": "ó",
550
+ "lstrip": false,
551
+ "normalized": false,
552
+ "rstrip": false,
553
+ "single_word": false,
554
+ "special": true
555
+ },
556
+ "69": {
557
+ "content": "À",
558
+ "lstrip": false,
559
+ "normalized": false,
560
+ "rstrip": false,
561
+ "single_word": false,
562
+ "special": true
563
+ },
564
+ "70": {
565
+ "content": "é",
566
+ "lstrip": false,
567
+ "normalized": false,
568
+ "rstrip": false,
569
+ "single_word": false,
570
+ "special": true
571
+ },
572
+ "71": {
573
+ "content": "È",
574
+ "lstrip": false,
575
+ "normalized": false,
576
+ "rstrip": false,
577
+ "single_word": false,
578
+ "special": true
579
+ },
580
+ "72": {
581
+ "content": "ò",
582
+ "lstrip": false,
583
+ "normalized": false,
584
+ "rstrip": false,
585
+ "single_word": false,
586
+ "special": true
587
+ },
588
+ "73": {
589
+ "content": "à",
590
+ "lstrip": false,
591
+ "normalized": false,
592
+ "rstrip": false,
593
+ "single_word": false,
594
+ "special": true
595
+ },
596
+ "74": {
597
+ "content": "í",
598
+ "lstrip": false,
599
+ "normalized": false,
600
+ "rstrip": false,
601
+ "single_word": false,
602
+ "special": true
603
+ }
604
+ },
605
+ "additional_special_tokens": [
606
+ "a",
607
+ "E",
608
+ "e",
609
+ "Ɛ",
610
+ "ɛ",
611
+ "Ə",
612
+ "ə",
613
+ "I",
614
+ "i",
615
+ "O",
616
+ "o",
617
+ "Ɔ",
618
+ "ɔ",
619
+ "U",
620
+ "u",
621
+ "ã",
622
+ "ẽ",
623
+ "ĩ",
624
+ "õ",
625
+ "ũ",
626
+ "B",
627
+ "b",
628
+ "D",
629
+ "d",
630
+ "Đ",
631
+ "ɖ",
632
+ "F",
633
+ "f",
634
+ "Ƒ",
635
+ "ƒ",
636
+ "G",
637
+ "g",
638
+ "Ɣ",
639
+ "ɣ",
640
+ "H",
641
+ "h",
642
+ "K",
643
+ "k",
644
+ "L",
645
+ "l",
646
+ "M",
647
+ "m",
648
+ "N",
649
+ "n",
650
+ "Ŋ",
651
+ "ŋ",
652
+ "P",
653
+ "p",
654
+ "R",
655
+ "r",
656
+ "S",
657
+ "s",
658
+ "T",
659
+ "t",
660
+ "V",
661
+ "v",
662
+ "Ʋ",
663
+ "ʋ",
664
+ "W",
665
+ "w",
666
+ "X",
667
+ "x",
668
+ "Y",
669
+ "y",
670
+ "Z",
671
+ "z",
672
+ "D",
673
+ "z",
674
+ "d",
675
+ "z",
676
+ "G",
677
+ "b",
678
+ "g",
679
+ "b",
680
+ "K",
681
+ "p",
682
+ "k",
683
+ "p",
684
+ "N",
685
+ "y",
686
+ "n",
687
+ "y",
688
+ "T",
689
+ "s",
690
+ "i",
691
+ "t",
692
+ "s",
693
+ "i",
694
+ "D",
695
+ "y",
696
+ "d",
697
+ "y",
698
+ "D",
699
+ "z",
700
+ "i",
701
+ "d",
702
+ "z",
703
+ "i",
704
+ "Ŋ",
705
+ "w",
706
+ "ŋ",
707
+ "w",
708
+ "ɔ",
709
+ "̃",
710
+ "ó",
711
+ "À",
712
+ "é",
713
+ "È",
714
+ "ò",
715
+ "à",
716
+ "í"
717
+ ],
718
+ "bos_token": "<|endoftext|>",
719
+ "clean_up_tokenization_spaces": true,
720
+ "eos_token": "<|endoftext|>",
721
+ "model_max_length": 1024,
722
+ "tokenizer_class": "GPT2Tokenizer",
723
+ "unk_token": "<|endoftext|>"
724
+ }
vocab.json ADDED
@@ -0,0 +1 @@
 
 
1
+ {"<|endoftext|>":0,"a":1,"E":2,"e":3,"Ɛ":4,"ɛ":5,"Ə":6,"ə":7,"I":8,"i":9,"O":10,"o":11,"Ɔ":12,"ɔ":13,"U":14,"u":15,"ã":16,"ẽ":17,"ĩ":18,"õ":19,"ũ":20,"B":21,"b":22,"D":23,"d":24,"Đ":25,"ɖ":26,"F":27,"f":28,"Ƒ":29,"ƒ":30,"G":31,"g":32,"Ɣ":33,"ɣ":34,"H":35,"h":36,"K":37,"k":38,"L":39,"l":40,"M":41,"m":42,"N":43,"n":44,"Ŋ":45,"ŋ":46,"P":47,"p":48,"R":49,"r":50,"S":51,"s":52,"T":53,"t":54,"V":55,"v":56,"Ʋ":57,"ʋ":58,"W":59,"w":60,"X":61,"x":62,"Y":63,"y":64,"Z":65,"z":66,"̃":67,"ó":68,"À":69,"é":70,"È":71,"ò":72,"à":73,"í":74,"!":75,"\"":76,"#":77,"$":78,"%":79,"&":80,"'":81,"(":82,")":83,"*":84,"+":85,",":86,"-":87,".":88,"/":89,"0":90,"1":91,"2":92,"3":93,"4":94,"5":95,"6":96,"7":97,"8":98,"9":99,":":100,";":101,"<":102,"=":103,">":104,"?":105,"@":106,"A":107,"C":108,"J":109,"Q":110,"[":111,"\\":112,"]":113,"^":114,"_":115,"`":116,"c":117,"j":118,"q":119,"{":120,"|":121,"}":122,"~":123,"¡":124,"¢":125,"£":126,"¤":127,"¥":128,"¦":129,"§":130,"¨":131,"©":132,"ª":133,"«":134,"¬":135,"®":136,"¯":137,"°":138,"±":139,"²":140,"³":141,"´":142,"µ":143,"¶":144,"·":145,"¸":146,"¹":147,"º":148,"»":149,"¼":150,"½":151,"¾":152,"¿":153,"Á":154,"Â":155,"Ã":156,"Ä":157,"Å":158,"Æ":159,"Ç":160,"É":161,"Ê":162,"Ë":163,"Ì":164,"Í":165,"Î":166,"Ï":167,"Ð":168,"Ñ":169,"Ò":170,"Ó":171,"Ô":172,"Õ":173,"Ö":174,"×":175,"Ø":176,"Ù":177,"Ú":178,"Û":179,"Ü":180,"Ý":181,"Þ":182,"ß":183,"á":184,"â":185,"ä":186,"å":187,"æ":188,"ç":189,"è":190,"ê":191,"ë":192,"ì":193,"î":194,"ï":195,"ð":196,"ñ":197,"ô":198,"ö":199,"÷":200,"ø":201,"ù":202,"ú":203,"û":204,"ü":205,"ý":206,"þ":207,"ÿ":208,"Ā":209,"ā":210,"Ă":211,"ă":212,"Ą":213,"ą":214,"Ć":215,"ć":216,"Ĉ":217,"ĉ":218,"Ċ":219,"ċ":220,"Č":221,"č":222,"Ď":223,"ď":224,"đ":225,"Ē":226,"ē":227,"Ĕ":228,"ĕ":229,"Ė":230,"ė":231,"Ę":232,"ę":233,"Ě":234,"ě":235,"Ĝ":236,"ĝ":237,"Ğ":238,"ğ":239,"Ġ":240,"ġ":241,"Ģ":242,"ģ":243,"Ĥ":244,"ĥ":245,"Ħ":246,"ħ":247,"Ĩ":248,"Ī":249,"ī":250,"Ĭ":251,"ĭ":252,"Į":253,"į":254,"İ":255,"ı":256,"IJ":257,"ij":258,"Ĵ":259,"ĵ":260,"Ķ":261,"ķ":262,"ĸ":263,"Ĺ":264,"ĺ":265,"Ļ":266,"ļ":267,"Ľ":268,"ľ":269,"Ŀ":270,"ŀ":271,"Ł":272,"ł":273,"Ń":274}