fnlp
/

txsun commited on
Commit
9db9cef
1 Parent(s): f1ab6b1

Upload added_tokens.json with huggingface_hub

Browse files
Files changed (1) hide show
  1. added_tokens.json +45 -0
added_tokens.json ADDED
@@ -0,0 +1,45 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "\t\t": 106066,
3
+ "\t\t\t": 106065,
4
+ "\t\t\t\t": 106064,
5
+ "\t\t\t\t\t": 106063,
6
+ "\t\t\t\t\t\t": 106062,
7
+ "\t\t\t\t\t\t\t": 106061,
8
+ "\t\t\t\t\t\t\t\t": 106060,
9
+ "\t\t\t\t\t\t\t\t\t": 106059,
10
+ " ": 106058,
11
+ " ": 106057,
12
+ " ": 106056,
13
+ " ": 106055,
14
+ " ": 106054,
15
+ " ": 106053,
16
+ " ": 106052,
17
+ " ": 106051,
18
+ " ": 106050,
19
+ " ": 106049,
20
+ " ": 106048,
21
+ " ": 106047,
22
+ " ": 106046,
23
+ " ": 106045,
24
+ " ": 106044,
25
+ " ": 106043,
26
+ " ": 106042,
27
+ " ": 106041,
28
+ " ": 106040,
29
+ " ": 106039,
30
+ " ": 106038,
31
+ " ": 106037,
32
+ " ": 106036,
33
+ " ": 106035,
34
+ " ": 106034,
35
+ " ": 106033,
36
+ " ": 106032,
37
+ " ": 106031,
38
+ " ": 106030,
39
+ " ": 106029,
40
+ "<eoc>": 106070,
41
+ "<eoh>": 106067,
42
+ "<eom>": 106068,
43
+ "<eor>": 106071,
44
+ "<eot>": 106069
45
+ }