weblab-GENIAC
/

Tanuki-8x8B-dpo-v1.0

Text Generation

Model card Files Files and versions Community

misdelivery commited on Sep 2

Commit

f058f6f

•

1 Parent(s): 16db186

Upload tokenizer.json

Files changed (1) hide show

tokenizer.json +3 -0

tokenizer.json CHANGED Viewed

@@ -124,6 +124,9 @@
   "normalizer": {
     "type": "Sequence",
     "normalizers": [
       {
         "type": "Replace",
         "pattern": {

   "normalizer": {
     "type": "Sequence",
     "normalizers": [
+      {
+        "type": "NFKC"
+      },
       {
         "type": "Replace",
         "pattern": {