Upload 4 files

README.md

@@ -1,3 +1,9 @@
----
-license: mit
----
+This is a fork of the GPT NeoX 20B tokenizer, edited to split every numerical digit into a separate token. This has the goal of making it easier for the model to learn arithmetic capabilities and to hopefully be more interpretable, and copies the idea from the [PaLM tokenizer](https://ai.googleblog.com/2022/04/pathways-language-model-palm-scaling-to.html). 
+
+This was done, extremely hackily, by just removing every token that contained "\d\d" (eg "2013"). All remaining digit containing tokens are "0" ... "9" and " 0" ... " 9"
+
+This comes at the cost of making modelling normal text harder, since eg dates like 2013 which naturally *should* be a single token are now 2|0|1|3.
+
+This has a reduced vocab size of 48252 (several of the tokens towards the end are special whitespace tokens copied in from GPT-NeoX to make tokenizing code easier - some of these are duplicated in the vocabulary and thus may not actually show up at train time).
+
+It includes a padding token (<|PAD|>) an End-Of-String token (<|EOS|>) and a Beginning-Of-String token (<|BOS|>)

special_tokens_map.json

@@ -0,0 +1,5 @@
+{
+  "bos_token": "<|BOS|>",
+  "eos_token": "<|EOS|>",
+  "pad_token": "<|PAD|>"
+}

tokenizer_config.json

@@ -0,0 +1,3 @@
+{
+  "tokenizer_class": "GPTNeoXTokenizer"
+}