infinitejoy's picture
add tokenizer
7f599b5
raw
history blame
406 Bytes
{"а": 1, "б": 2, "в": 3, "г": 4, "д": 5, "е": 6, "ж": 7, "з": 8, "и": 9, "й": 10, "к": 11, "л": 12, "м": 13, "н": 14, "о": 15, "п": 16, "р": 17, "с": 18, "т": 19, "у": 20, "ф": 21, "х": 22, "ц": 23, "ч": 24, "ш": 25, "щ": 26, "ъ": 27, "ы": 28, "ь": 29, "э": 30, "ю": 31, "я": 32, "ё": 33, "ң": 34, "ү": 35, "ӊ": 36, "ө": 37, "⅛": 38, "|": 0, "[UNK]": 39, "[PAD]": 40}