DrishtiSharma's picture
add tokenizer
7910fa3
raw
history blame
447 Bytes
{"(": 1, ")": 2, "/": 3, "a": 4, "b": 5, "c": 6, "d": 7, "e": 8, "f": 9, "g": 10, "h": 11, "i": 12, "j": 13, "k": 14, "l": 15, "m": 16, "n": 17, "o": 18, "p": 19, "q": 20, "r": 21, "s": 22, "t": 23, "u": 24, "v": 25, "w": 26, "x": 27, "y": 28, "z": 29, "«": 30, "»": 31, "½": 32, "à": 33, "â": 34, "ç": 35, "è": 36, "é": 37, "ê": 38, "ñ": 39, "ô": 40, "ù": 41, "û": 42, "ü": 43, "ʼ": 44, "…": 45, "|": 0, "[UNK]": 46, "[PAD]": 47}