SreyanG-NVIDIA's picture
add tokenizer
199c1cd
raw
history blame contribute delete
268 Bytes
{"O": 0, "J": 1, "'": 2, "P": 3, "F": 4, "V": 5, "U": 6, "R": 7, "C": 9, "W": 10, "A": 11, "I": 12, "Q": 13, "D": 14, "K": 15, "M": 16, "X": 17, "T": 18, "N": 19, "H": 20, "B": 21, "E": 22, "Y": 23, "Z": 24, "L": 25, "G": 26, "S": 27, "|": 8, "[UNK]": 28, "[PAD]": 29}