USER_TOKEN and START_OF_TURN_TOKEN marked not-special?

#2
by bartowski - opened

Any reason for this? I think it tends to break tokenization (especially in llama.cpp) when these are set to special: False, but also odd that most others are properly set to special: True?

Mild false alarm on breaking tokenization, seems fine in llama.cpp even with non-special

still curious if you have info on why these are specifically marked not special but the others are

@bartowski do these models quantise fine using the existing llama.cpp code?

yup! no issues with b3634

Sign up or log in to comment