Join the conversation

Join the community of Machine Learners and AI enthusiasts.

Sign Up
nroggendorff 
posted an update 18 days ago
Post
2212
I still think whitespace in tokenizers are so dumb.
Congrats, you just doubled your vocab size for no reason.

Any alternative ideas?🤔

·

merges.txt :spinnyhat: