LLMs have shown impressive capabilities in many downstream tasks in a zero/few-shot setting such as summarization, reading comprehension, translation, and commonsense reasoning Brown et al. (2020); Touvron et al. (2023). To train a LLM, the currently established approach is to employ a tokenizer that splits the training documents into tokens where a token represents a word Bengio et al. (2000), a sub-word Schuster and Nakajima (2012); Sennrich et al. (2015); Wang et al. (2020), or a single character Gao et al. (2020), and each token is represented in the model by an embedding vector that can be further processed.