**Tokenization.** Tokenization is also a crucial step for data preprocessing. It aims to segment raw text into sequences of individual tokens, which are subsequently used as the inputs of LLMs. In traditional NLP research (_e.g.,_ sequence labeling with conditional random fields [220]), word-based tokenization is the predominant approach, which is more aligned with human's language cognition. However, word-based tokenization can yield different segmentation results for the same input in some languages (_e.g.,_ Chinese word segmentation), generate a huge word vocabulary containing many low-frequency words, and also suffer from the _"out-of-vocabulary"_ issue. Thus, several neural network models employ _character_ as the minimum unit to derive the word representation (_e.g.,_ a CNN word encoder in ELMo [21]). Recently, _subword tokenizers_ have been widely used in Transformer based language models, typically including Byte-Pair Encoding tokenization, WordPiece tokenization and Unigram tokenization. HuggingFace has maintained an excellent online NLP course on tokenizer22 with running examples, and we refer to the beginners to this course. Next, we briefly describe the three representative tokenization methods.