**Stage 1 (new input embeddings):** Initially, our focus is narrow yet critical: to learn the input embeddings of the newly added tokens while freezing all other model parameters. This stage is foundational, allowing the model to adjust its recognition and processing of these tokens from the beginning. The pre-initialized embeddings serve as a starting point, guiding the model to better utilize these new tokens in its existing framework. Our principal hypothesis here is that if the input and output token sequences in causal language modeling can be differentiated, by utilizing both the old and new tokenizers at the same time, the model can more efficiently and effectively learn new vocabulary embeddings, as it could leverage its established knowledge in the embedding spaces from old tokens. However, employing distinct tokenizers for input and output sequences at once poses implementation challenges, such as the difficulty of applying teacher forcing due to mismatched input/output sequences. Here, the subword-based embedding initialization (Sec 2.2) provides a proxy for using the old tokenizer for output sequences, such that the model is tasked to generate the subword token (old) given the whole word token (new). In other words, the model could learn to align their representations for generating the new token with that for generating its first subword token, by optimizing only the input embeddings without any modification of input/output token sequences as described in Figure 2. However, at this stage, the model is not yet able to distinguish between tokens sharing the same hidden state.