**How Pre-Training Affects ICL?** ICL is first proposed in GPT-3 [55], and it has been shown that the ICL ability becomes more significant with a larger model size. Further, some studies reveal that small-scale PLMs can also demonstrate a strong ICL ability by continual pre-training [490] or fine-tuning [491] on specially designed training tasks, which typically involve additional task examples in the input during the training process. It suggests that the design of training tasks is an important influence factor on the ICL capability of LLMs. Besides training tasks, recent studies have also investigated the relationship between ICL and pre-training corpora [488, 492]. For example, ICL can be theoretically explained as the product of pre-training on documents that exhibit long-range coherence [488]. Further, another study [492] theoretically analyzes that when scaling parameters and data, LLMs based on next-word prediction can emerge the ability of ICL by learning from the compositional structure (_e.g._, how words and phrases are combined to form larger linguistic units like sentences) present in language data.