Utilizing one of the most advanced open-source models, Llama-2 7B, our research investigates the effects on LLMs when they undergo finetuning8 with data generated by other LLMs. To ensure the generation of high-quality data and to provide a relevant but not trivial downstream task, we employ the Wikitext-103 dataset. We segment this dataset into chunks of 128 tokens, between each with a stride of 64 tokens, resulting in approximately 2.2 million chunks. Denote this dataset as \(\mathcal{D}_{0}\). The task for generation involves producing the final 32 tokens given the initial 96 tokens from each chunk in the original dataset. In the initial generation (0-th generation), we use the Llama-2 7B FT model, which has been finetuned on \(\mathcal{D}_{0}\), applying a generation loss that focuses solely on the cross-entropy loss of the final 32 tokens. We denote this initial model as \(\mathcal{M}_{0}\), which demonstrates enhanced capacity for the generation task compared to the standard Llama-2 7B model. By querying \(\mathcal{M}_{0}\) with the original 96 tokens from \(\mathcal{D}_{0}\), we generate the dataset \(\mathcal{D}_{1}\) and subsequently finetune Llama-2 7B on this dataset to obtain \(\mathcal{M}_{1}\). This process is sequentially repeated to generate \(\mathcal{D}_{i}\) from \(\mathcal{M}_{i-1}\) and obtain \(\mathcal{M}_{i}\) through finetuning. By comparing the performance of various \(\mathcal{M}\) models on the test set derived from Wikitext-103, also segmented into 128-token chunks, we aim to investigate the model collapse in LLMs.