While fine-tuning LMs with diverse NLP tasks has shown promising results, it often falls short in aligning with the intentions of real-world users. OpenAI has pursued a different approach by soliciting human annotators to provide a large corpus of human instructions, encompassing diverse forms and a wide range of task types. Building upon this dataset, OpenAI trained its GPT3 [1] model to create InstructGPT [10], which better aligns with users' inputs. This line of development has even led to the impressive work known as ChatGPT. However, it is important to note that the dataset and model weights associated with these advancements are not publicly available. Alpaca [26] takes a different route by adopting the self-instruct method [27], leveraging ChatGPT to generate data for training. Vicuna [28] utilizes user-shared conversations collected from ShareGPT.com to train its models. WizardLM [29] introduces the _Evol-Instruct_ method, which involves evolving existing instruction data to generate more complex and diverse datasets. In contrast to these general instruction fine-tuning approaches, our _WizardCoder_ successfully applies the _Evol-Instruct_ method specifically in the domain of Code LLMs.