The immense evolution in Large Language Models (LLMs)  has underscored the importance of massive, heterogeneous, and high-quality data.
A data recipe is a mixture of data of different types and from different sources for training an LLM, which has been known as one of the most important factors that decide the LLM’s performance.
Existing open-source tools for LLM data processing are mostly tailored for preparing specific data recipes.
To continuously uncover the potential of LLMs, incorporate (after cleaning)  data from new sources, and improve LLMs’ general-purpose or domain-specific performance, we build a data processing system, namedData-Juicer, with which we can efficiently generate diverse data recipes, explore different possibilities in forming the data mixtures, and evaluate their effects on the model performance.
Different from traditional data-analytics pipelines,Data-Juicerfaces some unique challenges.
Firstly, the possible data sources for forming data recipes are truly heterogeneous and massive with various qualities (e.g., considering all web-pages on the Internet) .
Secondly, it is extremely expensive to precisely evaluate data recipes’ impact on the LLMs’ performance.
Thirdly, sufficient flexibility needs to be provided to the end users ofData-Juicer, model developers, to configure and evaluate different data recipes. Data-Juicerfeatures a fine-grained abstraction of the pipeline for constructing data recipes, with over 50 built-in operators that can be freely composed and extended. By incorporating visualization and auto-evaluation capabilities,Data-Juicerenables a timely feedback loop after data processing for both LLM pre-training and fine-tuning.
Further,Data-Juiceris optimized and integrated with ecosystems for LLM training, evaluation, and distributed computing.
With the help ofData-Juicer, we derive data recipes that achieve remarkable performance boosts on state-of-the-art LLMs, demonstrating up toincrease in averaged score acrossLLM benchmarks and% higher win rate in pair-wise GPT-4 evaluations. More importantly, we hope thatData-Juicerpromotes broader data-centric research on training and understanding LLMs.Data-Juicerand our data recipes are released and actively maintained at https://github.com/alibaba/data-juicer.