Update README.md

README.md

@@ -18,9 +18,20 @@ tags:
 
 # KStack-full models
 
-KStack-full models is a collection of fine-tuned open-source generative text models fine-tuned on KStack dataset. 
+KStack-full models is a collection of fine-tuned open-source generative text models fine-tuned on KStack dataset with rule-based filtering. 
 This is a repository for fine-tuned CodeLlama-7b model in the Hugging Face Transformers format.
 
+## Rule-based filtering
+To increase the quality of the dataset and filter out statistical outliers such as homework assignments, we filter out the dataset entries according to the following rules:
+* We filter out files which belong to the low-popular repos (the sum of stars and forks is less than 6)
+* Next, we filter out files which belong to the repos with less than 5 Kotlin files
+* Finally, we remove files which have less than 20 SLOC
+
+We clean the content of the remaining dataset entries according to the following rules:
+* We remove all non-ASCII entries
+* We remove all package lines such as _package kotlinx.coroutines.channels_
+* We remove half of the import lines.
+
 # Model use
 
 ```python