Update README.md

README.md

@@ -10,15 +10,38 @@ tags:
 - yuvraj17/Llama-3-8B-spectrum-25
 - ruggsea/Llama3-stanford-encyclopedia-philosophy-QA
 - arcee-ai/Llama-3.1-SuperNova-Lite
+license: apache-2.0
+language:
+- en
+pipeline_tag: text-classification
 ---
 
 # Llama3-8B-SuperNova-Spectrum-dare_ties
 
-Llama3-8B-SuperNova-Spectrum-dare_ties is a merge of the following models using [LazyMergekit](https://colab.research.google.com/drive/1obulZ1ROXHjYLn6PPZJwRR6GzgQogxxb?usp=sharing):
+Llama3-8B-SuperNova-Spectrum-dare_ties is a `DARE_TIES` merge of the following models using [LazyMergekit](https://colab.research.google.com/drive/1obulZ1ROXHjYLn6PPZJwRR6GzgQogxxb?usp=sharing):
 * [yuvraj17/Llama-3-8B-spectrum-25](https://huggingface.co/yuvraj17/Llama-3-8B-spectrum-25)
 * [ruggsea/Llama3-stanford-encyclopedia-philosophy-QA](https://huggingface.co/ruggsea/Llama3-stanford-encyclopedia-philosophy-QA)
 * [arcee-ai/Llama-3.1-SuperNova-Lite](https://huggingface.co/arcee-ai/Llama-3.1-SuperNova-Lite)
 
+## DARE_TIES Merging
+
+### TIES Merging
+
+[TIES](https://arxiv.org/abs/2306.01708) Merging, introduced by Yadav et al. (2023), is a method for merging multiple specialized models into one general-purpose model. It solves two key challenges:
+* **Redundancy Removal**: Identifies and eliminates overlapping or unnecessary information between models, making the final model more efficient.
+* **Conflict Resolution**: Reconciles differences between models by creating a unified sign vector that represents the most dominant direction of change across all models.
+
+### DARE Merging
+
+Introduced by Yu et al. (2023), [DARE](https://arxiv.org/abs/2311.03099) uses an approach similar to TIES with two main differences:
+
+* **Weight Pruning**: Randomly resets some fine-tuned weights to their original values, reducing model complexity.
+* **Weight Scaling**: Adjusts the remaining weights by scaling and combining them with the base model's weights to maintain consistent performance.
+
+Mergekit’s implementation of this method has two flavours: with the sign election step of TIES (`dare_ties`) or without (`dare_linear`).
+
+For more information refer this [Merge Large Language Models with MergeKit by Maxime Labonne](https://towardsdatascience.com/merge-large-language-models-with-mergekit-2118fb392b54)
+
 ## 🧩 Configuration
 
 ```yaml
@@ -65,4 +88,12 @@ pipeline = transformers.pipeline(
 
 outputs = pipeline(prompt, max_new_tokens=256, do_sample=True, temperature=0.7, top_k=50, top_p=0.95)
 print(outputs[0]["generated_text"])
-```
+```
+
+## 🏆 Evaluation Scores
+Coming soon
+
+
+## Special thanks & Reference
+- Maxime Labonne for their easy-to-use colab-notebook [Merging LLMs with MergeKit](https://github.com/mlabonne/llm-course/blob/main/Mergekit.ipynb) and [Blog](https://towardsdatascience.com/merge-large-language-models-with-mergekit-2118fb392b54)
+- Authors of [Mergekit](https://github.com/arcee-ai/mergekit)