Update README.md

README.md

@@ -25,6 +25,7 @@ It is designed to handle a variety of complex instructions and generate coherent
 - **Language(s) (NLP):** Korean, English
 - **License:** [Llama3](https://llama.meta.com/llama3/license)
 - **Finetuned from model:** [meta-llama/Meta-Llama-3-8B-Instruct](https://huggingface.co/meta-llama/Meta-Llama-3-8B-Instruct)
+- **Tokenizer Soucrce:** [saltlux/Ko-Llama3-Luxia-8B](https://huggingface.co/saltlux/Ko-Llama3-Luxia-8B)
 
 ## Model Sources
 
@@ -159,7 +160,7 @@ The model is trained on the [MarkrAI/KoCommercial-Dataset](https://huggingface.c
 
 ### Training Procedure
 
-The model training used LoRA for computational efficiency. 0.02 billion parameters(0.26% of total parameters) were trained.
+The model training used LoRA for computational efficiency. 0.04 billion parameters(0.51% of total parameters) were trained.
 
 #### Training Hyperparameters
 
@@ -177,17 +178,17 @@ bnb_4bit_use_double_quant=False
 ################################################################################
 task_type="CAUSAL_LM"
 target_modules=["q_proj", "k_proj", "v_proj", "o_proj", "gate_proj", "up_proj", "down_proj"]
-r=8
-lora_alpha=16
-lora_dropout=0.05
+r=16
+lora_alpha=32
+lora_dropout=0.1
 bias="none"
 
 ################################################################################
 # TrainingArguments parameters
 ################################################################################
-num_train_epochs=1
+num_train_epochs=2
 per_device_train_batch_size=1
-gradient_accumulation_steps=4
+gradient_accumulation_steps=1
 gradient_checkpointing=True
 learning_rate=2e-5
 lr_scheduler_type="cosine"
@@ -208,17 +209,6 @@ packing=True
 
 ### Metrics
 
-#### English
-
-- **AI2 Reasoning Challenge (25-shot):** a set of grade-school science questions.
-- **HellaSwag (10-shot):** a test of commonsense inference, which is easy for humans (~95%) but challenging for SOTA models.
-- **MMLU (5-shot):** a test to measure a text model's multitask accuracy. The test covers 57 tasks including elementary mathematics, US history, computer science, law, and more.
-- **TruthfulQA (0-shot):** a test to measure a model's propensity to reproduce falsehoods commonly found online. Note: TruthfulQA is technically a 6-shot task in the Harness because each example is prepended with 6 Q/A pairs, even in the 0-shot setting.
-- **Winogrande (5-shot):** an adversarial and difficult Winograd benchmark at scale, for commonsense reasoning.
-- **GSM8k (5-shot):** diverse grade school math word problems to measure a model's ability to solve multi-step mathematical reasoning problems.
-
-#### Korean
-
 - **Ko-HellaSwag:**
 - **Ko-MMLU:**
 - **Ko-Arc:**
@@ -227,68 +217,6 @@ packing=True
   
 ### Results
 
-#### English
-
-<table>
-  <tr>
-   <td><strong>Benchmark</strong>
-   </td>
-   <td><strong>Waktaverse Llama 3 8B</strong>
-   </td>
-   <td><strong>Llama 3 8B</strong>
-   </td>
-  </tr>
-  <tr>
-   <td>Average
-   </td>
-   <td>66.77
-   </td>
-   <td>66.87
-   </td>
-  </tr>
-  <tr>
-   <td>ARC
-   </td>
-   <td>60.32
-   </td>
-   <td>60.75
-   </td>
-  </tr>
-  <tr>
-   <td>HellaSwag
-   </td>
-   <td>78.55
-   </td>
-   <td>78.55
-   </td>
-  </tr>
-  <tr>
-   <td>MMLU
-   </td>
-   <td>67.9
-   </td>
-   <td>67.07
-   </td>
-  </tr>
-  <tr>
-   <td>Winograde
-   </td>
-   <td>74.27
-   </td>
-   <td>74.51
-   </td>
-  <tr>
-    <td>GSM8K
-   </td>
-   <td>70.36
-   </td>
-   <td>68.69
-   </td>
-  </tr>
-</table>
-
-#### Korean
-
 <table>
   <tr>
    <td><strong>Benchmark</strong>
@@ -365,7 +293,11 @@ packing=True
 **Waktaverse-Llama-3**
 
 ```
-TBD
+@article{waktaversellama3modelcard,
+  title={Waktaverse Llama 3 Model Card},
+  author={AI@Waktaverse},
+  year={2024},
+  url = {https://huggingface.co/PathFinderKR/Waktaverse-Llama-3-KO-8B-Instruct}
 ```
 
 **Llama-3**
@@ -379,6 +311,16 @@ TBD
 }
 ```
 
+**Ko-Llama3-Luxia-8B**
+```
+@article{kollama3luxiamodelcard,
+  title={Ko Llama 3 Luxia Model Card},
+  author={AILabs@Saltux},
+  year={2024},
+  url={https://huggingface.co/saltlux/Ko-Llama3-Luxia-8B/blob/main/README.md}
+}
+```
+
 
 
 ## Model Card Authors