Update README.md

README.md

@@ -6,10 +6,48 @@ language:
 - en
 pipeline_tag: text-generation
 ---
-# Limstral 7B
+# LIMSTAL
+
 ## Mistral 7B fine-tuned on LIMA
+This model is a fine-tuned version of [mistralai/Mistral-7B-v0.1](https://huggingface.co/mistralai/Mistral-7B-v0.1) on the LIMA dataset.
+
+## Training procedure
+
+The model was loaded on **8 bits** and fine-tuned on the LIMA dataset using the **LoRA** PEFT technique with the `huggingface/peft` library for 2 epochs on 1 x A100 (40GB) GPU.
+LoRA config:
+```
+config = LoraConfig(
+        lora_alpha=16,
+        lora_dropout=0.1,
+        r=64,
+        bias="none",
+        task_type="CAUSAL_LM",
+        target_modules = ['q_proj', 'k_proj', 'down_proj', 'v_proj', 'o_proj', 'gate_proj', 'up_proj']
+    )
+```
+
+### Training hyperparameters
+
+The following hyperparameters were used during training:
+- learning_rate: 0.0002
+- train_batch_size: 2
+- eval_batch_size: 8
+- seed: 66
+- gradient_accumulation_steps: 64
+- total_train_batch_size: 128
+- optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
+- lr_scheduler_type: constant
+- lr_scheduler_warmup_ratio: 0.03
+- num_epochs: 2
+- mixed_precision_training: Native AMP
+
+### Training results
+
+| Training Loss | Epoch | Step | Validation Loss |
+|:-------------:|:-----:|:----:|:---------------:|
+| 1.7917        | 0.72  | 5    | 1.7604          |
+| 1.7743        | 1.44  | 10   | 1.7217          |
 
-WIP
 
 ### Usage
 ```py
@@ -29,4 +67,11 @@ gen = pipeline("text-generation", model=model, tokenizer=tokenizer, device=0)
 instruction = "[INST] Write a email to day goodbye to me boss [\INST]"
 res = gen(instruction, max_new_tokens=512, temperature=0.3, top_p=0.75, top_k=40, repetition_penalty=1.2)
 print(res[0]['generated_text'])
-```
+```
+
+### Framework versions
+
+- Transformers 4.35.0.dev0
+- Pytorch 2.1.0+cu118
+- Datasets 2.14.6
+- Tokenizers 0.14.1