mrm8488 commited on
Commit
c0ab2b2
1 Parent(s): 30af980

Update README.md

Browse files
Files changed (1) hide show
  1. README.md +17 -1
README.md CHANGED
@@ -13,7 +13,23 @@ This model is a fine-tuned version of [mistralai/Mistral-7B-v0.1](https://huggin
13
 
14
  ## Training procedure
15
 
16
- The model was loaded on **8 bits** and fine-tuned on the LIMA dataset using the **LoRA** PEFT technique with the `huggingface/peft` library for 2 epochs on 1 x A100 (40GB) GPU.
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
17
  LoRA config:
18
  ```
19
  config = LoraConfig(
 
13
 
14
  ## Training procedure
15
 
16
+ The model was loaded on **8 bits** and fine-tuned on the LIMA dataset using the **LoRA** PEFT technique with the `huggingface/peft` library and `trl/sft` for 2 epochs on 1 x A100 (40GB) GPU.
17
+
18
+ SFT Trainer params:
19
+ ```
20
+ trainer = SFTTrainer(
21
+ model=model,
22
+ train_dataset=train_ds,
23
+ eval_dataset=test_ds,
24
+ peft_config=peft_config,
25
+ dataset_text_field="text",
26
+ max_seq_length=2048,
27
+ tokenizer=tokenizer,
28
+ args=training_arguments,
29
+ packing=False
30
+ )
31
+ ```
32
+
33
  LoRA config:
34
  ```
35
  config = LoraConfig(