| license: apache-2.0 | |
| # gemma-7b-ultrachat-sft | |
| gemma-7b-ultrachat-sft is an SFT fine-tuned version of [google/gemma-7b](https://huggingface.co/google/gemma-7b) using the [stingning/ultrachat](https://huggingface.co/datasets/stingning/ultrachat) dataset. | |
| ## Fine-tuning configuration | |
| ### LoRA | |
| - LoRA r: 8 | |
| - LoRA alpha: 16 | |
| - LoRA dropout: 0.1 | |
| ### Training arguments | |
| - Epochs: 1 | |
| - Batch size: 4 | |
| - Gradient accumulation steps: 6 | |
| - Optimizer: paged_adamw_32bit | |
| - Max steps: 100 | |
| - Learning rate: 0.0002 | |
| - Weight decay: 0.001 | |
| - Learning rate scheduler type: constant | |
| - Max seq length: 2048 |