metadata
license: apache-2.0
gemma-7b-ultrachat-sft
gemma-7b-ultrachat-sft is an SFT fine-tuned version of google/gemma-7b using the stingning/ultrachat dataset.
Fine-tuning configuration
LoRA
- LoRA r: 8
- LoRA alpha: 16
- LoRA dropout: 0.1
Training arguments
- Epochs: 1
- Batch size: 4
- Gradient accumulation steps: 6
- Optimizer: paged_adamw_32bit
- Max steps: 100
- Learning rate: 0.0002
- Weight decay: 0.001
- Learning rate scheduler type: constant
- Max seq length: 2048