license:mit region:us
---
license: mit
---

- epochs: 3
- learning rate: 3e-4
- lora rank: 8
- lora alpha: 16
- lora dropout: 0.05 for cutoff 1024 13B, otherwise no dropout due to gradient checkpointing
- masking: none
- mbatch size: 4 (1 for 30B)
- batch size: 8 (2 for 30B)
- val set size: 0.2
- sdp implementation: xformers
- optimizer: AdamW
- eval strategy: none

Cleaned combination of:

[https://huggingface.co/datasets/QingyiSi/Alpaca-CoT](https://huggingface.co/datasets/QingyiSi/Alpaca-CoT)
- Chain of thought QED
- Chain of thought Aqua
- CodeAlpaca

[https://huggingface.co/datasets/neulab/conala](https://huggingface.co/datasets/neulab/conala)
- Code snippets

[https://huggingface.co/datasets/yahma/alpaca-cleaned](https://huggingface.co/datasets/yahma/alpaca-cleaned)
- Alpaca GPT4

Used in https://huggingface.co/kaiokendev/SuperCOT-LoRA