license:apache-2.0 region:us
---
license: apache-2.0
---
This contains both the Math.json and GM8SK.jsonl, Converted to Alpaca format. GM8sk.jsonl was used for evaluating, and the math file was used for training.
MATH_Alpaca.json contains ~ 5,000 examples for evaluating.
gm8sk_Alpaca.json contains ~1,000 examples for evaluation.
nothing stops you from using this either one to train a model.
For ALPACA LORA users: Modules you can target with lora:"gate_proj", "down_proj", "up_proj", "q_proj", "v_proj", "k_proj", "o_proj" Most lora models use:"q_proj", "v_proj", "k_proj", "o_proj" Platypus which got terrific results: "gate_proj", "down_proj", "up_proj"

Research on targeting certain modules still needs to be done, but if you don't want to train over a previously trained models newly learned abilities, target different modules than the ones used for original training.

Hyper perameters used by Platypus: Hyperparameters for 13B and 70B Models Hyperparameter Platypus2-13B / 70B batch size 16 micro batch size 1 num epochs 1 learning rate 4e-4 / 3e-4 cutoff len 4096 lora rank 16 lora alpha 16 lora dropout 0.05 lora target modules gate_proj, down_proj, up_proj train on inputs False add eos token False group by length False prompt template alpaca lr scheduler cosine warmup steps 100

I would reccomend using a batch size of 4-10, and cutt off length to ≤ 2048 to avoid using vram issues. Load_in_4bit, Normal Float, and bf16. For single 24 gig card. If training with oobabooga you must edit the "training.py" file in the "oobabooga_windows\text-generation-webui\modules" folder. In line 49 edit standard modules to the modules you would like to target. If training with alpaca lora use the argument --lora_target_modules when running the train.py command. To load in 4bit you must edit the train file, adding load in 4 bit, bf16, and normal float quant.