metadata

base_model: bigcode/starcoderbase-7b
library_name: peft
license: bigcode-openrail-m
tags:
  - generated_from_trainer
model-index:
  - name: peft-starcoder-lora-cutlass
    results: []

peft-starcoder-lora-cutlass

This model is a fine-tuned version of bigcode/starcoderbase-7b on an unknown dataset. It achieves the following results on the evaluation set:

Loss: 0.4322

Model description

More information needed

Intended uses & limitations

More information needed

Training and evaluation data

More information needed

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

learning_rate: 0.0005
train_batch_size: 16
eval_batch_size: 16
seed: 42
optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
lr_scheduler_type: cosine
lr_scheduler_warmup_steps: 30
training_steps: 4000

Training results

Training Loss	Epoch	Step	Validation Loss
0.5409	0.025	100	0.4729
0.3672	0.05	200	0.4051
0.3432	0.075	300	0.3876
0.312	0.1	400	0.3548
0.2594	0.125	500	0.3507
0.2821	0.15	600	0.3428
0.1958	0.175	700	0.3391
0.2362	0.2	800	0.3405
0.209	0.225	900	0.3421
0.2114	0.25	1000	0.3481
0.1877	0.275	1100	0.3447
0.1903	0.3	1200	0.3533
0.1723	0.325	1300	0.3578
0.1465	0.35	1400	0.3643
0.1751	0.375	1500	0.3639
0.1213	0.4	1600	0.3749
0.119	0.425	1700	0.3692
0.1199	0.45	1800	0.3820
0.0986	0.475	1900	0.3816
0.1232	0.5	2000	0.3868
0.0819	0.525	2100	0.3923
0.096	0.55	2200	0.3964
0.0873	0.575	2300	0.3977
0.0892	0.6	2400	0.4007
0.0802	0.625	2500	0.4078
0.066	0.65	2600	0.4119
0.0863	0.675	2700	0.4125
0.0722	0.7	2800	0.4189
0.0888	0.725	2900	0.4201
0.0757	0.75	3000	0.4263
0.0666	0.775	3100	0.4234
0.0757	0.8	3200	0.4275
0.053	0.825	3300	0.4289
0.0772	0.85	3400	0.4290
0.0546	0.875	3500	0.4297
0.0601	0.9	3600	0.4311
0.0606	0.925	3700	0.4317
0.0544	0.95	3800	0.4314
0.065	0.975	3900	0.4320
0.0503	1.0	4000	0.4322

Framework versions

PEFT 0.11.1
Transformers 4.42.4
Pytorch 2.4.0a0+07cecf4168.nv24.05
Datasets 2.20.0
Tokenizers 0.19.1