goldsounds/jetpack-model-pt

sft

This model is a fine-tuned version of bigcode/starcoder2-7b on the starcoder_jetpack dataset. It achieves the following results on the evaluation set:

• Loss: 0.6761

Model description

More information needed

Intended uses & limitations

More information needed

Training and evaluation data

More information needed

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

• learning_rate: 5e-05
• train_batch_size: 1
• eval_batch_size: 1
• seed: 42
• optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
• lr_scheduler_type: cosine
• num_epochs: 3.0
• mixed_precision_training: Native AMP

Training results

Training Loss	Epoch	Step	Validation Loss
1.2469	0.02	100	0.8479
1.3584	0.03	200	0.8368
1.0651	0.05	300	0.8281
0.9209	0.06	400	0.8185
0.8306	0.08	500	0.8132
0.9175	0.1	600	0.8064
0.8157	0.11	700	0.8023
0.9469	0.13	800	0.7996
0.8872	0.14	900	0.7955
0.8842	0.16	1000	0.7913
0.7909	0.18	1100	0.7863
0.8196	0.19	1200	0.7844
0.9341	0.21	1300	0.7825
0.8801	0.22	1400	0.7787
0.9109	0.24	1500	0.7777
0.8964	0.26	1600	0.7759
0.9265	0.27	1700	0.7742
0.8632	0.29	1800	0.7699
1.0081	0.3	1900	0.7693
0.7651	0.32	2000	0.7664
1.0037	0.34	2100	0.7654
0.8768	0.35	2200	0.7642
0.8052	0.37	2300	0.7618
0.7271	0.38	2400	0.7595
0.9615	0.4	2500	0.7582
0.8284	0.42	2600	0.7555
0.8631	0.43	2700	0.7540
1.0507	0.45	2800	0.7518
0.8247	0.46	2900	0.7512
0.9835	0.48	3000	0.7496
0.8407	0.49	3100	0.7496
0.7417	0.51	3200	0.7467
0.7449	0.53	3300	0.7473
0.8562	0.54	3400	0.7437
0.9222	0.56	3500	0.7429
0.9242	0.57	3600	0.7413
0.8092	0.59	3700	0.7403
0.7279	0.61	3800	0.7394
0.7774	0.62	3900	0.7385
0.8942	0.64	4000	0.7364
0.9286	0.65	4100	0.7348
0.7703	0.67	4200	0.7354
0.8322	0.69	4300	0.7330
0.9851	0.7	4400	0.7324
0.8712	0.72	4500	0.7317
0.7871	0.73	4600	0.7310
0.7156	0.75	4700	0.7284
0.7856	0.77	4800	0.7277
0.7906	0.78	4900	0.7255
0.7917	0.8	5000	0.7250
0.6395	0.81	5100	0.7237
0.7567	0.83	5200	0.7232
0.8551	0.85	5300	0.7220
0.7392	0.86	5400	0.7226
0.9312	0.88	5500	0.7205
0.8323	0.89	5600	0.7196
0.7312	0.91	5700	0.7197
1.0	0.93	5800	0.7182
0.6164	0.94	5900	0.7177
0.7484	0.96	6000	0.7147
0.7924	0.97	6100	0.7144
0.9389	0.99	6200	0.7145
0.7108	1.01	6300	0.7136
0.8076	1.02	6400	0.7154
0.7232	1.04	6500	0.7147
0.6456	1.05	6600	0.7122
0.5862	1.07	6700	0.7133
0.6935	1.09	6800	0.7112
0.7522	1.1	6900	0.7103
1.0525	1.12	7000	0.7106
0.8285	1.13	7100	0.7099
0.6116	1.15	7200	0.7079
0.5617	1.17	7300	0.7087
0.6514	1.18	7400	0.7072
0.6729	1.2	7500	0.7052
0.6401	1.21	7600	0.7055
0.8089	1.23	7700	0.7052
0.8166	1.25	7800	0.7041
0.8685	1.26	7900	0.7026
0.6945	1.28	8000	0.7043
0.6955	1.29	8100	0.7010
0.734	1.31	8200	0.7022
0.5586	1.32	8300	0.7017
0.7299	1.34	8400	0.6999
1.089	1.36	8500	0.6994
0.5733	1.37	8600	0.6994
0.5409	1.39	8700	0.6987
0.8848	1.4	8800	0.6976
0.5739	1.42	8900	0.6971
0.728	1.44	9000	0.6963
0.7503	1.45	9100	0.6953
0.706	1.47	9200	0.6951
0.7926	1.48	9300	0.6945
0.6019	1.5	9400	0.6945
0.6707	1.52	9500	0.6943
0.7269	1.53	9600	0.6940
0.7216	1.55	9700	0.6923
0.6394	1.56	9800	0.6920
0.7608	1.58	9900	0.6909
1.034	1.6	10000	0.6908
0.7934	1.61	10100	0.6892
0.627	1.63	10200	0.6902
0.5849	1.64	10300	0.6897
0.7257	1.66	10400	0.6889
0.8931	1.68	10500	0.6890
0.6831	1.69	10600	0.6875
0.4995	1.71	10700	0.6879
0.757	1.72	10800	0.6873
0.4664	1.74	10900	0.6876
0.78	1.76	11000	0.6865
0.5564	1.77	11100	0.6865
0.7858	1.79	11200	0.6858
0.6989	1.8	11300	0.6851
0.705	1.82	11400	0.6841
0.5795	1.84	11500	0.6842
0.6989	1.85	11600	0.6837
0.6877	1.87	11700	0.6838
0.6484	1.88	11800	0.6835
0.8525	1.9	11900	0.6832
0.7547	1.92	12000	0.6823
0.8118	1.93	12100	0.6819
0.8859	1.95	12200	0.6823
0.738	1.96	12300	0.6811
0.7051	1.98	12400	0.6816
0.5598	2.0	12500	0.6802
0.6194	2.01	12600	0.6812
0.7101	2.03	12700	0.6817
0.7027	2.04	12800	0.6815
0.9432	2.06	12900	0.6810
0.5931	2.08	13000	0.6817
0.5412	2.09	13100	0.6810
0.6237	2.11	13200	0.6815
0.5871	2.12	13300	0.6812
0.8331	2.14	13400	0.6817
0.4528	2.15	13500	0.6812
0.6292	2.17	13600	0.6814
0.6219	2.19	13700	0.6800
0.6281	2.2	13800	0.6798
0.6949	2.22	13900	0.6803
0.6701	2.23	14000	0.6791
0.6467	2.25	14100	0.6795
0.6579	2.27	14200	0.6800
0.5978	2.28	14300	0.6802
0.7032	2.3	14400	0.6793
0.6347	2.31	14500	0.6787
0.9034	2.33	14600	0.6788
0.6166	2.35	14700	0.6781
0.7327	2.36	14800	0.6786
0.7159	2.38	14900	0.6777
0.6283	2.39	15000	0.6779
0.6113	2.41	15100	0.6776
0.5951	2.43	15200	0.6781
0.6747	2.44	15300	0.6777
0.5935	2.46	15400	0.6779
0.6435	2.47	15500	0.6776
0.637	2.49	15600	0.6772
0.4617	2.51	15700	0.6774
0.7937	2.52	15800	0.6771
0.7187	2.54	15900	0.6768
0.657	2.55	16000	0.6767
0.8606	2.57	16100	0.6767
0.4392	2.59	16200	0.6768
0.5675	2.6	16300	0.6769
0.6454	2.62	16400	0.6768
0.5787	2.63	16500	0.6767
0.6111	2.65	16600	0.6766
0.6106	2.67	16700	0.6767
0.5947	2.68	16800	0.6763
0.5576	2.7	16900	0.6763
0.659	2.71	17000	0.6762
0.787	2.73	17100	0.6761
0.5503	2.75	17200	0.6760
0.5558	2.76	17300	0.6760
0.6324	2.78	17400	0.6761
0.5846	2.79	17500	0.6761
0.9542	2.81	17600	0.6760
0.5755	2.83	17700	0.6761
0.7841	2.84	17800	0.6761
0.5662	2.86	17900	0.6761
0.8085	2.87	18000	0.6761
0.7389	2.89	18100	0.6761
0.736	2.91	18200	0.6761
0.5604	2.92	18300	0.6761
0.6156	2.94	18400	0.6761
0.5473	2.95	18500	0.6761
0.7286	2.97	18600	0.6761
0.5932	2.98	18700	0.6761

Framework versions

• PEFT 0.9.0
• Transformers 4.39.0.dev0
• Pytorch 2.1.0+cu121
• Datasets 2.18.0
• Tokenizers 0.15.2