esm2_t130_150M-lora-classifier_2024-04-26_00-25-40

This model is a fine-tuned version of facebook/esm2_t30_150M_UR50D on the None dataset. It achieves the following results on the evaluation set:

• Loss: 1.6470
• Accuracy: 0.8887

Model description

More information needed

Intended uses & limitations

More information needed

Training and evaluation data

More information needed

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

• learning_rate: 0.0005701568055793089
• train_batch_size: 28
• eval_batch_size: 28
• seed: 8893
• optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
• lr_scheduler_type: cosine
• num_epochs: 300
• mixed_precision_training: Native AMP

Training results

Training Loss	Epoch	Step	Validation Loss	Accuracy
0.7096	1.0	55	0.6718	0.6055
0.6769	2.0	110	0.6739	0.6055
0.579	3.0	165	0.6608	0.6484
0.5726	4.0	220	0.5777	0.7109
0.6381	5.0	275	0.5020	0.7676
0.183	6.0	330	0.3725	0.8320
0.3701	7.0	385	0.3508	0.8535
0.2147	8.0	440	0.3191	0.8711
0.1654	9.0	495	0.3036	0.875
0.1581	10.0	550	0.3761	0.8516
0.3459	11.0	605	0.3746	0.8594
0.3325	12.0	660	0.3025	0.8867
0.1237	13.0	715	0.2983	0.8770
0.5167	14.0	770	0.3044	0.8887
0.3541	15.0	825	0.2927	0.8906
0.0378	16.0	880	0.3669	0.8906
0.062	17.0	935	0.3298	0.8887
0.1695	18.0	990	0.2912	0.9004
0.0444	19.0	1045	0.3034	0.9004
0.1794	20.0	1100	0.3641	0.8828
0.0634	21.0	1155	0.3521	0.8867
0.0446	22.0	1210	0.3438	0.8887
0.0266	23.0	1265	0.4553	0.8867
0.2637	24.0	1320	0.4715	0.8867
0.159	25.0	1375	0.4323	0.8945
0.2401	26.0	1430	0.6019	0.8809
0.1317	27.0	1485	0.5549	0.8906
0.1223	28.0	1540	0.4819	0.8926
0.0015	29.0	1595	0.6432	0.8711
0.0007	30.0	1650	0.6480	0.8926
0.0774	31.0	1705	0.7596	0.8926
0.1262	32.0	1760	0.7614	0.8809
0.034	33.0	1815	0.7392	0.8789
0.0021	34.0	1870	0.9068	0.8848
0.0003	35.0	1925	0.8724	0.8711
0.0001	36.0	1980	0.9483	0.8867
0.0127	37.0	2035	0.9638	0.8828
0.0001	38.0	2090	0.9105	0.8926
0.0001	39.0	2145	0.9231	0.8809
0.0008	40.0	2200	1.0224	0.8867
0.0001	41.0	2255	1.0666	0.8848
0.0002	42.0	2310	1.1028	0.8848
0.0	43.0	2365	0.9653	0.8906
0.0006	44.0	2420	1.1108	0.8848
0.0001	45.0	2475	1.2919	0.8730
0.0002	46.0	2530	1.0834	0.8926
0.0002	47.0	2585	1.1240	0.8887
0.0135	48.0	2640	1.1466	0.8887
0.0008	49.0	2695	1.2674	0.8691
0.0	50.0	2750	1.1311	0.8887
0.0086	51.0	2805	1.0957	0.8887
0.0	52.0	2860	1.1336	0.8789
0.0007	53.0	2915	1.1494	0.875
0.0002	54.0	2970	1.0790	0.8848
0.0002	55.0	3025	1.1489	0.8809
0.0	56.0	3080	1.1479	0.8867
0.0022	57.0	3135	1.2092	0.8848
0.2415	58.0	3190	1.2060	0.8848
0.7813	59.0	3245	1.3750	0.8613
0.0	60.0	3300	1.1202	0.875
0.0	61.0	3355	1.0502	0.8848
0.0	62.0	3410	1.3270	0.8730
0.0015	63.0	3465	1.0082	0.875
0.0002	64.0	3520	0.9724	0.8867
0.0014	65.0	3575	1.0862	0.8770
0.0002	66.0	3630	1.1366	0.8730
0.1868	67.0	3685	1.1838	0.8770
0.0004	68.0	3740	1.2073	0.875
0.0007	69.0	3795	1.1793	0.8770
0.0	70.0	3850	1.2262	0.8652
0.2838	71.0	3905	1.2415	0.875
0.0	72.0	3960	1.2346	0.8770
0.0041	73.0	4015	1.0830	0.8789
0.0055	74.0	4070	1.0731	0.8867
0.0	75.0	4125	1.4096	0.8652
0.0034	76.0	4180	1.1142	0.8711
0.0	77.0	4235	1.0250	0.8848
0.0002	78.0	4290	1.0700	0.8691
0.0009	79.0	4345	0.9032	0.8789
0.0001	80.0	4400	1.0556	0.8730
0.0001	81.0	4455	1.0740	0.8770
0.0002	82.0	4510	1.2571	0.8691
0.0	83.0	4565	1.2007	0.8809
0.0	84.0	4620	1.2515	0.875
0.0001	85.0	4675	1.0750	0.8828
0.0006	86.0	4730	1.3016	0.8730
0.0001	87.0	4785	1.2393	0.8809
0.0	88.0	4840	1.2232	0.8848
0.0003	89.0	4895	1.2187	0.8789
0.0	90.0	4950	1.2328	0.8730
0.0	91.0	5005	1.3026	0.8848
0.0	92.0	5060	1.3152	0.8770
0.0	93.0	5115	1.4069	0.875
0.0	94.0	5170	1.3988	0.8770
0.0	95.0	5225	1.3675	0.8594
0.0	96.0	5280	1.3366	0.8770
0.0003	97.0	5335	1.2140	0.8848
0.0	98.0	5390	1.3585	0.8711
0.0	99.0	5445	1.1665	0.8672
0.0	100.0	5500	1.0947	0.8809
0.0099	101.0	5555	1.2993	0.8730
0.0	102.0	5610	1.3578	0.8789
0.0	103.0	5665	1.3596	0.8867
0.0006	104.0	5720	1.3164	0.8848
0.0	105.0	5775	1.4100	0.8770
0.0	106.0	5830	1.3459	0.875
0.0005	107.0	5885	1.3783	0.8809
0.0	108.0	5940	1.2698	0.8770
0.0	109.0	5995	1.3933	0.8848
0.0	110.0	6050	1.3813	0.8809
0.0	111.0	6105	1.5747	0.875
0.0001	112.0	6160	1.3368	0.8867
0.0486	113.0	6215	1.3833	0.8828
0.1476	114.0	6270	1.4943	0.8828
0.0002	115.0	6325	1.4725	0.8789
0.0	116.0	6380	1.4614	0.875
0.0047	117.0	6435	1.6313	0.8770
0.0	118.0	6490	1.4459	0.8848
0.0026	119.0	6545	1.4150	0.8730
0.0	120.0	6600	1.6055	0.8555
0.0001	121.0	6655	1.3710	0.8789
0.3319	122.0	6710	1.3940	0.8867
0.0001	123.0	6765	1.2486	0.875
0.0002	124.0	6820	1.2946	0.8711
0.0	125.0	6875	1.2341	0.8711
0.0	126.0	6930	1.1418	0.8887
0.0	127.0	6985	1.0713	0.8926
0.0001	128.0	7040	1.1391	0.8613
0.1624	129.0	7095	1.2195	0.8789
0.0	130.0	7150	1.1576	0.8770
0.0001	131.0	7205	1.2939	0.8730
0.0	132.0	7260	1.1568	0.8867
0.0	133.0	7315	1.2117	0.8848
0.0	134.0	7370	1.1264	0.8926
0.0	135.0	7425	1.1675	0.8848
0.0	136.0	7480	1.1983	0.8828
0.0	137.0	7535	1.2666	0.8770
0.0001	138.0	7590	1.1287	0.8848
0.0	139.0	7645	1.0505	0.8848
0.0	140.0	7700	1.1770	0.8770
0.0	141.0	7755	1.1749	0.8906
0.0	142.0	7810	1.1311	0.8711
0.0	143.0	7865	1.1114	0.8652
0.0	144.0	7920	1.1419	0.8691
0.0	145.0	7975	1.1666	0.8691
0.0	146.0	8030	1.1712	0.8711
0.0	147.0	8085	1.1831	0.8711
0.0	148.0	8140	1.1799	0.8711
0.0	149.0	8195	1.1876	0.8711
0.0	150.0	8250	1.1884	0.8730
0.0	151.0	8305	1.2389	0.8730
0.0	152.0	8360	1.3622	0.875
0.0	153.0	8415	1.2604	0.8789
0.0	154.0	8470	1.3336	0.875
0.0	155.0	8525	1.3496	0.8809
0.0	156.0	8580	1.3882	0.8555
0.1815	157.0	8635	1.3679	0.8789
0.288	158.0	8690	1.3804	0.8691
0.0	159.0	8745	1.2980	0.8770
0.0	160.0	8800	1.4075	0.8789
0.0	161.0	8855	1.4231	0.8789
0.0	162.0	8910	1.4730	0.875
0.0019	163.0	8965	1.5861	0.8672
0.0	164.0	9020	1.4080	0.8809
0.0005	165.0	9075	1.5852	0.8711
0.0	166.0	9130	1.5370	0.875
0.0	167.0	9185	1.5288	0.875
0.0	168.0	9240	1.5516	0.8711
0.0	169.0	9295	1.5268	0.8730
0.0	170.0	9350	1.5061	0.8672
0.0	171.0	9405	1.4843	0.875
0.0	172.0	9460	1.5478	0.8633
0.0	173.0	9515	1.4753	0.8730
0.0	174.0	9570	1.6709	0.8730
0.0	175.0	9625	1.6663	0.875
0.0	176.0	9680	1.6980	0.8672
0.0	177.0	9735	1.5563	0.8770
0.0	178.0	9790	1.6146	0.875
0.0	179.0	9845	1.5599	0.8770
0.0	180.0	9900	1.5558	0.8789
0.0	181.0	9955	1.8485	0.8633
0.0	182.0	10010	1.7223	0.8789
0.0	183.0	10065	1.7169	0.875
0.0	184.0	10120	1.7125	0.8711
0.0	185.0	10175	1.7065	0.8711
0.0	186.0	10230	1.7748	0.8730
0.0	187.0	10285	1.6861	0.8789
0.0	188.0	10340	1.7325	0.8887
0.0	189.0	10395	1.7658	0.8828
0.0	190.0	10450	1.7649	0.8809
0.0	191.0	10505	1.7555	0.8828
0.0162	192.0	10560	1.8313	0.8691
0.0001	193.0	10615	1.8314	0.8574
0.0	194.0	10670	1.7706	0.8672
0.0	195.0	10725	1.6568	0.8730
0.0	196.0	10780	1.6568	0.8770
0.0	197.0	10835	1.6185	0.8848
0.0	198.0	10890	1.6133	0.8848
0.0	199.0	10945	1.6129	0.8848
0.0	200.0	11000	1.6121	0.8848
0.0	201.0	11055	1.6104	0.8828
0.0	202.0	11110	1.6075	0.8828
0.0	203.0	11165	1.6153	0.8867
0.0	204.0	11220	1.6339	0.8828
0.0	205.0	11275	1.6164	0.8867
0.0	206.0	11330	1.6114	0.8848
0.0	207.0	11385	1.6122	0.8867
0.0	208.0	11440	1.6079	0.8867
0.0	209.0	11495	1.6132	0.8867
0.0	210.0	11550	1.6141	0.8867
0.0	211.0	11605	1.6122	0.8867
0.0	212.0	11660	1.6070	0.8867
0.0	213.0	11715	1.6010	0.8867
0.0	214.0	11770	1.6562	0.8789
0.0005	215.0	11825	1.6297	0.8887
0.0	216.0	11880	1.6070	0.8809
0.0	217.0	11935	1.6750	0.8770
0.0	218.0	11990	1.6822	0.8730
0.0	219.0	12045	1.6819	0.8730
0.0	220.0	12100	1.6846	0.8770
0.0	221.0	12155	1.6827	0.875
0.0	222.0	12210	1.6822	0.875
0.0	223.0	12265	1.6780	0.8770
0.0	224.0	12320	1.6813	0.8770
0.0	225.0	12375	1.6770	0.8770
0.0	226.0	12430	1.6878	0.8789
0.0	227.0	12485	1.8890	0.8672
0.0	228.0	12540	1.6978	0.8828
0.0	229.0	12595	1.6945	0.8867
0.0	230.0	12650	1.6960	0.8848
0.0	231.0	12705	1.6972	0.8867
0.0	232.0	12760	1.6929	0.8867
0.0	233.0	12815	1.6911	0.8848
0.0	234.0	12870	1.6887	0.8867
0.0	235.0	12925	1.6999	0.8848
0.0	236.0	12980	1.7000	0.8848
0.0	237.0	13035	1.6877	0.8867
0.0	238.0	13090	1.6858	0.8867
0.0	239.0	13145	1.6859	0.8867
0.0	240.0	13200	1.6842	0.8867
0.0	241.0	13255	1.6829	0.8867
0.0	242.0	13310	1.6800	0.8867
0.0	243.0	13365	1.6870	0.8848
0.0	244.0	13420	1.6856	0.8848
0.0	245.0	13475	1.6831	0.8848
0.0	246.0	13530	1.6864	0.8828
0.0	247.0	13585	1.6896	0.8828
0.0	248.0	13640	1.6900	0.8828
0.0	249.0	13695	1.6906	0.8848
0.0	250.0	13750	1.6928	0.8828
0.0	251.0	13805	1.6943	0.8828
0.0	252.0	13860	1.6902	0.8789
0.0	253.0	13915	1.6638	0.8887
0.0	254.0	13970	1.6632	0.8867
0.0	255.0	14025	1.6627	0.8867
0.0	256.0	14080	1.6631	0.8867
0.0	257.0	14135	1.6626	0.8867
0.0	258.0	14190	1.6629	0.8867
0.0	259.0	14245	1.6617	0.8867
0.0	260.0	14300	1.6606	0.8867
0.0	261.0	14355	1.6598	0.8867
0.0	262.0	14410	1.6559	0.8867
0.0	263.0	14465	1.6564	0.8867
0.0	264.0	14520	1.6555	0.8867
0.0	265.0	14575	1.6588	0.8867
0.0	266.0	14630	1.6565	0.8867
0.0	267.0	14685	1.6558	0.8867
0.0	268.0	14740	1.6564	0.8848
0.0	269.0	14795	1.6578	0.8848
0.0	270.0	14850	1.6566	0.8848
0.0	271.0	14905	1.6560	0.8867
0.0	272.0	14960	1.6587	0.8848
0.0	273.0	15015	1.6575	0.8867
0.0	274.0	15070	1.6575	0.8848
0.0	275.0	15125	1.6570	0.8867
0.0	276.0	15180	1.6586	0.8848
0.0	277.0	15235	1.6572	0.8887
0.0	278.0	15290	1.6577	0.8848
0.0	279.0	15345	1.6570	0.8867
0.0	280.0	15400	1.6567	0.8887
0.0	281.0	15455	1.6548	0.8887
0.0	282.0	15510	1.6558	0.8867
0.0	283.0	15565	1.6505	0.8887
0.0	284.0	15620	1.6515	0.8887
0.0	285.0	15675	1.6513	0.8887
0.0	286.0	15730	1.6456	0.8887
0.0	287.0	15785	1.6471	0.8887
0.0	288.0	15840	1.6451	0.8887
0.0	289.0	15895	1.6468	0.8887
0.0	290.0	15950	1.6470	0.8887
0.0	291.0	16005	1.6448	0.8887
0.0	292.0	16060	1.6478	0.8887
0.0	293.0	16115	1.6475	0.8887
0.0	294.0	16170	1.6471	0.8887
0.0	295.0	16225	1.6476	0.8887
0.0	296.0	16280	1.6475	0.8887
0.0	297.0	16335	1.6460	0.8887
0.0	298.0	16390	1.6471	0.8887
0.0	299.0	16445	1.6469	0.8887
0.0	300.0	16500	1.6470	0.8887

Framework versions

• PEFT 0.10.0
• Transformers 4.39.3
• Pytorch 2.2.1
• Datasets 2.16.1
• Tokenizers 0.15.2