GreenBitAI/yi-34b-w4a16g32

license: apache-2.0

GreenBit Yi

This is GreenBitAI's pretrained 4-bit Yi 34B model with extreme compression yet still strong performance.

Please refer to our Github page for the code to run the model and more information.

Model Description

• Developed by: GreenBitAI
• Evaluated By: 01-Yi official
• Model type: Causal (Llama 2/Yi 6B)
• Language(s) (NLP): English
• License: Apache 2.0, Llama 2 license agreement

Few Shot Evaluation (officially evaluated by 01-Yi)

Model	Yi-34B FP16	Yi-34B 4 bit	Yi-6B FP16	Yi-6B 4 bit
GroupSize	-	32	-	32
Model Size (GB)	68.79	19.89	12.12	4.04
AVG	70.64	69.7	60.11	59.14
Detailed Evaluation
MMLU	76.32	75.42	63.24	62.09
CMMLU	83.65	83.07	75.53	72.85
ARC-e	84.42	84.13	77.23	76.52
ARC-c	61.77	59.56	50.34	48.47
GAOKAO	82.8	81.37	72.2	72.87
GSM8K	67.24	63.61	32.52	28.05
HumanEval	25.6	25	15.85	15.85
BBH	54.3	52.3	42.8	41.47
WinoGrande	78.68	78.53	70.63	71.19
PIQA	82.86	82.75	78.56	79.05
SIQA	74.46	73.44	64.53	64.53
HellaSwag	83.64	83.02	74.91	73.27
OBQA	91.6	90.8	85.4	82.6
CSQA	83.37	83.05	76.9	75.43
TriviaQA	81.52	80.73	64.85	61.75
SquAD	92.46	91.12	88.95	88.39
BoolQ	88.25	88.17	76.23	77.1
MBPP	41	39.68	26.32	25.13
QUAC	48.61	47.43	40.92	40.16
Lambda	73.18	73.39	67.74	67.8
NaturalQuestion	27.67	27.21	16.69	17.42

Zero Shot Evaluation

Task	Metric	Yi-6B FP16	Yi-6B 4 bit	Yi-34B 4 bit
Openbookqa	acc	0.314	0.324	0.344
	ac_norm	0.408	0.42	0.474
arc_challenge	acc	0.462	0.4573	0.569
	ac_norm	0.504	0.483	0.5964
hellawswag	acc	0.553	0.5447	0.628
	ac_norm	0.749	0.7327	0.83
piqa	acc	0.777	0.7709	0.8079
	ac_norm	0.787	0.7894	0.828
arc_easy	acc	0.777	0.7697	0.835
	ac_norm	0.774	0.7659	0.84
Winogrande	acc	0.707	0.7095	0.7853
boolq	acc	0.755	0.7648	0.886
truthfulqa_mc	mc1	0.29	0.2729	0.4026
	mc2	0.419	0.4033	0.5528
anli_r1	acc	0.423	0.416	0.554
anli_r2	acc	0.409	0.409	0.518
anli_r3	acc	0.411	0.393	0.4983
wic	acc	0.529	0.545	0.5376
rte	acc	0.685	0.7039	0.7617
record	f1	0.904	0.9011	0.924
	em	0.8962	0.8927	0.916
Average		0.596	0.5937	0.6708