abhinavnmagic commited on
Commit
3991529
1 Parent(s): 98b631e

Update README.md

Browse files
Files changed (1) hide show
  1. README.md +16 -19
README.md CHANGED
@@ -4,28 +4,25 @@ tags:
4
  - vllm
5
  ---
6
 
 
7
 
 
8
  Meta-Llama-3-8B-Instruct quantized to FP8 weights and activations using per-tensor quantization, ready for inference with vLLM >= 0.5.0.
9
 
 
10
  Produced using [AutoFP8 with calibration samples from ultrachat](https://github.com/neuralmagic/AutoFP8/blob/147fa4d9e1a90ef8a93f96fc7d9c33056ddc017a/example_dataset.py).
11
 
12
- Accuracy on MMLU:
13
- ```
14
- vllm (pretrained=meta-llama/Meta-Llama-3-8B-Instruct,gpu_memory_utilization=0.4), gen_kwargs: (None), limit: None, num_fewshot: 5, batch_size: 16
15
- | Groups |Version|Filter|n-shot|Metric|Value | |Stderr|
16
- |------------------|-------|------|-----:|------|-----:|---|-----:|
17
- |mmlu |N/A |none | 0|acc |0.6569|± |0.0038|
18
- | - humanities |N/A |none | 5|acc |0.6049|± |0.0068|
19
- | - other |N/A |none | 5|acc |0.7203|± |0.0078|
20
- | - social_sciences|N/A |none | 5|acc |0.7663|± |0.0075|
21
- | - stem |N/A |none | 5|acc |0.5652|± |0.0085|
 
 
 
22
 
23
- vllm (pretrained=nm-testing/Meta-Llama-3-8B-Instruct-FP8,quantization=fp8,gpu_memory_utilization=0.4), gen_kwargs: (None), limit: None, num_fewshot: 5, batch_size: 16
24
- | Groups |Version|Filter|n-shot|Metric|Value | |Stderr|
25
- |------------------|-------|------|-----:|------|-----:|---|-----:|
26
- |mmlu |N/A |none | 0|acc |0.6567|± |0.0038|
27
- | - humanities |N/A |none | 5|acc |0.6072|± |0.0068|
28
- | - other |N/A |none | 5|acc |0.7206|± |0.0078|
29
- | - social_sciences|N/A |none | 5|acc |0.7618|± |0.0075|
30
- | - stem |N/A |none | 5|acc |0.5649|± |0.0085|
31
- ```
 
4
  - vllm
5
  ---
6
 
7
+ # Meta-Llama-3-8B-Instruct-FP8
8
 
9
+ ## Model Overview
10
  Meta-Llama-3-8B-Instruct quantized to FP8 weights and activations using per-tensor quantization, ready for inference with vLLM >= 0.5.0.
11
 
12
+ ## Usage and Creation
13
  Produced using [AutoFP8 with calibration samples from ultrachat](https://github.com/neuralmagic/AutoFP8/blob/147fa4d9e1a90ef8a93f96fc7d9c33056ddc017a/example_dataset.py).
14
 
15
+ ## Evaluation
16
+
17
+ ### Open LLM Leaderboard evaluation scores
18
+ | | Meta-Llama-3-8B-Instruct | Meta-Llama-3-8B-Instruct-FP8<br>(this model) |
19
+ | :------------------: | :----------------------: | :------------------------------------------------: |
20
+ | arc-c<br>25-shot | 62.54 | 61.77 |
21
+ | hellaswag<br>10-shot | 78.83 | 78.56 |
22
+ | mmlu<br>5-shot | 66.60 | 66.27 |
23
+ | truthfulqa<br>0-shot | 52.44 | 52.35 |
24
+ | winogrande<br>5-shot | 75.93 | 76.4 |
25
+ | gsm8k<br>5-shot | 75.96 | 73.99 |
26
+ | **Average<br>Accuracy** | **68.71** | **68.22** |
27
+ | **Recovery** | **100%** | **99.28%** |
28