Update README.md

README.md

@@ -4,28 +4,25 @@ tags:
 - vllm
 ---
 
+# Meta-Llama-3-8B-Instruct-FP8
 
+## Model Overview
 Meta-Llama-3-8B-Instruct quantized to FP8 weights and activations using per-tensor quantization, ready for inference with vLLM >= 0.5.0.
 
+## Usage and Creation
 Produced using [AutoFP8 with calibration samples from ultrachat](https://github.com/neuralmagic/AutoFP8/blob/147fa4d9e1a90ef8a93f96fc7d9c33056ddc017a/example_dataset.py).
 
-Accuracy on MMLU:
-```
-vllm (pretrained=meta-llama/Meta-Llama-3-8B-Instruct,gpu_memory_utilization=0.4), gen_kwargs: (None), limit: None, num_fewshot: 5, batch_size: 16
-|      Groups      |Version|Filter|n-shot|Metric|Value |   |Stderr|
-|------------------|-------|------|-----:|------|-----:|---|-----:|
-|mmlu              |N/A    |none  |     0|acc   |0.6569|±  |0.0038|
-| - humanities     |N/A    |none  |     5|acc   |0.6049|±  |0.0068|
-| - other          |N/A    |none  |     5|acc   |0.7203|±  |0.0078|
-| - social_sciences|N/A    |none  |     5|acc   |0.7663|±  |0.0075|
-| - stem           |N/A    |none  |     5|acc   |0.5652|±  |0.0085|
+## Evaluation
+
+### Open LLM Leaderboard evaluation scores
+|                      | Meta-Llama-3-8B-Instruct | Meta-Llama-3-8B-Instruct-FP8<br>(this model) |
+| :------------------: | :----------------------: | :------------------------------------------------: |
+| arc-c<br>25-shot     | 62.54                    | 61.77                                              |
+| hellaswag<br>10-shot | 78.83                    | 78.56                                              |
+| mmlu<br>5-shot       | 66.60                    | 66.27                                              |
+| truthfulqa<br>0-shot | 52.44                    | 52.35                                              |
+| winogrande<br>5-shot | 75.93                    | 76.4                                              |
+| gsm8k<br>5-shot      | 75.96                    | 73.99                                              |
+| **Average<br>Accuracy**  | **68.71**                    |              **68.22**                                     |
+| **Recovery**             | **100%**                     |              **99.28%**                                     |
 
-vllm (pretrained=nm-testing/Meta-Llama-3-8B-Instruct-FP8,quantization=fp8,gpu_memory_utilization=0.4), gen_kwargs: (None), limit: None, num_fewshot: 5, batch_size: 16
-|      Groups      |Version|Filter|n-shot|Metric|Value |   |Stderr|
-|------------------|-------|------|-----:|------|-----:|---|-----:|
-|mmlu              |N/A    |none  |     0|acc   |0.6567|±  |0.0038|
-| - humanities     |N/A    |none  |     5|acc   |0.6072|±  |0.0068|
-| - other          |N/A    |none  |     5|acc   |0.7206|±  |0.0078|
-| - social_sciences|N/A    |none  |     5|acc   |0.7618|±  |0.0075|
-| - stem           |N/A    |none  |     5|acc   |0.5649|±  |0.0085|
-```