---
license: apache-2.0
---
### Evaluation

|             Tasks             |Version|Filter|n-shot| Metric |Value |   |Stderr|
|-------------------------------|-------|------|-----:|--------|-----:|---|-----:|
| - medmcqa                     |Yaml   |none  |     0|acc     |0.5408|±  |0.0077|
|                               |       |none  |     0|acc_norm|0.5408|±  |0.0077|
| - medqa_4options              |Yaml   |none  |     0|acc     |0.5711|±  |0.0139|
|                               |       |none  |     0|acc_norm|0.5711|±  |0.0139|
| - anatomy (mmlu)              |      0|none  |     0|acc     |0.6815|±  |0.0402|
| - clinical_knowledge (mmlu)   |      0|none  |     0|acc     |0.7434|±  |0.0269|
| - college_biology (mmlu)      |      0|none  |     0|acc     |0.8056|±  |0.0331|
| - college_medicine (mmlu)     |      0|none  |     0|acc     |0.6647|±  |0.0360|
| - medical_genetics (mmlu)     |      0|none  |     0|acc     |0.7300|±  |0.0446|
| - professional_medicine (mmlu)|      0|none  |     0|acc     |0.7353|±  |0.0268|
|stem                           |N/A    |none  |     0|acc_norm|0.5478|±  |0.0067|
|                               |       |none  |     0|acc     |0.5909|±  |0.0058|
| - pubmedqa                    |      1|none  |     0|acc     |0.7620|±  |0.0191|

|Groups|Version|Filter|n-shot| Metric |Value |   |Stderr|
|------|-------|------|-----:|--------|-----:|---|-----:|
|stem  |N/A    |none  |     0|acc_norm|0.5478|±  |0.0067|
|      |       |none  |     0|acc     |0.5909|±  |0.0058|

![Comparison Image](https://huggingface.co/ChenWeiLi/MedPhi-3-mini_v1/resolve/main/compare.png)