Update README.md

README.md

@@ -4,6 +4,8 @@ language:
 - en
 tags:
 - Mathematical Reasoning
+datasets:
+- akjindal53244/Arithmo-Data
 ---
 # Model Card for Model ID
 
@@ -11,42 +13,144 @@ tags:
 [![Model Weight License](https://img.shields.io/badge/Model%20Weights%20License-Apache_2.0-green.svg)](LICENSE)
 [![Python 3.9+](https://img.shields.io/badge/python-3.9+-blue.svg)](https://www.python.org/downloads/release/python-390/)
 
-This model is instruction-tuned [Mistral-7B](https://huggingface.co/mistralai/Mistral-7B-v0.1) model using QLoRA on single RTX 4090 GPU. Model is tuned to reason and answer mathematical problems. Model is also capable of writing a Python program that upon compilation prints answer to the question. To generate python program, you can ask model to write a Python program (as part of prompt) along with question. Refer to **Results** section for examples.
+**P.S.:** Please reach out to [Ashvini Jindal](https://www.linkedin.com/in/ashvini-jindal-26653262/) if you would be interested in supporting compute need. We are looking for small-scale support so we'd appreciate any kind of help! :)
 
 ## Model Details
 
-It is a Instruction-tuned Mistral-7B and performs mathematical reasoning and optionally write a Python program.
+Arithmo-Mistral-7B is trained to reason and answer mathematical problems and is also capable of writing a Python program that upon execution prints answer to the question. We used [Mistral-7B](https://huggingface.co/mistralai/Mistral-7B-v0.1) as a base model and used QLoRA to fine-tune it on a single RTX 4090 GPU.
 
 ### Model Description
 
 - **Project GitHub Page:** https://github.com/akjindal53244/Arithmo-Mistral-7B
 - **Developed by:** [Ashvini Kumar Jindal](https://www.linkedin.com/in/ashvini-jindal-26653262/)
 - **Funded by:** self-work
-- **Model type:** Instruction-tuned
+- **Model type:** fine-tuned
 - **Language(s) (NLP):** English
 - **Finetuned from model:** mistralai/Mistral-7B-v0.1
 
+## Results
+
+Arithmo-Mistral-7B outperforms existing 7B and 13B state-of-the-art Mathematical Reasoning models. Refer to [Comparing Arithmo-Mistral-7B with other LLM models](https://github.com/akjindal53244/Arithmo-Mistral-7B/tree/master#comparing-arithmo-mistral-7b-with-other-llm-models) section for more details.
+
+<table>
+    <thead>
+        <tr>
+            <th>Prompt Approach</th>
+            <th>GSM8k</th>
+            <th>MATH</th>
+        </tr>
+    </thead>
+    <tbody>
+        <tr>
+            <td>Zero-Shot CoT</td>
+            <td><b>74.7</b></td>
+            <td><b>25.3</b></td>
+        </tr>
+        <tr>
+            <td>Zero-Shot PoT</td>
+            <td><b>71.2</b></td>
+            <td>-</td>
+        </tr>
+    </tbody>
+</table>
+
+- **Zero-Shot CoT**: On providing a question as prompt, model generates reasoning steps to solve the question along with answer. We check if answer matches with ground-truth.
+- **Zero-Shot PoT**: We prompt the model to generate a Python program for the given question. During inference, we execute the Python program generated by the model and check if the program output matches with ground-truth answer.
+
+
+## Installation
+
+```
+pip install transformers == 4.34.0
+pip install accelerate
+pip install sentencepiece
+pip install protobuf
+
+# If you are GPU poor like me
+pip install torch torchvision torchaudio --index-url https://download.pytorch.org/whl/cpu
+
+# If you have a GPU.
+pip install --pre torch --index-url https://download.pytorch.org/whl/nightly/cu118
+pip install scipy
+pip install bitsandbytes
+```
+
 
 ## How to query the model
 
-Arithmo-Mistral-7B is trained with the following format:
+```
+# Set `run_model_on_gpu` to `False` if you are running on CPU. Model will generate reasoning steps with answer for your question. If you want to generate Python program, uncomment line-69 that adds a Python prompt.
+# This script automatically does formatting for you, so you just need to type question (eg: `What is 2+2?`) without any prefix like `Question:`, etc.**
+
+$ python query_model.py
+```
+**Note:** Above script automatically does formatting for you, so you just need to type question (eg: `What is 2+2?`) without any prefix like `Question:`, etc. Checkout [query_model.py](https://github.com/akjindal53244/Arithmo-Mistral-7B/blob/master/query_model.py) for more details. <br><br>
 
-### CoT Format:
+##### Sample Input:
+```
+Question: There are total 10 children. I have to give 1 apple to first child, 2 apples to second child, 3 apples to third child, and so on. How many apples do I need?
+```
+##### Model Output:
+```
+Answer: The total number of apples needed is the sum of the first 10 positive integers.
+This can be calculated using the formula for the sum of an arithmetic series:
+\[S = \frac{n}{2}(a_1 + a_n),\]
+where $S$ is the sum, $n$ is the number of terms, $a_1$ is the first term, and $a_n$ is the last term.
+In this case, $n = 10$, $a_1 = 1$, and $a_n = 10$.
+Plugging these values into the formula, we get:
+\[S = \frac{10}{2}(1 + 10) = 5(11) = \boxed{55}.\]
+The answer is: 55
+```
+
+Arithmo-Mistral-7B is trained with the following format:
+#### CoT Format (generate reasoning steps with answer):
 ```
 Question: <question>
 
 Answer:
 ```
 
-### PoT Format:
+#### PoT Format (generate a python program):
 ```
 Question: <question> <python_prompt>
 
 Answer:
 ```
-
-It will perform best if queried in this way.
-
-## How to Get Started with the Model
-
-Model is compatibale with Huggingface. I will publish a generation/inference script soon. Model inference on CPU also works; I have tested it on Macbook M1 Pro. GPU inference is much faster than CPU inference.
+It will perform best if queried in this way with your own script.
+
+## Comparing Arithmo-Mistral-7B with other LLM models.
+Results for all models except `Arithmo-Mistral-7B` are taken from [MetaMath](https://github.com/meta-math/MetaMath/blob/main/README.MD) repository.
+
+| Model               | GSM8k Pass@1 | MATH Pass@1 |
+|---------------------|--------------|-------------|
+| MPT-7B              | 6.8          | 3.0         |
+| Falcon-7B           | 6.8          | 2.3         |
+| LLaMA-1-7B          | 11.0         | 2.9         |
+| LLaMA-2-7B          | 14.6         | 2.5         |
+| MPT-30B             | 15.2         | 3.1         |
+| LLaMA-1-13B         | 17.8         | 3.9         |
+| GPT-Neo-2.7B        | 19.5         | --          |
+| Falcon-40B          | 19.6         | 2.5         |
+| Baichuan-chat-13B   | 23.9         | --          |
+| Vicuna-v1.3-13B     | 27.6         | --          |
+| LLaMA-2-13B         | 28.7         | 3.9         |
+| InternLM-7B         | 31.2         | --          |
+| ChatGLM-2-6B        | 32.4         | --          |
+| GPT-J-6B            | 34.9         | --          |
+| LLaMA-1-33B         | 35.6         | 3.9         |
+| LLaMA-2-34B         | 42.2         | 6.24        |
+| RFT-7B              | 50.3         | --          |
+| LLaMA-1-65B         | 50.9         | 10.6        |
+| Qwen-7B             | 51.6         | --          |
+| WizardMath-7B       | 54.9         | 10.7        |
+| LLaMA-2-70B         | 56.8         | 13.5        |
+| WizardMath-13B      | 63.9         | 14.0        |
+| MetaMath-7B         | 66.5         | 19.8        |
+| MetaMath-13B        | 72.3         | 22.4        |
+| 🔥 **Arithmo-Mistral-7B Zero-Shot PoT**  | **71.2** | --       |
+| 🔥 **Arithmo-Mistral-7B Zero-Shot CoT**  | **74.7** | **25.3**       |
+| WizardMath-70B      | **81.6**     | 22.7        |
+| MetaMath-70B        | **82.3**     | **26.6**        |
+
+
+If you are interested in reproducing the resullts, visit https://github.com/akjindal53244/Arithmo-Mistral-7B#reproducing-results section.