Update README.md

README.md

@@ -1,17 +1,18 @@
 ---
 inference: false
 ---
+# weblab-10b-instruction-sft-GPTQ 
+
 original model [weblab-10b-instruction-sft](https://huggingface.co/matsuo-lab/weblab-10b-instruction-sft)
 
 This is 4bit GPTQ Version.
 
 The size is smaller and the execution speed is faster, but the inference performance may be a little worse.
 
-Benchmark results are in progress.
-I will upload it at a later date.
-
 
 ### sample code
+At least one GPU is currently required due to a limitation of the Accelerate library.  
+
 ```
 pip install auto-gptq
 ```
@@ -32,12 +33,30 @@ model = AutoGPTQForCausalLM.from_quantized(
         device="cuda:0")
 
 prompt = "スタジオジブリの作品を５つ教えてください"
-prompt_template = f"### Instruction: {prompt}\n### Response:"
+prompt_template = f"### 指示: {prompt}\n\n### 応答:"
 
 tokens = tokenizer(prompt_template, return_tensors="pt").to("cuda:0").input_ids
 output = model.generate(input_ids=tokens, max_new_tokens=100, do_sample=True, temperature=0.8)
 print(tokenizer.decode(output[0]))
 ```
 
-### See Also  
-https://github.com/PanQiWei/AutoGPTQ/blob/main/docs/tutorial/01-Quick-Start.md
+### See Also
+https://github.com/PanQiWei/AutoGPTQ/blob/main/docs/tutorial/01-Quick-Start.md
+
+
+### Benchmark
+
+The results below are preliminary. The blank part is under measurement.
+
+* **Japanese benchmark**
+
+    - *We used [Stability-AI/lm-evaluation-harness](https://github.com/Stability-AI/lm-evaluation-harness/tree/jp-stable) + gptq patch for evaluation.*
+    - *The 4-task average accuracy is based on results of JCommonsenseQA-1.1, JNLI-1.1, MARC-ja-1.1, and JSQuAD-1.1.*
+    - *model loading is performed with gptq_use_triton=True, and evaluation is performed with template version 0.3 using the few-shot in-context learning.*
+    - *The number of few-shots is 3,3,3,2.*
+   
+    | Model | Average | JCommonsenseQA | JNLI | MARC-ja | JSQuAD |
+    | :-- | :-- | :-- | :-- | :-- | :-- |
+    | weblab-10b-instruction-sft | 78.78 | 74.35 | 65.65 | 96.06 | 79.04 |
+    | weblab-10b | 66.38 | 65.86 | 54.19 | 84.49 | 60.98 |
+    | *weblab-10b-instruction-sft-GPTQ* | - | 74.53 | 41.70 | - | 72.69 |