Create README.md

README.md

@@ -0,0 +1,123 @@
+---
+language:
+- ru
+datasets:
+- IlyaGusev/saiga_scored
+- IlyaGusev/saiga_preferences
+license: apache-2.0
+---
+
+
+# Saiga/MistralNemo 12B, Russian fine-tune of Mistral Nemo
+
+Based on [an abliterated version](https://huggingface.co/natong19/Mistral-Nemo-Instruct-2407-abliterated) of [Mistral Nemo](https://huggingface.co/mistralai/Mistral-Nemo-Instruct-2407).
+
+Llama.cpp version: TBD
+
+Colab: [link](https://colab.research.google.com/drive/1qxgIPymzW6_H6s_wwXu3lknkkYM45Db4)
+
+## Prompt format
+
+% if messages[0]['role'] == 'system' %}{% set system_message = messages[0]['content'] | trim + '\n\n' %}{% set messages = messages[1:] %}{% else %}{% set system_message = '' %}{% endif %}{{- bos_token + system_message}}{% for message in messages %}{% if message['role'] == 'user' %}{{ '[INST] ' + message['content'] | trim + ' [/INST]' }}{% elif message['role'] == 'assistant' %}{{ ' ' + message['content'] | trim + eos_token }}{% endif %}{% endfor %}",
+  
+Original Misral Nemo prompt format, but the system prompt is in the beginning:
+```
+<s>Ты — Сайга, русскоязычный автоматический ассистент. Ты разговариваешь с людьми и помогаешь им.
+[INST]Как дела?[/INST]
+Отлично, а у тебя?</s>
+[INST]Шикарно. Как пройти в библиотеку?[/INST] 
+```
+
+## Code example
+```python
+# Исключительно ознакомительный пример.
+# НЕ НАДО ТАК ИНФЕРИТЬ МОДЕЛЬ В ПРОДЕ.
+# См. https://github.com/vllm-project/vllm или https://github.com/huggingface/text-generation-inference
+
+import torch
+from transformers import AutoModelForCausalLM, AutoTokenizer, GenerationConfig
+
+MODEL_NAME = "IlyaGusev/saiga_nemo_12b"
+
+model = AutoModelForCausalLM.from_pretrained(
+    MODEL_NAME,
+    load_in_8bit=True,
+    torch_dtype=torch.bfloat16,
+    device_map="auto"
+)
+model.eval()
+
+tokenizer = AutoTokenizer.from_pretrained(MODEL_NAME)
+generation_config = GenerationConfig.from_pretrained(MODEL_NAME)
+print(generation_config)
+
+inputs = ["Почему трава зеленая?", "Сочини длинный рассказ, обязательно упоминая следующие объекты. Дано: Таня, мяч"]
+for query in inputs:
+    prompt = tokenizer.apply_chat_template([{
+        "role": "user",
+        "content": query
+    }], tokenize=False, add_generation_prompt=True)
+    data = tokenizer(prompt, return_tensors="pt", add_special_tokens=False)
+    data = {k: v.to(model.device) for k, v in data.items()}
+    output_ids = model.generate(**data, generation_config=generation_config)[0]
+    output_ids = output_ids[len(data["input_ids"][0]):]
+    output = tokenizer.decode(output_ids, skip_special_tokens=True).strip()
+    print(query)
+    print(output)
+    print()
+    print("==============================")
+    print()
+```
+
+## Output examples
+```
+User: Почему трава зеленая? 
+Saiga: TBD
+```
+
+```
+User: Сочини длинный рассказ, обязательно упоминая следующие объекты. Дано: Таня, мяч
+Saiga: TBD
+```
+
+
+## Versions
+v1:
+- [87a83ce252ff0142cd4cc918fb3e6a9875ca4638](https://huggingface.co/IlyaGusev/saiga_nemo_12b/commit/87a83ce252ff0142cd4cc918fb3e6a9875ca4638)
+- Other name: saiga_nemo_12b_sft_m9_d14_simpo_m19_d31 
+- SFT dataset config: [sft_d14.json](https://github.com/IlyaGusev/saiga/blob/main/configs/datasets/sft_d14.json)
+- SFT model config: [saiga_nemo_12b_sft_m9.json](https://github.com/IlyaGusev/saiga/blob/main/configs/models/saiga_nemo_12b_sft_m9.json)
+- SimPO dataset config: [pref_d31.json](https://github.com/IlyaGusev/saiga/blob/main/configs/datasets/pref_d31.json)
+- SimPO model config: [saiga_nemo_12b_simpo_m19.json](https://github.com/IlyaGusev/saiga/blob/main/configs/models/saiga_nemo_12b_simpo_m19.json)
+- SFT wandb: [link](https://wandb.ai/ilyagusev/rulm_self_instruct/runs/2ympfu9y)
+- SimPO wandb: [link](https://wandb.ai/ilyagusev/rulm_self_instruct/runs/9zn4825e)
+
+
+## Evaluation
+
+* Dataset: https://github.com/IlyaGusev/rulm/blob/master/self_instruct/data/tasks.jsonl
+* Framework: https://github.com/tatsu-lab/alpaca_eval
+* Evaluator: alpaca_eval_cot_gpt4_turbo_fn
+
+Pivot: chatgpt_3_5_turbo
+| model | length_controlled_winrate |  win_rate | standard_error  | avg_length |
+|-----|-----|-----|-----|-----|
+|chatgpt_4_turbo | 76.04 | 90.00 |1.46 | 1270 |
+|chatgpt_3_5_turbo | 50.00 | 50.00 | 0.00  | 536 |
+|saiga_llama3_8b, v6 | 49.33 | 68.31 | 2.26  | 1262 |
+|sfr-iter-dpo | 49.11 | 74.94 | 2.13 | 1215 |
+|suzume | 49.05 | 71.57 | 2.20  | 1325 |
+|saiga_llama3_8b, v7| 48.95 | 69.40 | 2.25 | 1266 |
+|saiga_llama3_8b, v5  | 47.13  | 66.18 | 2.31 | 1194 |
+|saiga_llama3_8b, v4  | 43.64  | 65.90 | 2.31 | 1200 |
+|saiga_llama3_8b, v3  | 36.97  | 61.08 | 2.38 | 1162 |
+|saiga_llama3_8b, v2  | 33.07  | 48.19 | 2.45 | 1166 |
+|saiga_mistral_7b  | 23.38  | 35.99 | 2.34 | 949  |
+
+Pivot: sfr
+| model | length_controlled_winrate |  win_rate | standard_error  | avg_length |
+|-----|-----|-----|-----|-----|
+| sfr | 50.00 |  50.00 | 0.00 | 1215 |
+| saiga_llama3_8b, v7 |  48.95  |  49.16  | 2.46  | 1266 |
+| saiga_llama3_8b, v6 | 46.91 | 47.23 | 2.45 | 1262 |
+| suzume_8b | 43.69  | 48.19 | 2.46 | 1325 |