Upload folder using huggingface_hub

README.md

@@ -0,0 +1,83 @@
+---
+language:
+- fi
+base_model: LumiOpen/Viking-7B
+license: apache-2.0
+datasets:
+- pinzhenchen/alpaca-cleaned-fi
+---
+# Model Card for Finnish-Viking-Alpaca-V1-7B
+
+This is a merge of [mpasila/Finnish-Viking-Alpaca-V1-LoRA-7B](https://huggingface.co/mpasila/Finnish-Viking-Alpaca-V1-LoRA-7B/).
+
+LoRA trained with text-generation-webui in 4-bit using [LumiOpen/Viking-7B](https://huggingface.co/LumiOpen/Viking-7B/) as the base model for 1 epoch. Dataset used with the LoRA is [pinzhenchen/alpaca-cleaned-fi](https://huggingface.co/datasets/pinzhenchen/alpaca-cleaned-fi/).
+
+It uses Alpaca format but with a translated instruction at the start:
+```
+{
+    "instruction,output": "Alla on ohje, jossa kuvataan tehtävä. Kirjoita vastaus, joka täyttää pyynnön asianmukaisesti.\n\n### Instruction:\n%instruction%\n\n### Response:\n%output%",
+    "instruction,input,output": "Alla on ohje, jossa kuvataan tehtävä ja joka on yhdistetty kontekstia lisäävään syötteeseen. Kirjoita vastaus, joka täyttää pyynnön asianmukaisesti.\n\n### Instruction:\n%instruction%\n\n### Input:\n%input%\n\n### Response:\n%output%"
+}
+
+```
+
+Merged using this [Colab notebook](https://colab.research.google.com/drive/1a76Y21GfPtmVs71Uztlgk2xzPA4_vVjs?usp=sharing). It might not be the best way to merge a quantized LoRA on to a float16 model but I just wanted to quickly do something. You can try merging it better if you want.
+
+## Evaluation
+
+| Model | Size | Type | FIN-bench (score) |
+|-------|------|------|-------|
+| **mpasila/Finnish-Viking-Alpaca-V1-7B** | 7B | Instruct | 0.3943 |
+| [mpasila/NordicAlpaca-Finnish-V1-7B](https://huggingface.co/mpasila/NordicAlpaca-Finnish-V1-7B) | 7B | Instruct | 0.3891 |
+| [Finnish-NLP/llama-7b-finnish-instruct-v0.1](https://huggingface.co/Finnish-NLP/llama-7b-finnish-instruct-v0.1) | 7B | Instruct | **0.4365** |
+| [Finnish-NLP/llama-7b-finnish-instruct-v0.2](https://huggingface.co/Finnish-NLP/llama-7b-finnish-instruct-v0.2) | 7B | Instruct | 0.3993 |
+| [Finnish-NLP/llama-7b-finnish](https://huggingface.co/Finnish-NLP/llama-7b-finnish) | 7B | Base | 0.2350 |
+| [LumiOpen/Viking-7B (1000B)](https://huggingface.co/LumiOpen/Viking-7B) | 7B | Base | 0.3721 |
+| [HPLT/gpt-7b-nordic-prerelease](https://huggingface.co/HPLT/gpt-7b-nordic-prerelease) | 7B | Base | 0.3169 |
+
+[Source](https://docs.google.com/spreadsheets/d/1rqJb9dQVihg-Z1_Ras1L_-wuzPg9xNzpdmM2x5HueeY/edit?usp=sharing)
+
+#### FIN-bench scores:
+
+|                      Task                      |Version|       Metric        |Value |   |Stderr|
+|------------------------------------------------|------:|---------------------|-----:|---|-----:|
+|bigbench_analogies                              |      0|multiple_choice_grade|0.6308|±  |0.0425|
+|bigbench_arithmetic_1_digit_addition            |      0|multiple_choice_grade|0.6400|±  |0.0482|
+|bigbench_arithmetic_1_digit_division            |      0|multiple_choice_grade|0.7391|±  |0.0936|
+|bigbench_arithmetic_1_digit_multiplication      |      0|multiple_choice_grade|0.2800|±  |0.0451|
+|bigbench_arithmetic_1_digit_subtraction         |      0|multiple_choice_grade|0.5000|±  |0.0503|
+|bigbench_arithmetic_2_digit_addition            |      0|multiple_choice_grade|0.1800|±  |0.0386|
+|bigbench_arithmetic_2_digit_division            |      0|multiple_choice_grade|0.4800|±  |0.0502|
+|bigbench_arithmetic_2_digit_multiplication      |      0|multiple_choice_grade|0.0800|±  |0.0273|
+|bigbench_arithmetic_2_digit_subtraction         |      0|multiple_choice_grade|0.2500|±  |0.0435|
+|bigbench_arithmetic_3_digit_addition            |      0|multiple_choice_grade|0.1800|±  |0.0386|
+|bigbench_arithmetic_3_digit_division            |      0|multiple_choice_grade|0.2500|±  |0.0435|
+|bigbench_arithmetic_3_digit_multiplication      |      0|multiple_choice_grade|0.1700|±  |0.0378|
+|bigbench_arithmetic_3_digit_subtraction         |      0|multiple_choice_grade|0.5000|±  |0.0503|
+|bigbench_arithmetic_4_digit_addition            |      0|multiple_choice_grade|0.2600|±  |0.0441|
+|bigbench_arithmetic_4_digit_division            |      0|multiple_choice_grade|0.2500|±  |0.0435|
+|bigbench_arithmetic_4_digit_multiplication      |      0|multiple_choice_grade|0.2100|±  |0.0409|
+|bigbench_arithmetic_4_digit_subtraction         |      0|multiple_choice_grade|0.5200|±  |0.0502|
+|bigbench_arithmetic_5_digit_addition            |      0|multiple_choice_grade|0.3900|±  |0.0490|
+|bigbench_arithmetic_5_digit_division            |      0|multiple_choice_grade|0.1600|±  |0.0368|
+|bigbench_arithmetic_5_digit_multiplication      |      0|multiple_choice_grade|0.1000|±  |0.0302|
+|bigbench_arithmetic_5_digit_subtraction         |      0|multiple_choice_grade|0.6100|±  |0.0490|
+|bigbench_cause_and_effect_one_sentence          |      0|multiple_choice_grade|0.6471|±  |0.0676|
+|bigbench_cause_and_effect_one_sentence_no_prompt|      0|multiple_choice_grade|0.6863|±  |0.0656|
+|bigbench_cause_and_effect_two_sentences         |      0|multiple_choice_grade|0.3922|±  |0.0690|
+|bigbench_emotions                               |      0|multiple_choice_grade|0.2812|±  |0.0357|
+|bigbench_empirical_judgments                    |      0|multiple_choice_grade|0.2828|±  |0.0455|
+|bigbench_general_knowledge                      |      0|multiple_choice_grade|0.4000|±  |0.0590|
+|bigbench_hhh_alignment_harmless                 |      0|multiple_choice_grade|0.3621|±  |0.0637|
+|bigbench_hhh_alignment_helpful                  |      0|multiple_choice_grade|0.3559|±  |0.0629|
+|bigbench_hhh_alignment_honest                   |      0|multiple_choice_grade|0.3729|±  |0.0635|
+|bigbench_hhh_alignment_other                    |      0|multiple_choice_grade|0.5581|±  |0.0766|
+|bigbench_intent_recognition                     |      0|multiple_choice_grade|0.1879|±  |0.0149|
+|bigbench_misconceptions                         |      0|multiple_choice_grade|0.5373|±  |0.0432|
+|bigbench_paraphrase                             |      0|multiple_choice_grade|0.5150|±  |0.0354|
+|bigbench_sentence_ambiguity                     |      0|multiple_choice_grade|0.5000|±  |0.0651|
+|bigbench_similarities_abstraction               |      0|multiple_choice_grade|0.7368|±  |0.0508|
+
+### Framework versions
+
+- PEFT 0.8.2

config.json

@@ -0,0 +1,40 @@
+{
+    "_name_or_path": "LumiOpen/Viking-7B",
+    "architectures": [
+        "LlamaForCausalLM"
+    ],
+    "attention_bias": false,
+    "attention_dropout": 0.0,
+    "bos_token_id": 1,
+    "eos_token_id": 2,
+    "hidden_act": "silu",
+    "hidden_size": 4096,
+    "initializer_range": 0.02,
+    "intermediate_size": 11008,
+    "max_position_embeddings": 4096,
+    "model_type": "llama",
+    "num_attention_heads": 32,
+    "num_hidden_layers": 32,
+    "num_key_value_heads": 32,
+    "pretraining_tp": 1,
+    "rms_norm_eps": 1e-05,
+    "rope_scaling": null,
+    "rope_theta": 10000.0,
+    "tie_word_embeddings": false,
+    "torch_dtype": "float16",
+    "transformers_version": "4.38.2",
+    "untie_embeddings_and_output_weights": true,
+    "use_cache": true,
+    "vocab_size": 131072,
+    "quantization_config": {
+        "quant_method": "exl2",
+        "version": "0.0.17",
+        "bits": 4.0,
+        "head_bits": 6,
+        "calibration": {
+            "rows": 100,
+            "length": 2048,
+            "dataset": "(default)"
+        }
+    }
+}

generation_config.json

@@ -0,0 +1,6 @@
+{
+  "_from_model_config": true,
+  "bos_token_id": 1,
+  "eos_token_id": 2,
+  "transformers_version": "4.38.2"
+}

output.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:4302321ef876d1e23f02d126859c9745ef6684d5e884bf5c75dca0f7d54b5b36
+size 4737670824

special_tokens_map.json

@@ -0,0 +1,52 @@
+{
+  "additional_special_tokens": [
+    "<fim_prefix>",
+    "<fim_middle>",
+    "<fim_suffix>",
+    "<fim_pad>",
+    "<filename>",
+    "<gh_stars>",
+    "<issue_start>",
+    "<issue_comment>",
+    "<issue_closed>",
+    "<jupyter_start>",
+    "<jupyter_text>",
+    "<jupyter_code>",
+    "<jupyter_output>",
+    "<empty_output>",
+    "<commit_before>",
+    "<commit_msg>",
+    "<commit_after>",
+    "<reponame>",
+    "<|im_start|>",
+    "<|im_end|>"
+  ],
+  "bos_token": {
+    "content": "<s>",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  },
+  "eos_token": {
+    "content": "</s>",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  },
+  "pad_token": {
+    "content": "<pad>",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  },
+  "unk_token": {
+    "content": "<unk>",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  }
+}

tokenizer_config.json

@@ -0,0 +1,227 @@
+{
+  "add_prefix_space": false,
+  "added_tokens_decoder": {
+    "0": {
+      "content": "<unk>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "1": {
+      "content": "<s>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "2": {
+      "content": "</s>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "3": {
+      "content": "<pad>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "4": {
+      "content": "<fim_prefix>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "5": {
+      "content": "<fim_middle>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "6": {
+      "content": "<fim_suffix>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "7": {
+      "content": "<fim_pad>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "8": {
+      "content": "<filename>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "9": {
+      "content": "<gh_stars>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "10": {
+      "content": "<issue_start>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "11": {
+      "content": "<issue_comment>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "12": {
+      "content": "<issue_closed>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "13": {
+      "content": "<jupyter_start>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "14": {
+      "content": "<jupyter_text>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "15": {
+      "content": "<jupyter_code>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "16": {
+      "content": "<jupyter_output>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "17": {
+      "content": "<empty_output>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "18": {
+      "content": "<commit_before>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "19": {
+      "content": "<commit_msg>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "20": {
+      "content": "<commit_after>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "21": {
+      "content": "<reponame>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "22": {
+      "content": "<|im_start|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "23": {
+      "content": "<|im_end|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    }
+  },
+  "additional_special_tokens": [
+    "<fim_prefix>",
+    "<fim_middle>",
+    "<fim_suffix>",
+    "<fim_pad>",
+    "<filename>",
+    "<gh_stars>",
+    "<issue_start>",
+    "<issue_comment>",
+    "<issue_closed>",
+    "<jupyter_start>",
+    "<jupyter_text>",
+    "<jupyter_code>",
+    "<jupyter_output>",
+    "<empty_output>",
+    "<commit_before>",
+    "<commit_msg>",
+    "<commit_after>",
+    "<reponame>",
+    "<|im_start|>",
+    "<|im_end|>"
+  ],
+  "bos_token": "<s>",
+  "clean_up_tokenization_spaces": false,
+  "eos_token": "</s>",
+  "model_max_length": 1000000000000000019884624838656,
+  "pad_token": "<pad>",
+  "padding_side": "left",
+  "tokenizer_class": "BloomTokenizer",
+  "unk_token": "<unk>"
+}