First model version

README.md

@@ -1,3 +1,79 @@
 ---
 license: apache-2.0
 ---
+# **MoLM**
+MoLM is a collection of MoE-based language models ranging in scale from 4 billion to 8 billion parameters. This is the repository for the MoLM-700M-4B pretrained model, converted for the Hugging Face Transformers format. Links to other models can be found in the index at the bottom.
+
+**Model Usage**
+To load the model, you need install the [ModuleFormer package](https://github.com/IBM/ModuleFormer). Then you can load the model with the following code:
+```
+from transformers import AutoTokenizer, AutoModelForCausalLM, AutoConfig, AutoModelForSequenceClassification
+from moduleformer import ModuleFormerForCausalLM, ModuleFormerConfig, ModuleFormerForSequenceClassification
+AutoConfig.register("moduleformer", ModuleFormerConfig)
+AutoModelForCausalLM.register(ModuleFormerConfig, ModuleFormerForCausalLM)
+AutoModelForSequenceClassification.register(ModuleFormerConfig, ModuleFormerForSequenceClassification)
+
+tokenizer = AutoTokenizer.from_pretrained('ibm/MoLM-350M-4B')
+model = AutoModelForCausalLM.from_pretrained('ibm/MoLM-350M-4B')
+```
+
+**Model Details**
+MoLM-350M-4B is a MoE-based language models. It has 4 billion parameters, but each input token only use 350M parameteres during its inference. Thus, it's computationally equivelant to a 350M dense model.
+MoLM-700M-4B has 4 billion parameters and computationally equivelant to a 700M dense model.
+MoLM-700M-8B has 8 billion parameters and computationally equivelant to a 700M dense model.
+Both models are trained on 300 billion tokens from publicly available sources, with a learning rate of 3.0 x 10<sup>-4</sup> and a global batch-size of 3M tokens.
+
+**Model Developers** IBM
+
+**Variations** MoLM comes in two different parameter sizes — 4B and 8B. The 4B models has two variants with different computation cost — 350M and 700M.
+
+**Input** Models input text only.
+
+**Output** Models generate text only.
+
+**Model Architecture** MoLM is an auto-regressive language model that uses the ModuleFormer architecture. It has 16 attention modules in each attention layer and 32 MLP modules in each MLP layer. During inference, in each layer, MoLM-350M-4B and MoLM-700M-8B activate 2 modules for each token, while MoLM-700M-4B activate 4 modules. MoLM-350M-4B and MoLM-700M-4B has 24 blocks and MoLM-700M-8B has 48 blocks.
+
+**Status** This is a static model trained on an offline dataset. Future versions of the tuned models will be released as we improve model safety with community feedback.
+
+**Research Paper** ["ModuleFormer: Modularity Emerges from Mixture-of-Experts"](https://arxiv.org/abs/2306.04640)
+
+## Training Data
+MoLM models are pretrained on 300 billion tokens of data from publicly available sources.
+
+## Evaluation Results
+
+In this section, we report the results for the MoLM models on standard academic benchmarks. For all the evaluations, we use [LM evaluations Harness](https://github.com/EleutherAI/lm-evaluation-harness).
+
+|Model|Latency|Memory|Throughput|Hellaswag|PIQA|ARC-e|ARC-c|OBQA|
+|---|---|---|---|---|---|---|---|---|
+||ms|GB|tokens/sec|acc|acc|acc|acc|acc|
+|Pythia 410M|554|25|59594|33.72|66.70|51.89|21.42|18.2|
+|GPT-Neo 1.3B|991|23|32857|38.66|71.11|56.19|23.12|21.4|
+|Pythia 1.4B|918|42|35559|40.41|70.84|60.52|26.11|22.2|
+|MoLM-350M-4B|497|27|71017|39.21|70.13|56.44|23.55|20.8|
+|GPT-Neo 2.7B|1737|35|18788|42.71|72.2|61.07|27.47|23.2|
+|Pythia 2.8B|2111|70|15522|45.34|73.99|64.35|29.35|23.8|
+|MoLM-700M-4B|863|27|39931|42.20|73.01|60.82|25.94|22.6|
+|MoLM-700M-8B|939|38|37419|43.33|72.91|62.46|27.90|23.8|
+
+|Model| |TriviaQA| | | HumanEval| |Wikitext|
+|---|---|---|---|---|---|---|---|
+||0-shot |1-shot |5-shot |pass@1 |pass@10 |pass@100 |PPL|
+|Pythia 410M |2.32 |5.02 |6.42 |1.20 |3.85 |9.98 |20.09 |
+|GPT-Neo 1.3B |5.24 |8.01 |9.74 |3.62 |6.87 |14.50 |16.16 |
+|Pythia 1.4B |5.30 |9.87 |12.84 |2.19 |7.31 |14.33 |14.71|
+|MoLM-350M-4B |5.40 |11.12 |13.70 |3.04 |6.99 |13.79 |15.15 |
+|GPT-Neo 2.7B |4.82 |11.23 |13.67 |4.89 |9.54 |17.90 |13.93 |
+|Pythia 2.8B |7.38 |15.58 |18.98 |4.91 |11.76 |21.54 |12.68|
+|MoLM-700M-4B|9.07|14.24|16.49|5.50|10.65|20.27|13.20|
+|MoLM-700M-8B |11.47 |16.73 |20.75 |5.51 |12.58 |20.40 |12.97 |
+
+## Ethical Considerations and Limitations
+MoLM is a new technology that carries risks with use. Testing conducted to date has been in English, and has not covered, nor could it cover all scenarios. For these reasons, as with all LLMs, MoLM’s potential outputs cannot be predicted in advance, and the model may in some instances produce inaccurate, biased or other objectionable responses to user prompts. Therefore, before deploying any applications of MoLM, developers should perform safety testing and tuning tailored to their specific applications of the model.
+
+## MoLM Model Index
+|Model|MoLM|
+|---|---|
+|350M-4B| [Link](https://huggingface.co/ibm/MoLM-350M-4B) |
+|700M-4B| [Link](https://huggingface.co/ibm/MoLM-700M-4B) |
+|700M-8B| [Link](https://huggingface.co/ibm/MoLM-700M-8B) |

added_tokens.json

@@ -0,0 +1,40 @@
+{
+  "\t\t": 50294,
+  "\t\t\t": 50293,
+  "\t\t\t\t": 50292,
+  "\t\t\t\t\t": 50291,
+  "\t\t\t\t\t\t": 50290,
+  "\t\t\t\t\t\t\t": 50289,
+  "\t\t\t\t\t\t\t\t": 50288,
+  "\t\t\t\t\t\t\t\t\t": 50287,
+  "  ": 50286,
+  "   ": 50285,
+  "    ": 50284,
+  "     ": 50283,
+  "      ": 50282,
+  "       ": 50281,
+  "        ": 50280,
+  "         ": 50279,
+  "          ": 50278,
+  "           ": 50277,
+  "            ": 50276,
+  "             ": 50275,
+  "              ": 50274,
+  "               ": 50273,
+  "                ": 50272,
+  "                 ": 50271,
+  "                  ": 50270,
+  "                   ": 50269,
+  "                    ": 50268,
+  "                     ": 50267,
+  "                      ": 50266,
+  "                       ": 50265,
+  "                        ": 50264,
+  "                         ": 50263,
+  "                          ": 50262,
+  "                           ": 50261,
+  "                            ": 50260,
+  "                             ": 50259,
+  "                              ": 50258,
+  "                               ": 50257
+}

config.json

@@ -0,0 +1,44 @@
+{
+  "_name_or_path": "./MoLM-700M-4B",
+  "activation_function": "gelu_new",
+  "architectures": [
+    "ModuleFormerForCausalLM"
+  ],
+  "att_func": "stickbreaking",
+  "att_hidden": 1024,
+  "attn_pdrop": 0,
+  "aux_loss_type": "mi",
+  "aux_loss_weight": 0,
+  "block_size": 512,
+  "bos_token_id": 50256,
+  "embd_pdrop": 0,
+  "eos_token_id": 50256,
+  "ffd_hidden": 2048,
+  "gate_type": "mlp",
+  "gating_size": 256,
+  "history_length": 512,
+  "initializer_range": 0.02,
+  "k_att": 4,
+  "k_mlp": 4,
+  "layer_norm_epsilon": 1e-05,
+  "local_size": 1,
+  "model_type": "moduleformer",
+  "moe_pdrop": 0,
+  "moe_type": "moe",
+  "n_att_experts": 16,
+  "n_ctx": 12288,
+  "n_embd": 1024,
+  "n_head": 1,
+  "n_layer": 24,
+  "n_mlp_experts": 32,
+  "pre_norm": true,
+  "resid_pdrop": 0,
+  "sample_topk": 0,
+  "tie_word_embeddings": false,
+  "torch_dtype": "bfloat16",
+  "transformers_version": "4.28.1",
+  "universal": false,
+  "use_cache": true,
+  "vocab_size": 50295,
+  "world_size": null
+}

generation_config.json

@@ -0,0 +1,6 @@
+{
+  "_from_model_config": true,
+  "bos_token_id": 50256,
+  "eos_token_id": 50256,
+  "transformers_version": "4.28.1"
+}

pytorch_model.bin

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:5e08644f7a0d5fe671efea6e788a3b799f645b6a520790362315ef6c7ea99b74
+size 8360677801

special_tokens_map.json

@@ -0,0 +1,5 @@
+{
+  "bos_token": "<|endoftext|>",
+  "eos_token": "<|endoftext|>",
+  "unk_token": "<|endoftext|>"
+}

tokenizer_config.json

@@ -0,0 +1,9 @@
+{
+  "add_prefix_space": false,
+  "bos_token": "<|endoftext|>",
+  "clean_up_tokenization_spaces": true,
+  "eos_token": "<|endoftext|>",
+  "model_max_length": 2048,
+  "tokenizer_class": "CodeGenTokenizer",
+  "unk_token": "<|endoftext|>"
+}