Upload MixtralForCausalLM

README.md

@@ -0,0 +1,201 @@
+---
+library_name: transformers
+tags: []
+---
+
+# Model Card for Model ID
+
+<!-- Provide a quick summary of what the model is/does. -->
+
+
+
+## Model Details
+
+### Model Description
+
+<!-- Provide a longer summary of what this model is. -->
+
+This is the model card of a 🤗 transformers model that has been pushed on the Hub. This model card has been automatically generated.
+
+- **Developed by:** [More Information Needed]
+- **Funded by [optional]:** [More Information Needed]
+- **Shared by [optional]:** [More Information Needed]
+- **Model type:** [More Information Needed]
+- **Language(s) (NLP):** [More Information Needed]
+- **License:** [More Information Needed]
+- **Finetuned from model [optional]:** [More Information Needed]
+
+### Model Sources [optional]
+
+<!-- Provide the basic links for the model. -->
+
+- **Repository:** [More Information Needed]
+- **Paper [optional]:** [More Information Needed]
+- **Demo [optional]:** [More Information Needed]
+
+## Uses
+
+<!-- Address questions around how the model is intended to be used, including the foreseeable users of the model and those affected by the model. -->
+
+### Direct Use
+
+<!-- This section is for the model use without fine-tuning or plugging into a larger ecosystem/app. -->
+
+[More Information Needed]
+
+### Downstream Use [optional]
+
+<!-- This section is for the model use when fine-tuned for a task, or when plugged into a larger ecosystem/app -->
+
+[More Information Needed]
+
+### Out-of-Scope Use
+
+<!-- This section addresses misuse, malicious use, and uses that the model will not work well for. -->
+
+[More Information Needed]
+
+## Bias, Risks, and Limitations
+
+<!-- This section is meant to convey both technical and sociotechnical limitations. -->
+
+[More Information Needed]
+
+### Recommendations
+
+<!-- This section is meant to convey recommendations with respect to the bias, risk, and technical limitations. -->
+
+Users (both direct and downstream) should be made aware of the risks, biases and limitations of the model. More information needed for further recommendations.
+
+## How to Get Started with the Model
+
+Use the code below to get started with the model.
+
+[More Information Needed]
+
+## Training Details
+
+### Training Data
+
+<!-- This should link to a Dataset Card, perhaps with a short stub of information on what the training data is all about as well as documentation related to data pre-processing or additional filtering. -->
+
+[More Information Needed]
+
+### Training Procedure 
+
+<!-- This relates heavily to the Technical Specifications. Content here should link to that section when it is relevant to the training procedure. -->
+
+#### Preprocessing [optional]
+
+[More Information Needed]
+
+
+#### Training Hyperparameters
+
+- **Training regime:** [More Information Needed] <!--fp32, fp16 mixed precision, bf16 mixed precision, bf16 non-mixed precision, fp16 non-mixed precision, fp8 mixed precision -->
+
+#### Speeds, Sizes, Times [optional]
+
+<!-- This section provides information about throughput, start/end time, checkpoint size if relevant, etc. -->
+
+[More Information Needed]
+
+## Evaluation
+
+<!-- This section describes the evaluation protocols and provides the results. -->
+
+### Testing Data, Factors & Metrics
+
+#### Testing Data
+
+<!-- This should link to a Dataset Card if possible. -->
+
+[More Information Needed]
+
+#### Factors
+
+<!-- These are the things the evaluation is disaggregating by, e.g., subpopulations or domains. -->
+
+[More Information Needed]
+
+#### Metrics
+
+<!-- These are the evaluation metrics being used, ideally with a description of why. -->
+
+[More Information Needed]
+
+### Results
+
+[More Information Needed]
+
+#### Summary
+
+
+
+## Model Examination [optional]
+
+<!-- Relevant interpretability work for the model goes here -->
+
+[More Information Needed]
+
+## Environmental Impact
+
+<!-- Total emissions (in grams of CO2eq) and additional considerations, such as electricity usage, go here. Edit the suggested text below accordingly -->
+
+Carbon emissions can be estimated using the [Machine Learning Impact calculator](https://mlco2.github.io/impact#compute) presented in [Lacoste et al. (2019)](https://arxiv.org/abs/1910.09700).
+
+- **Hardware Type:** [More Information Needed]
+- **Hours used:** [More Information Needed]
+- **Cloud Provider:** [More Information Needed]
+- **Compute Region:** [More Information Needed]
+- **Carbon Emitted:** [More Information Needed]
+
+## Technical Specifications [optional]
+
+### Model Architecture and Objective
+
+[More Information Needed]
+
+### Compute Infrastructure
+
+[More Information Needed]
+
+#### Hardware
+
+[More Information Needed]
+
+#### Software
+
+[More Information Needed]
+
+## Citation [optional]
+
+<!-- If there is a paper or blog post introducing the model, the APA and Bibtex information for that should go in this section. -->
+
+**BibTeX:**
+
+[More Information Needed]
+
+**APA:**
+
+[More Information Needed]
+
+## Glossary [optional]
+
+<!-- If relevant, include terms and calculations in this section that can help readers understand the model or model card. -->
+
+[More Information Needed]
+
+## More Information [optional]
+
+[More Information Needed]
+
+## Model Card Authors [optional]
+
+[More Information Needed]
+
+## Model Card Contact
+
+[More Information Needed]
+
+

config.json

@@ -0,0 +1,31 @@
+{
+  "_name_or_path": "mistral-community/Mixtral-8x22B-v0.1",
+  "architectures": [
+    "MixtralForCausalLM"
+  ],
+  "attention_dropout": 0.0,
+  "bos_token_id": 1,
+  "eos_token_id": 2,
+  "hidden_act": "silu",
+  "hidden_size": 6144,
+  "initializer_range": 0.02,
+  "intermediate_size": 16384,
+  "max_position_embeddings": 65536,
+  "model_type": "mixtral",
+  "num_attention_heads": 48,
+  "num_experts_per_tok": 2,
+  "num_hidden_layers": 56,
+  "num_key_value_heads": 8,
+  "num_local_experts": 8,
+  "output_router_logits": false,
+  "rms_norm_eps": 1e-05,
+  "rope_theta": 1000000,
+  "router_aux_loss_coef": 0.001,
+  "router_jitter_noise": 0.0,
+  "sliding_window": null,
+  "tie_word_embeddings": false,
+  "torch_dtype": "float16",
+  "transformers_version": "4.39.3",
+  "use_cache": true,
+  "vocab_size": 32000
+}

generation_config.json

@@ -0,0 +1,6 @@
+{
+  "_from_model_config": true,
+  "bos_token_id": 1,
+  "eos_token_id": 2,
+  "transformers_version": "4.39.3"
+}

model-00001-of-00059.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:da21a4930a736d1df65e61f9985be95ac9fec38b317fcd4063126a6799eb8945
+size 4998663672

model-00002-of-00059.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:abe78f96dc9b1b14630819bd978b549e3649325443e63513e6e38df59913af2e
+size 4806799088

model-00003-of-00059.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:b0127023f59c77f32c62117e4be97621f64df2d55d6c84c037afade4f72fdfec
+size 4806799088

model-00004-of-00059.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:db9ecda4e46d15c05d27f3e3b5c682834ae09b09d24b15dc057134ee2c521683
+size 4806799088

model-00005-of-00059.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:793555ead8dc75c55313e02aa61c27a3e0da277dd942dc1118d3a9f7ef9be940
+size 4806799088

model-00006-of-00059.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:154c4f336a82086cd0e400c8f259f972d50a332df736a33ae6247dbbfc58f978
+size 4806799088

model-00007-of-00059.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:383276fd569779fd2c1e57d54d8e15d7d406be0d7f2d392c1bb7863cfda1f5d1
+size 4806799088

model-00008-of-00059.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:1b6303431cf14c775f2bb00492619f4379708aec18ce661cc6ecbdd63f659c3c
+size 4806799088

model-00009-of-00059.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:53c3add01ab5d85db0162264eb0ca8595b4cef91ab107a3854dc938b426678b0
+size 4806799088

model-00010-of-00059.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:da7687f9487f424713dd60865f3d7b9e194e4a66a0c8e8bc9be8ce7b6e0ac19f
+size 4806799088

model-00011-of-00059.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:e97f3d447651c20289d98c2454716e1dc699c9c807bfccea69c205ab70e6e731
+size 4806799112

model-00012-of-00059.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:644d0b89d18aba51a36294e6a4404189ae2b9de10ab56213f04ab36deeaa7e37
+size 4806799120

model-00013-of-00059.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:b2d5b8be1d79234208ad8c12fcd271846c68e9d14ee4785b84d474dce824d160
+size 4806799120

model-00014-of-00059.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:4f4a5b77bb19465a345fff1dce9a8a6a74a3f43f03dc7583ae13f2b92430dedf
+size 4806799120

model-00015-of-00059.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:f8384d987838d38ed44d8052525fe81202cdc8a82f92bbce2ef40408fd616e48
+size 4806799120

model-00016-of-00059.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:57e5633662da492acb69870e4cce11219fe6ef6635e1a19079df31fdea6327b3
+size 4806799120

model-00017-of-00059.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:5a84aab051ecd8cde8580965f967c6c8be301ee4792a5ceeb8b1258bf75c83a7
+size 4806799120

model-00018-of-00059.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:456f9c711d73c40226b4167bb6a88f470a65dddbc4bfba8fba70b9620d40b1c4
+size 4806799120

model-00019-of-00059.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:d3d961e7b31bf025189173dcfc4f867cb367a7f28932cdd5c8ede82c8ebf0146
+size 4806799120

model-00020-of-00059.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:5c1741f910777227f9b48081de1467a3168cf66757be193514c887c27e1b34b2
+size 4806799120

model-00021-of-00059.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:692baf1d0875e4a80d886097477ebda2acda9a2dac54f92a33ba6f7f2ed7d934
+size 4806799120

model-00022-of-00059.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:8fac567efcccd5b03527d0a7ac81cf241b2ad481535d072e410b7a3d7f8a7001
+size 4806799120

model-00023-of-00059.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:c44e0dd3ea2262c706b5724f5880fb656deb4162f90861cd05a7b6e9f3427ad3
+size 4806799120

model-00024-of-00059.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:45d633300930d3cae8c5f0a9e1c8ca138262aaf8d104d6623bb0ecb128b69540
+size 4932529832

model-00025-of-00059.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:b8ae50bc48d1363b0c899f4f8e402faf560829785c6fbf5e9e30c9c19abc66e1
+size 4995542816

model-00026-of-00059.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:76b83a25f54bb71cfcef8a07d457b2c092c45b0fd2aad83a0d6f2b064846c682
+size 4995542816

model-00027-of-00059.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:84148f266484f93958355de98355825127da971d6b3d940fc9b806fbf36c3f63
+size 4932628256

model-00028-of-00059.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:f4a3b1cce520cb1c32bfa7038b0476479a85470019febb3d69c130ff3ef280bd
+size 4806774312

model-00029-of-00059.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:06888432cec897e48c8343b93c168100e64ef85b205ba09661bf64dc3d40e54e
+size 4806799120

model-00030-of-00059.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:3aa2ff84625fce5b01772f87f9069045851c42413296c47e11b4c66840ef5ff9
+size 4806799120

model-00031-of-00059.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:735c5de881b2c07519ea8f3718c9d349221bdbb472ce7396a386af95efd5f7d4
+size 4806799120

model-00032-of-00059.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:ba06ea156a527f9603a2f6c10099c256e299e1fdbafd22bf2fbb78d0399c6be2
+size 4806799120

model-00033-of-00059.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:9b9171538022939a1645b666533d87c9c27a5997ac7de967ee5af148b46f3211
+size 4806799120

model-00034-of-00059.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:b22becd5f131b3afa15e469db962ce87cc953ae65152b97467a406cdc8e0b1db
+size 4806799120

model-00035-of-00059.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:94b0cb22c3bc35663c47d295389ef487a58d9a866705b5e101a2599d33d85291
+size 4806799120

model-00036-of-00059.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:af2dcaf564e081a684a16e10d7f235aaf03eea20c9ae3b1bddfa75b38e94a657
+size 4806799120

model-00037-of-00059.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:00cb419522bcb530e5c2c61d45af739d8a04b7cf429953766e8b406ecc50dc13
+size 4806799120

model-00038-of-00059.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:21997bf60cc4810baf01911d47dfc4a0dfd50223f2bdaed7f25fae2179f2b493
+size 4806799120

model-00039-of-00059.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:ddbb0da220f460116056e129e46985cec2aece53e79b6b1150b2d25a671f7327
+size 4806799120

model-00040-of-00059.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:a59168997237957ff3010166ab9c2e16603df53e0b1e46398572d816a71b0d7c
+size 4806799120

model-00041-of-00059.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:81d253babe1f5867872b137a76812f44dca52864a798afbd79bbec6155623957
+size 4806799120

model-00042-of-00059.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:fed6e4a0b3f9c50aed76360d965602229ff8a6e7df2497e13f247e4d2b638559
+size 4806799120

model-00043-of-00059.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:02b3543059f5707820984811885c4155a1b3501a3d9fb6bef6334d718cfd7f12
+size 4806799120

model-00044-of-00059.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:d82086c5349167ed0ef106fd8c6a0d82fbb3be0d772a9b7be5754d65d9c9994d
+size 4806799120

model-00045-of-00059.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:54482fed0a531e6c64241d428382463b1064edf9028966b58eb873c95df3d667
+size 4806799120

model-00046-of-00059.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:a339fe9f30f968826fa474eda4a1f27f383eb088ec2084eb8d41b2de8cf86e11
+size 4806799120

model-00047-of-00059.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:b313080f8ee8c269642cfa887aa487121a854070cd99dd660ffea6eba0ab258d
+size 4806799120