Update README.md - Add model Details
#8
by
Citaman
- opened
README.md
CHANGED
|
@@ -2,8 +2,62 @@
|
|
| 2 |
license: apache-2.0
|
| 3 |
---
|
| 4 |
# Grok-1
|
|
|
|
| 5 |
|
| 6 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 7 |
|
| 8 |
Make sure to download the `int8` checkpoint to the `checkpoints` directory and run
|
| 9 |
|
|
@@ -18,4 +72,4 @@ You should be seeing output from the language model.
|
|
| 18 |
|
| 19 |
Due to the large size of the model (314B parameters), a multi-GPU machine is required to test the model with the example code.
|
| 20 |
|
| 21 |
-
p.s. we're hiring: https://x.ai/career
|
|
|
|
| 2 |
license: apache-2.0
|
| 3 |
---
|
| 4 |
# Grok-1
|
| 5 |
+
_This repository contains the weights of the Grok-1 open-weights model._
|
| 6 |
|
| 7 |
+
╔══════════════════════════╗
|
| 8 |
+
║ _______ ║
|
| 9 |
+
║ /\ |_ _| ║
|
| 10 |
+
║ __ __ / \ | | ║
|
| 11 |
+
║ \ \/ / / /\ \ | | ║
|
| 12 |
+
║ > < / ____ \ _| |_ ║
|
| 13 |
+
║ /_/\_\/_/ \_\_____| ║
|
| 14 |
+
║ ║
|
| 15 |
+
║ Understand the Universe ║
|
| 16 |
+
║ [https://x.ai] ║
|
| 17 |
+
╚════════════╗╔════════════╝
|
| 18 |
+
╔════════╝╚═════════╗
|
| 19 |
+
║ xAI Grok-1 (314B) ║
|
| 20 |
+
╚════════╗╔═════════╝
|
| 21 |
+
╔═════════════════════╝╚═════════════════════╗
|
| 22 |
+
║ 314B parameter Mixture of Experts model ║
|
| 23 |
+
║ - Base model (not finetuned) ║
|
| 24 |
+
║ - 8 experts (2 active) ║
|
| 25 |
+
║ - 86B active parameters ║
|
| 26 |
+
║ - Apache 2.0 license ║
|
| 27 |
+
║ - Code: https://github.com/xai-org/grok-1 ║
|
| 28 |
+
║ - Happy coding! ║
|
| 29 |
+
╚════════════════════════════════════════════╝
|
| 30 |
+
|
| 31 |
+
|
| 32 |
+
## Model Configuration Details
|
| 33 |
+
|
| 34 |
+
**Vocabulary Size**: 131,072
|
| 35 |
+
|
| 36 |
+
**Special Tokens**:
|
| 37 |
+
- Pad Token: 0
|
| 38 |
+
- End of Sequence Token: 2
|
| 39 |
+
|
| 40 |
+
**Sequence Length**: 8192
|
| 41 |
+
|
| 42 |
+
### **Model Architecture**: MoE
|
| 43 |
+
- **Embedding Size**: 6,144
|
| 44 |
+
- **Layers**: 64
|
| 45 |
+
- **Experts**: 8
|
| 46 |
+
- **Selected Experts**: 2
|
| 47 |
+
- **Widening Factor**: 8
|
| 48 |
+
- **Key Size**: 128
|
| 49 |
+
- **Query Heads**: 48
|
| 50 |
+
- **Key Value Heads**: 8
|
| 51 |
+
- **Activation Sharding**: Data-wise, Model-wise
|
| 52 |
+
|
| 53 |
+
### **Inference Configuration**:
|
| 54 |
+
- Batch Size per Device: 0.125
|
| 55 |
+
- Tokenizer: `./tokenizer.model`
|
| 56 |
+
- Local Mesh: 1x8
|
| 57 |
+
- Between Hosts: 1x1
|
| 58 |
+
|
| 59 |
+
|
| 60 |
+
## Inference Details
|
| 61 |
|
| 62 |
Make sure to download the `int8` checkpoint to the `checkpoints` directory and run
|
| 63 |
|
|
|
|
| 72 |
|
| 73 |
Due to the large size of the model (314B parameters), a multi-GPU machine is required to test the model with the example code.
|
| 74 |
|
| 75 |
+
**p.s. we're hiring: https://x.ai/career**
|