readme update

README.md

@@ -1,3 +1,127 @@
 ---
 license: apache-2.0
+language:
+- zh
+- en
+pipeline_tag: text-generation
 ---
+
+<h4 align="center">
+    <p>
+        <b>English</b> |
+        <a href="https://huggingface.co/CofeAI/FLM-101B/blob/main/README_zh.md">简体中文</a> |
+    <p>
+</h4>
+
+
+# FLM-101B
+
+FLM-101B is an open-source decoder-only LLM with 101 billion parameters. During the training process, model growth technique was employed. The model rapidly acquires knowledge on a small-scale model(16B) in the early stages of training and gradually scales up to 101B, resulting in a cost-effective 100B-scale LLM training(costing approximately $100,000).
+FLM-101B supports both Chinese and English languages. It has a context window length of 2048 in training. Thanks to the use of [xPos](https://arxiv.org/pdf/2212.10554.pdf) rotary position embedding, it allows for efficient expansion of the window size during inference.
+
+To advance the development of 100B-scale Large Language Models (LLMs), FLM-101B has now been fully open-sourced.
+
+
+## Why use FLM-101B
+
+- It's an open-source 100B scaled Chinese-English bilingual model.
+- It's the largest known language model trained with xPos.
+- It's the largest known language model that successfully implements μp transfer and loss prediction.
+- It's the largest known language model that successfully implements progressive learning with model growth.
+
+## Quick Started with FLM-101B
+
+```python
+import torch
+from transformers import AutoModelForCausalLM, AutoTokenizer
+tokenizer = AutoTokenizer.from_pretrained("CofeAI/FLM-101B", trust_remote_code=True)
+model = AutoModelForCausalLM.from_pretrained("CofeAI/FLM-101B", torch_dtype=torch.bfloat16, low_cpu_mem_usage=True, device_map="auto", trust_remote_code=True)
+inputs = tokenizer('A house without books is like a body without a soul;', return_tensors='pt').to(model.device)
+generated = model.generate(**inputs, max_new_tokens=64, repetition_penalty=1.1)
+print(tokenizer.decode(generated.cpu()[0], skip_special_tokens=True))
+```
+
+## Model Details
+
+### Model Description
+
+- **Model type:** Decoder-only language model
+- **Language(s) (NLP):** zh, en
+- **License:** apache-2.0
+
+
+<!-- Provide a longer summary of what this model is. -->
+
+### Model size
+
+| Hyperparameter | Value |
+|----------------|-------|
+| n_parameters   | 101B  |
+| n_layers       | 80    |
+| n_heads        | 80    |
+| d_model        | 10240 |
+| vocab size     | 100256|
+| sequence length| 2048  |
+
+
+### Model Architecture and Objective
+
+- **[Extrapolatable Position Embedding (xPos)](https://arxiv.org/pdf/2212.10554.pdf)**
+- **[Flash Attention (In Training)](https://arxiv.org/pdf/2205.14135.pdf)**
+- **[Model Growth](https://arxiv.org/pdf/2305.02869.pdf)**
+- **[Loss Prediction](https://arxiv.org/abs/2304.06875)**
+
+
+
+### Training Details
+
+
+#### Training Hyper Parameter
+| **Hyperparameter** | **16b**           |**51b**           |**101b**           |
+|--------------------|-------------------|------------------|-------------------|
+| Optimizer          |                   | AdamW            |                   |
+| Precision          |                   | bfloat16         |                   |
+| Weight decay       |                   | 0.1              |                   |
+| Gradient clipping  |                   | 1.0              |                   |
+| Learning rate      | 4e-4              | 3.4e-4           | 2e-4              |
+| Batch size(M tokens)|4.72              | 4.72             | 4.31              |
+| Warmup(M samples)  | 4.61              | 0.23             | 0.23              |
+| Time(day)          | 9.63              | 5.37             | 6.54              |
+| Tokens(B)          | 245.37            | 39.64            | 26.54             |
+
+
+#### Parallel Strategies
+
+| **Params(billion)** | **TP Size**  | **PP Size**  | **DP Size**  | **Number of GPUs**  | **Batch Size**  | **TFLOP/s per GPU**  | **GPU Utilization**  |
+|---------------------|--------------|--------------|--------------|---------------------|-----------------|----------------------|----------------------|
+| 16                  | 2            | 1            | 96           | 192                 | 2304            | 162                  | 51.90%               | 
+| 51                  | 4            | 2            | 24           | 192                 | 2304            | 160                  | 51.30%               | 
+| 101                 | 4            | 4            | 12           | 192                 | 2160            | 165                  | 52.88%               | 
+
+#### Hardware
+
+FLM-101B is trained on a cluster of 24 DGX-A800 GPU (8×80G) servers for less than 26 days. Based on the growth strategy, we sequentially completed the model training for sizes 16B, 51B, and 101B on this cluster.
+
+#### Software
+
+FLM-101B was trained with Megatron-FLM(open source soon...) which is modified based on Megatron-LM.
+It uses a 3D(DP+TP+PP) parallelism approach and distributed optimizer.
+
+
+## Bias, Risks, and Limitations
+
+<!-- This section is meant to convey both technical and sociotechnical limitations. -->
+Although we've made extensive efforts to thoroughly clean and filter the training corpus for the model, due to the open nature of the dataset, the model may still have picked up on some unsafe examples. Consequently, the model may still generate unexpected content, including but not limited to discrimination, bias, or offensive language. We would like to strongly advise users not to spread any unsafe content generated by the model. The project developers cannot be held responsible for any repercussions stemming from the dissemination of harmful information.
+
+
+
+
+## Citation
+
+<!-- If there is a paper or blog post introducing the model, the APA and Bibtex information for that should go in this section. -->
+
+
+
+## Contact
+
+tshwangyequan at gmail.com

README_zh.md

@@ -0,0 +1,128 @@
+---
+license: apache-2.0
+language:
+- zh
+- en
+pipeline_tag: text-generation
+---
+
+<h4 align="center">
+    <p>
+        <a href="https://huggingface.co/CofeAI/FLM-101B/blob/main/README.md">English</a> |
+        <b>简体中文</b> |
+    <p>
+</h4>
+
+
+# FLM-101B
+
+FLM-101B是一个开源的decoder-only架构的语言模型,参数规模101B.训练过程采用模型生长技术，通过训练前期在小规模模型上快速学习知识，后期将模型逐步生长成大模型的方式，实现了千亿规模模型的低成本(~$100K)训练。
+FLM-101B支持中英双语，训练上下文窗口长度为2048，得益于使用了xPos旋转位置编码，推理时窗口大小可进行良好的拓展。
+为推动千亿规模LLM技术发展,FLM-101B现已全面开源.
+
+
+## 为什么使用FLM-101B
+
+- 开源的千亿级中英双语模型
+- 已知的最大规模的使用了xPos训练的语言模型
+- 已知的最大规模的成功实现了μp transfer以及loss prediction的语言模型
+- 已知的最大规模的成功实现了progressive learning with model growth的语言模型
+
+## 快速上手FLM-101B
+
+```python
+import torch
+from transformers import AutoModelForCausalLM, AutoTokenizer
+tokenizer = AutoTokenizer.from_pretrained("CofeAI/FLM-101B", trust_remote_code=True)
+model = AutoModelForCausalLM.from_pretrained("CofeAI/FLM-101B", torch_dtype=torch.bfloat16, low_cpu_mem_usage=True, device_map="auto", trust_remote_code=True)
+inputs = tokenizer('一幢没有书的房子，犹如一个没有灵魂的身体;', return_tensors='pt').to(model.device)
+generated = model.generate(**inputs, max_new_tokens=64, repetition_penalty=1.1)
+print(tokenizer.decode(generated.cpu()[0], skip_special_tokens=True))
+```
+
+
+
+
+## 模型细节
+
+### 模型简介
+
+- **模型类型:** 解码器语言模型
+- **支持语言:** 中文/英文
+- **开源协议:** apache-2.0
+
+
+<!-- Provide a longer summary of what this model is. -->
+
+### 模型大小
+
+| Hyperparameter | Value |
+|----------------|-------|
+| n_parameters   | 101B  |
+| n_layers       | 80    |
+| n_heads        | 80    |
+| d_model        | 10240 |
+| vocab size     | 100256|
+| sequence length| 2048  |
+
+
+### 模型架构
+
+- **[Extrapolatable Position Embedding (xPos)](https://arxiv.org/pdf/2212.10554.pdf)**
+- **[Flash Attention (In Training)](https://arxiv.org/pdf/2205.14135.pdf)**
+- **[Model Growth](https://arxiv.org/pdf/2305.02869.pdf)**
+- **[Loss Prediction](https://arxiv.org/abs/2304.06875)**
+
+
+
+### 训练情况
+
+
+#### 训练超参数
+
+| **Hyperparameter** | **16b**           |**51b**           |**101b**           |
+|--------------------|-------------------|------------------|-------------------|
+| Optimizer          |                   | AdamW            |                   |
+| Precision          |                   | bfloat16         |                   |
+| Weight decay       |                   | 0.1              |                   |
+| Gradient clipping  |                   | 1.0              |                   |
+| Learning rate      | 4e-4              | 3.4e-4           | 2e-4              |
+| Batch size(M tokens)|4.72              | 4.72             | 4.31              |
+| Warmup(M samples)  | 4.61              | 0.23             | 0.23              |
+| Time(day)          | 9.63              | 5.37             | 6.54              |
+| Tokens(B)          | 245.37            | 39.64            | 26.54             |
+
+
+#### 并行策略
+
+| **Params(billion)** | **TP Size**  | **PP Size**  | **DP Size**  | **Number of GPUs**  | **Batch Size**  | **TFLOP/s per GPU**  | **GPU Utilization**  |
+|---------------------|--------------|--------------|--------------|---------------------|-----------------|----------------------|----------------------|
+| 16                  | 2            | 1            | 96           | 192                 | 2304            | 162                  | 51.90%               | 
+| 51                  | 4            | 2            | 24           | 192                 | 2304            | 160                  | 51.30%               | 
+| 101                 | 4            | 4            | 12           | 192                 | 2160            | 165                  | 52.88%               | 
+
+#### 硬件
+
+FLM-101B在24节点DGX-A800 GPU(8×80G)集群上完成的训练,总耗时近26天.基于模型生长策略,我们依次在该集群上进行了16B, 51B和101B的模型的训练和生长.
+
+#### 软件
+
+FLM-101B的训练代码Megatron-FLM基于Megatron-LM框架修改,将在近期开源.
+框架支持3D并行策略以及分布式优化器.
+
+## 偏见、风险与限制
+
+<!-- This section is meant to convey both technical and sociotechnical limitations. -->
+
+尽管我们已经尽最大努力对模型训练语料进行了清洗过滤，但由于训练语料的开放性，模型仍有可能在一些不安全的语料上进行过学习。因此模型仍有可能生成不符合预期的文本，包括但不限于歧视、偏见、谩骂等。我们在此提醒模型使用者，请勿传播模型可能生成的不安全内容。由于传播不良信息导致的任何后果，本项目开发者不承担责任。
+
+
+## Citation
+
+<!-- If there is a paper or blog post introducing the model, the APA and Bibtex information for that should go in this section. -->
+
+
+
+## Contact
+
+tshwangyequan at gmail.com