Text Generation
Transformers
PyTorch
mosaic_gpt
custom_code
Files changed (1) hide show
  1. README.md +65 -0
README.md ADDED
@@ -0,0 +1,65 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ ---
2
+ license: apache-2.0
3
+ datasets:
4
+ - togethercomputer/RedPajama-Data-1T
5
+ ---
6
+
7
+ # Mosaic-1b-RedPajama-200b
8
+
9
+ Mosaic-1b-RedPajama-200b is a 1.4 billion parameter decoder-only transformer trained on the [RedPajama dataset](https://huggingface.co/datasets/togethercomputer/RedPajama-Data-1T).
10
+ The model was trained for 200B tokens by sampling from the subsets of the RedPajama dataset in the same proportions as were used by the [Llama series of models](https://arxiv.org/abs/2302.13971).
11
+ This model was trained by [MosaicML](https://www.mosaicml.com) and follows the a modified decoder-only transformer architecture.
12
+
13
+ ## Model Date
14
+
15
+ April 19, 2023
16
+
17
+ ## How to Use
18
+
19
+ Note: This model requires that `trust_remote_code=True` be passed to the `from_pretrained` method.
20
+ This is because we train using [FlashAttention (Dao et al. 2022)](https://arxiv.org/pdf/2205.14135.pdf), which is not part of the `transformers` library and depends on [Triton](https://github.com/openai/triton) and some custom PyTorch code.
21
+
22
+ ```python
23
+ import transformers
24
+ model = transformers.AutoModelForCausalLM.from_pretrained('mosaicml/mosaic-llama-redpajama-final-candidate', trust_remote_code=True)```
25
+ ```
26
+
27
+ ## Model Description
28
+
29
+ This model uses the MosaicML LLM codebase, which can be found in the [MosaicML Examples Repository](https://github.com/mosaicml/examples/tree/v0.0.4/examples/llm).
30
+ The architecture is a modification of a standard decoder-only transformer.
31
+ The transformer has 24 layers, 16 attention heads, and width 2048.
32
+ The model has been modified from a standard transformer in the following ways:
33
+ * It uses FlashAttention.
34
+ * It uses ALiBi position encodings.
35
+ * It does not use biases.
36
+
37
+ ## Training Data
38
+
39
+ The model was trained for 200B tokens (batch size 2200, sequence length 2048). It was trained on the following data mix:
40
+ * 67% RedPajama Common Crawl
41
+ * 15% [C4](https://huggingface.co/datasets/c4)
42
+ * 4.5% RedPajama GitHub
43
+ * 4.5% RedPajama Wikipedia
44
+ * 4.5% RedPajama Books
45
+ * 2.5% RedPajama Arxiv
46
+ * 2% RedPajama StackExchange
47
+
48
+ This is the same mix of data as was used in the Llama series of models](https://arxiv.org/abs/2302.13971).
49
+
50
+ Each sample was chosen from one of the datasets, with the dataset selected with the probability specified above.
51
+ The examples were shuffled within each dataset.
52
+ Each example was constructed from as many sequences from that dataset as were necessary to fill the 2048 sequence length.
53
+
54
+ The data was tokenized using the [EleutherAI/gpt-neox-20b](https://huggingface.co/EleutherAI/gpt-neox-20b) tokenizer.
55
+
56
+ ## Training Configuration
57
+
58
+ This model was trained on 440 A100-40GBs for about half a day using the [MosaicML Platform](https://www.mosaicml.com/platform). The model was trained in a data parallel manner using FSDP.
59
+
60
+ ## Acknowledgements
61
+
62
+ This model builds on the work of [Together](https://www.together.xyz), which created the RedPajama dataset with the goal of mimicking the training data used to create the Llama series of models.
63
+ We gratefully acknowledge the hard work of the team that put together this dataset, and we hope this model serves as a useful companion to that work.
64
+
65
+ We also gratefully acknowledge the work of the researchers who created the Llama series of models, which was the impetus for our efforts and those who worked on the RedPajama project.