Text Generation
Transformers
PyTorch
mosaic_gpt
custom_code
jfrankle commited on
Commit
e18637d
1 Parent(s): 8a14fde

Create README.md

Browse files
Files changed (1) hide show
  1. README.md +62 -0
README.md ADDED
@@ -0,0 +1,62 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ ---
2
+ license: apache-2.0
3
+ datasets:
4
+ - togethercomputer/RedPajama-Data-1T
5
+ ---
6
+
7
+ # MosaicML-1B-RedPajama-Llama
8
+
9
+ MosaicML-1B-RedPajama-Llama is a 1B parameter decoder-only transformer trained on the [RedPajama dataset](https://huggingface.co/datasets/togethercomputer/RedPajama-Data-1T).
10
+ The model was trained for 200B tokens by sampling from the subsets of the RedPajama dataset in the same proportions as were used by the [Llama series of models](https://arxiv.org/abs/2302.13971).
11
+ This model was trained by [MosaicML](https://www.mosaicml.com) and follows the a modified decoder-only transformer architecture.
12
+
13
+ ## Model Date
14
+
15
+ April 19, 2023
16
+
17
+ ## How to Use
18
+
19
+ Note: This model requires that `trust_remote_code=True` be passed to the `from_pretrained` method.
20
+ This is because we train using [FlashAttention (Dao et al. 2022)](https://arxiv.org/pdf/2205.14135.pdf), which is not part of the `transformers` library and depends on [Triton](https://github.com/openai/triton) and some custom PyTorch code.
21
+
22
+ ```python
23
+ import transformers
24
+ model = transformers.AutoModelForCausalLM.from_pretrained('mosaicml/mosaic-llama-redpajama-final-candidate', trust_remote_code=True)```
25
+ ```
26
+
27
+ ## Model Description
28
+
29
+ This model uses the MosaicML LLM codebase, which can be found in the [MosaicML Examples Repository](https://github.com/mosaicml/examples/tree/v0.0.4/examples/llm).
30
+ The architecture is a modification of a standard decoder-only transformer.
31
+ The transformer has 24 layers, 16 attention heads, and width 2048.
32
+ The model has been modified from a standard transformer in the following ways:
33
+ * It uses FlashAttention.
34
+ * It uses ALiBi position encodings.
35
+ * It does not use biases.
36
+ * It includes layernorm after the keys and queries of the attention operation.
37
+
38
+ ## Training Data
39
+
40
+ The model was trained for 200B tokens (batch size 2200, sequence length 2048). It was trained on the following data mix:
41
+ * 67% RedPajama Common Crawl
42
+ * 15% [C4](https://huggingface.co/datasets/c4)
43
+ * 4.5% RedPajama GitHub
44
+ * 4.5% RedPajama Wikipedia
45
+ * 4.5% RedPajama Books
46
+ * 2.5% RedPajama Arxiv
47
+ * 2% RedPajama StackExchange
48
+
49
+ This is the same mix of data as was used in the Llama series of models](https://arxiv.org/abs/2302.13971).
50
+
51
+ Each sample was chosen from one of the datasets, with the dataset selected with the probability specified above.
52
+ The examples were shuffled within each dataset.
53
+ Each example was constructed from as many sequences from that dataset as were necessary to fill the 2048 sequence length.
54
+
55
+ The data was tokenized using the GPT-NeoX tokenizer.
56
+
57
+ ## Acknowledgements
58
+
59
+ This model builds on the work of [Together](https://www.together.xyz), which created the RedPajama dataset with the goal of mimicking the training data used to create the Llama series of models.
60
+ We gratefully acknowledge the hard work of the team that put together this dataset, and we hope this model serves as a useful companion to that work.
61
+
62
+ We also gratefully acknowledge the work of the researchers who created the Llama series of models, which was the impetus for our efforts and those who worked on the RedPajama project.