togethercomputer
/

RedPajama-INCITE-7B-Base

Text Generation

text-generation-inference

Inference Endpoints

Model card Files Files and versions Community

danfu09 commited on May 5, 2023

Commit

c076b6d

•

1 Parent(s): 870f6da

Update README.md

Files changed (1) hide show

README.md +1 -1

README.md CHANGED Viewed

@@ -121,7 +121,7 @@ Please refer to [togethercomputer/RedPajama-Data-1T](https://huggingface.co/data
 - **Hardware:** 512 nodes of 6xV100 (IBM Power9), on the OLCF Summit cluster
 - **Optimizer:** Apex FusedAdam
-- **Parallelism:** Pipeline parallel 12, model parallel 2
 - **Gradient Accumulations**: 8 (global batch size 4M tokens)
 - **Num of Tokens:** 800B Tokens
 - **Learning rate:** 0.00012

 - **Hardware:** 512 nodes of 6xV100 (IBM Power9), on the OLCF Summit cluster
 - **Optimizer:** Apex FusedAdam
+- **Parallelism:** Pipeline parallel 12, tensor parallel 2
 - **Gradient Accumulations**: 8 (global batch size 4M tokens)
 - **Num of Tokens:** 800B Tokens
 - **Learning rate:** 0.00012