gradientai
/

Llama-3-8B-Instruct-Gradient-4194k

Text Generation

text-generation-inference

Inference Endpoints

Model card Files Files and versions Community

Update README.md

#4

by forrest-gradient - opened May 4

base: refs/heads/main

←

from: refs/pr/4

Discussion Files changed

Files changed (1) hide show

README.md +2 -2

README.md CHANGED Viewed

@@ -44,8 +44,8 @@ For training data, we generate long contexts by augmenting [SlimPajama](https://
 | RoPE theta             | 15.3 M    | 207.1 M   | 1.06B     | 2.80B     | 45.2B     |
 | Batch Size             | 1         | 1         | 16         | 16         | 2         |
 | Gradient Accumulation Steps | 32    | 16        | 1         | 1         | 2         |
-| Steps                  | 30        | 24        | 50        | 50        | 11        |
-| Total Tokens           | 62914560  | 100663296 | 419430400 | 838860800 | 184549376 |
 | Learning Rate          | 2.00E-05  | 2.00E-05  | 2.00E-05  | 2.00E-05  | 2.00E-05  |
 | # GPUs                 | 8         | 32        | 512       | 512       | 512       |
 | GPU Type               | NVIDIA L40S | NVIDIA L40S | NVIDIA L40S | NVIDIA L40S | NVIDIA L40S |

 | RoPE theta             | 15.3 M    | 207.1 M   | 1.06B     | 2.80B     | 45.2B     |
 | Batch Size             | 1         | 1         | 16         | 16         | 2         |
 | Gradient Accumulation Steps | 32    | 16        | 1         | 1         | 2         |
+| Steps                  | 30        | 24        | 50        | 50        | 12        |
+| Total Tokens           | 62914560  | 100663296 | 419430400 | 838860800 | 201326592 |
 | Learning Rate          | 2.00E-05  | 2.00E-05  | 2.00E-05  | 2.00E-05  | 2.00E-05  |
 | # GPUs                 | 8         | 32        | 512       | 512       | 512       |
 | GPU Type               | NVIDIA L40S | NVIDIA L40S | NVIDIA L40S | NVIDIA L40S | NVIDIA L40S |