Hugging Face's logo

tangledgroup
/
tangled-llama-108m-32k-base-v0.1

Text Generation
Transformers
Safetensors
llama
litgpt
litdata
conversational
text-generation-inference
Inference Endpoints
Model card Files Community
mtasic85 commited on
Commit
dc0b6f5
•
1 Parent(s): 05a4d11

cleanup

Browse files
Files changed (9) hide show
  1. README.md +115 -50
  2. out/pretrain/final/evaluate/config.json +0 -3
  3. out/pretrain/final/evaluate/model_config.yaml +0 -33
  4. out/pretrain/final/evaluate/pytorch_model.bin +0 -3
  5. out/pretrain/final/evaluate/results.json +0 -0
  6. out/pretrain/final/evaluate/tokenizer.json +0 -3
  7. out/pretrain/final/evaluate/tokenizer_config.json +0 -3
  8. scripts/TRAIN.md +4 -2
  9. scripts/requirements-lit.in +1 -1
README.md CHANGED
@@ -26,6 +26,12 @@ tags:
26
 
27
  ![logo](./misc/logo.png)
28
 
 
 
 
 
 
 
29
  [loss, val_loss](https://api.wandb.ai/links/mtasic85/m1gxynpw)
30
 
31
  [val_ppl](https://api.wandb.ai/links/mtasic85/xs21d6u2)
@@ -36,83 +42,142 @@ tags:
36
 
37
  ## lm-evaluation-harness
38
 
39
- | Tasks |Version| Filter |n-shot| Metric | |Value | |Stderr|
40
- |---------------------------------------|------:|----------------|-----:|-----------|---|-----:|---|-----:|
41
- |arc_challenge | 1|none | 0|acc |↑ |0.1937|± |0.0115|
42
- | | |none | 0|acc_norm |↑ |0.2363|± |0.0124|
43
- |gsm8k | 3|flexible-extract| 5|exact_match|↑ |0.0136|± |0.0032|
44
- | | |strict-match | 5|exact_match|↑ |0.0000|± |0.0000|
45
- |hellaswag | 1|none | 0|acc |↑ |0.2659|± |0.0044|
46
- | | |none | 0|acc_norm |↑ |0.2709|± |0.0044|
47
- |mmlu | 2|none | |acc |↑ |0.2309|± |0.0036|
48
- | - humanities | 2|none | |acc |↑ |0.2370|± |0.0062|
49
- | - formal_logic | 1|none | 0|acc |↑ |0.2778|± |0.0401|
50
- | - high_school_european_history | 1|none | 0|acc |↑ |0.2303|± |0.0329|
51
- | - high_school_us_history | 1|none | 0|acc |↑ |0.2402|± |0.0300|
52
- | - high_school_world_history | 1|none | 0|acc |↑ |0.2405|± |0.0278|
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
53
  | - international_law | 1|none | 0|acc |↑ |0.1983|± |0.0364|
54
  | - jurisprudence | 1|none | 0|acc |↑ |0.2315|± |0.0408|
55
  | - logical_fallacies | 1|none | 0|acc |↑ |0.1840|± |0.0304|
56
  | - moral_disputes | 1|none | 0|acc |↑ |0.2110|± |0.0220|
57
  | - moral_scenarios | 1|none | 0|acc |↑ |0.2380|± |0.0142|
58
- | - philosophy | 1|none | 0|acc |↑ |0.1994|± |0.0227|
59
  | - prehistory | 1|none | 0|acc |↑ |0.2315|± |0.0235|
60
- | - professional_law | 1|none | 0|acc |↑ |0.2510|± |0.0111|
61
  | - world_religions | 1|none | 0|acc |↑ |0.2865|± |0.0347|
62
- | - other | 2|none | |acc |↑ |0.2372|± |0.0076|
63
  | - business_ethics | 1|none | 0|acc |↑ |0.2900|± |0.0456|
64
  | - clinical_knowledge | 1|none | 0|acc |↑ |0.2113|± |0.0251|
65
- | - college_medicine | 1|none | 0|acc |↑ |0.2023|± |0.0306|
66
  | - global_facts | 1|none | 0|acc |↑ |0.1900|± |0.0394|
67
  | - human_aging | 1|none | 0|acc |↑ |0.3004|± |0.0308|
68
  | - management | 1|none | 0|acc |↑ |0.1748|± |0.0376|
69
  | - marketing | 1|none | 0|acc |↑ |0.2863|± |0.0296|
70
- | - medical_genetics | 1|none | 0|acc |↑ |0.2700|± |0.0446|
71
- | - miscellaneous | 1|none | 0|acc |↑ |0.2337|± |0.0151|
72
  | - nutrition | 1|none | 0|acc |↑ |0.2255|± |0.0239|
73
- | - professional_accounting | 1|none | 0|acc |↑ |0.2411|± |0.0255|
74
  | - professional_medicine | 1|none | 0|acc |↑ |0.1985|± |0.0242|
75
- | - virology | 1|none | 0|acc |↑ |0.2711|± |0.0346|
76
- | - social sciences | 2|none | |acc |↑ |0.2278|± |0.0076|
77
  | - econometrics | 1|none | 0|acc |↑ |0.2105|± |0.0384|
78
- | - high_school_geography | 1|none | 0|acc |↑ |0.1768|± |0.0272|
79
  | - high_school_government_and_politics| 1|none | 0|acc |↑ |0.2280|± |0.0303|
80
- | - high_school_macroeconomics | 1|none | 0|acc |↑ |0.2436|± |0.0218|
81
- | - high_school_microeconomics | 1|none | 0|acc |↑ |0.2395|± |0.0277|
82
- | - high_school_psychology | 1|none | 0|acc |↑ |0.2037|± |0.0173|
83
  | - human_sexuality | 1|none | 0|acc |↑ |0.2595|± |0.0384|
84
- | - professional_psychology | 1|none | 0|acc |↑ |0.2386|± |0.0172|
85
  | - public_relations | 1|none | 0|acc |↑ |0.2091|± |0.0390|
86
- | - security_studies | 1|none | 0|acc |↑ |0.2490|± |0.0277|
87
- | - sociology | 1|none | 0|acc |↑ |0.1990|± |0.0282|
88
  | - us_foreign_policy | 1|none | 0|acc |↑ |0.3100|± |0.0465|
89
- | - stem | 2|none | |acc |↑ |0.2185|± |0.0074|
90
- | - abstract_algebra | 1|none | 0|acc |↑ |0.2600|± |0.0441|
91
  | - anatomy | 1|none | 0|acc |↑ |0.1630|± |0.0319|
92
- | - astronomy | 1|none | 0|acc |↑ |0.2237|± |0.0339|
93
- | - college_biology | 1|none | 0|acc |↑ |0.2708|± |0.0372|
94
- | - college_chemistry | 1|none | 0|acc |↑ |0.2300|± |0.0423|
95
  | - college_computer_science | 1|none | 0|acc |↑ |0.2100|± |0.0409|
96
- | - college_mathematics | 1|none | 0|acc |↑ |0.2200|± |0.0416|
97
- | - college_physics | 1|none | 0|acc |↑ |0.2647|± |0.0439|
98
  | - computer_security | 1|none | 0|acc |↑ |0.3000|± |0.0461|
99
- | - conceptual_physics | 1|none | 0|acc |↑ |0.2000|± |0.0261|
100
- | - electrical_engineering | 1|none | 0|acc |↑ |0.2345|± |0.0353|
101
  | - elementary_mathematics | 1|none | 0|acc |↑ |0.2302|± |0.0217|
102
- | - high_school_biology | 1|none | 0|acc |↑ |0.1903|± |0.0223|
103
  | - high_school_chemistry | 1|none | 0|acc |↑ |0.1527|± |0.0253|
104
- | - high_school_computer_science | 1|none | 0|acc |↑ |0.2700|± |0.0446|
105
- | - high_school_mathematics | 1|none | 0|acc |↑ |0.1926|± |0.0240|
106
  | - high_school_physics | 1|none | 0|acc |↑ |0.2053|± |0.0330|
107
- | - high_school_statistics | 1|none | 0|acc |↑ |0.2130|± |0.0279|
108
  | - machine_learning | 1|none | 0|acc |↑ |0.2768|± |0.0425|
109
- |truthfulqa_mc2 | 2|none | 0|acc |↑ |0.4683|± |0.0160|
110
- |winogrande | 1|none | 0|acc |↑ |0.5075|± |0.0141|
111
 
112
  | Groups |Version|Filter|n-shot|Metric| |Value | |Stderr|
113
  |------------------|------:|------|------|------|---|-----:|---|-----:|
114
- |mmlu | 2|none | |acc |↑ |0.2309|± |0.0036|
115
- | - humanities | 2|none | |acc |↑ |0.2370|± |0.0062|
116
- | - other | 2|none | |acc |↑ |0.2372|± |0.0076|
117
- | - social sciences| 2|none | |acc |↑ |0.2278|± |0.0076|
118
- | - stem | 2|none | |acc |↑ |0.2185|± |0.0074|
 
26
 
27
  ![logo](./misc/logo.png)
28
 
29
+ A pretrained language model based on the Llama model with about **108M** parameters. This model has been trained on **9.7B** (`9,782,206,713`) tokens from more than **5.2M** (`5,285,575`) dataset rows.
30
+
31
+ This model **isn't** designed for immediate use but rather for Continued Pretraining and Finetuning on a downstream task. While it can handle a context length of up to **32K** (`32,768`) tokens, it was pretrained with sequences of **2K** (`2048`) tokens.
32
+
33
+ The objective is to streamline the cognitive or reasoning core, eliminating any redundant knowledge from the model.
34
+
35
  [loss, val_loss](https://api.wandb.ai/links/mtasic85/m1gxynpw)
36
 
37
  [val_ppl](https://api.wandb.ai/links/mtasic85/xs21d6u2)
 
42
 
43
  ## lm-evaluation-harness
44
 
45
+ ```bash
46
+ litgpt evaluate --tasks 'leaderboard' --out_dir 'evaluate-0/' --batch_size 4 --dtype 'bfloat16' out/pretrain/final/
47
+ ```
48
+
49
+ | Tasks |Version|Filter|n-shot| Metric | |Value | |Stderr|
50
+ |-----------------------------------------------------------|-------|------|-----:|-----------------------|---|-----:|---|------|
51
+ |leaderboard | N/A| | | | | | | |
52
+ | - leaderboard_bbh | N/A| | | | | | | |
53
+ | - leaderboard_bbh_boolean_expressions | 1|none | 3|acc_norm |↑ |0.5040|± |0.0317|
54
+ | - leaderboard_bbh_causal_judgement | 1|none | 3|acc_norm |↑ |0.5134|± |0.0366|
55
+ | - leaderboard_bbh_date_understanding | 1|none | 3|acc_norm |↑ |0.1880|± |0.0248|
56
+ | - leaderboard_bbh_disambiguation_qa | 1|none | 3|acc_norm |↑ |0.3000|± |0.0290|
57
+ | - leaderboard_bbh_formal_fallacies | 1|none | 3|acc_norm |↑ |0.4680|± |0.0316|
58
+ | - leaderboard_bbh_geometric_shapes | 1|none | 3|acc_norm |↑ |0.0560|± |0.0146|
59
+ | - leaderboard_bbh_hyperbaton | 1|none | 3|acc_norm |↑ |0.4960|± |0.0317|
60
+ | - leaderboard_bbh_logical_deduction_five_objects | 1|none | 3|acc_norm |↑ |0.1680|± |0.0237|
61
+ | - leaderboard_bbh_logical_deduction_seven_objects | 1|none | 3|acc_norm |↑ |0.1400|± |0.0220|
62
+ | - leaderboard_bbh_logical_deduction_three_objects | 1|none | 3|acc_norm |↑ |0.3360|± |0.0299|
63
+ | - leaderboard_bbh_movie_recommendation | 1|none | 3|acc_norm |↑ |0.2280|± |0.0266|
64
+ | - leaderboard_bbh_navigate | 1|none | 3|acc_norm |↑ |0.4720|± |0.0316|
65
+ | - leaderboard_bbh_object_counting | 1|none | 3|acc_norm |↑ |0.1000|± |0.0190|
66
+ | - leaderboard_bbh_penguins_in_a_table | 1|none | 3|acc_norm |↑ |0.1918|± |0.0327|
67
+ | - leaderboard_bbh_reasoning_about_colored_objects | 1|none | 3|acc_norm |↑ |0.1480|± |0.0225|
68
+ | - leaderboard_bbh_ruin_names | 1|none | 3|acc_norm |↑ |0.2920|± |0.0288|
69
+ | - leaderboard_bbh_salient_translation_error_detection | 1|none | 3|acc_norm |↑ |0.1520|± |0.0228|
70
+ | - leaderboard_bbh_snarks | 1|none | 3|acc_norm |↑ |0.4494|± |0.0374|
71
+ | - leaderboard_bbh_sports_understanding | 1|none | 3|acc_norm |↑ |0.4600|± |0.0316|
72
+ | - leaderboard_bbh_temporal_sequences | 1|none | 3|acc_norm |↑ |0.2480|± |0.0274|
73
+ | - leaderboard_bbh_tracking_shuffled_objects_five_objects | 1|none | 3|acc_norm |↑ |0.1920|± |0.0250|
74
+ | - leaderboard_bbh_tracking_shuffled_objects_seven_objects| 1|none | 3|acc_norm |↑ |0.1560|± |0.0230|
75
+ | - leaderboard_bbh_tracking_shuffled_objects_three_objects| 1|none | 3|acc_norm |↑ |0.3000|± |0.0290|
76
+ | - leaderboard_bbh_web_of_lies | 1|none | 3|acc_norm |↑ |0.5040|± |0.0317|
77
+ | - leaderboard_gpqa | N/A| | | | | | | |
78
+ | - leaderboard_gpqa_diamond | 1|none | 0|acc_norm |↑ |0.2222|± |0.0296|
79
+ | - leaderboard_gpqa_extended | 1|none | 0|acc_norm |↑ |0.2711|± |0.0190|
80
+ | - leaderboard_gpqa_main | 1|none | 0|acc_norm |↑ |0.2589|± |0.0207|
81
+ | - leaderboard_ifeval | 3|none | 0|inst_level_loose_acc |↑ |0.2050|± | N/A|
82
+ | | |none | 0|inst_level_strict_acc |↑ |0.1966|± | N/A|
83
+ | | |none | 0|prompt_level_loose_acc |↑ |0.1072|± |0.0133|
84
+ | | |none | 0|prompt_level_strict_acc|↑ |0.1035|± |0.0131|
85
+ | - leaderboard_math_hard | N/A| | | | | | | |
86
+ | - leaderboard_math_algebra_hard | 1|none | 4|exact_match |↑ |0.0000|± | 0|
87
+ | - leaderboard_math_counting_and_prob_hard | 1|none | 4|exact_match |↑ |0.0000|± | 0|
88
+ | - leaderboard_math_geometry_hard | 1|none | 4|exact_match |↑ |0.0000|± | 0|
89
+ | - leaderboard_math_intermediate_algebra_hard | 1|none | 4|exact_match |↑ |0.0000|± | 0|
90
+ | - leaderboard_math_num_theory_hard | 1|none | 4|exact_match |↑ |0.0000|± | 0|
91
+ | - leaderboard_math_prealgebra_hard | 1|none | 4|exact_match |↑ |0.0000|± | 0|
92
+ | - leaderboard_math_precalculus_hard | 1|none | 4|exact_match |↑ |0.0000|± | 0|
93
+ | - leaderboard_mmlu_pro | 0.1|none | 5|acc |↑ |0.1151|± |0.0029|
94
+ | - leaderboard_musr | N/A| | | | | | | |
95
+ | - leaderboard_musr_murder_mysteries | 1|none | 0|acc_norm |↑ |0.4840|± |0.0317|
96
+ | - leaderboard_musr_object_placements | 1|none | 0|acc_norm |↑ |0.3125|± |0.0290|
97
+ | - leaderboard_musr_team_allocation | 1|none | 0|acc_norm |↑ |0.3840|± |0.0308|
98
+
99
+
100
+ ```bash
101
+ litgpt evaluate --tasks 'hellaswag,gsm8k,truthfulqa_mc2,mmlu,winogrande,arc_challenge' --out_dir 'evaluate-1/' --batch_size 4 --dtype 'bfloat16' out/pretrain/final/
102
+ ```
103
+
104
+ | Tasks |Version| Filter |n-shot| Metric | |Value | |Stderr|
105
+ |---------------------------------------|------:|----------------|-----:|-----------|---|-----:|---|-----:|
106
+ |arc_challenge | 1|none | 0|acc |↑ |0.1911|± |0.0115|
107
+ | | |none | 0|acc_norm |↑ |0.2355|± |0.0124|
108
+ |gsm8k | 3|flexible-extract| 5|exact_match|↑ |0.0152|± |0.0034|
109
+ | | |strict-match | 5|exact_match|↑ |0.0000|± |0.0000|
110
+ |hellaswag | 1|none | 0|acc |↑ |0.2661|± |0.0044|
111
+ | | |none | 0|acc_norm |↑ |0.2708|± |0.0044|
112
+ |mmlu | 2|none | |acc |↑ |0.2315|± |0.0036|
113
+ | - humanities | 2|none | |acc |↑ |0.2372|± |0.0062|
114
+ | - formal_logic | 1|none | 0|acc |↑ |0.2937|± |0.0407|
115
+ | - high_school_european_history | 1|none | 0|acc |↑ |0.2424|± |0.0335|
116
+ | - high_school_us_history | 1|none | 0|acc |↑ |0.2451|± |0.0302|
117
+ | - high_school_world_history | 1|none | 0|acc |↑ |0.2321|± |0.0275|
118
  | - international_law | 1|none | 0|acc |↑ |0.1983|± |0.0364|
119
  | - jurisprudence | 1|none | 0|acc |↑ |0.2315|± |0.0408|
120
  | - logical_fallacies | 1|none | 0|acc |↑ |0.1840|± |0.0304|
121
  | - moral_disputes | 1|none | 0|acc |↑ |0.2110|± |0.0220|
122
  | - moral_scenarios | 1|none | 0|acc |↑ |0.2380|± |0.0142|
123
+ | - philosophy | 1|none | 0|acc |↑ |0.1961|± |0.0226|
124
  | - prehistory | 1|none | 0|acc |↑ |0.2315|± |0.0235|
125
+ | - professional_law | 1|none | 0|acc |↑ |0.2503|± |0.0111|
126
  | - world_religions | 1|none | 0|acc |↑ |0.2865|± |0.0347|
127
+ | - other | 2|none | |acc |↑ |0.2385|± |0.0076|
128
  | - business_ethics | 1|none | 0|acc |↑ |0.2900|± |0.0456|
129
  | - clinical_knowledge | 1|none | 0|acc |↑ |0.2113|± |0.0251|
130
+ | - college_medicine | 1|none | 0|acc |↑ |0.1965|± |0.0303|
131
  | - global_facts | 1|none | 0|acc |↑ |0.1900|± |0.0394|
132
  | - human_aging | 1|none | 0|acc |↑ |0.3004|± |0.0308|
133
  | - management | 1|none | 0|acc |↑ |0.1748|± |0.0376|
134
  | - marketing | 1|none | 0|acc |↑ |0.2863|± |0.0296|
135
+ | - medical_genetics | 1|none | 0|acc |↑ |0.2800|± |0.0451|
136
+ | - miscellaneous | 1|none | 0|acc |↑ |0.2350|± |0.0152|
137
  | - nutrition | 1|none | 0|acc |↑ |0.2255|± |0.0239|
138
+ | - professional_accounting | 1|none | 0|acc |↑ |0.2482|± |0.0258|
139
  | - professional_medicine | 1|none | 0|acc |↑ |0.1985|± |0.0242|
140
+ | - virology | 1|none | 0|acc |↑ |0.2771|± |0.0348|
141
+ | - social sciences | 2|none | |acc |↑ |0.2281|± |0.0076|
142
  | - econometrics | 1|none | 0|acc |↑ |0.2105|± |0.0384|
143
+ | - high_school_geography | 1|none | 0|acc |↑ |0.1818|± |0.0275|
144
  | - high_school_government_and_politics| 1|none | 0|acc |↑ |0.2280|± |0.0303|
145
+ | - high_school_macroeconomics | 1|none | 0|acc |↑ |0.2410|± |0.0217|
146
+ | - high_school_microeconomics | 1|none | 0|acc |↑ |0.2353|± |0.0276|
147
+ | - high_school_psychology | 1|none | 0|acc |↑ |0.2055|± |0.0173|
148
  | - human_sexuality | 1|none | 0|acc |↑ |0.2595|± |0.0384|
149
+ | - professional_psychology | 1|none | 0|acc |↑ |0.2418|± |0.0173|
150
  | - public_relations | 1|none | 0|acc |↑ |0.2091|± |0.0390|
151
+ | - security_studies | 1|none | 0|acc |↑ |0.2408|± |0.0274|
152
+ | - sociology | 1|none | 0|acc |↑ |0.2040|± |0.0285|
153
  | - us_foreign_policy | 1|none | 0|acc |↑ |0.3100|± |0.0465|
154
+ | - stem | 2|none | |acc |↑ |0.2195|± |0.0074|
155
+ | - abstract_algebra | 1|none | 0|acc |↑ |0.2700|± |0.0446|
156
  | - anatomy | 1|none | 0|acc |↑ |0.1630|± |0.0319|
157
+ | - astronomy | 1|none | 0|acc |↑ |0.2303|± |0.0343|
158
+ | - college_biology | 1|none | 0|acc |↑ |0.2569|± |0.0365|
159
+ | - college_chemistry | 1|none | 0|acc |↑ |0.2400|± |0.0429|
160
  | - college_computer_science | 1|none | 0|acc |↑ |0.2100|± |0.0409|
161
+ | - college_mathematics | 1|none | 0|acc |↑ |0.2100|± |0.0409|
162
+ | - college_physics | 1|none | 0|acc |↑ |0.2745|± |0.0444|
163
  | - computer_security | 1|none | 0|acc |↑ |0.3000|± |0.0461|
164
+ | - conceptual_physics | 1|none | 0|acc |↑ |0.1957|± |0.0259|
165
+ | - electrical_engineering | 1|none | 0|acc |↑ |0.2276|± |0.0349|
166
  | - elementary_mathematics | 1|none | 0|acc |↑ |0.2302|± |0.0217|
167
+ | - high_school_biology | 1|none | 0|acc |↑ |0.1968|± |0.0226|
168
  | - high_school_chemistry | 1|none | 0|acc |↑ |0.1527|± |0.0253|
169
+ | - high_school_computer_science | 1|none | 0|acc |↑ |0.2500|± |0.0435|
170
+ | - high_school_mathematics | 1|none | 0|acc |↑ |0.1963|± |0.0242|
171
  | - high_school_physics | 1|none | 0|acc |↑ |0.2053|± |0.0330|
172
+ | - high_school_statistics | 1|none | 0|acc |↑ |0.2269|± |0.0286|
173
  | - machine_learning | 1|none | 0|acc |↑ |0.2768|± |0.0425|
174
+ |truthfulqa_mc2 | 2|none | 0|acc |↑ |0.4681|± |0.0159|
175
+ |winogrande | 1|none | 0|acc |↑ |0.5146|± |0.0140|
176
 
177
  | Groups |Version|Filter|n-shot|Metric| |Value | |Stderr|
178
  |------------------|------:|------|------|------|---|-----:|---|-----:|
179
+ |mmlu | 2|none | |acc |↑ |0.2315|± |0.0036|
180
+ | - humanities | 2|none | |acc |↑ |0.2372|± |0.0062|
181
+ | - other | 2|none | |acc |↑ |0.2385|± |0.0076|
182
+ | - social sciences| 2|none | |acc |↑ |0.2281|± |0.0076|
183
+ | - stem | 2|none | |acc |↑ |0.2195|± |0.0074|
out/pretrain/final/evaluate/config.json DELETED
@@ -1,3 +0,0 @@
1
- version https://git-lfs.github.com/spec/v1
2
- oid sha256:d68796194460465eef693c3a2a4dfc2df61655c8a7e8c9119ac22c07bd9e7f27
3
- size 547
 
 
 
 
out/pretrain/final/evaluate/model_config.yaml DELETED
@@ -1,33 +0,0 @@
1
- attention_logit_softcapping: null
2
- attention_scores_scalar: null
3
- bias: false
4
- block_size: 32768
5
- final_logit_softcapping: null
6
- gelu_approximate: none
7
- head_size: 16
8
- hf_config: {}
9
- intermediate_size: 2048
10
- lm_head_bias: false
11
- mlp_class_name: LLaMAMLP
12
- n_embd: 512
13
- n_expert: 0
14
- n_expert_per_token: 0
15
- n_head: 32
16
- n_layer: 20
17
- n_query_groups: 4
18
- name: ''
19
- norm_class_name: RMSNorm
20
- norm_eps: 1.0e-05
21
- padded_vocab_size: 32768
22
- padding_multiple: 512
23
- parallel_residual: false
24
- post_attention_norm: false
25
- post_mlp_norm: false
26
- rope_base: 500000
27
- rope_condense_ratio: 1
28
- rotary_percentage: 1.0
29
- scale_embeddings: false
30
- shared_attention_norm: false
31
- sliding_window_layer_placing: null
32
- sliding_window_size: null
33
- vocab_size: 32768
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
out/pretrain/final/evaluate/pytorch_model.bin DELETED
@@ -1,3 +0,0 @@
1
- version https://git-lfs.github.com/spec/v1
2
- oid sha256:ab60761622286ea11d7ce9f33d6865aa01e9300c9cd59ebf201b20db15aee8ac
3
- size 216632102
 
 
 
 
out/pretrain/final/evaluate/results.json DELETED
File without changes
out/pretrain/final/evaluate/tokenizer.json DELETED
@@ -1,3 +0,0 @@
1
- version https://git-lfs.github.com/spec/v1
2
- oid sha256:5b496a30dc268bcb8adfd551f693e68e9eadd06b81cab385c088a61e7663649c
3
- size 1368561
 
 
 
 
out/pretrain/final/evaluate/tokenizer_config.json DELETED
@@ -1,3 +0,0 @@
1
- version https://git-lfs.github.com/spec/v1
2
- oid sha256:d6333d68c3280be6081b795cc160fd5872707562021f9889b2e2bd3ae508fa62
3
- size 23043
 
 
 
 
scripts/TRAIN.md CHANGED
@@ -57,7 +57,9 @@ model.save_pretrained('out/converted_model/')
57
  ## Evaluate
58
 
59
  ```bash
60
- # litgpt evaluate --tasks 'hellaswag,gsm8k,truthfulqa_mc2,mmlu,winogrande,arc_challenge' --batch_size 8 out/pretrain/final/
61
 
62
- litgpt evaluate --tasks 'hellaswag,gsm8k,truthfulqa_mc2,mmlu,mmlu_pro,winogrande,arc_challenge,leaderboard,ifeval,mgsm_direct,mathqa,gpqa' --batch_size 8 out/pretrain/final/
 
 
63
  ```
 
57
  ## Evaluate
58
 
59
  ```bash
60
+ litgpt evaluate --tasks 'leaderboard' --out_dir 'evaluate-0/' --batch_size 4 --dtype 'bfloat16' out/pretrain/final/
61
 
62
+ litgpt evaluate --tasks 'hellaswag,gsm8k,truthfulqa_mc2,mmlu,winogrande,arc_challenge' --out_dir 'evaluate-1/' --batch_size 4 --dtype 'bfloat16' out/pretrain/final/
63
+
64
+ litgpt evaluate --tasks 'mmlu_pro,ifeval,mgsm_direct,mathqa,gpqa' --out_dir 'evaluate-2/' --batch_size 4 --dtype 'bfloat16' out/pretrain/final/
65
  ```
scripts/requirements-lit.in CHANGED
@@ -6,6 +6,6 @@ transformers
6
  bitsandbytes
7
  wandb
8
  # litgpt[all]
9
- litgpt[all] @ git+https://github.com/mtasic85/litgpt.git
10
  litdata
11
  grokadamw
 
6
  bitsandbytes
7
  wandb
8
  # litgpt[all]
9
+ litgpt[all] @ git+https://github.com/Lightning-AI/litgpt.git
10
  litdata
11
  grokadamw