Hugging Face's logo

sanchit-gandhi
/
distil-zephyr-1.5b-ssft

Text Generation
Transformers
TensorBoard
Safetensors
mistral
alignment-handbook
Generated from Trainer
trl
sft
conversational
text-generation-inference
Inference Endpoints
Model card Files Metrics Community
sanchit-gandhi HF staff commited on
Commit
fa8bbea
•
1 Parent(s): 895829e

Training in progress, step 600

Browse files
Files changed (7) hide show
  1. model.safetensors +1 -1
  2. runs/Feb01_17-38-02_ip-26-0-165-24/events.out.tfevents.1706809106.ip-26-0-165-24.237059.0 +2 -2
  3. wandb/debug-internal.log +196 -0
  4. wandb/run-20240201_173828-py26nu6m/files/output.log +85 -0
  5. wandb/run-20240201_173828-py26nu6m/files/wandb-summary.json +1 -1
  6. wandb/run-20240201_173828-py26nu6m/logs/debug-internal.log +196 -0
  7. wandb/run-20240201_173828-py26nu6m/run-py26nu6m.wandb +0 -0
model.safetensors CHANGED
@@ -1,3 +1,3 @@
1
  version https://git-lfs.github.com/spec/v1
2
- oid sha256:cf25a398b08458de3374071b7cea8c385537c8f99b8770a28ab8afa8d7d36c2c
3
  size 3141646744
 
1
  version https://git-lfs.github.com/spec/v1
2
+ oid sha256:0dfc5f43bb6f4351d0eec3f0f5b16fc5d10e01761eb48e94e96056ef4bc9a6e8
3
  size 3141646744
runs/Feb01_17-38-02_ip-26-0-165-24/events.out.tfevents.1706809106.ip-26-0-165-24.237059.0 CHANGED
@@ -1,3 +1,3 @@
1
  version https://git-lfs.github.com/spec/v1
2
- oid sha256:a482b66b5a1fb18173d309270fccdcf33806f209221c9f936d35e777f13ad87b
3
- size 20154
 
1
  version https://git-lfs.github.com/spec/v1
2
+ oid sha256:3fbff98bb8fa9aa6501f0e21a4b854cbb237485139925550c0119d0e9b772b1f
3
+ size 23294
wandb/debug-internal.log CHANGED
@@ -1150,3 +1150,199 @@
1150
  2024-02-01 17:48:28,680 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/output.log
1151
  2024-02-01 17:48:28,876 DEBUG SenderThread:237521 [sender.py:send():382] send: stats
1152
  2024-02-01 17:48:28,876 DEBUG HandlerThread:237521 [handler.py:handle_request():146] handle_request: status_report
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1150
  2024-02-01 17:48:28,680 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/output.log
1151
  2024-02-01 17:48:28,876 DEBUG SenderThread:237521 [sender.py:send():382] send: stats
1152
  2024-02-01 17:48:28,876 DEBUG HandlerThread:237521 [handler.py:handle_request():146] handle_request: status_report
1153
+ 2024-02-01 17:48:31,685 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/output.log
1154
+ 2024-02-01 17:48:33,112 DEBUG HandlerThread:237521 [handler.py:handle_request():146] handle_request: partial_history
1155
+ 2024-02-01 17:48:33,114 DEBUG SenderThread:237521 [sender.py:send():382] send: history
1156
+ 2024-02-01 17:48:33,114 DEBUG SenderThread:237521 [sender.py:send_request():409] send_request: summary_record
1157
+ 2024-02-01 17:48:33,122 INFO SenderThread:237521 [sender.py:_save_file():1392] saving file wandb-summary.json with policy end
1158
+ 2024-02-01 17:48:33,691 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/wandb-summary.json
1159
+ 2024-02-01 17:48:33,691 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/output.log
1160
+ 2024-02-01 17:48:34,072 DEBUG HandlerThread:237521 [handler.py:handle_request():146] handle_request: status_report
1161
+ 2024-02-01 17:48:34,693 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/output.log
1162
+ 2024-02-01 17:48:35,788 DEBUG HandlerThread:237521 [handler.py:handle_request():146] handle_request: stop_status
1163
+ 2024-02-01 17:48:35,788 DEBUG HandlerThread:237521 [handler.py:handle_request():146] handle_request: internal_messages
1164
+ 2024-02-01 17:48:35,788 DEBUG SenderThread:237521 [sender.py:send_request():409] send_request: stop_status
1165
+ 2024-02-01 17:48:37,698 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/output.log
1166
+ 2024-02-01 17:48:37,900 DEBUG HandlerThread:237521 [handler.py:handle_request():146] handle_request: partial_history
1167
+ 2024-02-01 17:48:37,901 DEBUG SenderThread:237521 [sender.py:send():382] send: history
1168
+ 2024-02-01 17:48:37,902 DEBUG SenderThread:237521 [sender.py:send_request():409] send_request: summary_record
1169
+ 2024-02-01 17:48:37,904 INFO SenderThread:237521 [sender.py:_save_file():1392] saving file wandb-summary.json with policy end
1170
+ 2024-02-01 17:48:38,700 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/wandb-summary.json
1171
+ 2024-02-01 17:48:39,702 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/output.log
1172
+ 2024-02-01 17:48:39,818 DEBUG HandlerThread:237521 [handler.py:handle_request():146] handle_request: status_report
1173
+ 2024-02-01 17:48:40,703 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/output.log
1174
+ 2024-02-01 17:48:42,697 DEBUG HandlerThread:237521 [handler.py:handle_request():146] handle_request: partial_history
1175
+ 2024-02-01 17:48:42,698 DEBUG SenderThread:237521 [sender.py:send():382] send: history
1176
+ 2024-02-01 17:48:42,699 DEBUG SenderThread:237521 [sender.py:send_request():409] send_request: summary_record
1177
+ 2024-02-01 17:48:42,700 INFO SenderThread:237521 [sender.py:_save_file():1392] saving file wandb-summary.json with policy end
1178
+ 2024-02-01 17:48:42,707 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/wandb-summary.json
1179
+ 2024-02-01 17:48:43,708 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/output.log
1180
+ 2024-02-01 17:48:44,709 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/output.log
1181
+ 2024-02-01 17:48:45,550 DEBUG HandlerThread:237521 [handler.py:handle_request():146] handle_request: status_report
1182
+ 2024-02-01 17:48:47,442 DEBUG HandlerThread:237521 [handler.py:handle_request():146] handle_request: partial_history
1183
+ 2024-02-01 17:48:47,443 DEBUG SenderThread:237521 [sender.py:send():382] send: history
1184
+ 2024-02-01 17:48:47,443 DEBUG SenderThread:237521 [sender.py:send_request():409] send_request: summary_record
1185
+ 2024-02-01 17:48:47,445 INFO SenderThread:237521 [sender.py:_save_file():1392] saving file wandb-summary.json with policy end
1186
+ 2024-02-01 17:48:47,714 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/wandb-summary.json
1187
+ 2024-02-01 17:48:47,714 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/output.log
1188
+ 2024-02-01 17:48:48,715 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/output.log
1189
+ 2024-02-01 17:48:50,788 DEBUG HandlerThread:237521 [handler.py:handle_request():146] handle_request: status_report
1190
+ 2024-02-01 17:48:50,788 DEBUG HandlerThread:237521 [handler.py:handle_request():146] handle_request: stop_status
1191
+ 2024-02-01 17:48:50,788 DEBUG HandlerThread:237521 [handler.py:handle_request():146] handle_request: internal_messages
1192
+ 2024-02-01 17:48:50,788 DEBUG SenderThread:237521 [sender.py:send_request():409] send_request: stop_status
1193
+ 2024-02-01 17:48:51,719 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/output.log
1194
+ 2024-02-01 17:48:52,163 DEBUG HandlerThread:237521 [handler.py:handle_request():146] handle_request: partial_history
1195
+ 2024-02-01 17:48:52,164 DEBUG SenderThread:237521 [sender.py:send():382] send: history
1196
+ 2024-02-01 17:48:52,164 DEBUG SenderThread:237521 [sender.py:send_request():409] send_request: summary_record
1197
+ 2024-02-01 17:48:52,166 INFO SenderThread:237521 [sender.py:_save_file():1392] saving file wandb-summary.json with policy end
1198
+ 2024-02-01 17:48:52,722 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/wandb-summary.json
1199
+ 2024-02-01 17:48:53,723 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/output.log
1200
+ 2024-02-01 17:48:54,724 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/output.log
1201
+ 2024-02-01 17:48:55,950 DEBUG HandlerThread:237521 [handler.py:handle_request():146] handle_request: status_report
1202
+ 2024-02-01 17:48:56,895 DEBUG HandlerThread:237521 [handler.py:handle_request():146] handle_request: partial_history
1203
+ 2024-02-01 17:48:56,897 DEBUG SenderThread:237521 [sender.py:send():382] send: history
1204
+ 2024-02-01 17:48:56,897 DEBUG SenderThread:237521 [sender.py:send_request():409] send_request: summary_record
1205
+ 2024-02-01 17:48:56,899 INFO SenderThread:237521 [sender.py:_save_file():1392] saving file wandb-summary.json with policy end
1206
+ 2024-02-01 17:48:57,730 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/wandb-summary.json
1207
+ 2024-02-01 17:48:57,730 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/output.log
1208
+ 2024-02-01 17:48:58,731 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/output.log
1209
+ 2024-02-01 17:48:58,877 DEBUG SenderThread:237521 [sender.py:send():382] send: stats
1210
+ 2024-02-01 17:49:01,605 DEBUG HandlerThread:237521 [handler.py:handle_request():146] handle_request: status_report
1211
+ 2024-02-01 17:49:01,606 DEBUG HandlerThread:237521 [handler.py:handle_request():146] handle_request: partial_history
1212
+ 2024-02-01 17:49:01,608 DEBUG SenderThread:237521 [sender.py:send():382] send: history
1213
+ 2024-02-01 17:49:01,608 DEBUG SenderThread:237521 [sender.py:send_request():409] send_request: summary_record
1214
+ 2024-02-01 17:49:01,610 INFO SenderThread:237521 [sender.py:_save_file():1392] saving file wandb-summary.json with policy end
1215
+ 2024-02-01 17:49:01,736 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/wandb-summary.json
1216
+ 2024-02-01 17:49:01,736 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/output.log
1217
+ 2024-02-01 17:49:02,737 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/output.log
1218
+ 2024-02-01 17:49:05,741 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/output.log
1219
+ 2024-02-01 17:49:05,980 DEBUG HandlerThread:237521 [handler.py:handle_request():146] handle_request: internal_messages
1220
+ 2024-02-01 17:49:05,980 DEBUG HandlerThread:237521 [handler.py:handle_request():146] handle_request: stop_status
1221
+ 2024-02-01 17:49:05,980 DEBUG SenderThread:237521 [sender.py:send_request():409] send_request: stop_status
1222
+ 2024-02-01 17:49:07,214 DEBUG HandlerThread:237521 [handler.py:handle_request():146] handle_request: status_report
1223
+ 2024-02-01 17:49:07,744 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/output.log
1224
+ 2024-02-01 17:49:08,746 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/output.log
1225
+ 2024-02-01 17:49:12,336 DEBUG HandlerThread:237521 [handler.py:handle_request():146] handle_request: status_report
1226
+ 2024-02-01 17:49:12,751 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/output.log
1227
+ 2024-02-01 17:49:15,755 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/output.log
1228
+ 2024-02-01 17:49:17,828 DEBUG HandlerThread:237521 [handler.py:handle_request():146] handle_request: status_report
1229
+ 2024-02-01 17:49:19,760 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/output.log
1230
+ 2024-02-01 17:49:20,979 DEBUG HandlerThread:237521 [handler.py:handle_request():146] handle_request: stop_status
1231
+ 2024-02-01 17:49:20,979 DEBUG HandlerThread:237521 [handler.py:handle_request():146] handle_request: internal_messages
1232
+ 2024-02-01 17:49:20,980 DEBUG SenderThread:237521 [sender.py:send_request():409] send_request: stop_status
1233
+ 2024-02-01 17:49:21,618 DEBUG HandlerThread:237521 [handler.py:handle_request():146] handle_request: partial_history
1234
+ 2024-02-01 17:49:21,619 DEBUG SenderThread:237521 [sender.py:send():382] send: history
1235
+ 2024-02-01 17:49:21,620 DEBUG SenderThread:237521 [sender.py:send_request():409] send_request: summary_record
1236
+ 2024-02-01 17:49:21,621 INFO SenderThread:237521 [sender.py:_save_file():1392] saving file wandb-summary.json with policy end
1237
+ 2024-02-01 17:49:21,763 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/wandb-summary.json
1238
+ 2024-02-01 17:49:21,763 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/output.log
1239
+ 2024-02-01 17:49:22,765 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/output.log
1240
+ 2024-02-01 17:49:23,450 DEBUG HandlerThread:237521 [handler.py:handle_request():146] handle_request: status_report
1241
+ 2024-02-01 17:49:25,769 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/output.log
1242
+ 2024-02-01 17:49:26,250 DEBUG HandlerThread:237521 [handler.py:handle_request():146] handle_request: partial_history
1243
+ 2024-02-01 17:49:26,251 DEBUG SenderThread:237521 [sender.py:send():382] send: history
1244
+ 2024-02-01 17:49:26,251 DEBUG SenderThread:237521 [sender.py:send_request():409] send_request: summary_record
1245
+ 2024-02-01 17:49:26,253 INFO SenderThread:237521 [sender.py:_save_file():1392] saving file wandb-summary.json with policy end
1246
+ 2024-02-01 17:49:26,771 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/wandb-summary.json
1247
+ 2024-02-01 17:49:27,772 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/output.log
1248
+ 2024-02-01 17:49:28,774 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/output.log
1249
+ 2024-02-01 17:49:28,879 DEBUG SenderThread:237521 [sender.py:send():382] send: stats
1250
+ 2024-02-01 17:49:28,880 DEBUG HandlerThread:237521 [handler.py:handle_request():146] handle_request: status_report
1251
+ 2024-02-01 17:49:30,912 DEBUG HandlerThread:237521 [handler.py:handle_request():146] handle_request: partial_history
1252
+ 2024-02-01 17:49:30,913 DEBUG SenderThread:237521 [sender.py:send():382] send: history
1253
+ 2024-02-01 17:49:30,914 DEBUG SenderThread:237521 [sender.py:send_request():409] send_request: summary_record
1254
+ 2024-02-01 17:49:30,915 INFO SenderThread:237521 [sender.py:_save_file():1392] saving file wandb-summary.json with policy end
1255
+ 2024-02-01 17:49:31,779 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/wandb-summary.json
1256
+ 2024-02-01 17:49:31,779 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/output.log
1257
+ 2024-02-01 17:49:32,780 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/output.log
1258
+ 2024-02-01 17:49:34,646 DEBUG HandlerThread:237521 [handler.py:handle_request():146] handle_request: status_report
1259
+ 2024-02-01 17:49:35,579 DEBUG HandlerThread:237521 [handler.py:handle_request():146] handle_request: partial_history
1260
+ 2024-02-01 17:49:35,580 DEBUG SenderThread:237521 [sender.py:send():382] send: history
1261
+ 2024-02-01 17:49:35,581 DEBUG SenderThread:237521 [sender.py:send_request():409] send_request: summary_record
1262
+ 2024-02-01 17:49:35,583 INFO SenderThread:237521 [sender.py:_save_file():1392] saving file wandb-summary.json with policy end
1263
+ 2024-02-01 17:49:35,785 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/wandb-summary.json
1264
+ 2024-02-01 17:49:35,785 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/output.log
1265
+ 2024-02-01 17:49:35,979 DEBUG HandlerThread:237521 [handler.py:handle_request():146] handle_request: stop_status
1266
+ 2024-02-01 17:49:35,980 DEBUG HandlerThread:237521 [handler.py:handle_request():146] handle_request: internal_messages
1267
+ 2024-02-01 17:49:35,980 DEBUG SenderThread:237521 [sender.py:send_request():409] send_request: stop_status
1268
+ 2024-02-01 17:49:36,786 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/output.log
1269
+ 2024-02-01 17:49:39,790 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/output.log
1270
+ 2024-02-01 17:49:40,251 DEBUG HandlerThread:237521 [handler.py:handle_request():146] handle_request: status_report
1271
+ 2024-02-01 17:49:40,252 DEBUG HandlerThread:237521 [handler.py:handle_request():146] handle_request: partial_history
1272
+ 2024-02-01 17:49:40,253 DEBUG SenderThread:237521 [sender.py:send():382] send: history
1273
+ 2024-02-01 17:49:40,253 DEBUG SenderThread:237521 [sender.py:send_request():409] send_request: summary_record
1274
+ 2024-02-01 17:49:40,256 INFO SenderThread:237521 [sender.py:_save_file():1392] saving file wandb-summary.json with policy end
1275
+ 2024-02-01 17:49:40,792 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/wandb-summary.json
1276
+ 2024-02-01 17:49:41,794 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/output.log
1277
+ 2024-02-01 17:49:42,795 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/output.log
1278
+ 2024-02-01 17:49:44,910 DEBUG HandlerThread:237521 [handler.py:handle_request():146] handle_request: partial_history
1279
+ 2024-02-01 17:49:44,911 DEBUG SenderThread:237521 [sender.py:send():382] send: history
1280
+ 2024-02-01 17:49:44,911 DEBUG SenderThread:237521 [sender.py:send_request():409] send_request: summary_record
1281
+ 2024-02-01 17:49:44,913 INFO SenderThread:237521 [sender.py:_save_file():1392] saving file wandb-summary.json with policy end
1282
+ 2024-02-01 17:49:45,800 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/wandb-summary.json
1283
+ 2024-02-01 17:49:45,800 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/output.log
1284
+ 2024-02-01 17:49:45,841 DEBUG HandlerThread:237521 [handler.py:handle_request():146] handle_request: status_report
1285
+ 2024-02-01 17:49:46,801 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/output.log
1286
+ 2024-02-01 17:49:49,554 DEBUG HandlerThread:237521 [handler.py:handle_request():146] handle_request: partial_history
1287
+ 2024-02-01 17:49:49,555 DEBUG SenderThread:237521 [sender.py:send():382] send: history
1288
+ 2024-02-01 17:49:49,555 DEBUG SenderThread:237521 [sender.py:send_request():409] send_request: summary_record
1289
+ 2024-02-01 17:49:49,557 INFO SenderThread:237521 [sender.py:_save_file():1392] saving file wandb-summary.json with policy end
1290
+ 2024-02-01 17:49:49,806 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/wandb-summary.json
1291
+ 2024-02-01 17:49:49,806 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/output.log
1292
+ 2024-02-01 17:49:50,807 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/output.log
1293
+ 2024-02-01 17:49:50,979 DEBUG HandlerThread:237521 [handler.py:handle_request():146] handle_request: stop_status
1294
+ 2024-02-01 17:49:50,979 DEBUG HandlerThread:237521 [handler.py:handle_request():146] handle_request: internal_messages
1295
+ 2024-02-01 17:49:50,980 DEBUG HandlerThread:237521 [handler.py:handle_request():146] handle_request: status_report
1296
+ 2024-02-01 17:49:50,980 DEBUG SenderThread:237521 [sender.py:send_request():409] send_request: stop_status
1297
+ 2024-02-01 17:49:53,811 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/output.log
1298
+ 2024-02-01 17:49:54,212 DEBUG HandlerThread:237521 [handler.py:handle_request():146] handle_request: partial_history
1299
+ 2024-02-01 17:49:54,213 DEBUG SenderThread:237521 [sender.py:send():382] send: history
1300
+ 2024-02-01 17:49:54,214 DEBUG SenderThread:237521 [sender.py:send_request():409] send_request: summary_record
1301
+ 2024-02-01 17:49:54,216 INFO SenderThread:237521 [sender.py:_save_file():1392] saving file wandb-summary.json with policy end
1302
+ 2024-02-01 17:49:54,813 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/wandb-summary.json
1303
+ 2024-02-01 17:49:55,815 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/output.log
1304
+ 2024-02-01 17:49:56,082 DEBUG HandlerThread:237521 [handler.py:handle_request():146] handle_request: status_report
1305
+ 2024-02-01 17:49:56,816 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/output.log
1306
+ 2024-02-01 17:49:58,880 DEBUG HandlerThread:237521 [handler.py:handle_request():146] handle_request: partial_history
1307
+ 2024-02-01 17:49:58,882 DEBUG SenderThread:237521 [sender.py:send():382] send: stats
1308
+ 2024-02-01 17:49:58,883 DEBUG SenderThread:237521 [sender.py:send():382] send: history
1309
+ 2024-02-01 17:49:58,883 DEBUG SenderThread:237521 [sender.py:send_request():409] send_request: summary_record
1310
+ 2024-02-01 17:49:58,884 INFO SenderThread:237521 [sender.py:_save_file():1392] saving file wandb-summary.json with policy end
1311
+ 2024-02-01 17:49:59,821 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/wandb-summary.json
1312
+ 2024-02-01 17:49:59,821 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/output.log
1313
+ 2024-02-01 17:50:00,822 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/output.log
1314
+ 2024-02-01 17:50:01,678 DEBUG HandlerThread:237521 [handler.py:handle_request():146] handle_request: status_report
1315
+ 2024-02-01 17:50:02,825 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/output.log
1316
+ 2024-02-01 17:50:03,542 DEBUG HandlerThread:237521 [handler.py:handle_request():146] handle_request: partial_history
1317
+ 2024-02-01 17:50:03,543 DEBUG SenderThread:237521 [sender.py:send():382] send: history
1318
+ 2024-02-01 17:50:03,544 DEBUG SenderThread:237521 [sender.py:send_request():409] send_request: summary_record
1319
+ 2024-02-01 17:50:03,546 INFO SenderThread:237521 [sender.py:_save_file():1392] saving file wandb-summary.json with policy end
1320
+ 2024-02-01 17:50:03,827 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/wandb-summary.json
1321
+ 2024-02-01 17:50:04,828 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/output.log
1322
+ 2024-02-01 17:50:05,979 DEBUG HandlerThread:237521 [handler.py:handle_request():146] handle_request: stop_status
1323
+ 2024-02-01 17:50:05,979 DEBUG HandlerThread:237521 [handler.py:handle_request():146] handle_request: internal_messages
1324
+ 2024-02-01 17:50:05,980 DEBUG SenderThread:237521 [sender.py:send_request():409] send_request: stop_status
1325
+ 2024-02-01 17:50:06,831 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/output.log
1326
+ 2024-02-01 17:50:07,275 DEBUG HandlerThread:237521 [handler.py:handle_request():146] handle_request: status_report
1327
+ 2024-02-01 17:50:08,210 DEBUG HandlerThread:237521 [handler.py:handle_request():146] handle_request: partial_history
1328
+ 2024-02-01 17:50:08,211 DEBUG SenderThread:237521 [sender.py:send():382] send: history
1329
+ 2024-02-01 17:50:08,212 DEBUG SenderThread:237521 [sender.py:send_request():409] send_request: summary_record
1330
+ 2024-02-01 17:50:08,213 INFO SenderThread:237521 [sender.py:_save_file():1392] saving file wandb-summary.json with policy end
1331
+ 2024-02-01 17:50:08,835 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/wandb-summary.json
1332
+ 2024-02-01 17:50:08,835 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/output.log
1333
+ 2024-02-01 17:50:10,837 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/output.log
1334
+ 2024-02-01 17:50:12,840 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/output.log
1335
+ 2024-02-01 17:50:12,871 DEBUG HandlerThread:237521 [handler.py:handle_request():146] handle_request: status_report
1336
+ 2024-02-01 17:50:12,872 DEBUG HandlerThread:237521 [handler.py:handle_request():146] handle_request: partial_history
1337
+ 2024-02-01 17:50:12,873 DEBUG SenderThread:237521 [sender.py:send():382] send: history
1338
+ 2024-02-01 17:50:12,873 DEBUG SenderThread:237521 [sender.py:send_request():409] send_request: summary_record
1339
+ 2024-02-01 17:50:12,875 INFO SenderThread:237521 [sender.py:_save_file():1392] saving file wandb-summary.json with policy end
1340
+ 2024-02-01 17:50:13,842 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/wandb-summary.json
1341
+ 2024-02-01 17:50:14,843 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/output.log
1342
+ 2024-02-01 17:50:16,846 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/output.log
1343
+ 2024-02-01 17:50:17,540 DEBUG HandlerThread:237521 [handler.py:handle_request():146] handle_request: partial_history
1344
+ 2024-02-01 17:50:17,541 DEBUG SenderThread:237521 [sender.py:send():382] send: history
1345
+ 2024-02-01 17:50:17,541 DEBUG SenderThread:237521 [sender.py:send_request():409] send_request: summary_record
1346
+ 2024-02-01 17:50:17,543 INFO SenderThread:237521 [sender.py:_save_file():1392] saving file wandb-summary.json with policy end
1347
+ 2024-02-01 17:50:17,848 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/wandb-summary.json
1348
+ 2024-02-01 17:50:18,469 DEBUG HandlerThread:237521 [handler.py:handle_request():146] handle_request: status_report
wandb/run-20240201_173828-py26nu6m/files/output.log CHANGED
@@ -460,3 +460,88 @@
460
 
461
  52%|█████████████████████████████████████████▍ | 564/1090 [09:51<08:01, 1.09it/s]
462
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
460
 
461
  52%|█████████████████████████████████████████▍ | 564/1090 [09:51<08:01, 1.09it/s]
462
 
463
+
464
+
465
+ 52%|█████████████████████████████████████████▊ | 569/1090 [09:56<07:57, 1.09it/s]
466
+
467
+
468
+
469
+ 53%|██████████████████████████████████████████▏ | 575/1090 [10:03<08:41, 1.01s/it]
470
+
471
+
472
+ 53%|██████████████████████████████████████████▍ | 579/1090 [10:07<08:16, 1.03it/s]
473
+
474
+
475
+ 53%|██████████████████████████████████████████▊ | 583/1090 [10:11<08:05, 1.04it/s]
476
+
477
+
478
+
479
+ 54%|███████████████████████████████████████████▎ | 590/1090 [10:17<07:52, 1.06it/s]
480
+
481
+
482
+ 54%|███████████████████████████████████████████▌ | 594/1090 [10:21<07:50, 1.06it/s]
483
+
484
+
485
+ 55%|███████████████████████████████████████████▉ | 598/1090 [10:25<07:43, 1.06it/s]
486
+
487
+ 55%|████████████████████████████████████████████ | 600/1090 [10:27<07:42, 1.06it/s][INFO|trainer.py:2889] 2024-02-01 17:49:02,702 >> Saving model checkpoint to ./tmp-checkpoint-600
488
+ [INFO|configuration_utils.py:483] 2024-02-01 17:49:02,706 >> Configuration saved in ./tmp-checkpoint-600/config.json
489
+ [INFO|configuration_utils.py:594] 2024-02-01 17:49:02,708 >> Configuration saved in ./tmp-checkpoint-600/generation_config.json
490
+ [INFO|modeling_utils.py:2382] 2024-02-01 17:49:05,978 >> Model weights saved in ./tmp-checkpoint-600/pytorch_model.bin
491
+ [INFO|tokenization_utils_base.py:2432] 2024-02-01 17:49:05,993 >> tokenizer config file saved in ./tmp-checkpoint-600/tokenizer_config.json
492
+ [INFO|tokenization_utils_base.py:2441] 2024-02-01 17:49:05,996 >> Special tokens file saved in ./tmp-checkpoint-600/special_tokens_map.json
493
+ /fsx/sanchit/miniconda3/envs/venv/lib/python3.11/site-packages/torch/nn/modules/module.py:1879: UserWarning: Positional args are being deprecated, use kwargs instead. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details.
494
+ warnings.warn(
495
+ [2024-02-01 17:49:06,062] [INFO] [logging.py:96:log_dist] [Rank 0] [Torch] Checkpoint global_step600 is about to be saved!
496
+ [2024-02-01 17:49:06,071] [INFO] [logging.py:96:log_dist] [Rank 0] Saving model checkpoint: ./tmp-checkpoint-600/global_step600/zero_pp_rank_0_mp_rank_00_model_states.pt
497
+ [2024-02-01 17:49:06,071] [INFO] [torch_checkpoint_engine.py:21:save] [Torch] Saving ./tmp-checkpoint-600/global_step600/zero_pp_rank_0_mp_rank_00_model_states.pt...
498
+ [2024-02-01 17:49:06,208] [INFO] [torch_checkpoint_engine.py:23:save] [Torch] Saved ./tmp-checkpoint-600/global_step600/zero_pp_rank_0_mp_rank_00_model_states.pt.
499
+ [2024-02-01 17:49:06,213] [INFO] [torch_checkpoint_engine.py:21:save] [Torch] Saving ./tmp-checkpoint-600/global_step600/bf16_zero_pp_rank_0_mp_rank_00_optim_states.pt...
500
+ [2024-02-01 17:49:10,329] [INFO] [torch_checkpoint_engine.py:23:save] [Torch] Saved ./tmp-checkpoint-600/global_step600/bf16_zero_pp_rank_0_mp_rank_00_optim_states.pt.
501
+ [2024-02-01 17:49:10,334] [INFO] [engine.py:3393:_save_zero_checkpoint] zero checkpoint saved ./tmp-checkpoint-600/global_step600/bf16_zero_pp_rank_0_mp_rank_00_optim_states.pt
502
+ [2024-02-01 17:49:10,335] [INFO] [torch_checkpoint_engine.py:33:commit] [Torch] Checkpoint global_step600 is ready now!
503
+ [INFO|tokenization_utils_base.py:2432] 2024-02-01 17:49:13,152 >> tokenizer config file saved in ./tokenizer_config.json
504
+ [INFO|tokenization_utils_base.py:2441] 2024-02-01 17:49:13,155 >> Special tokens file saved in ./special_tokens_map.json
505
+ [INFO|trainer.py:2979] 2024-02-01 17:49:13,184 >> Deleting older checkpoint [checkpoint-500] due to args.save_total_limit
506
+
507
+ 55%|████████████████████████████████████████████▎ | 603/1090 [10:45<25:52, 3.19s/it]
508
+
509
+
510
+
511
+ 56%|████████████████████████████████████████████▊ | 610/1090 [10:51<08:57, 1.12s/it]
512
+
513
+
514
+ 56%|█████████████████████████████████████████████ | 614/1090 [10:55<07:45, 1.02it/s]
515
+
516
+
517
+ 57%|█████████████████████████████████████████████▎ | 618/1090 [10:59<07:25, 1.06it/s]
518
+
519
+
520
+
521
+ 57%|█████████████████████████████████████████████▊ | 625/1090 [11:05<07:14, 1.07it/s]
522
+
523
+
524
+ 58%|██████████████████████████████████████████████▏ | 629/1090 [11:09<07:09, 1.07it/s]
525
+
526
+
527
+ 58%|██████████████████████████████████████████████▍ | 633/1090 [11:13<07:05, 1.07it/s]
528
+
529
+
530
+
531
+ 59%|██████████████████████████████████████████████▉ | 640/1090 [11:19<06:59, 1.07it/s]
532
+
533
+
534
+ 59%|███████████████████████████████████████████████▎ | 644/1090 [11:23<06:55, 1.07it/s]
535
+
536
+
537
+ 59%|███████████████████████████████████████████████▌ | 648/1090 [11:27<06:52, 1.07it/s]
538
+
539
+
540
+
541
+ 60%|████████████████████████████████████████████████ | 655/1090 [11:33<06:45, 1.07it/s]
542
+
543
+
544
+ 60%|████████████████████████████████████████████████▎ | 659/1090 [11:37<06:41, 1.07it/s]
545
+
546
+
547
+ 61%|████████████████████████████████████████████████▋ | 663/1090 [11:41<06:38, 1.07it/s]
wandb/run-20240201_173828-py26nu6m/files/wandb-summary.json CHANGED
@@ -1 +1 @@
1
- {"train/loss": 2.9132, "train/learning_rate": 1.1102595071354471e-05, "train/epoch": 0.52, "train/global_step": 565, "_timestamp": 1706809707.2731094, "_runtime": 598.8212354183197, "_step": 113}
 
1
+ {"train/loss": 2.7137, "train/learning_rate": 7.917750740470116e-06, "train/epoch": 0.61, "train/global_step": 665, "_timestamp": 1706809817.5397475, "_runtime": 709.0878734588623, "_step": 133}
wandb/run-20240201_173828-py26nu6m/logs/debug-internal.log CHANGED
@@ -1150,3 +1150,199 @@
1150
  2024-02-01 17:48:28,680 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/output.log
1151
  2024-02-01 17:48:28,876 DEBUG SenderThread:237521 [sender.py:send():382] send: stats
1152
  2024-02-01 17:48:28,876 DEBUG HandlerThread:237521 [handler.py:handle_request():146] handle_request: status_report
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1150
  2024-02-01 17:48:28,680 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/output.log
1151
  2024-02-01 17:48:28,876 DEBUG SenderThread:237521 [sender.py:send():382] send: stats
1152
  2024-02-01 17:48:28,876 DEBUG HandlerThread:237521 [handler.py:handle_request():146] handle_request: status_report
1153
+ 2024-02-01 17:48:31,685 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/output.log
1154
+ 2024-02-01 17:48:33,112 DEBUG HandlerThread:237521 [handler.py:handle_request():146] handle_request: partial_history
1155
+ 2024-02-01 17:48:33,114 DEBUG SenderThread:237521 [sender.py:send():382] send: history
1156
+ 2024-02-01 17:48:33,114 DEBUG SenderThread:237521 [sender.py:send_request():409] send_request: summary_record
1157
+ 2024-02-01 17:48:33,122 INFO SenderThread:237521 [sender.py:_save_file():1392] saving file wandb-summary.json with policy end
1158
+ 2024-02-01 17:48:33,691 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/wandb-summary.json
1159
+ 2024-02-01 17:48:33,691 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/output.log
1160
+ 2024-02-01 17:48:34,072 DEBUG HandlerThread:237521 [handler.py:handle_request():146] handle_request: status_report
1161
+ 2024-02-01 17:48:34,693 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/output.log
1162
+ 2024-02-01 17:48:35,788 DEBUG HandlerThread:237521 [handler.py:handle_request():146] handle_request: stop_status
1163
+ 2024-02-01 17:48:35,788 DEBUG HandlerThread:237521 [handler.py:handle_request():146] handle_request: internal_messages
1164
+ 2024-02-01 17:48:35,788 DEBUG SenderThread:237521 [sender.py:send_request():409] send_request: stop_status
1165
+ 2024-02-01 17:48:37,698 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/output.log
1166
+ 2024-02-01 17:48:37,900 DEBUG HandlerThread:237521 [handler.py:handle_request():146] handle_request: partial_history
1167
+ 2024-02-01 17:48:37,901 DEBUG SenderThread:237521 [sender.py:send():382] send: history
1168
+ 2024-02-01 17:48:37,902 DEBUG SenderThread:237521 [sender.py:send_request():409] send_request: summary_record
1169
+ 2024-02-01 17:48:37,904 INFO SenderThread:237521 [sender.py:_save_file():1392] saving file wandb-summary.json with policy end
1170
+ 2024-02-01 17:48:38,700 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/wandb-summary.json
1171
+ 2024-02-01 17:48:39,702 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/output.log
1172
+ 2024-02-01 17:48:39,818 DEBUG HandlerThread:237521 [handler.py:handle_request():146] handle_request: status_report
1173
+ 2024-02-01 17:48:40,703 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/output.log
1174
+ 2024-02-01 17:48:42,697 DEBUG HandlerThread:237521 [handler.py:handle_request():146] handle_request: partial_history
1175
+ 2024-02-01 17:48:42,698 DEBUG SenderThread:237521 [sender.py:send():382] send: history
1176
+ 2024-02-01 17:48:42,699 DEBUG SenderThread:237521 [sender.py:send_request():409] send_request: summary_record
1177
+ 2024-02-01 17:48:42,700 INFO SenderThread:237521 [sender.py:_save_file():1392] saving file wandb-summary.json with policy end
1178
+ 2024-02-01 17:48:42,707 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/wandb-summary.json
1179
+ 2024-02-01 17:48:43,708 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/output.log
1180
+ 2024-02-01 17:48:44,709 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/output.log
1181
+ 2024-02-01 17:48:45,550 DEBUG HandlerThread:237521 [handler.py:handle_request():146] handle_request: status_report
1182
+ 2024-02-01 17:48:47,442 DEBUG HandlerThread:237521 [handler.py:handle_request():146] handle_request: partial_history
1183
+ 2024-02-01 17:48:47,443 DEBUG SenderThread:237521 [sender.py:send():382] send: history
1184
+ 2024-02-01 17:48:47,443 DEBUG SenderThread:237521 [sender.py:send_request():409] send_request: summary_record
1185
+ 2024-02-01 17:48:47,445 INFO SenderThread:237521 [sender.py:_save_file():1392] saving file wandb-summary.json with policy end
1186
+ 2024-02-01 17:48:47,714 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/wandb-summary.json
1187
+ 2024-02-01 17:48:47,714 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/output.log
1188
+ 2024-02-01 17:48:48,715 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/output.log
1189
+ 2024-02-01 17:48:50,788 DEBUG HandlerThread:237521 [handler.py:handle_request():146] handle_request: status_report
1190
+ 2024-02-01 17:48:50,788 DEBUG HandlerThread:237521 [handler.py:handle_request():146] handle_request: stop_status
1191
+ 2024-02-01 17:48:50,788 DEBUG HandlerThread:237521 [handler.py:handle_request():146] handle_request: internal_messages
1192
+ 2024-02-01 17:48:50,788 DEBUG SenderThread:237521 [sender.py:send_request():409] send_request: stop_status
1193
+ 2024-02-01 17:48:51,719 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/output.log
1194
+ 2024-02-01 17:48:52,163 DEBUG HandlerThread:237521 [handler.py:handle_request():146] handle_request: partial_history
1195
+ 2024-02-01 17:48:52,164 DEBUG SenderThread:237521 [sender.py:send():382] send: history
1196
+ 2024-02-01 17:48:52,164 DEBUG SenderThread:237521 [sender.py:send_request():409] send_request: summary_record
1197
+ 2024-02-01 17:48:52,166 INFO SenderThread:237521 [sender.py:_save_file():1392] saving file wandb-summary.json with policy end
1198
+ 2024-02-01 17:48:52,722 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/wandb-summary.json
1199
+ 2024-02-01 17:48:53,723 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/output.log
1200
+ 2024-02-01 17:48:54,724 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/output.log
1201
+ 2024-02-01 17:48:55,950 DEBUG HandlerThread:237521 [handler.py:handle_request():146] handle_request: status_report
1202
+ 2024-02-01 17:48:56,895 DEBUG HandlerThread:237521 [handler.py:handle_request():146] handle_request: partial_history
1203
+ 2024-02-01 17:48:56,897 DEBUG SenderThread:237521 [sender.py:send():382] send: history
1204
+ 2024-02-01 17:48:56,897 DEBUG SenderThread:237521 [sender.py:send_request():409] send_request: summary_record
1205
+ 2024-02-01 17:48:56,899 INFO SenderThread:237521 [sender.py:_save_file():1392] saving file wandb-summary.json with policy end
1206
+ 2024-02-01 17:48:57,730 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/wandb-summary.json
1207
+ 2024-02-01 17:48:57,730 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/output.log
1208
+ 2024-02-01 17:48:58,731 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/output.log
1209
+ 2024-02-01 17:48:58,877 DEBUG SenderThread:237521 [sender.py:send():382] send: stats
1210
+ 2024-02-01 17:49:01,605 DEBUG HandlerThread:237521 [handler.py:handle_request():146] handle_request: status_report
1211
+ 2024-02-01 17:49:01,606 DEBUG HandlerThread:237521 [handler.py:handle_request():146] handle_request: partial_history
1212
+ 2024-02-01 17:49:01,608 DEBUG SenderThread:237521 [sender.py:send():382] send: history
1213
+ 2024-02-01 17:49:01,608 DEBUG SenderThread:237521 [sender.py:send_request():409] send_request: summary_record
1214
+ 2024-02-01 17:49:01,610 INFO SenderThread:237521 [sender.py:_save_file():1392] saving file wandb-summary.json with policy end
1215
+ 2024-02-01 17:49:01,736 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/wandb-summary.json
1216
+ 2024-02-01 17:49:01,736 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/output.log
1217
+ 2024-02-01 17:49:02,737 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/output.log
1218
+ 2024-02-01 17:49:05,741 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/output.log
1219
+ 2024-02-01 17:49:05,980 DEBUG HandlerThread:237521 [handler.py:handle_request():146] handle_request: internal_messages
1220
+ 2024-02-01 17:49:05,980 DEBUG HandlerThread:237521 [handler.py:handle_request():146] handle_request: stop_status
1221
+ 2024-02-01 17:49:05,980 DEBUG SenderThread:237521 [sender.py:send_request():409] send_request: stop_status
1222
+ 2024-02-01 17:49:07,214 DEBUG HandlerThread:237521 [handler.py:handle_request():146] handle_request: status_report
1223
+ 2024-02-01 17:49:07,744 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/output.log
1224
+ 2024-02-01 17:49:08,746 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/output.log
1225
+ 2024-02-01 17:49:12,336 DEBUG HandlerThread:237521 [handler.py:handle_request():146] handle_request: status_report
1226
+ 2024-02-01 17:49:12,751 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/output.log
1227
+ 2024-02-01 17:49:15,755 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/output.log
1228
+ 2024-02-01 17:49:17,828 DEBUG HandlerThread:237521 [handler.py:handle_request():146] handle_request: status_report
1229
+ 2024-02-01 17:49:19,760 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/output.log
1230
+ 2024-02-01 17:49:20,979 DEBUG HandlerThread:237521 [handler.py:handle_request():146] handle_request: stop_status
1231
+ 2024-02-01 17:49:20,979 DEBUG HandlerThread:237521 [handler.py:handle_request():146] handle_request: internal_messages
1232
+ 2024-02-01 17:49:20,980 DEBUG SenderThread:237521 [sender.py:send_request():409] send_request: stop_status
1233
+ 2024-02-01 17:49:21,618 DEBUG HandlerThread:237521 [handler.py:handle_request():146] handle_request: partial_history
1234
+ 2024-02-01 17:49:21,619 DEBUG SenderThread:237521 [sender.py:send():382] send: history
1235
+ 2024-02-01 17:49:21,620 DEBUG SenderThread:237521 [sender.py:send_request():409] send_request: summary_record
1236
+ 2024-02-01 17:49:21,621 INFO SenderThread:237521 [sender.py:_save_file():1392] saving file wandb-summary.json with policy end
1237
+ 2024-02-01 17:49:21,763 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/wandb-summary.json
1238
+ 2024-02-01 17:49:21,763 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/output.log
1239
+ 2024-02-01 17:49:22,765 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/output.log
1240
+ 2024-02-01 17:49:23,450 DEBUG HandlerThread:237521 [handler.py:handle_request():146] handle_request: status_report
1241
+ 2024-02-01 17:49:25,769 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/output.log
1242
+ 2024-02-01 17:49:26,250 DEBUG HandlerThread:237521 [handler.py:handle_request():146] handle_request: partial_history
1243
+ 2024-02-01 17:49:26,251 DEBUG SenderThread:237521 [sender.py:send():382] send: history
1244
+ 2024-02-01 17:49:26,251 DEBUG SenderThread:237521 [sender.py:send_request():409] send_request: summary_record
1245
+ 2024-02-01 17:49:26,253 INFO SenderThread:237521 [sender.py:_save_file():1392] saving file wandb-summary.json with policy end
1246
+ 2024-02-01 17:49:26,771 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/wandb-summary.json
1247
+ 2024-02-01 17:49:27,772 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/output.log
1248
+ 2024-02-01 17:49:28,774 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/output.log
1249
+ 2024-02-01 17:49:28,879 DEBUG SenderThread:237521 [sender.py:send():382] send: stats
1250
+ 2024-02-01 17:49:28,880 DEBUG HandlerThread:237521 [handler.py:handle_request():146] handle_request: status_report
1251
+ 2024-02-01 17:49:30,912 DEBUG HandlerThread:237521 [handler.py:handle_request():146] handle_request: partial_history
1252
+ 2024-02-01 17:49:30,913 DEBUG SenderThread:237521 [sender.py:send():382] send: history
1253
+ 2024-02-01 17:49:30,914 DEBUG SenderThread:237521 [sender.py:send_request():409] send_request: summary_record
1254
+ 2024-02-01 17:49:30,915 INFO SenderThread:237521 [sender.py:_save_file():1392] saving file wandb-summary.json with policy end
1255
+ 2024-02-01 17:49:31,779 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/wandb-summary.json
1256
+ 2024-02-01 17:49:31,779 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/output.log
1257
+ 2024-02-01 17:49:32,780 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/output.log
1258
+ 2024-02-01 17:49:34,646 DEBUG HandlerThread:237521 [handler.py:handle_request():146] handle_request: status_report
1259
+ 2024-02-01 17:49:35,579 DEBUG HandlerThread:237521 [handler.py:handle_request():146] handle_request: partial_history
1260
+ 2024-02-01 17:49:35,580 DEBUG SenderThread:237521 [sender.py:send():382] send: history
1261
+ 2024-02-01 17:49:35,581 DEBUG SenderThread:237521 [sender.py:send_request():409] send_request: summary_record
1262
+ 2024-02-01 17:49:35,583 INFO SenderThread:237521 [sender.py:_save_file():1392] saving file wandb-summary.json with policy end
1263
+ 2024-02-01 17:49:35,785 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/wandb-summary.json
1264
+ 2024-02-01 17:49:35,785 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/output.log
1265
+ 2024-02-01 17:49:35,979 DEBUG HandlerThread:237521 [handler.py:handle_request():146] handle_request: stop_status
1266
+ 2024-02-01 17:49:35,980 DEBUG HandlerThread:237521 [handler.py:handle_request():146] handle_request: internal_messages
1267
+ 2024-02-01 17:49:35,980 DEBUG SenderThread:237521 [sender.py:send_request():409] send_request: stop_status
1268
+ 2024-02-01 17:49:36,786 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/output.log
1269
+ 2024-02-01 17:49:39,790 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/output.log
1270
+ 2024-02-01 17:49:40,251 DEBUG HandlerThread:237521 [handler.py:handle_request():146] handle_request: status_report
1271
+ 2024-02-01 17:49:40,252 DEBUG HandlerThread:237521 [handler.py:handle_request():146] handle_request: partial_history
1272
+ 2024-02-01 17:49:40,253 DEBUG SenderThread:237521 [sender.py:send():382] send: history
1273
+ 2024-02-01 17:49:40,253 DEBUG SenderThread:237521 [sender.py:send_request():409] send_request: summary_record
1274
+ 2024-02-01 17:49:40,256 INFO SenderThread:237521 [sender.py:_save_file():1392] saving file wandb-summary.json with policy end
1275
+ 2024-02-01 17:49:40,792 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/wandb-summary.json
1276
+ 2024-02-01 17:49:41,794 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/output.log
1277
+ 2024-02-01 17:49:42,795 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/output.log
1278
+ 2024-02-01 17:49:44,910 DEBUG HandlerThread:237521 [handler.py:handle_request():146] handle_request: partial_history
1279
+ 2024-02-01 17:49:44,911 DEBUG SenderThread:237521 [sender.py:send():382] send: history
1280
+ 2024-02-01 17:49:44,911 DEBUG SenderThread:237521 [sender.py:send_request():409] send_request: summary_record
1281
+ 2024-02-01 17:49:44,913 INFO SenderThread:237521 [sender.py:_save_file():1392] saving file wandb-summary.json with policy end
1282
+ 2024-02-01 17:49:45,800 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/wandb-summary.json
1283
+ 2024-02-01 17:49:45,800 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/output.log
1284
+ 2024-02-01 17:49:45,841 DEBUG HandlerThread:237521 [handler.py:handle_request():146] handle_request: status_report
1285
+ 2024-02-01 17:49:46,801 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/output.log
1286
+ 2024-02-01 17:49:49,554 DEBUG HandlerThread:237521 [handler.py:handle_request():146] handle_request: partial_history
1287
+ 2024-02-01 17:49:49,555 DEBUG SenderThread:237521 [sender.py:send():382] send: history
1288
+ 2024-02-01 17:49:49,555 DEBUG SenderThread:237521 [sender.py:send_request():409] send_request: summary_record
1289
+ 2024-02-01 17:49:49,557 INFO SenderThread:237521 [sender.py:_save_file():1392] saving file wandb-summary.json with policy end
1290
+ 2024-02-01 17:49:49,806 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/wandb-summary.json
1291
+ 2024-02-01 17:49:49,806 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/output.log
1292
+ 2024-02-01 17:49:50,807 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/output.log
1293
+ 2024-02-01 17:49:50,979 DEBUG HandlerThread:237521 [handler.py:handle_request():146] handle_request: stop_status
1294
+ 2024-02-01 17:49:50,979 DEBUG HandlerThread:237521 [handler.py:handle_request():146] handle_request: internal_messages
1295
+ 2024-02-01 17:49:50,980 DEBUG HandlerThread:237521 [handler.py:handle_request():146] handle_request: status_report
1296
+ 2024-02-01 17:49:50,980 DEBUG SenderThread:237521 [sender.py:send_request():409] send_request: stop_status
1297
+ 2024-02-01 17:49:53,811 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/output.log
1298
+ 2024-02-01 17:49:54,212 DEBUG HandlerThread:237521 [handler.py:handle_request():146] handle_request: partial_history
1299
+ 2024-02-01 17:49:54,213 DEBUG SenderThread:237521 [sender.py:send():382] send: history
1300
+ 2024-02-01 17:49:54,214 DEBUG SenderThread:237521 [sender.py:send_request():409] send_request: summary_record
1301
+ 2024-02-01 17:49:54,216 INFO SenderThread:237521 [sender.py:_save_file():1392] saving file wandb-summary.json with policy end
1302
+ 2024-02-01 17:49:54,813 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/wandb-summary.json
1303
+ 2024-02-01 17:49:55,815 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/output.log
1304
+ 2024-02-01 17:49:56,082 DEBUG HandlerThread:237521 [handler.py:handle_request():146] handle_request: status_report
1305
+ 2024-02-01 17:49:56,816 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/output.log
1306
+ 2024-02-01 17:49:58,880 DEBUG HandlerThread:237521 [handler.py:handle_request():146] handle_request: partial_history
1307
+ 2024-02-01 17:49:58,882 DEBUG SenderThread:237521 [sender.py:send():382] send: stats
1308
+ 2024-02-01 17:49:58,883 DEBUG SenderThread:237521 [sender.py:send():382] send: history
1309
+ 2024-02-01 17:49:58,883 DEBUG SenderThread:237521 [sender.py:send_request():409] send_request: summary_record
1310
+ 2024-02-01 17:49:58,884 INFO SenderThread:237521 [sender.py:_save_file():1392] saving file wandb-summary.json with policy end
1311
+ 2024-02-01 17:49:59,821 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/wandb-summary.json
1312
+ 2024-02-01 17:49:59,821 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/output.log
1313
+ 2024-02-01 17:50:00,822 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/output.log
1314
+ 2024-02-01 17:50:01,678 DEBUG HandlerThread:237521 [handler.py:handle_request():146] handle_request: status_report
1315
+ 2024-02-01 17:50:02,825 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/output.log
1316
+ 2024-02-01 17:50:03,542 DEBUG HandlerThread:237521 [handler.py:handle_request():146] handle_request: partial_history
1317
+ 2024-02-01 17:50:03,543 DEBUG SenderThread:237521 [sender.py:send():382] send: history
1318
+ 2024-02-01 17:50:03,544 DEBUG SenderThread:237521 [sender.py:send_request():409] send_request: summary_record
1319
+ 2024-02-01 17:50:03,546 INFO SenderThread:237521 [sender.py:_save_file():1392] saving file wandb-summary.json with policy end
1320
+ 2024-02-01 17:50:03,827 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/wandb-summary.json
1321
+ 2024-02-01 17:50:04,828 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/output.log
1322
+ 2024-02-01 17:50:05,979 DEBUG HandlerThread:237521 [handler.py:handle_request():146] handle_request: stop_status
1323
+ 2024-02-01 17:50:05,979 DEBUG HandlerThread:237521 [handler.py:handle_request():146] handle_request: internal_messages
1324
+ 2024-02-01 17:50:05,980 DEBUG SenderThread:237521 [sender.py:send_request():409] send_request: stop_status
1325
+ 2024-02-01 17:50:06,831 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/output.log
1326
+ 2024-02-01 17:50:07,275 DEBUG HandlerThread:237521 [handler.py:handle_request():146] handle_request: status_report
1327
+ 2024-02-01 17:50:08,210 DEBUG HandlerThread:237521 [handler.py:handle_request():146] handle_request: partial_history
1328
+ 2024-02-01 17:50:08,211 DEBUG SenderThread:237521 [sender.py:send():382] send: history
1329
+ 2024-02-01 17:50:08,212 DEBUG SenderThread:237521 [sender.py:send_request():409] send_request: summary_record
1330
+ 2024-02-01 17:50:08,213 INFO SenderThread:237521 [sender.py:_save_file():1392] saving file wandb-summary.json with policy end
1331
+ 2024-02-01 17:50:08,835 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/wandb-summary.json
1332
+ 2024-02-01 17:50:08,835 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/output.log
1333
+ 2024-02-01 17:50:10,837 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/output.log
1334
+ 2024-02-01 17:50:12,840 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/output.log
1335
+ 2024-02-01 17:50:12,871 DEBUG HandlerThread:237521 [handler.py:handle_request():146] handle_request: status_report
1336
+ 2024-02-01 17:50:12,872 DEBUG HandlerThread:237521 [handler.py:handle_request():146] handle_request: partial_history
1337
+ 2024-02-01 17:50:12,873 DEBUG SenderThread:237521 [sender.py:send():382] send: history
1338
+ 2024-02-01 17:50:12,873 DEBUG SenderThread:237521 [sender.py:send_request():409] send_request: summary_record
1339
+ 2024-02-01 17:50:12,875 INFO SenderThread:237521 [sender.py:_save_file():1392] saving file wandb-summary.json with policy end
1340
+ 2024-02-01 17:50:13,842 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/wandb-summary.json
1341
+ 2024-02-01 17:50:14,843 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/output.log
1342
+ 2024-02-01 17:50:16,846 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/output.log
1343
+ 2024-02-01 17:50:17,540 DEBUG HandlerThread:237521 [handler.py:handle_request():146] handle_request: partial_history
1344
+ 2024-02-01 17:50:17,541 DEBUG SenderThread:237521 [sender.py:send():382] send: history
1345
+ 2024-02-01 17:50:17,541 DEBUG SenderThread:237521 [sender.py:send_request():409] send_request: summary_record
1346
+ 2024-02-01 17:50:17,543 INFO SenderThread:237521 [sender.py:_save_file():1392] saving file wandb-summary.json with policy end
1347
+ 2024-02-01 17:50:17,848 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/wandb-summary.json
1348
+ 2024-02-01 17:50:18,469 DEBUG HandlerThread:237521 [handler.py:handle_request():146] handle_request: status_report
wandb/run-20240201_173828-py26nu6m/run-py26nu6m.wandb CHANGED
Binary files a/wandb/run-20240201_173828-py26nu6m/run-py26nu6m.wandb and b/wandb/run-20240201_173828-py26nu6m/run-py26nu6m.wandb differ