Hugging Face's logo

sanchit-gandhi
/
distil-zephyr-1.5b-ssft

Text Generation
Transformers
TensorBoard
Safetensors
mistral
alignment-handbook
Generated from Trainer
trl
sft
conversational
text-generation-inference
Inference Endpoints
Model card Files Metrics Community
sanchit-gandhi HF staff commited on
Commit
d64e239
•
1 Parent(s): fa8bbea

Training in progress, step 700

Browse files
Files changed (7) hide show
  1. model.safetensors +1 -1
  2. runs/Feb01_17-38-02_ip-26-0-165-24/events.out.tfevents.1706809106.ip-26-0-165-24.237059.0 +2 -2
  3. wandb/debug-internal.log +201 -0
  4. wandb/run-20240201_173828-py26nu6m/files/output.log +87 -0
  5. wandb/run-20240201_173828-py26nu6m/files/wandb-summary.json +1 -1
  6. wandb/run-20240201_173828-py26nu6m/logs/debug-internal.log +201 -0
  7. wandb/run-20240201_173828-py26nu6m/run-py26nu6m.wandb +0 -0
model.safetensors CHANGED
@@ -1,3 +1,3 @@
1
  version https://git-lfs.github.com/spec/v1
2
- oid sha256:0dfc5f43bb6f4351d0eec3f0f5b16fc5d10e01761eb48e94e96056ef4bc9a6e8
3
  size 3141646744
 
1
  version https://git-lfs.github.com/spec/v1
2
+ oid sha256:911afb52d97b158ceb161b2a10f3656ac90b8f25d44eda5dc9d73b4e3f7f9688
3
  size 3141646744
runs/Feb01_17-38-02_ip-26-0-165-24/events.out.tfevents.1706809106.ip-26-0-165-24.237059.0 CHANGED
@@ -1,3 +1,3 @@
1
  version https://git-lfs.github.com/spec/v1
2
- oid sha256:3fbff98bb8fa9aa6501f0e21a4b854cbb237485139925550c0119d0e9b772b1f
3
- size 23294
 
1
  version https://git-lfs.github.com/spec/v1
2
+ oid sha256:4b50156e4c610ce48fd0a866dcead6e9ec10d5d2c3134da1db0522d3eee63d89
3
+ size 26434
wandb/debug-internal.log CHANGED
@@ -1346,3 +1346,204 @@
1346
  2024-02-01 17:50:17,543 INFO SenderThread:237521 [sender.py:_save_file():1392] saving file wandb-summary.json with policy end
1347
  2024-02-01 17:50:17,848 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/wandb-summary.json
1348
  2024-02-01 17:50:18,469 DEBUG HandlerThread:237521 [handler.py:handle_request():146] handle_request: status_report
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1346
  2024-02-01 17:50:17,543 INFO SenderThread:237521 [sender.py:_save_file():1392] saving file wandb-summary.json with policy end
1347
  2024-02-01 17:50:17,848 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/wandb-summary.json
1348
  2024-02-01 17:50:18,469 DEBUG HandlerThread:237521 [handler.py:handle_request():146] handle_request: status_report
1349
+ 2024-02-01 17:50:18,849 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/output.log
1350
+ 2024-02-01 17:50:20,852 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/output.log
1351
+ 2024-02-01 17:50:20,979 DEBUG HandlerThread:237521 [handler.py:handle_request():146] handle_request: stop_status
1352
+ 2024-02-01 17:50:20,979 DEBUG HandlerThread:237521 [handler.py:handle_request():146] handle_request: internal_messages
1353
+ 2024-02-01 17:50:20,980 DEBUG SenderThread:237521 [sender.py:send_request():409] send_request: stop_status
1354
+ 2024-02-01 17:50:22,179 DEBUG HandlerThread:237521 [handler.py:handle_request():146] handle_request: partial_history
1355
+ 2024-02-01 17:50:22,189 DEBUG SenderThread:237521 [sender.py:send():382] send: history
1356
+ 2024-02-01 17:50:22,189 DEBUG SenderThread:237521 [sender.py:send_request():409] send_request: summary_record
1357
+ 2024-02-01 17:50:22,191 INFO SenderThread:237521 [sender.py:_save_file():1392] saving file wandb-summary.json with policy end
1358
+ 2024-02-01 17:50:22,856 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/wandb-summary.json
1359
+ 2024-02-01 17:50:22,856 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/output.log
1360
+ 2024-02-01 17:50:24,047 DEBUG HandlerThread:237521 [handler.py:handle_request():146] handle_request: status_report
1361
+ 2024-02-01 17:50:24,858 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/output.log
1362
+ 2024-02-01 17:50:26,823 DEBUG HandlerThread:237521 [handler.py:handle_request():146] handle_request: partial_history
1363
+ 2024-02-01 17:50:26,824 DEBUG SenderThread:237521 [sender.py:send():382] send: history
1364
+ 2024-02-01 17:50:26,825 DEBUG SenderThread:237521 [sender.py:send_request():409] send_request: summary_record
1365
+ 2024-02-01 17:50:26,828 INFO SenderThread:237521 [sender.py:_save_file():1392] saving file wandb-summary.json with policy end
1366
+ 2024-02-01 17:50:26,862 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/wandb-summary.json
1367
+ 2024-02-01 17:50:26,862 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/output.log
1368
+ 2024-02-01 17:50:28,864 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/output.log
1369
+ 2024-02-01 17:50:28,941 DEBUG SenderThread:237521 [sender.py:send():382] send: stats
1370
+ 2024-02-01 17:50:29,608 DEBUG HandlerThread:237521 [handler.py:handle_request():146] handle_request: status_report
1371
+ 2024-02-01 17:50:30,867 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/output.log
1372
+ 2024-02-01 17:50:31,471 DEBUG HandlerThread:237521 [handler.py:handle_request():146] handle_request: partial_history
1373
+ 2024-02-01 17:50:31,473 DEBUG SenderThread:237521 [sender.py:send():382] send: history
1374
+ 2024-02-01 17:50:31,473 DEBUG SenderThread:237521 [sender.py:send_request():409] send_request: summary_record
1375
+ 2024-02-01 17:50:31,475 INFO SenderThread:237521 [sender.py:_save_file():1392] saving file wandb-summary.json with policy end
1376
+ 2024-02-01 17:50:31,870 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/wandb-summary.json
1377
+ 2024-02-01 17:50:32,870 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/output.log
1378
+ 2024-02-01 17:50:34,718 DEBUG HandlerThread:237521 [handler.py:handle_request():146] handle_request: status_report
1379
+ 2024-02-01 17:50:34,873 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/output.log
1380
+ 2024-02-01 17:50:35,979 DEBUG HandlerThread:237521 [handler.py:handle_request():146] handle_request: stop_status
1381
+ 2024-02-01 17:50:35,980 DEBUG HandlerThread:237521 [handler.py:handle_request():146] handle_request: internal_messages
1382
+ 2024-02-01 17:50:35,980 DEBUG SenderThread:237521 [sender.py:send_request():409] send_request: stop_status
1383
+ 2024-02-01 17:50:36,618 DEBUG HandlerThread:237521 [handler.py:handle_request():146] handle_request: partial_history
1384
+ 2024-02-01 17:50:36,619 DEBUG SenderThread:237521 [sender.py:send():382] send: history
1385
+ 2024-02-01 17:50:36,620 DEBUG SenderThread:237521 [sender.py:send_request():409] send_request: summary_record
1386
+ 2024-02-01 17:50:36,621 INFO SenderThread:237521 [sender.py:_save_file():1392] saving file wandb-summary.json with policy end
1387
+ 2024-02-01 17:50:36,876 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/wandb-summary.json
1388
+ 2024-02-01 17:50:36,877 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/output.log
1389
+ 2024-02-01 17:50:38,879 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/output.log
1390
+ 2024-02-01 17:50:40,303 DEBUG HandlerThread:237521 [handler.py:handle_request():146] handle_request: status_report
1391
+ 2024-02-01 17:50:40,881 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/output.log
1392
+ 2024-02-01 17:50:41,229 DEBUG HandlerThread:237521 [handler.py:handle_request():146] handle_request: partial_history
1393
+ 2024-02-01 17:50:41,230 DEBUG SenderThread:237521 [sender.py:send():382] send: history
1394
+ 2024-02-01 17:50:41,231 DEBUG SenderThread:237521 [sender.py:send_request():409] send_request: summary_record
1395
+ 2024-02-01 17:50:41,233 INFO SenderThread:237521 [sender.py:_save_file():1392] saving file wandb-summary.json with policy end
1396
+ 2024-02-01 17:50:41,884 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/wandb-summary.json
1397
+ 2024-02-01 17:50:42,885 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/output.log
1398
+ 2024-02-01 17:50:44,888 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/output.log
1399
+ 2024-02-01 17:50:45,866 DEBUG HandlerThread:237521 [handler.py:handle_request():146] handle_request: status_report
1400
+ 2024-02-01 17:50:45,867 DEBUG HandlerThread:237521 [handler.py:handle_request():146] handle_request: partial_history
1401
+ 2024-02-01 17:50:45,868 DEBUG SenderThread:237521 [sender.py:send():382] send: history
1402
+ 2024-02-01 17:50:45,869 DEBUG SenderThread:237521 [sender.py:send_request():409] send_request: summary_record
1403
+ 2024-02-01 17:50:45,871 INFO SenderThread:237521 [sender.py:_save_file():1392] saving file wandb-summary.json with policy end
1404
+ 2024-02-01 17:50:45,890 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/wandb-summary.json
1405
+ 2024-02-01 17:50:46,891 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/output.log
1406
+ 2024-02-01 17:50:48,893 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/output.log
1407
+ 2024-02-01 17:50:50,521 DEBUG HandlerThread:237521 [handler.py:handle_request():146] handle_request: partial_history
1408
+ 2024-02-01 17:50:50,522 DEBUG SenderThread:237521 [sender.py:send():382] send: history
1409
+ 2024-02-01 17:50:50,522 DEBUG SenderThread:237521 [sender.py:send_request():409] send_request: summary_record
1410
+ 2024-02-01 17:50:50,525 INFO SenderThread:237521 [sender.py:_save_file():1392] saving file wandb-summary.json with policy end
1411
+ 2024-02-01 17:50:50,897 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/wandb-summary.json
1412
+ 2024-02-01 17:50:50,897 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/output.log
1413
+ 2024-02-01 17:50:50,980 DEBUG HandlerThread:237521 [handler.py:handle_request():146] handle_request: stop_status
1414
+ 2024-02-01 17:50:50,980 DEBUG HandlerThread:237521 [handler.py:handle_request():146] handle_request: status_report
1415
+ 2024-02-01 17:50:50,980 DEBUG HandlerThread:237521 [handler.py:handle_request():146] handle_request: internal_messages
1416
+ 2024-02-01 17:50:50,980 DEBUG SenderThread:237521 [sender.py:send_request():409] send_request: stop_status
1417
+ 2024-02-01 17:50:52,899 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/output.log
1418
+ 2024-02-01 17:50:56,673 DEBUG HandlerThread:237521 [handler.py:handle_request():146] handle_request: status_report
1419
+ 2024-02-01 17:50:56,905 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/output.log
1420
+ 2024-02-01 17:50:58,942 DEBUG SenderThread:237521 [sender.py:send():382] send: stats
1421
+ 2024-02-01 17:51:00,910 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/output.log
1422
+ 2024-02-01 17:51:02,660 DEBUG HandlerThread:237521 [handler.py:handle_request():146] handle_request: status_report
1423
+ 2024-02-01 17:51:02,913 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/output.log
1424
+ 2024-02-01 17:51:05,979 DEBUG HandlerThread:237521 [handler.py:handle_request():146] handle_request: stop_status
1425
+ 2024-02-01 17:51:05,980 DEBUG HandlerThread:237521 [handler.py:handle_request():146] handle_request: internal_messages
1426
+ 2024-02-01 17:51:05,980 DEBUG SenderThread:237521 [sender.py:send_request():409] send_request: stop_status
1427
+ 2024-02-01 17:51:06,918 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/output.log
1428
+ 2024-02-01 17:51:08,062 DEBUG HandlerThread:237521 [handler.py:handle_request():146] handle_request: status_report
1429
+ 2024-02-01 17:51:08,920 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/output.log
1430
+ 2024-02-01 17:51:09,907 DEBUG HandlerThread:237521 [handler.py:handle_request():146] handle_request: partial_history
1431
+ 2024-02-01 17:51:09,993 DEBUG SenderThread:237521 [sender.py:send():382] send: history
1432
+ 2024-02-01 17:51:09,993 DEBUG SenderThread:237521 [sender.py:send_request():409] send_request: summary_record
1433
+ 2024-02-01 17:51:09,995 INFO SenderThread:237521 [sender.py:_save_file():1392] saving file wandb-summary.json with policy end
1434
+ 2024-02-01 17:51:10,924 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/wandb-summary.json
1435
+ 2024-02-01 17:51:10,924 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/output.log
1436
+ 2024-02-01 17:51:12,926 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/output.log
1437
+ 2024-02-01 17:51:13,621 DEBUG HandlerThread:237521 [handler.py:handle_request():146] handle_request: status_report
1438
+ 2024-02-01 17:51:14,549 DEBUG HandlerThread:237521 [handler.py:handle_request():146] handle_request: partial_history
1439
+ 2024-02-01 17:51:14,550 DEBUG SenderThread:237521 [sender.py:send():382] send: history
1440
+ 2024-02-01 17:51:14,551 DEBUG SenderThread:237521 [sender.py:send_request():409] send_request: summary_record
1441
+ 2024-02-01 17:51:14,553 INFO SenderThread:237521 [sender.py:_save_file():1392] saving file wandb-summary.json with policy end
1442
+ 2024-02-01 17:51:14,930 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/wandb-summary.json
1443
+ 2024-02-01 17:51:14,930 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/output.log
1444
+ 2024-02-01 17:51:16,932 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/output.log
1445
+ 2024-02-01 17:51:18,935 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/output.log
1446
+ 2024-02-01 17:51:19,179 DEBUG HandlerThread:237521 [handler.py:handle_request():146] handle_request: status_report
1447
+ 2024-02-01 17:51:19,180 DEBUG HandlerThread:237521 [handler.py:handle_request():146] handle_request: partial_history
1448
+ 2024-02-01 17:51:19,182 DEBUG SenderThread:237521 [sender.py:send():382] send: history
1449
+ 2024-02-01 17:51:19,182 DEBUG SenderThread:237521 [sender.py:send_request():409] send_request: summary_record
1450
+ 2024-02-01 17:51:19,184 INFO SenderThread:237521 [sender.py:_save_file():1392] saving file wandb-summary.json with policy end
1451
+ 2024-02-01 17:51:19,937 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/wandb-summary.json
1452
+ 2024-02-01 17:51:20,939 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/output.log
1453
+ 2024-02-01 17:51:20,980 DEBUG HandlerThread:237521 [handler.py:handle_request():146] handle_request: stop_status
1454
+ 2024-02-01 17:51:20,980 DEBUG HandlerThread:237521 [handler.py:handle_request():146] handle_request: internal_messages
1455
+ 2024-02-01 17:51:20,980 DEBUG SenderThread:237521 [sender.py:send_request():409] send_request: stop_status
1456
+ 2024-02-01 17:51:22,941 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/output.log
1457
+ 2024-02-01 17:51:23,824 DEBUG HandlerThread:237521 [handler.py:handle_request():146] handle_request: partial_history
1458
+ 2024-02-01 17:51:23,825 DEBUG SenderThread:237521 [sender.py:send():382] send: history
1459
+ 2024-02-01 17:51:23,826 DEBUG SenderThread:237521 [sender.py:send_request():409] send_request: summary_record
1460
+ 2024-02-01 17:51:23,828 INFO SenderThread:237521 [sender.py:_save_file():1392] saving file wandb-summary.json with policy end
1461
+ 2024-02-01 17:51:23,944 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/wandb-summary.json
1462
+ 2024-02-01 17:51:24,747 DEBUG HandlerThread:237521 [handler.py:handle_request():146] handle_request: status_report
1463
+ 2024-02-01 17:51:24,945 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/output.log
1464
+ 2024-02-01 17:51:26,947 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/output.log
1465
+ 2024-02-01 17:51:28,465 DEBUG HandlerThread:237521 [handler.py:handle_request():146] handle_request: partial_history
1466
+ 2024-02-01 17:51:28,466 DEBUG SenderThread:237521 [sender.py:send():382] send: history
1467
+ 2024-02-01 17:51:28,466 DEBUG SenderThread:237521 [sender.py:send_request():409] send_request: summary_record
1468
+ 2024-02-01 17:51:28,468 INFO SenderThread:237521 [sender.py:_save_file():1392] saving file wandb-summary.json with policy end
1469
+ 2024-02-01 17:51:28,944 DEBUG SenderThread:237521 [sender.py:send():382] send: stats
1470
+ 2024-02-01 17:51:28,951 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/wandb-summary.json
1471
+ 2024-02-01 17:51:28,951 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/output.log
1472
+ 2024-02-01 17:51:30,320 DEBUG HandlerThread:237521 [handler.py:handle_request():146] handle_request: status_report
1473
+ 2024-02-01 17:51:30,954 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/output.log
1474
+ 2024-02-01 17:51:32,956 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/output.log
1475
+ 2024-02-01 17:51:33,105 DEBUG HandlerThread:237521 [handler.py:handle_request():146] handle_request: partial_history
1476
+ 2024-02-01 17:51:33,106 DEBUG SenderThread:237521 [sender.py:send():382] send: history
1477
+ 2024-02-01 17:51:33,106 DEBUG SenderThread:237521 [sender.py:send_request():409] send_request: summary_record
1478
+ 2024-02-01 17:51:33,108 INFO SenderThread:237521 [sender.py:_save_file():1392] saving file wandb-summary.json with policy end
1479
+ 2024-02-01 17:51:33,959 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/wandb-summary.json
1480
+ 2024-02-01 17:51:34,960 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/output.log
1481
+ 2024-02-01 17:51:35,884 DEBUG HandlerThread:237521 [handler.py:handle_request():146] handle_request: status_report
1482
+ 2024-02-01 17:51:35,980 DEBUG HandlerThread:237521 [handler.py:handle_request():146] handle_request: stop_status
1483
+ 2024-02-01 17:51:35,980 DEBUG HandlerThread:237521 [handler.py:handle_request():146] handle_request: internal_messages
1484
+ 2024-02-01 17:51:35,980 DEBUG SenderThread:237521 [sender.py:send_request():409] send_request: stop_status
1485
+ 2024-02-01 17:51:36,962 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/output.log
1486
+ 2024-02-01 17:51:37,739 DEBUG HandlerThread:237521 [handler.py:handle_request():146] handle_request: partial_history
1487
+ 2024-02-01 17:51:37,740 DEBUG SenderThread:237521 [sender.py:send():382] send: history
1488
+ 2024-02-01 17:51:37,740 DEBUG SenderThread:237521 [sender.py:send_request():409] send_request: summary_record
1489
+ 2024-02-01 17:51:37,742 INFO SenderThread:237521 [sender.py:_save_file():1392] saving file wandb-summary.json with policy end
1490
+ 2024-02-01 17:51:37,965 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/wandb-summary.json
1491
+ 2024-02-01 17:51:38,966 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/output.log
1492
+ 2024-02-01 17:51:40,968 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/output.log
1493
+ 2024-02-01 17:51:41,448 DEBUG HandlerThread:237521 [handler.py:handle_request():146] handle_request: status_report
1494
+ 2024-02-01 17:51:42,378 DEBUG HandlerThread:237521 [handler.py:handle_request():146] handle_request: partial_history
1495
+ 2024-02-01 17:51:42,380 DEBUG SenderThread:237521 [sender.py:send():382] send: history
1496
+ 2024-02-01 17:51:42,380 DEBUG SenderThread:237521 [sender.py:send_request():409] send_request: summary_record
1497
+ 2024-02-01 17:51:42,382 INFO SenderThread:237521 [sender.py:_save_file():1392] saving file wandb-summary.json with policy end
1498
+ 2024-02-01 17:51:42,972 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/wandb-summary.json
1499
+ 2024-02-01 17:51:42,972 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/output.log
1500
+ 2024-02-01 17:51:44,975 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/output.log
1501
+ 2024-02-01 17:51:46,977 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/output.log
1502
+ 2024-02-01 17:51:47,031 DEBUG HandlerThread:237521 [handler.py:handle_request():146] handle_request: status_report
1503
+ 2024-02-01 17:51:47,032 DEBUG HandlerThread:237521 [handler.py:handle_request():146] handle_request: partial_history
1504
+ 2024-02-01 17:51:47,034 DEBUG SenderThread:237521 [sender.py:send():382] send: history
1505
+ 2024-02-01 17:51:47,034 DEBUG SenderThread:237521 [sender.py:send_request():409] send_request: summary_record
1506
+ 2024-02-01 17:51:47,036 INFO SenderThread:237521 [sender.py:_save_file():1392] saving file wandb-summary.json with policy end
1507
+ 2024-02-01 17:51:47,979 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/wandb-summary.json
1508
+ 2024-02-01 17:51:48,981 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/output.log
1509
+ 2024-02-01 17:51:50,980 DEBUG HandlerThread:237521 [handler.py:handle_request():146] handle_request: stop_status
1510
+ 2024-02-01 17:51:50,980 DEBUG HandlerThread:237521 [handler.py:handle_request():146] handle_request: internal_messages
1511
+ 2024-02-01 17:51:50,980 DEBUG SenderThread:237521 [sender.py:send_request():409] send_request: stop_status
1512
+ 2024-02-01 17:51:50,983 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/output.log
1513
+ 2024-02-01 17:51:51,942 DEBUG HandlerThread:237521 [handler.py:handle_request():146] handle_request: partial_history
1514
+ 2024-02-01 17:51:51,943 DEBUG SenderThread:237521 [sender.py:send():382] send: history
1515
+ 2024-02-01 17:51:51,944 DEBUG SenderThread:237521 [sender.py:send_request():409] send_request: summary_record
1516
+ 2024-02-01 17:51:51,946 INFO SenderThread:237521 [sender.py:_save_file():1392] saving file wandb-summary.json with policy end
1517
+ 2024-02-01 17:51:51,985 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/wandb-summary.json
1518
+ 2024-02-01 17:51:52,870 DEBUG HandlerThread:237521 [handler.py:handle_request():146] handle_request: status_report
1519
+ 2024-02-01 17:51:52,987 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/output.log
1520
+ 2024-02-01 17:51:54,989 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/output.log
1521
+ 2024-02-01 17:51:56,605 DEBUG HandlerThread:237521 [handler.py:handle_request():146] handle_request: partial_history
1522
+ 2024-02-01 17:51:56,607 DEBUG SenderThread:237521 [sender.py:send():382] send: history
1523
+ 2024-02-01 17:51:56,607 DEBUG SenderThread:237521 [sender.py:send_request():409] send_request: summary_record
1524
+ 2024-02-01 17:51:56,609 INFO SenderThread:237521 [sender.py:_save_file():1392] saving file wandb-summary.json with policy end
1525
+ 2024-02-01 17:51:56,993 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/wandb-summary.json
1526
+ 2024-02-01 17:51:56,993 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/output.log
1527
+ 2024-02-01 17:51:58,470 DEBUG HandlerThread:237521 [handler.py:handle_request():146] handle_request: status_report
1528
+ 2024-02-01 17:51:58,946 DEBUG SenderThread:237521 [sender.py:send():382] send: stats
1529
+ 2024-02-01 17:51:58,995 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/output.log
1530
+ 2024-02-01 17:52:00,998 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/output.log
1531
+ 2024-02-01 17:52:01,254 DEBUG HandlerThread:237521 [handler.py:handle_request():146] handle_request: partial_history
1532
+ 2024-02-01 17:52:01,255 DEBUG SenderThread:237521 [sender.py:send():382] send: history
1533
+ 2024-02-01 17:52:01,256 DEBUG SenderThread:237521 [sender.py:send_request():409] send_request: summary_record
1534
+ 2024-02-01 17:52:01,258 INFO SenderThread:237521 [sender.py:_save_file():1392] saving file wandb-summary.json with policy end
1535
+ 2024-02-01 17:52:02,000 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/wandb-summary.json
1536
+ 2024-02-01 17:52:03,002 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/output.log
1537
+ 2024-02-01 17:52:04,035 DEBUG HandlerThread:237521 [handler.py:handle_request():146] handle_request: status_report
1538
+ 2024-02-01 17:52:05,004 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/output.log
1539
+ 2024-02-01 17:52:05,898 DEBUG HandlerThread:237521 [handler.py:handle_request():146] handle_request: partial_history
1540
+ 2024-02-01 17:52:05,899 DEBUG SenderThread:237521 [sender.py:send():382] send: history
1541
+ 2024-02-01 17:52:05,899 DEBUG SenderThread:237521 [sender.py:send_request():409] send_request: summary_record
1542
+ 2024-02-01 17:52:05,901 INFO SenderThread:237521 [sender.py:_save_file():1392] saving file wandb-summary.json with policy end
1543
+ 2024-02-01 17:52:05,980 DEBUG HandlerThread:237521 [handler.py:handle_request():146] handle_request: stop_status
1544
+ 2024-02-01 17:52:05,980 DEBUG HandlerThread:237521 [handler.py:handle_request():146] handle_request: internal_messages
1545
+ 2024-02-01 17:52:05,980 DEBUG SenderThread:237521 [sender.py:send_request():409] send_request: stop_status
1546
+ 2024-02-01 17:52:06,006 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/wandb-summary.json
1547
+ 2024-02-01 17:52:07,008 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/output.log
1548
+ 2024-02-01 17:52:09,010 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/output.log
1549
+ 2024-02-01 17:52:09,616 DEBUG HandlerThread:237521 [handler.py:handle_request():146] handle_request: status_report
wandb/run-20240201_173828-py26nu6m/files/output.log CHANGED
@@ -545,3 +545,90 @@
545
 
546
 
547
  61%|████████████████████████████████████████████████▋ | 663/1090 [11:41<06:38, 1.07it/s]
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
545
 
546
 
547
  61%|████████████████████████████████████████████████▋ | 663/1090 [11:41<06:38, 1.07it/s]
548
+
549
+
550
+
551
+ 61%|█████████████████████████████████████████████████▏ | 670/1090 [11:47<06:30, 1.08it/s]
552
+
553
+
554
+ 62%|█████████████████████████████████████████████████▍ | 674/1090 [11:51<06:25, 1.08it/s]
555
+
556
+
557
+ 62%|█████████████████████████████████████████████████▊ | 678/1090 [11:55<06:22, 1.08it/s]
558
+
559
+
560
+
561
+ 63%|██████████████████████████████████████████████████▏ | 684/1090 [12:01<06:36, 1.02it/s]
562
+
563
+
564
+ 63%|██████████████████████████████████████████████████▌ | 689/1090 [12:05<06:16, 1.07it/s]
565
+
566
+
567
+
568
+ 64%|███████████████████████████████████████████████████ | 695/1090 [12:11<06:07, 1.08it/s]
569
+
570
+
571
+ 64%|███████████████████████████████████████████████████▎ | 699/1090 [12:15<06:03, 1.08it/s]
572
+ 64%|███████████████████████████████████████████████████▍ | 700/1090 [12:16<06:02, 1.08it/s][INFO|trainer.py:2889] 2024-02-01 17:50:51,588 >> Saving model checkpoint to ./tmp-checkpoint-700
573
+ [INFO|configuration_utils.py:483] 2024-02-01 17:50:51,591 >> Configuration saved in ./tmp-checkpoint-700/config.json
574
+ [INFO|configuration_utils.py:594] 2024-02-01 17:50:51,593 >> Configuration saved in ./tmp-checkpoint-700/generation_config.json
575
+ [2024-02-01 17:50:54,656] [INFO] [logging.py:96:log_dist] [Rank 0] [Torch] Checkpoint global_step700 is about to be saved!
576
+ [2024-02-01 17:50:54,660] [INFO] [logging.py:96:log_dist] [Rank 0] Saving model checkpoint: ./tmp-checkpoint-700/global_step700/zero_pp_rank_0_mp_rank_00_model_states.pt
577
+ [2024-02-01 17:50:54,660] [INFO] [torch_checkpoint_engine.py:21:save] [Torch] Saving ./tmp-checkpoint-700/global_step700/zero_pp_rank_0_mp_rank_00_model_states.pt...
578
+ [2024-02-01 17:50:54,665] [INFO] [torch_checkpoint_engine.py:23:save] [Torch] Saved ./tmp-checkpoint-700/global_step700/zero_pp_rank_0_mp_rank_00_model_states.pt.
579
+ [2024-02-01 17:50:54,672] [INFO] [torch_checkpoint_engine.py:21:save] [Torch] Saving ./tmp-checkpoint-700/global_step700/bf16_zero_pp_rank_0_mp_rank_00_optim_states.pt...
580
+ [INFO|modeling_utils.py:2382] 2024-02-01 17:50:54,621 >> Model weights saved in ./tmp-checkpoint-700/pytorch_model.bin
581
+ [INFO|tokenization_utils_base.py:2432] 2024-02-01 17:50:54,624 >> tokenizer config file saved in ./tmp-checkpoint-700/tokenizer_config.json
582
+ [INFO|tokenization_utils_base.py:2441] 2024-02-01 17:50:54,626 >> Special tokens file saved in ./tmp-checkpoint-700/special_tokens_map.json
583
+ /fsx/sanchit/miniconda3/envs/venv/lib/python3.11/site-packages/torch/nn/modules/module.py:1879: UserWarning: Positional args are being deprecated, use kwargs instead. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details.
584
+ warnings.warn(
585
+ [2024-02-01 17:50:58,287] [INFO] [torch_checkpoint_engine.py:23:save] [Torch] Saved ./tmp-checkpoint-700/global_step700/bf16_zero_pp_rank_0_mp_rank_00_optim_states.pt.
586
+ [2024-02-01 17:50:58,292] [INFO] [engine.py:3393:_save_zero_checkpoint] zero checkpoint saved ./tmp-checkpoint-700/global_step700/bf16_zero_pp_rank_0_mp_rank_00_optim_states.pt
587
+ [2024-02-01 17:50:58,823] [INFO] [torch_checkpoint_engine.py:33:commit] [Torch] Checkpoint global_step700 is ready now!
588
+ [INFO|tokenization_utils_base.py:2432] 2024-02-01 17:51:01,627 >> tokenizer config file saved in ./tokenizer_config.json
589
+ [INFO|tokenization_utils_base.py:2441] 2024-02-01 17:51:01,629 >> Special tokens file saved in ./special_tokens_map.json
590
+ [INFO|trainer.py:2979] 2024-02-01 17:51:01,659 >> Deleting older checkpoint [checkpoint-600] due to args.save_total_limit
591
+
592
+
593
+ 65%|███████████████████████████████████████████████████▋ | 705/1090 [12:35<12:45, 1.99s/it]
594
+
595
+
596
+ 65%|████████████████████████████████████████████��███████ | 709/1090 [12:39<07:30, 1.18s/it]
597
+
598
+
599
+ 66%|████████████████████████████████████████████████████▍ | 714/1090 [12:43<06:04, 1.03it/s]
600
+
601
+
602
+
603
+ 66%|████████████████████████████████████████████████████▊ | 720/1090 [12:49<05:45, 1.07it/s]
604
+
605
+
606
+ 67%|█████████████████████████████████████████████████████▏ | 725/1090 [12:53<05:39, 1.08it/s]
607
+
608
+
609
+ 67%|█████████████████████████████████████████████████████▌ | 729/1090 [12:57<05:35, 1.08it/s]
610
+
611
+
612
+ 67%|█████████████████████████████████████████████████████▊ | 733/1090 [13:01<05:30, 1.08it/s]
613
+
614
+
615
+
616
+ 68%|██████████████████████████████████████████████████████▎ | 740/1090 [13:07<05:24, 1.08it/s]
617
+
618
+
619
+ 68%|██████████████████████████████████████████████████████▌ | 744/1090 [13:11<05:21, 1.07it/s]
620
+
621
+
622
+
623
+ 69%|███████████████████████████████████████████████████████ | 750/1090 [13:17<05:34, 1.02it/s]
624
+
625
+
626
+ 69%|███████████████████████████████████████████████████████▎ | 754/1090 [13:21<05:17, 1.06it/s]
627
+
628
+
629
+ 70%|███████████████████████████████████████████████████████▋ | 759/1090 [13:25<05:08, 1.07it/s]
630
+
631
+
632
+
633
+ 70%|████████████████████████████████████████████████████████▏ | 765/1090 [13:31<05:02, 1.08it/s]
634
+
wandb/run-20240201_173828-py26nu6m/files/wandb-summary.json CHANGED
@@ -1 +1 @@
1
- {"train/loss": 2.7137, "train/learning_rate": 7.917750740470116e-06, "train/epoch": 0.61, "train/global_step": 665, "_timestamp": 1706809817.5397475, "_runtime": 709.0878734588623, "_step": 133}
 
1
+ {"train/loss": 2.5865, "train/learning_rate": 4.9446350645759885e-06, "train/epoch": 0.7, "train/global_step": 765, "_timestamp": 1706809925.897822, "_runtime": 817.4459478855133, "_step": 153}
wandb/run-20240201_173828-py26nu6m/logs/debug-internal.log CHANGED
@@ -1346,3 +1346,204 @@
1346
  2024-02-01 17:50:17,543 INFO SenderThread:237521 [sender.py:_save_file():1392] saving file wandb-summary.json with policy end
1347
  2024-02-01 17:50:17,848 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/wandb-summary.json
1348
  2024-02-01 17:50:18,469 DEBUG HandlerThread:237521 [handler.py:handle_request():146] handle_request: status_report
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1346
  2024-02-01 17:50:17,543 INFO SenderThread:237521 [sender.py:_save_file():1392] saving file wandb-summary.json with policy end
1347
  2024-02-01 17:50:17,848 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/wandb-summary.json
1348
  2024-02-01 17:50:18,469 DEBUG HandlerThread:237521 [handler.py:handle_request():146] handle_request: status_report
1349
+ 2024-02-01 17:50:18,849 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/output.log
1350
+ 2024-02-01 17:50:20,852 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/output.log
1351
+ 2024-02-01 17:50:20,979 DEBUG HandlerThread:237521 [handler.py:handle_request():146] handle_request: stop_status
1352
+ 2024-02-01 17:50:20,979 DEBUG HandlerThread:237521 [handler.py:handle_request():146] handle_request: internal_messages
1353
+ 2024-02-01 17:50:20,980 DEBUG SenderThread:237521 [sender.py:send_request():409] send_request: stop_status
1354
+ 2024-02-01 17:50:22,179 DEBUG HandlerThread:237521 [handler.py:handle_request():146] handle_request: partial_history
1355
+ 2024-02-01 17:50:22,189 DEBUG SenderThread:237521 [sender.py:send():382] send: history
1356
+ 2024-02-01 17:50:22,189 DEBUG SenderThread:237521 [sender.py:send_request():409] send_request: summary_record
1357
+ 2024-02-01 17:50:22,191 INFO SenderThread:237521 [sender.py:_save_file():1392] saving file wandb-summary.json with policy end
1358
+ 2024-02-01 17:50:22,856 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/wandb-summary.json
1359
+ 2024-02-01 17:50:22,856 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/output.log
1360
+ 2024-02-01 17:50:24,047 DEBUG HandlerThread:237521 [handler.py:handle_request():146] handle_request: status_report
1361
+ 2024-02-01 17:50:24,858 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/output.log
1362
+ 2024-02-01 17:50:26,823 DEBUG HandlerThread:237521 [handler.py:handle_request():146] handle_request: partial_history
1363
+ 2024-02-01 17:50:26,824 DEBUG SenderThread:237521 [sender.py:send():382] send: history
1364
+ 2024-02-01 17:50:26,825 DEBUG SenderThread:237521 [sender.py:send_request():409] send_request: summary_record
1365
+ 2024-02-01 17:50:26,828 INFO SenderThread:237521 [sender.py:_save_file():1392] saving file wandb-summary.json with policy end
1366
+ 2024-02-01 17:50:26,862 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/wandb-summary.json
1367
+ 2024-02-01 17:50:26,862 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/output.log
1368
+ 2024-02-01 17:50:28,864 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/output.log
1369
+ 2024-02-01 17:50:28,941 DEBUG SenderThread:237521 [sender.py:send():382] send: stats
1370
+ 2024-02-01 17:50:29,608 DEBUG HandlerThread:237521 [handler.py:handle_request():146] handle_request: status_report
1371
+ 2024-02-01 17:50:30,867 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/output.log
1372
+ 2024-02-01 17:50:31,471 DEBUG HandlerThread:237521 [handler.py:handle_request():146] handle_request: partial_history
1373
+ 2024-02-01 17:50:31,473 DEBUG SenderThread:237521 [sender.py:send():382] send: history
1374
+ 2024-02-01 17:50:31,473 DEBUG SenderThread:237521 [sender.py:send_request():409] send_request: summary_record
1375
+ 2024-02-01 17:50:31,475 INFO SenderThread:237521 [sender.py:_save_file():1392] saving file wandb-summary.json with policy end
1376
+ 2024-02-01 17:50:31,870 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/wandb-summary.json
1377
+ 2024-02-01 17:50:32,870 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/output.log
1378
+ 2024-02-01 17:50:34,718 DEBUG HandlerThread:237521 [handler.py:handle_request():146] handle_request: status_report
1379
+ 2024-02-01 17:50:34,873 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/output.log
1380
+ 2024-02-01 17:50:35,979 DEBUG HandlerThread:237521 [handler.py:handle_request():146] handle_request: stop_status
1381
+ 2024-02-01 17:50:35,980 DEBUG HandlerThread:237521 [handler.py:handle_request():146] handle_request: internal_messages
1382
+ 2024-02-01 17:50:35,980 DEBUG SenderThread:237521 [sender.py:send_request():409] send_request: stop_status
1383
+ 2024-02-01 17:50:36,618 DEBUG HandlerThread:237521 [handler.py:handle_request():146] handle_request: partial_history
1384
+ 2024-02-01 17:50:36,619 DEBUG SenderThread:237521 [sender.py:send():382] send: history
1385
+ 2024-02-01 17:50:36,620 DEBUG SenderThread:237521 [sender.py:send_request():409] send_request: summary_record
1386
+ 2024-02-01 17:50:36,621 INFO SenderThread:237521 [sender.py:_save_file():1392] saving file wandb-summary.json with policy end
1387
+ 2024-02-01 17:50:36,876 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/wandb-summary.json
1388
+ 2024-02-01 17:50:36,877 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/output.log
1389
+ 2024-02-01 17:50:38,879 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/output.log
1390
+ 2024-02-01 17:50:40,303 DEBUG HandlerThread:237521 [handler.py:handle_request():146] handle_request: status_report
1391
+ 2024-02-01 17:50:40,881 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/output.log
1392
+ 2024-02-01 17:50:41,229 DEBUG HandlerThread:237521 [handler.py:handle_request():146] handle_request: partial_history
1393
+ 2024-02-01 17:50:41,230 DEBUG SenderThread:237521 [sender.py:send():382] send: history
1394
+ 2024-02-01 17:50:41,231 DEBUG SenderThread:237521 [sender.py:send_request():409] send_request: summary_record
1395
+ 2024-02-01 17:50:41,233 INFO SenderThread:237521 [sender.py:_save_file():1392] saving file wandb-summary.json with policy end
1396
+ 2024-02-01 17:50:41,884 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/wandb-summary.json
1397
+ 2024-02-01 17:50:42,885 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/output.log
1398
+ 2024-02-01 17:50:44,888 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/output.log
1399
+ 2024-02-01 17:50:45,866 DEBUG HandlerThread:237521 [handler.py:handle_request():146] handle_request: status_report
1400
+ 2024-02-01 17:50:45,867 DEBUG HandlerThread:237521 [handler.py:handle_request():146] handle_request: partial_history
1401
+ 2024-02-01 17:50:45,868 DEBUG SenderThread:237521 [sender.py:send():382] send: history
1402
+ 2024-02-01 17:50:45,869 DEBUG SenderThread:237521 [sender.py:send_request():409] send_request: summary_record
1403
+ 2024-02-01 17:50:45,871 INFO SenderThread:237521 [sender.py:_save_file():1392] saving file wandb-summary.json with policy end
1404
+ 2024-02-01 17:50:45,890 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/wandb-summary.json
1405
+ 2024-02-01 17:50:46,891 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/output.log
1406
+ 2024-02-01 17:50:48,893 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/output.log
1407
+ 2024-02-01 17:50:50,521 DEBUG HandlerThread:237521 [handler.py:handle_request():146] handle_request: partial_history
1408
+ 2024-02-01 17:50:50,522 DEBUG SenderThread:237521 [sender.py:send():382] send: history
1409
+ 2024-02-01 17:50:50,522 DEBUG SenderThread:237521 [sender.py:send_request():409] send_request: summary_record
1410
+ 2024-02-01 17:50:50,525 INFO SenderThread:237521 [sender.py:_save_file():1392] saving file wandb-summary.json with policy end
1411
+ 2024-02-01 17:50:50,897 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/wandb-summary.json
1412
+ 2024-02-01 17:50:50,897 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/output.log
1413
+ 2024-02-01 17:50:50,980 DEBUG HandlerThread:237521 [handler.py:handle_request():146] handle_request: stop_status
1414
+ 2024-02-01 17:50:50,980 DEBUG HandlerThread:237521 [handler.py:handle_request():146] handle_request: status_report
1415
+ 2024-02-01 17:50:50,980 DEBUG HandlerThread:237521 [handler.py:handle_request():146] handle_request: internal_messages
1416
+ 2024-02-01 17:50:50,980 DEBUG SenderThread:237521 [sender.py:send_request():409] send_request: stop_status
1417
+ 2024-02-01 17:50:52,899 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/output.log
1418
+ 2024-02-01 17:50:56,673 DEBUG HandlerThread:237521 [handler.py:handle_request():146] handle_request: status_report
1419
+ 2024-02-01 17:50:56,905 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/output.log
1420
+ 2024-02-01 17:50:58,942 DEBUG SenderThread:237521 [sender.py:send():382] send: stats
1421
+ 2024-02-01 17:51:00,910 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/output.log
1422
+ 2024-02-01 17:51:02,660 DEBUG HandlerThread:237521 [handler.py:handle_request():146] handle_request: status_report
1423
+ 2024-02-01 17:51:02,913 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/output.log
1424
+ 2024-02-01 17:51:05,979 DEBUG HandlerThread:237521 [handler.py:handle_request():146] handle_request: stop_status
1425
+ 2024-02-01 17:51:05,980 DEBUG HandlerThread:237521 [handler.py:handle_request():146] handle_request: internal_messages
1426
+ 2024-02-01 17:51:05,980 DEBUG SenderThread:237521 [sender.py:send_request():409] send_request: stop_status
1427
+ 2024-02-01 17:51:06,918 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/output.log
1428
+ 2024-02-01 17:51:08,062 DEBUG HandlerThread:237521 [handler.py:handle_request():146] handle_request: status_report
1429
+ 2024-02-01 17:51:08,920 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/output.log
1430
+ 2024-02-01 17:51:09,907 DEBUG HandlerThread:237521 [handler.py:handle_request():146] handle_request: partial_history
1431
+ 2024-02-01 17:51:09,993 DEBUG SenderThread:237521 [sender.py:send():382] send: history
1432
+ 2024-02-01 17:51:09,993 DEBUG SenderThread:237521 [sender.py:send_request():409] send_request: summary_record
1433
+ 2024-02-01 17:51:09,995 INFO SenderThread:237521 [sender.py:_save_file():1392] saving file wandb-summary.json with policy end
1434
+ 2024-02-01 17:51:10,924 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/wandb-summary.json
1435
+ 2024-02-01 17:51:10,924 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/output.log
1436
+ 2024-02-01 17:51:12,926 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/output.log
1437
+ 2024-02-01 17:51:13,621 DEBUG HandlerThread:237521 [handler.py:handle_request():146] handle_request: status_report
1438
+ 2024-02-01 17:51:14,549 DEBUG HandlerThread:237521 [handler.py:handle_request():146] handle_request: partial_history
1439
+ 2024-02-01 17:51:14,550 DEBUG SenderThread:237521 [sender.py:send():382] send: history
1440
+ 2024-02-01 17:51:14,551 DEBUG SenderThread:237521 [sender.py:send_request():409] send_request: summary_record
1441
+ 2024-02-01 17:51:14,553 INFO SenderThread:237521 [sender.py:_save_file():1392] saving file wandb-summary.json with policy end
1442
+ 2024-02-01 17:51:14,930 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/wandb-summary.json
1443
+ 2024-02-01 17:51:14,930 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/output.log
1444
+ 2024-02-01 17:51:16,932 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/output.log
1445
+ 2024-02-01 17:51:18,935 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/output.log
1446
+ 2024-02-01 17:51:19,179 DEBUG HandlerThread:237521 [handler.py:handle_request():146] handle_request: status_report
1447
+ 2024-02-01 17:51:19,180 DEBUG HandlerThread:237521 [handler.py:handle_request():146] handle_request: partial_history
1448
+ 2024-02-01 17:51:19,182 DEBUG SenderThread:237521 [sender.py:send():382] send: history
1449
+ 2024-02-01 17:51:19,182 DEBUG SenderThread:237521 [sender.py:send_request():409] send_request: summary_record
1450
+ 2024-02-01 17:51:19,184 INFO SenderThread:237521 [sender.py:_save_file():1392] saving file wandb-summary.json with policy end
1451
+ 2024-02-01 17:51:19,937 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/wandb-summary.json
1452
+ 2024-02-01 17:51:20,939 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/output.log
1453
+ 2024-02-01 17:51:20,980 DEBUG HandlerThread:237521 [handler.py:handle_request():146] handle_request: stop_status
1454
+ 2024-02-01 17:51:20,980 DEBUG HandlerThread:237521 [handler.py:handle_request():146] handle_request: internal_messages
1455
+ 2024-02-01 17:51:20,980 DEBUG SenderThread:237521 [sender.py:send_request():409] send_request: stop_status
1456
+ 2024-02-01 17:51:22,941 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/output.log
1457
+ 2024-02-01 17:51:23,824 DEBUG HandlerThread:237521 [handler.py:handle_request():146] handle_request: partial_history
1458
+ 2024-02-01 17:51:23,825 DEBUG SenderThread:237521 [sender.py:send():382] send: history
1459
+ 2024-02-01 17:51:23,826 DEBUG SenderThread:237521 [sender.py:send_request():409] send_request: summary_record
1460
+ 2024-02-01 17:51:23,828 INFO SenderThread:237521 [sender.py:_save_file():1392] saving file wandb-summary.json with policy end
1461
+ 2024-02-01 17:51:23,944 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/wandb-summary.json
1462
+ 2024-02-01 17:51:24,747 DEBUG HandlerThread:237521 [handler.py:handle_request():146] handle_request: status_report
1463
+ 2024-02-01 17:51:24,945 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/output.log
1464
+ 2024-02-01 17:51:26,947 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/output.log
1465
+ 2024-02-01 17:51:28,465 DEBUG HandlerThread:237521 [handler.py:handle_request():146] handle_request: partial_history
1466
+ 2024-02-01 17:51:28,466 DEBUG SenderThread:237521 [sender.py:send():382] send: history
1467
+ 2024-02-01 17:51:28,466 DEBUG SenderThread:237521 [sender.py:send_request():409] send_request: summary_record
1468
+ 2024-02-01 17:51:28,468 INFO SenderThread:237521 [sender.py:_save_file():1392] saving file wandb-summary.json with policy end
1469
+ 2024-02-01 17:51:28,944 DEBUG SenderThread:237521 [sender.py:send():382] send: stats
1470
+ 2024-02-01 17:51:28,951 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/wandb-summary.json
1471
+ 2024-02-01 17:51:28,951 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/output.log
1472
+ 2024-02-01 17:51:30,320 DEBUG HandlerThread:237521 [handler.py:handle_request():146] handle_request: status_report
1473
+ 2024-02-01 17:51:30,954 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/output.log
1474
+ 2024-02-01 17:51:32,956 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/output.log
1475
+ 2024-02-01 17:51:33,105 DEBUG HandlerThread:237521 [handler.py:handle_request():146] handle_request: partial_history
1476
+ 2024-02-01 17:51:33,106 DEBUG SenderThread:237521 [sender.py:send():382] send: history
1477
+ 2024-02-01 17:51:33,106 DEBUG SenderThread:237521 [sender.py:send_request():409] send_request: summary_record
1478
+ 2024-02-01 17:51:33,108 INFO SenderThread:237521 [sender.py:_save_file():1392] saving file wandb-summary.json with policy end
1479
+ 2024-02-01 17:51:33,959 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/wandb-summary.json
1480
+ 2024-02-01 17:51:34,960 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/output.log
1481
+ 2024-02-01 17:51:35,884 DEBUG HandlerThread:237521 [handler.py:handle_request():146] handle_request: status_report
1482
+ 2024-02-01 17:51:35,980 DEBUG HandlerThread:237521 [handler.py:handle_request():146] handle_request: stop_status
1483
+ 2024-02-01 17:51:35,980 DEBUG HandlerThread:237521 [handler.py:handle_request():146] handle_request: internal_messages
1484
+ 2024-02-01 17:51:35,980 DEBUG SenderThread:237521 [sender.py:send_request():409] send_request: stop_status
1485
+ 2024-02-01 17:51:36,962 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/output.log
1486
+ 2024-02-01 17:51:37,739 DEBUG HandlerThread:237521 [handler.py:handle_request():146] handle_request: partial_history
1487
+ 2024-02-01 17:51:37,740 DEBUG SenderThread:237521 [sender.py:send():382] send: history
1488
+ 2024-02-01 17:51:37,740 DEBUG SenderThread:237521 [sender.py:send_request():409] send_request: summary_record
1489
+ 2024-02-01 17:51:37,742 INFO SenderThread:237521 [sender.py:_save_file():1392] saving file wandb-summary.json with policy end
1490
+ 2024-02-01 17:51:37,965 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/wandb-summary.json
1491
+ 2024-02-01 17:51:38,966 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/output.log
1492
+ 2024-02-01 17:51:40,968 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/output.log
1493
+ 2024-02-01 17:51:41,448 DEBUG HandlerThread:237521 [handler.py:handle_request():146] handle_request: status_report
1494
+ 2024-02-01 17:51:42,378 DEBUG HandlerThread:237521 [handler.py:handle_request():146] handle_request: partial_history
1495
+ 2024-02-01 17:51:42,380 DEBUG SenderThread:237521 [sender.py:send():382] send: history
1496
+ 2024-02-01 17:51:42,380 DEBUG SenderThread:237521 [sender.py:send_request():409] send_request: summary_record
1497
+ 2024-02-01 17:51:42,382 INFO SenderThread:237521 [sender.py:_save_file():1392] saving file wandb-summary.json with policy end
1498
+ 2024-02-01 17:51:42,972 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/wandb-summary.json
1499
+ 2024-02-01 17:51:42,972 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/output.log
1500
+ 2024-02-01 17:51:44,975 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/output.log
1501
+ 2024-02-01 17:51:46,977 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/output.log
1502
+ 2024-02-01 17:51:47,031 DEBUG HandlerThread:237521 [handler.py:handle_request():146] handle_request: status_report
1503
+ 2024-02-01 17:51:47,032 DEBUG HandlerThread:237521 [handler.py:handle_request():146] handle_request: partial_history
1504
+ 2024-02-01 17:51:47,034 DEBUG SenderThread:237521 [sender.py:send():382] send: history
1505
+ 2024-02-01 17:51:47,034 DEBUG SenderThread:237521 [sender.py:send_request():409] send_request: summary_record
1506
+ 2024-02-01 17:51:47,036 INFO SenderThread:237521 [sender.py:_save_file():1392] saving file wandb-summary.json with policy end
1507
+ 2024-02-01 17:51:47,979 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/wandb-summary.json
1508
+ 2024-02-01 17:51:48,981 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/output.log
1509
+ 2024-02-01 17:51:50,980 DEBUG HandlerThread:237521 [handler.py:handle_request():146] handle_request: stop_status
1510
+ 2024-02-01 17:51:50,980 DEBUG HandlerThread:237521 [handler.py:handle_request():146] handle_request: internal_messages
1511
+ 2024-02-01 17:51:50,980 DEBUG SenderThread:237521 [sender.py:send_request():409] send_request: stop_status
1512
+ 2024-02-01 17:51:50,983 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/output.log
1513
+ 2024-02-01 17:51:51,942 DEBUG HandlerThread:237521 [handler.py:handle_request():146] handle_request: partial_history
1514
+ 2024-02-01 17:51:51,943 DEBUG SenderThread:237521 [sender.py:send():382] send: history
1515
+ 2024-02-01 17:51:51,944 DEBUG SenderThread:237521 [sender.py:send_request():409] send_request: summary_record
1516
+ 2024-02-01 17:51:51,946 INFO SenderThread:237521 [sender.py:_save_file():1392] saving file wandb-summary.json with policy end
1517
+ 2024-02-01 17:51:51,985 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/wandb-summary.json
1518
+ 2024-02-01 17:51:52,870 DEBUG HandlerThread:237521 [handler.py:handle_request():146] handle_request: status_report
1519
+ 2024-02-01 17:51:52,987 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/output.log
1520
+ 2024-02-01 17:51:54,989 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/output.log
1521
+ 2024-02-01 17:51:56,605 DEBUG HandlerThread:237521 [handler.py:handle_request():146] handle_request: partial_history
1522
+ 2024-02-01 17:51:56,607 DEBUG SenderThread:237521 [sender.py:send():382] send: history
1523
+ 2024-02-01 17:51:56,607 DEBUG SenderThread:237521 [sender.py:send_request():409] send_request: summary_record
1524
+ 2024-02-01 17:51:56,609 INFO SenderThread:237521 [sender.py:_save_file():1392] saving file wandb-summary.json with policy end
1525
+ 2024-02-01 17:51:56,993 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/wandb-summary.json
1526
+ 2024-02-01 17:51:56,993 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/output.log
1527
+ 2024-02-01 17:51:58,470 DEBUG HandlerThread:237521 [handler.py:handle_request():146] handle_request: status_report
1528
+ 2024-02-01 17:51:58,946 DEBUG SenderThread:237521 [sender.py:send():382] send: stats
1529
+ 2024-02-01 17:51:58,995 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/output.log
1530
+ 2024-02-01 17:52:00,998 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/output.log
1531
+ 2024-02-01 17:52:01,254 DEBUG HandlerThread:237521 [handler.py:handle_request():146] handle_request: partial_history
1532
+ 2024-02-01 17:52:01,255 DEBUG SenderThread:237521 [sender.py:send():382] send: history
1533
+ 2024-02-01 17:52:01,256 DEBUG SenderThread:237521 [sender.py:send_request():409] send_request: summary_record
1534
+ 2024-02-01 17:52:01,258 INFO SenderThread:237521 [sender.py:_save_file():1392] saving file wandb-summary.json with policy end
1535
+ 2024-02-01 17:52:02,000 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/wandb-summary.json
1536
+ 2024-02-01 17:52:03,002 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/output.log
1537
+ 2024-02-01 17:52:04,035 DEBUG HandlerThread:237521 [handler.py:handle_request():146] handle_request: status_report
1538
+ 2024-02-01 17:52:05,004 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/output.log
1539
+ 2024-02-01 17:52:05,898 DEBUG HandlerThread:237521 [handler.py:handle_request():146] handle_request: partial_history
1540
+ 2024-02-01 17:52:05,899 DEBUG SenderThread:237521 [sender.py:send():382] send: history
1541
+ 2024-02-01 17:52:05,899 DEBUG SenderThread:237521 [sender.py:send_request():409] send_request: summary_record
1542
+ 2024-02-01 17:52:05,901 INFO SenderThread:237521 [sender.py:_save_file():1392] saving file wandb-summary.json with policy end
1543
+ 2024-02-01 17:52:05,980 DEBUG HandlerThread:237521 [handler.py:handle_request():146] handle_request: stop_status
1544
+ 2024-02-01 17:52:05,980 DEBUG HandlerThread:237521 [handler.py:handle_request():146] handle_request: internal_messages
1545
+ 2024-02-01 17:52:05,980 DEBUG SenderThread:237521 [sender.py:send_request():409] send_request: stop_status
1546
+ 2024-02-01 17:52:06,006 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/wandb-summary.json
1547
+ 2024-02-01 17:52:07,008 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/output.log
1548
+ 2024-02-01 17:52:09,010 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/output.log
1549
+ 2024-02-01 17:52:09,616 DEBUG HandlerThread:237521 [handler.py:handle_request():146] handle_request: status_report
wandb/run-20240201_173828-py26nu6m/run-py26nu6m.wandb CHANGED
Binary files a/wandb/run-20240201_173828-py26nu6m/run-py26nu6m.wandb and b/wandb/run-20240201_173828-py26nu6m/run-py26nu6m.wandb differ