Hugging Face's logo

sanchit-gandhi
/
distil-zephyr-1.5b-ssft

Text Generation
Transformers
TensorBoard
Safetensors
mistral
alignment-handbook
Generated from Trainer
trl
sft
conversational
text-generation-inference
Inference Endpoints
Model card Files Metrics Community
sanchit-gandhi HF staff commited on
Commit
003ac82
•
1 Parent(s): 78bd673

Training in progress, step 200

Browse files
Files changed (7) hide show
  1. model.safetensors +1 -1
  2. runs/Feb01_17-38-02_ip-26-0-165-24/events.out.tfevents.1706809106.ip-26-0-165-24.237059.0 +2 -2
  3. wandb/debug-internal.log +192 -0
  4. wandb/run-20240201_173828-py26nu6m/files/output.log +82 -0
  5. wandb/run-20240201_173828-py26nu6m/files/wandb-summary.json +1 -1
  6. wandb/run-20240201_173828-py26nu6m/logs/debug-internal.log +192 -0
  7. wandb/run-20240201_173828-py26nu6m/run-py26nu6m.wandb +0 -0
model.safetensors CHANGED
@@ -1,3 +1,3 @@
1
  version https://git-lfs.github.com/spec/v1
2
- oid sha256:ea3988d778bcca2e85f7c4596420ea91e8b5ce55168e2a692fb57b0f4bb71c04
3
  size 3141646744
 
1
  version https://git-lfs.github.com/spec/v1
2
+ oid sha256:730cfbca6c4e3b40963642af1fc8e0f356f3ef5510c73ceb03c6fbc50fd93b27
3
  size 3141646744
runs/Feb01_17-38-02_ip-26-0-165-24/events.out.tfevents.1706809106.ip-26-0-165-24.237059.0 CHANGED
@@ -1,3 +1,3 @@
1
  version https://git-lfs.github.com/spec/v1
2
- oid sha256:16dd7fa95e3557c9f2227dd3fe7f108ee5d59052a40eeb5741db9bf340a8a65d
3
- size 7609
 
1
  version https://git-lfs.github.com/spec/v1
2
+ oid sha256:1603c97b69b7d58ce2c7ed76a2adf4b022070d189ec5b91fa161296300d04cb4
3
+ size 10734
wandb/debug-internal.log CHANGED
@@ -383,3 +383,195 @@
383
  2024-02-01 17:41:27,718 DEBUG HandlerThread:237521 [handler.py:handle_request():146] handle_request: status_report
384
  2024-02-01 17:41:28,848 DEBUG SenderThread:237521 [sender.py:send():382] send: stats
385
  2024-02-01 17:41:28,987 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/output.log
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
383
  2024-02-01 17:41:27,718 DEBUG HandlerThread:237521 [handler.py:handle_request():146] handle_request: status_report
384
  2024-02-01 17:41:28,848 DEBUG SenderThread:237521 [sender.py:send():382] send: stats
385
  2024-02-01 17:41:28,987 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/output.log
386
+ 2024-02-01 17:41:30,990 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/output.log
387
+ 2024-02-01 17:41:31,370 DEBUG HandlerThread:237521 [handler.py:handle_request():146] handle_request: partial_history
388
+ 2024-02-01 17:41:31,371 DEBUG SenderThread:237521 [sender.py:send():382] send: history
389
+ 2024-02-01 17:41:31,371 DEBUG SenderThread:237521 [sender.py:send_request():409] send_request: summary_record
390
+ 2024-02-01 17:41:31,373 INFO SenderThread:237521 [sender.py:_save_file():1392] saving file wandb-summary.json with policy end
391
+ 2024-02-01 17:41:31,992 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/wandb-summary.json
392
+ 2024-02-01 17:41:32,993 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/output.log
393
+ 2024-02-01 17:41:33,202 DEBUG HandlerThread:237521 [handler.py:handle_request():146] handle_request: status_report
394
+ 2024-02-01 17:41:34,920 DEBUG HandlerThread:237521 [handler.py:handle_request():146] handle_request: stop_status
395
+ 2024-02-01 17:41:34,920 DEBUG SenderThread:237521 [sender.py:send_request():409] send_request: stop_status
396
+ 2024-02-01 17:41:34,957 DEBUG HandlerThread:237521 [handler.py:handle_request():146] handle_request: internal_messages
397
+ 2024-02-01 17:41:34,996 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/output.log
398
+ 2024-02-01 17:41:35,982 DEBUG HandlerThread:237521 [handler.py:handle_request():146] handle_request: partial_history
399
+ 2024-02-01 17:41:35,984 DEBUG SenderThread:237521 [sender.py:send():382] send: history
400
+ 2024-02-01 17:41:35,984 DEBUG SenderThread:237521 [sender.py:send_request():409] send_request: summary_record
401
+ 2024-02-01 17:41:35,986 INFO SenderThread:237521 [sender.py:_save_file():1392] saving file wandb-summary.json with policy end
402
+ 2024-02-01 17:41:35,998 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/wandb-summary.json
403
+ 2024-02-01 17:41:36,999 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/output.log
404
+ 2024-02-01 17:41:38,688 DEBUG HandlerThread:237521 [handler.py:handle_request():146] handle_request: status_report
405
+ 2024-02-01 17:41:39,002 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/output.log
406
+ 2024-02-01 17:41:40,512 DEBUG HandlerThread:237521 [handler.py:handle_request():146] handle_request: partial_history
407
+ 2024-02-01 17:41:40,513 DEBUG SenderThread:237521 [sender.py:send():382] send: history
408
+ 2024-02-01 17:41:40,513 DEBUG SenderThread:237521 [sender.py:send_request():409] send_request: summary_record
409
+ 2024-02-01 17:41:40,515 INFO SenderThread:237521 [sender.py:_save_file():1392] saving file wandb-summary.json with policy end
410
+ 2024-02-01 17:41:41,005 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/wandb-summary.json
411
+ 2024-02-01 17:41:41,006 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/output.log
412
+ 2024-02-01 17:41:43,008 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/output.log
413
+ 2024-02-01 17:41:44,171 DEBUG HandlerThread:237521 [handler.py:handle_request():146] handle_request: status_report
414
+ 2024-02-01 17:41:45,011 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/output.log
415
+ 2024-02-01 17:41:45,090 DEBUG HandlerThread:237521 [handler.py:handle_request():146] handle_request: partial_history
416
+ 2024-02-01 17:41:45,091 DEBUG SenderThread:237521 [sender.py:send():382] send: history
417
+ 2024-02-01 17:41:45,092 DEBUG SenderThread:237521 [sender.py:send_request():409] send_request: summary_record
418
+ 2024-02-01 17:41:45,093 INFO SenderThread:237521 [sender.py:_save_file():1392] saving file wandb-summary.json with policy end
419
+ 2024-02-01 17:41:46,013 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/wandb-summary.json
420
+ 2024-02-01 17:41:47,014 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/output.log
421
+ 2024-02-01 17:41:49,017 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/output.log
422
+ 2024-02-01 17:41:49,653 DEBUG HandlerThread:237521 [handler.py:handle_request():146] handle_request: status_report
423
+ 2024-02-01 17:41:49,654 DEBUG HandlerThread:237521 [handler.py:handle_request():146] handle_request: partial_history
424
+ 2024-02-01 17:41:49,655 DEBUG SenderThread:237521 [sender.py:send():382] send: history
425
+ 2024-02-01 17:41:49,656 DEBUG SenderThread:237521 [sender.py:send_request():409] send_request: summary_record
426
+ 2024-02-01 17:41:49,658 INFO SenderThread:237521 [sender.py:_save_file():1392] saving file wandb-summary.json with policy end
427
+ 2024-02-01 17:41:49,920 DEBUG HandlerThread:237521 [handler.py:handle_request():146] handle_request: stop_status
428
+ 2024-02-01 17:41:49,921 DEBUG SenderThread:237521 [sender.py:send_request():409] send_request: stop_status
429
+ 2024-02-01 17:41:49,957 DEBUG HandlerThread:237521 [handler.py:handle_request():146] handle_request: internal_messages
430
+ 2024-02-01 17:41:50,019 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/wandb-summary.json
431
+ 2024-02-01 17:41:51,020 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/output.log
432
+ 2024-02-01 17:41:53,023 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/output.log
433
+ 2024-02-01 17:41:54,225 DEBUG HandlerThread:237521 [handler.py:handle_request():146] handle_request: partial_history
434
+ 2024-02-01 17:41:54,226 DEBUG SenderThread:237521 [sender.py:send():382] send: history
435
+ 2024-02-01 17:41:54,227 DEBUG SenderThread:237521 [sender.py:send_request():409] send_request: summary_record
436
+ 2024-02-01 17:41:54,228 INFO SenderThread:237521 [sender.py:_save_file():1392] saving file wandb-summary.json with policy end
437
+ 2024-02-01 17:41:55,026 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/wandb-summary.json
438
+ 2024-02-01 17:41:55,026 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/output.log
439
+ 2024-02-01 17:41:55,229 DEBUG HandlerThread:237521 [handler.py:handle_request():146] handle_request: status_report
440
+ 2024-02-01 17:41:57,029 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/output.log
441
+ 2024-02-01 17:41:58,859 DEBUG SenderThread:237521 [sender.py:send():382] send: stats
442
+ 2024-02-01 17:42:00,860 DEBUG HandlerThread:237521 [handler.py:handle_request():146] handle_request: status_report
443
+ 2024-02-01 17:42:01,035 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/output.log
444
+ 2024-02-01 17:42:04,920 DEBUG HandlerThread:237521 [handler.py:handle_request():146] handle_request: stop_status
445
+ 2024-02-01 17:42:04,921 DEBUG SenderThread:237521 [sender.py:send_request():409] send_request: stop_status
446
+ 2024-02-01 17:42:04,957 DEBUG HandlerThread:237521 [handler.py:handle_request():146] handle_request: internal_messages
447
+ 2024-02-01 17:42:05,040 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/output.log
448
+ 2024-02-01 17:42:06,567 DEBUG HandlerThread:237521 [handler.py:handle_request():146] handle_request: status_report
449
+ 2024-02-01 17:42:07,042 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/output.log
450
+ 2024-02-01 17:42:11,047 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/output.log
451
+ 2024-02-01 17:42:11,821 DEBUG HandlerThread:237521 [handler.py:handle_request():146] handle_request: status_report
452
+ 2024-02-01 17:42:13,050 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/output.log
453
+ 2024-02-01 17:42:13,639 DEBUG HandlerThread:237521 [handler.py:handle_request():146] handle_request: partial_history
454
+ 2024-02-01 17:42:13,641 DEBUG SenderThread:237521 [sender.py:send():382] send: history
455
+ 2024-02-01 17:42:13,641 DEBUG SenderThread:237521 [sender.py:send_request():409] send_request: summary_record
456
+ 2024-02-01 17:42:13,643 INFO SenderThread:237521 [sender.py:_save_file():1392] saving file wandb-summary.json with policy end
457
+ 2024-02-01 17:42:14,052 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/wandb-summary.json
458
+ 2024-02-01 17:42:15,053 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/output.log
459
+ 2024-02-01 17:42:17,056 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/output.log
460
+ 2024-02-01 17:42:17,301 DEBUG HandlerThread:237521 [handler.py:handle_request():146] handle_request: status_report
461
+ 2024-02-01 17:42:18,210 DEBUG HandlerThread:237521 [handler.py:handle_request():146] handle_request: partial_history
462
+ 2024-02-01 17:42:18,212 DEBUG SenderThread:237521 [sender.py:send():382] send: history
463
+ 2024-02-01 17:42:18,212 DEBUG SenderThread:237521 [sender.py:send_request():409] send_request: summary_record
464
+ 2024-02-01 17:42:18,214 INFO SenderThread:237521 [sender.py:_save_file():1392] saving file wandb-summary.json with policy end
465
+ 2024-02-01 17:42:19,059 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/wandb-summary.json
466
+ 2024-02-01 17:42:19,059 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/output.log
467
+ 2024-02-01 17:42:19,920 DEBUG HandlerThread:237521 [handler.py:handle_request():146] handle_request: stop_status
468
+ 2024-02-01 17:42:19,921 DEBUG SenderThread:237521 [sender.py:send_request():409] send_request: stop_status
469
+ 2024-02-01 17:42:19,957 DEBUG HandlerThread:237521 [handler.py:handle_request():146] handle_request: internal_messages
470
+ 2024-02-01 17:42:21,062 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/output.log
471
+ 2024-02-01 17:42:22,795 DEBUG HandlerThread:237521 [handler.py:handle_request():146] handle_request: status_report
472
+ 2024-02-01 17:42:22,796 DEBUG HandlerThread:237521 [handler.py:handle_request():146] handle_request: partial_history
473
+ 2024-02-01 17:42:22,797 DEBUG SenderThread:237521 [sender.py:send():382] send: history
474
+ 2024-02-01 17:42:22,797 DEBUG SenderThread:237521 [sender.py:send_request():409] send_request: summary_record
475
+ 2024-02-01 17:42:22,800 INFO SenderThread:237521 [sender.py:_save_file():1392] saving file wandb-summary.json with policy end
476
+ 2024-02-01 17:42:23,065 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/wandb-summary.json
477
+ 2024-02-01 17:42:23,065 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/output.log
478
+ 2024-02-01 17:42:25,068 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/output.log
479
+ 2024-02-01 17:42:27,070 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/output.log
480
+ 2024-02-01 17:42:27,347 DEBUG HandlerThread:237521 [handler.py:handle_request():146] handle_request: partial_history
481
+ 2024-02-01 17:42:27,348 DEBUG SenderThread:237521 [sender.py:send():382] send: history
482
+ 2024-02-01 17:42:27,349 DEBUG SenderThread:237521 [sender.py:send_request():409] send_request: summary_record
483
+ 2024-02-01 17:42:27,350 INFO SenderThread:237521 [sender.py:_save_file():1392] saving file wandb-summary.json with policy end
484
+ 2024-02-01 17:42:28,072 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/wandb-summary.json
485
+ 2024-02-01 17:42:28,262 DEBUG HandlerThread:237521 [handler.py:handle_request():146] handle_request: status_report
486
+ 2024-02-01 17:42:28,852 DEBUG SenderThread:237521 [sender.py:send():382] send: stats
487
+ 2024-02-01 17:42:29,074 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/output.log
488
+ 2024-02-01 17:42:31,076 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/output.log
489
+ 2024-02-01 17:42:31,920 DEBUG HandlerThread:237521 [handler.py:handle_request():146] handle_request: partial_history
490
+ 2024-02-01 17:42:31,921 DEBUG SenderThread:237521 [sender.py:send():382] send: history
491
+ 2024-02-01 17:42:31,922 DEBUG SenderThread:237521 [sender.py:send_request():409] send_request: summary_record
492
+ 2024-02-01 17:42:31,923 INFO SenderThread:237521 [sender.py:_save_file():1392] saving file wandb-summary.json with policy end
493
+ 2024-02-01 17:42:32,078 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/wandb-summary.json
494
+ 2024-02-01 17:42:33,080 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/output.log
495
+ 2024-02-01 17:42:33,747 DEBUG HandlerThread:237521 [handler.py:handle_request():146] handle_request: status_report
496
+ 2024-02-01 17:42:34,921 DEBUG HandlerThread:237521 [handler.py:handle_request():146] handle_request: stop_status
497
+ 2024-02-01 17:42:34,921 DEBUG SenderThread:237521 [sender.py:send_request():409] send_request: stop_status
498
+ 2024-02-01 17:42:34,958 DEBUG HandlerThread:237521 [handler.py:handle_request():146] handle_request: internal_messages
499
+ 2024-02-01 17:42:35,082 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/output.log
500
+ 2024-02-01 17:42:36,493 DEBUG HandlerThread:237521 [handler.py:handle_request():146] handle_request: partial_history
501
+ 2024-02-01 17:42:36,494 DEBUG SenderThread:237521 [sender.py:send():382] send: history
502
+ 2024-02-01 17:42:36,495 DEBUG SenderThread:237521 [sender.py:send_request():409] send_request: summary_record
503
+ 2024-02-01 17:42:36,496 INFO SenderThread:237521 [sender.py:_save_file():1392] saving file wandb-summary.json with policy end
504
+ 2024-02-01 17:42:37,086 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/wandb-summary.json
505
+ 2024-02-01 17:42:37,086 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/output.log
506
+ 2024-02-01 17:42:39,088 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/output.log
507
+ 2024-02-01 17:42:39,242 DEBUG HandlerThread:237521 [handler.py:handle_request():146] handle_request: status_report
508
+ 2024-02-01 17:42:41,072 DEBUG HandlerThread:237521 [handler.py:handle_request():146] handle_request: partial_history
509
+ 2024-02-01 17:42:41,073 DEBUG SenderThread:237521 [sender.py:send():382] send: history
510
+ 2024-02-01 17:42:41,074 DEBUG SenderThread:237521 [sender.py:send_request():409] send_request: summary_record
511
+ 2024-02-01 17:42:41,075 INFO SenderThread:237521 [sender.py:_save_file():1392] saving file wandb-summary.json with policy end
512
+ 2024-02-01 17:42:41,092 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/wandb-summary.json
513
+ 2024-02-01 17:42:41,092 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/output.log
514
+ 2024-02-01 17:42:43,094 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/output.log
515
+ 2024-02-01 17:42:44,723 DEBUG HandlerThread:237521 [handler.py:handle_request():146] handle_request: status_report
516
+ 2024-02-01 17:42:45,097 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/output.log
517
+ 2024-02-01 17:42:45,640 DEBUG HandlerThread:237521 [handler.py:handle_request():146] handle_request: partial_history
518
+ 2024-02-01 17:42:45,642 DEBUG SenderThread:237521 [sender.py:send():382] send: history
519
+ 2024-02-01 17:42:45,642 DEBUG SenderThread:237521 [sender.py:send_request():409] send_request: summary_record
520
+ 2024-02-01 17:42:45,644 INFO SenderThread:237521 [sender.py:_save_file():1392] saving file wandb-summary.json with policy end
521
+ 2024-02-01 17:42:46,099 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/wandb-summary.json
522
+ 2024-02-01 17:42:47,100 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/output.log
523
+ 2024-02-01 17:42:49,103 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/output.log
524
+ 2024-02-01 17:42:49,921 DEBUG HandlerThread:237521 [handler.py:handle_request():146] handle_request: stop_status
525
+ 2024-02-01 17:42:49,921 DEBUG SenderThread:237521 [sender.py:send_request():409] send_request: stop_status
526
+ 2024-02-01 17:42:49,958 DEBUG HandlerThread:237521 [handler.py:handle_request():146] handle_request: internal_messages
527
+ 2024-02-01 17:42:50,002 DEBUG HandlerThread:237521 [handler.py:handle_request():146] handle_request: status_report
528
+ 2024-02-01 17:42:50,259 DEBUG HandlerThread:237521 [handler.py:handle_request():146] handle_request: partial_history
529
+ 2024-02-01 17:42:50,260 DEBUG SenderThread:237521 [sender.py:send():382] send: history
530
+ 2024-02-01 17:42:50,260 DEBUG SenderThread:237521 [sender.py:send_request():409] send_request: summary_record
531
+ 2024-02-01 17:42:50,262 INFO SenderThread:237521 [sender.py:_save_file():1392] saving file wandb-summary.json with policy end
532
+ 2024-02-01 17:42:51,107 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/wandb-summary.json
533
+ 2024-02-01 17:42:51,107 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/output.log
534
+ 2024-02-01 17:42:53,109 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/output.log
535
+ 2024-02-01 17:42:54,791 DEBUG HandlerThread:237521 [handler.py:handle_request():146] handle_request: partial_history
536
+ 2024-02-01 17:42:54,793 DEBUG SenderThread:237521 [sender.py:send():382] send: history
537
+ 2024-02-01 17:42:54,793 DEBUG SenderThread:237521 [sender.py:send_request():409] send_request: summary_record
538
+ 2024-02-01 17:42:54,795 INFO SenderThread:237521 [sender.py:_save_file():1392] saving file wandb-summary.json with policy end
539
+ 2024-02-01 17:42:55,113 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/wandb-summary.json
540
+ 2024-02-01 17:42:55,113 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/output.log
541
+ 2024-02-01 17:42:55,708 DEBUG HandlerThread:237521 [handler.py:handle_request():146] handle_request: status_report
542
+ 2024-02-01 17:42:57,115 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/output.log
543
+ 2024-02-01 17:42:58,863 DEBUG SenderThread:237521 [sender.py:send():382] send: stats
544
+ 2024-02-01 17:42:59,118 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/output.log
545
+ 2024-02-01 17:42:59,371 DEBUG HandlerThread:237521 [handler.py:handle_request():146] handle_request: partial_history
546
+ 2024-02-01 17:42:59,372 DEBUG SenderThread:237521 [sender.py:send():382] send: history
547
+ 2024-02-01 17:42:59,373 DEBUG SenderThread:237521 [sender.py:send_request():409] send_request: summary_record
548
+ 2024-02-01 17:42:59,374 INFO SenderThread:237521 [sender.py:_save_file():1392] saving file wandb-summary.json with policy end
549
+ 2024-02-01 17:43:00,120 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/wandb-summary.json
550
+ 2024-02-01 17:43:01,121 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/output.log
551
+ 2024-02-01 17:43:01,196 DEBUG HandlerThread:237521 [handler.py:handle_request():146] handle_request: status_report
552
+ 2024-02-01 17:43:03,124 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/output.log
553
+ 2024-02-01 17:43:03,937 DEBUG HandlerThread:237521 [handler.py:handle_request():146] handle_request: partial_history
554
+ 2024-02-01 17:43:03,938 DEBUG SenderThread:237521 [sender.py:send():382] send: history
555
+ 2024-02-01 17:43:03,939 DEBUG SenderThread:237521 [sender.py:send_request():409] send_request: summary_record
556
+ 2024-02-01 17:43:03,941 INFO SenderThread:237521 [sender.py:_save_file():1392] saving file wandb-summary.json with policy end
557
+ 2024-02-01 17:43:04,126 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/wandb-summary.json
558
+ 2024-02-01 17:43:04,921 DEBUG HandlerThread:237521 [handler.py:handle_request():146] handle_request: stop_status
559
+ 2024-02-01 17:43:04,921 DEBUG SenderThread:237521 [sender.py:send_request():409] send_request: stop_status
560
+ 2024-02-01 17:43:04,957 DEBUG HandlerThread:237521 [handler.py:handle_request():146] handle_request: internal_messages
561
+ 2024-02-01 17:43:05,127 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/output.log
562
+ 2024-02-01 17:43:06,691 DEBUG HandlerThread:237521 [handler.py:handle_request():146] handle_request: status_report
563
+ 2024-02-01 17:43:07,130 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/output.log
564
+ 2024-02-01 17:43:08,525 DEBUG HandlerThread:237521 [handler.py:handle_request():146] handle_request: partial_history
565
+ 2024-02-01 17:43:08,526 DEBUG SenderThread:237521 [sender.py:send():382] send: history
566
+ 2024-02-01 17:43:08,527 DEBUG SenderThread:237521 [sender.py:send_request():409] send_request: summary_record
567
+ 2024-02-01 17:43:08,529 INFO SenderThread:237521 [sender.py:_save_file():1392] saving file wandb-summary.json with policy end
568
+ 2024-02-01 17:43:09,133 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/wandb-summary.json
569
+ 2024-02-01 17:43:09,134 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/output.log
570
+ 2024-02-01 17:43:11,136 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/output.log
571
+ 2024-02-01 17:43:12,188 DEBUG HandlerThread:237521 [handler.py:handle_request():146] handle_request: status_report
572
+ 2024-02-01 17:43:13,107 DEBUG HandlerThread:237521 [handler.py:handle_request():146] handle_request: partial_history
573
+ 2024-02-01 17:43:13,109 DEBUG SenderThread:237521 [sender.py:send():382] send: history
574
+ 2024-02-01 17:43:13,109 DEBUG SenderThread:237521 [sender.py:send_request():409] send_request: summary_record
575
+ 2024-02-01 17:43:13,111 INFO SenderThread:237521 [sender.py:_save_file():1392] saving file wandb-summary.json with policy end
576
+ 2024-02-01 17:43:13,140 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/wandb-summary.json
577
+ 2024-02-01 17:43:13,140 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/output.log
wandb/run-20240201_173828-py26nu6m/files/output.log CHANGED
@@ -129,3 +129,85 @@
129
 
130
  16%|████████████▍ | 169/1090 [02:51<14:01, 1.09it/s]
131
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
129
 
130
  16%|████████████▍ | 169/1090 [02:51<14:01, 1.09it/s]
131
 
132
+
133
+ 16%|████████████▋ | 173/1090 [02:55<13:57, 1.10it/s]
134
+
135
+
136
+
137
+ 17%|█████████████▏ | 180/1090 [03:01<13:51, 1.09it/s]
138
+
139
+
140
+ 17%|█████████████▌ | 184/1090 [03:05<13:47, 1.09it/s]
141
+
142
+
143
+ 17%|█████████████▊ | 188/1090 [03:08<13:44, 1.09it/s]
144
+
145
+
146
+ 18%|██████████████▏ | 193/1090 [03:13<13:39, 1.09it/s]
147
+
148
+
149
+
150
+ 18%|██████████████▌ | 199/1090 [03:18<13:34, 1.09it/s]
151
+ 18%|██████████████▋ | 200/1090 [03:19<13:32, 1.10it/s][INFO|trainer.py:2889] 2024-02-01 17:41:55,274 >> Saving model checkpoint to ./tmp-checkpoint-200
152
+ [INFO|configuration_utils.py:483] 2024-02-01 17:41:55,278 >> Configuration saved in ./tmp-checkpoint-200/config.json
153
+ [INFO|configuration_utils.py:594] 2024-02-01 17:41:55,280 >> Configuration saved in ./tmp-checkpoint-200/generation_config.json
154
+ [2024-02-01 17:41:58,337] [INFO] [logging.py:96:log_dist] [Rank 0] [Torch] Checkpoint global_step200 is about to be saved!
155
+ [2024-02-01 17:41:58,341] [INFO] [logging.py:96:log_dist] [Rank 0] Saving model checkpoint: ./tmp-checkpoint-200/global_step200/zero_pp_rank_0_mp_rank_00_model_states.pt
156
+ [2024-02-01 17:41:58,341] [INFO] [torch_checkpoint_engine.py:21:save] [Torch] Saving ./tmp-checkpoint-200/global_step200/zero_pp_rank_0_mp_rank_00_model_states.pt...
157
+ [2024-02-01 17:41:58,452] [INFO] [torch_checkpoint_engine.py:23:save] [Torch] Saved ./tmp-checkpoint-200/global_step200/zero_pp_rank_0_mp_rank_00_model_states.pt.
158
+ [2024-02-01 17:41:58,456] [INFO] [torch_checkpoint_engine.py:21:save] [Torch] Saving ./tmp-checkpoint-200/global_step200/bf16_zero_pp_rank_0_mp_rank_00_optim_states.pt...
159
+ [INFO|modeling_utils.py:2382] 2024-02-01 17:41:58,307 >> Model weights saved in ./tmp-checkpoint-200/pytorch_model.bin
160
+ [INFO|tokenization_utils_base.py:2432] 2024-02-01 17:41:58,310 >> tokenizer config file saved in ./tmp-checkpoint-200/tokenizer_config.json
161
+ [INFO|tokenization_utils_base.py:2441] 2024-02-01 17:41:58,312 >> Special tokens file saved in ./tmp-checkpoint-200/special_tokens_map.json
162
+ /fsx/sanchit/miniconda3/envs/venv/lib/python3.11/site-packages/torch/nn/modules/module.py:1879: UserWarning: Positional args are being deprecated, use kwargs instead. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details.
163
+ warnings.warn(
164
+ [2024-02-01 17:42:02,208] [INFO] [torch_checkpoint_engine.py:23:save] [Torch] Saved ./tmp-checkpoint-200/global_step200/bf16_zero_pp_rank_0_mp_rank_00_optim_states.pt.
165
+ [2024-02-01 17:42:02,213] [INFO] [engine.py:3393:_save_zero_checkpoint] zero checkpoint saved ./tmp-checkpoint-200/global_step200/bf16_zero_pp_rank_0_mp_rank_00_optim_states.pt
166
+ [2024-02-01 17:42:02,552] [INFO] [torch_checkpoint_engine.py:33:commit] [Torch] Checkpoint global_step200 is ready now!
167
+ [INFO|tokenization_utils_base.py:2432] 2024-02-01 17:42:05,523 >> tokenizer config file saved in ./tokenizer_config.json
168
+ [INFO|tokenization_utils_base.py:2441] 2024-02-01 17:42:05,525 >> Special tokens file saved in ./special_tokens_map.json
169
+ [INFO|trainer.py:2979] 2024-02-01 17:42:05,553 >> Deleting older checkpoint [checkpoint-100] due to args.save_total_limit
170
+
171
+ 19%|██████████████▉ | 203/1090 [03:37<45:45, 3.10s/it]
172
+
173
+
174
+
175
+ 19%|███████████████▎ | 209/1090 [03:42<17:11, 1.17s/it]
176
+
177
+
178
+ 20%|███████████████▋ | 214/1090 [03:47<14:00, 1.04it/s]
179
+
180
+
181
+ 20%|████████████████ | 218/1090 [03:51<13:24, 1.08it/s]
182
+
183
+
184
+
185
+ 21%|████████████████▌ | 225/1090 [03:57<13:10, 1.09it/s]
186
+
187
+
188
+ 21%|████████████████▊ | 229/1090 [04:01<13:08, 1.09it/s]
189
+
190
+
191
+ 21%|█████████████████▏ | 234/1090 [04:05<13:03, 1.09it/s]
192
+
193
+
194
+ 22%|█████████████████▍ | 238/1090 [04:09<12:57, 1.10it/s]
195
+
196
+
197
+
198
+ 22%|█████████████████▉ | 244/1090 [04:14<12:54, 1.09it/s]
199
+
200
+
201
+ 23%|██████████████████▎ | 249/1090 [04:19<12:51, 1.09it/s]
202
+
203
+
204
+ 23%|██████████████████▌ | 253/1090 [04:23<12:45, 1.09it/s]
205
+
206
+
207
+
208
+ 24%|███████████████████ | 260/1090 [04:29<12:38, 1.09it/s]
209
+
210
+
211
+ 24%|███████████████████▍ | 264/1090 [04:33<12:37, 1.09it/s]
212
+
213
+
wandb/run-20240201_173828-py26nu6m/files/wandb-summary.json CHANGED
@@ -1 +1 @@
1
- {"train/loss": 6.8326, "train/learning_rate": 1.9809800282473014e-05, "train/epoch": 0.16, "train/global_step": 170, "_timestamp": 1706809286.8025768, "_runtime": 178.35070276260376, "_step": 34}
 
1
+ {"train/loss": 5.3788, "train/learning_rate": 1.8700006548379898e-05, "train/epoch": 0.25, "train/global_step": 270, "_timestamp": 1706809393.1073587, "_runtime": 284.6554846763611, "_step": 54}
wandb/run-20240201_173828-py26nu6m/logs/debug-internal.log CHANGED
@@ -383,3 +383,195 @@
383
  2024-02-01 17:41:27,718 DEBUG HandlerThread:237521 [handler.py:handle_request():146] handle_request: status_report
384
  2024-02-01 17:41:28,848 DEBUG SenderThread:237521 [sender.py:send():382] send: stats
385
  2024-02-01 17:41:28,987 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/output.log
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
383
  2024-02-01 17:41:27,718 DEBUG HandlerThread:237521 [handler.py:handle_request():146] handle_request: status_report
384
  2024-02-01 17:41:28,848 DEBUG SenderThread:237521 [sender.py:send():382] send: stats
385
  2024-02-01 17:41:28,987 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/output.log
386
+ 2024-02-01 17:41:30,990 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/output.log
387
+ 2024-02-01 17:41:31,370 DEBUG HandlerThread:237521 [handler.py:handle_request():146] handle_request: partial_history
388
+ 2024-02-01 17:41:31,371 DEBUG SenderThread:237521 [sender.py:send():382] send: history
389
+ 2024-02-01 17:41:31,371 DEBUG SenderThread:237521 [sender.py:send_request():409] send_request: summary_record
390
+ 2024-02-01 17:41:31,373 INFO SenderThread:237521 [sender.py:_save_file():1392] saving file wandb-summary.json with policy end
391
+ 2024-02-01 17:41:31,992 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/wandb-summary.json
392
+ 2024-02-01 17:41:32,993 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/output.log
393
+ 2024-02-01 17:41:33,202 DEBUG HandlerThread:237521 [handler.py:handle_request():146] handle_request: status_report
394
+ 2024-02-01 17:41:34,920 DEBUG HandlerThread:237521 [handler.py:handle_request():146] handle_request: stop_status
395
+ 2024-02-01 17:41:34,920 DEBUG SenderThread:237521 [sender.py:send_request():409] send_request: stop_status
396
+ 2024-02-01 17:41:34,957 DEBUG HandlerThread:237521 [handler.py:handle_request():146] handle_request: internal_messages
397
+ 2024-02-01 17:41:34,996 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/output.log
398
+ 2024-02-01 17:41:35,982 DEBUG HandlerThread:237521 [handler.py:handle_request():146] handle_request: partial_history
399
+ 2024-02-01 17:41:35,984 DEBUG SenderThread:237521 [sender.py:send():382] send: history
400
+ 2024-02-01 17:41:35,984 DEBUG SenderThread:237521 [sender.py:send_request():409] send_request: summary_record
401
+ 2024-02-01 17:41:35,986 INFO SenderThread:237521 [sender.py:_save_file():1392] saving file wandb-summary.json with policy end
402
+ 2024-02-01 17:41:35,998 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/wandb-summary.json
403
+ 2024-02-01 17:41:36,999 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/output.log
404
+ 2024-02-01 17:41:38,688 DEBUG HandlerThread:237521 [handler.py:handle_request():146] handle_request: status_report
405
+ 2024-02-01 17:41:39,002 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/output.log
406
+ 2024-02-01 17:41:40,512 DEBUG HandlerThread:237521 [handler.py:handle_request():146] handle_request: partial_history
407
+ 2024-02-01 17:41:40,513 DEBUG SenderThread:237521 [sender.py:send():382] send: history
408
+ 2024-02-01 17:41:40,513 DEBUG SenderThread:237521 [sender.py:send_request():409] send_request: summary_record
409
+ 2024-02-01 17:41:40,515 INFO SenderThread:237521 [sender.py:_save_file():1392] saving file wandb-summary.json with policy end
410
+ 2024-02-01 17:41:41,005 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/wandb-summary.json
411
+ 2024-02-01 17:41:41,006 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/output.log
412
+ 2024-02-01 17:41:43,008 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/output.log
413
+ 2024-02-01 17:41:44,171 DEBUG HandlerThread:237521 [handler.py:handle_request():146] handle_request: status_report
414
+ 2024-02-01 17:41:45,011 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/output.log
415
+ 2024-02-01 17:41:45,090 DEBUG HandlerThread:237521 [handler.py:handle_request():146] handle_request: partial_history
416
+ 2024-02-01 17:41:45,091 DEBUG SenderThread:237521 [sender.py:send():382] send: history
417
+ 2024-02-01 17:41:45,092 DEBUG SenderThread:237521 [sender.py:send_request():409] send_request: summary_record
418
+ 2024-02-01 17:41:45,093 INFO SenderThread:237521 [sender.py:_save_file():1392] saving file wandb-summary.json with policy end
419
+ 2024-02-01 17:41:46,013 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/wandb-summary.json
420
+ 2024-02-01 17:41:47,014 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/output.log
421
+ 2024-02-01 17:41:49,017 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/output.log
422
+ 2024-02-01 17:41:49,653 DEBUG HandlerThread:237521 [handler.py:handle_request():146] handle_request: status_report
423
+ 2024-02-01 17:41:49,654 DEBUG HandlerThread:237521 [handler.py:handle_request():146] handle_request: partial_history
424
+ 2024-02-01 17:41:49,655 DEBUG SenderThread:237521 [sender.py:send():382] send: history
425
+ 2024-02-01 17:41:49,656 DEBUG SenderThread:237521 [sender.py:send_request():409] send_request: summary_record
426
+ 2024-02-01 17:41:49,658 INFO SenderThread:237521 [sender.py:_save_file():1392] saving file wandb-summary.json with policy end
427
+ 2024-02-01 17:41:49,920 DEBUG HandlerThread:237521 [handler.py:handle_request():146] handle_request: stop_status
428
+ 2024-02-01 17:41:49,921 DEBUG SenderThread:237521 [sender.py:send_request():409] send_request: stop_status
429
+ 2024-02-01 17:41:49,957 DEBUG HandlerThread:237521 [handler.py:handle_request():146] handle_request: internal_messages
430
+ 2024-02-01 17:41:50,019 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/wandb-summary.json
431
+ 2024-02-01 17:41:51,020 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/output.log
432
+ 2024-02-01 17:41:53,023 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/output.log
433
+ 2024-02-01 17:41:54,225 DEBUG HandlerThread:237521 [handler.py:handle_request():146] handle_request: partial_history
434
+ 2024-02-01 17:41:54,226 DEBUG SenderThread:237521 [sender.py:send():382] send: history
435
+ 2024-02-01 17:41:54,227 DEBUG SenderThread:237521 [sender.py:send_request():409] send_request: summary_record
436
+ 2024-02-01 17:41:54,228 INFO SenderThread:237521 [sender.py:_save_file():1392] saving file wandb-summary.json with policy end
437
+ 2024-02-01 17:41:55,026 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/wandb-summary.json
438
+ 2024-02-01 17:41:55,026 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/output.log
439
+ 2024-02-01 17:41:55,229 DEBUG HandlerThread:237521 [handler.py:handle_request():146] handle_request: status_report
440
+ 2024-02-01 17:41:57,029 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/output.log
441
+ 2024-02-01 17:41:58,859 DEBUG SenderThread:237521 [sender.py:send():382] send: stats
442
+ 2024-02-01 17:42:00,860 DEBUG HandlerThread:237521 [handler.py:handle_request():146] handle_request: status_report
443
+ 2024-02-01 17:42:01,035 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/output.log
444
+ 2024-02-01 17:42:04,920 DEBUG HandlerThread:237521 [handler.py:handle_request():146] handle_request: stop_status
445
+ 2024-02-01 17:42:04,921 DEBUG SenderThread:237521 [sender.py:send_request():409] send_request: stop_status
446
+ 2024-02-01 17:42:04,957 DEBUG HandlerThread:237521 [handler.py:handle_request():146] handle_request: internal_messages
447
+ 2024-02-01 17:42:05,040 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/output.log
448
+ 2024-02-01 17:42:06,567 DEBUG HandlerThread:237521 [handler.py:handle_request():146] handle_request: status_report
449
+ 2024-02-01 17:42:07,042 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/output.log
450
+ 2024-02-01 17:42:11,047 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/output.log
451
+ 2024-02-01 17:42:11,821 DEBUG HandlerThread:237521 [handler.py:handle_request():146] handle_request: status_report
452
+ 2024-02-01 17:42:13,050 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/output.log
453
+ 2024-02-01 17:42:13,639 DEBUG HandlerThread:237521 [handler.py:handle_request():146] handle_request: partial_history
454
+ 2024-02-01 17:42:13,641 DEBUG SenderThread:237521 [sender.py:send():382] send: history
455
+ 2024-02-01 17:42:13,641 DEBUG SenderThread:237521 [sender.py:send_request():409] send_request: summary_record
456
+ 2024-02-01 17:42:13,643 INFO SenderThread:237521 [sender.py:_save_file():1392] saving file wandb-summary.json with policy end
457
+ 2024-02-01 17:42:14,052 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/wandb-summary.json
458
+ 2024-02-01 17:42:15,053 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/output.log
459
+ 2024-02-01 17:42:17,056 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/output.log
460
+ 2024-02-01 17:42:17,301 DEBUG HandlerThread:237521 [handler.py:handle_request():146] handle_request: status_report
461
+ 2024-02-01 17:42:18,210 DEBUG HandlerThread:237521 [handler.py:handle_request():146] handle_request: partial_history
462
+ 2024-02-01 17:42:18,212 DEBUG SenderThread:237521 [sender.py:send():382] send: history
463
+ 2024-02-01 17:42:18,212 DEBUG SenderThread:237521 [sender.py:send_request():409] send_request: summary_record
464
+ 2024-02-01 17:42:18,214 INFO SenderThread:237521 [sender.py:_save_file():1392] saving file wandb-summary.json with policy end
465
+ 2024-02-01 17:42:19,059 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/wandb-summary.json
466
+ 2024-02-01 17:42:19,059 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/output.log
467
+ 2024-02-01 17:42:19,920 DEBUG HandlerThread:237521 [handler.py:handle_request():146] handle_request: stop_status
468
+ 2024-02-01 17:42:19,921 DEBUG SenderThread:237521 [sender.py:send_request():409] send_request: stop_status
469
+ 2024-02-01 17:42:19,957 DEBUG HandlerThread:237521 [handler.py:handle_request():146] handle_request: internal_messages
470
+ 2024-02-01 17:42:21,062 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/output.log
471
+ 2024-02-01 17:42:22,795 DEBUG HandlerThread:237521 [handler.py:handle_request():146] handle_request: status_report
472
+ 2024-02-01 17:42:22,796 DEBUG HandlerThread:237521 [handler.py:handle_request():146] handle_request: partial_history
473
+ 2024-02-01 17:42:22,797 DEBUG SenderThread:237521 [sender.py:send():382] send: history
474
+ 2024-02-01 17:42:22,797 DEBUG SenderThread:237521 [sender.py:send_request():409] send_request: summary_record
475
+ 2024-02-01 17:42:22,800 INFO SenderThread:237521 [sender.py:_save_file():1392] saving file wandb-summary.json with policy end
476
+ 2024-02-01 17:42:23,065 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/wandb-summary.json
477
+ 2024-02-01 17:42:23,065 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/output.log
478
+ 2024-02-01 17:42:25,068 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/output.log
479
+ 2024-02-01 17:42:27,070 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/output.log
480
+ 2024-02-01 17:42:27,347 DEBUG HandlerThread:237521 [handler.py:handle_request():146] handle_request: partial_history
481
+ 2024-02-01 17:42:27,348 DEBUG SenderThread:237521 [sender.py:send():382] send: history
482
+ 2024-02-01 17:42:27,349 DEBUG SenderThread:237521 [sender.py:send_request():409] send_request: summary_record
483
+ 2024-02-01 17:42:27,350 INFO SenderThread:237521 [sender.py:_save_file():1392] saving file wandb-summary.json with policy end
484
+ 2024-02-01 17:42:28,072 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/wandb-summary.json
485
+ 2024-02-01 17:42:28,262 DEBUG HandlerThread:237521 [handler.py:handle_request():146] handle_request: status_report
486
+ 2024-02-01 17:42:28,852 DEBUG SenderThread:237521 [sender.py:send():382] send: stats
487
+ 2024-02-01 17:42:29,074 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/output.log
488
+ 2024-02-01 17:42:31,076 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/output.log
489
+ 2024-02-01 17:42:31,920 DEBUG HandlerThread:237521 [handler.py:handle_request():146] handle_request: partial_history
490
+ 2024-02-01 17:42:31,921 DEBUG SenderThread:237521 [sender.py:send():382] send: history
491
+ 2024-02-01 17:42:31,922 DEBUG SenderThread:237521 [sender.py:send_request():409] send_request: summary_record
492
+ 2024-02-01 17:42:31,923 INFO SenderThread:237521 [sender.py:_save_file():1392] saving file wandb-summary.json with policy end
493
+ 2024-02-01 17:42:32,078 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/wandb-summary.json
494
+ 2024-02-01 17:42:33,080 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/output.log
495
+ 2024-02-01 17:42:33,747 DEBUG HandlerThread:237521 [handler.py:handle_request():146] handle_request: status_report
496
+ 2024-02-01 17:42:34,921 DEBUG HandlerThread:237521 [handler.py:handle_request():146] handle_request: stop_status
497
+ 2024-02-01 17:42:34,921 DEBUG SenderThread:237521 [sender.py:send_request():409] send_request: stop_status
498
+ 2024-02-01 17:42:34,958 DEBUG HandlerThread:237521 [handler.py:handle_request():146] handle_request: internal_messages
499
+ 2024-02-01 17:42:35,082 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/output.log
500
+ 2024-02-01 17:42:36,493 DEBUG HandlerThread:237521 [handler.py:handle_request():146] handle_request: partial_history
501
+ 2024-02-01 17:42:36,494 DEBUG SenderThread:237521 [sender.py:send():382] send: history
502
+ 2024-02-01 17:42:36,495 DEBUG SenderThread:237521 [sender.py:send_request():409] send_request: summary_record
503
+ 2024-02-01 17:42:36,496 INFO SenderThread:237521 [sender.py:_save_file():1392] saving file wandb-summary.json with policy end
504
+ 2024-02-01 17:42:37,086 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/wandb-summary.json
505
+ 2024-02-01 17:42:37,086 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/output.log
506
+ 2024-02-01 17:42:39,088 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/output.log
507
+ 2024-02-01 17:42:39,242 DEBUG HandlerThread:237521 [handler.py:handle_request():146] handle_request: status_report
508
+ 2024-02-01 17:42:41,072 DEBUG HandlerThread:237521 [handler.py:handle_request():146] handle_request: partial_history
509
+ 2024-02-01 17:42:41,073 DEBUG SenderThread:237521 [sender.py:send():382] send: history
510
+ 2024-02-01 17:42:41,074 DEBUG SenderThread:237521 [sender.py:send_request():409] send_request: summary_record
511
+ 2024-02-01 17:42:41,075 INFO SenderThread:237521 [sender.py:_save_file():1392] saving file wandb-summary.json with policy end
512
+ 2024-02-01 17:42:41,092 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/wandb-summary.json
513
+ 2024-02-01 17:42:41,092 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/output.log
514
+ 2024-02-01 17:42:43,094 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/output.log
515
+ 2024-02-01 17:42:44,723 DEBUG HandlerThread:237521 [handler.py:handle_request():146] handle_request: status_report
516
+ 2024-02-01 17:42:45,097 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/output.log
517
+ 2024-02-01 17:42:45,640 DEBUG HandlerThread:237521 [handler.py:handle_request():146] handle_request: partial_history
518
+ 2024-02-01 17:42:45,642 DEBUG SenderThread:237521 [sender.py:send():382] send: history
519
+ 2024-02-01 17:42:45,642 DEBUG SenderThread:237521 [sender.py:send_request():409] send_request: summary_record
520
+ 2024-02-01 17:42:45,644 INFO SenderThread:237521 [sender.py:_save_file():1392] saving file wandb-summary.json with policy end
521
+ 2024-02-01 17:42:46,099 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/wandb-summary.json
522
+ 2024-02-01 17:42:47,100 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/output.log
523
+ 2024-02-01 17:42:49,103 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/output.log
524
+ 2024-02-01 17:42:49,921 DEBUG HandlerThread:237521 [handler.py:handle_request():146] handle_request: stop_status
525
+ 2024-02-01 17:42:49,921 DEBUG SenderThread:237521 [sender.py:send_request():409] send_request: stop_status
526
+ 2024-02-01 17:42:49,958 DEBUG HandlerThread:237521 [handler.py:handle_request():146] handle_request: internal_messages
527
+ 2024-02-01 17:42:50,002 DEBUG HandlerThread:237521 [handler.py:handle_request():146] handle_request: status_report
528
+ 2024-02-01 17:42:50,259 DEBUG HandlerThread:237521 [handler.py:handle_request():146] handle_request: partial_history
529
+ 2024-02-01 17:42:50,260 DEBUG SenderThread:237521 [sender.py:send():382] send: history
530
+ 2024-02-01 17:42:50,260 DEBUG SenderThread:237521 [sender.py:send_request():409] send_request: summary_record
531
+ 2024-02-01 17:42:50,262 INFO SenderThread:237521 [sender.py:_save_file():1392] saving file wandb-summary.json with policy end
532
+ 2024-02-01 17:42:51,107 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/wandb-summary.json
533
+ 2024-02-01 17:42:51,107 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/output.log
534
+ 2024-02-01 17:42:53,109 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/output.log
535
+ 2024-02-01 17:42:54,791 DEBUG HandlerThread:237521 [handler.py:handle_request():146] handle_request: partial_history
536
+ 2024-02-01 17:42:54,793 DEBUG SenderThread:237521 [sender.py:send():382] send: history
537
+ 2024-02-01 17:42:54,793 DEBUG SenderThread:237521 [sender.py:send_request():409] send_request: summary_record
538
+ 2024-02-01 17:42:54,795 INFO SenderThread:237521 [sender.py:_save_file():1392] saving file wandb-summary.json with policy end
539
+ 2024-02-01 17:42:55,113 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/wandb-summary.json
540
+ 2024-02-01 17:42:55,113 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/output.log
541
+ 2024-02-01 17:42:55,708 DEBUG HandlerThread:237521 [handler.py:handle_request():146] handle_request: status_report
542
+ 2024-02-01 17:42:57,115 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/output.log
543
+ 2024-02-01 17:42:58,863 DEBUG SenderThread:237521 [sender.py:send():382] send: stats
544
+ 2024-02-01 17:42:59,118 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/output.log
545
+ 2024-02-01 17:42:59,371 DEBUG HandlerThread:237521 [handler.py:handle_request():146] handle_request: partial_history
546
+ 2024-02-01 17:42:59,372 DEBUG SenderThread:237521 [sender.py:send():382] send: history
547
+ 2024-02-01 17:42:59,373 DEBUG SenderThread:237521 [sender.py:send_request():409] send_request: summary_record
548
+ 2024-02-01 17:42:59,374 INFO SenderThread:237521 [sender.py:_save_file():1392] saving file wandb-summary.json with policy end
549
+ 2024-02-01 17:43:00,120 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/wandb-summary.json
550
+ 2024-02-01 17:43:01,121 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/output.log
551
+ 2024-02-01 17:43:01,196 DEBUG HandlerThread:237521 [handler.py:handle_request():146] handle_request: status_report
552
+ 2024-02-01 17:43:03,124 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/output.log
553
+ 2024-02-01 17:43:03,937 DEBUG HandlerThread:237521 [handler.py:handle_request():146] handle_request: partial_history
554
+ 2024-02-01 17:43:03,938 DEBUG SenderThread:237521 [sender.py:send():382] send: history
555
+ 2024-02-01 17:43:03,939 DEBUG SenderThread:237521 [sender.py:send_request():409] send_request: summary_record
556
+ 2024-02-01 17:43:03,941 INFO SenderThread:237521 [sender.py:_save_file():1392] saving file wandb-summary.json with policy end
557
+ 2024-02-01 17:43:04,126 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/wandb-summary.json
558
+ 2024-02-01 17:43:04,921 DEBUG HandlerThread:237521 [handler.py:handle_request():146] handle_request: stop_status
559
+ 2024-02-01 17:43:04,921 DEBUG SenderThread:237521 [sender.py:send_request():409] send_request: stop_status
560
+ 2024-02-01 17:43:04,957 DEBUG HandlerThread:237521 [handler.py:handle_request():146] handle_request: internal_messages
561
+ 2024-02-01 17:43:05,127 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/output.log
562
+ 2024-02-01 17:43:06,691 DEBUG HandlerThread:237521 [handler.py:handle_request():146] handle_request: status_report
563
+ 2024-02-01 17:43:07,130 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/output.log
564
+ 2024-02-01 17:43:08,525 DEBUG HandlerThread:237521 [handler.py:handle_request():146] handle_request: partial_history
565
+ 2024-02-01 17:43:08,526 DEBUG SenderThread:237521 [sender.py:send():382] send: history
566
+ 2024-02-01 17:43:08,527 DEBUG SenderThread:237521 [sender.py:send_request():409] send_request: summary_record
567
+ 2024-02-01 17:43:08,529 INFO SenderThread:237521 [sender.py:_save_file():1392] saving file wandb-summary.json with policy end
568
+ 2024-02-01 17:43:09,133 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/wandb-summary.json
569
+ 2024-02-01 17:43:09,134 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/output.log
570
+ 2024-02-01 17:43:11,136 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/output.log
571
+ 2024-02-01 17:43:12,188 DEBUG HandlerThread:237521 [handler.py:handle_request():146] handle_request: status_report
572
+ 2024-02-01 17:43:13,107 DEBUG HandlerThread:237521 [handler.py:handle_request():146] handle_request: partial_history
573
+ 2024-02-01 17:43:13,109 DEBUG SenderThread:237521 [sender.py:send():382] send: history
574
+ 2024-02-01 17:43:13,109 DEBUG SenderThread:237521 [sender.py:send_request():409] send_request: summary_record
575
+ 2024-02-01 17:43:13,111 INFO SenderThread:237521 [sender.py:_save_file():1392] saving file wandb-summary.json with policy end
576
+ 2024-02-01 17:43:13,140 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/wandb-summary.json
577
+ 2024-02-01 17:43:13,140 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/output.log
wandb/run-20240201_173828-py26nu6m/run-py26nu6m.wandb CHANGED
Binary files a/wandb/run-20240201_173828-py26nu6m/run-py26nu6m.wandb and b/wandb/run-20240201_173828-py26nu6m/run-py26nu6m.wandb differ