Hugging Face's logo

sanchit-gandhi
/
distil-zephyr-1.5b-ssft

Text Generation
Transformers
TensorBoard
Safetensors
mistral
alignment-handbook
Generated from Trainer
trl
sft
conversational
text-generation-inference
Inference Endpoints
Model card Files Metrics Community
sanchit-gandhi HF staff commited on
Commit
6299411
•
1 Parent(s): 003ac82

Training in progress, step 300

Browse files
Files changed (7) hide show
  1. model.safetensors +1 -1
  2. runs/Feb01_17-38-02_ip-26-0-165-24/events.out.tfevents.1706809106.ip-26-0-165-24.237059.0 +2 -2
  3. wandb/debug-internal.log +187 -0
  4. wandb/run-20240201_173828-py26nu6m/files/output.log +83 -0
  5. wandb/run-20240201_173828-py26nu6m/files/wandb-summary.json +1 -1
  6. wandb/run-20240201_173828-py26nu6m/logs/debug-internal.log +187 -0
  7. wandb/run-20240201_173828-py26nu6m/run-py26nu6m.wandb +0 -0
model.safetensors CHANGED
@@ -1,3 +1,3 @@
1
  version https://git-lfs.github.com/spec/v1
2
- oid sha256:730cfbca6c4e3b40963642af1fc8e0f356f3ef5510c73ceb03c6fbc50fd93b27
3
  size 3141646744
 
1
  version https://git-lfs.github.com/spec/v1
2
+ oid sha256:5726b95fb59862ee5de794319c14f7296f374c5d53f885a1aba0207b8074c149
3
  size 3141646744
runs/Feb01_17-38-02_ip-26-0-165-24/events.out.tfevents.1706809106.ip-26-0-165-24.237059.0 CHANGED
@@ -1,3 +1,3 @@
1
  version https://git-lfs.github.com/spec/v1
2
- oid sha256:1603c97b69b7d58ce2c7ed76a2adf4b022070d189ec5b91fa161296300d04cb4
3
- size 10734
 
1
  version https://git-lfs.github.com/spec/v1
2
+ oid sha256:4aecbf973c9bcd1ddac7d1105844774d635b2bcc5b63e337974801e429f7baa1
3
+ size 13874
wandb/debug-internal.log CHANGED
@@ -575,3 +575,190 @@
575
  2024-02-01 17:43:13,111 INFO SenderThread:237521 [sender.py:_save_file():1392] saving file wandb-summary.json with policy end
576
  2024-02-01 17:43:13,140 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/wandb-summary.json
577
  2024-02-01 17:43:13,140 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/output.log
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
575
  2024-02-01 17:43:13,111 INFO SenderThread:237521 [sender.py:_save_file():1392] saving file wandb-summary.json with policy end
576
  2024-02-01 17:43:13,140 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/wandb-summary.json
577
  2024-02-01 17:43:13,140 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/output.log
578
+ 2024-02-01 17:43:15,143 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/output.log
579
+ 2024-02-01 17:43:17,145 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/output.log
580
+ 2024-02-01 17:43:17,675 DEBUG HandlerThread:237521 [handler.py:handle_request():146] handle_request: status_report
581
+ 2024-02-01 17:43:17,676 DEBUG HandlerThread:237521 [handler.py:handle_request():146] handle_request: partial_history
582
+ 2024-02-01 17:43:17,678 DEBUG SenderThread:237521 [sender.py:send():382] send: history
583
+ 2024-02-01 17:43:17,678 DEBUG SenderThread:237521 [sender.py:send_request():409] send_request: summary_record
584
+ 2024-02-01 17:43:17,680 INFO SenderThread:237521 [sender.py:_save_file():1392] saving file wandb-summary.json with policy end
585
+ 2024-02-01 17:43:18,147 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/wandb-summary.json
586
+ 2024-02-01 17:43:19,149 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/output.log
587
+ 2024-02-01 17:43:19,921 DEBUG HandlerThread:237521 [handler.py:handle_request():146] handle_request: stop_status
588
+ 2024-02-01 17:43:19,921 DEBUG SenderThread:237521 [sender.py:send_request():409] send_request: stop_status
589
+ 2024-02-01 17:43:19,957 DEBUG HandlerThread:237521 [handler.py:handle_request():146] handle_request: internal_messages
590
+ 2024-02-01 17:43:21,151 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/output.log
591
+ 2024-02-01 17:43:22,250 DEBUG HandlerThread:237521 [handler.py:handle_request():146] handle_request: partial_history
592
+ 2024-02-01 17:43:22,251 DEBUG SenderThread:237521 [sender.py:send():382] send: history
593
+ 2024-02-01 17:43:22,252 DEBUG SenderThread:237521 [sender.py:send_request():409] send_request: summary_record
594
+ 2024-02-01 17:43:22,253 INFO SenderThread:237521 [sender.py:_save_file():1392] saving file wandb-summary.json with policy end
595
+ 2024-02-01 17:43:23,155 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/wandb-summary.json
596
+ 2024-02-01 17:43:23,155 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/output.log
597
+ 2024-02-01 17:43:23,164 DEBUG HandlerThread:237521 [handler.py:handle_request():146] handle_request: status_report
598
+ 2024-02-01 17:43:25,157 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/output.log
599
+ 2024-02-01 17:43:26,819 DEBUG HandlerThread:237521 [handler.py:handle_request():146] handle_request: partial_history
600
+ 2024-02-01 17:43:26,820 DEBUG SenderThread:237521 [sender.py:send():382] send: history
601
+ 2024-02-01 17:43:26,821 DEBUG SenderThread:237521 [sender.py:send_request():409] send_request: summary_record
602
+ 2024-02-01 17:43:26,823 INFO SenderThread:237521 [sender.py:_save_file():1392] saving file wandb-summary.json with policy end
603
+ 2024-02-01 17:43:27,161 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/wandb-summary.json
604
+ 2024-02-01 17:43:27,161 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/output.log
605
+ 2024-02-01 17:43:28,649 DEBUG HandlerThread:237521 [handler.py:handle_request():146] handle_request: status_report
606
+ 2024-02-01 17:43:28,856 DEBUG SenderThread:237521 [sender.py:send():382] send: stats
607
+ 2024-02-01 17:43:29,164 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/output.log
608
+ 2024-02-01 17:43:31,166 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/output.log
609
+ 2024-02-01 17:43:31,392 DEBUG HandlerThread:237521 [handler.py:handle_request():146] handle_request: partial_history
610
+ 2024-02-01 17:43:31,394 DEBUG SenderThread:237521 [sender.py:send():382] send: history
611
+ 2024-02-01 17:43:31,394 DEBUG SenderThread:237521 [sender.py:send_request():409] send_request: summary_record
612
+ 2024-02-01 17:43:31,396 INFO SenderThread:237521 [sender.py:_save_file():1392] saving file wandb-summary.json with policy end
613
+ 2024-02-01 17:43:32,169 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/wandb-summary.json
614
+ 2024-02-01 17:43:32,169 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/output.log
615
+ 2024-02-01 17:43:34,139 DEBUG HandlerThread:237521 [handler.py:handle_request():146] handle_request: status_report
616
+ 2024-02-01 17:43:34,921 DEBUG HandlerThread:237521 [handler.py:handle_request():146] handle_request: stop_status
617
+ 2024-02-01 17:43:34,921 DEBUG SenderThread:237521 [sender.py:send_request():409] send_request: stop_status
618
+ 2024-02-01 17:43:34,958 DEBUG HandlerThread:237521 [handler.py:handle_request():146] handle_request: internal_messages
619
+ 2024-02-01 17:43:35,172 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/output.log
620
+ 2024-02-01 17:43:36,014 DEBUG HandlerThread:237521 [handler.py:handle_request():146] handle_request: partial_history
621
+ 2024-02-01 17:43:36,016 DEBUG SenderThread:237521 [sender.py:send():382] send: history
622
+ 2024-02-01 17:43:36,016 DEBUG SenderThread:237521 [sender.py:send_request():409] send_request: summary_record
623
+ 2024-02-01 17:43:36,018 INFO SenderThread:237521 [sender.py:_save_file():1392] saving file wandb-summary.json with policy end
624
+ 2024-02-01 17:43:36,175 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/wandb-summary.json
625
+ 2024-02-01 17:43:37,176 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/output.log
626
+ 2024-02-01 17:43:38,177 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/output.log
627
+ 2024-02-01 17:43:39,634 DEBUG HandlerThread:237521 [handler.py:handle_request():146] handle_request: status_report
628
+ 2024-02-01 17:43:40,551 DEBUG HandlerThread:237521 [handler.py:handle_request():146] handle_request: partial_history
629
+ 2024-02-01 17:43:40,552 DEBUG SenderThread:237521 [sender.py:send():382] send: history
630
+ 2024-02-01 17:43:40,552 DEBUG SenderThread:237521 [sender.py:send_request():409] send_request: summary_record
631
+ 2024-02-01 17:43:40,554 INFO SenderThread:237521 [sender.py:_save_file():1392] saving file wandb-summary.json with policy end
632
+ 2024-02-01 17:43:41,182 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/wandb-summary.json
633
+ 2024-02-01 17:43:41,182 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/output.log
634
+ 2024-02-01 17:43:42,183 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/output.log
635
+ 2024-02-01 17:43:44,713 DEBUG HandlerThread:237521 [handler.py:handle_request():146] handle_request: status_report
636
+ 2024-02-01 17:43:46,190 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/output.log
637
+ 2024-02-01 17:43:49,818 DEBUG HandlerThread:237521 [handler.py:handle_request():146] handle_request: status_report
638
+ 2024-02-01 17:43:49,921 DEBUG HandlerThread:237521 [handler.py:handle_request():146] handle_request: stop_status
639
+ 2024-02-01 17:43:49,921 DEBUG SenderThread:237521 [sender.py:send_request():409] send_request: stop_status
640
+ 2024-02-01 17:43:49,958 DEBUG HandlerThread:237521 [handler.py:handle_request():146] handle_request: internal_messages
641
+ 2024-02-01 17:43:50,196 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/output.log
642
+ 2024-02-01 17:43:53,199 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/output.log
643
+ 2024-02-01 17:43:55,675 DEBUG HandlerThread:237521 [handler.py:handle_request():146] handle_request: status_report
644
+ 2024-02-01 17:43:57,204 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/output.log
645
+ 2024-02-01 17:43:58,858 DEBUG SenderThread:237521 [sender.py:send():382] send: stats
646
+ 2024-02-01 17:43:59,206 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/output.log
647
+ 2024-02-01 17:43:59,738 DEBUG HandlerThread:237521 [handler.py:handle_request():146] handle_request: partial_history
648
+ 2024-02-01 17:43:59,739 DEBUG SenderThread:237521 [sender.py:send():382] send: history
649
+ 2024-02-01 17:43:59,739 DEBUG SenderThread:237521 [sender.py:send_request():409] send_request: summary_record
650
+ 2024-02-01 17:43:59,742 INFO SenderThread:237521 [sender.py:_save_file():1392] saving file wandb-summary.json with policy end
651
+ 2024-02-01 17:44:00,209 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/wandb-summary.json
652
+ 2024-02-01 17:44:01,210 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/output.log
653
+ 2024-02-01 17:44:01,563 DEBUG HandlerThread:237521 [handler.py:handle_request():146] handle_request: status_report
654
+ 2024-02-01 17:44:02,211 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/output.log
655
+ 2024-02-01 17:44:04,300 DEBUG HandlerThread:237521 [handler.py:handle_request():146] handle_request: partial_history
656
+ 2024-02-01 17:44:04,301 DEBUG SenderThread:237521 [sender.py:send():382] send: history
657
+ 2024-02-01 17:44:04,301 DEBUG SenderThread:237521 [sender.py:send_request():409] send_request: summary_record
658
+ 2024-02-01 17:44:04,303 INFO SenderThread:237521 [sender.py:_save_file():1392] saving file wandb-summary.json with policy end
659
+ 2024-02-01 17:44:04,921 DEBUG HandlerThread:237521 [handler.py:handle_request():146] handle_request: stop_status
660
+ 2024-02-01 17:44:04,921 DEBUG SenderThread:237521 [sender.py:send_request():409] send_request: stop_status
661
+ 2024-02-01 17:44:04,958 DEBUG HandlerThread:237521 [handler.py:handle_request():146] handle_request: internal_messages
662
+ 2024-02-01 17:44:05,216 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/wandb-summary.json
663
+ 2024-02-01 17:44:05,216 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/output.log
664
+ 2024-02-01 17:44:06,217 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/output.log
665
+ 2024-02-01 17:44:07,030 DEBUG HandlerThread:237521 [handler.py:handle_request():146] handle_request: status_report
666
+ 2024-02-01 17:44:08,853 DEBUG HandlerThread:237521 [handler.py:handle_request():146] handle_request: partial_history
667
+ 2024-02-01 17:44:08,854 DEBUG SenderThread:237521 [sender.py:send():382] send: history
668
+ 2024-02-01 17:44:08,854 DEBUG SenderThread:237521 [sender.py:send_request():409] send_request: summary_record
669
+ 2024-02-01 17:44:08,856 INFO SenderThread:237521 [sender.py:_save_file():1392] saving file wandb-summary.json with policy end
670
+ 2024-02-01 17:44:09,222 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/wandb-summary.json
671
+ 2024-02-01 17:44:09,222 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/output.log
672
+ 2024-02-01 17:44:10,223 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/output.log
673
+ 2024-02-01 17:44:12,507 DEBUG HandlerThread:237521 [handler.py:handle_request():146] handle_request: status_report
674
+ 2024-02-01 17:44:13,227 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/output.log
675
+ 2024-02-01 17:44:13,417 DEBUG HandlerThread:237521 [handler.py:handle_request():146] handle_request: partial_history
676
+ 2024-02-01 17:44:13,419 DEBUG SenderThread:237521 [sender.py:send():382] send: history
677
+ 2024-02-01 17:44:13,419 DEBUG SenderThread:237521 [sender.py:send_request():409] send_request: summary_record
678
+ 2024-02-01 17:44:13,421 INFO SenderThread:237521 [sender.py:_save_file():1392] saving file wandb-summary.json with policy end
679
+ 2024-02-01 17:44:14,230 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/wandb-summary.json
680
+ 2024-02-01 17:44:14,230 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/output.log
681
+ 2024-02-01 17:44:17,234 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/output.log
682
+ 2024-02-01 17:44:17,978 DEBUG HandlerThread:237521 [handler.py:handle_request():146] handle_request: status_report
683
+ 2024-02-01 17:44:17,979 DEBUG HandlerThread:237521 [handler.py:handle_request():146] handle_request: partial_history
684
+ 2024-02-01 17:44:17,981 DEBUG SenderThread:237521 [sender.py:send():382] send: history
685
+ 2024-02-01 17:44:17,981 DEBUG SenderThread:237521 [sender.py:send_request():409] send_request: summary_record
686
+ 2024-02-01 17:44:17,983 INFO SenderThread:237521 [sender.py:_save_file():1392] saving file wandb-summary.json with policy end
687
+ 2024-02-01 17:44:18,236 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/wandb-summary.json
688
+ 2024-02-01 17:44:19,237 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/output.log
689
+ 2024-02-01 17:44:19,921 DEBUG HandlerThread:237521 [handler.py:handle_request():146] handle_request: stop_status
690
+ 2024-02-01 17:44:19,922 DEBUG SenderThread:237521 [sender.py:send_request():409] send_request: stop_status
691
+ 2024-02-01 17:44:19,958 DEBUG HandlerThread:237521 [handler.py:handle_request():146] handle_request: internal_messages
692
+ 2024-02-01 17:44:20,239 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/output.log
693
+ 2024-02-01 17:44:22,545 DEBUG HandlerThread:237521 [handler.py:handle_request():146] handle_request: partial_history
694
+ 2024-02-01 17:44:22,546 DEBUG SenderThread:237521 [sender.py:send():382] send: history
695
+ 2024-02-01 17:44:22,546 DEBUG SenderThread:237521 [sender.py:send_request():409] send_request: summary_record
696
+ 2024-02-01 17:44:22,548 INFO SenderThread:237521 [sender.py:_save_file():1392] saving file wandb-summary.json with policy end
697
+ 2024-02-01 17:44:23,243 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/wandb-summary.json
698
+ 2024-02-01 17:44:23,244 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/output.log
699
+ 2024-02-01 17:44:23,457 DEBUG HandlerThread:237521 [handler.py:handle_request():146] handle_request: status_report
700
+ 2024-02-01 17:44:24,245 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/output.log
701
+ 2024-02-01 17:44:27,120 DEBUG HandlerThread:237521 [handler.py:handle_request():146] handle_request: partial_history
702
+ 2024-02-01 17:44:27,121 DEBUG SenderThread:237521 [sender.py:send():382] send: history
703
+ 2024-02-01 17:44:27,121 DEBUG SenderThread:237521 [sender.py:send_request():409] send_request: summary_record
704
+ 2024-02-01 17:44:27,123 INFO SenderThread:237521 [sender.py:_save_file():1392] saving file wandb-summary.json with policy end
705
+ 2024-02-01 17:44:27,249 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/wandb-summary.json
706
+ 2024-02-01 17:44:27,250 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/output.log
707
+ 2024-02-01 17:44:28,251 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/output.log
708
+ 2024-02-01 17:44:28,860 DEBUG SenderThread:237521 [sender.py:send():382] send: stats
709
+ 2024-02-01 17:44:28,860 DEBUG HandlerThread:237521 [handler.py:handle_request():146] handle_request: status_report
710
+ 2024-02-01 17:44:31,255 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/output.log
711
+ 2024-02-01 17:44:31,688 DEBUG HandlerThread:237521 [handler.py:handle_request():146] handle_request: partial_history
712
+ 2024-02-01 17:44:31,689 DEBUG SenderThread:237521 [sender.py:send():382] send: history
713
+ 2024-02-01 17:44:31,690 DEBUG SenderThread:237521 [sender.py:send_request():409] send_request: summary_record
714
+ 2024-02-01 17:44:31,692 INFO SenderThread:237521 [sender.py:_save_file():1392] saving file wandb-summary.json with policy end
715
+ 2024-02-01 17:44:32,257 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/wandb-summary.json
716
+ 2024-02-01 17:44:32,257 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/output.log
717
+ 2024-02-01 17:44:34,432 DEBUG HandlerThread:237521 [handler.py:handle_request():146] handle_request: status_report
718
+ 2024-02-01 17:44:34,921 DEBUG HandlerThread:237521 [handler.py:handle_request():146] handle_request: stop_status
719
+ 2024-02-01 17:44:34,922 DEBUG SenderThread:237521 [sender.py:send_request():409] send_request: stop_status
720
+ 2024-02-01 17:44:34,958 DEBUG HandlerThread:237521 [handler.py:handle_request():146] handle_request: internal_messages
721
+ 2024-02-01 17:44:35,261 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/output.log
722
+ 2024-02-01 17:44:36,258 DEBUG HandlerThread:237521 [handler.py:handle_request():146] handle_request: partial_history
723
+ 2024-02-01 17:44:36,260 DEBUG SenderThread:237521 [sender.py:send():382] send: history
724
+ 2024-02-01 17:44:36,260 DEBUG SenderThread:237521 [sender.py:send_request():409] send_request: summary_record
725
+ 2024-02-01 17:44:36,262 INFO SenderThread:237521 [sender.py:_save_file():1392] saving file wandb-summary.json with policy end
726
+ 2024-02-01 17:44:36,263 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/wandb-summary.json
727
+ 2024-02-01 17:44:37,264 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/output.log
728
+ 2024-02-01 17:44:38,266 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/output.log
729
+ 2024-02-01 17:44:39,916 DEBUG HandlerThread:237521 [handler.py:handle_request():146] handle_request: status_report
730
+ 2024-02-01 17:44:40,829 DEBUG HandlerThread:237521 [handler.py:handle_request():146] handle_request: partial_history
731
+ 2024-02-01 17:44:40,830 DEBUG SenderThread:237521 [sender.py:send():382] send: history
732
+ 2024-02-01 17:44:40,830 DEBUG SenderThread:237521 [sender.py:send_request():409] send_request: summary_record
733
+ 2024-02-01 17:44:40,832 INFO SenderThread:237521 [sender.py:_save_file():1392] saving file wandb-summary.json with policy end
734
+ 2024-02-01 17:44:41,270 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/wandb-summary.json
735
+ 2024-02-01 17:44:41,271 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/output.log
736
+ 2024-02-01 17:44:42,272 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/output.log
737
+ 2024-02-01 17:44:45,276 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/output.log
738
+ 2024-02-01 17:44:45,408 DEBUG HandlerThread:237521 [handler.py:handle_request():146] handle_request: status_report
739
+ 2024-02-01 17:44:45,409 DEBUG HandlerThread:237521 [handler.py:handle_request():146] handle_request: partial_history
740
+ 2024-02-01 17:44:45,410 DEBUG SenderThread:237521 [sender.py:send():382] send: history
741
+ 2024-02-01 17:44:45,410 DEBUG SenderThread:237521 [sender.py:send_request():409] send_request: summary_record
742
+ 2024-02-01 17:44:45,412 INFO SenderThread:237521 [sender.py:_save_file():1392] saving file wandb-summary.json with policy end
743
+ 2024-02-01 17:44:46,278 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/wandb-summary.json
744
+ 2024-02-01 17:44:46,278 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/output.log
745
+ 2024-02-01 17:44:49,282 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/output.log
746
+ 2024-02-01 17:44:49,922 DEBUG HandlerThread:237521 [handler.py:handle_request():146] handle_request: stop_status
747
+ 2024-02-01 17:44:49,922 DEBUG SenderThread:237521 [sender.py:send_request():409] send_request: stop_status
748
+ 2024-02-01 17:44:49,958 DEBUG HandlerThread:237521 [handler.py:handle_request():146] handle_request: internal_messages
749
+ 2024-02-01 17:44:50,012 DEBUG HandlerThread:237521 [handler.py:handle_request():146] handle_request: partial_history
750
+ 2024-02-01 17:44:50,013 DEBUG SenderThread:237521 [sender.py:send():382] send: history
751
+ 2024-02-01 17:44:50,013 DEBUG SenderThread:237521 [sender.py:send_request():409] send_request: summary_record
752
+ 2024-02-01 17:44:50,015 INFO SenderThread:237521 [sender.py:_save_file():1392] saving file wandb-summary.json with policy end
753
+ 2024-02-01 17:44:50,284 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/wandb-summary.json
754
+ 2024-02-01 17:44:50,896 DEBUG HandlerThread:237521 [handler.py:handle_request():146] handle_request: status_report
755
+ 2024-02-01 17:44:51,286 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/output.log
756
+ 2024-02-01 17:44:52,287 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/output.log
757
+ 2024-02-01 17:44:54,548 DEBUG HandlerThread:237521 [handler.py:handle_request():146] handle_request: partial_history
758
+ 2024-02-01 17:44:54,549 DEBUG SenderThread:237521 [sender.py:send():382] send: history
759
+ 2024-02-01 17:44:54,549 DEBUG SenderThread:237521 [sender.py:send_request():409] send_request: summary_record
760
+ 2024-02-01 17:44:54,552 INFO SenderThread:237521 [sender.py:_save_file():1392] saving file wandb-summary.json with policy end
761
+ 2024-02-01 17:44:55,292 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/wandb-summary.json
762
+ 2024-02-01 17:44:55,292 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/output.log
763
+ 2024-02-01 17:44:56,293 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/output.log
764
+ 2024-02-01 17:44:56,373 DEBUG HandlerThread:237521 [handler.py:handle_request():146] handle_request: status_report
wandb/run-20240201_173828-py26nu6m/files/output.log CHANGED
@@ -211,3 +211,86 @@
211
  24%|███████████████████▍ | 264/1090 [04:33<12:37, 1.09it/s]
212
 
213
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
211
  24%|███████████████████▍ | 264/1090 [04:33<12:37, 1.09it/s]
212
 
213
 
214
+ 25%|███████████████████▋ | 269/1090 [04:37<12:32, 1.09it/s]
215
+
216
+
217
+ 25%|████████████████████ | 273/1090 [04:41<12:28, 1.09it/s]
218
+
219
+
220
+
221
+ 26%|████████████████████▍ | 279/1090 [04:46<12:22, 1.09it/s]
222
+
223
+
224
+ 26%|████████████████████▊ | 284/1090 [04:51<12:15, 1.10it/s]
225
+
226
+
227
+ 26%|█████████████████████▏ | 288/1090 [04:55<12:13, 1.09it/s]
228
+
229
+
230
+
231
+ 27%|█████████████████████▋ | 295/1090 [05:01<12:08, 1.09it/s]
232
+
233
+
234
+ 27%|█████████████████████▉ | 299/1090 [05:05<12:03, 1.09it/s]
235
+ 28%|██████████████████████ | 300/1090 [05:06<12:03, 1.09it/s][INFO|trainer.py:2889] 2024-02-01 17:43:41,617 >> Saving model checkpoint to ./tmp-checkpoint-300
236
+ [INFO|configuration_utils.py:483] 2024-02-01 17:43:41,621 >> Configuration saved in ./tmp-checkpoint-300/config.json
237
+ [INFO|configuration_utils.py:594] 2024-02-01 17:43:41,623 >> Configuration saved in ./tmp-checkpoint-300/generation_config.json
238
+ [2024-02-01 17:43:44,742] [INFO] [logging.py:96:log_dist] [Rank 0] [Torch] Checkpoint global_step300 is about to be saved!
239
+ [2024-02-01 17:43:44,746] [INFO] [logging.py:96:log_dist] [Rank 0] Saving model checkpoint: ./tmp-checkpoint-300/global_step300/zero_pp_rank_0_mp_rank_00_model_states.pt
240
+ [2024-02-01 17:43:44,747] [INFO] [torch_checkpoint_engine.py:21:save] [Torch] Saving ./tmp-checkpoint-300/global_step300/zero_pp_rank_0_mp_rank_00_model_states.pt...
241
+ [2024-02-01 17:43:44,751] [INFO] [torch_checkpoint_engine.py:23:save] [Torch] Saved ./tmp-checkpoint-300/global_step300/zero_pp_rank_0_mp_rank_00_model_states.pt.
242
+ [2024-02-01 17:43:44,759] [INFO] [torch_checkpoint_engine.py:21:save] [Torch] Saving ./tmp-checkpoint-300/global_step300/bf16_zero_pp_rank_0_mp_rank_00_optim_states.pt...
243
+ [INFO|modeling_utils.py:2382] 2024-02-01 17:43:44,712 >> Model weights saved in ./tmp-checkpoint-300/pytorch_model.bin
244
+ [INFO|tokenization_utils_base.py:2432] 2024-02-01 17:43:44,715 >> tokenizer config file saved in ./tmp-checkpoint-300/tokenizer_config.json
245
+ [INFO|tokenization_utils_base.py:2441] 2024-02-01 17:43:44,717 >> Special tokens file saved in ./tmp-checkpoint-300/special_tokens_map.json
246
+ /fsx/sanchit/miniconda3/envs/venv/lib/python3.11/site-packages/torch/nn/modules/module.py:1879: UserWarning: Positional args are being deprecated, use kwargs instead. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details.
247
+ warnings.warn(
248
+ [2024-02-01 17:43:48,812] [INFO] [torch_checkpoint_engine.py:23:save] [Torch] Saved ./tmp-checkpoint-300/global_step300/bf16_zero_pp_rank_0_mp_rank_00_optim_states.pt.
249
+ [2024-02-01 17:43:48,816] [INFO] [engine.py:3393:_save_zero_checkpoint] zero checkpoint saved ./tmp-checkpoint-300/global_step300/bf16_zero_pp_rank_0_mp_rank_00_optim_states.pt
250
+ [2024-02-01 17:43:48,817] [INFO] [torch_checkpoint_engine.py:33:commit] [Torch] Checkpoint global_step300 is ready now!
251
+ [INFO|tokenization_utils_base.py:2432] 2024-02-01 17:43:51,631 >> tokenizer config file saved in ./tokenizer_config.json
252
+ [INFO|tokenization_utils_base.py:2441] 2024-02-01 17:43:51,633 >> Special tokens file saved in ./special_tokens_map.json
253
+ [INFO|trainer.py:2979] 2024-02-01 17:43:51,660 >> Deleting older checkpoint [checkpoint-200] due to args.save_total_limit
254
+
255
+
256
+ 28%|██████████████████████▍ | 305/1090 [05:25<25:42, 1.96s/it]
257
+
258
+
259
+ 28%|██████████████████████▋ | 309/1090 [05:28<15:09, 1.16s/it]
260
+
261
+
262
+ 29%|███████████████████████ | 314/1090 [05:33<12:20, 1.05it/s]
263
+
264
+
265
+ 29%|███████████████████████▎ | 318/1090 [05:37<11:52, 1.08it/s]
266
+
267
+
268
+
269
+ 30%|███████████████████████▊ | 325/1090 [05:43<11:38, 1.09it/s]
270
+
271
+
272
+ 30%|████████████████████████▏ | 329/1090 [05:47<11:34, 1.10it/s]
273
+
274
+
275
+ 31%|████████████████████████▌ | 334/1090 [05:51<11:31, 1.09it/s]
276
+
277
+
278
+ 31%|████████████████████████▊ | 338/1090 [05:55<11:26, 1.09it/s]
279
+
280
+
281
+
282
+ 32%|█████████████████████████▏ | 344/1090 [06:00<11:22, 1.09it/s]
283
+
284
+
285
+ 32%|█████████████████████████▌ | 349/1090 [06:05<11:17, 1.09it/s]
286
+
287
+
288
+ 32%|█████████████████████████▉ | 353/1090 [06:09<11:15, 1.09it/s]
289
+
290
+
291
+
292
+ 33%|██████████████████████████▍ | 360/1090 [06:15<11:07, 1.09it/s]
293
+
294
+
295
+ 33%|██████████████████████████▋ | 364/1090 [06:19<11:02, 1.10it/s]
296
+
wandb/run-20240201_173828-py26nu6m/files/wandb-summary.json CHANGED
@@ -1 +1 @@
1
- {"train/loss": 5.3788, "train/learning_rate": 1.8700006548379898e-05, "train/epoch": 0.25, "train/global_step": 270, "_timestamp": 1706809393.1073587, "_runtime": 284.6554846763611, "_step": 54}
 
1
+ {"train/loss": 5.332, "train/learning_rate": 1.6823495987048922e-05, "train/epoch": 0.33, "train/global_step": 365, "_timestamp": 1706809494.5481908, "_runtime": 386.0963168144226, "_step": 73}
wandb/run-20240201_173828-py26nu6m/logs/debug-internal.log CHANGED
@@ -575,3 +575,190 @@
575
  2024-02-01 17:43:13,111 INFO SenderThread:237521 [sender.py:_save_file():1392] saving file wandb-summary.json with policy end
576
  2024-02-01 17:43:13,140 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/wandb-summary.json
577
  2024-02-01 17:43:13,140 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/output.log
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
575
  2024-02-01 17:43:13,111 INFO SenderThread:237521 [sender.py:_save_file():1392] saving file wandb-summary.json with policy end
576
  2024-02-01 17:43:13,140 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/wandb-summary.json
577
  2024-02-01 17:43:13,140 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/output.log
578
+ 2024-02-01 17:43:15,143 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/output.log
579
+ 2024-02-01 17:43:17,145 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/output.log
580
+ 2024-02-01 17:43:17,675 DEBUG HandlerThread:237521 [handler.py:handle_request():146] handle_request: status_report
581
+ 2024-02-01 17:43:17,676 DEBUG HandlerThread:237521 [handler.py:handle_request():146] handle_request: partial_history
582
+ 2024-02-01 17:43:17,678 DEBUG SenderThread:237521 [sender.py:send():382] send: history
583
+ 2024-02-01 17:43:17,678 DEBUG SenderThread:237521 [sender.py:send_request():409] send_request: summary_record
584
+ 2024-02-01 17:43:17,680 INFO SenderThread:237521 [sender.py:_save_file():1392] saving file wandb-summary.json with policy end
585
+ 2024-02-01 17:43:18,147 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/wandb-summary.json
586
+ 2024-02-01 17:43:19,149 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/output.log
587
+ 2024-02-01 17:43:19,921 DEBUG HandlerThread:237521 [handler.py:handle_request():146] handle_request: stop_status
588
+ 2024-02-01 17:43:19,921 DEBUG SenderThread:237521 [sender.py:send_request():409] send_request: stop_status
589
+ 2024-02-01 17:43:19,957 DEBUG HandlerThread:237521 [handler.py:handle_request():146] handle_request: internal_messages
590
+ 2024-02-01 17:43:21,151 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/output.log
591
+ 2024-02-01 17:43:22,250 DEBUG HandlerThread:237521 [handler.py:handle_request():146] handle_request: partial_history
592
+ 2024-02-01 17:43:22,251 DEBUG SenderThread:237521 [sender.py:send():382] send: history
593
+ 2024-02-01 17:43:22,252 DEBUG SenderThread:237521 [sender.py:send_request():409] send_request: summary_record
594
+ 2024-02-01 17:43:22,253 INFO SenderThread:237521 [sender.py:_save_file():1392] saving file wandb-summary.json with policy end
595
+ 2024-02-01 17:43:23,155 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/wandb-summary.json
596
+ 2024-02-01 17:43:23,155 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/output.log
597
+ 2024-02-01 17:43:23,164 DEBUG HandlerThread:237521 [handler.py:handle_request():146] handle_request: status_report
598
+ 2024-02-01 17:43:25,157 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/output.log
599
+ 2024-02-01 17:43:26,819 DEBUG HandlerThread:237521 [handler.py:handle_request():146] handle_request: partial_history
600
+ 2024-02-01 17:43:26,820 DEBUG SenderThread:237521 [sender.py:send():382] send: history
601
+ 2024-02-01 17:43:26,821 DEBUG SenderThread:237521 [sender.py:send_request():409] send_request: summary_record
602
+ 2024-02-01 17:43:26,823 INFO SenderThread:237521 [sender.py:_save_file():1392] saving file wandb-summary.json with policy end
603
+ 2024-02-01 17:43:27,161 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/wandb-summary.json
604
+ 2024-02-01 17:43:27,161 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/output.log
605
+ 2024-02-01 17:43:28,649 DEBUG HandlerThread:237521 [handler.py:handle_request():146] handle_request: status_report
606
+ 2024-02-01 17:43:28,856 DEBUG SenderThread:237521 [sender.py:send():382] send: stats
607
+ 2024-02-01 17:43:29,164 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/output.log
608
+ 2024-02-01 17:43:31,166 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/output.log
609
+ 2024-02-01 17:43:31,392 DEBUG HandlerThread:237521 [handler.py:handle_request():146] handle_request: partial_history
610
+ 2024-02-01 17:43:31,394 DEBUG SenderThread:237521 [sender.py:send():382] send: history
611
+ 2024-02-01 17:43:31,394 DEBUG SenderThread:237521 [sender.py:send_request():409] send_request: summary_record
612
+ 2024-02-01 17:43:31,396 INFO SenderThread:237521 [sender.py:_save_file():1392] saving file wandb-summary.json with policy end
613
+ 2024-02-01 17:43:32,169 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/wandb-summary.json
614
+ 2024-02-01 17:43:32,169 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/output.log
615
+ 2024-02-01 17:43:34,139 DEBUG HandlerThread:237521 [handler.py:handle_request():146] handle_request: status_report
616
+ 2024-02-01 17:43:34,921 DEBUG HandlerThread:237521 [handler.py:handle_request():146] handle_request: stop_status
617
+ 2024-02-01 17:43:34,921 DEBUG SenderThread:237521 [sender.py:send_request():409] send_request: stop_status
618
+ 2024-02-01 17:43:34,958 DEBUG HandlerThread:237521 [handler.py:handle_request():146] handle_request: internal_messages
619
+ 2024-02-01 17:43:35,172 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/output.log
620
+ 2024-02-01 17:43:36,014 DEBUG HandlerThread:237521 [handler.py:handle_request():146] handle_request: partial_history
621
+ 2024-02-01 17:43:36,016 DEBUG SenderThread:237521 [sender.py:send():382] send: history
622
+ 2024-02-01 17:43:36,016 DEBUG SenderThread:237521 [sender.py:send_request():409] send_request: summary_record
623
+ 2024-02-01 17:43:36,018 INFO SenderThread:237521 [sender.py:_save_file():1392] saving file wandb-summary.json with policy end
624
+ 2024-02-01 17:43:36,175 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/wandb-summary.json
625
+ 2024-02-01 17:43:37,176 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/output.log
626
+ 2024-02-01 17:43:38,177 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/output.log
627
+ 2024-02-01 17:43:39,634 DEBUG HandlerThread:237521 [handler.py:handle_request():146] handle_request: status_report
628
+ 2024-02-01 17:43:40,551 DEBUG HandlerThread:237521 [handler.py:handle_request():146] handle_request: partial_history
629
+ 2024-02-01 17:43:40,552 DEBUG SenderThread:237521 [sender.py:send():382] send: history
630
+ 2024-02-01 17:43:40,552 DEBUG SenderThread:237521 [sender.py:send_request():409] send_request: summary_record
631
+ 2024-02-01 17:43:40,554 INFO SenderThread:237521 [sender.py:_save_file():1392] saving file wandb-summary.json with policy end
632
+ 2024-02-01 17:43:41,182 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/wandb-summary.json
633
+ 2024-02-01 17:43:41,182 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/output.log
634
+ 2024-02-01 17:43:42,183 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/output.log
635
+ 2024-02-01 17:43:44,713 DEBUG HandlerThread:237521 [handler.py:handle_request():146] handle_request: status_report
636
+ 2024-02-01 17:43:46,190 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/output.log
637
+ 2024-02-01 17:43:49,818 DEBUG HandlerThread:237521 [handler.py:handle_request():146] handle_request: status_report
638
+ 2024-02-01 17:43:49,921 DEBUG HandlerThread:237521 [handler.py:handle_request():146] handle_request: stop_status
639
+ 2024-02-01 17:43:49,921 DEBUG SenderThread:237521 [sender.py:send_request():409] send_request: stop_status
640
+ 2024-02-01 17:43:49,958 DEBUG HandlerThread:237521 [handler.py:handle_request():146] handle_request: internal_messages
641
+ 2024-02-01 17:43:50,196 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/output.log
642
+ 2024-02-01 17:43:53,199 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/output.log
643
+ 2024-02-01 17:43:55,675 DEBUG HandlerThread:237521 [handler.py:handle_request():146] handle_request: status_report
644
+ 2024-02-01 17:43:57,204 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/output.log
645
+ 2024-02-01 17:43:58,858 DEBUG SenderThread:237521 [sender.py:send():382] send: stats
646
+ 2024-02-01 17:43:59,206 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/output.log
647
+ 2024-02-01 17:43:59,738 DEBUG HandlerThread:237521 [handler.py:handle_request():146] handle_request: partial_history
648
+ 2024-02-01 17:43:59,739 DEBUG SenderThread:237521 [sender.py:send():382] send: history
649
+ 2024-02-01 17:43:59,739 DEBUG SenderThread:237521 [sender.py:send_request():409] send_request: summary_record
650
+ 2024-02-01 17:43:59,742 INFO SenderThread:237521 [sender.py:_save_file():1392] saving file wandb-summary.json with policy end
651
+ 2024-02-01 17:44:00,209 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/wandb-summary.json
652
+ 2024-02-01 17:44:01,210 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/output.log
653
+ 2024-02-01 17:44:01,563 DEBUG HandlerThread:237521 [handler.py:handle_request():146] handle_request: status_report
654
+ 2024-02-01 17:44:02,211 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/output.log
655
+ 2024-02-01 17:44:04,300 DEBUG HandlerThread:237521 [handler.py:handle_request():146] handle_request: partial_history
656
+ 2024-02-01 17:44:04,301 DEBUG SenderThread:237521 [sender.py:send():382] send: history
657
+ 2024-02-01 17:44:04,301 DEBUG SenderThread:237521 [sender.py:send_request():409] send_request: summary_record
658
+ 2024-02-01 17:44:04,303 INFO SenderThread:237521 [sender.py:_save_file():1392] saving file wandb-summary.json with policy end
659
+ 2024-02-01 17:44:04,921 DEBUG HandlerThread:237521 [handler.py:handle_request():146] handle_request: stop_status
660
+ 2024-02-01 17:44:04,921 DEBUG SenderThread:237521 [sender.py:send_request():409] send_request: stop_status
661
+ 2024-02-01 17:44:04,958 DEBUG HandlerThread:237521 [handler.py:handle_request():146] handle_request: internal_messages
662
+ 2024-02-01 17:44:05,216 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/wandb-summary.json
663
+ 2024-02-01 17:44:05,216 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/output.log
664
+ 2024-02-01 17:44:06,217 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/output.log
665
+ 2024-02-01 17:44:07,030 DEBUG HandlerThread:237521 [handler.py:handle_request():146] handle_request: status_report
666
+ 2024-02-01 17:44:08,853 DEBUG HandlerThread:237521 [handler.py:handle_request():146] handle_request: partial_history
667
+ 2024-02-01 17:44:08,854 DEBUG SenderThread:237521 [sender.py:send():382] send: history
668
+ 2024-02-01 17:44:08,854 DEBUG SenderThread:237521 [sender.py:send_request():409] send_request: summary_record
669
+ 2024-02-01 17:44:08,856 INFO SenderThread:237521 [sender.py:_save_file():1392] saving file wandb-summary.json with policy end
670
+ 2024-02-01 17:44:09,222 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/wandb-summary.json
671
+ 2024-02-01 17:44:09,222 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/output.log
672
+ 2024-02-01 17:44:10,223 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/output.log
673
+ 2024-02-01 17:44:12,507 DEBUG HandlerThread:237521 [handler.py:handle_request():146] handle_request: status_report
674
+ 2024-02-01 17:44:13,227 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/output.log
675
+ 2024-02-01 17:44:13,417 DEBUG HandlerThread:237521 [handler.py:handle_request():146] handle_request: partial_history
676
+ 2024-02-01 17:44:13,419 DEBUG SenderThread:237521 [sender.py:send():382] send: history
677
+ 2024-02-01 17:44:13,419 DEBUG SenderThread:237521 [sender.py:send_request():409] send_request: summary_record
678
+ 2024-02-01 17:44:13,421 INFO SenderThread:237521 [sender.py:_save_file():1392] saving file wandb-summary.json with policy end
679
+ 2024-02-01 17:44:14,230 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/wandb-summary.json
680
+ 2024-02-01 17:44:14,230 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/output.log
681
+ 2024-02-01 17:44:17,234 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/output.log
682
+ 2024-02-01 17:44:17,978 DEBUG HandlerThread:237521 [handler.py:handle_request():146] handle_request: status_report
683
+ 2024-02-01 17:44:17,979 DEBUG HandlerThread:237521 [handler.py:handle_request():146] handle_request: partial_history
684
+ 2024-02-01 17:44:17,981 DEBUG SenderThread:237521 [sender.py:send():382] send: history
685
+ 2024-02-01 17:44:17,981 DEBUG SenderThread:237521 [sender.py:send_request():409] send_request: summary_record
686
+ 2024-02-01 17:44:17,983 INFO SenderThread:237521 [sender.py:_save_file():1392] saving file wandb-summary.json with policy end
687
+ 2024-02-01 17:44:18,236 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/wandb-summary.json
688
+ 2024-02-01 17:44:19,237 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/output.log
689
+ 2024-02-01 17:44:19,921 DEBUG HandlerThread:237521 [handler.py:handle_request():146] handle_request: stop_status
690
+ 2024-02-01 17:44:19,922 DEBUG SenderThread:237521 [sender.py:send_request():409] send_request: stop_status
691
+ 2024-02-01 17:44:19,958 DEBUG HandlerThread:237521 [handler.py:handle_request():146] handle_request: internal_messages
692
+ 2024-02-01 17:44:20,239 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/output.log
693
+ 2024-02-01 17:44:22,545 DEBUG HandlerThread:237521 [handler.py:handle_request():146] handle_request: partial_history
694
+ 2024-02-01 17:44:22,546 DEBUG SenderThread:237521 [sender.py:send():382] send: history
695
+ 2024-02-01 17:44:22,546 DEBUG SenderThread:237521 [sender.py:send_request():409] send_request: summary_record
696
+ 2024-02-01 17:44:22,548 INFO SenderThread:237521 [sender.py:_save_file():1392] saving file wandb-summary.json with policy end
697
+ 2024-02-01 17:44:23,243 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/wandb-summary.json
698
+ 2024-02-01 17:44:23,244 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/output.log
699
+ 2024-02-01 17:44:23,457 DEBUG HandlerThread:237521 [handler.py:handle_request():146] handle_request: status_report
700
+ 2024-02-01 17:44:24,245 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/output.log
701
+ 2024-02-01 17:44:27,120 DEBUG HandlerThread:237521 [handler.py:handle_request():146] handle_request: partial_history
702
+ 2024-02-01 17:44:27,121 DEBUG SenderThread:237521 [sender.py:send():382] send: history
703
+ 2024-02-01 17:44:27,121 DEBUG SenderThread:237521 [sender.py:send_request():409] send_request: summary_record
704
+ 2024-02-01 17:44:27,123 INFO SenderThread:237521 [sender.py:_save_file():1392] saving file wandb-summary.json with policy end
705
+ 2024-02-01 17:44:27,249 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/wandb-summary.json
706
+ 2024-02-01 17:44:27,250 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/output.log
707
+ 2024-02-01 17:44:28,251 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/output.log
708
+ 2024-02-01 17:44:28,860 DEBUG SenderThread:237521 [sender.py:send():382] send: stats
709
+ 2024-02-01 17:44:28,860 DEBUG HandlerThread:237521 [handler.py:handle_request():146] handle_request: status_report
710
+ 2024-02-01 17:44:31,255 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/output.log
711
+ 2024-02-01 17:44:31,688 DEBUG HandlerThread:237521 [handler.py:handle_request():146] handle_request: partial_history
712
+ 2024-02-01 17:44:31,689 DEBUG SenderThread:237521 [sender.py:send():382] send: history
713
+ 2024-02-01 17:44:31,690 DEBUG SenderThread:237521 [sender.py:send_request():409] send_request: summary_record
714
+ 2024-02-01 17:44:31,692 INFO SenderThread:237521 [sender.py:_save_file():1392] saving file wandb-summary.json with policy end
715
+ 2024-02-01 17:44:32,257 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/wandb-summary.json
716
+ 2024-02-01 17:44:32,257 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/output.log
717
+ 2024-02-01 17:44:34,432 DEBUG HandlerThread:237521 [handler.py:handle_request():146] handle_request: status_report
718
+ 2024-02-01 17:44:34,921 DEBUG HandlerThread:237521 [handler.py:handle_request():146] handle_request: stop_status
719
+ 2024-02-01 17:44:34,922 DEBUG SenderThread:237521 [sender.py:send_request():409] send_request: stop_status
720
+ 2024-02-01 17:44:34,958 DEBUG HandlerThread:237521 [handler.py:handle_request():146] handle_request: internal_messages
721
+ 2024-02-01 17:44:35,261 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/output.log
722
+ 2024-02-01 17:44:36,258 DEBUG HandlerThread:237521 [handler.py:handle_request():146] handle_request: partial_history
723
+ 2024-02-01 17:44:36,260 DEBUG SenderThread:237521 [sender.py:send():382] send: history
724
+ 2024-02-01 17:44:36,260 DEBUG SenderThread:237521 [sender.py:send_request():409] send_request: summary_record
725
+ 2024-02-01 17:44:36,262 INFO SenderThread:237521 [sender.py:_save_file():1392] saving file wandb-summary.json with policy end
726
+ 2024-02-01 17:44:36,263 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/wandb-summary.json
727
+ 2024-02-01 17:44:37,264 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/output.log
728
+ 2024-02-01 17:44:38,266 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/output.log
729
+ 2024-02-01 17:44:39,916 DEBUG HandlerThread:237521 [handler.py:handle_request():146] handle_request: status_report
730
+ 2024-02-01 17:44:40,829 DEBUG HandlerThread:237521 [handler.py:handle_request():146] handle_request: partial_history
731
+ 2024-02-01 17:44:40,830 DEBUG SenderThread:237521 [sender.py:send():382] send: history
732
+ 2024-02-01 17:44:40,830 DEBUG SenderThread:237521 [sender.py:send_request():409] send_request: summary_record
733
+ 2024-02-01 17:44:40,832 INFO SenderThread:237521 [sender.py:_save_file():1392] saving file wandb-summary.json with policy end
734
+ 2024-02-01 17:44:41,270 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/wandb-summary.json
735
+ 2024-02-01 17:44:41,271 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/output.log
736
+ 2024-02-01 17:44:42,272 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/output.log
737
+ 2024-02-01 17:44:45,276 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/output.log
738
+ 2024-02-01 17:44:45,408 DEBUG HandlerThread:237521 [handler.py:handle_request():146] handle_request: status_report
739
+ 2024-02-01 17:44:45,409 DEBUG HandlerThread:237521 [handler.py:handle_request():146] handle_request: partial_history
740
+ 2024-02-01 17:44:45,410 DEBUG SenderThread:237521 [sender.py:send():382] send: history
741
+ 2024-02-01 17:44:45,410 DEBUG SenderThread:237521 [sender.py:send_request():409] send_request: summary_record
742
+ 2024-02-01 17:44:45,412 INFO SenderThread:237521 [sender.py:_save_file():1392] saving file wandb-summary.json with policy end
743
+ 2024-02-01 17:44:46,278 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/wandb-summary.json
744
+ 2024-02-01 17:44:46,278 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/output.log
745
+ 2024-02-01 17:44:49,282 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/output.log
746
+ 2024-02-01 17:44:49,922 DEBUG HandlerThread:237521 [handler.py:handle_request():146] handle_request: stop_status
747
+ 2024-02-01 17:44:49,922 DEBUG SenderThread:237521 [sender.py:send_request():409] send_request: stop_status
748
+ 2024-02-01 17:44:49,958 DEBUG HandlerThread:237521 [handler.py:handle_request():146] handle_request: internal_messages
749
+ 2024-02-01 17:44:50,012 DEBUG HandlerThread:237521 [handler.py:handle_request():146] handle_request: partial_history
750
+ 2024-02-01 17:44:50,013 DEBUG SenderThread:237521 [sender.py:send():382] send: history
751
+ 2024-02-01 17:44:50,013 DEBUG SenderThread:237521 [sender.py:send_request():409] send_request: summary_record
752
+ 2024-02-01 17:44:50,015 INFO SenderThread:237521 [sender.py:_save_file():1392] saving file wandb-summary.json with policy end
753
+ 2024-02-01 17:44:50,284 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/wandb-summary.json
754
+ 2024-02-01 17:44:50,896 DEBUG HandlerThread:237521 [handler.py:handle_request():146] handle_request: status_report
755
+ 2024-02-01 17:44:51,286 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/output.log
756
+ 2024-02-01 17:44:52,287 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/output.log
757
+ 2024-02-01 17:44:54,548 DEBUG HandlerThread:237521 [handler.py:handle_request():146] handle_request: partial_history
758
+ 2024-02-01 17:44:54,549 DEBUG SenderThread:237521 [sender.py:send():382] send: history
759
+ 2024-02-01 17:44:54,549 DEBUG SenderThread:237521 [sender.py:send_request():409] send_request: summary_record
760
+ 2024-02-01 17:44:54,552 INFO SenderThread:237521 [sender.py:_save_file():1392] saving file wandb-summary.json with policy end
761
+ 2024-02-01 17:44:55,292 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/wandb-summary.json
762
+ 2024-02-01 17:44:55,292 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/output.log
763
+ 2024-02-01 17:44:56,293 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/output.log
764
+ 2024-02-01 17:44:56,373 DEBUG HandlerThread:237521 [handler.py:handle_request():146] handle_request: status_report
wandb/run-20240201_173828-py26nu6m/run-py26nu6m.wandb CHANGED
Binary files a/wandb/run-20240201_173828-py26nu6m/run-py26nu6m.wandb and b/wandb/run-20240201_173828-py26nu6m/run-py26nu6m.wandb differ