Hugging Face's logo

sanchit-gandhi
/
distil-zephyr-1.5b-ssft

Text Generation
Transformers
TensorBoard
Safetensors
mistral
alignment-handbook
Generated from Trainer
trl
sft
conversational
text-generation-inference
Inference Endpoints
Model card Files Metrics Community
sanchit-gandhi HF staff commited on
Commit
9f7777e
•
1 Parent(s): 6299411

Training in progress, step 400

Browse files
Files changed (7) hide show
  1. model.safetensors +1 -1
  2. runs/Feb01_17-38-02_ip-26-0-165-24/events.out.tfevents.1706809106.ip-26-0-165-24.237059.0 +2 -2
  3. wandb/debug-internal.log +193 -0
  4. wandb/run-20240201_173828-py26nu6m/files/output.log +84 -0
  5. wandb/run-20240201_173828-py26nu6m/files/wandb-summary.json +1 -1
  6. wandb/run-20240201_173828-py26nu6m/logs/debug-internal.log +193 -0
  7. wandb/run-20240201_173828-py26nu6m/run-py26nu6m.wandb +0 -0
model.safetensors CHANGED
@@ -1,3 +1,3 @@
1
  version https://git-lfs.github.com/spec/v1
2
- oid sha256:5726b95fb59862ee5de794319c14f7296f374c5d53f885a1aba0207b8074c149
3
  size 3141646744
 
1
  version https://git-lfs.github.com/spec/v1
2
+ oid sha256:e806ed22e6133d9b25f5c6096b7545b0a898553f8573e473a0cc2daf4b6eaa63
3
  size 3141646744
runs/Feb01_17-38-02_ip-26-0-165-24/events.out.tfevents.1706809106.ip-26-0-165-24.237059.0 CHANGED
@@ -1,3 +1,3 @@
1
  version https://git-lfs.github.com/spec/v1
2
- oid sha256:4aecbf973c9bcd1ddac7d1105844774d635b2bcc5b63e337974801e429f7baa1
3
- size 13874
 
1
  version https://git-lfs.github.com/spec/v1
2
+ oid sha256:e403dbd3805e13d02e41b8f40ddcc16a6de2d91b85ecc6c3c2c8d8ab489f60bb
3
+ size 17014
wandb/debug-internal.log CHANGED
@@ -762,3 +762,196 @@
762
  2024-02-01 17:44:55,292 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/output.log
763
  2024-02-01 17:44:56,293 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/output.log
764
  2024-02-01 17:44:56,373 DEBUG HandlerThread:237521 [handler.py:handle_request():146] handle_request: status_report
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
762
  2024-02-01 17:44:55,292 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/output.log
763
  2024-02-01 17:44:56,293 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/output.log
764
  2024-02-01 17:44:56,373 DEBUG HandlerThread:237521 [handler.py:handle_request():146] handle_request: status_report
765
+ 2024-02-01 17:44:58,861 DEBUG SenderThread:237521 [sender.py:send():382] send: stats
766
+ 2024-02-01 17:44:59,112 DEBUG HandlerThread:237521 [handler.py:handle_request():146] handle_request: partial_history
767
+ 2024-02-01 17:44:59,113 DEBUG SenderThread:237521 [sender.py:send():382] send: history
768
+ 2024-02-01 17:44:59,113 DEBUG SenderThread:237521 [sender.py:send_request():409] send_request: summary_record
769
+ 2024-02-01 17:44:59,115 INFO SenderThread:237521 [sender.py:_save_file():1392] saving file wandb-summary.json with policy end
770
+ 2024-02-01 17:44:59,297 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/wandb-summary.json
771
+ 2024-02-01 17:44:59,298 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/output.log
772
+ 2024-02-01 17:45:00,299 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/output.log
773
+ 2024-02-01 17:45:01,851 DEBUG HandlerThread:237521 [handler.py:handle_request():146] handle_request: status_report
774
+ 2024-02-01 17:45:03,302 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/output.log
775
+ 2024-02-01 17:45:03,686 DEBUG HandlerThread:237521 [handler.py:handle_request():146] handle_request: partial_history
776
+ 2024-02-01 17:45:03,687 DEBUG SenderThread:237521 [sender.py:send():382] send: history
777
+ 2024-02-01 17:45:03,688 DEBUG SenderThread:237521 [sender.py:send_request():409] send_request: summary_record
778
+ 2024-02-01 17:45:03,689 INFO SenderThread:237521 [sender.py:_save_file():1392] saving file wandb-summary.json with policy end
779
+ 2024-02-01 17:45:04,306 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/wandb-summary.json
780
+ 2024-02-01 17:45:04,306 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/output.log
781
+ 2024-02-01 17:45:04,922 DEBUG HandlerThread:237521 [handler.py:handle_request():146] handle_request: stop_status
782
+ 2024-02-01 17:45:04,922 DEBUG SenderThread:237521 [sender.py:send_request():409] send_request: stop_status
783
+ 2024-02-01 17:45:04,958 DEBUG HandlerThread:237521 [handler.py:handle_request():146] handle_request: internal_messages
784
+ 2024-02-01 17:45:07,310 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/output.log
785
+ 2024-02-01 17:45:07,339 DEBUG HandlerThread:237521 [handler.py:handle_request():146] handle_request: status_report
786
+ 2024-02-01 17:45:08,253 DEBUG HandlerThread:237521 [handler.py:handle_request():146] handle_request: partial_history
787
+ 2024-02-01 17:45:08,253 DEBUG SenderThread:237521 [sender.py:send():382] send: history
788
+ 2024-02-01 17:45:08,254 DEBUG SenderThread:237521 [sender.py:send_request():409] send_request: summary_record
789
+ 2024-02-01 17:45:08,256 INFO SenderThread:237521 [sender.py:_save_file():1392] saving file wandb-summary.json with policy end
790
+ 2024-02-01 17:45:08,312 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/wandb-summary.json
791
+ 2024-02-01 17:45:09,313 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/output.log
792
+ 2024-02-01 17:45:10,315 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/output.log
793
+ 2024-02-01 17:45:12,828 DEBUG HandlerThread:237521 [handler.py:handle_request():146] handle_request: status_report
794
+ 2024-02-01 17:45:12,829 DEBUG HandlerThread:237521 [handler.py:handle_request():146] handle_request: partial_history
795
+ 2024-02-01 17:45:12,830 DEBUG SenderThread:237521 [sender.py:send():382] send: history
796
+ 2024-02-01 17:45:12,830 DEBUG SenderThread:237521 [sender.py:send_request():409] send_request: summary_record
797
+ 2024-02-01 17:45:12,832 INFO SenderThread:237521 [sender.py:_save_file():1392] saving file wandb-summary.json with policy end
798
+ 2024-02-01 17:45:13,320 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/wandb-summary.json
799
+ 2024-02-01 17:45:13,320 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/output.log
800
+ 2024-02-01 17:45:14,321 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/output.log
801
+ 2024-02-01 17:45:17,325 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/output.log
802
+ 2024-02-01 17:45:17,396 DEBUG HandlerThread:237521 [handler.py:handle_request():146] handle_request: partial_history
803
+ 2024-02-01 17:45:17,397 DEBUG SenderThread:237521 [sender.py:send():382] send: history
804
+ 2024-02-01 17:45:17,398 DEBUG SenderThread:237521 [sender.py:send_request():409] send_request: summary_record
805
+ 2024-02-01 17:45:17,400 INFO SenderThread:237521 [sender.py:_save_file():1392] saving file wandb-summary.json with policy end
806
+ 2024-02-01 17:45:18,310 DEBUG HandlerThread:237521 [handler.py:handle_request():146] handle_request: status_report
807
+ 2024-02-01 17:45:18,327 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/wandb-summary.json
808
+ 2024-02-01 17:45:18,327 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/output.log
809
+ 2024-02-01 17:45:19,922 DEBUG HandlerThread:237521 [handler.py:handle_request():146] handle_request: stop_status
810
+ 2024-02-01 17:45:19,922 DEBUG SenderThread:237521 [sender.py:send_request():409] send_request: stop_status
811
+ 2024-02-01 17:45:19,958 DEBUG HandlerThread:237521 [handler.py:handle_request():146] handle_request: internal_messages
812
+ 2024-02-01 17:45:21,331 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/output.log
813
+ 2024-02-01 17:45:21,971 DEBUG HandlerThread:237521 [handler.py:handle_request():146] handle_request: partial_history
814
+ 2024-02-01 17:45:21,972 DEBUG SenderThread:237521 [sender.py:send():382] send: history
815
+ 2024-02-01 17:45:21,972 DEBUG SenderThread:237521 [sender.py:send_request():409] send_request: summary_record
816
+ 2024-02-01 17:45:21,974 INFO SenderThread:237521 [sender.py:_save_file():1392] saving file wandb-summary.json with policy end
817
+ 2024-02-01 17:45:22,333 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/wandb-summary.json
818
+ 2024-02-01 17:45:23,334 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/output.log
819
+ 2024-02-01 17:45:23,801 DEBUG HandlerThread:237521 [handler.py:handle_request():146] handle_request: status_report
820
+ 2024-02-01 17:45:24,336 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/output.log
821
+ 2024-02-01 17:45:26,837 DEBUG HandlerThread:237521 [handler.py:handle_request():146] handle_request: partial_history
822
+ 2024-02-01 17:45:26,838 DEBUG SenderThread:237521 [sender.py:send():382] send: history
823
+ 2024-02-01 17:45:26,839 DEBUG SenderThread:237521 [sender.py:send_request():409] send_request: summary_record
824
+ 2024-02-01 17:45:26,840 INFO SenderThread:237521 [sender.py:_save_file():1392] saving file wandb-summary.json with policy end
825
+ 2024-02-01 17:45:27,341 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/wandb-summary.json
826
+ 2024-02-01 17:45:27,341 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/output.log
827
+ 2024-02-01 17:45:28,342 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/output.log
828
+ 2024-02-01 17:45:28,901 DEBUG HandlerThread:237521 [handler.py:handle_request():146] handle_request: status_report
829
+ 2024-02-01 17:45:28,954 DEBUG SenderThread:237521 [sender.py:send():382] send: stats
830
+ 2024-02-01 17:45:32,347 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/output.log
831
+ 2024-02-01 17:45:34,027 DEBUG HandlerThread:237521 [handler.py:handle_request():146] handle_request: status_report
832
+ 2024-02-01 17:45:35,049 DEBUG HandlerThread:237521 [handler.py:handle_request():146] handle_request: stop_status
833
+ 2024-02-01 17:45:35,049 DEBUG HandlerThread:237521 [handler.py:handle_request():146] handle_request: internal_messages
834
+ 2024-02-01 17:45:35,050 DEBUG SenderThread:237521 [sender.py:send_request():409] send_request: stop_status
835
+ 2024-02-01 17:45:36,353 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/output.log
836
+ 2024-02-01 17:45:39,357 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/output.log
837
+ 2024-02-01 17:45:39,935 DEBUG HandlerThread:237521 [handler.py:handle_request():146] handle_request: status_report
838
+ 2024-02-01 17:45:45,310 DEBUG HandlerThread:237521 [handler.py:handle_request():146] handle_request: status_report
839
+ 2024-02-01 17:45:45,365 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/output.log
840
+ 2024-02-01 17:45:46,222 DEBUG HandlerThread:237521 [handler.py:handle_request():146] handle_request: partial_history
841
+ 2024-02-01 17:45:46,224 DEBUG SenderThread:237521 [sender.py:send():382] send: history
842
+ 2024-02-01 17:45:46,224 DEBUG SenderThread:237521 [sender.py:send_request():409] send_request: summary_record
843
+ 2024-02-01 17:45:46,226 INFO SenderThread:237521 [sender.py:_save_file():1392] saving file wandb-summary.json with policy end
844
+ 2024-02-01 17:45:46,368 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/wandb-summary.json
845
+ 2024-02-01 17:45:47,369 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/output.log
846
+ 2024-02-01 17:45:48,371 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/output.log
847
+ 2024-02-01 17:45:50,021 DEBUG HandlerThread:237521 [handler.py:handle_request():146] handle_request: stop_status
848
+ 2024-02-01 17:45:50,021 DEBUG SenderThread:237521 [sender.py:send_request():409] send_request: stop_status
849
+ 2024-02-01 17:45:50,049 DEBUG HandlerThread:237521 [handler.py:handle_request():146] handle_request: internal_messages
850
+ 2024-02-01 17:45:50,826 DEBUG HandlerThread:237521 [handler.py:handle_request():146] handle_request: partial_history
851
+ 2024-02-01 17:45:51,374 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/output.log
852
+ 2024-02-01 17:45:51,683 DEBUG SenderThread:237521 [sender.py:send():382] send: history
853
+ 2024-02-01 17:45:51,684 DEBUG HandlerThread:237521 [handler.py:handle_request():146] handle_request: status_report
854
+ 2024-02-01 17:45:51,684 DEBUG SenderThread:237521 [sender.py:send_request():409] send_request: summary_record
855
+ 2024-02-01 17:45:51,686 INFO SenderThread:237521 [sender.py:_save_file():1392] saving file wandb-summary.json with policy end
856
+ 2024-02-01 17:45:52,377 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/wandb-summary.json
857
+ 2024-02-01 17:45:52,377 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/output.log
858
+ 2024-02-01 17:45:55,347 DEBUG HandlerThread:237521 [handler.py:handle_request():146] handle_request: partial_history
859
+ 2024-02-01 17:45:55,348 DEBUG SenderThread:237521 [sender.py:send():382] send: history
860
+ 2024-02-01 17:45:55,349 DEBUG SenderThread:237521 [sender.py:send_request():409] send_request: summary_record
861
+ 2024-02-01 17:45:55,351 INFO SenderThread:237521 [sender.py:_save_file():1392] saving file wandb-summary.json with policy end
862
+ 2024-02-01 17:45:55,381 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/wandb-summary.json
863
+ 2024-02-01 17:45:55,382 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/output.log
864
+ 2024-02-01 17:45:56,383 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/output.log
865
+ 2024-02-01 17:45:57,169 DEBUG HandlerThread:237521 [handler.py:handle_request():146] handle_request: status_report
866
+ 2024-02-01 17:45:58,865 DEBUG SenderThread:237521 [sender.py:send():382] send: stats
867
+ 2024-02-01 17:45:59,386 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/output.log
868
+ 2024-02-01 17:45:59,911 DEBUG HandlerThread:237521 [handler.py:handle_request():146] handle_request: partial_history
869
+ 2024-02-01 17:45:59,912 DEBUG SenderThread:237521 [sender.py:send():382] send: history
870
+ 2024-02-01 17:45:59,913 DEBUG SenderThread:237521 [sender.py:send_request():409] send_request: summary_record
871
+ 2024-02-01 17:45:59,914 INFO SenderThread:237521 [sender.py:_save_file():1392] saving file wandb-summary.json with policy end
872
+ 2024-02-01 17:46:00,388 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/wandb-summary.json
873
+ 2024-02-01 17:46:01,390 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/output.log
874
+ 2024-02-01 17:46:02,391 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/output.log
875
+ 2024-02-01 17:46:02,657 DEBUG HandlerThread:237521 [handler.py:handle_request():146] handle_request: status_report
876
+ 2024-02-01 17:46:04,486 DEBUG HandlerThread:237521 [handler.py:handle_request():146] handle_request: partial_history
877
+ 2024-02-01 17:46:04,487 DEBUG SenderThread:237521 [sender.py:send():382] send: history
878
+ 2024-02-01 17:46:04,488 DEBUG SenderThread:237521 [sender.py:send_request():409] send_request: summary_record
879
+ 2024-02-01 17:46:04,489 INFO SenderThread:237521 [sender.py:_save_file():1392] saving file wandb-summary.json with policy end
880
+ 2024-02-01 17:46:05,021 DEBUG HandlerThread:237521 [handler.py:handle_request():146] handle_request: stop_status
881
+ 2024-02-01 17:46:05,021 DEBUG SenderThread:237521 [sender.py:send_request():409] send_request: stop_status
882
+ 2024-02-01 17:46:05,049 DEBUG HandlerThread:237521 [handler.py:handle_request():146] handle_request: internal_messages
883
+ 2024-02-01 17:46:05,396 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/wandb-summary.json
884
+ 2024-02-01 17:46:05,396 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/output.log
885
+ 2024-02-01 17:46:06,397 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/output.log
886
+ 2024-02-01 17:46:08,147 DEBUG HandlerThread:237521 [handler.py:handle_request():146] handle_request: status_report
887
+ 2024-02-01 17:46:09,058 DEBUG HandlerThread:237521 [handler.py:handle_request():146] handle_request: partial_history
888
+ 2024-02-01 17:46:09,059 DEBUG SenderThread:237521 [sender.py:send():382] send: history
889
+ 2024-02-01 17:46:09,059 DEBUG SenderThread:237521 [sender.py:send_request():409] send_request: summary_record
890
+ 2024-02-01 17:46:09,061 INFO SenderThread:237521 [sender.py:_save_file():1392] saving file wandb-summary.json with policy end
891
+ 2024-02-01 17:46:09,402 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/wandb-summary.json
892
+ 2024-02-01 17:46:09,402 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/output.log
893
+ 2024-02-01 17:46:10,403 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/output.log
894
+ 2024-02-01 17:46:13,410 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/output.log
895
+ 2024-02-01 17:46:13,629 DEBUG HandlerThread:237521 [handler.py:handle_request():146] handle_request: status_report
896
+ 2024-02-01 17:46:13,631 DEBUG HandlerThread:237521 [handler.py:handle_request():146] handle_request: partial_history
897
+ 2024-02-01 17:46:13,631 DEBUG SenderThread:237521 [sender.py:send():382] send: history
898
+ 2024-02-01 17:46:13,632 DEBUG SenderThread:237521 [sender.py:send_request():409] send_request: summary_record
899
+ 2024-02-01 17:46:13,636 INFO SenderThread:237521 [sender.py:_save_file():1392] saving file wandb-summary.json with policy end
900
+ 2024-02-01 17:46:14,412 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/wandb-summary.json
901
+ 2024-02-01 17:46:14,413 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/output.log
902
+ 2024-02-01 17:46:17,416 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/output.log
903
+ 2024-02-01 17:46:18,205 DEBUG HandlerThread:237521 [handler.py:handle_request():146] handle_request: partial_history
904
+ 2024-02-01 17:46:18,206 DEBUG SenderThread:237521 [sender.py:send():382] send: history
905
+ 2024-02-01 17:46:18,206 DEBUG SenderThread:237521 [sender.py:send_request():409] send_request: summary_record
906
+ 2024-02-01 17:46:18,208 INFO SenderThread:237521 [sender.py:_save_file():1392] saving file wandb-summary.json with policy end
907
+ 2024-02-01 17:46:18,419 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/wandb-summary.json
908
+ 2024-02-01 17:46:19,118 DEBUG HandlerThread:237521 [handler.py:handle_request():146] handle_request: status_report
909
+ 2024-02-01 17:46:19,420 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/output.log
910
+ 2024-02-01 17:46:20,021 DEBUG HandlerThread:237521 [handler.py:handle_request():146] handle_request: stop_status
911
+ 2024-02-01 17:46:20,021 DEBUG SenderThread:237521 [sender.py:send_request():409] send_request: stop_status
912
+ 2024-02-01 17:46:20,058 DEBUG HandlerThread:237521 [handler.py:handle_request():146] handle_request: internal_messages
913
+ 2024-02-01 17:46:20,423 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/output.log
914
+ 2024-02-01 17:46:22,775 DEBUG HandlerThread:237521 [handler.py:handle_request():146] handle_request: partial_history
915
+ 2024-02-01 17:46:22,777 DEBUG SenderThread:237521 [sender.py:send():382] send: history
916
+ 2024-02-01 17:46:22,777 DEBUG SenderThread:237521 [sender.py:send_request():409] send_request: summary_record
917
+ 2024-02-01 17:46:22,779 INFO SenderThread:237521 [sender.py:_save_file():1392] saving file wandb-summary.json with policy end
918
+ 2024-02-01 17:46:23,428 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/wandb-summary.json
919
+ 2024-02-01 17:46:23,428 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/output.log
920
+ 2024-02-01 17:46:24,429 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/output.log
921
+ 2024-02-01 17:46:24,605 DEBUG HandlerThread:237521 [handler.py:handle_request():146] handle_request: status_report
922
+ 2024-02-01 17:46:27,348 DEBUG HandlerThread:237521 [handler.py:handle_request():146] handle_request: partial_history
923
+ 2024-02-01 17:46:27,349 DEBUG SenderThread:237521 [sender.py:send():382] send: history
924
+ 2024-02-01 17:46:27,349 DEBUG SenderThread:237521 [sender.py:send_request():409] send_request: summary_record
925
+ 2024-02-01 17:46:27,351 INFO SenderThread:237521 [sender.py:_save_file():1392] saving file wandb-summary.json with policy end
926
+ 2024-02-01 17:46:27,434 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/wandb-summary.json
927
+ 2024-02-01 17:46:27,434 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/output.log
928
+ 2024-02-01 17:46:28,436 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/output.log
929
+ 2024-02-01 17:46:28,867 DEBUG SenderThread:237521 [sender.py:send():382] send: stats
930
+ 2024-02-01 17:46:30,089 DEBUG HandlerThread:237521 [handler.py:handle_request():146] handle_request: status_report
931
+ 2024-02-01 17:46:31,439 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/output.log
932
+ 2024-02-01 17:46:31,917 DEBUG HandlerThread:237521 [handler.py:handle_request():146] handle_request: partial_history
933
+ 2024-02-01 17:46:31,919 DEBUG SenderThread:237521 [sender.py:send():382] send: history
934
+ 2024-02-01 17:46:31,919 DEBUG SenderThread:237521 [sender.py:send_request():409] send_request: summary_record
935
+ 2024-02-01 17:46:31,921 INFO SenderThread:237521 [sender.py:_save_file():1392] saving file wandb-summary.json with policy end
936
+ 2024-02-01 17:46:32,442 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/wandb-summary.json
937
+ 2024-02-01 17:46:33,443 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/output.log
938
+ 2024-02-01 17:46:34,444 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/output.log
939
+ 2024-02-01 17:46:35,021 DEBUG HandlerThread:237521 [handler.py:handle_request():146] handle_request: stop_status
940
+ 2024-02-01 17:46:35,021 DEBUG SenderThread:237521 [sender.py:send_request():409] send_request: stop_status
941
+ 2024-02-01 17:46:35,054 DEBUG HandlerThread:237521 [handler.py:handle_request():146] handle_request: internal_messages
942
+ 2024-02-01 17:46:35,097 DEBUG HandlerThread:237521 [handler.py:handle_request():146] handle_request: status_report
943
+ 2024-02-01 17:46:36,483 DEBUG HandlerThread:237521 [handler.py:handle_request():146] handle_request: partial_history
944
+ 2024-02-01 17:46:36,484 DEBUG SenderThread:237521 [sender.py:send():382] send: history
945
+ 2024-02-01 17:46:36,485 DEBUG SenderThread:237521 [sender.py:send_request():409] send_request: summary_record
946
+ 2024-02-01 17:46:36,486 INFO SenderThread:237521 [sender.py:_save_file():1392] saving file wandb-summary.json with policy end
947
+ 2024-02-01 17:46:37,449 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/wandb-summary.json
948
+ 2024-02-01 17:46:37,449 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/output.log
949
+ 2024-02-01 17:46:38,450 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/output.log
950
+ 2024-02-01 17:46:40,139 DEBUG HandlerThread:237521 [handler.py:handle_request():146] handle_request: status_report
951
+ 2024-02-01 17:46:41,052 DEBUG HandlerThread:237521 [handler.py:handle_request():146] handle_request: partial_history
952
+ 2024-02-01 17:46:41,053 DEBUG SenderThread:237521 [sender.py:send():382] send: history
953
+ 2024-02-01 17:46:41,053 DEBUG SenderThread:237521 [sender.py:send_request():409] send_request: summary_record
954
+ 2024-02-01 17:46:41,055 INFO SenderThread:237521 [sender.py:_save_file():1392] saving file wandb-summary.json with policy end
955
+ 2024-02-01 17:46:41,455 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/wandb-summary.json
956
+ 2024-02-01 17:46:41,456 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/output.log
957
+ 2024-02-01 17:46:42,457 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/output.log
wandb/run-20240201_173828-py26nu6m/files/output.log CHANGED
@@ -294,3 +294,87 @@
294
 
295
  33%|██████████████████████████▋ | 364/1090 [06:19<11:02, 1.10it/s]
296
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
294
 
295
  33%|██████████████████████████▋ | 364/1090 [06:19<11:02, 1.10it/s]
296
 
297
+
298
+ 34%|███████████████████████████ | 369/1090 [06:23<10:58, 1.10it/s]
299
+
300
+
301
+ 34%|███████████████████████████▍ | 373/1090 [06:27<10:54, 1.10it/s]
302
+
303
+
304
+
305
+ 35%|███████████████████████████▉ | 380/1090 [06:33<10:48, 1.09it/s]
306
+
307
+
308
+ 35%|████████████████████████████▏ | 384/1090 [06:37<10:45, 1.09it/s]
309
+
310
+
311
+ 36%|████████████████████████████▍ | 388/1090 [06:41<10:41, 1.09it/s]
312
+
313
+
314
+
315
+ 36%|████████████████████████████▉ | 395/1090 [06:47<10:35, 1.09it/s]
316
+
317
+
318
+ 37%|█████████████████████████████▎ | 399/1090 [06:51<11:32, 1.00s/it]
319
+ 37%|█████████████████████████████▎ | 400/1090 [06:52<11:13, 1.02it/s][INFO|trainer.py:2889] 2024-02-01 17:45:27,895 >> Saving model checkpoint to ./tmp-checkpoint-400
320
+ [INFO|configuration_utils.py:483] 2024-02-01 17:45:27,898 >> Configuration saved in ./tmp-checkpoint-400/config.json
321
+ [INFO|configuration_utils.py:594] 2024-02-01 17:45:27,900 >> Configuration saved in ./tmp-checkpoint-400/generation_config.json
322
+ [2024-02-01 17:45:31,010] [INFO] [logging.py:96:log_dist] [Rank 0] [Torch] Checkpoint global_step400 is about to be saved!
323
+ [2024-02-01 17:45:31,014] [INFO] [logging.py:96:log_dist] [Rank 0] Saving model checkpoint: ./tmp-checkpoint-400/global_step400/zero_pp_rank_0_mp_rank_00_model_states.pt
324
+ [2024-02-01 17:45:31,014] [INFO] [torch_checkpoint_engine.py:21:save] [Torch] Saving ./tmp-checkpoint-400/global_step400/zero_pp_rank_0_mp_rank_00_model_states.pt...
325
+ [2024-02-01 17:45:31,018] [INFO] [torch_checkpoint_engine.py:23:save] [Torch] Saved ./tmp-checkpoint-400/global_step400/zero_pp_rank_0_mp_rank_00_model_states.pt.
326
+ [2024-02-01 17:45:31,026] [INFO] [torch_checkpoint_engine.py:21:save] [Torch] Saving ./tmp-checkpoint-400/global_step400/bf16_zero_pp_rank_0_mp_rank_00_optim_states.pt...
327
+ [INFO|modeling_utils.py:2382] 2024-02-01 17:45:30,979 >> Model weights saved in ./tmp-checkpoint-400/pytorch_model.bin
328
+ [INFO|tokenization_utils_base.py:2432] 2024-02-01 17:45:30,982 >> tokenizer config file saved in ./tmp-checkpoint-400/tokenizer_config.json
329
+ [INFO|tokenization_utils_base.py:2441] 2024-02-01 17:45:30,984 >> Special tokens file saved in ./tmp-checkpoint-400/special_tokens_map.json
330
+ /fsx/sanchit/miniconda3/envs/venv/lib/python3.11/site-packages/torch/nn/modules/module.py:1879: UserWarning: Positional args are being deprecated, use kwargs instead. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details.
331
+ warnings.warn(
332
+ [2024-02-01 17:45:35,048] [INFO] [torch_checkpoint_engine.py:23:save] [Torch] Saved ./tmp-checkpoint-400/global_step400/bf16_zero_pp_rank_0_mp_rank_00_optim_states.pt.
333
+ [2024-02-01 17:45:35,053] [INFO] [engine.py:3393:_save_zero_checkpoint] zero checkpoint saved ./tmp-checkpoint-400/global_step400/bf16_zero_pp_rank_0_mp_rank_00_optim_states.pt
334
+ [2024-02-01 17:45:35,055] [INFO] [torch_checkpoint_engine.py:33:commit] [Torch] Checkpoint global_step400 is ready now!
335
+ [INFO|tokenization_utils_base.py:2432] 2024-02-01 17:45:37,892 >> tokenizer config file saved in ./tokenizer_config.json
336
+ [INFO|tokenization_utils_base.py:2441] 2024-02-01 17:45:37,895 >> Special tokens file saved in ./special_tokens_map.json
337
+ [INFO|trainer.py:2979] 2024-02-01 17:45:37,923 >> Deleting older checkpoint [checkpoint-300] due to args.save_total_limit
338
+
339
+ 37%|█████████████████████████████▋ | 405/1090 [07:11<22:47, 2.00s/it]
340
+
341
+
342
+ 38%|██████████████████████████████ | 409/1090 [07:15<13:18, 1.17s/it]
343
+
344
+
345
+ 38%|██████████████████████████████▎ | 413/1090 [07:19<11:00, 1.02it/s]
346
+
347
+
348
+
349
+ 39%|██████████████████████████████▊ | 420/1090 [07:25<10:15, 1.09it/s]
350
+
351
+
352
+ 39%|███████████████████████████████ | 424/1090 [07:29<10:09, 1.09it/s]
353
+
354
+
355
+ 39%|███████████████████████████████▍ | 429/1090 [07:33<10:05, 1.09it/s]
356
+
357
+
358
+ 40%|███████████████████████████████▊ | 433/1090 [07:37<10:01, 1.09it/s]
359
+
360
+
361
+
362
+ 40%|████████████████████████████████▎ | 440/1090 [07:43<09:54, 1.09it/s]
363
+
364
+
365
+ 41%|████████████████████████████████▌ | 444/1090 [07:47<09:50, 1.09it/s]
366
+
367
+
368
+ 41%|████████████████████████████████▉ | 448/1090 [07:51<09:46, 1.09it/s]
369
+
370
+
371
+
372
+ 42%|█████████████████████████████████▍ | 455/1090 [07:57<09:39, 1.10it/s]
373
+
374
+
375
+ 42%|█████████████████████████████████▋ | 459/1090 [08:01<09:36, 1.09it/s]
376
+
377
+
378
+ 43%|██████████████████████████████████ | 464/1090 [08:05<09:31, 1.09it/s]
379
+
380
+
wandb/run-20240201_173828-py26nu6m/files/wandb-summary.json CHANGED
@@ -1 +1 @@
1
- {"train/loss": 5.332, "train/learning_rate": 1.6823495987048922e-05, "train/epoch": 0.33, "train/global_step": 365, "_timestamp": 1706809494.5481908, "_runtime": 386.0963168144226, "_step": 73}
 
1
+ {"train/loss": 3.2315, "train/learning_rate": 1.4175324596392075e-05, "train/epoch": 0.43, "train/global_step": 465, "_timestamp": 1706809601.0518904, "_runtime": 492.6000163555145, "_step": 93}
wandb/run-20240201_173828-py26nu6m/logs/debug-internal.log CHANGED
@@ -762,3 +762,196 @@
762
  2024-02-01 17:44:55,292 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/output.log
763
  2024-02-01 17:44:56,293 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/output.log
764
  2024-02-01 17:44:56,373 DEBUG HandlerThread:237521 [handler.py:handle_request():146] handle_request: status_report
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
762
  2024-02-01 17:44:55,292 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/output.log
763
  2024-02-01 17:44:56,293 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/output.log
764
  2024-02-01 17:44:56,373 DEBUG HandlerThread:237521 [handler.py:handle_request():146] handle_request: status_report
765
+ 2024-02-01 17:44:58,861 DEBUG SenderThread:237521 [sender.py:send():382] send: stats
766
+ 2024-02-01 17:44:59,112 DEBUG HandlerThread:237521 [handler.py:handle_request():146] handle_request: partial_history
767
+ 2024-02-01 17:44:59,113 DEBUG SenderThread:237521 [sender.py:send():382] send: history
768
+ 2024-02-01 17:44:59,113 DEBUG SenderThread:237521 [sender.py:send_request():409] send_request: summary_record
769
+ 2024-02-01 17:44:59,115 INFO SenderThread:237521 [sender.py:_save_file():1392] saving file wandb-summary.json with policy end
770
+ 2024-02-01 17:44:59,297 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/wandb-summary.json
771
+ 2024-02-01 17:44:59,298 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/output.log
772
+ 2024-02-01 17:45:00,299 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/output.log
773
+ 2024-02-01 17:45:01,851 DEBUG HandlerThread:237521 [handler.py:handle_request():146] handle_request: status_report
774
+ 2024-02-01 17:45:03,302 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/output.log
775
+ 2024-02-01 17:45:03,686 DEBUG HandlerThread:237521 [handler.py:handle_request():146] handle_request: partial_history
776
+ 2024-02-01 17:45:03,687 DEBUG SenderThread:237521 [sender.py:send():382] send: history
777
+ 2024-02-01 17:45:03,688 DEBUG SenderThread:237521 [sender.py:send_request():409] send_request: summary_record
778
+ 2024-02-01 17:45:03,689 INFO SenderThread:237521 [sender.py:_save_file():1392] saving file wandb-summary.json with policy end
779
+ 2024-02-01 17:45:04,306 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/wandb-summary.json
780
+ 2024-02-01 17:45:04,306 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/output.log
781
+ 2024-02-01 17:45:04,922 DEBUG HandlerThread:237521 [handler.py:handle_request():146] handle_request: stop_status
782
+ 2024-02-01 17:45:04,922 DEBUG SenderThread:237521 [sender.py:send_request():409] send_request: stop_status
783
+ 2024-02-01 17:45:04,958 DEBUG HandlerThread:237521 [handler.py:handle_request():146] handle_request: internal_messages
784
+ 2024-02-01 17:45:07,310 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/output.log
785
+ 2024-02-01 17:45:07,339 DEBUG HandlerThread:237521 [handler.py:handle_request():146] handle_request: status_report
786
+ 2024-02-01 17:45:08,253 DEBUG HandlerThread:237521 [handler.py:handle_request():146] handle_request: partial_history
787
+ 2024-02-01 17:45:08,253 DEBUG SenderThread:237521 [sender.py:send():382] send: history
788
+ 2024-02-01 17:45:08,254 DEBUG SenderThread:237521 [sender.py:send_request():409] send_request: summary_record
789
+ 2024-02-01 17:45:08,256 INFO SenderThread:237521 [sender.py:_save_file():1392] saving file wandb-summary.json with policy end
790
+ 2024-02-01 17:45:08,312 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/wandb-summary.json
791
+ 2024-02-01 17:45:09,313 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/output.log
792
+ 2024-02-01 17:45:10,315 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/output.log
793
+ 2024-02-01 17:45:12,828 DEBUG HandlerThread:237521 [handler.py:handle_request():146] handle_request: status_report
794
+ 2024-02-01 17:45:12,829 DEBUG HandlerThread:237521 [handler.py:handle_request():146] handle_request: partial_history
795
+ 2024-02-01 17:45:12,830 DEBUG SenderThread:237521 [sender.py:send():382] send: history
796
+ 2024-02-01 17:45:12,830 DEBUG SenderThread:237521 [sender.py:send_request():409] send_request: summary_record
797
+ 2024-02-01 17:45:12,832 INFO SenderThread:237521 [sender.py:_save_file():1392] saving file wandb-summary.json with policy end
798
+ 2024-02-01 17:45:13,320 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/wandb-summary.json
799
+ 2024-02-01 17:45:13,320 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/output.log
800
+ 2024-02-01 17:45:14,321 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/output.log
801
+ 2024-02-01 17:45:17,325 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/output.log
802
+ 2024-02-01 17:45:17,396 DEBUG HandlerThread:237521 [handler.py:handle_request():146] handle_request: partial_history
803
+ 2024-02-01 17:45:17,397 DEBUG SenderThread:237521 [sender.py:send():382] send: history
804
+ 2024-02-01 17:45:17,398 DEBUG SenderThread:237521 [sender.py:send_request():409] send_request: summary_record
805
+ 2024-02-01 17:45:17,400 INFO SenderThread:237521 [sender.py:_save_file():1392] saving file wandb-summary.json with policy end
806
+ 2024-02-01 17:45:18,310 DEBUG HandlerThread:237521 [handler.py:handle_request():146] handle_request: status_report
807
+ 2024-02-01 17:45:18,327 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/wandb-summary.json
808
+ 2024-02-01 17:45:18,327 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/output.log
809
+ 2024-02-01 17:45:19,922 DEBUG HandlerThread:237521 [handler.py:handle_request():146] handle_request: stop_status
810
+ 2024-02-01 17:45:19,922 DEBUG SenderThread:237521 [sender.py:send_request():409] send_request: stop_status
811
+ 2024-02-01 17:45:19,958 DEBUG HandlerThread:237521 [handler.py:handle_request():146] handle_request: internal_messages
812
+ 2024-02-01 17:45:21,331 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/output.log
813
+ 2024-02-01 17:45:21,971 DEBUG HandlerThread:237521 [handler.py:handle_request():146] handle_request: partial_history
814
+ 2024-02-01 17:45:21,972 DEBUG SenderThread:237521 [sender.py:send():382] send: history
815
+ 2024-02-01 17:45:21,972 DEBUG SenderThread:237521 [sender.py:send_request():409] send_request: summary_record
816
+ 2024-02-01 17:45:21,974 INFO SenderThread:237521 [sender.py:_save_file():1392] saving file wandb-summary.json with policy end
817
+ 2024-02-01 17:45:22,333 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/wandb-summary.json
818
+ 2024-02-01 17:45:23,334 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/output.log
819
+ 2024-02-01 17:45:23,801 DEBUG HandlerThread:237521 [handler.py:handle_request():146] handle_request: status_report
820
+ 2024-02-01 17:45:24,336 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/output.log
821
+ 2024-02-01 17:45:26,837 DEBUG HandlerThread:237521 [handler.py:handle_request():146] handle_request: partial_history
822
+ 2024-02-01 17:45:26,838 DEBUG SenderThread:237521 [sender.py:send():382] send: history
823
+ 2024-02-01 17:45:26,839 DEBUG SenderThread:237521 [sender.py:send_request():409] send_request: summary_record
824
+ 2024-02-01 17:45:26,840 INFO SenderThread:237521 [sender.py:_save_file():1392] saving file wandb-summary.json with policy end
825
+ 2024-02-01 17:45:27,341 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/wandb-summary.json
826
+ 2024-02-01 17:45:27,341 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/output.log
827
+ 2024-02-01 17:45:28,342 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/output.log
828
+ 2024-02-01 17:45:28,901 DEBUG HandlerThread:237521 [handler.py:handle_request():146] handle_request: status_report
829
+ 2024-02-01 17:45:28,954 DEBUG SenderThread:237521 [sender.py:send():382] send: stats
830
+ 2024-02-01 17:45:32,347 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/output.log
831
+ 2024-02-01 17:45:34,027 DEBUG HandlerThread:237521 [handler.py:handle_request():146] handle_request: status_report
832
+ 2024-02-01 17:45:35,049 DEBUG HandlerThread:237521 [handler.py:handle_request():146] handle_request: stop_status
833
+ 2024-02-01 17:45:35,049 DEBUG HandlerThread:237521 [handler.py:handle_request():146] handle_request: internal_messages
834
+ 2024-02-01 17:45:35,050 DEBUG SenderThread:237521 [sender.py:send_request():409] send_request: stop_status
835
+ 2024-02-01 17:45:36,353 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/output.log
836
+ 2024-02-01 17:45:39,357 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/output.log
837
+ 2024-02-01 17:45:39,935 DEBUG HandlerThread:237521 [handler.py:handle_request():146] handle_request: status_report
838
+ 2024-02-01 17:45:45,310 DEBUG HandlerThread:237521 [handler.py:handle_request():146] handle_request: status_report
839
+ 2024-02-01 17:45:45,365 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/output.log
840
+ 2024-02-01 17:45:46,222 DEBUG HandlerThread:237521 [handler.py:handle_request():146] handle_request: partial_history
841
+ 2024-02-01 17:45:46,224 DEBUG SenderThread:237521 [sender.py:send():382] send: history
842
+ 2024-02-01 17:45:46,224 DEBUG SenderThread:237521 [sender.py:send_request():409] send_request: summary_record
843
+ 2024-02-01 17:45:46,226 INFO SenderThread:237521 [sender.py:_save_file():1392] saving file wandb-summary.json with policy end
844
+ 2024-02-01 17:45:46,368 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/wandb-summary.json
845
+ 2024-02-01 17:45:47,369 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/output.log
846
+ 2024-02-01 17:45:48,371 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/output.log
847
+ 2024-02-01 17:45:50,021 DEBUG HandlerThread:237521 [handler.py:handle_request():146] handle_request: stop_status
848
+ 2024-02-01 17:45:50,021 DEBUG SenderThread:237521 [sender.py:send_request():409] send_request: stop_status
849
+ 2024-02-01 17:45:50,049 DEBUG HandlerThread:237521 [handler.py:handle_request():146] handle_request: internal_messages
850
+ 2024-02-01 17:45:50,826 DEBUG HandlerThread:237521 [handler.py:handle_request():146] handle_request: partial_history
851
+ 2024-02-01 17:45:51,374 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/output.log
852
+ 2024-02-01 17:45:51,683 DEBUG SenderThread:237521 [sender.py:send():382] send: history
853
+ 2024-02-01 17:45:51,684 DEBUG HandlerThread:237521 [handler.py:handle_request():146] handle_request: status_report
854
+ 2024-02-01 17:45:51,684 DEBUG SenderThread:237521 [sender.py:send_request():409] send_request: summary_record
855
+ 2024-02-01 17:45:51,686 INFO SenderThread:237521 [sender.py:_save_file():1392] saving file wandb-summary.json with policy end
856
+ 2024-02-01 17:45:52,377 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/wandb-summary.json
857
+ 2024-02-01 17:45:52,377 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/output.log
858
+ 2024-02-01 17:45:55,347 DEBUG HandlerThread:237521 [handler.py:handle_request():146] handle_request: partial_history
859
+ 2024-02-01 17:45:55,348 DEBUG SenderThread:237521 [sender.py:send():382] send: history
860
+ 2024-02-01 17:45:55,349 DEBUG SenderThread:237521 [sender.py:send_request():409] send_request: summary_record
861
+ 2024-02-01 17:45:55,351 INFO SenderThread:237521 [sender.py:_save_file():1392] saving file wandb-summary.json with policy end
862
+ 2024-02-01 17:45:55,381 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/wandb-summary.json
863
+ 2024-02-01 17:45:55,382 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/output.log
864
+ 2024-02-01 17:45:56,383 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/output.log
865
+ 2024-02-01 17:45:57,169 DEBUG HandlerThread:237521 [handler.py:handle_request():146] handle_request: status_report
866
+ 2024-02-01 17:45:58,865 DEBUG SenderThread:237521 [sender.py:send():382] send: stats
867
+ 2024-02-01 17:45:59,386 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/output.log
868
+ 2024-02-01 17:45:59,911 DEBUG HandlerThread:237521 [handler.py:handle_request():146] handle_request: partial_history
869
+ 2024-02-01 17:45:59,912 DEBUG SenderThread:237521 [sender.py:send():382] send: history
870
+ 2024-02-01 17:45:59,913 DEBUG SenderThread:237521 [sender.py:send_request():409] send_request: summary_record
871
+ 2024-02-01 17:45:59,914 INFO SenderThread:237521 [sender.py:_save_file():1392] saving file wandb-summary.json with policy end
872
+ 2024-02-01 17:46:00,388 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/wandb-summary.json
873
+ 2024-02-01 17:46:01,390 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/output.log
874
+ 2024-02-01 17:46:02,391 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/output.log
875
+ 2024-02-01 17:46:02,657 DEBUG HandlerThread:237521 [handler.py:handle_request():146] handle_request: status_report
876
+ 2024-02-01 17:46:04,486 DEBUG HandlerThread:237521 [handler.py:handle_request():146] handle_request: partial_history
877
+ 2024-02-01 17:46:04,487 DEBUG SenderThread:237521 [sender.py:send():382] send: history
878
+ 2024-02-01 17:46:04,488 DEBUG SenderThread:237521 [sender.py:send_request():409] send_request: summary_record
879
+ 2024-02-01 17:46:04,489 INFO SenderThread:237521 [sender.py:_save_file():1392] saving file wandb-summary.json with policy end
880
+ 2024-02-01 17:46:05,021 DEBUG HandlerThread:237521 [handler.py:handle_request():146] handle_request: stop_status
881
+ 2024-02-01 17:46:05,021 DEBUG SenderThread:237521 [sender.py:send_request():409] send_request: stop_status
882
+ 2024-02-01 17:46:05,049 DEBUG HandlerThread:237521 [handler.py:handle_request():146] handle_request: internal_messages
883
+ 2024-02-01 17:46:05,396 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/wandb-summary.json
884
+ 2024-02-01 17:46:05,396 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/output.log
885
+ 2024-02-01 17:46:06,397 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/output.log
886
+ 2024-02-01 17:46:08,147 DEBUG HandlerThread:237521 [handler.py:handle_request():146] handle_request: status_report
887
+ 2024-02-01 17:46:09,058 DEBUG HandlerThread:237521 [handler.py:handle_request():146] handle_request: partial_history
888
+ 2024-02-01 17:46:09,059 DEBUG SenderThread:237521 [sender.py:send():382] send: history
889
+ 2024-02-01 17:46:09,059 DEBUG SenderThread:237521 [sender.py:send_request():409] send_request: summary_record
890
+ 2024-02-01 17:46:09,061 INFO SenderThread:237521 [sender.py:_save_file():1392] saving file wandb-summary.json with policy end
891
+ 2024-02-01 17:46:09,402 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/wandb-summary.json
892
+ 2024-02-01 17:46:09,402 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/output.log
893
+ 2024-02-01 17:46:10,403 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/output.log
894
+ 2024-02-01 17:46:13,410 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/output.log
895
+ 2024-02-01 17:46:13,629 DEBUG HandlerThread:237521 [handler.py:handle_request():146] handle_request: status_report
896
+ 2024-02-01 17:46:13,631 DEBUG HandlerThread:237521 [handler.py:handle_request():146] handle_request: partial_history
897
+ 2024-02-01 17:46:13,631 DEBUG SenderThread:237521 [sender.py:send():382] send: history
898
+ 2024-02-01 17:46:13,632 DEBUG SenderThread:237521 [sender.py:send_request():409] send_request: summary_record
899
+ 2024-02-01 17:46:13,636 INFO SenderThread:237521 [sender.py:_save_file():1392] saving file wandb-summary.json with policy end
900
+ 2024-02-01 17:46:14,412 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/wandb-summary.json
901
+ 2024-02-01 17:46:14,413 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/output.log
902
+ 2024-02-01 17:46:17,416 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/output.log
903
+ 2024-02-01 17:46:18,205 DEBUG HandlerThread:237521 [handler.py:handle_request():146] handle_request: partial_history
904
+ 2024-02-01 17:46:18,206 DEBUG SenderThread:237521 [sender.py:send():382] send: history
905
+ 2024-02-01 17:46:18,206 DEBUG SenderThread:237521 [sender.py:send_request():409] send_request: summary_record
906
+ 2024-02-01 17:46:18,208 INFO SenderThread:237521 [sender.py:_save_file():1392] saving file wandb-summary.json with policy end
907
+ 2024-02-01 17:46:18,419 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/wandb-summary.json
908
+ 2024-02-01 17:46:19,118 DEBUG HandlerThread:237521 [handler.py:handle_request():146] handle_request: status_report
909
+ 2024-02-01 17:46:19,420 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/output.log
910
+ 2024-02-01 17:46:20,021 DEBUG HandlerThread:237521 [handler.py:handle_request():146] handle_request: stop_status
911
+ 2024-02-01 17:46:20,021 DEBUG SenderThread:237521 [sender.py:send_request():409] send_request: stop_status
912
+ 2024-02-01 17:46:20,058 DEBUG HandlerThread:237521 [handler.py:handle_request():146] handle_request: internal_messages
913
+ 2024-02-01 17:46:20,423 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/output.log
914
+ 2024-02-01 17:46:22,775 DEBUG HandlerThread:237521 [handler.py:handle_request():146] handle_request: partial_history
915
+ 2024-02-01 17:46:22,777 DEBUG SenderThread:237521 [sender.py:send():382] send: history
916
+ 2024-02-01 17:46:22,777 DEBUG SenderThread:237521 [sender.py:send_request():409] send_request: summary_record
917
+ 2024-02-01 17:46:22,779 INFO SenderThread:237521 [sender.py:_save_file():1392] saving file wandb-summary.json with policy end
918
+ 2024-02-01 17:46:23,428 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/wandb-summary.json
919
+ 2024-02-01 17:46:23,428 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/output.log
920
+ 2024-02-01 17:46:24,429 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/output.log
921
+ 2024-02-01 17:46:24,605 DEBUG HandlerThread:237521 [handler.py:handle_request():146] handle_request: status_report
922
+ 2024-02-01 17:46:27,348 DEBUG HandlerThread:237521 [handler.py:handle_request():146] handle_request: partial_history
923
+ 2024-02-01 17:46:27,349 DEBUG SenderThread:237521 [sender.py:send():382] send: history
924
+ 2024-02-01 17:46:27,349 DEBUG SenderThread:237521 [sender.py:send_request():409] send_request: summary_record
925
+ 2024-02-01 17:46:27,351 INFO SenderThread:237521 [sender.py:_save_file():1392] saving file wandb-summary.json with policy end
926
+ 2024-02-01 17:46:27,434 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/wandb-summary.json
927
+ 2024-02-01 17:46:27,434 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/output.log
928
+ 2024-02-01 17:46:28,436 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/output.log
929
+ 2024-02-01 17:46:28,867 DEBUG SenderThread:237521 [sender.py:send():382] send: stats
930
+ 2024-02-01 17:46:30,089 DEBUG HandlerThread:237521 [handler.py:handle_request():146] handle_request: status_report
931
+ 2024-02-01 17:46:31,439 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/output.log
932
+ 2024-02-01 17:46:31,917 DEBUG HandlerThread:237521 [handler.py:handle_request():146] handle_request: partial_history
933
+ 2024-02-01 17:46:31,919 DEBUG SenderThread:237521 [sender.py:send():382] send: history
934
+ 2024-02-01 17:46:31,919 DEBUG SenderThread:237521 [sender.py:send_request():409] send_request: summary_record
935
+ 2024-02-01 17:46:31,921 INFO SenderThread:237521 [sender.py:_save_file():1392] saving file wandb-summary.json with policy end
936
+ 2024-02-01 17:46:32,442 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/wandb-summary.json
937
+ 2024-02-01 17:46:33,443 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/output.log
938
+ 2024-02-01 17:46:34,444 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/output.log
939
+ 2024-02-01 17:46:35,021 DEBUG HandlerThread:237521 [handler.py:handle_request():146] handle_request: stop_status
940
+ 2024-02-01 17:46:35,021 DEBUG SenderThread:237521 [sender.py:send_request():409] send_request: stop_status
941
+ 2024-02-01 17:46:35,054 DEBUG HandlerThread:237521 [handler.py:handle_request():146] handle_request: internal_messages
942
+ 2024-02-01 17:46:35,097 DEBUG HandlerThread:237521 [handler.py:handle_request():146] handle_request: status_report
943
+ 2024-02-01 17:46:36,483 DEBUG HandlerThread:237521 [handler.py:handle_request():146] handle_request: partial_history
944
+ 2024-02-01 17:46:36,484 DEBUG SenderThread:237521 [sender.py:send():382] send: history
945
+ 2024-02-01 17:46:36,485 DEBUG SenderThread:237521 [sender.py:send_request():409] send_request: summary_record
946
+ 2024-02-01 17:46:36,486 INFO SenderThread:237521 [sender.py:_save_file():1392] saving file wandb-summary.json with policy end
947
+ 2024-02-01 17:46:37,449 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/wandb-summary.json
948
+ 2024-02-01 17:46:37,449 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/output.log
949
+ 2024-02-01 17:46:38,450 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/output.log
950
+ 2024-02-01 17:46:40,139 DEBUG HandlerThread:237521 [handler.py:handle_request():146] handle_request: status_report
951
+ 2024-02-01 17:46:41,052 DEBUG HandlerThread:237521 [handler.py:handle_request():146] handle_request: partial_history
952
+ 2024-02-01 17:46:41,053 DEBUG SenderThread:237521 [sender.py:send():382] send: history
953
+ 2024-02-01 17:46:41,053 DEBUG SenderThread:237521 [sender.py:send_request():409] send_request: summary_record
954
+ 2024-02-01 17:46:41,055 INFO SenderThread:237521 [sender.py:_save_file():1392] saving file wandb-summary.json with policy end
955
+ 2024-02-01 17:46:41,455 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/wandb-summary.json
956
+ 2024-02-01 17:46:41,456 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/output.log
957
+ 2024-02-01 17:46:42,457 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/output.log
wandb/run-20240201_173828-py26nu6m/run-py26nu6m.wandb CHANGED
Binary files a/wandb/run-20240201_173828-py26nu6m/run-py26nu6m.wandb and b/wandb/run-20240201_173828-py26nu6m/run-py26nu6m.wandb differ