Hugging Face's logo

sanchit-gandhi
/
distil-zephyr-1.5b-ssft

Text Generation
Transformers
TensorBoard
Safetensors
mistral
alignment-handbook
Generated from Trainer
trl
sft
conversational
text-generation-inference
Inference Endpoints
Model card Files Metrics Community
sanchit-gandhi HF staff commited on
Commit
895829e
•
1 Parent(s): 9f7777e

Training in progress, step 500

Browse files
Files changed (7) hide show
  1. model.safetensors +1 -1
  2. runs/Feb01_17-38-02_ip-26-0-165-24/events.out.tfevents.1706809106.ip-26-0-165-24.237059.0 +2 -2
  3. wandb/debug-internal.log +195 -0
  4. wandb/run-20240201_173828-py26nu6m/files/output.log +82 -0
  5. wandb/run-20240201_173828-py26nu6m/files/wandb-summary.json +1 -1
  6. wandb/run-20240201_173828-py26nu6m/logs/debug-internal.log +195 -0
  7. wandb/run-20240201_173828-py26nu6m/run-py26nu6m.wandb +0 -0
model.safetensors CHANGED
@@ -1,3 +1,3 @@
1
  version https://git-lfs.github.com/spec/v1
2
- oid sha256:e806ed22e6133d9b25f5c6096b7545b0a898553f8573e473a0cc2daf4b6eaa63
3
  size 3141646744
 
1
  version https://git-lfs.github.com/spec/v1
2
+ oid sha256:cf25a398b08458de3374071b7cea8c385537c8f99b8770a28ab8afa8d7d36c2c
3
  size 3141646744
runs/Feb01_17-38-02_ip-26-0-165-24/events.out.tfevents.1706809106.ip-26-0-165-24.237059.0 CHANGED
@@ -1,3 +1,3 @@
1
  version https://git-lfs.github.com/spec/v1
2
- oid sha256:e403dbd3805e13d02e41b8f40ddcc16a6de2d91b85ecc6c3c2c8d8ab489f60bb
3
- size 17014
 
1
  version https://git-lfs.github.com/spec/v1
2
+ oid sha256:a482b66b5a1fb18173d309270fccdcf33806f209221c9f936d35e777f13ad87b
3
+ size 20154
wandb/debug-internal.log CHANGED
@@ -955,3 +955,198 @@
955
  2024-02-01 17:46:41,455 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/wandb-summary.json
956
  2024-02-01 17:46:41,456 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/output.log
957
  2024-02-01 17:46:42,457 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/output.log
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
955
  2024-02-01 17:46:41,455 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/wandb-summary.json
956
  2024-02-01 17:46:41,456 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/output.log
957
  2024-02-01 17:46:42,457 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/output.log
958
+ 2024-02-01 17:46:45,461 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/output.log
959
+ 2024-02-01 17:46:45,624 DEBUG HandlerThread:237521 [handler.py:handle_request():146] handle_request: status_report
960
+ 2024-02-01 17:46:45,625 DEBUG HandlerThread:237521 [handler.py:handle_request():146] handle_request: partial_history
961
+ 2024-02-01 17:46:45,626 DEBUG SenderThread:237521 [sender.py:send():382] send: history
962
+ 2024-02-01 17:46:45,626 DEBUG SenderThread:237521 [sender.py:send_request():409] send_request: summary_record
963
+ 2024-02-01 17:46:45,628 INFO SenderThread:237521 [sender.py:_save_file():1392] saving file wandb-summary.json with policy end
964
+ 2024-02-01 17:46:46,463 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/wandb-summary.json
965
+ 2024-02-01 17:46:46,463 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/output.log
966
+ 2024-02-01 17:46:49,467 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/output.log
967
+ 2024-02-01 17:46:50,021 DEBUG HandlerThread:237521 [handler.py:handle_request():146] handle_request: stop_status
968
+ 2024-02-01 17:46:50,021 DEBUG SenderThread:237521 [sender.py:send_request():409] send_request: stop_status
969
+ 2024-02-01 17:46:50,054 DEBUG HandlerThread:237521 [handler.py:handle_request():146] handle_request: internal_messages
970
+ 2024-02-01 17:46:50,238 DEBUG HandlerThread:237521 [handler.py:handle_request():146] handle_request: partial_history
971
+ 2024-02-01 17:46:51,012 DEBUG SenderThread:237521 [sender.py:send():382] send: history
972
+ 2024-02-01 17:46:51,012 DEBUG SenderThread:237521 [sender.py:send_request():409] send_request: summary_record
973
+ 2024-02-01 17:46:51,013 DEBUG HandlerThread:237521 [handler.py:handle_request():146] handle_request: status_report
974
+ 2024-02-01 17:46:51,014 INFO SenderThread:237521 [sender.py:_save_file():1392] saving file wandb-summary.json with policy end
975
+ 2024-02-01 17:46:51,470 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/wandb-summary.json
976
+ 2024-02-01 17:46:51,471 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/output.log
977
+ 2024-02-01 17:46:52,472 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/output.log
978
+ 2024-02-01 17:46:54,767 DEBUG HandlerThread:237521 [handler.py:handle_request():146] handle_request: partial_history
979
+ 2024-02-01 17:46:54,768 DEBUG SenderThread:237521 [sender.py:send():382] send: history
980
+ 2024-02-01 17:46:54,768 DEBUG SenderThread:237521 [sender.py:send_request():409] send_request: summary_record
981
+ 2024-02-01 17:46:54,770 INFO SenderThread:237521 [sender.py:_save_file():1392] saving file wandb-summary.json with policy end
982
+ 2024-02-01 17:46:55,477 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/wandb-summary.json
983
+ 2024-02-01 17:46:55,477 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/output.log
984
+ 2024-02-01 17:46:56,478 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/output.log
985
+ 2024-02-01 17:46:56,591 DEBUG HandlerThread:237521 [handler.py:handle_request():146] handle_request: status_report
986
+ 2024-02-01 17:46:58,869 DEBUG SenderThread:237521 [sender.py:send():382] send: stats
987
+ 2024-02-01 17:46:59,332 DEBUG HandlerThread:237521 [handler.py:handle_request():146] handle_request: partial_history
988
+ 2024-02-01 17:46:59,333 DEBUG SenderThread:237521 [sender.py:send():382] send: history
989
+ 2024-02-01 17:46:59,333 DEBUG SenderThread:237521 [sender.py:send_request():409] send_request: summary_record
990
+ 2024-02-01 17:46:59,335 INFO SenderThread:237521 [sender.py:_save_file():1392] saving file wandb-summary.json with policy end
991
+ 2024-02-01 17:46:59,483 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/wandb-summary.json
992
+ 2024-02-01 17:46:59,483 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/output.log
993
+ 2024-02-01 17:47:00,484 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/output.log
994
+ 2024-02-01 17:47:02,072 DEBUG HandlerThread:237521 [handler.py:handle_request():146] handle_request: status_report
995
+ 2024-02-01 17:47:03,488 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/output.log
996
+ 2024-02-01 17:47:03,905 DEBUG HandlerThread:237521 [handler.py:handle_request():146] handle_request: partial_history
997
+ 2024-02-01 17:47:03,906 DEBUG SenderThread:237521 [sender.py:send():382] send: history
998
+ 2024-02-01 17:47:03,907 DEBUG SenderThread:237521 [sender.py:send_request():409] send_request: summary_record
999
+ 2024-02-01 17:47:03,909 INFO SenderThread:237521 [sender.py:_save_file():1392] saving file wandb-summary.json with policy end
1000
+ 2024-02-01 17:47:04,491 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/wandb-summary.json
1001
+ 2024-02-01 17:47:05,021 DEBUG HandlerThread:237521 [handler.py:handle_request():146] handle_request: stop_status
1002
+ 2024-02-01 17:47:05,021 DEBUG SenderThread:237521 [sender.py:send_request():409] send_request: stop_status
1003
+ 2024-02-01 17:47:05,054 DEBUG HandlerThread:237521 [handler.py:handle_request():146] handle_request: internal_messages
1004
+ 2024-02-01 17:47:05,492 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/output.log
1005
+ 2024-02-01 17:47:06,493 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/output.log
1006
+ 2024-02-01 17:47:07,565 DEBUG HandlerThread:237521 [handler.py:handle_request():146] handle_request: status_report
1007
+ 2024-02-01 17:47:08,481 DEBUG HandlerThread:237521 [handler.py:handle_request():146] handle_request: partial_history
1008
+ 2024-02-01 17:47:08,482 DEBUG SenderThread:237521 [sender.py:send():382] send: history
1009
+ 2024-02-01 17:47:08,483 DEBUG SenderThread:237521 [sender.py:send_request():409] send_request: summary_record
1010
+ 2024-02-01 17:47:08,485 INFO SenderThread:237521 [sender.py:_save_file():1392] saving file wandb-summary.json with policy end
1011
+ 2024-02-01 17:47:08,497 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/wandb-summary.json
1012
+ 2024-02-01 17:47:09,498 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/output.log
1013
+ 2024-02-01 17:47:10,499 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/output.log
1014
+ 2024-02-01 17:47:13,050 DEBUG HandlerThread:237521 [handler.py:handle_request():146] handle_request: status_report
1015
+ 2024-02-01 17:47:13,051 DEBUG HandlerThread:237521 [handler.py:handle_request():146] handle_request: partial_history
1016
+ 2024-02-01 17:47:13,052 DEBUG SenderThread:237521 [sender.py:send():382] send: history
1017
+ 2024-02-01 17:47:13,052 DEBUG SenderThread:237521 [sender.py:send_request():409] send_request: summary_record
1018
+ 2024-02-01 17:47:13,054 INFO SenderThread:237521 [sender.py:_save_file():1392] saving file wandb-summary.json with policy end
1019
+ 2024-02-01 17:47:13,504 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/wandb-summary.json
1020
+ 2024-02-01 17:47:13,504 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/output.log
1021
+ 2024-02-01 17:47:14,505 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/output.log
1022
+ 2024-02-01 17:47:18,373 DEBUG HandlerThread:237521 [handler.py:handle_request():146] handle_request: status_report
1023
+ 2024-02-01 17:47:18,511 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/output.log
1024
+ 2024-02-01 17:47:20,830 DEBUG HandlerThread:237521 [handler.py:handle_request():146] handle_request: internal_messages
1025
+ 2024-02-01 17:47:20,830 DEBUG HandlerThread:237521 [handler.py:handle_request():146] handle_request: stop_status
1026
+ 2024-02-01 17:47:20,831 DEBUG SenderThread:237521 [sender.py:send_request():409] send_request: stop_status
1027
+ 2024-02-01 17:47:22,518 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/output.log
1028
+ 2024-02-01 17:47:23,478 DEBUG HandlerThread:237521 [handler.py:handle_request():146] handle_request: status_report
1029
+ 2024-02-01 17:47:25,521 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/output.log
1030
+ 2024-02-01 17:47:27,524 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/output.log
1031
+ 2024-02-01 17:47:28,710 DEBUG HandlerThread:237521 [handler.py:handle_request():146] handle_request: status_report
1032
+ 2024-02-01 17:47:28,871 DEBUG SenderThread:237521 [sender.py:send():382] send: stats
1033
+ 2024-02-01 17:47:31,531 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/output.log
1034
+ 2024-02-01 17:47:32,347 DEBUG HandlerThread:237521 [handler.py:handle_request():146] handle_request: partial_history
1035
+ 2024-02-01 17:47:32,349 DEBUG SenderThread:237521 [sender.py:send():382] send: history
1036
+ 2024-02-01 17:47:32,349 DEBUG SenderThread:237521 [sender.py:send_request():409] send_request: summary_record
1037
+ 2024-02-01 17:47:32,351 INFO SenderThread:237521 [sender.py:_save_file():1392] saving file wandb-summary.json with policy end
1038
+ 2024-02-01 17:47:32,533 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/wandb-summary.json
1039
+ 2024-02-01 17:47:33,535 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/output.log
1040
+ 2024-02-01 17:47:34,171 DEBUG HandlerThread:237521 [handler.py:handle_request():146] handle_request: status_report
1041
+ 2024-02-01 17:47:34,537 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/output.log
1042
+ 2024-02-01 17:47:35,788 DEBUG HandlerThread:237521 [handler.py:handle_request():146] handle_request: internal_messages
1043
+ 2024-02-01 17:47:35,788 DEBUG HandlerThread:237521 [handler.py:handle_request():146] handle_request: stop_status
1044
+ 2024-02-01 17:47:35,788 DEBUG SenderThread:237521 [sender.py:send_request():409] send_request: stop_status
1045
+ 2024-02-01 17:47:36,906 DEBUG HandlerThread:237521 [handler.py:handle_request():146] handle_request: partial_history
1046
+ 2024-02-01 17:47:36,907 DEBUG SenderThread:237521 [sender.py:send():382] send: history
1047
+ 2024-02-01 17:47:36,908 DEBUG SenderThread:237521 [sender.py:send_request():409] send_request: summary_record
1048
+ 2024-02-01 17:47:36,912 INFO SenderThread:237521 [sender.py:_save_file():1392] saving file wandb-summary.json with policy end
1049
+ 2024-02-01 17:47:37,547 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/wandb-summary.json
1050
+ 2024-02-01 17:47:37,547 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/output.log
1051
+ 2024-02-01 17:47:38,549 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/output.log
1052
+ 2024-02-01 17:47:39,636 DEBUG HandlerThread:237521 [handler.py:handle_request():146] handle_request: status_report
1053
+ 2024-02-01 17:47:41,486 DEBUG HandlerThread:237521 [handler.py:handle_request():146] handle_request: partial_history
1054
+ 2024-02-01 17:47:41,487 DEBUG SenderThread:237521 [sender.py:send():382] send: history
1055
+ 2024-02-01 17:47:41,488 DEBUG SenderThread:237521 [sender.py:send_request():409] send_request: summary_record
1056
+ 2024-02-01 17:47:41,498 INFO SenderThread:237521 [sender.py:_save_file():1392] saving file wandb-summary.json with policy end
1057
+ 2024-02-01 17:47:41,560 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/wandb-summary.json
1058
+ 2024-02-01 17:47:41,560 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/output.log
1059
+ 2024-02-01 17:47:42,562 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/output.log
1060
+ 2024-02-01 17:47:45,135 DEBUG HandlerThread:237521 [handler.py:handle_request():146] handle_request: status_report
1061
+ 2024-02-01 17:47:45,570 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/output.log
1062
+ 2024-02-01 17:47:46,052 DEBUG HandlerThread:237521 [handler.py:handle_request():146] handle_request: partial_history
1063
+ 2024-02-01 17:47:46,053 DEBUG SenderThread:237521 [sender.py:send():382] send: history
1064
+ 2024-02-01 17:47:46,053 DEBUG SenderThread:237521 [sender.py:send_request():409] send_request: summary_record
1065
+ 2024-02-01 17:47:46,063 INFO SenderThread:237521 [sender.py:_save_file():1392] saving file wandb-summary.json with policy end
1066
+ 2024-02-01 17:47:46,575 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/wandb-summary.json
1067
+ 2024-02-01 17:47:47,578 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/output.log
1068
+ 2024-02-01 17:47:48,580 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/output.log
1069
+ 2024-02-01 17:47:50,626 DEBUG HandlerThread:237521 [handler.py:handle_request():146] handle_request: status_report
1070
+ 2024-02-01 17:47:50,632 DEBUG HandlerThread:237521 [handler.py:handle_request():146] handle_request: partial_history
1071
+ 2024-02-01 17:47:50,633 DEBUG SenderThread:237521 [sender.py:send():382] send: history
1072
+ 2024-02-01 17:47:50,634 DEBUG SenderThread:237521 [sender.py:send_request():409] send_request: summary_record
1073
+ 2024-02-01 17:47:50,639 INFO SenderThread:237521 [sender.py:_save_file():1392] saving file wandb-summary.json with policy end
1074
+ 2024-02-01 17:47:50,787 DEBUG HandlerThread:237521 [handler.py:handle_request():146] handle_request: internal_messages
1075
+ 2024-02-01 17:47:50,788 DEBUG HandlerThread:237521 [handler.py:handle_request():146] handle_request: stop_status
1076
+ 2024-02-01 17:47:50,788 DEBUG SenderThread:237521 [sender.py:send_request():409] send_request: stop_status
1077
+ 2024-02-01 17:47:51,596 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/wandb-summary.json
1078
+ 2024-02-01 17:47:51,596 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/output.log
1079
+ 2024-02-01 17:47:52,598 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/output.log
1080
+ 2024-02-01 17:47:55,212 DEBUG HandlerThread:237521 [handler.py:handle_request():146] handle_request: partial_history
1081
+ 2024-02-01 17:47:55,214 DEBUG SenderThread:237521 [sender.py:send():382] send: history
1082
+ 2024-02-01 17:47:55,214 DEBUG SenderThread:237521 [sender.py:send_request():409] send_request: summary_record
1083
+ 2024-02-01 17:47:55,224 INFO SenderThread:237521 [sender.py:_save_file():1392] saving file wandb-summary.json with policy end
1084
+ 2024-02-01 17:47:55,610 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/wandb-summary.json
1085
+ 2024-02-01 17:47:55,610 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/output.log
1086
+ 2024-02-01 17:47:56,124 DEBUG HandlerThread:237521 [handler.py:handle_request():146] handle_request: status_report
1087
+ 2024-02-01 17:47:56,613 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/output.log
1088
+ 2024-02-01 17:47:58,879 DEBUG SenderThread:237521 [sender.py:send():382] send: stats
1089
+ 2024-02-01 17:47:59,620 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/output.log
1090
+ 2024-02-01 17:47:59,797 DEBUG HandlerThread:237521 [handler.py:handle_request():146] handle_request: partial_history
1091
+ 2024-02-01 17:47:59,799 DEBUG SenderThread:237521 [sender.py:send():382] send: history
1092
+ 2024-02-01 17:47:59,799 DEBUG SenderThread:237521 [sender.py:send_request():409] send_request: summary_record
1093
+ 2024-02-01 17:47:59,801 INFO SenderThread:237521 [sender.py:_save_file():1392] saving file wandb-summary.json with policy end
1094
+ 2024-02-01 17:48:00,625 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/wandb-summary.json
1095
+ 2024-02-01 17:48:01,620 DEBUG HandlerThread:237521 [handler.py:handle_request():146] handle_request: status_report
1096
+ 2024-02-01 17:48:01,627 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/output.log
1097
+ 2024-02-01 17:48:02,629 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/output.log
1098
+ 2024-02-01 17:48:04,366 DEBUG HandlerThread:237521 [handler.py:handle_request():146] handle_request: partial_history
1099
+ 2024-02-01 17:48:04,367 DEBUG SenderThread:237521 [sender.py:send():382] send: history
1100
+ 2024-02-01 17:48:04,368 DEBUG SenderThread:237521 [sender.py:send_request():409] send_request: summary_record
1101
+ 2024-02-01 17:48:04,369 INFO SenderThread:237521 [sender.py:_save_file():1392] saving file wandb-summary.json with policy end
1102
+ 2024-02-01 17:48:04,634 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/wandb-summary.json
1103
+ 2024-02-01 17:48:05,635 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/output.log
1104
+ 2024-02-01 17:48:05,788 DEBUG HandlerThread:237521 [handler.py:handle_request():146] handle_request: stop_status
1105
+ 2024-02-01 17:48:05,788 DEBUG SenderThread:237521 [sender.py:send_request():409] send_request: stop_status
1106
+ 2024-02-01 17:48:05,830 DEBUG HandlerThread:237521 [handler.py:handle_request():146] handle_request: internal_messages
1107
+ 2024-02-01 17:48:06,637 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/output.log
1108
+ 2024-02-01 17:48:07,105 DEBUG HandlerThread:237521 [handler.py:handle_request():146] handle_request: status_report
1109
+ 2024-02-01 17:48:08,934 DEBUG HandlerThread:237521 [handler.py:handle_request():146] handle_request: partial_history
1110
+ 2024-02-01 17:48:08,935 DEBUG SenderThread:237521 [sender.py:send():382] send: history
1111
+ 2024-02-01 17:48:08,936 DEBUG SenderThread:237521 [sender.py:send_request():409] send_request: summary_record
1112
+ 2024-02-01 17:48:08,938 INFO SenderThread:237521 [sender.py:_save_file():1392] saving file wandb-summary.json with policy end
1113
+ 2024-02-01 17:48:09,646 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/wandb-summary.json
1114
+ 2024-02-01 17:48:09,647 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/output.log
1115
+ 2024-02-01 17:48:10,648 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/output.log
1116
+ 2024-02-01 17:48:12,590 DEBUG HandlerThread:237521 [handler.py:handle_request():146] handle_request: status_report
1117
+ 2024-02-01 17:48:13,507 DEBUG HandlerThread:237521 [handler.py:handle_request():146] handle_request: partial_history
1118
+ 2024-02-01 17:48:13,508 DEBUG SenderThread:237521 [sender.py:send():382] send: history
1119
+ 2024-02-01 17:48:13,509 DEBUG SenderThread:237521 [sender.py:send_request():409] send_request: summary_record
1120
+ 2024-02-01 17:48:13,511 INFO SenderThread:237521 [sender.py:_save_file():1392] saving file wandb-summary.json with policy end
1121
+ 2024-02-01 17:48:13,653 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/wandb-summary.json
1122
+ 2024-02-01 17:48:13,653 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/output.log
1123
+ 2024-02-01 17:48:14,654 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/output.log
1124
+ 2024-02-01 17:48:17,658 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/output.log
1125
+ 2024-02-01 17:48:18,084 DEBUG HandlerThread:237521 [handler.py:handle_request():146] handle_request: status_report
1126
+ 2024-02-01 17:48:18,085 DEBUG HandlerThread:237521 [handler.py:handle_request():146] handle_request: partial_history
1127
+ 2024-02-01 17:48:18,086 DEBUG SenderThread:237521 [sender.py:send():382] send: history
1128
+ 2024-02-01 17:48:18,087 DEBUG SenderThread:237521 [sender.py:send_request():409] send_request: summary_record
1129
+ 2024-02-01 17:48:18,089 INFO SenderThread:237521 [sender.py:_save_file():1392] saving file wandb-summary.json with policy end
1130
+ 2024-02-01 17:48:18,660 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/wandb-summary.json
1131
+ 2024-02-01 17:48:19,662 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/output.log
1132
+ 2024-02-01 17:48:20,663 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/output.log
1133
+ 2024-02-01 17:48:20,788 DEBUG HandlerThread:237521 [handler.py:handle_request():146] handle_request: stop_status
1134
+ 2024-02-01 17:48:20,788 DEBUG SenderThread:237521 [sender.py:send_request():409] send_request: stop_status
1135
+ 2024-02-01 17:48:20,830 DEBUG HandlerThread:237521 [handler.py:handle_request():146] handle_request: internal_messages
1136
+ 2024-02-01 17:48:22,694 DEBUG HandlerThread:237521 [handler.py:handle_request():146] handle_request: partial_history
1137
+ 2024-02-01 17:48:22,696 DEBUG SenderThread:237521 [sender.py:send():382] send: history
1138
+ 2024-02-01 17:48:22,696 DEBUG SenderThread:237521 [sender.py:send_request():409] send_request: summary_record
1139
+ 2024-02-01 17:48:22,698 INFO SenderThread:237521 [sender.py:_save_file():1392] saving file wandb-summary.json with policy end
1140
+ 2024-02-01 17:48:23,606 DEBUG HandlerThread:237521 [handler.py:handle_request():146] handle_request: status_report
1141
+ 2024-02-01 17:48:23,668 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/wandb-summary.json
1142
+ 2024-02-01 17:48:23,668 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/output.log
1143
+ 2024-02-01 17:48:24,670 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/output.log
1144
+ 2024-02-01 17:48:27,273 DEBUG HandlerThread:237521 [handler.py:handle_request():146] handle_request: partial_history
1145
+ 2024-02-01 17:48:27,274 DEBUG SenderThread:237521 [sender.py:send():382] send: history
1146
+ 2024-02-01 17:48:27,275 DEBUG SenderThread:237521 [sender.py:send_request():409] send_request: summary_record
1147
+ 2024-02-01 17:48:27,281 INFO SenderThread:237521 [sender.py:_save_file():1392] saving file wandb-summary.json with policy end
1148
+ 2024-02-01 17:48:27,678 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/wandb-summary.json
1149
+ 2024-02-01 17:48:27,678 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/output.log
1150
+ 2024-02-01 17:48:28,680 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/output.log
1151
+ 2024-02-01 17:48:28,876 DEBUG SenderThread:237521 [sender.py:send():382] send: stats
1152
+ 2024-02-01 17:48:28,876 DEBUG HandlerThread:237521 [handler.py:handle_request():146] handle_request: status_report
wandb/run-20240201_173828-py26nu6m/files/output.log CHANGED
@@ -378,3 +378,85 @@
378
  43%|██████████████████████████████████ | 464/1090 [08:05<09:31, 1.09it/s]
379
 
380
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
378
  43%|██████████████████████████████████ | 464/1090 [08:05<09:31, 1.09it/s]
379
 
380
 
381
+ 43%|██████████████████████████████████▎ | 468/1090 [08:09<09:29, 1.09it/s]
382
+
383
+
384
+
385
+ 43%|██████████████████████████████████▊ | 474/1090 [08:14<09:22, 1.09it/s]
386
+
387
+
388
+ 44%|███████████████████████████████████▏ | 479/1090 [08:19<09:18, 1.09it/s]
389
+
390
+
391
+ 44%|███████████████████████████████████▍ | 483/1090 [08:23<09:14, 1.09it/s]
392
+
393
+
394
+
395
+ 45%|███████████████████████████████████▉ | 490/1090 [08:29<09:09, 1.09it/s]
396
+
397
+
398
+ 45%|████████████████████████████████████▎ | 494/1090 [08:33<09:05, 1.09it/s]
399
+
400
+
401
+ 46%|████████████████████████████████████▌ | 499/1090 [08:37<09:00, 1.09it/s]
402
+ 46%|████████████████████████████████████▋ | 500/1090 [08:38<08:59, 1.09it/s][INFO|trainer.py:2889] 2024-02-01 17:47:14,118 >> Saving model checkpoint to ./tmp-checkpoint-500
403
+ [INFO|configuration_utils.py:483] 2024-02-01 17:47:14,121 >> Configuration saved in ./tmp-checkpoint-500/config.json
404
+ [INFO|configuration_utils.py:594] 2024-02-01 17:47:14,124 >> Configuration saved in ./tmp-checkpoint-500/generation_config.json
405
+ [2024-02-01 17:47:17,242] [INFO] [logging.py:96:log_dist] [Rank 0] [Torch] Checkpoint global_step500 is about to be saved!
406
+ [2024-02-01 17:47:17,247] [INFO] [logging.py:96:log_dist] [Rank 0] Saving model checkpoint: ./tmp-checkpoint-500/global_step500/zero_pp_rank_0_mp_rank_00_model_states.pt
407
+ [2024-02-01 17:47:17,247] [INFO] [torch_checkpoint_engine.py:21:save] [Torch] Saving ./tmp-checkpoint-500/global_step500/zero_pp_rank_0_mp_rank_00_model_states.pt...
408
+ [2024-02-01 17:47:17,368] [INFO] [torch_checkpoint_engine.py:23:save] [Torch] Saved ./tmp-checkpoint-500/global_step500/zero_pp_rank_0_mp_rank_00_model_states.pt.
409
+ [2024-02-01 17:47:17,373] [INFO] [torch_checkpoint_engine.py:21:save] [Torch] Saving ./tmp-checkpoint-500/global_step500/bf16_zero_pp_rank_0_mp_rank_00_optim_states.pt...
410
+ [INFO|modeling_utils.py:2382] 2024-02-01 17:47:17,211 >> Model weights saved in ./tmp-checkpoint-500/pytorch_model.bin
411
+ [INFO|tokenization_utils_base.py:2432] 2024-02-01 17:47:17,214 >> tokenizer config file saved in ./tmp-checkpoint-500/tokenizer_config.json
412
+ [INFO|tokenization_utils_base.py:2441] 2024-02-01 17:47:17,216 >> Special tokens file saved in ./tmp-checkpoint-500/special_tokens_map.json
413
+ /fsx/sanchit/miniconda3/envs/venv/lib/python3.11/site-packages/torch/nn/modules/module.py:1879: UserWarning: Positional args are being deprecated, use kwargs instead. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details.
414
+ warnings.warn(
415
+ [2024-02-01 17:47:20,828] [INFO] [torch_checkpoint_engine.py:23:save] [Torch] Saved ./tmp-checkpoint-500/global_step500/bf16_zero_pp_rank_0_mp_rank_00_optim_states.pt.
416
+ [2024-02-01 17:47:20,833] [INFO] [engine.py:3393:_save_zero_checkpoint] zero checkpoint saved ./tmp-checkpoint-500/global_step500/bf16_zero_pp_rank_0_mp_rank_00_optim_states.pt
417
+ [2024-02-01 17:47:21,476] [INFO] [torch_checkpoint_engine.py:33:commit] [Torch] Checkpoint global_step500 is ready now!
418
+ [INFO|tokenization_utils_base.py:2432] 2024-02-01 17:47:24,306 >> tokenizer config file saved in ./tokenizer_config.json
419
+ [INFO|tokenization_utils_base.py:2441] 2024-02-01 17:47:24,308 >> Special tokens file saved in ./special_tokens_map.json
420
+ [INFO|trainer.py:2979] 2024-02-01 17:47:24,337 >> Deleting older checkpoint [checkpoint-400] due to args.save_total_limit
421
+
422
+ 46%|████████████████████████████████████▉ | 504/1090 [08:56<23:43, 2.43s/it]
423
+
424
+
425
+ 47%|█████████████████████████████████████▎ | 509/1090 [09:01<11:17, 1.17s/it]
426
+
427
+
428
+ 47%|█████████████████████████████████████▋ | 513/1090 [09:05<09:20, 1.03it/s]
429
+
430
+
431
+
432
+ 48%|██████████��███████████████████████████▏ | 520/1090 [09:11<08:43, 1.09it/s]
433
+
434
+
435
+ 48%|██████████████████████████████████████▍ | 524/1090 [09:15<08:38, 1.09it/s]
436
+
437
+
438
+ 49%|██████████████████████████████████████▊ | 529/1090 [09:19<08:35, 1.09it/s]
439
+
440
+
441
+
442
+ 49%|███████████████████████████████████████▎ | 535/1090 [09:25<08:29, 1.09it/s]
443
+
444
+
445
+ 49%|███████████████████████████████████████▌ | 539/1090 [09:28<08:24, 1.09it/s]
446
+
447
+
448
+ 50%|███████████████████████████████████████▉ | 544/1090 [09:33<08:19, 1.09it/s]
449
+
450
+
451
+ 50%|████████████████████████████████████████▏ | 548/1090 [09:37<08:15, 1.09it/s]
452
+
453
+
454
+
455
+ 51%|████████████████████████████████████████▋ | 555/1090 [09:43<08:09, 1.09it/s]
456
+
457
+
458
+ 51%|█████████████████████████████████████████ | 559/1090 [09:47<08:10, 1.08it/s]
459
+
460
+
461
+ 52%|█████████████████████████████████████████▍ | 564/1090 [09:51<08:01, 1.09it/s]
462
+
wandb/run-20240201_173828-py26nu6m/files/wandb-summary.json CHANGED
@@ -1 +1 @@
1
- {"train/loss": 3.2315, "train/learning_rate": 1.4175324596392075e-05, "train/epoch": 0.43, "train/global_step": 465, "_timestamp": 1706809601.0518904, "_runtime": 492.6000163555145, "_step": 93}
 
1
+ {"train/loss": 2.9132, "train/learning_rate": 1.1102595071354471e-05, "train/epoch": 0.52, "train/global_step": 565, "_timestamp": 1706809707.2731094, "_runtime": 598.8212354183197, "_step": 113}
wandb/run-20240201_173828-py26nu6m/logs/debug-internal.log CHANGED
@@ -955,3 +955,198 @@
955
  2024-02-01 17:46:41,455 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/wandb-summary.json
956
  2024-02-01 17:46:41,456 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/output.log
957
  2024-02-01 17:46:42,457 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/output.log
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
955
  2024-02-01 17:46:41,455 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/wandb-summary.json
956
  2024-02-01 17:46:41,456 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/output.log
957
  2024-02-01 17:46:42,457 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/output.log
958
+ 2024-02-01 17:46:45,461 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/output.log
959
+ 2024-02-01 17:46:45,624 DEBUG HandlerThread:237521 [handler.py:handle_request():146] handle_request: status_report
960
+ 2024-02-01 17:46:45,625 DEBUG HandlerThread:237521 [handler.py:handle_request():146] handle_request: partial_history
961
+ 2024-02-01 17:46:45,626 DEBUG SenderThread:237521 [sender.py:send():382] send: history
962
+ 2024-02-01 17:46:45,626 DEBUG SenderThread:237521 [sender.py:send_request():409] send_request: summary_record
963
+ 2024-02-01 17:46:45,628 INFO SenderThread:237521 [sender.py:_save_file():1392] saving file wandb-summary.json with policy end
964
+ 2024-02-01 17:46:46,463 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/wandb-summary.json
965
+ 2024-02-01 17:46:46,463 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/output.log
966
+ 2024-02-01 17:46:49,467 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/output.log
967
+ 2024-02-01 17:46:50,021 DEBUG HandlerThread:237521 [handler.py:handle_request():146] handle_request: stop_status
968
+ 2024-02-01 17:46:50,021 DEBUG SenderThread:237521 [sender.py:send_request():409] send_request: stop_status
969
+ 2024-02-01 17:46:50,054 DEBUG HandlerThread:237521 [handler.py:handle_request():146] handle_request: internal_messages
970
+ 2024-02-01 17:46:50,238 DEBUG HandlerThread:237521 [handler.py:handle_request():146] handle_request: partial_history
971
+ 2024-02-01 17:46:51,012 DEBUG SenderThread:237521 [sender.py:send():382] send: history
972
+ 2024-02-01 17:46:51,012 DEBUG SenderThread:237521 [sender.py:send_request():409] send_request: summary_record
973
+ 2024-02-01 17:46:51,013 DEBUG HandlerThread:237521 [handler.py:handle_request():146] handle_request: status_report
974
+ 2024-02-01 17:46:51,014 INFO SenderThread:237521 [sender.py:_save_file():1392] saving file wandb-summary.json with policy end
975
+ 2024-02-01 17:46:51,470 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/wandb-summary.json
976
+ 2024-02-01 17:46:51,471 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/output.log
977
+ 2024-02-01 17:46:52,472 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/output.log
978
+ 2024-02-01 17:46:54,767 DEBUG HandlerThread:237521 [handler.py:handle_request():146] handle_request: partial_history
979
+ 2024-02-01 17:46:54,768 DEBUG SenderThread:237521 [sender.py:send():382] send: history
980
+ 2024-02-01 17:46:54,768 DEBUG SenderThread:237521 [sender.py:send_request():409] send_request: summary_record
981
+ 2024-02-01 17:46:54,770 INFO SenderThread:237521 [sender.py:_save_file():1392] saving file wandb-summary.json with policy end
982
+ 2024-02-01 17:46:55,477 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/wandb-summary.json
983
+ 2024-02-01 17:46:55,477 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/output.log
984
+ 2024-02-01 17:46:56,478 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/output.log
985
+ 2024-02-01 17:46:56,591 DEBUG HandlerThread:237521 [handler.py:handle_request():146] handle_request: status_report
986
+ 2024-02-01 17:46:58,869 DEBUG SenderThread:237521 [sender.py:send():382] send: stats
987
+ 2024-02-01 17:46:59,332 DEBUG HandlerThread:237521 [handler.py:handle_request():146] handle_request: partial_history
988
+ 2024-02-01 17:46:59,333 DEBUG SenderThread:237521 [sender.py:send():382] send: history
989
+ 2024-02-01 17:46:59,333 DEBUG SenderThread:237521 [sender.py:send_request():409] send_request: summary_record
990
+ 2024-02-01 17:46:59,335 INFO SenderThread:237521 [sender.py:_save_file():1392] saving file wandb-summary.json with policy end
991
+ 2024-02-01 17:46:59,483 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/wandb-summary.json
992
+ 2024-02-01 17:46:59,483 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/output.log
993
+ 2024-02-01 17:47:00,484 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/output.log
994
+ 2024-02-01 17:47:02,072 DEBUG HandlerThread:237521 [handler.py:handle_request():146] handle_request: status_report
995
+ 2024-02-01 17:47:03,488 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/output.log
996
+ 2024-02-01 17:47:03,905 DEBUG HandlerThread:237521 [handler.py:handle_request():146] handle_request: partial_history
997
+ 2024-02-01 17:47:03,906 DEBUG SenderThread:237521 [sender.py:send():382] send: history
998
+ 2024-02-01 17:47:03,907 DEBUG SenderThread:237521 [sender.py:send_request():409] send_request: summary_record
999
+ 2024-02-01 17:47:03,909 INFO SenderThread:237521 [sender.py:_save_file():1392] saving file wandb-summary.json with policy end
1000
+ 2024-02-01 17:47:04,491 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/wandb-summary.json
1001
+ 2024-02-01 17:47:05,021 DEBUG HandlerThread:237521 [handler.py:handle_request():146] handle_request: stop_status
1002
+ 2024-02-01 17:47:05,021 DEBUG SenderThread:237521 [sender.py:send_request():409] send_request: stop_status
1003
+ 2024-02-01 17:47:05,054 DEBUG HandlerThread:237521 [handler.py:handle_request():146] handle_request: internal_messages
1004
+ 2024-02-01 17:47:05,492 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/output.log
1005
+ 2024-02-01 17:47:06,493 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/output.log
1006
+ 2024-02-01 17:47:07,565 DEBUG HandlerThread:237521 [handler.py:handle_request():146] handle_request: status_report
1007
+ 2024-02-01 17:47:08,481 DEBUG HandlerThread:237521 [handler.py:handle_request():146] handle_request: partial_history
1008
+ 2024-02-01 17:47:08,482 DEBUG SenderThread:237521 [sender.py:send():382] send: history
1009
+ 2024-02-01 17:47:08,483 DEBUG SenderThread:237521 [sender.py:send_request():409] send_request: summary_record
1010
+ 2024-02-01 17:47:08,485 INFO SenderThread:237521 [sender.py:_save_file():1392] saving file wandb-summary.json with policy end
1011
+ 2024-02-01 17:47:08,497 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/wandb-summary.json
1012
+ 2024-02-01 17:47:09,498 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/output.log
1013
+ 2024-02-01 17:47:10,499 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/output.log
1014
+ 2024-02-01 17:47:13,050 DEBUG HandlerThread:237521 [handler.py:handle_request():146] handle_request: status_report
1015
+ 2024-02-01 17:47:13,051 DEBUG HandlerThread:237521 [handler.py:handle_request():146] handle_request: partial_history
1016
+ 2024-02-01 17:47:13,052 DEBUG SenderThread:237521 [sender.py:send():382] send: history
1017
+ 2024-02-01 17:47:13,052 DEBUG SenderThread:237521 [sender.py:send_request():409] send_request: summary_record
1018
+ 2024-02-01 17:47:13,054 INFO SenderThread:237521 [sender.py:_save_file():1392] saving file wandb-summary.json with policy end
1019
+ 2024-02-01 17:47:13,504 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/wandb-summary.json
1020
+ 2024-02-01 17:47:13,504 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/output.log
1021
+ 2024-02-01 17:47:14,505 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/output.log
1022
+ 2024-02-01 17:47:18,373 DEBUG HandlerThread:237521 [handler.py:handle_request():146] handle_request: status_report
1023
+ 2024-02-01 17:47:18,511 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/output.log
1024
+ 2024-02-01 17:47:20,830 DEBUG HandlerThread:237521 [handler.py:handle_request():146] handle_request: internal_messages
1025
+ 2024-02-01 17:47:20,830 DEBUG HandlerThread:237521 [handler.py:handle_request():146] handle_request: stop_status
1026
+ 2024-02-01 17:47:20,831 DEBUG SenderThread:237521 [sender.py:send_request():409] send_request: stop_status
1027
+ 2024-02-01 17:47:22,518 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/output.log
1028
+ 2024-02-01 17:47:23,478 DEBUG HandlerThread:237521 [handler.py:handle_request():146] handle_request: status_report
1029
+ 2024-02-01 17:47:25,521 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/output.log
1030
+ 2024-02-01 17:47:27,524 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/output.log
1031
+ 2024-02-01 17:47:28,710 DEBUG HandlerThread:237521 [handler.py:handle_request():146] handle_request: status_report
1032
+ 2024-02-01 17:47:28,871 DEBUG SenderThread:237521 [sender.py:send():382] send: stats
1033
+ 2024-02-01 17:47:31,531 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/output.log
1034
+ 2024-02-01 17:47:32,347 DEBUG HandlerThread:237521 [handler.py:handle_request():146] handle_request: partial_history
1035
+ 2024-02-01 17:47:32,349 DEBUG SenderThread:237521 [sender.py:send():382] send: history
1036
+ 2024-02-01 17:47:32,349 DEBUG SenderThread:237521 [sender.py:send_request():409] send_request: summary_record
1037
+ 2024-02-01 17:47:32,351 INFO SenderThread:237521 [sender.py:_save_file():1392] saving file wandb-summary.json with policy end
1038
+ 2024-02-01 17:47:32,533 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/wandb-summary.json
1039
+ 2024-02-01 17:47:33,535 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/output.log
1040
+ 2024-02-01 17:47:34,171 DEBUG HandlerThread:237521 [handler.py:handle_request():146] handle_request: status_report
1041
+ 2024-02-01 17:47:34,537 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/output.log
1042
+ 2024-02-01 17:47:35,788 DEBUG HandlerThread:237521 [handler.py:handle_request():146] handle_request: internal_messages
1043
+ 2024-02-01 17:47:35,788 DEBUG HandlerThread:237521 [handler.py:handle_request():146] handle_request: stop_status
1044
+ 2024-02-01 17:47:35,788 DEBUG SenderThread:237521 [sender.py:send_request():409] send_request: stop_status
1045
+ 2024-02-01 17:47:36,906 DEBUG HandlerThread:237521 [handler.py:handle_request():146] handle_request: partial_history
1046
+ 2024-02-01 17:47:36,907 DEBUG SenderThread:237521 [sender.py:send():382] send: history
1047
+ 2024-02-01 17:47:36,908 DEBUG SenderThread:237521 [sender.py:send_request():409] send_request: summary_record
1048
+ 2024-02-01 17:47:36,912 INFO SenderThread:237521 [sender.py:_save_file():1392] saving file wandb-summary.json with policy end
1049
+ 2024-02-01 17:47:37,547 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/wandb-summary.json
1050
+ 2024-02-01 17:47:37,547 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/output.log
1051
+ 2024-02-01 17:47:38,549 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/output.log
1052
+ 2024-02-01 17:47:39,636 DEBUG HandlerThread:237521 [handler.py:handle_request():146] handle_request: status_report
1053
+ 2024-02-01 17:47:41,486 DEBUG HandlerThread:237521 [handler.py:handle_request():146] handle_request: partial_history
1054
+ 2024-02-01 17:47:41,487 DEBUG SenderThread:237521 [sender.py:send():382] send: history
1055
+ 2024-02-01 17:47:41,488 DEBUG SenderThread:237521 [sender.py:send_request():409] send_request: summary_record
1056
+ 2024-02-01 17:47:41,498 INFO SenderThread:237521 [sender.py:_save_file():1392] saving file wandb-summary.json with policy end
1057
+ 2024-02-01 17:47:41,560 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/wandb-summary.json
1058
+ 2024-02-01 17:47:41,560 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/output.log
1059
+ 2024-02-01 17:47:42,562 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/output.log
1060
+ 2024-02-01 17:47:45,135 DEBUG HandlerThread:237521 [handler.py:handle_request():146] handle_request: status_report
1061
+ 2024-02-01 17:47:45,570 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/output.log
1062
+ 2024-02-01 17:47:46,052 DEBUG HandlerThread:237521 [handler.py:handle_request():146] handle_request: partial_history
1063
+ 2024-02-01 17:47:46,053 DEBUG SenderThread:237521 [sender.py:send():382] send: history
1064
+ 2024-02-01 17:47:46,053 DEBUG SenderThread:237521 [sender.py:send_request():409] send_request: summary_record
1065
+ 2024-02-01 17:47:46,063 INFO SenderThread:237521 [sender.py:_save_file():1392] saving file wandb-summary.json with policy end
1066
+ 2024-02-01 17:47:46,575 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/wandb-summary.json
1067
+ 2024-02-01 17:47:47,578 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/output.log
1068
+ 2024-02-01 17:47:48,580 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/output.log
1069
+ 2024-02-01 17:47:50,626 DEBUG HandlerThread:237521 [handler.py:handle_request():146] handle_request: status_report
1070
+ 2024-02-01 17:47:50,632 DEBUG HandlerThread:237521 [handler.py:handle_request():146] handle_request: partial_history
1071
+ 2024-02-01 17:47:50,633 DEBUG SenderThread:237521 [sender.py:send():382] send: history
1072
+ 2024-02-01 17:47:50,634 DEBUG SenderThread:237521 [sender.py:send_request():409] send_request: summary_record
1073
+ 2024-02-01 17:47:50,639 INFO SenderThread:237521 [sender.py:_save_file():1392] saving file wandb-summary.json with policy end
1074
+ 2024-02-01 17:47:50,787 DEBUG HandlerThread:237521 [handler.py:handle_request():146] handle_request: internal_messages
1075
+ 2024-02-01 17:47:50,788 DEBUG HandlerThread:237521 [handler.py:handle_request():146] handle_request: stop_status
1076
+ 2024-02-01 17:47:50,788 DEBUG SenderThread:237521 [sender.py:send_request():409] send_request: stop_status
1077
+ 2024-02-01 17:47:51,596 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/wandb-summary.json
1078
+ 2024-02-01 17:47:51,596 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/output.log
1079
+ 2024-02-01 17:47:52,598 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/output.log
1080
+ 2024-02-01 17:47:55,212 DEBUG HandlerThread:237521 [handler.py:handle_request():146] handle_request: partial_history
1081
+ 2024-02-01 17:47:55,214 DEBUG SenderThread:237521 [sender.py:send():382] send: history
1082
+ 2024-02-01 17:47:55,214 DEBUG SenderThread:237521 [sender.py:send_request():409] send_request: summary_record
1083
+ 2024-02-01 17:47:55,224 INFO SenderThread:237521 [sender.py:_save_file():1392] saving file wandb-summary.json with policy end
1084
+ 2024-02-01 17:47:55,610 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/wandb-summary.json
1085
+ 2024-02-01 17:47:55,610 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/output.log
1086
+ 2024-02-01 17:47:56,124 DEBUG HandlerThread:237521 [handler.py:handle_request():146] handle_request: status_report
1087
+ 2024-02-01 17:47:56,613 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/output.log
1088
+ 2024-02-01 17:47:58,879 DEBUG SenderThread:237521 [sender.py:send():382] send: stats
1089
+ 2024-02-01 17:47:59,620 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/output.log
1090
+ 2024-02-01 17:47:59,797 DEBUG HandlerThread:237521 [handler.py:handle_request():146] handle_request: partial_history
1091
+ 2024-02-01 17:47:59,799 DEBUG SenderThread:237521 [sender.py:send():382] send: history
1092
+ 2024-02-01 17:47:59,799 DEBUG SenderThread:237521 [sender.py:send_request():409] send_request: summary_record
1093
+ 2024-02-01 17:47:59,801 INFO SenderThread:237521 [sender.py:_save_file():1392] saving file wandb-summary.json with policy end
1094
+ 2024-02-01 17:48:00,625 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/wandb-summary.json
1095
+ 2024-02-01 17:48:01,620 DEBUG HandlerThread:237521 [handler.py:handle_request():146] handle_request: status_report
1096
+ 2024-02-01 17:48:01,627 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/output.log
1097
+ 2024-02-01 17:48:02,629 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/output.log
1098
+ 2024-02-01 17:48:04,366 DEBUG HandlerThread:237521 [handler.py:handle_request():146] handle_request: partial_history
1099
+ 2024-02-01 17:48:04,367 DEBUG SenderThread:237521 [sender.py:send():382] send: history
1100
+ 2024-02-01 17:48:04,368 DEBUG SenderThread:237521 [sender.py:send_request():409] send_request: summary_record
1101
+ 2024-02-01 17:48:04,369 INFO SenderThread:237521 [sender.py:_save_file():1392] saving file wandb-summary.json with policy end
1102
+ 2024-02-01 17:48:04,634 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/wandb-summary.json
1103
+ 2024-02-01 17:48:05,635 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/output.log
1104
+ 2024-02-01 17:48:05,788 DEBUG HandlerThread:237521 [handler.py:handle_request():146] handle_request: stop_status
1105
+ 2024-02-01 17:48:05,788 DEBUG SenderThread:237521 [sender.py:send_request():409] send_request: stop_status
1106
+ 2024-02-01 17:48:05,830 DEBUG HandlerThread:237521 [handler.py:handle_request():146] handle_request: internal_messages
1107
+ 2024-02-01 17:48:06,637 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/output.log
1108
+ 2024-02-01 17:48:07,105 DEBUG HandlerThread:237521 [handler.py:handle_request():146] handle_request: status_report
1109
+ 2024-02-01 17:48:08,934 DEBUG HandlerThread:237521 [handler.py:handle_request():146] handle_request: partial_history
1110
+ 2024-02-01 17:48:08,935 DEBUG SenderThread:237521 [sender.py:send():382] send: history
1111
+ 2024-02-01 17:48:08,936 DEBUG SenderThread:237521 [sender.py:send_request():409] send_request: summary_record
1112
+ 2024-02-01 17:48:08,938 INFO SenderThread:237521 [sender.py:_save_file():1392] saving file wandb-summary.json with policy end
1113
+ 2024-02-01 17:48:09,646 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/wandb-summary.json
1114
+ 2024-02-01 17:48:09,647 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/output.log
1115
+ 2024-02-01 17:48:10,648 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/output.log
1116
+ 2024-02-01 17:48:12,590 DEBUG HandlerThread:237521 [handler.py:handle_request():146] handle_request: status_report
1117
+ 2024-02-01 17:48:13,507 DEBUG HandlerThread:237521 [handler.py:handle_request():146] handle_request: partial_history
1118
+ 2024-02-01 17:48:13,508 DEBUG SenderThread:237521 [sender.py:send():382] send: history
1119
+ 2024-02-01 17:48:13,509 DEBUG SenderThread:237521 [sender.py:send_request():409] send_request: summary_record
1120
+ 2024-02-01 17:48:13,511 INFO SenderThread:237521 [sender.py:_save_file():1392] saving file wandb-summary.json with policy end
1121
+ 2024-02-01 17:48:13,653 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/wandb-summary.json
1122
+ 2024-02-01 17:48:13,653 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/output.log
1123
+ 2024-02-01 17:48:14,654 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/output.log
1124
+ 2024-02-01 17:48:17,658 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/output.log
1125
+ 2024-02-01 17:48:18,084 DEBUG HandlerThread:237521 [handler.py:handle_request():146] handle_request: status_report
1126
+ 2024-02-01 17:48:18,085 DEBUG HandlerThread:237521 [handler.py:handle_request():146] handle_request: partial_history
1127
+ 2024-02-01 17:48:18,086 DEBUG SenderThread:237521 [sender.py:send():382] send: history
1128
+ 2024-02-01 17:48:18,087 DEBUG SenderThread:237521 [sender.py:send_request():409] send_request: summary_record
1129
+ 2024-02-01 17:48:18,089 INFO SenderThread:237521 [sender.py:_save_file():1392] saving file wandb-summary.json with policy end
1130
+ 2024-02-01 17:48:18,660 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/wandb-summary.json
1131
+ 2024-02-01 17:48:19,662 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/output.log
1132
+ 2024-02-01 17:48:20,663 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/output.log
1133
+ 2024-02-01 17:48:20,788 DEBUG HandlerThread:237521 [handler.py:handle_request():146] handle_request: stop_status
1134
+ 2024-02-01 17:48:20,788 DEBUG SenderThread:237521 [sender.py:send_request():409] send_request: stop_status
1135
+ 2024-02-01 17:48:20,830 DEBUG HandlerThread:237521 [handler.py:handle_request():146] handle_request: internal_messages
1136
+ 2024-02-01 17:48:22,694 DEBUG HandlerThread:237521 [handler.py:handle_request():146] handle_request: partial_history
1137
+ 2024-02-01 17:48:22,696 DEBUG SenderThread:237521 [sender.py:send():382] send: history
1138
+ 2024-02-01 17:48:22,696 DEBUG SenderThread:237521 [sender.py:send_request():409] send_request: summary_record
1139
+ 2024-02-01 17:48:22,698 INFO SenderThread:237521 [sender.py:_save_file():1392] saving file wandb-summary.json with policy end
1140
+ 2024-02-01 17:48:23,606 DEBUG HandlerThread:237521 [handler.py:handle_request():146] handle_request: status_report
1141
+ 2024-02-01 17:48:23,668 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/wandb-summary.json
1142
+ 2024-02-01 17:48:23,668 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/output.log
1143
+ 2024-02-01 17:48:24,670 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/output.log
1144
+ 2024-02-01 17:48:27,273 DEBUG HandlerThread:237521 [handler.py:handle_request():146] handle_request: partial_history
1145
+ 2024-02-01 17:48:27,274 DEBUG SenderThread:237521 [sender.py:send():382] send: history
1146
+ 2024-02-01 17:48:27,275 DEBUG SenderThread:237521 [sender.py:send_request():409] send_request: summary_record
1147
+ 2024-02-01 17:48:27,281 INFO SenderThread:237521 [sender.py:_save_file():1392] saving file wandb-summary.json with policy end
1148
+ 2024-02-01 17:48:27,678 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/wandb-summary.json
1149
+ 2024-02-01 17:48:27,678 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/output.log
1150
+ 2024-02-01 17:48:28,680 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/output.log
1151
+ 2024-02-01 17:48:28,876 DEBUG SenderThread:237521 [sender.py:send():382] send: stats
1152
+ 2024-02-01 17:48:28,876 DEBUG HandlerThread:237521 [handler.py:handle_request():146] handle_request: status_report
wandb/run-20240201_173828-py26nu6m/run-py26nu6m.wandb CHANGED
Binary files a/wandb/run-20240201_173828-py26nu6m/run-py26nu6m.wandb and b/wandb/run-20240201_173828-py26nu6m/run-py26nu6m.wandb differ