| 2022-04-04 18:12:20,609 INFO MainThread:1244997 [internal.py:wandb_internal():92] W&B internal server running at pid: 1244997, started at: 2022-04-04 18:12:20.609124 | |
| 2022-04-04 18:12:20,612 DEBUG SenderThread:1244997 [sender.py:send():235] send: header | |
| 2022-04-04 18:12:20,612 DEBUG SenderThread:1244997 [sender.py:send():235] send: run | |
| 2022-04-04 18:12:20,617 INFO WriterThread:1244997 [datastore.py:open_for_write():77] open: /home/sanchitgandhi/flax-wav2vec2-2-bart-large-cnn/wandb/run-20220404_181219-4s004g1k/run-4s004g1k.wandb | |
| 2022-04-04 18:12:20,695 INFO SenderThread:1244997 [dir_watcher.py:__init__():169] watching files in: /home/sanchitgandhi/flax-wav2vec2-2-bart-large-cnn/wandb/run-20220404_181219-4s004g1k/files | |
| 2022-04-04 18:12:20,695 INFO SenderThread:1244997 [sender.py:_start_run_threads():812] run started: 4s004g1k with start time 1649095939 | |
| 2022-04-04 18:12:20,695 DEBUG SenderThread:1244997 [sender.py:send():235] send: summary | |
| 2022-04-04 18:12:20,695 INFO SenderThread:1244997 [sender.py:_save_file():947] saving file wandb-summary.json with policy end | |
| 2022-04-04 18:12:20,698 DEBUG HandlerThread:1244997 [handler.py:handle_request():141] handle_request: check_version | |
| 2022-04-04 18:12:20,698 DEBUG SenderThread:1244997 [sender.py:send_request():249] send_request: check_version | |
| 2022-04-04 18:12:20,759 DEBUG HandlerThread:1244997 [handler.py:handle_request():141] handle_request: run_start | |
| 2022-04-04 18:12:21,698 INFO Thread-7 :1244997 [dir_watcher.py:_on_file_created():217] file/dir created: /home/sanchitgandhi/flax-wav2vec2-2-bart-large-cnn/wandb/run-20220404_181219-4s004g1k/files/wandb-summary.json | |
| 2022-04-04 18:12:23,008 DEBUG HandlerThread:1244997 [meta.py:__init__():37] meta init | |
| 2022-04-04 18:12:23,008 DEBUG HandlerThread:1244997 [meta.py:__init__():51] meta init done | |
| 2022-04-04 18:12:23,008 DEBUG HandlerThread:1244997 [meta.py:probe():211] probe | |
| 2022-04-04 18:12:23,009 DEBUG HandlerThread:1244997 [meta.py:_setup_git():201] setup git | |
| 2022-04-04 18:12:23,041 DEBUG HandlerThread:1244997 [meta.py:_setup_git():208] setup git done | |
| 2022-04-04 18:12:23,041 DEBUG HandlerThread:1244997 [meta.py:_save_pip():55] save pip | |
| 2022-04-04 18:12:23,041 DEBUG HandlerThread:1244997 [meta.py:_save_pip():69] save pip done | |
| 2022-04-04 18:12:23,041 DEBUG HandlerThread:1244997 [meta.py:probe():249] probe done | |
| 2022-04-04 18:12:23,044 DEBUG SenderThread:1244997 [sender.py:send():235] send: files | |
| 2022-04-04 18:12:23,045 INFO SenderThread:1244997 [sender.py:_save_file():947] saving file wandb-metadata.json with policy now | |
| 2022-04-04 18:12:23,051 DEBUG HandlerThread:1244997 [handler.py:handle_request():141] handle_request: stop_status | |
| 2022-04-04 18:12:23,052 DEBUG SenderThread:1244997 [sender.py:send_request():249] send_request: stop_status | |
| 2022-04-04 18:12:23,584 DEBUG SenderThread:1244997 [sender.py:send():235] send: telemetry | |
| 2022-04-04 18:12:23,699 INFO Thread-7 :1244997 [dir_watcher.py:_on_file_created():217] file/dir created: /home/sanchitgandhi/flax-wav2vec2-2-bart-large-cnn/wandb/run-20220404_181219-4s004g1k/files/requirements.txt | |
| 2022-04-04 18:12:23,700 INFO Thread-7 :1244997 [dir_watcher.py:_on_file_created():217] file/dir created: /home/sanchitgandhi/flax-wav2vec2-2-bart-large-cnn/wandb/run-20220404_181219-4s004g1k/files/wandb-metadata.json | |
| 2022-04-04 18:12:23,700 INFO Thread-7 :1244997 [dir_watcher.py:_on_file_created():217] file/dir created: /home/sanchitgandhi/flax-wav2vec2-2-bart-large-cnn/wandb/run-20220404_181219-4s004g1k/files/output.log | |
| 2022-04-04 18:12:23,749 INFO Thread-11 :1244997 [upload_job.py:push():137] Uploaded file /tmp/tmpgn5_775mwandb/366y469x-wandb-metadata.json | |
| 2022-04-04 18:12:25,700 INFO Thread-7 :1244997 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/sanchitgandhi/flax-wav2vec2-2-bart-large-cnn/wandb/run-20220404_181219-4s004g1k/files/output.log | |
| 2022-04-04 18:12:38,890 DEBUG HandlerThread:1244997 [handler.py:handle_request():141] handle_request: stop_status | |
| 2022-04-04 18:12:38,891 DEBUG SenderThread:1244997 [sender.py:send_request():249] send_request: stop_status | |
| 2022-04-04 18:12:39,705 INFO Thread-7 :1244997 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/sanchitgandhi/flax-wav2vec2-2-bart-large-cnn/wandb/run-20220404_181219-4s004g1k/files/output.log | |
| 2022-04-04 18:12:41,706 INFO Thread-7 :1244997 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/sanchitgandhi/flax-wav2vec2-2-bart-large-cnn/wandb/run-20220404_181219-4s004g1k/files/output.log | |
| 2022-04-04 18:12:43,369 DEBUG SenderThread:1244997 [sender.py:send():235] send: telemetry | |
| 2022-04-04 18:12:43,369 DEBUG HandlerThread:1244997 [handler.py:handle_request():141] handle_request: partial_history | |
| 2022-04-04 18:12:43,707 INFO Thread-7 :1244997 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/sanchitgandhi/flax-wav2vec2-2-bart-large-cnn/wandb/run-20220404_181219-4s004g1k/files/output.log | |
| 2022-04-04 18:12:45,709 INFO Thread-7 :1244997 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/sanchitgandhi/flax-wav2vec2-2-bart-large-cnn/wandb/run-20220404_181219-4s004g1k/files/output.log | |
| 2022-04-04 18:12:45,972 DEBUG HandlerThread:1244997 [handler.py:handle_request():141] handle_request: poll_exit | |
| 2022-04-04 18:12:45,972 DEBUG SenderThread:1244997 [sender.py:send():235] send: exit | |
| 2022-04-04 18:12:45,972 INFO SenderThread:1244997 [sender.py:send_exit():371] handling exit code: 1 | |
| 2022-04-04 18:12:45,972 INFO SenderThread:1244997 [sender.py:send_exit():373] handling runtime: 25 | |
| 2022-04-04 18:12:45,973 INFO SenderThread:1244997 [sender.py:_save_file():947] saving file wandb-summary.json with policy end | |
| 2022-04-04 18:12:45,973 INFO SenderThread:1244997 [sender.py:send_exit():379] send defer | |
| 2022-04-04 18:12:45,973 DEBUG SenderThread:1244997 [sender.py:send_request():249] send_request: poll_exit | |
| 2022-04-04 18:12:45,974 DEBUG HandlerThread:1244997 [handler.py:handle_request():141] handle_request: defer | |
| 2022-04-04 18:12:45,974 INFO HandlerThread:1244997 [handler.py:handle_request_defer():164] handle defer: 0 | |
| 2022-04-04 18:12:45,974 DEBUG SenderThread:1244997 [sender.py:send_request():249] send_request: defer | |
| 2022-04-04 18:12:45,974 INFO SenderThread:1244997 [sender.py:send_request_defer():388] handle sender defer: 0 | |
| 2022-04-04 18:12:45,974 INFO SenderThread:1244997 [sender.py:transition_state():392] send defer: 1 | |
| 2022-04-04 18:12:45,974 DEBUG HandlerThread:1244997 [handler.py:handle_request():141] handle_request: defer | |
| 2022-04-04 18:12:45,974 INFO HandlerThread:1244997 [handler.py:handle_request_defer():164] handle defer: 1 | |
| 2022-04-04 18:12:45,979 DEBUG SenderThread:1244997 [sender.py:send_request():249] send_request: defer | |
| 2022-04-04 18:12:45,979 INFO SenderThread:1244997 [sender.py:send_request_defer():388] handle sender defer: 1 | |
| 2022-04-04 18:12:45,979 INFO SenderThread:1244997 [sender.py:transition_state():392] send defer: 2 | |
| 2022-04-04 18:12:45,979 DEBUG SenderThread:1244997 [sender.py:send():235] send: stats | |
| 2022-04-04 18:12:45,979 DEBUG HandlerThread:1244997 [handler.py:handle_request():141] handle_request: defer | |
| 2022-04-04 18:12:45,980 INFO HandlerThread:1244997 [handler.py:handle_request_defer():164] handle defer: 2 | |
| 2022-04-04 18:12:45,980 DEBUG SenderThread:1244997 [sender.py:send_request():249] send_request: defer | |
| 2022-04-04 18:12:45,980 INFO SenderThread:1244997 [sender.py:send_request_defer():388] handle sender defer: 2 | |
| 2022-04-04 18:12:45,980 INFO SenderThread:1244997 [sender.py:transition_state():392] send defer: 3 | |
| 2022-04-04 18:12:45,980 DEBUG HandlerThread:1244997 [handler.py:handle_request():141] handle_request: defer | |
| 2022-04-04 18:12:45,980 INFO HandlerThread:1244997 [handler.py:handle_request_defer():164] handle defer: 3 | |
| 2022-04-04 18:12:45,980 DEBUG SenderThread:1244997 [sender.py:send_request():249] send_request: defer | |
| 2022-04-04 18:12:45,980 INFO SenderThread:1244997 [sender.py:send_request_defer():388] handle sender defer: 3 | |
| 2022-04-04 18:12:45,980 INFO SenderThread:1244997 [sender.py:transition_state():392] send defer: 4 | |
| 2022-04-04 18:12:45,980 DEBUG HandlerThread:1244997 [handler.py:handle_request():141] handle_request: defer | |
| 2022-04-04 18:12:45,981 INFO HandlerThread:1244997 [handler.py:handle_request_defer():164] handle defer: 4 | |
| 2022-04-04 18:12:45,981 DEBUG SenderThread:1244997 [sender.py:send():235] send: summary | |
| 2022-04-04 18:12:45,981 INFO SenderThread:1244997 [sender.py:_save_file():947] saving file wandb-summary.json with policy end | |
| 2022-04-04 18:12:45,981 DEBUG SenderThread:1244997 [sender.py:send_request():249] send_request: defer | |
| 2022-04-04 18:12:45,981 INFO SenderThread:1244997 [sender.py:send_request_defer():388] handle sender defer: 4 | |
| 2022-04-04 18:12:45,981 INFO SenderThread:1244997 [sender.py:transition_state():392] send defer: 5 | |
| 2022-04-04 18:12:45,981 DEBUG HandlerThread:1244997 [handler.py:handle_request():141] handle_request: defer | |
| 2022-04-04 18:12:45,981 INFO HandlerThread:1244997 [handler.py:handle_request_defer():164] handle defer: 5 | |
| 2022-04-04 18:12:45,982 DEBUG SenderThread:1244997 [sender.py:send_request():249] send_request: defer | |
| 2022-04-04 18:12:45,982 INFO SenderThread:1244997 [sender.py:send_request_defer():388] handle sender defer: 5 | |
| 2022-04-04 18:12:46,039 INFO SenderThread:1244997 [sender.py:transition_state():392] send defer: 6 | |
| 2022-04-04 18:12:46,039 DEBUG HandlerThread:1244997 [handler.py:handle_request():141] handle_request: defer | |
| 2022-04-04 18:12:46,039 INFO HandlerThread:1244997 [handler.py:handle_request_defer():164] handle defer: 6 | |
| 2022-04-04 18:12:46,040 DEBUG SenderThread:1244997 [sender.py:send_request():249] send_request: defer | |
| 2022-04-04 18:12:46,040 INFO SenderThread:1244997 [sender.py:send_request_defer():388] handle sender defer: 6 | |
| 2022-04-04 18:12:46,040 INFO SenderThread:1244997 [dir_watcher.py:finish():283] shutting down directory watcher | |
| 2022-04-04 18:12:46,075 DEBUG HandlerThread:1244997 [handler.py:handle_request():141] handle_request: poll_exit | |
| 2022-04-04 18:12:46,709 INFO Thread-7 :1244997 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/sanchitgandhi/flax-wav2vec2-2-bart-large-cnn/wandb/run-20220404_181219-4s004g1k/files/config.yaml | |
| 2022-04-04 18:12:46,710 INFO SenderThread:1244997 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/sanchitgandhi/flax-wav2vec2-2-bart-large-cnn/wandb/run-20220404_181219-4s004g1k/files/wandb-summary.json | |
| 2022-04-04 18:12:46,710 INFO SenderThread:1244997 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/sanchitgandhi/flax-wav2vec2-2-bart-large-cnn/wandb/run-20220404_181219-4s004g1k/files/output.log | |
| 2022-04-04 18:12:46,710 INFO SenderThread:1244997 [dir_watcher.py:finish():313] scan: /home/sanchitgandhi/flax-wav2vec2-2-bart-large-cnn/wandb/run-20220404_181219-4s004g1k/files | |
| 2022-04-04 18:12:46,710 INFO SenderThread:1244997 [dir_watcher.py:finish():327] scan save: /home/sanchitgandhi/flax-wav2vec2-2-bart-large-cnn/wandb/run-20220404_181219-4s004g1k/files/requirements.txt requirements.txt | |
| 2022-04-04 18:12:46,710 INFO SenderThread:1244997 [dir_watcher.py:finish():327] scan save: /home/sanchitgandhi/flax-wav2vec2-2-bart-large-cnn/wandb/run-20220404_181219-4s004g1k/files/output.log output.log | |
| 2022-04-04 18:12:46,710 INFO SenderThread:1244997 [dir_watcher.py:finish():327] scan save: /home/sanchitgandhi/flax-wav2vec2-2-bart-large-cnn/wandb/run-20220404_181219-4s004g1k/files/wandb-metadata.json wandb-metadata.json | |
| 2022-04-04 18:12:46,710 INFO SenderThread:1244997 [dir_watcher.py:finish():327] scan save: /home/sanchitgandhi/flax-wav2vec2-2-bart-large-cnn/wandb/run-20220404_181219-4s004g1k/files/config.yaml config.yaml | |
| 2022-04-04 18:12:46,711 INFO SenderThread:1244997 [dir_watcher.py:finish():327] scan save: /home/sanchitgandhi/flax-wav2vec2-2-bart-large-cnn/wandb/run-20220404_181219-4s004g1k/files/wandb-summary.json wandb-summary.json | |
| 2022-04-04 18:12:46,715 INFO SenderThread:1244997 [sender.py:transition_state():392] send defer: 7 | |
| 2022-04-04 18:12:46,716 DEBUG SenderThread:1244997 [sender.py:send_request():249] send_request: poll_exit | |
| 2022-04-04 18:12:46,719 DEBUG HandlerThread:1244997 [handler.py:handle_request():141] handle_request: defer | |
| 2022-04-04 18:12:46,719 INFO HandlerThread:1244997 [handler.py:handle_request_defer():164] handle defer: 7 | |
| 2022-04-04 18:12:46,719 DEBUG SenderThread:1244997 [sender.py:send_request():249] send_request: defer | |
| 2022-04-04 18:12:46,719 INFO SenderThread:1244997 [sender.py:send_request_defer():388] handle sender defer: 7 | |
| 2022-04-04 18:12:46,720 INFO SenderThread:1244997 [file_pusher.py:finish():145] shutting down file pusher | |
| 2022-04-04 18:12:46,824 DEBUG HandlerThread:1244997 [handler.py:handle_request():141] handle_request: poll_exit | |
| 2022-04-04 18:12:46,824 DEBUG SenderThread:1244997 [sender.py:send_request():249] send_request: poll_exit | |
| 2022-04-04 18:12:46,926 DEBUG HandlerThread:1244997 [handler.py:handle_request():141] handle_request: poll_exit | |
| 2022-04-04 18:12:46,926 DEBUG SenderThread:1244997 [sender.py:send_request():249] send_request: poll_exit | |
| 2022-04-04 18:12:47,022 INFO Thread-12 :1244997 [upload_job.py:push():137] Uploaded file /home/sanchitgandhi/flax-wav2vec2-2-bart-large-cnn/wandb/run-20220404_181219-4s004g1k/files/requirements.txt | |
| 2022-04-04 18:12:47,028 DEBUG HandlerThread:1244997 [handler.py:handle_request():141] handle_request: poll_exit | |
| 2022-04-04 18:12:47,028 DEBUG SenderThread:1244997 [sender.py:send_request():249] send_request: poll_exit | |
| 2022-04-04 18:12:47,044 INFO Thread-13 :1244997 [upload_job.py:push():137] Uploaded file /home/sanchitgandhi/flax-wav2vec2-2-bart-large-cnn/wandb/run-20220404_181219-4s004g1k/files/output.log | |
| 2022-04-04 18:12:47,047 INFO Thread-15 :1244997 [upload_job.py:push():137] Uploaded file /home/sanchitgandhi/flax-wav2vec2-2-bart-large-cnn/wandb/run-20220404_181219-4s004g1k/files/wandb-summary.json | |
| 2022-04-04 18:12:47,130 DEBUG HandlerThread:1244997 [handler.py:handle_request():141] handle_request: poll_exit | |
| 2022-04-04 18:12:47,130 DEBUG SenderThread:1244997 [sender.py:send_request():249] send_request: poll_exit | |
| 2022-04-04 18:12:47,232 DEBUG HandlerThread:1244997 [handler.py:handle_request():141] handle_request: poll_exit | |
| 2022-04-04 18:12:47,234 DEBUG SenderThread:1244997 [sender.py:send_request():249] send_request: poll_exit | |
| 2022-04-04 18:12:47,335 DEBUG HandlerThread:1244997 [handler.py:handle_request():141] handle_request: poll_exit | |
| 2022-04-04 18:12:47,336 DEBUG SenderThread:1244997 [sender.py:send_request():249] send_request: poll_exit | |
| 2022-04-04 18:12:47,437 DEBUG HandlerThread:1244997 [handler.py:handle_request():141] handle_request: poll_exit | |
| 2022-04-04 18:12:47,437 DEBUG SenderThread:1244997 [sender.py:send_request():249] send_request: poll_exit | |
| 2022-04-04 18:12:47,440 INFO Thread-14 :1244997 [upload_job.py:push():137] Uploaded file /home/sanchitgandhi/flax-wav2vec2-2-bart-large-cnn/wandb/run-20220404_181219-4s004g1k/files/config.yaml | |
| 2022-04-04 18:12:47,539 DEBUG HandlerThread:1244997 [handler.py:handle_request():141] handle_request: poll_exit | |
| 2022-04-04 18:12:47,539 DEBUG SenderThread:1244997 [sender.py:send_request():249] send_request: poll_exit | |
| 2022-04-04 18:12:47,641 DEBUG HandlerThread:1244997 [handler.py:handle_request():141] handle_request: poll_exit | |
| 2022-04-04 18:12:47,641 INFO Thread-6 :1244997 [sender.py:transition_state():392] send defer: 8 | |
| 2022-04-04 18:12:47,641 DEBUG SenderThread:1244997 [sender.py:send_request():249] send_request: poll_exit | |
| 2022-04-04 18:12:47,642 DEBUG HandlerThread:1244997 [handler.py:handle_request():141] handle_request: defer | |
| 2022-04-04 18:12:47,642 INFO HandlerThread:1244997 [handler.py:handle_request_defer():164] handle defer: 8 | |
| 2022-04-04 18:12:47,642 DEBUG SenderThread:1244997 [sender.py:send_request():249] send_request: defer | |
| 2022-04-04 18:12:47,642 INFO SenderThread:1244997 [sender.py:send_request_defer():388] handle sender defer: 8 | |
| 2022-04-04 18:12:47,721 INFO SenderThread:1244997 [sender.py:transition_state():392] send defer: 9 | |
| 2022-04-04 18:12:47,722 DEBUG HandlerThread:1244997 [handler.py:handle_request():141] handle_request: defer | |
| 2022-04-04 18:12:47,722 INFO HandlerThread:1244997 [handler.py:handle_request_defer():164] handle defer: 9 | |
| 2022-04-04 18:12:47,722 DEBUG SenderThread:1244997 [sender.py:send_request():249] send_request: defer | |
| 2022-04-04 18:12:47,722 INFO SenderThread:1244997 [sender.py:send_request_defer():388] handle sender defer: 9 | |
| 2022-04-04 18:12:47,722 INFO SenderThread:1244997 [sender.py:transition_state():392] send defer: 10 | |
| 2022-04-04 18:12:47,723 DEBUG HandlerThread:1244997 [handler.py:handle_request():141] handle_request: defer | |
| 2022-04-04 18:12:47,723 INFO HandlerThread:1244997 [handler.py:handle_request_defer():164] handle defer: 10 | |
| 2022-04-04 18:12:47,723 DEBUG SenderThread:1244997 [sender.py:send():235] send: final | |
| 2022-04-04 18:12:47,723 DEBUG SenderThread:1244997 [sender.py:send():235] send: footer | |
| 2022-04-04 18:12:47,723 DEBUG SenderThread:1244997 [sender.py:send_request():249] send_request: defer | |
| 2022-04-04 18:12:47,723 INFO SenderThread:1244997 [sender.py:send_request_defer():388] handle sender defer: 10 | |
| 2022-04-04 18:12:47,743 DEBUG HandlerThread:1244997 [handler.py:handle_request():141] handle_request: poll_exit | |
| 2022-04-04 18:12:47,743 DEBUG SenderThread:1244997 [sender.py:send_request():249] send_request: poll_exit | |
| 2022-04-04 18:12:47,743 INFO SenderThread:1244997 [file_pusher.py:join():150] waiting for file pusher | |
| 2022-04-04 18:12:47,962 DEBUG HandlerThread:1244997 [handler.py:handle_request():141] handle_request: sampled_history | |
| 2022-04-04 18:12:47,963 DEBUG HandlerThread:1244997 [handler.py:handle_request():141] handle_request: get_summary | |
| 2022-04-04 18:12:47,964 DEBUG HandlerThread:1244997 [handler.py:handle_request():141] handle_request: shutdown | |
| 2022-04-04 18:12:47,964 INFO HandlerThread:1244997 [handler.py:finish():778] shutting down handler | |
| 2022-04-04 18:12:48,723 INFO WriterThread:1244997 [datastore.py:close():281] close: /home/sanchitgandhi/flax-wav2vec2-2-bart-large-cnn/wandb/run-20220404_181219-4s004g1k/run-4s004g1k.wandb | |
| 2022-04-04 18:12:48,861 INFO SenderThread:1244997 [sender.py:finish():1078] shutting down sender | |
| 2022-04-04 18:12:48,861 INFO SenderThread:1244997 [file_pusher.py:finish():145] shutting down file pusher | |
| 2022-04-04 18:12:48,861 INFO SenderThread:1244997 [file_pusher.py:join():150] waiting for file pusher | |
| 2022-04-04 18:12:48,863 INFO MainThread:1244997 [internal.py:handle_exit():82] Internal process exited | |