2022-04-04 18:10:28,689 INFO    MainThread:1241646 [internal.py:wandb_internal():92] W&B internal server running at pid: 1241646, started at: 2022-04-04 18:10:28.689690
2022-04-04 18:10:28,692 INFO    WriterThread:1241646 [datastore.py:open_for_write():77] open: /home/sanchitgandhi/flax-wav2vec2-2-bart-large-cnn/wandb/run-20220404_181027-mgg9caus/run-mgg9caus.wandb
2022-04-04 18:10:28,693 DEBUG   SenderThread:1241646 [sender.py:send():235] send: header
2022-04-04 18:10:28,693 DEBUG   SenderThread:1241646 [sender.py:send():235] send: run
2022-04-04 18:10:28,789 INFO    SenderThread:1241646 [dir_watcher.py:__init__():169] watching files in: /home/sanchitgandhi/flax-wav2vec2-2-bart-large-cnn/wandb/run-20220404_181027-mgg9caus/files
2022-04-04 18:10:28,789 INFO    SenderThread:1241646 [sender.py:_start_run_threads():812] run started: mgg9caus with start time 1649095827
2022-04-04 18:10:28,789 DEBUG   SenderThread:1241646 [sender.py:send():235] send: summary
2022-04-04 18:10:28,789 INFO    SenderThread:1241646 [sender.py:_save_file():947] saving file wandb-summary.json with policy end
2022-04-04 18:10:28,793 DEBUG   HandlerThread:1241646 [handler.py:handle_request():141] handle_request: check_version
2022-04-04 18:10:28,794 DEBUG   SenderThread:1241646 [sender.py:send_request():249] send_request: check_version
2022-04-04 18:10:28,854 DEBUG   HandlerThread:1241646 [handler.py:handle_request():141] handle_request: run_start
2022-04-04 18:10:29,791 INFO    Thread-7  :1241646 [dir_watcher.py:_on_file_created():217] file/dir created: /home/sanchitgandhi/flax-wav2vec2-2-bart-large-cnn/wandb/run-20220404_181027-mgg9caus/files/wandb-summary.json
2022-04-04 18:10:31,051 DEBUG   HandlerThread:1241646 [meta.py:__init__():37] meta init
2022-04-04 18:10:31,051 DEBUG   HandlerThread:1241646 [meta.py:__init__():51] meta init done
2022-04-04 18:10:31,051 DEBUG   HandlerThread:1241646 [meta.py:probe():211] probe
2022-04-04 18:10:31,052 DEBUG   HandlerThread:1241646 [meta.py:_setup_git():201] setup git
2022-04-04 18:10:31,084 DEBUG   HandlerThread:1241646 [meta.py:_setup_git():208] setup git done
2022-04-04 18:10:31,084 DEBUG   HandlerThread:1241646 [meta.py:_save_pip():55] save pip
2022-04-04 18:10:31,084 DEBUG   HandlerThread:1241646 [meta.py:_save_pip():69] save pip done
2022-04-04 18:10:31,084 DEBUG   HandlerThread:1241646 [meta.py:probe():249] probe done
2022-04-04 18:10:31,088 DEBUG   SenderThread:1241646 [sender.py:send():235] send: files
2022-04-04 18:10:31,088 INFO    SenderThread:1241646 [sender.py:_save_file():947] saving file wandb-metadata.json with policy now
2022-04-04 18:10:31,096 DEBUG   HandlerThread:1241646 [handler.py:handle_request():141] handle_request: stop_status
2022-04-04 18:10:31,096 DEBUG   SenderThread:1241646 [sender.py:send_request():249] send_request: stop_status
2022-04-04 18:10:31,139 DEBUG   SenderThread:1241646 [sender.py:send():235] send: telemetry
2022-04-04 18:10:31,335 INFO    Thread-11 :1241646 [upload_job.py:push():137] Uploaded file /tmp/tmpxnk1ubkiwandb/3n64l3r4-wandb-metadata.json
2022-04-04 18:10:31,791 INFO    Thread-7  :1241646 [dir_watcher.py:_on_file_created():217] file/dir created: /home/sanchitgandhi/flax-wav2vec2-2-bart-large-cnn/wandb/run-20220404_181027-mgg9caus/files/wandb-metadata.json
2022-04-04 18:10:31,792 INFO    Thread-7  :1241646 [dir_watcher.py:_on_file_created():217] file/dir created: /home/sanchitgandhi/flax-wav2vec2-2-bart-large-cnn/wandb/run-20220404_181027-mgg9caus/files/output.log
2022-04-04 18:10:31,792 INFO    Thread-7  :1241646 [dir_watcher.py:_on_file_created():217] file/dir created: /home/sanchitgandhi/flax-wav2vec2-2-bart-large-cnn/wandb/run-20220404_181027-mgg9caus/files/requirements.txt
2022-04-04 18:10:33,792 INFO    Thread-7  :1241646 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/sanchitgandhi/flax-wav2vec2-2-bart-large-cnn/wandb/run-20220404_181027-mgg9caus/files/output.log
2022-04-04 18:10:46,142 DEBUG   HandlerThread:1241646 [handler.py:handle_request():141] handle_request: stop_status
2022-04-04 18:10:46,143 DEBUG   SenderThread:1241646 [sender.py:send_request():249] send_request: stop_status
2022-04-04 18:10:47,800 INFO    Thread-7  :1241646 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/sanchitgandhi/flax-wav2vec2-2-bart-large-cnn/wandb/run-20220404_181027-mgg9caus/files/output.log
2022-04-04 18:10:49,801 INFO    Thread-7  :1241646 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/sanchitgandhi/flax-wav2vec2-2-bart-large-cnn/wandb/run-20220404_181027-mgg9caus/files/output.log
2022-04-04 18:10:50,725 DEBUG   HandlerThread:1241646 [handler.py:handle_request():141] handle_request: partial_history
2022-04-04 18:10:50,725 DEBUG   SenderThread:1241646 [sender.py:send():235] send: telemetry
2022-04-04 18:10:51,801 INFO    Thread-7  :1241646 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/sanchitgandhi/flax-wav2vec2-2-bart-large-cnn/wandb/run-20220404_181027-mgg9caus/files/output.log
2022-04-04 18:10:53,234 DEBUG   HandlerThread:1241646 [handler.py:handle_request():141] handle_request: poll_exit
2022-04-04 18:10:53,234 DEBUG   SenderThread:1241646 [sender.py:send():235] send: exit
2022-04-04 18:10:53,235 INFO    SenderThread:1241646 [sender.py:send_exit():371] handling exit code: 1
2022-04-04 18:10:53,235 INFO    SenderThread:1241646 [sender.py:send_exit():373] handling runtime: 24
2022-04-04 18:10:53,237 INFO    SenderThread:1241646 [sender.py:_save_file():947] saving file wandb-summary.json with policy end
2022-04-04 18:10:53,237 INFO    SenderThread:1241646 [sender.py:send_exit():379] send defer
2022-04-04 18:10:53,237 DEBUG   SenderThread:1241646 [sender.py:send_request():249] send_request: poll_exit
2022-04-04 18:10:53,238 DEBUG   HandlerThread:1241646 [handler.py:handle_request():141] handle_request: defer
2022-04-04 18:10:53,238 INFO    HandlerThread:1241646 [handler.py:handle_request_defer():164] handle defer: 0
2022-04-04 18:10:53,238 DEBUG   SenderThread:1241646 [sender.py:send_request():249] send_request: defer
2022-04-04 18:10:53,238 INFO    SenderThread:1241646 [sender.py:send_request_defer():388] handle sender defer: 0
2022-04-04 18:10:53,238 INFO    SenderThread:1241646 [sender.py:transition_state():392] send defer: 1
2022-04-04 18:10:53,238 DEBUG   HandlerThread:1241646 [handler.py:handle_request():141] handle_request: defer
2022-04-04 18:10:53,238 INFO    HandlerThread:1241646 [handler.py:handle_request_defer():164] handle defer: 1
2022-04-04 18:10:53,321 DEBUG   SenderThread:1241646 [sender.py:send_request():249] send_request: defer
2022-04-04 18:10:53,321 INFO    SenderThread:1241646 [sender.py:send_request_defer():388] handle sender defer: 1
2022-04-04 18:10:53,321 INFO    SenderThread:1241646 [sender.py:transition_state():392] send defer: 2
2022-04-04 18:10:53,321 DEBUG   SenderThread:1241646 [sender.py:send():235] send: stats
2022-04-04 18:10:53,322 DEBUG   HandlerThread:1241646 [handler.py:handle_request():141] handle_request: defer
2022-04-04 18:10:53,322 INFO    HandlerThread:1241646 [handler.py:handle_request_defer():164] handle defer: 2
2022-04-04 18:10:53,322 DEBUG   SenderThread:1241646 [sender.py:send_request():249] send_request: defer
2022-04-04 18:10:53,322 INFO    SenderThread:1241646 [sender.py:send_request_defer():388] handle sender defer: 2
2022-04-04 18:10:53,322 INFO    SenderThread:1241646 [sender.py:transition_state():392] send defer: 3
2022-04-04 18:10:53,322 DEBUG   HandlerThread:1241646 [handler.py:handle_request():141] handle_request: defer
2022-04-04 18:10:53,322 INFO    HandlerThread:1241646 [handler.py:handle_request_defer():164] handle defer: 3
2022-04-04 18:10:53,322 DEBUG   SenderThread:1241646 [sender.py:send_request():249] send_request: defer
2022-04-04 18:10:53,322 INFO    SenderThread:1241646 [sender.py:send_request_defer():388] handle sender defer: 3
2022-04-04 18:10:53,322 INFO    SenderThread:1241646 [sender.py:transition_state():392] send defer: 4
2022-04-04 18:10:53,323 DEBUG   HandlerThread:1241646 [handler.py:handle_request():141] handle_request: defer
2022-04-04 18:10:53,323 INFO    HandlerThread:1241646 [handler.py:handle_request_defer():164] handle defer: 4
2022-04-04 18:10:53,323 DEBUG   SenderThread:1241646 [sender.py:send():235] send: summary
2022-04-04 18:10:53,323 INFO    SenderThread:1241646 [sender.py:_save_file():947] saving file wandb-summary.json with policy end
2022-04-04 18:10:53,324 DEBUG   SenderThread:1241646 [sender.py:send_request():249] send_request: defer
2022-04-04 18:10:53,324 INFO    SenderThread:1241646 [sender.py:send_request_defer():388] handle sender defer: 4
2022-04-04 18:10:53,324 INFO    SenderThread:1241646 [sender.py:transition_state():392] send defer: 5
2022-04-04 18:10:53,324 DEBUG   HandlerThread:1241646 [handler.py:handle_request():141] handle_request: defer
2022-04-04 18:10:53,324 INFO    HandlerThread:1241646 [handler.py:handle_request_defer():164] handle defer: 5
2022-04-04 18:10:53,324 DEBUG   SenderThread:1241646 [sender.py:send_request():249] send_request: defer
2022-04-04 18:10:53,324 INFO    SenderThread:1241646 [sender.py:send_request_defer():388] handle sender defer: 5
2022-04-04 18:10:53,339 DEBUG   HandlerThread:1241646 [handler.py:handle_request():141] handle_request: poll_exit
2022-04-04 18:10:53,392 INFO    SenderThread:1241646 [sender.py:transition_state():392] send defer: 6
2022-04-04 18:10:53,392 DEBUG   SenderThread:1241646 [sender.py:send_request():249] send_request: poll_exit
2022-04-04 18:10:53,393 DEBUG   HandlerThread:1241646 [handler.py:handle_request():141] handle_request: defer
2022-04-04 18:10:53,393 INFO    HandlerThread:1241646 [handler.py:handle_request_defer():164] handle defer: 6
2022-04-04 18:10:53,393 DEBUG   SenderThread:1241646 [sender.py:send_request():249] send_request: defer
2022-04-04 18:10:53,393 INFO    SenderThread:1241646 [sender.py:send_request_defer():388] handle sender defer: 6
2022-04-04 18:10:53,393 INFO    SenderThread:1241646 [dir_watcher.py:finish():283] shutting down directory watcher
2022-04-04 18:10:53,494 DEBUG   HandlerThread:1241646 [handler.py:handle_request():141] handle_request: poll_exit
2022-04-04 18:10:53,802 INFO    Thread-7  :1241646 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/sanchitgandhi/flax-wav2vec2-2-bart-large-cnn/wandb/run-20220404_181027-mgg9caus/files/output.log
2022-04-04 18:10:53,803 INFO    SenderThread:1241646 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/sanchitgandhi/flax-wav2vec2-2-bart-large-cnn/wandb/run-20220404_181027-mgg9caus/files/config.yaml
2022-04-04 18:10:53,803 INFO    SenderThread:1241646 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/sanchitgandhi/flax-wav2vec2-2-bart-large-cnn/wandb/run-20220404_181027-mgg9caus/files/wandb-summary.json
2022-04-04 18:10:53,804 INFO    SenderThread:1241646 [dir_watcher.py:finish():313] scan: /home/sanchitgandhi/flax-wav2vec2-2-bart-large-cnn/wandb/run-20220404_181027-mgg9caus/files
2022-04-04 18:10:53,804 INFO    SenderThread:1241646 [dir_watcher.py:finish():327] scan save: /home/sanchitgandhi/flax-wav2vec2-2-bart-large-cnn/wandb/run-20220404_181027-mgg9caus/files/requirements.txt requirements.txt
2022-04-04 18:10:53,804 INFO    SenderThread:1241646 [dir_watcher.py:finish():327] scan save: /home/sanchitgandhi/flax-wav2vec2-2-bart-large-cnn/wandb/run-20220404_181027-mgg9caus/files/output.log output.log
2022-04-04 18:10:53,804 INFO    SenderThread:1241646 [dir_watcher.py:finish():327] scan save: /home/sanchitgandhi/flax-wav2vec2-2-bart-large-cnn/wandb/run-20220404_181027-mgg9caus/files/wandb-metadata.json wandb-metadata.json
2022-04-04 18:10:53,804 INFO    SenderThread:1241646 [dir_watcher.py:finish():327] scan save: /home/sanchitgandhi/flax-wav2vec2-2-bart-large-cnn/wandb/run-20220404_181027-mgg9caus/files/config.yaml config.yaml
2022-04-04 18:10:53,804 INFO    SenderThread:1241646 [dir_watcher.py:finish():327] scan save: /home/sanchitgandhi/flax-wav2vec2-2-bart-large-cnn/wandb/run-20220404_181027-mgg9caus/files/wandb-summary.json wandb-summary.json
2022-04-04 18:10:53,805 INFO    SenderThread:1241646 [sender.py:transition_state():392] send defer: 7
2022-04-04 18:10:53,805 DEBUG   SenderThread:1241646 [sender.py:send_request():249] send_request: poll_exit
2022-04-04 18:10:53,808 DEBUG   HandlerThread:1241646 [handler.py:handle_request():141] handle_request: defer
2022-04-04 18:10:53,808 INFO    HandlerThread:1241646 [handler.py:handle_request_defer():164] handle defer: 7
2022-04-04 18:10:53,808 DEBUG   SenderThread:1241646 [sender.py:send_request():249] send_request: defer
2022-04-04 18:10:53,808 INFO    SenderThread:1241646 [sender.py:send_request_defer():388] handle sender defer: 7
2022-04-04 18:10:53,808 INFO    SenderThread:1241646 [file_pusher.py:finish():145] shutting down file pusher
2022-04-04 18:10:53,907 DEBUG   HandlerThread:1241646 [handler.py:handle_request():141] handle_request: poll_exit
2022-04-04 18:10:53,907 DEBUG   SenderThread:1241646 [sender.py:send_request():249] send_request: poll_exit
2022-04-04 18:10:54,008 DEBUG   HandlerThread:1241646 [handler.py:handle_request():141] handle_request: poll_exit
2022-04-04 18:10:54,009 DEBUG   SenderThread:1241646 [sender.py:send_request():249] send_request: poll_exit
2022-04-04 18:10:54,056 INFO    Thread-13 :1241646 [upload_job.py:push():137] Uploaded file /home/sanchitgandhi/flax-wav2vec2-2-bart-large-cnn/wandb/run-20220404_181027-mgg9caus/files/output.log
2022-04-04 18:10:54,061 INFO    Thread-15 :1241646 [upload_job.py:push():137] Uploaded file /home/sanchitgandhi/flax-wav2vec2-2-bart-large-cnn/wandb/run-20220404_181027-mgg9caus/files/wandb-summary.json
2022-04-04 18:10:54,082 INFO    Thread-14 :1241646 [upload_job.py:push():137] Uploaded file /home/sanchitgandhi/flax-wav2vec2-2-bart-large-cnn/wandb/run-20220404_181027-mgg9caus/files/config.yaml
2022-04-04 18:10:54,102 INFO    Thread-12 :1241646 [upload_job.py:push():137] Uploaded file /home/sanchitgandhi/flax-wav2vec2-2-bart-large-cnn/wandb/run-20220404_181027-mgg9caus/files/requirements.txt
2022-04-04 18:10:54,110 DEBUG   HandlerThread:1241646 [handler.py:handle_request():141] handle_request: poll_exit
2022-04-04 18:10:54,110 DEBUG   SenderThread:1241646 [sender.py:send_request():249] send_request: poll_exit
2022-04-04 18:10:54,212 DEBUG   HandlerThread:1241646 [handler.py:handle_request():141] handle_request: poll_exit
2022-04-04 18:10:54,212 DEBUG   SenderThread:1241646 [sender.py:send_request():249] send_request: poll_exit
2022-04-04 18:10:54,303 INFO    Thread-6  :1241646 [sender.py:transition_state():392] send defer: 8
2022-04-04 18:10:54,303 DEBUG   HandlerThread:1241646 [handler.py:handle_request():141] handle_request: defer
2022-04-04 18:10:54,303 INFO    HandlerThread:1241646 [handler.py:handle_request_defer():164] handle defer: 8
2022-04-04 18:10:54,304 DEBUG   SenderThread:1241646 [sender.py:send_request():249] send_request: defer
2022-04-04 18:10:54,304 INFO    SenderThread:1241646 [sender.py:send_request_defer():388] handle sender defer: 8
2022-04-04 18:10:54,314 DEBUG   HandlerThread:1241646 [handler.py:handle_request():141] handle_request: poll_exit
2022-04-04 18:10:54,430 INFO    SenderThread:1241646 [sender.py:transition_state():392] send defer: 9
2022-04-04 18:10:54,431 DEBUG   SenderThread:1241646 [sender.py:send_request():249] send_request: poll_exit
2022-04-04 18:10:54,431 DEBUG   HandlerThread:1241646 [handler.py:handle_request():141] handle_request: defer
2022-04-04 18:10:54,431 INFO    HandlerThread:1241646 [handler.py:handle_request_defer():164] handle defer: 9
2022-04-04 18:10:54,431 DEBUG   SenderThread:1241646 [sender.py:send_request():249] send_request: defer
2022-04-04 18:10:54,431 INFO    SenderThread:1241646 [sender.py:send_request_defer():388] handle sender defer: 9
2022-04-04 18:10:54,431 INFO    SenderThread:1241646 [sender.py:transition_state():392] send defer: 10
2022-04-04 18:10:54,432 DEBUG   SenderThread:1241646 [sender.py:send():235] send: final
2022-04-04 18:10:54,432 DEBUG   SenderThread:1241646 [sender.py:send():235] send: footer
2022-04-04 18:10:54,432 DEBUG   HandlerThread:1241646 [handler.py:handle_request():141] handle_request: defer
2022-04-04 18:10:54,432 INFO    HandlerThread:1241646 [handler.py:handle_request_defer():164] handle defer: 10
2022-04-04 18:10:54,432 DEBUG   SenderThread:1241646 [sender.py:send_request():249] send_request: defer
2022-04-04 18:10:54,432 INFO    SenderThread:1241646 [sender.py:send_request_defer():388] handle sender defer: 10
2022-04-04 18:10:54,532 DEBUG   HandlerThread:1241646 [handler.py:handle_request():141] handle_request: poll_exit
2022-04-04 18:10:54,532 DEBUG   SenderThread:1241646 [sender.py:send_request():249] send_request: poll_exit
2022-04-04 18:10:54,533 INFO    SenderThread:1241646 [file_pusher.py:join():150] waiting for file pusher
2022-04-04 18:10:54,686 DEBUG   HandlerThread:1241646 [handler.py:handle_request():141] handle_request: sampled_history
2022-04-04 18:10:54,687 DEBUG   HandlerThread:1241646 [handler.py:handle_request():141] handle_request: get_summary
2022-04-04 18:10:54,688 DEBUG   HandlerThread:1241646 [handler.py:handle_request():141] handle_request: shutdown
2022-04-04 18:10:54,688 INFO    HandlerThread:1241646 [handler.py:finish():778] shutting down handler
2022-04-04 18:10:55,432 INFO    WriterThread:1241646 [datastore.py:close():281] close: /home/sanchitgandhi/flax-wav2vec2-2-bart-large-cnn/wandb/run-20220404_181027-mgg9caus/run-mgg9caus.wandb
2022-04-04 18:10:55,585 INFO    SenderThread:1241646 [sender.py:finish():1078] shutting down sender
2022-04-04 18:10:55,585 INFO    SenderThread:1241646 [file_pusher.py:finish():145] shutting down file pusher
2022-04-04 18:10:55,585 INFO    SenderThread:1241646 [file_pusher.py:join():150] waiting for file pusher
2022-04-04 18:10:55,587 INFO    MainThread:1241646 [internal.py:handle_exit():82] Internal process exited