wandb/debug-internal.log

2024-02-01 17:58:50,082 INFO    StreamThr :239784 [internal.py:wandb_internal():86] W&B internal server running at pid: 239784, started at: 2024-02-01 17:58:50.080918
2024-02-01 17:58:50,083 DEBUG   HandlerThread:239784 [handler.py:handle_request():146] handle_request: status
2024-02-01 17:58:50,087 INFO    WriterThread:239784 [datastore.py:open_for_write():85] open: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_175850-i93q0p12/run-i93q0p12.wandb
2024-02-01 17:58:50,088 DEBUG   SenderThread:239784 [sender.py:send():382] send: header
2024-02-01 17:58:50,109 DEBUG   SenderThread:239784 [sender.py:send():382] send: run
2024-02-01 17:58:50,401 INFO    SenderThread:239784 [dir_watcher.py:__init__():211] watching files in: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_175850-i93q0p12/files
2024-02-01 17:58:50,401 INFO    SenderThread:239784 [sender.py:_start_run_threads():1136] run started: i93q0p12 with start time 1706810330.083967
2024-02-01 17:58:50,409 DEBUG   HandlerThread:239784 [handler.py:handle_request():146] handle_request: check_version
2024-02-01 17:58:50,409 DEBUG   SenderThread:239784 [sender.py:send_request():409] send_request: check_version
2024-02-01 17:58:50,463 DEBUG   HandlerThread:239784 [handler.py:handle_request():146] handle_request: run_start
2024-02-01 17:58:50,558 DEBUG   HandlerThread:239784 [system_info.py:__init__():32] System info init
2024-02-01 17:58:50,558 DEBUG   HandlerThread:239784 [system_info.py:__init__():47] System info init done
2024-02-01 17:58:50,558 INFO    HandlerThread:239784 [system_monitor.py:start():194] Starting system monitor
2024-02-01 17:58:50,558 INFO    SystemMonitor:239784 [system_monitor.py:_start():158] Starting system asset monitoring threads
2024-02-01 17:58:50,559 INFO    HandlerThread:239784 [system_monitor.py:probe():214] Collecting system info
2024-02-01 17:58:50,559 INFO    SystemMonitor:239784 [interfaces.py:start():190] Started cpu monitoring
2024-02-01 17:58:50,559 INFO    SystemMonitor:239784 [interfaces.py:start():190] Started disk monitoring
2024-02-01 17:58:50,560 INFO    SystemMonitor:239784 [interfaces.py:start():190] Started gpu monitoring
2024-02-01 17:58:50,560 INFO    SystemMonitor:239784 [interfaces.py:start():190] Started memory monitoring
2024-02-01 17:58:50,561 INFO    SystemMonitor:239784 [interfaces.py:start():190] Started network monitoring
2024-02-01 17:58:50,615 DEBUG   HandlerThread:239784 [system_info.py:probe():196] Probing system
2024-02-01 17:58:50,618 DEBUG   HandlerThread:239784 [system_info.py:_probe_git():181] Probing git
2024-02-01 17:58:50,642 DEBUG   HandlerThread:239784 [system_info.py:_probe_git():189] Probing git done
2024-02-01 17:58:50,642 DEBUG   HandlerThread:239784 [system_info.py:probe():244] Probing system done
2024-02-01 17:58:50,642 DEBUG   HandlerThread:239784 [system_monitor.py:probe():223] {'os': 'Linux-5.15.0-1048-aws-x86_64-with-glibc2.31', 'python': '3.11.5', 'heartbeatAt': '2024-02-01T17:58:50.615903', 'startedAt': '2024-02-01T17:58:50.064651', 'docker': None, 'cuda': None, 'args': ('config_full.yaml',), 'state': 'running', 'program': '/fsx/sanchit/distil-zephyr-1.5b-ssft/run_sft.py', 'codePathLocal': 'run_sft.py', 'codePath': 'run_sft.py', 'git': {'remote': 'https://huggingface.co/sanchit-gandhi/distil-zephyr-1.5b-ssft', 'commit': '79a4ae874a71e67016ded927e7d23351e5c7dab8'}, 'email': None, 'root': '/fsx/sanchit/distil-zephyr-1.5b-ssft', 'host': 'ip-26-0-165-24', 'username': 'sanchit', 'executable': '/fsx/sanchit/miniconda3/envs/venv/bin/python', 'cpu_count': 96, 'cpu_count_logical': 96, 'cpu_freq': {'current': 2728.6971041666666, 'min': 0.0, 'max': 0.0}, 'cpu_freq_per_core': [{'current': 2650.0, 'min': 0.0, 'max': 0.0}, {'current': 2650.0, 'min': 0.0, 'max': 0.0}, {'current': 2650.0, 'min': 0.0, 'max': 0.0}, {'current': 2650.0, 'min': 0.0, 'max': 0.0}, {'current': 2650.0, 'min': 0.0, 'max': 0.0}, {'current': 2650.0, 'min': 0.0, 'max': 0.0}, {'current': 2650.0, 'min': 0.0, 'max': 0.0}, {'current': 3589.018, 'min': 0.0, 'max': 0.0}, {'current': 2650.0, 'min': 0.0, 'max': 0.0}, {'current': 2650.0, 'min': 0.0, 'max': 0.0}, {'current': 3587.597, 'min': 0.0, 'max': 0.0}, {'current': 2650.0, 'min': 0.0, 'max': 0.0}, {'current': 3596.958, 'min': 0.0, 'max': 0.0}, {'current': 2650.0, 'min': 0.0, 'max': 0.0}, {'current': 2650.0, 'min': 0.0, 'max': 0.0}, {'current': 2650.0, 'min': 0.0, 'max': 0.0}, {'current': 3597.83, 'min': 0.0, 'max': 0.0}, {'current': 2650.0, 'min': 0.0, 'max': 0.0}, {'current': 3595.048, 'min': 0.0, 'max': 0.0}, {'current': 2650.0, 'min': 0.0, 'max': 0.0}, {'current': 2650.0, 'min': 0.0, 'max': 0.0}, {'current': 2650.0, 'min': 0.0, 'max': 0.0}, {'current': 2650.0, 'min': 0.0, 'max': 0.0}, {'current': 3591.991, 'min': 0.0, 'max': 0.0}, {'current': 3598.381, 'min': 0.0, 'max': 0.0}, {'current': 3598.099, 'min': 0.0, 'max': 0.0}, {'current': 2650.0, 'min': 0.0, 'max': 0.0}, {'current': 2650.0, 'min': 0.0, 'max': 0.0}, {'current': 2650.0, 'min': 0.0, 'max': 0.0}, {'current': 2650.0, 'min': 0.0, 'max': 0.0}, {'current': 2650.0, 'min': 0.0, 'max': 0.0}, {'current': 2650.0, 'min': 0.0, 'max': 0.0}, {'current': 2650.0, 'min': 0.0, 'max': 0.0}, {'current': 2650.0, 'min': 0.0, 'max': 0.0}, {'current': 2650.0, 'min': 0.0, 'max': 0.0}, {'current': 2650.0, 'min': 0.0, 'max': 0.0}, {'current': 2650.0, 'min': 0.0, 'max': 0.0}, {'current': 2650.0, 'min': 0.0, 'max': 0.0}, {'current': 2650.0, 'min': 0.0, 'max': 0.0}, {'current': 2650.0, 'min': 0.0, 'max': 0.0}, {'current': 2650.0, 'min': 0.0, 'max': 0.0}, {'current': 2650.0, 'min': 0.0, 'max': 0.0}, {'current': 2650.0, 'min': 0.0, 'max': 0.0}, {'current': 2650.0, 'min': 0.0, 'max': 0.0}, {'current': 2650.0, 'min': 0.0, 'max': 0.0}, {'current': 2650.0, 'min': 0.0, 'max': 0.0}, {'current': 2650.0, 'min': 0.0, 'max': 0.0}, {'current': 2650.0, 'min': 0.0, 'max': 0.0}, {'current': 2650.0, 'min': 0.0, 'max': 0.0}, {'current': 2650.0, 'min': 0.0, 'max': 0.0}, {'current': 2650.0, 'min': 0.0, 'max': 0.0}, {'current': 2650.0, 'min': 0.0, 'max': 0.0}, {'current': 2650.0, 'min': 0.0, 'max': 0.0}, {'current': 2650.0, 'min': 0.0, 'max': 0.0}, {'current': 2650.0, 'min': 0.0, 'max': 0.0}, {'current': 2650.0, 'min': 0.0, 'max': 0.0}, {'current': 2650.0, 'min': 0.0, 'max': 0.0}, {'current': 2650.0, 'min': 0.0, 'max': 0.0}, {'current': 2650.0, 'min': 0.0, 'max': 0.0}, {'current': 2650.0, 'min': 0.0, 'max': 0.0}, {'current': 2650.0, 'min': 0.0, 'max': 0.0}, {'current': 2650.0, 'min': 0.0, 'max': 0.0}, {'current': 2650.0, 'min': 0.0, 'max': 0.0}, {'current': 2650.0, 'min': 0.0, 'max': 0.0}, {'current': 2650.0, 'min': 0.0, 'max': 0.0}, {'current': 2650.0, 'min': 0.0, 'max': 0.0}, {'current': 2650.0, 'min': 0.0, 'max': 0.0}, {'current': 2650.0, 'min': 0.0, 'max': 0.0}, {'current': 2650.0, 'min': 0.0, 'max': 0.0}, {'current': 2650.0, 'min': 0.0, 'max': 0.0}, {'current': 2650.0, 'min': 0.0, 'max': 0.0}, {'current': 2650.0, 'min': 0.0, 'max': 0.0}, {'current': 2650.0, 'min': 0.0, 'max': 0.0}, {'current': 2650.0, 'min': 0.0, 'max': 0.0}, {'current': 2650.0, 'min': 0.0, 'max': 0.0}, {'current': 2650.0, 'min': 0.0, 'max': 0.0}, {'current': 2650.0, 'min': 0.0, 'max': 0.0}, {'current': 2650.0, 'min': 0.0, 'max': 0.0}, {'current': 2650.0, 'min': 0.0, 'max': 0.0}, {'current': 2650.0, 'min': 0.0, 'max': 0.0}, {'current': 2650.0, 'min': 0.0, 'max': 0.0}, {'current': 2650.0, 'min': 0.0, 'max': 0.0}, {'current': 2650.0, 'min': 0.0, 'max': 0.0}, {'current': 2650.0, 'min': 0.0, 'max': 0.0}, {'current': 2650.0, 'min': 0.0, 'max': 0.0}, {'current': 2650.0, 'min': 0.0, 'max': 0.0}, {'current': 2650.0, 'min': 0.0, 'max': 0.0}, {'current': 2650.0, 'min': 0.0, 'max': 0.0}, {'current': 2650.0, 'min': 0.0, 'max': 0.0}, {'current': 2650.0, 'min': 0.0, 'max': 0.0}, {'current': 2650.0, 'min': 0.0, 'max': 0.0}, {'current': 2650.0, 'min': 0.0, 'max': 0.0}, {'current': 2650.0, 'min': 0.0, 'max': 0.0}, {'current': 2650.0, 'min': 0.0, 'max': 0.0}, {'current': 2650.0, 'min': 0.0, 'max': 0.0}, {'current': 2650.0, 'min': 0.0, 'max': 0.0}], 'disk': {'/': {'total': 290.7472343444824, 'used': 57.44965744018555}}, 'gpu': 'NVIDIA H100 80GB HBM3', 'gpu_count': 8, 'gpu_devices': [{'name': 'NVIDIA H100 80GB HBM3', 'memory_total': 85520809984}, {'name': 'NVIDIA H100 80GB HBM3', 'memory_total': 85520809984}, {'name': 'NVIDIA H100 80GB HBM3', 'memory_total': 85520809984}, {'name': 'NVIDIA H100 80GB HBM3', 'memory_total': 85520809984}, {'name': 'NVIDIA H100 80GB HBM3', 'memory_total': 85520809984}, {'name': 'NVIDIA H100 80GB HBM3', 'memory_total': 85520809984}, {'name': 'NVIDIA H100 80GB HBM3', 'memory_total': 85520809984}, {'name': 'NVIDIA H100 80GB HBM3', 'memory_total': 85520809984}], 'memory': {'total': 1999.9855346679688}}
2024-02-01 17:58:50,642 INFO    HandlerThread:239784 [system_monitor.py:probe():224] Finished collecting system info
2024-02-01 17:58:50,642 INFO    HandlerThread:239784 [system_monitor.py:probe():227] Publishing system info
2024-02-01 17:58:50,642 DEBUG   HandlerThread:239784 [system_info.py:_save_pip():52] Saving list of pip packages installed into the current environment
2024-02-01 17:58:50,644 DEBUG   HandlerThread:239784 [system_info.py:_save_pip():68] Saving pip packages done
2024-02-01 17:58:50,644 DEBUG   HandlerThread:239784 [system_info.py:_save_conda():75] Saving list of conda packages installed into the current environment
2024-02-01 17:58:51,404 INFO    Thread-12 :239784 [dir_watcher.py:_on_file_created():271] file/dir created: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_175850-i93q0p12/files/requirements.txt
2024-02-01 17:58:51,404 INFO    Thread-12 :239784 [dir_watcher.py:_on_file_created():271] file/dir created: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_175850-i93q0p12/files/conda-environment.yaml
2024-02-01 17:58:55,310 DEBUG   HandlerThread:239784 [system_info.py:_save_conda():87] Saving conda packages done
2024-02-01 17:58:55,312 INFO    HandlerThread:239784 [system_monitor.py:probe():229] Finished publishing system info
2024-02-01 17:58:55,339 DEBUG   SenderThread:239784 [sender.py:send():382] send: files
2024-02-01 17:58:55,339 INFO    SenderThread:239784 [sender.py:_save_file():1392] saving file wandb-metadata.json with policy now
2024-02-01 17:58:55,339 DEBUG   HandlerThread:239784 [handler.py:handle_request():146] handle_request: status_report
2024-02-01 17:58:55,347 DEBUG   HandlerThread:239784 [handler.py:handle_request():146] handle_request: stop_status
2024-02-01 17:58:55,347 DEBUG   SenderThread:239784 [sender.py:send_request():409] send_request: stop_status
2024-02-01 17:58:55,349 DEBUG   HandlerThread:239784 [handler.py:handle_request():146] handle_request: internal_messages
2024-02-01 17:58:55,408 INFO    Thread-12 :239784 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_175850-i93q0p12/files/conda-environment.yaml
2024-02-01 17:58:55,409 INFO    Thread-12 :239784 [dir_watcher.py:_on_file_created():271] file/dir created: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_175850-i93q0p12/files/wandb-metadata.json
2024-02-01 17:58:55,462 DEBUG   SenderThread:239784 [sender.py:send():382] send: telemetry
2024-02-01 17:58:55,463 DEBUG   SenderThread:239784 [sender.py:send():382] send: config
2024-02-01 17:58:55,463 DEBUG   SenderThread:239784 [sender.py:send():382] send: metric
2024-02-01 17:58:55,463 DEBUG   SenderThread:239784 [sender.py:send():382] send: telemetry
2024-02-01 17:58:55,463 DEBUG   SenderThread:239784 [sender.py:send():382] send: metric
2024-02-01 17:58:55,464 WARNING SenderThread:239784 [sender.py:send_metric():1343] Seen metric with glob (shouldn't happen)
2024-02-01 17:58:55,464 DEBUG   SenderThread:239784 [sender.py:send():382] send: telemetry
2024-02-01 17:58:55,631 INFO    wandb-upload_0:239784 [upload_job.py:push():131] Uploaded file /tmp/tmpr1qr0ps0wandb/r53qkhjh-wandb-metadata.json
2024-02-01 17:58:56,411 INFO    Thread-12 :239784 [dir_watcher.py:_on_file_created():271] file/dir created: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_175850-i93q0p12/files/output.log
2024-02-01 17:58:58,413 INFO    Thread-12 :239784 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_175850-i93q0p12/files/output.log
2024-02-01 17:59:00,471 DEBUG   HandlerThread:239784 [handler.py:handle_request():146] handle_request: status_report
2024-02-01 17:59:05,472 DEBUG   HandlerThread:239784 [handler.py:handle_request():146] handle_request: status_report
2024-02-01 17:59:06,418 DEBUG   HandlerThread:239784 [handler.py:handle_request():146] handle_request: partial_history
2024-02-01 17:59:06,420 DEBUG   SenderThread:239784 [sender.py:send():382] send: metric
2024-02-01 17:59:06,420 DEBUG   SenderThread:239784 [sender.py:send():382] send: metric
2024-02-01 17:59:06,421 DEBUG   SenderThread:239784 [sender.py:send():382] send: metric
2024-02-01 17:59:06,421 DEBUG   SenderThread:239784 [sender.py:send():382] send: history
2024-02-01 17:59:06,421 DEBUG   SenderThread:239784 [sender.py:send_request():409] send_request: summary_record
2024-02-01 17:59:06,424 INFO    SenderThread:239784 [sender.py:_save_file():1392] saving file wandb-summary.json with policy end
2024-02-01 17:59:06,424 INFO    Thread-12 :239784 [dir_watcher.py:_on_file_created():271] file/dir created: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_175850-i93q0p12/files/wandb-summary.json
2024-02-01 17:59:08,427 INFO    Thread-12 :239784 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_175850-i93q0p12/files/output.log
2024-02-01 17:59:10,347 DEBUG   HandlerThread:239784 [handler.py:handle_request():146] handle_request: stop_status
2024-02-01 17:59:10,348 DEBUG   HandlerThread:239784 [handler.py:handle_request():146] handle_request: internal_messages
2024-02-01 17:59:10,348 DEBUG   SenderThread:239784 [sender.py:send_request():409] send_request: stop_status
2024-02-01 17:59:10,493 DEBUG   HandlerThread:239784 [handler.py:handle_request():146] handle_request: status_report
2024-02-01 17:59:12,432 INFO    Thread-12 :239784 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_175850-i93q0p12/files/output.log
2024-02-01 17:59:15,576 DEBUG   HandlerThread:239784 [handler.py:handle_request():146] handle_request: status_report
2024-02-01 17:59:16,437 INFO    Thread-12 :239784 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_175850-i93q0p12/files/output.log
2024-02-01 17:59:18,439 INFO    Thread-12 :239784 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_175850-i93q0p12/files/output.log
2024-02-01 17:59:20,788 DEBUG   HandlerThread:239784 [handler.py:handle_request():146] handle_request: status_report
2024-02-01 17:59:21,444 INFO    Thread-12 :239784 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_175850-i93q0p12/files/config.yaml
2024-02-01 17:59:22,446 INFO    Thread-12 :239784 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_175850-i93q0p12/files/output.log
2024-02-01 17:59:25,347 DEBUG   HandlerThread:239784 [handler.py:handle_request():146] handle_request: stop_status
2024-02-01 17:59:25,348 DEBUG   HandlerThread:239784 [handler.py:handle_request():146] handle_request: internal_messages
2024-02-01 17:59:25,348 DEBUG   SenderThread:239784 [sender.py:send_request():409] send_request: stop_status
2024-02-01 17:59:26,451 INFO    Thread-12 :239784 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_175850-i93q0p12/files/output.log
2024-02-01 17:59:26,471 DEBUG   HandlerThread:239784 [handler.py:handle_request():146] handle_request: status_report
2024-02-01 17:59:30,456 INFO    Thread-12 :239784 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_175850-i93q0p12/files/output.log
2024-02-01 17:59:31,603 DEBUG   HandlerThread:239784 [handler.py:handle_request():146] handle_request: status_report
2024-02-01 17:59:34,461 INFO    Thread-12 :239784 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_175850-i93q0p12/files/output.log
2024-02-01 17:59:36,464 INFO    Thread-12 :239784 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_175850-i93q0p12/files/output.log
2024-02-01 17:59:37,209 DEBUG   HandlerThread:239784 [handler.py:handle_request():146] handle_request: status_report
2024-02-01 17:59:38,807 DEBUG   HandlerThread:239784 [handler.py:handle_request():146] handle_request: partial_history
2024-02-01 17:59:38,808 DEBUG   SenderThread:239784 [sender.py:send():382] send: history
2024-02-01 17:59:38,808 DEBUG   SenderThread:239784 [sender.py:send_request():409] send_request: summary_record
2024-02-01 17:59:38,812 INFO    SenderThread:239784 [sender.py:_save_file():1392] saving file wandb-summary.json with policy end
2024-02-01 17:59:39,469 INFO    Thread-12 :239784 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_175850-i93q0p12/files/wandb-summary.json
2024-02-01 17:59:40,347 DEBUG   HandlerThread:239784 [handler.py:handle_request():146] handle_request: stop_status
2024-02-01 17:59:40,348 DEBUG   SenderThread:239784 [sender.py:send_request():409] send_request: stop_status
2024-02-01 17:59:40,348 DEBUG   HandlerThread:239784 [handler.py:handle_request():146] handle_request: internal_messages
2024-02-01 17:59:40,470 INFO    Thread-12 :239784 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_175850-i93q0p12/files/output.log
2024-02-01 17:59:42,419 DEBUG   HandlerThread:239784 [handler.py:handle_request():146] handle_request: status_report
2024-02-01 17:59:44,475 INFO    Thread-12 :239784 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_175850-i93q0p12/files/output.log
2024-02-01 17:59:48,018 DEBUG   HandlerThread:239784 [handler.py:handle_request():146] handle_request: status_report
2024-02-01 17:59:48,480 INFO    Thread-12 :239784 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_175850-i93q0p12/files/output.log
2024-02-01 17:59:50,561 DEBUG   SystemMonitor:239784 [system_monitor.py:_start():172] Starting system metrics aggregation loop
2024-02-01 17:59:50,564 DEBUG   SenderThread:239784 [sender.py:send():382] send: stats
2024-02-01 17:59:52,485 INFO    Thread-12 :239784 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_175850-i93q0p12/files/output.log
2024-02-01 17:59:53,221 DEBUG   HandlerThread:239784 [handler.py:handle_request():146] handle_request: status_report
2024-02-01 17:59:54,488 INFO    Thread-12 :239784 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_175850-i93q0p12/files/output.log
2024-02-01 17:59:55,347 DEBUG   HandlerThread:239784 [handler.py:handle_request():146] handle_request: stop_status
2024-02-01 17:59:55,348 DEBUG   SenderThread:239784 [sender.py:send_request():409] send_request: stop_status
2024-02-01 17:59:55,348 DEBUG   HandlerThread:239784 [handler.py:handle_request():146] handle_request: internal_messages
2024-02-01 17:59:57,492 INFO    Thread-12 :239784 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_175850-i93q0p12/files/output.log
2024-02-01 17:59:58,833 DEBUG   HandlerThread:239784 [handler.py:handle_request():146] handle_request: status_report
2024-02-01 18:00:01,497 INFO    Thread-12 :239784 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_175850-i93q0p12/files/output.log
2024-02-01 18:00:04,040 DEBUG   HandlerThread:239784 [handler.py:handle_request():146] handle_request: status_report
2024-02-01 18:00:05,502 INFO    Thread-12 :239784 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_175850-i93q0p12/files/output.log
2024-02-01 18:00:09,507 INFO    Thread-12 :239784 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_175850-i93q0p12/files/output.log
2024-02-01 18:00:09,647 DEBUG   HandlerThread:239784 [handler.py:handle_request():146] handle_request: status_report
2024-02-01 18:00:10,348 DEBUG   HandlerThread:239784 [handler.py:handle_request():146] handle_request: stop_status
2024-02-01 18:00:10,348 DEBUG   HandlerThread:239784 [handler.py:handle_request():146] handle_request: internal_messages
2024-02-01 18:00:10,348 DEBUG   SenderThread:239784 [sender.py:send_request():409] send_request: stop_status
2024-02-01 18:00:11,509 INFO    Thread-12 :239784 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_175850-i93q0p12/files/output.log
2024-02-01 18:00:14,847 DEBUG   HandlerThread:239784 [handler.py:handle_request():146] handle_request: status_report
2024-02-01 18:00:14,849 DEBUG   HandlerThread:239784 [handler.py:handle_request():146] handle_request: partial_history
2024-02-01 18:00:14,850 DEBUG   SenderThread:239784 [sender.py:send():382] send: history
2024-02-01 18:00:14,851 DEBUG   SenderThread:239784 [sender.py:send_request():409] send_request: summary_record
2024-02-01 18:00:14,853 INFO    SenderThread:239784 [sender.py:_save_file():1392] saving file wandb-summary.json with policy end
2024-02-01 18:00:15,516 INFO    Thread-12 :239784 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_175850-i93q0p12/files/output.log
2024-02-01 18:00:15,516 INFO    Thread-12 :239784 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_175850-i93q0p12/files/wandb-summary.json
2024-02-01 18:00:16,517 INFO    Thread-12 :239784 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_175850-i93q0p12/files/output.log
2024-02-01 18:00:19,521 INFO    Thread-12 :239784 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_175850-i93q0p12/files/output.log
2024-02-01 18:00:20,459 DEBUG   HandlerThread:239784 [handler.py:handle_request():146] handle_request: status_report
2024-02-01 18:00:20,566 DEBUG   SenderThread:239784 [sender.py:send():382] send: stats
2024-02-01 18:00:23,526 INFO    Thread-12 :239784 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_175850-i93q0p12/files/output.log
2024-02-01 18:00:25,348 DEBUG   HandlerThread:239784 [handler.py:handle_request():146] handle_request: stop_status
2024-02-01 18:00:25,348 DEBUG   SenderThread:239784 [sender.py:send_request():409] send_request: stop_status
2024-02-01 18:00:25,348 DEBUG   HandlerThread:239784 [handler.py:handle_request():146] handle_request: internal_messages
2024-02-01 18:00:25,478 DEBUG   HandlerThread:239784 [handler.py:handle_request():146] handle_request: status_report
2024-02-01 18:00:27,531 INFO    Thread-12 :239784 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_175850-i93q0p12/files/output.log
2024-02-01 18:00:29,534 INFO    Thread-12 :239784 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_175850-i93q0p12/files/output.log
2024-02-01 18:00:31,279 DEBUG   HandlerThread:239784 [handler.py:handle_request():146] handle_request: status_report
2024-02-01 18:00:33,539 INFO    Thread-12 :239784 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_175850-i93q0p12/files/output.log
2024-02-01 18:00:36,505 DEBUG   HandlerThread:239784 [handler.py:handle_request():146] handle_request: status_report
2024-02-01 18:00:37,544 INFO    Thread-12 :239784 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_175850-i93q0p12/files/output.log
2024-02-01 18:00:40,348 DEBUG   HandlerThread:239784 [handler.py:handle_request():146] handle_request: stop_status
2024-02-01 18:00:40,348 DEBUG   SenderThread:239784 [sender.py:send_request():409] send_request: stop_status
2024-02-01 18:00:40,348 DEBUG   HandlerThread:239784 [handler.py:handle_request():146] handle_request: internal_messages
2024-02-01 18:00:41,549 INFO    Thread-12 :239784 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_175850-i93q0p12/files/output.log
2024-02-01 18:00:42,498 DEBUG   HandlerThread:239784 [handler.py:handle_request():146] handle_request: status_report
2024-02-01 18:00:45,554 INFO    Thread-12 :239784 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_175850-i93q0p12/files/output.log
2024-02-01 18:00:47,556 INFO    Thread-12 :239784 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_175850-i93q0p12/files/output.log
2024-02-01 18:00:48,306 DEBUG   HandlerThread:239784 [handler.py:handle_request():146] handle_request: status_report
2024-02-01 18:00:50,568 DEBUG   SenderThread:239784 [sender.py:send():382] send: stats
2024-02-01 18:00:50,912 DEBUG   HandlerThread:239784 [handler.py:handle_request():146] handle_request: partial_history
2024-02-01 18:00:50,913 DEBUG   SenderThread:239784 [sender.py:send():382] send: history
2024-02-01 18:00:50,914 DEBUG   SenderThread:239784 [sender.py:send_request():409] send_request: summary_record
2024-02-01 18:00:50,915 INFO    SenderThread:239784 [sender.py:_save_file():1392] saving file wandb-summary.json with policy end
2024-02-01 18:00:51,562 INFO    Thread-12 :239784 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_175850-i93q0p12/files/output.log
2024-02-01 18:00:51,563 INFO    Thread-12 :239784 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_175850-i93q0p12/files/wandb-summary.json
2024-02-01 18:00:52,564 INFO    Thread-12 :239784 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_175850-i93q0p12/files/output.log
2024-02-01 18:00:53,925 DEBUG   HandlerThread:239784 [handler.py:handle_request():146] handle_request: status_report
2024-02-01 18:00:55,348 DEBUG   HandlerThread:239784 [handler.py:handle_request():146] handle_request: stop_status
2024-02-01 18:00:55,348 DEBUG   SenderThread:239784 [sender.py:send_request():409] send_request: stop_status
2024-02-01 18:00:55,349 DEBUG   HandlerThread:239784 [handler.py:handle_request():146] handle_request: internal_messages
2024-02-01 18:00:55,567 INFO    Thread-12 :239784 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_175850-i93q0p12/files/output.log
2024-02-01 18:00:59,109 DEBUG   HandlerThread:239784 [handler.py:handle_request():146] handle_request: status_report
2024-02-01 18:00:59,572 INFO    Thread-12 :239784 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_175850-i93q0p12/files/output.log
2024-02-01 18:01:03,578 INFO    Thread-12 :239784 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_175850-i93q0p12/files/output.log
2024-02-01 18:01:04,714 DEBUG   HandlerThread:239784 [handler.py:handle_request():146] handle_request: status_report
2024-02-01 18:01:05,580 INFO    Thread-12 :239784 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_175850-i93q0p12/files/output.log
2024-02-01 18:01:09,585 INFO    Thread-12 :239784 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_175850-i93q0p12/files/output.log
2024-02-01 18:01:09,913 DEBUG   HandlerThread:239784 [handler.py:handle_request():146] handle_request: status_report
2024-02-01 18:01:10,348 DEBUG   HandlerThread:239784 [handler.py:handle_request():146] handle_request: stop_status
2024-02-01 18:01:10,348 DEBUG   SenderThread:239784 [sender.py:send_request():409] send_request: stop_status
2024-02-01 18:01:10,348 DEBUG   HandlerThread:239784 [handler.py:handle_request():146] handle_request: internal_messages
2024-02-01 18:01:13,590 INFO    Thread-12 :239784 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_175850-i93q0p12/files/output.log
2024-02-01 18:01:15,510 DEBUG   HandlerThread:239784 [handler.py:handle_request():146] handle_request: status_report
2024-02-01 18:01:17,595 INFO    Thread-12 :239784 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_175850-i93q0p12/files/output.log
2024-02-01 18:01:19,526 WARNING FileStreamThread:239784 [file_stream.py:request_with_retry():668] requests_with_retry encountered retryable exception: 502 Server Error: Bad Gateway for url: https://api.wandb.ai/files/sanchit-gandhi/huggingface/i93q0p12/file_stream. func: functools.partial(<bound method Session.post of <requests.sessions.Session object at 0x7fa81dfad490>>, timeout=180), args: ('https://api.wandb.ai/files/sanchit-gandhi/huggingface/i93q0p12/file_stream',), kwargs: {'json': {'files': {'wandb-summary.json': {'offset': 0, 'content': ['{"train/loss": 7.6246, "train/learning_rate": 4.3795620437956196e-05, "train/epoch": 0.07, "train/global_step": 20, "_timestamp": 1706810414.8489923, "_runtime": 84.76502537727356, "_step": 2}']}, 'wandb-history.jsonl': {'offset': 2, 'content': ['{"train/loss": 7.6246, "train/learning_rate": 4.3795620437956196e-05, "train/epoch": 0.07, "train/global_step": 20, "_timestamp": 1706810414.8489923, "_runtime": 84.76502537727356, "_step": 2}']}, 'output.log': {'offset': 5, 'content': ["2024-02-01T18:00:16.385360 {'loss': 7.6246, 'learning_rate': 4.3795620437956196e-05, 'epoch': 0.07}\n"]}}, 'dropped': 0}}
2024-02-01 18:01:20,570 DEBUG   SenderThread:239784 [sender.py:send():382] send: stats
2024-02-01 18:01:20,570 DEBUG   HandlerThread:239784 [handler.py:handle_request():146] handle_request: status_report
2024-02-01 18:01:21,600 INFO    Thread-12 :239784 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_175850-i93q0p12/files/output.log
2024-02-01 18:01:23,603 INFO    Thread-12 :239784 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_175850-i93q0p12/files/output.log
2024-02-01 18:01:25,348 DEBUG   HandlerThread:239784 [handler.py:handle_request():146] handle_request: stop_status
2024-02-01 18:01:25,349 DEBUG   HandlerThread:239784 [handler.py:handle_request():146] handle_request: internal_messages
2024-02-01 18:01:25,349 DEBUG   SenderThread:239784 [sender.py:send_request():409] send_request: stop_status
2024-02-01 18:01:26,448 DEBUG   HandlerThread:239784 [handler.py:handle_request():146] handle_request: status_report
2024-02-01 18:01:26,942 DEBUG   HandlerThread:239784 [handler.py:handle_request():146] handle_request: partial_history
2024-02-01 18:01:26,943 DEBUG   SenderThread:239784 [sender.py:send():382] send: history
2024-02-01 18:01:26,944 DEBUG   SenderThread:239784 [sender.py:send_request():409] send_request: summary_record
2024-02-01 18:01:26,945 INFO    SenderThread:239784 [sender.py:_save_file():1392] saving file wandb-summary.json with policy end
2024-02-01 18:01:27,609 INFO    Thread-12 :239784 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_175850-i93q0p12/files/output.log
2024-02-01 18:01:27,609 INFO    Thread-12 :239784 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_175850-i93q0p12/files/wandb-summary.json
2024-02-01 18:01:28,611 INFO    Thread-12 :239784 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_175850-i93q0p12/files/output.log
2024-02-01 18:01:31,497 DEBUG   HandlerThread:239784 [handler.py:handle_request():146] handle_request: status_report
2024-02-01 18:01:31,615 INFO    Thread-12 :239784 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_175850-i93q0p12/files/output.log
2024-02-01 18:01:35,620 INFO    Thread-12 :239784 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_175850-i93q0p12/files/output.log
2024-02-01 18:01:37,093 DEBUG   HandlerThread:239784 [handler.py:handle_request():146] handle_request: status_report
2024-02-01 18:01:39,625 INFO    Thread-12 :239784 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_175850-i93q0p12/files/output.log
2024-02-01 18:01:40,348 DEBUG   HandlerThread:239784 [handler.py:handle_request():146] handle_request: stop_status
2024-02-01 18:01:40,348 DEBUG   HandlerThread:239784 [handler.py:handle_request():146] handle_request: internal_messages
2024-02-01 18:01:40,349 DEBUG   SenderThread:239784 [sender.py:send_request():409] send_request: stop_status
2024-02-01 18:01:41,628 INFO    Thread-12 :239784 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_175850-i93q0p12/files/output.log
2024-02-01 18:01:42,310 DEBUG   HandlerThread:239784 [handler.py:handle_request():146] handle_request: status_report
2024-02-01 18:01:45,634 INFO    Thread-12 :239784 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_175850-i93q0p12/files/output.log
2024-02-01 18:01:47,915 DEBUG   HandlerThread:239784 [handler.py:handle_request():146] handle_request: status_report
2024-02-01 18:01:49,639 INFO    Thread-12 :239784 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_175850-i93q0p12/files/output.log
2024-02-01 18:01:50,572 DEBUG   SenderThread:239784 [sender.py:send():382] send: stats
2024-02-01 18:01:53,132 DEBUG   HandlerThread:239784 [handler.py:handle_request():146] handle_request: status_report
2024-02-01 18:01:53,644 INFO    Thread-12 :239784 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_175850-i93q0p12/files/output.log
2024-02-01 18:01:55,348 DEBUG   HandlerThread:239784 [handler.py:handle_request():146] handle_request: stop_status
2024-02-01 18:01:55,349 DEBUG   HandlerThread:239784 [handler.py:handle_request():146] handle_request: internal_messages
2024-02-01 18:01:55,349 DEBUG   SenderThread:239784 [sender.py:send_request():409] send_request: stop_status
2024-02-01 18:01:57,650 INFO    Thread-12 :239784 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_175850-i93q0p12/files/output.log
2024-02-01 18:01:58,727 DEBUG   HandlerThread:239784 [handler.py:handle_request():146] handle_request: status_report
2024-02-01 18:01:59,652 INFO    Thread-12 :239784 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_175850-i93q0p12/files/output.log
2024-02-01 18:02:02,945 DEBUG   HandlerThread:239784 [handler.py:handle_request():146] handle_request: partial_history
2024-02-01 18:02:02,947 DEBUG   SenderThread:239784 [sender.py:send():382] send: history
2024-02-01 18:02:02,947 DEBUG   SenderThread:239784 [sender.py:send_request():409] send_request: summary_record
2024-02-01 18:02:02,949 INFO    SenderThread:239784 [sender.py:_save_file():1392] saving file wandb-summary.json with policy end
2024-02-01 18:02:03,659 INFO    Thread-12 :239784 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_175850-i93q0p12/files/output.log
2024-02-01 18:02:03,659 INFO    Thread-12 :239784 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_175850-i93q0p12/files/wandb-summary.json
2024-02-01 18:02:03,950 DEBUG   HandlerThread:239784 [handler.py:handle_request():146] handle_request: status_report
2024-02-01 18:02:04,660 INFO    Thread-12 :239784 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_175850-i93q0p12/files/output.log
2024-02-01 18:02:07,664 INFO    Thread-12 :239784 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_175850-i93q0p12/files/output.log
2024-02-01 18:02:09,549 DEBUG   HandlerThread:239784 [handler.py:handle_request():146] handle_request: status_report
2024-02-01 18:02:10,348 DEBUG   HandlerThread:239784 [handler.py:handle_request():146] handle_request: stop_status
2024-02-01 18:02:10,349 DEBUG   HandlerThread:239784 [handler.py:handle_request():146] handle_request: internal_messages
2024-02-01 18:02:10,349 DEBUG   SenderThread:239784 [sender.py:send_request():409] send_request: stop_status
2024-02-01 18:02:11,669 INFO    Thread-12 :239784 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_175850-i93q0p12/files/output.log
2024-02-01 18:02:14,750 DEBUG   HandlerThread:239784 [handler.py:handle_request():146] handle_request: status_report
2024-02-01 18:02:15,675 INFO    Thread-12 :239784 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_175850-i93q0p12/files/output.log
2024-02-01 18:02:17,677 INFO    Thread-12 :239784 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_175850-i93q0p12/files/output.log
2024-02-01 18:02:20,345 DEBUG   HandlerThread:239784 [handler.py:handle_request():146] handle_request: status_report
2024-02-01 18:02:20,574 DEBUG   SenderThread:239784 [sender.py:send():382] send: stats
2024-02-01 18:02:21,683 INFO    Thread-12 :239784 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_175850-i93q0p12/files/output.log
2024-02-01 18:02:25,348 DEBUG   HandlerThread:239784 [handler.py:handle_request():146] handle_request: stop_status
2024-02-01 18:02:25,349 DEBUG   HandlerThread:239784 [handler.py:handle_request():146] handle_request: internal_messages
2024-02-01 18:02:25,349 DEBUG   SenderThread:239784 [sender.py:send_request():409] send_request: stop_status
2024-02-01 18:02:25,418 DEBUG   HandlerThread:239784 [handler.py:handle_request():146] handle_request: status_report
2024-02-01 18:02:25,688 INFO    Thread-12 :239784 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_175850-i93q0p12/files/output.log
2024-02-01 18:02:29,693 INFO    Thread-12 :239784 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_175850-i93q0p12/files/output.log
2024-02-01 18:02:31,181 DEBUG   HandlerThread:239784 [handler.py:handle_request():146] handle_request: status_report
2024-02-01 18:02:33,698 INFO    Thread-12 :239784 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_175850-i93q0p12/files/output.log
2024-02-01 18:02:35,701 INFO    Thread-12 :239784 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_175850-i93q0p12/files/output.log
2024-02-01 18:02:36,400 DEBUG   HandlerThread:239784 [handler.py:handle_request():146] handle_request: status_report
2024-02-01 18:02:39,023 DEBUG   HandlerThread:239784 [handler.py:handle_request():146] handle_request: partial_history
2024-02-01 18:02:39,024 DEBUG   SenderThread:239784 [sender.py:send():382] send: history
2024-02-01 18:02:39,024 DEBUG   SenderThread:239784 [sender.py:send_request():409] send_request: summary_record
2024-02-01 18:02:39,026 INFO    SenderThread:239784 [sender.py:_save_file():1392] saving file wandb-summary.json with policy end
2024-02-01 18:02:39,707 INFO    Thread-12 :239784 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_175850-i93q0p12/files/output.log
2024-02-01 18:02:39,707 INFO    Thread-12 :239784 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_175850-i93q0p12/files/wandb-summary.json
2024-02-01 18:02:40,348 DEBUG   HandlerThread:239784 [handler.py:handle_request():146] handle_request: stop_status
2024-02-01 18:02:40,349 DEBUG   SenderThread:239784 [sender.py:send_request():409] send_request: stop_status
2024-02-01 18:02:40,349 DEBUG   HandlerThread:239784 [handler.py:handle_request():146] handle_request: internal_messages
2024-02-01 18:02:40,708 INFO    Thread-12 :239784 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_175850-i93q0p12/files/output.log
2024-02-01 18:02:41,400 DEBUG   HandlerThread:239784 [handler.py:handle_request():146] handle_request: status_report
2024-02-01 18:02:43,712 INFO    Thread-12 :239784 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_175850-i93q0p12/files/output.log
2024-02-01 18:02:47,226 DEBUG   HandlerThread:239784 [handler.py:handle_request():146] handle_request: status_report
2024-02-01 18:02:47,718 INFO    Thread-12 :239784 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_175850-i93q0p12/files/output.log
2024-02-01 18:02:50,576 DEBUG   SenderThread:239784 [sender.py:send():382] send: stats
2024-02-01 18:02:51,723 INFO    Thread-12 :239784 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_175850-i93q0p12/files/output.log
2024-02-01 18:02:52,577 DEBUG   HandlerThread:239784 [handler.py:handle_request():146] handle_request: status_report
2024-02-01 18:02:53,726 INFO    Thread-12 :239784 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_175850-i93q0p12/files/output.log
2024-02-01 18:02:55,349 DEBUG   HandlerThread:239784 [handler.py:handle_request():146] handle_request: stop_status
2024-02-01 18:02:55,349 DEBUG   SenderThread:239784 [sender.py:send_request():409] send_request: stop_status
2024-02-01 18:02:55,349 DEBUG   HandlerThread:239784 [handler.py:handle_request():146] handle_request: internal_messages
2024-02-01 18:02:57,731 INFO    Thread-12 :239784 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_175850-i93q0p12/files/output.log
2024-02-01 18:02:58,047 DEBUG   HandlerThread:239784 [handler.py:handle_request():146] handle_request: status_report
2024-02-01 18:03:01,736 INFO    Thread-12 :239784 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_175850-i93q0p12/files/output.log
2024-02-01 18:03:03,651 DEBUG   HandlerThread:239784 [handler.py:handle_request():146] handle_request: status_report
2024-02-01 18:03:05,742 INFO    Thread-12 :239784 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_175850-i93q0p12/files/output.log
2024-02-01 18:03:08,868 DEBUG   HandlerThread:239784 [handler.py:handle_request():146] handle_request: status_report
2024-02-01 18:03:09,748 INFO    Thread-12 :239784 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_175850-i93q0p12/files/output.log
2024-02-01 18:03:10,349 DEBUG   HandlerThread:239784 [handler.py:handle_request():146] handle_request: stop_status
2024-02-01 18:03:10,349 DEBUG   HandlerThread:239784 [handler.py:handle_request():146] handle_request: internal_messages
2024-02-01 18:03:10,349 DEBUG   SenderThread:239784 [sender.py:send_request():409] send_request: stop_status
2024-02-01 18:03:11,750 INFO    Thread-12 :239784 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_175850-i93q0p12/files/output.log
2024-02-01 18:03:14,469 DEBUG   HandlerThread:239784 [handler.py:handle_request():146] handle_request: status_report
2024-02-01 18:03:15,092 DEBUG   HandlerThread:239784 [handler.py:handle_request():146] handle_request: partial_history
2024-02-01 18:03:15,093 DEBUG   SenderThread:239784 [sender.py:send():382] send: history
2024-02-01 18:03:15,094 DEBUG   SenderThread:239784 [sender.py:send_request():409] send_request: summary_record
2024-02-01 18:03:15,096 INFO    SenderThread:239784 [sender.py:_save_file():1392] saving file wandb-summary.json with policy end
2024-02-01 18:03:15,757 INFO    Thread-12 :239784 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_175850-i93q0p12/files/output.log
2024-02-01 18:03:15,757 INFO    Thread-12 :239784 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_175850-i93q0p12/files/wandb-summary.json
2024-02-01 18:03:16,758 INFO    Thread-12 :239784 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_175850-i93q0p12/files/output.log
2024-02-01 18:03:19,706 DEBUG   HandlerThread:239784 [handler.py:handle_request():146] handle_request: status_report
2024-02-01 18:03:19,762 INFO    Thread-12 :239784 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_175850-i93q0p12/files/output.log
2024-02-01 18:03:20,577 DEBUG   SenderThread:239784 [sender.py:send():382] send: stats
2024-02-01 18:03:23,767 INFO    Thread-12 :239784 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_175850-i93q0p12/files/output.log
2024-02-01 18:03:25,309 DEBUG   HandlerThread:239784 [handler.py:handle_request():146] handle_request: status_report
2024-02-01 18:03:25,349 DEBUG   HandlerThread:239784 [handler.py:handle_request():146] handle_request: stop_status
2024-02-01 18:03:25,349 DEBUG   SenderThread:239784 [sender.py:send_request():409] send_request: stop_status
2024-02-01 18:03:25,350 DEBUG   HandlerThread:239784 [handler.py:handle_request():146] handle_request: internal_messages
2024-02-01 18:03:27,773 INFO    Thread-12 :239784 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_175850-i93q0p12/files/output.log
2024-02-01 18:03:30,545 DEBUG   HandlerThread:239784 [handler.py:handle_request():146] handle_request: status_report
2024-02-01 18:03:31,778 INFO    Thread-12 :239784 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_175850-i93q0p12/files/output.log
2024-02-01 18:03:33,781 INFO    Thread-12 :239784 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_175850-i93q0p12/files/output.log
2024-02-01 18:03:36,152 DEBUG   HandlerThread:239784 [handler.py:handle_request():146] handle_request: status_report
2024-02-01 18:03:37,786 INFO    Thread-12 :239784 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_175850-i93q0p12/files/output.log
2024-02-01 18:03:40,349 DEBUG   HandlerThread:239784 [handler.py:handle_request():146] handle_request: stop_status
2024-02-01 18:03:40,349 DEBUG   SenderThread:239784 [sender.py:send_request():409] send_request: stop_status
2024-02-01 18:03:40,351 DEBUG   HandlerThread:239784 [handler.py:handle_request():146] handle_request: internal_messages
2024-02-01 18:03:41,404 DEBUG   HandlerThread:239784 [handler.py:handle_request():146] handle_request: status_report
2024-02-01 18:03:41,791 INFO    Thread-12 :239784 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_175850-i93q0p12/files/output.log
2024-02-01 18:03:45,797 INFO    Thread-12 :239784 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_175850-i93q0p12/files/output.log
2024-02-01 18:03:46,979 DEBUG   HandlerThread:239784 [handler.py:handle_request():146] handle_request: status_report
2024-02-01 18:03:49,802 INFO    Thread-12 :239784 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_175850-i93q0p12/files/output.log
2024-02-01 18:03:50,579 DEBUG   SenderThread:239784 [sender.py:send():382] send: stats
2024-02-01 18:03:51,183 DEBUG   HandlerThread:239784 [handler.py:handle_request():146] handle_request: partial_history
2024-02-01 18:03:51,184 DEBUG   SenderThread:239784 [sender.py:send():382] send: history
2024-02-01 18:03:51,185 DEBUG   SenderThread:239784 [sender.py:send_request():409] send_request: summary_record
2024-02-01 18:03:51,187 INFO    SenderThread:239784 [sender.py:_save_file():1392] saving file wandb-summary.json with policy end
2024-02-01 18:03:51,806 INFO    Thread-12 :239784 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_175850-i93q0p12/files/output.log
2024-02-01 18:03:51,806 INFO    Thread-12 :239784 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_175850-i93q0p12/files/wandb-summary.json
2024-02-01 18:03:52,187 DEBUG   HandlerThread:239784 [handler.py:handle_request():146] handle_request: status_report
2024-02-01 18:03:52,807 INFO    Thread-12 :239784 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_175850-i93q0p12/files/output.log
2024-02-01 18:03:55,349 DEBUG   HandlerThread:239784 [handler.py:handle_request():146] handle_request: stop_status
2024-02-01 18:03:55,349 DEBUG   SenderThread:239784 [sender.py:send_request():409] send_request: stop_status
2024-02-01 18:03:55,350 DEBUG   HandlerThread:239784 [handler.py:handle_request():146] handle_request: internal_messages
2024-02-01 18:03:55,811 INFO    Thread-12 :239784 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_175850-i93q0p12/files/output.log
2024-02-01 18:03:57,472 DEBUG   HandlerThread:239784 [handler.py:handle_request():146] handle_request: status_report
2024-02-01 18:03:59,817 INFO    Thread-12 :239784 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_175850-i93q0p12/files/output.log
2024-02-01 18:04:03,006 DEBUG   HandlerThread:239784 [handler.py:handle_request():146] handle_request: status_report
2024-02-01 18:04:03,822 INFO    Thread-12 :239784 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_175850-i93q0p12/files/output.log
2024-02-01 18:04:07,828 INFO    Thread-12 :239784 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_175850-i93q0p12/files/output.log
2024-02-01 18:04:08,620 DEBUG   HandlerThread:239784 [handler.py:handle_request():146] handle_request: status_report
2024-02-01 18:04:09,830 INFO    Thread-12 :239784 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_175850-i93q0p12/files/output.log
2024-02-01 18:04:10,349 DEBUG   HandlerThread:239784 [handler.py:handle_request():146] handle_request: stop_status
2024-02-01 18:04:10,350 DEBUG   HandlerThread:239784 [handler.py:handle_request():146] handle_request: internal_messages
2024-02-01 18:04:10,350 DEBUG   SenderThread:239784 [sender.py:send_request():409] send_request: stop_status
2024-02-01 18:04:13,836 DEBUG   HandlerThread:239784 [handler.py:handle_request():146] handle_request: status_report
2024-02-01 18:04:13,836 INFO    Thread-12 :239784 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_175850-i93q0p12/files/output.log
2024-02-01 18:04:17,841 INFO    Thread-12 :239784 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_175850-i93q0p12/files/output.log
2024-02-01 18:04:19,433 DEBUG   HandlerThread:239784 [handler.py:handle_request():146] handle_request: status_report
2024-02-01 18:04:20,581 DEBUG   SenderThread:239784 [sender.py:send():382] send: stats
2024-02-01 18:04:21,847 INFO    Thread-12 :239784 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_175850-i93q0p12/files/output.log
2024-02-01 18:04:24,633 DEBUG   HandlerThread:239784 [handler.py:handle_request():146] handle_request: status_report
2024-02-01 18:04:25,349 DEBUG   HandlerThread:239784 [handler.py:handle_request():146] handle_request: stop_status
2024-02-01 18:04:25,349 DEBUG   SenderThread:239784 [sender.py:send_request():409] send_request: stop_status
2024-02-01 18:04:25,351 DEBUG   HandlerThread:239784 [handler.py:handle_request():146] handle_request: internal_messages
2024-02-01 18:04:25,852 INFO    Thread-12 :239784 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_175850-i93q0p12/files/output.log
2024-02-01 18:04:27,274 DEBUG   HandlerThread:239784 [handler.py:handle_request():146] handle_request: partial_history
2024-02-01 18:04:27,275 DEBUG   SenderThread:239784 [sender.py:send():382] send: history
2024-02-01 18:04:27,275 DEBUG   SenderThread:239784 [sender.py:send_request():409] send_request: summary_record
2024-02-01 18:04:27,277 INFO    SenderThread:239784 [sender.py:_save_file():1392] saving file wandb-summary.json with policy end
2024-02-01 18:04:27,855 INFO    Thread-12 :239784 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_175850-i93q0p12/files/output.log
2024-02-01 18:04:27,856 INFO    Thread-12 :239784 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_175850-i93q0p12/files/wandb-summary.json
2024-02-01 18:04:28,857 INFO    Thread-12 :239784 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_175850-i93q0p12/files/output.log
2024-02-01 18:04:30,278 DEBUG   HandlerThread:239784 [handler.py:handle_request():146] handle_request: status_report
2024-02-01 18:04:31,861 INFO    Thread-12 :239784 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_175850-i93q0p12/files/output.log
2024-02-01 18:04:35,451 DEBUG   HandlerThread:239784 [handler.py:handle_request():146] handle_request: status_report
2024-02-01 18:04:35,866 INFO    Thread-12 :239784 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_175850-i93q0p12/files/output.log
2024-02-01 18:04:39,871 INFO    Thread-12 :239784 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_175850-i93q0p12/files/output.log
2024-02-01 18:04:40,349 DEBUG   HandlerThread:239784 [handler.py:handle_request():146] handle_request: stop_status
2024-02-01 18:04:40,350 DEBUG   SenderThread:239784 [sender.py:send_request():409] send_request: stop_status
2024-02-01 18:04:40,351 DEBUG   HandlerThread:239784 [handler.py:handle_request():146] handle_request: internal_messages
2024-02-01 18:04:40,479 DEBUG   HandlerThread:239784 [handler.py:handle_request():146] handle_request: status_report
2024-02-01 18:04:43,877 INFO    Thread-12 :239784 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_175850-i93q0p12/files/output.log
2024-02-01 18:04:45,879 INFO    Thread-12 :239784 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_175850-i93q0p12/files/output.log
2024-02-01 18:04:46,271 DEBUG   HandlerThread:239784 [handler.py:handle_request():146] handle_request: status_report
2024-02-01 18:04:49,884 INFO    Thread-12 :239784 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_175850-i93q0p12/files/output.log
2024-02-01 18:04:50,583 DEBUG   SenderThread:239784 [sender.py:send():382] send: stats
2024-02-01 18:04:51,584 DEBUG   HandlerThread:239784 [handler.py:handle_request():146] handle_request: status_report
2024-02-01 18:04:53,890 INFO    Thread-12 :239784 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_175850-i93q0p12/files/output.log
2024-02-01 18:04:55,349 DEBUG   HandlerThread:239784 [handler.py:handle_request():146] handle_request: stop_status
2024-02-01 18:04:55,350 DEBUG   SenderThread:239784 [sender.py:send_request():409] send_request: stop_status
2024-02-01 18:04:55,351 DEBUG   HandlerThread:239784 [handler.py:handle_request():146] handle_request: internal_messages
2024-02-01 18:04:57,096 DEBUG   HandlerThread:239784 [handler.py:handle_request():146] handle_request: status_report
2024-02-01 18:04:57,895 INFO    Thread-12 :239784 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_175850-i93q0p12/files/output.log
2024-02-01 18:05:01,900 INFO    Thread-12 :239784 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_175850-i93q0p12/files/output.log
2024-02-01 18:05:02,711 DEBUG   HandlerThread:239784 [handler.py:handle_request():146] handle_request: status_report
2024-02-01 18:05:03,319 DEBUG   HandlerThread:239784 [handler.py:handle_request():146] handle_request: partial_history
2024-02-01 18:05:03,320 DEBUG   SenderThread:239784 [sender.py:send():382] send: history
2024-02-01 18:05:03,320 DEBUG   SenderThread:239784 [sender.py:send_request():409] send_request: summary_record
2024-02-01 18:05:03,322 INFO    SenderThread:239784 [sender.py:_save_file():1392] saving file wandb-summary.json with policy end
2024-02-01 18:05:03,904 INFO    Thread-12 :239784 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_175850-i93q0p12/files/output.log
2024-02-01 18:05:03,904 INFO    Thread-12 :239784 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_175850-i93q0p12/files/wandb-summary.json
2024-02-01 18:05:04,905 INFO    Thread-12 :239784 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_175850-i93q0p12/files/output.log
2024-02-01 18:05:07,909 INFO    Thread-12 :239784 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_175850-i93q0p12/files/output.log
2024-02-01 18:05:07,930 DEBUG   HandlerThread:239784 [handler.py:handle_request():146] handle_request: status_report
2024-02-01 18:05:10,349 DEBUG   HandlerThread:239784 [handler.py:handle_request():146] handle_request: stop_status
2024-02-01 18:05:10,350 DEBUG   SenderThread:239784 [sender.py:send_request():409] send_request: stop_status
2024-02-01 18:05:10,350 DEBUG   HandlerThread:239784 [handler.py:handle_request():146] handle_request: internal_messages
2024-02-01 18:05:11,915 INFO    Thread-12 :239784 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_175850-i93q0p12/files/output.log
2024-02-01 18:05:13,541 DEBUG   HandlerThread:239784 [handler.py:handle_request():146] handle_request: status_report
2024-02-01 18:05:15,920 INFO    Thread-12 :239784 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_175850-i93q0p12/files/output.log
2024-02-01 18:05:18,751 DEBUG   HandlerThread:239784 [handler.py:handle_request():146] handle_request: status_report
2024-02-01 18:05:19,925 INFO    Thread-12 :239784 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_175850-i93q0p12/files/output.log
2024-02-01 18:05:20,595 DEBUG   SenderThread:239784 [sender.py:send():382] send: stats
2024-02-01 18:05:21,928 INFO    Thread-12 :239784 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_175850-i93q0p12/files/output.log
2024-02-01 18:05:24,364 DEBUG   HandlerThread:239784 [handler.py:handle_request():146] handle_request: status_report
2024-02-01 18:05:25,349 DEBUG   HandlerThread:239784 [handler.py:handle_request():146] handle_request: stop_status
2024-02-01 18:05:25,350 DEBUG   SenderThread:239784 [sender.py:send_request():409] send_request: stop_status
2024-02-01 18:05:25,351 DEBUG   HandlerThread:239784 [handler.py:handle_request():146] handle_request: internal_messages
2024-02-01 18:05:25,933 INFO    Thread-12 :239784 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_175850-i93q0p12/files/output.log
2024-02-01 18:05:29,563 DEBUG   HandlerThread:239784 [handler.py:handle_request():146] handle_request: status_report
2024-02-01 18:05:29,937 INFO    Thread-12 :239784 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_175850-i93q0p12/files/output.log
2024-02-01 18:05:33,942 INFO    Thread-12 :239784 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_175850-i93q0p12/files/output.log
2024-02-01 18:05:35,165 DEBUG   HandlerThread:239784 [handler.py:handle_request():146] handle_request: status_report
2024-02-01 18:05:37,947 INFO    Thread-12 :239784 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_175850-i93q0p12/files/output.log
2024-02-01 18:05:39,380 DEBUG   HandlerThread:239784 [handler.py:handle_request():146] handle_request: partial_history
2024-02-01 18:05:39,381 DEBUG   SenderThread:239784 [sender.py:send():382] send: history
2024-02-01 18:05:39,381 DEBUG   SenderThread:239784 [sender.py:send_request():409] send_request: summary_record
2024-02-01 18:05:39,383 INFO    SenderThread:239784 [sender.py:_save_file():1392] saving file wandb-summary.json with policy end
2024-02-01 18:05:39,951 INFO    Thread-12 :239784 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_175850-i93q0p12/files/output.log
2024-02-01 18:05:39,951 INFO    Thread-12 :239784 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_175850-i93q0p12/files/wandb-summary.json
2024-02-01 18:05:40,350 DEBUG   HandlerThread:239784 [handler.py:handle_request():146] handle_request: stop_status
2024-02-01 18:05:40,350 DEBUG   SenderThread:239784 [sender.py:send_request():409] send_request: stop_status
2024-02-01 18:05:40,350 DEBUG   HandlerThread:239784 [handler.py:handle_request():146] handle_request: internal_messages
2024-02-01 18:05:40,443 DEBUG   HandlerThread:239784 [handler.py:handle_request():146] handle_request: status_report
2024-02-01 18:05:40,952 INFO    Thread-12 :239784 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_175850-i93q0p12/files/output.log
2024-02-01 18:05:43,956 INFO    Thread-12 :239784 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_175850-i93q0p12/files/output.log
2024-02-01 18:05:45,987 DEBUG   HandlerThread:239784 [handler.py:handle_request():146] handle_request: status_report
2024-02-01 18:05:47,961 INFO    Thread-12 :239784 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_175850-i93q0p12/files/output.log
2024-02-01 18:05:50,587 DEBUG   SenderThread:239784 [sender.py:send():382] send: stats
2024-02-01 18:05:51,587 DEBUG   HandlerThread:239784 [handler.py:handle_request():146] handle_request: status_report
2024-02-01 18:05:51,966 INFO    Thread-12 :239784 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_175850-i93q0p12/files/output.log
2024-02-01 18:05:55,350 DEBUG   HandlerThread:239784 [handler.py:handle_request():146] handle_request: stop_status
2024-02-01 18:05:55,350 DEBUG   SenderThread:239784 [sender.py:send_request():409] send_request: stop_status
2024-02-01 18:05:55,351 DEBUG   HandlerThread:239784 [handler.py:handle_request():146] handle_request: internal_messages
2024-02-01 18:05:55,970 INFO    Thread-12 :239784 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_175850-i93q0p12/files/output.log
2024-02-01 18:05:57,430 DEBUG   HandlerThread:239784 [handler.py:handle_request():146] handle_request: status_report
2024-02-01 18:05:57,973 INFO    Thread-12 :239784 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_175850-i93q0p12/files/output.log
2024-02-01 18:06:01,978 INFO    Thread-12 :239784 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_175850-i93q0p12/files/output.log
2024-02-01 18:06:03,038 DEBUG   HandlerThread:239784 [handler.py:handle_request():146] handle_request: status_report
2024-02-01 18:06:05,983 INFO    Thread-12 :239784 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_175850-i93q0p12/files/output.log
2024-02-01 18:06:08,254 DEBUG   HandlerThread:239784 [handler.py:handle_request():146] handle_request: status_report
2024-02-01 18:06:09,988 INFO    Thread-12 :239784 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_175850-i93q0p12/files/output.log
2024-02-01 18:06:10,350 DEBUG   HandlerThread:239784 [handler.py:handle_request():146] handle_request: stop_status
2024-02-01 18:06:10,350 DEBUG   SenderThread:239784 [sender.py:send_request():409] send_request: stop_status
2024-02-01 18:06:10,352 DEBUG   HandlerThread:239784 [handler.py:handle_request():146] handle_request: internal_messages
2024-02-01 18:06:13,860 DEBUG   HandlerThread:239784 [handler.py:handle_request():146] handle_request: status_report
2024-02-01 18:06:13,993 INFO    Thread-12 :239784 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_175850-i93q0p12/files/output.log
2024-02-01 18:06:15,460 DEBUG   HandlerThread:239784 [handler.py:handle_request():146] handle_request: partial_history
2024-02-01 18:06:15,461 DEBUG   SenderThread:239784 [sender.py:send():382] send: history
2024-02-01 18:06:15,462 DEBUG   SenderThread:239784 [sender.py:send_request():409] send_request: summary_record
2024-02-01 18:06:15,464 INFO    SenderThread:239784 [sender.py:_save_file():1392] saving file wandb-summary.json with policy end
2024-02-01 18:06:15,996 INFO    Thread-12 :239784 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_175850-i93q0p12/files/output.log
2024-02-01 18:06:15,997 INFO    Thread-12 :239784 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_175850-i93q0p12/files/wandb-summary.json
2024-02-01 18:06:16,998 INFO    Thread-12 :239784 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_175850-i93q0p12/files/output.log
2024-02-01 18:06:19,075 DEBUG   HandlerThread:239784 [handler.py:handle_request():146] handle_request: status_report
2024-02-01 18:06:20,001 INFO    Thread-12 :239784 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_175850-i93q0p12/files/output.log
2024-02-01 18:06:20,589 DEBUG   SenderThread:239784 [sender.py:send():382] send: stats
2024-02-01 18:06:24,006 INFO    Thread-12 :239784 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_175850-i93q0p12/files/output.log
2024-02-01 18:06:24,688 DEBUG   HandlerThread:239784 [handler.py:handle_request():146] handle_request: status_report
2024-02-01 18:06:25,350 DEBUG   HandlerThread:239784 [handler.py:handle_request():146] handle_request: stop_status
2024-02-01 18:06:25,350 DEBUG   SenderThread:239784 [sender.py:send_request():409] send_request: stop_status
2024-02-01 18:06:25,352 DEBUG   HandlerThread:239784 [handler.py:handle_request():146] handle_request: internal_messages
2024-02-01 18:06:28,012 INFO    Thread-12 :239784 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_175850-i93q0p12/files/output.log
2024-02-01 18:06:29,911 DEBUG   HandlerThread:239784 [handler.py:handle_request():146] handle_request: status_report
2024-02-01 18:06:32,017 INFO    Thread-12 :239784 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_175850-i93q0p12/files/output.log
2024-02-01 18:06:34,019 INFO    Thread-12 :239784 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_175850-i93q0p12/files/output.log
2024-02-01 18:06:35,529 DEBUG   HandlerThread:239784 [handler.py:handle_request():146] handle_request: status_report
2024-02-01 18:06:38,025 INFO    Thread-12 :239784 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_175850-i93q0p12/files/output.log
2024-02-01 18:06:40,350 DEBUG   HandlerThread:239784 [handler.py:handle_request():146] handle_request: stop_status
2024-02-01 18:06:40,350 DEBUG   SenderThread:239784 [sender.py:send_request():409] send_request: stop_status
2024-02-01 18:06:40,351 DEBUG   HandlerThread:239784 [handler.py:handle_request():146] handle_request: internal_messages
2024-02-01 18:06:40,741 DEBUG   HandlerThread:239784 [handler.py:handle_request():146] handle_request: status_report
2024-02-01 18:06:42,029 INFO    Thread-12 :239784 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_175850-i93q0p12/files/output.log
2024-02-01 18:06:46,035 INFO    Thread-12 :239784 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_175850-i93q0p12/files/output.log
2024-02-01 18:06:46,345 DEBUG   HandlerThread:239784 [handler.py:handle_request():146] handle_request: status_report
2024-02-01 18:06:50,039 INFO    Thread-12 :239784 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_175850-i93q0p12/files/output.log
2024-02-01 18:06:50,590 DEBUG   SenderThread:239784 [sender.py:send():382] send: stats
2024-02-01 18:06:51,549 DEBUG   HandlerThread:239784 [handler.py:handle_request():146] handle_request: status_report
2024-02-01 18:06:51,550 DEBUG   HandlerThread:239784 [handler.py:handle_request():146] handle_request: partial_history
2024-02-01 18:06:51,551 DEBUG   SenderThread:239784 [sender.py:send():382] send: history
2024-02-01 18:06:51,552 DEBUG   SenderThread:239784 [sender.py:send_request():409] send_request: summary_record
2024-02-01 18:06:51,554 INFO    SenderThread:239784 [sender.py:_save_file():1392] saving file wandb-summary.json with policy end
2024-02-01 18:06:52,043 INFO    Thread-12 :239784 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_175850-i93q0p12/files/output.log
2024-02-01 18:06:52,043 INFO    Thread-12 :239784 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_175850-i93q0p12/files/wandb-summary.json
2024-02-01 18:06:53,044 INFO    Thread-12 :239784 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_175850-i93q0p12/files/output.log
2024-02-01 18:06:55,350 DEBUG   HandlerThread:239784 [handler.py:handle_request():146] handle_request: stop_status
2024-02-01 18:06:55,350 DEBUG   SenderThread:239784 [sender.py:send_request():409] send_request: stop_status
2024-02-01 18:06:55,352 DEBUG   HandlerThread:239784 [handler.py:handle_request():146] handle_request: internal_messages
2024-02-01 18:06:56,048 INFO    Thread-12 :239784 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_175850-i93q0p12/files/output.log
2024-02-01 18:06:57,499 DEBUG   HandlerThread:239784 [handler.py:handle_request():146] handle_request: status_report
2024-02-01 18:07:00,053 INFO    Thread-12 :239784 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_175850-i93q0p12/files/output.log
2024-02-01 18:07:03,326 DEBUG   HandlerThread:239784 [handler.py:handle_request():146] handle_request: status_report
2024-02-01 18:07:04,058 INFO    Thread-12 :239784 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_175850-i93q0p12/files/output.log
2024-02-01 18:07:08,063 INFO    Thread-12 :239784 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_175850-i93q0p12/files/output.log
2024-02-01 18:07:08,919 DEBUG   HandlerThread:239784 [handler.py:handle_request():146] handle_request: status_report
2024-02-01 18:07:10,065 INFO    Thread-12 :239784 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_175850-i93q0p12/files/output.log
2024-02-01 18:07:10,350 DEBUG   HandlerThread:239784 [handler.py:handle_request():146] handle_request: stop_status
2024-02-01 18:07:10,350 DEBUG   SenderThread:239784 [sender.py:send_request():409] send_request: stop_status
2024-02-01 18:07:10,352 DEBUG   HandlerThread:239784 [handler.py:handle_request():146] handle_request: internal_messages
2024-02-01 18:07:14,071 INFO    Thread-12 :239784 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_175850-i93q0p12/files/output.log
2024-02-01 18:07:14,115 DEBUG   HandlerThread:239784 [handler.py:handle_request():146] handle_request: status_report
2024-02-01 18:07:18,076 INFO    Thread-12 :239784 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_175850-i93q0p12/files/output.log
2024-02-01 18:07:19,718 DEBUG   HandlerThread:239784 [handler.py:handle_request():146] handle_request: status_report
2024-02-01 18:07:20,602 DEBUG   SenderThread:239784 [sender.py:send():382] send: stats
2024-02-01 18:07:22,081 INFO    Thread-12 :239784 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_175850-i93q0p12/files/output.log
2024-02-01 18:07:24,919 DEBUG   HandlerThread:239784 [handler.py:handle_request():146] handle_request: status_report
2024-02-01 18:07:25,350 DEBUG   HandlerThread:239784 [handler.py:handle_request():146] handle_request: stop_status
2024-02-01 18:07:25,350 DEBUG   SenderThread:239784 [sender.py:send_request():409] send_request: stop_status
2024-02-01 18:07:25,352 DEBUG   HandlerThread:239784 [handler.py:handle_request():146] handle_request: internal_messages
2024-02-01 18:07:26,086 INFO    Thread-12 :239784 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_175850-i93q0p12/files/output.log
2024-02-01 18:07:27,566 DEBUG   HandlerThread:239784 [handler.py:handle_request():146] handle_request: partial_history
2024-02-01 18:07:27,567 DEBUG   SenderThread:239784 [sender.py:send():382] send: history
2024-02-01 18:07:27,567 DEBUG   SenderThread:239784 [sender.py:send_request():409] send_request: summary_record
2024-02-01 18:07:27,569 INFO    SenderThread:239784 [sender.py:_save_file():1392] saving file wandb-summary.json with policy end
2024-02-01 18:07:28,089 INFO    Thread-12 :239784 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_175850-i93q0p12/files/output.log
2024-02-01 18:07:28,090 INFO    Thread-12 :239784 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_175850-i93q0p12/files/wandb-summary.json
2024-02-01 18:07:29,091 INFO    Thread-12 :239784 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_175850-i93q0p12/files/output.log
2024-02-01 18:07:30,570 DEBUG   HandlerThread:239784 [handler.py:handle_request():146] handle_request: status_report
2024-02-01 18:07:32,095 INFO    Thread-12 :239784 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_175850-i93q0p12/files/output.log
2024-02-01 18:07:35,723 DEBUG   HandlerThread:239784 [handler.py:handle_request():146] handle_request: status_report
2024-02-01 18:07:36,101 INFO    Thread-12 :239784 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_175850-i93q0p12/files/output.log
2024-02-01 18:07:40,107 INFO    Thread-12 :239784 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_175850-i93q0p12/files/output.log
2024-02-01 18:07:40,350 DEBUG   HandlerThread:239784 [handler.py:handle_request():146] handle_request: stop_status
2024-02-01 18:07:40,351 DEBUG   SenderThread:239784 [sender.py:send_request():409] send_request: stop_status
2024-02-01 18:07:40,352 DEBUG   HandlerThread:239784 [handler.py:handle_request():146] handle_request: internal_messages
2024-02-01 18:07:41,482 DEBUG   HandlerThread:239784 [handler.py:handle_request():146] handle_request: status_report
2024-02-01 18:07:44,119 INFO    Thread-12 :239784 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_175850-i93q0p12/files/output.log
2024-02-01 18:07:46,123 INFO    Thread-12 :239784 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_175850-i93q0p12/files/output.log
2024-02-01 18:07:47,458 DEBUG   HandlerThread:239784 [handler.py:handle_request():146] handle_request: status_report
2024-02-01 18:07:50,133 INFO    Thread-12 :239784 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_175850-i93q0p12/files/output.log
2024-02-01 18:07:50,594 DEBUG   SenderThread:239784 [sender.py:send():382] send: stats
2024-02-01 18:07:52,595 DEBUG   HandlerThread:239784 [handler.py:handle_request():146] handle_request: status_report
2024-02-01 18:07:54,144 INFO    Thread-12 :239784 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_175850-i93q0p12/files/output.log
2024-02-01 18:07:55,350 DEBUG   HandlerThread:239784 [handler.py:handle_request():146] handle_request: stop_status
2024-02-01 18:07:55,351 DEBUG   SenderThread:239784 [sender.py:send_request():409] send_request: stop_status
2024-02-01 18:07:55,352 DEBUG   HandlerThread:239784 [handler.py:handle_request():146] handle_request: internal_messages
2024-02-01 18:07:58,154 INFO    Thread-12 :239784 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_175850-i93q0p12/files/output.log
2024-02-01 18:07:58,183 DEBUG   HandlerThread:239784 [handler.py:handle_request():146] handle_request: status_report
2024-02-01 18:08:02,165 INFO    Thread-12 :239784 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_175850-i93q0p12/files/output.log
2024-02-01 18:08:03,329 DEBUG   HandlerThread:239784 [handler.py:handle_request():146] handle_request: status_report
2024-02-01 18:08:03,330 DEBUG   HandlerThread:239784 [handler.py:handle_request():146] handle_request: partial_history
2024-02-01 18:08:03,331 DEBUG   SenderThread:239784 [sender.py:send():382] send: history
2024-02-01 18:08:03,331 DEBUG   SenderThread:239784 [sender.py:send_request():409] send_request: summary_record
2024-02-01 18:08:03,346 INFO    SenderThread:239784 [sender.py:_save_file():1392] saving file wandb-summary.json with policy end
2024-02-01 18:08:04,175 INFO    Thread-12 :239784 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_175850-i93q0p12/files/output.log
2024-02-01 18:08:04,175 INFO    Thread-12 :239784 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_175850-i93q0p12/files/wandb-summary.json
2024-02-01 18:08:05,177 INFO    Thread-12 :239784 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_175850-i93q0p12/files/output.log
2024-02-01 18:08:08,184 INFO    Thread-12 :239784 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_175850-i93q0p12/files/output.log
2024-02-01 18:08:08,914 DEBUG   HandlerThread:239784 [handler.py:handle_request():146] handle_request: status_report
2024-02-01 18:08:10,350 DEBUG   HandlerThread:239784 [handler.py:handle_request():146] handle_request: stop_status
2024-02-01 18:08:10,351 DEBUG   SenderThread:239784 [sender.py:send_request():409] send_request: stop_status
2024-02-01 18:08:10,352 DEBUG   HandlerThread:239784 [handler.py:handle_request():146] handle_request: internal_messages
2024-02-01 18:08:12,193 INFO    Thread-12 :239784 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_175850-i93q0p12/files/output.log
2024-02-01 18:08:14,072 DEBUG   HandlerThread:239784 [handler.py:handle_request():146] handle_request: status_report
2024-02-01 18:08:16,203 INFO    Thread-12 :239784 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_175850-i93q0p12/files/output.log
2024-02-01 18:08:19,644 DEBUG   HandlerThread:239784 [handler.py:handle_request():146] handle_request: status_report
2024-02-01 18:08:20,210 INFO    Thread-12 :239784 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_175850-i93q0p12/files/output.log
2024-02-01 18:08:20,596 DEBUG   SenderThread:239784 [sender.py:send():382] send: stats
2024-02-01 18:08:22,213 INFO    Thread-12 :239784 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_175850-i93q0p12/files/output.log
2024-02-01 18:08:24,790 DEBUG   HandlerThread:239784 [handler.py:handle_request():146] handle_request: status_report
2024-02-01 18:08:25,351 DEBUG   HandlerThread:239784 [handler.py:handle_request():146] handle_request: stop_status
2024-02-01 18:08:25,351 DEBUG   SenderThread:239784 [sender.py:send_request():409] send_request: stop_status
2024-02-01 18:08:25,352 DEBUG   HandlerThread:239784 [handler.py:handle_request():146] handle_request: internal_messages
2024-02-01 18:08:26,218 INFO    Thread-12 :239784 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_175850-i93q0p12/files/output.log
2024-02-01 18:08:30,223 INFO    Thread-12 :239784 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_175850-i93q0p12/files/output.log
2024-02-01 18:08:30,353 DEBUG   HandlerThread:239784 [handler.py:handle_request():146] handle_request: status_report
2024-02-01 18:08:34,228 INFO    Thread-12 :239784 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_175850-i93q0p12/files/output.log
2024-02-01 18:08:35,552 DEBUG   HandlerThread:239784 [handler.py:handle_request():146] handle_request: status_report
2024-02-01 18:08:36,230 INFO    Thread-12 :239784 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_175850-i93q0p12/files/output.log
2024-02-01 18:08:39,143 DEBUG   HandlerThread:239784 [handler.py:handle_request():146] handle_request: partial_history
2024-02-01 18:08:39,144 DEBUG   SenderThread:239784 [sender.py:send():382] send: history
2024-02-01 18:08:39,144 DEBUG   SenderThread:239784 [sender.py:send_request():409] send_request: summary_record
2024-02-01 18:08:39,146 INFO    SenderThread:239784 [sender.py:_save_file():1392] saving file wandb-summary.json with policy end
2024-02-01 18:08:39,235 INFO    Thread-12 :239784 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_175850-i93q0p12/files/wandb-summary.json
2024-02-01 18:08:40,236 INFO    Thread-12 :239784 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_175850-i93q0p12/files/output.log
2024-02-01 18:08:40,351 DEBUG   HandlerThread:239784 [handler.py:handle_request():146] handle_request: stop_status
2024-02-01 18:08:40,351 DEBUG   SenderThread:239784 [sender.py:send_request():409] send_request: stop_status
2024-02-01 18:08:40,353 DEBUG   HandlerThread:239784 [handler.py:handle_request():146] handle_request: internal_messages
2024-02-01 18:08:41,238 INFO    Thread-12 :239784 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_175850-i93q0p12/files/output.log
2024-02-01 18:08:41,422 DEBUG   HandlerThread:239784 [handler.py:handle_request():146] handle_request: status_report
2024-02-01 18:08:44,242 INFO    Thread-12 :239784 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_175850-i93q0p12/files/output.log
2024-02-01 18:08:47,289 DEBUG   HandlerThread:239784 [handler.py:handle_request():146] handle_request: status_report
2024-02-01 18:08:48,247 INFO    Thread-12 :239784 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_175850-i93q0p12/files/output.log
2024-02-01 18:08:50,600 DEBUG   SenderThread:239784 [sender.py:send():382] send: stats
2024-02-01 18:08:52,260 INFO    Thread-12 :239784 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_175850-i93q0p12/files/output.log
2024-02-01 18:08:52,602 DEBUG   HandlerThread:239784 [handler.py:handle_request():146] handle_request: status_report
2024-02-01 18:08:54,264 INFO    Thread-12 :239784 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_175850-i93q0p12/files/output.log
2024-02-01 18:08:55,351 DEBUG   HandlerThread:239784 [handler.py:handle_request():146] handle_request: stop_status
2024-02-01 18:08:55,351 DEBUG   SenderThread:239784 [sender.py:send_request():409] send_request: stop_status
2024-02-01 18:08:55,355 DEBUG   HandlerThread:239784 [handler.py:handle_request():146] handle_request: internal_messages
2024-02-01 18:08:58,269 DEBUG   HandlerThread:239784 [handler.py:handle_request():146] handle_request: status_report
2024-02-01 18:08:58,272 INFO    Thread-12 :239784 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_175850-i93q0p12/files/output.log
2024-02-01 18:09:02,277 INFO    Thread-12 :239784 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_175850-i93q0p12/files/output.log
2024-02-01 18:09:03,969 DEBUG   HandlerThread:239784 [handler.py:handle_request():146] handle_request: status_report
2024-02-01 18:09:06,282 INFO    Thread-12 :239784 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_175850-i93q0p12/files/output.log
2024-02-01 18:09:09,260 DEBUG   HandlerThread:239784 [handler.py:handle_request():146] handle_request: status_report
2024-02-01 18:09:10,287 INFO    Thread-12 :239784 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_175850-i93q0p12/files/output.log
2024-02-01 18:09:10,351 DEBUG   HandlerThread:239784 [handler.py:handle_request():146] handle_request: stop_status
2024-02-01 18:09:10,351 DEBUG   SenderThread:239784 [sender.py:send_request():409] send_request: stop_status
2024-02-01 18:09:10,355 DEBUG   HandlerThread:239784 [handler.py:handle_request():146] handle_request: internal_messages
2024-02-01 18:09:14,292 INFO    Thread-12 :239784 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_175850-i93q0p12/files/output.log
2024-02-01 18:09:14,879 DEBUG   HandlerThread:239784 [handler.py:handle_request():146] handle_request: status_report
2024-02-01 18:09:15,517 DEBUG   HandlerThread:239784 [handler.py:handle_request():146] handle_request: partial_history
2024-02-01 18:09:15,520 DEBUG   SenderThread:239784 [sender.py:send():382] send: history
2024-02-01 18:09:15,520 DEBUG   SenderThread:239784 [sender.py:send_request():409] send_request: summary_record
2024-02-01 18:09:15,524 INFO    SenderThread:239784 [sender.py:_save_file():1392] saving file wandb-summary.json with policy end
2024-02-01 18:09:16,295 INFO    Thread-12 :239784 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_175850-i93q0p12/files/output.log
2024-02-01 18:09:16,296 INFO    Thread-12 :239784 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_175850-i93q0p12/files/wandb-summary.json
2024-02-01 18:09:17,297 INFO    Thread-12 :239784 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_175850-i93q0p12/files/output.log
2024-02-01 18:09:20,151 DEBUG   HandlerThread:239784 [handler.py:handle_request():146] handle_request: status_report
2024-02-01 18:09:20,301 INFO    Thread-12 :239784 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_175850-i93q0p12/files/output.log
2024-02-01 18:09:20,601 DEBUG   SenderThread:239784 [sender.py:send():382] send: stats
2024-02-01 18:09:24,306 INFO    Thread-12 :239784 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_175850-i93q0p12/files/output.log
2024-02-01 18:09:25,351 DEBUG   HandlerThread:239784 [handler.py:handle_request():146] handle_request: stop_status
2024-02-01 18:09:25,351 DEBUG   SenderThread:239784 [sender.py:send_request():409] send_request: stop_status
2024-02-01 18:09:25,353 DEBUG   HandlerThread:239784 [handler.py:handle_request():146] handle_request: internal_messages
2024-02-01 18:09:25,438 DEBUG   HandlerThread:239784 [handler.py:handle_request():146] handle_request: status_report
2024-02-01 18:09:28,311 INFO    Thread-12 :239784 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_175850-i93q0p12/files/output.log
2024-02-01 18:09:31,008 DEBUG   HandlerThread:239784 [handler.py:handle_request():146] handle_request: status_report
2024-02-01 18:09:32,315 INFO    Thread-12 :239784 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_175850-i93q0p12/files/output.log
2024-02-01 18:09:36,320 INFO    Thread-12 :239784 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_175850-i93q0p12/files/output.log
2024-02-01 18:09:36,632 DEBUG   HandlerThread:239784 [handler.py:handle_request():146] handle_request: status_report
2024-02-01 18:09:38,323 INFO    Thread-12 :239784 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_175850-i93q0p12/files/output.log
2024-02-01 18:09:40,351 DEBUG   HandlerThread:239784 [handler.py:handle_request():146] handle_request: stop_status
2024-02-01 18:09:40,351 DEBUG   SenderThread:239784 [sender.py:send_request():409] send_request: stop_status
2024-02-01 18:09:40,353 DEBUG   HandlerThread:239784 [handler.py:handle_request():146] handle_request: internal_messages
2024-02-01 18:09:41,854 DEBUG   HandlerThread:239784 [handler.py:handle_request():146] handle_request: status_report
2024-02-01 18:09:42,328 INFO    Thread-12 :239784 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_175850-i93q0p12/files/output.log
2024-02-01 18:09:46,333 INFO    Thread-12 :239784 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_175850-i93q0p12/files/output.log
2024-02-01 18:09:47,458 DEBUG   HandlerThread:239784 [handler.py:handle_request():146] handle_request: status_report
2024-02-01 18:09:50,338 INFO    Thread-12 :239784 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_175850-i93q0p12/files/output.log
2024-02-01 18:09:50,602 DEBUG   SenderThread:239784 [sender.py:send():382] send: stats
2024-02-01 18:09:51,687 DEBUG   HandlerThread:239784 [handler.py:handle_request():146] handle_request: partial_history
2024-02-01 18:09:51,689 DEBUG   SenderThread:239784 [sender.py:send():382] send: history
2024-02-01 18:09:51,689 DEBUG   SenderThread:239784 [sender.py:send_request():409] send_request: summary_record
2024-02-01 18:09:51,691 INFO    SenderThread:239784 [sender.py:_save_file():1392] saving file wandb-summary.json with policy end
2024-02-01 18:09:52,341 INFO    Thread-12 :239784 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_175850-i93q0p12/files/wandb-summary.json
2024-02-01 18:09:52,692 DEBUG   HandlerThread:239784 [handler.py:handle_request():146] handle_request: status_report
2024-02-01 18:09:53,343 INFO    Thread-12 :239784 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_175850-i93q0p12/files/output.log
2024-02-01 18:09:54,344 INFO    Thread-12 :239784 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_175850-i93q0p12/files/output.log
2024-02-01 18:09:55,420 DEBUG   HandlerThread:239784 [handler.py:handle_request():146] handle_request: internal_messages
2024-02-01 18:09:55,421 DEBUG   HandlerThread:239784 [handler.py:handle_request():146] handle_request: stop_status
2024-02-01 18:09:55,421 DEBUG   SenderThread:239784 [sender.py:send_request():409] send_request: stop_status
2024-02-01 18:09:56,347 INFO    Thread-12 :239784 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_175850-i93q0p12/files/output.log
2024-02-01 18:09:58,569 DEBUG   HandlerThread:239784 [handler.py:handle_request():146] handle_request: status_report
2024-02-01 18:10:00,352 INFO    Thread-12 :239784 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_175850-i93q0p12/files/output.log
2024-02-01 18:10:04,357 INFO    Thread-12 :239784 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_175850-i93q0p12/files/output.log
2024-02-01 18:10:04,511 DEBUG   HandlerThread:239784 [handler.py:handle_request():146] handle_request: status_report
2024-02-01 18:10:08,362 INFO    Thread-12 :239784 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_175850-i93q0p12/files/output.log
2024-02-01 18:10:09,721 DEBUG   HandlerThread:239784 [handler.py:handle_request():146] handle_request: status_report
2024-02-01 18:10:10,384 DEBUG   HandlerThread:239784 [handler.py:handle_request():146] handle_request: stop_status
2024-02-01 18:10:10,384 DEBUG   SenderThread:239784 [sender.py:send_request():409] send_request: stop_status
2024-02-01 18:10:10,407 DEBUG   HandlerThread:239784 [handler.py:handle_request():146] handle_request: internal_messages
2024-02-01 18:10:12,367 INFO    Thread-12 :239784 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_175850-i93q0p12/files/output.log
2024-02-01 18:10:14,369 INFO    Thread-12 :239784 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_175850-i93q0p12/files/output.log
2024-02-01 18:10:15,328 DEBUG   HandlerThread:239784 [handler.py:handle_request():146] handle_request: status_report
2024-02-01 18:10:18,374 INFO    Thread-12 :239784 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_175850-i93q0p12/files/output.log
2024-02-01 18:10:20,510 DEBUG   HandlerThread:239784 [handler.py:handle_request():146] handle_request: status_report
2024-02-01 18:10:20,604 DEBUG   SenderThread:239784 [sender.py:send():382] send: stats
2024-02-01 18:10:22,380 INFO    Thread-12 :239784 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_175850-i93q0p12/files/output.log
2024-02-01 18:10:25,383 DEBUG   HandlerThread:239784 [handler.py:handle_request():146] handle_request: stop_status
2024-02-01 18:10:25,384 DEBUG   SenderThread:239784 [sender.py:send_request():409] send_request: stop_status
2024-02-01 18:10:25,407 DEBUG   HandlerThread:239784 [handler.py:handle_request():146] handle_request: internal_messages
2024-02-01 18:10:26,387 INFO    Thread-12 :239784 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_175850-i93q0p12/files/output.log
2024-02-01 18:10:26,498 DEBUG   HandlerThread:239784 [handler.py:handle_request():146] handle_request: status_report
2024-02-01 18:10:27,754 DEBUG   HandlerThread:239784 [handler.py:handle_request():146] handle_request: partial_history
2024-02-01 18:10:27,755 DEBUG   SenderThread:239784 [sender.py:send():382] send: history
2024-02-01 18:10:27,755 DEBUG   SenderThread:239784 [sender.py:send_request():409] send_request: summary_record
2024-02-01 18:10:27,757 INFO    SenderThread:239784 [sender.py:_save_file():1392] saving file wandb-summary.json with policy end
2024-02-01 18:10:28,390 INFO    Thread-12 :239784 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_175850-i93q0p12/files/wandb-summary.json
2024-02-01 18:10:29,391 INFO    Thread-12 :239784 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_175850-i93q0p12/files/output.log
2024-02-01 18:10:30,393 INFO    Thread-12 :239784 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_175850-i93q0p12/files/output.log
2024-02-01 18:10:32,302 DEBUG   HandlerThread:239784 [handler.py:handle_request():146] handle_request: status_report
2024-02-01 18:10:32,395 INFO    Thread-12 :239784 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_175850-i93q0p12/files/output.log
2024-02-01 18:10:36,400 INFO    Thread-12 :239784 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_175850-i93q0p12/files/output.log
2024-02-01 18:10:37,916 DEBUG   HandlerThread:239784 [handler.py:handle_request():146] handle_request: status_report
2024-02-01 18:10:40,383 DEBUG   HandlerThread:239784 [handler.py:handle_request():146] handle_request: stop_status
2024-02-01 18:10:40,384 DEBUG   SenderThread:239784 [sender.py:send_request():409] send_request: stop_status
2024-02-01 18:10:40,405 INFO    Thread-12 :239784 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_175850-i93q0p12/files/output.log
2024-02-01 18:10:40,407 DEBUG   HandlerThread:239784 [handler.py:handle_request():146] handle_request: internal_messages
2024-02-01 18:10:43,116 DEBUG   HandlerThread:239784 [handler.py:handle_request():146] handle_request: status_report
2024-02-01 18:10:44,411 INFO    Thread-12 :239784 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_175850-i93q0p12/files/output.log
2024-02-01 18:10:48,416 INFO    Thread-12 :239784 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_175850-i93q0p12/files/output.log
2024-02-01 18:10:48,718 DEBUG   HandlerThread:239784 [handler.py:handle_request():146] handle_request: status_report
2024-02-01 18:10:50,418 INFO    Thread-12 :239784 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_175850-i93q0p12/files/output.log
2024-02-01 18:10:50,606 DEBUG   SenderThread:239784 [sender.py:send():382] send: stats
2024-02-01 18:10:53,902 DEBUG   HandlerThread:239784 [handler.py:handle_request():146] handle_request: status_report
2024-02-01 18:10:54,423 INFO    Thread-12 :239784 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_175850-i93q0p12/files/output.log
2024-02-01 18:10:55,383 DEBUG   HandlerThread:239784 [handler.py:handle_request():146] handle_request: stop_status
2024-02-01 18:10:55,384 DEBUG   SenderThread:239784 [sender.py:send_request():409] send_request: stop_status
2024-02-01 18:10:55,407 DEBUG   HandlerThread:239784 [handler.py:handle_request():146] handle_request: internal_messages
2024-02-01 18:10:58,428 INFO    Thread-12 :239784 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_175850-i93q0p12/files/output.log
2024-02-01 18:10:59,501 DEBUG   HandlerThread:239784 [handler.py:handle_request():146] handle_request: status_report
2024-02-01 18:11:02,433 INFO    Thread-12 :239784 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_175850-i93q0p12/files/output.log
2024-02-01 18:11:03,689 DEBUG   HandlerThread:239784 [handler.py:handle_request():146] handle_request: partial_history
2024-02-01 18:11:03,691 DEBUG   SenderThread:239784 [sender.py:send():382] send: history
2024-02-01 18:11:03,691 DEBUG   SenderThread:239784 [sender.py:send_request():409] send_request: summary_record
2024-02-01 18:11:03,693 INFO    SenderThread:239784 [sender.py:_save_file():1392] saving file wandb-summary.json with policy end
2024-02-01 18:11:04,437 INFO    Thread-12 :239784 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_175850-i93q0p12/files/wandb-summary.json
2024-02-01 18:11:04,693 DEBUG   HandlerThread:239784 [handler.py:handle_request():146] handle_request: status_report
2024-02-01 18:11:05,438 INFO    Thread-12 :239784 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_175850-i93q0p12/files/output.log
2024-02-01 18:11:06,440 INFO    Thread-12 :239784 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_175850-i93q0p12/files/output.log
2024-02-01 18:11:08,442 INFO    Thread-12 :239784 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_175850-i93q0p12/files/output.log
2024-02-01 18:11:10,295 DEBUG   HandlerThread:239784 [handler.py:handle_request():146] handle_request: status_report
2024-02-01 18:11:10,384 DEBUG   HandlerThread:239784 [handler.py:handle_request():146] handle_request: stop_status
2024-02-01 18:11:10,384 DEBUG   SenderThread:239784 [sender.py:send_request():409] send_request: stop_status
2024-02-01 18:11:10,407 DEBUG   HandlerThread:239784 [handler.py:handle_request():146] handle_request: internal_messages
2024-02-01 18:11:12,447 INFO    Thread-12 :239784 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_175850-i93q0p12/files/output.log
2024-02-01 18:11:15,474 DEBUG   HandlerThread:239784 [handler.py:handle_request():146] handle_request: status_report
2024-02-01 18:11:16,452 INFO    Thread-12 :239784 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_175850-i93q0p12/files/output.log
2024-02-01 18:11:20,457 INFO    Thread-12 :239784 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_175850-i93q0p12/files/output.log
2024-02-01 18:11:20,608 DEBUG   SenderThread:239784 [sender.py:send():382] send: stats
2024-02-01 18:11:20,609 DEBUG   HandlerThread:239784 [handler.py:handle_request():146] handle_request: status_report
2024-02-01 18:11:24,462 INFO    Thread-12 :239784 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_175850-i93q0p12/files/output.log
2024-02-01 18:11:25,384 DEBUG   HandlerThread:239784 [handler.py:handle_request():146] handle_request: stop_status
2024-02-01 18:11:25,384 DEBUG   SenderThread:239784 [sender.py:send_request():409] send_request: stop_status
2024-02-01 18:11:25,407 DEBUG   HandlerThread:239784 [handler.py:handle_request():146] handle_request: internal_messages
2024-02-01 18:11:26,465 INFO    Thread-12 :239784 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_175850-i93q0p12/files/output.log
2024-02-01 18:11:26,470 DEBUG   HandlerThread:239784 [handler.py:handle_request():146] handle_request: status_report
2024-02-01 18:11:30,470 INFO    Thread-12 :239784 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_175850-i93q0p12/files/output.log
2024-02-01 18:11:31,839 DEBUG   HandlerThread:239784 [handler.py:handle_request():146] handle_request: status_report
2024-02-01 18:11:34,475 INFO    Thread-12 :239784 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_175850-i93q0p12/files/output.log
2024-02-01 18:11:37,033 DEBUG   HandlerThread:239784 [handler.py:handle_request():146] handle_request: status_report
2024-02-01 18:11:38,480 INFO    Thread-12 :239784 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_175850-i93q0p12/files/output.log
2024-02-01 18:11:39,627 DEBUG   HandlerThread:239784 [handler.py:handle_request():146] handle_request: partial_history
2024-02-01 18:11:39,628 DEBUG   SenderThread:239784 [sender.py:send():382] send: history
2024-02-01 18:11:39,628 DEBUG   SenderThread:239784 [sender.py:send_request():409] send_request: summary_record
2024-02-01 18:11:39,630 INFO    SenderThread:239784 [sender.py:_save_file():1392] saving file wandb-summary.json with policy end
2024-02-01 18:11:40,384 DEBUG   HandlerThread:239784 [handler.py:handle_request():146] handle_request: stop_status
2024-02-01 18:11:40,384 DEBUG   SenderThread:239784 [sender.py:send_request():409] send_request: stop_status
2024-02-01 18:11:40,407 DEBUG   HandlerThread:239784 [handler.py:handle_request():146] handle_request: internal_messages
2024-02-01 18:11:40,483 INFO    Thread-12 :239784 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_175850-i93q0p12/files/output.log
2024-02-01 18:11:40,483 INFO    Thread-12 :239784 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_175850-i93q0p12/files/wandb-summary.json
2024-02-01 18:11:42,527 DEBUG   HandlerThread:239784 [handler.py:handle_request():146] handle_request: status_report
2024-02-01 18:11:44,488 INFO    Thread-12 :239784 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_175850-i93q0p12/files/output.log
2024-02-01 18:11:47,808 DEBUG   HandlerThread:239784 [handler.py:handle_request():146] handle_request: status_report
2024-02-01 18:11:48,493 INFO    Thread-12 :239784 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_175850-i93q0p12/files/output.log
2024-02-01 18:11:50,619 DEBUG   SenderThread:239784 [sender.py:send():382] send: stats
2024-02-01 18:11:52,498 INFO    Thread-12 :239784 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_175850-i93q0p12/files/output.log
2024-02-01 18:11:53,620 DEBUG   HandlerThread:239784 [handler.py:handle_request():146] handle_request: status_report
2024-02-01 18:11:55,384 DEBUG   HandlerThread:239784 [handler.py:handle_request():146] handle_request: stop_status
2024-02-01 18:11:55,385 DEBUG   SenderThread:239784 [sender.py:send_request():409] send_request: stop_status
2024-02-01 18:11:55,407 DEBUG   HandlerThread:239784 [handler.py:handle_request():146] handle_request: internal_messages
2024-02-01 18:11:56,503 INFO    Thread-12 :239784 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_175850-i93q0p12/files/output.log
2024-02-01 18:11:58,505 INFO    Thread-12 :239784 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_175850-i93q0p12/files/output.log
2024-02-01 18:11:59,589 DEBUG   HandlerThread:239784 [handler.py:handle_request():146] handle_request: status_report
2024-02-01 18:12:02,510 INFO    Thread-12 :239784 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_175850-i93q0p12/files/output.log
2024-02-01 18:12:04,771 DEBUG   HandlerThread:239784 [handler.py:handle_request():146] handle_request: status_report
2024-02-01 18:12:06,515 INFO    Thread-12 :239784 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_175850-i93q0p12/files/output.log
2024-02-01 18:12:10,369 DEBUG   HandlerThread:239784 [handler.py:handle_request():146] handle_request: status_report
2024-02-01 18:12:10,384 DEBUG   HandlerThread:239784 [handler.py:handle_request():146] handle_request: stop_status
2024-02-01 18:12:10,384 DEBUG   SenderThread:239784 [sender.py:send_request():409] send_request: stop_status
2024-02-01 18:12:10,407 DEBUG   HandlerThread:239784 [handler.py:handle_request():146] handle_request: internal_messages
2024-02-01 18:12:10,520 INFO    Thread-12 :239784 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_175850-i93q0p12/files/output.log
2024-02-01 18:12:14,525 INFO    Thread-12 :239784 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_175850-i93q0p12/files/output.log
2024-02-01 18:12:15,549 DEBUG   HandlerThread:239784 [handler.py:handle_request():146] handle_request: status_report
2024-02-01 18:12:15,550 DEBUG   HandlerThread:239784 [handler.py:handle_request():146] handle_request: partial_history
2024-02-01 18:12:15,551 DEBUG   SenderThread:239784 [sender.py:send():382] send: history
2024-02-01 18:12:15,552 DEBUG   SenderThread:239784 [sender.py:send_request():409] send_request: summary_record
2024-02-01 18:12:15,554 INFO    SenderThread:239784 [sender.py:_save_file():1392] saving file wandb-summary.json with policy end
2024-02-01 18:12:16,530 INFO    Thread-12 :239784 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_175850-i93q0p12/files/output.log
2024-02-01 18:12:16,530 INFO    Thread-12 :239784 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_175850-i93q0p12/files/wandb-summary.json
2024-02-01 18:12:20,534 INFO    Thread-12 :239784 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_175850-i93q0p12/files/output.log
2024-02-01 18:12:20,612 DEBUG   SenderThread:239784 [sender.py:send():382] send: stats
2024-02-01 18:12:20,613 DEBUG   HandlerThread:239784 [handler.py:handle_request():146] handle_request: status_report
2024-02-01 18:12:24,539 INFO    Thread-12 :239784 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_175850-i93q0p12/files/output.log
2024-02-01 18:12:25,384 DEBUG   HandlerThread:239784 [handler.py:handle_request():146] handle_request: stop_status
2024-02-01 18:12:25,385 DEBUG   SenderThread:239784 [sender.py:send_request():409] send_request: stop_status
2024-02-01 18:12:25,407 DEBUG   HandlerThread:239784 [handler.py:handle_request():146] handle_request: internal_messages
2024-02-01 18:12:26,333 DEBUG   HandlerThread:239784 [handler.py:handle_request():146] handle_request: status_report
2024-02-01 18:12:28,544 INFO    Thread-12 :239784 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_175850-i93q0p12/files/output.log
2024-02-01 18:12:31,919 DEBUG   HandlerThread:239784 [handler.py:handle_request():146] handle_request: status_report
2024-02-01 18:12:32,549 INFO    Thread-12 :239784 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_175850-i93q0p12/files/output.log
2024-02-01 18:12:34,552 INFO    Thread-12 :239784 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_175850-i93q0p12/files/output.log
2024-02-01 18:12:37,107 DEBUG   HandlerThread:239784 [handler.py:handle_request():146] handle_request: status_report
2024-02-01 18:12:38,557 INFO    Thread-12 :239784 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_175850-i93q0p12/files/output.log
2024-02-01 18:12:40,384 DEBUG   HandlerThread:239784 [handler.py:handle_request():146] handle_request: stop_status
2024-02-01 18:12:40,385 DEBUG   SenderThread:239784 [sender.py:send_request():409] send_request: stop_status
2024-02-01 18:12:40,408 DEBUG   HandlerThread:239784 [handler.py:handle_request():146] handle_request: internal_messages
2024-02-01 18:12:42,562 INFO    Thread-12 :239784 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_175850-i93q0p12/files/output.log
2024-02-01 18:12:42,706 DEBUG   HandlerThread:239784 [handler.py:handle_request():146] handle_request: status_report
2024-02-01 18:12:46,567 INFO    Thread-12 :239784 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_175850-i93q0p12/files/output.log
2024-02-01 18:12:47,896 DEBUG   HandlerThread:239784 [handler.py:handle_request():146] handle_request: status_report
2024-02-01 18:12:50,571 INFO    Thread-12 :239784 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_175850-i93q0p12/files/output.log
2024-02-01 18:12:50,614 DEBUG   SenderThread:239784 [sender.py:send():382] send: stats
2024-02-01 18:12:51,492 DEBUG   HandlerThread:239784 [handler.py:handle_request():146] handle_request: partial_history
2024-02-01 18:12:51,493 DEBUG   SenderThread:239784 [sender.py:send():382] send: history
2024-02-01 18:12:51,494 DEBUG   SenderThread:239784 [sender.py:send_request():409] send_request: summary_record
2024-02-01 18:12:51,496 INFO    SenderThread:239784 [sender.py:_save_file():1392] saving file wandb-summary.json with policy end
2024-02-01 18:12:51,574 INFO    Thread-12 :239784 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_175850-i93q0p12/files/wandb-summary.json
2024-02-01 18:12:52,575 INFO    Thread-12 :239784 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_175850-i93q0p12/files/output.log
2024-02-01 18:12:53,496 DEBUG   HandlerThread:239784 [handler.py:handle_request():146] handle_request: status_report
2024-02-01 18:12:55,384 DEBUG   HandlerThread:239784 [handler.py:handle_request():146] handle_request: stop_status
2024-02-01 18:12:55,385 DEBUG   SenderThread:239784 [sender.py:send_request():409] send_request: stop_status
2024-02-01 18:12:55,408 DEBUG   HandlerThread:239784 [handler.py:handle_request():146] handle_request: internal_messages
2024-02-01 18:12:56,580 INFO    Thread-12 :239784 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_175850-i93q0p12/files/output.log
2024-02-01 18:12:58,690 DEBUG   HandlerThread:239784 [handler.py:handle_request():146] handle_request: status_report
2024-02-01 18:13:00,585 INFO    Thread-12 :239784 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_175850-i93q0p12/files/output.log
2024-02-01 18:13:04,277 DEBUG   HandlerThread:239784 [handler.py:handle_request():146] handle_request: status_report
2024-02-01 18:13:04,590 INFO    Thread-12 :239784 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_175850-i93q0p12/files/output.log
2024-02-01 18:13:08,595 INFO    Thread-12 :239784 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_175850-i93q0p12/files/output.log
2024-02-01 18:13:09,448 DEBUG   HandlerThread:239784 [handler.py:handle_request():146] handle_request: status_report
2024-02-01 18:13:10,385 DEBUG   HandlerThread:239784 [handler.py:handle_request():146] handle_request: stop_status
2024-02-01 18:13:10,385 DEBUG   SenderThread:239784 [sender.py:send_request():409] send_request: stop_status
2024-02-01 18:13:10,408 DEBUG   HandlerThread:239784 [handler.py:handle_request():146] handle_request: internal_messages
2024-02-01 18:13:10,598 INFO    Thread-12 :239784 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_175850-i93q0p12/files/output.log
2024-02-01 18:13:14,603 INFO    Thread-12 :239784 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_175850-i93q0p12/files/output.log
2024-02-01 18:13:15,033 DEBUG   HandlerThread:239784 [handler.py:handle_request():146] handle_request: status_report
2024-02-01 18:13:18,608 INFO    Thread-12 :239784 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_175850-i93q0p12/files/output.log
2024-02-01 18:13:20,216 DEBUG   HandlerThread:239784 [handler.py:handle_request():146] handle_request: status_report
2024-02-01 18:13:20,616 DEBUG   SenderThread:239784 [sender.py:send():382] send: stats
2024-02-01 18:13:22,613 INFO    Thread-12 :239784 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_175850-i93q0p12/files/output.log
2024-02-01 18:13:25,385 DEBUG   HandlerThread:239784 [handler.py:handle_request():146] handle_request: stop_status
2024-02-01 18:13:25,385 DEBUG   SenderThread:239784 [sender.py:send_request():409] send_request: stop_status
2024-02-01 18:13:25,408 DEBUG   HandlerThread:239784 [handler.py:handle_request():146] handle_request: internal_messages
2024-02-01 18:13:25,501 DEBUG   HandlerThread:239784 [handler.py:handle_request():146] handle_request: status_report
2024-02-01 18:13:26,618 INFO    Thread-12 :239784 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_175850-i93q0p12/files/output.log
2024-02-01 18:13:27,430 DEBUG   HandlerThread:239784 [handler.py:handle_request():146] handle_request: partial_history
2024-02-01 18:13:27,431 DEBUG   SenderThread:239784 [sender.py:send():382] send: history
2024-02-01 18:13:27,432 DEBUG   SenderThread:239784 [sender.py:send_request():409] send_request: summary_record
2024-02-01 18:13:27,433 INFO    SenderThread:239784 [sender.py:_save_file():1392] saving file wandb-summary.json with policy end
2024-02-01 18:13:27,620 INFO    Thread-12 :239784 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_175850-i93q0p12/files/wandb-summary.json
2024-02-01 18:13:28,621 INFO    Thread-12 :239784 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_175850-i93q0p12/files/output.log
2024-02-01 18:13:30,976 DEBUG   HandlerThread:239784 [handler.py:handle_request():146] handle_request: status_report
2024-02-01 18:13:32,626 INFO    Thread-12 :239784 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_175850-i93q0p12/files/output.log
2024-02-01 18:13:36,561 DEBUG   HandlerThread:239784 [handler.py:handle_request():146] handle_request: status_report
2024-02-01 18:13:36,631 INFO    Thread-12 :239784 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_175850-i93q0p12/files/output.log
2024-02-01 18:13:40,385 DEBUG   HandlerThread:239784 [handler.py:handle_request():146] handle_request: stop_status
2024-02-01 18:13:40,385 DEBUG   SenderThread:239784 [sender.py:send_request():409] send_request: stop_status
2024-02-01 18:13:40,408 DEBUG   HandlerThread:239784 [handler.py:handle_request():146] handle_request: internal_messages
2024-02-01 18:13:40,636 INFO    Thread-12 :239784 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_175850-i93q0p12/files/output.log
2024-02-01 18:13:41,750 DEBUG   HandlerThread:239784 [handler.py:handle_request():146] handle_request: status_report
2024-02-01 18:13:44,641 INFO    Thread-12 :239784 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_175850-i93q0p12/files/output.log
2024-02-01 18:13:46,644 INFO    Thread-12 :239784 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_175850-i93q0p12/files/output.log
2024-02-01 18:13:47,330 DEBUG   HandlerThread:239784 [handler.py:handle_request():146] handle_request: status_report
2024-02-01 18:13:50,628 DEBUG   SenderThread:239784 [sender.py:send():382] send: stats
2024-02-01 18:13:50,649 INFO    Thread-12 :239784 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_175850-i93q0p12/files/output.log
2024-02-01 18:13:52,515 DEBUG   HandlerThread:239784 [handler.py:handle_request():146] handle_request: status_report
2024-02-01 18:13:54,654 INFO    Thread-12 :239784 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_175850-i93q0p12/files/output.log
2024-02-01 18:13:55,385 DEBUG   HandlerThread:239784 [handler.py:handle_request():146] handle_request: stop_status
2024-02-01 18:13:55,385 DEBUG   SenderThread:239784 [sender.py:send_request():409] send_request: stop_status
2024-02-01 18:13:55,408 DEBUG   HandlerThread:239784 [handler.py:handle_request():146] handle_request: internal_messages
2024-02-01 18:13:58,110 DEBUG   HandlerThread:239784 [handler.py:handle_request():146] handle_request: status_report
2024-02-01 18:13:58,659 INFO    Thread-12 :239784 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_175850-i93q0p12/files/output.log
2024-02-01 18:14:02,664 INFO    Thread-12 :239784 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_175850-i93q0p12/files/output.log
2024-02-01 18:14:03,307 DEBUG   HandlerThread:239784 [handler.py:handle_request():146] handle_request: status_report
2024-02-01 18:14:03,308 DEBUG   HandlerThread:239784 [handler.py:handle_request():146] handle_request: partial_history
2024-02-01 18:14:03,309 DEBUG   SenderThread:239784 [sender.py:send():382] send: history
2024-02-01 18:14:03,309 DEBUG   SenderThread:239784 [sender.py:send_request():409] send_request: summary_record
2024-02-01 18:14:03,311 INFO    SenderThread:239784 [sender.py:_save_file():1392] saving file wandb-summary.json with policy end
2024-02-01 18:14:03,666 INFO    Thread-12 :239784 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_175850-i93q0p12/files/wandb-summary.json
2024-02-01 18:14:04,667 INFO    Thread-12 :239784 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_175850-i93q0p12/files/output.log
2024-02-01 18:14:08,672 INFO    Thread-12 :239784 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_175850-i93q0p12/files/output.log
2024-02-01 18:14:08,892 DEBUG   HandlerThread:239784 [handler.py:handle_request():146] handle_request: status_report
2024-02-01 18:14:10,385 DEBUG   HandlerThread:239784 [handler.py:handle_request():146] handle_request: stop_status
2024-02-01 18:14:10,385 DEBUG   SenderThread:239784 [sender.py:send_request():409] send_request: stop_status
2024-02-01 18:14:10,408 DEBUG   HandlerThread:239784 [handler.py:handle_request():146] handle_request: internal_messages
2024-02-01 18:14:12,678 INFO    Thread-12 :239784 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_175850-i93q0p12/files/output.log
2024-02-01 18:14:14,071 DEBUG   HandlerThread:239784 [handler.py:handle_request():146] handle_request: status_report
2024-02-01 18:14:16,683 INFO    Thread-12 :239784 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_175850-i93q0p12/files/output.log
2024-02-01 18:14:18,685 INFO    Thread-12 :239784 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_175850-i93q0p12/files/output.log
2024-02-01 18:14:19,651 DEBUG   HandlerThread:239784 [handler.py:handle_request():146] handle_request: status_report
2024-02-01 18:14:20,619 DEBUG   SenderThread:239784 [sender.py:send():382] send: stats
2024-02-01 18:14:22,690 INFO    Thread-12 :239784 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_175850-i93q0p12/files/output.log
2024-02-01 18:14:24,814 DEBUG   HandlerThread:239784 [handler.py:handle_request():146] handle_request: status_report
2024-02-01 18:14:25,385 DEBUG   HandlerThread:239784 [handler.py:handle_request():146] handle_request: stop_status
2024-02-01 18:14:25,385 DEBUG   SenderThread:239784 [sender.py:send_request():409] send_request: stop_status
2024-02-01 18:14:25,408 DEBUG   HandlerThread:239784 [handler.py:handle_request():146] handle_request: internal_messages
2024-02-01 18:14:26,695 INFO    Thread-12 :239784 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_175850-i93q0p12/files/output.log
2024-02-01 18:14:30,407 DEBUG   HandlerThread:239784 [handler.py:handle_request():146] handle_request: status_report
2024-02-01 18:14:30,700 INFO    Thread-12 :239784 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_175850-i93q0p12/files/output.log
2024-02-01 18:14:34,705 INFO    Thread-12 :239784 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_175850-i93q0p12/files/output.log
2024-02-01 18:14:35,707 DEBUG   HandlerThread:239784 [handler.py:handle_request():146] handle_request: status_report
2024-02-01 18:14:38,711 INFO    Thread-12 :239784 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_175850-i93q0p12/files/output.log
2024-02-01 18:14:39,365 DEBUG   HandlerThread:239784 [handler.py:handle_request():146] handle_request: partial_history
2024-02-01 18:14:39,366 DEBUG   SenderThread:239784 [sender.py:send():382] send: history
2024-02-01 18:14:39,366 DEBUG   SenderThread:239784 [sender.py:send_request():409] send_request: summary_record
2024-02-01 18:14:39,368 INFO    SenderThread:239784 [sender.py:_save_file():1392] saving file wandb-summary.json with policy end
2024-02-01 18:14:39,713 INFO    Thread-12 :239784 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_175850-i93q0p12/files/wandb-summary.json
2024-02-01 18:14:40,385 DEBUG   HandlerThread:239784 [handler.py:handle_request():146] handle_request: stop_status
2024-02-01 18:14:40,385 DEBUG   SenderThread:239784 [sender.py:send_request():409] send_request: stop_status
2024-02-01 18:14:40,408 DEBUG   HandlerThread:239784 [handler.py:handle_request():146] handle_request: internal_messages
2024-02-01 18:14:40,715 INFO    Thread-12 :239784 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_175850-i93q0p12/files/output.log
2024-02-01 18:14:41,513 DEBUG   HandlerThread:239784 [handler.py:handle_request():146] handle_request: status_report
2024-02-01 18:14:44,720 INFO    Thread-12 :239784 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_175850-i93q0p12/files/output.log
2024-02-01 18:14:46,636 DEBUG   HandlerThread:239784 [handler.py:handle_request():146] handle_request: status_report
2024-02-01 18:14:48,724 INFO    Thread-12 :239784 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_175850-i93q0p12/files/output.log
2024-02-01 18:14:50,621 DEBUG   SenderThread:239784 [sender.py:send():382] send: stats
2024-02-01 18:14:51,729 INFO    Thread-12 :239784 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_175850-i93q0p12/files/output.log
2024-02-01 18:14:52,622 DEBUG   HandlerThread:239784 [handler.py:handle_request():146] handle_request: status_report
2024-02-01 18:14:55,385 DEBUG   HandlerThread:239784 [handler.py:handle_request():146] handle_request: stop_status
2024-02-01 18:14:55,386 DEBUG   SenderThread:239784 [sender.py:send_request():409] send_request: stop_status
2024-02-01 18:14:55,408 DEBUG   HandlerThread:239784 [handler.py:handle_request():146] handle_request: internal_messages
2024-02-01 18:14:55,734 INFO    Thread-12 :239784 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_175850-i93q0p12/files/output.log
2024-02-01 18:14:57,736 INFO    Thread-12 :239784 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_175850-i93q0p12/files/output.log
2024-02-01 18:14:58,544 DEBUG   HandlerThread:239784 [handler.py:handle_request():146] handle_request: status_report
2024-02-01 18:15:01,741 INFO    Thread-12 :239784 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_175850-i93q0p12/files/output.log
2024-02-01 18:15:04,165 DEBUG   HandlerThread:239784 [handler.py:handle_request():146] handle_request: status_report
2024-02-01 18:15:05,746 INFO    Thread-12 :239784 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_175850-i93q0p12/files/output.log
2024-02-01 18:15:09,426 DEBUG   HandlerThread:239784 [handler.py:handle_request():146] handle_request: status_report
2024-02-01 18:15:09,751 INFO    Thread-12 :239784 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_175850-i93q0p12/files/output.log
2024-02-01 18:15:10,385 DEBUG   HandlerThread:239784 [handler.py:handle_request():146] handle_request: stop_status
2024-02-01 18:15:10,386 DEBUG   SenderThread:239784 [sender.py:send_request():409] send_request: stop_status
2024-02-01 18:15:10,408 DEBUG   HandlerThread:239784 [handler.py:handle_request():146] handle_request: internal_messages
2024-02-01 18:15:13,756 INFO    Thread-12 :239784 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_175850-i93q0p12/files/output.log
2024-02-01 18:15:15,048 DEBUG   HandlerThread:239784 [handler.py:handle_request():146] handle_request: status_report
2024-02-01 18:15:15,680 DEBUG   HandlerThread:239784 [handler.py:handle_request():146] handle_request: partial_history
2024-02-01 18:15:15,681 DEBUG   SenderThread:239784 [sender.py:send():382] send: history
2024-02-01 18:15:15,681 DEBUG   SenderThread:239784 [sender.py:send_request():409] send_request: summary_record
2024-02-01 18:15:15,683 INFO    SenderThread:239784 [sender.py:_save_file():1392] saving file wandb-summary.json with policy end
2024-02-01 18:15:15,760 INFO    Thread-12 :239784 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_175850-i93q0p12/files/output.log
2024-02-01 18:15:15,760 INFO    Thread-12 :239784 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_175850-i93q0p12/files/wandb-summary.json
2024-02-01 18:15:16,761 INFO    Thread-12 :239784 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_175850-i93q0p12/files/output.log
2024-02-01 18:15:19,765 INFO    Thread-12 :239784 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_175850-i93q0p12/files/output.log
2024-02-01 18:15:20,298 DEBUG   HandlerThread:239784 [handler.py:handle_request():146] handle_request: status_report
2024-02-01 18:15:20,623 DEBUG   SenderThread:239784 [sender.py:send():382] send: stats
2024-02-01 18:15:23,770 INFO    Thread-12 :239784 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_175850-i93q0p12/files/output.log
2024-02-01 18:15:25,385 DEBUG   HandlerThread:239784 [handler.py:handle_request():146] handle_request: stop_status
2024-02-01 18:15:25,386 DEBUG   SenderThread:239784 [sender.py:send_request():409] send_request: stop_status
2024-02-01 18:15:25,409 DEBUG   HandlerThread:239784 [handler.py:handle_request():146] handle_request: internal_messages
2024-02-01 18:15:25,508 DEBUG   HandlerThread:239784 [handler.py:handle_request():146] handle_request: status_report
2024-02-01 18:15:27,775 INFO    Thread-12 :239784 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_175850-i93q0p12/files/output.log
2024-02-01 18:15:29,778 INFO    Thread-12 :239784 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_175850-i93q0p12/files/output.log
2024-02-01 18:15:30,539 DEBUG   HandlerThread:239784 [handler.py:handle_request():146] handle_request: status_report
2024-02-01 18:15:31,781 INFO    Thread-12 :239784 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_175850-i93q0p12/files/output.log
2024-02-01 18:15:33,783 INFO    Thread-12 :239784 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_175850-i93q0p12/files/output.log
2024-02-01 18:15:35,570 DEBUG   HandlerThread:239784 [handler.py:handle_request():146] handle_request: status_report
2024-02-01 18:15:35,786 INFO    Thread-12 :239784 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_175850-i93q0p12/files/output.log
2024-02-01 18:15:37,789 INFO    Thread-12 :239784 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_175850-i93q0p12/files/output.log
2024-02-01 18:15:39,791 INFO    Thread-12 :239784 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_175850-i93q0p12/files/output.log
2024-02-01 18:15:40,385 DEBUG   HandlerThread:239784 [handler.py:handle_request():146] handle_request: stop_status
2024-02-01 18:15:40,386 DEBUG   SenderThread:239784 [sender.py:send_request():409] send_request: stop_status
2024-02-01 18:15:40,409 DEBUG   HandlerThread:239784 [handler.py:handle_request():146] handle_request: internal_messages
2024-02-01 18:15:40,590 DEBUG   HandlerThread:239784 [handler.py:handle_request():146] handle_request: status_report
2024-02-01 18:15:41,794 INFO    Thread-12 :239784 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_175850-i93q0p12/files/output.log
2024-02-01 18:15:43,797 INFO    Thread-12 :239784 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_175850-i93q0p12/files/output.log
2024-02-01 18:15:45,601 DEBUG   HandlerThread:239784 [handler.py:handle_request():146] handle_request: status_report
2024-02-01 18:15:45,799 INFO    Thread-12 :239784 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_175850-i93q0p12/files/output.log
2024-02-01 18:15:47,802 INFO    Thread-12 :239784 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_175850-i93q0p12/files/output.log
2024-02-01 18:15:49,804 INFO    Thread-12 :239784 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_175850-i93q0p12/files/output.log
2024-02-01 18:15:50,626 DEBUG   SenderThread:239784 [sender.py:send():382] send: stats
2024-02-01 18:15:50,627 DEBUG   HandlerThread:239784 [handler.py:handle_request():146] handle_request: status_report
2024-02-01 18:15:51,807 INFO    Thread-12 :239784 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_175850-i93q0p12/files/output.log
2024-02-01 18:15:53,810 INFO    Thread-12 :239784 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_175850-i93q0p12/files/output.log
2024-02-01 18:15:55,386 DEBUG   HandlerThread:239784 [handler.py:handle_request():146] handle_request: stop_status
2024-02-01 18:15:55,386 DEBUG   SenderThread:239784 [sender.py:send_request():409] send_request: stop_status
2024-02-01 18:15:55,409 DEBUG   HandlerThread:239784 [handler.py:handle_request():146] handle_request: internal_messages
2024-02-01 18:15:55,640 DEBUG   HandlerThread:239784 [handler.py:handle_request():146] handle_request: status_report
2024-02-01 18:15:55,812 INFO    Thread-12 :239784 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_175850-i93q0p12/files/output.log
2024-02-01 18:15:57,215 DEBUG   HandlerThread:239784 [handler.py:handle_request():146] handle_request: partial_history
2024-02-01 18:15:57,219 DEBUG   SenderThread:239784 [sender.py:send():382] send: metric
2024-02-01 18:15:57,220 DEBUG   SenderThread:239784 [sender.py:send():382] send: metric
2024-02-01 18:15:57,220 DEBUG   SenderThread:239784 [sender.py:send():382] send: metric
2024-02-01 18:15:57,221 DEBUG   SenderThread:239784 [sender.py:send():382] send: metric
2024-02-01 18:15:57,221 DEBUG   SenderThread:239784 [sender.py:send():382] send: history
2024-02-01 18:15:57,221 DEBUG   SenderThread:239784 [sender.py:send_request():409] send_request: summary_record
2024-02-01 18:15:57,223 INFO    SenderThread:239784 [sender.py:_save_file():1392] saving file wandb-summary.json with policy end
2024-02-01 18:15:57,816 INFO    Thread-12 :239784 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_175850-i93q0p12/files/output.log
2024-02-01 18:15:57,816 INFO    Thread-12 :239784 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_175850-i93q0p12/files/wandb-summary.json
2024-02-01 18:15:58,817 INFO    Thread-12 :239784 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_175850-i93q0p12/files/output.log
2024-02-01 18:15:59,819 INFO    Thread-12 :239784 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_175850-i93q0p12/files/output.log
2024-02-01 18:16:01,327 DEBUG   HandlerThread:239784 [handler.py:handle_request():146] handle_request: status_report
2024-02-01 18:16:01,821 INFO    Thread-12 :239784 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_175850-i93q0p12/files/output.log
2024-02-01 18:16:02,823 INFO    Thread-12 :239784 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_175850-i93q0p12/files/output.log
2024-02-01 18:16:06,828 INFO    Thread-12 :239784 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_175850-i93q0p12/files/output.log
2024-02-01 18:16:06,938 DEBUG   HandlerThread:239784 [handler.py:handle_request():146] handle_request: status_report
2024-02-01 18:16:09,832 INFO    Thread-12 :239784 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_175850-i93q0p12/files/output.log
2024-02-01 18:16:10,386 DEBUG   HandlerThread:239784 [handler.py:handle_request():146] handle_request: stop_status
2024-02-01 18:16:10,386 DEBUG   SenderThread:239784 [sender.py:send_request():409] send_request: stop_status
2024-02-01 18:16:10,409 DEBUG   HandlerThread:239784 [handler.py:handle_request():146] handle_request: internal_messages
2024-02-01 18:16:12,143 DEBUG   HandlerThread:239784 [handler.py:handle_request():146] handle_request: status_report
2024-02-01 18:16:12,837 INFO    Thread-12 :239784 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_175850-i93q0p12/files/config.yaml
2024-02-01 18:16:13,838 INFO    Thread-12 :239784 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_175850-i93q0p12/files/output.log
2024-02-01 18:16:15,841 INFO    Thread-12 :239784 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_175850-i93q0p12/files/output.log
2024-02-01 18:16:17,724 DEBUG   HandlerThread:239784 [handler.py:handle_request():146] handle_request: status_report
2024-02-01 18:16:19,846 INFO    Thread-12 :239784 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_175850-i93q0p12/files/output.log
2024-02-01 18:16:20,627 DEBUG   SenderThread:239784 [sender.py:send():382] send: stats
2024-02-01 18:16:22,891 DEBUG   HandlerThread:239784 [handler.py:handle_request():146] handle_request: status_report
2024-02-01 18:16:23,851 INFO    Thread-12 :239784 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_175850-i93q0p12/files/output.log
2024-02-01 18:16:25,386 DEBUG   HandlerThread:239784 [handler.py:handle_request():146] handle_request: stop_status
2024-02-01 18:16:25,386 DEBUG   SenderThread:239784 [sender.py:send_request():409] send_request: stop_status
2024-02-01 18:16:25,409 DEBUG   HandlerThread:239784 [handler.py:handle_request():146] handle_request: internal_messages
2024-02-01 18:16:27,856 INFO    Thread-12 :239784 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_175850-i93q0p12/files/output.log
2024-02-01 18:16:28,490 DEBUG   HandlerThread:239784 [handler.py:handle_request():146] handle_request: status_report
2024-02-01 18:16:31,861 INFO    Thread-12 :239784 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_175850-i93q0p12/files/output.log
2024-02-01 18:16:33,645 DEBUG   HandlerThread:239784 [handler.py:handle_request():146] handle_request: status_report
2024-02-01 18:16:33,646 DEBUG   HandlerThread:239784 [handler.py:handle_request():146] handle_request: partial_history
2024-02-01 18:16:33,647 DEBUG   SenderThread:239784 [sender.py:send():382] send: history
2024-02-01 18:16:33,647 DEBUG   SenderThread:239784 [sender.py:send_request():409] send_request: summary_record
2024-02-01 18:16:33,650 INFO    SenderThread:239784 [sender.py:_save_file():1392] saving file wandb-summary.json with policy end
2024-02-01 18:16:33,865 INFO    Thread-12 :239784 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_175850-i93q0p12/files/output.log
2024-02-01 18:16:33,865 INFO    Thread-12 :239784 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_175850-i93q0p12/files/wandb-summary.json
2024-02-01 18:16:34,866 INFO    Thread-12 :239784 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_175850-i93q0p12/files/output.log
2024-02-01 18:16:37,870 INFO    Thread-12 :239784 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_175850-i93q0p12/files/output.log
2024-02-01 18:16:39,229 DEBUG   HandlerThread:239784 [handler.py:handle_request():146] handle_request: status_report
2024-02-01 18:16:40,386 DEBUG   HandlerThread:239784 [handler.py:handle_request():146] handle_request: stop_status
2024-02-01 18:16:40,386 DEBUG   SenderThread:239784 [sender.py:send_request():409] send_request: stop_status
2024-02-01 18:16:40,409 DEBUG   HandlerThread:239784 [handler.py:handle_request():146] handle_request: internal_messages
2024-02-01 18:16:41,875 INFO    Thread-12 :239784 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_175850-i93q0p12/files/output.log
2024-02-01 18:16:44,390 DEBUG   HandlerThread:239784 [handler.py:handle_request():146] handle_request: status_report
2024-02-01 18:16:45,881 INFO    Thread-12 :239784 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_175850-i93q0p12/files/output.log
2024-02-01 18:16:49,886 INFO    Thread-12 :239784 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_175850-i93q0p12/files/output.log
2024-02-01 18:16:49,982 DEBUG   HandlerThread:239784 [handler.py:handle_request():146] handle_request: status_report
2024-02-01 18:16:50,629 DEBUG   SenderThread:239784 [sender.py:send():382] send: stats
2024-02-01 18:16:51,888 INFO    Thread-12 :239784 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_175850-i93q0p12/files/output.log
2024-02-01 18:16:55,146 DEBUG   HandlerThread:239784 [handler.py:handle_request():146] handle_request: status_report
2024-02-01 18:16:55,386 DEBUG   HandlerThread:239784 [handler.py:handle_request():146] handle_request: stop_status
2024-02-01 18:16:55,386 DEBUG   SenderThread:239784 [sender.py:send_request():409] send_request: stop_status
2024-02-01 18:16:55,409 DEBUG   HandlerThread:239784 [handler.py:handle_request():146] handle_request: internal_messages
2024-02-01 18:16:55,893 INFO    Thread-12 :239784 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_175850-i93q0p12/files/output.log
2024-02-01 18:16:59,899 INFO    Thread-12 :239784 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_175850-i93q0p12/files/output.log
2024-02-01 18:17:00,735 DEBUG   HandlerThread:239784 [handler.py:handle_request():146] handle_request: status_report
2024-02-01 18:17:03,904 INFO    Thread-12 :239784 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_175850-i93q0p12/files/output.log
2024-02-01 18:17:05,905 DEBUG   HandlerThread:239784 [handler.py:handle_request():146] handle_request: status_report
2024-02-01 18:17:07,909 INFO    Thread-12 :239784 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_175850-i93q0p12/files/output.log
2024-02-01 18:17:09,494 DEBUG   HandlerThread:239784 [handler.py:handle_request():146] handle_request: partial_history
2024-02-01 18:17:09,495 DEBUG   SenderThread:239784 [sender.py:send():382] send: history
2024-02-01 18:17:09,495 DEBUG   SenderThread:239784 [sender.py:send_request():409] send_request: summary_record
2024-02-01 18:17:09,497 INFO    SenderThread:239784 [sender.py:_save_file():1392] saving file wandb-summary.json with policy end
2024-02-01 18:17:09,913 INFO    Thread-12 :239784 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_175850-i93q0p12/files/output.log
2024-02-01 18:17:09,913 INFO    Thread-12 :239784 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_175850-i93q0p12/files/wandb-summary.json
2024-02-01 18:17:10,386 DEBUG   HandlerThread:239784 [handler.py:handle_request():146] handle_request: stop_status
2024-02-01 18:17:10,387 DEBUG   SenderThread:239784 [sender.py:send_request():409] send_request: stop_status
2024-02-01 18:17:10,409 DEBUG   HandlerThread:239784 [handler.py:handle_request():146] handle_request: internal_messages
2024-02-01 18:17:10,914 INFO    Thread-12 :239784 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_175850-i93q0p12/files/output.log
2024-02-01 18:17:11,515 DEBUG   HandlerThread:239784 [handler.py:handle_request():146] handle_request: status_report
2024-02-01 18:17:13,918 INFO    Thread-12 :239784 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_175850-i93q0p12/files/output.log