| 2024-03-08 13:49:01,420 INFO StreamThr :37167 [internal.py:wandb_internal():86] W&B internal server running at pid: 37167, started at: 2024-03-08 13:49:01.420105 | |
| 2024-03-08 13:49:01,422 DEBUG HandlerThread:37167 [handler.py:handle_request():146] handle_request: status | |
| 2024-03-08 13:49:06,425 DEBUG HandlerThread:37167 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-08 13:49:06,425 INFO WriterThread:37167 [datastore.py:open_for_write():87] open: /content/wandb/run-20240308_134901-hlg2f18u/run-hlg2f18u.wandb | |
| 2024-03-08 13:49:06,451 DEBUG SenderThread:37167 [sender.py:send():378] send: header | |
| 2024-03-08 13:49:06,454 DEBUG SenderThread:37167 [sender.py:send():378] send: run | |
| 2024-03-08 13:49:06,939 INFO SenderThread:37167 [dir_watcher.py:__init__():211] watching files in: /content/wandb/run-20240308_134901-hlg2f18u/files | |
| 2024-03-08 13:49:06,939 INFO SenderThread:37167 [sender.py:_start_run_threads():1099] run started: hlg2f18u with start time 1709905741.420297 | |
| 2024-03-08 13:49:06,942 DEBUG HandlerThread:37167 [handler.py:handle_request():146] handle_request: check_version | |
| 2024-03-08 13:49:06,942 DEBUG SenderThread:37167 [sender.py:send_request():405] send_request: check_version | |
| 2024-03-08 13:49:07,153 DEBUG HandlerThread:37167 [handler.py:handle_request():146] handle_request: run_start | |
| 2024-03-08 13:49:07,163 DEBUG HandlerThread:37167 [system_info.py:__init__():26] System info init | |
| 2024-03-08 13:49:07,163 DEBUG HandlerThread:37167 [system_info.py:__init__():41] System info init done | |
| 2024-03-08 13:49:07,163 INFO HandlerThread:37167 [system_monitor.py:start():194] Starting system monitor | |
| 2024-03-08 13:49:07,164 INFO SystemMonitor:37167 [system_monitor.py:_start():158] Starting system asset monitoring threads | |
| 2024-03-08 13:49:07,164 INFO HandlerThread:37167 [system_monitor.py:probe():214] Collecting system info | |
| 2024-03-08 13:49:07,164 INFO SystemMonitor:37167 [interfaces.py:start():190] Started cpu monitoring | |
| 2024-03-08 13:49:07,165 INFO SystemMonitor:37167 [interfaces.py:start():190] Started disk monitoring | |
| 2024-03-08 13:49:07,166 INFO SystemMonitor:37167 [interfaces.py:start():190] Started gpu monitoring | |
| 2024-03-08 13:49:07,167 INFO SystemMonitor:37167 [interfaces.py:start():190] Started memory monitoring | |
| 2024-03-08 13:49:07,168 INFO SystemMonitor:37167 [interfaces.py:start():190] Started network monitoring | |
| 2024-03-08 13:49:07,172 DEBUG HandlerThread:37167 [system_info.py:probe():150] Probing system | |
| 2024-03-08 13:49:07,174 DEBUG HandlerThread:37167 [gitlib.py:_init_repo():56] git repository is invalid | |
| 2024-03-08 13:49:07,174 DEBUG HandlerThread:37167 [system_info.py:probe():198] Probing system done | |
| 2024-03-08 13:49:07,174 DEBUG HandlerThread:37167 [system_monitor.py:probe():223] {'os': 'Linux-6.1.58+-x86_64-with-glibc2.35', 'python': '3.10.12', 'heartbeatAt': '2024-03-08T13:49:07.172649', 'startedAt': '2024-03-08T13:49:01.413484', 'docker': None, 'cuda': None, 'args': (), 'state': 'running', 'program': 'Smart_Agriculture_Falcon.ipynb', 'codePathLocal': None, 'colab': 'https://colab.research.google.com/notebook#fileId=1kv2VO7q0c8xX_DQt5NERLeJJZtHW6zHO', 'host': '4997a2709b0f', 'username': 'root', 'executable': '/usr/bin/python3', 'cpu_count': 4, 'cpu_count_logical': 8, 'cpu_freq': {'current': 2199.998, 'min': 0.0, 'max': 0.0}, 'cpu_freq_per_core': [{'current': 2199.998, 'min': 0.0, 'max': 0.0}, {'current': 2199.998, 'min': 0.0, 'max': 0.0}, {'current': 2199.998, 'min': 0.0, 'max': 0.0}, {'current': 2199.998, 'min': 0.0, 'max': 0.0}, {'current': 2199.998, 'min': 0.0, 'max': 0.0}, {'current': 2199.998, 'min': 0.0, 'max': 0.0}, {'current': 2199.998, 'min': 0.0, 'max': 0.0}, {'current': 2199.998, 'min': 0.0, 'max': 0.0}], 'disk': {'/': {'total': 201.22513961791992, 'used': 58.57290267944336}}, 'gpu': 'Tesla T4', 'gpu_count': 1, 'gpu_devices': [{'name': 'Tesla T4', 'memory_total': 16106127360}], 'memory': {'total': 50.993690490722656}} | |
| 2024-03-08 13:49:07,174 INFO HandlerThread:37167 [system_monitor.py:probe():224] Finished collecting system info | |
| 2024-03-08 13:49:07,174 INFO HandlerThread:37167 [system_monitor.py:probe():227] Publishing system info | |
| 2024-03-08 13:49:07,174 INFO HandlerThread:37167 [system_monitor.py:probe():229] Finished publishing system info | |
| 2024-03-08 13:49:07,186 DEBUG SenderThread:37167 [sender.py:send():378] send: files | |
| 2024-03-08 13:49:07,186 INFO SenderThread:37167 [sender.py:_save_file():1365] saving file wandb-metadata.json with policy now | |
| 2024-03-08 13:49:07,646 DEBUG HandlerThread:37167 [handler.py:handle_request():146] handle_request: python_packages | |
| 2024-03-08 13:49:07,647 DEBUG SenderThread:37167 [sender.py:send_request():405] send_request: python_packages | |
| 2024-03-08 13:49:07,686 DEBUG HandlerThread:37167 [handler.py:handle_request():146] handle_request: internal_messages | |
| 2024-03-08 13:49:07,687 DEBUG HandlerThread:37167 [handler.py:handle_request():146] handle_request: stop_status | |
| 2024-03-08 13:49:07,688 DEBUG SenderThread:37167 [sender.py:send_request():405] send_request: stop_status | |
| 2024-03-08 13:49:07,940 INFO Thread-12 :37167 [dir_watcher.py:_on_file_created():271] file/dir created: /content/wandb/run-20240308_134901-hlg2f18u/files/wandb-metadata.json | |
| 2024-03-08 13:49:07,941 INFO Thread-12 :37167 [dir_watcher.py:_on_file_created():271] file/dir created: /content/wandb/run-20240308_134901-hlg2f18u/files/requirements.txt | |
| 2024-03-08 13:49:08,065 INFO wandb-upload_0:37167 [upload_job.py:push():131] Uploaded file /tmp/tmpgtstjuiqwandb/o7wr7mlg-wandb-metadata.json | |
| 2024-03-08 13:49:08,292 DEBUG SenderThread:37167 [sender.py:send():378] send: telemetry | |
| 2024-03-08 13:49:08,292 DEBUG SenderThread:37167 [sender.py:send():378] send: config | |
| 2024-03-08 13:49:08,293 DEBUG SenderThread:37167 [sender.py:send():378] send: metric | |
| 2024-03-08 13:49:08,293 DEBUG SenderThread:37167 [sender.py:send():378] send: telemetry | |
| 2024-03-08 13:49:08,293 DEBUG SenderThread:37167 [sender.py:send():378] send: metric | |
| 2024-03-08 13:49:08,293 WARNING SenderThread:37167 [sender.py:send_metric():1316] Seen metric with glob (shouldn't happen) | |
| 2024-03-08 13:49:08,293 DEBUG SenderThread:37167 [sender.py:send():378] send: telemetry | |
| 2024-03-08 13:49:12,294 DEBUG HandlerThread:37167 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-08 13:49:17,295 DEBUG HandlerThread:37167 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-08 13:49:22,296 DEBUG HandlerThread:37167 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-08 13:49:22,647 DEBUG HandlerThread:37167 [handler.py:handle_request():146] handle_request: stop_status | |
| 2024-03-08 13:49:22,647 DEBUG SenderThread:37167 [sender.py:send_request():405] send_request: stop_status | |
| 2024-03-08 13:49:22,688 DEBUG HandlerThread:37167 [handler.py:handle_request():146] handle_request: internal_messages | |
| 2024-03-08 13:49:27,889 DEBUG HandlerThread:37167 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-08 13:49:32,894 DEBUG HandlerThread:37167 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-08 13:49:33,949 INFO Thread-12 :37167 [dir_watcher.py:_on_file_modified():288] file/dir modified: /content/wandb/run-20240308_134901-hlg2f18u/files/config.yaml | |
| 2024-03-08 13:49:37,647 DEBUG HandlerThread:37167 [handler.py:handle_request():146] handle_request: stop_status | |
| 2024-03-08 13:49:37,648 DEBUG SenderThread:37167 [sender.py:send_request():405] send_request: stop_status | |
| 2024-03-08 13:49:37,688 DEBUG HandlerThread:37167 [handler.py:handle_request():146] handle_request: internal_messages | |
| 2024-03-08 13:49:38,897 DEBUG HandlerThread:37167 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-08 13:49:43,898 DEBUG HandlerThread:37167 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-08 13:49:48,899 DEBUG HandlerThread:37167 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-08 13:49:52,647 DEBUG HandlerThread:37167 [handler.py:handle_request():146] handle_request: stop_status | |
| 2024-03-08 13:49:52,648 DEBUG SenderThread:37167 [sender.py:send_request():405] send_request: stop_status | |
| 2024-03-08 13:49:52,688 DEBUG HandlerThread:37167 [handler.py:handle_request():146] handle_request: internal_messages | |
| 2024-03-08 13:49:54,851 DEBUG HandlerThread:37167 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-08 13:49:59,852 DEBUG HandlerThread:37167 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-08 13:50:04,853 DEBUG HandlerThread:37167 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-08 13:50:07,168 DEBUG SystemMonitor:37167 [system_monitor.py:_start():172] Starting system metrics aggregation loop | |
| 2024-03-08 13:50:07,169 DEBUG SenderThread:37167 [sender.py:send():378] send: stats | |
| 2024-03-08 13:50:07,647 DEBUG HandlerThread:37167 [handler.py:handle_request():146] handle_request: stop_status | |
| 2024-03-08 13:50:07,648 DEBUG SenderThread:37167 [sender.py:send_request():405] send_request: stop_status | |
| 2024-03-08 13:50:07,688 DEBUG HandlerThread:37167 [handler.py:handle_request():146] handle_request: internal_messages | |
| 2024-03-08 13:50:09,898 DEBUG HandlerThread:37167 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-08 13:50:14,899 DEBUG HandlerThread:37167 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-08 13:50:19,900 DEBUG HandlerThread:37167 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-08 13:50:22,648 DEBUG HandlerThread:37167 [handler.py:handle_request():146] handle_request: stop_status | |
| 2024-03-08 13:50:22,648 DEBUG SenderThread:37167 [sender.py:send_request():405] send_request: stop_status | |
| 2024-03-08 13:50:22,688 DEBUG HandlerThread:37167 [handler.py:handle_request():146] handle_request: internal_messages | |
| 2024-03-08 13:50:25,835 DEBUG HandlerThread:37167 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-08 13:50:30,836 DEBUG HandlerThread:37167 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-08 13:50:30,851 DEBUG HandlerThread:37167 [handler.py:handle_request():146] handle_request: partial_history | |
| 2024-03-08 13:50:30,853 DEBUG SenderThread:37167 [sender.py:send():378] send: metric | |
| 2024-03-08 13:50:30,854 DEBUG SenderThread:37167 [sender.py:send():378] send: metric | |
| 2024-03-08 13:50:30,854 DEBUG SenderThread:37167 [sender.py:send():378] send: metric | |
| 2024-03-08 13:50:30,854 DEBUG SenderThread:37167 [sender.py:send():378] send: metric | |
| 2024-03-08 13:50:30,854 DEBUG SenderThread:37167 [sender.py:send():378] send: history | |
| 2024-03-08 13:50:30,854 DEBUG SenderThread:37167 [sender.py:send_request():405] send_request: summary_record | |
| 2024-03-08 13:50:30,856 INFO SenderThread:37167 [sender.py:_save_file():1365] saving file wandb-summary.json with policy end | |
| 2024-03-08 13:50:30,969 INFO Thread-12 :37167 [dir_watcher.py:_on_file_created():271] file/dir created: /content/wandb/run-20240308_134901-hlg2f18u/files/output.log | |
| 2024-03-08 13:50:30,970 INFO Thread-12 :37167 [dir_watcher.py:_on_file_created():271] file/dir created: /content/wandb/run-20240308_134901-hlg2f18u/files/wandb-summary.json | |
| 2024-03-08 13:50:32,971 INFO Thread-12 :37167 [dir_watcher.py:_on_file_modified():288] file/dir modified: /content/wandb/run-20240308_134901-hlg2f18u/files/output.log | |
| 2024-03-08 13:50:35,899 DEBUG HandlerThread:37167 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-08 13:50:36,972 INFO Thread-12 :37167 [dir_watcher.py:_on_file_modified():288] file/dir modified: /content/wandb/run-20240308_134901-hlg2f18u/files/config.yaml | |
| 2024-03-08 13:50:37,170 DEBUG SenderThread:37167 [sender.py:send():378] send: stats | |
| 2024-03-08 13:50:37,648 DEBUG HandlerThread:37167 [handler.py:handle_request():146] handle_request: stop_status | |
| 2024-03-08 13:50:37,648 DEBUG HandlerThread:37167 [handler.py:handle_request():146] handle_request: internal_messages | |
| 2024-03-08 13:50:37,649 DEBUG SenderThread:37167 [sender.py:send_request():405] send_request: stop_status | |
| 2024-03-08 13:50:41,848 DEBUG HandlerThread:37167 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-08 13:50:46,849 DEBUG HandlerThread:37167 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-08 13:50:51,850 DEBUG HandlerThread:37167 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-08 13:50:52,648 DEBUG HandlerThread:37167 [handler.py:handle_request():146] handle_request: stop_status | |
| 2024-03-08 13:50:52,648 DEBUG SenderThread:37167 [sender.py:send_request():405] send_request: stop_status | |
| 2024-03-08 13:50:52,689 DEBUG HandlerThread:37167 [handler.py:handle_request():146] handle_request: internal_messages | |
| 2024-03-08 13:50:56,936 DEBUG HandlerThread:37167 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-08 13:51:01,937 DEBUG HandlerThread:37167 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-08 13:51:06,939 DEBUG HandlerThread:37167 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-08 13:51:07,171 DEBUG SenderThread:37167 [sender.py:send():378] send: stats | |
| 2024-03-08 13:51:07,648 DEBUG HandlerThread:37167 [handler.py:handle_request():146] handle_request: stop_status | |
| 2024-03-08 13:51:07,648 DEBUG SenderThread:37167 [sender.py:send_request():405] send_request: stop_status | |
| 2024-03-08 13:51:07,689 DEBUG HandlerThread:37167 [handler.py:handle_request():146] handle_request: internal_messages | |
| 2024-03-08 13:51:11,951 DEBUG HandlerThread:37167 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-08 13:51:16,952 DEBUG HandlerThread:37167 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-08 13:51:21,953 DEBUG HandlerThread:37167 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-08 13:51:22,648 DEBUG HandlerThread:37167 [handler.py:handle_request():146] handle_request: stop_status | |
| 2024-03-08 13:51:22,649 DEBUG SenderThread:37167 [sender.py:send_request():405] send_request: stop_status | |
| 2024-03-08 13:51:22,689 DEBUG HandlerThread:37167 [handler.py:handle_request():146] handle_request: internal_messages | |
| 2024-03-08 13:51:27,772 DEBUG HandlerThread:37167 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-08 13:51:32,773 DEBUG HandlerThread:37167 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-08 13:51:37,172 DEBUG SenderThread:37167 [sender.py:send():378] send: stats | |
| 2024-03-08 13:51:37,650 DEBUG HandlerThread:37167 [handler.py:handle_request():146] handle_request: stop_status | |
| 2024-03-08 13:51:37,650 DEBUG HandlerThread:37167 [handler.py:handle_request():146] handle_request: internal_messages | |
| 2024-03-08 13:51:37,651 DEBUG SenderThread:37167 [sender.py:send_request():405] send_request: stop_status | |
| 2024-03-08 13:51:37,939 DEBUG HandlerThread:37167 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-08 13:51:42,940 DEBUG HandlerThread:37167 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-08 13:51:47,942 DEBUG HandlerThread:37167 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-08 13:51:52,649 DEBUG HandlerThread:37167 [handler.py:handle_request():146] handle_request: stop_status | |
| 2024-03-08 13:51:52,649 DEBUG SenderThread:37167 [sender.py:send_request():405] send_request: stop_status | |
| 2024-03-08 13:51:52,689 DEBUG HandlerThread:37167 [handler.py:handle_request():146] handle_request: internal_messages | |
| 2024-03-08 13:51:54,693 DEBUG HandlerThread:37167 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-08 13:51:59,617 DEBUG HandlerThread:37167 [handler.py:handle_request():146] handle_request: partial_history | |
| 2024-03-08 13:51:59,619 DEBUG SenderThread:37167 [sender.py:send():378] send: history | |
| 2024-03-08 13:51:59,619 DEBUG SenderThread:37167 [sender.py:send_request():405] send_request: summary_record | |
| 2024-03-08 13:51:59,621 INFO SenderThread:37167 [sender.py:_save_file():1365] saving file wandb-summary.json with policy end | |
| 2024-03-08 13:52:00,002 INFO Thread-12 :37167 [dir_watcher.py:_on_file_modified():288] file/dir modified: /content/wandb/run-20240308_134901-hlg2f18u/files/wandb-summary.json | |
| 2024-03-08 13:52:00,659 DEBUG HandlerThread:37167 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-08 13:52:01,002 INFO Thread-12 :37167 [dir_watcher.py:_on_file_modified():288] file/dir modified: /content/wandb/run-20240308_134901-hlg2f18u/files/output.log | |
| 2024-03-08 13:52:05,660 DEBUG HandlerThread:37167 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-08 13:52:07,173 DEBUG SenderThread:37167 [sender.py:send():378] send: stats | |
| 2024-03-08 13:52:07,649 DEBUG HandlerThread:37167 [handler.py:handle_request():146] handle_request: stop_status | |
| 2024-03-08 13:52:07,650 DEBUG SenderThread:37167 [sender.py:send_request():405] send_request: stop_status | |
| 2024-03-08 13:52:07,651 DEBUG HandlerThread:37167 [handler.py:handle_request():146] handle_request: internal_messages | |
| 2024-03-08 13:52:10,920 DEBUG HandlerThread:37167 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-08 13:52:15,921 DEBUG HandlerThread:37167 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-08 13:52:20,922 DEBUG HandlerThread:37167 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-08 13:52:22,649 DEBUG HandlerThread:37167 [handler.py:handle_request():146] handle_request: stop_status | |
| 2024-03-08 13:52:22,650 DEBUG SenderThread:37167 [sender.py:send_request():405] send_request: stop_status | |
| 2024-03-08 13:52:22,690 DEBUG HandlerThread:37167 [handler.py:handle_request():146] handle_request: internal_messages | |
| 2024-03-08 13:52:26,913 DEBUG HandlerThread:37167 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-08 13:52:31,914 DEBUG HandlerThread:37167 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-08 13:52:36,915 DEBUG HandlerThread:37167 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-08 13:52:37,174 DEBUG SenderThread:37167 [sender.py:send():378] send: stats | |
| 2024-03-08 13:52:37,649 DEBUG HandlerThread:37167 [handler.py:handle_request():146] handle_request: stop_status | |
| 2024-03-08 13:52:37,650 DEBUG SenderThread:37167 [sender.py:send_request():405] send_request: stop_status | |
| 2024-03-08 13:52:37,690 DEBUG HandlerThread:37167 [handler.py:handle_request():146] handle_request: internal_messages | |
| 2024-03-08 13:52:42,883 DEBUG HandlerThread:37167 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-08 13:52:47,884 DEBUG HandlerThread:37167 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-08 13:52:52,649 DEBUG HandlerThread:37167 [handler.py:handle_request():146] handle_request: stop_status | |
| 2024-03-08 13:52:52,650 DEBUG SenderThread:37167 [sender.py:send_request():405] send_request: stop_status | |
| 2024-03-08 13:52:52,690 DEBUG HandlerThread:37167 [handler.py:handle_request():146] handle_request: internal_messages | |
| 2024-03-08 13:52:52,936 DEBUG HandlerThread:37167 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-08 13:52:57,937 DEBUG HandlerThread:37167 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-08 13:53:02,938 DEBUG HandlerThread:37167 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-08 13:53:07,174 DEBUG SenderThread:37167 [sender.py:send():378] send: stats | |
| 2024-03-08 13:53:07,650 DEBUG HandlerThread:37167 [handler.py:handle_request():146] handle_request: stop_status | |
| 2024-03-08 13:53:07,650 DEBUG SenderThread:37167 [sender.py:send_request():405] send_request: stop_status | |
| 2024-03-08 13:53:07,690 DEBUG HandlerThread:37167 [handler.py:handle_request():146] handle_request: internal_messages | |
| 2024-03-08 13:53:07,941 DEBUG HandlerThread:37167 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-08 13:53:12,942 DEBUG HandlerThread:37167 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-08 13:53:17,943 DEBUG HandlerThread:37167 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-08 13:53:22,650 DEBUG HandlerThread:37167 [handler.py:handle_request():146] handle_request: stop_status | |
| 2024-03-08 13:53:22,650 DEBUG SenderThread:37167 [sender.py:send_request():405] send_request: stop_status | |
| 2024-03-08 13:53:22,690 DEBUG HandlerThread:37167 [handler.py:handle_request():146] handle_request: internal_messages | |
| 2024-03-08 13:53:24,209 DEBUG HandlerThread:37167 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-08 13:53:29,210 DEBUG HandlerThread:37167 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-08 13:53:32,742 DEBUG HandlerThread:37167 [handler.py:handle_request():146] handle_request: partial_history | |
| 2024-03-08 13:53:32,744 DEBUG SenderThread:37167 [sender.py:send():378] send: history | |
| 2024-03-08 13:53:32,744 DEBUG SenderThread:37167 [sender.py:send_request():405] send_request: summary_record | |
| 2024-03-08 13:53:32,746 INFO SenderThread:37167 [sender.py:_save_file():1365] saving file wandb-summary.json with policy end | |
| 2024-03-08 13:53:33,038 INFO Thread-12 :37167 [dir_watcher.py:_on_file_modified():288] file/dir modified: /content/wandb/run-20240308_134901-hlg2f18u/files/wandb-summary.json | |
| 2024-03-08 13:53:34,785 DEBUG HandlerThread:37167 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-08 13:53:36,039 INFO Thread-12 :37167 [dir_watcher.py:_on_file_modified():288] file/dir modified: /content/wandb/run-20240308_134901-hlg2f18u/files/output.log | |
| 2024-03-08 13:53:37,175 DEBUG SenderThread:37167 [sender.py:send():378] send: stats | |
| 2024-03-08 13:53:37,650 DEBUG HandlerThread:37167 [handler.py:handle_request():146] handle_request: stop_status | |
| 2024-03-08 13:53:37,651 DEBUG HandlerThread:37167 [handler.py:handle_request():146] handle_request: internal_messages | |
| 2024-03-08 13:53:37,651 DEBUG SenderThread:37167 [sender.py:send_request():405] send_request: stop_status | |
| 2024-03-08 13:53:39,861 DEBUG HandlerThread:37167 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-08 13:53:44,862 DEBUG HandlerThread:37167 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-08 13:53:49,863 DEBUG HandlerThread:37167 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-08 13:53:52,650 DEBUG HandlerThread:37167 [handler.py:handle_request():146] handle_request: stop_status | |
| 2024-03-08 13:53:52,651 DEBUG SenderThread:37167 [sender.py:send_request():405] send_request: stop_status | |
| 2024-03-08 13:53:52,691 DEBUG HandlerThread:37167 [handler.py:handle_request():146] handle_request: internal_messages | |
| 2024-03-08 13:53:55,843 DEBUG HandlerThread:37167 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-08 13:54:00,844 DEBUG HandlerThread:37167 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-08 13:54:05,845 DEBUG HandlerThread:37167 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-08 13:54:07,176 DEBUG SenderThread:37167 [sender.py:send():378] send: stats | |
| 2024-03-08 13:54:07,650 DEBUG HandlerThread:37167 [handler.py:handle_request():146] handle_request: stop_status | |
| 2024-03-08 13:54:07,651 DEBUG SenderThread:37167 [sender.py:send_request():405] send_request: stop_status | |
| 2024-03-08 13:54:07,691 DEBUG HandlerThread:37167 [handler.py:handle_request():146] handle_request: internal_messages | |
| 2024-03-08 13:54:10,892 DEBUG HandlerThread:37167 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-08 13:54:15,893 DEBUG HandlerThread:37167 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-08 13:54:20,894 DEBUG HandlerThread:37167 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-08 13:54:22,650 DEBUG HandlerThread:37167 [handler.py:handle_request():146] handle_request: stop_status | |
| 2024-03-08 13:54:22,651 DEBUG SenderThread:37167 [sender.py:send_request():405] send_request: stop_status | |
| 2024-03-08 13:54:22,691 DEBUG HandlerThread:37167 [handler.py:handle_request():146] handle_request: internal_messages | |
| 2024-03-08 13:54:26,137 DEBUG HandlerThread:37167 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-08 13:54:31,138 DEBUG HandlerThread:37167 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-08 13:54:36,139 DEBUG HandlerThread:37167 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-08 13:54:37,177 DEBUG SenderThread:37167 [sender.py:send():378] send: stats | |
| 2024-03-08 13:54:37,651 DEBUG HandlerThread:37167 [handler.py:handle_request():146] handle_request: stop_status | |
| 2024-03-08 13:54:37,651 DEBUG SenderThread:37167 [sender.py:send_request():405] send_request: stop_status | |
| 2024-03-08 13:54:37,691 DEBUG HandlerThread:37167 [handler.py:handle_request():146] handle_request: internal_messages | |
| 2024-03-08 13:54:41,932 DEBUG HandlerThread:37167 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-08 13:54:46,933 DEBUG HandlerThread:37167 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-08 13:54:51,934 DEBUG HandlerThread:37167 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-08 13:54:52,651 DEBUG HandlerThread:37167 [handler.py:handle_request():146] handle_request: stop_status | |
| 2024-03-08 13:54:52,651 DEBUG SenderThread:37167 [sender.py:send_request():405] send_request: stop_status | |
| 2024-03-08 13:54:52,691 DEBUG HandlerThread:37167 [handler.py:handle_request():146] handle_request: internal_messages | |
| 2024-03-08 13:54:57,199 DEBUG HandlerThread:37167 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-08 13:55:02,200 DEBUG HandlerThread:37167 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-08 13:55:04,360 DEBUG HandlerThread:37167 [handler.py:handle_request():146] handle_request: partial_history | |
| 2024-03-08 13:55:04,362 DEBUG SenderThread:37167 [sender.py:send():378] send: history | |
| 2024-03-08 13:55:04,362 DEBUG SenderThread:37167 [sender.py:send_request():405] send_request: summary_record | |
| 2024-03-08 13:55:04,363 INFO SenderThread:37167 [sender.py:_save_file():1365] saving file wandb-summary.json with policy end | |
| 2024-03-08 13:55:05,072 INFO Thread-12 :37167 [dir_watcher.py:_on_file_modified():288] file/dir modified: /content/wandb/run-20240308_134901-hlg2f18u/files/wandb-summary.json | |
| 2024-03-08 13:55:06,073 INFO Thread-12 :37167 [dir_watcher.py:_on_file_modified():288] file/dir modified: /content/wandb/run-20240308_134901-hlg2f18u/files/output.log | |
| 2024-03-08 13:55:07,177 DEBUG SenderThread:37167 [sender.py:send():378] send: stats | |
| 2024-03-08 13:55:07,652 DEBUG HandlerThread:37167 [handler.py:handle_request():146] handle_request: stop_status | |
| 2024-03-08 13:55:07,652 DEBUG HandlerThread:37167 [handler.py:handle_request():146] handle_request: internal_messages | |
| 2024-03-08 13:55:07,653 DEBUG SenderThread:37167 [sender.py:send_request():405] send_request: stop_status | |
| 2024-03-08 13:55:07,845 DEBUG HandlerThread:37167 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-08 13:55:12,846 DEBUG HandlerThread:37167 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-08 13:55:17,847 DEBUG HandlerThread:37167 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-08 13:55:22,651 DEBUG HandlerThread:37167 [handler.py:handle_request():146] handle_request: stop_status | |
| 2024-03-08 13:55:22,652 DEBUG SenderThread:37167 [sender.py:send_request():405] send_request: stop_status | |
| 2024-03-08 13:55:22,692 DEBUG HandlerThread:37167 [handler.py:handle_request():146] handle_request: internal_messages | |
| 2024-03-08 13:55:22,923 DEBUG HandlerThread:37167 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-08 13:55:27,924 DEBUG HandlerThread:37167 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-08 13:55:32,925 DEBUG HandlerThread:37167 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-08 13:55:37,178 DEBUG SenderThread:37167 [sender.py:send():378] send: stats | |
| 2024-03-08 13:55:37,651 DEBUG HandlerThread:37167 [handler.py:handle_request():146] handle_request: stop_status | |
| 2024-03-08 13:55:37,652 DEBUG SenderThread:37167 [sender.py:send_request():405] send_request: stop_status | |
| 2024-03-08 13:55:37,692 DEBUG HandlerThread:37167 [handler.py:handle_request():146] handle_request: internal_messages | |
| 2024-03-08 13:55:38,904 DEBUG HandlerThread:37167 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-08 13:55:43,905 DEBUG HandlerThread:37167 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-08 13:55:48,906 DEBUG HandlerThread:37167 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-08 13:55:52,652 DEBUG HandlerThread:37167 [handler.py:handle_request():146] handle_request: stop_status | |
| 2024-03-08 13:55:52,652 DEBUG SenderThread:37167 [sender.py:send_request():405] send_request: stop_status | |
| 2024-03-08 13:55:52,692 DEBUG HandlerThread:37167 [handler.py:handle_request():146] handle_request: internal_messages | |
| 2024-03-08 13:55:54,850 DEBUG HandlerThread:37167 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-08 13:55:59,850 DEBUG HandlerThread:37167 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-08 13:56:04,851 DEBUG HandlerThread:37167 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-08 13:56:07,179 DEBUG SenderThread:37167 [sender.py:send():378] send: stats | |
| 2024-03-08 13:56:07,652 DEBUG HandlerThread:37167 [handler.py:handle_request():146] handle_request: stop_status | |
| 2024-03-08 13:56:07,652 DEBUG SenderThread:37167 [sender.py:send_request():405] send_request: stop_status | |
| 2024-03-08 13:56:07,692 DEBUG HandlerThread:37167 [handler.py:handle_request():146] handle_request: internal_messages | |
| 2024-03-08 13:56:09,877 DEBUG HandlerThread:37167 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-08 13:56:14,878 DEBUG HandlerThread:37167 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-08 13:56:19,880 DEBUG HandlerThread:37167 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-08 13:56:22,652 DEBUG HandlerThread:37167 [handler.py:handle_request():146] handle_request: stop_status | |
| 2024-03-08 13:56:22,652 DEBUG SenderThread:37167 [sender.py:send_request():405] send_request: stop_status | |
| 2024-03-08 13:56:22,692 DEBUG HandlerThread:37167 [handler.py:handle_request():146] handle_request: internal_messages | |
| 2024-03-08 13:56:24,927 DEBUG HandlerThread:37167 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-08 13:56:29,928 DEBUG HandlerThread:37167 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-08 13:56:34,929 DEBUG HandlerThread:37167 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-08 13:56:37,180 DEBUG SenderThread:37167 [sender.py:send():378] send: stats | |
| 2024-03-08 13:56:37,387 DEBUG HandlerThread:37167 [handler.py:handle_request():146] handle_request: partial_history | |
| 2024-03-08 13:56:37,389 DEBUG SenderThread:37167 [sender.py:send():378] send: history | |
| 2024-03-08 13:56:37,389 DEBUG SenderThread:37167 [sender.py:send_request():405] send_request: summary_record | |
| 2024-03-08 13:56:37,390 INFO SenderThread:37167 [sender.py:_save_file():1365] saving file wandb-summary.json with policy end | |
| 2024-03-08 13:56:37,652 DEBUG HandlerThread:37167 [handler.py:handle_request():146] handle_request: stop_status | |
| 2024-03-08 13:56:37,653 DEBUG SenderThread:37167 [sender.py:send_request():405] send_request: stop_status | |
| 2024-03-08 13:56:37,653 DEBUG HandlerThread:37167 [handler.py:handle_request():146] handle_request: internal_messages | |
| 2024-03-08 13:56:38,107 INFO Thread-12 :37167 [dir_watcher.py:_on_file_modified():288] file/dir modified: /content/wandb/run-20240308_134901-hlg2f18u/files/wandb-summary.json | |
| 2024-03-08 13:56:39,940 DEBUG HandlerThread:37167 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-08 13:56:40,108 INFO Thread-12 :37167 [dir_watcher.py:_on_file_modified():288] file/dir modified: /content/wandb/run-20240308_134901-hlg2f18u/files/output.log | |
| 2024-03-08 13:56:44,942 DEBUG HandlerThread:37167 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-08 13:56:49,943 DEBUG HandlerThread:37167 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-08 13:56:52,652 DEBUG HandlerThread:37167 [handler.py:handle_request():146] handle_request: stop_status | |
| 2024-03-08 13:56:52,653 DEBUG SenderThread:37167 [sender.py:send_request():405] send_request: stop_status | |
| 2024-03-08 13:56:52,693 DEBUG HandlerThread:37167 [handler.py:handle_request():146] handle_request: internal_messages | |
| 2024-03-08 13:56:55,868 DEBUG HandlerThread:37167 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-08 13:57:00,869 DEBUG HandlerThread:37167 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-08 13:57:05,870 DEBUG HandlerThread:37167 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-08 13:57:07,180 DEBUG SenderThread:37167 [sender.py:send():378] send: stats | |
| 2024-03-08 13:57:07,652 DEBUG HandlerThread:37167 [handler.py:handle_request():146] handle_request: stop_status | |
| 2024-03-08 13:57:07,653 DEBUG SenderThread:37167 [sender.py:send_request():405] send_request: stop_status | |
| 2024-03-08 13:57:07,694 DEBUG HandlerThread:37167 [handler.py:handle_request():146] handle_request: internal_messages | |
| 2024-03-08 13:57:10,899 DEBUG HandlerThread:37167 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-08 13:57:15,901 DEBUG HandlerThread:37167 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-08 13:57:20,902 DEBUG HandlerThread:37167 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-08 13:57:22,652 DEBUG HandlerThread:37167 [handler.py:handle_request():146] handle_request: stop_status | |
| 2024-03-08 13:57:22,653 DEBUG SenderThread:37167 [sender.py:send_request():405] send_request: stop_status | |
| 2024-03-08 13:57:22,693 DEBUG HandlerThread:37167 [handler.py:handle_request():146] handle_request: internal_messages | |
| 2024-03-08 13:57:26,888 DEBUG HandlerThread:37167 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-08 13:57:31,888 DEBUG HandlerThread:37167 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-08 13:57:36,890 DEBUG HandlerThread:37167 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-08 13:57:37,181 DEBUG SenderThread:37167 [sender.py:send():378] send: stats | |
| 2024-03-08 13:57:37,653 DEBUG HandlerThread:37167 [handler.py:handle_request():146] handle_request: stop_status | |
| 2024-03-08 13:57:37,653 DEBUG SenderThread:37167 [sender.py:send_request():405] send_request: stop_status | |
| 2024-03-08 13:57:37,693 DEBUG HandlerThread:37167 [handler.py:handle_request():146] handle_request: internal_messages | |
| 2024-03-08 13:57:42,876 DEBUG HandlerThread:37167 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-08 13:57:47,877 DEBUG HandlerThread:37167 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-08 13:57:52,653 DEBUG HandlerThread:37167 [handler.py:handle_request():146] handle_request: stop_status | |
| 2024-03-08 13:57:52,653 DEBUG SenderThread:37167 [sender.py:send_request():405] send_request: stop_status | |
| 2024-03-08 13:57:52,693 DEBUG HandlerThread:37167 [handler.py:handle_request():146] handle_request: internal_messages | |
| 2024-03-08 13:57:53,876 DEBUG HandlerThread:37167 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-08 13:57:58,877 DEBUG HandlerThread:37167 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-08 13:58:03,879 DEBUG HandlerThread:37167 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-08 13:58:07,182 DEBUG SenderThread:37167 [sender.py:send():378] send: stats | |
| 2024-03-08 13:58:07,653 DEBUG HandlerThread:37167 [handler.py:handle_request():146] handle_request: stop_status | |
| 2024-03-08 13:58:07,654 DEBUG SenderThread:37167 [sender.py:send_request():405] send_request: stop_status | |
| 2024-03-08 13:58:07,695 DEBUG HandlerThread:37167 [handler.py:handle_request():146] handle_request: internal_messages | |
| 2024-03-08 13:58:09,855 DEBUG HandlerThread:37167 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-08 13:58:13,676 DEBUG HandlerThread:37167 [handler.py:handle_request():146] handle_request: partial_history | |
| 2024-03-08 13:58:13,678 DEBUG SenderThread:37167 [sender.py:send():378] send: history | |
| 2024-03-08 13:58:13,678 DEBUG SenderThread:37167 [sender.py:send_request():405] send_request: summary_record | |
| 2024-03-08 13:58:13,679 INFO SenderThread:37167 [sender.py:_save_file():1365] saving file wandb-summary.json with policy end | |
| 2024-03-08 13:58:14,144 INFO Thread-12 :37167 [dir_watcher.py:_on_file_modified():288] file/dir modified: /content/wandb/run-20240308_134901-hlg2f18u/files/wandb-summary.json | |
| 2024-03-08 13:58:15,718 DEBUG HandlerThread:37167 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-08 13:58:16,144 INFO Thread-12 :37167 [dir_watcher.py:_on_file_modified():288] file/dir modified: /content/wandb/run-20240308_134901-hlg2f18u/files/output.log | |
| 2024-03-08 13:58:20,719 DEBUG HandlerThread:37167 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-08 13:58:22,653 DEBUG HandlerThread:37167 [handler.py:handle_request():146] handle_request: stop_status | |
| 2024-03-08 13:58:22,654 DEBUG HandlerThread:37167 [handler.py:handle_request():146] handle_request: internal_messages | |
| 2024-03-08 13:58:22,654 DEBUG SenderThread:37167 [sender.py:send_request():405] send_request: stop_status | |
| 2024-03-08 13:58:26,249 DEBUG HandlerThread:37167 [handler.py:handle_request():146] handle_request: status_report | |