2024-03-21 01:54:58,388 INFO    StreamThr :7197 [internal.py:wandb_internal():86] W&B internal server running at pid: 7197, started at: 2024-03-21 01:54:58.384167
2024-03-21 01:54:58,394 DEBUG   HandlerThread:7197 [handler.py:handle_request():146] handle_request: status
2024-03-21 01:55:03,409 DEBUG   HandlerThread:7197 [handler.py:handle_request():146] handle_request: status_report
2024-03-21 01:55:03,409 INFO    WriterThread:7197 [datastore.py:open_for_write():87] open: /content/gdrive/MyDrive/LLM/Mistral-7B-Finetuning-Insurance/wandb/run-20240321_015458-y11xrjxv/run-y11xrjxv.wandb
2024-03-21 01:55:03,434 DEBUG   SenderThread:7197 [sender.py:send():378] send: header
2024-03-21 01:55:03,442 DEBUG   SenderThread:7197 [sender.py:send():378] send: run
2024-03-21 01:55:04,065 INFO    SenderThread:7197 [dir_watcher.py:__init__():211] watching files in: /content/gdrive/MyDrive/LLM/Mistral-7B-Finetuning-Insurance/wandb/run-20240321_015458-y11xrjxv/files
2024-03-21 01:55:04,066 INFO    SenderThread:7197 [sender.py:_start_run_threads():1099] run started: y11xrjxv with start time 1710986098.384977
2024-03-21 01:55:04,082 DEBUG   HandlerThread:7197 [handler.py:handle_request():146] handle_request: check_version
2024-03-21 01:55:04,083 DEBUG   SenderThread:7197 [sender.py:send_request():405] send_request: check_version
2024-03-21 01:55:04,202 DEBUG   HandlerThread:7197 [handler.py:handle_request():146] handle_request: run_start
2024-03-21 01:55:04,211 DEBUG   HandlerThread:7197 [system_info.py:__init__():26] System info init
2024-03-21 01:55:04,211 DEBUG   HandlerThread:7197 [system_info.py:__init__():41] System info init done
2024-03-21 01:55:04,211 INFO    HandlerThread:7197 [system_monitor.py:start():194] Starting system monitor
2024-03-21 01:55:04,211 INFO    SystemMonitor:7197 [system_monitor.py:_start():158] Starting system asset monitoring threads
2024-03-21 01:55:04,212 INFO    HandlerThread:7197 [system_monitor.py:probe():214] Collecting system info
2024-03-21 01:55:04,212 INFO    SystemMonitor:7197 [interfaces.py:start():190] Started cpu monitoring
2024-03-21 01:55:04,213 INFO    SystemMonitor:7197 [interfaces.py:start():190] Started disk monitoring
2024-03-21 01:55:04,214 INFO    SystemMonitor:7197 [interfaces.py:start():190] Started gpu monitoring
2024-03-21 01:55:04,215 INFO    SystemMonitor:7197 [interfaces.py:start():190] Started memory monitoring
2024-03-21 01:55:04,215 INFO    SystemMonitor:7197 [interfaces.py:start():190] Started network monitoring
2024-03-21 01:55:04,228 DEBUG   HandlerThread:7197 [system_info.py:probe():150] Probing system
2024-03-21 01:55:04,234 DEBUG   HandlerThread:7197 [gitlib.py:_init_repo():56] git repository is invalid
2024-03-21 01:55:04,234 DEBUG   HandlerThread:7197 [system_info.py:probe():198] Probing system done
2024-03-21 01:55:04,234 DEBUG   HandlerThread:7197 [system_monitor.py:probe():223] {'os': 'Linux-6.1.58+-x86_64-with-glibc2.35', 'python': '3.10.12', 'heartbeatAt': '2024-03-21T01:55:04.228731', 'startedAt': '2024-03-21T01:54:58.281206', 'docker': None, 'cuda': None, 'args': (), 'state': 'running', 'program': 'Mistral-7B-Finetuning-Insurance.ipynb%EC%9D%98%20%EC%82%AC%EB%B3%B8', 'codePathLocal': None, 'colab': 'https://colab.research.google.com/notebook#fileId=1glmddDRkqHDhfUT8slC17FnAssneTUkr', 'host': 'd72aea89d4b6', 'username': 'root', 'executable': '/usr/bin/python3', 'cpu_count': 1, 'cpu_count_logical': 2, 'cpu_freq': {'current': 2000.176, 'min': 0.0, 'max': 0.0}, 'cpu_freq_per_core': [{'current': 2000.176, 'min': 0.0, 'max': 0.0}, {'current': 2000.176, 'min': 0.0, 'max': 0.0}], 'disk': {'/': {'total': 78.1898422241211, 'used': 42.871490478515625}}, 'gpu': 'Tesla T4', 'gpu_count': 1, 'gpu_devices': [{'name': 'Tesla T4', 'memory_total': 16106127360}], 'memory': {'total': 12.674781799316406}}
2024-03-21 01:55:04,235 INFO    HandlerThread:7197 [system_monitor.py:probe():224] Finished collecting system info
2024-03-21 01:55:04,235 INFO    HandlerThread:7197 [system_monitor.py:probe():227] Publishing system info
2024-03-21 01:55:04,254 INFO    HandlerThread:7197 [system_monitor.py:probe():229] Finished publishing system info
2024-03-21 01:55:04,260 DEBUG   SenderThread:7197 [sender.py:send():378] send: files
2024-03-21 01:55:04,260 INFO    SenderThread:7197 [sender.py:_save_file():1365] saving file wandb-metadata.json with policy now
2024-03-21 01:55:04,647 DEBUG   HandlerThread:7197 [handler.py:handle_request():146] handle_request: python_packages
2024-03-21 01:55:04,648 DEBUG   HandlerThread:7197 [handler.py:handle_request():146] handle_request: stop_status
2024-03-21 01:55:04,648 DEBUG   SenderThread:7197 [sender.py:send_request():405] send_request: python_packages
2024-03-21 01:55:04,649 DEBUG   HandlerThread:7197 [handler.py:handle_request():146] handle_request: internal_messages
2024-03-21 01:55:04,656 DEBUG   SenderThread:7197 [sender.py:send_request():405] send_request: stop_status
2024-03-21 01:55:05,019 DEBUG   SenderThread:7197 [sender.py:send():378] send: telemetry
2024-03-21 01:55:05,020 DEBUG   SenderThread:7197 [sender.py:send():378] send: config
2024-03-21 01:55:05,021 DEBUG   SenderThread:7197 [sender.py:send():378] send: metric
2024-03-21 01:55:05,022 DEBUG   SenderThread:7197 [sender.py:send():378] send: telemetry
2024-03-21 01:55:05,022 DEBUG   SenderThread:7197 [sender.py:send():378] send: metric
2024-03-21 01:55:05,022 WARNING SenderThread:7197 [sender.py:send_metric():1316] Seen metric with glob (shouldn't happen)
2024-03-21 01:55:05,022 DEBUG   SenderThread:7197 [sender.py:send():378] send: telemetry
2024-03-21 01:55:05,070 INFO    Thread-12 :7197 [dir_watcher.py:_on_file_created():271] file/dir created: /content/gdrive/MyDrive/LLM/Mistral-7B-Finetuning-Insurance/wandb/run-20240321_015458-y11xrjxv/files/wandb-metadata.json
2024-03-21 01:55:05,070 INFO    Thread-12 :7197 [dir_watcher.py:_on_file_created():271] file/dir created: /content/gdrive/MyDrive/LLM/Mistral-7B-Finetuning-Insurance/wandb/run-20240321_015458-y11xrjxv/files/requirements.txt
2024-03-21 01:55:05,160 INFO    wandb-upload_0:7197 [upload_job.py:push():131] Uploaded file /tmp/tmpe7x8kdblwandb/qivfhje6-wandb-metadata.json
2024-03-21 01:55:09,023 DEBUG   HandlerThread:7197 [handler.py:handle_request():146] handle_request: status_report
2024-03-21 01:55:14,024 DEBUG   HandlerThread:7197 [handler.py:handle_request():146] handle_request: status_report
2024-03-21 01:55:19,025 DEBUG   HandlerThread:7197 [handler.py:handle_request():146] handle_request: status_report
2024-03-21 01:55:19,648 DEBUG   HandlerThread:7197 [handler.py:handle_request():146] handle_request: stop_status
2024-03-21 01:55:19,648 DEBUG   SenderThread:7197 [sender.py:send_request():405] send_request: stop_status
2024-03-21 01:55:19,652 DEBUG   HandlerThread:7197 [handler.py:handle_request():146] handle_request: internal_messages
2024-03-21 01:55:24,884 DEBUG   HandlerThread:7197 [handler.py:handle_request():146] handle_request: status_report
2024-03-21 01:55:29,891 DEBUG   HandlerThread:7197 [handler.py:handle_request():146] handle_request: status_report
2024-03-21 01:55:31,114 INFO    Thread-12 :7197 [dir_watcher.py:_on_file_modified():288] file/dir modified: /content/gdrive/MyDrive/LLM/Mistral-7B-Finetuning-Insurance/wandb/run-20240321_015458-y11xrjxv/files/config.yaml
2024-03-21 01:55:34,648 DEBUG   HandlerThread:7197 [handler.py:handle_request():146] handle_request: stop_status
2024-03-21 01:55:34,648 DEBUG   SenderThread:7197 [sender.py:send_request():405] send_request: stop_status
2024-03-21 01:55:34,689 DEBUG   HandlerThread:7197 [handler.py:handle_request():146] handle_request: internal_messages
2024-03-21 01:55:35,977 DEBUG   HandlerThread:7197 [handler.py:handle_request():146] handle_request: status_report
2024-03-21 01:55:40,977 DEBUG   HandlerThread:7197 [handler.py:handle_request():146] handle_request: status_report
2024-03-21 01:55:45,980 DEBUG   HandlerThread:7197 [handler.py:handle_request():146] handle_request: status_report
2024-03-21 01:55:49,648 DEBUG   HandlerThread:7197 [handler.py:handle_request():146] handle_request: stop_status
2024-03-21 01:55:49,649 DEBUG   SenderThread:7197 [sender.py:send_request():405] send_request: stop_status
2024-03-21 01:55:49,689 DEBUG   HandlerThread:7197 [handler.py:handle_request():146] handle_request: internal_messages
2024-03-21 01:55:50,983 DEBUG   HandlerThread:7197 [handler.py:handle_request():146] handle_request: status_report
2024-03-21 01:55:55,984 DEBUG   HandlerThread:7197 [handler.py:handle_request():146] handle_request: status_report
2024-03-21 01:56:00,985 DEBUG   HandlerThread:7197 [handler.py:handle_request():146] handle_request: status_report
2024-03-21 01:56:04,216 DEBUG   SystemMonitor:7197 [system_monitor.py:_start():172] Starting system metrics aggregation loop
2024-03-21 01:56:04,217 DEBUG   SenderThread:7197 [sender.py:send():378] send: stats
2024-03-21 01:56:04,648 DEBUG   HandlerThread:7197 [handler.py:handle_request():146] handle_request: stop_status
2024-03-21 01:56:04,649 DEBUG   SenderThread:7197 [sender.py:send_request():405] send_request: stop_status
2024-03-21 01:56:04,689 DEBUG   HandlerThread:7197 [handler.py:handle_request():146] handle_request: internal_messages
2024-03-21 01:56:06,724 DEBUG   HandlerThread:7197 [handler.py:handle_request():146] handle_request: status_report
2024-03-21 01:56:11,725 DEBUG   HandlerThread:7197 [handler.py:handle_request():146] handle_request: status_report
2024-03-21 01:56:16,726 DEBUG   HandlerThread:7197 [handler.py:handle_request():146] handle_request: status_report
2024-03-21 01:56:19,648 DEBUG   HandlerThread:7197 [handler.py:handle_request():146] handle_request: stop_status
2024-03-21 01:56:19,649 DEBUG   SenderThread:7197 [sender.py:send_request():405] send_request: stop_status
2024-03-21 01:56:19,689 DEBUG   HandlerThread:7197 [handler.py:handle_request():146] handle_request: internal_messages
2024-03-21 01:56:21,983 DEBUG   HandlerThread:7197 [handler.py:handle_request():146] handle_request: status_report
2024-03-21 01:56:26,984 DEBUG   HandlerThread:7197 [handler.py:handle_request():146] handle_request: status_report
2024-03-21 01:56:31,985 DEBUG   HandlerThread:7197 [handler.py:handle_request():146] handle_request: status_report
2024-03-21 01:56:34,218 DEBUG   SenderThread:7197 [sender.py:send():378] send: stats
2024-03-21 01:56:34,650 DEBUG   HandlerThread:7197 [handler.py:handle_request():146] handle_request: stop_status
2024-03-21 01:56:34,651 DEBUG   SenderThread:7197 [sender.py:send_request():405] send_request: stop_status
2024-03-21 01:56:34,691 DEBUG   HandlerThread:7197 [handler.py:handle_request():146] handle_request: internal_messages
2024-03-21 01:56:37,948 DEBUG   HandlerThread:7197 [handler.py:handle_request():146] handle_request: status_report
2024-03-21 01:56:42,949 DEBUG   HandlerThread:7197 [handler.py:handle_request():146] handle_request: status_report
2024-03-21 01:56:47,949 DEBUG   HandlerThread:7197 [handler.py:handle_request():146] handle_request: status_report
2024-03-21 01:56:49,650 DEBUG   HandlerThread:7197 [handler.py:handle_request():146] handle_request: stop_status
2024-03-21 01:56:49,651 DEBUG   SenderThread:7197 [sender.py:send_request():405] send_request: stop_status
2024-03-21 01:56:49,691 DEBUG   HandlerThread:7197 [handler.py:handle_request():146] handle_request: internal_messages
2024-03-21 01:56:52,960 DEBUG   HandlerThread:7197 [handler.py:handle_request():146] handle_request: status_report
2024-03-21 01:56:57,961 DEBUG   HandlerThread:7197 [handler.py:handle_request():146] handle_request: status_report
2024-03-21 01:57:02,961 DEBUG   HandlerThread:7197 [handler.py:handle_request():146] handle_request: status_report
2024-03-21 01:57:04,219 DEBUG   SenderThread:7197 [sender.py:send():378] send: stats
2024-03-21 01:57:04,650 DEBUG   HandlerThread:7197 [handler.py:handle_request():146] handle_request: stop_status
2024-03-21 01:57:04,651 DEBUG   SenderThread:7197 [sender.py:send_request():405] send_request: stop_status
2024-03-21 01:57:04,691 DEBUG   HandlerThread:7197 [handler.py:handle_request():146] handle_request: internal_messages
2024-03-21 01:57:07,993 DEBUG   HandlerThread:7197 [handler.py:handle_request():146] handle_request: status_report
2024-03-21 01:57:12,994 DEBUG   HandlerThread:7197 [handler.py:handle_request():146] handle_request: status_report
2024-03-21 01:57:17,995 DEBUG   HandlerThread:7197 [handler.py:handle_request():146] handle_request: status_report
2024-03-21 01:57:19,650 DEBUG   HandlerThread:7197 [handler.py:handle_request():146] handle_request: stop_status
2024-03-21 01:57:19,651 DEBUG   SenderThread:7197 [sender.py:send_request():405] send_request: stop_status
2024-03-21 01:57:19,691 DEBUG   HandlerThread:7197 [handler.py:handle_request():146] handle_request: internal_messages
2024-03-21 01:57:23,885 DEBUG   HandlerThread:7197 [handler.py:handle_request():146] handle_request: status_report
2024-03-21 01:57:28,885 DEBUG   HandlerThread:7197 [handler.py:handle_request():146] handle_request: status_report
2024-03-21 01:57:33,886 DEBUG   HandlerThread:7197 [handler.py:handle_request():146] handle_request: status_report
2024-03-21 01:57:34,219 DEBUG   SenderThread:7197 [sender.py:send():378] send: stats
2024-03-21 01:57:34,650 DEBUG   HandlerThread:7197 [handler.py:handle_request():146] handle_request: stop_status
2024-03-21 01:57:34,651 DEBUG   SenderThread:7197 [sender.py:send_request():405] send_request: stop_status
2024-03-21 01:57:34,697 DEBUG   HandlerThread:7197 [handler.py:handle_request():146] handle_request: internal_messages
2024-03-21 01:57:36,218 DEBUG   HandlerThread:7197 [handler.py:handle_request():146] handle_request: partial_history
2024-03-21 01:57:36,221 DEBUG   SenderThread:7197 [sender.py:send():378] send: metric
2024-03-21 01:57:36,221 DEBUG   SenderThread:7197 [sender.py:send():378] send: metric
2024-03-21 01:57:36,221 DEBUG   SenderThread:7197 [sender.py:send():378] send: metric
2024-03-21 01:57:36,222 DEBUG   SenderThread:7197 [sender.py:send():378] send: metric
2024-03-21 01:57:36,224 DEBUG   SenderThread:7197 [sender.py:send():378] send: history
2024-03-21 01:57:36,225 DEBUG   SenderThread:7197 [sender.py:send_request():405] send_request: summary_record
2024-03-21 01:57:36,230 INFO    SenderThread:7197 [sender.py:_save_file():1365] saving file wandb-summary.json with policy end
2024-03-21 01:57:36,349 INFO    Thread-12 :7197 [dir_watcher.py:_on_file_created():271] file/dir created: /content/gdrive/MyDrive/LLM/Mistral-7B-Finetuning-Insurance/wandb/run-20240321_015458-y11xrjxv/files/wandb-summary.json
2024-03-21 01:57:39,233 DEBUG   HandlerThread:7197 [handler.py:handle_request():146] handle_request: status_report
2024-03-21 01:57:44,234 DEBUG   HandlerThread:7197 [handler.py:handle_request():146] handle_request: status_report
2024-03-21 01:58:03,817 DEBUG   HandlerThread:7197 [handler.py:handle_request():146] handle_request: status_report
2024-03-21 01:58:03,819 DEBUG   HandlerThread:7197 [handler.py:handle_request():146] handle_request: status_report
2024-03-21 01:58:03,819 DEBUG   HandlerThread:7197 [handler.py:handle_request():146] handle_request: status_report
2024-03-21 01:58:04,033 DEBUG   HandlerThread:7197 [handler.py:handle_request():146] handle_request: internal_messages
2024-03-21 01:58:04,103 DEBUG   HandlerThread:7197 [handler.py:handle_request():146] handle_request: stop_status
2024-03-21 01:58:04,105 DEBUG   SenderThread:7197 [sender.py:send_request():405] send_request: stop_status
2024-03-21 01:58:04,415 DEBUG   HandlerThread:7197 [handler.py:handle_request():146] handle_request: status_report
2024-03-21 01:58:05,362 DEBUG   SenderThread:7197 [sender.py:send():378] send: stats
2024-03-21 01:58:05,506 INFO    Thread-12 :7197 [dir_watcher.py:_on_file_modified():288] file/dir modified: /content/gdrive/MyDrive/LLM/Mistral-7B-Finetuning-Insurance/wandb/run-20240321_015458-y11xrjxv/files/config.yaml
2024-03-21 01:58:10,366 DEBUG   HandlerThread:7197 [handler.py:handle_request():146] handle_request: status_report
2024-03-21 01:58:15,367 DEBUG   HandlerThread:7197 [handler.py:handle_request():146] handle_request: status_report
2024-03-21 01:58:20,047 DEBUG   HandlerThread:7197 [handler.py:handle_request():146] handle_request: stop_status
2024-03-21 01:58:20,049 DEBUG   HandlerThread:7197 [handler.py:handle_request():146] handle_request: internal_messages
2024-03-21 01:58:20,049 DEBUG   SenderThread:7197 [sender.py:send_request():405] send_request: stop_status
2024-03-21 01:58:21,305 DEBUG   HandlerThread:7197 [handler.py:handle_request():146] handle_request: status_report
2024-03-21 01:58:26,306 DEBUG   HandlerThread:7197 [handler.py:handle_request():146] handle_request: status_report
2024-03-21 01:58:31,307 DEBUG   HandlerThread:7197 [handler.py:handle_request():146] handle_request: status_report
2024-03-21 01:58:53,904 DEBUG   HandlerThread:7197 [handler.py:handle_request():146] handle_request: stop_status
2024-03-21 01:58:53,911 DEBUG   HandlerThread:7197 [handler.py:handle_request():146] handle_request: internal_messages
2024-03-21 01:58:53,911 DEBUG   HandlerThread:7197 [handler.py:handle_request():146] handle_request: status_report
2024-03-21 01:58:53,911 DEBUG   HandlerThread:7197 [handler.py:handle_request():146] handle_request: keepalive
2024-03-21 01:58:53,913 DEBUG   HandlerThread:7197 [handler.py:handle_request():146] handle_request: status_report
2024-03-21 01:58:53,913 DEBUG   HandlerThread:7197 [handler.py:handle_request():146] handle_request: keepalive
2024-03-21 01:58:53,913 DEBUG   HandlerThread:7197 [handler.py:handle_request():146] handle_request: status_report
2024-03-21 01:58:53,913 DEBUG   HandlerThread:7197 [handler.py:handle_request():146] handle_request: keepalive
2024-03-21 01:58:53,913 DEBUG   HandlerThread:7197 [handler.py:handle_request():146] handle_request: status_report
2024-03-21 01:58:53,904 DEBUG   SenderThread:7197 [sender.py:send():378] send: stats
2024-03-21 01:58:53,914 DEBUG   SenderThread:7197 [sender.py:send_request():405] send_request: stop_status
2024-03-21 01:58:55,052 DEBUG   HandlerThread:7197 [handler.py:handle_request():146] handle_request: keepalive
2024-03-21 01:58:58,517 DEBUG   HandlerThread:7197 [handler.py:handle_request():146] handle_request: status_report
2024-03-21 01:59:00,166 DEBUG   HandlerThread:7197 [handler.py:handle_request():146] handle_request: keepalive
2024-03-21 01:59:03,518 DEBUG   HandlerThread:7197 [handler.py:handle_request():146] handle_request: status_report
2024-03-21 01:59:04,222 DEBUG   SenderThread:7197 [sender.py:send():378] send: stats
2024-03-21 01:59:05,167 DEBUG   HandlerThread:7197 [handler.py:handle_request():146] handle_request: keepalive
2024-03-21 01:59:09,223 DEBUG   HandlerThread:7197 [handler.py:handle_request():146] handle_request: status_report
2024-03-21 01:59:10,169 DEBUG   HandlerThread:7197 [handler.py:handle_request():146] handle_request: keepalive
2024-03-21 01:59:14,224 DEBUG   HandlerThread:7197 [handler.py:handle_request():146] handle_request: status_report
2024-03-21 01:59:15,170 DEBUG   HandlerThread:7197 [handler.py:handle_request():146] handle_request: keepalive
2024-03-21 01:59:19,225 DEBUG   HandlerThread:7197 [handler.py:handle_request():146] handle_request: status_report
2024-03-21 01:59:20,171 DEBUG   HandlerThread:7197 [handler.py:handle_request():146] handle_request: keepalive
2024-03-21 01:59:24,225 DEBUG   HandlerThread:7197 [handler.py:handle_request():146] handle_request: status_report
2024-03-21 01:59:25,172 DEBUG   HandlerThread:7197 [handler.py:handle_request():146] handle_request: keepalive
2024-03-21 01:59:29,226 DEBUG   HandlerThread:7197 [handler.py:handle_request():146] handle_request: status_report
2024-03-21 01:59:30,173 DEBUG   HandlerThread:7197 [handler.py:handle_request():146] handle_request: keepalive
2024-03-21 01:59:34,222 DEBUG   SenderThread:7197 [sender.py:send():378] send: stats
2024-03-21 01:59:35,176 DEBUG   HandlerThread:7197 [handler.py:handle_request():146] handle_request: keepalive
2024-03-21 01:59:35,223 DEBUG   HandlerThread:7197 [handler.py:handle_request():146] handle_request: status_report
2024-03-21 01:59:40,177 DEBUG   HandlerThread:7197 [handler.py:handle_request():146] handle_request: keepalive
2024-03-21 01:59:40,224 DEBUG   HandlerThread:7197 [handler.py:handle_request():146] handle_request: status_report
2024-03-21 01:59:45,178 DEBUG   HandlerThread:7197 [handler.py:handle_request():146] handle_request: keepalive
2024-03-21 01:59:45,225 DEBUG   HandlerThread:7197 [handler.py:handle_request():146] handle_request: status_report
2024-03-21 01:59:50,179 DEBUG   HandlerThread:7197 [handler.py:handle_request():146] handle_request: keepalive
2024-03-21 01:59:50,226 DEBUG   HandlerThread:7197 [handler.py:handle_request():146] handle_request: status_report
2024-03-21 01:59:55,179 DEBUG   HandlerThread:7197 [handler.py:handle_request():146] handle_request: keepalive
2024-03-21 01:59:55,227 DEBUG   HandlerThread:7197 [handler.py:handle_request():146] handle_request: status_report
2024-03-21 02:00:00,180 DEBUG   HandlerThread:7197 [handler.py:handle_request():146] handle_request: keepalive
2024-03-21 02:00:00,228 DEBUG   HandlerThread:7197 [handler.py:handle_request():146] handle_request: status_report
2024-03-21 02:00:04,223 DEBUG   SenderThread:7197 [sender.py:send():378] send: stats
2024-03-21 02:00:05,181 DEBUG   HandlerThread:7197 [handler.py:handle_request():146] handle_request: keepalive
2024-03-21 02:00:06,224 DEBUG   HandlerThread:7197 [handler.py:handle_request():146] handle_request: status_report
2024-03-21 02:00:10,182 DEBUG   HandlerThread:7197 [handler.py:handle_request():146] handle_request: keepalive
2024-03-21 02:00:11,226 DEBUG   HandlerThread:7197 [handler.py:handle_request():146] handle_request: status_report
2024-03-21 02:00:15,183 DEBUG   HandlerThread:7197 [handler.py:handle_request():146] handle_request: keepalive
2024-03-21 02:00:16,227 DEBUG   HandlerThread:7197 [handler.py:handle_request():146] handle_request: status_report
2024-03-21 02:00:20,184 DEBUG   HandlerThread:7197 [handler.py:handle_request():146] handle_request: keepalive
2024-03-21 02:00:21,228 DEBUG   HandlerThread:7197 [handler.py:handle_request():146] handle_request: status_report
2024-03-21 02:00:25,185 DEBUG   HandlerThread:7197 [handler.py:handle_request():146] handle_request: keepalive
2024-03-21 02:00:26,229 DEBUG   HandlerThread:7197 [handler.py:handle_request():146] handle_request: status_report
2024-03-21 02:00:30,187 DEBUG   HandlerThread:7197 [handler.py:handle_request():146] handle_request: keepalive
2024-03-21 02:00:31,230 DEBUG   HandlerThread:7197 [handler.py:handle_request():146] handle_request: status_report
2024-03-21 02:00:34,224 DEBUG   SenderThread:7197 [sender.py:send():378] send: stats
2024-03-21 02:00:35,188 DEBUG   HandlerThread:7197 [handler.py:handle_request():146] handle_request: keepalive
2024-03-21 02:00:37,010 DEBUG   HandlerThread:7197 [handler.py:handle_request():146] handle_request: partial_history
2024-03-21 02:00:37,013 DEBUG   SenderThread:7197 [sender.py:send():378] send: history
2024-03-21 02:00:37,014 DEBUG   SenderThread:7197 [sender.py:send_request():405] send_request: summary_record
2024-03-21 02:00:37,015 DEBUG   HandlerThread:7197 [handler.py:handle_request():146] handle_request: status_report
2024-03-21 02:00:37,019 INFO    SenderThread:7197 [sender.py:_save_file():1365] saving file wandb-summary.json with policy end
2024-03-21 02:00:37,893 INFO    Thread-12 :7197 [dir_watcher.py:_on_file_modified():288] file/dir modified: /content/gdrive/MyDrive/LLM/Mistral-7B-Finetuning-Insurance/wandb/run-20240321_015458-y11xrjxv/files/wandb-summary.json
2024-03-21 02:00:42,021 DEBUG   HandlerThread:7197 [handler.py:handle_request():146] handle_request: status_report
2024-03-21 02:01:05,507 DEBUG   HandlerThread:7197 [handler.py:handle_request():146] handle_request: status_report
2024-03-21 02:01:05,510 DEBUG   HandlerThread:7197 [handler.py:handle_request():146] handle_request: status_report
2024-03-21 02:01:05,510 DEBUG   HandlerThread:7197 [handler.py:handle_request():146] handle_request: status_report
2024-03-21 02:01:05,510 DEBUG   HandlerThread:7197 [handler.py:handle_request():146] handle_request: status_report
2024-03-21 02:01:05,511 DEBUG   SenderThread:7197 [sender.py:send():378] send: stats
2024-03-21 02:01:05,717 DEBUG   HandlerThread:7197 [handler.py:handle_request():146] handle_request: keepalive
2024-03-21 02:01:07,512 DEBUG   HandlerThread:7197 [handler.py:handle_request():146] handle_request: status_report
2024-03-21 02:01:11,560 DEBUG   HandlerThread:7197 [handler.py:handle_request():146] handle_request: keepalive
2024-03-21 02:01:12,513 DEBUG   HandlerThread:7197 [handler.py:handle_request():146] handle_request: status_report
2024-03-21 02:01:17,514 DEBUG   HandlerThread:7197 [handler.py:handle_request():146] handle_request: status_report
2024-03-21 02:01:20,668 DEBUG   HandlerThread:7197 [handler.py:handle_request():146] handle_request: keepalive
2024-03-21 02:01:22,516 DEBUG   HandlerThread:7197 [handler.py:handle_request():146] handle_request: status_report
2024-03-21 02:01:26,471 DEBUG   HandlerThread:7197 [handler.py:handle_request():146] handle_request: keepalive
2024-03-21 02:01:27,517 DEBUG   HandlerThread:7197 [handler.py:handle_request():146] handle_request: status_report
2024-03-21 02:01:31,472 DEBUG   HandlerThread:7197 [handler.py:handle_request():146] handle_request: keepalive
2024-03-21 02:01:32,518 DEBUG   HandlerThread:7197 [handler.py:handle_request():146] handle_request: status_report
2024-03-21 02:01:34,225 DEBUG   SenderThread:7197 [sender.py:send():378] send: stats
2024-03-21 02:01:36,473 DEBUG   HandlerThread:7197 [handler.py:handle_request():146] handle_request: keepalive
2024-03-21 02:01:38,227 DEBUG   HandlerThread:7197 [handler.py:handle_request():146] handle_request: status_report
2024-03-21 02:01:41,476 DEBUG   HandlerThread:7197 [handler.py:handle_request():146] handle_request: keepalive
2024-03-21 02:01:43,228 DEBUG   HandlerThread:7197 [handler.py:handle_request():146] handle_request: status_report
2024-03-21 02:01:46,479 DEBUG   HandlerThread:7197 [handler.py:handle_request():146] handle_request: keepalive
2024-03-21 02:01:48,229 DEBUG   HandlerThread:7197 [handler.py:handle_request():146] handle_request: status_report
2024-03-21 02:01:51,480 DEBUG   HandlerThread:7197 [handler.py:handle_request():146] handle_request: keepalive
2024-03-21 02:01:53,229 DEBUG   HandlerThread:7197 [handler.py:handle_request():146] handle_request: status_report
2024-03-21 02:01:56,482 DEBUG   HandlerThread:7197 [handler.py:handle_request():146] handle_request: keepalive
2024-03-21 02:01:58,230 DEBUG   HandlerThread:7197 [handler.py:handle_request():146] handle_request: status_report
2024-03-21 02:02:01,483 DEBUG   HandlerThread:7197 [handler.py:handle_request():146] handle_request: keepalive
2024-03-21 02:02:03,232 DEBUG   HandlerThread:7197 [handler.py:handle_request():146] handle_request: status_report
2024-03-21 02:02:04,226 DEBUG   SenderThread:7197 [sender.py:send():378] send: stats
2024-03-21 02:02:06,484 DEBUG   HandlerThread:7197 [handler.py:handle_request():146] handle_request: keepalive
2024-03-21 02:02:09,230 DEBUG   HandlerThread:7197 [handler.py:handle_request():146] handle_request: status_report
2024-03-21 02:02:11,490 DEBUG   HandlerThread:7197 [handler.py:handle_request():146] handle_request: keepalive
2024-03-21 02:02:14,231 DEBUG   HandlerThread:7197 [handler.py:handle_request():146] handle_request: status_report
2024-03-21 02:02:16,491 DEBUG   HandlerThread:7197 [handler.py:handle_request():146] handle_request: keepalive
2024-03-21 02:02:19,232 DEBUG   HandlerThread:7197 [handler.py:handle_request():146] handle_request: status_report
2024-03-21 02:02:21,492 DEBUG   HandlerThread:7197 [handler.py:handle_request():146] handle_request: keepalive
2024-03-21 02:02:24,232 DEBUG   HandlerThread:7197 [handler.py:handle_request():146] handle_request: status_report
2024-03-21 02:02:26,494 DEBUG   HandlerThread:7197 [handler.py:handle_request():146] handle_request: keepalive
2024-03-21 02:02:29,234 DEBUG   HandlerThread:7197 [handler.py:handle_request():146] handle_request: status_report
2024-03-21 02:02:31,495 DEBUG   HandlerThread:7197 [handler.py:handle_request():146] handle_request: keepalive
2024-03-21 02:02:34,227 DEBUG   SenderThread:7197 [sender.py:send():378] send: stats
2024-03-21 02:02:35,228 DEBUG   HandlerThread:7197 [handler.py:handle_request():146] handle_request: status_report
2024-03-21 02:02:36,496 DEBUG   HandlerThread:7197 [handler.py:handle_request():146] handle_request: keepalive
2024-03-21 02:02:39,633 DEBUG   HandlerThread:7197 [handler.py:handle_request():146] handle_request: partial_history
2024-03-21 02:02:39,634 DEBUG   SenderThread:7197 [sender.py:send():378] send: history
2024-03-21 02:02:39,635 DEBUG   SenderThread:7197 [sender.py:send_request():405] send_request: summary_record
2024-03-21 02:02:39,645 INFO    SenderThread:7197 [sender.py:_save_file():1365] saving file wandb-summary.json with policy end
2024-03-21 02:02:40,359 INFO    Thread-12 :7197 [dir_watcher.py:_on_file_modified():288] file/dir modified: /content/gdrive/MyDrive/LLM/Mistral-7B-Finetuning-Insurance/wandb/run-20240321_015458-y11xrjxv/files/wandb-summary.json
2024-03-21 02:02:40,651 DEBUG   HandlerThread:7197 [handler.py:handle_request():146] handle_request: status_report
2024-03-21 02:02:41,496 DEBUG   HandlerThread:7197 [handler.py:handle_request():146] handle_request: keepalive
2024-03-21 02:02:45,651 DEBUG   HandlerThread:7197 [handler.py:handle_request():146] handle_request: status_report
2024-03-21 02:02:50,652 DEBUG   HandlerThread:7197 [handler.py:handle_request():146] handle_request: status_report
2024-03-21 02:02:58,007 DEBUG   HandlerThread:7197 [handler.py:handle_request():146] handle_request: status_report
2024-03-21 02:02:58,222 DEBUG   HandlerThread:7197 [handler.py:handle_request():146] handle_request: keepalive
2024-03-21 02:03:00,654 DEBUG   HandlerThread:7197 [handler.py:handle_request():146] handle_request: status_report
2024-03-21 02:03:04,228 DEBUG   SenderThread:7197 [sender.py:send():378] send: stats
2024-03-21 02:03:11,188 DEBUG   HandlerThread:7197 [handler.py:handle_request():146] handle_request: status_report
2024-03-21 02:03:11,195 DEBUG   HandlerThread:7197 [handler.py:handle_request():146] handle_request: keepalive
2024-03-21 02:03:16,191 DEBUG   HandlerThread:7197 [handler.py:handle_request():146] handle_request: status_report
2024-03-21 02:03:16,981 DEBUG   HandlerThread:7197 [handler.py:handle_request():146] handle_request: keepalive
2024-03-21 02:03:21,191 DEBUG   HandlerThread:7197 [handler.py:handle_request():146] handle_request: status_report
2024-03-21 02:03:22,003 DEBUG   HandlerThread:7197 [handler.py:handle_request():146] handle_request: keepalive
2024-03-21 02:03:26,192 DEBUG   HandlerThread:7197 [handler.py:handle_request():146] handle_request: status_report
2024-03-21 02:03:27,009 DEBUG   HandlerThread:7197 [handler.py:handle_request():146] handle_request: keepalive
2024-03-21 02:03:31,193 DEBUG   HandlerThread:7197 [handler.py:handle_request():146] handle_request: status_report
2024-03-21 02:03:32,010 DEBUG   HandlerThread:7197 [handler.py:handle_request():146] handle_request: keepalive
2024-03-21 02:03:34,229 DEBUG   SenderThread:7197 [sender.py:send():378] send: stats
2024-03-21 02:03:36,230 DEBUG   HandlerThread:7197 [handler.py:handle_request():146] handle_request: status_report
2024-03-21 02:03:37,011 DEBUG   HandlerThread:7197 [handler.py:handle_request():146] handle_request: keepalive
2024-03-21 02:03:41,233 DEBUG   HandlerThread:7197 [handler.py:handle_request():146] handle_request: status_report
2024-03-21 02:03:42,012 DEBUG   HandlerThread:7197 [handler.py:handle_request():146] handle_request: keepalive
2024-03-21 02:03:46,234 DEBUG   HandlerThread:7197 [handler.py:handle_request():146] handle_request: status_report
2024-03-21 02:03:47,012 DEBUG   HandlerThread:7197 [handler.py:handle_request():146] handle_request: keepalive
2024-03-21 02:03:51,235 DEBUG   HandlerThread:7197 [handler.py:handle_request():146] handle_request: status_report
2024-03-21 02:03:52,013 DEBUG   HandlerThread:7197 [handler.py:handle_request():146] handle_request: keepalive
2024-03-21 02:03:56,236 DEBUG   HandlerThread:7197 [handler.py:handle_request():146] handle_request: status_report
2024-03-21 02:03:57,014 DEBUG   HandlerThread:7197 [handler.py:handle_request():146] handle_request: keepalive
2024-03-21 02:04:01,237 DEBUG   HandlerThread:7197 [handler.py:handle_request():146] handle_request: status_report
2024-03-21 02:04:02,015 DEBUG   HandlerThread:7197 [handler.py:handle_request():146] handle_request: keepalive
2024-03-21 02:04:04,230 DEBUG   SenderThread:7197 [sender.py:send():378] send: stats
2024-03-21 02:04:07,017 DEBUG   HandlerThread:7197 [handler.py:handle_request():146] handle_request: keepalive
2024-03-21 02:04:07,236 DEBUG   HandlerThread:7197 [handler.py:handle_request():146] handle_request: status_report
2024-03-21 02:04:12,018 DEBUG   HandlerThread:7197 [handler.py:handle_request():146] handle_request: keepalive
2024-03-21 02:04:12,237 DEBUG   HandlerThread:7197 [handler.py:handle_request():146] handle_request: status_report
2024-03-21 02:04:17,020 DEBUG   HandlerThread:7197 [handler.py:handle_request():146] handle_request: keepalive
2024-03-21 02:04:17,238 DEBUG   HandlerThread:7197 [handler.py:handle_request():146] handle_request: status_report
2024-03-21 02:04:22,021 DEBUG   HandlerThread:7197 [handler.py:handle_request():146] handle_request: keepalive
2024-03-21 02:04:22,239 DEBUG   HandlerThread:7197 [handler.py:handle_request():146] handle_request: status_report
2024-03-21 02:04:27,022 DEBUG   HandlerThread:7197 [handler.py:handle_request():146] handle_request: keepalive
2024-03-21 02:04:27,240 DEBUG   HandlerThread:7197 [handler.py:handle_request():146] handle_request: status_report
2024-03-21 02:04:32,022 DEBUG   HandlerThread:7197 [handler.py:handle_request():146] handle_request: keepalive
2024-03-21 02:04:32,241 DEBUG   HandlerThread:7197 [handler.py:handle_request():146] handle_request: status_report
2024-03-21 02:04:34,231 DEBUG   SenderThread:7197 [sender.py:send():378] send: stats
2024-03-21 02:04:37,023 DEBUG   HandlerThread:7197 [handler.py:handle_request():146] handle_request: keepalive
2024-03-21 02:04:38,233 DEBUG   HandlerThread:7197 [handler.py:handle_request():146] handle_request: status_report
2024-03-21 02:04:42,025 DEBUG   HandlerThread:7197 [handler.py:handle_request():146] handle_request: keepalive
2024-03-21 02:04:43,234 DEBUG   HandlerThread:7197 [handler.py:handle_request():146] handle_request: status_report
2024-03-21 02:04:47,026 DEBUG   HandlerThread:7197 [handler.py:handle_request():146] handle_request: keepalive
2024-03-21 02:04:48,235 DEBUG   HandlerThread:7197 [handler.py:handle_request():146] handle_request: status_report
2024-03-21 02:04:52,027 DEBUG   HandlerThread:7197 [handler.py:handle_request():146] handle_request: keepalive
2024-03-21 02:04:53,236 DEBUG   HandlerThread:7197 [handler.py:handle_request():146] handle_request: status_report
2024-03-21 02:04:57,028 DEBUG   HandlerThread:7197 [handler.py:handle_request():146] handle_request: keepalive
2024-03-21 02:04:58,237 DEBUG   HandlerThread:7197 [handler.py:handle_request():146] handle_request: status_report
2024-03-21 02:05:02,029 DEBUG   HandlerThread:7197 [handler.py:handle_request():146] handle_request: keepalive
2024-03-21 02:05:03,237 DEBUG   HandlerThread:7197 [handler.py:handle_request():146] handle_request: status_report
2024-03-21 02:05:04,232 DEBUG   SenderThread:7197 [sender.py:send():378] send: stats
2024-03-21 02:05:07,030 DEBUG   HandlerThread:7197 [handler.py:handle_request():146] handle_request: keepalive
2024-03-21 02:05:09,235 DEBUG   HandlerThread:7197 [handler.py:handle_request():146] handle_request: status_report
2024-03-21 02:05:12,031 DEBUG   HandlerThread:7197 [handler.py:handle_request():146] handle_request: keepalive
2024-03-21 02:05:14,236 DEBUG   HandlerThread:7197 [handler.py:handle_request():146] handle_request: status_report
2024-03-21 02:05:17,032 DEBUG   HandlerThread:7197 [handler.py:handle_request():146] handle_request: keepalive
2024-03-21 02:05:19,237 DEBUG   HandlerThread:7197 [handler.py:handle_request():146] handle_request: status_report
2024-03-21 02:05:20,556 DEBUG   HandlerThread:7197 [handler.py:handle_request():146] handle_request: partial_history
2024-03-21 02:05:20,559 DEBUG   SenderThread:7197 [sender.py:send():378] send: history
2024-03-21 02:05:20,561 DEBUG   SenderThread:7197 [sender.py:send_request():405] send_request: summary_record
2024-03-21 02:05:20,567 INFO    SenderThread:7197 [sender.py:_save_file():1365] saving file wandb-summary.json with policy end
2024-03-21 02:05:21,134 INFO    Thread-12 :7197 [dir_watcher.py:_on_file_modified():288] file/dir modified: /content/gdrive/MyDrive/LLM/Mistral-7B-Finetuning-Insurance/wandb/run-20240321_015458-y11xrjxv/files/wandb-summary.json
2024-03-21 02:05:24,569 DEBUG   HandlerThread:7197 [handler.py:handle_request():146] handle_request: status_report
2024-03-21 02:05:47,162 DEBUG   HandlerThread:7197 [handler.py:handle_request():146] handle_request: status_report
2024-03-21 02:05:47,164 DEBUG   HandlerThread:7197 [handler.py:handle_request():146] handle_request: status_report
2024-03-21 02:05:47,165 DEBUG   HandlerThread:7197 [handler.py:handle_request():146] handle_request: status_report
2024-03-21 02:05:47,164 DEBUG   SenderThread:7197 [sender.py:send():378] send: stats
2024-03-21 02:05:47,165 DEBUG   HandlerThread:7197 [handler.py:handle_request():146] handle_request: status_report
2024-03-21 02:05:47,381 DEBUG   HandlerThread:7197 [handler.py:handle_request():146] handle_request: keepalive
2024-03-21 02:05:50,167 DEBUG   HandlerThread:7197 [handler.py:handle_request():146] handle_request: status_report
2024-03-21 02:05:55,168 DEBUG   HandlerThread:7197 [handler.py:handle_request():146] handle_request: status_report
2024-03-21 02:06:01,501 DEBUG   HandlerThread:7197 [handler.py:handle_request():146] handle_request: status_report
2024-03-21 02:06:01,532 DEBUG   HandlerThread:7197 [handler.py:handle_request():146] handle_request: keepalive
2024-03-21 02:06:04,233 DEBUG   SenderThread:7197 [sender.py:send():378] send: stats
2024-03-21 02:06:05,235 DEBUG   HandlerThread:7197 [handler.py:handle_request():146] handle_request: status_report
2024-03-21 02:06:07,238 DEBUG   HandlerThread:7197 [handler.py:handle_request():146] handle_request: keepalive
2024-03-21 02:06:10,236 DEBUG   HandlerThread:7197 [handler.py:handle_request():146] handle_request: status_report
2024-03-21 02:06:12,238 DEBUG   HandlerThread:7197 [handler.py:handle_request():146] handle_request: keepalive
2024-03-21 02:06:15,237 DEBUG   HandlerThread:7197 [handler.py:handle_request():146] handle_request: status_report
2024-03-21 02:06:17,247 DEBUG   HandlerThread:7197 [handler.py:handle_request():146] handle_request: keepalive
2024-03-21 02:06:20,239 DEBUG   HandlerThread:7197 [handler.py:handle_request():146] handle_request: status_report
2024-03-21 02:06:22,248 DEBUG   HandlerThread:7197 [handler.py:handle_request():146] handle_request: keepalive
2024-03-21 02:06:25,239 DEBUG   HandlerThread:7197 [handler.py:handle_request():146] handle_request: status_report
2024-03-21 02:06:27,249 DEBUG   HandlerThread:7197 [handler.py:handle_request():146] handle_request: keepalive
2024-03-21 02:06:30,240 DEBUG   HandlerThread:7197 [handler.py:handle_request():146] handle_request: status_report
2024-03-21 02:06:32,250 DEBUG   HandlerThread:7197 [handler.py:handle_request():146] handle_request: keepalive
2024-03-21 02:06:34,234 DEBUG   SenderThread:7197 [sender.py:send():378] send: stats
2024-03-21 02:06:36,235 DEBUG   HandlerThread:7197 [handler.py:handle_request():146] handle_request: status_report
2024-03-21 02:06:37,251 DEBUG   HandlerThread:7197 [handler.py:handle_request():146] handle_request: keepalive
2024-03-21 02:06:41,236 DEBUG   HandlerThread:7197 [handler.py:handle_request():146] handle_request: status_report
2024-03-21 02:06:42,252 DEBUG   HandlerThread:7197 [handler.py:handle_request():146] handle_request: keepalive
2024-03-21 02:06:46,237 DEBUG   HandlerThread:7197 [handler.py:handle_request():146] handle_request: status_report
2024-03-21 02:06:47,253 DEBUG   HandlerThread:7197 [handler.py:handle_request():146] handle_request: keepalive
2024-03-21 02:06:51,238 DEBUG   HandlerThread:7197 [handler.py:handle_request():146] handle_request: status_report
2024-03-21 02:06:52,254 DEBUG   HandlerThread:7197 [handler.py:handle_request():146] handle_request: keepalive
2024-03-21 02:06:56,238 DEBUG   HandlerThread:7197 [handler.py:handle_request():146] handle_request: status_report
2024-03-21 02:06:57,255 DEBUG   HandlerThread:7197 [handler.py:handle_request():146] handle_request: keepalive
2024-03-21 02:07:01,239 DEBUG   HandlerThread:7197 [handler.py:handle_request():146] handle_request: status_report
2024-03-21 02:07:02,256 DEBUG   HandlerThread:7197 [handler.py:handle_request():146] handle_request: keepalive
2024-03-21 02:07:04,235 DEBUG   SenderThread:7197 [sender.py:send():378] send: stats
2024-03-21 02:07:07,238 DEBUG   HandlerThread:7197 [handler.py:handle_request():146] handle_request: status_report
2024-03-21 02:07:07,257 DEBUG   HandlerThread:7197 [handler.py:handle_request():146] handle_request: keepalive
2024-03-21 02:07:12,239 DEBUG   HandlerThread:7197 [handler.py:handle_request():146] handle_request: status_report
2024-03-21 02:07:12,258 DEBUG   HandlerThread:7197 [handler.py:handle_request():146] handle_request: keepalive
2024-03-21 02:07:17,240 DEBUG   HandlerThread:7197 [handler.py:handle_request():146] handle_request: status_report
2024-03-21 02:07:17,259 DEBUG   HandlerThread:7197 [handler.py:handle_request():146] handle_request: keepalive
2024-03-21 02:07:22,240 DEBUG   HandlerThread:7197 [handler.py:handle_request():146] handle_request: status_report
2024-03-21 02:07:22,260 DEBUG   HandlerThread:7197 [handler.py:handle_request():146] handle_request: keepalive
2024-03-21 02:07:27,241 DEBUG   HandlerThread:7197 [handler.py:handle_request():146] handle_request: status_report
2024-03-21 02:07:27,261 DEBUG   HandlerThread:7197 [handler.py:handle_request():146] handle_request: keepalive
2024-03-21 02:07:32,242 DEBUG   HandlerThread:7197 [handler.py:handle_request():146] handle_request: status_report
2024-03-21 02:07:32,262 DEBUG   HandlerThread:7197 [handler.py:handle_request():146] handle_request: keepalive
2024-03-21 02:07:34,236 DEBUG   SenderThread:7197 [sender.py:send():378] send: stats
2024-03-21 02:07:37,263 DEBUG   HandlerThread:7197 [handler.py:handle_request():146] handle_request: keepalive
2024-03-21 02:07:38,237 DEBUG   HandlerThread:7197 [handler.py:handle_request():146] handle_request: status_report
2024-03-21 02:07:42,263 DEBUG   HandlerThread:7197 [handler.py:handle_request():146] handle_request: keepalive
2024-03-21 02:07:43,238 DEBUG   HandlerThread:7197 [handler.py:handle_request():146] handle_request: status_report
2024-03-21 02:07:47,264 DEBUG   HandlerThread:7197 [handler.py:handle_request():146] handle_request: keepalive
2024-03-21 02:07:48,239 DEBUG   HandlerThread:7197 [handler.py:handle_request():146] handle_request: status_report
2024-03-21 02:07:52,265 DEBUG   HandlerThread:7197 [handler.py:handle_request():146] handle_request: keepalive
2024-03-21 02:07:53,239 DEBUG   HandlerThread:7197 [handler.py:handle_request():146] handle_request: status_report
2024-03-21 02:07:57,266 DEBUG   HandlerThread:7197 [handler.py:handle_request():146] handle_request: keepalive
2024-03-21 02:07:58,240 DEBUG   HandlerThread:7197 [handler.py:handle_request():146] handle_request: status_report
2024-03-21 02:08:02,267 DEBUG   HandlerThread:7197 [handler.py:handle_request():146] handle_request: keepalive
2024-03-21 02:08:03,241 DEBUG   HandlerThread:7197 [handler.py:handle_request():146] handle_request: status_report
2024-03-21 02:08:04,165 DEBUG   HandlerThread:7197 [handler.py:handle_request():146] handle_request: partial_history
2024-03-21 02:08:04,170 DEBUG   SenderThread:7197 [sender.py:send():378] send: history
2024-03-21 02:08:04,171 DEBUG   SenderThread:7197 [sender.py:send_request():405] send_request: summary_record
2024-03-21 02:08:04,176 INFO    SenderThread:7197 [sender.py:_save_file():1365] saving file wandb-summary.json with policy end
2024-03-21 02:08:04,236 DEBUG   SenderThread:7197 [sender.py:send():378] send: stats
2024-03-21 02:08:04,534 INFO    Thread-12 :7197 [dir_watcher.py:_on_file_modified():288] file/dir modified: /content/gdrive/MyDrive/LLM/Mistral-7B-Finetuning-Insurance/wandb/run-20240321_015458-y11xrjxv/files/wandb-summary.json
2024-03-21 02:08:09,237 DEBUG   HandlerThread:7197 [handler.py:handle_request():146] handle_request: status_report
2024-03-21 02:08:36,561 DEBUG   HandlerThread:7197 [handler.py:handle_request():146] handle_request: status_report
2024-03-21 02:08:36,563 DEBUG   HandlerThread:7197 [handler.py:handle_request():146] handle_request: status_report
2024-03-21 02:08:36,563 DEBUG   HandlerThread:7197 [handler.py:handle_request():146] handle_request: status_report
2024-03-21 02:08:36,564 DEBUG   HandlerThread:7197 [handler.py:handle_request():146] handle_request: status_report
2024-03-21 02:08:36,564 DEBUG   HandlerThread:7197 [handler.py:handle_request():146] handle_request: status_report
2024-03-21 02:08:36,565 DEBUG   SenderThread:7197 [sender.py:send():378] send: stats
2024-03-21 02:08:36,826 DEBUG   HandlerThread:7197 [handler.py:handle_request():146] handle_request: keepalive
2024-03-21 02:08:39,566 DEBUG   HandlerThread:7197 [handler.py:handle_request():146] handle_request: status_report
2024-03-21 02:08:44,567 DEBUG   HandlerThread:7197 [handler.py:handle_request():146] handle_request: status_report
2024-03-21 02:08:49,568 DEBUG   HandlerThread:7197 [handler.py:handle_request():146] handle_request: status_report
2024-03-21 02:08:50,457 DEBUG   HandlerThread:7197 [handler.py:handle_request():146] handle_request: keepalive
2024-03-21 02:08:54,568 DEBUG   HandlerThread:7197 [handler.py:handle_request():146] handle_request: status_report
2024-03-21 02:08:56,068 DEBUG   HandlerThread:7197 [handler.py:handle_request():146] handle_request: keepalive
2024-03-21 02:08:59,569 DEBUG   HandlerThread:7197 [handler.py:handle_request():146] handle_request: status_report
2024-03-21 02:09:01,070 DEBUG   HandlerThread:7197 [handler.py:handle_request():146] handle_request: keepalive
2024-03-21 02:09:04,238 DEBUG   SenderThread:7197 [sender.py:send():378] send: stats
2024-03-21 02:09:05,239 DEBUG   HandlerThread:7197 [handler.py:handle_request():146] handle_request: status_report
2024-03-21 02:09:06,071 DEBUG   HandlerThread:7197 [handler.py:handle_request():146] handle_request: keepalive
2024-03-21 02:09:10,240 DEBUG   HandlerThread:7197 [handler.py:handle_request():146] handle_request: status_report
2024-03-21 02:09:11,072 DEBUG   HandlerThread:7197 [handler.py:handle_request():146] handle_request: keepalive
2024-03-21 02:09:15,241 DEBUG   HandlerThread:7197 [handler.py:handle_request():146] handle_request: status_report
2024-03-21 02:09:16,073 DEBUG   HandlerThread:7197 [handler.py:handle_request():146] handle_request: keepalive
2024-03-21 02:09:20,241 DEBUG   HandlerThread:7197 [handler.py:handle_request():146] handle_request: status_report
2024-03-21 02:09:21,074 DEBUG   HandlerThread:7197 [handler.py:handle_request():146] handle_request: keepalive
2024-03-21 02:09:25,242 DEBUG   HandlerThread:7197 [handler.py:handle_request():146] handle_request: status_report
2024-03-21 02:09:26,075 DEBUG   HandlerThread:7197 [handler.py:handle_request():146] handle_request: keepalive
2024-03-21 02:09:30,243 DEBUG   HandlerThread:7197 [handler.py:handle_request():146] handle_request: status_report
2024-03-21 02:09:31,076 DEBUG   HandlerThread:7197 [handler.py:handle_request():146] handle_request: keepalive
2024-03-21 02:09:34,241 DEBUG   SenderThread:7197 [sender.py:send():378] send: stats
2024-03-21 02:09:36,077 DEBUG   HandlerThread:7197 [handler.py:handle_request():146] handle_request: keepalive
2024-03-21 02:09:36,242 DEBUG   HandlerThread:7197 [handler.py:handle_request():146] handle_request: status_report
2024-03-21 02:09:41,078 DEBUG   HandlerThread:7197 [handler.py:handle_request():146] handle_request: keepalive
2024-03-21 02:09:41,243 DEBUG   HandlerThread:7197 [handler.py:handle_request():146] handle_request: status_report
2024-03-21 02:09:46,083 DEBUG   HandlerThread:7197 [handler.py:handle_request():146] handle_request: keepalive
2024-03-21 02:09:46,244 DEBUG   HandlerThread:7197 [handler.py:handle_request():146] handle_request: status_report
2024-03-21 02:09:51,084 DEBUG   HandlerThread:7197 [handler.py:handle_request():146] handle_request: keepalive
2024-03-21 02:09:51,244 DEBUG   HandlerThread:7197 [handler.py:handle_request():146] handle_request: status_report
2024-03-21 02:09:56,085 DEBUG   HandlerThread:7197 [handler.py:handle_request():146] handle_request: keepalive
2024-03-21 02:09:56,245 DEBUG   HandlerThread:7197 [handler.py:handle_request():146] handle_request: status_report
2024-03-21 02:10:01,086 DEBUG   HandlerThread:7197 [handler.py:handle_request():146] handle_request: keepalive
2024-03-21 02:10:01,246 DEBUG   HandlerThread:7197 [handler.py:handle_request():146] handle_request: status_report
2024-03-21 02:10:04,239 DEBUG   SenderThread:7197 [sender.py:send():378] send: stats
2024-03-21 02:10:06,087 DEBUG   HandlerThread:7197 [handler.py:handle_request():146] handle_request: keepalive
2024-03-21 02:10:07,241 DEBUG   HandlerThread:7197 [handler.py:handle_request():146] handle_request: status_report
2024-03-21 02:10:11,088 DEBUG   HandlerThread:7197 [handler.py:handle_request():146] handle_request: keepalive
2024-03-21 02:10:12,242 DEBUG   HandlerThread:7197 [handler.py:handle_request():146] handle_request: status_report
2024-03-21 02:10:15,267 DEBUG   HandlerThread:7197 [handler.py:handle_request():146] handle_request: partial_history
2024-03-21 02:10:15,269 DEBUG   SenderThread:7197 [sender.py:send():378] send: history
2024-03-21 02:10:15,269 DEBUG   SenderThread:7197 [sender.py:send_request():405] send_request: summary_record
2024-03-21 02:10:15,278 INFO    SenderThread:7197 [sender.py:_save_file():1365] saving file wandb-summary.json with policy end
2024-03-21 02:10:15,898 INFO    Thread-12 :7197 [dir_watcher.py:_on_file_modified():288] file/dir modified: /content/gdrive/MyDrive/LLM/Mistral-7B-Finetuning-Insurance/wandb/run-20240321_015458-y11xrjxv/files/wandb-summary.json
2024-03-21 02:10:16,089 DEBUG   HandlerThread:7197 [handler.py:handle_request():146] handle_request: keepalive
2024-03-21 02:10:17,280 DEBUG   HandlerThread:7197 [handler.py:handle_request():146] handle_request: status_report
2024-03-21 02:10:22,281 DEBUG   HandlerThread:7197 [handler.py:handle_request():146] handle_request: status_report
2024-03-21 02:10:27,282 DEBUG   HandlerThread:7197 [handler.py:handle_request():146] handle_request: status_report
2024-03-21 02:10:27,477 DEBUG   HandlerThread:7197 [handler.py:handle_request():146] handle_request: keepalive
2024-03-21 02:10:32,283 DEBUG   HandlerThread:7197 [handler.py:handle_request():146] handle_request: status_report
2024-03-21 02:10:40,003 DEBUG   HandlerThread:7197 [handler.py:handle_request():146] handle_request: status_report
2024-03-21 02:10:40,004 DEBUG   SenderThread:7197 [sender.py:send():378] send: stats
2024-03-21 02:10:40,011 DEBUG   HandlerThread:7197 [handler.py:handle_request():146] handle_request: keepalive
2024-03-21 02:10:43,007 DEBUG   HandlerThread:7197 [handler.py:handle_request():146] handle_request: status_report
2024-03-21 02:10:44,253 DEBUG   SenderThread:7197 [sender.py:send_request():405] send_request: summary_record
2024-03-21 02:10:44,255 DEBUG   HandlerThread:7197 [handler.py:handle_request():146] handle_request: partial_history
2024-03-21 02:10:44,271 INFO    SenderThread:7197 [sender.py:_save_file():1365] saving file wandb-summary.json with policy end
2024-03-21 02:10:44,276 DEBUG   SenderThread:7197 [sender.py:send_request():405] send_request: summary_record
2024-03-21 02:10:44,285 INFO    SenderThread:7197 [sender.py:_save_file():1365] saving file wandb-summary.json with policy end
2024-03-21 02:10:44,287 DEBUG   SenderThread:7197 [sender.py:send_request():405] send_request: summary_record
2024-03-21 02:10:44,296 INFO    SenderThread:7197 [sender.py:_save_file():1365] saving file wandb-summary.json with policy end
2024-03-21 02:10:44,298 DEBUG   SenderThread:7197 [sender.py:send_request():405] send_request: summary_record
2024-03-21 02:10:44,307 INFO    SenderThread:7197 [sender.py:_save_file():1365] saving file wandb-summary.json with policy end
2024-03-21 02:10:44,309 DEBUG   SenderThread:7197 [sender.py:send_request():405] send_request: summary_record
2024-03-21 02:10:44,317 INFO    SenderThread:7197 [sender.py:_save_file():1365] saving file wandb-summary.json with policy end
2024-03-21 02:10:44,319 DEBUG   SenderThread:7197 [sender.py:send():378] send: history
2024-03-21 02:10:44,319 DEBUG   SenderThread:7197 [sender.py:send_request():405] send_request: summary_record
2024-03-21 02:10:44,327 INFO    SenderThread:7197 [sender.py:_save_file():1365] saving file wandb-summary.json with policy end
2024-03-21 02:10:45,011 INFO    Thread-12 :7197 [dir_watcher.py:_on_file_modified():288] file/dir modified: /content/gdrive/MyDrive/LLM/Mistral-7B-Finetuning-Insurance/wandb/run-20240321_015458-y11xrjxv/files/wandb-summary.json
2024-03-21 02:10:45,483 DEBUG   HandlerThread:7197 [handler.py:handle_request():146] handle_request: keepalive
2024-03-21 02:10:45,484 DEBUG   HandlerThread:7197 [handler.py:handle_request():146] handle_request: keepalive
2024-03-21 02:10:48,330 DEBUG   HandlerThread:7197 [handler.py:handle_request():146] handle_request: status_report
2024-03-21 02:10:50,484 DEBUG   HandlerThread:7197 [handler.py:handle_request():146] handle_request: keepalive
2024-03-21 02:10:53,331 DEBUG   HandlerThread:7197 [handler.py:handle_request():146] handle_request: status_report
2024-03-21 02:10:55,485 DEBUG   HandlerThread:7197 [handler.py:handle_request():146] handle_request: keepalive
2024-03-21 02:10:58,332 DEBUG   HandlerThread:7197 [handler.py:handle_request():146] handle_request: status_report
2024-03-21 02:11:00,486 DEBUG   HandlerThread:7197 [handler.py:handle_request():146] handle_request: keepalive
2024-03-21 02:11:03,333 DEBUG   HandlerThread:7197 [handler.py:handle_request():146] handle_request: status_report
2024-03-21 02:11:04,246 DEBUG   SenderThread:7197 [sender.py:send():378] send: stats
2024-03-21 02:11:05,487 DEBUG   HandlerThread:7197 [handler.py:handle_request():146] handle_request: keepalive
2024-03-21 02:11:09,247 DEBUG   HandlerThread:7197 [handler.py:handle_request():146] handle_request: status_report
2024-03-21 02:11:10,488 DEBUG   HandlerThread:7197 [handler.py:handle_request():146] handle_request: keepalive
2024-03-21 02:11:10,544 DEBUG   HandlerThread:7197 [handler.py:handle_request():146] handle_request: pause
2024-03-21 02:11:10,545 INFO    HandlerThread:7197 [handler.py:handle_request_pause():708] stopping system metrics thread
2024-03-21 02:11:10,545 INFO    HandlerThread:7197 [system_monitor.py:finish():203] Stopping system monitor
2024-03-21 02:11:10,545 DEBUG   SystemMonitor:7197 [system_monitor.py:_start():179] Finished system metrics aggregation loop
2024-03-21 02:11:10,549 DEBUG   SystemMonitor:7197 [system_monitor.py:_start():183] Publishing last batch of metrics
2024-03-21 02:11:10,547 INFO    HandlerThread:7197 [interfaces.py:finish():202] Joined cpu monitor
2024-03-21 02:11:10,554 INFO    HandlerThread:7197 [interfaces.py:finish():202] Joined disk monitor
2024-03-21 02:11:10,563 INFO    HandlerThread:7197 [interfaces.py:finish():202] Joined gpu monitor
2024-03-21 02:11:10,563 INFO    HandlerThread:7197 [interfaces.py:finish():202] Joined memory monitor
2024-03-21 02:11:10,564 INFO    HandlerThread:7197 [interfaces.py:finish():202] Joined network monitor
2024-03-21 02:11:10,564 DEBUG   SenderThread:7197 [sender.py:send():378] send: stats
2024-03-21 02:11:14,565 DEBUG   HandlerThread:7197 [handler.py:handle_request():146] handle_request: status_report
2024-03-21 02:11:15,489 DEBUG   HandlerThread:7197 [handler.py:handle_request():146] handle_request: keepalive
2024-03-21 02:11:19,566 DEBUG   HandlerThread:7197 [handler.py:handle_request():146] handle_request: status_report
2024-03-21 02:11:20,490 DEBUG   HandlerThread:7197 [handler.py:handle_request():146] handle_request: keepalive
2024-03-21 02:11:24,567 DEBUG   HandlerThread:7197 [handler.py:handle_request():146] handle_request: status_report
2024-03-21 02:11:25,491 DEBUG   HandlerThread:7197 [handler.py:handle_request():146] handle_request: keepalive
2024-03-21 02:11:29,568 DEBUG   HandlerThread:7197 [handler.py:handle_request():146] handle_request: status_report
2024-03-21 02:11:30,492 DEBUG   HandlerThread:7197 [handler.py:handle_request():146] handle_request: keepalive
2024-03-21 02:11:34,569 DEBUG   HandlerThread:7197 [handler.py:handle_request():146] handle_request: status_report
2024-03-21 02:11:35,493 DEBUG   HandlerThread:7197 [handler.py:handle_request():146] handle_request: keepalive
2024-03-21 02:11:38,632 DEBUG   HandlerThread:7197 [handler.py:handle_request():146] handle_request: resume
2024-03-21 02:11:38,633 INFO    HandlerThread:7197 [handler.py:handle_request_resume():699] starting system metrics thread
2024-03-21 02:11:38,634 INFO    HandlerThread:7197 [system_monitor.py:start():194] Starting system monitor
2024-03-21 02:11:38,634 INFO    SystemMonitor:7197 [system_monitor.py:_start():158] Starting system asset monitoring threads
2024-03-21 02:11:38,637 INFO    SystemMonitor:7197 [interfaces.py:start():190] Started cpu monitoring
2024-03-21 02:11:38,640 INFO    SystemMonitor:7197 [interfaces.py:start():190] Started disk monitoring
2024-03-21 02:11:38,642 INFO    SystemMonitor:7197 [interfaces.py:start():190] Started gpu monitoring
2024-03-21 02:11:38,649 INFO    SystemMonitor:7197 [interfaces.py:start():190] Started memory monitoring
2024-03-21 02:11:38,651 INFO    SystemMonitor:7197 [interfaces.py:start():190] Started network monitoring
2024-03-21 02:11:39,570 DEBUG   HandlerThread:7197 [handler.py:handle_request():146] handle_request: status_report
2024-03-21 02:11:40,615 DEBUG   HandlerThread:7197 [handler.py:handle_request():146] handle_request: keepalive