parler-tts-mini-v0.1-expresso-concatenated-combined
/
wandb
/run-20240513_204652-m0g0ap7d
/logs
/debug-internal.log
| 2024-05-13 20:46:52,823 INFO StreamThr :1244775 [internal.py:wandb_internal():85] W&B internal server running at pid: 1244775, started at: 2024-05-13 20:46:52.823043 | |
| 2024-05-13 20:46:52,825 DEBUG HandlerThread:1244775 [handler.py:handle_request():158] handle_request: status | |
| 2024-05-13 20:46:52,826 INFO WriterThread:1244775 [datastore.py:open_for_write():87] open: /raid/sanchit/parler-tts-mini-v0.1-expresso-concatenated-combined/wandb/run-20240513_204652-m0g0ap7d/run-m0g0ap7d.wandb | |
| 2024-05-13 20:46:52,828 DEBUG SenderThread:1244775 [sender.py:send():378] send: header | |
| 2024-05-13 20:46:52,829 DEBUG SenderThread:1244775 [sender.py:send():378] send: run | |
| 2024-05-13 20:46:53,223 INFO SenderThread:1244775 [dir_watcher.py:__init__():211] watching files in: /raid/sanchit/parler-tts-mini-v0.1-expresso-concatenated-combined/wandb/run-20240513_204652-m0g0ap7d/files | |
| 2024-05-13 20:46:53,223 INFO SenderThread:1244775 [sender.py:_start_run_threads():1123] run started: m0g0ap7d with start time 1715626012.822173 | |
| 2024-05-13 20:46:53,229 DEBUG HandlerThread:1244775 [handler.py:handle_request():158] handle_request: check_version | |
| 2024-05-13 20:46:53,229 DEBUG SenderThread:1244775 [sender.py:send_request():405] send_request: check_version | |
| 2024-05-13 20:46:53,293 DEBUG HandlerThread:1244775 [handler.py:handle_request():158] handle_request: run_start | |
| 2024-05-13 20:46:53,323 DEBUG HandlerThread:1244775 [system_info.py:__init__():26] System info init | |
| 2024-05-13 20:46:53,323 DEBUG HandlerThread:1244775 [system_info.py:__init__():41] System info init done | |
| 2024-05-13 20:46:53,323 INFO HandlerThread:1244775 [system_monitor.py:start():194] Starting system monitor | |
| 2024-05-13 20:46:53,323 INFO SystemMonitor:1244775 [system_monitor.py:_start():158] Starting system asset monitoring threads | |
| 2024-05-13 20:46:53,324 INFO HandlerThread:1244775 [system_monitor.py:probe():214] Collecting system info | |
| 2024-05-13 20:46:53,324 INFO SystemMonitor:1244775 [interfaces.py:start():188] Started cpu monitoring | |
| 2024-05-13 20:46:53,325 INFO SystemMonitor:1244775 [interfaces.py:start():188] Started disk monitoring | |
| 2024-05-13 20:46:53,325 INFO SystemMonitor:1244775 [interfaces.py:start():188] Started gpu monitoring | |
| 2024-05-13 20:46:53,327 INFO SystemMonitor:1244775 [interfaces.py:start():188] Started memory monitoring | |
| 2024-05-13 20:46:53,329 INFO SystemMonitor:1244775 [interfaces.py:start():188] Started network monitoring | |
| 2024-05-13 20:46:53,365 DEBUG HandlerThread:1244775 [system_info.py:probe():150] Probing system | |
| 2024-05-13 20:46:53,366 DEBUG HandlerThread:1244775 [system_info.py:_probe_git():135] Probing git | |
| 2024-05-13 20:46:53,372 DEBUG HandlerThread:1244775 [system_info.py:_probe_git():143] Probing git done | |
| 2024-05-13 20:46:53,372 DEBUG HandlerThread:1244775 [system_info.py:probe():198] Probing system done | |
| 2024-05-13 20:46:53,372 DEBUG HandlerThread:1244775 [system_monitor.py:probe():223] {'os': 'Linux-5.4.0-166-generic-x86_64-with-glibc2.31', 'python': '3.11.9', 'heartbeatAt': '2024-05-13T18:46:53.365083', 'startedAt': '2024-05-13T18:46:52.816759', 'docker': None, 'cuda': None, 'args': ('finetuning_concatenated_config.json',), 'state': 'running', 'program': '/raid/sanchit/parler-tts-mini-v0.1-expresso-concatenated-combined/run_parler_tts_training.py', 'codePathLocal': 'run_parler_tts_training.py', 'codePath': 'run_parler_tts_training.py', 'git': {'remote': 'https://huggingface.co/sanchit-gandhi/parler-tts-mini-v0.1-expresso-concatenated-combined', 'commit': '50ba4323d7b8bb052629aa1b88283b9df081a821'}, 'email': '[email protected]', 'root': '/raid/sanchit/parler-tts-mini-v0.1-expresso-concatenated-combined', 'host': 'hf-dgx-01', 'username': 'sanchit', 'executable': '/home/sanchit/miniconda3/envs/venv/bin/python', 'cpu_count': 64, 'cpu_count_logical': 128, 'cpu_freq': {'current': 2257.736234375, 'min': 1500.0, 'max': 2250.0}, 'cpu_freq_per_core': [{'current': 1795.281, 'min': 1500.0, 'max': 2250.0}, {'current': 1794.292, 'min': 1500.0, 'max': 2250.0}, {'current': 1795.78, 'min': 1500.0, 'max': 2250.0}, {'current': 1792.55, 'min': 1500.0, 'max': 2250.0}, {'current': 1742.094, 'min': 1500.0, 'max': 2250.0}, {'current': 3026.54, 'min': 1500.0, 'max': 2250.0}, {'current': 1786.214, 'min': 1500.0, 'max': 2250.0}, {'current': 1742.547, 'min': 1500.0, 'max': 2250.0}, {'current': 1728.916, 'min': 1500.0, 'max': 2250.0}, {'current': 1734.023, 'min': 1500.0, 'max': 2250.0}, {'current': 3195.219, 'min': 1500.0, 'max': 2250.0}, {'current': 1733.722, 'min': 1500.0, 'max': 2250.0}, {'current': 3341.911, 'min': 1500.0, 'max': 2250.0}, {'current': 3325.608, 'min': 1500.0, 'max': 2250.0}, {'current': 3233.258, 'min': 1500.0, 'max': 2250.0}, {'current': 1669.847, 'min': 1500.0, 'max': 2250.0}, {'current': 1791.948, 'min': 1500.0, 'max': 2250.0}, {'current': 1796.794, 'min': 1500.0, 'max': 2250.0}, {'current': 1791.49, 'min': 1500.0, 'max': 2250.0}, {'current': 1793.945, 'min': 1500.0, 'max': 2250.0}, {'current': 3342.943, 'min': 1500.0, 'max': 2250.0}, {'current': 1669.791, 'min': 1500.0, 'max': 2250.0}, {'current': 1669.593, 'min': 1500.0, 'max': 2250.0}, {'current': 1694.312, 'min': 1500.0, 'max': 2250.0}, {'current': 1873.727, 'min': 1500.0, 'max': 2250.0}, {'current': 1724.813, 'min': 1500.0, 'max': 2250.0}, {'current': 2354.471, 'min': 1500.0, 'max': 2250.0}, {'current': 1718.662, 'min': 1500.0, 'max': 2250.0}, {'current': 1670.588, 'min': 1500.0, 'max': 2250.0}, {'current': 1665.577, 'min': 1500.0, 'max': 2250.0}, {'current': 1616.671, 'min': 1500.0, 'max': 2250.0}, {'current': 2080.81, 'min': 1500.0, 'max': 2250.0}, {'current': 1670.666, 'min': 1500.0, 'max': 2250.0}, {'current': 1652.559, 'min': 1500.0, 'max': 2250.0}, {'current': 3323.654, 'min': 1500.0, 'max': 2250.0}, {'current': 1671.311, 'min': 1500.0, 'max': 2250.0}, {'current': 1726.286, 'min': 1500.0, 'max': 2250.0}, {'current': 1670.365, 'min': 1500.0, 'max': 2250.0}, {'current': 3320.57, 'min': 1500.0, 'max': 2250.0}, {'current': 1669.941, 'min': 1500.0, 'max': 2250.0}, {'current': 1791.021, 'min': 1500.0, 'max': 2250.0}, {'current': 1796.246, 'min': 1500.0, 'max': 2250.0}, {'current': 1793.946, 'min': 1500.0, 'max': 2250.0}, {'current': 1794.848, 'min': 1500.0, 'max': 2250.0}, {'current': 3339.327, 'min': 1500.0, 'max': 2250.0}, {'current': 3344.315, 'min': 1500.0, 'max': 2250.0}, {'current': 3338.901, 'min': 1500.0, 'max': 2250.0}, {'current': 1668.541, 'min': 1500.0, 'max': 2250.0}, {'current': 1794.526, 'min': 1500.0, 'max': 2250.0}, {'current': 1792.886, 'min': 1500.0, 'max': 2250.0}, {'current': 1796.844, 'min': 1500.0, 'max': 2250.0}, {'current': 1793.81, 'min': 1500.0, 'max': 2250.0}, {'current': 1724.861, 'min': 1500.0, 'max': 2250.0}, {'current': 2294.458, 'min': 1500.0, 'max': 2250.0}, {'current': 1720.835, 'min': 1500.0, 'max': 2250.0}, {'current': 1720.155, 'min': 1500.0, 'max': 2250.0}, {'current': 1668.96, 'min': 1500.0, 'max': 2250.0}, {'current': 1976.5, 'min': 1500.0, 'max': 2250.0}, {'current': 2241.578, 'min': 1500.0, 'max': 2250.0}, {'current': 1671.964, 'min': 1500.0, 'max': 2250.0}, {'current': 3319.623, 'min': 1500.0, 'max': 2250.0}, {'current': 1670.777, 'min': 1500.0, 'max': 2250.0}, {'current': 1670.389, 'min': 1500.0, 'max': 2250.0}, {'current': 1669.629, 'min': 1500.0, 'max': 2250.0}, {'current': 1794.19, 'min': 1500.0, 'max': 2250.0}, {'current': 1794.138, 'min': 1500.0, 'max': 2250.0}, {'current': 1796.317, 'min': 1500.0, 'max': 2250.0}, {'current': 1792.821, 'min': 1500.0, 'max': 2250.0}, {'current': 1794.716, 'min': 1500.0, 'max': 2250.0}, {'current': 1793.624, 'min': 1500.0, 'max': 2250.0}, {'current': 1796.346, 'min': 1500.0, 'max': 2250.0}, {'current': 1793.897, 'min': 1500.0, 'max': 2250.0}, {'current': 1735.424, 'min': 1500.0, 'max': 2250.0}, {'current': 1738.64, 'min': 1500.0, 'max': 2250.0}, {'current': 1979.998, 'min': 1500.0, 'max': 2250.0}, {'current': 1737.286, 'min': 1500.0, 'max': 2250.0}, {'current': 3313.748, 'min': 1500.0, 'max': 2250.0}, {'current': 3337.223, 'min': 1500.0, 'max': 2250.0}, {'current': 1671.416, 'min': 1500.0, 'max': 2250.0}, {'current': 1670.005, 'min': 1500.0, 'max': 2250.0}, {'current': 1794.276, 'min': 1500.0, 'max': 2250.0}, {'current': 1738.22, 'min': 1500.0, 'max': 2250.0}, {'current': 1742.737, 'min': 1500.0, 'max': 2250.0}, {'current': 1770.535, 'min': 1500.0, 'max': 2250.0}, {'current': 3320.252, 'min': 1500.0, 'max': 2250.0}, {'current': 1671.037, 'min': 1500.0, 'max': 2250.0}, {'current': 1669.549, 'min': 1500.0, 'max': 2250.0}, {'current': 1670.948, 'min': 1500.0, 'max': 2250.0}, {'current': 2843.391, 'min': 1500.0, 'max': 2250.0}, {'current': 2348.589, 'min': 1500.0, 'max': 2250.0}, {'current': 3287.915, 'min': 1500.0, 'max': 2250.0}, {'current': 2340.192, 'min': 1500.0, 'max': 2250.0}, {'current': 2426.358, 'min': 1500.0, 'max': 2250.0}, {'current': 2415.833, 'min': 1500.0, 'max': 2250.0}, {'current': 2419.416, 'min': 1500.0, 'max': 2250.0}, {'current': 2277.433, 'min': 1500.0, 'max': 2250.0}, {'current': 2365.562, 'min': 1500.0, 'max': 2250.0}, {'current': 2400.6, 'min': 1500.0, 'max': 2250.0}, {'current': 2075.143, 'min': 1500.0, 'max': 2250.0}, {'current': 2382.295, 'min': 1500.0, 'max': 2250.0}, {'current': 3066.339, 'min': 1500.0, 'max': 2250.0}, {'current': 2466.631, 'min': 1500.0, 'max': 2250.0}, {'current': 3100.81, 'min': 1500.0, 'max': 2250.0}, {'current': 2421.93, 'min': 1500.0, 'max': 2250.0}, {'current': 3233.829, 'min': 1500.0, 'max': 2250.0}, {'current': 2234.583, 'min': 1500.0, 'max': 2250.0}, {'current': 2452.089, 'min': 1500.0, 'max': 2250.0}, {'current': 2975.985, 'min': 1500.0, 'max': 2250.0}, {'current': 3301.512, 'min': 1500.0, 'max': 2250.0}, {'current': 3336.905, 'min': 1500.0, 'max': 2250.0}, {'current': 2984.87, 'min': 1500.0, 'max': 2250.0}, {'current': 2384.306, 'min': 1500.0, 'max': 2250.0}, {'current': 2965.197, 'min': 1500.0, 'max': 2250.0}, {'current': 1929.067, 'min': 1500.0, 'max': 2250.0}, {'current': 1986.731, 'min': 1500.0, 'max': 2250.0}, {'current': 1999.412, 'min': 1500.0, 'max': 2250.0}, {'current': 2477.541, 'min': 1500.0, 'max': 2250.0}, {'current': 3111.851, 'min': 1500.0, 'max': 2250.0}, {'current': 2009.907, 'min': 1500.0, 'max': 2250.0}, {'current': 1993.784, 'min': 1500.0, 'max': 2250.0}, {'current': 2144.459, 'min': 1500.0, 'max': 2250.0}, {'current': 3337.426, 'min': 1500.0, 'max': 2250.0}, {'current': 3320.114, 'min': 1500.0, 'max': 2250.0}, {'current': 2169.719, 'min': 1500.0, 'max': 2250.0}, {'current': 3308.644, 'min': 1500.0, 'max': 2250.0}, {'current': 2111.633, 'min': 1500.0, 'max': 2250.0}, {'current': 2123.71, 'min': 1500.0, 'max': 2250.0}, {'current': 2153.49, 'min': 1500.0, 'max': 2250.0}], 'disk': {'/': {'total': 1757.8785285949707, 'used': 1663.5005989074707}}, 'gpu': 'NVIDIA A100-SXM4-80GB', 'gpu_count': 5, 'gpu_devices': [{'name': 'NVIDIA A100-SXM4-80GB', 'memory_total': 85899345920}, {'name': 'NVIDIA A100-SXM4-80GB', 'memory_total': 85899345920}, {'name': 'NVIDIA A100-SXM4-80GB', 'memory_total': 85899345920}, {'name': 'NVIDIA DGX Display', 'memory_total': 4294967296}, {'name': 'NVIDIA A100-SXM4-80GB', 'memory_total': 85899345920}], 'memory': {'total': 503.5396919250488}} | |
| 2024-05-13 20:46:53,372 INFO HandlerThread:1244775 [system_monitor.py:probe():224] Finished collecting system info | |
| 2024-05-13 20:46:53,372 INFO HandlerThread:1244775 [system_monitor.py:probe():227] Publishing system info | |
| 2024-05-13 20:46:53,372 DEBUG HandlerThread:1244775 [system_info.py:_save_conda():207] Saving list of conda packages installed into the current environment | |
| 2024-05-13 20:46:53,387 ERROR gpu :1244775 [interfaces.py:monitor():142] Failed to sample metric: Not Supported | |
| 2024-05-13 20:46:53,400 ERROR gpu :1244775 [interfaces.py:monitor():142] Failed to sample metric: Not Supported | |
| 2024-05-13 20:46:54,224 INFO Thread-12 :1244775 [dir_watcher.py:_on_file_created():271] file/dir created: /raid/sanchit/parler-tts-mini-v0.1-expresso-concatenated-combined/wandb/run-20240513_204652-m0g0ap7d/files/conda-environment.yaml | |
| 2024-05-13 20:46:55,418 ERROR gpu :1244775 [interfaces.py:monitor():142] Failed to sample metric: Not Supported | |
| 2024-05-13 20:46:55,429 ERROR gpu :1244775 [interfaces.py:monitor():142] Failed to sample metric: Not Supported | |
| 2024-05-13 20:46:55,741 DEBUG HandlerThread:1244775 [system_info.py:_save_conda():222] Saving conda packages done | |
| 2024-05-13 20:46:55,742 INFO HandlerThread:1244775 [system_monitor.py:probe():229] Finished publishing system info | |
| 2024-05-13 20:46:55,750 DEBUG SenderThread:1244775 [sender.py:send():378] send: files | |
| 2024-05-13 20:46:55,750 INFO SenderThread:1244775 [sender.py:_save_file():1389] saving file wandb-metadata.json with policy now | |
| 2024-05-13 20:46:55,863 DEBUG HandlerThread:1244775 [handler.py:handle_request():158] handle_request: python_packages | |
| 2024-05-13 20:46:55,863 DEBUG HandlerThread:1244775 [handler.py:handle_request():158] handle_request: stop_status | |
| 2024-05-13 20:46:55,863 DEBUG SenderThread:1244775 [sender.py:send_request():405] send_request: python_packages | |
| 2024-05-13 20:46:55,865 DEBUG SenderThread:1244775 [sender.py:send_request():405] send_request: stop_status | |
| 2024-05-13 20:46:56,093 DEBUG SenderThread:1244775 [sender.py:send():378] send: telemetry | |
| 2024-05-13 20:46:56,093 DEBUG SenderThread:1244775 [sender.py:send():378] send: config | |
| 2024-05-13 20:46:56,224 INFO Thread-12 :1244775 [dir_watcher.py:_on_file_modified():288] file/dir modified: /raid/sanchit/parler-tts-mini-v0.1-expresso-concatenated-combined/wandb/run-20240513_204652-m0g0ap7d/files/conda-environment.yaml | |
| 2024-05-13 20:46:56,224 INFO Thread-12 :1244775 [dir_watcher.py:_on_file_created():271] file/dir created: /raid/sanchit/parler-tts-mini-v0.1-expresso-concatenated-combined/wandb/run-20240513_204652-m0g0ap7d/files/wandb-metadata.json | |
| 2024-05-13 20:46:56,224 INFO Thread-12 :1244775 [dir_watcher.py:_on_file_created():271] file/dir created: /raid/sanchit/parler-tts-mini-v0.1-expresso-concatenated-combined/wandb/run-20240513_204652-m0g0ap7d/files/requirements.txt | |
| 2024-05-13 20:46:56,224 INFO Thread-12 :1244775 [dir_watcher.py:_on_file_created():271] file/dir created: /raid/sanchit/parler-tts-mini-v0.1-expresso-concatenated-combined/wandb/run-20240513_204652-m0g0ap7d/files/output.log | |
| 2024-05-13 20:46:56,261 INFO wandb-upload_0:1244775 [upload_job.py:push():130] Uploaded file /tmp/tmpewnm1an9wandb/l0duo91p-wandb-metadata.json | |
| 2024-05-13 20:46:58,225 INFO Thread-12 :1244775 [dir_watcher.py:_on_file_modified():288] file/dir modified: /raid/sanchit/parler-tts-mini-v0.1-expresso-concatenated-combined/wandb/run-20240513_204652-m0g0ap7d/files/output.log | |
| 2024-05-13 20:46:58,331 ERROR gpu :1244775 [interfaces.py:monitor():142] Failed to sample metric: Not Supported | |
| 2024-05-13 20:46:58,343 ERROR gpu :1244775 [interfaces.py:monitor():142] Failed to sample metric: Not Supported | |
| 2024-05-13 20:46:58,386 DEBUG HandlerThread:1244775 [handler.py:handle_request():158] handle_request: status_report | |
| 2024-05-13 20:47:00,362 ERROR gpu :1244775 [interfaces.py:monitor():142] Failed to sample metric: Not Supported | |
| 2024-05-13 20:47:00,375 ERROR gpu :1244775 [interfaces.py:monitor():142] Failed to sample metric: Not Supported | |
| 2024-05-13 20:47:03,387 DEBUG HandlerThread:1244775 [handler.py:handle_request():158] handle_request: status_report | |
| 2024-05-13 20:47:03,431 ERROR gpu :1244775 [interfaces.py:monitor():142] Failed to sample metric: Not Supported | |
| 2024-05-13 20:47:03,443 ERROR gpu :1244775 [interfaces.py:monitor():142] Failed to sample metric: Not Supported | |
| 2024-05-13 20:47:05,466 ERROR gpu :1244775 [interfaces.py:monitor():142] Failed to sample metric: Not Supported | |
| 2024-05-13 20:47:05,478 ERROR gpu :1244775 [interfaces.py:monitor():142] Failed to sample metric: Not Supported | |
| 2024-05-13 20:47:08,388 DEBUG HandlerThread:1244775 [handler.py:handle_request():158] handle_request: status_report | |
| 2024-05-13 20:47:08,679 ERROR gpu :1244775 [interfaces.py:monitor():142] Failed to sample metric: Not Supported | |
| 2024-05-13 20:47:08,692 ERROR gpu :1244775 [interfaces.py:monitor():142] Failed to sample metric: Not Supported | |
| 2024-05-13 20:47:10,713 ERROR gpu :1244775 [interfaces.py:monitor():142] Failed to sample metric: Not Supported | |
| 2024-05-13 20:47:10,724 ERROR gpu :1244775 [interfaces.py:monitor():142] Failed to sample metric: Not Supported | |
| 2024-05-13 20:47:10,863 DEBUG HandlerThread:1244775 [handler.py:handle_request():158] handle_request: stop_status | |
| 2024-05-13 20:47:10,863 DEBUG SenderThread:1244775 [sender.py:send_request():405] send_request: stop_status | |
| 2024-05-13 20:47:12,746 ERROR gpu :1244775 [interfaces.py:monitor():142] Failed to sample metric: Not Supported | |
| 2024-05-13 20:47:12,757 ERROR gpu :1244775 [interfaces.py:monitor():142] Failed to sample metric: Not Supported | |
| 2024-05-13 20:47:14,094 DEBUG HandlerThread:1244775 [handler.py:handle_request():158] handle_request: status_report | |
| 2024-05-13 20:47:15,627 ERROR gpu :1244775 [interfaces.py:monitor():142] Failed to sample metric: Not Supported | |
| 2024-05-13 20:47:15,637 ERROR gpu :1244775 [interfaces.py:monitor():142] Failed to sample metric: Not Supported | |
| 2024-05-13 20:47:17,658 ERROR gpu :1244775 [interfaces.py:monitor():142] Failed to sample metric: Not Supported | |
| 2024-05-13 20:47:17,668 ERROR gpu :1244775 [interfaces.py:monitor():142] Failed to sample metric: Not Supported | |
| 2024-05-13 20:47:19,096 DEBUG HandlerThread:1244775 [handler.py:handle_request():158] handle_request: status_report | |
| 2024-05-13 20:47:20,779 ERROR gpu :1244775 [interfaces.py:monitor():142] Failed to sample metric: Not Supported | |
| 2024-05-13 20:47:20,799 ERROR gpu :1244775 [interfaces.py:monitor():142] Failed to sample metric: Not Supported | |
| 2024-05-13 20:47:22,817 ERROR gpu :1244775 [interfaces.py:monitor():142] Failed to sample metric: Not Supported | |
| 2024-05-13 20:47:22,830 ERROR gpu :1244775 [interfaces.py:monitor():142] Failed to sample metric: Not Supported | |
| 2024-05-13 20:47:24,099 DEBUG HandlerThread:1244775 [handler.py:handle_request():158] handle_request: status_report | |
| 2024-05-13 20:47:24,858 ERROR gpu :1244775 [interfaces.py:monitor():142] Failed to sample metric: Not Supported | |
| 2024-05-13 20:47:24,870 ERROR gpu :1244775 [interfaces.py:monitor():142] Failed to sample metric: Not Supported | |
| 2024-05-13 20:47:25,233 INFO Thread-12 :1244775 [dir_watcher.py:_on_file_modified():288] file/dir modified: /raid/sanchit/parler-tts-mini-v0.1-expresso-concatenated-combined/wandb/run-20240513_204652-m0g0ap7d/files/config.yaml | |
| 2024-05-13 20:47:25,863 DEBUG HandlerThread:1244775 [handler.py:handle_request():158] handle_request: stop_status | |
| 2024-05-13 20:47:25,863 DEBUG SenderThread:1244775 [sender.py:send_request():405] send_request: stop_status | |
| 2024-05-13 20:47:27,736 ERROR gpu :1244775 [interfaces.py:monitor():142] Failed to sample metric: Not Supported | |
| 2024-05-13 20:47:27,747 ERROR gpu :1244775 [interfaces.py:monitor():142] Failed to sample metric: Not Supported | |
| 2024-05-13 20:47:30,092 DEBUG HandlerThread:1244775 [handler.py:handle_request():158] handle_request: status_report | |
| 2024-05-13 20:47:31,414 ERROR gpu :1244775 [interfaces.py:aggregate():159] Failed to serialize metric: division by zero | |
| 2024-05-13 20:47:31,434 ERROR gpu :1244775 [interfaces.py:monitor():142] Failed to sample metric: Not Supported | |
| 2024-05-13 20:47:31,462 ERROR gpu :1244775 [interfaces.py:monitor():142] Failed to sample metric: Not Supported | |
| 2024-05-13 20:47:34,494 ERROR gpu :1244775 [interfaces.py:monitor():142] Failed to sample metric: Not Supported | |
| 2024-05-13 20:47:34,518 ERROR gpu :1244775 [interfaces.py:monitor():142] Failed to sample metric: Not Supported | |
| 2024-05-13 20:47:35,093 DEBUG HandlerThread:1244775 [handler.py:handle_request():158] handle_request: status_report | |
| 2024-05-13 20:47:36,569 ERROR gpu :1244775 [interfaces.py:monitor():142] Failed to sample metric: Not Supported | |
| 2024-05-13 20:47:36,599 ERROR gpu :1244775 [interfaces.py:monitor():142] Failed to sample metric: Not Supported | |
| 2024-05-13 20:47:38,635 ERROR gpu :1244775 [interfaces.py:monitor():142] Failed to sample metric: Not Supported | |
| 2024-05-13 20:47:38,658 ERROR gpu :1244775 [interfaces.py:monitor():142] Failed to sample metric: Not Supported | |
| 2024-05-13 20:47:40,093 DEBUG HandlerThread:1244775 [handler.py:handle_request():158] handle_request: status_report | |
| 2024-05-13 20:47:40,863 DEBUG HandlerThread:1244775 [handler.py:handle_request():158] handle_request: stop_status | |
| 2024-05-13 20:47:40,864 DEBUG SenderThread:1244775 [sender.py:send_request():405] send_request: stop_status | |
| 2024-05-13 20:47:41,560 ERROR gpu :1244775 [interfaces.py:monitor():142] Failed to sample metric: Not Supported | |
| 2024-05-13 20:47:41,584 ERROR gpu :1244775 [interfaces.py:monitor():142] Failed to sample metric: Not Supported | |
| 2024-05-13 20:47:43,631 ERROR gpu :1244775 [interfaces.py:monitor():142] Failed to sample metric: Not Supported | |
| 2024-05-13 20:47:43,652 ERROR gpu :1244775 [interfaces.py:monitor():142] Failed to sample metric: Not Supported | |
| 2024-05-13 20:47:46,084 DEBUG HandlerThread:1244775 [handler.py:handle_request():158] handle_request: status_report | |
| 2024-05-13 20:47:46,570 ERROR gpu :1244775 [interfaces.py:monitor():142] Failed to sample metric: Not Supported | |
| 2024-05-13 20:47:46,604 ERROR gpu :1244775 [interfaces.py:monitor():142] Failed to sample metric: Not Supported | |
| 2024-05-13 20:47:48,647 ERROR gpu :1244775 [interfaces.py:monitor():142] Failed to sample metric: Not Supported | |
| 2024-05-13 20:47:48,659 ERROR gpu :1244775 [interfaces.py:monitor():142] Failed to sample metric: Not Supported | |
| 2024-05-13 20:47:51,084 DEBUG HandlerThread:1244775 [handler.py:handle_request():158] handle_request: status_report | |
| 2024-05-13 20:47:51,664 ERROR gpu :1244775 [interfaces.py:monitor():142] Failed to sample metric: Not Supported | |
| 2024-05-13 20:47:51,686 ERROR gpu :1244775 [interfaces.py:monitor():142] Failed to sample metric: Not Supported | |
| 2024-05-13 20:47:53,329 DEBUG SystemMonitor:1244775 [system_monitor.py:_start():172] Starting system metrics aggregation loop | |
| 2024-05-13 20:47:53,333 DEBUG SenderThread:1244775 [sender.py:send():378] send: stats | |
| 2024-05-13 20:47:53,709 ERROR gpu :1244775 [interfaces.py:monitor():142] Failed to sample metric: Not Supported | |
| 2024-05-13 20:47:53,724 ERROR gpu :1244775 [interfaces.py:monitor():142] Failed to sample metric: Not Supported | |
| 2024-05-13 20:47:55,863 DEBUG HandlerThread:1244775 [handler.py:handle_request():158] handle_request: stop_status | |
| 2024-05-13 20:47:55,864 DEBUG SenderThread:1244775 [sender.py:send_request():405] send_request: stop_status | |
| 2024-05-13 20:47:56,593 ERROR gpu :1244775 [interfaces.py:monitor():142] Failed to sample metric: Not Supported | |
| 2024-05-13 20:47:56,607 ERROR gpu :1244775 [interfaces.py:monitor():142] Failed to sample metric: Not Supported | |
| 2024-05-13 20:47:57,080 DEBUG HandlerThread:1244775 [handler.py:handle_request():158] handle_request: status_report | |
| 2024-05-13 20:47:58,627 ERROR gpu :1244775 [interfaces.py:monitor():142] Failed to sample metric: Not Supported | |
| 2024-05-13 20:47:58,641 ERROR gpu :1244775 [interfaces.py:monitor():142] Failed to sample metric: Not Supported | |
| 2024-05-13 20:48:01,653 ERROR gpu :1244775 [interfaces.py:monitor():142] Failed to sample metric: Not Supported | |
| 2024-05-13 20:48:01,664 ERROR gpu :1244775 [interfaces.py:monitor():142] Failed to sample metric: Not Supported | |
| 2024-05-13 20:48:02,081 DEBUG HandlerThread:1244775 [handler.py:handle_request():158] handle_request: status_report | |
| 2024-05-13 20:48:03,684 ERROR gpu :1244775 [interfaces.py:monitor():142] Failed to sample metric: Not Supported | |
| 2024-05-13 20:48:03,696 ERROR gpu :1244775 [interfaces.py:monitor():142] Failed to sample metric: Not Supported | |
| 2024-05-13 20:48:06,665 ERROR gpu :1244775 [interfaces.py:monitor():142] Failed to sample metric: Not Supported | |
| 2024-05-13 20:48:06,679 ERROR gpu :1244775 [interfaces.py:monitor():142] Failed to sample metric: Not Supported | |
| 2024-05-13 20:48:07,082 DEBUG HandlerThread:1244775 [handler.py:handle_request():158] handle_request: status_report | |
| 2024-05-13 20:48:10,344 ERROR gpu :1244775 [interfaces.py:aggregate():159] Failed to serialize metric: division by zero | |
| 2024-05-13 20:48:10,366 ERROR gpu :1244775 [interfaces.py:monitor():142] Failed to sample metric: Not Supported | |
| 2024-05-13 20:48:10,380 ERROR gpu :1244775 [interfaces.py:monitor():142] Failed to sample metric: Not Supported | |
| 2024-05-13 20:48:10,863 DEBUG HandlerThread:1244775 [handler.py:handle_request():158] handle_request: stop_status | |
| 2024-05-13 20:48:10,864 DEBUG SenderThread:1244775 [sender.py:send_request():405] send_request: stop_status | |
| 2024-05-13 20:48:13,048 DEBUG HandlerThread:1244775 [handler.py:handle_request():158] handle_request: status_report | |
| 2024-05-13 20:48:13,506 ERROR gpu :1244775 [interfaces.py:monitor():142] Failed to sample metric: Not Supported | |
| 2024-05-13 20:48:13,529 ERROR gpu :1244775 [interfaces.py:monitor():142] Failed to sample metric: Not Supported | |
| 2024-05-13 20:48:15,558 ERROR gpu :1244775 [interfaces.py:monitor():142] Failed to sample metric: Not Supported | |
| 2024-05-13 20:48:15,586 ERROR gpu :1244775 [interfaces.py:monitor():142] Failed to sample metric: Not Supported | |
| 2024-05-13 20:48:18,050 DEBUG HandlerThread:1244775 [handler.py:handle_request():158] handle_request: status_report | |
| 2024-05-13 20:48:18,552 ERROR gpu :1244775 [interfaces.py:monitor():142] Failed to sample metric: Not Supported | |
| 2024-05-13 20:48:18,572 ERROR gpu :1244775 [interfaces.py:monitor():142] Failed to sample metric: Not Supported | |
| 2024-05-13 20:48:20,626 ERROR gpu :1244775 [interfaces.py:monitor():142] Failed to sample metric: Not Supported | |
| 2024-05-13 20:48:20,644 ERROR gpu :1244775 [interfaces.py:monitor():142] Failed to sample metric: Not Supported | |
| 2024-05-13 20:48:23,050 DEBUG HandlerThread:1244775 [handler.py:handle_request():158] handle_request: status_report | |
| 2024-05-13 20:48:23,336 DEBUG SenderThread:1244775 [sender.py:send():378] send: stats | |
| 2024-05-13 20:48:23,683 ERROR gpu :1244775 [interfaces.py:monitor():142] Failed to sample metric: Not Supported | |
| 2024-05-13 20:48:23,707 ERROR gpu :1244775 [interfaces.py:monitor():142] Failed to sample metric: Not Supported | |
| 2024-05-13 20:48:25,750 ERROR gpu :1244775 [interfaces.py:monitor():142] Failed to sample metric: Not Supported | |
| 2024-05-13 20:48:25,769 ERROR gpu :1244775 [interfaces.py:monitor():142] Failed to sample metric: Not Supported | |
| 2024-05-13 20:48:25,863 DEBUG HandlerThread:1244775 [handler.py:handle_request():158] handle_request: stop_status | |
| 2024-05-13 20:48:25,864 DEBUG SenderThread:1244775 [sender.py:send_request():405] send_request: stop_status | |
| 2024-05-13 20:48:28,681 ERROR gpu :1244775 [interfaces.py:monitor():142] Failed to sample metric: Not Supported | |
| 2024-05-13 20:48:28,701 ERROR gpu :1244775 [interfaces.py:monitor():142] Failed to sample metric: Not Supported | |
| 2024-05-13 20:48:29,005 DEBUG HandlerThread:1244775 [handler.py:handle_request():158] handle_request: status_report | |
| 2024-05-13 20:48:30,725 ERROR gpu :1244775 [interfaces.py:monitor():142] Failed to sample metric: Not Supported | |
| 2024-05-13 20:48:30,747 ERROR gpu :1244775 [interfaces.py:monitor():142] Failed to sample metric: Not Supported | |
| 2024-05-13 20:48:33,782 ERROR gpu :1244775 [interfaces.py:monitor():142] Failed to sample metric: Not Supported | |
| 2024-05-13 20:48:33,801 ERROR gpu :1244775 [interfaces.py:monitor():142] Failed to sample metric: Not Supported | |
| 2024-05-13 20:48:34,006 DEBUG HandlerThread:1244775 [handler.py:handle_request():158] handle_request: status_report | |
| 2024-05-13 20:48:35,835 ERROR gpu :1244775 [interfaces.py:monitor():142] Failed to sample metric: Not Supported | |
| 2024-05-13 20:48:35,858 ERROR gpu :1244775 [interfaces.py:monitor():142] Failed to sample metric: Not Supported | |
| 2024-05-13 20:48:38,877 ERROR gpu :1244775 [interfaces.py:monitor():142] Failed to sample metric: Not Supported | |
| 2024-05-13 20:48:38,904 ERROR gpu :1244775 [interfaces.py:monitor():142] Failed to sample metric: Not Supported | |
| 2024-05-13 20:48:39,007 DEBUG HandlerThread:1244775 [handler.py:handle_request():158] handle_request: status_report | |
| 2024-05-13 20:48:40,863 DEBUG HandlerThread:1244775 [handler.py:handle_request():158] handle_request: stop_status | |
| 2024-05-13 20:48:40,864 DEBUG SenderThread:1244775 [sender.py:send_request():405] send_request: stop_status | |
| 2024-05-13 20:48:40,932 ERROR gpu :1244775 [interfaces.py:monitor():142] Failed to sample metric: Not Supported | |
| 2024-05-13 20:48:40,946 ERROR gpu :1244775 [interfaces.py:monitor():142] Failed to sample metric: Not Supported | |
| 2024-05-13 20:48:42,969 ERROR gpu :1244775 [interfaces.py:monitor():142] Failed to sample metric: Not Supported | |
| 2024-05-13 20:48:42,980 ERROR gpu :1244775 [interfaces.py:monitor():142] Failed to sample metric: Not Supported | |
| 2024-05-13 20:48:44,036 DEBUG HandlerThread:1244775 [handler.py:handle_request():158] handle_request: status_report | |
| 2024-05-13 20:48:45,801 ERROR gpu :1244775 [interfaces.py:monitor():142] Failed to sample metric: Not Supported | |
| 2024-05-13 20:48:45,831 ERROR gpu :1244775 [interfaces.py:monitor():142] Failed to sample metric: Not Supported | |
| 2024-05-13 20:48:49,037 DEBUG HandlerThread:1244775 [handler.py:handle_request():158] handle_request: status_report | |
| 2024-05-13 20:48:49,512 ERROR gpu :1244775 [interfaces.py:aggregate():159] Failed to serialize metric: division by zero | |
| 2024-05-13 20:48:49,550 ERROR gpu :1244775 [interfaces.py:monitor():142] Failed to sample metric: Not Supported | |
| 2024-05-13 20:48:49,567 ERROR gpu :1244775 [interfaces.py:monitor():142] Failed to sample metric: Not Supported | |
| 2024-05-13 20:48:52,479 ERROR gpu :1244775 [interfaces.py:monitor():142] Failed to sample metric: Not Supported | |
| 2024-05-13 20:48:52,494 ERROR gpu :1244775 [interfaces.py:monitor():142] Failed to sample metric: Not Supported | |
| 2024-05-13 20:48:53,338 DEBUG SenderThread:1244775 [sender.py:send():378] send: stats | |
| 2024-05-13 20:48:54,339 DEBUG HandlerThread:1244775 [handler.py:handle_request():158] handle_request: status_report | |
| 2024-05-13 20:48:54,518 ERROR gpu :1244775 [interfaces.py:monitor():142] Failed to sample metric: Not Supported | |
| 2024-05-13 20:48:54,530 ERROR gpu :1244775 [interfaces.py:monitor():142] Failed to sample metric: Not Supported | |
| 2024-05-13 20:48:55,864 DEBUG HandlerThread:1244775 [handler.py:handle_request():158] handle_request: stop_status | |
| 2024-05-13 20:48:55,864 DEBUG SenderThread:1244775 [sender.py:send_request():405] send_request: stop_status | |
| 2024-05-13 20:48:57,576 ERROR gpu :1244775 [interfaces.py:monitor():142] Failed to sample metric: Not Supported | |
| 2024-05-13 20:48:57,589 ERROR gpu :1244775 [interfaces.py:monitor():142] Failed to sample metric: Not Supported | |
| 2024-05-13 20:48:59,615 ERROR gpu :1244775 [interfaces.py:monitor():142] Failed to sample metric: Not Supported | |
| 2024-05-13 20:48:59,628 ERROR gpu :1244775 [interfaces.py:monitor():142] Failed to sample metric: Not Supported | |
| 2024-05-13 20:49:00,007 DEBUG HandlerThread:1244775 [handler.py:handle_request():158] handle_request: status_report | |
| 2024-05-13 20:49:02,716 ERROR gpu :1244775 [interfaces.py:monitor():142] Failed to sample metric: Not Supported | |
| 2024-05-13 20:49:02,730 ERROR gpu :1244775 [interfaces.py:monitor():142] Failed to sample metric: Not Supported | |
| 2024-05-13 20:49:04,748 ERROR gpu :1244775 [interfaces.py:monitor():142] Failed to sample metric: Not Supported | |
| 2024-05-13 20:49:04,763 ERROR gpu :1244775 [interfaces.py:monitor():142] Failed to sample metric: Not Supported | |
| 2024-05-13 20:49:05,008 DEBUG HandlerThread:1244775 [handler.py:handle_request():158] handle_request: status_report | |
| 2024-05-13 20:49:07,719 ERROR gpu :1244775 [interfaces.py:monitor():142] Failed to sample metric: Not Supported | |
| 2024-05-13 20:49:07,741 ERROR gpu :1244775 [interfaces.py:monitor():142] Failed to sample metric: Not Supported | |
| 2024-05-13 20:49:09,766 ERROR gpu :1244775 [interfaces.py:monitor():142] Failed to sample metric: Not Supported | |
| 2024-05-13 20:49:09,776 ERROR gpu :1244775 [interfaces.py:monitor():142] Failed to sample metric: Not Supported | |
| 2024-05-13 20:49:10,009 DEBUG HandlerThread:1244775 [handler.py:handle_request():158] handle_request: status_report | |
| 2024-05-13 20:49:10,864 DEBUG HandlerThread:1244775 [handler.py:handle_request():158] handle_request: stop_status | |
| 2024-05-13 20:49:10,864 DEBUG SenderThread:1244775 [sender.py:send_request():405] send_request: stop_status | |
| 2024-05-13 20:49:12,861 ERROR gpu :1244775 [interfaces.py:monitor():142] Failed to sample metric: Not Supported | |
| 2024-05-13 20:49:12,875 ERROR gpu :1244775 [interfaces.py:monitor():142] Failed to sample metric: Not Supported | |
| 2024-05-13 20:49:14,916 ERROR gpu :1244775 [interfaces.py:monitor():142] Failed to sample metric: Not Supported | |
| 2024-05-13 20:49:14,934 ERROR gpu :1244775 [interfaces.py:monitor():142] Failed to sample metric: Not Supported | |
| 2024-05-13 20:49:15,085 DEBUG HandlerThread:1244775 [handler.py:handle_request():158] handle_request: status_report | |
| 2024-05-13 20:49:17,938 ERROR gpu :1244775 [interfaces.py:monitor():142] Failed to sample metric: Not Supported | |
| 2024-05-13 20:49:17,957 ERROR gpu :1244775 [interfaces.py:monitor():142] Failed to sample metric: Not Supported | |
| 2024-05-13 20:49:20,002 ERROR gpu :1244775 [interfaces.py:monitor():142] Failed to sample metric: Not Supported | |
| 2024-05-13 20:49:20,012 ERROR gpu :1244775 [interfaces.py:monitor():142] Failed to sample metric: Not Supported | |
| 2024-05-13 20:49:20,086 DEBUG HandlerThread:1244775 [handler.py:handle_request():158] handle_request: status_report | |
| 2024-05-13 20:49:22,621 ERROR gpu :1244775 [interfaces.py:monitor():142] Failed to sample metric: Not Supported | |
| 2024-05-13 20:49:22,641 ERROR gpu :1244775 [interfaces.py:monitor():142] Failed to sample metric: Not Supported | |
| 2024-05-13 20:49:23,339 DEBUG SenderThread:1244775 [sender.py:send():378] send: stats | |
| 2024-05-13 20:49:24,675 ERROR gpu :1244775 [interfaces.py:monitor():142] Failed to sample metric: Not Supported | |
| 2024-05-13 20:49:24,686 ERROR gpu :1244775 [interfaces.py:monitor():142] Failed to sample metric: Not Supported | |
| 2024-05-13 20:49:25,341 DEBUG HandlerThread:1244775 [handler.py:handle_request():158] handle_request: status_report | |
| 2024-05-13 20:49:25,864 DEBUG HandlerThread:1244775 [handler.py:handle_request():158] handle_request: stop_status | |
| 2024-05-13 20:49:25,865 DEBUG SenderThread:1244775 [sender.py:send_request():405] send_request: stop_status | |
| 2024-05-13 20:49:28,370 ERROR gpu :1244775 [interfaces.py:aggregate():159] Failed to serialize metric: division by zero | |
| 2024-05-13 20:49:28,612 ERROR gpu :1244775 [interfaces.py:monitor():142] Failed to sample metric: Not Supported | |
| 2024-05-13 20:49:28,633 ERROR gpu :1244775 [interfaces.py:monitor():142] Failed to sample metric: Not Supported | |
| 2024-05-13 20:49:30,656 ERROR gpu :1244775 [interfaces.py:monitor():142] Failed to sample metric: Not Supported | |
| 2024-05-13 20:49:30,678 ERROR gpu :1244775 [interfaces.py:monitor():142] Failed to sample metric: Not Supported | |
| 2024-05-13 20:49:31,038 DEBUG HandlerThread:1244775 [handler.py:handle_request():158] handle_request: status_report | |
| 2024-05-13 20:49:33,121 ERROR gpu :1244775 [interfaces.py:monitor():142] Failed to sample metric: Not Supported | |
| 2024-05-13 20:49:33,150 ERROR gpu :1244775 [interfaces.py:monitor():142] Failed to sample metric: Not Supported | |
| 2024-05-13 20:49:35,177 ERROR gpu :1244775 [interfaces.py:monitor():142] Failed to sample metric: Not Supported | |
| 2024-05-13 20:49:35,217 ERROR gpu :1244775 [interfaces.py:monitor():142] Failed to sample metric: Not Supported | |
| 2024-05-13 20:49:36,039 DEBUG HandlerThread:1244775 [handler.py:handle_request():158] handle_request: status_report | |
| 2024-05-13 20:49:37,932 ERROR gpu :1244775 [interfaces.py:monitor():142] Failed to sample metric: Not Supported | |
| 2024-05-13 20:49:37,967 ERROR gpu :1244775 [interfaces.py:monitor():142] Failed to sample metric: Not Supported | |
| 2024-05-13 20:49:40,002 ERROR gpu :1244775 [interfaces.py:monitor():142] Failed to sample metric: Not Supported | |
| 2024-05-13 20:49:40,032 ERROR gpu :1244775 [interfaces.py:monitor():142] Failed to sample metric: Not Supported | |
| 2024-05-13 20:49:40,864 DEBUG HandlerThread:1244775 [handler.py:handle_request():158] handle_request: stop_status | |
| 2024-05-13 20:49:40,864 DEBUG SenderThread:1244775 [sender.py:send_request():405] send_request: stop_status | |
| 2024-05-13 20:49:42,037 DEBUG HandlerThread:1244775 [handler.py:handle_request():158] handle_request: status_report | |
| 2024-05-13 20:49:42,295 ERROR gpu :1244775 [interfaces.py:monitor():142] Failed to sample metric: Not Supported | |
| 2024-05-13 20:49:42,620 ERROR gpu :1244775 [interfaces.py:monitor():142] Failed to sample metric: Not Supported | |
| 2024-05-13 20:49:44,641 ERROR gpu :1244775 [interfaces.py:monitor():142] Failed to sample metric: Not Supported | |
| 2024-05-13 20:49:44,688 ERROR gpu :1244775 [interfaces.py:monitor():142] Failed to sample metric: Not Supported | |
| 2024-05-13 20:49:47,037 DEBUG HandlerThread:1244775 [handler.py:handle_request():158] handle_request: status_report | |
| 2024-05-13 20:49:47,242 ERROR gpu :1244775 [interfaces.py:monitor():142] Failed to sample metric: Not Supported | |
| 2024-05-13 20:49:47,286 ERROR gpu :1244775 [interfaces.py:monitor():142] Failed to sample metric: Not Supported | |
| 2024-05-13 20:49:49,336 ERROR gpu :1244775 [interfaces.py:monitor():142] Failed to sample metric: Not Supported | |
| 2024-05-13 20:49:49,362 ERROR gpu :1244775 [interfaces.py:monitor():142] Failed to sample metric: Not Supported | |
| 2024-05-13 20:49:51,898 ERROR gpu :1244775 [interfaces.py:monitor():142] Failed to sample metric: Not Supported | |
| 2024-05-13 20:49:51,927 ERROR gpu :1244775 [interfaces.py:monitor():142] Failed to sample metric: Not Supported | |
| 2024-05-13 20:49:52,038 DEBUG HandlerThread:1244775 [handler.py:handle_request():158] handle_request: status_report | |
| 2024-05-13 20:49:53,343 DEBUG SenderThread:1244775 [sender.py:send():378] send: stats | |
| 2024-05-13 20:49:54,378 ERROR gpu :1244775 [interfaces.py:monitor():142] Failed to sample metric: Not Supported | |
| 2024-05-13 20:49:54,396 ERROR gpu :1244775 [interfaces.py:monitor():142] Failed to sample metric: Not Supported | |
| 2024-05-13 20:49:55,864 DEBUG HandlerThread:1244775 [handler.py:handle_request():158] handle_request: stop_status | |
| 2024-05-13 20:49:55,864 DEBUG SenderThread:1244775 [sender.py:send_request():405] send_request: stop_status | |
| 2024-05-13 20:49:56,417 ERROR gpu :1244775 [interfaces.py:monitor():142] Failed to sample metric: Not Supported | |
| 2024-05-13 20:49:56,465 ERROR gpu :1244775 [interfaces.py:monitor():142] Failed to sample metric: Not Supported | |
| 2024-05-13 20:49:58,028 DEBUG HandlerThread:1244775 [handler.py:handle_request():158] handle_request: status_report | |
| 2024-05-13 20:49:58,997 ERROR gpu :1244775 [interfaces.py:monitor():142] Failed to sample metric: Not Supported | |
| 2024-05-13 20:49:59,011 ERROR gpu :1244775 [interfaces.py:monitor():142] Failed to sample metric: Not Supported | |
| 2024-05-13 20:50:01,057 ERROR gpu :1244775 [interfaces.py:monitor():142] Failed to sample metric: Not Supported | |
| 2024-05-13 20:50:01,097 ERROR gpu :1244775 [interfaces.py:monitor():142] Failed to sample metric: Not Supported | |
| 2024-05-13 20:50:03,029 DEBUG HandlerThread:1244775 [handler.py:handle_request():158] handle_request: status_report | |
| 2024-05-13 20:50:05,200 ERROR gpu :1244775 [interfaces.py:aggregate():159] Failed to serialize metric: division by zero | |
| 2024-05-13 20:50:05,253 ERROR gpu :1244775 [interfaces.py:monitor():142] Failed to sample metric: Not Supported | |
| 2024-05-13 20:50:05,272 ERROR gpu :1244775 [interfaces.py:monitor():142] Failed to sample metric: Not Supported | |
| 2024-05-13 20:50:07,311 ERROR gpu :1244775 [interfaces.py:monitor():142] Failed to sample metric: Not Supported | |
| 2024-05-13 20:50:07,330 ERROR gpu :1244775 [interfaces.py:monitor():142] Failed to sample metric: Not Supported | |
| 2024-05-13 20:50:08,029 DEBUG HandlerThread:1244775 [handler.py:handle_request():158] handle_request: status_report | |
| 2024-05-13 20:50:09,887 ERROR gpu :1244775 [interfaces.py:monitor():142] Failed to sample metric: Not Supported | |
| 2024-05-13 20:50:09,918 ERROR gpu :1244775 [interfaces.py:monitor():142] Failed to sample metric: Not Supported | |
| 2024-05-13 20:50:10,864 DEBUG HandlerThread:1244775 [handler.py:handle_request():158] handle_request: stop_status | |
| 2024-05-13 20:50:10,864 DEBUG SenderThread:1244775 [sender.py:send_request():405] send_request: stop_status | |
| 2024-05-13 20:50:11,934 ERROR gpu :1244775 [interfaces.py:monitor():142] Failed to sample metric: Not Supported | |
| 2024-05-13 20:50:11,942 ERROR gpu :1244775 [interfaces.py:monitor():142] Failed to sample metric: Not Supported | |
| 2024-05-13 20:50:13,047 DEBUG HandlerThread:1244775 [handler.py:handle_request():158] handle_request: status_report | |
| 2024-05-13 20:50:14,680 ERROR gpu :1244775 [interfaces.py:monitor():142] Failed to sample metric: Not Supported | |
| 2024-05-13 20:50:14,697 ERROR gpu :1244775 [interfaces.py:monitor():142] Failed to sample metric: Not Supported | |
| 2024-05-13 20:50:16,719 ERROR gpu :1244775 [interfaces.py:monitor():142] Failed to sample metric: Not Supported | |
| 2024-05-13 20:50:16,738 ERROR gpu :1244775 [interfaces.py:monitor():142] Failed to sample metric: Not Supported | |
| 2024-05-13 20:50:18,047 DEBUG HandlerThread:1244775 [handler.py:handle_request():158] handle_request: status_report | |
| 2024-05-13 20:50:19,232 ERROR gpu :1244775 [interfaces.py:monitor():142] Failed to sample metric: Not Supported | |
| 2024-05-13 20:50:19,254 ERROR gpu :1244775 [interfaces.py:monitor():142] Failed to sample metric: Not Supported | |
| 2024-05-13 20:50:21,272 ERROR gpu :1244775 [interfaces.py:monitor():142] Failed to sample metric: Not Supported | |
| 2024-05-13 20:50:21,282 ERROR gpu :1244775 [interfaces.py:monitor():142] Failed to sample metric: Not Supported | |
| 2024-05-13 20:50:23,048 DEBUG HandlerThread:1244775 [handler.py:handle_request():158] handle_request: status_report | |
| 2024-05-13 20:50:23,346 DEBUG SenderThread:1244775 [sender.py:send():378] send: stats | |
| 2024-05-13 20:50:23,802 ERROR gpu :1244775 [interfaces.py:monitor():142] Failed to sample metric: Not Supported | |
| 2024-05-13 20:50:23,817 ERROR gpu :1244775 [interfaces.py:monitor():142] Failed to sample metric: Not Supported | |
| 2024-05-13 20:50:25,839 ERROR gpu :1244775 [interfaces.py:monitor():142] Failed to sample metric: Not Supported | |
| 2024-05-13 20:50:25,850 ERROR gpu :1244775 [interfaces.py:monitor():142] Failed to sample metric: Not Supported | |
| 2024-05-13 20:50:25,864 DEBUG HandlerThread:1244775 [handler.py:handle_request():158] handle_request: stop_status | |
| 2024-05-13 20:50:25,865 DEBUG SenderThread:1244775 [sender.py:send_request():405] send_request: stop_status | |
| 2024-05-13 20:50:28,061 DEBUG HandlerThread:1244775 [handler.py:handle_request():158] handle_request: status_report | |
| 2024-05-13 20:50:28,256 ERROR gpu :1244775 [interfaces.py:monitor():142] Failed to sample metric: Not Supported | |
| 2024-05-13 20:50:28,265 ERROR gpu :1244775 [interfaces.py:monitor():142] Failed to sample metric: Not Supported | |
| 2024-05-13 20:50:30,284 ERROR gpu :1244775 [interfaces.py:monitor():142] Failed to sample metric: Not Supported | |
| 2024-05-13 20:50:30,296 ERROR gpu :1244775 [interfaces.py:monitor():142] Failed to sample metric: Not Supported | |
| 2024-05-13 20:50:32,293 INFO Thread-12 :1244775 [dir_watcher.py:_on_file_modified():288] file/dir modified: /raid/sanchit/parler-tts-mini-v0.1-expresso-concatenated-combined/wandb/run-20240513_204652-m0g0ap7d/files/output.log | |
| 2024-05-13 20:50:32,787 ERROR gpu :1244775 [interfaces.py:monitor():142] Failed to sample metric: Not Supported | |
| 2024-05-13 20:50:32,796 ERROR gpu :1244775 [interfaces.py:monitor():142] Failed to sample metric: Not Supported | |
| 2024-05-13 20:50:33,152 DEBUG HandlerThread:1244775 [handler.py:handle_request():158] handle_request: status_report | |
| 2024-05-13 20:50:33,158 DEBUG SenderThread:1244775 [sender.py:send():378] send: exit | |
| 2024-05-13 20:50:33,158 INFO SenderThread:1244775 [sender.py:send_exit():585] handling exit code: 1 | |
| 2024-05-13 20:50:33,159 INFO SenderThread:1244775 [sender.py:send_exit():587] handling runtime: 219 | |
| 2024-05-13 20:50:33,159 INFO SenderThread:1244775 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end | |
| 2024-05-13 20:50:33,159 INFO SenderThread:1244775 [sender.py:send_exit():593] send defer | |
| 2024-05-13 20:50:33,159 DEBUG HandlerThread:1244775 [handler.py:handle_request():158] handle_request: defer | |
| 2024-05-13 20:50:33,159 INFO HandlerThread:1244775 [handler.py:handle_request_defer():184] handle defer: 0 | |
| 2024-05-13 20:50:33,159 DEBUG SenderThread:1244775 [sender.py:send_request():405] send_request: defer | |
| 2024-05-13 20:50:33,160 INFO SenderThread:1244775 [sender.py:send_request_defer():609] handle sender defer: 0 | |
| 2024-05-13 20:50:33,160 INFO SenderThread:1244775 [sender.py:transition_state():613] send defer: 1 | |
| 2024-05-13 20:50:33,160 DEBUG HandlerThread:1244775 [handler.py:handle_request():158] handle_request: defer | |
| 2024-05-13 20:50:33,160 INFO HandlerThread:1244775 [handler.py:handle_request_defer():184] handle defer: 1 | |
| 2024-05-13 20:50:33,160 DEBUG SenderThread:1244775 [sender.py:send_request():405] send_request: defer | |
| 2024-05-13 20:50:33,160 INFO SenderThread:1244775 [sender.py:send_request_defer():609] handle sender defer: 1 | |
| 2024-05-13 20:50:33,160 INFO SenderThread:1244775 [sender.py:transition_state():613] send defer: 2 | |
| 2024-05-13 20:50:33,160 DEBUG HandlerThread:1244775 [handler.py:handle_request():158] handle_request: defer | |
| 2024-05-13 20:50:33,160 INFO HandlerThread:1244775 [handler.py:handle_request_defer():184] handle defer: 2 | |
| 2024-05-13 20:50:33,160 INFO HandlerThread:1244775 [system_monitor.py:finish():203] Stopping system monitor | |
| 2024-05-13 20:50:33,161 DEBUG SystemMonitor:1244775 [system_monitor.py:_start():179] Finished system metrics aggregation loop | |
| 2024-05-13 20:50:33,161 DEBUG SystemMonitor:1244775 [system_monitor.py:_start():183] Publishing last batch of metrics | |
| 2024-05-13 20:50:33,161 INFO HandlerThread:1244775 [interfaces.py:finish():200] Joined cpu monitor | |
| 2024-05-13 20:50:33,164 INFO HandlerThread:1244775 [interfaces.py:finish():200] Joined disk monitor | |
| 2024-05-13 20:50:33,293 INFO Thread-12 :1244775 [dir_watcher.py:_on_file_created():271] file/dir created: /raid/sanchit/parler-tts-mini-v0.1-expresso-concatenated-combined/wandb/run-20240513_204652-m0g0ap7d/files/wandb-summary.json | |
| 2024-05-13 20:50:34,293 INFO Thread-12 :1244775 [dir_watcher.py:_on_file_modified():288] file/dir modified: /raid/sanchit/parler-tts-mini-v0.1-expresso-concatenated-combined/wandb/run-20240513_204652-m0g0ap7d/files/output.log | |
| 2024-05-13 20:50:34,797 ERROR gpu :1244775 [interfaces.py:aggregate():159] Failed to serialize metric: division by zero | |
| 2024-05-13 20:50:34,797 INFO HandlerThread:1244775 [interfaces.py:finish():200] Joined gpu monitor | |
| 2024-05-13 20:50:34,797 INFO HandlerThread:1244775 [interfaces.py:finish():200] Joined memory monitor | |
| 2024-05-13 20:50:34,797 INFO HandlerThread:1244775 [interfaces.py:finish():200] Joined network monitor | |
| 2024-05-13 20:50:34,798 DEBUG HandlerThread:1244775 [handler.py:handle_request():158] handle_request: poll_exit | |
| 2024-05-13 20:50:34,799 DEBUG SenderThread:1244775 [sender.py:send_request():405] send_request: defer | |
| 2024-05-13 20:50:34,799 INFO SenderThread:1244775 [sender.py:send_request_defer():609] handle sender defer: 2 | |
| 2024-05-13 20:50:34,799 INFO SenderThread:1244775 [sender.py:transition_state():613] send defer: 3 | |
| 2024-05-13 20:50:34,800 DEBUG HandlerThread:1244775 [handler.py:handle_request():158] handle_request: defer | |
| 2024-05-13 20:50:34,800 DEBUG SenderThread:1244775 [sender.py:send():378] send: stats | |
| 2024-05-13 20:50:34,800 INFO HandlerThread:1244775 [handler.py:handle_request_defer():184] handle defer: 3 | |
| 2024-05-13 20:50:34,800 DEBUG SenderThread:1244775 [sender.py:send_request():405] send_request: poll_exit | |
| 2024-05-13 20:50:34,801 DEBUG SenderThread:1244775 [sender.py:send_request():405] send_request: defer | |
| 2024-05-13 20:50:34,801 INFO SenderThread:1244775 [sender.py:send_request_defer():609] handle sender defer: 3 | |
| 2024-05-13 20:50:34,801 INFO SenderThread:1244775 [sender.py:transition_state():613] send defer: 4 | |
| 2024-05-13 20:50:34,801 DEBUG HandlerThread:1244775 [handler.py:handle_request():158] handle_request: defer | |
| 2024-05-13 20:50:34,801 INFO HandlerThread:1244775 [handler.py:handle_request_defer():184] handle defer: 4 | |
| 2024-05-13 20:50:34,801 DEBUG SenderThread:1244775 [sender.py:send_request():405] send_request: defer | |
| 2024-05-13 20:50:34,801 INFO SenderThread:1244775 [sender.py:send_request_defer():609] handle sender defer: 4 | |
| 2024-05-13 20:50:34,801 INFO SenderThread:1244775 [sender.py:transition_state():613] send defer: 5 | |
| 2024-05-13 20:50:34,802 DEBUG HandlerThread:1244775 [handler.py:handle_request():158] handle_request: defer | |
| 2024-05-13 20:50:34,802 INFO HandlerThread:1244775 [handler.py:handle_request_defer():184] handle defer: 5 | |
| 2024-05-13 20:50:34,802 DEBUG SenderThread:1244775 [sender.py:send():378] send: summary | |
| 2024-05-13 20:50:34,802 INFO SenderThread:1244775 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end | |
| 2024-05-13 20:50:34,802 DEBUG SenderThread:1244775 [sender.py:send_request():405] send_request: defer | |
| 2024-05-13 20:50:34,802 INFO SenderThread:1244775 [sender.py:send_request_defer():609] handle sender defer: 5 | |
| 2024-05-13 20:50:34,802 INFO SenderThread:1244775 [sender.py:transition_state():613] send defer: 6 | |
| 2024-05-13 20:50:34,802 DEBUG HandlerThread:1244775 [handler.py:handle_request():158] handle_request: defer | |
| 2024-05-13 20:50:34,802 INFO HandlerThread:1244775 [handler.py:handle_request_defer():184] handle defer: 6 | |
| 2024-05-13 20:50:34,803 DEBUG SenderThread:1244775 [sender.py:send_request():405] send_request: defer | |
| 2024-05-13 20:50:34,803 INFO SenderThread:1244775 [sender.py:send_request_defer():609] handle sender defer: 6 | |
| 2024-05-13 20:50:34,803 INFO SenderThread:1244775 [sender.py:transition_state():613] send defer: 7 | |
| 2024-05-13 20:50:34,803 DEBUG HandlerThread:1244775 [handler.py:handle_request():158] handle_request: status_report | |
| 2024-05-13 20:50:34,803 DEBUG HandlerThread:1244775 [handler.py:handle_request():158] handle_request: defer | |
| 2024-05-13 20:50:34,803 INFO HandlerThread:1244775 [handler.py:handle_request_defer():184] handle defer: 7 | |
| 2024-05-13 20:50:34,803 DEBUG SenderThread:1244775 [sender.py:send_request():405] send_request: defer | |
| 2024-05-13 20:50:34,803 INFO SenderThread:1244775 [sender.py:send_request_defer():609] handle sender defer: 7 | |
| 2024-05-13 20:50:35,159 DEBUG HandlerThread:1244775 [handler.py:handle_request():158] handle_request: poll_exit | |
| 2024-05-13 20:50:35,294 INFO Thread-12 :1244775 [dir_watcher.py:_on_file_modified():288] file/dir modified: /raid/sanchit/parler-tts-mini-v0.1-expresso-concatenated-combined/wandb/run-20240513_204652-m0g0ap7d/files/wandb-summary.json | |
| 2024-05-13 20:50:38,152 INFO SenderThread:1244775 [sender.py:transition_state():613] send defer: 8 | |
| 2024-05-13 20:50:38,152 DEBUG SenderThread:1244775 [sender.py:send_request():405] send_request: poll_exit | |
| 2024-05-13 20:50:38,152 DEBUG HandlerThread:1244775 [handler.py:handle_request():158] handle_request: defer | |
| 2024-05-13 20:50:38,153 INFO HandlerThread:1244775 [handler.py:handle_request_defer():184] handle defer: 8 | |
| 2024-05-13 20:50:38,153 DEBUG SenderThread:1244775 [sender.py:send_request():405] send_request: defer | |
| 2024-05-13 20:50:38,153 INFO SenderThread:1244775 [sender.py:send_request_defer():609] handle sender defer: 8 | |
| 2024-05-13 20:50:38,153 INFO SenderThread:1244775 [job_builder.py:build():432] Attempting to build job artifact | |
| 2024-05-13 20:50:38,153 INFO SenderThread:1244775 [job_builder.py:_get_source_type():565] is repo sourced job | |
| 2024-05-13 20:50:38,160 DEBUG HandlerThread:1244775 [handler.py:handle_request():158] handle_request: poll_exit | |
| 2024-05-13 20:50:38,179 INFO SenderThread:1244775 [job_builder.py:build():541] adding wandb-job metadata file | |
| 2024-05-13 20:50:38,181 INFO SenderThread:1244775 [sender.py:transition_state():613] send defer: 9 | |
| 2024-05-13 20:50:38,181 DEBUG SenderThread:1244775 [sender.py:send_request():405] send_request: poll_exit | |
| 2024-05-13 20:50:38,181 DEBUG HandlerThread:1244775 [handler.py:handle_request():158] handle_request: defer | |
| 2024-05-13 20:50:38,182 INFO HandlerThread:1244775 [handler.py:handle_request_defer():184] handle defer: 9 | |
| 2024-05-13 20:50:38,182 DEBUG SenderThread:1244775 [sender.py:send():378] send: artifact | |
| 2024-05-13 20:50:38,294 INFO Thread-12 :1244775 [dir_watcher.py:_on_file_modified():288] file/dir modified: /raid/sanchit/parler-tts-mini-v0.1-expresso-concatenated-combined/wandb/run-20240513_204652-m0g0ap7d/files/output.log | |
| 2024-05-13 20:50:39,160 DEBUG HandlerThread:1244775 [handler.py:handle_request():158] handle_request: poll_exit | |
| 2024-05-13 20:50:39,209 INFO wandb-upload_0:1244775 [upload_job.py:push():88] Uploaded file /home/sanchit/.local/share/wandb/artifacts/staging/tmp34vs1_ku | |
| 2024-05-13 20:50:39,238 INFO wandb-upload_1:1244775 [upload_job.py:push():88] Uploaded file /tmp/tmp_ne7l6g3/wandb-job.json | |
| 2024-05-13 20:50:40,085 INFO SenderThread:1244775 [sender.py:send_artifact():1467] sent artifact job-https___huggingface.co_sanchit-gandhi_parler-tts-mini-v0.1-expresso-concatenated-combined_run_parler_tts_training.py - {'id': 'QXJ0aWZhY3Q6ODM0NzI5NzMx', 'state': 'PENDING', 'artifactSequence': {'id': 'QXJ0aWZhY3RDb2xsZWN0aW9uOjE3NDIzMTI3NQ==', 'latestArtifact': None}} | |
| 2024-05-13 20:50:40,086 DEBUG SenderThread:1244775 [sender.py:send_request():405] send_request: defer | |
| 2024-05-13 20:50:40,086 INFO SenderThread:1244775 [sender.py:send_request_defer():609] handle sender defer: 9 | |
| 2024-05-13 20:50:40,086 INFO SenderThread:1244775 [dir_watcher.py:finish():358] shutting down directory watcher | |
| 2024-05-13 20:50:40,295 INFO SenderThread:1244775 [dir_watcher.py:finish():388] scan: /raid/sanchit/parler-tts-mini-v0.1-expresso-concatenated-combined/wandb/run-20240513_204652-m0g0ap7d/files | |
| 2024-05-13 20:50:40,295 INFO SenderThread:1244775 [dir_watcher.py:finish():402] scan save: /raid/sanchit/parler-tts-mini-v0.1-expresso-concatenated-combined/wandb/run-20240513_204652-m0g0ap7d/files/conda-environment.yaml conda-environment.yaml | |
| 2024-05-13 20:50:40,295 INFO SenderThread:1244775 [dir_watcher.py:finish():402] scan save: /raid/sanchit/parler-tts-mini-v0.1-expresso-concatenated-combined/wandb/run-20240513_204652-m0g0ap7d/files/wandb-summary.json wandb-summary.json | |
| 2024-05-13 20:50:40,295 INFO SenderThread:1244775 [dir_watcher.py:finish():402] scan save: /raid/sanchit/parler-tts-mini-v0.1-expresso-concatenated-combined/wandb/run-20240513_204652-m0g0ap7d/files/output.log output.log | |
| 2024-05-13 20:50:40,297 INFO SenderThread:1244775 [dir_watcher.py:finish():402] scan save: /raid/sanchit/parler-tts-mini-v0.1-expresso-concatenated-combined/wandb/run-20240513_204652-m0g0ap7d/files/requirements.txt requirements.txt | |
| 2024-05-13 20:50:40,300 INFO SenderThread:1244775 [dir_watcher.py:finish():402] scan save: /raid/sanchit/parler-tts-mini-v0.1-expresso-concatenated-combined/wandb/run-20240513_204652-m0g0ap7d/files/config.yaml config.yaml | |
| 2024-05-13 20:50:40,301 INFO SenderThread:1244775 [dir_watcher.py:finish():402] scan save: /raid/sanchit/parler-tts-mini-v0.1-expresso-concatenated-combined/wandb/run-20240513_204652-m0g0ap7d/files/wandb-metadata.json wandb-metadata.json | |
| 2024-05-13 20:50:40,301 INFO SenderThread:1244775 [sender.py:transition_state():613] send defer: 10 | |
| 2024-05-13 20:50:40,301 DEBUG SenderThread:1244775 [sender.py:send_request():405] send_request: poll_exit | |
| 2024-05-13 20:50:40,301 DEBUG HandlerThread:1244775 [handler.py:handle_request():158] handle_request: defer | |
| 2024-05-13 20:50:40,302 INFO HandlerThread:1244775 [handler.py:handle_request_defer():184] handle defer: 10 | |
| 2024-05-13 20:50:40,306 DEBUG SenderThread:1244775 [sender.py:send_request():405] send_request: defer | |
| 2024-05-13 20:50:40,306 INFO SenderThread:1244775 [sender.py:send_request_defer():609] handle sender defer: 10 | |
| 2024-05-13 20:50:40,306 INFO SenderThread:1244775 [file_pusher.py:finish():169] shutting down file pusher | |
| 2024-05-13 20:50:40,632 INFO wandb-upload_0:1244775 [upload_job.py:push():130] Uploaded file /raid/sanchit/parler-tts-mini-v0.1-expresso-concatenated-combined/wandb/run-20240513_204652-m0g0ap7d/files/conda-environment.yaml | |
| 2024-05-13 20:50:40,639 INFO wandb-upload_1:1244775 [upload_job.py:push():130] Uploaded file /raid/sanchit/parler-tts-mini-v0.1-expresso-concatenated-combined/wandb/run-20240513_204652-m0g0ap7d/files/output.log | |
| 2024-05-13 20:50:40,758 INFO wandb-upload_3:1244775 [upload_job.py:push():130] Uploaded file /raid/sanchit/parler-tts-mini-v0.1-expresso-concatenated-combined/wandb/run-20240513_204652-m0g0ap7d/files/requirements.txt | |
| 2024-05-13 20:50:40,773 INFO wandb-upload_2:1244775 [upload_job.py:push():130] Uploaded file /raid/sanchit/parler-tts-mini-v0.1-expresso-concatenated-combined/wandb/run-20240513_204652-m0g0ap7d/files/wandb-summary.json | |
| 2024-05-13 20:50:40,918 INFO wandb-upload_4:1244775 [upload_job.py:push():130] Uploaded file /raid/sanchit/parler-tts-mini-v0.1-expresso-concatenated-combined/wandb/run-20240513_204652-m0g0ap7d/files/config.yaml | |
| 2024-05-13 20:50:41,118 INFO Thread-11 (_thread_body):1244775 [sender.py:transition_state():613] send defer: 11 | |
| 2024-05-13 20:50:41,119 DEBUG HandlerThread:1244775 [handler.py:handle_request():158] handle_request: defer | |
| 2024-05-13 20:50:41,119 INFO HandlerThread:1244775 [handler.py:handle_request_defer():184] handle defer: 11 | |
| 2024-05-13 20:50:41,119 DEBUG SenderThread:1244775 [sender.py:send_request():405] send_request: defer | |
| 2024-05-13 20:50:41,119 INFO SenderThread:1244775 [sender.py:send_request_defer():609] handle sender defer: 11 | |
| 2024-05-13 20:50:41,120 INFO SenderThread:1244775 [file_pusher.py:join():175] waiting for file pusher | |
| 2024-05-13 20:50:41,120 INFO SenderThread:1244775 [sender.py:transition_state():613] send defer: 12 | |
| 2024-05-13 20:50:41,120 DEBUG HandlerThread:1244775 [handler.py:handle_request():158] handle_request: defer | |
| 2024-05-13 20:50:41,120 INFO HandlerThread:1244775 [handler.py:handle_request_defer():184] handle defer: 12 | |
| 2024-05-13 20:50:41,120 DEBUG SenderThread:1244775 [sender.py:send_request():405] send_request: defer | |
| 2024-05-13 20:50:41,120 INFO SenderThread:1244775 [sender.py:send_request_defer():609] handle sender defer: 12 | |
| 2024-05-13 20:50:41,120 INFO SenderThread:1244775 [file_stream.py:finish():601] file stream finish called | |
| 2024-05-13 20:50:41,161 DEBUG HandlerThread:1244775 [handler.py:handle_request():158] handle_request: poll_exit | |
| 2024-05-13 20:50:41,510 INFO SenderThread:1244775 [file_stream.py:finish():605] file stream finish is done | |
| 2024-05-13 20:50:41,510 INFO SenderThread:1244775 [sender.py:transition_state():613] send defer: 13 | |
| 2024-05-13 20:50:41,510 DEBUG SenderThread:1244775 [sender.py:send_request():405] send_request: poll_exit | |
| 2024-05-13 20:50:41,510 DEBUG HandlerThread:1244775 [handler.py:handle_request():158] handle_request: defer | |
| 2024-05-13 20:50:41,510 INFO HandlerThread:1244775 [handler.py:handle_request_defer():184] handle defer: 13 | |
| 2024-05-13 20:50:41,510 DEBUG SenderThread:1244775 [sender.py:send_request():405] send_request: defer | |
| 2024-05-13 20:50:41,510 INFO SenderThread:1244775 [sender.py:send_request_defer():609] handle sender defer: 13 | |
| 2024-05-13 20:50:41,510 INFO SenderThread:1244775 [sender.py:transition_state():613] send defer: 14 | |
| 2024-05-13 20:50:41,511 DEBUG HandlerThread:1244775 [handler.py:handle_request():158] handle_request: defer | |
| 2024-05-13 20:50:41,511 INFO HandlerThread:1244775 [handler.py:handle_request_defer():184] handle defer: 14 | |
| 2024-05-13 20:50:41,511 DEBUG SenderThread:1244775 [sender.py:send():378] send: final | |
| 2024-05-13 20:50:41,511 DEBUG SenderThread:1244775 [sender.py:send():378] send: footer | |
| 2024-05-13 20:50:41,511 DEBUG SenderThread:1244775 [sender.py:send_request():405] send_request: defer | |
| 2024-05-13 20:50:41,511 INFO SenderThread:1244775 [sender.py:send_request_defer():609] handle sender defer: 14 | |
| 2024-05-13 20:50:41,512 DEBUG HandlerThread:1244775 [handler.py:handle_request():158] handle_request: poll_exit | |
| 2024-05-13 20:50:41,512 DEBUG SenderThread:1244775 [sender.py:send_request():405] send_request: poll_exit | |
| 2024-05-13 20:50:41,512 DEBUG HandlerThread:1244775 [handler.py:handle_request():158] handle_request: poll_exit | |
| 2024-05-13 20:50:41,512 DEBUG SenderThread:1244775 [sender.py:send_request():405] send_request: poll_exit | |
| 2024-05-13 20:50:41,513 DEBUG HandlerThread:1244775 [handler.py:handle_request():158] handle_request: server_info | |
| 2024-05-13 20:50:41,513 DEBUG HandlerThread:1244775 [handler.py:handle_request():158] handle_request: get_summary | |
| 2024-05-13 20:50:41,513 DEBUG SenderThread:1244775 [sender.py:send_request():405] send_request: server_info | |
| 2024-05-13 20:50:41,515 DEBUG HandlerThread:1244775 [handler.py:handle_request():158] handle_request: sampled_history | |
| 2024-05-13 20:50:41,515 DEBUG HandlerThread:1244775 [handler.py:handle_request():158] handle_request: internal_messages | |
| 2024-05-13 20:50:41,651 INFO MainThread:1244775 [wandb_run.py:_footer_history_summary_info():3994] rendering history | |
| 2024-05-13 20:50:41,651 INFO MainThread:1244775 [wandb_run.py:_footer_history_summary_info():4026] rendering summary | |
| 2024-05-13 20:50:41,651 INFO MainThread:1244775 [wandb_run.py:_footer_sync_info():3953] logging synced files | |
| 2024-05-13 20:50:41,651 DEBUG HandlerThread:1244775 [handler.py:handle_request():158] handle_request: shutdown | |
| 2024-05-13 20:50:41,651 INFO HandlerThread:1244775 [handler.py:finish():882] shutting down handler | |
| 2024-05-13 20:50:42,513 INFO WriterThread:1244775 [datastore.py:close():296] close: /raid/sanchit/parler-tts-mini-v0.1-expresso-concatenated-combined/wandb/run-20240513_204652-m0g0ap7d/run-m0g0ap7d.wandb | |
| 2024-05-13 20:50:42,651 INFO SenderThread:1244775 [sender.py:finish():1545] shutting down sender | |
| 2024-05-13 20:50:42,651 INFO SenderThread:1244775 [file_pusher.py:finish():169] shutting down file pusher | |
| 2024-05-13 20:50:42,651 INFO SenderThread:1244775 [file_pusher.py:join():175] waiting for file pusher | |