Spaces:
Runtime error
Runtime error
| 2023-04-21 23:07:06,088 INFO StreamThr :22944 [internal.py:wandb_internal():86] W&B internal server running at pid: 22944, started at: 2023-04-21 23:07:06.088184 | |
| 2023-04-21 23:07:06,090 DEBUG HandlerThread:22944 [handler.py:handle_request():144] handle_request: status | |
| 2023-04-21 23:07:06,090 INFO WriterThread:22944 [datastore.py:open_for_write():85] open: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230421_230706-1i2p1xzz\run-1i2p1xzz.wandb | |
| 2023-04-21 23:07:06,094 DEBUG SenderThread:22944 [sender.py:send():375] send: header | |
| 2023-04-21 23:07:06,157 DEBUG SenderThread:22944 [sender.py:send():375] send: run | |
| 2023-04-21 23:07:06,955 DEBUG HandlerThread:22944 [handler.py:handle_request():144] handle_request: check_version | |
| 2023-04-21 23:07:06,956 INFO SenderThread:22944 [dir_watcher.py:__init__():219] watching files in: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230421_230706-1i2p1xzz\files | |
| 2023-04-21 23:07:06,956 INFO SenderThread:22944 [sender.py:_start_run_threads():1124] run started: 1i2p1xzz with start time 1682089626.089182 | |
| 2023-04-21 23:07:06,956 DEBUG SenderThread:22944 [sender.py:send_request():402] send_request: summary_record | |
| 2023-04-21 23:07:06,957 INFO SenderThread:22944 [sender.py:_save_file():1378] saving file wandb-summary.json with policy end | |
| 2023-04-21 23:07:06,957 DEBUG SenderThread:22944 [sender.py:send_request():402] send_request: check_version | |
| 2023-04-21 23:07:07,646 DEBUG HandlerThread:22944 [handler.py:handle_request():144] handle_request: run_start | |
| 2023-04-21 23:07:07,693 DEBUG HandlerThread:22944 [system_info.py:__init__():31] System info init | |
| 2023-04-21 23:07:07,693 DEBUG HandlerThread:22944 [system_info.py:__init__():46] System info init done | |
| 2023-04-21 23:07:07,693 INFO HandlerThread:22944 [system_monitor.py:start():181] Starting system monitor | |
| 2023-04-21 23:07:07,694 INFO SystemMonitor:22944 [system_monitor.py:_start():145] Starting system asset monitoring threads | |
| 2023-04-21 23:07:07,694 INFO HandlerThread:22944 [system_monitor.py:probe():201] Collecting system info | |
| 2023-04-21 23:07:07,700 INFO SystemMonitor:22944 [interfaces.py:start():190] Started cpu monitoring | |
| 2023-04-21 23:07:07,701 INFO SystemMonitor:22944 [interfaces.py:start():190] Started disk monitoring | |
| 2023-04-21 23:07:07,701 INFO SystemMonitor:22944 [interfaces.py:start():190] Started gpu monitoring | |
| 2023-04-21 23:07:07,715 INFO SystemMonitor:22944 [interfaces.py:start():190] Started memory monitoring | |
| 2023-04-21 23:07:07,748 INFO SystemMonitor:22944 [interfaces.py:start():190] Started network monitoring | |
| 2023-04-21 23:07:07,763 DEBUG HandlerThread:22944 [system_info.py:probe():195] Probing system | |
| 2023-04-21 23:07:07,765 DEBUG HandlerThread:22944 [system_info.py:_probe_git():180] Probing git | |
| 2023-04-21 23:07:07,768 ERROR gpu :22944 [interfaces.py:monitor():144] Failed to sample metric: Not Supported | |
| 2023-04-21 23:07:07,862 DEBUG HandlerThread:22944 [system_info.py:_probe_git():188] Probing git done | |
| 2023-04-21 23:07:07,862 DEBUG HandlerThread:22944 [system_info.py:probe():240] Probing system done | |
| 2023-04-21 23:07:07,862 DEBUG HandlerThread:22944 [system_monitor.py:probe():210] {'os': 'Windows-10-10.0.19041-SP0', 'python': '3.8.10', 'heartbeatAt': '2023-04-21T15:07:07.763382', 'startedAt': '2023-04-21T15:07:06.078182', 'docker': None, 'cuda': None, 'args': ('--do_train', '--train_file', '.\\datasets\\chat\\train.json', '--validation_file', '.\\datasets\\chat\\dev.json', '--prompt_column', 'content', '--response_column', 'summary', '--overwrite_cache', '--model_name_or_path', '..\\models\\chatglm-6b-int4', '--output_dir', 'output\\adgen-chatglm-6b-pt-128-2e-2', '--overwrite_output_dir', '--max_source_length', '64', '--max_target_length', '64', '--per_device_train_batch_size', '1', '--per_device_eval_batch_size', '1', '--gradient_accumulation_steps', '16', '--predict_with_generate', '--max_steps', '1000', '--logging_steps', '10', '--save_steps', '10', '--learning_rate', '2e-2', '--pre_seq_len', '128', '--quantization_bit', '4'), 'state': 'running', 'program': 'main.py', 'codePath': 'ptuning\\main.py', 'git': {'remote': 'https://github.com/THUDM/ChatGLM-6B', 'commit': '01e6313abf4122d789d6e68128856af52847b355'}, 'cpu_count': 6, 'cpu_count_logical': 12, 'cpu_freq': {'current': 2592.0, 'min': 0.0, 'max': 2592.0}, 'cpu_freq_per_core': [{'current': 2592.0, 'min': 0.0, 'max': 2592.0}], 'disk': {'total': 500.32030868530273, 'used': 235.76929092407227}, 'gpu': 'NVIDIA GeForce RTX 2060', 'gpu_count': 1, 'gpu_devices': [{'name': 'NVIDIA GeForce RTX 2060', 'memory_total': 6442450944}], 'memory': {'total': 63.87089538574219}} | |
| 2023-04-21 23:07:07,862 INFO HandlerThread:22944 [system_monitor.py:probe():211] Finished collecting system info | |
| 2023-04-21 23:07:07,862 INFO HandlerThread:22944 [system_monitor.py:probe():214] Publishing system info | |
| 2023-04-21 23:07:07,862 DEBUG HandlerThread:22944 [system_info.py:_save_pip():51] Saving list of pip packages installed into the current environment | |
| 2023-04-21 23:07:07,863 DEBUG HandlerThread:22944 [system_info.py:_save_pip():67] Saving pip packages done | |
| 2023-04-21 23:07:07,864 INFO HandlerThread:22944 [system_monitor.py:probe():216] Finished publishing system info | |
| 2023-04-21 23:07:07,876 DEBUG SenderThread:22944 [sender.py:send():375] send: files | |
| 2023-04-21 23:07:07,876 INFO SenderThread:22944 [sender.py:_save_file():1378] saving file wandb-metadata.json with policy now | |
| 2023-04-21 23:07:07,889 DEBUG HandlerThread:22944 [handler.py:handle_request():144] handle_request: stop_status | |
| 2023-04-21 23:07:07,890 DEBUG SenderThread:22944 [sender.py:send_request():402] send_request: stop_status | |
| 2023-04-21 23:07:07,968 INFO Thread-16 :22944 [dir_watcher.py:_on_file_created():278] file/dir created: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230421_230706-1i2p1xzz\files\wandb-summary.json | |
| 2023-04-21 23:07:07,968 INFO Thread-16 :22944 [dir_watcher.py:_on_file_created():278] file/dir created: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230421_230706-1i2p1xzz\files\wandb-metadata.json | |
| 2023-04-21 23:07:07,969 INFO Thread-16 :22944 [dir_watcher.py:_on_file_created():278] file/dir created: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230421_230706-1i2p1xzz\files\requirements.txt | |
| 2023-04-21 23:07:08,283 DEBUG SenderThread:22944 [sender.py:send():375] send: telemetry | |
| 2023-04-21 23:07:08,283 DEBUG SenderThread:22944 [sender.py:send():375] send: config | |
| 2023-04-21 23:07:08,284 DEBUG SenderThread:22944 [sender.py:send():375] send: metric | |
| 2023-04-21 23:07:08,284 DEBUG SenderThread:22944 [sender.py:send():375] send: telemetry | |
| 2023-04-21 23:07:08,285 DEBUG SenderThread:22944 [sender.py:send():375] send: metric | |
| 2023-04-21 23:07:08,285 WARNING SenderThread:22944 [sender.py:send_metric():1329] Seen metric with glob (shouldn't happen) | |
| 2023-04-21 23:07:08,973 INFO Thread-16 :22944 [dir_watcher.py:_on_file_created():278] file/dir created: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230421_230706-1i2p1xzz\files\output.log | |
| 2023-04-21 23:07:09,801 ERROR gpu :22944 [interfaces.py:monitor():144] Failed to sample metric: Not Supported | |
| 2023-04-21 23:07:10,978 INFO Thread-16 :22944 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230421_230706-1i2p1xzz\files\output.log | |
| 2023-04-21 23:07:11,033 INFO wandb-upload_0:22944 [upload_job.py:push():137] Uploaded file C:\Users\Lenovo\AppData\Local\Temp\tmp75xykfhvwandb\u6pe2ft2-wandb-metadata.json | |
| 2023-04-21 23:07:11,502 DEBUG HandlerThread:22944 [handler.py:handle_request():144] handle_request: status_report | |
| 2023-04-21 23:07:11,845 ERROR gpu :22944 [interfaces.py:monitor():144] Failed to sample metric: Not Supported | |
| 2023-04-21 23:07:11,988 INFO Thread-16 :22944 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230421_230706-1i2p1xzz\files\output.log | |
| 2023-04-21 23:07:13,897 ERROR gpu :22944 [interfaces.py:monitor():144] Failed to sample metric: Not Supported | |
| 2023-04-21 23:07:15,933 ERROR gpu :22944 [interfaces.py:monitor():144] Failed to sample metric: Not Supported | |
| 2023-04-21 23:07:16,543 DEBUG HandlerThread:22944 [handler.py:handle_request():144] handle_request: status_report | |
| 2023-04-21 23:07:17,966 ERROR gpu :22944 [interfaces.py:monitor():144] Failed to sample metric: Not Supported | |
| 2023-04-21 23:07:20,016 ERROR gpu :22944 [interfaces.py:monitor():144] Failed to sample metric: Not Supported | |
| 2023-04-21 23:07:21,595 DEBUG HandlerThread:22944 [handler.py:handle_request():144] handle_request: status_report | |
| 2023-04-21 23:07:22,059 ERROR gpu :22944 [interfaces.py:monitor():144] Failed to sample metric: Not Supported | |
| 2023-04-21 23:07:22,903 DEBUG HandlerThread:22944 [handler.py:handle_request():144] handle_request: stop_status | |
| 2023-04-21 23:07:22,904 DEBUG SenderThread:22944 [sender.py:send_request():402] send_request: stop_status | |
| 2023-04-21 23:07:24,099 ERROR gpu :22944 [interfaces.py:monitor():144] Failed to sample metric: Not Supported | |
| 2023-04-21 23:07:26,146 ERROR gpu :22944 [interfaces.py:monitor():144] Failed to sample metric: Not Supported | |
| 2023-04-21 23:07:27,182 DEBUG HandlerThread:22944 [handler.py:handle_request():144] handle_request: status_report | |
| 2023-04-21 23:07:28,194 ERROR gpu :22944 [interfaces.py:monitor():144] Failed to sample metric: Not Supported | |
| 2023-04-21 23:07:30,238 ERROR gpu :22944 [interfaces.py:monitor():144] Failed to sample metric: Not Supported | |
| 2023-04-21 23:07:32,238 DEBUG HandlerThread:22944 [handler.py:handle_request():144] handle_request: status_report | |
| 2023-04-21 23:07:32,283 ERROR gpu :22944 [interfaces.py:monitor():144] Failed to sample metric: Not Supported | |
| 2023-04-21 23:07:34,336 ERROR gpu :22944 [interfaces.py:monitor():144] Failed to sample metric: Not Supported | |
| 2023-04-21 23:07:35,333 INFO Thread-16 :22944 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230421_230706-1i2p1xzz\files\output.log | |
| 2023-04-21 23:07:36,373 ERROR gpu :22944 [interfaces.py:monitor():144] Failed to sample metric: Not Supported | |
| 2023-04-21 23:07:37,915 DEBUG HandlerThread:22944 [handler.py:handle_request():144] handle_request: stop_status | |
| 2023-04-21 23:07:37,918 DEBUG HandlerThread:22944 [handler.py:handle_request():144] handle_request: status_report | |
| 2023-04-21 23:07:38,466 ERROR gpu :22944 [interfaces.py:monitor():144] Failed to sample metric: Not Supported | |
| 2023-04-21 23:07:38,765 DEBUG SenderThread:22944 [sender.py:send_request():402] send_request: stop_status | |
| 2023-04-21 23:07:39,434 INFO Thread-16 :22944 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230421_230706-1i2p1xzz\files\config.yaml | |
| 2023-04-21 23:07:40,488 ERROR gpu :22944 [interfaces.py:monitor():144] Failed to sample metric: Not Supported | |
| 2023-04-21 23:07:42,518 ERROR gpu :22944 [interfaces.py:monitor():144] Failed to sample metric: Not Supported | |
| 2023-04-21 23:07:44,055 DEBUG HandlerThread:22944 [handler.py:handle_request():144] handle_request: status_report | |
| 2023-04-21 23:07:44,543 ERROR gpu :22944 [interfaces.py:monitor():144] Failed to sample metric: Not Supported | |
| 2023-04-21 23:07:46,579 ERROR gpu :22944 [interfaces.py:monitor():144] Failed to sample metric: Not Supported | |
| 2023-04-21 23:07:48,620 ERROR gpu :22944 [interfaces.py:monitor():144] Failed to sample metric: Not Supported | |
| 2023-04-21 23:07:49,108 DEBUG HandlerThread:22944 [handler.py:handle_request():144] handle_request: status_report | |
| 2023-04-21 23:07:50,680 ERROR gpu :22944 [interfaces.py:monitor():144] Failed to sample metric: Not Supported | |
| 2023-04-21 23:07:52,722 ERROR gpu :22944 [interfaces.py:monitor():144] Failed to sample metric: Not Supported | |
| 2023-04-21 23:07:52,926 DEBUG HandlerThread:22944 [handler.py:handle_request():144] handle_request: stop_status | |
| 2023-04-21 23:07:52,926 DEBUG SenderThread:22944 [sender.py:send_request():402] send_request: stop_status | |
| 2023-04-21 23:07:54,416 DEBUG HandlerThread:22944 [handler.py:handle_request():144] handle_request: status_report | |
| 2023-04-21 23:07:54,759 INFO Thread-16 :22944 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230421_230706-1i2p1xzz\files\output.log | |
| 2023-04-21 23:07:54,770 ERROR gpu :22944 [interfaces.py:monitor():144] Failed to sample metric: Not Supported | |
| 2023-04-21 23:07:56,808 ERROR gpu :22944 [interfaces.py:monitor():144] Failed to sample metric: Not Supported | |
| 2023-04-21 23:07:58,842 ERROR gpu :22944 [interfaces.py:monitor():144] Failed to sample metric: Not Supported | |
| 2023-04-21 23:07:59,455 DEBUG HandlerThread:22944 [handler.py:handle_request():144] handle_request: status_report | |
| 2023-04-21 23:08:00,896 ERROR gpu :22944 [interfaces.py:monitor():144] Failed to sample metric: Not Supported | |
| 2023-04-21 23:08:02,931 ERROR gpu :22944 [interfaces.py:monitor():144] Failed to sample metric: Not Supported | |
| 2023-04-21 23:08:04,505 DEBUG HandlerThread:22944 [handler.py:handle_request():144] handle_request: status_report | |
| 2023-04-21 23:08:04,980 ERROR gpu :22944 [interfaces.py:monitor():144] Failed to sample metric: Not Supported | |
| 2023-04-21 23:08:07,008 ERROR gpu :22944 [interfaces.py:monitor():144] Failed to sample metric: Not Supported | |
| 2023-04-21 23:08:07,761 DEBUG SystemMonitor:22944 [system_monitor.py:_start():159] Starting system metrics aggregation loop | |
| 2023-04-21 23:08:07,762 DEBUG SenderThread:22944 [sender.py:send():375] send: stats | |
| 2023-04-21 23:08:07,935 DEBUG HandlerThread:22944 [handler.py:handle_request():144] handle_request: stop_status | |
| 2023-04-21 23:08:07,935 DEBUG SenderThread:22944 [sender.py:send_request():402] send_request: stop_status | |
| 2023-04-21 23:08:09,129 ERROR gpu :22944 [interfaces.py:monitor():144] Failed to sample metric: Not Supported | |
| 2023-04-21 23:08:10,193 DEBUG HandlerThread:22944 [handler.py:handle_request():144] handle_request: status_report | |
| 2023-04-21 23:08:11,145 ERROR gpu :22944 [interfaces.py:monitor():144] Failed to sample metric: Not Supported | |
| 2023-04-21 23:08:13,163 ERROR gpu :22944 [interfaces.py:monitor():144] Failed to sample metric: Not Supported | |
| 2023-04-21 23:08:15,189 INFO Thread-16 :22944 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230421_230706-1i2p1xzz\files\output.log | |
| 2023-04-21 23:08:15,201 ERROR gpu :22944 [interfaces.py:monitor():144] Failed to sample metric: Not Supported | |
| 2023-04-21 23:08:15,926 DEBUG HandlerThread:22944 [handler.py:handle_request():144] handle_request: status_report | |
| 2023-04-21 23:08:17,242 ERROR gpu :22944 [interfaces.py:monitor():144] Failed to sample metric: Not Supported | |
| 2023-04-21 23:08:19,283 ERROR gpu :22944 [interfaces.py:monitor():144] Failed to sample metric: Not Supported | |
| 2023-04-21 23:08:20,957 DEBUG HandlerThread:22944 [handler.py:handle_request():144] handle_request: status_report | |
| 2023-04-21 23:08:21,338 ERROR gpu :22944 [interfaces.py:monitor():144] Failed to sample metric: Not Supported | |
| 2023-04-21 23:08:22,941 DEBUG HandlerThread:22944 [handler.py:handle_request():144] handle_request: stop_status | |
| 2023-04-21 23:08:22,941 DEBUG SenderThread:22944 [sender.py:send_request():402] send_request: stop_status | |
| 2023-04-21 23:08:23,382 ERROR gpu :22944 [interfaces.py:monitor():144] Failed to sample metric: Not Supported | |
| 2023-04-21 23:08:25,430 ERROR gpu :22944 [interfaces.py:monitor():144] Failed to sample metric: Not Supported | |
| 2023-04-21 23:08:26,226 DEBUG HandlerThread:22944 [handler.py:handle_request():144] handle_request: status_report | |
| 2023-04-21 23:08:27,477 ERROR gpu :22944 [interfaces.py:monitor():144] Failed to sample metric: Not Supported | |
| 2023-04-21 23:08:29,527 ERROR gpu :22944 [interfaces.py:monitor():144] Failed to sample metric: Not Supported | |
| 2023-04-21 23:08:31,272 DEBUG HandlerThread:22944 [handler.py:handle_request():144] handle_request: status_report | |
| 2023-04-21 23:08:31,566 ERROR gpu :22944 [interfaces.py:monitor():144] Failed to sample metric: Not Supported | |
| 2023-04-21 23:08:33,615 ERROR gpu :22944 [interfaces.py:monitor():144] Failed to sample metric: Not Supported | |
| 2023-04-21 23:08:35,654 ERROR gpu :22944 [interfaces.py:monitor():144] Failed to sample metric: Not Supported | |
| 2023-04-21 23:08:36,324 DEBUG HandlerThread:22944 [handler.py:handle_request():144] handle_request: status_report | |
| 2023-04-21 23:08:37,713 ERROR gpu :22944 [interfaces.py:monitor():144] Failed to sample metric: Not Supported | |
| 2023-04-21 23:08:37,764 DEBUG SenderThread:22944 [sender.py:send():375] send: stats | |
| 2023-04-21 23:08:37,952 DEBUG HandlerThread:22944 [handler.py:handle_request():144] handle_request: stop_status | |
| 2023-04-21 23:08:37,953 DEBUG SenderThread:22944 [sender.py:send_request():402] send_request: stop_status | |
| 2023-04-21 23:08:39,840 ERROR gpu :22944 [interfaces.py:monitor():144] Failed to sample metric: Not Supported | |
| 2023-04-21 23:08:41,854 ERROR gpu :22944 [interfaces.py:monitor():144] Failed to sample metric: Not Supported | |
| 2023-04-21 23:08:42,244 DEBUG HandlerThread:22944 [handler.py:handle_request():144] handle_request: status_report | |
| 2023-04-21 23:08:43,869 ERROR gpu :22944 [interfaces.py:monitor():144] Failed to sample metric: Not Supported | |
| 2023-04-21 23:08:45,894 ERROR gpu :22944 [interfaces.py:monitor():144] Failed to sample metric: Not Supported | |
| 2023-04-21 23:08:47,301 DEBUG HandlerThread:22944 [handler.py:handle_request():144] handle_request: status_report | |
| 2023-04-21 23:08:47,938 ERROR gpu :22944 [interfaces.py:monitor():144] Failed to sample metric: Not Supported | |
| 2023-04-21 23:08:48,941 INFO Thread-16 :22944 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230421_230706-1i2p1xzz\files\output.log | |
| 2023-04-21 23:08:49,988 ERROR gpu :22944 [interfaces.py:monitor():144] Failed to sample metric: Not Supported | |
| 2023-04-21 23:08:52,025 ERROR gpu :22944 [interfaces.py:monitor():144] Failed to sample metric: Not Supported | |
| 2023-04-21 23:08:52,958 DEBUG HandlerThread:22944 [handler.py:handle_request():144] handle_request: stop_status | |
| 2023-04-21 23:08:52,959 DEBUG SenderThread:22944 [sender.py:send_request():402] send_request: stop_status | |
| 2023-04-21 23:08:53,218 DEBUG HandlerThread:22944 [handler.py:handle_request():144] handle_request: status_report | |
| 2023-04-21 23:08:54,058 ERROR gpu :22944 [interfaces.py:monitor():144] Failed to sample metric: Not Supported | |
| 2023-04-21 23:08:56,101 ERROR gpu :22944 [interfaces.py:monitor():144] Failed to sample metric: Not Supported | |
| 2023-04-21 23:08:58,148 ERROR gpu :22944 [interfaces.py:monitor():144] Failed to sample metric: Not Supported | |
| 2023-04-21 23:08:58,254 DEBUG HandlerThread:22944 [handler.py:handle_request():144] handle_request: status_report | |
| 2023-04-21 23:09:00,185 ERROR gpu :22944 [interfaces.py:monitor():144] Failed to sample metric: Not Supported | |
| 2023-04-21 23:09:02,240 ERROR gpu :22944 [interfaces.py:monitor():144] Failed to sample metric: Not Supported | |
| 2023-04-21 23:09:03,300 DEBUG HandlerThread:22944 [handler.py:handle_request():144] handle_request: status_report | |
| 2023-04-21 23:09:04,279 ERROR gpu :22944 [interfaces.py:monitor():144] Failed to sample metric: Not Supported | |
| 2023-04-21 23:09:06,339 ERROR gpu :22944 [interfaces.py:monitor():144] Failed to sample metric: Not Supported | |
| 2023-04-21 23:09:07,780 DEBUG SenderThread:22944 [sender.py:send():375] send: stats | |
| 2023-04-21 23:09:07,966 DEBUG HandlerThread:22944 [handler.py:handle_request():144] handle_request: stop_status | |
| 2023-04-21 23:09:07,967 DEBUG SenderThread:22944 [sender.py:send_request():402] send_request: stop_status | |
| 2023-04-21 23:09:08,380 ERROR gpu :22944 [interfaces.py:monitor():144] Failed to sample metric: Not Supported | |
| 2023-04-21 23:09:09,236 DEBUG HandlerThread:22944 [handler.py:handle_request():144] handle_request: status_report | |
| 2023-04-21 23:09:10,514 ERROR gpu :22944 [interfaces.py:monitor():144] Failed to sample metric: Not Supported | |
| 2023-04-21 23:09:12,532 ERROR gpu :22944 [interfaces.py:monitor():144] Failed to sample metric: Not Supported | |
| 2023-04-21 23:09:14,274 DEBUG HandlerThread:22944 [handler.py:handle_request():144] handle_request: status_report | |
| 2023-04-21 23:09:14,548 ERROR gpu :22944 [interfaces.py:monitor():144] Failed to sample metric: Not Supported | |
| 2023-04-21 23:09:16,577 ERROR gpu :22944 [interfaces.py:monitor():144] Failed to sample metric: Not Supported | |
| 2023-04-21 23:09:18,607 ERROR gpu :22944 [interfaces.py:monitor():144] Failed to sample metric: Not Supported | |
| 2023-04-21 23:09:19,326 DEBUG HandlerThread:22944 [handler.py:handle_request():144] handle_request: status_report | |
| 2023-04-21 23:09:20,671 ERROR gpu :22944 [interfaces.py:monitor():144] Failed to sample metric: Not Supported | |
| 2023-04-21 23:09:22,712 ERROR gpu :22944 [interfaces.py:monitor():144] Failed to sample metric: Not Supported | |
| 2023-04-21 23:09:22,971 DEBUG HandlerThread:22944 [handler.py:handle_request():144] handle_request: stop_status | |
| 2023-04-21 23:09:22,972 DEBUG SenderThread:22944 [sender.py:send_request():402] send_request: stop_status | |
| 2023-04-21 23:09:23,703 INFO Thread-16 :22944 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230421_230706-1i2p1xzz\files\output.log | |
| 2023-04-21 23:09:24,769 ERROR gpu :22944 [interfaces.py:monitor():144] Failed to sample metric: Not Supported | |
| 2023-04-21 23:09:25,229 DEBUG HandlerThread:22944 [handler.py:handle_request():144] handle_request: status_report | |
| 2023-04-21 23:09:26,838 ERROR gpu :22944 [interfaces.py:monitor():144] Failed to sample metric: Not Supported | |
| 2023-04-21 23:09:28,863 ERROR gpu :22944 [interfaces.py:monitor():144] Failed to sample metric: Not Supported | |
| 2023-04-21 23:09:30,281 DEBUG HandlerThread:22944 [handler.py:handle_request():144] handle_request: status_report | |
| 2023-04-21 23:09:30,912 ERROR gpu :22944 [interfaces.py:monitor():144] Failed to sample metric: Not Supported | |
| 2023-04-21 23:09:32,958 ERROR gpu :22944 [interfaces.py:monitor():144] Failed to sample metric: Not Supported | |
| 2023-04-21 23:09:35,001 ERROR gpu :22944 [interfaces.py:monitor():144] Failed to sample metric: Not Supported | |
| 2023-04-21 23:09:35,334 DEBUG HandlerThread:22944 [handler.py:handle_request():144] handle_request: status_report | |
| 2023-04-21 23:09:37,035 ERROR gpu :22944 [interfaces.py:monitor():144] Failed to sample metric: Not Supported | |
| 2023-04-21 23:09:37,784 DEBUG SenderThread:22944 [sender.py:send():375] send: stats | |
| 2023-04-21 23:09:37,971 DEBUG HandlerThread:22944 [handler.py:handle_request():144] handle_request: stop_status | |
| 2023-04-21 23:09:37,972 DEBUG SenderThread:22944 [sender.py:send_request():402] send_request: stop_status | |
| 2023-04-21 23:09:39,073 ERROR gpu :22944 [interfaces.py:monitor():144] Failed to sample metric: Not Supported | |
| 2023-04-21 23:09:41,187 ERROR gpu :22944 [interfaces.py:monitor():144] Failed to sample metric: Not Supported | |
| 2023-04-21 23:09:41,263 DEBUG HandlerThread:22944 [handler.py:handle_request():144] handle_request: status_report | |
| 2023-04-21 23:09:43,213 ERROR gpu :22944 [interfaces.py:monitor():144] Failed to sample metric: Not Supported | |
| 2023-04-21 23:09:45,208 INFO Thread-16 :22944 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230421_230706-1i2p1xzz\files\output.log | |
| 2023-04-21 23:09:45,231 ERROR gpu :22944 [interfaces.py:monitor():144] Failed to sample metric: Not Supported | |
| 2023-04-21 23:09:46,350 DEBUG HandlerThread:22944 [handler.py:handle_request():144] handle_request: status_report | |
| 2023-04-21 23:09:47,261 ERROR gpu :22944 [interfaces.py:monitor():144] Failed to sample metric: Not Supported | |
| 2023-04-21 23:09:49,301 ERROR gpu :22944 [interfaces.py:monitor():144] Failed to sample metric: Not Supported | |
| 2023-04-21 23:09:51,356 ERROR gpu :22944 [interfaces.py:monitor():144] Failed to sample metric: Not Supported | |
| 2023-04-21 23:09:51,405 DEBUG HandlerThread:22944 [handler.py:handle_request():144] handle_request: status_report | |
| 2023-04-21 23:09:52,976 DEBUG HandlerThread:22944 [handler.py:handle_request():144] handle_request: stop_status | |
| 2023-04-21 23:09:52,977 DEBUG SenderThread:22944 [sender.py:send_request():402] send_request: stop_status | |
| 2023-04-21 23:09:53,419 ERROR gpu :22944 [interfaces.py:monitor():144] Failed to sample metric: Not Supported | |
| 2023-04-21 23:09:55,457 ERROR gpu :22944 [interfaces.py:monitor():144] Failed to sample metric: Not Supported | |
| 2023-04-21 23:09:57,263 DEBUG HandlerThread:22944 [handler.py:handle_request():144] handle_request: status_report | |
| 2023-04-21 23:09:57,507 ERROR gpu :22944 [interfaces.py:monitor():144] Failed to sample metric: Not Supported | |
| 2023-04-21 23:09:59,551 ERROR gpu :22944 [interfaces.py:monitor():144] Failed to sample metric: Not Supported | |
| 2023-04-21 23:10:01,595 ERROR gpu :22944 [interfaces.py:monitor():144] Failed to sample metric: Not Supported | |
| 2023-04-21 23:10:02,303 DEBUG HandlerThread:22944 [handler.py:handle_request():144] handle_request: status_report | |
| 2023-04-21 23:10:03,642 ERROR gpu :22944 [interfaces.py:monitor():144] Failed to sample metric: Not Supported | |
| 2023-04-21 23:10:05,687 ERROR gpu :22944 [interfaces.py:monitor():144] Failed to sample metric: Not Supported | |
| 2023-04-21 23:10:07,336 DEBUG HandlerThread:22944 [handler.py:handle_request():144] handle_request: status_report | |
| 2023-04-21 23:10:07,723 ERROR gpu :22944 [interfaces.py:monitor():144] Failed to sample metric: Not Supported | |
| 2023-04-21 23:10:07,788 DEBUG SenderThread:22944 [sender.py:send():375] send: stats | |
| 2023-04-21 23:10:07,988 DEBUG HandlerThread:22944 [handler.py:handle_request():144] handle_request: stop_status | |
| 2023-04-21 23:10:07,988 DEBUG SenderThread:22944 [sender.py:send_request():402] send_request: stop_status | |
| 2023-04-21 23:10:09,761 INFO Thread-16 :22944 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230421_230706-1i2p1xzz\files\output.log | |
| 2023-04-21 23:10:09,787 ERROR gpu :22944 [interfaces.py:monitor():144] Failed to sample metric: Not Supported | |
| 2023-04-21 23:10:11,878 ERROR gpu :22944 [interfaces.py:monitor():144] Failed to sample metric: Not Supported | |
| 2023-04-21 23:10:13,283 DEBUG HandlerThread:22944 [handler.py:handle_request():144] handle_request: status_report | |
| 2023-04-21 23:10:13,897 ERROR gpu :22944 [interfaces.py:monitor():144] Failed to sample metric: Not Supported | |
| 2023-04-21 23:10:15,922 ERROR gpu :22944 [interfaces.py:monitor():144] Failed to sample metric: Not Supported | |
| 2023-04-21 23:10:17,939 ERROR gpu :22944 [interfaces.py:monitor():144] Failed to sample metric: Not Supported | |
| 2023-04-21 23:10:18,316 DEBUG HandlerThread:22944 [handler.py:handle_request():144] handle_request: status_report | |
| 2023-04-21 23:10:19,986 ERROR gpu :22944 [interfaces.py:monitor():144] Failed to sample metric: Not Supported | |
| 2023-04-21 23:10:22,039 ERROR gpu :22944 [interfaces.py:monitor():144] Failed to sample metric: Not Supported | |
| 2023-04-21 23:10:22,997 DEBUG HandlerThread:22944 [handler.py:handle_request():144] handle_request: stop_status | |
| 2023-04-21 23:10:22,998 DEBUG SenderThread:22944 [sender.py:send_request():402] send_request: stop_status | |
| 2023-04-21 23:10:24,086 ERROR gpu :22944 [interfaces.py:monitor():144] Failed to sample metric: Not Supported | |
| 2023-04-21 23:10:24,270 DEBUG HandlerThread:22944 [handler.py:handle_request():144] handle_request: status_report | |
| 2023-04-21 23:10:26,123 ERROR gpu :22944 [interfaces.py:monitor():144] Failed to sample metric: Not Supported | |
| 2023-04-21 23:10:28,164 ERROR gpu :22944 [interfaces.py:monitor():144] Failed to sample metric: Not Supported | |
| 2023-04-21 23:10:29,322 DEBUG HandlerThread:22944 [handler.py:handle_request():144] handle_request: status_report | |
| 2023-04-21 23:10:30,225 ERROR gpu :22944 [interfaces.py:monitor():144] Failed to sample metric: Not Supported | |
| 2023-04-21 23:10:31,213 INFO Thread-16 :22944 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230421_230706-1i2p1xzz\files\output.log | |
| 2023-04-21 23:10:32,253 ERROR gpu :22944 [interfaces.py:monitor():144] Failed to sample metric: Not Supported | |
| 2023-04-21 23:10:34,287 ERROR gpu :22944 [interfaces.py:monitor():144] Failed to sample metric: Not Supported | |
| 2023-04-21 23:10:34,840 DEBUG HandlerThread:22944 [handler.py:handle_request():144] handle_request: status_report | |
| 2023-04-21 23:10:36,334 ERROR gpu :22944 [interfaces.py:monitor():144] Failed to sample metric: Not Supported | |
| 2023-04-21 23:10:37,793 DEBUG SenderThread:22944 [sender.py:send():375] send: stats | |
| 2023-04-21 23:10:38,012 DEBUG HandlerThread:22944 [handler.py:handle_request():144] handle_request: stop_status | |
| 2023-04-21 23:10:38,012 DEBUG SenderThread:22944 [sender.py:send_request():402] send_request: stop_status | |
| 2023-04-21 23:10:38,372 ERROR gpu :22944 [interfaces.py:monitor():144] Failed to sample metric: Not Supported | |
| 2023-04-21 23:10:40,304 DEBUG HandlerThread:22944 [handler.py:handle_request():144] handle_request: status_report | |
| 2023-04-21 23:10:40,418 ERROR gpu :22944 [interfaces.py:monitor():144] Failed to sample metric: Not Supported | |
| 2023-04-21 23:10:42,525 ERROR gpu :22944 [interfaces.py:monitor():144] Failed to sample metric: Not Supported | |
| 2023-04-21 23:10:44,542 ERROR gpu :22944 [interfaces.py:monitor():144] Failed to sample metric: Not Supported | |
| 2023-04-21 23:10:45,338 DEBUG HandlerThread:22944 [handler.py:handle_request():144] handle_request: status_report | |
| 2023-04-21 23:10:46,566 ERROR gpu :22944 [interfaces.py:monitor():144] Failed to sample metric: Not Supported | |
| 2023-04-21 23:10:48,605 ERROR gpu :22944 [interfaces.py:monitor():144] Failed to sample metric: Not Supported | |
| 2023-04-21 23:10:50,397 DEBUG HandlerThread:22944 [handler.py:handle_request():144] handle_request: status_report | |
| 2023-04-21 23:10:50,642 ERROR gpu :22944 [interfaces.py:monitor():144] Failed to sample metric: Not Supported | |
| 2023-04-21 23:10:52,684 ERROR gpu :22944 [interfaces.py:monitor():144] Failed to sample metric: Not Supported | |
| 2023-04-21 23:10:53,024 DEBUG HandlerThread:22944 [handler.py:handle_request():144] handle_request: stop_status | |
| 2023-04-21 23:10:53,024 DEBUG SenderThread:22944 [sender.py:send_request():402] send_request: stop_status | |
| 2023-04-21 23:10:53,688 INFO Thread-16 :22944 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230421_230706-1i2p1xzz\files\output.log | |
| 2023-04-21 23:10:54,725 ERROR gpu :22944 [interfaces.py:monitor():144] Failed to sample metric: Not Supported | |
| 2023-04-21 23:10:56,303 DEBUG HandlerThread:22944 [handler.py:handle_request():144] handle_request: status_report | |
| 2023-04-21 23:10:56,767 ERROR gpu :22944 [interfaces.py:monitor():144] Failed to sample metric: Not Supported | |
| 2023-04-21 23:10:58,806 ERROR gpu :22944 [interfaces.py:monitor():144] Failed to sample metric: Not Supported | |
| 2023-04-21 23:11:00,868 ERROR gpu :22944 [interfaces.py:monitor():144] Failed to sample metric: Not Supported | |
| 2023-04-21 23:11:01,347 DEBUG HandlerThread:22944 [handler.py:handle_request():144] handle_request: status_report | |
| 2023-04-21 23:11:02,904 ERROR gpu :22944 [interfaces.py:monitor():144] Failed to sample metric: Not Supported | |
| 2023-04-21 23:11:04,940 ERROR gpu :22944 [interfaces.py:monitor():144] Failed to sample metric: Not Supported | |
| 2023-04-21 23:11:06,405 DEBUG HandlerThread:22944 [handler.py:handle_request():144] handle_request: status_report | |
| 2023-04-21 23:11:06,978 ERROR gpu :22944 [interfaces.py:monitor():144] Failed to sample metric: Not Supported | |
| 2023-04-21 23:11:07,805 DEBUG SenderThread:22944 [sender.py:send():375] send: stats | |
| 2023-04-21 23:11:08,034 DEBUG HandlerThread:22944 [handler.py:handle_request():144] handle_request: stop_status | |
| 2023-04-21 23:11:08,035 DEBUG SenderThread:22944 [sender.py:send_request():402] send_request: stop_status | |
| 2023-04-21 23:11:09,030 ERROR gpu :22944 [interfaces.py:monitor():144] Failed to sample metric: Not Supported | |
| 2023-04-21 23:11:11,071 ERROR gpu :22944 [interfaces.py:monitor():144] Failed to sample metric: Not Supported | |
| 2023-04-21 23:11:12,314 DEBUG HandlerThread:22944 [handler.py:handle_request():144] handle_request: status_report | |
| 2023-04-21 23:11:13,165 ERROR gpu :22944 [interfaces.py:monitor():144] Failed to sample metric: Not Supported | |
| 2023-04-21 23:11:14,341 DEBUG HandlerThread:22944 [handler.py:handle_request():144] handle_request: partial_history | |
| 2023-04-21 23:11:14,344 DEBUG SenderThread:22944 [sender.py:send():375] send: metric | |
| 2023-04-21 23:11:14,344 DEBUG SenderThread:22944 [sender.py:send():375] send: metric | |
| 2023-04-21 23:11:14,345 DEBUG SenderThread:22944 [sender.py:send():375] send: metric | |
| 2023-04-21 23:11:14,345 DEBUG SenderThread:22944 [sender.py:send():375] send: history | |
| 2023-04-21 23:11:14,345 DEBUG SenderThread:22944 [sender.py:send_request():402] send_request: summary_record | |
| 2023-04-21 23:11:14,347 INFO SenderThread:22944 [sender.py:_save_file():1378] saving file wandb-summary.json with policy end | |
| 2023-04-21 23:11:15,144 INFO Thread-16 :22944 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230421_230706-1i2p1xzz\files\wandb-summary.json | |
| 2023-04-21 23:11:15,184 ERROR gpu :22944 [interfaces.py:monitor():144] Failed to sample metric: Not Supported | |
| 2023-04-21 23:11:16,150 INFO Thread-16 :22944 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230421_230706-1i2p1xzz\files\output.log | |
| 2023-04-21 23:11:17,196 INFO Thread-16 :22944 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230421_230706-1i2p1xzz\files\output.log | |
| 2023-04-21 23:11:17,204 ERROR gpu :22944 [interfaces.py:monitor():144] Failed to sample metric: Not Supported | |
| 2023-04-21 23:11:17,641 DEBUG HandlerThread:22944 [handler.py:handle_request():144] handle_request: status_report | |
| 2023-04-21 23:11:19,249 ERROR gpu :22944 [interfaces.py:monitor():144] Failed to sample metric: Not Supported | |
| 2023-04-21 23:11:21,305 ERROR gpu :22944 [interfaces.py:monitor():144] Failed to sample metric: Not Supported | |
| 2023-04-21 23:11:22,682 DEBUG HandlerThread:22944 [handler.py:handle_request():144] handle_request: status_report | |
| 2023-04-21 23:11:23,036 DEBUG HandlerThread:22944 [handler.py:handle_request():144] handle_request: stop_status | |
| 2023-04-21 23:11:23,374 DEBUG SenderThread:22944 [sender.py:send_request():402] send_request: stop_status | |
| 2023-04-21 23:11:23,376 ERROR gpu :22944 [interfaces.py:monitor():144] Failed to sample metric: Not Supported | |
| 2023-04-21 23:11:24,342 INFO Thread-16 :22944 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230421_230706-1i2p1xzz\files\config.yaml | |
| 2023-04-21 23:11:25,395 ERROR gpu :22944 [interfaces.py:monitor():144] Failed to sample metric: Not Supported | |
| 2023-04-21 23:11:27,423 ERROR gpu :22944 [interfaces.py:monitor():144] Failed to sample metric: Not Supported | |
| 2023-04-21 23:11:28,664 DEBUG HandlerThread:22944 [handler.py:handle_request():144] handle_request: status_report | |
| 2023-04-21 23:11:29,479 ERROR gpu :22944 [interfaces.py:monitor():144] Failed to sample metric: Not Supported | |
| 2023-04-21 23:11:31,524 ERROR gpu :22944 [interfaces.py:monitor():144] Failed to sample metric: Not Supported | |
| 2023-04-21 23:11:33,567 ERROR gpu :22944 [interfaces.py:monitor():144] Failed to sample metric: Not Supported | |
| 2023-04-21 23:11:33,703 DEBUG HandlerThread:22944 [handler.py:handle_request():144] handle_request: status_report | |
| 2023-04-21 23:11:35,617 ERROR gpu :22944 [interfaces.py:monitor():144] Failed to sample metric: Not Supported | |
| 2023-04-21 23:11:37,655 ERROR gpu :22944 [interfaces.py:monitor():144] Failed to sample metric: Not Supported | |
| 2023-04-21 23:11:37,816 DEBUG SenderThread:22944 [sender.py:send():375] send: stats | |
| 2023-04-21 23:11:38,032 DEBUG HandlerThread:22944 [handler.py:handle_request():144] handle_request: stop_status | |
| 2023-04-21 23:11:38,032 DEBUG SenderThread:22944 [sender.py:send_request():402] send_request: stop_status | |
| 2023-04-21 23:11:39,286 DEBUG HandlerThread:22944 [handler.py:handle_request():144] handle_request: status_report | |
| 2023-04-21 23:11:39,698 INFO Thread-16 :22944 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230421_230706-1i2p1xzz\files\output.log | |
| 2023-04-21 23:11:39,711 ERROR gpu :22944 [interfaces.py:monitor():144] Failed to sample metric: Not Supported | |
| 2023-04-21 23:11:41,752 ERROR gpu :22944 [interfaces.py:monitor():144] Failed to sample metric: Not Supported | |
| 2023-04-21 23:11:43,869 ERROR gpu :22944 [interfaces.py:monitor():144] Failed to sample metric: Not Supported | |
| 2023-04-21 23:11:44,306 DEBUG HandlerThread:22944 [handler.py:handle_request():144] handle_request: status_report | |
| 2023-04-21 23:11:45,882 ERROR gpu :22944 [interfaces.py:monitor():144] Failed to sample metric: Not Supported | |
| 2023-04-21 23:11:47,899 ERROR gpu :22944 [interfaces.py:monitor():144] Failed to sample metric: Not Supported | |
| 2023-04-21 23:11:49,363 DEBUG HandlerThread:22944 [handler.py:handle_request():144] handle_request: status_report | |
| 2023-04-21 23:11:49,966 ERROR gpu :22944 [interfaces.py:monitor():144] Failed to sample metric: Not Supported | |
| 2023-04-21 23:11:52,023 ERROR gpu :22944 [interfaces.py:monitor():144] Failed to sample metric: Not Supported | |
| 2023-04-21 23:11:53,041 DEBUG HandlerThread:22944 [handler.py:handle_request():144] handle_request: stop_status | |
| 2023-04-21 23:11:53,041 DEBUG SenderThread:22944 [sender.py:send_request():402] send_request: stop_status | |
| 2023-04-21 23:11:54,070 ERROR gpu :22944 [interfaces.py:monitor():144] Failed to sample metric: Not Supported | |
| 2023-04-21 23:11:55,307 DEBUG HandlerThread:22944 [handler.py:handle_request():144] handle_request: status_report | |
| 2023-04-21 23:11:56,111 ERROR gpu :22944 [interfaces.py:monitor():144] Failed to sample metric: Not Supported | |
| 2023-04-21 23:11:58,171 ERROR gpu :22944 [interfaces.py:monitor():144] Failed to sample metric: Not Supported | |
| 2023-04-21 23:12:00,250 ERROR gpu :22944 [interfaces.py:monitor():144] Failed to sample metric: Not Supported | |
| 2023-04-21 23:12:00,352 DEBUG HandlerThread:22944 [handler.py:handle_request():144] handle_request: status_report | |
| 2023-04-21 23:12:02,285 ERROR gpu :22944 [interfaces.py:monitor():144] Failed to sample metric: Not Supported | |
| 2023-04-21 23:12:04,331 ERROR gpu :22944 [interfaces.py:monitor():144] Failed to sample metric: Not Supported | |
| 2023-04-21 23:12:05,863 DEBUG HandlerThread:22944 [handler.py:handle_request():144] handle_request: status_report | |
| 2023-04-21 23:12:06,371 INFO Thread-16 :22944 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230421_230706-1i2p1xzz\files\output.log | |
| 2023-04-21 23:12:06,385 ERROR gpu :22944 [interfaces.py:monitor():144] Failed to sample metric: Not Supported | |
| 2023-04-21 23:12:07,820 DEBUG SenderThread:22944 [sender.py:send():375] send: stats | |
| 2023-04-21 23:12:08,055 DEBUG HandlerThread:22944 [handler.py:handle_request():144] handle_request: stop_status | |
| 2023-04-21 23:12:08,055 DEBUG SenderThread:22944 [sender.py:send_request():402] send_request: stop_status | |
| 2023-04-21 23:12:08,421 ERROR gpu :22944 [interfaces.py:monitor():144] Failed to sample metric: Not Supported | |
| 2023-04-21 23:12:10,470 ERROR gpu :22944 [interfaces.py:monitor():144] Failed to sample metric: Not Supported | |
| 2023-04-21 23:12:11,327 DEBUG HandlerThread:22944 [handler.py:handle_request():144] handle_request: status_report | |
| 2023-04-21 23:12:12,509 ERROR gpu :22944 [interfaces.py:monitor():144] Failed to sample metric: Not Supported | |
| 2023-04-21 23:12:14,620 ERROR gpu :22944 [interfaces.py:monitor():144] Failed to sample metric: Not Supported | |
| 2023-04-21 23:12:16,381 DEBUG HandlerThread:22944 [handler.py:handle_request():144] handle_request: status_report | |
| 2023-04-21 23:12:16,641 ERROR gpu :22944 [interfaces.py:monitor():144] Failed to sample metric: Not Supported | |
| 2023-04-21 23:12:18,661 ERROR gpu :22944 [interfaces.py:monitor():144] Failed to sample metric: Not Supported | |
| 2023-04-21 23:12:20,687 ERROR gpu :22944 [interfaces.py:monitor():144] Failed to sample metric: Not Supported | |
| 2023-04-21 23:12:21,437 DEBUG HandlerThread:22944 [handler.py:handle_request():144] handle_request: status_report | |
| 2023-04-21 23:12:22,741 ERROR gpu :22944 [interfaces.py:monitor():144] Failed to sample metric: Not Supported | |
| 2023-04-21 23:12:23,073 DEBUG HandlerThread:22944 [handler.py:handle_request():144] handle_request: stop_status | |
| 2023-04-21 23:12:23,073 DEBUG SenderThread:22944 [sender.py:send_request():402] send_request: stop_status | |
| 2023-04-21 23:12:24,782 ERROR gpu :22944 [interfaces.py:monitor():144] Failed to sample metric: Not Supported | |
| 2023-04-21 23:12:26,666 DEBUG HandlerThread:22944 [handler.py:handle_request():144] handle_request: status_report | |
| 2023-04-21 23:12:26,824 ERROR gpu :22944 [interfaces.py:monitor():144] Failed to sample metric: Not Supported | |
| 2023-04-21 23:12:27,818 INFO Thread-16 :22944 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230421_230706-1i2p1xzz\files\output.log | |
| 2023-04-21 23:12:28,859 ERROR gpu :22944 [interfaces.py:monitor():144] Failed to sample metric: Not Supported | |
| 2023-04-21 23:12:30,907 ERROR gpu :22944 [interfaces.py:monitor():144] Failed to sample metric: Not Supported | |
| 2023-04-21 23:12:31,711 DEBUG HandlerThread:22944 [handler.py:handle_request():144] handle_request: status_report | |
| 2023-04-21 23:12:32,955 ERROR gpu :22944 [interfaces.py:monitor():144] Failed to sample metric: Not Supported | |
| 2023-04-21 23:12:34,995 ERROR gpu :22944 [interfaces.py:monitor():144] Failed to sample metric: Not Supported | |
| 2023-04-21 23:12:36,751 DEBUG HandlerThread:22944 [handler.py:handle_request():144] handle_request: status_report | |
| 2023-04-21 23:12:37,035 ERROR gpu :22944 [interfaces.py:monitor():144] Failed to sample metric: Not Supported | |
| 2023-04-21 23:12:37,824 DEBUG SenderThread:22944 [sender.py:send():375] send: stats | |
| 2023-04-21 23:12:38,086 DEBUG HandlerThread:22944 [handler.py:handle_request():144] handle_request: stop_status | |
| 2023-04-21 23:12:38,087 DEBUG SenderThread:22944 [sender.py:send_request():402] send_request: stop_status | |
| 2023-04-21 23:12:39,088 ERROR gpu :22944 [interfaces.py:monitor():144] Failed to sample metric: Not Supported | |
| 2023-04-21 23:12:41,129 ERROR gpu :22944 [interfaces.py:monitor():144] Failed to sample metric: Not Supported | |
| 2023-04-21 23:12:42,379 DEBUG HandlerThread:22944 [handler.py:handle_request():144] handle_request: status_report | |
| 2023-04-21 23:12:43,184 ERROR gpu :22944 [interfaces.py:monitor():144] Failed to sample metric: Not Supported | |
| 2023-04-21 23:12:45,276 ERROR gpu :22944 [interfaces.py:monitor():144] Failed to sample metric: Not Supported | |
| 2023-04-21 23:12:47,304 ERROR gpu :22944 [interfaces.py:monitor():144] Failed to sample metric: Not Supported | |
| 2023-04-21 23:12:47,419 DEBUG HandlerThread:22944 [handler.py:handle_request():144] handle_request: status_report | |
| 2023-04-21 23:12:49,327 ERROR gpu :22944 [interfaces.py:monitor():144] Failed to sample metric: Not Supported | |
| 2023-04-21 23:12:50,318 INFO Thread-16 :22944 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230421_230706-1i2p1xzz\files\output.log | |
| 2023-04-21 23:12:51,372 ERROR gpu :22944 [interfaces.py:monitor():144] Failed to sample metric: Not Supported | |
| 2023-04-21 23:12:52,918 DEBUG HandlerThread:22944 [handler.py:handle_request():144] handle_request: status_report | |
| 2023-04-21 23:12:53,105 DEBUG HandlerThread:22944 [handler.py:handle_request():144] handle_request: stop_status | |
| 2023-04-21 23:12:53,105 DEBUG SenderThread:22944 [sender.py:send_request():402] send_request: stop_status | |
| 2023-04-21 23:12:53,412 ERROR gpu :22944 [interfaces.py:monitor():144] Failed to sample metric: Not Supported | |
| 2023-04-21 23:12:55,472 ERROR gpu :22944 [interfaces.py:monitor():144] Failed to sample metric: Not Supported | |
| 2023-04-21 23:12:57,516 ERROR gpu :22944 [interfaces.py:monitor():144] Failed to sample metric: Not Supported | |
| 2023-04-21 23:12:58,422 DEBUG HandlerThread:22944 [handler.py:handle_request():144] handle_request: status_report | |
| 2023-04-21 23:12:59,560 ERROR gpu :22944 [interfaces.py:monitor():144] Failed to sample metric: Not Supported | |
| 2023-04-21 23:13:01,606 ERROR gpu :22944 [interfaces.py:monitor():144] Failed to sample metric: Not Supported | |
| 2023-04-21 23:13:03,464 DEBUG HandlerThread:22944 [handler.py:handle_request():144] handle_request: status_report | |
| 2023-04-21 23:13:03,653 ERROR gpu :22944 [interfaces.py:monitor():144] Failed to sample metric: Not Supported | |
| 2023-04-21 23:13:05,695 ERROR gpu :22944 [interfaces.py:monitor():144] Failed to sample metric: Not Supported | |
| 2023-04-21 23:13:07,731 ERROR gpu :22944 [interfaces.py:monitor():144] Failed to sample metric: Not Supported | |
| 2023-04-21 23:13:07,827 DEBUG SenderThread:22944 [sender.py:send():375] send: stats | |
| 2023-04-21 23:13:08,119 DEBUG HandlerThread:22944 [handler.py:handle_request():144] handle_request: stop_status | |
| 2023-04-21 23:13:08,120 DEBUG SenderThread:22944 [sender.py:send_request():402] send_request: stop_status | |
| 2023-04-21 23:13:09,380 DEBUG HandlerThread:22944 [handler.py:handle_request():144] handle_request: status_report | |
| 2023-04-21 23:13:09,801 ERROR gpu :22944 [interfaces.py:monitor():144] Failed to sample metric: Not Supported | |
| 2023-04-21 23:13:11,831 ERROR gpu :22944 [interfaces.py:monitor():144] Failed to sample metric: Not Supported | |
| 2023-04-21 23:13:13,859 INFO Thread-16 :22944 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230421_230706-1i2p1xzz\files\output.log | |
| 2023-04-21 23:13:13,867 ERROR gpu :22944 [interfaces.py:monitor():144] Failed to sample metric: Not Supported | |
| 2023-04-21 23:13:14,392 DEBUG HandlerThread:22944 [handler.py:handle_request():144] handle_request: status_report | |
| 2023-04-21 23:13:15,967 ERROR gpu :22944 [interfaces.py:monitor():144] Failed to sample metric: Not Supported | |
| 2023-04-21 23:13:17,991 ERROR gpu :22944 [interfaces.py:monitor():144] Failed to sample metric: Not Supported | |
| 2023-04-21 23:13:19,433 DEBUG HandlerThread:22944 [handler.py:handle_request():144] handle_request: status_report | |
| 2023-04-21 23:13:20,047 ERROR gpu :22944 [interfaces.py:monitor():144] Failed to sample metric: Not Supported | |
| 2023-04-21 23:13:22,089 ERROR gpu :22944 [interfaces.py:monitor():144] Failed to sample metric: Not Supported | |
| 2023-04-21 23:13:23,122 DEBUG HandlerThread:22944 [handler.py:handle_request():144] handle_request: stop_status | |
| 2023-04-21 23:13:23,122 DEBUG SenderThread:22944 [sender.py:send_request():402] send_request: stop_status | |
| 2023-04-21 23:13:24,134 ERROR gpu :22944 [interfaces.py:monitor():144] Failed to sample metric: Not Supported | |
| 2023-04-21 23:13:25,392 DEBUG HandlerThread:22944 [handler.py:handle_request():144] handle_request: status_report | |
| 2023-04-21 23:13:26,178 ERROR gpu :22944 [interfaces.py:monitor():144] Failed to sample metric: Not Supported | |
| 2023-04-21 23:13:28,220 ERROR gpu :22944 [interfaces.py:monitor():144] Failed to sample metric: Not Supported | |
| 2023-04-21 23:13:30,291 ERROR gpu :22944 [interfaces.py:monitor():144] Failed to sample metric: Not Supported | |
| 2023-04-21 23:13:30,421 DEBUG HandlerThread:22944 [handler.py:handle_request():144] handle_request: status_report | |
| 2023-04-21 23:13:32,324 ERROR gpu :22944 [interfaces.py:monitor():144] Failed to sample metric: Not Supported | |
| 2023-04-21 23:13:34,379 ERROR gpu :22944 [interfaces.py:monitor():144] Failed to sample metric: Not Supported | |
| 2023-04-21 23:13:35,477 DEBUG HandlerThread:22944 [handler.py:handle_request():144] handle_request: status_report | |
| 2023-04-21 23:13:36,418 INFO Thread-16 :22944 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230421_230706-1i2p1xzz\files\output.log | |
| 2023-04-21 23:13:36,428 ERROR gpu :22944 [interfaces.py:monitor():144] Failed to sample metric: Not Supported | |
| 2023-04-21 23:13:37,840 DEBUG SenderThread:22944 [sender.py:send():375] send: stats | |
| 2023-04-21 23:13:38,138 DEBUG HandlerThread:22944 [handler.py:handle_request():144] handle_request: stop_status | |
| 2023-04-21 23:13:38,139 DEBUG SenderThread:22944 [sender.py:send_request():402] send_request: stop_status | |
| 2023-04-21 23:13:38,464 ERROR gpu :22944 [interfaces.py:monitor():144] Failed to sample metric: Not Supported | |
| 2023-04-21 23:13:40,505 ERROR gpu :22944 [interfaces.py:monitor():144] Failed to sample metric: Not Supported | |
| 2023-04-21 23:13:41,158 DEBUG HandlerThread:22944 [handler.py:handle_request():144] handle_request: status_report | |
| 2023-04-21 23:13:42,562 ERROR gpu :22944 [interfaces.py:monitor():144] Failed to sample metric: Not Supported | |
| 2023-04-21 23:13:44,599 ERROR gpu :22944 [interfaces.py:monitor():144] Failed to sample metric: Not Supported | |
| 2023-04-21 23:13:46,199 DEBUG HandlerThread:22944 [handler.py:handle_request():144] handle_request: status_report | |
| 2023-04-21 23:13:46,696 ERROR gpu :22944 [interfaces.py:monitor():144] Failed to sample metric: Not Supported | |
| 2023-04-21 23:13:48,718 ERROR gpu :22944 [interfaces.py:monitor():144] Failed to sample metric: Not Supported | |
| 2023-04-21 23:13:50,737 ERROR gpu :22944 [interfaces.py:monitor():144] Failed to sample metric: Not Supported | |
| 2023-04-21 23:13:51,242 DEBUG HandlerThread:22944 [handler.py:handle_request():144] handle_request: status_report | |
| 2023-04-21 23:13:52,762 ERROR gpu :22944 [interfaces.py:monitor():144] Failed to sample metric: Not Supported | |
| 2023-04-21 23:13:53,147 DEBUG HandlerThread:22944 [handler.py:handle_request():144] handle_request: stop_status | |
| 2023-04-21 23:13:53,148 DEBUG SenderThread:22944 [sender.py:send_request():402] send_request: stop_status | |
| 2023-04-21 23:13:54,805 ERROR gpu :22944 [interfaces.py:monitor():144] Failed to sample metric: Not Supported | |
| 2023-04-21 23:13:56,433 DEBUG HandlerThread:22944 [handler.py:handle_request():144] handle_request: status_report | |
| 2023-04-21 23:13:56,857 ERROR gpu :22944 [interfaces.py:monitor():144] Failed to sample metric: Not Supported | |
| 2023-04-21 23:13:58,899 INFO Thread-16 :22944 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230421_230706-1i2p1xzz\files\output.log | |
| 2023-04-21 23:13:58,901 ERROR gpu :22944 [interfaces.py:monitor():144] Failed to sample metric: Not Supported | |
| 2023-04-21 23:14:00,945 ERROR gpu :22944 [interfaces.py:monitor():144] Failed to sample metric: Not Supported | |
| 2023-04-21 23:14:01,478 DEBUG HandlerThread:22944 [handler.py:handle_request():144] handle_request: status_report | |
| 2023-04-21 23:14:02,988 ERROR gpu :22944 [interfaces.py:monitor():144] Failed to sample metric: Not Supported | |
| 2023-04-21 23:14:05,049 ERROR gpu :22944 [interfaces.py:monitor():144] Failed to sample metric: Not Supported | |
| 2023-04-21 23:14:06,539 DEBUG HandlerThread:22944 [handler.py:handle_request():144] handle_request: status_report | |
| 2023-04-21 23:14:07,087 ERROR gpu :22944 [interfaces.py:monitor():144] Failed to sample metric: Not Supported | |
| 2023-04-21 23:14:07,844 DEBUG SenderThread:22944 [sender.py:send():375] send: stats | |
| 2023-04-21 23:14:08,157 DEBUG HandlerThread:22944 [handler.py:handle_request():144] handle_request: stop_status | |
| 2023-04-21 23:14:08,158 DEBUG SenderThread:22944 [sender.py:send_request():402] send_request: stop_status | |
| 2023-04-21 23:14:09,129 ERROR gpu :22944 [interfaces.py:monitor():144] Failed to sample metric: Not Supported | |
| 2023-04-21 23:14:11,170 ERROR gpu :22944 [interfaces.py:monitor():144] Failed to sample metric: Not Supported | |
| 2023-04-21 23:14:11,597 DEBUG HandlerThread:22944 [handler.py:handle_request():144] handle_request: status_report | |
| 2023-04-21 23:14:13,227 ERROR gpu :22944 [interfaces.py:monitor():144] Failed to sample metric: Not Supported | |
| 2023-04-21 23:14:15,264 ERROR gpu :22944 [interfaces.py:monitor():144] Failed to sample metric: Not Supported | |
| 2023-04-21 23:14:16,647 DEBUG HandlerThread:22944 [handler.py:handle_request():144] handle_request: status_report | |
| 2023-04-21 23:14:17,374 ERROR gpu :22944 [interfaces.py:monitor():144] Failed to sample metric: Not Supported | |
| 2023-04-21 23:14:19,388 ERROR gpu :22944 [interfaces.py:monitor():144] Failed to sample metric: Not Supported | |
| 2023-04-21 23:14:21,443 ERROR gpu :22944 [interfaces.py:monitor():144] Failed to sample metric: Not Supported | |
| 2023-04-21 23:14:21,998 DEBUG HandlerThread:22944 [handler.py:handle_request():144] handle_request: status_report | |
| 2023-04-21 23:14:22,435 INFO Thread-16 :22944 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230421_230706-1i2p1xzz\files\output.log | |
| 2023-04-21 23:14:23,165 DEBUG HandlerThread:22944 [handler.py:handle_request():144] handle_request: stop_status | |
| 2023-04-21 23:14:23,165 DEBUG SenderThread:22944 [sender.py:send_request():402] send_request: stop_status | |
| 2023-04-21 23:14:23,475 ERROR gpu :22944 [interfaces.py:monitor():144] Failed to sample metric: Not Supported | |
| 2023-04-21 23:14:25,513 ERROR gpu :22944 [interfaces.py:monitor():144] Failed to sample metric: Not Supported | |
| 2023-04-21 23:14:27,463 DEBUG HandlerThread:22944 [handler.py:handle_request():144] handle_request: status_report | |
| 2023-04-21 23:14:27,566 ERROR gpu :22944 [interfaces.py:monitor():144] Failed to sample metric: Not Supported | |
| 2023-04-21 23:14:29,611 ERROR gpu :22944 [interfaces.py:monitor():144] Failed to sample metric: Not Supported | |
| 2023-04-21 23:14:31,668 ERROR gpu :22944 [interfaces.py:monitor():144] Failed to sample metric: Not Supported | |
| 2023-04-21 23:14:32,512 DEBUG HandlerThread:22944 [handler.py:handle_request():144] handle_request: status_report | |
| 2023-04-21 23:14:33,715 ERROR gpu :22944 [interfaces.py:monitor():144] Failed to sample metric: Not Supported | |
| 2023-04-21 23:14:35,759 ERROR gpu :22944 [interfaces.py:monitor():144] Failed to sample metric: Not Supported | |
| 2023-04-21 23:14:37,563 DEBUG HandlerThread:22944 [handler.py:handle_request():144] handle_request: status_report | |
| 2023-04-21 23:14:37,813 ERROR gpu :22944 [interfaces.py:monitor():144] Failed to sample metric: Not Supported | |
| 2023-04-21 23:14:37,856 DEBUG SenderThread:22944 [sender.py:send():375] send: stats | |
| 2023-04-21 23:14:38,181 DEBUG HandlerThread:22944 [handler.py:handle_request():144] handle_request: stop_status | |
| 2023-04-21 23:14:38,182 DEBUG SenderThread:22944 [sender.py:send_request():402] send_request: stop_status | |
| 2023-04-21 23:14:39,853 ERROR gpu :22944 [interfaces.py:monitor():144] Failed to sample metric: Not Supported | |
| 2023-04-21 23:14:41,899 ERROR gpu :22944 [interfaces.py:monitor():144] Failed to sample metric: Not Supported | |
| 2023-04-21 23:14:43,474 DEBUG HandlerThread:22944 [handler.py:handle_request():144] handle_request: status_report | |
| 2023-04-21 23:14:43,937 ERROR gpu :22944 [interfaces.py:monitor():144] Failed to sample metric: Not Supported | |
| 2023-04-21 23:14:45,987 ERROR gpu :22944 [interfaces.py:monitor():144] Failed to sample metric: Not Supported | |
| 2023-04-21 23:14:46,981 INFO Thread-16 :22944 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230421_230706-1i2p1xzz\files\output.log | |
| 2023-04-21 23:14:48,092 ERROR gpu :22944 [interfaces.py:monitor():144] Failed to sample metric: Not Supported | |
| 2023-04-21 23:14:49,157 DEBUG HandlerThread:22944 [handler.py:handle_request():144] handle_request: status_report | |
| 2023-04-21 23:14:50,113 ERROR gpu :22944 [interfaces.py:monitor():144] Failed to sample metric: Not Supported | |
| 2023-04-21 23:14:52,133 ERROR gpu :22944 [interfaces.py:monitor():144] Failed to sample metric: Not Supported | |
| 2023-04-21 23:14:53,180 DEBUG HandlerThread:22944 [handler.py:handle_request():144] handle_request: stop_status | |
| 2023-04-21 23:14:53,180 DEBUG SenderThread:22944 [sender.py:send_request():402] send_request: stop_status | |
| 2023-04-21 23:14:54,181 ERROR gpu :22944 [interfaces.py:monitor():144] Failed to sample metric: Not Supported | |
| 2023-04-21 23:14:54,437 DEBUG HandlerThread:22944 [handler.py:handle_request():144] handle_request: status_report | |
| 2023-04-21 23:14:56,229 ERROR gpu :22944 [interfaces.py:monitor():144] Failed to sample metric: Not Supported | |
| 2023-04-21 23:14:58,266 ERROR gpu :22944 [interfaces.py:monitor():144] Failed to sample metric: Not Supported | |
| 2023-04-21 23:14:59,497 DEBUG HandlerThread:22944 [handler.py:handle_request():144] handle_request: status_report | |
| 2023-04-21 23:15:00,362 ERROR gpu :22944 [interfaces.py:monitor():144] Failed to sample metric: Not Supported | |
| 2023-04-21 23:15:02,399 ERROR gpu :22944 [interfaces.py:monitor():144] Failed to sample metric: Not Supported | |
| 2023-04-21 23:15:04,439 ERROR gpu :22944 [interfaces.py:monitor():144] Failed to sample metric: Not Supported | |
| 2023-04-21 23:15:04,537 DEBUG HandlerThread:22944 [handler.py:handle_request():144] handle_request: status_report | |
| 2023-04-21 23:15:06,479 ERROR gpu :22944 [interfaces.py:monitor():144] Failed to sample metric: Not Supported | |
| 2023-04-21 23:15:07,866 DEBUG SenderThread:22944 [sender.py:send():375] send: stats | |
| 2023-04-21 23:15:08,192 DEBUG HandlerThread:22944 [handler.py:handle_request():144] handle_request: stop_status | |
| 2023-04-21 23:15:08,192 DEBUG SenderThread:22944 [sender.py:send_request():402] send_request: stop_status | |
| 2023-04-21 23:15:08,509 DEBUG HandlerThread:22944 [handler.py:handle_request():144] handle_request: partial_history | |
| 2023-04-21 23:15:08,512 DEBUG SenderThread:22944 [sender.py:send():375] send: history | |
| 2023-04-21 23:15:08,513 DEBUG SenderThread:22944 [sender.py:send_request():402] send_request: summary_record | |
| 2023-04-21 23:15:08,513 INFO SenderThread:22944 [sender.py:_save_file():1378] saving file wandb-summary.json with policy end | |
| 2023-04-21 23:15:08,515 INFO Thread-16 :22944 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230421_230706-1i2p1xzz\files\wandb-summary.json | |
| 2023-04-21 23:15:08,523 ERROR gpu :22944 [interfaces.py:monitor():144] Failed to sample metric: Not Supported | |
| 2023-04-21 23:15:09,529 INFO Thread-16 :22944 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230421_230706-1i2p1xzz\files\output.log | |
| 2023-04-21 23:15:09,750 DEBUG HandlerThread:22944 [handler.py:handle_request():144] handle_request: status_report | |
| 2023-04-21 23:15:10,555 INFO Thread-16 :22944 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230421_230706-1i2p1xzz\files\output.log | |
| 2023-04-21 23:15:10,567 ERROR gpu :22944 [interfaces.py:monitor():144] Failed to sample metric: Not Supported | |
| 2023-04-21 23:15:12,615 ERROR gpu :22944 [interfaces.py:monitor():144] Failed to sample metric: Not Supported | |
| 2023-04-21 23:15:14,651 ERROR gpu :22944 [interfaces.py:monitor():144] Failed to sample metric: Not Supported | |
| 2023-04-21 23:15:14,805 DEBUG HandlerThread:22944 [handler.py:handle_request():144] handle_request: status_report | |
| 2023-04-21 23:15:16,702 ERROR gpu :22944 [interfaces.py:monitor():144] Failed to sample metric: Not Supported | |
| 2023-04-21 23:15:18,805 ERROR gpu :22944 [interfaces.py:monitor():144] Failed to sample metric: Not Supported | |
| 2023-04-21 23:15:19,858 DEBUG HandlerThread:22944 [handler.py:handle_request():144] handle_request: status_report | |
| 2023-04-21 23:15:20,833 ERROR gpu :22944 [interfaces.py:monitor():144] Failed to sample metric: Not Supported | |
| 2023-04-21 23:15:22,850 ERROR gpu :22944 [interfaces.py:monitor():144] Failed to sample metric: Not Supported | |
| 2023-04-21 23:15:23,215 DEBUG HandlerThread:22944 [handler.py:handle_request():144] handle_request: stop_status | |
| 2023-04-21 23:15:23,215 DEBUG SenderThread:22944 [sender.py:send_request():402] send_request: stop_status | |
| 2023-04-21 23:15:24,888 ERROR gpu :22944 [interfaces.py:monitor():144] Failed to sample metric: Not Supported | |
| 2023-04-21 23:15:25,480 DEBUG HandlerThread:22944 [handler.py:handle_request():144] handle_request: status_report | |
| 2023-04-21 23:15:26,923 ERROR gpu :22944 [interfaces.py:monitor():144] Failed to sample metric: Not Supported | |
| 2023-04-21 23:15:28,973 ERROR gpu :22944 [interfaces.py:monitor():144] Failed to sample metric: Not Supported | |
| 2023-04-21 23:15:30,522 DEBUG HandlerThread:22944 [handler.py:handle_request():144] handle_request: status_report | |
| 2023-04-21 23:15:31,021 ERROR gpu :22944 [interfaces.py:monitor():144] Failed to sample metric: Not Supported | |
| 2023-04-21 23:15:33,061 INFO Thread-16 :22944 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230421_230706-1i2p1xzz\files\output.log | |
| 2023-04-21 23:15:33,069 ERROR gpu :22944 [interfaces.py:monitor():144] Failed to sample metric: Not Supported | |
| 2023-04-21 23:15:35,114 ERROR gpu :22944 [interfaces.py:monitor():144] Failed to sample metric: Not Supported | |
| 2023-04-21 23:15:36,483 DEBUG HandlerThread:22944 [handler.py:handle_request():144] handle_request: status_report | |
| 2023-04-21 23:15:37,153 ERROR gpu :22944 [interfaces.py:monitor():144] Failed to sample metric: Not Supported | |
| 2023-04-21 23:15:37,869 DEBUG SenderThread:22944 [sender.py:send():375] send: stats | |
| 2023-04-21 23:15:38,212 DEBUG HandlerThread:22944 [handler.py:handle_request():144] handle_request: stop_status | |
| 2023-04-21 23:15:38,212 DEBUG SenderThread:22944 [sender.py:send_request():402] send_request: stop_status | |
| 2023-04-21 23:15:39,194 ERROR gpu :22944 [interfaces.py:monitor():144] Failed to sample metric: Not Supported | |
| 2023-04-21 23:15:41,237 ERROR gpu :22944 [interfaces.py:monitor():144] Failed to sample metric: Not Supported | |
| 2023-04-21 23:15:41,488 DEBUG HandlerThread:22944 [handler.py:handle_request():144] handle_request: status_report | |
| 2023-04-21 23:15:43,277 ERROR gpu :22944 [interfaces.py:monitor():144] Failed to sample metric: Not Supported | |
| 2023-04-21 23:15:45,319 ERROR gpu :22944 [interfaces.py:monitor():144] Failed to sample metric: Not Supported | |
| 2023-04-21 23:15:46,538 DEBUG HandlerThread:22944 [handler.py:handle_request():144] handle_request: status_report | |
| 2023-04-21 23:15:47,363 ERROR gpu :22944 [interfaces.py:monitor():144] Failed to sample metric: Not Supported | |
| 2023-04-21 23:15:49,462 ERROR gpu :22944 [interfaces.py:monitor():144] Failed to sample metric: Not Supported | |
| 2023-04-21 23:15:51,476 ERROR gpu :22944 [interfaces.py:monitor():144] Failed to sample metric: Not Supported | |
| 2023-04-21 23:15:51,601 DEBUG HandlerThread:22944 [handler.py:handle_request():144] handle_request: status_report | |
| 2023-04-21 23:15:53,226 DEBUG HandlerThread:22944 [handler.py:handle_request():144] handle_request: stop_status | |
| 2023-04-21 23:15:53,226 DEBUG SenderThread:22944 [sender.py:send_request():402] send_request: stop_status | |
| 2023-04-21 23:15:53,501 ERROR gpu :22944 [interfaces.py:monitor():144] Failed to sample metric: Not Supported | |
| 2023-04-21 23:15:54,498 INFO Thread-16 :22944 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230421_230706-1i2p1xzz\files\output.log | |
| 2023-04-21 23:15:55,551 ERROR gpu :22944 [interfaces.py:monitor():144] Failed to sample metric: Not Supported | |
| 2023-04-21 23:15:56,785 DEBUG HandlerThread:22944 [handler.py:handle_request():144] handle_request: status_report | |
| 2023-04-21 23:15:57,598 ERROR gpu :22944 [interfaces.py:monitor():144] Failed to sample metric: Not Supported | |
| 2023-04-21 23:15:59,631 ERROR gpu :22944 [interfaces.py:monitor():144] Failed to sample metric: Not Supported | |
| 2023-04-21 23:16:01,696 ERROR gpu :22944 [interfaces.py:monitor():144] Failed to sample metric: Not Supported | |
| 2023-04-21 23:16:01,829 DEBUG HandlerThread:22944 [handler.py:handle_request():144] handle_request: status_report | |
| 2023-04-21 23:16:03,738 ERROR gpu :22944 [interfaces.py:monitor():144] Failed to sample metric: Not Supported | |
| 2023-04-21 23:16:05,801 ERROR gpu :22944 [interfaces.py:monitor():144] Failed to sample metric: Not Supported | |
| 2023-04-21 23:16:06,882 DEBUG HandlerThread:22944 [handler.py:handle_request():144] handle_request: status_report | |
| 2023-04-21 23:16:07,844 ERROR gpu :22944 [interfaces.py:monitor():144] Failed to sample metric: Not Supported | |
| 2023-04-21 23:16:07,877 DEBUG SenderThread:22944 [sender.py:send():375] send: stats | |
| 2023-04-21 23:16:08,248 DEBUG HandlerThread:22944 [handler.py:handle_request():144] handle_request: stop_status | |
| 2023-04-21 23:16:08,249 DEBUG SenderThread:22944 [sender.py:send_request():402] send_request: stop_status | |
| 2023-04-21 23:16:09,894 ERROR gpu :22944 [interfaces.py:monitor():144] Failed to sample metric: Not Supported | |
| 2023-04-21 23:16:11,935 ERROR gpu :22944 [interfaces.py:monitor():144] Failed to sample metric: Not Supported | |
| 2023-04-21 23:16:12,557 DEBUG HandlerThread:22944 [handler.py:handle_request():144] handle_request: status_report | |
| 2023-04-21 23:16:13,974 ERROR gpu :22944 [interfaces.py:monitor():144] Failed to sample metric: Not Supported | |
| 2023-04-21 23:16:16,021 ERROR gpu :22944 [interfaces.py:monitor():144] Failed to sample metric: Not Supported | |
| 2023-04-21 23:16:18,080 ERROR gpu :22944 [interfaces.py:monitor():144] Failed to sample metric: Not Supported | |
| 2023-04-21 23:16:18,474 DEBUG HandlerThread:22944 [handler.py:handle_request():144] handle_request: status_report | |
| 2023-04-21 23:16:19,078 INFO Thread-16 :22944 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230421_230706-1i2p1xzz\files\output.log | |
| 2023-04-21 23:16:20,183 ERROR gpu :22944 [interfaces.py:monitor():144] Failed to sample metric: Not Supported | |
| 2023-04-21 23:16:22,207 ERROR gpu :22944 [interfaces.py:monitor():144] Failed to sample metric: Not Supported | |
| 2023-04-21 23:16:23,259 DEBUG HandlerThread:22944 [handler.py:handle_request():144] handle_request: stop_status | |
| 2023-04-21 23:16:23,260 DEBUG SenderThread:22944 [sender.py:send_request():402] send_request: stop_status | |
| 2023-04-21 23:16:23,542 DEBUG HandlerThread:22944 [handler.py:handle_request():144] handle_request: status_report | |
| 2023-04-21 23:16:24,238 ERROR gpu :22944 [interfaces.py:monitor():144] Failed to sample metric: Not Supported | |
| 2023-04-21 23:16:26,277 ERROR gpu :22944 [interfaces.py:monitor():144] Failed to sample metric: Not Supported | |
| 2023-04-21 23:16:28,318 ERROR gpu :22944 [interfaces.py:monitor():144] Failed to sample metric: Not Supported | |
| 2023-04-21 23:16:28,581 DEBUG HandlerThread:22944 [handler.py:handle_request():144] handle_request: status_report | |
| 2023-04-21 23:16:30,378 ERROR gpu :22944 [interfaces.py:monitor():144] Failed to sample metric: Not Supported | |
| 2023-04-21 23:16:32,421 ERROR gpu :22944 [interfaces.py:monitor():144] Failed to sample metric: Not Supported | |
| 2023-04-21 23:16:33,637 DEBUG HandlerThread:22944 [handler.py:handle_request():144] handle_request: status_report | |
| 2023-04-21 23:16:34,476 ERROR gpu :22944 [interfaces.py:monitor():144] Failed to sample metric: Not Supported | |
| 2023-04-21 23:16:36,522 ERROR gpu :22944 [interfaces.py:monitor():144] Failed to sample metric: Not Supported | |
| 2023-04-21 23:16:37,881 DEBUG SenderThread:22944 [sender.py:send():375] send: stats | |
| 2023-04-21 23:16:38,275 DEBUG HandlerThread:22944 [handler.py:handle_request():144] handle_request: stop_status | |
| 2023-04-21 23:16:38,276 DEBUG SenderThread:22944 [sender.py:send_request():402] send_request: stop_status | |
| 2023-04-21 23:16:38,569 ERROR gpu :22944 [interfaces.py:monitor():144] Failed to sample metric: Not Supported | |
| 2023-04-21 23:16:39,576 DEBUG HandlerThread:22944 [handler.py:handle_request():144] handle_request: status_report | |
| 2023-04-21 23:16:40,620 ERROR gpu :22944 [interfaces.py:monitor():144] Failed to sample metric: Not Supported | |
| 2023-04-21 23:16:41,615 INFO Thread-16 :22944 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230421_230706-1i2p1xzz\files\output.log | |
| 2023-04-21 23:16:42,675 ERROR gpu :22944 [interfaces.py:monitor():144] Failed to sample metric: Not Supported | |
| 2023-04-21 23:16:44,630 DEBUG HandlerThread:22944 [handler.py:handle_request():144] handle_request: status_report | |
| 2023-04-21 23:16:44,719 ERROR gpu :22944 [interfaces.py:monitor():144] Failed to sample metric: Not Supported | |
| 2023-04-21 23:16:46,769 ERROR gpu :22944 [interfaces.py:monitor():144] Failed to sample metric: Not Supported | |
| 2023-04-21 23:16:48,805 ERROR gpu :22944 [interfaces.py:monitor():144] Failed to sample metric: Not Supported | |
| 2023-04-21 23:16:49,678 DEBUG HandlerThread:22944 [handler.py:handle_request():144] handle_request: status_report | |
| 2023-04-21 23:16:50,907 ERROR gpu :22944 [interfaces.py:monitor():144] Failed to sample metric: Not Supported | |
| 2023-04-21 23:16:52,922 ERROR gpu :22944 [interfaces.py:monitor():144] Failed to sample metric: Not Supported | |
| 2023-04-21 23:16:53,299 DEBUG HandlerThread:22944 [handler.py:handle_request():144] handle_request: stop_status | |
| 2023-04-21 23:16:53,300 DEBUG SenderThread:22944 [sender.py:send_request():402] send_request: stop_status | |
| 2023-04-21 23:16:54,942 ERROR gpu :22944 [interfaces.py:monitor():144] Failed to sample metric: Not Supported | |
| 2023-04-21 23:16:55,583 DEBUG HandlerThread:22944 [handler.py:handle_request():144] handle_request: status_report | |
| 2023-04-21 23:16:56,968 ERROR gpu :22944 [interfaces.py:monitor():144] Failed to sample metric: Not Supported | |
| 2023-04-21 23:16:59,012 ERROR gpu :22944 [interfaces.py:monitor():144] Failed to sample metric: Not Supported | |
| 2023-04-21 23:17:00,673 DEBUG HandlerThread:22944 [handler.py:handle_request():144] handle_request: status_report | |
| 2023-04-21 23:17:01,051 INFO Thread-16 :22944 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230421_230706-1i2p1xzz\files\output.log | |
| 2023-04-21 23:17:01,059 ERROR gpu :22944 [interfaces.py:monitor():144] Failed to sample metric: Not Supported | |
| 2023-04-21 23:17:03,096 ERROR gpu :22944 [interfaces.py:monitor():144] Failed to sample metric: Not Supported | |
| 2023-04-21 23:17:05,148 ERROR gpu :22944 [interfaces.py:monitor():144] Failed to sample metric: Not Supported | |
| 2023-04-21 23:17:05,705 DEBUG HandlerThread:22944 [handler.py:handle_request():144] handle_request: status_report | |
| 2023-04-21 23:17:07,185 ERROR gpu :22944 [interfaces.py:monitor():144] Failed to sample metric: Not Supported | |
| 2023-04-21 23:17:07,889 DEBUG SenderThread:22944 [sender.py:send():375] send: stats | |
| 2023-04-21 23:17:08,315 DEBUG HandlerThread:22944 [handler.py:handle_request():144] handle_request: stop_status | |
| 2023-04-21 23:17:08,316 DEBUG SenderThread:22944 [sender.py:send_request():402] send_request: stop_status | |
| 2023-04-21 23:17:09,233 ERROR gpu :22944 [interfaces.py:monitor():144] Failed to sample metric: Not Supported | |
| 2023-04-21 23:17:11,266 ERROR gpu :22944 [interfaces.py:monitor():144] Failed to sample metric: Not Supported | |
| 2023-04-21 23:17:11,606 DEBUG HandlerThread:22944 [handler.py:handle_request():144] handle_request: status_report | |
| 2023-04-21 23:17:13,320 ERROR gpu :22944 [interfaces.py:monitor():144] Failed to sample metric: Not Supported | |
| 2023-04-21 23:17:15,364 ERROR gpu :22944 [interfaces.py:monitor():144] Failed to sample metric: Not Supported | |
| 2023-04-21 23:17:16,653 DEBUG HandlerThread:22944 [handler.py:handle_request():144] handle_request: status_report | |
| 2023-04-21 23:17:17,403 ERROR gpu :22944 [interfaces.py:monitor():144] Failed to sample metric: Not Supported | |
| 2023-04-21 23:17:19,439 INFO Thread-16 :22944 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230421_230706-1i2p1xzz\files\output.log | |
| 2023-04-21 23:17:19,453 ERROR gpu :22944 [interfaces.py:monitor():144] Failed to sample metric: Not Supported | |
| 2023-04-21 23:17:21,541 ERROR gpu :22944 [interfaces.py:monitor():144] Failed to sample metric: Not Supported | |
| 2023-04-21 23:17:22,124 DEBUG HandlerThread:22944 [handler.py:handle_request():144] handle_request: status_report | |
| 2023-04-21 23:17:23,329 DEBUG HandlerThread:22944 [handler.py:handle_request():144] handle_request: stop_status | |
| 2023-04-21 23:17:23,330 DEBUG SenderThread:22944 [sender.py:send_request():402] send_request: stop_status | |
| 2023-04-21 23:17:23,555 ERROR gpu :22944 [interfaces.py:monitor():144] Failed to sample metric: Not Supported | |
| 2023-04-21 23:17:25,566 ERROR gpu :22944 [interfaces.py:monitor():144] Failed to sample metric: Not Supported | |
| 2023-04-21 23:17:27,612 ERROR gpu :22944 [interfaces.py:monitor():144] Failed to sample metric: Not Supported | |
| 2023-04-21 23:17:27,656 DEBUG HandlerThread:22944 [handler.py:handle_request():144] handle_request: status_report | |
| 2023-04-21 23:17:29,652 ERROR gpu :22944 [interfaces.py:monitor():144] Failed to sample metric: Not Supported | |
| 2023-04-21 23:17:31,686 ERROR gpu :22944 [interfaces.py:monitor():144] Failed to sample metric: Not Supported | |
| 2023-04-21 23:17:32,952 DEBUG HandlerThread:22944 [handler.py:handle_request():144] handle_request: status_report | |
| 2023-04-21 23:17:33,723 INFO Thread-16 :22944 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230421_230706-1i2p1xzz\files\output.log | |
| 2023-04-21 23:17:33,726 ERROR gpu :22944 [interfaces.py:monitor():144] Failed to sample metric: Not Supported | |
| 2023-04-21 23:17:35,782 ERROR gpu :22944 [interfaces.py:monitor():144] Failed to sample metric: Not Supported | |
| 2023-04-21 23:17:37,824 ERROR gpu :22944 [interfaces.py:monitor():144] Failed to sample metric: Not Supported | |
| 2023-04-21 23:17:37,893 DEBUG SenderThread:22944 [sender.py:send():375] send: stats | |
| 2023-04-21 23:17:38,331 DEBUG HandlerThread:22944 [handler.py:handle_request():144] handle_request: stop_status | |
| 2023-04-21 23:17:38,331 DEBUG SenderThread:22944 [sender.py:send_request():402] send_request: stop_status | |
| 2023-04-21 23:17:38,588 DEBUG HandlerThread:22944 [handler.py:handle_request():144] handle_request: status_report | |
| 2023-04-21 23:17:39,876 ERROR gpu :22944 [interfaces.py:monitor():144] Failed to sample metric: Not Supported | |
| 2023-04-21 23:17:41,914 ERROR gpu :22944 [interfaces.py:monitor():144] Failed to sample metric: Not Supported | |
| 2023-04-21 23:17:43,630 DEBUG HandlerThread:22944 [handler.py:handle_request():144] handle_request: status_report | |
| 2023-04-21 23:17:43,958 ERROR gpu :22944 [interfaces.py:monitor():144] Failed to sample metric: Not Supported | |
| 2023-04-21 23:17:46,001 ERROR gpu :22944 [interfaces.py:monitor():144] Failed to sample metric: Not Supported | |
| 2023-04-21 23:17:47,001 INFO Thread-16 :22944 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230421_230706-1i2p1xzz\files\output.log | |
| 2023-04-21 23:17:48,044 ERROR gpu :22944 [interfaces.py:monitor():144] Failed to sample metric: Not Supported | |
| 2023-04-21 23:17:48,791 DEBUG HandlerThread:22944 [handler.py:handle_request():144] handle_request: status_report | |
| 2023-04-21 23:17:50,091 ERROR gpu :22944 [interfaces.py:monitor():144] Failed to sample metric: Not Supported | |
| 2023-04-21 23:17:52,177 ERROR gpu :22944 [interfaces.py:monitor():144] Failed to sample metric: Not Supported | |
| 2023-04-21 23:17:53,339 DEBUG HandlerThread:22944 [handler.py:handle_request():144] handle_request: stop_status | |
| 2023-04-21 23:17:53,340 DEBUG SenderThread:22944 [sender.py:send_request():402] send_request: stop_status | |
| 2023-04-21 23:17:54,200 ERROR gpu :22944 [interfaces.py:monitor():144] Failed to sample metric: Not Supported | |
| 2023-04-21 23:17:54,613 DEBUG HandlerThread:22944 [handler.py:handle_request():144] handle_request: status_report | |
| 2023-04-21 23:17:56,221 ERROR gpu :22944 [interfaces.py:monitor():144] Failed to sample metric: Not Supported | |
| 2023-04-21 23:17:58,245 ERROR gpu :22944 [interfaces.py:monitor():144] Failed to sample metric: Not Supported | |
| 2023-04-21 23:17:59,655 DEBUG HandlerThread:22944 [handler.py:handle_request():144] handle_request: status_report | |
| 2023-04-21 23:18:00,316 ERROR gpu :22944 [interfaces.py:monitor():144] Failed to sample metric: Not Supported | |
| 2023-04-21 23:18:01,307 INFO Thread-16 :22944 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230421_230706-1i2p1xzz\files\output.log | |
| 2023-04-21 23:18:02,356 ERROR gpu :22944 [interfaces.py:monitor():144] Failed to sample metric: Not Supported | |
| 2023-04-21 23:18:04,405 ERROR gpu :22944 [interfaces.py:monitor():144] Failed to sample metric: Not Supported | |
| 2023-04-21 23:18:04,818 DEBUG HandlerThread:22944 [handler.py:handle_request():144] handle_request: status_report | |
| 2023-04-21 23:18:06,450 ERROR gpu :22944 [interfaces.py:monitor():144] Failed to sample metric: Not Supported | |
| 2023-04-21 23:18:07,906 DEBUG SenderThread:22944 [sender.py:send():375] send: stats | |
| 2023-04-21 23:18:08,354 DEBUG HandlerThread:22944 [handler.py:handle_request():144] handle_request: stop_status | |
| 2023-04-21 23:18:08,355 DEBUG SenderThread:22944 [sender.py:send_request():402] send_request: stop_status | |
| 2023-04-21 23:18:08,494 ERROR gpu :22944 [interfaces.py:monitor():144] Failed to sample metric: Not Supported | |
| 2023-04-21 23:18:10,530 ERROR gpu :22944 [interfaces.py:monitor():144] Failed to sample metric: Not Supported | |
| 2023-04-21 23:18:10,646 DEBUG HandlerThread:22944 [handler.py:handle_request():144] handle_request: status_report | |
| 2023-04-21 23:18:12,576 ERROR gpu :22944 [interfaces.py:monitor():144] Failed to sample metric: Not Supported | |
| 2023-04-21 23:18:13,903 DEBUG HandlerThread:22944 [handler.py:handle_request():144] handle_request: partial_history | |
| 2023-04-21 23:18:13,904 DEBUG SenderThread:22944 [sender.py:send():375] send: history | |
| 2023-04-21 23:18:13,905 DEBUG SenderThread:22944 [sender.py:send_request():402] send_request: summary_record | |
| 2023-04-21 23:18:13,906 INFO SenderThread:22944 [sender.py:_save_file():1378] saving file wandb-summary.json with policy end | |
| 2023-04-21 23:18:14,620 INFO Thread-16 :22944 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230421_230706-1i2p1xzz\files\output.log | |
| 2023-04-21 23:18:14,620 INFO Thread-16 :22944 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230421_230706-1i2p1xzz\files\wandb-summary.json | |
| 2023-04-21 23:18:14,621 ERROR gpu :22944 [interfaces.py:monitor():144] Failed to sample metric: Not Supported | |
| 2023-04-21 23:18:15,634 INFO Thread-16 :22944 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230421_230706-1i2p1xzz\files\output.log | |
| 2023-04-21 23:18:16,145 DEBUG HandlerThread:22944 [handler.py:handle_request():144] handle_request: status_report | |
| 2023-04-21 23:18:16,672 ERROR gpu :22944 [interfaces.py:monitor():144] Failed to sample metric: Not Supported | |
| 2023-04-21 23:18:18,720 ERROR gpu :22944 [interfaces.py:monitor():144] Failed to sample metric: Not Supported | |
| 2023-04-21 23:18:20,758 ERROR gpu :22944 [interfaces.py:monitor():144] Failed to sample metric: Not Supported | |
| 2023-04-21 23:18:21,200 DEBUG HandlerThread:22944 [handler.py:handle_request():144] handle_request: status_report | |
| 2023-04-21 23:18:22,862 ERROR gpu :22944 [interfaces.py:monitor():144] Failed to sample metric: Not Supported | |
| 2023-04-21 23:18:23,371 DEBUG HandlerThread:22944 [handler.py:handle_request():144] handle_request: stop_status | |
| 2023-04-21 23:18:23,372 DEBUG SenderThread:22944 [sender.py:send_request():402] send_request: stop_status | |
| 2023-04-21 23:18:24,881 ERROR gpu :22944 [interfaces.py:monitor():144] Failed to sample metric: Not Supported | |
| 2023-04-21 23:18:26,676 DEBUG HandlerThread:22944 [handler.py:handle_request():144] handle_request: status_report | |
| 2023-04-21 23:18:26,904 ERROR gpu :22944 [interfaces.py:monitor():144] Failed to sample metric: Not Supported | |
| 2023-04-21 23:18:28,948 ERROR gpu :22944 [interfaces.py:monitor():144] Failed to sample metric: Not Supported | |
| 2023-04-21 23:18:29,953 INFO Thread-16 :22944 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230421_230706-1i2p1xzz\files\output.log | |
| 2023-04-21 23:18:30,989 ERROR gpu :22944 [interfaces.py:monitor():144] Failed to sample metric: Not Supported | |
| 2023-04-21 23:18:32,440 DEBUG HandlerThread:22944 [handler.py:handle_request():144] handle_request: status_report | |
| 2023-04-21 23:18:33,032 ERROR gpu :22944 [interfaces.py:monitor():144] Failed to sample metric: Not Supported | |
| 2023-04-21 23:18:35,075 ERROR gpu :22944 [interfaces.py:monitor():144] Failed to sample metric: Not Supported | |
| 2023-04-21 23:18:37,108 ERROR gpu :22944 [interfaces.py:monitor():144] Failed to sample metric: Not Supported | |
| 2023-04-21 23:18:37,487 DEBUG HandlerThread:22944 [handler.py:handle_request():144] handle_request: status_report | |
| 2023-04-21 23:18:37,909 DEBUG SenderThread:22944 [sender.py:send():375] send: stats | |
| 2023-04-21 23:18:38,387 DEBUG HandlerThread:22944 [handler.py:handle_request():144] handle_request: stop_status | |
| 2023-04-21 23:18:38,387 DEBUG SenderThread:22944 [sender.py:send_request():402] send_request: stop_status | |
| 2023-04-21 23:18:39,155 ERROR gpu :22944 [interfaces.py:monitor():144] Failed to sample metric: Not Supported | |
| 2023-04-21 23:18:41,194 ERROR gpu :22944 [interfaces.py:monitor():144] Failed to sample metric: Not Supported | |
| 2023-04-21 23:18:43,236 ERROR gpu :22944 [interfaces.py:monitor():144] Failed to sample metric: Not Supported | |
| 2023-04-21 23:18:43,244 DEBUG HandlerThread:22944 [handler.py:handle_request():144] handle_request: status_report | |
| 2023-04-21 23:18:44,235 INFO Thread-16 :22944 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230421_230706-1i2p1xzz\files\output.log | |
| 2023-04-21 23:18:45,278 ERROR gpu :22944 [interfaces.py:monitor():144] Failed to sample metric: Not Supported | |
| 2023-04-21 23:18:47,329 ERROR gpu :22944 [interfaces.py:monitor():144] Failed to sample metric: Not Supported | |
| 2023-04-21 23:18:48,279 DEBUG HandlerThread:22944 [handler.py:handle_request():144] handle_request: status_report | |
| 2023-04-21 23:18:49,371 ERROR gpu :22944 [interfaces.py:monitor():144] Failed to sample metric: Not Supported | |
| 2023-04-21 23:18:51,407 ERROR gpu :22944 [interfaces.py:monitor():144] Failed to sample metric: Not Supported | |
| 2023-04-21 23:18:53,315 DEBUG HandlerThread:22944 [handler.py:handle_request():144] handle_request: status_report | |
| 2023-04-21 23:18:53,452 DEBUG HandlerThread:22944 [handler.py:handle_request():144] handle_request: stop_status | |
| 2023-04-21 23:18:53,453 DEBUG SenderThread:22944 [sender.py:send_request():402] send_request: stop_status | |
| 2023-04-21 23:18:53,500 ERROR gpu :22944 [interfaces.py:monitor():144] Failed to sample metric: Not Supported | |
| 2023-04-21 23:18:55,526 ERROR gpu :22944 [interfaces.py:monitor():144] Failed to sample metric: Not Supported | |
| 2023-04-21 23:18:57,541 INFO Thread-16 :22944 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230421_230706-1i2p1xzz\files\output.log | |
| 2023-04-21 23:18:57,548 ERROR gpu :22944 [interfaces.py:monitor():144] Failed to sample metric: Not Supported | |
| 2023-04-21 23:18:59,228 DEBUG HandlerThread:22944 [handler.py:handle_request():144] handle_request: status_report | |
| 2023-04-21 23:18:59,609 ERROR gpu :22944 [interfaces.py:monitor():144] Failed to sample metric: Not Supported | |
| 2023-04-21 23:19:01,665 ERROR gpu :22944 [interfaces.py:monitor():144] Failed to sample metric: Not Supported | |
| 2023-04-21 23:19:03,701 ERROR gpu :22944 [interfaces.py:monitor():144] Failed to sample metric: Not Supported | |
| 2023-04-21 23:19:04,272 DEBUG HandlerThread:22944 [handler.py:handle_request():144] handle_request: status_report | |
| 2023-04-21 23:19:05,743 ERROR gpu :22944 [interfaces.py:monitor():144] Failed to sample metric: Not Supported | |
| 2023-04-21 23:19:07,792 ERROR gpu :22944 [interfaces.py:monitor():144] Failed to sample metric: Not Supported | |
| 2023-04-21 23:19:07,917 DEBUG SenderThread:22944 [sender.py:send():375] send: stats | |
| 2023-04-21 23:19:08,416 DEBUG HandlerThread:22944 [handler.py:handle_request():144] handle_request: stop_status | |
| 2023-04-21 23:19:08,416 DEBUG SenderThread:22944 [sender.py:send_request():402] send_request: stop_status | |
| 2023-04-21 23:19:09,688 DEBUG HandlerThread:22944 [handler.py:handle_request():144] handle_request: status_report | |
| 2023-04-21 23:19:09,846 ERROR gpu :22944 [interfaces.py:monitor():144] Failed to sample metric: Not Supported | |
| 2023-04-21 23:19:11,882 INFO Thread-16 :22944 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230421_230706-1i2p1xzz\files\output.log | |
| 2023-04-21 23:19:11,891 ERROR gpu :22944 [interfaces.py:monitor():144] Failed to sample metric: Not Supported | |
| 2023-04-21 23:19:13,932 ERROR gpu :22944 [interfaces.py:monitor():144] Failed to sample metric: Not Supported | |
| 2023-04-21 23:19:15,188 DEBUG HandlerThread:22944 [handler.py:handle_request():144] handle_request: status_report | |
| 2023-04-21 23:19:15,976 ERROR gpu :22944 [interfaces.py:monitor():144] Failed to sample metric: Not Supported | |
| 2023-04-21 23:19:18,026 ERROR gpu :22944 [interfaces.py:monitor():144] Failed to sample metric: Not Supported | |
| 2023-04-21 23:19:20,071 ERROR gpu :22944 [interfaces.py:monitor():144] Failed to sample metric: Not Supported | |
| 2023-04-21 23:19:20,227 DEBUG HandlerThread:22944 [handler.py:handle_request():144] handle_request: status_report | |
| 2023-04-21 23:19:22,105 ERROR gpu :22944 [interfaces.py:monitor():144] Failed to sample metric: Not Supported | |
| 2023-04-21 23:19:23,437 DEBUG HandlerThread:22944 [handler.py:handle_request():144] handle_request: stop_status | |
| 2023-04-21 23:19:23,437 DEBUG SenderThread:22944 [sender.py:send_request():402] send_request: stop_status | |
| 2023-04-21 23:19:24,197 ERROR gpu :22944 [interfaces.py:monitor():144] Failed to sample metric: Not Supported | |
| 2023-04-21 23:19:26,011 DEBUG HandlerThread:22944 [handler.py:handle_request():144] handle_request: status_report | |
| 2023-04-21 23:19:26,172 INFO Thread-16 :22944 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230421_230706-1i2p1xzz\files\output.log | |
| 2023-04-21 23:19:26,210 ERROR gpu :22944 [interfaces.py:monitor():144] Failed to sample metric: Not Supported | |
| 2023-04-21 23:19:28,234 ERROR gpu :22944 [interfaces.py:monitor():144] Failed to sample metric: Not Supported | |
| 2023-04-21 23:19:30,272 ERROR gpu :22944 [interfaces.py:monitor():144] Failed to sample metric: Not Supported | |
| 2023-04-21 23:19:31,059 DEBUG HandlerThread:22944 [handler.py:handle_request():144] handle_request: status_report | |
| 2023-04-21 23:19:32,311 ERROR gpu :22944 [interfaces.py:monitor():144] Failed to sample metric: Not Supported | |
| 2023-04-21 23:19:34,356 ERROR gpu :22944 [interfaces.py:monitor():144] Failed to sample metric: Not Supported | |
| 2023-04-21 23:19:36,097 DEBUG HandlerThread:22944 [handler.py:handle_request():144] handle_request: status_report | |
| 2023-04-21 23:19:36,401 ERROR gpu :22944 [interfaces.py:monitor():144] Failed to sample metric: Not Supported | |
| 2023-04-21 23:19:37,927 DEBUG SenderThread:22944 [sender.py:send():375] send: stats | |
| 2023-04-21 23:19:38,452 DEBUG HandlerThread:22944 [handler.py:handle_request():144] handle_request: stop_status | |
| 2023-04-21 23:19:38,453 DEBUG SenderThread:22944 [sender.py:send_request():402] send_request: stop_status | |
| 2023-04-21 23:19:38,461 ERROR gpu :22944 [interfaces.py:monitor():144] Failed to sample metric: Not Supported | |
| 2023-04-21 23:19:40,493 INFO Thread-16 :22944 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230421_230706-1i2p1xzz\files\output.log | |
| 2023-04-21 23:19:40,504 ERROR gpu :22944 [interfaces.py:monitor():144] Failed to sample metric: Not Supported | |
| 2023-04-21 23:19:41,715 DEBUG HandlerThread:22944 [handler.py:handle_request():144] handle_request: status_report | |
| 2023-04-21 23:19:42,539 ERROR gpu :22944 [interfaces.py:monitor():144] Failed to sample metric: Not Supported | |
| 2023-04-21 23:19:44,587 ERROR gpu :22944 [interfaces.py:monitor():144] Failed to sample metric: Not Supported | |
| 2023-04-21 23:19:46,635 ERROR gpu :22944 [interfaces.py:monitor():144] Failed to sample metric: Not Supported | |
| 2023-04-21 23:19:46,761 DEBUG HandlerThread:22944 [handler.py:handle_request():144] handle_request: status_report | |
| 2023-04-21 23:19:48,674 ERROR gpu :22944 [interfaces.py:monitor():144] Failed to sample metric: Not Supported | |
| 2023-04-21 23:19:50,707 ERROR gpu :22944 [interfaces.py:monitor():144] Failed to sample metric: Not Supported | |
| 2023-04-21 23:19:51,783 DEBUG HandlerThread:22944 [handler.py:handle_request():144] handle_request: status_report | |
| 2023-04-21 23:19:52,749 ERROR gpu :22944 [interfaces.py:monitor():144] Failed to sample metric: Not Supported | |
| 2023-04-21 23:19:53,457 DEBUG HandlerThread:22944 [handler.py:handle_request():144] handle_request: stop_status | |
| 2023-04-21 23:19:53,458 DEBUG SenderThread:22944 [sender.py:send_request():402] send_request: stop_status | |
| 2023-04-21 23:19:53,753 INFO Thread-16 :22944 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230421_230706-1i2p1xzz\files\output.log | |
| 2023-04-21 23:19:54,861 ERROR gpu :22944 [interfaces.py:monitor():144] Failed to sample metric: Not Supported | |
| 2023-04-21 23:19:56,880 ERROR gpu :22944 [interfaces.py:monitor():144] Failed to sample metric: Not Supported | |
| 2023-04-21 23:19:57,767 DEBUG HandlerThread:22944 [handler.py:handle_request():144] handle_request: status_report | |
| 2023-04-21 23:19:58,900 ERROR gpu :22944 [interfaces.py:monitor():144] Failed to sample metric: Not Supported | |
| 2023-04-21 23:20:00,935 ERROR gpu :22944 [interfaces.py:monitor():144] Failed to sample metric: Not Supported | |
| 2023-04-21 23:20:02,811 DEBUG HandlerThread:22944 [handler.py:handle_request():144] handle_request: status_report | |
| 2023-04-21 23:20:03,000 ERROR gpu :22944 [interfaces.py:monitor():144] Failed to sample metric: Not Supported | |
| 2023-04-21 23:20:05,030 ERROR gpu :22944 [interfaces.py:monitor():144] Failed to sample metric: Not Supported | |
| 2023-04-21 23:20:07,066 ERROR gpu :22944 [interfaces.py:monitor():144] Failed to sample metric: Not Supported | |
| 2023-04-21 23:20:07,934 DEBUG SenderThread:22944 [sender.py:send():375] send: stats | |
| 2023-04-21 23:20:07,935 DEBUG HandlerThread:22944 [handler.py:handle_request():144] handle_request: status_report | |
| 2023-04-21 23:20:08,075 INFO Thread-16 :22944 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230421_230706-1i2p1xzz\files\output.log | |
| 2023-04-21 23:20:08,466 DEBUG HandlerThread:22944 [handler.py:handle_request():144] handle_request: stop_status | |
| 2023-04-21 23:20:08,467 DEBUG SenderThread:22944 [sender.py:send_request():402] send_request: stop_status | |
| 2023-04-21 23:20:09,107 ERROR gpu :22944 [interfaces.py:monitor():144] Failed to sample metric: Not Supported | |
| 2023-04-21 23:20:11,145 ERROR gpu :22944 [interfaces.py:monitor():144] Failed to sample metric: Not Supported | |
| 2023-04-21 23:20:13,181 ERROR gpu :22944 [interfaces.py:monitor():144] Failed to sample metric: Not Supported | |
| 2023-04-21 23:20:13,731 DEBUG HandlerThread:22944 [handler.py:handle_request():144] handle_request: status_report | |
| 2023-04-21 23:20:15,223 ERROR gpu :22944 [interfaces.py:monitor():144] Failed to sample metric: Not Supported | |
| 2023-04-21 23:20:17,266 ERROR gpu :22944 [interfaces.py:monitor():144] Failed to sample metric: Not Supported | |
| 2023-04-21 23:20:18,759 DEBUG HandlerThread:22944 [handler.py:handle_request():144] handle_request: status_report | |
| 2023-04-21 23:20:19,313 ERROR gpu :22944 [interfaces.py:monitor():144] Failed to sample metric: Not Supported | |
| 2023-04-21 23:20:21,366 ERROR gpu :22944 [interfaces.py:monitor():144] Failed to sample metric: Not Supported | |
| 2023-04-21 23:20:22,372 INFO Thread-16 :22944 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230421_230706-1i2p1xzz\files\output.log | |
| 2023-04-21 23:20:23,417 ERROR gpu :22944 [interfaces.py:monitor():144] Failed to sample metric: Not Supported | |
| 2023-04-21 23:20:23,480 DEBUG HandlerThread:22944 [handler.py:handle_request():144] handle_request: stop_status | |
| 2023-04-21 23:20:23,481 DEBUG SenderThread:22944 [sender.py:send_request():402] send_request: stop_status | |
| 2023-04-21 23:20:24,733 DEBUG HandlerThread:22944 [handler.py:handle_request():144] handle_request: status_report | |
| 2023-04-21 23:20:25,525 ERROR gpu :22944 [interfaces.py:monitor():144] Failed to sample metric: Not Supported | |
| 2023-04-21 23:20:27,548 ERROR gpu :22944 [interfaces.py:monitor():144] Failed to sample metric: Not Supported | |
| 2023-04-21 23:20:29,562 ERROR gpu :22944 [interfaces.py:monitor():144] Failed to sample metric: Not Supported | |
| 2023-04-21 23:20:29,774 DEBUG HandlerThread:22944 [handler.py:handle_request():144] handle_request: status_report | |
| 2023-04-21 23:20:31,598 ERROR gpu :22944 [interfaces.py:monitor():144] Failed to sample metric: Not Supported | |
| 2023-04-21 23:20:33,641 ERROR gpu :22944 [interfaces.py:monitor():144] Failed to sample metric: Not Supported | |
| 2023-04-21 23:20:34,809 DEBUG HandlerThread:22944 [handler.py:handle_request():144] handle_request: status_report | |
| 2023-04-21 23:20:35,696 ERROR gpu :22944 [interfaces.py:monitor():144] Failed to sample metric: Not Supported | |
| 2023-04-21 23:20:36,777 DEBUG HandlerThread:22944 [handler.py:handle_request():144] handle_request: partial_history | |
| 2023-04-21 23:20:36,778 DEBUG SenderThread:22944 [sender.py:send():375] send: history | |
| 2023-04-21 23:20:36,778 DEBUG SenderThread:22944 [sender.py:send_request():402] send_request: summary_record | |
| 2023-04-21 23:20:36,779 INFO SenderThread:22944 [sender.py:_save_file():1378] saving file wandb-summary.json with policy end | |
| 2023-04-21 23:20:37,728 INFO Thread-16 :22944 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230421_230706-1i2p1xzz\files\wandb-summary.json | |
| 2023-04-21 23:20:37,740 ERROR gpu :22944 [interfaces.py:monitor():144] Failed to sample metric: Not Supported | |
| 2023-04-21 23:20:37,934 DEBUG SenderThread:22944 [sender.py:send():375] send: stats | |
| 2023-04-21 23:20:38,480 DEBUG HandlerThread:22944 [handler.py:handle_request():144] handle_request: stop_status | |
| 2023-04-21 23:20:38,480 DEBUG SenderThread:22944 [sender.py:send_request():402] send_request: stop_status | |
| 2023-04-21 23:20:38,730 INFO Thread-16 :22944 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230421_230706-1i2p1xzz\files\output.log | |
| 2023-04-21 23:20:39,779 INFO Thread-16 :22944 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230421_230706-1i2p1xzz\files\output.log | |
| 2023-04-21 23:20:39,797 ERROR gpu :22944 [interfaces.py:monitor():144] Failed to sample metric: Not Supported | |
| 2023-04-21 23:20:40,754 DEBUG HandlerThread:22944 [handler.py:handle_request():144] handle_request: status_report | |
| 2023-04-21 23:20:41,840 ERROR gpu :22944 [interfaces.py:monitor():144] Failed to sample metric: Not Supported | |
| 2023-04-21 23:20:43,883 ERROR gpu :22944 [interfaces.py:monitor():144] Failed to sample metric: Not Supported | |
| 2023-04-21 23:20:45,809 DEBUG HandlerThread:22944 [handler.py:handle_request():144] handle_request: status_report | |
| 2023-04-21 23:20:45,933 ERROR gpu :22944 [interfaces.py:monitor():144] Failed to sample metric: Not Supported | |
| 2023-04-21 23:20:47,984 ERROR gpu :22944 [interfaces.py:monitor():144] Failed to sample metric: Not Supported | |
| 2023-04-21 23:20:50,036 ERROR gpu :22944 [interfaces.py:monitor():144] Failed to sample metric: Not Supported | |
| 2023-04-21 23:20:50,858 DEBUG HandlerThread:22944 [handler.py:handle_request():144] handle_request: status_report | |
| 2023-04-21 23:20:52,082 ERROR gpu :22944 [interfaces.py:monitor():144] Failed to sample metric: Not Supported | |
| 2023-04-21 23:20:53,493 DEBUG HandlerThread:22944 [handler.py:handle_request():144] handle_request: stop_status | |
| 2023-04-21 23:20:53,494 DEBUG SenderThread:22944 [sender.py:send_request():402] send_request: stop_status | |
| 2023-04-21 23:20:54,130 ERROR gpu :22944 [interfaces.py:monitor():144] Failed to sample metric: Not Supported | |
| 2023-04-21 23:20:56,255 ERROR gpu :22944 [interfaces.py:monitor():144] Failed to sample metric: Not Supported | |
| 2023-04-21 23:20:56,772 DEBUG HandlerThread:22944 [handler.py:handle_request():144] handle_request: status_report | |
| 2023-04-21 23:20:58,277 ERROR gpu :22944 [interfaces.py:monitor():144] Failed to sample metric: Not Supported | |
| 2023-04-21 23:21:00,302 ERROR gpu :22944 [interfaces.py:monitor():144] Failed to sample metric: Not Supported | |
| 2023-04-21 23:21:01,818 DEBUG HandlerThread:22944 [handler.py:handle_request():144] handle_request: status_report | |
| 2023-04-21 23:21:02,327 ERROR gpu :22944 [interfaces.py:monitor():144] Failed to sample metric: Not Supported | |
| 2023-04-21 23:21:04,353 INFO Thread-16 :22944 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230421_230706-1i2p1xzz\files\output.log | |
| 2023-04-21 23:21:04,365 ERROR gpu :22944 [interfaces.py:monitor():144] Failed to sample metric: Not Supported | |
| 2023-04-21 23:21:06,404 ERROR gpu :22944 [interfaces.py:monitor():144] Failed to sample metric: Not Supported | |
| 2023-04-21 23:21:07,176 DEBUG HandlerThread:22944 [handler.py:handle_request():144] handle_request: status_report | |
| 2023-04-21 23:21:07,940 DEBUG SenderThread:22944 [sender.py:send():375] send: stats | |
| 2023-04-21 23:21:08,448 ERROR gpu :22944 [interfaces.py:monitor():144] Failed to sample metric: Not Supported | |
| 2023-04-21 23:21:08,498 DEBUG HandlerThread:22944 [handler.py:handle_request():144] handle_request: stop_status | |
| 2023-04-21 23:21:08,498 DEBUG SenderThread:22944 [sender.py:send_request():402] send_request: stop_status | |
| 2023-04-21 23:21:10,491 ERROR gpu :22944 [interfaces.py:monitor():144] Failed to sample metric: Not Supported | |
| 2023-04-21 23:21:12,528 ERROR gpu :22944 [interfaces.py:monitor():144] Failed to sample metric: Not Supported | |
| 2023-04-21 23:21:12,789 DEBUG HandlerThread:22944 [handler.py:handle_request():144] handle_request: status_report | |
| 2023-04-21 23:21:14,571 ERROR gpu :22944 [interfaces.py:monitor():144] Failed to sample metric: Not Supported | |
| 2023-04-21 23:21:16,615 ERROR gpu :22944 [interfaces.py:monitor():144] Failed to sample metric: Not Supported | |
| 2023-04-21 23:21:17,831 DEBUG HandlerThread:22944 [handler.py:handle_request():144] handle_request: status_report | |
| 2023-04-21 23:21:18,657 ERROR gpu :22944 [interfaces.py:monitor():144] Failed to sample metric: Not Supported | |
| 2023-04-21 23:21:20,701 ERROR gpu :22944 [interfaces.py:monitor():144] Failed to sample metric: Not Supported | |
| 2023-04-21 23:21:21,785 DEBUG SenderThread:22944 [sender.py:send():375] send: exit | |
| 2023-04-21 23:21:21,785 INFO SenderThread:22944 [sender.py:send_exit():598] handling exit code: 255 | |
| 2023-04-21 23:21:21,785 INFO SenderThread:22944 [sender.py:send_exit():600] handling runtime: 854 | |
| 2023-04-21 23:21:21,786 INFO SenderThread:22944 [sender.py:_save_file():1378] saving file wandb-summary.json with policy end | |
| 2023-04-21 23:21:21,786 INFO SenderThread:22944 [sender.py:send_exit():606] send defer | |
| 2023-04-21 23:21:21,787 DEBUG HandlerThread:22944 [handler.py:handle_request():144] handle_request: defer | |
| 2023-04-21 23:21:21,787 INFO HandlerThread:22944 [handler.py:handle_request_defer():170] handle defer: 0 | |
| 2023-04-21 23:21:21,787 DEBUG SenderThread:22944 [sender.py:send_request():402] send_request: defer | |
| 2023-04-21 23:21:21,787 INFO SenderThread:22944 [sender.py:send_request_defer():622] handle sender defer: 0 | |
| 2023-04-21 23:21:21,787 INFO SenderThread:22944 [sender.py:transition_state():626] send defer: 1 | |
| 2023-04-21 23:21:21,787 DEBUG HandlerThread:22944 [handler.py:handle_request():144] handle_request: defer | |
| 2023-04-21 23:21:21,787 INFO HandlerThread:22944 [handler.py:handle_request_defer():170] handle defer: 1 | |
| 2023-04-21 23:21:21,788 DEBUG SenderThread:22944 [sender.py:send_request():402] send_request: defer | |
| 2023-04-21 23:21:21,788 INFO SenderThread:22944 [sender.py:send_request_defer():622] handle sender defer: 1 | |
| 2023-04-21 23:21:21,788 INFO SenderThread:22944 [sender.py:transition_state():626] send defer: 2 | |
| 2023-04-21 23:21:21,788 DEBUG HandlerThread:22944 [handler.py:handle_request():144] handle_request: defer | |
| 2023-04-21 23:21:21,788 INFO HandlerThread:22944 [handler.py:handle_request_defer():170] handle defer: 2 | |
| 2023-04-21 23:21:21,788 INFO HandlerThread:22944 [system_monitor.py:finish():190] Stopping system monitor | |
| 2023-04-21 23:21:21,788 DEBUG SystemMonitor:22944 [system_monitor.py:_start():166] Finished system metrics aggregation loop | |
| 2023-04-21 23:21:21,788 INFO HandlerThread:22944 [interfaces.py:finish():202] Joined cpu monitor | |
| 2023-04-21 23:21:21,789 DEBUG SystemMonitor:22944 [system_monitor.py:_start():170] Publishing last batch of metrics | |
| 2023-04-21 23:21:21,799 INFO HandlerThread:22944 [interfaces.py:finish():202] Joined disk monitor | |
| 2023-04-21 23:21:21,841 INFO HandlerThread:22944 [interfaces.py:finish():202] Joined gpu monitor | |
| 2023-04-21 23:21:21,841 INFO HandlerThread:22944 [interfaces.py:finish():202] Joined memory monitor | |
| 2023-04-21 23:21:21,841 INFO HandlerThread:22944 [interfaces.py:finish():202] Joined network monitor | |
| 2023-04-21 23:21:21,842 DEBUG SenderThread:22944 [sender.py:send_request():402] send_request: defer | |
| 2023-04-21 23:21:21,842 INFO SenderThread:22944 [sender.py:send_request_defer():622] handle sender defer: 2 | |
| 2023-04-21 23:21:21,842 INFO SenderThread:22944 [sender.py:transition_state():626] send defer: 3 | |
| 2023-04-21 23:21:21,842 DEBUG SenderThread:22944 [sender.py:send():375] send: stats | |
| 2023-04-21 23:21:21,842 DEBUG HandlerThread:22944 [handler.py:handle_request():144] handle_request: defer | |
| 2023-04-21 23:21:21,842 INFO HandlerThread:22944 [handler.py:handle_request_defer():170] handle defer: 3 | |
| 2023-04-21 23:21:21,843 DEBUG SenderThread:22944 [sender.py:send_request():402] send_request: defer | |
| 2023-04-21 23:21:21,843 INFO SenderThread:22944 [sender.py:send_request_defer():622] handle sender defer: 3 | |
| 2023-04-21 23:21:21,843 INFO SenderThread:22944 [sender.py:transition_state():626] send defer: 4 | |
| 2023-04-21 23:21:21,843 DEBUG HandlerThread:22944 [handler.py:handle_request():144] handle_request: defer | |
| 2023-04-21 23:21:21,843 INFO HandlerThread:22944 [handler.py:handle_request_defer():170] handle defer: 4 | |
| 2023-04-21 23:21:21,843 DEBUG SenderThread:22944 [sender.py:send_request():402] send_request: defer | |
| 2023-04-21 23:21:21,843 INFO SenderThread:22944 [sender.py:send_request_defer():622] handle sender defer: 4 | |
| 2023-04-21 23:21:21,844 INFO SenderThread:22944 [sender.py:transition_state():626] send defer: 5 | |
| 2023-04-21 23:21:21,844 DEBUG HandlerThread:22944 [handler.py:handle_request():144] handle_request: defer | |
| 2023-04-21 23:21:21,844 INFO HandlerThread:22944 [handler.py:handle_request_defer():170] handle defer: 5 | |
| 2023-04-21 23:21:21,844 DEBUG SenderThread:22944 [sender.py:send():375] send: summary | |
| 2023-04-21 23:21:21,845 INFO SenderThread:22944 [sender.py:_save_file():1378] saving file wandb-summary.json with policy end | |
| 2023-04-21 23:21:21,846 DEBUG SenderThread:22944 [sender.py:send_request():402] send_request: defer | |
| 2023-04-21 23:21:21,846 INFO SenderThread:22944 [sender.py:send_request_defer():622] handle sender defer: 5 | |
| 2023-04-21 23:21:21,846 INFO SenderThread:22944 [sender.py:transition_state():626] send defer: 6 | |
| 2023-04-21 23:21:21,846 DEBUG HandlerThread:22944 [handler.py:handle_request():144] handle_request: defer | |
| 2023-04-21 23:21:21,846 INFO HandlerThread:22944 [handler.py:handle_request_defer():170] handle defer: 6 | |
| 2023-04-21 23:21:21,846 DEBUG SenderThread:22944 [sender.py:send_request():402] send_request: defer | |
| 2023-04-21 23:21:21,846 INFO SenderThread:22944 [sender.py:send_request_defer():622] handle sender defer: 6 | |
| 2023-04-21 23:21:21,846 INFO SenderThread:22944 [sender.py:transition_state():626] send defer: 7 | |
| 2023-04-21 23:21:21,846 DEBUG HandlerThread:22944 [handler.py:handle_request():144] handle_request: status_report | |
| 2023-04-21 23:21:21,847 DEBUG HandlerThread:22944 [handler.py:handle_request():144] handle_request: defer | |
| 2023-04-21 23:21:21,847 INFO HandlerThread:22944 [handler.py:handle_request_defer():170] handle defer: 7 | |
| 2023-04-21 23:21:21,847 DEBUG SenderThread:22944 [sender.py:send_request():402] send_request: defer | |
| 2023-04-21 23:21:21,848 INFO SenderThread:22944 [sender.py:send_request_defer():622] handle sender defer: 7 | |
| 2023-04-21 23:21:22,699 INFO Thread-16 :22944 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230421_230706-1i2p1xzz\files\output.log | |
| 2023-04-21 23:21:22,699 INFO Thread-16 :22944 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230421_230706-1i2p1xzz\files\wandb-summary.json | |
| 2023-04-21 23:21:22,850 DEBUG HandlerThread:22944 [handler.py:handle_request():144] handle_request: poll_exit | |
| 2023-04-21 23:21:24,718 INFO Thread-16 :22944 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230421_230706-1i2p1xzz\files\output.log | |
| 2023-04-21 23:21:25,028 INFO SenderThread:22944 [sender.py:transition_state():626] send defer: 8 | |
| 2023-04-21 23:21:25,028 DEBUG SenderThread:22944 [sender.py:send_request():402] send_request: poll_exit | |
| 2023-04-21 23:21:25,028 DEBUG HandlerThread:22944 [handler.py:handle_request():144] handle_request: defer | |
| 2023-04-21 23:21:25,028 INFO HandlerThread:22944 [handler.py:handle_request_defer():170] handle defer: 8 | |
| 2023-04-21 23:21:25,029 DEBUG SenderThread:22944 [sender.py:send_request():402] send_request: defer | |
| 2023-04-21 23:21:25,029 INFO SenderThread:22944 [sender.py:send_request_defer():622] handle sender defer: 8 | |
| 2023-04-21 23:21:25,061 INFO SenderThread:22944 [sender.py:transition_state():626] send defer: 9 | |
| 2023-04-21 23:21:25,063 DEBUG HandlerThread:22944 [handler.py:handle_request():144] handle_request: defer | |
| 2023-04-21 23:21:25,063 DEBUG SenderThread:22944 [sender.py:send():375] send: artifact | |
| 2023-04-21 23:21:25,063 INFO HandlerThread:22944 [handler.py:handle_request_defer():170] handle defer: 9 | |
| 2023-04-21 23:21:25,733 INFO Thread-16 :22944 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230421_230706-1i2p1xzz\files\output.log | |
| 2023-04-21 23:21:26,441 INFO SenderThread:22944 [sender.py:send_artifact():1474] sent artifact job-https___github.com_THUDM_ChatGLM-6B_ptuning_main.py - {'id': 'QXJ0aWZhY3Q6NDI5OTE5NTc5', 'digest': 'ab8fa958b17d8260d4017e4943372340', 'state': 'COMMITTED', 'aliases': [{'artifactCollectionName': 'job-https___github.com_THUDM_ChatGLM-6B_ptuning_main.py', 'alias': 'latest'}, {'artifactCollectionName': 'job-https___github.com_THUDM_ChatGLM-6B_ptuning_main.py', 'alias': 'v5'}], 'artifactSequence': {'id': 'QXJ0aWZhY3RDb2xsZWN0aW9uOjYzNDcyMzc3', 'latestArtifact': {'id': 'QXJ0aWZhY3Q6NDI5OTE5NTc5', 'versionIndex': 5}}, 'version': 'v5'} | |
| 2023-04-21 23:21:26,442 DEBUG SenderThread:22944 [sender.py:send_request():402] send_request: defer | |
| 2023-04-21 23:21:26,442 INFO SenderThread:22944 [sender.py:send_request_defer():622] handle sender defer: 9 | |
| 2023-04-21 23:21:26,442 INFO SenderThread:22944 [dir_watcher.py:finish():365] shutting down directory watcher | |
| 2023-04-21 23:21:26,748 INFO SenderThread:22944 [dir_watcher.py:finish():395] scan: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230421_230706-1i2p1xzz\files | |
| 2023-04-21 23:21:26,748 INFO SenderThread:22944 [dir_watcher.py:finish():409] scan save: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230421_230706-1i2p1xzz\files\config.yaml config.yaml | |
| 2023-04-21 23:21:26,749 INFO SenderThread:22944 [dir_watcher.py:finish():409] scan save: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230421_230706-1i2p1xzz\files\output.log output.log | |
| 2023-04-21 23:21:26,751 INFO SenderThread:22944 [dir_watcher.py:finish():409] scan save: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230421_230706-1i2p1xzz\files\requirements.txt requirements.txt | |
| 2023-04-21 23:21:26,753 INFO SenderThread:22944 [dir_watcher.py:finish():409] scan save: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230421_230706-1i2p1xzz\files\wandb-metadata.json wandb-metadata.json | |
| 2023-04-21 23:21:26,753 INFO SenderThread:22944 [dir_watcher.py:finish():409] scan save: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230421_230706-1i2p1xzz\files\wandb-summary.json wandb-summary.json | |
| 2023-04-21 23:21:26,756 INFO SenderThread:22944 [sender.py:transition_state():626] send defer: 10 | |
| 2023-04-21 23:21:26,758 DEBUG HandlerThread:22944 [handler.py:handle_request():144] handle_request: defer | |
| 2023-04-21 23:21:26,758 INFO HandlerThread:22944 [handler.py:handle_request_defer():170] handle defer: 10 | |
| 2023-04-21 23:21:26,760 DEBUG SenderThread:22944 [sender.py:send_request():402] send_request: defer | |
| 2023-04-21 23:21:26,760 INFO SenderThread:22944 [sender.py:send_request_defer():622] handle sender defer: 10 | |
| 2023-04-21 23:21:26,760 INFO SenderThread:22944 [file_pusher.py:finish():168] shutting down file pusher | |
| 2023-04-21 23:21:27,904 DEBUG HandlerThread:22944 [handler.py:handle_request():144] handle_request: keepalive | |
| 2023-04-21 23:21:30,784 DEBUG HandlerThread:22944 [handler.py:handle_request():144] handle_request: status_report | |
| 2023-04-21 23:21:32,157 INFO wandb-upload_3:22944 [upload_job.py:push():137] Uploaded file E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230421_230706-1i2p1xzz\files\wandb-summary.json | |
| 2023-04-21 23:21:32,827 INFO wandb-upload_0:22944 [upload_job.py:push():137] Uploaded file E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230421_230706-1i2p1xzz\files\config.yaml | |
| 2023-04-21 23:21:32,933 DEBUG HandlerThread:22944 [handler.py:handle_request():144] handle_request: keepalive | |
| 2023-04-21 23:21:33,023 INFO wandb-upload_2:22944 [upload_job.py:push():137] Uploaded file E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230421_230706-1i2p1xzz\files\requirements.txt | |
| 2023-04-21 23:21:34,949 INFO wandb-upload_1:22944 [upload_job.py:push():137] Uploaded file E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230421_230706-1i2p1xzz\files\output.log | |
| 2023-04-21 23:21:35,157 INFO Thread-15 :22944 [sender.py:transition_state():626] send defer: 11 | |
| 2023-04-21 23:21:35,157 DEBUG HandlerThread:22944 [handler.py:handle_request():144] handle_request: defer | |
| 2023-04-21 23:21:35,157 INFO HandlerThread:22944 [handler.py:handle_request_defer():170] handle defer: 11 | |
| 2023-04-21 23:21:35,158 DEBUG SenderThread:22944 [sender.py:send_request():402] send_request: defer | |
| 2023-04-21 23:21:35,158 INFO SenderThread:22944 [sender.py:send_request_defer():622] handle sender defer: 11 | |
| 2023-04-21 23:21:35,158 INFO SenderThread:22944 [file_pusher.py:join():173] waiting for file pusher | |
| 2023-04-21 23:21:35,158 INFO SenderThread:22944 [sender.py:transition_state():626] send defer: 12 | |
| 2023-04-21 23:21:35,158 DEBUG HandlerThread:22944 [handler.py:handle_request():144] handle_request: defer | |
| 2023-04-21 23:21:35,158 INFO HandlerThread:22944 [handler.py:handle_request_defer():170] handle defer: 12 | |
| 2023-04-21 23:21:35,159 DEBUG SenderThread:22944 [sender.py:send_request():402] send_request: defer | |
| 2023-04-21 23:21:35,159 INFO SenderThread:22944 [sender.py:send_request_defer():622] handle sender defer: 12 | |
| 2023-04-21 23:21:36,260 INFO SenderThread:22944 [sender.py:transition_state():626] send defer: 13 | |
| 2023-04-21 23:21:36,260 DEBUG HandlerThread:22944 [handler.py:handle_request():144] handle_request: defer | |
| 2023-04-21 23:21:36,260 INFO HandlerThread:22944 [handler.py:handle_request_defer():170] handle defer: 13 | |
| 2023-04-21 23:21:36,260 DEBUG HandlerThread:22944 [handler.py:handle_request():144] handle_request: status_report | |
| 2023-04-21 23:21:36,260 DEBUG SenderThread:22944 [sender.py:send_request():402] send_request: defer | |
| 2023-04-21 23:21:36,260 INFO SenderThread:22944 [sender.py:send_request_defer():622] handle sender defer: 13 | |
| 2023-04-21 23:21:36,260 INFO SenderThread:22944 [sender.py:transition_state():626] send defer: 14 | |
| 2023-04-21 23:21:36,260 DEBUG HandlerThread:22944 [handler.py:handle_request():144] handle_request: defer | |
| 2023-04-21 23:21:36,261 DEBUG SenderThread:22944 [sender.py:send():375] send: final | |
| 2023-04-21 23:21:36,261 INFO HandlerThread:22944 [handler.py:handle_request_defer():170] handle defer: 14 | |
| 2023-04-21 23:21:36,261 DEBUG SenderThread:22944 [sender.py:send():375] send: footer | |
| 2023-04-21 23:21:36,261 DEBUG SenderThread:22944 [sender.py:send_request():402] send_request: defer | |
| 2023-04-21 23:21:36,261 INFO SenderThread:22944 [sender.py:send_request_defer():622] handle sender defer: 14 | |
| 2023-04-21 23:21:36,261 DEBUG HandlerThread:22944 [handler.py:handle_request():144] handle_request: poll_exit | |
| 2023-04-21 23:21:36,262 DEBUG HandlerThread:22944 [handler.py:handle_request():144] handle_request: server_info | |
| 2023-04-21 23:21:36,262 DEBUG HandlerThread:22944 [handler.py:handle_request():144] handle_request: get_summary | |
| 2023-04-21 23:21:36,262 DEBUG SenderThread:22944 [sender.py:send_request():402] send_request: poll_exit | |
| 2023-04-21 23:21:36,262 DEBUG HandlerThread:22944 [handler.py:handle_request():144] handle_request: sampled_history | |
| 2023-04-21 23:21:36,262 DEBUG SenderThread:22944 [sender.py:send_request():402] send_request: server_info | |
| 2023-04-21 23:21:36,509 INFO MainThread:22944 [wandb_run.py:_footer_history_summary_info():3476] rendering history | |
| 2023-04-21 23:21:36,509 INFO MainThread:22944 [wandb_run.py:_footer_history_summary_info():3508] rendering summary | |
| 2023-04-21 23:21:36,512 INFO MainThread:22944 [wandb_run.py:_footer_sync_info():3434] logging synced files | |
| 2023-04-21 23:21:36,513 DEBUG HandlerThread:22944 [handler.py:handle_request():144] handle_request: shutdown | |
| 2023-04-21 23:21:36,513 INFO HandlerThread:22944 [handler.py:finish():845] shutting down handler | |
| 2023-04-21 23:21:37,278 INFO WriterThread:22944 [datastore.py:close():298] close: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230421_230706-1i2p1xzz\run-1i2p1xzz.wandb | |
| 2023-04-21 23:21:37,522 INFO SenderThread:22944 [sender.py:finish():1550] shutting down sender | |
| 2023-04-21 23:21:37,522 INFO SenderThread:22944 [file_pusher.py:finish():168] shutting down file pusher | |
| 2023-04-21 23:21:37,522 INFO SenderThread:22944 [file_pusher.py:join():173] waiting for file pusher | |