2023-04-21 03:58:43,953 INFO StreamThr :38724 [internal.py:wandb_internal():86] W&B internal server running at pid: 38724, started at: 2023-04-21 03:58:43.953038 2023-04-21 03:58:43,954 DEBUG HandlerThread:38724 [handler.py:handle_request():144] handle_request: status 2023-04-21 03:58:43,956 INFO WriterThread:38724 [datastore.py:open_for_write():85] open: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230421_035843-kyzl8vjo\run-kyzl8vjo.wandb 2023-04-21 03:58:43,960 DEBUG SenderThread:38724 [sender.py:send():375] send: header 2023-04-21 03:58:44,024 DEBUG SenderThread:38724 [sender.py:send():375] send: run 2023-04-21 03:58:44,776 INFO SenderThread:38724 [dir_watcher.py:__init__():219] watching files in: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230421_035843-kyzl8vjo\files 2023-04-21 03:58:44,776 INFO SenderThread:38724 [sender.py:_start_run_threads():1124] run started: kyzl8vjo with start time 1682020723.954036 2023-04-21 03:58:44,776 DEBUG SenderThread:38724 [sender.py:send_request():402] send_request: summary_record 2023-04-21 03:58:44,777 INFO SenderThread:38724 [sender.py:_save_file():1378] saving file wandb-summary.json with policy end 2023-04-21 03:58:44,778 DEBUG HandlerThread:38724 [handler.py:handle_request():144] handle_request: check_version 2023-04-21 03:58:44,779 DEBUG SenderThread:38724 [sender.py:send_request():402] send_request: check_version 2023-04-21 03:58:45,315 DEBUG HandlerThread:38724 [handler.py:handle_request():144] handle_request: run_start 2023-04-21 03:58:45,368 DEBUG HandlerThread:38724 [system_info.py:__init__():31] System info init 2023-04-21 03:58:45,368 DEBUG HandlerThread:38724 [system_info.py:__init__():46] System info init done 2023-04-21 03:58:45,368 INFO HandlerThread:38724 [system_monitor.py:start():181] Starting system monitor 2023-04-21 03:58:45,368 INFO SystemMonitor:38724 [system_monitor.py:_start():145] Starting system asset monitoring threads 2023-04-21 03:58:45,368 INFO HandlerThread:38724 [system_monitor.py:probe():201] Collecting system info 2023-04-21 03:58:45,376 INFO SystemMonitor:38724 [interfaces.py:start():190] Started cpu monitoring 2023-04-21 03:58:45,377 INFO SystemMonitor:38724 [interfaces.py:start():190] Started disk monitoring 2023-04-21 03:58:45,377 INFO SystemMonitor:38724 [interfaces.py:start():190] Started gpu monitoring 2023-04-21 03:58:45,387 INFO SystemMonitor:38724 [interfaces.py:start():190] Started memory monitoring 2023-04-21 03:58:45,418 INFO SystemMonitor:38724 [interfaces.py:start():190] Started network monitoring 2023-04-21 03:58:45,445 DEBUG HandlerThread:38724 [system_info.py:probe():195] Probing system 2023-04-21 03:58:45,448 DEBUG HandlerThread:38724 [system_info.py:_probe_git():180] Probing git 2023-04-21 03:58:45,449 ERROR gpu :38724 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-21 03:58:45,555 DEBUG HandlerThread:38724 [system_info.py:_probe_git():188] Probing git done 2023-04-21 03:58:45,555 DEBUG HandlerThread:38724 [system_info.py:probe():240] Probing system done 2023-04-21 03:58:45,555 DEBUG HandlerThread:38724 [system_monitor.py:probe():210] {'os': 'Windows-10-10.0.19041-SP0', 'python': '3.8.10', 'heartbeatAt': '2023-04-20T19:58:45.445809', 'startedAt': '2023-04-20T19:58:43.942038', 'docker': None, 'cuda': None, 'args': ('--do_train', '--train_file', '..\\AdvertiseGen\\train.json', '--validation_file', '..\\AdvertiseGen\\dev.json', '--prompt_column', 'content', '--response_column', 'summary', '--overwrite_cache', '--model_name_or_path', '..\\models\\chatglm-6b-int4-1\\models--THUDM--chatglm-6b-int4\\snapshots\\e02ba894cf18f3fd9b2526c795f983683c4ec732', '--output_dir', 'output\\adgen-chatglm-6b-pt-128-2e-2', '--overwrite_output_dir', '--max_source_length', '64', '--max_target_length', '64', '--per_device_train_batch_size', '1', '--per_device_eval_batch_size', '1', '--gradient_accumulation_steps', '16', '--predict_with_generate', '--max_steps', '10', '--logging_steps', '10', '--save_steps', '5', '--learning_rate', '2e-2', '--pre_seq_len', '128', '--quantization_bit', '4'), 'state': 'running', 'program': 'main.py', 'codePath': 'ptuning\\main.py', 'git': {'remote': 'https://github.com/THUDM/ChatGLM-6B', 'commit': '01e6313abf4122d789d6e68128856af52847b355'}, 'cpu_count': 6, 'cpu_count_logical': 12, 'cpu_freq': {'current': 2592.0, 'min': 0.0, 'max': 2592.0}, 'cpu_freq_per_core': [{'current': 2592.0, 'min': 0.0, 'max': 2592.0}], 'disk': {'total': 500.32030868530273, 'used': 483.4673271179199}, 'gpu': 'NVIDIA GeForce RTX 2060', 'gpu_count': 1, 'gpu_devices': [{'name': 'NVIDIA GeForce RTX 2060', 'memory_total': 6442450944}], 'memory': {'total': 63.87089538574219}} 2023-04-21 03:58:45,556 INFO HandlerThread:38724 [system_monitor.py:probe():211] Finished collecting system info 2023-04-21 03:58:45,556 INFO HandlerThread:38724 [system_monitor.py:probe():214] Publishing system info 2023-04-21 03:58:45,556 DEBUG HandlerThread:38724 [system_info.py:_save_pip():51] Saving list of pip packages installed into the current environment 2023-04-21 03:58:45,557 DEBUG HandlerThread:38724 [system_info.py:_save_pip():67] Saving pip packages done 2023-04-21 03:58:45,558 INFO HandlerThread:38724 [system_monitor.py:probe():216] Finished publishing system info 2023-04-21 03:58:45,571 DEBUG SenderThread:38724 [sender.py:send():375] send: files 2023-04-21 03:58:45,572 INFO SenderThread:38724 [sender.py:_save_file():1378] saving file wandb-metadata.json with policy now 2023-04-21 03:58:45,584 DEBUG HandlerThread:38724 [handler.py:handle_request():144] handle_request: stop_status 2023-04-21 03:58:45,584 DEBUG SenderThread:38724 [sender.py:send_request():402] send_request: stop_status 2023-04-21 03:58:45,788 INFO Thread-16 :38724 [dir_watcher.py:_on_file_created():278] file/dir created: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230421_035843-kyzl8vjo\files\requirements.txt 2023-04-21 03:58:45,788 INFO Thread-16 :38724 [dir_watcher.py:_on_file_created():278] file/dir created: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230421_035843-kyzl8vjo\files\wandb-summary.json 2023-04-21 03:58:45,789 INFO Thread-16 :38724 [dir_watcher.py:_on_file_created():278] file/dir created: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230421_035843-kyzl8vjo\files\wandb-metadata.json 2023-04-21 03:58:46,002 DEBUG SenderThread:38724 [sender.py:send():375] send: telemetry 2023-04-21 03:58:46,003 DEBUG SenderThread:38724 [sender.py:send():375] send: config 2023-04-21 03:58:46,004 DEBUG SenderThread:38724 [sender.py:send():375] send: metric 2023-04-21 03:58:46,004 DEBUG SenderThread:38724 [sender.py:send():375] send: telemetry 2023-04-21 03:58:46,004 DEBUG SenderThread:38724 [sender.py:send():375] send: metric 2023-04-21 03:58:46,004 WARNING SenderThread:38724 [sender.py:send_metric():1329] Seen metric with glob (shouldn't happen) 2023-04-21 03:58:46,617 INFO wandb-upload_0:38724 [upload_job.py:push():137] Uploaded file C:\Users\Lenovo\AppData\Local\Temp\tmpeu4l8b7twandb\d2ycakdr-wandb-metadata.json 2023-04-21 03:58:46,790 INFO Thread-16 :38724 [dir_watcher.py:_on_file_created():278] file/dir created: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230421_035843-kyzl8vjo\files\output.log 2023-04-21 03:58:47,503 ERROR gpu :38724 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-21 03:58:48,818 INFO Thread-16 :38724 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230421_035843-kyzl8vjo\files\output.log 2023-04-21 03:58:49,142 DEBUG HandlerThread:38724 [handler.py:handle_request():144] handle_request: status_report 2023-04-21 03:58:49,546 ERROR gpu :38724 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-21 03:58:49,829 INFO Thread-16 :38724 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230421_035843-kyzl8vjo\files\output.log 2023-04-21 03:58:51,601 ERROR gpu :38724 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-21 03:58:53,647 ERROR gpu :38724 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-21 03:58:54,181 DEBUG HandlerThread:38724 [handler.py:handle_request():144] handle_request: status_report 2023-04-21 03:58:55,689 ERROR gpu :38724 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-21 03:58:57,732 ERROR gpu :38724 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-21 03:58:59,234 DEBUG HandlerThread:38724 [handler.py:handle_request():144] handle_request: status_report 2023-04-21 03:58:59,773 ERROR gpu :38724 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-21 03:59:00,613 DEBUG HandlerThread:38724 [handler.py:handle_request():144] handle_request: stop_status 2023-04-21 03:59:00,614 DEBUG SenderThread:38724 [sender.py:send_request():402] send_request: stop_status 2023-04-21 03:59:01,821 ERROR gpu :38724 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-21 03:59:03,882 ERROR gpu :38724 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-21 03:59:04,912 DEBUG HandlerThread:38724 [handler.py:handle_request():144] handle_request: status_report 2023-04-21 03:59:05,942 ERROR gpu :38724 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-21 03:59:07,975 ERROR gpu :38724 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-21 03:59:09,999 DEBUG HandlerThread:38724 [handler.py:handle_request():144] handle_request: status_report 2023-04-21 03:59:10,029 ERROR gpu :38724 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-21 03:59:12,081 ERROR gpu :38724 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-21 03:59:14,142 ERROR gpu :38724 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-21 03:59:15,049 DEBUG HandlerThread:38724 [handler.py:handle_request():144] handle_request: status_report 2023-04-21 03:59:15,621 DEBUG HandlerThread:38724 [handler.py:handle_request():144] handle_request: stop_status 2023-04-21 03:59:15,622 DEBUG SenderThread:38724 [sender.py:send_request():402] send_request: stop_status 2023-04-21 03:59:16,222 INFO Thread-16 :38724 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230421_035843-kyzl8vjo\files\config.yaml 2023-04-21 03:59:16,259 ERROR gpu :38724 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-21 03:59:18,284 ERROR gpu :38724 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-21 03:59:20,314 ERROR gpu :38724 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-21 03:59:20,975 DEBUG HandlerThread:38724 [handler.py:handle_request():144] handle_request: status_report 2023-04-21 03:59:22,358 ERROR gpu :38724 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-21 03:59:24,414 ERROR gpu :38724 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-21 03:59:26,020 DEBUG HandlerThread:38724 [handler.py:handle_request():144] handle_request: status_report 2023-04-21 03:59:26,454 ERROR gpu :38724 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-21 03:59:28,502 ERROR gpu :38724 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-21 03:59:30,553 ERROR gpu :38724 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-21 03:59:30,638 DEBUG HandlerThread:38724 [handler.py:handle_request():144] handle_request: stop_status 2023-04-21 03:59:30,639 DEBUG SenderThread:38724 [sender.py:send_request():402] send_request: stop_status 2023-04-21 03:59:31,887 DEBUG HandlerThread:38724 [handler.py:handle_request():144] handle_request: status_report 2023-04-21 03:59:32,600 ERROR gpu :38724 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-21 03:59:34,663 ERROR gpu :38724 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-21 03:59:36,708 ERROR gpu :38724 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-21 03:59:36,918 DEBUG HandlerThread:38724 [handler.py:handle_request():144] handle_request: status_report 2023-04-21 03:59:38,769 ERROR gpu :38724 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-21 03:59:40,827 ERROR gpu :38724 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-21 03:59:41,961 DEBUG HandlerThread:38724 [handler.py:handle_request():144] handle_request: status_report 2023-04-21 03:59:42,900 ERROR gpu :38724 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-21 03:59:45,001 ERROR gpu :38724 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-21 03:59:45,433 DEBUG SystemMonitor:38724 [system_monitor.py:_start():159] Starting system metrics aggregation loop 2023-04-21 03:59:45,435 DEBUG SenderThread:38724 [sender.py:send():375] send: stats 2023-04-21 03:59:45,648 DEBUG HandlerThread:38724 [handler.py:handle_request():144] handle_request: stop_status 2023-04-21 03:59:45,648 DEBUG SenderThread:38724 [sender.py:send_request():402] send_request: stop_status 2023-04-21 03:59:47,109 ERROR gpu :38724 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-21 03:59:47,929 DEBUG HandlerThread:38724 [handler.py:handle_request():144] handle_request: status_report 2023-04-21 03:59:49,122 ERROR gpu :38724 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-21 03:59:51,146 ERROR gpu :38724 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-21 03:59:52,971 DEBUG HandlerThread:38724 [handler.py:handle_request():144] handle_request: status_report 2023-04-21 03:59:53,193 ERROR gpu :38724 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-21 03:59:55,245 ERROR gpu :38724 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-21 03:59:57,294 ERROR gpu :38724 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-21 03:59:58,023 DEBUG HandlerThread:38724 [handler.py:handle_request():144] handle_request: status_report 2023-04-21 03:59:59,335 ERROR gpu :38724 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-21 04:00:00,660 DEBUG HandlerThread:38724 [handler.py:handle_request():144] handle_request: stop_status 2023-04-21 04:00:00,660 DEBUG SenderThread:38724 [sender.py:send_request():402] send_request: stop_status 2023-04-21 04:00:01,393 ERROR gpu :38724 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-21 04:00:03,437 ERROR gpu :38724 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-21 04:00:03,965 DEBUG HandlerThread:38724 [handler.py:handle_request():144] handle_request: status_report 2023-04-21 04:00:05,490 ERROR gpu :38724 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-21 04:00:07,538 ERROR gpu :38724 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-21 04:00:09,004 DEBUG HandlerThread:38724 [handler.py:handle_request():144] handle_request: status_report 2023-04-21 04:00:09,586 ERROR gpu :38724 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-21 04:00:11,647 ERROR gpu :38724 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-21 04:00:13,695 ERROR gpu :38724 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-21 04:00:14,063 DEBUG HandlerThread:38724 [handler.py:handle_request():144] handle_request: status_report 2023-04-21 04:00:15,443 DEBUG SenderThread:38724 [sender.py:send():375] send: stats 2023-04-21 04:00:15,658 DEBUG HandlerThread:38724 [handler.py:handle_request():144] handle_request: stop_status 2023-04-21 04:00:15,659 DEBUG SenderThread:38724 [sender.py:send_request():402] send_request: stop_status 2023-04-21 04:00:15,741 ERROR gpu :38724 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-21 04:00:17,898 ERROR gpu :38724 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-21 04:00:19,916 ERROR gpu :38724 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-21 04:00:19,937 DEBUG HandlerThread:38724 [handler.py:handle_request():144] handle_request: status_report 2023-04-21 04:00:21,968 ERROR gpu :38724 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-21 04:00:24,008 ERROR gpu :38724 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-21 04:00:25,205 DEBUG HandlerThread:38724 [handler.py:handle_request():144] handle_request: status_report 2023-04-21 04:00:26,076 ERROR gpu :38724 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-21 04:00:27,067 INFO Thread-16 :38724 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230421_035843-kyzl8vjo\files\output.log 2023-04-21 04:00:28,115 ERROR gpu :38724 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-21 04:00:30,169 ERROR gpu :38724 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-21 04:00:30,239 DEBUG HandlerThread:38724 [handler.py:handle_request():144] handle_request: status_report 2023-04-21 04:00:30,663 DEBUG HandlerThread:38724 [handler.py:handle_request():144] handle_request: stop_status 2023-04-21 04:00:30,664 DEBUG SenderThread:38724 [sender.py:send_request():402] send_request: stop_status 2023-04-21 04:00:32,238 ERROR gpu :38724 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-21 04:00:34,282 ERROR gpu :38724 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-21 04:00:35,961 DEBUG HandlerThread:38724 [handler.py:handle_request():144] handle_request: status_report 2023-04-21 04:00:36,323 ERROR gpu :38724 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-21 04:00:38,382 ERROR gpu :38724 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-21 04:00:40,434 ERROR gpu :38724 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-21 04:00:41,018 DEBUG HandlerThread:38724 [handler.py:handle_request():144] handle_request: status_report 2023-04-21 04:00:42,479 ERROR gpu :38724 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-21 04:00:44,530 ERROR gpu :38724 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-21 04:00:45,451 DEBUG SenderThread:38724 [sender.py:send():375] send: stats 2023-04-21 04:00:45,670 DEBUG HandlerThread:38724 [handler.py:handle_request():144] handle_request: stop_status 2023-04-21 04:00:45,671 DEBUG SenderThread:38724 [sender.py:send_request():402] send_request: stop_status 2023-04-21 04:00:46,577 ERROR gpu :38724 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-21 04:00:46,949 DEBUG HandlerThread:38724 [handler.py:handle_request():144] handle_request: status_report 2023-04-21 04:00:48,705 ERROR gpu :38724 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-21 04:00:50,720 ERROR gpu :38724 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-21 04:00:51,993 DEBUG HandlerThread:38724 [handler.py:handle_request():144] handle_request: status_report 2023-04-21 04:00:52,755 ERROR gpu :38724 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-21 04:00:54,790 INFO Thread-16 :38724 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230421_035843-kyzl8vjo\files\output.log 2023-04-21 04:00:54,797 ERROR gpu :38724 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-21 04:00:56,849 ERROR gpu :38724 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-21 04:00:57,625 DEBUG HandlerThread:38724 [handler.py:handle_request():144] handle_request: status_report 2023-04-21 04:00:58,902 ERROR gpu :38724 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-21 04:01:00,680 DEBUG HandlerThread:38724 [handler.py:handle_request():144] handle_request: stop_status 2023-04-21 04:01:00,681 DEBUG SenderThread:38724 [sender.py:send_request():402] send_request: stop_status 2023-04-21 04:01:00,943 ERROR gpu :38724 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-21 04:01:02,989 DEBUG HandlerThread:38724 [handler.py:handle_request():144] handle_request: status_report 2023-04-21 04:01:02,997 ERROR gpu :38724 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-21 04:01:05,037 ERROR gpu :38724 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-21 04:01:07,105 ERROR gpu :38724 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-21 04:01:08,826 DEBUG HandlerThread:38724 [handler.py:handle_request():144] handle_request: status_report 2023-04-21 04:01:09,148 INFO Thread-16 :38724 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230421_035843-kyzl8vjo\files\output.log 2023-04-21 04:01:09,160 ERROR gpu :38724 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-21 04:01:11,225 ERROR gpu :38724 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-21 04:01:13,283 ERROR gpu :38724 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-21 04:01:13,867 DEBUG HandlerThread:38724 [handler.py:handle_request():144] handle_request: status_report 2023-04-21 04:01:15,365 ERROR gpu :38724 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-21 04:01:15,463 DEBUG SenderThread:38724 [sender.py:send():375] send: stats 2023-04-21 04:01:15,693 DEBUG HandlerThread:38724 [handler.py:handle_request():144] handle_request: stop_status 2023-04-21 04:01:15,694 DEBUG SenderThread:38724 [sender.py:send_request():402] send_request: stop_status 2023-04-21 04:01:17,413 ERROR gpu :38724 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-21 04:01:18,961 DEBUG HandlerThread:38724 [handler.py:handle_request():144] handle_request: status_report 2023-04-21 04:01:19,535 ERROR gpu :38724 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-21 04:01:21,557 INFO Thread-16 :38724 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230421_035843-kyzl8vjo\files\output.log 2023-04-21 04:01:21,574 ERROR gpu :38724 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-21 04:01:23,628 ERROR gpu :38724 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-21 04:01:24,359 DEBUG HandlerThread:38724 [handler.py:handle_request():144] handle_request: status_report 2023-04-21 04:01:25,715 ERROR gpu :38724 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-21 04:01:27,806 ERROR gpu :38724 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-21 04:01:29,433 DEBUG HandlerThread:38724 [handler.py:handle_request():144] handle_request: status_report 2023-04-21 04:01:29,872 ERROR gpu :38724 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-21 04:01:30,700 DEBUG HandlerThread:38724 [handler.py:handle_request():144] handle_request: stop_status 2023-04-21 04:01:30,700 DEBUG SenderThread:38724 [sender.py:send_request():402] send_request: stop_status 2023-04-21 04:01:31,961 ERROR gpu :38724 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-21 04:01:34,012 ERROR gpu :38724 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-21 04:01:35,005 DEBUG HandlerThread:38724 [handler.py:handle_request():144] handle_request: status_report 2023-04-21 04:01:36,083 ERROR gpu :38724 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-21 04:01:38,130 INFO Thread-16 :38724 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230421_035843-kyzl8vjo\files\output.log 2023-04-21 04:01:38,139 ERROR gpu :38724 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-21 04:01:39,136 INFO Thread-16 :38724 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230421_035843-kyzl8vjo\files\output.log 2023-04-21 04:01:40,204 ERROR gpu :38724 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-21 04:01:40,469 DEBUG HandlerThread:38724 [handler.py:handle_request():144] handle_request: status_report 2023-04-21 04:01:42,267 ERROR gpu :38724 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-21 04:01:44,324 ERROR gpu :38724 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-21 04:01:45,464 DEBUG SenderThread:38724 [sender.py:send():375] send: stats 2023-04-21 04:01:45,714 DEBUG HandlerThread:38724 [handler.py:handle_request():144] handle_request: stop_status 2023-04-21 04:01:45,714 DEBUG HandlerThread:38724 [handler.py:handle_request():144] handle_request: status_report 2023-04-21 04:01:45,715 DEBUG SenderThread:38724 [sender.py:send_request():402] send_request: stop_status 2023-04-21 04:01:46,376 ERROR gpu :38724 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-21 04:01:48,446 ERROR gpu :38724 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-21 04:01:50,552 ERROR gpu :38724 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-21 04:01:51,005 DEBUG HandlerThread:38724 [handler.py:handle_request():144] handle_request: status_report 2023-04-21 04:01:52,570 ERROR gpu :38724 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-21 04:01:53,539 INFO Thread-16 :38724 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230421_035843-kyzl8vjo\files\output.log 2023-04-21 04:01:54,584 ERROR gpu :38724 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-21 04:01:56,633 ERROR gpu :38724 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-21 04:01:56,701 DEBUG HandlerThread:38724 [handler.py:handle_request():144] handle_request: status_report 2023-04-21 04:01:58,686 ERROR gpu :38724 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-21 04:02:00,727 DEBUG HandlerThread:38724 [handler.py:handle_request():144] handle_request: stop_status 2023-04-21 04:02:00,728 DEBUG SenderThread:38724 [sender.py:send_request():402] send_request: stop_status 2023-04-21 04:02:00,733 ERROR gpu :38724 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-21 04:02:01,991 DEBUG HandlerThread:38724 [handler.py:handle_request():144] handle_request: status_report 2023-04-21 04:02:02,776 ERROR gpu :38724 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-21 04:02:04,829 ERROR gpu :38724 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-21 04:02:06,866 ERROR gpu :38724 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-21 04:02:07,034 DEBUG HandlerThread:38724 [handler.py:handle_request():144] handle_request: status_report 2023-04-21 04:02:08,929 ERROR gpu :38724 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-21 04:02:09,923 INFO Thread-16 :38724 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230421_035843-kyzl8vjo\files\output.log 2023-04-21 04:02:11,007 ERROR gpu :38724 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-21 04:02:12,654 DEBUG HandlerThread:38724 [handler.py:handle_request():144] handle_request: status_report 2023-04-21 04:02:13,041 ERROR gpu :38724 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-21 04:02:15,129 ERROR gpu :38724 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-21 04:02:15,467 DEBUG SenderThread:38724 [sender.py:send():375] send: stats 2023-04-21 04:02:15,728 DEBUG HandlerThread:38724 [handler.py:handle_request():144] handle_request: stop_status 2023-04-21 04:02:15,729 DEBUG SenderThread:38724 [sender.py:send_request():402] send_request: stop_status 2023-04-21 04:02:17,177 ERROR gpu :38724 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-21 04:02:18,030 DEBUG HandlerThread:38724 [handler.py:handle_request():144] handle_request: status_report 2023-04-21 04:02:19,221 ERROR gpu :38724 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-21 04:02:21,336 ERROR gpu :38724 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-21 04:02:23,179 DEBUG HandlerThread:38724 [handler.py:handle_request():144] handle_request: status_report 2023-04-21 04:02:23,330 INFO Thread-16 :38724 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230421_035843-kyzl8vjo\files\output.log 2023-04-21 04:02:23,359 ERROR gpu :38724 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-21 04:02:25,388 ERROR gpu :38724 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-21 04:02:27,441 ERROR gpu :38724 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-21 04:02:28,226 DEBUG HandlerThread:38724 [handler.py:handle_request():144] handle_request: status_report 2023-04-21 04:02:29,496 ERROR gpu :38724 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-21 04:02:30,753 DEBUG HandlerThread:38724 [handler.py:handle_request():144] handle_request: stop_status 2023-04-21 04:02:30,753 DEBUG SenderThread:38724 [sender.py:send_request():402] send_request: stop_status 2023-04-21 04:02:31,555 ERROR gpu :38724 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-21 04:02:33,589 ERROR gpu :38724 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-21 04:02:34,032 DEBUG HandlerThread:38724 [handler.py:handle_request():144] handle_request: status_report 2023-04-21 04:02:35,639 ERROR gpu :38724 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-21 04:02:37,680 ERROR gpu :38724 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-21 04:02:39,720 DEBUG HandlerThread:38724 [handler.py:handle_request():144] handle_request: status_report 2023-04-21 04:02:39,721 INFO Thread-16 :38724 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230421_035843-kyzl8vjo\files\output.log 2023-04-21 04:02:39,730 ERROR gpu :38724 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-21 04:02:41,781 ERROR gpu :38724 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-21 04:02:43,830 ERROR gpu :38724 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-21 04:02:44,830 DEBUG HandlerThread:38724 [handler.py:handle_request():144] handle_request: status_report 2023-04-21 04:02:45,468 DEBUG SenderThread:38724 [sender.py:send():375] send: stats 2023-04-21 04:02:45,770 DEBUG HandlerThread:38724 [handler.py:handle_request():144] handle_request: stop_status 2023-04-21 04:02:45,771 DEBUG SenderThread:38724 [sender.py:send_request():402] send_request: stop_status 2023-04-21 04:02:45,866 ERROR gpu :38724 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-21 04:02:47,956 ERROR gpu :38724 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-21 04:02:49,987 ERROR gpu :38724 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-21 04:02:50,048 DEBUG HandlerThread:38724 [handler.py:handle_request():144] handle_request: status_report 2023-04-21 04:02:51,159 DEBUG HandlerThread:38724 [handler.py:handle_request():144] handle_request: partial_history 2023-04-21 04:02:51,160 DEBUG SenderThread:38724 [sender.py:send():375] send: metric 2023-04-21 04:02:51,161 DEBUG SenderThread:38724 [sender.py:send():375] send: metric 2023-04-21 04:02:51,161 DEBUG SenderThread:38724 [sender.py:send():375] send: metric 2023-04-21 04:02:51,161 DEBUG SenderThread:38724 [sender.py:send():375] send: history 2023-04-21 04:02:51,161 DEBUG SenderThread:38724 [sender.py:send_request():402] send_request: summary_record 2023-04-21 04:02:51,162 INFO SenderThread:38724 [sender.py:_save_file():1378] saving file wandb-summary.json with policy end 2023-04-21 04:02:51,713 DEBUG HandlerThread:38724 [handler.py:handle_request():144] handle_request: partial_history 2023-04-21 04:02:51,716 DEBUG SenderThread:38724 [sender.py:send():375] send: metric 2023-04-21 04:02:51,716 DEBUG SenderThread:38724 [sender.py:send():375] send: metric 2023-04-21 04:02:51,717 DEBUG SenderThread:38724 [sender.py:send():375] send: metric 2023-04-21 04:02:51,717 DEBUG SenderThread:38724 [sender.py:send():375] send: metric 2023-04-21 04:02:51,718 DEBUG SenderThread:38724 [sender.py:send():375] send: metric 2023-04-21 04:02:51,718 DEBUG SenderThread:38724 [sender.py:send():375] send: history 2023-04-21 04:02:51,720 DEBUG SenderThread:38724 [sender.py:send_request():402] send_request: summary_record 2023-04-21 04:02:51,721 INFO SenderThread:38724 [sender.py:_save_file():1378] saving file wandb-summary.json with policy end 2023-04-21 04:02:51,743 DEBUG SenderThread:38724 [sender.py:send():375] send: exit 2023-04-21 04:02:51,743 INFO SenderThread:38724 [sender.py:send_exit():598] handling exit code: 0 2023-04-21 04:02:51,743 INFO SenderThread:38724 [sender.py:send_exit():600] handling runtime: 246 2023-04-21 04:02:51,744 INFO SenderThread:38724 [sender.py:_save_file():1378] saving file wandb-summary.json with policy end 2023-04-21 04:02:51,744 INFO SenderThread:38724 [sender.py:send_exit():606] send defer 2023-04-21 04:02:51,745 DEBUG HandlerThread:38724 [handler.py:handle_request():144] handle_request: defer 2023-04-21 04:02:51,745 INFO HandlerThread:38724 [handler.py:handle_request_defer():170] handle defer: 0 2023-04-21 04:02:51,745 DEBUG SenderThread:38724 [sender.py:send_request():402] send_request: defer 2023-04-21 04:02:51,745 INFO SenderThread:38724 [sender.py:send_request_defer():622] handle sender defer: 0 2023-04-21 04:02:51,745 INFO SenderThread:38724 [sender.py:transition_state():626] send defer: 1 2023-04-21 04:02:51,745 DEBUG HandlerThread:38724 [handler.py:handle_request():144] handle_request: defer 2023-04-21 04:02:51,746 INFO HandlerThread:38724 [handler.py:handle_request_defer():170] handle defer: 1 2023-04-21 04:02:51,746 DEBUG SenderThread:38724 [sender.py:send_request():402] send_request: defer 2023-04-21 04:02:51,746 INFO SenderThread:38724 [sender.py:send_request_defer():622] handle sender defer: 1 2023-04-21 04:02:51,746 INFO SenderThread:38724 [sender.py:transition_state():626] send defer: 2 2023-04-21 04:02:51,746 DEBUG HandlerThread:38724 [handler.py:handle_request():144] handle_request: defer 2023-04-21 04:02:51,746 INFO HandlerThread:38724 [handler.py:handle_request_defer():170] handle defer: 2 2023-04-21 04:02:51,747 INFO HandlerThread:38724 [system_monitor.py:finish():190] Stopping system monitor 2023-04-21 04:02:51,747 DEBUG SystemMonitor:38724 [system_monitor.py:_start():166] Finished system metrics aggregation loop 2023-04-21 04:02:51,747 INFO HandlerThread:38724 [interfaces.py:finish():202] Joined cpu monitor 2023-04-21 04:02:51,757 DEBUG SystemMonitor:38724 [system_monitor.py:_start():170] Publishing last batch of metrics 2023-04-21 04:02:51,758 INFO HandlerThread:38724 [interfaces.py:finish():202] Joined disk monitor 2023-04-21 04:02:51,806 INFO HandlerThread:38724 [interfaces.py:finish():202] Joined gpu monitor 2023-04-21 04:02:51,806 INFO HandlerThread:38724 [interfaces.py:finish():202] Joined memory monitor 2023-04-21 04:02:51,806 INFO HandlerThread:38724 [interfaces.py:finish():202] Joined network monitor 2023-04-21 04:02:51,806 DEBUG SenderThread:38724 [sender.py:send_request():402] send_request: defer 2023-04-21 04:02:51,807 INFO SenderThread:38724 [sender.py:send_request_defer():622] handle sender defer: 2 2023-04-21 04:02:51,807 INFO SenderThread:38724 [sender.py:transition_state():626] send defer: 3 2023-04-21 04:02:51,807 DEBUG SenderThread:38724 [sender.py:send():375] send: stats 2023-04-21 04:02:51,807 DEBUG HandlerThread:38724 [handler.py:handle_request():144] handle_request: defer 2023-04-21 04:02:51,807 INFO HandlerThread:38724 [handler.py:handle_request_defer():170] handle defer: 3 2023-04-21 04:02:51,808 DEBUG SenderThread:38724 [sender.py:send_request():402] send_request: defer 2023-04-21 04:02:51,808 INFO SenderThread:38724 [sender.py:send_request_defer():622] handle sender defer: 3 2023-04-21 04:02:51,808 INFO SenderThread:38724 [sender.py:transition_state():626] send defer: 4 2023-04-21 04:02:51,809 DEBUG HandlerThread:38724 [handler.py:handle_request():144] handle_request: defer 2023-04-21 04:02:51,809 INFO HandlerThread:38724 [handler.py:handle_request_defer():170] handle defer: 4 2023-04-21 04:02:51,809 DEBUG SenderThread:38724 [sender.py:send_request():402] send_request: defer 2023-04-21 04:02:51,809 INFO SenderThread:38724 [sender.py:send_request_defer():622] handle sender defer: 4 2023-04-21 04:02:51,809 INFO SenderThread:38724 [sender.py:transition_state():626] send defer: 5 2023-04-21 04:02:51,810 DEBUG HandlerThread:38724 [handler.py:handle_request():144] handle_request: defer 2023-04-21 04:02:51,810 INFO HandlerThread:38724 [handler.py:handle_request_defer():170] handle defer: 5 2023-04-21 04:02:51,810 DEBUG SenderThread:38724 [sender.py:send():375] send: summary 2023-04-21 04:02:51,810 INFO SenderThread:38724 [sender.py:_save_file():1378] saving file wandb-summary.json with policy end 2023-04-21 04:02:51,811 DEBUG SenderThread:38724 [sender.py:send_request():402] send_request: defer 2023-04-21 04:02:51,811 INFO SenderThread:38724 [sender.py:send_request_defer():622] handle sender defer: 5 2023-04-21 04:02:51,811 INFO SenderThread:38724 [sender.py:transition_state():626] send defer: 6 2023-04-21 04:02:51,811 DEBUG HandlerThread:38724 [handler.py:handle_request():144] handle_request: defer 2023-04-21 04:02:51,811 INFO HandlerThread:38724 [handler.py:handle_request_defer():170] handle defer: 6 2023-04-21 04:02:51,812 DEBUG SenderThread:38724 [sender.py:send_request():402] send_request: defer 2023-04-21 04:02:51,812 INFO SenderThread:38724 [sender.py:send_request_defer():622] handle sender defer: 6 2023-04-21 04:02:51,816 DEBUG HandlerThread:38724 [handler.py:handle_request():144] handle_request: status_report 2023-04-21 04:02:51,985 INFO Thread-16 :38724 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230421_035843-kyzl8vjo\files\wandb-summary.json 2023-04-21 04:02:52,432 INFO SenderThread:38724 [sender.py:transition_state():626] send defer: 7 2023-04-21 04:02:52,433 DEBUG HandlerThread:38724 [handler.py:handle_request():144] handle_request: defer 2023-04-21 04:02:52,433 INFO HandlerThread:38724 [handler.py:handle_request_defer():170] handle defer: 7 2023-04-21 04:02:52,433 DEBUG SenderThread:38724 [sender.py:send_request():402] send_request: defer 2023-04-21 04:02:52,433 INFO SenderThread:38724 [sender.py:send_request_defer():622] handle sender defer: 7 2023-04-21 04:02:52,812 DEBUG HandlerThread:38724 [handler.py:handle_request():144] handle_request: poll_exit 2023-04-21 04:02:52,998 INFO Thread-16 :38724 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230421_035843-kyzl8vjo\files\config.yaml 2023-04-21 04:02:52,999 INFO Thread-16 :38724 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230421_035843-kyzl8vjo\files\output.log 2023-04-21 04:02:54,014 INFO Thread-16 :38724 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230421_035843-kyzl8vjo\files\output.log 2023-04-21 04:02:54,311 INFO SenderThread:38724 [sender.py:transition_state():626] send defer: 8 2023-04-21 04:02:54,311 DEBUG SenderThread:38724 [sender.py:send_request():402] send_request: poll_exit 2023-04-21 04:02:54,311 DEBUG HandlerThread:38724 [handler.py:handle_request():144] handle_request: defer 2023-04-21 04:02:54,311 INFO HandlerThread:38724 [handler.py:handle_request_defer():170] handle defer: 8 2023-04-21 04:02:54,311 DEBUG SenderThread:38724 [sender.py:send_request():402] send_request: defer 2023-04-21 04:02:54,311 INFO SenderThread:38724 [sender.py:send_request_defer():622] handle sender defer: 8 2023-04-21 04:02:54,338 INFO SenderThread:38724 [sender.py:transition_state():626] send defer: 9 2023-04-21 04:02:54,338 DEBUG HandlerThread:38724 [handler.py:handle_request():144] handle_request: defer 2023-04-21 04:02:54,338 INFO HandlerThread:38724 [handler.py:handle_request_defer():170] handle defer: 9 2023-04-21 04:02:54,338 DEBUG SenderThread:38724 [sender.py:send():375] send: artifact 2023-04-21 04:02:55,024 INFO Thread-16 :38724 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230421_035843-kyzl8vjo\files\output.log 2023-04-21 04:02:55,676 INFO SenderThread:38724 [sender.py:send_artifact():1474] sent artifact job-https___github.com_THUDM_ChatGLM-6B_ptuning_main.py - {'id': 'QXJ0aWZhY3Q6NDI5MTMzMDE1', 'digest': '651bcc3a27fe0b9435b558b5f4a1dbed', 'state': 'COMMITTED', 'aliases': [{'artifactCollectionName': 'job-https___github.com_THUDM_ChatGLM-6B_ptuning_main.py', 'alias': 'latest'}, {'artifactCollectionName': 'job-https___github.com_THUDM_ChatGLM-6B_ptuning_main.py', 'alias': 'v1'}], 'artifactSequence': {'id': 'QXJ0aWZhY3RDb2xsZWN0aW9uOjYzNDcyMzc3', 'latestArtifact': {'id': 'QXJ0aWZhY3Q6NDI5MjY1Nzg0', 'versionIndex': 2}}, 'version': 'v1'} 2023-04-21 04:02:55,677 DEBUG SenderThread:38724 [sender.py:send_request():402] send_request: defer 2023-04-21 04:02:55,677 INFO SenderThread:38724 [sender.py:send_request_defer():622] handle sender defer: 9 2023-04-21 04:02:55,677 INFO SenderThread:38724 [dir_watcher.py:finish():365] shutting down directory watcher 2023-04-21 04:02:56,025 INFO SenderThread:38724 [dir_watcher.py:finish():395] scan: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230421_035843-kyzl8vjo\files 2023-04-21 04:02:56,026 INFO SenderThread:38724 [dir_watcher.py:finish():409] scan save: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230421_035843-kyzl8vjo\files\config.yaml config.yaml 2023-04-21 04:02:56,026 INFO SenderThread:38724 [dir_watcher.py:finish():409] scan save: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230421_035843-kyzl8vjo\files\output.log output.log 2023-04-21 04:02:56,029 INFO SenderThread:38724 [dir_watcher.py:finish():409] scan save: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230421_035843-kyzl8vjo\files\requirements.txt requirements.txt 2023-04-21 04:02:56,033 INFO SenderThread:38724 [dir_watcher.py:finish():409] scan save: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230421_035843-kyzl8vjo\files\wandb-metadata.json wandb-metadata.json 2023-04-21 04:02:56,033 INFO SenderThread:38724 [dir_watcher.py:finish():409] scan save: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230421_035843-kyzl8vjo\files\wandb-summary.json wandb-summary.json 2023-04-21 04:02:56,036 INFO SenderThread:38724 [sender.py:transition_state():626] send defer: 10 2023-04-21 04:02:56,036 DEBUG HandlerThread:38724 [handler.py:handle_request():144] handle_request: defer 2023-04-21 04:02:56,037 INFO HandlerThread:38724 [handler.py:handle_request_defer():170] handle defer: 10 2023-04-21 04:02:56,037 DEBUG SenderThread:38724 [sender.py:send_request():402] send_request: defer 2023-04-21 04:02:56,039 INFO SenderThread:38724 [sender.py:send_request_defer():622] handle sender defer: 10 2023-04-21 04:02:56,039 INFO SenderThread:38724 [file_pusher.py:finish():168] shutting down file pusher 2023-04-21 04:02:56,691 INFO wandb-upload_1:38724 [upload_job.py:push():137] Uploaded file E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230421_035843-kyzl8vjo\files\output.log 2023-04-21 04:02:57,069 INFO wandb-upload_0:38724 [upload_job.py:push():137] Uploaded file E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230421_035843-kyzl8vjo\files\config.yaml 2023-04-21 04:02:57,249 INFO wandb-upload_2:38724 [upload_job.py:push():137] Uploaded file E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230421_035843-kyzl8vjo\files\requirements.txt 2023-04-21 04:02:57,296 INFO wandb-upload_3:38724 [upload_job.py:push():137] Uploaded file E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230421_035843-kyzl8vjo\files\wandb-summary.json 2023-04-21 04:02:57,503 INFO Thread-15 :38724 [sender.py:transition_state():626] send defer: 11 2023-04-21 04:02:57,503 DEBUG HandlerThread:38724 [handler.py:handle_request():144] handle_request: defer 2023-04-21 04:02:57,503 INFO HandlerThread:38724 [handler.py:handle_request_defer():170] handle defer: 11 2023-04-21 04:02:57,504 DEBUG SenderThread:38724 [sender.py:send_request():402] send_request: defer 2023-04-21 04:02:57,504 INFO SenderThread:38724 [sender.py:send_request_defer():622] handle sender defer: 11 2023-04-21 04:02:57,504 INFO SenderThread:38724 [file_pusher.py:join():173] waiting for file pusher 2023-04-21 04:02:57,504 INFO SenderThread:38724 [sender.py:transition_state():626] send defer: 12 2023-04-21 04:02:57,504 DEBUG HandlerThread:38724 [handler.py:handle_request():144] handle_request: defer 2023-04-21 04:02:57,504 INFO HandlerThread:38724 [handler.py:handle_request_defer():170] handle defer: 12 2023-04-21 04:02:57,504 DEBUG SenderThread:38724 [sender.py:send_request():402] send_request: defer 2023-04-21 04:02:57,504 INFO SenderThread:38724 [sender.py:send_request_defer():622] handle sender defer: 12 2023-04-21 04:02:57,864 DEBUG HandlerThread:38724 [handler.py:handle_request():144] handle_request: keepalive 2023-04-21 04:02:58,146 INFO SenderThread:38724 [sender.py:transition_state():626] send defer: 13 2023-04-21 04:02:58,146 DEBUG HandlerThread:38724 [handler.py:handle_request():144] handle_request: defer 2023-04-21 04:02:58,146 INFO HandlerThread:38724 [handler.py:handle_request_defer():170] handle defer: 13 2023-04-21 04:02:58,146 DEBUG SenderThread:38724 [sender.py:send_request():402] send_request: defer 2023-04-21 04:02:58,146 INFO SenderThread:38724 [sender.py:send_request_defer():622] handle sender defer: 13 2023-04-21 04:02:58,146 INFO SenderThread:38724 [sender.py:transition_state():626] send defer: 14 2023-04-21 04:02:58,147 DEBUG HandlerThread:38724 [handler.py:handle_request():144] handle_request: defer 2023-04-21 04:02:58,147 DEBUG SenderThread:38724 [sender.py:send():375] send: final 2023-04-21 04:02:58,147 INFO HandlerThread:38724 [handler.py:handle_request_defer():170] handle defer: 14 2023-04-21 04:02:58,147 DEBUG SenderThread:38724 [sender.py:send():375] send: footer 2023-04-21 04:02:58,147 DEBUG SenderThread:38724 [sender.py:send_request():402] send_request: defer 2023-04-21 04:02:58,147 INFO SenderThread:38724 [sender.py:send_request_defer():622] handle sender defer: 14 2023-04-21 04:02:58,148 DEBUG HandlerThread:38724 [handler.py:handle_request():144] handle_request: poll_exit 2023-04-21 04:02:58,148 DEBUG HandlerThread:38724 [handler.py:handle_request():144] handle_request: server_info 2023-04-21 04:02:58,148 DEBUG HandlerThread:38724 [handler.py:handle_request():144] handle_request: get_summary 2023-04-21 04:02:58,148 DEBUG SenderThread:38724 [sender.py:send_request():402] send_request: poll_exit 2023-04-21 04:02:58,148 DEBUG HandlerThread:38724 [handler.py:handle_request():144] handle_request: sampled_history 2023-04-21 04:02:58,148 DEBUG SenderThread:38724 [sender.py:send_request():402] send_request: server_info 2023-04-21 04:02:58,399 INFO MainThread:38724 [wandb_run.py:_footer_history_summary_info():3476] rendering history 2023-04-21 04:02:58,400 INFO MainThread:38724 [wandb_run.py:_footer_history_summary_info():3508] rendering summary 2023-04-21 04:02:58,410 INFO MainThread:38724 [wandb_run.py:_footer_sync_info():3434] logging synced files 2023-04-21 04:02:58,412 DEBUG HandlerThread:38724 [handler.py:handle_request():144] handle_request: shutdown 2023-04-21 04:02:58,412 INFO HandlerThread:38724 [handler.py:finish():845] shutting down handler 2023-04-21 04:02:59,160 INFO WriterThread:38724 [datastore.py:close():298] close: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230421_035843-kyzl8vjo\run-kyzl8vjo.wandb 2023-04-21 04:02:59,411 INFO SenderThread:38724 [sender.py:finish():1550] shutting down sender 2023-04-21 04:02:59,411 INFO SenderThread:38724 [file_pusher.py:finish():168] shutting down file pusher 2023-04-21 04:02:59,411 INFO SenderThread:38724 [file_pusher.py:join():173] waiting for file pusher