| 2024-09-21 11:52:59,195 INFO MainThread:1311189 [wandb_setup.py:_flush():76] Current SDK version is 0.17.5 | |
| 2024-09-21 11:52:59,196 INFO MainThread:1311189 [wandb_setup.py:_flush():76] Configure stats pid to 1311189 | |
| 2024-09-21 11:52:59,196 INFO MainThread:1311189 [wandb_setup.py:_flush():76] Loading settings from /home/yangyaodong/.config/wandb/settings | |
| 2024-09-21 11:52:59,196 INFO MainThread:1311189 [wandb_setup.py:_flush():76] Loading settings from /aifs4su/yaodong/projects/hantao/dev_cham/align-anything/scripts/wandb/settings | |
| 2024-09-21 11:52:59,196 INFO MainThread:1311189 [wandb_setup.py:_flush():76] Loading settings from environment variables: {'api_key': '***REDACTED***'} | |
| 2024-09-21 11:52:59,196 INFO MainThread:1311189 [wandb_setup.py:_flush():76] Applying setup settings: {'_disable_service': False} | |
| 2024-09-21 11:52:59,196 WARNING MainThread:1311189 [wandb_setup.py:_flush():76] Could not find program at -m align_anything.trainers.tiv_to_t.dpo | |
| 2024-09-21 11:52:59,196 INFO MainThread:1311189 [wandb_setup.py:_flush():76] Inferring run settings from compute environment: {'program_relpath': None, 'program': '-m align_anything.trainers.tiv_to_t.dpo'} | |
| 2024-09-21 11:52:59,196 INFO MainThread:1311189 [wandb_setup.py:_flush():76] Applying login settings: {} | |
| 2024-09-21 11:52:59,196 INFO MainThread:1311189 [wandb_init.py:_log_setup():529] Logging user logs to ../outputs/dpo_tiv2t_1.5k_base/wandb/run-20240921_115259-p9bvnzls/logs/debug.log | |
| 2024-09-21 11:52:59,196 INFO MainThread:1311189 [wandb_init.py:_log_setup():530] Logging internal logs to ../outputs/dpo_tiv2t_1.5k_base/wandb/run-20240921_115259-p9bvnzls/logs/debug-internal.log | |
| 2024-09-21 11:52:59,196 INFO MainThread:1311189 [wandb_init.py:init():569] calling init triggers | |
| 2024-09-21 11:52:59,197 INFO MainThread:1311189 [wandb_init.py:init():576] wandb.init called with sweep_config: {} | |
| config: {'train_cfgs': {'ds_cfgs': 'ds_z3_config.json', 'epochs': 3, 'seed': 42, 'per_device_train_batch_size': 1.0, 'per_device_eval_batch_size': 1.0, 'gradient_accumulation_steps': 1.0, 'gradient_checkpointing': True, 'learning_rate': 1e-06, 'lr_scheduler_type': 'cosine', 'lr_warmup_ratio': 0.01, 'weight_decay': 0.0, 'adam_betas': [0.9, 0.95], 'bf16': True, 'fp16': False, 'eval_strategy': 'epoch', 'eval_interval': 10, 'regularization': 0.001, 'scale_coeff': 0.1, 'freeze_mm_proj': False, 'freeze_vision_tower': True, 'freeze_language_model': False}, 'data_cfgs': {'train_datasets': '/aifs4su/yaodong/datasets/aaa_dataset/TV2T-preference/extracted', 'train_template': 'NExTQA_preference', 'train_size': None, 'train_split': 'train', 'train_subset': None, 'train_data_files': 'extracted_preference_1.5k_washed.json', 'train_optional_args': [], 'eval_datasets': None, 'eval_template': None, 'eval_size': None, 'eval_split': None, 'eval_subset': None, 'eval_data_files': None, 'eval_optional_args': []}, 'logger_cfgs': {'log_type': 'wandb', 'log_project': 'align-anything', 'log_run_name': 'dpo', 'output_dir': '../outputs/dpo_tiv2t_1.5k_base', 'cache_dir': None, 'save_interval': 100000}, 'model_cfgs': {'model_name_or_path': '/aifs4su/yaodong/models/Qwen2-VL-7B-Instruct', 'trust_remote_code': True, 'model_max_length': 4096}, 'special_tokens': None} | |
| 2024-09-21 11:52:59,197 INFO MainThread:1311189 [wandb_init.py:init():619] starting backend | |
| 2024-09-21 11:52:59,197 INFO MainThread:1311189 [wandb_init.py:init():623] setting up manager | |
| 2024-09-21 11:52:59,198 INFO MainThread:1311189 [backend.py:_multiprocessing_setup():105] multiprocessing start_methods=fork,spawn,forkserver, using: spawn | |
| 2024-09-21 11:52:59,201 INFO MainThread:1311189 [wandb_init.py:init():631] backend started and connected | |
| 2024-09-21 11:52:59,204 INFO MainThread:1311189 [wandb_init.py:init():720] updated telemetry | |
| 2024-09-21 11:52:59,226 INFO MainThread:1311189 [wandb_init.py:init():753] communicating run to backend with 90.0 second timeout | |
| 2024-09-21 11:52:59,742 INFO MainThread:1311189 [wandb_run.py:_on_init():2435] communicating current version | |
| 2024-09-21 11:52:59,955 INFO MainThread:1311189 [wandb_run.py:_on_init():2444] got version response upgrade_message: "wandb version 0.18.1 is available! To upgrade, please run:\n $ pip install wandb --upgrade" | |
| 2024-09-21 11:52:59,956 INFO MainThread:1311189 [wandb_init.py:init():804] starting run threads in backend | |
| 2024-09-21 11:53:06,573 INFO MainThread:1311189 [wandb_run.py:_console_start():2413] atexit reg | |
| 2024-09-21 11:53:06,574 INFO MainThread:1311189 [wandb_run.py:_redirect():2255] redirect: wrap_raw | |
| 2024-09-21 11:53:06,574 INFO MainThread:1311189 [wandb_run.py:_redirect():2320] Wrapping output streams. | |
| 2024-09-21 11:53:06,574 INFO MainThread:1311189 [wandb_run.py:_redirect():2345] Redirects installed. | |
| 2024-09-21 11:53:06,633 INFO MainThread:1311189 [wandb_init.py:init():847] run started, returning control to user process | |