| 2024-05-13 20:46:52,817 INFO MainThread:1244577 [wandb_setup.py:_flush():76] Current SDK version is 0.17.0 | |
| 2024-05-13 20:46:52,817 INFO MainThread:1244577 [wandb_setup.py:_flush():76] Configure stats pid to 1244577 | |
| 2024-05-13 20:46:52,817 INFO MainThread:1244577 [wandb_setup.py:_flush():76] Loading settings from /home/sanchit/.config/wandb/settings | |
| 2024-05-13 20:46:52,817 INFO MainThread:1244577 [wandb_setup.py:_flush():76] Loading settings from /raid/sanchit/parler-tts-mini-v0.1-expresso-concatenated-combined/wandb/settings | |
| 2024-05-13 20:46:52,817 INFO MainThread:1244577 [wandb_setup.py:_flush():76] Loading settings from environment variables: {} | |
| 2024-05-13 20:46:52,817 INFO MainThread:1244577 [wandb_setup.py:_flush():76] Applying setup settings: {'_disable_service': False} | |
| 2024-05-13 20:46:52,817 INFO MainThread:1244577 [wandb_setup.py:_flush():76] Inferring run settings from compute environment: {'program_relpath': 'run_parler_tts_training.py', 'program_abspath': '/raid/sanchit/parler-tts-mini-v0.1-expresso-concatenated-combined/run_parler_tts_training.py', 'program': '/raid/sanchit/parler-tts-mini-v0.1-expresso-concatenated-combined/run_parler_tts_training.py'} | |
| 2024-05-13 20:46:52,817 INFO MainThread:1244577 [wandb_setup.py:_flush():76] Applying login settings: {} | |
| 2024-05-13 20:46:52,817 INFO MainThread:1244577 [wandb_init.py:_log_setup():520] Logging user logs to /raid/sanchit/parler-tts-mini-v0.1-expresso-concatenated-combined/wandb/run-20240513_204652-m0g0ap7d/logs/debug.log | |
| 2024-05-13 20:46:52,818 INFO MainThread:1244577 [wandb_init.py:_log_setup():521] Logging internal logs to /raid/sanchit/parler-tts-mini-v0.1-expresso-concatenated-combined/wandb/run-20240513_204652-m0g0ap7d/logs/debug-internal.log | |
| 2024-05-13 20:46:52,818 INFO MainThread:1244577 [wandb_init.py:init():560] calling init triggers | |
| 2024-05-13 20:46:52,818 INFO MainThread:1244577 [wandb_init.py:init():567] wandb.init called with sweep_config: {} | |
| config: {} | |
| 2024-05-13 20:46:52,818 INFO MainThread:1244577 [wandb_init.py:init():610] starting backend | |
| 2024-05-13 20:46:52,818 INFO MainThread:1244577 [wandb_init.py:init():614] setting up manager | |
| 2024-05-13 20:46:52,821 INFO MainThread:1244577 [backend.py:_multiprocessing_setup():105] multiprocessing start_methods=fork,spawn,forkserver, using: spawn | |
| 2024-05-13 20:46:52,822 INFO MainThread:1244577 [wandb_init.py:init():622] backend started and connected | |
| 2024-05-13 20:46:52,823 INFO MainThread:1244577 [wandb_init.py:init():711] updated telemetry | |
| 2024-05-13 20:46:52,826 INFO MainThread:1244577 [wandb_init.py:init():744] communicating run to backend with 90.0 second timeout | |
| 2024-05-13 20:46:53,228 INFO MainThread:1244577 [wandb_run.py:_on_init():2396] communicating current version | |
| 2024-05-13 20:46:53,287 INFO MainThread:1244577 [wandb_run.py:_on_init():2405] got version response | |
| 2024-05-13 20:46:53,287 INFO MainThread:1244577 [wandb_init.py:init():795] starting run threads in backend | |
| 2024-05-13 20:46:55,863 INFO MainThread:1244577 [wandb_run.py:_console_start():2374] atexit reg | |
| 2024-05-13 20:46:55,863 INFO MainThread:1244577 [wandb_run.py:_redirect():2229] redirect: wrap_raw | |
| 2024-05-13 20:46:55,863 INFO MainThread:1244577 [wandb_run.py:_redirect():2294] Wrapping output streams. | |
| 2024-05-13 20:46:55,863 INFO MainThread:1244577 [wandb_run.py:_redirect():2319] Redirects installed. | |
| 2024-05-13 20:46:55,864 INFO MainThread:1244577 [wandb_init.py:init():838] run started, returning control to user process | |
| 2024-05-13 20:46:55,864 INFO MainThread:1244577 [wandb_run.py:_config_callback():1376] config_cb None None {'learning_rate': 8e-05, 'model_name_or_path': 'parler-tts/parler_tts_mini_v0.1', 'num_train_epochs': 8, 'gradient_accumulation_steps': 8, 'per_device_train_batch_size': 16, 'global_batch_size': 16, 'mixed_precision': 'bf16', 'lr_scheduler_type': 'SchedulerType.COSINE', 'warmup_steps': 250, 'freeze_text_encoder': True, 'max_duration_in_seconds': 30.0, 'weight_decay': 0.01, 'adam_beta1': 0.9, 'adam_beta2': 0.99, 'temperature': 1.0} | |
| 2024-05-13 20:50:42,653 WARNING MsgRouterThr:1244577 [router.py:message_loop():77] message_loop has been closed | |