[2024-01-05 20:02:29,789] [INFO] [real_accelerator.py:161:get_accelerator] Setting ds_accelerator to cuda (auto detect)
[2024-01-05 20:02:34,703] [WARNING] [runner.py:202:fetch_hostfile] Unable to find hostfile, will proceed with training with local resources only.
Detected CUDA_VISIBLE_DEVICES=0,1,2,3 but ignoring it because one or several of --include/--exclude/--num_gpus/--num_nodes cl args were used. If you want to use CUDA_VISIBLE_DEVICES don't pass any of these arguments to deepspeed.
[2024-01-05 20:02:34,703] [INFO] [runner.py:571:main] cmd = /data/jiongxiao_wang/anaconda3/envs/safe-rlhf/bin/python -u -m deepspeed.launcher.launch --world_info=eyJsb2NhbGhvc3QiOiBbMCwgMSwgMiwgM119 --master_addr=127.0.0.1 --master_port=47607 --module --enable_each_rank_log=None safe_rlhf.values.reward --train_datasets PKU-SafeRLHF/train:1.0:PKU-SafeRLHF-harmless-only-30k --eval_datasets PKU-SafeRLHF/test --model_name_or_path output/sft --max_length 512 --trust_remote_code True --loss_type sequence-wise --epochs 2 --per_device_train_batch_size 16 --per_device_eval_batch_size 16 --gradient_accumulation_steps 2 --gradient_checkpointing --normalize_score_during_training False --normalizer_type ExponentialMovingAverage --normalizer_momentum 0.9 --learning_rate 2e-5 --lr_scheduler_type cosine --lr_warmup_ratio 0.03 --weight_decay 0.1 --seed 42 --eval_strategy epoch --output_dir /data/jiongxiao_wang/rlhf_attack/safe-rlhf/output/rm_30k --log_type wandb --log_project Safe-RLHF-RM --zero_stage 3 --bf16 True --tf32 True
[2024-01-05 20:02:36,745] [INFO] [real_accelerator.py:161:get_accelerator] Setting ds_accelerator to cuda (auto detect)
[2024-01-05 20:02:39,776] [INFO] [launch.py:145:main] WORLD INFO DICT: {'localhost': [0, 1, 2, 3]}
[2024-01-05 20:02:39,776] [INFO] [launch.py:151:main] nnodes=1, num_local_procs=4, node_rank=0
[2024-01-05 20:02:39,776] [INFO] [launch.py:162:main] global_rank_mapping=defaultdict(<class 'list'>, {'localhost': [0, 1, 2, 3]})
[2024-01-05 20:02:39,776] [INFO] [launch.py:163:main] dist_world_size=4
[2024-01-05 20:02:39,776] [INFO] [launch.py:165:main] Setting CUDA_VISIBLE_DEVICES=0,1,2,3
[2024-01-05 20:02:42,171] [INFO] [real_accelerator.py:161:get_accelerator] Setting ds_accelerator to cuda (auto detect)
[2024-01-05 20:02:42,172] [INFO] [real_accelerator.py:161:get_accelerator] Setting ds_accelerator to cuda (auto detect)
[2024-01-05 20:02:42,174] [INFO] [real_accelerator.py:161:get_accelerator] Setting ds_accelerator to cuda (auto detect)
[2024-01-05 20:02:42,175] [INFO] [real_accelerator.py:161:get_accelerator] Setting ds_accelerator to cuda (auto detect)
[2024-01-05 20:02:52,431] [INFO] [comm.py:637:init_distributed] cdb=None
[2024-01-05 20:02:52,431] [INFO] [comm.py:668:init_distributed] Initializing TorchBackend in DeepSpeed with backend nccl
[2024-01-05 20:02:52,437] [INFO] [comm.py:637:init_distributed] cdb=None
[2024-01-05 20:02:52,442] [INFO] [comm.py:637:init_distributed] cdb=None
[2024-01-05 20:02:52,443] [INFO] [comm.py:637:init_distributed] cdb=None
Set logger level to WARNING.
ninja: no work to do.
Time to load fused_adam op: 0.13281846046447754 seconds
Time to load fused_adam op: 0.20154452323913574 seconds
Time to load fused_adam op: 0.2013847827911377 seconds
Time to load fused_adam op: 0.2015242576599121 seconds
Parameter Offload: Total persistent parameters: 270336 in 66 params
***** Running training *****
Saving model to "/data/jiongxiao_wang/rlhf_attack/safe-rlhf/output/rm_30k" ...
Saving DeepSpeed Checkpoints...
Converting DeepSpeed Checkpoints to Hugging Face format...
[2024-01-05 20:56:50,901] [INFO] [real_accelerator.py:161:get_accelerator] Setting ds_accelerator to cuda (auto detect)
Processing zero checkpoint './global_step420'
Detected checkpoint of type zero stage 3, world_size: 4
Parsing checkpoint created by deepspeed==0.12.6
Reconstructed Trainable fp32 state dict with 291 params 6607351808 elements
Saving fp32 state dict to pytorch_model.bin
Model saved!
[2024-01-05 20:58:03,159] [INFO] [launch.py:347:main] Process 250383 exits successfully.
[2024-01-05 20:58:03,159] [INFO] [launch.py:347:main] Process 250381 exits successfully.
[2024-01-05 20:58:04,160] [INFO] [launch.py:347:main] Process 250382 exits successfully.
[2024-01-05 20:58:11,167] [INFO] [launch.py:347:main] Process 250380 exits successfully.