[2023-01-11 06:14:39,279][18976] Saving configuration to /home/andrew_huggingface_co/sample-factory/train_dir/ant_test/config.json... [2023-01-11 06:14:39,292][18976] Rollout worker 0 uses device cpu [2023-01-11 06:14:39,292][18976] Rollout worker 1 uses device cpu [2023-01-11 06:14:39,293][18976] Rollout worker 2 uses device cpu [2023-01-11 06:14:39,293][18976] Rollout worker 3 uses device cpu [2023-01-11 06:14:39,293][18976] Rollout worker 4 uses device cpu [2023-01-11 06:14:39,293][18976] Rollout worker 5 uses device cpu [2023-01-11 06:14:39,293][18976] Rollout worker 6 uses device cpu [2023-01-11 06:14:39,293][18976] Rollout worker 7 uses device cpu [2023-01-11 06:14:39,293][18976] In synchronous mode, we only accumulate one batch. Setting num_batches_to_accumulate to 1 [2023-01-11 06:14:39,316][18976] Using GPUs [0] for process 0 (actually maps to GPUs [0]) [2023-01-11 06:14:39,316][18976] InferenceWorker_p0-w0: min num requests: 2 [2023-01-11 06:14:39,350][18976] Starting all processes... [2023-01-11 06:14:39,350][18976] Starting process learner_proc0 [2023-01-11 06:14:39,356][18976] Starting all processes... [2023-01-11 06:14:39,362][18976] Starting process inference_proc0-0 [2023-01-11 06:14:39,362][18976] Starting process rollout_proc0 [2023-01-11 06:14:39,363][18976] Starting process rollout_proc1 [2023-01-11 06:14:39,363][18976] Starting process rollout_proc2 [2023-01-11 06:14:39,363][18976] Starting process rollout_proc3 [2023-01-11 06:14:39,363][18976] Starting process rollout_proc4 [2023-01-11 06:14:39,364][18976] Starting process rollout_proc5 [2023-01-11 06:14:39,364][18976] Starting process rollout_proc6 [2023-01-11 06:14:39,364][18976] Starting process rollout_proc7 [2023-01-11 06:14:41,133][19078] Worker 3 uses CPU cores [3] [2023-01-11 06:14:41,149][19080] Worker 4 uses CPU cores [4] [2023-01-11 06:14:41,173][19081] Worker 6 uses CPU cores [6] [2023-01-11 06:14:41,236][19062] Using GPUs [0] for process 0 (actually maps to GPUs [0]) [2023-01-11 06:14:41,236][19062] Set environment var CUDA_VISIBLE_DEVICES to '0' (GPU indices [0]) for learning process 0 [2023-01-11 06:14:41,237][19079] Worker 2 uses CPU cores [2] [2023-01-11 06:14:41,419][19077] Worker 0 uses CPU cores [0] [2023-01-11 06:14:41,431][19076] Worker 1 uses CPU cores [1] [2023-01-11 06:14:41,458][19075] Using GPUs [0] for process 0 (actually maps to GPUs [0]) [2023-01-11 06:14:41,458][19075] Set environment var CUDA_VISIBLE_DEVICES to '0' (GPU indices [0]) for inference process 0 [2023-01-11 06:14:41,495][19082] Worker 5 uses CPU cores [5] [2023-01-11 06:14:41,525][19083] Worker 7 uses CPU cores [7] [2023-01-11 06:14:41,971][19062] Num visible devices: 1 [2023-01-11 06:14:41,971][19075] Num visible devices: 1 [2023-01-11 06:14:42,022][19062] Starting seed is not provided [2023-01-11 06:14:42,022][19062] Using GPUs [0] for process 0 (actually maps to GPUs [0]) [2023-01-11 06:14:42,022][19062] Initializing actor-critic model on device cuda:0 [2023-01-11 06:14:42,023][19062] RunningMeanStd input shape: (27,) [2023-01-11 06:14:42,023][19062] RunningMeanStd input shape: (1,) [2023-01-11 06:14:42,127][19062] Created Actor Critic model with architecture: [2023-01-11 06:14:42,128][19062] ActorCriticSharedWeights( (obs_normalizer): ObservationNormalizer( (running_mean_std): RunningMeanStdDictInPlace( (running_mean_std): ModuleDict( (obs): RunningMeanStdInPlace() ) ) ) (returns_normalizer): RecursiveScriptModule(original_name=RunningMeanStdInPlace) (encoder): MultiInputEncoder( (encoders): ModuleDict( (obs): MlpEncoder( (mlp_head): RecursiveScriptModule( original_name=Sequential (0): RecursiveScriptModule(original_name=Linear) (1): RecursiveScriptModule(original_name=Tanh) (2): RecursiveScriptModule(original_name=Linear) (3): RecursiveScriptModule(original_name=Tanh) ) ) ) ) (core): ModelCoreIdentity() (decoder): MlpDecoder( (mlp): Identity() ) (critic_linear): Linear(in_features=64, out_features=1, bias=True) (action_parameterization): ActionParameterizationContinuousNonAdaptiveStddev( (distribution_linear): Linear(in_features=64, out_features=8, bias=True) ) ) [2023-01-11 06:14:47,316][19062] Using optimizer [2023-01-11 06:14:47,317][19062] No checkpoints found [2023-01-11 06:14:47,317][19062] Did not load from checkpoint, starting from scratch! [2023-01-11 06:14:47,318][19062] Initialized policy 0 weights for model version 0 [2023-01-11 06:14:47,321][19062] LearnerWorker_p0 finished initialization! [2023-01-11 06:14:47,322][19062] Using GPUs [0] for process 0 (actually maps to GPUs [0]) [2023-01-11 06:14:47,429][19075] RunningMeanStd input shape: (27,) [2023-01-11 06:14:47,429][19075] RunningMeanStd input shape: (1,) [2023-01-11 06:14:50,677][18976] Inference worker 0-0 is ready! [2023-01-11 06:14:50,678][18976] All inference workers are ready! Signal rollout workers to start! [2023-01-11 06:14:50,880][19081] Decorrelating experience for 0 frames... [2023-01-11 06:14:50,881][19076] Decorrelating experience for 0 frames... [2023-01-11 06:14:50,882][19081] Decorrelating experience for 64 frames... [2023-01-11 06:14:50,883][19076] Decorrelating experience for 64 frames... [2023-01-11 06:14:50,883][19077] Decorrelating experience for 0 frames... [2023-01-11 06:14:50,884][19080] Decorrelating experience for 0 frames... [2023-01-11 06:14:50,884][19079] Decorrelating experience for 0 frames... [2023-01-11 06:14:50,885][19082] Decorrelating experience for 0 frames... [2023-01-11 06:14:50,885][19078] Decorrelating experience for 0 frames... [2023-01-11 06:14:50,885][19083] Decorrelating experience for 0 frames... [2023-01-11 06:14:50,885][19077] Decorrelating experience for 64 frames... [2023-01-11 06:14:50,886][19080] Decorrelating experience for 64 frames... [2023-01-11 06:14:50,886][19079] Decorrelating experience for 64 frames... [2023-01-11 06:14:50,887][19082] Decorrelating experience for 64 frames... [2023-01-11 06:14:50,887][19078] Decorrelating experience for 64 frames... [2023-01-11 06:14:50,887][19083] Decorrelating experience for 64 frames... [2023-01-11 06:14:50,938][19076] Decorrelating experience for 128 frames... [2023-01-11 06:14:50,939][19081] Decorrelating experience for 128 frames... [2023-01-11 06:14:50,940][19077] Decorrelating experience for 128 frames... [2023-01-11 06:14:50,941][19080] Decorrelating experience for 128 frames... [2023-01-11 06:14:50,943][19083] Decorrelating experience for 128 frames... [2023-01-11 06:14:50,943][19078] Decorrelating experience for 128 frames... [2023-01-11 06:14:50,943][19079] Decorrelating experience for 128 frames... [2023-01-11 06:14:50,944][19082] Decorrelating experience for 128 frames... [2023-01-11 06:14:51,044][19076] Decorrelating experience for 192 frames... [2023-01-11 06:14:51,047][19081] Decorrelating experience for 192 frames... [2023-01-11 06:14:51,047][19077] Decorrelating experience for 192 frames... [2023-01-11 06:14:51,050][19083] Decorrelating experience for 192 frames... [2023-01-11 06:14:51,050][19080] Decorrelating experience for 192 frames... [2023-01-11 06:14:51,051][19078] Decorrelating experience for 192 frames... [2023-01-11 06:14:51,051][19082] Decorrelating experience for 192 frames... [2023-01-11 06:14:51,054][19079] Decorrelating experience for 192 frames... [2023-01-11 06:14:51,235][19076] Decorrelating experience for 256 frames... [2023-01-11 06:14:51,238][19083] Decorrelating experience for 256 frames... [2023-01-11 06:14:51,239][19077] Decorrelating experience for 256 frames... [2023-01-11 06:14:51,242][19078] Decorrelating experience for 256 frames... [2023-01-11 06:14:51,243][19081] Decorrelating experience for 256 frames... [2023-01-11 06:14:51,247][19080] Decorrelating experience for 256 frames... [2023-01-11 06:14:51,248][19082] Decorrelating experience for 256 frames... [2023-01-11 06:14:51,251][19079] Decorrelating experience for 256 frames... [2023-01-11 06:14:51,333][18976] Fps is (10 sec: nan, 60 sec: nan, 300 sec: nan). Total num frames: 0. Throughput: 0: nan. Samples: 0. Policy #0 lag: (min: -1.0, avg: -1.0, max: -1.0) [2023-01-11 06:14:51,335][19062] Saving /home/andrew_huggingface_co/sample-factory/train_dir/ant_test/checkpoint_p0/checkpoint_000000000_0.pth... [2023-01-11 06:14:51,445][19083] Decorrelating experience for 320 frames... [2023-01-11 06:14:51,447][19077] Decorrelating experience for 320 frames... [2023-01-11 06:14:51,450][19076] Decorrelating experience for 320 frames... [2023-01-11 06:14:51,452][19078] Decorrelating experience for 320 frames... [2023-01-11 06:14:51,453][19081] Decorrelating experience for 320 frames... [2023-01-11 06:14:51,457][19082] Decorrelating experience for 320 frames... [2023-01-11 06:14:51,458][19080] Decorrelating experience for 320 frames... [2023-01-11 06:14:51,465][19079] Decorrelating experience for 320 frames... [2023-01-11 06:14:51,706][19083] Decorrelating experience for 384 frames... [2023-01-11 06:14:51,712][19076] Decorrelating experience for 384 frames... [2023-01-11 06:14:51,716][19081] Decorrelating experience for 384 frames... [2023-01-11 06:14:51,717][19078] Decorrelating experience for 384 frames... [2023-01-11 06:14:51,717][19077] Decorrelating experience for 384 frames... [2023-01-11 06:14:51,723][19082] Decorrelating experience for 384 frames... [2023-01-11 06:14:51,724][19080] Decorrelating experience for 384 frames... [2023-01-11 06:14:51,737][19079] Decorrelating experience for 384 frames... [2023-01-11 06:14:52,017][19083] Decorrelating experience for 448 frames... [2023-01-11 06:14:52,029][19076] Decorrelating experience for 448 frames... [2023-01-11 06:14:52,036][19077] Decorrelating experience for 448 frames... [2023-01-11 06:14:52,037][19082] Decorrelating experience for 448 frames... [2023-01-11 06:14:52,038][19081] Decorrelating experience for 448 frames... [2023-01-11 06:14:52,041][19078] Decorrelating experience for 448 frames... [2023-01-11 06:14:52,053][19080] Decorrelating experience for 448 frames... [2023-01-11 06:14:52,064][19079] Decorrelating experience for 448 frames... [2023-01-11 06:14:56,333][18976] Fps is (10 sec: 819.2, 60 sec: 819.2, 300 sec: 819.2). Total num frames: 4096. Throughput: 0: 177.6. Samples: 888. Policy #0 lag: (min: 7.0, avg: 7.0, max: 7.0) [2023-01-11 06:14:56,333][18976] Avg episode reward: [(0, '-126.473')] [2023-01-11 06:14:59,307][18976] Heartbeat connected on Batcher_0 [2023-01-11 06:14:59,311][18976] Heartbeat connected on LearnerWorker_p0 [2023-01-11 06:14:59,322][18976] Heartbeat connected on RolloutWorker_w0 [2023-01-11 06:14:59,326][18976] Heartbeat connected on InferenceWorker_p0-w0 [2023-01-11 06:14:59,331][18976] Heartbeat connected on RolloutWorker_w1 [2023-01-11 06:14:59,335][18976] Heartbeat connected on RolloutWorker_w2 [2023-01-11 06:14:59,340][18976] Heartbeat connected on RolloutWorker_w3 [2023-01-11 06:14:59,341][18976] Heartbeat connected on RolloutWorker_w5 [2023-01-11 06:14:59,341][18976] Heartbeat connected on RolloutWorker_w4 [2023-01-11 06:14:59,351][18976] Heartbeat connected on RolloutWorker_w6 [2023-01-11 06:14:59,351][18976] Heartbeat connected on RolloutWorker_w7 [2023-01-11 06:15:01,333][18976] Fps is (10 sec: 3276.8, 60 sec: 3276.8, 300 sec: 3276.8). Total num frames: 32768. Throughput: 0: 2408.4. Samples: 24084. Policy #0 lag: (min: 4.0, avg: 4.0, max: 4.0) [2023-01-11 06:15:01,333][18976] Avg episode reward: [(0, '-130.992')] [2023-01-11 06:15:03,917][19075] Updated weights for policy 0, policy_version 80 (0.0008) [2023-01-11 06:15:06,333][18976] Fps is (10 sec: 4096.0, 60 sec: 3003.7, 300 sec: 3003.7). Total num frames: 45056. Throughput: 0: 3008.0. Samples: 45120. Policy #0 lag: (min: 4.0, avg: 4.0, max: 4.0) [2023-01-11 06:15:06,334][18976] Avg episode reward: [(0, '-227.535')] [2023-01-11 06:15:06,339][19062] Saving /home/andrew_huggingface_co/sample-factory/train_dir/ant_test/checkpoint_p0/checkpoint_000000088_45056.pth... [2023-01-11 06:15:11,333][18976] Fps is (10 sec: 4096.0, 60 sec: 3686.4, 300 sec: 3686.4). Total num frames: 73728. Throughput: 0: 3020.8. Samples: 60416. Policy #0 lag: (min: 7.0, avg: 7.0, max: 7.0) [2023-01-11 06:15:11,333][18976] Avg episode reward: [(0, '-297.337')] [2023-01-11 06:15:12,628][19075] Updated weights for policy 0, policy_version 160 (0.0007) [2023-01-11 06:15:16,332][18976] Fps is (10 sec: 5324.9, 60 sec: 3932.2, 300 sec: 3932.2). Total num frames: 98304. Throughput: 0: 3737.8. Samples: 93444. Policy #0 lag: (min: 2.0, avg: 2.0, max: 2.0) [2023-01-11 06:15:16,333][18976] Avg episode reward: [(0, '-127.822')] [2023-01-11 06:15:20,209][19075] Updated weights for policy 0, policy_version 240 (0.0007) [2023-01-11 06:15:21,333][18976] Fps is (10 sec: 5324.8, 60 sec: 4232.5, 300 sec: 4232.5). Total num frames: 126976. Throughput: 0: 4195.5. Samples: 125864. Policy #0 lag: (min: 5.0, avg: 5.0, max: 5.0) [2023-01-11 06:15:21,333][18976] Avg episode reward: [(0, '-105.405')] [2023-01-11 06:15:21,340][19062] Saving /home/andrew_huggingface_co/sample-factory/train_dir/ant_test/checkpoint_p0/checkpoint_000000248_126976.pth... [2023-01-11 06:15:21,347][19062] Removing /home/andrew_huggingface_co/sample-factory/train_dir/ant_test/checkpoint_p0/checkpoint_000000000_0.pth [2023-01-11 06:15:21,348][19062] Saving new best policy, reward=-105.405! [2023-01-11 06:15:26,333][18976] Fps is (10 sec: 5734.3, 60 sec: 4447.1, 300 sec: 4447.1). Total num frames: 155648. Throughput: 0: 4065.3. Samples: 142284. Policy #0 lag: (min: 7.0, avg: 7.0, max: 7.0) [2023-01-11 06:15:26,333][18976] Avg episode reward: [(0, '-109.426')] [2023-01-11 06:15:27,645][19075] Updated weights for policy 0, policy_version 320 (0.0007) [2023-01-11 06:15:31,332][18976] Fps is (10 sec: 5324.9, 60 sec: 4505.6, 300 sec: 4505.6). Total num frames: 180224. Throughput: 0: 4384.3. Samples: 175372. Policy #0 lag: (min: 7.0, avg: 7.0, max: 7.0) [2023-01-11 06:15:31,333][18976] Avg episode reward: [(0, '-72.518')] [2023-01-11 06:15:31,333][19062] Saving new best policy, reward=-72.518! [2023-01-11 06:15:35,206][19075] Updated weights for policy 0, policy_version 400 (0.0007) [2023-01-11 06:15:36,333][18976] Fps is (10 sec: 5324.8, 60 sec: 4642.1, 300 sec: 4642.1). Total num frames: 208896. Throughput: 0: 4618.4. Samples: 207828. Policy #0 lag: (min: 7.0, avg: 7.0, max: 7.0) [2023-01-11 06:15:36,333][18976] Avg episode reward: [(0, '-35.778')] [2023-01-11 06:15:36,339][19062] Saving /home/andrew_huggingface_co/sample-factory/train_dir/ant_test/checkpoint_p0/checkpoint_000000408_208896.pth... [2023-01-11 06:15:36,347][19062] Removing /home/andrew_huggingface_co/sample-factory/train_dir/ant_test/checkpoint_p0/checkpoint_000000088_45056.pth [2023-01-11 06:15:36,347][19062] Saving new best policy, reward=-35.778! [2023-01-11 06:15:41,333][18976] Fps is (10 sec: 5734.3, 60 sec: 4751.4, 300 sec: 4751.4). Total num frames: 237568. Throughput: 0: 4961.7. Samples: 224164. Policy #0 lag: (min: 3.0, avg: 3.0, max: 3.0) [2023-01-11 06:15:41,333][18976] Avg episode reward: [(0, '-30.077')] [2023-01-11 06:15:41,334][19062] Saving new best policy, reward=-30.077! [2023-01-11 06:15:42,735][19075] Updated weights for policy 0, policy_version 480 (0.0007) [2023-01-11 06:15:46,333][18976] Fps is (10 sec: 5324.9, 60 sec: 4766.3, 300 sec: 4766.3). Total num frames: 262144. Throughput: 0: 5175.8. Samples: 256996. Policy #0 lag: (min: 3.0, avg: 3.0, max: 3.0) [2023-01-11 06:15:46,333][18976] Avg episode reward: [(0, '-40.853')] [2023-01-11 06:15:50,214][19075] Updated weights for policy 0, policy_version 560 (0.0007) [2023-01-11 06:15:51,333][18976] Fps is (10 sec: 5324.8, 60 sec: 4846.9, 300 sec: 4846.9). Total num frames: 290816. Throughput: 0: 5434.8. Samples: 289688. Policy #0 lag: (min: 7.0, avg: 7.0, max: 7.0) [2023-01-11 06:15:51,333][18976] Avg episode reward: [(0, '-38.381')] [2023-01-11 06:15:51,340][19062] Saving /home/andrew_huggingface_co/sample-factory/train_dir/ant_test/checkpoint_p0/checkpoint_000000568_290816.pth... [2023-01-11 06:15:51,347][19062] Removing /home/andrew_huggingface_co/sample-factory/train_dir/ant_test/checkpoint_p0/checkpoint_000000248_126976.pth [2023-01-11 06:15:56,333][18976] Fps is (10 sec: 5734.3, 60 sec: 5256.5, 300 sec: 4915.2). Total num frames: 319488. Throughput: 0: 5459.1. Samples: 306076. Policy #0 lag: (min: 4.0, avg: 4.0, max: 4.0) [2023-01-11 06:15:56,333][18976] Avg episode reward: [(0, '-20.183')] [2023-01-11 06:15:56,334][19062] Saving new best policy, reward=-20.183! [2023-01-11 06:15:57,590][19075] Updated weights for policy 0, policy_version 640 (0.0006) [2023-01-11 06:16:01,333][18976] Fps is (10 sec: 5324.9, 60 sec: 5188.3, 300 sec: 4915.2). Total num frames: 344064. Throughput: 0: 5477.0. Samples: 339908. Policy #0 lag: (min: 1.0, avg: 1.0, max: 1.0) [2023-01-11 06:16:01,333][18976] Avg episode reward: [(0, '7.282')] [2023-01-11 06:16:01,335][19062] Saving new best policy, reward=7.282! [2023-01-11 06:16:06,333][18976] Fps is (10 sec: 4096.0, 60 sec: 5256.5, 300 sec: 4806.0). Total num frames: 360448. Throughput: 0: 5188.2. Samples: 359332. Policy #0 lag: (min: 4.0, avg: 4.0, max: 4.0) [2023-01-11 06:16:06,333][18976] Avg episode reward: [(0, '20.695')] [2023-01-11 06:16:06,339][19062] Saving /home/andrew_huggingface_co/sample-factory/train_dir/ant_test/checkpoint_p0/checkpoint_000000704_360448.pth... [2023-01-11 06:16:06,346][19062] Removing /home/andrew_huggingface_co/sample-factory/train_dir/ant_test/checkpoint_p0/checkpoint_000000408_208896.pth [2023-01-11 06:16:06,347][19062] Saving new best policy, reward=20.695! [2023-01-11 06:16:07,412][19075] Updated weights for policy 0, policy_version 720 (0.0007) [2023-01-11 06:16:11,332][18976] Fps is (10 sec: 4505.6, 60 sec: 5256.5, 300 sec: 4864.0). Total num frames: 389120. Throughput: 0: 5189.2. Samples: 375796. Policy #0 lag: (min: 7.0, avg: 7.0, max: 7.0) [2023-01-11 06:16:11,333][18976] Avg episode reward: [(0, '42.463')] [2023-01-11 06:16:11,334][19062] Saving new best policy, reward=42.463! [2023-01-11 06:16:14,150][18976] Keyboard interrupt detected in the event loop EvtLoop [Runner_EvtLoop, process=main process 18976], exiting... [2023-01-11 06:16:14,150][18976] Runner profile tree view: main_loop: 94.8007 [2023-01-11 06:16:14,150][19080] Stopping RolloutWorker_w4... [2023-01-11 06:16:14,151][19079] Stopping RolloutWorker_w2... [2023-01-11 06:16:14,151][18976] Collected {0: 401408}, FPS: 4234.2 [2023-01-11 06:16:14,151][19082] Stopping RolloutWorker_w5... [2023-01-11 06:16:14,151][19079] Loop rollout_proc2_evt_loop terminating... [2023-01-11 06:16:14,151][19080] Loop rollout_proc4_evt_loop terminating... [2023-01-11 06:16:14,152][19082] Loop rollout_proc5_evt_loop terminating... [2023-01-11 06:16:14,151][19083] Stopping RolloutWorker_w7... [2023-01-11 06:16:14,151][19076] Stopping RolloutWorker_w1... [2023-01-11 06:16:14,151][19078] Stopping RolloutWorker_w3... [2023-01-11 06:16:14,154][19083] Loop rollout_proc7_evt_loop terminating... [2023-01-11 06:16:14,154][19078] Loop rollout_proc3_evt_loop terminating... [2023-01-11 06:16:14,154][19076] Loop rollout_proc1_evt_loop terminating... [2023-01-11 06:16:14,151][19062] Stopping Batcher_0... [2023-01-11 06:16:14,151][19081] Stopping RolloutWorker_w6... [2023-01-11 06:16:14,151][19077] Stopping RolloutWorker_w0... [2023-01-11 06:16:14,157][19062] Loop batcher_evt_loop terminating... [2023-01-11 06:16:14,157][19077] Loop rollout_proc0_evt_loop terminating... [2023-01-11 06:16:14,157][19081] Loop rollout_proc6_evt_loop terminating... [2023-01-11 06:16:14,211][19075] Weights refcount: 2 0 [2023-01-11 06:16:14,213][19075] Stopping InferenceWorker_p0-w0... [2023-01-11 06:16:14,214][19075] Loop inference_proc0-0_evt_loop terminating... [2023-01-11 06:16:14,224][19062] Saving /home/andrew_huggingface_co/sample-factory/train_dir/ant_test/checkpoint_p0/checkpoint_000000792_405504.pth... [2023-01-11 06:16:14,232][19062] Removing /home/andrew_huggingface_co/sample-factory/train_dir/ant_test/checkpoint_p0/checkpoint_000000568_290816.pth [2023-01-11 06:16:14,233][19062] Stopping LearnerWorker_p0... [2023-01-11 06:16:14,233][19062] Loop learner_proc0_evt_loop terminating...