| [2023-01-11 06:14:39,279][18976] Saving configuration to /home/andrew_huggingface_co/sample-factory/train_dir/ant_test/config.json... | |
| [2023-01-11 06:14:39,292][18976] Rollout worker 0 uses device cpu | |
| [2023-01-11 06:14:39,292][18976] Rollout worker 1 uses device cpu | |
| [2023-01-11 06:14:39,293][18976] Rollout worker 2 uses device cpu | |
| [2023-01-11 06:14:39,293][18976] Rollout worker 3 uses device cpu | |
| [2023-01-11 06:14:39,293][18976] Rollout worker 4 uses device cpu | |
| [2023-01-11 06:14:39,293][18976] Rollout worker 5 uses device cpu | |
| [2023-01-11 06:14:39,293][18976] Rollout worker 6 uses device cpu | |
| [2023-01-11 06:14:39,293][18976] Rollout worker 7 uses device cpu | |
| [2023-01-11 06:14:39,293][18976] In synchronous mode, we only accumulate one batch. Setting num_batches_to_accumulate to 1 | |
| [2023-01-11 06:14:39,316][18976] Using GPUs [0] for process 0 (actually maps to GPUs [0]) | |
| [2023-01-11 06:14:39,316][18976] InferenceWorker_p0-w0: min num requests: 2 | |
| [2023-01-11 06:14:39,350][18976] Starting all processes... | |
| [2023-01-11 06:14:39,350][18976] Starting process learner_proc0 | |
| [2023-01-11 06:14:39,356][18976] Starting all processes... | |
| [2023-01-11 06:14:39,362][18976] Starting process inference_proc0-0 | |
| [2023-01-11 06:14:39,362][18976] Starting process rollout_proc0 | |
| [2023-01-11 06:14:39,363][18976] Starting process rollout_proc1 | |
| [2023-01-11 06:14:39,363][18976] Starting process rollout_proc2 | |
| [2023-01-11 06:14:39,363][18976] Starting process rollout_proc3 | |
| [2023-01-11 06:14:39,363][18976] Starting process rollout_proc4 | |
| [2023-01-11 06:14:39,364][18976] Starting process rollout_proc5 | |
| [2023-01-11 06:14:39,364][18976] Starting process rollout_proc6 | |
| [2023-01-11 06:14:39,364][18976] Starting process rollout_proc7 | |
| [2023-01-11 06:14:41,133][19078] Worker 3 uses CPU cores [3] | |
| [2023-01-11 06:14:41,149][19080] Worker 4 uses CPU cores [4] | |
| [2023-01-11 06:14:41,173][19081] Worker 6 uses CPU cores [6] | |
| [2023-01-11 06:14:41,236][19062] Using GPUs [0] for process 0 (actually maps to GPUs [0]) | |
| [2023-01-11 06:14:41,236][19062] Set environment var CUDA_VISIBLE_DEVICES to '0' (GPU indices [0]) for learning process 0 | |
| [2023-01-11 06:14:41,237][19079] Worker 2 uses CPU cores [2] | |
| [2023-01-11 06:14:41,419][19077] Worker 0 uses CPU cores [0] | |
| [2023-01-11 06:14:41,431][19076] Worker 1 uses CPU cores [1] | |
| [2023-01-11 06:14:41,458][19075] Using GPUs [0] for process 0 (actually maps to GPUs [0]) | |
| [2023-01-11 06:14:41,458][19075] Set environment var CUDA_VISIBLE_DEVICES to '0' (GPU indices [0]) for inference process 0 | |
| [2023-01-11 06:14:41,495][19082] Worker 5 uses CPU cores [5] | |
| [2023-01-11 06:14:41,525][19083] Worker 7 uses CPU cores [7] | |
| [2023-01-11 06:14:41,971][19062] Num visible devices: 1 | |
| [2023-01-11 06:14:41,971][19075] Num visible devices: 1 | |
| [2023-01-11 06:14:42,022][19062] Starting seed is not provided | |
| [2023-01-11 06:14:42,022][19062] Using GPUs [0] for process 0 (actually maps to GPUs [0]) | |
| [2023-01-11 06:14:42,022][19062] Initializing actor-critic model on device cuda:0 | |
| [2023-01-11 06:14:42,023][19062] RunningMeanStd input shape: (27,) | |
| [2023-01-11 06:14:42,023][19062] RunningMeanStd input shape: (1,) | |
| [2023-01-11 06:14:42,127][19062] Created Actor Critic model with architecture: | |
| [2023-01-11 06:14:42,128][19062] ActorCriticSharedWeights( | |
| (obs_normalizer): ObservationNormalizer( | |
| (running_mean_std): RunningMeanStdDictInPlace( | |
| (running_mean_std): ModuleDict( | |
| (obs): RunningMeanStdInPlace() | |
| ) | |
| ) | |
| ) | |
| (returns_normalizer): RecursiveScriptModule(original_name=RunningMeanStdInPlace) | |
| (encoder): MultiInputEncoder( | |
| (encoders): ModuleDict( | |
| (obs): MlpEncoder( | |
| (mlp_head): RecursiveScriptModule( | |
| original_name=Sequential | |
| (0): RecursiveScriptModule(original_name=Linear) | |
| (1): RecursiveScriptModule(original_name=Tanh) | |
| (2): RecursiveScriptModule(original_name=Linear) | |
| (3): RecursiveScriptModule(original_name=Tanh) | |
| ) | |
| ) | |
| ) | |
| ) | |
| (core): ModelCoreIdentity() | |
| (decoder): MlpDecoder( | |
| (mlp): Identity() | |
| ) | |
| (critic_linear): Linear(in_features=64, out_features=1, bias=True) | |
| (action_parameterization): ActionParameterizationContinuousNonAdaptiveStddev( | |
| (distribution_linear): Linear(in_features=64, out_features=8, bias=True) | |
| ) | |
| ) | |
| [2023-01-11 06:14:47,316][19062] Using optimizer <class 'torch.optim.adam.Adam'> | |
| [2023-01-11 06:14:47,317][19062] No checkpoints found | |
| [2023-01-11 06:14:47,317][19062] Did not load from checkpoint, starting from scratch! | |
| [2023-01-11 06:14:47,318][19062] Initialized policy 0 weights for model version 0 | |
| [2023-01-11 06:14:47,321][19062] LearnerWorker_p0 finished initialization! | |
| [2023-01-11 06:14:47,322][19062] Using GPUs [0] for process 0 (actually maps to GPUs [0]) | |
| [2023-01-11 06:14:47,429][19075] RunningMeanStd input shape: (27,) | |
| [2023-01-11 06:14:47,429][19075] RunningMeanStd input shape: (1,) | |
| [2023-01-11 06:14:50,677][18976] Inference worker 0-0 is ready! | |
| [2023-01-11 06:14:50,678][18976] All inference workers are ready! Signal rollout workers to start! | |
| [2023-01-11 06:14:50,880][19081] Decorrelating experience for 0 frames... | |
| [2023-01-11 06:14:50,881][19076] Decorrelating experience for 0 frames... | |
| [2023-01-11 06:14:50,882][19081] Decorrelating experience for 64 frames... | |
| [2023-01-11 06:14:50,883][19076] Decorrelating experience for 64 frames... | |
| [2023-01-11 06:14:50,883][19077] Decorrelating experience for 0 frames... | |
| [2023-01-11 06:14:50,884][19080] Decorrelating experience for 0 frames... | |
| [2023-01-11 06:14:50,884][19079] Decorrelating experience for 0 frames... | |
| [2023-01-11 06:14:50,885][19082] Decorrelating experience for 0 frames... | |
| [2023-01-11 06:14:50,885][19078] Decorrelating experience for 0 frames... | |
| [2023-01-11 06:14:50,885][19083] Decorrelating experience for 0 frames... | |
| [2023-01-11 06:14:50,885][19077] Decorrelating experience for 64 frames... | |
| [2023-01-11 06:14:50,886][19080] Decorrelating experience for 64 frames... | |
| [2023-01-11 06:14:50,886][19079] Decorrelating experience for 64 frames... | |
| [2023-01-11 06:14:50,887][19082] Decorrelating experience for 64 frames... | |
| [2023-01-11 06:14:50,887][19078] Decorrelating experience for 64 frames... | |
| [2023-01-11 06:14:50,887][19083] Decorrelating experience for 64 frames... | |
| [2023-01-11 06:14:50,938][19076] Decorrelating experience for 128 frames... | |
| [2023-01-11 06:14:50,939][19081] Decorrelating experience for 128 frames... | |
| [2023-01-11 06:14:50,940][19077] Decorrelating experience for 128 frames... | |
| [2023-01-11 06:14:50,941][19080] Decorrelating experience for 128 frames... | |
| [2023-01-11 06:14:50,943][19083] Decorrelating experience for 128 frames... | |
| [2023-01-11 06:14:50,943][19078] Decorrelating experience for 128 frames... | |
| [2023-01-11 06:14:50,943][19079] Decorrelating experience for 128 frames... | |
| [2023-01-11 06:14:50,944][19082] Decorrelating experience for 128 frames... | |
| [2023-01-11 06:14:51,044][19076] Decorrelating experience for 192 frames... | |
| [2023-01-11 06:14:51,047][19081] Decorrelating experience for 192 frames... | |
| [2023-01-11 06:14:51,047][19077] Decorrelating experience for 192 frames... | |
| [2023-01-11 06:14:51,050][19083] Decorrelating experience for 192 frames... | |
| [2023-01-11 06:14:51,050][19080] Decorrelating experience for 192 frames... | |
| [2023-01-11 06:14:51,051][19078] Decorrelating experience for 192 frames... | |
| [2023-01-11 06:14:51,051][19082] Decorrelating experience for 192 frames... | |
| [2023-01-11 06:14:51,054][19079] Decorrelating experience for 192 frames... | |
| [2023-01-11 06:14:51,235][19076] Decorrelating experience for 256 frames... | |
| [2023-01-11 06:14:51,238][19083] Decorrelating experience for 256 frames... | |
| [2023-01-11 06:14:51,239][19077] Decorrelating experience for 256 frames... | |
| [2023-01-11 06:14:51,242][19078] Decorrelating experience for 256 frames... | |
| [2023-01-11 06:14:51,243][19081] Decorrelating experience for 256 frames... | |
| [2023-01-11 06:14:51,247][19080] Decorrelating experience for 256 frames... | |
| [2023-01-11 06:14:51,248][19082] Decorrelating experience for 256 frames... | |
| [2023-01-11 06:14:51,251][19079] Decorrelating experience for 256 frames... | |
| [2023-01-11 06:14:51,333][18976] Fps is (10 sec: nan, 60 sec: nan, 300 sec: nan). Total num frames: 0. Throughput: 0: nan. Samples: 0. Policy #0 lag: (min: -1.0, avg: -1.0, max: -1.0) | |
| [2023-01-11 06:14:51,335][19062] Saving /home/andrew_huggingface_co/sample-factory/train_dir/ant_test/checkpoint_p0/checkpoint_000000000_0.pth... | |
| [2023-01-11 06:14:51,445][19083] Decorrelating experience for 320 frames... | |
| [2023-01-11 06:14:51,447][19077] Decorrelating experience for 320 frames... | |
| [2023-01-11 06:14:51,450][19076] Decorrelating experience for 320 frames... | |
| [2023-01-11 06:14:51,452][19078] Decorrelating experience for 320 frames... | |
| [2023-01-11 06:14:51,453][19081] Decorrelating experience for 320 frames... | |
| [2023-01-11 06:14:51,457][19082] Decorrelating experience for 320 frames... | |
| [2023-01-11 06:14:51,458][19080] Decorrelating experience for 320 frames... | |
| [2023-01-11 06:14:51,465][19079] Decorrelating experience for 320 frames... | |
| [2023-01-11 06:14:51,706][19083] Decorrelating experience for 384 frames... | |
| [2023-01-11 06:14:51,712][19076] Decorrelating experience for 384 frames... | |
| [2023-01-11 06:14:51,716][19081] Decorrelating experience for 384 frames... | |
| [2023-01-11 06:14:51,717][19078] Decorrelating experience for 384 frames... | |
| [2023-01-11 06:14:51,717][19077] Decorrelating experience for 384 frames... | |
| [2023-01-11 06:14:51,723][19082] Decorrelating experience for 384 frames... | |
| [2023-01-11 06:14:51,724][19080] Decorrelating experience for 384 frames... | |
| [2023-01-11 06:14:51,737][19079] Decorrelating experience for 384 frames... | |
| [2023-01-11 06:14:52,017][19083] Decorrelating experience for 448 frames... | |
| [2023-01-11 06:14:52,029][19076] Decorrelating experience for 448 frames... | |
| [2023-01-11 06:14:52,036][19077] Decorrelating experience for 448 frames... | |
| [2023-01-11 06:14:52,037][19082] Decorrelating experience for 448 frames... | |
| [2023-01-11 06:14:52,038][19081] Decorrelating experience for 448 frames... | |
| [2023-01-11 06:14:52,041][19078] Decorrelating experience for 448 frames... | |
| [2023-01-11 06:14:52,053][19080] Decorrelating experience for 448 frames... | |
| [2023-01-11 06:14:52,064][19079] Decorrelating experience for 448 frames... | |
| [2023-01-11 06:14:56,333][18976] Fps is (10 sec: 819.2, 60 sec: 819.2, 300 sec: 819.2). Total num frames: 4096. Throughput: 0: 177.6. Samples: 888. Policy #0 lag: (min: 7.0, avg: 7.0, max: 7.0) | |
| [2023-01-11 06:14:56,333][18976] Avg episode reward: [(0, '-126.473')] | |
| [2023-01-11 06:14:59,307][18976] Heartbeat connected on Batcher_0 | |
| [2023-01-11 06:14:59,311][18976] Heartbeat connected on LearnerWorker_p0 | |
| [2023-01-11 06:14:59,322][18976] Heartbeat connected on RolloutWorker_w0 | |
| [2023-01-11 06:14:59,326][18976] Heartbeat connected on InferenceWorker_p0-w0 | |
| [2023-01-11 06:14:59,331][18976] Heartbeat connected on RolloutWorker_w1 | |
| [2023-01-11 06:14:59,335][18976] Heartbeat connected on RolloutWorker_w2 | |
| [2023-01-11 06:14:59,340][18976] Heartbeat connected on RolloutWorker_w3 | |
| [2023-01-11 06:14:59,341][18976] Heartbeat connected on RolloutWorker_w5 | |
| [2023-01-11 06:14:59,341][18976] Heartbeat connected on RolloutWorker_w4 | |
| [2023-01-11 06:14:59,351][18976] Heartbeat connected on RolloutWorker_w6 | |
| [2023-01-11 06:14:59,351][18976] Heartbeat connected on RolloutWorker_w7 | |
| [2023-01-11 06:15:01,333][18976] Fps is (10 sec: 3276.8, 60 sec: 3276.8, 300 sec: 3276.8). Total num frames: 32768. Throughput: 0: 2408.4. Samples: 24084. Policy #0 lag: (min: 4.0, avg: 4.0, max: 4.0) | |
| [2023-01-11 06:15:01,333][18976] Avg episode reward: [(0, '-130.992')] | |
| [2023-01-11 06:15:03,917][19075] Updated weights for policy 0, policy_version 80 (0.0008) | |
| [2023-01-11 06:15:06,333][18976] Fps is (10 sec: 4096.0, 60 sec: 3003.7, 300 sec: 3003.7). Total num frames: 45056. Throughput: 0: 3008.0. Samples: 45120. Policy #0 lag: (min: 4.0, avg: 4.0, max: 4.0) | |
| [2023-01-11 06:15:06,334][18976] Avg episode reward: [(0, '-227.535')] | |
| [2023-01-11 06:15:06,339][19062] Saving /home/andrew_huggingface_co/sample-factory/train_dir/ant_test/checkpoint_p0/checkpoint_000000088_45056.pth... | |
| [2023-01-11 06:15:11,333][18976] Fps is (10 sec: 4096.0, 60 sec: 3686.4, 300 sec: 3686.4). Total num frames: 73728. Throughput: 0: 3020.8. Samples: 60416. Policy #0 lag: (min: 7.0, avg: 7.0, max: 7.0) | |
| [2023-01-11 06:15:11,333][18976] Avg episode reward: [(0, '-297.337')] | |
| [2023-01-11 06:15:12,628][19075] Updated weights for policy 0, policy_version 160 (0.0007) | |
| [2023-01-11 06:15:16,332][18976] Fps is (10 sec: 5324.9, 60 sec: 3932.2, 300 sec: 3932.2). Total num frames: 98304. Throughput: 0: 3737.8. Samples: 93444. Policy #0 lag: (min: 2.0, avg: 2.0, max: 2.0) | |
| [2023-01-11 06:15:16,333][18976] Avg episode reward: [(0, '-127.822')] | |
| [2023-01-11 06:15:20,209][19075] Updated weights for policy 0, policy_version 240 (0.0007) | |
| [2023-01-11 06:15:21,333][18976] Fps is (10 sec: 5324.8, 60 sec: 4232.5, 300 sec: 4232.5). Total num frames: 126976. Throughput: 0: 4195.5. Samples: 125864. Policy #0 lag: (min: 5.0, avg: 5.0, max: 5.0) | |
| [2023-01-11 06:15:21,333][18976] Avg episode reward: [(0, '-105.405')] | |
| [2023-01-11 06:15:21,340][19062] Saving /home/andrew_huggingface_co/sample-factory/train_dir/ant_test/checkpoint_p0/checkpoint_000000248_126976.pth... | |
| [2023-01-11 06:15:21,347][19062] Removing /home/andrew_huggingface_co/sample-factory/train_dir/ant_test/checkpoint_p0/checkpoint_000000000_0.pth | |
| [2023-01-11 06:15:21,348][19062] Saving new best policy, reward=-105.405! | |
| [2023-01-11 06:15:26,333][18976] Fps is (10 sec: 5734.3, 60 sec: 4447.1, 300 sec: 4447.1). Total num frames: 155648. Throughput: 0: 4065.3. Samples: 142284. Policy #0 lag: (min: 7.0, avg: 7.0, max: 7.0) | |
| [2023-01-11 06:15:26,333][18976] Avg episode reward: [(0, '-109.426')] | |
| [2023-01-11 06:15:27,645][19075] Updated weights for policy 0, policy_version 320 (0.0007) | |
| [2023-01-11 06:15:31,332][18976] Fps is (10 sec: 5324.9, 60 sec: 4505.6, 300 sec: 4505.6). Total num frames: 180224. Throughput: 0: 4384.3. Samples: 175372. Policy #0 lag: (min: 7.0, avg: 7.0, max: 7.0) | |
| [2023-01-11 06:15:31,333][18976] Avg episode reward: [(0, '-72.518')] | |
| [2023-01-11 06:15:31,333][19062] Saving new best policy, reward=-72.518! | |
| [2023-01-11 06:15:35,206][19075] Updated weights for policy 0, policy_version 400 (0.0007) | |
| [2023-01-11 06:15:36,333][18976] Fps is (10 sec: 5324.8, 60 sec: 4642.1, 300 sec: 4642.1). Total num frames: 208896. Throughput: 0: 4618.4. Samples: 207828. Policy #0 lag: (min: 7.0, avg: 7.0, max: 7.0) | |
| [2023-01-11 06:15:36,333][18976] Avg episode reward: [(0, '-35.778')] | |
| [2023-01-11 06:15:36,339][19062] Saving /home/andrew_huggingface_co/sample-factory/train_dir/ant_test/checkpoint_p0/checkpoint_000000408_208896.pth... | |
| [2023-01-11 06:15:36,347][19062] Removing /home/andrew_huggingface_co/sample-factory/train_dir/ant_test/checkpoint_p0/checkpoint_000000088_45056.pth | |
| [2023-01-11 06:15:36,347][19062] Saving new best policy, reward=-35.778! | |
| [2023-01-11 06:15:41,333][18976] Fps is (10 sec: 5734.3, 60 sec: 4751.4, 300 sec: 4751.4). Total num frames: 237568. Throughput: 0: 4961.7. Samples: 224164. Policy #0 lag: (min: 3.0, avg: 3.0, max: 3.0) | |
| [2023-01-11 06:15:41,333][18976] Avg episode reward: [(0, '-30.077')] | |
| [2023-01-11 06:15:41,334][19062] Saving new best policy, reward=-30.077! | |
| [2023-01-11 06:15:42,735][19075] Updated weights for policy 0, policy_version 480 (0.0007) | |
| [2023-01-11 06:15:46,333][18976] Fps is (10 sec: 5324.9, 60 sec: 4766.3, 300 sec: 4766.3). Total num frames: 262144. Throughput: 0: 5175.8. Samples: 256996. Policy #0 lag: (min: 3.0, avg: 3.0, max: 3.0) | |
| [2023-01-11 06:15:46,333][18976] Avg episode reward: [(0, '-40.853')] | |
| [2023-01-11 06:15:50,214][19075] Updated weights for policy 0, policy_version 560 (0.0007) | |
| [2023-01-11 06:15:51,333][18976] Fps is (10 sec: 5324.8, 60 sec: 4846.9, 300 sec: 4846.9). Total num frames: 290816. Throughput: 0: 5434.8. Samples: 289688. Policy #0 lag: (min: 7.0, avg: 7.0, max: 7.0) | |
| [2023-01-11 06:15:51,333][18976] Avg episode reward: [(0, '-38.381')] | |
| [2023-01-11 06:15:51,340][19062] Saving /home/andrew_huggingface_co/sample-factory/train_dir/ant_test/checkpoint_p0/checkpoint_000000568_290816.pth... | |
| [2023-01-11 06:15:51,347][19062] Removing /home/andrew_huggingface_co/sample-factory/train_dir/ant_test/checkpoint_p0/checkpoint_000000248_126976.pth | |
| [2023-01-11 06:15:56,333][18976] Fps is (10 sec: 5734.3, 60 sec: 5256.5, 300 sec: 4915.2). Total num frames: 319488. Throughput: 0: 5459.1. Samples: 306076. Policy #0 lag: (min: 4.0, avg: 4.0, max: 4.0) | |
| [2023-01-11 06:15:56,333][18976] Avg episode reward: [(0, '-20.183')] | |
| [2023-01-11 06:15:56,334][19062] Saving new best policy, reward=-20.183! | |
| [2023-01-11 06:15:57,590][19075] Updated weights for policy 0, policy_version 640 (0.0006) | |
| [2023-01-11 06:16:01,333][18976] Fps is (10 sec: 5324.9, 60 sec: 5188.3, 300 sec: 4915.2). Total num frames: 344064. Throughput: 0: 5477.0. Samples: 339908. Policy #0 lag: (min: 1.0, avg: 1.0, max: 1.0) | |
| [2023-01-11 06:16:01,333][18976] Avg episode reward: [(0, '7.282')] | |
| [2023-01-11 06:16:01,335][19062] Saving new best policy, reward=7.282! | |
| [2023-01-11 06:16:06,333][18976] Fps is (10 sec: 4096.0, 60 sec: 5256.5, 300 sec: 4806.0). Total num frames: 360448. Throughput: 0: 5188.2. Samples: 359332. Policy #0 lag: (min: 4.0, avg: 4.0, max: 4.0) | |
| [2023-01-11 06:16:06,333][18976] Avg episode reward: [(0, '20.695')] | |
| [2023-01-11 06:16:06,339][19062] Saving /home/andrew_huggingface_co/sample-factory/train_dir/ant_test/checkpoint_p0/checkpoint_000000704_360448.pth... | |
| [2023-01-11 06:16:06,346][19062] Removing /home/andrew_huggingface_co/sample-factory/train_dir/ant_test/checkpoint_p0/checkpoint_000000408_208896.pth | |
| [2023-01-11 06:16:06,347][19062] Saving new best policy, reward=20.695! | |
| [2023-01-11 06:16:07,412][19075] Updated weights for policy 0, policy_version 720 (0.0007) | |
| [2023-01-11 06:16:11,332][18976] Fps is (10 sec: 4505.6, 60 sec: 5256.5, 300 sec: 4864.0). Total num frames: 389120. Throughput: 0: 5189.2. Samples: 375796. Policy #0 lag: (min: 7.0, avg: 7.0, max: 7.0) | |
| [2023-01-11 06:16:11,333][18976] Avg episode reward: [(0, '42.463')] | |
| [2023-01-11 06:16:11,334][19062] Saving new best policy, reward=42.463! | |
| [2023-01-11 06:16:14,150][18976] Keyboard interrupt detected in the event loop EvtLoop [Runner_EvtLoop, process=main process 18976], exiting... | |
| [2023-01-11 06:16:14,150][18976] Runner profile tree view: | |
| main_loop: 94.8007 | |
| [2023-01-11 06:16:14,150][19080] Stopping RolloutWorker_w4... | |
| [2023-01-11 06:16:14,151][19079] Stopping RolloutWorker_w2... | |
| [2023-01-11 06:16:14,151][18976] Collected {0: 401408}, FPS: 4234.2 | |
| [2023-01-11 06:16:14,151][19082] Stopping RolloutWorker_w5... | |
| [2023-01-11 06:16:14,151][19079] Loop rollout_proc2_evt_loop terminating... | |
| [2023-01-11 06:16:14,151][19080] Loop rollout_proc4_evt_loop terminating... | |
| [2023-01-11 06:16:14,152][19082] Loop rollout_proc5_evt_loop terminating... | |
| [2023-01-11 06:16:14,151][19083] Stopping RolloutWorker_w7... | |
| [2023-01-11 06:16:14,151][19076] Stopping RolloutWorker_w1... | |
| [2023-01-11 06:16:14,151][19078] Stopping RolloutWorker_w3... | |
| [2023-01-11 06:16:14,154][19083] Loop rollout_proc7_evt_loop terminating... | |
| [2023-01-11 06:16:14,154][19078] Loop rollout_proc3_evt_loop terminating... | |
| [2023-01-11 06:16:14,154][19076] Loop rollout_proc1_evt_loop terminating... | |
| [2023-01-11 06:16:14,151][19062] Stopping Batcher_0... | |
| [2023-01-11 06:16:14,151][19081] Stopping RolloutWorker_w6... | |
| [2023-01-11 06:16:14,151][19077] Stopping RolloutWorker_w0... | |
| [2023-01-11 06:16:14,157][19062] Loop batcher_evt_loop terminating... | |
| [2023-01-11 06:16:14,157][19077] Loop rollout_proc0_evt_loop terminating... | |
| [2023-01-11 06:16:14,157][19081] Loop rollout_proc6_evt_loop terminating... | |
| [2023-01-11 06:16:14,211][19075] Weights refcount: 2 0 | |
| [2023-01-11 06:16:14,213][19075] Stopping InferenceWorker_p0-w0... | |
| [2023-01-11 06:16:14,214][19075] Loop inference_proc0-0_evt_loop terminating... | |
| [2023-01-11 06:16:14,224][19062] Saving /home/andrew_huggingface_co/sample-factory/train_dir/ant_test/checkpoint_p0/checkpoint_000000792_405504.pth... | |
| [2023-01-11 06:16:14,232][19062] Removing /home/andrew_huggingface_co/sample-factory/train_dir/ant_test/checkpoint_p0/checkpoint_000000568_290816.pth | |
| [2023-01-11 06:16:14,233][19062] Stopping LearnerWorker_p0... | |
| [2023-01-11 06:16:14,233][19062] Loop learner_proc0_evt_loop terminating... | |