| 2024-08-07 19:01:43,343 - INFO: Calling run.. | |
| 2024-08-07 19:01:43,344 - INFO: Environment configuration: ConfigNLPCausalClassificationEnvironment(gpus=['0'], mixed_precision=False, compile_model=False, use_deepspeed=False, deepspeed_reduce_bucket_size=10000000.0, deepspeed_stage3_prefetch_bucket_size=10000000.0, deepspeed_stage3_param_persistence_threshold=10000000.0, deepspeed_offload_optimizer=False, deepspeed_stage3_max_live_parameters=10000000.0, deepspeed_stage3_max_reuse_distance=10000000.0, find_unused_parameters=False, trust_remote_code=False, huggingface_branch='main', number_of_workers=8, seed=-1, _seed=0, _distributed=False, _distributed_inference=True, _local_rank=0, _world_size=1, _curr_step=0, _curr_val_step=0, _rank=0, _device='cuda', _cpu_comm=None, _model_card_template='text_causal_classification_model_card_template.md', _summary_card_template='text_causal_classification_experiment_summary_card_template.md') | |
| 2024-08-07 19:01:43,344 - INFO: cfg.environment._distributed set to False | |
| 2024-08-07 19:01:43,345 - INFO: Problem Type: text_causal_classification_modeling | |
| 2024-08-07 19:01:43,345 - INFO: Global random seed: 288462 | |
| 2024-08-07 19:01:43,345 - INFO: Preparing the data... | |
| 2024-08-07 19:01:43,346 - INFO: Setting up automatic validation split... | |
| 2024-08-07 19:01:43,381 - INFO: The dataframe has following columns: Index(['Description', 'category', 'sub_category', 'label'], dtype='object') | |
| 2024-08-07 19:01:43,387 - INFO: Preparing train and validation data, dataset config to be used: ConfigNLPCausalClassificationDataset(dataset_class=<class 'llm_studio.src.datasets.text_causal_classification_ds.CustomDataset'>, personalize=False, chatbot_name='OI_AI', chatbot_author='openinnovation.ai', train_dataframe='/app/train_df.csv', validation_strategy='automatic', validation_dataframe='', validation_size=0.2, data_sample=1.0, data_sample_choice=('Train', 'Validation'), system_column='None', prompt_column=('Description',), answer_column='category', parent_id_column='None', text_system_start='', text_prompt_start='', text_answer_separator='', limit_chained_samples=False, add_eos_token_to_system=False, add_eos_token_to_prompt=False, add_eos_token_to_answer=False, mask_prompt_labels=True, _allowed_file_extensions=('csv', 'pq', 'parquet'), num_classes=10) | |
| 2024-08-07 19:01:43,387 - INFO: Loading train dataset... | |
| 2024-08-07 19:01:43,387 - INFO: Columns found: Index(['Description', 'category', 'sub_category', 'label'], dtype='object') | |
| 2024-08-07 19:01:44,557 - INFO: Loading validation dataset... | |
| 2024-08-07 19:01:44,706 - INFO: Number of observations in train dataset: 9600 | |
| 2024-08-07 19:01:44,706 - INFO: Number of observations in validation dataset: 2400 | |
| 2024-08-07 19:01:44,965 - WARNING: PAD token id not matching between config and tokenizer. Overwriting with tokenizer id. | |
| 2024-08-07 19:01:44,973 - INFO: Using float16 for backbone | |
| 2024-08-07 19:01:44,973 - INFO: Loading tiiuae/falcon-7b. This may take a while. | |
| 2024-08-07 19:03:58,808 - INFO: Loaded tiiuae/falcon-7b. | |
| 2024-08-07 19:03:58,810 - WARNING: PAD token id not matching between generation config and tokenizer. Overwriting with tokenizer id. | |
| 2024-08-07 19:03:58,810 - INFO: Lora module names: ['query_key_value', 'dense', 'dense_h_to_4h', 'dense_4h_to_h'] | |
| 2024-08-07 19:03:58,943 - INFO: Enough space available for saving model weights.Required space: 13632.81MB, Available space: 368261.23MB. | |
| 2024-08-07 19:03:58,947 - INFO: Optimizer AdamW has been provided with parameters {'weight_decay': 0.0, 'eps': 1e-08, 'betas': (0.8999999762, 0.9990000129), 'lr': 0.0001} | |
| 2024-08-07 19:03:59,367 - INFO: started process: 0, can_track: True, tracking_mode: TrackingMode.AFTER_EPOCH | |
| 2024-08-07 19:03:59,368 - INFO: Training Epoch: 1 / 10 | |
| 2024-08-07 19:03:59,369 - INFO: train loss: 0%| | 0/300 [00:00<?, ?it/s] | |
| 2024-08-07 19:03:59,533 - INFO: Evaluation step: 300 | |
| 2024-08-07 19:04:21,347 - INFO: train loss: 42.78: 1%|1 | 3/300 [00:21<36:15, 7.33s/it] | |
| 2024-08-07 19:04:26,648 - INFO: train loss: 66.63: 2%|2 | 6/300 [00:27<19:52, 4.06s/it] | |
| 2024-08-07 19:04:31,837 - INFO: train loss: 88.90: 3%|3 | 9/300 [00:32<14:31, 2.99s/it] | |
| 2024-08-07 19:04:36,782 - INFO: train loss: 92.28: 4%|4 | 12/300 [00:37<11:49, 2.46s/it] | |
| 2024-08-07 19:04:41,794 - INFO: train loss: 76.47: 5%|5 | 15/300 [00:42<10:20, 2.18s/it] | |
| 2024-08-07 19:04:47,149 - INFO: train loss: 43.56: 6%|6 | 18/300 [00:47<09:36, 2.04s/it] | |
| 2024-08-07 19:04:51,908 - INFO: train loss: 18.78: 7%|7 | 21/300 [00:52<08:48, 1.89s/it] | |
| 2024-08-07 19:04:56,631 - INFO: train loss: 8.20: 8%|8 | 24/300 [00:57<08:14, 1.79s/it] | |
| 2024-08-07 19:05:01,294 - INFO: train loss: 5.85: 9%|9 | 27/300 [01:01<07:48, 1.72s/it] | |
| 2024-08-07 19:05:06,377 - INFO: train loss: 3.56: 10%|# | 30/300 [01:07<07:41, 1.71s/it] | |
| 2024-08-07 19:05:11,288 - INFO: train loss: 3.24: 11%|#1 | 33/300 [01:11<07:30, 1.69s/it] | |
| 2024-08-07 19:05:16,310 - INFO: train loss: 3.20: 12%|#2 | 36/300 [01:16<07:24, 1.68s/it] | |
| 2024-08-07 19:05:21,802 - INFO: train loss: 2.91: 13%|#3 | 39/300 [01:22<07:31, 1.73s/it] | |
| 2024-08-07 19:05:26,717 - INFO: train loss: 2.94: 14%|#4 | 42/300 [01:27<07:18, 1.70s/it] | |
| 2024-08-07 19:05:31,630 - INFO: train loss: 2.52: 15%|#5 | 45/300 [01:32<07:08, 1.68s/it] | |
| 2024-08-07 19:05:36,474 - INFO: train loss: 2.30: 16%|#6 | 48/300 [01:37<06:58, 1.66s/it] | |
| 2024-08-07 19:05:41,478 - INFO: train loss: 1.99: 17%|#7 | 51/300 [01:42<06:54, 1.66s/it] | |
| 2024-08-07 19:05:46,191 - INFO: train loss: 1.84: 18%|#8 | 54/300 [01:46<06:42, 1.64s/it] | |
| 2024-08-07 19:05:51,465 - INFO: train loss: 1.82: 19%|#9 | 57/300 [01:52<06:46, 1.67s/it] | |
| 2024-08-07 19:05:56,388 - INFO: train loss: 2.09: 20%|## | 60/300 [01:57<06:39, 1.66s/it] | |
| 2024-08-07 19:06:01,888 - INFO: train loss: 2.33: 21%|##1 | 63/300 [02:02<06:46, 1.71s/it] | |
| 2024-08-07 19:06:06,594 - INFO: train loss: 2.30: 22%|##2 | 66/300 [02:07<06:30, 1.67s/it] | |
| 2024-08-07 19:06:11,394 - INFO: train loss: 2.12: 23%|##3 | 69/300 [02:12<06:20, 1.65s/it] | |
| 2024-08-07 19:06:16,189 - INFO: train loss: 1.77: 24%|##4 | 72/300 [02:16<06:12, 1.63s/it] | |
| 2024-08-07 19:06:20,994 - INFO: train loss: 1.69: 25%|##5 | 75/300 [02:21<06:05, 1.62s/it] | |
| 2024-08-07 19:06:25,706 - INFO: train loss: 1.54: 26%|##6 | 78/300 [02:26<05:57, 1.61s/it] | |
| 2024-08-07 19:06:30,764 - INFO: train loss: 1.55: 27%|##7 | 81/300 [02:31<05:57, 1.63s/it] | |
| 2024-08-07 19:06:35,446 - INFO: train loss: 1.55: 28%|##8 | 84/300 [02:36<05:47, 1.61s/it] | |
| 2024-08-07 19:06:40,842 - INFO: train loss: 1.58: 29%|##9 | 87/300 [02:41<05:55, 1.67s/it] | |
| 2024-08-07 19:06:45,584 - INFO: train loss: 1.60: 30%|### | 90/300 [02:46<05:44, 1.64s/it] | |
| 2024-08-07 19:06:50,936 - INFO: train loss: 1.95: 31%|###1 | 93/300 [02:51<05:48, 1.68s/it] | |
| 2024-08-07 19:06:55,947 - INFO: train loss: 1.98: 32%|###2 | 96/300 [02:56<05:42, 1.68s/it] | |
| 2024-08-07 19:07:01,137 - INFO: train loss: 2.24: 33%|###3 | 99/300 [03:01<05:40, 1.69s/it] | |
| 2024-08-07 19:07:05,556 - INFO: train loss: 1.71: 34%|###4 | 102/300 [03:06<05:22, 1.63s/it] | |
| 2024-08-07 19:07:10,290 - INFO: train loss: 1.80: 35%|###5 | 105/300 [03:10<05:14, 1.61s/it] | |
| 2024-08-07 19:07:15,450 - INFO: train loss: 1.43: 36%|###6 | 108/300 [03:16<05:15, 1.65s/it] | |
| 2024-08-07 19:07:20,182 - INFO: train loss: 1.65: 37%|###7 | 111/300 [03:20<05:07, 1.62s/it] | |
| 2024-08-07 19:07:25,095 - INFO: train loss: 1.56: 38%|###8 | 114/300 [03:25<05:02, 1.63s/it] | |
| 2024-08-07 19:07:30,293 - INFO: train loss: 1.41: 39%|###9 | 117/300 [03:30<05:03, 1.66s/it] | |
| 2024-08-07 19:07:35,139 - INFO: train loss: 1.21: 40%|#### | 120/300 [03:35<04:56, 1.65s/it] | |
| 2024-08-07 19:07:40,126 - INFO: train loss: 1.03: 41%|####1 | 123/300 [03:40<04:52, 1.65s/it] | |
| 2024-08-07 19:07:45,181 - INFO: train loss: 1.09: 42%|####2 | 126/300 [03:45<04:49, 1.66s/it] | |
| 2024-08-07 19:07:50,224 - INFO: train loss: 1.01: 43%|####3 | 129/300 [03:50<04:45, 1.67s/it] | |
| 2024-08-07 19:07:55,119 - INFO: train loss: 1.06: 44%|####4 | 132/300 [03:55<04:38, 1.66s/it] | |
| 2024-08-07 19:08:00,282 - INFO: train loss: 0.90: 45%|####5 | 135/300 [04:00<04:36, 1.68s/it] | |
| 2024-08-07 19:08:05,376 - INFO: train loss: 0.78: 46%|####6 | 138/300 [04:06<04:32, 1.68s/it] | |
| 2024-08-07 19:08:10,200 - INFO: train loss: 0.68: 47%|####6 | 141/300 [04:10<04:23, 1.66s/it] | |
| 2024-08-07 19:08:15,038 - INFO: train loss: 0.77: 48%|####8 | 144/300 [04:15<04:16, 1.65s/it] | |
| 2024-08-07 19:08:20,027 - INFO: train loss: 1.08: 49%|####9 | 147/300 [04:20<04:12, 1.65s/it] | |
| 2024-08-07 19:08:25,059 - INFO: train loss: 1.07: 50%|##### | 150/300 [04:25<04:08, 1.66s/it] | |
| 2024-08-07 19:08:30,150 - INFO: train loss: 1.16: 51%|#####1 | 153/300 [04:30<04:05, 1.67s/it] | |
| 2024-08-07 19:08:35,214 - INFO: train loss: 0.99: 52%|#####2 | 156/300 [04:35<04:01, 1.68s/it] | |
| 2024-08-07 19:08:40,371 - INFO: train loss: 0.99: 53%|#####3 | 159/300 [04:41<03:58, 1.69s/it] | |
| 2024-08-07 19:08:45,405 - INFO: train loss: 1.09: 54%|#####4 | 162/300 [04:46<03:52, 1.69s/it] | |
| 2024-08-07 19:08:50,067 - INFO: train loss: 0.97: 55%|#####5 | 165/300 [04:50<03:42, 1.65s/it] | |
| 2024-08-07 19:08:55,289 - INFO: train loss: 1.05: 56%|#####6 | 168/300 [04:55<03:41, 1.67s/it] | |
| 2024-08-07 19:09:00,586 - INFO: train loss: 0.80: 57%|#####6 | 171/300 [05:01<03:39, 1.70s/it] | |
| 2024-08-07 19:09:05,547 - INFO: train loss: 0.78: 58%|#####8 | 174/300 [05:06<03:32, 1.69s/it] | |
| 2024-08-07 19:09:10,638 - INFO: train loss: 0.64: 59%|#####8 | 177/300 [05:11<03:27, 1.69s/it] | |
| 2024-08-07 19:09:15,498 - INFO: train loss: 0.63: 60%|###### | 180/300 [05:16<03:20, 1.67s/it] | |
| 2024-08-07 19:09:20,402 - INFO: train loss: 0.63: 61%|######1 | 183/300 [05:21<03:14, 1.66s/it] | |
| 2024-08-07 19:09:26,507 - INFO: train loss: 0.70: 62%|######2 | 186/300 [05:27<03:21, 1.77s/it] | |
| 2024-08-07 19:09:31,511 - INFO: train loss: 0.79: 63%|######3 | 189/300 [05:32<03:13, 1.74s/it] | |
| 2024-08-07 19:09:36,109 - INFO: train loss: 0.87: 64%|######4 | 192/300 [05:36<03:01, 1.68s/it] | |
| 2024-08-07 19:09:41,658 - INFO: train loss: 0.68: 65%|######5 | 195/300 [05:42<03:01, 1.73s/it] | |
| 2024-08-07 19:09:46,689 - INFO: train loss: 0.87: 66%|######6 | 198/300 [05:47<02:54, 1.71s/it] | |
| 2024-08-07 19:09:51,832 - INFO: train loss: 0.92: 67%|######7 | 201/300 [05:52<02:49, 1.71s/it] | |
| 2024-08-07 19:09:56,511 - INFO: train loss: 1.01: 68%|######8 | 204/300 [05:57<02:40, 1.67s/it] | |
| 2024-08-07 19:10:01,305 - INFO: train loss: 1.04: 69%|######9 | 207/300 [06:01<02:33, 1.65s/it] | |
| 2024-08-07 19:10:06,511 - INFO: train loss: 0.90: 70%|####### | 210/300 [06:07<02:30, 1.67s/it] | |
| 2024-08-07 19:10:11,465 - INFO: train loss: 0.80: 71%|#######1 | 213/300 [06:12<02:25, 1.67s/it] | |
| 2024-08-07 19:10:16,589 - INFO: train loss: 0.80: 72%|#######2 | 216/300 [06:17<02:21, 1.68s/it] | |
| 2024-08-07 19:10:21,452 - INFO: train loss: 0.82: 73%|#######3 | 219/300 [06:22<02:14, 1.66s/it] | |
| 2024-08-07 19:10:26,417 - INFO: train loss: 0.85: 74%|#######4 | 222/300 [06:27<02:09, 1.66s/it] | |
| 2024-08-07 19:10:31,503 - INFO: train loss: 0.87: 75%|#######5 | 225/300 [06:32<02:05, 1.67s/it] | |
| 2024-08-07 19:10:36,396 - INFO: train loss: 0.70: 76%|#######6 | 228/300 [06:37<01:59, 1.66s/it] | |
| 2024-08-07 19:10:41,514 - INFO: train loss: 0.65: 77%|#######7 | 231/300 [06:42<01:55, 1.67s/it] | |
| 2024-08-07 19:10:46,341 - INFO: train loss: 0.52: 78%|#######8 | 234/300 [06:46<01:49, 1.65s/it] | |
| 2024-08-07 19:10:51,510 - INFO: train loss: 0.56: 79%|#######9 | 237/300 [06:52<01:45, 1.67s/it] | |
| 2024-08-07 19:10:56,430 - INFO: train loss: 0.69: 80%|######## | 240/300 [06:57<01:39, 1.66s/it] | |
| 2024-08-07 19:11:01,795 - INFO: train loss: 0.88: 81%|########1 | 243/300 [07:02<01:36, 1.70s/it] | |
| 2024-08-07 19:11:07,429 - INFO: train loss: 0.84: 82%|########2 | 246/300 [07:08<01:34, 1.75s/it] | |
| 2024-08-07 19:11:12,323 - INFO: train loss: 0.74: 83%|########2 | 249/300 [07:12<01:27, 1.72s/it] | |
| 2024-08-07 19:11:17,576 - INFO: train loss: 0.60: 84%|########4 | 252/300 [07:18<01:22, 1.73s/it] | |
| 2024-08-07 19:11:22,955 - INFO: train loss: 0.60: 85%|########5 | 255/300 [07:23<01:18, 1.75s/it] | |
| 2024-08-07 19:11:27,660 - INFO: train loss: 0.84: 86%|########6 | 258/300 [07:28<01:11, 1.69s/it] | |
| 2024-08-07 19:11:32,389 - INFO: train loss: 0.86: 87%|########7 | 261/300 [07:33<01:04, 1.66s/it] | |
| 2024-08-07 19:11:37,357 - INFO: train loss: 0.81: 88%|########8 | 264/300 [07:37<00:59, 1.66s/it] | |
| 2024-08-07 19:11:42,244 - INFO: train loss: 0.65: 89%|########9 | 267/300 [07:42<00:54, 1.65s/it] | |
| 2024-08-07 19:11:47,713 - INFO: train loss: 0.72: 90%|######### | 270/300 [07:48<00:51, 1.70s/it] | |
| 2024-08-07 19:11:52,431 - INFO: train loss: 0.78: 91%|#########1| 273/300 [07:53<00:44, 1.66s/it] | |
| 2024-08-07 19:11:57,157 - INFO: train loss: 0.96: 92%|#########2| 276/300 [07:57<00:39, 1.64s/it] | |
| 2024-08-07 19:12:02,221 - INFO: train loss: 0.81: 93%|#########3| 279/300 [08:02<00:34, 1.65s/it] | |
| 2024-08-07 19:12:07,076 - INFO: train loss: 0.86: 94%|#########3| 282/300 [08:07<00:29, 1.64s/it] | |
| 2024-08-07 19:12:12,287 - INFO: train loss: 0.73: 95%|#########5| 285/300 [08:12<00:25, 1.67s/it] | |
| 2024-08-07 19:12:17,309 - INFO: train loss: 0.91: 96%|#########6| 288/300 [08:17<00:20, 1.67s/it] | |
| 2024-08-07 19:12:22,028 - INFO: train loss: 1.20: 97%|#########7| 291/300 [08:22<00:14, 1.64s/it] | |
| 2024-08-07 19:12:27,110 - INFO: train loss: 1.26: 98%|#########8| 294/300 [08:27<00:09, 1.66s/it] | |
| 2024-08-07 19:12:31,838 - INFO: train loss: 1.01: 99%|#########9| 297/300 [08:32<00:04, 1.63s/it] | |
| 2024-08-07 19:12:37,289 - INFO: train loss: 0.93: 100%|##########| 300/300 [08:37<00:00, 1.69s/it] | |
| 2024-08-07 19:12:37,289 - INFO: train loss: 0.93: 100%|##########| 300/300 [08:37<00:00, 1.73s/it] | |
| 2024-08-07 19:12:37,289 - INFO: Saving last model checkpoint to /app/output | |
| 2024-08-07 19:12:37,289 - INFO: Saving checkpoint.. | |
| 2024-08-07 19:13:03,140 - INFO: Starting validation inference | |
| 2024-08-07 19:13:03,141 - INFO: validation progress: 0%| | 0/75 [00:00<?, ?it/s] | |
| 2024-08-07 19:13:08,314 - INFO: validation progress: 4%|4 | 3/75 [00:05<02:04, 1.72s/it] | |
| 2024-08-07 19:13:09,837 - INFO: validation progress: 8%|8 | 6/75 [00:06<01:09, 1.01s/it] | |
| 2024-08-07 19:13:11,269 - INFO: validation progress: 12%|#2 | 9/75 [00:08<00:50, 1.31it/s] | |
| 2024-08-07 19:13:12,770 - INFO: validation progress: 16%|#6 | 12/75 [00:09<00:41, 1.51it/s] | |
| 2024-08-07 19:13:14,375 - INFO: validation progress: 20%|## | 15/75 [00:11<00:36, 1.62it/s] | |
| 2024-08-07 19:13:16,027 - INFO: validation progress: 24%|##4 | 18/75 [00:12<00:33, 1.68it/s] | |
| 2024-08-07 19:13:17,755 - INFO: validation progress: 28%|##8 | 21/75 [00:14<00:31, 1.70it/s] | |
| 2024-08-07 19:13:19,293 - INFO: validation progress: 32%|###2 | 24/75 [00:16<00:28, 1.77it/s] | |
| 2024-08-07 19:13:20,991 - INFO: validation progress: 36%|###6 | 27/75 [00:17<00:27, 1.77it/s] | |
| 2024-08-07 19:13:22,668 - INFO: validation progress: 40%|#### | 30/75 [00:19<00:25, 1.78it/s] | |
| 2024-08-07 19:13:24,299 - INFO: validation progress: 44%|####4 | 33/75 [00:21<00:23, 1.80it/s] | |
| 2024-08-07 19:13:25,932 - INFO: validation progress: 48%|####8 | 36/75 [00:22<00:21, 1.81it/s] | |
| 2024-08-07 19:13:27,679 - INFO: validation progress: 52%|#####2 | 39/75 [00:24<00:20, 1.78it/s] | |
| 2024-08-07 19:13:29,263 - INFO: validation progress: 56%|#####6 | 42/75 [00:26<00:18, 1.81it/s] | |
| 2024-08-07 19:13:30,858 - INFO: validation progress: 60%|###### | 45/75 [00:27<00:16, 1.83it/s] | |
| 2024-08-07 19:13:32,367 - INFO: validation progress: 64%|######4 | 48/75 [00:29<00:14, 1.88it/s] | |
| 2024-08-07 19:13:33,874 - INFO: validation progress: 68%|######8 | 51/75 [00:30<00:12, 1.91it/s] | |
| 2024-08-07 19:13:35,532 - INFO: validation progress: 72%|#######2 | 54/75 [00:32<00:11, 1.88it/s] | |
| 2024-08-07 19:13:37,126 - INFO: validation progress: 76%|#######6 | 57/75 [00:33<00:09, 1.88it/s] | |
| 2024-08-07 19:13:38,658 - INFO: validation progress: 80%|######## | 60/75 [00:35<00:07, 1.90it/s] | |
| 2024-08-07 19:13:40,459 - INFO: validation progress: 84%|########4 | 63/75 [00:37<00:06, 1.82it/s] | |
| 2024-08-07 19:13:42,139 - INFO: validation progress: 88%|########8 | 66/75 [00:38<00:04, 1.81it/s] | |
| 2024-08-07 19:13:43,821 - INFO: validation progress: 92%|#########2| 69/75 [00:40<00:03, 1.80it/s] | |
| 2024-08-07 19:13:45,282 - INFO: validation progress: 96%|#########6| 72/75 [00:42<00:01, 1.87it/s] | |
| 2024-08-07 19:13:47,023 - INFO: validation progress: 100%|##########| 75/75 [00:43<00:00, 1.82it/s] | |
| 2024-08-07 19:13:47,026 - INFO: validation progress: 100%|##########| 75/75 [00:43<00:00, 1.71it/s] | |
| 2024-08-07 19:13:47,250 - INFO: Validation AUC: 0.95527 | |
| 2024-08-07 19:13:47,251 - INFO: Mean validation loss: 0.80791 | |
| 2024-08-07 19:13:47,370 - INFO: Training Epoch: 2 / 10 | |
| 2024-08-07 19:13:47,370 - INFO: train loss: 0%| | 0/300 [00:00<?, ?it/s] | |
| 2024-08-07 19:13:50,662 - INFO: Evaluation step: 300 | |
| 2024-08-07 19:13:55,857 - INFO: train loss: 0.61: 1%|1 | 3/300 [00:08<14:00, 2.83s/it] | |
| 2024-08-07 19:14:00,671 - INFO: train loss: 0.58: 2%|2 | 6/300 [00:13<10:19, 2.11s/it] | |
| 2024-08-07 19:14:05,790 - INFO: train loss: 0.55: 3%|3 | 9/300 [00:18<09:20, 1.92s/it] | |
| 2024-08-07 19:14:10,424 - INFO: train loss: 0.59: 4%|4 | 12/300 [00:23<08:31, 1.77s/it] | |
| 2024-08-07 19:14:15,917 - INFO: train loss: 0.58: 5%|5 | 15/300 [00:28<08:31, 1.80s/it] | |
| 2024-08-07 19:14:20,795 - INFO: train loss: 0.53: 6%|6 | 18/300 [00:33<08:09, 1.74s/it] | |
| 2024-08-07 19:14:25,501 - INFO: train loss: 0.53: 7%|7 | 21/300 [00:38<07:49, 1.68s/it] | |
| 2024-08-07 19:14:30,895 - INFO: train loss: 0.46: 8%|8 | 24/300 [00:43<07:54, 1.72s/it] | |
| 2024-08-07 19:14:35,892 - INFO: train loss: 0.72: 9%|9 | 27/300 [00:48<07:44, 1.70s/it] | |
| 2024-08-07 19:14:41,067 - INFO: train loss: 1.49: 10%|# | 30/300 [00:53<07:41, 1.71s/it] | |
| 2024-08-07 19:14:46,191 - INFO: train loss: 1.54: 11%|#1 | 33/300 [00:58<07:36, 1.71s/it] | |
| 2024-08-07 19:14:51,170 - INFO: train loss: 1.49: 12%|#2 | 36/300 [01:03<07:27, 1.69s/it] | |
| 2024-08-07 19:14:56,391 - INFO: train loss: 0.86: 13%|#3 | 39/300 [01:09<07:25, 1.71s/it] | |
| 2024-08-07 19:15:01,709 - INFO: train loss: 0.94: 14%|#4 | 42/300 [01:14<07:25, 1.73s/it] | |
| 2024-08-07 19:15:06,886 - INFO: train loss: 0.91: 15%|#5 | 45/300 [01:19<07:20, 1.73s/it] | |
| 2024-08-07 19:15:11,549 - INFO: train loss: 0.86: 16%|#6 | 48/300 [01:24<07:02, 1.67s/it] | |
| 2024-08-07 19:15:16,858 - INFO: train loss: 0.63: 17%|#7 | 51/300 [01:29<07:04, 1.70s/it] | |
| 2024-08-07 19:15:22,131 - INFO: train loss: 0.70: 18%|#8 | 54/300 [01:34<07:03, 1.72s/it] | |
| 2024-08-07 19:15:27,297 - INFO: train loss: 1.06: 19%|#9 | 57/300 [01:39<06:58, 1.72s/it] | |
| 2024-08-07 19:15:32,362 - INFO: train loss: 1.05: 20%|## | 60/300 [01:44<06:50, 1.71s/it] | |
| 2024-08-07 19:15:37,259 - INFO: train loss: 1.12: 21%|##1 | 63/300 [01:49<06:39, 1.69s/it] | |
| 2024-08-07 19:15:42,011 - INFO: train loss: 0.78: 22%|##2 | 66/300 [01:54<06:27, 1.66s/it] | |
| 2024-08-07 19:15:47,220 - INFO: train loss: 0.55: 23%|##3 | 69/300 [01:59<06:28, 1.68s/it] | |
| 2024-08-07 19:15:52,099 - INFO: train loss: 0.51: 24%|##4 | 72/300 [02:04<06:19, 1.66s/it] | |
| 2024-08-07 19:15:57,267 - INFO: train loss: 0.65: 25%|##5 | 75/300 [02:09<06:18, 1.68s/it] | |
| 2024-08-07 19:16:02,338 - INFO: train loss: 0.74: 26%|##6 | 78/300 [02:14<06:13, 1.68s/it] | |
| 2024-08-07 19:16:07,875 - INFO: train loss: 0.88: 27%|##7 | 81/300 [02:20<06:19, 1.73s/it] | |
| 2024-08-07 19:16:13,320 - INFO: train loss: 0.74: 28%|##8 | 84/300 [02:25<06:19, 1.76s/it] | |
| 2024-08-07 19:16:18,154 - INFO: train loss: 0.53: 29%|##9 | 87/300 [02:30<06:04, 1.71s/it] | |
| 2024-08-07 19:16:24,044 - INFO: train loss: 0.39: 30%|### | 90/300 [02:36<06:15, 1.79s/it] | |
| 2024-08-07 19:16:29,104 - INFO: train loss: 0.34: 31%|###1 | 93/300 [02:41<06:03, 1.76s/it] | |
| 2024-08-07 19:16:34,700 - INFO: train loss: 0.41: 32%|###2 | 96/300 [02:47<06:05, 1.79s/it] | |
| 2024-08-07 19:16:40,060 - INFO: train loss: 0.44: 33%|###3 | 99/300 [02:52<05:59, 1.79s/it] | |
| 2024-08-07 19:16:45,111 - INFO: train loss: 0.53: 34%|###4 | 102/300 [02:57<05:47, 1.76s/it] | |
| 2024-08-07 19:16:51,649 - INFO: train loss: 0.51: 35%|###5 | 105/300 [03:04<06:07, 1.88s/it] | |
| 2024-08-07 19:16:57,221 - INFO: train loss: 0.45: 36%|###6 | 108/300 [03:09<06:00, 1.88s/it] | |
| 2024-08-07 19:17:01,876 - INFO: train loss: 0.43: 37%|###7 | 111/300 [03:14<05:36, 1.78s/it] | |
| 2024-08-07 19:17:06,879 - INFO: train loss: 0.40: 38%|###8 | 114/300 [03:19<05:24, 1.75s/it] | |
| 2024-08-07 19:17:11,988 - INFO: train loss: 0.43: 39%|###9 | 117/300 [03:24<05:17, 1.73s/it] | |
| 2024-08-07 19:17:16,949 - INFO: train loss: 0.36: 40%|#### | 120/300 [03:29<05:07, 1.71s/it] | |
| 2024-08-07 19:17:21,540 - INFO: train loss: 0.51: 41%|####1 | 123/300 [03:34<04:53, 1.66s/it] | |
| 2024-08-07 19:17:26,517 - INFO: train loss: 0.58: 42%|####2 | 126/300 [03:39<04:48, 1.66s/it] | |
| 2024-08-07 19:17:31,848 - INFO: train loss: 0.62: 43%|####3 | 129/300 [03:44<04:49, 1.69s/it] | |
| 2024-08-07 19:17:36,859 - INFO: train loss: 0.69: 44%|####4 | 132/300 [03:49<04:43, 1.69s/it] | |
| 2024-08-07 19:17:42,055 - INFO: train loss: 0.55: 45%|####5 | 135/300 [03:54<04:40, 1.70s/it] | |
| 2024-08-07 19:17:46,785 - INFO: train loss: 0.59: 46%|####6 | 138/300 [03:59<04:29, 1.66s/it] | |
| 2024-08-07 19:17:51,609 - INFO: train loss: 0.59: 47%|####6 | 141/300 [04:04<04:21, 1.65s/it] | |
| 2024-08-07 19:17:56,419 - INFO: train loss: 0.59: 48%|####8 | 144/300 [04:09<04:14, 1.63s/it] | |
| 2024-08-07 19:18:01,455 - INFO: train loss: 0.58: 49%|####9 | 147/300 [04:14<04:11, 1.65s/it] | |
| 2024-08-07 19:18:06,183 - INFO: train loss: 0.59: 50%|##### | 150/300 [04:18<04:03, 1.63s/it] | |
| 2024-08-07 19:18:11,341 - INFO: train loss: 0.61: 51%|#####1 | 153/300 [04:23<04:03, 1.65s/it] | |
| 2024-08-07 19:18:16,482 - INFO: train loss: 0.60: 52%|#####2 | 156/300 [04:29<04:00, 1.67s/it] | |
| 2024-08-07 19:18:21,284 - INFO: train loss: 0.71: 53%|#####3 | 159/300 [04:33<03:52, 1.65s/it] | |
| 2024-08-07 19:18:26,276 - INFO: train loss: 0.68: 54%|#####4 | 162/300 [04:38<03:48, 1.65s/it] | |
| 2024-08-07 19:18:31,378 - INFO: train loss: 0.76: 55%|#####5 | 165/300 [04:44<03:45, 1.67s/it] | |
| 2024-08-07 19:18:36,552 - INFO: train loss: 0.64: 56%|#####6 | 168/300 [04:49<03:42, 1.69s/it] | |
| 2024-08-07 19:18:41,487 - INFO: train loss: 0.63: 57%|#####6 | 171/300 [04:54<03:35, 1.67s/it] | |
| 2024-08-07 19:18:46,411 - INFO: train loss: 0.54: 58%|#####8 | 174/300 [04:59<03:29, 1.66s/it] | |
| 2024-08-07 19:18:51,024 - INFO: train loss: 0.48: 59%|#####8 | 177/300 [05:03<03:19, 1.63s/it] | |
| 2024-08-07 19:18:56,135 - INFO: train loss: 0.47: 60%|###### | 180/300 [05:08<03:17, 1.65s/it] | |
| 2024-08-07 19:19:00,934 - INFO: train loss: 0.48: 61%|######1 | 183/300 [05:13<03:11, 1.63s/it] | |
| 2024-08-07 19:19:06,044 - INFO: train loss: 0.53: 62%|######2 | 186/300 [05:18<03:08, 1.65s/it] | |
| 2024-08-07 19:19:10,627 - INFO: train loss: 0.67: 63%|######3 | 189/300 [05:23<02:59, 1.62s/it] | |
| 2024-08-07 19:19:15,307 - INFO: train loss: 0.62: 64%|######4 | 192/300 [05:27<02:52, 1.60s/it] | |
| 2024-08-07 19:19:20,224 - INFO: train loss: 0.58: 65%|######5 | 195/300 [05:32<02:49, 1.61s/it] | |
| 2024-08-07 19:19:25,369 - INFO: train loss: 0.49: 66%|######6 | 198/300 [05:37<02:47, 1.64s/it] | |
| 2024-08-07 19:19:30,555 - INFO: train loss: 0.42: 67%|######7 | 201/300 [05:43<02:45, 1.67s/it] | |
| 2024-08-07 19:19:35,560 - INFO: train loss: 0.53: 68%|######8 | 204/300 [05:48<02:40, 1.67s/it] | |
| 2024-08-07 19:19:40,911 - INFO: train loss: 0.60: 69%|######9 | 207/300 [05:53<02:38, 1.70s/it] | |
| 2024-08-07 19:19:45,894 - INFO: train loss: 0.73: 70%|####### | 210/300 [05:58<02:32, 1.69s/it] | |
| 2024-08-07 19:19:50,909 - INFO: train loss: 0.70: 71%|#######1 | 213/300 [06:03<02:26, 1.68s/it] | |
| 2024-08-07 19:19:55,567 - INFO: train loss: 0.72: 72%|#######2 | 216/300 [06:08<02:18, 1.65s/it] | |
| 2024-08-07 19:20:00,750 - INFO: train loss: 0.71: 73%|#######3 | 219/300 [06:13<02:15, 1.67s/it] | |
| 2024-08-07 19:20:05,857 - INFO: train loss: 0.61: 74%|#######4 | 222/300 [06:18<02:11, 1.68s/it] | |
| 2024-08-07 19:20:10,434 - INFO: train loss: 0.60: 75%|#######5 | 225/300 [06:23<02:02, 1.63s/it] | |
| 2024-08-07 19:20:15,355 - INFO: train loss: 0.54: 76%|#######6 | 228/300 [06:27<01:57, 1.64s/it] | |
| 2024-08-07 19:20:21,092 - INFO: train loss: 0.52: 77%|#######7 | 231/300 [06:33<01:58, 1.72s/it] | |
| 2024-08-07 19:20:26,018 - INFO: train loss: 0.45: 78%|#######8 | 234/300 [06:38<01:51, 1.70s/it] | |
| 2024-08-07 19:20:31,252 - INFO: train loss: 0.60: 79%|#######9 | 237/300 [06:43<01:47, 1.71s/it] | |
| 2024-08-07 19:20:36,386 - INFO: train loss: 0.76: 80%|######## | 240/300 [06:49<01:42, 1.71s/it] | |
| 2024-08-07 19:20:41,776 - INFO: train loss: 0.72: 81%|########1 | 243/300 [06:54<01:38, 1.74s/it] | |
| 2024-08-07 19:20:46,915 - INFO: train loss: 0.82: 82%|########2 | 246/300 [06:59<01:33, 1.73s/it] | |
| 2024-08-07 19:20:51,388 - INFO: train loss: 0.44: 83%|########2 | 249/300 [07:04<01:24, 1.66s/it] | |
| 2024-08-07 19:20:56,415 - INFO: train loss: 0.68: 84%|########4 | 252/300 [07:09<01:19, 1.66s/it] | |
| 2024-08-07 19:21:01,275 - INFO: train loss: 0.78: 85%|########5 | 255/300 [07:13<01:14, 1.65s/it] | |
| 2024-08-07 19:21:06,486 - INFO: train loss: 0.77: 86%|########6 | 258/300 [07:19<01:10, 1.68s/it] | |
| 2024-08-07 19:21:11,351 - INFO: train loss: 0.71: 87%|########7 | 261/300 [07:23<01:04, 1.66s/it] | |
| 2024-08-07 19:21:16,069 - INFO: train loss: 0.46: 88%|########8 | 264/300 [07:28<00:58, 1.63s/it] | |
| 2024-08-07 19:21:20,879 - INFO: train loss: 0.46: 89%|########9 | 267/300 [07:33<00:53, 1.62s/it] | |
| 2024-08-07 19:21:26,072 - INFO: train loss: 0.50: 90%|######### | 270/300 [07:38<00:49, 1.66s/it] | |
| 2024-08-07 19:21:31,164 - INFO: train loss: 0.54: 91%|#########1| 273/300 [07:43<00:45, 1.67s/it] | |
| 2024-08-07 19:21:35,863 - INFO: train loss: 0.80: 92%|#########2| 276/300 [07:48<00:39, 1.64s/it] | |
| 2024-08-07 19:21:40,944 - INFO: train loss: 0.74: 93%|#########3| 279/300 [07:53<00:34, 1.65s/it] | |
| 2024-08-07 19:21:45,764 - INFO: train loss: 0.86: 94%|#########3| 282/300 [07:58<00:29, 1.64s/it] | |
| 2024-08-07 19:21:50,668 - INFO: train loss: 0.61: 95%|#########5| 285/300 [08:03<00:24, 1.64s/it] | |
| 2024-08-07 19:21:55,646 - INFO: train loss: 0.85: 96%|#########6| 288/300 [08:08<00:19, 1.64s/it] | |
| 2024-08-07 19:22:00,327 - INFO: train loss: 0.79: 97%|#########7| 291/300 [08:12<00:14, 1.62s/it] | |
| 2024-08-07 19:22:04,848 - INFO: train loss: 0.76: 98%|#########8| 294/300 [08:17<00:09, 1.59s/it] | |
| 2024-08-07 19:22:09,450 - INFO: train loss: 0.51: 99%|#########9| 297/300 [08:22<00:04, 1.57s/it] | |
| 2024-08-07 19:22:14,517 - INFO: train loss: 0.74: 100%|##########| 300/300 [08:27<00:00, 1.61s/it] | |
| 2024-08-07 19:22:14,517 - INFO: train loss: 0.74: 100%|##########| 300/300 [08:27<00:00, 1.69s/it] | |
| 2024-08-07 19:22:14,517 - INFO: Saving last model checkpoint to /app/output | |
| 2024-08-07 19:22:14,518 - INFO: Saving checkpoint.. | |
| 2024-08-07 19:22:40,850 - INFO: Starting validation inference | |
| 2024-08-07 19:22:40,851 - INFO: validation progress: 0%| | 0/75 [00:00<?, ?it/s] | |
| 2024-08-07 19:22:46,789 - INFO: validation progress: 4%|4 | 3/75 [00:05<02:22, 1.98s/it] | |
| 2024-08-07 19:22:48,304 - INFO: validation progress: 8%|8 | 6/75 [00:07<01:16, 1.11s/it] | |
| 2024-08-07 19:22:49,749 - INFO: validation progress: 12%|#2 | 9/75 [00:08<00:54, 1.21it/s] | |
| 2024-08-07 19:22:51,249 - INFO: validation progress: 16%|#6 | 12/75 [00:10<00:43, 1.44it/s] | |
| 2024-08-07 19:22:52,858 - INFO: validation progress: 20%|## | 15/75 [00:12<00:38, 1.57it/s] | |
| 2024-08-07 19:22:54,468 - INFO: validation progress: 24%|##4 | 18/75 [00:13<00:34, 1.66it/s] | |
| 2024-08-07 19:22:56,098 - INFO: validation progress: 28%|##8 | 21/75 [00:15<00:31, 1.71it/s] | |
| 2024-08-07 19:22:57,589 - INFO: validation progress: 32%|###2 | 24/75 [00:16<00:28, 1.80it/s] | |
| 2024-08-07 19:22:59,293 - INFO: validation progress: 36%|###6 | 27/75 [00:18<00:26, 1.79it/s] | |
| 2024-08-07 19:23:00,869 - INFO: validation progress: 40%|#### | 30/75 [00:20<00:24, 1.82it/s] | |
| 2024-08-07 19:23:02,505 - INFO: validation progress: 44%|####4 | 33/75 [00:21<00:23, 1.82it/s] | |
| 2024-08-07 19:23:04,190 - INFO: validation progress: 48%|####8 | 36/75 [00:23<00:21, 1.81it/s] | |
| 2024-08-07 19:23:05,909 - INFO: validation progress: 52%|#####2 | 39/75 [00:25<00:20, 1.79it/s] | |
| 2024-08-07 19:23:07,486 - INFO: validation progress: 56%|#####6 | 42/75 [00:26<00:18, 1.82it/s] | |
| 2024-08-07 19:23:09,034 - INFO: validation progress: 60%|###### | 45/75 [00:28<00:16, 1.86it/s] | |
| 2024-08-07 19:23:10,493 - INFO: validation progress: 64%|######4 | 48/75 [00:29<00:14, 1.91it/s] | |
| 2024-08-07 19:23:12,006 - INFO: validation progress: 68%|######8 | 51/75 [00:31<00:12, 1.93it/s] | |
| 2024-08-07 19:23:13,687 - INFO: validation progress: 72%|#######2 | 54/75 [00:32<00:11, 1.89it/s] | |
| 2024-08-07 19:23:15,257 - INFO: validation progress: 76%|#######6 | 57/75 [00:34<00:09, 1.89it/s] | |
| 2024-08-07 19:23:16,698 - INFO: validation progress: 80%|######## | 60/75 [00:35<00:07, 1.95it/s] | |
| 2024-08-07 19:23:18,403 - INFO: validation progress: 84%|########4 | 63/75 [00:37<00:06, 1.89it/s] | |
| 2024-08-07 19:23:20,037 - INFO: validation progress: 88%|########8 | 66/75 [00:39<00:04, 1.87it/s] | |
| 2024-08-07 19:23:21,764 - INFO: validation progress: 92%|#########2| 69/75 [00:40<00:03, 1.83it/s] | |
| 2024-08-07 19:23:23,241 - INFO: validation progress: 96%|#########6| 72/75 [00:42<00:01, 1.89it/s] | |
| 2024-08-07 19:23:24,969 - INFO: validation progress: 100%|##########| 75/75 [00:44<00:00, 1.84it/s] | |
| 2024-08-07 19:23:24,971 - INFO: validation progress: 100%|##########| 75/75 [00:44<00:00, 1.70it/s] | |
| 2024-08-07 19:23:25,300 - INFO: Validation AUC: 0.97725 | |
| 2024-08-07 19:23:25,300 - INFO: Mean validation loss: 1.16324 | |
| 2024-08-07 19:23:25,706 - INFO: Training Epoch: 3 / 10 | |
| 2024-08-07 19:23:25,706 - INFO: train loss: 0%| | 0/300 [00:00<?, ?it/s] | |
| 2024-08-07 19:23:29,575 - INFO: Evaluation step: 300 | |
| 2024-08-07 19:23:34,831 - INFO: train loss: 0.59: 1%|1 | 3/300 [00:09<15:03, 3.04s/it] | |
| 2024-08-07 19:23:39,501 - INFO: train loss: 0.48: 2%|2 | 6/300 [00:13<10:37, 2.17s/it] | |
| 2024-08-07 19:23:44,342 - INFO: train loss: 0.64: 3%|3 | 9/300 [00:18<09:17, 1.91s/it] | |
| 2024-08-07 19:23:48,851 - INFO: train loss: 0.58: 4%|4 | 12/300 [00:23<08:24, 1.75s/it] | |
| 2024-08-07 19:23:54,092 - INFO: train loss: 0.52: 5%|5 | 15/300 [00:28<08:18, 1.75s/it] | |
| 2024-08-07 19:23:58,855 - INFO: train loss: 0.50: 6%|6 | 18/300 [00:33<07:58, 1.70s/it] | |
| 2024-08-07 19:24:03,561 - INFO: train loss: 0.52: 7%|7 | 21/300 [00:37<07:41, 1.65s/it] | |
| 2024-08-07 19:24:08,537 - INFO: train loss: 0.55: 8%|8 | 24/300 [00:42<07:36, 1.66s/it] | |
| 2024-08-07 19:24:13,683 - INFO: train loss: 0.47: 9%|9 | 27/300 [00:47<07:37, 1.67s/it] | |
| 2024-08-07 19:24:19,175 - INFO: train loss: 0.44: 10%|# | 30/300 [00:53<07:45, 1.72s/it] | |
| 2024-08-07 19:24:24,253 - INFO: train loss: 0.50: 11%|#1 | 33/300 [00:58<07:37, 1.71s/it] | |
| 2024-08-07 19:24:29,065 - INFO: train loss: 0.50: 12%|#2 | 36/300 [01:03<07:23, 1.68s/it] | |
| 2024-08-07 19:24:33,566 - INFO: train loss: 0.52: 13%|#3 | 39/300 [01:07<07:04, 1.63s/it] | |
| 2024-08-07 19:24:38,556 - INFO: train loss: 0.33: 14%|#4 | 42/300 [01:12<07:02, 1.64s/it] | |
| 2024-08-07 19:24:43,747 - INFO: train loss: 0.52: 15%|#5 | 45/300 [01:18<07:04, 1.67s/it] | |
| 2024-08-07 19:24:48,683 - INFO: train loss: 0.44: 16%|#6 | 48/300 [01:22<06:58, 1.66s/it] | |
| 2024-08-07 19:24:53,798 - INFO: train loss: 0.48: 17%|#7 | 51/300 [01:28<06:56, 1.67s/it] | |
| 2024-08-07 19:24:59,069 - INFO: train loss: 0.38: 18%|#8 | 54/300 [01:33<06:57, 1.70s/it] | |
| 2024-08-07 19:25:04,460 - INFO: train loss: 0.38: 19%|#9 | 57/300 [01:38<06:59, 1.73s/it] | |
| 2024-08-07 19:25:09,478 - INFO: train loss: 0.38: 20%|## | 60/300 [01:43<06:50, 1.71s/it] | |
| 2024-08-07 19:25:14,373 - INFO: train loss: 0.44: 21%|##1 | 63/300 [01:48<06:39, 1.69s/it] | |
| 2024-08-07 19:25:19,465 - INFO: train loss: 0.76: 22%|##2 | 66/300 [01:53<06:35, 1.69s/it] | |
| 2024-08-07 19:25:24,162 - INFO: train loss: 0.80: 23%|##3 | 69/300 [01:58<06:21, 1.65s/it] | |
| 2024-08-07 19:25:29,792 - INFO: train loss: 1.08: 24%|##4 | 72/300 [02:04<06:32, 1.72s/it] | |
| 2024-08-07 19:25:34,796 - INFO: train loss: 0.87: 25%|##5 | 75/300 [02:09<06:23, 1.70s/it] | |
| 2024-08-07 19:25:40,054 - INFO: train loss: 0.74: 26%|##6 | 78/300 [02:14<06:21, 1.72s/it] | |
| 2024-08-07 19:25:45,549 - INFO: train loss: 0.46: 27%|##7 | 81/300 [02:19<06:23, 1.75s/it] | |
| 2024-08-07 19:25:50,736 - INFO: train loss: 0.35: 28%|##8 | 84/300 [02:25<06:17, 1.75s/it] | |
| 2024-08-07 19:25:55,329 - INFO: train loss: 0.52: 29%|##9 | 87/300 [02:29<05:58, 1.68s/it] | |
| 2024-08-07 19:26:00,146 - INFO: train loss: 0.53: 30%|### | 90/300 [02:34<05:48, 1.66s/it] | |
| 2024-08-07 19:26:05,185 - INFO: train loss: 0.45: 31%|###1 | 93/300 [02:39<05:44, 1.66s/it] | |
| 2024-08-07 19:26:10,016 - INFO: train loss: 0.61: 32%|###2 | 96/300 [02:44<05:36, 1.65s/it] | |
| 2024-08-07 19:26:14,846 - INFO: train loss: 0.41: 33%|###3 | 99/300 [02:49<05:29, 1.64s/it] | |
| 2024-08-07 19:26:20,159 - INFO: train loss: 0.49: 34%|###4 | 102/300 [02:54<05:32, 1.68s/it] | |
| 2024-08-07 19:26:25,375 - INFO: train loss: 0.30: 35%|###5 | 105/300 [02:59<05:30, 1.70s/it] | |
| 2024-08-07 19:26:30,313 - INFO: train loss: 0.41: 36%|###6 | 108/300 [03:04<05:22, 1.68s/it] | |
| 2024-08-07 19:26:36,208 - INFO: train loss: 0.33: 37%|###7 | 111/300 [03:10<05:33, 1.77s/it] | |
| 2024-08-07 19:26:40,662 - INFO: train loss: 0.48: 38%|###8 | 114/300 [03:14<05:12, 1.68s/it] | |
| 2024-08-07 19:26:45,404 - INFO: train loss: 0.50: 39%|###9 | 117/300 [03:19<05:02, 1.65s/it] | |
| 2024-08-07 19:26:51,145 - INFO: train loss: 0.54: 40%|#### | 120/300 [03:25<05:11, 1.73s/it] | |
| 2024-08-07 19:26:56,365 - INFO: train loss: 0.56: 41%|####1 | 123/300 [03:30<05:06, 1.73s/it] | |
| 2024-08-07 19:27:01,889 - INFO: train loss: 0.53: 42%|####2 | 126/300 [03:36<05:07, 1.77s/it] | |
| 2024-08-07 19:27:06,744 - INFO: train loss: 0.44: 43%|####3 | 129/300 [03:41<04:54, 1.72s/it] | |
| 2024-08-07 19:27:11,914 - INFO: train loss: 0.45: 44%|####4 | 132/300 [03:46<04:49, 1.72s/it] | |
| 2024-08-07 19:27:16,646 - INFO: train loss: 0.29: 45%|####5 | 135/300 [03:50<04:36, 1.68s/it] | |
| 2024-08-07 19:27:21,536 - INFO: train loss: 0.30: 46%|####6 | 138/300 [03:55<04:29, 1.66s/it] | |
| 2024-08-07 19:27:27,003 - INFO: train loss: 0.31: 47%|####6 | 141/300 [04:01<04:32, 1.71s/it] | |
| 2024-08-07 19:27:32,177 - INFO: train loss: 0.27: 48%|####8 | 144/300 [04:06<04:27, 1.72s/it] | |
| 2024-08-07 19:27:37,259 - INFO: train loss: 0.22: 49%|####9 | 147/300 [04:11<04:21, 1.71s/it] | |
| 2024-08-07 19:27:42,604 - INFO: train loss: 0.23: 50%|##### | 150/300 [04:16<04:19, 1.73s/it] | |
| 2024-08-07 19:27:47,759 - INFO: train loss: 0.25: 51%|#####1 | 153/300 [04:22<04:13, 1.73s/it] | |
| 2024-08-07 19:27:52,999 - INFO: train loss: 0.27: 52%|#####2 | 156/300 [04:27<04:09, 1.73s/it] | |
| 2024-08-07 19:27:58,082 - INFO: train loss: 0.30: 53%|#####3 | 159/300 [04:32<04:02, 1.72s/it] | |
| 2024-08-07 19:28:03,115 - INFO: train loss: 0.36: 54%|#####4 | 162/300 [04:37<03:55, 1.71s/it] | |
| 2024-08-07 19:28:08,035 - INFO: train loss: 0.36: 55%|#####5 | 165/300 [04:42<03:47, 1.69s/it] | |
| 2024-08-07 19:28:12,639 - INFO: train loss: 0.39: 56%|#####6 | 168/300 [04:46<03:36, 1.64s/it] | |
| 2024-08-07 19:28:17,843 - INFO: train loss: 0.38: 57%|#####6 | 171/300 [04:52<03:35, 1.67s/it] | |
| 2024-08-07 19:28:23,175 - INFO: train loss: 0.38: 58%|#####8 | 174/300 [04:57<03:34, 1.70s/it] | |
| 2024-08-07 19:28:27,933 - INFO: train loss: 0.40: 59%|#####8 | 177/300 [05:02<03:25, 1.67s/it] | |
| 2024-08-07 19:28:33,051 - INFO: train loss: 0.40: 60%|###### | 180/300 [05:07<03:21, 1.68s/it] | |
| 2024-08-07 19:28:38,074 - INFO: train loss: 0.42: 61%|######1 | 183/300 [05:12<03:16, 1.68s/it] | |
| 2024-08-07 19:28:42,833 - INFO: train loss: 0.34: 62%|######2 | 186/300 [05:17<03:08, 1.65s/it] | |
| 2024-08-07 19:28:47,430 - INFO: train loss: 0.34: 63%|######3 | 189/300 [05:21<02:59, 1.61s/it] | |
| 2024-08-07 19:28:52,617 - INFO: train loss: 0.34: 64%|######4 | 192/300 [05:26<02:58, 1.65s/it] | |
| 2024-08-07 19:28:57,845 - INFO: train loss: 0.41: 65%|######5 | 195/300 [05:32<02:56, 1.68s/it] | |
| 2024-08-07 19:29:02,658 - INFO: train loss: 0.49: 66%|######6 | 198/300 [05:36<02:48, 1.66s/it] | |
| 2024-08-07 19:29:08,210 - INFO: train loss: 0.49: 67%|######7 | 201/300 [05:42<02:49, 1.71s/it] | |
| 2024-08-07 19:29:12,848 - INFO: train loss: 0.40: 68%|######8 | 204/300 [05:47<02:39, 1.66s/it] | |
| 2024-08-07 19:29:17,976 - INFO: train loss: 0.35: 69%|######9 | 207/300 [05:52<02:35, 1.68s/it] | |
| 2024-08-07 19:29:22,891 - INFO: train loss: 0.37: 70%|####### | 210/300 [05:57<02:29, 1.67s/it] | |
| 2024-08-07 19:29:27,636 - INFO: train loss: 0.46: 71%|#######1 | 213/300 [06:01<02:22, 1.64s/it] | |
| 2024-08-07 19:29:32,488 - INFO: train loss: 0.46: 72%|#######2 | 216/300 [06:06<02:17, 1.63s/it] | |
| 2024-08-07 19:29:37,637 - INFO: train loss: 0.54: 73%|#######3 | 219/300 [06:11<02:14, 1.66s/it] | |
| 2024-08-07 19:29:42,650 - INFO: train loss: 0.39: 74%|#######4 | 222/300 [06:16<02:09, 1.66s/it] | |
| 2024-08-07 19:29:48,009 - INFO: train loss: 0.40: 75%|#######5 | 225/300 [06:22<02:07, 1.70s/it] | |
| 2024-08-07 19:29:53,119 - INFO: train loss: 0.39: 76%|#######6 | 228/300 [06:27<02:02, 1.70s/it] | |
| 2024-08-07 19:29:57,838 - INFO: train loss: 0.35: 77%|#######7 | 231/300 [06:32<01:54, 1.66s/it] | |
| 2024-08-07 19:30:02,931 - INFO: train loss: 0.32: 78%|#######8 | 234/300 [06:37<01:50, 1.67s/it] | |
| 2024-08-07 19:30:07,740 - INFO: train loss: 0.40: 79%|#######9 | 237/300 [06:42<01:44, 1.65s/it] | |
| 2024-08-07 19:30:12,602 - INFO: train loss: 0.53: 80%|######## | 240/300 [06:46<01:38, 1.64s/it] | |
| 2024-08-07 19:30:17,604 - INFO: train loss: 0.58: 81%|########1 | 243/300 [06:51<01:34, 1.65s/it] | |
| 2024-08-07 19:30:22,725 - INFO: train loss: 0.49: 82%|########2 | 246/300 [06:57<01:30, 1.67s/it] | |
| 2024-08-07 19:30:27,864 - INFO: train loss: 0.30: 83%|########2 | 249/300 [07:02<01:25, 1.68s/it] | |
| 2024-08-07 19:30:34,111 - INFO: train loss: 0.28: 84%|########4 | 252/300 [07:08<01:26, 1.80s/it] | |
| 2024-08-07 19:30:39,163 - INFO: train loss: 0.27: 85%|########5 | 255/300 [07:13<01:19, 1.77s/it] | |
| 2024-08-07 19:30:44,370 - INFO: train loss: 0.33: 86%|########6 | 258/300 [07:18<01:13, 1.76s/it] | |
| 2024-08-07 19:30:49,655 - INFO: train loss: 0.32: 87%|########7 | 261/300 [07:23<01:08, 1.76s/it] | |
| 2024-08-07 19:30:54,837 - INFO: train loss: 0.35: 88%|########8 | 264/300 [07:29<01:02, 1.75s/it] | |
| 2024-08-07 19:30:59,962 - INFO: train loss: 0.38: 89%|########9 | 267/300 [07:34<00:57, 1.74s/it] | |
| 2024-08-07 19:31:04,838 - INFO: train loss: 0.33: 90%|######### | 270/300 [07:39<00:51, 1.70s/it] | |
| 2024-08-07 19:31:09,868 - INFO: train loss: 0.34: 91%|#########1| 273/300 [07:44<00:45, 1.70s/it] | |
| 2024-08-07 19:31:15,082 - INFO: train loss: 0.36: 92%|#########2| 276/300 [07:49<00:40, 1.71s/it] | |
| 2024-08-07 19:31:19,851 - INFO: train loss: 0.35: 93%|#########3| 279/300 [07:54<00:35, 1.67s/it] | |
| 2024-08-07 19:31:25,179 - INFO: train loss: 0.40: 94%|#########3| 282/300 [07:59<00:30, 1.70s/it] | |
| 2024-08-07 19:31:30,292 - INFO: train loss: 0.44: 95%|#########5| 285/300 [08:04<00:25, 1.70s/it] | |
| 2024-08-07 19:31:34,977 - INFO: train loss: 0.35: 96%|#########6| 288/300 [08:09<00:19, 1.66s/it] | |
| 2024-08-07 19:31:40,691 - INFO: train loss: 0.29: 97%|#########7| 291/300 [08:14<00:15, 1.73s/it] | |
| 2024-08-07 19:31:46,068 - INFO: train loss: 0.37: 98%|#########8| 294/300 [08:20<00:10, 1.75s/it] | |
| 2024-08-07 19:31:50,951 - INFO: train loss: 0.38: 99%|#########9| 297/300 [08:25<00:05, 1.71s/it] | |
| 2024-08-07 19:31:55,957 - INFO: train loss: 0.47: 100%|##########| 300/300 [08:30<00:00, 1.70s/it] | |
| 2024-08-07 19:31:55,957 - INFO: train loss: 0.47: 100%|##########| 300/300 [08:30<00:00, 1.70s/it] | |
| 2024-08-07 19:31:55,957 - INFO: Saving last model checkpoint to /app/output | |
| 2024-08-07 19:31:55,957 - INFO: Saving checkpoint.. | |
| 2024-08-07 19:32:27,078 - INFO: Starting validation inference | |
| 2024-08-07 19:32:27,079 - INFO: validation progress: 0%| | 0/75 [00:00<?, ?it/s] | |
| 2024-08-07 19:32:32,655 - INFO: validation progress: 4%|4 | 3/75 [00:05<02:13, 1.86s/it] | |
| 2024-08-07 19:32:34,219 - INFO: validation progress: 8%|8 | 6/75 [00:07<01:13, 1.07s/it] | |
| 2024-08-07 19:32:35,650 - INFO: validation progress: 12%|#2 | 9/75 [00:08<00:52, 1.25it/s] | |
| 2024-08-07 19:32:37,165 - INFO: validation progress: 16%|#6 | 12/75 [00:10<00:43, 1.46it/s] | |
| 2024-08-07 19:32:38,769 - INFO: validation progress: 20%|## | 15/75 [00:11<00:37, 1.59it/s] | |
| 2024-08-07 19:32:40,325 - INFO: validation progress: 24%|##4 | 18/75 [00:13<00:33, 1.69it/s] | |
| 2024-08-07 19:32:41,955 - INFO: validation progress: 28%|##8 | 21/75 [00:14<00:31, 1.74it/s] | |
| 2024-08-07 19:32:43,448 - INFO: validation progress: 32%|###2 | 24/75 [00:16<00:28, 1.81it/s] | |
| 2024-08-07 19:32:45,091 - INFO: validation progress: 36%|###6 | 27/75 [00:18<00:26, 1.82it/s] | |
| 2024-08-07 19:32:46,670 - INFO: validation progress: 40%|#### | 30/75 [00:19<00:24, 1.84it/s] | |
| 2024-08-07 19:32:48,303 - INFO: validation progress: 44%|####4 | 33/75 [00:21<00:22, 1.84it/s] | |
| 2024-08-07 19:32:49,939 - INFO: validation progress: 48%|####8 | 36/75 [00:22<00:21, 1.84it/s] | |
| 2024-08-07 19:32:51,601 - INFO: validation progress: 52%|#####2 | 39/75 [00:24<00:19, 1.83it/s] | |
| 2024-08-07 19:32:53,177 - INFO: validation progress: 56%|#####6 | 42/75 [00:26<00:17, 1.85it/s] | |
| 2024-08-07 19:32:54,726 - INFO: validation progress: 60%|###### | 45/75 [00:27<00:15, 1.88it/s] | |
| 2024-08-07 19:32:56,182 - INFO: validation progress: 64%|######4 | 48/75 [00:29<00:14, 1.93it/s] | |
| 2024-08-07 19:32:57,686 - INFO: validation progress: 68%|######8 | 51/75 [00:30<00:12, 1.95it/s] | |
| 2024-08-07 19:32:59,264 - INFO: validation progress: 72%|#######2 | 54/75 [00:32<00:10, 1.93it/s] | |
| 2024-08-07 19:33:00,842 - INFO: validation progress: 76%|#######6 | 57/75 [00:33<00:09, 1.92it/s] | |
| 2024-08-07 19:33:02,285 - INFO: validation progress: 80%|######## | 60/75 [00:35<00:07, 1.97it/s] | |
| 2024-08-07 19:33:04,031 - INFO: validation progress: 84%|########4 | 63/75 [00:36<00:06, 1.89it/s] | |
| 2024-08-07 19:33:05,661 - INFO: validation progress: 88%|########8 | 66/75 [00:38<00:04, 1.87it/s] | |
| 2024-08-07 19:33:07,338 - INFO: validation progress: 92%|#########2| 69/75 [00:40<00:03, 1.85it/s] | |
| 2024-08-07 19:33:08,800 - INFO: validation progress: 96%|#########6| 72/75 [00:41<00:01, 1.90it/s] | |
| 2024-08-07 19:33:10,543 - INFO: validation progress: 100%|##########| 75/75 [00:43<00:00, 1.84it/s] | |
| 2024-08-07 19:33:10,546 - INFO: validation progress: 100%|##########| 75/75 [00:43<00:00, 1.73it/s] | |
| 2024-08-07 19:33:10,819 - INFO: Validation AUC: 0.97624 | |
| 2024-08-07 19:33:10,819 - INFO: Mean validation loss: 0.50932 | |
| 2024-08-07 19:33:11,230 - INFO: Training Epoch: 4 / 10 | |
| 2024-08-07 19:33:11,231 - INFO: train loss: 0%| | 0/300 [00:00<?, ?it/s] | |
| 2024-08-07 19:33:15,018 - INFO: Evaluation step: 300 | |
| 2024-08-07 19:33:20,645 - INFO: train loss: 0.39: 1%|1 | 3/300 [00:09<15:32, 3.14s/it] | |
| 2024-08-07 19:33:25,436 - INFO: train loss: 1.11: 2%|2 | 6/300 [00:14<10:56, 2.23s/it] | |
| 2024-08-07 19:33:30,384 - INFO: train loss: 0.87: 3%|3 | 9/300 [00:19<09:31, 1.97s/it] | |
| 2024-08-07 19:33:36,617 - INFO: train loss: 0.93: 4%|4 | 12/300 [00:25<09:38, 2.01s/it] | |
| 2024-08-07 19:33:42,136 - INFO: train loss: 0.64: 5%|5 | 15/300 [00:30<09:15, 1.95s/it] | |
| 2024-08-07 19:33:47,102 - INFO: train loss: 0.46: 6%|6 | 18/300 [00:35<08:41, 1.85s/it] | |
| 2024-08-07 19:33:51,875 - INFO: train loss: 0.42: 7%|7 | 21/300 [00:40<08:12, 1.76s/it] | |
| 2024-08-07 19:33:57,004 - INFO: train loss: 0.28: 8%|8 | 24/300 [00:45<08:02, 1.75s/it] | |
| 2024-08-07 19:34:02,259 - INFO: train loss: 0.38: 9%|9 | 27/300 [00:51<07:57, 1.75s/it] | |
| 2024-08-07 19:34:06,937 - INFO: train loss: 0.35: 10%|# | 30/300 [00:55<07:36, 1.69s/it] | |
| 2024-08-07 19:34:11,642 - INFO: train loss: 0.37: 11%|#1 | 33/300 [01:00<07:21, 1.65s/it] | |
| 2024-08-07 19:34:16,615 - INFO: train loss: 0.35: 12%|#2 | 36/300 [01:05<07:16, 1.65s/it] | |
| 2024-08-07 19:34:21,481 - INFO: train loss: 0.20: 13%|#3 | 39/300 [01:10<07:09, 1.64s/it] | |
| 2024-08-07 19:34:26,315 - INFO: train loss: 0.17: 14%|#4 | 42/300 [01:15<07:01, 1.63s/it] | |
| 2024-08-07 19:34:31,265 - INFO: train loss: 0.16: 15%|#5 | 45/300 [01:20<06:57, 1.64s/it] | |
| 2024-08-07 19:34:36,194 - INFO: train loss: 0.27: 16%|#6 | 48/300 [01:24<06:53, 1.64s/it] | |
| 2024-08-07 19:34:41,610 - INFO: train loss: 0.28: 17%|#7 | 51/300 [01:30<07:00, 1.69s/it] | |
| 2024-08-07 19:34:46,602 - INFO: train loss: 0.33: 18%|#8 | 54/300 [01:35<06:53, 1.68s/it] | |
| 2024-08-07 19:34:51,869 - INFO: train loss: 0.31: 19%|#9 | 57/300 [01:40<06:54, 1.70s/it] | |
| 2024-08-07 19:34:57,393 - INFO: train loss: 0.27: 20%|## | 60/300 [01:46<06:58, 1.75s/it] | |
| 2024-08-07 19:35:02,286 - INFO: train loss: 0.27: 21%|##1 | 63/300 [01:51<06:45, 1.71s/it] | |
| 2024-08-07 19:35:07,319 - INFO: train loss: 0.19: 22%|##2 | 66/300 [01:56<06:38, 1.70s/it] | |
| 2024-08-07 19:35:12,464 - INFO: train loss: 0.20: 23%|##3 | 69/300 [02:01<06:33, 1.71s/it] | |
| 2024-08-07 19:35:17,863 - INFO: train loss: 0.29: 24%|##4 | 72/300 [02:06<06:35, 1.73s/it] | |
| 2024-08-07 19:35:22,551 - INFO: train loss: 0.44: 25%|##5 | 75/300 [02:11<06:18, 1.68s/it] | |
| 2024-08-07 19:35:27,155 - INFO: train loss: 0.49: 26%|##6 | 78/300 [02:15<06:03, 1.64s/it] | |
| 2024-08-07 19:35:32,257 - INFO: train loss: 0.43: 27%|##7 | 81/300 [02:21<06:02, 1.66s/it] | |
| 2024-08-07 19:35:37,455 - INFO: train loss: 0.25: 28%|##8 | 84/300 [02:26<06:02, 1.68s/it] | |
| 2024-08-07 19:35:42,698 - INFO: train loss: 0.16: 29%|##9 | 87/300 [02:31<06:02, 1.70s/it] | |
| 2024-08-07 19:35:47,073 - INFO: train loss: 0.13: 30%|### | 90/300 [02:35<05:41, 1.63s/it] | |
| 2024-08-07 19:35:52,294 - INFO: train loss: 0.10: 31%|###1 | 93/300 [02:41<05:43, 1.66s/it] | |
| 2024-08-07 19:35:57,477 - INFO: train loss: 0.21: 32%|###2 | 96/300 [02:46<05:42, 1.68s/it] | |
| 2024-08-07 19:36:02,234 - INFO: train loss: 0.26: 33%|###3 | 99/300 [02:51<05:32, 1.65s/it] | |
| 2024-08-07 19:36:06,754 - INFO: train loss: 0.25: 34%|###4 | 102/300 [02:55<05:18, 1.61s/it] | |
| 2024-08-07 19:36:11,705 - INFO: train loss: 0.29: 35%|###5 | 105/300 [03:00<05:16, 1.62s/it] | |
| 2024-08-07 19:36:16,678 - INFO: train loss: 0.34: 36%|###6 | 108/300 [03:05<05:13, 1.63s/it] | |
| 2024-08-07 19:36:21,317 - INFO: train loss: 0.34: 37%|###7 | 111/300 [03:10<05:03, 1.61s/it] | |
| 2024-08-07 19:36:26,149 - INFO: train loss: 0.39: 38%|###8 | 114/300 [03:14<04:59, 1.61s/it] | |
| 2024-08-07 19:36:31,358 - INFO: train loss: 0.33: 39%|###9 | 117/300 [03:20<05:01, 1.65s/it] | |
| 2024-08-07 19:36:36,841 - INFO: train loss: 0.51: 40%|#### | 120/300 [03:25<05:06, 1.70s/it] | |
| 2024-08-07 19:36:42,087 - INFO: train loss: 0.39: 41%|####1 | 123/300 [03:30<05:03, 1.72s/it] | |
| 2024-08-07 19:36:47,409 - INFO: train loss: 0.41: 42%|####2 | 126/300 [03:36<05:01, 1.73s/it] | |
| 2024-08-07 19:36:52,441 - INFO: train loss: 0.30: 43%|####3 | 129/300 [03:41<04:53, 1.72s/it] | |
| 2024-08-07 19:36:58,205 - INFO: train loss: 0.25: 44%|####4 | 132/300 [03:46<04:58, 1.78s/it] | |
| 2024-08-07 19:37:03,268 - INFO: train loss: 0.33: 45%|####5 | 135/300 [03:52<04:48, 1.75s/it] | |
| 2024-08-07 19:37:08,495 - INFO: train loss: 0.46: 46%|####6 | 138/300 [03:57<04:43, 1.75s/it] | |
| 2024-08-07 19:37:13,355 - INFO: train loss: 0.57: 47%|####6 | 141/300 [04:02<04:31, 1.71s/it] | |
| 2024-08-07 19:37:18,077 - INFO: train loss: 0.57: 48%|####8 | 144/300 [04:06<04:20, 1.67s/it] | |
| 2024-08-07 19:37:23,259 - INFO: train loss: 0.47: 49%|####9 | 147/300 [04:12<04:18, 1.69s/it] | |
| 2024-08-07 19:37:28,419 - INFO: train loss: 0.38: 50%|##### | 150/300 [04:17<04:14, 1.70s/it] | |
| 2024-08-07 19:37:33,391 - INFO: train loss: 0.30: 51%|#####1 | 153/300 [04:22<04:07, 1.68s/it] | |
| 2024-08-07 19:37:38,529 - INFO: train loss: 0.18: 52%|#####2 | 156/300 [04:27<04:03, 1.69s/it] | |
| 2024-08-07 19:37:43,360 - INFO: train loss: 0.20: 53%|#####3 | 159/300 [04:32<03:55, 1.67s/it] | |
| 2024-08-07 19:37:48,457 - INFO: train loss: 0.24: 54%|#####4 | 162/300 [04:37<03:51, 1.68s/it] | |
| 2024-08-07 19:37:53,495 - INFO: train loss: 0.24: 55%|#####5 | 165/300 [04:42<03:46, 1.68s/it] | |
| 2024-08-07 19:37:58,752 - INFO: train loss: 0.30: 56%|#####6 | 168/300 [04:47<03:44, 1.70s/it] | |
| 2024-08-07 19:38:03,605 - INFO: train loss: 0.25: 57%|#####6 | 171/300 [04:52<03:36, 1.68s/it] | |
| 2024-08-07 19:38:08,535 - INFO: train loss: 0.24: 58%|#####8 | 174/300 [04:57<03:29, 1.67s/it] | |
| 2024-08-07 19:38:13,758 - INFO: train loss: 0.26: 59%|#####8 | 177/300 [05:02<03:27, 1.69s/it] | |
| 2024-08-07 19:38:18,430 - INFO: train loss: 0.23: 60%|###### | 180/300 [05:07<03:17, 1.65s/it] | |
| 2024-08-07 19:38:23,276 - INFO: train loss: 0.23: 61%|######1 | 183/300 [05:12<03:11, 1.64s/it] | |
| 2024-08-07 19:38:27,629 - INFO: train loss: 0.22: 62%|######2 | 186/300 [05:16<03:00, 1.58s/it] | |
| 2024-08-07 19:38:32,161 - INFO: train loss: 0.20: 63%|######3 | 189/300 [05:20<02:53, 1.56s/it] | |
| 2024-08-07 19:38:37,103 - INFO: train loss: 0.17: 64%|######4 | 192/300 [05:25<02:51, 1.59s/it] | |
| 2024-08-07 19:38:42,118 - INFO: train loss: 0.14: 65%|######5 | 195/300 [05:30<02:49, 1.61s/it] | |
| 2024-08-07 19:38:47,364 - INFO: train loss: 0.13: 66%|######6 | 198/300 [05:36<02:48, 1.65s/it] | |
| 2024-08-07 19:38:52,135 - INFO: train loss: 0.24: 67%|######7 | 201/300 [05:40<02:41, 1.63s/it] | |
| 2024-08-07 19:38:57,313 - INFO: train loss: 0.28: 68%|######8 | 204/300 [05:46<02:39, 1.66s/it] | |
| 2024-08-07 19:39:02,442 - INFO: train loss: 0.25: 69%|######9 | 207/300 [05:51<02:35, 1.68s/it] | |
| 2024-08-07 19:39:07,498 - INFO: train loss: 0.17: 70%|####### | 210/300 [05:56<02:31, 1.68s/it] | |
| 2024-08-07 19:39:12,446 - INFO: train loss: 0.21: 71%|#######1 | 213/300 [06:01<02:25, 1.67s/it] | |
| 2024-08-07 19:39:17,397 - INFO: train loss: 0.23: 72%|#######2 | 216/300 [06:06<02:19, 1.66s/it] | |
| 2024-08-07 19:39:22,203 - INFO: train loss: 0.31: 73%|#######3 | 219/300 [06:10<02:13, 1.65s/it] | |
| 2024-08-07 19:39:27,278 - INFO: train loss: 0.36: 74%|#######4 | 222/300 [06:16<02:09, 1.66s/it] | |
| 2024-08-07 19:39:32,219 - INFO: train loss: 0.29: 75%|#######5 | 225/300 [06:20<02:04, 1.66s/it] | |
| 2024-08-07 19:39:37,287 - INFO: train loss: 0.38: 76%|#######6 | 228/300 [06:26<01:59, 1.67s/it] | |
| 2024-08-07 19:39:41,725 - INFO: train loss: 0.33: 77%|#######7 | 231/300 [06:30<01:51, 1.61s/it] | |
| 2024-08-07 19:39:46,445 - INFO: train loss: 0.44: 78%|#######8 | 234/300 [06:35<01:45, 1.60s/it] | |
| 2024-08-07 19:39:51,907 - INFO: train loss: 0.36: 79%|#######9 | 237/300 [06:40<01:44, 1.67s/it] | |
| 2024-08-07 19:39:56,707 - INFO: train loss: 0.34: 80%|######## | 240/300 [06:45<01:38, 1.65s/it] | |
| 2024-08-07 19:40:02,407 - INFO: train loss: 0.34: 81%|########1 | 243/300 [06:51<01:38, 1.72s/it] | |
| 2024-08-07 19:40:07,586 - INFO: train loss: 0.36: 82%|########2 | 246/300 [06:56<01:33, 1.72s/it] | |
| 2024-08-07 19:40:12,053 - INFO: train loss: 0.39: 83%|########2 | 249/300 [07:00<01:24, 1.65s/it] | |
| 2024-08-07 19:40:16,976 - INFO: train loss: 0.35: 84%|########4 | 252/300 [07:05<01:19, 1.65s/it] | |
| 2024-08-07 19:40:22,063 - INFO: train loss: 0.31: 85%|########5 | 255/300 [07:10<01:14, 1.66s/it] | |
| 2024-08-07 19:40:27,136 - INFO: train loss: 0.34: 86%|########6 | 258/300 [07:15<01:10, 1.67s/it] | |
| 2024-08-07 19:40:31,927 - INFO: train loss: 0.31: 87%|########7 | 261/300 [07:20<01:04, 1.65s/it] | |
| 2024-08-07 19:40:37,124 - INFO: train loss: 0.32: 88%|########8 | 264/300 [07:25<01:00, 1.67s/it] | |
| 2024-08-07 19:40:42,257 - INFO: train loss: 0.29: 89%|########9 | 267/300 [07:31<00:55, 1.69s/it] | |
| 2024-08-07 19:40:47,569 - INFO: train loss: 0.21: 90%|######### | 270/300 [07:36<00:51, 1.71s/it] | |
| 2024-08-07 19:40:52,816 - INFO: train loss: 0.25: 91%|#########1| 273/300 [07:41<00:46, 1.72s/it] | |
| 2024-08-07 19:40:57,781 - INFO: train loss: 0.35: 92%|#########2| 276/300 [07:46<00:40, 1.70s/it] | |
| 2024-08-07 19:41:02,562 - INFO: train loss: 0.33: 93%|#########3| 279/300 [07:51<00:35, 1.67s/it] | |
| 2024-08-07 19:41:07,783 - INFO: train loss: 0.37: 94%|#########3| 282/300 [07:56<00:30, 1.69s/it] | |
| 2024-08-07 19:41:12,864 - INFO: train loss: 0.22: 95%|#########5| 285/300 [08:01<00:25, 1.69s/it] | |
| 2024-08-07 19:41:17,798 - INFO: train loss: 0.27: 96%|#########6| 288/300 [08:06<00:20, 1.68s/it] | |
| 2024-08-07 19:41:23,540 - INFO: train loss: 0.37: 97%|#########7| 291/300 [08:12<00:15, 1.75s/it] | |
| 2024-08-07 19:41:28,276 - INFO: train loss: 0.36: 98%|#########8| 294/300 [08:17<00:10, 1.70s/it] | |
| 2024-08-07 19:41:33,224 - INFO: train loss: 0.41: 99%|#########9| 297/300 [08:21<00:05, 1.68s/it] | |
| 2024-08-07 19:41:39,216 - INFO: train loss: 0.36: 100%|##########| 300/300 [08:27<00:00, 1.78s/it] | |
| 2024-08-07 19:41:39,216 - INFO: train loss: 0.36: 100%|##########| 300/300 [08:27<00:00, 1.69s/it] | |
| 2024-08-07 19:41:39,216 - INFO: Saving last model checkpoint to /app/output | |
| 2024-08-07 19:41:39,216 - INFO: Saving checkpoint.. | |
| 2024-08-07 19:42:06,037 - INFO: Starting validation inference | |
| 2024-08-07 19:42:06,038 - INFO: validation progress: 0%| | 0/75 [00:00<?, ?it/s] | |
| 2024-08-07 19:42:11,243 - INFO: validation progress: 4%|4 | 3/75 [00:05<02:04, 1.73s/it] | |
| 2024-08-07 19:42:12,763 - INFO: validation progress: 8%|8 | 6/75 [00:06<01:09, 1.01s/it] | |
| 2024-08-07 19:42:14,196 - INFO: validation progress: 12%|#2 | 9/75 [00:08<00:50, 1.30it/s] | |
| 2024-08-07 19:42:15,696 - INFO: validation progress: 16%|#6 | 12/75 [00:09<00:41, 1.51it/s] | |
| 2024-08-07 19:42:17,301 - INFO: validation progress: 20%|## | 15/75 [00:11<00:36, 1.62it/s] | |
| 2024-08-07 19:42:18,857 - INFO: validation progress: 24%|##4 | 18/75 [00:12<00:33, 1.71it/s] | |
| 2024-08-07 19:42:20,493 - INFO: validation progress: 28%|##8 | 21/75 [00:14<00:30, 1.75it/s] | |
| 2024-08-07 19:42:21,990 - INFO: validation progress: 32%|###2 | 24/75 [00:15<00:27, 1.83it/s] | |
| 2024-08-07 19:42:23,631 - INFO: validation progress: 36%|###6 | 27/75 [00:17<00:26, 1.83it/s] | |
| 2024-08-07 19:42:25,210 - INFO: validation progress: 40%|#### | 30/75 [00:19<00:24, 1.85it/s] | |
| 2024-08-07 19:42:26,849 - INFO: validation progress: 44%|####4 | 33/75 [00:20<00:22, 1.84it/s] | |
| 2024-08-07 19:42:28,525 - INFO: validation progress: 48%|####8 | 36/75 [00:22<00:21, 1.83it/s] | |
| 2024-08-07 19:42:30,182 - INFO: validation progress: 52%|#####2 | 39/75 [00:24<00:19, 1.82it/s] | |
| 2024-08-07 19:42:31,759 - INFO: validation progress: 56%|#####6 | 42/75 [00:25<00:17, 1.85it/s] | |
| 2024-08-07 19:42:33,355 - INFO: validation progress: 60%|###### | 45/75 [00:27<00:16, 1.86it/s] | |
| 2024-08-07 19:42:34,826 - INFO: validation progress: 64%|######4 | 48/75 [00:28<00:14, 1.91it/s] | |
| 2024-08-07 19:42:36,330 - INFO: validation progress: 68%|######8 | 51/75 [00:30<00:12, 1.93it/s] | |
| 2024-08-07 19:42:37,907 - INFO: validation progress: 72%|#######2 | 54/75 [00:31<00:10, 1.92it/s] | |
| 2024-08-07 19:42:39,480 - INFO: validation progress: 76%|#######6 | 57/75 [00:33<00:09, 1.92it/s] | |
| 2024-08-07 19:42:40,919 - INFO: validation progress: 80%|######## | 60/75 [00:34<00:07, 1.97it/s] | |
| 2024-08-07 19:42:42,624 - INFO: validation progress: 84%|########4 | 63/75 [00:36<00:06, 1.90it/s] | |
| 2024-08-07 19:42:44,257 - INFO: validation progress: 88%|########8 | 66/75 [00:38<00:04, 1.88it/s] | |
| 2024-08-07 19:42:45,889 - INFO: validation progress: 92%|#########2| 69/75 [00:39<00:03, 1.87it/s] | |
| 2024-08-07 19:42:47,351 - INFO: validation progress: 96%|#########6| 72/75 [00:41<00:01, 1.92it/s] | |
| 2024-08-07 19:42:49,081 - INFO: validation progress: 100%|##########| 75/75 [00:43<00:00, 1.86it/s] | |
| 2024-08-07 19:42:49,083 - INFO: validation progress: 100%|##########| 75/75 [00:43<00:00, 1.74it/s] | |
| 2024-08-07 19:42:49,341 - INFO: Validation AUC: 0.97709 | |
| 2024-08-07 19:42:49,342 - INFO: Mean validation loss: 0.49407 | |
| 2024-08-07 19:42:49,692 - INFO: Training Epoch: 5 / 10 | |
| 2024-08-07 19:42:49,693 - INFO: train loss: 0%| | 0/300 [00:00<?, ?it/s] | |
| 2024-08-07 19:42:53,545 - INFO: Evaluation step: 300 | |
| 2024-08-07 19:42:59,059 - INFO: train loss: 0.27: 1%|1 | 3/300 [00:09<15:27, 3.12s/it] | |
| 2024-08-07 19:43:04,254 - INFO: train loss: 0.22: 2%|2 | 6/300 [00:14<11:17, 2.30s/it] | |
| 2024-08-07 19:43:09,394 - INFO: train loss: 0.24: 3%|3 | 9/300 [00:19<09:51, 2.03s/it] | |
| 2024-08-07 19:43:14,267 - INFO: train loss: 0.19: 4%|4 | 12/300 [00:24<08:59, 1.87s/it] | |
| 2024-08-07 19:43:19,273 - INFO: train loss: 0.19: 5%|5 | 15/300 [00:29<08:32, 1.80s/it] | |
| 2024-08-07 19:43:24,126 - INFO: train loss: 0.17: 6%|6 | 18/300 [00:34<08:09, 1.74s/it] | |
| 2024-08-07 19:43:28,745 - INFO: train loss: 0.19: 7%|7 | 21/300 [00:39<07:46, 1.67s/it] | |
| 2024-08-07 19:43:33,851 - INFO: train loss: 0.18: 8%|8 | 24/300 [00:44<07:44, 1.68s/it] | |
| 2024-08-07 19:43:38,913 - INFO: train loss: 0.19: 9%|9 | 27/300 [00:49<07:39, 1.68s/it] | |
| 2024-08-07 19:43:43,893 - INFO: train loss: 0.19: 10%|# | 30/300 [00:54<07:32, 1.68s/it] | |
| 2024-08-07 19:43:48,883 - INFO: train loss: 0.12: 11%|#1 | 33/300 [00:59<07:26, 1.67s/it] | |
| 2024-08-07 19:43:53,384 - INFO: train loss: 0.13: 12%|#2 | 36/300 [01:03<07:07, 1.62s/it] | |
| 2024-08-07 19:43:58,510 - INFO: train loss: 0.15: 13%|#3 | 39/300 [01:08<07:09, 1.65s/it] | |
| 2024-08-07 19:44:03,554 - INFO: train loss: 0.16: 14%|#4 | 42/300 [01:13<07:07, 1.66s/it] | |
| 2024-08-07 19:44:08,495 - INFO: train loss: 0.17: 15%|#5 | 45/300 [01:18<07:01, 1.65s/it] | |
| 2024-08-07 19:44:13,812 - INFO: train loss: 0.24: 16%|#6 | 48/300 [01:24<07:05, 1.69s/it] | |
| 2024-08-07 19:44:18,769 - INFO: train loss: 0.29: 17%|#7 | 51/300 [01:29<06:57, 1.68s/it] | |
| 2024-08-07 19:44:25,098 - INFO: train loss: 0.25: 18%|#8 | 54/300 [01:35<07:24, 1.81s/it] | |
| 2024-08-07 19:44:31,233 - INFO: train loss: 0.25: 19%|#9 | 57/300 [01:41<07:36, 1.88s/it] | |
| 2024-08-07 19:44:36,038 - INFO: train loss: 0.24: 20%|## | 60/300 [01:46<07:11, 1.80s/it] | |
| 2024-08-07 19:44:41,114 - INFO: train loss: 0.26: 21%|##1 | 63/300 [01:51<06:58, 1.76s/it] | |
| 2024-08-07 19:44:46,314 - INFO: train loss: 0.34: 22%|##2 | 66/300 [01:56<06:50, 1.76s/it] | |
| 2024-08-07 19:44:51,077 - INFO: train loss: 0.17: 23%|##3 | 69/300 [02:01<06:33, 1.70s/it] | |
| 2024-08-07 19:44:56,250 - INFO: train loss: 0.26: 24%|##4 | 72/300 [02:06<06:30, 1.71s/it] | |
| 2024-08-07 19:45:01,152 - INFO: train loss: 0.24: 25%|##5 | 75/300 [02:11<06:19, 1.69s/it] | |
| 2024-08-07 19:45:06,159 - INFO: train loss: 0.20: 26%|##6 | 78/300 [02:16<06:13, 1.68s/it] | |
| 2024-08-07 19:45:11,262 - INFO: train loss: 0.18: 27%|##7 | 81/300 [02:21<06:09, 1.69s/it] | |
| 2024-08-07 19:45:16,237 - INFO: train loss: 0.27: 28%|##8 | 84/300 [02:26<06:02, 1.68s/it] | |
| 2024-08-07 19:45:21,794 - INFO: train loss: 0.25: 29%|##9 | 87/300 [02:32<06:08, 1.73s/it] | |
| 2024-08-07 19:45:26,743 - INFO: train loss: 0.19: 30%|### | 90/300 [02:37<05:58, 1.71s/it] | |
| 2024-08-07 19:45:31,472 - INFO: train loss: 0.20: 31%|###1 | 93/300 [02:41<05:45, 1.67s/it] | |
| 2024-08-07 19:45:36,431 - INFO: train loss: 0.20: 32%|###2 | 96/300 [02:46<05:39, 1.66s/it] | |
| 2024-08-07 19:45:41,053 - INFO: train loss: 0.23: 33%|###3 | 99/300 [02:51<05:26, 1.63s/it] | |
| 2024-08-07 19:45:46,010 - INFO: train loss: 0.22: 34%|###4 | 102/300 [02:56<05:23, 1.63s/it] | |
| 2024-08-07 19:45:50,795 - INFO: train loss: 0.19: 35%|###5 | 105/300 [03:01<05:16, 1.62s/it] | |
| 2024-08-07 19:45:55,775 - INFO: train loss: 0.21: 36%|###6 | 108/300 [03:06<05:13, 1.63s/it] | |
| 2024-08-07 19:46:00,725 - INFO: train loss: 0.23: 37%|###7 | 111/300 [03:11<05:09, 1.64s/it] | |
| 2024-08-07 19:46:05,596 - INFO: train loss: 0.19: 38%|###8 | 114/300 [03:15<05:03, 1.63s/it] | |
| 2024-08-07 19:46:11,054 - INFO: train loss: 0.16: 39%|###9 | 117/300 [03:21<05:09, 1.69s/it] | |
| 2024-08-07 19:46:15,886 - INFO: train loss: 0.21: 40%|#### | 120/300 [03:26<04:59, 1.67s/it] | |
| 2024-08-07 19:46:20,979 - INFO: train loss: 0.24: 41%|####1 | 123/300 [03:31<04:56, 1.68s/it] | |
| 2024-08-07 19:46:25,838 - INFO: train loss: 0.30: 42%|####2 | 126/300 [03:36<04:48, 1.66s/it] | |
| 2024-08-07 19:46:31,170 - INFO: train loss: 0.26: 43%|####3 | 129/300 [03:41<04:49, 1.69s/it] | |
| 2024-08-07 19:46:36,361 - INFO: train loss: 0.23: 44%|####4 | 132/300 [03:46<04:46, 1.71s/it] | |
| 2024-08-07 19:46:40,804 - INFO: train loss: 0.18: 45%|####5 | 135/300 [03:51<04:30, 1.64s/it] | |
| 2024-08-07 19:46:45,981 - INFO: train loss: 0.23: 46%|####6 | 138/300 [03:56<04:29, 1.66s/it] | |
| 2024-08-07 19:46:50,791 - INFO: train loss: 0.20: 47%|####6 | 141/300 [04:01<04:21, 1.65s/it] | |
| 2024-08-07 19:46:55,915 - INFO: train loss: 0.25: 48%|####8 | 144/300 [04:06<04:19, 1.66s/it] | |
| 2024-08-07 19:47:00,823 - INFO: train loss: 0.36: 49%|####9 | 147/300 [04:11<04:13, 1.66s/it] | |
| 2024-08-07 19:47:06,227 - INFO: train loss: 0.33: 50%|##### | 150/300 [04:16<04:14, 1.70s/it] | |
| 2024-08-07 19:47:10,822 - INFO: train loss: 0.33: 51%|#####1 | 153/300 [04:21<04:02, 1.65s/it] | |
| 2024-08-07 19:47:15,565 - INFO: train loss: 0.28: 52%|#####2 | 156/300 [04:25<03:54, 1.63s/it] | |
| 2024-08-07 19:47:20,505 - INFO: train loss: 0.27: 53%|#####3 | 159/300 [04:30<03:50, 1.63s/it] | |
| 2024-08-07 19:47:25,483 - INFO: train loss: 0.26: 54%|#####4 | 162/300 [04:35<03:46, 1.64s/it] | |
| 2024-08-07 19:47:30,516 - INFO: train loss: 0.29: 55%|#####5 | 165/300 [04:40<03:43, 1.65s/it] | |
| 2024-08-07 19:47:35,611 - INFO: train loss: 0.32: 56%|#####6 | 168/300 [04:45<03:39, 1.67s/it] | |
| 2024-08-07 19:47:40,979 - INFO: train loss: 0.30: 57%|#####6 | 171/300 [04:51<03:39, 1.70s/it] | |
| 2024-08-07 19:47:45,953 - INFO: train loss: 0.35: 58%|#####8 | 174/300 [04:56<03:32, 1.69s/it] | |
| 2024-08-07 19:47:50,964 - INFO: train loss: 0.28: 59%|#####8 | 177/300 [05:01<03:27, 1.68s/it] | |
| 2024-08-07 19:47:56,046 - INFO: train loss: 0.17: 60%|###### | 180/300 [05:06<03:22, 1.69s/it] | |
| 2024-08-07 19:48:01,586 - INFO: train loss: 0.15: 61%|######1 | 183/300 [05:11<03:22, 1.73s/it] | |
| 2024-08-07 19:48:06,425 - INFO: train loss: 0.12: 62%|######2 | 186/300 [05:16<03:13, 1.70s/it] | |
| 2024-08-07 19:48:11,543 - INFO: train loss: 0.14: 63%|######3 | 189/300 [05:21<03:08, 1.70s/it] | |
| 2024-08-07 19:48:16,500 - INFO: train loss: 0.13: 64%|######4 | 192/300 [05:26<03:02, 1.69s/it] | |
| 2024-08-07 19:48:22,043 - INFO: train loss: 0.20: 65%|######5 | 195/300 [05:32<03:02, 1.73s/it] | |
| 2024-08-07 19:48:27,388 - INFO: train loss: 0.14: 66%|######6 | 198/300 [05:37<02:58, 1.75s/it] | |
| 2024-08-07 19:48:32,725 - INFO: train loss: 0.21: 67%|######7 | 201/300 [05:43<02:54, 1.76s/it] | |
| 2024-08-07 19:48:37,695 - INFO: train loss: 0.22: 68%|######8 | 204/300 [05:48<02:45, 1.73s/it] | |
| 2024-08-07 19:48:42,860 - INFO: train loss: 0.25: 69%|######9 | 207/300 [05:53<02:40, 1.73s/it] | |
| 2024-08-07 19:48:48,009 - INFO: train loss: 0.20: 70%|####### | 210/300 [05:58<02:35, 1.72s/it] | |
| 2024-08-07 19:48:53,005 - INFO: train loss: 0.21: 71%|#######1 | 213/300 [06:03<02:28, 1.71s/it] | |
| 2024-08-07 19:48:57,718 - INFO: train loss: 0.21: 72%|#######2 | 216/300 [06:08<02:19, 1.67s/it] | |
| 2024-08-07 19:49:02,947 - INFO: train loss: 0.23: 73%|#######3 | 219/300 [06:13<02:16, 1.69s/it] | |
| 2024-08-07 19:49:09,310 - INFO: train loss: 0.16: 74%|#######4 | 222/300 [06:19<02:21, 1.82s/it] | |
| 2024-08-07 19:49:14,469 - INFO: train loss: 0.19: 75%|#######5 | 225/300 [06:24<02:14, 1.79s/it] | |
| 2024-08-07 19:49:19,617 - INFO: train loss: 0.16: 76%|#######6 | 228/300 [06:29<02:07, 1.77s/it] | |
| 2024-08-07 19:49:24,787 - INFO: train loss: 0.17: 77%|#######7 | 231/300 [06:35<02:01, 1.75s/it] | |
| 2024-08-07 19:49:30,512 - INFO: train loss: 0.23: 78%|#######8 | 234/300 [06:40<01:58, 1.80s/it] | |
| 2024-08-07 19:49:35,720 - INFO: train loss: 0.26: 79%|#######9 | 237/300 [06:46<01:52, 1.78s/it] | |
| 2024-08-07 19:49:40,492 - INFO: train loss: 0.31: 80%|######## | 240/300 [06:50<01:43, 1.72s/it] | |
| 2024-08-07 19:49:45,402 - INFO: train loss: 0.27: 81%|########1 | 243/300 [06:55<01:36, 1.70s/it] | |
| 2024-08-07 19:49:50,438 - INFO: train loss: 0.25: 82%|########2 | 246/300 [07:00<01:31, 1.69s/it] | |
| 2024-08-07 19:49:55,774 - INFO: train loss: 0.27: 83%|########2 | 249/300 [07:06<01:27, 1.72s/it] | |
| 2024-08-07 19:50:00,819 - INFO: train loss: 0.34: 84%|########4 | 252/300 [07:11<01:21, 1.71s/it] | |
| 2024-08-07 19:50:05,794 - INFO: train loss: 0.35: 85%|########5 | 255/300 [07:16<01:16, 1.69s/it] | |
| 2024-08-07 19:50:10,782 - INFO: train loss: 0.33: 86%|########6 | 258/300 [07:21<01:10, 1.68s/it] | |
| 2024-08-07 19:50:15,516 - INFO: train loss: 0.23: 87%|########7 | 261/300 [07:25<01:04, 1.65s/it] | |
| 2024-08-07 19:50:20,696 - INFO: train loss: 0.15: 88%|########8 | 264/300 [07:31<01:00, 1.67s/it] | |
| 2024-08-07 19:50:25,191 - INFO: train loss: 0.17: 89%|########9 | 267/300 [07:35<00:53, 1.62s/it] | |
| 2024-08-07 19:50:30,131 - INFO: train loss: 0.12: 90%|######### | 270/300 [07:40<00:48, 1.63s/it] | |
| 2024-08-07 19:50:34,990 - INFO: train loss: 0.13: 91%|#########1| 273/300 [07:45<00:43, 1.63s/it] | |
| 2024-08-07 19:50:40,272 - INFO: train loss: 0.14: 92%|#########2| 276/300 [07:50<00:39, 1.67s/it] | |
| 2024-08-07 19:50:45,388 - INFO: train loss: 0.13: 93%|#########3| 279/300 [07:55<00:35, 1.68s/it] | |
| 2024-08-07 19:50:50,418 - INFO: train loss: 0.15: 94%|#########3| 282/300 [08:00<00:30, 1.68s/it] | |
| 2024-08-07 19:50:55,431 - INFO: train loss: 0.18: 95%|#########5| 285/300 [08:05<00:25, 1.68s/it] | |
| 2024-08-07 19:51:00,418 - INFO: train loss: 0.12: 96%|#########6| 288/300 [08:10<00:20, 1.67s/it] | |
| 2024-08-07 19:51:05,401 - INFO: train loss: 0.12: 97%|#########7| 291/300 [08:15<00:15, 1.67s/it] | |
| 2024-08-07 19:51:10,634 - INFO: train loss: 0.11: 98%|#########8| 294/300 [08:20<00:10, 1.69s/it] | |
| 2024-08-07 19:51:15,379 - INFO: train loss: 0.10: 99%|#########9| 297/300 [08:25<00:04, 1.66s/it] | |
| 2024-08-07 19:51:19,991 - INFO: train loss: 0.13: 100%|##########| 300/300 [08:30<00:00, 1.62s/it] | |
| 2024-08-07 19:51:19,991 - INFO: train loss: 0.13: 100%|##########| 300/300 [08:30<00:00, 1.70s/it] | |
| 2024-08-07 19:51:19,991 - INFO: Saving last model checkpoint to /app/output | |
| 2024-08-07 19:51:19,992 - INFO: Saving checkpoint.. | |
| 2024-08-07 19:51:48,053 - INFO: Starting validation inference | |
| 2024-08-07 19:51:48,053 - INFO: validation progress: 0%| | 0/75 [00:00<?, ?it/s] | |
| 2024-08-07 19:51:53,417 - INFO: validation progress: 4%|4 | 3/75 [00:05<02:08, 1.79s/it] | |
| 2024-08-07 19:51:54,934 - INFO: validation progress: 8%|8 | 6/75 [00:06<01:11, 1.03s/it] | |
| 2024-08-07 19:51:56,369 - INFO: validation progress: 12%|#2 | 9/75 [00:08<00:51, 1.28it/s] | |
| 2024-08-07 19:51:57,877 - INFO: validation progress: 16%|#6 | 12/75 [00:09<00:42, 1.49it/s] | |
| 2024-08-07 19:51:59,531 - INFO: validation progress: 20%|## | 15/75 [00:11<00:37, 1.59it/s] | |
| 2024-08-07 19:52:01,132 - INFO: validation progress: 24%|##4 | 18/75 [00:13<00:33, 1.68it/s] | |
| 2024-08-07 19:52:02,775 - INFO: validation progress: 28%|##8 | 21/75 [00:14<00:31, 1.72it/s] | |
| 2024-08-07 19:52:04,267 - INFO: validation progress: 32%|###2 | 24/75 [00:16<00:28, 1.81it/s] | |
| 2024-08-07 19:52:05,909 - INFO: validation progress: 36%|###6 | 27/75 [00:17<00:26, 1.81it/s] | |
| 2024-08-07 19:52:07,490 - INFO: validation progress: 40%|#### | 30/75 [00:19<00:24, 1.84it/s] | |
| 2024-08-07 19:52:09,121 - INFO: validation progress: 44%|####4 | 33/75 [00:21<00:22, 1.84it/s] | |
| 2024-08-07 19:52:10,762 - INFO: validation progress: 48%|####8 | 36/75 [00:22<00:21, 1.84it/s] | |
| 2024-08-07 19:52:12,423 - INFO: validation progress: 52%|#####2 | 39/75 [00:24<00:19, 1.83it/s] | |
| 2024-08-07 19:52:14,011 - INFO: validation progress: 56%|#####6 | 42/75 [00:25<00:17, 1.85it/s] | |
| 2024-08-07 19:52:15,600 - INFO: validation progress: 60%|###### | 45/75 [00:27<00:16, 1.86it/s] | |
| 2024-08-07 19:52:17,059 - INFO: validation progress: 64%|######4 | 48/75 [00:29<00:14, 1.91it/s] | |
| 2024-08-07 19:52:18,565 - INFO: validation progress: 68%|######8 | 51/75 [00:30<00:12, 1.94it/s] | |
| 2024-08-07 19:52:20,141 - INFO: validation progress: 72%|#######2 | 54/75 [00:32<00:10, 1.93it/s] | |
| 2024-08-07 19:52:21,758 - INFO: validation progress: 76%|#######6 | 57/75 [00:33<00:09, 1.90it/s] | |
| 2024-08-07 19:52:23,211 - INFO: validation progress: 80%|######## | 60/75 [00:35<00:07, 1.95it/s] | |
| 2024-08-07 19:52:24,919 - INFO: validation progress: 84%|########4 | 63/75 [00:36<00:06, 1.89it/s] | |
| 2024-08-07 19:52:26,550 - INFO: validation progress: 88%|########8 | 66/75 [00:38<00:04, 1.87it/s] | |
| 2024-08-07 19:52:28,181 - INFO: validation progress: 92%|#########2| 69/75 [00:40<00:03, 1.86it/s] | |
| 2024-08-07 19:52:29,644 - INFO: validation progress: 96%|#########6| 72/75 [00:41<00:01, 1.92it/s] | |
| 2024-08-07 19:52:31,374 - INFO: validation progress: 100%|##########| 75/75 [00:43<00:00, 1.86it/s] | |
| 2024-08-07 19:52:31,378 - INFO: validation progress: 100%|##########| 75/75 [00:43<00:00, 1.73it/s] | |
| 2024-08-07 19:52:31,713 - INFO: Validation AUC: 0.98151 | |
| 2024-08-07 19:52:31,713 - INFO: Mean validation loss: 0.52399 | |
| 2024-08-07 19:52:32,061 - INFO: Training Epoch: 6 / 10 | |
| 2024-08-07 19:52:32,062 - INFO: train loss: 0%| | 0/300 [00:00<?, ?it/s] | |
| 2024-08-07 19:52:35,503 - INFO: Evaluation step: 300 | |
| 2024-08-07 19:52:40,151 - INFO: train loss: 0.06: 1%|1 | 3/300 [00:08<13:20, 2.70s/it] | |
| 2024-08-07 19:52:44,848 - INFO: train loss: 0.10: 2%|2 | 6/300 [00:12<09:57, 2.03s/it] | |
| 2024-08-07 19:52:50,109 - INFO: train loss: 0.09: 3%|3 | 9/300 [00:18<09:14, 1.90s/it] | |
| 2024-08-07 19:52:55,242 - INFO: train loss: 0.11: 4%|4 | 12/300 [00:23<08:46, 1.83s/it] | |
| 2024-08-07 19:53:00,339 - INFO: train loss: 0.15: 5%|5 | 15/300 [00:28<08:27, 1.78s/it] | |
| 2024-08-07 19:53:05,567 - INFO: train loss: 0.13: 6%|6 | 18/300 [00:33<08:18, 1.77s/it] | |
| 2024-08-07 19:53:10,909 - INFO: train loss: 0.11: 7%|7 | 21/300 [00:38<08:14, 1.77s/it] | |
| 2024-08-07 19:53:15,755 - INFO: train loss: 0.09: 8%|8 | 24/300 [00:43<07:55, 1.72s/it] | |
| 2024-08-07 19:53:22,147 - INFO: train loss: 0.10: 9%|9 | 27/300 [00:50<08:25, 1.85s/it] | |
| 2024-08-07 19:53:27,108 - INFO: train loss: 0.10: 10%|# | 30/300 [00:55<08:03, 1.79s/it] | |
| 2024-08-07 19:53:32,237 - INFO: train loss: 0.08: 11%|#1 | 33/300 [01:00<07:51, 1.77s/it] | |
| 2024-08-07 19:53:36,828 - INFO: train loss: 0.08: 12%|#2 | 36/300 [01:04<07:27, 1.69s/it] | |
| 2024-08-07 19:53:41,548 - INFO: train loss: 0.05: 13%|#3 | 39/300 [01:09<07:12, 1.66s/it] | |
| 2024-08-07 19:53:46,136 - INFO: train loss: 0.08: 14%|#4 | 42/300 [01:14<06:57, 1.62s/it] | |
| 2024-08-07 19:53:51,389 - INFO: train loss: 0.14: 15%|#5 | 45/300 [01:19<07:02, 1.66s/it] | |
| 2024-08-07 19:53:56,349 - INFO: train loss: 0.14: 16%|#6 | 48/300 [01:24<06:57, 1.66s/it] | |
| 2024-08-07 19:54:01,609 - INFO: train loss: 0.16: 17%|#7 | 51/300 [01:29<06:59, 1.69s/it] | |
| 2024-08-07 19:54:06,547 - INFO: train loss: 0.15: 18%|#8 | 54/300 [01:34<06:51, 1.67s/it] | |
| 2024-08-07 19:54:11,629 - INFO: train loss: 0.12: 19%|#9 | 57/300 [01:39<06:48, 1.68s/it] | |
| 2024-08-07 19:54:16,770 - INFO: train loss: 0.12: 20%|## | 60/300 [01:44<06:45, 1.69s/it] | |
| 2024-08-07 19:54:21,544 - INFO: train loss: 0.16: 21%|##1 | 63/300 [01:49<06:33, 1.66s/it] | |
| 2024-08-07 19:54:26,496 - INFO: train loss: 0.22: 22%|##2 | 66/300 [01:54<06:27, 1.66s/it] | |
| 2024-08-07 19:54:31,316 - INFO: train loss: 0.25: 23%|##3 | 69/300 [01:59<06:19, 1.64s/it] | |
| 2024-08-07 19:54:36,453 - INFO: train loss: 0.28: 24%|##4 | 72/300 [02:04<06:19, 1.66s/it] | |
| 2024-08-07 19:54:41,257 - INFO: train loss: 0.15: 25%|##5 | 75/300 [02:09<06:10, 1.64s/it] | |
| 2024-08-07 19:54:46,235 - INFO: train loss: 0.10: 26%|##6 | 78/300 [02:14<06:06, 1.65s/it] | |
| 2024-08-07 19:54:51,120 - INFO: train loss: 0.09: 27%|##7 | 81/300 [02:19<05:59, 1.64s/it] | |
| 2024-08-07 19:54:56,442 - INFO: train loss: 0.08: 28%|##8 | 84/300 [02:24<06:03, 1.68s/it] | |
| 2024-08-07 19:55:01,897 - INFO: train loss: 0.12: 29%|##9 | 87/300 [02:29<06:07, 1.72s/it] | |
| 2024-08-07 19:55:06,884 - INFO: train loss: 0.13: 30%|### | 90/300 [02:34<05:58, 1.70s/it] | |
| 2024-08-07 19:55:11,905 - INFO: train loss: 0.16: 31%|###1 | 93/300 [02:39<05:50, 1.70s/it] | |
| 2024-08-07 19:55:16,948 - INFO: train loss: 0.20: 32%|###2 | 96/300 [02:44<05:44, 1.69s/it] | |
| 2024-08-07 19:55:21,933 - INFO: train loss: 0.16: 33%|###3 | 99/300 [02:49<05:38, 1.68s/it] | |
| 2024-08-07 19:55:26,751 - INFO: train loss: 0.24: 34%|###4 | 102/300 [02:54<05:28, 1.66s/it] | |
| 2024-08-07 19:55:31,702 - INFO: train loss: 0.16: 35%|###5 | 105/300 [02:59<05:23, 1.66s/it] | |
| 2024-08-07 19:55:36,914 - INFO: train loss: 0.15: 36%|###6 | 108/300 [03:04<05:22, 1.68s/it] | |
| 2024-08-07 19:55:42,618 - INFO: train loss: 0.05: 37%|###7 | 111/300 [03:10<05:30, 1.75s/it] | |
| 2024-08-07 19:55:47,962 - INFO: train loss: 0.08: 38%|###8 | 114/300 [03:15<05:26, 1.76s/it] | |
| 2024-08-07 19:55:52,860 - INFO: train loss: 0.08: 39%|###9 | 117/300 [03:20<05:14, 1.72s/it] | |
| 2024-08-07 19:55:57,510 - INFO: train loss: 0.09: 40%|#### | 120/300 [03:25<05:00, 1.67s/it] | |
| 2024-08-07 19:56:02,668 - INFO: train loss: 0.09: 41%|####1 | 123/300 [03:30<04:58, 1.68s/it] | |
| 2024-08-07 19:56:07,648 - INFO: train loss: 0.06: 42%|####2 | 126/300 [03:35<04:51, 1.68s/it] | |
| 2024-08-07 19:56:12,724 - INFO: train loss: 0.08: 43%|####3 | 129/300 [03:40<04:47, 1.68s/it] | |
| 2024-08-07 19:56:17,586 - INFO: train loss: 0.05: 44%|####4 | 132/300 [03:45<04:39, 1.66s/it] | |
| 2024-08-07 19:56:23,014 - INFO: train loss: 0.12: 45%|####5 | 135/300 [03:50<04:41, 1.71s/it] | |
| 2024-08-07 19:56:27,863 - INFO: train loss: 0.16: 46%|####6 | 138/300 [03:55<04:32, 1.68s/it] | |
| 2024-08-07 19:56:32,610 - INFO: train loss: 0.14: 47%|####6 | 141/300 [04:00<04:22, 1.65s/it] | |
| 2024-08-07 19:56:38,024 - INFO: train loss: 0.11: 48%|####8 | 144/300 [04:05<04:24, 1.70s/it] | |
| 2024-08-07 19:56:42,919 - INFO: train loss: 0.12: 49%|####9 | 147/300 [04:10<04:16, 1.68s/it] | |
| 2024-08-07 19:56:47,811 - INFO: train loss: 0.12: 50%|##### | 150/300 [04:15<04:09, 1.66s/it] | |
| 2024-08-07 19:56:52,735 - INFO: train loss: 0.12: 51%|#####1 | 153/300 [04:20<04:03, 1.66s/it] | |
| 2024-08-07 19:56:58,101 - INFO: train loss: 0.13: 52%|#####2 | 156/300 [04:26<04:04, 1.70s/it] | |
| 2024-08-07 19:57:03,301 - INFO: train loss: 0.04: 53%|#####3 | 159/300 [04:31<04:00, 1.71s/it] | |
| 2024-08-07 19:57:08,129 - INFO: train loss: 0.06: 54%|#####4 | 162/300 [04:36<03:51, 1.68s/it] | |
| 2024-08-07 19:57:12,945 - INFO: train loss: 0.09: 55%|#####5 | 165/300 [04:40<03:43, 1.66s/it] | |
| 2024-08-07 19:57:18,450 - INFO: train loss: 0.06: 56%|#####6 | 168/300 [04:46<03:45, 1.71s/it] | |
| 2024-08-07 19:57:23,372 - INFO: train loss: 0.06: 57%|#####6 | 171/300 [04:51<03:37, 1.69s/it] | |
| 2024-08-07 19:57:28,187 - INFO: train loss: 0.08: 58%|#####8 | 174/300 [04:56<03:29, 1.66s/it] | |
| 2024-08-07 19:57:33,906 - INFO: train loss: 0.06: 59%|#####8 | 177/300 [05:01<03:33, 1.74s/it] | |
| 2024-08-07 19:57:38,763 - INFO: train loss: 0.08: 60%|###### | 180/300 [05:06<03:24, 1.70s/it] | |
| 2024-08-07 19:57:44,154 - INFO: train loss: 0.16: 61%|######1 | 183/300 [05:12<03:22, 1.73s/it] | |
| 2024-08-07 19:57:49,263 - INFO: train loss: 0.17: 62%|######2 | 186/300 [05:17<03:16, 1.72s/it] | |
| 2024-08-07 19:57:54,682 - INFO: train loss: 0.18: 63%|######3 | 189/300 [05:22<03:13, 1.75s/it] | |
| 2024-08-07 19:57:59,696 - INFO: train loss: 0.08: 64%|######4 | 192/300 [05:27<03:06, 1.72s/it] | |
| 2024-08-07 19:58:04,539 - INFO: train loss: 0.09: 65%|######5 | 195/300 [05:32<02:57, 1.69s/it] | |
| 2024-08-07 19:58:09,574 - INFO: train loss: 0.07: 66%|######6 | 198/300 [05:37<02:52, 1.69s/it] | |
| 2024-08-07 19:58:14,399 - INFO: train loss: 0.08: 67%|######7 | 201/300 [05:42<02:44, 1.66s/it] | |
| 2024-08-07 19:58:19,098 - INFO: train loss: 0.05: 68%|######8 | 204/300 [05:47<02:36, 1.63s/it] | |
| 2024-08-07 19:58:23,987 - INFO: train loss: 0.04: 69%|######9 | 207/300 [05:51<02:31, 1.63s/it] | |
| 2024-08-07 19:58:28,915 - INFO: train loss: 0.07: 70%|####### | 210/300 [05:56<02:27, 1.64s/it] | |
| 2024-08-07 19:58:34,033 - INFO: train loss: 0.09: 71%|#######1 | 213/300 [06:01<02:24, 1.66s/it] | |
| 2024-08-07 19:58:38,472 - INFO: train loss: 0.09: 72%|#######2 | 216/300 [06:06<02:14, 1.60s/it] | |
| 2024-08-07 19:58:43,909 - INFO: train loss: 0.15: 73%|#######3 | 219/300 [06:11<02:14, 1.67s/it] | |
| 2024-08-07 19:58:48,658 - INFO: train loss: 0.11: 74%|#######4 | 222/300 [06:16<02:08, 1.64s/it] | |
| 2024-08-07 19:58:53,939 - INFO: train loss: 0.11: 75%|#######5 | 225/300 [06:21<02:05, 1.68s/it] | |
| 2024-08-07 19:58:58,561 - INFO: train loss: 0.09: 76%|#######6 | 228/300 [06:26<01:57, 1.64s/it] | |
| 2024-08-07 19:59:04,000 - INFO: train loss: 0.05: 77%|#######7 | 231/300 [06:31<01:56, 1.69s/it] | |
| 2024-08-07 19:59:08,977 - INFO: train loss: 0.09: 78%|#######8 | 234/300 [06:36<01:50, 1.68s/it] | |
| 2024-08-07 19:59:14,056 - INFO: train loss: 0.07: 79%|#######9 | 237/300 [06:41<01:46, 1.68s/it] | |
| 2024-08-07 19:59:19,292 - INFO: train loss: 0.09: 80%|######## | 240/300 [06:47<01:42, 1.70s/it] | |
| 2024-08-07 19:59:24,034 - INFO: train loss: 0.06: 81%|########1 | 243/300 [06:51<01:34, 1.67s/it] | |
| 2024-08-07 19:59:28,841 - INFO: train loss: 0.06: 82%|########2 | 246/300 [06:56<01:28, 1.65s/it] | |
| 2024-08-07 19:59:33,475 - INFO: train loss: 0.09: 83%|########2 | 249/300 [07:01<01:22, 1.62s/it] | |
| 2024-08-07 19:59:38,769 - INFO: train loss: 0.14: 84%|########4 | 252/300 [07:06<01:19, 1.66s/it] | |
| 2024-08-07 19:59:45,661 - INFO: train loss: 0.13: 85%|########5 | 255/300 [07:13<01:23, 1.85s/it] | |
| 2024-08-07 19:59:50,852 - INFO: train loss: 0.12: 86%|########6 | 258/300 [07:18<01:16, 1.82s/it] | |
| 2024-08-07 19:59:55,773 - INFO: train loss: 0.10: 87%|########7 | 261/300 [07:23<01:08, 1.76s/it] | |
| 2024-08-07 20:00:00,844 - INFO: train loss: 0.06: 88%|########8 | 264/300 [07:28<01:02, 1.74s/it] | |
| 2024-08-07 20:00:05,670 - INFO: train loss: 0.06: 89%|########9 | 267/300 [07:33<00:56, 1.70s/it] | |
| 2024-08-07 20:00:10,560 - INFO: train loss: 0.06: 90%|######### | 270/300 [07:38<00:50, 1.68s/it] | |
| 2024-08-07 20:00:15,214 - INFO: train loss: 0.04: 91%|#########1| 273/300 [07:43<00:44, 1.64s/it] | |
| 2024-08-07 20:00:20,459 - INFO: train loss: 0.05: 92%|#########2| 276/300 [07:48<00:40, 1.67s/it] | |
| 2024-08-07 20:00:25,706 - INFO: train loss: 0.04: 93%|#########3| 279/300 [07:53<00:35, 1.70s/it] | |
| 2024-08-07 20:00:30,506 - INFO: train loss: 0.04: 94%|#########3| 282/300 [07:58<00:30, 1.67s/it] | |
| 2024-08-07 20:00:35,468 - INFO: train loss: 0.04: 95%|#########5| 285/300 [08:03<00:24, 1.66s/it] | |
| 2024-08-07 20:00:40,906 - INFO: train loss: 0.07: 96%|#########6| 288/300 [08:08<00:20, 1.71s/it] | |
| 2024-08-07 20:00:46,065 - INFO: train loss: 0.09: 97%|#########7| 291/300 [08:14<00:15, 1.71s/it] | |
| 2024-08-07 20:00:51,141 - INFO: train loss: 0.09: 98%|#########8| 294/300 [08:19<00:10, 1.71s/it] | |
| 2024-08-07 20:00:56,616 - INFO: train loss: 0.08: 99%|#########9| 297/300 [08:24<00:05, 1.74s/it] | |
| 2024-08-07 20:01:01,959 - INFO: train loss: 0.06: 100%|##########| 300/300 [08:29<00:00, 1.75s/it] | |
| 2024-08-07 20:01:01,959 - INFO: train loss: 0.06: 100%|##########| 300/300 [08:29<00:00, 1.70s/it] | |
| 2024-08-07 20:01:01,959 - INFO: Saving last model checkpoint to /app/output | |
| 2024-08-07 20:01:01,960 - INFO: Saving checkpoint.. | |
| 2024-08-07 20:01:27,878 - INFO: Starting validation inference | |
| 2024-08-07 20:01:27,879 - INFO: validation progress: 0%| | 0/75 [00:00<?, ?it/s] | |
| 2024-08-07 20:01:33,273 - INFO: validation progress: 4%|4 | 3/75 [00:05<02:09, 1.80s/it] | |
| 2024-08-07 20:01:34,791 - INFO: validation progress: 8%|8 | 6/75 [00:06<01:11, 1.04s/it] | |
| 2024-08-07 20:01:36,225 - INFO: validation progress: 12%|#2 | 9/75 [00:08<00:51, 1.28it/s] | |
| 2024-08-07 20:01:37,727 - INFO: validation progress: 16%|#6 | 12/75 [00:09<00:42, 1.49it/s] | |
| 2024-08-07 20:01:39,331 - INFO: validation progress: 20%|## | 15/75 [00:11<00:37, 1.61it/s] | |
| 2024-08-07 20:01:40,888 - INFO: validation progress: 24%|##4 | 18/75 [00:13<00:33, 1.70it/s] | |
| 2024-08-07 20:01:42,520 - INFO: validation progress: 28%|##8 | 21/75 [00:14<00:30, 1.75it/s] | |
| 2024-08-07 20:01:44,012 - INFO: validation progress: 32%|###2 | 24/75 [00:16<00:27, 1.82it/s] | |
| 2024-08-07 20:01:45,653 - INFO: validation progress: 36%|###6 | 27/75 [00:17<00:26, 1.82it/s] | |
| 2024-08-07 20:01:47,236 - INFO: validation progress: 40%|#### | 30/75 [00:19<00:24, 1.85it/s] | |
| 2024-08-07 20:01:48,871 - INFO: validation progress: 44%|####4 | 33/75 [00:20<00:22, 1.84it/s] | |
| 2024-08-07 20:01:50,510 - INFO: validation progress: 48%|####8 | 36/75 [00:22<00:21, 1.84it/s] | |
| 2024-08-07 20:01:52,169 - INFO: validation progress: 52%|#####2 | 39/75 [00:24<00:19, 1.83it/s] | |
| 2024-08-07 20:01:53,742 - INFO: validation progress: 56%|#####6 | 42/75 [00:25<00:17, 1.85it/s] | |
| 2024-08-07 20:01:55,302 - INFO: validation progress: 60%|###### | 45/75 [00:27<00:16, 1.87it/s] | |
| 2024-08-07 20:01:56,798 - INFO: validation progress: 64%|######4 | 48/75 [00:28<00:14, 1.91it/s] | |
| 2024-08-07 20:01:58,300 - INFO: validation progress: 68%|######8 | 51/75 [00:30<00:12, 1.94it/s] | |
| 2024-08-07 20:01:59,925 - INFO: validation progress: 72%|#######2 | 54/75 [00:32<00:11, 1.91it/s] | |
| 2024-08-07 20:02:01,517 - INFO: validation progress: 76%|#######6 | 57/75 [00:33<00:09, 1.90it/s] | |
| 2024-08-07 20:02:02,957 - INFO: validation progress: 80%|######## | 60/75 [00:35<00:07, 1.95it/s] | |
| 2024-08-07 20:02:04,662 - INFO: validation progress: 84%|########4 | 63/75 [00:36<00:06, 1.89it/s] | |
| 2024-08-07 20:02:06,292 - INFO: validation progress: 88%|########8 | 66/75 [00:38<00:04, 1.88it/s] | |
| 2024-08-07 20:02:07,923 - INFO: validation progress: 92%|#########2| 69/75 [00:40<00:03, 1.86it/s] | |
| 2024-08-07 20:02:09,386 - INFO: validation progress: 96%|#########6| 72/75 [00:41<00:01, 1.92it/s] | |
| 2024-08-07 20:02:11,117 - INFO: validation progress: 100%|##########| 75/75 [00:43<00:00, 1.86it/s] | |
| 2024-08-07 20:02:11,121 - INFO: validation progress: 100%|##########| 75/75 [00:43<00:00, 1.73it/s] | |
| 2024-08-07 20:02:11,406 - INFO: Validation AUC: 0.98634 | |
| 2024-08-07 20:02:11,406 - INFO: Mean validation loss: 0.59212 | |
| 2024-08-07 20:02:11,743 - INFO: Training Epoch: 7 / 10 | |
| 2024-08-07 20:02:11,743 - INFO: train loss: 0%| | 0/300 [00:00<?, ?it/s] | |
| 2024-08-07 20:02:15,215 - INFO: Evaluation step: 300 | |
| 2024-08-07 20:02:20,317 - INFO: train loss: 0.03: 1%|1 | 3/300 [00:08<14:08, 2.86s/it] | |
| 2024-08-07 20:02:25,440 - INFO: train loss: 0.05: 2%|2 | 6/300 [00:13<10:41, 2.18s/it] | |
| 2024-08-07 20:02:30,534 - INFO: train loss: 0.03: 3%|3 | 9/300 [00:18<09:30, 1.96s/it] | |
| 2024-08-07 20:02:35,772 - INFO: train loss: 0.02: 4%|4 | 12/300 [00:24<09:00, 1.88s/it] | |
| 2024-08-07 20:02:41,171 - INFO: train loss: 0.03: 5%|5 | 15/300 [00:29<08:46, 1.85s/it] | |
| 2024-08-07 20:02:46,361 - INFO: train loss: 0.01: 6%|6 | 18/300 [00:34<08:29, 1.81s/it] | |
| 2024-08-07 20:02:51,986 - INFO: train loss: 0.02: 7%|7 | 21/300 [00:40<08:30, 1.83s/it] | |
| 2024-08-07 20:02:56,682 - INFO: train loss: 0.03: 8%|8 | 24/300 [00:44<08:01, 1.75s/it] | |
| 2024-08-07 20:03:01,644 - INFO: train loss: 0.03: 9%|9 | 27/300 [00:49<07:48, 1.72s/it] | |
| 2024-08-07 20:03:07,004 - INFO: train loss: 0.03: 10%|# | 30/300 [00:55<07:49, 1.74s/it] | |
| 2024-08-07 20:03:12,281 - INFO: train loss: 0.02: 11%|#1 | 33/300 [01:00<07:45, 1.74s/it] | |
| 2024-08-07 20:03:17,109 - INFO: train loss: 0.01: 12%|#2 | 36/300 [01:05<07:29, 1.70s/it] | |
| 2024-08-07 20:03:21,979 - INFO: train loss: 0.01: 13%|#3 | 39/300 [01:10<07:18, 1.68s/it] | |
| 2024-08-07 20:03:26,998 - INFO: train loss: 0.03: 14%|#4 | 42/300 [01:15<07:12, 1.68s/it] | |
| 2024-08-07 20:03:32,329 - INFO: train loss: 0.08: 15%|#5 | 45/300 [01:20<07:15, 1.71s/it] | |
| 2024-08-07 20:03:38,048 - INFO: train loss: 0.09: 16%|#6 | 48/300 [01:26<07:25, 1.77s/it] | |
| 2024-08-07 20:03:43,786 - INFO: train loss: 0.09: 17%|#7 | 51/300 [01:32<07:30, 1.81s/it] | |
| 2024-08-07 20:03:48,612 - INFO: train loss: 0.08: 18%|#8 | 54/300 [01:36<07:10, 1.75s/it] | |
| 2024-08-07 20:03:53,591 - INFO: train loss: 0.05: 19%|#9 | 57/300 [01:41<06:58, 1.72s/it] | |
| 2024-08-07 20:03:58,413 - INFO: train loss: 0.05: 20%|## | 60/300 [01:46<06:45, 1.69s/it] | |
| 2024-08-07 20:04:03,882 - INFO: train loss: 0.04: 21%|##1 | 63/300 [01:52<06:49, 1.73s/it] | |
| 2024-08-07 20:04:09,110 - INFO: train loss: 0.08: 22%|##2 | 66/300 [01:57<06:45, 1.73s/it] | |
| 2024-08-07 20:04:14,332 - INFO: train loss: 0.06: 23%|##3 | 69/300 [02:02<06:40, 1.74s/it] | |
| 2024-08-07 20:04:19,200 - INFO: train loss: 0.06: 24%|##4 | 72/300 [02:07<06:27, 1.70s/it] | |
| 2024-08-07 20:04:24,256 - INFO: train loss: 0.03: 25%|##5 | 75/300 [02:12<06:21, 1.70s/it] | |
| 2024-08-07 20:04:29,059 - INFO: train loss: 0.02: 26%|##6 | 78/300 [02:17<06:10, 1.67s/it] | |
| 2024-08-07 20:04:33,717 - INFO: train loss: 0.08: 27%|##7 | 81/300 [02:21<05:57, 1.63s/it] | |
| 2024-08-07 20:04:38,820 - INFO: train loss: 0.09: 28%|##8 | 84/300 [02:27<05:57, 1.65s/it] | |
| 2024-08-07 20:04:43,646 - INFO: train loss: 0.09: 29%|##9 | 87/300 [02:31<05:49, 1.64s/it] | |
| 2024-08-07 20:04:48,597 - INFO: train loss: 0.05: 30%|### | 90/300 [02:36<05:45, 1.64s/it] | |
| 2024-08-07 20:04:53,615 - INFO: train loss: 0.02: 31%|###1 | 93/300 [02:41<05:41, 1.65s/it] | |
| 2024-08-07 20:04:58,559 - INFO: train loss: 0.02: 32%|###2 | 96/300 [02:46<05:36, 1.65s/it] | |
| 2024-08-07 20:05:03,532 - INFO: train loss: 0.02: 33%|###3 | 99/300 [02:51<05:32, 1.65s/it] | |
| 2024-08-07 20:05:08,245 - INFO: train loss: 0.02: 34%|###4 | 102/300 [02:56<05:22, 1.63s/it] | |
| 2024-08-07 20:05:13,434 - INFO: train loss: 0.01: 35%|###5 | 105/300 [03:01<05:23, 1.66s/it] | |
| 2024-08-07 20:05:18,070 - INFO: train loss: 0.01: 36%|###6 | 108/300 [03:06<05:11, 1.62s/it] | |
| 2024-08-07 20:05:23,459 - INFO: train loss: 0.02: 37%|###7 | 111/300 [03:11<05:16, 1.68s/it] | |
| 2024-08-07 20:05:28,562 - INFO: train loss: 0.02: 38%|###8 | 114/300 [03:16<05:13, 1.68s/it] | |
| 2024-08-07 20:05:33,874 - INFO: train loss: 0.02: 39%|###9 | 117/300 [03:22<05:12, 1.71s/it] | |
| 2024-08-07 20:05:39,427 - INFO: train loss: 0.01: 40%|#### | 120/300 [03:27<05:15, 1.75s/it] | |
| 2024-08-07 20:05:44,155 - INFO: train loss: 0.01: 41%|####1 | 123/300 [03:32<05:00, 1.70s/it] | |
| 2024-08-07 20:05:49,161 - INFO: train loss: 0.01: 42%|####2 | 126/300 [03:37<04:54, 1.69s/it] | |
| 2024-08-07 20:05:54,090 - INFO: train loss: 0.02: 43%|####3 | 129/300 [03:42<04:46, 1.68s/it] | |
| 2024-08-07 20:05:59,132 - INFO: train loss: 0.02: 44%|####4 | 132/300 [03:47<04:41, 1.68s/it] | |
| 2024-08-07 20:06:03,890 - INFO: train loss: 0.05: 45%|####5 | 135/300 [03:52<04:32, 1.65s/it] | |
| 2024-08-07 20:06:08,624 - INFO: train loss: 0.06: 46%|####6 | 138/300 [03:56<04:23, 1.63s/it] | |
| 2024-08-07 20:06:13,985 - INFO: train loss: 0.07: 47%|####6 | 141/300 [04:02<04:26, 1.68s/it] | |
| 2024-08-07 20:06:18,920 - INFO: train loss: 0.04: 48%|####8 | 144/300 [04:07<04:20, 1.67s/it] | |
| 2024-08-07 20:06:23,726 - INFO: train loss: 0.03: 49%|####9 | 147/300 [04:11<04:12, 1.65s/it] | |
| 2024-08-07 20:06:28,750 - INFO: train loss: 0.02: 50%|##### | 150/300 [04:17<04:08, 1.66s/it] | |
| 2024-08-07 20:06:33,461 - INFO: train loss: 0.04: 51%|#####1 | 153/300 [04:21<03:59, 1.63s/it] | |
| 2024-08-07 20:06:38,503 - INFO: train loss: 0.03: 52%|#####2 | 156/300 [04:26<03:56, 1.65s/it] | |
| 2024-08-07 20:06:43,401 - INFO: train loss: 0.03: 53%|#####3 | 159/300 [04:31<03:51, 1.64s/it] | |
| 2024-08-07 20:06:48,369 - INFO: train loss: 0.03: 54%|#####4 | 162/300 [04:36<03:47, 1.65s/it] | |
| 2024-08-07 20:06:53,627 - INFO: train loss: 0.02: 55%|#####5 | 165/300 [04:41<03:46, 1.68s/it] | |
| 2024-08-07 20:06:58,654 - INFO: train loss: 0.02: 56%|#####6 | 168/300 [04:46<03:41, 1.68s/it] | |
| 2024-08-07 20:07:04,980 - INFO: train loss: 0.03: 57%|#####6 | 171/300 [04:53<03:53, 1.81s/it] | |
| 2024-08-07 20:07:10,211 - INFO: train loss: 0.02: 58%|#####8 | 174/300 [04:58<03:45, 1.79s/it] | |
| 2024-08-07 20:07:15,070 - INFO: train loss: 0.02: 59%|#####8 | 177/300 [05:03<03:33, 1.74s/it] | |
| 2024-08-07 20:07:19,764 - INFO: train loss: 0.04: 60%|###### | 180/300 [05:08<03:22, 1.69s/it] | |
| 2024-08-07 20:07:24,692 - INFO: train loss: 0.03: 61%|######1 | 183/300 [05:12<03:15, 1.67s/it] | |
| 2024-08-07 20:07:29,838 - INFO: train loss: 0.06: 62%|######2 | 186/300 [05:18<03:12, 1.69s/it] | |
| 2024-08-07 20:07:34,578 - INFO: train loss: 0.04: 63%|######3 | 189/300 [05:22<03:03, 1.65s/it] | |
| 2024-08-07 20:07:39,734 - INFO: train loss: 0.04: 64%|######4 | 192/300 [05:27<03:00, 1.67s/it] | |
| 2024-08-07 20:07:44,805 - INFO: train loss: 0.02: 65%|######5 | 195/300 [05:33<02:56, 1.68s/it] | |
| 2024-08-07 20:07:49,962 - INFO: train loss: 0.03: 66%|######6 | 198/300 [05:38<02:52, 1.69s/it] | |
| 2024-08-07 20:07:55,180 - INFO: train loss: 0.03: 67%|######7 | 201/300 [05:43<02:48, 1.71s/it] | |
| 2024-08-07 20:08:00,138 - INFO: train loss: 0.03: 68%|######8 | 204/300 [05:48<02:42, 1.69s/it] | |
| 2024-08-07 20:08:05,303 - INFO: train loss: 0.03: 69%|######9 | 207/300 [05:53<02:38, 1.70s/it] | |
| 2024-08-07 20:08:09,990 - INFO: train loss: 0.03: 70%|####### | 210/300 [05:58<02:29, 1.66s/it] | |
| 2024-08-07 20:08:15,237 - INFO: train loss: 0.02: 71%|#######1 | 213/300 [06:03<02:26, 1.69s/it] | |
| 2024-08-07 20:08:20,011 - INFO: train loss: 0.03: 72%|#######2 | 216/300 [06:08<02:19, 1.66s/it] | |
| 2024-08-07 20:08:25,493 - INFO: train loss: 0.01: 73%|#######3 | 219/300 [06:13<02:18, 1.71s/it] | |
| 2024-08-07 20:08:30,240 - INFO: train loss: 0.01: 74%|#######4 | 222/300 [06:18<02:10, 1.67s/it] | |
| 2024-08-07 20:08:35,411 - INFO: train loss: 0.01: 75%|#######5 | 225/300 [06:23<02:06, 1.69s/it] | |
| 2024-08-07 20:08:40,752 - INFO: train loss: 0.01: 76%|#######6 | 228/300 [06:29<02:03, 1.71s/it] | |
| 2024-08-07 20:08:45,759 - INFO: train loss: 0.02: 77%|#######7 | 231/300 [06:34<01:57, 1.70s/it] | |
| 2024-08-07 20:08:50,785 - INFO: train loss: 0.02: 78%|#######8 | 234/300 [06:39<01:51, 1.69s/it] | |
| 2024-08-07 20:08:55,851 - INFO: train loss: 0.02: 79%|#######9 | 237/300 [06:44<01:46, 1.69s/it] | |
| 2024-08-07 20:09:01,235 - INFO: train loss: 0.03: 80%|######## | 240/300 [06:49<01:43, 1.72s/it] | |
| 2024-08-07 20:09:06,816 - INFO: train loss: 0.08: 81%|########1 | 243/300 [06:55<01:40, 1.76s/it] | |
| 2024-08-07 20:09:11,993 - INFO: train loss: 0.08: 82%|########2 | 246/300 [07:00<01:34, 1.75s/it] | |
| 2024-08-07 20:09:16,995 - INFO: train loss: 0.07: 83%|########2 | 249/300 [07:05<01:28, 1.73s/it] | |
| 2024-08-07 20:09:22,471 - INFO: train loss: 0.04: 84%|########4 | 252/300 [07:10<01:24, 1.76s/it] | |
| 2024-08-07 20:09:27,640 - INFO: train loss: 0.03: 85%|########5 | 255/300 [07:15<01:18, 1.75s/it] | |
| 2024-08-07 20:09:33,109 - INFO: train loss: 0.03: 86%|########6 | 258/300 [07:21<01:14, 1.77s/it] | |
| 2024-08-07 20:09:37,615 - INFO: train loss: 0.04: 87%|########7 | 261/300 [07:25<01:05, 1.69s/it] | |
| 2024-08-07 20:09:42,610 - INFO: train loss: 0.05: 88%|########8 | 264/300 [07:30<01:00, 1.68s/it] | |
| 2024-08-07 20:09:47,847 - INFO: train loss: 0.06: 89%|########9 | 267/300 [07:36<00:56, 1.70s/it] | |
| 2024-08-07 20:09:52,913 - INFO: train loss: 0.04: 90%|######### | 270/300 [07:41<00:50, 1.70s/it] | |
| 2024-08-07 20:09:57,770 - INFO: train loss: 0.04: 91%|#########1| 273/300 [07:46<00:45, 1.67s/it] | |
| 2024-08-07 20:10:02,584 - INFO: train loss: 0.03: 92%|#########2| 276/300 [07:50<00:39, 1.65s/it] | |
| 2024-08-07 20:10:07,523 - INFO: train loss: 0.01: 93%|#########3| 279/300 [07:55<00:34, 1.65s/it] | |
| 2024-08-07 20:10:12,848 - INFO: train loss: 0.01: 94%|#########3| 282/300 [08:01<00:30, 1.69s/it] | |
| 2024-08-07 20:10:18,059 - INFO: train loss: 0.01: 95%|#########5| 285/300 [08:06<00:25, 1.70s/it] | |
| 2024-08-07 20:10:22,998 - INFO: train loss: 0.01: 96%|#########6| 288/300 [08:11<00:20, 1.69s/it] | |
| 2024-08-07 20:10:28,501 - INFO: train loss: 0.05: 97%|#########7| 291/300 [08:16<00:15, 1.73s/it] | |
| 2024-08-07 20:10:33,431 - INFO: train loss: 0.07: 98%|#########8| 294/300 [08:21<00:10, 1.70s/it] | |
| 2024-08-07 20:10:38,856 - INFO: train loss: 0.08: 99%|#########9| 297/300 [08:27<00:05, 1.74s/it] | |
| 2024-08-07 20:10:43,577 - INFO: train loss: 0.07: 100%|##########| 300/300 [08:31<00:00, 1.69s/it] | |
| 2024-08-07 20:10:43,577 - INFO: train loss: 0.07: 100%|##########| 300/300 [08:31<00:00, 1.71s/it] | |
| 2024-08-07 20:10:43,577 - INFO: Saving last model checkpoint to /app/output | |
| 2024-08-07 20:10:43,577 - INFO: Saving checkpoint.. | |
| 2024-08-07 20:11:09,827 - INFO: Starting validation inference | |
| 2024-08-07 20:11:09,827 - INFO: validation progress: 0%| | 0/75 [00:00<?, ?it/s] | |
| 2024-08-07 20:11:15,580 - INFO: validation progress: 4%|4 | 3/75 [00:05<02:18, 1.92s/it] | |
| 2024-08-07 20:11:17,097 - INFO: validation progress: 8%|8 | 6/75 [00:07<01:15, 1.09s/it] | |
| 2024-08-07 20:11:18,540 - INFO: validation progress: 12%|#2 | 9/75 [00:08<00:53, 1.23it/s] | |
| 2024-08-07 20:11:20,041 - INFO: validation progress: 16%|#6 | 12/75 [00:10<00:43, 1.45it/s] | |
| 2024-08-07 20:11:21,646 - INFO: validation progress: 20%|## | 15/75 [00:11<00:37, 1.58it/s] | |
| 2024-08-07 20:11:23,314 - INFO: validation progress: 24%|##4 | 18/75 [00:13<00:34, 1.65it/s] | |
| 2024-08-07 20:11:24,946 - INFO: validation progress: 28%|##8 | 21/75 [00:15<00:31, 1.71it/s] | |
| 2024-08-07 20:11:26,443 - INFO: validation progress: 32%|###2 | 24/75 [00:16<00:28, 1.79it/s] | |
| 2024-08-07 20:11:28,095 - INFO: validation progress: 36%|###6 | 27/75 [00:18<00:26, 1.80it/s] | |
| 2024-08-07 20:11:29,673 - INFO: validation progress: 40%|#### | 30/75 [00:19<00:24, 1.83it/s] | |
| 2024-08-07 20:11:31,390 - INFO: validation progress: 44%|####4 | 33/75 [00:21<00:23, 1.80it/s] | |
| 2024-08-07 20:11:33,039 - INFO: validation progress: 48%|####8 | 36/75 [00:23<00:21, 1.81it/s] | |
| 2024-08-07 20:11:34,699 - INFO: validation progress: 52%|#####2 | 39/75 [00:24<00:19, 1.81it/s] | |
| 2024-08-07 20:11:36,272 - INFO: validation progress: 56%|#####6 | 42/75 [00:26<00:17, 1.84it/s] | |
| 2024-08-07 20:11:37,827 - INFO: validation progress: 60%|###### | 45/75 [00:27<00:16, 1.86it/s] | |
| 2024-08-07 20:11:39,285 - INFO: validation progress: 64%|######4 | 48/75 [00:29<00:14, 1.92it/s] | |
| 2024-08-07 20:11:40,791 - INFO: validation progress: 68%|######8 | 51/75 [00:30<00:12, 1.94it/s] | |
| 2024-08-07 20:11:42,372 - INFO: validation progress: 72%|#######2 | 54/75 [00:32<00:10, 1.93it/s] | |
| 2024-08-07 20:11:43,953 - INFO: validation progress: 76%|#######6 | 57/75 [00:34<00:09, 1.92it/s] | |
| 2024-08-07 20:11:45,434 - INFO: validation progress: 80%|######## | 60/75 [00:35<00:07, 1.95it/s] | |
| 2024-08-07 20:11:47,188 - INFO: validation progress: 84%|########4 | 63/75 [00:37<00:06, 1.87it/s] | |
| 2024-08-07 20:11:48,833 - INFO: validation progress: 88%|########8 | 66/75 [00:39<00:04, 1.86it/s] | |
| 2024-08-07 20:11:50,465 - INFO: validation progress: 92%|#########2| 69/75 [00:40<00:03, 1.85it/s] | |
| 2024-08-07 20:11:51,930 - INFO: validation progress: 96%|#########6| 72/75 [00:42<00:01, 1.91it/s] | |
| 2024-08-07 20:11:53,661 - INFO: validation progress: 100%|##########| 75/75 [00:43<00:00, 1.85it/s] | |
| 2024-08-07 20:11:53,663 - INFO: validation progress: 100%|##########| 75/75 [00:43<00:00, 1.71it/s] | |
| 2024-08-07 20:11:53,937 - INFO: Validation AUC: 0.98353 | |
| 2024-08-07 20:11:53,937 - INFO: Mean validation loss: 0.77703 | |
| 2024-08-07 20:11:54,279 - INFO: Training Epoch: 8 / 10 | |
| 2024-08-07 20:11:54,279 - INFO: train loss: 0%| | 0/300 [00:00<?, ?it/s] | |
| 2024-08-07 20:11:57,790 - INFO: Evaluation step: 300 | |
| 2024-08-07 20:12:02,761 - INFO: train loss: 0.06: 1%|1 | 3/300 [00:08<13:59, 2.83s/it] | |
| 2024-08-07 20:12:07,850 - INFO: train loss: 0.03: 2%|2 | 6/300 [00:13<10:35, 2.16s/it] | |
| 2024-08-07 20:12:12,951 - INFO: train loss: 0.03: 3%|3 | 9/300 [00:18<09:27, 1.95s/it] | |
| 2024-08-07 20:12:18,228 - INFO: train loss: 0.01: 4%|4 | 12/300 [00:23<09:00, 1.88s/it] | |
| 2024-08-07 20:12:22,904 - INFO: train loss: 0.01: 5%|5 | 15/300 [00:28<08:21, 1.76s/it] | |
| 2024-08-07 20:12:27,761 - INFO: train loss: 0.01: 6%|6 | 18/300 [00:33<08:03, 1.71s/it] | |
| 2024-08-07 20:12:32,963 - INFO: train loss: 0.01: 7%|7 | 21/300 [00:38<07:59, 1.72s/it] | |
| 2024-08-07 20:12:38,069 - INFO: train loss: 0.00: 8%|8 | 24/300 [00:43<07:53, 1.71s/it] | |
| 2024-08-07 20:12:42,760 - INFO: train loss: 0.01: 9%|9 | 27/300 [00:48<07:35, 1.67s/it] | |
| 2024-08-07 20:12:47,632 - INFO: train loss: 0.02: 10%|# | 30/300 [00:53<07:26, 1.65s/it] | |
| 2024-08-07 20:12:52,887 - INFO: train loss: 0.02: 11%|#1 | 33/300 [00:58<07:29, 1.68s/it] | |
| 2024-08-07 20:12:57,814 - INFO: train loss: 0.01: 12%|#2 | 36/300 [01:03<07:21, 1.67s/it] | |
| 2024-08-07 20:13:02,513 - INFO: train loss: 0.01: 13%|#3 | 39/300 [01:08<07:07, 1.64s/it] | |
| 2024-08-07 20:13:07,348 - INFO: train loss: 0.00: 14%|#4 | 42/300 [01:13<07:00, 1.63s/it] | |
| 2024-08-07 20:13:12,463 - INFO: train loss: 0.01: 15%|#5 | 45/300 [01:18<07:01, 1.65s/it] | |
| 2024-08-07 20:13:17,454 - INFO: train loss: 0.02: 16%|#6 | 48/300 [01:23<06:57, 1.66s/it] | |
| 2024-08-07 20:13:22,641 - INFO: train loss: 0.02: 17%|#7 | 51/300 [01:28<06:57, 1.68s/it] | |
| 2024-08-07 20:13:27,472 - INFO: train loss: 0.02: 18%|#8 | 54/300 [01:33<06:47, 1.66s/it] | |
| 2024-08-07 20:13:32,480 - INFO: train loss: 0.00: 19%|#9 | 57/300 [01:38<06:43, 1.66s/it] | |
| 2024-08-07 20:13:37,073 - INFO: train loss: 0.01: 20%|## | 60/300 [01:42<06:29, 1.62s/it] | |
| 2024-08-07 20:13:42,574 - INFO: train loss: 0.02: 21%|##1 | 63/300 [01:48<06:39, 1.69s/it] | |
| 2024-08-07 20:13:47,327 - INFO: train loss: 0.02: 22%|##2 | 66/300 [01:53<06:27, 1.66s/it] | |
| 2024-08-07 20:13:52,481 - INFO: train loss: 0.02: 23%|##3 | 69/300 [01:58<06:26, 1.67s/it] | |
| 2024-08-07 20:13:57,520 - INFO: train loss: 0.01: 24%|##4 | 72/300 [02:03<06:22, 1.68s/it] | |
| 2024-08-07 20:14:02,908 - INFO: train loss: 0.00: 25%|##5 | 75/300 [02:08<06:25, 1.71s/it] | |
| 2024-08-07 20:14:07,360 - INFO: train loss: 0.00: 26%|##6 | 78/300 [02:13<06:04, 1.64s/it] | |
| 2024-08-07 20:14:12,444 - INFO: train loss: 0.00: 27%|##7 | 81/300 [02:18<06:03, 1.66s/it] | |
| 2024-08-07 20:14:17,443 - INFO: train loss: 0.00: 28%|##8 | 84/300 [02:23<05:58, 1.66s/it] | |
| 2024-08-07 20:14:22,297 - INFO: train loss: 0.00: 29%|##9 | 87/300 [02:28<05:51, 1.65s/it] | |
| 2024-08-07 20:14:27,180 - INFO: train loss: 0.01: 30%|### | 90/300 [02:32<05:44, 1.64s/it] | |
| 2024-08-07 20:14:31,597 - INFO: train loss: 0.01: 31%|###1 | 93/300 [02:37<05:29, 1.59s/it] | |
| 2024-08-07 20:14:36,922 - INFO: train loss: 0.01: 32%|###2 | 96/300 [02:42<05:35, 1.65s/it] | |
| 2024-08-07 20:14:41,847 - INFO: train loss: 0.00: 33%|###3 | 99/300 [02:47<05:30, 1.64s/it] | |
| 2024-08-07 20:14:47,270 - INFO: train loss: 0.01: 34%|###4 | 102/300 [02:52<05:35, 1.69s/it] | |
| 2024-08-07 20:14:52,339 - INFO: train loss: 0.01: 35%|###5 | 105/300 [02:58<05:30, 1.69s/it] | |
| 2024-08-07 20:14:57,282 - INFO: train loss: 0.01: 36%|###6 | 108/300 [03:03<05:22, 1.68s/it] | |
| 2024-08-07 20:15:01,768 - INFO: train loss: 0.01: 37%|###7 | 111/300 [03:07<05:06, 1.62s/it] | |
| 2024-08-07 20:15:07,162 - INFO: train loss: 0.01: 38%|###8 | 114/300 [03:12<05:11, 1.68s/it] | |
| 2024-08-07 20:15:12,529 - INFO: train loss: 0.02: 39%|###9 | 117/300 [03:18<05:12, 1.71s/it] | |
| 2024-08-07 20:15:17,943 - INFO: train loss: 0.02: 40%|#### | 120/300 [03:23<05:12, 1.74s/it] | |
| 2024-08-07 20:15:22,732 - INFO: train loss: 0.02: 41%|####1 | 123/300 [03:28<05:00, 1.70s/it] | |
| 2024-08-07 20:15:27,789 - INFO: train loss: 0.02: 42%|####2 | 126/300 [03:33<04:54, 1.69s/it] | |
| 2024-08-07 20:15:32,907 - INFO: train loss: 0.01: 43%|####3 | 129/300 [03:38<04:50, 1.70s/it] | |
| 2024-08-07 20:15:38,273 - INFO: train loss: 0.01: 44%|####4 | 132/300 [03:43<04:49, 1.72s/it] | |
| 2024-08-07 20:15:43,918 - INFO: train loss: 0.02: 45%|####5 | 135/300 [03:49<04:52, 1.77s/it] | |
| 2024-08-07 20:15:48,616 - INFO: train loss: 0.02: 46%|####6 | 138/300 [03:54<04:37, 1.71s/it] | |
| 2024-08-07 20:15:53,974 - INFO: train loss: 0.01: 47%|####6 | 141/300 [03:59<04:35, 1.73s/it] | |
| 2024-08-07 20:16:00,296 - INFO: train loss: 0.02: 48%|####8 | 144/300 [04:06<04:47, 1.85s/it] | |
| 2024-08-07 20:16:04,958 - INFO: train loss: 0.00: 49%|####9 | 147/300 [04:10<04:28, 1.76s/it] | |
| 2024-08-07 20:16:09,938 - INFO: train loss: 0.00: 50%|##### | 150/300 [04:15<04:19, 1.73s/it] | |
| 2024-08-07 20:16:15,388 - INFO: train loss: 0.00: 51%|#####1 | 153/300 [04:21<04:17, 1.75s/it] | |
| 2024-08-07 20:16:20,326 - INFO: train loss: 0.00: 52%|#####2 | 156/300 [04:26<04:08, 1.72s/it] | |
| 2024-08-07 20:16:25,522 - INFO: train loss: 0.01: 53%|#####3 | 159/300 [04:31<04:03, 1.73s/it] | |
| 2024-08-07 20:16:30,797 - INFO: train loss: 0.01: 54%|#####4 | 162/300 [04:36<03:59, 1.74s/it] | |
| 2024-08-07 20:16:35,492 - INFO: train loss: 0.01: 55%|#####5 | 165/300 [04:41<03:47, 1.68s/it] | |
| 2024-08-07 20:16:40,745 - INFO: train loss: 0.01: 56%|#####6 | 168/300 [04:46<03:44, 1.70s/it] | |
| 2024-08-07 20:16:45,355 - INFO: train loss: 0.01: 57%|#####6 | 171/300 [04:51<03:33, 1.65s/it] | |
| 2024-08-07 20:16:50,156 - INFO: train loss: 0.01: 58%|#####8 | 174/300 [04:55<03:26, 1.64s/it] | |
| 2024-08-07 20:16:55,009 - INFO: train loss: 0.00: 59%|#####8 | 177/300 [05:00<03:20, 1.63s/it] | |
| 2024-08-07 20:16:59,859 - INFO: train loss: 0.01: 60%|###### | 180/300 [05:05<03:15, 1.63s/it] | |
| 2024-08-07 20:17:05,028 - INFO: train loss: 0.02: 61%|######1 | 183/300 [05:10<03:13, 1.66s/it] | |
| 2024-08-07 20:17:09,693 - INFO: train loss: 0.01: 62%|######2 | 186/300 [05:15<03:05, 1.63s/it] | |
| 2024-08-07 20:17:14,655 - INFO: train loss: 0.01: 63%|######3 | 189/300 [05:20<03:01, 1.63s/it] | |
| 2024-08-07 20:17:19,897 - INFO: train loss: 0.02: 64%|######4 | 192/300 [05:25<03:00, 1.67s/it] | |
| 2024-08-07 20:17:25,024 - INFO: train loss: 0.02: 65%|######5 | 195/300 [05:30<02:56, 1.68s/it] | |
| 2024-08-07 20:17:29,873 - INFO: train loss: 0.01: 66%|######6 | 198/300 [05:35<02:49, 1.66s/it] | |
| 2024-08-07 20:17:34,662 - INFO: train loss: 0.00: 67%|######7 | 201/300 [05:40<02:42, 1.64s/it] | |
| 2024-08-07 20:17:40,024 - INFO: train loss: 0.00: 68%|######8 | 204/300 [05:45<02:41, 1.69s/it] | |
| 2024-08-07 20:17:45,230 - INFO: train loss: 0.00: 69%|######9 | 207/300 [05:50<02:38, 1.70s/it] | |
| 2024-08-07 20:17:49,819 - INFO: train loss: 0.02: 70%|####### | 210/300 [05:55<02:28, 1.65s/it] | |
| 2024-08-07 20:17:55,232 - INFO: train loss: 0.02: 71%|#######1 | 213/300 [06:00<02:27, 1.70s/it] | |
| 2024-08-07 20:18:00,534 - INFO: train loss: 0.02: 72%|#######2 | 216/300 [06:06<02:24, 1.72s/it] | |
| 2024-08-07 20:18:05,513 - INFO: train loss: 0.01: 73%|#######3 | 219/300 [06:11<02:17, 1.70s/it] | |
| 2024-08-07 20:18:09,898 - INFO: train loss: 0.00: 74%|#######4 | 222/300 [06:15<02:07, 1.63s/it] | |
| 2024-08-07 20:18:14,500 - INFO: train loss: 0.01: 75%|#######5 | 225/300 [06:20<02:00, 1.60s/it] | |
| 2024-08-07 20:18:19,901 - INFO: train loss: 0.01: 76%|#######6 | 228/300 [06:25<01:59, 1.66s/it] | |
| 2024-08-07 20:18:24,874 - INFO: train loss: 0.02: 77%|#######7 | 231/300 [06:30<01:54, 1.66s/it] | |
| 2024-08-07 20:18:30,311 - INFO: train loss: 0.03: 78%|#######8 | 234/300 [06:36<01:52, 1.71s/it] | |
| 2024-08-07 20:18:35,272 - INFO: train loss: 0.02: 79%|#######9 | 237/300 [06:40<01:46, 1.69s/it] | |
| 2024-08-07 20:18:39,913 - INFO: train loss: 0.01: 80%|######## | 240/300 [06:45<01:38, 1.65s/it] | |
| 2024-08-07 20:18:44,501 - INFO: train loss: 0.01: 81%|########1 | 243/300 [06:50<01:31, 1.61s/it] | |
| 2024-08-07 20:18:49,281 - INFO: train loss: 0.00: 82%|########2 | 246/300 [06:55<01:26, 1.61s/it] | |
| 2024-08-07 20:18:54,155 - INFO: train loss: 0.00: 83%|########2 | 249/300 [06:59<01:22, 1.61s/it] | |
| 2024-08-07 20:18:59,680 - INFO: train loss: 0.01: 84%|########4 | 252/300 [07:05<01:20, 1.68s/it] | |
| 2024-08-07 20:19:04,938 - INFO: train loss: 0.01: 85%|########5 | 255/300 [07:10<01:16, 1.70s/it] | |
| 2024-08-07 20:19:10,024 - INFO: train loss: 0.01: 86%|########6 | 258/300 [07:15<01:11, 1.70s/it] | |
| 2024-08-07 20:19:14,899 - INFO: train loss: 0.01: 87%|########7 | 261/300 [07:20<01:05, 1.68s/it] | |
| 2024-08-07 20:19:20,212 - INFO: train loss: 0.00: 88%|########8 | 264/300 [07:25<01:01, 1.71s/it] | |
| 2024-08-07 20:19:25,336 - INFO: train loss: 0.00: 89%|########9 | 267/300 [07:31<00:56, 1.71s/it] | |
| 2024-08-07 20:19:30,534 - INFO: train loss: 0.00: 90%|######### | 270/300 [07:36<00:51, 1.71s/it] | |
| 2024-08-07 20:19:35,732 - INFO: train loss: 0.00: 91%|#########1| 273/300 [07:41<00:46, 1.72s/it] | |
| 2024-08-07 20:19:40,862 - INFO: train loss: 0.00: 92%|#########2| 276/300 [07:46<00:41, 1.72s/it] | |
| 2024-08-07 20:19:45,387 - INFO: train loss: 0.00: 93%|#########3| 279/300 [07:51<00:34, 1.65s/it] | |
| 2024-08-07 20:19:50,372 - INFO: train loss: 0.01: 94%|#########3| 282/300 [07:56<00:29, 1.66s/it] | |
| 2024-08-07 20:19:55,253 - INFO: train loss: 0.01: 95%|#########5| 285/300 [08:00<00:24, 1.65s/it] | |
| 2024-08-07 20:20:00,260 - INFO: train loss: 0.01: 96%|#########6| 288/300 [08:05<00:19, 1.65s/it] | |
| 2024-08-07 20:20:05,453 - INFO: train loss: 0.01: 97%|#########7| 291/300 [08:11<00:15, 1.68s/it] | |
| 2024-08-07 20:20:10,743 - INFO: train loss: 0.01: 98%|#########8| 294/300 [08:16<00:10, 1.70s/it] | |
| 2024-08-07 20:20:16,042 - INFO: train loss: 0.03: 99%|#########9| 297/300 [08:21<00:05, 1.72s/it] | |
| 2024-08-07 20:20:20,700 - INFO: train loss: 0.04: 100%|##########| 300/300 [08:26<00:00, 1.67s/it] | |
| 2024-08-07 20:20:20,700 - INFO: train loss: 0.04: 100%|##########| 300/300 [08:26<00:00, 1.69s/it] | |
| 2024-08-07 20:20:20,700 - INFO: Saving last model checkpoint to /app/output | |
| 2024-08-07 20:20:20,700 - INFO: Saving checkpoint.. | |
| 2024-08-07 20:20:47,550 - INFO: Starting validation inference | |
| 2024-08-07 20:20:47,551 - INFO: validation progress: 0%| | 0/75 [00:00<?, ?it/s] | |
| 2024-08-07 20:20:52,956 - INFO: validation progress: 4%|4 | 3/75 [00:05<02:09, 1.80s/it] | |
| 2024-08-07 20:20:54,473 - INFO: validation progress: 8%|8 | 6/75 [00:06<01:11, 1.04s/it] | |
| 2024-08-07 20:20:55,907 - INFO: validation progress: 12%|#2 | 9/75 [00:08<00:51, 1.28it/s] | |
| 2024-08-07 20:20:57,418 - INFO: validation progress: 16%|#6 | 12/75 [00:09<00:42, 1.49it/s] | |
| 2024-08-07 20:20:59,061 - INFO: validation progress: 20%|## | 15/75 [00:11<00:37, 1.59it/s] | |
| 2024-08-07 20:21:00,620 - INFO: validation progress: 24%|##4 | 18/75 [00:13<00:33, 1.69it/s] | |
| 2024-08-07 20:21:02,302 - INFO: validation progress: 28%|##8 | 21/75 [00:14<00:31, 1.72it/s] | |
| 2024-08-07 20:21:03,796 - INFO: validation progress: 32%|###2 | 24/75 [00:16<00:28, 1.80it/s] | |
| 2024-08-07 20:21:05,438 - INFO: validation progress: 36%|###6 | 27/75 [00:17<00:26, 1.81it/s] | |
| 2024-08-07 20:21:07,031 - INFO: validation progress: 40%|#### | 30/75 [00:19<00:24, 1.83it/s] | |
| 2024-08-07 20:21:08,662 - INFO: validation progress: 44%|####4 | 33/75 [00:21<00:22, 1.83it/s] | |
| 2024-08-07 20:21:10,298 - INFO: validation progress: 48%|####8 | 36/75 [00:22<00:21, 1.83it/s] | |
| 2024-08-07 20:21:11,957 - INFO: validation progress: 52%|#####2 | 39/75 [00:24<00:19, 1.83it/s] | |
| 2024-08-07 20:21:13,531 - INFO: validation progress: 56%|#####6 | 42/75 [00:25<00:17, 1.85it/s] | |
| 2024-08-07 20:21:15,081 - INFO: validation progress: 60%|###### | 45/75 [00:27<00:16, 1.87it/s] | |
| 2024-08-07 20:21:16,537 - INFO: validation progress: 64%|######4 | 48/75 [00:28<00:14, 1.93it/s] | |
| 2024-08-07 20:21:18,046 - INFO: validation progress: 68%|######8 | 51/75 [00:30<00:12, 1.95it/s] | |
| 2024-08-07 20:21:19,623 - INFO: validation progress: 72%|#######2 | 54/75 [00:32<00:10, 1.93it/s] | |
| 2024-08-07 20:21:21,196 - INFO: validation progress: 76%|#######6 | 57/75 [00:33<00:09, 1.92it/s] | |
| 2024-08-07 20:21:22,636 - INFO: validation progress: 80%|######## | 60/75 [00:35<00:07, 1.97it/s] | |
| 2024-08-07 20:21:24,343 - INFO: validation progress: 84%|########4 | 63/75 [00:36<00:06, 1.90it/s] | |
| 2024-08-07 20:21:25,983 - INFO: validation progress: 88%|########8 | 66/75 [00:38<00:04, 1.88it/s] | |
| 2024-08-07 20:21:27,654 - INFO: validation progress: 92%|#########2| 69/75 [00:40<00:03, 1.85it/s] | |
| 2024-08-07 20:21:29,115 - INFO: validation progress: 96%|#########6| 72/75 [00:41<00:01, 1.91it/s] | |
| 2024-08-07 20:21:30,892 - INFO: validation progress: 100%|##########| 75/75 [00:43<00:00, 1.84it/s] | |
| 2024-08-07 20:21:30,895 - INFO: validation progress: 100%|##########| 75/75 [00:43<00:00, 1.73it/s] | |
| 2024-08-07 20:21:31,175 - INFO: Validation AUC: 0.98508 | |
| 2024-08-07 20:21:31,175 - INFO: Mean validation loss: 0.64293 | |
| 2024-08-07 20:21:31,522 - INFO: Training Epoch: 9 / 10 | |
| 2024-08-07 20:21:31,522 - INFO: train loss: 0%| | 0/300 [00:00<?, ?it/s] | |
| 2024-08-07 20:21:35,983 - INFO: Evaluation step: 300 | |
| 2024-08-07 20:21:41,085 - INFO: train loss: 0.03: 1%|1 | 3/300 [00:09<15:46, 3.19s/it] | |
| 2024-08-07 20:21:46,070 - INFO: train loss: 0.01: 2%|2 | 6/300 [00:14<11:13, 2.29s/it] | |
| 2024-08-07 20:21:51,064 - INFO: train loss: 0.01: 3%|3 | 9/300 [00:19<09:43, 2.00s/it] | |
| 2024-08-07 20:21:55,990 - INFO: train loss: 0.00: 4%|4 | 12/300 [00:24<08:56, 1.86s/it] | |
| 2024-08-07 20:22:00,464 - INFO: train loss: 0.00: 5%|5 | 15/300 [00:28<08:12, 1.73s/it] | |
| 2024-08-07 20:22:05,384 - INFO: train loss: 0.00: 6%|6 | 18/300 [00:33<07:58, 1.70s/it] | |
| 2024-08-07 20:22:10,373 - INFO: train loss: 0.00: 7%|7 | 21/300 [00:38<07:50, 1.69s/it] | |
| 2024-08-07 20:22:15,589 - INFO: train loss: 0.00: 8%|8 | 24/300 [00:44<07:50, 1.70s/it] | |
| 2024-08-07 20:22:20,810 - INFO: train loss: 0.00: 9%|9 | 27/300 [00:49<07:48, 1.71s/it] | |
| 2024-08-07 20:22:26,006 - INFO: train loss: 0.00: 10%|# | 30/300 [00:54<07:44, 1.72s/it] | |
| 2024-08-07 20:22:31,261 - INFO: train loss: 0.00: 11%|#1 | 33/300 [00:59<07:41, 1.73s/it] | |
| 2024-08-07 20:22:36,122 - INFO: train loss: 0.00: 12%|#2 | 36/300 [01:04<07:27, 1.70s/it] | |
| 2024-08-07 20:22:40,522 - INFO: train loss: 0.00: 13%|#3 | 39/300 [01:08<07:04, 1.63s/it] | |
| 2024-08-07 20:22:45,484 - INFO: train loss: 0.00: 14%|#4 | 42/300 [01:13<07:01, 1.64s/it] | |
| 2024-08-07 20:22:50,381 - INFO: train loss: 0.00: 15%|#5 | 45/300 [01:18<06:56, 1.63s/it] | |
| 2024-08-07 20:22:55,456 - INFO: train loss: 0.00: 16%|#6 | 48/300 [01:23<06:56, 1.65s/it] | |
| 2024-08-07 20:23:00,381 - INFO: train loss: 0.00: 17%|#7 | 51/300 [01:28<06:50, 1.65s/it] | |
| 2024-08-07 20:23:05,319 - INFO: train loss: 0.00: 18%|#8 | 54/300 [01:33<06:45, 1.65s/it] | |
| 2024-08-07 20:23:10,174 - INFO: train loss: 0.00: 19%|#9 | 57/300 [01:38<06:38, 1.64s/it] | |
| 2024-08-07 20:23:15,054 - INFO: train loss: 0.00: 20%|## | 60/300 [01:43<06:32, 1.64s/it] | |
| 2024-08-07 20:23:20,225 - INFO: train loss: 0.00: 21%|##1 | 63/300 [01:48<06:33, 1.66s/it] | |
| 2024-08-07 20:23:25,740 - INFO: train loss: 0.00: 22%|##2 | 66/300 [01:54<06:41, 1.71s/it] | |
| 2024-08-07 20:23:30,691 - INFO: train loss: 0.00: 23%|##3 | 69/300 [01:59<06:31, 1.70s/it] | |
| 2024-08-07 20:23:35,634 - INFO: train loss: 0.00: 24%|##4 | 72/300 [02:04<06:23, 1.68s/it] | |
| 2024-08-07 20:23:40,599 - INFO: train loss: 0.00: 25%|##5 | 75/300 [02:09<06:16, 1.67s/it] | |
| 2024-08-07 20:23:45,461 - INFO: train loss: 0.00: 26%|##6 | 78/300 [02:13<06:07, 1.66s/it] | |
| 2024-08-07 20:23:50,584 - INFO: train loss: 0.00: 27%|##7 | 81/300 [02:19<06:06, 1.67s/it] | |
| 2024-08-07 20:23:55,339 - INFO: train loss: 0.00: 28%|##8 | 84/300 [02:23<05:55, 1.65s/it] | |
| 2024-08-07 20:23:59,954 - INFO: train loss: 0.00: 29%|##9 | 87/300 [02:28<05:43, 1.61s/it] | |
| 2024-08-07 20:24:05,052 - INFO: train loss: 0.00: 30%|### | 90/300 [02:33<05:44, 1.64s/it] | |
| 2024-08-07 20:24:10,095 - INFO: train loss: 0.00: 31%|###1 | 93/300 [02:38<05:41, 1.65s/it] | |
| 2024-08-07 20:24:15,656 - INFO: train loss: 0.00: 32%|###2 | 96/300 [02:44<05:49, 1.71s/it] | |
| 2024-08-07 20:24:20,679 - INFO: train loss: 0.00: 33%|###3 | 99/300 [02:49<05:41, 1.70s/it] | |
| 2024-08-07 20:24:25,731 - INFO: train loss: 0.00: 34%|###4 | 102/300 [02:54<05:35, 1.70s/it] | |
| 2024-08-07 20:24:30,595 - INFO: train loss: 0.00: 35%|###5 | 105/300 [02:59<05:26, 1.67s/it] | |
| 2024-08-07 20:24:35,802 - INFO: train loss: 0.00: 36%|###6 | 108/300 [03:04<05:24, 1.69s/it] | |
| 2024-08-07 20:24:41,025 - INFO: train loss: 0.00: 37%|###7 | 111/300 [03:09<05:22, 1.71s/it] | |
| 2024-08-07 20:24:46,397 - INFO: train loss: 0.01: 38%|###8 | 114/300 [03:14<05:22, 1.73s/it] | |
| 2024-08-07 20:24:51,763 - INFO: train loss: 0.01: 39%|###9 | 117/300 [03:20<05:20, 1.75s/it] | |
| 2024-08-07 20:24:56,472 - INFO: train loss: 0.01: 40%|#### | 120/300 [03:24<05:05, 1.70s/it] | |
| 2024-08-07 20:25:01,294 - INFO: train loss: 0.00: 41%|####1 | 123/300 [03:29<04:55, 1.67s/it] | |
| 2024-08-07 20:25:06,422 - INFO: train loss: 0.00: 42%|####2 | 126/300 [03:34<04:52, 1.68s/it] | |
| 2024-08-07 20:25:11,324 - INFO: train loss: 0.00: 43%|####3 | 129/300 [03:39<04:45, 1.67s/it] | |
| 2024-08-07 20:25:16,582 - INFO: train loss: 0.00: 44%|####4 | 132/300 [03:45<04:44, 1.69s/it] | |
| 2024-08-07 20:25:21,783 - INFO: train loss: 0.00: 45%|####5 | 135/300 [03:50<04:41, 1.70s/it] | |
| 2024-08-07 20:25:26,857 - INFO: train loss: 0.00: 46%|####6 | 138/300 [03:55<04:35, 1.70s/it] | |
| 2024-08-07 20:25:31,802 - INFO: train loss: 0.01: 47%|####6 | 141/300 [04:00<04:27, 1.69s/it] | |
| 2024-08-07 20:25:36,498 - INFO: train loss: 0.01: 48%|####8 | 144/300 [04:04<04:17, 1.65s/it] | |
| 2024-08-07 20:25:41,315 - INFO: train loss: 0.01: 49%|####9 | 147/300 [04:09<04:10, 1.64s/it] | |
| 2024-08-07 20:25:46,328 - INFO: train loss: 0.00: 50%|##### | 150/300 [04:14<04:06, 1.65s/it] | |
| 2024-08-07 20:25:51,604 - INFO: train loss: 0.00: 51%|#####1 | 153/300 [04:20<04:06, 1.68s/it] | |
| 2024-08-07 20:25:56,342 - INFO: train loss: 0.00: 52%|#####2 | 156/300 [04:24<03:57, 1.65s/it] | |
| 2024-08-07 20:26:01,821 - INFO: train loss: 0.00: 53%|#####3 | 159/300 [04:30<04:00, 1.70s/it] | |
| 2024-08-07 20:26:07,553 - INFO: train loss: 0.00: 54%|#####4 | 162/300 [04:36<04:03, 1.77s/it] | |
| 2024-08-07 20:26:12,182 - INFO: train loss: 0.00: 55%|#####5 | 165/300 [04:40<03:49, 1.70s/it] | |
| 2024-08-07 20:26:17,532 - INFO: train loss: 0.00: 56%|#####6 | 168/300 [04:46<03:47, 1.72s/it] | |
| 2024-08-07 20:26:23,050 - INFO: train loss: 0.00: 57%|#####6 | 171/300 [04:51<03:46, 1.76s/it] | |
| 2024-08-07 20:26:28,175 - INFO: train loss: 0.00: 58%|#####8 | 174/300 [04:56<03:39, 1.74s/it] | |
| 2024-08-07 20:26:33,293 - INFO: train loss: 0.00: 59%|#####8 | 177/300 [05:01<03:33, 1.73s/it] | |
| 2024-08-07 20:26:38,011 - INFO: train loss: 0.00: 60%|###### | 180/300 [05:06<03:22, 1.68s/it] | |
| 2024-08-07 20:26:42,706 - INFO: train loss: 0.00: 61%|######1 | 183/300 [05:11<03:12, 1.65s/it] | |
| 2024-08-07 20:26:47,159 - INFO: train loss: 0.00: 62%|######2 | 186/300 [05:15<03:02, 1.60s/it] | |
| 2024-08-07 20:26:51,986 - INFO: train loss: 0.00: 63%|######3 | 189/300 [05:20<02:57, 1.60s/it] | |
| 2024-08-07 20:26:56,988 - INFO: train loss: 0.00: 64%|######4 | 192/300 [05:25<02:55, 1.62s/it] | |
| 2024-08-07 20:27:02,143 - INFO: train loss: 0.00: 65%|######5 | 195/300 [05:30<02:53, 1.65s/it] | |
| 2024-08-07 20:27:07,318 - INFO: train loss: 0.00: 66%|######6 | 198/300 [05:35<02:50, 1.67s/it] | |
| 2024-08-07 20:27:12,563 - INFO: train loss: 0.00: 67%|######7 | 201/300 [05:41<02:47, 1.70s/it] | |
| 2024-08-07 20:27:17,271 - INFO: train loss: 0.00: 68%|######8 | 204/300 [05:45<02:39, 1.66s/it] | |
| 2024-08-07 20:27:21,565 - INFO: train loss: 0.00: 69%|######9 | 207/300 [05:50<02:27, 1.59s/it] | |
| 2024-08-07 20:27:26,687 - INFO: train loss: 0.00: 70%|####### | 210/300 [05:55<02:26, 1.63s/it] | |
| 2024-08-07 20:27:31,823 - INFO: train loss: 0.00: 71%|#######1 | 213/300 [06:00<02:23, 1.65s/it] | |
| 2024-08-07 20:27:36,961 - INFO: train loss: 0.00: 72%|#######2 | 216/300 [06:05<02:20, 1.67s/it] | |
| 2024-08-07 20:27:41,879 - INFO: train loss: 0.00: 73%|#######3 | 219/300 [06:10<02:14, 1.66s/it] | |
| 2024-08-07 20:27:46,529 - INFO: train loss: 0.00: 74%|#######4 | 222/300 [06:15<02:06, 1.63s/it] | |
| 2024-08-07 20:27:51,831 - INFO: train loss: 0.00: 75%|#######5 | 225/300 [06:20<02:05, 1.67s/it] | |
| 2024-08-07 20:27:56,966 - INFO: train loss: 0.00: 76%|#######6 | 228/300 [06:25<02:01, 1.68s/it] | |
| 2024-08-07 20:28:01,999 - INFO: train loss: 0.00: 77%|#######7 | 231/300 [06:30<01:55, 1.68s/it] | |
| 2024-08-07 20:28:06,994 - INFO: train loss: 0.00: 78%|#######8 | 234/300 [06:35<01:50, 1.68s/it] | |
| 2024-08-07 20:28:12,321 - INFO: train loss: 0.00: 79%|#######9 | 237/300 [06:40<01:47, 1.71s/it] | |
| 2024-08-07 20:28:18,126 - INFO: train loss: 0.00: 80%|######## | 240/300 [06:46<01:46, 1.77s/it] | |
| 2024-08-07 20:28:23,306 - INFO: train loss: 0.00: 81%|########1 | 243/300 [06:51<01:40, 1.76s/it] | |
| 2024-08-07 20:28:27,880 - INFO: train loss: 0.00: 82%|########2 | 246/300 [06:56<01:31, 1.69s/it] | |
| 2024-08-07 20:28:32,786 - INFO: train loss: 0.00: 83%|########2 | 249/300 [07:01<01:25, 1.67s/it] | |
| 2024-08-07 20:28:37,605 - INFO: train loss: 0.00: 84%|########4 | 252/300 [07:06<01:19, 1.65s/it] | |
| 2024-08-07 20:28:42,333 - INFO: train loss: 0.00: 85%|########5 | 255/300 [07:10<01:13, 1.63s/it] | |
| 2024-08-07 20:28:47,654 - INFO: train loss: 0.00: 86%|########6 | 258/300 [07:16<01:10, 1.67s/it] | |
| 2024-08-07 20:28:52,363 - INFO: train loss: 0.00: 87%|########7 | 261/300 [07:20<01:04, 1.64s/it] | |
| 2024-08-07 20:28:56,865 - INFO: train loss: 0.00: 88%|########8 | 264/300 [07:25<00:57, 1.60s/it] | |
| 2024-08-07 20:29:01,962 - INFO: train loss: 0.00: 89%|########9 | 267/300 [07:30<00:53, 1.63s/it] | |
| 2024-08-07 20:29:06,987 - INFO: train loss: 0.00: 90%|######### | 270/300 [07:35<00:49, 1.64s/it] | |
| 2024-08-07 20:29:12,440 - INFO: train loss: 0.00: 91%|#########1| 273/300 [07:40<00:45, 1.70s/it] | |
| 2024-08-07 20:29:17,750 - INFO: train loss: 0.00: 92%|#########2| 276/300 [07:46<00:41, 1.72s/it] | |
| 2024-08-07 20:29:22,608 - INFO: train loss: 0.00: 93%|#########3| 279/300 [07:51<00:35, 1.69s/it] | |
| 2024-08-07 20:29:28,596 - INFO: train loss: 0.00: 94%|#########3| 282/300 [07:57<00:32, 1.78s/it] | |
| 2024-08-07 20:29:34,487 - INFO: train loss: 0.00: 95%|#########5| 285/300 [08:02<00:27, 1.84s/it] | |
| 2024-08-07 20:29:39,896 - INFO: train loss: 0.00: 96%|#########6| 288/300 [08:08<00:21, 1.83s/it] | |
| 2024-08-07 20:29:44,719 - INFO: train loss: 0.00: 97%|#########7| 291/300 [08:13<00:15, 1.76s/it] | |
| 2024-08-07 20:29:49,663 - INFO: train loss: 0.00: 98%|#########8| 294/300 [08:18<00:10, 1.73s/it] | |
| 2024-08-07 20:29:54,606 - INFO: train loss: 0.00: 99%|#########9| 297/300 [08:23<00:05, 1.70s/it] | |
| 2024-08-07 20:29:59,956 - INFO: train loss: 0.00: 100%|##########| 300/300 [08:28<00:00, 1.73s/it] | |
| 2024-08-07 20:29:59,956 - INFO: train loss: 0.00: 100%|##########| 300/300 [08:28<00:00, 1.69s/it] | |
| 2024-08-07 20:29:59,956 - INFO: Saving last model checkpoint to /app/output | |
| 2024-08-07 20:29:59,957 - INFO: Saving checkpoint.. | |
| 2024-08-07 20:30:29,425 - INFO: Starting validation inference | |
| 2024-08-07 20:30:29,426 - INFO: validation progress: 0%| | 0/75 [00:00<?, ?it/s] | |
| 2024-08-07 20:30:34,701 - INFO: validation progress: 4%|4 | 3/75 [00:05<02:06, 1.76s/it] | |
| 2024-08-07 20:30:36,218 - INFO: validation progress: 8%|8 | 6/75 [00:06<01:10, 1.02s/it] | |
| 2024-08-07 20:30:37,655 - INFO: validation progress: 12%|#2 | 9/75 [00:08<00:51, 1.29it/s] | |
| 2024-08-07 20:30:39,155 - INFO: validation progress: 16%|#6 | 12/75 [00:09<00:41, 1.50it/s] | |
| 2024-08-07 20:30:40,762 - INFO: validation progress: 20%|## | 15/75 [00:11<00:37, 1.62it/s] | |
| 2024-08-07 20:30:42,319 - INFO: validation progress: 24%|##4 | 18/75 [00:12<00:33, 1.71it/s] | |
| 2024-08-07 20:30:43,951 - INFO: validation progress: 28%|##8 | 21/75 [00:14<00:30, 1.75it/s] | |
| 2024-08-07 20:30:45,447 - INFO: validation progress: 32%|###2 | 24/75 [00:16<00:27, 1.82it/s] | |
| 2024-08-07 20:30:47,091 - INFO: validation progress: 36%|###6 | 27/75 [00:17<00:26, 1.82it/s] | |
| 2024-08-07 20:30:48,675 - INFO: validation progress: 40%|#### | 30/75 [00:19<00:24, 1.85it/s] | |
| 2024-08-07 20:30:50,310 - INFO: validation progress: 44%|####4 | 33/75 [00:20<00:22, 1.84it/s] | |
| 2024-08-07 20:30:51,946 - INFO: validation progress: 48%|####8 | 36/75 [00:22<00:21, 1.84it/s] | |
| 2024-08-07 20:30:53,607 - INFO: validation progress: 52%|#####2 | 39/75 [00:24<00:19, 1.83it/s] | |
| 2024-08-07 20:30:55,181 - INFO: validation progress: 56%|#####6 | 42/75 [00:25<00:17, 1.85it/s] | |
| 2024-08-07 20:30:56,732 - INFO: validation progress: 60%|###### | 45/75 [00:27<00:15, 1.88it/s] | |
| 2024-08-07 20:30:58,204 - INFO: validation progress: 64%|######4 | 48/75 [00:28<00:14, 1.92it/s] | |
| 2024-08-07 20:30:59,710 - INFO: validation progress: 68%|######8 | 51/75 [00:30<00:12, 1.94it/s] | |
| 2024-08-07 20:31:01,769 - INFO: validation progress: 72%|#######2 | 54/75 [00:32<00:11, 1.77it/s] | |
| 2024-08-07 20:31:03,388 - INFO: validation progress: 76%|#######6 | 57/75 [00:33<00:10, 1.79it/s] | |
| 2024-08-07 20:31:04,843 - INFO: validation progress: 80%|######## | 60/75 [00:35<00:08, 1.86it/s] | |
| 2024-08-07 20:31:06,549 - INFO: validation progress: 84%|########4 | 63/75 [00:37<00:06, 1.83it/s] | |
| 2024-08-07 20:31:08,186 - INFO: validation progress: 88%|########8 | 66/75 [00:38<00:04, 1.83it/s] | |
| 2024-08-07 20:31:09,819 - INFO: validation progress: 92%|#########2| 69/75 [00:40<00:03, 1.83it/s] | |
| 2024-08-07 20:31:11,283 - INFO: validation progress: 96%|#########6| 72/75 [00:41<00:01, 1.89it/s] | |
| 2024-08-07 20:31:13,013 - INFO: validation progress: 100%|##########| 75/75 [00:43<00:00, 1.84it/s] | |
| 2024-08-07 20:31:13,016 - INFO: validation progress: 100%|##########| 75/75 [00:43<00:00, 1.72it/s] | |
| 2024-08-07 20:31:13,295 - INFO: Validation AUC: 0.98502 | |
| 2024-08-07 20:31:13,295 - INFO: Mean validation loss: 0.62766 | |
| 2024-08-07 20:31:13,655 - INFO: Training Epoch: 10 / 10 | |
| 2024-08-07 20:31:13,655 - INFO: train loss: 0%| | 0/300 [00:00<?, ?it/s] | |
| 2024-08-07 20:31:17,753 - INFO: Evaluation step: 300 | |
| 2024-08-07 20:31:22,773 - INFO: train loss: 0.00: 1%|1 | 3/300 [00:09<15:02, 3.04s/it] | |
| 2024-08-07 20:31:27,855 - INFO: train loss: 0.00: 2%|2 | 6/300 [00:14<11:00, 2.25s/it] | |
| 2024-08-07 20:31:34,058 - INFO: train loss: 0.00: 3%|3 | 9/300 [00:20<10:30, 2.17s/it] | |
| 2024-08-07 20:31:38,902 - INFO: train loss: 0.00: 4%|4 | 12/300 [00:25<09:21, 1.95s/it] | |
| 2024-08-07 20:31:44,119 - INFO: train loss: 0.00: 5%|5 | 15/300 [00:30<08:53, 1.87s/it] | |
| 2024-08-07 20:31:49,123 - INFO: train loss: 0.00: 6%|6 | 18/300 [00:35<08:28, 1.80s/it] | |
| 2024-08-07 20:31:53,861 - INFO: train loss: 0.00: 7%|7 | 21/300 [00:40<08:02, 1.73s/it] | |
| 2024-08-07 20:31:58,863 - INFO: train loss: 0.00: 8%|8 | 24/300 [00:45<07:51, 1.71s/it] | |
| 2024-08-07 20:32:03,567 - INFO: train loss: 0.00: 9%|9 | 27/300 [00:49<07:34, 1.67s/it] | |
| 2024-08-07 20:32:08,541 - INFO: train loss: 0.00: 10%|# | 30/300 [00:54<07:29, 1.66s/it] | |
| 2024-08-07 20:32:13,658 - INFO: train loss: 0.00: 11%|#1 | 33/300 [01:00<07:27, 1.68s/it] | |
| 2024-08-07 20:32:18,172 - INFO: train loss: 0.00: 12%|#2 | 36/300 [01:04<07:08, 1.62s/it] | |
| 2024-08-07 20:32:23,681 - INFO: train loss: 0.00: 13%|#3 | 39/300 [01:10<07:20, 1.69s/it] | |
| 2024-08-07 20:32:28,564 - INFO: train loss: 0.00: 14%|#4 | 42/300 [01:14<07:10, 1.67s/it] | |
| 2024-08-07 20:32:33,779 - INFO: train loss: 0.00: 15%|#5 | 45/300 [01:20<07:11, 1.69s/it] | |
| 2024-08-07 20:32:38,734 - INFO: train loss: 0.00: 16%|#6 | 48/300 [01:25<07:03, 1.68s/it] | |
| 2024-08-07 20:32:43,643 - INFO: train loss: 0.00: 17%|#7 | 51/300 [01:29<06:54, 1.67s/it] | |
| 2024-08-07 20:32:49,121 - INFO: train loss: 0.00: 18%|#8 | 54/300 [01:35<07:01, 1.71s/it] | |
| 2024-08-07 20:32:54,109 - INFO: train loss: 0.00: 19%|#9 | 57/300 [01:40<06:52, 1.70s/it] | |
| 2024-08-07 20:32:59,083 - INFO: train loss: 0.00: 20%|## | 60/300 [01:45<06:44, 1.69s/it] | |
| 2024-08-07 20:33:03,709 - INFO: train loss: 0.00: 21%|##1 | 63/300 [01:50<06:29, 1.64s/it] | |
| 2024-08-07 20:33:08,781 - INFO: train loss: 0.00: 22%|##2 | 66/300 [01:55<06:27, 1.66s/it] | |
| 2024-08-07 20:33:13,989 - INFO: train loss: 0.00: 23%|##3 | 69/300 [02:00<06:28, 1.68s/it] | |
| 2024-08-07 20:33:18,529 - INFO: train loss: 0.00: 24%|##4 | 72/300 [02:04<06:11, 1.63s/it] | |
| 2024-08-07 20:33:23,203 - INFO: train loss: 0.00: 25%|##5 | 75/300 [02:09<06:01, 1.61s/it] | |
| 2024-08-07 20:33:28,215 - INFO: train loss: 0.00: 26%|##6 | 78/300 [02:14<06:01, 1.63s/it] | |
| 2024-08-07 20:33:33,197 - INFO: train loss: 0.00: 27%|##7 | 81/300 [02:19<05:58, 1.64s/it] | |
| 2024-08-07 20:33:38,164 - INFO: train loss: 0.00: 28%|##8 | 84/300 [02:24<05:54, 1.64s/it] | |
| 2024-08-07 20:33:43,257 - INFO: train loss: 0.00: 29%|##9 | 87/300 [02:29<05:53, 1.66s/it] | |
| 2024-08-07 20:33:48,455 - INFO: train loss: 0.00: 30%|### | 90/300 [02:34<05:53, 1.68s/it] | |
| 2024-08-07 20:33:53,404 - INFO: train loss: 0.00: 31%|###1 | 93/300 [02:39<05:46, 1.67s/it] | |
| 2024-08-07 20:33:58,486 - INFO: train loss: 0.00: 32%|###2 | 96/300 [02:44<05:42, 1.68s/it] | |
| 2024-08-07 20:34:03,474 - INFO: train loss: 0.00: 33%|###3 | 99/300 [02:49<05:36, 1.67s/it] | |
| 2024-08-07 20:34:08,411 - INFO: train loss: 0.00: 34%|###4 | 102/300 [02:54<05:29, 1.67s/it] | |
| 2024-08-07 20:34:13,419 - INFO: train loss: 0.00: 35%|###5 | 105/300 [02:59<05:24, 1.67s/it] | |
| 2024-08-07 20:34:18,516 - INFO: train loss: 0.00: 36%|###6 | 108/300 [03:04<05:21, 1.68s/it] | |
| 2024-08-07 20:34:23,360 - INFO: train loss: 0.00: 37%|###7 | 111/300 [03:09<05:13, 1.66s/it] | |
| 2024-08-07 20:34:28,372 - INFO: train loss: 0.00: 38%|###8 | 114/300 [03:14<05:09, 1.66s/it] | |
| 2024-08-07 20:34:33,621 - INFO: train loss: 0.00: 39%|###9 | 117/300 [03:19<05:08, 1.69s/it] | |
| 2024-08-07 20:34:38,503 - INFO: train loss: 0.00: 40%|#### | 120/300 [03:24<05:00, 1.67s/it] | |
| 2024-08-07 20:34:43,505 - INFO: train loss: 0.00: 41%|####1 | 123/300 [03:29<04:55, 1.67s/it] | |
| 2024-08-07 20:34:48,661 - INFO: train loss: 0.00: 42%|####2 | 126/300 [03:35<04:53, 1.68s/it] | |
| 2024-08-07 20:34:53,573 - INFO: train loss: 0.00: 43%|####3 | 129/300 [03:39<04:45, 1.67s/it] | |
| 2024-08-07 20:34:59,045 - INFO: train loss: 0.00: 44%|####4 | 132/300 [03:45<04:48, 1.72s/it] | |
| 2024-08-07 20:35:04,183 - INFO: train loss: 0.00: 45%|####5 | 135/300 [03:50<04:42, 1.72s/it] | |
| 2024-08-07 20:35:09,347 - INFO: train loss: 0.00: 46%|####6 | 138/300 [03:55<04:38, 1.72s/it] | |
| 2024-08-07 20:35:14,334 - INFO: train loss: 0.00: 47%|####6 | 141/300 [04:00<04:30, 1.70s/it] | |
| 2024-08-07 20:35:19,823 - INFO: train loss: 0.00: 48%|####8 | 144/300 [04:06<04:31, 1.74s/it] | |
| 2024-08-07 20:35:24,659 - INFO: train loss: 0.00: 49%|####9 | 147/300 [04:11<04:20, 1.70s/it] | |
| 2024-08-07 20:35:30,031 - INFO: train loss: 0.00: 50%|##### | 150/300 [04:16<04:19, 1.73s/it] | |
| 2024-08-07 20:35:34,793 - INFO: train loss: 0.00: 51%|#####1 | 153/300 [04:21<04:07, 1.69s/it] | |
| 2024-08-07 20:35:39,878 - INFO: train loss: 0.00: 52%|#####2 | 156/300 [04:26<04:03, 1.69s/it] | |
| 2024-08-07 20:35:44,837 - INFO: train loss: 0.00: 53%|#####3 | 159/300 [04:31<03:56, 1.68s/it] | |
| 2024-08-07 20:35:49,843 - INFO: train loss: 0.00: 54%|#####4 | 162/300 [04:36<03:51, 1.68s/it] | |
| 2024-08-07 20:35:54,615 - INFO: train loss: 0.00: 55%|#####5 | 165/300 [04:40<03:42, 1.65s/it] | |
| 2024-08-07 20:35:59,471 - INFO: train loss: 0.00: 56%|#####6 | 168/300 [04:45<03:36, 1.64s/it] | |
| 2024-08-07 20:36:04,296 - INFO: train loss: 0.00: 57%|#####6 | 171/300 [04:50<03:30, 1.63s/it] | |
| 2024-08-07 20:36:09,285 - INFO: train loss: 0.00: 58%|#####8 | 174/300 [04:55<03:26, 1.64s/it] | |
| 2024-08-07 20:36:13,895 - INFO: train loss: 0.00: 59%|#####8 | 177/300 [05:00<03:17, 1.61s/it] | |
| 2024-08-07 20:36:18,875 - INFO: train loss: 0.00: 60%|###### | 180/300 [05:05<03:14, 1.62s/it] | |
| 2024-08-07 20:36:23,555 - INFO: train loss: 0.00: 61%|######1 | 183/300 [05:09<03:07, 1.61s/it] | |
| 2024-08-07 20:36:28,927 - INFO: train loss: 0.00: 62%|######2 | 186/300 [05:15<03:09, 1.66s/it] | |
| 2024-08-07 20:36:34,270 - INFO: train loss: 0.00: 63%|######3 | 189/300 [05:20<03:08, 1.70s/it] | |
| 2024-08-07 20:36:38,927 - INFO: train loss: 0.00: 64%|######4 | 192/300 [05:25<02:58, 1.65s/it] | |
| 2024-08-07 20:36:44,145 - INFO: train loss: 0.00: 65%|######5 | 195/300 [05:30<02:56, 1.68s/it] | |
| 2024-08-07 20:36:49,154 - INFO: train loss: 0.00: 66%|######6 | 198/300 [05:35<02:50, 1.68s/it] | |
| 2024-08-07 20:36:53,844 - INFO: train loss: 0.00: 67%|######7 | 201/300 [05:40<02:42, 1.64s/it] | |
| 2024-08-07 20:37:00,123 - INFO: train loss: 0.00: 68%|######8 | 204/300 [05:46<02:50, 1.78s/it] | |
| 2024-08-07 20:37:05,665 - INFO: train loss: 0.00: 69%|######9 | 207/300 [05:52<02:47, 1.80s/it] | |
| 2024-08-07 20:37:10,768 - INFO: train loss: 0.00: 70%|####### | 210/300 [05:57<02:39, 1.77s/it] | |
| 2024-08-07 20:37:15,538 - INFO: train loss: 0.00: 71%|#######1 | 213/300 [06:01<02:29, 1.72s/it] | |
| 2024-08-07 20:37:20,535 - INFO: train loss: 0.00: 72%|#######2 | 216/300 [06:06<02:22, 1.70s/it] | |
| 2024-08-07 20:37:25,562 - INFO: train loss: 0.00: 73%|#######3 | 219/300 [06:11<02:17, 1.69s/it] | |
| 2024-08-07 20:37:30,959 - INFO: train loss: 0.00: 74%|#######4 | 222/300 [06:17<02:14, 1.72s/it] | |
| 2024-08-07 20:37:36,096 - INFO: train loss: 0.00: 75%|#######5 | 225/300 [06:22<02:09, 1.72s/it] | |
| 2024-08-07 20:37:41,524 - INFO: train loss: 0.00: 76%|#######6 | 228/300 [06:27<02:05, 1.75s/it] | |
| 2024-08-07 20:37:46,362 - INFO: train loss: 0.00: 77%|#######7 | 231/300 [06:32<01:57, 1.71s/it] | |
| 2024-08-07 20:37:51,208 - INFO: train loss: 0.00: 78%|#######8 | 234/300 [06:37<01:50, 1.68s/it] | |
| 2024-08-07 20:37:56,321 - INFO: train loss: 0.00: 79%|#######9 | 237/300 [06:42<01:46, 1.69s/it] | |
| 2024-08-07 20:38:01,325 - INFO: train loss: 0.00: 80%|######## | 240/300 [06:47<01:40, 1.68s/it] | |
| 2024-08-07 20:38:06,370 - INFO: train loss: 0.00: 81%|########1 | 243/300 [06:52<01:35, 1.68s/it] | |
| 2024-08-07 20:38:11,090 - INFO: train loss: 0.00: 82%|########2 | 246/300 [06:57<01:29, 1.65s/it] | |
| 2024-08-07 20:38:16,276 - INFO: train loss: 0.00: 83%|########2 | 249/300 [07:02<01:25, 1.67s/it] | |
| 2024-08-07 20:38:21,081 - INFO: train loss: 0.00: 84%|########4 | 252/300 [07:07<01:19, 1.65s/it] | |
| 2024-08-07 20:38:26,445 - INFO: train loss: 0.00: 85%|########5 | 255/300 [07:12<01:16, 1.69s/it] | |
| 2024-08-07 20:38:31,447 - INFO: train loss: 0.00: 86%|########6 | 258/300 [07:17<01:10, 1.68s/it] | |
| 2024-08-07 20:38:36,844 - INFO: train loss: 0.00: 87%|########7 | 261/300 [07:23<01:07, 1.72s/it] | |
| 2024-08-07 20:38:41,909 - INFO: train loss: 0.00: 88%|########8 | 264/300 [07:28<01:01, 1.71s/it] | |
| 2024-08-07 20:38:46,957 - INFO: train loss: 0.00: 89%|########9 | 267/300 [07:33<00:56, 1.70s/it] | |
| 2024-08-07 20:38:51,537 - INFO: train loss: 0.00: 90%|######### | 270/300 [07:37<00:49, 1.65s/it] | |
| 2024-08-07 20:38:56,346 - INFO: train loss: 0.00: 91%|#########1| 273/300 [07:42<00:44, 1.64s/it] | |
| 2024-08-07 20:39:01,231 - INFO: train loss: 0.00: 92%|#########2| 276/300 [07:47<00:39, 1.63s/it] | |
| 2024-08-07 20:39:05,855 - INFO: train loss: 0.00: 93%|#########3| 279/300 [07:52<00:33, 1.61s/it] | |
| 2024-08-07 20:39:10,535 - INFO: train loss: 0.00: 94%|#########3| 282/300 [07:56<00:28, 1.59s/it] | |
| 2024-08-07 20:39:15,560 - INFO: train loss: 0.00: 95%|#########5| 285/300 [08:01<00:24, 1.62s/it] | |
| 2024-08-07 20:39:20,898 - INFO: train loss: 0.00: 96%|#########6| 288/300 [08:07<00:19, 1.67s/it] | |
| 2024-08-07 20:39:25,862 - INFO: train loss: 0.00: 97%|#########7| 291/300 [08:12<00:14, 1.66s/it] | |
| 2024-08-07 20:39:30,817 - INFO: train loss: 0.00: 98%|#########8| 294/300 [08:17<00:09, 1.66s/it] | |
| 2024-08-07 20:39:36,277 - INFO: train loss: 0.00: 99%|#########9| 297/300 [08:22<00:05, 1.71s/it] | |
| 2024-08-07 20:39:40,891 - INFO: train loss: 0.00: 100%|##########| 300/300 [08:27<00:00, 1.66s/it] | |
| 2024-08-07 20:39:40,894 - INFO: train loss: 0.00: 100%|##########| 300/300 [08:27<00:00, 1.69s/it] | |
| 2024-08-07 20:39:40,894 - INFO: Saving last model checkpoint to /app/output | |
| 2024-08-07 20:39:40,894 - INFO: Saving checkpoint.. | |
| 2024-08-07 20:40:08,676 - INFO: Starting validation inference | |
| 2024-08-07 20:40:08,678 - INFO: validation progress: 0%| | 0/75 [00:00<?, ?it/s] | |
| 2024-08-07 20:40:15,254 - INFO: validation progress: 4%|4 | 3/75 [00:06<02:37, 2.19s/it] | |
| 2024-08-07 20:40:16,775 - INFO: validation progress: 8%|8 | 6/75 [00:08<01:22, 1.20s/it] | |
| 2024-08-07 20:40:18,262 - INFO: validation progress: 12%|#2 | 9/75 [00:09<00:58, 1.14it/s] | |
| 2024-08-07 20:40:19,824 - INFO: validation progress: 16%|#6 | 12/75 [00:11<00:46, 1.36it/s] | |
| 2024-08-07 20:40:21,434 - INFO: validation progress: 20%|## | 15/75 [00:12<00:39, 1.50it/s] | |
| 2024-08-07 20:40:22,999 - INFO: validation progress: 24%|##4 | 18/75 [00:14<00:35, 1.62it/s] | |
| 2024-08-07 20:40:24,632 - INFO: validation progress: 28%|##8 | 21/75 [00:15<00:32, 1.69it/s] | |
| 2024-08-07 20:40:26,126 - INFO: validation progress: 32%|###2 | 24/75 [00:17<00:28, 1.78it/s] | |
| 2024-08-07 20:40:27,817 - INFO: validation progress: 36%|###6 | 27/75 [00:19<00:27, 1.78it/s] | |
| 2024-08-07 20:40:29,457 - INFO: validation progress: 40%|#### | 30/75 [00:20<00:25, 1.79it/s] | |
| 2024-08-07 20:40:31,090 - INFO: validation progress: 44%|####4 | 33/75 [00:22<00:23, 1.81it/s] | |
| 2024-08-07 20:40:32,727 - INFO: validation progress: 48%|####8 | 36/75 [00:24<00:21, 1.81it/s] | |
| 2024-08-07 20:40:34,393 - INFO: validation progress: 52%|#####2 | 39/75 [00:25<00:19, 1.81it/s] | |
| 2024-08-07 20:40:35,972 - INFO: validation progress: 56%|#####6 | 42/75 [00:27<00:17, 1.84it/s] | |
| 2024-08-07 20:40:37,524 - INFO: validation progress: 60%|###### | 45/75 [00:28<00:16, 1.86it/s] | |
| 2024-08-07 20:40:38,989 - INFO: validation progress: 64%|######4 | 48/75 [00:30<00:14, 1.92it/s] | |
| 2024-08-07 20:40:40,531 - INFO: validation progress: 68%|######8 | 51/75 [00:31<00:12, 1.92it/s] | |
| 2024-08-07 20:40:42,159 - INFO: validation progress: 72%|#######2 | 54/75 [00:33<00:11, 1.90it/s] | |
| 2024-08-07 20:40:43,750 - INFO: validation progress: 76%|#######6 | 57/75 [00:35<00:09, 1.90it/s] | |
| 2024-08-07 20:40:45,191 - INFO: validation progress: 80%|######## | 60/75 [00:36<00:07, 1.95it/s] | |
| 2024-08-07 20:40:46,900 - INFO: validation progress: 84%|########4 | 63/75 [00:38<00:06, 1.89it/s] | |
| 2024-08-07 20:40:48,533 - INFO: validation progress: 88%|########8 | 66/75 [00:39<00:04, 1.87it/s] | |
| 2024-08-07 20:40:50,171 - INFO: validation progress: 92%|#########2| 69/75 [00:41<00:03, 1.86it/s] | |
| 2024-08-07 20:40:51,636 - INFO: validation progress: 96%|#########6| 72/75 [00:42<00:01, 1.91it/s] | |
| 2024-08-07 20:40:53,364 - INFO: validation progress: 100%|##########| 75/75 [00:44<00:00, 1.86it/s] | |
| 2024-08-07 20:40:53,368 - INFO: validation progress: 100%|##########| 75/75 [00:44<00:00, 1.68it/s] | |
| 2024-08-07 20:40:53,666 - INFO: Validation AUC: 0.98502 | |
| 2024-08-07 20:40:53,666 - INFO: Mean validation loss: 0.62995 | |