|
2024-05-15 13:34:20,264 - INFO: Calling run.. |
|
2024-05-15 13:34:20,264 - INFO: Problem Type: text_causal_language_modeling |
|
2024-05-15 13:34:20,264 - INFO: Global random seed: 916310 |
|
2024-05-15 13:34:20,264 - INFO: Preparing the data... |
|
2024-05-15 13:34:20,265 - INFO: Setting up automatic validation split... |
|
2024-05-15 13:34:20,271 - INFO: Preparing train and validation data |
|
2024-05-15 13:34:20,271 - INFO: Loading train dataset... |
|
2024-05-15 13:34:21,196 - INFO: Stop token ids: [tensor([ 27, 91, 41681, 91, 29]), tensor([ 27, 91, 9399, 91, 29]), tensor([ 27, 91, 9125, 91, 29])] |
|
2024-05-15 13:34:21,210 - INFO: Loading validation dataset... |
|
2024-05-15 13:34:21,608 - INFO: Stop token ids: [tensor([ 27, 91, 41681, 91, 29]), tensor([ 27, 91, 9399, 91, 29]), tensor([ 27, 91, 9125, 91, 29])] |
|
2024-05-15 13:34:21,625 - INFO: Number of observations in train dataset: 15 |
|
2024-05-15 13:34:21,625 - INFO: Number of observations in validation dataset: 1 |
|
2024-05-15 13:34:22,161 - INFO: Stop token ids: [tensor([ 27, 91, 41681, 91, 29], device='cuda:0'), tensor([ 27, 91, 9399, 91, 29], device='cuda:0'), tensor([ 27, 91, 9125, 91, 29], device='cuda:0')] |
|
2024-05-15 13:34:22,173 - WARNING: PAD token id not matching between config and tokenizer. Overwriting with tokenizer id. |
|
2024-05-15 13:34:22,173 - INFO: Setting pretraining_tp of model config to 1. |
|
2024-05-15 13:34:22,188 - INFO: Using bfloat16 for backbone |
|
2024-05-15 13:34:22,188 - INFO: Loading meta-llama/Meta-Llama-3-8B. This may take a while. |
|
2024-05-15 13:35:50,131 - INFO: Loaded meta-llama/Meta-Llama-3-8B. |
|
2024-05-15 13:35:50,134 - WARNING: PAD token id not matching between generation config and tokenizer. Overwriting with tokenizer id. |
|
2024-05-15 13:35:50,135 - INFO: Lora module names: ['q_proj', 'k_proj', 'v_proj', 'o_proj', 'gate_proj', 'up_proj', 'down_proj'] |
|
2024-05-15 13:35:50,379 - INFO: Enough space available for saving model weights.Required space: 15817.20MB, Available space: 983440.06MB. |
|
2024-05-15 13:35:50,387 - INFO: Optimizer AdamW has been provided with parameters {'eps': 1e-08, 'weight_decay': 0.0, 'betas': (0.8999999762, 0.9990000129), 'lr': 0.0001} |
|
2024-05-15 13:35:51,788 - INFO: started process: 0, can_track: True, tracking_mode: TrackingMode.AFTER_EPOCH |
|
2024-05-15 13:35:51,788 - INFO: Training Epoch: 1 / 2 |
|
2024-05-15 13:35:51,789 - INFO: train loss: 0%| | 0/3 [00:00<?, ?it/s] |
|
2024-05-15 13:35:51,922 - INFO: Evaluation step: 3 |
|
2024-05-15 13:35:53,635 - INFO: train loss: 6.76: 33%|###3 | 1/3 [00:01<00:03, 1.85s/it] |
|
2024-05-15 13:35:54,389 - INFO: train loss: 6.14: 67%|######6 | 2/3 [00:02<00:01, 1.20s/it] |
|
2024-05-15 13:35:55,354 - INFO: train loss: 4.83: 100%|##########| 3/3 [00:03<00:00, 1.09s/it] |
|
2024-05-15 13:35:55,354 - INFO: train loss: 4.83: 100%|##########| 3/3 [00:03<00:00, 1.19s/it] |
|
2024-05-15 13:35:55,354 - INFO: Saving last model checkpoint to /app/output |
|
2024-05-15 13:35:55,354 - INFO: Saving checkpoint.. |
|
2024-05-15 13:36:45,100 - INFO: Starting validation inference |
|
2024-05-15 13:36:45,100 - INFO: validation progress: 0%| | 0/1 [00:00<?, ?it/s] |
|
2024-05-15 13:36:45,330 - INFO: validation progress: 100%|##########| 1/1 [00:00<00:00, 4.35it/s] |
|
2024-05-15 13:36:45,334 - INFO: validation progress: 100%|##########| 1/1 [00:00<00:00, 4.29it/s] |
|
2024-05-15 13:36:45,371 - INFO: Validation Perplexity: 1.52548 |
|
2024-05-15 13:36:45,371 - INFO: Mean validation loss: 0.42231 |
|
2024-05-15 13:36:45,908 - INFO: Training Epoch: 2 / 2 |
|
2024-05-15 13:36:45,908 - INFO: train loss: 0%| | 0/3 [00:00<?, ?it/s] |
|
2024-05-15 13:36:45,995 - INFO: Evaluation step: 3 |
|
2024-05-15 13:36:46,947 - INFO: train loss: 0.44: 33%|###3 | 1/3 [00:01<00:02, 1.04s/it] |
|
2024-05-15 13:36:47,767 - INFO: train loss: 0.29: 67%|######6 | 2/3 [00:01<00:00, 1.10it/s] |
|
2024-05-15 13:36:49,228 - INFO: train loss: 0.23: 100%|##########| 3/3 [00:03<00:00, 1.16s/it] |
|
2024-05-15 13:36:49,228 - INFO: train loss: 0.23: 100%|##########| 3/3 [00:03<00:00, 1.11s/it] |
|
2024-05-15 13:36:49,228 - INFO: Saving last model checkpoint to /app/output |
|
2024-05-15 13:36:49,229 - INFO: Saving checkpoint.. |
|
2024-05-15 13:37:33,992 - INFO: Starting validation inference |
|
2024-05-15 13:37:33,993 - INFO: validation progress: 0%| | 0/1 [00:00<?, ?it/s] |
|
2024-05-15 13:37:34,208 - INFO: validation progress: 100%|##########| 1/1 [00:00<00:00, 4.65it/s] |
|
2024-05-15 13:37:34,210 - INFO: validation progress: 100%|##########| 1/1 [00:00<00:00, 4.59it/s] |
|
2024-05-15 13:37:34,242 - INFO: Validation Perplexity: 1.07148 |
|
2024-05-15 13:37:34,242 - INFO: Mean validation loss: 0.06904 |
|
|
