Model save

README.md

@@ -0,0 +1,67 @@
+---
+library_name: transformers
+license: llama3
+base_model: meta-llama/Meta-Llama-3-8B
+tags:
+- llama-factory
+- generated_from_trainer
+model-index:
+- name: llama3_8b_baseline_dcft_oh_v3
+  results: []
+---
+
+<!-- This model card has been generated automatically according to the information the Trainer had access to. You
+should probably proofread and complete it, then remove this comment. -->
+
+# llama3_8b_baseline_dcft_oh_v3
+
+This model is a fine-tuned version of [meta-llama/Meta-Llama-3-8B](https://huggingface.co/meta-llama/Meta-Llama-3-8B) on an unknown dataset.
+It achieves the following results on the evaluation set:
+- Loss: 0.6458
+
+## Model description
+
+More information needed
+
+## Intended uses & limitations
+
+More information needed
+
+## Training and evaluation data
+
+More information needed
+
+## Training procedure
+
+### Training hyperparameters
+
+The following hyperparameters were used during training:
+- learning_rate: 5e-06
+- train_batch_size: 16
+- eval_batch_size: 8
+- seed: 42
+- distributed_type: multi-GPU
+- num_devices: 32
+- total_train_batch_size: 512
+- total_eval_batch_size: 256
+- optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
+- lr_scheduler_type: constant
+- lr_scheduler_warmup_ratio: 0.1
+- lr_scheduler_warmup_steps: 1738
+- num_epochs: 3.0
+
+### Training results
+
+| Training Loss | Epoch | Step | Validation Loss |
+|:-------------:|:-----:|:----:|:---------------:|
+| 0.6524        | 1.0   | 423  | 0.6508          |
+| 0.6057        | 2.0   | 846  | 0.6412          |
+| 0.577         | 3.0   | 1269 | 0.6458          |
+
+
+### Framework versions
+
+- Transformers 4.44.2
+- Pytorch 2.4.0
+- Datasets 2.21.0
+- Tokenizers 0.19.1

generation_config.json

@@ -0,0 +1,9 @@
+{
+  "bos_token_id": 128000,
+  "do_sample": true,
+  "eos_token_id": 128001,
+  "max_length": 4096,
+  "temperature": 0.6,
+  "top_p": 0.9,
+  "transformers_version": "4.44.2"
+}

model-00001-of-00004.safetensors

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:982f1c5b42195d197dbb3331de198374ff61835fe9ab6d78caf4f61007d76185
+oid sha256:b04a8e21e9c154354a748905526266c7fccaaea76d106919ed57afd7f820a0bc
 size 4976698672

model-00002-of-00004.safetensors

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:0b97af9d4b86001d7affa3555abe036f95c9aea8d732ec3d80230d8da8a52689
+oid sha256:cdd3d90cdf2a74b45c5487042420a9720e26ea3ceca7a22483656eaf48230c2b
 size 4999802720

model-00003-of-00004.safetensors

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:7a990d9454e170744a40253e045ed580824a99dfc09ca1a408395684d7674408
+oid sha256:3ee13d919059baf7234c6011b1258db2a7cc7bc8d9571068b78a73398c7b8c12
 size 4915916176

model-00004-of-00004.safetensors

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:434b04fe3e7aa39def57253dc62adc0f80b16bba66fea26b5d9fdadc016eb86d
+oid sha256:b695c4006fa5740b3cb271719734a0f3afb4585cae5c13e6306434fe86e3f2df
 size 1168138808

trainer_log.jsonl

@@ -87,3 +87,44 @@
 {"current_steps": 850, "total_steps": 1269, "loss": 0.5827, "learning_rate": 5e-06, "epoch": 2.0094562647754137, "percentage": 66.98, "elapsed_time": "1:39:32", "remaining_time": "0:49:04"}
 {"current_steps": 860, "total_steps": 1269, "loss": 0.5511, "learning_rate": 5e-06, "epoch": 2.033096926713948, "percentage": 67.77, "elapsed_time": "1:40:35", "remaining_time": "0:47:50"}
 {"current_steps": 870, "total_steps": 1269, "loss": 0.5581, "learning_rate": 5e-06, "epoch": 2.0567375886524824, "percentage": 68.56, "elapsed_time": "1:41:48", "remaining_time": "0:46:41"}
+{"current_steps": 880, "total_steps": 1269, "loss": 0.5513, "learning_rate": 5e-06, "epoch": 2.0803782505910164, "percentage": 69.35, "elapsed_time": "1:42:57", "remaining_time": "0:45:30"}
+{"current_steps": 890, "total_steps": 1269, "loss": 0.5585, "learning_rate": 5e-06, "epoch": 2.104018912529551, "percentage": 70.13, "elapsed_time": "1:43:59", "remaining_time": "0:44:17"}
+{"current_steps": 900, "total_steps": 1269, "loss": 0.5571, "learning_rate": 5e-06, "epoch": 2.127659574468085, "percentage": 70.92, "elapsed_time": "1:45:02", "remaining_time": "0:43:04"}
+{"current_steps": 910, "total_steps": 1269, "loss": 0.5632, "learning_rate": 5e-06, "epoch": 2.1513002364066196, "percentage": 71.71, "elapsed_time": "1:46:11", "remaining_time": "0:41:53"}
+{"current_steps": 920, "total_steps": 1269, "loss": 0.5613, "learning_rate": 5e-06, "epoch": 2.1749408983451537, "percentage": 72.5, "elapsed_time": "1:47:26", "remaining_time": "0:40:45"}
+{"current_steps": 930, "total_steps": 1269, "loss": 0.5611, "learning_rate": 5e-06, "epoch": 2.198581560283688, "percentage": 73.29, "elapsed_time": "1:48:28", "remaining_time": "0:39:32"}
+{"current_steps": 940, "total_steps": 1269, "loss": 0.5572, "learning_rate": 5e-06, "epoch": 2.2222222222222223, "percentage": 74.07, "elapsed_time": "1:49:31", "remaining_time": "0:38:20"}
+{"current_steps": 950, "total_steps": 1269, "loss": 0.561, "learning_rate": 5e-06, "epoch": 2.2458628841607564, "percentage": 74.86, "elapsed_time": "1:50:34", "remaining_time": "0:37:07"}
+{"current_steps": 960, "total_steps": 1269, "loss": 0.5593, "learning_rate": 5e-06, "epoch": 2.269503546099291, "percentage": 75.65, "elapsed_time": "1:51:36", "remaining_time": "0:35:55"}
+{"current_steps": 970, "total_steps": 1269, "loss": 0.5677, "learning_rate": 5e-06, "epoch": 2.293144208037825, "percentage": 76.44, "elapsed_time": "1:52:39", "remaining_time": "0:34:43"}
+{"current_steps": 980, "total_steps": 1269, "loss": 0.5586, "learning_rate": 5e-06, "epoch": 2.3167848699763596, "percentage": 77.23, "elapsed_time": "1:53:42", "remaining_time": "0:33:31"}
+{"current_steps": 990, "total_steps": 1269, "loss": 0.5568, "learning_rate": 5e-06, "epoch": 2.3404255319148937, "percentage": 78.01, "elapsed_time": "1:54:44", "remaining_time": "0:32:20"}
+{"current_steps": 1000, "total_steps": 1269, "loss": 0.5559, "learning_rate": 5e-06, "epoch": 2.3640661938534278, "percentage": 78.8, "elapsed_time": "1:55:47", "remaining_time": "0:31:08"}
+{"current_steps": 1010, "total_steps": 1269, "loss": 0.5578, "learning_rate": 5e-06, "epoch": 2.3877068557919623, "percentage": 79.59, "elapsed_time": "1:56:49", "remaining_time": "0:29:57"}
+{"current_steps": 1020, "total_steps": 1269, "loss": 0.5666, "learning_rate": 5e-06, "epoch": 2.4113475177304964, "percentage": 80.38, "elapsed_time": "1:57:52", "remaining_time": "0:28:46"}
+{"current_steps": 1030, "total_steps": 1269, "loss": 0.5598, "learning_rate": 5e-06, "epoch": 2.434988179669031, "percentage": 81.17, "elapsed_time": "1:59:03", "remaining_time": "0:27:37"}
+{"current_steps": 1040, "total_steps": 1269, "loss": 0.562, "learning_rate": 5e-06, "epoch": 2.458628841607565, "percentage": 81.95, "elapsed_time": "2:00:13", "remaining_time": "0:26:28"}
+{"current_steps": 1050, "total_steps": 1269, "loss": 0.5595, "learning_rate": 5e-06, "epoch": 2.482269503546099, "percentage": 82.74, "elapsed_time": "2:01:15", "remaining_time": "0:25:17"}
+{"current_steps": 1060, "total_steps": 1269, "loss": 0.5643, "learning_rate": 5e-06, "epoch": 2.5059101654846336, "percentage": 83.53, "elapsed_time": "2:02:18", "remaining_time": "0:24:06"}
+{"current_steps": 1070, "total_steps": 1269, "loss": 0.5527, "learning_rate": 5e-06, "epoch": 2.5295508274231677, "percentage": 84.32, "elapsed_time": "2:03:23", "remaining_time": "0:22:56"}
+{"current_steps": 1080, "total_steps": 1269, "loss": 0.563, "learning_rate": 5e-06, "epoch": 2.5531914893617023, "percentage": 85.11, "elapsed_time": "2:04:38", "remaining_time": "0:21:48"}
+{"current_steps": 1090, "total_steps": 1269, "loss": 0.5601, "learning_rate": 5e-06, "epoch": 2.5768321513002364, "percentage": 85.89, "elapsed_time": "2:05:41", "remaining_time": "0:20:38"}
+{"current_steps": 1100, "total_steps": 1269, "loss": 0.559, "learning_rate": 5e-06, "epoch": 2.6004728132387704, "percentage": 86.68, "elapsed_time": "2:06:48", "remaining_time": "0:19:28"}
+{"current_steps": 1110, "total_steps": 1269, "loss": 0.5671, "learning_rate": 5e-06, "epoch": 2.624113475177305, "percentage": 87.47, "elapsed_time": "2:08:00", "remaining_time": "0:18:20"}
+{"current_steps": 1120, "total_steps": 1269, "loss": 0.5621, "learning_rate": 5e-06, "epoch": 2.6477541371158395, "percentage": 88.26, "elapsed_time": "2:09:02", "remaining_time": "0:17:10"}
+{"current_steps": 1130, "total_steps": 1269, "loss": 0.5649, "learning_rate": 5e-06, "epoch": 2.6713947990543736, "percentage": 89.05, "elapsed_time": "2:10:05", "remaining_time": "0:16:00"}
+{"current_steps": 1140, "total_steps": 1269, "loss": 0.5607, "learning_rate": 5e-06, "epoch": 2.6950354609929077, "percentage": 89.83, "elapsed_time": "2:11:09", "remaining_time": "0:14:50"}
+{"current_steps": 1150, "total_steps": 1269, "loss": 0.563, "learning_rate": 5e-06, "epoch": 2.7186761229314422, "percentage": 90.62, "elapsed_time": "2:12:24", "remaining_time": "0:13:42"}
+{"current_steps": 1160, "total_steps": 1269, "loss": 0.5645, "learning_rate": 5e-06, "epoch": 2.7423167848699763, "percentage": 91.41, "elapsed_time": "2:13:26", "remaining_time": "0:12:32"}
+{"current_steps": 1170, "total_steps": 1269, "loss": 0.5595, "learning_rate": 5e-06, "epoch": 2.7659574468085104, "percentage": 92.2, "elapsed_time": "2:14:29", "remaining_time": "0:11:22"}
+{"current_steps": 1180, "total_steps": 1269, "loss": 0.5664, "learning_rate": 5e-06, "epoch": 2.789598108747045, "percentage": 92.99, "elapsed_time": "2:15:32", "remaining_time": "0:10:13"}
+{"current_steps": 1190, "total_steps": 1269, "loss": 0.5651, "learning_rate": 5e-06, "epoch": 2.813238770685579, "percentage": 93.77, "elapsed_time": "2:16:34", "remaining_time": "0:09:04"}
+{"current_steps": 1200, "total_steps": 1269, "loss": 0.5553, "learning_rate": 5e-06, "epoch": 2.8368794326241136, "percentage": 94.56, "elapsed_time": "2:17:37", "remaining_time": "0:07:54"}
+{"current_steps": 1210, "total_steps": 1269, "loss": 0.5632, "learning_rate": 5e-06, "epoch": 2.8605200945626477, "percentage": 95.35, "elapsed_time": "2:18:39", "remaining_time": "0:06:45"}
+{"current_steps": 1220, "total_steps": 1269, "loss": 0.5657, "learning_rate": 5e-06, "epoch": 2.884160756501182, "percentage": 96.14, "elapsed_time": "2:19:42", "remaining_time": "0:05:36"}
+{"current_steps": 1230, "total_steps": 1269, "loss": 0.5617, "learning_rate": 5e-06, "epoch": 2.9078014184397163, "percentage": 96.93, "elapsed_time": "2:20:44", "remaining_time": "0:04:27"}
+{"current_steps": 1240, "total_steps": 1269, "loss": 0.5632, "learning_rate": 5e-06, "epoch": 2.9314420803782504, "percentage": 97.71, "elapsed_time": "2:21:47", "remaining_time": "0:03:18"}
+{"current_steps": 1250, "total_steps": 1269, "loss": 0.5646, "learning_rate": 5e-06, "epoch": 2.955082742316785, "percentage": 98.5, "elapsed_time": "2:22:49", "remaining_time": "0:02:10"}
+{"current_steps": 1260, "total_steps": 1269, "loss": 0.577, "learning_rate": 5e-06, "epoch": 2.978723404255319, "percentage": 99.29, "elapsed_time": "2:23:52", "remaining_time": "0:01:01"}
+{"current_steps": 1269, "total_steps": 1269, "eval_loss": 0.6458395719528198, "epoch": 3.0, "percentage": 100.0, "elapsed_time": "2:29:14", "remaining_time": "0:00:00"}
+{"current_steps": 1269, "total_steps": 1269, "epoch": 3.0, "percentage": 100.0, "elapsed_time": "2:32:36", "remaining_time": "0:00:00"}