SmolLM-1.7B-Instruct_fsdp_qlora_nf4_adapter

README.md

@@ -20,7 +20,7 @@ should probably proofread and complete it, then remove this comment. -->
 
 This model is a fine-tuned version of [HuggingFaceTB/SmolLM-1.7B-Instruct](https://huggingface.co/HuggingFaceTB/SmolLM-1.7B-Instruct) on the generator dataset.
 It achieves the following results on the evaluation set:
-- Loss: 1.6111
+- Loss: 1.6524
 
 ## Model description
 
@@ -51,32 +51,22 @@ The following hyperparameters were used during training:
 - optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
 - lr_scheduler_type: cosine
 - lr_scheduler_warmup_ratio: 0.03
-- num_epochs: 20
+- num_epochs: 10
 
 ### Training results
 
-| Training Loss | Epoch   | Step | Validation Loss |
-|:-------------:|:-------:|:----:|:---------------:|
-| 2.0769        | 0.9524  | 10   | 1.9176          |
-| 1.8602        | 2.0     | 21   | 1.7910          |
-| 1.7729        | 2.9524  | 31   | 1.7320          |
-| 1.7147        | 4.0     | 42   | 1.6913          |
-| 1.6753        | 4.9524  | 52   | 1.6662          |
-| 1.6518        | 6.0     | 63   | 1.6477          |
-| 1.6228        | 6.9524  | 73   | 1.6361          |
-| 1.6118        | 8.0     | 84   | 1.6274          |
-| 1.5843        | 8.9524  | 94   | 1.6214          |
-| 1.5805        | 10.0    | 105  | 1.6173          |
-| 1.5712        | 10.9524 | 115  | 1.6151          |
-| 1.5524        | 12.0    | 126  | 1.6133          |
-| 1.5491        | 12.9524 | 136  | 1.6121          |
-| 1.5445        | 14.0    | 147  | 1.6113          |
-| 1.5397        | 14.9524 | 157  | 1.6113          |
-| 1.5392        | 16.0    | 168  | 1.6114          |
-| 1.5337        | 16.9524 | 178  | 1.6111          |
-| 1.5347        | 18.0    | 189  | 1.6111          |
-| 1.5337        | 18.9524 | 199  | 1.6111          |
-| 1.5351        | 19.0476 | 200  | 1.6111          |
+| Training Loss | Epoch  | Step | Validation Loss |
+|:-------------:|:------:|:----:|:---------------:|
+| 2.0516        | 0.9756 | 10   | 1.8985          |
+| 1.8456        | 1.9512 | 20   | 1.7876          |
+| 1.7672        | 2.9268 | 30   | 1.7327          |
+| 1.712         | 4.0    | 41   | 1.6967          |
+| 1.6874        | 4.9756 | 51   | 1.6761          |
+| 1.6643        | 5.9512 | 61   | 1.6636          |
+| 1.6426        | 6.9268 | 71   | 1.6565          |
+| 1.6388        | 8.0    | 82   | 1.6530          |
+| 1.6387        | 8.9756 | 92   | 1.6524          |
+| 1.6343        | 9.7561 | 100  | 1.6524          |
 
 
 ### Framework versions

adapter_config.json

@@ -20,13 +20,13 @@
   "rank_pattern": {},
   "revision": null,
   "target_modules": [
-    "k_proj",
-    "v_proj",
-    "gate_proj",
     "o_proj",
+    "v_proj",
+    "k_proj",
     "up_proj",
     "q_proj",
-    "down_proj"
+    "down_proj",
+    "gate_proj"
   ],
   "task_type": "CAUSAL_LM",
   "use_dora": false,

adapter_model.safetensors

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:c3a4c40f6a10d6fc3272a401bd4d0f52278f41ae12d179f9cadc4402d40e0279
+oid sha256:cefc889333611152b6295d43b9f9f88a837b53912f9925dbb6c9ba32a36d3978
 size 36220744

runs/Sep05_16-31-17_algo-3/events.out.tfevents.1725553937.algo-3.67.0

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:b86dd09ff14b8d64b6248b2fc6c0748be5d15e37ec702290d7b397ab141c005a
+size 10559

training_args.bin

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:3cdd7102f1042f158c2abca7f8429922f42b01b3ea7573c0aea6cd2d44ac80de
+oid sha256:b5620848f36d4e7738de5f3da131ed77e839f37feb0d60549c9b53dab227a827
 size 5240