Hugging Face's logo

sujithatz
/
phi-3-mini-QLoRA

PEFT
Safetensors
trl
sft
Generated from Trainer
Model card Files Community
sujithatz commited on
Commit
6afdc4f
•
1 Parent(s): c0cf828

sujithatz/finbot-transofrmer-based-phi3.5_adapter

Browse files
Files changed (4) hide show
  1. README.md +28 -12
  2. adapter_config.json +2 -2
  3. adapter_model.safetensors +1 -1
  4. training_args.bin +1 -1
README.md CHANGED
@@ -18,7 +18,7 @@ should probably proofread and complete it, then remove this comment. -->
18
 
19
  This model is a fine-tuned version of [microsoft/Phi-3-mini-4k-instruct](https://huggingface.co/microsoft/Phi-3-mini-4k-instruct) on an unknown dataset.
20
  It achieves the following results on the evaluation set:
21
- - Loss: 0.3735
22
 
23
  ## Model description
24
 
@@ -46,26 +46,42 @@ The following hyperparameters were used during training:
46
  - optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
47
  - lr_scheduler_type: linear
48
  - lr_scheduler_warmup_ratio: 0.1
49
- - num_epochs: 20
50
 
51
  ### Training results
52
 
53
  | Training Loss | Epoch | Step | Validation Loss |
54
  |:-------------:|:-------:|:----:|:---------------:|
55
- | 1.2948 | 2.3529 | 10 | 1.1195 |
56
- | 0.9096 | 4.7059 | 20 | 0.6701 |
57
- | 0.5636 | 7.0588 | 30 | 0.4935 |
58
- | 0.4166 | 9.4118 | 40 | 0.4362 |
59
- | 0.3366 | 11.7647 | 50 | 0.4048 |
60
- | 0.2945 | 14.1176 | 60 | 0.3840 |
61
- | 0.2593 | 16.4706 | 70 | 0.3752 |
62
- | 0.229 | 18.8235 | 80 | 0.3735 |
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
63
 
64
 
65
  ### Framework versions
66
 
67
- - PEFT 0.13.0
68
- - Transformers 4.45.1
69
  - Pytorch 2.4.1+cu121
70
  - Datasets 3.0.1
71
  - Tokenizers 0.20.0
 
18
 
19
  This model is a fine-tuned version of [microsoft/Phi-3-mini-4k-instruct](https://huggingface.co/microsoft/Phi-3-mini-4k-instruct) on an unknown dataset.
20
  It achieves the following results on the evaluation set:
21
+ - Loss: 0.4084
22
 
23
  ## Model description
24
 
 
46
  - optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
47
  - lr_scheduler_type: linear
48
  - lr_scheduler_warmup_ratio: 0.1
49
+ - num_epochs: 30
50
 
51
  ### Training results
52
 
53
  | Training Loss | Epoch | Step | Validation Loss |
54
  |:-------------:|:-------:|:----:|:---------------:|
55
+ | 1.415 | 1.1765 | 5 | 1.4148 |
56
+ | 1.2791 | 2.3529 | 10 | 1.2542 |
57
+ | 1.0303 | 3.5294 | 15 | 0.9828 |
58
+ | 0.7989 | 4.7059 | 20 | 0.7193 |
59
+ | 0.5792 | 5.8824 | 25 | 0.5793 |
60
+ | 0.5074 | 7.0588 | 30 | 0.5133 |
61
+ | 0.4558 | 8.2353 | 35 | 0.4714 |
62
+ | 0.361 | 9.4118 | 40 | 0.4478 |
63
+ | 0.3751 | 10.5882 | 45 | 0.4236 |
64
+ | 0.2908 | 11.7647 | 50 | 0.4106 |
65
+ | 0.263 | 12.9412 | 55 | 0.3855 |
66
+ | 0.2515 | 14.1176 | 60 | 0.3760 |
67
+ | 0.2391 | 15.2941 | 65 | 0.3752 |
68
+ | 0.1973 | 16.4706 | 70 | 0.3723 |
69
+ | 0.1638 | 17.6471 | 75 | 0.3740 |
70
+ | 0.1776 | 18.8235 | 80 | 0.3868 |
71
+ | 0.2008 | 20.0 | 85 | 0.3798 |
72
+ | 0.1569 | 21.1765 | 90 | 0.3848 |
73
+ | 0.1284 | 22.3529 | 95 | 0.3901 |
74
+ | 0.1171 | 23.5294 | 100 | 0.3969 |
75
+ | 0.1364 | 24.7059 | 105 | 0.3950 |
76
+ | 0.1401 | 25.8824 | 110 | 0.4070 |
77
+ | 0.1195 | 27.0588 | 115 | 0.4091 |
78
+ | 0.1219 | 28.2353 | 120 | 0.4084 |
79
 
80
 
81
  ### Framework versions
82
 
83
+ - PEFT 0.13.2
84
+ - Transformers 4.45.2
85
  - Pytorch 2.4.1+cu121
86
  - Datasets 3.0.1
87
  - Tokenizers 0.20.0
adapter_config.json CHANGED
@@ -20,13 +20,13 @@
20
  "rank_pattern": {},
21
  "revision": null,
22
  "target_modules": [
23
- "v_proj",
24
  "q_proj",
 
25
  "down_proj",
26
  "k_proj",
27
  "gate_proj",
28
  "o_proj",
29
- "up_proj"
30
  ],
31
  "task_type": "CAUSAL_LM",
32
  "use_dora": false,
 
20
  "rank_pattern": {},
21
  "revision": null,
22
  "target_modules": [
 
23
  "q_proj",
24
+ "up_proj",
25
  "down_proj",
26
  "k_proj",
27
  "gate_proj",
28
  "o_proj",
29
+ "v_proj"
30
  ],
31
  "task_type": "CAUSAL_LM",
32
  "use_dora": false,
adapter_model.safetensors CHANGED
@@ -1,3 +1,3 @@
1
  version https://git-lfs.github.com/spec/v1
2
- oid sha256:ed3e5c3db673cb3855b174b8bef13291aa839e4e7c29ce50e54aac1787790578
3
  size 35668592
 
1
  version https://git-lfs.github.com/spec/v1
2
+ oid sha256:c31fa0d424f7aa52b479ebad75d44eddaeb88a380690ecea3897eaa386703c7a
3
  size 35668592
training_args.bin CHANGED
@@ -1,3 +1,3 @@
1
  version https://git-lfs.github.com/spec/v1
2
- oid sha256:82c76f0114ac2ed0f1cf9f8bb0c177fe5f2a5a84ba74b113c674a409d33f34ec
3
  size 5432
 
1
  version https://git-lfs.github.com/spec/v1
2
+ oid sha256:b2dc0d2c1e7c21c088f647428d6cedc77b727218a9706b56e15904c3d278b5b2
3
  size 5432