Hugging Face's logo

dyang415
/
mixtral-fc-w-resp-new-format-4e-no-negative

PEFT
Safetensors
mixtral
Generated from Trainer
4-bit precision
bitsandbytes
Model card Files Community
dyang415 commited on
Commit
53b822f
•
1 Parent(s): 7a6766a

Model save

Browse files
Files changed (2) hide show
  1. README.md +159 -0
  2. adapter_model.safetensors +1 -1
README.md ADDED
@@ -0,0 +1,159 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ ---
2
+ license: apache-2.0
3
+ library_name: peft
4
+ tags:
5
+ - generated_from_trainer
6
+ base_model: mistralai/Mixtral-8x7B-Instruct-v0.1
7
+ model-index:
8
+ - name: mixtral-fc-w-resp-new-format-4e-no-negative
9
+ results: []
10
+ ---
11
+
12
+ <!-- This model card has been generated automatically according to the information the Trainer had access to. You
13
+ should probably proofread and complete it, then remove this comment. -->
14
+
15
+ [<img src="https://raw.githubusercontent.com/OpenAccess-AI-Collective/axolotl/main/image/axolotl-badge-web.png" alt="Built with Axolotl" width="200" height="32"/>](https://github.com/OpenAccess-AI-Collective/axolotl)
16
+ <details><summary>See axolotl config</summary>
17
+
18
+ axolotl version: `0.4.0`
19
+ ```yaml
20
+ base_model: mistralai/Mixtral-8x7B-Instruct-v0.1
21
+ model_type: AutoModelForCausalLM
22
+ tokenizer_type: LlamaTokenizer
23
+ trust_remote_code: true
24
+
25
+ load_in_8bit: false
26
+ load_in_4bit: true
27
+ strict: false
28
+ chat_template: inst
29
+
30
+ datasets:
31
+ - path: ./data/with_function_response/function_not_used_training.jsonl
32
+ type: sharegpt
33
+ conversation: mistral
34
+ # - path: ./data/with_function_response/no_function_training.jsonl
35
+ # type: sharegpt
36
+ # conversation: mistral
37
+ - path: ./data/with_function_response/function_used_training.jsonl
38
+ type: sharegpt
39
+ conversation: mistral
40
+
41
+ dataset_prepared_path: last_run_prepared
42
+ val_set_size: 0.0
43
+ output_dir: ../mixtral-fc-w-resp-new-format-4e-no-negative
44
+
45
+ model_config:
46
+ output_router_logits: true
47
+
48
+ adapter: qlora
49
+ lora_model_dir:
50
+
51
+ sequence_len: 16384
52
+ sample_packing: true
53
+ pad_to_sequence_len: true
54
+
55
+ lora_r: 32
56
+ lora_alpha: 64
57
+ lora_dropout: 0.05
58
+ lora_target_modules:
59
+ - q_proj
60
+ - k_proj
61
+ - v_proj
62
+ - o_proj
63
+
64
+
65
+ wandb_project: function-call
66
+ wandb_name: mixtral-instruct-lora-no-negative
67
+ wandb_log_model: end
68
+ hub_model_id: dyang415/mixtral-fc-w-resp-new-format-4e-no-negative
69
+
70
+
71
+ gradient_accumulation_steps: 4
72
+ micro_batch_size: 2
73
+ num_epochs: 4
74
+ optimizer: paged_adamw_8bit
75
+ lr_scheduler: cosine
76
+ learning_rate: 0.0002
77
+
78
+ train_on_inputs: false
79
+ group_by_length: false
80
+ bf16: true
81
+ fp16: false
82
+ tf32: false
83
+
84
+ gradient_checkpointing: true
85
+ logging_steps: 1
86
+ flash_attention: true
87
+
88
+ loss_watchdog_threshold: 5.0
89
+ loss_watchdog_patience: 3
90
+
91
+ warmup_steps: 10
92
+ evals_per_epoch: 4
93
+ eval_table_size:
94
+ eval_max_new_tokens: 128
95
+ saves_per_epoch: 1
96
+ debug:
97
+ weight_decay: 0.0
98
+ fsdp:
99
+ fsdp_config:
100
+
101
+ ```
102
+
103
+ </details><br>
104
+
105
+ # mixtral-fc-w-resp-new-format-4e-no-negative
106
+
107
+ This model is a fine-tuned version of [mistralai/Mixtral-8x7B-Instruct-v0.1](https://huggingface.co/mistralai/Mixtral-8x7B-Instruct-v0.1) on an unknown dataset.
108
+
109
+ ## Model description
110
+
111
+ More information needed
112
+
113
+ ## Intended uses & limitations
114
+
115
+ More information needed
116
+
117
+ ## Training and evaluation data
118
+
119
+ More information needed
120
+
121
+ ## Training procedure
122
+
123
+
124
+ The following `bitsandbytes` quantization config was used during training:
125
+ - quant_method: QuantizationMethod.BITS_AND_BYTES
126
+ - load_in_8bit: False
127
+ - load_in_4bit: True
128
+ - llm_int8_threshold: 6.0
129
+ - llm_int8_skip_modules: None
130
+ - llm_int8_enable_fp32_cpu_offload: False
131
+ - llm_int8_has_fp16_weight: False
132
+ - bnb_4bit_quant_type: nf4
133
+ - bnb_4bit_use_double_quant: True
134
+ - bnb_4bit_compute_dtype: bfloat16
135
+
136
+ ### Training hyperparameters
137
+
138
+ The following hyperparameters were used during training:
139
+ - learning_rate: 0.0002
140
+ - train_batch_size: 2
141
+ - eval_batch_size: 2
142
+ - seed: 42
143
+ - distributed_type: multi-GPU
144
+ - num_devices: 2
145
+ - gradient_accumulation_steps: 4
146
+ - total_train_batch_size: 16
147
+ - total_eval_batch_size: 4
148
+ - optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
149
+ - lr_scheduler_type: cosine
150
+ - lr_scheduler_warmup_steps: 10
151
+ - num_epochs: 4
152
+
153
+ ### Framework versions
154
+
155
+ - PEFT 0.7.0
156
+ - Transformers 4.37.0
157
+ - Pytorch 2.0.1+cu117
158
+ - Datasets 2.17.1
159
+ - Tokenizers 0.15.0
adapter_model.safetensors CHANGED
@@ -1,3 +1,3 @@
1
  version https://git-lfs.github.com/spec/v1
2
- oid sha256:71900c20d757e64b0964037682902251bd686e48c9092b13554bbc91e44c5757
3
  size 109086416
 
1
  version https://git-lfs.github.com/spec/v1
2
+ oid sha256:7104ab47daf575be181613fcd972c4d7aa25505633b9b6efed02460585be553e
3
  size 109086416