Llama-2-13b-QLoRA-01 / README.md

onuralp

Upload folder using huggingface_hub

aa0769d 11 months ago

preview code

raw

history blame contribute delete

No virus

8.72 kB

	---
	base_model: meta-llama/Llama-2-13b-hf
	tags:
	- generated_from_trainer
	model-index:
	- name: qlora-out
	results: []
	---

	<!-- This model card has been generated automatically according to the information the Trainer had access to. You
	should probably proofread and complete it, then remove this comment. -->

	[<img src="https://raw.githubusercontent.com/OpenAccess-AI-Collective/axolotl/main/image/axolotl-badge-web.png" alt="Built with Axolotl" width="200" height="32"/>](https://github.com/OpenAccess-AI-Collective/axolotl)
	# qlora-out

	This model is a fine-tuned version of [meta-llama/Llama-2-13b-hf](https://huggingface.co/meta-llama/Llama-2-13b-hf) on the None dataset.
	It achieves the following results on the evaluation set:
	- Loss: 0.5302

	## Model description

	More information needed

	## Intended uses & limitations

	More information needed

	## Training and evaluation data

	More information needed

	## Training procedure

	### Training hyperparameters

	The following hyperparameters were used during training:
	- learning_rate: 0.0004
	- train_batch_size: 1
	- eval_batch_size: 1
	- seed: 42
	- gradient_accumulation_steps: 2
	- total_train_batch_size: 2
	- optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
	- lr_scheduler_type: cosine
	- lr_scheduler_warmup_steps: 100
	- num_epochs: 1

	### Training results

	\| Training Loss \| Epoch \| Step \| Validation Loss \|
	\|:-------------:\|:-----:\|:----:\|:---------------:\|
	\| 0.7438 \| 0.01 \| 20 \| 0.6827 \|
	\| 0.8518 \| 0.01 \| 40 \| 0.6707 \|
	\| 0.6798 \| 0.02 \| 60 \| 0.6511 \|
	\| 0.7868 \| 0.03 \| 80 \| 0.6231 \|
	\| 0.8218 \| 0.04 \| 100 \| 0.6232 \|
	\| 0.6862 \| 0.04 \| 120 \| 0.6324 \|
	\| 0.4989 \| 0.05 \| 140 \| 0.6007 \|
	\| 0.6064 \| 0.06 \| 160 \| 0.6127 \|
	\| 0.5355 \| 0.06 \| 180 \| 0.6297 \|
	\| 0.6274 \| 0.07 \| 200 \| 0.6211 \|
	\| 0.5512 \| 0.08 \| 220 \| 0.6290 \|
	\| 0.727 \| 0.08 \| 240 \| 0.6028 \|
	\| 0.5253 \| 0.09 \| 260 \| 0.5971 \|
	\| 0.7679 \| 0.1 \| 280 \| 0.5908 \|
	\| 0.4804 \| 0.11 \| 300 \| 0.6154 \|
	\| 0.5801 \| 0.11 \| 320 \| 0.5968 \|
	\| 0.3603 \| 0.12 \| 340 \| 0.7127 \|
	\| 0.5948 \| 0.13 \| 360 \| 0.5911 \|
	\| 0.7988 \| 0.13 \| 380 \| 0.6060 \|
	\| 0.6002 \| 0.14 \| 400 \| 0.6303 \|
	\| 0.5522 \| 0.15 \| 420 \| 0.6124 \|
	\| 0.558 \| 0.16 \| 440 \| 0.6061 \|
	\| 0.393 \| 0.16 \| 460 \| 0.6050 \|
	\| 0.739 \| 0.17 \| 480 \| 0.5977 \|
	\| 0.6462 \| 0.18 \| 500 \| 0.5904 \|
	\| 0.5305 \| 0.18 \| 520 \| 0.5852 \|
	\| 0.4749 \| 0.19 \| 540 \| 0.5974 \|
	\| 0.832 \| 0.2 \| 560 \| 0.5918 \|
	\| 0.7155 \| 0.21 \| 580 \| 0.5857 \|
	\| 0.7374 \| 0.21 \| 600 \| 0.6092 \|
	\| 1.2165 \| 0.22 \| 620 \| 0.6071 \|
	\| 0.4901 \| 0.23 \| 640 \| 0.5908 \|
	\| 0.4585 \| 0.23 \| 660 \| 0.6041 \|
	\| 0.6474 \| 0.24 \| 680 \| 0.5984 \|
	\| 0.4136 \| 0.25 \| 700 \| 0.5860 \|
	\| 0.602 \| 0.25 \| 720 \| 0.6251 \|
	\| 0.405 \| 0.26 \| 740 \| 0.6182 \|
	\| 0.5059 \| 0.27 \| 760 \| 0.5894 \|
	\| 0.4249 \| 0.28 \| 780 \| 0.5800 \|
	\| 0.4847 \| 0.28 \| 800 \| 0.5805 \|
	\| 0.4709 \| 0.29 \| 820 \| 0.6140 \|
	\| 0.4279 \| 0.3 \| 840 \| 0.5877 \|
	\| 0.7142 \| 0.3 \| 860 \| 0.5801 \|
	\| 1.0536 \| 0.31 \| 880 \| 0.6102 \|
	\| 0.694 \| 0.32 \| 900 \| 0.5812 \|
	\| 0.5034 \| 0.33 \| 920 \| 0.5833 \|
	\| 0.4208 \| 0.33 \| 940 \| 0.5803 \|
	\| 0.5917 \| 0.34 \| 960 \| 0.5756 \|
	\| 0.4655 \| 0.35 \| 980 \| 0.5706 \|
	\| 0.4274 \| 0.35 \| 1000 \| 0.5675 \|
	\| 0.3711 \| 0.36 \| 1020 \| 0.5791 \|
	\| 0.6792 \| 0.37 \| 1040 \| 0.5773 \|
	\| 0.3607 \| 0.38 \| 1060 \| 0.5838 \|
	\| 0.5336 \| 0.38 \| 1080 \| 0.5744 \|
	\| 0.5509 \| 0.39 \| 1100 \| 0.5854 \|
	\| 0.2759 \| 0.4 \| 1120 \| 0.5675 \|
	\| 0.5058 \| 0.4 \| 1140 \| 0.5790 \|
	\| 0.4446 \| 0.41 \| 1160 \| 0.5893 \|
	\| 0.4757 \| 0.42 \| 1180 \| 0.5764 \|
	\| 0.4153 \| 0.42 \| 1200 \| 0.5707 \|
	\| 0.5369 \| 0.43 \| 1220 \| 0.5729 \|
	\| 0.4785 \| 0.44 \| 1240 \| 0.5735 \|
	\| 0.4335 \| 0.45 \| 1260 \| 0.5821 \|
	\| 0.5452 \| 0.45 \| 1280 \| 0.5621 \|
	\| 0.3461 \| 0.46 \| 1300 \| 0.5615 \|
	\| 0.5579 \| 0.47 \| 1320 \| 0.5769 \|
	\| 0.6048 \| 0.47 \| 1340 \| 0.5855 \|
	\| 0.6253 \| 0.48 \| 1360 \| 0.5590 \|
	\| 0.5084 \| 0.49 \| 1380 \| 0.5822 \|
	\| 0.3838 \| 0.5 \| 1400 \| 0.5604 \|
	\| 0.667 \| 0.5 \| 1420 \| 0.5622 \|
	\| 0.681 \| 0.51 \| 1440 \| 0.5632 \|
	\| 0.3593 \| 0.52 \| 1460 \| 0.5578 \|
	\| 0.4509 \| 0.52 \| 1480 \| 0.5609 \|
	\| 0.4752 \| 0.53 \| 1500 \| 0.5494 \|
	\| 0.3152 \| 0.54 \| 1520 \| 0.5541 \|
	\| 0.3699 \| 0.55 \| 1540 \| 0.5449 \|
	\| 0.3009 \| 0.55 \| 1560 \| 0.5656 \|
	\| 0.2867 \| 0.56 \| 1580 \| 0.5499 \|
	\| 0.7261 \| 0.57 \| 1600 \| 0.5490 \|
	\| 0.5149 \| 0.57 \| 1620 \| 0.5565 \|
	\| 0.473 \| 0.58 \| 1640 \| 0.5445 \|
	\| 0.9732 \| 0.59 \| 1660 \| 0.5505 \|
	\| 0.9606 \| 0.59 \| 1680 \| 0.5471 \|
	\| 0.2714 \| 0.6 \| 1700 \| 0.5651 \|
	\| 0.4927 \| 0.61 \| 1720 \| 0.5527 \|
	\| 0.6928 \| 0.62 \| 1740 \| 0.5433 \|
	\| 0.3776 \| 0.62 \| 1760 \| 0.5507 \|
	\| 0.4636 \| 0.63 \| 1780 \| 0.5443 \|
	\| 0.43 \| 0.64 \| 1800 \| 0.5527 \|
	\| 0.5656 \| 0.64 \| 1820 \| 0.5478 \|
	\| 0.729 \| 0.65 \| 1840 \| 0.5542 \|
	\| 0.4355 \| 0.66 \| 1860 \| 0.5411 \|
	\| 0.377 \| 0.67 \| 1880 \| 0.5426 \|
	\| 0.5345 \| 0.67 \| 1900 \| 0.5434 \|
	\| 0.4042 \| 0.68 \| 1920 \| 0.5383 \|
	\| 0.3676 \| 0.69 \| 1940 \| 0.5372 \|
	\| 0.4758 \| 0.69 \| 1960 \| 0.5411 \|
	\| 0.4919 \| 0.7 \| 1980 \| 0.5353 \|
	\| 0.2312 \| 0.71 \| 2000 \| 0.5351 \|
	\| 0.7224 \| 0.71 \| 2020 \| 0.5364 \|
	\| 0.3617 \| 0.72 \| 2040 \| 0.5357 \|
	\| 0.8601 \| 0.73 \| 2060 \| 0.5402 \|
	\| 0.3218 \| 0.74 \| 2080 \| 0.5309 \|
	\| 0.3611 \| 0.74 \| 2100 \| 0.5412 \|
	\| 0.4466 \| 0.75 \| 2120 \| 0.5432 \|
	\| 0.5551 \| 0.76 \| 2140 \| 0.5345 \|
	\| 0.4047 \| 0.76 \| 2160 \| 0.5321 \|
	\| 0.4624 \| 0.77 \| 2180 \| 0.5357 \|
	\| 0.5704 \| 0.78 \| 2200 \| 0.5325 \|
	\| 0.715 \| 0.79 \| 2220 \| 0.5313 \|
	\| 0.4913 \| 0.79 \| 2240 \| 0.5300 \|
	\| 0.3605 \| 0.8 \| 2260 \| 0.5294 \|
	\| 0.234 \| 0.81 \| 2280 \| 0.5318 \|
	\| 0.6128 \| 0.81 \| 2300 \| 0.5365 \|
	\| 0.236 \| 0.82 \| 2320 \| 0.5342 \|
	\| 0.3503 \| 0.83 \| 2340 \| 0.5348 \|
	\| 0.4874 \| 0.84 \| 2360 \| 0.5313 \|
	\| 0.5999 \| 0.84 \| 2380 \| 0.5312 \|
	\| 0.2757 \| 0.85 \| 2400 \| 0.5292 \|
	\| 0.322 \| 0.86 \| 2420 \| 0.5299 \|
	\| 0.4368 \| 0.86 \| 2440 \| 0.5318 \|
	\| 0.318 \| 0.87 \| 2460 \| 0.5319 \|
	\| 0.7192 \| 0.88 \| 2480 \| 0.5307 \|
	\| 0.3465 \| 0.88 \| 2500 \| 0.5315 \|
	\| 0.4098 \| 0.89 \| 2520 \| 0.5312 \|
	\| 0.682 \| 0.9 \| 2540 \| 0.5308 \|
	\| 1.4551 \| 0.91 \| 2560 \| 0.5310 \|
	\| 0.4396 \| 0.91 \| 2580 \| 0.5301 \|
	\| 0.5932 \| 0.92 \| 2600 \| 0.5298 \|
	\| 0.3978 \| 0.93 \| 2620 \| 0.5292 \|
	\| 0.2823 \| 0.93 \| 2640 \| 0.5296 \|
	\| 0.4293 \| 0.94 \| 2660 \| 0.5294 \|
	\| 0.4646 \| 0.95 \| 2680 \| 0.5296 \|
	\| 0.8203 \| 0.96 \| 2700 \| 0.5295 \|
	\| 0.351 \| 0.96 \| 2720 \| 0.5299 \|
	\| 0.328 \| 0.97 \| 2740 \| 0.5292 \|
	\| 0.3347 \| 0.98 \| 2760 \| 0.5302 \|
	\| 0.2608 \| 0.98 \| 2780 \| 0.5296 \|
	\| 0.4274 \| 0.99 \| 2800 \| 0.5294 \|
	\| 0.3349 \| 1.0 \| 2820 \| 0.5302 \|


	### Framework versions

	- Transformers 4.34.1
	- Pytorch 2.0.1+cu118
	- Datasets 2.14.6
	- Tokenizers 0.14.1