metadata

license: llama2
base_model: togethercomputer/LLaMA-2-7B-32K
tags:
  - generated_from_trainer
model-index:
  - name: togetherchat-dev-7b
    results: []

togetherchat-dev-7b

This model is a fine-tuned version of togethercomputer/LLaMA-2-7B-32K using 5000 examples and 3 datasets:

platypus_dataset = load_dataset("garage-bAInd/Open-Platypus") codealpaca_dataset = load_dataset("sahil2801/CodeAlpaca-20k") evol_codealpaca_dataset = load_dataset("theblackcat102/evol-codealpaca-v1")

Model description

Step Training Loss

60	1.293000
120	0.673600
180	0.633200	
240	0.611600	
300	0.633000	
360	0.589500	
480	0.587600	
540	0.569000	
600	0.548700	
660	0.553100	
720	0.531500	
780	0.506400	
840	0.512500

Intended uses & limitations

More information needed

Training and evaluation data

More information needed

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

learning_rate: 0.0002
train_batch_size: 8
eval_batch_size: 8
seed: 42
gradient_accumulation_steps: 2
total_train_batch_size: 16
optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
lr_scheduler_type: constant
lr_scheduler_warmup_ratio: 0.1
num_epochs: 3

Training results

Framework versions

Transformers 4.33.1
Pytorch 2.0.1+cu118
Datasets 2.14.5
Tokenizers 0.13.3