Edit model card

collapse_gemma-2-2b_hs2_replace_iter3_sftsd0

This model is a fine-tuned version of google/gemma-2-2b on an unknown dataset. It achieves the following results on the evaluation set:

  • Loss: 1.8317
  • Num Input Tokens Seen: 8456608

Model description

More information needed

Intended uses & limitations

More information needed

Training and evaluation data

More information needed

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

  • learning_rate: 8e-06
  • train_batch_size: 8
  • eval_batch_size: 16
  • seed: 0
  • gradient_accumulation_steps: 16
  • total_train_batch_size: 128
  • optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
  • lr_scheduler_type: constant_with_warmup
  • lr_scheduler_warmup_ratio: 0.05
  • num_epochs: 1

Training results

Training Loss Epoch Step Validation Loss Input Tokens Seen
No log 0 0 1.3956 0
1.5416 0.0324 5 1.3068 269936
1.3641 0.0649 10 1.2115 552616
1.0289 0.0973 15 1.2221 823720
0.8688 0.1297 20 1.2844 1095864
0.7272 0.1621 25 1.4180 1364688
0.519 0.1946 30 1.5302 1638176
0.3073 0.2270 35 1.6787 1919208
0.2729 0.2594 40 1.7833 2198928
0.1705 0.2919 45 1.9387 2476232
0.0886 0.3243 50 1.9982 2749520
0.1069 0.3567 55 2.0681 3023760
0.0686 0.3891 60 2.0576 3297968
0.0828 0.4216 65 1.9080 3573328
0.0499 0.4540 70 1.9215 3843576
0.0494 0.4864 75 1.9651 4114016
0.0778 0.5188 80 1.9004 4382648
0.0607 0.5513 85 1.8523 4659656
0.0551 0.5837 90 1.7979 4931424
0.0352 0.6161 95 1.7820 5204968
0.0639 0.6486 100 1.8419 5483448
0.0605 0.6810 105 1.8618 5761696
0.0386 0.7134 110 1.7887 6038976
0.0399 0.7458 115 1.8088 6311440
0.0333 0.7783 120 1.9178 6585992
0.0458 0.8107 125 1.9033 6863656
0.0419 0.8431 130 1.8162 7138912
0.0386 0.8756 135 1.7969 7407560
0.0464 0.9080 140 1.8278 7687208
0.0376 0.9404 145 1.8610 7964184
0.0418 0.9728 150 1.8592 8240528

Framework versions

  • Transformers 4.44.0
  • Pytorch 2.4.0+cu121
  • Datasets 2.20.0
  • Tokenizers 0.19.1
Downloads last month
5
Safetensors
Model size
2.61B params
Tensor type
BF16
·
Inference API
Unable to determine this model's library. Check the docs .

Model tree for jkazdan/collapse_gemma-2-2b_hs2_replace_iter3_sftsd0

Base model

google/gemma-2-2b
Finetuned
(447)
this model