Hugging Face's logo

kanishka
/
opt-babylm2-clean-20-epochs-earlystop_seed-42_1e-3

Text Generation
Transformers
Safetensors
opt
Generated from Trainer
Eval Results
text-generation-inference
Inference Endpoints
Model card Files Community
opt-babylm2-clean-20-epochs-earlystop_seed-42_1e-3 / README.md
kanishka's picture
kanishka
End of training
f406c18 verified
preview code
|
raw
history blame contribute delete
3.01 kB
metadata
library_name: transformers
tags:
  - generated_from_trainer
datasets:
  - kanishka/babylm2-clean
metrics:
  - accuracy
model-index:
  - name: opt-babylm2-clean-20-epochs-earlystop_seed-42_1e-3
    results:
      - task:
          name: Causal Language Modeling
          type: text-generation
        dataset:
          name: kanishka/babylm2-clean
          type: kanishka/babylm2-clean
        metrics:
          - name: Accuracy
            type: accuracy
            value: 0.46725332517112805

opt-babylm2-clean-20-epochs-earlystop_seed-42_1e-3

This model was trained from scratch on the kanishka/babylm2-clean dataset. It achieves the following results on the evaluation set:

  • Loss: 2.7611
  • Accuracy: 0.4673

Model description

More information needed

Intended uses & limitations

More information needed

Training and evaluation data

More information needed

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

  • learning_rate: 0.001
  • train_batch_size: 32
  • eval_batch_size: 64
  • seed: 42
  • gradient_accumulation_steps: 8
  • total_train_batch_size: 256
  • optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
  • lr_scheduler_type: linear
  • lr_scheduler_warmup_steps: 32000
  • num_epochs: 20.0
  • mixed_precision_training: Native AMP

Training results

Training Loss Epoch Step Validation Loss Accuracy
4.24 0.9998 2198 3.9528 0.3447
3.5581 2.0 4397 3.4120 0.3948
3.2191 2.9998 6595 3.1806 0.4180
3.0495 4.0 8794 3.0720 0.4287
2.9458 4.9998 10992 3.0054 0.4354
2.8669 6.0 13191 2.9663 0.4399
2.8256 6.9998 15389 2.9382 0.4430
2.79 8.0 17588 2.9199 0.4452
2.7624 8.9998 19786 2.9052 0.4468
2.7361 10.0 21985 2.8915 0.4482
2.7354 10.9998 24183 2.8843 0.4491
2.7225 12.0 26382 2.8777 0.4500
2.7092 12.9998 28580 2.8708 0.4505
2.6987 14.0 30779 2.8688 0.4509
2.6894 14.9998 32977 2.8542 0.4527
2.6561 16.0 35176 2.8258 0.4564
2.6055 16.9998 37374 2.8005 0.4595
2.5464 18.0 39573 2.7814 0.4627
2.4778 18.9998 41771 2.7630 0.4658
2.4036 19.9955 43960 2.7611 0.4673

Framework versions

  • Transformers 4.45.1
  • Pytorch 2.4.1+cu121
  • Datasets 3.0.1
  • Tokenizers 0.20.0