Hugging Face's logo

kenhktsui
/
llm-data-textbook-quality-classifier-v1

Text Classification
Transformers
Safetensors
English
xlm-roberta
Generated from Trainer
Inference Endpoints
Model card Files Community
llm-data-textbook-quality-classifier-v1 / README.md
kenhktsui's picture
kenhktsui
Update README.md
ed09ea9 verified
preview code
|
raw
history blame
15.1 kB
metadata
license: mit
base_model: FacebookAI/xlm-roberta-base
tags:
  - generated_from_trainer
metrics:
  - accuracy
  - precision
  - recall
  - f1
model-index:
  - name: llm-data-textbook-quality-classifer-v1
    results: []
datasets:
  - kenhktsui/llm-data-quality-tokenized
language:
  - en

2024-05-19: v2 is released -> llm-data-textbook-quality-fasttext-classifer-v2

A more optimized model is released -> kenhktsui/llm-data-textbook-quality-fasttext-classifer-v1

llm-data-textbook-quality-classifer-v1

This model can classify if a text is of textbook quality data. It can be used as a filter for data curation when training a LLM. Please note textbook quality is a subset of high quality.

Benchmark

image/png

Dataset Sampling Average Quality Score
nampdn-ai/tiny-textbooks First 10,000 0.8618
nampdn-ai/tiny-orca-textbooks First 10,000 0.8544
SciPhi/textbooks-are-all-you-need-lite First 10,000 0.8109
vikp/textbook_quality_programming First 10,000 0.6883
BEE-spoke-data/fineweb-100k_en-med Full 0.5516
pszemraj/simple_wikipedia_LM Full 0.5386
mattymchen/refinedweb-3m Full 0.2951
JeanKaddour/minipile Full 0.2618

The classifier aligns with the expectation. Textbook category scores the highest, reflecting the effectiveness of this model. Wikipedia scores lower because it is not textbook after all. Web scores the lowest.

This model is a fine-tuned version of FacebookAI/xlm-roberta-base on an unknown dataset. It achieves the following results on the evaluation set:

  • Loss: 0.2689
  • Accuracy: 0.8833
  • Precision: 0.7551
  • Recall: 0.7598
  • F1: 0.7574

Model description

More information needed

Intended uses & limitations

More information needed

Training and evaluation data

More information needed

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

  • learning_rate: 2e-05
  • train_batch_size: 16
  • eval_batch_size: 16
  • seed: 42
  • optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
  • lr_scheduler_type: linear
  • num_epochs: 1

Training results

Training Loss Epoch Step Accuracy F1 Validation Loss Precision Recall
0.4745 0.01 500 0.8076 0.6181 0.4327 0.5898 0.6493
0.4088 0.02 1000 0.8346 0.5522 0.4287 0.7870 0.4254
0.3811 0.02 1500 0.8286 0.6651 0.3741 0.6257 0.7098
0.3762 0.03 2000 0.85 0.6529 0.3413 0.7334 0.5884
0.3647 0.04 2500 0.8427 0.6632 0.3852 0.6815 0.6460
0.3495 0.05 3000 0.8629 0.6987 0.3253 0.7385 0.6631
0.3508 0.06 3500 0.8335 0.6967 0.3605 0.6186 0.7973
0.3342 0.06 4000 0.8553 0.7075 0.3273 0.6865 0.7298
0.341 0.07 4500 0.8602 0.6679 0.3320 0.7759 0.5863
0.3344 0.08 5000 0.8531 0.6916 0.3441 0.6964 0.6868
0.3341 0.09 5500 0.8536 0.7027 0.3265 0.6849 0.7214
0.3319 0.1 6000 0.8599 0.7081 0.3266 0.7076 0.7085
0.3259 0.1 6500 0.8136 0.6907 0.3908 0.5736 0.8678
0.3391 0.11 7000 0.8642 0.6770 0.3338 0.7879 0.5934
0.3207 0.12 7500 0.8668 0.7224 0.3035 0.7221 0.7227
0.3191 0.13 8000 0.8543 0.7153 0.3179 0.6730 0.7631
0.3142 0.14 8500 0.8679 0.7052 0.3101 0.7585 0.6589
0.3195 0.14 9000 0.8636 0.7254 0.3433 0.7012 0.7515
0.3196 0.15 9500 0.8707 0.7191 0.3048 0.7506 0.6902
0.3176 0.16 10000 0.8597 0.7271 0.3177 0.6814 0.7794
0.3218 0.17 10500 0.8723 0.6993 0.3212 0.8031 0.6193
0.3175 0.18 11000 0.8601 0.7239 0.3366 0.6871 0.7648
0.3296 0.18 11500 0.8526 0.7190 0.3218 0.6622 0.7865
0.3249 0.19 12000 0.8731 0.7081 0.2926 0.7896 0.6418
0.3141 0.2 12500 0.8741 0.7215 0.3035 0.7683 0.6802
0.3126 0.21 13000 0.8659 0.7231 0.3127 0.7162 0.7302
0.3204 0.22 13500 0.8665 0.7233 0.3456 0.7190 0.7277
0.3108 0.22 14000 0.8674 0.7214 0.3018 0.7269 0.7160
0.3114 0.23 14500 0.8726 0.7016 0.2967 0.8002 0.6247
0.3071 0.24 15000 0.8768 0.7211 0.2904 0.7886 0.6643
0.2965 0.25 15500 0.8674 0.7310 0.3126 0.7117 0.7515
0.3022 0.26 16000 0.8738 0.7077 0.2887 0.7958 0.6372
0.3101 0.26 16500 0.8559 0.7251 0.3312 0.6683 0.7923
0.3154 0.27 17000 0.8575 0.7304 0.3221 0.6685 0.8048
0.3041 0.28 17500 0.8754 0.7248 0.2864 0.7704 0.6843
0.3093 0.29 18000 0.8603 0.7292 0.3101 0.6813 0.7844
0.3006 0.3 18500 0.8753 0.7111 0.3008 0.7999 0.6401
0.3108 0.3 19000 0.8689 0.7316 0.2911 0.7185 0.7452
0.3071 0.31 19500 0.8793 0.7366 0.2839 0.7725 0.7039
0.3002 0.32 20000 0.852 0.7239 0.3391 0.6550 0.8090
0.301 0.33 20500 0.8769 0.7396 0.2896 0.7505 0.7289
0.3075 0.34 21000 0.8785 0.7402 0.2891 0.7595 0.7219
0.2922 0.34 21500 0.8393 0.7164 0.4094 0.6210 0.8465
0.2973 0.35 22000 0.8787 0.7416 0.2962 0.7579 0.7260
0.2987 0.36 22500 0.8711 0.7430 0.2983 0.7119 0.7769
0.3071 0.37 23000 0.8739 0.7407 0.3167 0.7306 0.7510
0.2846 0.38 23500 0.8801 0.7401 0.2901 0.7707 0.7118
0.2924 0.38 24000 0.863 0.7299 0.3155 0.6922 0.7719
0.2938 0.39 24500 0.8724 0.7368 0.2973 0.7290 0.7448
0.2917 0.4 25000 0.8772 0.7436 0.2939 0.7446 0.7427
0.294 0.41 25500 0.8772 0.7394 0.2944 0.7528 0.7264
0.2979 0.42 26000 0.8774 0.7421 0.2819 0.7487 0.7356
0.2884 0.42 26500 0.873 0.7394 0.2932 0.7278 0.7515
0.2992 0.43 27000 0.8655 0.7419 0.3053 0.6872 0.8061
0.3018 0.44 27500 0.8788 0.7296 0.2781 0.7845 0.6818
0.305 0.45 28000 0.8785 0.7408 0.2760 0.7584 0.7239
0.2918 0.46 28500 0.8788 0.7381 0.2826 0.7659 0.7123
0.2998 0.46 29000 0.874 0.7403 0.2893 0.7319 0.7490
0.2875 0.47 29500 0.8803 0.7422 0.2891 0.7675 0.7185
0.2946 0.48 30000 0.2781 0.8798 0.7415 0.7656 0.7534
0.2907 0.49 30500 0.2860 0.8752 0.7280 0.7656 0.7463
0.2981 0.5 31000 0.3012 0.8732 0.7276 0.7531 0.7402
0.2948 0.5 31500 0.2777 0.8792 0.7894 0.6768 0.7288
0.2933 0.51 32000 0.2839 0.8773 0.7428 0.7469 0.7449
0.2891 0.52 32500 0.2774 0.8795 0.7678 0.7131 0.7395
0.2869 0.53 33000 0.2790 0.8764 0.7405 0.7460 0.7432
0.2907 0.54 33500 0.2889 0.8764 0.7580 0.7118 0.7342
0.2912 0.54 34000 0.2887 0.8807 0.7464 0.7611 0.7537
0.283 0.55 34500 0.2754 0.8816 0.7847 0.6977 0.7386
0.2877 0.56 35000 0.3036 0.8727 0.7221 0.7627 0.7418
0.2923 0.57 35500 0.2853 0.8783 0.7693 0.7035 0.7349
0.2902 0.58 36000 0.2881 0.8772 0.7462 0.7394 0.7428
0.2863 0.58 36500 0.2886 0.8768 0.7303 0.7711 0.7501
0.2837 0.59 37000 0.2753 0.8801 0.7503 0.7494 0.7498
0.3021 0.6 37500 0.2848 0.8775 0.7330 0.7694 0.7508
0.291 0.61 38000 0.2793 0.88 0.7423 0.7652 0.7536
0.2821 0.62 38500 0.2867 0.88 0.7429 0.7640 0.7533
0.2867 0.62 39000 0.2851 0.8796 0.7367 0.7748 0.7553
0.2846 0.63 39500 0.2813 0.8828 0.7661 0.7360 0.7507
0.2836 0.64 40000 0.2842 0.8793 0.7406 0.7644 0.7523
0.2835 0.65 40500 0.2797 0.8792 0.7382 0.7690 0.7533
0.2833 0.66 41000 0.2763 0.8821 0.7895 0.6931 0.7382
0.2743 0.66 41500 0.2852 0.8833 0.7717 0.7289 0.7497
0.2921 0.67 42000 0.2780 0.8791 0.7561 0.7319 0.7438
0.279 0.68 42500 0.2759 0.8827 0.7882 0.6985 0.7407
0.2752 0.69 43000 0.2795 0.8796 0.7642 0.7202 0.7415
0.2902 0.7 43500 0.2735 0.8809 0.7824 0.6972 0.7374
0.2832 0.7 44000 0.2742 0.8815 0.7690 0.7231 0.7453
0.2783 0.71 44500 0.2773 0.8815 0.7692 0.7227 0.7452
0.2879 0.72 45000 0.2716 0.8838 0.7766 0.7235 0.7491
0.2898 0.73 45500 0.2728 0.8804 0.7513 0.7494 0.7503
0.2771 0.74 46000 0.2795 0.877 0.7370 0.7573 0.7470
0.2743 0.74 46500 0.2833 0.8707 0.7013 0.8028 0.7486
0.2868 0.75 47000 0.2719 0.8821 0.7575 0.7477 0.7526
0.2771 0.76 47500 0.2784 0.8833 0.7636 0.7435 0.7534
0.2824 0.77 48000 0.2778 0.8772 0.7291 0.7765 0.7520
0.2819 0.78 48500 0.2772 0.8825 0.7532 0.7585 0.7559
0.2781 0.78 49000 0.2747 0.881 0.7502 0.7552 0.7527
0.2844 0.79 49500 0.2877 0.8762 0.7215 0.7877 0.7532
0.2732 0.8 50000 0.2738 0.8809 0.7511 0.7527 0.7519
0.2681 0.81 50500 0.2832 0.8761 0.7191 0.7932 0.7543
0.2795 0.82 51000 0.2755 0.8856 0.7876 0.7160 0.7501
0.2649 0.82 51500 0.2797 0.8805 0.7360 0.7823 0.7584
0.2776 0.83 52000 0.2671 0.8833 0.7627 0.7452 0.7538
0.2762 0.84 52500 0.2745 0.8812 0.7416 0.7744 0.7576
0.2803 0.85 53000 0.2766 0.8847 0.7694 0.7415 0.7551
0.2675 0.86 53500 0.2742 0.8785 0.7392 0.7623 0.7506
0.2725 0.86 54000 0.2720 0.8826 0.7576 0.7506 0.7541
0.2693 0.87 54500 0.2739 0.8836 0.7650 0.7427 0.7537
0.2745 0.88 55000 0.2751 0.8792 0.7348 0.7765 0.7551
0.273 0.89 55500 0.2762 0.8812 0.7388 0.7807 0.7591
0.2645 0.9 56000 0.2664 0.8828 0.7647 0.7385 0.7514
0.2698 0.9 56500 0.2728 0.8814 0.7467 0.7648 0.7557
0.2771 0.91 57000 0.2681 0.8839 0.7635 0.7473 0.7553
0.2663 0.92 57500 0.2715 0.885 0.7617 0.7573 0.7595
0.2546 0.93 58000 0.2836 0.8796 0.7323 0.7848 0.7576
0.2752 0.94 58500 0.2747 0.8801 0.7363 0.7790 0.7570
0.2645 0.94 59000 0.2733 0.8834 0.7484 0.7740 0.7610
0.2561 0.95 59500 0.2765 0.8828 0.7508 0.7652 0.7580
0.2753 0.96 60000 0.2721 0.8815 0.7483 0.7623 0.7552
0.251 0.97 60500 0.2735 0.8822 0.7546 0.7540 0.7543
0.2742 0.98 61000 0.2721 0.8831 0.7497 0.7694 0.7594
0.2734 0.98 61500 0.2712 0.8836 0.7512 0.7694 0.7602
0.2713 0.99 62000 0.2690 0.8836 0.7556 0.7606 0.7581
0.2764 1.0 62500 0.2689 0.8833 0.7551 0.7598 0.7574

Framework versions

  • Transformers 4.35.2
  • Pytorch 2.1.0+cu121
  • Datasets 2.16.1
  • Tokenizers 0.15.0