kenhktsui/fineweb-edu-fasttext-classifier

FineWeb-Edu FastText classifier

Model summary

This is a FastText classifier for judging the educational value of web pages based on training data fineweb-edu-llama3-annotations. There are two objectives:

• ⚡ throughput optimisation: It can classify more than 2000 examples per second in CPU, and so it can be used on-the-fly during pretraining/ to process huge data with CPU.
• 🧪fasttext vs transformer based model: How does this lightweight model with limited capacity compare to the original model HuggingFaceFW/fineweb-edu-classifier?

The FastText approach is inspired by my independent development of educational classifier based on a different definition of educational value, which can be found at kenhktsui/llm-data-textbook-quality-fasttext-classifier-v2.

🛠️Usage

from typing import List
import re
from huggingface_hub import hf_hub_download
import fasttext


model_hf = fasttext.load_model(hf_hub_download("kenhktsui/fineweb-edu-fasttext-classifier", "model.bin"))


def replace_newlines(text: str) -> str:
  return re.sub("\n+", " ", text)


def predict(text_list: List[str]) -> List[dict]:
  text_list = [replace_newlines(text) for text in text_list]
  pred = model_hf.predict(text_list)
  return [{"label": int(l[0].lstrip("__label__")), "score": s[0]}
           for l, s in zip(*pred)]


predict(["Hi"])
# Output: [{'label': 0, 'score': 1.00001}]

📊Evaluation

The last 46867 samples are used as test data, but it's not the exact test data as in HuggingFaceFW/fineweb-edu-classifier

Classification Report

              precision    recall  f1-score   support

           0       0.72      0.44      0.55      5704
           1       0.73      0.87      0.80     26595
           2       0.52      0.49      0.50     10350
           3       0.48      0.33      0.39      3397
           4       0.69      0.03      0.06       819
           5       0.00      0.00      0.00         2

    accuracy                           0.68     46867
   macro avg       0.52      0.36      0.38     46867
weighted avg       0.67      0.68      0.66     46867

The below table compares FastText model vs transformer based model.

Label	This Model	HuggingFaceFW/fineweb-edu-classifier
0	0.55	0.59
1	0.80	0.81
2	0.50	0.59
3	0.39	0.53
4	0.06	0.44
5	0.00	0.02

Label 0, 1, 2 are comparable to the original model. The performance degradation starts to be noticeable in label 3, and widen further in 4, which is due to limited capacity of fasttext model.
So, this classifer can perform reasonably well in label 0, 1, 2, and also 3 with some degradation.

Confusion Matrix

       [ 2537  3098    65     4     0     0]
       [  944 23037  2491   123     0     0]
y_true [   26  4742  5048   533     1     0]
       [    4   434  1846  1105     8     0]
       [    0    38   213   544    24     0]
       [    0     0     0     0     2     0]
                       y_pred

The model has a accuracy of 68%, and it is more likely to underpredict educational value than overpredict so. The exhibited conservatism is good for filtering large amount of data.

Predicted - Actual Rating	Frequency	%
0	31751	67.7%
-1	8078	17.2%
+1	6130	13.1%
-2	673	1.4%
+2	189	0.4%
-3	42	0.1%
+3	4	0.0%

Alignment with HuggingFaceFW/fineweb-edu-classifier

Spearman rank-order correlation coefficient is 0.5881 in MiniPile train split and 0.5832 in test split, indicating a moderately strong monotonic relationship in over 1 million representative document in web data.