trueparagraph.ai-ELECTRA

This model is a fine-tuned version of google/electra-base-discriminator on the None dataset. It achieves the following results on the evaluation set:

• Accuracy: 0.9430
• F1: 0.9421
• Precision: 0.9528
• Recall: 0.9316
• Mcc: 0.8862
• Roc Auc: 0.9429
• Pr Auc: 0.9217
• Log Loss: 0.8825
• Loss: 0.2952

Model description

TrueParagraph ELECTRA is a transformer-based model designed for detecting AI-generated text within academic and technical domains, particularly focusing on STEM (Science, Technology, Engineering, and Mathematics) texts. It leverages the ELECTRA architecture, which is known for its efficiency and accuracy in understanding complex text patterns and semantics. ELECTRA uses a novel training approach where it is trained as a discriminator rather than a generator, enhancing its ability to differentiate between real and rephrased text with higher precision. This makes TrueParagraph ELECTRA particularly effective in maintaining academic integrity by identifying potential AI-generated content.

Intended uses & limitations

AI-Generated Text Detection: TrueParagraph ELECTRA is optimized to detect AI-generated paragraphs within academic documents, theses, and research papers. Academic Integrity Enforcement: Useful for educators, researchers, and publishers in verifying the authenticity of written content.

Limitations:

Domain-Specific Performance: While highly effective in STEM-related texts, performance may vary in non-STEM fields due to the specific training dataset used. Potential Bias: The model's predictions might reflect biases present in the training data, particularly in edge cases where AI-generated and human-written text are indistinguishable. False Positives/Negatives: As with any AI model, there may be instances of misclassification, leading to false positives or false negatives, which users should account for when interpreting results.

Training and evaluation data

The model was trained and evaluated on the "pffaundez/16k-trueparagraph-STEM" dataset available on Hugging Face. This dataset comprises 16,000 paragraphs extracted from academic papers and theses across various STEM disciplines. The data includes both human-authored and AI-generated content, providing a balanced and representative sample for training a robust classification model. The dataset is preprocessed to maintain the integrity of technical terminologies, formulas, and citations, ensuring that the model is well-equipped to handle the intricacies of STEM literature.

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

• learning_rate: 5e-05
• train_batch_size: 16
• eval_batch_size: 16
• seed: 42
• optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
• lr_scheduler_type: linear
• lr_scheduler_warmup_steps: 500
• num_epochs: 5

Training results

Training Loss	Epoch	Step	Accuracy	F1	Precision	Recall	Mcc	Roc Auc	Pr Auc	Log Loss	Validation Loss
0.5401	0.6297	500	0.7694	0.7044	0.9732	0.5519	0.5963	0.7684	0.7602	3.5789	0.6109
0.3122	1.2594	1000	0.9225	0.9231	0.9122	0.9342	0.8452	0.9225	0.8850	1.1485	0.2368
0.2301	1.8892	1500	0.8670	0.8811	0.7942	0.9892	0.7573	0.8676	0.7910	1.9476	0.3654
0.1608	2.5189	2000	0.9348	0.9364	0.9103	0.9639	0.8711	0.9349	0.8955	1.0090	0.2677
0.1146	3.1486	2500	0.9430	0.9421	0.9528	0.9316	0.8862	0.9429	0.9217	0.8825	0.2952

Framework versions

• Transformers 4.42.4
• Pytorch 2.3.1+cu121
• Datasets 2.20.0
• Tokenizers 0.19.1