<jupyter_start><jupyter_text>Example 1: Training the Pseudoscientific Videos Detection Classifier<jupyter_code>import os
os.chdir('../')

from dataset.DatasetUtils import DatasetUtils
from classifier.featureengineering.FeatureEngineeringModels import FeatureEngineeringModels
from classifier.training.ClassifierTraining import ClassifierTraining

# Create Objects
dataset = DatasetUtils()
featureEngineeringModels = FeatureEngineeringModels(dataset_object=dataset)<jupyter_output>/Users/matus/opt/miniconda3/lib/python3.8/site-packages/requests/__init__.py:87: RequestsDependencyWarning: urllib3 (1.25.8) or chardet (2.0.3) doesn't match a supported version!
  warnings.warn("urllib3 ({}) or chardet ({}) doesn't match a supported "<jupyter_text>Step 1. Fine-tune separate fastText models for each Video Metadata TypeIn this step, we will fine-tune four separate fastText models, one for each different video metadata type,which we use during the training of Deep Learning model to generate embeddings for each different video metadata type.This step is only required to run once. 1.1. Video Snippet<jupyter_code># Generate Video Snippet fastText input features
featureEngineeringModels.prepare_fasttext_data(model_type='video_snippet')

# Fine-tune a fastText model for Video Snippet
featureEngineeringModels.finetune_model(model_type='video_snippet')<jupyter_output><empty_output><jupyter_text>1.2. Video Tags<jupyter_code># Generate Video Tags fastText input features
featureEngineeringModels.prepare_fasttext_data(model_type='video_tags')

# Fine-tune a fastText model for Video Tags
featureEngineeringModels.finetune_model(model_type='video_tags')<jupyter_output><empty_output><jupyter_text>1.3. Video Transcript<jupyter_code># Generate Video Transcript fastText input features
featureEngineeringModels.prepare_fasttext_data(model_type='video_transcript')

# Fine-tune a fastText model for Video Transcript
featureEngineeringModels.finetune_model(model_type='video_transcript')<jupyter_output><empty_output><jupyter_text>1.4. Video Comments<jupyter_code># Generate Video Comments fastText input features
featureEngineeringModels.prepare_fasttext_data(model_type='video_comments')

# Fine-tune a fastText model for Video Comments
featureEngineeringModels.finetune_model(model_type='video_comments')<jupyter_output><empty_output><jupyter_text>Step 2. Train the Pseudoscience Deep Learning ModelAt this step, we train and validate the Pseudoscientific Content Detection Deep Learning model using 10-fold cross-validation.At the end of the training, the best model will be stored in: ```pseudoscientificvideosdetection\models\pseudoscience_model_final.hdf5```. 2.1. Create a Classifier Training ObjectThe creation of this object automatically creates the deep learning model<jupyter_code># Create ClassifierTraining Object
classifierTrainingObject = ClassifierTraining(dataset_object=dataset)<jupyter_output>/Users/matus/opt/miniconda3/lib/python3.8/site-packages/keras/optimizer_v2/optimizer_v2.py:355: UserWarning: The `lr` argument is deprecated, use `learning_rate` instead.
  warnings.warn(<jupyter_text>2.2. Train the Classifier<jupyter_code>classifierTrainingObject.train_model()<jupyter_output>/n---Training the Model with 1971 videos.

--- [K-FOLD 1/10] TRAIN: 1773, TEST: 198
TOTAL VIDEOS: 1773 | TRAIN: 631, TEST: 159
--- Oversampling Train set...
--- [AFTER OVER-SAMPLING] TRAIN: 918, VAL: 159, TEST: 198

--- Classifier Training started...
3
[array([[-0.00154017,  0.00425594, -0.01610176, ...,  0.0909829 ,
         0.00650647, -0.00792344],
       [-0.02839425,  0.00769057, -0.00898624, ...,  0.08284736,
         0.00765291, -0.00735889],
       [ 0.00877337,  0.0220746 , -0.02116582, ...,  0.07865869,
         0.01977451, -0.0105785 ],
       ...,
       [ 0.00041521,  0.00509564, -0.02090871, ...,  0.08547921,
         0.00670899, -0.01469666],
       [-0.00853482,  0.00643449, -0.02708934, ...,  0.0946865 ,
         0.01245062, -0.00099109],
       [-0.00478722,  0.01052219, -0.01604749, ...,  0.09192502,
         0.00564561, -0.00994288]], dtype=float32), array([[-0.00168346,  0.00280175, -0.01627938, ...,  0.09156334,
         0.01212652, -0.00420917],
       [-0.006592[...]
{'fraction_non_alphanumeric': 0.10693916349809886, 'fraction_numerical': 0.10860266159695818, 'mean_word_length': 5.226331360946745, 'pattern_counts': {'":': 0, '<': 26, '<?xml version=': 0, '>': 26, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '7', 'dir': 'jupyter-structured-clean-dedup', 'id': '459589', 'license': 'MIT', 'n_tokens_mistral': 1661, 'n_tokens_neox': 1371, 'n_words': 314, 'path': 'examples/train_pseudoscience_classifier.ipynb', 'repo_name': 'kinit-sk/pseudoscience-paper', 'size': '4208'}