<jupyter_start><jupyter_text>Models and FeaturesWe are currently using the following models to train against the dataset with the associated feature engineering:1. C-SVM    - The overviews are using a **bag of words** model and have been vectorized and transformed using **TF_IDF**.2. Naive Bayes    - The overviews are using a **bag of words** model and have been vectorized with a **Count Vectorizer**.3. Simple neural network (not deep)    - The overviews were tokenized with a **white space tokenizer**. Stop words were removed. Overviews were treated as **bag of words**, which each word being converted to a vector, using the GoogleNews-vectors-negative300.bin model.  The **arithmetic mean** of the words represented the overview. Taking the top 3 genres predicted for each movie.<jupyter_code>with open(f'{ROOT}/data/processed/genre_id_to_name_dict.pkl','rb') as f:
    genre_id_to_name=pickle.load(f)  

genre_names=list(genre_id_to_name.values())
import json

with open(f"{ROOT}/models/model_scores.json", "r") as f:
    scores = json.load(f)

with open(f'{ROOT}/data/processed/target_test.pkl','rb') as f:
    target_test=pickle.load(f)<jupyter_output><empty_output><jupyter_text>Naive Bayes with Raw Bag of Word Features Metrics for each genre<jupyter_code>with open(f'{ROOT}/models/classifier_nb.pkl','rb') as f:
    classifnb = pickle.load(f)

with open(f'{ROOT}/data/processed/raw_count_features_test.pkl','rb') as f:
    raw_count_features_test=pickle.load(f)

predsnb=classifnb.predict(raw_count_features_test)
print (classification_report(target_test, predsnb, target_names=genre_names))<jupyter_output>precision    recall  f1-score   support

         Action       0.55      0.54      0.55        48
      Adventure       0.41      0.58      0.48        31
      Animation       0.50      0.42      0.46        31
         Comedy       0.67      0.60      0.63        87
          Crime       0.50      0.34      0.41        29
    Documentary       0.65      0.64      0.64        61
          Drama       0.45      0.42      0.43        50
         Family       0.30      0.33      0.32        18
        Fantasy       0.25      0.08      0.12        12
        History       0.50      0.49      0.50        53
         Horror       0.52      0.44      0.48        25
          Music       0.59      0.71      0.65        14
        Mystery       0.55      0.50      0.52        34
        Romance       0.41      0.29      0.34        24
Science Fiction       0.33      0.50      0.40        14
       TV Movie       0.44      0.47      0.45        30
       Thriller       0.48    [...]<jupyter_text>Precision and Recall for the overall model<jupyter_code>nb_scores = scores["naive_bayes"]

md('''
Precision: {prec_mean}

Recall: {rec_mean}
'''.format(prec_mean=nb_scores["prec"], rec_mean=nb_scores["rec"]))<jupyter_output><empty_output><jupyter_text>C-SVM with TF-IDF features Metrics for each genre<jupyter_code>with open(f'{ROOT}/models/classifier_svc.pkl','rb') as f:
    svc_classifier=pickle.load(f)

with open(f'{ROOT}/data/processed/tfidf_count_features_test.pkl','rb') as f:
    tfidf_count_features=pickle.load(f)

predstfidf=svc_classifier.predict(tfidf_count_features)
print(classification_report(target_test, predstfidf, target_names=genre_names))<jupyter_output>precision    recall  f1-score   support

         Action       0.00      0.00      0.00        48
      Adventure       0.27      0.52      0.35        31
      Animation       0.00      0.00      0.00        31
         Comedy       0.58      0.57      0.58        87
          Crime       0.42      0.38      0.40        29
    Documentary       0.60      0.74      0.66        61
          Drama       0.00      0.00      0.00        50
         Family       0.44      0.39      0.41        18
        Fantasy       0.25      0.08      0.12        12
        History       0.46      0.55      0.50        53
         Horror       0.58      0.56      0.57        25
          Music       0.73      0.79      0.76        14
        Mystery       0.47      0.47      0.47        34
        Romance       0.35      0.38      0.36        24
Science Fiction       0.43      0.43      0.43        14
       TV Movie       0.40      0.27      0.32        30
       Thriller       0.47    [...]<jupyter_text>Precision and Recall for the overall model<jupyter_code>svc_scores = scores["svc"]

md('''
Precision: {prec_mean}

Recall: {rec_mean}
'''.format(prec_mean=svc_scores["prec"], rec_mean=svc_scores["rec"]))<jupyter_output><empty_output><jupyter_text>Simple Neural Network with Word2Vec features<jupyter_code>import keras

with open(f'{ROOT}/data/processed/w2v_features_test.pkl','rb') as f:
    w2v_features=pickle.load(f)

w2v_nn = keras.models.load_model(f"{ROOT}/models/classifier_nn.h5")
with open(f'{ROOT}/models/mlb.pkl','rb') as f:
    mlb=pickle.load(f)

# score = w2v_nn.evaluate(w2v_features, target_test, batch_size=249)
Y_preds=w2v_nn.predict(w2v_features)<jupyter_output><empty_output><jupyter_text>Precision and Recall for the overall model<jupyter_code>nn_scores = scores["neural_network"]

md('''
Precision: {prec_mean}

Recall: {rec_mean}
'''.format(prec_mean=nn_scores["prec"], rec_mean=nn_scores["rec"]))<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.09996191926884997, 'fraction_numerical': 0.07216298552932217, 'mean_word_length': 2.0991150442477875, 'pattern_counts': {'":': 0, '<': 27, '<?xml version=': 0, '>': 27, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '7', 'dir': 'jupyter-structured-clean-dedup', 'id': '359268', 'license': 'MIT', 'n_tokens_mistral': 1966, 'n_tokens_neox': 1793, 'n_words': 417, 'path': 'notebooks/comparison.ipynb', 'repo_name': 'JoshBroomberg/MovieGenrePrediction', 'size': '5252'}