<jupyter_start><jupyter_text>Import validation data for running a baseline model<jupyter_code>val = np.load('../data/interim/val_data.npy', allow_pickle=True)<jupyter_output><empty_output><jupyter_text>Upload columns from raw dataset<jupyter_code>df = pd.read_csv('../data/raw/Tweets.csv')
cols = df.columns
#create dev DataFrame
val = pd.DataFrame(val, columns=cols)
val.shape<jupyter_output><empty_output><jupyter_text>Apply textblob.sentiment method to get sentiment from text column `TextBlob.sentiment` returns a tuple of (*polarity*, *subjectivity*).  Not worried about the subjectivity right now so we'll just pull the polarity score.<jupyter_code>val['polarity'] = val.text.apply(lambda x: TextBlob(x).sentiment[0])<jupyter_output><empty_output><jupyter_text>Based on descriptive stats, bin data based on IQR 0 >= neg, and 0.2 <= pos<jupyter_code>val['polarity'].describe()<jupyter_output><empty_output><jupyter_text>Create labels of target data based on polarity score<jupyter_code>def polarity_to_label(polarity_score):
    if polarity_score >= pos_cutoff:
        return 2
    elif polarity_score > neg_cutoff:
        return 1
    else: 
        return 0

val['labels'] = val['polarity'].apply(lambda x: polarity_to_label(x))<jupyter_output><empty_output><jupyter_text>Create true target labels based on airline sentiment<jupyter_code>def sent_to_label(airline_sentiment):
    if airline_sentiment == 'positive':
        return 2
    elif airline_sentiment == 'neutral':
        return 1
    elif airline_sentiment == 'negative':
        return 0
val['y_label'] = val['airline_sentiment'].apply(lambda x: sent_to_label(x))
val.rename({'labels':'predicted', 'y_label':'actual'}, axis=1, inplace=True)<jupyter_output><empty_output><jupyter_text>Create and interpret Classification Report<jupyter_code>print(classification_report(val['actual'], val['predicted'], target_names=['negative', 'neutral', 'positive']))<jupyter_output>precision    recall  f1-score   support

    negative       0.71      0.73      0.72       913
     neutral       0.22      0.10      0.14       304
    positive       0.41      0.65      0.50       247

    accuracy                           0.58      1464
   macro avg       0.45      0.49      0.45      1464
weighted avg       0.56      0.58      0.56      1464<jupyter_text>$\qquad$ Overall accuracy for a naive model isn't horrible, it's almost twice as good as random guessing, which would be about 1 out of 3 correct or 33%.  This model captured 58%.  Not surprisingly, the precision for negative responses was highest (there were a lot of negative responses), while the worst scores were in the neutral category.  $\qquad$  We'll need a more nuanced model to capture the "neutral" sentiment.  Our recall for neutral is an abysymal 10%, meaning we are systematically misclassifying this category.  The main driver, of course, for this model are the pos and neg cutoff thresholds that we defined earlier.  No doubt, the model predictions will likely differ greatly depending on the thresholds that we define.<jupyter_code>confusion_matrix(val['actual'], val['predicted'], labels=[0, 1, 2])<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.0979772439949431, 'fraction_numerical': 0.029077117572692796, 'mean_word_length': 4.214168039538715, 'pattern_counts': {'":': 0, '<': 33, '<?xml version=': 0, '>': 35, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 2, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '8', 'dir': 'jupyter-structured-clean-dedup', 'id': '484393', 'license': 'FTL', 'n_tokens_mistral': 1028, 'n_tokens_neox': 942, 'n_words': 289, 'path': 'notebooks/Simple_Baseline.ipynb', 'repo_name': 'americanthinker/Twitter_project', 'size': '3164'}