<jupyter_start><jupyter_text>Annotation and Sentiment Analysis with TextBlob **Initiate libraries:**<jupyter_code>import pandas as pd
import nltk
from textblob import TextBlob
import matplotlib.pyplot as plt
plt.rcParams['figure.figsize'] = [10, 10]
nltk.download('punkt')
nltk.download('averaged_perceptron_tagger')
nltk.download('brown')<jupyter_output>[nltk_data] Downloading package punkt to /home/tz5b44/nltk_data...
[nltk_data]   Package punkt is already up-to-date!
[nltk_data] Downloading package averaged_perceptron_tagger to
[nltk_data]     /home/tz5b44/nltk_data...
[nltk_data]   Package averaged_perceptron_tagger is already up-to-
[nltk_data]       date!
[nltk_data] Downloading package brown to /home/tz5b44/nltk_data...
[nltk_data]   Package brown is already up-to-date!<jupyter_text>**Read in data from file:**<jupyter_code>df = pd.read_csv("../data/processed/20191209233601.19044.gkg.Labeled.txt", sep='\t', index_col=0)
df.Date = pd.to_datetime(df.Date, format="%Y%m%d")
df.info()<jupyter_output><class 'pandas.core.frame.DataFrame'>
Int64Index: 4089 entries, 0 to 4088
Data columns (total 13 columns):
Date             4089 non-null datetime64[ns]
NumArticles      4089 non-null int64
Counts           586 non-null object
Themes           4001 non-null object
Locations        4089 non-null object
Persons          3578 non-null object
Organizations    3771 non-null object
ToneData         4089 non-null object
CAMEOEvents      2222 non-null object
Sources          4089 non-null object
SourceURLs       4089 non-null object
text             3478 non-null object
label            4089 non-null int64
dtypes: datetime64[ns](1), int64(2), object(10)
memory usage: 447.2+ KB<jupyter_text>**Annotate one record:**<jupyter_code>blob = TextBlob(df['text'][3])<jupyter_output><empty_output><jupyter_text>**Print out tags:**<jupyter_code>blob.tags<jupyter_output><empty_output><jupyter_text>**Print out noun phrases:**<jupyter_code>blob.noun_phrases<jupyter_output><empty_output><jupyter_text>**Print sentiment polarity for this example:**<jupyter_code>for sentence in blob.sentences:
    print(sentence.sentiment.polarity, end = ", ")<jupyter_output>-0.025, 0.17857142857142858, -0.0625, 0.15625, 0.0, 0.0, 0.1, 0.0, 0.0, -0.15, 0.0, 0.125, 0.3, 0.1875, 0.2, 0.05000000000000001, 0.1, 0.0, 0.1,<jupyter_text>**Print sentiment for entire document:**<jupyter_code>blob.sentiment<jupyter_output><empty_output><jupyter_text>**Print sentiment polarity for entire document:**<jupyter_code>blob.sentiment.polarity<jupyter_output><empty_output><jupyter_text>**Print sentiment subjectivity for entire document:**<jupyter_code>blob.sentiment.subjectivity<jupyter_output><empty_output><jupyter_text>**Define functions to calculate polarity and subjectivity for input text:**<jupyter_code>def TB_Polarity (text):
    text_ = str(text)[:500]
    try:
        tb_polarity = TextBlob(text_).sentiment.polarity
    except:
        tb_polarity = -200.0
    return tb_polarity
def TB_Subjectivity (text):
    text_ = str(text)[:500]
    try:
        tb_subjectivity = TextBlob(text_).sentiment.subjectivity
        tb_subjectivity
    except:
        tb_subjectivity = -200.0
    return tb_subjectivity<jupyter_output><empty_output><jupyter_text>**Calculate polarity and subjectivity for all records:**<jupyter_code>%time df['tb_polarity'] = df['text'].apply(TB_Polarity)
df_tb_polarity01  = df[ (df.tb_polarity > -100) & ((df.label == 0) | (df.label == 1)) ]
bydatedf_tb_polarity01 = df_tb_polarity01.groupby(['Date']).mean()
axe = bydatedf_tb_polarity01[['tb_polarity']].resample("1d").mean().plot(figsize=(15,6), marker='o')
axe.vlines(pd.Timestamp('2018-03-18'), 0.2, -0.0, linestyles='dashed', color='black')
axe.set_ylabel('Polarity', fontsize=14)
axe.set_xlabel('Date', fontsize=14)
axe.set_title('Sentiment Polarity with TextBlob', fontsize=20)
axe.legend(["TextBlob Polarity"], fontsize=20)
axe.grid()
plt.savefig("TB_Polarity_March2018.png")
df_tb_polarity01[ df_tb_polarity01['Date'] < pd.Timestamp('2018-03-18') ].shape
df_tb_polarity01[ df_tb_polarity01['Date'] >= pd.Timestamp('2018-03-18') ].shape
plt.hist(df_tb_polarity01[ df_tb_polarity01['Date'] < pd.Timestamp('2018-03-18') ]['tb_polarity'], bins=30, density=True)
plt.hist(df_tb_polarity01[ df_tb_polarity01['Date'] >= pd.Timestamp('2018-03-18') ]['tb_polarity'], bins=30, density=True, color="orange", alpha=0.5)
plt.xlabel('TextBlob Polarity', fontsize=14)
plt.ylabel('Normalized Frequency', fontsize=14)
plt.title('Distributions of TextBlob Sentiment Polarity',fontsize=20)
plt.legend(["before March 18 2018","after March 18 2018"],fontsize=20)
plt.savefig("TB_PolarityHist_March2018.png")
plt.show()
%time df['tb_subjectivity'] = df['text'].apply(TB_Subjectivity)
df_tb_subjectivity = df[ (df.tb_subjectivity > -100) & ((df.label == 0) | (df.label == 1)) ]
bydatedf_tb_subjectivity=df_tb_subjectivity.groupby(['Date']).mean()
axe = bydatedf_tb_subjectivity[['tb_subjectivity']].resample("1d").median().plot(figsize=(15,6), marker='o')
axe.vlines(pd.Timestamp('2018-03-18'), 0.6, 0.0, linestyles='dashed', color='black')
axe.set_ylabel('Subjectivity', fontsize=14)
axe.set_xlabel('Date', fontsize=14)
axe.set_title('Sentiment Subjectivity with TextBlob', fontsize=20)
axe.legend(["TextBlob Subjectivity"], fontsize=20)
axe.grid()
plt.savefig("TB_Subjectivity_March2018.png")<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.1472175693281221, 'fraction_numerical': 0.06830448538991253, 'mean_word_length': 6.089709762532982, 'pattern_counts': {'":': 0, '<': 45, '<?xml version=': 0, '>': 47, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 1, 'word_list_counts': {'cursed_substrings.json': 6, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '11', 'dir': 'jupyter-structured-clean-dedup', 'id': '522751', 'license': 'MIT', 'n_tokens_mistral': 2241, 'n_tokens_neox': 1985, 'n_words': 350, 'path': 'notebooks/04_AnnotateTextBlob.ipynb', 'repo_name': 'pakhotin/NLP_AutonomousCars', 'size': '5373'}