<jupyter_start><jupyter_text>Sentiment Analysis with Google **Initiate libraries:**<jupyter_code>import pandas as pd
import matplotlib.pyplot as plt
plt.rcParams['figure.figsize'] = [10, 10]<jupyter_output><empty_output><jupyter_text>**Read in data from file:**<jupyter_code>df = pd.read_csv("../data/processed/20191209233601.19044.gkg.Labeled.txt", sep='\t', index_col=0)
df.Date = pd.to_datetime(df.Date, format="%Y%m%d")
df.info()<jupyter_output><class 'pandas.core.frame.DataFrame'>
Int64Index: 4089 entries, 0 to 4088
Data columns (total 13 columns):
Date             4089 non-null datetime64[ns]
NumArticles      4089 non-null int64
Counts           586 non-null object
Themes           4001 non-null object
Locations        4089 non-null object
Persons          3578 non-null object
Organizations    3771 non-null object
ToneData         4089 non-null object
CAMEOEvents      2222 non-null object
Sources          4089 non-null object
SourceURLs       4089 non-null object
text             3478 non-null object
label            4089 non-null int64
dtypes: datetime64[ns](1), int64(2), object(10)
memory usage: 447.2+ KB<jupyter_text>**Extract sentiment tone from sentiment data stored in `ToneData` column:**<jupyter_code>df['Tone'] = df['ToneData'].apply(lambda x: float(x.split(',')[0]))
df_tone  = df[ (df.label == 0) | (df.label == 1) ]
bydatedf_tone = df_tone.groupby(['Date']).mean()
axe = bydatedf_tone[['Tone']].resample("1d").mean().plot(figsize=(15,6), marker='o')
axe.vlines(pd.Timestamp('2018-03-18'), 2, -4, linestyles='dashed', color='black')
axe.set_ylabel('Sentiment Tone', fontsize=14)
axe.set_xlabel('Date', fontsize=14)
axe.set_title('Sentiment Tone by Google', fontsize=20)
axe.legend(["Google Sentiment Tone"], fontsize=20)
axe.grid()
plt.savefig("Ggl_Tone_March2018.png")
df_tone[ df_tone['Date'] < pd.Timestamp('2018-03-18') ].shape
df_tone[ df_tone['Date'] >= pd.Timestamp('2018-03-18') ].shape
plt.hist(df_tone[ df_tone['Date'] < pd.Timestamp('2018-03-18') ]['Tone'], bins=30, density=True)
plt.hist(df_tone[ df_tone['Date'] >= pd.Timestamp('2018-03-18') ]['Tone'], bins=30, density=True, color="orange", alpha=0.5)
plt.xlabel('Google Sentiment Tone', fontsize=14)
plt.ylabel('Normalized Frequency', fontsize=14)
plt.title('Distributions of Google Sentiment Tone',fontsize=20)
plt.legend(["before March 18 2018","after March 18 2018"],fontsize=20)
plt.savefig("Ggl_ToneHist_March2018.png")
plt.show()<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.16084484159220147, 'fraction_numerical': 0.08001624695369618, 'mean_word_length': 5.331619537275064, 'pattern_counts': {'":': 0, '<': 15, '<?xml version=': 0, '>': 15, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '3', 'dir': 'jupyter-structured-clean-dedup', 'id': '529331', 'license': 'MIT', 'n_tokens_mistral': 1041, 'n_tokens_neox': 888, 'n_words': 186, 'path': 'notebooks/05_AnnotateGoogle.ipynb', 'repo_name': 'pakhotin/NLP_AutonomousCars', 'size': '2462'}