<jupyter_start><jupyter_text>Whatsapp analysisthis notebook is created for the Mamba python course as an example for data analysis. The data was obtained by exporting a single Whatsapp chat. See https://faq.whatsapp.com/en/android/23756533/ to export your own data. developed by MAMBA  table of content:1. [import files](1)2. [read whatsapp data](2)3. [anonymize data](3)4. [plot hourly messages](4)5. [wordcloud](5)6. [text length](6)  1. import files<jupyter_code>import matplotlib.pyplot as plt
import zipfile
import os
import re
import pandas as pd
import wordcloud
import whatsapp_func as wf
import re
#settings
%matplotlib inline
plt.style.use('seaborn')<jupyter_output><empty_output><jupyter_text>2. read whatsapp data because of privacy issues I did not include the zip file in the GIT repository. You can use your own export zip file or use the anonymised pandas dataframe file available in the excercise folder: "_chat_df.csv"<jupyter_code># read whatsapp data
whatsapp_zip = r'XXX.zip'

time_user_df = wf.read_whatsapp(whatsapp_zip, strftime_format='%d/%m/%Y, %H:%M')<jupyter_output><empty_output><jupyter_text>3. anonymize data because of privacy issues I anonymised my data. I did this with the following cells. If you are running your own analysis, you can skip these lines.<jupyter_code># save anonymised dataframe
time_user_df[['user','message']].to_csv(r'data\_chat_df.csv')
# read anonymised dataframe
time_user_df = pd.read_csv(r'data\_chat_df.csv', index_col=0, parse_dates=True)
time_user_df.head()<jupyter_output><empty_output><jupyter_text>4. plot hourly messages<jupyter_code>time_user_df['hour'] = time_user_df.index.hour

hourly_messages_df = pd.DataFrame(index=range(24))
for user in time_user_df.user.unique():
    hourly_message_per_user = time_user_df[time_user_df.user==user].groupby(by='hour').sum()
    hourly_message_per_user.rename(columns={'message': user}, inplace=True)
    hourly_messages_df[user] = hourly_message_per_user[user]
# plot
fig = plt.figure(figsize=(16,16/2.75))
ax = plt.subplot()

# # format axes
ax.set_xlabel('Time (hour)')
ax.set_ylabel('number of messages', fontsize=12)

# create title
start_date = time_user_df.index[0].strftime('%d-%b-%Y')
end_date   = time_user_df.index[-1].strftime('%d-%b-%Y')
ax.set_title('sum of the message send per hour between %s and %s'%(start_date, end_date))

hourly_messages_df.plot(kind='bar', ax=ax)
fig.savefig(r'figures\messages_per_hour_per_user.png', bbox_inches='tight')<jupyter_output><empty_output><jupyter_text>5. plot wordcloud<jupyter_code>wf.check_anonymized_dataset(time_user_df)

wd = wordcloud.WordCloud().generate(' '.join(time_user_df['text'].values))
plt.imshow(wd, interpolation='bilinear')
plt.axis("off")
plt.savefig(r'figures\word_cloud.png', bbox_inches='tight')<jupyter_output><empty_output><jupyter_text>6. plot number of characters<jupyter_code>wf.check_anonymized_dataset(time_user_df)

for user in time_user_df.user.unique():
    print('average text length for %s is %i characters'%(user, time_user_df[time_user_df.user==user].text.str.len().mean()))
wf.check_anonymized_dataset(time_user_df)

time_user_df['hour'] = time_user_df.index.hour
time_user_df['text_length'] = time_user_df.text.str.len()

hourly_text_length_df = pd.DataFrame(index=range(24))
for user in time_user_df.user.unique():
    hourly_text_length_per_user = time_user_df[time_user_df.user==user].groupby(by='hour').mean()
    hourly_text_length_per_user.rename(columns={'text_length': user}, inplace=True)
    hourly_text_length_df[user] = hourly_text_length_per_user[user]
wf.check_anonymized_dataset(time_user_df)

# plot
fig = plt.figure(figsize=(16,16/2.75))
ax = plt.subplot()

# # format axes
ax.set_xlabel('Time (hour)')
ax.set_ylabel('characters per message', fontsize=12)

# create title
start_date = time_user_df.index[0].strftime('%d-%b-%Y')
end_date   = time_user_df.index[-1].strftime('%d-%b-%Y')
ax.set_title('average number of characters per message per hour of the day between %s and %s'%(start_date, end_date))

hourly_text_length_df.plot(kind='bar', ax=ax)
fig.savefig(r'figures\characters_per_message_per_user.png', bbox_inches='tight')
time_user_df.sort_values('text_length').tail()<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.13077469793887705, 'fraction_numerical': 0.012556266287609572, 'mean_word_length': 7.294695481335953, 'pattern_counts': {'":': 0, '<': 25, '<?xml version=': 0, '>': 25, 'https://': 1, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '6', 'dir': 'jupyter-structured-clean-dedup', 'id': '334717', 'license': 'MIT', 'n_tokens_mistral': 1593, 'n_tokens_neox': 1543, 'n_words': 307, 'path': 'Exercise_notebooks/On_topic/01_Pandas/whatsapp_data_analysis.ipynb', 'repo_name': 'teije01/course-material', 'size': '4221'}