<jupyter_start><jupyter_text>Data Challenge 2**Author:** <NAME>**Date:** February 26, 2018<jupyter_code>#loading libraries
import numpy as np
import pandas as pd
import matplotlib.pyplot as plt
%matplotlib inline<jupyter_output><empty_output><jupyter_text>* First I will load the data and make the necessary adjustments and convertions to it, in order to produce the relevant plots<jupyter_code>#loading the relevant data
rawEvents = pd.read_csv('yammer_events.csv')
print(rawEvents.shape)
rawEvents.head()
#convert to datetime data types
rawEvents['occurred_at']=rawEvents['occurred_at'].astype('M')
rawEvents.dtypes<jupyter_output><empty_output><jupyter_text>* Below I define a function that allows me to group events by week, setting Mondays as the start of a week* Afterwards I add a column to the dataframe with the week where an event took place<jupyter_code>def getWeek(date):
    '''get week starting on mondays'''
    n=int(np.floor((date
                    -pd.Timestamp('2012-12-31'))/pd.Timedelta('7 day')))
    return (n*pd.Timedelta('7 day') +pd.Timestamp('2012-12-31'))
rawEvents['week']=rawEvents['occurred_at'].apply(lambda x:getWeek(x))
#check that calculations are correct with a small sample
rawEvents.tail()<jupyter_output><empty_output><jupyter_text>* The plot below is a reproduction of the plot presented in the challenge:<jupyter_code>plt.figure(figsize=(9,6))
plt.plot(rawEvents.groupby(['week'])['user_id']
         .unique().apply(lambda x:len(x)))
plt.title('Weekly active users', fontsize=16);
plt.axvline(pd.to_datetime('2014-07-28'), color='r', linestyle='--', lw=2)
plt.xticks(rotation=60);<jupyter_output><empty_output><jupyter_text>* The drop in weekly users coincides with the beggining of the last moth of the summer, when many people in the norther hemisphere go to vacation. * To confirm that this is the most likely explanation of the drop in users, I present two plots below. The firs one shows the daily active users in the United States, around **the weekend of the 4th of July**:<jupyter_code>#need to group events by day for the next plot
rawEvents['day']=rawEvents['occurred_at'].dt.date
plt.figure(figsize=(10,6))
plt.plot(rawEvents[rawEvents['location']=='United States'].groupby(['day'])['user_id'].unique().apply(lambda x:len(x)),
             label='United States');
plt.axvline(pd.to_datetime('2014-07-07'), color='r', linestyle='--', lw=2)
plt.title('Active users in the United States', fontsize=16);
plt.xlim(pd.to_datetime('2014-06-23'),pd.to_datetime('2014-07-21'))
plt.xticks(rotation=60)
plt.legend();<jupyter_output><empty_output><jupyter_text>* Note that there is a drop in user engagement at the start of the week immediately after the 4th of July weekend of 2014, which was a Friday* Note also that there is a marked slower rate of user engament during the days after that weekend, compared with the weeks before and after the holiday. * This shows that Holidays have a measurable effect on user engagementBelow I create a plot which groups users according to world regions, in order to test the two-hemisphere hypothesis<jupyter_code>#need to define regions of the world to group according to region
regions_map = { 'North America' : ['United States', 'Canada', 'Mexico'], 
               'Asia' : ['Japan', 'India', 'Thailand', 'Taiwan', 'Indonesia', 'Singapore', 'Hong Kong', 'Philippines', 'Malaysia'], 
               'South America' : ['Brazil', 'Colombia', 'Korea', 'Venezuela', 'Chile', 'Argentina'], 
               'Europe' : ['Netherlands', 'Austria', 'Finland', 'United Kingdom', 'France', 'Germany', 'Russia', 'Spain', 'Greece', 'Norway', 'Belgium', 'Poland', 'Sweden', 'Denmark', 'Italy', 'Portugal', 'Switzerland', 'Ireland'], 
               'Middle East' : ['Iran', 'Israel', 'Iraq', 'United Arab Emirates', 'Saudi Arabia', 'Pakistan', 'Turkey'], 
               'Africa' : ['Nigeria', 'Egypt', 'South Africa'], 'Australia' : ['Australia'] }
def getRegion(country):
    for key,value in regions_map.items():
        if country in value:
            return key
#defining a region column for the dataset
rawEvents['region']=rawEvents['location'].apply(lambda x:getRegion(x))
plt.figure(figsize=(12,8))
plt.plot(rawEvents[rawEvents['region']=='North America'].groupby(['week'])['user_id'].unique().apply(lambda x:len(x)),
             label='North America');
plt.plot(rawEvents[rawEvents['region']=='Europe'].groupby(['week'])['user_id'].unique().apply(lambda x:len(x)),
             label='Europe');
plt.plot(rawEvents[rawEvents['region']=='Asia'].groupby(['week'])['user_id'].unique().apply(lambda x:len(x)),
             label='Asia');
plt.plot(rawEvents[rawEvents['region']=='South America'].groupby(['week'])['user_id'].unique().apply(lambda x:len(x)),
             label='South America');
plt.plot(rawEvents[rawEvents['region']=='Africa'].groupby(['week'])['user_id'].unique().apply(lambda x:len(x)),
             label='Africa');
plt.plot(rawEvents[rawEvents['region']=='Australia'].groupby(['week'])['user_id'].unique().apply(lambda x:len(x)),
             label='Australia');
plt.plot(rawEvents[rawEvents['region']=='Middle East'].groupby(['week'])['user_id'].unique().apply(lambda x:len(x)),
             label='Middle East');
plt.title('Active users in the northern and souther hemispheres', fontsize=16);
plt.xticks(rotation=60)
plt.legend(loc='center left', bbox_to_anchor=(1, 0.5));<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.16083657227466222, 'fraction_numerical': 0.01628724782528225, 'mean_word_length': 5.590243902439024, 'pattern_counts': {'":': 0, '<': 26, '<?xml version=': 0, '>': 26, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '6', 'dir': 'jupyter-structured-clean-dedup', 'id': '78510', 'license': 'BSD-3-Clause', 'n_tokens_mistral': 1792, 'n_tokens_neox': 1640, 'n_words': 463, 'path': 'others/Data_Challenge2/Leonardo_Espin_data_challenge_2.ipynb', 'repo_name': 'davestanley-cogrhythms/insight-datachallenge1', 'size': '5411'}