<jupyter_start><jupyter_text>Data Challenge 2: <NAME>  Problem The number of weekly active users increased more than 100% between the beginning of May and the end of July, with only a few time periods showing even mild decreases. However, weekly active users decreased rougly 18% between the beginning and end of August, with the largest decrease occurring in the first week of August. What happened?  Hypotheses and Results **Hypothesis 1:** Something happened in the account creation process that reduced conversion from pending to active.**Results:** The rate of conversion from user creation to completing signup were similar in August as compared to May/June/July (52% vs. 50%), and the rate of attrition was similar at each step. The user activation funnel does not appear to be suffering from any new barriers to user activation.     **Hypothesis 2:** There is a drop in a particular type of user-driven engagement and, as a result, overall engagement has decreased.**Results:** Even though overall user-driven engagements are down in August, the relative frequency of each type of engagement is remarkably consistent (less than 1% difference in each category), indicating that once users are on Yammer there is no signficant difference in their behavior. **Bottom line:** I was unable to find any explanatory patterns for the decrease in customer engagement. With additional time, I would explore whether there are particular locations or companies where employee engagement is particularly sufferring and whether there has been a significant decrease in any particular types of Yammer-generated engagement events.  **Takeaway:** I should have invested in more EDA up front. I thought I was taking the right approach by starting with hypotheses, but I neglected to base the hypothesis on EDA and therefore wasted a significant amount of time. As a result, I was unable to reach a useful conclusion in the time given. My goal for this weekend is to increase my skills in EDA in Python so that I can explore datasets more quickly and efficiently.  Analysis  Hypothesis 1: Something happened in the account creation process that reduced conversion from pending to active. Approach: Split the dataset into before and after August 1, and use population pyramid to visualize the account creation/activation funnel with the goal of determining whether a problem occurred in August that is reducing the number of account activation and, if so, at what point in the process that problem occurred. Method: Read in 'Events' yammer_events.csv dataset. Create a dummy variable that identifies whether an event occurred on or after August 1 (aug_event where 1 Yes, 0 No). *(Note that this does not identify cases where user authentification began in June or July but was completed in August, but should still include a reasonable approximation of the process since edge cases on either end are also included.)* Select only events that occur in the user authentication process (event_type == signup_flow). Visualize user authentication funnel (create_user, enter_email, enter_info, complete_signup) comparing before August to August using a population pyramid. If no bottleneck is visible, merge in user authentication date from 'Users' yammer_users.csv as new rows to check the number of activations (activated_at).<jupyter_code># Import likely packages
import pandas as pd
import matplotlib.pylab as plt
import numpy as np
%matplotlib inline
from sklearn.linear_model import LinearRegression
import seaborn as sns
# Read in 'Events' yammer_events.csv and look at data structure of first few rows
events = pd.read_csv('yammer_events.csv')
events.head()
events.shape
# Create a dummy variable that identifies whether an event occurred 
# on or after August 1 (aug_event where 1 Yes, 0 No).
events['in_august'] = 0
events['in_august'] = np.where((pd.to_datetime(pd.Series(events['occurred_at']))) >= (pd.to_datetime('20140801', format='%Y%m%d')), 1, 0)
events.head()
events.tail()
events.shape
# Select only events that occur in the user authentication process (event_type == signup_flow).
auth_process = events[events.event_type == 'signup_flow']
auth_process.head()
auth_process.tail()
auth_process.shape
funnel_data = auth_process.groupby(['in_august','event_name'], as_index=False).count()
funnel_data
funnel_data = funnel_data.sort_values(['user_id'], ascending = False)
funnel_data
# Visualize user authentication funnel: create_user, enter_email, enter_info, complete_signup.
# Draw Plot
plt.figure(figsize=(13,10), dpi= 80)
group_col = 'in_august'
order_of_bars = funnel_data.event_name.unique()[::-1]
colors = [plt.cm.Spectral(i/float(len(funnel_data[group_col].unique())-1)) for i in range(len(funnel_data[group_col].unique()))]
for c, group in zip(colors, funnel_data[group_col].unique()):
    sns.barplot(x='user_id', y='event_name', data=funnel_data.loc[funnel_data[group_col]==group, :], order=order_of_bars, color=c, label=group)

# Decorations    
plt.xlabel("$Users$")
plt.ylabel("Stage of Authentication")
plt.yticks(fontsize=12)
plt.title("Authentication Funnel", fontsize=22)
plt.legend()
plt.show()<jupyter_output><empty_output><jupyter_text>The rate of conversion from user creation to completing signup were similar in August as compared to May/June/July (52% vs. 50%), and the rate of attrition was similar at each step, so the next step will be checking whether there was a significant dropoff in activation in August as compared to May/June/July. Add activation into dataset<jupyter_code># Read in 'Events' yammer_events.csv and look at data structure of first few rows
users = pd.read_csv('yammer_users.csv')
users.head()
# Create a dummy variable that identifies whether an event occurred on or after
# August 1 (aug_event where 1 Yes, 0 No)
users['in_august'] = 0
users['in_august'] = np.where((pd.to_datetime(pd.Series(users['activated_at']))) >= (pd.to_datetime('20140801', format='%Y%m%d')), 1, 0)
users.head()
users.tail()
# Create a dummy variable that identifies whether an event occurred before
# May 1 (before where 1 Yes, 0 No)
users['before'] = 0
users['before'] = np.where((pd.to_datetime(pd.Series(users['activated_at']))) < (pd.to_datetime('20140501', format='%Y%m%d')), 1, 0)
users.head()
users.shape
# Drop users that activated before May 1
new_users = users[users.before == 0]
new_users.shape
# Select only active users
active_users = new_users[new_users.state == 'active']
active_users.head()
active_users.tail()
active_users.shape
# Drop unneeded columns
active_users.columns.values
mask = ['user_id','in_august']
filtered_active_users = active_users[mask]
filtered_active_users.head()
filtered_active_users.tail()
activate = filtered_active_users.groupby('in_august', as_index = False).count()
activate
# Create new variable event_name = activate_user
activate['event_name'] = 'activate_user'
activate<jupyter_output><empty_output><jupyter_text>Apparently, complete_signup is equivalent to activation, so adding an activation level based on the activation date and time didn't add any additional information. There did not appear to be any significant difference in the user authentication funnel between August and May-June, so a problem in the user authentication process is unlikely to be the issue. **Note:** I realized at this point that some additional EDA might have kept me from spending too much time going down this road; for example, a simple line graph representing the number of new users over time would have shown me that there were no significant decreases in account creation.  Hypothesis 2: There is a drop in a particular type of user-driven engagement and, as a result, overall engagement has decreased. Approach: Split the dataset into before and after August 1 and compare the distribution of engagement activities in each time period. Method: Read in 'Events' yammer_events.csv dataset. Create a dummy variable that identifies whether an event occurred on or after August 1 (aug_event where 1 Yes, 0 No). *(Note that this does not identify cases where user authentification began in June or July but was completed in August, but should still include a reasonable approximation of the process since edge cases on either end are also included.)* Select only engagement events (event_type == engagement). Compare the distribution of engagement activities (home_page, like_message, login, search_autocomplete, search_run, search_click_result_X, send_message, view_inbox) comparing before August to August.<jupyter_code># Read in 'Events' yammer_events.csv and look at data structure of first few rows
events = pd.read_csv('yammer_events.csv')
events.head()
# Create a dummy variable that identifies whether an event occurred on or after
# August 1 (aug_event where 1 Yes, 0 No)
# Note that this dataset does not include events that occurred before May 1 (other than activation)
# So there is no need to filter out old events
events['in_august'] = 0
events['in_august'] = np.where((pd.to_datetime(pd.Series(events['occurred_at']))) >= (pd.to_datetime('20140801', format='%Y%m%d')), 1, 0)
events.head()
events.tail()
# Select only events user engagement process (event_type == engagement).
engagement = events[events.event_type == 'engagement']
engagement.head()
# Group data to compare August vs other
engagement_data = engagement.groupby(['in_august','event_name'], as_index=False).count()
engagement_data
engagement_data = engagement_data.sort_values(['in_august', 'event_name'], ascending = False)
engagement_data
    # Group data to calculate total engagement in both groups
    total_engagement = engagement.groupby(['in_august'], as_index=False).count()
    total_engagement<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.07912701696317749, 'fraction_numerical': 0.010136532892014894, 'mean_word_length': 5.823570924488355, 'pattern_counts': {'":': 0, '<': 15, '<?xml version=': 0, '>': 17, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '3', 'dir': 'jupyter-structured-clean-dedup', 'id': '543541', 'license': 'BSD-3-Clause', 'n_tokens_mistral': 2671, 'n_tokens_neox': 2522, 'n_words': 1181, 'path': 'others/Data_Challenge2/Sarah_Rustan_Data_Challenge_2.ipynb', 'repo_name': 'davestanley-cogrhythms/insight-datachallenge1', 'size': '9681'}