<jupyter_start><jupyter_text>Starbucks Capstone Challenge IntroductionThis data set contains simulated data that mimics customer behavior on the Starbucks rewards mobile app. Once every few days, Starbucks sends out an offer to users of the mobile app. An offer can be merely an advertisement for a drink or an actual offer such as a discount or BOGO (buy one get one free). Some users might not receive any offer during certain weeks. Not all users receive the same offer, and that is the challenge to solve with this data set.Your task is to combine transaction, demographic and offer data to determine which demographic groups respond best to which offer type. This data set is a simplified version of the real Starbucks app because the underlying simulator only has one product whereas Starbucks actually sells dozens of products.Every offer has a validity period before the offer expires. As an example, a BOGO offer might be valid for only 5 days. You'll see in the data set that informational offers have a validity period even though these ads are merely providing information about a product; for example, if an informational offer has 7 days of validity, you can assume the customer is feeling the influence of the offer for 7 days after receiving the advertisement.You'll be given transactional data showing user purchases made on the app including the timestamp of purchase and the amount of money spent on a purchase. This transactional data also has a record for each offer that a user receives as well as a record for when a user actually views the offer. There are also records for when a user completes an offer. Keep in mind as well that someone using the app might make a purchase through the app without having received an offer or seen an offer. ExampleTo give an example, a user could receive a discount offer buy 10 dollars get 2 off on Monday. The offer is valid for 10 days from receipt. If the customer accumulates at least 10 dollars in purchases during the validity period, the customer completes the offer.However, there are a few things to watch out for in this data set. Customers do not opt into the offers that they receive; in other words, a user can receive an offer, never actually view the offer, and still complete the offer. For example, a user might receive the "buy 10 dollars get 2 dollars off offer", but the user never opens the offer during the 10 day validity period. The customer spends 15 dollars during those ten days. There will be an offer completion record in the data set; however, the customer was not influenced by the offer because the customer never viewed the offer. CleaningThis makes data cleaning especially important and tricky.You'll also want to take into account that some demographic groups will make purchases even if they don't receive an offer. From a business perspective, if a customer is going to make a 10 dollar purchase without an offer anyway, you wouldn't want to send a buy 10 dollars get 2 dollars off offer. You'll want to try to assess what a certain demographic group will buy when not receiving any offers. Final AdviceBecause this is a capstone project, you are free to analyze the data any way you see fit. For example, you could build a machine learning model that predicts how much someone will spend based on demographics and offer type. Or you could build a model that predicts whether or not someone will respond to an offer. Or, you don't need to build a machine learning model at all. You could develop a set of heuristics that determine what offer you should send to each customer (i.e., 75 percent of women customers who were 35 years old responded to offer A vs 40 percent from the same demographic to offer B, so send offer A).  Data SetsThe data is contained in three files:* portfolio.json - containing offer ids and meta data about each offer (duration, type, etc.)* profile.json - demographic data for each customer* transcript.json - records for transactions, offers received, offers viewed, and offers completedHere is the schema and explanation of each variable in the files:**portfolio.json*** id (string) - offer id* offer_type (string) - type of offer ie BOGO, discount, informational* difficulty (int) - minimum required spend to complete an offer* reward (int) - reward given for completing an offer* duration (int) - time for offer to be open, in days* channels (list of strings)**profile.json*** age (int) - age of the customer * became_member_on (int) - date when customer created an app account* gender (str) - gender of the customer (note some entries contain 'O' for other rather than M or F)* id (str) - customer id* income (float) - customer's income**transcript.json*** event (str) - record description (ie transaction, offer received, offer viewed, etc.)* person (str) - customer id* time (int) - time in hours since start of test. The data begins at time t=0* value - (dict of strings) - either an offer id or transaction amount depending on the record**Note:** If you are using the workspace, you will need to go to the terminal and run the command `conda update pandas` before reading in the files. This is because the version of pandas in the workspace cannot read in the transcript.json file correctly, but the newest version of pandas can. You can access the termnal from the orange icon in the top left of this notebook.  You can see how to access the terminal and how the install works using the two images below.  First you need to access the terminal:Then you will want to run the above command:Finally, when you enter back into the notebook (use the jupyter icon again), you should be able to run the below cell without any errors.<jupyter_code>import pandas as pd
import numpy as np
import json
from datetime import datetime
import matplotlib.pyplot as plt
from matplotlib.colors import ListedColormap
import seaborn as sns
import warnings
warnings.filterwarnings("ignore")
from sklearn.linear_model import LogisticRegression
from sklearn.ensemble import AdaBoostClassifier
from sklearn.metrics import accuracy_score,f1_score, classification_report, confusion_matrix
from sklearn.model_selection import train_test_split, GridSearchCV, RandomizedSearchCV
from sklearn.preprocessing import StandardScaler, FunctionTransformer
from sklearn.pipeline import Pipeline, FeatureUnion
from sklearn.dummy import DummyClassifier
plt.style.use('seaborn')
%matplotlib inline<jupyter_output><empty_output><jupyter_text>Read data from files<jupyter_code># read in the json files
portfolios_data = pd.read_json('portfolio.json', orient='records', lines=True)
profiles_data = pd.read_json('profile.json', orient='records', lines=True)
transcripts_data = pd.read_json('transcript.json', orient='records', lines=True)<jupyter_output><empty_output><jupyter_text>Look if any of datasets have a null values<jupyter_code># look if dataset has an empty values
portfolios_data.isnull().any()
profiles_data.isnull().any()
transcripts_data.isnull().any()<jupyter_output><empty_output><jupyter_text>**1. Data Cleaning** **Cleaning the portfolio data.*** Rename id col name to offer_id.* Apply one-hot encode.* Apply One-hot encode in offer_type Column<jupyter_code>portfolios_data.head()
# one-hot encode offer type
portfolios_data = pd.concat((portfolios_data, pd.get_dummies(portfolios_data['offer_type'])), axis=1)
portfolios_data.drop(['offer_type'], inplace=True, axis=1)
# rename id column
portfolios_data.rename(columns={'id': 'offer_id'}, inplace=True)
df_channels = pd.DataFrame(portfolios_data['channels'].apply(lambda x: ' '.join(x)).str.split(' ', expand=True))
channels = np.unique(df_channels.dropna().values)
# one-hot encode channels
df_channels.columns = channels
for col in channels:
    df_channels[col] = df_channels[col].apply(lambda x: 1 if x else 0)
portfolios_data = pd.concat((portfolios_data, df_channels), axis=1)
portfolios_data.drop(['channels'], inplace=True, axis=1)
portfolios_data.head()<jupyter_output><empty_output><jupyter_text>**B. Clean Profile data*** profile: rename id col name to customer_id.* profile: change age value of 118 to nan.* profile: create readable date format in became_member_on column.* profile: dropping rows that has no gender, income, age data* profile: convert gender values to numeric (0s , 1s)* profile: add starting years and months columns.<jupyter_code>profiles_data.head()
# convert time to datetime
def apply_datetime(x):
    return datetime.strptime(str(x), '%Y%m%d')
profiles_data['became_member_on'] = profiles_data['became_member_on'].apply(lambda x: apply_datetime(x))

profiles_data['age'] = profiles_data['age'].apply(lambda x: None if x == 118 else x)
profiles_data.describe()
profiles_data.dropna(axis=0, inplace=True)
profiles_data['gender'] = profiles_data['gender'].apply(lambda x: 0 if x == 'F' else 1)
profiles_data.dropna(inplace=True)
profiles_data.rename(columns={'id': 'customer_id'}, inplace=True)
profiles_data.reset_index(drop=True, inplace=True)
profiles_data.head()
profiles_data.isnull().any()
profiles_data['year'] = profiles_data['became_member_on'].dt.year
profiles_data['month'] = profiles_data['became_member_on'].dt.month
profiles_data.head()<jupyter_output><empty_output><jupyter_text>**C. Clean Transcript Data*** transcript: rename person col name to customer_id.* transcript: creating separate columns for amount and offer_id from value col.* transcript: dropping transaction rows whose customer_id is not in profile:customer_id* transcript: converting time in hours to time in days* segregate offer and transaction data* transcript: finally drop duplicates.<jupyter_code>transcripts_data.head()
transcripts_data.rename(columns={'person': 'customer_id'}, inplace=True)
# add two new columns - id for offer and amount spent by user - based on value column
transcripts_data['offer_id'] = None
transcripts_data['amount'] = None

for i in range(len(transcripts_data)):
    key, value = list(transcripts_data.iloc[i]['value'].items())[0]
    if key in['offer_id', 'offer id']:
        transcripts_data.at[i, 'offer_id'] = value
    elif key == 'amount':
        transcripts_data.at[i, 'amount'] = float(value)


transcripts_data.drop(['value'], inplace=True, axis=1)
# delete duplicates and set time in days
transcripts_data['time'] = transcripts_data['time'].apply(lambda x: x / 24)
transcripts_data.drop_duplicates(inplace=True)
transcripts_data.reset_index(drop=True, inplace=True)
transcripts_data.head()
# filte transcript data by users' ids, which are present in profiles dataset
transcripts_data = transcripts_data[transcripts_data['customer_id'].isin(profiles_data['customer_id'])]
# look at unique events
events = np.unique(transcripts_data['event'].values)
events
# create dataset with only offers in it
offers_data = transcripts_data[transcripts_data['event'] != 'transaction']
# one-hot encode offer event
offers_data ['received'] = offers_data.event.apply(lambda x: 1 if x == 'offer received' else 0)
offers_data ['completed'] = offers_data.event.apply(lambda x: 1 if x == 'offer completed' else 0)
offers_data ['viewed'] = offers_data.event.apply(lambda x: 1 if x == 'offer viewed' else 0)

offers_data.drop(columns=['event', 'amount'], inplace=True, axis=1)
offers_data[['viewed', 'completed', 'received']].apply(pd.Series.value_counts)<jupyter_output><empty_output><jupyter_text>Exploratory Data Analysis  1. Profile data<jupyter_code>profiles_data.head()
# Checking the null vlaues
profiles_data.isnull().any()
# Compute gender distribution in customer's data
profiles_data.gender.value_counts()
ax = profiles_data.gender.value_counts().plot(kind='bar', title='Gender distribution')
ax.set_xticklabels(['Males', 'Females']);<jupyter_output><empty_output><jupyter_text>Among the customers prevail men<jupyter_code># Stats of customer age and income attributes when income is specified
profiles_data[profiles_data['income'] != None].describe()
#Compute what percent of customers vs which year enrolled as a member
membership_each_year = profiles_data.groupby('year')['year'].count()

membership_each_year = (membership_each_year / membership_each_year.sum()) * 100
membership_each_year
# Compute what percent of customers vs which month enrolled as a member
membership_each_month = profiles_data['became_member_on'].dt.month.value_counts()

membership_each_month = (membership_each_month /  membership_each_year.sum()) * 100
membership_each_month.sort_index()<jupyter_output><empty_output><jupyter_text>Profile General Distribution — Age, Income, Members<jupyter_code># to be able to draw three subplots in a row
fig, ax = plt.subplots(ncols=3, nrows=1, figsize=(20, 6))
# plot a age distribution in data
profiles_data['age'].plot(kind='hist', ax=ax[0])
ax[0].set_xlabel('Age')
ax[0].set_ylabel('Count')
ax[0].set_title('Age Distribution')

# plot a income distribution in data
profiles_data['income'].plot(kind='hist', ax=ax[1])
ax[1].set_xlabel('Income')
ax[1].set_ylabel('Count')
ax[1].set_title('Income Distribution')

# plot a start_year distribution
profiles_data['year'].plot(kind='hist', ax=ax[2])
ax[2].set_xlabel('Year')
ax[2].set_ylabel('Count')
ax[2].set_title('Membership distribution by year');<jupyter_output><empty_output><jupyter_text>On the age distribution plot we see, that median age of customers is 60 and most of them belong to age between 40 to 70. Income distribution shows that the number of customers whose average salary is less than 70K is highThe third plot shows that number of customers was growing year by year, and only on last, 2018 is a gap, when number of customers is lower than in 2017, but higher than in 2016.  Income distribution as a function a gender<jupyter_code>sns.distplot(profiles_data[profiles_data['gender'] == 1]['income'], kde=False, label='Male')
sns.distplot(profiles_data[profiles_data['gender'] == 0]['income'], kde=False, label='Female')
plt.xticks(rotation=90)
plt.title('Income distribution hue gender')
plt.legend();<jupyter_output><empty_output><jupyter_text>In this plot we can see, that the minimum and maximum income for both male and female are almost the same, but the count of male customers in low-income level is higher than that of female customers.  Age distribution as a function of gender<jupyter_code>sns.distplot(profiles_data[profiles_data['gender'] == 0]['age'], kde=False, label='Female')
sns.distplot(profiles_data[profiles_data['gender'] == 1]['age'], kde=False, label='Male')
plt.title('Age distribution hue gender')
plt.legend();<jupyter_output><empty_output><jupyter_text>Number of males and females are distributed equaly - maximum and minimum age for both genders is the same, but we have more younger men.<jupyter_code># groupby start_year and gender to plot a graph
g = profiles_data.groupby(['year', 'gender'], as_index=False).size()
g = g.reset_index()
g.columns = ['year', 'gender', 'count']
# plot a bar graph for age distribution as a function of gender in membership program
plt.figure(figsize=(10, 5))
sns.barplot(x='year', y='count', hue='gender', data=g)
plt.xlabel('Membership Start Year')
plt.ylabel('Count');<jupyter_output><empty_output><jupyter_text>We see, that the Starbucks rewards program is more interesting for men rather for women.  2.Transaction data  Compute the event type distribution<jupyter_code>events = transcripts_data['event'].value_counts()
events
transactions = events.sum()

transactions_in_percent = 100 * events[0] / transactions
offers_in_percent = 100 * events[1:].sum() / transactions

transactions_in_percent, offers_in_percent<jupyter_output><empty_output><jupyter_text>Combine offer, customer and transactions data<jupyter_code># get all users' ids
users_ids = offers_data['customer_id'].unique()
# merge all datasets in one
data = []
# for every unique id of users
for id in users_ids:
    # get profile of user
    user = profiles_data[profiles_data['customer_id'] == id]
    # get actions of user
    actions = transcripts_data[transcripts_data['customer_id'] == id]
    # get payments
    payments = actions[actions['event'] == 'transaction']
    # get offers with diven id oedered to customer
    offers = offers_data[offers_data['customer_id'] == id]
    # get offers which were received, viewed and completed 
    offer_received_data = offers[offers['received'] == 1]
    offer_completed_data = offers[offers['completed'] == 1]
    offer_viewed_data = offers[offers['viewed'] == 1]
    rows = []
    # for each offer received
    for ind in range(offer_received_data.shape[0]):
        # get id
        offer_id = offer_received_data.iloc[ind]['offer_id']
        # get portfolio
        portfolio = portfolios_data[portfolios_data['offer_id'] == offer_id]
        # get offers with current id
        offer = offers[offers['offer_id'] == offer_id]
        # calculate start and end time
        start_offer = offer_received_data.iloc[ind]['time']
        end_offer = start_offer + portfolio['duration'].values[0]
        # was offer completed    
        filter_completed = (
            offer_completed_data['time'] >= start_offer) & (offer_completed_data['time'] <= end_offer)
        # was osser viewed in time (before offer ended)
        filter_viewed = (
                offer_viewed_data['time'] >= start_offer) & (offer_viewed_data['time'] <= end_offer)
        # was offer successful (viewed and completed within end time)
        offer_successful = filter_completed.sum() > 0 and filter_viewed.sum() > 0
        # get transaction amount in time of the offer
        transactions_filter = (payments['time'] >= start_offer) & (payments['time'] <= end_offer)
        transaction_in_time = payments[transactions_filter]
        # get total transaction
        transaction_total = transaction_in_time['amount'].sum()
        dict_row = {
                'id_offer': offer_id,
                'id_user': id,
                'time_start': start_offer,
                'time_end': end_offer,
                'total_spent': transaction_total,
                'is_successful': int(offer_successful),
            }
        dict_row.update(user.iloc[0].to_dict())
        dict_row.update(portfolio.iloc[0].to_dict())
        rows.append(dict_row)
    data.extend(rows)
df = pd.DataFrame(data)
df  
df['is_successful'].value_counts()
df.columns
df[['time_start', 'time_end', 'total_spent',
       'is_successful', 'gender', 'age', 'customer_id', 'became_member_on',
       'income', 'year', 'month', 'reward', 'difficulty', 'duration',
       'offer_id', 'bogo', 'discount', 'informational', 'email', 'mobile',
       'social', 'web']].to_csv('offer_analysis.csv', index=False)
df = pd.read_csv('offer_analysis.csv', parse_dates=['became_member_on'])
df.head()
g = df.groupby('offer_id')['is_successful'].agg(['count', 'sum'])
g['percent_success'] = (g['sum'] / g['count']) * 100
g<jupyter_output><empty_output><jupyter_text>Generate a plot that illustrates: How many customers were provided with a specific offer? Offer success rate (percentsuccess)<jupyter_code>fig, ax = plt.subplots(ncols=2, nrows=1, figsize=(20, 8), sharey=True)
g['percent_success'].plot(kind='barh', ax=ax[0])
ax[0].set_title('Percentage of successful offers')
g['count'].plot(kind='barh', ax=ax[1])
ax[1].set_title('Number customers each offer was given to');
ax = sns.countplot('gender', hue='is_successful', data=df)
ax.set_xticklabels(['Males', 'Females'])
ax.set_title('Number of successful and unsuccessful offers hue gender');<jupyter_output><empty_output><jupyter_text>We see, that offers, which were sent to men, are more successful than offers sent to women.    Build models<jupyter_code># look at columns
print(df.columns)
columns_for_prediction = ['gender', 'age', 
       'income', 'reward', 'difficulty', 'duration',
       'bogo', 'discount', 'informational', 'email', 'mobile',
       'social', 'web']
binary_columns = ['bogo', 'discount', 'informational', 'email', 'mobile',
       'social', 'web', 'gender']
num_columns = ['age', 'income', 'reward', 'difficulty', 'duration']

# separate features from targets
X = df[columns_for_prediction]
y = df['is_successful']
# create pipline to preprocess features. There are binary and numeric features, so they need to be preprocessed
# in different way. For example, we can leave the binary features without changing, but numeric scale
pipeline = Pipeline(steps = [       
    ('feature_processing', FeatureUnion(
        transformer_list = [        
            ('binary_processing', FunctionTransformer(lambda data: data[binary_columns])), 
               
            ('numeric_processing', Pipeline(steps = [
                ('selecting', FunctionTransformer(lambda data: data[num_columns])),
                ('scaling', StandardScaler())            
                        ])),
        ]))])
X_train, X_test, y_train,  y_test = train_test_split(X, y, test_size=0.2, random_state=17)
X_train = pipeline.fit_transform(X_train)
X_test = pipeline.transform(X_test)<jupyter_output><empty_output><jupyter_text>Construct Adaboost Classifier<jupyter_code># find best Adaboost
parameters = {'n_estimators': [100, 150, 200, 500, 1000]}
gs_adaboost = GridSearchCV(AdaBoostClassifier(), param_grid=parameters, cv=5, scoring='f1',
                          verbose=1, n_jobs=-1)
gs_adaboost.fit(X_train, y_train)
gs_adaboost.best_score_, gs_adaboost.best_params_
# save model
adaboost = gs_adaboost.best_estimator_<jupyter_output><empty_output><jupyter_text>Construct LogisticRegressionClassifier<jupyter_code># find best LogisticRegression
parameters = {'penalty': ['l1', 'l2', 'elasticnet', None], 'C': [0.001, 0.01, 0.1, 1]}
gs_lr = GridSearchCV(LogisticRegression(), param_grid=parameters, cv=5, scoring='f1',
                          verbose=1, n_jobs=-1)
gs_lr.fit(X_train, y_train)
gs_lr.best_score_, gs_lr.best_params_
lr = gs_lr.best_estimator_<jupyter_output><empty_output><jupyter_text>Construct Dummy Classifier<jupyter_code>dummy = DummyClassifier()
dummy.fit(X_train, y_train)
f1_score(y_test, dummy.predict(X_test))<jupyter_output><empty_output><jupyter_text>Compare two models<jupyter_code># get prediction for train and test dataset
adaboost_prediction_test = adaboost.predict(X_test)
lr_prediction_test = lr.predict(X_test)
adaboost_prediction_train = adaboost.predict(X_train)
lr_prediction_train = lr.predict(X_train)<jupyter_output><empty_output><jupyter_text>Plot confusion matrixes<jupyter_code>fig, axes = plt.subplots(2, 2, figsize=(12, 12))
sns.heatmap(confusion_matrix(y_train, lr_prediction_train), annot=True, fmt='d', ax=axes[0,0])
bot, top = axes[0,0].get_ylim() 
axes[0, 0].set_ylim(bot + 0.5, top - 0.5)
axes[0, 0].set_xlabel('Predicted label')
axes[0, 0].set_ylabel('True label')
axes[0, 0].set_title('Confusion matrix LogisticRegression (train set)')
sns.heatmap(confusion_matrix(y_test, lr_prediction_test), annot=True, fmt='d', ax=axes[0, 1])
bot, top = axes[0, 1].get_ylim() 
axes[0, 1].set_ylim(bot + 0.5, top - 0.5)
axes[0, 1].set_xlabel('Predicted label')
axes[0, 1].set_ylabel('True label')
axes[0, 1].set_title('Confusion matrix LogisticRegression (test set)')

sns.heatmap(confusion_matrix(y_train, adaboost_prediction_train), annot=True, fmt='d', ax=axes[1, 0])
bot, top = axes[1, 0].get_ylim() 
axes[1, 0].set_ylim(bot + 0.5, top - 0.5)
axes[1, 0].set_xlabel('Predicted label')
axes[1, 0].set_ylabel('True label')
axes[1, 0].set_title('Confusion matrix Adaboost (train set)')

axes[1, 1] = sns.heatmap(confusion_matrix(y_test, adaboost_prediction_test), annot=True, fmt='d', ax=axes[1, 1])
bot, top = axes[1, 1].get_ylim() 
axes[1, 1].set_ylim(bot + 0.5, top - 0.5)
axes[1, 1].set_xlabel('Predicted label')
axes[1, 1].set_ylabel('True label')
axes[1, 1].set_title('Confusion matrix Adaboost (test set)');<jupyter_output><empty_output><jupyter_text>Print classification report<jupyter_code>print('Classification report for Adaboost classifier (train)')
print(classification_report(y_train, adaboost_prediction_train))
print()
print('Classification report for Adaboost classifier (test)')
print(classification_report(y_test, adaboost_prediction_test))
print()
print()
print('Classification report for LogisticRegression classifier (train)')
print(classification_report(y_train, lr_prediction_train))
print()
print('Classification report for LogisticRegression classifier classifier (test)')
print(classification_report(y_test, lr_prediction_test))
print()
print('Classification report for Dummy classifier (test)')
print(classification_report(y_test, dummy.predict(X_test)))<jupyter_output>Classification report for Adaboost classifier (train)
              precision    recall  f1-score   support

           0       0.75      0.65      0.70     28183
           1       0.66      0.76      0.70     25017

    accuracy                           0.70     53200
   macro avg       0.70      0.70      0.70     53200
weighted avg       0.71      0.70      0.70     53200


Classification report for Adaboost classifier (test)
              precision    recall  f1-score   support

           0       0.74      0.66      0.70      6953
           1       0.67      0.75      0.71      6348

    accuracy                           0.70     13301
   macro avg       0.70      0.70      0.70     13301
weighted avg       0.71      0.70      0.70     13301



Classification report for LogisticRegression classifier (train)
              precision    recall  f1-score   support

           0       0.73      0.67      0.70     28183
           1       0.66      0.72      0.69     25017

    accu[...]
{'fraction_non_alphanumeric': 0.10441830271078412, 'fraction_numerical': 0.01644568595821694, 'mean_word_length': 4.9792989884733005, 'pattern_counts': {'":': 0, '<': 91, '<?xml version=': 0, '>': 93, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 6, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '22', 'dir': 'jupyter-structured-clean-dedup', 'id': '285840', 'license': 'MIT', 'n_tokens_mistral': 8032, 'n_tokens_neox': 7498, 'n_words': 2522, 'path': 'Starbucks_new.ipynb', 'repo_name': 'Abdol9900/Starbucks-Capstone-Challenge', 'size': '25417'}