<jupyter_start><jupyter_text>Starbucks Capstone Challenge IntroductionThis data set contains simulated data that mimics customer behavior on the Starbucks rewards mobile app. Once every few days, Starbucks sends out an offer to users of the mobile app. An offer can be merely an advertisement for a drink or an actual offer such as a discount or BOGO (buy one get one free). Some users might not receive any offer during certain weeks. Not all users receive the same offer, and that is the challenge to solve with this data set.Your task is to combine transaction, demographic and offer data to determine which demographic groups respond best to which offer type. This data set is a simplified version of the real Starbucks app because the underlying simulator only has one product whereas Starbucks actually sells dozens of products.Every offer has a validity period before the offer expires. As an example, a BOGO offer might be valid for only 5 days. You'll see in the data set that informational offers have a validity period even though these ads are merely providing information about a product; for example, if an informational offer has 7 days of validity, you can assume the customer is feeling the influence of the offer for 7 days after receiving the advertisement.You'll be given transactional data showing user purchases made on the app including the timestamp of purchase and the amount of money spent on a purchase. This transactional data also has a record for each offer that a user receives as well as a record for when a user actually views the offer. There are also records for when a user completes an offer. Keep in mind as well that someone using the app might make a purchase through the app without having received an offer or seen an offer. ExampleTo give an example, a user could receive a discount offer buy 10 dollars get 2 off on Monday. The offer is valid for 10 days from receipt. If the customer accumulates at least 10 dollars in purchases during the validity period, the customer completes the offer.However, there are a few things to watch out for in this data set. Customers do not opt into the offers that they receive; in other words, a user can receive an offer, never actually view the offer, and still complete the offer. For example, a user might receive the "buy 10 dollars get 2 dollars off offer", but the user never opens the offer during the 10 day validity period. The customer spends 15 dollars during those ten days. There will be an offer completion record in the data set; however, the customer was not influenced by the offer because the customer never viewed the offer. CleaningThis makes data cleaning especially important and tricky.You'll also want to take into account that some demographic groups will make purchases even if they don't receive an offer. From a business perspective, if a customer is going to make a 10 dollar purchase without an offer anyway, you wouldn't want to send a buy 10 dollars get 2 dollars off offer. You'll want to try to assess what a certain demographic group will buy when not receiving any offers. Final AdviceBecause this is a capstone project, you are free to analyze the data any way you see fit. For example, you could build a machine learning model that predicts how much someone will spend based on demographics and offer type. Or you could build a model that predicts whether or not someone will respond to an offer. Or, you don't need to build a machine learning model at all. You could develop a set of heuristics that determine what offer you should send to each customer (i.e., 75 percent of women customers who were 35 years old responded to offer A vs 40 percent from the same demographic to offer B, so send offer A).  Data SetsThe data is contained in three files:* portfolio.json - containing offer ids and meta data about each offer (duration, type, etc.)* profile.json - demographic data for each customer* transcript.json - records for transactions, offers received, offers viewed, and offers completedHere is the schema and explanation of each variable in the files:**portfolio.json*** id (string) - offer id* offer_type (string) - type of offer ie BOGO, discount, informational* difficulty (int) - minimum required spend to complete an offer* reward (int) - reward given for completing an offer* duration (int) - time for offer to be open, in days* channels (list of strings)**profile.json*** age (int) - age of the customer * became_member_on (int) - date when customer created an app account* gender (str) - gender of the customer (note some entries contain 'O' for other rather than M or F)* id (str) - customer id* income (float) - customer's income**transcript.json*** event (str) - record description (ie transaction, offer received, offer viewed, etc.)* person (str) - customer id* time (int) - time in hours since start of test. The data begins at time t=0* value - (dict of strings) - either an offer id or transaction amount depending on the record**Note:** If you are using the workspace, you will need to go to the terminal and run the command `conda update pandas` before reading in the files. This is because the version of pandas in the workspace cannot read in the transcript.json file correctly, but the newest version of pandas can. You can access the termnal from the orange icon in the top left of this notebook.  You can see how to access the terminal and how the install works using the two images below.  First you need to access the terminal:Then you will want to run the above command:Finally, when you enter back into the notebook (use the jupyter icon again), you should be able to run the below cell without any errors.<jupyter_code># import the necessary libraries
import pandas as pd
import numpy as np
import matplotlib.pyplot as plt
import seaborn as sns
import plotly as py
import plotly.graph_objs as go
import math
import json
import re

from sklearn.preprocessing import LabelBinarizer, MinMaxScaler, MultiLabelBinarizer
from sklearn.metrics import accuracy_score, f1_score
from sklearn.ensemble import RandomForestClassifier
from sklearn.linear_model import LogisticRegression
from sklearn.metrics import fbeta_score, make_scorer, confusion_matrix, classification_report
from sklearn.model_selection import GridSearchCV, RandomizedSearchCV, train_test_split
from sklearn.svm import SVC

% matplotlib inline
# read in the json files
portfolio = pd.read_json('data/portfolio.json', orient='records', lines=True)
profile = pd.read_json('data/profile.json', orient='records', lines=True)
transcript = pd.read_json('data/transcript.json', orient='records', lines=True)
# lets check out few records in the portfolio df
portfolio.head()
portfolio.shape<jupyter_output><empty_output><jupyter_text>**Based on the data in the portfolia data set:**  - Channels variable is multi-label (can split using MultiLabelBinarizer)  - Rename 'id' variable to 'offer_id'  - One Hot Encode the 'offer_type' variable  There are only 10 records.<jupyter_code># Lets create a new df with the Channels data split into binary values 
# https://scikit-learn.org/stable/modules/generated/sklearn.preprocessing.MultiLabelBinarizer.html
binarizer_obj = MultiLabelBinarizer()
binarizer_obj.fit(portfolio['channels'])

offer_channels_df = pd.DataFrame(binarizer_obj.transform(portfolio['channels']),columns=binarizer_obj.classes_)

offer_channels_df
# Lets raneme the id variable to "offer_id"
portfolio.rename(columns={"id": "offer_id"}, inplace=True)
portfolio.head()
#Lets one hot encode the offert_type column
# http://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.get_dummies.html
offer_type_df = pd.get_dummies(portfolio['offer_type'])
offer_type_df.head()
#lets concantenate all the data frames together to create a new df
portfolio = pd.concat([portfolio, offer_type_df, offer_channels_df], axis=1)

# remove duplicate columns
#portfolio.drop(columns=['offer_type', 'channels'], inplace=True)

portfolio.head()
# lets re-order the columns

portfolio_df = portfolio[ ['offer_id','difficulty','duration','reward','offer_type','bogo','discount',
                         'informational','channels','email','mobile','social','web'] ]

portfolio_df.head()<jupyter_output><empty_output><jupyter_text>Profile DataLets take a look at the profile and perform EDA<jupyter_code># lets check out some data in the profile df
profile.head()
profile.shape
# Interesting to see rows with age as 118. Lets take a look at those
age_outlier_df = profile[profile.age == 118]
print("Unique Gender Values : ",age_outlier_df.gender.nunique())
print("Unique Age Values : ",age_outlier_df.age.nunique())
print("Unique income Values : ",age_outlier_df.income.nunique())<jupyter_output>Unique Gender Values :  0
Unique Age Values :  1
Unique income Values :  0<jupyter_text>Outlier values in Profile Data Frame  - There are many records with age value as 118  - On closer examination you will notice that these records have no gender and income values  - My assumption is that Date of Birth is not a mandatory field in the app. And anyone who doesnt have an age is given a default age by the app. I am keeping this data in a separate data frame and removing these values from the profile data frame<jupyter_code>age_outlier_df.shape
# remove the records with age values of 118. This should give us a total of 14825 records
profile = profile[profile.age < 118]
profile.shape<jupyter_output><empty_output><jupyter_text>Lets do the following for the profile data  - Change the date format of the became_member_on column  - Lets get the year from the date. A new column "membershipstart" will be added  - Lets hot encode the membership years  - Lets create an age range from the ages column  - Lets hot encode the age range values  - Lets one hot encode the gender column  - rename the id column to customer_id  - remove any null values<jupyter_code># The date when the customer became a member is in YYYYMMDD format. Lets convert into a date format
profile.became_member_on = pd.to_datetime(profile.became_member_on, format = '%Y%m%d')
profile.head()
profile['membershipstart'] = profile['became_member_on'].apply(lambda x: x.year)
profile.head()
# lets check for any nan values
profile.isna().sum(axis=0)

#seems there are no nan values. Hurray!!
## It would have been easier to plot the age distribution to see the anamoly. Better late than never. So here you go!!
ax = sns.distplot(profile.age,color="g")
## Lets check the gender distribution
profile.groupby('gender')['age'].count().plot(x='gender',y='count',kind="bar",title="Gender Distribution");
age_df = profile.groupby('gender',as_index=False).count()
#Lets check the distribution of users by gender
(profile.groupby('gender').size() / profile.shape[0]) * 100<jupyter_output><empty_output><jupyter_text>**There are 212 records with gender as "O". Lets remove that.** We have to make sure records belongning to these customers are also removed from the transcript data set.<jupyter_code>profile = profile[profile['gender'] != 'O']
profile = profile.reset_index(drop=True)
(profile.groupby('gender').size() / profile.shape[0]) * 100
# lets rename the id column to customer_id
profile.rename(columns={"id": "customer_id"}, inplace=True)
profile.head()
## Lets check the income distribution
ax = sns.distplot(profile.income,color="r")
# Lets look at memberships by year
fig, ax = plt.subplots(figsize=(10,6))
profile.groupby('membershipstart')['age'].count().plot(ax=ax , kind="bar")
ax.set_xlabel("Year")
ax.set_ylabel("No of Registrations")
ax.set_title("Memberships By Year");
# Memberships by Year by Gender
fig, ax = plt.subplots(figsize=(10,6))
profile.groupby(['membershipstart','gender'])['age'].count().unstack().plot(ax=ax ,kind="bar")
ax.set_xlabel("Year")
ax.set_ylabel("No of Registrations")
ax.set_title("Memberships By Year by Gender");
# lets check out the customers based on gender more closely
female_customers = profile[profile.gender == 'F']
male_customers = profile[profile.gender == 'M']
# lets do a combined income distribution of the two data sets so that it is easy to analyse side by side
current_palette = sns.color_palette()

sns.set(font_scale=1.5)
sns.set_style('white')

fig, ax = plt.subplots(figsize=(15, 6),
                       nrows=1,
                       ncols=2,
                       sharex=True,
                       sharey=True)

plt.sca(ax[0])
sns.distplot(male_customers['income'] * 1E-3,
             color=current_palette[1])
plt.xlabel('Income [000]')
plt.ylabel('P(Income)')
plt.title('Male Customer Income')

plt.sca(ax[1])
sns.distplot(female_customers['income'] * 1E-3,
             color=current_palette[0])
plt.xlabel('Income [000]')
plt.ylabel('P(Income)')
plt.title('Female Customer Income')
plt.tight_layout()<jupyter_output><empty_output><jupyter_text>Income Analysis Based on GenderFrom the above charts it seems that the minimum and maximum income for both male and female customers is approximately the same. However, male customer income is slightly biased towards lower values compared to female customer income.<jupyter_code># One hot encode a customer's age range
min_age = np.int(np.floor(np.min(profile['age'])/10)*10)
max_age = np.int(np.ceil(np.max(profile['age'])/10)*10)

profile['agerange'] =   pd.cut(profile['age'],(range(min_age,max_age + 10, 10)),\
           right=False)

profile['agerange'] = profile['agerange'].astype('str')

# remove the ",", "[" and ")" from the age range values
profile.agerange = profile.agerange.str.replace(',',' -')
profile.agerange = profile.agerange.str.replace('[','')
profile.agerange = profile.agerange.str.replace(')','')

#check the data
profile.head()
profile.agerange.unique().tolist()

# cleaned up data looks good
fig, ax = plt.subplots(figsize=(10,6))
profile.groupby(['agerange','gender'])['age'].count().unstack().plot(ax=ax ,kind="bar")
ax.set_xlabel("Year")
ax.set_ylabel("No of Registrations")
ax.set_title("Memberships By Age Range by Gender");<jupyter_output><empty_output><jupyter_text>**The most members are in the 50 - 60 age range, followed by 60 -70 and 40 - 50 age ranges.**<jupyter_code>fig, ax = plt.subplots(figsize=(15,6))
profile.groupby(['membershipstart','agerange'])['age'].count().unstack().plot(ax=ax ,kind="bar")
ax.set_xlabel("Year")
ax.set_ylabel("No of Registrations")
ax.set_title("Memberships By Age Range by Year");<jupyter_output><empty_output><jupyter_text>**2017 saw the most number of registrations**<jupyter_code># One Hot Encode Data
cols = ['customerid','gender','income']

#one hot encode membership start year
membershipyear_df = pd.get_dummies(profile['membershipstart'])
cols.extend(membershipyear_df.columns.values)

# One hot encode Age range values
agerange_df = pd.get_dummies(profile['agerange'])
cols.extend(agerange_df.columns.values)

# Transform a customer's gender from a character to a number
binarizerobj = LabelBinarizer()
profile['gender_value'] = binarizerobj.fit_transform(profile['gender'])
# Check the gender mapping
gender_map = {}
for g in binarizerobj.classes_:
    gender_map[g] = binarizerobj.transform([g])[0,0]
gender_map    
# Appened the one hot encoded columns to the original data frame
profile = pd.concat([profile,
                     agerange_df,
                     membershipyear_df], axis=1)


# Remove depcreated columns
profile = profile.drop(columns=['age',
                                'agerange',
                                'became_member_on',
                                'membershipstart'])
profile.head()
# lets check out some data from the transcript df
transcript.head()
# Lets see event count distribution
event_counts = transcript['event'].value_counts()
event_counts = pd.DataFrame(list(zip(event_counts.index.values, event_counts)),
                            columns=['event', 'count'])
event_counts
print(event_counts.event.tolist())
print(event_counts['count'].tolist())
total_transactions = event_counts['count'].sum()

percentage_transactions = 100 * event_counts.iloc[0]['count'] / total_transactions
percentage_offers = 100 * event_counts.iloc[1:]['count'].sum() / total_transactions

print("Percentage of customer transaction events: %.1f %%" % (percentage_transactions))
print("Percentage of customer offer events: %.1f %%" % (percentage_offers))<jupyter_output>Percentage of customer transaction events: 45.3 %
Percentage of customer offer events: 54.7 %<jupyter_text>Transcript Data  - Lets change the person column name to customer_id  - change time in hours to days  - change time column name to days<jupyter_code>transcript.rename(columns={"person": "customer_id"}, inplace=True)
transcript.head()
# Remove customer id's that are not in the customer profile DataFrame
select_data = transcript['customer_id'].isin(profile['customer_id'])
transcript = transcript[select_data]

percent_removed = 100 * (1 - select_data.sum() / select_data.shape[0])
print("Percentage of transactions removed: %.2f %%" % percent_removed)
transcript['time'] /= 24.0
transcript.rename(columns={"time": "days"}, inplace=True)
transcript.head()
# Select customer offers
pattern_obj = re.compile('^offer (?:received|viewed|completed)')
h_is_offer = lambda elem: pattern_obj.match(elem) != None
is_offer = transcript['event'].apply(h_is_offer)

offers_data = transcript[is_offer].copy()
offers_data = offers_data.reset_index(drop=True)
cols = ['offer_id', 'customer_id', 'days']
offers_data['offer_id'] = offers_data['value'].apply(lambda elem: list(elem.values())[0])
offers_data.head()<jupyter_output><empty_output><jupyter_text>Perform the following  - Offer received column value to be changed to received  - Offer viewed column to be changed to viewed<jupyter_code># Transform a column that describes a customer offer event
# https://docs.python.org/3/library/re.html
pattern_obj = re.compile('^offer ([a-z]+$)')
h_transform = lambda elem: pattern_obj.match(elem).groups(1)[0]
offers_data['event'] = offers_data['event'].apply(h_transform)
offers_data.groupby('event').size()
# One hot encode customer offer events
event_df = pd.get_dummies(offers_data['event'])
event_df.head()
cols.extend(event_df.columns.values)
cols
# Create a DataFrame that describes customer offer events
offers_data = pd.concat([offers_data, event_df], axis=1)
offers_data.drop(columns=['event', 'value'])
offers_data = offers_data[cols]
offers_data.head()<jupyter_output><empty_output><jupyter_text>Transactions performed by usersThe actual transactions performed by the users are also available in the transcript df.  - Lets create a new data frame to capture these records.   - We will create a new column called "amount" to capture the actual amount for each transaction from the "value" column.  - Drop duplicate columns from the data frame (value, event)<jupyter_code># Select customer transaction events
transactions = transcript[is_offer == False]
transactions = transactions.reset_index(drop=True)
transactions.head()
# Transform customer transaction event values
transactions['amount'] = transactions['value'].apply(lambda elem: list(elem.values())[0])
transactions.head()
# Create a DataFrame that describes customer transactions
transactions = transactions.drop(columns=['event', 'value'])
cols = ['customer_id', 'days', 'amount']
transactions = transactions[cols]
transactions.head()<jupyter_output><empty_output><jupyter_text>**Before we head further lets take a look at the data frames we have**.  - **profile** : DataFrame that contains demographic data for each customer  - **offer_data** : Contains offer ids and meta data (duration, difficulty, type)  - **transactions** : DataFrame that describes customer transactions  - **portfolio** : DataFrame that describes customer offer data **For each customer, we will do the following:**  - Select a customer profile  - Select offer data for the specific customer  - Select transactions for the specific customer  - Identify when a customer receives, views, and completes an offer  - Iterate through each offer that the selected customer receives      - Get the time period (start date, end date) for the offer      - select customer transactions that fall within the valid offer time window      - Identify when a customer completes an offer      - Identify when a customer views an offer      - Determine whether the current offer was successful. For an offer to be successful a customer has to view and complete it      - Select customer transactions that occurred within the current offer valid time period      Once all customer transactions have been evaluated, create a new pandas DataFrame.<jupyter_code>def combine_records(customer_id, portfolio, profile, offer_data, transactions):
    """ 
    Creates a list of dictionaries that describes the effectiveness of
    offers to a specific customer
    INPUT:
        customer_id: Customer Id
        portfolio: DataFrame containing offer ids and meta data about each offer (duration, type, etc.)
        offer_data: DataFrame that describes customer offer data
        transactions: DataFrame that describes customer transactions
    
    OUTPUT:
        rows: customer information, offer infomration and whether the offer was successful
    """
    
    # Select a customer's profile from the profile data frame
    customer = profile[profile['customer_id'] == customer_id]

    # Select offer data for a specific customer
    valid_offer_data = offer_data['customer_id'] == customer_id
    customer_offers = offer_data[valid_offer_data]
    customer_offers = customer_offers.drop(columns='customer_id')
    customer_offers = customer_offers.reset_index(drop=True)

    # Select transactions for a specific customer
    valid_transaction = transactions['customer_id'] == customer_id
    customer_transactions = transactions[valid_transaction]
    customer_transactions = customer_transactions.drop(columns='customer_id')
    customer_transactions = customer_transactions.reset_index(drop=True)    

    # select received offer data
    # drop event type columns
    received = customer_offers[customer_offers['received'] == 1]
    received = received.drop(columns=['completed','received','viewed'])
    received = received.reset_index(drop=True)

    # select viewed offer data
    # drop event type columns
    viewed = customer_offers[customer_offers['viewed'] == 1]
    viewed = viewed.drop(columns=['completed','received','viewed'])
    viewed = viewed.reset_index(drop=True)

    # select completed offer data
    # drop event type columns
    completed = customer_offers[customer_offers['completed'] == 1]
    completed = completed.drop(columns=['completed','received','viewed'])
    completed = completed.reset_index(drop=True)    
    
    #Process each offer data
    rows = []
    for idx in range(received.shape[0]):

        # Initialize the current offer id
        cur_offer_id = received.iloc[idx]['offer_id']

        # Get the current offer row and the days
        cur_offer = portfolio.loc[portfolio['offer_id'] == cur_offer_id]
        durationdays = cur_offer['duration'].values[0]

        # Get the time period when an offer is valid, (start time and end time)
        cur_offer_startime = received.iloc[idx]['days']
        cur_offer_endtime = received.iloc[idx]['days'] + durationdays

        # Initialize an array to hold customer transcations that fall within the valid offer time period
        # https://docs.scipy.org/doc/numpy/reference/generated/numpy.logical_and.html
        valid_transaction = np.logical_and(customer_transactions['days'] >= \
                           cur_offer_startime,
                           customer_transactions['days'] <=
                           cur_offer_endtime)

        # Initialize a array with information about when a customer completes an offer
        select_offer_completed = np.logical_and(completed['days'] >= cur_offer_startime, \
                           completed['days'] <= cur_offer_endtime)

        # Initialize a array with information about when a customer views an offer
        select_offer_viewed = np.logical_and(viewed['days'] >= cur_offer_startime, \
                           viewed['days'] <= cur_offer_endtime)

        # Determine whether the current offer was successful
        cur_offer_successful = select_offer_completed.sum() > 0 and select_offer_viewed.sum() > 0

        # Select customer transcations that occurred within the current offer's valid time period
        cur_offer_transactions = customer_transactions[valid_transaction]

        # Initialize a dictionary with the current customer offer
        cur_row = {'offer_id': cur_offer_id,
                   'customer_id': customer_id,
                   'time': cur_offer_startime,
                   'offer_successful': int(cur_offer_successful),
                   'total_amount': cur_offer_transactions['amount'].sum()}
        
        cur_row.update(cur_offer.iloc[0,0:].to_dict())
        cur_row.update(customer.iloc[0,0:].to_dict())
        rows.append(cur_row)
        
    return rows    
clean_data = []
customerid_list = offer_data['customer_id'].unique()
#customerid_list
# lets look at one record to see how it comes out
data = combine_records(customerid_list[6],portfolio,profile,offers_data,transactions)
clean_df = pd.DataFrame(data)
clean_df[['age','offer_id','customer_id','total_amount','agerange','gender','offer_successful','duration','time','offer_type','income']]
clean_data = []
customerid_list = offer_data['customer_id'].unique()

for idx in range(len(customerid_list)):
    clean_data.extend(combine_records(customerid_list[idx],portfolio,profile,offers_data,transactions))
    
clean_data = pd.DataFrame(clean_data)

clean_data = clean_data.sort_values('time')
clean_data.reset_index(drop=True)

clean_data.head()
clean_data.to_csv('cleaned_data.csv',index=False)<jupyter_output><empty_output><jupyter_text>**Now that we have the combined data frame, lets do some more EDA**<jupyter_code>df = pd.read_csv('cleaned_data.csv')
df.columns
df1 = df.copy().drop(columns=['agerange','became_member_on', 'bogo', 'channels', 'customer_id','gender','membershipstart',
                             'offer_id','offer_successful', 'offer_type','time','total_amount'])
# lets plot the correlation of different numerical features to the targe (offer_successful)
df1.corrwith(df.offer_successful).plot.bar(figsize=(20,10),
                        title='Correlation with Response Variable',
                        fontsize=15, rot=45, grid=True);<jupyter_output><empty_output><jupyter_text>From the above chart we find that:  - difficulty, duration, discount, reard, income has positive corelation to the target  lets also plot histogram for some of the feature columns. I am gonna include difficulty, duration, discount, reard, income, age, gender_value for analysis<jupyter_code># create a new data frame to plot the histograms
df2 = df1.copy().drop(columns=['10 - 20', '100 - 110', '20 - 30', '30 - 40', '40 - 50', '50 - 60',
       '60 - 70', '70 - 80', '80 - 90', '90 - 100','2013','2014','2015','2016','2017','2018','web',
                               'social','mobile','email','informational'])

# lets also plot histogram for some of the feature columns.
fig=plt.figure(figsize=(10, 30), facecolor='w', edgecolor='k')
fig.suptitle("Histogram of Numerical Columns", fontsize=20)
for i in range(1, df2.shape[1] + 1):
    plt.subplot(5, 2, i)
    f = plt.gca()
    f.set_title(df2.columns.values[i - 1])
    
    # get the binsize
    vals = np.size(df2.iloc[:,i - 1].unique())
    plt.hist(df2.iloc[:, i -1 ], bins=vals, color='#3F5D7D')<jupyter_output><empty_output><jupyter_text>**Some of the inferences from the histogram**  - seems to be more male customers compared to female customers  - rewards between 2 - 7 seems to be more popular  - offers with duration of 7 days seems to be more  - offers with diffficulty 5 - 10 seems to be more  - there is a spike in age around 60, otherwise it is evenly distributed  **Now lets look at successful offers (where offer_successful = 1)**<jupyter_code># create a new data frame with successful records
df_success = df[ df.offer_successful == 1 ]
df_success.shape
# get the count of unique customers to compare the male vs female ratio
custs_df = df [['customer_id','gender']]
unique_custs = custs_df.drop_duplicates()
unique_custs.groupby('gender')['customer_id'].count()/unique_custs.shape[0]*100
# lets plot the total amount spend by each gender
df_success.groupby(['gender'],as_index=False)['total_amount'].sum().plot(x="gender", y=["total_amount"], 
                    kind="bar", figsize=(8,6), 
                    title="Total Amount spend vs Gender");<jupyter_output><empty_output><jupyter_text>**Nice observation :**  - 42% female customers  - 58% male customers  - The total amount spend by female customers are more than male customersDoes that women they drink more coffee / tea? Not necessarily. Because starbucks has other offerings as well.So they could be spending more money on those. ***However it seems on average women spend more money at Starbucks.***<jupyter_code># Lets look at the various age group for the female customers and how much they spend
femme_df = df_success[ df_success.gender == 'F']
femme_total_df = femme_df.groupby(['agerange'],as_index=False)['total_amount'].sum()
femme_total_df.plot(x="agerange", y=["total_amount"], 
                    kind="bar", figsize=(8,6), 
                    title="Total Amount Spend vs Age Range of Female Customers", 
                    grid=True);
# Lets look at the various age group for the male customers and how much they spend
homme_df = df_success[ df_success.gender == 'M']
homme_total_df = homme_df.groupby(['agerange'],as_index=False)['total_amount'].sum()
homme_total_df.plot(x="agerange", y=["total_amount"], 
                    kind="bar", figsize=(8,6), 
                    title="Total Amount Spend vs Age Range of Male Customers", 
                    grid=True);<jupyter_output><empty_output><jupyter_text>**Seems folks belonging to 50 - 60 age range in both groups (female , male) spend more money. 40 - 50 and 60 - 70 age groups also spend more money compared to other age groups*****If you are wondering what femme and homme is, those are french equivalent words for woman and man.***  Duration, Discount and RewardLets look the duration, discount and reward and how many offers were completed for each<jupyter_code># group by duration and plot
df_success.groupby('duration')['time'].count().plot(x='duration',y='count',
                                                    kind="bar",title="Successful Offers Completed vs Duration");
# group by difficulty and plot
df_success.groupby('difficulty')['time'].count().plot(x='difficulty',y='count',
                                                    kind="bar",title="Successful Offers Completed vs Duration");
# group by reward and plot
df_success.groupby('reward')['time'].count().plot(x='reward',y='count',
                                                    kind="bar",title="Successful Offers Completed vs Duration");<jupyter_output><empty_output><jupyter_text>**Duration and Successful Offers :**  - Offers with duration of 5, 7 and 10 has more successful offers  - the highest being offers with duration of 7 days**Difficulty and Successful Offers :**  - Offers with duration of 5, 7 and 10 has more successful offers  - the highest being offers with difficulty of 10 days  - this could be becuase more the difficulty more the duration and more the reward**Reward and Successful Offers :**  - Offers with reward of 2, 5 and 10 has more successful offers  - the highest being offers with a reward of 5  Train, Test and Evaluate ModelWe need to   - drop the below columns    - customer_id, offer_id, became_member_on, gender, age, agerange, channels, offert_type, time    membershipstart, total_amount  - perform a min max feature scaling on the below columns:    - difficulty, duration, reward, income  - create train and test data set  - train the model  - evauluate the results  - use grid search to find the best parameters  - improve the model<jupyter_code>dataset = df.copy().drop( columns = ['customer_id','offer_id','became_member_on','gender',\
                                     'channels','age','agerange','offer_type','membershipstart','total_amount','time'])
dataset.columns
# https://scikit-learn.org/stable/modules/generated/sklearn.preprocessing.MinMaxScaler.html
scaler = MinMaxScaler()
dataset[['difficulty', 'duration', 'reward', 'income']] = \
                        scaler.fit_transform(dataset[['difficulty', 'duration', 'reward', 'income']])
# Lets check out the data and it looks good.
dataset[['difficulty', 'duration', 'reward', 'income']].head()
# lets check out the columns
dataset.columns
# Lets create test and train data
X = dataset.copy().drop(columns=['offer_successful'])
y = dataset['offer_successful']
X.head()
y.head()<jupyter_output><empty_output><jupyter_text>Split the data into training and testing data<jupyter_code># https://scikit-learn.org/stable/modules/generated/sklearn.model_selection.train_test_split.html
random_state = 123456789

(X_train,
 X_test,
 y_train,
 y_test) = train_test_split(X,
                            y,
                            test_size=0.2,
                            random_state=random_state)

print("X_Train : ",X_train.shape)
print("X_Test : ",X_test.shape)
print("y_Train : ",y_train.shape)
print("y_Test : ",y_test.shape)<jupyter_output>X_Train :  (52468, 28)
X_Test :  (13117, 28)
y_Train :  (52468,)
y_Test :  (13117,)<jupyter_text>SVM Model[sklearn SVM](https://scikit-learn.org/stable/modules/generated/sklearn.svm.SVC.html)[Support Vector Machine](https://www.analyticsvidhya.com/blog/2017/09/understaing-support-vector-machine-example-code/)Common Parameters that can be changed to tune performance of the model  - C (the higher the C value the more time it takes to train the model)  - gamma (the higher the gamma value the more time it takes to train the model)  - kernel (‘linear’, ‘poly’, ‘rbf’, ‘sigmoid’, ‘precomputed’ or a callable). default is 'rbf'<jupyter_code># Train using default parameters (C, gamma and use the default kernel "rbf")
svc_model = SVC(random_state=random_state)
svc_model.fit(X_train, y_train)
#predict the accuracy
y_pred = svc_model.predict(X_test)
print(accuracy_score(y_test,y_pred))
cm = confusion_matrix( y_test, y_pred)
sns.heatmap(cm, annot=True);
print(classification_report(y_test,y_pred))<jupyter_output>precision    recall  f1-score   support

          0       0.74      0.73      0.74      6937
          1       0.70      0.71      0.71      6180

avg / total       0.72      0.72      0.72     13117<jupyter_text>Even for such a small data set SVM model took lot of time to train the model. We could try tuning the C and gamma variables and try to fine tune the model. The higher the C and gamma values, the more it is going to take to train the model.So let's try out few other algorithms like Logistic Regression and Random Forest Classifier.  Logistic Regression model[sklearn Logistic Regression](https://scikit-learn.org/stable/modules/generated/sklearn.linear_model.LogisticRegression.html)We will try the following parameters to tune the model:  - penalty: Used to specify the norm used in the penalization.  - solver: We will choose 'liblinear' as the sovler since this is not a big data set  - C: Inverse of regularization strength; must be a positive float.  - Perform random search of model hyperparameter space    - Accuracy :  0.718    - F1-score :  0.712<jupyter_code>lr_model = LogisticRegression(random_state=random_state,
                            solver='liblinear')
lr_model.fit(X_train, y_train)
#predict the accuracy
y_lr_pred = lr_model.predict(X_test)
print(accuracy_score(y_test,y_lr_pred))
cm = confusion_matrix( y_test, y_lr_pred)
sns.heatmap(cm, annot=True);
print(classification_report(y_test,y_lr_pred))<jupyter_output>precision    recall  f1-score   support

          0       0.76      0.71      0.73      6937
          1       0.69      0.75      0.72      6180

avg / total       0.73      0.73      0.73     13117<jupyter_text>Tune Parameters to improve model performance  - We will use Grid Search to find the relatiely best values for the different parameters  - We will use the following    - penalty    - solver    - C    - max_iter<jupyter_code>lr_param_grid = {'penalty':['l1','l2'] ,'C':[0.1, 1, 10,100], 'solver':['liblinear'],'max_iter':[100,200,300] }
lr_grid = GridSearchCV(LogisticRegression(random_state= random_state), lr_param_grid, refit = True, verbose = 4)
lr_grid.fit(X_train, y_train)
lr_grid.best_params_
lr_opt_y_pred = lr_grid.predict(X_test)

print(accuracy_score(y_test,lr_opt_y_pred))

lr_cm1 = confusion_matrix( y_test, lr_opt_y_pred)
sns.heatmap(lr_cm1, annot=True);
print(classification_report(y_test,lr_opt_y_pred))<jupyter_output>precision    recall  f1-score   support

          0       0.76      0.70      0.73      6937
          1       0.69      0.75      0.72      6180

avg / total       0.73      0.73      0.73     13117<jupyter_text>**Even after tuning the selected parameters, we dont see much of improvement in the precision.**. There are other parameters that we can try.  Construct Random Forest Model[sklearn Random Forest Classifier](https://scikit-learn.org/stable/modules/generated/sklearn.ensemble.RandomForestClassifier.html)[Random Forest Blog](https://medium.com/machine-learning-101/chapter-5-random-forest-classifier-56dc7425c3e1)We will try the below parameters:  - n_estimators : The number of trees in the forest.  - criterion : The function to measure the quality of a split.  - min_samples_split : The minimum number of samples required to split an internal node  - min_samples_leaf : The minimum number of samples required to be at a leaf node.    - Perform random search of model hyperparameter space    - Accuracy : 0.742    - F1-score : 0.735<jupyter_code>rf_model = RandomForestClassifier(random_state=random_state)
rf_model.fit(X_train, y_train)
y_rf_pred = rf_model.predict(X_test)
print(accuracy_score(y_test,y_rf_pred))
rf_cm = confusion_matrix( y_test, y_rf_pred)
sns.heatmap(rf_cm, annot=True);
print(classification_report(y_test,y_rf_pred))
rf_clf = RandomForestClassifier(random_state=random_state)

# Number of trees in random forest
n_estimators = [100, 200, 300]

# Maximum number of levels in tree
max_depth = [int(x) for x in range(5, 11)]

# Minimum number of samples required to split a node
min_samples_split = [2, 4, 6]

# Minimum number of samples required at each leaf node
min_samples_leaf = [ 2, 4]

# Create the random grid
random_grid = {'n_estimators': n_estimators,
               'max_depth': max_depth,
               'min_samples_split': min_samples_split,
               'min_samples_leaf': min_samples_leaf}

randf = GridSearchCV(estimator = rf_clf,
                               param_grid  = random_grid,
                               cv = 3,
                               verbose=2,
                               n_jobs = 3)

randf.fit(X_train, y_train)
y_rf_opt_pred = randf.best_estimator_.predict(X_test)
print(accuracy_score(y_test,y_rf_opt_pred))
rf_cm1 = confusion_matrix( y_test, y_rf_opt_pred)
sns.heatmap(rf_cm1, annot=True);
print(classification_report(y_test, y_rf_opt_pred))<jupyter_output>precision    recall  f1-score   support

          0       0.77      0.72      0.74      6937
          1       0.70      0.76      0.73      6180

avg / total       0.74      0.74      0.74     13117<jupyter_text>Random Forest Classifier  - We use grid search to find the best parameter values  - there was 5% improvement in the precision, which is good  - GridSearch took more time to evaluate the optimal parameter combinations  - As an improvement we could user RandomSearchCV instead of GridSearchCV  Random Forest Classifier performs feature importance and selection internally while constructing the model. The model can also report on the variable importance determined during the model preparation process. Now lets check out the Feature importance.<jupyter_code>variable_names = X_train.columns
variable_names<jupyter_output><empty_output><jupyter_text>Plot Estimated Feature ImportanceThe top features for this problem are:  - Offer difficulty (how much money a customer must spend to complete an offer. this is dependent on reward and duration as well. Even if the difficulty is more, if the offer is valid for a longer time period and if the reward is higher, there is a higher chance that offer will be completed by customers)  - Offer reward (more the reward the better the probability of customer completing an offer)  - Offer duration (more the duration the better the probability of customer completing an offer  - Customer income   - 2018 (Year when the user created the account)  - social (whether the offers area available through social media. this make sense as more and more people are using social media like instagra, twitter, FB, Snapchat etc)  - informational (the offer was informational message. This could mean that if it is informational people generally may not complete the offer. and could be a major feature in identifying what type of offer is more effective)  **I am surprised that bogo and discount are not in the top 5 features.**<jupyter_code>relative_importance = randf.best_estimator_.feature_importances_
relative_importance = relative_importance / np.sum(relative_importance)

feature_importance =\
    pd.DataFrame(list(zip(variable_names,
                          relative_importance)),
                 columns=['feature', 'relativeimportance'])

feature_importance = feature_importance.sort_values('relativeimportance',
                                                    ascending=False)

feature_importance = feature_importance.reset_index(drop=True)

palette = sns.color_palette("Blues_r", feature_importance.shape[0])

plt.figure(figsize=(8, 8))
sns.barplot(x='relativeimportance',
            y='feature',
            data=feature_importance,
            palette=palette)
plt.xlabel('Relative Importance')
plt.ylabel('Feature')
plt.title('Random Forest Estimated Feature Importance')
feature_importance.head(n=10)<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.09455900149491019, 'fraction_numerical': 0.014830458201836604, 'mean_word_length': 4.27459324155194, 'pattern_counts': {'":': 4, '<': 102, '<?xml version=': 0, '>': 103, 'https://': 10, 'lorem ipsum': 0, 'www.': 1, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 18, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '26', 'dir': 'jupyter-structured-clean-dedup', 'id': '184436', 'license': 'MIT', 'n_tokens_mistral': 12533, 'n_tokens_neox': 11809, 'n_words': 4215, 'path': 'Starbucks_Project.ipynb', 'repo_name': 'raj-arun/dsnd_starbucks_captstone_project', 'size': '42143'}