<jupyter_start><jupyter_text>Starbucks Capstone Challenge IntroductionThis data set contains simulated data that mimics customer behavior on the Starbucks rewards mobile app. Once every few days, Starbucks sends out an offer to users of the mobile app. An offer can be merely an advertisement for a drink or an actual offer such as a discount or BOGO (buy one get one free). Some users might not receive any offer during certain weeks. Not all users receive the same offer, and that is the challenge to solve with this data set.Your task is to combine transaction, demographic and offer data to determine which demographic groups respond best to which offer type. This data set is a simplified version of the real Starbucks app because the underlying simulator only has one product whereas Starbucks actually sells dozens of products.Every offer has a validity period before the offer expires. As an example, a BOGO offer might be valid for only 5 days. You'll see in the data set that informational offers have a validity period even though these ads are merely providing information about a product; for example, if an informational offer has 7 days of validity, you can assume the customer is feeling the influence of the offer for 7 days after receiving the advertisement.You'll be given transactional data showing user purchases made on the app including the timestamp of purchase and the amount of money spent on a purchase. This transactional data also has a record for each offer that a user receives as well as a record for when a user actually views the offer. There are also records for when a user completes an offer. Keep in mind as well that someone using the app might make a purchase through the app without having received an offer or seen an offer. ExampleTo give an example, a user could receive a discount offer buy 10 dollars get 2 off on Monday. The offer is valid for 10 days from receipt. If the customer accumulates at least 10 dollars in purchases during the validity period, the customer completes the offer.However, there are a few things to watch out for in this data set. Customers do not opt into the offers that they receive; in other words, a user can receive an offer, never actually view the offer, and still complete the offer. For example, a user might receive the "buy 10 dollars get 2 dollars off offer", but the user never opens the offer during the 10 day validity period. The customer spends 15 dollars during those ten days. There will be an offer completion record in the data set; however, the customer was not influenced by the offer because the customer never viewed the offer. CleaningThis makes data cleaning especially important and tricky.You'll also want to take into account that some demographic groups will make purchases even if they don't receive an offer. From a business perspective, if a customer is going to make a 10 dollar purchase without an offer anyway, you wouldn't want to send a buy 10 dollars get 2 dollars off offer. You'll want to try to assess what a certain demographic group will buy when not receiving any offers. Final AdviceBecause this is a capstone project, you are free to analyze the data any way you see fit. For example, you could build a machine learning model that predicts how much someone will spend based on demographics and offer type. Or you could build a model that predicts whether or not someone will respond to an offer. Or, you don't need to build a machine learning model at all. You could develop a set of heuristics that determine what offer you should send to each customer (i.e., 75 percent of women customers who were 35 years old responded to offer A vs 40 percent from the same demographic to offer B, so send offer A).  Data SetsThe data is contained in three files:* portfolio.json - containing offer ids and meta data about each offer (duration, type, etc.)* profile.json - demographic data for each customer* transcript.json - records for transactions, offers received, offers viewed, and offers completedHere is the schema and explanation of each variable in the files:**portfolio.json*** id (string) - offer id* offer_type (string) - type of offer ie BOGO, discount, informational* difficulty (int) - minimum required spend to complete an offer* reward (int) - reward given for completing an offer* duration (int) - time for offer to be open, in days* channels (list of strings)**profile.json*** age (int) - age of the customer * became_member_on (int) - date when customer created an app account* gender (str) - gender of the customer (note some entries contain 'O' for other rather than M or F)* id (str) - customer id* income (float) - customer's income**transcript.json*** event (str) - record description (ie transaction, offer received, offer viewed, etc.)* person (str) - customer id* time (int) - time in hours since start of test. The data begins at time t=0* value - (dict of strings) - either an offer id or transaction amount depending on the record**Note:** If you are using the workspace, you will need to go to the terminal and run the command `conda update pandas` before reading in the files. This is because the version of pandas in the workspace cannot read in the transcript.json file correctly, but the newest version of pandas can. You can access the termnal from the orange icon in the top left of this notebook.  You can see how to access the terminal and how the install works using the two images below.  First you need to access the terminal:Then you will want to run the above command:Finally, when you enter back into the notebook (use the jupyter icon again), you should be able to run the below cell without any errors.  Udacity Starbucks Capstone Project- Student provides a high-level overview of the project. Background information such as the problem domain, the project origin, and related data sets or input data is provided.- The problem which needs to be solved is clearly defined. A strategy for solving the problem, including discussion of the expected solution, has been made.- Metrics used to measure performance of a model or result are clearly defined. Metrics are justified based on the characteristics of the problem.----  Install dependencies<jupyter_code>!pip install -r requirements.txt
!pip install ipython-autotime
# load execution time measurement tools 
%load_ext autotime<jupyter_output>Requirement already satisfied: ipython-autotime in c:\users\victo\anaconda3\envs\starbucks-capstone\lib\site-packages (0.3.1)
Requirement already satisfied: ipython in c:\users\victo\anaconda3\envs\starbucks-capstone\lib\site-packages (from ipython-autotime) (7.20.0)
Requirement already satisfied: pickleshare in c:\users\victo\anaconda3\envs\starbucks-capstone\lib\site-packages (from ipython->ipython-autotime) (0.7.5)
Requirement already satisfied: setuptools>=18.5 in c:\users\victo\anaconda3\envs\starbucks-capstone\lib\site-packages (from ipython->ipython-autotime) (52.0.0.post20210125)
Requirement already satisfied: traitlets>=4.2 in c:\users\victo\anaconda3\envs\starbucks-capstone\lib\site-packages (from ipython->ipython-autotime) (5.0.5)
Requirement already satisfied: colorama in c:\users\victo\anaconda3\envs\starbucks-capstone\lib\site-packages (from ipython->ipython-autotime) (0.4.4)
Requirement already satisfied: jedi>=0.16 in c:\users\victo\anaconda3\envs\starbucks-capstone\lib[...]<jupyter_text>Import dependencies<jupyter_code>import pandas as pd
import numpy as np
import math
import json
import matplotlib.pyplot as plt
import seaborn as sns

# module for prediction
from sklearn.preprocessing import LabelEncoder
from sklearn.model_selection import train_test_split
from sklearn.svm import LinearSVC
from sklearn.neighbors import KNeighborsClassifier
from sklearn.ensemble import AdaBoostClassifier
from sklearn.metrics import accuracy_score
from sklearn.metrics import f1_score

# Hyperparameterization
from sklearn.model_selection import GridSearchCV

# Model Evaluation
from numpy import mean
from numpy import std
from sklearn.model_selection import KFold
from sklearn.model_selection import cross_val_score<jupyter_output>time: 16 ms (started: 2021-03-22 21:01:11 +01:00)<jupyter_text>Loading dataset<jupyter_code>%matplotlib inline

# read in the json files
portfolio = pd.read_json('data/portfolio.json', orient='records', lines=True)
profile = pd.read_json('data/profile.json', orient='records', lines=True)
transcript = pd.read_json('data/transcript.json', orient='records', lines=True)<jupyter_output>time: 3.3 s (started: 2021-03-22 20:58:42 +01:00)<jupyter_text>Helper method<jupyter_code>def print_isna_proportion(ser: pd.Series):
    """
    Printing the proportion of isna value from a panda series.

    :param ser: panda series to be calculated for the proportion
    :return: None
    """
    print(f'Amount of isna value : {ser.isna().sum()} and its proportion : {ser.isna().sum() / len(ser)}')
    
print_isna_proportion(profile['gender'])
def print_metrics(y_test: pd.DataFrame, y_pred: pd.DataFrame):
    """
    Printing the accuracy and f1 score for the given dataframes.
    :param y_test: panda test dataframe
    :param y_pred: panda prediction dataframe
    """
    print("Accuracy for the model : ", accuracy_score(y_test, y_pred, normalize = True))
    print("F1-score for the model : ", f1_score(y_test, y_pred, average='weighted'))
def plot_distribution(df: pd.DataFrame, column: str, target_column: str = 'offer_successful'):
    """
    Plotting a distribution plot of a given dataframe based on given column and target_column
    :param df: panda dataframe to be ploted
    :param column: column to be plotted (x-axis)
    :param target_column: target column to be plotted (y-axis)

    """
    sns.displot(merged_dataframe, x=column, hue=target_column, multiple="dodge").set(title=f'Distribution by {column}')<jupyter_output>time: 15 ms (started: 2021-03-22 00:24:07 +01:00)<jupyter_text>1. Business Understanding Following questions will be examined and answered:- How much starbucks customer completed a reward ?- What kind of offer type most customer view and completed?- Could a prediction be made, either a customer will take a reward or not? So a targeted marketing could be done!  2. Data Understanding  2.1. Portfolio Dataset * id (string) - offer id* offer_type (string) - type of offer ie BOGO, discount, informational* difficulty (int) - minimum required spend to complete an offer* reward (int) - reward given for completing an offer* duration (int) - time for offer to be open, in days* channels (list of strings)<jupyter_code>portfolio.head()
portfolio.shape
portfolio.dtypes<jupyter_output><empty_output><jupyter_text>2.1.1 `reward` feature<jupyter_code>portfolio['reward'].value_counts()<jupyter_output><empty_output><jupyter_text>2.1.2 `channels` feature<jupyter_code>portfolio['channels']<jupyter_output><empty_output><jupyter_text>2.1.3 `difficulty` feature<jupyter_code>portfolio['difficulty'].value_counts()<jupyter_output><empty_output><jupyter_text>2.1.4 `duration` feature<jupyter_code>portfolio['duration'].value_counts()<jupyter_output><empty_output><jupyter_text>2.1.5 `over_type` feature<jupyter_code>portfolio['offer_type'].value_counts()<jupyter_output><empty_output><jupyter_text>2.1.5 `id` feature<jupyter_code>portfolio['id'].nunique()<jupyter_output><empty_output><jupyter_text>=> There are no duplicate value (by id)  **Summary**- has **10** entries and no duplicated `id`- `channels` has list of a string => has a potential to be splitted into multiple column - `offer type` has only 3 possible value (`discount`, `bogo`, `informational`) => One Hot Encoding- `reward`, `diffculty`, `duration` contains numerical data  2.2. Profile Dataset * age (int) - age of the customer * became_member_on (int) - date when customer created an app account* gender (str) - gender of the customer (note some entries contain 'O' for other rather than M or F)* id (str) - customer id* income (float) - customer's income<jupyter_code>profile.head()
profile.shape
profile.dtypes
profile.info()<jupyter_output><class 'pandas.core.frame.DataFrame'>
RangeIndex: 17000 entries, 0 to 16999
Data columns (total 5 columns):
 #   Column            Non-Null Count  Dtype  
---  ------            --------------  -----  
 0   gender            14825 non-null  object 
 1   age               17000 non-null  int64  
 2   id                17000 non-null  object 
 3   became_member_on  17000 non-null  int64  
 4   income            14825 non-null  float64
dtypes: float64(1), int64(2), object(2)
memory usage: 664.2+ KB
time: 15 ms (started: 2021-03-22 00:24:07 +01:00)<jupyter_text>=> Check gender and income for missing value  2.2.1 `gender` feature * gender (str) - gender of the customer (note some entries contain 'O' for other rather than M or F)<jupyter_code>print_isna_proportion(profile['gender'])
profile[(profile['gender'] != 'F') & (profile['gender'] != 'M') & (profile['gender'].notna())]<jupyter_output><empty_output><jupyter_text>=> in `gender` feature, there are 212 data containing **O**  2.2.2 `age` feature => From the `.head()` above, age 118 looks like strange number for a customer ageprofile['age'].hist()<jupyter_code>profile['age'].hist()
profile[profile['age'] >= 118].count()
profile[profile['age'] >= 118].income.isna().count()<jupyter_output><empty_output><jupyter_text>age > 100 will be an outlier => we will remove it<jupyter_code>profile['age'].min() # minimum age in the dataset<jupyter_output><empty_output><jupyter_text>2.2.3 `id` feature<jupyter_code># count unique id
profile['id'].nunique()<jupyter_output><empty_output><jupyter_text>=> there are no dupplicate id  2.2.4 `became_member_on` feature<jupyter_code>profile['became_member_on'].dtype<jupyter_output><empty_output><jupyter_text>=> the `became_member_on` is a int-value. It need to be converted to a *datetime*  2.2.5 `income` feature<jupyter_code>profile['income'].dtype
print_isna_proportion(profile['income'])<jupyter_output>Amount of isna value : 2175 and its proportion : 0.12794117647058822
time: 0 ns (started: 2021-03-22 00:24:08 +01:00)<jupyter_text>Profile `income` outliers<jupyter_code>older_than_100 = profile[profile['age']>100]
older_than_100.head()
older_than_100.shape[0]
older_than_100[pd.notnull(older_than_100["income"])]<jupyter_output><empty_output><jupyter_text>Summary- There are **17000** entries, but with **2180** entries with `age` higher than 100 years old. => `age` need to be cleaned- `gender`,  `income` has null values => need to be cleaned- `became_member_on` has only string value => convert to DateTime/Timestamp  2.3 Transcript dataset * event (str) - record description (ie transaction, offer received, offer viewed, etc.)* person (str) - customer id* time (int) - time in hours since start of test. The data begins at time t=0* value - (dict of strings) - either an offer id or transaction amount depending on the record<jupyter_code>transcript.head()
transcript.shape
transcript.dtypes
transcript.info()<jupyter_output><class 'pandas.core.frame.DataFrame'>
RangeIndex: 306534 entries, 0 to 306533
Data columns (total 4 columns):
 #   Column  Non-Null Count   Dtype 
---  ------  --------------   ----- 
 0   person  306534 non-null  object
 1   event   306534 non-null  object
 2   value   306534 non-null  object
 3   time    306534 non-null  int64 
dtypes: int64(1), object(3)
memory usage: 9.4+ MB
time: 62 ms (started: 2021-03-22 00:24:08 +01:00)<jupyter_text>2.3.1 `person` feature<jupyter_code>transcript['person'].nunique()<jupyter_output><empty_output><jupyter_text>=> - the amount equals the profile amounts => each profile has at least 1 transaction- if some profiles are dropped, the corresponding transaction need to be deleted too  2.3.2 `event` feature<jupyter_code>transcript['event'].value_counts()<jupyter_output><empty_output><jupyter_text>2.3.3 `value` feature<jupyter_code>transcript.head()
transcript.tail()<jupyter_output><empty_output><jupyter_text>**=> it looks like that the value is dependent on the event type. => investigate further!**<jupyter_code>event_types = list(transcript['event'].value_counts().index)
event_types
for event_type in event_types:
    display(transcript[transcript['event'] == str(event_type)].head(5))<jupyter_output><empty_output><jupyter_text>The transaction with the eventtype `transaction` need to be deleted, because in our case we will not use the accumulated amount of money a customer used.  2.3.4 `time` feature<jupyter_code>transcript['time'].nunique()<jupyter_output><empty_output><jupyter_text>Summary- cleaning the `transcript` dataset based on deleted `profile` dataset- `value` depends on the `event` and contains either an `offer id` or `amount` depending on the record- drop the `transcript` data with event `transaction`- `value` feature contains dictionary => convert `value` to new colum(s)  Cleaning transcript `value`<jupyter_code>transcript['value'].dtypes
def has_multiple_key(x):
    if len(x.keys()) > 1: 
        return x
transcript['value'].apply(lambda x: has_multiple_key(x))<jupyter_output><empty_output><jupyter_text>3. Data cleaning & Data preparation In this section, following dataset will be cleaned:1. `Portfolio` dataset    - splitting `channels` feature into multiple column     - one-hot-encoding of the `offer type`  2. `Profile` dataset    - drop `age` which > 100 years old    - drop `gender` and `income` feature that has null values    - convert `became_member_on` to DateTime    3. `Transcript` dataset    - convert `value` to new colum(s)    - dropping `transcript` row based on deleted `profile` dataset    This will be done in 2 phases. - The first phases is to converting and cleaning the column and after that `profile` and `transcripts` will be joined. - The second phases will be more concentrated on converting the categorical data with **one-hot-encoding**.  3.1 Prepare `porfolio` dataset<jupyter_code>portfolio.head()<jupyter_output><empty_output><jupyter_text>3.1.1 splitting channels feature into multiple column<jupyter_code># list of all possible channel
all_channels = set()

# iterate each element of a list and append it to a set
def get_all_channels(x: list):
    for el in x:
        all_channels.add(el)
        
portfolio['channels'].apply(get_all_channels)

all_channels = list(all_channels)
all_channels
contains_channel = []
for portfolio_channels in portfolio['channels']:
    # iterate for each possible channel and append 1 if 
    # the channel inside the list of string from the `channels`
    # column otherwise append 0
    temp_list = [] # a container for each row of the `channel` column
    for channel in all_channels:
        temp_list.append( 1 if channel in portfolio_channels else 0)
    contains_channel.append(temp_list)

channels = pd.DataFrame(np.array(contains_channel),
                    columns=all_channels)
channels 
# merged splited channels dataframe to portfolio dataframe
cleaned_portfolio = pd.concat([portfolio, channels], axis=1)
cleaned_portfolio
# dropping unused `channels` column
cleaned_portfolio.drop(columns=['channels'], inplace=True)
cleaned_portfolio<jupyter_output><empty_output><jupyter_text>3.2 Prepare `profile` dataset- drop `gender` and `income` feature that has null values- drop `age` which > 100 years old- convert `became_member_on` to DateTime  3.2.1 drop `gender` and `income` feature that has null values From previous section (2.2) summary => there are only 2 features that have NAN-value. Therefore [`dropna()`](https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.dropna.html) function could be executed.<jupyter_code># copy `profile` dataset and drop the NAN-value
cleaned_profile = profile.copy()
cleaned_profile = cleaned_profile.dropna()
cleaned_profile
cleaned_profile.info()<jupyter_output><class 'pandas.core.frame.DataFrame'>
Int64Index: 14825 entries, 1 to 16999
Data columns (total 5 columns):
 #   Column            Non-Null Count  Dtype  
---  ------            --------------  -----  
 0   gender            14825 non-null  object 
 1   age               14825 non-null  int64  
 2   id                14825 non-null  object 
 3   became_member_on  14825 non-null  int64  
 4   income            14825 non-null  float64
dtypes: float64(1), int64(2), object(2)
memory usage: 694.9+ KB
time: 16 ms (started: 2021-03-22 00:24:09 +01:00)<jupyter_text>3.2.2 drop `age` which > 100 years old<jupyter_code>cleaned_profile['age'].hist()<jupyter_output><empty_output><jupyter_text>=> The age > 100 years old are in the same row with the empy `gender` and `income`. This could be because of the customer does not want to specify their personal information.The distribution of `age` feature are looks correct and therefore does not need to be cleaned any futher.  3.2.3 grouping `age` & drop `age`In this section, I will group the age based on folowing information: - adolescence (13-18 years)- adult (19-59 years) - senior adult (60 years and above)<jupyter_code>cleaned_profile['age_group'] = pd.cut(cleaned_profile['age'], bins=[18, 59, 60, np.inf], labels = ['adolesence','adult','senior'], right=False)
cleaned_profile.head()
cleaned_profile[cleaned_profile['age']==18]
cleaned_profile.drop(columns=['age'], inplace=True)
cleaned_profile.head()<jupyter_output><empty_output><jupyter_text>3.2.4 convert `became_member_on` to DateTime<jupyter_code>cleaned_profile['became_member_on'].head()
# convert `becaome_member_on` to datetime
cleaned_profile['became_member_on'] = pd.to_datetime(cleaned_profile['became_member_on'], format='%Y%m%d')
cleaned_profile.info()
cleaned_profile.head()
print(f'Previous shape : {profile.shape}. After cleaning: {cleaned_profile.shape}')
print(f"Min date: {cleaned_profile['became_member_on'].min()} & Maximum Date: {cleaned_profile['became_member_on'].max()}")<jupyter_output>Min date: 2013-07-29 00:00:00 & Maximum Date: 2018-07-26 00:00:00
time: 0 ns (started: 2021-03-22 00:24:09 +01:00)<jupyter_text>Summary<jupyter_code>cleaned_profile.describe()
cleaned_profile.info()<jupyter_output><class 'pandas.core.frame.DataFrame'>
Int64Index: 14825 entries, 1 to 16999
Data columns (total 5 columns):
 #   Column            Non-Null Count  Dtype         
---  ------            --------------  -----         
 0   gender            14825 non-null  object        
 1   id                14825 non-null  object        
 2   became_member_on  14825 non-null  datetime64[ns]
 3   income            14825 non-null  float64       
 4   age_group         14825 non-null  category      
dtypes: category(1), datetime64[ns](1), float64(1), object(2)
memory usage: 593.7+ KB
time: 16 ms (started: 2021-03-22 00:24:10 +01:00)<jupyter_text>3.3. Prepare `transcript` dataset<jupyter_code>transcript.head()<jupyter_output><empty_output><jupyter_text>3.3.1 convert `value` to new colum(s)<jupyter_code># create a copy of `transcript` dataframe
cleaned_transcript = transcript.copy()
# get all possible dictionary key from the `value` column
value_keys = set()
def get_keys(x):
    value_keys.update(x.keys())
cleaned_transcript['value'].apply(get_keys)
value_keys<jupyter_output><empty_output><jupyter_text>=> there are only 3 possible values from the `value`-Feature. We will assume that the `offer id` and `offer_id` are the same value.<jupyter_code>cleaned_transcript.head()
# creating a new feature `offer_id` from cleaned `value` feature
cleaned_transcript['offerid'] = cleaned_transcript['value'].apply(lambda elem: list(elem.values())[0])
cleaned_transcript.head(10)
# drop value column 
cleaned_transcript.drop('value', axis=1, inplace=True)
cleaned_transcript.head(10)<jupyter_output><empty_output><jupyter_text>3.3.2 drop if event-type `transaction`<jupyter_code>cleaned_transcript = cleaned_transcript[cleaned_transcript['event'] != 'transaction']
cleaned_transcript.head()
cleaned_transcript.shape<jupyter_output><empty_output><jupyter_text>3.3.3 drop if person is deletedThe `transcript` does not to be dropt if the person is deleted. The reason is, while merging the DataFrame, we will use join based on the `person_id`. So if the person already deleted, the following `transcript` data will not be included in the merged dataframe.  3.3.4 Rename column `person`<jupyter_code>cleaned_transcript.rename(columns={'person': 'customer_id', 'offerid' : 'offer_id', 'event': 'event_type' }, inplace=True)
cleaned_transcript.head()<jupyter_output><empty_output><jupyter_text>3.3.5 One-hot-encoding `event_type` and concat into transcript<jupyter_code>event_temp = pd.get_dummies(cleaned_transcript['event_type'], prefix='event')
print(event_temp.head())
event_temp.rename(columns={'event_offer completed': 'event_offer_completed', 'event_offer received' : 'event_offer_received', 'event_offer viewed': 'event_offer_viewed' }, inplace=True)
event_temp.head()
cleaned_transcript = pd.concat([cleaned_transcript, event_temp], axis=1)
cleaned_transcript.head()
# cleaned_transcript = cleaned_transcript.drop('event_type', axis = 1)
#cleaned_transcript.head()<jupyter_output>time: 0 ns (started: 2021-03-22 00:24:10 +01:00)<jupyter_text>3.3.6 Creating feature `offer_successful`I will use the new feature `offer_successful` as target-feature for the prediction model. The feature will describe if a customer viewed the offer and therefore completed the reward.<jupyter_code>cleaned_transcript[(cleaned_transcript['customer_id'] == '0009655768c64bdeb2e877511632db8f') & (cleaned_transcript['offer_id'] == '2906b810c7d4411798c6938adc9daaa5')]
# Group by `person_id` and `offer_id`
cleaned_transcript_group = cleaned_transcript.groupby(['customer_id', 'offer_id'])['event_type'].agg(['sum']).reset_index()
cleaned_transcript_group.head()
# add new feature `offer_successful`
cleaned_transcript_group['offer_successful'] = cleaned_transcript_group['sum'].str.contains('viewed') \
    & cleaned_transcript_group['sum'].str.contains('completed')
cleaned_transcript_group.head()
cleaned_transcript_group.drop(columns=['sum'], inplace=True)
cleaned_transcript_group<jupyter_output><empty_output><jupyter_text>Merge the `offer_successful` feature into the  `cleaned_transcript`.<jupyter_code>#cleaned_transcript = pd.merge(cleaned_transcript, received_completed,  how='left', left_on=['customer_id','offer_id'], right_on = ['customer_id','offer_id']).drop(columns=['sum'])
#cleaned_transcript.head()<jupyter_output>time: 0 ns (started: 2021-03-22 00:24:17 +01:00)<jupyter_text>Summary<jupyter_code>cleaned_transcript_group.describe()
cleaned_transcript_group.info()<jupyter_output><class 'pandas.core.frame.DataFrame'>
RangeIndex: 63288 entries, 0 to 63287
Data columns (total 3 columns):
 #   Column            Non-Null Count  Dtype 
---  ------            --------------  ----- 
 0   customer_id       63288 non-null  object
 1   offer_id          63288 non-null  object
 2   offer_successful  63288 non-null  bool  
dtypes: bool(1), object(2)
memory usage: 1.0+ MB
time: 15 ms (started: 2021-03-22 00:24:17 +01:00)<jupyter_text>3.4 Merging all 3 dataframes<jupyter_code>cleaned_portfolio.head()
cleaned_profile.head()
cleaned_transcript_group.head()<jupyter_output><empty_output><jupyter_text>3.4.1 Merge `profile` and `transcript`<jupyter_code>merged_dataframe = cleaned_profile.merge(cleaned_transcript_group , left_on='id', right_on='customer_id')
merged_dataframe.head(10)<jupyter_output><empty_output><jupyter_text>Because of the **left join** between `profile` and `transcript`, the transcript data with deleted `profile` will be automatically not merged.  3.4.2 Merge `portfolio` with `profile` and `transcript`<jupyter_code>merged_dataframe = cleaned_portfolio.merge(merged_dataframe , left_on='id', right_on='offer_id')
merged_dataframe.head(10)
merged_dataframe.shape
# Remove double column, customer_id and offer_id
merged_dataframe = merged_dataframe.drop(['id_x','id_y', 'offer_id', 'customer_id'], axis=1, errors='ignore') # the flag errors to make sure that this command could be called twice
merged_dataframe<jupyter_output><empty_output><jupyter_text>3.5 Encoding categorical column<jupyter_code>merged_dataframe.info()
categorical_column = ['offer_type', 'gender', 'age_group']
merge_dataframe_encoded = merged_dataframe.copy()
merge_dataframe_encoded.head()
#import the necessary module
from sklearn import preprocessing
# create the Labelencoder object
le = preprocessing.LabelEncoder()
for cat in categorical_column:
    merge_dataframe_encoded[cat] = le.fit_transform(merge_dataframe_encoded[cat])

merge_dataframe_encoded.head()<jupyter_output><empty_output><jupyter_text>3.6 Correlation of the feature column<jupyter_code>merge_dataframe_encoded.head()
# plotting heatmap
plt.figure(figsize=(30,30))
cor = merge_dataframe_encoded.corr()
sns.heatmap(cor, annot=True, cmap=plt.cm.Reds)
plt.show()<jupyter_output><empty_output><jupyter_text>As we can see, the higest correlation is from the feature `duration`, followed by `social` and `reward`.   3.7 Dropping `became_member_on` I will drop the feature `became_member_on` in this project, with the reason that there is not enough information, when the data collected at. So we could not get an absolute timepoint.But for further investigation, or the next project, this feature could be divided into `year` and `month`, and could be used for the prediction of the model.<jupyter_code>merge_dataframe_encoded.drop(columns=['became_member_on'], inplace=True)
merged_dataframe.drop(columns=['became_member_on'], inplace=True)<jupyter_output>time: 0 ns (started: 2021-03-22 00:24:19 +01:00)<jupyter_text>3.5 Create target feature<jupyter_code>target_cols = 'offer_successful'
feature_cols = merge_dataframe_encoded.columns.tolist()
feature_cols.remove(target_cols)
feature_cols
data = merge_dataframe_encoded[feature_cols]
#assigning the `offer_successful` column as target
target = merge_dataframe_encoded[target_cols]<jupyter_output>time: 0 ns (started: 2021-03-22 00:24:19 +01:00)<jupyter_text>4. Data analysis and data model Following questions will be examined and answered:- How much starbucks customer completed a reward ?- What kind of offer type most customer view and completed?- Could a prediction be made, either a customer will take a reward or not? So a targeted marketing could be done! Until now there are only 2 dataframes that we could use for analysis:- **merged_dataframe** : dataframe that still containing `offer_type`, `gender` as text, and- **merged_dataframe_encoded** : dataframe that contains `offer_type` and `gender` as encoded information. This dataframe will be used for the model prediction<jupyter_code>merged_dataframe.head()<jupyter_output><empty_output><jupyter_text>4.1 Demographic distribution of starbucks customer<jupyter_code>offer_taken = merged_dataframe[merged_dataframe['offer_successful']]
offer_taken
offer_not_taken = merged_dataframe[merged_dataframe['offer_successful'] != True]
offer_not_taken
print(f'The proportion of customer that taken taken the offer are : {offer_taken.shape[0]/merged_dataframe.shape[0]}')
sns.displot(merged_dataframe, x="age_group", hue="offer_successful", multiple="dodge").set(title='Distribution by age group')
sns.displot(merged_dataframe, x="gender", hue="offer_successful", multiple="dodge").set(title='Distribution by gender')
sns.displot(merged_dataframe, x="income", hue="offer_successful", multiple="dodge").set(title='Distribution by income')<jupyter_output><empty_output><jupyter_text>4.2 Distribution of offer<jupyter_code>sns.displot(merged_dataframe, x="offer_type", hue="offer_successful", multiple="dodge").set(title='Distribution by offer type')<jupyter_output><empty_output><jupyter_text>4.3 Model to predict if a customer take an offer In this part, I will train a model with following classifier: - Linear SVC- KNeighbourhood- AdaBoost**Linear SVC** and **KNeighbourhood** are chosen based on scikit-learn [page](https://scikit-learn.org/stable/tutorial/machine_learning_map/). AdaBoost are chosen, because previous experience with the classifier and has a good performace and speed for training a model.To measure the performance of the model, I will use **F-Score** and **accuracy**.  4.3.1 Splitting into train and test data<jupyter_code>X_train, X_test, y_train, y_test = train_test_split(data, target, test_size=0.33, random_state=42)
X_train.head()
y_train.head()<jupyter_output><empty_output><jupyter_text>4.3.2 Linear SVC  Training model<jupyter_code>#create an object of type LinearSVC
svc_model = LinearSVC(random_state=0)
svc_model.fit(X_train, y_train)

#train the algorithm on training data and predict using the testing data
model_svc = svc_model.fit(X_train, y_train)<jupyter_output>C:\Users\victo\anaconda3\envs\starbucks-capstone\lib\site-packages\sklearn\svm\_base.py:976: ConvergenceWarning: Liblinear failed to converge, increase the number of iterations.
  warnings.warn("Liblinear failed to converge, increase "<jupyter_text>Prediction & evaluation<jupyter_code>y_pred = model_svc.predict(X_test)
print_metrics(y_test, y_pred)<jupyter_output>Accuracy for the model :  0.5763279192273925
F1-score for the model :  0.42142737742509156
time: 16 ms (started: 2021-03-22 00:30:54 +01:00)<jupyter_text>4.3.3. KNeighbourhood  Training model<jupyter_code>#create object of the classifier
neigh_model = KNeighborsClassifier(n_neighbors=3)

#Train the algorithm
model_neighbourhood = neigh_model.fit(X_train, y_train)<jupyter_output>time: 187 ms (started: 2021-03-22 00:31:46 +01:00)<jupyter_text>Prediction & evaluation<jupyter_code>y_pred = model_neighbourhood.predict(X_test)
print_metrics(y_test, y_pred)<jupyter_output>Accuracy for the model :  0.6927129060579456
F1-score for the model :  0.6934513838220744
time: 797 ms (started: 2021-03-22 00:31:47 +01:00)<jupyter_text>4.3.4 AdaBoost  Training model The most important parameters for AdaBoost are [**base_estimator, n_estimators, and learning_rate.**](https://chrisalbon.com/machine_learning/trees_and_forests/adaboost_classifier/). But I will just use the **n_estimators** and let the other parameter as it is (default).<jupyter_code>#create object of the classifier
ada_model = AdaBoostClassifier(n_estimators=100, random_state=0)

#Train the algorithm
model_ada = ada_model.fit(X_train, y_train)<jupyter_output>time: 2.23 s (started: 2021-03-22 00:31:47 +01:00)<jupyter_text>Prediction & evaluation<jupyter_code>y_pred = model_ada.predict(X_test)
print_metrics(y_test, y_pred)<jupyter_output>Accuracy for the model :  0.7378182616330115
F1-score for the model :  0.7386484432107018
time: 344 ms (started: 2021-03-22 00:31:50 +01:00)<jupyter_text>From the 3 classifier above, only AdaBoost that could predict with astounding **accuracy (73.78%)** and **F1-Score (73.86%)**.  4.3.5 Hyperparameter Tuning with  GridSearchCVTo achieve a better result / metrics, I will use **GridSearchCV** for finding the best parameter.<jupyter_code># defining the parameter grid
param_grid={
    'n_estimators' : [10, 50, 100, 500],
    'learning_rate' : [0.0001, 0.001, 0.01, 0.1, 1.0],
    'algorithm' : ['SAMME', 'SAMME.R']
}
grid_ada_model = AdaBoostClassifier(random_state=0)

grid_search = GridSearchCV(estimator=grid_ada_model,
             param_grid=param_grid, n_jobs=-1,scoring=['accuracy','f1'], refit='accuracy')
# execute the grid search
grid_result = grid_search.fit(X_train, y_train)
print("Best score: %f using following parameters %s" % (grid_result.best_score_, grid_result.best_params_))
#create object of the classifier with best parameter
best_ada_model = AdaBoostClassifier(n_estimators=500, algorithm = 'SAMME.R', learning_rate = 0.1, random_state=0)

#Train the algorithm with best parameter
best_model_ada = best_ada_model.fit(X_train, y_train)
y_pred = best_model_ada.predict(X_test)
print_metrics(y_test, y_pred)<jupyter_output>Accuracy for the model :  0.7397936786654961
F1-score for the model :  0.7406589710964917
time: 969 ms (started: 2021-03-22 19:00:54 +01:00)<jupyter_text>With parameter tuning, the classifier could only achieve from **accuracy (73.78%)** and **F1-Score (73.86%)** a bit of improvement to **accuracy (73.98%)** and **F1-Score (74.07%)**  4.3.6 Evaluating the model with K-Fold To test the robustness of the AdaBoostModel, I will evaluate the model with help of K-Fold.<jupyter_code>cv = KFold(n_splits=10, random_state=1, shuffle=True)

# evaluate model
acc_scores = cross_val_score(best_model_ada, data, target, scoring='accuracy', cv=cv, n_jobs=-1)

# report performance
print('Accuracy: %.4f (%.4f)' % (mean(acc_scores), std(acc_scores)))

# evaluate model
f1_scores = cross_val_score(best_model_ada, data, target, scoring='f1', cv=cv, n_jobs=-1)

# report performance
print('F1-Score: %.4f (%.4f)' % (mean(f1_scores), std(f1_scores)))<jupyter_output>Accuracy: 0.7388 (0.0059)
F1-Score: 0.7020 (0.0071)
time: 1min 48s (started: 2021-03-22 21:15:35 +01:00)
{'fraction_non_alphanumeric': 0.10268562401263823, 'fraction_numerical': 0.037015852263441736, 'mean_word_length': 5.567978533094812, 'pattern_counts': {'":': 0, '<': 227, '<?xml version=': 0, '>': 263, 'https://': 3, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '61', 'dir': 'jupyter-structured-clean-dedup', 'id': '409202', 'license': 'FTL', 'n_tokens_mistral': 12358, 'n_tokens_neox': 11191, 'n_words': 3669, 'path': 'Starbucks_Capstone_notebook.ipynb', 'repo_name': 'veectro/starbucks-capstone', 'size': '36714'}