<jupyter_start><jupyter_text>Starbucks Data Science Project  1. Introduction: This data set contains simulated data that mimics customer behavior on the Starbucks rewards mobile app. Once every few days, Starbucks sends out an offer to users of the mobile app. An offer can be merely an advertisement for a drink or an actual offer such as a discount or BOGO (buy one get one free).   2. Datasets: The data is contained in three files:1. portfolio.json - containing offer ids and meta data about each offer (duration, type, etc.)2. profile.json - demographic data for each customer3. transcript.json - records for transactions, offers received, offers viewed, and offers completedHere is the schema and explanation of each variable in the files:1. portfolio.json    - id (string) - offer id    - offer_type (string) - type of offer ie BOGO, discount, informational    - difficulty (int) - minimum required spend to complete an offer    - reward (int) - reward given for completing an offer    - duration (int) - time for offer to be open, in days    - channels (list of strings)        2. profile.json    - age (int) - age of the customer    - became_member_on (int) - date when customer created an app account    - gender (str) - gender of the customer (note some entries contain 'O' for other rather than M or F)    - id (str) - customer id    - income (float) - customer's income    3. transcript.json    - event (str) - record description (ie transaction, offer received, offer viewed, etc.)    - person (str) - customer id    - time (int) - time in hours since start of test. The data begins at time t=0    -value - (dict of strings) - either an offer id or transaction amount depending on the record  3. Importing libraries and data<jupyter_code>import pandas as pd
import numpy as np
import matplotlib.pyplot as plt
import seaborn as sns
from datetime import datetime
import re
import os
import pickle
from sklearn.model_selection import train_test_split
from sklearn.preprocessing import LabelBinarizer
from sklearn.ensemble import RandomForestClassifier
from sklearn.preprocessing import MinMaxScaler
from sklearn.metrics import accuracy_score, f1_score
from sklearn.ensemble import GradientBoostingClassifier, RandomForestClassifier
from sklearn.linear_model import LogisticRegression
from sklearn.metrics import fbeta_score, make_scorer
from sklearn.model_selection import GridSearchCV
from sklearn.model_selection import RandomizedSearchCV
from joblib import dump, load
%matplotlib inline

# Keeping only 2 digits after decimal
pd.options.display.float_format = '{:.2f}'.format
# read in the json files
portfolio = pd.read_json('portfolio.json', orient='records', lines=True)
profile = pd.read_json('profile.json', orient='records', lines=True)
transcript = pd.read_json('transcript.json', orient='records', lines=True)<jupyter_output><empty_output><jupyter_text>With reference to the python file other_functions.py lets work on cleaning and analyzing the data  4. Data Wrangling, Data Analysis and Visualization  - Working with the portfolio data:<jupyter_code># Viewing the offer portfolio data
portfolio
portfolio.info()
portfolio.describe()
portfolio['offer_type'].value_counts()
portfolio.groupby('offer_type')['duration'].value_counts()<jupyter_output><empty_output><jupyter_text>Steps to clean this portfolio data:- Change the name of the 'id' column to 'offerid'- Change the name of the 'duration' column to 'durationdays'- Remove underscores from column names- One hot encode the 'offertype' column- One hot encode the 'channels' columns- Replace the 'offertype' and 'channels' columns with their respective encoded values<jupyter_code># change the duration from day to hour and then apply one hot encoding to channels, offer_type column
# combine the portfolio and offer_type dataframe to form a cleaned dataframe

def clean_portfolio(portfolio):
    '''
    data cleaning the portfolio dataframe
    
    INPUT:
    portfolio - the portfolio dataframe to be cleaned
    
    OUTPUT:
    portfolio - the cleaned portfolio dataframe
    
    '''
    
    cleaned_portfolio = portfolio.copy()
    cleaned_portfolio['duration'] = cleaned_portfolio['duration'] * 24

    cleaned_portfolio['web'] = cleaned_portfolio['channels'].apply(lambda x: 1 if 'web' in x else 0)
    cleaned_portfolio['email'] = cleaned_portfolio['channels'].apply(lambda x: 1 if 'email' in x else 0)
    cleaned_portfolio['mobile'] = cleaned_portfolio['channels'].apply(lambda x: 1 if 'mobile' in x else 0)
    cleaned_portfolio['social'] = cleaned_portfolio['channels'].apply(lambda x: 1 if 'social' in x else 0)
    
    offer_type = pd.get_dummies(cleaned_portfolio['offer_type'])
    
    cleaned_portfolio.drop(['channels', 'offer_type'], axis=1, inplace=True)
    
    cleaned_portfolio = pd.concat([cleaned_portfolio, offer_type], axis=1, sort=False)
    
    return cleaned_portfolio
# Code: To clean the portfolio data
cleaned_portfolio = clean_portfolio(portfolio)
# Test:
cleaned_portfolio<jupyter_output><empty_output><jupyter_text>- Working with the profile data:<jupyter_code># Viewing the profile data
profile.head()
profile.info()
profile.describe()
profile.isnull().sum()
# All the missing age values are encoded as 62.5 (mean age taken from the statistics above) 
# and then drop all missing data. Convert the became_member_on into a datetime format and calculate the number of days 
# being a member with respect to today's date


def clean_profile(profile):
    '''
    data cleaning the profile dataframe
    
    INPUT:
    profile - the profile dataframe to be cleaned
    
    OUTPUT:
    profile - the cleaned profile dataframe
    
    '''
    profile['age'] = profile['age'].apply(lambda x: np.nan if x == 62.5 else x)
    
    profile.dropna(inplace=True)
    
    profile['memberdays'] = pd.datetime.today().date() - pd.to_datetime(profile['became_member_on'], format='%Y%m%d').dt.date
    profile['memberdays'] = profile['memberdays'].dt.days
    
    
    return profile
cleaned_profile = clean_profile(profile)
cleaned_profile.head()
cleaned_profile.describe()
cleaned_profile.info()<jupyter_output><class 'pandas.core.frame.DataFrame'>
Int64Index: 14825 entries, 1 to 16999
Data columns (total 6 columns):
 #   Column            Non-Null Count  Dtype  
---  ------            --------------  -----  
 0   gender            14825 non-null  object 
 1   age               14825 non-null  int64  
 2   id                14825 non-null  object 
 3   became_member_on  14825 non-null  int64  
 4   income            14825 non-null  float64
 5   memberdays        14825 non-null  int64  
dtypes: float64(1), int64(3), object(2)
memory usage: 810.7+ KB<jupyter_text>The users have an annual income range of 60k to 80k<jupyter_code>plt.figure(figsize=(12,8))
cleaned_profile.income.hist(bins=15)
plt.xlabel("Income Range")
plt.title("Income Range of the users")
cleaned_profile['gender'].value_counts()
# Visualization for the age distributions of the customers

user_age = cleaned_profile['age'].plot(kind='hist', bins=20, title='Distribution of Customer Age')
user_age.set_xlabel("Customer Age")
# Visualization for the Number of days being a member

memberdays = cleaned_profile['memberdays'].plot(kind='hist', bins=50, title='Number of days since the users became members')
memberdays.set_xlabel("Days")
# Visualization for the distribution of the users' income

user_income = cleaned_profile['income'].plot(kind='hist', bins=20, title='Distribution of Customer Income')
user_income.set_xlabel("Income")<jupyter_output><empty_output><jupyter_text>- Working with the transcript data<jupyter_code># Viewing the transcript data
transcript
# Number of event types

transcript['event'].unique()
# Randomly selecting one user and exploring the transcript records and flow of events
transcript[transcript['person']=='a03223e636434f42ac4c3df47e8bac43']
# extract the rows that are related to offer action (e.g. offer received, offer viewed, offer completed)
# Excluding 'transaction' of the events
# extract the offer id from value column
    
def clean_transcript(transcript):
    '''
    data cleaning the transcript dataframe, return usable dataframe need contain offer events
    
    INPUT:
    transcript - the transcript dataframe to be cleaned
    
    OUTPUT:
    offer - the cleaned offer dataframe
    
    '''
    offer = transcript[transcript['value'].apply(lambda x: True if ('offer id' in x) or ('offer_id' in x) else False)]
    
    offer['offer_id'] = offer['value'].apply(lambda x: x['offer id'] if ('offer id' in x) else x['offer_id'])
    
    return offer
offer = clean_transcript(transcript)
offer['event'].unique()<jupyter_output><empty_output><jupyter_text>We can see below that the offer received per person in the transactional data could range from 1 to 6 offers received.<jupyter_code>#group event=offer received per person in transactional records
print(offer[offer['event']=='offer received'].groupby('person')['event'].count().head())

#visualise offers received per person
offer[offer['event']=='offer received'].groupby('person')['event'].count().hist()<jupyter_output>person
0009655768c64bdeb2e877511632db8f    5
00116118485d4dfda04fdbaba9a87b5c    2
0011e0d4e6b944f998e987f904e8c1e5    5
0020c2b971eb4e9188eac86d93036a77    5
0020ccbbb6d84e358d3414a3ff76cffd    4
Name: event, dtype: int64<jupyter_text>Customer gender distribution attribute: Greater male users<jupyter_code>gender_counts = cleaned_profile['gender'].value_counts()
print(gender_counts)
gender_counts.plot(kind = 'bar')<jupyter_output>M    8484
F    6129
O     212
Name: gender, dtype: int64<jupyter_text>Which year attracted maximum subscribers : 2017 attracted maximum customers<jupyter_code>became_member_on = cleaned_profile['became_member_on'].apply(convert_to_datetime)

start_year = became_member_on.apply(lambda elem: elem.year).value_counts()
start_year *= 100 / start_year.sum()
print(start_year)
start_year.plot(kind = 'bar')<jupyter_output>2017   37.77
2018   24.75
2016   20.40
2015   10.77
2014    4.47
2013    1.85
Name: became_member_on, dtype: float64<jupyter_text>Which month attracts most subscribers: August, October and December usually attract maximum customers<jupyter_code>start_month = became_member_on.apply(lambda elem: elem.month).value_counts()
start_month *= 100 / start_month.sum()
print(start_month)
start_month.plot(kind='bar')<jupyter_output>8    9.41
10   9.35
12   9.25
1    8.95
9    8.76
11   8.55
7    8.02
5    7.77
4    7.72
3    7.72
6    7.45
2    7.03
Name: became_member_on, dtype: float64<jupyter_text>5. Data Modeling<jupyter_code># Concentrating only on BOGO and discounts first


def create_user_item_matrix(offer):
    '''
    Return the user item matrix that indicate the number of offer complete of a particular user
    
    INPUT:
    offer - a cleaned transcript dataframe
    filename(string) - the file name that save the user item matrix
    
    OUTPUT:
    user_item_matrix - the user item matrix which 
        - row is user 
        - column is offer
        - value is the number of offer complete by the user (NaN means no offer given)
    
    '''   
    user_item_matrix = offer.groupby(['person', 'offer_id'])['event'].agg(lambda x: np.nan).unstack()
    
    user_item_matrix.drop(list(portfolio[portfolio['offer_type']=='informational']['id']), axis=1, inplace=True)
    
    for offer_id in user_item_matrix.columns:
        print("Now processing: ", offer_id)
        num = 0
        for person in user_item_matrix.index:
            num += 1
            if num % 1000 == 0:
                print("finished ", num/16994*100, '%')
            events = []
            for event in offer[(offer['offer_id']==offer_id) & (offer['person']==person)]['event']:
                events.append(event)
            if len(events) >= 3:
                user_item_matrix.loc[person, offer_id] = 0
                for i in range(len(events)-2):
                    # check if the transaction sequence is offer received -> offer viewed -> offer completed
                    # if yes, we assume the user reacted positively with the offer we provided.
                    if (events[i] == 'offer received') & (events[i+1] == 'offer viewed') & (events[i+2] == 'offer completed'):
                        user_item_matrix.loc[person, offer_id] += 1
            elif len(events) > 0:
                user_item_matrix.loc[person, offer_id] = 0
    
    # store the large martix into file
 #   fh = open(filename, 'wb')
  #  pickle.dump(user_item_matrix,fh)
  #  fh.close()
    
    return user_item_matrix
df = create_user_item_matrix(offer)
df.head()
# Checking the size of the training and test sets
train_size = int(offer.shape[0]*0.7)
test_size = offer.shape[0] - train_size
train_size, test_size
train_offer = offer[:train_size]
test_offer = offer[train_size:]

train_df = create_user_item_matrix(train_offer)
train_df.head()
test_df = create_user_item_matrix(test_offer)
test_df.head()

def matrxfact(complete_mat, latent_features=12, learning_rate=0.0001, iters=100):
    '''
    This function performs matrix factorization using a basic form of FunkSVD with no regularization
    
    INPUT:
    complete_mat - (numpy array) a matrix with users as rows, offers as columns, and offer completed as values
    latent_features - (int) the number of latent features used
    learning_rate - (float) the learning rate 
    iters - (int) the number of iterations
    
    OUTPUT:
    user_mat - (numpy array) a user by latent feature matrix
    offer_mat - (numpy array) a latent feature by offer matrix
    '''
    
    # Set up useful values to be used through the rest of the function
    n_users = complete_mat.shape[0]
    n_offers = complete_mat.shape[1]
    num_complete = np.count_nonzero(~np.isnan(complete_mat))
    
    # initialize the user and movie matrices with random values
    user_mat = np.random.rand(n_users, latent_features)
    offer_mat = np.random.rand(latent_features, n_offers)
    
    # initialize sse at 0 for first iteration
    sse_accum = 0
    
    # keep track of iteration and MSE
    print("Optimizaiton Statistics")
    print("Iterations | Mean Squared Error ")
    
    # for each iteration
    for iteration in range(iters):

        # update our sse
        old_sse = sse_accum
        sse_accum = 0
        
        # For each user-offer pair
        for i in range(n_users):
            for j in range(n_offers):
                
                # if the rating exists
                if complete_mat[i, j] >= 0:
                    # compute the error as the actual minus the dot product of the user and offer latent features
                    diff = complete_mat[i, j] - np.dot(user_mat[i, :], offer_mat[:, j])
                    
                    # Keep track of the sum of squared errors for the matrix
                    sse_accum += diff**2
                    
                    # update the values in each matrix in the direction of the gradient
                    for k in range(latent_features):
                        user_mat[i, k] += learning_rate * (2*diff*offer_mat[k, j])
                        offer_mat[k, j] += learning_rate * (2*diff*user_mat[i, k])

        # print results
        print("%d \t\t %f" % (iteration+1, sse_accum / num_complete))
        
    return user_mat, offer_mat
train_data_np = np.array(train_df)
user_mat, offer_mat = matrxfact(train_data_np, latent_features=15, learning_rate=0.005, iters=250)
# Test for the best number of latent feature. (with latent features 10)
user_mat_10, offer_mat_10 = matrxfact(train_data_np, latent_features=10, learning_rate=0.005, iters=250)
def predict_reaction(user_matrix, offer_matrix, user_id, offer_id):
    '''
    We can use this to make predictions for how users would react with the offers
    
    INPUT:
    user_matrix - user by latent factor matrix
    offer_matrix - latent factor by offer matrix
    user_id - the user_id from the reviews df
    offer_id - the offer_id according the offers df
    
    OUTPUT:
    pred - the predicted reaction for user_id-offer_id according to FunkSVD
    '''
    try:
        # Use the training data to create a series of users and movies that matches the ordering in training data
        user_ids_series = np.array(train_df.index)
        offer_ids_series = np.array(train_df.columns)

        # User row and Movie Column
        user_row = np.where(user_ids_series == user_id)[0][0]
        offer_col = np.where(offer_ids_series == offer_id)[0][0]

        # Take dot product of that row and column in U and V to make prediction
        pred = np.dot(user_matrix[user_row, :], offer_matrix[:, offer_col])
        
        return pred
    
    except:
        return None

def validation(test_df, user_mat, offer_mat):
    '''Measure the squared errors for the prediction'''
    num_complete = np.count_nonzero(~np.isnan(test_df))
    
    sse_accum = 0
    
    for user_id in test_df.index:
        for offer_id in test_df.columns:
            if ~np.isnan(test_df.loc[user_id, offer_id]):
                predict_value = predict_reaction(user_mat, offer_mat, user_id, offer_id)
                if predict_value != None:
                    # compute the error as the actual minus the dot product of the user and offer latent features
                    diff = test_df.loc[user_id, offer_id] - predict_reaction(user_mat, offer_mat, user_id, offer_id)

                    # Keep track of the sum of squared errors for the matrix
                    sse_accum += diff**2
    
    print(sse_accum / num_complete)
# Evaluation for latent features of 10
validation(test_df, user_mat_10, offer_mat_10)
def recommend_offer(user_id, user_mat, offer_mat):
    recommend = {}
    for offer_id in train_df.columns:
        pred_val = predict_reaction(user_mat, offer_mat, user_id, offer_id)
        if pred_val != None:
            recommend[offer_id] = pred_val
        else:
            break
    if pred_val == None:
        print("Because this user wasn't in our database, we are giving back the top best sale offer recommendations for all users.")
        top_offer = best_sale_offer(full_df)
        for offer_id in top_offer['offer_id']:
            print("offer id: ", offer_id)
    else:
        print("recommend offer for user ", user_id)
        for offer_id, pred_val in sorted(recommend.items(), key=lambda kv:(kv[1], kv[0]), reverse=True):
            print("offer id: ", offer_id, " predicted value: ", round(pred_val,2))
# recommend offer for a particular user
recommend_offer('0610b486422d4921ae7d2bf64640c50b', user_mat, offer_mat)
def gender_offer(full_df):
    '''
    Find out which type of offer (bogo/discount) attracts most to different gender
    Return with a plotted bar chart
    '''
    
    # initialize the gender response and the gender sent dataframe
    gender_response = pd.DataFrame(data=[[0,0],[0,0],[0,0]], index=['M', 'F', 'O'], columns=['bogo', 'discount'])
    gender_sent = pd.DataFrame(data=[[0,0],[0,0],[0,0]], index=['M', 'F', 'O'], columns=['bogo', 'discount'])

    # We count the number of offer completed base on the customer gender
    for offer_type in ['bogo', 'discount']:
        for offer_id in portfolio[portfolio['offer_type']==offer_type]['id']:
            # count the total number of customers responsed our offers.
            gender_response[offer_type] = gender_response[offer_type].add(profile[profile['id'].isin(full_df[full_df[offer_id]>0].index)]['gender'].value_counts(), fill_value=0)
            # count the total number of offer sent
            gender_sent[offer_type] = gender_sent[offer_type].add(profile[profile['id'].isin(full_df[full_df[offer_id]>=0].index)]['gender'].value_counts(), fill_value=0)
        
    gender_offer = (gender_response / gender_sent * 100).plot(kind='bar', title='Percentage of customer response')
    gender_offer.set_xlabel("Gender")
    gender_offer.set_ylabel("Percentage")
gender_offer(df)
def gender_offer_channel(full_df):
    '''
    Find out which type of offer (bogo/discount) attracts most to different gender
    Return with a plotted bar chart
    '''
    
    # initialize the gender response and the gender sent dataframe
    gender_response = pd.DataFrame(data=[[0,0,0,0],[0,0,0,0],[0,0,0,0]], index=['M', 'F', 'O'], columns=['web', 'email', 'mobile', 'social'])
    gender_sent = pd.DataFrame(data=[[0,0,0,0],[0,0,0,0],[0,0,0,0]], index=['M', 'F', 'O'], columns=['web', 'email', 'mobile', 'social'])

    # We count the number of offer completed base on the customer gender
    for channel_type in ['web', 'email', 'mobile', 'social']:
        for offer_id in cleaned_portfolio[(cleaned_portfolio[channel_type]==1)&(cleaned_portfolio['informational']!=1)]['id']:
            # count the total number of customers responsed our offers.
            gender_response[channel_type] = gender_response[channel_type].add(profile[profile['id'].isin(full_df[full_df[offer_id]>0].index)]['gender'].value_counts(), fill_value=0)
            # count the total number of offer sent
            gender_sent[channel_type] = gender_sent[channel_type].add(profile[profile['id'].isin(full_df[full_df[offer_id]>=0].index)]['gender'].value_counts(), fill_value=0)
        
    gender_offer = (gender_response / gender_sent * 100).plot(kind='bar', title='Percentage of customer response')
    gender_offer.set_xlabel("Gender")
    gender_offer.set_ylabel("Percentage")
gender_offer_channel(df)
def best_sale_offer(user_item_matrix):
    # Find out which offer sells the best

    offer_gain = []
    for offer_id in user_item_matrix.columns:
        offer_gain.append([offer_id, transcript[(transcript['person'].isin(list(user_item_matrix[user_item_matrix[offer_id]>=1].index)))&(transcript['event']=='transaction')]['value'].apply(lambda x: x['amount']).sum()])

    offer_value = pd.DataFrame(offer_gain, columns=['offer_id', 'gain'])
    offer_value['gain'] = pd.to_numeric(offer_value['gain'])
    offer_value.sort_values(by='gain', ascending=False, inplace=True)
    print(offer_value)
    
    best_sale = offer_value.plot(kind='bar', title='Sale Amount of Different Offers')
    best_sale.set_xlabel('Offer number')
    best_sale.set_ylabel('Sale Amount')
    
    return offer_value
best_sale_offer(df)
# offer_if of the best sale
offer[offer.values=='fafdcd668e3743c1bb461111dcafc2a4']<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.10021464985242823, 'fraction_numerical': 0.02464001430999016, 'mean_word_length': 3.6319386909693456, 'pattern_counts': {'":': 0, '<': 41, '<?xml version=': 0, '>': 51, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '11', 'dir': 'jupyter-structured-clean-dedup', 'id': '165090', 'license': 'FTL', 'n_tokens_mistral': 7205, 'n_tokens_neox': 6655, 'n_words': 2046, 'path': 'Starbucks_Capstone_Project.ipynb', 'repo_name': 'kalwadhavisha/Starbucks_Capstone', 'size': '22362'}