<jupyter_start><jupyter_text>1.3  -- Creating Customer ProfilesDunnHumby The Complete Journey<NAME>  Data Pipeline PackageIn order to simplify or manage the workflow of a data product, we can employ a package of scripts, or functions.  Working in a Jupyter Notebook feels like a much safer place to experiment with data pipelining for me right now; but as far as I understand, bash and SQL will be primary tools in constructing data pipelines in industry. The process thus far has been an educational one for me and helped to solidify my foundation with my primary (first, and only) scripting language, Python, while working my way through some basic comprehension of this critical data infrastructure piece.Obviously, we want to be able to load the relevant data (without any null values, only of the relevant type, etc.) -- and hopefully not spend too much unnecessary computing power or memory to do so, especially on pay-as-you-go cloud software. I know I still have a ton to learn about what that might look like for any given organization; I'm confident that I will learn quickly on the job. Just like when you're making bread, every kitchen is different. Things that work in one environment may not work well in another.We might employ a 'Unit Test'; to check that our data is exactly what we expect it to be. I still have a lot to learn about what this might look like in any given data workflow, but since I'm the only person working on this project and everything is hosted locally, I am bypassing this (extremely important) step with caution and humility. I hope that this limited exposure to project API development will further my comprehension of workflows and systems in future workplaces. Using the package for this project (labeled dtcj, for dunnhumby the complete journey), I load the cleaned version of our data below.<jupyter_code># import modules
import pandas as pd
import numpy as np
import matplotlib.pyplot as plt
import seaborn as sns
plt.style.use('seaborn')
plt.rcParams['figure.figsize']==(6,6)

import dtcj

# load data
merged = dtcj.load_merged()<jupyter_output><empty_output><jupyter_text>Continuing on... Now that we have transactions with appropriate products labels, we can also begin the process of distinguishing our households into meaningful groups. I'm banking on the idea that there is some sort of relationship between the attributes of each household (which we are about to define), and the products that they purchase (what we hope to predict).If this proves to be the case, we can make more intelligent and cost-effective recommendations for our households. By examining the association rules for different market segments, we can determine the most effective strategy for product placement and direct marketing.  Developing Customer Profiles  RFM Score In this case, I'll start by ranking households using RFM score -- the Recency, Frequency, and Monetary value of their transactions.We rank each household on a scale of 1-5, with 5 being the most 'valuable' households in each of the Recency, Frequency, and Monetary domains.By using pandas .qcut(), we can easily create even-sized quantile bins.  Recency We should take note that our sample group are 'frequent purchasers'; so this metric could be misleading in terms of pushing a model or pipeline onto live or new data -- we would have to develop a different means of defining recency in that case.Instead of using .qcut(), I will assign a score of 5 to any household which made a purchase within the last two weeks of the data.<jupyter_code># find the last day of purchase for each household
last_days = merged.groupby('household_key')['DAY'].max()
last_days.head()<jupyter_output><empty_output><jupyter_text>Plotting Original Recency Distribution<jupyter_code>print(merged['DAY'].min(),merged['DAY'].max())
# we have 608 days of data<jupyter_output>101 708<jupyter_text>We have 608 days of data left after truncations. We know our households are supposed to be 'frequent' shoppers at our stores. How recently did each one purchase?<jupyter_code># plot the range 101 to 708+
plt.figure(figsize=(14,4))
plt.title('Distribution of Last day of Purchase')
last_days.hist(bins=61)
plt.xticks(range(100,730,15), rotation=45)
plt.axvline(last_days.mean(), color='red', label=f'mean: {last_days.mean()}')
plt.axvline(last_days.median(), color='orange', label=f'median: {last_days.median()}')
plt.legend()
plt.show()
#<jupyter_output><empty_output><jupyter_text>last_days.describe() This is clearly not a normal distribution. But before we panic, let's think about what the data is telling us. In this case, we have a lot of hits at the top end of the spectrum; right up close to the last days in the data. We see that **50% of the households actually had a last purchase within the past week, divided at day 702**. Since we're considering sorting using the pd.qcut() method (which creates even class distributions for our scores by separating the column by quantiles), we might be placing certain households with fairly similar recency of purchase into different categories.It would make more sense to me if we classified all households who purchased in the past week or so as very-high frequency shoppers.  This issue is more so a problem with the methodology of RFM score in this instance; but we shouldn't corrupt our data simply to make things appear "normal" on the graph -- the underlying phenomena is what we seek to describe -- in an interpretable, accurate, and relevant way. Let's compare the two options visually, and then I'll describe the meaning of boundaries I would use.  Plotting Distributions for alternative Recency Score Calculations<jupyter_code>plt.subplots(2,1,figsize=(20,5))
plt.suptitle('Comparing Potential Recency-Metric Ranking Distributions')

plt.subplot(2,1,1)
plt.title('Quantile Based')
# ALTERNATIVE ONE; USE THE QUANTILES
R = pd.qcut(last_days, 5, labels=[1,2,3,4,5])
sns.histplot(R)

plt.ticks = range(1,5)
plt.subplot(2,1,2)
# ALTERNATIVE TWO; USE BUSINESS-LANGUAGE DECISION BOUNDARIES
plt.title('Using Arbitrary Boundaries')
## within 1 year; within 3 months; within one month; within 1 weeks...
R = pd.cut(last_days, [0, 525, 615, 675, 700, np.inf], labels=[1,2,3,4,5], ordered=True)
sns.histplot(R)

plt.tight_layout()
plt.show()<jupyter_output><empty_output><jupyter_text>In the first plot, since our distribution is so dense near the end, we have boundary conflicts on the discrete DAY axis! This results in improper binning; moreover, we have a significant number of households classified with the lowest possible score. This does not correlate with the real underlying phenomena. We could employ a transformation to reduce the boundary conflict, but I have opted with the second strategy because I believe they better represent business-facing values as the relevant R Score metric. They represent customers who purchased:- more than 183 days ago; (more than 6 months ago)- within the past 183 days; (in the past 6 months)- in the past 93 days;  (in the past 3 months)- the past 33 days;  (in the past month)- and the past 8 days, (in the past week)respectively. Basically I'm saying that if they've made a purchase **within the past week**, they should be a high-frequency customer, for sure! They're coming into our stores **very often**. Unfortunately (Fortunately!) for us as data scientists, that actually accounts for half of our data; a very high participation/frequency is displayed by this sample group. We have to move the boundary out to one month to pick up another quarter of the households, and to get almost half of that, we have to move to three months. If we bin this way we can explain our group labels much more easily to stakeholders. However, it bears re-iterating that **we don't know if this is representative of the behaviour of the underlying population**. In fact, based on the nature of the experiment, we're pretty sure it's not. The distribution of these scores is an artifact of our sampling method, but accurately represents the data we have.   Frequency Just like Recency, the frequency of purchase for our customers is likely to be high.We should have data which represents all the transactions by our distinct households over this same timeframe -- so there's an argument to be made to use number of baskets outright. However, this method of binning would punish customers who have come on over the course of the data, and are now become steady and frequent shoppers.This is easily a debatable approach, as we are supposed to have consistent information from all 2500 households over the data.<jupyter_code>num_baskets = merged.groupby('household_key')['BASKET_ID'].nunique()
sns.histplot(num_baskets); # first pass<jupyter_output><empty_output><jupyter_text>Another method of developing this rank (although it introduces a different kind of bias) would be to count the number of days each household is in the data, and divide by the number of baskets they purchased to find their purchases per day.  Potential Errors...For any customers who randomly made several purchases but did not become frequent shoppers, this method may be misleading; for customers who didn't purchase for a long stretch of time bounding on the beginning or the end of our dataset it might be misleading.The impacts of these errors should be somewhat accounted for by using the quantile separation of total spend for monetary scores, below.<jupyter_code>#take each household's day column
grouper = merged.groupby('household_key')['DAY']
# find their duration in transactions table
days_in_data = grouper.max() - grouper.min() + 1 #(no day 0 in our data)
# find their baskets per day value
transactions_per_day = num_baskets/days_in_data
sns.histplot(transactions_per_day)
plt.title('Histogram of Transactions per Day for households');<jupyter_output><empty_output><jupyter_text>This looks a lot like our original data -- I think that's a good thing? We can now apply a quantile separation to create an even class distribution for our Frequency metric.<jupyter_code>F = pd.qcut(transactions_per_day, 5, labels=[1,2,3,4,5])
sns.histplot(F)
plt.title('Distribution of Frequency rank for households');<jupyter_output><empty_output><jupyter_text>Monetary Finally, we should identify the Monetary rank for each household. Since we have other ranks for frequency and recency of purchase, this should probably just be some distribution of the total sales value.<jupyter_code>sns.histplot(merged.groupby('household_key')['SALES_VALUE'].sum())
plt.title('Histogram of Total Spend by household');<jupyter_output><empty_output><jupyter_text>To prevent any boundary conflicts etc., I'm going to log transform the sum of sales first. This doesn't seem necessary because .qcut() is percentile based, but I don't think it will hurt, either.<jupyter_code>sns.histplot(np.log(merged.groupby('household_key')['SALES_VALUE'].sum()))
plt.title('Log Transformed Total Spend by household');
M = pd.qcut(np.log(merged.groupby('household_key')['SALES_VALUE'].sum()), 5, labels=[1,2,3,4,5])
sns.histplot(M)
plt.title('Distribution of households by Monetary Rank (1-5)');<jupyter_output><empty_output><jupyter_text>RFM Table<jupyter_code>df = pd.concat([R, F, M], axis=1)
df.columns = ['R', 'F', 'M']
df['RFM'] = df.sum(axis=1).astype(int)
df<jupyter_output><empty_output><jupyter_text>tada! Let's take a look at what our best customers distribution looks like.<jupyter_code>sns.histplot(df['RFM'], discrete=True)
plt.title('RFM Score Histogram for households in the data')
plt.show()<jupyter_output><empty_output><jupyter_text>There is tons of discussion to be had about what RFM labels indicate to us about our sample population. Most likely we would want a uniform distribution of scores; but I've made some assumptions in order to classify this data as accurately as I can, instead of pushing the data into a normal or uniform distribution. This is introducing bias, in exchange for reducing the error in our data collection process (some very low-recency customers vs. many very-high-recency customers).   Plotting the Category Sales by Customer RFM Score Ranks Now that we have defined the RFM score for each of our ~2500 households, let's take a cursory look at each group's sales participation over time...<jupyter_code>def compare_group_sales(merged, 
                        hh_agg, 
                        label_col='RFM',
                       date_col='datetime', 
                       resample_rule='BM',
                       sales_col='SALES_VALUE',
                      mean=True):
    
    """Using `household_key`,
        plot the sales for each unique label in `label_col` (from `hh_agg`), 

        using the `sales_col` `and date_col` found in `merged`

                                    """
    
    ## show the total sales at the top; adjust height ratios to emphasize the bottom plot
    plt.subplots(2,1,figsize=(16,6), sharex=True,gridspec_kw={'height_ratios': [1,3]})
    plt.suptitle(f'Sales by {resample_rule}')
    
    # total sales for all households in merged
    plt.subplots_adjust(hspace=0.1)
    plt.subplot(2,1,1)
    plt.title(f'Total Sum {sales_col}')
    merged.resample(resample_rule, on=date_col)[sales_col].sum().plot(label='Total')

    
    if mean == True:
    # divide the sales total over `resample_rule` by the number of households in each group, and plot.
        plt.subplot(2,1,2)
        plt.title(f'Mean Sum of Sales by {label_col}') 
        for rank in reversed(np.sort(hh_agg[label_col].unique())):
            lst = list(hh_agg[hh_agg[label_col]==rank].index)
            try:
                (merged[merged['household_key'].isin(lst)].resample(resample_rule, on=date_col)[sales_col].sum()/len(lst)).plot(label=f'{rank}')
            except:
                pass
            
    elif mean == False:
        plt.subplot(2,1,2)
        plt.title(f'Total Sum of Sales by {label_col}') 
        for rank in reversed(np.sort(hh_agg[label_col].unique())):
            lst = list(hh_agg[hh_agg[label_col]==rank].index)
            try:
                merged[merged['household_key'].isin(lst)].resample(resample_rule, on=date_col)[sales_col].sum().plot(label=f'{rank}')
            except:
                pass
        
    plt.legend(loc='upper left')

    plt.tight_layout()
    plt.show()
    
    
compare_group_sales(merged, df, mean=False)<jupyter_output><empty_output><jupyter_text>As we might expect, our most valuable customers do quite well; at the highest sales peak there, customers with a perfect score of 15 are making up around 1/3 of our sales.We see the echo of our sales spikes effectively 'filtered' down; our most valuable customers appear to be the ones who drive the spikes in our sales data. Focusing advertising on these customers makes sense; we might also consider that offering a 'lifestyle' of products similar to those purchased by our highest-grossing households might encourage other customers to purchase in a similar fashion.If we parsed the scores down into 5 bins once again into a normal distribution:<jupyter_code>customer_ranks = pd.cut(df['RFM'], bins=[0,6,9,13,15,np.inf], labels=[1,2,3,4,5], right=False).astype(int)
customer_ranks = pd.DataFrame(customer_ranks) ## THIS IS HACKY BRO
sns.histplot(customer_ranks, discrete=True)

print(customer_ranks.value_counts())<jupyter_output>RFM
3      924
2      579
4      506
5      263
1      213
dtype: int64<jupyter_text>And then compare their mean sales by score;<jupyter_code>compare_group_sales(merged, customer_ranks,mean=True);<jupyter_output><empty_output><jupyter_text>We see that our definition is maintained. However if we examine the total sales;<jupyter_code>compare_group_sales(merged, customer_ranks,mean=False);<jupyter_output><empty_output><jupyter_text>We can see that RFM bin 4 (score 13+14) has a nearly equivalent total sales value as RFM bin 5 (score 15). Bin 3 (score 10, 11, 12) has a similar total sales contribution as well. This is despite or because of significant class imbalances. I find this to be promising, as an effective way to group our customers; there is proportionality and distinction.  I'm going to tag these labels on as well:<jupyter_code>df['RFM Bins'] = customer_ranks<jupyter_output><empty_output><jupyter_text>RFM Function And the function; we'll bring all of our profile-development functions into one dataframe at the bottom of the notebook.<jupyter_code>def RFM_Score(merged):
    last_days = merged.groupby('household_key')['DAY'].max()
    R = pd.cut(last_days, [0, 525, 615, 675, 700, np.inf], labels=[1,2,3,4,5], ordered=True)
    num_baskets = merged.groupby('household_key')['BASKET_ID'].nunique()
    grouper = merged.groupby('household_key')['DAY']
    days_in_data = grouper.max() - grouper.min() + 1 #(no day 0 in our data)
    transactions_per_day = num_baskets/days_in_data
    F = pd.qcut(transactions_per_day, 5, labels=[1,2,3,4,5])
    M = pd.qcut(np.log(merged.groupby('household_key')['SALES_VALUE'].sum()), 5, labels=[1,2,3,4,5])
    df = pd.concat([R, F, M], axis=1)
    df.columns = ['R', 'F', 'M']
    df['RFM'] = df.sum(axis=1).astype(int)
    customer_ranks = pd.cut(df['RFM'], bins=[0,6,9,13,15, np.inf], labels=[1,2,3,4,5], right=False).astype(int)
    customer_ranks = pd.DataFrame(customer_ranks) ## THIS IS HACKY BRO
    df['RFM Bins'] = customer_ranks

    return df

all(df == RFM_Score(merged))<jupyter_output><empty_output><jupyter_text>Feature Engineering To create some more definition for our customer profiles, I'd like to aggregate the transaction data we have for each household. While it might be interesting to compare households using deep learning (perhaps a tensor composed of sales by month for each category, for example), I am still learning.I propose to add in the sales by category for each household.<jupyter_code>import warnings
warnings.filterwarnings('ignore')<jupyter_output><empty_output><jupyter_text>Section Sales<jupyter_code>def get_section_sales(merged) -> pd.DataFrame():

    # get dummies for each transaction row
    section_dummies = pd.get_dummies(merged['Section Labels'])
    # multiply each row by it's SALES VALUE
    section_sales = section_dummies.apply(lambda x: x * merged['SALES_VALUE'])

#             # check
#     idx = merged.index
#     print(all(section_sales.index == idx))

    section_sales.join(merged[['household_key']]).groupby('household_key').agg(sum)

    # add and group by household key, sum all rows from the dummy columns. this one looks funny but I think it's right
    section_sales = section_sales.join(merged[['household_key']]).groupby('household_key').sum()



    return section_sales

section_sales = get_section_sales(merged)
section_sales<jupyter_output><empty_output><jupyter_text>Because these product labels are meaningful, we could directly bin households based on their total spend on some indicative categories:    - alcohol- junk food- meat- home/family- produce etc.  Plotting Section Sales Distributions This function allows us to pass a boolean argument to the `log` parameter.<jupyter_code>def plot_category_distributions(section_sales, log=False):
    """hardcoded to 15 section categories; accepts hh_agg[[section_sales_cols]]"""
    
    plt.subplots(5,3,figsize=(16,6)) 
    # iterate over columns in section_sales...
    for idx, col in enumerate(section_sales):
        plt.subplot(5,3,idx+1)
        
        
        # show the log distributions
        if log==True:
            plt.title(f'log distribution of {col}')
            sns.histplot(np.log(section_sales[col]), label=f'{col}')
            
        # else show the untransformed distributions
        else:
            plt.title(f'distribution of {col}')
            sns.histplot(section_sales[col], label=f'{col}')

    plt.tight_layout()
    plt.show()

plot_category_distributions(section_sales, log=True)<jupyter_output><empty_output><jupyter_text>Once log-transformed, these distributions are fairly normal. I am shooting in the dark here, and would love the chance to learn from an experienced data scientist; my impression is that this engineered variance will allow our model to better distinguish the differences between households.   log transformed section sales, duration-based mean daily spend per category, mean basket spend As additional engineered features, I'm going to add the log-transformed sum of total sales, as well as the mean daily purchase per section.<jupyter_code># log_transformed sum of section sales
log_transformed_section_sales = np.log(section_sales).replace([np.inf, -np.inf], np.nan).fillna(0)
log_transformed_section_sales.columns = [x + '_log_transformed_sales_total' for x in log_transformed_section_sales.columns]
# daily mean spend per section 
# based on # of days in data;
# add Days in Data for each household; similar to Frequency transformation
grouper = merged.groupby('household_key')['DAY']
duration = (grouper.max() - grouper.min()) + 1 # no day 0
duration.name = 'Days in Data'
# divide each column by the household's Days in Data value
test = section_sales.merge(duration, on='household_key').drop('Days in Data', axis=1)
daily_mean_spend_per_category = test.iloc[:,1:].div(duration, axis=0)
daily_mean_spend_per_category.columns = [x + '_daily_mean_spend_per_category' for x in daily_mean_spend_per_category.columns]
features = pd.concat([duration, 
                      log_transformed_section_sales, 
                      daily_mean_spend_per_category, 
                     ], axis=1)
features.head(1)
df = df.merge(features, on='household_key')
df.head()<jupyter_output><empty_output><jupyter_text>Since we're including the mean and total spend per category in some form, I won't include the section sales proper.  `make_hh_agg` Now that we've generated some features about section spend, let's parse some more basic information about each household.<jupyter_code>def make_hh_agg(df) -> pd.DataFrame:


    # group transactions by household key and aggregate:
    hh_agg = df.groupby('household_key').agg(

    # total spend 
        total_spend=pd.NamedAgg(column="SALES_VALUE", aggfunc="sum"),
    # the total loyalty discount  
        total_loyalty=pd.NamedAgg(column="RETAIL_DISC", aggfunc="sum"), 
    # the total coupon match discount  
        total_coupon=pd.NamedAgg(column="COUPON_MATCH_DISC", aggfunc="sum"),
    # total quantity of items purchased   
        total_quantity=pd.NamedAgg(column='QUANTITY', aggfunc="sum"),

    # the number of baskets  
        num_baskets=pd.NamedAgg(column='BASKET_ID', aggfunc='nunique'),
    # number of unique of items purchased   
        unique_products=pd.NamedAgg(column='PRODUCT_ID', aggfunc='nunique'),

    # first day of purchase
        first_purchase=pd.NamedAgg(column='DAY', aggfunc='min'),
    # last day of purchase
        last_purchase=pd.NamedAgg(column='DAY', aggfunc='max'), 
                                                                )

    ### Deriving Avg. Basket Spend and Avg. Item Cost --

    # average basket spend
    hh_agg['avg_basket_spend'] = hh_agg['total_spend']/hh_agg['num_baskets']

    # average item cost (total spend/quantity)
    hh_agg['avg_item_cost'] = hh_agg['total_spend']/hh_agg['total_quantity']
    return hh_agg

make_hh_agg(merged).head(1)
df = df.merge(make_hh_agg(merged), on='household_key')

df.head()<jupyter_output><empty_output><jupyter_text>`load_hh_agg` Let's bring it all together.  Below is the function I've added to `dtcj` package.<jupyter_code>def load_hh_agg(df) -> pd.DataFrame:
    '''accept the merged table or a subset thereof and return relevant house-level aggregations'''

    def make_hh_agg(df) -> pd.DataFrame:            
        # group transactions by household key and aggregate:
        hh_agg = df.groupby('household_key').agg(

        # total spend 
            total_spend=pd.NamedAgg(column="SALES_VALUE", aggfunc="sum"),
        # the total loyalty discount  
            total_loyalty=pd.NamedAgg(column="RETAIL_DISC", aggfunc="sum"), 
        # the total coupon match discount  
            total_coupon=pd.NamedAgg(column="COUPON_MATCH_DISC", aggfunc="sum"),
        # total quantity of items purchased   
            total_quantity=pd.NamedAgg(column='QUANTITY', aggfunc="sum"),

        # the number of baskets  
            num_baskets=pd.NamedAgg(column='BASKET_ID', aggfunc='nunique'),
        # number of unique of items purchased   
            unique_products=pd.NamedAgg(column='PRODUCT_ID', aggfunc='nunique'),

        # first day of purchase
            first_purchase=pd.NamedAgg(column='DAY', aggfunc='min'),
        # last day of purchase
            last_purchase=pd.NamedAgg(column='DAY', aggfunc='max'), 
                                                                    )

        ### Deriving Avg. Basket Spend and Avg. Item Cost --

        # average basket spend
        hh_agg['avg_basket_spend'] = hh_agg['total_spend']/hh_agg['num_baskets']

        # average item cost (total spend/quantity)
        hh_agg['avg_item_cost'] = hh_agg['total_spend']/hh_agg['total_quantity']
        return hh_agg


    def add_RFM(merged):
        last_days = merged.groupby('household_key')['DAY'].max()
        R = pd.cut(last_days, [0, 525, 615, 675, 700, np.inf], labels=[1,2,3,4,5], ordered=True)
        num_baskets = merged.groupby('household_key')['BASKET_ID'].nunique()
        grouper = merged.groupby('household_key')['DAY']
        days_in_data = grouper.max() - grouper.min() + 1 #(no day 0 in our data)
        transactions_per_day = num_baskets/days_in_data
        F = pd.qcut(transactions_per_day, 5, labels=[1,2,3,4,5])
        M = pd.qcut(np.log(merged.groupby('household_key')['SALES_VALUE'].sum()), 5, labels=[1,2,3,4,5])
        df = pd.concat([R, F, M], axis=1)
        df.columns = ['R', 'F', 'M']
        df['RFM'] = df.sum(axis=1).astype(int)
        customer_ranks = pd.cut(df['RFM'], bins=[0,6,9,13,15, np.inf], labels=[1,2,3,4,5], right=False).astype(int)
        customer_ranks = pd.DataFrame(customer_ranks) ## THIS IS HACKY BRO
        df['RFM Bins'] = customer_ranks

        return df
    
   
    def transformed_section_sales(merged):
        
        def get_section_sales(df) -> pd.DataFrame():

            idx = df.index
            # get dummies for each transaction row
            section_dummies = pd.get_dummies(df['Section Labels'])

            # multiply each row by it's SALES VALUE
            section_sales = section_dummies.apply(lambda x: x * df['SALES_VALUE'])
        #     print(all(section_sales.index == idx))
            # add and group by household key, sum all rows from the dummy columns
            section_sales = section_sales.join(df[['household_key']]).groupby('household_key').agg(sum)
            return section_sales
        
        # create section sales
        section_sales = get_section_sales(merged)
        
       # log_transformed sum of section sales
        log_transformed_section_sales = np.log(section_sales).replace([np.inf, -np.inf], np.nan).fillna(0)
        log_transformed_section_sales.columns = [x + '_log_transformed_sales_total' for x in log_transformed_section_sales.columns]

        # daily mean spend per section 
        # based on # of days in data;
        # add Days in Data for each household; similar to Frequency transformation
        grouper = merged.groupby('household_key')['DAY']
        duration = (grouper.max() - grouper.min()) + 1 # no day 0
        duration.name = 'Days in Data'
        # divide each column by the household's Days in Data value
        test = section_sales.merge(duration, on='household_key').drop('Days in Data', axis=1)
        daily_mean_spend_per_category = test.div(duration, axis=0)
        daily_mean_spend_per_category.columns = [x + '_daily_mean_spend_per_category' for x in daily_mean_spend_per_category.columns]
        return pd.concat([log_transformed_section_sales, daily_mean_spend_per_category], axis=1)
      
    
    ### RUNTIME
    
    hh_agg = pd.concat([make_hh_agg(df), transformed_section_sales(df), add_RFM(df)], axis=1)

    return hh_agg

hh_agg = load_hh_agg(merged)
hh_agg.shape, df.shape<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.0921127562642369, 'fraction_numerical': 0.011247152619589976, 'mean_word_length': 4.132809645597369, 'pattern_counts': {'":': 0, '<': 96, '<?xml version=': 0, '>': 101, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 6, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '24', 'dir': 'jupyter-structured-clean-dedup', 'id': '441073', 'license': 'MIT', 'n_tokens_mistral': 8599, 'n_tokens_neox': 8034, 'n_words': 2959, 'path': '1.3 -- Creating Customer Profiles as `hh_agg`.ipynb', 'repo_name': 'np1919/DTCJ', 'size': '28106'}