<jupyter_start><jupyter_text>Attribute Description:Attribute | Description----------|-------------`Invoice No` | Invoice ID, encoded as Label`StockCode` | Unique code per stock, encoded as Label`Description` | The Description, encoded as Label`Quantity` | Quantity purchased`InvoiceDate` | Date of purchase`UnitPrice` | The target value, price of every product`CustomerID` | Unique Identifier for every country`Country` | Country of sales, encoded as Label Target`UnitPrice` is the target. Performance MeasureRMSE (Root Mean Square Error)<jupyter_code>import pandas as pd
import numpy as np
import matplotlib.pyplot as plt
import seaborn as sns

from scipy import stats

from sklearn import preprocessing

import utility_functions as uf<jupyter_output><empty_output><jupyter_text>Initial Data Analysis<jupyter_code>data = pd.read_csv("data/Train.csv")
test = pd.read_csv("data/Test.csv")
data.head()
print("Shape of:")
print("-"*10)
print("\t Training data:", data.shape)
print("\t Test data:", test.shape)<jupyter_output>Shape of:
----------
	 Training data: (284780, 8)
	 Test data: (122049, 7)<jupyter_text>Drop duplicates<jupyter_code>print(f"There are {data[data.duplicated(keep=False)].shape[0]} duplicates in training data")
data = data.drop_duplicates(ignore_index=True)<jupyter_output><empty_output><jupyter_text>Missing Value Check and Type casting<jupyter_code>print("Number of Nulls in Training data:",
      data.isna().sum()\
      .sum())
print("Number of Nulls in Test data:",
        test.isna().sum()\
        .sum())
data.info()
###### change dtypes to appropriate data types as applicable #######
categorical_cols = ['InvoiceNo', 'StockCode', 'Description', 'CustomerID', 'Country']

# convert to string
data[categorical_cols] = data.loc[:,categorical_cols].astype('object')
# convert to datetime
data['InvoiceDate'] = pd.to_datetime(data.loc[:,'InvoiceDate'])
data.describe()
# drop(columns=['year','month','day_of_week','hour','minutes','day_of_month']).describe()
# corr = data[['Quantity', 'UnitPrice']].corr(method='spearman')
corr = data.corr(method='spearman')
mask = np.triu(np.ones_like(corr, dtype=bool))

sns.heatmap(corr, annot=True, mask=mask, cbar=False)


plt.title("Spearman Correlation plot")
plt.tight_layout()
plt.xticks(ticks=[0,1], labels=['Quantity', ''], )
plt.yticks(ticks=[0,1], labels=['','UnitPrice'])
plt.show()<jupyter_output><empty_output><jupyter_text>Observations- There were $5093$ duplicate records in training data taht have been dropped.- No missing values have been found.- Both `Quantity` and `UnitPrice` have outliers. The severity of which needs to be further analysed. Although we will be limited in our ability to deal with outliers in `UnitPrice` since it is the target.- Minimum for `Quantity` is -80995 which seems improbable and also observe that maximum is 80995. It will be further analysed.  Engineer Temporal Features from `InvoiceDate`<jupyter_code>data['year'] = data.InvoiceDate.dt.year
data['month'] = data.InvoiceDate.dt.month
data['day'] = data.InvoiceDate.dt.dayofweek # Monday=0, Sunday=6
data['hour'] = data.InvoiceDate.dt.hour
data['minutes'] = data.InvoiceDate.dt.minute
data['day_of_month'] = data.InvoiceDate.dt.day
data.head()<jupyter_output><empty_output><jupyter_text>General Hypothesis  $Q$: Does every Invoice has only one Associated customer?$A:$ Yes every Invoice has only one unique customer associated with it. As shown below:<jupyter_code># No of unique InvoiceID
print("Number of unique invoices: ", data.InvoiceNo.nunique())

print('-'*70)

print("Number of Invoices with Number of unique customers !=1 :",
     (data.groupby('InvoiceNo')['CustomerID'].nunique() != 1).sum()
     )<jupyter_output>Number of unique invoices:  20971
----------------------------------------------------------------------
Number of Invoices with Number of unique customers !=1 : 0<jupyter_text>UNivariate ANalysis of Variables  `UnitPrice` $\Longrightarrow$  Target variable<jupyter_code>data.UnitPrice.describe()
# iqr 
# UNITPrice
q1, q3 = np.percentile(data.UnitPrice, [25, 75])
UP_iqr = q3 - q1
UP_max_threshold = q3 + 1.5 * UP_iqr

## LOG of UnitPrice
q1, q3 = np.percentile(np.log(data.UnitPrice + np.full_like(data.UnitPrice, fill_value=0.0001)),
                       [25, 75])
iqr = q3 - q1

log_max_threshold = q3 + 1.5 * iqr
log_median = np.percentile(np.log(data.UnitPrice + np.full_like(data.UnitPrice, fill_value=0.0001)),
                      50)
fig, ax = plt.subplots(1, 4, figsize=(20,5))


plt.subplot(ax[0])
# sns.boxplot(x=data.UnitPrice)
sns.kdeplot(x=data.UnitPrice)
plt.yticks([])
plt.ylabel('')
plt.title("Many Outliers present.\n Median = 1.95, Q3 = 3.75")

plt.subplot(ax[1])
sns.boxplot(x=np.log(data.UnitPrice + np.full_like(data.UnitPrice, fill_value=0.0001)))
plt.xlabel('')
plt.title("Box plot for Log of UnitPrice")


plt.subplot(ax[2])
plt.plot(data.UnitPrice, np.ones_like(data.UnitPrice),
         'o', alpha=0.5)

plt.vlines(UP_max_threshold, 0.96, 1.04, linestyles='dashed', colors='g', label='q3+1.5*IQR = 7.5')
plt.yticks([])
plt.legend()
# plt.xlim(right=5000)
plt.title("UnitPrice distribution")

plt.subplot(ax[3])
plt.plot(np.log(data.UnitPrice + np.full_like(data.UnitPrice, fill_value=0.0001)),
         np.ones_like(data.UnitPrice),
         'o', alpha=0.4, linewidth=0.5)

plt.vlines(log_max_threshold, 0.96, 1.04, linestyles='dashed', colors='g', label=f'q3+1.5*IQR = {round(max_threshold,2)}')
plt.vlines(log_median, 0.96, 1.04, linestyles='dashed', colors='lime', label=f'median = {round(median,2)}')
plt.yticks([]) # remove y-ticks  
plt.legend(loc='upper left')
# plt.xlim(right=5000)
plt.title(f"UnitPrice (Log transformed) distribution")

plt.show()

a = np.log(data.UnitPrice + np.full_like(data.UnitPrice, fill_value=0.0001))

print(f"{round(len(data.UnitPrice[data.UnitPrice>UP_max_threshold])/len(data.UnitPrice) * 100,0)}\
% of observations are above Max threshold in UnitPrice")

print(f"{round(len(a[a>log_max_threshold])/len(a) * 100,2)}\
% of observations are above Max threshold in Log of UnitPrice")<jupyter_output>9.0% of observations are above Max threshold in UnitPrice
0.37% of observations are above Max threshold in Log of UnitPrice<jupyter_text>Observation from analysis of `UnitPrice`- Taking Log reduced the Outliers from $\approx 9\%$ to $0.37\%$.- Log transformation will be used on UnitPrice, $\because$ it reduces the number of outliers significantly.<jupyter_code># jitter added
data['log_UnitPrice'] = np.log(data.UnitPrice + np.full_like(data.UnitPrice, fill_value=0.01))<jupyter_output><empty_output><jupyter_text>`Quantity`From above Initaial analysis wkt. there are some discrepencies in Quantity values. We will explore them in depth.<jupyter_code>print(f"Number records where quantity is zero: {data[data.Quantity==0].shape[0]}")
print(f"Number of records where quantity < zero: {data[data.Quantity<0].shape[0]}")<jupyter_output>Number records where quantity is zero: 0
Number of records where quantity < zero: 6153<jupyter_text>Analysing `-ve` order QuantitiesPossibilities are that the negative quantities are either- mistakenly recorded as negative quantity or- result of negative proration such as  return orders [(refer)](https://knowledgecenter.zuora.com/BB_Introducing_Z_Business/How_Do_I_._._./How_do_I_handle_a_negative_invoice%3F).It has been noticed that negative quantities are present even in the test set, thus reducing the chances of it being being data entry error.\This also takes away the possibility of dropping such records from analysis. Although the negative quatities are very small fraction of both the training as well as test datasets(as shown below).<jupyter_code>print("Percentage of observations with negative quantity in Training data:",
     round(data[data.Quantity < 0].shape[0] / data.shape[0] * 100
         ,2)
     )

print("-"*100)

print("Percentage of observations with negative quantity in Test data:",
     round(test[test.Quantity < 0].shape[0] / test.shape[0] * 100
         ,2)
     )
data['Quantity_pos'] = data.Quantity.apply(lambda x: 'POS' if x>=0 else 'NEG')
fig, ax = plt.subplots(1,2, figsize=(10, 4), sharey=True)
plt.subplot(ax[0])
sns.boxplot(y='Quantity_pos', x='UnitPrice', data=data, hue='Quantity_pos')
plt.xlim(right=5000)
plt.title("Box plot of UnitPrice")

plt.subplot(ax[1])
sns.boxplot(y='Quantity_pos', x='log_UnitPrice', data=data, hue='Quantity_pos')
plt.ylabel('')
plt.yticks([])
plt.title("Box plot of UnitPrice(log applied)")

plt.suptitle("Difficult to visually differenciate between the Groups (Positive and negative Quantity)")

plt.tight_layout()
plt.show()
data.groupby('Quantity_pos')['UnitPrice'].describe().round(3)<jupyter_output><empty_output><jupyter_text>Statistical test (Mann-Whitney U Test)- Although Relationship is not exactly Linear (that would indicate Normal distribution), but it is Linear enough give the large sample size.- Levene's test of Homogeneity of variance is violated. But if ratio of Largest: Smallest group is reasonable ($\approx$ 1.5 times), then violation of this assumption should not cause any major issue.However, the ratio is too high ($\approx$ 45 times). Thus violation is serious.Keeping in view above points the Non - parametric variant of the t-test, **Mann-Whitney U Test** is used.  Normality test<jupyter_code>stats.probplot(data.log_UnitPrice, plot=plt)
plt.title(f"Normal Probability Plot\nNo of Observations: {data.log_UnitPrice.shape[0]}")
plt.ylabel("Log (UnitPrice)")
plt.show()<jupyter_output><empty_output><jupyter_text>Levene's test of Homogeneity<jupyter_code>stat, p = stats.levene(data.UnitPrice[data.Quantity_pos=='POS'],
                      data.UnitPrice[data.Quantity_pos=='NEG'])
is_significant = lambda p_value: 'Significant' if p_value<=0.05 else 'NOT Significant'
print(f"Levenes test is {is_significant(p)}, with p = {round(p, 4)}.")

if is_significant(p).strip().lower() == 'significant':
    print("Thus, assumption of Homogeneity of Variance is Violated.")
else:
    print("Thus, assumption of Homogeneity of Variance is Not violated. ")
print(f"Ratio of Largest: Smallest = \
{round(data[data.Quantity_pos=='POS'].shape[0]/data[data.Quantity_pos=='NEG'].shape[0],0)}"
)<jupyter_output>Ratio of Largest: Smallest = 45.0<jupyter_text>Mann-Whitney U Test$H_0$: The distribution of UnitPrice is **same across categories** ie. POS and NEG Quantitites.\$H_a$: The distribution of UnitPrice is **not same across categories** ie. POS and NEG Quantitites.<jupyter_code>stat, p = stats.mannwhitneyu(x=data.UnitPrice[data.Quantity_pos=='POS'], 
                   y=data.UnitPrice[data.Quantity_pos=='NEG'],
                   alternative='two-sided')
if p < 0.05:
    print(f"Mann-Whitney test is Significant (p={round(p, 4)}), meaning the distribution of UnitPrice is sepearate across groups.")
else:
    print(f"The test is Not significant (p={round(p, 4)}), meaning the distribution of UnitPrice is same across groups. ")<jupyter_output>Mann-Whitney test is Significant (p=0.0), meaning the distribution of UnitPrice is sepearate across groups.<jupyter_text>Observation from Quantities divided into POS and NEG categories- Negative quantities are result of **negative proration**.- There is **statistically significant difference** in distribution of UnitPrice for **Postive and negative quantities**.- Thus `Quantity_pos` **may** prove to be a **useful feature** to any future predcitive model.<jupyter_code>plt.scatter(data.Quantity, data.log_UnitPrice, alpha=0.3, color='r')
plt.xlabel('Quantity')
plt.ylabel("Log (UnitPrice)")
plt.xlim(-1000, 1500)
rho = r"$\rho$"
plt.title(f"Very weak Linear correlation with Target variable\n{rho} = {round(stats.pearsonr(data.Quantity, data.log_UnitPrice)[0], 2)}")
plt.show()<jupyter_output><empty_output><jupyter_text>Treating `Quantity` distribution$\rightarrow$ Check Natural distribution \$\rightarrow$ Scale by Mean/Median (depending on outliers in distribution) \$\rightarrow$ Power Transformation \$\rightarrow$ Thresholding\$\rightarrow$ Which observations are above Threshold\$\rightarrow$ **Log Transformation** is **not an option** due to Negative values in the quantity data.<jupyter_code>quant_YJ = preprocessing.PowerTransformer().fit_transform(data.Quantity.values.reshape(-1,1))
quant_scaled = preprocessing.StandardScaler().fit_transform(data.Quantity.values.reshape(-1,1))
def median_transformation(variable):
    '''
         z = (x - u) / s
    '''
    median = np.median(variable)
    mad = stats.median_abs_deviation(variable, nan_policy='omit')
    
    return pd.Series(variable).apply(lambda x: (x-median)/mad)
    
quant_med_scaled = median_transformation(data.Quantity)
### Box plots #########

fig, ax = plt.subplots(1,3, figsize=(15, 5))

plt.subplot(ax[0])
sns.boxplot(x=data.Quantity)
x_lim = (-100, 100)
plt.xlim(x_lim)
extreme_outliers = r"$\bf{Extreme \ Outliers}$"
plt.title(f"Quantitiy Distribution - zoomed in {x_lim}.\n \
Presence of {extreme_outliers}.\n \
Min = {data.Quantity.min().round(2)}, Median = {data.Quantity.median()}, Max = {data.Quantity.max()}")
plt.xlabel('')

plt.subplot(ax[1])
sns.boxplot(x=quant_YJ)
x_lim = (-10, 20)
plt.xlim(x_lim)
plt.title(f"Quantity Yeo Johnson transformed. - zoomed in {x_lim}.\n \
Min = {quant_YJ.min().round(2)}, Median = {np.median(quant_YJ).round(2)}, Max = {quant_YJ.max().round(2)}"
)
plt.xlabel('')

plt.subplot(ax[2])
sns.boxplot(x=quant_med_scaled)
x_lim = (-100, 100)
plt.xlim(x_lim)
plt.title(f"Quantity Median scaled - zoomed in {x_lim}.\n \
Min = {quant_med_scaled.min().round(2)}, Median = {quant_med_scaled.median().round(2)}, Max = {quant_med_scaled.max().round(2)}"
)
plt.xlabel('')

plt.tight_layout()
# plt.show()


### Distribution Plots ####
fig, ax = plt.subplots(1,4, figsize=(20,5))
plt.subplot(ax[0])
# x_lim = (-100, 100) 
sns.kdeplot(data.Quantity)
plt.xlabel('')
# plt.xlim(x_lim)
plt.yticks([])
plt.ylabel('')
plt.title("Quantity")

plt.subplot(ax[1])
x_lim = (-10, 20) 
sns.kdeplot(pd.Series(quant_YJ.reshape(len(quant_YJ))))
plt.xlim(x_lim)
plt.yticks([])
plt.ylabel('')
plt.title("Yeo Johnson Transformed")

plt.subplot(ax[2])
x_lim = (-10, 20) 
sns.kdeplot(pd.Series(quant_scaled.reshape(len(quant_scaled))))
plt.xlim(x_lim)
plt.yticks([])
plt.ylabel('')
plt.title("Standard scaled")

plt.subplot(ax[3])
# x_lim = (-100, 100) 
sns.kdeplot(quant_med_scaled)
plt.xlabel('')
# plt.xlim(x_lim)
plt.yticks([])
plt.ylabel('')
plt.title("Median Scaled")
# sns.kdeplot(pd.Series(quant_med_scaled.reshape(len(quant_med_scaled))))

plt.suptitle("Most of the observations are concentrated around a small range. Extreme Outliers need to be treated")
plt.show()
print('Range for Quantity: ', data.Quantity.max() - data.Quantity.min())
print('Range for Quantity - Yeo Johnson Transformed: ', (quant_YJ.max() - quant_YJ.min()).round(2))
print('Range for Quantity - Median Scaled: ', quant_med_scaled.max() - quant_med_scaled.min())
print('Range for Quantity - Standard Scaled: ', (quant_scaled.max() - quant_scaled.min()).round(2))<jupyter_output>Range for Quantity:  161990
Range for Quantity - Yeo Johnson Transformed:  543.86
Range for Quantity - Median Scaled:  40497.5
Range for Quantity - Standard Scaled:  546.41<jupyter_text>Thresholded data- Thresholded data- Yeo Johnson Transformed thresholded data- Median transformed thresholded data<jupyter_code># Thresholded Quantity data
quant_thresholded = uf.threshold_data(data.Quantity)['th_data']

# Yeo Johnson Transformed thresholded data
quant_th_YJ = preprocessing.PowerTransformer().fit_transform(quant_thresholded.values.reshape(-1,1))\
            .reshape(len(quant_thresholded))

# median scaled quantity thresholded
quant_med_th = uf.threshold_data(quant_med_scaled)['th_data'].values.reshape(len(quant_med_scaled))
fig, ax = plt.subplots(1,3, figsize=(15,5))

plt.subplot(ax[0])
sns.kdeplot(quant_thresholded)
plt.title(f"Thresholded Quantity data.\n\
Minimum = {quant_thresholded.min().round(2)}, Median = {quant_thresholded.median().round(2)}, Maximum = {quant_thresholded.max().round(2)}"
)
plt.yticks([])
plt.ylabel('')


plt.subplot(ax[1])
sns.kdeplot(quant_th_YJ)
plt.title(f"Yeo Johnson Transformed Thresholded data.\n\
Minimum = {quant_th_YJ.min().round(2)}, Median = {np.median(quant_th_YJ).round(2)}, Maximum = {quant_th_YJ.max().round(2)}"
)
plt.yticks([])
plt.ylabel('')


plt.subplot(ax[2])
sns.kdeplot(quant_med_th, color='lime')
plt.title(f"Median Transformed Thresholded data.\n\
Minimum = {quant_med_th.min().round(2)}, Median = {np.median(quant_med_th).round(2)}, Maximum = {quant_med_th.max().round(2)}"
)
plt.yticks([])
plt.ylabel('')

plt.tight_layout()
plt.show()<jupyter_output><empty_output><jupyter_text>Takeaway from `Quantity`- Quantity NEG/POS category can be a good feature.- Standard scaled and Median Transformed Quantity thresholded data will be experimented with while modelling.<jupyter_code>data['Quantity_pos'] = data.Quantity.apply(lambda x: 'POS' if x>=0 else 'NEG')<jupyter_output><empty_output><jupyter_text>`Country`<jupyter_code>print(f"There are data from {data.Country.nunique()} countries.")
data.InvoiceNo.nunique()
len(set(data.InvoiceNo.unique())\
    .intersection(set(test.InvoiceNo.unique()))
   )
data.InvoiceDate = pd.to_datetime(data.InvoiceDate)
# data.rename(columns={'day':'day_of_week'}, inplace=True)

data.Description.nunique()
data[data.InvoiceNo==6141].shape
data[data.InvoiceNo==6141]<jupyter_output><empty_output><jupyter_text>Basic Mean modelling<jupyter_code>data.UnitPrice.mean()
predicted =<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.15224597096811063, 'fraction_numerical': 0.02091667619156475, 'mean_word_length': 5.67136866183759, 'pattern_counts': {'":': 0, '<': 77, '<?xml version=': 0, '>': 74, 'https://': 1, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '19', 'dir': 'jupyter-structured-clean-dedup', 'id': '259964', 'license': 'MIT', 'n_tokens_mistral': 6410, 'n_tokens_neox': 5886, 'n_words': 1407, 'path': 'analysis.ipynb', 'repo_name': 'gunnerVivek/Foretelling-Retail-price-using-Multivariate-Regression', 'size': '17498'}