<jupyter_start><jupyter_text>Code SectionI will use this Jupyter notebook to stage & review all data transformations & formulas needed for this assignment<jupyter_code>import pandas as pd
import requests
import io
import numpy as np
import os
import matplotlib.pyplot as plt
%matplotlib inline
import seaborn as sns
os.chdir(r'C:\Users\lwhieldon\OneDrive - SC&H Group\Administrative\School Materials\DATA 602 - Into to Data Analysis and ML\IntroToDataAnalysis\Images')<jupyter_output><empty_output><jupyter_text>Getting the data & Initial ObservationsLet's import the csv file containing the data into a pandas dataframe. As you can see with the first 5 rows of data, each observation consists of a single order id. There are 14 attributes (Order Date, Customer ID, Segment, City, State, Postal Code, Region, Product ID, Category, Sub-Category, Sales, Quantity of units sold, Discount, and Profit). We will drop Row ID from the dataset as it is redundant.<jupyter_code>url = 'https://raw.githubusercontent.com/Lwhieldon/IntroToDataAnalysis/master/Assignments/Assignment%201/Sample%20-%20Superstore%20Data.csv'

download = requests.get(url).content

df = pd.read_csv(io.StringIO(download.decode('utf-8')))
df = df.drop(["Row ID"], axis=1)
df.head()<jupyter_output><empty_output><jupyter_text>Let's convert order date to a datetime format and create a new column for Year since the business would rather look at trends for future profit years. We are going to treat year as a categorical variable (even though it's numeric in the dataframe)<jupyter_code>df['Order Year']=pd.DatetimeIndex(df['Order Date']).year
df = df.drop(['Order Date'], axis = 1)

#  Here's a snapshot of the info method so you can see the data type of each attribute, 
# the number of records, and the memory of the df.
# Notice also that there are no non-null values which is great from a data clean up perspective
df.info()
#Let's describe the data here to better understand the summary of the numerical attributes
df.describe()<jupyter_output><empty_output><jupyter_text>Let's look specifically look at profit to see the distribution of profit in our dataset<jupyter_code>df['Profit'].describe()<jupyter_output><empty_output><jupyter_text>Whoa! It looks like some prices are super high & low; this could throw off our model. Let's reduce the dataset to only include prices that fall with the 95% percentile to help reduce the outliers.<jupyter_code>q_low = df["Profit"].quantile(0.05)
q_hi  = df["Profit"].quantile(0.94)
df = df[(df["Profit"] < q_hi) & (df["Profit"] > q_low)]
df['Profit'].describe()
df_profit= df['Profit']
df_profit.hist(bins=20,grid=False)
plt.title('Distribution Profit',fontweight='bold')
plt.xlabel('Profit',fontsize=10)
plt.ylabel('Frequency',fontsize=10)
plt.locator_params(axis='x', nbins=20)
# plt.show()
plt.savefig('profit_histogram.png')<jupyter_output><empty_output><jupyter_text>Ok, now we see somewhat of a right skewed distribution but not too bad. Let's continue on... Let's look at a histograms of each data element to get a better sense of our features against profit.<jupyter_code>df.hist(bins=75, figsize=(20,15))
plt.show()<jupyter_output><empty_output><jupyter_text>Here are a few things I notice right away:  The majority of orders have about 2-3 items, but the population has a slightly right skewed distribution.  Sales has a large right skewed distribution. This makes sense since most of the units sold are small in value.  The total number of orders by year slightly increases with each year. (we will treat this as categorical)  Most of the orders are not provided a discount; others are given a 2% discount  We can ignore Postal Code as it is qualitative in nature rather than quantitativeLet's look at sales a little more closely so we can see how detailed the skew is<jupyter_code>df_profit= df['Sales']
df_profit.hist(bins=30,grid=False, figsize=(10,10))

plt.title('Distribution Sales',fontweight='bold')
plt.xlabel('Sales',fontsize=8)
plt.ylabel('Frequency',fontsize=10)
plt.locator_params(axis='x', nbins=25)
# plt.show()
plt.savefig('sales_histogram.png')<jupyter_output><empty_output><jupyter_text>Analyze Quantitative AttributesIn our dataset, let's look at how each attribute correlates with profit<jupyter_code>corr_matrix = df.corr()
corr_matrix["Profit"].sort_values(ascending=False)<jupyter_output><empty_output><jupyter_text>Correlation Observations:  Not surprisingly, Sales is positively correlated to profit; discounts are negatively correlated to profit, which also makes sense in that the amount of discounts you give a customer will deter higher profits.  Interestingly, quantity is slightly correlated. One would think there would be more correlation in that the more you sell products the more profit you would gain but our dataset does not provide us with that conclusion.   Order Year appears to have no real correlation to Profit.Let's also check for correlation using a scatter plot. Let's only look at profit, sales, quantity, and discount<jupyter_code>from pandas.plotting import scatter_matrix
attributes_to_plot = ["Profit","Sales","Quantity","Discount"]

scatter_matrix(df[attributes_to_plot], figsize=(12,8))<jupyter_output><empty_output><jupyter_text>Interesting results. Strong horizontal lines exist for quantity, discount, and year. Let's keep this in mind as we begin to build our model.Otherwise based on what we can see, the attribute that looks best to predict profit in this dataset is sales. Let's look at this histogram a little closer<jupyter_code>hist = df.plot(kind="scatter",x="Sales",y="Profit",alpha=0.1)
hist.figure.savefig('sales_to_profit_histogram.png')<jupyter_output><empty_output><jupyter_text>The scatterplot reveals a few things:  You can see the makings of a STEEP trend, but there's a lot of data that causes a faning affect. There are a few outliers that dip under the main trend, but for the most part there's a vague but perceptible upward trend.  Given the slight 'v' shape of the data, we might want to consider adjusting our final model to account for this interesting trend  Analyze Qualitative AttributesIn our data, we have several qualitative attributes:  Order ID  Customer ID  Customer Segment  Postal Code  City  State  Region  Product ID  Product Category  Product SubCategory  Order Year Some of these attributes are 'hierarchical' in nature, meaning the attributes assigned to an observation can be grouped into categories. For example, we can remove the product id and customer id because of its collinearity with product & customer categories. Since Order id and rowid is unique per order, we can drop that as well. We can remove the 'Product Sub-category' as this is colinear to Product Category too. Another element we will remove from this model is lower level location attributes, such as city, state, and postal code; they too have a colinear relationship with region. Now let's look at Customer Segment, Region, Product Category, and Order Year the new qualitative attributes we will use in our model:<jupyter_code>cleaned_df = df.drop(['Product ID','Sub-Category','Order ID','City','State','Postal Code','Customer ID'], axis = 1)
print("Segment Categories: ",cleaned_df['Segment'].unique())
print("Region Categories: ",cleaned_df['Region'].unique())
print("Product Categories: ",cleaned_df['Category'].unique())
print("Order Years: ",cleaned_df['Order Year'].unique())<jupyter_output>Segment Categories:  ['Consumer' 'Corporate' 'Home Office']
Region Categories:  ['South' 'West' 'Central' 'East']
Product Categories:  ['Furniture' 'Office Supplies' 'Technology']
Order Years:  [2018 2017 2016 2019]<jupyter_text>Let's explore the features<jupyter_code>fig, axes = plt.subplots(1, 4,figsize=(15,5),sharey='row')
fig.text(0.04, 0.5, 'common Y', va='center', rotation='vertical')
plt.subplots_adjust(wspace=0)
sns.violinplot(df["Segment"],df["Profit"], ax=axes[0], cut=0) #ztest
sns.violinplot(df["Region"],df["Profit"], ax=axes[1], cut=0) #ztest
sns.violinplot(df["Category"],df["Profit"], ax=axes[2], cut=0) #ztest
sns.violinplot(df["Order Year"],df["Profit"], ax=axes[3], cut=0) #ztest
plt.suptitle('Distribution of Profit by features', ha = 'left',fontweight='bold',fontsize=25,x=0.125,y=.9,va='baseline')

plt.savefig('profit_distributiion_violins.png')

#Code body for violin plots respectfully derived from this project: https://github.com/a-woodbury/A-House-with-a-View<jupyter_output><empty_output><jupyter_text>Violin plots are great when you want to review numeric data (Profit) and its distribution among groups of data; it also shows the interquartile range of the profit data within each group & sets them side by side for easy comparisons. I love how easily you can see density or frequency of the data for each group.For the most part, the means & interquartile ranges for profit are similar for all categories in Segment, Region, and Order Year. The shapes of the distributions between each category in Segment, Region, and Order are also similar, indicating that there are is a small frequency of really large values making the smoothed distributions (i.e. the shape of the violins) look really stretched.The category feature contains some interesting insights: It looks like Office Supplies have really low profits for the most part (hence the fat shape of the violin); furniture & technology are very thin and give the indication that they have higher price points than that of office supplies. This make sense given that furniture & technology tend to be more expensive.Since the product category plays a huge factor in the behavior of the data, we will consider this when we create dummy variables  Creating dummy variables for our model Since most machine learning algorithms like to work with numbers, let's convert these categories from text to dummy arrays, joining them back onto our main training dataframeNote that because the product categories act differently when they are office supplies versus Technology/Furniture, let's only include a feature to denote that it's office supplies<jupyter_code>#Added Dummy Variables to NoofStories & DwellingType
Segments = pd.get_dummies(cleaned_df['Segment'])
Regions = pd.get_dummies(cleaned_df['Region'])
Categories = pd.get_dummies(cleaned_df['Category'])
OrderYears = pd.get_dummies(cleaned_df['Order Year'])

#Joined to dummy variables & original continuous variables to new dataframe. Drop data we do not need in the model
processed_results = pd.concat([cleaned_df, Segments ,Regions,Categories["Office Supplies"],OrderYears], axis=1, join='inner')
processed_results = processed_results.drop(['Region','Segment','Category','Order Year'], axis = 1)
processed_results.head()<jupyter_output><empty_output><jupyter_text>Preprocessing Quantiative VariablesNow that we have the qualitative & quantitative attributes selected, we should look at how to standardize the quatitative attributes. Attributes like 'Sales' and 'Order Year' could become more influential than 'Quantity' and 'Discount' in our model given the larger size observations. Let's use Sci-Kit Learn's Standard Scaler to ensure that certain variables do not have more weight in our model<jupyter_code>from sklearn import preprocessing
scaler = preprocessing.StandardScaler()

processed_results[['Sales','Quantity','Discount','Profit_Processed']]= scaler.fit_transform(processed_results[['Sales','Quantity','Discount','Profit']])
processed_results.head()<jupyter_output><empty_output><jupyter_text>Create a Test SetBefore we continue further, let's isolate our target variable (Profit) from our features. After that, let's break out our dataset into a training & test set. Given that the data is close to 100k records, I will use 10% of the data to test, giving me ~90k to train and ~10k to test my model.I will use Scikit-Learn to break out data into train & test; since I might need to rerun my code a few times, I'm using seeding to ensure that results in my test & train set do not overlap<jupyter_code>from sklearn.model_selection import train_test_split

target = processed_results[['Profit','Profit_Processed']]
features = processed_results.drop(['Profit','Profit_Processed'], axis = 1)
data_train,data_test,target_train,target_test = train_test_split(features,target['Profit'],test_size=0.1,random_state=42)<jupyter_output><empty_output><jupyter_text>Select and Train Linear Regression ModelNow that we have our explored, preprocessed, and split our dataset into a training & testing set, let's first train a linear regression model.<jupyter_code>from sklearn.linear_model import LinearRegression
lin_reg = LinearRegression()
lin_reg.fit(data_train,target_train)<jupyter_output><empty_output><jupyter_text>Evaluating Linear Regression ModelLet's measure this regression model using the score in the LinearRegression().score() method to return the coefficient of determination R^2 of the prediction:<jupyter_code>lin_reg.score(data_test, target_test)<jupyter_output><empty_output><jupyter_text>Not the best R^2 score; we're looking to get closer to 1.In a real world use case, I would go back to the business and ask for additonal information about these orders, such as quality of feedback from the sale (enhancing it using a feedback form associated with the order). But since this is the data we have, let's continue to tweak our model.  Tweaking our ModelLet's see if we can tweak our model to help improve it<jupyter_code>from sklearn.preprocessing import PolynomialFeatures

quad = PolynomialFeatures (degree = 2 )
x_quad = quad.fit_transform(features)

data_train,data_test,target_train,target_test = train_test_split(x_quad,target['Profit'],test_size=0.1,random_state=42)

plr = LinearRegression().fit(data_train,target_train)

Y_train_pred = plr.predict(data_train)
Y_test_pred = plr.predict(data_test)

print(plr.score(data_test,target_test))<jupyter_output>0.4727783991275235<jupyter_text>By incorporating polynomial features into our model, it gets a little bit better R^2 but not by much. Let's try RandomForestRegressor instead<jupyter_code>from sklearn.ensemble import RandomForestRegressor
from sklearn.metrics import r2_score,mean_squared_error

forest = RandomForestRegressor(n_estimators = 100, criterion = 'mse', random_state = 1, n_jobs = -1)
forest.fit(data_train,target_train)
forest_train_pred = forest.predict(data_train)
forest_test_pred = forest.predict(data_test)

print('MSE train data: %.3f, MSE test data: %.3f' % (
mean_squared_error(target_train,forest_train_pred),
mean_squared_error(target_test,forest_test_pred)))
print('R2 train data: %.3f, R2 test data: %.3f' % (
r2_score(target_train,forest_train_pred),
r2_score(target_test,forest_test_pred)))<jupyter_output>MSE train data: 42.538, MSE test data: 344.352
R2 train data: 0.954, R2 test data: 0.686<jupyter_text>Wow! Much better! Looks like Random Forest does a wayyy better job determining our model than linear or logistic regression.  Plotting our resultsSince Random Forest performed the best out of the 3 models, let's show how our training set compares to our testing set<jupyter_code>plt.figure(figsize=(10,6))

plt.scatter(forest_train_pred,forest_train_pred - target_train,
          c = 'black', marker = 'o', s = 35, alpha = 0.5,
          label = 'Train data')
plt.scatter(forest_test_pred,forest_test_pred - target_test,
          c = 'c', marker = 'o', s = 35, alpha = 0.7,
          label = 'Test data')
plt.xlabel('Predicted Values')
plt.ylabel('Variance from Predicted Value')
plt.legend(loc = 'upper left')
plt.hlines(y = 0, xmin = -50, xmax = 200, lw = 2, color = 'red')
plt.savefig('test_train_scatter.png')<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.08849557522123894, 'fraction_numerical': 0.012697191227395151, 'mean_word_length': 6.311298640412565, 'pattern_counts': {'":': 0, '<': 79, '<?xml version=': 0, '>': 79, 'https://': 2, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 2, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '20', 'dir': 'jupyter-structured-clean-dedup', 'id': '82935', 'license': 'MIT', 'n_tokens_mistral': 4751, 'n_tokens_neox': 4410, 'n_words': 1738, 'path': 'Assignments/Assignment 1/Code.ipynb', 'repo_name': 'Sandeep-Bansal1/IntroToDataAnalysis', 'size': '15594'}