<jupyter_start><jupyter_text>Importing the necesssary Libraries<jupyter_code>import pandas as pd
import numpy as np

import seaborn as sns
import matplotlib.pyplot as plt
%matplotlib inline

import psycopg2 as pg2
import datetime as dt
# package used for converting the data into datetime format
from sklearn.preprocessing import LabelEncoder

from sklearn.feature_selection import RFE, f_regression
from sklearn.linear_model import (LinearRegression, Ridge, Lasso, RandomizedLasso)
from sklearn.preprocessing import MinMaxScaler
from sklearn.ensemble import RandomForestRegressor

from sklearn.metrics import r2_score
from sklearn.metrics import mean_squared_error
import math

import warnings
warnings.filterwarnings("ignore")

## Execution time counter
#%load_ext autotime<jupyter_output><empty_output><jupyter_text>Establish connection to the PostgreSQL database<jupyter_code>conn= pg2.connect('dbname = Booksville user= postgres password =<PASSWORD> host= 127.0.0.1')
cur=conn.cursor()
df_raw = pd.read_sql_query('select * from public."keepa" limit 1000', conn)
#Check the dimension of the raw data to see if its properly imported
print('Starting size of our Dataset ')
df_raw.shape
# Print out count of each datatype in the dataframe
df_raw.dtypes.value_counts()<jupyter_output><empty_output><jupyter_text>Price AggregatorAmazon's listing price for books is stored in one of the three columns in the sql dump we obtained from Keepa. If Amazon is fullfilling the sale, the price is stored in amazon_price column. But if the book is sold by a third party seller, the listing price would be marketplace_new_price for new books and marketplace_used_price for used books.We are combining the three columns in to one column called 'price' and assign its values based on the three given price assignment information.The aggregator function adds the new column to the dataset and assigns the value that appears first from the following list and finally drops the three columns from the dataset.     * amazon_Price             * marketplace_new             * marketplace_used_price<jupyter_code>def PriceAggregator(original_df):
    
    df=original_df
    # create a copy of the three columns to choose amazon price from
    df_copy=df[['amazon_price','marketplace_new_price','marketplace_used_price']]
    
    # Replace missing price denoted by -1 to Null in all three price columns   
    for item in df_copy:
        df_copy[item].replace('-1',np.nan, inplace=True)
        
    # Add a new column to store the aggregated price with default value of 'amazon_price'         
    df.insert(79,'price',df_copy['amazon_price'].astype('float'))
    
    
    #Loop throgh all three columns to assign non-null value to the newly created price column. 
    #Keep amazon_price as it is if it is not null, otherwise assign marketplace_new_price as the new price. 
    #Where both 'amazon_price' and 'marketplace_new_price' are null, price will be set to 
    #'marketplace_used_price' regardless of its value.
    
    for i in range(df['price'].size):
        if pd.isnull(df['price'][i]):
            if pd.isnull(df_copy['marketplace_new_price'][i]):
                if pd.isnull(df_copy['marketplace_used_price'][i]):
                    pass
                else:
                    df['price'][i]=df_copy['marketplace_used_price'][i]
            else:
                df['price'][i]=df_copy['marketplace_new_price'][i]
        else:
            pass
    
    # Delete records where price value is missing since that is what we are trying to predict 
    df.dropna(subset=['price'], axis=0, inplace=True)
    
    #Reset index after dropping rows with missing price
    df.reset_index(drop= True, inplace=True)
    
    #Delete old price columns after assigning aggregated price to a brand new column    
    df.drop(['amazon_price','marketplace_new_price','marketplace_used_price'], axis=1 , inplace=True)
    
    #Return the a dataframe with a new price column added to the original dataframe
    return df   
    
    
df=PriceAggregator(df_raw)
df.shape<jupyter_output><empty_output><jupyter_text>Delete duplicate records, if there are any.<jupyter_code># data size before deleting duplicates
df.shape
df.drop_duplicates(inplace = True)
# data size after deleting duplicates
df.shape<jupyter_output><empty_output><jupyter_text>We can see from .shape() function there are 99600 unique records with 77 features in the dataset.  Understanding the Data<jupyter_code># Highlight of the dataframe
df.head(3)<jupyter_output><empty_output><jupyter_text>Descriptive statistics that summarize the central tendency, dispersion and shape of a dataset’s distribution,<jupyter_code>df.describe(include= 'all' )<jupyter_output><empty_output><jupyter_text>The data was stored in the PostgreSQL database as text(string) type and running discriptive statistics with .describe().The dataframe doesn't tell us much until we do the proper type of conversion.   Data Wrangling  Data wrangling is the process of converting data from the initial format to a format that may be better for analysis. As part of the wrangling process we are applying different techniques to come up with a cleaner and complete dataset to apply machine learning. Here are some of the steps we are following;    - Identify Missing Values    - Replace or Delete Missing Values    - Correct Data format    - Aggregate highly related categorical values where necessary  Replace missing values with Null Values. We will replace every missing values with Numpy Null in order to keep uniformity and computational speed.<jupyter_code>df.fillna(np.NaN)
df.replace('', np.NaN, inplace=True)
## count and sort null values in every coulumn in descending order
df.isna().sum().sort_values(ascending=False).to_frame(name='Count of Null Values')<jupyter_output><empty_output><jupyter_text>Null or missing value implies that we dont have information about the feature. We can delete the features that contain Null values for the majority of the records, because keeping them will not provide anything about the data.<jupyter_code>## create a list of features that contain null value for more than 50% of the records based on the the above observation
Null_features=['coupon','offers','liveOffersOrder','promotions','buyBoxSellerIdHistory','features','upcList','variations',
               'hazardousMaterialType','genre','platform','variationCSV','parentAsin','department','size','model','color'
               ,'partNumber','mpn','brand','edition','format']
# Column names with their number of null values
df[Null_features].isna().sum()<jupyter_output><empty_output><jupyter_text>We can delete these features without losing any useful information from our data since more than 50% of the records in the above list contain null values.<jupyter_code>## delete columns that contain very high count of null values 
df.drop(Null_features, axis=1, inplace=True)
# check the shape of the dataset to confirm the features are dropped
df.shape<jupyter_output><empty_output><jupyter_text>For the remaining Null values in our data where the total count is relatively small, we will replace them by a statistically representative values like mean or mode.* Mode value is for categorical columns where there is a clear majority of null values or will be replaced by 'Unknown'* Mean value is used for numerical columns<jupyter_code>#Assigns column names that contain null values to a list
with_Nulls=df.loc[:, df.isna().sum()!=0].columns.tolist()
#Lists down the number of null values in every column in descending order
df[with_Nulls].isna().sum().sort_values(ascending=False)
# let's see what kind of information is in each column
df[with_Nulls].head(5)<jupyter_output><empty_output><jupyter_text>The sample shows that the records are mainly comprised of string or categorical values. Lets further divide the series based on the number of missing (Null) values.<jupyter_code>Nulls2Unknown=['categoryTree_4','categoryTree_3','categoryTree_2','author','studio','publisher','manufacturer',
              'label']
#print out the highest frequency value(Mode) in the given list of features, it only shows the count not the value. 
#Based on the count we can tell if there's a statistical representative mode value to replace the nulls.  
for item in with_Nulls:
    print(f'{item}\t\t{df[item].value_counts().max()}')<jupyter_output><empty_output><jupyter_text>Given that our data contains 100,000 records we can clearly see the high mode value will replace the null values for some of the features.<jupyter_code># The following 3 features have very high Mode value, therefore we'll replace nulls by mode
Nulls2Mode=['languages_0','categoryTree_0','categoryTree_1']
mode = df.filter(['languages_0','categoryTree_0','categoryTree_1']).mode()
df[Nulls2Mode]=df[Nulls2Mode].fillna(df.mode().iloc[0])<jupyter_output><empty_output><jupyter_text>For the following features since there is no one single category with a high frequency(Mode) in the group, we are filling the missing(Null) values with 'Unknown'.<jupyter_code>NullswithNoMode=df.loc[:, df.isna().sum()!=0].columns.tolist()

#Based on the top 3 most frequent records in each column, it shows that there is no dominant value that can be used
#as a mode to replace null values. Therefore, we are replacing null values with 'Unknown'.
for item in NullswithNoMode:
    print(item)
    print(df[item].value_counts().nlargest(3))
    print('Total Number of null values in %s = %d' % (item,df[item].isna().sum()))
    print('')
# Replace nulls with 'Unknown' for multimodel features
df[NullswithNoMode]=df[NullswithNoMode].fillna('Unknown')
# Check if there are still missing or null values in the dataset
df[df.loc[:, df.isna().sum()!=0].columns].isna().sum()<jupyter_output><empty_output><jupyter_text>We have entirely replaced the null and missing values in the dataset by statistically representative values.  Data Type Conversion<jupyter_code>df.dtypes.value_counts()<jupyter_output><empty_output><jupyter_text>The data imported from postgreSQL to pandas dataframe contain columns as object type(string). Most of those features are actually nemerical values, and we will convert the object data type in to the proper format. Lets group all those features that are in string (object) format and convert them to numeric<jupyter_code>df.dtypes.value_counts()
#Convert columns that contain numerical values to numeric data type using pandas to_numeric
numeric=['availabilityAmazon',
       'ean','hasReviews', 'isEligibleForSuperSaverShipping', 'isEligibleForTradeIn',
       'isRedirectASIN', 'isSNS', 'lastPriceChange','lastRatingUpdate', 'lastUpdate', 'listedSince', 
       'newPriceIsMAP', 'numberOfItems','numberOfPages', 'offersSuccessful', 'packageHeight',
       'packageLength', 'packageQuantity', 'packageWeight', 'packageWidth',
       'publicationDate', 'releaseDate', 'rootCategory','stats_atIntervalStart', 'stats_avg', 'stats_avg30', 'stats_avg90',
       'stats_avg180', 'stats_current', 'stats_outOfStockPercentage30',
       'stats_outOfStockPercentage90', 'stats_outOfStockPercentageInInterval',
       'trackingSince','sales_rank', 'price']
#cols = ['productType','rootCategory','stats_atIntervalStart','availabilityAmazon','hasReviews','isRedirectASIN','isSNS','isEligibleForTradeIn','isEligibleForSuperSaverShipping', 'ean','hasReviews', 'availabilityAmazon','isEligibleForTradeIn','lastPriceChange','lastRatingUpdate','lastUpdate','lastRatingUpdate','lastUpdate','listedSince',"newPriceIsMAP", "numberOfItems", "numberOfPages","packageHeight", "packageLength","packageQuantity", "packageWeight", "packageWidth",'stats_avg', 'stats_avg30', 'stats_avg90', 'stats_avg180', 'stats_current',"stats_outOfStockPercentage30", "stats_outOfStockPercentage90","stats_outOfStockPercentageInInterval","trackingSince",'upc','price','amazon_price', 'marketplace_new_price', 'marketplace_used_price', 'sales_rank']
df[numeric] = df[numeric].apply(pd.to_numeric, errors='coerce', axis=1)
df.dtypes.value_counts()
strings=df.loc[:, df.dtypes == np.object].columns.tolist()
print('\n'+ 'Sample of the dataset with only categorical information'+'\n')
df[strings].head(3)<jupyter_output><empty_output><jupyter_text>We can delete 'asin', 'ean' and 'imageCSV' columns since the information they contain is not characteristic discription of books.<jupyter_code>df.drop(['asin','imagesCSV','ean', 'upc'], axis=1, inplace=True)
df.shape
df.dtypes.value_counts()
# Print features that are object type
df.loc[:, df.dtypes == np.object].columns<jupyter_output><empty_output><jupyter_text>Information regarding what language a book is written in and whether its an original print or translated version is recorded in the 'language_0' column. These two information are separated by comma, this column can be split into two parts and stored separately in order to reduce categorical data that we have to encode later.<jupyter_code>df['languages_0'].head(5)
new = df['languages_0'].str.split(",", n = 1, expand = True)
df['language_1']=new[0]
df['language_2']=new[1]
#reduce categories from 9 to 6 groupes by combining related categories 
#df['language_1'].value_counts().to_frame()
#group English, english and Middle English to one categry
df['language_1'].replace(('English', 'english','Middle English'),'English', inplace = True)

#group Spanish,Portuguese and Latin under "Spanish"
df['language_1'].replace(('Spanish', 'Portuguese','Latin'),'Spanish', inplace = True)

#group Chinese, mandarin Chinese and simplified chinese under Chinese 
df['language_1'].replace(('Simplified Chinese', 'Mandarin Chinese','Chinese'),'Chinese', inplace = True)

#group Arabic,Hebrew and Turkish under Middle Eastern
df['language_1'].replace(('Arabic', 'Hebrew','Turkish'),'Middle Eastern', inplace = True)

# group languages with single entry record in to one group called 'Others'
df['language_1'].replace(('Hindi', 'Scots','Filipino','Malay','Dutch','Greek','Korean','Romanian','Czech'),'Others', inplace = True)


#group Danish and Norwegian under 'Scandinavian'  
df['language_1'].replace(('Danish', 'Norwegian'),'Scandinavian', inplace=True)
#replace ('published','Published,Dolby Digital 1.0','Published,DTS-HD 5.1') by Published
df['language_2'].replace(('published','Published,Dolby Digital 1.0','Published,DTS-HD 5.1'),'Published', inplace=True)
df[['language_1','language_2']].head(5)
#Since we have copied the information into new columns we can delete the languages_0 column
df.drop(['languages_0'], axis=1 , inplace=True)
df.columns
df.shape
#Cleaning binding column
df.binding.value_counts()<jupyter_output><empty_output><jupyter_text>The binding column contains 73 differnt categories that are mostly related and some of them contain very small elements. We will aggregate closely related categories to reduce the dimension of our variables to avoid curse of dimensionality<jupyter_code>df.binding.nunique()
# create a dictionary of identical items to create a group an aggregated category 
dict={'Unknown':['Printed Access Code', 'Unknown','Health and Beauty', 'Lawn & Patio', 'Workbook', 'Kitchen', 'Automotive', 'Jewelry'],
     'spiral':[ 'Spiral-bound', 'Staple Bound', 'Ring-bound', 'Plastic Comb', 'Loose Leaf', 'Thread Bound'],
     'magazines':[ 'Journal', 'Single Issue Magazine', 'Print Magazine'],
     'audios':[ 'Audible Audiobook', 'Audio CD', 'DVD', 'Album', 'MP3 CD', 'Audio CD Library Binding'],
     'digital_prints':[ 'CD-ROM', 'Blu-ray', 'DVD-ROM', 'Kindle Edition', 'Video Game', 'Sheet music', 'Software Download',
 'Personal Computers', 'Electronics', 'Game', 'Wireless Phone Accessory'],
     'hardcovers':['Hardcover', 'Hardcover-spiral', 'Turtleback', 'Roughcut'],
     'others':[ 'Cards', 'Pamphlet', 'Calendar', 'Map', 'Stationery', 'Accessory', 'Misc. Supplies', 'Office Product', 'Poster',
 'Wall Chart', 'Bookmark', 'JP Oversized'],
     'paperbacks':[ 'Paperback', 'Perfect Paperback', 'Mass Market Paperback', 'Flexibound', 'Print on Demand (Paperback)',
 'Comic', 'Puzzle', 'Paperback Bunko'],
     'leather_bonded':[ 'Bonded Leather', 'Leather Bound', 'Imitation Leather', 'Vinyl Bound'],
     'board_book':[ 'Board book', 'Baby Product', 'Toy', 'Rag Book', 'Card Book', 'Bath Book', 'Pocket Book'],
     'schoolLibrary_binding':[ 'School & Library Binding', 'Library Binding', 'Textbook Binding']}
for key,val in dict.items():
    df.binding.replace(val,key, inplace=True)
df.binding.value_counts()
df.head()
#catTree_under10.categoryTree_2.values= 'Other'
def groupUnder10(x):
    cond = df[x].value_counts()
    threshold = 10
    df[x] = np.where(df[x].isin(cond.index[cond > threshold ]), df[x], 'Others')
    print('All the different categories that contain less than 10 items in the %s column are grouped together and renamed to "Others".' %x)
df[['categoryTree_1','categoryTree_2','categoryTree_3','categoryTree_4']].nunique()
groupUnder10('categoryTree_2')
#group under 10 counts in to one for categoryTree_3 column
groupUnder10('categoryTree_3')
groupUnder10('categoryTree_4')
df[['categoryTree_0','categoryTree_1','categoryTree_2','categoryTree_3','categoryTree_4']].nunique()
## Some features are duplicated within the dataset, lets delete those duplicated columns
## Delete duplicated features

duplicates=df[['label', 'manufacturer', 'publisher', 'studio']]

df['label'].equals(df['manufacturer'])
df['label'].equals(duplicates['publisher'])
df['label'].equals(duplicates['studio'])
df[df.duplicated(['label', 'manufacturer', 'publisher', 'studio'])]
duplicates.describe(include='all')
df.duplicated(subset=['label', 'manufacturer', 'publisher', 'studio'],keep='first').value_counts()<jupyter_output><empty_output><jupyter_text>Since the above 4 columns contain 89493 duplicated informartion out of 99600 total records we can keep one of those and drop the reamining ones without losing useful information.<jupyter_code># Keep publisher and drop the rest
df.drop(['label', 'manufacturer','studio'], axis =1, inplace=True)
df.shape
df.describe(include='all').transpose()<jupyter_output><empty_output><jupyter_text>Exploratory Data Analysis  Outlier detection and transformation Before we decide whether to use standard deviation or interquntile range to identify outliers, lets plot the data points using a distribution plot.<jupyter_code>def distWithBox(data):
    import numpy as np
    import seaborn as sns
    import matplotlib.pyplot as plt

    sns.set(style="ticks")

    x = df[data]

    f, (ax_box, ax_hist) = plt.subplots(2, sharex=True, 
                                        gridspec_kw={"height_ratios": (.15, .85)})

    sns.boxplot(x, ax=ax_box)
    sns.distplot(x, ax=ax_hist)

    ax_box.set(yticks=[])
    sns.despine(ax=ax_hist)
    sns.despine(ax=ax_box, left=True)
## Distribution and box plot of the raw data with outliers
distWithBox('price')<jupyter_output><empty_output><jupyter_text>For normally distributed data, the skewness should be about 0. A skewness value > 0 means that there is more weight in the left tail of the distribution. The function skewtest can be used to determine if the skewness value is close enough to 0, statistically speaking.<jupyter_code>from scipy.stats import skew
from scipy.stats import skewtest
skew(df['price'])

import seaborn as sns
sns.set(style="whitegrid")
ax = sns.boxplot(x=df['price'])<jupyter_output><empty_output><jupyter_text>We can see from the the distribution plot, the skewtest and the box plot that price is not normally distributed. The price  data is right skewed and there are outlier values that need to be handled.When a data set has outliers or extreme values, we summarize a typical value using the median as opposed to the mean.  When a data set has outliers, variability is often summarized by a statistic called the interquartile range, which is the difference between the first and third quartiles. The first quartile, denoted Q1, is the value in the data set that holds 25% of the values below it. The third quartile, denoted Q3, is the value in the data set that holds 25% of the values above it. The quartiles can be determined following the same approach that we used to determine the median, but we now consider each half of the data set separately. The interquartile range is defined as follows:Interquartile Range(IQR) = Q3-Q1Outliers are values  1.5*IQR below Q1 or above Q3 or equivalently, values below Q1-1.5 IQR or above Q3+1.5 IQR.These are referred to as Tukey fences.<jupyter_code>from numpy import percentile
data=df['price']
q25, q75 = percentile(data, 25), percentile(data, 75)
iqr = q75 - q25
print('Percentiles:\n\t25th=%.3f \n\t75th=%.3f \n\tIQR=%.3f' % (q25, q75, iqr))
# calculate the outlier cutoff
cut_off = iqr * 1.5
lower, upper = q25 - cut_off, q75 + cut_off
# identify outliers
outliers = [x for x in data if x < lower or x > upper]
print('Identified outliers: %d' % len(outliers) )
outliers_removed = [x for x in data if x >= lower and x <= upper]
print('Non-outlier observations: %d' % len(outliers_removed))
outliers=[] 
data_1=df['price'] 
for item in data_1:
    if item <lower or item>upper:
        outliers.append(item)

x=df['price']
outlier_indices=list(data_1.index[(x<lower) | (x> upper)])
len(outlier_indices)
df.drop(axis=0,index=outlier_indices, inplace=True)
df.shape
## lets plot distribution with and box plot to see the change after we trim down the outliers
distWithBox('price')<jupyter_output><empty_output><jupyter_text>Correlation Between Numerical Features We are running pearson correlation between numeric valued features to see if there is any linear dependence between the variables.<jupyter_code>cor=df.corr()
sns.heatmap(cor,cmap="PiYG")<jupyter_output><empty_output><jupyter_text>Boxplots will show us the distribution of categorical data against a continuous variable. We are using a boxplot to visualize the distribution of values in binding and language columns against price. Based on the visualization we can see that there is not so much overlap in the binding category, which implies that it is a good predictor of price. But when it comes to language_1, books in almost in every language fall within a price range of 500 to 2000( which is $5 to $20). It implies that knowing what language a book is written in doesn't tell us how much it would worth.<jupyter_code>sns.boxplot(x=df['price'],y=df['binding'], data=df)
sns.boxplot(y=df['language_1'],x=df['price'], data=df)<jupyter_output><empty_output><jupyter_text>Encoding categorical columns<jupyter_code>cat_cols=['author','language_1','language_2','binding','categoryTree_0', 'categoryTree_1', 'categoryTree_2', 'categoryTree_3',
       'categoryTree_4','productGroup','publisher','title','type','language_1','language_2']
for item in cat_cols:
    df[item]=df[item].astype(str)
df[cat_cols].head()
# Label encoding to convert string to representative numeric values
df[cat_cols]= df[cat_cols].apply(LabelEncoder().fit_transform)
# Display top 5 records from the dataset to check if all the records are converted to numbers
df.head(5)<jupyter_output><empty_output><jupyter_text>Feature Selection VarianceThreshold is a simple baseline approach to feature selection. It removes all features whose variance doesn’t meet some threshold. By default, it removes all zero-variance features, i.e. features that have the same value in all samples.threshold .8 * (1 - .8)Using 0.8 as a threshhold, we will remove features with less than 20 percent variation within itself.<jupyter_code>df_X=df.loc[:, df.columns != 'price']
df_y=df['price']
from sklearn.feature_selection import VarianceThreshold
print('%s Number of features before VarianceThreshhold'%len(df_X.columns))

selector=VarianceThreshold(threshold=(.8*(1-.8)))
FeaturesTransformed=selector.fit_transform(df_X)

## print the support and shape of the transformed features
print(selector.get_support())

data=df_X[df_X.columns[selector.get_support(indices=True)]]
cols=data.columns
df_reduced=pd.DataFrame(FeaturesTransformed, columns=cols)
df_reduced.shape
data=df_reduced
target=df_y
data.shape<jupyter_output><empty_output><jupyter_text>Yellowbrick for Feature Selectionwe are using yellowbrick's feature selection method for finding and selecting the most useful features and eliminate zero importance features from the dataset.  Important Features for Random Forest Regressor<jupyter_code>#Using yellowbrick feature selection method with random forest regressor
from sklearn.ensemble import RandomForestRegressor

from yellowbrick.features.importances import FeatureImportances

fig = plt.figure(figsize=(10,10))
ax = fig.add_subplot()

viz = FeatureImportances(RandomForestRegressor(), ax=ax)
viz.fit(data, target)
viz.poof()
feature_importances = pd.DataFrame(viz.feature_importances_,
                                   index=data.columns,
                                   columns=['importance']).sort_values('importance', ascending=False)
## important features for Random Forest Regression
RF_importants=feature_importances.index[feature_importances.importance!=0]
df[RF_importants].shape
# predicting price using random forest regressor
from sklearn.ensemble import RandomForestRegressor
from sklearn.model_selection import train_test_split as split
X=df[RF_importants]
Y=df['price']
model=RandomForestRegressor()
X_train, X_test, Y_train, Y_test= split(X,Y,test_size=0.25, random_state=42)
model.fit(X_train,Y_train)
Y_test=model.predict(X_test)
ax1=sns.distplot(target,hist=False, color='r',label="Actual price")
sns.distplot(Y_test,hist=False,color='b', label="Predicted price", ax=ax1)
plt.title(" Actual Vs Predicted Price ")
plt.xlabel('Price')
plt.ylabel('Proportion of Books')
plt.show()<jupyter_output><empty_output><jupyter_text>Important Features for Gradient Boosting Regressor<jupyter_code>from sklearn.ensemble import GradientBoostingRegressor

from yellowbrick.features.importances import FeatureImportances

fig = plt.figure(figsize=(20,20))
ax = fig.add_subplot()

viz = FeatureImportances(GradientBoostingRegressor(), ax=ax)
viz.fit(data, target)
viz.poof()
feature_importances = pd.DataFrame(viz.feature_importances_,
                                   index=data.columns,
                                   columns=['importance']).sort_values('importance', ascending=False)
## important features for gradient boosting regression
GBR_importants=feature_importances.index[feature_importances.importance!=0]
df[GBR_importants].shape
from sklearn.ensemble import GradientBoostingRegressor
from sklearn.model_selection import train_test_split as split
X=df[GBR_importants]
Y=df['price']
model=GradientBoostingRegressor()
X_train, X_test, Y_train, Y_test= split(X,Y,test_size=0.25, random_state=42)
model.fit(X_train,Y_train)
Y_test=model.predict(X_test)
ax1=sns.distplot(target,hist=False, color='r',label="Actual price")
sns.distplot(Y_test,hist=False,color='b', label="Predicted price", ax=ax1)
plt.title(" Actual Vs Predicted Price ")
plt.xlabel('Price')
plt.ylabel('Proportion of Books')
plt.show()<jupyter_output><empty_output><jupyter_text>Important Features for Decision Tree Regressor<jupyter_code>from sklearn.tree import DecisionTreeRegressor

from yellowbrick.features.importances import FeatureImportances

fig = plt.figure(figsize=(20,20))
ax = fig.add_subplot()

viz = FeatureImportances( DecisionTreeRegressor(), ax=ax)
viz.fit(data, target)
viz.poof()
feature_importances = pd.DataFrame(viz.feature_importances_,
                                   index=data.columns,
                                   columns=['importance']).sort_values('importance', ascending=False)
## important features for decision tree regression
DTR_importants=feature_importances.index[feature_importances.importance!=0]
df[DTR_importants].shape
from sklearn.tree import DecisionTreeRegressor
from sklearn.model_selection import train_test_split as split
X=df[DTR_importants]
Y=df['price']
model=DecisionTreeRegressor()
X_train, X_test, Y_train, Y_test= split(X,Y,test_size=0.25, random_state=42)
model.fit(X_train,Y_train)
Y_test=model.predict(X_test)
ax1=sns.distplot(target,hist=False, color='r',label="Actual price")
sns.distplot(Y_test,hist=False,color='b', label="Predicted price", ax=ax1)
plt.title(" Actual Vs Predicted Price ")
plt.xlabel('Price')
plt.ylabel('Proportion of Books')
plt.show()<jupyter_output><empty_output><jupyter_text>Model DevelopmentIn this section we will implement several models that will predict price using the dependent variables and compare the accuracy, r-score, goodness of fit and plot residuals. Based on the scores and visual comparison of the plots, we will refine the best performing models using grid search to fine tune the hyperparameters to generate a better predictive model.<jupyter_code># This function applies multiple models on the data and returns model name with r2-score and mean squared error value
def ModelScores(data,target):
    X=data
    Y=target
    
    from sklearn.metrics import r2_score
    from sklearn.metrics import mean_squared_error
    import math
    from sklearn.model_selection import train_test_split as split
    X_train, X_test, Y_train, Y_test= split(X,Y,test_size=0.25, random_state=42)
    
    from sklearn.ensemble import GradientBoostingRegressor
    from sklearn.ensemble import RandomForestRegressor
    from sklearn.tree import DecisionTreeRegressor
    from sklearn.linear_model import LinearRegression
    from sklearn.neural_network import MLPRegressor
    from sklearn.linear_model import RidgeCV
    from sklearn.linear_model import LassoLars
    from sklearn.linear_model import Lasso
    from sklearn.linear_model import ElasticNet
    from sklearn.linear_model import BayesianRidge
    from sklearn.linear_model import RANSACRegressor

    models={'Gradient Boost': GradientBoostingRegressor(),
            'Random Forest': RandomForestRegressor(),
            'Decision Tree': DecisionTreeRegressor(),
            'Linear Regression': LinearRegression(),
            'MLP': MLPRegressor(),
            'Ridge CV': RidgeCV(),
            'LassoLars':LassoLars(),
            'Lasso':Lasso(),
            'Elastic Search': ElasticNet(),
            'Bayesian Ridge':BayesianRidge(),
            'Ransac':RANSACRegressor()      
           }
    for name,model in models.items():
        mdl=model
        mdl.fit(X_train, Y_train)
        prediction = mdl.predict(X_test)
        print(name)
        print("Accuracy Score", r2_score(Y_test, prediction))
        mse3 = mean_squared_error(Y_test, prediction)
        print("The root mean square value", math.sqrt(mse3))
data=data
target=df['price']
ModelScores(data, target)
%matplotlib inline
from yellowbrick.classifier import ClassificationReport
from yellowbrick.classifier import ClassPredictionError
from yellowbrick.regressor import ResidualsPlot
from sklearn.svm import SVR
from sklearn.ensemble import RandomForestRegressor
from sklearn.ensemble import GradientBoostingRegressor
from sklearn.tree import DecisionTreeRegressor

regressors = {
    "Gradient Boost": GradientBoostingRegressor(),
    "Random Forest": RandomForestRegressor(),
    "Decision Tree": DecisionTreeRegressor()
    
}

for _, regressor in regressors.items():
    visualizer = ResidualsPlot(regressor)
    visualizer.fit(X_train, Y_train)
    visualizer.score(X_test, Y_test)
    visualizer.poof()
from yellowbrick.target import FeatureCorrelation
feature_names = np.array(df.columns)
data=df.loc[:, df.columns != 'price']
target=df['price']
figsize=(20, 20)
visualizer = FeatureCorrelation(labels=feature_names)
visualizer.fit(data, target)
visualizer.poof()
#validation curve for decision tree regression and Random forest regression models

from yellowbrick.model_selection import ValidationCurve

# Extract the instances and target
X = df_X
y = df_y

regressors = {
        "Gradient Boost": GradientBoostingRegressor(),
        "Decision Tree": DecisionTreeRegressor(),
        "Random Forest": RandomForestRegressor()

}
for _, regressor in regressors.items():
    viz = ValidationCurve(
        regressor, param_name="max_depth",
        param_range=np.arange(1, 11), cv=10, scoring="r2"
    )
    # Fit and poof the visualizer
    viz.fit(X, y)
    viz.poof()<jupyter_output><empty_output><jupyter_text>We will use the validation curve from the above three figures to narrow down the optimal 'max_depth' value range to use, for hyperparameter tuning in a  grid search.<jupyter_code># Cross Validation Score for Random Forest Regressor
from sklearn.ensemble import RandomForestRegressor
from sklearn.model_selection import KFold
from yellowbrick.model_selection import CVScores

ind=df[RF_importants].values
dep=df['price'].values

_, ax = plt.subplots()
cv = KFold(10)
oz = CVScores(
    RandomForestRegressor(), ax=ax, cv=cv, scoring='r2'
)

oz.fit(ind, dep)
oz.poof()

# CV score for Gradiet Boosting Regresor
from sklearn.ensemble import GradientBoostingRegressor
from sklearn.model_selection import StratifiedKFold
from yellowbrick.model_selection import CVScores

ind=df[GBR_importants].values
dep=df['price'].values

_, ax = plt.subplots()
cv = StratifiedKFold(n_splits=10, random_state=42)
oz = CVScores(
    GradientBoostingRegressor(), ax=ax, cv=cv, scoring = 'r2'
)

oz.fit(ind, dep)
oz.poof()


# CV score for Decision Tree Regressor
from sklearn.tree import DecisionTreeRegressor
from sklearn.model_selection import KFold
from yellowbrick.model_selection import CVScores

ind=df[DTR_importants].values
dep=df['price'].values

_, ax = plt.subplots()
cv = KFold(10)
oz = CVScores(
    DecisionTreeRegressor(), ax=ax, cv=cv, scoring = 'r2'
)

oz.fit(ind, dep)
oz.poof()<jupyter_output><empty_output><jupyter_text>Hyperparameter Tuning<jupyter_code># hyper-parameter tunung for decision tree
from sklearn.model_selection import GridSearchCV
from sklearn.tree import DecisionTreeRegressor

DecisionTree = DecisionTreeRegressor(random_state = 40)

min_samples_split = [2,3,4,5,6,7,8]
min_samples_leaf = [1,2,3,4,5]
max_depth = [4,5,6,7,8,9]
tuned_params = [{'min_samples_split': min_samples_split}, {'min_samples_leaf': min_samples_leaf},{'max_depth': max_depth}]
n_folds = 5

X=df[DTR_importants]
Y=df['price']

grid = GridSearchCV(
    DecisionTree, tuned_params, cv=n_folds
)

grid.fit(X, Y)
print(grid.best_estimator_)
from sklearn.model_selection import GridSearchCV
from sklearn.ensemble import GradientBoostingRegressor

GradientBoosting = GradientBoostingRegressor(random_state = 40)
alphas = [0.001, 0.01, 0.1, 0.5, 0.9]
sample_split = [2,3,4,5,6,7,8]
max_depth = [4,5,6,7,8,9]
learning_rate = [0.1, 0.3, 0.5, 0.7]
tuned_params = [{'alpha': alphas}, {'min_samples_split': sample_split}, {'max_depth': max_depth}, {'learning_rate':learning_rate}]
n_folds = 5

X=df[GBR_importants]
Y=df['price']

grid = GridSearchCV(
    GradientBoosting, tuned_params, cv=n_folds
)

grid.fit(X, Y)
print(grid.best_estimator_)



from sklearn.model_selection import GridSearchCV
from sklearn.ensemble import RandomForestRegressor

RandomForest = RandomForestRegressor(random_state = 40)

estimators = [10,50,100]
sample_split = [2,3,4,5,6,7,8]
sample_leaf = [1,2,3,4,5]
max_depth = [4,5,6,7,8,9]
tuned_params = [{'n_estimators': estimators}, {'min_samples_split': sample_split}, {'min_samples_leaf': sample_leaf},{'max_leaf_nodes': max_depth}]
n_folds = 5

X=df[RF_importants]
Y=df['price']

grid = GridSearchCV(
    RandomForest, tuned_params, cv=n_folds
)

grid.fit(X, Y)
print(grid.best_estimator_)<jupyter_output><empty_output><jupyter_text>Model EvaluationIn this part we are using the result that we obtained from the grid search as an input to retrain our models. The grid search is applied with cross validation by taking the average score over 5 folds.<jupyter_code>X=df[DTR_importants]
Y=df['price']
X_train, X_test, Y_train, Y_test= split(X,Y,test_size=0.25, random_state=42)

from sklearn.tree import DecisionTreeRegressor
model=DecisionTreeRegressor(criterion='mse', max_depth=9, max_features=None,
           max_leaf_nodes=None, min_impurity_decrease=0.0,
           min_impurity_split=None, min_samples_leaf=1,
           min_samples_split=2, min_weight_fraction_leaf=0.0,
           presort=False, random_state=40, splitter='best')

model.fit(X_train,Y_train)
prediction13 = model.predict(X_test)
print("Accuracy Score", r2_score(Y_test, prediction13))
mse = mean_squared_error(Y_test, prediction13)
print("The root mean square value", math.sqrt(mse))
X=df[GBR_importants]
Y=df['price']
X_train, X_test, Y_train, Y_test= split(X,Y,test_size=0.25, random_state=42)

from sklearn.ensemble import GradientBoostingRegressor
model=GradientBoostingRegressor(alpha=0.9, criterion='friedman_mse', init=None,
             learning_rate=0.1, loss='ls', max_depth=5, max_features=None,
             max_leaf_nodes=None, min_impurity_decrease=0.0,
             min_impurity_split=None, min_samples_leaf=1,
             min_samples_split=2, min_weight_fraction_leaf=0.0,
             n_estimators=100, n_iter_no_change=None, presort='auto',
             random_state=40, subsample=1.0, tol=0.0001,
             validation_fraction=0.1, verbose=0, warm_start=False)
model.fit(X_train,Y_train)
prediction13 = model.predict(X_test)
print("Accuracy Score", r2_score(Y_test, prediction13))
mse = mean_squared_error(Y_test, prediction13)
print("The root mean square value", math.sqrt(mse))
X=df[RF_importants]
Y=df['price']
X_train, X_test, Y_train, Y_test= split(X,Y,test_size=0.25, random_state=42)

from sklearn.ensemble import RandomForestRegressor
rfg = RandomForestRegressor(bootstrap=True, criterion='mse', max_depth=None,
           max_features='auto', max_leaf_nodes=None,
           min_impurity_decrease=0.0, min_impurity_split=None,
           min_samples_leaf=1, min_samples_split=2,
           min_weight_fraction_leaf=0.0, n_estimators=100, n_jobs=None,
           oob_score=False, random_state=40, verbose=0, warm_start=False)
rfg.fit(X_train, Y_train)
prediction14 = rfg.predict(X_test)
print("Accuracy Score", r2_score(Y_test, prediction14))
mse2 = mean_squared_error(Y_test, prediction14)
print("The root mean square value", math.sqrt(mse2))<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.10510810668630649, 'fraction_numerical': 0.01299384502077963, 'mean_word_length': 5.726645435244161, 'pattern_counts': {'":': 7, '<': 138, '<?xml version=': 0, '>': 140, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 1, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 9, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '33', 'dir': 'jupyter-structured-clean-dedup', 'id': '47610', 'license': 'MIT', 'n_tokens_mistral': 12177, 'n_tokens_neox': 11556, 'n_words': 3417, 'path': '.ipynb_checkpoints/CAPSTONE-FINAL-checkpoint.ipynb', 'repo_name': 'suchitsharma/Booksville', 'size': '38014'}