<jupyter_start><jupyter_text>Price AggregatorPrice column is divided into three different coumns depending on availability and condition of books carried by Amazon.According to keepa, prise is set based on availability of information in the following order. The aggregator function addsa new column  called 'price' to the dataset and assigns the value that appers first from the following list and finally deletes the three price columns.               * amazon_Price                 * marketplace_new                 * marketplace_used_price<jupyter_code>def PriceAggregator(original_df):
    
    df=original_df
    # create a copy of the three columns to choose amazon price from
    df_copy=df[['amazon_price','marketplace_new_price','marketplace_used_price']]
    
    # Replace missing price denoted by -1 to Null in all three price columns   
    for item in df_copy:
        df_copy[item].replace('-1',np.nan, inplace=True)
        
    # Add a new column to store the aggregated price with default value of 'amazon_price'         
    df.insert(79,'price',df_copy['amazon_price'].astype('float'))
    
    
    #Loop throgh all three columns to assign non-null value to the newly created price column. 
    #Keep amazon_price as is if not null, otherwise assign marketplace_new_price as the new price. 
    #Where both 'amazon_price' and 'marketplace_new_price' are null, price will be set to 
    #'marketplace_used_price' regardless of its value.
    
    for i in range(df['price'].size):
        if pd.isnull(df['price'][i]):
            if pd.isnull(df_copy['marketplace_new_price'][i]):
                if pd.isnull(df_copy['marketplace_used_price'][i]):
                    pass
                else:
                    df['price'][i]=df_copy['marketplace_used_price'][i]
            else:
                df['price'][i]=df_copy['marketplace_new_price'][i]
        else:
            pass
    
    #Delete records where price record is missing since there is no value to cross check
    #the accuracy of the model in the test set. 
    df.dropna(subset=['price'], axis=0, inplace=True)
    
    #Reset index after dropping rows with missing price
    df.reset_index(drop= True, inplace=True)
    
    #Delete old price columns after assigning aggregated price to a brand new column    
    df.drop(['amazon_price','marketplace_new_price','marketplace_used_price'], axis=1 , inplace=True)
    
    #Return the a dataframe with a new price column added to the original dataframe
    return df   
    
    
df=PriceAggregator(df_raw)
df.shape<jupyter_output><empty_output><jupyter_text>Delete duplicate records if there are any<jupyter_code>df.shape
df.drop_duplicates(inplace = True)
df.shape
## categorical valued features
cat=['author', 'binding','brand','categoryTree_0','categoryTree_1','categoryTree_2','categoryTree_3','categoryTree_4',
'color','edition','features','format','genre','label','languages_0', 'manufacturer','productGroup','publisher','studio',
'title','type']
df[cat].head(5)<jupyter_output><empty_output><jupyter_text>Replace every missing value with Null Values for  further analysis.<jupyter_code>df.replace('', np.NaN, inplace=True)
#df.fillna(np.NaN)
df.isna().sum().sort_values(ascending=False).to_frame(name='Count of Null Values')<jupyter_output><empty_output><jupyter_text>We can delete those columns that contain Null values for the majority of the record set since those features are not common behaviour to our instances.<jupyter_code>Null_features=['coupon','offers','liveOffersOrder','promotions','buyBoxSellerIdHistory','features','upcList','variations',
               'hazardousMaterialType','genre','platform','variationCSV','parentAsin','department','size','model','color'
               ,'partNumber','mpn','brand','edition','format']
df[Null_features].isna().sum()<jupyter_output><empty_output><jupyter_text>We can delete these features without losing any useful information from our data since more than 50% of the records in the above list contain null values.<jupyter_code>df.drop(Null_features, axis=1, inplace=True)
df.shape<jupyter_output><empty_output><jupyter_text>For the remaining null values in our data where the total count is relatively small, we will replace them by  a statistically representative values like mean or mode.  * Mode for categorical values where there is a clear majority or filled with 'Unknown'  * Mean value is used for numerical columns<jupyter_code>with_Nulls=df.loc[:, df.isna().sum()!=0].columns.tolist()
df[with_Nulls].isna().sum().sort_values(ascending=False)
## For our records sets mainly comprised of string or categorival data
Nulls2Unknown=['categoryTree_4','categoryTree_3','categoryTree_2','author','studio','publisher','manufacturer',
              'label']
df[with_Nulls].head(3)
for item in with_Nulls:
   print (f'{item}\t\t{df[item].value_counts().max()}')<jupyter_output>author		319
binding		44619
categoryTree_0		99576
categoryTree_1		99576
categoryTree_2		17441
categoryTree_3		8216
categoryTree_4		2704
imagesCSV		4
label		4267
languages_0		95316
manufacturer		4267
publisher		4267
studio		4267<jupyter_text>Given that our data contains 100,000 records we can clearly see the high mode value for some of the features to replace the nu<jupyter_code>Nulls2Mode=['languages_0','categoryTree_0','categoryTree_1']
mode = df.filter(['languages_0','categoryTree_0','categoryTree_1']).mode()
df[Nulls2Mode]=df[Nulls2Mode].fillna(df.mode().iloc[0])<jupyter_output><empty_output><jupyter_text>For the following features since there is no one single category with a high frequency(Mode) in the group, we are filling the missing(Null) values with 'Unknown'.<jupyter_code>NullswithNoMode=df.loc[:, df.isna().sum()!=0].columns.tolist()
for item in NullswithNoMode:
    print(item)
    print(df[item].value_counts().nlargest(3))
df[NullswithNoMode]=df[NullswithNoMode].fillna('Unknown')
# Check if there are still missing or null values in the dataset
df[df.loc[:, df.isna().sum()!=0].columns].isna().sum()<jupyter_output><empty_output><jupyter_text>We have entirely replaced the null and missing values in the dataset by statistically representative values.  Data Type Conversion<jupyter_code>df.dtypes.value_counts()<jupyter_output><empty_output><jupyter_text>Lets group all those features that are in string (object) format and convert them to numeric<jupyter_code>#df[strings] = df[strings].apply(pd.to_numeric, errors='coerce', axis=1)
df.dtypes.value_counts()
#Convert columns that contain numerical values to numeric data type using pandsas to_numeric
numeric=['availabilityAmazon',
       'ean','hasReviews', 'isEligibleForSuperSaverShipping', 'isEligibleForTradeIn',
       'isRedirectASIN', 'isSNS', 'lastPriceChange','lastRatingUpdate', 'lastUpdate', 'listedSince', 
       'newPriceIsMAP', 'numberOfItems','numberOfPages', 'offersSuccessful', 'packageHeight',
       'packageLength', 'packageQuantity', 'packageWeight', 'packageWidth',
       'publicationDate', 'releaseDate', 'rootCategory','stats_atIntervalStart', 'stats_avg', 'stats_avg30', 'stats_avg90',
       'stats_avg180', 'stats_current', 'stats_outOfStockPercentage30',
       'stats_outOfStockPercentage90', 'stats_outOfStockPercentageInInterval',
       'trackingSince','sales_rank', 'price']
#cols = ['productType','rootCategory','stats_atIntervalStart','availabilityAmazon','hasReviews','isRedirectASIN','isSNS','isEligibleForTradeIn','isEligibleForSuperSaverShipping', 'ean','hasReviews', 'availabilityAmazon','isEligibleForTradeIn','lastPriceChange','lastRatingUpdate','lastUpdate','lastRatingUpdate','lastUpdate','listedSince',"newPriceIsMAP", "numberOfItems", "numberOfPages","packageHeight", "packageLength","packageQuantity", "packageWeight", "packageWidth",'stats_avg', 'stats_avg30', 'stats_avg90', 'stats_avg180', 'stats_current',"stats_outOfStockPercentage30", "stats_outOfStockPercentage90","stats_outOfStockPercentageInInterval","trackingSince",'upc','price','amazon_price', 'marketplace_new_price', 'marketplace_used_price', 'sales_rank']
df[numeric] = df[numeric].apply(pd.to_numeric, errors='coerce', axis=1)
df.dtypes.value_counts()
strings=df.loc[:, df.dtypes == np.object].columns.tolist()
print('\n'+ 'Sample of the dataset with only categorical information'+'\n')
df[strings].head(3)<jupyter_output>Sample of the dataset with only categorical information<jupyter_text>We can delete asin, ean and imageCSV columns since the information contained in them is not characteristic discription of books.<jupyter_code>df.drop(['asin','imagesCSV','ean', 'upc'], axis=1, inplace=True)
#upc  might break code watch for it
df.shape
df.dtypes.value_counts()
df.loc[:, df.dtypes == np.object].columns<jupyter_output><empty_output><jupyter_text>The language_0 column contains aggregated information separated by comma, we are going to split it into 2 parts;<jupyter_code>df['languages_0'].head(5)
new = df['languages_0'].str.split(",", n = 1, expand = True)
df['language_1']=new[0]
df['language_2']=new[1]
# reduced categories froom 9 to 6 grouping related categories together
#df['language_1'].value_counts().to_frame()
#group English, english and Middle English to one categry
df['language_1'].replace(('English', 'english','Middle English'),'English', inplace = True)

#grouping Spanish,Portuguese and Latin under "Spanish"
df['language_1'].replace(('Spanish', 'Portuguese','Latin'),'Spanish', inplace = True)

#grouping Chinese, mandarin Chinese and simplified chinese to Chinese 
df['language_1'].replace(('Simplified Chinese', 'Mandarin Chinese','Chinese'),'Chinese', inplace = True)

#grouping Arabic,Hebrew and Turkish under Middle Eastern
df['language_1'].replace(('Arabic', 'Hebrew','Turkish'),'Middle Eastern', inplace = True)

# group languages with single entry record in to one group called 'Others'
df['language_1'].replace(('Hindi', 'Scots','Filipino','Malay','Dutch','Greek','Korean','Romanian','Czech'),'Others', inplace = True)


#grouping Danish and Norwegian into one group of 'Scandinavian'  
df['language_1'].replace(('Danish', 'Norwegian'),'Scandinavian', inplace=True)
#replaced ('published','Published,Dolby Digital 1.0','Published,DTS-HD 5.1') by Published
df['language_2'].replace(('published','Published,Dolby Digital 1.0','Published,DTS-HD 5.1'),'Published', inplace=True)
df[['language_1','language_2']].head(5)
#Since we have copied the information into new columns we can delete the languages_0 column
df.drop(['languages_0'], axis=1 , inplace=True)
df.columns
df.shape
df.binding.value_counts()<jupyter_output><empty_output><jupyter_text>The binding column contains 73 differnt categories that are mostly related and some of them contain very small elements, we will aggregate closely related categories to reduce the dimension of our variables to avoid curse of dimentioanlity<jupyter_code>df.binding.nunique()
dict={'Unknown':['Printed Access Code', 'Unknown','Health and Beauty', 'Lawn & Patio', 'Workbook', 'Kitchen', 'Automotive', 'Jewelry'],
     'spiral':[ 'Spiral-bound', 'Staple Bound', 'Ring-bound', 'Plastic Comb', 'Loose Leaf', 'Thread Bound'],
     'magazines':[ 'Journal', 'Single Issue Magazine', 'Print Magazine'],
     'audios':[ 'Audible Audiobook', 'Audio CD', 'DVD', 'Album', 'MP3 CD', 'Audio CD Library Binding'],
     'digital_prints':[ 'CD-ROM', 'Blu-ray', 'DVD-ROM', 'Kindle Edition', 'Video Game', 'Sheet music', 'Software Download',
 'Personal Computers', 'Electronics', 'Game', 'Wireless Phone Accessory'],
     'hardcovers':['Hardcover', 'Hardcover-spiral', 'Turtleback', 'Roughcut'],
     'others':[ 'Cards', 'Pamphlet', 'Calendar', 'Map', 'Stationery', 'Accessory', 'Misc. Supplies', 'Office Product', 'Poster',
 'Wall Chart', 'Bookmark', 'JP Oversized'],
     'paperbacks':[ 'Paperback', 'Perfect Paperback', 'Mass Market Paperback', 'Flexibound', 'Print on Demand (Paperback)',
 'Comic', 'Puzzle', 'Paperback Bunko'],
     'leather_bonded':[ 'Bonded Leather', 'Leather Bound', 'Imitation Leather', 'Vinyl Bound'],
     'board_book':[ 'Board book', 'Baby Product', 'Toy', 'Rag Book', 'Card Book', 'Bath Book', 'Pocket Book'],
     'schoolLibrary_binding':[ 'School & Library Binding', 'Library Binding', 'Textbook Binding']}
for key,val in dict.items():
    df.binding.replace(val,key, inplace=True)
df.binding.value_counts()
df.head()
#catTree_under10.categoryTree_2.values= 'Other'
def groupUnder10(x):
    cond = df[x].value_counts()
    threshold = 10
    df[x] = np.where(df[x].isin(cond.index[cond > threshold ]), df[x], 'Others')
    return('All the different categories that contain less than 10 items in the %s column are renamed to Others \n inorder to avoid curse of dimensionality' %x)
df[['categoryTree_1','categoryTree_2','categoryTree_3','categoryTree_4']].nunique()
groupUnder10('categoryTree_2')
#group under 10 counts in to one for categoryTree_3 column
groupUnder10('categoryTree_3')
groupUnder10('categoryTree_4')
df[['categoryTree_0','categoryTree_1','categoryTree_2','categoryTree_3','categoryTree_4']].nunique()
## Some features are duplicated within the dataset, lets delete those duplicated columns
## Delete duplicated features

duplicates=df[['label', 'manufacturer', 'publisher', 'studio']]

df['label'].equals(df['manufacturer'])
df['label'].equals(duplicates['publisher'])
df['label'].equals(duplicates['studio'])
#df[df.duplicated(['label', 'manufacturer', 'publisher', 'studio'])]
duplicates.describe(include='all')
df.duplicated(subset=['label', 'manufacturer', 'publisher', 'studio'],keep='first').value_counts()<jupyter_output><empty_output><jupyter_text>Since the above 4 columns contain duplicated informartion in 89493 out of 99600 total records we can keep one of those and drop the reamining ones without losing useful information.<jupyter_code># Keep publisher and drop the rest
df.drop(['label', 'manufacturer','studio'], axis =1, inplace=True)
df.shape
df.describe(include='all').transpose()<jupyter_output><empty_output><jupyter_text>Encoding categorical columns<jupyter_code>cat_cols=['author','language_1','language_2','binding','categoryTree_0', 'categoryTree_1', 'categoryTree_2', 'categoryTree_3',
       'categoryTree_4','productGroup','publisher','title','type','language_1','language_2']
df[cat_cols].head()
#might not be necessary
df['author']=df['author'].astype(str)
df['language_2']=df['language_2'].astype(str)

df['categoryTree_1']=df['categoryTree_1'].astype(str)
df['categoryTree_2']=df['categoryTree_2'].astype(str)
df['categoryTree_3']=df['categoryTree_3'].astype(str)
df['categoryTree_4']=df['categoryTree_4'].astype(str)<jupyter_output><empty_output><jupyter_text>Outlier detection and transformation Before we decide whether to use standard deviation or interquntile range to identify outliers, lets plot the data points using a distribution plot.<jupyter_code>def distWithBox(data):
    import numpy as np
    import seaborn as sns
    import matplotlib.pyplot as plt

    sns.set(style="ticks")

    x = df[data]

    f, (ax_box, ax_hist) = plt.subplots(2, sharex=True, 
                                        gridspec_kw={"height_ratios": (.15, .85)})

    sns.boxplot(x, ax=ax_box)
    sns.distplot(x, ax=ax_hist)

    ax_box.set(yticks=[])
    sns.despine(ax=ax_hist)
    sns.despine(ax=ax_box, left=True)
## Distribution and box plot of the raw data with outliers
distWithBox('price')<jupyter_output><empty_output><jupyter_text>We can see from the graph that the distribution is not normal so we have to use interquantile range to cutoff outliers<jupyter_code>from numpy import percentile
data=df['price']
q25, q75 = percentile(data, 25), percentile(data, 75)
iqr = q75 - q25
print('Percentiles: 25th=%.3f, 75th=%.3f, IQR=%.3f' % (q25, q75, iqr))
# calculate the outlier cutoff
cut_off = iqr * 1.5
lower, upper = q25 - cut_off, q75 + cut_off
# identify outliers
outliers = [x for x in data if x < lower or x > upper]
print('Identified outliers: %d' % len(outliers))
outliers_removed = [x for x in data if x >= lower and x <= upper]
print('Non-outlier observations: %d' % len(outliers_removed))
outliers=[] 
data_1=df['price'] 
for item in data_1:
    if item <lower or item>upper:
        outliers.append(item)

x=df['price']
outlier_indices=list(data_1.index[(x<lower) | (x> upper)])
len(outlier_indices)
df.drop(axis=0,index=outlier_indices, inplace=True)
df.shape
## lets plot distribution with and box plot to see the change after we trim down the outliers
distWithBox('price')<jupyter_output><empty_output><jupyter_text>Label Encoding<jupyter_code>df[cat_cols]= df[cat_cols].apply(LabelEncoder().fit_transform)<jupyter_output><empty_output><jupyter_text>Feature Selection VarianceThreshold is a simple baseline approach to feature selection. It removes all features whose variance doesn’t meet some threshold. By default, it removes all zero-variance features, i.e. features that have the same value in all samples.so we can select using the threshold .8 * (1 - .8):<jupyter_code>df_X=df.loc[:, df.columns != 'price']
df_y=df['price']
from sklearn.feature_selection import VarianceThreshold
print('%s Number of features before VarianceThreshhold'%len(df_X.columns))

selector=VarianceThreshold(threshold=(.8*(1-.8)))
FeaturesTransformed=selector.fit_transform(df_X)

## print the support and shape of the transformed features
print(selector.get_support())

data=df_X[df_X.columns[selector.get_support(indices=True)]]
cols=data.columns
df_reduced=pd.DataFrame(FeaturesTransformed, columns=cols)
df_reduced.shape
df_X=df_reduced
from sklearn.ensemble import RandomForestRegressor

from yellowbrick.features.importances import FeatureImportances

fig = plt.figure(figsize=(20,20))
ax = fig.add_subplot()

viz = FeatureImportances(RandomForestRegressor(), ax=ax)
viz.fit(df_X, df_y)
viz.poof()
feature_importances = pd.DataFrame(viz.feature_importances_,
                                   index=df_X.columns,
                                   columns=['importance']).sort_values('importance', ascending=False)
## important features for Random Forest Regression
importants=feature_importances.index[feature_importances.importance!=0]
importants
from sklearn.ensemble import RandomForestRegressor
from sklearn.model_selection import train_test_split as split
X=df[['language_1', 'sales_rank', 'type', 'trackingSince', 'title',
       'stats_outOfStockPercentageInInterval', 'stats_outOfStockPercentage90',
       'stats_outOfStockPercentage30', 'stats_current', 'stats_avg180',
       'stats_avg90', 'stats_avg30', 'stats_avg', 'stats_atIntervalStart',
       'rootCategory', 'releaseDate', 'publisher', 'publicationDate',
       'productGroup', 'packageWidth', 'packageWeight', 'packageQuantity',
       'packageLength', 'packageHeight', 'numberOfPages', 'numberOfItems',
       'listedSince', 'lastUpdate', 'lastRatingUpdate', 'lastPriceChange',
       'isEligibleForSuperSaverShipping', 'categoryTree_4', 'categoryTree_3',
       'categoryTree_2', 'binding', 'author', 'productType']]
Y=df['price']
model=RandomForestRegressor()
X_train, X_test, Y_train, Y_test= split(X,Y,test_size=0.25, random_state=42)
model.fit(X_train,Y_train)
from sklearn.ensemble import GradientBoostingRegressor

from yellowbrick.features.importances import FeatureImportances

fig = plt.figure(figsize=(20,20))
ax = fig.add_subplot()

viz = FeatureImportances(GradientBoostingRegressor(), ax=ax)
viz.fit(df_X, df_y)
viz.poof()
feature_importances = pd.DataFrame(viz.feature_importances_,
                                   index=df_X.columns,
                                   columns=['importance']).sort_values('importance', ascending=False)
## important features for gradient boosting regression
importants=feature_importances.index[feature_importances.importance!=0]
importants
from sklearn.ensemble import GradientBoostingRegressor
from sklearn.metrics import r2_score
from sklearn.metrics import mean_squared_error
import math
from sklearn.model_selection import train_test_split as split
X=df[['language_1', 'sales_rank', 'type', 'trackingSince', 'title',
       'stats_outOfStockPercentageInInterval', 'stats_outOfStockPercentage90',
       'stats_outOfStockPercentage30', 'stats_current', 'stats_avg180',
       'stats_avg90', 'stats_avg30', 'stats_avg', 'stats_atIntervalStart',
       'rootCategory', 'releaseDate', 'publisher', 'publicationDate',
       'productGroup', 'packageWidth', 'packageWeight', 'packageQuantity',
       'packageLength', 'packageHeight', 'numberOfPages', 'numberOfItems',
       'listedSince', 'lastUpdate', 'lastRatingUpdate', 'lastPriceChange',
       'isEligibleForSuperSaverShipping', 'categoryTree_4']]
Y=df['price']
model=GradientBoostingRegressor()
X_train, X_test, Y_train, Y_test= split(X,Y,test_size=0.25, random_state=42)
model.fit(X_train,Y_train)
prediction1 = model.predict(X_test)
print("Accuracy Score", r2_score(Y_test, prediction1))
mse = mean_squared_error(Y_test, prediction1)
print("The root mean square value", math.sqrt(mse))




from sklearn.ensemble import RandomForestRegressor
rfg = RandomForestRegressor()
rfg.fit(X_train, Y_train)
prediction2 = rfg.predict(X_test)
print("Accuracy Score", r2_score(Y_test, prediction2))
mse2 = mean_squared_error(Y_test, prediction2)
print("The root mean square value", math.sqrt(mse2))

from sklearn.tree import DecisionTreeRegressor
dtg = DecisionTreeRegressor()
dtg.fit(X_train, Y_train)
prediction3 = dtg.predict(X_test)
print("Accuracy Score", r2_score(Y_test, prediction3))
mse3 = mean_squared_error(Y_test, prediction3)
print("The root mean square value", math.sqrt(mse3))
from sklearn.linear_model import LinearRegression
lr = LinearRegression()
lr.fit(X_train, Y_train)
prediction4 = lr.predict(X_test)
print("Accuracy Score", r2_score(Y_test, prediction4))
mse4 = mean_squared_error(Y_test, prediction4)
print("The root mean square value", math.sqrt(mse4))
from sklearn.neural_network import MLPRegressor
mlp = MLPRegressor()
mlp.fit(X_train, Y_train)
prediction5 = mlp.predict(X_test)
print("Accuracy Score", r2_score(Y_test, prediction5))
mse5 = mean_squared_error(Y_test, prediction5)
print("The root mean square value", math.sqrt(mse5))
from sklearn.linear_model import RidgeCV
rcv = RidgeCV()
rcv.fit(X_train, Y_train)
prediction6 = rcv.predict(X_test)
print("Accuracy Score", r2_score(Y_test, prediction6))
mse6 = mean_squared_error(Y_test, prediction6)
print("The root mean square value", math.sqrt(mse6))
from sklearn.linear_model import ElasticNet
lm_elastic = ElasticNet()
lm_elastic.fit(X_train, Y_train)
prediction7 = lm_elastic.predict(X_test)
print("Accuracy Score", r2_score(Y_test, prediction7))
mse7 = mean_squared_error(Y_test, prediction7)
print("The root mean square value", math.sqrt(mse7))
from sklearn.linear_model import LassoLars
llrs = LassoLars()
llrs.fit(X_train, Y_train)
prediction8 = llrs.predict(X_test)
print("Accuracy Score", r2_score(Y_test, prediction8))
mse8 = mean_squared_error(Y_test, prediction8)
print("The root mean square value", math.sqrt(mse8))
from sklearn.linear_model import Lasso
lass = Lasso()
lass.fit(X_train, Y_train)
prediction9 = lass.predict(X_test)
print("Accuracy Score", r2_score(Y_test, prediction9))
mse9 = mean_squared_error(Y_test, prediction9)
print("The root mean square value", math.sqrt(mse9))
from sklearn.linear_model import BayesianRidge
br = BayesianRidge()
br.fit(X_train, Y_train)
prediction10 = br.predict(X_test)
print("Accuracy Score", r2_score(Y_test, prediction10))
mse10 = mean_squared_error(Y_test, prediction10)
print("The root mean square value", math.sqrt(mse10))
from sklearn import linear_model
ransac = linear_model.RANSACRegressor()
ransac.fit(X_train, Y_train)
prediction11 = ransac.predict(X_test)
print("Accuracy Score", r2_score(Y_test, prediction11))
mse11 = mean_squared_error(Y_test, prediction11)
print("The root mean square value", math.sqrt(mse11))
from sklearn.svm import SVR
svr_rbf = SVR(kernel = 'rbf')
svr_rbf.fit(X_train, Y_train)
prediction12 = svr_rbf.predict(X_test)
print("Accuracy Score", r2_score(Y_test, prediction12))
mse12 = mean_squared_error(Y_test, prediction12)
print("The root mean square value", math.sqrt(mse12))
%matplotlib inline
from sklearn.linear_model import LassoLars
from sklearn.tree import DecisionTreeRegressor
from sklearn.ensemble import RandomForestRegressor
from sklearn.ensemble import GradientBoostingRegressor
from yellowbrick.classifier import ClassificationReport
from yellowbrick.classifier import ClassPredictionError
from yellowbrick.regressor import ResidualsPlot


regressors = {
    "Gradient Boost": GradientBoostingRegressor(),
    "Random Forest": RandomForestRegressor(),
    "Decision Tree": DecisionTreeRegressor(),
    "Lasso Lars": LassoLars()
    
    
}

for _, regressor in regressors.items():
    visualizer = ResidualsPlot(regressor)
    visualizer.fit(X_train, Y_train)
    visualizer.score(X_test, Y_test)
    visualizer.poof()


from sklearn.ensemble import GradientBoostingRegressor
from sklearn.model_selection import KFold
from yellowbrick.model_selection import CVScores

ind=df[['language_1', 'sales_rank', 'type', 'trackingSince', 'title',
       'stats_outOfStockPercentageInInterval', 'stats_outOfStockPercentage90',
       'stats_outOfStockPercentage30', 'stats_current', 'stats_avg180',
       'stats_avg90', 'stats_avg30', 'stats_avg', 'stats_atIntervalStart',
       'rootCategory', 'releaseDate', 'publisher', 'publicationDate',
       'productGroup', 'packageWidth', 'packageWeight', 'packageQuantity',
       'packageLength', 'packageHeight', 'numberOfPages', 'numberOfItems',
       'listedSince', 'lastUpdate', 'lastRatingUpdate', 'lastPriceChange',
       'isEligibleForSuperSaverShipping', 'categoryTree_4']].values
dep=df['price'].values

_, ax = plt.subplots()
cv = KFold(15)
oz = CVScores(
    GradientBoostingRegressor(), ax=ax, cv=cv, scoring = 'r2'
)

oz.fit(ind, dep)
oz.poof()

from sklearn.ensemble import RandomForestRegressor
from sklearn.model_selection import KFold
from yellowbrick.model_selection import CVScores

ind=df[['language_1', 'sales_rank', 'type', 'trackingSince', 'title',
       'stats_outOfStockPercentageInInterval', 'stats_outOfStockPercentage90',
       'stats_outOfStockPercentage30', 'stats_current', 'stats_avg180',
       'stats_avg90', 'stats_avg30', 'stats_avg', 'stats_atIntervalStart',
       'rootCategory', 'releaseDate', 'publisher', 'publicationDate',
       'productGroup', 'packageWidth', 'packageWeight', 'packageQuantity',
       'packageLength', 'packageHeight', 'numberOfPages', 'numberOfItems',
       'listedSince', 'lastUpdate', 'lastRatingUpdate', 'lastPriceChange',
       'isEligibleForSuperSaverShipping', 'categoryTree_4']].values
dep=df['price'].values

_, ax = plt.subplots()
cv = KFold(15)
oz = CVScores(
    RandomForestRegressor(), ax=ax, cv=cv, scoring='r2'
)

oz.fit(ind, dep)
oz.poof()

from sklearn.model_selection import GridSearchCV
from sklearn.ensemble import RandomForestRegressor
from sklearn.ensemble import GradientBoostingRegressor

GradientBoosting = GradientBoostingRegressor(random_state = 40)
alphas = [0.001, 0.01, 0.1, 0.5, 0.9]
sample_split = [2,3,4,5,6,7,8]
max_depth = [4,5,6,7,8,9]
learning_rate = [0.1, 0.3, 0.5, 0.7]
tuned_params = [{'alpha': alphas}, {'min_samples_split': sample_split}, {'max_depth': max_depth}, {'learning_rate':learning_rate}]
n_folds = 5

X=df[['language_1', 'sales_rank', 'type', 'trackingSince', 'title',
       'stats_outOfStockPercentageInInterval', 'stats_outOfStockPercentage90',
       'stats_outOfStockPercentage30', 'stats_current', 'stats_avg180',
       'stats_avg90', 'stats_avg30', 'stats_avg', 'stats_atIntervalStart',
       'rootCategory', 'releaseDate', 'publisher', 'publicationDate',
       'productGroup', 'packageWidth', 'packageWeight', 'packageQuantity',
       'packageLength', 'packageHeight', 'numberOfPages', 'numberOfItems',
       'listedSince', 'lastUpdate', 'lastRatingUpdate', 'lastPriceChange',
       'isEligibleForSuperSaverShipping', 'categoryTree_4']]
Y=df['price']

grid = GridSearchCV(
    GradientBoosting, tuned_params, cv=n_folds
)

grid.fit(X, Y)
print(grid.best_estimator_)



from sklearn.model_selection import GridSearchCV
from sklearn.ensemble import RandomForestRegressor
from sklearn.ensemble import GradientBoostingRegressor

RandomForest = RandomForestRegressor(random_state = 40)

estimators = [10,50,100]
sample_split = [2,3,4,5,6,7,8]
sample_leaf = [1,2,3,4,5]
max_depth = [4,5,6,7,8,9]
tuned_params = [{'n_estimators': estimators}, {'min_samples_split': sample_split}, {'min_samples_leaf': sample_leaf},{'max_leaf_nodes': max_depth}]
n_folds = 5

X=df[['language_1', 'sales_rank', 'type', 'trackingSince', 'title',
       'stats_outOfStockPercentageInInterval', 'stats_outOfStockPercentage90',
       'stats_outOfStockPercentage30', 'stats_current', 'stats_avg180',
       'stats_avg90', 'stats_avg30', 'stats_avg', 'stats_atIntervalStart',
       'rootCategory', 'releaseDate', 'publisher', 'publicationDate',
       'productGroup', 'packageWidth', 'packageWeight', 'packageQuantity',
       'packageLength', 'packageHeight', 'numberOfPages', 'numberOfItems',
       'listedSince', 'lastUpdate', 'lastRatingUpdate', 'lastPriceChange',
       'isEligibleForSuperSaverShipping', 'categoryTree_4']]
Y=df['price']

grid = GridSearchCV(
    RandomForest, tuned_params, cv=n_folds
)

grid.fit(X, Y)
print(grid.best_estimator_)



from sklearn.ensemble import GradientBoostingRegressor
model=GradientBoostingRegressor(alpha=0.9, criterion='friedman_mse', init=None,
             learning_rate=0.1, loss='ls', max_depth=7, max_features=None,
             max_leaf_nodes=None, min_impurity_decrease=0.0,
             min_impurity_split=None, min_samples_leaf=1,
             min_samples_split=2, min_weight_fraction_leaf=0.0,
             n_estimators=100, n_iter_no_change=None, presort='auto',
             random_state=40, subsample=1.0, tol=0.0001,
             validation_fraction=0.1, verbose=0, warm_start=False)
model.fit(X_train,Y_train)
prediction13 = model.predict(X_test)
print("Accuracy Score", r2_score(Y_test, prediction13))
mse = mean_squared_error(Y_test, prediction13)
print("The root mean square value", math.sqrt(mse))
from sklearn.ensemble import RandomForestRegressor
rfg = RandomForestRegressor(bootstrap=True, criterion='mse', max_depth=None,
           max_features='auto', max_leaf_nodes=None,
           min_impurity_decrease=0.0, min_impurity_split=None,
           min_samples_leaf=5, min_samples_split=2,
           min_weight_fraction_leaf=0.0, n_estimators=10, n_jobs=None,
           oob_score=False, random_state=40, verbose=0, warm_start=False)
rfg.fit(X_train, Y_train)
prediction14 = rfg.predict(X_test)
print("Accuracy Score", r2_score(Y_test, prediction14))
mse2 = mean_squared_error(Y_test, prediction14)
print("The root mean square value", math.sqrt(mse2))

from sklearn.preprocessing import StandardScaler
X=df[['language_1', 'sales_rank', 'type', 'trackingSince', 'title',
       'stats_outOfStockPercentageInInterval', 'stats_outOfStockPercentage90',
       'stats_outOfStockPercentage30', 'stats_current', 'stats_avg180',
       'stats_avg90', 'stats_avg30', 'stats_avg', 'stats_atIntervalStart',
       'rootCategory', 'releaseDate', 'publisher', 'publicationDate',
       'productGroup', 'packageWidth', 'packageWeight', 'packageQuantity',
       'packageLength', 'packageHeight', 'numberOfPages', 'numberOfItems',
       'listedSince', 'lastUpdate', 'lastRatingUpdate', 'lastPriceChange',
       'isEligibleForSuperSaverShipping', 'categoryTree_4', 'categoryTree_3',
       'categoryTree_2', 'binding', 'author', 'productType']]
names = X.columns
scale = StandardScaler()
X_df = scale.fit_transform(X)
X_df = pd.DataFrame(X_df, columns=names)
Y=df['price']
X_df.head(5)

from sklearn.model_selection import train_test_split
X_train, X_test, y_train, y_test = train_test_split(X_df,Y, test_size = 0.25, random_state = 42)
modelScale=GradientBoostingRegressor()
modelScale.fit(X_train,y_train)
predictionScale = modelScale.predict(X_test)
print("Accuracy Score", r2_score(y_test, predictionScale))
mseScale = mean_squared_error(y_test, predictionScale)
print("The root mean square value", math.sqrt(mseScale))

from sklearn.ensemble import RandomForestRegressor
rfgScale = RandomForestRegressor()
rfgScale.fit(X_train, y_train)
predictionrfg = rfgScale.predict(X_test)
print("Accuracy Score", r2_score(y_test, predictionrfg))
mseSca = mean_squared_error(y_test, predictionrfg)
print("The root mean square value", math.sqrt(mseSca))

from sklearn.model_selection import GridSearchCV
from sklearn.ensemble import RandomForestRegressor
from sklearn.ensemble import GradientBoostingRegressor

GradientBoosting = GradientBoostingRegressor(random_state = 40)
alphas = [0.001, 0.01, 0.1, 0.5, 0.9]
sample_split = [2,3,4,5,6,7,8]
max_depth = [4,5,6,7,8,9]
learning_rate = [0.1, 0.3, 0.5, 0.7]
tuned_params = [{'alpha': alphas}, {'min_samples_split': sample_split}, {'max_depth': max_depth}, {'learning_rate':learning_rate}]
n_folds = 5


grid = GridSearchCV(
    GradientBoosting, tuned_params, cv=n_folds
)

grid.fit(X_df, Y)
print(grid.best_estimator_)



from sklearn.ensemble import GradientBoostingRegressor
modelS=GradientBoostingRegressor(alpha=0.9, criterion='friedman_mse', init=None,
             learning_rate=0.1, loss='ls', max_depth=6, max_features=None,
             max_leaf_nodes=None, min_impurity_decrease=0.0,
             min_impurity_split=None, min_samples_leaf=1,
             min_samples_split=2, min_weight_fraction_leaf=0.0,
             n_estimators=100, n_iter_no_change=None, presort='auto',
             random_state=40, subsample=1.0, tol=0.0001,
             validation_fraction=0.1, verbose=0, warm_start=False)
modelS.fit(X_train,Y_train)
predictionS1 = modelS.predict(X_test)
print("Accuracy Score", r2_score(Y_test, predictionS1))
mse = mean_squared_error(Y_test, predictionS1)
print("The root mean square value", math.sqrt(mse))
from sklearn.ensemble import RandomForestRegressor
rfg = RandomForestRegressor(bootstrap=True, criterion='mse', max_depth=None,
           max_features='auto', max_leaf_nodes=None,
           min_impurity_decrease=0.0, min_impurity_split=None,
           min_samples_leaf=5, min_samples_split=2,
           min_weight_fraction_leaf=0.0, n_estimators=10, n_jobs=None,
           oob_score=False, random_state=40, verbose=0, warm_start=False)
rfg.fit(X_train, Y_train)
prediction14 = rfg.predict(X_test)
print("Accuracy Score", r2_score(Y_test, prediction14))
mse2 = mean_squared_error(Y_test, prediction14)
print("The root mean square value", math.sqrt(mse2))

from sklearn.metrics import r2_score
from sklearn.metrics import mean_squared_error

def ModelScores(data,target):
    X = data
    Y=target
    from sklearn.model_selection import train_test_split as split
    X_train, X_test, Y_train, Y_test= split(X_df,Y,test_size=0.25, random_state=40)
    
    from sklearn.ensemble import GradientBoostingRegressor
    from sklearn.ensemble import RandomForestRegressor
    from sklearn.tree import DecisionTreeRegressor
    from sklearn.linear_model import LinearRegression
    from sklearn.neural_network import MLPRegressor
    from sklearn.linear_model import RidgeCV
    from sklearn.linear_model import LassoLars
    from sklearn.linear_model import Lasso
    from sklearn.linear_model import ElasticNet
    from sklearn.svm import SVR
    from sklearn.linear_model import BayesianRidge
    from sklearn.linear_model import RANSACRegressor

    models={'Gradient Boost': GradientBoostingRegressor(),
            'Random Forest': RandomForestRegressor(),
            'Decision Tree': DecisionTreeRegressor(),
            'Linear Regression': LinearRegression(),
            'MLP': MLPRegressor(),
            'Ridge CV': RidgeCV(),
            'LassoLars':LassoLars(),
            'Lasso':Lasso(),
            'Elastic Search': ElasticNet(),
            'Bayesian Ridge':BayesianRidge(),
            'Ransac':RANSACRegressor()      
           }
    for name,model in models.items():
        mdl=model
        mdl.fit(X_train, Y_train)
        prediction = mdl.predict(X_test)
        print(name)
        print("Accuracy Score", r2_score(Y_test, prediction))
        mse3 = mean_squared_error(Y_test, prediction)
        print("The root mean square value", math.sqrt(mse3))
ModelScores(X_df,Y)
from sklearn.decomposition import PCA
pca = PCA()
X = pca.fit_transform(X_df)
exp_variance = pca.explained_variance_ratio_
print(exp_variance)
X_train, X_test, Y_train, Y_test = train_test_split(X,Y, test_size = 0.25, random_state = 40)
from sklearn.ensemble import GradientBoostingRegressor
modelS=GradientBoostingRegressor(alpha=0.9, criterion='friedman_mse', init=None,
             learning_rate=0.1, loss='ls', max_depth=6, max_features=None,
             max_leaf_nodes=None, min_impurity_decrease=0.0,
             min_impurity_split=None, min_samples_leaf=1,
             min_samples_split=2, min_weight_fraction_leaf=0.0,
             n_estimators=100, n_iter_no_change=None, presort='auto',
             random_state=40, subsample=1.0, tol=0.0001,
             validation_fraction=0.1, verbose=0, warm_start=False)
modelS.fit(X_train,Y_train)
predictionS1 = modelS.predict(X_test)
print("Accuracy Score", r2_score(Y_test, predictionS1))
mse = mean_squared_error(Y_test, predictionS1)
print("The root mean square value", math.sqrt(mse))<jupyter_output>[2.64089811e-01 1.33347693e-01 8.74779248e-02 3.71342098e-02
 3.00158716e-02 2.85265644e-02 2.79891570e-02 2.71173766e-02
 2.69522094e-02 2.68891066e-02 2.64068270e-02 2.61079369e-02
 2.56749961e-02 2.55578893e-02 2.48376914e-02 2.34310623e-02
 2.27857499e-02 2.13663066e-02 1.97996837e-02 1.83007131e-02
 1.66211202e-02 1.42005906e-02 1.11698379e-02 9.01006757e-03
 6.22404654e-03 5.30523640e-03 3.43552165e-03 2.35637720e-03
 1.96343025e-03 1.76120747e-03 1.36246293e-03 1.09939634e-03
 4.97719030e-04 4.20423923e-04 3.90835138e-04 3.20459894e-04
 5.24860289e-05]
Accuracy Score 0.9345579211268843
The root mean square value 192.070058627823
{'fraction_non_alphanumeric': 0.12491766131794588, 'fraction_numerical': 0.030853950939319686, 'mean_word_length': 5.961482024944974, 'pattern_counts': {'":': 5, '<': 78, '<?xml version=': 0, '>': 79, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 6, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '19', 'dir': 'jupyter-structured-clean-dedup', 'id': '96706', 'license': 'MIT', 'n_tokens_mistral': 13365, 'n_tokens_neox': 12431, 'n_words': 2721, 'path': '.ipynb_checkpoints/Standardized_Gridsearch_Notebook-checkpoint.ipynb', 'repo_name': 'suchitsharma/Booksville', 'size': '37953'}